Moving away from cron jobs to some workflow manager

[Ari Becker]

Hi Rabin,

Did you consider using Jenkins? It may be a little heavyweight, but it
should be relatively easy to set up and configure. You can use the same
scripts you're using today, the ability to state which jobs run on which
nodes, set up dependencies between them, set timeouts, set cron triggers to
start the initial job... seems to answer your requirements.

-Ari

On Tue, Jun 19, 2018, 09:07 Rabin Yasharzadehe <rabin at rabin.io> wrote:

> Hi all,
>
> I need some advice, currently I have a huge cron file which schedules
> tasks one after anther, and each task is position precisely (with some room
> for error) to start after it predecessor.
>
> So if one job start at 00:00 and it will go and fetch some files and it
> takes 3minutes
> the next job will be after start right after at ~00:05
> and so on
>
> the problem is that if one job fails, all other jobs which are depend on
> him will fail as well, and then I get a shitload of alerts, and the worst
> part is that if i have to manually start a batch process I need to go to
> each machine and manually start each job in the right order,
>
> I was looking to resolve this problem with a tool which can manage this
> "pipe line"
> and I cam across several tools like Luigi and (apache-)AirFlow, I started
> with Luigi but It didn't look
> right for the job, and then I tried airflow, but was not able to make it
> to work, the jobs queue never executed. =(
>
> Has any one have experience with airflow, or other tool like it which they
> can recommend ?
> My needs are to be able to execute my CURRENT shell/python/php scripts and
> build the dependency between them, and I perfer the option for remote exec
> so that I will have central
> place to manage and monitor all work flow whichs are executed on several
> nodes.
>
> Thanks in advance,
> Rabin
> _______________________________________________
> Linux-il mailing list
> Linux-il at cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20180619/4ce74716/attachment.html>