Moving away from cron jobs to some workflow manager

[Moish]

Try GNUbatch. 


On June 19, 2018 9:42:35 AM GMT+03:00, Omer Zak <w1 at zak.co.il> wrote:
>For dependency management, you may want to use 'make' or modern
>equivalents ('ant', 'gradle', etc.).
>For controlling remote nodes, 'ansible' may be able to do the work.
>
>--- Omer Zak
>
>
>On Tue, 2018-06-19 at 09:06 +0300, Rabin Yasharzadehe wrote:
>> Hi all, 
>> 
>> I need some advice, currently I have a huge cron file which schedules
>> tasks one after anther, and each task is position precisely (with
>> some room for error) to start after it predecessor.
>> 
>> So if one job start at 00:00 and it will go and fetch some files and
>> it takes 3minutes
>> the next job will be after start right after at ~00:05 
>> and so on 
>> 
>> the problem is that if one job fails, all other jobs which are depend
>> on him will fail as well, and then I get a shitload of alerts, and
>> the worst part is that if i have to manually start a batch process I
>> need to go to each machine and manually start each job in the right
>> order,
>> 
>> I was looking to resolve this problem with a tool which can manage
>> this "pipe line" 
>> and I cam across several tools like Luigi and (apache-)AirFlow, I
>> started with Luigi but It didn't look
>> right for the job, and then I tried airflow, but was not able to make
>> it to work, the jobs queue never executed. =(
>> 
>> Has any one have experience with airflow, or other tool like it which
>> they can recommend ? 
>> My needs are to be able to execute my CURRENT shell/python/php
>> scripts and build the dependency between them, and I perfer the
>> option for remote exec so that I will have central 
>> place to manage and monitor all work flow whichs are executed on
>> several nodes.
>-- 
>More proof the End of the World has started. Just saw this online:
>I think it's beginning! Ten minutes ago there was a group of people
>waiting at the bus stop outside my house. Now, they're all gone!
>My own blog is at https://tddpirate.zak.co.il/
>
>My opinions, as expressed in this E-mail message, are mine alone.
>They do not represent the official policy of any organization with
>which
>I may be affiliated in any way.
>WARNING TO SPAMMERS:  at https://www.zak.co.il/spamwarning.html
>
>_______________________________________________
>Linux-il mailing list
>Linux-il at cs.huji.ac.il
>http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20180619/c81fd751/attachment.html>