Moving away from cron jobs to some workflow manager

[Dimid Duchovny]

Hi Rabin,


I'm far from being a linux expert, but isn't dependency between services
handled by systemd?
E.g. https://wiki.archlinux.org/index.php/Systemd/Timers

HTH

2018-06-19 12:12 GMT+03:00 Moish <moish at mln.co.il>:

> Try GNUbatch.
>
>
> On June 19, 2018 9:42:35 AM GMT+03:00, Omer Zak <w1 at zak.co.il> wrote:
>
>> For dependency management, you may want to use 'make' or modern
>> equivalents ('ant', 'gradle', etc.).
>> For controlling remote nodes, 'ansible' may be able to do the work.
>>
>> --- Omer Zak
>>
>>
>> On Tue, 2018-06-19 at 09:06 +0300, Rabin Yasharzadehe wrote:
>>
>>>  Hi all,
>>>
>>>  I need some advice, currently I have a huge cron file which schedules
>>>  tasks one after anther, and each task is position precisely (with
>>>  some room for error) to start after it predecessor.
>>>
>>>  So if one job start at 00:00 and it will go and fetch some files and
>>>  it takes 3minutes
>>>  the next job will be after start right after at ~00:05
>>>  and so on
>>>
>>>  the problem is that if one job fails, all other jobs which are depend
>>>  on him will fail as well, and then I get a shitload of alerts, and
>>>  the worst part is that if i have to manually start a batch process I
>>>  need to go to each machine and manually start each job in the right
>>>  order,
>>>
>>>  I was looking to resolve this problem with a tool which can manage
>>>  this "pipe line"
>>>  and I cam across several tools like Luigi and (apache-)AirFlow, I
>>>  started with Luigi but It didn't look
>>>  right for the job, and then I tried airflow, but was not able to make
>>>  it to work, the jobs queue never executed. =(
>>>
>>>  Has any one have experience with airflow, or other tool like it which
>>>  they can recommend ?
>>>  My needs are to be able to execute my CURRENT shell/python/php
>>>  scripts and build the dependency between them, and I perfer the
>>>  option for remote exec so that I will have central
>>>  place to manage and monitor all work flow whichs are executed on
>>>  several nodes.
>>>
>> --
>> More proof the End of the World has started. Just saw this online:
>> I think it's beginning! Ten minutes ago there was a group of people
>> waiting at the bus stop outside my house. Now, they're all gone!
>> My own blog is at https://tddpirate.zak.co.il/
>>
>> My opinions, as expressed in this E-mail message, are mine alone.
>> They do not represent the official policy of any organization with which
>> I may be affiliated in any way.
>> WARNING TO SPAMMERS:  at https://www.zak.co.il/spamwarning.html
>>
>> ------------------------------
>>
>> Linux-il mailing list
>> Linux-il at cs.huji.ac.il
>> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>>
>>
> _______________________________________________
> Linux-il mailing list
> Linux-il at cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20180619/dc5d18f3/attachment-0001.html>