Moving away from cron jobs to some workflow manager

[Rabin Yasharzadehe]

systemd is a complete different tool, which was not designed for this kinda
purpose.
(maybe in the future it will grow to be something like that ;-) )

I'm looking for something a bit more sophisticated then "go to this
machine" and "run this script" and "expect this result"
i like to define execution time limits (finish in 3 minute) and maybe some
grace time (can go up to 5 minute) and have the orchestrator monitor the
process and have a nice dashboard where i can see every thing from above
(this is why Airflow was looked so appealing, but the installation process
and the documentation are still lagging behind).


--
Rabin


On Tue, 19 Jun 2018 at 12:26, Dimid Duchovny <dimidd at gmail.com> wrote:

> Hi Rabin,
>
>
> I'm far from being a linux expert, but isn't dependency between services
> handled by systemd?
> E.g. https://wiki.archlinux.org/index.php/Systemd/Timers
>
> HTH
>
> 2018-06-19 12:12 GMT+03:00 Moish <moish at mln.co.il>:
>
>> Try GNUbatch.
>>
>>
>> On June 19, 2018 9:42:35 AM GMT+03:00, Omer Zak <w1 at zak.co.il> wrote:
>>
>>> For dependency management, you may want to use 'make' or modern
>>> equivalents ('ant', 'gradle', etc.).
>>> For controlling remote nodes, 'ansible' may be able to do the work.
>>>
>>> --- Omer Zak
>>>
>>>
>>> On Tue, 2018-06-19 at 09:06 +0300, Rabin Yasharzadehe wrote:
>>>
>>>>  Hi all,
>>>>
>>>>  I need some advice, currently I have a huge cron file which schedules
>>>>  tasks one after anther, and each task is position precisely (with
>>>>  some room for error) to start after it predecessor.
>>>>
>>>>  So if one job start at 00:00 and it will go and fetch some files and
>>>>  it takes 3minutes
>>>>  the next job will be after start right after at ~00:05
>>>>  and so on
>>>>
>>>>  the problem is that if one job fails, all other jobs which are depend
>>>>  on him will fail as well, and then I get a shitload of alerts, and
>>>>  the worst part is that if i have to manually start a batch process I
>>>>  need to go to each machine and manually start each job in the right
>>>>  order,
>>>>
>>>>  I was looking to resolve this problem with a tool which can manage
>>>>  this "pipe line"
>>>>  and I cam across several tools like Luigi and (apache-)AirFlow, I
>>>>  started with Luigi but It didn't look
>>>>  right for the job, and then I tried airflow, but was not able to make
>>>>  it to work, the jobs queue never executed. =(
>>>>
>>>>  Has any one have experience with airflow, or other tool like it which
>>>>  they can recommend ?
>>>>  My needs are to be able to execute my CURRENT shell/python/php
>>>>  scripts and build the dependency between them, and I perfer the
>>>>  option for remote exec so that I will have central
>>>>  place to manage and monitor all work flow whichs are executed on
>>>>  several nodes.
>>>>
>>> --
>>> More proof the End of the World has started. Just saw this online:
>>> I think it's beginning! Ten minutes ago there was a group of people
>>> waiting at the bus stop outside my house. Now, they're all gone!
>>> My own blog is at https://tddpirate.zak.co.il/
>>>
>>> My opinions, as expressed in this E-mail message, are mine alone.
>>> They do not represent the official policy of any organization with which
>>> I may be affiliated in any way.
>>> WARNING TO SPAMMERS:  at https://www.zak.co.il/spamwarning.html
>>>
>>> ------------------------------
>>>
>>> Linux-il mailing list
>>> Linux-il at cs.huji.ac.il
>>> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>>>
>>>
>> _______________________________________________
>> Linux-il mailing list
>> Linux-il at cs.huji.ac.il
>> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>>
>>
> _______________________________________________
> Linux-il mailing list
> Linux-il at cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20180619/741d47df/attachment.html>