Moving away from cron jobs to some workflow manager

[Rabin Yasharzadehe]

Hi all,

I need some advice, currently I have a huge cron file which schedules tasks
one after anther, and each task is position precisely (with some room for
error) to start after it predecessor.

So if one job start at 00:00 and it will go and fetch some files and it
takes 3minutes
the next job will be after start right after at ~00:05
and so on

the problem is that if one job fails, all other jobs which are depend on
him will fail as well, and then I get a shitload of alerts, and the worst
part is that if i have to manually start a batch process I need to go to
each machine and manually start each job in the right order,

I was looking to resolve this problem with a tool which can manage this
"pipe line"
and I cam across several tools like Luigi and (apache-)AirFlow, I started
with Luigi but It didn't look
right for the job, and then I tried airflow, but was not able to make it to
work, the jobs queue never executed. =(

Has any one have experience with airflow, or other tool like it which they
can recommend ?
My needs are to be able to execute my CURRENT shell/python/php scripts and
build the dependency between them, and I perfer the option for remote exec
so that I will have central
place to manage and monitor all work flow whichs are executed on several
nodes.

Thanks in advance,
Rabin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20180619/f7cc558a/attachment.html>