Moving away from cron jobs to some workflow manager

[Omer Zak]

For dependency management, you may want to use 'make' or modern
equivalents ('ant', 'gradle', etc.).
For controlling remote nodes, 'ansible' may be able to do the work.

--- Omer Zak


On Tue, 2018-06-19 at 09:06 +0300, Rabin Yasharzadehe wrote:
> Hi all, 
> 
> I need some advice, currently I have a huge cron file which schedules
> tasks one after anther, and each task is position precisely (with
> some room for error) to start after it predecessor.
> 
> So if one job start at 00:00 and it will go and fetch some files and
> it takes 3minutes
> the next job will be after start right after at ~00:05 
> and so on 
> 
> the problem is that if one job fails, all other jobs which are depend
> on him will fail as well, and then I get a shitload of alerts, and
> the worst part is that if i have to manually start a batch process I
> need to go to each machine and manually start each job in the right
> order,
> 
> I was looking to resolve this problem with a tool which can manage
> this "pipe line" 
> and I cam across several tools like Luigi and (apache-)AirFlow, I
> started with Luigi but It didn't look
> right for the job, and then I tried airflow, but was not able to make
> it to work, the jobs queue never executed. =(
> 
> Has any one have experience with airflow, or other tool like it which
> they can recommend ? 
> My needs are to be able to execute my CURRENT shell/python/php
> scripts and build the dependency between them, and I perfer the
> option for remote exec so that I will have central 
> place to manage and monitor all work flow whichs are executed on
> several nodes.
-- 
More proof the End of the World has started. Just saw this online:
I think it's beginning! Ten minutes ago there was a group of people
waiting at the bus stop outside my house. Now, they're all gone!
My own blog is at https://tddpirate.zak.co.il/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS:  at https://www.zak.co.il/spamwarning.html