Moving away from cron jobs to some workflow manager

[Lior Okman]

Hi,

You could try something like Concourse ( https://concourse-ci.org/ ). It
allows you to define a pipeline which is comprised of jobs and the order in
which they should be invoked.


--
Lior

On Tue, Jun 19, 2018 at 9:06 AM, Rabin Yasharzadehe <rabin at rabin.io> wrote:

> Hi all,
>
> I need some advice, currently I have a huge cron file which schedules
> tasks one after anther, and each task is position precisely (with some room
> for error) to start after it predecessor.
>
> So if one job start at 00:00 and it will go and fetch some files and it
> takes 3minutes
> the next job will be after start right after at ~00:05
> and so on
>
> the problem is that if one job fails, all other jobs which are depend on
> him will fail as well, and then I get a shitload of alerts, and the worst
> part is that if i have to manually start a batch process I need to go to
> each machine and manually start each job in the right order,
>
> I was looking to resolve this problem with a tool which can manage this
> "pipe line"
> and I cam across several tools like Luigi and (apache-)AirFlow, I started
> with Luigi but It didn't look
> right for the job, and then I tried airflow, but was not able to make it
> to work, the jobs queue never executed. =(
>
> Has any one have experience with airflow, or other tool like it which they
> can recommend ?
> My needs are to be able to execute my CURRENT shell/python/php scripts and
> build the dependency between them, and I perfer the option for remote exec
> so that I will have central
> place to manage and monitor all work flow whichs are executed on several
> nodes.
>
> Thanks in advance,
> Rabin
>
> _______________________________________________
> Linux-il mailing list
> Linux-il at cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20180620/68b5a136/attachment-0001.html>