Moving away from cron jobs to some workflow manager

[Rabin Yasharzadehe]

never heard of it,
but from reading the manual and the 10minute presentation ,
it's seems like it is more suitable for data crunching, where you have a
pool
of compute resources and you submit jobs to it.

my case is a bit different, where I have many jobs which need to run
(orchestrated) on there own hosts
with a specific environment and setup.


--
Rabin


On Tue, 19 Jun 2018 at 09:10, Marc Volovic <marcvolovic at icloud.com> wrote:

> Why not a minimal deploy of SGE - which would also allow you to make
> multi-executor?
>
> https://arc.liv.ac.uk/trac/SGE
>
> —mav
> Marc Volovic
> marcvolovic at me.com
>
>
>
> > On 19 Jun 2018, at 9:06, Rabin Yasharzadehe <rabin at rabin.io> wrote:
> >
> > Hi all,
> >
> > I need some advice, currently I have a huge cron file which schedules
> tasks one after anther, and each task is position precisely (with some room
> for error) to start after it predecessor.
> >
> > So if one job start at 00:00 and it will go and fetch some files and it
> takes 3minutes
> > the next job will be after start right after at ~00:05
> > and so on
> >
> > the problem is that if one job fails, all other jobs which are depend on
> him will fail as well, and then I get a shitload of alerts, and the worst
> part is that if i have to manually start a batch process I need to go to
> each machine and manually start each job in the right order,
> >
> > I was looking to resolve this problem with a tool which can manage this
> "pipe line"
> > and I cam across several tools like Luigi and (apache-)AirFlow, I
> started with Luigi but It didn't look
> > right for the job, and then I tried airflow, but was not able to make it
> to work, the jobs queue never executed. =(
> >
> > Has any one have experience with airflow, or other tool like it which
> they can recommend ?
> > My needs are to be able to execute my CURRENT shell/python/php scripts
> and build the dependency between them, and I perfer the option for remote
> exec so that I will have central
> > place to manage and monitor all work flow whichs are executed on several
> nodes.
> >
> > Thanks in advance,
> > Rabin
> > _______________________________________________
> > Linux-il mailing list
> > Linux-il at cs.huji.ac.il
> > http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20180619/c507c058/attachment-0001.html>