<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
try GNUbatch<br>
<br>
<div class="moz-cite-prefix">On 19/06/2018 09:06, Rabin Yasharzadehe
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CABKuJEDcM6a7LBVDWV7Q3TMQg9yaxMuJbRU4MPzc_QFymfzDFQ@mail.gmail.com">
<div dir="ltr">
<div class="gmail_default" style="font-family:tahoma,sans-serif">Hi
all, <br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">I
need some advice, currently I have a huge <span class=""
style="" id=":zs9.1" tabindex="-1">cron</span> file which
schedules tasks one after anther, and each task is position
precisely (with some room for error) to start after it
predecessor.</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">So
if one job start at 00:00 and it will go and fetch some files
and it takes 3minutes</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">the
next job will be after start right after at ~00:05 <br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">and
so on <br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">the
problem is that if one job fails, all other jobs which are
depend on him will fail as well, and then I get a shitload of
alerts, and the worst part is that if i have to manually start
a batch process I need to go to each machine and manually
start each job in the right order,</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">I
was looking to resolve this problem with a tool which can
manage this "pipe line" <br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">and
I cam across several tools like Luigi and (<span class=""
style="" id=":zs9.3" tabindex="-1">apache</span>-)<span
class="" style="" id=":zs9.4" tabindex="-1">AirFlow</span>,
I started with Luigi but It didn't look</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">right
for the job, and then I tried airflow, but was not able to
make it to work, the jobs queue never executed. =(</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">Has
any one have experience with airflow, or other tool like it
which they can recommend ? <br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">My
needs are to be able to execute my CURRENT shell/python/php
scripts and build the dependency between them, and I perfer
the option for remote exec so that I will have central <br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">place
to manage and monitor all work flow whichs are executed on
several nodes.<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">Thanks
in advance,<br>
</div>
<div>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr"><span style="font-family:tahoma,sans-serif">Rabin</span></div>
</div>
</div>
</div>
<!--'"--><br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Linux-il mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Linux-il@cs.huji.ac.il">Linux-il@cs.huji.ac.il</a>
<a class="moz-txt-link-freetext" href="http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il">http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Moish
0546484411</pre>
</body>
</html>