Moving away from cron jobs to some workflow manager

[Omer Zak]

1. Execution time limits:

Ansible has async with polling intervals. I did not research for
methods to kill hung tasks.
https://docs.ansible.com/ansible/latest/user_guide/playbooks_async.html

2. Dashboard-like functionality

According to:
https://www.reddit.com/r/ansible/comments/5ksphc/best_web_gui_for_run_a
nsible_playbooks/

There are the following options:
- ansible-tower
- remote-task-executor (did not look into it)
- nci-ansible-ui
- Jenkins (normally used for CI/CD setups)

In addition to the above, you may want to look into Ansible
alternatives:
- Puppet
- Chef
- SaltStack
A quick Google search yielded:
https://www.intigua.com/blog/puppet-vs.-chef-vs.-ansible-vs.-saltstack




On Tue, 2018-06-19 at 12:32 +0300, Rabin Yasharzadehe wrote:
> systemd is a complete different tool, which was not designed for this
> kinda purpose.
> (maybe in the future it will grow to be something like that ;-) )
> 
> I'm looking for something a bit more sophisticated then "go to this
> machine" and "run this script" and "expect this result"
> i like to define execution time limits (finish in 3 minute) and maybe
> some grace time (can go up to 5 minute) and have the orchestrator
> monitor the process and have a nice dashboard where i can see every
> thing from above (this is why Airflow was looked so appealing, but
> the installation process and the documentation are still lagging
> behind).
> 
> 
> --
> Rabin
> 
> 
> On Tue, 19 Jun 2018 at 12:26, Dimid Duchovny <dimidd at gmail.com>
> wrote:
> > Hi Rabin,
> > 
> > 
> > I'm far from being a linux expert, but isn't dependency between
> > services handled by systemd?
> > E.g. https://wiki.archlinux.org/index.php/Systemd/Timers
> > 
> > HTH
> > 
> > 2018-06-19 12:12 GMT+03:00 Moish <moish at mln.co.il>:
> > > Try GNUbatch. 
> > > 
> > > 
> > > On June 19, 2018 9:42:35 AM GMT+03:00, Omer Zak <w1 at zak.co.il>
> > > wrote:
> > > > For dependency management, you may want to use 'make' or modern
> > > > equivalents ('ant', 'gradle', etc.).
> > > > For controlling remote nodes, 'ansible' may be able to do the
> > > > work.
> > > > 
> > > > --- Omer Zak
> > > > 
> > > > 
> > > > On Tue, 2018-06-19 at 09:06 +0300, Rabin Yasharzadehe wrote:
> > > > >  Hi all, 
> > > > >  
> > > > >  I need some advice, currently I have a huge cron file which
> > > > > schedules
> > > > >  tasks one after anther, and each task is position precisely
> > > > > (with
> > > > >  some room for error) to start after it predecessor.
> > > > >  
> > > > >  So if one job start at 00:00 and it will go and fetch some
> > > > > files and
> > > > >  it takes 3minutes
> > > > >  the next job will be after start right after at ~00:05 
> > > > >  and so on 
> > > > >  
> > > > >  the problem is that if one job fails, all other jobs which
> > > > > are depend
> > > > >  on him will fail as well, and then I get a shitload of
> > > > > alerts, and
> > > > >  the worst part is that if i have to manually start a batch
> > > > > process I
> > > > >  need to go to each machine and manually start each job in
> > > > > the right
> > > > >  order,
> > > > >  
> > > > >  I was looking to resolve this problem with a tool which can
> > > > > manage
> > > > >  this "pipe line" 
> > > > >  and I cam across several tools like Luigi and (apache-
> > > > > )AirFlow, I
> > > > >  started with Luigi but It didn't look
> > > > >  right for the job, and then I tried airflow, but was not
> > > > > able to make
> > > > >  it to work, the jobs queue never executed. =(
> > > > >  
> > > > >  Has any one have experience with airflow, or other tool like
> > > > > it which
> > > > >  they can recommend ? 
> > > > >  My needs are to be able to execute my CURRENT
> > > > > shell/python/php
> > > > >  scripts and build the dependency between them, and I perfer
> > > > > the
> > > > >  option for remote exec so that I will have central 
> > > > >  place to manage and monitor all work flow whichs are
> > > > > executed on
> > > > >  several nodes.
-- 
What happens if one mixes together evolution with time travel to the
past?  See: https://www.zak.co.il/ideas/stuff/opinions/eng/evol_tm.html
My own blog is at https://tddpirate.zak.co.il/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with
which
I may be affiliated in any way.
WARNING TO SPAMMERS:  at https://www.zak.co.il/spamwarning.html