From geoff at QuiteLikely.com Thu Jun 7 12:22:05 2018 From: geoff at QuiteLikely.com (Geoff Shang) Date: Thu, 7 Jun 2018 12:22:05 +0300 (IDT) Subject: VDSL modem recommendation Message-ID: Hi everyone, I'm in the market for a VDSL modem, and after looking at various options, I'm unsure which I should buy. So I thought I'd ask here for help/recommendations. Last year, I posted to the list about ISP recommendations. As part of that discussion, I was sent a link to a site listing modems compatible with Israeli service. Until then, I didn't know that buying your own device was even an option here. As previously mentioned, ISP-supplied modems/routers are an issue for me as a blind person, as many seem to have visual CAPTCHAs which make it difficult/impossible to log into the interface. And I don't think my choice of ISP should be determined by whether or not I can log into their equipment, I'd prefer to choose based on service (like everyone else). So I'm now looking to buy a modem. I have a Ubiquiti Edge Router which I intend to use as the router, so routing/wi-fi component is not important. I started looking at options. The things I thought were important were: * Good/best modem performance * Support for all/most relevant modem connectivity standards * Support for open firmware When I started looking, I was looking for modem-only devices. The closest I could find were modem/routers with only one ethernet port. The device that seemed to fit the bill was the Netgear DM200. It has 1 port (albeit 100 MBPS but VDSL connections aren't exceeding that at the moment anyway), and it appears to be supported by OpenWRT. But then I started seeing some bad reviews. In particular, with the help of Google Translate, I saw some unhappy people on hometheatre.co.il, which I considered important as I will be using it here. The othere 1-port device I found was the Draytech Vigor 130. This is more expensive, but I don't mind about that if the device is good. It seemed to generally get better performance reviews than the Netgear. However, it's not supported by OpenWRT as far as I can see. There were a couple of other things that I also saw time and again. Some people say that Broadcom chips are better than others , though I've also seem comment that compatibility with the upstream chipset is the most important. Also, I saw references to things like vectoring and g.imp, things I know nothing about but which sound important, and g.fast which seems like it's going to be the next big thing but which nothing much seems to support yet. People also pointed out that there's really no difference between using a 1-port router in bridged mode and using a 4-port router in bridged mode. As neither of the above had the Broadcom chips, and since I had seen quite a few recommendations for the TP Link 9970/9977, I looked into these. The TP Link 9970 and 9977 (9977 is apparently newer) both seem to have the Broadcom chips and support g.imp. But neither seem to support open firmware from what I can see. Otheres like the 9980 do support it, but they don't have Broadcom chips. Some people make the argument that if you're using a device in bridged mode, open firmware is less important, but I'm not sure if I buy that argument. So now I'm a bit confused. I'm starting to lose track of which devices support what, and I don't really know which of these factors are specifically important here in Israel and which are more or less important generally. Since I believe that most problems have been solved by others before me, I thought I'd ask here. To repeat, I'm after a good modem. I'm not so concerned about price and I really don't care about the routing and wi-fi specs as I don't plan on using them. Any advice anyone can provide would be most helpful. Thanks, Geoff. From idokan at gmail.com Thu Jun 7 19:14:16 2018 From: idokan at gmail.com (ik) Date: Thu, 7 Jun 2018 19:14:16 +0300 Subject: VDSL modem recommendation In-Reply-To: References: Message-ID: I just bought a router actually named VR600 of TP-LINK, and it works great even on vectoring for example. I actually just activated it, and this email is using that router. I like the UI, and the features, but no WRT support for example. But then again, it's a router with a A/VDSL modem, and not just a modem. Ido On Thu, Jun 7, 2018 at 12:22 PM, Geoff Shang wrote: > Hi everyone, > > I'm in the market for a VDSL modem, and after looking at various options, > I'm unsure which I should buy. So I thought I'd ask here for > help/recommendations. > > Last year, I posted to the list about ISP recommendations. As part of > that discussion, I was sent a link to a site listing modems compatible with > Israeli service. Until then, I didn't know that buying your own device was > even an option here. > > As previously mentioned, ISP-supplied modems/routers are an issue for me > as a blind person, as many seem to have visual CAPTCHAs which make it > difficult/impossible to log into the interface. And I don't think my > choice of ISP should be determined by whether or not I can log into their > equipment, I'd prefer to choose based on service (like everyone else). > > So I'm now looking to buy a modem. I have a Ubiquiti Edge Router which I > intend to use as the router, so routing/wi-fi component is not important. > > I started looking at options. The things I thought were important were: > > * Good/best modem performance > > * Support for all/most relevant modem connectivity standards > > * Support for open firmware > > When I started looking, I was looking for modem-only devices. The closest > I could find were modem/routers with only one ethernet port. > > The device that seemed to fit the bill was the Netgear DM200. It has 1 > port (albeit 100 MBPS but VDSL connections aren't exceeding that at the > moment anyway), and it appears to be supported by OpenWRT. > > But then I started seeing some bad reviews. In particular, with the help > of Google Translate, I saw some unhappy people on hometheatre.co.il, > which I considered important as I will be using it here. > > The othere 1-port device I found was the Draytech Vigor 130. This is more > expensive, but I don't mind about that if the device is good. It seemed to > generally get better performance reviews than the Netgear. However, it's > not supported by OpenWRT as far as I can see. > > There were a couple of other things that I also saw time and again. Some > people say that Broadcom chips are better than others , though I've also > seem comment that compatibility with the upstream chipset is the most > important. Also, I saw references to things like vectoring and g.imp, > things I know nothing about but which sound important, and g.fast which > seems like it's going to be the next big thing but which nothing much seems > to support yet. > > People also pointed out that there's really no difference between using a > 1-port router in bridged mode and using a 4-port router in bridged mode. > As neither of the above had the Broadcom chips, and since I had seen quite > a few recommendations for the TP Link 9970/9977, I looked into these. > > The TP Link 9970 and 9977 (9977 is apparently newer) both seem to have the > Broadcom chips and support g.imp. But neither seem to support open > firmware from what I can see. Otheres like the 9980 do support it, but > they don't have Broadcom chips. > > Some people make the argument that if you're using a device in bridged > mode, open firmware is less important, but I'm not sure if I buy that > argument. > > So now I'm a bit confused. I'm starting to lose track of which devices > support what, and I don't really know which of these factors are > specifically important here in Israel and which are more or less important > generally. > > Since I believe that most problems have been solved by others before me, I > thought I'd ask here. > > To repeat, I'm after a good modem. I'm not so concerned about price and I > really don't care about the routing and wi-fi specs as I don't plan on > using them. > > Any advice anyone can provide would be most helpful. > > > Thanks, > Geoff. > > > _______________________________________________ > Linux-il mailing list > Linux-il at cs.huji.ac.il > http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rabin at rabin.io Thu Jun 7 20:51:36 2018 From: rabin at rabin.io (Rabin Yasharzadehe) Date: Thu, 7 Jun 2018 20:51:36 +0300 Subject: VDSL modem recommendation In-Reply-To: References: Message-ID: About TD-W9980, I have it, and I was happy with at first, but not so much any more, TP-LINK dose not publish any fimware updates for it, and the WRT support is defiantly not for common users. I also think that having a modem even as closed source and not WRT supported is not a biggie if you're going to use it in bridge mode. my 2c -- Rabin On Thu, 7 Jun 2018 at 12:23, Geoff Shang wrote: > Hi everyone, > > I'm in the market for a VDSL modem, and after looking at various options, > I'm unsure which I should buy. So I thought I'd ask here for > help/recommendations. > > Last year, I posted to the list about ISP recommendations. As part of > that discussion, I was sent a link to a site listing modems compatible > with Israeli service. Until then, I didn't know that buying your own > device was even an option here. > > As previously mentioned, ISP-supplied modems/routers are an issue for me > as a blind person, as many seem to have visual CAPTCHAs which make it > difficult/impossible to log into the interface. And I don't think my > choice of ISP should be determined by whether or not I can log into their > equipment, I'd prefer to choose based on service (like everyone else). > > So I'm now looking to buy a modem. I have a Ubiquiti Edge Router which I > intend to use as the router, so routing/wi-fi component is not important. > > I started looking at options. The things I thought were important were: > > * Good/best modem performance > > * Support for all/most relevant modem connectivity standards > > * Support for open firmware > > When I started looking, I was looking for modem-only devices. The closest > I could find were modem/routers with only one ethernet port. > > The device that seemed to fit the bill was the Netgear DM200. It has 1 > port (albeit 100 MBPS but VDSL connections aren't exceeding that at the > moment anyway), and it appears to be supported by OpenWRT. > > But then I started seeing some bad reviews. In particular, with the > help of Google Translate, I saw some unhappy people on hometheatre.co.il, > which I considered important as I will be using it here. > > The othere 1-port device I found was the Draytech Vigor 130. This is more > expensive, but I don't mind about that if the device is good. It seemed > to generally get better performance reviews than the Netgear. However, > it's not supported by OpenWRT as far as I can see. > > There were a couple of other things that I also saw time and again. Some > people say that Broadcom chips are better than others , though I've also > seem comment that compatibility with the upstream chipset is the most > important. Also, I saw references to things like vectoring and g.imp, > things I know nothing about but which sound important, and g.fast which > seems like it's going to be the next big thing but which nothing much > seems to support yet. > > People also pointed out that there's really no difference between using > a 1-port router in bridged mode and using a 4-port router in bridged > mode. As neither of the above had the Broadcom chips, and since I had > seen quite a few recommendations for the TP Link 9970/9977, I looked into > these. > > The TP Link 9970 and 9977 (9977 is apparently newer) both seem to have the > Broadcom chips and support g.imp. But neither seem to support open > firmware from what I can see. Otheres like the 9980 do support it, but > they don't have Broadcom chips. > > Some people make the argument that if you're using a device in bridged > mode, open firmware is less important, but I'm not sure if I buy that > argument. > > So now I'm a bit confused. I'm starting to lose track of which devices > support what, and I don't really know which of these factors are > specifically important here in Israel and which are more or less > important generally. > > Since I believe that most problems have been solved by others before me, I > thought I'd ask here. > > To repeat, I'm after a good modem. I'm not so concerned about price and I > really don't care about the routing and wi-fi specs as I don't plan on > using them. > > Any advice anyone can provide would be most helpful. > > > Thanks, > Geoff. > > > _______________________________________________ > Linux-il mailing list > Linux-il at cs.huji.ac.il > http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il > -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at QuiteLikely.com Sun Jun 10 01:19:46 2018 From: geoff at QuiteLikely.com (Geoff Shang) Date: Sat, 9 Jun 2018 23:19:46 +0100 (BST) Subject: VDSL modem recommendation In-Reply-To: References: Message-ID: On Thu, 7 Jun 2018, ik wrote: > I just bought a router actually named VR600 of TP-LINK, and it works great > even on vectoring for example. The VR600 seems to have the Broadcom chips, like the 9970/9977. https://www.tp-link.com/au/products/details/cat-15_Archer-VR600.html I think that if I were to buy a router for use in bridged mode, this would be overkill as it has lots of fancy stuff that I would pay for and wouldn't use. But this does seem to be another vote for TP Link's Broadcom chip devices. Thanks, Geoff. From geoff at QuiteLikely.com Sun Jun 10 01:26:14 2018 From: geoff at QuiteLikely.com (Geoff Shang) Date: Sat, 9 Jun 2018 23:26:14 +0100 (BST) Subject: VDSL modem recommendation In-Reply-To: References: Message-ID: On Thu, 7 Jun 2018, Rabin Yasharzadehe wrote: > About TD-W9980, I have it, and I was happy with at first, but not so much > any more, > TP-LINK dose not publish any fimware updates for it, and the WRT support is > defiantly not for common users. good to know. Are these the only reasons why you're not so keen, or does it not perform as well as you would like? > I also think that having a modem even as closed source and not WRT > supported is not a biggie if you're going to use it in bridge mode. Thanks for your opinion on this. I'm interested to know what others think about this. Cheers, Geoff. From 0 at zeltak.com Sun Jun 17 11:05:45 2018 From: 0 at zeltak.com (Shahar Tamir) Date: Sun, 17 Jun 2018 11:05:45 +0300 Subject: [OT] [JOB] Linux sysadmin Message-ID: <59b53d3a-c0b9-ed64-7c27-aaacf57b68cc@zeltak.com> My employer, Jungo Connectivity, is looking for a system administrator. Transcribed from https://www.jungo.com/st/jobpost/system-administrator/ Job description: Administer servers and network infrastructure Handle workstation issues Requirements: 2+ years experience in a similar role Ability to work individually and collaborate with engineering teams Good knowledge of Linux environments (mostly Ubuntu and Debian) Good knowledge of shell scripting Understanding of networking concepts (e.g. subnets, vlans, routing) Self-learning skills and desire to know more Good communication skills Optional: Windows administration skills Familiarity with versioning systems (cvs / svn, git) Familiarity with CI systems (Jenkins) MySQL administration skills PHP Perl Familiarity with virtualization systems Familiarity with Juniper equipment Part-time job possible Location: Poleg, Netanya Contact me off-list or at shahart at jungo.com Shahar. From rabin at rabin.io Tue Jun 19 09:06:56 2018 From: rabin at rabin.io (Rabin Yasharzadehe) Date: Tue, 19 Jun 2018 09:06:56 +0300 Subject: Moving away from cron jobs to some workflow manager Message-ID: Hi all, I need some advice, currently I have a huge cron file which schedules tasks one after anther, and each task is position precisely (with some room for error) to start after it predecessor. So if one job start at 00:00 and it will go and fetch some files and it takes 3minutes the next job will be after start right after at ~00:05 and so on the problem is that if one job fails, all other jobs which are depend on him will fail as well, and then I get a shitload of alerts, and the worst part is that if i have to manually start a batch process I need to go to each machine and manually start each job in the right order, I was looking to resolve this problem with a tool which can manage this "pipe line" and I cam across several tools like Luigi and (apache-)AirFlow, I started with Luigi but It didn't look right for the job, and then I tried airflow, but was not able to make it to work, the jobs queue never executed. =( Has any one have experience with airflow, or other tool like it which they can recommend ? My needs are to be able to execute my CURRENT shell/python/php scripts and build the dependency between them, and I perfer the option for remote exec so that I will have central place to manage and monitor all work flow whichs are executed on several nodes. Thanks in advance, Rabin -------------- next part -------------- An HTML attachment was scrubbed... URL: From rabin at rabin.io Tue Jun 19 09:41:30 2018 From: rabin at rabin.io (Rabin Yasharzadehe) Date: Tue, 19 Jun 2018 09:41:30 +0300 Subject: Moving away from cron jobs to some workflow manager In-Reply-To: <6635350D-8925-4509-8246-AB2454268DCF@icloud.com> References: <6635350D-8925-4509-8246-AB2454268DCF@icloud.com> Message-ID: never heard of it, but from reading the manual and the 10minute presentation , it's seems like it is more suitable for data crunching, where you have a pool of compute resources and you submit jobs to it. my case is a bit different, where I have many jobs which need to run (orchestrated) on there own hosts with a specific environment and setup. -- Rabin On Tue, 19 Jun 2018 at 09:10, Marc Volovic wrote: > Why not a minimal deploy of SGE - which would also allow you to make > multi-executor? > > https://arc.liv.ac.uk/trac/SGE > > ?mav > Marc Volovic > marcvolovic at me.com > > > > > On 19 Jun 2018, at 9:06, Rabin Yasharzadehe wrote: > > > > Hi all, > > > > I need some advice, currently I have a huge cron file which schedules > tasks one after anther, and each task is position precisely (with some room > for error) to start after it predecessor. > > > > So if one job start at 00:00 and it will go and fetch some files and it > takes 3minutes > > the next job will be after start right after at ~00:05 > > and so on > > > > the problem is that if one job fails, all other jobs which are depend on > him will fail as well, and then I get a shitload of alerts, and the worst > part is that if i have to manually start a batch process I need to go to > each machine and manually start each job in the right order, > > > > I was looking to resolve this problem with a tool which can manage this > "pipe line" > > and I cam across several tools like Luigi and (apache-)AirFlow, I > started with Luigi but It didn't look > > right for the job, and then I tried airflow, but was not able to make it > to work, the jobs queue never executed. =( > > > > Has any one have experience with airflow, or other tool like it which > they can recommend ? > > My needs are to be able to execute my CURRENT shell/python/php scripts > and build the dependency between them, and I perfer the option for remote > exec so that I will have central > > place to manage and monitor all work flow whichs are executed on several > nodes. > > > > Thanks in advance, > > Rabin > > _______________________________________________ > > Linux-il mailing list > > Linux-il at cs.huji.ac.il > > http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From w1 at zak.co.il Tue Jun 19 09:42:35 2018 From: w1 at zak.co.il (Omer Zak) Date: Tue, 19 Jun 2018 09:42:35 +0300 Subject: Moving away from cron jobs to some workflow manager In-Reply-To: References: Message-ID: <1529390555.14444.140.camel@zak.co.il> For dependency management, you may want to use 'make' or modern equivalents ('ant', 'gradle', etc.). For controlling remote nodes, 'ansible' may be able to do the work. --- Omer Zak On Tue, 2018-06-19 at 09:06 +0300, Rabin Yasharzadehe wrote: > Hi all,? > > I need some advice, currently I have a huge cron file which schedules > tasks one after anther, and each task is position precisely (with > some room for error) to start after it predecessor. > > So if one job start at 00:00 and it will go and fetch some files and > it takes 3minutes > the next job will be after start right after at ~00:05? > and so on? > > the problem is that if one job fails, all other jobs which are depend > on him will fail as well, and then I get a shitload of alerts, and > the worst part is that if i have to manually start a batch process I > need to go to each machine and manually start each job in the right > order, > > I was looking to resolve this problem with a tool which can manage > this "pipe line"? > and I cam across several tools like Luigi and (apache-)AirFlow, I > started with Luigi but It didn't look > right for the job, and then I tried airflow, but was not able to make > it to work, the jobs queue never executed. =( > > Has any one have experience with airflow, or other tool like it which > they can recommend ?? > My needs are to be able to execute my CURRENT shell/python/php > scripts and build the dependency between them, and I perfer the > option for remote exec so that I will have central? > place to manage and monitor all work flow whichs are executed on > several nodes. -- More proof the End of the World has started. Just saw this online: I think it's beginning! Ten minutes ago there was a group of people waiting at the bus stop outside my house. Now, they're all gone! My own blog is at https://tddpirate.zak.co.il/ My opinions, as expressed in this E-mail message, are mine alone. They do not represent the official policy of any organization with which I may be affiliated in any way. WARNING TO SPAMMERS: at https://www.zak.co.il/spamwarning.html From rabin at rabin.io Tue Jun 19 11:20:48 2018 From: rabin at rabin.io (Rabin Yasharzadehe) Date: Tue, 19 Jun 2018 11:20:48 +0300 Subject: Moving away from cron jobs to some workflow manager In-Reply-To: <26827B9C-C4E6-498B-BD68-B2A54FBA1564@icloud.com> References: <6635350D-8925-4509-8246-AB2454268DCF@icloud.com> <26827B9C-C4E6-498B-BD68-B2A54FBA1564@icloud.com> Message-ID: I'll have to read the documentation to learn more, but this project seems barely maintained as only minor versions each year or two (last release was 2 years ago), that doesn't give a lot of confidence. but i'll check it out thanks. -- Rabin On Tue, 19 Jun 2018 at 09:43, Marc Volovic wrote: > Hi, > > It is intended for submitting multiple jobs for crunching. But you can use > it (SOGE) or SLURM for issuing job and dependent jobs, even on a single > machine issuer/execution host. It can be used as a resource aware job > scheduler. > > ?mav > Marc Volovic > marcvolovic at me.com > > > > > On 19 Jun 2018, at 9:41, Rabin Yasharzadehe wrote: > > > > never heard of it, > > but from reading the manual and the 10minute presentation , > > it's seems like it is more suitable for data crunching, where you have a > pool > > of compute resources and you submit jobs to it. > > > > my case is a bit different, where I have many jobs which need to run > (orchestrated) on there own hosts > > with a specific environment and setup. > > > > > > -- > > Rabin > > > > > > On Tue, 19 Jun 2018 at 09:10, Marc Volovic > wrote: > > Why not a minimal deploy of SGE - which would also allow you to make > multi-executor? > > > > https://arc.liv.ac.uk/trac/SGE > > > > ?mav > > Marc Volovic > > marcvolovic at me.com > > > > > > > > > On 19 Jun 2018, at 9:06, Rabin Yasharzadehe wrote: > > > > > > Hi all, > > > > > > I need some advice, currently I have a huge cron file which schedules > tasks one after anther, and each task is position precisely (with some room > for error) to start after it predecessor. > > > > > > So if one job start at 00:00 and it will go and fetch some files and > it takes 3minutes > > > the next job will be after start right after at ~00:05 > > > and so on > > > > > > the problem is that if one job fails, all other jobs which are depend > on him will fail as well, and then I get a shitload of alerts, and the > worst part is that if i have to manually start a batch process I need to go > to each machine and manually start each job in the right order, > > > > > > I was looking to resolve this problem with a tool which can manage > this "pipe line" > > > and I cam across several tools like Luigi and (apache-)AirFlow, I > started with Luigi but It didn't look > > > right for the job, and then I tried airflow, but was not able to make > it to work, the jobs queue never executed. =( > > > > > > Has any one have experience with airflow, or other tool like it which > they can recommend ? > > > My needs are to be able to execute my CURRENT shell/python/php scripts > and build the dependency between them, and I perfer the option for remote > exec so that I will have central > > > place to manage and monitor all work flow whichs are executed on > several nodes. > > > > > > Thanks in advance, > > > Rabin > > > _______________________________________________ > > > Linux-il mailing list > > > Linux-il at cs.huji.ac.il > > > http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From moish at mln.co.il Tue Jun 19 12:12:16 2018 From: moish at mln.co.il (Moish) Date: Tue, 19 Jun 2018 12:12:16 +0300 Subject: Moving away from cron jobs to some workflow manager In-Reply-To: <1529390555.14444.140.camel@zak.co.il> References: <1529390555.14444.140.camel@zak.co.il> Message-ID: Try GNUbatch. On June 19, 2018 9:42:35 AM GMT+03:00, Omer Zak wrote: >For dependency management, you may want to use 'make' or modern >equivalents ('ant', 'gradle', etc.). >For controlling remote nodes, 'ansible' may be able to do the work. > >--- Omer Zak > > >On Tue, 2018-06-19 at 09:06 +0300, Rabin Yasharzadehe wrote: >> Hi all,? >> >> I need some advice, currently I have a huge cron file which schedules >> tasks one after anther, and each task is position precisely (with >> some room for error) to start after it predecessor. >> >> So if one job start at 00:00 and it will go and fetch some files and >> it takes 3minutes >> the next job will be after start right after at ~00:05? >> and so on? >> >> the problem is that if one job fails, all other jobs which are depend >> on him will fail as well, and then I get a shitload of alerts, and >> the worst part is that if i have to manually start a batch process I >> need to go to each machine and manually start each job in the right >> order, >> >> I was looking to resolve this problem with a tool which can manage >> this "pipe line"? >> and I cam across several tools like Luigi and (apache-)AirFlow, I >> started with Luigi but It didn't look >> right for the job, and then I tried airflow, but was not able to make >> it to work, the jobs queue never executed. =( >> >> Has any one have experience with airflow, or other tool like it which >> they can recommend ?? >> My needs are to be able to execute my CURRENT shell/python/php >> scripts and build the dependency between them, and I perfer the >> option for remote exec so that I will have central? >> place to manage and monitor all work flow whichs are executed on >> several nodes. >-- >More proof the End of the World has started. Just saw this online: >I think it's beginning! Ten minutes ago there was a group of people >waiting at the bus stop outside my house. Now, they're all gone! >My own blog is at https://tddpirate.zak.co.il/ > >My opinions, as expressed in this E-mail message, are mine alone. >They do not represent the official policy of any organization with >which >I may be affiliated in any way. >WARNING TO SPAMMERS: at https://www.zak.co.il/spamwarning.html > >_______________________________________________ >Linux-il mailing list >Linux-il at cs.huji.ac.il >http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il -------------- next part -------------- An HTML attachment was scrubbed... URL: From dimidd at gmail.com Tue Jun 19 12:25:21 2018 From: dimidd at gmail.com (Dimid Duchovny) Date: Tue, 19 Jun 2018 12:25:21 +0300 Subject: Moving away from cron jobs to some workflow manager In-Reply-To: References: <1529390555.14444.140.camel@zak.co.il> Message-ID: Hi Rabin, I'm far from being a linux expert, but isn't dependency between services handled by systemd? E.g. https://wiki.archlinux.org/index.php/Systemd/Timers HTH 2018-06-19 12:12 GMT+03:00 Moish : > Try GNUbatch. > > > On June 19, 2018 9:42:35 AM GMT+03:00, Omer Zak wrote: > >> For dependency management, you may want to use 'make' or modern >> equivalents ('ant', 'gradle', etc.). >> For controlling remote nodes, 'ansible' may be able to do the work. >> >> --- Omer Zak >> >> >> On Tue, 2018-06-19 at 09:06 +0300, Rabin Yasharzadehe wrote: >> >>> Hi all, >>> >>> I need some advice, currently I have a huge cron file which schedules >>> tasks one after anther, and each task is position precisely (with >>> some room for error) to start after it predecessor. >>> >>> So if one job start at 00:00 and it will go and fetch some files and >>> it takes 3minutes >>> the next job will be after start right after at ~00:05 >>> and so on >>> >>> the problem is that if one job fails, all other jobs which are depend >>> on him will fail as well, and then I get a shitload of alerts, and >>> the worst part is that if i have to manually start a batch process I >>> need to go to each machine and manually start each job in the right >>> order, >>> >>> I was looking to resolve this problem with a tool which can manage >>> this "pipe line" >>> and I cam across several tools like Luigi and (apache-)AirFlow, I >>> started with Luigi but It didn't look >>> right for the job, and then I tried airflow, but was not able to make >>> it to work, the jobs queue never executed. =( >>> >>> Has any one have experience with airflow, or other tool like it which >>> they can recommend ? >>> My needs are to be able to execute my CURRENT shell/python/php >>> scripts and build the dependency between them, and I perfer the >>> option for remote exec so that I will have central >>> place to manage and monitor all work flow whichs are executed on >>> several nodes. >>> >> -- >> More proof the End of the World has started. Just saw this online: >> I think it's beginning! Ten minutes ago there was a group of people >> waiting at the bus stop outside my house. Now, they're all gone! >> My own blog is at https://tddpirate.zak.co.il/ >> >> My opinions, as expressed in this E-mail message, are mine alone. >> They do not represent the official policy of any organization with which >> I may be affiliated in any way. >> WARNING TO SPAMMERS: at https://www.zak.co.il/spamwarning.html >> >> ------------------------------ >> >> Linux-il mailing list >> Linux-il at cs.huji.ac.il >> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il >> >> > _______________________________________________ > Linux-il mailing list > Linux-il at cs.huji.ac.il > http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rabin at rabin.io Tue Jun 19 12:32:30 2018 From: rabin at rabin.io (Rabin Yasharzadehe) Date: Tue, 19 Jun 2018 12:32:30 +0300 Subject: Moving away from cron jobs to some workflow manager In-Reply-To: References: <1529390555.14444.140.camel@zak.co.il> Message-ID: systemd is a complete different tool, which was not designed for this kinda purpose. (maybe in the future it will grow to be something like that ;-) ) I'm looking for something a bit more sophisticated then "go to this machine" and "run this script" and "expect this result" i like to define execution time limits (finish in 3 minute) and maybe some grace time (can go up to 5 minute) and have the orchestrator monitor the process and have a nice dashboard where i can see every thing from above (this is why Airflow was looked so appealing, but the installation process and the documentation are still lagging behind). -- Rabin On Tue, 19 Jun 2018 at 12:26, Dimid Duchovny wrote: > Hi Rabin, > > > I'm far from being a linux expert, but isn't dependency between services > handled by systemd? > E.g. https://wiki.archlinux.org/index.php/Systemd/Timers > > HTH > > 2018-06-19 12:12 GMT+03:00 Moish : > >> Try GNUbatch. >> >> >> On June 19, 2018 9:42:35 AM GMT+03:00, Omer Zak wrote: >> >>> For dependency management, you may want to use 'make' or modern >>> equivalents ('ant', 'gradle', etc.). >>> For controlling remote nodes, 'ansible' may be able to do the work. >>> >>> --- Omer Zak >>> >>> >>> On Tue, 2018-06-19 at 09:06 +0300, Rabin Yasharzadehe wrote: >>> >>>> Hi all, >>>> >>>> I need some advice, currently I have a huge cron file which schedules >>>> tasks one after anther, and each task is position precisely (with >>>> some room for error) to start after it predecessor. >>>> >>>> So if one job start at 00:00 and it will go and fetch some files and >>>> it takes 3minutes >>>> the next job will be after start right after at ~00:05 >>>> and so on >>>> >>>> the problem is that if one job fails, all other jobs which are depend >>>> on him will fail as well, and then I get a shitload of alerts, and >>>> the worst part is that if i have to manually start a batch process I >>>> need to go to each machine and manually start each job in the right >>>> order, >>>> >>>> I was looking to resolve this problem with a tool which can manage >>>> this "pipe line" >>>> and I cam across several tools like Luigi and (apache-)AirFlow, I >>>> started with Luigi but It didn't look >>>> right for the job, and then I tried airflow, but was not able to make >>>> it to work, the jobs queue never executed. =( >>>> >>>> Has any one have experience with airflow, or other tool like it which >>>> they can recommend ? >>>> My needs are to be able to execute my CURRENT shell/python/php >>>> scripts and build the dependency between them, and I perfer the >>>> option for remote exec so that I will have central >>>> place to manage and monitor all work flow whichs are executed on >>>> several nodes. >>>> >>> -- >>> More proof the End of the World has started. Just saw this online: >>> I think it's beginning! Ten minutes ago there was a group of people >>> waiting at the bus stop outside my house. Now, they're all gone! >>> My own blog is at https://tddpirate.zak.co.il/ >>> >>> My opinions, as expressed in this E-mail message, are mine alone. >>> They do not represent the official policy of any organization with which >>> I may be affiliated in any way. >>> WARNING TO SPAMMERS: at https://www.zak.co.il/spamwarning.html >>> >>> ------------------------------ >>> >>> Linux-il mailing list >>> Linux-il at cs.huji.ac.il >>> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il >>> >>> >> _______________________________________________ >> Linux-il mailing list >> Linux-il at cs.huji.ac.il >> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il >> >> > _______________________________________________ > Linux-il mailing list > Linux-il at cs.huji.ac.il > http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wickedpheonix at gmail.com Tue Jun 19 16:09:35 2018 From: wickedpheonix at gmail.com (Ari Becker) Date: Tue, 19 Jun 2018 16:09:35 +0300 Subject: Moving away from cron jobs to some workflow manager In-Reply-To: References: Message-ID: Hi Rabin, Did you consider using Jenkins? It may be a little heavyweight, but it should be relatively easy to set up and configure. You can use the same scripts you're using today, the ability to state which jobs run on which nodes, set up dependencies between them, set timeouts, set cron triggers to start the initial job... seems to answer your requirements. -Ari On Tue, Jun 19, 2018, 09:07 Rabin Yasharzadehe wrote: > Hi all, > > I need some advice, currently I have a huge cron file which schedules > tasks one after anther, and each task is position precisely (with some room > for error) to start after it predecessor. > > So if one job start at 00:00 and it will go and fetch some files and it > takes 3minutes > the next job will be after start right after at ~00:05 > and so on > > the problem is that if one job fails, all other jobs which are depend on > him will fail as well, and then I get a shitload of alerts, and the worst > part is that if i have to manually start a batch process I need to go to > each machine and manually start each job in the right order, > > I was looking to resolve this problem with a tool which can manage this > "pipe line" > and I cam across several tools like Luigi and (apache-)AirFlow, I started > with Luigi but It didn't look > right for the job, and then I tried airflow, but was not able to make it > to work, the jobs queue never executed. =( > > Has any one have experience with airflow, or other tool like it which they > can recommend ? > My needs are to be able to execute my CURRENT shell/python/php scripts and > build the dependency between them, and I perfer the option for remote exec > so that I will have central > place to manage and monitor all work flow whichs are executed on several > nodes. > > Thanks in advance, > Rabin > _______________________________________________ > Linux-il mailing list > Linux-il at cs.huji.ac.il > http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il > -------------- next part -------------- An HTML attachment was scrubbed... URL: From w1 at zak.co.il Tue Jun 19 16:17:51 2018 From: w1 at zak.co.il (Omer Zak) Date: Tue, 19 Jun 2018 16:17:51 +0300 Subject: Moving away from cron jobs to some workflow manager In-Reply-To: References: <1529390555.14444.140.camel@zak.co.il> Message-ID: <1529414271.14444.148.camel@zak.co.il> 1. Execution time limits: Ansible has async with polling intervals. I did not research for methods to kill hung tasks. https://docs.ansible.com/ansible/latest/user_guide/playbooks_async.html 2. Dashboard-like functionality According to: https://www.reddit.com/r/ansible/comments/5ksphc/best_web_gui_for_run_a nsible_playbooks/ There are the following options: - ansible-tower - remote-task-executor (did not look into it) - nci-ansible-ui - Jenkins (normally used for CI/CD setups) In addition to the above, you may want to look into Ansible alternatives: - Puppet - Chef - SaltStack A quick Google search yielded: https://www.intigua.com/blog/puppet-vs.-chef-vs.-ansible-vs.-saltstack On Tue, 2018-06-19 at 12:32 +0300, Rabin Yasharzadehe wrote: > systemd is a complete different tool, which was not designed for this > kinda purpose. > (maybe in the future it will grow to be something like that ;-) ) > > I'm looking for something a bit more sophisticated then "go to this > machine" and "run this script" and "expect this result" > i like to define execution time limits (finish in 3 minute) and maybe > some grace time (can go up to 5 minute) and have the orchestrator > monitor the process and have a nice dashboard where i can see every > thing from above (this is why Airflow was looked so appealing, but > the installation process and the documentation are still lagging > behind). > > > -- > Rabin > > > On Tue, 19 Jun 2018 at 12:26, Dimid Duchovny > wrote: > > Hi Rabin, > > > > > > I'm far from being a linux expert, but isn't dependency between > > services handled by systemd? > > E.g. https://wiki.archlinux.org/index.php/Systemd/Timers > > > > HTH > > > > 2018-06-19 12:12 GMT+03:00 Moish : > > > Try GNUbatch.? > > > > > > > > > On June 19, 2018 9:42:35 AM GMT+03:00, Omer Zak > > > wrote: > > > > For dependency management, you may want to use 'make' or modern > > > > equivalents ('ant', 'gradle', etc.). > > > > For controlling remote nodes, 'ansible' may be able to do the > > > > work. > > > > > > > > --- Omer Zak > > > > > > > > > > > > On Tue, 2018-06-19 at 09:06 +0300, Rabin Yasharzadehe wrote: > > > > > ?Hi all,? > > > > > ? > > > > > ?I need some advice, currently I have a huge cron file which > > > > > schedules > > > > > ?tasks one after anther, and each task is position precisely > > > > > (with > > > > > ?some room for error) to start after it predecessor. > > > > > ? > > > > > ?So if one job start at 00:00 and it will go and fetch some > > > > > files and > > > > > ?it takes 3minutes > > > > > ?the next job will be after start right after at ~00:05? > > > > > ?and so on? > > > > > ? > > > > > ?the problem is that if one job fails, all other jobs which > > > > > are depend > > > > > ?on him will fail as well, and then I get a shitload of > > > > > alerts, and > > > > > ?the worst part is that if i have to manually start a batch > > > > > process I > > > > > ?need to go to each machine and manually start each job in > > > > > the right > > > > > ?order, > > > > > ? > > > > > ?I was looking to resolve this problem with a tool which can > > > > > manage > > > > > ?this "pipe line"? > > > > > ?and I cam across several tools like Luigi and (apache- > > > > > )AirFlow, I > > > > > ?started with Luigi but It didn't look > > > > > ?right for the job, and then I tried airflow, but was not > > > > > able to make > > > > > ?it to work, the jobs queue never executed. =( > > > > > ? > > > > > ?Has any one have experience with airflow, or other tool like > > > > > it which > > > > > ?they can recommend ?? > > > > > ?My needs are to be able to execute my CURRENT > > > > > shell/python/php > > > > > ?scripts and build the dependency between them, and I perfer > > > > > the > > > > > ?option for remote exec so that I will have central? > > > > > ?place to manage and monitor all work flow whichs are > > > > > executed on > > > > > ?several nodes. -- What happens if one mixes together evolution with time travel to the past???See:?https://www.zak.co.il/ideas/stuff/opinions/eng/evol_tm.html My own blog is at https://tddpirate.zak.co.il/ My opinions, as expressed in this E-mail message, are mine alone. They do not represent the official policy of any organization with which I may be affiliated in any way. WARNING TO SPAMMERS:??at https://www.zak.co.il/spamwarning.html From moish at mln.co.il Tue Jun 19 16:19:23 2018 From: moish at mln.co.il (Moish) Date: Tue, 19 Jun 2018 16:19:23 +0300 Subject: Moving away from cron jobs to some workflow manager In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From linux.il at gmail.com Tue Jun 19 16:35:47 2018 From: linux.il at gmail.com (linux.il) Date: Tue, 19 Jun 2018 16:35:47 +0300 Subject: Moving away from cron jobs to some workflow manager In-Reply-To: References: Message-ID: I suggest to check Jenkins (as already suggested) and Rundeck. Vitaly On Tue, Jun 19, 2018 at 9:06 AM, Rabin Yasharzadehe wrote: > Hi all, > > I need some advice, currently I have a huge cron file which schedules > tasks one after anther, and each task is position precisely (with some room > for error) to start after it predecessor. > > So if one job start at 00:00 and it will go and fetch some files and it > takes 3minutes > the next job will be after start right after at ~00:05 > and so on > > the problem is that if one job fails, all other jobs which are depend on > him will fail as well, and then I get a shitload of alerts, and the worst > part is that if i have to manually start a batch process I need to go to > each machine and manually start each job in the right order, > > I was looking to resolve this problem with a tool which can manage this > "pipe line" > and I cam across several tools like Luigi and (apache-)AirFlow, I started > with Luigi but It didn't look > right for the job, and then I tried airflow, but was not able to make it > to work, the jobs queue never executed. =( > > Has any one have experience with airflow, or other tool like it which they > can recommend ? > My needs are to be able to execute my CURRENT shell/python/php scripts and > build the dependency between them, and I perfer the option for remote exec > so that I will have central > place to manage and monitor all work flow whichs are executed on several > nodes. > > Thanks in advance, > Rabin > > _______________________________________________ > Linux-il mailing list > Linux-il at cs.huji.ac.il > http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From slitt at troubleshooters.com Wed Jun 20 04:29:43 2018 From: slitt at troubleshooters.com (Steve Litt) Date: Tue, 19 Jun 2018 21:29:43 -0400 Subject: Moving away from cron jobs to some workflow manager In-Reply-To: <1529390555.14444.140.camel@zak.co.il> References: <1529390555.14444.140.camel@zak.co.il> Message-ID: <20180619212943.3fe6a25b@mydesk.domain.cxm> On Tue, 19 Jun 2018 09:42:35 +0300 Omer Zak wrote: > For dependency management, you may want to use 'make' If you can depend on each task to create specific files, yeah, that sounds like a great idea. I should have thought of it. And then you just put it in a loop so things are always being progressed upon, and alarms to warn if a step takes too long. SteveT Steve Litt June 2018 featured book: Twenty Eight Tales of Troubleshooting http://www.troubleshooters.com/28 From slitt at troubleshooters.com Wed Jun 20 04:38:04 2018 From: slitt at troubleshooters.com (Steve Litt) Date: Tue, 19 Jun 2018 21:38:04 -0400 Subject: Moving away from cron jobs to some workflow manager In-Reply-To: References: <1529390555.14444.140.camel@zak.co.il> Message-ID: <20180619213804.671315be@mydesk.domain.cxm> On Tue, 19 Jun 2018 12:25:21 +0300 Dimid Duchovny wrote: > Hi Rabin, > > > I'm far from being a linux expert, but isn't dependency between > services handled by systemd? > E.g. https://wiki.archlinux.org/index.php/Systemd/Timers If you drive on that side of the road :-) More seriously, I think you're intermixing these systemd timers and systemd's (sort of) ability to delay running long running daemon B until daemon A, which it depends on, is running. But the OP's needs were much greater. Apparently he couldn't depend on any of consecutively run programs to conclude in a certain amount of time, and if he grants each one a crazy long amount of time, it would exceed 24 hours. What's needed is for each process to provide some clue that it's finished. Assuming its output files are those clues, Omer's right: make could be used to not only do the job, but add some parallellization so that if two processes' input is each complete, those two processes can be run in tandem. It's brilliant. SteveT Steve Litt June 2018 featured book: Twenty Eight Tales of Troubleshooting http://www.troubleshooters.com/28 From slitt at troubleshooters.com Wed Jun 20 08:01:08 2018 From: slitt at troubleshooters.com (Steve Litt) Date: Wed, 20 Jun 2018 01:01:08 -0400 Subject: Moving away from cron jobs to some workflow manager In-Reply-To: <1529390555.14444.140.camel@zak.co.il> References: <1529390555.14444.140.camel@zak.co.il> Message-ID: <20180620010108.1aa6b868@mydesk.domain.cxm> On Tue, 19 Jun 2018 09:42:35 +0300 Omer Zak wrote: > For dependency management, you may want to use 'make' or modern Hi Omer, While corresponding with someone offlist, I had another idea maybe as good as using make. I could make a customized installation of the process supervisor part of either the runit or s6 inits, or maybe even use just plain daemontools, to make sure apps don't run until apps upon which they depend have finished. So, for appC that depends on output from appA and appB, then the run script for appC would look something like the following: #!/bin/sh if appAnot_finished; then sleep 60 # prevent excessive polling elif appBnot_finished; then sleep 60 # prevent excessive polling else exec appC fi If every app expresses its immediate prerequisites that way, the whole thing will run very efficiently, and in many cases, in parallel where not prevented by unfinished prerequisites. Some more complexity would need to be added in order that appA and appB don't start again before the entire bucket brigade finishes. SteveT Steve Litt June 2018 featured book: Twenty Eight Tales of Troubleshooting http://www.troubleshooters.com/28 From lior at okman.name Wed Jun 20 08:41:55 2018 From: lior at okman.name (Lior Okman) Date: Wed, 20 Jun 2018 08:41:55 +0300 Subject: Moving away from cron jobs to some workflow manager In-Reply-To: References: Message-ID: Hi, You could try something like Concourse ( https://concourse-ci.org/ ). It allows you to define a pipeline which is comprised of jobs and the order in which they should be invoked. -- Lior On Tue, Jun 19, 2018 at 9:06 AM, Rabin Yasharzadehe wrote: > Hi all, > > I need some advice, currently I have a huge cron file which schedules > tasks one after anther, and each task is position precisely (with some room > for error) to start after it predecessor. > > So if one job start at 00:00 and it will go and fetch some files and it > takes 3minutes > the next job will be after start right after at ~00:05 > and so on > > the problem is that if one job fails, all other jobs which are depend on > him will fail as well, and then I get a shitload of alerts, and the worst > part is that if i have to manually start a batch process I need to go to > each machine and manually start each job in the right order, > > I was looking to resolve this problem with a tool which can manage this > "pipe line" > and I cam across several tools like Luigi and (apache-)AirFlow, I started > with Luigi but It didn't look > right for the job, and then I tried airflow, but was not able to make it > to work, the jobs queue never executed. =( > > Has any one have experience with airflow, or other tool like it which they > can recommend ? > My needs are to be able to execute my CURRENT shell/python/php scripts and > build the dependency between them, and I perfer the option for remote exec > so that I will have central > place to manage and monitor all work flow whichs are executed on several > nodes. > > Thanks in advance, > Rabin > > _______________________________________________ > Linux-il mailing list > Linux-il at cs.huji.ac.il > http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il > > -------------- next part -------------- An HTML attachment was scrubbed... URL: