Gradual installation of debian packages

Sat Aug 6 02:27:09 IDT 2016

What kind of hardware is this that's connected to the servers, and what
does the software do that you can't test before installing on production
servers?

On 6 August 2016 at 02:14, Elazar Leibovich <elazarl at gmail.com> wrote:

> All real servers, with custom hardware attached, geographically
> distributed across the planet.
>
> Real people actually use the hardware attached to this computers, and it's
> not obvious to test whether or not it failed.
>
> The strategy therefor is, deploy randomly to small percentage of the
> machines, wait to see if you get complains from those customers using these
> hardware devices, and if everything went well, update the rest of the
> servers.
>
> The provisioning solution is chef, but I'm open to changing it. As I said,
> I don't think it makes too much difference.
>
> As of immutable server images, I'd do it with ZFS/brtfs snapshots
> (+docker/machinectl/systemd-nspawn if you must have some sort of virtual
> environment), but it's probably a better idea than apt-get install
> pkg=oldversion. Immutable filesystem for execution is of course not enough,
> since you might have migrations for the mutable part, etc. In this
> particular case, I don't think it's a big deal.
>
> You see, not everything is a web startup with customer facing website ;-)
>
> Thanks,
> Appreciate you sharing your experience.
> I'm not disagreeing with your points, but in this particular case, where
> testing is expensive, not all of them seems valid.
>
> On Fri, Aug 5, 2016 at 3:15 PM, Amos Shapira <amos.shapira at gmail.com>
> wrote:
>
>> What provisioning tools do you use to manage these servers? Please tell
>> me you aren't doing all of this manually.
>> Also what's your environment? All hardware servers? Any virtualisation
>> involved? Cloud servers?
>>
>> Reading your question it feels like you are setting yourself up to fail
>> instead of minimising the failure altogether.
>>
>> What I suggest is that you test your package automatically in a test
>> environment (to me, Vagrant + Rspec/ServerSpec would be first candidates to
>> check) then rollout the package to the repository for the servers to pick
>> it up.
>>
>> As for "roll-back" - with comprehensive automatic testing this concept is
>> becoming obsolete, there is no such thing as "roll-back" only
>> "roll-forward", i.e. since the testing and rolling out are small and
>> "cheap", it should be feasible to fix whatever problem was found instead of
>> having to revert the change altogether.
>>
>> If you are in a properly supported virtual environment then I'd even go
>> for immutable server images (e.g. Packer building AMI's, or Docker
>> containers), then it's a matter of just firing up an instance of the new
>> image both when testing and in production.
>>
>> --Amos
>>
>> On 3 August 2016 at 16:55, Elazar Leibovich <elazarl at gmail.com> wrote:
>>
>>> How exactly you connect to the server is not in the scope of the
>>> discussion, and I agree that ansible is a sensible solution.
>>>
>>> But what you're proposing is to manually update the package on a small
>>> percent of the machines.
>>>
>>> Manual solution is fine, but I would like to hear experience of people
>>> who actually did that on many servers.
>>>
>>> There are many other issues, for example, how to you roll back?
>>>
>>> apt-get remove exposes you to the risk that the uninstallation script
>>> would be buggy. There are other solutions, e.g., btrfs snapshots on root
>>> partitions, but I'm curious to hear someone experienced with it to expose
>>> issues I didn't even thought of.
>>>
>>> Another issue is, how do you select the servers you try it?
>>>
>>> You suggested a static "beta" list, and I think it's better to select
>>> the candidates randomly on each update.
>>>
>>> Anyhow, how exactly you connect to the server is not the essence of the
>>> issue.
>>>
>>> On Wed, Aug 3, 2016 at 9:30 AM, Evgeniy Ginzburg <nad.oby at gmail.com>
>>> wrote:
>>>
>>>> Hello.
>>>> I'm assuming that you have paswordless ssh to the servers in question
>>>> as root.
>>>> Also I assume that you don't use central management/deployment software
>>>> (ansible/puppet/chef)
>>>> In similar cases I usully use parallel-ssh (gnu-parallel is another
>>>> alternative).
>>>> First stage install the package manually on one server to see that
>>>> configuration is OK, daemons restart, etc...
>>>> If this stage is ok second step will be creating list of servers for
>>>> "complain" list and install package on them trough parallel-ssh.
>>>> Instead of waiting for complains, one can define metrics to check and
>>>> use some monitoring appliance for verification.
>>>> I case of failure remove package from repository and remove-install
>>>> again.
>>>> Third will be parallel-ssh install on all the servers.
>>>>
>>>> P. S. In case of few tens of servers I'd prefer to work with ansible or
>>>> alternative, it's worh it in most cases/
>>>>
>>>> Best Regards, Evgeniy.
>>>>
>>>>
>>>> On Tue, Aug 2, 2016 at 8:50 PM, Elazar Leibovich <elazarl at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm having a few (say, a few tens) Debian machines, with a local
>>>>> repository defined.
>>>>>
>>>>> In the local repository I have some home made packages I'm building
>>>>> and pushing to the local repository.
>>>>>
>>>>> When I'm upgrading my package, I want to be sure the update wouldn't
>>>>> cause a problem.
>>>>>
>>>>> So I wish to install them on a few percentage of the machines, wait
>>>>> for complaints.
>>>>>
>>>>> If complaints arrive - roll back.
>>>>> Otherwise keep upgrading the whole machines.
>>>>>
>>>>> I'll appreciate your advice and experience of similar situation,
>>>>> I'll appreciate if someone who had actual real life experience with
>>>>> this situation would mention it in the comments.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> _______________________________________________
>>>>> Linux-il mailing list
>>>>> Linux-il at cs.huji.ac.il
>>>>> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> So long, and thanks for all the fish.
>>>>
>>>
>>>
>>> _______________________________________________
>>> Linux-il mailing list
>>> Linux-il at cs.huji.ac.il
>>> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>>>
>>>
>>
>>
>> --
>> <http://au.linkedin.com/in/gliderflyer>
>>
>
>

-- 
<http://au.linkedin.com/in/gliderflyer>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20160806/d0be7e27/attachment-0001.html>