slightly OT - backup strategy

[Rabin Yasharzadehe]

Using ZFS with sanoid <https://github.com/jimsalterjrs/sanoid>

ZFS will give you all the benefits of COW filesystem, compression, and
snapshots (and much more),
combined with sanoid utility will allow you to automate the snapshots and
send them to a remote system,
and because ZFS is block-level aware of the changes between snapshots,
send&recive is much more efficient,
because unlike Rsync which needs to stats and compare each file to
determine what to sync,
ZFS only need to compile a list of block which have changed between 2
snapshots and send only them.
which also works if the volume is encrypted, which allows you to have a
remote system, which is encrypted on rest,
and keep pushing/sending snapshots to it without having to unlock it.


--
Rabin


On Sun, 17 Jul 2022 at 16:50, Shlomo Solomon <shlomo.solomon at gmail.com>
wrote:

> I recently lost some files because of a bad disk - hardware problem.
>
> I do regular backups so I was not really worried, but I now see that I
> have a problem with my backup strategy so I'd like to know how others
> handle/prevent what happened to me.
>
> I backup files using rsync and I basically have 2 types of backups.
>
> My most important files are backed up every night. I do incremental
> backups using:    rsync -aqrlvtogS --ignore-errors  --backup
> I keep about 4 months of backups. So if a file is damaged,
> missing or accidentally deleted, I can find a good file - even if, for
> example I screwed up the file and only discovered the problem a few
> days later.
>
> BUT, all the rest of my files - music, videos, pictures, etc are backed
> up daily and weekly on 2 different physical drives using:
> rsync -qrlvtogS --delete --ignore-errors
> I use --delete to prevent accumulating garbage on my backup disks.
>
> So here's the problem: Because of a hardware problem, several files on
> one of my disks were lost. As a result, the daily backup script
> "thought" that those files should be deleted from the daily backup.
> Unfortunately, I did not notice the problem. A few days later, those
> same files were also deleted from the weekly backup. So they are lost.
>
> So on one hand, I need --delete to avoid keeping backups of old
> garbage, but on the other hand, the --delete option does not know if I
> deleted the file or if it's gone because of a hardware problem.
>
>
>
> --
> Shlomo Solomon
> http://the-solomons.net
> Claws Mail 3.17.5 - KDE Plasma 5.18.5 - Kubuntu 20.04
>
> _______________________________________________
> Linux-il mailing list
> Linux-il at cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20220720/ecaf9bab/attachment-0001.html>