Creating Backup Storage Sucks
smarmelling.com17 points by smarmelling 3 days ago
17 points by smarmelling 3 days ago
Why wouldn't you use NTFS for backing up a windows system?
It sounds like most of your frustration can be eliminated by actually using your NAS as the ground-truth for all data, using something like SMB/NFS instead of SyncThing.
Then you only need to backup the NAS.
Oh interesting I hadn’t considered this. I guess the only trouble would be if I am out with my laptop and I have no internet connection but this seems like a good tradeoff for simplicity
What the author describes here is hard because it's "simple."
What's simple because it's "hard" is replacing parts 2 & 3 with a network appliance like TrueNAS running a zfs pool that syncs to backblaze every night. Yeah you have to learn a bit but it won't fall in weird ways like the hard drive part here will just fail to mount one night and not back things up for 3 months until you notice. My 2¢
> like the hard drive part here will just fail to mount one night and not back things up for 3 months until you notice
I think the author is having trouble because he is conflating concepts and roles that should be distinct. Sync, rotating snapshots, and deduplicated backups need to be kept entirely separate if you want any hope of maintaining your sanity.
So he's got sync but he's missing some sort of rotating snapshot system which would solve the stated concern of guarding against syncthing replicating corrupted data. Such automated snapshots can then be used as the source to feed the backup pipeline.
That hard drive doesn't make a good backup because it seems that it is always online. You need an offline backup that you manually plug in to run the job once every so often.
He's also making this more difficult than it needs to be by insisting that the backup drive be compatible with windows. Plug the drives for both snapshots and backups into a linux box, format them with a modern filesystem, and get on with life.
Sync is its own clusterfuck and I have yet to arrive at a satisfactory solution myself despite wasting inordinate amounts of time on it. IMO you either go with a network share or you make due with the "least bad" option of syncthing. Personally I've more or less settled on sshfs at this point not because it's particularly good but because it works well enough and doesn't add any additional complexity.
Personally I use btrfs snapshots on all my devices, those get streamed across the network to a NAS, and the contents of the NAS are periodically (every few months) stuffed into borgbackup on redundant offline devices. Aside from sync the other problem you'll run into if you're a data hoarder is how to split backups across multiple drives once you exceed a few TB. Because external drives only get so large but the NAS will inevitably keep ballooning.
I agree I think one of the main things I learned from all this was that I should probably buy / set up a real NAS. I’ll look into zfs pool thanks for the comment!