One of the comments that got lost in the recent database restore was a pointer to a backup program I can’t offhand remember the name of. I remember that it’s a command-line tool written in Perl (alas) and meant to be done by a cron job; what it does underneath is rsync with hardlinks to the remote target, so you get a Time-Machine-like effect for not much beyond the space requirement of the initial dump. Can someone remind me what this is. please?
Also, I’m in the market for a dropbox-like service that I can rsync to and from, for off-site backup. Any suggestions?
UPDATE: rsnapshot is what I was trying to remember. A very elegant little tool, thoughtfully written and handy. I may go with rsync.net for offsite backup.
rsnapshot? I know that uses hardlinks, and I saw it mentioned. Not a bad program for most uses.
Scott beat me to it, although I’ve been squeamish about using it because its effectiveness depends entirely on what strategy the modifying program chooses to take with the file.
Rsnapshot is the most popular, but I’ve used FAUbackup in the past, which does the same sort of thing as well.
I’ve never used it but I recently investigated the possibility of using Amazon’s S3 service alongside rsync for backup. IIRC Amazon supply their own rsync-to-S3 script, here: http://www.s3rsync.com/
I never used it so I can’t vouch for it, but the cost is very low and one (probably) can’t fault Amazon’s infrastructure.
EDIT: seems that service isn’t supplied by Amazon. I’m pretty sure there are alternatives out there which don’t cost (or you could roll your own…)
Tarsnap by Colin Percival may meet your last request, but I can’t speak from personal experience. Be sure to check out the “About”.
It’s not the one you were thinking of, but take a look at rdiff-backup: http://www.nongnu.org/rdiff-backup/
As others said, rsnapshot was the solution linked in a lost comment.
I’ve been using a custom script based on the article that apparently inspired the rsnapshot author.
For the offsite service, I have heard of rsync.net but I have not tried it.
I’ve used rdiff-backup briefly, it seems pretty satisfactory.
On the other hand, this really seems like solving the problem at the wrong level; surely it would be better to have snapshots and filesystem level replication? Ceph is apparently capable of this, but I don’t think I’d trust it with my home directory yet – is anything more mature available with those features?
For that, matter, why not check your homedir into a VCS? I suppose there’s some bothersome issue with that I’ve forgotten (metadata?), but it’s would be an elegant solution… of course, you’ld need to some kind of tool to selectively delete certain older portions of the history to save space, if only such a tool existed :)
>For that, matter, why not check your homedir into a VCS?
Well, one good reason is that that doesn’t solve the problem. You want your backups on a volume physically separate from the original, so one disk failure can’t take originals and backups both out.
Another good reason is that there are VCSes under my home directory. What happens when, say, git finds an instance of .git below or above another one is territory I think best not explored.
You might be looking for Dirvish. It’s written in Perl, runs on top of rsync, uses hardlinks and it’s meant to be run as a cron job.
Also, I have used rsync.net and highly recommend them.
I mentioned rsnapshot in the comments. I love it. I’ve got it setup to run scripts to dump remote MySQL DBs and it backs up my local and remote web servers, my desktops, laptop, my Nokia N810 and even my wife’s WinXP desktop (via Cygwin). All over SSH.
ESR says: I fixed the link.
+1 in favor of Tarsnap. It’s saved my butt a couple times, and cperciva is the best hacker whose code I’ve ever had the pleasure of reviewing.
I’ve been using rsync.net for offsite backup myself, and enthusiastically recommend them.
I’m currently setting up rsnapshot for my home machine (onto an external drive); it seems to work reasonably well.
I was under the impression that one could check files into a VCS stored on a server halfway ’round the world.
On the contrary; it’s best that someone does explore this (on a test system of course). A properly-written VCS ought to be able to deal with filename collisions like this and not skip a beat. Everyone who maintains VCS code ought to do testing to prove they can handle it. If they can’t handle it, they ought to code into their VCS some way to loudly complain about exactly what they can’t handle, so that users know the limitations.
I use rsync.net — simple, straightforward, and reliable. Also, none of this “we don’t support Linux — can that even connect to the internet?” garbage. They know their stuff.
There is one dropbox-like system called iFolder: http://ifolder.com/ifolder
The downside is that you have to run your own server, but then the data stays under your control.
…An elegant little tool, from a more civilized age…
May I ask why you found DropBox itself insufficient? IIRC it has a Linux client.
>May I ask why you found DropBox itself insufficient? IIRC it has a Linux client.
The 2GB quota for free users is too low for my needs. If I have to pay for the service, rsync.net’s technical base and guarantees seem stronger.
Long time lurker here…this posting is perfectly timed for a situation we are having at a lab at work. Thanks for all the info.
I’ve been using Amazons s3 (simple storage service) via s3sync: http://http://www.s3sync.net/wiki for several years. Amazon’s s3 is a pay service, but is very cheap for small amounts of data. Check it out.
Not sure where to put it so I’ll ask here:
you’ve been pretty happy and excited about android (I think it suxx from technical PoV but nevertheless) – what do you think of recent MeeGo-related (which so far looks awesome to my mind) announcements?
>what do you think of recent MeeGo-related (which so far looks awesome to my mind) announcements?
If you attach this question to one of my smartphone posts, where it belongs, I might answer it.
I haven’t had a problem with nested git repositories. I can’t speak for other VCSes though.