Some years back I wrote a book titled The Art of Unix Programming. My goal in that book was to convey the Zen of Unix to today’s generations of eager young Linux and *BSD programmers. In the spirit of that book, I feel impelled to point out out a program I’ve recently learned as a striking, near-perfect example of Unix style in the modern day. rsnapshot, you’re doing it right!
rsnapshot is a filesystem snapshot utility for making backups of local and remote systems; you save your stuff with it. I can describe it pretty well by just listing the ways that it fulfills classic Unix design patterns.
1. Everything the program does is driven by a single configuration file in a plain-ASCII text format – easily readable by the Mark One Eyeball, easily editable without special tools.
2. The configuration file is a specification in a declarative minilanguage. That is a bit different and stronger statement than point 1; it implies that, rather than being a series of tweaks of opaque parameters that cannot be understood without intimate knowledge of the rsnapshot code, the configuration is largely self-describing given a certain fairly minimal understanding of the application’s domain (making backups).
3. rsnapshot delegates everything it can and doesn’t reinvent wheels. The actual backup engine is rsync, possibly assisted by ssh (two bog-standard Unix tools every sysadmin will have installed). Backup scheduling is handled by putting rsnapshot jobs in a cron file. Notifications to the sysadmin are done through normal email.
4. While high performance of the backup engine is important (and rsync is tuned for it) performance of rsnaphot itself is not. So it’s written in a scripting language for best flexibility and maintainability. Perl wouldn’t have been my first choice, but for this application it is a big win over C or any compiled language.
5. rsnapshot doesn’t have a GUI because it doesn’t need one; the real “interface” to it is a text editor running on its config file. If somebody wanted to give it a GUI, the GUI code could be a separate small program that simply does structured modifications of the config file while remaining completely isolated from the logic of rsnaphot itself. Such separation lowers global complexity, reduces bugs, and increases the options to improve both programs.
6. Mechanism is clearly separated from policy. In this particular case, the portions of the configuration language that describe the tools to use for doing backups are clearly separated from scheduling and the specification of archive locations.
7. rsnapshot run on a config file with a syntactic or other detectable error emits a useful error message including the line number of the error to standard error. The real point is that errors lead to more understanding rather than some bland “A problem occurred” pointing nowhere in particular.
8. rsnapshot includes a -t (test) option that allows you to see the rsync and other copy commands a given configuration would emit without committing them. It also includes a -V option to show more detail about how these actions were generated. These options, especially in combination with the previous point, make the program discoverable; they lower the cost of becoming familiar with it.
9. rsnapshot makes as few wired-in assumptions as it can get away with. This future-proofs it against, for example, changes in the preferred backup tools. If rsync were to be obsolesced by a faster and better file-tree-synchronization tool tomorrow, rsnapshot installations would be using it without fuss the next day.
10. rsnapshot isn’t just good, it’s unobtrusively good. The design doesn’t repeatedly hit you over the head with how clever it is or try to impress you with flashiness signifying nothing. It is spare, clean, elegant, and gets out of your way. This is correct Unix style!
I’m not saying rsnapshot is absolutely perfect – I could quarrel with the reliance on tabs as field separators in the config file, even though the parser is helpful about telling you when you’ve used the wrong kind of whitespace. But it’s near perfect, and a worthy model for programmers trying to improve their Unix style.
“I could quarrel with the reliance on tabs as field separators in the config file,”
I read your objections against tab-separated values in the Unix book. And I must admit that I have been bitten several times by the silent conversion of tabs to spaces.
But I always end up using tabs again as the alternatives end up unreadable. At least tabs give a reasonable layout and they are automatically used by cut. And normal text does contain comma’s and semi-colons.
The problem isn’t allowing the use of a tab as a separator, the problem is in requiring a tab to be used rather than a space. Proper Unix tradition holds that tab and space characters should be treated more or less equivalently, ala /etc/fstab or /etc/hosts.
This can be problematic when space characters might be present in paths, however. For example, if you have, say, a CIFS path in your fstab like ‘//server/this is my share’, you’ll have to write it using the ugly octal notation like ‘//server/this40is40\my40\share’. This gets worse if there is a number in the path, like ‘//server/this is my share 2’ -> ‘//server/this40is40my40\share402’
Probably this is why they allow only tabs and not spaces.
Wow, did WordPress mangle that or what? where you see ’40’, should be ‘\ 0 4 0’ without the spaces.
Hi Eric,
I am the original author and former maintainer of rsnapshot. To hear this coming from you is high praise indeed. I am truly honored.
By the way, I was definitely reading The Art of Unix Programming right around the time I was originally writing the first versions of rsnapshot. It’s a great book!
Thanks!
>By the way, I was definitely reading The Art of Unix Programming right around the time I was originally writing the first versions of rsnapshot.
I’d ask whether you learned relevant design lessons from the book or they just reinforced what you already knew, but I don’t actually think the answer to that question matters very much. I didn’t create the tradition, I just transmitted it. Credit belongs rather to the originators of the Unix style, and to you for having the good taste and judgment to execute it properly.
>line number of the error to standard output.
Standard output? Why not standard error?
And did you mean to have two number 2 comments?
ESR says: Good catches, thanks. Now fixed.
One thing I’ve noticed is missing from the configuration file is a way to ‘#include’ other files. Perhaps I’ve been spending too much time admining Apache and Debian/Ubuntu boxes lately, but I think all configuration files should allow some way to include other configuration files. This lets you do dynamic things, like having a central configuration for all machines on mounted NFS or CIFS filesystem that #includes local changes. This is very necessary in larger multi-server environments.
Hi Morgan,
The include_conf parameter in rsnapshot.conf allows you to include additional files, and for exactly the purpose you described.
Pet peeve: If you know enough to complain this specifically, you probably know enough to fix the problem. If so, why complain?
>Pet peeve: If you know enough to complain this specifically, you probably know enough to fix the problem. If so, why complain?
Because I’m not actually certain it’s a design bug yet. There might be a constraint I’m unaware of; Morgan hinted at one possibility.
This is somewhat offtopic, but are you still working on “Why C++ is not my favorite programming language”?
>This is somewhat offtopic, but are you still working on “Why C++ is not my favorite programming language”?
Still stalled on my collaborator. I may have to give up on having a collaborator.
Woo Hoo! Nathan Rosenquist might be worthy!
When is he going to write a book?
Because other people who don’t know how to fix the problem will be bitten by it.
Treating tabs and spaces differently is very much against the Unix tradition that all sequences of one or more whitespace characters are syntactically equivalent.
I often email/IM snippets of shell commands to co-workers. I will go out of my way to put in something like
`printf '\t'`inside a string where a tab is desired, just to make sure that there’s no problem with the comm software re-rendering the tab as spaces. Fortunately, it’s very rare to need to do that.As to filenames with embedded spaces, you should be able to either quote or escape them:
“/home/monster/This is a filename with spaces in it”
/home/monster/This\ is\ a\ filename\ with\ spaces\ in\ it
If your config file parser can’t handle those, it’s not written very well.
There are real security concerns here. Note that ‘;rm -rf X’ is a valid filename in Unix.
Filenames and Pathnames in Shell: How to do it correctly
David A. Wheeler
http://www.dwheeler.com/essays/filenames-in-shell.html
I see I was insufficiently precise:
@Jay Maynard:
As Eric mentioned, he hasn’t thought about enough to figure out if it’s a design bug. It might be that there is more than one possible user mistake that maps into the same parse error. If that is indeed the case, then, as The Zen of Python says, “In the face of ambiguity, refuse the temptation to guess.”
Indeed. I believe I’ve mentioned here before the really nasty trick of doing
touch ./-rfin a directory, which then makes
rm -i *not do what you meant for it to do, as “
-” sorts before pretty much everything else.Of course, the reverse is true as well:
echo "You really should be more careful" >./-iwhich will override the -f switch in
rm -f *This precaution can be handy in an important directory that is difficult to quickly restore from backups.
But when it comes to config files, I just don’t like using a tab as a separator. I’ll use | if I have to have whitespace inside fields, because there is rarely a need for it in a field (but quoting/escaping/both should be provided if there is any chance it could be a valid character within a field), and it visually suggests separation of fields, making it easier for the humans who edit the files to grok. In essence, selecting a good field separator character is also good UI.
@esr:
@Winter and Monster:
That’s what ‘–‘ is for.
As in
rm -i — *
As Wheeler not quite states outright in the blockquote, it’s not the shells that are broken, it’s their users brains that are broken. Unix shells provide great power, but with that great power comes great responsibility to ensure that you really, really do know what you’re doing.
Unfortunately, in a mixed Windows/Unix environment, Windows admins with Microsoft brain damage like to have spaces in CIFS share names and directories. So you sometimes end up having to have spaces in fields in files like /etc/fstab.
Note: That em dash should be two hypens, obviously. More WP mangling…
Actually I prefer
rm -i ./*
Much simpler and works better in most cases. But still – yes, this aspect of Unix (no forbidden symbols in filename and flags in most programs plus wildcards expansion is shell, not in a program) is really troublesome.
P.S. May be it’s good idea to add yet-another-option to bash ? If “*” will expand to “./-rf ./foo ./bar” it’ll break few badly written scripts but will much safer… and most scripts should really never use globs with * in first position…
But that still doesn’t mean you have to use tabs as field separators. You can either quote or escape those fields with spaces in them, or use a character like | as a field separator.
First off, whitespace is the best delimiter and should be used wherever possilble. It’s clean and makes reading very, very easy. Why do you suppose that so many programming languages — and all Unix shells — use it as a delimiter?
Actually, I think this is one thing Unix gets wrong. While Unix — and programs written in the Unix tradition — uniformly uses plain ASCII text as a standard file format, there is little uniformity otherwise. For example, /etc/fstab, /etc/hosts, crontabs, etc., all use space and tab as a field delimiter, but /etc/passwd and /etc/group use the ‘:’ as a field separator. Some files get really weird, like /etc/nsswitch.conf, which requires that the name of each database ends in a colon (‘:’), but otherwise uses tabs and spaces and field separators. (WTF?)
Then there’s files like Samba’s smb.conf and wgetrc that use name = value pairs. And /etc/termcap and /etc/printcap, which variously use ‘|’, ‘:’ and whitespace as a separator.
I won’t even talk about the gods-awful ugliness of the Xdefaults file. I refused to use an X-based Emacs for years because of that stupid confusing bletcherous pile of garbage.
And, of course, some well-meaning but very misguided individuals have tried to fix all this inconsistency by using everything from XML to SQL databases as configuration files.
If we’d all just use simple mini-languages like rsnapshot here, we wouldn’t have all this confusing mess.
@Morgan
I think we’re talking past each other on the issue of “whitespace-delimited fields” vs. “tab-delimited fields”. Using “whitespace” (a series of one or more consecutive spaces, tabs, and other whitespace characters not including linefeed) as a field separator is GOOD. Singling out a tab as a delimiter, despite the fact that the tab looks like a bunch of spaces, is BAD.
The reason why different config files use different field separators is that there are different constraints on what can go in the fields. Take
/etc/passwd. The GECOS/comment field is expected to have spaces in it, and it is not the last field (as is the case with the sole field in crontab expected to contain spaces), whitespace is a bad choice for field separator.In practice, when you have one of these files in a text editor, there is no question what the delimiter is (unless it’s one that uses Tab as delimiter so as to allow spaces inside fields). I have no problem moving back and forth between whitespace/:/| delimiters.
I don’t have a problem with all the different styles of config files, provided that they are all editable in any text editor (SQL DB fails this test). I see nothing “misguided” about using XML for config files. The tools to parse XML are ubiquitous, and the tags can be designed so as to be very self-documenting, which is always a Good Thing.
Why not? It’s the sole field allowed to have spaces. It was a purely arbitrary decision. /etc/passwd looks like it does for purely historical reasons.
True enough, but do you remember learning Unix for the first time? Didn’t the /etc/passwd file seem scary and untouchable? (Or was it just me?)
Parsing an XML file correctly, however, requires a metric crapton of code.
“But that still doesn’t mean you have to use tabs as field separators. You can either quote or escape those fields with spaces in them, or use a character like | as a field separator.”
Who volunteers to do the helpdesk for such a system? Remember this application is also used by Windows admins and other people without a strong Unix background.
But that code exists in multiple reusable forms. You don’t have to reinvent that wheel.
No, but you do have to link that code. In many cases, the fewer external dependencies you have, the better.
I’m not opposed to using XML files for what they were intended for — data. I’m just opposed to using XML files for configuration for systems where it makes no sense.
@Jay Maynard:
> But [XML parsing] code exists in multiple reusable forms. You don’t have to reinvent that wheel.
This is absolutely true. The problem of XML is not the readers — it’s the writers. You either have to create your own writer utility, or direct your users to use a special schema-aware editor to write the XML, or (and I’ve seen this way too many times) expect your users to directly enter XML into a regular editor.
A huge benefit of XML is that it was designed to be human-readable, but it’s a mistake to believe that implies that it’s very human-writable.
Unfortunately, I think the heavy use of Java and the mindset that goes with it has helped contribute to the misconception that XML is human-writable, because judicious use of XML actually makes Java systems much more dynamic than they would be otherwise, and the mindset of the typical Java programmer is that Java is his hammer and XML is his saw — two complementary, indispensable tools.
Yes, that’s it. That’s the biggest problem. Getting users to write XML that actually complies with the schema and is actually XML-compliant isn’t so easy. Handling such failures is even more tricky. If you’re using XML for data, it’s not such a huge problem because your program is writing the XML programmatically. But using XML for configuration is, IMHO, a very big mistake.
@Morgan
The : character can’t be a part of a local user ID, so it was a sound choice as field delimiter. But once that historical arbitrary decision was made, it couldn’t be unmade without breaking existing tools that parse the file. That’s perhaps a downside, part of the price paid for the Unix Tao’s embrace of human-readable config files; once you’ve created a format, you can’t just change it without providing some mechanism for backward compatibility for programs that don’t know about your change.
It was just you. The very first time I looked at
/etc/passwdI was 100% comfortable with what I saw. In fact, the only real confusion I’ve ever had was in the different implementations of GECOS/Comment sub-fields (separated by commas). I sometimes forget which ones go with which *nix.I know I can’t resolve the debate between tabs and spaces, but I can describe the reasons why I originally decided to go with tabs in the rsnapshot config file.
I have worked in a lot of mixed computing environments (Linux/Unix, Mac, Windows). While I’m sure it’s possible to put a tab in an important file or directory name, I have yet to see it done. On the other hand, I have frequently seen spaces, single quotes, double quotes, backslashes, backticks, exclamation points, hash marks, curly and square brackets, and all sorts of other things show up in file and directory names, especially on Mac and Windows. While good Unix admins always name everything in lowercase without any spaces, almost anything goes on other platforms.
Additionally, rsnapshot allows the system administrator to specify custom arguments to both rsync and ssh. Some of these arguments can sometimes contain either single quotes or double quotes.
By forcing the use of tabs, it completely sidesteps any and all escaping issues. Essentially, the only restriction now is that you can’t have a tab in a directory name that is set as a backup point (although you still can have files or directories under that backup point contain tabs in their names, of course).
Since rsnapshot typically runs from cron, I thought it would be easier for people to know that, if they are willing to use tabs as the official separator, they can avoid any escaping issues anywhere else. `rsnapshot configtest` will tell you if the config file is valid, and once you get that sorted out, you can rest assured that what you see in the config file is exactly what you get. To me it seemed like a good trade-off. And hey, Makefiles have the same restrictions, and probably for the same exact reasons. Wwould you really want to escape every single space in a Makefile?
If I were writing another program that wasn’t so intimately associated with potentially weird file paths and such, I would have gone with a more typical Unix tabs-or-spaces approach.
As for the potential security issue Winter describes above, I don’t believe it’s a concern in this case for the following reasons:
1) rsnapshot sidesteps the shell interpreting any special characters for the majority of the commands in the program. It does this by calling the Perl system() call with each command argument as elements in an array. When used in this fashion, Perl’s system() call avoids any character interpretation whatsoever and simply runs the command as executed. As a side note, if you need to remove a file called ‘ -rf /’, typing `perl` and hitting enter, then typing (without quotes) ‘system(‘rm’, ‘ -rf /’);’, hit enter, and then ^D is a pretty good way to go about it. You could also use unlink() to be even safer, which is how rsnapshot removes individual files whenever possible.
2) Even if rsnapshot did just pass commands all willy nilly to the shell, the only way a malicious path like that would be passed to the system to execute as a command line argument would be if the system administrator of the backup server configured rsnapshot.conf to use that path. Aside from the fact that it would be at the end of a path already, and wouldn’t be interpreted specially by the shell, the system administrator would only be sabotaging themselves. If a user created any files or directories with funny names like that, rsync would also just handle them without any special shell magic.
Thanks for the comments everyone!
> True enough, but do you remember learning Unix for the first time? Didn’t the /etc/passwd file seem scary and untouchable? (Or was it just me?)
I learned Unix recently enough that I’ve never needed to touch /etc/passwd.
@Nathan, I’d have probably done it by implementing string support in the configuration language. You’d have to escape quotes, but surely quotes aren’t as common as spaces. Or you could add heredoc support. But whatever, it’s easy enough to set my editor in tab-mode for just one file.
Oh. NICE (in both senses of the word). Thank you . From a ‘consumer’ POV, if it works as advertised, it will be very, very useful to my company.
rsnapshot has been my backup tool of choice for a couple of years now and I like it very much, except that I found this section:
#########################################
# BACKUP INTERVALS #
# Must be unique and in ascending order #
# i.e. hourly, daily, weekly, etc. #
#########################################
# an “interval” in this context is a unit of time defined by the time
# between the consecutive runs of rsnapshot (or the time between cron
# runs). The number is the number of backups to make with this time
# interval.
#interval hourly 6
interval daily 7
interval weekly 4
interval monthly 3
to be somewhat confusing until you it dawns on you that “daily” and “weekly” are strictly arbitrary labels and don’t mean what you think they mean to the program itself. It’s the cron job that gives the labels meaning.
I’ve been eager for this document as well.
X11 is the textbook definition of “gods-awful ugly”. The sooner we bin this monstrosity and get some sanity in the GUI realm the better.
This is why I was amused to find that the lead Wayland developers (and advocates on Fedora) are all Xorg lead devs. They know what they’re replacing!
I’m generally pretty impressed by anything Morgan has to say, but on
> First off, whitespace is the best delimiter and should be used wherever possilble. It’s clean and makes reading very, very easy.
I gotta call horsecrap.
Delimters *by definition* should be characters which are the least likely to appear in the data in the fields which you need to delimit.
I’ve been doing this, man and boy, for three decades, this year, and the number of times I have had a vertical-bar character in a data file that I (would have) needed to escape because/if that was the field delimiter …
is *once*.
It’s a valid ASCII7 character, but — except for pipes in unix shells and delimiters — I’ve never seen it used as part of data except just that one time.
Damned if I remember what it was; it was at least a decade ago, and I think it was a program-building tool of some kind.
Whitespace is *really* hard to do well as a delimiter, *because there are decisions to make* (same problem that Unicode parsers have, I gather); it’s not how well the bear waltzes, it’s that the bear has to waltz at all.
Most arguments in favor amount to “it’s easier to vgrep”. You’re not *writing* the config file for the humans; you’re writing it for the *program*.
Of course, since little languages *usually* beat out raw data config files anyway, it shouldn’t matter as much, right? :-)
So… surprising to me, I had to backup a directory where the name was ungodly because it was chuck full of spaces. so… I thought I’d give it a try, and it worked… so now I understand why the tab character is used in the config file. Here’s the my backup statement:
backup[tab]/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/[tab]samms-ubuntu/
the tab characters are represented as [tab] and the embedded spaces are the actual space in the fullpath directory