The following is a very lightly edited version of email I wrote to my apprentice Ian Bruene after he wrote documentation for his new Kommandant project that was, alas, as awful as I generally expect from programmers. I’m not training Ian for mere coding competence; he’s too talented for that and anyway I have higher standards. This is my way of insisting that he do documentation well – and it was he who suggested it would make a good blog post.
Here’s how to write narrative documentation for a thing like Kommandant.
You want to do this well because people will pre-judge the quality of your software by how clearly you can write about it. They’re right to do so; explaining it clearly is their best warrant that your thinking was not muddy and confused when you wrote it.
In fact, writing good documentation is an excellent way to ensure that you really understand the problem space you’re in, and to throw light into corners of your software where defects might lurk. Do not underestimate the power of this effect! Often enough to matter, it will save you from serious embarrassment.
Doing this right needn’t be difficult. The quality of your documentation is like the quality of your code – less a result of how much effort you put in than it is of having the right mental habits to begin with.
One important mental habit is to not be terrified by the blank sheet of paper. One of my goals in giving you a procedure to follow is so when you start something like this you can do it on automatic pilot. That will make it easier to deal when you have to write the more difficult, more project-specific parts. Over time you will evolve the procedure to suit your tastes, and that’s fine. It’s meant to a springboard, not a straitjacket.
First choose a title. Make it simple and direct. Use the name of the product in it. The phrase “how to” may occur. Examples: “Writing Applications with Kommandant”, “Kommandant Client how-to”.
Then write an introductory paragraph that explains why the product exists. A good section title might be “Introduction” or “Motivation”. It is often good for the first, topic sentence to read something like “Foo is a library for doing Bar”. If the product is derived from or modeled on another piece of software, nod in that direction. If there are examples of similar products your userbase is likely to know about, mention them.
The reader should exit the introduction with a clear sense of how the software will help him and why he cares.
Next, a section on theory of operation. This explains the problem your software solves in more detail, and describes the strategy it uses. Here is where you want to sketch the relationship between the Kommandant core and the class that implements the user’s commands. Don’t get too far into the API weeds here, but *do* establish terminology – like “core”, “helper” and the different categories of user-defined hooks.
Be clear about why things are done as they are. Sometime this is best done by nodding at alternatives. Like: “We could have required the user to explicitly register command handlers, but by introspecting on the client class we both simplify the problem and eliminate a potential source of defects. Python Cmd set a good example here.”
The user should exit the “Theory of operation” section grasping the concepts he needs to understand a detailed description of the UI/API.
The next two sections should be a detailed API description and a set of simple worked examples with explanation. Which order to do this in depends on your product. For something as simple as Kommandant, examples first probably works better. On the other hand, if the examples are necessarily so complex that they’re hard to read without having seen the API description first, they go second.
(If you find yourself in the latter situation, consider it as a clue that your design may be overcomplicated and need a rethink.)
The reason it’s good to put worked examples first if you can is that expository arc in a document like this should always try to supply *motivation* at each step before requiring the reader to do more mental work. The examples are motivation for grappling with the API description.
(Notice that my explanation has followed the same kind of expository arc I’m describing…)
Because you’ll have an API reference nearby there is little need to annotate the examples heavily. Focus on details the reader might find surprising or a little tricky, like the difference between a basic-mode application and one that calls readline.
Your API section should walk through each call, ideally in the order they would occur in a typical application. You’re trying to do two things here: fill in details and tell a story about how an application uses your library
Next section: tricks and traps. Anticipate what might confuse your user about the API and address it directly. A good example in Kommandant is the fact that if you want to customize your prompt, it’s not enough to do it in the PreCmd hook; you have to do in PreLoop too, or the very first prompt won’t be what you expect. Explain why each gotcha is the way it is.
Next section: Credits and connections. Who contributed to this? Who sponsored it? What was its proof-of-concept application?
Finally, a brief revision history. Tie the document to versions of your software. Summarize changes. A user who is revisiting this document should be able to look here and tell what parts to read for the updates.
Once you have written a document like this, a one-stop shop for what users need to know, it attracts updates from them the way your software attracts patches. *You* will find it easier to keep in sync with your software than if its information were scattered across multiple files.
Now read and think about this document
http://www.catb.org/gpsd/client-howto.html
as an example of the form. You don’t need to write anything so long or elaborate, because Kommandant’s problem domain is simpler.
When you get this right, the result will feel a little like art; this is not an accident. I’ve been using words like “story” and “narrative arc” for a reason. By engaging the human brain in the way it wants to be engaged, you lower the friction cost for your users of acquiring the information they need. This is a functional virtue and they will love you for it, giving your software long legs.
You have an advantage here. You have a sharp sense of humor that pervades what you write and say. Let that work for you.
Now go do it right; make me proud.
Some time later Ian popped up on IRC saying
* | ianbruene has enough teeth into HOWTO to start enjoying the process
To which I replied:
esr | ianbruene: *Excellent.* That was part of my evil plan.
Which of course it was. You’ll never write decent documentation if you think of it as a deadly chore. You need to learn how to take pride in your documentation and enjoy the process so your users will, too.
UPDATE: And this is what Ian wrote after absorbing the above advice.
@esr: Then write an introductory paragraph that explains why the product exists.
Bless you. I’ve lost count of the number of offerings I’ve seen, shareware, freeware, and open source, starting in the MSDOS days and going forward, where I couldn’t find a simple description of what it was and why I might want to use it.
I don’t believe any of them still exist, and that lack is a major reason why.
>Dennis
The 2018 version – web pages pitching hosted software applications that require you to create a username and password without first providing that simple description of what it does and why you should care (maybe everything will be explained clearly once you’re logged in, but I’ll never know…)
Yes! what the heck is up with that? in the early days of the web lots of clueless business owners thought a website was basically a pamphlet and I’m not happy to see this useless idea making a return.
Almost as boggling as websites I’ve come across which are allegedly trying to sell common goods such as. clothing but won’t let me see them until I’ve created a login. Do the proprietors of these websites not realise they’re up against competitors, some of whom are actually not placing obstacles in my way?
Hmmm… Needs a simple up-vote function in the comments.
Routinizing documentation creation sets stones on the path to the flow/play state when writing; the appropriately-minded writer, in the flow state and utilizing human deep psychology via use of narrative structures, then has the potential to produce words that echo in the reader, assisting him in achieving a concordant flow/play state and, subsequently, freeing the mind to not merely understand the project but also to more easily envision its potential applications.
Or something. Seems to suggest documentation as code’s mirror twin, functionally as well as conceptually. That is, documentation in whatever form – man page, README, or discoverable interface – should serve to assist the user in entering the ideal mental state for interacting with the software. A ritual in every repo. I assume TAoUP is an attempt at this kind of documagic at a meta-project scale.
I notice how your formulation resembles a good man page. Not an accident, I’m sure.
If the documentation is written before the code is, it forms a specification for the coders to implement. Of course, the map is not the territory, so all regressions should be resolved either by fixing the code or rewriting the documentation.
I’m thinking about one particular application I admin where I’ve kept updating my documentation to reflect the ever-changing reality on the ground. I can look at the current state alongside the initial architecture diagram and note that they bear virtually no resemblance whatsoever.
A narrative documentation of how narrative documentation works — neat!
Cursedly recursive?
One of my best friendships was formed when GJ and I were teamed in a class project to write a baby operating system for an IBM 1120 (I think). He was a whiz programmer, I was a plodder – I could code but much slower. He said “I’ll do it, don’t bother me.”, I said “I’ll understand it and do the documentation”. So we did.
This is not a bad approach, with the caveat that any code you can’t understand should be rewritten until you can. Note that I didn’t say Smart Guy™ should explain the code to you. If he has to explain it, he’s coding too cleverly for the codebase to be maintainable.
That was excellent.
Word
LaTeX
I’m curious enough to ask, and this seems on-topic enough:
Eric, how do you manage to have the sort of vocabulary that includes words like “gallimaufry”?? I think it’s not quite enough that you read a lot; it’s apparently a matter of what you read. (Like, I read for hours a day, but it’s primarily non-fiction, which probably explains why I don’t escape the confines of a – I’m guessing – 25,000 word bolus.)
>Eric, how do you manage to have the sort of vocabulary that includes words like “gallimaufry”?? I think it’s not quite enough that you read a lot; it’s apparently a matter of what you read.
I dunno. I always thought it was reading a lot, myself.
My “a lot” has been dominated by science fiction and non-fiction. The non-fiction has heavily featured hard sciences, history, anthropology, linguistics, and sources for comparative religion and mythology.
I probably picked up “gallimaufry” from historical fiction, in which I have a minor but significant interest. I think it’s not so much that other people don’t encounter words like this as that they’re less likely than I am to retain them and integrate them into production (and not merely recognition) vocabulary.
Broad and frequent reading certainly helps but I think that sort of exposure is only half the story. The other half is one of linguistic skill: how sharp of a distinction do you retain on words—for example, without looking up either definition, do you know the precise difference in meaning between saying a thing “glitters” or “glistens” (or how both differ from the much more common “shines”)? The phrenic distance one places on synonyms determines if, when encountering a new word, you see it as filling a “gap” [perhaps not just in meaning, but also grammatical usage or prosody] in your existing vocabulary or simply being duplicative. In my opinion, the best touchstone for what sort of skill you possess with language is probably the range of metrics you wield in this consideration, followed shortly by enjoyment of language play—not puns, except perhaps as true equivoque, but other forms such as alliteration—and rhetorical flourishes such as ploce.
This touches on a long-time demand I’ve had for:
1. a reverse dictionary – given the meaning, provide the term – that is higher quality than what is currently available
2. a thesaurus that explains fine distinctions between synonyms; e.g. why or where you would use big vs. enormous vs. stupendous vs. brobdingnagian
This might as yet be hopeless pie in the sky, but I also keep hoping for an app that automates this to some degree. Using WordNet as a reference, perhaps, and producing a higher level editing tool. Even if it’s for entertainment. Can you imagine taking a paragraph of mundane text and generating purple prose for it? I mean, there are already apps that generate deconstructivist essays and Thomas Friedman articles…
>do you know the precise difference in meaning between saying a thing “glitters” or “glistens” (or how both differ from the much more common “shines”)?
Interesting example. The distinction was immediately available to me, but it’s not something I knew that I knew.
If something nearby glitters, it has specular reflection from small facets or grain-sized inclusions, but does not reflect over its whole surface; also a distant object may be said to glitter if it is emitting light but atomospheric conditions cause random variations in perceived luminance. If it glistens, it has a liquid or semi-liquid surface layer that may reflect light. If it shines, it may be doing either light reflection or light emission.
I think the only way one really learns these is by reading a lot of use contexts.
What’s also sort of intriguing is that the fact that these words have retained these precise denotations implies that they are generally only used by people who understand the denotations. If 90% of the people using “glisten” thought it was a mere synonym of “shining”, it would be a mere synonym of “shining”.
I think the ability to make these fine distinctions is one of the most obvious hallmarks of intelligence.
>I think the ability to make these fine distinctions is one of the most obvious hallmarks of intelligence.
Psychometricians agree with you. Vocabulary in one’s primary language is the most g-loaded measure of intelligence known.
We home school, and saw a presentation by a teacher on Poetics. His main claim was that most of poetics is taught as rhyme / rhythm / meter / scansion what have you, when it poetics education should be about picking the right words for the meaning / emotion the poet is trying to evoke. And that poetics can be worked into daily use, not just as a special form. He pointed out a lot of great speeches (Gettysburg Address, MLKjr, Kennedy, Reagan, US Founding Documents) all are quite poetic, even though the aren’t poems. And that in documentation (!), business memos, advertising and even just reporting can all be made much more effective if the writer is versed in poetics.
Of course, he was selling a poetics course…
Still, the idea that effective word choice can enrich AND clarify writing has stuck with me as self-evidently true. (Aside: George Carlin on Shell Shock gradually morphing to PTSD is depressingly funny).
How to write documentation in a way that when future humans want to process them by code, like doclifter, they won’t hate your guts? Please add this.
LaTeX
I think Eric set out here to separate the content from the mechanics of writing documentation.The techniques he mentions apply equally to a manpage and an HTML readme.
It’s the examples that make documentation immediately useful. Alphabetical lists of cryptic API calls are fully as useful as Doxygen robo-commented spaghetti code, which has a sub-Torr level of asymptotic suckatude. Approaching the docs as a tutorial, rather than as a recitation of features that the coder was forced to write up, is what sells the code.
One alteration I’d suggest is to group your API calls by theme and sort the groups by complexity of the application that uses them rather than sort ungrouped calls by the order an application would use them. For example, let’s say a typical workflow is to take input, manipulate it, and then produce output – it’s probably more intuitive to have an I/O subsection followed by subsections for increasingly arcane modification tasks than it is to have input, chronological modification calls, and then output. In the latter case, I will read it to build the application you describe and then tweak it until it does what I want – but in the former, I can have a design in my head or on paper and implement it directly. I also won’t miss any boilerplate needed to clean up, prevent memory leaks, etc., which I otherwise might if it’s buried at the bottom. And if I go back to add a new feature, it will be easier to find the commands I need.
>One alteration I’d suggest is to group your API calls by theme and sort the groups by complexity of the application that uses them rather than sort ungrouped calls by the order an application would use them.
It depends. The approach you’re describing is better for complex libraries in which there isn’t a stereotyped usage pattern. Kommandant is simple and has a stereotyped usage pattern, so a sequential walk-through works well.
I’m constantly looking up stuff in Vim’s help, and to the extent it has an API, it’s organized by theme, and it works well, since I’m often browsing features conceptually adjacent to the specific one that led me to seek help. So I’m in favor of grouping by theme.
This is for when you’re in the reference phase of documentation, however. It’s no longer really narrative – to extend the analogy, this is for when you’ve told the core story of your product, and the reader is now looking for related tales. (Here’s what’s been going on with all those secondary and tertiary characters…)
My usual way of thinking of intro docs is to start with the examples as Eric et al. have described, and solidify them with explanations of the features in order of how often they’re expected to be used. So in vim, for example, it might begin with insert and append mode, then movement by characters, then by words, then lines, then screens; then :w to save the file, then how to delete, then paste, then copy, then undo and redo; then how to search. To me, these are the most used features, so they come first. Then chapter 2 delivers the next most frequent ring of features, such as moving to matching delimiters, setting word wrap and tab size, how to join lines, and then how to set up a very rudimentary vimrc. These are all conceptually scattered features, except in terms of (hopefully, IMO) frequency of use.
My metaphor is that of an island. The reader is dropped in the middle. Once you’ve explained why he’s even there, you quickly give the reader the means to understand everything within arm’s reach, then within ten feet, then ten yards, and so on. Within reason and with a little luck, he knows the whole island, perhaps minus a few deep caves.
Also note that there are really three types of documentation to consider, and you need to be clear which one is being written.
1 – tutorial/intro to explain how to use this to people who are new, either to the idea itself or to this expression of it. This needs to explain the material and flow naturally.
2 – reference manual for people who use the tool, regularly or irregularly. The important thing here is to make sure that it’s as easy as possible to look up the information on each command, and that each entry is as complete as possible in itself to minimize flipping-around.
3 – update for a new version, “what’s new”?
Obviously with a totally new program type 1 documentation takes priority, but one may need to consider over the evolution of a program when to add type 2 or 3 documentation.
Sounds like another Master Foo entry: Thus, was Ian enlightened.
>Sounds like another Master Foo entry: Thus, was Ian enlightened.
Funny idea, but I don’t think this lesson quite fits in the koan form.
Eric, I don’t know what you would call this kind of ‘HOWTO’ (probably a recipe, to achieve a specific task, not fully documented elsewhere but each individual component can be found in different locations and may be in a different context) http://harishankar.org/blog/entry.php/go-how-to-use-glade-files-as-resources (I sometimes document these kinds of ‘little’ things for myself, as much as for others, since it serves as a handy reference at some later date)
I would like your feedback on how I can improve. I deliberately wanted to keep it simple and not repeat instructions found elsewhere, rather only wanted to give general pointers.
>I deliberately wanted to keep it simple and not repeat instructions found elsewhere, rather only wanted to give general pointers.
Mission accomplished then. It made me intersted enough in go-bindata to go read up on that.
It seems that the link I shared is an unmaintained repository. But others appears to have the project picked up and actively maintained elsewhere as different forks.
This has the ring of a great and deep truth with wide applicability.
Especially when writing an API, I try to write the documentation before I’ve written the bulk of the code behind it. When I write the documentation, it forces me to think about how someone using the API will expect it work, which frequently alters my thinking about the fundamental structure and how things should be organized. Names of API routines, order and types of parameters, etc, seem to come to me more intuitively when I switch my thinking into this mode too.