doclifter 2.9 is released

I’ve released doclifter 2.9, and as part of that process I’ve been testing it on the entire collection of manual pages on my system again. Because doclifter does mechanical translation of troff-based markups to DocBook-XML, one of the side effects of testing it is that I find lots of broken markup. I’ll ship over 700 fix patches back to maintainers this time, though maybe not until after I get back from World Boardgaming Chapionships next week.

Release here, report on markup bugs found is here. Yes, over 700 patches, but that’s actually a drop from previous passes.

My last rampage through the man-page universe with fire and sword was in 2007. Most (I’d say about 85%) of the patches I shipped then were accepted. One particularly noticeable change is that in 2007, only a handful of pages identifiably had DocBook masters and could thus be skipped; in 2012 fully 7% of the entire corpus is like that.

Which is good news – why, at that rate, we’ll be fully converted before the end of this century. :-)

(For any of those who are wondering what the practical consequence is, think Web availability. DocBook renders into HTML much more cleanly than conventional manual-page markup does.)

20 comments

  1. We’ve started creating manpages for the Xen Cloud Platform’s xe command and we’ve chosen to use DocBook as the the source format and XSLT to create the html, pdf and manpages. Do you see this as the best way forward for a new project? I see a lot of tex formatted documentation out there.

    1. >Do you see [DocBook] as the best way forward for a new project?

      Yes. Tex is nice for typesetting mathematics for print, but it’s not nearly as Web-friendly as DocBook.

  2. >> Do you see [DocBook] as the best way forward for a new project?

    > Yes. TeX is nice for typesetting mathematics for print, but it’s not nearly as Web-friendly as DocBook.

    Well, there is always latex2html, which produces quite nice HTML… though I cannot compare it to DocBook to HTML conversion.

    There is always option of using lightweight markup language with man and HTML output (and info, and PDF) like AsciiDoc.

  3. Is there any human-readable (e.g., RST-ish) format that could possibly replace DocBook for man page sources, in your opinion? This reminds me a bit of the initial attempt to make single-master etexts for Project Gutenberg using TEI (a very, very heavy-weight XML encoding system); efforts have moved to a slightly-extended RST, which is much easier to work with and very nearly as powerful. (The missing pieces, like sidenotes, are unimplemented, but not inherently impossible.)

    (There’s an rst2man project out there with some samples.)

    1. >Is there any human-readable (e.g., RST-ish) format that could possibly replace DocBook for man page sources, in your opinion?

      Yes, asciidoc. It generates DocBook but has much lighter-weight markup resembling wiki markup or RST. As with RST you gove up some power. The advantage over RST is that you co-opt the DocBook back ends.

  4. Does the toolchain for converting from Docbook to hardcopy suck any less than it did ten years ago when I last dealt with it?

  5. Documentation should be read proportionally spaced, and written proportionally spaced. It should contain headers, bullets, and such.

    Creating documentation in and for mono-spaced ASCII is archaic, and should be treated as a legacy burden that we are trying to escape from as rapidly as possible. Back when more flexible formats, such as Microsoft Word, were all proprietary, and the only satisfactory way to write HTML was in a mono-spaced unicode editor, (because all available wysiwyg tools produced abominable source) this was (barely) defensible, but now that there are free tools that edit HTML as wysiwyg, and produce HTML source code that does not suck, everyone should produce documentation that is actually readable, using a tool that displays it readably.

    And, similarly, the literate code movement sucked because the literate source code was written in monospaced ASCII, though designed to be read by humans in more readable format.

    As literate programming was envisaged by Knuth, you would edit your literate code in unreadable monospaced ASCII source, which is then compiled into two representations, one designed to be read by humans (weave), and the other by computers(tangle) – but all your debug information necessarily references the one designed to be read by computers, requiring you to be edit the tangled version during the debug session.

    Thus for literate programming to work, you need to be able to decompile the tangled version back into literate source, and thence into the woven version. Similarly, for the documentation to be writeable, you need to be able to edit the woven version in a wysiwyg editor, and have it be decompiled back into source, and thence into the tangled version – which implies there is no reason why we should ever see or touch the literate source version.

  6. I think personally that, even today, standards complaint and structured HTML with CSS is easier to write in code than in WYSIWYG. Mainly because of stylesheets having made things simple for separating content from presentation, WYSIWYG offers no significant advances but might cause headaches by introducing inadverdant local style overrides. E.g. a tag might be inserted when you meant

    Perhaps not all documentation benefits from being written in a higher level structured format like DocBook XML or SGML, but it still makes sense because of toolchain availability, open standards and facilities for conversion. Though I detest the verbosity of XML or SGML like formats, I think the idea behind DocBook is, in principle, sound.

    Monospaced font or lack of it is a different issue and I don’t think it is preferred in any printed or printable format, but for reading on a computer terminal or even e-book reader without the advanced e-ink facilities, either monospaced or clean Sans Serif fonts work better.

  7. The above comment ate up my HTML tag example:

    I mean <span style=”align: right”> might be introduced instead of <span class=”signature-line”>

  8. I must be weird, then, because I like monospaced fonts better than proportional for a lot of things. Perhaps it’s because I find it easier to do things like put comments above/below a line of code that point to specific elements to help understand it better.

    Or I’m an old-school typist who sees the value in being able to present information in tabular format very easily by judicious use of spacing. I set my work email to use Courier New as the default font so that I can do that sort of thing.

  9. Aha; asciidoc–yes, that looks about right. Maybe it’s just an aesthetic preference, but seeing people use DocBook as a master format for, for instance, manpages gives me a feeling of inelegance and bleah. Far be it from me to complain about the modernization of the X.org documentation for R7.7, but something about heavyweight XML formats being used as hand-edited masters just doesn’t sit well with me.

    Are my aesthetics totally out of sync with everyone else’s here? Is it a hardcore-systems-programmer vs Python-user thing?

    1. >but something about heavyweight XML formats being used as hand-edited masters just doesn’t sit well with me.

      For me it depends on the length and complexity of the document. There’s nothing I’d rather do a book in than XML-DocBook, but I prefer asciidoc for short pieces with simple structure.

      I think XML-DocBook is a very good choice for the X documentation.

  10. grendelkhan, in our case we have a lot of output formats so writing in a “source” format that converts well to others is key. Hard coding a manpage and then converting it to html, pdf, epub etc. doesn’t make a lot of sense. So the $64 question is what “source” format should we use. Docbook seems pretty clean and XSLT works wonderfully.

  11. Grant: asciidoc has automated conversion to manpages, HTML, PDF, and even ePub. Like I said before, it’s mainly an aesthetic choice, but it is possible to use asciidoc (or RST, or something else wiki-markup-ish) as a master format.

  12. BobW> Do you set your color scheme to bright green on black?

    No. Although my Coronamatic had swappable cartridges, including a green one for when I wanted to type in green, I generally used white paper and black ink, which coincidentally is the color scheme I use in emails.

    I do, however, set my desktop background color to black, but I rarely see much of it, on any of my monitors.

    It also bugs me that people will take a GRAPHICAL screenshot of a text-mode app such as a telnet client, paste it into an MSWord doc, then attach the doc to an email that itself has heavy GUIfication, instead of doing a text-mode copy/paste of the screen to capture the same information in orders of magnitude less bytes. But I’m also a recovering BBS sysop who had to pay LD to connect to the hub on [redacted]net, and grew up listening to my parents tell me stories of how far they stretched everything during the Depression.

  13. @The Monster

    I set terminal window color schemes to green-on-black.

    My typewriter ribbons were all black-on-white. Every so often I miss my old Hermes 3000. I never had an electric typewriter.

    I don’t understand people sending bitmaps instead of characters either. It seems like so much trouble.

    On the other hand, most people just want to do their work with as little thought as possible about anything peripheral. For most people computers are peripheral concerns at most.

  14. > Docbook seems pretty clean and XSLT works wonderfully.

    Meh. Maybe I’m too old-school. I’m with grendelkhan and The Monster on this. I think that a source document should itself be eminently readable, easily editable with any text editor, easily diffable with any diff utility, and easily version controlled with any VCS, even with tables and whatnot, so I tend to use rst.

    We may be a minority, but we’re not alone:

    http://www.goer.org/Journal/2010/11/xml_not_great_for_documents_actually.html

    BTW, I sometimes contribute to the rst2pdf project, which wraps docutils and reportlab into a pure Python system that can generate PDFs from rst.

  15. Yet more people agree:

    http://yannramin.com/2011/06/22/semantic-end-user-documentation-docbook-or-sphinx/

    Me, I was burned a long time ago on non-ASCII proprietary programming formats such as LabView and Lotus Notes. I do schematic stuff, too, but I always consider the source to be ASCII.

    Yes, XML is ASCII, and yes, it’s better than those other things. But if XML is your source document, then you really need special editors, especially if you want mere mortals to edit the source.

    At work, I have a system that generates lots of techical stuff from a single source. In general, the source format is excel spreadsheets. I hate spreadsheets, but I support this because these are what the users (who are all hardware engineers) want. But I want an intermediate format that is more suitable for viewing and version control than either the spreadsheets or XML, so my first step is a conversion from the spreadsheet into a customized RSON format.

  16. Thank you for your continued work in this area. It will be of huge benefit to the unix ecosystem, particularly for newcomers. I look forward to the day where there is beautiful, clear, complete unix documentation available on the internet.

Leave a Reply to Jakub Narebski Cancel reply

Your email address will not be published. Required fields are marked *