In response to a bug report that was relatively easily fixed, I’ve just shipped release 2.8 of doclifter, a program that takes troff-based document markups – including man page markup – and lifts them to DocBook XML.
In doclifter I’ve come as close as I probably ever will to building an AI :-). Automatically lifting the grotty presentation-level goo in man pages (and other troff macro sets) to structural markup is hard – actually, the DocBook user community thought high-quality translations without extensive manual intervention by a human were impossible until I did it. But it turns out that clever parsing and a whole lot of cliche analysis are good enough for about 97% of the real-world cases, and doclifter can throw useful warnings for the other 3%.
This is, by the way, a useful tool even if you’re not interested in DocBook, because DocBook is kind of like Earth orbit – it’s halfway to anywhere. In particular, man to DocBook via doclifter followed by
Docbook to HTML with the stock stylesheets produces better HTML than any of the half-dozen direct man-to-HTML converters out there. This is because none of them actually do much structural analysis – they’re mostly converting presentation-level cliches in troff to presentation-level cliches in HTML. They’re also deficient in handling troff special characters, which doclifter maps to XML Unicode literals.
If you run an open-source project, and your documentation masters are still in troff, please use doclifter to fix this (yes, you will be able to make man pages from the XML after lifting them). That’s why I wrote it – every project that switches to DocBook XML improves our ability to present good-looking and properly hyperlinked documentation over the Web.
If you haven’t seen it, you might want to look at Pandoc: http://johnmacfarlane.net/pandoc/
“If you need to convert files from one markup format into another, pandoc is your swiss-army knife. Pandoc can convert documents in markdown, reStructuredText, textile, HTML, DocBook, or LaTeX to
HTML formats: XHTML, HTML5, and HTML slide shows using Slidy, Slideous, S5, or DZSlides.
Word processor formats: Microsoft Word docx, OpenOffice/LibreOffice ODT, OpenDocument XML
Ebooks: EPUB
Documentation formats: DocBook, GNU TexInfo, Groff man pages
TeX formats: LaTeX, ConTeXt, LaTeX Beamer slides
PDF via LaTeX
Lightweight markup formats: Markdown, reStructuredText, AsciiDoc, MediaWiki markup, Emacs Org-Mode, Textile”
I’ve just begin to play with it, but it looks quite promising. (Written in Haskell, no less.)
Doclifter to get things *to* DocBook, and PanDoc to get them to other things might be a nice combo.
>If you haven’t seen it, you might want to look at Pandoc: http://johnmacfarlane.net/pandoc/
I have. The problem with Pandoc, which is generic to all tools going through an intermediate markup, is that translation quality is bounded by whatever subset of the source language(s) the intermediate markup can express. So you can end up with situations like: DocBook supports tables, ODT supports tables, but the translator doesn’t cleanly map DocBook tables to ODT tables. (I don’t know that this is the case – maybe Pandoc supports tables just fine – but it’s pretty certain that there are problems of this kind.)
OT (but somehow related): i have a question about the hacker mindset behind such feats.
To someone like me, coming to amateur programming from formal theoretical linguistics, very abstract programming languages, and especially functional languages, are exciting, even intoxicating. Before discovering lisp I already knew about the lamba-calculus from my semantics courses, but I had no idea that that formal system could be turned into a tool that _did_ things!
On the contrary, low-level languages, not to mention task-oriented languages like troff proved too much for me. The main obstacle was boredom: on the basis of your own writings (howto be a hacker etc.) I started to study my C, perl and so on, but even the simplest of scripts proved mind-numbingly boring. Even shell-programming was too awkard for me, after lisp.
I can imagine the amount of detailed knowledge of arbitrary facts a program like this requires and I wonder how you can reconcile programming like this with the more abstract reasoning that programming in languages like Lisp or Python offers, and you yourself advocate.
>I wonder how you can reconcile programming like this with the more abstract reasoning that programming in languages like Lisp or Python offers, and you yourself advocate.
Well…doclifter embodies a huge amount of grubby domain-specific knowledge (your one extreme) and it’s written in Python (your other). Sometimes the job requires one extreme, sometimes the other, sometimes both at once. You use the tools that the task requires.
Check out http://mdocml.bsd.lv/ It can output very nice (X)HTML/PDF/PS.
@esr:
>I have. The problem with Pandoc, which is generic to all tools going through an intermediate
What stack would you recommend for someone writing documentation that will need to be generated into PDF, HTML, XHTML, man etc..?
>What stack would you recommend for someone writing documentation that will need to be generated into PDF, HTML, XHTML, man etc..?
Today? I’d recommend asciidoc for simple documents and DocBook for more control over complex ones.
DocBook is just one kind of SGML/XML document. It’s not a meta-format. Of course, that’s the reason why DocBook cannot be directly comparable to say OpenOffice/LibreOffice ODT because ODT is just a file format while DocBook is the format (XML/SGML) + the structure, and makes it difficult to map structure correctly all the time.
I think it’s unreasonable to expect a single structure to map every kind of document out there. That’s why generic document conversion tools won’t be satisfactory in every sense (say when a non-structured document is required to be mapped correctly into a structured format – there just isn’t enough context for the tool to do the job)
Off-topic:
Young People Like Tech but Not Enough to Work in IT
“About 97% of teens and young adults between the ages of 13 and 24 say they like tech. But only 18% of them expressed strong interest in a career in information technology, according to a recent study released by CompTIA. About 26% of male respondents said they are definitely interested in an IT career while only 9% of females said the same.
“If the industry thinks it’s difficult to find workers for certain kinds of IT jobs now, just wait a few years. The Bureau of Labor Statistics projects that computer and information technology jobs will grow by 22%, adding about 758,800 jobs by 2020. Workers in these occupations will be needed to develop software, increase cybersecurity and update existing network infrastructure, according to BLS. During that same period, the labor force is expected to grow only 6.8%.”
http://blogs.wsj.com/cio/2012/06/27/young-people-like-tech-but-not-enough-to-work-in-it/