{"id":7032,"date":"2016-02-17T18:39:16","date_gmt":"2016-02-17T23:39:16","guid":{"rendered":"http:\/\/esr.ibiblio.org\/?p=7032"},"modified":"2016-03-09T10:32:29","modified_gmt":"2016-03-09T15:32:29","slug":"automatons-judgment-amplifiers-and-dsls","status":"publish","type":"post","link":"http:\/\/esr.ibiblio.org\/?p=7032","title":{"rendered":"Automatons, judgment amplifiers, and DSLs"},"content":{"rendered":"<p>Do we make too many of our software tools automatons when they should be judgment amplifiers? And why don&#8217;t we write more DSLs?<\/p>\n<p>Back in the Renaissance there was a literary tradition of explaining natural philosophy via conversations among imaginary characters.  I&#8217;m going to revive that this evening because I had an IRC conversation this afternoon, about the design insights behind reposurgeon, that pretty much begs to be presented this way.<\/p>\n<p>The person of &#8220;Simplicio&#8221; was Galileo&#8217;s invention in his <cite>Dialogue Concerning the Two Chief World Systems<\/cite>.  Here he represents four different people, but almost everything he says is something one of them in fact said or very plausibly might have.   I&#8217;ve cleaned it up, edited, and amplified only a little.<\/p>\n<p>For those of you coming in late, reposurgeon is a tool I wrote for editing version-control histories.  It has many applications, including highest-quality repository conversions. Simplicio needed to excise some security-sensitive credentials from a DHS code repository &#8211; not just from the tip version but from the entire history.  Reposurgeon is pretty much the only practical way to do this.<\/p>\n<p>So, without further ado&#8230;<\/p>\n<p><!--more--><\/p>\n<p>SIMPLICIO pounce-hugs ESR.<\/p>\n<p>SIMPLICIO: I have to run back to work, but I just wanted to say&#8230;reposurgeon is FREAKING AWESOME.<\/p>\n<p>ESR: I take it you figured out how to do the necessary.<\/p>\n<p>SIMPLICIO: Yep.  How did you *imagine* that?<\/p>\n<p>ESR: I love designing DSLs (domain-specific languages).  What you are seeing as &#8220;awesome&#8221; is the result of proper attention to keeping the language primitives in the DSL mutually orthogonal, so they can have fruitful combinations I did not anticipate. This is a design style that is difficult to do well, but when you pull it off the payoff is <em>huge<\/em>.<\/p>\n<p>SIMPLICIO nods.<\/p>\n<p>ESR: You might be entertained to know what the model for reposurgeon&#8217;s DSL was.  Brace for it: &#8230; ed(1).<\/p>\n<p>SIMPLICIO: LOL<\/p>\n<p>ESR: I&#8217;m not joking. Think about how reposurgeon selections interact with the command verbs.  Pick a collection of records, do something to it &#8211; possibly with auxiliary following arguments.<\/p>\n<p>SIMPLICIO: I know you&#8217;re not joking, it&#8217;s still amusing.  Heh. The original Patriarchs of Unix were truly worthy of their mantles.<\/p>\n<p>ESR: There were two big insights behind the design of reposurgeon:<\/p>\n<p>(1) Attempts to fully automate repo-conversion tools are doomed by ontological mismatches between VCSes. Bridging those requires case-by-case human judgment; <em>therefore<\/em>, the best tool should seek to amplify human judgment rather than futilely attempting to remove it from the process.<\/p>\n<p>(2) The structure implied by a deserialized git-fast-import stream resembles a line sequence in an editor <em>just enough<\/em> that the orthogonal ed model of &#8220;apply a command verb to a selection&#8221; is applicable.<\/p>\n<p>SIMPLICIO: Everything old is new again.<\/p>\n<p>ESR: Everything since those insights has been in some sense mere details.  In particular, mating premise (2) to the properties of tne Python Cmd.cmd library class implies quite a lot of the reposurgeon implementation.<\/p>\n<p>But premise (1) suggests a larger question: where else are we <em>making the same mistake?<\/em> Are there other domains where we should be trying to write judgment amplifiers rather than automatons?<\/p>\n<p>If I ever again write a DSL as effective as reposurgeon it will be because I found a specific answer to that question. I would love to do this again and again and again.<\/p>\n<p>SIMPLICIO: Blog post, or a new chapter for <cite>The Art of Unix Programming<\/cite>?<\/p>\n<p>ESR: Hm.  Blog post for sure.  Not sure premise (2) is Unix-specific enough to deserve a chapter in TAoUP.<\/p>\n<p>SIMPLICIO: I was thinking of &#8220;judgment amplifiers vs. automatons&#8221;.  That demands to be a chapter title. :-)<\/p>\n<p>ESR: It&#8217;s a good design question to notice.  Whether it&#8217;s a <em>Unix<\/em> design question is another matter.<\/p>\n<p>SIMPLICIO: Seriously, I don&#8217;t understand how to know when a DSL is necessary.<\/p>\n<p>ESR: I&#8217;m pretty sure &#8220;when is it necessary&#8221; is the wrong way to frame the question. &#8220;When is it possible&#8221; would be a better one.<\/p>\n<p>SIMPLICIO: That may be my problem then.<\/p>\n<p>ESR: If you can figure out a proper set of orthogonal primitives to build it around, a DSL is <em>always<\/em> better than a more rigid design.  At worst, it becomes one of the soft layers in an <a href=\"http:\/\/c2.com\/cgi\/wiki?AlternateHardAndSoftLayers\">alternating hard and soft stack<\/a>.<\/p>\n<p>If I have a DSL, I can front-end it as a GUI or some other kind of more rigid interface. But the reverse is not true; if you don&#8217;t design in DSL-like flexibility to begin with, it&#8217;s almost impossible to retrofit.<\/p>\n<p>SIMPLICIO: That does make sense. In the past I&#8217;ve compared DSLs to more general-purpose programming languages and mainly seen their limitations.  Now&#8230;I&#8217;m intrigued.<\/p>\n<p>ESR: A good example is basically any modern E-CAD package.  Look past the GUI and you&#8217;re going to find some kind of DSL for hardware descriptions underneath.  Going directly from the GUI&#8217;s data representation to silicon would be doomed, but the soft layer in the middle gives it a way to chunk the design process that captures domain primitives like logic gates, vias, or entire functional blocks.<\/p>\n<p>SIMPLICIO: Oh. Oh! I bet you&#8217;re going to bring up SQL next.<\/p>\n<p>ESR: I certainly could.  Mathematica, there&#8217;s another one.<\/p>\n<p>Yet another example is Emacs.  You have to sort of forget that Lisp is theoretically general-purpose for a moment; if you do you&#8217;ll see the same hard-over-soft pattern, DSL underpinning something that doesn&#8217;t look like one.<\/p>\n<p>This is an <em>extremely powerful<\/em> way to design. You&#8217;d see it more often, but there&#8217;s no tradition about teaching the practice.  So programmers have to re-invent it nearly from scratch almost every time they do it.<\/p>\n<p>SIMPLICIO: If you know of any good teaching materials, I&#8217;d be very grateful. If not, I&#8217;ll go googling at some point.<\/p>\n<p>ESR: I <em>wish<\/em> there had been teaching materials when I was a noob &#8211; I had to spend a quarter-century learning how to do it right.  Sadly, there still aren&#8217;t; all we have is a handful of people who audodidacticated themselves. RMS.  Steven Wolfram.  Me.  A few others.<\/p>\n<p>SIMPLICIO: There are a bunch of things I wish there were teaching materials for.  I&#8217;ve noticed that if they&#8217;re engineering-useful but not interesting to academics they tend not to get written.<\/p>\n<p>ESR: The really hard part is carving the domain operations into orthogonal primitives. Then you sort of clothe them in semi-generic DSL machinery.<\/p>\n<p>So, listen to this carefully: the reason git fast-import streams were essential to the design of reposurgeon is because they concretized the problem.  They reduced the abstract question &#8220;what is an orthogonal set of primitive operations on repository histories&#8221; to a more concrete one: &#8220;What is an orthogonal set of primitives for editing the attributed graph implied by a fast-import stream?&#8221;<\/p>\n<p>The first question, the abstract one, is fundamentally difficult because it&#8217;s ill-defined.  The second question, the concrete one, has an answer that may be somewhat complex but is well-defined and not fundamentally difficult. You can get it by staring at diagrams of nodes and links, and thinking up every possible way to screw with them.<\/p>\n<p>SIMPLICIO nods.<\/p>\n<p>ESR: But for a great many interesting cases, any answer to the second question implies an answer to the first.  You write your attributed-graph editor and you <em>have<\/em> a repository editor.<\/p>\n<p>SIMPLICIO: That makes sense :)<\/p>\n<p>ESR: Export\/import to actual repositories is still an issue of course but it&#8217;s one you can keep well isolated from the rest of the design.<\/p>\n<p>SIMPLICIO: Is there some way to generalize reposurgeon&#8217;s design pattern?  I think I get it now, but I don&#8217;t see how you map it to other application domains.<\/p>\n<p>ESR: A first thing to notice is how the agenda of amplifying human judgment rather than fully automating fits with writing a DSL.  You&#8217;re not really writing a judgment amplifier if the tool is incapable of doing things the designer didn&#8217;t anticipate.  You need the flexibility, the ability to generate and collect options.<\/p>\n<p>A second thing is that you can get a hell of a jump on grasping the problem domain well enough to write a DSL over it if there is some kind of declarative markup that captures all of its entities.  Then there&#8217;s a mapping &#8211; mathematicians would call it a functor &#8211; between operations on the markup and operations on the problem domain.<\/p>\n<p>So I&#8217;d say a good first question to ask is: is there a declarative markup &#8211; an analogue of git fast-import streams &#8211; that captures everything in the problem domain I want to hack?  And if not, <em>can I invent one?<\/em><\/p>\n<p>The process of knowledge capture that needs to happen for such a markup to exist is exactly the one that will tell you, or at least imply, what the primitives for your DSL are.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Do we make too many of our software tools automatons when they should be judgment amplifiers? And why don&#8217;t we write more DSLs? Back in the Renaissance there was a literary tradition of explaining natural philosophy via conversations among imaginary characters. I&#8217;m going to revive that this evening because I had an IRC conversation this&hellip; <a class=\"more-link\" href=\"http:\/\/esr.ibiblio.org\/?p=7032\">Continue reading <span class=\"screen-reader-text\">Automatons, judgment amplifiers, and DSLs<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[33],"class_list":["post-7032","post","type-post","status-publish","format-standard","hentry","category-software","tag-reposurgeon","entry"],"_links":{"self":[{"href":"http:\/\/esr.ibiblio.org\/index.php?rest_route=\/wp\/v2\/posts\/7032","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/esr.ibiblio.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/esr.ibiblio.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/esr.ibiblio.org\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/esr.ibiblio.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=7032"}],"version-history":[{"count":6,"href":"http:\/\/esr.ibiblio.org\/index.php?rest_route=\/wp\/v2\/posts\/7032\/revisions"}],"predecessor-version":[{"id":7053,"href":"http:\/\/esr.ibiblio.org\/index.php?rest_route=\/wp\/v2\/posts\/7032\/revisions\/7053"}],"wp:attachment":[{"href":"http:\/\/esr.ibiblio.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=7032"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/esr.ibiblio.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=7032"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/esr.ibiblio.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=7032"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}