irker is feature-complete

I’ve just shipped irker 1.8, and I think this brings the wild ride I’ve been on for the last eleven days approximately to a close. I consider this release feature-complete; it achieves all the goals I had in mind when the CIA service died and I decided it was up to me to rescue the situation. I expect the development pace to slow down a lot from the almost daily release I’ve been doing.

The last really major feature was irkerhook support for Mercurial repositories. I’d be mildly interested in a bzr extractor class if anyone wanted to contribute one, and that probably wouldn’t be hard – the git and hg extractors are about 70 lines each. But with git, hg, and Subversion covered it’s good enough.

Uptake of irker continues at a pleasingly rapid pace. There’s now a second symbiote application, a poller daemon that watches the log of a specified Subversion repository and uses irkerd to ship notifications from it. This can be useful if you don’t have write access to the repo hooks and thus cannot install irkerhook.

Time for a pause and some reflection on lessons to be learned.

I think the most interesting aspect of the project, after I got the basic irkerd design in working shape, was hardening the implementation against denial-of-service attacks. I got some high-powered volunteer help with this, including from A&D regulars Daniel Franke and Peter Scott. Reasoning out the attack paths and the simplest possible countermeasures was just plain fun, possibly more fun because I’ve never had to do this kind of analysis before.

The willingness of hackers to step up and contribute to a project like this continues to be a wonderful thing about which I hope never to become jaded. Because this project spun up so fast and still doesn’t have a mailing list, I know many of my contributers mainly by IRC nicks. Thank you, AI0867, birkenfeld, KingPin, laurentb, dak180, nenolod, and everybody else who showed up on #irker to help and contribute and critique and report bugs and ask questions. It has been a pleasure working with all of you and watching a healthy micro-community form around this project within hours of my first release.

irker does have some has competition for its goal of replacing the CIA notification service. Some Debian people dusted off a project called “KGB”; I gave its principal designer a bit of a hard time when he showed up in A&D’s comments because KGB is heavier and more elaborate than it needs to be, but in truth it’s not actually horrible. CIA’s spaghetti architecture was horrible; KGB is merely somewhat overweight.

KGB is also, judging by the traffic on freenode #commits, losing the adoption race. They got to the party late with software that’s more difficult to grok and get running than irker is. Though, to be fair, irker’s minimalistic design might not have gained an advantage without also having better documentation. I never consider high-quality documentation a mere optional extra on my projects. This is certainly a lesson more developers could stand to learn…

I will admit, however, that one KGB feature I initially scoffed at turned out to be sufficiently useful and lightweight that I added it – that is, support for color-highlighting notication lines. The key to that decision was that I found a way to implement it in a handful of lines of code in irkerhook.py; the feature doesn’t touch irkerd at all.

This is an example of a larger theme in irker’s design – policy/mechanism separation. The irker daemon is pure mechanism; it has no options or control knobs of any kind (other than one to enable debugging messages and another to dump the version and exit). It’s just a message bus. All the policy stuff (choices about what to put in a notification) lives in irkerhook.py. And you damn betcha that I plan to keep it that way!

Because all irkerhook.py does is gather information that it ships as JSON, the addition of one simple option – to dump the JSON to stdout rather than trying to ship to the configured irker instance – makes changes in the the policy stuff very easy to test. Which is as unlike the huge, nigh-untestable and thus extremely failure-prone hairball that was CIA as it is possible to be and do an even remotely similar job.

Generally, policy needs to change more often than than mechanism. So, when you partition your system into a mechanism part (irkerd) and a policy part (irkerhook.py), the policy part is the part where there is greater need for testability. Or, to put it differently: when you partition your code and you find that the least stable part is most amenable to testing, you’re doing it right.

Yes, this is the old-time Unix gospel of minimalism and design for testability I’m preaching here, brothers and sisters, and I’m doin’ it for a reason. The total line count of irkerd and irkerhook.py code is 689 LOC (measured by sloccount), compared to 1957 LOC for KGB and probably tens of thousands of lines for CIA (I haven’t measured that). When you pay proper attention to separation of function and separation of policy from mechanism, your code gets not larger but smaller. With virtuous consequences, the most important of which is fewer failure modes.

Comprehensibility helps deployment, too. It’s not that I think every project administrator who has adopted irker has actually read the daemon code, but I’m certain its smallness and lightness – and the fact that you can read through irkerhook.py and grok what what it’s doing in just a couple of minutes – has made it an easier sale to people who are chronically short of time and attention to get all their tasks done.

26 comments

  1. Sing it, brother.

    Minimalism and focus (and an itch to scratch) are what it takes.

    This might not make much sense to people without the experience, but every morning when I heat up a pot of water, I marvel that the designers of my stove thought a ‘cancel’ button was a good idea.

  2. > There’s now a second symbiote application, a poller daemon that watches the log of a specified Subversion repository and uses irkerd to ship notifications from it.

    It would be nice if there was a place where those symbiote applications were listed, for example irker homepage, or “SEE ALSO” section of irkerhook(1) manpage.

    1. >It would be nice if there was a place where those symbiote applications were listed

      install.txt in the distribution.

  3. >> It would be nice if there was a place where those symbiote applications were listed

    > install.txt in the distribution.

    That’s nice and good.

    It would be even better if HTML version of install.txt were linked to on irker homepage (http://www.catb.org/esr/irker/), just like security.txt and hacking.txt are.

  4. Comprehensibility helps deployment, too.

    Yes, yet it does. Helps the poor fools who (I jokingly call them ‘webmonkeys’) are responsible for keeping things up and running, too. Thank you!

  5. The only feature I can imagine wanting in irker that seems likely to require a change to irkerd is the ability to authenticate with irc servers; even though this was not something that CIA did I would consider it a more useful feature for irker because it does not use a centralized server and therefore a predictable address.

  6. The only feature I can imagine wanting in irker that seems likely to require a change to irkerd is the ability to authenticate with irc server

    Given that irkerhook.py will need to be the driver of this (at a guess i’d start with the tried and tested username@host structure), how would you securely set-up the password? Is .loginrc still considered safe (and fair game)?

    I would consider it a more useful feature for irker because it does not use a centralized server and therefore a predictable address.

    Wouldn’t this give it a lower priority for irker than for CIA? Reason being that allowing CIA you, by necessity, automatically allow in any project that wants to spam your IRC server. Whereas if you allow in a specific irkerd then the project would need access to that irkerd (see security.txt about the assumed security model).

    1. >Whereas if you allow in a specific irkerd then the project would need access to that irkerd (see security.txt about the assumed security model).

      Password authentication would be a rather bad fit for irkerd’s usage pattern. The problem isn’t that credentials would be difficult to pass it – I could always add an optional password field to the JSON. No, the problem is that once irkerd has a credential, it would have to do source-address IP checking to know whether the source host of any given notification request is the same as one that has presented the password. I’m not gong to go there; the potential for access controls becoming leaky seems too high.

  7. No, the problem is that once irkerd has a credential, it would have to do source-address IP checking to know whether the source host of any given notification request is the same as one that has presented the password.

    Assuming that we’re talking about the same thing, wouldn’t it be enough to require each notification request to send the credentials that it wants its message to be posted with? If the dispatcher map was extended to include login credentials then “irc://irc.somewhere.com:6669” wouldn’t match “irc://irker:p4ssw0rd@irc.somewhere.com:6669” which also wouldn’t match “irc://irker:wrongp4ssw0rd@irc.somewhere.com:6669”.

    There is a deeper issue as well. From memory(which is a good 10-20 years old) irc itself didn’t care about authentication. That was deferred to a separate component which killed any connection which didn’t send the magical bits within a defined time limit.

    1. >wouldn’t it be enough to require each notification request to send the credentials that it wants its message to be posted with?

      Might work. I’ll take a look at the relevant portions of the IRCs.

  8. Just to correct myself, my 6:50pm comment should mention .netrc rather than .loginrc.
    Not that it matters if no-one is disturbed by passwords being sent to irkerd.

  9. > There is a deeper issue as well. From memory(which is a good 10-20 years old) irc itself didn’t care about authentication. That was deferred to a separate component which killed any connection which didn’t send the magical bits within a defined time limit.

    This separate component is NickServ bot.

    But it should be not a problem, thanks to irkerd automatically dealing with nick collisions, and I hope also with being kicked out by NickServ, by semi-randomly changing password. Well, unless channel is configured in such way to give voice only to users registered with NickServ. Hmmm…

    1. >Well, unless channel is configured in such way to give voice only to users registered with NickServ.

      If that happens, irkerd will fail. Fixing this problem is out of scope for the project.

  10. Just read the source, a nice 20-minute romp, as a lesson in what good code documentation looks like. What isn’t made obvious by the code itself is spelled out cleanly. Not knowing much about what’s under-the-hood for IRC, it was easy following along with what was happening. Also gleaned some insight into a better function structure, aka dealing with eveything that could go wrong before getting to the meat. I’m sure that’s my noobishness showing, but kudos nonetheless, Eric.

  11. @Morgan

    True; like I said, I know next to nothing about the innards of IRC. I was thinking more generally in terms of dealing with abnormalities (malformed input, missing resources, etc) before doing what the function was meant to do. I have a bad habit of doing it the other way around, and it’s gotten me into trouble in the past. Fortunately for the rest of the world, this is my hobby, so the worst that happens is one toy program or another fails in weird ways, and I improve in direct proportion to how much time I’m able to put into it.

  12. There is a deeper issue as well. From memory(which is a good 10-20 years old) irc itself didn’t care about authentication. That was deferred to a separate component which killed any connection which didn’t send the magical bits within a defined time limit.

    This separate component is NickServ bot.

    But it should be not a problem, thanks to irkerd automatically dealing with nick collisions, and I hope also with being kicked out by NickServ, by semi-randomly changing password. Well, unless channel is configured in such way to give voice only to users registered with NickServ. Hmmm…

    Yes and no. By and large, the problem of user authentication is delegated out to the NickServ service bot on most IRC networks. But there is a mechanism that can be used for requiring authenticaion on connection. Usualy, this is reserved for server connections ’cause having to hand edit the ircd.conf file for every new user sucks. But I’ve seen it done for users well.

    The problem of authenticating Irker aginst NickServ should be as easy as sending irker a message in the form of {“to”: “irc://irc.freenode.net/nickserv,isnick”, “privmsg”: “identify supersecretpassword”}, as all intereaction with the service bots is done with the same PRIVMSG command used to send private messages to users and public messages to channels.

    Connection-time passwords would be trickier. Per the interenet draft linked to in irker’s source, an IRC url can be appended with ,needpass to indicate that a connection will require a password. The message passed to irker would look something like {“to”: irc://irc.freeenode.net/commits,needpass“, “privmsg”: “Hello World!”}. Per the linked draft, the user should be prompted for the password before attempting to dereference the URL. I’m not sure how you’d do that with irker, since it’s just a simple relay. Though not conforming strictly to the draft, a url in the form of irc://irker:password@irc.freenode.net/commits,needpass” might do the trick. It follows the form used in other URLs, at least. In fact, I think one could argue that ,needpass is unnecessary with the username and password in the URL like that.

    Then there’s the issue of channel passwords (irc://irc.freenode.net/commits,needkey), which has the same prompting issue as connection-time passwords. I’m not sure how to address that.

    Admittedly, this all hinges on the assumption that one wants irkerd to be able to authenticate to irc networks where such authentication mechanisms are required by the admins.

  13. /me smacks forehead.

    I think I’m done posting for now. I’m going back to my corner.

    An “Edit” button would be glorious.

  14. On this collective guilt stuff: it is actually about something slightly different, but still not a good thing.

    Basically, in normal logic, doing something wrong means using whatever power you have in a wrong way, for a wrong purpose, using wrong methods and so on.

    Now these kind of Critical Theory folks want us to think that having power is in _itself_ wrong.

    There is some kind of logic to it – of course people having power over others is often not a desirable social outcome, but their mistake is jumping from desirable social outcomes to personal morality – they cannot be matched 1:1.

    As a thought experiment, if someone has the powers of a totalitarian dictator, but does not actually use them, is he still ethically in the wrong merely by having that power i.e. not abdicating? By moral intuitions say that even though this situation is socially undesirable, for us, for his it is either not a moral wrong, or a very small one, clearly much, much smaller than actually using that power. I also have the feeling it is related to the gun debate – is it morally wrong to be able to (have power to) kill easily? I say no – a good friend of mine was killed on his motorbike by an idiot car driver, yet we would not say all car drivers are doing something wrong.

    It is an interesting dilemma – basically it means that even though you may be in a situation that is socially not desirable, not the ideal utilitarian outcome, there is still no moral obligation for an individual to change it. This is the case with almost every kind of power. We want kings to abdicate but we can’t really morally blame them for not doing so.

    Anyway, it would be interesting for someone to take a more formal philosophical analysis of it: when and why or why not is an individual morally obligated to abdicate from some kind of power which is not socially useful or desirable for him to have?

Leave a Reply to Jakub Narebski Cancel reply

Your email address will not be published. Required fields are marked *