The CIA commit-notification service abruptly died two days ago, a development that surprised nobody who has been paying attention to the recent history of the codebase and its one public server site. A screwup at the cloud service hosting the CIA virtual machine irretrievably destroyed the instance data; please don’t ask me for details, I don’t know how it happened and don’t care. The CIA codebase is so messed up that even reconstituting a virgin instance would be way too much work – and that I will talk about a bit later in this post.
Fortunately, I saw this coming and had started work on a CIA replacement in late August. I had been holding off releasing it because there was some effort going on to salvage the CIA code, but that possibility effectively vanished when the only instance was erased. I shipped my replacement just a few minutes ago, and expect to spend much of the next week helping forge-site operators install it so we can have our notification service back.
The remainder of this post is a finished version of a design analysis of CIA I started a couple of weeks ago when the death of the service was still only a theoretical possibility. Since that theory has become actuality, the message should be heard loudly and clearly: this was a truly classic case of over-engineering, code bloat, excessive centralization, and bad practice. Read on for the cautionary tale.
I always liked the idea of CIA, the “version control informant” that relays commit notifications to IRC channels. I’ve maintained the (now obsolete) git hook scripts that talk to CIA for several years now. But recently I have been looking more closely at the design of CIA and how it’s implemented, and have concluded that it was a pretty horrible example of how not to do things.
First, a review of what CIA did and how it did it. What you saw from the outside when a CIA setup was working for a project was simple: whenever a developer commited code to the project’s public repository, the commit summary was shipped to an IRC channel associated with the project. It became part of that conversational stream, and was also echoed to a special channel (#commits on freenode) where you could watch all the open-source world’s commits flow by like a river.
A notification service like this is a very useful aid to collaboration. It makes IRC conversations among a development group more productive. It also does something unquantifiable but good to the coherence of the development groups that use it, and the coherence of the open-source community as a whole – when the service was live it was hard to watch #commits for any length of time without being impressed and encouraged.
Looking a little deeper, here’s what happened when a commit was made. The repository’s checkin procedure fires a “commit hook” – a small program, usually written in shell or Perl or Python – that is passed various metadata such as the commit’s ID, its list of files modified, and its change comment. The hook assembled an XML message in a particular format containing this information. It then used XML-RPC to call a central CIA server at cia.vc and ship it the notification.
The CIA server was then responsible for turning the XML notification into a text line that got shipped to the project channel and to #commits. It also updated a bunch of statistical summaries that could be browsed at the CIA website (now defunct).
Unfortunately, as in the old proverb about law and sausage, those who loved CIA notifications were best advised not to look too much more closely than this at how they were made. The service was notoriously subject to random outages and stalls; but that, bad as it is, is only symptomatic. Underlying this were several layers of unfortunate history, poor design decisions and shoddy implementation.
CIA hadn’t been actively maintained in several years before its collapse – the originator, one Micah Dowty, disappeared around 2007. One Karsten Behrmann, aka “BearPerson”, stepped in around 2008 but was unable to solve the problems with the software. The sole running public instance was hosted by a third party prone to loudly complaining on the #cia channel that the host box was an insecure hairball full of flaky and obsolete software that he couldn’t fix because the CIA code had dependencies on now-obsolete software versions.
That running instance is what’s now vanished. If you examine the repo of the CIA software, you’ll discover that it’s a mixture of parts in mostly Python but some Erlang, using (a) a custom web framework, (b) some Twisted, and (c) some Django. I’m told by people who have examined all this more closely than me that the individual subsystems (such as the Django code that generates most of the visible web pages in the site) aren’t too bad, but the interactions among them are messy and leaky.
The more experienced software engineers in my audience will already be getting a clue to what went wrong here, if not yet quite why. This is what software that has undergone a collapse into rubble under the weight of its own complexity looks like, complete with maintainers who have run away from their own inability to manage the resulting mess.
But the indictment wouldn’t be complete without noticing that their development practices sucked, too. My hair stood on end when BearPerson let drop on the #cia channel that the code in the running instance didn’t match the head state of the project’s CIA repository on GoogleCode – he admitted he’d “been lazy” and patched things on the site without propagating the changes back to the repo. I nudged him into fixing this. Or, at least, claiming to have fixed it, but the historical record didn’t do a lot to reassure me on that score. And it’s why those close to the problem have given up on attempting to resuscitate CIA without a running instance to look at.
Yes, before the VM was wiped there was a crew on the #cia channel trying to salvage the codebase. I helped a bit on this, but my estimate of their odds was never very optimistic. It is notoriously difficult to un-collapse a rubble pile, especially when the author and one previous rescue attempt (BearPerson’s) have already manifestly failed. Thus, I directed most of the limited energy I could spend on this problem into a different strategy.
That strategy began with asking why CIA suffered a complexity collapse, and whether much simpler code could do the job its users expect of it. This architecture diagram, proudly displayed on the author’s blog, is a pretty good clue.
But long before I saw this diagram there were several aspects of the design that seemed rather iffy. Why one centralized server? Why the elaboration of XML-RPC? Why do you have to register your project on cia.vc to use the notification, with the mapping from your project to IRC channels lurking in an opaque database on a distant server, rather than being simply declared in the (arguments to your) repository hook?
The answer seems to be that the original designer fell in love with the idea of data-mining and filtering the notification stream. It is quite visible on the CIA site how much of the code is concerned with automatically massaging the commit stream into pretty reports. I’m told there is a complicated and clever feature involving XML rewrite rules that allows one to filter commit reports from any number of projects by the file subtrees they touch, then aggregate the result into a synthetic notification channel distinct from any of the ones those projects declared themselves.
Bletch! Bloat, feature creep, and overkill! With chrome like this piled on top of the original simple concept of a notication relay, the resulting complexity collapse should no longer be any surprise. Additionally, this is a near-perfect case study in how to make your service scale up poorly and be maximally vulnerable to single-point failures – if that one database gets lost or corrupted, everybody’s notifications will go haywire. The CIA design would have been over-centralized even if the implementation weren’t broken.
Of course the way to prove this kind of indictment is to do better. But once I got this far in my thinking, I realized that wouldn’t be difficult. And started to write code. The result is irkerd, a simple service daemon. One end of it listens on a socket for JSON requests that specify a server/channel pair and a message string. The other end behaves like a specialized IRC client that maintains concurrent session state for any number of IRC-server instances. All irkerd is, really, is a message bus that routes notification requests to the right servers. (And is multithreaded so it won’t block on a server stall, and times out inactive sessions.)
That’s it. Less than 400 lines of Python replaces CIA’s core notification service. The code for a repo hook to talk to it is simpler than any existing CIA hook. And it doesn’t require a centralized server. The right way to deploy this thing will be to host multiple instances of irker on repository sites, not publicly visible (because otherwise they could too easily be used to spam IRC channels) but available to the repository’s hooks running inside the site firewall.
Filtering? Aggregation? As previously noted, they don’t need to be in the transmission path. One or more IRC bots could be watching #commits, generating reports visible on the web, and aggregating synthetic feeds. The only agreement needed to make this happen is minimal regularity in the commit message formats that the hooks ship to IRC, which is really no more onerous than the current requirement to gin up an XML-RPC blob in a documented format.
I must note one drawback to this way of partitioning things. Because IRC has a message length limit, naively shipping commits with very long metadata (due to for example, large lists of modified files) would make only a truncated version available on IRC (and thus, to an IRC watcher bot gathering statistics).
It might be that this was the original motivation for using an XML-RPC transport on CIA’s input end. Indeed, when I first recognized the problem I started sketching a design for a auxiliary daemon that would do nothing but accept XML-RPC requests in something very close to CIA’s preferred format, then forward short digests of them to an irker instance for shipping to IRC. This auxiliary could collect statistics based on the un-truncated metadata…
Fortunately, I experienced a rush of good sense before I actually started coding this thing. It would have hugely complicated deployment and testing to handle an unusual case – observably from #commits, most commit messages are short and touch few files. We get a much simpler system if we accept two reduction rules:
1. If a commit notification would be longer than 510 bytes, we omit the filenames list. An empty filenames list is to be interpreted by filtering software as “may touch any file in the project”.
2. Then…we just ship it. If the IRC server truncates it at 510 bytes, so be it. Humans watching the commit stream won’t need more than that to put the commit in context (especially not for projects which use git’s first-line-is-a-summary convention) and the hypothetical statistics-gathering bots won’t understand natural language well enough to care that it’s truncated.
This is how you keep things simple. And that is how you prevent your projects from collapsing under complexity.
I wrote irkerd to accomplish two things: (1) Light a fire under the CIA salvage crew, attempting to speed up their success, and (2) provide a viable alternative in case they didn’t succeed. To this I now add (3) illustrate what healthy minimalism in software design looks like. Antoine de St-Exupéry said it best: Perfection (in the design of software, as well as his airplanes) is achieved not when there is nothing more to add, but rather when there is nothing more to take away.
Accordingly, note the nonexistence of irkerd configuration options and the complete absence of anything resembling a control dotfile. I even, quite deliberately, omitted the usual option to change the port that irker listens on. Because if you think you need an option like that, you actually have a problem you need to solve at your firewall.
But releasing irkerd, of course, is not the end of the story. For it to do any good, instances of the daemon and its repo hook will need to be running and documented at sites like SourceForge, GitHub, Gitorious, Gna, and Savannah. As I noted at the beginning of this essay, I expect pushing the deployment along will eat up a lot of my time in the near future – probably more time than it took to write and test the code. These forge sites are all chronically understaffed and have long issue backlogs.
Still, at least we now have a simple and robust design, and working code. And – this can’t be emphasized enough – single-site outages will no longer be fatal. If there’s one thing the history of the Internet should have taught us, it’s that you get robust and scalable services not by centralizing but by distributing them. It’s too bad the designer of CIA never internalized that lesson, and there can be no better finish to this tale of failure than by reinforcing it.
UPDATE: I’ve changed my mind about statistics-gathering. I no longer think a bot watching the #commits channel is any kind of good idea – the notifications are too easily spoofed, or could just be garbled by software or configuration errors. If you want activity statistics, Ohloh shows the way – analysis tools operating on the repositories.
I understand the case for minimalism in software, but the lack of an option to change the port seems positively avant-garde.
@Nathan, quite a few months back, Eric dropped a comment here or on Google+ about getting a port number allocated from the appropriate internet authority. So, really, there’s no need to change the port. It’s not like you’d be running multiple copies with different configurations, or anything, or doing ‘security through translucency’ by running on a non-standard port. (It’s not even obscurity; anyone who doesn’t understand port scans has no business using the word security.)
I’m not familiar with CIA myself, but reading your description in paragraph 7-8, my first thought was “why is it trying to do it as message -> analysis server -> irc, instead of message -> irc -> analysis server or irc analysis server? That makes no sense.”
Out of curiosity, why truncate long commit messages instead of breaking them into multiple parts? And is there still a magic pointer to #commits, or are the clients expected to specify that?
>why truncate long commit messages instead of breaking them into multiple parts?
Someone nicely anticipated my answer later in the thread.
>And is there still a magic pointer to #commits, or are the clients expected to specify that?
Clients specify that. It’s an overrideable default.
It sounds like irker can be generalized to push most anything to an IRC channel, and the source needn’t be a VCS. Write a commit-hook script for whatever you are monitoring changes on, assuming that does something you *can* hook..
Agreed on truncating the commit message: if the user is sufficiently interested they can simply hop over to the repository and look at the actual commit records.
But I started to think “too complex and over-engineered” when I saw modules in “perl, python, Erlang, Twisted, and Django.” WTF? Yeah, if your different modules are suitably encapsulated and communicate via a defined message passing mechanism that is language independent, you shouldn’t have to care what host language a module was in. But this just sounds like people adding functionality they thought was desirable in a language they happened to know, with no strong maintainer to enforce sanity and see that complexity was minimized. Python for the daemon. Python or Perl for the commit_hooks. Anything else? Not unless you provide very convincing evidence it’s the only way to do it, and that what it is is sufficiently important to justify the variance. Better yet, just don’t go there.
>It sounds like irker can be generalized to push most anything to an IRC channel, and the source needn’t be a VCS. Write a commit-hook script for whatever you are monitoring changes on, assuming that does something you *can* hook..
That is correct.
I’d say the rationale is that you should be able to work out the context of the commit from 500 characters which is the point whereas you don’t really get anything by guaranteeing the full commit message at the cost of complicating the line protocol to the point that you might compromise the ability for humans to work out the context of the commit.
The lack of configurable port smells a bit to me like personal preference gone too far. It feels a bit strange, that on one hand, there’s a fixed port number for interoperability, but on the other hand, you recommend locking every single instance of the service down behind closed doors.
If it’s supposed to be private, why not let the entities involved set it up to fit their environment?
What if you have an existing service running on the port number? If every piece of software would go and reserve all the ports it uses, and everyone would honor the reservations, we would’ve run out of ports in the registry long ago. The only reason the port registration system “works” is because no-one really cares about it.
I guess that you can just force your way through inertia, and leave the painful reconfiguration or deployment of more machines to the end users.
xmpp > irc for this sort of thing but I guess few open source devs are using xmpp and have flocked to freenode. Other than freenode other irc usage have dropped 60% since 2003 and my favorite (grove.io) just went bye bye (it didn’t last very long…less than a year from launch to *poof*).
We tend to use some combo of redmine, jenkins, gerrit or the atlassian suite (free for open source last I checked) with git or svn and integrated with mylin with little reliance on irc.
It’s probably just a personal preference thing but I never really liked XMPP because sending messages over xml (when the message itself doesn’t need to be xml) seems like the Wrong Thing.
Nigel: XMPP support wouldn’t be that hard to add. The messages sent to irkerd are of the form
{"to": [url1, url2, ...], "privmsg": "Commit message here."}Those URLs are currently IRC channel URLs, like “irc://chat.freenode.net/commits”, but they could just as easily point to XMPP people or group chat channels (e.g. “xmpp:commits@myserver.com”). To make irkerd support this, you would need to add another couple of hundred lines of code (or so) to handle the XMPP connections. It would take some work, but not a lot of work. Crucially, it wouldn’t be a confusing nightmare where everything breaks because you touched it.
>To make irkerd support this, you would need to add another couple of hundred lines of code (or so) to handle the XMPP connections.
I’d take that patch.
>Those URLs are currently IRC channel URLs, like “irc://chat.freenode.net/commits”, but they could just as easily point to XMPP people or group chat channels
When I was designing this, I contemplated splitting the target designations as follows:
{"server":"chat.freenode,org", "channel":"#gpsd", "privmsg":"Hello world!"}But when I thought about it, my design intuition said to me rather loudly “Wrong. Use URLs.” I didn’t know why at the time, but I’ve learned to trust my instincts about stuff like this. I think you just told me why – somewhere in my backbrain I had already figured out that someday being able to speak non-IRC URLs would be useful.
Creativity can be an odd and mysterious thing sometimes.
Because nobody in this thread has mentioned it yet: the code in the release tarball definitely seems to have a configurable PORT parameter, set to 6659 by default. And if it didn’t, adding it would be a trivial five-line change. A nice rule for writing simple software is to not fret too much about this sort of thing when it’s so easy to change if you need to.
Actually, you could probably come up with a handful of quick-and-dirty rules that would prevent most common complexity problems. The kind of server configuration issues that plagued CIA, for example, can be prevented by doing server setup only through a script, which is checked into your repo along with the rest of the code. For a typical Python project, such a script would apt-get install anything it needs that’s not Python code, create a virtualenv, set up the most recent version of the code (and fetch dependencies) with a pip one-liner, and then restart any daemons that need updating. The map is never out of date, because it is the territory.
(Another good rule is, never use Twisted. I know that’s not a very broadly useful aphorism, but boy, would it have saved me a lot of headache. Too many callbacks, too much spaghetti code, and too damn many factory classes. Something relatively sane like Eventlet is a real breath of fresh air.)
Why do you even consider collecting statistics from commit notifications? Notifications are there for humans. Why go through all the format conversions, trying to workaround message truncating, delay and possible loss, when you can just pull the data in full from the source – the repo?
>Why go through all the format conversions, trying to workaround message truncating, delay and possible loss, when you can just pull the data in full from the source – the repo?
I wondered about this myself. I think it was a case of designer self-abuse. :-)
I wonder if instead of having to lock down irker behind firewall, why not use some standard authentication mechanism. I think OAuth (1.0 or 2.0) is now standard for tools authorization.
>I wonder if instead of having to lock down irker behind firewall, why not use some standard authentication mechanism.
Too heavyweight to be worthwhile in this case, or so I judge.
ESR said : “mostly Python but some Erlang, using (a) a custom web framework, (b) some Twisted, and (c) some Django”.
Argh. This alone makes your point to anyone having faced mixes of technologies. Especially for a not-that-big system(the thing it does is, mmmh, well, not really complex. You re-made it in 400 lines, seems like).
Someone was having fun creating this. Fun shall not be dismissed, though. Fun fuels productivity & creativity(probably even more for open source). A lot. I tend to believe more & more that being able to stop when fun becomes counter-productive is one of the toughest things for being a professional coder. Not enough fun, & you’ll do bland things(i.e. corporate, repetitive, unefficient, boring, slow, unintuitive to use, lotus-notish…). Too much fun & you end up with astronautic architectures(as described here) where 2 or 3 small components are enough.
Typo: “Unfortunately, as in the old proverb about law and sausage, […] advised not look to much more closely”
Should be, say, “advised not to look”, or “advised to not look”.
BTW @esr would you be replacing ‘contrib/ciabot/’ in git sources by equivalent irkerd solution?
>BTW @esr would you be replacing ‘contrib/ciabot/’ in git sources by equivalent irkerd solution?
No, because the git hook code ships with the irker distribution. I’ve sent mail to the git development list explaining the situation and requesting that contrib/ciabot/ be deleted.
> Why go through all the format conversions, trying to workaround message truncating, delay and possible loss, when you can just pull the data in full from the source – the repo?
That (pulling data from repo) is, among other things, what Ohloh does:
https://www.ohloh.net/
Thje one team I’m actively involved in as a developer these days uses Skype as a communications channel. None of us particularly likes IRC, and Skype gives us the ability to join the chat and get access to what has gone before.
It shouldn’t be hard to add the same functionality for Skype chats, either. I may look into this.
As one of many former CIA users, I thank you for this.
Let me ask a question, though: why couldn’t a candidate CIA replacement simply take the form of a commit script hook that talks directly to the IRC channel instead of through a daemon?
>why couldn’t a candidate CIA replacement simply take the form of a commit script hook that talks directly to the IRC channel instead of through a daemon?
Because a CIA client that opened up a new connection for each message would produce huge volumes of join/leave spam.
As someone who has spawned his fair share of poorly conceived and executed code “blobs”, I think your restraint and brevity while describing the work of the originator and the maintainers is admirable. Critique not criticism! And I think your handling of the sitution is also admirable in that, rather than complain about the sticky soda spilled on the kitchen floor, you are one of those persons who is on the floor mopping up the mess while also asking the children how they might avoid making such a mess in the future. Bravo!
@esr –
In the OP, you talk briefly about the history of CIA.vc. It seems (from a little link chasing), that the real original author was M. Elizabeth Scott, who wrote the first code back in 2003ish. When she dropped it, atheme.org took over. Her Twitter feed from 26 Sept. talks about re-hosting the service and re-pointing the DNS entry. You might want to correspond with her or her successor maintainers about your work.
Hm. Looks like the original architecture came about before DVCSs. That might explain a few design choices.
But that Ms. Scott was a college student with plenty of free time explains many more.
IGnatius: There’s an efficiency advantage for a forge site in routing all this through one central daemon instead of having 40 zillion IRC scripts with separate state and network connections for each one. Since this is aimed at forge sites, it’s a natural design choice.
My bad – more careful link chasing reveals “M. Elizabeth Scott” and “Micah Dowty” are the same person.
My sincere apologies to anyone upset by my statements.
GitHub offers that; see ‹https://github.com/github/github-services/blob/master/services/irc.rb›. It requires the GitHub framework though, so it’s (probably) not portable to other forges.
Eric,
I understand the logic of not being able to specify a port. But should it at least not take an option to specify an address on which to listen (defaulting to 127.0.0.1)? Or does it default to 127.0.0.1 already? This can be done at the firewall, but why should it?
>Or does it default to 127.0.0.1 already?
No. I actually wanted to do this, but couldn’t figure out how to beat the Python SocketServer library into it soon enough for 1.0.
>>why couldn’t a candidate CIA replacement simply take the form of a commit script hook that talks directly to the IRC channel instead of through a daemon?
>Because a CIA client that opened up a new connection for each message would produce huge volumes of join/leave spam.
I was going to say “Why not a generalized daemon that listens on a port for messages and sends them on to IRC”, then realized… that’s basically what it is. I think that regardless of the initial idea being a CIA replacement for commit hooks, saying it’s “for” that undersells its usefulness as a general tool. It could be used for _any time ever_ you want to script sending a notification to IRC – without making the tool generating the notification deal with maintaining IRC state.
Now, it’s probably something that a thousand people have tacked on a half-assed version of to a hundred different IRC bot codebases with no standard protocol, but I think this has the potential to become a unix-philosophy IRC bot.
>I was going to say “Why not a generalized daemon that listens on a port for messages and sends them on to IRC”, then realized… that’s basically what it is.
I’m wary of the tendency to over-generalise. Start out writing an IRC client, then generalise that into a socket library, then end up re-inventing netcat..
>but I think this has the potential to become a unix-philosophy IRC bot.
Despite my one reservation above, I actually agree with you on this count, and I think ESR’s solution is truly beautiful.
Actually, it would take minimal work to turn it into a general chat notification bot: with minimal hacking, the channel URL can be *any* URL, and then all you need to do is write the specific handler for that URL scheme. A handler for an xmpp: URL would probably plug right in, and Skype (plus or minus library licensing, an annoyance I need to think about) isn’t much harder.
Lewis: when in doubt, look at the code; it’s the ultimate authority on itself. Host and port are configurable, and default to localhost:6659.
Am I missing something here?
“Why do you even consider collecting statistics from commit notifications?” was mentioned, but CIA.VC WAS in my book the data! I only found that it was down when I went to check up on my across several projects yesterday. CIA.VC used to provide a good CENTRAL record that I could use. One does not realize how much one relies on something until it’s not there – I had several years on ‘history’ until a break caused by pigging DVCS’s not being fully developed before they were blindly adopted (there are better things than git) and took some time for all the infrastructure to catch up!
The current developments are simply replacing the code? There is currently no sign of a replacement ‘home’?
I can’t say that I am surprised that ‘virtualization’ and ‘the cloud’ feature in the problem. Personally I still prefer to know where my data is :) and I run Firebird on all my customer projects which runs background backups at appropriate times so that when something does break – which it does occasionally – there is always a crutch to fall back on.
Oh and we don’t want ‘one’ of the ‘social networks’ picking this up, the nice thing was the independence and access to code across many systems!
@ESR, why “privmsg” here, and not just “message”, “payload” or even “body”?
Jakub, Eric had IRC in mind when writing it. PRIVMSG is the command an IRC client sends the server to send a message to another user or to a channel. (Both. Don’t ask me why; it’s probably for hysterical raisins relating to the evolution of IRC in the distant past.) The irkerd source code also uses heavily IRCish variable names.
@,b>Jay, I was referring to this when asking about “privmsg”:
So we have URLs in generic “to”, but IRC-specific “privmsg”…
>So we have URLs in generic “to”, but IRC-specific “privmsg”…
Yes, this is a minor design flaw. I was rushing to get the code out the door following the death of CIA, and did not think everything through as thoroughly as I would have at more leisure.
Quite coincidentally, I spent part of last week simplifying an overengineered codebase, as well.
The difference is that (a) the original codebase was mine rather than someone else’s, and (b) for some uses, it isn’t really overengineered. But for many potential uses and users, it is definitely overengineered, if for no other reason than that the learning curve might be too steep if you need to perform customizations like subclassing things.
The codebase was rson, which was born out of my unhappiness with the severely overengineered YAML. And rson has proven itself very useful over the last couple of years, both as an example file format with a robust parser, and as a highly configurable parser that can be easily tweaked as necessary.
But reconfiguring the parser involves understanding how and where to subclass things, and not everything is configurable. I was about to add yet another configuration option when I decided to go the other way, and separate out really low-level container parsing (indentation, comments, etc.) from the application-specific higher level stuff.
So, I now have a new rsonlite parser, which (AFAIK) is the easiest way to transform a text file that has meaningful indentation into a corresponding Python data structure. Also, since rsonlite is a simpler format than rson, it was easy to generate an encoder for it, as well. This is something I never did for stock rson, because rson is so flexible it’s hard to decide what the canonical format should be.
I’m really glad someone is finally replacing CIA.
As you’ve gathered, the architecture was a total mess, and it’s been limping along on life support for many years. I feel somewhat compelled to add some historical perspective, though. Despite what you may think, CIA was not designed by a total idiot- it just wasn’t ever designed to be as big as it became.
When I started CIA, it was quite literally just a few shell scripts. I think the very first IRC bot was a pretty simple Perl or Python script, and there was a commit hook which would send it messages over a pipe. As simple as possible. It was only intended to serve a single project- a little embedded GUI I was working on at the time. I just wanted something that would notify the IRC channel, and while it was at it I could collect some simple checkin stats.
A few other people were interested in using it, so I extended it very incrementally at first. From the very beginning, it was “production” code in the sense that people were relying on it, but alas it wasn’t anyone’s primary responsibility. I still considered it to be a little side-project that I mostly just used to help with developing the embedded GUI.
There was one major rewrite which replaced the mess of shell scripts with a daemon written in Python using the Twisted framework. I now know this was a horrible idea- the resulting codebase was complex, monolithic, hard to debug and maintain. But for me, it was a side-project and a learning experience. Something to do instead of going to classes. CIA *never* had a proper development team behind it. I was begrudgingly maintaining it to the extent that I had to, but it was never really my primary focus.
And that was the problem. One big rewrite that left the system in a hard-to-maintain spot, followed by very little time and energy for follow-up rearchitecture. This all happened way before the service got as big as it is now. To help with the scaling, I ended up splitting the monolithic Twisted daemon into a few daemons. Then I started the process of rewriting the web frontend using something standard (Django). But I never had time to finish that rewrite, so there was always part of the site in Django and part of it in the original Twisted codebase.
I made an attempt to do one last rewrite to make the codebase actually scalable and decentralized. Some of what you see in the repository (like the Erlang bits) were remnants of experiments I did, to see what technologies would work well for such a rewrite. But I was long past out of steam, and I never completed that rearchitecture.
Anyway, hopefully this is helpful. I think many people can vouch for my competence as a developer. (For example, I’m pretty happy with my resume: http://scanlime.org/resume/). To paint this purely as a case of some idiot developing a recklessly complex architecture would be misleading, and I think much less interesting than the truth: This is a great case study of what happens when a service is well-liked but nobody steps up to maintain it. It grows organically to the extent that it can, then collapses.
–beth
>To paint this purely as a case of some idiot developing a recklessly complex architecture would be misleading, and I think much less interesting than the truth: This is a great case study of what happens when a service is well-liked but nobody steps up to maintain it. It grows organically to the extent that it can, then collapses.
I won’t argue with that description; it’s completely consistent with what I saw in my one look at the code. I’m not going to let you off the hook completely, though; you may not have intended reckless complexity, but that is where you ended up through not having enough discipline and persistence to do the hard work of keeping things simple. Perhaps you have that kind of craftsmanship now as a more experienced developer; I hope you do.
> I was going to say “Why not a generalized daemon that listens on a port for messages > and sends them on to IRC”, then realized… that’s basically what it is. I think that
> regardless of the initial idea being a CIA replacement for commit hooks, saying it’s “for” > that undersells its usefulness as a general tool. It could be used for _any time ever_
> you want to script sending a notification to IRC – without making the tool generating the > notification deal with maintaining IRC state.
That was my initial impression as well. It brought me back to the days of having nagios alerts going to our in-house irc network, and what a pain it was…
> Now, it’s probably something that a thousand people have tacked on a half-assed
> version of to a hundred different IRC bot codebases with no standard protocol, but I
> think this has the potential to become a unix-philosophy IRC bot.
Well spoken, and my thoughts too. I’m envisioning a few uses already, none having anything to do with its intended purpose.
Hello,
I understand the architecture is a mess and the source is very complicated. I have started working to update and rewrite some of the architecture for CIA.vc. This will take some time to do but I have many ideas for CIA and the remaining codebase.
I have started a new CIA server at http://cia.stacksmash.net/ for everyone to take a look at.
I plan on maintaining this for a long time as I am obsessed with the project and many of the alternatives do not do what I would like them to do. I have become upset with the idea that there is no longer a centrailized server for all commits to go to and all the replacements I have seen since do not allow “subscribing” like CIA did. I followed many project updates in my IRC server/channel which I am now unable to follow due to the lack of ability to get access to commit hooks.
I have plans to completely remove twisted with a better IRC implementation. I have already begun finishing the rewrite of the web front that Beth Scott was doing as well as updating many of the libraries CIA depends on. Most (if not all) bugs have been fixed with the restart of the server. The front “overview” page has been rewritten into django and I am working on rewriting the stats functions as well as updating from django 0.96 to django 1.0 (and further upgrading from there). Once I get CIA fully upgraded as well as cleaning up the code base, it will be a bit more understandable than it is now.
You can follow my progress at https://github.com/Justasic/cia-vc or on the blog on http://cia.stacksmash.net/
– Justin
>I have become upset with the idea that there is no longer a centrailized server for all commits to go to and all the replacements I have seen since do not allow “subscribing” like CIA did. I followed many project updates in my IRC server/channel which I am now unable to follow due to the lack of ability to get access to commit hooks.
If you must do this, get it right this time. Cooperate with irkerd. If you show me a credible plan, I’ll add a side channel to irkerd that relays JSON commit notifications to a CIA aggregation site. That way you never have to maintain or ship your own hooks and can specialize in aggregation and subscription feeds. You wouldn’t even have to have your own database of project-to-IRC-channel mappings; the incoming JSON would tell you that.