It was just three days ago that I shipped irker 1.0, but the project is already a huge hit out there in hackerland. It’s clear from traffic on the freenode #commits channel that irker installations are springing up everywhere. There’s already one symbiote, a proxy that takes XML-RPC requests in the CIA format and passes them to an irker instance (you have to supply your own mapping of projects to IRC channels for it to use). And at least one custom hook already written and in production – by the Python development list, as it happens.
I’m a bit boggled, actually. I don’t think I’ve ever had a project go from launch to all over the freakin’ landscape this fast before. Guess that’ll happen when you step up with a clean replacement for a service that lots of people were habituated to and have suddenly lost.
There’s more work to be done, of course. (There’s a public repository, and an #irker IRC channel, for people interested in following development.)
A&D regular Daniel Franke did a really good security-vulnerability analysis, which I’ve expanded on and is now in the repository. Launching from that I just added some DoS prevention to the code.
The repo hook component needs work as well (this is the Python script you make your repository’s post-commit hook call in order to generate a notification). At present it has support for git and Subversion; it could support Mercurial, CVS, and other version-control systems as well. An important feature of the code is that the VCS-dependent stuff lives in extractor classes well separated from the generic stuff; thus, adding support for more VCSes will be easy when someone steps up to do it.
I’m also a little worried about the multithreading – a technique I normally fight shy of because it’s so prone to subtle race conditions, but it had to be done here. It seems to work, but…if any of you reading have experience at reviewing this kind of design, please critique mercilessly.
We’ve had one annoying deployment issue. Some people have been reporting irker crashes on session disconnect due to a bug in the stale version of irclib up on SourceForge. The actual project home has moved to PyPI, the Python Package Index, but the maintainer hasn’t gotten around to updating his documentation and web pages yet. If you want to run this code, get the PyPI version of the IRC library to do it with.
But, these relatively minor issues aside…three cheers for classic Unix minimalism! Total LOC of the codebase is just 503 lines exclusive of comments. I chased St. Exupéry’s definition of perfection (“…when there is nothing left to take away.”) pretty hard this time, and it seems to have worked out well.
UPDATE: I just shipped 1.2.
Bear in mind, it’s going to be rolled out in places you’re not going to get to see. I’m going to set it up for our in-house project, taking opportunity of the deployment and setup of our new development servers. If no one’s done so by the time, I might need to write the XMPP communications bit, which will certainly be interesting.
Anyhow, congrats!
There shouldn’t be many problems with multi-threading in my opinion, as the daemon runs only locally with a limited number of users.
But I don’t see why you write that it had to be done here, writing the daemon to use asynchronous I/O techniques would be possible.
>writing the daemon to use asynchronous I/O techniques would be possible.
The Python API for that is obscure and thinly documented. And even if I understood it perfectly, it probably would have required messing with the internals of the irc client library I was using.
This is a bit off topic. I am getting back into teaching myself Python both Console and Gui programming (PyQt for the Gui) and I am using Eric version 5, and QtDesigner. I am curious what tools other python programmers are using to code with. Also what tools would be suitable for doing the gui side of things with the wxPython library for GTK gui code.
I’ve got a fork of the irker repo using asynchronous I/O, in a single thread. I know that using Eventlet for this feels like cheating, but it works. This is a three-line change to the code (!), makes Ctrl-C work correctly, and prevents the thread-flooding DoS attack on 32-bit machines. It also tends to make a lot of subtle race conditions simply not happen, because there are a lot fewer points where execution can leave a greenlet — hooray for cooperative scheduling.
The only frightening aspect of this is that it’s magic, and involves replacing some standard library modules with wrappers. Once you understand the magic, though, it’s a remarkably useful library that’s seen heavy production use.
>I’ve got a fork of the irker repo
Send me a diff and explanation, please.
By the way, is there any testing setup for this? What I did was just start a local IRC server (ngircd with the default config file), connect to a couple of channels with an IRC client, and have a short Python script send sample messages to irker, aimed at both of those channels. This works, but it’s a little ad-hoc for my tastes.
>By the way, is there any testing setup for this?
Not yet. I’m doing live testing to #irker and #commits on freenode.
@esr:
Nah. I’ve used it. It’s quite easy; all event-driven. A programmer of your caliber would have little difficulty in working with it. See http://docs.python.org/library/asyncore.html
But messing with the IRC client library is probably not worth it. I don’t anticipate a single use case where something like irker would be using enough CPU power that it would matter at all. Not even the Linux kernel has that many commits.
Morgan,
I thought the benefit of single threading was that it made DOS attacks harder. Am I wrong about that? Not feeling really sharp today.
Yours,
Tom
Off-topic: more risks to innovation and freedom in America.
The former Register of Copyrights, Ralph Oman, is arguing to the court in the Aereo case that copyright law should require the innovator of any new technology to apply to Congress for approval before deplioyment, just to be sure that it doesn’t damage existing copyright. Prior restraint, anyone?
Article
Brief to the court
Are all codebase doomed to increase in LOC?
> writing the daemon to use asynchronous I/O techniques would be possible.
Using eventlet *is* using asynchronous I/O; that’s what eventlet is for, to make programming with asynchronous I/O look like programming with threads. See the top of the page at http://eventlet.net/:
“It uses epoll or libevent for highly scalable non-blocking I/O.”
> Using eventlet *is* using asynchronous I/O
Ah, disregard my last — should have looked more closely at the diffs in the repository first. :redface:
kiba said: Are all codebase doomed to increase in LOC?
No. ESR reassumed the lead on giflib last April (after a gap of 18 years), and issued a new release (4.2.0) in July. He threw out “lots of useless old cruft” and reduced the codebase from 10.8KLoC to 8.6KLoC.
OT:
http://apple.slashdot.org/story/12/10/03/0357223/apple-acknowledges-iphone-5-camera-flaw
kiba said: Are all codebase doomed to increase in LOC?
In my experience, when a project is learning to walk, yes. There are many features and considerations to be incorporated to irker, so most likely the codebase is growing.
But in the aforementioned case, with giflib, the project was already mature; features needed not to be added, but removed, because the code was (out of need) not lean. In the best cases, the need for excess code (to support different platforms or, as I believe was the case with giflib, conversion from quirky, now unused formats) is not present. In those cases, the codebase does grow. _But_ in an orderly fashion, with an architecture that makes it clear where the growth should be. Such as irker. No unnecessary growth, and easily maintainable growth, that’s the goal “As few LOC as needed to clearly and cleanly provide the features” is the maxim.
> There are many features and considerations to be incorporated to irker, so most likely the codebase is growing.
Actually, the last real code change I made significantly reduced the line count of irkerhook.py.
The program main – the part that sequences the code for discovering the repository type, then instantiating and applying the extractor class – was longer and more complicated than it needed to be. I found an almost terrifyingly elegant way to code it that walks through a list of extractor classes, applying for each one a static is_repository() method that checks for the associated repo type, then instantiating the class only if the check returned True.
“Are all codebase[s] doomed to increase in LOC?”
If you’re talking about commercial products, whether shrink-wrap or enterprise, I’d say “yes”. Generally companies want to compete on features offered, not price or quality of code.
Microsoft is the classic example of this; instead of fixing their long accumulated list of bugs, they keep introducing new features that hardly anyone wants, and redesigning a user interface that everyone understands and don’t want to see changed.
Open Source is a different matter.
“Open Source is a different matter.”
Really? My last visit to ohloh.net seems to confirm that all open source codebase grow in LOC too.
@kiba : depends on the project, what’s being done, and the developers.
esr managed to thwart my attempt to explain stuff merely by using stages, although what he did counts as maintenance in my book; just as you managed to thwart Cathy with ohloh stats.
I think we can agree that the general case is that adding features (especially features that are not related to each other) grows the codebase. If you do maintenance while adding features… well, that’s your fault, not mine.
We can also agree (I think) that the only way to reduce a codebase without functionality loss is maintenance (refactoring, reimplementing).
So, growing software, adding new features, tends to grow the codebase. I predict irker (not necessarily the daemon, but the project as a whole) will grow. That ain’t bad by itself… bleh, feel like I’m making a moot point.
>I predict irker (not necessarily the daemon, but the project as a whole) will grow.
A safe prediction, and you got an important detail right. The daemon, I think, is going to grow very slowly. Most of the new features and LOC increase will be in the repo hook, irkerhook.py, as we add support for more VCSes.
> I found an almost terrifyingly elegant way to code it that walks through a list of extractor classes, applying for each one a static is_repository() method that checks for the associated repo type, then instantiating the class only if the check returned True.
sorry to join late — would you say this is a case of functional programming in a non-functional language? As I read your description, I mentally translated it into a “map” function applied to a function and a list of data.
>As I read your description, I mentally translated it into a “map” function applied to a function and a list of data.
No, the pure-functional implementation of this would rely on filter rather than map.