I’ve been thinking about how to build a better IRC client recently.
The proximate cause is that I switched to irssi from chatzilla recently. In most ways it’s better, but it has some annoying UI quirks. Thinking they’d be easy to fix, I dug into the codebase and discovered that it’s a nasty hairball. We’ve seen projects where a great deal of enthusiasm and overengineering resulted in code that is full of locally-clever bits but excessively difficult to modify or maintain as a whole; irssi is one of those. Even its maintainers have mostly abandoned it; there hasn’t neen an actual release since 2010.
This is a shame, because despite its quirks it’s probably the best client going for serious IRC users. I say this because I’ve tried the other major contenders (chatzilla, BitchX, XChat, ircii) in the past. None of them really match irsii’s feature set, which makes it particularly unfortunate that the codebase resembles a rubble pile.
I’m nor capable of stumbling over a situation like this without thinking about how to fix it. And yesterday…I had an insight.
Probably the single most annoying thing about today’s IRC clients is that if you don’t leave them on all the time you miss some of the channel traffic. There’s no way to join a favorite channel and look at the traffic for the last half hour to get context that happened while you were gone, other than leaving your client actively watching it. And sometimes you don’t want a client distracting you with chat and urgent notifications.
So, I thought, OK, what if I built a client that logs all your IRC traffic for you? You’d still have the dropout problem, but at least it could always use the log to show you the last part of the conversation you were actually present for. Hm…but what about when you weren’t there?
That’s when I got it. I realized that because people think of IRC clients as ways to watch network traffic, they build them all wrong. Here’s how to do it right…
First, build a little client daemon whose job it is to watch channels for you and log their traffic, aggregating it into a message timeline that’s stored as a logfile on disk. The daemon gets started if it’s not already running, whenever you fire up your client. But exiting the client doesn’t kill the daemon. If you really don’t want to miss anything, you launch the daemon from your login profile well before you start your client.
Your client is just a browser for the message timeline. It doesn’t actually talk to IRC servers because it no longer has to. When it wants to send traffic, or join a channel, or leave a channel, it ships a request to the daemon, which is managing all the actual server connections. The response gets appended to the message timeline just like every other traffic and is then visible by the client.
Then I realized…I’ve already written this daemon! Almost all of it, anyway. It’s irker, my replacement for the defunct CIA service. Add an option to log traffic. Add options to set your nick and its nickserv password. Done!
Those features are in the irker repo now. Not released yet because the code for nickserv authentication is untested, but that’s a detail. The point is that adding about 20 lines of trivial code has amped up irker so that it’s now a generic chat-logging back end that could be used by a whole family of IRC clients – every one of which could be functionally superior to what’s now out there.
To paraphrase XKCD: Code reuse. It works, bitches!
This is currently already implemented and in-use in Quassel ( http://www.quassel-irc.org/about ). The core connects to IRC and stores its backlog, which can be viewed using the Client (which is also available for Android and iOS). Maybe that is what you are looking for :)
I do not fully understand how the irker approach will work — but that’s due to not knowing irker’s inner workings in specific terms.
Anyway, another solution would be Quassel which also fulfils your requirements.
What you seem to be describing as your perfect client sounds like Quassel.
With Quassel you run a “core” daemon, which is the actual IRC client connecting to IRC networks, and then connect with one or more clients (available for plain old Linux, Android, iOS, OS X, Windows, Haiku, or whatever tickles your fancy).
When the clients connect to the core, they fetch the unread backlog from your core to display to you (configurable amount, of course). One nice feature from this is that you have basically infinite backlog available in the client, you just scroll upwards in your client and it fetches more from the core.
Linkinus, the for-pay Mac-only IRC client that I’ve been using for the last few years, does exactly that. The “Linkinus Agent” process communicates to the IRC servers, and the “Linkinus” app does the UI. Launching “Linkinus” launches “Linkinus Agent”, but you can choose to quit “Linkinus” and leave “Linkinus Agent” running.
Neat idea. I immediately wanted to add FR for Freenode’s cloaking support but didn’t managed to find bug tracker for irker.
Having support for http://www.wikihow.com/Get-a-Cloak-on-IRC would make irker into great backend for making irc clients.
I’ve probably missed it but why bsd license all of a sudden?
If you’re interested in clients using a distributed topology approach, you should really check out Quassel IRC. It’s cross-platform and has a GUI interface which is nice even if you spend most of your time in a terminal. Quassel has been around for a number of years and was previously included as the default IRC client of Kubuntu. Anyways, you check it out at http://www.quassel-irc.org/ .
IRC, the worlds most readable Pub-Sub framework.
What about making the client talk to the daemon optionally over IP? That would be useful to someone like me. My primary computer is a laptop, and it goes into deep sleep a lot of the time, as it’s always with me. If I could run the daemon on my VPS, and then connect to it whenever my laptop wakes up, and get all of my logs.
>What about making the client talk to the daemon optionally over IP?
That was already the plan.
Aaron: You could consider using a bouncer such as znc.
Esr: This seems to be basically what ii does: http://tools.suckless.org/ii/
But if you want to log IRC, then you want to log it 24/7 and probably you are not going to leave your desktop running 24/7. Especially if it is a laptop. Logging should be done by an always-on machine i.e. a server. Having a not-always-on machine log a service provided by an always on server sounds like a category mismatch to me. How many daemons you use provide you with incomplete data, depending on when you turned off your machine?
The most obvious candidate for logging would be the the IRC server itself. Why not just extend the IRC protocol with requesting a log from the server?
Or getting such an extension through is hard, semi-official extensions, much like Python PEP’s or RFC’s?
@esr : I’ve been IRC’ing for decades. irssi is my client of choice, and has been that way for as long as I can remember.
But, lol, aren’t you basically describing the features of the eggdrop IRC bot?
Also, screen/tmux + irssi, and you don’t have to bother writing something new – unless of course you really, really want to – which I imagine you still do ;)
Regards.
screen + irrsi; seems like will do almost everything that is required?
>screen + irrsi; seems like will do almost everything that is required?
No. I started thinking about this because I don’t like the way irsii handles multiple channels. What I specifically dislke is getting activity cues by window number rather than channel name. Chatzilla’s UI (tabs with the channel on them that light up for different activity levels) is better.
Besides, a suite based on this design idea ought to have at least three clients: one GUI, one TUI (working roughly like irssi) and one line-oriented to be used for scripting and regression testing.
I will investigate ii and quassel, though. They might make it unnecessary for me to write anything.
@esr
What’s the format of the log file? In particular, does formatting live in the “log file browser” (with the daemon just storing stuff like “:foo!bar@baz PRIVMSG #channel :some text” or some more easily parseable intermediate format), or the “daemon” (log looks like “<foo> some text”)? What’s responsible for rules-based stuff like ignoring, highlighting (notification triggering), etc? Where do advanced channel administration features live? It’s a cool concept for an IRC client, but “None of them really match irsii’s feature set” rings hollow if you don’t even match any of their feature sets.
Speaking of this, someone else had the same idea. I haven’t looked at how it answers these questions, though I suspect it dodges some of them by being “minimalist”.
@Aaron Traas
> What about making the client talk to the daemon optionally over IP?
Why not just have the files live in a remote filesystem like NFS or sshfs, given you’re already buying a file-based architecture. If you’re going to have the client talk to the daemon over IP, you might as well just use a bouncer and connect a normal IRC client to it (I’ve occasionally spent time thinking about how one would design a bouncer, a client, and extensions to the protocol to seamlessly preserve scrollback history), since if you have to design a protocol you might as well use the IRC protocol itself.
>What’s the format of the log file?
Three fields per message: timestamp, source server FQDN, raw message line. It’s the front end’s job to interpret this, perform highlighting, filter, etc.
@Kevin Cave
> screen/tmux + irssi
Honestly? The reason I, as I said above, “occasionally spent time thinking about how one would design a bouncer, a client, and extensions to the protocol to seamlessly preserve scrollback history”, is because I’m unhappy with needing to use something as heavyweight as tmux for this use case (dtach has intermittent bugs that stop me from using it), since having tmux around causes other bad habits for me. And honestly, I’m not happy with being forced to use a textmode IRC client, either.
@Random832 : I did say tmux or screen ;)
I’ve never considered tmux or screen to be particularly heavyweight – perhaps I’m wrong or my perspective differs from yours.
As for the client, well, IRC being a text-only medium, I prefer to use a text-only client – running in an xterm if you want the GUI bit ;)
I honestly think the “irc problem” is being over-thought and over-engineered here. Why do all that when screen+irssi automatically does it for you?
Regards.
I don’t use IRC much, so maybe this is a dumb question but:
Wouldn’t having the daemon running to log channel traffic when a client isn’t running mean that other people would see you as online even though you weren’t actually available for contact?
>Wouldn’t having the daemon running to log channel traffic when a client isn’t running mean that other people would see you as online even though you weren’t actually available for contact?
Yes, but the protocol has a feature that addresses this. You can declare yourself “away”, which is a status other peoples’ clients can discover.
@Kevin Cave
Screen is more heavyweight than tmux, I figured that went without saying.
Monospace fonts are ugly, and no-one’s bothered to design a way to make a terminal – particularly one that you can run fullscreen apps on – that can sensibly use a proportional font.
ii is basically only the backend (i.e. just writes log files), the frontend is tail -f. It would be instead of what you’ve already written, rather than instead of what you would need to write.
@esr
> >What about making the client talk to the daemon optionally over IP?
> That was already the plan.
Then I’ll redirect my “why not just use a bouncer” (or write one, if you want to be doing something) question to you – instead of inventing a new protocol for IP-based communication between the two pieces, use the IRC protocol itself (plus extensions to request logs, etc) so that any existing client (or a new client, if you don’t like the UI of any existing one) can connect to it.
>Then I’ll redirect my “why not just use a bouncer” (or write one, if you want to be doing something) question
Because I didn’t know what a bouncer was. But I think I’ve deduced what it must be from your comment. Can you point me at an example?
For some time I have used KVirc, then XChat (with its plugins: C, Python, Perl, Tcl).
Nowadays I usually use what is close at hand: ChatZilla when using Firefox, Thunderbird IRC support, or one of GNU Emacs IRC clients: ERC or rcirc.
http://en.wikipedia.org/wiki/BNC_(software)
Dircproxy is the name of another one I’ve heard of that’s not mentioned on that wikipedia page. I’ve never actually used one, so I don’t know which ones are any good or not (and I don’t know if any support stuff like retrieving back logs etc)
I think that some bouncers have a bad reputation due to being used as open relays or as the wikipedia article mentions “Although psyBNC is like any other IRC proxy, it is often installed on compromised systems by the attacker.[7]”
I use ZNC and bitlbee as my irc proxies which I then feed into emacs etc.
This gives me the always on chat log and keeps emacs from locking up if my connection goes away.
> Add options to set your nick and its nickserv password. Done!
Actually, I needed NickServ in order to make Irker work for its original purpose, otherwise it may have gotten kicked automatically by our channel bot for floodng. I’d been planning to kluge them in myself, thanks for saving me the trouble. :)
weechat supports relay too and is a terminal client like irssi so it might be more up your alley
http://www.weechat.org/files/doc/stable/weechat_relay_protocol.en.html
tmux/screen + irssi or irssi + znc should do the job just fine as well.
Why not increase irssi’s buffer limit, giving you a longer buffer to scroll back through channels text for however long required. ALso run irssi via screen, close screen and you can come back when you want, no need for the long work around :)
I’ve been using bip (http://bip.milkypond.org/) as a caching irc proxy.
Use putty or SecureCRT to login to a SERVER that is running a screen’d session of irssi, that has an increased buffer limit, and you’ll never have a problem. I have been doing this for 15 years (starting out with bitchx) and i have scrollback for over a year in some chans.
no need for all that bullshit you speak of.
Could notre escape thinking reading this that you are a kind of Miles Davis of thé code.
Great article !
Preceding zone was nice read also.
ESR> It’s the front end’s job to interpret this, perform highlighting, filter, etc.
Damned right it is. IRC must be thought of as another “layer” in the protocol stack, and interpreting the content of a chat transmission belongs at the top of that stack.
I second shenpen’s comment; the most intuitive way to achieve this seems to be a recent-history command for the server, at least to me. Is there a specific reason not to do it that way? My guess: simpler to make one client that achives the goal with current servers than to convince all servers to add history functionality. One is a clearly solvable technical problem, the other a possibly-unsolvable coordination problem.
I have a project of my own that, once I stripped away the magic, boiled down to adding video support to IRC — preferably in a way that doesn’t murder server bandwidth. Anyone know if that problem has already been solved somewhere, well? Might save me some trouble.
@The Monster:
If the client is going to do anything but pull down the _entire_ log (however many months back the session began) of _all_ channels, the server has to do some interpretation to be able to get information such as what channels a message (which may not contain the channel name, such as QUIT) affects.
Weechat. It’s better in almost every way. Curses interface, full feature set, and still being updated
>Weechat. It’s better in almost every way. Curses interface, full feature set, and still being updated
This is true. I switched a couple of days ago.
Since you seem to be interested in IRC lately, perhaps you’d like to take a look at my client ‘quIRC’ . Downside: it’s unnecessarily written in C (I started it before I learned Python. Also, strings without encodings aargh). Upside: I generally work on it at the same time as re-reading TAOUP (I am fairly confident that my (not yet fully implemented) design for scripting has the Unix-nature).
And as others have pointed out, a BNC (bouncer) does seem to be what you’re after, with the advantage that it can be (almost) completely transparent to the client.
Sounds like another advantage of this design is that if you have two computers, you can have only one signon to the IRC channel, rather than nelson-desktop, nelson-laptop, nelson-cell, and nelson-nexus7.
I’m confused. The article subject matter is all existing code with multiple open source implementations in the field, some of which are on their second or third decade of widespread deployment, but the text is full of language that suggests recent innovations or even new inventions have occurred. Did I miss something, or does esr live under a rock?
IRC proxies with all of these capabilities are not new, so what is? Is this a new and awesome implementation? If so, what new quantum of awesomeness is added (or quantum of suck removed)? Is there a chart to enumerate what’s out there now and why a new implementation is necessary?
Mobile IRC (i.e. IRC while your IRC client is physically moving over hundred-mile distances) requires a proxy on a well-connected server somewhere to isolate the IRC network from all the horrible things happening to your IRC client as it passes from one Internet uplink to another (or to nothing at all). The better IRC proxies available play back the messages you missed while you were disconnected, and allow multiple IRC clients (or multiple IRC-speaking robots and an IRC client) to share one upstream connection, while still using vanilla IRC protocol to talk to your client. The proxy often provides an embedded IRC bot that you can chat with to control the proxy itself.
This kind of setup also handles intentionally disconnecting and reconnecting your IRC client, if you wanted to do that too. My IRC client notifications are filtered through a list of matching rules, and I just change which subset of those rules is active to control the desired noise level.
IRC proxies don’t usually turn the IRC protocol into something that is efficient for phone-type devices (which is why they usually have their own), but that doesn’t seem to be the target environment here.
Completely off-topic:
I’ve just learned about the “Kerbal Space Program” game. From what people have written about it, it sounds as though the authors did a brilliant job of playability, depth, etc. Simulation gamers may find it interesting; I will definitely be giving it a whirl.
http://www.shamusyoung.com/twentysidedtale/?p=19396
The actual kerbalspaceprogram.com site seems to be temporarily down.
@Cathy: One of my cow-orkers strongly recommended it, but then I looked at the EULA…
If you run KSP on Linux (eg. with Wine), you’re violating the license. (Also, running it on a pirated copy of Windows is apparently a violation of the KSP license. That seems kinda… redundant)
I think I’ll stick to open source games.
It seems the comment system eats <link>s. Maybe using <a> — https://github.com/ec429/quIRC — will work.
(Eric, if you can’t give us a preview function — or at least an edit function — at least put some kind of indication of what markup is accepted on the comment form.)
@Edward Cree – it seems to accept (a subset of) HTML. <link> is not the HTML markup tag for a clickable link, it does exist in HTML but only as a metadata link for related files (e.g. stylesheets). I don’t know what markup language you are used to where it represents a clickable link.
@Random832: It’s a fairly common convention in plain text files to use <www.example.org> to indicate a URL. That’s what I was referring to, and the format I used initially, not realising that the <> would be parsed as HTML. (I assumed they’d get converted to < and >. Too much exposure to comment boxes that take Markdown or BBcode.)
I think what you describe there is called a ‘bouncer’. They’ve been around fir ages.
It seems to me that while a bouncer can be configured to spurt some level of scrollback out to the client when it first connects, this isn’t necessarily the behaviour that esr is after. A client/server could be designed so that the client can request that scrollback as it needs it, instead of having a big dump at the start.
As I had understood it, it was possible to send a command to existing bouncers to request scrollback on demand, but that this functionality is not standardized.
My suggestion was to make a systematic extension to the IRC protocol to allow for these requests, rather than invent a whole new protocol
The problem with the attitude that “It’s the front end’s job to interpret this” is that if all you’re storing is raw IRC lines, the front-end has to do an unbounded amount of processing to tell which lines belong to what channel. Most lines do include the channel name in them, but some do not. For example, QUIT messages (which do not contain a channel name) are expected to be displayed in the scrollback for the channels the user was in at the time of the quit. Also, the front-end needs to know the channel topic and list of users in the channel when you join – esr and Monster’s “idealized” program won’t keep track of these – will it make fresh requests for them to the server? Or will it expect the client to pull down the entire log and process it from when this initially happened weeks ago, applying every join/part/quit that’s happened since then to its notion of the channel state?
IRC is fundamentally a stateful protocol. If the daemon won’t keep track of this state (both the present state and the state at the time of any message to determine what channels it should appear in the scrollback of), then there’s no way for the client to reasonably work with a partial log.
In practice it doesn’t matter, since the client will request a set of fresh information when it connects to a server anyway, just as if it was connecting to any other server with unknown state.
That said, it isn’t perfect. Multiplexing bouncers tend to repeat the fresh channel lists to every connected client, including clients that didn’t ask for channel list information (to fix this the proxy would have to know which server reply corresponded to which client request, but that’s ambiguous in the IRC protocol even in the single-client case). Join/part/quit messages get transmuted into notices or in-band messages (assuming the user cares about these, and doesn’t just turn them off).
The solution is for the bouncer to keep track of the state and answer the clients directly rather than forwarding their requests to the server.
> Join/part/quit messages get transmuted into notices or in-band messages (assuming the user cares about these, and doesn’t just turn them off).
That’s fine for joins and parts, but quits aren’t associated with a channel – the bouncer has to know what channels they apply to. And it can’t ask the server, since it doesn’t know anymore after the quit.
@Random832
I don’t get this at all. If the back end is logging all IRC traffic, then it has all of the TOPIC commands logged. So all you have to do is read the log from the bottom up, and find the most recent TOPIC command for each channel. Reconstructing the list of visible users is probably not worth the trouble to track all the JOINs and QUITs, so it’s easier to just send a WHO request to the server if you really want to know who’s logged into the channel right now. That’s what a conventional client would do when you join the channel anyway, so it’s not like it’s doing any more work.
Remember, ESR didn’t say this was supposed to maintain the full state of the channel. It’s just supposed to log all the message traffic so that you don’t miss anything by not being logged in.
> Reconstructing the list of visible users is probably not worth the trouble to track all the JOINs and QUITs, so it’s easier to just send a WHO request to the server if you really want to know who’s logged into the channel right now.
Yes, and you can do that with the topic too. The problem is you don’t know which quits from the log to display in which channel window – for that you have to know who was logged in to the channel at the time the portion of the log you are displaying started.
I don’t know why you have such a strong objection to having any intelligence exist in the daemon.
@Random 832
> I don’t know why you have such a strong objection to having any intelligence exist in the daemon.
It’s not that I don’t want it to have any intelligence, but I want it to have only the intelligence it needs to have for the same reason I don’t want the IP layer to know anything about TCP ACKs or HTTP GETs. I want a daemon like this to be as lightweight as possible, so that it doesn’t bog down my computer with extra work interpreting information I may never want to see. That work should be done in the client when I ask for it.
If knowing who was on a particular channel when a specific message was logged is that important to you, then feel free to write a daemon that bothers to log such information as well as the actual traffic it’s receiving from the servers.
But it’s not the use case we’re discussing here. It’s silly to add a lot of complexity we don’t need to accomplish the job we’re trying to do.
> But it’s not the use case we’re discussing here.
How is that not the use case we’re discussing here? Seeing quit messages in the buffers for the channels the user was in at the time they quit, and not in the other channels, is implicit in the whole idea of getting log information for the past N minutes or whatever on connecting.
@Random832
“There’s no way to join a favorite channel and look at the traffic for the last half hour to get context that happened while you were gone, other than leaving your client actively watching it”
I read that as the message traffic, not peripheral matters like QUITs.
It looks as if an irssi release candidate came out in September 2013. Weechat seemed a bit bloated to me in comparison with irssi. How does weechat’s code compare?