This is a tease.

Yesterday I applied for allocation of a new public port number from IANA. It’s 6659. When the allocation is confirmed, I’ll publish the source code for a reference implementation of the server. It’s a bit over 300 lines of Python.

Let the speculation begin. :-)

Published
Categorized as General

56 comments

    1. >“Federation”

      Oh, no. That I’m not anywhere near ready to deploy. The service for this one is already written.

    1. >Something to do with ForgePlucker or something else VCS-related?

      Heh. I can confirm that it’s VCS-related. But not to any particular VCS.

  1. Hmm… something VCS-related that wants an IANA-assigned port number…

    Sounds like some sort of notification sever à la cia.vc, but that seems to work fine over IRC so I’m not going to guess any more.

  2. My money’s on Joel’s guess, but if he’s wrong, then is it a way to automatically keep VCSs of different types in sync?

  3. Well, let’s see.

    Here we’ve got a confirmation it’s VCS-related.

    We know Our Gracious Host was recently working on ciabot. As Joel points out, cia.vc uses IRC.

    Over on the g+ feed, there was a guess that it was “Something related to instant messaging, since it’s so close to 6667 (and other not-so-well-known irc ports).” Our Gracious Host responded was that that guess was “remarkably shrewd.”?

    (Looking on the other side, port 6657 seems to be assigned to a buzzword-heavy European Union project that produced a bunch of research papers and some model code, and subsequently has sat on the shelf. Maybe there’s something interesting in PalCom, but I got an allergic reaction to the writing too soon to dig anything interesting out.)

    So. Something to monitor publicly-readable VCSes that don’t have a cia.vc hook or which the admins didn’t set up so it could report to cia.vc (like, apparently, Savannah fails to allow), and report activity on them? A distributed version of cia.vc, so that it can’t get killed like it did for a short while in May of last year?

    Hmmmm.

  4. I like Mr. Ehrbar’s answer, but would like to point out that CVSup uses registered port 5999. So maybe this is something like that. The whole 66 numbering may, in fact, be a red herring.

  5. > Over on the g+ feed, there was a guess that it was “Something related to instant messaging, since it’s so close to 6667 (and other not-so-well-known irc ports).” Our Gracious Host responded was that that guess was “remarkably shrewd.”?

    Now that you mention it, I do remember esr discussing one of his new projects with Freenode IRC staff…

    ” It’a listener daemon that accepts requests like this on a socket: {“to”:”irc://chat.freenode.net/freenode”,”privmsg”:”Hellow World!”} and posts them. It’s intended as a lighter-weight competitor for the CIA notification service.”

    1. >It’s intended as a lighter-weight competitor for the CIA notification service.

      Heh. OK, I’m busted.

      Coming soon: an essay on what’s wrong with CIA, and what it teaches us about how to avoid complexity collapse in software projects. I was going to wait until my IANA port allocation was confirmed, but now I may not.

  6. The 300 lines of Python is only a front to make everyone think that it’s innocuous. After a secret handshake is performed on the port, the global undernet of the Eric Conspiracy runs on that port.

  7. Well duh! The answer is obvious. It is a server that listens on that port. You send text commands indicating your proficiency at various fighting styles, weight, stength etc. along with your location (obtained from gpsd of course),and the information is tested against a database, and responds with the perfect martial arts school for you and your wife.

    Come on Eric, give us a hard one…

  8. When ESR pointed to CIAbot the other day, I hadn’t heard of the CIA service at all, and it took me a while to wrap my head around. Some of this may be due to my lack of experience with IRC (I experimented with IRC a time or two, and I didn’t like it; I didn’t inhale, and and never tried it again), but the whole idea seemed poorly matched for anything I might want. When I want to know what’s going on with a project, I follow the RSS feed from its GitHub account. Asking a chatbot on a different machine for information the forge sent it previously feels “wrong” to me somehow.

    For the sort of notifications that a VCS system might create, IRC seems both too much and too little: does the forge site really need to know how to log in to a chat room? and is a chat string really enough to hold the information that I do want?

    1. >Asking a chatbot on a different machine for information the forge sent it previously feels “wrong” to me somehow.

      It’s not there for queries. The point of CIA is so your project activity generates notifications on the channel where your real-time development conversations take place.

  9. Well if a new unique public port needs to be allocated it sounds like a new protocol or a fork of a previous one. If this is the case is there going to be RFC draft for peer review? Or a new version of Nethack that uses VCS to store configuration and profile data?

  10. That “PalCom” project that Steven Ehrbar mentioned is actually pretty entertaining, if you have a taste for high-budget Lovecraftian farce. From what I could gather — and they never really come out and say what it’s for — it’s a ridiculously heavyweight middleware framework for connecting portable devices together in a way that can be configured by a “user friendly” GUI. And when I say user friendly, I mean that they claim to have developed it “through empirical ethnographic and ethnomethodological studies”, but from what I can make out from the poorly-JPEG-compressed screenshots, it looks to be about as usable as Clippy. Services (of which they steadfastly refuse to give concrete examples) can be defined as subclasses in some Java class hierarchy, and then compiled to bytecode for their own virtual machine that they made for some reason. All configuration, obviously, happens through a mixture of XML and some binary protocols that provide a very efficient, optimized container for their large, verbose, uncompressed XML payloads. All of this is meticulously described in hundreds and hundreds of pages of PDF documents. It is, of course, irreconcilably incompatible with everything else in existence.

    The EU spent about $9 million on this. Of course, nobody actually uses it.

  11. I must admit, I see Justin’s point – what justification is there for consuming a port number for this project?

    1. >what justification is there for consuming a port number for this project?

      It is a new protocol. Well, an application of JSON, but semantically a new protocol.

  12. “through empirical ethnographic and ethnomethodological studies”

    Tell me you made that shit up, *please*….. :)

  13. It is a new protocol. Well, an application of JSON, but semantically a new protocol.

    OK…so…why does this justify reserving a port number? There must be any number of “applications of JSON” out there…we can’t give ’em all their own port…?

    1. >There must be any number of “applications of JSON” out there…we can’t give ‘em all their own port…?

      Your lack of clue about this mystifies and astounds me. I would have thought you knew better than this. I would have thought anybody who’d been a programmer or sysadmin around Unix for longer than three weeks would know better.

      A new port allocation is justified whenever a service protocol is functionally distinct from all previous protocols and thus needs to have its own listener with its own capabilities. The fact that it may have a common syntactic form with other protocols (such as JSON, or XML, or the SMTPish form of a number of early Internet protocols) is completely irrelevant.

      Yes, in fact, every application of JSON that’s going to be a visible protocol on the public net does require its own port. To see why this is so, imagine the havoc if service daemons for two different JSON applications had to share one. That’s not even possible – one of the socket binds would fail. But if it were, how would the objects get routed to the right daemon?

  14. By the way ESR, I’m not asking the above question to be an assclown about it ;)

    I am genuinely interested in the criteria/justifications for obtaining such ports.

  15. @Dan:

    OK…so…why does this justify reserving a port number? There must be any number of “applications of JSON” out there…we can’t give ‘em all their own port…?

    Because as a sysadmin, I can tell you that I expect that any two applications that speak on port foo will speak either the same protocol or at least a compatible protocol.

    It annoys me to no end when a vendor picks a well-known port for some proprietary BS that isn’t supposed to be there. It annoys me even more when I see some open source program doing the same thing.

    Call me a bitter old man, go ahead, but there was a day when crap that like would get you laughed out the room.

  16. “Lack of clue” is a tad harsh, although perhaps I should have worded my query better….I understand perfectly why we have reserved ports, yet do not see why this new JSON dialect you’ve designed requires one.

    Of course I understand that different protocols need to speak on different ports, but that doesn’t mean that you need a *reserved* port.

    My understanding was that reserved ports were for ubiquitous services that required a standard ‘point of contact’ known in advance. However, in these glorious days of loosely coupled services, that seems less critical. A port 80 web service request to *discover* a dynamic port to talk to your new JSON service would seem more in tune with this…rather than hoovering up another morsel of a finite resource.

    Which was really what I was getting at – why is this so critical that it requires a ‘Net-wide port reservation over other approaches for service discovery?

    1. >Which was really what I was getting at – why is this so critical that it requires a ‘Net-wide port reservation over other approaches for service discovery?

      OK, that’s a much more intelligent question. I am relieved that you are not so clueless :-).

      The answer, in this case, is that “other approaches for service discovery” would have significantly complicated implementation, deployment, and use.

      Presently, irker (that’s the name of the daemon) is 361 lines of code and has no dependencies on other services. Code to ship it JSON packets from a hook script can be written in a handful of lines, with the only magic cookie in them being the host/port combination. Because it runs as a daemon, maintaining the persistant state required to talk to IRC servers is simple.

      Everybody’s favorite approach to service discovery these days is magic URLs. Yes, you get to multiplex a lot of services over port 80 that way, but persistent state is significantly trickier and requires more code – you have to futz with session cookies, databases, and it’s pretty difficult to multithread the handler service.

      And it isn’t really the case that port numbers are scarce, anyway. There are presently about 5900 allocated public TCP ports out of a range of 48127 available numbers; that’s 12% utilization, with only a handful of new ones allocated every year. The rate of new requests is probably dropping due to port 80 multiplexing via HTTP – irker doesn’t fit the batchy model that’s easy to do in a webserver, but lots of things do. I don’t think there will be anything even close to a scarcity of port numbers for centuries.

  17. @Morgan – I wholeheartedly agree. If you stomp on a reserved port with your own traffic, you deserve a knuckle sandwich ;)

    No, you’re not a bitter old man, not yet…well…not much :)

  18. Tell me you made that shit up, *please*….. :)

    I don’t see anything that is wrong with it. Europeans spend a lot more time around cultures different from their own than do Americans, and any notions about what constitutes an “easy” or “intuitive” interface have to be viewed through a multicultural lens.

    Anyway, better the EU’s $9M boondoggle that results a vague messaging framework that nobody uses than what the USA has been up to — boondoggles orders of magnitude more costly that result in partially successful underclass surveillance programs.

  19. Yes, in fact, every application of JSON that’s going to be a visible protocol on the public net does require its own port. To see why this is so, imagine the havoc if service daemons for two different JSON applications had to share one. That’s not even possible – one of the socket binds would fail. But if it were, how would the objects get routed to the right daemon?

    That’s simple: all the application daemons live as services — subclasses in some Java class hierarchy. A single daemon listens for incoming messages on the single reserved port; each message is wrapped in an “envelope” with header information describing what kind of message it is; the dispatcher daemon strips off and reads the envelope and forwards the payload to the appropriate service.

    :)

  20. I don’t see anything that is wrong with it.

    ROFLCOPTERS

    I should have known Jeff would respond in such a manner.

    You never let me down, Jeff ;)

  21. @Jeff – Exactly. SOAP/web services, J2EE containers etc have been multiplexing services for quite some time now without any need for individual reserved ports.

    1. >SOAP/web services, J2EE containers etc have been multiplexing services for quite some time now without any need for individual reserved ports.

      And are, in consequence, complex and heavyweight and have dependencies out the wazoo.

  22. > SOAP/web services, J2EE containers etc have been multiplexing services for
    > quite some time now without any need for individual reserved ports.

    Whether those solutions are in any way sane is probably out of scope, however…

  23. @jsk – Sure, the merits and demerits of any particular solution are always up for much debate on the intertubes :)

    The point being that the separation of service discovery from service has been kicked around for some time….as being *arguably* preferable to strongly binding services to a given comms channel.

    I recall the dark days of the [DCE] RPC, CORBA, XML RPC, DCOM et al…..nowadays we actually have some very flexible and workable solutions – with solid open source frameworks – to accomplish such things.

  24. And are, in consequence, complex and heavyweight and have dependencies out the wazoo.

    That is an implementation issue. They don’t have to be, and in fact there are lightweight and/or more performant solutions.

    Not that I’m wishing for a religious war here ;) I just recognize their utility.

    1. >I just recognize their utility

      So do I. But using them would have made irker stupidly and unnecessarily complicated.

  25. I don’t think there will be anything even close to a scarcity of port numbers for centuries.

    Really? With the explosion of mobile iOS/Android computing platforms? You might end up sitting next to Bill Gates’ infamous “640Kb” quote ;)

    I don’t think you’ve really given me an answer that justifies locking down a port from general use – unless ‘convenience/simplicity’ and ‘port abundance’ are to be considered good justification…..but it seems to me that almost anyone could make that argument for their pet project – and then we may end up having to deal with that ‘scarcity’ problem after all, when it might otherwise be avoided.

    Fixed ports seem a bit too ‘hosts file’ in a world where we have dynamic DNS and service brokering. Like IANA is playing FCC with ports, when we should bust through it all much like digital broadcast did. *shrug*

    Anyway, I hope it all works out and is a success.

    1. >Really? With the explosion of mobile iOS/Android computing platforms? You might end up sitting next to Bill Gates’ infamous “640Kb” quote ;)

      Snarky question, but it deserves a serious answer.The infamous Gates quote was stupid because it ignored trend curves; my projection is non-stupid because I’m actually paying attention to them.

      We’ve had five years of history with mass-market mobile computing; if it were going to change the rate of consumption of new port numbers, that would be obvious just by looking at the names in /etc/services. But that isn’t happening, and it’s fairly clear why it’s not happening. Many of the new services that might otherwise eat port numbers are multiplexing through port 80 using HTTP as a transport. This can be expected to continue.

      >unless ‘convenience/simplicity’ and ‘port abundance’ are to be considered good justification

      That’s what I’m saying, yes.

      >but it seems to me that almost anyone could make that argument for their pet project

      Yes. And your point is?

      Dan, it’s supposed to be like that. I’ve never heard of IANA turning down a port allocation request – which doesn’t mean it’s never happened, but does suggest it’s pretty rare. We have thirty years of experience with a regime in which “anyone could make that argument for their pet project”, and the outcome has not been galloping consumption or scarcity.

  26. But using them would have made irker stupidly and unnecessarily complicated.

    You realize you’re throwing down a glove right there, dontcha? ;)

  27. Even IANA recognizes the slow consumption of port numbers, and also that this does not deny the finite nature of the resource – thus requiring good management.

    It does seem that it has been fairly laid-back in granting allocations, but has polished its game somewhat over time – rfc 5526 & 6335 kinda give some insight into how they judge applications.

  28. Dan, it’s supposed to be like that. I’ve never heard of IANA turning down a port allocation request – which doesn’t mean it’s never happened, but does suggest it’s pretty rare. We have thirty years of experience with a regime in which “anyone could make that argument for their pet project”, and the outcome has not been galloping consumption or scarcity.

    I think I see where Dan is coming from. If you’ll pardon, it smacks of arrogance and selfishness to think that “my little client-server app deserves the eternal use of a limited and scarce resource because I’m just that special”. Yes I know, it’s exaggeration for effect.

    You say that reserved ports exist for a reason, well the flip side of that is that *unreserved* ports exist for a reason too. I don’t know a sysadmin who hasn’t set up a service on an unreserved port- generally on an ad-hoc basic, but it can be permanent too. The mind-set here is “what’s so special about your app/server/protocol that it deserves to be classified as well-known?” (and consume the corresponding resource).

    Not that I agree with that in this case, but I understand the mind-set. I think sysadmins (especially ones used to having to be stingy with resources, and who tend to only work with at most an enterprise-wide scope) would approach this differently from developers (especially ones used to working on major projects, the kind that are ubiquitous and enduring).

    1. >I think sysadmins (especially ones used to having to be stingy with resources, and who tend to only work with at most an enterprise-wide scope) would approach this differently from developers (especially ones used to working on major projects, the kind that are ubiquitous and enduring).

      Interesting point!

      I’m very much coming from the “developer used to working on major ubiquitous/enduring projects” mindset. Even so, I only own one IANA port number – gpsd’s 2947. (Anybody going to argue that’s not special enough?)

      It would be a dark day if IANA public port numbers became scarce and were hoarded like jewels. The whole ethos of the Internet’s design is that anyone can play, anyone can invent new services, anyone can have a port. The looseness of the allocation system is a feature, not a bug.

      Fortunately, we won’t run out. Even with today’s range my best guess is we have four centuries of slack. Long before then the size of a port number will change from 2 bytes to 8 bytes or larger, and IANA will have plenty of room to declare new private and public ranges.

  29. Well, I wasn’t really thinking of “arrogance” but rather that accepting ‘convenience/simplicity’ and ‘port abundance’ seems like a weak and open-ended practice potentially vulnerable to abuse, of a “tragedy of the commons” style. Not really the kind of approach I would consider to be a good steward of such a finite resource.

    I’m not trashing this project – it sounds like a great idea – but I have always wondered about the rationale behind port allocation (just never really bothered digging into the details before). In such an open system as the Internet, it seems to me that slamming the door shut on little slices of it should be taken a bit more seriously than “fuggedaboutit….we gotsa lotta ports” ;)

  30. The IANA has no power to compel anyone to obey their port allocations. Their port allocating power comes entirely from people’s implicit approval. If they tighten the criteria and make it more difficult to get ports, then there’s a point at which they start losing legitimacy — if they become too much of a bureaucratic burden then nobody will bother with them, and they’ll become irrelevant.

    Considering the abundance of port numbers, I think it’s practically inevitable that their most popular allocation process doles out ports with a minimum of hassle.

  31. if they become too much of a bureaucratic burden then nobody will bother with them, and they’ll become irrelevant.

    If it hasn’t happened already. Look at all the “unofficial” port numbers on the Wikipedia list.

  32. ROFLCOPTERS

    I should have known Jeff would respond in such a manner.

    You never let me down, Jeff ;)

    Har, har, har.

    While we’re all laughing, a co-worker of mine told a joke: An American was trying to explain what American politics was like to a local while in the UK. He said, “Well, you have the Republicans, which are the equivalent of your Conservative Party; and the Democrats, which are the equivalent of your Conservative Party.”

    From a global perspective American politics is uniformly right wing. There is only one extrinsically visible party: the party of the elite. A correlated phenomenon is a profound tunnel vision that Americans seem to have when considering matters of a political or cultural nature, though it’s difficult to determine which is the cart and which the horse here.

    Personally I find that the EU government funded such humanitarian computing research fascinating. Contrast that with Murka, where the bulk of significant government funded computing research is directed toward warfare.

    I did some reading up on PalCom, and it’s not quite the ploy for grant money with no clear purpose that you made it out to be. PalCom is short for “palpable computing”, where “palpable” means “easily accessible through a physical interface”. The idea is that since in the future computers are going to be integrated into many of the tools we use, there is a clear and present danger that they may function less like tools and more like computers. A simple example is the fact that it has become increasingly difficult to diagnose car trouble because the running of a car is increasingly microprocessor-mediated. Sure, you can buy a diagnostic tool from the manufacturer that hooks into their proprietary connector and can decipher their proprietary error codes, but that’s a far cry from the days where you could look at an engine and SEE which part had failed and why. In PalCom parlance modern cars are not “palpable”. The goal of PalCom is to reverse this trend, so that embedded systems can smoothly integrate into the lives of ordinary people — regardless of their cultural backgrounds, hence the “ethnographic and ethnomethodological” bits of the research.

    The potential applications are not specified in detail in part because part of the research entails finding applications for this technology. Yes, it costs more than it should and is smothered in red tape, but so is any university research. Again, it’s a much, much better way to spend taxpayer dollars than, say, the American government’s recent resurgence of interest in AI, which is motivated entirely by potential surveillance and anti-“terrorist” applications.

  33. Jeff, as far as I can tell, your whole post comes down to arguing that the PalCom guys had good intentions and a genuinely important problem to solve. I don’t dispute this! Greater transparency and easier debugging in software systems is a Good Thing. Great honor and glory await anyone who can make a serious contribution to getting us out of the twisty maze of protocols and buses and user interfaces, all different. Lovely intentions, and a worthy problem.

    The problem is, none of that matters; results do. And what are the results of this research? They came up with what looks like a decent concept for presenting a plug-blocks-together view of a collection of devices, and an implementation that nobody will use, which somehow manages to be both half-assed and overly elaborate. Oh, and they got a fair number of papers out of it, which is nice, I suppose. The thing is, these are not new ideas; while their particular spin on it might have some novel and interesting ideas, (almost?) all of what they’re proposing is available via a bunch of other protocols, or not available because nobody bothered. Adding one more incompatible middleware system doesn’t improve this situation! It doesn’t help that their approach is so inelegant, and their scope so creepingly over-broad.

    Unless the primary goal here is research funding and paper publication, the whole projects seems predictably, inevitably ineffective.

  34. He said, “Well, you have the Republicans, which are the equivalent of your Conservative Party; and the Democrats, which are the equivalent of your Conservative Party.”

    Nah. Bad joke. Misses the mark entirely.

    Labour Party == Democrats
    Conservative Party == Labour Party + nicer suits

    Republicans == exploding euroheads ;)

    From a global perspective American politics is uniformly right wing

    Bunk. The rest of the world can identify, and relate to, the Democrats for what they are – ‘Old World’ pinko statists.

  35. Republicans == exploding euroheads ;)

    If by “exploding” you mean “striking atavistic terror in” you might be right.

    Scientific research has confirmed what many of us who read A&D have long suspected: free-market ideology is STRONGLY correlated with anti-scientific willful ignorance.

    The “Old World pinko statists” are winning all the arguments. Usually because the facts are on their side.

  36. esr writes:
    > >Which was really what I was getting at – why is this so critical that it requires a ‘Net-wide port reservation over other approaches for service discovery?
    >
    > OK, that’s a much more intelligent question. I am relieved that you are not so clueless :-).
    >
    > The answer, in this case, is that “other approaches for service discovery” would have significantly complicated implementation, deployment, and use.

    How is `use a DNS SRV record’ not an acceptable fit for this application?

  37. Another gem from Jeff.

    You truly are the gift that keeps on giving, brother ;)

    Keep ’em coming…..just don’t CC-NC-whatever them….I may wish to compile a book of lefty balderdash one day :)

  38. “I” “have” port 18. Message Send Protocol. :-)

    Oh, and Jeff: “free market” is not an ideology. It’s a fact that markets regulated by customers work out better for customers, and markets regulated by governments work out better for governments. But you keep advocating for the latter if you really want unhappy customers and happy governments.

Leave a comment

Your email address will not be published. Required fields are marked *