Rust and the limits of swarm design

In my last blog post I expressed my severe disappointment with the gap between the Rust language in theory (as I had read about it) and the Rust language in practice, as I encountered it when I actually tried to write something in it.

Part of what I hoped for was a constructive response from the Rust community. I think I got that. Amidst the expected volume of flamage from rather clueless Rust fanboys, several people (I’m going to particularly call out Brian Campbell, Eric Kidd, and Scott Lamb) had useful and thoughtful things to say.

I understand now that I tested the language too soon. My use case – foundational network infrastructure with planning horizons on a decadal scale – needs stability guarantees that Rust is not yet equipped to give. But I’m somewhat more optimistic about Rust’s odds of maturing into a fully production-quality tool than I was.

Still, I think I see a problem in the assumptions behind Rust’s development model. The Rust community, as I now understand it, seems to me to be organized on a premise that is false, or at least incomplete. I fear I am partly responsible for that false premise, so I feel a responsibility to address it square on and attempt to correct it.

Technically, I greatly admire Rust’s crate system. I see its continuity with earlier experiments in the same general direction – Perl CPAN, Python Pip. But crates are easier to use than their predecessors and get more details right. The differences are evolutionary, but they matter – they add up to making crates a near ideal tool for enabling code re-use and a decentralized swarm attack on the design space around Rust.

It’s “let a thousand modules bloom”, and exactly the kind of approach I advocated in my foundational work on the open-source development model. I pushed hard in that direction back around the turn of the century because that’s what the times demanded; our main problem then was getting out from under overcontrolling practices that were holding back software engineering and serving both its craftsmen and its customers rather poorly.

Now, confronting a purer expression of those decentralist ideas than we then knew how to build, I worry that I may have encouraged a kind of utopianism – a belief that if we just cultivate enough decentralization and divergence of approach, good whole systems will just sort of coalesce out of the chaos without anyone having to make hard decisions.

But evolution doesn’t work that way. Ratcheting up adaptive fitness requires not just mutation but selective pressure. Alternatives need to be winnowed as well as generated. Markets (even reputation markets) have penalties as well as rewards – if you offer an inferior product, people won’t buy it and eventually you need to go do something else to survive.

Which brings me directly to what bothers me about the crate system and the sociology behind it – I don’t see any pruning. Worse, I don’t see much understanding of the need for it. A lot of Rustaceans don’t seem to grasp why, when the question is “where do I get feature X?” the answer “oh, there are 23 crates for that” is objectively terrifying.

How many of those crates are student exercises or plain junk? How do I tell which ones are still maintained? How do I audit for quality? How do I form expectations about which will still be maintained in ten years? How do I even find all the crates that might be relevant to my interests?

This isn’t a question that comes up so often with respect to (say) Python because Python has an effective form of curation – blessing things into the standard library, at which point their alternatives generally disappear. In effect, Python modules are quality-filtered on the taste of the BDFL and the devteam.

(This is, in fact, why Benevolent Dictator For Life is such a common governance model in our otherwise rather anarchic community. Experience has shown that curation by one person with a clear design vision generally beats attempts to design by committee.)

The same question comes up at other levels. At the whole-program level, we have operating-system distributions precisely in order to curate the riotous diversity of software into semi-coherent wholes with an implied minimum quality threshold. Again: selective pressure.

Rust seems to be missing any analogous mechanism, and that worries me a lot. It’s what I meant when I said in my last post that the swarm attack seems to be failing.

To sharpen this point, I’ll tell you what I think “success” looks like. As a potential Rust user, what I want to do is be able to go from a given feature requirement to one high-quality implementation with an implicit long-term stability guarantee. This, folks, is what a high-quality production language looks like, and not by accident – it minimizes discovery costs for actual production users.

Getting to this from the crate ecology as it is now conceived is not just a technology problem, it’s a a challenge to how the Rust community imagines itself. And, as I trust I’ve made clear, not a unique one. Any sort of open-source ecology eventually has to face a similar transition to maturity.

It could be as conceptually simple as adding a rating system to crates.io, allowing the user to filter on ratings, and having one of the ratings being “Approved by the core team; we take responsibility for this.” But someone is going to have to decide. Who is it going to be? And who decides who will decide?

Backing up a bit: in scheduling theory and operations research there’s a class of problems called “explore/exploit dilemmas”. They show up everywhere where you have limited resources to invest and have to choose between generating options so you’ll have good ones later and harvesting the rewards from those you already have. They show up in the strategy games I like to play, especially in pick-up-and carry games where (say) you’re running a simulated railroad to try to make the money the fastest. Early in the game, you’re mainly interested in investing so you can build an efficient network. But later on you have to back off building track and pile up money using what you have, otherwise you’ll be overtaken by players who chose that transition point more shrewdly.

Rust has built a good machine for exploring. But you guys have competition out there: Go, Nim, Swift, D, and maybe other languages that aren’t on anyone’s radar yet. I think you’re going to need to make that turnover into exploit mode before you know it, and that this is tied into your challenges about curation and governance.

Here’s my advice, if you’ll take it: get conscious!. Grapple with what success looks like from a potential user’s point of view, the challenge of curation, and the problem of who decides. It’s either that, or remain a toy language with an inward-facing community forever.

69 comments

  1. Rust is definitely in the rapid-expansion phase (for relative definitions of rapid); the notion of general blessings of which crates are best suited for which tasks is one that is on many people’s minds (and crates.io is working on a means of showing this), but the Rust ecosystem isn’t yet at a point where that’s the case for a large number of crates. There are a handful of cases, like regex and serde, but the most prominent case at this particular moment, tokio, is nowhere near ready to commit to being the lingua franca of Rust’s async story.

    Rust’s team is also specifically *not* aggregating such crates into the standard library, since that freezes them far more than does saying “these are the community-recommended crates for these purposes”. The community-standard crates will eventually become a stable tier below the standard library, but for now everything’s still growing.

    Which is exciting, if you’re in a position not to be put off by that, but definitely off-putting if you’re in a position where stability is more necessary.

  2. I think it’s worth mentioning that it is SPECIFICALLY on the roadmap for Rust to have “1.0 level” (read: stable) crates for “common tasks” by the end of this year, which will be highlighted and well-advertised.

    I would liken the current period to Python 2’s early period, when distributing packages was still relatively difficult, PyPI did’t exist, and we didn’t have anywhere near the number of high-quality packages that we do now.

  3. It’s a little odd to me that you say this in the context of select and epoll. The former is part of POSIX and the latter is Linux specific, neither one is blessed by ISO C. You don’t even get a hash table with C. And there’s no package system.

    But that’s not to say you don’t have a point. It’s just easy to get distracted by the comparison to C, which doesn’t make sense to me.

    A few days ago, I saw a link to the Rust roadmap: https://github.com/aturon/rfcs/blob/roadmap-2017/text/0000-roadmap-2017.md On it, is “Rust should provide easy access to high quality crates”. So at least I can say this is on their radar, even before your blog post.

    My understanding, and I could be wrong, is they are doing what you suggest, and you just happened to look too early. The libraries that will eventually be blessed aren’t ready yet, and it’s not yet clear which they will be, at least in some cases, but things are well on their way.

    I certainly understand why having 7 packages that all do the same thing can be terrifying. But you can ask on forums and mailing lists and reddit what people are using. You can see which projects have the most stars and commits on github or the most downloads on crates.

    But ultimately I think it’s a curse of early adoption. No one truly knows the future, and Rust hasn’t been around for 30 years yet.

    As an aside, you should check out the npm and the Javascript ecosystem. Unless I’m missing something, they’re a larger, even more chaotic ecosystem.

  4. This isn’t a question that comes up so often with respect to (say) Python because Python has an effective form of curation – blessing things into the standard library, at which point their alternatives generally disappear. In effect, Python modules are quality-filtered on the taste of the BDFL and the devteam.

    I’d argue that there are two major ways in which you’re mischaracterizing the situation:

    First, the crates.io developers are well aware of its shortcomings and are currently designing improvements. In fact, within the last month or two, I remember taking a survey they presented to make sure they understood how highly various factors rate when people are judging a dependency (in any language) for suitability to a given purpose.

    Rust also has https://github.com/rust-lang-nursery/ as a place where candidates for eventual inclusion into the standard library (or some similar aggregation of blessed-and-API-frozen code) are being given time to mature in the spotlight in case any unexpected warts are still lurking that would require a new major version before stdlib inclusion prevents new and old versions from coexisting in the same ecosystem.

    Second, my experience with the Python community is that Python follows a pattern similar to Rust, but less conservative …and they’re trending toward Rust’s conservatism after getting burned so many times by code that was revealed as deeply suboptimal in by later 3rd-party packages. (It wasn’t too long ago that much drama was had over the fact that finally enabling SSL/TLS certificate validation in stdlib would cause many breakages in deployed code and, because it’s stdlib, “depend on an older major version” wasn’t an option.)

    For quite a while now, the Python ecosystem has taken the view that “stdlib is where packages go to die” and, as a result, they’ve been trying to retrofit a cargo/crates-style packaging story into the language.

    Even ignoring that, stdlib inclusion in Python is reactive, not proactive. I’ve been programming in Python long enough to clearly remember that modules like sqlite3 and ElementTree were merged into the standard library AFTER they approached de facto standard status.

    Furthermore, there are quite a few modules which, despite being in the standard library and not listed as deprecated, are “de facto deprecated” by community consensus… often with a single third-party alternative that refuses to become part of stdlib because “stdlib is where packages go to die”:

    asyncore/asynchat/imaplib/etc. are in stdlib but everyone agrees that you should use Twisted instead
    BaseHTTPServer and its subclasses are in stdlib but, regardless of whether you choose a WSGI host, an HTTP API mocking framework, or what have you, the consensus in places like #python @ FreeNode is “Do NOT use the stdlib HTTP servers. It will only end in tears.”
    Pickle/cPickle are in stdlib, but the consensus is to use anything else because their approach to serialization and deserialization is inherently insecure and “arbitrary code execution by design”.
    ctypes got into the stdlib because people were so eager to have a C foreign function interface that didn’t require writing bindings in C, but then PyPy came along and developed CFFI which people now agree provides a more robust and portable way to bind to C libraries.
    the standard library provides the distutils packaging framework, but the ecosystem is now so dependent on the setuptools superset of it that I’m not even sure how viable it is to use distutils on its own anymore. (To the point where the standard library now contains an `ensurepip` module which you can use to install `pip`, `virtualenv`, and `setuptools` because they’re not part of the standard library but the ecosystem depends on them.)
    Tkinter is included in the standard library, but people generally avoid it unless they don’t know any better because it’s greatly inferior to 3rd-party widget toolkit bindings in multiple ways and, on Linux, doesn’t even have the “guaranteed to be installed by default” advantage. (Debian-family distros split out into an optional package like the development headers, documentation, debugging symbols, code examples, etc.)
    urllib and urllib2 (to use their more illustrative Python 2.x names) are strongly recommended against by the community, instead advising people to use the Python Requests library or, at the very least, the out-of-stdlib urllib3 that it wraps.
    Not really a “don’t use stdlib” moment, but the Python standard library contains three different XML APIs, only one of which is recommended and, even then, a lot of people use a third-party drop-in replacement called LXML which provides a much richer feature set.
    …not to mention the packages which were officially deprecated for ages (eg. the image-related ones) and finally removed in Python 3.x in favour of 3rd-party packages like Pillow (the friendly fork of PIL, the Python Imaging Library).

    …and those are just the packages I’ve found to be “de facto deprecated” because I needed their functionality. Most of stdlib I’ve never even touched.

  5. Ugh. It appears this blog ISN’T operating on WordPress’s “we’ve stopped advertising it, but we still allow HTML blockquote/ul/li/etc. tags” rules.

    The first paragraph I posted is supposed to be a quote and here’s a more manually marked-up version of that big blob:

    * asyncore/asynchat/imaplib/etc. are in stdlib but everyone agrees that you should use Twisted instead

    * BaseHTTPServer and its subclasses are in stdlib but, regardless of whether you choose a WSGI host, an HTTP API mocking framework, or what have you, the consensus in places like #python @ FreeNode is “Do NOT use the stdlib HTTP servers. It will only end in tears.”

    * Pickle/cPickle are in stdlib, but the consensus is to use anything else because their approach to serialization and deserialization is inherently insecure and “arbitrary code execution by design”.

    * ctypes got into the stdlib because people were so eager to have a C foreign function interface that didn’t require writing bindings in C, but then PyPy came along and developed CFFI which people now agree provides a more robust and portable way to bind to C libraries.

    * the standard library provides the distutils packaging framework, but the ecosystem is now so dependent on the setuptools superset of it that I’m not even sure how viable it is to use distutils on its own anymore. (To the point where the standard library now contains an `ensurepip` module which you can use to install `pip`, `virtualenv`, and `setuptools` because they’re not part of the standard library but the ecosystem depends on them.)

    * Tkinter is included in the standard library, but people generally avoid it unless they don’t know any better because it’s greatly inferior to 3rd-party widget toolkit bindings in multiple ways and, on Linux, doesn’t even have the “guaranteed to be installed by default” advantage. (Debian-family distros split out into an optional package like the development headers, documentation, debugging symbols, code examples, etc.)

    * urllib and urllib2 (to use their more illustrative Python 2.x names) are strongly recommended against by the community, instead advising people to use the Python Requests library or, at the very least, the out-of-stdlib urllib3 that it wraps.

    * Not really a “don’t use stdlib” moment, but the Python standard library contains three different XML APIs, only one of which is recommended and, even then, a lot of people use a third-party drop-in replacement called LXML which provides a much richer feature set.

    * …not to mention the packages which were officially deprecated for ages (eg. the image-related ones) and finally removed in Python 3.x in favour of 3rd-party packages like Pillow (the friendly fork of PIL, the Python Imaging Library).

  6. > Technically, I greatly admire Rust’s crate system. I see its continuity with earlier experiments in the same general direction – Perl CPAN, Python Pip.

    It was influenced from NPM, not the other two.

    > Which brings me directly to what bothers me about the crate system and the sociology behind it – I don’t see any pruning. Worse, I don’t see much understanding of the need for it. A lot of Rustaceans don’t seem to grasp why, when the question is “where do I get feature X?” the answer “oh, there are 23 crates for that” is objectively terrifying.

    I’ve been using Rust for two years and I have never come across this issue. Others in the community also do not have issues with this.

    > This isn’t a question that comes up so often with respect to (say) Python because Python has an effective form of curation – blessing things into the standard library, at which point their alternatives generally disappear. In effect, Python modules are quality-filtered on the taste of the BDFL and the devteam.

    Python doesn’t have any curation. The standard library is littered with horrible designs and implementations, where it’s better to ignore the standard library and use external libraries which are more efficient.

    Python doesn’t do this, but Rust does. During the 1.0 release, many parts of the Rust standard library were left out to be developed outside the standard library until a date when these libraries had matured. Over time, these official crates are re-incorporated into the standard library after they have matured.

    > To sharpen this point, I’ll tell you what I think “success” looks like. As a potential Rust user, what I want to do is be able to go from a given feature requirement to one high-quality implementation with an implicit long-term stability guarantee. This, folks, is what a high-quality production language looks like, and not by accident – it minimizes discovery costs for actual production users.

    It’s a terrible idea to only allow for one high quality implementation. In order for open source to move forward, it is better to allow for competing libraries.

    > It could be as conceptually simple as adding a rating system to crates.io, allowing the user to filter on ratings, and having one of the ratings being “Approved by the core team; we take responsibility for this.” But someone is going to have to decide. Who is it going to be? And who decides who will decide?

    If you had been following Rust’s Reddit forum, you would know that there has already been much consideration on displaying better results for Crates searches with a new rating system.

    > Rust has built a good machine for exploring. But you guys have competition out there: Go, Nim, Swift, D, and maybe other languages that aren’t on anyone’s radar yet. I think you’re going to need to make that turnover into exploit mode before you know it, and that this is tied into your challenges about curation and governance.

    Go isn’t really competing in the same space. Go only aimed for developing web services at Google, and it pretty much bombs for any task that isn’t that. The tooling is terrible, and the build system is vastly inferior to Cargo.

    D has serious issues with split implementations of the standard library, a completely disorganized community, and serious issues with packaging in Linux distributions.

    As for Swift, it’s just Apple’s poor man’s version of Rust. It has pretty much zero reception outside of the Apple camp.

    > Here’s my advice, if you’ll take it: get conscious!. Grapple with what success looks like from a potential user’s point of view, the challenge of curation, and the problem of who decides. It’s either that, or remain a toy language with an inward-facing community forever.

    Here’s my advice: spend time with the community — that means engaging on Reddit — so you know what is or isn’t being considered or worked on. Additionally, dial down your trolling because calling the hard work of many a ‘toy language’ is highly insulting.

  7. There are a few points with which I’d disagree here, but I agree with you pointing out this central challenge of community curation. There are a few efforts to figure this out, one of which is proposed here:

    https://github.com/integer32llc/rfcs/blob/crates-io-ranking/text/0000-crates.io-default-ranking.md

    With a conversation around it here:

    https://github.com/rust-lang/rfcs/pull/1824

    Certainly not perfect, but the community is definitely aware that it’s a challenge but is eager to figure something out which improves the situation. I think this is another situation where there’s a lot of “tribal knowledge” among existing users (do use this library, don’t use that) and it needs to be figured out how to disseminate and formalize that.

  8. Thank you for an honest assessment of the limitations of collective intelligence. Have you read Jaron Lanier’s “Digital Maoism”? Your observations are similar, albeit concerning different fields. (I know you two once had a tense debate, but I admire both of you.)

    And the part about explore/exploit dilemmas reminds me of “A secret of game-fu”. I’ll try to apply your insights to Freeciv. ;-)

    Only typos I found in the present thread’s OP: “it’s a a challenge” and “pick-up-and carry games” (there should be a hyphen between “and” and “carry” as well).

    I see [Rust’s crate system’s] continuity with earlier experiments in the same general direction – Perl CPAN, Python Pip.

    I suppose Ruby’s “Gems” and Chicken Scheme’s “Eggs” fall into the same category.

  9. I think it’s important to make a distinction between what a newcomer perceives and what the ecosystem *is*. From the inside of the Rust community, the swarm doesn’t really seem like a swarm. It’s never “there are 23 crates for this”, it’s usually “use crate X”. In some cases, there may be a couple choices but they’re all reasonable and cater to different use cases. For example, for your previous post, the suggestions would be to use epoll if you want to be very low level, mio if you want to be higher level, and tokio (new and incomplete, but uses mio under the hood) if you really want a very Rusty API to use. The ecosystem is converging on hooking into tokio for everything async.

    C has the same problem, really; you can find all kinds of libraries out there but only a few are the ones you’re *actually* supposed to be looking at. C doesn’t have this kind of curation at all. And C is certainly a “production” language. C doesn’t even have a central index of libraries that people have published, so it’s even worse. But the nebulous C “community” does know; you can ask people and you will get an idea of the library space.

    Of course, this is not to say that it isn’t a *problem*. It’s just not as bad as it sounds. It’s not a fundamental issue with the crates model that will take incredible amounts of effort to extricate from the ecosystem, it’s just a disconnect in what the community “knows” and what information is presented to newcomers, and that’s a disconnect that’s easier to fix.

    The Rust community is well aware of this! It’s been a known problem for a while, but the community has grown only in the last year to the point where “ask on a Rust forum” is an unacceptable answer to “how should I figure out how to do things”. The situation has improved on this quite a lot since 1.0, but crate recommendations are one of those issues that still need to be tackled.

    While this has been a while in the making, recently an RFC was created to address this, https://github.com/rust-lang/rfcs/pull/1824 . A lot of the issues of ranking crates were discussed there. I’m not faulting you for not knowing this, since we should not expect newcomers to be aware of all the goings-on of the community. But you might want to have a look at that and leave your comments.

    For the web stack in particular, http://www.arewewebyet.org/ exists, though it’s not as “official” as whatever that rfc ends up proposing is. But it’s an experiment in what could be done, potentially for the whole ecosystem.

    Regarding “Worse, I don’t see much understanding of the need for it.”, I think that’s just community members who are not affected by this (i.e. they already know what crates to use by virtue of being in the in-crowd). Plenty of people care about this, and this is something that folks have been pushing for. In particular, Carol and Jake Goulding have been continually running experiments and tinkering with this, and probably will continue to do so in pursuit of a really good solution.

    ——

    Regarding the Python comparison, Rust’s version of “blessed thing within the standard library” is https://github.com/rust-lang/ ,e.g. https://github.com/rust-lang/regex . There are more crates in https://github.com/rust-lang-nursery/ , the “nursery” is where these crates start out and are somewhat blessed (in that they’re maintained by the Rust core team with others, and are on track to become “blessed”, but not yet moved to rust-lang). Python has a problem with “the stdlib is where modules go to die”, which this avoids. Versioning libraries separate from the compiler allows for rapid improvement.

    > what I want to do is be able to go from a given feature requirement to one high-quality implementation

    I think it’s important to mention here that “feature requirements” can be vague at times. For example, if you want a parser library, there are many parsing libraries in Rust, but the number usually gets whittled down to 1 or 2 if you start being more specific like “parser combinator” or “LALR parser generator” or whatever. I am hard pressed to think of two libraries in Rust that do the exact same thing, but it’s easier to find libraries that are good and work in the same space and for many use-cases may work as well as the other (but for other use cases only one will be applicable, it depends on how constrained your needs are)

    I also don’t really find Rust to be very different from a BDFL model. I find that the core team of 8 people are always in agreement over the general direction and philosophy of Rust, which is really what a good BDFL does anyway. They may disagree on smaller decisions but that’s okay. Rust has an RFC model like the PEP model, which is overseen by the core team and subteams (which are led by core team members).

    ——

    When I started using Rust last year I came across a similar stumbling block when it came to crates. I just hopped on IRC and asked in #rust, and people gave good recommendations. They discussed my needs with me and usually it boiled down to a single crate. I think there’s also a #rust-beginners channel now which is even more helpful. Like I said, this isn’t to say that this isn’t a problem, but I find the solution of asking on a Rust forum adequate for my own needs for now. You might too. In the long run this is not something we should have and fixing it is being actively worked on, but in the meantime it’s fine IMO.

  10. “As for Swift, it’s just Apple’s poor man’s version of Rust. It has pretty much zero reception outside of the Apple camp.”

    Tell it to IBM.

  11. I’m not sure that I buy the premise of a binary choice between “one way of doing everything”, and the wild west of JavaScript’s NPM.

    My primary professional language is Java. It’s syntax draws criticism both from older C/C++ programmers as well as the younger dynamic crowd… but everyone agrees that it has one of the most successful ECOSYSTEMS in history.

    This ecosystem is not curated by Java’s dictator (Sun/Oracle). In fact, Oracle “blessing” a particular solution is a good predictor that the solution will be rejected by the ecosystem! The crates-like platform driving Java’s ecosystem (Maven Central) itself arose from the community without any involvement from Sun or Oracle. Anyone can publish a new library to the global Maven Central repository in less than a day.

    Still, there are clear winners, best practices, and standards in the Java ecosystem. No matter what you’re looking for, the odds are likely that you should be using something from the Apache Foundation, the Eclipse Foundation, or Spring. There are a hundred niche options in web development, but two major frameworks hold probably 90% of the market share. If you need a database ORM library, then you can count the serious options on one hand (and Hibernate will probably always be the default). Those are really common needs… for more niche use cases, usually there’s one clear winner. The “discovery cost” for a newbie trying to find the most appropriate framework or library is usually 5 minutes of Googling or community chat.

    So I really don’t believe that community standards fall into place because a BDFL declares them. Guido van Rossum can’t make people migrate to Python 3 after nine years of trying (!), so his curation of Pip modules feels a bit exaggerated.

    Maybe the real issue is that Java and Python are over 20 years old, and the community has had time to vet, choose, and solidify around winners and losers, defaults standards and niche options. I’ve been dabbling with Go for about 5 or 6 years now, and a similar maturation process in that ecosystem has been obvious over the past year or two.

    I see no indication of philosophical differences or process impediments that would prevent the Rust community from heading in the same direction in due time. Perhaps the problem is that it simply isn’t reasonable to expect 20 years of ecosystem maturity to already be manifest 20 MONTHS after its first stable release?

  12. Sheesh. There are like 6 comments to this post and I’m already having flashbacks to the “Hey, Democrats!” thread. As someone who occasionally writes scripts and software, but is not a programmer, I’m always amazed at how flippantly people reject the advice of those with more experience.

  13. > Technically, I greatly admire Rust’s crate system. I see its continuity with earlier experiments in the same general direction – Perl CPAN, Python Pip. But crates are easier to use than their predecessors and get more details right. The differences are evolutionary, but they matter – they add up to making crates a near ideal tool for enabling code re-use and a decentralized swarm attack on the design space around Rust.

    I’m not terribly familiar with the Python ecosystem, unfortunately—I mostly just use Python for NumPy, SciPy, Pandas and Jupyter. I do spend a fair bit of time in the JavaScript and Ruby worlds, and I have a feel for how their ecosystems work.

    In the Ruby world, you have rubygems.org, which is very well done. There are thousands of gems. But everybody understands that the Ruby world typically has three tiers of gems:

    – The “consensus” solution. This is often what DHH has blessed by inclusion with Rails. For unit testing, this would be Test::Unit. Perhaps 70% of Ruby programmers will typically use the consensus solution.

    – The “alternative” solution. This is the choice of maybe 25% of the community who can’t stand the “consensus” solution. For unit testing, this would be RSpec. It’s usually mature and supported on at least a half-decade timescale.

    – Everything else. This is what you use if you’re terminally hip or have weird special requirements. If it integrates with Rails, then it will break for good after the the next major release of Rails.

    The situation with JavaScript and Node.js is more complex. At the time of writing, there are 399,773 packages on npmjs.com. And yet, there are really obvious categories with no good alternatives. For example, I needed a CLI option parser the other day, and I needed it to support git-style subcommands. I searched npmjs.com for 30 minutes and spoke to our in-house Node guru, and everybody said, “No, every single Node package for your use case is terrible.” This makes me sad and frustrated. I eventually found something that was, frankly, pretty bad and cudgeled it until it worked.

    If you’re “on the inside of” the Rust ecosystem, it currently feels more like the Ruby world than JavaScript one. For example, the “consensus” and “alternative” argument parsers are clap and docopt with 300,000+ downloads each. clap is the right choice if you have complex, git-style arguments. docopt is less powerful, but it’s elegantly simple. Anything else you can safely ignore. Both are high-quality, extensively tested and actively supported, and arguably better than anything I could find for Node.

    Sadly, docopt doesn’t show up if you search for “argument parser” on crates.io. Not sure what’s up there, but I agree that it’s a problem. When in doubt, drop by the #rust channel on Mozilla IRC and ask what crate to use. One of the big roadmap goals for 2017 is to make it easier to identify high-quality crates on crates.io. People are brainstorming ideas.

    But culturally, the Rust community does tend to want to pick one or two “winners”. Now that tokio is out, and the consensus is that this it the right way forward for async I/O, dozens of other libraries are being overhauled to support it. The underlying psychological dynamic feels almost as if the Rust community believes in a “the right thing” in the old Lisp sense, and likes the idea of there being one excellent library in most categories. People would rather submit PRs than rewrite things from scratch. Your problem, perhaps, was that you simply looked for an async I/O library shortly after the consensus formed but before the implementation work was ready.

    I know that none of this apparent from where you’re sitting. This is definitely a bug in my opinion. Anyway, thank you for writing up your experiences getting started with Rust—it’s always valuable to know where people are getting stuck or where the communication needs to improve.

  14. There is a tension between having the best standards, and having the most ubiquitous standards. I’m a little surprised that nobody has mentioned this here or in the previous posting. It is nearly impossible to standardize on the best implementation of something. And if you think you have, just wait a little bit.

    I know that everybody who is running “Python” has urllib2. They don’t need to fetch and install Requests in order to run my code. Yes, it looks a little ugly in the way it constructs the HTTP request, but people can just let their eyes glaze over unless they need to understand how it works. Although ugly, it’s well documented, so …. I’m not going to worry about Python dilettantes not understanding the core part of my code that they’ll never need to change anyway.

    So yeah. urllib2 is my go-to tool for TCP connections, not Requests, and not urllib3. Because …. standards. Because …. Batteries Included.

    BTW, this posting is not about urllib2. If you think it is, read it again.

  15. @D. T. Anonymous:

    * asyncore/asynchat/imaplib/etc. are in stdlib but everyone agrees that you should use Twisted instead

    Or tornado, or eventlet, or gevent, or…

    This is actually an area where Python’s stdlib AFAIK has never gotten much traction. (Even asyncore was based on a third-party library, medusa, that got a fair bit of usage as a third-party library, but not much as a stdlib module.) The latest standard Python attempt in this area, the async/coroutine changes in 3.4 and 3.5, struck me as disappointing (for one thing, adding a new keyword seemed both unnecessary and counterproductive).

    However, I think it’s worth noting that the Python stdlib modules that give rise to the “go there to die” meme are not what I think Eric was referring to as “core functions”. They are more like higher level API wrappers. For example, asyncore is really a wrapper (a clunky one) around the select stdlib module; but the latter module works just fine on its own.

  16. A couple points about Ruby, since it’s been mentioned a few times.

    Ten years ago, the latest stable version of Ruby was 1.8.7 and Rails was at 1.2. If you tried setting up a major application built on that stack on a modern system:

    1) The Ruby runtime may or may not compile. Expect problems. If it won’t compile you’re screwed because:
    2) Rails 1.2 will not run on an a modern Ruby (2.X).
    3) Event if the above goes well, chances are, your native dependencies will not compile.

    Given Rust’s aims, permitting the above would be unacceptable.

  17. As many pointed out, there is no reason to think that Python is doing a much better job in defining the core set of libraries. The real difference here is one’s age: Python is 25 years old, Go is 7 years old, and Rust is not yet 2. So it is hardly surprising that Python is much more mature than Rust, but I believe Rust is doing fine for its age. Obviously Rust still has a lot to gain in terms of ecosystem maturity, but the Rust team is aware of that.

  18. Josh Kaplan:
    C has the same problem, really; you can find all kinds of libraries out there but only a few are the ones you’re *actually* supposed to be looking at. C doesn’t have this kind of curation at all. And C is certainly a “production” language. C doesn’t even have a central index of libraries that people have published, so it’s even worse. But the nebulous C “community” does know; you can ask people and you will get an idea of the library space.

    That was exactly my reaction to the original article. Given the amount of success that C still enjoys, is seems that an officially accepted and well-publicized procedure to select a set of “best” libraries is not strictly necessary. On the other hand, the need for such a procedure is clearly championed in the article (and I agree with that). I can reconcile both aspects only by considering the existence of a procedure that is widely known by insiders, but not well publicized to newcomers.

    My question is: how does the C community select the “best” library for a task, and how does it communicate those libraries to new components of the community?

    I am not comparing C and Rust communities, and I am not saying that the two ecosystems should be held to the same standards. Since C is an accepted systems language for long-term programs, how does its community route around a defect that is believed to be fatal for Rust?

    1. >Given the amount of success that C still enjoys, is seems that an officially accepted and well-publicized procedure to select a set of “best” libraries is not strictly necessary.

      I think that’s an indirect consequence of C’s weak type ontology. It’s so difficult to build objects and frameworks in C that the “library binding” of C is just POSIX. Which is (at least compared to the package ecology of a more modern language) small, simple, well tested, and has public standards, so the discovery problem doesn’t arise in quite the same way,

  19. “Ask on a forum/IRC channel” is not something that scales. That requires knowledgeable people available 24/7 to answer questions. And you’re dealing with developers whose opportunity cost is several hundred dollars per hour.

    Automated browsing/searching/ranking systems do scale much better.

  20. Perhaps the problem is that there aren’t enough “consumers” of Rust libraries. That the proportion of “writers” of Rust libraries in the Rust community is too high.

    Like Steve Perkins above, I’m a Java programmer. The Java ecosystem is pretty good, but there is no individual curating it. But the vast, vast majority of Java programmers use external, non-JDK libraries, but don’t write them. We’re all working on our own business software. I think that large “consumer programmer” community provides the selective pressure that winnows the libraries. Discussion and word of mouth among the consumer programmer community identify and push up the good libraries, and real-world use-cases push adoption of new features.

    Maybe what Rust really needs is fewer programmers writing Rust crates and features, and more programmers who use those features to write end-user programs that won’t be redistributed.

    1. >Maybe what Rust really needs is fewer programmers writing Rust crates and features, and more programmers who use those features to write end-user programs that won’t be redistributed.

      This.

      The Rust community looks, from the outside, a bit ingrown and subject to a kind of “can you top this (complexity)” syndrome. I don’t mean this observation in a hostile way; if I were eyeball-deep in the development of a young language I would probably behave the same way – and arguably I did, back in Python’s earlier days when I was briefly on that devteam.

      But yes, I think more feedback from people who just want to get stuff done and have a lower complexity tolerance would be good for the language.

  21. I think that’s an indirect consequence of C’s weak type ontology. It’s so difficult to build objects and frameworks in C that the “library binding” of C is just POSIX.

    I think that has more to do with the fact that Unix was developed in C, C was blessed as an implementation language in Unix’s early history, and hence the POSIX API is expressed in C terms and provided as a C library.

    If Rust achieves its goals, we will see the day when kernels written in C just aren’t taken seriously for production use. When that happens and you scrape Linux off and replace it with Redox, you will find a single standard Rust interface to the OS’s capabilities, just as Linux had a single standard C interface.

  22. > But yes, I think more feedback from people who just want to get stuff done and have a lower complexity tolerance would be good for the language.

    I’ll see what I can do… I’ve been lurking in the community since a little before the 1.0 release but I haven’t really had much free time to get experience yet and I am definitely very “just want to get stuff done” oriented.

  23. @Michael Aaron Murphy

    > D has serious issues with split implementations of the standard library, a completely disorganized community, and serious issues with packaging in Linux distributions.

    I won’t comment on the “disorganized community and serious issues with packaging” because they just feel too subjective, but I must point out that the split implementation of the standard library has litteraly been fixed for years now. It might be time to update your views on the language.

  24. Jeff Read wrote: “When that happens and you scrape Linux off and replace it with Redox”.

    Redox is just another experimental microkernel, which is fine if you want a toy OS to play with, but it’s never going to replace a general purpose kernel like Linux, and this has nothing to do with Rust success.

    Moreover, it does not really matter in what language the kernel is written if you write an user space program. If the goal was to wrap all syscalls in Rust, there is already a crate for that. But this API is never going to be a part of the standard library, because Rust wants to be portable across a wide range of platforms. So the goal is to build good abstractions, which can work efficiently on different systems.

  25. @TomDeGisis Jeff Read almost nailed it; what Jeff Read described won’t happen in our life times, because good enough is the enemy of perfect. C is good enough; POSIX is good enough; Unix/Linux/BSD is good enough. Plan9 and BeOS couldn’t break through that barrier, why would Redox/Rust?

  26. Mycroft Jones: were/are Plan9 and BeOS “10 times” “better” than UNIX™/Linux/BSD? Plus it wasn’t until 2000, after AT&T had shifted its focus to Inferno, that Plan9 was released under an open source license according to Wikipedia, and from 2002 to early 2014 it was the GPL incompatible and obscure Lucent Public License, then it was GPLed. The article even quotes our host on the lack of enough extra merit:

    [I]t looks like Plan 9 failed simply because it fell short of being a compelling enough improvement on Unix to displace its ancestor. Compared to Plan 9, Unix creaks and clanks and has obvious rust spots, but it gets the job done well enough to hold its position. There is a lesson here for ambitious system architects: the most dangerous enemy of a better solution is an existing codebase that is just good enough.

    Again using Wikipedia, except for a few components BeOS was tied to the fate of its owners and never open sourced, and as of now there’s one clone still under development and it’s not exited Alpha state. Despite e.g. hearing good things about its multi-threaded GUI, I don’t see how it ever got a real chance to supplant the existing incumbents.

    I’m hoping that we’ll have more people getting concerned about security and reliability that something that’s 10 times or better in those can some day become a viable competitor to the incumbents, and I certainly hope it’ll be in my lifetime.

  27. ESR:

    > The Rust community looks, from the outside, a bit ingrown and subject to a kind of “can you top this (complexity)” syndrome. I don’t mean this observation in a hostile way; if I were eyeball-deep in the development of a young language I would probably behave the same way – and arguably I did, back in Python’s earlier days when I was briefly on that devteam.

    Complexity is even more insidious than that, I’m sad to say. It’s not enough to avoid being excessively clever. Even if you restrain yourself, complexity will still creep on you anyway. You need to actively fight it whenever your can, and periodically review your “complexity budget.”

    Rust has already, in my opinion as a production user, overdrawn the complexity budget in one place. This happened accidentally, very early on. Specifically, it’s the interaction between several Rust features: (1) The auto-deref features provided by Deref. (2) The split between “owned” types like String and reference/slice types like &str. (3) The unification-based type inference that allows you to write let i: u32 = "10".parse()? and automatically choose the right version of parse. (4) The Into trait (and related traits), which allows you to write functions that take arguments of type Into<String>, which essentially means, “Oh, just give me any type that advertises conversion to a String.”

    Any one of these features individually makes Rust much nicer to program in. But if several of them all gang up in the same piece of code, a novice Rust user will get burnt. If I’m reading between the lines correctly, this might actually be what bit you with bind. The bind function takes an argument of type ToSocketAddrs, which is basically one of those Into-like traits which allow you to pass in several different ways of representing a socket address. It’s cute and clever (yuck), and it works most of the time. But if it gets combined with (1-3) above, it’s going to generate crappy error messages. The fix is relatively simple for an experienced Rust programmer, if annoying: Add explicit type declarations all over the place. But at work, my guideline is “just don’t use Into-style conversion traits in your APIs unless you have a damn good reason. They don’t actually improve API ergonomics.”

    If this is the only place where Rust exceeds it’s complexity budget, well, users will just learn this one bad interaction and go about their lives. C++ has dozens of warts worse than this. But any further expansions of Rust in this area need to be handled very carefully, and production Rust users need to participate in the RFC process.

    But let’s dig deeper.

    There are two libraries in the Rust space which worry me: Diesel and Tokio. Diesel looks like an ORM, but it’s really not—it’s just a typed version of the relational algebra which can dump output into Rust data types. It results in really concise and efficient code once it’s working. But the error messages are just horrendous (though not yet in modern C++ territory, though that’s nothing to be proud of). Diesel has chosen to push Rust’s type system to its limits in the name of speed and expressiveness. I’m not sure it’s worth it. We had a long debate at work and I paired on Diesel code with one of our other Ruby backend guys, and he said the tradeoffs with Diesel’s error messages were worth it. I’m not 100% sold.

    Where I’m more concerned is tokio. As everybody has told you, tokio is central to the Rust async I/O story, and lots of popular crates are moving to tokio-based backends (though most will still export synchronous APIs). And from what I’m hearing, tokio is currently generating bad error messages for some common use cases. In my opinion, this needs to be fixed—and the core team is discussing pushing up a language feature that’s been in the RFC process for a while now, which will hopefully make the error messages much clearer.

    Still, I’m holding off on tokio-based async I/O for at least 6 months in production code, to see how this all plays out.

  28. C is good enough; POSIX is good enough; Unix/Linux/BSD is good enough. Plan9 and BeOS couldn’t break through that barrier, why would Redox/Rust?

    The long, long list of CVEs for buffer overflows, stack smashes, use-after-free vulnerabilities, etc. put the lie to the notion that C is “good enough”. We’ve used it for so long largely for hysterical raisins, and because for a long time it and C++ were the only widely available languages that made it easy to do certain types of system programming. But C trusts the programmer too much to be conscientious, and on a large enough code base it’s easy to fall into dangerous traps.

    That all changes with Rust. In Rust, you can still do all the dangerous stuff C can do, but it’s safe by default; unsafe code can be written but must be flagged as such (and should be used sparingly). You have to tell the compiler “I know what I’m doing here, and I take full responsibility for it”. Anything outside blocks of code flagged unsafe will be checked for the kind of commonplace memory errors that make C so buggy and leaky.

    And look — we’re living in the age of always-on connectivity, Internet of Things, etc. So security is much more important because the stakes are so much higher now than they were in the past. If Rust gets you all the speed and flexibility of C plus additional security, then it will be a win. And eventually, any large code base still written in C should be treated with suspicion, and not trusted in security-sensitive applications.

  29. @Jeff Read
    What you described is probably true for a lot of user-space code, but it is not true for the kernel. If you look at Linux kernel CVEs, you will see that they have nothing to do buffer overflows, stack smashes, use-after-free vulnerabilities. They are more likely to be logical errors, which can be exploited under some condition. Even when you hear a data race condition found in the kernel, it is unlikely to be caused by a missing lock. In most cases, it is a logical error of some kind. For example, here is the patch for a recently discovered data race in Linux (CVE-2016-5195): https://lkml.org/lkml/2016/10/19/860. As you see, it was not about locking, but about proper checking different flags.

    Moreover, Linux developers actively use a static analyzer to catch the most common mistakes in their code. So Rust can’t help there much. Also, it is completely unrealistic to write a general purpose kernel without unsafe code. Even if you look at the Rust standard library, you will find a lot of unsafe code there (e.g. all collections). When you work with hardware at low-level, you have to do a lot of things that can be potentially unsafe. But let’s suppose that it was possible to write the whole kernel in a safe programming language, would it eliminate security exploits? Of course, not.

    You may look at long list of CVEs for many web-frameworks written in safe languages (such as PHP, Python, Ruby). By its nature, the kernel works at a higher privilege than the user-code, which means many logical errors in the kernel can be exploited to do something that the user should not be allowed to do. So writing a safe kernel is a far more complex task then writing a kernel in a safe language. First of all, you have to have a formal definition of what you mean by “safe” and then decide how you are going to prove that. It is very tricky even for a very small toy kernel.

    1. >And then there’s – god – RCU. I’ve read so much about it and only understand how it works for short periods.

      Likewise, so don’t feel bad. I think the knowledge might stick if I actually used it for something.

  30. Dmitry,

    I will have to concede that you can get about 50% of the way to Rust — maybe a little more — with a good static analyzer for C that screams at you if it detects an attempt to alias pointers, access outside of array bounds, or access freed memory, except in functions which are marked with magical comments or perhaps something like ‘#pragma unsafe’.

    Again, for hysterical raisins — largely because of RMS’s obstructionism w.r.t. letting other software connect to GCC’s front or back end — the tooling for that took too long to emerge in the open source world — and that’s a crying shame.

    And there’s only so much a static analyzer can catch; it cannot guarantee total memory safety. You can’t determine in the general case whether a given program will access out-of-bounds memory from the source alone. To guarantee memory safety, you have to check that array accesses are within bounds at runtime, and abort or throw on out-of-bounds. This is one of the things that people bitched about with respect to Ada in the early days; they didn’t like incurring the performance hit of a runtime check on every array access. Of course, nowadays Ada compilers will automatically remove checking code if it can statically determine the array accesses are safe; and they also provide knobs to let you manually turn checking off.

    Of course Rust has that same problem; afaict Rust actually does no bounds checking at compile time, but does do the check at runtime.

  31. Jeff Read:

    > Of course Rust has that same problem; afaict Rust actually does no bounds checking at compile time, but does do the check at runtime.

    There are two interesting wrinkles to keep in mind here:

    1. Rust emits runtime bounds check for array access (unless you use get_unchecked inside an unsafe block), but I believe that LLVM can eliminate them at least some of them time, using standard optimizations.

    2. Idiomatic Rust discourages indexed loops in favor of iterators. Rust iterators usually compile to very efficient code (if you use --release like you should), and iterators do not require bounds checks. I’ve only written a handful of indexed loops in Rust in the last 6 months.

    A more interesting issue is that Rust only checks for integer overflow in debug builds by default. Integer overflow errors are less serious in Rust (because array access is checked), but they’re still an issue.

    If you actually want wrapping integer overflow in Rust, you’re supposed to use Wrapping or a similar API.

  32. > A more interesting issue is that Rust only checks for integer overflow in debug builds by default. Integer overflow errors are less serious in Rust (because array access is checked), but they’re still an issue.

    My guess would be that in non-debug builds you can get the same mysterious so-called optimizer bugs that you do in modern C/C++ implementations if your code has an integer overflow. I wouldn’t call that less serious, and I don’t see what it has to do with array access at all.

  33. @Jeff Read,
    A good static analyzer specifically written for needs of kernel developers can catch a lot of things. For example, recently there was a patch to check the _assembler_ code to ensure that it properly aligns the stack before call. More importantly, it enforces many other kernel-specific policies on which developers rely. Can Rust do that? I doubt.

    Rust code is only safe as long as you don’t use “unsafe”, which is impractical if you work at low-level. Kernel developers use assembler in many places and many other things that Rust considers unsafe, though they are perfectly safe. Currently, you cannot write something simple as HashMap in safe Rust code. Or maybe you can, but it will be vastly inefficient, and that is the reason why the Rust library uses unsafe code in many places. So if you want to be _practical_, you cannot eliminate all unsafe code with Rust. Thus there will NOT be memory safety in any case. You have to make certain trade-offs to have something practical.

    Moreover, in practice, memory safety bugs are responsible for minority exploits. So even if the whole kernel is written only in a safe language, it does not eliminate most security vulnerabilities. So it makes no practical sense to make huge investments in re-writing everything while most security problems will remain anyway.

    The fact is that Rust has been developed with different priorities than those that kernel developers have. It’s written to replace C/C++ in such programs as a web-browser, where you have to deal a lot of complex formats, which often results in subtle memory bugs. The kernel avoids dealing with complex formats by delegating complex tasks to the user space. So most kernel complexity comes from the fact that it has to be fast and scalable. Therefore it utilizes a lot of specialized collections, algorithms, and different tricks to make things fast and efficient. I don’t see how Rust can help here, because most of those things cannot be written in safe Rust now. (Perhaps, it is possible to extend the type system to verify some of those things formally, but it will bring immensely amount of complexity, which will make the language nearly unusable for everyone but a few experts in the field.)

    Of course, you can use Rust to a simple “toy OS” for some real-time embedded device, and it could be even some practical value in doing so, because often such embedded devices are used for more than a decade and they never receive any software update. However, this is a completely different story than replacing a general purpose kernel like Linux.

  34. @Dimitry

    > A good static analyzer specifically written for needs of kernel developers can catch a lot of things. For example, recently there was a patch to check the _assembler_ code to ensure that it properly aligns the stack before call. More importantly, it enforces many other kernel-specific policies on which developers rely. Can Rust do that? I doubt.

    Rust supports custom lints through compiler plugins. See Clippy as an example, which is usually packed inside every serious Rust development environment.

    https://github.com/Manishearth/rust-clippy

    > Rust code is only safe as long as you don’t use “unsafe”, which is impractical if you work at low-level.

    Time and time again, this has been stated to be false. The unsafe keyword does not disable all of Rust’s safety features. You still have the borrow checker, for example, making sure that you aren’t doing anything potentially dangerous with your memory.

    Working at an unsafe level is never needed unless you are writing a wrapper around a C library with the FFI or you are doing some potentially dangerous optimizations when designing some efficient data structures. The great thing about working unsafely in Rust is that the safety nets make working with extreme optimizations much easier.

    > Kernel developers use assembler in many places and many other things that Rust considers unsafe, though they are perfectly safe.

    The Redox kernel / OS is written almost entirely without unsafe.

    > Currently, you cannot write something simple as HashMap in safe Rust code. Or maybe you can, but it will be vastly inefficient, and that is the reason why the Rust library uses unsafe code in many places. So if you want to be _practical_, you cannot eliminate all unsafe code with Rust. Thus there will NOT be memory safety in any case. You have to make certain trade-offs to have something practical.

    Why would you need to? Being optimized with unsafe features doesn’t make it unsafe. The exposed interfaces to these data structures are perfectly memory safe. It is merely telling the compiler that although it is dangerous, the designers of the data structures know what they are doing and what they are doing is safe.

    > Moreover, in practice, memory safety bugs are responsible for minority exploits. So even if the whole kernel is written only in a safe language, it does not eliminate most security vulnerabilities. So it makes no practical sense to make huge investments in re-writing everything while most security problems will remain anyway.

    Rust isn’t about memory safety in itself though. There is more to Rust’s safety than mere memory violations.

    > The fact is that Rust has been developed with different priorities than those that kernel developers have.

    I’ll stop you right there. There are kernel developers in the Rust team. Anything that makes Rust a better language for writing kernels is always welcome, and many changes have been made in the past to make Rust better at writing kernels and OSs.

    > I don’t see how Rust can help here, because most of those things cannot be written in safe Rust now.

    There’s no reason to write efficient data structures that are already safe exclusively in safe Rust. Swapping pointers and transmuting values, for example, is a valid strategy for fast optimizations. Much of the Rust community is hellbent on writing the most efficient solutions possible, and this means combining the safety features of Rust with unsafe optimizations. It would be stupid to avoid using the unsafe keyword entirely just because you don’t like the keyword.

  35. @Random832

    > My guess would be that in non-debug builds you can get the same mysterious so-called optimizer bugs that you do in modern C/C++ implementations if your code has an integer overflow. I wouldn’t call that less serious, […]

    No, the Rust designers specifically did not want to fall into this trap.

    As previously mentioned, in debug builds, integer overflows cause a runtime panic.

    Meanwhile, in release builds, integer overflows have a *well-defined* (wrapping as two’s complement) behavior. This means Rust “misses out” on some hypothetical optimizations when compared with C’s choice to leave signed integer overflow undefined. But it also means that even if you do allow overflow, you will not get burned by compiler optimizations.

    For further reading, if you are interested:

    http://huonw.github.io/blog/2016/04/myths-and-legends-about-integer-overflow-in-rust/

    https://github.com/rust-lang/rfcs/pull/560

  36. > (This is, in fact, why Benevolent Dictator For Life is such a common governance model in our otherwise rather anarchic community. Experience has shown that curation by one person with a clear design vision generally beats attempts to design by committee.)

    Hmm… this reminds me of a comment on a certain Arstechnica article from three years ago…

    > (theotherjim) It’s been almost 30 years since a colleague suggested to me that all the keyboard designers in the world – except one, any one – should be shot.

    (Funny from link diving: the tacitly nominated Christopher Latham Sholes, who designed QWERTY, …um… “was instrumental in the successful movement to abolish capital punishment in Wisconsin” according to Wikipedia.)

    Then I scanned the comments to see how off-topic this one, in a bunch of comments I haven’t been following, would be. It’s pretty bad, but I found a neat little coincidence:

    > (Tim) But ultimately I think it’s a curse of early adoption. No one truly knows the future, and Rust hasn’t been around for 30 years yet.

    For the final word, don’t confuse “anomy” with “anarchy”. AFAIK, the former word doesn’t exist, but I made it up following the etymology of other similar words (including “etymology”, lol), like “theonomy” and “autonomy” which are God-law and self-law. “a-” goes “no-” for words such as “atheist” (no-God) and “anomia” (no-law but, unlike “anomy”, is an actual Greek word.) Anomy is no law, which is bad. But anarchy is no government, which can be good. I think BDFL is a side-effect of the best aspects of anarchy clashing with the least bad aspects of anomy and coming up with an anarchy-without-anomy solution.

    (Sorry for my tardiness, but I had to watch a few 2015 Bahnsen Conference lectures to realize there was something in this post that I GAF enough to comment on.)

  37. > Anomy is no law, which is bad. But anarchy is no government, which can be good

    Yet anarchy cannot have law, because law is enacted and enforced by an entity. A law that is not enforced is not a law. Anarchy creates a power vacuum, and power vacuums are quick to find their next leader. Basically, you end up with mafia law, where your government is your local mafia. These mafia gangs eventually converge as they consume each other, becoming more and more powerful as they expand.

  38. > Yet anarchy cannot have law, because law is enacted and enforced by an entity

    While we can debate its stability and the difficulty of achieving it, one could argue that anarchy without anomy can be approached by having a situation where shared community mores operate in lieu of overarching governance and, while there’s no central authority, anyone who refuses to assimilate will quickly find them selves independently banned from dozens of chat rooms/forums/etc. by dozens of independently-operating moderators, leaving them invisible to everyone except those who allow unsolicited private messages and haven’t yet added them to their killfiles.

  39. @Terry

    > For the final word, don’t confuse “anomy” with “anarchy”. AFAIK, the former word doesn’t exist, but I made it up

    You’ve independently invented a word that already exists. It’s used by sociologists, usually under a different spelling, to mean something quite different (https://en.wikipedia.org/wiki/Anomie).

  40. Yet anarchy cannot have law, because law is enacted and enforced by an entity.

    Oh, good grief, you’re going to come and drop a line like that here? Which entity enacted and enforces international law? (Hint: By “enforces”, I mean enforces. Like with guns, not sternly worded letters.)

  41. @Michael Aaron Murphy

    > you are doing some potentially dangerous optimizations when designing some efficient data structures.

    Well, “efficient data structures” can describe a lot of what the kernel does, because it is one of its key features. As I said before, the kernel can avoid dealing with many complex formats by delegating this task to the user-space, but it has to be scalable and efficient, so this is one main sources of complexity.

    > The Redox kernel / OS is written almost entirely without unsafe.

    I have never questioned that you can write a “toy OS” mostly in safe Rust. But if you want to replace Linux, this is a completely different task. From engineering point of view, the difference may be as large as the difference between building a bridge that is 50ft long and one of the longest bridge in the world — they are completely different engineering challenges.

    > There are kernel developers in the Rust team.

    I am not sure whom you mean here. Certainly, if you want to build a microkernel for some embedded device, you can use Rust for that (and it helps to avoid many bugs that usually such devices have). However, right now, I do not see how Rust can help when it comes to a big project like the Linux kernel. Of course, Rust is still a very young language, so this may change…

    > Much of the Rust community is hellbent on writing the most efficient solutions possible

    I think the same thing can be said C++ community, and both communities have aspiration for having a more high-level language than C. What can be wrong with having “zero-cost abstractions”? They make your code more readable and also they allow to hide some low-level unsafe code behind high-level safe abstractions. Unfortunately, they may have some practical cost:
    1. Compilation time is significantly larger than what you have with C.
    2. It may be difficult to reason about efficiency of some code by looking at some of those abstractions. (It matters a lot when you have strict constraints, such as only 8Kb stack space).
    3. What is theoretical should be zero-cost may not be so in practice. (Again, it is difficult to spot such problems, and then it is unclear how to fix.).
    4. Regression in some optimization can be difficult to diagnose and deal with, especially if you want to support different versions of the compiler.

    So, the fact that C can be much more verbose is not always minus, it can be also plus for something low-level like the kernel, where explicitness and predictability matter a lot.

    I clearly remember how 15 years ago C++ fans were absolutely convinced that C++ would replace C everywhere. To their mind, C++ was better in every respect than C. Now, I must admit that Rust is clearly a better language than C++ when it comes to writing a kernel. Still, it is far from certain that Rust can win the hearts and minds of kernel developers.

  42. International law is not really law. It’s really a hybrid of law and custom. People want it to be law, so we call it law in the hope it will become so.

    Yours,
    Tom

  43. Eric Kidd’s survey of the ecosystems for Ruby on Rails and JavaScript stirs me to think it might be a worthwhile undertaking to map and document the ecosystems for as many programming languages as we have the energy and substantiated sources.

    In practice, I’d expect such a meta-resource to be highly organic – any given language’s adherents would talk about what’s in the main distro, where people go to get it, who seems to be driving most of the main development, the IDE(s) of choice, the usual sites, irc channels, et al. to go with bugs and questions, what the main applications and “killer apps” are, etc. Possibly with a rating system to indicate stability, code quality, and traffic (or just tie it in with Alexa or something if it’s a website).

    The result would quickly guide newcomers to the places veterans and intermediates expect to go for corelibs, popular add-ons, and support.

  44. A puzzle:

    If moz://a had spent as much money on appointing (read: hiring) Rust stewardship as they did on choosing a new logo, would the maturity problems still exist?

    I get the feeling that the Rust community is employing what Tom Schelling called precommitment: getting as many people onboard with Rust while it was still immature, so that it seemed more like a fait accompli to the wider community and there would be more emotional investment in making it good than there would be in considering a competitor.

    And Eric, you can’t expect anything to last ten years in open source anymore. Somebody somewhere is going to want to burn your work to the ground and start completely afresh, and odds are they’ll have an in at Red Hat. My money is on ntpsec being uprooted and replaced with systemd-timed in all major distros within six years.

  45. @Tom DeGisi:

    International law is not really law. It’s really a hybrid of law and custom.

    As persuasive arguments go, this deserves a grade of “F for (non-)effort”. Until you specify what critical aspects of “law-ness” are lost in the described hybridization, how can I determine if I agree or disagree with your thesis that “international law is not really law”?

  46. Eric: a small request. Recently the issue of Jallikattu has become a big issue here in my native place.

    Jallikattu is the age old traditional bull-taming sport of the farming community of South India which happens once a year in the harvest season, particularly Tamil Nadu. The bulls that form part of this tradition are native breed and help in propagating the species. Now PETA India, by its activism has got a legal ban on the tradition which will lead to the loss of livelihood of the bull rearers and ultimately lead to the slaughter of the bulls. Recently huge protests have erupted all over our state to revive this tradition which is vital in keeping these breeds alive as these stud bulls keep the gene pool strong. You can read more about the issue on Wikipedia. https://en.wikipedia.org/wiki/Jallikattu

    Please consider if you can pitch in with your views. I was hoping whether you can share some more information about the activities of this subversive organization PETA in your part of the world which I can share on the social media to discredit their brand of activism. The whole “Cruelty” aspect to the bulls is over-stated and utterly false. The game involves more danger to the humans than the bulls and the youth who indulge in this game are basically unarmed.

  47. Here’s something that should help your Go programming: HelloGopher, a Makefile that lets you build Go projects with conventional ‘make’ commands — no GOPATH necessary, even!

    The developers appear to not recommend this solution if you intend to become a serious Go developer. They recommend you suck it up and learn to use GOPATH and the Go build tools. Still, this is a neat onboarding option if you are used to Unix.

  48. Which entity enacted and enforces international law?

    International law is not really law.

    War crimes tribunals, such as the International Criminal Tribunal for the Former Yugoslavia, do in fact enforce a specific portion of international law — namely, the Law of War (that is, they punish people for violations of it). The military commissions at Guantanamo Bay are also designed to enforce the Laws of War, which are a part of international law. The Nuremberg and Far East tribunals did the same. People do get punished for violating it; it’s law.

    (Not really relevant to the question of whether an anarchic society — such as saga Iceland — can have laws.)

  49. (Not really relevant to the question of whether an anarchic society — such as saga Iceland — can have laws.)

    Saga Iceland had laws; I believe the Althing, the legislature of the Icelandic Commonwealth, is the oldest legislature still in operation today. (Though the form of government has changed since the time of the sagas.)

    Under the Commonwealth, enforcement of the laws was delegated to the various gothorth, or chieftainships. A chieftainship was property; the title of chieftain could be inherited, bought, or sold. Each chieftain could choose to enforce or not enforce the laws, be as strict or as lenient as he pleased. Chieftainships were non-geographical; any person could change their allegiance any time they pleased.

    It was, all in all, a bit like the libertarian fantasy of multiple competing police departments operating in the same region. It fell apart, as these things tend to do; but the fact that there was no single body enforcing all the laws didn’t mean that there were no laws in saga Iceland.

  50. jpq327, that’s very nice.

    Unfortunately, getting short GC pause times necessarily incurs significant memory overhead. Hertz and Berger (2005) concluded that the same application requires five times as much RAM to run with a GC as to run with explicit memory management.

    It’s 2017. Tracing GCs are close to 50 years old, and in that time we’ve developed better methods of automatic memory management — value semantics in C++ and Rust, and ARC in Objective-C and Swift, for example — that get you 90% (or more) of the way to where a tracing GC gets you with close to zero runtime or memory cost.

    Tracing GCs are, in a word, obsolete. It was a foolish mistake for a “systems language” like Go to adopt one.

    Again, cowboy up, and learn some goddamn C++ or Rust. Don’t complain that it’s too hard. Programming is hard, Highness; anyone who tells you otherwise is selling something.

  51. Hertz and Berger actually concluded that 5x the RAM is neede to run the same application with equivalent performance to explicit memory management.

    You can run anything you like with a GC — as long as you don’t mind it being not only slow, but nondeterministic in its runtime.

  52. Eric, you might want to read “The Lisp Curse”, whose thesis is that in highly expressive languages, what would otherwise be technical problems become social problems. In particular he points out the dangers of an ever-expanding set of broken, undocumented, half-finished libraries that did exactly what the original author needed them to do, and little else. A few people, notably Olin Shivers (and in a subsidiary way, me) are fighting this trend, but it isn’t easy.

    The extreme case is Stalin, which does whole-program interprocedural analysis before translating to C, and is (depending on the benchmark) either the fastest or the second-fastest Scheme there is, based on execution speed. But socially it is utterly autistic. For example, if you type “stalin file.scm” to compile a file of Scheme code, it tells you that running Stalin without the -i option is not yet supported. Now a one-line patch to always have the -i option turned on would be trivial. But having made the patch, what would I do with it? There is no Stalin mailing list, or forum, or IRC channel. The code is monolithic and completely comment-free; the documentation is a single README file. The original developer is no longer working on it and won’t accept patches. The users (who are people for whom Gambit is not fast enough) have made their peace with Stalin or made changes which they have zero interest in sharing. They are in effect a bunch of Solarians from The Naked Sun and the more extreme version in Foundation and Earth.

    Jeff Read: In 1958, when Eric and I were babies (I don’t know how old you are), it was absolutely taken for granted that every high-level language had to have static typing and static memory lifetime. Lisp has survived the 58 years since then without either, and performance has done nothing but improve. (Don’t believe the Lisp-is-inherently-slow myth.) And what’s more, it has a whole host of dynamically typed GC-ful epigones. (Lisp programmers are smug weenies, because what we had in the 1970s is almost available in other languages today, and people get tired of hearing us say so.)

    1. >Eric, you might want to read “The Lisp Curse”

      Wow. That’s good analysis. And, although the author doesn’t realize it, it implies a way to beat the curse.

      Damn, now I’m tempted. I know how to solve this problem. But it would take a multi-year commitment and some serious social engineering. I need to finish NTPsec first.

    1. >Zig is an exciting competitor : http://ziglang.org/

      At the project scale Go is designed to address, manual memory management won’t cut it any more. And Zig doesn’t have Rust’s story about provable type safety either. So, no it isn’t.

Leave a comment

Your email address will not be published. Required fields are marked *