Rust severely disappoints me

I wanted to like Rust. I really did. I’ve been investigating it for months, from the outside, as a C replacement with stronger correctness guarantees that we could use for NTPsec.

I finally cleared my queue enough that I could spend a week learning Rust. I was evaluating it in contrast with Go, which I learned in order to evaluate as a C replacement a couple of weeks back.

My chosen learning project in Rust was to write a simple IRC server. As a service daemon with no real-time requirements written around a state machine of the kind I can code practically in my sleep, I thought this would make a good warmup for NTP.

In practice, I found Rust painful to the point of unusability. The learning curve was far worse than I expected; it took me those four days of struggling with inadequate documentation to write 67 lines of wrapper code for the server.

Even things that should be dirt-simple, like string concatenation, are unreasonably difficult. The language demands a huge amount of fussy, obscure ritual before you can get anything done.

The contrast with Go is extreme. By four days in of exploring Go I had mastered most of the language, had a working program and tests, and was adding features to taste.

Then I found out that a feature absolutely critical for writing network servers is plain missing from Rust. Contemplate this bug report: Is there some API like “select/poll/epoll_wait”? and get a load of this answer:

We do not currently have an epoll/select abstraction. The current answer is “spawn a task per socket”.

Upon further investigation I found that there are no proposals to actually fix this problem in the core language. The comments acknowledge that there are a welter of half-solutions in third-party crates but describe no consensus about which to adopt.

Not only that, but it seems the CSP implementation in Rust – the most tractable concurrency primitive – has limits that make it unusable for NTPsec’s purposes (only selection over a static set of channels is possible) and there is some danger that it might be removed entirely!

I have discovered that Rust is not, or not yet, a language suitable for long-term infrastructure work. It suffers from a damning combination of high barriers to entry and technical deficiency. What is worse is that Rust’s community seems to be unable to fix or even clearly grasp these problems.

I think one problem here is the absence of a strong BDFL providing tasteful design direction and setting priorities. The Rust community appears to have elected to use the decentralized nature of their crate system (which undeniably has some very nice technical properties) to execute a swarm attack on the design space around the language. Which is wonderful in theory but seems to be failing in practice.

UPDATE: This post attracted two kinds of pro-Rust response. One was stupid flamage from zealots. The other was thoughtful commentary from a few people close to the core of the Rust community. The latter group has convinced me that there is considerable awareness of the problems I ran into; a couple even agreed, after analysis, that Rust is at present a poor fit for NTPsec’s requirements. This gives me hope that the Rust of five years from now may become the mature and effective replacement for C that it is not yet.

224 comments

  1. “Even things that should be dirt-simple, like string concatenation, are unreasonably difficult.”

    Could you elaborate further.

    Hope you’ll write more later on your view of Go.

  2. Eric, a year ago I came to similar conclusions. Watching the chaos on the mailing lists and IRC channel, I saw that it was a combination of arrogance and SJWism from the Netscape/Mozilla crowd, in combination with a huge influx from the Ruby on Rails crowd… Ruby on Rails may have been great for setting up a website in 5 minutes, but their mentality toward security, performance, and bugs made my hair stand on end for many years. And that same chaotic mentality followed them to Rust. It is almost like Rust is their attempt to fix the problems they had with Ruby on Rails, but they don’t quite have what it takes.

    This is not a slam on Ruby as a language, per se. Although even that has problems. Mainly the whole Ruby on Rails scene. Magic layered on magic, cavalier attitudes towards crashes and security, etc. And also beautiful looking user interfaces that your heart aches for them to work well.

  3. To an extent, I have learned I can reliably evaluate a technology based on WHO developed it, and WHAT their development style has been like in the past. Lennart Poettering for instance. PulseAudio begat Systemd. Ruby on Rails + Netscape begat Rust. Plan 9 begat Go. NeWS begat Java.

    Go is a nice language, but like Plan9, it is a bit too self-contained, almost what you need to interoperate, but not 100% there. Like a beautiful jewel that stands best on its own.

    1. >This is not really difficult:

      Right. Then you run into the problem that the socket bind primitive won’t actually take a String for its address. And that there seem to be two different kinds of String things, one from std and one from collections, and who knows what function that takes something stringlike will take which?

      >So you have three possibilities.

      Yes, that was a warning flare all by itself. Which of those three will be maintained in ten years?

      1. Tokio uses futures-rs; as far as I know it has always used futures-rs. So futures-rs will be maintained if either is maintained.

        I know you’re unlikely to see this response since this post is so old, but I hope you will take another look at Rust when the Rust 2018 release occurs later this year; the team plans to incorporate async/await support directly into the language.

  4. You should know better than to just throw a “I didn’t understood the concept and didn’t want to understand new one” as an argument to say “this language sucks”.

    Go is pretty close to C. Rust in comparison is very far from it and asks you to read and understand a lot of new concepts, and I don’t think you tried a lot.

    If you had read the official book you would have found things like “how to do a string concatenation” (https://doc.rust-lang.org/stable/book/strings.html#concatenation) which can be done with a simple add operation.

    Also why are you looking for an epoll in the standard library ? Is it in the standard library of C or C++ ? No, because it’s a Linux API. A bit of googling would have showed you the epoll crate (https://github.com/nathansizemore/epoll) or even better the mio crate (https://github.com/carllerche/mio). There’s even a whole tutorial on writting a TCP server: https://github.com/carllerche/mio/blob/getting-started/doc/getting-started.md

    I think Go is a fine language and if you prefer it, stick with it. But if you want something without garbage collector and with a much more complex but more powerful type system (not everyone or every project need those), then rust is arguably better.

    1. >If you had read the official book

      I read the official book twice. I liked the language when all I knew about it was the book.

      Then I tried to actually use it…

  5. If you’re looking for a C replacement, take a look at Nim: http://nim-lang.org/. It is tastefully designed by a BDFL, has a modern python flavored syntax, excellent performance (optimized C level and definitely better than Go), good documentation, modern package management, nice C lib FFI, optional and/or on-demand GC, responsive core team, and a good cross-platform story. The existing concurrency story is a reasonable (http://forum.nim-lang.org/t/714) with a model for deadlock avoidance at compiletime being actively worked: http://nim-lang.org/blog/concurrency.html

  6. To an extent, I have learned I can reliably evaluate a technology based on WHO developed it, and WHAT their development style has been like in the past.

    Same here. I decided to keep my distance from Rust the minute I heard Steve Klabnik was heavily involved with the language.

  7. > there are no proposals to actually fix this problem in the core language

    1. C doesn’t have any either in the core language, technically.
    2. As I understand it, Rust uses the C ABI. If you want select/poll, you know where to find them. It looks like the discussions you linked are focused on trying to implement something better, which is harder than it sounds because it has to handle things that aren’t file descriptors.

    And that best solution might well end up looking like “spawn a task per socket” if tasks aren’t necessarily OS threads. Do you want an API that looks like poll, or do you want to be able to perform as well as poll without having to write to an API that looks like poll? And if the former, why?

    1. >And that best solution might well end up looking like “spawn a task per socket” if tasks aren’t necessarily OS threads. Do you want an API that looks like poll, or do you want to be able to perform as well as poll without having to write to an API that looks like poll? And if the former, why?

      IRC and NTP are alike – and unusual – in that the most natural way to write them is as a single-threaded server. (I can’t easily imagine how to write NTP as a thread-per-socket server at all.) Which means, yes, I want an API that looks exactly like poll/select, so all the code for handling client connections shares the same memory space.

  8. I just read up on Rust Strings here.

    https://doc.rust-lang.org/1.5.0/book/strings.html

    You got to be kidding me. Got save programmers from religious devotion to an idea.

    There is no reason to do the malarkey they do with strings. If you are getting a way from C, you should do what it takes to make it as simple as

    let hello = “Hello”;
    let world = “world!”;

    let hello_world = hello + ” ” + world;

    why anybody thinks something like “Hello “.to_string(); is good idea escapes me.

    1. You are getting confused on naming of “String”. In Java lang equate “hello”.to_string() creating a “StringBuffer” or “StringBuilder” — while “hello” is just a a string.

  9. Same here. I decided to keep my distance from Rust the minute I heard Steve Klabnik was heavily involved with the language.

    Who was very specifically a Ruby on Rails “web programmer” of little note outside the community per his own reported history, who Mozilla hired to document a systems language, and to no surprise did that badly from every report I’ve heard by a systems programmer. He appears to have ended up being the public face of the language as well.

    Another example of the costs of SJW convergence.

  10. Robert, I don’t intend to be a Rust apologist, but what would you expect the operation of concatenation on strings to perform? The goal of C-level languages ought to be to avoid intermediate temporary allocations and that is what I suspect your suggested code amounts to. I am content to suffer additional machinery to prevent the need for intermediate results in memory.

    I am also surprised to see people, ostensibly grown men, use the term “SJW” without apparent irony in here. This whole attitude reminds me of the sort of cultural “othering” that produced the red scare.

    1. bruh, many of the people mccarthy accused of communism, people who gaslighted him as insane? Years later, it turned out those individuals were in fact communists *all along*.

      Rust is a cargo cult.

      1. That might be the most intellectually vapid response I have seen on any message board, technical or political.

  11. I am also surprised to see people, ostensibly grown men, use the term “SJW” without apparent irony in here.

    Oh, excuse me, I actually mean Social Justice Convergence.

  12. >>inadequate documentation to write 67 lines of wrapper code for the server. Even things that should be dirt-simple, like string concatenation, are unreasonably difficult.

    Could you please suggest issues with the standard library documentation? I’ve found the string library’s documentation fairly comprehensive the team would love suggestions on how to improve.

    https://doc.rust-lang.org/std/string/struct.String.html

    >>We do not currently have an epoll/select abstraction. The current answer is “spawn a task per socket”.

    There is no epoll abstraction in the core language as epoll is not offered on all platforms. The goal of the standard library is to be usable on all platforms.

    There are 3rd parties library to resolve these pain points. Epoll abstraction: https://docs.rs/mio/0.6.1/mio/

    >>There is some danger that it might be removed entirely

    What? Can you provide a source to this discussion? Because I’ve never seen it. Are you talking about this? https://github.com/rust-lang/rust/issues/27800 Where one person suggest this in passing and was shot down immediately almost 2 years ago?

    The MPSC channel library isn’t great (1 allocation per item inputted). But there are better implementations on crates.io A quick search returns several https://crates.io/search?q=queue

    Also MIO (linked earlier when discussion epoll) offers the same facility with it’s channels letting you monitor as many as you wish at once.

    The only feature to be removed from Rust’ standard library was scoped threads and that is because they were leaking and failing their guarantees they provided. Also they were never formally stabilized (only existed in non-stable builds).

    >>What is worse is that Rust’s community seems to be unable to fix or even clearly grasp these problems.

    Clearly the biggest issue is the so called *tribal elder* of FOSS either being totally ignorant, or totally obeisant to using FOSS libraries.

    >> The Rust community appears to have elected to use the decentralized nature of their crate system (which undeniably has some very nice technical properties) to execute a swarm attack on the design space around the language. Which is wonderful in theory but seems to be failing in practice.

    How so?

    You say there are no abstractions around items there are abstractions for.

    Did you only find out about crates.io after you abounded your project? Or did you never find it’s search feature? Is the only search you can use the github issue tracker?

    1. >Could you please suggest issues with the standard library documentation?

      Here’s a representative example of a point problem. It took me an unreasonably long time to get basic handling of command-line options working because the one example on the getopts page never shows how to get the contents of a string-valued option in the context of the option handling. Instead the magic option cookie gets passed whole to a function, which then unwraps it in a way that is nearly as unhelpful as possible to figuring out what to do in the normal case. (I did eventually figure it out.)

      The more general problem is that the existing documentation is great at conveying theory but terrible at telling what you need to do in practice. After reading the Rust Book twice (I do my homework) I was pumped – I felt like I understood what Rust was trying to do and liked a lot of it. The generalization of RAII offered by lifetimes and the borrow checker seemed like a very clever idea; I was ready.

      Then I tried to use the language in practice, and it was like constantly being stung by nettles. I went beck to the documentation over and over again, bashing my head against it repeatedly, and it just … didn’t … help. There’s some basic disconnect there that can’t tell you how to fix because I don’t comprehend it myself. I know language documentation doesn’t have to be that bad, because a few weeks ago I learned Go in less time and effort than it took me to give up Rust as a bad job.

      >There is no epoll abstraction in the core language as epoll is not offered on all platforms. The goal of the standard library is to be usable on all platforms.

      So, yeah, you just said RUST IS USELESS FOR WRITING SINGLE-THREADED SERVERS.

      I don’t care why you don’t have select/epoll. What I care about is being able to realize an architecture in which the handling for all my client instances shares state without having to go through inter-task IPC. I wouldn’t want to pay the complexity overhead for that, even if I we are willing to ignore the implied latency problems.

      If your answer is “we don’t do that”, then you’ve disqualified yourself from a large range of service architectures that Go and even fuddy-duddy archaic C handle nicely.

      >Where one person suggest this in passing and was shot down immediately almost 2 years ago?

      One person noted that someone else had also advocated removing it, and I saw no shootdown.

      That would have alarmed me less if I hadn’t been starting to figure out the Conway’s Law effect of your crate system on the language design. Your crate design is lovely in a lot of ways, but the blitheness of people who answer “what do I do about the absence of abstraction X in the language core?” with “oh, there are 23 crates for that” is actually quite frightening to someone trying to plan for ten-year timescales. I have to ask myself which crates will be maintained in ten years?

      Contrast this with Go, wherein I don’t have to wonder if any of the core abstractions will be gone in ten years exactly because they’re core abstractions. There’s a stability commitment there that Rust is not equipped to give me.

      >Did you only find out about crates.io after you abounded your project? Or did you never find it’s search feature? Is the only search you can use the github issue tracker?

      No to any of those. I used crates.io all right. Found caesar, which I was ready to drop in for TLS support once I had the IRC server prototyped with the standard library server framework.

      (Please don’t assume I’m stupid or inattentive or don’t do my research. That assumption has never cashed out well for anybody who made it.)

      Let me repeat: “There are 23 crates for that” is a horrible answer. I need a language with a sound core architecture, reasonably coextensive with a full ANSI binding of C, in which I can be reasonably confident that the features I need to rely on will still be there in a decade. Go gives me that because it’s run by ramrod-up-the-ass authoritarians who have no trouble being normative; I may not like all their decisions, but at least I know what the decisions are.

      Rust’s freewheeling throw-another-crate-at-it style (what I meant by “swarm attack”) is great for exploring the design space, but at some point you have to stop generating an inchoate cloud of options and actually make decisions. And those decisions have to be reflected in your core documentation. Well, that is, if you want the language to move beyond “toy” to a real production tool.

      All these abstractions can be focused back down to practicality with a really simple question, to whit: where’s my goddamned POSIX select(2) primitive?

      1. You are comparing apples and oranges.

        ISO C doesn’t specify epoll(), select(), kqueue() or anything of the kind. POSIX does for select, and Linux and *BSD do for the other two, all with C bindings, but a conforming C implementation doesn’t have to implement POSIX or any de facto standards of Linux or BSD. The situation is the same for C++. Criticizing Rust because it doesn’t implement something outside of its purview while holding up C as a model when C *also* doesn’t specify the exact same thing is inconsistent.

        >>There is no epoll abstraction in the core language as epoll is not
        >>offered on all platforms. The goal of the standard library is to be
        >>usable on all platforms.
        >
        > So, yeah, you just said RUST IS USELESS FOR WRITING
        > SINGLE-THREADED SERVERS.

        No, that wasn’t at all what he said. What he said is that the Rust standard library implements a subset of functionality that is intended to be common across platforms. You can build additional abstractions on top of that for other useful bits of functionality if you want, but if it isn’t universal, it’s not there in the standard library. C is much the same way, for what it’s worth.

        Also you keep mentioning Go but it seems only fair to mention that Go *encourages* what you’d undoubtedly call “multithreaded” programming through the use of goroutines and channels. Of course, it’s rather elaborate runtime abstracts the details of how it does it, and it’s essentially event driven under the hood, but from the programmer’s perspective you’re writing synchronous concurrent code.

  13. ESR, any opionion on Shapiro’s BitC, from his (I presume defunct) Coyotos project? It’s the only language I recall hearing of that was actually designed as a C replacement for tasks where currently only C is suitable. (Which is not exactly your use case for NTPsec, of course.)

    1. >ESR, any opionion on Shapiro’s BitC

      I know shap slightly, but I never got a close enough look at BitC to evaluate it.

  14. @DTH, I agree that one of the goals of C is to avoid intermediate temporary allocations. But one of the goals of Rust is safe allocation of variables. Once you start doing that then idea of needing to know where strings are stored and how they are laid in order to assign one string to another or to combine them is irrelevant.

    Hence my objection stands. You should not have to use “Hello “.tostring(); ever in a language. The compilier has all the information to infer what the programmer wants to do with “Hello ” alone and with “Hello ” + “World!”

    Remember Rust has type inference built in. So it already doing automatic stuff behind the scene unlike C.

  15. This was my experience with rust when I tried to use it a few years ago for a small toy project. It felt unfinished and needlessly obtuse, and at the time was breaking compatibility and still trying new things with each release. I guess it hasn’t improved much.

    There are things I really love about the language, like immutibility by default. It pushes you towards a lot of good practices, but gives you a lot of ways of telling it “no, I really know what I’m doing here”. I also like the string formatting and slicing syntax being built into a C-like language, which I feel is table stakes for any new language these days.

    But overall, it feels like C with memory safety and syntactic sugar. That’s not really good language design. Go, by contrast, is stylistically less to my liking than Rust, but it’s extremely useful today. And part of that is because it was created by someone at Google to solve a problem, and it was then adopted by a bunch of teams, and then lots of people outside of Google. I’ve used it to write a few web services, where I think it really shines. I think it’s about nearly as easy as Python to just get up and start coding.

  16. @DTH for example from here
    http://rustbyexample.com/cast/inference.html

    You mean to tell me that the designer of the compiler could figure out this

    // Because of the annotation, the compiler knows that `elem` has type u8.
    let elem = 5u8;

    That the designer couldn’t have the compiler figure out what this means.

    let Hello = “Hello “;
    let World = “World!”;

    let HelloWorld = Hello + World;

    If you want more complex options for String storage then fine. Have .toString(); available and use the & symbol if you need to make the distinction. But for the common cases use the most elegant syntax possible and have the compiler handle the rest.

  17. > Right. Then you run into the problem that the socket bind primitive Won’t actually take a String for its address.

    It will take a &str, which is a kind of string.

    > And that there seem to be two different kinds of String things, one from std and one from collections, and who knows what function that takes something stringlike will take which?

    There is `String`, which is owned, growable, and heap-allocated and `&str`, which is an immutable “view” into a string. Almost always, a function taking something stringlike will take a `&str`. The only reason for a function to take `String` is if it wants to take ownership of the memory that’s backing the string.

    I think the distinction (and the motivation for it) is explained quite well here: https://doc.rust-lang.org/stable/book/strings.html

    1. I think the distinction (and the motivation for it) is explained quite well here: https://doc.rust-lang.org/stable/book/strings.html

      Which exemplifies what I mean by inadequate documentation. I read and reread that page until I had it engraved on the inside of my eyeballs. I understand he theory; I get the difference between references and allocated string objects, I know why it’s there.

      In practice, the theory didn’t help.

      1. I don’t honestly see that the issue is. In C++, is a char* the same as a std::string? No, because one is a pointer to an already-allocated space (hopefully, but not required (!) to point to a null-terminated array of bytes), the other is a class which contains a lot of info about the string, such as encoding and length.

        Rust’s &str is a slice. It’s a view into an already-allocated string (often on the stack, but static heap memory is common as well). Because it is a reference, it can be passed around (i.e. “borrowed”), but changing it in one place can change it in another place, hence it’s dangerous to just be throwing around. Furthermore, that memory can be removed at any time and any dereference after that will blow up. Rust prevents this (it’s the whole point of the language’s borrow-checker after all), and forces you to use a lifetime to prevent one function from deleting it while another function is borrowing it. Same as passing around char*s everywhere. You pass in a pointer to some bad memory (because some other deuced thread freed it) a few times, and you realize how much you’d love a language that prevents you from doing that.

        String is a heap-owned string and that ownership is passed around and easily understood.

        Seems like a pretty trivial thing for a software engineer, especially one not beholden to a “JVM-view” of things (where memory seems to be seen as some “magic-space” to do with whatever one wills, at the expense of good programming) to understand, so I’m confused as to the difficulty in understanding the distinction.

  18. > I am also surprised to see people, ostensibly grown men, use the term “SJW” without apparent irony in here.

    I am surprised to see ostensibly grown men like Klabnik who still think Communism is a good idea.

    > This whole attitude reminds me of the sort of cultural “othering” that produced the red scare.

    The reds murdered 100 million people in the last century. I for one would prefer to avoid a repeat of that. “Red scare”, indeed.

    1. WTF does “Social Justice Warrior” have to do with Communism? I mean, I’m not super up-to-date on the lingo of either the Alt Right or the millennial progressives, but I didn’t realize that one was exclusively related to the other. Did I miss a connection where a “SJW” necessarily supports a particular form of totalitarian government?

      And WTF does either have to do with Rust? I have to admit the OP insinuating that anyone who defends Rust is a “SJW” seemed pretty non-congruent to me as well.

      1. >WTF does “Social Justice Warrior” have to do with Communism?

        Origin. SJWism is a kind of memetic infection with theoretical roots in Marxism (power relations between identity groups replace power relationships between classes) and very strong rhetorical roots in old Soviet propaganda tropes.

        >I have to admit the OP insinuating that anyone who defends Rust is a “SJW” seemed pretty non-congruent to me as well.

        I don’t think I said that, because I don’t believe it.

        Some technical subcultures seem more SJW-infected than others. Self-identified Rustaceans are high-ish on that scale, though not the worst I’ve seen. “Anyone who defends Rust” is a much larger category about which I won’t generalize.

  19. @DTH you try to criticize us for “othering” by using a term like SJW, then you immediately try to “other” us by questioning our manhood. Go pull the other one, it has bells on.

  20. Quote: Clearly the biggest issue is the so called *tribal elder* of FOSS either being totally ignorant, or totally obeisant to using FOSS libraries.

    And there is that Netscape/Mozilla arrogance I was referring to. It reeked to high heaven back in the 1990’s, and it reeks now. Some things don’t change.

  21. >You should not have to use “Hello “.tostring(); ever in a language.

    Agreed. Certainly not in a language that purports to do type inference.

  22. I am also surprised to see people, ostensibly grown men, use the term “SJW” without apparent irony in here. This whole attitude reminds me of the sort of cultural “othering” that produced the red scare.

    Right, because it’s the people using the term “SJW” (without irony!) who are demanding nonpolitical organizations adopt “codes of conduct” against unacceptable opinions, engaging in name-and-shame tactics, blacklisting people from development conferences for their private opinions expressed in private correspondence, and hounding people like Brandon Eich from their jobs for having taken controversial political positions unrelated to their jobs.

    Yes, DTH, there’s a cultural “othering” just like the Red Scare going on, certainly. And you are defending the modern McCarthyists.

    1. 1) There is no such thing as a non-political organization that is made of human beings.

      2) Eich’s job was to be the public face of Mozilla, like any other CEO. When he brought that public face into disrepute (whether by his fault or not is irrelevant) he had to go.

      1. Disrepute among whom? The problem with SJWs is they think their clown world bubble is bigger than it is – in reality the wider world finds them far more disturbing and disreputable than the people they raise communist lynch mobs (both digital and meatspace) against.

  23. >Right. Then you run into the problem that the socket bind primitive Won’t actually take a String for its address. And that there seem to be two different kinds of String things, one from std and one from collections, and who knows what function that takes something stringlike will take which?

    Hello Eric, I’ve dabbled with Rust and Go and I think I can answer your question though I don’t know where in the documentation this feature is discussed. The short of it, I recall, is that a “String”, the allocated type, can always be coerced to a “str”, the more restricted type. I don’t know how to describe this latter type precisely… it’s somewhat like a Go slice and a bit like a C++ string_view. This coercion should not require further allocation, so the runtime characteristics are favorable. I believe the standard library prefers “str” in all cases due to these properties.

    >For epoll: There is a crate. You could also use future-rs or tokio. So you have three possibilities.
    >>Which of those three will be maintained in then years?

    I can coincidentally answer this question, because yesterday the first public release of “Tokio” occurred, of which “futures-rs” is a building block, and the announcement was widely circulated. This project appears to be a significant effort of the Rust project developers themselves, and might be the “blessed” library for which you so long in your essay above.

    However, I say this not to dissuade you from your trajectory. I think that Go will be a solid platform upon which to extend NTPSec. I think that Rust may deserve more credit than you give it here, though I would wait until the platform has had as much time as Go has had to mature. Another five years would seem about correct to me.

  24. > Is “swift” on your list of things to try?

    +1

    I haven’t used it a lot myself, but it seems reasonably pleasant.

    IBM has released a cross-platform socket library called BlueSocket that our host might find useful.

  25. > In practice, the theory didn’t help.

    When learning something new and different, I always struggle to put it into practice the first few times. I don’t think your experience proves that the docs are bad, just that rust does things differently from most other languages.

    FWIW, the steepness of the learning curve is something that the rust community is very aware of. For example, people are very patient with beginner questions on IRC…

  26. > You should not have to use “Hello “.tostring(); ever in a language.

    You never ever need to use “Hello”.to_string() in rust. Just use “Hello”.to_owned() instead: it does exactly the same thing (namely, puts “Hello” into a growable, heap-allocated String), but with a clearer name.

  27. To answer your question about how to write asynchronous services, the community is actually coalescing around a set of crates for asynchronous programming; Tokio provides a few layers, the lowest just providing a reactor and I/O primitives, and higher layers providing traits for defining codecs and services, which uses mio as a low-level wrapper around select/poll/epoll/etc, and Futures as the core abstraction for deferred work.

    mio has been around for a couple of years, and has been the de-facto low-level wrapper in the ecosystem; tokio just released 0.1 yesterday, though it was announced a few months ago and already many projects have moved to it, so it is well on its way to becoming the standard.

    As far as setting an overall direction, I think the Rust team and community do a pretty good job at that; the issue is, it’s not communicated well if you’re not following the community closely. The Rust team just set a roadmap for 2017 that aims to address several of the issues you’ve run into; reducing the learning curve, making high-quality crates in the ecosystem more easily discoverable, getting more of that ecosystem to 1.0 status, and making it easier to write robust, high-scale servers, including things like async libraries and better language support for making async code easier to write.

    Finally, I’m curious what your difficulty with concatenating strings was. It is the second example in the docs on String, as well as in the book.

    I found, when first learning Rust, that it took a little while to figure out where I had to look for documentation, but once I’d gotten the hang of it, I can now find things quite easily. One convenient way to get over that hurdle is to ask questions, on the users forum, on Reddit, on IRC (#rust-beginners or #rust on irc.mozilla.org), or on Stack Overflow.

    1. >As far as setting an overall direction, I think the Rust team and community do a pretty good job at that; the issue is, it’s not communicated well if you’re not following the community closely.

      That’s nice to know, but it’s not very helpful to my situation. If it’s not obvious why not, I can explain…

      >making high-quality crates in the ecosystem more easily discoverable

      I think you just put your finger on a large issue that I was trying unsuccessfully to pin down about why the documentation is so damned unhelpful. It tells you a lot about what the compiler does, and I got that part. But there’s an explanatory gap between the core language and the crate system that the documentation doesn’t bridge. (I actually feel better having figured this out.)

  28. Not sure how anyone could take this post seriously when it begins with

    > I finally cleared my queue enough that I could spend a week learning Rust. I was evaluating it in contrast with Go, which I learned in order to evaluate as a C replacement a couple of weeks back.

    Go never was nor will ever be a C replacement. It’s an entirely different category. It’s up there in the Java and Python spectrum, sure, but C? Not even close.

    > Then I found out that a feature absolutely critical for writing
    network servers is plain missing from Rust. Contemplate this bug report: Is there some API like “select/poll/epoll_wait”? and get a load of this answer:

    Your entire argument is rendered invalid and is proof that you haven’t done enough research. There are many great libraries that can do just this, and as posted in your issue report, tokio.rs is the result of much work that has gone into developing a powerful solution that’s superior to existing solutions.

  29. I must also add the following:

    > In practice, I found Rust painful to the point of unusability. The learning curve was far worse than I expected; it took me those four days of struggling with inadequate documentation to write 67 lines of wrapper code for the server.

    So you’re basing your entire opinion of a language on four days of learning, and you decided to tackle a problem that would require a higher level understanding of the basics out of the gate. That’s ridiculous.

    Rust isn’t Python — you need to do some studying, read the book, and read the documentation. There are a lot of concepts in here that you won’t find in most languages you are familiar with. There’s algebraic types, lazy iterators, higher-order functions, traits, lifetimes, and a borrowing and ownership model. These are all necessary and have allowed Rust to become the powerful language that it is — no garbage collector, no runtimes, no overhead.

    > Even things that should be dirt-simple, like string concatenation, are unreasonably difficult. The language demands a huge amount of fussy, obscure ritual before you can get anything done.

    Because typing this is too hard?

    string.push_str(other_string);

    Geez you’re a baby.

    > The contrast with Go is extreme. By four days in of exploring Go I had mastered most of the language, had a working program and tests, and was adding features to taste.

    Go is not a low level language nor a replacement for C.

  30. All this business about the various ways to get “Hello” into something usable as a string ignores the basic issue: there’s exactly zero reason for “Hello” to be anything but a string constant! If you can’t use “Hello” anywhere you can use a string, you’re doing it wrong!

  31. Btw. thanks to this piece of critique, the compiler error messages shown when concatenating strings has type clashes will be improved: https://github.com/rust-lang/rust/issues/39018

    Also, the roadmap for 2017 specifically addresses many of the problems/concerns presented: https://github.com/aturon/rfcs/blob/c59aa270f02f37f908e55b664c82321efc7fbfbe/text/0000-roadmap-2017.md

    Furthermore, just yesterday there was the initial release of Tokio, a framework for writing network applications in Rust, that is projected to be the “blessed” one.

    You might want to check Rust out again in a year or so – it seems that “fixes” to many of the problems you encountered are underway, and there’s increasingly work towards ergonomics, mature libraries, saner learning curve etc.

  32. >Because typing this is too hard?

    > string.push_str(other_string);

    Once? No. Every time I want to concatenate strings, for the rest of my life? Damn straight.

  33. This critique is disappointing. It looks that C is what you know, and C is what you want to stick with. Therefore Go being similar to C is what you enjoy. And that’s fine. But then critique of Rust is just trying hard to explain how all you want is just a little better C.

    The “high barrier to entry” is just you being long time C coder reluctant to change your habits, investigate and understand things with open mind. You’d like to get more secure, reliable C but without “obscure ritual before you can get anything done” which translates to “anything I don’t understand already”. So you’re fine with allocating everything on heap, GC, heavy runtime, and therefore lack of ability to export a pure C API. I wonder how long will it take you to get sick of `if err != nil { }` ritual after every operation. Maybe you don’t mind, since that’s what we’ve been doing in C (I’m also C dev) and beauty of Mondic error handling is not interesting to you so you missed it. You probably also missed the way how `cargo` version management works which makes it perfect for reliably reusing 3rd party libraries that solve all other issues you’ve mentioned (like mio for portable epoll, mioco for green threads). So you think standard library is the only thing you can rely on and if something isn’t there, it isn’t “in the language” and can’t be used/trusted (as this is how it was in C).

    Rust has it pain points like ecosystem immaturity, but your critique is mostly mislead. With all due respect: It’s not Rust, it’s you. You want a bit better C, and Go is exactly what you need. Rust is “disapointing” for you, only because it’s not just a little better C.

    1. >This critique is disappointing. It looks that C is what you know, and C is what you want to stick with,

      Guess again, meathead. I’m actually an old LISP hacker who would be delighted to have a replacement systems language with the higher-order abstractions I miss from LISP. I didn’t have any trouble understanding Rust’s theory of operation; the problem is I don’t yet see that theory cashing out to a language I can use for production.

      1. I’d have to concur with this. As a fellow lisp hacker, I thought rust look great from reading the book. In use, though, I found it tough going; for me the main problem was the pain around creating a circular data structure.

        The core ideas behind rust still seem good. Perhaps it will develop into something really cool.

  34. ESR, you wrote:
    > So, yeah, [William Cody Laeder] just said RUST IS USELESS FOR WRITING SINGLE-THREADED SERVERS.

    I would rephrase as “Rust’s standard library is insufficient for writing single-threaded servers; you need a crate”. It’s frustrating that this decision isn’t clearly spelled out in the old bugs you looked through, but IMHO it’s the right decision for Rust. It’s not as opinionated a language as Go. That means some things are harder than with Go or require looking beyond the standard library, but also that some things are possible with Rust that aren’t with Go.

    You also wrote:

    > Let me repeat: “There are 23 crates for that” is a horrible answer. I need a language with a sound core architecture, reasonably coextensive with a full ANSI binding of C, in which I can be reasonably confident that the features I need to rely on will still be there in a decade.

    I agree this is a valid reason for avoiding Rust.

    My best guess is that tokio will become the crate you’re looking for, but it literally just had its first release two days ago, so I can’t say with a straight face that I’m confident it’s what everyone will be using in a decade or that its API in a decade will be anything like its API today. Personally I love the language and community and think it’s worth going through this evolution with them, but I can understand why you’d make a different choice.

    As to one of your frustrations about strings:

    > And that there seem to be two different kinds of String things, one from std and one from collections, and who knows what function that takes something stringlike will take which?

    I think collections::string::String and std::string::String are the same type. I don’t know why there are two names for it, but the top-level docs for collections says “Unstable (collections #27783): library is unlikely to be stabilized with the current layout and name, use std::collections instead”. I agree this could be improved; the collections crate probably should be hidden entirely if it’s an implementation detail, or at least every page in the collections docs should be plastered with the same warning.

    I think the right way to do all the string stuff is simple once you figure it out; I bet the Rust docs folks would love a list of all the things that misled you along the way…

    1. >I agree this is a valid reason for avoiding Rust.

      Perhaps the actionable advice for the Rust community, then, is that you need to think harder about when you’ll move from generating options to cashing them out into a stable, standardized language. This question is generic to a large class of explore vs. exploit problems in scheduling theory and operations research – it even shows up in the strategy games I play for fun. Time for you guys to be a little more conscious about it.

      When I filter out the responses from the people who think I’m a dumbass because I didn’t instantly drink ALL TEH KOOL-AID, the common theme I’m seeing is that I tried this language too soon.

  35. “IRC and NTP are alike – and unusual – in that the most natural way to write them is as a single-threaded server.”

    I don’t know NTP, but that doesn’t make any sense for IRC. IRC is line-based protocol, and the most natural way to parse such a protocol is to read characters, with what either actually is or at least looks like* a blocking getc call, in a loop until you reach a newline.

    The “ideal” IRC server implementation would have two “threads” per connection – one that reads and handles messages, and one that picks up outgoing messages from a queue and writes them. The reason not to do this is cost, not the fact that a select loop is more “natural”. A language that can transform code that looks like this into a select/poll loop or equivalent seems like it would be massively useful for writing networking code in general, and it’s the direction that higher-level languages like C#, Python, Javascript, seem to be going.

    > (I can’t easily imagine how to write NTP as a thread-per-socket server at all.) Which means, yes, I want an API that looks exactly like poll/select, so all the code for handling client connections shares the same memory space.

    I’m not even sure what you mean by “same memory space” in this context, and in what sense threads don’t.

    *i.e. “the task sets its continuation as what to do when the file descriptor becomes ready for reading, then stops executing. when the file descriptor becomes ready, the select loop internal to the implementation will restore the task’s context”

    1. >The “ideal” IRC server implementation would have two “threads” per connection – one that reads and handles messages, and one that picks up outgoing messages from a queue and writes them.

      Yeah, you could do it that way. It would be unnecessarily complicated, though, and incur pointless thread-switching overhead.

      I have a simpler design in mind. I was going to describe it in detail, but I think that would be mostly a distraction. Because I can guess easily what parts you’d object to, and they’re places where things could be done your way in a TCP server but can’t be in a UDP server like NTP. We’d have wound up way off in the weeds arguing about things that aren’t germane to Rust at all.

  36. Jay Maynard on 2017-01-12 at 16:56:02 said:
    > All this business about the various ways to get “Hello” into something usable as a string ignores the basic issue: there’s exactly zero reason for “Hello” to be anything but a string constant! If you can’t use “Hello” anywhere you can use a string, you’re doing it wrong!

    Rust encodes lifetimes and storage type into the type system. This means that there can be multiple different types of “hello”, that need to be kept separate because what can be done to them differ. A simple “hello” in the code is a &’static, or, in other words, a pointer+length to the readonly static space of the executable. Upsides is that it costs nothing to allocate, and does not need to be deallocated, downside is that you cannot modify it in any way. What you usually want returned is a String, which is a pointer+lenght to heap-allocated memory. What you usually want to pass up is &str, which is a pointer+lenght to any memory managed by the caller.

    This complexity makes rust a bad language for string processing. However, the system does have it’s upsides, especially when optimizing huge workloads.

    Personally, a change I would have made is make the standard String class be either the current String, or point to statically allocated memory with allocate on write. It would handwave away most of the complexities of managing strings with no performance impact, except moving some allocations to the point of use.

  37. I would like to echo an above comment in that the select loop is suboptimal for something like an irc server when you can have clean r/w threads and CSP instead. select is hopelessly single core. threads and csp are not.

    I too was very put off by chasing “new/old documentation” and churn in trying to use rust for something real, and gave up. It felt like rust was a new and maybe less perverse version of the BSDM modern C++ programmers are used to, but was differently perverse enough to make me quit trying.

    /me goes back to fiddling with nim. Oh look! generics! Cross compilation to javascript

  38. > Perhaps the actionable advice for the Rust community, then, is that you need to think harder about when you’ll move from generating options to cashing them out into a stable, standardized language.

    I think they’re doing a great job of this. You want something that’s finished, and that’s fine, but unfinished doesn’t mean they’re going about it wrong or too slowly. Keep in mind that you’re comparing a language that hit 1.0 in 2012 (Go) to that hit 1.0 in 2015 (Rust). I’m not surprised that the one that’s three years older and was developed specifically with async network servers in mind is more mature for that use case.

    IMHO, the most interesting questions are which language will be the better one for your project in 10 years and, if that’s Rust, will it be enough better that it’s worth going through the extra pain of immaturity now? I don’t know the answers.

  39. > Right. Then you run into the problem that the socket bind primitive won’t actually take a String for its address.

    Since a String is an owned, heap-allocated buffer, you can always get an &str (string slice) from it.

    String dereferences to str, which so you can dereference it, and then take a reference to the result to get an &str: let string: String = ...; let s: &str = &*string;. This is a little cumbersome, so there’s a convenience feature known as deref coercions that allows you to abbreviate that as &string;. So if you have any function which takes an &str, and you have a String, you can just call func(&string).

    Using deref coercion to get an &str from a String is covered in the docs on String.

    The way to think about this is that everything that can take an immutably borrowed type should; and anything that can take a slice should. So, if you just need a string slice to create a socket (based on an IP address or hostname), then you take &str, and if someone happens to have a String, they just pass in a reference to that and everything works fine. If you did it the other way around, had APIs take an owned string, then you would need to allocate a new one each time you had a string slice, but needed to pass it in. This way, you are just passing in a reference, which is statically checked at compile time to be valid.

    > And that there seem to be two different kinds of String things, one from std and one from collections, and who knows what function that takes something stringlike will take which?

    This is actually the same type. Much of std is a facade over a few other internal crates, and it just re-exports from those internal crates. Normally, this isn’t supposed to be exposed, for instance it shows up in the docs as std::string::String, but there are some places in error messages that will show the original path rather than the one you’re accessing it from.

    >>So you have three possibilities.
    >
    > Yes, that was a warning flare all by itself. Which of those three will be maintained in then years?

    mio has been maintained for over two years (longer than Rust has been stable), and become the de-facto low-level abstraction over select/poll/epoll/IOCP that a lot of projects use over the past couple of years. For a while, however, there wasn’t one higher level abstraction that people had settled on.

    Mozilla then funded the author of mio to work on higher level abstractions, and two of the core Rust developers joined him, and have produced Tokio and futures. There is already work on HTTP, DNS, gRPC, Cap’n Proto, SSH, and more built on top of Tokio, despite 0.1 just being released yesterday. So, Tokio (and by extension, mio as the lowest level component) is looking like that core system that the community is coalescing around. Of course, it was just released yesterday; that’s no guarantee it’ll be supported in 10 years. But it’s got a pretty good amount of support behind it, and mio has a pretty good track record over the couple of years that Rust has been viable to work with.

    The choices are really: do you want just a thin wrapper around epoll (which is Linux specific), and write all of your event loop handling and dealing with deferred operations yourself? Then maybe you want the epoll crate (can’t tell you much about how well maintained that is, but it does look like it is at least recently maintained).

    Do you want a low-level cross-platform abstraction over event driven interfaces? Use mio. Do you want higher level interfaces over network I/O? Use tokio-core and futures (these aren’t really separate choices, futures is how tokio-core provides deferred operations). Do you want abstractions for services, request-response pairs, things like that? Use tokio-protocol. All of these are part of the same stack, just at different levels of abstraction, so the question isn’t really which is likely to be there in 10 years, but what level of abstraction is appropriate for your use case (I’m guessing the tokio-core level for NTP, but that’s just a hunch).

    1. > All of these are part of the same stack, just at different levels of abstraction

      And the core docs don’t presently offer me anything like the survey of options you just laid out.

      I trust you understand why this is a problem without my having to spell it out.

  40. >All these abstractions can be focused back down to practicality with a really simple question, to whit: where’s my goddamned POSIX select(2) primitive?

    In the libc system call. I fail to see how not being grafted to POSIX is Rust’s fault. If C made any sense, everything POSIX-y would be in a `libposix`, not in `libc`.

    1. >In the libc system call.

      That’s an obtuse answer. I meant, of course: “Where is the Rust core operation I can use analogously to select(2) in a Rust context? And if there is no such thing, why in the hell is there no such thing?”

  41. > Where is the Rust core operation I can use analogously to select(2) in a Rust context? And if there is no such thing, why in the hell is there no such thing?

    I’m not sure what “core operation” means, but here’s my understanding of what exists today. Part of the reason there are several crates is that they provide different levels of abstraction:

    * the “libc” crate, which is intended to wrap whatever is in the system’s libc; on Linux this includes epoll. This just gives you the C interface. It’s not going anywhere; you could use it today and keep your code the same for 10 years if you were so inclined.

    * the “epoll” crate, which is a very thin wrapper around libc. It’s only 145 lines including comments: . I don’t know if it will be maintained, but there’s not much to rot, and it’d be easy to fork or rewrite.

    * the “mio” crate, which is a higher-level interface, based on libc. I think it’s been around for a couple years and abstracts epoll vs kqueue and the like, similar to libevent or libuv in C. I haven’t used it myself. I’m guessing it will be around for a while given that tokio uses it.

    * the “tokio” crate, which provides a still higher-level futures-based interface, based on mio. It’s brand new, but lots of people are excited about using it. As I mentioned before, my best guess is that this is what people will be coding to in 10 years, but that’s just a guess.

  42. < And there is that Netscape/Mozilla arrogance I was referring to. It reeked to high heaven back in the 1990’s, and it reeks now. Some things don’t change.

    Amen brothers! Try Nim, here's an entertaining start:
    http://forum.nim-lang.org/t/2687

  43. > When I filter out the responses from the people who think I’m a dumbass because I didn’t instantly drink ALL TEH KOOL-AID, the common theme I’m seeing is that I tried this language too soon.

    No kidding. Even if I *liked* Rust (I tried it and didn’t), getting this kind of attitude from people would put me off from it. I’ve seen this movie before with other trendy projects. ESR’s not doing a buzzword-compliant undergraduate research project here, he’s working on foundational infrastructure of the internet. Rust isn’t ready for prime time in that space.

    1. This so much.

      Rust as a language has some good ideas, albeit buried under a bunch of bad ones, but my god, if “Rustaceans” aren’t the most obnoxious and self-righteous programming community out there today. Quite frankly, if in five years Rust was the perfect systems language, and the _only_ reason to use C over it was to keep the Rust zealots out of my codebase, that would be a HUGE reason to use C over Rust. I remember seeing someone respond to de Raadt’s criticism that rustc can’t compile itself on i386 because it exhausts the entire address space with “This isn’t a real problem, who still uses i386 anyway lol.”

      In college, one of my professors went on a rant about how he would _fire_ any TA who responded to a mention of a bug with “just upgrade your system.” For their sake and mine, I hope these people don’t have jobs.

  44. > Contrast this with Go, wherein I don’t have to wonder if any of the core abstractions will be gone in ten years exactly because they’re core abstractions. There’s a stability commitment there that Rust is not equipped to give me.

    You’re right. The Rust language and standard library are not yet ready to give you async IO primitives. And the Rust philosophy is that if it’s not essential to almost any program, or ready to be committed to for long-term stability, that development and experimentation should happen outside of the standard library. This helps to avoid issues like you see in the Python world, where it’s “batteries included”, but there are many tasks, such as fetching a URL, where the standard library API is awful, but can’t change due to stability, and there are much better third party libraries, but people just use the standard library one because it’s already there.

    > And those decisions have to be reflected in your core documentation. Well, that is, if you want the language to move beyond “toy” to a real production tool.

    Where in the C or C++ “core documentation” are the async IO primitives? Are those toy languages? They don’t provide standardized async I/O facilities. Different platforms do, but even on POSIX platforms, the only ones standardized are the older ones with various issues like select and poll, the modern interfaces like epoll, kqueue, and IOCP on Windows are all different. However, that doesn’t stop these from being production languages; some people just use the platform primitives directly, others use cross-platform wrappers like libev, libuv, or asio.

    > All these abstractions can be focused back down to practicality with a really simple question, to whit: where’s my goddamned POSIX select(2) primitive?

    If that’s all you want, libc is the crate you want; this provides the raw bindings to C library functions on POSIX-ish platforms. This is maintained by the Rust core team; it’s maintained outside of std since the APIs differ between platforms and so can’t be stabilized as part of the standard library, and also so that it can be quicker and easier to add more platform bindings in an external crate. Of course, since it’s raw bindings, it’s all unsafe, but you can use it to write a standard C-style select or poll loop.

    If you go this route, you’ll need to use AsRawFd or IntoRawFd to extract the raw file descriptors from sockets, and write a lot of unsafe code that calls C APIs directly. But if you don’t want safe, cross platform abstractions, and just want to call the POSIX APIs, you can do that just fine in Rust.

    > I think you just put your finger on a large issue that I was trying unsuccessfully to pin down about why the documentation is so damned unhelpful. It tells you a lot about what the compiler does, and I got that part. But there’s an explanatory gap between the core language and the crate system that the documentation doesn’t bridge. (I actually feel better having figured this out.)

    Yes. This is a very common complaint, and addressing this is one of the big goals in the Rust community right now. Apologies that you were bitten by this, and hopefully we can do better on this soon.

    > common theme I’m seeing is that I tried this language too soon.

    Yep, if you’re expecting to have stable, cross-platform, safe abstractions over async networking, that’s not there yet. If you’re willing to use stuff that is unstable (in the sense that APIs may change over the next year or two, though on the scale of “one or two backwards-incompatible releases in the next year or two” not “every week something breaks”), but has a lot of momentum behind and support behind it including funding from Mozilla and contributions from the Rust core team, Tokio is probably the way to go. If you just want low-level, unsafe POSIX bindings, those are all available.

    To give a little bit of history, Rust actually originally (long before the language stabilized with the 1.0 release) had an I/O story similar to Go’s. It had green threads, and non-blocking I/O primitives. However, green threads and non-blocking I/O primitives limit you in a lot of ways; they can be slower, they require a runtime complicating things like writing libraries that could be called from C in Rust, and calling conventions that make calling C from Rust more painful, that runtime also makes running on bare-metal platforms more difficult, etc. So the standard library was written such that tasks could be either green threads with non-blocking I/O primitives, or native threads with blocking I/O primitives. This allowed you to trade off between these two models, but imposed costs of its own, in complexity, in code bloat, and in performance due to the extra dynamic dispatch between the two systems that needed to happen whenever you did I/O.

    Eventually, the Rust developers realized that for the kind of systems space that Rust was targeting, not adding overhead on top of the OS provided primitives was important, and the abstractions that Rust provides could be used for providing good async I/O as a library feature rather than requiring specific language support, so the green-threaded, non-blocking I/O code was jettisoned, making the Rust standard library I/O now be a very thin portable layer over system I/O.

    Once that happened, the Rust developers focused on stabilizing Rust and releasing 1.0, so not a lot of new work was done in this space, mostly a lot of work on the final cleanups and refactors of the API that were now a lot easier, and getting everything stabilized and ready for release. After 1.0, there was still a good amount of work of fixing up and stabilizing more functionality that missed the cutoff, and getting the ecosystem off the ground; in the meantime, other people started experimenting with various abstractions in the ecosystem.

    In the middle of last year, however, Mozilla funded the developer of mio, which had become the de-facto low-level wrapper, to work on higher level APIs, and several of the core Rust developers got involved as well. That brings us to now, when 0.1 of that work has been released.

    So yeah, right now, that space in Rust is not quite ready but developing rapidly. Good time to get involved if you want to influence the direction it goes; bad time if you just want something that will be stable for the next 10 years.

  45. > I know language documentation doesn’t have to be that bad, because a few weeks ago I learned Go in less time and effort than it took me to give up Rust as a bad job.

    Go is an easy language to learn. Rust is a hard language to learn. The docs can only do so much to change that. Programming in Rust is enough of a PITA that I’ve switched off of using Rust to implement a (relatively simple) GitHub webhook-based service*. And I’ve used Rust pretty much since 1.0.

    Rust has a lot of infatuated fanboys who oversell it. Real scoop: it isn’t going to have higher programmer productivity than C or C++ until your codebase is big enough that you can’t keep all the complexity in your head at once**, because that’s the point where you’re going to need to rely on the compiler to catch would-be Undefined Behavior. Alternatively, if you’re used to copy-heavy and reference-counting-heavy safe-C++, it’s going to be a matter of performance rather than productivity.

    It’s not going to have higher programmer productivity than garbage collected languages like Go at any point, because GC allows you to rely on the runtime the same way as Rust allows you to rely on the compiler, except the runtime is far more forgiving.

    I should probably be explicit with my point: if Go’s shortcomings aren’t eating you alive, then it’s probably not worth switching to Rust. Stuff like Servo and Ripgrep play to Rust’s strengths, because ripgrep isn’t a network app, and while Servo is sort of a networking app, it doesn’t need to do C10K.

    A REST API that spends most of its time waiting for other REST APIs plays to Rust’s weaknesses. You’re never going to see the advantage of optimizing low-level memory access patterns in an application like that. NTPsec is at an uncomfortable point in the middle, which is actually the very niche Go exists for.

    * To Elixir, if anybody cares: https://github.com/bors-ng/bors-ng. I’ve used Go before, and it sounded interesting and more appropriate to this project. I’ve actually never used Ruby, because I never saw a reason to pick it over Python.

    ** I’ve used C and C++ before, but I know Rust better than either of those two languages. It helps that actual human beings can remember things like the syntax for Rust’s function pointers. :-)

  46. Let me repeat: “There are 23 crates for that” is a horrible answer. I need a language with a sound core architecture, reasonably coextensive with a full ANSI binding of C, in which I can be reasonably confident that the features I need to rely on will still be there in a decade.

    “There are 23 crates (read: libraries) for that” is just the answer you’d get from C, though. As Brian Campbell carefully and extensively pointed out (as is his style), I/O primitives on different operating systems differ widely and will have different C APIs; why would you expect a single API in Rust to cover all possible applications on all possible platforms?

    The thing to watch out for is a reasonably widely accepted set of bindings for your chosen OS’s I/O primitives. Maybe this doesn’t exist for Rust yet, in which case your choices are to wait for it to emerge, or to chip in and start hacking on it yourself.

    Alternatively, you could use another “safe” language like Ada that’s mature enough to have community standards for the I/O calls you need. (Ada actually does have an ISO-standard POSIX API, and there’s an open-source implementation called Florist that works with the GNU Ada compiler.)

    Can you tell I actually like Ada, and have learned to stop worrying and love the verbose Pascal-esque syntax? :)

  47. Reading these comments has been unsettling. It seems like people have just decided to band together into disgusting little cults when it comes to Rust, each accusing the other of evil and incompetence. Just mindless shit-slinging drivel, by people who clearly can’t tolerate anymore. Can’t tolerate having to learn something new and not quite mature, can’t tolerate having to deal with other people, and can’t tolerate just being criticized for having no substance in their arguments, only thinly-veiled personal attacks.

    I’m going to need to lie down and reevaluate my opinions on a lot of people I’ve worked with and revered over the last 30+ years. All because they’re such colossal babies you’d think they haven’t actually matured over time. Congratulations all around.

    1. Welcome to humanity, pal. It hasn’t changed much from when we were LITERALLY flinging shit at each other, except at least these days we don’t meet arguments we don’t like to hear with blades and fire (at least not all the time).

      I don’t mind sitting down and explaining why Rust has some damn fine points and why some decisions were really the right ones (from a FP and a memory-handling point of view), but I also don’t mind admitting there are quite a few shortcomings so far, so watching people just blather insults at people who are questioning some of Rust’s strange quirks is pretty frustrating as well.

      However, just talking to a wall of willful ignorance when people WANT to believe that Rust sucks and won’t listen to any point answering their gripes, but just return one excuse after another why they’re still going to believe the way they do despite clear evidence to the contrary, well… it just becomes yet another political issue then, doesn’t it? Just like real politics (with “fake news” basically meaning any news that doesn’t reinforce one’s already-set and immutable internal beliefs), this discussion has shown that EVERYthing is a political fight (and that’s not new, just go back to message boards circa 1980 regarding PC vs. Mac or C vs. Lisp, etc. etc.).

  48. ESR, any opionion on Shapiro’s BitC, from his (I presume defunct) Coyotos project? It’s the only language I recall hearing of that was actually designed as a C replacement for tasks where currently only C is suitable. (Which is not exactly your use case for NTPsec, of course.)

    BitC, Cyclone, ooc, etc. All of those were interesting research projects that got nowhere. BitC struck me as really cool because it was a cut-down Lisp dialect that actually reminds me of my favorite systems-programming language — Pre-Scheme. But without developer — or, particularly monetary — interest in the Coyotos research, BitC is DOA.

    A general rule is that you use the tools, libraries, and languages favored by what I call the Open Source Hivemind, because those are the ones that are going to get developer love, a large support community, and commercial interest (and therefore testing and refinement from real-world use cases). The Hivemind is fully committed to systemd for service management, Wayland for graphical display, and PulseAudio for audio, which is why it doesn’t matter if you personally find them distasteful, they will become the de facto standard. For systems programming, Rust looks increasingly like it will be the Hivemind’s choice to replace C, as it has the backing of the Mozilla organization, is being used for real work (the servo engine), and meets the necessary criteria of being a strongly-typed, non-GC’d, safe, bit-level language that supports generic programming.

    1. You’re right about the hive-mind. Except that Rust is so obtuse that nobody will ever develop anything worthwhile in it, and the hive-mind is clearly choosing Go. There are literally zero workstation or server applications written in Rust that anybody uses (save for a tiny bit of Firefox’s internals). No command line utilities, no web frameworks, nothing that anybody would choose to use (or foist upon their poor employer) for any reason other than being a Rust fanboy. Meanwhile in the real world: C, Go and Python (and unfortunately, Java) are still being used to solve real problems.

      Ruby was a miserable failure of language+ecosystem+community based on some truly bizarre ideas that don’t pass the sniff test to any engineer worth his/her paycheck, and that same community is also screwing up any chance Rust had to be successful. Ruby’s claim to fame was being the bad alternative to Python, and Rust will be remembered as the bad alternative to Go, but it’s questionable that Rust will even be as successful as Ruby was.

      1. >Ruby was a miserable failure of language+ecosystem+community

        Be fair. The language design itself isn’t bad – as much as I like Python I have to concede that Ruby has better and more consistent functional-programming support. Matz himself seems to have his head on straight and I have a lot of respect for him. It’s the crap around Ruby – especially Rails and its entertainingly dysfunctional community – that’s a problem.

  49. >Rust encodes lifetimes and storage type into the type system. This means that there can be multiple different types of “hello”, that need to be kept separate because what can be done to them differ.

    In other words, the language emphasizes theoretical programming language wankery over usability. If your language has that many different types for friggin’ “hello” it is fundamentally broken. Sorry, it just is.

    I’ve seen this movie before (a half-dozen times, at least).

    “Hey, Rocky! Watch me pull a Hindley–Milner type system out of my ass!”
    “But that trick never works!”

    > Even if I *liked* Rust (I tried it and didn’t), getting this kind of attitude from people would put me off from it.

    No kidding. Before this thread I though Lisp people had an arrogance/poor social skills problem.

    At least with Lisp the arrogance is somewhat justified. I have seen no evidence for such with Rust.

  50. where’s my goddamned POSIX select(2) primitive?

    Fun fact: select(2) — and any “async” I/O model that relies on synchronous-but-nonblocking I/O primitives and polling fds — performs like absolute dogshit under Windows. When you’re writing async I/O for Windows, you want to use overlapped I/O calls and I/O completion ports to notify your thread pool when operations complete. Do that and your application’s performance can significantly exceed anything on Linux.

    So, select(2) is not a universal primitive across operating systems or environments, and because it’s not applicable in all scenarios, the Rust developers kept your goddamned select(2) out of the motherfucking core language, as well they should.

    Wait for a library that provides the abstraction you need to become standardized, or write one yourself.

  51. “can’t tolerate having to learn something new and not quite mature”

    Where are you getting that? He said that he read the book twice. That doesn’t sound like “can’t tolerate having to learn something new” to me.

    1. >He said that he read the book twice.

      I did – and, as I’ve also said repeatedly. I found the theory of Rust very interesting. And not that difficult; there’s really only three primitive notions in the theory (mutability, single ownership, lifetime) and if you’ve been around the track as often as I have they’re not particularly daunting.

      The immaturity bothers me because of my mission constraints. I’m not choosing a language to just randomly screw around with; NTPsec has very specific requirements and a need for long-term stability of API that Rust cannot yet meet.

      I am still a little skeptical about the social machine reflected in the crate system being able to converge to a language with the properties I need, but a few of the more thoughtful advocates to show up here have at least made the right kinds of noises about that.

  52. I’ve added an update to the OP reflecting some of the education I got from the most mature and thoughtful pro-Rust respondents.

  53. I’d like to read about rust downsides, but real downsides, not the “I tried, and failed”. Rust is not C, Rust has its own abstractions, which make learning curve steep. I agree, that such a learning curve is not good, but its not a point, while we don’t get into consideration reasons behind that.
    Unusual abstractions of rust make it hard to understand. While learning rust I have feeling like in old good days, when I learned assembler and lisp: I was able to understand, but unable to code. If you do not pass this phase of learning, do not move forward, then you opinion on rust is pretty useless.

  54. @Doctor Locketopus There is only one kind of “hello” in Rust.

    let hello = “hello”;

    The above is an &str, more specifically an &str with a static lifetime. A static lifetime in this case means that it is compiled and located within a region of memory that is binary. You can do anything with an &str that you would a String, except enact dangerous consequences such as updating the string or growing it. You are free to slice a string, and this is recommended. It would make perfect sense if you understood how UTF8 string encoding works from a systems software perspective.

    The str type is basically a UTF-8 character array with characters of varying sizes. The str type will never be interfaced with directly because this type is managed by a heap-allocated String or compiled into the binary as a &’static str. They are dynamically destroyed and allocated by the String when the String needs to grow. There is zero allocation overhead to operating with &str variables.

    When you want a collection, or a heap allocated data structure, then you are looking for a String. It is merely a data structure that manages internally-created str data. When you reference a String, you are given an &str.

    let mut hello_heap = String::from(“hello”);
    let world_stack = ” world”;
    hello_heap.push_str(world_stack);

    The `hello_heap` is a String and is located on the heap. The world_stack is an `&str` and is located on the stack. Pushing an &str into a String will migrate the data within into the heap location owned by the String.

    This is all simple stuff for a systems software engineer. It is a great thing that Rust makes this distinction instead of trying to enact terrible engineering mistakes that most other languages do, which typically require a garbage collector running in the background in order to abstract and dumb down the type system. Rust is able to promote zero allocation and zero cost claims because of details like this.

    > Where are you getting that? He said that he read the book twice. That doesn’t sound like “can’t tolerate having to learn something new” to me.

    Reading a book twice isn’t the same as comprehending the book. Additionally, he stated that he only spent four days, and after four days he decided to basically troll post.

    1. >he decided to basically troll post.

      With advocates like you Rust scarcely needs any enemies,

      Now go read the update I added just before your last comment.

  55. Freudian slip on your opinion whether Rust will “become the mature and effective replacement for C that it is not yet”? ;-)

  56. Rust definitely has a learning curve, but give it a few months and you’ll begin to understand it. It’s a very intuitive and consistent language, but it does require a new intuition to be learned. For example, It’s best to wrap impl’d traits in the `impl` for a new object so that there is one well-defined public interface for code calling the object from outside the module, solving many of the early accessibility errors I would receive.

    One of the greatest assets of a young language like Rust is adaptability. It certainly lacks infrastructure, graphic libraries are still a work in progress for example, but these things are in development and growing now that the core of the language has solidified. As you learn, perhaps consider becoming a regular contributor to Rust and you’ll quickly find yourself in a position to help steer it.

    To echo your issues with Rust, a major adoption pain I had with Go was that it had a near-complete lack of native windowing toolkits; most available libraries at the time were graphics servers on localhost. Infrastructure is an absolutely fair reason to avoid adopting a language, but also one of the things that can change faster than the language itself does. Python, for example, has developed an absolutely massive amount of infrastructure because users contributed to a language that had potential!

    1. >Rust definitely has a learning curve, but give it a few months and you’ll begin to understand it.

      In what universe is that supposed to be persuasive? Let’s see, I can write in Go which I effectively mastered in four days, or I can invest months in Rust. In order for that tradeoff to work out to Rust’s advantage, Rust has to offer me enough that Go does not to justify the additional effort.

      It’s hard to imagine a use case in which that’s plausible, outside of writing a direct-on-the-metal OS kernel or hard-realtime system.

  57. “This is all simple stuff for a systems software engineer. ”

    Sure it is. So are Turing machines. That doesn’t mean I want to start writing all my code for one.

    Unless I’m writing a device driver or some such, which should I give a shit how my string is being stored? I don’t actually care if it’s on the heap, on the stack, or on a USB thumb drive in orbit around Pluto. That’s why I have a compiler, dude.

    Requiring this sort of excessive attention to detail is a sure sign that a language has been designed by theoreticians rather than actual programmers, and it usually gets worse over time, not better (I don’t recall any instance where it’s ever gotten better, offhand).

    In languages that are type-happy it winds up with every UTF-8 character being its own incompatible type. In Java-style OO languages it winds up with ThingFactoryFactoryFactoryFactories all over the place. In languages that emphasize pure functions, it winds up being impossible to output a character or set a pixel on the screen, because ZOMG!!! STATE!!! EVILL!!!, plus you have to teach your data entry people to input all the data in the form of Church numerals or some shit.

    Obviously the previous paragraph was exaggerated for effect, but it is a real problem.

    String concatenation came up earlier in this thread. I’d almost say that “how straightforward is it to do string concatenation in this language” is a definitive tell for whether a language was designed by (and for) programmers or theoreticians.

  58. “With advocates like you Rust scarcely needs any enemies”

    I love the blithe assumption that we need to be lectured on the differences between the stack, the heap, and static storage, how UTF-8 strings work, and so on.

    I suspect that I understood most of those concepts before Michael Aaron Murphy was even born.

  59. >Where are you getting that? He said that he read the book twice

    …because surely I was talking about him and only him, yes?

    But go ahead and keep making this an us-vs-them thing, you’ve made it clear which camp you side with and just how little you think of “the other guys”.

  60. “…because surely I was talking about him and only him, yes?”

    Considering that you were responding to a post on his blog, I don’t think that was an unreasonable assumption.

  61. >In what universe is that supposed to be persuasive? Let’s see, I can write in Go which I effectively mastered in four days, or I can invest months in Rust.

    Exactly, all languages come with adoption costs. Rust is new, but the basic syntax can be learned in days. Go is familiar but inherited familiar problems too. I really didn’t see reason to adopt Go with C++ in my toolbox. In the same way, you have Go in your toolbox now and are content without Rust.

    It’s perfectly fair to say that what you have already fulfills needs that something new could. But that’s also not a valid reason to make any argument besides that the new thing isn’t needed by you. Your criticism of networking and concurrency infrastructure is a much more valid reason to refute Rust itself, rather than your need of it.

    I really respect that you cited specific problems with the language, and that you also have the open mind to consider when your problems are unique to you.

    I think your premise, that Rust is missing critical features, is absolutely spot on! The conclusion that Rust is failing, however, is quite premature.

  62. >It’s hard to imagine a use case in which that’s plausible, outside of writing a direct-on-the-metal OS kernel or hard-realtime system.

    Interestingly this was one of the first use cases I found when I googled Rust. The project is called Redox and is based on Minix. It’s definitely not ready for production.

  63. @Jeff Read:


    > Can you tell I actually like Ada, and have learned to stop worrying and love the verbose Pascal-esque syntax? :)

    Oberon is easily the best designed language in the pascal family. Go is in many respects a glorified version of Oberon

  64. The harping on Rust for missing select(2) seems a bit excessive, considering that even Go offers only thin syscall wrappers for it (the deprecated syscall.Select and golang.org/x/sys/unix.Select, outside the standard library) that are not integrated with os.File or net.Conn; and indeed won’t be, since the authors prefer to push I/O multiplexing to the channel level. Do you also count this as a point against Go, or does the fact that they are available by default at all make all the difference?

  65. let mut hello = String::from(“Hello, “);
    hello.push(‘w’);
    hello.push_str(“orld!”);

    this can be alternatively written as

    let mut hello = String::from(“Hello, “);
    hello += “world”;

  66. @Jeff Read
    “A general rule is that you use the tools, libraries, and languages favored by what I call the Open Source Hivemind, because those are the ones that are going to get developer love, a large support community, and commercial interest (and therefore testing and refinement from real-world use cases). ”

    That is saying that technological developments follow actual developer use. It sounds as if you are surprised that technology is a social phenomenon?

    As far as I know technology has behaved as a Hivemind from before the invention of fire. They even describe paleolithic technology in this fashion.

    Anyhow, a computer language is a tool for performing specific tasks. The more difficult a task is done in a language and the more lines of code is needed, the more bugs will crop up. If the tool is bad at doing a task, you look for a better tool. Eric cannot do the task he wants with Rust easily or even at all, so I see no point to keep trying when a better tool is available.

  67. As a relatively inexperienced rust and C programmer, currentlly rust definately does have a steep learning curve. But C, despite being a small language, is far more difficult to learn to program *well*.

    Regarding the comments about strings, they do indeed take a few days to wrap your head around. But I think we’re forgetting that dealing with strings in C is arguably just as hard or harder to learn, less readable, and moreover prone to errors even for experienced programmers.

    C has only dumb pointers, ascii string literals, and standard library functions that make manipulation simple but dangerous. The programmer is responsible for allocation & deallocation, bounds and validity checking, reallocation in the case of a resize etc.

    Rust has Strings, smart pointers (&str), and a comparable standard library, all UTF8. String arguments are usually &str but you can pass an &String where an &str argument is expected and it will be coerced. Literals are &str pointers to statically allocated constant strings. That’s the gist.

    How a string is allocated is more explicit in the syntax of rust but still has to be *understood* to program with properly in C. Moreover if you get it wrong in C, your program will not work properly, crash, corrupt your data, or be a security risk. It is very hard to get it wrong in rust, the typechecker will stop you and often tell you how to fix your problem.

    Rust isn’t a language you can start programming straight away like many modern gc languages, but I still think having invested the time to read the book, it is well worth spending a few weeks with the language to get your head around the typesystems and the idioms of the language which in the end is a small cost in the lifecycle of a large project compared to the costs of debugging and maintenance, and the dividends of code re-use, quality, and performance.

  68. Comparing Rust to C in terms of usability is missing the point entirely, or so I see it. C is nearly 50 years old!

    There is no justification for a MODERN language to require using a bunch of ugly method calls to concatenate two strings. That is such a fundamental (and common) task that there should be syntax (or at least syntactic sugar) for it.

    When your language compares unfavorably in usability to quick hack jobs like PHP and Javascript, it’s a sign that something is very, very wrong.

    I’d hate to imagine what string interpolation looks like in Rust. Does it even have string interpolation?

    > which in the end is a small cost in the lifecycle of a large project compared to the costs of debugging and maintenance,

    What “large projects” have actually been written in Rust? Any? Can you point to any actual data showing lower debugging and maintenance costs, or is this merely an opinion?

    1. >Can you point to any actual data showing lower debugging and maintenance costs, or is this merely an opinion?

      I must say that for all my criticisms of the language as it now is I find the claim of lower debugging and maintenance costs very credible. Most of those are driven by downstream defect rates. I expect that if you can pay the high up-front costs of doing a project in Rust at all, the ownership system and other features will in fact lower the defect rate significantly.

      Whether and when that defect reduction is worth paying for in the way Rust requires you to pay is a different question. Other approaches, like having a garbage collector, can be better; it depends on where you are in the stack and how much latency spikes due to stop-the-world GC pauses actually cost you.

      But just because Rust is immature and sometimes overhyped as a solution for everything doesn’t mean it’s not a potential solution for anything. Yes, the Rustaceans need to make some hard decisions about defining and stabilizing a core API, and yes the language needs some serious documentation and usability improvements. These things are possible, at least in principle.

      The social question is whether they’re too drunk on their own kool-aid to actually fix these things. Some of the more measured Rust advocates that have shown up here seem not to be.

  69. “It’s hard to imagine a use case in which that’s plausible, outside of writing a direct-on-the-metal OS kernel or hard-realtime system.”

    These days, I write hard-realtime systems code for a living.
    In C. With no stdlib or any other library.
    On embedded controllers with 512K (for the later version) of ROM and 128K of RAM.

    Something tells me that I couldn’t cram a rewrite of our code in either Rust or Go into that box with a steam-powered piledriver.

  70. > On embedded controllers with 512K (for the later version) of ROM and 128K of RAM.

    One of the test boards zinc.rs targets is NXP LPC1768 which comes with a maximum of 512K ROM and 64K SRAM.

  71. > Unless I’m writing a device driver or some such, which should I give a shit how my string is being stored? I don’t actually care if it’s on the heap, on the stack, or on a USB thumb drive in orbit around Pluto. That’s why I have a compiler, dude.
    > Requiring this sort of excessive attention to detail is a sure sign that a language has been designed by theoreticians rather than actual programmers, and it usually gets worse over time, not better (I don’t recall any instance where it’s ever gotten better, offhand).

    You seem to be fundamentally against the idea that systems programming languages (specifically, non-GC’d languages) should even exist (outside device drivers). That is a ridiculous claim — things that need a non-GC’d language are literally *everywhere*: operating systems, performance-critical networking software, large-scale systems with critical memory usage, and even high-performance video games.

    Yes, usability is a great thing, and that is why GC’d languages are so popular — they’re much easier to reason about, and have much more room to be ergonomic. But to claim GC-less is fundamentally broken just because GC is good is absurd.

  72. “One of the test boards zinc.rs targets is NXP LPC1768 which comes with a maximum of 512K ROM and 64K SRAM.”

    Yes, and?

    What’s zinc.rs, and why is that relevant?

  73. >What’s zinc.rs, and why is that relevant?

    Sorry, seems I forgot the context. zinc.rs is a project to write an RTOS stack in Rust. It currently has no C code, and has almost no ASM.

    1. >Sorry, seems I forgot the context. zinc.rs is a project to write an RTOS stack in Rust. It currently has no C code, and has almost no ASM.

      That’s a deployment that makes sense to me. In a way that using Rust as an application language or even for most other kinds of systems programming does not.

  74. Rust is being billed as a “systems programming language” but the standard library calls abort on OOM. I guess that’s just another critical and pervasive mistake in the language that’s waiting for some fairy dust and positive thinking to come along and fix it — like the other 100 or so glaring fuckups that are still in the language.

    1. >like the other 100 or so glaring fuckups that are still in the language.

      Can you be more specific about these?

  75. > I’d hate to imagine what string interpolation looks like in Rust. Does it even have string interpolation?

    format!("Welcome to {} jungle", "the")

    > When your language compares unfavorably in usability to quick hack jobs like PHP and Javascript, it’s a sign that something is very, very wrong.

    You can’t write a device driver in PHP, or any language with the same kind of memory model, for reasons that should be obvious. You can do that in Rust.

    Also, if the same program is written in both languages, the Rust program will probable outperform the PHP one by orders of magnitude.

    If PHP and JS are even up for consideration, then you don’t need to use Rust, for the same reason that I don’t commute in a tank.

    > Unless I’m writing a device driver or some such, which should I give a shit how my string is being stored?

    Do you even know why Rust exists? You’re supposed to be able to write device drivers in it. Rust has concepts baked in that are only applicable to low-priority level programming tasks because that’s what it’s for. It’s the same reason Go has syntax for CSP baked in, and PHP has so many ways to concatenate strings together.

    Comparing PHP and JavaScript to Rust is a bad rhetorical choice, because if PHP is up for discussion, then you probably shouldn’t use Rust, and you definitely don’t need to. The reason people still use C, despite it being a 20 year old language with terrible ergonomics and even worse theoretical underpinnings, is because there haven’t been any better alternatives. Unless you know of a language that has better ergonomics than Rust and can be run in an interrupt handler.

  76. Doctor Locketopus here’s how you concatenate two Strings in Rust:

    let mut hello = String::from(“Hello, “);
    let world = String::from(“world”);
    hello += &world;

    The only part that is ugly is that you have to write a method call to make a String. The reason is that Strings allocate, which is why it’s more verbose to use a String than a slice. This is a design choice.

    In the future, you might be able to do:

    let mut hello = “Hello, “;
    hello += “world”; //type inference can guess that hello is of type String here

    But this isn’t a high priority issue for the Rust team because it’s just saving a few characters, the program actually doesn’t act any differently.

  77. String interpolation in Rust works like this:

    let hello = “Hello”;
    let world = “world”;
    let hello_world = format!(“{}, {}!”, hello, world); //hello_world is now a String

  78. I’m a former colleague of Brian Campbell (hi!), and one of the relatively few people currently using Rust for production code. First, some background, so you know where I’m coming from: I’ve got a couple decades of C/C++ under my belt, a decade plus of professional Lisp/Scheme and several years of Haskell (hobby only). I’ve run Linux as my main desktop for a long time.

    I completely agree that the learning curve for Rust can be brutal, especially during the first week.

    The very first problem is the memory model: You need to know the difference between the types `Foo` (a value) and `&Foo` (a reference to a value). You need to know about `String` (owned strings which allocate memory), `&String` (a reference to a `String`) and `&str` (a string “slice” which does not allocate, basically a pointer and a length). Until you understand why Rust distinguishes between strings and string slices, you’re going to have a bad day. (Answer: Rust cares very much about making memory allocation and memory copying explicit. Never mind garbage collection, as far as Rust is concerned, even `malloc` and `memcpy` are expensive enough to be called out explicitly.)

    The second obstacle is the borrow checker. If you’re a senior C/C++ programmer, you’re relatively lucky: You’ll spend a week having a knock-down, drag-out struggle with the borrow checker, but once you’ve settled your differences, you’ll be friends. (If you’re a junior programmer who has never worked with pointers, stacks or heaps, it will take longer.) Once you make friends with the borrow checker, you’ll barely ever hear from it. And then a month or two later, the borrow checker will say, “That won’t work.” And you’ll respond, “Sure, it works. I know you can’t see it, but my superior human brain knows that this code is safe.” And then you’ll stare at the code for 5 minutes, and finally say, “Shit… You’re right. That’s a really bad and subtle bug, and it would only show up on alternate Tuesdays under heavy load, resulting in subtle memory corruption. This would take me weeks to reproduce, and days to analyze.”

    Once you figure out the memory model and the borrow checker, the remaining obstacles are mostly minor cultural stuff. One cultural thing, as you’ve discovered, is knowing that the Rust community relies very heavily on crates.io, and that there are several “semi-standard” crates that nearly everybody uses when needed. This happened more-or-less organically: The compiler team is very cautious about growing the standard library, but `cargo` and crates.io work surprisingly well, so that’s where the action happens. And once a version of a library is published to crates.io, it’s permanent, and you’ll be able to fetch the exact version you tested against years later. Plus the Rust community generally uses semver correctly, which makes upgrading dependencies less painful.

    So, those are the downsides (or at least the ones related to getting started). What are the upsides?

    The biggest upside is that Rust allows you to get very close to the hardware while maintaining strict safety rules for footguns. Imagine that you want to write a CLI tool that munges 250 MB of data per second per core on a modern laptop. You benchmark a bunch of old-school GNU C utilities, and many of them top out at 50 MB/sec on the laptop. To reach 250 MB/sec per code, you’ll going to need to avoid malloc and memcpy as much as possible, and do all your processing using pointers into raw I/O buffers. And you’re going to need to use threads. Of course, as every C programmer knows, this is dangerous. In Rust, it’s not necessarily easy, because you’ll be picking a whole new fight with the borrow checker, and you might need to use the lower-level I/O routines. But you can make it work in Rust, and it won’t segfault or have weird race conditions, and it will be pretty fast.

    Similarly, you can use Rust with no runtime at all, by writing `#![no_std]`. This means that you’ll lose the collection types and the I/O libraries. (But not the slice types or all the handy iterator APIs—it’s still a surprisingly good language.) But at this point, you can run in kernel space or even on bare metal, or you can build *.so files that work anywhere C would work.

    So those are the low-level benefits of Rust. I think these are the best reasons to _learn_ Rust. But if you already know Rust, there are a couple of reasons why you might want to use it for higher-level tasks.

    – Rust has that property which Haskell users describe as “If it compiles, it works.” In practice, this is a slight exaggeration. But if I spend 2 hours writing tricky Rust code and tweak it until it compiles, then there’s about a 90% chance it will work flawlessly on the first try. (At work, we have one minor internal service that’s extremely difficult to test locally. So once it compiles, I just deploy it immediately. Yes, I’m a bad person, but I trust the Rust compiler that far.)

    – The tooling is surprisingly good for a young language, at least as far as most “Unix people” will be concerned. Cargo is a great build tool and dependency manager. Rustfmt settles coding style debates. I have auto-completion in Emacs and Visual Studio Code. If I write CLI tools, I can assume that they’ll work cross-platform with minimal effort. The weak spot right now is IDE support, for people who like rich IDEs. (If you want an IDE, try the Linux version of Visual Studio Code with RustyCode and Rust racer; it’s all open source and has some basic IDE features, and it’s way better than you’d expect from Microsoft.)

    – Rust refactors really well. Now that I’ve climbed the learning curve, I can write Rust about half as fast as I write Ruby. But I can refactor Rust maybe 5 times faster than I refactor Ruby, with a lot more confidence. Once a project gets above 10,000 lines, this tends to be a good trade off. Above 25,000 lines, it’s a clear win.

    – Rust enforces a high degree of “data structure clarity”. If I’m working with complex data types, with lots of optional fields and union types, I’m going to be tempted to use Rust, because Rust’s `Option` type makes all the NULLs explicit, and the `enum` type (which is basically a tagged union) forces code to deal with all the possible values that might occur. This is handy when dealing with complex, badly-designed JSON schemas.

    So to sum up, I don’t think Rust is a good match for your project. If I understand your goals, you’re looking for green threads, async I/O and reasonable performance. You’re not especially worried about either GC overhead or malloc overhead. And—this is important—you’re not already up to speed in Rust. If you were already a Rust programmer, I’d say, “Try tokio-protocol. There are some rough edges but it looks really promising.” But nothing in Rust-land provides a sufficiently good solution to your needs for you to climb the learning curve right now. You almost certainly want to consider Go; it probably meets your performance requirements and it’s good at the kind of async I/O you want.

    To see Rust at it’s strongest, check out Philipp Oppermann’s tutorial on writing a kernel in Rust or look at BurntSushi’s ripgrep. For an example of a simple, fast Rust tool that I wrote for real-world use, check out scrubcsv, which took a couple of hours to write and which runs at a respectable 75 MB/s throughput on my laptop. (I know how to get it up near 250 MB/s, but that would take another half-page of code to eliminate malloc in the inner loop.)

  79. > The social question is whether they’re too drunk on their own kool-aid to actually fix these things.

    Given that a fair amount of the conversation on Reddit is about whether you (esr) should even be allowed to USE Rust under the terms of their “CoC” (because thoughtcrimes), I’d say no. That type of discussion alone would convince me to stay far, far away from this language.

    > You can’t write a device driver in PHP, or any language with the same kind of memory model

    That is utterly irrelevant to the point under discussion. I am talking about the language syntax.

    > the reason people still use C, despite it being a 20 year old language

    47, actually.

    > Unless you know of a language that has better ergonomics than Rust and can be run in an interrupt handler.

    I’ll bet you could do it in Erlang, if you put your mind to it.

    >Doctor Locketopus here’s how you concatenate two Strings in Rust:

    > let mut hello = String::from(“Hello, “);
    > let world = String::from(“world”);
    > hello += &world;

    That’s nice. Here’s how you do it in languages that someone would actually want to use:

    “hello, ” + “world”

    Or:

    “Hello, “.”world”

    Or:

    “Hello, ” ++ “world”

    Or something else along those lines.

    > But this isn’t a high priority issue for the Rust team because it’s just saving a few characters

    “Hello, ” + “world” is 19 characters. Your Rust example is 92. I don’t call a 4.8x increase in code size “a few”.

  80. “Hello, ” + “world” is 19 characters. Your Rust example is 92. I don’t call a 4.8x increase in code size “a few”.

    What exactly does “hello, ” + “world” mean? Does it mean you allocate a new string and copy the two strings’ contents into it? Does it mean you extend the buffer of “hello, ” and copy “world” at the end? Or does it mean that you create a new data structure, a rope say, that contains pointer+length spans that reference the two parent strings?

    Before you say “it doesn’t matter; just give me the fucking concatenated string!!!111”, stop and think: in memory-constrained environments it may damn well matter which you choose. The fact that Rust makes you stop and think about what you actually want from the machine, and be explicit about it, is a feature — not a bug.

  81. The social question is whether they’re too drunk on their own kool-aid to actually fix these things. Some of the more measured Rust advocates that have shown up here seem not to be.

    You really don’t understand how the hivemind works. The idea is to get the hypetrain moving for a promising solution first, then once hivemind buy-in occurs, crowdsourcing enough programming talent to make the solution complete becomes a doddle. This is why Red Hat committed to systemd, and then Wayland, long before either were ready for prime time: once the hivemind realizes that transition to these solutions is a fait accompli, it will put in the work to make them good. This is easily derivable from principles of open source development that you wrote about decades ago. In order to get to the point where Linus’s Law kicks in you need to have developer mindshare, and in order to get that you need good old-fashioned marketing. Which necessarily involves a bit of “kool-aid drinking” rhetoric.

    So here we are with Rust. It’s not finished yet, but most everybody knows that Rust will replace C in the coming years. The activation energy to develop an alternative systems-programming language and toolset that ticks all the checkboxes is way higher than the activation energy contributing to the Rust ecosystem.

    1. >So here we are with Rust. It’s not finished yet, but most everybody knows that Rust will replace C in the coming years.

      Oh, bullshit. Your own “activation energy” argument points at Go, which is an easier transition from C and more mature.

  82. Doctor Locketopus wrote:

    That’s nice. Here’s how you do it in languages that someone would actually want to use:

    “hello, ” + “world”

    Here’s how you do it in C (I hope there are no bugs):

    /* Create a malloc'ed string from the input string. */
    char *string_new(const char *s) {
        size_t len;
        char *result;
        len = strlen(s);
        result = (char *) malloc(len+1);
        if (result)
          strncpy(result, s, len+1);
        return result;
    }

    /* Append s2 to s1, reallocating s1 as necessay. */
    char *string_append(char *s1, const char *s2) {
      size_t s1len, s2len;
      char *s1new;
      s1len = strlen(s1);
      s2len = strlen(s2);
      s1new = (char *) realloc(s1, s1len + s2len + 1);
      if (s1new)
        strncpy(&s1new[s1len], s2, s2len + 1);
      return s1new;
    }

    int main() {
      char *s;
      s = string_new("hello, ");
      s = string_append(s, "world");
      printf("%s", s);
      free(s);
    }

    You’ll notice that my C program has two string types: owned char*, and regular char*. If it’s an owned char* you can append to it using realloc and you must delete it using free. Regular char* is best treated as a read-only view into somebody else’s string.

    Here’s how you do it in C++ and Rust (without the in-place append this time):

    // C++
    std::string("hello, ") + "world"
    // Rust
    String::from("hello, ") + "world"

    This is the same idea as the C code: We have “owned” strings (std::string and String), which support appending. And we have string literals (const char * in C++ and the slice type &str in Rust). C++ and Rust make this distinction because they want to make memory allocation explicit.

    If your response to this is “What an ugly nuisance. Who cares?” then C, C++ and Rust are all poor choices for your project. But if you need precise control over memory allocation, then you’ll want a language which makes this distinction.

    There’s an extra wrinkle affecting Rust. If you have two owned strings, you can’t pass them directly to + without a tiny change:

    // C++
    std::string("hello, ") + std::string("world")
    // Rust
    String::from("hello, ") + &String::from("world")

    The Rust example contains an extra &. This is necessary because passing a parameter by value in Rust transfers ownership, which—in this case—would consume and deallocate the second string, forcing you to pay for an extra free. So in Rust, the current implementation of Add takes arguments of type &mut String and &str. The first is a mutable reference to an owned String object, and the second is a string slice that we can read data from but not change. There’s one bit of magic in Rust which transforms the &String to a &str for you using the Deref trait. This is pretty much the only implicit magic in Rust, and I’m still not entirely convinced it was a good idea.

    This particular case could be fixed by about 5 lines of code in the standard library, if people philosophically wanted you to be able to omit that extra &. But right now, the community preference is to make memory management explicit and visible. And even if this issue was fixed here, you’d still have to understand the general ideas to work in Rust.

    I think of this as the “Rust tax”: Your code will contain a bunch of extra & and .to_owned(), and sometimes even the ugly &* when somebody got too clever with type parameters. Rust makes memory handling visible, whether you want it or not. If you don’t want it, then Rust is a bad match for your problem.

  83. @Doctor Locketopus

    > There is no justification for a MODERN language to require using a bunch of ugly method calls to concatenate two strings. That is such a fundamental (and common) task that there should be syntax (or at least syntactic sugar) for it.

    I fail to see how there are any “ugly method calls” when handling Strings. All a String requires is that following arguments are of the &str type so that the original String may copy the &str’s data into itself without destroying the content of the other String from whence the &str came from.

    Additionally, converting an &str into a String via String::from(), to_owned(), or to_string() does not take much effort. The &str type cannot grow, and it is immutable, so naturally you cannot add anything directly to it. That &str could be a reference to a heap-allocated string or it could be located within the binary data. The only way to safely add two &str together is to add them into a heap-allocated String in a separate location of memory.

    > I’d hate to imagine what string interpolation looks like in Rust. Does it even have string interpolation?

    Comments like these are just proof that you’re trolling. It’s taught in Chapter 1 of the book that string interpolation is as simple as:

    let interpolated = format!(“ABC {} GHI {}”, “DEF”, 123);

    > What “large projects” have actually been written in Rust? Any? Can you point to any actual data showing lower debugging and maintenance costs, or is this merely an opinion?

    Redox and Servo are the top two. I’ve written some projects myself and can verify that debugging has never been easier than it is with Rust. Rust catches all the gotchas early in the design of the software so that you don’t make those mistakes later. Your IDE is quick to highlight areas of issue right after you save due to the compiler’s lints (at least with Atom), which can be expanded with Clippy. Any issues you have from that point will merely be logic errors that can easily be fixed by creating unit tests as you go.

    @Jay Maynard

    Seems you haven’t heard of Redox or similar projects. In Rust, you may also drop the standard library and write software at a lower level than C+stdlib. Rust will fit in all the same places that C can.

    > Craig T

    If you’re using the ralloc memory allocator, written in Rust and used by Redox, you can configure the behavior of OOM errors, among a number of other useful features that the standard glibc isn’t capable of.

  84. Oh, bullshit. Your own “activation energy” argument points at Go, which is an easier transition from C and more mature.

    Go has an unsound type system and a garbage collector. Also, in general, it’s much more difficult to exercise the explicit control over how memory is used in Go than it is in C, C++, or Rust.

    Maybe you don’t need that. From the sound of things you would almost be as well off writing ntpd in Node.js.

  85. I’ve used Rust for the last couple of months for my toy projects.
    The language itself is fine if you don’t mind the quirky syntax (see string handling). It feels like Haskell meets C++ (in a good way). You get the speed of C++ and the high level abstractions of Haskell. Words cannot describe how great the Option and Result types are compared to exceptions or error codes. The Option type contains a value or is empty. The Result type contains a value or an Error. Iterators, variable shadowing, the from trait are but a few great features Rust has.
    However, Rust faces a major problem in my opinion: the community. I don’t mean the SJW part (which is a completely different story), but the lack of direction. Like ESR pointed out, the Rust ecosystem lives in crates maintained by god-knows-who. For example: searching for XML brings up a bunch of crates which are all in their 0.x phase. Which one do you pick? The one with the most downloads? The most actively developed one? I feel like such trivial features should live in the Rust standard library like Go has. Another example is the segmentation between the stable and nightly compiler. Libraries like Diesel (ORM), Clippy (static analysis) and Rocket (web framework) all require you to use the nightly compiler. Seriously? Do you really expect me to use a nightly compiler to build a library? The lack of direction can also be found in the programming language itself. There’s no BDFL or a board to steer it, it’s just an RFC posted on GitHub and, again, god-knows-who that shines their light on it.
    As much as I hate Go, at least they got one thing right: the feature rich standard library.

  86. @Jay Maynard > On embedded controllers with 512K (for the later version) of ROM and 128K of RAM.

    Luxurious! I envy you. Still trying to wrap my head around how that guy managed to get LISP running in 2k of ram for Arduino. ;)

  87. Still trying to wrap my head around how that guy managed to get LISP running in 2k of ram for Arduino. ;)

    The first implementation of Lisp was a set of subroutines called by FORTRAN (I assume they were in assembly) running on a vacuum tube computer. And it was generally resource constrained for a long time, so running on tiny systems is in its DNA. Although 2KiB writable data space does sound a tiny bit too small, at least to do development in REPL style, instead of putting all your code in read-only flash….

  88. @Craig > Rust is being billed as a “systems programming language” but the standard library calls abort on OOM. I guess that’s just another critical and pervasive mistake in the language that’s waiting for some fairy dust and positive thinking to come along and fix it — like the other 100 or so glaring fuckups that are still in the language.

    Yep, that is the kind of hair raising stuff I expect from the Ruby on Rails crowd. Well meaning, earnest…

  89. > Libraries like Diesel (ORM), Clippy (static analysis) and Rocket (web framework) all require you to use the nightly compiler.

    Diesel works on stable as well, but Clippy and Rocket require compiler plugins which aren’t a stable feature.

    > Yep, that is the kind of hair raising stuff I expect from the Ruby on Rails crowd. Well meaning, earnest…

    Remind me, who are these people from the “RoR crowd”? Klabnik? He was hired to write the docs relatively recently.

    Either way, if you don’t like the default allocator’s behaviour you can write your own. (There’s an example on this towards the end of the book.) Anyway, while I certainly agree that it would be nice to have the default allocator handle OOM situations more nicely (though it’s rarely a problem on most systems Rust is likely to run on these days due to overcommit generally being the default), it is hard to handle it in a well-working and sound way that doesn’t require going back to calling malloc and free everywhere.

  90. Oh, and him and Katz both joined the core team just over two years ago, and none of the things people have raised as issues in the comments have really been related to either of them.

  91. ESR:

    > Oh, bullshit. Your own “activation energy” argument points at Go, which is an easier transition from C and more mature.

    Go will absolutely win anywhere you want a simple, GCed language and green threads, and where you don’t care about (1) type checking strong enough to eliminate unexpected NULLs or (2) generic types that allow you to implement MyContainer<T>. Also, you have to be able to live with if err != nil checks everywhere, though it doesn’t bother some people and anyway, they could fix it trivially by adopting the ? syntax that other exception-free languages are using in this space.

    But I wouldn’t use Go to add new features a random C library like librsvg, because it would require a runtime and a GCed heap. I wouldn’t write a garbage collector or kernel driver in Go. There’s a whole space of low-level stuff where you really don’t want a runtime or a GCed heap, but Go has (rightfully) decided to target one level up, because it’s a bigger market. And Go is already doing just fine in that space.

    Rust is most compelling for two groups of people:

    1. People who don’t know Rust, but who want a low-level systems language (no GC, no runtime) with high-level abstractions, solid tooling, and strong safety guarantees. And they need to be able to afford to burn a week or two on learning curve. This is almost certainly a smaller niche than Go, but it’s where a lot of critical infrastructure lives.

    2. People who already know Rust and who want a compiled language. Seriously, now that I’m up the learning curve, I write Rust code about half as fast as I write Ruby code, and the gap is still narrowing. At this point, if I need to bang out a fast command-line tool or a native module for Node.js, I just reach for Rust, and it’s a total blast. But I could never justify learning Rust for any of these projects, if you see what I mean.

    Probably by the end of this year, my default stack will be TypeScript+Node for “scripting” stuff, and Rust for compiled stuff. Modern JavaScript is actually a pretty good language, and TypeScript means that if it compiles, the odds of it working are very good. It feels really weird to have a couple of open source Microsoft tools in my Linux stack, but they’re popular and they work. But it’s taking me a few weeks to page the Node ecosystem back into working memory. There’s an infinite number of Node.js libraries available through NPM, but a lot of the popular ones are just awful.

  92. “And they need to be able to afford to burn a week or two on learning curve.”

    This seems to be a great big YMMV. For some folks, it’s a week. For others, it seems to be several months.

  93. > Rust is being billed as a “systems programming language” but the standard library calls abort on OOM.

    It also calls abort when stack allocation generates an OOM. Guess what C does when it runs out of stack space.

    Or, if you prefer to focus specifically on heap allocation, what most C programs do when they run out of heap space. GLib aborts explicitly. Git will also abort after trying to free the pack cache. Qt Extended monitors the amount of memory available in the system and tries to preemptively avoid running out of memory. So does Android (I know that Android is technically Java), lighttpd will just dereference the null pointer, and . Running out of memory on a multitasking system just doesn’t work that well, simply because of the nondeterministic “last allocator loses” behavior of it; if you’re a hard-real-time or system-critical process, you should allocate all the memory you need at start and never touch the allocator again afterward.

    1. “Most programs” aborting and a language runtime aborting are not comparable at all. This is why Rust adoption is so low — when faced with glaringly obvious and unnecessary flaws, the Rust crowd immediately starts with whataboutism.

      What about GLib…?
      What about Git…?
      What about Qt…?
      What about Android…?

      Those projects were all given the choice and decided that aborting was acceptable for their specific circumstances. Having the language runtime abort from under you takes away that choice and in many cases aborting is categorically the WRONG choice, no matter how much whataboutism you throw around.

  94. > if you’re a hard-real-time or system-critical process, you should allocate all the memory you need at start and never touch the allocator again afterward.

    I’ve done this in Rust with #![no_std] and programming directly against the core library, which is normal on an embedded system. Obviously it’s frustrating to lose Vec and String, but Rust works surprisingly well without a heap at all. If you just need some one-shot allocation at startup, you can write a custom allocator. But what you really ought to do in kernel space is to write your own collection types that return errors on allocation failures. The Linux kernel reimplements anything it needs from libc; there’s no reason why a Rust kernel couldn’t do the same thing.

    Personally, I’m OK with Rust’s std panicking if malloc fails. Linux has already broken malloc’s return value, and checking every allocation for a NULL return value is tedious everywhere except kernel space and in very tiny, specialized programs. Still, somebody should write a crate for this. :-)

    > For some folks, it’s a week. For others, it seems to be several months.

    Yup, I can believe this. The Rust learning curve consists of several things:

    1. Memory, pointers, stacks and heaps. Much of Rust’s design will ultimately make sense if you already understand these things, but it’s hard to justify otherwise.

    2. Designing code around a “single ownership” principle, with a one mutable borrow or multiple immutable borrows. For some people (and programs) this comes naturally. But for others, it’s a big adjustment. For example, if you’re a newbie Rust programmer, avoid doubly-linked lists and graph processing at all costs: You can do these, and even write nice libraries for them, but they’re genuinely hard if you’re still internalizing ownership.

    3. Basic functional programming. If you’ve ever used Ruby or Lisp or JavaScript with lodash, you’ll be fine. But if you’ve never used map or reduce with a custom function argument, things will be a bit weird. Bonus points if you’ve ever used a language with algebraic data types (basically tagged, type safe unions) and a match statement.

    4. Specific Rust tips and tricks, like you’d expect from any language, and some knowledge of where to look for libraries.

    If you have no experience with either (1) or (3) at all, Rust is going to be hard and alien. (2) is going to be a pain in the neck for everybody, and (4) is inevitable with any language.

    As a production Rust user, if I had to hire somebody and teach them Rust, I’d currently look for an above average programmer with experience in one of {C, C++} and one of {Ruby, Lisp, a strongly-typed functional language}, and I’d expect to pair with them a lot in the first week or two. After that, I’d expect to answer occasional questions on Slack.

    Probably ESR could get up to speed in Rust in a week or two if he had a problem which was an especially bad match for Python or Go. IIRC, he has extensive experience with C and Lisp, so there’s nothing really alien in Rust except maybe ownership. People with our host’s profile tend to pick up Rust in a week or two if they’re motivated. But if you already know Go, like Go, and if you’re happy using it for your current project, I can’t see the point of learning Rust. If you’re chronically grumpy about Go’s error-handling and lack of generics, or if you secretly wish for a functional language, or if you just need raw zero-copy speed, then Rust might be tempting.

    Me? I have heavy experience with C++, Lisp and Haskell, and I’ve made friends with borrow checker. Rust is crack and my boss needs to stage interventions to make me stop coding and sleep.

    1. > Linux has already broken malloc’s return value,

      No it hasn’t. There are many, many situations and configurations on Linux where malloc still returns NULL.

      People who don’t check for NULL returns from malloc (“because muh overcommit hurrr”) are an absolute cancer.

  95. >stop and think: in memory-constrained environments it may damn well matter which you choose.

    If memory is so constrained that I actually give a fuck about the amount of space taken up by “Hello, world”, I’m going to be using asm, not Rust.

    > I fail to see how there are any “ugly method calls” when handling Strings.

    If really you don’t see why it’s preferable to write:

    “Hello, ” + “world”

    to:

    let mut hello = String::from(“Hello, “);
    let world = String::from(“world”);
    hello += &world;

    I seriously question whether you have written enough actual code in your lifetime to have a valid opinion on the matter.

    Why stop there? Why not go whole hog and put the burden of assigning the individual CPU registers back on programmers, too? That would be maximally efficient, and programmer time is free, right?

  96. > But if you need precise control over memory allocation, then you’ll want a language which makes this distinction.

    As I noted above, I suppose this means we can expect the “register” keyword to make a reappearance in Rust at some point in the future.

  97. > The first implementation of Lisp was a set of subroutines called by FORTRAN (I assume they were in assembly)

    On the IBM 704, CAR and CDR were each fifteen bit wide slices of a raw machine word, so, yeah. :-)

  98. Doctor Locketopus, if you have two static &str you don’t need to convert BOTH of them to Strings, I was showing how to add two Strings.

    If you want a &str you do this:

    let hello_world = concat!(“Hello, “, “world”); //&str

    If you want a String you do this:

    let mut hello_world = String::with_capacity(15);
    hello_world += “Hello, “;
    hello_world += “world”;

    notice I actually allocated enough capacity so that the String doesn’t have to re-allocate when “world” is added. This is the whole point of Rust, it encourages the programmer to write efficient code.

    But it seems you’re really stuck on the number of characters that you have to write. This is a weird mindset because a programmer only writes maybe a few hundred lines of code a day. From a pure speed perspective, it only takes a few minutes to TYPE what you commit in a day. It takes the rest of the time to actually think how to approach a problem, solve it, and to test that your code works.

    But please tell me, what should “Hello, ” + “world” even yield, a String with a capacity of 12, or a &str that’s static? Those are not the same types in Rust or C++

  99. Eric Kidd: I’m used to working without a heap. There is no malloc or the like anywhere at all in my code. (And when I’m building HTML on the fly in a (statically-allocated) buffer to be fed to uip’s HTTP server, there’s a ton of calls of the form d=Append(d, "...");.) I spend lots and lots of my time dealing with pointers.

    (And I’ll excuse C’s lack of a string concatenation operator, because C doesn’t have strings as first-class pieces of data to begin with.)

    But I don’t know a thing about functional programming. It’s always struck me as something beloved of computer scientists but with limited practical application in the real world.

    My weapons of choice are C and Python. I can slog my way through C++, though I despise it viscerally. In former lives, I’ve spent years working in assembler and even a little PL/I.

    Guess Rust would be a loss for me, then…

  100. On the IBM 704, CAR and CDR were each fifteen bit wide slices of a raw machine word, so, yeah. :-)

    Contents of the Address and Decrement Registers. One reason we keep them instead of wholesale changing to first and rest is that they compose, e.g. the 2nd Lisp Machine design and the first to go into production was dubbed the CADR, which is (CAR (CDR list)), or the 2nd element in a list.

  101. > I finally cleared my queue enough that I could spend a week learning Rust.

    IMHO, learning Rust in one week is unrealistic. If you had very strong C++ and ML background, perhaps you could learn it in a couple weeks or so, but otherwise it is likely to take much longer.

    > I was evaluating it in contrast with Go

    You’re comparing apples and oranges here. Those languages have been designed with very different priorities in mind. Rust tries to compete with C/C++, while Go tries to be as easy to use as Python while providing efficiency similar to Java. Consequently, Rust emphasis choice and control, while Go opts for simplicity. Consequently Rust opts for manual memory management (similar to C++ but safe by default), while Go relies on GC. Similar to C/C++, Rust provides bare minimum functionality in the _standard_ library and expects programmers to use some third-party libraries accordingly their needs. Moreover, the whole standard library is optional, so you can compile your program to run on bare metal. Go heavily relies on runtime, and it uses many abstractions that have noticeable runtime cost, which is unacceptable for a language that wants to compete with C or C++. For example, Rust used to have “green threads”, which are similar to Go routines, but this feature was removed because it would cause some runtime overhead for code that does not use green threads. This is just one example of many trade-offs that were resolved differently.

    > Even things that should be dirt-simple, like string concatenation, are unreasonably difficult.

    The Rust book shows how to do that, and it is certainly much easier to manipulate strings in Rust than in C. So I am not sure what is your problem with strings. Moreover, making concatenation strings too easy may be not so good idea, because it may hide memory allocation, which has noticeable runtime cost. After all, the primary goal of Rust is to encourage to write safe and efficient code, and runtime overhead that is not obvious while looking at the source code should be avoided.

    > We do not currently have an epoll/select abstraction.

    Neither C nor C++ provides this abstraction. You have to either use some platform specific functions (like epoll on Linux) or download and use a third party library, which provides a cross-platform abstraction. If anything, Cargo makes it much easier to download and install third-party libraries. This is a real improvement over C/C++ where you have to manage dependencies manually. Of course, discoverability of packages could be better, but there is nothing wrong with asking people on IRC.

    > The Rust community appears to have elected to use the decentralized nature of their crate system

    I think most arguments that you used in “The Cathedral and the Bazaar” can be applied here. A decentralized approach offers many advantages, but you need to communicate with the community to take advantage of that.

    > Which is wonderful in theory but seems to be failing in practice.

    Things tend to be failing when you do not understand basics, and it takes time and effort to learn something new, especially when it comes to something very different than what you used to. Also more powerful and flexible systems tend to require more learning. For example, it takes more time to learn Git than Subversion, but then you can do a lot of more things with Git.

  102. To add to my last post, if you know all of the Strings ahead of time and they’re all static, you can just write:

    let hello_world = String::from(concat!(“Hello, “, “world”));

    There are only two things that can be improved from a syntax standpoint:

    1. “Hello, ” + “world” could actually be syntax sugar for concat!(“Hello, “, “world”)
    2. Type inference could allocate a String when you need one, and leave it a &str if you don’t

    I think only the first point would be pretty non-controversial and easy to implement. Then you could write

    let hello_world = String::from(“Hello, ” + “world”);

    which wouldn’t be too bad

  103. “let hello_world = String::from(concat!(“Hello, “, “world”));”

    Not responding to anyone in particular …

    With all the explanations and apologetics for Rust (e.g. string concatenation), it brings to mind the Alan Kay quote “Simple things should be simple, complex things should be possible.”

    It appears there are no simple things in Rust, that the case where someone just wants a String and would like to extend it and is not particularly concerned about how/where that is done is a case Rust cares not at all for. Certainly it is true that Rust can handle the the complex things, but the simple things won’t be simple.

  104. Jay Maynard:

    > I’m used to working without a heap. There is no malloc or the like anywhere at all in my code

    Good times. :-)

    > But I don’t know a thing about functional programming. It’s always struck me as something beloved of computer scientists but with limited practical application in the real world.

    At this point, basically all the popular scripting languages except Python (JavaScript, Ruby, everything in Microsoft land) have an increasingly strong functional component. I’m thinking of map, and anonymous functions that capture their environment, and stuff like that. Python can do functional stuff, but Guido has never really been enthusiastic about it, so Python’s the major holdout among scripting languages. JavaScript has gone especially far in a functional direction. Overall, we’re talking about millions and millions of real-world programmers here, if we use “functional” in the watered-down sense—and that’s all it takes to reduce the Rust learning curve a bit. But it’s far from everyone.

    Rust is more or less what you’d get if you said “Can I have a C++ without any footguns?” or “Could I have a practical ML-family language that doesn’t need a garbage collector or runtime?” and experimented until you found something that was a compromise between those two goals.

    > My weapons of choice are C and Python. I can slog my way through C++, though I despise it viscerally.

    If you had said that you had a “complex love-hate relationship with C++”, then I would’ve encouraged you to try Rust. But if (for example) you despise C++, prefer Python over Ruby, really enjoy programming in Go, and could never see the point of even a watered-down functional language, then Rust may just never be to your taste.

    Personally, I feel much the same way about Go. I really like it on paper. When I write anything less than 1,000 lines of Go, I enjoy myself. But as my program grows, so does my frustration. The if err != nil business slowly drives me nuts. I write more and more boilerplate code that could be avoided with generic types. I’m sad every time nil appears somewhere the type system should have prevented it. From a strictly personal perspective, the ergonomics are subtly wrong; there’s a mismatch between between thinking and coding. This doesn’t mean Go is a bad language—many extremely talented programmers with excellent taste like it a lot—but it means I’m just as likely to reach for JavaScript instead. I do, however, submit Go code to other people’s projects!

    I can totally imagine the a non-trivial number of programmers will feel the same way about Rust. Personally, I love it, and I’m extremely productive in it. But there’s this whole Rust aesthetic mixing careful control over memory and performance with safe, high-level abstractions. If you aren’t at least a little bit seduced by both halves of that idea, I’m not sure that it’s necessarily worth climbing the Rust learning curve. Rust is a smaller language than C++, but I’ve never heard of anybody—no matter what their talent or background—truly feeling at home in less than a week. Fundamentally, the combination of memory safety without garbage collection is new, and it requires subtly different programming techniques that trip most people up for a while.

  105. Michael:

    > It appears there are no simple things in Rust, that the case where someone just wants a String and would like to extend it and is not particularly concerned about how/where that is done is a case Rust cares not at all for.

    I think this is actually a fair observation. If you fundamentally don’t care about the difference between “a String that I own and can use as I wish” and “an &str slice pointing into somebody else’s string”, then Rust might be a bad match for your problem. At the very least, Rust forces you to think for half a second to decide, and the distinction has consequences that ripple through most APIs in the system. Once you realize the underlying principle, the details are generally pretty easy.

    What I can say is that the “Rust ownership tax” is both finite—you only need to learn a certain number of things and you’re basically done—and not hugely burdensome in practice once you internalize it. Like I said, I code Rust at about 50% the speed of Ruby at this point, which is not bad for a systems language.

  106. “But it seems you’re really stuck on the number of characters that you have to write.”

    Yeah, I am “really stuck” on that. Not only because it means the difference between tedious drudgery and a language that is actually pleasant to use, but because software development cost (and number of bugs) per line of code are proportional. More code = more cost and more bugs.

    Note that this is almost the opposite of the popular (in some circles) method of measuring software *productivity* in terms of lines of code.

    If language A requires twice as many lines of code to do the job as language B, chances are writing your code in language A is going to cost twice as much and have twice the number of bugs.

    “But please tell me, what should “Hello, ” + “world” even yield, a String with a capacity of 12, or a &str that’s static?:

    Please tell me why I should CARE, any more than I care which specific CPU registers are being used to perform the computation, or exactly where the operands are in the system’s memory (are they in the on-chip cache? In an external cache? In RAM? Paged out to disk?). Why doesn’t Rust require you to specify those things too, if it’s so all-fired bare-metal?

  107. > […] the case where someone just wants a String and would like to extend it and is not particularly concerned about how/where that is done is a case Rust cares not at all for.

    That is exactly right. If you really don’t want to care about allocation, then use any GC’d language you prefer.

    If you want to absolutely go to town and allocate everything yourself, use C.

    Rust is in the middle. Allocation and mutability and ownership matter in Rust, but the compiler is doing the bookkeeping for you. You can’t shoot yourself in the foot, though you can still paint yourself into a corner and end up with something awkward. (Eric Kidd’s suggestion to avoid graph processing in your first Rust program is spot on. Graphs are super easy when you have garbage collection, not so easy otherwise.)

    Furthermore, this was an explicit design goal from the beginning; it’s Rust’s desired state of being. You shouldn’t feel any need to justify choosing to use a garbage-collected language over Rust, because not everybody is writing a soft-realtime system. Or maybe you’re NASA or Casey of handmadehero.com fame (I’m being slightly facetious, but you should really look that one up anyway) and you can actually write a complex program in C without shooting yourself in the foot.

    Personally the largest independent program I wrote in the last month is in Bash; Rust wasn’t even in the running for that one.

  108. > If memory is so constrained that I actually give a fuck about the amount of space taken up by “Hello, world”, I’m going to be using asm, not Rust.

    > If you don’t understand why it’s preferrable to type “Hello, ” + “world”

    Personally, I think it’s preferable to type “Hello, world” (note the lack of a plus sign).

    Actual programs are doing this stuff in a loop, with dynamic input, so it matters a great deal what the cost is. The way string concatenation is done can make the difference between quadratic time and amortized linear time.

    this version runs in O(n^2) time. You can tell because it’s calling operator delete inside the loop (asm lines 96 and 147; the former is called on repeat, and the latter is called in the tail end of the loop), and the string constructor (asm line 107):


    #include <string>
    #include <vector>
    using namespace std;
    string concat_together(vector<string>& input) {
    string ret_val = "";
    vector<string>::iterator e = input.end();
    for (vector<string>::iterator i = input.begin(); i != e; ++i) {
    ret_val = ret_val + *i;
    }
    return ret_val;
    }

    this version runs in amortized linear time. Interestingly, the assembler in the first version is 174 instructions long, while the second is only 54; it’s literally just a call to the append function in a loop.


    string concat_together(vector<string>& input) {
    string ret_val = "";
    vec<string>::iterator e = input.end();
    for (vec<string>::iterator i = input.begin(); i != e; ++i) {
    ret_val.append(*i);
    }
    return ret_val;
    }

    In contrast, both of these Rust functions will run in amortized linear time:

    fn concat_together(input: &[String]) -> String {
    let mut ret_val = String::new();
    for i in input {
    ret_val = ret_val + &i[..];
    }
    return ret_val;
    }

    In fact, their asm ends up almost identical.

    fn concat_together(input: &[String]) -> String {
    let ret_val = String::new();
    for i in input {
    ret_val.append(&i[..]);
    }
    return ret_val;
    }

    The Rust assembler is … hard to read, because rustc inlined memcpy into this function, so the logic of evaluating the loop itself is smooshed together with a partially unrolled SIMD-accelerated implementation of memcpy. The smoking gun, as it were, is that the only call to __rust_deallocate is inside the unwind table (the string buffer needs to be freed in case an exception is thrown a panic occurs). Because String::new doesn’t actually allocate anything, there ends up being a call to __rust_allocate inside the loop body as well.

    In any case, it’s not copying the entire thing to a whole new string every single time it iterates.

    The equivalent to the O(n^2) C++ version is here:

    fn concat_together(input: &[String]) -> String {
    let ret_val = String::new();
    for i in input {
    ret_val = ret_val.clone() + &i[..];
    }
    return ret_val;
    }

    Don’t use this version. It wastes CPU cycles and memory and it’s the most unreadable one out of the three.

    BTW: I’m using <pre>. Why is the indentation being stripped?

    BTW2: Neither “Hello, ” + “world” nor my concat loops are perfectly realistic, but I think the latter is more realistic than the former. It’s certainly more interesting. I would profile this to see if I could make it any faster, but there’s already a concat() in the standard library that does this, so it’s kinda pointless.

  109. @db48x

    “If you really don’t want to care about allocation, then use any GC’d language you prefer.”
    “If you want to absolutely go to town and allocate everything yourself, use C.”
    “Rust is in the middle.”
    “Furthermore, this was an explicit design goal from the beginning”

    Understood. But this would seem to put Rust in a fairly narrow niche. IMHO, Go has shown that a language with the easy expressiveness of Python/Ruby can in fact approach C-like performance and can therefore occupy a fairly “tall” slice of the language hierarchy. IMHO, few C aficionados will ever give it up for Rust. And no-one will choose Rust where Go would work fine for all the reasons stated in this thread. So does that leave Rust with sufficient territory to really sustain it?

    Stated differently, would it hurt Rust that much to allow for the “simple things” and therefore make itself more useful to a broader (er, taller) base?

  110. Michael:

    > IMHO, Go has shown that a language with the easy expressiveness of Python/Ruby can in fact approach C-like performance and can therefore occupy a fairly “tall” slice of the language hierarchy.

    My use case for Rust does not involve approaching C-like performance. :-) I have literally tens of billions of dirty records to clean and transform, and probably over a trillion values. As far as I’m concerned, old school C apps like the GNU CLI tools and/or PostgreSQL’s query engine are where performance goes to die.

    Now, it’s certainly possible to write C code that’s much faster than typical old-school C tools or PostgreSQL. (GNU cat, for example, is ridiculously fast. So is git.) But as you well know—given the projects you mentioned up thread—to make C fast, you usually need to use lots of pointers into buffers and you need to be very careful. So eventually most C programmers get lazy, and settle for malloc and single threading.

    Even my quick-and-dirty Rust code is usually a fair match for C. But if I’m willing to have a long, philosophical argument with the borrow-checker, it will allow me to make extensive use of vector slices and threads, and I’ll probably be able to match aggressively hand-tuned C without sacrificing safety. As I mentioned before, the best public example of something like this is ripgrep. (Seriously, everybody who uses grep -r should try out ripgrep.)

    To be fair, this sort of heavily optimized Rust code is far from easy to write. If you’ve already made friends with the borrow checker, you’re sacrificing that hard-won amiability. Also, Rust is still very weak with respect to MMX intrinsics, which is a limiting factor for some programs.

    > Stated differently, would it hurt Rust that much to allow for the “simple things” and therefore make itself more useful to a broader (er, taller) base?

    It’s certainly possible to fix some individual rough edges in Rust. But overall? Rust is trying to provide strict, provable memory safety without a GC. And that comes at a price: You need to know about pointers, stacks, heaps and ownership, and you need to scatter the occasional &, .to_owned() and .clone() throughout your code. To me, this seems like a fair price to pay, but not everyone will agree.

    When I code in Rust, I split my screen down the middle. Emacs fills the left half, and a terminal running cargo watch fills the right. I type code, and the Rust compiler fills my terminal with errors. I insert an occasional & or fix some stupid mistake. When cargo watch finally goes green, I’m pretty sure my program works. When my coworkers watch me code in Rust, they say, “It’s like you’re having a conversation with the compiler.” Rust has some pretty nice error messages, but even so, it helps to be familiar with common problems and how to fix them.

    Rust basically takes the whole burden of debugging and testing your code, and shifts 90% of it back to compile time. You’ll fight to get your code to compile. But once you make the compiler, nearly all the remaining errors will be conceptual ones.

    The interesting question is what happens if you do make it up the Rust learning curve for some reason. Once you understand how memory works, how to appease the borrow checker, and how to write elegant Rust code, the language actually works surprisingly well for a broad range of tasks. The niches where Rust is overwhelmingly compelling are relatively narrow (especially this early), but once you know the language, the niches in which it’s an effective tool are surprisingly large.

    But again, keep in mind that Rust is an unusually good match for my brain. I’ve always wanted a cross between C++ and Haskell with Ruby’s library system and strict safety rules for footguns. Rust gets me about 80% of what I want and I’m drunk on code, having shipped three production Rust tools in the last month. (A CLI tool, a multithreaded server, and a Node.js extension.) Sleep? What’s sleep?

  111. Doctor Locketopus, you care because Rust makes a distinction between owned things and borrowed things. &str is borrowed, while String is owned.

    It’s not about performance, it’s about whether it’s safe to share said thing between multiple threads. It’s about safety in the general case. You can easily write yourself into a data race in Go, while such a thing in Rust is prevented by the compiler.

    I would use Rust in any project where I want to deal with concurrency or parallelism over Go. Even if it takes me longer to write the thing in Rust, I’m less concerned about these hazards so I’m more sure that whatever I wrote actually works every time.

  112. “Personally, I think it’s preferable to type “Hello, world” (note the lack of a plus sign)”

    Except that doesn’t actually, you know, concatenate two separate strings. Other than that, great point!

    (followed by a tedious lecture regurgitating material that most of us saw in Introduction to Algorithms decades ago).

    What is it with you guys and your incessant lecturing on basic concepts? Did you just learn about computational complexity like, a year ago in CS 200, and now feel the need to share your EXPERT GENIUS LEVEL KNOWLEDGE with the world?

    Sheeesh.

    Judging from your picture, I would wager that I wrote my first compiler before you were even born, and learned about algorithmic complexity years before that.

  113. > You can easily write yourself into a data race in Go, while such a thing in Rust is prevented by the compiler.

    If I’m worried about that, I’m going to use Erlang, or Clojure, or something else that manages to handle data integrity issues while somehow, magically, at the same time, manages to have syntax that lets you concatenate strings without requiring half a page of code.

  114. Doctor Locketopus says “If memory is so constrained that I actually give a fuck about the amount of space taken up by “Hello, world”, I’m going to be using asm, not Rust.”

    No, you’re not. You’re going to use C, because C is little more than portable assembly language. As somebody who watched his assembly language library get thrown away in favor of a C library doing more or less the same thing, yeah, if you write it in asm, then you deserve all the pain you will suffer.

  115. Michael Howell says “Or, if you prefer to focus specifically on heap allocation, what most C programs do when they run out of heap space.” It’s possible to write C programs which pause until more memory becomes available. For example, qmail doesn’t crash when it runs out of memory.

    So, really, your objection is not to C, the language, but the C library and programmers who don’t care if their program crashes, or loses data, or silently fails to produce the right results simply because it has run out of memory.

    You now have an example of C running reliably in the face of a lack of memory.
    If you want to be “critical infrastructure”, then THAT is your target.

  116. > I’ll bet you could do it in Erlang, if you put your mind to it.

    I’m going to have to cut this conversation. If it keeps going like this I’m going to start ALLCAPSing.

  117. I learned more about rust from this thread that I’d learned from exploring the theory and manual. I too was intimidated and gave up on it, but I’m desperately tired of working in C, and for most of the stuff I have been doing lately, a GC language is the wrong thing. I will give it a shot again in the coming weeks. (I was exploring eric’s loccount in trying to understand why my C version was 10x faster than the go). So perhaps I’ll try a rewrite of that in rust…. for CSP, what’s a good crate to pick? The existing chan facility?

    https://github.com/servo/ipc-channel?

  118. > The “ideal” IRC server implementation would have two “threads” per connection – one that reads and handles messages, and one that picks up outgoing messages from a queue and writes them.

    I’ve implemented IRC servers (and clients and other IRC-speaking agents) both ways, and any nonblocking (or asynchronous if your OS has one) IO API is the clear winner even if you use threads for everything other part of the application.

    The trouble with blocking IO calls is that every thread can do only one thing while it’s blocked, and if you want to change your mind while a thread is blocked you have to figure out some way to unblock it: use some sort of inter-thread signalling mechanism, or sprinkle timeout parameters on your OS-level API, or embed implicit timeout state in every thread, or cancel the thread…no, wait, that’s a special, unnecessarily-hard-to-get-right case of the first thing.

    If your OS and library dependencies can implement any of that correctly, they can just implement a non-blocking API instead.

    An IRC server has to be able to preemptively drop dead client connections, and that’s just harder to do if you first have to get rid of threads blocked on IO with them.

  119. @Michael Howell

    That was just a whole bunch of rambling. Examples of other projects that make similar mistakes are non-arguments. Also, your examples are all GUI libraries or platforms. GUIs can generally get away with calling abort in those circumstances. There’s a huge difference between a UI library calling abort and the core routines of a standard library calling abort.

    If you came here to parade your knowledge of pointless trivia – good job!

  120. @Johannes Löthberg

    Why do so many people seem to think memory overcommit makes malloc practically immune to failure?

    Put a resource limit on a process, malloc above that limit and watch malloc fail (return NULL) instantly. I hope the next line of code doesn’t unconditionally dereference that NULL pointer…

    Next line of reasoning from the RoR crowd: “but who actually uses resource limits these days anyway?!”

  121. > I will give it a shot again in the coming weeks. (I was exploring eric’s loccount in trying to understand why my C version was 10x faster than the go). So perhaps I’ll try a rewrite of that in rust…. for CSP, what’s a good crate to pick? The existing chan facility?

    The important design choices depend on how fast you want to go. :-) If your goal is “I just want it to work and be tolerably fast”, just choose any crates with high download counts, and allocate String objects whenever necessary to avoid thinking too hard about the borrow checker.

    If your goal is “faster than any exiting line-count tool written in C”, you’ll require more specialized knowledge, and “which threading abstraction to use” isn’t really the important question. The important questions are probably:

    1. What’s the fastest way to walk a directory tree while honoring .gitignore? For this, your best bet is to read the ripgrep article and look at the source code. Doing so reveals that ripgrep uses the ignore crate for fast directory walking. This has a somewhat tricky parallel directory-traversal API, but I think you’ll want to look at the ripgrep source to figure out how to use it efficiently. Ripgrep is very fast, and you should steal design ideas liberally. I think it uses some kind of queuing library for cross-thread communication, but I’d need to dig through the code to confirm. (You don’t necessarily want to limit yourself to CSP in Rust. In general, Rust’s threading abstractions are surprisingly good, and you can use mutexes and even more exotic abstractions without shooting yourself in the foot.)

    2. What’s the fastest way to read files into memory? If you get this wrong, it will kill your performance. Three rules: (1) Benchmark your I/O speed. Rust I/O speed varies tremendously depending on what API you use, and you need to confirm you’re doing it right. (2) Always buffer your I/O with BufReader, so you do large-block I/O. (3) Never use the lines iterator, because it returns a fresh String for each line, and thus pays for malloc overhead, capping you at about 75 MB/s throughput per core. Instead, consider either read_line, which writes to a &mut String that you provide, or ideally just plain read, which will allow you to write zero-copy code. Using read will give you extra grief from the borrow checker, but it will allow you to reach at least 250 MB/s throughput and possibly into the multiple GB/s range depending on your disk type. (Also, according to the ripgrep article, you shouldn’t use memmap during parallel iteration; just do regular block I/O.)

    3. How do you want to write the line-counting code? This is probably the closest to straight C, and if you get (1) and (2) correct, this will be your performance bottleneck. For maximum performance, you probably want to write a custom lexer and state machine. (Be sure to look at the generated code to make sure you’re not getting killed by array bounds checks on the buffer, and visit #rust IRC if you are.) If that’s too much work, try the regex crate maybe?

    4. How many threads should you use? Ripgrep uses one thread per CPU (see the num_cpus crate), so you should probably start there, which makes sense since you’re likely to be CPU bound.

    5. Always compile with --release. Rust’s default debug optimization is terrible.

    If you use these rules as a starting point, you have a chance of writing a state-of-the-art line counting tool that gives any existing C implementations a run for their money. Expect your first week or two of Rust development to involve lots of struggling with the compiler, and expect to re-aggravate your wounds if you resort to read. Don’t hesitate to ask questions on #rust, and once you have something that sorta works, feel free to ask for a code review on r/rust on reddit.

  122. > Next line of reasoning from the RoR crowd: “but who actually uses resource limits these days anyway?!”

    Pretty much everybody, actually. The RoR crowd typically deploys to Heroku or similar managed infrastructure with strict memory limits. More serious production sites—at least the ones built in the last 12 months or so—are increasingly using containers on top of a cluster container scheduler like Amazon’s ECS or the open source Kubernetes. (Docker Swarm is trendy, but I’m not hearing nice things about it yet.) If you exhaust your specified memory limit, the cluster scheduler just kills your process and reschedules it. If you’re clever, you’ve already set up auto-scaling and you’ll spin up new server instances and workers as needed.

    Honestly, when you run completely out of RAM in a modern server application, there’s often not much you can do. Sure, in C, you can check for a NULL result from malloc, but let’s face it: Virtually nobody tests those code paths, and in my experience, they’re buggy garbage in anything not written by djb or one of a tiny handful of other ultra-paranoid C programmers. If that C code was written by a mere mortal? I want it to crash, log and restart, because once malloc has failed, I don’t trust anything the program does. The median C program is buggy garbage.

    C++’s new throws bad_alloc, which means you’ll be unwinding the stack and calling destructors with no memory available. Again, I’m not feeling much hope for the code in question, though there’s at least some chance it will work. As for Go, I think it will issue fatal error: runtime: out of memory under some circumstances, but I’ve never done ops for a large Go program, so I don’t know.

    At some abstract philosophical level, I think the right answer is for processes to monitor their own memory consumption and start refusing connections before they run out of memory. This can then be used to apply backpressure to the load balancers or queuing system.

    Actually, that would be a fun crate to write, and we could use it at work. I need to go read the ulimit man pages and see what the OS facilities actually look like.

  123. >No, you’re not. You’re going to use C, because C is little more than portable assembly language.”

    No, I’m going to use asm. If I’m worried about 13 friggin’ bytes, there isn’t going to be any space for the C runtime library. Don’t tell me what tool I’d use in a specific situation. Thanks.

    > I’m going to have to cut this conversation. If it keeps going like this I’m going to start ALLCAPSing.

    Oh, and I was so looking forward to your mini-lecture on Turing equivalence, and the one on whether P=NP, and anything else you could come up with to attempt to show us how much attention you paid in class while studiously avoiding the real issue. Free clue: we’re not impressed. Most of us took those classes too. Really.

    The real issue, for those who got lost in Michael’s content-free sideshow about algorithmic complexity, is that Rust’s syntax for concatenating strings is pure shit. The syntax. SYN-TAX.

    That is a completely separate issue from what string concatenation algorithm might be running behind that syntax. The str1 + str2 syntax might be using some whizzy log log log log log n algorithm imported from the Andromeda Galaxy. The “COBOL fingers” Rust syntax might be doing something like allocating enough space for the result, filling it with random numbers, checking to see if the result is correct, and repeating if not. Or it could be the other way round. That has nothing to do with the actual issue under discussion, which is that if your language’s syntax is shit, no one outside your little bubble is going to use it, no matter how wonderful its semantics might be, or what other advantages it might have. The Rust people appear to be largely in denial about this basic fact of real-world language adoption, so at this stage I’m going to write off the language entirely. If anyone actually manages to produce a working, widely-used operating system using that pile of bollocks, I will be the first to congratulate that person and admit that I was wrong. I don’t expect that to happen, though.

    However, since we’ve gotten off on that tangent, let me repay Michael’s mini-lecture on algorithmic complexity with one of my own.

    1) You know how your professor told you that the constant factor in an algorithm “doesn’t matter”? Your professor lied. If one piece of code takes 2n operations to process n data items, and another one takes 10n, that actually does matter. It matters A LOT. Your professor pretended that it didn’t matter because getting from 10n to 2n isn’t something you can “analyze” with pseudomathematical handwaving wankery. You have to actually know what you’re doing.

    2) Yes, if n is sufficiently large, an O(n^2) algorithm is going to be horribly worse than an O(n) algorithm. But you know what? If n is never going to be larger than 4, it probably doesn’t matter. Maybe not even if n is never going to be larger than 400. If the O(n^2) algorithm is clearer, shorter, or easier to understand, and n isn’t going to be very big, you might want to use it anyway. Again, this requires that you actually know what you’re doing.

    3) Optimizing the shit out of code when you first write it is almost always a horrible, horrible mistake. If the optimization comes at the cost of clarity, or requires taking 10x longer to write the code, or 10x the number of lines of code, or it makes the code less generalizable/reusable for other purposes, it is DEFINITELY a mistake.

  124. Doctor Locketopus: “>No, you’re not. You’re going to use C, because C is little more than portable assembly language.”

    No, I’m going to use asm. If I’m worried about 13 friggin’ bytes, there isn’t going to be any space for the C runtime library. Don’t tell me what tool I’d use in a specific situation. Thanks.”

    Uhm, C does not imply C runtime library. As mentioned above, I do embedded development without a single reference to the C runtime.

    And I think there’s a pretty good case to be made that C compilers are good enough to outperform, say, ARM assembler programmers.

    “Free clue: we’re not impressed. Most of us took those classes too. Really.”

    And those of us who didn’t have learned in the school of hard knocks.

    “Optimizing the shit out of code when you first write it is almost always a horrible, horrible mistake.”

    Premature optimization is the root of all evil.

  125. >Uhm, C does not imply C runtime library. As mentioned above, I do embedded development without a single reference to the C runtime.

    Good point. I still think I’d likely be using asm in a “13 bytes matters” situation, though.

  126. While it’s easy to quibble on the details, Esr’s *experience* _is_ valid, just as you might have a holiday and end up getting robbed in an otherwise lovely destination, picking the *wrong* project in Rust (wrong in the sense that the ecosystem may have short comings).

    I had a roughly similar experience trying to learn Rust (going on 6 months now), and while I’m more patient and accepting that there will be many stumbling blocks. I continue to be surprised at how many times I attempt something which I would hope to be *simple*, which ends up taking an afternoons research to find what is even the best approach given all the constraints. While this can be said about moving to any new technology, I’m surprised at the number of times I’ll ask a question in IRC or StackOverflow about something I would assume to be possible, only to be linked to an RFC or some discussion… of course its great there is an RFC at all… and that Rust continues to develop… but it does add real stumbling blocks which IMHO shouldn’t be ignored.

    In short, my experience is often that I attempt something that I’d be able to accomplish in C or Python… an hour later still no solution with the conclusion “it’s complicated” … *(depends on macros 1.1… requires HKT, depends on an unstable feature… can reduce 3 ugly lines to 1 but needs to pull in some crate.. etc)*.

  127. @Eric Kidd

    The Lua language does a full garbage collection cycle if malloc fails and then retries. On “embedded” systems, where a decent portion of system memory may be marked but not yet freed, this actually works out quite well.

    Consider that this wouldn’t be possible if malloc just forcefully aborted instead of returning NULL. If you actually step outside your own small slice of the world for a second, there are actually plenty of other legitimate cases where not only can you “do something useful” after malloc failure, but can actually do something to allow a successful retry.

    > “because once malloc has failed, I don’t trust anything the program does”

    All this comment does is illustrate your clear lack of understanding.

  128. “> “because once malloc has failed, I don’t trust anything the program does”

    All this comment does is illustrate your clear lack of understanding.”

    Those two are not mutually exclusive. The problem with lack of trust in a program that does anything after a malloc failure is that the program is continuing to execute as though it had gotten the memory it requested. If the application doesn’t regain control until after the failure has been caught and retried, as in your Lua example, then there’s no reason not to trust it – but that’s a different case from what Eric Kidd was raising.

  129. I wrote:

    > Actually, that would be a fun crate to write, and we could use it at work. I need to go read the ulimit man pages and see what the OS facilities actually look like.

    So it turns out that the right way check the available RAM inside of a cgroup is to compare /sys/fs/cgroup/memory/memory.limit_in_bytes and /sys/fs/cgroup/memory/memory.usage_in_bytes. If your server is running under a cgroup, this is the number that’s most likely to result in your code getting forcibly killed with an OOM error. Since nearly all of my production deploys are containerized these days, this seems like the right API.

    So I wrote an experimental Rust crate to check the RAM available to the current container and stuck it on GitHub. You use it like this:


    // Check to see if we have enough RAM to handle another request.
    if Resource::Ram.available()? < MINIMUM_REQUEST_RAM {
        warn!("Dangerously low on RAM, rejecting request");
        // Return an HTTP 503 or whatever.
    }

    This will cause your server to shed load before it crashes, and if your load balancers are configured correctly, this should apply backpressure upstream (or notify your auto-scaling setup to spin up more instances and worker processes).

    If there are any other limits (get_rlimit, swap, quotas, etc.) that you think I should handle, please feel free to explain them or send pull requests. Once this crate has been reviewed, I’m happy to stick a release up on crates.io.

  130. Campbell Barton: I’m sorry to hear you’ve been having such a rough time. Some thoughts which might or might not help; take them with a grain of salt.

    > depends on macros 1.1…

    This cannot arrive soon enough. It’s on the beta channel and should be stable around February 5th. This gets serde and diesel onto stable Rust with no need for a build.rs script. I’m going to throw a party. ????

    > requires HKT,

    Honestly, if the answers to your Rust problems are “requires higher-kinded types”, there’s a good chance that you’re asking too much of Rust. (Maybe not in your case, but that’s usually what makes people ask for HKT.) Rust is not Haskell. If you try to treat Rust like an advanced functional programming language, you will struggle and suffer unnecessarily. For example, Rust does not have monads. And sadly, its Iterator type doesn’t support a particular streaming pattern that both BurntSushi and I would love. Adding HKT types to Rust would push the language’s overall complexity budget hard. It might happen, but the core team is being very cautious and will probably only consider it this year if production Rust users are pushing hard for something which requires it.

    Basically, my advice here is like the doctor who says “Well, don’t do that then.” If you find that you need HKT, just give up and try something less clever. Part of the secret of making friends with the borrow checker is knowing when to keep your code simple.

    > depends on an unstable feature

    Anything major other than Macros 1.1? Macros 1.1 is the only unstable feature I’m using across a half-dozen production Rust projects. The only exception to this is my toy Rust kernel, which uses a handful of unstable features. In general, nightly Rust is not worth it. Just say no.

    Anyway, as somebody’s who’s been doing lots of work in Rust lately, those are my gut reactions to your problems. If I’ve misinterpreted your situation, please feel free to elaborate!

  131. “Those two are not mutually exclusive. The problem with lack of trust in a program that does anything after a malloc failure is that the program is continuing to execute as though it had gotten the memory it requested.”

    What the fuck am I even reading here…?! A program that continues like that is completely and utterly broken and you shouldn’t trust it in any state.

  132. Craig T.:

    > The Lua language does a full garbage collection cycle if malloc fails and then retries.

    This sounds really cool. Could you point me to the code? The only code I can find in the Lua runtime converts a NULL result from frealloc into a LUA_ERRMEM, which doesn’t seem to be handled in any special way according to a very cursory search. Are you thinking about a different Lua implementation?

    Anyway, if you’re writing a GC, I’d recommend just writing extern crate libc; use libc::{malloc, free}; or directly asking the kernel for some pages of memory. Also, Rust’s allocator is replacable, so you can provide your allocator with an emergency buffer if that’s what you want.

    Jay Maynard:

    > The problem with lack of trust in a program that does anything after a malloc failure is that the program is continuing to execute as though it had gotten the memory it requested.

    Actually, my complaint is even more subtle than that. Let’s imagine that our hypothetical C program does check for NULL and tries to handle it. What then? Well, it’s almost certainly going to have to unwind the stack and clean up any resources it was using.

    I’ve seen a lot of C programs in my life. Outside of a few tiny embedded systems and a few ultra-paranoid people like djb, those error-recovery code paths are garbage. They leak resources, segfault, leave data structures in inconsistent states, and so on.

    If I were a code reviewer and you wanted me to trust C error-handling code, my first question would be, “Do we have a test harness that simulates malloc failure? Does our test suite cover every possible malloc failure in the program? Do we run it under valgrind and kcov?” If the answer is “No”, then I’m not going to believe that those error-handling code paths work. I’ve audited a lot of those error-handling paths in my life, and it’s almost always resulted in a long list of bugs.

    This is why I think that allocation failures should—by default—kill the program. If you’re clever enough to recover from malloc failures correctly, you can always call a specialized allocator API.

    Yes, this is a radical and cynical position. :-) And it certainly doesn’t apply to all imaginable programs.

  133. Eric Kidd, could something like the borrower checker enable conversations which would clean up resources properly in the event of a failed malloc?

    As usual, lurking on one of these graduate level discussions is fascinating.

    Yours,
    Tom

  134. I feel like a lot of posters in here have delusions of Torvalds-esque “genius telling it like it is” grandeur that they simply haven’t achieved. Falling short of the standard set by the man himself lands you in toxicity, plain and simple. It’s not a good look.

    I like Rust but I agree with the OP (if I am understanding his points correctly) that crate discovery is a huge issue. My experience has been that a lot of crates, especially ones in the “nursery” (I am not quite sure how it all works) are of higher quality than a lot of standard library code I’ve used in other languages, but that’s no help to anyone if those crates are not easily discoverable from the core documentation, and right now for the most part they are not.

    I also agree that long-term stability is a valid concern that the Rust core team has not yet provided a fully satisfactory answer to, though I do appreciate the discipline they clearly have in separating “stable” from unstable features, and in communicating that distinction to users.

    As the OP says, I think the inner Rust community sees these issues fairly clearly and wants to resolve them, but seem to be balancing that goal with a desire to move slowly and get things right for the long term. Rust is pretty young still, so I appreciate that. Good luck to them.

    Thanks to everyone in this thread who made thoughtful and respectful comments on either side of the discussion.

  135. Interesting. I’ve not played around with Rust. From the conversation here, it looks like it might have been a good choice for my last job. I did a lot of I/O-intensive in-kernel work. I definitively relate to avoiding malloc/free because of costs. (Lots of per-CPU allocators because they avoid locking requirements). So it definitely looks like it could fill a niche.

  136. @Inkstain
    > were those OS level threads?

    Yes and no (I’ve done both). Given a stack-based implementation language, I’m not aware of any practical difference that would arise by adding or removing layers between the application and OS or rearranging the thread implementation between them. (If we’re starting from a language that doesn’t use stacks or threads, then the terms “blocking” and “nonblocking” IO API need to be redefined in terms of whatever the language is using to maintain CPU state, and we’re also so far off topic we can’t see it with a telescope).

    If we call a function like read(), and it goes into a kernel and blocks there, it’s not much different from calling read(), going into a green thread library on top of a kernel, and blocking somewhere between the application and the kernel. We still have to force an early return of the read() call, somehow, if it turns out that we need the thread in question to change its mind and do something different within a function call (not necessarily an OS call–we could be calling a compression or encryption library). In practice, every mature OS reaches the point where someone says “hey, wait, we need a way to get out of this stack hole we’ve fallen into,” and a mechanism is provided that behaves just a little differently from the way every other OS behaves when encountering the same problem.

    In the IRC server case this is a hard requirement. We need to be able to drop undesirable clients at any time. If we have a nonblocking IO API it’s trivial to switch between waiting for an internal state change and closing a socket. If we have a blocking IO API, we have to figure out how to stop some other thread doing a blocking read() and start closing a socket, without undefined behavior. In cases where nonblocking IO is unavailable and blocking IO cannot be interrupted safely, it’s not possible to implement a working IRC server.

  137. If you don’t want to abort on OOM, don’t use the standard library. It’s just a library, it doesn’t fit everyone’s use case. You can make your own collections that don’t abort. In fact, someone was working on this.

  138. > I like Rust but, yes, the string concatenation situation is annoying. Why not `+`?

    It is `+`.

    let mut a = String::new();
    a += “something”; //works

    it doesn’t work in a few situations:

    1. You have two static strings like “Hello, ” and “world” and you don’t want to write “Hello, world” another time. You can do concat!(str1, str2); This is somewhat useless and nobody actually does this. Why not just use the two strings directly since they’re already going to be written into the binary instead of concatenating? If there’s a real use case for this, then str1 + str2 can easily be added to the language.

    2. You have two owned Strings and you want to grow the first one by the second. You should borrow the second String so you don’t have to pass ownership. str1 + &str2 works, but str1 + str2 is forbidden. There is a real use case for allowing it, because you might want to do container.fold(String::Add) to concatenate all of the strings into one. But there’s also concerns that it would encourage people to do a + b.clone() which will do an unnecessary allocation instead of the correct a + &b. Currently a + b.clone() doesn’t compile so you have to do a + &b.

    Not sure which one is more annoying for you, for me #2 is definitely more annoying.

  139. Eric Kidd, my goto tool for dealing with files is mmap, and avoiding conventional I/O entirely. vs loccount.go I already had a C prototype (nloc on my github) that was about as fast as possible. The stumbling blocks were understanding how to apply a similar technique to the go version (it IS doing inefficient char/string lookup), or to find a language as expressive as go to handle the needed parallelism for dir/file lookup.

    On the linux kernel, to handle the csp, this ended up 10x faster than the go version and a heck of a lot shorter to write.

    P=`getconf _NPROCESSORS_ONLN`

    N=512

    find * -type f -name \*.c -o -name \*.h | xargs -P $P -n $N nloc | grep Total | awk -F : ‘{sum+=$2} END {print sum}’

    :)

    as for state of the art-edness, no, it was mostly just cussedness. I had a meta-goal of deeply explaining to myself why DOCOMO was such a bad model…

    thx for the tips, tho, I’ll try ’em.

  140. @Zygo I just read “spawn 2 threads per client connection” and instantly had visions of server melting under heavy load, is all.

  141. @Eric Kidd, thanks for the response.

    Here are some examples:

    re: HKT, reasons I ran into this are:

    Wanting to write a vector API that could generalize over 2D and 3D vectors (other sizes too if needed). eg:

    http://stackoverflow.com/questions/38941025

    Another time I ran into this I was looking for ability to have a function that takes and returns either a `const` or a `mut`, without having to copy-paste the code.

    http://stackoverflow.com/questions/41436525

    … I realize these are the kinds of features that seem like they should be simple, but probably aren’t, just noting that I wasn’t attempting Haskell level complexity… both are possible in C++ for example.

    re: Unstable features:

    – How to quiet a warning for a single statement in Rust? http://stackoverflow.com/questions/39269408
    – Get the number of items in an enum needs procedural macros – http://stackoverflow.com/questions/41637978
    – Type constants, functions that returns constants don’t always cut it – when the value needs to be known at compile time (more of a nice-to-have).

  142. @Eric Kidd:

    > “This sounds really cool. Could you point me to the code?”

    I can link you to the author’s slides (http://www.inf.puc-rio.br/~roberto/talks/novelties-5.2.pdf – slide #5). The code is there to be found somewhere.

    > “If I were a code reviewer and you wanted me to trust C error-handling code, my first question would be, “Do we have a test harness that simulates malloc failure? Does our test suite cover every possible malloc failure in the program?”

    The SQLite project uses a custom allocator for running tests, that is set up to fail after 1 allocation, then 2, 3, 4 etc. until it eventually completes normally. With sufficient test coverage, this can simulate malloc failure at every call site without any invasive instrumentation. I’d trust SQLite in a mission critical application over any Rust project. Rust doesn’t do anything at all to prevent logic errors or ward off shitty programmers. Sooner or later you have to accept the fact that the attitude and methodology of the programmer is more important than any novel language feature.

  143. Tom DeGisi:

    > Eric Kidd, could something like the borrower checker enable conversations which would clean up resources properly in the event of a failed malloc?

    If you’re asking about Rust: Yes! If you use #![no_std] and include just the core library, you’ll get a version of Rust that doesn’t allocate at all. Then you can write your own collection types that return an error when no memory can be allocated (somebody was working on this). This is an excellent idea for kernel-space or embedded Rust, where you don’t want most of std anyway. (Alternatively, if you want a Rust with a hidden “emergency memory reserve”, you can just replace the allocator.)

    If you’re asking about C or another language: If you were to take C, C++ or a similar language, and tried to add a Rust-style borrow checker, then you’d wind up with a language that looks a whole lot like Rust. A lot of the weird design decisions in Rust flow directly from the constraints “memory safe”, “no GC” and “fast.” Once you accept those goals, you’re at least 75% of the way to Rust.

    Dave Taht:

    > Eric Kidd, my goto tool for dealing with files is mmap, and avoiding conventional I/O entirely.

    Check out the BurntSushi ripgrep blog post I linked upthread. Apparently, according to his benchmarks, mmap is faster on Linux for reading a single file. But once you are using enough cores to processing multiple files at once, conventional I/O eventually beats mmap. This surprises me, but he lives to benchmark and tune this stuff. Ripgrep, of course, uses both mmap and conventional I/O depending on which is faster for a given task. Really do check out the source code; it’s a perfect example of very fast systems Rust.

    Campbell Barton:

    > … I realize these are the kinds of features that seem like they should be simple, but probably aren’t, just noting that I wasn’t attempting Haskell level complexity… both are possible in C++ for example.

    Yeah, OK, I see the kind of stuff that’s tripping you up. And yeah, you could do most of this stuff in modern C++.

    The underlying issue here is that while Rust is a fairly complex language, it’s still not as complex as modern C++. And the core team is frankly a bit hesitant to go there. So right now, you can’t completely generalize over mut and non-mut types, and you can’t completely generalize over 2D and 3D vectors. Similarly, as much as BurntSushi and I would love to support a certain kind of streaming I/O iterator API, that requires higher-kinded lifetimes (if not full HKT). And there are certain aspects of collection types you can’t quite generalize over, either.

    The core team could give us what we want. But the complexity problems that ESR is complaining about would probably get worse.

    I’m going to reiterate my earlier advice: “Just don’t do that, then.” When you start straying into that territory, and the compiler starts getting cranky, back off and write simpler code. You might not always achieve the level of abstraction you want, but you’ll have a much more mellow Rust experience.

    > Get the number of items in an enum needs procedural macros

    Not any more! Macros 1.1 is on the beta channel and should ship in stable the first week of February. You can definitely handle this case with Macros 1.1 and a custom `#![derive(EnumSize)]`. Stick it in a crate and call it done for everybody once and for all. :-) If you want help, hit me up on #rust IRC.

    The lack of Macros 1.1 was a real hole in the Rust story. Rust needed the ability to write custom `derive` implementations that generated code. It’s the only way to handle serialization APIs with acceptable ergonomics. And it’s far better than trying to push more complexity into the type system via more powerful generics and HKT.

    (Go was not entirely wrong to eliminate generics completely. Generics have a real cost in complexity. Rust has generics, but they’re still a lot more limited that C++ templates. There’s constant pressure to add the missing C++ features, but the compiler team is in no rush. Better to be cautious and get this stuff right, and to keep an eye on the complexity budget.)

  144. @Eric Kidd, thanks again, and I don’t want to give the impression that these issues cause continuous frustration. More like regular niggles when learning the limits of what Rust can/can’t do… and edge cases where you can, but better not :)

    Am quite happy that Rust stays non-OOP and closer to C then C++ in terms of complexity. I’d just assumed generic functions would be capable of generalizing over size fo eg (was hoping to do away with a bunch of boiler plate 2d,3d,4d functions… no matter).

    As for “Just don’t do that, then.” – right, its what I’m doing, but it takes time to figure out these limits. Or simply the limits of what is *reasonable*, when it might makes sense to use macros… unsafe… etc.

  145. Campbell Barton:

    > Am quite happy that Rust stays non-OOP and closer to C then C++ in terms of complexity. I’d just assumed generic functions would be capable of generalizing over size fo eg (was hoping to do away with a bunch of boiler plate 2d,3d,4d functions… no matter).

    Yeah, Rust might get something like C++’s template <unsigned int N> at some point. My intuition says that shouldn’t affect the global “complexity budget” very much, but somebody would probably need to look hard at the [pre-RFC discussion of this issue](https://github.com/rust-lang/rfcs/issues/1038), work out a detailed design, and address any objections. The RFC process mostly works quite well, but it relies on somebody actually stepping up to do the work.

    In the meantime, your best bet is probably to give up on representing this idea in the type system, and just resort to code generation. You can probably do this with regular Rust macros in a reasonably clear and mostly pain-free fashion.

    This is also the only way to deal with the lack of C++-style partial template specialization. Some of these C++ features are so tempting, but if Rust supports all of them, it will start to look an awful lot like the worst parts of C++ template metaprogramming. C++ had the best of intentions, and each individual feature looked justified at the time…

  146. @Inkstain

    > I just read “spawn 2 threads per client connection” and instantly had visions of server melting under heavy load, is all.

    Wasn’t my idea, it came from further upthread. ;)

    I did it once as an exercise to understand why other people keep advocating it. I can only conclude from that experiment that proponents of this architecture never tried the same experiment themselves, or just ignored the result.

  147. > Then you could write
    >
    > let hello_world = String::from(“Hello, ” + “world”);
    >
    > which wouldn’t be too bad

    I think if Rust was to be extended in this direction, it should be simply

    let mut hello_world = “Hello, ” + “world”;

    (note the mut)

  148. Campbell Barton, type level integers and higher kinded polymorphism are not the same feature. I think type level integers might get added before HKP. This is because when you use macros you’re limited to a certain number. You might stop at 1-10 and call it a day. This is very limiting.

    However, get_foo vs. get_foo_mut is solved with macros with Rust. Macros have a weirder syntax than C++ templates, but they are equivalent in this case. This is because they get type checked at expansion, like C++ templates. There will probably be no solution with generics ever. But it’s the kind of thing that’s perfect for a macro, since you will never need another type of access modifier.

    Jakub Narebski, this also works currently:

    let mut a = “Hello”;
    a = “world”;

    notice that the mut refers to the binding, not the contents – the “Hello” part is unmodified, and might still be somewhere in the executable if you actually use it, and “world” is written into a different part of the executable

    what’s mutated is the variable itself, the pointer changes from one address to the other since the type of a is &str

    I fully expect that

    let mut hello_world = “Hello, ” + “world”;

    would give you a &’static str “Hello, world” instead of a String since that’s what would happen if you impl’d Add on &str

  149. @eric kidd

    I thought the ripgrep article was fascinating (@esr, go read that!), but I think he might have missed the non-standard MAP_POPULATE option to linux mmap which cuts down on context switches by a lot. That said I learned a lot from it. My end goal is to be working on a tool (with a better model than DOCOMO) for estimating the amount of work (in the churn), love, and even quality, in a codebase, and I was groping around for a tool to write that kind of stuff in – perl being my former goto language for that kind of stuff.

    (I do wish there was a highly parallel version of etags, btw, and I enjoyed learning about remacs (a rewrite in rust). Emacs could really use threads (or at least, nonblocking io)

  150. @Igor Polyakov, Ah, good to know re: HKT/type-level-ints.

    As for using macros for const correctness, this works in principle, in practice I found it more involved – for example I ended up needing to pass in function id’s `${foobar}()` vs `${foobar}_mut()` and passing in type to be able to use `${type}::from(var)`.

    See before/after example: https://bitbucket.org/snippets/ideasman42/74nAA

    While I wouldn’t fault Rusts macro system here, this kind of macro takes extra time to write, changes to the code are more involved too.

    The resulting macro is quite a bit harder to read & reason about then the example with a single unsafe cast which is happens to be safe in practice AFAICS.

  151. Re: string concatenation, while the OP doesn’t suggest string addition, its been mentioned many times in replies.

    I’m glad Rust *doesn’t* have this, since its often abused by inexperienced developers (in Python for eg).
    A while back I reviewed a file-format exporter that built the entire file (could be many MB) using string addition – re-allocating an entire new string for _every_ element added to the string.

    Of course `a = a + b` could be coerced into `a += b` (which doesn’t have Python’s down-side of creating a new string every time)… however this only works if the values used in addition aren’t used later on in the code.

  152. Campbell Barton and others:

    OK, Team Rust does not pick one sane thing for string concatenation syntactic sugar, and Tuna noted more generally about strings types, references etc.:

    This complexity makes rust a bad language for string processing.

    And that strikes me as suboptimal for a language that wants to survive, say, in competition with C++, in which string processing can be less obnoxious than C.

  153. the summary of features speaks for itself

    That suggests you didn’t pay attention to the actual post.

  154. Harold: C++ strings are similar. You can’t concatenate two string_views in C++ either, why would you be able to do this in Rust with the equivalent &str type?

    The only annoying part is you have to borrow the second string to concatenate it, so a + &b where in C++ it’s a + b

  155. > What exactly does “hello, ” + “world” mean?

    It means “I want a string that is the concatenation of these two string literals”, which means the compiler can optimize it into “hello, world” if it likes.

    However, if you set variables hello=“hello, ” and world=“world” then

    hello_world=hello + world
    means “allocate a new string named hello_world and assign the concatenation of the strings named hello and world to it”.

    Whether a compiler would notice that neither hello nor world were referenced in any other way, and optimized them away, is another question.

  156. > Whether a compiler would notice that neither hello nor world were referenced in any other way, and optimized them away, is another question.

    For that matter, it could still optimize away the concatenation (by allocating all three strings) even if it doesn’t optimize away those strings.

  157. > Oberon is easily the best designed language in the pascal family. Go is in many respects a glorified version of Oberon

    There’s also Modula 3, which allows you to do both garbage-collected and explicitly allocated and freed storage. It’s another very well-designed language in the Pascal family. It is based loosely on Modula 2, but was not designed by Wirth.

  158. My own experience with Rust was extremely negative, for similar reasons. I too ran into the missing select() abstraction, which felt like a massive betrayal of Rust’s marketing, but I sucked it up and spent a bunch of time using the FFI to create my own bindings. (None of the crates looked promising).

    I also wanted to use POSIX pseudoterminal APIs; this led me to try to use Rust’s FFI, which was really not up to the task. The last straw was https://www.reddit.com/r/rust/comments/47a0s3/dealing_with_variation_in_c_ffi_interfaces/ – basically there’s a C function that I needed to call, and it’s impossible to call in a correct and portable way without using the platform’s C header file. There is a “posix” crate, but it’s unmaintained and only supports x86, so it doesn’t solve the problem. There’s a project rust-bindgen that’s theoretically supposed to parse C header files so you can do that sort of thing, but it was so badly maintained that I couldn’t even get it to compile.

    So I gave up on Rust entirely.

    Rust advertises itself as a language that can go anywhere C can go. To live up to that claim, it would need to be able to either parse C header files, or have enough built-in bindings to not need to do so in practice.

  159. Guess again, meathead. I’m actually an old LISP hacker who would be delighted to have a replacement systems language with the higher-order abstractions I miss from LISP.

    Lisp (Common Lisp, Scheme, Clojure) lacks features which are critically important for a language in 2017:

    1) static typing

    2) statically determined object lifetimes

    C++ and Rust give you these. Rust gives you a third: statically determined object ownership. Both give you higher-order abstractions at zero runtime cost (unlike Lisp).

    Bottom line: there are two, maybe three, choices for a language if you’re doimg serious systems programming: C++, Rust, and (maybe) Ada. Swift may mature into a fourth. If you can’t be bothered to learn Rust, at least do yourself, your dev team, and the world a favor and use C++ instead. Contrary to popular belief, there is a commonly accepted subset of C++ you should hew to. It basically boils down to: DO use the STL and Boost; DON’T use bare pointers, bare arrays, C strings, C stdio functions, or operator[] except in absolutely performance critical sections of code, and then check the shit out of those. The outliers here are libraries (like MFC and Qt) which were written before compilers for C++11 and 14 got good.

    Suck it up. You’re a talented hacker. Learn Rust, or else just use C++. If you follow the rules it’s as joyous as writing Python, and your code will be safe and performant.

  160. “just use C++. If you follow the rules it’s as joyous as writing Python, and your code will be safe and performant.”

    C++ is as joyous to me as driving a 1925 Maxwell on the Katy Freeway in Houston: it’s fragile, slow, painful, and annoying as hell both to me and everyone around me. The only reason I mess with it at all is because I work on a 1.2 MLOC monstrosity written in it, and I hate its guts. When I lived in Houston, I would sometimes daydream about driving the 90 miles to College Station so I could string Bjarne Stroustrup up by his gonads and cut his head off with a bamboo saw, slowly.

    So no, no matter how much lipstick you layer on that pig, I’m still not going to kiss it.

  161. I’m replying to the person wwho was stuck with megacode in C++ and won’t kiss the C++ pig even with lipstick on it.

    You might be interested in a paper in a Montreal conference a few
    years ago. It describes a project in which they decided to add a
    scripting language to a C++ software pile of several hundred
    thousand lines of code.

    The one they picked was Gambit, a Scheme implementation with both an interpreter and a compiler. The compiler compiles to C or C++, making it very compatible with their existing code base.

    After they added this on to their pile of code, they started using
    their new scripting feature. And they would find that it was often
    easier to reimplement a buggy component in Gambit than to fix it in
    C++.

    Over the next few years, the C++ component of their codebase shrunk
    by almost an order of magnituse. The Gambit code was much smaller
    than the code it replaced. And the increased flexibility and
    greater understanding they had with the replacement code meant that their system was faster than the original.

    I don’t have a link, unfortunately, but the paper was presented at
    a conference in Montreal on the subject of something like Scheme,
    Lisp, and/or functional programming.

    Marc Feely is in charge of the Gambit project, and I’m sure he could
    inform you further. He reads the Gambit mailing list, so you could
    find him there if you are so inclined.

    — hendrik

  162. > a project in which they decided to add a scripting language to a C++ software pile of several hundred thousand lines of code.

    This was how a lot of CAD/CAM software worked in the 1990’s. Older projects implemented a homebrew LISP variant; newer ones used Tcl. The C++ code would implement a data model and a viewer, and the Tcl or LISP would provide the UI and data manipulation functions.

    Times have changed since then, though. The working parts of old C++ were little more than you’d get from your C compiler, so it made sense to pretend you were writing code for C with Classes, and just implement a language interpreter on top of C.

    C++11 can now do things like closures, threads, and exceptions without having to resort to a second programming language. I did a few projects in C++11 recently that started with scripting language interpreter bindings because they seemed like a good fit for the C-and-a-second-language model, but I ended up never writing any code in the second language. I eventually dropped them and just wrote the whole project in C++11.

  163. I have to agree with the other poster that there are two fundamental problems in the Rust “community” (never mind the language, which is good for small programs but begins to go under its own weight in larger scope):
    1). The ridiculous Feminism/SJW vibe of the whole community with its own version of affirmative action, and
    2). The whole Rubyist mentality of the whole crowd. No wonder no C and C++ programmers are migrating to Rust.

    I do believe the story will the same as what happened to Clojure – no real usage by actual Lispers, and plenty of webdev people who turned the language into a joke.

  164. Hendrik Boom,

    Gambit is a fine project, and I’ve used it extensively myself and even made the odd small contribution.

    But it still has a GC, which makes it completely unsuitable for systems programming with strict space or time constraints. And it still lacks a static type system (though the compiler can and does perform optimizations based on inferred types).

    There are only a handful of choices when it comes to doing this sort of programming. Of these, when it comes to the vector sum of maturity, ecosystem diversity, safety, abstractive power, and performance — C++ is the clear winner (though one day soon, Rust may overtake it).

  165. I actually love Rust, although it has a big learning curve. I’d really love if Rust gets some nice syntactic sugar for common actions. For example I’d love not to specify io::stdin()::read_to_string. I’d really love to have a read! macro by default in parallel to print!
    People have their choices. But I don’t think it makes that much of a difference. No one learns Idris in 4 days and yet Idris such a nice language, it’s worth it!

    These are minor inconveniences really. As with your string concatenation example you are right. However I have seen Rust users actually justified that steps and even provided you alternative approaches to that. (I’ll add one more):
    let (x, y) = ("hello", "world");
    let z = format!("{} {}", x, y);

    Disclaimer: Before you call me a communist/leftist zionist shill “promoting” Rust to extinguish the white race by enabling “race-mixers” etc etc something something marxism propaganda (what a sad time to live!), I don’t quite use Rust myself, I like D better. But How many languages have you see that:

    1. Has no GC
    2. (As) Low level (as C++)
    3. Fast and compiles to native LLVM
    that has convenient features like pattern matching, modules, lazy iterators, ADT’s, UFCS, Higher order functions, etc these days? Of course Rust is the best language for everyone but it’s the best that came out recently.

  166. I was a bit shocked reading this post and the comments; both of the pro Rust people and the others, at least most of them. Yes, Rust is special, and it’s not convenient at all. It’s very different from C, it’s actually very different from pretty every language I’ve ever worked with. And I guess that people that are looking for a substitute for ‘C’ will be disappointed in way that they think they’ll find a language that is ‘C-like’. It’s not. But it can be used in the same field as C is being used, and that’s the whole point of the phrase ‘a potential replacement for C’. So I don’t really get it what all this childish flame-war should be good for. People that don’t like the language: don’t touch it. But writing that the language is not good is absolutely wrong.

    I personally started developing C with the age of 12 and never really “switched” wholeheartedly to anything else. Of course, I used many different languages in the meantime like Java, C#, C++, Ruby, Lua etc. but C has been my favorite language because of its (in my opinion) simple and clear design.

    Then I read about Go, which sounded very interesting and similar to C meaning that it is similarly simple, there are not many complicated core language parts, you can get a complete overview of the language in a very short time. I enjoyed learning Go very much and I still use it a lot (mostly for web services or when I need things implemented quickly). And I still follow its progress. And during the one year of using Go every day I read about Rust, and I thought: “Should I have had a look at Rust too, instead of just learning Go without looking left nor right?”. I was never a fan of complicated languages or complicated language designs (like for instance C++). Therefore I didn’t care much after reading up on Rust.

    But at some point I met difficulties arising with Go. One was its C-interface (CGO), the other one was related to handling arbitrary JSON, which is pain beyond pain in Go. So I decided to peak into Rust, installed the tool-chain, did some initial tests and started working on my ‘problems’. And it turned out to be a real success! I implemented the problematic micro services in Rust and they work very well, and damn fast, and they are stable. Yes, it was a lot of pain getting the code to compile ;-)

    Long story short: I could start coding in Go in 2 days time, no problem. It’s not a C substitute as it doesn’t give me any control over memory allocation, it’s using a GC, the C interface is not what I expected.

    I’m coding some performance critical micro services in Rust now, I’m reading 3 books in parallel (because every book shows a different angle on certain subjects), and after 2 months I’m still in the learning phase and far from having mastered the language. But the final results are very convincing, so I think it’s worth it.

    To the OP: It somehow strikes me that you had the impression that you would be able to learn Rust in a short time, similar to Go, and that you are frustrated because it didn’t work out; and maybe that makes you feel that you are not as smart as you thought after all. You are having a go at Rust because it doesn’t offer you anything similar to ‘epoll/poll/selected’ in the standard library. These functions are part of the operating system you are working with and are therefore not part of the standard library (as many others already pointed out). They are not even part of the C standard library. But as with C, you can use Rust to interface with ‘epoll’. No problem. So you might have a lot of programming experience, but that doesn’t entitle you to “either I learn a new language within xxx days or it’s crap/unfit for the world/whatever”. That’s simply not true. You’ll have to invest a lot more time on Rust. I also have a lot of programming experience, and I never thought that learning Rust would be easy. But learning C wasn’t easy either, was it? Rust is a beautiful language I think. So is Go, C, and many others. Open your mind and stop being frustrated.

    And if you don’t think that it’s worth it to spend that much time on learning the language: don’t learn it. And state it that way: It’s not worth the effort in your opinion because …

    But that doesn’t make the language a ‘bad language’.

  167. It is funny how people complain about SJWs who might be involved in Rust, while looking for their names, who they are and arguing that rust is bad because of this and this and this person and especially the supposed group of people they say they hate, which in itself is the core of SJW ideology, looking at names rather then looking at the actual and factual code. Seems like the Intelligence agencies finally realized their hoarded 0days will be worth zilch in no time

  168. I think I fall in the category of mostly loving rust. From my perspective, one of the worst things about rust is that it takes weeks to learn even after you have “learned rust.” New ideas (borrowing, lifetimes & safety) create a moire effect that allows, forbids and detours hundreds of common programming approaches and paradigms. And the rust compiler forces correctness over productivity so “pointer loving object oriented programmer,” abandon ye hope and embrace the rust path, found blindly in the first months by way of minimum compiler complaint.
    And yet, I mostly love rust: The trade-offs are grim but fair.

    1. Something I’ve noticed in Rust programmers: prior knowledge of C appears to be a hindrance to Rust enlightenment. Which means the legions of C and C++ programmers out there won’t have to be converted to Rust religion — they will simply be replaced, by Ruby and JavaScript kiddies who will have no trouble writing the kernels, device drivers, game engines, and high-performance databases of tomorrow in Rust because they haven’t been exposed to C’s broken semantics of how memory works. Perhaps in another decade or two we will regard, per Dijkstra the teaching of C as a criminal offense, for.it mutilates the mind beyond recovery.

  169. i find myself in the same boat. i think the biggest thing rust has going for it is “cool” factor. popular with the kids.
    but any look at a mature rust project and you get an obfuscated mess of wraps and unwraps and angle brackets and question marks and macros that do who knows what… the things are completely unreadable.

    compare that to a mature go project and everything is crystal clear. you can quick scan a piece of code and know exactly what it does.

    1. >compare that to a mature go project and everything is crystal clear. you can quick scan a piece of code and know exactly what it does.

      I think Go is not quite as lucid as Python that way, but I can see Python’s influence in some of the ways they worked at making the language more transparent.

  170. I couldn’t agree with this article more. I love Go. I can use most of the language’s features in a single application. I work in a large corporation and deal with infrastructure. Our code base is mostly Python. The word around is that a lot will newer POCs will be done with Golang. I’ve been hearing this to be the case with many enterprises.

    Rust looks does not look elegant whatsoever. VS Code with Go is a dream come true. I like how easy it is to use bytes, byte by byte. I pretty much know my code is going to run before compiling it thanks to VS Code’s linting and suggestions.

    I like that Golang uses tabs as it’s default. It’s great to have code auto-formatted by VS Code and the enforcement of syntax styling is what really does it for me. I can’t tell you how many Java programmers came over to the Python world at my workplace and continued writing Java (LOL).

    I agree with the author on his C/C++ views … however I cannot stand Java. Golang just got everything right, and is what Java COULD have been as far as cross-platform compatibility and syntax goes.

    Go does it right in a minimal way. I personally like the missing try/except. Half the time my try/excepts are buried as far away from the higher-level main program/logic as possible, purposefully. The fact that Go can return the result plus any error is AWESOME in my opinion: I would have written my Python code like that anyways.

    Rust just does not appeal to me. And the syntax is not enough for me to use it over a well proven language like C or even Erlang (I also really agree network servers comment).

    In my optinion, Go is everything a language should be. I just hope they don’t start adding a lot of bs/multiple ways of doing the same thing. They’ve borderline already done that regarding bytes/buffer io. But other than that, there’s typically a right way to do a task in Go. I’m hooked myself.

  171. is it true, that if I use wrong pronoun for Rust mascot I will get banned from using Rust for the rest of my life?

  172. I don’t think it’s fair to compare Rust and Go. They are languages developed for different purposes. Go was designed to be easy to learn, allow quick production, and fast compile times. These things come at the cost of a language which is frankly slow. Yes the familiar C style syntax will make developers new to language feel cozy, but I have found that for this same reason Go is often misrepresented as a language which is appropriate for systems development.

    Rust was designed for large complex highly optimized systems level code projects, whereas Go was designed for new developers to be able to quickly hack new features onto a web server, spin up a micro service, or other such I/O bound tasks, where hardware optimization essentially doesn’t matter.

    I don’t love some of Rusts decisions when it comes to syntax etc.. and the language is far from mature, but when you have found yourself trying to implement channels, closures, etc… in C enough times, then Rust become a very attractive option.

    1. >I don’t think it’s fair to compare Rust and Go.

      I think it’s completely fair to compare them when you have tasks both of them can do. Which is my situation.

      >when you have found yourself trying to implement channels, closures, etc… in C enough times, then Rust become a very attractive option.

      And Go doesn’t win bigger along that axis? Go’s CSP stuff is very easy to drive and well integrated into the rest of the language. Last I heard Rust still had some catching up to do in that area.

Leave a comment

Your email address will not be published. Required fields are marked *