A low-performance mystery: the adventure continues

The mystery I described two posts back has actually been mostly solved (I think) but I’m having a great deal of fun trying to make cvs-fast-export run even faster, and my regulars are not only kibitzing with glee but have even thrown money at me so I can upgrade my PC and run tests on a machine that doesn’t resemble (as one of them put it) a slack-jawed yokel at a hot-dog-eating contest.

Hey, a 2.66Ghz Intel Core 2 Duo with 4GB was hot shit when I bought it, and because I avoid bloatware (my window manager is i3) it has been sufficient unto my needs up to now. I’m a cheap bastard when it comes to hardware; tend to hold onto it until I actually need to upgrade. This is me loftily ignoring the snarking from the peanut gallery, except from the people who actually donated money to the Help Stamp Out CVS In Your Lifetime hardware fund.

(For the rest of you, the PayPal and Gratipay buttons should be clearly visible to your immediate right. Just sayin’…)

Ahem. Where was I? Yes. The major mystery – the unexplained slowdown in stage 3 of the threaded version – appears to have been solved. It appears this was due to a glibc feature, which is that if you link with threads support it tries to detect use of threads and use thread locks in stdio to make it safe. Which slows it down.

A workaround exists and has been applied. With that in place the threaded performance numbers are now roughly comparable to the unthreaded ones – a bit slower but nothing that isn’t readily explicable by normal cache- and disk-contention issues on hardware that is admittedly weak for this job. Sometime soon I’ll upgrade to some beastly hexacore monster with lots of RAM by today’s standards and then we’re see what we’ll see.

But the quest to make cvs-fast export faster, ever faster, continues. Besides the practical utility, I’m quite enjoying doing down-and-dirty systemsy stuff on something that isn’t GPSD. And, my regulars are having so much fun looking over my shoulder and offering suggestions that I’d kind of hate to end the party.

Here’s where things are, currently:

1. On the input side, I appear to have found a bug – or at least some behavior severely enough misdocumented that it’s tantamount to a bug – in the way flex-generated scanners handle EOF. The flex maintainer has acknowledged that something odd is going on and discussion has begun on flex-help.

A fix for this problem – which is presently limiting cvs-fast-export to character-by-character input when parsing master files, and preventing use of some flex speedups for non-interactive scanners – is probably the best near-term bet for significant performance gains.

If you feel like working on it, grab a copy of the code, delete the custom YY_INPUT macro from lex.l, rebuild, and watch what happens. Here’s a more detailed description of the problem.

Diagnosing this more precisely would actually be a good project for somebody who wants to help out but is wary of getting involved with the black magic in the CVS-analysis stuff. Whatever is wrong here is well separated from that.

2. I have given up for the moment on trying to eliminate the shared counter for making blob IDs. It turns out that the assumption those are small sequential numbers is important to the stage 3 logic – it’s used as an array index into a map of external marks.

I may revisit this after I’ve collected the optimizations with a higher expected payoff, like the flex fix and tuning.

3. A couple of my commenters are advocating a refinement of the current design that delegates writing the revision snapshots in stage 1 to a separate thread accessed by a bounded queue. This is clever, but it doesn’t attack the root of the problem, which is that currently the snapshots have to be written to disk, persist, and then be copied to standard output as blobs when the stream is being generated.

If I can refactor the main loop in generate() into a setup()/make-next-snapshot()/wrapup() sequence, something much more radical is possible. It might be that snapshot generation could be deferred until after the big black-magic branch merge in stage 2, which only needs each CVS revision’s metadata rather than its contents. Then make-next-snapshot() could be called iteratively on each master to generate snapshots just in time to be shipped on stdout as blobs.

This would be huge, possibly cutting runtime by 40%. There would be significant complications,though. A major one is that a naive implementation would have a huge working set, containing the contents of the last revision generated of all master files. There would have to be some sort of LRU scheme to hold that size down.

4. I have a to-do item to run more serious profiling with cachegrind and iostat, but I’ve been putting that off because (a) there are obvious wins to be had (like anything that reduces I/O traffic), (b) those numbers will be more interesting on the new monster machine I’ll be shopping for shortly.

68 thoughts on “A low-performance mystery: the adventure continues

  1. The Stamp Out CVS In Your Lifetime hardware fund has reaches $250 thanks to $25 from James Richardson.

  2. If you’re updating the Hardware Buyer HOWTO, I strongly suggest expanding the section on keyboards to include some advice on ergonomics, mechanical keyboards, etc. Some sort of disclaimer along the lines of “getting this purchase wrong may literally cripple you” wouldn’t be hyperbolic.

  3. @esr: “Hey, a 2.66Ghz Intel Core 2 Duo with 4GB was hot shit when I bought it, and because I avoid bloatware (my window manager is i3) it has been sufficient unto my needs up to now. I’m a cheap bastard when it comes to hardware; tend to hold onto it until I actually need to upgrade.”

    It’s a reasonable development workstation if you just want to create, edit, and build code. It’s not a reasonable machine for trying to process enormous data sets using multi-threading. That’s what servers are for, and I can’t help wondering how much of “It’s too slow” has origins in trying to make it run acceptably.on a machine never intended to do that sort of thing.

    I’ll be real curious to see what your numbers look like when you do have a more powerful machine in place to run it.

  4. In relation to your point 2. this may be a naive suggestion about a naive approach that would require non-trivial code modification, but I throw it out for consideration…

    Step 1 modified to read masters only to get metadata

    Step 3 modified to re-read masters to apply changes to generate the required blobs.

    It would add some disk-re-reading but reduce disk-writing in step 1.

  5. Actually, depending on the size of masters versus all the blobs from a master, it might actually reduce the reading in step 3 (in addtion to reducing writing in step 1).

  6. I, too, am running on a dual core machine with 4GB of RAM. Much too slow for today’s software. And I, too, am too cheap to upgrade until I really HAVE to upgrade.

  7. I am running Fedora 20 with XFCE (rather than Gnome) on a dual-core box at 3.1 GHz with 4 GB of ram and it does what I want to do just fine.

  8. >Step 1 modified to read masters only to get metadatata Step 3 modified to re-read masters to apply changes to generate the required blobs.

    Yes, this is a strong possibility. I’m thinking along similar lines.

    There would be significant problems with this, however. Working-set size is a big issue.

  9. And thank you Duncan Bayne for contributing $10 to the Help Stamp Out CVS In Your Lifetime fund.

  10. Don’t feel bad about the Core 2 Duo. I just rebuilt an old Lenovo Thinkpad T61 running a 2.0ghz core 2 duo as a secondary portable machine. It works quite well for that purpose, but the screen is rather dim from age.

  11. On the topic of hardware, I’m surprised no-one has mentioned that it can actually be a _good thing_ to have an older machine as your primary workstation, at least depending on what your primary area of focus is. It can help enforce discipline and focus around the way you work (e.g. not filling up your workspace with distractions because you simply can’t spare the resources) and around writing more efficient code. I don’t mean premature optimisation here, but rather about how being forced to work on a slow machine can get you thinking about smarter ways to implement what you’re trying to do from day one, where otherwise you might have taken the lazy approach just to get it done, thinking you’ll go back to it later.

  12. I have a to-do item to run more serious profiling with cachegrind and iostat, but I’ve been putting that off because (a) there are obvious wins to be had (like anything that reduces I/O traffic), (b) those numbers will be more interesting on the new monster machine I’ll be shopping for shortly.

    I suspect you’ll find that you get more performance wins by reducing I/O traffic than virtually everything else listed. While somewhat underpowered and very outdated, the Core 2 Duo setup you describe isn’t horrible. I have some production databases running in VMs with 4 GB and only two 2.66 Ghz cores assigned to them with no real performance issues. Of course, these are running on RHEL with no desktop and no X server and nothing else really running on it and the processors are Xeons rather than Core 2 Duos, but their workload is not all that dissimilar.

    In fact, I’ll even go so far as to say I don’t think you’ll get as much performance gain as you hope for by throwing more hardware at it.

  13. Wow! Jason Azze sets a new record by donating $250 to the Help Stamp Out CVS In Your Lifetime hardware fund. Thank you, Jason!

  14. When you update Hardward HOWTO, you should look at RAID. In the past two years a lot of motherboards are coming out with RAID support out of the box. Allow you to use RAID 1 to mirror two Hard Drives for extra redundancy.

    There also been a lot of folks who swear by having two HDs (or two sets with RAID) a smaller solid state drive for the Operating system and a larger drive where your data and software reside. I tried it myself using WIndows and is seems to give the system more pep especially with bootup. The ability to RAID two arbitrary HDs has been the outstanding feature so far.

  15. >I suspect you’ll find that you get more performance wins by reducing I/O traffic than virtually everything else listed.

    More, yes, but a reasonable person could still expect the effect of better memory and wider caches on multithreaded performance to be significant.

  16. I second the multi-HD setup advice.

    SSD for the OS, spinny disk for everything else (including swap).

    Also – NUMA. Thread and memory affinity is a big win.

  17. If allocating unique numbers is a source of contention, you can allocate blocks of e.g. 1024 unique numbers indeed. Contention gone. Only works if you can tolerate holes in the sequence and out-of-order allocation between threads.

  18. I am reading all these complaints about only having a dual core processor and 4 GB of RAM on a Toughbook CF-W2 with 256 *MB* and but a single core.

  19. >Only works if you can tolerate holes in the sequence and out-of-order allocation between threads.

    Alas, in this case I cannot.

    Progress is being made, though. I’m just wrapping up ten hours of truly frenzied hacking during which I successfully separated master file analysis from snapshot generation. Snapshot generation now takes place after the magic branch merge, immediately before export. This is well after the analysis worker threads have terminated, so the mutex guarding the snapshot counter can go away.

    Of course this means the snapshot output no longer benefits from concurrency, but that doesn’t seem to to slow things down any (which just means that phase is completely I/O dominated). It is really remarkable how fast the analysis phase is now; I’m seeing it parse the over 15K masters in the groff repository in a hair over one second.

    The goal is to merge snapshot generation into exports so that snapshots are generated on demand and written to stdout – that is, they never get dropped in a tempfile and recopied the way it’s done now. All this would have nbeen impossible before the big data structure refactoring I did to support threading in the analysis stage.

  20. The estimable Phillip Rhodes has donated $10 to Help Stamp Out CVS In Your Lifetime, bringing the current total to $400.

  21. Don’t overlook the value of the current machine. “ESR hacked here” etched on the front panel has got to be worth a few dollars. I’ll start the bidding at $50.

  22. I am enjoying these threads in the same manner as a sports fan who played the sport in college a little bit; that is to say I know the rules and enough of the domain to appreciate the talent and skills on display, but could never actually do same thing.

  23. In the past two years a lot of motherboards are coming out with RAID support out of the box.

    Nearly always with a proprietary format, and usually rather buggy. MD RAID is just fine and a lot more flexible.

  24. @esr

    Step 1 modified to read masters only to get metadatata Step 3 modified to re-read masters to apply changes to generate the required blobs.

    …There would be significant problems with this, however. Working-set size is a big issue.

    You have moved snapshot generation to after the step-2, so I am asking this just out of curiosity…. For the “naive approach” I assumed that the large working-set size would be required to hold all the blobs in core during steps 1 and 2. But if step 1 ignores everything but metadata and blobs/snapshots are created after step 2 (by re-reading masters), why a large working-set?

  25. I’m squeezing some more useful life from my workstation/game platform by dropping a modded Xeon into the LGA 771 socket. Useful info on Google but here’s a good starter: http://www.delidded.com/lga-771-to-775-adapter/

    I am of course looking at an all new system, but until the planets align, my budget is flush, and my cheap bastardenss is in remission I’ll make due, hopefully for another year or so.

  26. >But if step 1 ignores everything but metadata and blobs/snapshots are created after step 2 (by re-reading masters), why a large working-set?

    There isn’t a large-working-set problem yet. The problem arises if I go to making blobs on demand. In that case, the context for each master will need to keep around whatever revision it last built in order to have a place to start when it applies the next delta.

  27. >Don’t overlook the value of the current machine. “ESR hacked here” etched on the front panel has got to be worth a few dollars. I’ll start the bidding at $50.

    Are you physically near enough to me that you can come get it? Shipping would be kind of a pain. Also more that I may need to salvage one of the drives (there are 3).

  28. >Put the machine up on eBay, make the winner pay shipping

    Something like this has happened before. Years ago – I think two upgrades past – I decomissioned a machine and sold it to a local second-hand computer store. A week later I was told that some local gamer group calling itself “Clan of the Penguin” had been thrilled to get their hands on my old metal.

    *blink*

    Uh, OK. I hope they got years of use out if it.

  29. @esr –

    >Don’t overlook the value of the current machine.

    NO! Keep your old one as [1] your backup destination (continuous rsync from your new desktop, so you can’t lose anything), and [2] your “hot spare” (in case of OMFG failure of your shiny new box).

    Yes, there’s hacker cred value in ESR’s old computer – but I think it’s more valuable to you if still in your possession.

  30. >Yes, there’s hacker cred value in ESR’s old computer – but I think it’s more valuable to you if still in your possession.

    You make a telling point.

  31. You might be able to save some money on your old machine, if you don’t put a video card in it and run it headless all the time. You could use it for compute, but retain your current one as primary workstation.

    On the other hand, it’s nice having that power available without jumping through hoops. You could save the same money by salvaging the video card from your current box (I think you mentioned upgrading it recently, so it’s probably new enough to be worth reusing.)

  32. > On the other hand, it’s nice having that power available without jumping through hoops.

    Actually, it could *reduce* the hoop-jumping, if done carefully.

    E.g. my current systems all have good hardware support in Linux and FreeBSD (my current choice of OS). If I wanted to buy a powerful box for computing-intensive tasks, I’d set it up as a headless server and go for the best performance/price ratio I could buy, without having to worry about compatibility details. Will it connect to Ethernet? Job done :)

    Then I could sit on my swing-seat overlooking the trees, drink a cold beverage, and use ssh, tmux, Emacs and tramp to make use of my remote power-house.

  33. Thanks you, Daniel Sharpe, for donating $10 to the Help Stamp Out CVS In Your Lifetime fund.

  34. Pretty much the entire section 3 of the hardware howto is massively outdated. Plus, I think you managed a double typo in this sentence: “In fact, SCSI has effectively nerged into SCSI”.

  35. @Rick C:

    One I noticed was “One final, important tip: that audio cable from your CD-ROM back to the sound card is used only when you play audio CD-ROMs through your speakers. Software-generated sound goes through the system bus, so you can play games with sound even if your sound board or motherboard won’t accept the audio cable connector.”

    Also, with a few exceptions, the whole section on modems.

  36. Also, section 7 (Special Considerations When Buying Laptops and Netbboks) should be considered the most relevant to buyers these days. I don’t know anyone IRL who uses a desktop system as a primary development environment – for *NIX, OSX, iOS … everyone I know uses a portable of some sort.

    Happy to see you making a point of good keyboards, and vouching for the ThinkPad range (although my experience is that the customer service offered by Lenovo, at least in Australia, is pretty poor).

    I’d suggest the following advice for buying a *NIX portable: take a live distribution on USB to a store, and ask to test out any portables you’re interested in. Any decent store will be okay with that. Shortlist those with suitable specs and a good keyboard feel, then boot them to Linux / BSD / whatever, and have a poke around to see how the hardware works.

  37. Put something like this:
    http://www.ebay.com/itm/HP-Proliant-DL580-G7-Server-4x-E7550-Eight-Core-2GHz-64GB-/331343792854?pt=COMP_EN_Servers
    or
    http://www.ebay.com/itm/HP-Proliant-DL380-G7-Dual-Xeon-Quad-Core-2-53GHz-32GB-RAM-6-x-146GB-SAS-/281463476287

    in your basement, or some place cool, dry and sound isolated.

    Get one of the new Dell laptops of a size and thickness that makes you happy. They do a decent job of supporting Linux, and most have some sort of docking station. Reuse your existing monitor and keyboard.

    Set up your firewall so you can VPN in from anywhere.

    Now you have a machine that has multiple cores and can do NUMA to develop on. It’s got *lots* of memory, and can handle a decent amount of storage.

    Loud as f*k, but keep it in the basement.

    Remember, all hardware sucks and all software sucks, you’re just trying to find the combination that sucks the longest and hardest for the least money.

    That’s three variables. You ought to be able to fit an elephant.

  38. If laptops are under consideration, I’m willing to recommend something based on the Clevo W740SU base. I bought mine from AVADirect since they were a bit cheaper and I was going to be doing a custom install anyway (LVM+LUKS and all that fun), but System76 sells equivalent models with Ubuntu preinstalled. I’ve been quite pleasantly surprised with the processing and graphics performance of what’s supposed to be a low-but-not-ultra-low-power i7-4760HQ.

  39. > It appears this was due to a glibc feature, which is that if you link with threads support it tries to
    > detect use of threads and use thread locks in stdio to make it safe. Which slows it down.
    > A workaround exists and has been applied.

    I’ve never heard of this feature. Do you have a link to the documentation?
    And how does your workaround look like?

  40. >I don’t know anyone IRL who uses a desktop system as a primary development environment – for *NIX, OSX, iOS … everyone I know uses a portable of some sort.

    That’s bizarre. How do they live with the small screens and crappy keyboards?

  41. >One I noticed was “One final, important tip: that audio cable from your CD-ROM back to the sound card is used only when you play audio CD-ROMs through your speakers. Software-generated sound goes through the system bus, so you can play games with sound even if your sound board or motherboard won’t accept the audio cable connector.”

    What’s wrong with this?

  42. A person who wishes to be known as ‘A’ has contributed the Discordian amount of $23 to the Help Stamp Out CVS In Your Lifetime fund. ‘A’, I thank you and the Goddess Eris thanks you.

  43. How do they live with the small screens and crappy keyboards?

    In my case, by having a high-resolution built-in screen for portable work and a DisplayPort port for at the office, and by selecting carefully for the keyboard. I personally prefer a high-quality scissor to a mechanical, and the keyboard on my recently-retired Dell M6600 was near perfect. The keyboard on the W740SU is medium (about like that of a MacBook), but not a full 104, and so I have a full-size external SIIG scissor keyboard for the office.

    What’s wrong with this?

    Nothing wrong per se, it’s just like having a chapter on carburetors in an introductory book on modern cars.

  44. > I thank you and the Goddess Eris thanks you.

    Surely not? I’d have thought “the Greek goddess of chaos, strife and discord” would have been pushing CVS on everyone, from the start ;)

    > That’s bizarre. How do they live with the small screens and crappy keyboards?

    The same way I do – by buying something with a half decent keyboard (MacBook, ThinkPad) and having an external keyboard and mouse on their desk(s) for when not mobile.

    E.g. on my three-dev team, we have:

    – one of the smaller ThinkPads (an X220?) coupled to a docking station, Das Keyboard and an external monitor

    – a MacBook air, with an external monitor and Apple keyboard

    – my ThinkPad L530, which is currently residing on a standing station at work; at home, I have an L520 mated to an external monitor and a Unicomp On-The-Ball ( https://www.flickr.com/photos/duncanbayne/12295182336/ )

    I’m particularly enamored of the ThinkPads’ anti-reflective screens. Makes a big difference when working in environments with poor ambient light (e.g. trains, cafes).

  45. > Surely not? I’d have thought “the Greek goddess of chaos, strife and discord” would have been pushing CVS on everyone, from the start ;)

    It may be worth noting that the ddate program contains, in a comment, in obvious parody of RCS IDs, a “[i]DVCS[/i] ID”.

  46. John D Bell beat me to it with his very sound advice to use your old machine as a constant backup of your current machine using rsync. For the last 15 years, I have always relegated the “old” machine to that very purpose. Old machine is always built with two identical sized hard drives and nightly scripts run that tar ball up all the data. You can’t imagine how many times I’ve needed to roll back to data found in those tar balls.Once you get a decent machine, then you start realizing the power and utility of virtualbox too!

  47. > That’s bizarre. How do they live with the small screens and crappy keyboards?

    My work provided lapdog is a 15″ MacBook Pro. At work I’ve got a 24″ Dell monitor and a Dell Keyboard (I don’t care for the Apple keyboards, the dells are ok) and the laptop is open giving me 2 screens with the 24″ being the primary.

    At home it’s reversed–I plug in the 24 inch extra screen that sometimes hangs off the iMac (27″ screen) and just “suffer” with the laptop keyboard. This is just until get a little more out of the hole financially and get a better desk.

    Modern lapdogs can drive really high res screens. Everyone in the Dev/Ops group I’m in is using lapdogs as their primary machines. Most of the dev guys are driving 27 inch monitors at full resolution.

  48. Modern lapdogs can drive really high res screens. Everyone in the Dev/Ops group I’m in is using lapdogs as their primary machines. Most of the dev guys are driving 27 inch monitors at full resolution.

    For reference, the internal panel on the current MacBook Pro is 2880×1800, and Apple doesn’t even use a discrete graphics card, just the i7’s integrated Iris Pro. AMD’s integrated graphics are weaker but probably still fine for the (as I understand) non-3D workload you do.

  49. Since we seem to be on a laptop discussion, let me modestly mention the laptop I have. It’s a gaming laptop, an MSi GX70 with 12GB RAM, 4 2.5Ghz AMD cores with 320KB L1 cache and 8MB L2. It also has a 1TB spinning disk and a 256MB SSD, AMD Radeon GPU, a 1920×1050 resolution 17″ display and the ability to drive at least 2 other monitors (I believe I can do 4 displays total, but have no desire to test this). I usually use an external keyboard and mouse but the built in keyboard and trackpad are adequate for those times when I’m not docked.

    Amazingly it also has reasonable battery life if you don’t stress it too much – as in I’ve used it for basic web/doc processing/perl dev for a good 4 hours before it told me to connect the power supply (which was the moment when I realized I’d failed to ensure the power supply was plugged into something with actual electricity). I typically have at least 2 VMs running (one linux for dev, one win 7 for office tasks) and spin up more for specific demos and the like.

    And yes it weighs 8lbs and doesn’t really work as a machine to use when flying in cattle class.

    It isn’t suitable for ESR because I cannot get the thing to reliably boot into Linux. I’ve tried on and off a number of times and have had it work for a bit until some update somewhere breaks it again. The root problem seems to be the GPU drivers but UEFI also gives me grief (since I have real work to do, I’ve limited my time endlessly rebooting to try and get it to consistently boot linux). However as a tool for me it is great, despite having to run windows 8.1. There are similar Dell (Alienware) gamer laptops that do run Linux properly and that’s the sort of machine I’d recommend for ESR.

  50. The MSI laptops have an obnoxious keyboard layout, unfortunately. I liked the rest of their specs, but I have a hard filter on (1080p+ display && “correct” keyboard).

  51. > but System76 sells equivalent models with Ubuntu preinstalled.

    Hmmm… 17.3 inch lapdog with 16 gig of ram, 4 cores and 2x250G SSDs for around $1500.

    Not bad. Not bad at all.

    And the high end laptop (4 cores and up to 32 gig of ram) isn’t bad on price either.

  52. “Are you physically near enough to me that you can come get it?.”

    Seattle, so no. And you’ve heard better suggestions. Get something BIG & FAST!

  53. @esr:
    >What’s wrong with this?

    I don’t think I’ve seen a system that didn’t play CD audio through software in the last decade at least.

    Also, who actually plays audio from CD’s anymore? Nowadays, if you even buy a CD, you usually rip it and play it from a centralized music folder on your hard drive.

  54. Ahem. Where was I? Yes. The major mystery – the unexplained slowdown in stage 3 of the threaded version – appears to have been solved. It appears this was due to a glibc feature, which is that if you link with threads support it tries to detect use of threads and use thread locks in stdio to make it safe. Which slows it down.

    Ah, GNU. It’s as if they’re trying to hammer home the point that free software is inherently ideological by convincing us to accept bloated, overcomplicated solutions because they’re free.

    Consider compiling against musl. It is smaller and less buggy than glibc while remaining relatively complete, and is BSD-licensed!

  55. > Consider compiling against musl. It is smaller and less buggy than glibc while remaining relatively complete, and is BSD-licensed!

    And also has locking in its stdio primitives. It’s almost like this stuff is actually necessary for a POSIX-compliant C library.

    I’m still not sure I understand how a single lock acquisition at the top of fwrite can have a significant performance impact, yet the one in (for example) the fprintf directly before that fwrite doesn’t.

Leave a Reply

Your email address will not be published. Required fields are marked *