OK, I’m officially coming out of my cave now, after what amounted to a two-week coding orgy. I’ve shipped reposurgeon 0.5; the code looks and feels pretty solid, the documentation is written, the test suite is in place, and I’ve got working repo-rebuild support for two systems, one of which is not git.
The rest is cleanup and polishing. Likely the next release or the one after will be 1.0. It’s time for an after-action report. As usual, I learned a few things from this project. Some are worth sharing.
The basic concept was to edit repositories by (1) serializing them to git-import streams with export tools, (2) deserializing that representation into a bunch of Python objects in the front end of reposurgeon, (3) hacking the objects, (4) serializing them to a modified git-import stream with the back end of reposurgeon, and (5) feeding that to importers to recreate a repository.
Did it work? Oh, yes, indeed it did. By decoupling reposurgeon from the internals of any particular VCS I was able to write a program that, for its power and range, is ridiculously short and simple. 2665 lines of Python can perform complex editing operations on git and bzr, and will add more VCSes to its range as fast as they grow importers. (Next one will almost certainly be hg; its exporter works, but its importer is still somewhat flaky.)
In fact I exceeded my planned feature-list – found myself implementing a couple of extra operations (topological cut and commit split), simply because the representation-as-Python-objects was so easy to work with that it suggested possibilities.
Another thumping success was the interface pattern – a line-oriented interpreter with the property that command-line arguments are simply treated as lines to be passed to it, with one magic cookie “-” meaning “go into interactive mode and take commands until EOF”. This proved extremely flexible and easy to script, especially after I write the “script” command (execute command lines from a named file).
With this interface in place, adding stuff to the regression-test suite was almost absurdly easy. Which meant that I wrote a lot of tests. Which is good!
What’s the biggest drawback? Speed, or rather lack of it. On large repos, waiting for that repo read to finish is going to take a while. Can’t be helped, really; first, the exporter has to serialize the repo, then reposurgeon has to read and digest that whole input stream. I profiled, and as is normal with code like this it spends most of its time either on disk I/O or in the string library. There just aren’t any well-defined optimization targets sticking up to be flattened.
The performance issue raises an obvious question: was Python a mistake here? I love the language, but there is no denying that it is a high-overhead, poor-performing language even relative to others in it class, let alone relative to a compiled language like C.
On the other hand…no way this would have been a 16-day project in C. I’d have been lucky to get it done in sixteen weeks in C, probably with 14 of them spent chasing resource-management bugs and a grievously high maintenance burden down the road. Python I can fire and, more or less, forget – no core-dump heisenbugs for me, baby.
You pays your money and you takes your choice. I think Python was right here – heck, if I’d had to do in C I might not have been brave enough to try it at all, and my other scripting languages are now rusty enough that I would have lost a lot of project time to relearning them. But a slowish language is a relatively easy tradeoff here because this is not a program I expect to be in daily production use by anyone; it’s a coping mechanism for unusual situations.
Jay Maynard happened to be staying at my place during the first frenzied week of hacking. He actually argued semi-seriously that it’s a good thing reposurgeon is slow, as repo surgery is such a fraught and potentially shady proposition that it should be difficult and give you plenty of time to reflect. Less chance of fingers getting ahead of brain that way, maybe.
Most fun part of the project? That’s a toss-up, I think. One candidate was the graph-coloring algorithm I wrote to check whether breaking a given parent/child link would actually accomplish a topological cut of the commit graph. Another candidate is the bit of reduction algebra that I wrote to simplify sequences of file operations after a commit deletion. Both held challenges pleasing to my ex-mathematicianly self.
The least fun part was being dependent on flaky import-export tools. And I found out that, even though it’s the first non-git system I have support for in reposurgeon, I don’t like bzr. I don’t like it at all.
In your bog-standard generic DVCS – of which git and hg are the two leading examples – the unit of work (the thing that’s passed around when you clone and push and pull) is an entire repository that can contain multiple branches and merges and looks like a general DAG (directed acyclic graph). Sync operations between repositories are merge operations aimed at coming up with a reconciled history DAG that both sides of the merge can agree on.
(Yes, I love thinking about this stuff. Shows, doesn’t it?)
bzr has a conceptual problem. It can’t decide whether it wants its unit of work to be a repo or a branch. The way it wants you to work is to create a branch which is copied from and tied back to a branch in a remote repo. Your branch isn’t the entire project history, just one strictly linear segment of it that you occasionally push and merge to other peoples’ branches (or collections of branches).
Maybe this model has virtues I’m oblivious to, but it seems both restrictive and overcomplicated to me. Here’s where it bites reposurgeon: the bzr exporter can only export branches, but the importer creates an (implicitly multi-branch) repo with your branch living in a subdirectory named ‘trunk’.
It seems deeply wrong to me that the import and export operations aren’t lossless inversions of each other as they are in git (well, except for cryptosigned tags) and will be in hg when it gets its importing act together. I’m now trying to persuade the relevant bzr dev that they’ve missed the point of import streams and should be exporting entire repo graphs rather than just branches. We’ll see how that goes.
Late nights, monomaniacal concentration, and code code code. Ahhh. Not the life for everyone, but it suits me fine.
I (almost) always think Python is the right answer.
Sometimes you have to code a bit of C (or Pyrex/Cython) for performance.
Oftentimes, though, there are other things you can do before you go down that path. One thing to watch out for is that sometimes the profiler lies about where overhead is being incurred. Another thing is that, if you are doing a lot of string functions, something like re.split() can often give you a huge parsing win.
I wish I had time to download and play with this now — in my experience, Python programs can sometimes be made *faster* than C programs, for the simple reason that you alluded to, namely, that nobody has time to carefully hand-craft C well enough to be both fast and correct.
It’s interesting to see your analysis of bzr. I’ve been wondering whether that’s useful to add to my arsenal, but I wholeheartedly agree that the ability to round-trip is eminently useful. I think that’s true for a lot of problem domains. For example, with audio/video compression. you certainly don’t expect the first compression to be lossless, but a decompression and subsequent recompression with the same algorithm ought to yield the same compressed bits.
I’ve been heads-down coding lately, as well, which is good and bad. Good, because it’s fun; bad, because I can’t share my results with the world. Ah, well, you can’t always have everything.
You forgot a couple of games in there, too.
>You forgot a couple of games in there, too.
And damn fine games they were, too. Jay here’s got a knack for small-unit tactics. Didn’t beat me, but might have if we’d been able to play out that third one.
>I’ve been wondering whether [bzr would be] useful to add to my arsenal, but I wholeheartedly agree that the ability to round-trip is eminently useful.
I’m going to say “don’t”. IMO either git or hg beats it like a gong. The bzr people like to talk up their supposed ease of use, but I ain’t buying it – not as long as their branching model makes my head hurt when I try to think about it. Wrong wrong wrong.
Phew. Boy am I glad I’m not the only one. And here I thought with the bzr folks yapping about how much “easier” bzr is to use, maybe, just maybe, *I* was the stupid one. But if the branching model gives *you* headaches, well, apparently I’m not so crazy.
OTOH, that which does not kill you makes you stronger, right? Learning bzr may be good from that standpoint, I suppose.
Well, I have to admit that, when bazaar first came out, I read the papers, wanted to like it, played with it a bit, but found it somewhat difficult, and since I tend not to be an early adopter on mission critical stuff when I already have a working solution, set it aside and never looked at it again.
Eric, I get the impression that you don’t really rate C++ (please correct me if this isn’t true), but surely that has to be the solution for a task that requires the performance of C together with the ease of programming of a scripting language.
Since I’ve been working with the combination of C++ with a comprehensive class library (first STL then Qt), I’ve never had to worry about checking that the result of malloc isn’t NULL, or whether a buffer is big enough before copying in a string, or whether an index is valid before using it with an array, or think about how to hand-code the management of any data structure that is more complex than a simple array. Yes, that has all been shifted to behind the scenes with more overhead, but with today’s technology that is not a serious problem.
>ease of programming of a scripting language
There may be a universe in which C++ has the “ease of programming of a scripting language” somewhere, but it ain’t this one. C++ is a hideous pile of overcomplexity, non-orthogonal features, and cruft. I’ve had way too much experience with it working on Battle for Wesnoth and don’t want any more.
@esr: Did you check out the Qt library?
When I use it I feel like I am programming in a programming language other than C++. They added a lot of features (a foreach loop for instance, as well as a couple of memory management techniques for starters) due to the extensibility of C++, thus it’s no longer plain C++ we’re talking about here.
I agree about C++. I learned C++ in school and I tell you that it feels too complex to wrap your head around especially its Object-Oriented programming features.
I ended up coding mostly in procedural style even in C++ which is not that much different from C programming (except the standard libraries).
There are some libraries which are honourable exceptions like Qt kit, but generally most OOP programming in C++ is horrible. C++ is why I started hating OOP competely to the extent that I hesitate to use OOP features on other languages as well, including Python to this day.
@hari:
> I agree about C++. I learned C++ in school and I tell you that it feels too
> complex to wrap your head around especially its Object-Oriented programming
> features.
> I ended up coding mostly in procedural style even in C++ which is not that
> much different from C programming (except the standard libraries).
You don’t have to use OOP – there is nothing wrong with writing a procedural program in C++, but don’t underestimate the power of the libraries even if you do so. They will help greatly with memory management and data structures (and avoid having to do many the checks that I mentioned in my first message).
I agree with you that OOP may seem difficult and unnatural when presented through artificial didactic examples, but certainly for GUI programming with Qt it just feels like the natural way to do things. Even its main rival GTK is OOP-based but implemented in “pure C”; I’ve no experience of using this myself, but I can’t imagine that it would be any easier/clearer than using a language designed for OOP.
> You don’t have to use OOP – there is nothing wrong with writing a procedural program in C++, but don’t underestimate the power of the libraries even if you do so. They will help greatly with memory management and data structures (and avoid having to do many the checks that I mentioned in my first message).
I would tend to agree with this in theory except that libraries vary hugely in quality and documentation in C++.
Adding to the complexity is that some C++ libraries often tend to be a huge spider’s web of class and object dependencies which make deciphering their usage much much more difficult than actually handling the problem you set out to solve. Java seems to have similar problems though of a lesser degree though it suffers from other faults as well.
But yes, when you get a well designed library it can be a boon in any programming language and especially C++.
The reason I went to spend a few days with Eric (that turned into a week due to car issues (and, especially a pair of E4 Torx screws!)) was that I wanted guidance and a bit of mentoring in dealing with an open source program I’m helping with. It’s north of 750K lines of code, and all C++/Qt, with a healthy dose of Boost thrown in.
Merciful $DEITY, what a pain in the ass to work with. It often takes me, literally, hours to find where something is done in the code. I’m the lead on the OS X port of the program, and I’m sure Xcode can help in there somewhere, but holy cow…trying to track down a spurious int->float->int conversion somewhere in the code took me more than a day – and I still haven’t found it. I got lost in a twisty maze of templates, all different.
Then there’s Boost. I needed to rebuild it to make sure I had a clean copy. I could not, for the life of me, get it to build with the options I needed. I finally gave up, took a copy from somewhere else trusted, and crossed my fingers.
Sorry, but C++ is evil. “Ease of programming” only applies to it in the same way that “sweetens coffee better than sucrose” applies to Tabasco.
@esr:
Speaking of C++, have you made any progress on the proposed critique you mentioned some time ago? I know I am not the only one waiting patiently to read your full thoughts. Last I recall you mentioning, it was on hold due to collaborator distance. Still spinning in sleep(3) mode?
ESR: Yes, alas. At some point Rob Landley will fly in so we can finish it.
@esr, could you share those algorithms (in truest sense ;-)), perhaps in next blog post? I think that it could be interesting.
>@esr, could you share those algorithms (in truest sense ;-)), perhaps in next blog post? I think that it could be interesting.
For the reduction algebra, look under CANONICALIZATION RULES on the manual page.
The graph-coloring went like this. Suppose you want to break a parent repo into two pieces (say, to split off a subproject) but preserve the branch structure in the child repos. In general, this is possible only if your cut wouldn’t leave the repo connected through another path (e.g. if you’re not cutting a link in one of two or more commit chains that end in the same merge). You want to do the cut if possible, but if it isn’t you want to give an error message rather than making the cut.
Here’s what you do. You make the cut, then you tag the commits before and after it with different values of a color attribute. Then, for each neighbor of a colored commit, you apply the following rule: ignore commits of the same color, give your color to uncolored nodes, and stop on a node of the other color. You keep doing this until one of two things happens. Either you run out of uncolored nodes, in which case you have successfully partitioned the graph into disjoint subgraphs with different colors and you’re done. Or, you stop on a neighbor of the other color, in which case your cut was on one side of a bubble in the DAG and the DAG is still connected through the node you just found. You undo the cut and throw an error.
This algorithm is lifted straight from a common proof tactic in graph theory. I wish I got to do stuff like it more often.
If you like Python but imagine you might be faster in C, you might look at Pyrex (http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/), which basically lets you write C in Python. I don’t know about doing the whole thing in Pyrex, but it does make it easy to drop certain bits into C.
In other news, Fossil SCM (http://www.fossil-scm.org/) is now growing GIT import/export. It’s brand new functionality, so probably a little rough, but there’s another potential target.
>In other news, Fossil SCM (http://www.fossil-scm.org/) is now growing GIT import/export. It’s brand new functionality, so probably a little rough, but there’s another potential target.
Only if it has export, too.
But I gotta say fossil is looking pretty interesting. I’ll have to try it out for a small project.
Did you consider writing it in Haskell? Haskell performance rivals C, and the task just screams for Parsec.
>Did you consider writing it in Haskell?
Actually, I did…for a microsecond or two. Then I decided I wanted the tool quickly more than I wanted the Haskell learning experience.
I’ve used Pyrex to great advantage in the past. I haven’t used Cython, but if I had a performance problem today, I might use that instead. Cython is a friendly fork of Pyrex, which is more cutting-edge. (Pyrex is just one very smart but very part-time guy without a public repository; Cython is a lot of people who couldn’t get their patches into Pyrex quickly enough.)
@Hari: The OOP features in Python aren’t very much like the OOP features in C++. The Python object model is somewhat unique due to its dynamic typing features. In C++, if you pass an object that’s expected to be of type foo, it really, really has to be type foo. OTOH, in Python, so long as the object looks enough like type foo, it’ll work.
Thing is, even if you never ever define a class in Python, in a sense you’re still using OOP because in Python, everything is a class, even exceptions. Even a simple string is class with methods. For example, this is a common method of joining two strings with a comma in-between:
>>> a=”foo”
>>> b=”bar”
>>> “,”.join([a, b])
‘foo,bar’
That ‘.join’ is a method call! The coolest thing about Python objects, though, is introspection. If you don’t know what attributes a class has, you can find out. Try ‘str.__dict__.keys()’ at a Python prompt. (Yes, ‘join’ is in there. :))
I learned OOP (and pretty much everything else) in Python, so that’s become my standard in terms of readability and intuitiveness. I tried (and am still trying) to learn C++, and found it to be the complete opposite of the spectrum, and thus I have not tried to build anything useful with it. I found Java to be somewhat better, and it’s easier to build GUIs with it. However, the day someone comes up with a GUI builder that can match the quality of NetBeans or Eclipse in Python is the day I give up trying to learn yet another language to get a decent interface, because its just too darn easy to write the backend in Python.
>However, the day someone comes up with a GUI builder that can match the quality of NetBeans or Eclipse in Python is the day I give up trying to learn yet another language
Look into pyglade, if you haven’t. I don’t know NetBeans or Eclipse, so I can’t compare, but it’s worked well for me.
I’d be interested to know what a NetBeans/Eclipse user thinks of it.
@esr: Does it mean that you keep whole repository (or representation of it) in memory? Or do you use some kind of lazy-loading or streaming?
>Does it mean that you keep whole repository (or representation of it) in memory? Or do you use some kind of lazy-loading or streaming?
Everything except the blobs goes in memory; the blobs live in tempfiles during the edit. I’ve metered, and the resulting memory footprint is tolerable even for repositories the size of, say, git’s
It has to be this way, as some of the edit operations require random access.
@Daniel: About Haskell – I have heard that one of reasons behind low popularity of Darcs, an DVCS written in Haskell, was that Haskell requires fairly exotic toolchain to compile (and improve) and to use. Python is fairly widespread and portable.
Jakub: the dependencies are no more onerous than for any other language. Granted, that you’re much more likely to have already installed Python than to have already installed GHC, but it’s still no more than an ‘apt-get install’ away.
Gah. Haskell strikes me as formalistic and theoretical, much more than practical, much like pure Pascal. I’m not interested in learning lots of mathematical theory. I just want to solve problems with computers. If I wanted to learn a language just as a mind expansion, I’d tackle INTERCAL.
Haskell can expand the mind, INTERCAL will twist it. It’s not quite the same thing, though I will freely concede the early stages of both may feel the same.
Eric, we need to make a Haskell compiler in INTERCAL for that upcoming proj…*phht*thud*slump*
>Eric, we need to make a Haskell compiler in INTERCAL for that upcoming proj…*phht*thud*slump*
Well done, my ninjas!
> Haskell can expand the mind, INTERCAL will twist it. It’s not quite the same thing, though I will freely concede the early stages of both may feel the same.
Ah. In that case, I guess I won’t try starting to learn INTERCAL.
@Jakub:
> About Haskell – I have heard that one of reasons behind low popularity of Darcs, an DVCS written in Haskell, was that Haskell requires fairly exotic toolchain to compile (and improve) and to use. Python is fairly widespread and portable.
GHC is not particularly exotic, and with the advent of the Haskell Platform it’s become even easier to install. My feeling is that the low popularity of Darcs is because, while it appeals to mathematically-inclined people, the patch theory stuff it does isn’t terribly practical, and the tool itself is mostly only known in the Haskell/academic world. If git/hg/bzr had never shown up, I’m sure Darcs would be more popular, but as it stands, I’d recommend the use of nearly any other DVCS over Darcs, and I’m sure I’m not the only one.
You don’t actually need the patch-theory stuff to use darcs. Darcs is plenty easy to use without grokking it, and usually does the right thing. It’s also the first open-source DVCS to come to prominence.
The freakin’ problem with darcs became manifest to me when I tried building it on my OLPC. GHC fell over hard many times in the struggle to compile.
Git compiled without a hitch. It’s written in C.
I’d love to use something as cool as darcs, but pragmatic concerns force me to use git — itself a fine tool.
Jay Maynard Says: Sorry, but C++ is evil. “Ease of programming†only applies to it in the same way that “sweetens coffee better than sucrose†applies to Tabasco.
I respectfully disagree. C++ is very easy to program in. The hard part is getting that program you wrote so easily to actually work. Almost nothing in the language seems to work the way you thought it would…
This definition of “easy to program in” only counts if you consider getting the compiler to emit an executable instead of error messages to be “programming”.
Do you mean this pyglade? I haven’t tried it, but I’ve used something similar called GladeGen , which I think is about the same thing. Both are now completely obsolete with GTK Builder.
“Thing is, even if you never ever define a class in Python, in a sense you’re still using OOP because in Python, everything is a class, even exceptions.”
What blows my is that every function is an object, and every executable object is a function. It made me understand that concepts like OOP or functional programming have much more to with limitations in less flexible programming languages, and are not really categories conceptually cast in stone.
@Morgan Grey
Good point there.
I feel very comfortable coding in Python anyway. I agree that OOP is much cleaner in Python than C++. I’ve written more programs in Python these days than in C or C++ in the last 10 years. Practically all my apps are coded in Python except the PHP ones I write for my website. And with a bit of care, most Python code can be as cross-platform as Java.
I do have an occasional need for compiled code though. I don’t want to give up C entirely.
Sorry, but C++ is evil. “Ease of programming†only applies to it in the same way that “sweetens coffee better than sucrose†applies to Tabasco.
…
“This definition of “easy to program in†only counts if you consider getting the compiler to emit an executable instead of error messages to be “programmingâ€.
I don’t know what your (and others’) problem is with C++. I have always found it to be a flexible and expressive language, and have built a helluva lot of high-quality production code with it. I also use many other languages as needs dictate – Java, Python, Ada, Objective-C blah blah blah – and don’t have any difficulties doing so. They’re just media for conveying design. I’ve never really understood all the pissing contests that inevitably flare up from time-to-time regarding ‘which language is better’…I kinda stand back and observe them with a bemused expression on my face.
Modern IDEs are a godsend for helping beginners get over that seemingly insurmountable hump where nothing you type seems to compile – syntax highlighting, code completion, auto-referencing etc….heck, even I like to lean on them; it frees my brain up for more important things.
As far as I know, there is no integrated Python GUI module for Netbeans (my IDE of choice). The ‘PyGlade’ combo of using the Glade UI builder and Python/GTK+ bindings to integrate your Python code, seems to be a good approach, if that’s what you need to use.
C++ absolutely is a flexible and expressive language. That’s half the problem. When somebody hands you a pile of code to work with, chances are that if it is in C, it’s pretty straightforward, but if it’s in C++, that’s often because the original author felt C wasn’t expressive enough, and they wanted to do something really clever.
The bottom line is that you can write obfuscated cruft in any language, but a few languages like C++, perl, and PHP make this much easier. There’s somewhat of a chicken and egg thing going on there — people who tend to write obfuscated cruft are naturally attracted to these languages, while people who learn one of these languages (and its idioms from other practitioners) as their introduction to programming can wind up thinking that obfuscated cruft is the one true way to program. It’s a positive feedback look, and the result is that bad programming practices propagate in programming communities in much the same way that child abuse does in families.
So, even though you can certainly use C++ to write nice stuff, it’s just not the done thing in some quarters.
Are you one of those people who thinks all cultures are created equal? Seriously, if you can’t look at languages and rate them both objectively and subjectively on different attributes, and relate those attributes to the task you have at hand, and then pick the best language based on the task, your familiarity with the language, and the language’s attributes, then you have no business programming. If you can look at languages and rate them in this fashion, then you by definition have an opinion about which language is best for you for a given task. If you’re just bemused by people having strong opinions about these things and sharing those opinions, and have no idea how those opinions came to be formed, then, for you, programming is just a day job — something that you’re not interested in enough to delve into and learn deeply with any sort of passion.
Ah. Day job then.
> C++ absolutely is a flexible and expressive language. That’s half the problem. When somebody hands you a pile of code to work with, chances are
> that if it is in C, it’s pretty straightforward, but if it’s in C++, that’s often because the original author felt C wasn’t expressive enough, and they
> wanted to do something really clever.
I’m not a C++ fan by any stretch of the imagination, but something about that argument makes it seem like it shouldn’t ought to be valid. Expressiveness and flexibility are not bad things.
I think more of a problem with C++ is its inconsistency, redundancy, and often cringe-inducing syntax.
So you actually like being unproductive? :)
> So you actually like being unproductive? :)
Can’t speak for Patrick, but I certainly do. Why *do* you think I started programming? My grades have never been the same since. ;)
“Expressiveness and flexibility are not bad things.”
No, they are good things….but…aren’t Python (and a bunch of other languages) just as flexible and expressive as C++?…without all that interaction amongst its conglomeration of ‘features’?
For the non-geezers reading this, (most of you) I’ll just mention APL. That one was expressiveness and flexibility taken to the max. The judgement of history has been pretty hard on it.
“ALGOL was a big improvement upon its successors.”
(I forgot who said that, and quick Googling did not reveal the source. It was either Hoare or Dijkstra.)
FWIW, my definition of the ‘better language’ is one that expands with the user. If you’re a rank noob (or a curious 11-year old nerd) and you sit down to the Python interpreter, within a half hour you can write simple programs that might do something useful. As your experience and curiosity grows, the language grows with you. For longest time, the programs I wrote were basically text re-formatters for a game mod I was working on (a GUI-based game script editor). Python excels at text processing, and because EVERYTHING in Python is an object, the models in your head translate pretty easily onto the screen. Not long ago, I decided to create an economics simulator, and discovered very quickly that trying to process large numbers of identical objects serially is SLOW, and that to parallelize it I needed threads. I had avoided threads for a long time because they were ‘scary’, but after working through a few examples, discovered they and the synchronization that goes with them aren’t that bad. The libraries for threading in Python are straightforward, easy-to-use, and generally lack gotchas. After having had enough of Eric Schmidt’s creepy brand of socialism, I thought “how would I write a search engine?” You need a web crawler (threaded), something to extract links (also threaded), and a place to put and index all the data. The first two, at least, are almost trivial in Python with the urllib module. Just yesterday, while pondering how to build GUIs the *nix way (with a front-end and back-end and communication in between), I finally grokked the client-server model of IPC, using sockets. At each step, you literally feel your brain getting bigger, from both the ‘hey, I get it now’ feeling and from all the possibilities that present themselves that you never would have considered before. I can’t imagine doing ANY of this in C++. I would have thrown my hands up in disgust and gone off and done something like…mechanical engineering *shudder*
Sometimes I wish I did like being unproductive. That would make it a lot easier to separate my work and home life — just leave work at work; never, ever think about it at home. I long ago realized that, due to my monomaniacal focus, if I wanted that kind of separation (and especially if I still wanted to be well-paid) I would have to find a job that I thoroughly despised. Unwilling to devote that many hours to something I don’t like, I have the next-best compromise — a job where the deadlines are usually very reasonable, which gives me plenty of time to optimize those things which need it, and gives me some slack time both on the job and off between major projects.
@hari:
C is a language I like a lot. C++, not so much. C programs tend to pretty straightfoward, unless someone is going out of their way to write obfuscated code.
I agree with Patrick Maupin’s hypothesis regarding the expressiveness of languages like C++ and Perl: these languages tend to attract people looking to do “cool tricks.” The problem is that most “cool tricks” make for hard-to-read code. TMTOWTDI often means that you end up with lines of code that resemble line noise. That’s a big reason why I like Python; Guido and his band of merry Python hackers go out of their way to make sure that there is one — hopefully only one — obvious way to do it.
I’ll need a strong cup of coffee to wash down all that pretentious condescension, Patrick ;)
You, unsurprisingly, are almost 180 degrees out of phase with my reality. The ‘pissing contests’ I speak of invariably have no aspect of context to them. They simply seek to establish that “Java > C” or “Python > Perl”. They are silly.
Yes, I have a day job. It pays very well. Because I am an expert engineer. My expertise stems from the fact that I have spent many years studying such languages (and their relationship with software engineering) at considerable depth…as mathematical artefacts, for example. My love of software engineering is fundamentally a love of math.
It is said that a bad craftsman blames his tools, yet it is also true that expert craftsmen have much respect for the finest tools…hence my comment regarding IDEs. My wetware bandwidth is, sadly, finite, and I do not wish to waste one soggy neural flop on remembering every operation of every software entity in every API on every platform…code completion FTW! Syntax highlighting? ahhh…my aging eyes bless the day you were born. People don’t pay me to be a walking encyclopaedia of APIs….they pay me to think and create.
I want to free my brain to spend as much time as possible thinking about abstractions, then have excellent tools assist me in crafting concrete representations of them in a suitable language (or multiple languages) appropriate for the target environment[s].
I don’t evangelize particular languages, but I do take the time to understand exactly at what level of abstraction they enable me to craft software, and hence what they are bets suited for. The fact that humans can make a dogs dinner out of any language is hardly a potent criticism of the languages themselves…here’s an idea – try being a better software engineer and you won’t make such a mess of things. A radical concept!
It depends on how they are presented. One of the Python mantras is “there should be one obvious way to do it,” in direct and reactionary contrast to perl’s “There’s more than one way to do it.”
But, what you call inconsistency and redundancy is what allows for the kind of extra expressiveness that makes it a PITA to debug someone else’s code. Even the cringe-inducing syntax doesn’t usually have to be used, if you restrict yourself to a reasonable superset of C. But people who like C++ often do so because, for example, they like being able to express themselves in just a few characters of modem line noise.
Humans, in general, only deal with alphabets and rules of a certain size very well. When you get below that size, expressiveness suffers; when you get above that size, understanding suffers. But everything is a compromise, and the question is whether the expressiveness is worth the extra difficulty in understanding. Obviously, there are other considerations that go into the design of a language besides the decision of how many key words and how many special symbols the language is going to have, but don’t discount the size decision — it is very important, and one that (IMO) Python manages to hit the sweet spot on.
> No, they are good things….but…aren’t Python (and a bunch of other languages) just as flexible and expressive as C++?…without all that interaction
> amongst its conglomeration of ‘features’?
Absolutely yes. Seems to me that this would make them *more* “expressive and flexible.” Which is good! :)
That’s a big reason why I like Python; Guido and his band of merry Python hackers go out of their way to make sure that there is one — hopefully only one — obvious way to do it.
I like this property too…for projects where Python is well suited to the task at hand (I’ve used it increasingly over the last few years). However, as in life, there are often many ways to skin a cat, and preempting all but one of them can itself be a flaw at times.
@Dan:
Well, the comment about IDEs was meant to be more flippant the condescending, but I do have to admit that I hate IDEs. In my opinion, an IDE is a crutch for a bad language. Some of the other things you mention, like syntax highlighting, are indisputably nice. Code completion usually gets in my way and is done wrongly/badly. Python usually doesn’t need it. I am not an “encyclopedia of APIs”, but if I need something that I don’t use very often, I will read the doc, because automatic templating will probably screw it up for me. I spend most of my time writing my own stuff, not calling other stuff, so that’s not really a huge time hit.
As far as language comparisions go, I’m sorry, but it is eminently true that “Python > perl” for most classes of problems. If you are writing a throw-away 20 line text processor, then “perl > Python”. But if it’s not really throw-away, and it might need to be extended later, then “Python > perl.”
This, in my universe, is a fact. I understand there are other universes out there, and for somebody who completely refuses to learn Python because of the whitespace, or whatever, the rules of physics in his universe may indicate that it is immutable that “perl > Python.” But that’s not a universe I inhabit. So I understand how/why people get into these animated discussions. On the few occasions when I decide to engage in a language war, it is usually because I am trying to pull people out of their comfort zones and get them to try something new. But most of the language wars I have seen are between combatants who all refuse to leave their own comfort zones and think that everybody else ought to change to suit them.
Dan:
> here’s an idea – try being a better software engineer and you won’t make such a mess of things. A radical concept!
Ah, if only every hired programmer was a good enough software engineer. The problems I’ve seen with languages like Perl (which I happen to enjoy in my day-to-day) and C++ is entirely on the heads of the folks who aren’t good or knowledgeable enough to keep line noise from their code.
In the long term, code maintenance is essentially communication. Can Person A communicate clearly to Person B in Language C, given a natural sideband signal loss increasing over distance (time)?
Perl, C++, and other canonically noisy expressive languages potentially make that sideband loss far, far more egregious, in my experience.
>Even the cringe-inducing syntax doesn’t usually have to be used, if you restrict yourself to a reasonable superset of C.
I agree. If you stick to a certain subset of C++ you can write readable code. For me, this subset was, well, C. This is why I tell people “I learned C by learning C++ and then forgetting most of it.” I wish C had function and operator overloading, but otherwise it has as much as I’ll really ever need to use.
>But people who like C++ often do so because, for example, they like being able to express themselves in just a few characters of modem line noise.
The thing is, I don’t call that “expressive.” It has to express itself to the reader as well. If you start a conversation with the words “blargle argle ook ook,” I don’t care what that means to *you,* you still haven’t expressed yourself effectively.
That being said, this is a case of two definitions for the same word, not genuine disagreement. :)
In the long term, code maintenance is essentially communication.
I think I can agree with this. Given that a language is a medium for expressing a design, that expression must of course be understandable by a compiler/interpreter (or it’s useless!) but also by other people – if it is to be maintained/collaborated upon.
Right around now is when I typically encounter “code = documentation” flames ;) when I say that the person-to-person communicative quality of source code is built from more than just the language itself. Good inline annotations serve to explain intent, or to help untangle those ‘clever’ bits of C/C++ hackery.
I can look at pretty much any piece of code and understand what it does – the language is self describing in that sense – but what I cannot necessarily gather from raw code is the rationale behind it. Why did the author craft it this way? Is there a larger significance to this pattern that I don’t appreciate? The expertise required to emit good documentation artefacts alongside code, is such a valuable asset. I don’t write code – in any language – that appears to be ‘modem noise’ (nice turn of phrase btw ;) . I write code to be as plainly understandable as possible – the compiler can take care of the rest :)
I have no doubt that C++ is loaded with expressive power. That’s great for the programmer who’s trying to establish himself as a l33t h4x0r. For the poor slob who has to come along and fix the bugs in his code, though, it’s nothing less than unmitigated pain.
The major cost of a piece of software is not in the creation, but rather in the maintenance. Why optimize for the former at the cost of the latter?
I write code, any more, with the idea at the forefront of my mind that I’m going to have to come back to it in five years and answer the question “why the ever loving fuck did you write it THAT way?!”. Cute, clever, expressive power is almost always the wrong way to approach the answer to that question. Simple, straightforward, clean, clear, understandable code wins every time.
Show me how to do that in C++ and I may consider writing in it for those things that it is good at. As it stands, though, the prime example I have to work with is an advertisement for C. The same goes for Perl. I know it’s possible to write clean, understandable Perl code. I’ve done it. I’ve also hacked on far too much Perl that looks like the output of a random ASCII code generator. Why should I subject myself to that, when Python is much more readable?
If you want a write-only language, don’t use half measures like C++. Go get APL and be done with it. Just don’t inflict it on the rest of us.
@Patrick Maupin:
I think you think that because the most popular IDEs are *cough*VB*cough* crutches for bad languages *cough*dotnet*cough*. :) Check out PyDev. I’ve used it off and on; the thing I like most about it is its wonderful Python debugger. But then again, I typically prefer visually-oriented debuggers over command-line debuggers.
@Morgan:
I actually got a free license to the WingWare IDE after sprinting at PyCon a few years ago.
I might have installed it, and said “oh, look — buttons”
The thing is, whether through luck, skill, or simply not tackling hard enough problems, I’ve never really needed a Python debugger.
That’s not to say that I never had a problem in Python that one skilled in the use of a debugger couldn’t have tracked down more quickly than I did. However, it is a statement that the time it would take me to learn a Python debugger would be unlikely to pay for itself. This assessment is not without experience. I have used debuggers and emulators in the past. Currently I have an oscilloscope at my desk I use on a regular basis. I write testbenches and use verilog simulators on a regular basis.
But the first thing I reach for when confronted with a problem is the source code. Back in 1984, one day I went into my boss’ office and asked him to show me how to use one of the Z80 emulators in the lab. He gave me a really funny look and asked me what I had been doing during my year and a half of employment there :-)
I’ve been a professional C++ programmer for, oh, about 15 years now, ever since C++ was the “hot new thing”. I have to say the language is a pile of crap. It makes performance-vs-ease-of-use trade offs that barely made sense in the 1980’s and make zero sense today.
The main reasons I use C++ are:
1) The available selection of third-party libraries is second to none.
2) High pay because the language is so hard to understand.
3) Occasionally, performance.
It’s possible to write good programs in C++, but you have to be an expert to write a large C++ program without the code imploding into a black hole under it’s own weight. In contrast, it’s slick and easy to write good large programs in, for example, Python.
I’d say C++’s primary reason for being is OO polymorphism and generic (template) programming. But C++ doesn’t actually support either of those things very well. To succeed in using those features you have to write lots of boilerplate code (clone member functions being the most famous example), use “required” third-party libraries like Boost’s pointer containers, and know what insane land mines to avoid such as lack of multiple-dispatch function calls, and no obvious way to force a template symbol to resolve at template instantiation time instead of template definition time. Then there’s lack of reflection which continually drives me nuts. Every time I add a member to a class I have to remember to update several other redundant pieces of code wherever the idea of “do something to every member of the class” is needed. In Python I’d simply run the class member list in a loop and never touch that code again. Then there’s the pages of error messages that fly by when you make a minor mistake like, declare one variable as non-const when it should have been const.
But don’t worry, a few minor problems with C++ might be fixed next year with C++0x. (But not any of the problems I listed above).
I could invert a matrix with a single line of code (because matrix-inversion is a built-in operator).
Good times.
Maybe using APL back in the day is what kept me from running away screaming when I discovered perl. I’m not sure whether that was a good thing or not.
As much as I want to like PyDev, i find it at best a luke-warm IDE. It’s confused far too often.
If you don’t want to go back to smalltalk just to see an interesting IDE, try IntelliJ for java. While this definately counts in my mind as a crutch for a defective language IntelliJ makes every other IDE i’ve ever used seem like i might as well just use the command line to edit and compile my stuff.(“echo ‘public static void maon(string args[]) {‘ >> main.java” Aaah bugger typo… “cat main.java | sed -c ‘5/public static void main(string args[]) {/’ > main.java”)
Apparantly they’ve release a python ide (pycharm) thats similar but i haven’t used it yet.
*cough* -c = -e *cough*
i always make that mistake.
@techtech – I have made use of C++ professionally since its earliest entrance into the marketplace…beforehand I was mostly using assembler, C and COBOL.
It was the conceptual model of ‘objects’ that most intrigued me. They were the first clear example of higher-level abstraction beyond assembler that I had witnessed. Other languages simply ‘wrapped’ the functional transformations of assembler with a more ‘human’ veneer – which was/is fantastic – but to be able to think about systems in an entirely different way was very startling…and it led me, over the years, to indulge my interest in the mathematical formulation of such languages.
For the record, my favorite language is Ada95…jury is still out on the recent incarnation
I currently enjoy the freedom to use many languages to accomplish my projects. Python is looming large, followed by Java, with C++ nipping at its heels.
However, in contrast to the seemingly-fashionable C++ hatin’ going on around here…I have never had a problem writing clean, leak-free (hat-tip to the awesome Valgrind) code that I have never had problems revisiting, extending or reworking. I have always taken great efforts to ensure my code is hospitable to all.
It must be said, however, that I am atypical of ‘certain’ C++ programmers, in that I do not make use of the full range of C++ language features. I basically tend to stick to a ‘C with objects’ model unless I truly need to incorporate deeper features. I also make judicial use of Boost where necessary. As soon as I can practically split my architecture from such low-level code, I do, and move to more practical higher-level languages to build surface components. I am very grateful to the designers of such languages for giving me such tools to simplify my work!
I much preferred Modula-2 to Ada (though I never looked at Ada95). It was interesting to see C get enough better, between ANSI-fication and compilers getting smart enough to ask “did you *really* mean to do if (i=1) or did you forget the ==?”, for these other languages to fade away. C already had quite a bit of momentum at that time, but its position was solidified by adopting just enough of the best ideas from other languages to keep them at bay.
Once C achieved critical mass, it was inevitable that something called “C” would be enhanced. Personally, I think one of the reasons that C++ became popular was the name.
There’s a quote I remember from a decade and a half or so ago, when a couple of contenders were fighting it out for top networking marketshare:
“I don’t know what the next interconnect technology will look like, but it will be called ethernet.”
@Morgan Greywolf: Not if you are writing in Modern Perl (see also Higher-Order Perl).
If you want whipituptude, TMTOWTDI is very useful…
Gah:
This sounds like a recipe for less readable, less maintainable code, not more. Taking advantage of CPAN is especially deadly, as it’s the closest thing to DLL hell you’ll find on Unix.
Dan Says:
> I basically tend to stick to a ‘C with objects’ model
Which is fine as far as it goes. C++ carefully used can be nothing more than a slightly-better C.
As soon as you try to use real polymorphism with those objects, with a complicated inheritance hierarchy — supposedly the main reason for C++ to exist in the first place — that’s when the real hurt beings. When I learned to use Python objects it was like a breath of fresh air compared to C++.
Huh…it seems the WordPress fairies ate my last post….
…that’s when the real hurt beings
Of course, it could be said that “if it hurts, you’re doing it wrong” ;)
Personally, I have rarely encountered models that seem to demand multiple inheritance that couldn’t be well modelled with a ‘multiple interfaces on a container’ pattern…
> This sounds like a recipe for less readable, less maintainable code, not more. Taking advantage of
> CPAN is especially deadly, as it’s the closest thing to DLL hell you’ll find on Unix.
If that’s true, that’s saying a lot of good about unix. In my experience, CPAN doesn’t take much getting used to and it isn’t particularly hard to use. Getting duplicates of packages lying around is theoretically possible but, for some reason, doesn’t seem to cause problems.
With all this C++ bashing (discussion?), I find that the conclusion I’ve come to : there is no place for C++ seems consistent with others:
Use a dynamic language(my favorite is Python) for most things, and drop down to C (actually, usually Cython for me) if you really need to get close the metal for performance or access to device drivers or whatever. Part of that conclusion is due to the fact that if you try to use C++ with all its features, you lose some of the close to the metal stuff (complete control of memory management, pointer slinging, etc) that C gives you.
An interesting observation to me that is there still is no even semi-standard n-dimensional array or linear algebra library for C++. Why is that? it’s a gaping hole for scientific/computation programming. I suspect it’s because it’s so hard to write a good one C++, and even if you do it won’t be compatible with anything else (due to static typing thing, and the hiding of pointers by higher level objects).
But a whole new question: What about D? it looks like it’s trying to be a C++ without the warts — i.e. an extension to C that adds the useful programming paradigms without all the ugly cruft. Any folks here have thoughts on it?
@Jay Maynard: If you don’t have admin rights, and want to install CPAN module, use local::lib; you wouldn’t have nor need multiple copies of CPAN modules lying around.
But that gets more and more off topic…
At first glance, it looks like a cross between C and Python. That strikes me as a good thing. My concerns about adopting it would center on bindings for GUIs and the like, and on its possible longevity.
>My concerns about adopting it would center on bindings for GUIs and the like, and on its possible longevity
It looks like one of the design goals of the language is the ability to link to C – certainly the type ontology will support that.
As for longevity, now that there’s an open-source compiler I think that has ceased to be a big issue.
“But a whole new question: What about D?”
I don’t know anything about the language, but, at the bookstore I noticed that “The D Programming Language” has been written by Andrei Alexandrescu. That says a lot….
But wasn’t the next language supposed to be called “P”?
It does? What does it say?
Jay or Eric or Other Gamers,
Phil, on the Don’t Let Them Give You To The Women Thread, wants a recommendation for a good introductory table top game.
Yours,
Tom
Dear freakin’ lord, Eric!
One huge ~3000 file of Python? I shudder a little.
(Python does too — when you put your code in .py modules, the bytecode compiler doesn’t have to recompile the same 3000 lines over and over again. But forget about the machine, have mercy on the brains of the people who’ll need to maintain it after you!)
LS Says:I noticed that “The D Programming Language” has been written by Andrei Alexandrescu. That says a lot….
Jay Maynard Says: It does? What does it say?
He authored another book called “Modern C++ Design”. Back when the C/C++ Users Journal was still being published, all the C++ gurus there fell all over themselves praising that book. If Andrei has switched over to D, either the head C++ guru thinks the language is much better, or maybe he really needs the money. I dunno….
I had a Eureka! moment last night, while struggling with that C++ monstrosity:
I understand OOP. I learned it from Python.
I’ve been struggling with C++ because it takes the underlying elegance and trowels a rail car load of horse exhaust on top of it. Yech bleh pooey.
Eric, when’s Rob coming out there?
>I’ve been struggling with C++ because it takes the underlying elegance and trowels a rail car load of horse exhaust on top of it.
Well put.
>Eric, when’s Rob coming out there?
Unknown. He’s unemployed, so plane fare is an issue.
The upcoming release of PyPy should be mature enough to run your code. It should be substantially faster than CPython when applied to your problem. It has a JIT which should be able to optimize the string manipulations. Benchmarks can be found at http://speed.pypy.org. Memory consumption will be larger than for CPython, but this is being worked on.