Six years ago, in The Art of Unix Programming, I observed some interesting trends in the deployment of programming languages. One Christer Nyfält mailed me this morning reporting that he had followed up by collecting the analogous statistics from SourceForge for present time. Here’s what he said (lightly copy-edited as his English is a bit shaky):
I took data from Freshmeat, but its interface has changed recently, so I had to search for how many projects were tagged with the name of each language. My data is from 2009-05-27, a little over six years since you wrote that chapter.
My observations:
C: from 4845 to 8944, 184%
C++: from 2098 to 4824, 230%
Emacs Lisp: from 31 to 60, 194%These are the language you claimed have reached stability, so let’s call a doubling of projects a stable growth rate.
Perl: from 2508 to 3730, 149%
Here we see the noted stagnation of Perl, only 50% growth instead of doubling expected.
Tcl: from 328 to 480, 146%
Same situation as Perl. Also, Ruby has reached similar numbers (469) as Tcl.
Python: from 948 to 3161, 333%
A tripling. We can start to expect Python to pass Perl in a couple of years.
Java: from 1900 to 5316, 280%
Also a big growth, and it has passed both C++ and Perl in numbers.
Shell: from 487 to 1064, 218%
Normal growth here.
So, the losers are Perl and Tcl, and the winners are Java and Python. Two losers that you predicted, one winner that you predicted, and one winner that’s probably due to policy changes by Sun.
I’m interested in your take on this subject. A blog entry from you on it would be interesting. Are there any other trends caused by new languages? Anything surprising? Would Tcl still be listed as a major language in second edition? Will Perl survive in the long run?
Your wish is my command, Christer, especially when you’re making me look prescient. Not that it took a lot of clue to foresee these trends; I’d been watching since ’97 and there was a lot of momentum on them. In answer to your questions…
1. Ruby is probably the biggest disruptor since 2003. For a while there I thought it might do to Python what Python did to Perl, but it didn’t sustain its initial growth surge and seems to be having trouble getting design wins nowadays outside a small community of very hard-core supporters. I’m not knocking the language, mind you – it has some attractive features – but it looks like Ruby is turning out not to be quite enough stronger than Python to take share away from it.
2. Tcl looks to me like it’s on life-support at this point, with the twin iron lungs being Expect and Tkinter. I bet if you dug deep you’d find that’s what most of the new “Tcl” projects are actually using it for.
3. Yeah, Perl will survive – because there’s huge dark masses of legacy out there that we’ll be dealing with for decades. Perl has become the COBOL of web design.
Ruby is a strange case. Take a look at this project: http://god.rubyforge.org/
I can’t tell whether it’s a parody or an ironical self-reflection of the general culture of Macintosh-like “coolness” and “awesomeness” associated with Ruby or simply an extreme example of that, anyway, this culture certainly generates lots of attention. However, it seems, then, that this attention doesn’t mean an actually skyrocketing popularity.
What about PHP by the way? I’d SO like to see it go away, but it won’t, I don’t think anyone’s willing to rewrite Drupal + Views module (with Views Calendar, Views Calc etc.) + CCK module which is arguably the best toolkit to design new web applications with very little bespoke programming, it’s the FoxPro/Access/CA-Clipper of the web, in Python or Ruby. Nor WordPress. Etc. etc.
Meanwhile, I keep seeing all these new languages which are supposed to appeal to scientists who want distributed computing and multiprocessing, except the hardcore ones can’t tolerate a 1% slower solution so they continue to handcraft HPC solutions in hyper-optimized FORTRAN and C, and they seldom post their stuff to any open repository.
You didn’t mention the new trend of semi-scripted languages that run on the JVM, either. There’s even a Jython, Jruby, and others like Groovy and Scala. This is an interesting hybrid driven by the growing ubiquity of web servers with java engines.
Finally there’s a new trend of pushing projects to github and rubyforge, etc and not to use sourceforge. Does Freshmeat accurately represent the alternative repository farms?
>You didn’t mention the new trend of semi-scripted languages that run on the JVM, either.
I’m not getting the impression those are making a lot of headway. One reason for that may be a flaw they inherit from Java – mark-and-sweep GC, which both causes unpredictable latency spikes and means the languages don’t have deterministic destructor timing. CPython gets this right, though sort of accidentally so – makes it more competitive against C++.
>Does Freshmeat accurately represent the alternative repository farms?
A good question. The existence of rubyforge might skew Ruby’s results some, but github probably doesn’t.
Sniff… TCL was fun. Fare thee well
> One reason for that may be a flaw they inherit from Java – mark-and-sweep GC,
The amusing thing about this is that jruby using the underlying JVM gc has better performance that the regular (C) Ruby’s gc, making it the fastest implementation of Ruby, especially for Rails apps. JVM gc tuning has become an art form of its own, especially with the proliferation of JSPs.
I think the “god” project could have been written in any language. I wouldn’t draw goofy conclusions as you have based upon a single data point (oops, I just did that about you!).
> The existence of rubyforge might skew Ruby’s results some, but github probably doesn’t.
I’d argue that most programmers have moved to alternate locations to publish their code and the results may vary dramatically if you include sites like Google Code, GitHub, BitBucket, and Launchpad. As an example, Freshmeat has under 500 ruby projects whereas GitHub has over 20,000 ruby projects (not including forks).
Groovy, which is part of the Grails/Spring stack, might surprise a lot of people.
Another source for this type of metric is Ohloh (which was just picked up by SourceForge, Inc.)
Ohloh Language Comparison
The interesting bit from Ohloh is that it tracks the number of source commits for each language. Since you can then compare by the % of commits, it’s relatively easy to compare. On this basis, Python has left Perl in the dust, and Ruby has caught Perl and roughly leveled off. I’d argue that C & C++ are hard to separate, so they’re still ahead of Java when you combine them.
“the regular (C) Ruby’s gc” is mark-and-sweep too, AFAIK.
It seems that any epitaph of Perl that fails to account for the CPAN (which is larger than all of RubyForge with 7,846, all of PyPI with 6660, and probably a handful of other languages library repos combined), and its growth over that span of time, is probably at least a little premature. Growth at CPAN has accelerated over those years, and the current count of distributions is 15850.
In raw number of distributions, it’s likely that CPAN has grown more than either RubyForge or PyPI, though the rate of growth may be higher with the latter two. PyPI is still pretty new, and didn’t even exist six years ago, so it’s certainly grown very rapidly. I’m not sure about RubyForge and its age or rate of growth. But, if we assume (as I think is reasonable) that a large percentage of PyPI distributions existed before PyPI and were then packaged up for PyPI in a rush once some sort of standard was established, then that rate of growth is likely to be unsustainable without a steady stream of new developers coming to Python.
Ruby and Python are very fine languages, but claiming they’ve killed Perl when Perl still has dramatically more developers and dramatically more code and those developers are also writing new code, is just silly.
And, of course, any discussion that leaves out PHP (as much as we all might like to) is probably missing the biggest growth story during that time span.
> The existence of rubyforge might skew Ruby’s results some, but github probably doesn’t.
I have never, to my knowledge, used a Ruby project from Freshmeat or SourceForge. Of the dozens I do use, about three quarters live on Rubyforge, the rest on GitHub. For my personal projects, I have stopped bothering to work with Rubyforge. GitHub is just easier.
Something else that may skew results: Full-blown Rubyists, for better or worse, tend to create so many projects that they can’t maintain them all. It can be difficult to distinguish the projects with stamina from the clever idea-lets.
> “the regular (C) Ruby’s gc†is mark-and-sweep too, AFAIK
C Ruby’s garbage collection is notoriously poor. There’s even a fork partially devoted to lessening the pain (Enterprise Ruby).
tcl i’ve seen have momentum in C/++ environments for the sheer ease of integration. the universal reaction was “it’s horrible, but it works and it works easy”. for adding-in guis in a previously unix-only script-only environment, it is a good no-risk default.
like shenpen, i’m a bit surprised PHP didn’t make an appearance. it does my head in by having called modules control Scope, but there’s no arguing its enormous uptake in the web-coding world.
but i think the core take-away point is: social factors (including momentum and laziness) are the primary factors in a language’s take-up, not its “worth”
gotta say, though:
>The easiest way to understand how god will make your life better is by looking at a sample config file.
, has to be one of the best amusing doco lines i’ve ever read.
“Perl has become the COBOL of web design.”… ouch.
I have to agree with Matthew King, rubyforge has currently 7846 projects hosted which is a good number … Sourceforge is definitely not the primary repository of ruby programs
>Ruby and Python are very fine languages, but claiming they’ve killed Perl
I haven’t seen anyone make that claim here.
>I haven’t seen anyone make that claim here.
What pray tell does “[Ruby] might do to Python what Python did to Perl,” mean?
>What pray tell does “[Ruby] might do to Python what Python did to Perl,†mean?
Cream off many of its brightest users and grab a huge chunk of what would have been its future application domain. This is not equivalent to killing it.
Yes, doing a more complete analysis and summary (by including data from CPAN, Rubyforge, PyPI, PLaneT, Hackage … Github, Bitbucket, Launchpad, Google code … number and frequency of commits to core language …) plus seeing some functional languages like Haskell and PLT Scheme included would be pretty neat. Oh, and also include Python 3 and Perl 6.
But I get the feeling that that rabbit hole can get pretty deep.
Oooh, could also factor in devel mailing-list traffic; and also non-spam newsgroup posts/day, and relevant IRC channel traffic, and … :)
The absence of PHP points to one flaw in the logic of the survey. The survey aims to measure, not how much code is being _written_ in each language, but how many projects in each language are being _shared_ in public repositories.
Now of course, if that’s really the statistic you want to measure, that’s perfectly fine. But in the modern era, where most of the “interesting” applications run on a web server rather than an end-user’s computer, and as a result (and not without good reason) a lot of hackers see open standards and open APIs as being more important to their interests than widely-distributed source code, it may be the wrong statistic from which to try to draw more significant conclusions about language popularity.
“I haven’t seen anyone make that claim here.”
and yet…
“Perl has become the COBOL of web design.”
COBOL is (for all intents and purposes) a dead language with lots of legacy code still out there. Perl is a language with lots of legacy code, but lots of new code, too (as the accelerating growth of CPAN attests). Which, interestingly, is just like Python and every other language that has ever reached a reasonable level of popularity.
If you didn’t mean to imply that Perl is like COBOL in the one way that matters in this discussion, apologies for inferring otherwise. Given that Perl and Python and Ruby and dramatically more alike in pretty much every way, than Perl is to COBOL, I made an assumption that it seem you didn’t intend. So, what is actually intended by “Perl has become the COBOL of web design”? There is, I presume some meaning other than, “I don’t like Perl, like I don’t like COBOL”.
>COBOL is (for all intents and purposes) a dead language with lots of legacy code still out there.
COBOL isn’t actually dead. Alas. Nor is Perl. Alas. (OK, Perl isn’t as bad as COBOL, so these cases aren’t really parallel; I’m just being snarky.)
I meant “Whether or not Perl is ‘alive’, we’re going to be dealing with the Perl legacy for a long time.”
BTW are there any general lessons to learn from the popularity of PHP? (Why isn’t that included here?) It doesn’t have many advantages, except that it’s beginner-friendly. But beginner-friendliness shouldn’t count for much in such big and well-polished projects as WordPress and Drupal. Either it does count (then why?), or there might be another reason?
The ohlo statistics show that there are about twice as many lines of Java code committed, compared to lines of Python code. This does not mean that twice the amount of Java software gets produced. On the average, I would say that a Python program that does exactly the same as a Java program takes up half the number of lines of code. This means that Python and Java are actually at par with each other.
PHP is not included because it wasn’t included in “The Art of UNIX Programming”, so I had no data point from 2003, and because I see it as a special purpose language for WWW scripts. Current numbers from Freshmeat for PHP gives 4345.
About the Freshmeat versus other sites issue, as I see it a project has matured to release status once it has been announced on Freshmeat. Only having it on a language-specific hosting site seems to give a message that the project is not ready. The case of Rubyforge seems to give a message to me that there are many started Ruby projects, but few that reach release quality, or it might be the case that the Ruby community only writes for the Ruby community.
So, if a language has a lot of activity, but few projects reach release status, what can be concluded? Just that it’s a scripting language?
Also, the fact that there are no real newcomers that have become popular, is it a good or bad thing? Sure we don’t have to learn new languages, but it also gives a message about lack of paradigm shifts.
Well, at least the programming language fauna is good compared to documentation formats, where the future is COBOL, I mean DocBook.
Jacob, what a weird idea. Python and Java are probably similarly wordy, but Python’s syntax is much more proscriptive, so it is likely that more lines of python code == less functionality than the equivalent amount of Java code. But taking a cue from your idea, since Perl’s syntax can be so concise (though some would use less favourable terms for it), it must be 3x the Java, right?
Of course this all ignores the fact that stable projects need less changes than unstable, buggy projects. The ohloh figures are really not indicative of much.
Although not in your world, C# has become more and more of a dominant language in the Microsoft ecosystem. And for good reason, some of its features are very powerful. It is kind of like what C++ should have been (absent GC) had Stroustroup not been such a stick in the mud.
A current story at slashdot links an interesting study at:
http://gmarceau.qc.ca/blog/2009/05/speed-size-and-dependability-of.html
Quantiatively rating a very wide variety of languages on speed and conciseness on a wide set of benchmarks (nothing about maintainabiliy, etc.). Perl and Python are remarkably similar in this study, and lua does strikingly well.
One thing I note is that if you look at Ubuntu (and debian relatives) then you see a heck of a lot more gui work done in PYthon than perl. I’m not sure why the equivalents of pygtk are so much less used to wrap scripts but it certainly seems to be the case.
@C.J. Nyfält re: “So, if a language has a lot of activity, but few projects reach release status, what can be concluded? Just that it’s a scripting language?”
There are so many factors connected to Freshmeat usage the one thing you can only conclude from the data presented is that you cannot conclude anything is applicable outside of Freshmeat!
http://en.wikipedia.org/wiki/Nonprobability_sampling
>I’m not sure why the equivalents of pygtk are so much less used to wrap scripts but it certainly seems to be the case.
GUIs are one of the areas where OO is a big win, and Python’s class system is nicer than Perl’s (which imitates it, but is infected by the grottiness of the rest of the language.)
>http://gmarceau.qc.ca/blog/2009/05/speed-size-and-dependability-of.html
I was going to bring up this one myself. Fascinating statistics and analysis.
“Fascinating statistics and analysis.”
Er, better than the usual stuff, yes. But I wonder whether we might want more from programming languages than execution speed and code size.
For example, what a lot of people want is the easiness of distribution: it should be very easy to turn the finished software into a next/next/OK installer on Windows, given that Windows still dominates the desktop/laptop space. Preferably, it should compile into one executable with some dlls. I think this is what harms the popularity of Python, for example. Perhaps there are ways to easily to do – py2exe and suchlike – but people who want to reach Windows users just don’t see _obvious_ ways to do so, and that makes them wary of the whole thing. It’s waaaay different than the Unix way of thinking about distributing software. (Well, even on Unix, there can be problems of distribution. 3 years ago Rails didn’t work with the version of Ruby OSX Tiger came with. This can be confusing – that you have to change thing X before you install a software. Not too user-friendly.)
Another consideration can be that code size doesn’t always correspond to development time. It only does if you are familiar with that given language (you just type, don’t spend much time looking up library documentation), and if it’s a fairly long-running project where maintenance is an issue.
> GUIs are one of the areas where OO is a big win, and Python’s class system is nicer than Perl’s (which imitates it, but is infected by the grottiness of the rest of the language.)
Moose seems to be the current best-practice OO system for Perl (IMO). In addition to that, have you seen MooseX::Declare http://search.cpan.org/~flora/MooseX-Declare-0.22/lib/MooseX/Declare.pm ?
>But I wonder whether we might want more from programming languages than execution speed and code size.
Of course we do. But good statistics on these two things are valuable and rare; don’t poor-mouth the effort involved in getting them simply because there are N other things we might have wanted to measure.
When it comes to deployed apps out there and running, COBOL is still #1.
Not bad for a 50-year-old language.
Jeff:
Actually, when it comes to ‘deployed apps running’, I’d bet on Excel, not COBOL, for many definitions of deployed apps.
It would be an interesting excercise to take the declarative language living in Excel, fighting to get out, and detach it from the GUI table framework it lives in into something more generic – some sort of plain-text source. But maybe something like that already exists?
“Python’s syntax is much more proscriptive, so it is likely that more lines of python code == less functionality than the equivalent amount of Java code.”
Um, no, not even close. If that’s the case there is something horribly wrong with the Python code, for example trying to use Java-style idioms where they’re unnecessary.
@esr: Funny yoiu should say that about Tcl. In fact, the only two things I use Tcl for these days are Expect and TKinter. And much less TKinter these days, since I think PyGTK is about the greatest thing to happen to GUI programming on Unix ever.
@Shenpen: PHP+MySQL are to the Web what VB and ODBC were to Windows in the 90s, in both ways good and bad. PHP is drop dead simple to learn and actively encourages poor programming practices. Look at the quality level of some of the crappier open source PHP CMS systems, many of which are very, very poorly designed and written.
Everyone writing about Web programming being more interesting seems to forget that Python Is a language that is extremely well suited to Web development, among other things.
You’re neglecting Cocoa, which is probably the greatest thing to happen to GUI programming, full stop.
It has bindings for most of your favorite languages, too.
> Cream off many of its brightest users and grab a huge chunk of what would have been its future application
> domain. This is not equivalent to killing it.
No, its the equivalent of sleeping with Perl’s mother, impregnating her, and having she and Larry raise the bastard child.
Stripping resources is warfare.
I said it was the greatest thing to happen to GUI programming on Unix. Since Cocoa is specific to one particular vendor’s Unix, Cocoa programming is decidedly not Unix programming*.
(*) Yes, I fully realize that there are open source implementations of parts of the Cocoa API, but since there is no complete implementation, most Cocoa apps will not run on other flavors of Unix.
>Yes, I fully realize that there are open source implementations of parts of the Cocoa API, but since there is no complete implementation, most Cocoa apps will not run on other flavors of Unix.
More generally, the world has changed in the last ten years. It is no longer possible for any closed-source code to be honestly described as “the greatest thing to happen to ${FOO} programming”, for any value of FOO. Jeff Read is trolling again.
Agreed, and what I meant, albeit I was trying to be more — diplomatic. ;)
To be completely fair, Cocoa and GNUStep each have played a separate roll in popularizing the model-view-controller model of program design, something else I’ve found very helpful. On the gripping hand, MVC probably would have caught on without Cocoa.
s/roll/role
I wish this thing had a preview button.
Joe Cooper: >>“Perl has become the COBOL of web design.â€
>COBOL is (for all intents and purposes) a dead language with lots of legacy code still out there.
not sure what the current stateofplay is but even 10 years ago COBOL remained the fastest environment for highspeed file processing.
Mike Earl: >Actually, when it comes to ‘deployed apps running’, I’d bet on Excel, not COBOL, for many definitions of deployed apps.
>It would be an interesting excercise to take the declarative language living in Excel, fighting to get out, and detach it from the GUI table framework it lives in into something more generic – some sort of plain-text source. But maybe something like that already exists?
actually excel IS a declarative language, with embedded database.
but for ability to simply drop that to sequential text, yes it’s been done. WingZ. as good as excel, and written in its own macro language. frankly, one of the more awesome technical feats i’ve come across in my life. it died a commercial death 15 years ago, but then, so did BeOS.
Mike Earl: but yeah, when you consider just how MUCH of what we regard as normal commercial let alone financial markets’ normal functioning rests absolutely on this that or the other spreadsheet, i’m 100% behind you re COBOL vs Excel “deployed apps running”.
Eric, I wasn’t trolling up there. The Cocoa APIs and tools really are that awesome — far more intuitive than GTK, Qt, or anything of the ilk. You should try them. :)
GNUstep is great, but appears to be suffering from neglect as (ugh) GNOME appears to be the desktop of choice nowadays. Last time I tried it it wasn’t quite 64-bit clean.
Do you think functional programming languages will start to take a stronger hold due to the abundance of multicore consumer desktops and increasingly cheaper computational clusters?
>Do you think functional programming languages will start to take a stronger hold due to the abundance of multicore consumer desktops and increasingly cheaper computational clusters?
Dunno. As an old LISP-head, I don’t really get why FP hasn’t broken out of its ghetto before this, so I don’t feel I can make that kind of prediction with any confidence.
well, it kinda actually HAS. but it’s called “Excel”.
“…I don’t really get why FP hasn’t broken out of its ghetto before this….”
As someone who’s currently studying the Wizard Book for the first time, I am also puzzled by this. I think it might be because:
1. Imperative languages have a lower learning curve, especially for the casual programmer without much background in mathematics. It’s easier to put in low quality code in imperative languages than functional ones.
2. Imperative languages work well enough.
Due to the emerging abundance of multicores, 2 might change.
However, I wouldn’t exactly say that functional languages are locked into a ghetto. Python has successfully created a hybrid imperative-functional language.
seriously, Excel is a declarative language
don’t be fooled by it being 2-dimensionally represented rather than 1-dimensionally (text stream)
Exactly language and a translation of web pages has always been a serious problem
I add that I am Polish and I use the interpreter from GOOGLE