I just released the 2.0 version of loccount.
This is a major release with many new features and upgrades. It’s gone well beyond just being a faster, cleaner, bug-fixed port of David A. Wheeler’s sloccount. The count of supported languages is now up to 74 from sloccount’s 30. But the bigger change is that for 33 of those languages the tool can now deliver a statement count (LLOC = Logical Lines Of Code) as well as opposed to a line count (SLOC = Source Lines of Code, ignoring whitespace and comments)
To go with this, the tool can now perform COCOMO II cost and schedule estimation based on LLOC as well as COCOMO I based on SLOC.
The manual page includes the following cautions:
SLOC/LLOC figures should be used with caution. While they do predict project costs and defect incidence reasonably well, they are not appropriate for use as ‘productivity’ measures; good code is often less bulky than bad code. Comparing SLOC across languages is also dubious, as differing languages can have very different complexity per line.
With these qualifications, SLOC/LLOC does have some other uses. It is quite effective for tracking changes in complexity and attack surface as a codebase evolves over time.
That’s how I’ve used it – to track the drastic reduction in NTPsec’s codebase size. It was also my finger exercise in learning Go. For those of you who enjoy reading code, there are a couple of points of interest…
One is how much of the program’s intelligence is in tables rather than executable code. Adding support for a new language is usually as simple as adding a new table entry and a test load. Some of the internal changes in 2.0 relate to enriching the table format – for example, two things you can now declare are that a language uses the C preprocessor and that C-style backslash escapes are implemented in its string constants. I’ve often advocated this kind of programming by declarative specification in my writings; here you can see it in action.
The contrast with sloccount is extreme; sloccount was clever, but it’s a messy pile of scripting-language hacks that makes adding a new language – or, really, changing anything about it at all – rather difficult. This is probably why it hasn’t been updated since 2004.
Another point of interest is the way the program uses Go’s CSP concurrency primitives. It’s a pattern that could generalize to other ways of data-mining file trees. Walk the tree, spawning a thread to gather stats on each file; each thread writes a summary to one single rendezvous channel; the main thread reads summary blocks off the rendezvous channel and aggregates them into a report. There’s no explicit lock or release anywhere, all the synchronization is supplied by the channel’s atomicity guarantees.
That’s pretty simple, but as a readily accessible demonstration of why CSP rocks compared to conventional mailbox-and-mutex synchronization its very simplicity makes it hard to beat.
Ironically, one of the source languages this tool written in Go cannot deliver LLOC reports for is Go itself. That’s because Go doesn’t have an explicit end-of-statement marker; counting statements would in principle require a full parse. Something might be done with the assumption that the input source in in the canonical form that go fmt produces.
Plans for future releases:
* Beat Go into not barfing on ABC comments? (We have a test load)
* Tell Objective-C .m from MUMPS .m with a verifier?
* sloccount handles different asm comment conventions. We should too.
* How to weight EAF: https://dwheeler.com/sloccount/sloccount.html#cocomo
* Check that we handle every extension in sloccount’s list.
* Detect inline assembler in C? https://en.wikipedia.org/wiki/Inline_assembler.
Here’s the set of languages supported: ada algol60 asm autotools awk c c# c++ c-header clojure clojurescript clu cobol cobra csh css d dart eiffel elisp erlang expect f# fortran fortran03 fortran90 fortran95 go haskell icon java javascript kotlin lex lisp lua m4 makefile ml modula2 modula3 mumps oberon objective-c occam pascal perl php php3 php4 php5 php6 php7 pl/1 pop11 prolog python rebol ruby rust sather scheme scons sed shell simula sql swift tcl verilog vhdl vrml waf yacc.
If something you actually use isn’t on there, send me a code sample with the name of the language in it. The code samples should contain a block comment and a winged comment as well as code comprising at least one full statement. “Hello, world” will do.
One thing that might be worth considering is some kind of metric for complexity of the statements, though I really have little idea how you would do it. After all, a simple constant assignment is less likely to be problematic than a complex equation, or even opening a for block with side effects in use intentionally.
>One thing that might be worth considering is some kind of metric for complexity of the statements, though I really have little idea how you would do it.
And applied to whole programs there is cyclomatic_complexity. The general problem with all metrics so far discovered that are more complex than LLOC is that they don’t actually predict outcomes like defect rates significantly better than LLOC.
> While they do predict project costs and defect incidence reasonably well, they are not appropriate for use as ‘productivity’ measures; good code is often less bulky than bad code.
A semi-famous story has to do with Apple management instituting a “lines of code written” field on the weekly report that all programmers were required to submit.
Bill Atkinson, the author of QuickDraw, thought this was a stupid metric, so when he figured out a technique that not only sped up QuickDraw by a factor of six, but also eliminated 2,000 lines of code, he naturally turned in the form with “-2,000” in the lines-of-code field.
That story is available here:
https://www.folklore.org/StoryView.py?project=Macintosh&story=Negative_2000_Lines_Of_Code.txt
folklore.org is a great source of tales from the early days of Apple, and the Macintosh project in particular.
Does it support QuakeC?
>Does it support QuakeC?
No. Until I websearched for it just now I had no dial this language existed.
Looks like it has C syntax. While extension should one expect on QuakeC files?
Can it provide a percentage of LOC copy’n’pasted without thought from Stack Overflow by ‘highly qualified’ indian code monkeys? Gotta love that ‘indian gold’ ;)
I bet that would correlate tightly with defect rate….
What’s stopping you from a doing a full parse of Go when Go has a Go parser right in its standard library?
How much Go would a Go parser parse if a Go parser could parse Go?
>What’s stopping you from a doing a full parse of Go when Go has a Go parser right in its standard library?
Zounds. I had forgotten that. Guess I’ll have to see if the AST is sufficiently well documented for this.
I’m not sure what the right thing is for Haskell, but what you’re doing is definitely wrong. Haskell, being purely functional, doesn’t have “statements” at all; just declarations, which can have very complex substructure. Also, like in Python, whitespace is significant for bracketing, though its rules are lot more complex than Python’s. It’s another language where you need a full parse to extract anything meaningful.
>I’m not sure what the right thing is for Haskell, but what you’re doing is definitely wrong.
Is it? Haskell isn’t one of the languages for which I try to count LLOC.
I may have misread the intent of that table then.
>I may have misread the intent of that table then.
The “terminator” member of the go entry is an empty string. LLOC count was only attempted if it’s nonempty. The usual nonempty value is “;”.
As of a couple minutes ago Go is a special case – the terminator fields is still empty but LLOC is computed from the AST.
Do “loccount -l” to see languages with LLOC support.
Regarding your second point in your plans for future releases, in my field files with the .m extension are almost invariably MATLAB functions or scripts. Here’s a sample showing the comment types and what features demarcate the boundaries between statements. I tried to be reasonably thorough but no doubt there’s other tricks that aren’t coming to mind right now.
function syntaxDemoMatlab
%{
A percent sign and a left brace alone on a line begin a block comment
%{
Repeating the same begins a nested block comment
%}
Ending the nested comment does NOT end the top level comment
%}
a = 1 % A percent sign indicates the rest of the line is a comment
b = 2; % Terminal semicolon is optional but suppresses echo
c = 3; d = 4; % Midline semicolon joins statements without echo
e = 5, f = 6 % Midline comma joins statements with echo
g = [',...''', '%;'] % These characters appear in other contexts too
fprintf(['Hello, ', ... Three dots continue execution on the next line
'World: %d %d %d %d %d %d %s\n'], ... AND they begin a comment
a, b, c, d, e, f,
g); % Within parentheses a terminal comma continues the line
end
>Regarding your second point in your plans for future releases, in my field files with the .m extension are almost invariably MATLAB functions or scripts.
Bletch. Collides directly with Objective C and MUMPS.
>Regarding your second point in your plans for future releases, in my field files with the .m extension are almost invariably MATLAB functions or scripts.
I added a verifier that assumes any MATLAB file will contain the string “end” somewhere in it. Is this safe?
I’m not so sure; Objective-C terminates class and protocol definitions with
@end. You should probably check that the previous token isn’t ‘@’ before assuming it’s a MATLAB file. Note that this is kinda tricky, because Objective-C’s grammar lets you put whitespace between the ‘@’ and the rest of the keyword.>I’m not so sure; Objective-C terminates class and protocol definitions with @end.
That’s OK, there’s a different and more restrictive verifier for Objective-C and I have a test load for it. d5xtgr’s rexample is now the MATLAB test, and autodetection seems to be working properly.
Unfortunately not. Scripts only use “end” to terminate flow control structures; if there are no loops, conditionals, etc. the file won’t contain that keyword outside of maybe comments or string literals. This isn’t as silly as it sounds because MATLAB’s paradigm favours avoiding these features in favour of logical indexing and vectorisation.
In fact, MATLAB takes a permissive stance on omitting “end” even for functions unless things get complicated (e.g. nested function declarations). Using it anyway is considered good practice, but the above example will run just fine if you remove it.
Groovy would be lovely (especially for comparison with LOC in Java and Kotlin), but as it has no required terminator (and it has inline closures, which I suspect might be challenging to handle in several languages), it sounds unlikely.
>Groovy would be lovely (especially for comparison with LOC in Java and Kotlin)
It’s on the to-do list. Save me a step: What’s the file extension?
The extension is .groovy.
Last sentence of the second paragraph of loccount (1) contains a mismatched paren.
I’ve bolded the mismatch below:
A line of code is counted in SLOC if it (a) includes characters other than whitespace and a terminating newline, and (b) is not composed solely of a comment or part of a comment. Comment leaders and trailers in string literals (including multiline string literals) in languages that have them) are ignored.
Since this is the ESR version, if the tool is asked to measure LOC of a C program, it simply outputs “You should have written this in Go.”
>Since this is the ESR version, if the tool is asked to measure LOC of a C program, it simply outputs “You should have written this in Go.”
No. The actual algorithm:
1. Roll a d6. On 1-5, say “You should have written this in Go.” Done.
2. Roll a d6. On 1-5, say “You should have written this in Lisp.” Done.
3. On a 6, “Maybe you should have written this in Ocaml?”
Obviously it should handle brainfuck.
Actually that might be rather easy…
Proof of this algorithm’s incorrectness is trivial: The correct behavior would be to output “You should have written this in Rust.”
>The correct behavior would be to output “You should have written this in Rust.”
…but only if you crave pain. :-)
:) Okay, fine. “You should have had a millennial write it in Rust”, then.
…again, only if you crave pain ;-)
Nonsense. Real programmers write in INTERCAL.
M-X Butterfly.
I saw that you had extensions for .php3 through .php7. Does anyone actually use versioned extensions in PHP other than .php3?
I have seen .phtml, .phps, and .phpt are in somewhat common use. .phtml assumes it’s a crappy situation of PHP embedded in HTML, so maybe that’s not too easy for the program to deal with? .phpt is for unit tests — is this relevant to include in loccount?
>I saw that you had extensions for .php3 through .php7. Does anyone actually use versioned extensions in PHP other than .php3?
I’m not sure which I’ve seen in live use, but the extras do no harm.
What does “.phps” mean?
It’s a PHP source file. If a web server is configured normally, it doesn’t execute the PHP source, but it’s supposed to serve up a color syntax highlighted version of that PHP source to the user agent.
The PHP world is… strange. I’m glad that I’m mostly out of it.
My local shop joked that PHP stands for “Perkele! Helvete! Perkele!”, because anyone forced to deal with it will soon be shouting that, whether or not they normally curse in Finnish.
I do recall having seen php3 and php4 files in the wild, in the past, basically forcing specific major-versions of PHP. Version 7 is somewhat backwards-compatible with version 5, and maybe that has died out now.
For what it’s worth though, php6 is probably never going to be seen. PHP went straight from 5 to 7.
>For what it’s worth though, php6 is probably never going to be seen. PHP went straight from 5 to 7.
Noted. I’ve removed that entry.
esr are you a narcissist?
>esr are you a narcissist?
That’s kind of like asking “Are you asleep?” Having the condition more or less precludes answering it affirmatively, So when I tell you “No.” I have conveyed zero information.
Having the condition more or less precludes answering it affirmatively
Exactly the opposite of the truth. One question test for NPD.
Narcissists don’t mind telling everyone they’re a narcissist. What they do mind and never want to do is change.
>Narcissists don’t mind telling everyone they’re a narcissist. What they do mind and never want to do is change.
Well, that’s interesting. OK, then I guess my previous answer conveyed information after all.
This Dilbert pretty much answers your question …
https://dilbert.com/strip/2018-01-26
:)