There is a kind of programming trap I occasionally fall into that is so damn irritating that it needs a name.
The task is easy to specify and apparently easy to write tests for. The code can be instrumented so that you can see exactly what is going on during every run. You think you have a complete grasp on the theory. It’s the kind of thing you think you’re normally good at, and ought to be able to polish off in 20 LOC and 45 minutes.
And yet, success eludes you for an insanely long time. Edge cases spring up out of nowhere to mug you. Every fix you try drags you further off into the weeds. You stare at dumps from the instrumentation until you’re dizzy and numb, and no enlightenment occurs. Even as you are bashing your head against a wall of incomprehension, consciousness grows that when you find the solution, it will be damningly simple and you will feel utterly moronic, like you should have gotten there days ago.
Welcome to programmer hell. This is your shtoopid problem.
Yes, I have a real example. I just spent the better part of three days debugging code to close block scopes in Python, for a little tool that translates Python into a different language with explicit end brackets. Three days! Of feeling shtoooopid.
It left me wondering what it is about some apparently simple conundrums like this that repels solution. I can say that there are certain recurring patterns in shtoopid problems – when you’re experienced enough, you will notice them partway in and get that sinking oh-no-not-again feeling.
A big one is when your algorithm is such that off-by-one and fencepost errors that are easy to make and hard to spot. The defects that go with this are reversed sign on an increment, or trying to step through an array or list or code loop by bumping the wrong counter.
Another recurring shtoopidness trap is near data sentinels – that mysteriously don’t. And a third is any situation where you are mutating (especially deleting) parts of a serial data structure while iterating through it. (Some languages have mutability restrictions in loops to try to prevent this one. The Dread God Finagle can easily contrive a way for you to screw yourself despite them,)
Copy-paste code that is just slightly wrong in a way that is difficult to spot is a common source of shtoopidity. So are undetected namespace collisions. And variable shadowing. And reversed test in conditional guards.
If you ever find yourself staring at your instrumentation results and thinking “It…can’t…possibly…be…doing…that”, welcome to shtoopidland. Here’s your mallet, have fun pounding your own head. (Cue cartoon sound effects.)
Solutions to shtoopidity traps are not conceptually hard, but they’re slippery and evasive. They’re difficult to climb out of even when you know you’re in one. You’re not defeated by what you don’t know so much as by what you think you do know. It follows that the most effective way to get out of your trap is…
…instrument everything. I mean EVERYTHING, especially the places where you think you are sure what is going on. Your assumptions are your enemy; printf-equivalents are your friend. If you track every state change in the your code down to a sufficient level of detail, you will eventually have that forehead-slapping moment of why didn’t-I-see-this-sooner that is the terminal characteristic of a shtoopid problem.
@ESR “There is a kind of programming trap I occasionally fall into that is so damn irritating …”
Thank you. I feel ever-so-slightly less inferior now. If this sort of stuff happens even to The Outlier Programmer, then maybe there’s hope for me.
>Thank you. I feel ever-so-slightly less inferior now. If this sort of stuff happens even to The Outlier Programmer, then maybe there’s hope for me.
Indeed, providing such reassurance is part of why I posted.
Being The Outlier Programmer does not immunize one; it mainly makes one feel more stupid when one succumbs.
(It may lower the frequency, but I’m not certain of that. I seem to hit one of these about every 9-12 months.)
“Being rather cleverer than most men, my mistakes tend to be correspondingly huger.” –Albus Dumbledore
I have often said that the difference between a moron and a genius is the number of people their mistakes kill.
murderer -> conqueror -> god
hat tip Megadeth
This strangely makes me feel underemployed. I have killed a few small companies, does that count? Not through mistakes, but by implementing business software my boss sold, which demanded precise and organized business processes, you know, the kind that actually can be programmed for, and it seemed they worked better on their previous chaotic ad hoc level and could not actually make money this way… when three years later I heard they closed up shop, I felt like I am being Darwin’s invisible hand. Or maybe boss was too good and too greedy at selling. Whatever.
“undetected namespace collisions”
Oh man…if I wasn’t laughing so hard I’d be crying until my eyes bled.
Been there. Lost a few days of my life. Went hunting for the guilty asshat. There should be a justifiable homicide clause for pulling shit like that.
>Oh man…if I wasn’t laughing so hard I’d be crying until my eyes bled.
My work here is done.
Word, brother.
I had a particularly painful one of these last year.
I’d be writing in scheme for quite a lot of stuff for quite a while and then had to do something in an imperative language. The meat of the problem took about a day and a half and then I spent the rest of the week debugging a single for-loop. There were about 995 lines that came out right first time and then these last 5 lines that I just couldn’t crack.
Scheme isn’t a language where I write a lot of for-loops and I just kind of figured that I’d gotten out of practice. I knew how to do it. I knew how to specify and maintain all the invariants and avoid all the gotchas with mutated state. …but I just couldn’t get it right and I couldn’t see why it wasn’t working. …and as I debugged and searched for the problem I ended up with so much instrumentation telemetry pouring out of stdout that I had to get a bit more formal with how I was marking that up as well!
It was so unbelievably painful and a big hit to the ego to realise that I’d become too stupid to even write a humble for-loop anymore! The rest of the code that worked was solving far bigger, more difficult and conceptually intricate problems. This bit was merely maintaining a little index for something.
When I read this post it bought all that pain right back to me.
I’m not sure if there’s a lesson in here about language design and power but I fear that if we try to find it then we’ll end up in a holy flame war!
Isn’t this the canonical argument given for pair programming? One person’s blind spot is (hopefully) another person’s point-of-focused-comprehension.
The other classic “trick” for solving this is rubber-ducking.
(Caveat – I’ve never peer programmed, so I may be talking through my hat. OTOH, I have rubber-ducked successfully. I even have a 4″ tall rubber penguin labeled ibm.com/linux sitting on my desk to help me out in the other role.)
EXPN “rubber ducking”?
You explain your problem to some{one|thing} with absolutely no background in it (but enough interest, possibly feigned, to keep listening). In the course of formulating your description precisely enough for them to understand what the fsck you are talking about, you stumble over the key issue, and often the solution.
(Obvious targets for the explanation are a non-programming spouse or SO, or even an inanimate icon – hence the term “rubber duck”.)
I’m so glad this has a name I can use for it. I have lost count of how many times I’ve done this. Just forcing you to explain the problem to someone else clarifies the problem.
And now I have a practical purpose for the stuffed Tux in my home office.
I never had a name for this either. When I play the rubber duck part for colleague, after s/he has the aha moment I say, “See I’m so good my mere presence solved your problem.”
Agreed; Tux is one of three “ducks” who live in my home studio/office space.
John’s explanation is accurate and concise, but I still much prefer the original source that tells the story.
Paul –
Glad you linked to this – I’ll remember this story (and give proper credit) in the future!
Wikipedia credits this phrase to The Practical Programmer (1999). I first heard the concept as “talk to the bear’ in the book Showstopper (1994), about the development of Windows NT. I think the best credit for the idea at this point is “legend”.
The idea of “talk to the bear” is also in The Psychology of Computer Programming, a book by Weinberg. It was published in 1971. And I wouldn’t be surprised if there were even earlier references to the idea.
I intuited that this was what you meant, having never heard the term before.
Couldn’t agree more. There have been several times I’ve been talking through a problem with a colleague or my imaginary accomplice, and I suddenly stop talking, turn back to my workstation and emit a lengthy “fuuuuuuuuuuuuuuuuudge”. Problem solved.
Having to explain something in sufficient detail can reveal both misunderstanding or [unknown unknown] lack of understanding.
Related: I have lost track of the number of times I helped a slower student “get” something and in the process of explaining it to them, felt that I improved my own understanding of it. Put another way, I wasn’t just teaching the other person, I was also teaching myself.
There’s a reason that the standard method of learning a surgical procedure is “see one, do one, teach one”.
About a week ago I had a “great” idea for an enhancement for this tool we built. It would let us do thing with a lot fewer errors.
Yesterday explaining it to our site mangler I realized that my simple solution was *way* underspecified and it would be about an order of magnitude more work, and essentially create a whole new class of mistakes.
As a manager of software developers, rubber ducking is one of the most valuable services I provide my staff.
Another approach, that sometimes works, is putting it aside, and sleeping on it, or taking a long shower. Often this loosens the cogs enough that insight appears.
Rubber ducking is great too – but I hadn’t heard that term in all of my 50+ years of coding!
I used to call it putting the problem into batch mode and letting my subconscious work on it as resources allow. My first programming tasks in school where in COBOL(yes, you can pity me) using dumb terminals.
I’ve written before about the programmer who preceded me in my current job. He called this kind of error “too stupid to live”…as in how can anyone who manages to live in the modern world make an error that st00pid?
Instrumenting everything on an embedded platform gets a lot simpler once you have a web server built in. Otherwise, it’s time to do JTAG debugging with gdb, and that’s loads of fun.
I try to get a GDB server running on target over a serial or tcp socket as soon as I can. GDB via JTAG works and is great for the initial bring-up but it’s too inconvenient for more detailed stuff and too expensive if you want to run a big zoo of boxes for testing things on.
That’s fine if you have enough space to provide the infrastructure GDB needs to run natively. When you’re working with a grand total of 256kB of ROM, the calculus changes.
gdb over jtag would be a dream come true at the present
This week’s lesson (not from my code):
Do not, under any circumstances, create more than one class with the same name, even if they are in different modules. And definitely do not make one of them a subclass of the other. The resulting code is just too damn hard for someone else to debug. Specifically, DO NOT DO THIS:
a.py:
class Foo(object): pass
b.py:
import a
class Foo(a.Foo): pass
A (now former) coworker left this sort of crap lying around for us, only with lots of modules, and with a bunch of different systems dependent on different modules, so we can’t easily refactor it.
It occurs to me that perhaps a code-wide index of ALL named variables, containing both name and location, would provide at least a sanity check on that sort of thing.
PS: apparently I can create replies with {craig@trader.name} but not {craig.trader@gmail.com}. What’s up with that?
>PS: apparently I can create replies with {craig@trader.name} but not {craig.trader@gmail.com}. What’s up with that?
I have no idea. I didn’t know WordPress could discriminate that way.
I reported that here some time ago, that if I tried to post under my gmail address it would disappear.
I’m not sure but I have major problems getting posts to show up at all. Once I get one to show up it ends up waiting for moderation. Once the moderation is successful further followups in the same post work straight away but this dance is necessary for every post. :-(
How about Fnords for the name. No rational consideration is possible…
The accumulation of moments like this are what hounded me into TDD pastures.
In your experience, how do the trade-offs work?
It seems we’re looking at a cost of ~5 days per year of pain for the non-TDD version. What does this “major pain” factor reduce to with TDD? Is there any corresponding “minor pain” that’s more frequent instead? What other benefits does it bring?
Great questions. I don’t think I have an answer for all of them.
In my experience is isn’t simply and exchange of, for example, 5 days of shtoopidness for n number of days spent with TDD. I can recall several cases where deadlines loomed and being stuck in a shtoopid loop was only “fixed” by a some ugly hacking and incurring an uncomfortable amount of technical debt. Fast forward quite a few years during which some fluency with TDD was acquired, it was much easier to isolate where the source of the agony was. This didn’t eliminate shtoopidness, but subjectively speaking, it did seem to minimize it considerably. Working up against deadlines has been much less stressful. (I AM leery of this being a false sense of security, so I’m usually scanning for signs of trouble.)
This may have also been influenced by an improved ability to estimate the level of effort required to complete the work in the first place. My experience with using TDD has made it easier to scope out how much work I needed to do – Where’s the unique coding effort? Where am I going to need to mock an API or 3rd party vendor? How much of the deliverable do I have actual control over? So I’m less likely to estimate myself into a corner these days. In other words, when tasked with giving an estimate for the amount of work to be done, I think first in terms of how it can or will need to be tested. I find this easier than thinking how I’m going to code something. “How can I break it?” before “How can I build it?”
Perhaps these are the more frequent “minor pains” that have been reduced with TDD.
Thanks Gregory!
There is another factor here beyond bug incidence rates: Confidence in your code.
When you have proper tests you spend less time (even sub-consciously) worrying if your code is really doing what you think it is, and isn’t going to randomly blow up in your face.
This effect is strong enough that even not doing properly rigorous TDD such as writing your functions before the tests still calls it into being.
@ESR … for a little tool that translates Python into a different language with explicit end brackets.”
OT: I’d be curious to hear more about this.
>OT: I’d be curious to hear more about this.
Coming soon. Probably next blog entry.
…to sleep on it. Works every time.
>…to sleep on it. Works every time.
No. No it doesn’t. If that were reliable I’d never get stuck for days.
Maybe not every time, as esr says, but you’re not real wrong, either – distance and time help.
I’ve solved many a problem or come up with a much better refactoring or solution either a day or two on after doing something else, or on the drive home, or anything but staring at that damned code.
That’s what I mean. A brain reboot is effective. I’ll bet ESR’s brain doesn’t reboot when he sleeps. Damn uptime fanatics… :)
>Your assumptions are your enemy
On my cubicle wall, I have printed out in the largest font that Landscape 11×8.5″ will accommodate:
A S S | U | M E
under which is
NVMQAM
PONENDA EST
PLVRALITAS SINE
NECESSITATE
-William of Ockham
these are my top two memento mori.
But it’s not really that I don’t assume. Instead, I assume that we are asses, who will screw things up from time to time, so I have to anticipate all of the different ways we’ll do that, and figure out ways to detect and if possible correct them.
[One of the things asses do is to try to make things more complicated than they need to be, which is why Ockham is there to remind me of that particular form of assitude.]
Amen.
More than once I’ve been working on some code that’s become monstrous and overcompliated, and thought “no, I’m too close in and I’m overthinking this”.
Take a break, come at it fresh, rewrite it clean and simple with a quarter (or less) complexity.
(Also, YAGNI.)
One assumption that bit me hard in my early years, was to assume that reality matched the specs/documentation.
Oh foolish naive twentysomething me.
Always be alert for discrepancies between what docs believe/intend/imagine it to do, and what reality demonstrates that it actually does.
“The map is not the territory.”
I spend a lot of time updating “maps” (documentation) because of this.
Having had to go through the process of developing and certifying interface code to financial services, I have been in this same situation. Only place I found which was more crazy was code talking to an ATM or HSM.
If I would do such things, I would go for The Royal Society’s Nullius In Verba: take no ones word for it. While being the only proper definition of the scientific method, it is also hilarously useful for analysing business requirements. People tell me about a difficult time consuming task they want automated, and with enough investigation, it turns out they were never even supposed to do that, or do that entirely differently, or various other outcomes. They spend hours a week assembling a spreadsheet report that their boss needed at one time 3 years ago but does not even read anymore and if I ask him he is surprised and just tells us all to forget it.
A’propos translating from implicit block (like the indentation-defined from Python) to explicit block (like e.g. { } delimited block in C), I have recently read blog post on similar problem with parsing:
http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2018/05/combinator2.html
(about parsing implicit and explicit blocks in Haskell code).
This one bites shell scripting pretty hard. One of a thousand reasons it’s not really a nice language to work with…
Small plug: I have reflac that is written in shell script, and even at ~152 lines, it approaches the maximum maintainability limit for shell (at least IMO). I’ve considered rewriting it in Python, but most of the work is actually in calling other commands, which happens to be shell’s strength, of course. I have actually had shtoopid problems while adding a couple features to it. Recursive operation and being able to handle any arbitrary file names are big ones.
My largest bash script has currently 1042 lines, however it’s just processing one text file with specific format and nothing else (my private base of movies I’ve watched).
However, I had to rewrite it few times to make it more maintainable.
I’m maintaining several 1k line bash scripts. Some might be even more.
Genuinely curious :) Are any of them public, or just private ones?
reflac is 90% branching, and that’s where it’s become difficult to add new stuff to it… it can be surprisingly tricky, really. It’s not hard to imagine a larger one being more maintainable.
Private.
Mostly they revolve around a single script (just under 1k lines) that runs “workflows” consisting of a list of other scripts to run, and whether to do them in series or parallel (yeah, it does *really simple* multiprocessing in bash).
Several of the 1k scripts were written to create the lists for that script to run–for some reason it was deemed smarter by previous “developers” to make individual copies of the scripts for different projects rather than a single script. It’s been…horrible trying to clean that up.
I should add that the algorithmic complexity is pretty low.
Choosing the tool for the job depends upon the nature of the job. Shell scripts are perfectly cromulent for certain jobs. I have a present project takes html *text* files, strips the unnecessary java display code, cleans and standardizes the tagging in the text, adds tags, and creates epubs. Input: 15272 files. Maximum file count in any one input set is 4129. Ten input sets. Output is 15272 epub files. Maximum line count in one script is 1475.
Check out: mandamus.ca
Total line count in shell scripts: 25110 in 44 scripts, of which about 4K lines are 100 line chunks at the start of each running step, doing setup and copying etc. Then many chunks of ‘for file in *.htm; do…’ . The scripts cut the job into steps where the output can be tested for failures.
The real work is done by calling sed: about 8,100 times! Without looking, I would suspect that 100% are of the form “sed -i ‘s|this \(thing\)|that \1|g’ $file” but with masses of regex parsing in determining the ‘this’ and the ‘thing’. Far too many lines just take care of formatting errors in the text. One file, with a completely different format structure to the others, has its own 625 line cleanup script as part of the main line of scripts.
I tried xsl and awk. I don’t think that there is anything out there which can do *this particular job* better. C might be faster, but it has been *years* since I dabbled in C. (It just took me 20 minutes to find my copy of Kernighan and Ritchie.. and I was fairly sure about which box it was in.. and it wasn’t!). And C is not particularly replete with built-in regex functions.
My last couple of shtoopid problems have involved external (unseen) constraints.
Why didn’t “rsync -a /opt/test1/* /tmp/test1/ work? Turns out that the shell doesn’t like trying to do that, when there are 2015 files in the folder: raising ulimit might help, but dropping the ‘*’ in the ‘from’ phrase works… Hmmm, wildcard expansion *causes* problems…
So I’m quite happy to do this in shell scripts, calling basic programs. They work and can be checked.
Yeah, lots of times. There is a OS/Shell limit to how long the command can be. Had to look it up, but “getconf ARG_MAX” will tell you what it is.
Redhat 6 system:
$ getconf ARG_MAX
4611686018427387903
Cygwin on Windows 7:
$ getconf ARG_MAX
32000
A Devuan system I can’t cut and paste from gives 2097152
This number *includes* environmental variables.
So it’s possible to overflow *that* limit.
*twitch*
21,000 lines of bash running sed 8,100 times with regex parsing?
I … erg.
But it works. The ‘beauty’ of it is (for certain meanings of the word!), that I can fire it up, to run the sets through the steps, and go to bed. If I find that it had snarfled something and is not right, there is a 99 and 44 one-hundredths possibility that the error is a brain fart by the guy n the chair.
But the tagging in the files is lax. There are 21 variants! for one type of entry, which is just an html tag and a section number, and may include name and href tags and extra spaces and periods. These all have to be standardized to a strict format.
Regex is nice. It’s really all search and replace but you can’t do it in Word, especially when there are 15,000 files.
Just don’t drop the trailing slash.
rsync /opt/test1/ /tmp/test1 means copy contents of /opt/test1 into /tmp/test1.
rsync /opt/test1 /tmp/test1 means copy the directory path /opt/test1 to /tmp/test1/test1.
But that only matters on the source path – having or not having a trailing slash on the destination doesn’t matter. I have to check the man page for this every time.
Ohhh yes! Been there, done that, scratched my head a lot!
I personally don’t have to look that up…. any more. I’m sure that I learned it through my finger tips by osmosis!
“It cannot possibly be doing what it is doing” means, in my experience, one of two things.
Usually, almost always, it means your assumptions about what it can be doing (the inputs) are wrong.
Rarely it means there’s actually some bug in the environment that’s making the tools behave wrong or inserting corruption.
(One benefit of the old-school Unix dev system compared to modern IDEs is that that one’s a lot harder to have happen.
Of course, there are also significant costs associated with that …)
“It cannot possibly be doing what it is doing” is logically equivalent to “But that’s impossible”.
Meet the Holmes Corollary to Murphy’s Law.
Murphy’s First Law: Anything that can go wrong, will go wrong.
Holmes’ Corollary: Notwithstanding any rule or law of chemistry or physics to the contrary, if it happened, it must be possible.
The derivation of the ‘law’, resulting from an epiphany, over-hearing one end of a telephone conversation: ‘But that’s impossible!’. The name from Holmes’ comment, “when you have eliminated the impossible, whatever remains, however improbable, must be the truth?”
OT : I’m experiencing problems with Safari failing to submit comments to your blog. FYI.
My natural reaction to such problems is that I should just keep a checklist of shtoopid things to check for.
And then the checklist gets too long to check in practice. Also, some items aren’t trivial to check for, or are time consuming (yes, I could write a test for all these hash code functions for all eighty of these classes, but…).
Ever get the feeling you’re hitting the Dunbar number of potential problems when programming? I’m serious. If so, when? For which languages? A remedy for this – or even the search for one, without finding it – could be illuminating.
For the name, how about “glamour trap?” It seems so easy, but that’s just an illusion to lure you in…
EDIT: Something like “honeypot” could also work, I suppose.
I have a quite complicated Linux server type application that I want to make easy to install and update. Linux suffers from a peculiarly recursive version of DLL hell, which surprised me.
Anyway I wrote some scripts that grab archives from my web space and do the right thing.
And it was frustrating and then embarrassing when in trying to test stuff the update script that was downloaded, because it was changed, kept disappearing, because the greater download overwrote it with an earlier version. Stupid and obvious. In retrospect.
(I’ve been touching Unix and its relations since 1986, but most of my expertise ended up being in Windows. It’s just the way life works out. And the Visual Studio debugger makes many of these shtoopid problems blinding obvious in the first pass, if we’re in a world it understands, which includes Linux nowadays. But better not mention that.)
It has never occurred to me that one can use VS against *nix stuff, but I suppose one can now, can’t one?
(I love having a Real Modern[ish?] Debugger and, wonder of wonders, interpreter, for my development environment.
I’ve done it ye Olde Fashioned Waye with gdb, and … no, thank you.)
In the sysadmin world, this feels like one’s normal operating mode. The most common causes are “assuming a given system actually does what’s documented” and “a previous admin did the systems-equivalent of a monkeypatch somewhere you don’t know about.”
On the code end, I think my most memorable shtoopid experience was the time I typoed a variable access in such a way that it matched a different variable in an outer scope. The typo wasn’t immediately obvious reading the code, and said outer scope was *the calling function*, way out of sight.
(in related news, fuck powershell’s default scoping behavior. Fuck it with a chain-sickle attached to the top end of a space elevator.)
Even worse when the “previous sysadmin” was you two years ago.
I’ve been inside the Intelligence Community and the Defense Community, and it’s almost scary how much of it is held together by monkey patches that have been working long enough to have earned a pension.
Two Year Ago Me was a dumbass. I keep seeing things he did that I have to rewrite.
I worked for a company for a while, then left. Came back 6 years after I left to find a python scripted I’d tossed off in an afternoon to monitor some things still in use.
What was even odder was I could tell what was my original code, and where someone else had adjusted it to deal with changes in the environment.
I’m reminded of a “Life of a Bug” flowchart I saw back in the mid 90’s that included the stages “How did this ever work” and “Oh, that was stupid”.
I have literally written comments saying “how did this ever work?”.
Against code I wrote.
(That made it through QA and was tested and verifiably DID work, too.
I don’t think we ever bothered to expend effort trying to figure out what externality changed that broke it…)
If you ever find yourself staring at your instrumentation results and thinking “It…can’t…possibly…be…doing…that”, welcome to shtoopidland. Here’s your mallet, have fun pounding your own head. (Cue cartoon sound effects.)
Oh, yes, indeed. Been there, done that, got the scars.
…instrument everything. I mean EVERYTHING, especially the places where you think you are sure what is going on.
Excellent advice. This is what I do whenever I find myself in the “here’s your mallet” situation. It almost always turns out that, indeed, somewhere where I was sure what was going on…something else was going on.
“And a third is any situation where you are mutating (especially deleting) parts of a serial data structure while iterating through it.”
This screams “defect attractor!!!!111eleven”. Immutable data structures, iterating backwards, whatever. Nothing is truly safe. I try to avoid this like the plague.
This is basically how I feel about XSLT, which has been an issue for me this week. I wind up spending a couple hours and ending up with a ten- or twenty-line stylesheet. It not being a imperative language, even the most trivial things trip up my attempts to reason about the flow of the program. I’d really like to learn an approach for improving my ability to deal with declarative languages like that.
In the case of XSLT, the correct answer is to stop using it. Go grab your favorite language, find an XML and XPath library, and write it “by hand”. It’s easier, faster, and more maintainable over time. All the bibblebabble about how wonderful XSLT is just propaganda, and here in 2018 it’s easy to see for anybody willing to look past the propaganda that not only did it not meet its promises, but that it did not even come close. There is effectively no way in which XSLT is superior to conventional programming languages in reality.
When you force a design decision on to something from the beginning, with no consideration for whether it’s a good idea, you always end up with a suboptimal product. XSLT is broken in that way on multiple levels. Forcing an XML serialization as the canonical serialization was a bad idea twice over. First, because it’s just a noisy serialization anyhow for anything other than marked-up text, and second, because dealing with XML in XML itself means you better understand how to encode XML intimately because you’re often working two layers down in the encoding.
The idea that it could ever be a “pure functional” transform wasn’t a great idea. This was further exacerbated by the design committee’s incorrect understanding of what “pure functional” actually is; there’s a very real way in which XSLT is the true manifestation of the things that Haskell’s critics often accuse it of. The subsequent features that reality forced them to bodge on are amazingly difficult and inconvenient as a result.
Anyhow, it’s not you. XSLT is a fundamentally shoopid technology. It’s best to just abandon it.
I say this as someone who actually respects XML and thinks that the primary problem it had was people forcing it out of its niche, or using it in situations where it’s just too powerful for the problem. When it’s the right solution, it’s really right, and there are places I see people boning themselves by not using it when they should because of an ignorant revulsion for it.
But XSLT is still a steaming pile that should be avoided. I initially had “avoided at all costs”, but in reality, that’s almost impossible, because avoiding XSLT is a huge net gain; there’s very few situations in which avoiding XSLT can be a “cost”.
Heh! Tell us what you really think!
I’ve been doing a lot of XML translations out to HTML for about 15 years now in JOB1, and other than some PowerShell once to hook into Excel and Outlook, I’ve never really been tempted to break out Java or Python or XXX. XSLT does what I need pretty well there.
I look at XSLT as being somewhat like Perl, honestly. You can get certain things done very quickly, but if you’re not careful, the language will let you produce a lot of unmanageable crap. That’s a bit of a point against it, I suppose.
If your point is that trying to use it for anything past document to document transforms would be a foolish FlusterCuck, I fully agree.
@d5xtgr: If you are trying to look for program flow by looking at the XSLT, that might be your problem. XSLT doesn’t give any sense of a linear programmatic path (Unless you’re doing full-on “Pull Style” stylesheets.) The path is variable depending on the input document. (If you already know that, sorry for the unnecessary explanation.)
No, I understand this. I just can’t mentally model how it works for some reason – perhaps it is more descriptive to say I struggle to reason about cause and effect in XSLT: it is difficult for me to understand what change to the output would result from a given change to the stylesheet, or what change must be made to the stylesheet to cause a given change in the output.
The specific problem that took more time than I felt justified this week was the following: given an input XML document, produce an output XML document where:
– Any
<foo>element whose attributes satisfy certain criteria should be deleted with all its descendants– All other
<foo>elements should be removed, with their immediate children promoted into their place, recursing if such elements are nested– Certain attributes of a removed
<foo>element should be propagated to its children, unless an attribute of matching name already exists on that child– In all other respects, the output must match the input.
This sample input
<foo deleteme="yes">
<bar/>
<bar>
<baz/>
</bar>
</foo>
<foo deleteme="no" myattribute="value">
<bar myattribute="something else"/>
<bar>
<baz/>
</bar>
</foo>
results in this output.
<bar myattribute="something else"/>
<bar myattribute="value">
<baz/>
</bar>
>No, I understand this. I just can’t mentally model how it works for some reason
If it makes you feel any better, I too find XSLT brutally difficult. Take more comfort because this evaluation is coming from someone with as much experience of weird declarative DSLs and text-transformation tools as I have; it’s not you, XSLT really is that rebarbative.
XML: “How would we make a truly shitty knockoff of S-expressions? Hmm… first, you know how people are always complaining about too many parentheses? We’ll get rid of those, but add metric tons of OTHER unnecessary brackets. Then, take away the ability to write executable code, or macros, or…”
XSLT: “But now we want to write executable code. I know, we’ll hack the ability to write code back in, but do it in a way that makes the result even MOAR verbose, confusing, and ugly.”
There’s probably a good conspiracy theory to be ginned up to the effect that XML, XPATH, XSALT, and all their buddies were deliberately designed to destroy productivity.
Lisp for COBOL programmers, as someone described it in this lengthy discussion:
http://wiki.c2.com/?XmlIsaPoorCopyOfEssExpressions
> How would we make a truly shitty knockoff of S-expressions?
Its parent DSSSL was actual s-expressions, and foolishly decided to have a fling with that up-and-coming XML.
(Not that DSSSL was much easier to use.)
Thanks for the pointer; I had somehow missed the rise and fall of that technology. Saved for later study — as it happens, I’m actually working on a Scheme-based web framework in my copious free time.
You are right, but the comparison isn’t really 1:1.
XML *would* be a terrible replacement for s-expressions. But they weren’t, to my knowledge, meant to be used in the same problem space. XML was essentially a redesign of SGML, which was a nightmare to parse. Much of the data expressed in those language is mixed-structure document data.
XSLT is useless as a general-purpose executable. It’s not really meant to be used as one. But if you want to transform one or more XML documents to some other form, it’s pretty straightforward.
Comparing it to LISP is pointless. It’d be like saying HTML5 sucks because Java can do polymorphism better.
Nail-guns make horrible screwdrivers.
I recommend you outsource any concerns involving XSLT to an enterprise Java developer less than 35 years of age. It’s all intuitively obvious to them.
Every “enterprise Java” (or “enterprise” anything, for that matter) developer I know is in his 40s, at least, so that might be kinda tough.
They’re out there; for greater densities of such young enterprise programmers you may have to look outside the Western hemisphere.
The key is to find someone for whom Java, IDEs, and XML — rather than C, command line tools, and unstructured text — are the norm, and who has had little if any exposure to Unix before the late 90s or so. XML is much easier to grok without the disadvantage of having to unlearn all you think you know about text-stream object serialization.
JSON is easier still, but there are no schema, validation, or even encoding standards for JSON.
I’ve noticed a similar effect with Rust. While seasoned C and C++ programmers fulminate about the borrow checker, teens and twenty somethings using Rust — who come from Ruby or JavaScript backgrounds with little exposure to C or C++ except maybe for horror stories about buffer overflows — code circles around the C and C++ programmers even in their preferred language, and their code is that much safer for having used Rust.
There’s a reason why they say programming is a young person’s game.
Yes, it’s intuitively obvious that the first backwards time-traveler has a stopoff to make before he goes back to shoot Wilson.
I am assuming that there was some root element around both the posted input and output (they’re not well formed as is, and you said you had XML, so I must assume they’re there, and not shown for brevity) My example assumes a “root” element around both input and output.
29 lines, if comments and extra whitespace are excluded.
Forgive the sloppy formatting, I’m not good with the formatting in these forums…
<xsl:stylesheet version="2.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" encoding="utf-8" indent="yes" omit-xml-declaration="yes"/>
<!--
Root element wasn't in the example, but it was
implied to be there
-->
<xsl:template match="root">
<root>
<xsl:apply-templates/>
</root>
</xsl:template>
<!--
In all other respects, the output must match the input.
-->
<xsl:template match="*">
<xsl:copy-of select="."/>
<xsl:apply-templates/>
</xsl:template>
<!--
Any element whose attributes satisfy certain criteria should be deleted with all its descendants
-->
<xsl:template match="foo[@deleteme='yes']"/>
<!-- All other elements should be removed, ... -->
<xsl:template match="foo">
<!--
with their immediate children promoted into their
place, recursing if such elements are nested
-->
<xsl:apply-templates/>
</xsl:template>
<!--
Certain attributes of a removed element should
be propagated to its children, unless an attribute of
matching name already exists on that child
CAVEAT: Only handling the @myattribute in this example
-->
<xsl:template match="foo[not(@deleteme='yes')]/*">
<xsl:copy>
<xsl:attribute name="myattribute">
<xsl:choose>
<xsl:when test="@myattribute">
<xsl:value-of select="@myattribute"/>
</xsl:when>
<xsl:otherwise>
<xsl:value-of select="../@myattribute"/>
</xsl:otherwise>
</xsl:choose>
</xsl:attribute>
<xsl:apply-templates/>
</xsl:copy>
</xsl:template>
</xsl:stylesheet>
Well, here’s a story which happened more than once to me. It’s not the same kind of problem ESR described, only vaguely related by feeling incredibly stupid after finding a reason, and the reason was NOT within a program.
It has first happened many years ago when I was first time teaching assistant. A student called me to help him with a code which should work, but hadn’t. I sat next the guy, immediately found the mistake, save, compile. Didn’t work. Tried another one, didn’t work. Within few minutes I started to sweat; I was very young, that was my first year, some of the students were actually older than me, and they all gathered around me looking at what I was doing.
The solution came when I started to wonder why the f* printfs are not showing. I stopped looking at the code and reexamined all the steps I did. When I found out the reasons I started to laugh. The student opened two windows; in one there was vi with a C code, in second a bash shell in which he hit arrow up, compile.
The thing was the student had a program from the previous exercise called the same as the new program and the bash shell window was opened in a directory with a previous exercise. For fifteen minutes I was changing the one program, while compiling another.
Even more pernicious was accident when student was above block quota, though at that time I was more experienced, so by sheer luck I find out why the code which should work, doesn’t, within less than a minute. Or students calling their programs “pi”, “test” and similar :D
“For fifteen minutes I was changing the one program, while compiling another”
Yep, been there. Have never given the same name to 2 different programs since.
Early on I wrote a supposedly simple program that just would not work. 3 days later after steadily escalating importance of eyes looking at it, the bug turned out to be an several-layers-deep nested ENDIF that was off by one line. A valuable lesson in how hard it can be to see a shtoopid problem.
Writing code after a good night’s sleep, ideal amounts of caffeine, mind is bright and effective. Shtoopid error commences. Can’t and can’t find it. Two days later suffering from lack of sleep, hangover or post-chili stomach for the teetotalers here, and a cold coming on. Decide to not take on other, difficult, thinking required tasks, just do something easy you can do on autopilot, so decide to track it down. Immediately spot the problem. Smart-me makes shtoopid mistakes because it is thinking about something else, like the next problem or the big picture, shtoopid-me spots it because it does not have capacity left over to pay attention to anything else but letters on the screen. Funny how this works.
There’s also the inverse experience: when you look over someone else’s shoulder and spot the shtoopid mistake they’ve spent hours or days trying to fix.
You don your cape and strut down the hall.
A salty old programmer referred to these kinds of problems as “intuitively obvious to the most casual observer”.
> when you look over someone else’s shoulder and spot the shtoopid mistake they’ve spent hours or days trying to fix.
Heh. True story.
It’s my first professional computer gig. (Summer of ’76, just as I was crapping out of CWRU.) I’m a “Student Operator” for a Chi Corporation remote job entry lab, which is full of O29 keypunches and 1004 “Data Line Terminals” (a combined card-reader and line-printer). My sole responsibility was to add boxes of paper to the printers and clear card jams, but I often got asked questions about programming or RJE.
So I’m sleepily sitting there, reading a trash SciFi paperback, when this guy in a suit walks up with a big stack of greenbar printout and an extremely worried expression. Ooops, he’s important, can’t blow him off. And his English wasn’t all that good, so that was strike two. He drops the listing in front of me, and I start to page through it – COBOL. Strike three! I had never studied nor programmed in “COBBLE”, all I knew were the snide jokes and stereotypes.
One trick I had learned – start near the end of the listing, go backwards until you find the first error message. (The compilers they used had an annoying habit of cascading error messages, so pretty much only that first one was useful.) I made a stab-in-the-dark guess. Every line of the listing ended in a period, except the one just before the first error. “Uh, isn’t there supposed to be a period here?” I pointed and asked.
His eyes grew wide, and a big smile beamed across his face. “Ah, yes! Thank you, thank you!!” He scurried off to repunch his card. And that was the first and last time I ever had to debug any COBOL.
No cape, no strut. I’ll admit to a bit of a smirk. Mostly glad I was able to not look like an idiot.
Hah!
My first IT job (after fliunking out of U of Illinois) was in an IBM mainframe shop, breaking out and distributing the individual copies of long reports printed on multi-carbon paper.
So for some reason I was working late one night, and there was an operator who I think was studying COBOL, and doing homework while the system chugged. He had a problem he couldn’t fix, and IIRC, he knew I had been studying programming, so he asked me to look at it.. I had zero knowledge of COBOL, but somehow I spotted his error,
This is not at all hard to believe. I never got any formal training in COBOL, and never wrote any programs in it, but from what could see of it, RDML Hopper earned her nickname (“Amazing Grace”) by creating a language that looked a hell of a lot like plain English (right down to each “sentence” having to end in a period) so that even if I couldn’t write it, I could read it, and understand exactly what the programmer had in mind. (This is sort of the opposite of APL or Perl, which are easier to write than to read, and therefore difficult to maintain without copious commenting. COBOL’s syntax makes self-documenting code easy.) That non-technical business analysts could also read COBOL programs to follow the logic was probably the main reason COBOL was the dominant language in mainframe business computing.
I call it “embarrassment factor”.
(Related to previous comments about “too close”, naturally.
Over the Shoulder guy lacks your mental state of what the code “is” doing, and can see where it … isn’t.)
It’s bad when the program doesn’t work when you are sure that it should.
But what keeps me awake and sweating at night is when a program I am *certain* shouldn’t work, does. Makes my teeth itch!
Or, even worse: when a program fixes itself.
…or even worse – the dreaded Heisenbug!
>the dreaded Heisenbug!
You talking about the uncertainty principle, the cat, or the drug lord?
It’s a superposition of all three, of course.
“It’s just timing dependent and involves cross-threading, how hard could it be?”
or actual differences in VM behavior upon hitting a breakpoint.
Almost as bad: When the first fix you try appears to work.
Ugh, I hate these. I sometimes call them “1+1=3” errors, because they feel like you’re staring at lines of code that say the equivalent of:
x=1;
y=1;
z=x+y;
print z
>>> 3
Basically if I’m tempted to screenshot a bit of code and the output, and put it on facebook with the caption, “My computer is possessed by a demon”, then I’m facing one of these problems.
“Who redefined the + operator to add twice?!”
In later math, we learn that 1+1=3 for sufficiently large values of 1.
The feeling of shtoopid is amplified in some cases.
1) When you stubbornly stay up specifically to fix it until 4AM getting nowhere, go to bed in total frustration, can’t get to sleep for another hour just thinking about it, get only fitful sleep. To then, wake up and find the shtoopid problem in under 10 minutes. I name these “bitter sweet shtoopid”.
2) When you build additional instrumentation that takes ten times longer to create and test than the original code itself. Only to find out that the original problem was a shtoopid one staring you in the face. And the entire instrumentation was a complete and total waste of time and effort. I name these “shtoopid bursters”
3) There turns out to have been a faster, smaller, better, brilliant way of doing whatever you were trying to do which you found out long after having gone through the entire shtoopid process. And had you done it that way, you would never have entered shtoopid land to begin with. I name these “shmarter shtoopid warpers”.
The only redeeming quality of dealing with shtoopid problems is that you do get a mini ‘ahah!’ triumph out of it. But they never feel as good as a real ‘ahah’ the best of which is reserved for grasping new abstract concepts. Those taste really good!
Heh. Now I know what to call that PHP form entry problem I ran into last month. (I spent days working over the logic flow and making small changes in the script processing the form data. Turns out the error was in the initial form set-up that didn’t assign the right values to some of the boxes. Lesson learned.)
“…instrument everything. I mean EVERYTHING, especially the places where you think you are sure what is going on.”
This policy can sometimes make the experience a net future positive, even if it sucks horribly at the time. I like to include any extra output used to work a shtoopid problem in the `–verbose` or (more often) `–debug` switches, and just leave it there afterward. I know I’ll thank myself later.
I don’t know how Real Developers decide what to put in those switches, but in my own projects it sometimes feels like half the content of –debug comes from past episodes of shtoopidity. :-(
I would be ashamed to admit how much time I lost because a function in the Win 3.1 API that was declared in the header file as return type BOOL could actually return 0,1, or -1 (which, of course, is true under C’s conventions).
I don’t consider that one entirely my fault, though.
Oooh, Windows 3.1! A delightful circus of obscure failure modes and subtle traps.
A few years ago I decided to get back into Windows 3.1 programming (with a bit of help from DOSBOX). I found that the program I was writing would not display correctly when minimized, and worse, would eventually crash windows.
It turns out that you’re not supposed to pass an hwnd to the GetMessage call in the main message loop. Rather you’re supposed to pass NULL. This retrieves messages destined for any window attached to your current hinstance. This is important because “minimizing a window” doesn’t mean what you think it means. When you minimize, the system hides your application’s main window and puts up a new, small window that displays the icon you registered to it. If you do not pass NULL (or otherwise pump the message loop for that window), then eventually this window’s message queue will fill up and crash Windows.
Fun times. Makes me feel like a kid again.
Not checking for -1 from GetMessage() (which I believe is the trap Locketopus described above) and passing a specific window to GetMessage() are two of the most common traps in Windows programming–and it’s as true today, even on Windows 10, as it was in 1991. (At least they fixed it so Windows doesn’t crash.)
To me all this means the OS is shtoopid, not the programmer. But maybe that’s just me…
> A few years ago I decided to get back into Windows 3.1 programming (with a bit of help from DOSBOX).
There really is something DEEPLY wrong with you.
> worse, would eventually crash windows.
Windows 3.1 did that all by itself.
Eric, do I understand correctly that your debugging is mostly printf-or-equivalents-based? Up until now I thought it’s a problem I’m still doing it that way. Or do you only use that as a debugging/instrumentation device in certain circumstances and not others?
>Up until now I thought it’s a problem I’m still doing it that way.
Oh hell no. I sometimes use symbolic debuggers on C programs, but for interpreted languages it’s printfs all the way down. Linus Torvalds is a notorious refusenik when it comes to symbolic debuggers, says he doesn’t trustthem and won’t use them.
/me uses printfs too :(
/me ‘s preferred programming language doesn’t even support arrays
> but for interpreted languages it’s printfs all the way down
Yeah, but printf what. The most common use of a Microsofty debugger is simply having it automatically launch at an unhandled exception and displaying all the variable values. Now it is perfectly replaceable by putting an exception handler on the top level of your code that iterates through dir() and pretty-prints everything and their values. And then putting exceptions into the critical places, as basically reverse unit testing: when a function is called with unexpected, wrong parameters, throw up, revealing the culprit caller. But are you doing that or really just printfs?
(I find reverse unit testing more useful than unit testing. Unit testing is typically about checking what a function does with unexpected parameters. I don’t want unexpected parameters in the first place, I want to know which function is calling this function with wrong parameters and why.)
In the classical sense, using a debugger means single-stepping through the code. Yeah, who has the time for that. But they most typical use case of a Microsofty debugger is this: when something bad happens, tell me everything about the current state of things.
SFConservancy misleads in “update” explaining GPLv2.
Section 4 is not operative against the grantor of the license, and makes no claim to be so.
It is speaking only of licensees and what might be described as sub-licensees.
Section 0 confirms that “You” refers to licensees.
Section 4 simply states that if a licensee loses his license, that does not cause the sub-licensee to lose his license in-turn.
My worst shtoopid :
I once wrote a Javascript program in which I defined numerical constants. Just to make it nice on screen I aligned them by adding superfluous zeros in front of the small values, like :
const a = 0001;
const b = 0002;
const c = 1024;
etc.
Hell is paved with good intentions.
When I started to test my program all went wrong. I ended up after hours banging my head having logs saying me 1 + 12 was equal to 15 or something like that… The devil was in the machine. And I completely lost my mind.
I finally could get out of this hell when I figured out that in that language numerical constants starting with zeros are interpreted as octal numbers, making my whole computations wrong.
Please never ever again.
That bit me in the butt when I was extracting the month from the current date and doing arithmetic on it. The script failed the first time it ran in August, because 08 and 09 are not valid numbers.
If I could go back in time and kill Baby Hitler, I’d stop on the way back to have a little chat with K&R about this nutty idea of a leading 0 meaning “octal”. I’d have tried to persuade them to use something like Ada’s approach (2#10011010, 8#232, 16#9A, and 154 all represent the same number).
IIRC, dmr has said he’d avoid that particular feature if he had it to do over again.
Besides, octal went out with the PDP-11. :-)
I remember walking down the hallway in college, reading the room numbers, looking for room 110, and expecting it to be right next to room 107.
I don’t think I’d get caught directly by that. But once we had IP4 addresses in a UI and the chap who was paying wanted it all neat and tidy and specified padding with leading 0… that then got passed back into inet_addr. Cue mysterious failures, some careful reading of the documentation and a reversion to standard behaviour.
Almost every time I have shtoopid problem, it’s because I’ve been looking in the wrong place for hours and hours, so when I find myself in that situation, I’ve eventually learned that I need to think about “Where else could the problem be?”
Sometimes stepping away from the computer for a few minutes is enough for me to realize what I’ve missed. There are many times I’ve solved a problem, or at least come up with a new idea for a solution on the elevator ride down to leaving work. The funny thing is that when I have those moments, it’s far more likely to happen immediately after leaving work than on the ride home.
“it’s far more likely to happen immediately after leaving work than on the ride home”
Yeah, I always get them about 500m away from the office on the home straight to the train station. Then the challenge is to hold on to the insight for the next 5 minutes while I negotiate the commuter crush.
I find that speaking it aloud helps with that.
By defining the problem as simple, you may be restricting the amount of resources your mind will allocate to the problem. Any discrepancies will be smoothed over before you see the resolution to the problem.
By talking to the duck, you may focus on the sense data rather than the model. A Doh rather than an Aha!
I find it surprising that Eric has never posted about the mindset behind debugging code.
The mentality you need to even approach the subject.
For me, I find it most useful to create stories to explain why what you expect isn’t what you get, and examine each story for consistency. If the story is consistent, it’ll have certain side effects that you can isolate and examine.
If your story is consistent and the side effects it might produce are detectable and detected, you can then prod the system a bit in a way that’ll create side effects that your story predicts. This is usually a confirmation that the story you’ve concocted to explain the errors is probably correct.
This puts you on the right path. Usually.
Some of this stuff can be easily detected (aliasing, shadowing) and yet we often instrument the compiler to stop bothering us with (mostly) false positives and then shoot ourselves into respective legs.
I can’t but to wonder, why nobody came up with compiler warning out of source ignoring, you know simple switch that tells the compiler that all currently existing warnings are okay, only show me the ones I introduce next time.
-Wno-bother-me-again and -Wno-bother-me-reset or even interactive mode that lets you discard/fix stuff one thing at the time instead of showing you 100s of warnings whenever you dare to use -Wall -Wextra on existing code (without polluting a source code with non-portable object attributes).
I can’t but to wonder, why nobody came up with compiler warning out of source ignoring, you know simple switch that tells the compiler that all currently existing warnings are okay, only show me the ones I introduce next time.
This exists in the Eclipse IDE for Java, and might also exist in IntelliJ, and in various Eclipse plugins for other languages. You can turn off whole classes of warnings, such as variables that are never used (e.g. because they’re debugging variables used in currently commented out code). Every warning is categorized as something, and you can hide that category by unchecking a box.
I generally frown on this. My view is that warnings exist for a reason, and that reason is usually ultimately either a defect, or a defect attractor. If I hide them, I’m just putting off the day when my code will break in a way I can’t hide, and now the fix will be harder. That said, I can sympathize with the problem of existing code written by Someone Else (including You In A Former Life) with potentially thousands of warnings. That code might be currently doing its job; fixing all those warnings could very easily cause defects unless you’re willing to spend effort you could have spent on other improvements instead.
I can also see the value in reporting only new warnings. It gets around the legacy working code problem I just described. “I’m aware of the pitfalls of leaving the current warnings alone, but I’m having to triage here; just keep me from making the problem any worse.”
The main reason I can see this not having been a feature already is that compilers have traditionally been unintegrated with version control. I don’t know of any compiler that automatically digs back and sorts warnings by earliest version in which they appeared (even in nontrivial cases), for example, although I’m ready to be pleasantly surprised.
To solve Shtoopid issues, I’ve found value in asking myself “what would I know if this wasn’t a problem?”. Usually I find and fix the cause within 5 -20 minutes.
I had one that consumed six weeks and my employment at the firm. The RFC for TKEY says that you MUST sign the TKEY that the server returns to the client. That’s nice, but Windows in fact requires that you MUST NOT sign the TKEY that the server returns to the client. Naturally, a completely-backwards clause in the RFC was not the first thing I looked at. And no, printfs didn’t help me, because there was nothing wrong with my code.