In the wake of the Shellshock bug, I guess I need to repeat in public some things I said at the time of the Heartbleed bug.
The first thing to notice here is that these bugs were found – and were findable – because of open-source scrutiny.
There’s a “things seen versus things unseen” fallacy here that gives bugs like Heartbleed and Shellshock false prominence. We don’t know – and can’t know – how many far worse exploits lurk in proprietary code known only to crackers or the NSA.
What we can project based on other measures of differential defect rates suggests that, however imperfect “many eyeballs” scrutiny is, “few eyeballs” or “no eyeballs” is far worse.
I’m not handwaving when I say this; we have statistics from places like Coverity that do defect-rate measurements on both open-source and proprietary closed source products, we have academic research like the UMich fuzz papers, we have CVE lists for Internet-exposed programs, we have multiple lines of evidence.
Everything we know tells us that while open source’s security failures may be conspicuous its successes, though invisible, are far larger.
Typo: “In said at the time of the Hearbleed bug.”
Why is it that being able to pass arbitrary environment strings to bash to cause it to do something is considered to be a vulnerability in bash, rather than in whatever program is passing through arbitrary environment strings?
You could, for example, use the feature as designed to make an “echo” function (for example) that carries your payload. The question becomes, is it part of the open source culture that suggests to people that it is a good idea to have a program have an obscure feature that causes it to automatically execute strings obtained from environment variables, and for other programs (not knowing about this feature) to call this program without aggressively sanitizing the environment?
@Random832
Because the problem is sending strings to bash via environment variables, intending for them to get interpolated as functions (good) while containing a hidden bug that led bash to *immediately execute* any commands contained in those variables after the closing brace of the function it just interpolated.
The CGI specification explicitly defines the “pass user defined data in environment variables” behavior, so servers implementing this behavior are operating correctly. Scripts did not expect to be executing user input based on these variables (and vulnerable ones do not), thus script developers felt no need to sanitize or taint check those variables. The bug lay within bash itself.
Yeah, people seem to be forgetting the very basic fact that you can see the code and change it if you have the time and ability to spare. I really don’t understand why situations like this result in people saying that it’s a failing of open source that bugs occur. I think that it has something to do with the myth that open source means that the code doesn’t have paid man hours being put towards it and thus no reviews of it. Which is pretty silly given that even I (a freelancer currently) get paid to write patches for open source stuff and am almost always allowed by the contract to send them to upstream.
and before someone asks yes that is allowed, the people who I work for generally do not redistribute this stuff so even considering GPL it’s within their right to pay me to write a patch for it for their internal usage. However I can count the number of times that has actually occurred on half a hand because companies have very high incentive to get their stuff merged upstream, not only is it free maintenance but it also frees them from conflicts that scale linearly with their patchset. Which makes the arguments for proprietary software that much stupider.
Granted, it really isn’t helping matters with the way the FSF is going to the other side of the extreme with stuff like http://www.fsf.org/news/free-software-foundation-statement-on-the-gnu-bash-shellshock-vulnerability and twisting it into something political and a feature when it’s not appropriate. That crap is what got me disillusioned with them in the first place.
@BJP:
Which was around first, bash or CGI?
>Which was around first, bash or CGI?
bash is much older.
@Lagg:
> Which makes the arguments for proprietary software that much stupider.
Exactly. If someone could wave a magic wand and permissively license the Linux kernel, it wouldn’t matter. Everybody in their right mind, even those who, like Google with Android, make sometimes incompatible patches, wants to push them upstream so they don’t have to deal with re-merging every time.
> Granted, it really isn’t helping matters with the way the FSF is going to the other side of the extreme.
Yeah, but this is not news. Hell, I could have written that press release, and with a couple months effort (using open source components, of course) could probably even write an FSF press release generator that could have written it.
And now, a reading from the Holy Book, from the Lord Stallman’s First Epistle to the Press, chapter 41, verses 1 to 39:
The Lord sayeth…
So endeth the reading.
I think my confusion comes from the fact that the actual vulnerability does not require the attacker to be able to specify an arbitrary variable name (such as “echo”), whereas my suggested version does.
I think there may be an inelegant grammatical construction here:
“We don’t know – and can’t know – how many far worse exploits lurk in proprietary code known only to crackers or the NSA.”
That would seem to mean the proprietary code is known only to crackers and/or intelligence agencies. Given that acquiring source of proprietary products is within the capabilities of both groups, that seems to be a legitimate parse.
That said, I suspect you intended it to be parsed more as, “We don’t know — and can’t know — how many far worse exploits, known only to crackers or intelligence agencies, lurk in proprietary code.”
Not that this changes the thrust of your post, of course, but the presence of two equally valid but semantically distinct parses always throws my mind for a bit of a loop while I try to figure out which one was intended.
@esr:
> bash is much older.
Yeah, it was a rhetorical question. Saying something is secure because it follows a spec, when that spec allows that something to hand semi-random data to something else that predated the spec and was purposely designed to be able to do whatever a given user can do just doesn’t make sense to me.
Looking at shellshock, it seems likely that most programmers would probably classify the bash behavior as buggy, but the assumptions that escalated that buggy behavior into a security violation were actually made outside of bash, in a spec that was written after the buggy behavior was added to bash.
I have no problem with the idea that the bug should be fixed in bash muy pronto, or the idea that going forward, bash should be vetted and expected to not have this particular issue any more, because that is the logical place to fix this security issue.
But I have a huge problem with the attitude that, simply because this is the logical place to fix the issue, bash itself had a security bug. Because even if it was a bug and not somebody’s feature, it wasn’t a security bug until somebody decided that they expected the shell to enforce a security model that it was never originally designed to enforce.
Of course, the very fact that bash is part of an open source stack is what allows the community to decide, ex post facto that bash should now support this security model, and also allows the fixes to be instantly and widely propagated.
What’s more, you can expect that various entities will be testing this new bash invariant from here out out.
I’m somewhat surprised that there hasn’t been a reasonably thorough security audit of more mature tools until now. Sure – the cat command isn’t exactly big on the list of targets, but it also is reasonably stable to the point that a security audit wouldn’t be invalidated in short order by new feature development, either. Other commands like ‘sudo’ and ‘su’ are probably closer to the top of the list.
Aren’t these issues doomed to extinction anyway as encryption becomes ubiquitous and robust., e.g. backdoors and buggy whips will have about the same utility.
@Patrick Maupin
“Saying something is secure because it follows a spec”
Since you quoted me before, please note I didn’t say anything was secure as a result of following a spec. The spec itself was flawed. But specs are the lifeblood of interoperability and that is how web servers implementing the Common Gateway Interface are meant to operate.
A webserver implementing CGI on an operating system that does not use GNU bash is not at risk from this issue. The websame server running on an operating system that DOES have GNU bash is at risk. The same server on the same OS with bash removed and replaced with sash or dash or any other option is not at risk (from this specific issue). The error in bash permitted exploitation via CGI scripts. CGI scripts are not the only vector to exploit this bug. Running sshd alone is enough. Running other daemons are likely also enough. A poorly designed email client could do the same.
A server implementing CGI under any operating system that ceases to populate environment variables becomes a broken CGI server that will not function. This will not fix the shellshock via ssh problem. bash had the bug.
The interesting part of this to me is that in all this time, so few people used the environment-variable-contents-to-enable-function-passing construct that nobody until now copy/pasted a line of source code accidentally onto the end of their function, post curly brace, and saw it execute when they didn’t mean for it to do so. There I would argue this functionality was a misfeature.
@Garrett:
I think there has been a lot of security audits on things in isolation, e.g. privilege escalation in su or sudo. But if you test bash in isolation, you might find this to be a bug, but not a security bug.
@BJP:
I can well believe this was a bash bug. If it was a feature, it arguably wasn’t very well thought out.
But still, it wasn’t a bash security bug, and from one viewpoint, it would practically be impossible to have a bash security bug. Nobody considers an SQL injection vulnerability to reflect on the database program; why should a command injection vulnerability reflect on the command processing program?
Now, the fact that bash didn’t meet everybody’s expectations means that we need to either alter expectations or alter bash. In this case, it’s easier to alter bash. With SQL, the opposite happens: “Doc, it hurts when I do this!” is routinely answered with “Then don’t do that!.”
You could certainly do the exact same sort of filtration that is regularly done for SQL, and that exact same sort of filtration would certainly have fixed the issue. Yes, it’s more than likely a bug, because its utility is not taken advantage of regularly (unlike SQL commands), but it’s extremely disingenuous to keep calling it a bash security bug. The only thing that makes it a security bug is the fact that other programs allow random strangers to execute bash with an arbitrary environment.
Having said that, now that we are busy incorporating everybodys’ expectations into the bash codebase, from henceforth, if there is a regression on this bug, then yes, it will be a security bug owned by bash.
My entire point is that the mindset that says “this is and always has been a bash security bug” is not the best mindset for reducing security vulnerabilities in the future. That mindset requires one to say “I am creating something new” and then to ask “What are my assumptions about the other programs I am relying on, and how do I know those assumptions are valid?”
@TomA:
Although there has been interesting work in processing data while it’s still encrypted, today in the real world, data is always unencrypted before it’s processed. Hence, backdoors are still relevant.
I’ve said it before and I’ll say it again:
The Stallmanites are fucking things up? WHAT A SHOCK!!!
@Patrick Maupin “Yeah, it was a rhetorical question. Saying something is secure because it follows a spec, when that spec allows that something to hand semi-random data to something else that predated the spec and was purposely designed to be able to do whatever a given user can do just doesn’t make sense to me.”
The problem is that, if someone told me last week that there is a program, unrelated to CGI, that will do anything at all with the contents of the HTTP_COOKIE variable, I would have said that is insane. If the functions were stored in “BASH_FUNCTION_name” variables instead of just “name”, that would never have made it into any kind of whitelist the way LC_ALL and LANG do, and certainly no program would deliberately store something in such a variable without intending for Bash to use it.
tl;dr: It is insane to process every environment variable for anything other than the most generic harmless environment processing (printing them, or removing any that don’t match a whitelist, in other words), and nobody should have expected any program to do so.
I’m not convinced the function as-designed isn’t still a privilege escalation vulnerability under some circumstances.
The first thing to notice here is that these bugs were found – and were findable – because of open-source scrutiny.
Given that the shellshock bug has existed since Bash 1.14 (circa 1994) and the heart bleed bug existed for years this is not a compelling argument.
Yes, shellshock was found eventually. 20 years later.
There’s a “things seen versus things unseen” fallacy here that gives bugs like Heartbleed and Shellshock false prominence. We don’t know – and can’t know – how many far worse exploits lurk in proprietary code known only to crackers or the NSA.
Given that the bug was found not by looking at bash source code (which evidently only had 2 eyeballs looking at it for a very long time) but by searching for exploitable bugs…just like in proprietary code.
Likewise heartbleed found via fuzz testing.
How many zero days the black hats have in their pockets is unknown but these two latest significant ones found by white hats have been in open source code using the same general techniques that would also ferret out proprietary code vulnerabilities.
So the fallacy is assuming that there’s some much larger hidden trove of proprietary exploits because the code is not freely available. It seems to make little difference.
Search code for vulnerabilities is an art…a very slow art requiring a lot of expertise. Slamming test servers with fuzz testing is brute force approach requiring far less expertise and leverages what computers are good at.
What we can project based on other measures of differential defect rates suggests that, however imperfect “many eyeballs” scrutiny is, “few eyeballs” or “no eyeballs” is far worse.
Given that these open source bugs had only a very few eyeballs looking at the code there is little to suggest that open source projects, even major ones, enjoy any great number of eyeballs.
Proprietary code has the advantage that if there is a business (or liability) case then someone will pay for those eyeballs and there’s a revenue model to afford such reviews.
>Proprietary code has the advantage that if there is a business (or liability) case then someone will pay for those eyeballs and there’s a revenue model to afford such reviews.
If this actually worked, large-scale audits like Coverity’s wouldn’t find a significant advantage in defect rates of open-source projects over closed.
Aren’t these issues doomed to extinction anyway as encryption becomes ubiquitous and robust., e.g. backdoors and buggy whips will have about the same utility.
NO.
See Apple leaking famous people’s nude selfies
There are bugs that you can exploit simply by connecting appropriately to some Internet connected device. If there are no more shellshock like bugs lurking then we can’t just arbitarily get control of remote devices by sending a magic message to them. But if we can guess the password then we’re off to the races. Assume there is another bug like heartbleed lurking somewhere in a popular library or program then the trick to gaining access is to either look for the passwords in what you exploit or, more likely, to do a phish kind of thing where you convince a user of the target to do something that gets you his access on the target. He just authenticated himself the target computer thinks he’s legit and that he must mean to do whatever thing it is that allows the target to now ask you for instructions on what to do next.
This is (in general terms) how banking trojans like ZeuS work. They keylog as you login to your bank account, fill in a different page than the one you thought you were seeing, and so get you to transfer your money to Russian Business Network Inc in Ukraine instead of Joe Blogs LLC next door.
If this actually worked, large-scale audits like Coverity’s wouldn’t find a significant advantage in defect rates of open-source projects over closed.
Finding defects and fixing them using static code analyzers are great. In this case, however, the issue is in the architecture of linux tools with many small tools chained together via shell commands (and shell hooks). Given that many security defects occur at the interface the more monolithic aspect of other design philosophies offer fewer entry points for this kind of vulnerability to appear or these kinds of support features to exist.
Why should DHCP invoke bash to do stuff? Because it’s the unix way.
The fact remains that you can find bugs in proprietary code the same way heartbleed and shellshock was found and that security bugs in open source code has largely been found the same way the proprietary bugs have been found…and it’s not via coverity scans.
@Random832:
Hindsight is, of course, 20/20.
When bash is invoked from bash, this nifty hack allows reconstitution of functions from the first bash into the second bash. When bash is invoked from something else, well, where did the environment variables come from? In 1992, they probably didn’t come from a different user on a different machine.
You won’t be a good security researcher if you don’t question everything.
I don’t think scripts are allowed to run setuid, so I’m not sure how that would happen without yet another program’s misfeature being involved. But I’m glad you’re questioning it :-)
@Nigel:
> Why should DHCP invoke bash to do stuff? Because it’s the unix way.
This just says that auditing tools need to get smart enough to analyze multiple programs running together. Seems like an opportunity; doesn’t seem like an insurmountable one.
@Random832:
I just realized I may not have been clear when I was discussing hindsight. The question isn’t what you would have thought last week; it’s what you would have thought 22 years ago…
@Patrick Maupin
‘My entire point is that the mindset that says “this is and always has been a bash security bug” is not the best mindset for reducing security vulnerabilities in the future. That mindset requires one to say “I am creating something new” and then to ask “What are my assumptions about the other programs I am relying on, and how do I know those assumptions are valid?”’
I agree fully, just from a different angle. I question assumptions about the specifications we rely on, and the specifications for which we carry backwards-compatible cruft. A webserver implementing a spec as defined doesn’t bother me per se, the spec itself does now though. We need to go through and re-audit every old protocol remaining in use to find the other leftover gotchas that made sense on a cooperative network but surely don’t now.
I never learned to write any web apps since CGIs in the 90s so it doesn’t please me to kill it with fire but precisely that needs doing now. Broken as designed, it must die.
None of that changes that bash immediately executing code appearing in an environment variable where it expected a function declaration yet outside the function definition indicates that this code path received little stress testing.
bash executes stuff it should ignore or barf at with an error. I can understand why RMS might want to blame anyone but GNU bash for this massive fail. But blaming those who implemented broken specs that defined the internet decades ago seems wrong.
When bash is invoked from bash, this nifty hack allows reconstitution of functions from the first bash into the second bash.
The hacker culture glorifies “nifty hacks”. There are upsides and downsides to this.
Shellshock was one of the downsides.
it’s what you would have thought 22 years ago…
22 years ago I was still young but even then I had discovered (the hard way as it usually happens) that being too clever in code has a tendency to bite you in the ass.
> When bash is invoked from bash, this nifty hack allows reconstitution of functions from the first bash into the second bash. When bash is invoked from something else, well, where did the environment variables come from? In 1992, they probably didn’t come from a different user on a different machine.
The feature would have worked just fine with namespaced environment variables, would not have enabled this exploit, and the fact that it processes every environment variable rather than those in a particular namespace was (fortunately, as this means they can contemplate changing it) not documented.
> The question isn’t what you would have thought last week; it’s what you would have thought 22 years ago…
After the concept of environment variables was explained to me, I would indeed have thought it was insane. My entire point is that it’s insane even without the security implications, because those names don’t belong to you, you should not be assigning special meaning to the entire set of environment variables, based on content instead of name.
The whole reason this wasn’t prevented by whitelisting is because LANG is on the whitelist. Most systems prevent user input from defining arbitrarily-named environment variables, because there are loads of interesting things like LD_PRELOAD that they could define if allowed to. Systems do not prevent user input from defining HTTP_COOKIE or SSH_ORIGINAL_COMMAND because remote-defined content is what it is for.
“You won’t be a good security researcher if you don’t question everything.”
The point isn’t whether someone should suspect it happening (because a lot of people do insane things), the point is that it is insane and therefore the program doing it is to blame for the security hole caused by it, rather than the program storing user input in environment variables. This is not a bug in ssh or apache or whatever else, it is a bug (or rather a misfeature, since even without the execution bug it’s insane. Today, bash doesn’t import functions from the environment at all if executed from a setuid program. Was that true last week? Was it true in 1991?) in bash, because it flies in the face of how environment variables are supposed to be used. You’re supposed to assign them meaning based on their names, not their values, otherwise why even have names?
@Nigel:
Cannot disagree. Obviously, 22 years ago, some people were writing overly clever hacks that are biting us in the ass now.
@Random832:
> The feature would have worked just fine with namespaced environment variables
Possibly. But bash_xxx would usually be for bash. The assumption was apparently that anybody could write a bash script using function names of his choosing, and those functions should magically appear in bash sub-scripts. It’s not usually the mindset that I’m going to restrict the names of things that you use.
> The point isn’t whether someone should suspect it happening (because a lot of people do insane things)…
tl;dr — Everybody knows of others who are insane, yet when programs A, B, C, and D stupidly rely on insane program E, the author of E is rightly blamed, not only for being insane, but also for the fact that the authors of A-D stupidly assumed he wasn’t. Meanwhile, no blame attaches to the authors of A-D even though everybody knows some programs were written by lunatics.
I suppose that works. Perhaps bash, like a swimming pool, was an “attractive nuisance.”
> Possibly. But bash_xxx would usually be for bash.
I’m not sure what your point is.
> The assumption was apparently that anybody could write a bash script using function names of his choosing, and those functions should magically appear in bash sub-scripts.
This could be accomplished by putting a prefix on when exporting, and taking it off when importing.
As documented, the feature is this: You define a function foo. You export it. It gets encoded in the environment in an implementation-defined way (i.e. the documentation doesn’t say how). When starting another shell, if it finds this implementation-defined thing in the environment, it imports the function. There’s nothing saying function “foo” has to become environment variable “foo” – that’s just an implementation detail.
And let’s face it, “some shell on some system somewhere might look for environment variables with a certain string in their value no matter what their name is and do crazy stuff based on that” isn’t a threat that anyone can really analyze. What if it had been a different shell, a different string? What if this feature had not existed in 1991 but were implemented tomorrow? It is the height of arrogance for one program – even if it is the shell – to think it owns the meaning of the entire environment namespace, rather than just picking a particular set of names that they’re pretty sure no-one else is already using to mean something else.
@Random832:
> This could be accomplished by putting a prefix on when exporting, and taking it off when importing.
Ummm, sure, but if the functions all live in the environment namespace in any case, that (a) would seem like a lot of extra work; and (b) to use your own terminology, might have actually seemed like the height of arrogance (at the time) for bash to declare certain variable names off-limits for other uses.
> in an implementation-defined way
That’s just asking for a different kind of trouble. The OS defines what valid variable names are, so no matter how you slice it, if you are mapping functions into that space, you could have a name collision with some other program. “In an implementation-defined way” just means you have no way up-front of knowing when that collision is going to happen. That’s just awesome, too.
> What if this feature had not existed in 1991 but were implemented tomorrow?
At least the pre-execution of some code would certainly be considered bad. But that’s just helping to make my point about hindsight. You’re getting awfully worked up over something that happened 22 years ago, and (IMO) not assigning enough blame or credit to others since then.
@Random832:
The thing is, if it was merely “some shell on some system somewhere,” it wouldn’t be in the news and we wouldn’t be having this conversation, thus rendering your conclusion a somewhat uninteresting counter-factual.
> Ummm, sure, but if the functions all live in the environment namespace in any case, that (a) would seem like a lot of extra work; and (b) to use your own terminology, might have actually seemed like the height of arrogance (at the time) for bash to declare certain variable names off-limits for other uses.
It is entirely reasonable for an application to define certain names, that no-one has to anyone’s knowledge used for any other purpose before, for its own usage. Everyone does it. It’s not at all reasonable for an application to go poking around in the whole environment and saying that every variable that has a value matching a certain pattern belongs to it.
> At least the pre-execution of some code would certainly be considered bad.
That’s not the feature, that’s the bug. The feature is the actual feature of storing functions in the environment, and loading them from it.
@ FrancisT – “There are bugs that you can exploit simply by connecting appropriately to some Internet connected device.”
Anything sensitive should be encrypted, stored, and utilized on a non-connectable platform. The greatest threat to privacy is the assumption that it exists in the connected world. Never assume privacy on a connected device.
@Random832:
> It is entirely reasonable… It’s not at all reasonable…
There’s that hindsight thing again.
> That’s not the feature, that’s the bug.
How do you know that? Unless the guy who inserted it said “oops, that was an accident” then it’s entirely possible it was a way-too-clever hack that was designed to allow code injection into a bash shell. The fact that we can strip this out without anybody caring too much (but can’t strip out the function definition stuff) doesn’t mean that it wasn’t done deliberately.