I like code-validation tools, because I hate defects in my software and I know that there are lots of kinds of defects that are difficult for an unaided human brain to notice.
On my projects, I throw every code validater I can find at my code. Standbys are cppcheck for C code, pylint for Python, and go lint for Go code. I run these frequently – usually they’re either part of the “make check” I use to run regression tests, or part of the hook script run when I push changes to the public repository.
A few days ago I found another validator that I now really like: shellcheck Yes, it’s a lint/validator for shell scripts – and in retrospect shell, as spiky and irregular and suffused with multilevel quoting as it is, has needed something like this for a long time.
I haven’t done a lot of shell scripting in the last couple of decades. It’s not a good language for programming at larger orders of magnitude than 10 lines or so – too many tool dependencies, too difficult to track what’s going on. These problems are why Perl and later scripting language became important; if shell had scaled up better the space they occupy would have remained shell code as far as the eye can see.
But sometimes you write a small script, and then it starts to grow, and you can end up in an awkward size range where it isn’t quite unmanageable enough to drive you to port it to (say) Python yet. I have some cases like this in the reposurgeon suite.
For this sort of thing a shell validater/linter can be a real boon, enabling you to have much more confidence that you’ll catch silly errors when you modify the script, and actually increasing the upper limit of source-line count at which shell remains a viable programming language.
So it is an excellent thing that shellcheck is a solid and carefully-thought-out piece of work. It does catch lot of nits and potential errors, hardening your script against cases you probably haven’t tested yet. For example. it’s especially good at flagging constructs that will break if a shell variable like $1 gets set to a value with embedded whitspace.
It has other features you want in a code validator, too. You can do line-by-line suppression of specific spellcheck warnings with magic comments, telling the tool “Yes, I really meant to do that” so it will shut up. This means when you get new warnings they are really obvious.
Also, it’s fast. Fast enough that you can run it on all your shellscripts up front of all your regular regression tests and probably barely ever notice the time cost.
It’s standard practice for me to have a “make check” that runs code validators and then the regression tests. I’m going back and adding shellcheck validation to those check productions on all my projects that ship shell scripts. I recommend this as a good habit to everybody.
Good article.
I had noticed some time ago that the JetBrains IDEs (e.g. GoLand, WebStorm, PyCharm) had become really helpful at suggesting improvements to my shell scripts. Turns out ShellCheck is bundled (along with Shfmt and ExplainShell) in the ‘Shell Script’ plugin.
I’ve now installed ShellCheck on my laptop for general use.
> It’s not a good language for programming at larger orders of magnitude than 10 lines or so
Coward. :)
You’d be AMAZED at what you can do in BASH should you choose to throw sanity and caution to the wind and f*king go for it.
I used to maintain a application that consisted of a nearly thousand line BASH script that sourced a function file that clocked in at over a thousand lines. The config file for this masterpiece was about the same size.
It was regularly fingered by about 10 different people.
It is the engine that drives workflows that ingest about a terabyte of data a day, and then runs additional workflows to process all of that data.
A couple different people tried to rebuild this system in Python but couldn’t get it to run fast enough (I suspect most of them tried to get too fancy).
I’ve been using shellcheck for about three years now, it is quite excellent. On a technical level, it’s astounding that it even works at all. It basically has to reimplement parsing rules of shell entirely in Haskell just so it can examine shell scripts!
On a related note, another tool I find useful is bashdb, a debugger for shell scripts with breakpoints, single-step execution, and examination of variables anywhere in the program (like gdb!).
I’d run across ShellCheck probably about a year ago, at then-$DAYJOB, and as I had a project which required some shell scripting, I started using it (via a commit hook).
In the end, I was using this script for host-level install/upgrade/management scripting on a server we did not control, so needed minimal dependencies…otherwise I might have used Python instead.
I haven’t had anything too complex in a shell script that wasn’t better to break out and build elsewhere, but e.g. bash prompt extensions like bash-git-prompt and other semi-complex tools probably should be using BATS, the Bash Automated Testing System.
> needed minimal dependencies
This is a reason I write a lot of shell code, both in the form of actual scripts (saved somewhere on the server on which it’s to run) and “recipes” (snippets I can quickly paste into a bash prompt to get something done quickly and reproducibly).
I remember something Bruce Schneier said about the machines his company places on clients’ networks. They strip out even stock Linux binaries that aren’t actually used by those machines (including shells), because doing so reduces the attack surface. Anything I can do with a bash script rather than installing some binary on which it’s dependent means one less thing InfoSec’s vulnerability scans can turn up for me to remediate.
For example, I hate having to upgrade JVMs just because Oracle has a new version, the mere existence thereof is deemed to render all older versions “vulnerable”. I have better things to do with my time than upgrade a bunch of JVMs every month or two to get off the InfoSec Schidt List. I have an application running on a Windows server that had some Java middleware that I rewrote in venerable (and pathological) cmd.exe scripting language, and then got rid of the JVM and off the Remediation Treadmill. The script isn’t awfully long, but it’s definitely more than 10 lines (probably not more than 100 though). When we got new servers (due to EoL of the previous servers’ Windows versions) and a newer version of the software that script has to interface with, the only change I had to make was changing slashes to backslashes on one line, due to that software changing what it expected to see as input.
I’m looking forward to when bash and GNU tools come standard on Windows servers, so that I can write scripts more sanely (and portably).
If you haven’t already, look into PowerShell. I’ve replaced all of my cmd.exe scripts with it. It’s not quite as terse as the old batch scripts, but it’s able to do a lot more, since it integrates with the .NET classes.
Is there a lint checker for PowerShell?
Not to my knowledge, but if The Monster is writing windows batch code, (s)he doesn’t have a lint checker now, either.
If you ever do run across a lint checker for PowerShell please let me know. I’ll name my next kid after you.
There is indeed: https://github.com/powershell/psscriptanalyzer
psscriptanalyzer reminds me of a recent discovery that makes me happy. I learned about shellcheck about a month ago. The vow: every shell script in every one of my projects will be shellchecked on every regression-test run. That’s how much I like what it does.
I find PowerShell very handy to run an SQL stored procedure, put it into an Excel file, throw a Pivot on and email it to a manager. And since .xslx is now just a bunch of really simple XML files zipped, very transparent and open, now I don’t have that old “I sold my soul to the Devil” feeling with it anymore. My Android phone displays these .xlsx files perfectly. What more openness would I want?
I’d only recently learned about a nifty web-based tool named explainshell. Does what it says on the box – enter a Linux command, get its documentation. I don’t just mean it’s a man gateway, either; I mean you could copypaste any command you’ve seen in a Unix shell script, no matter how complicated, and it will show you what it does, broken down by command line options, split by pipes, handling builtins, etc. in a clean visual style.
Probably the best accompaniment to shellcheck I can think of, short of having Linus himself sitting next to you with an HR intern between to filter out all the cussing.
>Probably the best accompaniment to shellcheck I can think of, short of having Linus himself sitting next to you with an HR intern between to filter out all the cussing.
Harrumph. I’m probably better at arcane shell hacking than Linus is – I mean, even he would probably expect me to be more skilled, given that I’ve been doing it longer and it’s the kind of programming where accumulating tricks by experience matters a great deal.
Now get off my lawn.
:-)
Linus is pretty famously narrow-visioned. He has blinders on that keep him on kernel hacking and … not a whole lot else. Undoubtably brilliant, but I wouldn’t trust him to install a distro, much less maintain shell scripts. It just isn’t in his scope.
It’s shocking the kinds of things that get written in unreasonably-grungy shell when the sheer quantity of paperwork and yak shaving that surrounds the installation of anything else gets high enough.
At my last job, we had a ton of icky ksh and nawk code around, because you could count on those to be there. A lot of it got replaced with Perl over time when we finally got approval to install Perl and some reasonable modules. Python had never made it in by the time I left, to say nothing of Rust or Go.
Rust is too fiddly as any kind of reasonable alternative to shell scripting, though I think Go is posed to take that position rather well. Especially with the golang compiler producing static executables, it makes a pretty compelling Python alternative whose output runs anywhere.
>Especially with the golang compiler producing static executables, it makes a pretty compelling Python alternative whose output runs anywhere.
I’m gradually moving all my Python work to Go, though some of my older Python programs may never make it over the hump. The static binaries are indeed a big deal, but that way of describing the feature almost distracts fron thereal impact – the absence of runtime dependency hell.
The output of Go does not run anywhere. You first have to (cross) compile for the correct target architecture. If you have a Go program on your PC, maybe amd64, you cannot run it on your embedded device, maybe armel or armhf. In contrast, a shell script or Python program would run on any architecture, if the interpreter is installed, i.e. /bin/sh or /usr/bin/python3 with its libraries. What is easier in the end, depends on the use case.
Will introduce this to $dayjob. There are a ton of places where you have shell/sed/awk but no perl/python – pretty much every cut down IOT device with busybox for example – so we have shell scripts that run on them. Some of these scripts are large.
Visual Studio Code Editor also have such a linter of quality as extension for shell scripting that does all the static analysis on the on the go, as well linter per language for almost every language in existence all from the Open source Community as third party extensions.
For most of my scripts, I tend to do:
#!/usr/bin/env bash
set -euo pipefail
shellcheck “$0”
… thereby, among other things, preventing the script from running if it doesn’t pass Shellcheck.
@Duncan Bayne
Thanks for that.
My pleasure :) It’s saved me from a few of my own brain-farts over the years.
I don’t usually auto-run shellcheck on script execute, but I version control and have it set up as a hook with pre-commit.
If you don’t always have shellcheck installed, your command will fail…I’d probably use the following instead, in case I have to run it on machines I don’t control:
command -v shellcheck >/dev/null 2>&1 && shellcheck "$0"I usually also use
set -E; it doesn’t make too much difference in most scripts, but if trapping ERR you don’t have to remember when the trap is not inherited.Rather than trapping ERR, I’m more likely to trap SIGINT and SIGTERM to allow for cleanly exiting, especially if I create temporary files in the script.
Yeah, I’ll take that approach if the deployment environment isn’t under my control.
But in most cases the scripts I’m writing are utility or automation scripts for my own environments (especially to set up new dev machines), so I’m quite happy to make the first step in environment setup installing git, tmux, and shellcheck :)
I think you’re right about -E, I might adopt that as standard practice.
Author of the tool is fairly critical of himself, his code quality and generally of lessions learned https://www.vidarholen.net/contents/blog/?p=859
In this age of essentially infinite computing power, why have the shell languages not improved substantially? Why isn’t Python or similar the standard shell language now? I know their is inertia but we have the resources to have much more powerful and expressive languages as standard now but it doesn’t seem to be happening. Why?
My theory is that it is much easier to design and implement an ergonomic language for one of: interactive use, large(r)-scale programming.
Toy example:
$ ls | grep "\.py"The best equivalent I could come up with in Python is:
from subprocess import check_outputcheck_output(["grep", "\\.py"],
input=check_output(["ls", "-a"]))
Which, arguably, is better in that it is more explicit about what is going on, but clearly worse from an ease-of-writing point of view.
I can easily imagine improvements to, say, bash that preserve the essential characteristics of a shell language and I can easily imagine modifications to Python that make it more suitable for interactive use, but I find it much harder imagining a language that preserves all of the advantages of both.
Closh is an interesting attempt at something like this, but I don’t think it’s ready for prime-time yet.
https://github.com/dundalek/closh
I guess what’s happening here is that shell is a domain-specific minilanguage for process control and I/O redirection, that grew programming features in an ad-hoc manner as people pushed it harder.
In theory one could create a general-purpose scripting language with a shell-like embedded minilanguage for shellouts; but I think the last time someone tried that the result was Perl.
Incidentally, your Python example doesn’t match the pipeline: in the shell version, the two processes run simultaneously; ls’s write()s to stdout can block if grep is slow and the pipebuf fills up; and if grep segfaults ls will get a SIGPIPE or something. Whereas your Python version sponges up into memory, so doesn’t have any of that behaviour. All arguably irrelevant subtleties for such a simple pipeline, but in the general case the distinction between a true pipeline and an emulation like this is quite important (consider “ps -F | grep quetzlcoatl”).
>I guess what’s happening here is that shell is a domain-specific minilanguage for process control and I/O redirection, that grew programming features in an ad-hoc manner as people pushed it harder.
While this is correct, it’s worth noting that in the case of the shell “pushed it harder” happened very early. The Mashey shell was already deliberately designed to have a pretty full suite of programming-language features by its first release in 1974 or 1975 (there is uncertainty about its ship date).