Argument passing in Bash scripts

My guess would be that most development teams have at least one or two Bash shell scripts that they use to avoid repetitive, error-prone tasks, or otherwise automate part of their day-to-day tasks. A common requirement is to pass arguments from one script/function/command to another. While in other languages, this is an unambiguous, simple task, in Bash, there are subtleties that it is important to be aware of.

Reading the manual, we can see that there are two special parameters that expand to "positional parameters" (a.k.a arguments): $* and $@. Judging by the length of the descriptions of theses parameters (684 and 878 characters, respectively) we can assume there are some nuances, and as we will see, there certainly are.

Recently here at Bamboo we wanted to write a Bash function that took optional arguments and interpolated them into an invocation of another command (in our case, curl). Think of passing flags such as --dry-run --description 'this is a dry run' to a command so the user can confirm the output is as they expect before making a configuration change, without having to duplicate the command (imagine if it already took 10 flags/arguments).

An argument-printing Ruby script

To help illustrate the problem, I'll be using a one-line Ruby script that prints the count and contents of the arguments passed to it by using ARGV:

$ ruby -e 'p ARGV.size; p ARGV'
0
[]
$ ruby -e 'p ARGV.size; p ARGV' foo
1
["foo"]
$ ruby -e 'p ARGV.size; p ARGV' foo bar
2
["foo", "bar"]
$ ruby -e 'p ARGV.size; p ARGV' foo 'bar baz'
2
["foo", "bar baz"]

Notice that the 'bar baz' argument is treated as a single argument since we quoted it to avoid word splitting (a complicated related topic in its own right, see the Bash manual or this informative wiki for more).

Four Bash functions: demonstrating $*, $@, "$*" and "$@"

Now, we set up functions that pass their arguments to our Ruby "argument printer":

dollar_star() {
    ruby -e 'p ARGV.size; p ARGV' -- $*
}
dollar_at() {
    ruby -e 'p ARGV.size; p ARGV' -- $@
}
dollar_star_quoted() {
    ruby -e 'p ARGV.size; p ARGV' -- "$*"
}
dollar_at_quoted() {
    ruby -e 'p ARGV.size; p ARGV' -- "$@"
}

N.B. that we're using -- to prevent the Ruby interpreter interpreting flag-like arguments.

Calling the Bash functions

Calling these functions shows the differences in output. When unquoted, $* and $@ both behave the same (badly, for our purposes!):

$ dollar_star --dry-run --flag 'with space'
4
["--dry-run", "--flag", "with", "space"]

$ dollar_at --dry-run --flag 'with space'
4
["--dry-run", "--flag", "with", "space"]

Notice how in both cases, we pass four arguments - our originally quoted 'with space' argument has been split into two separate arguments. For "$*", we get a different result:

$ quoted_dollar_star --dry-run --flag 'with space'
1
["--dry-run --flag with space"]

Now we are passing a single argument, which contains all the original arguments seemingly pasted together with spaces, which again, isn't what we want.

Finally, we see that "$@" gives us the behaviour we desire - ensuring that any whitespace within the original arguments is preserved:

$ quoted_dollar_at --dry-run --flag 'with space'
3
["--dry-run", "--flag", "with space"]

It's worth noting that my earlier "pasted together with spaces" description of "$*" isn't quite accurate, which we can demonstrate by setting a different $IFS variable value:

$ IFS='!' dollar_star_quoted --verbose --flag 'with space'
1
["--verbose!--flag!with space"]

With a modified $IFS, we are still passing a single argument, but it is formed of the three original arguments pasted together, separated by ! (the first character of $IFS) N.B. also that the space in the 'with space' quoted argument has been preserved as we are now using a different separator.

Summary

The different behaviours of two simple-looking variables can be a little bit tricky to understand, and it's certainly worth reading the manual carefully. For alternative further reading, there are plenty of Stack Exchange questions on this exact topic. The executive summary for our original purpose (pass flags to another command, preserving quoted arguments) is simple: just use "$@", but your requirements may differ.

Writing Bash scripts can be daunting vs say, Ruby. To help us avoid bugs and issues in our Bash scripts, we use the excellent Shellcheck tool to alert us to potential problems. Indeed, for the above example Bash functions Shellcheck warns on both of the unquoted variables - for $* SC2048 is triggered (which correctly suggests that we use "$@") and for $@ SC2068 (which warns us that our elements may be re-split, as we saw, when we ended up with 4 instead of 3 arguments).