Quoting, in reverse

While quoting inputs to Bash requires some care, little is talked about quoting output in a way that is safe for Bash to evaluate.

This opens the gate to metaprogramming: if programs like Bash can generate correctly quoted Bash code, we can safely evaluate the output of those programs.

This functionality silently snuck into popular tools, making Bash a valid output format, which opens up exciting possibilities.

The testing function q

Let's define a function q (for quote) that will evaluate its first argument and compare it to the second. We can use this to test the behavior of various quoted strings. Here is q:

q() {
  [[ $(eval $1) == "$2" ]] || {
    printf "eval(%s) != %s\n" "${1@Q}" "$2" >&2
    return 1
  }
}

Here's q in action:

exit:0 $ q 'printf 1' 1
exit:0 $ q 'printf 2' 1
eval('printf 2') != 1
exit:1 $

When evaluating the first argument doesn't match the second, we get an error message and q returns 1.

Bash

Bash comes with various ways of quoting code, each of them yielding slightly different results.

declare -p

This one we've covered in depth already here, and it's listed here just for completeness.

Using declare -p we actually get back another declare command that we can feed back into bash to redeclare a variable.

Let's test that it works:

exit:0 $ x='hello, world'
exit:0 $ declare -p x
declare -- x="hello, world"
exit:0 $ q "$(declare -p x); echo \$x" 'hello, world'

Since declare -p outputs a command, we need to explicitly print the value of x. Note that the substitution happens before q invoked, so the arguments that q sees are actually this:

q 'declare -- x="hello, world"; echo $x' 'hello, world'

printf's %q

The builtin printf function has additional format specifiers compared to the C stdlib's printf. This one is interesting specifically:

%q quote the argument in a way that can be reused as shell input

Let's see it in action:

exit:0 $ printf '%q\n' '$x'
\$x
exit:0 $ printf '%q\n' '$(rm -rf /)'
\$\(rm\ -rf\ /\)
exit:0 $ q "echo $(printf '%q' '$(date)')" '$(date)'
exit:0 $

parameter expansion: the Q attribute

Bash supports various modifiers for strings when expanding variables. These modifiers are indicated with an @, e.g, ${var@U} will upcase the value of var.

One such modifier is Q for quoting strings in a way that is safe for evaluating them as shell input.

Here is how that looks like:

exit:0 $ cmd='$(date)'
exit:0 $ echo $cmd
$(date)
exit:0 $ echo ${cmd@Q}
'$(date)'
exit:0 $ q "echo $cmd" '$(date)'
eval('echo $(date)') != $(date)
exit:1 $ q "echo ${cmd@Q}" '$(date)'
exit:0 $

We can see that passing an unquoted version of $cmd will actually expand to echo followed by the current date, which is obviously not equal to the string $(date).

Using ${cmd@Q} we do get back the string $(date) however.

jq

JSON has become the lingua franca of data exchange in the last decade and jq is to JSON like awk is to unstructured text.

It supports format specifiers to convert JSON literals for use as part of other common formats. One of these format specifiers is @sh, quoting for output in a shell.

We can use this to convert a JSON object into a series of variable assignments and then evaluate them in the shell.

Let's use this to define a function for invoking each script defined in a JavaScript project's package.json. We'll be using this one as an example:

{
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
 "#": "rest of the file omitted"
}

Using jq we can convert these entries into a series of function definitions:

$ jq -r '.scripts | to_entries[] | .key + "() { eval " + (.value | @sh) + "; }"' <package.json
dev() { eval 'next dev'; }
build() { eval 'next build'; }
start() { eval 'next start'; }
lint() { eval 'next lint'; }

This is now valid Bash code, which we can evaluate:

$ eval "$(jq -r '.scripts | to_entries[] | .key + "() { eval " + (.value | @sh) + "; }"' <package.json)"
$ declare -pf lint
lint () {
  eval 'next lint'
}

Let's define next to just print its command line, so that we can plug this into our q function:

exit:0 $ next() { printf "next %s\n" "$*"; }
exit:0 $ lint
next lint
exit:0 $ q lint 'next lint'
exit:0 $