3

Given a string variable (not a string) in bash, how can one expand its curly braces (and glob any file wildcards like asterisks), and then store the resulting expansion in an array?

I know this is possible with eval:

#!/usr/bin/env bash

string="{a,b}{1,2}"
array=( $( eval echo $string ) )
echo "${array[@]}"
#a1 a2 b1 b2

But eval could be dangerous if we don't trust the contents of string (say, if it is an argument to a script). A string with embedded semicolons, quotes, spaces, arbitrary commands, etc. could open the script to attack.

Is there a safer way to do this expansion in bash? I suspect that all of the following are equally unsafe ways of expanding the string:

#!/usr/bin/env bash

eval echo $string

bash <<< "echo $string"

echo echo $string | bash

bash <(echo echo $string)

Is there perhaps a command other than bash that we can pipe the string to, that would do the expansion?

Aside: Note that in csh/tcsh this is easy, since brace expansion happens after variable substitution (unlike in bash):

#!/usr/bin/env csh

set string = "{a,b}{1,2}"
set array = ( $string )
echo "$array"
#a1 a2 b1 b2
AWitt
  • 41
  • 3
  • Array assignments *do* expand the brace expansions, no `eval` required! Try `arr=({1,2}); declare -p arr` – Benjamin W. Nov 12 '20 at 19:50
  • Oh wait, you're starting with a string variable. That won't work, because brace expansion happens before parameter expansion. – Benjamin W. Nov 12 '20 at 19:51
  • Where do you get `string` from? Maybe there's a way to have brace expansion take place in a separate step. – Benjamin W. Nov 12 '20 at 19:53
  • Indeed, I'm starting with a string variable, not a string, and want to expand it into an array. In my application the string variable is typically long and complex with braces, wildcards, etc., and expands into a very long array of files. The string is also used in its non-expanded form elsewhere in the script, e.g. passed to other shell commands that will later do the expansion after prepending paths etc., so refactoring my code to avoid this would be hard. – AWitt Nov 12 '20 at 20:17
  • 1
    Sounds like passing the already expanded list instead of something like `'a{b,c}*'` would be preferable. Of course, this makes other things more difficult, but it's the non-hacky approach. Other than that, I'd really to love to see an answer for the question instead of comments like mine (trying to talk you into not doing what you wanted :) – Socowi Nov 12 '20 at 20:23
  • Thanks for the support. Yes, on StackOverflow there are a few closely-related questions, but the answers either used `eval` or refactored the user's specific problem to avoid the brace-laden string in the first place. That's not easy in my application. – AWitt Nov 12 '20 at 20:39
  • The only options I can think of involve doing workarounds to try to make `eval` less unsafe. That, or reimplementing the shell's brace expansion logic in your own code, and that's going to be very unpleasant. – Charles Duffy Nov 13 '20 at 15:00
  • @AWitt, ...actually... would you accept an answer that used a Python interpreter to do the work? – Charles Duffy Nov 13 '20 at 15:06
  • @CharlesDuffy, thanks for all the effort you put into the python solution. It seems like a lot of overhead though; let's see if anyone can come up with a native way to do it in bash. If not, I may accept something like your answer. – AWitt Nov 13 '20 at 21:47
  • _nod_. I could write something in native bash that worked in the simple cases (using native regex-matching to look for the construct in question and expanding them by hand), but I don't have a lot of faith that its behavior would be a good match to the shell's "real" behavior in trickier situations... or that it would be any shorter than the Python approach. – Charles Duffy Nov 13 '20 at 22:02

1 Answers1

1

The braceexpand Python module, in combination with the standard-library glob module, can be used to accomplish what you're looking for. Wrapped in shell for use from bash, this might look like:

#!/usr/bin/env bash
case $BASH_VERSION in
  ''|[123].*|4.[012].*) echo "ERROR: bash 4.3+ required" >&2; exit 1;;
esac

# store Python code in a string
expand_py=$(cat <<'EOF'
try:
  import sys, glob, braceexpand
except ImportError as e:
  sys.stderr.write("Did you remember to install the braceexpand Python module?\n")
  raise e

for arg in sys.argv[1:]:
  for globexp in braceexpand.braceexpand(arg):
    glob_results = glob.glob(globexp)
    if len(glob_results) == 0:
      sys.stdout.write(globexp)
      sys.stdout.write('\0')
      continue
    else:
      sys.stdout.write(''.join(['%s\0' % result for result in glob_results]))
EOF
)

# shell wrapper for the above Python program
expand() {
  local item
  local -n dest_array=$1; shift

  dest_array=( )
  while IFS= read -r -d '' item; do
    dest_array+=( "$item" )
  done < <(python3 -c "$expand_py" "$@")
  unset -n dest_array
}

# actually demonstrate usage
expand yourArray '{a,b}{1,2}'
declare -p yourArray

...outputs when run the intended array value:

declare -a yourArray=([0]="a1" [1]="a2" [2]="b1" [3]="b2")
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441