IFS and command substitution

Question

I am writing a shell script to read input csv files and run a java program accordingly.

#!/usr/bin/ksh
CSV_FILE=${1}
myScript="/usr/bin/java -version"
while read row
do
     $myScript
     IFS=$"|"
     for column in $row
     do 
        $myScript
     done
done < $CSV_FILE

csv file:

a|b|c

Interestingly, $myScript outside the for loop works but the $myScript inside the for loop says "/usr/bin/java -version: not found [No such file or directory]". I have come to know that it is because I am setting IFS. If I comment IFS, and change the csv file to

a b c

It works ! I imagine the shell using the default IFS to separate the command /usr/bin/java and then apply the -version argument later. Since I changed the IFS, it is taking the entire string as a single command - or that is what I think is happening.

But this is my requirement: I have a csv file with a custom delimiter, and the command has arguments in it, separated by space. How can I do this correctly?

Why are you storing the command in a variable in the first place? See: [Bash FAQ #50](https://mywiki.wooledge.org/BashFAQ/050). — PesaThe, Sep 08 '18 at 08:48
@PesaThe Thanks a lot, I am new to shell scripting and that link was helpful. — Mayavi, Sep 08 '18 at 10:16

score 1 · Accepted Answer · answered Sep 08 '18 at 09:39

IFS indicates how to split the values of variables in unquoted substitutions. It applies to both $row and $myscript.

If you want to use IFS to do the splitting, which is convenient in plain sh, then you need to change the value of IFS or arrange to need the same value. In this particular case, you can easily arrange to need the same value, by defining myScript as myScript="/usr/bin/java|-version". Alternatively, you can change the value of IFS just in time. In both cases, note that an unquoted substitution doesn't just split the value using IFS, it also interprets each part as a wildcard pattern and replaces it by the list of matching file names if there are any. This means that if your CSV file contains a line like

foo|*|bar

then the row won't be foo, *, bar but foo, each file name in the current directory, bar. To process the data like this, you need to turn off with set -f. Also remember that read reads continuation lines when a line ends with a backslash, and strips leading and trailing IFS characters. Use IFS= read -r to turn off these two behaviors.

myScript="/usr/bin/java -version"
set -f
while IFS= read -r row
do
    $myScript
    IFS='|'
    for column in $row
    do 
        IFS=' '
        $myScript
    done
done

However there are better ways that avoid IFS-splitting altogether. Don't store a command in a space-separated string: it fails in complex cases, like commands that need an argument that contains a space. There are three robust ways to store a command:

Store the command in a function. This is the most natural approach. Running a command is code; you define code in a function. You can refer to the function's arguments collectively as "$@".
```
myScript () {
    /usr/bin/java -version "$@"
}
…
myScript extra_argument_1 extra_argument_2
```

Store an executable command name and its arguments in an array.

myScript=(/usr/bin/java -version)
…
"${myScript[@]}" extra_argument_1 extra_argument_2

Store a shell command, i.e. something that is meant to be parsed by the shell. To evaluate the shell code in a string, use eval. Be sure to quote the argument, like any other variable expansion, to avoid premature wildcard expansion. This approach is more complex since it requires careful quoting. It's only really useful when you have to store the command in a string, for example because it comes in as a parameter to your script. Note that you can't sensibly pass extra arguments this way.
```
myScript='/usr/bin/java -version'
…
eval "$myScript"
```

Also, since you're using ksh and not plain sh, you don't need to use IFS to split the input line. Use read -A instead to directly split into an array.

#!/usr/bin/ksh
CSV_FILE=${1}
myScript=(/usr/bin/java -version)
while IFS='|' read -r -A columns
do
    "${myScript[@]}"
    for column in "${columns[@]}"
    do 
        "${myScript[@]}"
    done
done <"$CSV_FILE"

score 1 · Answer 2 · answered Sep 08 '18 at 09:48

The simplest soultion is to avoid changing IFS and do the splitting with read -d <delimiter> like this:

#!/usr/bin/ksh
CSV_FILE=${1}
myScript="/usr/bin/java -version"
while read -A -d '|' columns
do
     $myScript
     for column in "${columns[@]}"
     do 
        echo next is "$column"
        $myScript
     done
done < $CSV_FILE

score 0 · Answer 3 · answered Sep 08 '18 at 09:08

0

IFS tells the shell which characters separate "words", that is, the different components of a command. So when you remove the space character from IFS and run foo bar, the script sees a single argument "foo bar" rather than "foo" and "bar".

answered Sep 08 '18 at 09:08

l0b0

55,365
30
138
223

score -1 · Answer 4 · answered Sep 08 '18 at 09:35

-1

the IFS should be placed behind of "while"

#!/usr/bin/ksh
CSV_FILE=${1}
myScript="/usr/bin/java -version"
while IFS="|" read row
do
 $myScript
 for column in $row
 do 
    $myScript
 done
done < $CSV_FILE

answered Sep 08 '18 at 09:35

Ivan

459
1
3
9

While this is somewhat on the right track, it doesn't work. Since `read row` is only reading a single variable, setting `IFS` just for it has no effect on splitting. The row is still split at the beginning of the for loop. See my answer for some ways this can work. – Gilles 'SO- stop being evil' Sep 08 '18 at 09:40

IFS and command substitution

4 Answers4