Bash IFS is ignoring the delimiter at the end of line

Question

I have a file with full of key value pairs. I wrote this shell script which reads each line and split key value.

while IFS='=' read -r key value
do
   something

done < < application.properties.

One of the property looks like this Connections/Database/Token=#!VWdg5neXrFiIbMxtAzOwmH+fM2FNtk6QPLhgOHw=

As run my script, its splitting it okay but its ignoring the character = at the end of the line.

It gives

key = Connections/Database/Token
value = #!VWdg5neXrFiIbMxtAzOwmH+fM2FNtk6QPLhgOHw

but it should be giving like:

 key = Connections/Database/Token
 value = #!VWdg5neXrFiIbMxtAzOwmH+fM2FNtk6QPLhgOHw=

Answer 1

TL;DR Add an explicit = to the end of each input line, then remove it from the resulting value before using it.

---

Why it works the way it does

See https://mywiki.wooledge.org/BashPitfalls#pf47. In short, the = in IFS is not treated as a field separator, but a field terminator, according to the POSIX definition of field-splitting.

When you write

IFS== read -r key value <<< "foo=var="

the input is first split into two fields, "foo" and "var" (not "foo", "var", and ""). There are exactly as many variables as fields, so you just get key=foo and value=var

If you have

IFS== read -r key value <<< "foo=var=="

now there are three fields: "foo", "var", and "". Because there are only two variables, then key=foo, and value is assigned:

1. the value "var", as normal
2. The delimiter "=" immediately after "var" in the input
3. The field "" from the input
4. The delimiter "=" following the "" in the input

See the POSIX specification for read for details about each variable to read is assigned a value after the input undergoes field-splitting.

So, there is never a trailing null field that results from field-splitting the input, only a trailing delimiter that gets added back to the final variable.

---

How to preserve the input

To work around this, add an explicit = to your input, and then remove it from the resulting value.

$ for input in "foo=bar" "foo=bar=" "foo=bar=="; do
> IFS== read -r name value <<< "$input="
> echo "${value%=}"
> done
bar
bar=
bar==

In your case, this means using something

while IFS='=' read -r key value
do
   value=${value%=}
   ...    
done < < (sed 's/$/=/' application.properties)

Or, as suggested first by Ivan, use parameter expansion operators to split the input instead of let read do it.

while read -r input; do
    key=${input%%=*}
    value=${input#*=}
    ...
done < application.properties

Either way, though, keep in mind that only = is considered as the delimiter here; you many need to trim trailing whitespace from the key and leading whitespace from the value if your properties look like name = value rather than name=value.

Answer 2

IFS method is definitely not good here. Try this instead.

while read -r item
do
    key="${item%%=*}"
    val="${item#*=}"
    echo "key = $key"
    echo "value = $val"
done < file

Well maybe IFS can work too like so

cat -E file | while IFS== read -r key value
do
    echo "key = $key"
    echo "value = ${value%$}"
done

I'm on Ubuntu 18.04.3 LTS, GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu) Used this test file

$ cat file
Connections/Database/Token=#!VWdg5neXrFiIbMxtAzOwmH+fM2FNtk6QPLhgOHw1=
Conn/Database/Token=#!VWdg5neXrFiIbMxtAzOwmH+fM2FNtk6QPLhgOHw2
Connections/Data/Token=#!VWdg5neXrFiIbMxtAzOwmH+fM2FNtk6QPLhgOHw3=
Connection/Data/Token=#!VWdg5neXrFiIbMxtAzOwmH+fM2FNtk6QPLhgOHw3=test

And i've used this commands to see what is going on

cat -E file | while IFS== read -r key value last; do echo "key = $key"; echo "value = $value"; echo "last = ${last-empty}"; done
cat -E file | while IFS== read -r key value;      do echo "key = $key"; echo "value = $value"; echo "last = ${last-empty}"; done
cat    file | while IFS== read -r key value last; do echo "key = $key"; echo "value = $value"; echo "last = ${last-empty}"; done
cat    file | while IFS== read -r key value;      do echo "key = $key"; echo "value = $value"; echo "last = ${last-empty}"; done