69

I am trying to read a properties file from a shell script which contains a period (.) character like below:

# app.properties
db.uat.user=saple user
db.uat.passwd=secret


#/bin/sh
function pause(){
   read -p "$*"
}

file="./app.properties"

if [ -f "$file" ]
then
    echo "$file found."
 . $file

echo "User Id " $db.uat.user
echo "user password =" $db.uat.passwd
else
    echo "$file not found."
fi

I have tried to parse the file after sourcing the file but it is not working since the keys contains the "." character and there are spaces in that value also.

My properties file always resides in the same directory of the script or somewhere in /usr/share/doc

bakoyaro
  • 2,550
  • 3
  • 36
  • 63
Kiran
  • 921
  • 1
  • 11
  • 23

7 Answers7

114

I use simple grep inside function in bash script to receive properties from .properties file.

This properties file I use in two places - to setup dev environment and as application parameters.

I believe that grep may work slow in big loops but it solves my needs when I want to prepare dev environment.

Hope, someone will find this useful.

Example:

File: setup.sh

#!/bin/bash

ENV=${1:-dev}

function prop {
    grep "${1}" env/${ENV}.properties|cut -d'=' -f2
}

docker create \
    --name=myapp-storage \
    -p $(prop 'app.storage.address'):$(prop 'app.storage.port'):9000 \
    -h $(prop 'app.storage.host') \
    -e STORAGE_ACCESS_KEY="$(prop 'app.storage.access-key')" \
    -e STORAGE_SECRET_KEY="$(prop 'app.storage.secret-key')" \
    -e STORAGE_BUCKET="$(prop 'app.storage.bucket')" \
    -v "$(prop 'app.data-path')/storage":/app/storage \
    myapp-storage:latest

docker create \
    --name=myapp-database \
    -p "$(prop 'app.database.address')":"$(prop 'app.database.port')":5432 \
    -h "$(prop 'app.database.host')" \
    -e POSTGRES_USER="$(prop 'app.database.user')" \
    -e POSTGRES_PASSWORD="$(prop 'app.database.pass')" \
    -e POSTGRES_DB="$(prop 'app.database.main')" \
    -e PGDATA="/app/database" \
    -v "$(prop 'app.data-path')/database":/app/database \
    postgres:9.5

File: env/dev.properties

app.data-path=/apps/myapp/

#==========================================================
# Server properties
#==========================================================
app.server.address=127.0.0.70
app.server.host=dev.myapp.com
app.server.port=8080

#==========================================================
# Backend properties
#==========================================================
app.backend.address=127.0.0.70
app.backend.host=dev.myapp.com
app.backend.port=8081
app.backend.maximum.threads=5

#==========================================================
# Database properties
#==========================================================
app.database.address=127.0.0.70
app.database.host=database.myapp.com
app.database.port=5432
app.database.user=dev-user-name
app.database.pass=dev-password
app.database.main=dev-database

#==========================================================
# Storage properties
#==========================================================
app.storage.address=127.0.0.70
app.storage.host=storage.myapp.com
app.storage.port=4569
app.storage.endpoint=http://storage.myapp.com:4569
app.storage.access-key=dev-access-key
app.storage.secret-key=dev-secret-key
app.storage.region=us-east-1
app.storage.bucket=dev-bucket

Usage:

./setup.sh dev
Nicolai
  • 5,489
  • 1
  • 24
  • 31
  • 12
    I like this way. An improvement for your grep: Now it matches the string anywhere in the line. You could use `grep "^\\\s*${1}=" env/${ENV}.properties|cut -d'=' -f2` to match whole keys at the beginning of the line (excluding optional whitespace). – jhyot Aug 04 '16 at 13:57
  • @jhyot, Thank you, You are right. Additional this method has limits, for example it doesn't allow to refer to other properties @app.v1=test-${app.v2}@. It solves my need for now, but I believe I'll have to enhance it very soon. – Nicolai Aug 04 '16 at 14:11
  • Very nice approach! I think I'll adopt it in my own work. – bakoyaro Aug 25 '16 at 15:30
  • 2
    @jhyot Nice regexp. It should start with "^\\s*" rather than "^\\\s*" isn't it ? – Bludwarf Jun 28 '17 at 15:17
  • @Bludwarf yes I think you're right. Also at the end it should be `-f2-` so that equal signs (=) in the value will not get cut off. In my code I used something like this in the end: `grep '^\s*'"$1"'=' "env/${ENV}.properties" | cut -d'=' -f2-` – jhyot Jul 03 '17 at 14:50
  • 4
    Grep's `-w` argument is a simpler way to improve the match: `grep -w "${1}" env/${ENV}.properties|cut -d'=' -f2` – Mike Slinn Jul 31 '17 at 22:46
  • 1
    If you want to use only the last occurrence of a key in the file, use `grep "${1}"|cut -d'=' -f2| tail -1` – muelleth May 03 '18 at 14:44
  • i liked this - in my case i don't want to take into account any spaces and also don't want to match `#commented_out_prop`'s `cat ${properties_file} | tr -d "[:blank:]" | grep '^'${1} | cut -d'=' -f2` – Nicholas DiPiazza Jun 19 '18 at 21:43
  • 3
    if you use the property key as value inside the same file, you will also get the lines with the values as results. I modified it like so, to get only the keys: ``grep "^\\s*${1}=" $propsfile|cut -d'=' -f2` note the equal sign at the end of the key matching – Manticore May 02 '19 at 10:17
  • With this approach is, along with the property value, it is bringing key as well. Anybody solved this? – Suryaprakash Pisay Jul 22 '19 at 09:30
  • 2
    My property values have some '=' character. So, I replaced the `|cut -d'=' -f2` by `| sed -E 's/^[^=]*=(.*)$/\1/' ` – xsenechal Jan 04 '20 at 06:29
  • 1
    In order to skip comments you could: `grep -v "^#" env/${ENV}.properties|grep "${1}" |cut -d'=' -f2` – Marco Montel Feb 18 '20 at 11:00
  • This answer is gold. I was scratching my head over this for hours with no success. This answer is simple and it works well. – Akash Yadav Dec 31 '21 at 11:18
64

As (Bourne) shell variables cannot contain dots you can replace them by underscores. Read every line, translate . in the key to _ and evaluate.

#/bin/sh

file="./app.properties"

if [ -f "$file" ]
then
  echo "$file found."

  while IFS='=' read -r key value
  do
    key=$(echo $key | tr '.' '_')
    eval ${key}=\${value}
  done < "$file"

  echo "User Id       = " ${db_uat_user}
  echo "user password = " ${db_uat_passwd}
else
  echo "$file not found."
fi

Note that the above only translates . to _, if you have a more complex format you may want to use additional translations. I recently had to parse a full Ant properties file with lots of nasty characters, and there I had to use:

key=$(echo $key | tr .-/ _ | tr -cd 'A-Za-z0-9_')
fork2execve
  • 1,561
  • 11
  • 16
  • Is there way i can save the db_uat_user in a variable and then pass the variable to retrieve the value of it. Something like first var_key=db_uat_user and then echo "user id =" ${"$var_key"} anyhow intern the value var_key is db_uat_user is it possible to achieve something like this. – Adarsh H D Dev Dec 06 '16 at 04:28
  • @AdarshHDDev asking different questions is discouraged on SO. As to your question, its been asked and answered many times, for instance: http://unix.stackexchange.com/questions/68035/foo-and-zsh – fork2execve Jan 01 '17 at 14:06
  • the drawback of this answer is if value contains "=" character. – Ashwani Nov 02 '20 at 18:37
  • @Ashwani see my note, this explains how you can filter extra nasty characters like = – fork2execve Feb 09 '21 at 23:32
  • How to skip empty strings? – Rafis Ganeev Mar 04 '21 at 11:04
  • With `IFS='= '` (i.e. adding a space behind the equal sign) you can also remove leading spaces from the value. – Udo Mar 16 '23 at 11:46
20

Since variable names in the BASH shell cannot contain a dot or space it is better to use an associative array in BASH like this:

#!/bin/bash

# declare an associative array
declare -A arr

# read file line by line and populate the array. Field separator is "="
while IFS='=' read -r k v; do
   arr["$k"]="$v"
done < app.properties

Testing:

Use declare -p to show the result:

  > declare -p arr  

        declare -A arr='([db.uat.passwd]="secret" [db.uat.user]="saple user" )'
Steven the Easily Amused
  • 730
  • 9
  • 23
anubhava
  • 761,203
  • 64
  • 569
  • 643
  • 2
    I like this. You can use a check like this to ignore empty lines: `if [ ! -z $k ] ; then arr["$k"]="$v"; fi` – Datz Jul 24 '20 at 14:27
9

For a very high performance, and BASH 3.0 compatible solution:

file: loadProps.sh

function loadProperties() {
  local fileName=$1
  local prefixKey=$2

  if [ ! -f "${fileName}" ]; then
    echo "${fileName} not found!"
    return 1
  fi

  while IFS='=' read -r origKey value; do
    local key=${origKey}
    key=${key//[!a-zA-Z0-9_]/_} 
    if [[ "${origKey}" == "#"*   ]]; then
      local ignoreComments
    elif [ -z "${key}" ]; then
      local emptyLine
    else
      if [[ "${prefixKey}${key}" =~ ^[0-9].* ]]; then
        key=_${key}
      fi
      eval ${prefixKey}${key}=\${value}
    fi
  done < <(grep "" ${fileName})
}

The other solutions provided here are great and elegant, but

  • @fork2execve: slow when dealing with large properties files
  • @Nicolai: slow when reading lots of properties
  • @anubhava: require BASH 4.0 (for the array)

I needed something working on bash 3, dealing with properties files of ~1k entries, reading ~200 properties, and whole script called MANY times.

this function also deals with

  • empty lines
  • commented code
  • duplicated entries (last one wins)
  • normalize property names
  • last line without a proper new line

Testing

file: my.properties

a=value
a=override value
b=what about `!@#$%^&*()_+[]\?
c=${a} no expansion
d=another = (equal sign)
e=     5 spaces front and back     
f=
#g=commented out
#ignore new line below

.@a%^=who named this???
a1=A-ONE
1a=ONE-A
X=lastLine with no new line!

test script

. loadProps.sh

loadProperties my.properties PROP_
echo "a='${PROP_a}'"
echo "b='${PROP_b}'"
echo "c='${PROP_c}'"
echo "d='${PROP_d}'"
echo "e='${PROP_e}'"
echo "f='${PROP_f}'"
echo "g='${PROP_g}'"
echo ".@a%^='${PROP___a__}'"
echo "a1='${PROP_a1}'"
echo "1a='${PROP_1a}'"
echo "X='${PROP_X}'"

loadProperties my.properties
echo "a='${a}'"
echo "1a='${_1a}'"

output

a='override value'
b='what about `!@#$%^&*()_+[]\?'
c='${a} no expansion'
d='another = (equal sign)'
e='     5 spaces front and back     '
f=''
g=''
.@a%^='who named this???'
a1='A-ONE'
1a='ONE-A'
X='lastLine with no new line!'
a='override value'
1a='ONE-A'

Performance Test

. loadProps.sh

function fork2execve() {
  while IFS='=' read -r key value; do
    key=$(echo $key | tr .-/ _ | tr -cd 'A-Za-z0-9_')
    eval ${key}=\${value}
  done < "$1"
}

function prop {
  grep '^\s*'"$2"'=' "$1" | cut -d'=' -f2-
}

function Nicolai() {
  for i in $(seq 1 $2); do 
    prop0000=$(prop $1 "property_0000")
  done
}

function perfCase() {
  echo "perfCase $1, $2, $3"
  time for i in $(seq 1 1); do 
    eval $1 $2 $3
  done
}

function perf() {
  perfCase $1 0001.properties $2
  perfCase $1 0010.properties $2
  perfCase $1 0100.properties $2
  perfCase $1 1000.properties $2
}

perf "loadProperties"
perf "fork2execve"
perf "Nicolai" 1
perf "Nicolai" 10
perf "Nicolai" 100

with 4 NNNN.properties files with entries such as

property_0000=value_0000
property_0001=value_0001
...
property_NNNN=value_NNNN

resulted with

function   , file,   #,     real,    user,     sys
loadPropert, 0001,    ,    0.058,   0.002,   0.005
loadPropert, 0010,    ,    0.032,   0.003,   0.005
loadPropert, 0100,    ,    0.041,   0.013,   0.006
loadPropert, 1000,    ,    0.140,   0.106,   0.013

fork2execve, 0001,    ,    0.053,   0.003,   0.007
fork2execve, 0010,    ,    0.211,   0.021,   0.051
fork2execve, 0100,    ,    2.146,   0.214,   0.531
fork2execve, 1000,    ,   21.375,   2.151,   5.312

Nicolai    , 0001,   1,    0.048,   0.003,   0.009
Nicolai    , 0010,   1,    0.047,   0.003,   0.009
Nicolai    , 0100,   1,    0.044,   0.003,   0.010
Nicolai    , 1000,   1,    0.044,   0.004,   0.009

Nicolai    , 0001,  10,    0.240,   0.020,   0.056
Nicolai    , 0010,  10,    0.263,   0.021,   0.059
Nicolai    , 0100,  10,    0.272,   0.023,   0.062
Nicolai    , 1000,  10,    0.295,   0.027,   0.059

Nicolai    , 0001, 100,    2.218,   0.189,   0.528
Nicolai    , 0010, 100,    2.213,   0.193,   0.537
Nicolai    , 0100, 100,    2.247,   0.196,   0.543
Nicolai    , 1000, 100,    2.323,   0.253,   0.534
Salva
  • 91
  • 1
  • 5
  • Very good function, I just wish it handled properties ending with `\\` for values split over multiple lines. – raffian Jun 14 '23 at 18:59
1

@fork2x

I have tried like this .Please review and update me whether it is right approach or not. 

#/bin/sh
function pause(){
   read -p "$*"
}

file="./apptest.properties"


if [ -f "$file" ]
then
    echo "$file found."

dbUser=`sed '/^\#/d' $file | grep 'db.uat.user'  | tail -n 1 | cut -d "=" -f2- | sed 's/^[[:space:]]*//;s/[[:space:]]*$//'`
dbPass=`sed '/^\#/d' $file | grep 'db.uat.passwd'  | tail -n 1 | cut -d "=" -f2- | sed 's/^[[:space:]]*//;s/[[:space:]]*$//'`

echo database user = $dbUser
echo database pass = $dbPass

else
    echo "$file not found."
fi
Kiran
  • 921
  • 1
  • 11
  • 23
1

I found using while IFS='=' read -r to be a bit slow (I don't know why, maybe someone could briefly explain in a comment or point to a SO answer?). I also found @Nicolai answer very neat as a one-liner, but very inefficient as it will scan the entire properties file over and over again for every single call of prop.

I found a solution that answers the question, performs well and it is a one-liner (bit verbose line though).

The solution does sourcing but massages the contents before sourcing:

#!/usr/bin/env bash

source <(grep -v '^ *#' ./app.properties | grep '[^ ] *=' | awk '{split($0,a,"="); print gensub(/\./, "_", "g", a[1]) "=" a[2]}')

echo $db_uat_user

Explanation:

grep -v '^ *#': discard comment lines grep '[^ ] *=': discards lines without = split($0,a,"="): splits line at = and stores into array a, i.e. a[1] is the key, a[2] is the value gensub(/\./, "_", "g", a[1]): replaces . with _ print gensub... "=" a[2]} concatenates the result of gensub above with = and value.

Edit: As others pointed out, there are some incompatibilities issues (awk) and also it does not validate the contents to see if every line of the property file is actually a kv pair. But the goal here is to show the general idea for a solution that is both fast and clean. Sourcing seems to be the way to go as it loads the properties once that can be used multiple times.

L. Holanda
  • 4,432
  • 1
  • 36
  • 44
  • Your code gives me the error `/dev/fd/63: line 1: user: command not found` using `./app.properties` with contents as given in question. – TomRoche Jan 25 '18 at 05:46
  • I'm guessing this is due to your not handling the space in the file's first "value"=`sample user`. – TomRoche Jan 25 '18 at 06:15
  • 1
    This solution appears to assume `gawk` and is incompatible with the `awk` variants used in BSD/macOS, since `gensub` is not available there, see [The GNU Awk User's Guide: String Functions](https://www.gnu.org/software/gawk/manual/html_node/String-Functions.html) – Ernst de Haan Feb 07 '19 at 14:02
  • 1
    Instead `gsub` can be used, which seems standard/portable – Ernst de Haan Feb 07 '19 at 14:20
0

I think Nicolai's answer is good. However, sometimes people write

app.server.address = 127.0.0.70

instead of

app.server.address=127.0.0.70

In this situation. If we directly use

function prop {
    grep "${1}" env/${ENV}.properties|cut -d'=' -f2
}

it will produce " 127.0.0.70" instead of "127.0.0.70", take some error in string combination. To solve this, we could add "| xargs". And it will be

grep "${1}" ${ENV}.properties|cut -d'=' -f2 | xargs

And we will get what we want.

fwhdzh
  • 21
  • 3