3

Guys I'm new to awk and I'm struggling with awk command to find the standard deviation.

I have got the mean using the following:

echo ${GfieldList[@]} | awk 'NF {sum=0;for (i=1;i<=NF;i++)sum+=$i; print "Mean= " sum / NF; }'

Standard Deviation formula is:

sqrt((1/N)*(sum of (value - mean)^2))

I have found the mean using the above formula

Can you guys help me with the awk command for this one?

Farvardin
  • 5,336
  • 5
  • 33
  • 54
Karthik
  • 69
  • 2
  • 8

2 Answers2

4

An alternate formula for the standard deviation is the square root of the quantity: (the mean square minus the square of the mean). This is used below:

$ echo 20 21 22 | awk 'NF {sum=0;ssq=0;for (i=1;i<=NF;i++){sum+=$i;ssq+=$i**2}; print "Std Dev=" (ssq/NF-(sum/NF)**2)**0.5}'
Std Dev=0.816497

Notes:

  • In awk, NF is the number of "fields" on a line. In our case, every field is a number, so NF is the number of numbers on a given line.

  • ssq is the sum of the squares of each number on the line. Thus, ssq/NF is the mean square.

  • sum is the sum of the numbers on the line. Thus sum/NF is the mean and (sum/NF)**2 is the square of the mean.

  • As per the formular, then, the standard deviation is (ssq/NF-(sum/NF)**2)**0.5.

The awk code

  • NF

    This serves as a condition: the statements which follow will only be executed if the number of fields on this line, NF, evaluates to true, meaning non-zero. In other words, this condition will cause empty lines to be skipped.

  • sum=0;ssq=0;

    This initializes sum and ssq to zero. This is only needed if there is more than one line of input.

  • for (i=1;i<=NF;i++){sum+=$i;ssq+=$i**2}

    This puts the sum of all the numbers in sum and the sum of the square of the numbers in ssq.

  • print "Std Dev=" (ssq/NF-(sum/NF)**2)**0.5

    This prints out the standard deviation.

John1024
  • 109,961
  • 14
  • 137
  • 171
2

Once you know the mean:

awk '{
    for (i = 1;i <= NF; i++) {
        sum += $i
    };
    print sum / NF
}' # for 2, 4, 4, 4, 5, 5, 7, 9 gives 5

then the standard deviation can be found thus:

awk -vM=5 '{
    for (i = 1; i <= NF; i++) {
        sum += ($i-M) * ($i-M)
    };
    print sqrt (sum / NF)
}' # for 2, 4, 4, 4, 5, 5, 7, 9 gives 2

In "compressed" form:

awk '{for(i=1;i<=NF;i++){sum+=$i};print sum/NF}'
awk -vM=5 '{for(i=1;i<=NF;i++){sum+=($i-M)*($i-M)};print sqrt(sum/NF)}'

(changing the value for M to the actual mean extracted from the first command).

paxdiablo
  • 854,327
  • 234
  • 1,573
  • 1,953