Preserving leading white space while reading>>writing a file line by line in bash

Question

I am trying to loop through a directory of text files and combine them into one document. This works great, but the text files contain code snippets, and all of my formatting is getting collapsed to the left. All leading whitespace on a line is stripped.

#!/bin/sh
OUTPUT="../best_practices.textile"
FILES="../best-practices/*.textile"
for f in "$FILES"
do
  echo "Processing $f file..."
  echo "">$OUTPUT

  cat $f | while read line; do 
      echo "$line">>$OUTPUT
  done
  echo >>$OUTPUT
  echo >>$OUTPUT
done

I am admittedly a bash noob, but after searching high and low I couldn't find a proper solution. Apparently BASH hates the leading white space in general.

Answer 1

As others have pointed out, using cat or awk instead of a read-echo loop is a much better way to do this -- avoids the whitespace-trimming problem (and a couple of others you haven't stumbled upon), runs faster, and at least with cat, is simply cleaner code. Nonetheless, I'd like to take a stab at getting the read-echo loop to work right.

First, the whitespace-trimming problem: the read command automatically trims leading and trailing whitespace; this can be fixed by changing its definition of whitespace by setting the IFS variable to blank. Also, read assumes that a backslash at the end of the line means the next line is a continuation, and should be spliced together with this one; to fix this, use its -r (raw) flag. The third problem here is that many implementations of echo interpret escape sequences in the string (e.g. they may turn \n into an actual newline); to fix this, use printf instead. Finally, just as a general scripting hygiene rule, you shouldn't use cat when you don't actually need to; use input redirection instead. With those changes, the inner loop looks like this:

while IFS='' read -r line; do 
  printf "%s\n" "$line">>$OUTPUT
done <$f

...there are also a couple of other problems with the surrounding script: the line that tries to define FILES as the list of available .textile files has quotes around it, meaning it never gets expanded into an actual list of files. The best way to do this is to use an array:

FILES=(../best-practices/*.textile)
...
for f in "${FILES[@]}"

(and all occurrences of $f should be in double-quotes in case any of the filenames have spaces or other funny characters in them -- should really do this with $OUTPUT as well, though since that's defined in the script it's actually safe to leave off.)

Finally, there's a echo "">$OUTPUT near the top of the loop-over-files that's going to erase the output file every time through (i.e. at the end, it only contains the last .textile file); this needs to be moved to before the loop. I'm not sure if the intent here was to put a single blank line at the beginning of the file, or three blank lines between files (and one at the beginning and two at the end), so I'm not sure exactly what the appropriate replacement is. Anyway, here's what I can up with after fixing all of these problems:

#!/bin/sh
OUTPUT="../best_practices.textile"
FILES=(../best-practices/*.textile)

: >"$OUTPUT"
for f in "${FILES[@]}"
do
  echo "Processing $f file..."
  echo >>"$OUTPUT"

  while IFS='' read -r line; do 
    printf "%s\n" "$line">>"$OUTPUT"
  done <"$f"

  echo >>"$OUTPUT"
  echo >>"$OUTPUT"
done

Answer 2

Instead of:

cat $f | while read line; do 
    echo "$line">>$OUTPUT
done

Do this:

cat $f >>$OUTPUT

(If there's a reason you need to do things line by line it'd be good to include that in the question.)

Answer 3

that's an overly expensive way of combining files.

cat ../best-practices/*.textile >  ../best_practices.textile

if you want to add a blank( newline) to each file as you concatenate, use awk

awk 'FNR==1{print "">"out.txt"}{print > "out.txt" }' *.textile

OR

awk 'FNR==1{print ""}{print}' file* > out.txt

Answer 4

This allows you to intersperse newlines between each input file as you have done in your original script:

for f in $FILES; do echo -ne '\n\n' | cat "$f" -; done > $OUTPUT

Note that $FILES is unquoted for this to work (otherwise the extra newlines appear only once at the end of all the output), but $f must be quoted to protect spaces in filenames, if they exist.

Answer 5

The correct answer, imo, is this, reproduced below:

while IFS= read line; do
    check=${line:0:1}
done < file.txt

Note that it'll take care of situations where the input is piped from another command, and not just from an actual file.

Note that you can also simplify the redirection as shown below.

#!/bin/bash
OUTPUT="../best_practices.textile"
FILES="../best-practices/*.textile"
for f in "$FILES"
do
  echo "Processing $f file..."
  {
  echo

  while IFS= read line; do 
      echo "$line"
  done < $f
  echo
  echo;
  } > $OUTPUT
done