0

I want to create a shell that reads a text file.

The file, users_test.txt, looks like:

Headers - Group : Team : User

Group_A:Team_Red Team_Green:worker_1 worker_2 worker_2 worker_3
Group_B:Team_Blue Team_Black Team_Grey:worker_4 worker_2 worker_5 worker_6

I want to loop through the .txt file above which is colon separated. Here is my current solution.

#!/bin/bash
# set filename
FILENAME="users_test.txt"

# loop through file, each  line is divided into 3 fields separated by a colon (:) 
while IFS=':' read -r groups teams users
do 
    echo "group $groups has teams $teams and user $users"
    
    for team in $teams; do
        echo "hey: "$team
    done
    
    for group in $groups; do
        echo "hi: "$group
    done

    # Iterate the string variable using for loop
    for user in $users; do
        echo "hello: "$user
    done

done < "$FILENAME"

Why does it end after the first loop iteration?

output:

admin@computer:$ ./setup-test1.sh
group  has teams Team_Red Team_Green and user worker_1 worker_2 worker_2 worker_3
hey: Team_Red
hey: Team_Green
hi: Group_A
hello: worker_1
hello: worker_2
hello: worker_2
hello: worker_3
admin@computer5:$

It seems to work fine but it ends after the first line, as a result Group B isn't printed.

Can anyone tell my why the script only reads the first line containing Group_A but not the second line with Group_B?

Zizi96
  • 459
  • 1
  • 6
  • 23
  • 2
    I'm unable to reproduce your output with the script and input you provided. Recheck your script/data, maybe you fixed something as you put together the question? The only typo I notice off-hand is `worker 5` instead of `worker_5`. :) – Erwin Jun 27 '22 at 19:21
  • 2
    Make sure the file uses `\n` as the newline, not `\r`. – Barmar Jun 27 '22 at 19:22
  • @Erwin What output do you get? I've just repasted the code producing that output to be sure. – Zizi96 Jun 27 '22 at 19:31
  • 1
    Bit tricky to share that in a comment. BTW, another typo is that `echo "group $group has teams $teams and user $users"` should say `$groups` instead of `$group` – Erwin Jun 27 '22 at 19:34
  • You can paste here and set it to expire immediately or an hour - https://pastebin.com/ – Zizi96 Jun 27 '22 at 19:36
  • 1
    https://pastebin.com/WjZcBFFh – Erwin Jun 27 '22 at 19:37
  • Much appreciated, that's the exact output I'm looking for so my code is sound. No idea why I'm not getting that, I guess its probably has something to do with the AWS environment I'm running in. – Zizi96 Jun 27 '22 at 19:41
  • 1
    As @Barmar commented, there may be newline or whitespace issues. Compare with this `xxd` output, see if you have some difference that you did not expect: https://pastebin.com/k94Hef39 – Erwin Jun 27 '22 at 19:53
  • @Erwin Had a look and the only differences I'm seeing are at the very end where your `xxd` output has `0a` at the end where mine doesn't. Not sure why this is as I've copied the txt sample from here back into my .txt file but still cant replicated your `xxd` output - https://pastebin.com/qBp5LQmS – Zizi96 Jun 27 '22 at 20:22
  • I had to struggle to _not_ get a newline at the end of my `users_test.txt` file but once I did, I was able to reproduce your behavior, so that seems to be the exact issue – Erwin Jun 27 '22 at 20:30
  • I can't reproduce your problem with the script and text file provided. – tink Jun 27 '22 at 20:49
  • @Erwin ok thanks, I'm still confused as to how omitting the feed line character `0a` at the end of my `users_test.txt` equates to producing newline issue. More importantly, how do I rectify it? Currently I do the following: openNotepad-`Group_A:Team_Red Team_Green:worker_1 worker_2 worker_2 worker_3`-enterKey-`Group_B:Team_Blue Team_Black Team_Grey:worker_4 worker_2 worker_5 worker_6`-save. What should I do to create the txt file which runs without error, excuse my ignorance but I'm missing something obvious here. – Zizi96 Jun 27 '22 at 21:37
  • **Are you running on Windows** using one of the Unix-on-Windows environments like WSL, cygwin/mingw/git4win, gnuwin32, etc? Or are you creating the file on Windows and copying to Unix? While _Unix_ (including its shells) has a rule about ending the last line of a textfile with LF, _Windows_ does not and Windows programs like e.g. notepad will very easily create a file with no CRLF on the last line, and converting the actually-present CRLFs with dos2unix of tr or sed or similar doesn't change this. – dave_thompson_085 Jun 28 '22 at 00:31
  • Running in a unix environment, I was creating the text files through jupyter notebook which I suspect was the cause of my issues. Appreciate the explanation. – Zizi96 Jun 28 '22 at 10:16

1 Answers1

1

Given your comment that your file doesn't end in a linefeed, 0a:

$ printf 'foo\n' | xxd
00000000: 666f 6f0a                                foo.

$ printf 'foo' | xxd
00000000: 666f 6f                                  foo

here's the difference when trying to read such a file:

$ while IFS= read -r line; do echo "line=$line"; done < <(printf 'foo\n')
line=foo

$ while IFS= read -r line; do echo "line=$line"; done < <(printf 'foo')
$

A file that doesn't have a linefeed at the end is not a valid text file and so trying to read it with any text processing tool (e.g. read - note "The standard input shall be a text file.") is undefined behavior. It's not worth wasting time on what any tool does with such a file and why, just don't create files like that if you want to do any further text processing on them. If your editor creates files without terminating linefeeds then get a new editor, e.g. vi.

Ed Morton
  • 188,023
  • 17
  • 78
  • 185