Shell Script - list files, read files and write data to new file

Question

I have a special question to shell scripting.
Simple scripting is no Problem for me but I am new on this and want to make me a simple database file.

So, what I want to do is:

- Search for filetypes (i.e. .nfo) <-- should be no problem :)
- read inside of each found file and use some strings inside
- these string of each file should be written in a new file. Each found file informations 

should be one row in new file

I hope I explained my "project" good.

My problem is now, to understand how I can tell the script it has to search for files and then use each of this files to read in it and use some information in it to write this in a new file.

I will explain a bit better.
I am searching for files and that gives me back:

file1.nfo
file2.nfo
file3.nfo

Ok now in each of that file I need the information between 2 lines. i.e.
file1.nfo:

<user>test1</user>

file2.nfo:

<user>test2</user>

so in the new file there should now be:

file1.nfo:user1
file2.nfo:user2

OK so:

find -name *.nfo  > /test/database.txt

is printing out the list of files. and

sed -n '/<user*/,/<\/user>/p' file1.nfo

gives me back the complete file and not only the information between <user> and </user>

I try to go on step by step and I am reading a lot but it seems to be very difficult.

What am I doing wrong and what should be the best way to list all files, and write the files and the content between two strings to a file?

EDIT-NEW:

Ok here is an update for more informations. I learned now a lot and searched the web for my problems. I can find a lot of informations but i don´t know how to put them together so that i can use it.

Working now with awk is that i get back filename and the String.

Here now the complete Informations (i thought i can go on by myself with a bit of help but i can´t :( )

Here is an example of: /test/file1.nfo

<string1>STRING 1</string1>
<string2>STRING 2</string2>
<string3>STRING 3</string3>
<string4>STRING 4</string4>
<personal informations>
<hobby>Baseball</hobby>
<hobby>Baskeball</hobby>
</personal informations>

Here an example of /test/file2.nof

<string1>STRING 1</string1>
<string2>STRING 2</string2>
<string3>STRING 3</string3>
<string4>STRING 4</string4>
<personal informations>
<hobby>Soccer</hobby>
<hobby>Traveling</hobby>
</personal informations>

The File i want to create has to look like this.

STRING 1:::/test/file1.nfo:::Date of file:::STRING 4:::STRING 3:::Baseball, Basketball:::STRING 2
STRING 1:::/test/file2.nfo:::Date of file:::STRING 4:::STRING 3:::Baseball, Basketball:::STRING 2

"Date of file" should be the creation date of the file. So that i can see how old is the file.

So, that´s what i need and it seems not easy.

Thanks a lot.

UPATE ERROR -printf

find: unrecognized: -printf

Usage: find [PATH]... [OPTIONS] [ACTIONS]

Search for files and perform actions on them.
First failed action stops processing of current file.
Defaults: PATH is current directory, action is '-print'

    -follow         Follow symlinks
    -xdev           Don't descend directories on other filesystems
    -maxdepth N     Descend at most N levels. -maxdepth 0 applies
                    actions to command line arguments only
    -mindepth N     Don't act on first N levels
    -depth          Act on directory *after* traversing it

Actions:
    ( ACTIONS )     Group actions for -o / -a
    ! ACT           Invert ACT's success/failure
    ACT1 [-a] ACT2  If ACT1 fails, stop, else do ACT2
    ACT1 -o ACT2    If ACT1 succeeds, stop, else do ACT2
                    Note: -a has higher priority than -o
    -name PATTERN   Match file name (w/o directory name) to PATTERN
    -iname PATTERN  Case insensitive -name
    -path PATTERN   Match path to PATTERN
    -ipath PATTERN  Case insensitive -path
    -regex PATTERN  Match path to regex PATTERN
    -type X         File type is X (one of: f,d,l,b,c,...)
    -perm MASK      At least one mask bit (+MASK), all bits (-MASK),
                    or exactly MASK bits are set in file's mode
    -mtime DAYS     mtime is greater than (+N), less than (-N),
                    or exactly N days in the past
    -mmin MINS      mtime is greater than (+N), less than (-N),
                    or exactly N minutes in the past
    -newer FILE     mtime is more recent than FILE's
    -inum N         File has inode number N
    -user NAME/ID   File is owned by given user
    -group NAME/ID  File is owned by given group
    -size N[bck]    File size is N (c:bytes,k:kbytes,b:512 bytes(def.))
                    +/-N: file size is bigger/smaller than N
    -links N        Number of links is greater than (+N), less than (-N),
                    or exactly N
    -prune          If current file is directory, don't descend into it
If none of the following actions is specified, -print is assumed
    -print          Print file name
    -print0         Print file name, NUL terminated
    -exec CMD ARG ; Run CMD with all instances of {} replaced by
                    file name. Fails if CMD exits with nonzero
    -delete         Delete current file/directory. Turns on -depth option

Answer 1

The pat1,pat2 notation of sed is line based. Think of it like this, pat1 sets an enable flag for its commands and pat2 disables the flag. If both pat1 and pat2 are on the same line the flag will be set, and thus in your case print everything following and including the <user> line. See grymoire's sed howto for more.

An alternative to sed, in this case, would be to use a grep that supports look-around assertions, e.g. GNU grep:

find . -type f -name '*.nfo' | xargs grep -oP '(?<=<user>).*(?=</user>)'

If grep doesn't support -P, you can use a combination of grep and sed:

find . -type f -name '*.nfo' | xargs grep -o '<user>.*</user>' | sed 's:</\?user>::g'

Output:

./file1.nfo:test1
./file2.nfo:test2

Note, you should be aware of the issues involved with passing files on to xargs and perhaps use -exec ... instead.

Answer 2

It so happens that grep outputs in the format you need and is enough for an one-liner.

By default a grep '' *.nfo will output something like:

file1.nfo:random data  
file1.nfo:<user>test1</user>  
file1.nfo:some more random data  
file2.nfo:not needed  
file2.nfo:<user>test2</user>  
file2.nfo:etc etc  

By adding the -P option (Perl RegEx) you can restrict the output to matches only:

grep -P "<user>\w+<\/user>" *.nfo

output:

file1.nfo:<user>test1</user>  
file2.nfo:<user>test2</user>  

Now the -o option (only show what matched) saves the day, but we'll need a bit more advanced RegEx since the tags are not needed:

grep -oP "(?<=<user>)\w+(?=<\/user>)" *.nfo > /test/database.txt

output of cat /test/database.txt:

file1.nfo:test1 
file2.nfo:test2  

Explained RegEx here: http://regex101.com/r/oU2wQ1

And your whole script just became a single command.

Update:

If you don't have the --perl-regexp option try:

grep -oE "<user>\w+<\/user>" *.nfo|sed 's#</?user>##g' > /test/database.txt

Answer 3

All you need is:

find -name '*.nfo' | xargs awk -F'[><]' '{print FILENAME,$3}'

If you have more in your file than just what you show in your sample input then this is probably all you need:

... awk -F'[><]' '/<user>/{print FILENAME,$3}' file

Try this (untested):

> outfile
find -name '*.nfo' -printf "%p %Tc\n" |
while IFS= read -r fname tstamp
do
      awk -v tstamp="$tstamp" -F'[><]' -v OFS=":::" '
          { a[$2] = a[$2] sep[$2] $3; sep[$2] = ", " }
          END {
              print a["string1"], FILENAME, tstamp, a["string4"], a["string3"], a["hobby"], a["string2"]
          }
      ' "$fname" >> outfile
done

The above will only work if your file names do not contain spaces. If they can, we'd need to tweak the loop.

Alternative if your find doesn't support -printf (suggestion - seriously consider getting a modern "find"!):

> outfile
find -name '*.nfo' -print |
while IFS= read -r fname
do
      tstamp=$(stat -c"%x" "$fname")
      awk -v tstamp="$tstamp" -F'[><]' -v OFS=":::" '
          { a[$2] = a[$2] sep[$2] $3; sep[$2] = ", " }
          END {
              print a["string1"], FILENAME, tstamp, a["string4"], a["string3"], a["hobby"], a["string2"]
          }
      ' "$fname" >> outfile
done

If you don't have "stat" then google for alternatives to get a timestamp from a file or consider parsing the output of ls -l - it's unreliable but if it's all you've got...