extract words from a file

Question

I'm trying to create a dictionary of words from a collection of files. Is there a simple way to print all the words in a file, one per line?

Answer 1

You could use grep:

• -E '\w+' searches for words
• -o only prints the portion of the line that matches

% cat temp
Some examples use "The quick brown fox jumped over the lazy dog,"
rather than "Lorem ipsum dolor sit amet, consectetur adipiscing elit"
for example text.
# if you don't care whether words repeat
% grep -o -E '\w+' temp
Some
examples
use
The
quick
brown
fox
jumped
over
the
lazy
dog
rather
than
Lorem
ipsum
dolor
sit
amet
consectetur
adipiscing
elit
for
example
text

If you want to only print each word once, disregarding case, you can use sort

• -u only prints each word once
• -f tells sort to ignore case when comparing words

# if you only want each word once
% grep -o -E '\w+' temp | sort -u -f
adipiscing
amet
brown
consectetur
dog
dolor
elit
example
examples
for
fox
ipsum
jumped
lazy
Lorem
over
quick
rather
sit
Some
text
than
The
use

Answer 2

A good start is to simply use sed to replace all spaces with newlines, strip out the empty lines (again with sed), then sort with the -u (uniquify) flag to remove duplicates, as in this example:

$ echo "the quick brown dog and fox jumped
over the lazy   dog" | sed 's/ /\n/g' | sed '/^$/d' | sort -u

and
brown
dog
fox
jumped
lazy
over
quick
the

Then you can start worrying about punctuation and the likes.

Answer 3

assuming words separated by white spaces

awk '{for(i=1;i<=NF;i++)print $i}' file

or

 tr ' ' "\n" < file

if you want uniqueness:

awk '{for(i=1;i<=NF;i++)_[$i]++}END{for(i in _) print i}' file

tr ' ' "\n" < file | sort -u

with some punctuations removed.

awk '{
    gsub(/["*^&()#@$,?~]/,"")
    for(i=1;i<=NF;i++){  _[$i]  }
}
END{    for(o in _){ print o }  }' file

Answer 4

Ken Church's "Unix(TM) for Poets" (PDF) describes exactly this type of application - extracting words out of text files, sorting and counting them, etc.

Answer 5

The tr command can do this...

tr [:blank:] '\n' < test.txt

This asks the tr program to replace white space with a new line. The output is stdout, but it could be redirected to another file, result.txt:

tr [:blank:] '\n' < test.txt > result.txt

Refer here.