Regular Expression to strip away single quotes and preserve apostrophes

Question

I want to parse words from a text file. Apostrophes should be preserved, but single quotes should be removed. Here is some test data:

john's apostrophe is a 'challenge'

I am experimenting with grep as follows:

grep -o "[a-z'A-Z]*" file.txt

and it produces:

john's
apostrophe
is
a
'challenge'

Need to get rid of those quotes around the word challenge.

The correct/desired output should be:

john's
apostrophe
is
a
challenge

EDIT: As the consensus seems to be that apostrophes are problematic to recognize, I am now seeking a way to strip any kind of apostrophe (leading, trailing, embedded) out of all words. The words are to be added to a vocabulary index. The phrase searching should also strip out apostrophes. This may need another question.

Answer 1

Do you need to use grep? Here's a sed example just in case:

$ echo "john's apostrophe is a 'challenge'" | sed -re "s/'(\S*)'/\1/g"
john's apostrophe is a challenge

sed is a stream editor, I used it to perform a substitution (the format is s/pattern/subst/, g stands for global. I'm matching an arbitrary number (*) of non-whitespace characters (\S) and substitute it by the same group of characters, referring to it as \1 (I captured it with round brackets (...).

Edit: All right, here's an ugly Perl-like grep example:

$ echo "john's apostrophe is a 'challenge'" | grep -oP "(?<=')\S*(?=')|\w+'?\w*"
john's
apostrophe
is
a
challenge

I have no idea what I've done, so unexpected behavior is likely :)

With grep I used positive lookaround assertions to match either a word in single quotes (the assertions are used for the quotes not to be a part of the match) or (|) a word with an optional apostrophe, which is represented with "one or more word characters" (\w+) followed by ' (or not) and then optionally some word characters again.

More edit: here's a sed command that seems to do the job and copes with @tchrist's example:

$ echo "john's apostrophe is a 'challenge'" | sed -re "s/(\W|^) '(\w*)'(\W|$)/\1\2\3/g"
john's apostrophe is a challenge
$ echo "’Tis especially hard, ’tisn’t it now, to leave it for the dogs’ breakfast, let a lone for the cats'" | sed -re "s/(\W|^)'(\w*)'(\W|$)/\1\2\3/g"
’Tis especially hard, ’tisn’t it now, to leave it for the dogs’ breakfast, let a lone for the cats'

Answer 2

Here's a simpler grep-only approach:

grep -E -o "[a-zA-Z]([a-z'A-Z]*[a-zA-Z])?" file.txt

which in Java is:

Pattern.compile("[a-zA-Z]([a-z'A-Z]*[a-zA-Z])?")

(Both of those mean "an ASCII letter, optionally followed by a mixture of ASCII letters and/or apostrophes and an ASCII letter". The idea being that the matched substring has to start with a letter and end with a letter, but if it's more than two characters long, then it can contain apostrophes.)

To accept non-ASCII letters, the Java could be written as:

Pattern.compile("\\p{L}([\\p{L}']*\\p{L})?")

---

Edit for updated question (stripping out apostrophes): I don't think you can do that with just grep; but expanding our repertoire a bit, you can write:

tr -d "'" file.txt | grep -E -o "[a-zA-Z]+"

or in Java:

String apostrippedStr = str.replace("'", "");

Pattern.compile("[a-zA-Z]+") // or "\\p{L}+" for non-ASCII support
// ... apply pattern to apostrippedStr