grep exclude multiline pattern

Question

Searching for multiline patterns in files with grep is trivial. Inverting that pattern not so much.

Background: I want to create clean variable files without passwords so I can commit them into a repository.

Example of a variable file containing an encrypted password:

ansible_user: rick
ansible_become_password: !vault |
        $ANSIBLE_VAULT;1.1;AES256
        35623732646263636163383738353230626565383533626261313564383832643334363632383134
        3833316539376436333462303564636236646662376535300a356631346166626632333365353465
        30343138313363666434343938393464343861666234633434383037393230633333333364383835
        3962383339373731610a316362326239386539633638646331636633333330633231383730323634
        33653332353239353662366631373037653135303163663365633532643535663933
never: 'gonna,give,you,up'

Intended result:

ansible_user: rick
never: 'gonna,give,you,up'

I can easily match the lines containing the passwords with this command:

grep -Pz '.+\: !vault \|(\n\s+.+){2,}' host_vars/host.yml

The problem: The parameter -v, which usually inverts the result, doesn't work with -P

How can I create a copy of the variable files without the password lines?

score 0 · Answer 1 · answered May 09 '19 at 20:04

0

It's hackish because it wouldn't work in you have other multiline variables, but in this case:

grep -v -e password -e '^\s'

So it would just skip anything that starts with white space. If you want a more elegant solution I would use a command line program for dealing with yaml files and see if you can exclude values. Dealing with yaml files using the cli is covered in detail in other posts on this site: https://stackoverflow.com/questions/5014632/how-can-i-parse-a-yaml-file-from-a-linux-shell-script

answered May 09 '19 at 20:04

Some Linux Nerd

3,327
3
19
22

Sadly I do have more multiline variables, so this is out. Parsing and recreating the yaml was something I hoped to avoid. – Gerald Schneider May 10 '19 at 05:35
This will also remove lists and dicts. – Gerald Schneider May 10 '19 at 05:36

score 0 · Accepted Answer · answered May 16 '19 at 10:30

0

I got it working using pcregrep:

pcregrep -vM "\!vault.+\n(\s{4,}.+\n){2,}" host_vars/host.yml

But that requires to install pcregrep, which at least on ubuntu is not standard. Therefore I'm still open for better suggestions.

answered May 16 '19 at 10:30

Gerald Schneider

23,274
8
57
89

grep exclude multiline pattern

2 Answers2