Deleting a text block out of a text file by shell script

Question

I have a big LDIF file (OpenLDAP) and I want to remove one attribute out of every dataset.

Example:

dn: uid=axx,ou=People,dc=myfirma,dc=net
uid: axx
jpegPhoto:: /9j/4AAQSkZJRgABAQEAAAAAAAD/4QBaRXhpZgAATU0AKgAAAAgAAYKYAAIAAAA3
 AAAAGgAAAABUaW1vQ29tICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA
 gICAgICAAAP/sABFEdWNreQABAAQAAABkAAD/4QRuaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS
 4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENlaGlIenJlU3pOVGN6a2M5ZCI/P
 g0KPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iQWRvYmUgWE1Q
 IENvcmUgNS4zLWMwMTEgNjYuMTQ1NjYxLCAyMDEyLzAyLzA2LTE0OjU2OjI3ICAgICAgICAiPg0
 KCTxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3
 ludGF4LW5zIyI+DQoJCTxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHhtbG5zOnhtcE1NP
 pRRX7d7JHn7s/RD/gnRrMcnwyhXKqqxnGT719IrerjgjJ96KK/Hc+iljqiXc9Cn8KIL67xYytlf
 umvzT/ao1dG+MVznjbEMnPP3jxmiivQ4VivrL9Cah5wb8T3e75tpyDzwa9n/Y7vIn+LenkbwuGG
 OvbvRRX2mawSws/QKe6P0w0uPFjDt/uL79q0AcHPtRRX4vLc66a6ngn7fT4+E90N2GaFx+lfnjc
 Q7oEUsrLtAI9TRRX3nCOtGXqYVirf2MUso28bcc+9foj+wJEYvhXb7ufkGPbmiit+Kb/VkvMVM+
 g8YUVT18f8S+bt8h5/Ciivzpbmp+XH7VVqqfGvUNrN80a5HYda83u7aOe02q37wfhRRX7Flb/2a
 n6I5am56h+xrcfZ/jHaxsNy7D26Gv0406INYQnH8A/lRRXyPFn8eL8jSDZ//9k=
postalCode: 12345
mail: xxxx@myfirma.com
sn: Hotzenplotz
c: DE
street: Waldstr 2-4
givenName: Maik
o: myfirma Soft- und Hardware GmbH
l: Entenhausen
telephoneNumber: +49 1234 5678 5678
facsimileTelephoneNumber: +49 1234 5678 5676
roomNumber: 03.034
co: Deutschland
employeeActive: TRUE
cn: Maik Hotzenplotz
description: 9400
st: NRW
objectClass: top
objectClass: person
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: extperson
displayName: Maik Hotzenplotz
structuralObjectClass: inetOrgPerson
entryUUID: c43b735c-2d85-1037-85c2-45672a55bec29
createTimestamp: 20170914104654Z
labeledURI: ldap:///ou=Organization,dc=myfirma,dc=net??sub?(&(objectClass=or
 ganizationalRole)(roleOccupant=uid=axx,ou=People,dc=myfirma,dc=net))
userPassword:: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
employeeID: 3564
entryCSN: 20170914105425.815554Z#000000#000#000000
modifiersName: cn=Manager,dc=myfirma,dc=net
modifyTimestamp: 20170914105425Z

dn: uid=yyy,ou=People,dc=myfirma,dc=net
uid: yyy
jpegPhoto:: /9j/4AAQSkZJRgABAQEAAAAAAAD/4QBaRXhpZgAATU0AKgAAAAgAAYKYAAIAAAA3
 AAAAGgAAAABUaW1vQ29tICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA
 gICAgICAAAP/sABFEdWNreQABAAQAAABkAAD/4QRuaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS
 4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENlaGlIenJlU3pOVGN6a2M5ZCI/P
 g0KPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iQWRvYmUgWE1Q
 IENvcmUgNS4zLWMwMTEgNjYuMTQ1NjYxLCAyMDEyLzAyLzA2LTE0OjU2OjI3ICAgICAgICAiPg0
 KCTxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3
 jV8JtK/wARKja+D2z0ooq+a71FOV2yK9lNurOqBi3GRw+B79P8c1jwSq7OzBvM5YgHaw9BRRW0G
 eZitZJdDH1S2a/J87EXmEZ8sfdwOMiseG3Z7iK3kXdJI24cjhR0NFFdlPY8rEL925DvE+qzaZ4f
 uhJ8y3jiD5h2zkmvnn4g+JW1LxpJZ28MO2P5iUGDjPrRRW8VdHwedVGotGlJq8M/iFbFdPhF3fT
 IfOVjuZVAHSvbrLxNc2FlDCsLbY0AGIh6UUVMYo+c9tLlSP/Z
postalCode: 12345
mail: NDimpfelmoser@myfirma.com
sn: Dimpfelmoser
c: DE
street: Waldstr 2-4
givenName: Nadine
o: myfirma Soft- und Hardware GmbH
l: Entenhausen
telephoneNumber: +49 1234 5678 5672
facsimileTelephoneNumber: +49 1234 5678 5673
co: Deutschland
employeeActive: TRUE
cn: Nadine Dimpfelmoser
description: 9800
st: NRW
objectClass: top
objectClass: person
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: extperson
displayName: Nadine Dimpfelmoser
structuralObjectClass: inetOrgPerson
entryUUID: c4c664da-2d85-1437-85c3-4122a55bec29
createTimestamp: 20170914104654Z
labeledURI: ldap:///ou=Organization,dc=myfirma,dc=net??sub?(&(objectClass=or
 ganizationalRole)(roleOccupant=uid=yyy,ou=People,dc=myfirma,dc=net))
userPassword:: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
employeeID: 3978
entryCSN: 20170914105425.922291Z#000000#000#000000
modifiersName: cn=Manager,dc=myfirma,dc=net
modifyTimestamp: 20170914105425Z

The field jpegPhoto shall be deleted.

I tried in an regular expression emulator this term

    jpegPhoto::[^:]*(?=\n\w*:)

and it's worked in the emulator.

But I cannot get it to work with sed. Or shall I use another tool?

Thanks for any hints.

Ciao Peter Schütt

score 1 · Answer 1 · answered Sep 19 '17 at 15:45

1

sed does not support lookahead. You can do it with a loop:

sed '/^jpegPhoto::/{:a;N;/\n[^:]*:/!ba;s/.*\n//;}' file

answered Sep 19 '17 at 15:45

SLePort

15,211
3
34
44

You could settle for `:a; n; /^ /ba` and use `-n` and `p` at the end – Thor Sep 19 '17 at 20:31
When I use this line only the first dataset is processed and then it stops (seems to wait for input). I use sed (GNU sed) 4.2.2 – user1195024 Sep 20 '17 at 06:23
What is your OS? – SLePort Sep 20 '17 at 12:28

RomanPerekhrest · Accepted Answer · 2017-09-20T09:50:18.577

Yes, there's another powerful tool called awk:

awk '/jpegPhoto/{ f=1 }f && /^[^:]+: /{ f=0 }f{next}1' file

The output:

dn: uid=axx,ou=People,dc=myfirma,dc=net
uid: axx
postalCode: 12345
mail: xxxx@myfirma.com
sn: Hotzenplotz
c: DE
street: Waldstr 2-4
givenName: Maik
o: myfirma Soft- und Hardware GmbH
l: Entenhausen
telephoneNumber: +49 1234 5678 5678
facsimileTelephoneNumber: +49 1234 5678 5676
roomNumber: 03.034
co: Deutschland
employeeActive: TRUE
cn: Maik Hotzenplotz
description: 9400
st: NRW
objectClass: top
objectClass: person
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: extperson
displayName: Maik Hotzenplotz
structuralObjectClass: inetOrgPerson
entryUUID: c43b735c-2d85-1037-85c2-45672a55bec29
createTimestamp: 20170914104654Z
labeledURI: ldap:///ou=Organization,dc=myfirma,dc=net??sub?(&(objectClass=or
 ganizationalRole)(roleOccupant=uid=axx,ou=People,dc=myfirma,dc=net))
userPassword:: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
employeeID: 3564
entryCSN: 20170914105425.815554Z#000000#000#000000
modifiersName: cn=Manager,dc=myfirma,dc=net
modifyTimestamp: 20170914105425Z

dn: uid=yyy,ou=People,dc=myfirma,dc=net
uid: yyy
postalCode: 12345
mail: NDimpfelmoser@myfirma.com
sn: Dimpfelmoser
c: DE
street: Waldstr 2-4
givenName: Nadine
o: myfirma Soft- und Hardware GmbH
l: Entenhausen
telephoneNumber: +49 1234 5678 5672
facsimileTelephoneNumber: +49 1234 5678 5673
co: Deutschland
employeeActive: TRUE
cn: Nadine Dimpfelmoser
description: 9800
st: NRW
objectClass: top
objectClass: person
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: extperson
displayName: Nadine Dimpfelmoser
structuralObjectClass: inetOrgPerson
entryUUID: c4c664da-2d85-1437-85c3-4122a55bec29
createTimestamp: 20170914104654Z
labeledURI: ldap:///ou=Organization,dc=myfirma,dc=net??sub?(&(objectClass=or
 ganizationalRole)(roleOccupant=uid=yyy,ou=People,dc=myfirma,dc=net))
userPassword:: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
employeeID: 3978
entryCSN: 20170914105425.922291Z#000000#000#000000
modifiersName: cn=Manager,dc=myfirma,dc=net
modifyTimestamp: 20170914105425Z

Awfully not in every case the field "postalCode" follows on "jpegPhoto". Sometimes another field follows. — user1195024, Sep 20 '17 at 07:03
Is it possible to use a regular expression instead of a string constant as "postalCode"? In my tries it does not work. — user1195024, Sep 20 '17 at 07:12

score 0 · Answer 3 · answered Sep 19 '17 at 20:32

0

If you know postaCode always is the next field, something like this works:

sed '/^postalCode/p; /^jpegPhoto/,/^postalCode/d' infile

answered Sep 19 '17 at 20:32

Thor

45,082
11
119
130

user1195024 · Answer 4 · 2017-09-20T11:44:28.987

I give up with sed and awk.

But RomanPerekhrest answer is fine.

For archivation my slow solution:

This solution works but it is very slow for a big file:

#!/bin/sh

inJpeg=0
while IFS= read -r line
do
  if [ "${line:0:9}" == "jpegPhoto" ]; then
    inJpeg=1 ; 
  elif [ $inJpeg == 1 ]; then
    if [ "${line:0:1}" != " " ]; then
      inJpeg=0 ; 
      echo "${line}"
    fi
  else  
    echo "${line}"
  fi
done

If you have a better solution I would appreciate.

Thanks for any hint.

Ciao Peter Schütt

Deleting a text block out of a text file by shell script

4 Answers4