1

I am using slapadd to restore a backup. That backup contains 45k entries which takes a while to restore so I need to get some progress update from slapadd. Luckily for me there is the -v switch which gives an output similar to this one:

added: "mail=1@test.org,ou=People,dc=example,dc=org" (00003d53)
added: "mail=2@test.org,ou=People,dc=example,dc=org" (00003d54)
added: "mail=3@test.org,ou=People,dc=example,dc=org" (00003d55)
.########              44.22% eta 05m05s elapsed             04m spd  29.2 k/s 
added: "mail=4@test.org,ou=People,dc=example,dc=org" (00003d56)
added: "mail=5@test.org,ou=People,dc=example,dc=org" (00003d57)
added: "mail=6@test.org,ou=People,dc=example,dc=org" (00003d58)
added: "mail=7@test.org,ou=People,dc=example,dc=org" (00003d59)

Every N entries added, slapadd writes a progress update output line (.######## 44.22% eta 05m05s elapsed ...) which I want to keep and an output line for every entry created which I want to hide because it exposes people's email address but still want to count them to know how many users were imported

The way I thought about hiding emails and showing the progress update is this:

$ slapadd -v ... 2>&1 | tee log.txt | grep '########'
# => would give me real-time progress update
$ grep "added" log.txt | wc -l
# => once backup has been restored I would know how many users were added

I tried different variations of the above, and whatever I try I can't grep the progress update output line.

I traced slapadd as follows:

sudo strace slapadd -v ...

And here is what I get:

write(2, "added: \"mail=3@test.org"..., 78added: "mail=3@test.org,ou=People,dc=example,dc=org" (00000009)
) = 78
gettimeofday({1322645227, 253338}, NULL) = 0
_########    44.22% eta 05m05s elapsed      04m spd  29.2 k/s    ) = 80
write(2, "\n", 1
)

As you can see, the percentage line isn't sent to either stdout or stderr (FYI I have validated with known working and failing commands that 2 is stderr and 1 is stdout)

Q1: Where is the progress update output line going?
Q2: How can I grep on it while sending stderr to a file?

Additional info: I'm running Openldap 2.4.21 on ubuntu server 10.04

1st update: potentially relevant info

From http://www.openldap.org/lists/openldap-bugs/200903/msg00235.html

("meter" being what I called above the "progress update output line")

> No, the code only enables the meter if stdout (was: stderr) is a tty, so
> redirecting it anywhere disables it.  The meter was enabled both by
> "slapadd -l file" and "slapadd<  file", but "cat file | slapadd" did
> disable it.  IMHO not exactly an obvious way..

Still my questions remain as I don't know what to do with that additional info.

Max
  • 3,523
  • 16
  • 53
  • 71
  • strace again and look which file/device belongs to filehandle 2. – mailq Nov 30 '11 at 10:09
  • Don't know how to do it so it will take me a bit of time to find out. In the meantime can you look at the update I put in my question to see if you can advise further from there? Thanks. – Max Nov 30 '11 at 10:17

1 Answers1

3

You've been bitten by a funny behaviour of slapadd. There was a discussion on the developer's mailing list about it. Bottom line is that the pogress meter seems to get disabled when redirecting stderr or is only enabled when STDIN is a tty (I wasn't able to reach a definitive conclusion based on a few google searches - seems you need to dive into the code!).

Other than that, you're on the right track. For Q2; I'm not so sure about your question here, but to redirect STDERR to a file, just do slapadd -v ... 2>YOURFILE. This way you can still grep on STDOUT (1).

Roman
  • 3,907
  • 3
  • 21
  • 34
  • I can indeed redirect `stderr` to a file in which case I get the `added: "mail=...` lines and I can count the number of users imported. However I loose the meter line :( Honestly why would you output the meter line to something different than `stderr` or `stdout`, what's the point of doing this? – Max Nov 30 '11 at 10:27
  • Because it is useless for other programs in the pipe. But I see that you have a use case for that. I haven't tried if you could grab the data from /dev/console or /dev/ttyX while you _not_ redirect the output? – mailq Nov 30 '11 at 11:37
  • I see your point, but in my opinion it would be better to have the meter sent to `stdout` and the emails to `stderr` to facilitate the parsing of these 2 outputs. Anyway, could you give me directions as to how reading data from `dev/console` or `/dev/ttyX` please? – Max Nov 30 '11 at 13:23