0

I need to debug some issue on a production server, and I'm trying to decide whether to use strace or the log4cxx logging library; one crucial characteristic I need to know before, is which of the two would have lower impact on performance (speed, mostly)?

For strace, I'm planning for a commandline roughly like below:

strace -p PID -tt -f -e SOME,SELECTED,SYSCALLS -o FILENAME
# or maybe with "-ff" instead? which one would be faster?

For log4cxx, for simplicity we can probably assume I'm calling the logging functions just before doing each syscall I want to trace. Assume version log4cxx-0.10.0, 64 bit.

If what you would suggest is benchmarking this by myself, then I'd be grateful for help on how to write the benchmarks.

akavel
  • 357
  • 3
  • 10
  • See http://superuser.com/questions/583023/strace-vs-log4cxx-performance-hit – Hennes Apr 15 '13 at 12:55
  • @Hennes: that's also my question, I asked on superuser first, but then I remembered about serverfault and though maybe here is the better place for it as it is related to on-line debugging of production issue? – akavel Apr 15 '13 at 13:37

1 Answers1

1

Strace uses a moderately expensive mechanism, while the logging is normally more lightweight. One thing to look out for, though, is DIY logging that uses synchronous writes without buffering much. That tends to make the application wait while the logging completes for every operation logged. A colleague did that once, while measuring performance, and wondered why his performance was suddenly so bad (;-))

--dave

davecb
  • 211
  • 2
  • 5
  • "Strace uses a moderately expensive mechanism" - could you provide any details, references? Or, most desirably, benchmarks?.. – akavel Apr 18 '13 at 18:05
  • 1
    Strace (or truss) uses ptrace, which to oversimplify sets breakpoints in the code and traps to the tracing process when they're struck. This has start-up costs in finding and setting them, and two process-switches when they're used. As a rule of thumb, think of them as being like a system call. Ltrace/apptrace is lighter-weight, and logging more so again. I was one of the apptrace authors, which we wrote to be able to trace production program. Customers found truss too heavy to allow us to use on production machines. – davecb Apr 19 '13 at 16:50
  • Oh, I supposed there's rather some hook point in syscalls which allows for such a mechanism! Probably argument against this is that it would incur unnecessary cost on the "frequent path". Thank you very much for caring to elaborate! And actually I thus expected ltrace to be *less* efficient, interesting that you say it's opposite! – akavel Apr 19 '13 at 18:46
  • Ltrace sticks an "interposer" function between the application and the library that implements the syscall, so the cost is one extra call and return, plus one printf. Plus linking the library at start-up, of course! – davecb Apr 20 '13 at 16:07