3

I want to debug a C program running multiple TCP server threads. I cannot set up a sophisticated Debugging tool as I have to debug in an embedded linux(busy box). I hoped it would natively support gdb.

SO I started with gdb. Once I type run, the server seems to run in the background but gdb returns the prompt and "Program received signal SIG64 (Real-time event 64)" message(related to pthread I guess). I know it has to do be something with the main getting forked into several threads. But I have no clue how to debug this. Any starting points would be so helpful.

Also, is there someother "trace" like debuggers, small footprint that I can use?

Please help

user489152
  • 907
  • 1
  • 23
  • 42

5 Answers5

9

Most often, debugging a multi-threaded application is difficult with a debugger. The best way is to either try and isolate the bug to a single-threaded case, or use debug prints in suspect locations until the bug is discovered.

It's no help with your specific issue, but it's the best advice I've learned while working with multi-threaded applications, especially embedded.

Eli Iser
  • 2,788
  • 1
  • 19
  • 29
  • 2
    Given the multithreaded nature of the program, it makes sense to add that, while using debug prints, do not rely on the order of the prints accross different threads. – Alok Save Sep 14 '11 at 07:54
  • 5
    @als : it is a good idea in this case to have a "logger" thread, where you send the message you want to print (via msgqueue for example). The logger thread timestamps message and print them in the order in receive them from the msg queue. – Xavier T. Sep 14 '11 at 08:01
  • Nice suggestion Xavier. Perhaps I have to try that out :) – user489152 Sep 14 '11 at 08:04
  • if the problem is just coming due to just keeping multithread then its useless to isolate the bug to a single-threaded case & study – Jeegar Patel Sep 14 '11 at 08:13
  • @Xavier: It's a good suggestion indeed, It won't help much with Timing sensitive issues though, but worth a try for sure. – Alok Save Sep 14 '11 at 08:30
  • @Als : I agree, the precise timing will probably be off, but the sequence of event should be respected, which might be enough to solve some problems. – Xavier T. Sep 14 '11 at 08:51
  • Mr.32 has given the link of a thread which discusses this logger thread... Check out. It doesnt seem easy. I can see a lot of disclaimers :) http://stackoverflow.com/questions/7360391/in-multi-thread-application-how-can-i-redirect-stderr-stdout-in-separate-file-a – user489152 Sep 14 '11 at 12:58
3

See in such case I usually do this:

  1. Make a thread wise log file and have all stdout and stderr output redirect in that log file ... maybe this will help you for that: In multi thread application how can i redirect stderr & stdout in separate file as per thread?

  2. Keep track of global variables between all threads. Improper use of global variables tends to cause problems.

  3. If you are using a mutex then check that it will not create deadlocks. In conditional & semaphore design always try to track of all those threads on paper.

Community
  • 1
  • 1
Jeegar Patel
  • 26,264
  • 51
  • 149
  • 222
1

I recommend you to use memory access check program like valgrind. In my case, many of bugs are caused by illegal memory handling. It's hard to find bug on multi-threaded program so using memory leak checking program is a better way to figure out bugs causes.

BЈовић
  • 62,405
  • 41
  • 173
  • 273
JayMuzie
  • 341
  • 1
  • 4
  • 16
  • Valrgrind probably won't be able to run on an embedded Linux with BusyBox. – Eli Iser Sep 14 '11 at 07:27
  • @Eli lser I have tried to use valgrind on s3c6410 embedded board. compiling is success but actually i've failed to load valgrind because some of shared object version is mismatched with valgrind. if busybox he has contains proper so files it would work. – JayMuzie Sep 14 '11 at 07:43
  • 1
    I compiled my code with -lmcheck and resolved several "leaky" pointers :) – user489152 Sep 14 '11 at 13:22
0

GDB has a few functions for debugging with multiple threads. For example, you can run the command i threads to view the currently existing threads. Here is a link that may be helpful: https://ftp.gnu.org/old-gnu/Manuals/gdb/html_node/gdb_24.html.

Aloha Churchill
  • 549
  • 4
  • 5
0

some tricky idea ..

  1. add sleep(some times) into the thread to debug as starting point
  2. after program is running, check pid of thread (using ps with -L option)
  3. run gdb program {pid} or call attach {pid} in gdb prompt
  4. after sleep, u can trace next step for that thread

Don't forget attaching must be done before sleep time is gone.

As remarked above, testing on single thread or using text logging facility is good choice.

nop
  • 51
  • 1