3

I have an application that I recently split to run in separate processes that communicate with each other via local sockets. I split it out in hopes of increasing stability, as the core "watcher" process can detect a failure and restart the afflicted sub-process.

However, now my watcher process frequently crashes with only the message "Segmentation Fault". I've surrounded all threaded operations in try/catch blocks to try to dump any output, but I still get the same results.

I have been unable to get the debugger to work in MonoDevelop (so development has been difficult enough without these ghost issues).

Isn't Mono supposed to be in a managed environment to prevent issues like this? Is there any way I can narrow down the source of the issue?

Louis Ingenthron
  • 1,267
  • 8
  • 21
  • 2
    Sounds like you should be trying to get the debugger working. – ricochet1k Apr 25 '12 at 17:47
  • 1
    When a debugger is not available, your option to obtaining one is typically to load up the code with lots of logging output to narrow in on what it was trying to do right before the crash, and the speculate as to what have gone wrong. – Chris Stratton Apr 25 '12 at 17:50
  • I tried to get the debugger working for a few days. Short of building the monodevelop binaries myself (which is a step I really want to avoid), it seemed impossible. – Louis Ingenthron Apr 25 '12 at 19:13
  • I'm going to try logging, but this code is really very compact. A few short networking and monitoring threads with a simple text IO GUI. The amount of time that passes before the fault seems to be random (30secs to hours), so I'm worried it is a data-based error. – Louis Ingenthron Apr 25 '12 at 19:15
  • Ah, and the synchronous nature of the program is going to cause problems because regardless of the output, it could occur in any thread that's tracing... – Louis Ingenthron Apr 25 '12 at 19:19
  • Well, maybe it was a timing issue... I threw in a lot of tracing and a few Thread.Sleep()s, and all of the sudden it's running a lot better... – Louis Ingenthron Apr 25 '12 at 21:29
  • 2
    What version of Mono? Anything less than 2.10.1 and you should upgrade. – Andrew T Finnell Apr 26 '12 at 01:18
  • Ah, I'm running Lucid which only has an official release of 2.4. I'm downloading an unofficial distro of 2.10.8.1. We'll see if that helps. – Louis Ingenthron Apr 26 '12 at 13:35
  • Updating didn't seem to do it... – Louis Ingenthron Apr 27 '12 at 14:15

3 Answers3

9

Segmentation faults must (1) be debugged using gdb. To debug mono using gdb you first need to read this.

Once that's done, start your program, run ps auxf to find the pid of your program, and then execute:

gdb program PID

This will attach gdb to your program. You should be presented with a gdb prompt:

$ (gdb) 

execute the following (from the link you should have read by now):

$ (gdb) handle SIGXCPU SIG33 SIG35 SIGPWR nostop noprint
$ (gdb) continue

and now wait until your program stops responding. When that happens, return to gdb, and you'll hopefully find that your program has stopped at the segmentation fault (SIGSEGV), and you should be able to get more information about the crash. In particular this is useful:

$ (gdb) thread apply all backtrace

which will show the stack trace for all threads.

(1) You can also use the more brute way of sprinkling your code with calls to Console.WriteLine. This is your last resort when everything else fails :)

Rolf Bjarne Kvinge
  • 19,253
  • 2
  • 42
  • 86
  • Okay, finally got this to work, and not one callstack appears to be in my code. There are a lot of references to threads and mutexes (mutices?) – Louis Ingenthron Apr 27 '12 at 14:15
  • Yeah, native stack traces won't necessarily contain your code. Also managed frames aren't symbolicated by default, they usually show up as a lot of ????'s (you can get the symbol though by using `call mono_pmip ()` in gdb, this is explained in the document I linked to). – Rolf Bjarne Kvinge Apr 27 '12 at 20:29
0

Mono is a managed environment, which makes your question really peculiar to me. Under what circumstances are you dealing with memory in a way to actually cause a seg fault? Do you have any portions of code marked unsafe? If you do, I would look there first. Otherwise, there is a way to attach a debugger to an already running process. I would attempt to do that if you aren't debugging out of Monodevelop.

The more I think about it, the seg fault can't be in your code if you never do any memory manipulation yourself. It is probably in the runtime, and if so, try/catches aren't going to help you.

my suggestion would be to make good use of text and/or console logging and get a line by line detailed look at whats going on. Or hop in visual studio and do some traces.

Dabloons
  • 1,462
  • 2
  • 17
  • 30
  • I wish I had Visual Studio. Unfortunately I'm working on an Ubuntu box. I'm not using any unsafe code, and I'm watching the RAM on my machine... I never drop below 70% of the 8GB free, so it can't be that either. – Louis Ingenthron Apr 25 '12 at 19:11
  • A seg fault can happen for example if the program uses WinForms controls from Windows, which is not fully Mono compatible. I met that once when using DockPanel Suite. So even if you don't use unsafe it can occur. – Lex Li Apr 26 '12 at 00:49
  • No, I'm using Gtk#. Thanks, though. – Louis Ingenthron Apr 26 '12 at 13:25
-1

Well, it must have been something with my threading of sockets. I switched to using the Begin/End functions instead of handling the threading myself and that fixed the issue.

Louis Ingenthron
  • 1,267
  • 8
  • 21