Signal passing to managed processes using supervisord

Question

I am using supervisord to spawn and manage a FastCGI application that I am writing in C for a linux target. I have a signal handler that gracefully exits my application when SIGINT is received. I have verified that the signal handler works as desired by running the app in a terminal window and issuing Ctrl-C to exit.

When issuing a "shutdown" command to supervisord (via supervisorctl), it appears that supervisord is unable to force the app to exit without invoking SIGKILL:

2010-08-20 10:02:49,661 INFO waiting for cse to die
2010-08-20 10:02:52,665 INFO waiting for cse to die
2010-08-20 10:02:55,669 INFO waiting for cse to die
2010-08-20 10:02:58,672 INFO waiting for cse to die
2010-08-20 10:02:59,673 WARN killing 'cse' (2031) with SIGKILL
2010-08-20 10:02:59,674 INFO stopped: cse (terminated by SIGKILL)

I have the following in my supervisord.conf file

stopsignal=INT

It is my assumption that supervisord issues "stopsignal" at the invocation of the shutdown command, so I take the INFO statements as an indication that my app is not responding to the SIGINT issued by supervisord.

How do I go about debugging the signal passing between supervisord and my app?

I confirmed that INT is issued at the invocation of the shutdown command, so the "...waiting for cse to die" log entries are definitely indicative of the application not seeing SIGINT. — HikeOnPast, Aug 20 '10 at 20:44

score 5 · Accepted Answer · answered Aug 31 '10 at 15:14

5

After some more digging, it appears that the issue is not that supervisord is failing to pass signals to child processes. This seems to be working normally.

Rather, it appears that supervisord stops logging child process stderr output once it receives a request to invoke stopsignal. As a result, any stderr-based logging of the child process shutdown will not be processed by supervisord. In my case, this made it appear that the child process was not receiving SIGINT, since it wasn't logging anything to stderr after the signal was invoked.

answered Aug 31 '10 at 15:14

HikeOnPast

456
4
13

1

Is there a way to get `supervisord` to continue collecting log output after a process has been sent `SIGINT` from a `stop` command? – Neil May 17 '12 at 18:15
1

+1 on @Neil's question. Not seeing `stdout` and `stderr` makes impossible to debug problems during shutdown. – Yevgeniy Brikman Dec 09 '15 at 05:38

score 3 · Answer 2 · edited Oct 25 '12 at 10:07

3

This was a bug that has since been fixed in Supervisor's 3.0a10 (2011-03-30) release as per this commit: https://github.com/Supervisor/supervisor/commit/e19cbc185dfad045c8775750d36ab8ceed4c4dfb

edited Oct 25 '12 at 10:07

Community

1
1

answered Sep 06 '12 at 20:14

Neil

24,551
15
60
81

score 3 · Answer 3 · answered Aug 23 '10 at 19:40

3

You can run supervisord on the command line in debug mode and get more information.

supervisord -n -e debug

answered Aug 23 '10 at 19:40

Roger Hoover

596
4
5

Thanks. The output that I originally posted was generated using the supervisord.conf equivalent of "-n -e debug" (don't daemonize and use debug level logging). – HikeOnPast Aug 23 '10 at 20:54

Signal passing to managed processes using supervisord

3 Answers3