7

I have a CherryPy script that I frequently run to start a server. Today I was having to start and stop it a few times to fix some bugs in a config file, and I guess the socket didn't close all the way because when I tried to start it up again I got this issue:

[23/Mar/2015:14:08:00] ENGINE Listening for SIGHUP.
[23/Mar/2015:14:08:00] ENGINE Listening for SIGTERM.
[23/Mar/2015:14:08:00] ENGINE Listening for SIGUSR1.
[23/Mar/2015:14:08:00] ENGINE Bus STARTING
CherryPy Checker:
The Application mounted at '' has an empty config.

[23/Mar/2015:14:08:00] ENGINE Started monitor thread 'Autoreloader'.
[23/Mar/2015:14:08:00] ENGINE Started monitor thread '_TimeoutMonitor'.
[23/Mar/2015:14:08:00] ENGINE Error in HTTP server: shutting down
Traceback (most recent call last):
  File "/home/andrew/virtualenvs/mikernels/lib/python2.7/site-packages/cherrypy/process/servers.py", line 188, in _start_http_thread
    self.httpserver.start()
  File "/home/andrew/virtualenvs/mikernels/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 1848, in start
    raise socket.error(msg)
error: No socket could be created

I edited CherryPy's wsgiserver2.py to see the details of the socket.error and error.strerror was 

98  (98, 'Address already in use') Address already in use

Meanwhile my socket is constructed as:

af = 2
socktype = 1
proto = 6
canonname = ''
sa = ('0.0.0.0', 2112)
self.bind(af, socktype, proto)

(that's not exact code but that's what the values are when the error is fired)

I checked netstat and didn't see anything listening on port 2112, what could be causing the problem and how can I go about diagnosing it?

Thanks!

Andrew Latham
  • 5,982
  • 14
  • 47
  • 87

3 Answers3

21

You can try the following

from socket import *

sock=socket()
sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
# then bind

From the docs:

The SO_REUSEADDR flag tells the kernel to reuse a local socket in TIME_WAIT state, without waiting for its natural timeout to expire.

Here's the complete explanation:

Running an example several times with too small delay between executions, could lead to this error:

socket.error: [Errno 98] Address already in use

This is because the previous execution has left the socket in a TIME_WAIT state, and can’t be immediately reused.

There is a socket flag to set, in order to prevent this, socket.SO_REUSEADDR:

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
ForceBru
  • 43,482
  • 10
  • 63
  • 98
  • Thanks, but I see in the code I'm running that CherryPy actually does this. Line 1884: self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) I put a print statement after that line to see if it wasn't being executed but it seems like it is. – Andrew Latham Mar 23 '15 at 18:24
  • The problem seems to be on self.socket.bind(bind_addr) with bind_addr ('0.0.0.0', 2112) – Andrew Latham Mar 23 '15 at 18:26
  • @AndrewLatham, is `self.socket` the socket that is used for binding? – ForceBru Mar 23 '15 at 18:26
  • Yes. Also I tried netstat -n -A inet and saw a lot of TIME_WAIT, CLOSE_WAIT, and ESTABLISHED stuff on 2112. – Andrew Latham Mar 23 '15 at 18:27
  • @AndrewLatham, then it's not a Python's problem. You can use `ps ax | grep 2112` to try to find programs using this port. After that, you can try to kill all python's processes: `killall python`. – ForceBru Mar 23 '15 at 18:32
  • OK, thanks. There was a telnet listening, I killed it, things still aren't starting, maybe I need to wait a few minutes... although netstat -n A inet still has tons of stuff on 2112 even thouse ps ax | grep 2112 comes up with nothing (except the grep itself) – Andrew Latham Mar 23 '15 at 18:40
  • @AndrewLatham, well if you want this socket to be free right now, reboot the device you're running the script on. You can also wait for some time, but this can be too long. – ForceBru Mar 23 '15 at 18:42
  • Ends up they were all caused by the same process that just failed to die every time I started it. ps -ax -U andrew | grep server.py and killing all the related pids did that. Thanks for your help though! – Andrew Latham Mar 23 '15 at 20:10
  • Thank you so much. This fixed the problem entirely for me. I'm glad my test suite caught this. – byxor Nov 28 '16 at 23:26
  • Not work for `cli = socket.socket(socket.AF_UNIX, socket.SO_REUSEADDR, 1)` – Mithril Dec 28 '16 at 08:59
  • @Mithril, the second argument of `socket.socket` is the socket's _type_, one of the `SOCK_` constants. You can't set socket's options at initialization, you have to use `setsockopt` as stated in my answer. – ForceBru Dec 28 '16 at 09:29
  • @ForceBru Is it possible to clear the address immediately after use i.e. to put the socket not in a TIME_WAIT state after finishing listening? Some kind of `close()` method that "frees" the address? I tried `socket.close()` which doesn't work – black May 28 '20 at 15:48
16

You could find the process and kill it by doing:

ps aux | grep python

, finding the process ID, and stopping it manually by doing:

sudo kill -9 PID

replacing PID with your PID.

I often have to do this while testing with Flask/CherryPy. Would be interested to see if there's an easier way (for e.g. to prevent it in the first place)

Ryan
  • 3,555
  • 1
  • 22
  • 36
12

Much more easier to do it by:

Check the PID(:5000 is the host since I've been running on 127.0.0.1:5000):
$ lsof -i :5000
Then kill it:
$ sudo kill -9 PID

Lord Salforis
  • 572
  • 1
  • 6
  • 16