3

I have posted to the python and eventlet mailing list already so I apologize if I seem impatient.

I am running eventlet 0.9.16 on a Small (not micro) reserved ubuntu 11.10 aws instance.

I have a socketserver that is similar to the echo server from the examples in the eventlet documentation. When I first start running the code, everything seems fine, but I have been noticing that after 10 or 15 hours the cpu usage goes from about 1% to 99+%. At that point I am unable to make further connections to the socketserver.

This is the code that I am running:

    def socket_listener(self, port, socket_type): 
        L.LOGG(self._CONN, 0, H.func(), 'Action:Starting|SocketType:%s' % socket_type)   
        listener = eventlet.listen((self._host, port)) 
        listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        pool = eventlet.GreenPool(20000)
        while True: 
            connection, address = listener.accept() 
            connection.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
            L.LOGG(self._CONN, 0, H.func(), 'IPAddress:%s|GreenthreadsFree:%s|GreenthreadsRunning:%s' % (str(address[0]), str(pool.free()),str(pool.running())))
            pool.spawn_n(self.spawn_socketobject, connection, address, socket_type)
        listener.shutdown(socket.SHUT_RDWR)
        listener.close()

The L.LOGG method simply logs the supplied parameters to a mysql table.

I am running the socket_listener in a thread like so:

def listen_phones(self):  
    self.socket_listener(self._port_phone, 'phone') 

t_phones = Thread(target = self.listen_phones)
t_phones.start()

From my initial google searches I thought the issue might be similar to the bug reported at https://lists.secondlife.com/pipermail/eventletdev/2008-October/000140.html but I am using a new version of eventlet so surely that cannot be it?

tedtoy
  • 168
  • 2
  • 12
  • Ok, I'll ask. Are you sure that the blocking operation within your `while True` is actually blocking? Perhaps it's returning immediately due to an error condition or the exhaustion of some resource. – phs Feb 03 '12 at 05:43
  • Well I guess I am not ruling out an error condition. However that seems unlikely to me given that clients will connect successfully multiple times before they become unable to connect. The cpu will stay around 1-3% during this time before suddenly hitting 100%. I am not sure what resource it could be exhausting as there is very little in the way of other applications being run on this server. There is an apache and mysql instance but they are both doing virtually no work. Also I am not necessarily sure that it is a socket related blocking problem either. It is perplexing. – tedtoy Feb 03 '12 at 06:29
  • tedtoy, did you finally found the problem? i have similar issue with eventlet - http://stackoverflow.com/questions/9124120/python-consumes-99-of-cpu-running-eventlet – Andrew Mar 20 '12 at 11:05

2 Answers2

2

If listener.accept() is non-blocking, you should put the thread to sleep for a small amount of time, so that the os scheduler can dispatch work to other processes. Do this by putting

time.sleep(0.03)

at the end of your while True loop.

Chris
  • 1,613
  • 1
  • 18
  • 27
  • I'm not sure I like the idea of delaying the loop since I want it to run quickly, but I will try this to see if it helps and update later.. – tedtoy Feb 03 '12 at 08:20
  • You only need to wait for a few milliseconds. This is by the way something that you should do every time you run an endlees loop, because this is exactly what causes the high cpu load. – Chris Feb 03 '12 at 08:22
1

Sorry for late reply.

There was no code like listener.setblocking(0), therefore, it MUST behave as blocking and no sleep must be required.

Also, please use a tool like ps or top to at least ensure that it's python process who eats all CPU.

If the issue still persists, please, report it to one of these channels, whichever you like:

temoto
  • 5,394
  • 3
  • 34
  • 50