Asyncore client in thread makes the whole program crash when sending data immediately

Question

I write a simple program in python, with asyncore and threading. I want to implement a asynchorous client without blocking anything, like this:

How to handle asyncore within a class in python, without blocking anything?

Here is my code:

import socket, threading, time, asyncore
class Client(asyncore.dispatcher):
    def __init__(self, host, port):
        asyncore.dispatcher.__init__(self)
        self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
        self.connect((host, port))
mysocket = Client("",8888)
onethread = threading.Thread(target=asyncore.loop)
onethread.start()
# time.sleep(5)
mysocket.send("asfas\n")
input("End")

Now a exception will be throwed in send("asfas\n"), because I didn't open any server.

I think the exception in send function will call the handle_error function and won't affect the main program, but most of the time it crashes the whole program, and sometimes it works! And if I uncomment the time.sleep(5), it will only crash the thread. Why does it behave like this? Could I write a program that won't crash the whole program and don't use time.sleep() ? Thanks! Error message:

Traceback (most recent call last):
  File "thread.py", line 13, in <module>
    mysocket.send("asfas\n")
  File "/usr/lib/python2.7/asyncore.py", line 374, in send
    result = self.socket.send(data)
socket.error: [Errno 111] Connection refused

mguijarr · Accepted Answer · 2014-12-11T14:28:10.163

1

First of all, I would suggest not using the old asyncore module but to look into more modern and more efficient solutions: gevent, or going along the asyncio module (Python 3.4), which has been backported somehow to Python 2.

If you want to use asyncore, then you have to know:

be careful when using sockets created in one thread (the main thread, in your case), and dispatched by another thread (managed by "onethread", in your case), sockets cannot be shared like this between threads it is not threadsafe objects by themselves
for the same reason, you can't use the global map created by default in asyncore module, you have to create a map by thread
when connecting to a server, connection may not be immediate you have to wait for it to be connected (hence your "sleep 5"). When using asyncore, "handle_write" is called when socket is ready to send data.

Here is a newer version of your code, hopefully it fixes those issues:

import socket, threading, time, asyncore

class Client(threading.Thread, asyncore.dispatcher):
    def __init__(self, host, port):
        threading.Thread.__init__(self)
        self.daemon = True
        self._thread_sockets = dict()
        asyncore.dispatcher.__init__(self, map=self._thread_sockets)

        self.host = host
        self.port = port
        self.output_buffer = []
        self.start()

    def run(self):
        self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
        self.connect((self.host, self.port))
        asyncore.loop(map=self._thread_sockets)

    def send(self, data):
        self.output_buffer.append(data)

    def handle_write(self):
        all_data = "".join(self.output_buffer)
        bytes_sent = self.socket.send(all_data)
        remaining_data = all_data[bytes_sent:]
        self.output_buffer = [remaining_data]

mysocket = Client("",8888)
mysocket.send("asfas\n")

If you have only 1 socket by thread (i.e a dispatcher's map with size 1), there is no point using asyncore at all. Just use a normal, blocking socket in your threads. The benefit of async i/o comes with a lot of sockets.

EDIT: answer has been edited following comments.

edited Dec 11 '14 at 14:28

answered Dec 10 '14 at 15:23

mguijarr

7,641
6
45
72

Thanks for your answer, but I tried your code and the problem still exists. Sometimes the error is limited in the thread and sometimes it makes the whole program crash. What I need is to control a asyncore client without any blocking and it's error could be handle in handle_error function. So even I need only one socket I think I need to use thread. – Paul Dec 11 '14 at 01:14
I mean I need to use asyncore in thread. Because I think handle_read is convenient. – Paul Dec 11 '14 at 02:31
Hi! Which problem still exists exactly? – mguijarr Dec 11 '14 at 05:26
It seems that send is sometimes called before the asyncore.loop, and result in "socket.error: [Errno 111] Connection refused", and handle_error won't be called. – Paul Dec 11 '14 at 05:34
And I tried to move the connected_event.set after the asyncore.loop, and found the program was blocked at wait_connected. – Paul Dec 11 '14 at 05:42
And I added time.sleep(0.01) after wait_connected and everything worked as I expected. But this seems ugly. Maybe I need to use some other library. Do you have a better solution? – Paul Dec 11 '14 at 06:13
And wait_connected is blocking? And the blocking of send seems solvable， due to existence of handle_write. – Paul Dec 11 '14 at 06:30
Oh it seems the problem is resolved! I could use handle_write to send data. – Paul Dec 11 '14 at 06:39
Good to know your problem is solved. What did you do to solve it? – mguijarr Dec 11 '14 at 10:28
I define a handle_write function in Client to send data instead of using mysocket.send(""). Like this:https://docs.python.org/2/library/asyncore.html#asyncore-example-basic-http-client This could make sure the data is sent after the connection is established. I also removed wait_connected because it's blocking, and I defined a class that inherit Thread and dispatcher class like yours. – Paul Dec 11 '14 at 11:24
Ok! Indeed, I forgot about handle_write it's the good solution for not sending too early. I updated my answer. – mguijarr Dec 11 '14 at 14:29
Now my program meets another problem. http://stackoverflow.com/questions/27427960/define-writable-method-in-asyncore-client-makes-sending-data-very-slow :-) – Paul Dec 12 '14 at 01:21

Asyncore client in thread makes the whole program crash when sending data immediately

1 Answers1

Linked