Design question on Python network programming

Question

I'm currently writing a project in Python which has a client and a server part. I have troubles with the network communication, so I need to explain some things...

The client mainly does operations the server tells him to and sends the results of the operations back to the server. I need a way to communicate bidirectional on a TCP socket.

Current Situation

I currently use a LineReceiver of the Twisted framework on the server side, and a plain Python socket (and ssl) on client side (because I was unable to correctly implement a Twisted PushProducer). There is a Queue on the client side which gets filled with data which should be sent to the server; a subprocess continuously pulls data from the queue and sends it to the server (see code below).

This scenario works well, if only the client pushes its results to the manager. There is no possibility the server can send data to the client. More accurate, there is no way for the client to receive data the server has sent.

The Problem

I need a way to send commands from the server to the client.

I thought about listening for incoming data in the client loop I use to send data from the queue:

def run(self):
    while True:
        data = self.queue.get()
        logger.debug("Sending: %s", repr(data))
        data = cPickle.dumps(data)
        self.socket.write(data + "\r\n")
        # Here would be a good place to listen on the socket

But there are several problems with this solution:

the SSLSocket.read() method is a blocking one
if there is no data in the queue, the client will never receive any data

Yes, I could use Queue.get_nowait() instead of Queue.get(), but all in all it's not a good solution, I think.

The Question

Is there a good way to achieve this requirements with Twisted? I really do not have that much skills on Twisted to find my way round in there. I don't even know if using the LineReceiver is a good idea for this kind of problem, because it cannot send any data, if it does not receive data from the client. There is only a lineReceived event.

Is Twisted (or more general any event driven framework) able to solve this problem? I don't even have real event on the communication side. If the server decides to send data, it should be able to send it; there should not be a need to wait for any event on the communication side, as possible.

Good questions, but you accepted a non-answer. :-( I don’t care if you call it server/client/whatever. – P2P is not a twisted thing? — Robert Siemer, Nov 09 '13 at 09:27

score 2 · Answer 1 · answered Sep 28 '10 at 21:28

2

"I don't even know if using the LineReceiver is a good idea for this kind of problem, because it cannot send any data, if it does not receive data from the client. There is only a lineReceived event."

You can send data using protocol.transport.write from anywhere, not just in lineReceived.

answered Sep 28 '10 at 21:28

Forrest Voight

2,249
16
21

Yes, this is the problem I encountered with the `LineReceiver`. – Manuel Faux Oct 01 '10 at 07:42
And ... I'm proposing a solution - call `protocol.transport.write` from somewhere other than `lineReceived`. – Forrest Voight Oct 04 '10 at 01:35

S.Lott · Accepted Answer · 2010-09-22T21:38:33.313

-1

"I need a way to send commands from the server to the client."

Don't do this. It inverts the usual meaning of "client" and "server". Clients take the active role and send stuff or request stuff from the server.

Is Twisted (or more general any event driven framework) able to solve this problem?

It shouldn't. You're inverting the role of client and server.

If the server decides to send data, it should be able to send it;

False, actually.

The server is constrained to wait for clients to request data. That's generally the accepted meaning of "client" and "server".

"One to send commands to the client and one to transmit the results to the server. Does this solution sound more like a standard client-server communication for you?"

No.

If a client sent messages to a server and received responses from the server, it would meet more usual definitions.

Sometimes, this sort of thing is described as having "Agents" which are -- each -- a kind of server and a "Controller" which is a single client of all these servers.

The controller dispatches work to the agents. The agents are servers -- they listen on a port, accept work from the controller, and do work. Each Agent must do two concurrent things (usually via the select API):

Monitor a well-known socket on which it will receive work from the one-and-only client.
Do the work (in the background).

This is what Client-Server usually means.

If each Agent is a Server, you'll find lots of libraries will support this. This is the way everyone does it.

edited Sep 22 '10 at 21:38

answered Sep 22 '10 at 19:25

S.Lott

384,516
81
508
779

1

Your answer is, not to do that this way, because no one does it this way? I have a problem and need a solution, i don't think questioning the problem is straightforward here. I've thought about connecting two TCP sockets, like FTP does. One to send commands to the client and one to transmit the results to the server. Does this solution sound more like a _standard_ client-server communication for you? – Manuel Faux Sep 22 '10 at 19:47
2

If you don't like the words "client" and "server" then how about "master" and "slave"? Or "manager" and "worker"? There's nothing wrong with a design that many processes connecting to one and then receiving commands from it. Your "accept meanings" are needlessly restricted. – Jean-Paul Calderone Sep 22 '10 at 20:13
2

S.Lott, you might want to read up on the design and terminology of the X protocol. The words "client" and "server" do not always mean what you seem to think they mean. – ʇsәɹoɈ Sep 22 '10 at 20:17
1

Having a clear semantics which side is supposed to do what actually helps in keeping the software implementation sane, and problem free. Having a server push data to the client (vs. the client polling from the server) will bring you problems down the road. – knitti Sep 22 '10 at 20:20
If you don't like the way I use servers and clients, please tell me another way to solve my problem: A manager tells its workers to do a job and they continuously send it the results of the work until the manager tells them to stop or to do something else. – Manuel Faux Sep 22 '10 at 20:25
Request from the manager a chunk of work, do it and report back when you're done. The difference seems subtle, but your approach will have scaling issues and hard to troubleshoot communication issues (been there, done that) – knitti Sep 22 '10 at 20:29
I actually this a bit back: it depends. Pull architectures are easier to build up, I think. – knitti Sep 22 '10 at 20:37
_The difference seems subtle..._ The difference to what? What is the difference between our approaches? I currently do exactly what you are describing, except there is no _done_. The workers get jobs which are not done until the manager tells them to stop. – Manuel Faux Sep 22 '10 at 20:47
@forest: The X client applications make requests of the X server. The X server (running on your desktop) has events queued which the X client application must accept. I think the meanings are the same. The X server can't communicate randomly to a client. My understanding is that the client -- effectively -- polls the X server for mouse and keyboard events. – S.Lott Sep 22 '10 at 21:35
How about P2P? Or let me rephrase the questions: Is Twisted good for P2P? Is any other event driven framework ideal for P2P? – Robert Siemer Nov 09 '13 at 09:28

Design question on Python network programming

Current Situation

The Problem

The Question

2 Answers2