3

The following code does not work correctly on Windows (but does on Linux):

    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.setblocking(True)
    sock.connect(address)
    gobject.io_add_watch(
            sock.fileno(),
            gobject.IO_OUT | gobject.IO_ERR | gobject.IO_HUP,
            callback)

Snippets of comments in various places in the glib source, and other places mention that in Windows, sockets are put in non-blocking mode during polling. As a result the callback self.outgoing_cb is constantly called, and writing to the socket fails with this error message:

[Errno 10035] A non-blocking socket operation could not be completed immediately

Calling sock.setblocking(True) prior to writing does not seem to circumvent this. By lowering the priority of the polling, and ignoring the error message, it works as expected, but throws far to many events, and consumes a lot of CPU. Is there a way around this limitation in Windows?

Update

I might point out, that the whole point of polling for POLLOUT is that when you make the write call you won't get EAGAIN/EWOULDBLOCK. The strange error message that I'm getting, I believe would be the Windows equivalent of those 2 error codes. In other words, I'm getting gobject.IO_OUT events when the socket will not let me write successfully, and putting it into blocking mode still gives me this inappropriate error.

Another update

On Linux, where this works correctly, the socket is not switched to non-blocking mode, and I receive IO_OUT, when the socket will let me write without blocking, or throwing an error. It's this functionality I want to best emulate/restore under Windows.

Further notes

From man poll:

   poll()  performs a similar task to select(2): it waits for one of a set
   of file descriptors to become ready to perform I/O.
          POLLOUT
                 Writing now will not block.

From man select:

A file descriptor  is considered ready if it is possible to perform the corre‐
sponding I/O operation (e.g., read(2)) without blocking.
Matt Joiner
  • 112,946
  • 110
  • 377
  • 526
  • Regarding what you've cited from the manpage: It will not block for a certain amount of bytes, correct. HOWEVER... you will still get EAGAIN at the very end of your successful operations. And it doesn't tell you what that amount of bytes is. So the proper reponse to one of these events is to successfully read or write as much as you can, then get EAGAIN (also known as EWOULDBLOCK) when you have exhausted that number. – asveikau Nov 21 '09 at 00:55
  • @asveikau: EAGAIN only applies for nonblocking sockets. If select() returns that a blocking socket is ready to be written to, it will receive as many bytes as it can and return that number, immediately. – Matt Joiner Nov 21 '09 at 02:23
  • @Anacrolix I thought we were talking about write() here. If you select() on a blocking socket, and it told you it's time to write, then you write() more than can be immediately sent, it will block. Further, in the read case, since you mentioned it... First of all I don't think what you wrote is true.. But supposing it is... How do you know the upper bound of what can be read? Just pass in a large buffer? That doesn't make sense. Better to read() in a loop and not be bounded by arbitrary sizes. – asveikau Nov 21 '09 at 03:59

4 Answers4

1

Is there a problem with doing non-blocking I/O? It seems kind of strange to use polling loops if you're using blocking I/O.

When I write programs like this I tend to do the following:

  • Buffer the bytes I want to send to the file descriptor.

  • Only ask for IO_OUT (or the poll() equivalent, POLLOUT) events when said buffer is non-empty.

  • When poll() (or equivalent) has signaled that you're ready to write, issue the write. If you get EAGAIN/EWOULDBLOCK, remove the bytes you successfully wrote from the buffer and wait for the next time you get signaled. If you successfully wrote the entire buffer, then stop asking for POLLOUT so you don't spuriously wake up.

(My guess is that the Win32 bindings are using WSAEventSelect and WaitForMultipleObjects() to simulate poll(), but the result is the same...)

I'm not sure how your desired approach with blocking sockets would work. You are "waking up" constantly because you asked to wake you up when you can write. You only want to specify that when you have data to write... But then, when it wakes you up, the system won't really tell you how much data you can write without blocking, so that's a good reason to use non-blocking I/O.

asveikau
  • 39,039
  • 2
  • 53
  • 68
  • 1
    yes, i guess the difference is that windows returns that the socket is writable, even if it isn't. this isn't my desired behaviour, as writing to it then fails, and then i immediately get teh same IO_OUT event again, failing again... etc. – Matt Joiner Nov 14 '09 at 13:44
  • that hasn't been my experience working with WSAEventSelect(). maybe there is something else you've omitted? maybe more code would shed more light? – asveikau Nov 14 '09 at 22:31
  • I've updated my question to point out an apparent misunderstanding around EAGAIN/EWOULDBLOCK. – Matt Joiner Nov 15 '09 at 08:36
  • Your answer is inaccurate regarding justification for using non-blocking I/O. – Matt Joiner Nov 20 '09 at 23:25
  • 1
    It's not inaccruate. It's a bad idea to use blocking I/O in a poll loop where multiple sockets are served by the same thread. The write notifaction tells you a write is possible, but it does not tell you excactly how many bytes can be written. This is what EAGAIN is for. If you issue a blocking write() where the number of bytes exceeds what you are actually able to write, you will hold up other sockets in your polling loop from being serviced. Try it. But you're right, don't listen to me, I only know what I'm talking about. :P – asveikau Nov 21 '09 at 00:32
  • By the way, from a Google search, I think your issue is actually caused by a limitation of how the GLIB folks ported their polling stuff to Windows. I saw something saying they advocate doing your writes in response to a read event. Or something like this. – asveikau Nov 21 '09 at 00:34
  • Yes, limitations in the Win32, or shortcomings, are the reasons for Glib not working the same way on Windows. – Matt Joiner Feb 07 '10 at 17:37
1

GIO contains GSocket, a "lowlevel network socket object" since 2.22. However this is yet to be ported to pygobject on Windows.

Matt Joiner
  • 112,946
  • 110
  • 377
  • 526
0

I'm not sure if this helps (I'm not proficient with the poll function or the MFC sockets and don't know the polling is a requirement of your program structure), so take this with a grain of salt:

But to avoid a blocking or EAGAIN on write, we use select, i.e. add the socket to the write set that is passed to select, and if select() comes back with rc=0 the socket will accept writes right away ...

The write loop we use in our app is (in pseudocode): 

set_nonblocking.
count= 0.
do {
   FDSET writefds;
   add skt to writefds.
   call select with writefds and a reaonsable timeout.
   if (select fails with timeout) {
       die with some error;
   } 

   howmany= send(skt, buf+count, total-count).
   if (howmany>0) {
       count+= howmany.
   }
} while (howmany>0 && count<total);
Nicholaz
  • 1,419
  • 9
  • 11
  • @Nicholaz, this is what I'm trying to achieve. The `io_add_watch()` call should be the equivalent of a `select()` as you've described. In your example `howmany` is returning effectively returning `-1` for me. – Matt Joiner Nov 19 '09 at 22:48
  • 1
    Again. You cannot avoid properly handling EAGAIN with poll() nor select() nor WaitForMultipleObjects(). Even if poll() or select() says, there is some room in socket's output buffer, it doesn't say how much room. Which means you don't know how much you can write. That's why you usually don't use blocking IO with poll() or select() - the whole point is to avoid blocking. And with non-blocking IO you get EAGAIN when this internal socket buffer fills up. This is not an error. It's a correct usage. – Tomek Szpakowicz Nov 20 '09 at 08:08
  • @tomekszpakowicz: to quote `select()` from the manpage, (`poll()` and friends are very similar but more platform-specific): "A file descriptor is considered ready if it is possible to perform the corresponding I/O operation (e.g., read(2)) without blocking.". Please RTFM before you declare correct usage. – Matt Joiner Nov 20 '09 at 10:53
  • @Anacrolix: Sorry. It's been a long time since I last programmed this staff so I might be talking s**t. Good luck. – Tomek Szpakowicz Nov 20 '09 at 20:02
  • @Anacrolix -- Sorry, Tom is right. poll() and select() tell you what you can do without blocking, but they do not tell you how many bytes. Your citation of the manpage does not contradict this. – asveikau Nov 21 '09 at 00:20
  • poll() is a replacement for the older select(). poll() offers higher granularity on the types of events to wait for, and it is not limited to FD_SETSIZE file descriptors, but otherwise there is not much difference between them. on unix, GLIB wraps poll() for its event loops. hence we _are_ talking about poll() semantics. the Windows equivalent is WaitForMultipleObjects(). to use that with sockets you need WSAEventSelect() which offers poll()-like granularity on events that can be passed into WaitForMultipleObjects(). – asveikau Nov 21 '09 at 00:50
  • Argh! If select returns a file descriptor as being writable, it means you can write _at least one byte_. It _will not block_ regardless of how much you try to write, instead it will write as much as it can, and return how many bytes were written. This is always how write()/send() work, even without select being used. – Matt Joiner Feb 07 '10 at 17:39
0

You could use Twisted, which includes support for GTK (even on Windows) and will handle all the various error conditions that non-blocking sockets on Windows like to raise.

Glyph
  • 31,152
  • 11
  • 87
  • 129