8

When writing a non-blocking program (handling multiple sockets) which at a certain point needs to open files using open(2), stat(2) files or open directories using opendir(2), how can I ensure that the system calls do not block?

To me it seems that there's no other alternative than using threads or fork(2).

tshepang
  • 12,111
  • 21
  • 91
  • 136
watain
  • 4,838
  • 4
  • 34
  • 35
  • I wouldn't expect it to be possible. In order to do that you'd have to halt all file system activity on the system (to ensure nothing blocks your next call due to a race condition between the time you check and the time you actually call the function), which no sane desktop OS would let you do. – user541686 Jan 08 '13 at 08:18
  • Why does it matter to you that much? In most cases a process is doing much less `open` syscalls than e.g. `read` & `write`, and these 2 common syscalls can be non-blocking! And very often `open` is quite fast.... – Basile Starynkevitch Jan 08 '13 at 08:20
  • Other systems (Windows for one) actually handle this by allowing you to request I/O and then get notified when it is done. Like watain, I've never figured out how to get linux to do it nicely (+1). – DrC Jan 08 '13 at 08:22
  • 1
    And you can also set a timer and get the `open` syscall interrupted (failing with `errno == EINTR`) if timed out... – Basile Starynkevitch Jan 08 '13 at 08:30
  • @BasileStarynkevitch I understand what you mean, in most cases open _is_ quite fast, but still I don't want to run into possible exceptions. Setting a timer to interrupt the system call seems reasonable, I need to think about that, thanks! (although it might add unnecessary complexity). – watain Jan 08 '13 at 08:53
  • @BasileStarynkevitch: For certain cases `open()` can be a severe bottleneck. This includes reading lots of tiny files, or something as simple as an NFS timeout. – Brendan Feb 11 '14 at 22:09
  • @Mehrdad: Most OSs (internally) do all IO asynchronously (because they have to handle many processes opening files at the same time). The problem is the user-space libraries - e.g. standard C library which was designed before anyone cared much about threads, the POSIX asych IO library which fails to handle "asynchronous open", the standard C++ library that's so bad it can't even tell you why open failed, etc. – Brendan Feb 11 '14 at 22:14
  • Isn't asynchronous I/O the thing you want? Although I can't tell how good is it for Linux/OS X. – Dmytro Sirenko Feb 11 '14 at 22:16

3 Answers3

4

As Mel Nicholson replied, for everything file descriptor based you can use select/poll/epoll. For everything else you can have a proxy thread-per-item (or a thread pool) with the small stack that would convert (by means of the kernel scheduler) any synchronous blocking waits to select/poll/epoll-able asynchronous events using eventfd or a unix pipe (where portability is required).

The proxy thread shall block till the operation completes and then write to the eventfd or to the pipe to wake up the select/poll/epoll.

Community
  • 1
  • 1
bobah
  • 18,364
  • 2
  • 37
  • 70
  • Regular files (also descriptor-based) always return ready from poll. And than `read` blocks actually reading the data (`write` generally does not wait for actual write to disk, but it might also block _reading_ the data if you are doing random access) – Jan Hudec Jan 08 '13 at 08:41
  • @Jan Hudec - then eventfd/kicksocket(pipe) is a way to go, otherwise for high performance file IO mmap is usually used, which gives deterministic latencies – bobah Jan 08 '13 at 08:42
  • I don't think it is. If you make each operation asynchronous separately using worker thread, the additional context switching is going to add significant overhead. The threads should be given large chunks of work they can do independently to keep the need for synchronization reasonable. – Jan Hudec Jan 08 '13 at 08:47
  • file IO mmap does not give deterministic latencies, because it still has to go through the disk access layer that does not. But it it fastest. And can only be done with threads as the blocking is in page faults, which are uninterruptible. – Jan Hudec Jan 08 '13 at 08:50
2

Indeed there is no other method.

Actually there is another kind of blocking that can't be dealt with other than by threads and that is page faults. Those may happen in program code, program data, memory allocation or data mapped from files. It's almost impossible to avoid them (actually you can lock some pages to memory, but it's privileged operation and would probably backfire by making the kernel do a poor job of memory management somewhere else). So:

  1. You can't really weed out every last chance of blocking for a particular client, so don't bother with the likes of open and stat. The network will probably add larger delays than these functions anyway.
  2. For optimal performance you should have enough threads so some can be scheduled if the others are blocked on page fault or similar difficult blocking point.

Also if you need to read and process or process and write data during handling a network request, it's faster to access the file using memory-mapping, but that's blocking and can't be made non-blocking. So modern network servers tend to stick with the blocking calls for most stuff and simply have enough threads to keep the CPU busy while other threads are waiting for I/O.

The fact that most modern servers are multi-core is another reason why you need multiple threads anyway.

Jan Hudec
  • 73,652
  • 13
  • 125
  • 172
  • I get your point. Too bad there's no _real_ non-blocking library for file system related operations (it might make things easier), but since it's not really the same as with sockets I can understand why using threads is a better choice. – watain Jan 08 '13 at 09:04
0

You can use the poll( ) command to check any number of sockets for data using a single thread.

See here for linux details, or man poll for the details on your system.

open( ) and stat( ) will block in the thread they are called from in all POSIX compliant systems unless called via an asynchronous tactic (like in a fork)

Mel Nicholson
  • 3,225
  • 14
  • 24
  • So how do you use `poll` to check whether `open`, or `stat` will not need to wait for inode cache to populate? – Jan Hudec Jan 08 '13 at 08:16