ruby non-blocking line read

Question

I'm trying to read a line from an io in a non-blocking way.

Unfortunately readline blocks. I think I can solve this with read_nonblock with an additional buffer where I store partial result, check if there are multiple lines in the buffer, etc.. but it seems a bit complicated for a simple task like this. Is there a better way to do this?

Note: I'm using event demultiplexing (select) and I'm quite happy with it, I don't want to create threads, use EventMachine, etc...

score 5 · Accepted Answer · edited Jan 02 '14 at 14:45

5

I think the read_nonblock solution is probably the way to go. Simple, not maximally efficient monkey-patch version:

class IO
  def readline_nonblock
    rlnb_buffer = ""
    while ch = self.read_nonblock(1) 
      rlnb_buffer << ch
      if ch == "\n" then
        result = rlnb_buffer
        return result
      end
    end     
  end
end

That throws an exception if there's no data ready, just like read_nonblock, so you have to rescue that to just get a nil back, etc.

edited Jan 02 '14 at 14:45

Edward Anderson

13,591
4
52
48

answered Mar 28 '12 at 03:08

Mark Reed

91,912
16
138
175

Yeah, that was my original idea, and that's what I finally did, but I read as much as I can (not just a single char), I think that's better for performance. Thanks for the monke-patch tip ;) – Karoly Horvath Sep 21 '12 at 15:39

Edward Anderson · Answer 2 · 2014-10-06T17:30:57.430

3

This implementation improves on Mark Reed's answer by not discarding data read that does not end in a newline:

class IO
  def readline_nonblock
    buffer = ""
    buffer << read_nonblock(1) while buffer[-1] != "\n"

    buffer
  rescue IO::WaitReadable => blocking
    raise blocking if buffer.empty?

    buffer
  end
end

edited Oct 06 '14 at 17:30

answered Jan 02 '14 at 13:08

Edward Anderson

13,591
4
52
48

1

"no guarantee" - that's kinda scary. don't rely on undocumenteded 'features'. "I tested this" - thorougly? with different input methods? It may only worked because your input was line buffered... – Karoly Horvath Jan 02 '14 at 17:26
I wasn't brave enough to rely on it either, which is why I included the second part. :-) – Edward Anderson Jan 02 '14 at 21:09
I verified that `read_nonblock(4096)` does not return a single line on ruby 1.9.3, 2.0.0, and 2.1.3. Better take this suggestion out. Easy enough to verify with `ruby19 -e 'x = IO.popen("cat /etc/fstab"); sleep 0.2; p x.read_nonblock(4096)'`. – chutz Oct 06 '14 at 13:47
Thank you. Editing out that "no guarantee" suggestion. – Edward Anderson Oct 06 '14 at 17:30
2

So what about encoding? If the buffered input ends in the middle of a multi-byte character sequence, and then Ruby transcodes during reading, and then we concatenate another transcoded block that starts in the middle of a multi-byte character sequence, what happens? Does Ruby IO prevent that by internally buffering the partial last character in the external encoding and then prepend it to the next chunk? I can't find anything in the Ruby docs that talks about that. – Steve Jorgensen Apr 30 '16 at 01:58
Tried an experment. Given a foo.txt file containing "123456789Ž12345678", … `2.2.2 :001 > io = File.open('foo.txt','r') // => # // 2.2.2 :002 > a=io.readpartial(10) // => "123456789\xC5" // 2.2.2 :003 > b=io.readpartial(10) // => "\xBD12345678\n" // 2.2.2 :004 > a+b // => "123456789\xC5\xBD12345678\n"` – Steve Jorgensen Apr 30 '16 at 02:20

ruby non-blocking line read

2 Answers2