0

The aim is interact with an OpenEthereum server using json-rpc.

The problem is once connected, I need to react only when receving data as the aim is to subscribe to an event so I need the recv() function to be blocking. But in that case, if I ask to read more in the buffer than what the server sent the request will be blocking.

The OpenEthereum server is separating it s requests with a linefeed \n character but I don t know how this can help.
I know about simply waiting recv() to timeout. But I using C++ and ipc for having a better latency than my competitors on arbitrage. This also means I need to have the fewest number of context switches as possible.

How to effciently read a message whoes length cannot be determined in advance?

user2284570
  • 2,891
  • 3
  • 26
  • 74
  • What is `receive`? Do you by any chance mean `recv` or `recvmsg`? – Brian Bi Dec 08 '20 at 18:33
  • 1
    If I were you I would use bosst asio, specificaly with the async function. As they are really performant. If are determined use unix sockets, I encourage you to read "Beej's Guide to Network Programming" https://beej.us/guide/bgnet/html/. –  Dec 08 '20 at 18:45
  • You know you can set your sockets to non-blocking? Because if you didn't know that, then firstly you should, and secondly I wouldn't bet _too_ much money on you having the lowest latency on the market. – Useless Dec 08 '20 at 18:52
  • @V0_1D this is not really network programming. As explained, it s json rpc over ipc not over tcp. This is why I failed to find an example and no information if the boost unix domain functions support getting just what was written on the socket so far (with nothing more or nothing less). Are there really no modern c++ for handling such problem on streams? – user2284570 Dec 08 '20 at 18:54
  • @Useless that s why I wrote `I need to react only when receving data as the aim is to subscribe to an event so I need the recv() function to be blocking` in the question. – user2284570 Dec 08 '20 at 18:56
  • @Brian ok I edited the question. Sorry, I was meaning `recv()` – user2284570 Dec 08 '20 at 19:02
  • @Useless based on blockchain data, there s a single competitor for this dapps so I m likely to succeed. – user2284570 Dec 08 '20 at 19:04
  • "ipc" is not a transport type, it just means any inter-process communication. It looks like this is actually an AF_UNIX socket, and non-blocking will work fine - as will using `select` or `poll` to wait until something arrives. – Useless Dec 08 '20 at 19:07
  • @Useless how to use `select` and `poll` for waiting something arrives? Are websocket libaries doing the same thing for returning an array of the right size matching just what was sent? – user2284570 Dec 08 '20 at 19:25
  • Can you clarify exactly what you mean by IPC? You've stated that this is a unix domain socket but is it `SOCK_STREAM` or `SOCK_DGRAM` (with apologies if I've missed something)? – G.M. Dec 08 '20 at 20:09
  • @G.M. absolutely no idea. This is something which have to be seen inside the documentation or the code. In Ethereum, it s simply reffered as doing json rpc over ipc (it s not related to OpenEthereum as a client over ipc may work unmodified using Geth or any other full node Ethereum implementation). I just ran `netstat -lp` and sow that a process named OpenEthereum was listening on a file under my home directory and this is the only reason I m thinking it s using unix domain as things like python or v8 js hides this logic in their use library dependencies (which is why I found nothing). – user2284570 Dec 08 '20 at 20:22
  • @G.M. that s those times I wish I could use more than 5 tags on Stack Overflow questions in order to use [tag:ethereum]. – user2284570 Dec 08 '20 at 20:24

1 Answers1

0

Is there a function for determining how many bytes are left to read on a unix domain socket?

No - just keep doing non-blocking reads until one returns EAGAIN or EWOULDBLOCK.

There may be a platform-specific ioctl or fcntl - but you haven't named a platform, and it's neither portable nor necessary.

How to effciently read a message whoes length cannot be determined in advance?

Just do a non-blocking read into a buffer large enough to contain the largest message you might receive.

I need to react only when receving data as the aim is to subscribe to an event so I need the recv() function to be blocking

You're confusing two things.

  1. How to be notified when the socket becomes readable:

    by using select or poll to wait until the socket is readable. Just read their manuals, that's their most common use case.

  2. How to read everything available to read without blocking indefinitely:

    by doing non-blocking reads until EWOULDBLOCK or EAGAIN is returned.

There is logically a third step, for stream-based protocols like this, which is correctly managing buffers in case of partial messages. Oh, and actually parsing the messages, but I assume you have a JSON library already.

This is entirely normal, basic UNIX I/O design. It is not an exotic optimization.

Useless
  • 64,155
  • 6
  • 88
  • 132
  • I added Linux to the tags before you posted this answer. Concerning portability, the code won t he disclosed to competitors. – user2284570 Dec 08 '20 at 20:00
  • Ah, so you did. In that case I _think_ the first answer is just "no", but I haven't done a deep dive to check. And anyway the more important takeaway is that it doesn't matter and you don't need one. – Useless Dec 08 '20 at 20:01
  • Performance wise, I m seeing the fastest even for a single connection is `kqueue()` but I m failing to find an example for Unix domain which as far I understand don t use file descriptors for sockets. – user2284570 Dec 08 '20 at 20:06
  • You IPC transport has a pathname, which means it is almost certainly a UNIX-domain socket, specifically a stream-oriented socket with an AF_UNIX address. It works for most purposes identically to a TCP socket on the loopback interface. All sockets use file descriptors, including this one. – Useless Dec 08 '20 at 20:09
  • what about the `SIOCINQ` ioctl which can be used to return the amount of unread data? – user2284570 Dec 09 '20 at 09:45
  • It's still not very helpful. If you do the ioctl and get zero, the buffer could be filled immediately afterwards. So what use was it? – Useless Dec 09 '20 at 10:17
  • I was thinking about waiting something arrives prior to it of course. – user2284570 Dec 09 '20 at 10:19
  • 1
    Right, but calling ioctl in a busy loop is much more expensive than just using select or poll, which are actually intended for this purpose. Why are you so keen on finding a different way to wait for a socket to become readable? – Useless Dec 09 '20 at 10:24
  • I didn t said I would use a busy loop for that. – user2284570 Dec 09 '20 at 10:29