0

I am investigating a bit UDS for logging from an app and then using another process to send the logs to an external server. Overall it seems to work fine but when I was testing it I discovered that if I send some logs in a for loop, when stream is read from the socket it contains more "messages".

you can find the code for receving logs here https://github.com/MattBlack85/alf (after installing you can run it with alf /tmp/alf.sock http://127.0.0.1:8080)

you can find a small example to send logs here https://gist.github.com/MattBlack85/86d620a306f16416a7f96a1a035984dc

you can find a small webserver to let alf send over the logs here https://gist.github.com/MattBlack85/0638ef87eb077eb46879d6c90a30cf7a

if the for loop has no sleep, the result is something like that

[2018-12-18 13:12:39,798] [DEBUG] alf.worker - MSG from queue: b'{"time":"2018-12-18 13:12:39,797","name":"test","levelname":"DEBUG","message":"test 0","pathname":"logalf.py"}{"time":"2018-12-18 13:12:39,798","name":"test","levelname":"DEBUG","message":"test 1","pathname":"logalf.py"}{"time":"2018-12-18 13:12:39,798","name":"test","levelname":"DEBUG","message":"test 2","pathname":"logalf.py"}{"time":"2018-12-18 13:12:39,798","name":"test","levelname":"DEBUG","message":"test 3","pathname":"logalf.py"}{"time":"2018-12-18 13:12:39,798","name":"test","levelname":"DEBUG","message":"test 4","pathname":"logalf.py"}{"time":"2018-12-18 13:12:39,798","name":"test","levelname":"DEBUG","message":"test 5","pathname":"logalf.py"}'

while if I put a small break of 1ms all the messages are received one by one. I tried to close all heavy process on my OS and leave the CPU free but it didn't work. This is not a big issue as I can add a terminator when formatting the JSON log and the split what is read from the socket and put every item of the resulting list into the queue, but why I am seeing this at all?

Mattia Procopio
  • 631
  • 8
  • 16
  • 1
    You need to send something to tell the message is finished to separate your messages. For example '\n'. You will never be able to know how the network will cut your message – Ôrel Dec 18 '18 at 13:25
  • @Ôrel so how it is possible that a little sleep leaves me with an alwas reproducible case where all messages are received one by one? – Mattia Procopio Dec 18 '18 at 13:30
  • The sleep can trigger a split into different network packet – Ôrel Dec 18 '18 at 13:34
  • shouldn't communication over unix domain sockets happen in RAM entirely? – Mattia Procopio Dec 18 '18 at 14:27
  • 1
    this is another point, you use network stack which is in charge to split the packet. the tube (local or real network) is another point. The way you write on socket cannot be used as message separator, you need deliminator for your message. But why use so low level socket manipulation instead of using an existing protocol ? – Ôrel Dec 18 '18 at 14:51
  • let's say a bit of case study and the scope of that app – Mattia Procopio Dec 19 '18 at 08:58
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/185471/discussion-between-orel-and-mattia-procopio). – Ôrel Dec 19 '18 at 09:00

0 Answers0