0

My program on linux got a drastic speed increase when I wrote fcntl(fd, F_SETPIPE_SZ, size). fd is a pipe to a child process I created with fork+execv. I raised the pipe from 64K to 1MB which seems to be linux max without root permission.

I wrote a test to see how big it is on mac, it's also 64K, but I can't seem to figure out how to increase the pipe size. Does anyone know? I'm using an M2 and ventura

Zac Anger
  • 6,983
  • 2
  • 15
  • 42
Stan
  • 161
  • 8
  • Take a look at https://unix.stackexchange.com/questions/11946/how-big-is-the-pipe-buffer , I think this is the only aswer you might have – Arthur Bulakaiev Dec 31 '22 at 17:43

1 Answers1

1

As far as I know macOS doesn't have a similar option available. However if your end goal is to speed up your program, it might be worth taking a closer look at why increasing the pipe capacity on Linux improves performance, as it seems to indicate some underlying problem elsewhere in the program.

An example off the top of my head: if you are sending large chunks of data (>64K), it may be that the reading part of the code doesn't correctly handle truncated data and hangs for some amount of time when a partial chunk is read. With a small buffer this would happen more often, so increasing the buffer size would improve performance, but wouldn't actually fix the root problem.

moriarty
  • 41
  • 5
  • Another example: If the writing part of the code is in a loop and sleeping for a short time whenever it gets a buffer full error, then with a low buffer size that easily becomes full the program might spend a non-negligible amount of time sleeping. A large buffer would rarely become full and thus run "faster". – moriarty Jan 06 '23 at 13:14
  • I think the process I launch takes 50ms to launch and warm up and my process writes to stdout once it has 4K of data. It writes 64K really fast and with a 2MB buffer it almost never blocks. It doesn't sound like much but its about 50-100ms of time saved but the total runtime is 1second. 900 after I made the change on linux – Stan Jan 08 '23 at 19:35
  • Sorry I wasn't around the last few days. it looks like my bounty expired and the points vanished :( I didn't know anyone replied and that it wouldn't auto award anyone – Stan Jan 08 '23 at 19:37