2

Context :

Linux 64bits.

I am very well versed in shared memories, different IPCs mechanisms.

Currently, I am seeking something that could be used in userspace to notify change between two polled processes, to save cpu cycles.

I already know about futexes, eventfd, slow signals and even char devices.

But I am concerned about the latency involved.

Question :

I would like a very lightweight way to notify between two processes in userspace, without the context switch if possible, and a very low latency ( µs order of magnitude max or about).

No major constraints : no race condition prevention, no locking needed, but no busy waiting please (I manage the rushes and the slow pace periods already, and busy waiting would seriously deceive any optimizations on the long run).

If mmap could notify the changes without a msync()... I would not be asking.

Do you have an idea in your toolbox ?

Thanks

idea :

Just after asking, may come an idea : putting a file in a tmpfs and poll it... i expect the latency to be quite low, but I am not very sure..

Community
  • 1
  • 1
Larry
  • 1,735
  • 1
  • 18
  • 46
  • wouldn't polling be the opposite of what you want (ie. busy waiting)? Wouldn't you be less effective than `select` on that? Isn't that analogous to getting `shmget`, and the stuff you already mention, considering the man pages for the shm functionality explicitely tell you that tmpfs is a shmfs thing? – Marcus Müller Feb 25 '15 at 21:35
  • 2
    I'm missing it. Surely any mechanism for *inter-process* communication involves a context switch. A message dispatched by (a context associated with) one process must be received in (a context associated with) a different process to effect IPC at all. – John Bollinger Feb 25 '15 at 22:21
  • @JohnBollinger: I think OP is reffering to mode switching: userland-kernel-userland, which he wants to avoid. – Marcus Müller Feb 25 '15 at 22:26
  • @MarcusMüller, I could buy that, but I'm still missing how such a mechanism makes sense. IPC inherently breaks process isolation. If you can do that without ever entering kernel mode then you have a serious problem. – John Bollinger Feb 25 '15 at 22:39
  • I know the kernel _is_ responsible for any ipc since _it_ knows the different addresses in the table. I guess my mind went the wrong way on this one.. It happens just once in a while in my design after all. Shame on me. – Larry Feb 26 '15 at 06:27
  • 1
    Not possible without busy waiting. You cannot tell the scheduler, living in the kernel, to make another process runnable without communicating with the kernel and incurring a context switch. So you're left with finding a mechanism that have the lowest latency - you might get even lower latency if you run your processes under one of the realtime schedulers though. – nos Feb 26 '15 at 12:50
  • @nos : Thanks nos, yeah, I went the wrong path on this one. – Larry Feb 26 '15 at 17:35
  • 1
    > very low latency ( µs order of magnitude max or about) < It took years for microkernel guys and required complete redisgn (take a closer look at L4 or L4-Linux) - see here: [Towards Real Microkernels](http://homes.cs.washington.edu/~bershad/590s/papers/towards-ukernels.pdf). Don't expect that from general purpose OS like Linux. – myaut Feb 28 '15 at 11:17
  • @myaut: you are definitely right. Linux is not my man here. + 1 – Larry Feb 28 '15 at 11:42

2 Answers2

1

The solution is a complete change in the logic; Forget about poll and all this fuss.

Busy polling will be the way to go with no ops assembly operations.

Larry
  • 1,735
  • 1
  • 18
  • 46
0

I finally went the eventfd route;

What I wish cannot be done in the unmodified gnu/linux.

To satisfy Nominal Animal appetite : the context switch matters in the question i asked. Since it cannot be done without it, there are no other ways to do the job the way I would like.

So I will keep using eventfd, and other traditional tools.

Larry
  • 1,735
  • 1
  • 18
  • 46
  • You mean, *you have decided you cannot do so*. Just for giggles, I wrote a named pipe test program pair. On a stock Ubuntu 3.13.0-46-generic x86_64 kernel running on an Intel Core i5-4200U processor (a cheap HP laptop) using single-character messages, this yields a change notify latency of about 3.6µs, median. (Measured a few hundred thousand roundtrips, i.e. notification pairs, of which half were under 7.2 µs). *This is already well within the microsecond order of magnitude you specified*. So, your claim is therefore proven wrong. Or, are you just trolling? – Nominal Animal Feb 26 '15 at 16:38
  • @NominalAnimal: "Trolling" ? Seriously ? Which claim ? – Larry Feb 26 '15 at 17:32
  • Your problem statement was *"I would like a very lightweight way to notify between two processes in userspace, without the context switch if possible, and a very low latency (µs order of magnitude max or about)"*. Even a trivial test code seems to fulfill the latency requirement, and as other commenters to the original question have mentioned, context switches cannot be avoided. Instead, you gave up, with a sweeping generalization stated as if it were a fact: *"What I wish cannot be done in the unmodified gnu/linux"*. That falsehood irks me. What is it, if not trolling? Just lying? – Nominal Animal Feb 27 '15 at 19:35
  • Nope. You are very despising with people trying to set things up, and that is _not_ servicing you. The context is unavoidable. As I mentioned all the things I tried, you should have known that i am aware of the context switch thing. What i want is not possible with the linux kernel, except if i code a kernel module which I don't want because i want the userspace. So please, stop being sarcastic. My requirements cannot be met. And by the way, you should try relaxation. Irritation doesn't get anyone anywhere. – Larry Feb 28 '15 at 10:59
  • You misunderstand. Because you don't want to poll, you will have to have a blocking syscall to wait for a change notification. When a change arrives, the kernel has to wake up the blocking syscall. In other words, your own rules, not kernel design or anything else, make at least one context switch necessary -- regardless of which OS or kernel or kernel module you use; this is not Linux-specific. Context switch thing is irrelevant. What irks me is you claiming that this is impossible, when it is not. Even a trivial named pipe test program on a stock kernel yields < 4us latencies here. – Nominal Animal Feb 28 '15 at 16:54
  • I ended up recreating a new kernel eventfd module anyway + atomics. Will try to turn it into a framework. It will still consume some cpu cycles but it enables to reduce the system call volume over multiple applications. A system call is so expensive. I will make some tests and report. But the more I look into the kernel, the more I find some places that are (relatively) slow and inefficient (linked lists..) due to the generic thing "it has to work for everyone. In a sense, it is the way to go. But it all add up latency. Let's see if we can reach the nano order of magnitude. – Larry Feb 28 '15 at 17:46
  • By the way, bare metal programming is an alternative. But too huge of a project. – Larry Feb 28 '15 at 17:52
  • Using `fcntl()` file locks (unlocking used to notify the event so you can use `F_SETLKW` to block until the event occurs), seems to yield the smallest latencies (just under 1µs between unrelated processes) here. Did you already rule this approach out? (I used two one-byte write locks, flip-flopping in sync; median period was about 1850 ns. It should be correct, but I haven't fully verified it. You'll probably want to use a more complicated locking scheme; it will have higher runtime cost and limit the event rate, but the latency of a single event should not increase.) – Nominal Animal Feb 28 '15 at 19:42
  • Scratch that. I rewrote the test using three locks (so no risk of slippage), and with that, I get 5.960 µs median latencies comparing `CLOCK_REALTIME` measurements on sender and receiver separately (unrelated processes, three-record `fcntl()` locks). The full notification cycle on the event sender was a stable 12.088 µs median. However, I still think your claim *"... cannot be done"* is nonsense; just because you don't know how to do it (in a vanilla kernel), does not mean it is impossible. Hubris, you know? – Nominal Animal Mar 01 '15 at 03:42
  • Thanks God. Citius, Altius, Fortius. You are right that someday someone could tell me "it is possible", but not right now in this context; It seems you don't know either how to make it possible because it _is_ not possible by design. But since you love having the last word, I can, if you will let you talk there while I take my family out. Life, you know. Of course you know, God knows it all. Bye, thanks nonetheless. – Larry Mar 01 '15 at 10:41