Which audio APIs can support speculative evaluation and correction?

Question

Speculative evaluation (aka speculative execution) is an effective approach to achieving low-latency code at the cost of occasional rework (inefficiency). Speculative evaluation is a common low-level technique for modern computer architectures, but it can be supported at higher level programming models (cf. time warp protocol, temporal logic, reactive programming models).

One place it seems speculative evaluation would be especially useful is real-time computation of audio, e.g. for live coding or games. The idea is simple: we can speculatively fill audio buffers to prevent buffer under-runs, but then occasionally correct those buffers if we must react to any last-instant changes. Such a technique can still glitch. A straggling update might chop a little off the front. But since most of the speculated sounds should still be mostly correct, this is a different - and potentially more graceful - mode for glitches than typical under-runs.

Now, what I'm wondering is which audio APIs or libraries would most effectively to support these last-instant updates to existing buffers. I'm not an expert at sound programming, but most example code I've seen seems to assume commitment to buffers. If I'm committed to a buffer after I load it, then there's no choice but to make a tradeoff between latency and risk of under-runs. Which audio APIs do not require commitment?

Answer 1

Once a buffer containing sound has been passed off to your soundcard it is generally too late to modify it in any way. I seem to remember that Microsoft tried to introduce an audio driver model which allowed you to directly write into the soundcard's memory buffer (WaveRT), but it didn't gain too much traction as USB soundcards didn't support it.

All audio APIs will have a way that you give a buffer to be played, or fill a buffer that you are given. Once you have done that, there is no guarantee that changing the memory at that location will have any effect (it might even cause a crash). With the windows waveOut API, you can queue up multiple buffers, so you could possibly change one before the soundcard had got around to playing it. However, it is not a low latency API, so the one that is playing will likely be around 50ms.

Most applications that need very responsive sound simply work at low latencies. From an audio processing point of view it would often be extremely hard to retrospectively modify part of a buffer that has already been written, because swapping out audio can result in pops and clicks if you don't fade it in and out, and any DSP you were passing the audio through may have state that you can't reproduce.