I'm trying to stream audio from a QAudioInput to a QAudioOutput (Qt 5.15) but with a few seconds of artificial delay. In an effort to keep the code simple I implemented a delay filter based on a QIODevice, which sits between the input and output. The audio i/o is initialized like so:
QAudioFormat audioformat = ...;
QAudioInput *audioin = new QAudioInput(..., audioformat, this);
QAudioOutput *audioout = new QAudioOutput(..., audioformat, this);
DelayFilter *delay = new DelayFilter(this);
delay->setDelay(3.0, audioformat);
audioout->start(delay); // output reads from 'delay', writes to speakers
audioin->start(delay); // input reads from line input, writes to 'delay'
The delay filter is:
class DelayFilter : public QIODevice {
Q_OBJECT
public:
explicit DelayFilter (QObject *parent = nullptr);
void setDelay (int bytes);
void setDelay (double seconds, const QAudioFormat &format);
int delay () const { return delay_; }
protected:
qint64 readData (char *data, qint64 maxlen) override;
qint64 writeData (const char *data, qint64 len) override;
private:
int delay_; // delay length in bytes
QByteArray buffer_; // buffered data for delaying
int leadin_; // >0 = need to increase output delay, <0 = need to decrease
// debugging:
qint64 totalread_, totalwritten_;
};
And implemented like this:
DelayFilter::DelayFilter (QObject *parent)
: QIODevice(parent), delay_(0), leadin_(0), totalread_(0), totalwritten_(0)
{
open(QIODevice::ReadWrite);
}
void DelayFilter::setDelay (double seconds, const QAudioFormat &format) {
setDelay(format.bytesForFrames(qRound(seconds * format.sampleRate())));
}
void DelayFilter::setDelay (int bytes) {
bytes = std::max(0, bytes);
leadin_ += (bytes - delay_);
delay_ = bytes;
}
qint64 DelayFilter::writeData (const char *data, qint64 len) {
qint64 written = -1;
if (len >= 0) {
try {
buffer_.append(data, len);
written = len;
} catch (std::bad_alloc) {
}
}
if (written > 0) totalwritten_ += written;
//qDebug() << "wrote " << written << leadin_ << buffer_.size();
return written;
}
qint64 DelayFilter::readData (char *dest, qint64 maxlen) {
//qDebug() << "reading" << maxlen << leadin_ << buffer_.size();
qint64 w, bufpos;
for (w = 0; leadin_ > 0 && w < maxlen; --leadin_, ++w)
dest[w] = 0;
for (bufpos = 0; bufpos < buffer_.size() && leadin_ < 0; ++bufpos, ++leadin_)
;
// todo if needed: if upper limit is ok on buffered data, use a fixed size ring instead
if (leadin_ == 0) {
const char *bufdata = buffer_.constData();
for ( ; bufpos < buffer_.size() && w < maxlen; ++bufpos, ++w)
dest[w] = bufdata[bufpos];
buffer_ = buffer_.mid(bufpos);
}
totalread_ += w;
qDebug() << "read " << w << leadin_ << buffer_.size()
<< bufpos << totalwritten_ << totalread_ << (totalread_ - totalwritten_);
return w;
}
Where the fundamental idea is:
- If I want to delay 3 seconds, I write out 3 seconds of silence and then start piping the data as usual.
And delay changes are handled like this (for completeness, although it's not relevant to this question because I'm seeing issues without changing delays):
- If I decrease that 3 second delay to 2 seconds then I have to skip 1 second worth of input to catch up.
- If I increase that 3 second delay to 4 seconds then I have to write 1 second of silence to fall behind.
That is all implemented via the leadin_ counter, which contains the number of frames I have to delay (> 0) or skip (< 0) to get to the desired delay. In my example case, it's set > 0 when the 3 second delay is configured, and that will provide 3 seconds of silence to the QAudioOutput before it starts passing along the buffered data.
The problem is that the delay is there when the app starts, but over the course of a few seconds, the delay completely disappears and there is no delay any more. I can hear that it's skipping samples here and there to catch up, with an occasional light click or pop in the audio output.
The debug printouts show some things are working:
- Seemingly matched read / write timings
- Smooth decrease in
leadin_to 0 over first 3 seconds - Smooth increase in total bytes read and written
- Constant bytes_read - bytes_written value equal to the delay time after initial ramp up
But they also show some stuff isn't:
- The buffer size is filled up by the
QAudioInputand initially increases, but then begins decreasing (onceleadin_is exhausted) and stays stable at a low value. But I expect the buffer size to grow and then stay constant, equal to the delay time. The decrease means reads are happening faster than writes.
I can't make any sense of it. I added some debugging code to watch for state changes in the input / output to see if they were popping into Idle state (the output will do this to avoid buffer underruns) but they're not, they're just happily handling data with no apparent hiccups.
I expected this to work because both the input and output are using the same sample rate, and so once I initially get 3 seconds behind (or whatever delay time) I expected it to stay that way forever. I can't understand why, given that the input and output are configured at the same sample rate, the output is skipping samples and eating up the delay, and then playing smoothly again.
Am I missing some important override in my QIODevice implementation? Or is there some weird thing that Qt Multimedia does with audio buffering that is breaking this? Or am I just doing something fundamentally wrong here? Since this QIODevice-based delay is all very passive, I don't think I'm doing anything to drive the timing forward faster than it should be going, am I?
I hope this is clear; my definition of "read" and "write" kind of flip/flops above depending on context but I did my best.
Initial debug output (2nd number is leadin_, 3rd number is amount of buffered data, last number is read-written):
read 19200 268800 0 0 0 19200 19200
read 16384 252416 11520 0 11520 35584 24064
read 16384 236032 19200 0 19200 51968 32768
read 16384 219648 26880 0 26880 68352 41472
read 16384 203264 34560 0 34560 84736 50176
read 16384 186880 46080 0 46080 101120 55040
read 16384 170496 53760 0 53760 117504 63744
read 16384 154112 61440 0 61440 133888 72448
read 16384 137728 69120 0 69120 150272 81152
read 16384 121344 80640 0 80640 166656 86016
read 16384 104960 88320 0 88320 183040 94720
read 16384 88576 96000 0 96000 199424 103424
read 16384 72192 103680 0 103680 215808 112128
read 16384 55808 115200 0 115200 232192 116992
read 16384 39424 122880 0 122880 248576 125696
read 16384 23040 130560 0 130560 264960 134400
read 16384 6656 138240 0 138240 281344 143104
read 16384 0 140032 9728 149760 297728 147968
read 16384 0 131328 16384 157440 314112 156672
read 16384 0 122624 16384 165120 330496 165376
read 16384 0 113920 16384 172800 346880 174080
read 16384 0 109056 16384 184320 363264 178944
read 16384 0 100352 16384 192000 379648 187648
read 16384 0 91648 16384 199680 396032 196352
read 16384 0 82944 16384 207360 412416 205056
read 16384 0 78080 16384 218880 428800 209920
read 16384 0 69376 16384 226560 445184 218624
read 16384 0 60672 16384 234240 461568 227328
read 16384 0 51968 16384 241920 477952 236032
read 16384 0 47104 16384 253440 494336 240896
read 16384 0 38400 16384 261120 510720 249600
read 16384 0 29696 16384 268800 527104 258304
read 16384 0 20992 16384 276480 543488 267008
read 16384 0 16128 16384 288000 559872 271872
read 16384 0 7424 16384 295680 576256 280576
read 15104 0 0 15104 303360 591360 288000
read 7680 0 0 7680 311040 599040 288000
read 3840 0 0 3840 314880 602880 288000
read 3840 0 0 3840 318720 606720 288000
read 3840 0 0 3840 322560 610560 288000
read 3840 0 0 3840 326400 614400 288000
read 3840 0 0 3840 330240 618240 288000
read 3840 0 0 3840 334080 622080 288000
read 3840 0 0 3840 337920 625920 288000
