RTMP adaptive bitrate algorithm

Question

I searched online but there is very little information about this.

I have a live broadcasting app where I send encoded H264 video frames and AAC audio chunks resulting from camera & mic using the Android MediaCodec SDK, over a RTMP stack.

My live streams are 720p and I aim for great quality with 2500Kbps. This obviously requires a very good network connection which means 4G if you use a data plan.

Problem is even with the greatest connection there will be low peaks and congestion, so there will be moments where the network cant hold such heavy stream. Because I want to offer high reliability, I want to include automatic adaptive bitrate on my app so that the image quality is dropped in favor or reliability.

The thing is -- how to achieve this automatic adaptation to the network conditions without losing frames? Is it even possible? I've used professional encoding devices like Cerevo and they dont ever lose frames -- however with my app I always get some awful dragging due to p-frames being lost in the network.

This is what I currently have:

private long adaptBitrate(long idleNanos, Frame frame) {
        int bytes = frame.getSize();
        long nowNanos = System.nanoTime();
        if (nowNanos - mLastNanos > 1000L * 1000 * 1000) {
            double idle = (double) idleNanos / (double) (nowNanos - mLastNanos);
            float actualBitrate = newBitrate;

            int size = mBuffer.size();
            String s = "Bitrate: " + actualBitrate / 1000
                    + " kbps In-Flight:" + bytes
                    + " idle: " + idle;
            if (size > MAX_BUF_SIZE && size > mLastSize) {
                Log.i(TAG, "adaptBitrate: Dropping bitrate");
                newBitrate = (int) ((double) actualBitrate * BITRATE_DROP_MULTIPLIER);
                if (newBitrate < MIN_BITRATE) {
                    newBitrate = MIN_BITRATE;
                }
                s += "   late => " + newBitrate;
                mRtmpHandler.requestBitrate(newBitrate);
            } else if (size <= 2 && idle > IDLE_THRESHOLD) {
                mIdleFrames++;
                if(mIdleFrames >= MIN_IDLE_FRAMES){
                    Log.i(TAG, "adaptBitrate: Raising bitrate");
                    newBitrate = (int) ((double) newBitrate * BITRATE_RAISE_MULTIPLIER);
                    if (newBitrate > MAX_BITRATE) {
                        newBitrate = MAX_BITRATE;
                    }
                    s += "   idle => " + newBitrate;
                    mRtmpHandler.requestBitrate(newBitrate);
                    mIdleFrames = 0;
                }
            }
            debugThread(Log.VERBOSE, s);
            mLastNanos = System.nanoTime();
            mLastSize = size;
            idleNanos = 0;
        }
        return idleNanos;
    }

So if my buffer is exceeding a threshold, I lower the bitrate. If my app is spending too much time waiting for a new frame, for a number of consecutive frames, then I raise the bitrate.

No matter how cautious I am with the threshold values, I am always losing important information and my stream breaks until the next keyframe arrives (2 secs). Sometimes it seems that the network can hold a certain bitrate (stable at 1500kbps, for instance) but the image will still have some dragging as though a frame was lost in the way. With good network conditions, everything is smooth.

How do these streaming devices handle these situations? It always looks great with them, no dragging or skipped frames at all...

So do you know whether the device encoder itself actually support mode switching? In other words, what happens when you emulate a switch over a perfect connection? Also, is this a real-time streaming scenario, like a chat, or just typically broadcast? — ThomasRS, Apr 13 '17 at 18:42
real-time broadcast. dunno what you mean by "mode switching", if you mean adapting the bitrate, I do measure the output bitstream and it matches the desired bitrate, so it is effectively changing the bitrate midstream. Also all devices from 4.3 on support this if I am not mistaken. — M Rajoy, Apr 13 '17 at 19:36

score 0 · Answer 1 · answered Feb 02 '18 at 22:58

There is indeed no information online about adaptive bitrate from the broadcaster side, surprisingly. When I had to implement something like this with RTSP and two rtp sockets, I took a similar approach, creating a polling class that would modestly increase the bitrate of the mediacodec when the packet buffer was >$GOOD_PCT free, aggressively halve it when the queue was less than $BAD_PCT free, and do nothing if it was in between. Partially seen here. I'm not sure I have a complete picture of your solution based on the posted code, but you are adjusting the mediacodec bitrate directly, correct? The only time I had corruption is when I requested a sync frame from the mediacodec, so avoid that if it's in your code. Hope this helps.

RTMP adaptive bitrate algorithm

1 Answers1