Low-latency RTSP and HTTP streaming with FFmpeg: AVIOInterruptCB, fflags and low_delay

The target is “camera light blinks, pixel shows up next frame”. FFmpeg’s defaults aren’t tuned for that — it buffers, reorders, and waits. The latency budget is a handful of flags and one callback; missing them costs hundreds of ms.

The flags

opts.cpp

AVDictionary* opts = nullptr;
av_dict_set(&opts, "rtsp_transport", "tcp", 0);   // or udp — lossy
                                                  //   but lower latency
av_dict_set(&opts, "fflags",       "nobuffer", 0); // skip the AVIO
                                                   //   input buffer
av_dict_set(&opts, "flags",        "low_delay", 0);// decoder flag:
                                                   //   no reordering
av_dict_set(&opts, "max_delay",    "0",         0);
av_dict_set(&opts, "stimeout",     "5000000",   0); // RTSP socket
                                                  //   timeout, µs
av_dict_set(&opts, "reconnect",    "1",         0); // http keepalive

int r = avformat_open_input(&fmt, url, nullptr, &opts);

nobuffer is the big one — without it FFmpeg buffers a probe window to guess the format, adding startup latency.

The timeout that actually works — AVIOInterruptCB

interrupt.cpp

// stimeout only covers the socket layer; a stalled TLS handshake or
// a hung TCP stream can still block av_read_frame forever. The
// interrupt callback is the real deadline:
static int interruptCb(void* ctx) {
    auto* d = static_cast<Deadline*>(ctx);
    return d->expired();  // non-zero → FFmpeg aborts the blocked op
}
fmt->interrupt_callback = {interruptCb, &deadline};

deadline.arm(5000ms);
av_read_frame(fmt, pkt);   // returns EAGAIN/EXIT when the cb fires

Works for RTSP and HTTP — stimeout is RTSP-only, timeout is HTTP-only; the interrupt callback covers both.

Decoder-side latency

decode.cpp

avctx->flags |= AV_CODEC_FLAG_LOW_DELAY;    // no frame reordering
avctx->thread_count = 1;                    // frame threading queues
                                            //   frames = latency
avctx->flags2 |= AV_CODEC_FLAG2_FAST;       // skip some checks
// for H.264: request IDR-aligned output, drop b-frames if the
// encoder allows — b-frame reordering is per-codec latency

Frame threading (thread_count > 1) is the stealth latency source — the decoder queues thread_count frames internally before yielding the first.

The end-to-end budget

budget.txt

network read      stimeout/interruptCb → stall detected in ~100ms
demux             fflags=nobuffer       → no probe-window delay
decode            low_delay, threads=1  → no reorder queue, no frame
                                          threading backlog
upload/present    latest-frame mailbox  → stale frames dropped,
                                          newest shown immediately

Practical rules

rtsp_transport=tcp for reliability, udp for lowest latency — UDP drops packets instead of retransmitting (right choice for live walls; a torn frame beats a 500ms retransmit stall).
stimeout is connect+read timeout on the RTSP socket — the interrupt callback covers the HTTP path and deadlocks it can’t.
decode_threads = 1 + nobuffer + low_delay is the baseline for a live camera wall; each piece trades a bit of throughput for latency.
Test with a real camera — testsrc never stalls; the interrupt path only exercises on a flaky network or a real camera’s keepalive pattern.
Watch av_read_frame return codes — AVERROR(EAGAIN) from the interrupt is a recoverable stall, AVERROR_EOF is a real stream end.