FFmpeg AV_PIX_FMT_VAAPI and AV_PIX_FMT_DRM_PRIME: the hwaccel frame formats

When FFmpeg’s hwaccel decodes on the GPU, the returned AVFrame’s data[] isn’t pixel bytes — it’s a handle to a GPU surface. AV_PIX_FMT_VAAPI and AV_PIX_FMT_DRM_PRIME are the two flavors that matter for zero-copy to Vulkan: one is the opaque VA-API surface, the other is a DMA-Buf descriptor you can turn into a VkImage directly.

What each is

fmts.txt

AV_PIX_FMT_VAAPI      — frame->data[0] = VASurfaceID (uint32 cast).
                        Opaque handle into the VADisplay's surface
                        pool; only meaningful to VA-API calls.
AV_PIX_FMT_DRM_PRIME  — frame->data[0] = AVDRMFrameDescriptor*:
                        { nb_objects, objects[fd], nb_layers,
                          layers[planes] } — the DMA-Buf fd + per-
                          plane offset/stride/format. Vulkan-
                        importable.

VAAPI is the native decode format; DRM_PRIME is the exported one — you get the latter from av_hwframe_map or by requesting it directly.

The map step

map.cpp

// vaapiFrame = decoded AVFrame (AV_PIX_FMT_VAAPI):
AVFrame* drm = av_frame_alloc();
av_hwframe_map(drm, vaapiFrame, AV_HWFRAME_MAP_READ);
// drm->format == AV_PIX_FMT_DRM_PRIME
// drm->data[0] → AVDRMFrameDescriptor:
auto* desc = (AVDRMFrameDescriptor*)drm->data[0];
for (int i = 0; i < desc->nb_layers; i++)
    for (int p = 0; p < desc->layers[i].nb_planes; p++) {
        auto& pl = desc->layers[i].planes[p];
        // pl.object_index → desc->objects[oi].fd   (the DMA-Buf)
        // pl.offset, pl.pitch, pl.drm_format / modifier
    }

Each plane’s fd goes to vkImportMemoryFdKHR — the external_memory_dma_buf post covers the Vulkan side.

Getting to DRM_PRIME

paths.cpp

// path A — decode into VAAPI, then map (the usual default):
avctx->pix_fmt = AV_PIX_FMT_VAAPI;
// decode …
av_hwframe_map(drmFrame, vaapiFrame, AV_HWFRAME_MAP_READ);

// path B — hwframe_ctx defaults to DRM_PRIME (skip VAAPI entirely):
//   av_hwdevice_ctx_create(&hwdev, AV_HWDEVICE_TYPE_DRM, devNode, …)
//   frames come out AV_PIX_FMT_DRM_PRIME directly

// fallback — download to CPU (hwaccel without export):
av_hwframe_transfer_data(swFrame, vaapiFrame, 0);
// swFrame->format = AV_PIX_FMT_NV12 etc — real pixel bytes

av_hwframe_map keeps the GPU memory (zero-copy); av_hwframe_transfer_data is the readback — slow path.

Format negotiation

getfmt.cpp

// get_format callback — pick AV_PIX_FMT_VAAPI if offered:
static enum AVPixelFormat getFmt(AVCodecContext*, const enum AVPixelFormat* f) {
    for (; *f != AV_PIX_FMT_NONE; f++)
        if (*f == AV_PIX_FMT_VAAPI) return *f;
    return f[0];  // fallback to SW
}
// FFmpeg calls this once hw params are known; returning VAAPI makes
// the decoder allocate hw frames

Practical rules

VAAPI frames are opaque — you can’t mmap them; av_hwframe_map to DRM_PRIME is the export step, av_hwframe_transfer_data is the CPU download.
AVDRMFrameDescriptor may describe one multi-plane layer or one single-plane layer per plane — handle both layouts in your per-slot texture store.
Keep the AVFrame alive until the VkImage is done — av_frame_free drops the descriptor ref; the DMA-Buf fd’s lifetime follows it.
AV_HWFRAME_MAP_READ vs MAP_WRITE — decode output only needs READ; requesting WRITE when you don’t need it can force a format the driver doesn’t support.
The fds are inherited, not duplicated — dup() before handing to Vulkan, per the import post.