How to decode AV1 on the GPU with VK_KHR_video_decode_av1

H.264 and HEVC decode (VK_KHR_video_decode_h264/h265) cover the legacy codecs — AV1 (VK_KHR_video_decode_av1) is the modern one: the codec YouTube/Netflix are shipping, the one whose 8K streams choke software decoders. The Vulkan path is the same pattern as H.264 — a video session, a DPB, vkCmdDecodeVideoKHR — with AV1’s own parameters plumbed through VkVideoDecodeAV1* structs.

The session setup

session.cpp

// Codec-specific profile + capabilities:
vk::VideoDecodeAV1ProfileInfoKHR av1prof{};
av1prof.stdProfile = STD_VIDEO_AV1_PROFILE_MAIN;   // or _HIGH/_PROFESSIONAL
av1prof.filmGrainSupport = VK_FALSE;              // film grain synthesis
                                                  // on GPU? expensive.

vk::VideoProfileInfoKHR prof{};
prof.pNext = &av1prof;
prof.videoCodecOperation =
    vk::VideoCodecOperationFlagBitsKHR::eDecodeAV1;

auto caps = phys.getVideoCapabilitiesKHR(prof);
// caps.maxDpbSlots, caps.maxActiveReferencePictures,
//   caps.maxCodedExtent, …

vk::VideoSessionCreateInfoKHR vs{};
vs.queueFamilyIndex = decQueue;   // queue with eVideoDecodeKHR
vs.pVideoProfile    = &prof;
vs.pictureFormat    = vk::Format::eG8B8R82Plane420Unorm;
vs.referencePictureFormat = vs.pictureFormat;
vs.maxDpbSlots      = caps.maxDpbSlots;
vs.maxActiveReferencePictures = caps.maxActiveReferencePictures;
auto sess = dev.createVideoSessionKHR(vs);

Decoding a frame

decode.cpp

// AV1's frame parameters come from the STD header
// (vulkan_video_codec_av1std.h — StdVideoAV1SequenceHeader, …):
vk::VideoDecodeAV1PictureInfoKHR pic{};
pic.pStdPictureInfo = &stdPic;    // parsed frame header
pic.pNext           = nullptr;
pic.pReferenceNameSlotIndices…  // which DPB slots are refs

vk::VideoDecodeInfoKHR dec{};
dec.pNext = &pic;
dec.srcBuffer          = bitstreamBuf;   // OBU frame data
dec.srcBufferOffset    = off;
dec.srcBufferRange     = len;
dec.dstPictureResource = {imgView, eVideoDecodeDstKHR};
dec.referenceSlotCount = n;
dec.pReferenceSlots    = refs;           // VkVideoReferenceSlotInfoKHR

cmd.beginVideoCodingKHR(begin);
cmd.decodeVideoKHR(dec);                 // THE call
cmd.endVideoCodingKHR({});

What differs from H.264

diffs.txt

segmentation map     — AV1 has per-segment feature flags; the
                       decoder emits it; check
                       caps.needsExtraSourceImage? / segment map
film grain           — AV1's signature feature: synthetic grain applied
                       post-decode. filmGrainSupport=false = decoder
                       leaves grain off; true = GPU does it (extra pass)
tile/strip decode    — AV1's tile model maps differently to the
                       decode command's slice/segment breakdown
frame header parsing — you still parse the OBU header CPU-side and
                       feed StdVideoAV1SequenceHeader/FrameHeader

The pragmatic path — FFmpeg does it

ffmpeg.sh

# FFmpeg ≥7 exposes AV1 via hwcontext_vulkan decode:
ffmpeg -init_hw_device vulkan=vk0 -hwaccel vulkan \
       -hwaccel_output_format vulkan -i in.av1.webm -f null -
#   → fully on GPU, frames land as AV_PIX_FMT_VULKAN VkImages

See the hwcontext_vulkan post for how AVVkFrame hands you the VkImage directly — same plumbing, codec agnostic.

Practical rules

AV1 needs a decode-capable queue — check QueueFlagBitsKHR::eVideoDecodeKHR and vkGetPhysicalDeviceVideoCapabilitiesKHR with eDecodeAV1; support landed late on some drivers.
Film grain (filmGrainSupport) is the expensive part — a software grain pass post-decode is often cheaper than asking the driver.
AV1’s reference model (8 ref frames, show_existing_frame tricks) means maxActiveReferencePictures on AV1 is larger than H.264’s — don’t hardcode 2.
Session parameters (VkVideoDecodeAV1SessionParametersCreateInfoKHR) carry the sequence header — parse OBUs yourself or via FFmpeg’s av1 parser.
Pair with hwcontext_vulkan for actual use — hand-rolling AV1 decode plumbing is days of work; FFmpeg’s hwaccel already solves it.