VK_KHR_sampler_ycbcr_conversion explained: hardware YUV to RGB in the sampler

Video frames come as YUV (NV12, I420, P010). The straightforward approach is in-shader conversion — sample Y/U/V planes separately, do the BT.709 matrix manually. That’s fine, but Vulkan has a dedicated path: VK_KHR_sampler_ycbcr_conversion makes the sampler itself do the YUV→RGB conversion — one texture() call returns RGB, using the format’s multi-plane layout and the broadcast standards (BT.601/709/ 2020, limited/full range) as sampler state.

What it does for you

compare.txt

manual (grid shader):
    sample Y plane, sample UV plane → vec2, do matrix math, scale
    — full control, but every frame is shader work

ycbcr sampler:
    texture(ycbcrSampler, uv) → vec4 RGB — sampler hardware does the
    matrix; the "image" is a multi-planar VkImage the sampler reads
    as one

Setting it up

ycbcr.cpp

// 1. the conversion object — the matrix lives here:
vk::SamplerYcbcrConversionCreateInfo yci{};
yci.format  = vk::Format::eG8B8R82Plane420Unorm;   // NV12 as a format
yci.ycbcrModel  = vk::SamplerYcbcrModelConversion::eYcbcr709;
yci.ycbcrRange  = vk::SamplerYcbcrRange::eItuNarrow;  // limited range
yci.components  = {};   // identity swizzle
yci.xChromaOffset = vk::ChromaLocation::eMidpoint;
yci.yChromaOffset = vk::ChromaLocation::eMidpoint;
auto conv = dev.createSamplerYcbcrConversion(yci);

// 2. a sampler bound to the conversion:
vk::SamplerYcbcrConversionInfo convInfo{conv};
vk::SamplerCreateInfo sci{};
sci.pNext = &convInfo;
auto samp = dev.createSampler(sci);

// 3. a multi-plane VkImage with the ycbcr format, bound to the DMA-Buf
//    or uploaded planes — the *image* holds the planes, the *sampler*
//    does the math

frag.frag

// the shader just samples:
layout(set=0,binding=0) uniform sampler2D tex;   // ycbcr sampler
vec4 rgb = texture(tex, uv);                     // returns RGBA

Where each path fits

choose.txt

ycbcr sampler:   standard video formats (BT.601/709/2020, limited/
                 full, common chroma sitings) — the driver knows the
                 spec, uses native HW blocks when they exist
manual shader:   non-standard formats (P010→RGB with custom scaling),
                 per-frame color-matrix changes, filters that need
                 per-plane control (rgb_gain/hue/sat-style filters),
                 or when the format isn't in the ycbcr format list

Practical rules

Format must be a _2Plane/_3Plane ycbcr format (eG8B8R82Plane420Unorm, eG10X6B10X6R10X62Plane420Unorm3Pack16, …) — packed YUYV formats need eG8B8G8R8422Unorm style instead.
ycbcrRange is limited-vs-full — the classic “washed-out video” bug is eItuNarrow vs eItuFull mismatch with the stream’s color_range.
xChromaOffset/yChromaOffset (eMidpoint/eCositedEven) must match the stream’s chroma siting — MPEG-2 is co-sited, H.264 is midpoint; wrong siting = half-pixel chroma shift.
The conversion object is immutable — bake per-stream parameters (matrix, range, siting) into it; a different stream needs a different VkSamplerYcbcrConversion.
Keep the manual shader path anyway — the ycbcr sampler covers the common cases; a camera app’s cutout/hue/saturation/rgb_gain filters need per-pixel control the sampler can’t give.