VK_EXT_image_drm_format_modifier explained: importing tiled images from other APIs

A GPU-decoded video frame isn’t a linear buffer — it’s a vendor-tiled, possibly compressed image whose layout is described by a DRM format modifier (DRM_FORMAT_MOD_*). When FFmpeg/VAAPI hands you a DMA-Buf, the modifier tells you how the pixels are laid out. VK_EXT_image_drm_format_modifier lets Vulkan express that same layout — create a VkImage that is the imported frame, no copy.

Why modifiers exist

why.txt

linear:   simplest layout, no compression, poor cache behavior
tiled:    vendor-specific swizzle — good cache locality
modifier: a 64-bit tag that says WHICH tiling — e.g.
          DRM_FORMAT_MOD_LINEAR                    (plain)
          I915_FORMAT_MOD_X_TILED
          DRM_FORMAT_MOD_AMD_GFX9_64K_S
          + per-vendor compression variants (DCC, FBC…)

The exporter (VAAPI, V4L2, GBM, another Vulkan driver) tags the DMA-Buf with a modifier; the importer must declare it understands exactly that modifier — a mismatched modifier means the pixels decode wrong, silently.

The import path

import.cpp

// 1. query what modifiers the driver supports for the format:
vk::DrmFormatModifierPropertiesListEXT list{};
auto fmtProps = phys.getFormatProperties2(eR8, &list);
for (auto& p : list.drmFormatModifierProperties)
    // p.drmFormatModifier, p.drmFormatModifierPlaneCount,
    //   p.drmFormatModifierTilingFeatures

// 2. image creation — declare the modifier + plane layouts:
vk::ImageDrmFormatModifierListCreateInfoEXT modList{};
modList.drmFormatModifierCount = 1;
modList.pDrmFormatModifiers    = &mod;

vk::ImageCreateInfo ici{};
ici.tiling = vk::ImageTiling::eDrmFormatModifierEXT;
ici.pNext  = &modList;   // or explicit: VkImageDrmFormatModifierExplicitCreateInfoEXT

// 3. bind the DMA-Buf's fd via vkGetMemoryFdPropertiesKHR +
//    ImportMemoryFdInfoKHR (external_memory_dma_buf, other post)

eDrmFormatModifierEXT replaces eOptimal/eLinear — the driver interprets the modifier’s layout directly.

A real import pipeline

flow.txt

VAAPI decodes MJPEG/H.264 → AVDRMFrameDescriptor
    ├─ layer 0: NV12 plane 0 (Y)   + modifier
    └─ layer 1: NV12 plane 1 (UV)  + same modifier
→ each plane → one VkImage (R8 for Y, R8G8 for UV)
   tiling=eDrmFormatModifierEXT, modifier from the descriptor
→ sampled by the grid shader — zero copies

The imported images are foreign-typed: usage is limited to SAMPLED/TRANSFER_SRC typically, and the foreign queue family (VK_EXT_queue_family_foreign, separate post) hands ownership from the importer to your queue.

Practical rules

Always query VkDrmFormatModifierPropertiesListEXT — the modifier the decoder emits must be in the supported set for that format.
drmFormatModifierPlaneCount tells you how many VkImages a multi-plane format needs — NV12 → 2 planes, I420 → 3, each a separate VkImage sharing the modifier.
Modifier-tiled images only allow eGeneral/ePresentSrcKHR layouts — eOptimal transitions are invalid; transition to eGeneral once.
VK_EXT_image_drm_format_modifier is necessary but not sufficient — you also need VK_EXT_external_memory_dma_buf for the fd import and VK_EXT_queue_family_foreign for the ownership transfer.
A failed modifier match is a fallback signal, not a bug — fall back to eLinear staging + vkCmdCopyBufferToImage when the modifier isn’t in the device’s list.