V4L2 to Vulkan zero-copy: importing camera DMA-Bufs as textures

A V4L2 camera doesn’t have to land in CPU memory. With V4L2_MEMORY_DMABUF the camera driver exports each capture buffer as a DMA-Buf — the same fd type VAAPI exports. Import it to Vulkan and the camera sensor feeds the grid shader directly. It’s the capture side of the zero-copy pipeline the decode posts built for VAAPI frames.

The V4L2 side — request DMA-Buf buffers

v4l2.cpp

int fd = open("/dev/video0", O_RDWR);

v4l2_format fmt{};
fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
fmt.fmt.pix_mp.pixelformat = V4L2_PIX_FMT_NV12;  // or V4L2_PIX_FMT_…
fmt.fmt.pix_mp.width  = 1920;
fmt.fmt.pix_mp.height = 1080;
ioctl(fd, VIDIOC_S_FMT, &fmt);

v4l2_requestbuffers req{};
req.count   = 4;
req.type    = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
req.memory  = V4L2_MEMORY_DMABUF;            // ← export as DMA-Buf
ioctl(fd, VIDIOC_REQBUFS, &req);

// per-frame dequeue → the buffer's fd is the DMA-Buf:
v4l2_buffer buf{};
buf.type   = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
buf.memory = V4L2_MEMORY_DMABUF;
buf.length = nPlanes;
buf.m.planes = planes;
ioctl(fd, VIDIOC_DQBUF, &buf);
// buf.m.planes[i].m.fd — the DMA-Buf fds for the frame's planes

The Vulkan side — the same import as decode

import.cpp

// identical to the decoder path — the fd → VkDeviceMemory → VkImage:
vk::ImportMemoryFdInfoKHR imp{};
imp.handleType = vk::ExternalMemoryHandleTypeFlagBits::eDmaBufEXT;
imp.fd         = dup(buf.m.planes[0].m.fd);

// image tiling = eDrmFormatModifierEXT if the camera exposes a
// modifier, else eLinear with explicit stride

The mechanism is exactly the VK_EXT_external_memory_dma_buf + VK_EXT_image_drm_format_modifier pair from the decoder posts — a camera frame is just another DMA-Buf.

The whole pipeline

pipe.txt

/dev/video0 ──► V4L2 DQBUF → DMA-Buf fd
            ──► vkImportMemoryFdKHR → VkImage
            ──► sample in grid shader
            ──► render → swapchain
no CPU pixel path — same as VAAPI decode, but the *source* is the
sensor, not a decoder.

Where it fits vs. VAAPI

compare.txt

V4L2 DMA-Buf:     USB/CSI cameras, capture cards — raw or
                  vendor-compressed sensor data, uncompressed YUV/RGB
VAAPI DMA-Buf:    IP cameras, files — compressed bitstream decoded on
                  the VCN/VCU block, output is YUV planes
both:             same import API, same modifier mechanism, same
                  foreign-queue transfer

A camera wall mixing IP cams (VAAPI decode) and a local USB cam (V4L2) shares the whole Vulkan side — the only difference is where the fd comes from.

Practical rules

V4L2_MEMORY_DMABUF on REQBUFS — not the default MMAP; the camera driver must support exporting (most modern ones do).
V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE for multi-planar — NV12/YUV cameras expose planes via buf.m.planes[]; single-plane formats use _CAPTURE + buf.m.fd.
Reuse the decoder’s import code — the AVDRMFrameDescriptor from FFmpeg and the v4l2_buffer planes are the same data in different wrappers; normalize to {fd, offset, stride, format, modifier}.
The foreign-queue transfer (VK_EXT_queue_family_foreign) applies identically — the V4L2 driver wrote the buffer on its own context.
Return buffers promptly — VIDIOC_QBUF puts the DMA-Buf back for the next frame; holding fds starves the camera.