Vulkan-CUDA interop: sharing buffers and semaphores between CUDA and Vulkan

CUDA has the compute ecosystem; Vulkan has the display. Interop is the bridge: CUDA runs its kernel on a buffer, hands the result to Vulkan as a VkImage for rendering, synchronized by a semaphore both APIs understand. The mechanism is external memory + external semaphores — the same extensions used for DMA-Buf import, pointed at CUDA’s allocation instead of a kernel buffer.

Two directions

CUDA → Vulkan (render CUDA output):
    cudaMalloc → export as opaque fd/handle
    → vkImportMemoryFd / vkGetMemoryFdPropertiesKHR
    → bind to VkImage / VkBuffer — zero-copy
    sync: cudaExternalSemaphore_t ↔ VkSemaphore (timeline or binary)

Vulkan → CUDA (post-process a render):
    vkGetMemoryFdKHR on the VkDeviceMemory → cudaImportExternalMemory
    → cudaExternalMemoryGetMappedBuffer / …GetMappedMipmappedArray

Buffer sharing — the CUDA-export side

// CUDA allocates + exports:
cudaExternalMemoryHandleDesc d{};
d.type = cudaExternalMemoryHandleTypeOpaqueFd;   // or Win32 handle
cudaExternalMemoryGetExportedMemory... // via cuMemExportToShareableHandle

// Vulkan side: import as device memory
vk::ImportMemoryFdInfoKHR imp{};
imp.handleType = vk::ExternalMemoryHandleTypeFlagBits::eOpaqueFd;
imp.fd = cudaFd;
vk::MemoryAllocateInfo mai{};
mai.pNext = &imp;
// bind to a VkBuffer or VkImage as usual — same DMA-Buf mechanics

The same memory is a CUdeviceptr in CUDA and a VkDeviceMemory in Vulkan — one allocation, two views.

Synchronization — the semaphore bridge

// a Vulkan semaphore exported as an fd:
vk::SemaphoreCreateInfo sci{};
vk::ExportSemaphoreCreateInfo esci{};
esci.handleTypes = vk::ExternalSemaphoreHandleTypeFlagBits::eOpaqueFd;
sci.pNext = &esci;
auto sem = dev.createSemaphore(sci);
auto fd = dev.getSemaphoreFdKHR({sem, eOpaqueFd});

// CUDA side: cudaImportExternalSemaphore + cudaSignalExternalSemaphoresAsync
//            / cudaWaitExternalSemaphoresAsync — timeline semaphores map
//            to CUDA's stream-ordered signal/wait

A timeline VkSemaphore maps to CUDA’s 64-bit stream-ordered semaphore — the same wait/signal values on both sides, so a frame counter semaphore works for both “CUDA done computing” and “Vulkan done presenting” without stalling either.

The real-world flow

frame N:
  Vulkan renders to offscreen VkImage (in exported memory)
  ──► signal timeline sem (value N) → CUDA waits on it
  CUDA kernel reads image memory, writes into its buffer
  ──► signal sem (value N+1) → Vulkan waits on it
  Vulkan presents / re-encodes

Neither side blocks the CPU — it’s all device-side fences.

Practical rules

• VK_KHR_external_memory_fd + VK_KHR_external_semaphore_fd + VK_KHR_external_fence_fd (for binary sems) — the full KHR trio; on Windows it’s Win32 variants instead of OpaqueFd.
• cudaExternalMemoryHandleTypeOpaqueFd is the Linux path; the fd is a POSIX fd you dup() into the Vulkan import.
• Timeline VkSemaphore → cudaExternalSemaphoreHandleTypeTimelineSemaphore*; binary → cudaExternalSemaphoreHandleTypeOpaqueFd. Timeline is the right primitive for frame counting.
• The CUDA side needs cuCtxEnablePeerAccess/mempool-compatible allocations — plain cudaMalloc can’t be exported, use cuMemCreate/cuMemExportToShareableHandle.
• One memory allocation = two API objects; destroying the Vulkan side doesn’t free the CUDA allocation — ownership is a contract, both sides must agree who calls vkFreeMemory vs cuMemRelease.