How to ingest HTTP MJPEG camera streams into Vulkan (multipart/x-mixed-replace)

Post 58. The dumbest camera protocol still alive: multipart/x-mixed-replace — one endless HTTP response, one JPEG per chunk, delimited by a MIME boundary. Every cheap IP cam speaks it. Here’s the whole ingest: raw socket → boundary parser → libavcodec MJPEG decode → Vulkan upload.

example.sh

$ ./app --headless out.ppm http://127.0.0.1:8137/cam
got 320x240 MJPEG frame from http://127.0.0.1:8137/cam
wrote out.ppm

Example: VulkanExamples/p58_httpmjpeg

(check.py serves a synthetic MJPEG stream in-process — no camera needed).

The wire format

example.txt

HTTP/1.0 200 OK
Content-Type: multipart/x-mixed-replace;boundary=frame

--frame
Content-Type: image/jpeg
Content-Length: 3129

<jpeg bytes>
--frame
...

Frames are delimited by -- + the boundary from the response header, each preceded by per-part MIME headers ending in \r\n\r\n. Most cameras also send Content-Length — trust the boundary, not the length; some cams emit wrong lengths.

Parser essentials

main.cpp (boundary scan)

std::string delim = "\r\n--" + boundary;
auto pos  = find(delim);            // this part's trailing delimiter
auto start = pos + delim.size();
auto end  = find(delim, start);     // next part's leading delimiter
std::vector<uint8_t> part(buf.begin()+start, buf.begin()+end);
// strip per-part headers at first \r\n\r\n

Two traps I hit for real:

The first boundary has no leading \r\n. It sits at byte 0 of the body. Searching only for "\r\n--" skips part #1 — harmless for a live stream (you drop one frame), but don’t be surprised.
std::search(part, "\xff\xd8") never matches. The pattern is a const char*: 0xff as signed char is -1, which never equals uint8_t(255). Use a uint8_t pattern:

example.cpp

static const uint8_t soiPat[] = {0xff, 0xd8};
auto soi = std::search(part.begin(), part.end(), soiPat, soiPat + 2);

Decode: MJPEG via libavcodec, no demuxer needed

main.cpp (decode)

const AVCodec* codec = avcodec_find_decoder(AV_CODEC_ID_MJPEG);
auto* dec = avcodec_alloc_context3(codec);
avcodec_open2(dec, codec, nullptr);

// per frame: wrap the jpeg bytes as a packet
av_new_packet(pkt, jpg.size());
memcpy(pkt->data, jpg.data(), jpg.size());
avcodec_send_packet(dec, pkt);
avcodec_receive_frame(dec, frame);   // -> yuvj420p / yuvj422p usually

yuvj* formats are full-range YUV — the correct conversion is a sws_scale to AV_PIX_FMT_RGBA, or (better for the GPU path) upload the planes directly and convert in the fragment shader, exactly like post 49’s VA-API import but through a staging buffer.

The upload

Nothing new if you’ve followed the series: R8G8B8A8Unorm image, staging buffer, vkCmdCopyBufferToImage, eShaderReadOnlyOptimal, one fullscreen triangle. Where it gets interesting is when to upload: MJPEG cams emit 5–30 fps and every frame is a full JPEG — decode cost is real. The production pattern is a latest-frame mailbox: the network/decode thread owns the socket, always overwrites the mailbox with the newest decoded frame, and the render thread picks up at most one frame per draw. Never queue — a queue is just latency with extra steps.

When to stop here

MJPEG ingest is 100% CPU work — fine for a fallback or a 1-cam debug view. For a 3×3 wall you want H.264/HEVC where possible: each decoded frame is ~2.7MB of YUV at 1080p, and at 30fps×9 cams the JPEG decode alone eats a whole core. This example is the simplest stream type a camera app must support — in a real multi-camera decoder it is the “always works” fallback path.