How to decode JPEG entirely on the GPU — parallel Huffman entropy decode in a compute shader

Post 80. The decoder counterpart to post 51’s GPU JPEG encoder: a complete baseline JPEG decoder in three compute passes. The CPU only walks the JFIF markers (DQT/DHT/SOF/DRI/SOS) and uploads tables — entropy decode, dequant+IDCT, chroma upsampling and YCbCr→RGB all run on the GPU.

example.sh
$ ./app --verify photo.jpg
coefs: 0/6144000 mismatch
rgb: max |diff| 0  mean 0.0000

Bit-exact vs libjpeg-turbo on a 192-file corpus (gray, 4:4:4, 4:2:2, 4:2:0, restart intervals, odd dimensions, quality 5–100, optimized Huffman tables, real-world photos) — same integer islow IDCT, same “fancy” upsampling, same fixed-point color conversion.

Example: VulkanExamples/p80_gpujpegdec

Structure

example.txt
JPEG → host: marker walk → DQT/DHT/sof tables, scan span,
             restart-interval boundaries
  dispatch 1: entropy  — one invocation per restart interval
      VLC bitreader → 16-bit fused Huffman LUT → quantized coefs
      (2048-MCU chunks, bitstate+DC-pred handoff between generations)
  dispatch 2: recon    — one invocation per 8x8 block
      dequant + libjpeg islow integer IDCT → component planes
  dispatch 3: upsample — one invocation per pixel
      fancy h2v1/h1v2/h2v2 triangle upsampling + fixed-point YCbCr→RGB

Restart intervals are the parallelism

The encoder post manufactured parallelism by emitting restart markers every MCU. The decoder inherits whatever the file says — and it turns out the same RSTm markers are the only legal sync points in the stream: at a restart, the bitstream re-aligns to a byte boundary and the DC predictor resets to 0. So the work item is the interval, not the MCU: an interval table (built by the host from the DRI and the SOS scan) maps each invocation to a byte offset and an MCU range. A 3000×2000 photo with DRI=1 has 23500 independent intervals — dispatched in a single wave, the whole 4 MB entropy stream decodes in ~14 ms (~430 MP/s, >10× libjpeg-turbo).

DRI=0 files are the opposite extreme: one interval, one serial bitstream. There a single invocation walks ~2M symbols — latency-bound at ~1.9 MP/s on a Renoir iGPU. (Production GPU decoders handle this with speculative boundary resync — decode chunks at guessed offsets, keep the chains that tile the stream exactly. That’s the known next step; every other stage is already parallel either way.)

The watchdog problem — chunked generations

A serial invocation decoding 115K MCUs takes seconds — long enough for the driver to kill the submission mid-flight, leaving an untouched zero tail (the failure is non-deterministic: the cutoff moves between runs). The fix: split each interval’s MCU range into 2048-MCU chunks forming an (interval × chunk) grid, and hand off the bit-reader state (pos, buf, cnt, end) plus the four DC predictors through an SSBO — each generation of chunks is a separate queue submission, so no single submission runs long.

kEntropy (state resume)
if (w.w == 0u) {              // first chunk of an interval
    b.pos = w.x; b.buf = 0u; b.cnt = 0; b.end = false;
    pred = ivec4(0);
} else {                      // resume predecessor's state
    uint s = (idx - nIv) * 8u;
    b.pos = st[s]; b.buf = st[s+1]; b.cnt = int(st[s+2]);
    b.end = st[s+3] != 0u;
    pred = ivec4(int(st[s+4]), int(st[s+5]), int(st[s+6]), int(st[s+7]));
}

One load per symbol: the fused 16-bit LUT

Classic Huffman decode walks mincode/maxcode/valptr length by length — several dependent loads per symbol, fatal on a latency-bound serial path. Instead the host precomputes a flat 64K-entry LUT per table: index it with the next 16 bits of the stream, get everything back in one u32:

  • bit 31 set → fused: [4:0] total bit length, [8:5] run, [12:9] size, [28:13] the already sign-extended coefficient value — valid whenever huffLen + extLen ≤ 16, which covers the overwhelming majority of symbols;
  • bit 31 clear → plain: [4:0] code length, [15:8] symbol — rare long codes still work via a second rext() peek.
kEntropy (AC decode)
int e = flut[uint(at) * 65536u + peekN(b, 16u)];
if (e < 0) {                 // fused: one load, everything resolved
    eat(b, uint(e & 0x1F));
    rr = (e >> 5) & 0xF; ss = (e >> 9) & 0xF;
    val = ((e >> 13) & 0xFFFF) - 32768;
} else {                     // long code: symbol + explicit extension
    int len = e & 0x1F;
    if (len == 0) break;
    eat(b, uint(len));
    int sym = (e >> 8) & 0xFF;
    rr = sym >> 4; ss = sym & 15;
    val = (ss > 0) ? rext(b, ss) : 0;
}

Coef writeback is similarly latency-aware: the buffer is pre-zeroed with vkCmdFillBuffer, and each block accumulates one pending u32 in registers — one plain store per word, no read-modify-write.

Bit-exactness is the hard part, not the shaders

The IDCT/upsample/color stages reproduce libjpeg-turbo’s arithmetic exactly: the islow integer IDCT (CONST_BITS=13, PASS1_BITS=2), the “fancy” triangle upsampler with its alternating +1/+2 / +7/+8 rounding biases and edge replication, and the 16-bit fixed-point YCbCr→RGB tables. Any deviation is instantly visible in the verifier, which compares against jpeg_read_coefficients (entropy stage) and jpeg_start_decompress (final pixels):

example.sh
$ ./check.py
ok   corpus/cj_rst1.jpg   97280 coefs
ok   corpus/photo_1023x767_s2q75.jpg  1179648 coefs
...
192 pass, 0 fail  (137 generated + 40 real + 15 reject)
ALL BIT-EXACT vs libjpeg

Comparing coefficients has one gotcha: libjpeg’s virt_barray dims are ceil(downsampled/8) — not MCU-padded — while the GPU grid is MCU-padded. The verifier maps each reference block into the padded layout; read past libjpeg’s array and you’re diffing heap garbage.

Support matrix & honest limits

Baseline SOF0/SOF1, 8-bit precision, 1 or 3 components, all sampling factors, restart markers, fill bytes, optimized Huffman tables. Progressive, multi-scan, arithmetic-coded, >3-component and non-8-bit inputs are rejected cleanly by the marker walk — no out-of-bounds GPU access on malformed input.

  • restart-heavy files: ~430 MP/s — intervals dispatch in parallel
  • DRI=0: ~1.9 MP/s — serial VLC, memory-latency bound
  • libjpeg-turbo (CPU): ~33 MP/s regardless of restart markers

For an image pipeline that controls its encoder (like post 51 or VulkanoCam’s captures), emitting restart markers makes GPU decode 13× faster than libjpeg. For arbitrary camera files it’s a mixed bag: many cameras do write restart intervals, but DRI=0 needs the speculative-resync scheme to shine — a different article.


Check out similar posts by category: C++ Vulkan Video