LPDDR5X-PIM: 8× bandwidth became 3× tokens — where the other 5× went

The partWhat was shown

A standard LPDDR5X die has 16 banks. Samsung put a PIM block in each: a MAC tree plus an ALU that handles both integer and floating-point formats. At 8-bit precision each block does 4 multiply-accumulates per data clock, at 4-bit weights 8 per cycle; across the package that adds up to about 2.4 TOPS with INT4 weights or ~1.2 TFLOPS at FP8. The interesting number is not the compute, it is where the operands come from:

The package is a JEDEC-standard 561-ball LPDDR5X footprint — 16 GB across four dies per rank — so it is meant to drop into an existing board. Programming it goes through ordinary LPDDR5X commands aimed at reserved row addresses: a 1 024-bit instruction register holds up to 64 sixteen-bit instructions, a 4 kbit register file holds the activation vector, a 2 kbit file holds scales, and every register write is broadcast to all 16 banks. One operand comes from the register file, the other from the bank the block sits on. In other words: a tiny SIMD machine that you talk to by writing to memory addresses.

The demo: Llama 3.1 8B, 320-token context, SINT8 activations and SINT4 weights with 32-bit accumulation, on a Samsung edge AI accelerator SoC. Runtime went from 12.3 s to 5.4 s (2.28×); token throughput from 27.0 to 81.3 tokens/s (3.01×). Samsung calls the results preliminary and expects total energy not to rise, since the point of the exercise is to stop moving bytes across the bus.

RooflineWhy it works: decode is a weight stream

Generating one token in an 8B model at batch size one means reading every active weight once and doing almost nothing with it — a matrix-vector product per layer, a few flops per byte. At INT4 the weights are about 4.0 GB. The KV cache at 320 tokens is ~40 MiB and doesn't matter yet. So the per-token floor on a single x64 rank is:

# bytes per token / external bandwidth
4.0 GB / 76.8 GB/s  =  52 ms   →  ceiling ≈ 19 tok/s
# same weights through the in-bank MAC trees
4.0 GB / 614 GB/s   =  6.5 ms  →  ceiling ≈ 153 tok/s

Two things fall out. First, the baseline of 27 tok/s is above the single-rank roofline — so the test platform has more than one channel feeding the model, or the reported rate is not pure decode. Samsung hasn't published the channel count, which is one concrete reason to treat these as preliminary. Second, whatever the channel count, the ratio between the two paths is fixed at 8×, and the ratio Samsung measured is 3.01×. That gap is Amdahl's law, and it is quantifiable.

AmdahlWhere the other 5× went

If a fraction f of the per-token time is the weight stream and gets 8× faster, and the rest does not change:

speedup = 1 / ((1 − f) + f / 8) = 3.01
      f = (1 − 1/3.01) / (1 − 1/8) = 0.763

So on the conventional path, about 76 % of decode time was the GEMV weight stream — which is the textbook profile for a bandwidth-bound decoder, and a sanity check that the 3.01× is a real bandwidth win rather than a compiler artefact. After PIM, the picture inverts: the weight stream is now ~29 % of the shorter token time and the untouched 24 % has grown to ~71 %. With that remainder unchanged, no amount of extra in-bank bandwidth gets the workload past 4.2×. The next 2× is not in the DRAM; it is in whatever the remainder is made of.

What the remainder is, from the programming model above: host-side attention, normalisation and sampling; the round trip of writing an activation vector into the register file, issuing the instruction stream and reading results back over the same 76.8 GB/s bus; and the software constraints that make this a real platform decision — PIM regions must be mapped uncacheable and non-speculative, and a channel cannot serve PIM and ordinary traffic across threads at the same time, so channels get reserved for it. The 2.28× runtime figure against the 3.01× throughput figure says the same thing from the other side: prefill is compute-bound GEMM, not GEMV, and the MAC trees do not help it.

ComparisonPIM vs unified memory

It is tempting to line 614 GB/s up next to the 410–546 GB/s that an M4 Max delivers from its unified LPDDR5X, and conclude that a phone's DRAM has caught up with a workstation. It hasn't, because the two numbers measure different things. Unified memory delivers its bandwidth to any consumer for any operation — that is what made step 9 of the case study go zero-copy between CPU and GPU. PIM's 614 GB/s exists only inside the package, only for a MAC tree that multiplies a broadcast vector against the bank it lives in. Send it a softmax and it is a 76.8 GB/s part again.

That makes PIM a specialised accelerator that happens to be hidden inside a DRAM package, not a faster memory. Which is fine — matrix-vector against static weights is the dominant term in single-stream decode and the one worth attacking — but it means the product is the compiler and runtime that carve a model into "in-bank" and "everything else", not the die.