How kimi-k3-in-c Runs a 2.8-Trillion-Parameter Model on 8 GB of RAM

TL;DR

kimi-k3-in-c never loads the full model into memory. It keeps a small fixed part (the dense trunk) permanently resident, and fetches the rest (the experts) from disk on every token, pulling only the 16 experts out of 896 that actually matter. Experts sit pre-compressed at 4 bits on disk, a cache keeps the most requested ones within reach, and the RAM budget only tunes speed: the output stays identical whether you give it 8 GB or 224 GB.

---

The problem: a file that fits nowhere

The Kimi K3 checkpoint weighs 1.56 TB on disk, spread across 96 files. No consumer machine, and no server under a terabyte of RAM, has enough memory to load it whole. The question kimi-k3-in-c answers isn't "how do I compute faster," it's "how do I fit a model 200 times bigger than available RAM without changing its output."

The answer comes down to four architectural decisions, applied in order.

A quick refresher: mixture-of-experts

Kimi K3 is a Mixture-of-Experts (MoE) model. Instead of one large network that processes every token with all its parameters, the model holds 896 specialized sub-networks, the experts, and a router that picks 16 of them for each token it produces.

A dense model of the same size would use all 2.8 trillion parameters on every computation. An MoE model like Kimi K3 activates only a fraction of that: about 104 billion active parameters per token, or 3.7% of the 2.8 trillion stored on disk. That property, not a compression trick, is what makes the rest of the architecture possible.

Reduction one: the experts are already half-compressed

On disk, each expert sits in MXFP4 format, a 4-bit float instead of the usual 16 or 32 bits. The engine multiplies directly on this compact representation, without decompressing it into memory first. First gain: 1.45 TB of routed experts fit in a quarter of their original weight.

Reduction two: separating what stays fixed from what streams

The model splits into two parts of a different nature. The dense trunk, 93 attention layers (MLA, a single shared latent instead of 96 separate heads, and KDA, a linear attention mechanism whose memory doesn't grow with the text) stays resident in memory permanently or gets re-read from a compact file depending on the chosen budget. The 1.45 TB of routed experts, on the other hand, are never resident: they're read and multiplied directly from disk, on demand.

The trunk gets rewritten once, into a 109 GB file where each layer lives at a known offset and reads in a single disk call. That rewrite, the packing step, is what turns the memory constraint into a tunable dial rather than a fixed ceiling.

The LRU cache: which experts to keep hot

For every token generated, the engine already knows which 16 experts out of 896 it needs, the router has just picked them. An LRU cache (least recently used, the least recently used entry is evicted first) keeps the most frequently requested experts in memory, to avoid re-reading them every time they come back into play.

The size of that cache depends directly on the RAM budget chosen through the engine's presets (laptop, workstation, server). The bigger it is, the less the engine has to re-read the disk for the same experts, and the faster generation runs.

The real bottleneck: disk, not compute

The measurements published in the repository are unambiguous. With the laptop preset, 8.24 GB of RAM, the engine produces one token every 32.7 seconds on average. With the server preset, 127.92 GB of RAM, that delay drops to 10.7 seconds per token.

The computation itself, a matrix multiplication over 104 billion active parameters, would take a few hundred milliseconds on a modern CPU. The gap comes from elsewhere: every expert missing from the cache has to be read from disk before it can be multiplied, and that read dominates the total time. Giving the engine more RAM doesn't make it compute faster, it just spares it from re-reading disk as often.

Why the output never changes

One detail sets this project apart from classic lossy compression: the memory budget changes neither the weights nor the computations, only where the data sits before it gets used. Whether the trunk is fully resident or re-read layer by layer, whether the expert cache holds 100 or 10,000 entries, every expert ends up multiplied in its complete, exact form. The repository proves it with a test suite that compares the engine's output against a PyTorch reference, position by position, on a scaled-down model that shares the same tensor graph as Kimi K3.

What generalizes beyond this project

The general idea, split a model into a small, always-useful part and a large, rarely-consulted part, then load the latter only on demand, applies well beyond this project. It's the principle behind llama.cpp's on-demand loading and expert offloading for DeepSeek. kimi-k3-in-c offers a particularly readable implementation of it, in dependency-free C99, with tests that prove every claim rather than just stating it.

That leaves the question of real-world use: 10 to 32 seconds per token against what a GPU already does today, that gap isn't a rounding error. Why a GPU still matters works through that gap in detail.