Blog

Every guide to running a model locally arrives at the same table: FP32 takes this much memory, FP16 half as much, 4-bit a quarter, and then everyone moves on to which file to download. The table is not wrong, but it is a summary of a process that has several decisions inside it, and each of those decisions costs you something different — memory, accuracy, or time. Worth reading properly, because the choice you make changes depending on whether your constraint is a card with 6 GB or a workstation you are not allowed to open.

What quantization does, precisely

A model’s weights are numbers. Quantization replaces those numbers with lower-precision representations: fewer bits per weight, and correspondingly less memory and less memory traffic, which is what actually limits token generation on consumer hardware. A neural network in FP32 uses 32 bits per parameter; an INT8 version uses a byte; a 4-bit version uses under half of that. The model does not change architecture — only the storage format of its weights and, in some schemes, the numbers needed to decode them.

Two caveats the summary tables skip:

• Not everything gets quantized. Layer norms, embeddings, rotary tables and small 1-D tensors stay at

fp32 or fp16. That is why a “4-bit” file does not weigh four bits per parameter times N — it weighs about 4.5–4.85.

• The quoted sizes are weight sizes. KV cache, activations and context buffers are not in them. At long

context, the KV cache can be the difference between fitting and not fitting.

The ladder, with corrected numbers

A table circulating around recently gives this shape for a 7B model: FP32 ~28 GB, FP16/BF16 ~14 GB, INT8 ~7 GB, Q4_0 ~3.8 GB. It is right about the orders of magnitude and slightly wrong in ways that matter when you are choosing a file.

format	bits stored per weight	effective bits/weight	7B file, weights only	what it is for
FP32	32	32.0	~28 GB	training state; nobody runs this to generate text
BF16 / FP16	16	16.0	~14 GB	the honest baseline for local inference
INT8 (per-tensor)	8	~8.5	~7.2 GB	rarely used directly; llama.cpp’s q8_0 is 8.5 bpw
Q4_0	4	4.5	~4.3 GB	the oldest 4-bit ggml format, block-32 with one fp16 scale per block
Q4_K_M	4	~4.85	~4.1–4.5 GB	default “good enough” choice for most cards
Q5_K_M	5	~5.66	~4.8 GB	noticeably better than Q4 at modest extra cost
Q6_K	6	~6.56	~5.5 GB	near-BF16 quality, large file
Q8_0	8	8.5	~7.2 GB	quality ceiling of the ordinary formats

The correction that matters is effective bits per weight: side information — scales, and in K-quants extra quantized min/max levels — is part of the format, so a “4-bit” quant needs 4.5 to 4.85 bits per weight of storage. If you budget memory from the name rather than from the size, you will be wrong by about ten percent, and ten percent is how a model fails to fit on a card with no headroom.

KV cache arithmetic is where people get surprised twice. A rough estimate: two bytes per token per layer per head times the number of layers and heads, doubled because there are keys and values. On a 32-layer, 32-head, 128-dimension model that is 0.5 MB per token in BF16 — meaning a 32k-token context costs about 16 GB before you notice it. This is why quantizing the cache (--cache-type-k/--cache-type-v, q8_0 or q4_0) buys back a lot on long-context runs, and why “the file fits” and “the run fits” are separate questions.

Anatomy of a block: why group size costs memory

A ggml quant does not quantize the whole tensor at once; it chops it into blocks — 32 weights for Q4_0, larger super-blocks for K-quants — and stores, for each block, a scale (and in richer formats a second-level scale plus quantized minimum/offset values). Decoding multiplies the stored integer by the scale.

Three consequences worth internalising:

1. Small groups cost memory. A 16-bit scale per 32 weights is 0.5 bits per weight of overhead — half a whole bit of payload. Wider blocks amortise better, which is part of why K-quants can beat Q4_0 at similar size: they spend their bits differently rather than having more of them. 2. Two-level scales help quality. Q4_K and friends add sub-block scales; Q4_1-style formats add a stored offset per block. Same data, more side information, better reconstruction of outliers — the reason the “same 4 bits” variants have different sizes. 3. Some formats are not fixed-width at all. The IQ types store their values entropy-coded (arithmetic coding) with an importance matrix (--imatrix, from a calibration corpus), so their bit-width depends on the content, and they need lookup-table decoders. They can hold more accuracy per bit; they cost time to run.

That third point is not theoretical. The model card behind the recent GSQ-RCO builds measured it directly: two builds of nearly identical size, one using ordinary Q2_0 blocks and one using IQ2_XS, at 367.49 vs 108.19 prompt tokens/s and 93.79 vs 70.30 decode tokens/s on the same prompts — with the lookup-table build varying by 45 tokens/s across task categories versus 2 for the ordinary one. The smaller file was the fast one. If your interest is speed rather than benchmark scores, format choice matters more than bit width.

What a filename does and does not tell you

The name of a GGUF file states the main type — Q4_K_M, IQ3_XS — not the composition of the file. Real files hold many tensors at higher types and some small tensors at fp32; and labels like UD, _XL or GSQ-RCO are recipes, produced by someone’s process, not formats with a fixed bit-width. Two files with the same type label can be different sizes with different quality.

Which is exactly why the newest approach is to stop naming formats at all. In the GSQ-RCO builds, one quantization type is chosen per tensor by a constrained optimiser under an explicit size budget, and the resulting file carries an allocation record (rco-allocation.txt) listing which type every tensor got. The name in the filename becomes the target average bit-width — 2.40, 3.00, 3.50 — because that is what was actually controlled.

That distinction is the practical difference between picking a quant today and in a year: you stop guessing which hand-tuned recipe is best for your card and start telling something your memory budget.

Choosing on real hardware

The honest answer depends on which part of the trade you are paying for.

• You have 6–8 GB. 4-bit K-quants (Q4_K_M) are the starting point; expect a 7–8B model to fit with a

short context. Use IQ types only if you have the patience for decode cost and want the extra accuracy.

• You have 12 GB. Q5_K_M or a Q4 variant with more headroom for context. More context than precision is

usually the better trade at this size.

• You have 16–24 GB. Q6_K, or a quantized large model instead — often the big-model, lower-bit choice

beats a small high-quality model on reasoning tasks. If you are near 24 GB and want the large models, quantizing KV cache is how you keep enough context.

• You are deciding between “quantized” and “original.” Original weights are not automatically better;

careful low-bit quantization often matches it on standard benchmarks. The recent GSQ-RCO cards report a 3.00 bpw build at 92.57 vs the BF16 base’s 93.12 on the task average, at 4.7× smaller — and are candid that recoverings above 100% on zero-shot averages are noise, not improvement.

Then measure on your own workload rather than trusting anybody’s table: prompt length, batch size and which layers a request touches change throughput more than the bit-width does. The two formats at the same size in the example above differed by more than 3× in prompt speed. That is the useful number to know about quantization: the parts you can see in the benchmark table are the least interesting part of what you will actually run.

Read also

• GSQ and RCO: the non-uniform quantization now sitting in GGUF files — posts/gsq-rco-blog.md, published at

https://blog.localai.hex7.com/

• Llama.cpp – What is it? (the engine that runs these formats) — same blog.

Sources

• llama.cpp quantization format notes and effective bit-widths: https://github.com/ggml-org/llama.cpp
• GGUF quant name explainer: https://openclawdc.com/blog/gguf-quant-names-explained/
• GSQ (Gumbel-Softmax Quantization), arXiv:2604.18556; RCO, arXiv:2605.00649; reference implementations at

IST-DASLab/GSQ and IST-DASLab/RCO — method details in the model cards at https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF

A GGUF file used to tell you one thing about its precision: the type in its name. Q4_K_M meant most weights were 4-bit with a medium-quality recipe, and that was the whole story. Recent repositories break that convention. Files named Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf carry a recipe label rather than a bit-width, and they are worth understanding, because the trick they encode — one quantization type per tensor instead of one for the whole model — is how a 354 GB model becomes 75.8 GB without a noticeable fight about it.

The two methods are called GSQ (Gumbel-Softmax Quantization) and RCO (Riemannian Constrained Optimization), both from the Deep Algorithms and Systems Lab at the Institute of Science and Technology Austria. Their model cards describe them plainly: “GSQ provides accurate low-bit scalar quantization of each tensor at a given quantization type; RCO assigns the per-tensor quantization types under a size budget.” That division of labour is the whole design.

GSQ: making 2 and 3 bits behave like vector quantization

Post-training quantization has a long-standing split. Scalar quantization rounds each weight to the nearest value on a grid — fast, and it is what GGUF formats actually store. Vector quantization (codebooks, k-means) is usually more accurate per bit but needs extra decode machinery, which is why formats like IQ types need lookup tables.

GSQ sits in between by learning the grid instead of using a fixed one. Concretely: it jointly learns the per-coordinate assignments — which weight goes to which codebook entry — and the per-group scales that decode them, optimising both at once through a Gumbel-Softmax relaxation, a differentiable stand-in for the discrete choice you cannot back-propagate through. The result the authors claim is that scalar formats at 2–3 bits get close to what vector quantization would achieve, and stay in standard GGUF form so existing engines can run them unchanged.

That last clause matters more than it sounds. A method that only works inside a research runtime is not a practical deployment story; a method that produces ordinary GGUF blocks and scales is.

RCO: choosing where precision goes, with the budget actually enforced

RCO is the allocation problem. Given N tensors and K candidate quantization types (Q4_K, Q5_K, IQ4_NL, IQ3_S, …), which type should each tensor get so that the total file size hits your target — say 4.0 bits per weight average — while task loss is as low as possible?

The usual approach is a heuristic or a knapsack after thought: pick sizes, then hope the model still works. RCO’s move is to make the constraint itself geometrically tractable. It reformulates the exact size budget as a smooth Riemannian manifold in logit space, then optimises directly on that manifold, so every step it takes is a step that satisfies the budget exactly. No penalty weights to tune, no “close enough on size” slop. Gradients come from the task loss, which means the search is sensitive to what the model actually does badly at low precision rather than to a proxy like weight-space error.

Practically, this changes what you can ask for: build me a 75 GB version of this model, and get one.

Put together, the procedure in the model cards is three steps:

1. Per-tensor database. Every weight tensor is quantized at every candidate type with GSQ. This is the search space, and it costs GPU time to build — for a Qwen3.8-Flash-Next build the card says 352 tensors (304 dense plus 48 fused routed-expert matrices). 2. RCO search. Assign one type per tensor on the manifold, so the whole-file average bit-width meets the target exactly. 3. Assembly. Stitch the selected per-tensor variants into one standard GGUF.

The released files ship an audit artifact for this: tensor-allocation/<model>.rco-allocation.txt lists the type assigned to every tensor, with a histogram of types and the target bit-width — so you can inspect what the search decided without loading the model at all. That is unusually honest packaging.

What the numbers actually look like

The Qwen3.8-Flash-Next card is the clearest worked example. It is a sparse mixture-of-experts model: 512 routed experts per layer across 48 layers, ten active per token, and the expert matrices dominate both the parameter count and therefore the size budget — “95% of the searchable weights live in the experts.” The search assigns a type per layer, not per expert, because GGUF cannot express per-expert types within a fused matrix.

The shipped variants, with average bit-width over transformer weights:

build	bpw	shard 1 (resident)	shard 2 (n-gram table)	total	note
Q2_0	2.40	37.6 GB	28.8 GB	66.4 GB	built for throughput, avoids lookup-table formats
IQ2_XS	2.50	39.2 GB	28.8 GB	68.0 GB	smallest at equal quality
IQ3_XXS	3.00	47.0 GB	28.8 GB	75.8 GB	matches the base on AIME25
IQ3_S	3.50	54.8 GB	28.8 GB	83.6 GB	recommended
BF16 base	16.00	—	—	354 GB	reference

Quality numbers from the same card, evaluated against the BF16 base (task average is the mean of AIME25, GPQA-Diamond and LiveCodeBench v6): at 3.00 bpw, IQ3_XXS reaches 92.57 against 93.12 — 4.7× smaller — with AIME25 exactly equal to the base. At 2.50 bpw, IQ2_XS keeps 89.16 of 93.12 at 68 GB. And note what the card says about recoveries above 100% on zero-shot averages: they are “within roughly one standard error” and should be read as parity, not improvement. That caveat is worth quoting, because marketing copy does not usually include it.

The part that matters for someone running this locally

Two things in these files are easy to miss and both are deployment advice.

The second shard is a lookup table, not weights. Every build carries the same 28.8 GB shard: a per-layer n-gram embedding table (51.2B parameters) held at IQ4_NL, fixed at 4.5 bpw in every build and excluded from the search. It is read sparsely — one row per token — so it does not have to be resident. The card’s instruction is -lm mmap --lazy-mode on, plan for shard 1 in VRAM, and keep shard 2 on an SSD. That is how a 75.8 GB model becomes something you can actually run on a big card rather than something you cannot run at all.

Format choice is a speed decision, not only a quality one. The card measures Q2_0 against IQ2_XS on the same 55-prompt suite: 367.49 prompt t/s and 93.79 decode t/s versus 108.19 and 70.30, average latency 6.70 s versus 12.68 s — with Q2_0 being the smaller file by 1.6 GB. The difference is decoding cost: IQ formats pack more accuracy per bit but decode through lookup tables, and on this model that cost dominates. Q2_0’s decode rate stays within a 2.0 t/s spread across categories; IQ2_XS swings 49.4 to 94.5 t/s, a 45.1 t/s spread, because how much of each layer a request touches varies with the prompt. Long-context work is where the gap widens: 9.6× faster prefill on RAG, 6.8× on writing. Quality-wise the two are 0.09 apart on task average. If you want throughput rather than bragging rights, Q2_0 is the honest pick at that size.

Two more practical notes from the cards. The DeepSeek-V4.1-Flash build needed “llama.cpp b11870 or newer” to load its layout — these builds move faster than the engines that run them, so check your engine version before assuming a GGUF will work. And ffn_down_exps has its own constraint: 640 rows, not divisible by 256, which rules out every block-256 K and I format and leaves Q2_0 (block 64) and IQ4_NL (block 32). In the smaller builds the search puts every such layer on Q2_0; in IQ3_XXS it lifts 18 of 48 layers to IQ4_NL. Constraint like that is what “quantizing a model” really looks like in practice: not a formula, a set of hard edges the optimiser works around.

Why this is worth your attention

Per-tensor mixed precision has been available in inference engines for a while and invisible in GGUF. GSQ-RCO moves it into the file format itself: ordinary blocks, ordinary scales, ordinary engine support, plus a sizeable allocation that a gradient search picked. The naming change is the tell — UD-Q4_K_XL was always a hand-tuned recipe; GSQ-RCO-IQ3_XXS is the output of an optimiser told exactly how big to be.

For local use, the useful consequence is that a memory budget stops being a guess. You say 8 GB, 12 GB, 24 GB, and something decides which weights keep precision. The model cards claim Strata runs these GGUFs on a single NVIDIA GPU from 8 GB of VRAM — on our machines that is where these files start to matter, and it is also where I would urge you to measure rather than believe: the two builds at the same size differ by 3× in prompt throughput.

The paper references are GSQ (arXiv:2604.18556) and RCO (arXiv:2605.00649); reference implementations are at IST-DASLab/GSQ and IST-DASLab/RCO, and tooling is picking it up — GPTQModel’s changelog mentions a GSQ/RCO teacher-KL optimisation path. The model cards list the per-build benchmark tables if you want to check the numbers rather than take them as given.

Sources

• ISTA-DASLab Qwen3.8-Flash-Next-GSQ-RCO-GGUF model card: https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
• ISTA-DASLab DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF
• GSQ-RCO-GGUF collection: https://huggingface.co/collections/ISTA-DASLab/gsq-rco-gguf
• GGUF quant name notes: https://openclawdc.com/blog/gguf-quant-names-explained/

PolyStrata (github.com/simplexai-labs/PolyStrata, MIT; created by VecSzn) is a fork of Strata that grew a second engine. Where Strata serves one model family well, and where Avalon doubles down on a single model inside a llama.cpp fork, PolyStrata went the other way: it kept Strata’s engine untouched for Qwen3.8-Flash-Next, and wrote its own engine for eleven more models – GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Flint, Qwen3.6, Clover, Ornith 1.5, Gemma 4, Tiel Coder, MiniCPM-V and Kolibri 1.

The project’s own framing is the most useful part: "Run large AI models on your own PC – models that usually need a server." Its README describes what that costs: the largest model it hosts, GLM 5.3 Flash, is a file of 79 to 114 GB whose bulk stays in RAM, and needs a 32 GB card to run at all.

How it differs from the other three

Against llama.cpp: PolyStrata’s comparison claim is specific and self-reported – on an RTX 4070 Laptop with 8 GB and 32 GB of RAM it read prompts 2.7 to 3.5 times faster than llama.cpp b11540, and wrote 1.5 to 1.8 times faster, tested on Qwen3.6 and Tiel Coder. The mechanism is the same one Strata and Avalon push: stop letting the general runtime decide where weights live. Its engine also ships a benchmark you can run on your own machine – python polystrata.py benchmark --model <name> --prompt-tokens 24000 – and a community results file, which is an unusually good habit for a project of this age.

Against Strata: PolyStrata keeps Strata’s engine unchanged for Qwen3.8-Flash-Next ("its engine is the Strata project’s, unchanged") and adds its own backends for everything else, marked experimental – they have been run on one PC with a 32 GB card, and eight of them on an 8 GB laptop card. That split is why it is a separate repository rather than a pull request: different models, different architecture handling, a different maintenance path. Its installer also does things Strata does not: START-HERE.bat or ./setup.sh, model selection by family flag, Mac (Apple Silicon) builds as a beta, multi-GPU on two or three cards, an MCP server so an AI assistant can install and start the thing itself, and a docs/AI_SETUP.md page aimed at pasting into Claude Code or Cursor.

Against Avalon: Avalon is the opposite bet – one model, hand-tuned kernels, custom quantisation types mainline cannot load, RDNA3 first. PolyStrata bets on breadth with a shared app and API in front of many models. On the same laptop-class hardware that means PolyStrata’s numbers are good but not surgical: 94-100 tokens/s for Qwen3.6-35B-A3B on an RTX 4070 Laptop, 251-485 on an RTX 5090. Where Avalon fights for every token on old AMD cards, PolyStrata is trying to make a dozen MoE models runnable at all – including ones with unusual memory profiles: LFM2.5 keeps almost no attention layers, so a 128k context costs 2 GB more than 32k; Flint looks back only 513 tokens for most layers; DeepSeek V4 Flash costs 0.7 GB extra VRAM between 32k and 128k.

The memory control is the engineering worth reading

PolyStrata’s recent work is the same lesson Avalon taught, from the other direction: how much RAM an engine keeps for itself. Since 0.1.45-poly.7 its engine no longer holds parts of the model file that are already on the graphics card – Qwen3.6-35B-A3B after a 32,000-token prompt went from holding 17.9 GiB to 13.1 GiB, at the same speed. There’s an explicit knob for it, --ram-experts-mib, which trades held RAM against write speed with numbers attached rather than promises:

• holding 6,144 MiB instead of everything: RAM held after load drops from 11.7 to 6.5 GiB, peak during a 32k prompt from 14.0 to 7.3 GiB – at the cost of 3% on the first short chat and 13-14% on later turns.
• when the experts outside VRAM do not fit in free RAM, it now budgets a hot-expert set sized to free RAM minus 4 GiB and reads the rest from SSD, instead of mapping everything and taking page faults one at a time: measured on a Linux box with an RTX PRO 6000 limited to ~8 GB and the engine capped to what a 16 GB machine has, first short chat went from 51 to 57-60 tokens/s, after a 24k prompt from 75-78 to 77-85, and in the chat after that from 94-98 to 131-139.
• --lock-experts pins those experts so the OS cannot take them back; under Linux it respects ulimit -l and reports how much it locked.

That is a direct echo of the problem Avalon wrote an entire design around (page-cache eviction and driver registration limits), and it is encouraging that both projects landed in the same place: an engine must own its memory, or the machine will take it away mid-token.

The entry point is a terminal script – and a measurement tool

The file every description of this project should mention is polystrata.py, 232 lines at the root of the repository. It is the terminal front end: python polystrata.py --list-models shows what is installed and which model is in memory; --model <name> chat talks to it; benchmark --model <name> --prompt-tokens 24000 starts (or reuses) a local server on that model’s own port and reports honest numbers: time-to-first-token, prefill and decode tokens per second, cache-hit rate, how many MTP drafts were offered versus accepted, peak VRAM and RAM of the whole machine, and average GPU and CPU load. The script is explicit about its own guesses – "Four characters a token is a guess, and a text with tables comes to more" – and trims a prompt that does not fit rather than pretending it did.

That is worth copying into any local stack: if the engine cannot be measured from a script in one command, people will not measure it.

The ecosystem signal

PolyStrata has already been forked for minimalism: robertkarl/UnbiasedStrata keeps "its two inference engines unchanged and replaces everything around them: no installer, no downloads at build or first run" – which is the counter-argument to PolyStrata’s one-click approach, and worth reading before you decide which trade you want. KVMem now lists PolyStrata among its backends, alongside llama.cpp and NInfer, which is a sign that the serving layer – not any single engine – is where these tools get plugged in.

What I’d take from all four

1. General runtimes lose on memory policy, not on models. llama.cpp’s page-cache approach is exactly what both forks fix, and they quote crash-free or slower-before-versus-faster-now numbers to prove it.
2. Forks are cheap now; the expensive part is correctness. PolyStrata’s README documents per-model licences, which sizes were measured on which machines, and what is untested. So does Avalon’s, complete with its own corrections. That documentation discipline is the thing I’d copy into our own stack.
3. Pick by scope. For "whichever model today", llama.cpp plus a router. For a desktop appliance that runs one flagship model well: Strata. For squeezing one MoE out of old AMD hardware: Avalon. For a broad set of large MoEs on an ordinary PC, with an app and API in front: PolyStrata.

PolyStrata: https://github.com/simplexai-labs/PolyStrata The terminal entry point and benchmark harness: https://github.com/simplexai-labs/PolyStrata/blob/main/polystrata.py Its benchmark and community results: https://github.com/simplexai-labs/PolyStrata/blob/main/docs/COMMUNITY_BENCHMARKS.md The minimal fork of it: https://github.com/robertkarl/UnbiasedStrata Strata: https://github.com/Niko1221/Strata · Avalon: https://github.com/freriadoc/Avalon · llama.cpp: https://github.com/ggml-org/llama.cpp

Three engines now wear the same badge – "runs GGUF models on your own machine" – and they are not the same tool. llama.cpp is a general-purpose runtime. Strata is a desktop application that serves a short list of models well. Avalon (freriadoc/Avalon, MIT) is something different again: a fork of llama.cpp tuned for exactly one model, Qwen3.8-Flash-Next, on one consumer GPU with the experts living in system RAM. Its README states the intent plainly: "It is not tuned on a flagship machine: every number below comes from a desktop most people would call old."

That sentence is the whole design brief, and once you read it that way the rest of the project makes sense.

What Avalon actually is

Avalon is a llama.cpp fork – base: llama.cpp master merged up to 31 October 2026 – that started life as a branch of buun-llama-cpp, which is where the GPU expert cache and its LFU admission policy come from. It borrows the IQ3 KS and IQ3 KS-hybrid quantisation types (the type names in the repository use underscores) and the shared MTP head support from ik_llama.cpp, takes the expert-ranking format and preload idea from Strata, and ships a 89 GiB model in 33 files plus an MTP draft head.

The target model is Qwen3.8-Flash-Next: 512 experts per layer, 10 active, 48 layers. The GGUF files Avalon uses are built from quantised types and a packed layout that stock llama.cpp cannot load – so the README is candid that other models run "as in llama.cpp, without the expert cache work". It is one model, optimised within an inch of its life; everything else is compatibility, not tuning.

The difference is where the experts live

A mixture-of-experts model keeps around 49 GiB of expert weights that are mostly not needed by any single token. llama.cpp maps the model file and lets the operating system’s page cache deal with that. Avalon’s README explains exactly why that fails:

• the page cache is shared, so any other program – a browser, a compile – pushes experts out, and they return from disk at the next page fault in the middle of a token;
• the GPU driver limits memory it will register for all processes to RAM minus RAM/64 minus 1.5 GiB, so on a 16 or 32 GB machine most experts simply cannot be registered, and a long prompt ends in a crash.

Avalon instead keeps experts in RAM it owns. If everything fits (64 GB here) they are read once with direct I/O into its own address space, registered in place, and nothing else can evict them – the server is at full speed about 20 seconds after a cold start. If they do not fit, it builds an arena of pre-registered slots sized to what RAM allows (about 23 GiB on a 32 GB machine, 9 GiB on 16), seeds it at startup from a shipped ranking file, and keeps it LRU-managed, reading missing experts straight off the disk with direct I/O. That is a completely different contract from "hope the page cache is warm", and it is why an 89 GiB model runs at about 49 tokens per second on 16 GB of RAM.

The second part of the design is tiering: the GPU expert cache takes whatever VRAM nothing else is using – compute buffers are lent to it between phases, the unused tail of the KV cache is given to it, and a calculated 400 MiB stays outside; hosted MoE runs the cached experts on the GPU while the CPU computes the missing ones and moves one or two across the PCIe link when that finishes sooner; during prefill, expert prefetch runs on four slots borrowed from the cache. The stated goal is blunt: "RAM, VRAM, CPU, PCIe link and GPU should all be busy, all the time, at their limit."

Strata does the same job differently: it is an application with installers and a launcher (strata launch, strata connect), keeps per-expert VRAM caches across all layers, reads what does not fit from SSD, and serves a short list of model families across a wide set of hardware. Avalon has no app and no installer philosophy – you build it with CMake or run its ROCm installer script, then avalon-server gives you an OpenAI-compatible API on port 8080 and the "fit" option decides placement for you. Strata is a product; Avalon is a tuned engine you assemble yourself.

The numbers, with the caveats attached

Avalon’s own machine is deliberately old: Ryzen 7 3800XT (Zen 2, 2019), DDR4-3200 dual channel measured at 33-35 GB per second, an RX 7900 XT with 20 GB on PCIe 3.0 – no CUDA, no flagship. With MTP speculative decoding and two draft tokens it measures:

• 32k context, short prompt, warm cache: 66.1 tokens/s without speculation, 98.4 with it; prefill up to 1,854.
• 32k context, 32k filled: 62.3 / 88.0; prefill 1,680.
• 256k context, 131k filled: 60.5 / 88.0; prefill 1,521.
• 256k context, 250k filled: 57.3 / 81.1; prefill 1,342.

The full 256k context fits – with the MTP draft model, a quantised KV cache and the expert cache – under 19.7 GiB of VRAM.

Against Strata v0.2.9 on the same card, same protocol, both with tail-skip on, Avalon reports 98.4 vs 68.4 tokens per second on a short prompt (44 percent faster) and 81.1 vs 62.6 at 250k filled (30 percent), with prefill 1,854 vs 1,453. On NVIDIA the picture changes: on a rented RTX 4090 it is effectively even at 141.9 vs 141.4 with MTP, and Strata’s prefill is faster there (3,422 vs 3,113) because Avalon’s kernels are still tuned for RDNA3. That is a fair thing to say out loud, and the README does say it.

The low-RAM numbers are the most useful part of the document, because they show what the RAM arena buys: on 32 GB, 85-88 at short prompt falling to 40-49 after a 19k prompt; on 16 GB, about 49 and 30-38 after a long prompt – while the old page-cache path crashed at the 19k prompt and reached only 13-17. The README also corrects its own earlier claims: the previous 90 and 89 figures were measured under a memory cgroup only, with the driver’s registration limit still that of a 64 GB machine. That is an unusually honest piece of self-reporting.

Quality is where the argument "speed comes from a smaller model" dies. Perplexity on external text – English, French, code, lower is better:

• Unsloth UD-Q4 K XL (104 GB): 1.316 / 3.581 / 1.325
• Unsloth UD-IQ4 XS: 1.401 / 3.841 / 1.343
• Avalon IQ3 KS hybrid: 1.443 / 3.857 / 1.331
• ISTA GSQ-RCO IQ3 S (what Strata serves): 1.469 / 3.932 / 1.338

The hybrid keeps ISTA’s experts and requantises the up and gate tensors of 27 layers from original BF16 weights using an importance matrix – which is how it passes the model Strata runs while staying close to Unsloth’s UD-IQ4 XS at a smaller size. On GSM8K (first 200 problems, greedy): Avalon 196 of 200 and Strata 196 of 200. So the speed is not bought with a dumber model; it is bought with memory placement, kernel work and speculation.

The part neither of the others does: two GPUs as a pipeline

Avalon’s most distinctive feature is not on Strata’s list: with two GPUs visible it runs the model as a two-stage pipeline by default. Layers are split 23/25 with a flag moving the boundary, each GPU keeps its own KV cache and expert cache, prompt chunks are pipelined so both PCIe links are busy, and only hidden state crosses between stages. On two RTX A4000 that turns a plain layer split’s 1,079-1,190 prefill into 1,669-1,761 for a 19k prompt. It is untested on two AMD cards and beyond two GPUs – the README says so – but it is a genuinely different answer to "what do I do with two old cards".

What to take from this

For our kind of hardware – one box, fixed cards, models we choose ourselves – the three tools map onto three different questions. llama.cpp answers "which model can I run today, and how stable is my serving layer?". Strata answers "what does this model cost me in tokens per second on this card, off the shelf". Avalon answers "how far can one model be pushed on hardware that used to be considered too old for it", and it pays for that with a strict scope: one model family, quantisation types mainline cannot load, a ROCm installer only, RDNA3 kernels tuned and NVIDIA un-tuned.

Three honest conclusions from reading all three READMEs:

1. Scope is the price of speed. Avalon’s 44 percent over Strata comes with a single-model mandate; Strata’s breadth comes with its own limits; llama.cpp’s generality is what lets it run everything and be the thing you build a stack around.
2. The memory tier is the interesting engineering, not the tokens per second. Owning your RAM instead of trusting the page cache is the fix that turns a crash into a working 16 GB machine – whichever engine you use.
3. Read the caveats in the document, not just the headline table. Avalon corrected its own README numbers mid-flight, and Strata’s author documents where each figure came from. That is the difference between a project you would build on and one you would only quote.

If you want to try Avalon on the machine you already have: the installer checks your hardware, builds llama-server for the detected GPU, downloads about 98 GiB of disk worth of model and MTP head with SHA-256 verification, and then avalon-server serves an OpenAI-compatible API. For a serving layer that treats engines as interchangeable – which is how I would run any of these – it slots under an existing launcher too: Lemonade will take Avalon’s binary in place of its own ROCm build with one environment variable and the model listed where it lives.

Avalon: https://github.com/freriadoc/Avalon Strata: https://github.com/Niko1221/Strata llama.cpp: https://github.com/ggml-org/llama.cpp

Strata gets described as "a fork of llama.cpp", and that shorthand is worth clearing up first, because the precise version of it is more interesting. Strata (Niko1221/Strata, MIT licensed) is a separate inference runtime with its own engine and desktop app; what it inherits from llama.cpp is the raw materials – GGUF files, the quantisation formats, the kernel lineage from ggml – rather than the codebase and its breadth. Third-party projects do fork llama.cpp directly (llamAmpere) or fork those forks (Project Maya, kervia-Strata). So the accurate sentence is: Strata is an overfit engine built on llama.cpp’s formats, not a branch of llama.cpp’s source.

That distinction is the whole point of the project.

What it actually does

Strata ships installers for Windows, Linux and Apple Silicon, plus a terminal command:

curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
strata launch --family deepseek41 --model ist-das-2.00bpw --plan

You pick a model family and a quantisation set, it downloads them, and serves them. It exposes OpenAI- and Anthropic-compatible APIs, so coding tools – Claude Code, Codex CLI, OpenCode, and the rest – attach to it through strata connect without knowing which engine is underneath.

Where llama.cpp tries to run thousands of models across every backend it can reach, Strata supports a short list of model families on a specific set of cards: consumer GeForce RTX 20-50 series, Radeon RX 7700/7800/7900 and the AI PRO R9700, Intel Arc on Linux, Strix Halo, DGX Spark, and Apple Silicon with 64 GB or more.

The trick that makes it fast is memory placement. Expert weights for a mixture-of-experts model are cached per-expert in VRAM across all layers, native MTP (multi-token prediction) is used for speculation instead of a bolted-on draft model, and whatever does not fit is read from SSD – which is why the install asks for a lot of free disk: the 2.00 bpw set is about 257 GB of download and asks for roughly 265 GB free.

What that buys you

On one published test on the same hardware, an RTX 4070 with 64 GB of DDR5 running a 125B MoE: llama.cpp after tuning reached about 27 tokens/s decode, while Strata served about 53 tokens/s at 60,000 tokens of context – roughly double, with no hardware change. The same writer’s notes make the caveat explicit and it’s a fair one: the quantisations and settings differ between the two engines, so that is a comparison of complete serving setups rather than a controlled experiment.

Strata’s own numbers are published per card, which is unusual and useful. A 12 GB RTX 3060 with 46 GB of system RAM: 9-11 tokens/s output, around 155 tokens/s prompt processing at 4K context. A single Radeon AI PRO R9700 with 251 GB RAM: 21-25 tokens/s. Four of them: 74-77 tokens/s. Those are the kinds of figures that change what people think is possible on a desktop.

The engineering reason is boring and true: generality is a performance tax. A change to how experts move across PCIe in llama.cpp must not break Metal’s unified-memory path or twenty other backends, so it goes through review, slowly. A runtime with one model family and one card can make the radical change on a Tuesday.

What you give up

The trade is honest, and Strata’s own documentation is blunt about it: today this is effectively a short-list-of-models engine, while llama.cpp runs thousands of them across CUDA, ROCm, Metal, Vulkan, SYCL and a long list of CPUs. If your workload is "whichever model shipped this week", breadth matters more than speed, and llama.cpp is the right answer.

There’s also a trust question. These engines are young, mostly one-person projects, shipping native code that talks directly to your GPU driver. A sane minimum if you run one: build from source, pin a commit, skim the diff before updating, and don’t run it on a machine holding anything you can’t reconstruct.

So the practical advice is to stop treating the engine as the permanent part of your stack. Keep the serving layer stable – an OpenAI-compatible endpoint, a model router such as llama-swap – and treat the runtime underneath as replaceable. That way trying Strata is one model name away in your config, and llama.cpp is still one model name away when the experiment needs to be undone.

For hardware like ours – a single box, fixed cards, models we choose ourselves – this is exactly the profile where an overfit engine makes sense: the target doesn’t change, so specializing at it is free performance you were paying for in generality you didn’t need.

Strata: https://github.com/Niko1221/Strata The comparison write-up I lean on (with its caveats in full): https://carteakey.dev/blog/local-inference/the-rise-of-overfit-inference-engines/

Llama.cpp is an inference engine for large language models, written in plain C and C++ with no framework dependencies. Its stated goal is to run LLM and vision-model inference "with minimal setup and state-of-the-art performance on a wide range of hardware – locally and in the cloud". In practice that means you take a model file, point a command-line tool at it, and tokens come out of your own machine instead of someone else’s datacentre.

The repository lives at https://github.com/ggml-org/llama.cpp and is built on top of ggml, a general tensor-library with its own operator kernels. llama.cpp adds the model loading, quantisation, sampling, prompt templating and server layers above it.

How you actually use it

There are several ways in, and they all do the same thing:

# installer script
curl -LsSf https://llama.app/install.sh | sh

# download and run a model straight from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# start an OpenAI-compatible API server plus a built-in web UI
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

The last command is the one that makes it usable as a building block: llama serve exposes a local OpenAI-compatible endpoint, so anything written against that API – agent loops, chat UIs, the little assistant that answers questions on this site – can be aimed at a model running on a machine you control. Pre-built binaries and a Docker image are also published, and building from source is a matter of cloning the repository and following the build guide.

Models are usually obtained as GGUF files, which llama.cpp’s convert_hf_to_gguf.py script produces from Hugging Face checkpoints when a publisher doesn’t ship a GGUF build.

Why people reach for it

• Hardware breadth. The backend table in the README covers Metal, CUDA, HIP, Vulkan, SYCL,

OpenCL, BLAS, BLIS, CANN for Huawei Ascend, Hexagon for Snapdragon, IBM zDNN, MUSA, RPC, WebGPU and ZenDNN, plus plain CPU paths with ARM NEON, Accelerate, AVX/AVX2/AVX512/AMX and RISC-V extensions. Apple silicon is treated as a first-class target rather than an afterthought.

• Quantisation. 1.5-, 2-, 3-, 4-, 5-, 6- and 8-bit integer quantisation, so a model that

would not otherwise fit in memory or run acceptably fast can still run at all. The k-quants and i-quants families let you choose where the quality/size tradeoff sits.

• Hybrid offload. When a model is bigger than your VRAM, llama.cpp keeps part of it on the

GPU and the rest in system RAM (--n-gpu-layers for layer offload). For mixture-of-experts models there are dedicated flags: --cpu-moe keeps all expert weights in CPU memory, and --n-cpu-moe N keeps only the experts of the first N layers there while attention stays on the GPU. That distinction is what makes a large MoE usable on a single consumer card.

• No dependency chain. No CUDA toolkit installation for the CPU path, no Python runtime

required to serve a model. That matters when you maintain a small number of machines rather than a cluster.

What it does not do: it isn’t a training stack, and it isn’t trying to be the fastest possible answer on a rack of datacentre GPUs. It is optimised for the hardware people actually have.

What has changed recently

The most visible addition is speculative decoding, now documented in docs/speculative.md. The --spec-type flag accepts a list of implementations: draft models (draft-simple), EAGLE-3 drafts that read the target model’s hidden states (draft-eagle3, with supported speculator checkpoints for Llama 3.x, Qwen3 and gpt-oss), block-diffusion drafts (draft-dflash, draft-dspark), Multi Token Prediction heads from the main model itself (draft-mtp), and a set of n-gram self-speculation implementations (ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod) that need no extra model at all because they draft from patterns already in the token history. --spec-default turns on the hash-pool n-gram variant, and each implementation prints its own acceptance statistics so you can see whether it is actually helping. The same docs point at a SPEED-Bench client in tools/server/bench/speed-bench for measuring end-to-end gain.

Other current ground: OpenAI-compatible server plus a built-in web UI for chat and vision models, the llama command wrapper and llama.app installer, native MXFP4 support for the gpt-oss models across CUDA, Vulkan, Metal and CPU (see the discussion in ggml-org/llama.cpp#15095), and MoE offload options beyond the layer-split – community benchmarks such as developersdigest.tech’s October 2026 post compare --moe-cache-mib against --n-cpu-moe and treat them as different trades for short chat turns versus long contexts.

Where that leaves us

The honest way to use llama.cpp is as an infrastructure piece: it converts a model file into a local API endpoint, and everything else – retrieval, prompts, tooling – sits on top of that endpoint. That is exactly how this site’s assistant works: a model on our own hardware behind a local API, rather than a per-token bill against someone else’s server. If you want to try it, the shortest path is the installer above and a small GGUF; the next step is llama serve and then whatever you already have that speaks the OpenAI API.