The Working Set That Never Saturated

A new experiment on count tables used by LLMs demonstrates that the working set never reaches a plateau, even after thousands of lookups. The study refutes the idea that only a small resident set is needed and shows that paging costs are higher than previously assumed.

Tue, 15 Sep 2026 03:00:00 GMT

Large language models (LLMs) rely on massive tables of token counts to predict the next word. Traditional wisdom has treated each byte of these tables as resident memory, assuming that the working set – the portion of the table that must be in RAM at any one time – is small enough to fit comfortably. A recent series of experiments challenges that assumption, revealing that the working set keeps growing even after thousands of lookups, and that paging costs may be higher than expected.

Re‑examining the Resident Memory Assumption

Previous work on mixture‑of‑experts models showed that a dense core can stay in memory while experts stream from SSD, achieving 43‑70% cache hit rates. Those models rely on a router that selects a handful of experts per layer, reusing weights across tokens. In contrast, a plain count table is a random‑access lookup structure: every forward pass reads every weight, so the entire table is theoretically needed. The new study built the tables as fixed‑stride, memory‑mappable arrays, with an open‑addressed hash table per order and no auxiliary in‑memory index. Each record stores its own context tokens, ensuring that hash collisions cannot return incorrect distributions.

To verify accuracy, the researchers performed 13,500 sampled lookups, matching in‑memory tables exactly and confirming that every evaluation position produced the same flat probability vector. This rigorous correctness gate proved that the pageable rewrite did not alter model accuracy.

Working Set Never Saturates

The decisive test involved completing roughly 2,700 consecutive positions inside a single real file while monitoring which table pages were touched. Across eight files, seven continued to climb, adding between 1 and 3.8 MB to the resident set even after 2,700 completions. The last quarter of the session added 1‑3 MB, indicating no small plateau to hold resident memory.

A bounded cache experiment confirmed this finding: the hit rate plateaued at 0.744 with 16 MB resident, compared to 0.656 at 4 MB and 0.473 at 1 MB. Thus, the notion that a 143 MB table can be served from a 2 MB resident set is untenable. At 2 MB, 43% of page reads miss, underscoring the importance of locality.

Locality and Context Recurrence

The study highlighted a key distinction: code has a 0.88 repeat rate for recurring tokens, but the table is indexed by context. Within a single document, the cache saturates at order 4, yet verbatim 5‑grams never recur at any tested length. Novel order‑3 contexts keep arriving as typing continues, meaning each position is a fresh key into a 1.93 M‑context table. The runtime that inspired this research works because a router selects experts that serve many tokens, creating a reuse structure. An n‑gram table lacks such a router, so locality is weaker.

One file did plateau at 3.9 MB, six times smaller than its siblings, illustrating that locality varies enormously between files. A single hot‑set figure is therefore misleading, and a tool cannot rely on the best‑case scenario. If only one file had been tested, there was a one‑in‑eight chance of publishing the opposite result.

Paging Costs and Latency Assumptions

The researchers measured the latency constant that underpinned their verdict. They assumed 100 µs per page read, leading to an estimated 0.4 ms per position against a 10 ms code‑completer budget. However, actual device reads were slower: a 16 KB page read took 139 µs, while a 4 KB page read took 179 µs. Eight threads reduced effective latency by roughly tenfold, showing that read cost is not the binding constraint for a prefetching pager.

The experiment used F_NOCACHE to bypass the OS cache, reading random page‑aligned offsets over a 12 GB blob. Two gates ensured that the first pass proved nothing without a second pass over the same offsets. The results showed a 1.40 ratio for the F_NOCACHE two‑pass test and a 62.5× speed‑up for the cached control, confirming that the latency assumption was optimistic by 1.8×.

Implications for LLM Deployment

These findings suggest that paging a count table is not free: the flat format costs 142.7 MB versus 63 MB for a pickled representation. The working set never saturates, so designers cannot rely on a small resident footprint. Future work might explore cache policies that combine recency and frequency or layout strategies that cluster co‑occurring contexts onto the same page.

In practice, a 60 M‑token table would have a larger absolute working set, and whether the fraction falls remains untested. Sessions of 2,700 sequential positions in one file may overestimate locality compared to real editing sessions that revisit regions. Nonetheless, the study provides a clear measurement that challenges assumptions about memory usage in LLMs.

Ultimately, the experiment underscores the value of measuring assumptions. Even if the measured risk does not change the verdict, identifying the real bottleneck—whether it is working‑set growth or paging latency—guides future optimizations.

Key Takeaways

• Count tables in LLMs do not reach a resident memory plateau after thousands of lookups.
• Locality depends on context recurrence, not token recurrence.
• Assumed page‑read latency was too optimistic; real reads are slower.
• Eight‑threaded prefetching can reduce effective latency by an order of magnitude.
• Designing for small resident sets may be unrealistic; alternative cache policies or data layouts are needed.

Frequently Asked Questions

• What is a count table? A data structure that stores token frequencies used by LLMs to predict the next word.
• Why does the working set keep growing? Because each new context key is unique, the table continually loads new pages into memory.
• Can we reduce the resident footprint? Possible approaches include better caching policies or clustering contexts on the same page.
• How does this affect LLM deployment? It suggests that memory‑constrained environments may need to stream more data from disk, impacting latency.

Why it matters

Understanding the true memory demands of LLMs is critical for deploying them on resource‑limited hardware and for designing efficient caching strategies.

Key points

• Count tables grow without plateauing even after thousands of lookups.
• Locality is driven by context recurrence, not token recurrence.
• Assumed page‑read latency was underestimated; real reads are slower.
• Eight threads dramatically cut effective paging latency.
• Optimizing cache policy or data layout may be the only way to reduce resident memory.
• The study validates the need to measure assumptions in ML system design.

Frequently asked questions

What is a count table?

A data structure that stores token frequencies used by LLMs to predict the next word.

Why does the working set keep growing?

Because each new context key is unique, the table continually loads new pages into memory.

Can we reduce the resident footprint?

Possible approaches include better caching policies or clustering contexts on the same page.

How does this affect LLM deployment?

It suggests that memory‑constrained environments may need to stream more data from disk, impacting latency.

Reporting drawn from

• dev.to