Why Local LLMs Don't Need C++ or Python: Building a 15MB Native AOT Inference Engine in . NET 10

A new .NET 10 inference engine called Glacier bypasses traditional C++ runtimes and CUDA toolkits, using pure C# to run large language models directly on GPUs. By eliminating external binaries, reducing PCIe traffic, and leveraging Direct3D 12 compute, it achieves comparable or better performance t…

Sun, 13 Sep 2026 10:27:57 GMT

For years, the AI community has treated C++ and CUDA as the only viable paths to high‑performance local language‑model inference. The prevailing belief was that managed languages like C# had to rely on heavy, multi‑gigabyte toolchains and native DLL wrappers to reach the speed of dedicated GPUs. Glacier, a new .NET 10 inference engine, overturns that assumption by running entire models in pure C# while still matching or exceeding the speed of native C++ solutions.

How Glacier Eliminates the CUDA Runtime Overhead

Typical C# inference pipelines call into the CUDA runtime through cudart64.dll and cublas64.dll. Glacier sidesteps these layers entirely, communicating directly with the Windows GPU driver via low‑level P/Invoke calls to nvcuda.dll. By compiling pre‑built fat binaries that contain code for multiple SM architectures (sm_75, sm_80, sm_86, sm_89, sm_90), the engine keeps all de‑quantization multipliers and accumulators in registers, avoiding any DRAM stack spills. The result is a single, 15 MB native AOT executable that replaces a 4.5 GB CUDA toolkit and associated binaries.

Reducing PCIe Bottlenecks with In‑VRAM Argmax Reduction

Standard inference workflows perform greedy token selection by transferring a 152 k‑vocabulary logit tensor from GPU to CPU every step—a 608 KB transfer that can stall the PCIe bus. Glacier fuses the final linear projection and reduction into a single warp‑shuffle kernel that runs entirely on the GPU. The kernel outputs only a 4‑byte token ID back to the host, cutting the transfer time from hundreds of microseconds to roughly 3 µs per token.

Direct3D 12 Compute for Integrated GPUs

Running inference on laptop GPUs that double as display adapters often triggers Windows Timeout Detection and Recovery (TDR), resetting the driver. Glacier implements a pure Direct3D 12 compute engine using the Vortice.D3D12 wrapper and HLSL Wave32 shaders. This approach keeps command dispatches small, cooperates with the Desktop Window Manager, and maps weights directly into unified LPDDR5X memory on AMD RDNA 3.5 or Intel Arc GPUs, eliminating the need for PCIe staging copies.

Speculative Decoding to Beat Bandwidth Limits

Autoregressive generation is memory‑bandwidth bound: each token requires streaming the entire model. On a 256 GB/s GDDR6 bus, a 4.68 GB model can theoretically deliver only ~55 tokens per second. Glacier introduces a speculative decoding engine that proposes up to four candidate tokens in under a microsecond and verifies them in a single batch pass. This technique reduces the number of full‑model passes from K to one, boosting generation rates to 72–104 tokens per second on an RTX 4060 laptop GPU.

Head‑to‑Head Benchmarks

• Cold start to first token: 1.5 s for Glacier vs. 3.5 s for a Go‑based C++ CUDA daemon.
• Serial generation speed: 41.9 tokens/s (208 GB/s) for Glacier vs. 43.2 tokens/s (216 GB/s) for the daemon.
• Speculative generation: 72.5–104.8 tokens/s for Glacier, with no comparable native baseline.
• PCIe transfer overhead: 3.2 µs in‑VRAM vs. ~800 µs host‑to‑device.
• Integrated GPU performance: 21.68 tokens/s on an AMD Radeon 890M, beating 0.89 tokens/s from a CPU‑based AVX‑512 implementation.

These results show that a managed, pure‑C# engine can deliver competitive performance while dramatically simplifying deployment. Glacier’s single‑file, 15 MB binary eliminates the need for large CUDA toolchains, reduces distribution complexity, and keeps the inference pipeline stable on consumer hardware.

Why It Matters

By proving that managed languages can handle the low‑level demands of LLM inference, Glacier opens the door for developers who prefer .NET ecosystems to build and deploy high‑performance AI applications without wrestling with C++ or CUDA.

Key Takeaways

• Direct driver P/Invoke removes CUDA runtime bloat.
• In‑VRAM warp‑shuffle reduces PCIe traffic to 4 bytes per token.
• Direct3D 12 compute keeps integrated GPUs from timing out.
• Speculative decoding pushes token rates beyond theoretical bandwidth limits.
• All of this fits into a single 15 MB native AOT executable.

FAQ

• What hardware does Glacier support? NVIDIA, AMD, and Intel GPUs that expose a Windows nvcuda.dll driver, including integrated GPUs on laptops.
• Does it require CUDA toolkits? No. Glacier communicates directly with the GPU driver and does not depend on CUDA runtime libraries.
• Is the engine open source? Yes. The full code, benchmarks, and CLI binaries are available on GitHub under the name Glacier.Inference.

Why it matters

Glacier demonstrates that managed languages can achieve bare‑metal inference performance, simplifying deployment and expanding the developer base that can build and run large language models locally.

Key points

• Direct driver calls eliminate CUDA runtime overhead
• In‑VRAM reduction cuts PCIe traffic to 4 bytes per token
• Direct3D 12 compute avoids TDR on integrated GPUs
• Speculative decoding boosts token rates beyond bandwidth limits
• Single 15 MB binary replaces multi‑gigabyte toolchains

Frequently asked questions

What hardware does Glacier support?

NVIDIA, AMD, and Intel GPUs that expose a Windows nvcuda.dll driver, including integrated GPUs on laptops.

Does it require CUDA toolkits?

No. Glacier communicates directly with the GPU driver and does not depend on CUDA runtime libraries.

Is the engine open source?

Yes. The full code, benchmarks, and CLI binaries are available on GitHub under the name Glacier.Inference.

Reporting drawn from

• dev.to