basalt

Basalt is a local AI inference engine for running large mixture-of-experts language models on NVIDIA Blackwell workstations. It supports Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash, combining GPU memory with system RAM so the model's experts do not all have to fit in VRAM.

It is aimed at Linux users and developers who want to serve these models from their own hardware. Its OpenAI- and Anthropic-compatible APIs provide a familiar interface for client applications, while a built-in web interface lets you chat with the model.

Making room for large models

Basalt's defining choice is how it distributes experts, the model components selected during inference. Experts that do not fit on the main GPU can live in system RAM, with additional GPUs optionally providing another storage and compute tier. This makes the workstation's RAM capacity relevant alongside its graphics card.

The practical implication is that buying a supported GPU is only part of planning a Basalt system. You also need enough RAM for your chosen model pack and workload. Offloading reduces the requirement to hold everything in VRAM; it does not remove the memory requirement.

Models use the project's .basalt format. The model documentation describes support for chat, tool calls, reasoning controls and image input, with capabilities and settings differing between the two architectures.

Serving applications and watching resource use

Several configured models can share a single server address. The router loads one model at a time and switches when requests select another. This is useful for offering clients a choice of models, but switching involves stopping one engine and starting another, so it introduces a wait.

Concurrent requests use configurable parallel slots. By default, those slots divide the configured context capacity; giving every slot the full context requires more KV-cache memory. The stats page displays GPU load, VRAM, RAM, temperatures and generation rates, helping you understand how a workload uses the machine. Optional conversation caching on disk can preserve reusable conversation state across restarts.

For tuning, basalt-probe measures the local machine and prints suggested settings. It does not apply or save them automatically, leaving the configuration under your control.

When Basalt fits

Basalt merits consideration when you already have compatible Blackwell hardware and specifically want to run its supported MoE models locally. Its memory tiers, API interfaces and diagnostics address that focused workload.

For a wider choice of hardware backends, consider llama.cpp. For a packaged Linux installation and a workflow centered on downloading and managing models, consider Ollama. These are alternatives for local inference; support for a particular model and format still needs checking.