llama.cpp

llama.cpp is an open-source C and C++ inference engine for running large language models (LLMs) locally. Built on the ggml tensor library, it gives Linux users and developers direct control over model execution, hardware backends, and server deployment.

Use its command-line tools to run supported models, including GGUF models, or start llama-server to provide inference through an HTTP API. The server supports OpenAI-compatible clients, so you can connect local models to applications and development tools. You can also run llama.cpp in Docker, use prebuilt binaries, or build it from source.

llama.cpp can run inference on CPUs and supported accelerators, including NVIDIA GPUs with CUDA, AMD GPUs with HIP, and Apple Silicon with Metal. Vulkan and SYCL are also supported. CPU and GPU inference can work together, allowing partial GPU offload when a model does not fit entirely in available video memory. Quantized models can reduce memory use, though performance depends on the model, quantization, hardware and settings.

Main features

• Run local LLM inference from the command line or through a server.
• Load supported GGUF models, including models from Hugging Face.
• Serve models through an HTTP API with OpenAI-compatible support.
• Use CPU inference or hardware acceleration through supported backends such as CUDA, HIP, Metal, Vulkan and SYCL.
• Combine CPU and GPU resources with partial GPU offload.
• Deploy using Docker, prebuilt binaries or a source build.