Local AI Glossary. Real hardware. Zero cloud lock-in.

A neural network trained on massive text corpora to predict the next token. Modern LLMs (transformers with billions of parameters) can generate coherent text, follow instructions, reason, and assist with code and analysis. Local versions run entirely on your hardware.

Model parameters that the creators have released for public download. This is what makes local AI possible — you can run, fine-tune, or modify the model without relying on a vendor API. Not always fully open-source (training data or code may remain private).

The process of running a trained model to generate outputs from prompts. Distinct from training. All local AI usage is inference.

The basic units of text that models process (subwords, words, or characters). Roughly 1,000 tokens ≈ 750 English words. Context length, speed, and memory use are all measured in tokens.

The maximum number of tokens a model can attend to in a single pass, including the system prompt, conversation history, retrieved documents, and the response being generated. Exceeding it truncates earlier context.

The numerical weights that define the model’s behavior, usually counted in billions (7B, 14B, 32B, 70B, etc.). Larger models are generally more capable but require more memory and compute.

Fixed-length numerical vectors that represent the semantic meaning of text (or images). Used for similarity search, RAG, clustering, and classification. Local embedding models (e.g., nomic-embed, BGE) run alongside chat models.

Base models are the raw pretrained next-token predictors. Instruct (or chat) models have been further fine-tuned to follow user instructions and maintain conversational format. Almost all practical local use is with instruct models.

Persistent instructions that define the model’s persona, rules, and behavior. Usually applied at the start of every conversation via the chat template.

The structured format (often Jinja2) that wraps user messages, assistant responses, and system prompts so the model sees conversation turns correctly. Each model family has its preferred template.

Dedicated video memory on a discrete GPU. The primary limiting factor for how large a model (and how long a context) you can run locally. A 24 GB card can comfortably run many 32B Q4 models; 8–12 GB cards are limited to smaller models or heavier quantization.

Shared memory pool on Apple Silicon (and some other SoCs) that both the CPU and GPU can access without copying. Enables efficient use of the full system RAM for models via MLX or llama.cpp Metal — a major advantage for Mac users.

NVIDIA’s parallel computing platform. Required for high-performance GPU inference on NVIDIA cards with most local engines (llama.cpp, vLLM, ExLlama, etc.).

AMD’s open alternative to CUDA. Allows running many local AI stacks on AMD GPUs, though support and performance still lag NVIDIA in many tools.

Apple’s low-level GPU framework. Used by llama.cpp and MLX for acceleration on Macs.

The speed (in GB/s) at which data can be moved between memory and compute units. Token generation is usually memory-bandwidth bound rather than compute-bound, so higher bandwidth directly improves tokens/second.

Technique of placing some model layers on system RAM instead of VRAM when the GPU cannot hold the full model. Enables running larger models but significantly reduces speed.

The memory structure that stores keys and values from previous tokens so the model doesn’t recompute them. It grows linearly with context length and number of concurrent sequences and is often the second-largest VRAM consumer after the model weights.

The error that occurs when VRAM (or unified memory) is exhausted. Usually fixed by using a more aggressive quant, reducing context length, or enabling offloading.

Memory-mapping a model file so the operating system loads pages on demand rather than reading the entire model into RAM at once. Improves load times and reduces peak RAM use.

Methods for splitting a model across multiple GPUs. Tensor parallelism splits individual layers; pipeline parallelism assigns whole layers to different GPUs.

The process of reducing the numerical precision of model weights (typically from 16-bit floating point to 4–8 bits) so the model uses far less memory and runs faster, with a controlled quality trade-off.

The standard single-file format for quantized models used by llama.cpp, Ollama, LM Studio, and most local tools. It contains the weights, tokenizer, and metadata in one portable file.

The most widely recommended balanced 4-bit GGUF quantization. It offers excellent size reduction (~4.8 bits per weight average) while retaining strong quality for most use cases.

llama.cpp’s modern mixed-precision quantization scheme (the “K” in Q4_K_M, Q5_K_M, etc.) that assigns higher precision to more important tensors.

GGUF quantizations (e.g., IQ2_XS, IQ3_M, IQ4_XS) that use an importance matrix to allocate bits more carefully. Provide better quality than standard K-quants at the same or lower bit rate, at the cost of slower CPU decoding. Popular for squeezing large models onto limited VRAM.

A calibration file that records which weights are most sensitive to quantization error. Used when creating IQ-quants or refining other low-bit GGUF files. Improves quality especially below 4 bits.

Activation-aware Weight Quantization. A high-quality 4-bit method that protects important weights based on activation patterns. Often preferred for GPU serving with vLLM.

Popular post-training quantization technique that uses calibration data and error compensation to produce good 4-bit models.

16-bit floating-point formats. FP16 is the common baseline for full-precision weights; BF16 has a wider dynamic range and is often used in training.

8-bit floating-point format with hardware support on newer NVIDIA (Ada, Hopper, Blackwell) and some AMD GPUs. Offers near-FP16 quality at half the memory. Increasingly available in vLLM, SGLang, and some MLX builds.

NormalFloat 4-bit format optimized for the distribution of neural network weights. Used heavily in QLoRA and the bitsandbytes library.

Library that provides 4-bit and 8-bit quantization (including NF4) primarily for training and fine-tuning inside the Hugging Face ecosystem. Core component of QLoRA. Less common for pure inference compared with GGUF/AWQ.

On-the-fly quantization (often 4-bit or 8-bit) applied at load or runtime rather than baked into the model file. Used by bitsandbytes and some serving stacks. Convenient but usually slower or less optimized than pre-quantized formats.

A safe, fast, memory-mappable format for storing tensors. Preferred over older pickle-based formats because it cannot execute arbitrary code when loaded.

Variable-bitrate quantization formats used by ExLlama. They allow precise control over the final VRAM footprint while maximizing quality.

The average number of bits used to store each parameter after quantization. Directly determines file size and VRAM requirements (rough rule: size in GB ≈ parameters in billions × BPW / 8).

The point at which further reducing bit width causes a sharp drop in usable quality (often around 3 bits or lower for many models). Below this point the model may still produce text but loses coherence, instruction following, or factual accuracy. Larger models usually tolerate lower bits better.

The original Transformer attention mechanism in which every head has its own full set of query, key, and value projections. Highest quality but largest KV cache. Most modern models use more efficient variants instead.

Attention design where multiple query heads share a smaller number of key/value heads. Cuts KV cache size versus full multi-head attention while retaining most quality. Standard in Llama 3, Qwen2.5, and many modern models.

DeepSeek’s attention variant that compresses the key-value cache into a low-rank latent representation. Dramatically reduces KV cache memory (often 5–10× smaller) compared with standard multi-head attention, enabling longer contexts on the same hardware. Used in DeepSeek models.

Positional encoding method that applies rotations to query and key vectors based on token position. Enables strong relative position awareness and is the foundation for most modern context-extension techniques. Nearly universal in current local models.

Efficient method for extending a model’s trained context window (often 4–32×) by adjusting RoPE frequencies. Commonly used to turn a 8k–32k model into a 128k+ model with minimal quality loss on local hardware.

A RoPE-based context extension technique that scales frequencies differently across dimensions. Good for modest (2–4×) zero-shot extensions with less quality loss than simple linear interpolation. Often a component inside YaRN.

Attention mechanism that restricts each token to attending only to a fixed window of recent tokens (plus sometimes a few global tokens). Reduces KV cache growth for very long sequences. Used in Mistral and some other architectures.

The foundational open-source C/C++ inference engine that made efficient local LLMs practical on consumer hardware. It supports GGUF models, extensive quantization, and backends for CPU, CUDA, Metal, ROCm, and Vulkan. Almost every other local tool is built on or inspired by it.

A user-friendly CLI and server that wraps llama.cpp. It manages model downloads from its library, lets you create custom Modelfiles, and exposes both a simple API and an OpenAI-compatible endpoint. The most popular starting point for running local models with minimal setup.

A polished desktop application with a graphical interface for discovering, downloading, chatting with, and serving local GGUF models. Built on llama.cpp; ideal for users who prefer not to use the terminal.

Apple’s machine learning framework optimized specifically for Apple Silicon’s unified memory architecture. It delivers excellent performance for both inference and fine-tuning on Macs and is often the fastest option available on M-series chips.

A high-throughput serving engine designed for concurrent users and production workloads. It introduced PagedAttention for efficient KV-cache management and continuous batching. Best suited for NVIDIA and AMD GPUs when multiple people need to hit the same model.

A fast runtime focused on structured generation, tool calling, and agentic workflows. It offers strong performance for multi-turn conversations and constrained outputs.

Highly optimized inference engines (especially ExLlamaV2) known for outstanding speed and efficiency on NVIDIA GPUs with quantized models.

A popular open-source web-based chat interface that sits in front of Ollama, llama.cpp, or other backends. It adds features such as RAG document upload, multi-model support, and a clean ChatGPT-like experience.

An all-in-one local platform focused on document chat, RAG pipelines, and agent workflows. It bundles embedding, vector storage, and chat so you can talk to your private files with relatively little configuration.

The central hub of the open-model ecosystem. It hosts model weights, datasets, tokenizers, and the widely used Transformers library. Most local models originate here or are converted from Hugging Face repositories.

Ollama’s recipe format for customizing models. A Modelfile can set a system prompt, chat template, temperature defaults, and other parameters, then package them into a reusable model tag.

A standardized HTTP interface (especially /v1/chat/completions) that most local tools expose. It lets you point existing apps and frameworks written for OpenAI at your local model with minimal changes.

A single-file executable format that packages a model + llama.cpp into one portable binary. Useful for distributing or running models with almost zero setup on many platforms.

A memory management technique (pioneered in vLLM) that stores the KV cache in non-contiguous blocks (pages). It reduces memory waste and fragmentation, enabling higher concurrency and longer contexts.

Hugging Face’s production-oriented serving framework. Supports high-performance inference, continuous batching, and various quantization methods.

A lightweight proxy and gateway that provides a unified OpenAI-compatible interface across many different model providers (local and cloud). Useful for routing or falling back between local models and APIs.

A high-performance C/C++ port of OpenAI’s Whisper automatic speech recognition model. It runs efficiently on CPU, Apple Silicon (Metal), and GPUs, enabling fully offline local speech-to-text. Widely used for voice interfaces and transcription.

A node-based graphical interface originally built for Stable Diffusion image generation. It has become a popular local workflow tool for diffusion models, video, and multimodal pipelines that combine LLMs with image or audio processing.

Language models that can process both text and images (or other media). Local versions typically require a language model GGUF plus a separate multimodal projector file (mmproj). Examples include LLaVA, Qwen-VL, and Gemma vision variants. They allow local image description, OCR-like tasks, and visual reasoning.

The small additional model file required by many local vision LLMs (especially llama.cpp-based). It projects image features into the language model’s embedding space so the LLM can “see” the image. Must match the specific base model.

The process of running a vision-language model entirely on local hardware to describe, analyze, or answer questions about images. Common in privacy-sensitive workflows and offline setups.

A technique that retrieves relevant chunks from a knowledge base (using embeddings) and injects them into the prompt so the model can answer using your private or external documents. Reduces hallucination on domain-specific topics.

A specialized database (Chroma, Qdrant, Weaviate, LanceDB, etc.) optimized for storing and searching embeddings via approximate nearest-neighbor algorithms (commonly HNSW). Core component of local RAG systems.

Further training a pretrained model on your own data to adapt its behavior, style, or knowledge. Full fine-tuning is expensive; most local users use PEFT methods instead.

Parameter-efficient fine-tuning that freezes the base model and trains small adapter matrices. Dramatically reduces memory and compute needs compared with full fine-tuning.

LoRA performed on a quantized (usually 4-bit) base model. Makes fine-tuning large models practical on consumer GPUs.

Family of methods (LoRA, adapters, prompt tuning, etc.) that update only a small fraction of parameters. Essential for local fine-tuning.

Speed technique in which a smaller “draft” model proposes several tokens that the larger target model verifies in parallel. Can increase generation speed 1.5–3× with no quality loss.

Memory-efficient exact attention algorithm that reduces the VRAM required for attention computation, enabling longer contexts and faster training/inference.

Architecture in which the model contains multiple specialized “expert” sub-networks and only activates a subset for each token. Allows very large total parameter counts while keeping active compute lower.

Combining weights from multiple models or fine-tunes (via techniques like SLERP, TIES, or DARE) to create a single model with blended capabilities. Popular in the local community for creating specialized hybrids.

Post-training techniques that steer model behavior toward being more helpful, honest, and harmless. RLHF uses human preference data and a reward model; DPO optimizes directly on preference pairs.

When a model confidently generates incorrect, fabricated, or unsupported information. Local models exhibit the same behavior as cloud models; RAG and careful prompting help mitigate it.

Training a smaller “student” model to mimic a larger “teacher” model’s outputs. Used to create efficient local models that retain much of the capability of bigger ones.

The model’s ability to emit structured requests (usually JSON) to external tools or APIs which the runtime then executes and feeds back into context.

The formal definitions (often JSON Schema) of available tools that are provided to the model so it knows what functions exist, their parameters, and expected types. Proper schemas dramatically improve the reliability of tool use.

The ability of a model (and runtime) to generate responses that strictly follow a predefined format, such as valid JSON, XML, or a custom schema. Critical for reliable tool calling and downstream automation.

Runtime features (available in llama.cpp, Ollama, vLLM, SGLang, etc.) that force the model’s output to conform to a specific grammar or JSON schema. Prevents malformed tool calls and makes structured generation reliable.

Systems that allow models to plan, use tools, maintain state across steps, and take multi-step autonomous actions toward a goal.

An open standard for connecting models to tools, data sources, and services in a portable, secure way. Designed to make agent integrations work consistently across local and remote environments and reduce custom glue code.

Running models completely offline or on isolated networks so that no prompts or data ever leave the controlled environment. A major reason many people and organizations choose local AI.

The primary speed metric for generation. Measures how many tokens the model produces per second during the decode phase. 15–30 t/s feels responsive for interactive use.

Latency from sending the prompt until the first token of the response appears. Dominated by the prefill phase and prompt length. Critical for perceived responsiveness.

The two phases of autoregressive generation. Prefill processes the entire prompt in parallel (compute-bound). Decode generates one token at a time (usually memory-bandwidth bound).

A measure of how well a model predicts a held-out text sample (lower is better). The standard metric for measuring quantization quality loss. A Q4_K_M model might show 1–3% higher perplexity than FP16; larger jumps indicate noticeable degradation.

Serving technique that dynamically adds new requests into the batch as earlier ones finish, maximizing GPU utilization when multiple users or requests are active.

Massive Multitask Language Understanding benchmarks that test knowledge across dozens of academic subjects via multiple-choice questions. Widely used for comparing models, though results can be sensitive to prompt format.

A coding benchmark in which models write Python functions from docstrings and are scored by unit tests. Useful for judging coding ability of local models.

A long-context test that hides a specific fact (“needle”) inside a long document and asks the model to retrieve it. Measures whether the model can actually use its full claimed context window. Frequently used to evaluate context-extension techniques such as YaRN.

Meta’s open-weight models. Strong general performance, excellent tooling support, and the foundation for a huge ecosystem of fine-tunes and merges. Llama 3.x series uses GQA and is widely available in high-quality GGUF and MLX quantizations.

Alibaba’s models. Frequently among the strongest open-weight models in their size class for coding, multilingual, and reasoning tasks. Excellent MLX and GGUF support; many variants include strong long-context configurations via YaRN.

Mistral’s dense models and Mixtral MoE models. Efficient, high-quality, and often competitive with larger dense models. Mixtral’s MoE design keeps active compute lower while total parameters are high.

Google’s open models. Strong performance for their size, good instruction following, and solid local support. Often preferred when you want a well-behaved smaller model.

Microsoft’s small, high-quality models. Excellent capability-per-parameter. Popular for resource-constrained local setups (laptops, edge devices) where larger models won’t fit.

Models known for strong reasoning and coding performance, often at competitive sizes. DeepSeek-V2/V3 introduced MLA and advanced MoE designs that improve efficiency. Good GGUF availability; R1-style reasoning variants are popular locally.