GGUF

GGUF is a file format that bundles model weights, metadata, and tokenizer data for efficient loading in local inference engines.

GGUF replaced the older GGML layout for most community models on Hugging Face. When you download a *.gguf file, you are choosing both the base model and the quantization profile encoded in the filename.

Local apps map GGUF tensors to GPU or CPU backends. Matching the quant to your VRAM budget is the fastest way to avoid OOM errors on first load.