Local AI Model Tracker (2026)

Local AI Model Tracker (2026)

A "local AI model tracker" is a running shortlist of open-weight model releases and runner updates worth retesting on your own hardware, so you're not chasing every Hugging Face drop blindly. The short version for 2026: start with a small GGUF quant of whatever family you trust, confirm it actually fits and runs fast on your GPU, then scale model size only after you've verified the VRAM math holds. Everything below is the watchlist I keep open, plus the decision rules I use to decide what's actually worth downloading.

This page is the cornerstone for the WikiWayne local-AI cluster. When a news post here covers a new quant recipe or a GGUF release, it links back here. I bump the updatedAt date whenever a runner ships a breaking change or a model family drops a version worth your bandwidth.

What is a local AI model tracker?

A local AI model tracker is a curated list of open-weight LLMs and inference runners, tagged by how well they run on consumer hardware. It is not a leaderboard. Leaderboards rank models on benchmarks run on datacenter GPUs; a tracker tells you which quant fits your 12GB card and whether the runner you use already supports it. Open-weight means the model's weights are published for download (Qwen, Llama, Gemma, DeepSeek, Mistral, GLM, Phi), so you can run them offline without an API key.

The distinction matters because a model topping a benchmark at full BF16 precision on an H100 is useless to you if the only quant that fits your GPU is a brain-damaged 2-bit version. The tracker question is always: does a usable quant of this run well on my actual box?

Which open-weight model families should I watch in 2026?

These are the families I retest first whenever a new version lands. Sizes and use cases are ballpark; verify speed and fit on your own stack before committing.

Family	Sizes you'll see (GGUF)	Sweet spot	Notes
Qwen	0.5B–72B, plus MoE variants	General chat, coding, multilingual	Strong all-rounder; the 7B/14B quants are my daily drivers
Llama	8B, 70B class	General assistant, broad tooling support	Best ecosystem support; nearly every runner handles it first
Gemma	~2B, 9B, 27B class	Efficient mid-size, good on Apple Silicon	Punches above its size; 27B is a strong single-GPU pick
DeepSeek	7B up to large MoE	Reasoning, code	MoE versions need lots of RAM/VRAM but only activate a slice
Mistral	7B, plus Mixtral MoE	Fast, lean, great base for fine-tunes	The 7B is still a great low-VRAM workhorse
GLM	9B and up	Bilingual, agentic tasks	Worth testing if you do tool-calling workflows
Phi	~3B–14B class	Small-footprint reasoning, CPU-friendly	Good when VRAM is tight or you're CPU-only

If you only have time to track one thing: watch the Qwen and Llama GGUF repos on Hugging Face. They get quantized fastest and have the widest runner support.

How do I read a model release to know if it's worth my time?

Three signals tell me whether to download or skip:

• Is there a GGUF yet? GGUF is the single-file quantized format that llama.cpp, Ollama, LM Studio, and KoboldCpp all consume. No GGUF means waiting for the community to quantize it, or doing it yourself. See what is GGUF if the format is new to you.
• What's the parameter count and architecture? A dense 14B and a 30B MoE have very different memory profiles. MoE models load all experts into memory but only compute a few per token, so they're RAM-hungry but can be fast.
• Is the license actually open? "Open-weight" doesn't always mean "do whatever." Check the license file before you build anything commercial on it.

If GGUF exists and the license fits, I pull the smallest sensible quant first and benchmark before scaling up.

What quantization should I download first?

Quantization is compressing model weights to fewer bits per parameter to shrink memory use, trading a little quality for a lot less VRAM. Q4_K_M (roughly 4-bit) is the default I reach for: it cuts memory to about a quarter of full precision while keeping quality close enough for most work. Q8 (8-bit) is near-lossless but roughly double the size of Q4.

My rule of thumb:

• If you're tight on VRAM, start at Q4_K_M. It's the best size-to-quality ratio for most local work.
• If quality matters more than fitting a bigger model, go Q5_K_M or Q6_K. Noticeably better, still much smaller than Q8.
• If you have headroom and want max fidelity, use Q8_0. Beyond Q8 the returns are tiny.
• Avoid 2-bit and 3-bit quants unless you have no other option. They degrade fast, especially on smaller models.

I dig into the tradeoffs in Q4 vs Q8 quant quality and the broader quantization explainer.

How much VRAM do I actually need?

Rough math: a model's GGUF file size on disk is close to its base memory footprint, then add a bit for the KV cache, which grows with context length. So a 7B model at Q4_K_M lands around 4–5GB of weights, and you want a few GB of headroom on top for context and overhead.

A practical sizing guide, all approximate:

Model size	Q4_K_M weights (approx)	Comfortable GPU VRAM
3B	~2GB	6GB+
7–8B	~4–5GB	8GB+
13–14B	~8–9GB	12GB+
27–32B	~18–20GB	24GB+
70B	~40GB	2x 24GB or 48GB, or heavy offload

These are ballparks — your context length, runner, and OS overhead all move the number, so verify on your own card. The full method is in the VRAM requirements guide and the focused how much VRAM for Llama 3 8B. If a model doesn't fully fit, you can split layers between GPU and CPU — that's covered in GPU offload layers explained.

How do I pull and test a new model in five minutes?

Fastest path with Ollama — pull, run, done:

# Pull a small quant and chat with it
ollama pull qwen2.5:7b
ollama run qwen2.5:7b "Give me three test prompts to benchmark a local model."

If you'd rather grab a specific GGUF and run it directly with llama.cpp:

# Run any GGUF you downloaded from Hugging Face
./llama-cli -m ./qwen2.5-7b-instruct-q4_k_m.gguf \
  -p "Summarize why Q4_K_M is a sensible default." \
  -ngl 999   # offload all layers to GPU; lower this if VRAM is tight

LM Studio is the click-to-download route if you prefer a GUI — see downloading models in LM Studio step by step. For the absolute fastest hands-on, pull your first open-weight model in 5 minutes walks the whole loop.

To sanity-check speed, watch the tokens/sec the runner reports. Interactive feel starts somewhere around 15–20 tokens/sec for chat; below ~5 it gets painful. Your numbers depend entirely on GPU, quant, and context — measure, don't guess.

Which runner should I track for updates?

The runner matters as much as the model, because a new model is useless until your runner supports its architecture. Quick guide:

• If you want the simplest install and a model registry, use Ollama. New tags show up fast. Start with install Ollama on Windows, Mac, or Linux.
• If you want a GUI with a built-in model browser, use LM Studio.
• If you want maximum control and the newest architectures first, build llama.cpp from source. See the CUDA build quickstart.
• If you're on Apple Silicon and want native speed, also track MLX alongside GGUF — MLX on Apple Silicon.

llama.cpp usually gains support for brand-new model architectures first, and Ollama follows shortly after since it builds on llama.cpp. So if a fresh release won't load in Ollama yet, a llama.cpp update often unblocks it. The full comparison lives in LM Studio vs Ollama vs llama.cpp.

What's on the June 2026 watchlist?

The recurring things I retest each refresh cycle:

• New Qwen, Llama, Gemma, and DeepSeek quants landing on Hugging Face — pull the Q4_K_M first.
• Ollama registry tag updates, especially for any model family that just shipped a new version.
• llama.cpp releases that add a new model architecture or fix a quant kernel.
• LM Studio server and OpenAI-compatible API compatibility notes.
• MLX community ports of popular models for Apple Silicon users.

When this watchlist changes, I bump updatedAt and log the edit so the cluster stays current.

Do I need a GPU for any of this?

No, but it changes what's comfortable. CPU-only inference works fine for smaller quants and privacy experiments — it's just slower, especially past 7B. If you're running headless or privacy-first, the CPU-only local LLM privacy tradeoff lays out the limits. For GPU buyers, start with the best GPU for local AI or the budget used-GPU pick.

Bottom line

Don't chase every release. Track a handful of trusted open-weight families, watch for the GGUF to appear, pull the smallest sensible quant, and verify VRAM and tokens/sec on your own hardware before scaling up. Start at Q4_K_M, keep your runner updated, and let this page point you to the install steps and runner-specific commands in the rest of the cluster. Verify every number on your stack — the only benchmark that counts is the one running on your box.