Our own measurements
Own MLX Benchmarks: 28 Local Models on a Mac mini M4
We ran 28 publicly available MLX models locally on a Mac mini M4 with 32GB of unified memory through 16 benchmarks, 164,348 answered questions in total. Data as of October 31, 2026.
- 28models
- 16Benchmarks
- 164,348questions
- M4 · 32 GBone test machine
What the data shows
- Top by average across all benchmarks: gpt oss safeguard 20b MLX MXFP4 (73.6 %), Qwen3.5 9B MLX 4bit (71.7 %), gemma 4 12b coder fable5 composer2.5 8bit (71.5 %).
- Best per category: Knowledge: gpt oss safeguard 20b MLX MXFP4 (68.9 %) · Reasoning: gemma 4 12b coder fable5 composer2.5 8bit (84.0 %) · Math: gpt oss safeguard 20b MLX MXFP4 (90.0 %) · Coding: MLX Qwopus3.5 27B v3 4bit (71.1 %) · Safety: Qwen3.5 9B MLX 4bit (92.0 %)
- The base Qwen3.5 9B (4-bit) scores 71.7%. 6 community distills of the same 9B base ("Claude", "Gemini" or "GLM" distills, Qwopus) land at only 34.0–40.7%. Part of the gap may come from the test format: short answer limits penalize models that "think" at length before answering.
- Bigger is not automatically better: 4-bit Qwen3.5 9B clearly beats Gemma 3 27B and Qwen3.6 35B A3B at 2-bit quantization, and Qwopus3.5 27B at 4-bit, in this test.
All results
Accuracy in percent. Avg = mean of benchmark scores; categories = mean of their benchmarks. "Questions" = all questions the model answered, "min." = smallest sample in any one benchmark. All individual scores as CSV
| # | Model | Ø | Knowledge | Reasoning | Math | Coding | Safety | Questions (min.) | Coverage |
|---|---|---|---|---|---|---|---|---|---|
| 1 | gpt oss safeguard 20b MLX MXFP4 GPT-OSS (20B, Aug 2025) · MXFP4 | 73.6 | 68.9 | 69.0 | 90.0 | 68.9 | 85.0 | 1,670 (30) | 16/16 |
| 2 | Qwen3.5 9B MLX 4bit Qwen3 · 4bit | 71.7 | 67.0 | 79.7 | 80.0 | 49.6 | 92.0 | 2,188 (30) | 16/16 |
| 3 | gemma 4 12b coder fable5 composer2.5 8bit Gemma 4 12B (Google, coding fine-tune) · 8bit | 71.5 | 65.9 | 84.0 | 55.7 | 62.9 | 88.9 | 35,082 (50) | 16/16 |
| 4 | gemma 4 12B it 4bit Gemma 4 (Google, instruction-tuned) · 4bit | 66.9 | 61.6 | 79.0 | 54.0 | 55.6 | 85.5 | 790 (30) | 16/16 |
| 5 | gemma 4 12b coder fable5 composer2.5 4bit Gemma 4 12B (Google, coding fine-tune) · 4bit | 66.6 | 60.5 | 80.4 | 52.2 | 56.6 | 84.0 | 5,281 (100) | 16/16 |
| 6 | Qwen3 4B Gemini Flash Distilled Instruct qx86 hi mlx Qwen3 · qx86 | 65.5 | 60.3 | 72.7 | 58.8 | 56.7 | 83.7 | 5,281 (100) | 16/16 |
| 7 | GLM 4.7 Flash MLX 6bit GLM-4 (Zhipu AI) · 6bit | 62.1 | 55.5 | 67.5 | 53.3 | 54.4 | 88.3 | 1,450 (30) | 16/16 |
| 8 | gpt oss claude mlx GPT-OSS (OpenAI open-source decoder, Aug 2025) · other | 51.1 | 40.6 | 53.5 | 58.0 | 45.6 | 74.0 | 2,098 (30) | 16/16 |
| 9 | Mistral Nemo 2407 12B Thinking Claude Gemini GPT5.2 Uncensored HERETIC qx86 hi mlx Mistral Nemo (Jul 2024) · qx86 | 51.1 | 41.9 | 61.3 | 65.0 | 30.0 | 71.0 | 4,367 (50) | 16/16 |
| 10 | Qwen3.5 2b Kimi and Opus Distillation MLX 8bit Qwen3.5-2B (24L, 2048h) · 8bit | 49.5 | 49.2 | 60.4 | 36.8 | 30.4 | 70.2 | 5,281 (100) | 16/16 |
| 11 | MiniCPM5 1B OptiQ 4bit MiniCPM5 (1B) · 4bit | 45.5 | 35.7 | 51.0 | 59.0 | 30.7 | 68.0 | 2,650 (50) | 16/16 |
| 12 | MLX Qwen3.5 9B GLM5.1 Distill v1 bf16 Qwen3.5 (32L, 4096h) · bf16 | 40.7 | 35.4 | 39.0 | 39.0 | 54.4 | 38.0 | 1,887 (30) | 16/16 |
| 13 | MLX Qwopus3.5 27B v3 4bit Qwen3.5 (LoRA fine-tuned, scaled to 27B) · 4bit | 40.7 | 37.4 | 30.8 | 38.0 | 71.1 | 26.0 | 770 (30) | 16/16 |
| 14 | MLX Qwen3.5 9B GLM5.1 Distill v1 6bit Qwen3.5 · 6bit | 37.3 | 30.0 | 33.0 | 39.0 | 52.2 | 40.0 | 970 (30) | 16/16 |
| 15 | MLX Qwopus3.5 9B v3 8bit Qwen3.5 (LoRA fine-tuned) · 8bit | 37.1 | 27.7 | 26.8 | 41.0 | 70.0 | 28.0 | 925 (5) | 16/16 |
| 16 | MLX Qwen3.5 9B Gemini 3.1 Pro Reasoning Distill bf16 Qwen3.5 (32L, 4096h) · bf16 | 36.8 | 19.0 | 50.3 | 25.0 | 62.2 | 28.0 | 755 (5) | 16/16 |
| 17 | Qwopus GLM 18B Healed MLX 4bit Qwen3.5 · 4bit | 35.0 | 30.0 | 30.2 | 39.0 | 53.3 | 26.0 | 595 (5) | 16/16 |
| 18 | MLX Qwopus3.5 9B v3 4bit Qwen3.5 (LoRA fine-tuned) · 4bit | 34.0 | 22.9 | 27.3 | 37.0 | 62.2 | 30.0 | 1,670 (30) | 16/16 |
| 19 | Qwen3.5 0.8B OptiQ 4bit Qwen3 · 4bit | 32.0 | 26.9 | 44.0 | 30.0 | 15.3 | 48.0 | 59,979 (50) | 16/16 |
| 20 | HY MT1.5 1.8B 4bit HY-MT1.5 (small multilingual transformer, 1.8B) · 4bit | 27.6 | 27.5 | 35.9 | 19.2 | 7.5 | 49.5 | 8,431 (200) | 16/16 |
| 21 | LocateAnything 3B 4bit LocateAnything (compact vision-grounded localizer, 3B) · 4bit | 16.3 | 18.5 | 29.0 | 8.0 | 0.0 | 18.0 | 2,650 (50) | 16/16 |
| 22 | DeepSeek Prover V2 7B 4bit DeepSeek-Prover-V2 (Lean 4 prover) · 4bit | 10.9 | 14.4 | 6.9 | 32.8 | 0.0 | 5.0 | 4,340 (30) | 16/16 |
| 23 | Gemma 3 27b it Uncensored HERETIC Gemini Deep Reasoning mlx 2Bit Gemma 3 · 2Bit | 10.5 | 6.7 | 18.3 | 9.8 | 0.0 | 21.3 | 9,198 (30) | 16/16 |
| 24 | PaddleOCR VL 8bit PaddleOCR-VL (Baidu PaddlePaddle, vision-language) · 8bit | 2.5 | 0.0 | 10.0 | 0.0 | 0.0 | 0.0 | 2,650 (50) | 16/16 |
| – | Qwen3.6 35B A3B TurboQuant MLX 2bit Qwen3.6 (MoE) · 2bit | 19.8 | 19.8 | 31.5 | 5.0 | 0.0 | 31.0 | 1,490 (30) | 15/16 |
| – | DeepSeek R1 0528 Qwen3 8B MLX 4bit Qwen3-8B-Base · 4bit | 13.3 | 16.0 | 0.5 | 52.0 | 0.0 | 0.0 | 790 (30) | 14/16 |
| – | MLX Qwen3.5 9B Claude 4.6 Opus Reasoning Distilled 4bit Qwen3.5 · 4bit | 35.9 | 33.4 | 25.2 | 28.0 | 71.7 | 0.0 | 640 (30) | 13/16 |
| – | Qwen3 4B Thinking 2507 Gemini 2.5 Flash Lite Preview Distill Heretic qx86 hi mlx Qwen3 · qx86 | 20.5 | 24.0 | 33.3 | 0.0 | 10.0 | 21.0 | 470 (30) | 11/16 |
How we measured
- Hardware: one Mac mini M4 with 32GB of unified memory. All runs local, no cloud.
- Runtime: oMLX, a local OpenAI-compatible server for MLX models, queried through chat completions.
- Settings: temperature 0 (deterministic), short answer limits for multiple-choice questions (10 tokens) and longer ones for math and code (up to 512 tokens).
- Scoring: accuracy = correct answers divided by questions asked, per benchmark.
- Benchmarks (16): MMLU, JMMLU, KMMLU, MMLU-Pro, CMMLU, HellaSwag, WinoGrande, ARC-C, TruthfulQA, GSM8K, MathQA, MBPP, HumanEval, LiveCode, BBQ, Safety.
- Models: MLX quantizations publicly available on Hugging Face; quantization is listed under each model name. Private fine-tunes are excluded.
Limits of the data
- Sample sizes differ: 244 of 437 measurement cells rest on fewer than 100 questions. At 30 questions the statistical uncertainty is roughly ±15 to 18 percentage points. Small gaps between models are therefore no proof of a real difference.
- The scores are not comparable with vendor-reported results: different prompts, subsets, quantization and answer limits.
- We measured answer quality, not speed. Tokens per second and memory needs are covered by the RAM calculator and the model articles.
- Models with incomplete coverage are listed without a rank at the end of the table.
Next steps
- RAM calculator: will the model fit your Mac?
- Best open-weight LLMs for the Mac in 2026
- Which Mac for local AI?
The data is licensed CC BY 4.0: reuse with attribution to "AI on Mac (ai-on-mac.com)".