🐺🐦‍⬛ LLM Comparison/Test: 25 SOTA LLMs (including QwQ) through 59 MMLU-Pro CS benchmark runs

Introduction

I've been working tirelessly on my latest research project, comparing 25 state-of-the-art large language models by running them through the respected MMLU-Pro benchmark's computer science category. This involved:

• 59 separate benchmark runs
• Over 70 hours of total runtime
• Testing 25 different LLMs, including:
  • Latest models from Anthropic, Google, Alibaba, OpenAI, Mistral, Meta, and others
  • Multiple model sizes (parameters and quantization)
  • With and without speculative decoding (a technique that can speed up inference without compromising output quality)

The goal was to thoroughly and systematically evaluate these models to:

1. Determine which performs best on computer science tasks as a proxy for general intelligence
2. Compare open vs closed source models
3. Analyze the impact of model size and quantization choices
4. Measure the benefits of speculative decoding for inference speed
5. Provide a detailed analysis of the results (and surprises!)

I started this project in November and have been continuously expanding the models tested while updating my findings.

The release of QwQ particularly caught my attention, as this unique model demonstrated exceptional performance in preliminary testing, warranting deeper analysis and more extensive evaluation.

While I could continue refining and expanding this research indefinitely, I've chosen to consolidate my key findings into a focused blog post that reflects the current state of my research. This analysis represents one of the most comprehensive evaluations of large language models to date, providing valuable insights for researchers and practitioners looking to assess these models for their specific needs or implement them in real-world applications.

About the Benchmark

The MMLU-Pro benchmark is a comprehensive evaluation of large language models across various categories, including computer science, mathematics, physics, chemistry, and more. It's designed to assess a model's ability to understand and apply knowledge across a wide range of subjects, providing a robust measure of general intelligence. While it is a multiple choice test, instead of 4 answer options like in its predecessor MMLU, there are now 10 options per question, which drastically reduces the probability of correct answers by chance. Additionally, the focus is increasingly on complex reasoning tasks rather than pure factual knowledge.

For my benchmarks, I currently limit myself to the Computer Science category with its 410 questions. This pragmatic decision is based on several factors: First, I place particular emphasis on responses from my usual work environment, since I frequently use these models in this context during my daily work. Second, with local models running on consumer hardware, there are practical constraints around computation time - a single run already takes several hours with larger models, and I generally conduct at least two runs to ensure consistency.

Unlike typical benchmarks that only report single scores, I conduct multiple test runs for each model to capture performance variability. This comprehensive approach delivers a more accurate and nuanced understanding of each model's true capabilities. By executing at least two benchmark runs per model, I establish a robust assessment of both performance levels and consistency. The results feature error bars that show standard deviation, illustrating how performance varies across different test runs.

The benchmarks for this study alone required over 70 hours of runtime. With additional categories or runs, the testing duration would have become so long with the available resources that the tested models would have been outdated by the time the study was completed. Therefore, establishing practical framework conditions and boundaries is essential to achieve meaningful results within a reasonable timeframe.

Best Models

While what's best depends on the specific use case, these benchmarks offer a comprehensive overview of the current state-of-the-art in Large Language Models. Let's examine the graph at the top of this page highlighting the performance comparison among leading models:

The graph ranks models by their average score, with error bars indicating standard deviation. "Online" models are exclusively accessible through API providers such as Anthropic, Google, or OpenAI, while "Local" models can be downloaded directly from Hugging Face and run on your own hardware. The "Both" category indicates that these LLMs are available for both local deployment and through cloud API services like Azure, IONOS (a German provider especially relevant for GDPR-compliant applications requiring national cloud infrastructure), or Mistral.

1. Claude 3.5 Sonnet (20241022) stands out as the current top performer, which perfectly matches my hands-on experience. I've continuously used both versions of Claude 3.5 Sonnet (the original 20240620 and the updated 20241022) since their respective launches and consistently find it to be the most reliable and versatile solution across diverse applications. Based on its exceptional performance, I recommend it as the go-to model for most of my clients, provided online models are an option.

2. Gemini 1.5 Pro 002 demonstrates excellent performance, ranking second overall. While Google's latest experimental models reportedly achieve even better results, rate limits during benchmark testing prevented a proper evaluation of their capabilities.

3. QwQ 32B Preview is the best local model, surpassing many online models in performance. This is as amazing as it is surprising, as it's only a (relatively) small 32B model but outperforms all other local models in these benchmarks, including much larger 70B, 123B, or even 405B models. It even surpasses the online models from OpenAI (I could only test ChatGPT/GPT-4o) as well as the excellent Mistral models (which have always been among my personal favorites due to their outstanding multilingual capabilities).

   The graph shows QwQ 32B Preview in various configurations with different settings and parameters. The 8.0bpw (8.0 bits per weight) version performs best (it's the largest available in EXL2 format), provided - and this is a major finding - the model is given enough room (max_tokens=16K) to "think"! This is QwQ's unique ability: It's capable of using chain of thought and self-reflection to arrive at the correct answer, without being specifically prompted to do so.

   Consequently, QwQ performs worse in MMLU-Pro (and likely other benchmarks) if its output is truncated prematurely, which can easily happen with smaller output limits - MMLU-Pro's default is max_tokens=2K! This affects smaller quants more severely, as they aren't as intelligent as the 8.0bpw version and need to think longer (i.e., write more tokens) to arrive at the correct answer.

4. Athene V2 Chat is another excellent model, but it's not as stable as QwQ 32B Preview at 8-bit with max_tokens=16K. Its highest score slightly surpasses QwQ 32B Preview's, but QwQ is more consistent and has less variance, ranking higher in the graph based on the average score. It's also a 72B model, so much larger than QwQ 32B Preview.

5. Qwen 2.5 72B Instruct, from the same Alibaba team behind QwQ, performs exceptionally well. Even quantized down to 4.65bpw to fit my 48 GB VRAM, it outperforms most other models in these benchmarks. The Qwen team is clearly leading in open-weights models, ahead of Meta and Mistral.

6. GPT-4o (2024-08-06) appears lower than expected, and surprisingly, this older version performed better in the benchmark than the latest ChatGPT version or its more recent iteration (2024-11-20).

7. Mistral Large 2407, a 123B model, follows GPT-4o. Like GPT-4o, this older version outperformed the latest version (2411) in the benchmark. This raises questions about whether newer models are trading intelligence for better writing or speed.

8. Llama 3.1 405B Instruct (FP8) is the next best local model. As the largest local model, its performance falls short of expectations, especially considering the resources it requires to run.

9. Mistral Large 2411, the newer version, slightly trails its older counterpart. While I appreciate their models for their excellent writing and multilingual capabilities, Qwen has taken the lead, especially considering Mistral's size and research-only license.

10. ChatGPT-4o (latest) is the API version of the current ChatGPT website model. Its benchmark was conducted on 2024-11-18, using the version available at that time.

    Online models can be updated at any time, making versioned models a more reliable choice. Even with versioned models, providers may still modify parameters like quantization, settings, and safety guardrails without notice. For maximum consistency and full control, running models locally remains the only option!

11. GPT-4o (2024-11-20) is the latest version of GPT-4o. Again, it's curious that a newer version shows lower benchmark performance compared to its predecessor. Looks like they traded quality for speed.

12. Llama 3.1 70B Instruct is the next best local model. As a 70B model, it's relatively large but still runnable locally, especially when quantized. However, this benchmark used an online, unquantized version, representing its maximum performance.

13. Gemini 1.5 Flash 002, Google's compact model, delivers performance that reflects its smaller size - falling short of its Pro counterpart. Nevertheless, it impressively outperforms Meta's Llama 3.2 90B, demonstrating that smaller models can achieve remarkable results.

14. Llama 3.2 90B Vision Instruct represents Meta's multimodal evolution of Llama - essentially an enhanced Llama 3.1 70B with integrated vision capabilities. While its performance varies slightly, it maintains comparable effectiveness to the 70B version.

15. Qwen Coder 32B Instruct is another outstanding model in the Qwen family, specifically optimized for coding tasks. While it shares the same 32B parameter size as QwQ 32B Preview, it scores lower on this benchmark. This difference in performance is natural, as computer science knowledge and coding capabilities are distinct skill sets - specialized models often show reduced performance in broader domains outside their focus area.

16. Mistral Small 2409, a 22B parameter model, ranks last behind the QwQ and Mistral Large variants in these tests. I established a minimum threshold of 50% correct answers for inclusion in this benchmark set, making it the final model to qualify for analysis.

Detailed Results

Now that we've reviewed the rankings, let's explore the detailed results and uncover additional insights. Here's the complete table:

• Model: Model name (with relevant parameter and setting details)
• HF Main Model Name: Full name of the tested model as listed on Hugging Face
• HF Draft Model Name (speculative decoding): Draft model used for speculative decoding (if applicable)
• Size: Parameter count
• Format: Model format type (HF, EXL2, etc.)
• API: Service provider (TabbyAPI indicates local deployment)
• GPU: Graphics card used for this benchmark run
• GPU Mem: VRAM allocated to model and configuration
• Run: Benchmark run sequence number
• Duration: Total runtime of benchmark
• Total: Number of correct answers (determines ranking!)
• %: Percentage of correct answers
• TIGER-Lab: Comparison between TIGER-Lab (the makers of MMLU-Pro)'s CS benchmark results and mine
• Correct Random Guesses: When MMLU-Pro cannot definitively identify a model's answer choice, it defaults to random guessing and reports both the number of these random guesses and their accuracy (a high proportion of random guessing indicates problems with following the response format)
• Prompt tokens: Token count of input text
• tk/s: Tokens processed per second
• Completion tokens: Token count of generated response
• tk/s: Tokens generated per second

Speculative Decoding: Turbocharging Large Language Models

Speculative decoding represents a groundbreaking acceleration technique for LLMs that follows a "generate first, verify later" approach. This method employs a smaller, faster draft model to make preliminary token predictions, which are then validated by the main model.

The process operates through parallel processing where the draft model generates multiple token predictions simultaneously. These predictions are then batch-verified by the main model, significantly reducing processing time compared to sequential token generation.

This innovative approach can accelerate text generation up to 3x while maintaining output quality. The effectiveness stems from batch processing, where the main model evaluates multiple predictions at once rather than processing individual tokens sequentially.

The system's performance heavily depends on prediction accuracy. If the acceptance rate of speculative tokens is too low, the system might actually perform slower than traditional processing. For optimal performance, the draft model should be architecturally similar to the main model to make accurate predictions, while being small enough to run quickly and fit in VRAM alongside the main model.

Think of speculative decoding as a turbocharger for AI - it significantly boosts LLM performance without compromising quality, creating a win-win situation for all LLM applications. The key is finding the right balance between draft model size and prediction accuracy - it needs to be lightweight enough to provide speed benefits while maintaining sufficient similarity to the main model for reliable predictions.

Speculative Decoding & Qwen/QwQ

I successfully increased the speed of Qwen2.5-Coder-32B-Instruct (8.0bpw EXL2) from 24.86 to 43.76 tokens per second by using Qwen2.5-Coder-0.5B-Instruct as its draft model.

And applying the same 0.5B draft model to QwQ-32B-Preview (8.0bpw EXL2) - a different model - improved its speed from 26.60 to 46.29 tokens per second.

No, Speculative Decoding doesn't improve the quality of the output, only the speed

Initially, I was surprised to see QwQ-32B-Preview (8.0bpw EXL2) achieving only 75% accuracy without speculative decoding, while reaching 79.02% with it. Magic? Not quite - just statistical variance! Speculative decoding only improves processing speed, not output quality. This became clear when a second benchmark run with identical settings yielded just 76.59% accuracy.

The real game-changer was adjusting the benchmark software's max_tokens parameter. Setting it to 16384 consistently improved accuracy - first to 79.27%, then to 79.02% in a second run. These stable results make perfect sense: with the higher token limit, responses weren't truncated prematurely, allowing for more reliable answer identification.

While benchmarks can yield surprising results, always verify any seemingly impossible findings before jumping to conclusions. The explanation could be either an anomaly or a technical error.

Speed Demons

gpt-4o-2024-11-20 and gemini-1.5-flash-002 emerge as the clear performance leaders in this benchmark, both achieving impressive speeds over 100 tokens per second. Notably, gemini-1.5-flash-002 demonstrated inconsistent performance across two test runs - reaching peak speed in one but dropping to 59.29 tokens per second in the other.

The latest GPT-4o release (2024-11-20) reveals a fascinating trade-off: While it delivers dramatically improved speed exceeding 100 tokens per second - more than doubling its predecessor's throughput - the enhanced performance comes at a cost. The model appears to be a quantized or distilled variant, resulting in lower benchmark scores compared to the previous version.

Beyond the Benchmarks

Benchmark results show us what LLMs can do right now, but they're just one piece of the puzzle. Different real-world uses will get different results, since benchmarks can't capture every aspect of how these models actually perform.

Benchmarks are useful for comparing models, but they're just a starting point. If you find a model that looks good, try it out yourself to see how well it works for your needs.

I'm really excited about QwQ and what's coming next. It looks like these might be the first local models that can actually go toe-to-toe with the big cloud-based ones.

I've used QwQ alongside Claude 3.5 Sonnet and GPT-4 for real work projects, and QwQ did better than both in some situations. I'm really looking forward to seeing a QwQ 70B version - with a stronger foundation and further refinement, QwQ's unique approach could give us Sonnet-level performance right on our own machines. Sounds too good to be true? Maybe, but we're getting there - and probably, hopefully, faster than we think!

Closing Thoughts

This deep dive into MMLU-Pro benchmarking has revealed fascinating insights about the current state of LLMs. From the impressive performance of Claude 3.5 Sonnet and Gemini Pro to the speed demons like GPT-4o-2024-11-20, each model brings its own strengths to the table.

We've seen how architectural choices like speculative decoding can dramatically improve performance without sacrificing quality, and how careful parameter tuning (like adjusting max_tokens) can significantly impact results. The trade-offs between speed and accuracy, particularly evident in the latest GPT-4o release, highlight the complex balancing act in LLM development.

But most excitingly, we're witnessing the rise of powerful local models like QwQ that can compete with or even outperform much larger models. This democratization of AI capabilities is a promising trend for the future of accessible, high-performance language models.

As we continue to push the boundaries of what's possible with LLMs, these benchmarks serve as valuable waypoints in our journey toward more capable, efficient, and accessible AI systems. The rapid pace of innovation in this field ensures that what seems impressive today may be baseline tomorrow - and I, for one, can't wait to see what comes next.

Wolfram Ravenwolf is a German AI Engineer and an internationally active consultant and renowned researcher who's particularly passionate about local language models. You can follow him on X and Bluesky, read his previous LLM tests and comparisons on HF and Reddit, check out his models on Hugging Face, tip him on Ko-fi, or book him for a consultation.