# How we test language models · AI in Bed

Intermediate level · 7 min · https://www.aiinbed.com/learn/intermediate/testing-language-models

*Benchmarks, leaderboards, human votes and AI judges: how anyone decides one model is better than another, and why it's harder than it looks.*

For a spam filter, testing is easy: count right and wrong answers on held-out emails. For a chatbot that writes essays and code, "right" is fuzzy. The field uses several kinds of test, each with blind spots. A test suite for models is usually called an **eval**.

### Benchmarks

A fixed set of questions with known answers: maths problems, multiple-choice exams, coding tasks with unit tests. A **benchmark** gives one comparable number.

Weakness: once a benchmark is famous, its questions leak onto the web and into training data. A model can score well by memory, not skill. This is called **contamination**.

### Human preference votes

Show people two anonymous answers to the same prompt and ask which is better. Thousands of votes produce a ranking, like chess ratings.

Weakness: people favour confident, long, nicely formatted answers, even when a shorter one is more correct.

**Quick check:** Why can a model score highly on a famous benchmark without really having the skill?

A. Benchmarks are always too easy
B. The test questions may have leaked into its training data
C. Scores are made up by companies
D. Models can see the answer key at test time

Answer: B. If the exact questions were in the training data, the model may recall answers rather than reason. Good evals keep fresh, private questions.

### AI as the judge

A strong model grades another model's answers against a rubric. It's cheap and fast, so teams can test thousands of cases on every change.

Weakness: judges have biases too, such as preferring answers that sound like themselves. Teams spot-check a sample by hand.

### Your own evals beat leaderboards

If you build with AI, collect 50 to 100 real examples from your use case with what a good answer looks like. Run every model or prompt change against them. That tells you more than any public ranking.

**A tiny eval harness**

```python
# Pretend model: answers arithmetic questions, but badly at big numbers.
def model(q):
    a, b = map(int, q.split(" + "))
    return a + b if a < 50 else a + b + 1

tests = [("2 + 2", 4), ("10 + 7", 17), ("40 + 3", 43), ("75 + 5", 80), ("99 + 1", 100)]

passed = 0
for q, expected in tests:
    got = model(q)
    ok = got == expected
    passed += ok
    print(f"{'PASS' if ok else 'FAIL'}  {q} = {got}  (expected {expected})")

print(f"score: {passed}/{len(tests)}")
```

Add your own test cases. Notice how a score alone hides where the model fails; reading the failures is where you learn.

**Quick check:** You're building a customer-support bot. What's the most useful way to compare two models?

A. Pick the one higher on a public leaderboard
B. Run both on a set of your real support questions and compare the answers
C. Choose the bigger one
D. Ask each model which is better

Answer: B. Your own representative examples measure what you actually care about. Leaderboards measure someone else's task.

**Key takeaways**

- Benchmarks give comparable numbers but can be contaminated.
- Human votes and AI judges measure quality, each with their own biases.
- For real products, a small set of your own examples is the best eval.

**Go deeper (free sources)**

- [HELM: Holistic Evaluation of Language Models](https://crfm.stanford.edu/helm/) by Stanford CRFM
- [LMArena](https://lmarena.ai/) by LMArena
- [The LLM Evaluation Guidebook](https://github.com/huggingface/evaluation-guidebook) by Hugging Face