Domain 8 — Eval, Testing and Debugging (2.6%)
2.6% of the exam, roughly 1 question. Do not over-invest — but the thinking here is genuinely useful for the rest of the paper.
8.1 Isolating a failure
The first question for any misbehaving LLM feature is which layer broke. Three candidates, and they need completely different fixes:
| Layer | Signals | Fix lives in |
|---|---|---|
| Integration | HTTP errors, malformed requests, missing tool_result blocks, unhandled stop_reason, timeouts | Your code |
| Model output | Well-formed responses whose content is wrong, incomplete or off-format | The prompt, the context, the schema, or the model tier |
| Tool execution | Correct tool chosen, wrong arguments or a failing downstream system | The tool schema, or the tool itself |
Logging the full request and response, with the request_id, is what makes that
distinction possible after the fact. A stack trace alone rarely tells you which layer was at fault.
8.2 Building evals
Three principles: make them task-specific and representative of your real distribution including edge cases; automate so you can run them often; and prefer volume over hand-crafted perfection, because more cases with slightly noisier grading beats a handful of beautifully graded ones. Aim for a hundred or more test cases, and more for anything in production.
| Grading method | Character | Use for |
|---|---|---|
| Code-based | Fastest, most reliable, least nuanced | Exact match, string containment, schema validity, numeric tolerance |
| Model-based (LLM as judge) | Fast, flexible, scalable; must itself be validated | Tone, coherence, safety, whether an answer covers required points |
| Human | Highest quality, slowest, most expensive | Validating the other two, and genuinely ambiguous cases |
For LLM-as-judge, give a specific empirical rubric rather than "is this good", fix the output format to something parseable, and let the judge reason before scoring. Success criteria should be specific, measurable and tied to what the application is for — "accurate sentiment classification at 90% or better on our labelled set", not "performs well".
8.3 Regression discipline
Prompt changes, model version changes and tool schema changes are all deployments. Run the eval suite against each before shipping, and keep the suite in version control beside the prompts it grades. This is the concrete reason to pin an explicit model version rather than a floating alias: you want the model to change when you decide it does.
← Previous: Domain 7 — Claude Code | Next: The 13 anti-patterns →
Explore Other Certification Study Guides
Switch to another exam guide to prepare for your next Claude certification:
Claude Certified Associate — Foundations
Entry-level certification for non-coders: prompting, output evaluation, model selection, workflow integration, configuration, governance & responsible use.
Claude Certified Architect — Foundations
Core architectural principles: agentic loops & orchestration, tool design & MCP, Claude Code workflows, prompt engineering, and context reliability.
Claude Certified Architect — Professional
Advanced enterprise architecture: solution design, system integration, evaluation frameworks, governance, safety & developer enablement.