Domain 4 — Evaluation, Testing and Optimisation (16%)

16% of the exam, roughly 10 questions. The domain that most reliably separates Professional candidates from Foundations ones.

4.1 Evals are the deliverable, not the afterthought

At this level an eval suite is part of the architecture. Without one you cannot tell whether a prompt change helped, whether a model version regressed anything, or whether quality is drifting. The three design principles: make them task-specific and representative of the real distribution including edge cases; automate so they run on every change; and favour volume over hand-crafted perfection, because a hundred cases with slightly noisy grading beats a dozen beautifully graded ones. Production systems want a thousand or more.

For an LLM judge: give a specific empirical rubric rather than "is this good", fix the output format so it parses, and let the judge reason before scoring. A judge you have not checked against human labels is an unvalidated instrument, and building on it is a professional error the exam will punish.

4.2 Success criteria that survive a stakeholder conversation

Specific, measurable, achievable and relevant. "Accurate classification" is not a criterion; "macro-F1 at or above 0.88 on the labelled holdout set, with escalation on anything below the confidence threshold" is. Most real systems need several criteria at once — quality, latency, cost per transaction, escalation rate — because optimising one alone reliably damages another.

4.3 Changing things safely in production

• Offline evals first. Run the suite against the change before anything reaches users.
• A/B or shadow testing next. Shadow running — the new configuration processes real traffic without its output being served — gets you production distribution with no user risk, which is the right answer when the stem worries about exposure. A/B splitting gives you real user outcomes when you can afford the exposure.
• Change one variable. Prompt, model and retrieval configuration moved together produce a result you cannot attribute.
• Watch cost and latency alongside quality. An accuracy win that doubles cost per request is a decision, not an improvement, and it needs to be taken deliberately.

4.4 Monitoring a live system

Quality degrades without anything breaking. The signals to watch are behavioural rather than infrastructural: escalation and human-override rates, validation failure rates, guardrail trigger rates, retrieval hit rates, refusal rates, distribution shift in the incoming requests, and cost per transaction. A rise in human overrides is often the earliest honest indicator that the system is drifting.

Sample real traffic into a periodically re-graded eval set, so the suite tracks what users actually send rather than what they sent when it was built.

4.5 Diagnosing a production problem

Isolate the layer before you change anything:

Retrieval is the layer people skip, and in a RAG system it is where a surprising share of "the model is hallucinating" reports actually originate: the model answered faithfully from chunks that never contained the answer.

← Previous: Domain 3 — Integration  |  Next: Domain 5 — Governance, Safety and Risk →