CCAR-P questions and answers — Domain 4: Evaluation, Testing and Optimisation

10 practice questions on Evaluation, Testing and Optimisation, the domain weighted at 16% of the CCAR-P exam. Every answer explains why the distractors fail — that is the part that transfers to questions you have not seen before.

Question 1

A team wants to improve output quality and proposes rewriting the prompt. What should happen first?

1. Rewrite the prompt and compare outputs by eye
2. Build a representative eval set and establish a baseline, so the change can be measured
3. Switch to a more capable model as a control
4. Survey users about quality

Answer: B. Without a baseline there is no way to tell whether a change helped, hurt, or moved quality on one dimension while damaging another. Eye-comparison on a handful of cases is how teams ship regressions confidently.

Question 2

Which is the correct approach to an LLM-as-judge grader?

1. Use it directly; model graders are reliable at scale
2. Validate it against human labels first, with a specific empirical rubric and a parseable output format
3. Use it only for code-gradable criteria
4. Avoid it; only human grading is defensible

Answer: B. An unvalidated judge produces confident numbers about nothing. Validate against human judgement, give it a concrete rubric rather than "is this good", and fix the output format. It is then genuinely useful for nuanced criteria that code cannot grade and humans cannot grade at volume.

Question 3

A team wants to test a new retrieval configuration against production traffic without exposing users to it. Which approach?

1. An A/B test with 5% of users
2. Shadow running — the new configuration processes real traffic while its output is not served
3. A staged rollout by region
4. Testing in a staging environment with synthetic data

Answer: B. Shadow running gives you the real production distribution with zero user exposure, which is precisely what the stem asked for. A/B testing exposes some users by design; synthetic staging data does not reproduce the distribution.

Question 4

Which signal is often the earliest honest indicator that a live system's quality is drifting?

1. API error rate
2. The human override or escalation rate
3. CPU utilisation
4. Total request volume

Answer: B. Quality degrades without anything breaking, so infrastructure metrics stay flat. A rise in humans overriding or escalating the system's output is behavioural evidence that its answers have stopped being good enough.

Question 5

A change improves accuracy by two points and doubles cost per request. How should this be treated?

1. As an improvement, since accuracy is the primary metric
2. As a decision requiring explicit trade-off against the cost target, not an automatic improvement
3. As a regression
4. As neutral, since one metric rose and one fell

Answer: B. Most real systems have several success criteria at once, and optimising one alone reliably damages another. Doubling unit cost is a business decision that someone must take deliberately with the numbers in front of them.

Question 6

Why should an eval set be refreshed periodically from production traffic?

1. To increase its size
2. Because the incoming request distribution shifts, and a static suite eventually measures a workload you no longer have
3. To reduce grading cost
4. Because old cases expire

Answer: B. Drift happens on the input side as well as the model side. Sampling real traffic into a periodically re-graded suite keeps the definition of correct aligned with what users actually send.

Question 7

A team ships a new prompt, a new model version and a new chunking strategy together, and quality improves. What is the problem?

1. Nothing, since quality improved
2. The result cannot be attributed, so you cannot roll back intelligently or repeat the gain
3. Three changes exceed the safe limit
4. The model version should always change alone

Answer: B. An unattributable improvement is fragile: you do not know which change to keep, which to revert when something regresses later, or which to apply elsewhere. One measured variable at a time.

Question 8

Which is an appropriately specified success criterion?

1. High-quality summaries
2. Macro-F1 at or above 0.88 on the labelled holdout set, with escalation below the confidence threshold
3. Better than the previous system
4. Users are satisfied

Answer: B. Specific, measurable, achievable and relevant — and it names both the metric and what happens to the cases that fall short. The others cannot be tested, so they cannot gate a deployment or anchor an SLA.

Question 9

Costs are higher than forecast in a multi-component system. What is the correct first step?

1. Move to a cheaper model tier
2. Instrument cost per request by component and find where it concentrates
3. Reduce the eval suite to save spend
4. Shorten all prompts

Answer: B. The right lever depends entirely on where the cost actually sits, and it is frequently retrieval breadth, redundant context or an uncached prefix rather than the model tier. Measuring first is the professional habit the domain tests repeatedly.

Question 10

In a RAG system producing well-formed but ungrounded answers, which layer should be investigated before the prompt?

1. The integration layer
2. The retrieval layer
3. The tool execution layer
4. The transport layer

Answer: B. Well-formed responses mean the integration is working. In a RAG system the next candidate is retrieval: if the answer was never in the retrieved chunks, no prompt change will produce it. Isolating the layer before changing anything is the discipline being tested.

← Previous: Questions & answers — Domain 3  |  Next: Questions & answers — Domain 5 →