CCAO-F questions and answers — Domain 2: Output Evaluation and Validation

13 practice questions on Output Evaluation and Validation, the domain weighted at 21% of the CCAO-F exam. Every answer explains why the distractors fail — that is the part that transfers to questions you have not seen before.

Question 1

Claude produces a market analysis containing several specific percentages and a reference to an industry report. You did not supply any source documents. What should you do before circulating it?

1. Circulate it, noting that it was AI-assisted
2. Verify each figure and confirm the referenced report exists and says what is claimed
3. Ask Claude whether the figures are accurate
4. Check the writing for tone and grammar

Answer: B. Specific figures and citations produced without supplied sources are exactly where fabrication occurs, and a fabricated reference looks perfectly formatted. Asking Claude to self-assess is not independent verification. Noting AI assistance discloses the process without checking the content.

Question 2

Which of these is the clearest sign that an output needs closer verification?

1. It is longer than you expected
2. It contains precise statistics and named sources that were not in any material you provided
3. It is written more formally than you asked
4. It took longer than usual to generate

Answer: B. Unsourced specificity is the classic hallucination signature. Length, register and generation time say nothing about factual reliability.

Question 3

A summary of an attached report is accurate in everything it says, but omits the one finding your director specifically asked about. Which evaluation dimension has failed?

1. Accuracy
2. Completeness
3. Appropriateness
4. Tone

Answer: B. Everything present is correct, so accuracy holds. The output failed to answer the whole question, which is completeness — and it is the dimension people skip, because a well-written answer feels finished.

Question 4

You are preparing an internal brainstorm list of campaign ideas. What level of verification is appropriate?

1. Verify every claim against primary sources
2. Read it through, then use it
3. Have a colleague independently review it before use
4. Route it to your legal team

Answer: B. Proportionality runs both ways. A brainstorm carries almost no consequence if an idea is weak, so heavyweight verification wastes time and, in practice, is why teams abandon otherwise good workflows. Save the heavy checks for output that carries consequence.

Question 5

Which instruction most directly reduces the chance of Claude filling a gap with something plausible but false?

1. "Be thorough."
2. "If the attached documents do not contain the answer, say that you don't have enough information."
3. "Double-check your work."
4. "Write in a confident, authoritative tone."

Answer: B. Giving explicit permission to say "I don't know" removes the pressure to produce something for every question. It is one of the highest-value sentences you can add. Instructing confidence does the opposite — it encourages exactly the fluent-but-unsupported output you are trying to avoid.

Question 6

An HR coordinator uses Claude to summarise candidate applications for a hiring panel. What is the correct handling?

1. Use the summaries as the shortlist, since they are based on the applications
2. Use the summaries as preparation, with the panel reading the underlying applications and owning every decision
3. Ask Claude to rank the candidates and have the panel approve the ranking
4. Use the summaries, and disclose to candidates that AI was used

Answer: B. Employment is one of the high-risk domains where a qualified human must review and own the decision. Summaries as preparation are fine; a Claude-produced shortlist or ranking that the panel merely ratifies puts the decision in the wrong place. Disclosure is required too, but it does not license skipping human judgement.

Question 7

You ask the same analytical question twice and get materially different answers. What does this tell you?

1. The model is malfunctioning and you should report it
2. The areas where the answers diverge are the ones least well supported and need verification
3. You should always use the first answer
4. You should average the two responses

Answer: B. Divergence between runs is a useful signal: where the model is working from solid grounding it tends to be stable, and where it is not, it varies. Treat the differences as a map of what to check, not as a fault to report or a thing to average.

Question 8

Claude produces a comparison of two vendors that is presented as balanced but consistently frames one more favourably. What is the best response?

1. Accept it, since a comparison has to reach a conclusion
2. Ask explicitly for the strongest case for the other vendor and the main risks of the favoured one, then reassess
3. Ask Claude to be unbiased
4. Discard it and write the comparison yourself

Answer: B. Surfacing the counter-argument is a concrete test of whether the framing survives scrutiny, and it often reveals that one option's weaknesses were simply not mentioned. "Be unbiased" is not a mechanism, and discarding the work throws away a usable structure.

Question 9

Which technique makes an output auditable rather than simply more likely to be correct?

1. Asking for chain-of-thought reasoning
2. Requiring a citation or supporting quote for each claim
3. Using a more capable model
4. Asking for a longer answer

Answer: B. Citations create a trail you can follow, so you can check the claims that matter rather than re-researching the whole document. Reasoning helps you spot faulty logic, which is valuable but different; model tier and length affect quality, not auditability.

Question 10

A financial analyst asks Claude to check a draft investment memo. What is the correct framing of what Claude's review is worth?

1. It is a sufficient review if Claude finds no problems
2. It is a useful first pass; a qualified person still reviews and owns the memo
3. It is only useful for grammar and formatting
4. It should not be used at all in financial contexts

Answer: B. Financial advice is a high-risk domain requiring human review, so Claude's pass never closes the loop. But it is genuinely useful for far more than grammar — catching inconsistencies, unsupported claims and gaps — and refusing to use it at all is as wrong as over-relying on it.

Question 11

You gave Claude three source documents and asked it to use only those. The output includes a claim that is not in any of them. What is the right conclusion?

1. The instruction failed, so restriction techniques do not work
2. These techniques reduce the error rate but do not eliminate it, so verification is still required
3. The documents must have been uploaded incorrectly
4. The model needs a longer context window

Answer: B. Grounding techniques materially reduce fabrication without guaranteeing its absence. That is precisely why the exam never accepts an option claiming a technique makes review unnecessary. The single failure is expected behaviour, not evidence the approach is worthless.

Question 12

An output is factually correct but written in a register that would read as flippant to the customers receiving it. How should this be scored?

1. It passes, because it is accurate
2. It fails on appropriateness and needs revision before sending
3. It fails on completeness
4. It is acceptable if you add a disclaimer

Answer: B. Appropriateness — right tone for the right audience — is a separate dimension from accuracy, and an output can pass one and fail another. Nothing is missing, so completeness holds; a disclaimer does not fix a register problem.

Question 13

Which output carries the lowest inherent accuracy risk?

1. A list of recent regulatory changes in your industry
2. A rewritten version of a paragraph you supplied, made more concise
3. A competitor's likely pricing strategy
4. A summary of a conference that happened last month

Answer: B. Rewriting supplied text is transformation: the facts come from you, so there is little for the model to invent. The other three all require the model to supply information from its own knowledge, and two of them concern recent events, where risk is highest.

← Previous: Questions & answers — Domain 1  |  Next: Questions & answers — Domain 3 →