Evaluation, Testing & Optimization (16%)

What the exam tests: proving a Claude system works, finding why it fails, and making it cheaper and faster without making it worse.

Official task statements: define metrics for accuracy, latency, cost, safety, and security; design evaluation datasets and test frameworks using mixed methods; conduct A/B testing and iterative improvement; diagnose prompt failures, hallucinations, and model mismatch; optimize token usage, latency, and cost-performance; monitor with logging and observability tools.

Define success before you build

Write success criteria that are specific and measurable, then pick a metric for each.

Dimension Example metric Example target

Accuracy Exact match on extracted fields; grader score on answers 97% field accuracy Groundedness Share of claims supported by cited sources 99%

Latency p50 and p95 end to end; time to first token p95 under 4 s Cost Cost per successful task, not per call Under $0.05

Safety Policy-violation rate; correct refusal and escalation rate 0 critical violations Security Prompt-injection success rate in red-team suite Under 1% Business Deflection, handle time, reviewer edit rate 30% deflection

Build the evaluation set

Draw cases from real traffic (redacted), not only invented ones, and stratify by intent, difficulty, and language. Include edge cases, adversarial inputs, and cases where the right answer is "I don't know" or "escalate."

Label with domain experts and record why each label is correct.

Start with 50 to 200 cases and grow it from every production failure. Keep a frozen holdout to catch overfitting.

Mixed-method grading

• Method Use for Limits
• Code-based checks Schema validity, exact fields, required cita‐
• tions, length
• Cannot judge meaning

Model-based grading (LLM-asjudge)Tone, helpfulness, faithfulness against a rubric, at scale

• Must be calibrated against human
• labels
• Human review Ground truth, high-stakes cases, calibrating
• the judge
• Slow and costly; sample it

Good practice: a clear rubric, the judge reasons before scoring, the judge is checked for agreement with humans on a sample, and code checks run first because they are cheap.

Testing in the delivery pipeline

Treat prompts, tools, model versions, and retrieval config as code. Every change runs the regression suite in CI and must not drop any key metric below its threshold. Release behind a flag, canary to a small share of traffic, then compare.

A/B testing and iteration

Change one variable at a time, define the primary metric and guardrail metrics up front, size the sample for the effect you care about, and run long enough to cover weekly patterns. Offline evaluation tells you whether to ship; online A/B tells you whether it helped users.

Diagnosis: read the symptom

• Symptom Likely cause First check
• Confident wrong answers after a
• document refresh

Retrieval or indexing Retrieved chunks for failing queries (official sample)

Prompt structure or conflicting rules Simplify, reorder, split into steps Format breaks downstream No enforced schema Structured output or tool schema plus validation

Cost and latency optimization

Work through the levers in order of risk, checking evaluations after each.

Prompt caching for static prefixes.

Batch processing for work that can wait (the batch API is discounted, about half price at the time of writing, with results within 24 hours).

Routing easy traffic to a smaller model.

Trim context: fewer, better chunks; compact history; shorter tool outputs.

Limit output: tighter formats and max tokens; thinking only where it pays.

Streaming to cut perceived latency.

Parallelize independent calls.

Measure cost per successful outcome. A cheaper model that doubles retries or human rework is not cheaper.

Production monitoring

Dashboards for volume, latency percentiles, error and rate-limit responses, token spend by feature and team, cache hit rate, grader scores on sampled traffic, escalation and refusal rates, and user feedback. Alert on change, not just thresholds, and attach prompt and model versions to every trace so a regression maps to a release.

Deep dive: the evaluation toolkit

Defining metrics precisely

A metric is only useful if two people would compute it the same way. Write each one down. Metric Definition Watch out for

• Task success
• rate
• Share of cases where the output meets all must-have
• criteria in the rubric
• Vague rubrics; define "must-have"

Field accuracy Share of extracted fields that exactly match the label after normalization

• Normalization rules (dates, currency)
• must be fixed
• Groundedness Share of factual claims supported by the provided
• sources
• Needs claim splitting; judge calibration
• Abstention cor‐
• rectness
• Share of unanswerable cases where the system cor‐
• rectly declines or escalates
• Systems that abstain too often look "safe"

Critical error rate Share of outputs containing an error from a defined critical list

Keep the list short and concrete

p95 latency 95th percentile of end-to-end time Measure from the user's view, including retrieval and tools

Cost per success Total cost (model, tools, review) divided by successful tasks

Include retries and human rework

Metric Definition Watch out for

Escalation rate Share of cases routed to humans Not always bad; compare with risk tier Designing an LLM judge

One criterion per judge call where possible (accuracy, tone, completeness separately). Multi-criterion judges blur scores.

Concrete rubric with anchored levels: describe what a 1, 3, and 5 look like, with examples. Reference answers when available; the judge compares rather than guesses.

Reasoning before the score, so the judgment is explained and auditable.

Binary or small scales (pass/fail, 1 to 5) are more reliable than 1 to 100.

Calibrate against expert labels and re-check periodically, especially after changing the judge model or rubric.

Guard against bias: judges can prefer longer answers or answers in a familiar style. Test for it.

Statistics, briefly

You do not need to be a statistician, but the exam expects judgment.

Small evaluation sets give noisy results. A 2-point difference on 50 cases is often noise; on 1,000 cases it may be real.

Run each evaluation more than once if outputs vary, and compare the spread.

For A/B tests, decide the sample size and duration before starting, and avoid stopping as soon as the result looks good.

Report the result with its uncertainty ("91% ± 2%") when it matters to a decision.

Red teaming as a program

Red teaming is not a one-time event before launch.

Element Practice

Scope Prompt injection (direct and indirect), data exfiltration, jailbreaks of the system's own rules, harmful advice in the domain, privacy leaks, tool misuse

People Internal security staff, domain experts, and external testers for high-risk systems Assets A growing library of attack cases, versioned like the evaluation set Automa‐

tion

Run the attack library on every release, like a regression suite Metrics Attack success rate by category, time to fix

Feed‐

back

Every successful attack becomes a test case and, where possible, a design fix (not only a prompt patch)

Load and capacity testing

Before launch, test the whole system at expected peak and at two to three times peak. Measure throughput, latency percentiles, error rates, and rate-limit responses from the provider. Confirm that retries with backoff and queueing behave well under pressure, without retry storms.

Check quotas and rate limits with your provider or cloud platform, and request increases ahead of launch if needed.

Confirm that the fallback path (secondary region, platform, or human queue) works under load.

Cost modeling

Build a simple model before launch and compare it with reality each month.

\text{cost per task} = \frac{\text{model cost} + \text{tool and infrastructure cost} + \text{human review cost}}{\text{successful tasks}}

Model cost depends on input tokens (split into cached and uncached), output tokens (including thinking), and the number of calls per task. Track each term per feature, so a spike can be traced to its cause: a new prompt, a traffic shift, a loop, or a cache that stopped hitting.

Worked example 5.1: building an evaluation plan from scratch

Brief. A bank is launching an assistant that answers customer questions about mortgage products and drafts pre-qualification summaries for loan officers. Launch is in ten weeks. Step 1: success criteria by stakeholder.

Stakehold‐

er Criterion Metric Threshold

Customers Correct, clear answers Rubric score from calibrated judge 4.2 of 5 Compli‐

Summaries need little editing Edit distance and officer rating Median under 15% edit Operations Fast enough p95 end-to-end latency under 5 s

• Finance Affordable Cost per conversation under budget line
• Security Resistant to injection and data
• leakage
• Red-team success rate under 1%

Step 2: dataset. 500 cases: 300 sampled from redacted call-center transcripts (stratified by product, question type, and customer segment), 100 edge cases written by mortgage specialists

(self-employed income, recent bankruptcy, non-resident buyers), 60 "should refuse or escalate" cases (requests for a guaranteed rate, legal advice), 40 adversarial cases (injection in uploaded documents, attempts to get another customer's data). Hold out 100 as a frozen test set. Step 3: grading stack.

Code checks: required disclaimers present, no rate quoted without the rate-table tool, output schema valid for summaries.

LLM judge with a rubric for accuracy, clarity, and compliance tone; the judge writes its reasoning before scoring. Human calibration: mortgage specialists grade 100 cases; the judge must agree with them on at least 85% of pass/fail decisions before it is trusted.

Weekly human audit of a random sample from production after launch.

Step 4: gates. Pull requests that change prompts, tools, retrieval, or model version run the full suite in CI. A drop of more than 2 points on any key metric, or any critical violation, blocks the merge.

Step 5: launch plan. Internal staff only for two weeks, then 5% of web traffic, then 25%, then full. Each step requires metrics at or above threshold for a full week.

Worked example 5.2: diagnosing a regression from traces

Symptom. On Tuesday morning, user thumbs-down rates on a product-support assistant doubled. Latency and error rates look normal.

Investigation path.

What changed? Release log: no prompt or model change. Ingestion log: a new product catalog was loaded Monday night. Segment the failures. Thumbs-down clustered on questions about three product lines. Inspect traces for those questions: retrieved chunks come from the new catalog but list prices as "see table," because tables failed to parse.

Confirm by re-running the evaluation set for those product lines: accuracy dropped from 93% to 61%.

Fix: roll back to the previous index version, fix table parsing, re-ingest, re-run evaluations, redeploy.

Prevent: add an ingestion quality check (sample chunks per document type, alert on empty tables) and add the failing questions to the evaluation set.

What the exam wants you to see. The symptom (quality drop, latency unchanged, model unchanged) pointed to data. Changing the model or prompt would have wasted time. Trace logging with index versions made the root cause visible in minutes.

Worked example 5.3: optimizing cost without losing quality

Brief. A document-analysis service costs three times its budget. Profile per request: 22,000 input tokens (16,000 is a fixed analysis guide, 6,000 the document), 2,500 output tokens, extended thinking on every call, most-capable tier, all requests real-time even though 70% are nightly bulk uploads.

Plan, with an evaluation after each step.

Step Change Effect to expect

1 Move the 16,000-token guide to a cached prefix Large cut in input cost and time to first token on repeated traffic

2 Send the 70% bulk uploads through batch processing Discount on that share (about half price at the time of writing)

3 Test the balanced tier with the improved prompt Lower unit price if evaluation holds 4 Turn thinking on only for documents flagged complex by

• a classifier
• Fewer output tokens on simple documents
• 5 Tighten the output format (structured findings instead of
• prose)
• Fewer output tokens, easier validation

Do not cut the guide itself; it carries required instructions. The exam frequently offers "truncate the guide" as a distractor.

Worked example 5.4: designing an A/B test

Question. Does adding a reranker improve a support assistant enough to justify 300 ms of extra latency?

Design.

Hypothesis: reranking increases answer resolution rate.

Primary metric: resolution rate (user did not contact support again within 72 hours). Guardrail metrics: p95 latency, cost per conversation, thumbs-down rate.

Unit: user (not request), so one user always sees one variant.

Split: 50/50 after an offline evaluation shows no regression.

Duration: at least two full weeks to cover weekly patterns; sample size calculated for the smallest improvement that matters (say 2 points).

Decision rule: ship if resolution improves by at least 2 points with statistical confidence and no guardrail breaches.

Scenario drills

Your LLM judge gives the same score to every answer that sounds confident. What do you do?

A team proudly reports 99% accuracy on 40 test cases they wrote last week. What is your concern?

After switching to a cheaper model, cost per call dropped 60% but total spend dropped only 10%. Possible reasons?

Production answers in Spanish have more complaints, but your evaluation scores are high. Why?

A model starts adding unsupported claims after a prompt change that asked for "more detailed" answers. What is the fix and the lesson? Leadership wants one number on a dashboard for "AI quality." How do you respond? A team wants to skip offline evaluation and "just A/B test in production." When is that acceptable? Answers.

The judge is not calibrated: tighten the rubric with concrete criteria and examples, require reasoning before the score, check agreement with human labels, and add code checks for facts that can be verified.

Too few cases, written by the builders, likely not representative and probably overfitted. Expand with real traffic and expert labels; keep a holdout. More retries, longer outputs, more escalations to humans, or more conversation turns. Measure cost per successful task.

The evaluation set has few or no Spanish cases. Stratify by language.

Revert or rewrite the prompt to ask for detail grounded in sources, require citations, and add a groundedness check. Lesson: every prompt change runs through evaluation, including ones that seem harmless.

Offer a small scorecard (quality, safety, latency, cost, business outcome) with one headline business metric; a single blended number hides critical failures.

Rarely for anything customer-facing or regulated. Offline evaluation catches regressions before users see them; A/B tests confirm user impact. Use both.

Common traps

Measuring cost per call instead of cost per successful task.

Using an LLM judge without checking it against human labels.

Changing prompt and model in the same release.

Treating a single happy-path demo as an evaluation.

Check yourself

Name the three grading methods and the order you would run them.

A summarizer passes offline evals but users rate it lower after launch. What next? Monthly report generation costs too much and is not urgent. Best lever?

Answers started citing outdated policies last week. First place to look?

Answers: (1) Code checks, then model-based grading, with sampled human review to calibrate. (2) Check whether the eval set represents real traffic, add failing cases, run an online A/B on a fix. (3) Batch processing, plus caching of the shared template. (4) Ingestion and index freshness: stale or duplicate documents in the index.