Questions? Call 1.800.716.8955

Chapter 7

The six scenarios, worked through

You will meet four of these six scenarios, so know each one's architecture and its favourite failure modes before exam day. For each, the table lists the symptoms the questions tend to describe and the fix the guide's principles point to.

Scenario 1: Customer Support Resolution Agent

An Agent SDK agent with MCP tools get_customer, lookup_order, process_refund and escalate_to_human, aiming for 80%+ first-contact resolution.

  • Symptom Fix
  • Agent sometimes skips identity check
  • before refunds

Programmatic gate: block lookup_order and process_refund until get_customer returns a verified ID

Refunds above the limit slip through PreToolUse hook that blocks and redirects to escalate_to_human Order questions routed to get_customer Rewrite both tool descriptions with formats, examples and boundaries Easy cases escalated, hard ones attemp‐

  • ted
  • Explicit escalation criteria with few-shot examples
  • Several customers match the name Ask for another identifier
  • Human agents ask customers to repeat
  • everything

Structured hand-off: ID, root cause, amounts, actions tried, recommendation

Agent forgets amounts late in long chats Case facts block outside the summarised history; trim lookup results Case study: Northwind Outfitters' returns agent

A composite case built from the patterns in the exam guide. Names and numbers are illustrative. Starting point. Northwind, an online outdoor retailer, launched a support agent on the Agent SDK with four MCP tools. In week one, first-contact resolution was 55%, against an 80% target. The team pulled 500 transcripts and sorted the failures.

  • Failure
  • Share of fail‐
  • ures Root cause
  • Escalated simple damage
  • claims

31% No explicit escalation criteria; agent escalated anything that "felt"

risky

Wrong tool for order questions 18% Minimal tool descriptions

Lost track in multi-issue chats 16% No decomposition, no case facts Failure

Share of fail‐

ures Root cause

Retried non-retryable errors 12% Generic "Operation failed" errors Refunded without verified iden‐

  • tity
  • 4% Prompt-only rule
  • Other 19% Mixed

Fixes, in the order they shipped.

Day 1: identity gate. Even at 4%, refunds to the wrong account are the most damaging failure, so a programmatic prerequisite went in first: no lookup_order or process_refund until get_customer has returned exactly one verified match. A second hook blocks refunds over US$500 and redirects to escalation.

Day 2: tool descriptions. Each tool got input formats, example requests, outputs and a "use this, not that" line. Order-question misrouting fell from 18% of failures to under 2%. Day 3: structured errors. Errors now carry errorCategory, isRetryable and a customersafe message. Wasted retries disappeared, and customers stopped hearing "our system is down" for a declined card.

Week 2: escalation criteria. The system prompt gained explicit triggers (explicit request, policy gap, no progress, amounts over the limit) and four few-shot examples, two showing frustrated customers whose issues the agent resolves.

Week 2: case facts and trimming. A PostToolUse hook trims order lookups from 43 fields to 6, and a case facts block tracks each issue's ID, amount, status and promises. Week 3: decomposition. The prompt now requires listing every issue, resolving or escalating each, and a single closing reply covering all of them. Outcome. By week four, first-contact resolution reached 82%. Escalations fell by half, but the escalations that remained were the right ones: policy gaps and explicit requests, each with a structured hand-off that let human agents resolve them without re-asking.

What the exam would ask. Every row of the failure table maps to a likely exam item, and every fix is the "most effective" answer for that item. Notice what the team did not do: build a classifier, add sentiment analysis, or upgrade the model.

Scenario 2: Code Generation with Claude Code

A team uses Claude Code for generation, refactoring, debugging and docs, with shared commands and CLAUDE.md.

Symptom Fix

New hire does not get team conventions Move them from ~/.claude/CLAUDE.md to the project CLAUDE.md Team command missing for others Put it in .claude/commands/ in the repo Test conventions ignored for scattered test files .claude/rules/ file with paths: ["**/*.test.tsx"]

Symptom Fix

Huge refactor goes wrong halfway Plan mode first, then execute the plan Main session bloated by exploration Explore subagent, or a skill with context: fork Prose description of a transform misread Give 2 or 3 input/output examples Case study: rolling Claude Code out to a 40-person team

A composite case; details are illustrative.

Starting point. A fintech platform team of 40 engineers adopted Claude Code informally. Six months later, results varied wildly between people. A survey found three causes: conventions lived in a few senior engineers' personal files, nobody knew which commands existed, and large changes were attempted in direct execution and frequently abandoned.

The rollout plan.

Week Change Exam concept

1 Gather conventions from personal ~/.claude/CLAUDE.md files into a 50-line project CLAUDE.md plus .claude/rules/ files

Hierarchy and shar‐

ing

1 Scope rules: **/*.test.ts for tests, services/payments/**/* for PCI-sensitive code, infra/**/*.tf for Terraform

Path-specific rules

2 Publish /review, /changelog and /migration-plan in .claude/commands/ Project-scoped commands

2 Add a dependency-audit skill with context: fork and read-only allowed-tools Skill frontmatter 3 Add .mcp.json with Jira and GitHub servers using ${JIRA_TOKEN} and $

{github_token}

MCP scope and

secrets

3 Team guideline: plan mode for anything touching more than 5 files or more than one service

Plan mode vs direct

4 Teach the interview pattern and test-first iteration in a lunch session Iterative refinement A problem during rollout. Two weeks in, engineers on the payments team reported that Claude ignored the PCI rules. /memory showed the rules file loading for some files and not others. The cause: the paths glob was services/payments/*, which matched only the top directory. Changing it to services/payments/**/* fixed it. This is the kind of diagnosis the guide expects: verify what is loaded before rewriting instructions. A second problem. One engineer customised the team /review command in place to add personal checks, and teammates started getting his checks. The fix was to copy it to ~/.claude/ commands/review-strict.md under a new name and restore the team version.

Outcome. Abandoned large changes dropped sharply once plan mode became the norm for multi-file work, and review comments about convention violations fell because every session now loaded the same rules.

Scenario 3: Multi-Agent Research System

A coordinator delegates to search, document analysis, synthesis and report subagents and produces cited reports.

Symptom Fix

Report misses whole subtopics Coordinator's decomposition too narrow; broaden it and add a gapcheck loop Synthesis lacks findings or citations Pass full findings with structured source metadata in the subagent prompt

Subagents run one after another Emit all Task calls in one coordinator response Synthesis round-trips for simple fact checks Scoped verify_fact tool for the synthesis agent; deep checks via coordinator

Search timeout breaks or silently empties the

report

Structured error context to the coordinator; coverage annotations in the output

Conflicting statistics averaged or dropped Keep both values with sources and dates; mark as contested Case study: a policy research desk

A composite case; details are illustrative.

Starting point. A consultancy built a research system to brief clients on regulation. Version 1 used a fixed pipeline: one search agent, one analysis agent, one writer, always in that order, each passed a one-line instruction. Reports read well but analysts found three recurring problems: missing jurisdictions, unattributed statistics, and reports that silently omitted topics when a source was down.

Version 2 architecture.

Coordinator with a dimension step. Before delegating, the coordinator lists the question's dimensions (for regulation: jurisdictions, timelines, enforcement, penalties, open consultations) and writes them to a plan the analysts can inspect. Parallel, partitioned search. One search subagent per jurisdiction, all launched in one response, each with a distinct scope so they do not duplicate effort. Structured findings. Every subagent returns claims with excerpt, URL or document, page and publication date, plus an errors list and a gaps list.

Local recovery. Search agents retry transient failures once and try an alternative source before reporting a failure, with what they attempted. Synthesis with provenance. The synthesis agent receives all structured findings in its prompt, keeps claim-source mappings, separates settled from contested points, and renders penalty comparisons as a table.

Coverage loop. After synthesis, the coordinator compares the report against its dimension list. Missing or thin dimensions trigger targeted searches and one more synthesis round, at most twice.

Scoped verification. The synthesis agent has a verify_fact tool for simple checks such as a date or an article number; deeper questions go back through the coordinator.

A conflict handled well. Two credible sources gave different maximum fines for the same regulation. The analysis agent recorded both with dates; one was the original text and the other an amendment. The report showed both, explained the amendment and cited each. Version 1 would have picked one.

A failure handled well. The legislative database was unavailable for one jurisdiction. The search agent reported the failure, the attempted queries and partial results from a secondary source. The final report said so in a coverage note instead of implying there was no relevant law.

Exam lens. Version 1 shows nearly every Domain 1 and Domain 5 anti-pattern at once: fixed routing for an open-ended task, thin subagent briefs, sequential execution, and silent failure. Version 2 is the exam's model answer.

Scenario 4: Developer Productivity with Claude

An Agent SDK tool that helps engineers explore unfamiliar and legacy code with Read, Write, Bash, Grep, Glob and MCP servers.

Symptom Fix

Agent reads every file and runs out of context Grep for entry points, then Read along imports Wrong tool for finding files by name Glob for paths, Grep for contents Edit fails on a repeated snippet Read the file, then Write the whole file Agent prefers Grep over a better MCP search

  • tool
  • Improve the MCP tool's description
  • Answers drift to "typical patterns" late in the
  • session

Scratchpad file, phase summaries, subagents for narrow questions, / compact

Needs to compare two refactoring strategies fork_session from the shared analysis Case study: a legacy-code navigator

A composite case; details are illustrative.

Starting point. An insurer's platform team built an Agent SDK tool to help engineers understand a 15-year-old Java policy-administration system: 4,000 files, inconsistent naming and several layers of wrappers. Early versions failed in predictable ways: reading huge files up front, losing track of findings after an hour, and missing call sites hidden behind wrappers. Design choices that fixed it.

Problem Design choice

Reads too much up front System prompt and few-shot examples teach Grep for entry points, then Read along the call chain

Problem Design choice

Misses usages behind wrappers A standard procedure: list every exported alias of a symbol, then Grep for each Repeated Edit failures on boil‐

erplate

Fall back to Read plus Write when the anchor is not unique

Forgets findings after an hour A findings.md scratchpad, updated after each discovery and re-read before answering

Verbose exploration crowds out

reasoning

Subagents for narrow questions ("find every caller of RatingEngine.calculate"), returning short summaries

  • Two engineers want to compare
  • refactoring options
  • fork_session from the shared analysis

Engineers want ticket context A community Jira MCP server in .mcp.json; a custom MCP server only for the team's internal rating-rules catalogue, exposed as a resource

Agent prefers Grep over the se‐

mantic code index

Rewrote the index tool's description to say what it finds that Grep cannot A typical session. An engineer asks: "What happens when a policy is cancelled mid-term?" The agent greps for cancel in service classes, finds PolicyLifecycleService.cancel, reads it, follows calls into ProRataCalculator and RefundQueue, and delegates "find every consumer of RefundQueue messages" to a subagent. It records each finding in findings.md with file and line references. The final answer names actual classes and lines, not "typical patterns". Exam lens. This scenario tests Domain 2's built-in tool selection, Domain 5's context management for large codebases, and Domain 1's session management. Most wrong answers propose "read everything" or "use a bigger context window".

Scenario 5: Claude Code for Continuous Integration

Automated PR review, test generation and feedback with minimal false positives.

Symptom Fix

Job hangs waiting for input claude -p

Script cannot parse findings --output-format json with --json-schema Developers ignore the bot Explicit report/skip criteria; disable the noisiest category while you fix it

Duplicate comments after each push Pass prior findings; ask for new or unaddressed issues only Generated tests duplicate existing ones Provide existing tests; document test standards and fixtures in CLAUDE.md

Large PRs get uneven, contradictory reviews Per-file passes plus an integration pass Manager wants Batches API for pre-merge

  • checks
  • No: batch only latency-tolerant jobs such as overnight reports
  • Case study: winning back trust in a review bot

A composite case; details are illustrative.

Starting point. A SaaS company added Claude Code to CI for PR review and test generation. Within a month, developers resolved 94% of bot comments without changes, and a real SQL-injection finding was dismissed with the rest. Test generation added hundreds of trivial tests. Diagnosis.

The review prompt said "review for quality, security and best practices; be thorough". There were no criteria for what to report or skip.

The job ran in the same session that the coding agent used, so the reviewer inherited the author's reasoning. Each push triggered a full re-review that re-posted old comments.

Large PRs were reviewed in one pass, giving uneven depth.

Test generation never saw the existing tests or any statement of what a valuable test was. Changes.

A fresh claude -p invocation per review, independent of any authoring session. Review criteria in CLAUDE.md: report bugs, security issues and comment contradictions; skip style, naming and patterns already used in the module. Severity defined with a code example per level. Few-shot pairs showing acceptable patterns next to genuine issues.

Style and documentation categories switched off for a month while precision was measured. Per-file passes, then an integration pass for cross-file data flow.

Prior findings passed into each re-run with "report only new or still-unaddressed issues". Findings emitted with --output-format json --json-schema and a detected_pattern field; dismissals were logged against patterns weekly.

Test generation moved to a nightly batch job, with existing tests, a coverage report and testing standards from CLAUDE.md in context. Outcome. Comments per PR fell from 12 to about 3, and the share acted on rose steadily. When style checks were re-enabled, they returned with explicit criteria and examples and stayed below the agreed false-positive rate. Exam lens. This is Scenario 5 in miniature: precision through criteria not caution, independence of the reviewer, multi-pass structure, structured output flags, and batch versus synchronous choices.

Scenario 6: Structured Data Extraction

Extraction from unstructured documents, validated against JSON schemas, feeding downstream systems.

Symptom Fix

Malformed JSON Tool use with a JSON schema

Symptom Fix

Fabricated values for missing fields Make those fields nullable

Totals do not add up despite valid JSON Semantic validation; calculated_total vs stated_total, conflict_detected

Validation fails Retry with the document, the failed output and the specific errors Retries keep failing Check whether the data exists in the source at all Unknown document type tool_choice: "any" across the extraction tools 97% accuracy, but one form type is poor Segment accuracy by type and field; calibrated confidence; stratified sampling

  • 100,000 documents overnight, some too
  • long
  • Batches API, retry failed custom_ids with chunking
  • Case study: automating accounts-payable intake

A composite case; details are illustrative.

Starting point. A logistics company receives 60,000 supplier invoices a month as PDFs and email bodies. A first Claude pipeline asked for JSON in the prompt and parsed the reply. About 3% of replies were not valid JSON, totals sometimes did not add up, and missing purchase-order numbers were "filled in" with plausible values.

Version 2 pipeline.

Stage Design

Classify Three extraction tools (invoice, credit note, statement); tool_choice: "any" so a tool is always called Extract Strict JSON schemas; nullable fields for anything that may be absent; other plus detail for expense categories; unclear for ambiguous ones

Normal‐

ise

Prompt rules for dates (ISO 8601), currency (code plus number) and supplier names; few-shot examples covering scanned, tabular and narrative layouts

Validate Code checks: line items plus tax minus discount equals stated total; dates plausible; supplier exists; conflict_detected when the document contradicts itself

Retry One retry with the document, the failed extraction and specific errors; no retry when the data refers to an attachment not supplied

Process Message Batches API every 4 hours for the 30-hour SLA; failures resubmitted by custom_id, oversized PDFs split by page range

Review Field-level confidence, calibrated per document type on a labelled set; low confidence, nulls in required business fields, or conflicts go to a human queue

Monitor A stratified random sample of automated extractions reviewed weekly per supplier segment What the numbers showed. Aggregate accuracy reached 98.6%. Broken down, credit notes from two suppliers sat at 88% because they placed the credit as a negative line item. Two fewshot examples showing that layout raised them to 98%. Without segmentation, the team would have automated a weak segment.

Exam lens. This case touches nearly every Domain 4 task statement and the human-review and confidence-calibration parts of Domain 5. If an exam item mentions a schema, a retry, a batch or an accuracy figure, picture this pipeline and ask which stage the question is about.

Use ← → to move between sections