Prompt Engineering & Structured Output (20%)

What the exam tests: whether you can cut false positives with explicit criteria, use few-shot examples where instructions fail, get guaranteed-valid JSON through tool use, build validationretry loops that know their limits, choose batch versus synchronous calls, and design independent multi-pass reviews. About 12 of 60 items, centred on the CI and extraction scenarios.

5.1 Explicit criteria beat vague caution (Task 4.1)

"Be conservative" and "only report high-confidence findings" do not raise precision. Specific, categorical criteria do.

• Vague Explicit
• Check that comments are
• accurate

Flag a comment only when the behaviour it claims contradicts what the code does Report important issues Report bugs and security issues. Skip style, naming and patterns local to this file Rate severity High: data loss or auth bypass (example). Medium: wrong result on an edge case (example). Low: misleading message (example)

False positives are expensive beyond their own count: one noisy category teaches developers to ignore the accurate ones too. If a category is mostly noise, switch it off while you fix its prompt, to restore trust.

5.2 Few-shot examples (Task 4.2)

When detailed instructions still give inconsistent output, few-shot examples are the most effective fix. Use 2 to 4 targeted examples, aimed at the ambiguous cases, and show the reasoning for the choice over plausible alternatives. They work for:

Format: every finding with location, issue, severity and suggested fix.

Judgment: an acceptable pattern next to a genuine bug, so Claude generalises the line between them rather than matching keywords.

Extraction from varied documents: inline citations versus a bibliography, a methods section versus details embedded in prose, informal measurements. This reduces both hallucinated values and wrongly empty fields.

5.3 Structured output through tool use (Task 4.3)

The most reliable way to get schema-valid JSON is to define a tool whose input_schema is the shape you want and read Claude's tool_use input. Strict schemas remove syntax errors entirely. They do not remove semantic errors: line items that do not add up to the total, or a value placed in the wrong field.

"invoice_date": {"type": ["string", "null"], "description": "ISO 8601. null if not stated."},

"category": {"type": "string", "enum": ["software", "hardware", "services", "other", "unclear"]},

"category_detail": {"type": ["string", "null"], "description": "Required when category is other."},

• "stated_total": {"type": "number"},
• "calculated_total": {"type": "number"},
• "conflict_detected": {"type": "boolean"}
• },

"required": ["vendor", "category", "stated_total", "calculated_total", "conflict_detected"]

• }
• }
• Schema design rules from the guide:

Make a field nullable when the source may not contain it. A required field the document cannot fill invites the model to invent a value.

Add "other" plus a detail string for categories you cannot list completely, and "unclear" for genuinely ambiguous cases.

Put normalisation rules in the prompt (dates to ISO 8601, currency without symbols) alongside the schema. Use tool_choice: "any" when several extraction tools exist and the document type is unknown; a forced tool when one step, such as extract_metadata, must run first.

5.4 Validation, retry and feedback (Task 4.4)

When validation (for example Pydantic) fails, send a follow-up that contains the original document, the failed extraction and the specific errors. This retry-with-error-feedback loop fixes format and structural problems well.

Figure 6. Retry with error feedback fixes structure, not missing information

It cannot fix missing information. If the value lives in a document you did not provide, retrying just invites fabrication; return null or flag it.

Build self-checks into the schema: extract stated_total and also calculated_total, and set conflict_detected when the source contradicts itself. For review tools, add a detected_pattern field to each finding so you can analyse which patterns developers dismiss most.

5.5 Batch processing (Task 4.5)

• Message Batches API Synchronous API
• 50% cheaper Full price
• Up to 24 hours, no latency guarantee Immediate
• No multi-turn tool calls inside a request Full agentic loops
• Correlate results by custom_id One response per call

Overnight reports, weekly audits, nightly test generation Blocking pre-merge checks, live agents Three calculations and habits the exam uses:

SLA arithmetic. With up to 24 hours processing and a 30-hour SLA, submit batches at least every 4 hours (worst case 4 + 24 = 28 hours, leaving margin).

Failure handling. Resubmit only the failed custom_ids, changed as needed, for example chunking documents that exceeded the context limit.

Pilot first. Refine the prompt on a sample before batching thousands of documents, to raise first-pass success.

PROMPT ENGINEERING & STRUCTURED OUTPUT (20%)

5.6 Multi-instance and multi-pass review (Task 4.6)

A model reviewing its own output in the same session keeps its earlier reasoning and tends to agree with itself. An independent instance without that context catches more than a "review your work carefully" instruction or extended thinking does.

For large reviews, use per-file passes for local issues plus a separate integration pass for crossfile data flow. A bigger context window does not fix attention dilution, and "flag only what two of three runs agree on" suppresses real but intermittent findings. Having the reviewer attach a confidence to each finding helps route which ones a human checks.

Worked examples

Worked example 5.A: Rescuing a review bot nobody trusts

Situation. A CI review bot posts about 12 comments per PR. A sample of 200 comments shows: 38% style and naming, 21% "consider adding a comment", 17% comment-accuracy claims (half of them wrong), 15% real bugs, 9% security. Developers now resolve all bot comments without reading them.

Step 1: stop the bleeding. Disable the two noisiest categories (style, documentation suggestions) and the comment-accuracy check until it is fixed. Trust returns faster than any prompt rewrite can be validated.

Step 2: replace vague guidance with criteria.

Report only:

- Bugs: code that produces a wrong result, crashes, or leaks resources on a reachable path. - Security: injection, auth bypass, secrets in code, unsafe deserialisation.

- Comment contradictions: a comment that states behaviour the code does not have. Do not report:

- Style, naming, formatting, or preferences already enforced by the linter.

- Patterns used consistently elsewhere in this file or module.

- Missing comments or documentation.

Step 3: define severity with examples. One short code example each for critical, major and minor, so the classification is consistent.

Step 4: add few-shot pairs that show an acceptable pattern next to a real issue, with the reasoning ("any here is acceptable because the value comes from a validated schema; any in parsePayment is a bug because...").

Step 5: measure. Add detected_pattern to each finding and log which ones developers dismiss. Re-enable a category only when its precision is acceptable. What would not have worked: "be conservative", "only report high-confidence findings", or capping comments at five. None of these tells the model which findings matter.

Worked example 5.B: A schema that stops invention

Situation. A contract extractor fills termination_date, renewal_term and governing_law for every contract, including the 30% that state none of them. Auditors find plausible but invented values.

Diagnosis. All three fields are required strings. The model must put something there. Fix.

• {
• "name": "record_contract",
• "input_schema": {
• "type": "object",
• "properties": {

"parties": {"type": "array", "items": {"type": "string"}, "minItems": 2}, "effective_date": {"type": "string", "description": "ISO 8601"}, "termination_date": {"type": ["string", "null"],

"description": "ISO 8601. null if the contract states no end date."}, "renewal_term": {"type": ["string", "null"]},

"governing_law": {"type": ["string", "null"]},

"contract_type": {"type": "string",

"enum": ["nda", "msa", "sow", "lease", "employment", "other", "unclear"]}, "contract_type_detail": {"type": ["string", "null"]}

},

"required": ["parties", "effective_date", "termination_date", "renewal_term", "governing_law", "contract_type"]

}

}

Note the pattern: the key is required, but its value may be null. Claude must make an explicit decision for every field, and "not stated" is a legal answer. The prompt adds: "Use null when the document does not state a value. Never infer dates from context."

Worked example 5.C: Valid JSON, wrong numbers

Situation. Invoice extraction is schema-valid 100% of the time, but 6% of invoices have line items that do not sum to the total, and 2% put the tax amount in the discount field. Fix. Semantic validation in code, then retry with specifics:

def validate(inv):

errors = []

calc = round(sum(li["amount"] for li in inv["line_items"]) + inv["tax"] - inv["discount"], 2)

if abs(calc - inv["stated_total"]) > 0.01:

errors.append(f"Line items + tax - discount = {calc}, but stated_total = {inv['stated_total']}.")

if inv["discount"] < 0:

errors.append("discount must be zero or positive; check whether this is tax.") return errors

• errors = validate(result)
• if errors:
• PROMPT ENGINEERING & STRUCTURED OUTPUT (20%)

retry_prompt = (f"<document>{doc}</document>\n<previous_extraction>{json.dumps(result)}</

previous_extraction>\n"

"Validation errors:\n- " + "\n- ".join(errors) +

"\nRe-extract. If the document itself is inconsistent, set conflict_detected to true.")

The retry includes the document, the failed extraction and the specific errors, which is the pattern the guide names. The conflict_detected flag handles the case where the source itself is wrong, so the model does not "fix" the numbers to make them add up.

Worked example 5.D: When retrying is pointless

Situation. 4% of extractions fail validation on po_number. Retries never fix them. Investigation. Every failing invoice says "PO as per attached purchase order". The purchase order is a separate file that was never sent. Conclusion. No prompt or retry can extract information that is not in the input. Stop retrying these, return null with a reason ("po_number_source": "external_document"), and either fetch the purchase order and include it or route to a human. Track retry outcomes by error type, so you learn which failures are recoverable.

Worked example 5.E: Batch or not?

Workload Choice Reason

Pre-merge security check developers wait on Synchronous Blocking; no latency SLA on batches Overnight technical-debt report Batch Read next morning; 50% cheaper Weekly re-extraction of 80,000 archived invoices Batch Latency tolerant, large volume Agent that looks up orders mid-task Synchronous Batches cannot run tool calls mid-request Nightly test generation for changed modules Batch Ready by the next working day Live chat summarisation Synchronous User is waiting

SLA arithmetic, worked. Documents arrive continuously; the business needs each processed within 30 hours. Batches can take up to 24 hours. If you submit every N hours, a document can wait up to N hours for the next submission, then up to 24 hours in processing. You need N + 24 ≤ 30, so N ≤ 6; submitting every 4 hours leaves a 2-hour margin for retries, which is the guide's example. Failures, worked. Of 10,000 requests, 37 come back errored: 30 exceeded the context window and 7 hit transient errors. Match them by custom_id, split the 30 long documents into chunks, resend the 7 unchanged, and do not resubmit the 9,963 that succeeded.

Worked example 5.F: An independent reviewer

Situation. A coding agent writes a migration script and is then asked, in the same session, to "carefully review your migration for errors". It approves it. A human finds that it drops a column still read by a reporting job.

Diagnosis. Self-review in the same context inherits the generator's assumptions. The model is, in effect, re-reading its own argument.

Fix. Send the diff, the schema and the list of consumers to a second Claude instance with no access to the generation conversation, and a reviewer prompt with explicit criteria ("for each dropped or renamed column, list every reader in the provided consumer list"). Have it attach a confidence to each finding so a human checks the uncertain ones first.

Deep dive: anatomy of a good few-shot example

The guide stresses that few-shot examples should target ambiguous cases and show reasoning. A strong example has four parts:

A realistic input close to the boundary, not an easy case.

The output in exactly the target format.

A short reason naming the deciding factor.

The tempting alternative and why it was rejected.

<example>

Request: "My package says delivered but it isn't here."

Action: lookup_order, then offer a carrier trace; do not escalate yet.

Reason: Delivery disputes under $200 are within policy for the agent to resolve. Not chosen: escalate_to_human. Escalation is for policy gaps or explicit requests, and neither applies.

</example>

Two to four such examples, covering different sides of the boundary, usually beat a page of rules. The model generalises the reasoning to cases you did not list.

Deep dive: prompt chaining for multi-file review

A 14-file PR reviewed in one pass shows attention dilution: thorough comments on early files, thin ones on later files, and contradictory verdicts on identical code. The chain that fixes it: Per-file pass (can run in parallel): each file with its diff and immediate context; report local bugs and security issues using the shared criteria.

Integration pass: the list of changed interfaces, data flows and the per-file findings; report cross-file issues only, such as a renamed field still read elsewhere.

Consolidation: deduplicate, apply severity criteria consistently, and format for posting. This is a fixed pipeline, because the steps are the same for every PR.

Common traps

Choosing "tell it to be more conservative" to reduce false positives.

PROMPT ENGINEERING & STRUCTURED OUTPUT (20%)

Believing a strict schema guarantees correct values.

Making every field required.

Retrying when the data is simply not in the document.

Moving a blocking check to the Batches API to save money.

Asking the generating session to review itself.

Check yourself

Your invoice extractor invents due dates for invoices that have none. What schema change helps most?

Which tool_choice do you use when the document could be an invoice, a receipt or a contract, each with its own tool?

A nightly job must deliver results within 30 hours. How often do you submit batches? Why can strict JSON schemas still produce wrong data?