Wrap the model’s probabilities in engineering certainty

How it works

1. Constrained decoding for complianceStructured outputs make call arguments match the schema exactly, removing the whole class of “malformed call” failures from the system.
2. Dependency-graph parallelismCalls are compiled into a DAG and independent branches run concurrently — measured latency drops to about a third and cost to about a sixth, with a slight accuracy gain.
3. Retrieve before you callWith thousands of interfaces, first recall candidate tools, then generate the call — rather than stuffing every tool into the context (see C2).
4. Observe-and-call loopResults are fed back and the agent decides again — the prototype of every multi-step agent behavior; keep failed observations in the context so mistakes aren’t repeated.

Parallelism is not only faster and cheaper — it is also more accurate, because shorter chains accumulate fewer errors.

How it works

1. Context costEach tool definition takes roughly 150–400 tokens; 30 tools take about 8,000 tokens, 100 tools about 35,000, and connecting seven MCP servers can eat half the window.
2. Accuracy costGoing from 10 to 50 tools lowers tool-selection accuracy by 15–25% — choice overload is real and measurable.
3. Cache costAdding and removing tools dynamically invalidates the prefix and throws away the cache; instead, keep the tool set fixed and restrict what is available with output masking (logit masking).
4. Semantic tool searchWhen a large tool count is unavoidable, retrieve candidates first and then call — treat the tool catalog itself as a searchable knowledge base.

Engineering guidance: trigger compaction once context utilization passes about 70%; compress history 3:1–5:1 and tool output 10:1–20:1. The numbers vary by scenario, but “there must be a budget and there must be thresholds” is the consensus.

How it works

1. Predictive routingA difficulty classifier trained on preference data picks the tier before the request runs; measured results keep about 95% of quality while saving about 85% of cost.
2. Cascade fallbackA cheap model answers first with a confidence score and escalates only when it falls short — where simple requests dominate, cost can drop by up to about 98%.
3. Routing as a productFrontier models now route compute by difficulty in real time and gateway products offer unified routing; the era of “one application, one model” is over.
4. Routers need upkeepThe difficulty mix drifts as the business changes; routers need regular retraining and online monitoring, or the savings erode month by month.

A conservative caveat: public figures for routing savings range widely (25%–85%) and depend heavily on the traffic profile. Measure with A/B tests on the bank’s own traffic rather than taking vendors’ upper bounds at face value.

How it works

1. Speculative decoding (lossless)A small model drafts several tokens and the large model verifies them in one parallel pass, accepting or rolling back; the output distribution is identical to the original model, with measured 2–3× speed-ups and more than 6× for advanced methods.
2. Prefix-cache reuseOnce static prefixes such as the system prompt and tool definitions hit the cache, reads cost about an order of magnitude less than misses.
3. Continuous batchingThe serving layer interleaves decoding steps from different requests, markedly raising GPU utilization and overall throughput.
4. Quantization trade-offs8-bit weights and activations save about half the memory almost losslessly; 4-bit saves more but loses 1–3% accuracy — weigh it per scenario.

How it works

1. Generations one to threeNaive vector search → re-ranking and query rewriting → graphs and community summaries, step by step fixing “retrieves the wrong thing” and “can’t answer global questions”.
2. Generation four · self-directed retrievalThe model judges whether the evidence is sufficient, whether to ask a follow-up and whether to change angle, forming a multi-hop loop.
3. Reflective retrievalThe model emits reflection tokens — retrieve or not, is it relevant, is the answer supported — retrieving on demand and critiquing itself.
4. Rule one · don’t put everything in a graphFor simple factual questions, vector search is faster, more accurate and cheaper; only multi-hop and global questions justify the cost of building a graph.

The three approaches coexist rather than replace each other: choose by question type, not by “always use the newest method”.

How it works

1. Community summaries for global questionsThe entity graph is clustered hierarchically and summarized in advance, and thematic questions are answered by merging summaries — saving several to dozens of times the tokens of reading everything.
2. Deterministic multi-hop traversalTraverse typed relationships deterministically (equity → guarantee → related-party transaction) — something probabilistic retrieval cannot do.
3. Temporal graphsFacts carry valid-from and valid-to stamps; marking them invalid instead of deleting them lets the system answer “what was the state at the time?”
4. Costs have fallen sharplyLightweight graph approaches cut indexing and query cost to a small fraction of traditional ones — graphs are no longer a luxury.

A shared gap: neither natively supports governance — access control can only live in the application layer, and freshness is bound by the ingestion cycle. That is what A6, the knowledge and retrieval layer of the end-to-end solution, sets out to solve.

How it works

1. Writes are asynchronousAfter an event is written, extraction, consolidation and reflection run in the background with observable lag — assume long-term memory is eventually consistent, and never rely within the same turn on something just written.
2. Namespaces are isolation boundariesOrganize namespaces in three levels — subject, session, strategy — and use the namespace as a permission condition, so multi-tenant isolation is enforced at the authorization layer rather than in application code.
3. Writes are rate-limitedEvent writes for one subject’s session are usually limited to single digits per second, which naturally serializes writes (in line with read–write asymmetry).
4. Forgetting must be provableConfigurable event retention (say 7–365 days) plus deletion by ID is the basis for compliant deletion; but provable deletion from vector indexes is still an industry gap.

How it works

1. Keep the prefix stableA single-token difference invalidates the cache for everything after it — so never put timestamps or anything else that changes per call in the system prompt.
2. Append, don’t rewriteContext grows by appending, and serialization must be deterministic (an unstable key order silently destroys the cache).
3. Fix the assembly orderSystem instructions → tool definitions → long-term memory → retrieved knowledge → conversation history → current request; a stable order maximizes prefix reuse.
4. Never compress the red-line zoneCompliance constraints and permission boundaries sit in a fixed block that is re-injected unconditionally whenever context is rebuilt (the constraint pinning behind the judgment that “compaction erases safety constraints”).

How it works

1. Central orchestrationA lead agent breaks down the task, dispatches parallel subtasks and synthesizes the results, keeping a task ledger and a progress ledger so it doesn’t lose its way. Bank prototype: the customer-service workbench.
2. Hierarchical pipelineStandard operating procedures are encoded as an agent pipeline with structured hand-offs. Bank prototype: the centralized operations line.
3. Debate to consensusSeveral agents answer independently, read each other, revise and converge — markedly improving factuality. Bank prototype: the bull-and-bear debate of a credit committee.
4. Dynamic routingControl is handed to the best-suited specialist agent according to conversation state. Bank prototype: the customer-request triage desk.

How it works

1. Fan-out readsSub-agents search, gather evidence and verify in parallel, each in its own context, returning only condensed summaries (typically 1,000–2,000 tokens).
2. Single-point writesEvery persist, submit, send and order action converges on one writer (an agent or a human node), eliminating write conflicts by design.
3. Concurrency controlDatabase-style concurrency control (optimistic locks, version numbers, transaction boundaries) for shared state — dedicated 2026 work brings this into multi-agent systems.
4. State visibilityReaders see a versioned snapshot and the writer checks the version on commit — avoiding writes made on stale information.

How it works

1. Externalize the planWrite the plan to an external to-do list and restate it periodically, countering long tasks that “forget the goal along the way”.
2. Expensive plans, cheap executionPlanning on a strong model and executing on cheaper ones is now a standard cost lever; “plan mode” is a product feature in mainstream coding agents.
3. Context isolationSub-agents explore in clean contexts and may use tens of thousands of tokens, but return only a 1,000–2,000-token summary to the lead agent.
4. Re-plan on failureA failed step doesn’t mean giving up; control returns to the planner to adjust the route — the main source of robustness on long tasks.

How it works

1. Checkpoint and resumePersist computation and state at key points and resume from the latest checkpoint after a crash, instead of rerunning from scratch (reruns are costly and may repeat side effects).
2. Suspend for human approvalA workflow can be serialized and suspended while it waits for approval, then resume in place once approved — the technical precondition for human–machine collaboration.
3. Deterministic replayRecord the results of external calls and reuse them on replay instead of calling again, so the recovered run behaves exactly like the original path.
4. Session-length limitsMainstream runtimes cap a single session at about 8 hours; anything longer must be designed as a segmented workflow that can resume.

How it works

1. External signals are a mustWithout external feedback (test results, validators, retrieved evidence), purely internal self-correction is usually useless or even harmful — reflection needs something to reflect against.
2. Identity relabelingPresenting the content to be checked as “externally submitted material” markedly raises explicit correction rates (by tens of percentage points in most model–domain combinations tested).
3. Cap the roundsOver-checking suppresses correct answers and bloats reasoning; cap it at one or two rounds with an early-stop condition.
4. Keep the lessonsPut failure lessons into words and write them to the experience store (see T4 / C7), so reflection pays off across tasks, not just this once.

How it works

1. Calibrate before go-liveCalibrate on 200–500 expert-labeled samples, aiming for a correlation above 0.85 with the human baseline, and revalidate every quarter.
2. Control three biasesPosition bias (favoring earlier or later answers), verbosity bias (overrating long answers) and self-preference (favoring outputs from the same model family) — mitigated respectively by random swapping, length normalization or penalties, and cross-model judging.
3. Score by dimensionDon’t ask for an overall quality score; score accuracy, completeness, compliance, readability and other dimensions separately and output structured results.
4. Tiered deploymentReal-time blocking checks (injection detection, personal-data detection) must finish within a few hundred milliseconds and need small models; in-depth quality assessment runs asynchronously.

Even at 85%–88% agreement, more than one judgment in ten still differs from a human’s — so judges are for sampling and ranking, never for the final ruling.

How it works

1. Plan confirmationConfirming the plan and its boundaries once before execution costs far less than step-by-step correction (see C11).
2. Dangerous-action confirmationIrreversible actions — delete, send, pay, submit externally — require explicit confirmation, and the prompt must summarize the consequences.
3. Low-confidence escalationBelow a confidence threshold, or after failed retries, escalate to a person — and monitor the escalation rate itself as a quality metric.
4. Final sign-offDecisions involving funds or significant customer rights must have a human final sign-off — in China this is an explicit regulatory requirement, not an option.

How it works

1. ① Input sideAll external content is marked untrusted and physically separated from system instructions — tool output included (a tool description can itself hide instructions).
2. ② Architecture sideTwo models with a restricted interpreter that explicitly tracks data and control flow, attaches a capability label to every value and rules on actions by policy.
3. ③ Execution sideSandboxed execution, outbound allow-lists and a ban on dangerous tool combinations (never put shell, file system, browser and email in one unconstrained workflow).
4. ④ Output side and ⑤ human reviewOutput filtering and data-loss prevention; high-risk actions require human confirmation as the last gate.

How it works

1. Trajectory synthesisRun tasks with a strong model in real or simulated environments and keep the successful trajectories as training data; quality matters more than quantity.
2. Trajectory SFTImitation learning lays the foundation, teaching the model roughly how this kind of task is done.
3. RL with verifiable rewardsGRPO-family algorithms refine behavior over multi-turn interaction, directly optimizing task completion.
4. Distill downDistill the capability into small models for narrow tasks; with techniques such as first-thought prefix guidance, small models can gain retrieval and code-tool skills.

How it works

1. Multi-lens verificationGive each conclusion different lenses — compliance, risk, reproducibility — which find more problems than several identical verifiers.
2. Debate to convergenceTwo sides answer independently, read each other and revise, making disagreement visible — especially suited to credit and valuation, where judgment is subjective.
3. Majority voteUse majority rule for judgments that can be discretized, and record minority opinions as leads for review.
4. Heterogeneous cross-checksVerify with different models and data sources so models of the same lineage don’t share one blind spot (same-family models show self-preference bias).

How it works

1. ① Offline evaluationBuild stratified evaluation sets (report by query type to avoid Simpson’s paradox) and report pass^k and variance rather than a single best run.
2. ② Shadow runRun in parallel with the human process for one full business cycle, comparing without deciding, and focus on the samples where they disagree.
3. ③ Staged rolloutOpen up traffic step by step with automatic rollback thresholds; platform products already offer traffic splitting and versioned configuration.
4. ④ Online monitoringContinuous sampled evaluation plus drift detection against a validated golden baseline, turning evaluation from “once before launch” into “always on”.

A key reminder: with 100 samples and an 80% pass rate, the confidence interval is roughly 72%–88% — a small evaluation set simply cannot support claims like “we improved by 3 points”. A paired design (the same samples before and after) cuts the sample requirement by an order of magnitude.

How it works

1. Four levelsSession (one customer interaction) → task (one goal achieved) → step (planning / retrieval / generation) → tool call (including downstream system spans), each level drillable.
2. Standard fieldsThe industry now has tracing semantic conventions for generative AI, covering operation type, model, token usage, cache reads, compaction events, time to first token and other key attributes.
3. A family of metrics, not oneMonitor task success rate, pass^k, cost per task, P95 latency, human escalation rate, complaint rate, hallucination rate, cache hit rate and tool-call success rate side by side.
4. Optional capture of sensitive fieldsPrompt and output content is an optional capture item; in finance it needs a separate compliance switch and masking — observability does not mean retaining all raw text.

Dashboard discipline: quality, cost, experience and risk must appear on the same screen. Watching only success rate hides runaway cost; watching only cost hides a worsening experience — optimizing a single metric always has side effects.