Harnesses Enter the "Attribution Era": When an Agent Fails, Is It the Model or the Framework? (In-Depth Review)

1. Background: agent = model × harness

A harness (the agent's runtime framework) is the layer of code wrapped around the model. It decides what the model sees (prompts, how tool results are serialized, how much history is kept), what it can do (tool interfaces, action parsing), and when it stops (budgets, verification, submission). Our October 27 review discussed how "harnesses can be optimized." This week's research goes a step further: a harness's contribution can be measured and attributed.

This matters now because almost every leaderboard reports the score of a "model + some harness" combination. A full-factorial experiment on lightweight coding agents already showed that the same model in different harnesses can differ by up to 44 percentage points in SWE-bench Verified resolve rate, and that harness rankings change with the model [11]. If credit and blame can't be separated, enterprises can neither choose correctly nor improve in a targeted way.

The details of that study are worth noting too. Per the paper, under unified inference serving, metering and scoring, the authors fully crossed six open-source harnesses with three lightweight coding models on 100 SWE-bench Verified tasks, for 1,800 scored runs. Evidence-based failure classification of 1,100 failed runs showed that a model's dominant failure mode in one harness isn't necessarily dominant in another. Efficiency also depends on the combination: even among harnesses that all use a "model-driven loop," billed tokens per resolved task can differ by more than 20× [11]. In other words, "which harness is better" has no fixed answer; it only makes sense for a given model and a given task.

October's research falls into four threads:

1. Attribution: is a failure the model's, the harness's, or a mismatch between them [1][2][3][7][8]?
2. Loss: in which segment inside the harness does information get lost [4][6]?
3. Verification: at which layer of the lifecycle should checking and revision go [5]?
4. Evolution: does a harness that modifies itself from experience actually help [9][10]?

2. Attribution: first separate "finished" from "done right"

2.1 Valid completion rate: an overlooked first dimension

When Agents Fail: Model or Harness (2026-10-23) [1] argues that evaluating a harness requires looking not only at task scores but also at the valid completion rate (VCR): whether an attempt ends normally and hands an evaluable result to the evaluator, regardless of whether the answer is right. VCR decomposes into "normal termination × valid handoff." On this basis, the paper defines model–harness fit, cross-model fit (the geometric mean across models), interaction residuals (strengths and weaknesses specific to a combination), and token load (including what failed attempts consume).

Per the paper, the authors crossed four general-purpose harnesses (OpenCode, Codex CLI, Pi CLI, DeepAgents) with four core models, plus two native model–harness systems, running 111 settings and 52,168 trajectories across six benchmarks. Findings include: harness rankings change with the model and benchmark; the combination with the highest task score may not be the one with the highest valid completion rate; and configurations with similar scores can have very different execution profiles. OpenCode ranked first in cross-model performance, and DeepSeek V4 Flash Preview kept a relatively high average VCR across harnesses on the shared four-benchmark panel [1].

Trial by Trace (2026-10-23) [2] comes at it from the other side. It splits harness reliability into seven components — context, execution, governance, lifecycle, observability, tools and verification — designs 95 expert-level, mechanism-oriented tasks, and uses three modules (trace tracking, multi-source evidence gathering and failure attribution) to give mechanism-chain attribution at the level of individual runs. Per the paper, across 3,420 runs (4 harnesses × 9 models × 95 tasks), harness macro-average scores range from 59.8% to 72.0%; 31.1% of runs succeeded at the task but violated at least one harness guideline [2]. In other words, nearly a third of "successes" are successes with hidden problems.

2.2 Vendor harnesses away from their own models

kw-bench (2026-10-26) [7] tests harnesses with a very "business" task: reconstruct five years of VAT data across three generations of SAP systems (legacy ECC, a partially migrated S/4HANA, and a consultant-redesigned version). The input is raw system exports — headers, local number formats, broken relational tables — with no documentation at all. Scoring is fully deterministic, with no LLM judge.

Per the paper, with the same model, DeepSeek v4.1 Flash, and a two-hour limit, two independent model-agnostic harnesses delivered results (OpenCode 0.542, micro-cc 0.528), while two labs' own harnesses scored 0 away from their own models. The author attributes the failure to "inertia": insufficient ability to keep many weak signals alive at once in a long session. With the frontier model Claude Opus 5.5, micro-cc and Claude Code reached 0.9986 and 1.000 in 22 and 25 minutes respectively [7]. The author's summary: below the frontier, the harness determines how far a model can go.

Free to Look, Free to Reason (2026-10-26) [8] reaches a similar conclusion for microservice root-cause analysis. Three simple harnesses with no preset call flow (one call over an evidence bundle, an SQL agent, and the Codex coding agent) achieve PR@1 of 65%–77% on three mid-tier models, above five published hand-orchestrated RCA agents (16%–53%) and statistical and causal methods (6%–39%). Case analysis shows that preset flows skip decisive evidence [8].

2.3 Algorithm breakdown ①: ceiling minus mismatch — why "comparing harnesses with frozen weights" misleads

Intuition. The usual practice is to take a trained model, put it in the old harness and the new harness, and keep whichever scores higher. Your Policy Is Secretly a Harness (2026-10-27) [3] points out that this "frozen comparison" conflates two things: how much the new harness itself can offer (the ceiling), and how well this particular model uses it (the mismatch).

Analogy. A skilled left-hand-drive driver switches to a right-hand-drive car and drives worse for the first week. That doesn't mean the right-hand-drive car is slower. The car's performance limit (the ceiling) hasn't gotten worse; the driver's habits (the mismatch) just haven't adjusted yet. After some practice, they may drive the right-hand-drive car better.

Formalization. Under KL-regularized control, the optimal policy is a softmax over action scores:

F(s,a) = Q(s,a) + τ·log π_ref(a|s), Π(·|s) = softmax(F(s,·)/τ)

The paper proves that for a policy trained to optimality under some harness, the log-odds between two actions equal exactly the score difference induced by that harness. The policy "carries" the harness's preferences in its weights, even though it contains none of the harness's code; that is what the title "your policy is secretly a harness" means. This yields the identity [3]:

Deployed return J = ceiling C − mismatch M

Here C is the best return any policy can reach under that harness, independent of model weights; M is the KL divergence of the policy from the optimal policy, accumulated along its own trajectories. Training changes only M; switching harnesses changes both C and M.

Small examples (per the paper).

• Changing only the order: reverse the field order in the web observation JSON. Content, actions, rewards and the reference policy stay the same, so every optimal decision stays the same (C unchanged). But the trained model changes its preferred action in about half the states, and success drops from 85.0% to 19.2%. The entire loss comes from M rising because of "readout drift" [3].
• Adding a better action: a new harness supports "one decision executes two clicks." The best achievable return rises, but the frozen model's mismatch increase is about ten times the ceiling increase, so success drops from 85.0% to 72.9% and a frozen comparison would reject it. After continued training with the same budget in both harnesses, the model in the new harness reaches 88.2%, above both the original checkpoint's 85.0% and the 83.4% from continued training in the old harness. The harness that initially "hurt" the model ended up producing the best agent [3].

How to use it. On Terminal-Bench Lite, the paper uses a stronger model as a stand-in for the unknown optimal policy to estimate C and M. "C − M" reproduces the accuracy ranking of five coding harnesses, while C alone or M alone does not [3].

# Evaluate a candidate harness H_new (relative to the existing H_old)
C_new  = estimate_ceiling(H_new, proxy=stronger_model)   # approximate optimum with a strong model
M_new  = estimate_mismatch(policy, H_new, proxy)         # current model's KL deviation
J_frozen = C_new - M_new
if J_frozen < J_old and (C_new > C_old):
# a frozen comparison would reject it, but the ceiling is higher:
# worth continued training or prompt tuning under H_new
candidate_for_adaptation(H_new)

When it fails. The mechanism experiments compile the control problem on MiniWoB++ into a finite graph, so the optimum can be computed exactly. On real coding tasks only a strong-model proxy is available, and estimation error depends on the gap between the proxy and the true optimum [3]. For enterprises that don't post-train and can only switch harnesses, the main value of this decomposition is a reminder: winning or losing one A/B test is not the same as the harness itself being good or bad.

3. Loss: context is a pipeline, not a window

3.1 Algorithm breakdown ②: the five-stage pipeline and "row-aware truncation"

Observation. Context Is a Pipeline, Not a Window (2026-10-26) [4] studies a tool-using agent that diagnoses injected faults from Linux kernel traces. Across 720 runs, the largest single prompt was only 23,000 tokens, 5.75% of a 400,000-token window. Yet of 11,189 tool results, 3,827 (about a third) were truncated before reaching the model, and every one of them became broken JSON. The cause: the harness cut off the tail of the JSON text at a fixed character count, and since JSON key order is fixed, later fields were deleted wholesale. In the same batch of results, one container list reached the model in only 4.7% of calls, while an adjacent list reached it 99.3% of the time [4].

More importantly, which list carries the fault evidence depends on when the faulty process started in the trace. For "newly started process" faults, the evidence is in the earlier list, scoring 40–55/60; for "already-running container" faults, the evidence is in the later list, scoring 0–24/60. The authors originally assumed this was a limitation of kernel tracing itself; only after measuring did they find the harness was to blame [4].

The five-stage model. The paper models context as five boundaries, each with a measured failure:

Analogy. A parcel goes from warehouse to recipient through five steps: sorting, loading, transfer, delivery and sign-off. Looking only at the sign-off rate won't tell you at which station the box got crushed; you have to weigh it at every station.

Fix: row-aware truncation. Change "cut a prefix by characters" to "keep whole record rows and note how many were omitted." Replaying both rules on 864 real tool results, the paper finds that over-budget results were all corrupted under prefix truncation and none were corrupted under row-aware truncation [4].

def deliver(result, budget):
rows = flatten_to_records(result)            # each record of each field
keep, used = [], 0
for field in result.fields:                  # rotate across fields so later fields aren't dropped wholesale
for row in round_robin(rows[field]):
if used + len(row) > budget: break
keep.append(row); used += len(row)
note = {f: f"{len(rows[f]) - kept(f)} rows omitted" for f in result.fields}
return valid_json(keep, note)                # always emit a parseable structure

Metrics. The authors recommend reporting the "survival amount" at each boundary next to accuracy, plus a summary metric, "yield" = findings recorded / computations run [4].

The controlled experiments are subtler than intuition suggests. In a preregistered experiment that reintroduced defects one at a time, the real cost of S1 truncation showed up first as extra work, not wrong answers: 43% more tool calls, double the compute, and yield falling from 0.61 to 0.31. When the agent still had other paths to the evidence, it could still find the right answer. The authors acknowledge that whether truncation lowers accuracy remains an open question [4].

How widespread it is. The authors examined the tool-output trimming code in current versions of seven agent frameworks. Two general-purpose trimmers cut by position (one keeps only the first 500 characters; one deletes the middle of observations over 20,000 characters). Three don't trim at all, which just postpones the same loss to the context window. Only one implementation cut by whole rows and noted the omission, and it wasn't the default [4].

When it fails. Row-aware truncation guarantees structural integrity, but you still have to decide which rows matter more. For results where length itself is a signal (for example, log frequencies), round-robin retention changes the distribution, so a statistical summary should be noted as well.

3.2 Stopping "hallucinated tool calls" before execution

StateMirror (2026-10-23) [6] handles the reverse direction of S1: the actions the model sends to the environment. Coding agents often propose actions that don't hold in the current environment, such as accessing paths that don't exist or importing packages that aren't installed. Analyzing 4,398 trajectories, the authors find these "hallucinated tool calls" are common and expensive: each recovery costs a median of 3 extra interaction steps and 22,400 tokens. StateMirror intercepts actions at the harness boundary, extracts preconditions ("path exists," "package installed") via AST and command analysis, and checks them deterministically against the live repository. Per the paper, on 2,508 actions it achieves precision 0.940 and F1 0.817, 35.3% above the strongest LLM judge (F1); detection latency drops from 17.73 s to 1.50 ms with zero token cost; more than 95% of hallucinated calls were corrected after pre-execution feedback; and in live mitigation on 79 tasks, execution time fell by 15.8% and token consumption by 11.8%, with the resolve rate unchanged [6].

4. Verification: which layer you check at matters more than how much you check

4.1 Algorithm breakdown ③: the four-layer verification framework

Rethinking Verification in Agent Harnesses (2026-10-25) [5] treats verification as an axis of test-time scaling and divides it into four layers along the agent lifecycle:

Why the split is needed. The authors observe that only 6.2%–13.6% of detected editing turns are followed by a recognizable check command before the next edit or the end of the run [5]. Agents don't instinctively like to check.

Results (per the paper). The authors compare 17 designs and 7 frontier coding models on Terminal-Bench (TB) and SWE-bench Verified (SWE):

• L0 and L1 give small, model-dependent average gains, with output tokens about 0.90–1.00× and 0.75–0.93× baseline.
• L2 can give larger gains, but "rejection" gates cost 1.60–1.76× baseline on TB and 1.48–1.56× on SWE.
• L3 independent review is more cost-effective: average accuracy rises 5.9–9.5 points on TB and 3.9–6.6 points on SWE, at only 1.13–1.25× and 1.00–1.04× baseline cost respectively (including review and revision).
• On VisualBench, made up of 18 visual and interactive tasks, the four independent-review designs have net preferences of +13.2 to +23.8 points, all better than L0–L2, and review based on "changes" beats review based on the "full trajectory." Adding review rounds or independent reviewers raises TB average accuracy by up to another 3.5 points [5].

Analogy. Take a manuscript. L0 is telling the author "read it over once when you're done"; L1 is a reminder after every paragraph; L2 is the editor saying "check it again" when the author says "done"; L3 is handing it to another editor. Experience says a fresh pair of eyes often works better than having the author reread it three times, and the data supports that intuition.

When it fails. Independent reviewers do better when they look at "changes" rather than the "full trajectory," which shows that the reviewer's input needs to be designed. If the changes themselves are scattered (for example, a refactor across dozens of files), review cost rises quickly. Also, all seven models are frontier coding models from Chinese vendors; how strongly the conclusions hold for other model families remains to be verified [5].

5. Evolution: a self-modifying harness has to prove it helps

How Do Coding-Agent Harnesses Self-Evolve? (2026-10-26) [9] studies a skill that has the agent "review the session and propose harness changes." The authors ran Claude Code on 30 SWE-bench Verified issues, qualitatively coded 125 change proposals, implemented 24, and evaluated 12 on later issues in the same repositories using fresh sessions with and without the change. Per the paper, most proposals pointed to a specific event in the session, but 18 of the 43 proposals related to code-style checks were "generic good habits" that would be recommended no matter what happened. In 7 evaluations where the session without the change actually ran into the target problem, 2 changes eliminated the target action, 1 reduced it and 4 did not; some changes also added work on unrelated issues [9].

Lessons Outlive Their Failures (2026-10-23) [10] is a six-week personal case study: three AI tools edited a pure-Markdown repository daily under the same set of rules. The author kept a ledger of 118 agent failures. Per the paper, documentation drift (outdated statements) was the largest failure category (48/118). A single LLM documentation audit found about half of the known drift (Claude Sonnet 5.5 averaged 50%), and at least 93% of its findings were correct. Of five historical failures replayed on Sonnet 5.5, four tasks completed, and in three of them the failure no longer occurred with or without the fix [10]. Only one recurred after its check was removed — a "stale count" failure — and the corresponding mechanical check caught it in every run [10]. The author's conclusion: a rule written to fix one failure may outlive the failure itself. After a model upgrade the failure no longer happens, but the rule stays in the file, while documentation drift doesn't disappear with model upgrades and still needs mechanical checks. Self-evolution tools add rules faster than they retire them, so rules should be retested regularly and outdated ones retired [10].

Both studies point to the same gap: harness evolution needs more than "generating and implementing changes"; it also needs to prove that changes help on later tasks, and to revise or delete the ones that don't [9][10].

6. Side-by-side comparison

Note: tasks, models and metrics differ across studies, so the numbers are not directly comparable.

7. Our view

First, harness evaluation has to move from "one score" to "a health report." Task score, valid completion rate, guideline violations, token load and survival rate at each boundary — at least these five dimensions should be reported side by side [1][2][4]. The 31.1% of "successes with hidden problems" [2] means that enterprises looking only at success rates will carry latent risks into production.

Second, the advantage of a "vendor harness + vendor model" pairing can turn into a disadvantage when you switch models. kw-bench's zero scores [7] and the ceiling–mismatch decomposition [3] say the same thing: models and harnesses adapt to each other. Enterprises with a multi-model strategy should prefer model-agnostic harnesses with transparent behavior, and accept each combination separately.

Third, low-cost improvements are often in the pipeline, not the model. Row-aware truncation [4], pre-execution checks [6] and independent review [5] require no model change or retraining, yet directly cut waste and errors. We recommend that enterprises instrument every harness boundary before "upgrading the model."

Fourth, self-evolution must come with controlled experiments. Having an agent "review and change the rules" is easy; proving the change helps is hard [9]. Rules that only accumulate eventually become a new source of drift [10]. A harness's rule library should be treated like code: with tests, owners and a retirement process.

8. Outlook

6–12 months:

• Mainstream harnesses will offer structure-aware tool-output trimming by default and log both "what the tool produced" and "what the model received" [4].
• Leaderboards will start reporting "model × harness" matrices and valid completion rates, not just the score of a single combination [1][11].
• Independent review (L3) will become a common default for coding agents, with "changes" as the main review input [5].

1–3 years:

• Post-training will explicitly account for harness transfer: rotating harness presentations during training to reduce "readout drift" and teach models transferable harness skills [1][3].
• Harnesses will gain "continuous integration for rules": each rule carries the historical failure cases that triggered it, is replayed regularly, and is automatically flagged for retirement when it no longer has an effect [9][10].
• For high-value investigative knowledge work (audit, tax, due diligence), "investigative benchmarks + deterministic scoring" will become a common way to evaluate purchases [7].

Practical checklist: six steps for a harness health check

1. Fix the model and rotate harnesses; then fix the harness and rotate models. Run at least a 2×2 cross to see interaction residuals [1][11].
2. Record four kinds of results together: whether the task succeeded, whether it terminated normally with a valid handoff, whether it violated harness guidelines, and how many tokens it consumed (including failed attempts) [1][2].
3. Measure survival at every context boundary: how many records the tool produced, how many the model received, how many were retained and how many were handed off [4].
4. Run deterministic checks before actions are issued, so errors that rules can catch are blocked before execution [6].
5. Set up independent review before accepting results, with changes as the main review input, and record the extra cost review adds [5].
6. Attach the triggering historical cases to every new rule, replay them regularly, and send rules that no longer have an effect to retirement review [9][10].

These six steps need no model change and no training; any existing agent project can complete a first round in one to two weeks. The output isn't a higher score but a health report that can answer "where is the problem?"

9. Implications for financial risk control and enterprise deployment

1. Treat the harness as an audited object. For agent-assisted credit document checks and anti-money-laundering lead analysis, record not just the model version but also the harness version, tool-output trimming rules and verification configuration; otherwise failures can't be attributed afterward [1][2].
2. Measure "evidence survival" at every boundary. Long structured results such as credit reports and transaction statements are easily "deleted wholesale" by prefix truncation; trim by record rows instead and note how many were omitted [4].
3. Use deterministic rules for pre-execution checks. Queries on nonexistent fields and calls to unauthorized APIs should be blocked by rules before execution, not recovered from after an error [6]. Great Unicorn Intelligence's rule-engine practice shows such checks are low-cost with clear benefits.
4. Configure independent review for high-risk conclusions. Having reviewers look only at "changes and conclusions," not the whole reasoning trajectory, is more cost-effective [5].
5. Give the rule library a retirement mechanism. Rules added temporarily for an incident should be replayed and verified in later versions, and retired promptly when no longer needed, to avoid rule buildup causing conflicting standards [10].

10. Conclusion

The engineering question of the agent era is shifting from "is the model strong enough?" to "can the system be seen clearly?" This batch of October research provides tools to make harnesses attributable: separating finished from done right, ceiling from mismatch, the loss in each pipeline segment, and which verification layer actually works. Only once you can see clearly can you improve.

To learn more about Great Unicorn Intelligence's work on rule engines and intelligent decision systems, visit www.hxallai.com.

References

1. [1]Shao Zhang, Hangfan Zhang, Yuanyuan Shi, Hao Li et al. When Agents Fail: Model or Harness—Who's to Blame? arXiv, 2026-10-23. https://arxiv.org/abs/2610.32483
2. [2]Zhiyong Cao, Hengjie Zhu, Yongxin Wang, Xuchen Li et al. Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution. arXiv, 2026-10-23. https://arxiv.org/abs/2610.32285
3. [3]Jian Mu, Ruibo Shen, Yao Shu. Your Policy Is Secretly a Harness: The Decision Geometry of Agent Interfaces. arXiv, 2026-10-27. https://arxiv.org/abs/2610.37255
4. [4]Yuvraj Sehgal, Mahsa Panahandeh, Naser Ezzati-Jivan. Context Is a Pipeline, Not a Window: Information Attrition in LLM Agent Harnesses. arXiv, 2026-10-26. https://arxiv.org/abs/2610.36114
5. [5]Zhengxin Zhang, Erkang Zhu, Yin Lin, Claire Cardie et al. Rethinking Verification in Agent Harnesses. arXiv, 2026-10-25. https://arxiv.org/abs/2610.34577
6. [6]Shiwen Shan, Yintong Huo, Zhiying Wu, Yuxin Su. StateMirror: Pre-Execution Detection of Hallucinated Tool Calls in Coding Agents. arXiv, 2026-10-23. https://arxiv.org/abs/2610.32648
7. [7]George Juraj Salapa. kw-bench: Breaking Frontier Harnesses on Real-World Tasks. arXiv, 2026-10-26. https://arxiv.org/abs/2610.36405
8. [8]Tingzhu Bi, Xun Zhang, Jinglin Li, Ping Wang, Meng Ma. Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents. arXiv, 2026-10-26. https://arxiv.org/abs/2610.35767
9. [9]Christoph Treude, Sebastian Baltes. How Do Coding-Agent Harnesses Self-Evolve? An Empirical Study of Retrospective Adaptation. arXiv, 2026-10-26. https://arxiv.org/abs/2610.34938
10. [10]Hassan Hassanzadeh Aliabadi. Lessons Outlive Their Failures: A Case Study of Rules for AI Coding Agents. arXiv, 2026-10-23. https://arxiv.org/abs/2610.32578
11. [11]Lu Zhao, Christopher M. Poskitt, David Lo. Pairing Matters: Performance, Failures, and Efficiency in Lightweight Coding Agents. arXiv, 2026-10-23. (background) https://arxiv.org/abs/2610.32131