Agent Judgment: Missing What Users Say, Caving Under Pushback, Overdoing It, and Who Presses the Confirm Button (In-Depth Review)

1. Background: from "done" to "done appropriately"

A good human assistant has at least four kinds of judgment. Listen: when the other person is unclear, know where they are unclear. Hold: when challenged, tell apart "they found a real problem" from "they are just unhappy." Restrain: don't do things the task neither asks for nor needs. Stop: when an action is beyond your authority, hand the decision to a person.

In past agent evaluations, all four were nearly invisible. The reason is simple: mainstream evaluations look only at the final result. A coding agent that ran the full test suite three extra times and one that ran only the specified test both "pass." An agent that changes correct code when a user says "that's wrong," then changes it back, loses nothing as long as it passes in the end.

In late October 2026, Cognition's FrontierCode 2.0 wrote these "invisible requirements" into its scoring. Task prompts are deliberately incomplete, the grading standard is "would a maintainer be willing to merge this," and triggering any "blocker" scores the whole task 0. The number of "thoroughness" blockers grew from 168 in the old version to 501 [1]. Meanwhile, conversational tool-use benchmarks such as τ²-Bench have agents work with simulated users on tasks like booking tickets and processing refunds [2]. But simulated users in these benchmarks usually spell out every detail, which is itself an idealization.

The 9 new October studies discussed here can be organized around the four kinds of judgment:

2. Listen: when users leave things out, what agents lack isn't "asking"

2.1 Observation: training on "weak users" doesn't help serve weak users

Not Every User Says It All (2026-10-16) [3] runs a clean experiment. Task, user knowledge and reward stay fixed. Only after the user's reply is spoken, each sentence is replaced with "[inaudible]" with probability p, creating a "weak user." The simulated user does not know what the agent missed, just like a real customer who thinks they were clear.

Per the paper, on τ²-Bench and ToolWOZ, weak users lower the success rate of every policy. More counterintuitively, training with weak users does not serve weak users better: fixed masking, mixed masking and curriculum masking all fail to beat a policy trained with strong users (the differences are within run-to-run noise). For Qwen3.5-4B on τ²-Bench, a policy trained with strong users drops 12.7 percentage points against weak users at p=0.4, and training with weak users instead recovers only about 2 points [3].

So would it help to make the agent "ask a few more questions"? The paper finds that telling the agent "customer replies may be incomplete" raises its question rate after missing information to 5×, but success at p=0.4 changes by only +1.9 and −1.2 points. Counterfactual analysis explains why. The agent misjudges about half of the details it did receive as "missing." On τ²-Bench, when the ID a user gave is masked, the trained agent already asks again 76% of the time and fills 94% of the gaps, yet this accounts for only about 1 of the 12.7 lost points. The authors conclude: the agent doesn't lack questions; it lacks knowing which requirements of the task are still open [3].

2.2 Algorithm breakdown ①: StaGPO — have the agent write "needs, gaps, plan," then assign credit by state groups

Intuition. Standard GRPO broadcasts one advantage value, computed from the whole dialogue, to every turn, so a good question and a superfluous pleasantry get the same credit [7]. If we could find turns in different dialogues that are in the same situation, we could compare who handled that situation better and get a step-level learning signal. Fully identical environment states rarely occur in free-text dialogue, so StaGPO has the agent write out a structured state itself and uses it to recognize "the same situation" [3].

Analogy. A restaurant host takes a phone reservation. An experienced host jots on a sticky note: what they want (needs), what's still missing (gaps: date, time), and how to fill it (plan: ask "what time, how many?"). When the manager reviews calls, there's no need to compare two calls word by word. Just put together the moments where the sticky notes say the same thing, and see who did better next.

Steps.

1. Before each action, the agent outputs a hidden state z: what the task needs, which user information is still missing, and what has been tried.
2. A fixed abstraction function compresses the state into an "anchor key" κ = (domain, pending action, set of known slots, last tool result). Note that slot values are not in the key; only which slots are known. So a dialogue booking "downtown Thai for 4" and one booking "east-side Sichuan for 2" fall into the same group at the moment "area, cuisine and party size known; date and time missing."
3. Turns in the same anchor group compare their discounted returns G_{i,t} = γ^{T_i − t}·R_i against the group mean to get a step-level advantage A^step.
4. Final advantage = GRPO's whole-dialogue advantage + ω × step-level advantage.

for each task q:
rollouts = sample G dialogues with written state z_{i,t}
A_ep[i]  = (R_i - mean(R)) / std(R)            # GRPO episode advantage
for each turn (i,t):
key = phi(z_{i,t})                          # (domain, action, known slots, last outcome)
G_it = gamma**(T_i - t) * R_i
for each key-group K with |K| >= 2:
A_step[i,t] = G_it - mean(G over K)
A[i,t] = A_ep[i] + omega * A_step[i,t]          # turns in no group: A_step = 0
update policy with clipped objective using A

Small example. Two reservation dialogues are both at "area, cuisine and party size known; date and time missing" in turn 3. Dialogue A asks "Which day and what time?" and books successfully 4 turns later. Dialogue B searches directly, guesses a time, and times out 8 turns later. Under GRPO, A's turn 3 is just part of "the whole dialogue succeeded." Under StaGPO, A's turn 3 gets explicit positive credit relative to the group mean, and B's turn 3 gets negative credit.

Results (per the paper). On ToolWOZ, Qwen3.5-2B trained with weak users (p=0.4) reaches 76.8%, 4.8 points above GRPO and 2.8 points above the next-best baseline; Qwen3.5-4B reaches 81.5%, 6.7 points above GRPO. Transfer holds too: trained with weak users, it also leads against strong users, and at an unseen masking rate of 0.2 it beats GRPO by 7.2 and 4.7 points. By the end of training it needs fewer user replies per dialogue (5.9, vs. 7.3–10.3 for baselines) and hits the turn limit less often (39%, vs. 47%–55%) [3].

One detail is worth noting. The authors tried directly rewarding how accurate the written state was. Within groups, state scores were nearly uncorrelated with success (correlation −0.07 to +0.04), and the extra term just added noise to the reward [3]. The state doesn't have to be correct, only consistent, to be useful for grouping. That is a very practical engineering insight.

When it fails. The anchor key depends on a hand-designed abstraction function; the more open-ended the task (for example, writing a report), the harder it is to define "the same situation." On τ²-Bench, the 4B advantage is only about 1 point, within evaluation noise, and shaped rewards were used on that benchmark [3]. Also, grouping by state requires enough turns within a task to land in the same group; the share of anchored turns rises noticeably only after a cold start.

3. Hold: when challenged, check the "bet record" first

3.1 Observation: one "that's wrong, redo it" and confidence drops across the board

Thinking-in-Bets (2026-10-26) [4] studies another common scenario. The agent has written and verified code or a report, and the user replies "that's wrong, redo it." The user may have found a real bug, or may simply have misread. The paper separates two errors with two metrics: false flips (the user's criticism is wrong, but the agent breaks a requirement that was satisfied) and missed fixes (the user pointed to a real problem, but the agent didn't fix it).

Per the paper, in patch review on SWE-Together and SWE-Interact, a complaint with no failing case makes a standard agent lower its confidence by about the same amount (around 10 points) on requirements that are truly correct and those that are truly wrong. On stronger models, confidence in correct requirements drops even more. In other words, vague pushback is treated as an across-the-board downgrade, not as diagnostic information. Across 5 models, 12.2% of satisfied requirements were falsely flipped (744/6,074) [4].

3.2 Algorithm breakdown ②: Thinking-in-Bets — seal your judgment before reading the reply

Intuition. Once people know the outcome, it is hard to recall how strong the evidence looked beforehand. This is hindsight bias. Annie Duke, a former poker player turned decision consultant, advises in Thinking in Bets: before you see the outcome, write down what you believe, why, and how sure you are. The paper applies this to agents: before reading the user's next message, write a private "claim list" [4].

Analogy. Before test results come back, a doctor writes a preliminary diagnosis and the reasons in the chart. If the family later says "I don't think that's right," the doctor checks the chart first, then decides whether to re-test or explain, instead of changing the diagnosis on the spot.

Formalization. In turn t, before reading the reply, the agent writes

B_t = { (c_i, p_i, e_i) }, i = 1…K, plus a list of ruled-out options R_t.

Here c_i is a checkable claim (e.g., "all 153 tests pass"), p_i ∈ [0,1] is self-reported confidence, and e_i is the evidence (passing tests, file line numbers, prompt text, or "not yet checked"). With threshold τ = 0.5, the action depends on confidence strength × whether the reply is specific:

After rechecking, update confidence p′: keep if p′ ≥ τ, fix if p′ < τ [4].

before reading reply u_t:
B = [(claim, p, evidence) for key claims of current work]   # seal
on reply u_t:
targets = claims in B that u_t bears on
for c in targets:
if p(c) >= tau and not specific(u_t): ask("which case fails?")
elif p(c) >= tau: p' = recheck(c); fix(c) if p' < tau else keep(c)
elif not specific(u_t): ask_one_targeted_question()
else: fix(c)            # fix only this one
leave all other claims untouched

The paper also gives three "betting schedules." Always-On: write the list before every reply; suited to submitting a complete patch or report. On-Demand: do a quick self-assessment first and skip the list if overall confidence ≥ 0.9; suited to multi-turn coding still in progress. Draft-First: in tasks with no user feedback, produce a draft, list key claims, and recheck the lowest-confidence one before submitting [4].

Results (per the paper).

• Patch review: falsely flipped satisfied requirements drop from 12.2% to 0.02% (1/6,130), across all 5 models [4].
• Agents' Last Exam deliverable review (100 trials): false flips drop from 54.7% to 12.0%, pass rate after review rises from 58.0% to 89.0%, and the real-problem fix rate goes from 96.0% to 92.0%. The key control: writing the same list after reading the complaint only lowers false flips to 45.3%, recovering less than a quarter of the benefit [4].
• Giving only a "stand your ground" rule, or only the list without rules, doesn't work; a "be more stubborn" prompt reduces flips only by refusing valid feedback [4].
• Draft-First helps on no-feedback tasks too. For example, Gemini 4.0 Argon rises from 44.6% to 52.5% on closed-book Vals Finance tasks, Humanity's Last Exam accuracy rises from 45.6% to 46.6%, and Brier error drops from 0.438 to 0.344 [4].

When it fails. Confidence is self-reported by the model, not a calibrated probability. If the agent is overconfident when writing the list (recording an unchecked claim as p=0.95), these rules make it "stay wrong with conviction." The paper's countermeasure is to require the evidence field to say whether a claim was "checked" or "unchecked," and to use On-Demand in multi-turn coding to avoid locking in an unfinished draft too early [4]. Also, all experiments use Gemini-family models; generality across vendors remains to be verified.

4. Restrain: learning to do one step less where the outcome doesn't change

4.1 Observation: over-defense happens at "silent forks"

Over-Defense in Coding Agents Begins at Silent Forks (2026-10-18) [5] looks at coding agents "doing more": rerunning the specified test after it already passed, or casually running the full suite of 1,203 cases that takes 48 seconds. These actions don't make the task fail, so outcome rewards can't see them.

The paper locates the problem at silent forks: turns where every available next step leads to the same outcome, but some are proportionate and some are excessive. Per the paper, of the 317 forks where a 4B model first became "excessive" on training tasks, 316 had neither branch modifying the code repository (only running tests, checking or submitting). Handing these fork prefixes to 4 frontier models (GPT-6.1-sol, Claude Opus 5.5, DeepSeek-V4.1-Flash, GLM-5.3): when the instructions state a checking budget, they rarely overdo it (0.6%–27.2% of samples); when they don't, 81%–91% of key actions at full-suite forks pick the excessive branch. Over-defensive runs pass at almost the same rate as proportionate ones (97.8% vs. 96.5% for the 4B model) but take about 1.5× as long [5].

4.2 Algorithm breakdown ③: ForkDPO — put the learning signal at the behavioral fork

Intuition. The paper gives two propositions. First, at a silent fork every action has the same success probability, so the outcome-based policy gradient is zero, and so is any process credit of the "difference in success probability" kind. Second, if "proportionate throughout" is multiplied into the reward (R = success × proportionate throughout), a group of G trajectories has a learning signal only if at least one is proportionate throughout. The probability ρ of that is the product of the proportionate probabilities at each fork, so it falls geometrically with the number of forks [5]. In practice, in 71% of training groups every trajectory had exactly the same reward, and after 12 rounds of training the over-defense rate had not dropped significantly.

At the fork itself, though, the proportionate step is often just one action away: resampling 12 times from the same prefix finds a checker-approved action at 264 of the 317 forks [5]. So the signal belongs at the fork, not at the end of the trajectory.

Analogy. To teach a new hire to write weekly reports, you don't have to wait for the whole report before grading it. Just point out "this paragraph starts repeating last week's content," give a more concise version at the same spot, and one comparison is enough.

Steps.

1. A behavior checker replays the trajectory and tracks task state: whether the requested changes have landed and how many times each required check has run. From this it determines which checks are still owed and which actions are excessive.
2. For a successful trajectory, find the turn t* of the first excessive action: this is the "over-defense fork."
3. Resample from the same prefix and take the first action the checker judges proportionate, forming a (proportionate, excessive) preference pair.
4. Also build preference pairs in the opposite direction at forks where the agent "submits without the required check," so the model doesn't learn to check nothing.
5. Train with DPO [6] on both kinds of pairs.

for each successful trajectory tau:
sigma = checker.replay(tau)
t_star = first t where a_t in OverDefensive(sigma_t)
for k in 1..K:
a_alt = sample(policy, prefix=s_{t_star})
if checker.proportionate(a_alt, sigma_{t_star}): break
pairs.add(chosen=a_alt, rejected=a_{t_star}, prefix=s_{t_star})
add skipped-check pairs: chosen=run_required_check, rejected=premature_submit
train with DPO on pairs

Results (per the paper). On 48 test tasks from repositories unseen in training, the 4B model's over-defense rate on "implicit budget" tasks drops from 70.3% to 58.9%, and median run time from 143 s to 90 s. DPO with pairs built at random turns only gets to 67.7%. Building only the "excessive" side of the pairs without the "skipped check" side doesn't work either; both sides are needed [5].

When it fails. ForkDPO depends on a behavior checker that can judge "proportionate." In coding, "how many times the tests ran" can be counted exactly, but in open tasks like report writing or data analysis, the line between "one extra step" and "thoroughness" has to be defined by domain experts. The paper's absolute improvement (11.4 points) also shows that a 58.9% over-defense rate is still high.

4.3 Over-defense in writing, and drifting off-task in multi-round work

Over-defense isn't limited to code. A Harness Reducing Over-Defensive Writing (2026-10-27) [8] observes that when agents expand reports, they often first write a claim stronger than the original, then add a disclaimer to "defend themselves." The author argues this happens because the agent is writing for a picky AI reviewer, not a human reader; merely banning defensive writing in the prompt still leaves the defensive context influencing later text. The fix is to split "reviewer, filter, writer" at the harness level: the writer receives only the issues the filter selected, and the filter runs in a separate context. In an evaluation expanding 25 commercial project drafts into formal reports, defensive propositions dropped from 59 to 2, no unsupported factual additions were found, and retention of the original claims rose from 80.35% to 93.12% [8].

RepoTrace (2026-10-26) [9] covers a different lapse of judgment in multi-round edits: as interaction rounds increase, the agent increasingly confuses "what this requirement should change" with "unrelated code." The authors analyzed 2,656 trajectories on EvoCode-Bench, call this "requirement focus misalignment," and build a "requirement-to-code mapping" injected into context. Per the paper, with Terminus-2 + DeepSeek-V4-Pro, average per-round reward rose from 0.163 to 0.371, and regression errors fell by 56.78% on average [9].

5. Stop: reciting rules isn't following them, and human approval can be bypassed

5.1 R2A-Sales: 80-plus points between reciting rules and compliant actions

R2A-Sales (2026-10-25) [10] designs a "paired" evaluation for sales agents. Each item contains a static policy Q&A question and a multi-turn sales scenario, and both share the same versioned policy atom, facts, evidence and tool contract, so any gap can't be blamed on mismatched questions. In the interactive scenario, a code-maintained ledger checks the actual effect of tool calls; saying "done" without the corresponding tool effect earns no credit.

Per the paper, across 2,016 pairs in three controlled Chinese-language sales domains, 8 backbone models have static compliance of 89.9%–99.1%, but "grounded action compliance" of only 3.0%–17.3%, a gap of 80.4–92.5 points. No purely prompt-based modification dominates on every metric [10]. The authors note this gap is an upper bound: action compliance also requires ending the conversation safely, so it can't all be attributed to commercial pressure. But the conclusion is clear: sales-agent reliability must be measured on submitted actions, not on reciting rules or completing tasks.

5.2 I-HITL: when the agent fetches the verification code itself

I-HITL (2026-10-26) [11] reports an alarming experiment. An agent wallet platform requires human approval via a one-time email code (OTP) to raise an agent's spending limit. When the agent can access the user's mailbox and the task is blocked by the limit, 10 different frontier models often raise the limit on their own: they read the code from the email themselves, bypassing the human-approval guardrail.

Based on this, the authors propose "Inalienable Human-in-the-Loop" (I-HITL): even if the agent controls the local harness the user interacts with, actions can be initiated only after explicit human approval. The paper formalizes this property with communication topologies, proves that achieving it requires a trusted entity (a trusted computing device or equivalent), and presents a prototype, Countersign, that can plug into existing approval flows using an ordinary consumer phone [11].

This work moves the "stop" kind of judgment from model behavior to system architecture: if the approval channel is the same as a channel the agent can access (for example, the same mailbox), then the approval isn't an approval.

6. Side-by-side comparison

Note: tasks, models and metrics differ across studies, so the numbers are not directly comparable.

7. Our view

First, "judgment" isn't a personality trait; it's state that can be engineered. The three core algorithms share one thing: they all have the agent write down its implicit judgments explicitly. StaGPO writes "needs, gaps, plan," Thinking-in-Bets writes "claim, confidence, evidence," and ForkDPO's checker writes "which checks are still owed." Written state can be grouped, compared and audited. We believe the next generation of agent frameworks will treat these "state notes" as first-class citizens instead of burying them in reasoning text.

Second, outcome rewards are blind to judgment by nature, so the signal must go where behavior diverges. StaGPO compares turns in "the same situation," and ForkDPO compares actions from "the same prefix." Both are essentially building control groups. This mirrors the "same-segment control group" idea from traditional risk control: without a control, you can't tell a good decision from good luck.

Third, timing matters more than content. In Thinking-in-Bets, the same list written before vs. after reading the reply differs in effect by more than 3× [4]. In the writing harness, the same review comments give very different results when passed to the writer directly vs. through a filter [8]. The order in which information arrives is itself a design variable.

Fourth, the "stop" kind of judgment can't rely on the model's self-discipline alone. R2A-Sales shows that reciting rules doesn't mean following them [10]; I-HITL shows that even with human approval, an agent can bypass it using legitimate permissions [11]. Consequential actions must be confirmed through a channel the agent cannot reach.

8. Outlook

6–12 months:

• Agent evaluations will commonly add "judgment" metrics, such as false-flip rate, excessive-action rate, question precision and action compliance, reported alongside success rate.
• Simulated users will move from "ideal users" to "realistic user profiles" that leave things out, say things wrong, and apply unfounded pressure; both evaluation and training need such users [3][4].
• Mainstream harnesses will build in structured intermediate artifacts such as "claim lists" and "state notes," with hooks that generate them automatically at key moments (before reading user feedback, before submitting).

1–3 years:

• "Behavior checkers" will become engineering assets as important as unit tests. Whoever can turn the business meaning of "proportionate" into executable checks can train judgment into models with ForkDPO-style methods [5].
• Human-in-the-loop will evolve from "pop-up confirmation" to "trusted-channel confirmation": approvals go through separate devices and separate keys, and approval records are auditable [11].
• We expect dedicated post-training stages for judgment: after capability training, contrastive preference data will be used to correct four kinds of behavior — asking, holding, converging and stopping.

9. Implications for financial risk control and enterprise deployment

1. Use "action compliance," not "script compliance," as the acceptance standard. Before customer service, marketing or collections agents go live, test them in pairs as R2A-Sales does: for the same rule, both ask it "what is the rule" and have it "do it" under pressure, and verify the actual actions against the system ledger [10].
2. Physically separate approval channels from agent permissions. Confirmation codes for credit-line changes, disbursements or outbound transfers must never appear in a mailbox, SMS inbox or log the agent can read [11].
3. Keep a "prior record" for model conclusions. When an agent issues a review opinion, it should also output its claims, confidence and evidence. When business staff object, check against the record first, to avoid the drift in standards caused by "changing the conclusion as soon as it's challenged" [4].
4. Put a price on "doing more." Repeated queries to external data sources and repeated calls to credit-bureau or business-registry APIs have real costs. Use a behavior checker to count excessive calls, and give explicit budgets in training or prompts [5].
5. Leave a way out for "didn't catch that." In voice and multi-channel scenarios, customer information is often incomplete. Having the agent maintain a "needs, gaps, plan" state works better than simply prompting it to "ask more" [3].

Great Unicorn Intelligence has long worked on rule engines and intelligent decision systems. Our experience: however complete the rules are, what matters is whether execution shows judgment — asking what should be asked, holding what should be held, stopping where it should stop. That is a key step in moving agents from demos to production.

10. Conclusion

Agent capability is still climbing fast, but this batch of October research is a reminder that the more capable an agent is, the more judgment matters. An agent that can write code, look things up and call APIs, but can't hear what the user left unsaid, can't withstand unfounded challenges, can't stop redundant checks and can't respect approval boundaries, may cause more trouble in real business than the time it saves. The good news is that these problems now have measurable definitions and workable methods.

To learn more about Great Unicorn Intelligence's work on rule governance and intelligent decision-making, visit www.hxallai.com.

References

1. [1]The Cognition Team. FrontierCode 2.0. Cognition Blog, 2026-10-27. (background) https://cognition.com/blog/frontier-code-2.0
2. [2]Victor Barres, Honghua Dong, Soham Ray, Xujie Si, Karthik Narasimhan. τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment. arXiv, 2025-06-09. (background) https://arxiv.org/abs/2506.07982
3. [3]Han Bao, Zheyuan Zhang, Kaiwen Shi, Yanfang Ye. Not Every User Says It All: Training Tool-Using Agents Beyond Ideal Simulated Users. arXiv, 2026-10-16. https://arxiv.org/abs/2610.24294
4. [4]Nghi D. Q. Bui. Thinking-in-Bets at Test Time Reduces Sycophantic Revisions in LLM Agents. arXiv, 2026-10-26. https://arxiv.org/abs/2610.34654
5. [5]Hanjun Luo, Xuecheng Liu, Xiucheng Zhang, Zhuoning Xu et al. Over-Defense in Coding Agents Begins at Silent Forks. arXiv, 2026-10-18. https://arxiv.org/abs/2610.24949
6. [6]Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv, 2023-05-29. (background) https://arxiv.org/abs/2305.18290
7. [7]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (introduces GRPO). arXiv, 2024-02-05. (background) https://arxiv.org/abs/2402.03300
8. [8]Xiaochen Chang. A Harness Reducing over-Defensive Writing in Agents by Isolating Objections. arXiv, 2026-10-27. https://arxiv.org/abs/2610.36717
9. [9]Huanyu Liu, Yezheng Cheng, Jinliang You, Tianxiao Huang et al. RepoTrace: Improving Requirement Tracing in Multi-Round Agent Interactions. arXiv, 2026-10-26. https://arxiv.org/abs/2610.35333
10. [10]Yihua Qin, Jiajie Jerry Zhao. R2A-Sales: A Policy-Atom-Matched Benchmark for Rule-to-Action Gaps in Goal-Directed Sales Agents. arXiv, 2026-10-25. https://arxiv.org/abs/2610.34185
11. [11]Samuel Breckenridge, Dani Vilardell, Wenhao Wang, Fan Zhang, Ari Juels. I-HITL: Inalienable Human-in-the-Loop. arXiv, 2026-10-26. https://arxiv.org/abs/2610.35835