Data Lineage for Financial Agents: The Number Is Right, the Account Is Wrong (In-Depth Review)

1. Background: finance asks something different of numbers

In general Q&A, an answer that's "basically right" is often good enough. In finance, a number must satisfy at least five conditions at once: the right entity (which company, which institution), the right line item (which accounting concept), the right period (which reporting period, which point in time), the right unit (dollars or thousands of dollars, shares or thousands of shares), and the right basis (consolidated or standalone, sole filing or joint filing). Get any one wrong, and however precise the number is, the books are wrong.

A second feature is timeliness. Prices, rates and disclosures are each true only on a given day. Most financial benchmarks fix the answer at labeling time; once the market moves, the gold answer is out of date. And once answers leak into training data, the benchmark rewards memorization rather than retrieval and analysis [2].

A third feature is adversarial pressure. Trading agents treat news as evidence, and because they read the same sources and use similar strategies, a single fake post can make many agents trade the same way at the same moment [3].

October's research falls into four threads:

1. Provenance: where each number comes from and what meaning it's bound to [1][7].
2. Timeliness: how gold answers are updated as markets move [2].
3. Adversarial robustness and consistency: how fake information affects decisions, and whether decisions agree with execution and reports [3][4].
4. Process and experience: how strategy development, experience memory and multi-agent division of labor can be made reliable [5][6][8][9][10][11].

2. Provenance: Algorithm breakdown ① — NumTrace's triple verdict

2.1 Intuition

Reference-answer checks can tell you an answer is wrong but not where it went wrong. NumTrace (2026-10-26) [1] gives every numeric claim from a financial analyst agent a "birth certificate": which passage of which document it came from, and what financial meaning it was assigned.

2.2 Analogy

Think of reconciling books to records in an accounting audit. A number in the statements must trace to a voucher, and the voucher to source documents; and the voucher's line item, period and unit must match the statement. Checking only that the amounts are equal, without checking the line item, is exactly how "right number, wrong account" happens.

2.3 Steps

As described in the paper [1]:

1. Build the benchmark. 540 questions covering 2,163 SEC EDGAR filings split into 41,726 text chunks; 2,306 answer slots; reference answers generated on a frozen snapshot according to accounting rules (216 deterministic rules), with 36,345 source anchors. The questions cover content previously little covered by numeric benchmarks, such as institutional holdings (13F tables) and executive compensation (Item 402(i)).
2. Build a provenance record for each claim. It includes: the source (the exact document and offset in the snapshot); the binding stated in the claim (entity, line item, period, unit, basis, and any multiplier the agent applied to the raw number); the reference binding of the raw number at that source; the derivation (direct quote, or computed from logged inputs); and the first stage where a deviation appears.
3. Issue three independent verdicts:
   • Value: does the number equal the reference value?
   • Binding: are the entity, line item, period, unit and basis correct?
   • Support: does the cited authoritative source support both the value and the binding?
4. Compute the RNWA rate. A claim that passes the value check but fails the binding check is "right number, wrong account" (RNWA). The RNWA rate measures the gap between the pass rate when "only the value is checked" and when "value and binding are both correct."

for claim in agent_answer.numeric_claims():
rec = provenance(claim)                  # source span, stated binding, reference binding, derivation, first deviating stage
v = (claim.value == ref.value(rec.slot))
b = (rec.stated_binding == ref.binding(rec.slot))   # entity / line item / period / unit / basis
s = source_supports(rec.span, claim.value, rec.stated_binding)
if v and not b: rnwa += 1

2.4 Small examples (per the paper)

• The unit trap: amounts in 13F filings were reported in thousands of dollars before 2023-01-03 and in dollars afterward [1]. The same "12,345" (illustrative) differs by a factor of a thousand between the two periods.
• A unit error in the other direction: the paper gives an example where the agent treated a column in a 402(i) table, reported in dollars, as thousands of dollars and wrote $542 as "$0.54 million" [1].

2.5 Results (per the paper)

The authors audited three agents — Claude Opus 5.5, GPT-6 Astra and DeepSeek-V4.1-Flash. RNWA errors made up 67.5%–81.8% of each agent's failed claims. On questions about 13F joint-filing groups, about 91% of claims passed the value check, but fewer than half passed the binding check. The reliability of the reference answers was also tested: a programmer with no access to NumTrace's code or reference answers reimplemented it independently from the written rules, and on 110 sampled questions, 443 of 444 slots agreed; the only disagreement was the spelling of one member's name [1].

2.6 When it fails

The triple verdict depends on structured reference bindings. For well-formatted filings like SEC EDGAR, the rules can be written clearly. For unstructured research reports, news or verbal information, the "basis" is often itself disputed, and binding verdicts become subjective. The authors also note that 194 of the 216 rules are conventions, with no inter-annotator agreement check [1].

3. Timeliness: Algorithm breakdown ② — writing questions as executable programs

3.1 Intuition

If the answer changes every day, don't store the answer; store "the method for computing it." LiveSearch (2026-10-27) [2] stores each question as executable verifier code: a typed call program against a professional financial data service, plus a declarative extractor. The question text is frozen; the gold answer is obtained by executing the program at evaluation time and is re-executed daily by a scheduled job, with no humans or language models involved at any point [2].

3.2 Analogy

The exam question is no longer "what was index X's close on October 27, 2026?" but "look up the close on the most recent trading day." The grader holds not an answer but instructions for "how to look up the answer," and follows them on grading day. A memorized old answer is naturally useless.

3.3 Steps

As described in the paper [2]:

1. Enumerate programs. Enumerate call programs and extractors from real tool responses; the extractor language has no branches or expressions, so it can be checked mechanically.
2. Verbalize. A language model only "translates" an already-fixed program into a natural-language question; it never chooses the answer.
3. Execute daily. A scheduled job re-executes the program to get that day's gold answer.
4. Match with tolerance. Numeric answers are judged with a relative-error tolerance; text answers have separate matching rules.

question = {
"text":    "(question verbalized from the program)",
"program": [call("quote", symbol=S, field="close", date="latest_trading_day")],
"extract": path("data[0].close"),        # no branches, no expressions
"match":   relative_tolerance(eps)
}
gold_today = extract(execute(question.program))

3.4 Results (per the paper)

With general web search, six contemporary tool-calling models achieve answer accuracy of 70.9%–76.7%. Over three consecutive days, the gold answer changed for 70% of questions. Even with tolerance-aware matching, reusing the previous day's snapshot fails on most questions within a single day. The authors also find that per-question tracking reveals dynamic changes in correctness that aggregate accuracy hides [2].

3.5 When it fails

The method depends on professional data services with programmatic access. "Answers" that need human judgment to determine (for example, whether a disclosure constitutes a material event) can't be written as branch-free extractors. Also, definitional changes in the data service itself can make "the same program" produce incomparable results on different dates.

4. Adversarial robustness and consistency

4.1 Information injection: attacking the evidence, not the instructions

Fake News, Real Crashes (2026-10-19) [3] systematically studies "information injection" attacks on trading agents. Unlike prompt injection, which attacks the model's instructions, this attacks the evidence the agent trusts. The benchmark includes five injection methods and seven event types, evaluating about ten thousand decisions on 15 US stocks. Per the paper:

• Prompt injection has an attack success rate of 14%, while fake news reaches 62%; two 30-word templates reach 100% on all 15 stocks.
• For "fact poisoning" templates, surface form matters more than content: removing the named source or the "BREAKING" label drops success from 100% to 0%, with the false content itself completely unchanged.
• Success depends on the agent's state: a fake metrics dashboard succeeds 91% of the time against agents "about to hold" but only 12% against agents "about to buy"; the choice of stock affects success (34%–87%) more than the injection method.
• Larger models only partly protect: GPT-4o drops structural injection to 0%, but fact poisoning still succeeds 93% of the time, and frontier reasoning models do no better. Adding agents helps only in hierarchically aggregated pipelines: a star pipeline drops success to 9%, while peer debate still allows 74%. In all three topologies, the fake narrative still shows up in intermediate outputs.
• A safety-tuned model has 0% success when the content carries an explicit adversarial marker, but 46% when the same content is mixed into a real information feed. Each injection drives the agent to commit about $16,000 in the attacker's chosen direction, about 80% of its $20,000 buying power [3].

4.2 Behavioral concordance: decisions, orders, fills, ledger, reports

FIDES-Live (2026-10-27) [4] points out that even when the final portfolio value is computed correctly, the decisions, orders, fill receipts, account state and reports produced by a trading agent aren't necessarily consistent. It splits the audit into structured decision validity, optional adherence to numeric commitments, execution-to-ledger reconciliation, and receipt-grounded reporting, and explicitly labels coverage and "unscorable" results. Per the paper, the authors retrospectively audited ten reasoning lines, 7,299 crypto account turns and a 310-turn equities pilot. Crypto decision validity was 97.35%, and structured report accuracy among scorable responses was 90.81%, but performance varied widely across lines. Commitment adherence was 94.36% overall, dropping to 62.63% on constraints with non-empty order proposals, vs. 97.56% for valid empty proposals; all retained cash and position changes reconciled with fill records [4]. The authors stress that these results don't prove long-term trading ability, semantic faithfulness of explanations, or causal benefits of audit feedback [4].

5. Process and experience

5.1 Algorithm breakdown ③: MemTrial — putting memory on trial

The problem. Portfolio-management agents learn from experience by recording each memory's value as "the outcome of the decisions that used it." But in financial markets, that outcome mainly reflects the market move that all decisions faced together that day, so the credit tracks the market rather than the experience itself. Such agents often do worse than simply holding an equal-weight (1/N) portfolio [5].

Intuition. The approach of MemTrial (2026-10-08) [5] is to "put memory on trial": for the same decision, draft multiple versions "with this experience" and "without this experience," and have them face the same market. The shared market effect cancels in the difference, and what's left is the experience's contribution.

Analogy. To know whether an analyst's advice is worth anything, you can't just look at whether the market rose on the day they made a recommendation. You need to compare, on the same day, the difference between "following them" and "not following them."

Steps [5]:

1. For each decision, retrieve 4 experiences, pick 8 combinations by fractional factorial design, and draft a decision for each.
2. For each experience, compute its Banzhaf value: the average difference in outcome between drafts that "include it" and those that "don't."
3. Since each date occurs only once and each draft is a noisy model sample, pool these credits across dates and similar experiences with a hierarchical Bayesian model.
4. Act on the learned values only once they predict effectively on unseen dates; otherwise, anchor near a conservative reference (such as 1/N).

E = retrieve(memory, state, k=4)
drafts = fractional_factorial(E, n=8)             # 8 combinations of experiences
R = {d: evaluate(decide(state, d), market_today) for d in drafts}
for e in E:
banzhaf[e] = mean(R[d] for d if e in d) - mean(R[d] for d if e not in d)
posterior = hierarchical_bayes_update(banzhaf, by=[date, similar_experience])
action = act_on(posterior) if validated_on_unseen_dates(posterior) else anchor(1/N)

Results (per the paper). On a semi-synthetic benchmark with known experience quality, MemTrial leads among 15 methods. It also limits losses when the value doesn't hold: on PortBench and InvestorBench it is at most 2.2% below 1/N, while the best experience-learning agent is 15%–38% below. Averaged over five settings, it improves the utility of the best experience-learning agent by 21.2%, and across eight models it outperforms all LLM-based baselines on InvestorBench [5].

When it fails. Each decision requires 8 drafts, so inference cost is several times that of a single decision. When the market environment changes structurally, experience values validated on historical dates may suddenly stop working; the method's protection then is to fall back to the anchor, not to keep profiting.

5.2 Strategy development: "reliable before profitable"

QuantHarness (2026-10-19) [6] argues that a successful strategy-development run may still have implemented the wrong rules, used information not available at decision time, or hidden how fixes and candidate screening changed the strategy. It designs a two-layer harness: a finance-specific workflow splits development into specification, data grounding, building, testing and review, and the harness core enforces boundaries on information, actions and commitments at every stage. Per the paper, on the authors' FinRelBench, QuantHarness achieves a task success rate of 90.72% vs. 74.23%–84.54% for comparison systems; stress tests show the boundaries block targeted violations, and structured run records help with evidence retrieval and decision replay [6].

Auditing LLM Alpha-Factor Mining (2026-10-21) [7] is a retrospective. The author ran a three-layer audit of a well-preserved factor-mining project and found that historical comparisons mixed metric definitions (the mean of retained candidates vs. the mean of per-round maxima), and that the published catalog of 1,837 expressions excluded non-positive out-of-sample results. Recomputing all 1,047 unique expressions in the frozen pool, 923 were scorable: 671 had positive mean rank IC and 252 negative. A simple untrained reference also showed positive correlation, so this distribution can't prove that the agent has incremental ability or profitability [7].

5.3 Skills, spreadsheets and division of labor

• FinSkillBench (2026-10-02) [8]: 17,820 evaluations over 2,603 point-in-time scenarios and 12 subtasks. Per the paper, curated skill packs raise the average score from 0.366 to 0.528 (+16.2 points), while skills self-generated within a single scenario add only 0.5 points. Human documentation alone adds 5.6 points and executable tools alone add 19.5 points; together they add less than the sum. The authors conclude that the "skill premium" is a property of the whole model–resource–harness system, not just of the model [8].
• MBABench-2.0 (2026-10-22) [9]: 101 tasks building financial spreadsheets from scratch; the hardest is estimated to take a modeler with three-plus years of experience 33 hours. 129 criteria cover 12 categories. Among 20-plus agents from 8 models and 4 harness types, the leading agent's pass rate is only 16% [9].
• Single agent or multiple agents (2026-10-20) [10]: on 300 synthetic credit-assessment memos, one agent checking six requirements is compared with three agents checking two each. Per the paper, on the three models where the single agent missed defects, the team found 0.08–0.10 more explained defects (reaching corrected significance on two models), but produced more false positives on all four models. Balanced accuracy changed by no more than 0.02, and token consumption was 2.2–2.4×. In an exploratory comparison, sampling the single agent three times and reporting if any run flagged an issue gave higher balanced accuracy than the team on some models [10].

5.4 Reliability profiles

Agent Reliability Profiles (2026-10-02) [11] proposes a reliability evidence unit for financial services, "one per agent." The profile records a bounded, falsifiable claim: that the agent system operates reliably within its operating boundary. The operating boundary is defined by four items: autonomy level, operational design domain, action categories and control envelope. Assurance comes in three levels: institutional self-attestation (Level 1), testing and validation in the institution's own environment (Level 2), and the same tests run by a qualified independent assessor (Level 3), plus benchmark profiles that can be compared across institutions [11].

The value of this framework is "a fixed profile format with escalating levels of evidence": the same profile starts as something the institution fills in itself, then gradually adds evidence from local testing and independent assessment, without rewriting the whole package at each level [11]. For financial institutions, this resembles existing model risk management: first define use and boundaries, then provide validation evidence proportionate to the risk. Putting the number provenance, timeliness evaluation and consistency audit results described above into this profile creates a complete chain of evidence from "a single number" to "the whole agent."

6. Side-by-side comparison

Note: tasks, models and metrics differ across studies, so the numbers are not directly comparable.

Practice: seven questions before a number goes live

Drawing on the research above, we recommend using these seven questions as a quality checklist for every key number in a financial agent's output:

The first four questions can mostly be automated with deterministic rules; the fifth goes to executable queries; the sixth needs source grading combined with human confirmation; the seventh relies on end-to-end reconciliation. The checklist itself isn't complicated; the key is to embed it in the agent's output pipeline rather than spot-checking afterward. Teams just starting a pilot can get it working on one report type and one data source, then expand gradually.

7. Our view

First, the unit of evaluation for financial agents should move down from "the answer" to "the number." One answer may contain a dozen numbers, each with five dimensions — entity, line item, period, unit and basis [1]. Looking only at whether the final answer is right leaves seventy to eighty percent of failure causes invisible.

Second, a "right number" is more dangerous than a wrong one. RNWA errors pass value checks and are easily missed in human review too [1]. Agent quality checks at financial institutions should make "binding checks" a mandatory item on par with "value checks."

Third, timeliness should be modeled explicitly in evaluation. Evaluating financial search agents on benchmarks with fixed answers amounts to rewarding memorization [2]. Internal evaluation sets should be stored as "query programs" rather than "answer snapshots" wherever possible.

Fourth, more agents doesn't mean more reliable. Peer debate doesn't stop fake news [3], and a three-agent split catches more defects but also raises more false alarms [10]. The structures that work are hierarchical aggregation and independent checks, not simply more agents.

Fifth, reliability needs a profile. Operating boundaries, controls, test evidence and independent assessments should be recorded in a uniform format [11]. This is needed for internal governance and is also the basis for communicating with regulators.

8. Outlook

6–12 months:

• Financial agent outputs will come with per-number provenance records as a standard report attachment [1].
• Financial institutions' internal evaluation sets will move to "questions as programs," refreshing gold answers automatically every day [2].
• Trading and research agents will add source verification and credibility stripping on news inputs, so they aren't swayed by surface markers like "BREAKING" or "according to [an institution]" [3].

1–3 years:

• Agent reliability profiles may become common material when financial institutions procure and deploy agents, and will gradually be combined with independent assessment [11].
• Experience memory will commonly adopt a "counterfactual credit + conservative anchor" design to avoid mistaking market moves for experience value [5].
• Strategy-development agents will write "reliable before profitable" into the process: information-availability checks, definition-consistency checks and full candidate registration will become default steps [6][7].

9. Deployment recommendations for financial institutions and Great Unicorn Intelligence clients

1. Build number-level lineage. For every key number an agent generates, record the source document, location, entity, line item, period, unit, basis and derivation; check value and binding separately during quality review [1].
2. Write unit and basis rules as deterministic rules. Rules like "reported in thousands before date X" are well suited to central maintenance in a rule engine rather than relying on the model to judge each time [1]. In Great Unicorn Intelligence's rule-engine practice, these basis rules are among the first things to be codified.
3. Store evaluation sets as programs. For questions involving market data, rates and disclosures, store the query program and extraction rules and refresh answers daily [2].
4. Grade news inputs by source. Prevent "breaking" information from unverified sources from directly triggering trading or credit actions; use hierarchical aggregation and human confirmation [3].
5. Reconcile the whole chain. Reconcile decisions, orders, fills, ledger and reports turn by turn, and track adherence to commitment-type constraints (such as position limits) separately [4].
6. Build a reliability profile for every deployed agent. Specify the autonomy level, scope, action categories and controls, and gradually bring in independent testing [11].

10. Conclusion

The financial industry's demands on numbers have never been just "computed correctly"; they must also be "clearly explained, traceable and able to withstand scrutiny." October's research turns these demands into measurable metrics one by one: the triple verdict leaves "right number, wrong account" nowhere to hide, executable questions keep gold answers in step with the market, behavioral concordance audits let decisions and ledgers corroborate each other, and reliability profiles put boundaries and evidence on the record. Data lineage is a required course for financial agents on the way to production.

To learn more about Great Unicorn Intelligence's work on rule engines and intelligent decision systems, visit www.hxallai.com.

References

1. [1]Tianlei Zhu, Matt White, Xue Liu, Xiao-Yang Liu. NumTrace: Auditing the Number Provenance of Financial Analyst Agents. arXiv, 2026-10-26. https://arxiv.org/abs/2610.34887
2. [2]Sinuo Wang, Wentao Zhao, Jiacheng Lu, Rui Sun et al. LiveSearch: Questions as Executable Verifier Code for a Live Benchmark of Professional-Source Financial Search Agents. arXiv, 2026-10-27. https://arxiv.org/abs/2610.37166
3. [3]Mengxiao Wang, Nitesh Saxena. Fake News, Real Crashes: Information Injection Attacks on LLM Financial Trading Agents. arXiv, 2026-10-19. https://arxiv.org/abs/2610.26438
4. [4]Arther Tian, Alex Ding, Simon Wu, Aaron Chan. FIDES-Live: Evaluating Behavioral Concordance in Sequential Trading Agents. arXiv, 2026-10-27. https://arxiv.org/abs/2610.36476
5. [5]Guanghao Wu, Zhuo Cai, Shoujin Wang. MemTrial: Learning When to Trust Memory in LLM Portfolio Agents. arXiv, 2026-10-08. https://arxiv.org/abs/2610.11732
6. [6]Jiasheng Mao, Jiacheng Liu, Hengwei Ye, Zheng Tian. Reliable Before Profitable: QuantHarness for Governing Financial Strategy Development with LLM Agents. arXiv, 2026-10-19. https://arxiv.org/abs/2610.26101
7. [7]Yufeng He. Auditing LLM Alpha-Factor Mining: Metric Provenance and Full-Cohort Reanalysis. arXiv, 2026-10-21. https://arxiv.org/abs/2610.29500
8. [8]Jermyn Zhen Yong Bek, Zhuang Qiang Bok, Zhongtian Sun. Knowledge or Calculator? Decomposing the Skill Premium in Verifiable Financial Agent Workflows. arXiv, 2026-10-02. https://arxiv.org/abs/2610.03564
9. [9]Julian Poeltl, Thomson Yen, Patrick Shea, Shubham Anil Varma et al. MBABench-2.0: Evaluating AI Agents on Constructing Financial Spreadsheets Beyond Simple Verifiability. arXiv, 2026-10-22. https://arxiv.org/abs/2610.31711
10. [10]Pei Liu. The More the Better? A Comparison of Single- and Multi-Agent LLM Setups on a Financial Review Task. arXiv, 2026-10-20. https://arxiv.org/abs/2610.27488
11. [11]Mike Hsu, Medha Bankhwal, Béatrice Moissinac, Kevin Werbach et al. Agent Reliability Profiles in Financial Services. arXiv, 2026-10-02. https://arxiv.org/abs/2610.04123