Context-Orchestrated Research Math Agents
Abstract
Long-horizon mathematical reasoning fails less often because a model cannot produce a valid next step than because an agent fails to maintain and expose the right semantic state across many iterations. Left unmanaged, this produces research-level proofs that are locally convincing yet globally incomplete: a key lemma unproved, an assumption unchecked, a citation unsupported, or a computational claim unverified. We present Research Math Agents (RMA), an agentic framework for long-horizon proof development built around a persistent, typed research store and an orchestrator that compiles operation-specific context from that store. The Research Context Orchestrator is the central state-management layer between the persistent research store and each locally scoped proof operation: it retrieves task-relevant artifacts, compiles them into a bounded context, invokes the appropriate operation, and writes the resulting proof edits, issue updates, literature notes, plans, or evaluations back to the store. This process is designed to keep proof revisions, unresolved issues, prior attempts, literature, and evaluations available across rounds while exposing only task-relevant state to each local operation. We evaluate RMA across complementary research-level settings using independent expert evaluation, blind mathematician review, LLM-based benchmark evaluation, and Lean 4 kernel verification. RMA achieves a 42.5% solve rate on the independently evaluated SOOHAK Challenge Hard set, obtains 8 of 10 correct solutions on First Proof B1 and 8 of 10 passing solutions on B2 under human-expert evaluation, and verifies 213 of 300 sampled Research Solved targets in Formal Conjectures with the Lean 4 kernel. These results support structured state and context orchestration as useful components for long-horizon proof development, while comparisons against independently developed systems remain subject to differences in inference budgets, backbones, and evaluation protocols.
Method
Research-level proofs usually fail by omission: the text reads as finished, but a required part of the argument is still missing. Repairing such a gap needs more than another critique of the current draft. It needs the unresolved claim itself, what depends on it, what has already been tried, and which literature applies. RMA therefore treats proof development as a state-management problem and organizes it into three layers.
Persistent Research Store. Proof development is organized into research rounds; a round examines the current proof, identifies and prioritizes unresolved obligations, performs a bounded collection of research and repair operations, revises the proof, and evaluates the result. Each problem has one store : proof artifacts and their revision history, open and resolved issues, structured meeting records, literature records (when literature grounding is enabled), mathematical concepts and background knowledge, strategic insights, and evaluation records. The store is initialized from the problem and persists across rounds; execution may stop early when the proof is judged complete or the system ceases to make measurable progress across two consecutive rounds.
Research Context Orchestrator. An individual LM call should not receive the entire store: context windows are bounded, and models do not reliably use all information in long inputs. The orchestrator takes a problem , a round budget , a per-round issue budget , and a per-call context budget , and for every call compiles the operation’s instructions, its task, the current proof, the records linked to it, and the operation’s own recent outputs. then removes whole, addressable records in ascending priority order until the observation fits within , so an operation never receives a lemma statement cut mid-sentence. Issues are discovered by four complementary analyses: an LM gap critic, a deterministic structural and dependency checker, a semantic completeness judge, and a fidelity analysis that asks whether the proof’s reading of the problem is the intended one rather than one that trivializes it. Each issue receives a severity (P0–P3) by its effect on proof validity, and open issues are ordered by severity, then by dependency impact (the number of unresolved downstream claims resting on the affected claim), both computed deterministically from a claim-dependency graph rebuilt every round. The solver repairs the first issues one at a time with anchored patches; the next critic round re-scans the full proof, so any remaining or reintroduced defect is recorded again. Literature grounding is an optional module; meetings and coordinated revision handle strategy-level changes that local repairs cannot reach; a concept glossary standardizes terminology; and the loop terminates through only when the semantic completeness gate passes and no open issue is correctness-critical.
Locally scoped operations. The critic, solver, optional literature researcher, meeting, reviser, and evaluator each receive only their compiled observation and return a typed artifact that passes a deterministic write-back gate into the corresponding component of . This common interface keeps the operations modular while their outputs accumulate into a single persistent, auditable record of proof development.
Interactive Demo
Step through the operations of Algorithm 1 across the first two research rounds of an illustrative run on First Proof B1 problem 6: how the issue ledger evolves, what the orchestrator compiles into each session’s context, and what each operation writes back. The seven-step proof summary shown in the figure below, the constant 1/42 and the expert verdict are real, though the delivered proof’s own numbering differs; the individual records are a reconstruction, not a log. Switch to Generic iterative revision to compare what the matched-budget control gives each of its calls under its protocol. That control was evaluated on First Proof B2 and the held-out ablation set, not on this problem, so the comparison is one of mechanism, not of outputs.
Research Store inspector
The store is a set of typed, linked records rather than a transcript. Illustrative records from the walkthrough above; ids and contents are not from a run log.
Results
We evaluate on four benchmarks with complementary evaluators: independent third-party expert grading (SOOHAK Challenge Hard), blind review by mathematicians (First Proof B1 and B2), the benchmark’s own LLM-judge protocol (ResearchMath), and Lean 4 kernel verification (Formal Conjectures). The default backbone is Claude Opus 4.8, except for First Proof B1, which was run earlier with Claude Opus 4.6. Sampling for SOOHAK, First Proof, and ResearchMath is fixed by their organizers, not by us.
SOOHAK Challenge Hard
SOOHAK Challenge Hard has 40 non-public, contamination-resistant graduate and research-adjacent problems. We submitted RMA’s API endpoint, and the benchmark organizers independently ran three seeded repetitions of RMA and of every reported baseline under their official expert protocol; the solve rates below are means over those organizer-run repetitions. Seeds, execution, and grading were controlled by the organizers, so this is a third-party evaluation, protocol-matched but not compute-matched across systems.
| Method | Solve rate (%) ↑ | Runs |
|---|---|---|
| RMA (ours, Opus 4.8) | 42.5 | 3 |
| GPT-5.5 | 35.0 | 3 |
| Gemini 3.1 Pro | 17.5 | 3 |
| Claude Opus 4.8 | 12.5 | 3 |
| Kimi K2.7 Code | 7.5 | 3 |
| DeepSeek-V4-Flash | 5.0 | 3 |
First Proof B1
First Proof B1 contains ten research problems contributed by expert mathematicians. Ten mathematicians, at least three per problem, grade all compared systems blind. A solution counts as correct (✓) or incorrect (×) only when its graders are unanimous; anything else is inconclusive (≈). Both RMA and its standalone Claude baseline use the Opus 4.6 backbone on this batch. Aletheia released solutions for only 6 of the 10 problems (5 unanimously correct, 1 inconclusive); the remaining 4 had no released solution (N/A).
| Method | ✓ ↑ | ≈ | × ↓ |
|---|---|---|---|
| RMA (ours, Opus 4.6) | 8 | 1 | 1 |
| Aletheia | 5 | 1 | N/A |
| GPT-5.2R | 3 | 4 | 3 |
| Opus 4.6 | 0 | 1 | 9 |
| GPT-DR | 0 | 0 | 10 |
| Gemini-DR | 0 | 0 | 10 |
The figure below shows problem 6, the example used in the interactive demo. Aletheia released no solution for it. GPT-5.2R derives a looser bound and exhibits minor issues, such as reference hallucinations, as identified by mathematicians (2 correct and 3 inconclusive verdicts). RMA produces a better bound than GPT-5.2R and passes the mathematician checks: 5 of 5 experts judged its solution correct. Both systems answer the same problem statement, so the constants are directly comparable and a larger constant is the stronger result.
First Proof B2
B2 submissions are graded by human mathematicians as Pass (flawless or needing only minor revisions) or Fail, following the comparison set adopted by the benchmark’s authors, with total run cost reported alongside.
| Method | Pass ↑ | Fail | Total cost ($) |
|---|---|---|---|
| RMA (ours, Opus 4.8) | 8 | 2 | 1,850 |
| ProofCouncil | 7 | 3 | 3,186 |
| UCLA Moonshot | 3 | 7 | 4,799 |
| ChatGPT 5.5 Pro | 3 | 7 | 117 |
| Princeton Momus | 1 | 9 | 1,014 |
The headline RMA run costs $1,850, below ProofCouncil and UCLA Moonshot though substantially above ChatGPT 5.5 Pro. A separate higher-budget run realizes $3,571 while reaching the same 8 of 10 passes, so increasing spend beyond $1,850 produces no additional pass over the tested budget points.
Formal Conjectures (Lean 4 kernel)
Inside the benchmark’s research category (excluding textbook, API-stub, and build entries) we uniformly sample 300 Research Solved targets with a fixed dataset-sampling seed and no AI-solvability filtering. All methods are evaluated on the same 300 target IDs under the corresponding pinned Lean 4/Mathlib environments. External web search and fetch are disabled for every reported RMA run, so the agent cannot retrieve publicly hosted target solutions; the literature module is off, and the rest of the pipeline is unchanged, with compiler errors routed back to the solver as ordinary repair issues. A candidate counts only if the generated proof term type-checks against the pinned target proposition, contains no sorry or admit, does not depend on sorryAx or project-local custom axioms, and does not use native_decide or ofReduceBool to bypass the obligation; #print axioms is run on every accepted RMA proof.
| Method | Setting | Verified ↑ |
|---|---|---|
| RMA (ours, Opus 4.8) | agentic | 213 / 300 (71.0%) |
| Seed-Prover | prover | 193 / 300 (64.3%) |
| Hilbert | prover | 164 / 300 (54.7%) |
| NanoProof | prover | 135 / 300 (45.0%) |
| Opus 4.8 direct | direct | 121 / 300 (40.3%) |
The direct-backbone result is a same-backbone reference without the research store or orchestration layer, while the specialized provers, run from their released implementations, provide contemporary cross-system comparisons. Because those systems differ in backbone and test-time inference procedure, the cross-system comparison is descriptive rather than a matched-compute control.
ResearchMath
On the benchmark authors’ 90-problem sample, the benchmark’s LLM-judge protocol scores whether a trace explicitly decomposes a research-level problem into useful intermediate lemmas. This is a complementary LLM-judged measure of lemma decomposition, not comparable to the human pass counts on the held-out set below, which evaluate complete proofs on a separate subset.
| Model | Positive decompositions ↑ | Rate (%) ↑ |
|---|---|---|
| RMA (ours, Opus 4.8) | 5 / 90 | 5.6 |
| Opus 4.8 | 3 / 90 | 3.3 |
| Qwen3.5-397B | 3 / 90 | 3.3 |
| DeepSeek V4-Pro | 2 / 90 | 2.2 |
Cost, scaling, and ablations
The studies below use 30 held-out ResearchMath problems that the benchmark annotates as solved, drawn from a pool of 1,169 such questions with a fixed seed and checked to be disjoint from every ResearchMath problem the system was ever run on. Every configuration, including Generic Iterative Revision, runs under the same hard budget of $100 per problem and is scored by a continuous 0–10 LLM judge (the mean of Claude Opus 4.8, GPT-5.2, and Gemini 3.1 Pro).
Cost and scalability. Over the tested B2 budget points, the pass count reaches its maximum of 8 of 10 at the $1,850 headline run and stays there at $3,571. Reading off what a run capped at rounds would deliver, the judge score rises from 4.5 at one round to 8.5 at six (4.5, 5.7, 6.7, 7.5, 8.1, 8.5), with per-round gains falling from 1.2 to 0.4.
Matched-harness memory ablation. To isolate the contribution of structured persistent memory, the RMA harness is held fixed (backbone, prompts, operations, tools, issue ranking, stopping rule, selection policy, context budget, and the $100 per-problem budget) while only the representation and retrieval of cross-round memory change: full RMA keeps typed, individually addressable records with record-level retrieval, and the summary-memory variant keeps the research state in summarized form. The LLM score is the mean and standard deviation over three repeated runs; the human column is a separate blinded mathematician Pass/Fail evaluation of one paired set of 30 outputs per configuration.
| Configuration | LLM score (0–10) ↑ | Human Pass ↑ |
|---|---|---|
| Full RMA | 8.53 ± 0.15 | 25 / 30 |
| Summary memory | 6.63 ± 0.15 | 16 / 30 |
Among the 17 discordant problem pairs, full RMA passes in 13 cases and the summary-memory variant in 4 (two-sided exact McNemar ). This is the one contrast that does not rest on the LLM judge alone.
Broader architectural control. Generic Iterative Revision (GIR) removes the store and the orchestrator entirely and keeps the backbone, tools, and budget: each iteration is one LM call that sees only the problem, its own previous output, and a fixed window of recent iterations, and the loop runs until a call returns NO ISSUES or the budget is exhausted. On the held-out set it scores 4.5 under its default output selection and 5.50 ± 0.10 when given RMA’s best-proof selection rule. On First Proof B2, under the same $1,850 total budget as the headline run ($185 per problem) and the same blind Pass/Fail protocol, GIR passes 4 of 10 problems against RMA’s 8 of 10.
Component ablations. The largest degradations from the full score of 8.5 occur when the system loses the mechanisms that preserve and expose research state: a stateless store gives 2.6, and dumping the full state into context instead of orchestrating it gives 3.4. Disabling structured literature grounding gives 4.2, FIFO issue repair 5.4, and removing meetings and coordinated revision 6.4. Disabling one critic analysis at a time gives 5.4 (structural checker), 5.8 (semantic completeness judge), 6.1 (LM gap critic), and 6.4 (fidelity analysis); dropping the concept glossary, strategic insights, or evaluator feedback gives 6.8, 7.2, and 7.5. Generic Iterative Revision at the same budget scores 4.5, so exactly the three configurations that break persistent state, bounded context selection, or literature grounding fall below a plain critique-and-revise loop.
Graded design choices. Along each axis the score rises monotonically as the mechanism is strengthened:
| Axis | Basic | Improved | Full |
|---|---|---|---|
| Cross-round memory | Stateless: 2.6 | Last round only: 4.8 | Full persistence: 8.5 |
| Context handling | Dump entire state: 3.4 | Naive truncation: 5.7 | RMA orchestration: 8.5 |
| Issue ordering | FIFO: 5.4 | Severity only: 7.1 | Severity + dependency: 8.5 |
On the memory and context ladders the step from Improved to Full (3.7 and 2.8) is larger than the step from Basic to Improved (2.2 and 2.3): keeping only the last round, or truncating the full state to fit the budget, recovers less than half of the gap. Repeated runs of the decisive configurations are consistent with the plotted ordering: full RMA 8.53 ± 0.15, severity-only ordering 7.23 ± 0.15, context dumping 3.47 ± 0.21, and the stateless variant 2.63 ± 0.25. Two interventions need qualification: context dumping changes request size and completion behavior as well as information selection, and disabling the literature operation removes structured retrieval and persistence rather than all web access, since the remaining operations keep their web tools.
Interpretation and scope. These ablations are mechanism-sensitivity analyses under a fixed $100-per-problem regime, not estimates of deployment-scale effect sizes or statistical rankings among nearby variants; conclusions are not drawn from small differences between single-run configurations. The central claim about structured persistent state does not rely on the LLM judge alone, since the matched-harness comparison is independently evaluated by blinded mathematicians (25 of 30 against 16 of 30 passes).
BibTeX
@inproceedings{anonymous2027rma, title = {Context-Orchestrated Research Math Agents}, author = {Anonymous}, booktitle = {Submitted to the International Conference on Learning Representations (ICLR)}, year = {2027}, note = {Under review}}