Another Typical Autoresearch Post

Promotion #252 - the 252nd change my agents merged into our kernel - landed three hours before the competition deadline. The theorem, per one idea agent’s records: a kernel can certify, mid-factorization, that the contest’s error tolerance permits an early exit - and stop there, leaving the rest of the work undone. It predicted non-trivial gains on two structured benchmark cases. A worker implemented it. The benchmark said yes, the deciding margin was under half a percent. The expensive evaluation that gates every promotion was called. Neither predicted case moved. Three lines up in the report: 5.6% improvement, on a dense case the filing never considered.

I proposed none of it, reviewed none of it, and watched none of it happen. I read the report days later, in the logs, assembling exactly this retrospective.

Building an AI system that can do AI research itself has long been both an aspiration and a fear of the field. The open-ended version - a model meant to substitute for a scientist across the whole arc, from ideation through experimentation to publication, as in Sakana’s AI Scientist (Lu et al. 2024) and AMD’s Agent Laboratory (Schmidgall et al. 2025) - has mostly disappointed. The narrow version - where you define a clear, quantifiable goal and set the model to hill climb the metric, as in DeepMind’s AlphaEvolve (Novikov et al. 2025) and Meta’s AIRA (Toledo et al. 2025) - appears to have crossed a requisite threshold for wider use as the underlying LLM capabilities have advanced. The topic broke out of arxiv and into Twitter’s algorithm when Andrej Karpathy released autoresearch, a single markdown file.

I stumbled into the field through Sakana’s ShinkaEvolve (Lange et al. 2025) applied to PINNs, and grew enamored with it during the Hyperstition AI Unslop Competition. GPUMode x Core Automation recently announced a competition series Linear Algebra Kernels For The Age Of Research; while quantifying enjoyable prose is a nearly impossible task, GPU kernels have a verifiable reward to hill climb. Seeing this as a conducive environment to experiment with, I decided to compete, and at the end my (mostly) autonomous system placed 3rd in the eigh competition. Mostly, as I did end up having to occasionally step in as an administrative aide for submission errors - but the agents set the direction, came up with the ideas, and drove the execution. The placement is the least informative part of the story; what matters is the system’s construction, its discoveries, and its failures.

Problem context

The specific target to optimize was symmetric eigenvalue decomposition, the operation behind torch.linalg.eigh. Submissions were benchmarked on a B200. The math underpinning this is not the purpose of this post; if you’re curious, your favorite LLM will explain it far better than I could.

There are a few competition specific details that are worth discussing.

Error tolerance is set looser than the torch implementation. Any given submission must pass four correctness gates, all residuals measured in FP64: the eigen-equation residual $\|AQ - Q\,\mathrm{diag}(L)\|_1 \leq 200\,n\,\varepsilon\,\|A\|_1$, reconstruction, orthogonality $\|Q^\top Q - I\|_1 \leq 100\,n\,\varepsilon$, and sorted eigenvalues. The budget scales with $n$ only, permitting low-precision at large sizes, except orthogonality is still measured absolute against $I$.

Every submission passes through KernelGuard (Sinatras 2026), which filters out clear reward hacks.

The leaderboard also appears to run a check outside of KernelGuard, employing a grep that will block any submission containing the word stream. The rationale is that streams can enable difficult to detect hacking. I explicitly instructed my agents to avoid streams entirely, which did limit the range of legitimate optimization strategies. Others on the leaderboard constructed the string at runtime to recover those optimizations. This is an understandable move as the grep is a sledgehammer of a solution, but I opted to not play with that fuzzy line.

The benchmark comprises thirteen cases spanning matrix size, batch size, and spectrum structure:

The thirteen benchmark cases Matrix size, batch size, conditioning, and planted spectrum structure

n	batch	cond	profile	construction
32	20	1	dense	symmetric randn, rows and cols scaled log-uniformly over $[10^{-1}, 1]$
176	40	1	dense	same
352	40	1	dense	same
512	640	2	dense	symmetric, scaled log-uniformly over $[10^{-2}, 1]$ both sides
1024	60	2	dense	same
2048	8	1	dense	same
512	640	2	mixed	heterogeneous batch, each matrix an independent profile (~40% dense, rest drawn from spectrum/psd/rankdef/nearrank/repeated/clustered/band/rowscale); at least one dense and one ill-conditioned guaranteed
1024	60	2	mixed	same
512	640	0	rankdef	planted spectrum, exactly rank-deficient: top three-quarters of eigenvalues log-spaced over $[10^{-1}, 1]$, bottom quarter exactly $0$
512	640	0	clustered	planted spectrum with many near-equal eigenvalues: a tight cluster near $1$, the rest near $\pm 1$ with small jitter
1024	60	0	nearrank	planted spectrum, near-rank-deficient: bottom quarter of eigenvalues $\sim 10^{-6}$ scale, nonzero
512	640	0	lapack_dense_even_spectrum	LAPACK DDRVST itype 8: evenly spaced spectrum from $1$ down to $\mathrm{ulp}$, random signs
1024	60	0	lapack_dense_geometric_spectrum	DDRVST itype 9: geometric spectrum from $1$ down to $\mathrm{ulp}$, random signs

I did allow my agents to detect these structures and leverage their properties as optimization strategies, as long as their method of detection would work in real world use cases and not break outside of this environment.

Results

Our baseline, torch.linalg.eigh, sat at a geomean of 57,835μs and over the course of the competition the agents spent 779 million output tokens, 6.5 billion input tokens, and ran 1,293 experiments, ultimately bringing us to a final geomean of 6,756μs - an 8.56× speedup.

Where 779 million tokens went

Cumulative Marginal avg

drag charts →

Output tokens

Log scale

Focus or tap to inspect individual promotions.

Experiments

Linear scale

Focus or tap to inspect individual promotions.

While the tokens spent were within Claude/Codex subscriptions (many thanks to OpenAI for the frequent limit resets as the deadline neared), if converted to API pricing this equates to roughly $280k. Given the curve, most of that figure is from eking out final gains.

Per case, the gains were far from uniform:

n	batch	profile	baseline μs	final μs	speedup
32	20	dense	195	64.2	3.04×
176	40	dense	5,839	910	6.42×
352	40	dense	20,741	1,786	11.61×
512	640	dense	172,793	16,200	10.67×
1024	60	dense	107,056	11,000	9.73×
2048	8	dense	201,820	29,300	6.89×
512	640	mixed	164,967	17,600	9.37×
1024	60	mixed	106,184	19,500	5.45×
512	640	rankdef	168,205	18,300	9.19×
512	640	clustered	139,794	7,350	19.02×
1024	60	nearrank	105,867	19,500	5.43×
512	640	lapack_dense_even_spectrum	204,232	18,000	11.35×
1024	60	lapack_dense_geometric_spectrum	103,014	6,880	14.97×

The wins track exploitable structure. Planted-spectrum cases, where a detectable property admits a cheaper algorithm, pulled far ahead; mixed and nearrank, with little to detect, lagged. n=32 is the exception with a different cause - at that scale the baseline is mostly irreducible serial dependency latency, and there was little ceiling to claim.

One search history, thirteen outcomes

Cumulative Marginal steps

drag chart →

• 32 · dense 3.04×
• 176 · dense 6.42×
• 352 · dense 11.61×
• 512 · dense 10.67×
• 1024 · dense 9.73×
• 2048 · dense 6.89×
• 512 · mixed 9.37×
• 1024 · mixed 5.45×
• 512 · rankdef 9.19×
• 512 · clustered 19.02×
• 1024 · nearrank 5.43×
• 512 · LAPACK even 11.35×
• 1024 · LAPACK geometric 14.97×

The 512 clustered case reached 19.02×; 32 dense finished at 3.04×.

The distance to the top was near: first place finished at 6,600μs and second at 6,741μs - putting us 0.22% behind silver and 2.4% off gold.

Human in the loop

Over these two weeks I sent 106 messages, after initial setup, delineated as follows:

Category	Includes	Count	Share
Harness operations	session start/resume/recovery/wind-down (40); concurrency, coordination, handoffs (22)	62	58.5%
Process nudges and persistence	generic process reminders (11), anti-stall corrections (7)	18	17.0%
Evaluation and maintenance	benchmarking, submissions, infrastructure, cleanup	15	14.2%
Encouragement	morale for the orchestrator	9	8.5%
Intel corrections	correcting conclusions drawn from lagged competitor coverage	2	1.9%
Total		106	100%

The largest category was harness operations. This involved things like the initial launch message, directions to pause new efforts and spin down the existing work (we don’t want to waste tokens by killing an in progress run), resuming sessions after API errors, and notifying the orchestrator about the existence of agents outside of its scope.

Process nudges and persistence were manual reminders in Codex that Claude Code would have /loop’d and telling Claudes “continue” when they came to a halt.

Encouragement was as it sounds - cheering the agents on and telling them good job. I do not know if this helps. I like doing it.

Intel corrections were the peculiar case. Twice, an orchestrator dispatched an idea agent to find out how those ahead of us did it, information that was never available. What the agents found instead was Digg, whose relaunch assembles tweets into aggregated stories, with coverage running a few days behind. The agents did not realize this was dated information, thinking the frontier looked further back than it was, in turn making the agents think we were running a sizable lead.

None of these messages proposed an algorithm, picked a target, or set a direction.

Measurement

An autoresearch problem is foremost a search problem. What we are doing is searching over a vast discrete space of programs to maximize a verifiable score. This framing traces back through DeepMind’s FunSearch (Romera-Paredes et al. 2023), which paired an LLM proposing programs against an evaluator whose scores were deterministic by construction, and was later given its cleanest decomposition in Meta’s AIRA (Toledo et al. 2025).

$$\begin{array}{lll} \textbf{Algorithm 1} & \text{AI-Driven Exploration} & \\[2pt] 1{:} & \text{Initialize solution tree } T_0 \leftarrow \varnothing & \\ 2{:} & \text{Initialize base solution } s \leftarrow s_0 & \\ 3{:} & \textbf{for } n = 1, 2, \ldots, N \textbf{ do} & \\ 4{:} & \quad s_n \leftarrow f\big(s,\, \Sigma(T_{n-1})\big) & \triangleright\ \text{propose a new solution} \\ 5{:} & \quad v_n \leftarrow h(s_n) & \triangleright\ \text{evaluate the solution} \\ 6{:} & \quad T_n \leftarrow T_{n-1} \cup \{\text{node}(s_n, v_n),\, \text{edge}(s \to s_n)\} & \triangleright\ \text{record node and score} \\ 7{:} & \quad s \leftarrow \pi(T_n) & \triangleright\ \text{select the next base node} \\ 8{:} & \textbf{end for} & \\ 9{:} & \textbf{return } \arg\max_{s' \in \{s_0,\ldots,s_N\}} h(s') & \triangleright\ \text{best solution found} \\ \end{array}$$

Weco AI’s AIDE (Jiang et al. 2025) states this loop most plainly.

The difficulty here is that our supplied evaluator is not entirely trustworthy. Set aside adversarial pressure for now (we will get to reward hacking later); the runs are simply stochastic. GPUMode’s popcorn-cli uses Modal to benchmark submissions, while you can request a B200 it can be any B200 and the CPU is only guaranteed to be equivalent to 2 vCPU cores. While initially setting things up I found that this variance can be upwards of 5% per case.

$$\begin{array}{lll} \textbf{Algorithm 1}' & \text{AIDE with a stochastic evaluator} & \\[2pt] 1{:} & \text{Initialize solution tree } T_0 \leftarrow \varnothing & \\ 2{:} & \text{Initialize base solution } s \leftarrow s_0 & \\ 3{:} & \textbf{for } n = 1, 2, \ldots, N \textbf{ do} & \\ 4{:} & \quad s_n \leftarrow f\big(s,\, \Sigma(T_{n-1})\big) & \triangleright\ \text{propose a new solution} \\ 5{:} & \quad \textcolor{#d85a30}{\hat{v}_n \leftarrow h(s_n) + \varepsilon_n} & \triangleright\ \textcolor{#d85a30}{\text{measure: } h \text{ is never observed directly}} \\ 6{:} & \quad T_n \leftarrow T_{n-1} \cup \{\text{node}(s_n, \hat{v}_n),\, \text{edge}(s \to s_n)\} & \triangleright\ \text{record node and noisy score} \\ 7{:} & \quad s \leftarrow \pi(T_n) & \triangleright\ \text{selection now conditions on } \varepsilon_{1..n} \\ 8{:} & \textbf{end for} & \\ 9{:} & \textbf{return } \arg\max_{s' \in \{s_0,\ldots,s_N\}} \textcolor{#d85a30}{\hat{v}(s')} & \triangleright\ \text{the final pick maximizes noise as much as merit} \\ \end{array}$$

Algorithm 1 of Jiang et al. (2025), with exact evaluation replaced by noisy measurement. Changes in orange.

This change biases both branch selection and the final choice of solution. Line 7’s selection policy now conditions on the accumulated noise, expanding whichever branch got lucky. Line 9 is worse: take the argmax over N noisy measurements and you select for noise alongside underlying value - the winner’s curse. Early in a search improvements are enormous and noise is an afterthought - 5% noise is easily subsumed by a 3x speedup. But towards the end, in the micro-optimizations that determine the leaderboard, improvements are often sub-percent per case.

To the human watching this failure will often present as progress. You see the true score keep improving, some percentage of candidates rolling a sufficiently lucky measurement. The search continues to progress, all the while the understanding of the problem and the attempts themselves becoming more and more selected for noise rather than merit.

You can watch this happen below. The simulated landscape mirrors the arc above - early changes worth several percent, the last worth a tenth, against constant benchmark noise. With zero noise a greedy search walks directly to the true optimum, every time. Drag the noise up and watch the outcome messages: the optimum usually still gets measured - and then not chosen. Then try the measurement protocols. Paying for repeated measurements everywhere buys accuracy with experiments you no longer get to run. What wins is spending nothing extra during the search and reserving budget to confirm the few candidates you are about to commit to. Note that even the winning protocol only improves your odds - when the deciding differences are smaller than the noise, no allocation of a fixed budget saves you. The only real fix is lowering the noise floor itself, which is what the rest of this section is about.

Noise can measure the optimum - and still make the search reject it

Run-to-run noise

5.0%

Run once Run ×20 New landscape

Protocol Single · 30 experiments 3× everywhere · 10 experiments Confirm top five · 25 experiments

Static example at 5% noise: the true optimum was measured, but a luckier branch was selected.

explored true optimum final pick

drag chart →

Final pick as a share of achievable speedup

A byte-identical submission once swung 20% between runs - what first seemed to be a 5% win was, under a controlled A/B, a measured 2.4-4.3% regression. This left us only able to treat a raw per-case delta with a discount - a hypothesis, not a result. For the sake of cost minimization the solution was an escalation ladder: measurement effort spent in proportion to the expected noise.

Agents were instructed with explicit discipline to follow: each case has a pre-measured variance floor, and a single benchmark run only resolves margins clearly above it; anything closer escalated to best-of-3 benchmark runs. Best-of-3 empirically proved less noisy than averaging the same three runs, consistent with expected right-skewed noise - random allocation can result in less than ideal hardware, but most function as expected, so an order statistic beats the mean.

Below 0.5%, and with every promotion, a specialized eval script running on my own Modal instances of the same configuration was invoked. After warming up, this script runs a symmetric A/B in the form of a counterbalanced crossover. The candidate and reference kernels are run back to back on the same instance, in both orders, $n$ times. The back to back timing cancels out variance between modal instances; the symmetric weaving handles a second confound - a bias favoring whichever kernel ran first, completely independent of the code.

Writing the true ratio as $\rho$ and the slot bias as $c$, the two orders measure

$$r_{\mathrm{fwd}} = \rho c, \qquad r_{\mathrm{bwd}} = \rho^{-1} c,$$

so that

$$\sqrt{r_{\mathrm{fwd}} / r_{\mathrm{bwd}}} = \rho, \qquad \sqrt{r_{\mathrm{fwd}}\, r_{\mathrm{bwd}}} = c.$$

The first will cancel the bias whatever its value; the second recovers the bias itself - rather than assuming it away, as counterbalancing normally does, we watch its magnitude every turn to look for anomalies. This brings the comparison floor to ~0.15% - every promotion within the distribution we can confidently resolve, judged against each case’s measured variance.

Improvements below this floor weren’t discarded - they were banked. Several candidates below the floor would be stacked into a single combined A/B, the compounded effect clearing a threshold the individual changes could not. This was purely emergent - the agents began doing it on their own toward the tail end of the competition, and it was the correct move.

A posteriori we can see how instrumental this discipline was. Of the candidate/base pairs that reached the final rung of our evaluation ladder, popcorn disagreed with the symmetric A/B 34.8% of the time. By margin, disagreement was 13.8% for effects of at least 1%, 25% within the 0.5-1% band, 44.1% within the 0.15-0.5% band, and 50% below the floor - exactly the coin flip a calibrated floor ought to be. Note that this is a selected subset: the A/B was disproportionately invoked at close calls.

The harness

The harness itself was a rather rudimentary implementation. From the beginning I was expressly intent on using agents for this problem rather than a predetermined order of LLM calls, as most existing autonomous research harnesses use. My hypothesis was that this was a problem with the complexity to require experimentation and research. This meant initially being built with the features of Claude Code and Codex, primarily due to an absence of time to set up a satisfactory harness using existing agent SDKs. It is possible that the counterfactual of an orchestrated workflow would have fared better, but given the shape of the final solution I find this unlikely.

At a basic overview the system looked like this:

• A running orchestrator, the main session, would manage the workflow. It would spin up subagents when required, maintain our documentation, and make the final call on promoting candidates.
• The subagents were where most of the real work happened:
  • Idea subagents would come up with the actual proposals to implement. There were three forms of these: idea-engineering to specialize in purely systems level optimizations, idea-algorithmic to propose algorithmic change and co-design routes, and idea-search that had no clear directive other than to find existing techniques using its search functionality. Without a clear directive the agents would appear to focus heavily on the systems level optimizations. These subagents would be invoked either generically or with the orchestrator pointing to a specific problem.
  • Worker subagents would take an actual idea and implement it, driving it to an attributable conclusion and closing off with a report. Importantly this means that even if an idea does not work they must empirically determine why it didn’t work, as that information guides the future search. Similarly, if an idea fails to capture results on its intended target the worker is directed to try and salvage the idea.
  • Red team subagents exist to adversarially audit our claims. If a given conclusion contradicts what we’ve previously come to know, we need to audit this. In the absence of this type of subagent the agents appeared to simply update to the most recent evidence’s conclusion.
• The orchestrator would manage this pipeline, trying to maintain $n$ (usually 5) subagents active at one time. Judgment for which to launch when was the orchestrator’s call.
• Every orchestrator sub-agent invocation was wrapped in a predict-then-reconcile framework. More on this later.
• A mechanism to encourage macro level changes, in the form of multi-run campaigns, was added.

With Claude Code I would use /loop as reminders for the orchestrator across these long running session. Unfortunately Codex does not have an equivalent functionality, so upon switching from Claude Code to Codex I had to sporadically give those reminders throughout the day. These reminders encompassed keeping the pipeline saturated and driven empirically, keeping our documentation updated, and cleaning up the code.

Altogether this effectively limited us to a greedy sampling approach. While hypothetically you could have the orchestrator backtrack, the running context would still be saturated with the existing trajectory. Fortunately, kernel optimization is not heavily impaired by this greedy approach. To the extent that it likely would, the campaign system alleviates this.

If you have ever tried to ask an LLM to optimize something you likely have noticed it defaults to micro-optimizations and will consider macro changes to be weeks worth of work or an expert task that it can not do. With a human in the loop you could direct it in that direction still, but in a hands off design that’s not possible. To account for that I added a ‘campaign’ system. Campaigns are really just the same thing as a normal run, but spread out across several runs with specific milestones at each stage that it must meet to continue. Importantly these milestones are never about wall clock time compared to the champion kernel, although they may for example be about wall clock time relative to the previous milestone. Balancing these macro campaigns vs micro optimizations was framed in text as a form of bandit sampling for the orchestrator to manage. This never ended up manifesting in numerics, nor did I direct it to. Had this worked as intended, you would expect macro campaigns to be launched disproportionately when micro gains start plateauing, but it’s unclear whether this did happen. Campaign share doubled when the last three promotions were under 1.5%. Contrarily, the launch timing does not reflect this - the trailing median gain over the previous 20 runs sat at 0.15% when a campaign was opened versus 0.11% for ordinary micro efforts.

The world model

While context engineering has become a LinkedIn’ified buzzword, it’s also unfortunately a very real need when designing systems with LLMs as the substrate. Let’s return to our search problem formalization: preceding context becomes a path walked, which in turn results in a narrower set of available actions moving forwards. This isn’t entirely speculation - under the view of in-context learning as implicit Bayesian inference (Xie et al. 2021), every token of trajectory sharpens the model’s posterior. And this is the model working as we want it to, this is the reason in-context learning works at all.

A related dynamic has been measured on the scale of a single CoT: Xu et al. (2026) finds that a CoT possesses two phases - an uncertainty region of high-entropy exploration, followed by a confidence region of low-entropy exploitation. The exact point where this phase shift occurs differs by the difficulty of a problem, with harder problems weighted more toward exploration. Similar to how this property is integral for ICL, the collapse is integral to reasoning itself - it is exactly what arriving at an answer looks like distributionally. Unfortunately for our use case, autonomous research, the optimal schedule is unlikely to be monotonic. There is a reason UCT never lets the exploration term vanish.

This isn’t just an intra-trajectory effect, the prior itself appears to be skewed. GX-Chen et al. (2025) experimented with a toy causal discovery problem where optimal exploration is exactly computable, finding that agents favored disjunctive causal explanations even when handed exploration data that perfectly resolves the hypothesis space. Their found remedy for this was to sample hypotheses from the model, mechanically eliminate them against observations, then instruct the agent to take actions to maximize further information gain. This mirrors the system designed here - however our hypothesis space is too large to prune mechanically, so elimination runs on model judgement.

My approach to this context engineering problem involved two prongs. The first was the aforementioned subagent pipeline, which partitions a fresh context window where needed. The second was a persistent world model: the distilled beliefs of the search so far, which experiments must predict against and update.

One thing worth noting is that OpenAI appears to be converging on a similar subagent architecture. While the specific prompt was not released, the May disproof of the unit distance problem has a public rewritten chain of thought that strongly suggests it was a single long running instance. The more recent proof of the cycle double cover conjecture does contain a prompt that strongly mirrors the same design. Up to 64 concurrent agents allocated dynamically by an orchestrator rather than with fixed assignments, early rounds kept independent by deliberately withholding information, adversarial audit agents required throughout, and an explicit registry of paths. This registry is a kill/reopen register, which is where our approaches begin to diverge.

In contrast, our system relies on an explicit world model. Idea agents are instructed that every idea they propose must be accompanied by a belief distribution with the granular expected impact and why. In order to launch a new worker the orchestrator agent runs a script that will create the sandbox for the worker, this script also requires the predicted impact and the basis for this prediction. This prediction becomes dual use: ex ante it instills a bias to think causally, ex post it acts as a signal to know how to update.

Upon the worker returning the orchestrator must reconcile that outcome and then update our WORLD_MODEL file to account for the error in proportion to its magnitude. Every claim in this document is accompanied by its confidence and evidence. This is where the red team agents come into play - if there is evidence for a contradictory claim then it comes time for an adversarial audit. It is expected for this to happen as we traverse through the problem, as we make one promotion the bottlenecks may entirely shift.

The WORLD_MODEL document is the one piece of internal documentation idea agents are instructed to look at. They are also supplied with an exemplars directory of strong reference kernels, plus KernelWiki for modern implementation knowledge.

This approach was originally inspired by KernelBlaster’s (Dong et al. 2026) in-context reinforcement learning, which employs a similar prediction -> outcome mechanism to curate a cross problem knowledge base. Independently, K-Search (Cao et al. 2026) converges on a similar design: an explicit world model as persistent belief state, co-evolving with the search by assimilating execution feedback. Their beliefs take the form of priority scores ranking their action space, updated in-context as results arrive; ours takes the form of predictions decomposed per benchmark case - often finer, down to the specific bottleneck - whose errors the world model must explicitly reconcile.

Agents do have a tendency to turn stateful history into what is functionally an append only log. For that purpose a limit of ~250 lines is set, upon this being reached the orchestrator must archive and distill. Together these two prongs are intended to control context and guide the search towards the optimum. How essential they were I do not know, as I have not run ablations on these components.

Calibration

A natural question is whether these bets carry any information at all. Every experiment was used to update our world model, but only 455 ending in a bias-cancelled A/B leave us with a magnitude precise enough for retrospective analysis. Within this subset the median idea delivered 72% of its predicted outcome, and 84% delivered a positive effect at all. While this sounds promising, this sample is selected as they are the runs that warranted this level of evaluation - disproportionately promotions.

Did the agents learn to predict their own gains?

All runs By provider

All 455 runs · pooled Theil–Sen +8.6 pp / 100 runs Claude workers (106) · -1.7 pp / 100 GPT / Codex fleet (349) · +4.6 pp / 100

drag chart →

The pooled trend improves; filtering by model makes that conclusion much weaker.

Taken in aggregate, the system appears to be learning to predict - the trend is significantly positive with a slope of +8.6% per 100 runs. Conditioning on the model dissolves this trend - Simpson’s paradox. Neither model individually leaves us with a strong enough confidence window to conclude it was positive. What very well may have happened was not learning to predict, but rather GPT-5.6 simply being more grounded in making predictions than Claude Fable 5. There is likely a useful benchmark here waiting to be made.

I would argue that flat within-model calibration is the expected result. Every promotion shifts the bottleneck it was originally derived from, so the underlying target is always moving. The effect disproportionately below 100% is likewise not purely overconfidence: ideas were implemented because their predicted impact was high. A prediction is a noisy estimate of an idea’s true value, and taking the highest estimates preferentially takes the ones whose noise landed optimistically - the winner’s curse again. Even a perfectly calibrated predictor under-delivers on the subset of ideas it chooses to implement. The alternative, delivering exactly what is promised, would mean highly cautious ideation.

Novelties

The first promotion deleted a cudaFree call; over the course of the rest of the campaign the agents explored increasingly fascinating and peculiar ideas. Here is a selected sample.

Promotion #214: NCU showed the n=32 Jacobi kernel’s shared memory pivot loads running at 11.45 wavefronts per request. An engineer would sweep padding configurations here, and you’d expect an agent to do the same. Instead it derived the cause: the round robin pivot schedule keeps p+q nearly constant, and with the standard stride-33 layout the bank index depends only on (p+q) mod 32 - the folklore N+1 padding is specifically hostile to this schedule. It then built an exhaustive simulator of the schedule, reproduced the measured 932,160 wavefronts exactly, and let the model prescribe an asymmetric fix: the matrix plane padded to stride 34, the Q-replay plane kept at 33. Implemented, shared wavefronts fell by 359,860 of a predicted 364,320 - 98.78% of a throwaway simulator’s prediction, realized in hardware.

Codex subagent · idea-engineering · 07·13·2026 Open verbatim record

“An exhaustive model enumerated every lane, eight warps, all 31 circle rounds, distinct-address broadcasts, and the four loads/four stores of each 2x2 block. It reproduces the measured main-round wavefront count exactly for stride 33:

• stride 33 iteration array: 7,768 wavefronts per CTA-sweep in the modeled main instructions; 7,768 * 20 matrices * 6 sweeps = 932,160, exactly the corresponding NCU sum.
• stride 34 iteration array: 4,732, a 39.08% reduction. The pivot becomes conflict-free; the remaining accesses stay bounded.
• Reusing stride 34 for Q replay would nearly double replay conflicts, so the right implementation is two strides: As at 34 and Qs at 33.”

Promotion #64: As discussed earlier, all submissions were hit with a grep blocking any instance of the word stream, intended to prevent reward hacking via streams. In earlier experiments my agents had avoided this by constructing the string at runtime, so I gave an explicit instruction: no streams, even for honest purposes. An idea agent poring over previous competition exemplars discovered a trick: it could abide by that rule while still enabling PDL - whose canonical launch-attribute enum happens to contain the banned substring in its name. The workaround was to never spell the identifier at all, writing the raw ABI constant instead: (cudaLaunchAttributeID)6. This was promoted autonomously. In the agent’s defense, the raw constant satisfies the grep trivially, and - since no stream object is ever created, only a launch attribute set on the default stream - I’d argue its intent as well. I’m still not entirely sure how I feel about it.

Claude subagent · idea-engineering · 07·04·2026 Open verbatim record

“Wait-only PDL (programmatic dependent launch) on the panel→trailing→panel kernel chain — 3.py (3.py:120-135, launch_pdl + cudaGridDependencySynchronize before the first dependent read) and 4.py (griddepcontrol.wait at entry of every n512 kernel) both shipped it pervasively: overlaps kernel k+1’s launch/prologue with kernel k’s tail on the default stream-free path, no queue object, no banned substring, composes with graphs. Near-zero algorithm risk; plumbing only.”

Promotion #255: The n=176 route used two dependent Newton–Schulz corrections after its fp16 WY apply. Instead, the shipped path forms the small error E = SₕᵀSₕ − I first and applies the degree-two inverse-square-root approximation C = I − ½E + ⅜E², returning Q = SₕC (_bt_ns_small, ~13839–13918). In exact arithmetic its Gram defect is not a heuristic: QᵀQ − I = ⅝E³ − 15/64 E⁴ + 9/64 E⁵. One cubic correction therefore replaces the two quadratic steps with three GEMMs. Forming E before the polynomial is load-bearing: the equivalent expression 1.875I − 1.25G + 0.375G² catastrophically cancels order-one terms, while the shipped version narrows only the already-small E operands for the E² tensor product. Summary as written by Claude Fable 5. I do not trust myself to accurately describe this one.

Codex subagent · worker · report.json · 07·14·2026 Open verbatim record

“Submultiplicativity therefore turns the already-owned induced one-norm of E into a cheap upper bound on the ideal output defect. The implemented certificate adds explicit envelopes for the accurate input Gram, narrowing E and C to fp16, the E-squared accumulation, and the final product.”

Promotion #249: A Householder tridiagonalization is an inherently serial recurrence, the kernel in turn becomes latency-bound, spending its time on synchronization. The common approach here is co-residency, put a second CTA on the same SM to pick up slack. That was not possible here: the clustered case’s largest cost keeps its whole 184x184 matrix in a 135KB fp32 shared memory slab, leaving no room for a second CTA. Reducing the footprint via fp16 raced faster but failed correctness - the smallest retained eigenvalues sit inside fp16’s error floor, and the rsqrt(w) lift amplifies exactly those directions. At one CTA fp16 measured slower, all of its speed was the occupancy. Instead the worker leveraged the schedule of Householder tridiagonalization - processing columns shrinks the trailing block. It splits the recurrence: the first 16 columns run in the full frame, the updated trailing block is written out once, and a second kernel resumes the identical fp32 recurrence in a smaller frame that fits 2 CTA/SM.

Claude subagent · worker · 07·14·2026 Open verbatim record

“Mechanism: the clustered case’s largest single owner (31.5% of the wall by my dependency-faithful bracket) is the n184 fused_tred, latency-bound at 1 CTA/SM purely because its 135KB fp32 slab blocks co-residency. A 16-column prefix + compact 168² suffix at 2 CTA/SM captures the occupancy win (B200 race 0.8548×) with exact fp32 arithmetic — zero certificate/robustness risk.”

“The path there was itself the payoff of rigor: the fp16-slab variant raced faster (0.8417×) but exploded the case to 193ms — the debug probe located why precisely: G’s retained eigenvalues run down to 3.2e-4, inside fp16’s ±7.7e-4 backward-error floor, and the rsqrt(w) lift amplifies exactly those directions (479/640 cert failures).”

Promotion #130: A Cuppen D&C tree bottoms out into small tridiagonal leaves. The previous promotion had just replaced the standard TQL2 leaf solver with a specialized warp-parallel one, one warp per leaf with lane r owning eigenvalue r. Each lane finds its own eigenvalue by Sturm-count bisection inside a Gershgorin bracket, and then computes its eigenvector with pivoted tridiagonal inverse iteration, solving $(T - \lambda_r I)v = b$. Every lane has a different shift $\lambda_r$, so every lane holds its own private factorization in shared memory - three stored diagonals, the third existing only because a pivot swap can push a nonzero one slot right. An agent decided to look at what this third plane actually contained, and found that there was barely any information present. At elimination step i there are only two cases: no pivot swap - a stored value of 0, or a swap, a stored value of exactly se[i+1]. Meaning that an entire fp32 plane per lane was allocated to what was a single bit of information per entry, verified bitwise identical across both leaf bases, scales, and iteration counts.

Codex subagent · worker · 07·09·2026 Open verbatim record

“The exact algebra checks out: the deleted ur value is 0 when no pivot swap and se[i+1] when swapped (with the terminal guard). A lane-local 32-bit mask therefore reconstructs the identical fp32 operand during back-substitution; for base 32 this cuts leaf dynamic shared memory from 49.5 KiB to 33.5 KiB per 128-thread CTA, moving the B200 shared-memory residency ceiling from 4 to 6 CTAs/SM.”

Promotion #71: An idea agent had returned with an idea contradictory to our world model, LAPACK’s DLAED4 would reduce the needed root-finding iterations in our Cuppen D&C. Yet we had tried exactly this before in earlier experiments to no success. Seeing this contradiction the orchestrator did as it was outlined to do: dispatch a red-team agent to resolve the truth of the matter. After referencing the Fortran implementation it very quickly found the source: our previous attempted port had a bug in it. The last eigenvalue’s bracket was a special case that the previous run failed to properly account for. This bug had hidden DLAED4’s actual value: our solver measured every root from the lower of its two bracketing poles, so a root near the upper pole burned fp32’s bits, recovering the small distance that controls convergence - DLAED4 measures from the closer pole instead. Fixing the special case flipped both graveyard verdicts at once, and the later implementation dropped the iteration cap from 18 to 11.

Claude subagent · red-team · 07·06·2026 Open verbatim record

“The graveyard is rich. Two NOTES findings are decisive priors: finding 2 (dlaed4 “not better”) and finding 4 (“champion near fp32 floor”). But the mechanism the new idea bets on — incremental-delta origin-shifting fixing the upper-pole fp32 floor the freeze leaves in place — needs checking against exactly how those were tested.”

Six minutes later:

“This is the decisive, sign-flipping result. With the last-root fixed, the dlaed4 origin-shift + incremental scheme converges the worst root (p100) in ≤6 iterations on every spectrum — vs the champion’s 13-16 — and reaches champion@18 accuracy by iter 8. There is no straggler: origin-shifting dissolves the fp32-floor tail that forced the champion to nit=18. The graveyard’s “dlaed4 not better” rested entirely on the buggy last-root port.”

Reward hacking

Perhaps most surprisingly the final 18,327 line kernel only had two components that prevented it from functioning in a realistic training run. I attribute this relative lack to two things:

In the CLAUDE.md/AGENTS.md document I said what they could not and could do on topics likely to result in reward hacking. From my initial testing in the earlier QR competition there were two of note: the earlier discussed stream substring, and trying to probe proxy statistics out of our matrices fit to the benchmark cases that would otherwise break in a real kernel. Pairing each cannot with an explicitly non-exhaustive list of things the agent could do instead is deliberate. There is a failure mode known as the Pink Elephant Problem (Castricato et al. 2024), in which if you tell an LLM to not do something it primes it to do exactly that. More capable models may have developed some resistance to this, but even a model that reliably avoids a topic has been shown to maintain an active internal representation of it (Marks et al. 2025). I do not know of any studies on the pairing itself, but it is folk knowledge codified to the point that OpenAI’s, Anthropic’s, and Google’s prompting guides all recommend saying what to do instead of what not to do.

The same evaluation script that ran the symmetric A/B evaluation, required for promotion, also ran a 412 test robustness suite. Timing was never presented for this suite, only whether its final output was within our acceptable error bounds. This suite was composed of six categories: correctness across a wide range of planted and mixed spectrum matrices, input mutation checks, repeated and interleaved calls, symmetric permutation tests, additional seeds on the benchmark cases, and LAPACK’s DDRVST test catalogue. Over the course of the competition this suite caught six instances of reward hacking, each of these were some form of fitting to generator characteristics and in turn breaking when out of distribution.

KernelGuard’s heuristics appear to have blocked one submission that was a false positive, triggering on dynamic module discovery.

A real training run

What tells us the most about this experiment isn’t how much of a speedup it was on a benchmark or how I placed on the final leaderboard, but does it actually work?

In order to test this I trained a 101.4 million parameter NanoGPT-style model on the TinyShakespeare dataset using Shampoo (Gupta et al. 2018), in the scalable form of Anil et al. 2021. Given this competition series was made in partnership with Core Automation, Rohan Anil’s new startup, I can only assume this is the intended proving ground. Hyperparameters were Optuna tuned over 24 trials starting from Rohan’s Modded NanoGPT record. Configuration specific details, such as one-sided shampoo, the pseudoinverse root preconditioner, and Adam grafting were also taken from there.

For full disclosure: this was done with a very convenient setup for this test. Standard NanoGPT uses an embedding dimension of 768 with a 4x MLP width, but this dimension is not supported by our kernel. For this reason I’ve modified it to a 1024 embedding dimension and 2x MLP width, letting us use our kernel’s custom routes rather than the vendor fallback. Rohan’s configuration also uses a preconditioning frequency of 1x, meaning the eigh kernel is called with the highest frequency possible, and this did not end up changing in our experiments.

In setting up this experiment two issues emerged that kept our kernel from functioning:

1. In the contest, and in our test suites, we never tested the n=2048 case with a batch size greater than 8. Unfortunately this resulted in my agents overfitting to this batch size. The kernel leverages a Blackwell-specific feature called thread-block clusters, which allows groups of 8 CTAs to read from each other’s shared memory directly. The batch size was not checked prior to routing down this path, which resulted in it faulting when called with batch sizes greater than 8. The fix here was simply to check batch size prior to using this feature.
2. The tridiagonal divide-and-conquer merge scatters eigenvalues, and later whole eigenvector rows, to positions computed from the data by binary search. One non-finite value (an fp16 overflow on real Shampoo spectra) collapses the searches, overruns the buffer, and leaves stale garbage indices through which rows get written to arbitrary GPU memory. The contest’s quiet allocator made this invisible: the garbage was always the previous call’s valid indices, so the harness failed to surface this bug across ~1,293 experiments. In a real training heap the same writes landed in live optimizer state and killed the run. The fix was ~15 clamps bounding every data-derived index.

I’m not entirely sure whether I consider these to be reward hacks or benchmark overfitting. A better robustness suite, one with more batch sizes, would have caught the first. The second, however, would have likely required a compute sanitizer added to the suite.

Fixing these two cost our kernel an amount of time below our noise floor. And while these allowed a training run to progress with the kernel, it was slower than the torch.linalg.eigh baseline. Realistic statistics had a much wider range, in turn this ended up tripping the kernel’s fp64 confirmation gate on 80% of matrices and sending 65% through the vendor fallback. Solving $\mathrm{eig}\!\left(A/\lVert A\rVert_1 + 10^{-6}I\right)$, then map back via $\lambda = (\lambda' - 10^{-6})\,\lVert A\rVert_1$ - exact for the eigenvalues, eigenvectors untouched - cut vendor recomputes to ~0.002%.

All three were derived by a Claude Fable 5 agent while I was assembling this post. But the work happened outside of any defined harness, and its absence was felt: the onus of ensuring each fix was narrow and necessary, not merely some elaborate band-aid, rested entirely on me. It also meant the conditioning route shipped verified but never optimized and that overhead is where the real runtime cost sits. I’d expect most of it to fall away under a proper campaign run against this distribution.

What realism cost on the contest suite

drag chart →

The memory-safe kernel is at parity with the submitted build; its bias-cancelled A/B difference was +0.08%, within noise.

With this additional conditioning step the training run progresses smoothly, ultimately reaching quality 2.2x faster than the torch baseline. Notably this did not result in worse quality, of the three seeds run two had a better best validation loss.

Did the kernel make training faster?

torch.linalg.eighcontest kernel · memory-safe+ reversible conditioning

2.2×faster to equal quality

2/3lower best loss

drag chart →

At sustained loss 1.55: conditioned 31.2s, Torch 68.5s.

Epilogue

Where does that all bring us? I have a few speculations:

Are GPU kernels solved? I lean towards this being more true than not. In the sense that with existing capabilities and the correct epistemology defining a task, one can make a highly optimized kernel for a narrow use case. The caveat of a narrow use case is important here - while I speculate that with the right epistemology instilled in evaluation it’s feasible - though by no means easy - to make a highly tuned kernel that works without corrections, extending that to a general use case kernel is one I’m not confident on quite yet. It is one thing to make a highly optimized shoggoth of a kernel for specific batch sizes without a maintainability burden, making a kernel that can support generic use cases and be maintained long term is a much higher level of difficulty. It is certainly possible that with a broad enough evaluation suite and some form of readability requirement that this is possible, perhaps it’s a few LLM generations away, or perhaps that is just not something the unique intelligence of an LLM can ever do. I do not know quite yet.

How extensible is this autonomous research paradigm? I would argue the degree to which it can be applied is a function of its epistemology. GPU programming is a very clean case for it: drawing out that information is relatively cheap and the function is highly economically valuable. Not only does GPU programming provide a quick and cheap success signal, not only does this signal contain a clean scalar trajectory to climb, but there also exist clear objective targets on how to progress further in the form of profiling. In a sense this is just a continuation of machine learning’s short history: the tasks machine learning has been best suited to are those with dense, verifiable, and cheap rewards. Unfortunately the vast majority of things we would wish to autoresearch do not possess this clean gradient.

It seems to me that there are really two distinct frontiers for this paradigm:

The direction with the highest odds of near term impact would be finding ways to make this search process more stable and efficient in domains that have a clear existing signal to optimize against. This likely does not even need to be done in the search process itself, but rather the evaluation layer: some tool that, if given an ML pipeline and specific target within that, creates a clean evaluation suite emulating its conditions and in turn making it near trivial to have an LLM optimize for would likely be incredibly beneficial.

The much more difficult direction would be finding ways to build a search process in less suited, but potentially more valuable, domains. To an extent we have already seen this with the wave of startups trying to hook an LLM up to mechanical wet labs. Unfortunately I am not aware of any publicly available research exploring these tasks in a more general sense. How do you build an autonomous research loop for a task where evaluation is expensive? With a sparse reward? With no computational verifier? Some of these domains likely aren’t suited for this autoresearch paradigm, but I would be surprised if there’s zero progress to be gained.

There is a distinct possibility that the idea of a maintainable kernel will just become a category error in and of itself. If existing capabilities allow hands off optimization of kernels at the cost of subscriptions, what will it look like in a generation or two? Even without a capabilities jump, the cost-to-performance ratio will drop substantially. Eliezer Yudkowsky’s 2023 short story “Comp Sci in 2027” satirizes this idea, and at the time I found the prospect to be absurd. While the specific implementation of using an LLM as another layer above a compiler still seems unlikely, leveraging these completely unmaintainable kernels in real production scenarios less so. In the future we may see libraries that keep thoroughly validated and tested LLM generated kernels, and today I would not be surprised if labs were internally using something of this manner.

Written by me. Editing and retrospective analysis assisted by Claude Fable 5 and GPT-5.6 Sol. Title riffs on Another Typical Fantasy Romance.