🔍 Read the full analysis: FrontierCode: How Maintainers Approach Pull Requests From AI on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, testing 23 models on repository changes graded for maintainability as well as correctness. Claude Opus 5.5 led with a 52.2% score and cleared every blocking check in 57.7% of attempts; the results suggest human review remains necessary, while performance and cost vary widely.
Artificial Analysis added FrontierCode results to its model evaluations on October 12, reporting tests of 23 AI models on repository tasks judged against maintainer-style criteria. Claude Opus 5.5 led with a 52.2% score, but cleared all blocking checks in 57.7% of attempts—evidence that even the top result does not make review-free AI pull requests dependable on this benchmark.
FrontierCode asks whether a change would be accepted, rather than treating passing tests as enough. The benchmark checks patches across six areas: hidden tests, command checks for regressions and cleanliness, tests that must fail on the original code, adaptive tests for complex behavior, limits on files and diff size, and an LLM review based on maintainers’ criteria. A failure on any blocking check gives an attempt a score of zero; patches that pass all blocking checks receive a score from 70% to 100%, depending on quality checks.
In Artificial Analysis’s results, GPT-6.1 Sol scored 49.1% and Claude Fable 5.1 scored 49.0%, placing them second and third. Nine models from six labs scored between 43% and 52%. The reported pass rate measures the share of attempts that cleared every blocking check, not the overall FrontierCode score: Opus reached 57.7%, while GPT-6.1 Sol reached 54.7% and Fable 5.1 reached 54.3%. The source says nine of 100 tasks were not solved by any model on any attempt.
Artificial Analysis ran its tests with the same open-source agent harness, Stirrup, for every model, allowing up to 250 turns per attempt and three attempts per task. GPT-6 Luna (medium) served as the judge for the review step. A trajectory scanner flags attempts that look up solutions; those attempts score zero. The published figures therefore reflect performance under this benchmark setup, rather than a guarantee about how a model will perform in a particular team’s workflow.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
The results matter because they measure a part of AI coding that ordinary correctness benchmarks can miss: whether a patch is bounded, clean, and persuasive to a reviewer. A change can pass tests and still add irrelevant files, include weak tests, or fail to meet project-specific standards. FrontierCode makes those concerns part of the score, reflecting the work maintainers face when reviewing generated pull requests.
For teams considering more autonomous coding agents, the leading model’s 57.7% blocking-check pass rate is a caution against treating benchmark leadership as evidence that review can be removed. The figure does not show that every real-world patch fails the rest of the time, but it does show that the top model did not clear all mandatory checks in more than four in ten benchmark attempts. Human review remains relevant under the conditions tested.
The cost figures add another practical trade-off. Artificial Analysis lists Claude Opus 5.5 at $10.15 per task and GPT-6.1 Sol at $0.81; Opus scores 3.1 percentage points higher. Those results do not establish which model is best for a specific organization, but they show why buyers may weigh marginal performance against task costs and the cost of reviewing failed or low-quality changes.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Tests Repository Work
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026. Version 1.1 followed in July with rules against agents looking up solutions online. More than 20 maintainers created tasks drawn from 36 repositories, including Celery, Budibase, uppy, and Mattermost. The source says task authors spent more than 40 hours on each task.
The full benchmark contains 150 private tasks; Artificial Analysis ran its 100-task Main set. Keeping tasks private is intended to reduce the chance that models encountered their solutions during training. The benchmark also compares models under a shared harness and task limit, but its results remain tied to that task set, scoring design, and evaluation date.
Cost and performance do not move in lockstep. Artificial Analysis identifies GPT-6 Luna (max) at 41.9% for $0.09 per task, Xiaomi’s MiMo-V2.6-Pro at 45.4% for $0.18, GPT-6.1 Sol at 49.1% for $0.81, and Opus at 52.2% for $10.15 as points on its cost frontier. The source also reports that several models, including Gemini 4 Argon and Mistral Large 4 Preview, had no FrontierCode result as of October 13.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The results do not show how the models would perform across every codebase or under each organization’s review policies. FrontierCode covers a 100-task Main set in this evaluation, with a particular harness, turn cap, and review judge; the source does not establish that its scores predict acceptance rates in production repositories.
Some prominent models had no result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full, and Mistral Large 4 Preview. Their absence means the current table is not a complete comparison of all models mentioned in the source. The source also does not provide confidence intervals or a detailed account of variation across repeated runs, so small score differences should not be read as definitive rankings.
Artificial Analysis reports costs per task, but the supplied material does not specify how those figures would change with different usage patterns or deployment arrangements. It is also not clear from the reported figures how much of each model’s remaining failure rate comes from correctness, scope violations, testing quality, or the review judgment.
As an affiliate, we earn on qualifying purchases.
Awaiting More Model Results
The next comparison point is whether models without published scores are added to the leaderboard, particularly the absent systems named in the source. Further updates may also clarify how scores vary across repeated attempts and which blocking checks most often stop otherwise promising patches.
For now, readers can use the October results as a dated snapshot of performance on FrontierCode’s Main set, not as a standing guarantee about current models or an automatic deployment recommendation. Teams applying the findings to their own repositories would need to test agents against their projects and retain review practices appropriate to their risks.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is FrontierCode?
FrontierCode is a coding benchmark developed by Cognition with open-source maintainers. It evaluates repository patches for correctness, scope, test quality, and maintainer-style review criteria.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s October 2026 results, with a 52.2% FrontierCode score. It cleared all blocking checks in 57.7% of attempts.
Does a 52.2% score mean the model passed 52.2% of its tasks?
Not exactly. The FrontierCode score is distinct from the reported blocking-check pass rate. For Opus, the score was 52.2%, while the pass rate for clearing every blocking check was 57.7%.
Does the benchmark show that AI pull requests no longer need human review?
No. In this evaluation, the top model failed to clear every blocking check in 42.3% of attempts. The results support retaining review under the tested conditions; they do not determine how every team should handle every AI-generated change.
Which models were missing from the reported comparison?
As of October 13, the source said there were no FrontierCode results for Gemini 4 Argon, Muse Wonder, Muse Wonder Full, and Mistral Large 4 Preview. Their eventual results could change the comparison.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
