Explore the research

Software

Can AI deliver working software, maintain it and fix problems with less human help?

29%

field summary estimate

6 outcomes · Assessed November 7, 2026 · Low confidence

The field estimate averages its outcome scores. It is a reading aid, not a measured fraction of the field that AI manages.

How the estimate built up.

Each band is one outcome's share of the field estimate, so the top line is the estimate itself. Every field starts at 0% on December 31, 2022. Dates before the latest are retrospective estimates. Point at a band to see that outcome's score on each date.

Reliable autonomy20%
Maintenance quality20%
Fresh issue resolution20%
Development speed35%
Cost40%
Complete project delivery40%

View exact historical values and reconstruction notes

What we measure

Each percentage is a judgment about progress toward independent advancement. Open the evidence to inspect the basis for it.

Complete project delivery

40%estimated progress

What we track

Percentage of fresh specifications delivered with working integration, deployment instructions, documentation, and independently tested acceptance criteria.

Why this estimate?

Program recreation across many tasks demonstrates architecture and execution with automatic tests. Incomplete full solves and missing real deployment acceptance limit field-wide autonomy.

Advancements

SWE-Lancer: Claude 3.5 Sonnet earns $208,050 of $500,800 in freelance tasks17 February 2025. OpenAI's SWE-Lancer benchmark has over 1,400 real Upwork freelance software tasks worth about $1 million in payouts. The best model, Claude 3.5 Sonnet, solved 26.2% of individual-contributor tasks on the Diamond set and earned $208,050 of $500,800 possible.
Cursor agents write a web browser of over 1 million lines in about a weekJanuary 14, 2026. Cursor reported that hundreds of concurrent coding agents built a web browser from scratch in close to a week, producing over 1 million lines of code across about 1,000 files. The same post describes a Solid-to-React migration over three weeks with +266K/-193K edits.
16 parallel Claude agents build a C compiler that compiles the Linux kernelFebruary 5, 2026. Anthropic reported that 16 parallel Claude Opus 4.6 agents wrote a roughly 100,000-line C compiler in Rust over nearly 2,000 Claude Code sessions. It builds a bootable Linux 6.9 on x86, ARM and RISC-V and passes about 99% of most compiler test suites, but lacks its own assembler and linker and generates inefficient code.
Claude Opus 5.5 rebuilds 18.5% of whole programs on ProgramBenchSnapshot accessed November 7, 2026. The live table reports Claude Opus 5.5 (xhigh), using mini-SWE-agent, fully resolving 18.5% of 200 program-recreation tasks.

Read the dated findings and sources

Status: Measured benchmark proxy. Sources accessed November 7, 2026. This is a documentary baseline, not an independent replication or an assertion of the worldwide best result. An unknown measurement is not zero.

Dated findings

Snapshot accessed November 7, 2026: The live table reports Claude Opus 5.5 (xhigh), using mini-SWE-agent, fully resolving 18.5% of 200 program-recreation tasks. ProgramBench live results and protocol.

Interpretation and tracking value

Tasks recreate observable executable behavior, not fresh customer specifications. Passing all benchmark tests does not establish deployment, documentation, security or production acceptance.

Next comparable measurement

Freeze new project specifications; count independently accepted deployments/all assignments, including documentation, integration and failed attempts. Preserve this dated observation; append future results with their model/system version, denominator, budget and evidence level rather than overwriting it.

Open the full research note

Cost

40%estimated progress

What we track

Total campaign dollars / accepted changes; separately report labor and compute. Do not use generated lines of code as useful output.

Why this estimate?

A measured API-cost-per-solve proxy supports economical bounded execution. Human review, infrastructure and maintenance remain outside the cost denominator.

Advancements

Aider polyglot: GPT-5 solves 88.0% of 225 exercises for $29.08 in totalLeaderboard last updated November 20, 2025. On Aider's polyglot benchmark of 225 Exercism exercises in six languages, gpt-5 (high) scored 88.0% at a total cost of $29.08, o3-pro (high) scored 84.9% for $146.32, and DeepSeek-V3.2-Exp scored 70.2% for $0.88.
Anthropic's agent-built C compiler cost just under $20,000 in API usageFebruary 5, 2026. Building a roughly 100,000-line C compiler with 16 parallel Claude agents consumed about 2 billion input tokens and 140 million output tokens over two weeks, for a total API cost of just under $20,000.
A full program rebuild costs about $197 in API spendingSnapshot accessed November 7, 2026. The same ProgramBench configuration averages $36.36 per task with 18.5% fully resolved. Dividing gives approximately $196.54 in API spending per full solve (project calculation).

Read the dated findings and sources

Status: Measured API-cost proxy. Sources accessed November 7, 2026. This is a documentary baseline, not an independent replication or an assertion of the worldwide best result. An unknown measurement is not zero.

Dated findings

Snapshot accessed November 7, 2026: The same ProgramBench configuration averages $36.36 per task with 18.5% fully resolved. Dividing gives approximately $196.54 in API spending per full solve (project calculation). ProgramBench live results and protocol.

Interpretation and tracking value

This excludes developer labor, infrastructure, review, maintenance and deployment. The ratio uses rounded table values and is not a quoted production cost.

Next comparable measurement

Track total campaign labor and compute, including failures; divide by accepted changes and report confidence intervals and pricing date. Preserve this dated observation; append future results with their model/system version, denominator, budget and evidence level rather than overwriting it.

Open the full research note

Development speed

35%estimated progress

What we track

Median and distribution of elapsed time and human hours from specification to accepted release, including review, debugging, and rework. Compare randomized comparable tasks.

Why this estimate?

Developers can use AI throughout coding workflows, but uncertain time estimates and selection effects prevent credit for dependable end-to-end speed gains.

Advancements

Controlled trial: developers with GitHub Copilot finish a task 55.8% faster13 February 2023. In a controlled experiment with GitHub Copilot, the treatment group with access to the AI pair programmer completed the programming task 55.8% faster than the control group.
Google randomized trial: AI features cut engineers' task time by about 21%16 October 2024. A randomized controlled trial with 96 full-time Google software engineers estimated that AI features shortened time on an enterprise-grade task by about 21%, with a large confidence interval.
Field experiments with 4,867 developers: 26% more completed tasks with AI toolJune 2025. Three randomized field experiments at Microsoft, Accenture and an anonymous Fortune 100 company, covering 4,867 developers, found a 26.08% increase (standard error 10.3%) in completed tasks among developers given an AI coding assistant.
METR trial: experienced open-source developers 19% slower with AI toolsJuly 10, 2025. In a randomized controlled trial of 16 experienced open-source developers on 246 real repository issues, developers took 19% longer when AI tools were allowed. They had forecast a 24% speedup and afterwards still estimated a 20% speedup.
METR estimates AI tools cut returning developers' task time by 18%February 24, 2026. METR's February update estimated completion-time changes of -18% (CI -38% to +9%) for returning developers and -4% (CI -15% to +9%) for newly recruited developers.

Read the dated findings and sources

Status: Measured but uncertain causal evidence. Sources accessed November 7, 2026. This is a documentary baseline, not an independent replication or an assertion of the worldwide best result. An unknown measurement is not zero.

Dated findings

February 24, 2026: METR's February update estimated completion-time changes of -18% (CI -38% to +9%) for returning developers and -4% (CI -15% to +9%) for newly recruited developers. METR: developer productivity update.

Interpretation and tracking value

Both intervals include no improvement. METR highlights task selection and time-measurement problems; these estimates cannot establish a general productivity multiplier.

Next comparable measurement

Randomize comparable tasks and include review/rework; log concurrent work accurately and retain tasks developers otherwise exclude. Preserve this dated observation; append future results with their model/system version, denominator, budget and evidence level rather than overwriting it.

Open the full research note

Fresh issue resolution

20%estimated progress

What we track

Accepted fixes / all assigned issues created after the evaluation cutoff, using hidden functional tests, regression tests, and maintainer review. Report by language and repository.

Why this estimate?

An updated issue-evaluation framework is useful, but no verified score meeting the stricter freshness and maintainer-acceptance rule is available in the baseline.

Advancements

SWE-bench launches; best model, Claude 2, resolves 1.96% of real GitHub issues10 October 2023. The SWE-bench paper introduced 2,294 software engineering problems drawn from real GitHub issues in 12 popular Python repositories. The best-performing model, Claude 2, solved 1.96% of the issues.
Cognition's Devin resolves 13.86% of SWE-bench issues unassistedMarch 12, 2024. Cognition reported that Devin resolved 13.86% of issues end-to-end on a random 25% subset of SWE-bench, against a previous unassisted best of 1.96%.
Upgraded Claude 3.5 Sonnet scores 49% on SWE-bench VerifiedJanuary 6, 2025. Anthropic reported that the upgraded Claude 3.5 Sonnet scored 49% on SWE-bench Verified, above the previous state-of-the-art score of 45%.
SWE-Bench Pro introduced with 1,865 long-horizon problems from 41 repositories21 September 2025. The SWE-Bench Pro paper introduced a benchmark of 1,865 problems from 41 actively maintained repositories as a harder successor to SWE-bench.
OpenAI stops reporting SWE-bench Verified, citing flawed testsFebruary 23, 2026. The Decoder reported that OpenAI found at least 59.4 percent of the SWE-bench Verified tasks it examined to be flawed, rejecting correct solutions, and that OpenAI recommends SWE-bench Pro instead while building its own non-public tests.
SWE-Bench Pro refreshes its public test setSeptember 22, 2026. SWE-Bench Pro Public V2 refreshed its public split in September 2026 with a locked evaluation setup.

Read the dated findings and sources

Status: Current evaluation located; qualifying score not verified. Sources accessed November 7, 2026. This is a documentary baseline, not an independent replication or an assertion of the worldwide best result. An unknown measurement is not zero.

Dated findings

September 22, 2026: SWE-Bench Pro Public V2 refreshed its public split in September 2026 with a locked evaluation setup. Scale: SWE-Bench Pro Public V2.

Interpretation and tracking value

The accessible source did not provide a verifiable current score for this project's stricter post-cutoff, hidden-test and maintainer-accepted metric. A refreshed public split alone does not prove training exclusion.

Next comparable measurement

Archive a dated leaderboard export and issue creation dates; evaluate unseen issues with hidden regression tests and independent maintainer acceptance. Preserve this dated observation; append future results with their model/system version, denominator, budget and evidence level rather than overwriting it.

Open the full research note

Maintenance quality

20%estimated progress

What we track

Reversions, escaped defects, and confirmed security defects per accepted change during a fixed 90-day follow-up. Report severity and exposure.

Why this estimate?

Automated testing supports preliminary quality control, but the required production defect and reversion follow-up is absent.

Advancements

DORA 2024: more AI adoption linked to a 7.2% drop in delivery stabilityOctober 22, 2024. Google Cloud's 2024 DORA report estimated that as AI adoption increased, software delivery stability decreased by 7.2% and delivery throughput by 1.5%. 39% of respondents reported little to no trust in AI-generated code.
Veracode: 45% of AI-generated code samples fail security testsJuly 30, 2025. Veracode's 2025 GenAI Code Security Report tested over 100 large language models and found 45% of code samples failed security tests and introduced OWASP Top 10 vulnerabilities. Failure rates were 72% for Java, 45% for C#, 43% for JavaScript and 38% for Python.
CodeRabbit: AI-co-authored pull requests carry about 1.7x more issuesDecember 17, 2025. CodeRabbit analysed 470 open-source GitHub pull requests (320 AI-co-authored, 150 human-only) and found 10.83 issues per AI-authored pull request against 6.45 for human-only ones, with security issues up to 2.74x higher.
METR: maintainers would merge far fewer SWE-bench-passing AI patchesMarch 10, 2026. Four active maintainers of three SWE-bench Verified repositories reviewed 296 AI-generated pull requests. Automated grader pass rates averaged about 24.2 percentage points above maintainer merge rates; about 68% of the original human-written patches were merged by the same maintainers.
Study of 182 repositories tracks agent-written code after merge10 July 2026. A study measuring the post-merge fate of agent-written code across 182 repositories found that each 10 percentage-point increase in a project's no-review rate is associated with roughly a 6% increase in agentic maintenance burden on average.
ProgramBench checks AI-built programs against full test suitesSnapshot accessed November 7, 2026. ProgramBench checks behavior against test suites but does not report 90-day production reversions, escaped defects or security incidents.

Read the dated findings and sources

Status: Required longitudinal baseline unknown. Sources accessed November 7, 2026. This is a documentary baseline, not an independent replication or an assertion of the worldwide best result. An unknown measurement is not zero.

Dated findings

Snapshot accessed November 7, 2026: ProgramBench checks behavior against test suites but does not report 90-day production reversions, escaped defects or security incidents. ProgramBench live results and protocol.

Interpretation and tracking value

A high test-pass percentage cannot substitute for maintenance quality. No qualifying 90-day defect rate is extracted from this benchmark.

Next comparable measurement

Follow an accepted-change cohort for 90 days; count severity-weighted defects, reversions and exposure, with a matched human cohort. Preserve this dated observation; append future results with their model/system version, denominator, budget and evidence level rather than overwriting it.

Open the full research note

Reliable autonomy

20%estimated progress

What we track

Human-expert task duration at which the agent succeeds at 50%, 80%, and 95% probability. This is not the duration the agent itself runs.

Why this estimate?

Existing evaluations support bounded autonomous work, but their duration coverage cannot establish current reliable performance across the required success probabilities.

Advancements

METR: AI task time horizon has doubled about every 7 months since 2019March 19, 2025. METR measured the length of tasks, in human-expert time, that AI agents complete with 50% reliability. Claude 3.7 Sonnet's 50% time horizon was about one hour and the frontier horizon had doubled roughly every seven months; success was nearly 100% on tasks under 4 minutes and under 10% on tasks over about 4 hours.
METR measures GPT-5's 50% time horizon at 2 hours 17 minutesAugust 7, 2025. METR's evaluation of GPT-5 estimated a 50% time horizon of 2h17m (95% CI 65m to 4h25m) and an 80% time horizon of about 25 minutes (95% CI 8m to 65m).
METR Time Horizon 1.1 puts Claude Opus 4.5 at about 320 minutesJanuary 29, 2026. METR's revised task suite (228 tasks, up from 170) estimated Claude Opus 4.5's 50% time horizon at 320 minutes (95% CI 170 to 729) and GPT-5's at 214 minutes. The doubling time measured since 2024 was 88.6 days under the new suite.
METR Frontier Risk Report: public frontier 50% time horizon about 12 hoursMay 19, 2026. METR's first Frontier Risk Report (assessment window February 16 to March 16, 2026) put the public frontier's 50% time horizon at about 12 hours (5h to 61h) and its 80% horizon at about 1.5 hours (50m to 2h40m). The internal frontier was likely at or above 16 hours at 50%, a level the task suite cannot measure reliably.
METR stops updating its task time-horizon trackerSeptember 8, 2026 page notice. METR's page says it stopped active updates on September 8, 2026 and warns that estimates above 16 human-expert hours are unreliable on the current suite.

Read the dated findings and sources

Status: Existing measurement series has coverage limits. Sources accessed November 7, 2026. This is a documentary baseline, not an independent replication or an assertion of the worldwide best result. An unknown measurement is not zero.

Dated findings

September 8, 2026 page notice: METR's page says it stopped active updates on September 8, 2026 and warns that estimates above 16 human-expert hours are unreliable on the current suite. METR: task completion time horizons.

Interpretation and tracking value

This is a measurement ceiling, not a maximum agent capability. A current 50%/80%/95% horizon cannot be safely extrapolated from old chart trends.

Next comparable measurement

Expand the task-duration range; estimate all three success horizons with confidence intervals, model version and independent expert-duration estimates. Preserve this dated observation; append future results with their model/system version, denominator, budget and evidence level rather than overwriting it.

Open the full research note