Research library
Software engineering
View the four-date completion graphs and reconstruction notes.
Promise and evidence
AI could shorten the creation, repair, and maintenance of reliable software, including tools used to develop AI. SWE-bench measures resolution of repository issues; METR measures task horizons in terms of human task duration. Public benchmark scores require caution: OpenAI's 2026 audit reports contamination and test defects in SWE-bench Verified.
Apply the shared measurement protocol. Targets below are proposals.
Dated baseline coverage
All linked measurable-outcome files below contain an evidence baseline reviewed November 7, 2026, with source dates, findings, limitations and follow-up measurements. These distinguish measured results from proxies and unknown values; they do not assert that proposed milestones are achieved. See the baseline coverage notes.
Progress estimates
Judgment-based estimates as of November 7, 2026, using the same 0–100% scale and five-part rubric as mathematics. 0% anchors the period before ChatGPT's public release in 2022; 100% requires AI to manage this field's continuing advancement without human participation. Confidence is low. These are evidence-based judgments, not empirical completion measurements, forecasts or percentages of jobs replaced.
| Measurable outcome | Estimated progress | Execution / breadth / agenda / validation / improvement (each out of 20) |
|---|---|---|
| Complete project delivery | 40% | 10 / 10 / 5 / 10 / 5 |
| Cost | 40% | 10 / 10 / 5 / 10 / 5 |
| Development speed | 35% | 10 / 10 / 5 / 5 / 5 |
| Fresh issue resolution | 20% | 5 / 5 / 5 / 5 / 0 |
| Maintenance quality | 20% | 5 / 5 / 5 / 5 / 0 |
| Reliable autonomy | 20% | 5 / 5 / 5 / 5 / 0 |
Each linked outcome explains its score, cites its dated evidence baseline and identifies the next measurement. Zero indicates no qualifying post-anchor progress established by the reviewed evidence for that outcome, not proof of no relevant capability. Do not average these overlapping indicators into a field-wide percentage or interpret their differences as a precise ranking of fields. All outcomes reaching 100% would still require a separate field-wide demonstration of sustained autonomous advancement.
Measurable features
| Feature | Measurement and acceptance rule |
|---|---|
| Fresh issue resolution | Accepted fixes / all assigned issues created after the evaluation cutoff, using hidden functional tests, regression tests, and maintainer review. Report by language and repository. |
| Development speed | Median and distribution of elapsed time and human hours from specification to accepted release, including review, debugging, and rework. Compare randomized comparable tasks. |
| Reliable autonomy | Human-expert task duration at which the agent succeeds at 50%, 80%, and 95% probability. This is not the duration the agent itself runs. |
| Maintenance quality | Reversions, escaped defects, and confirmed security defects per accepted change during a fixed 90-day follow-up. Report severity and exposure. |
| Cost | Total campaign dollars / accepted changes; separately report labor and compute. Do not use generated lines of code as useful output. |
| Complete project delivery | Percentage of fresh specifications delivered with working integration, deployment instructions, documentation, and independently tested acceptance criteria. |
Evidence requirements
Archive specifications, initial repository states, patches, independent tests, run logs, reviewer decisions, and follow-up defects. Reject test deletion, weakened acceptance criteria, hard-coded hidden answers, and fixes that pass narrow tests while violating the specification. Evaluate the tests themselves. Separate public benchmark results from private prospective evaluations.
Proposed milestones
Resolve at least 80 of 100 fresh issues across ten repositories under a fixed budget, with independent maintainer acceptance and no unreviewed test changes.
In a powered comparative study, reduce total human hours by at least 25% while remaining within a preregistered non-inferiority margin for serious defects over 90 days.
Deliver five independently commissioned projects involving work estimated at a human week each, meeting at least 90% of mandatory acceptance criteria and every critical criterion; disclose all interventions.
Show that AI-created development tools measurably improve subsequent AI research productivity on a new task set. Ordinary application delivery is useful impact but does not by itself establish this feedback.