Research library
Research areas and measurement rules
Prepared November 7, 2026. These documents propose ways to track consequential AI progress. They are measurement specifications, not claims that the milestones have already been achieved. Numerical targets below are project proposals, not scientific consensus or forecasts. Baselines remain unmeasured unless explicitly sourced.
View the research progress summary for all areas, measurable outcomes, percentage estimates and links to the detailed files. The historical graph index links to one page per area with a four-line comparison graph and reconstruction notes.
Areas
| Document | Promise | Primary signal |
|---|---|---|
| Mathematics | New theorems and dependable reasoning | Substantially new, expert-reviewed, Lean-verified proofs |
| Software | Faster creation and maintenance of reliable software | Accepted work per total dollar and human hour |
| AI Research | AI improving the process that creates AI | Successive, independently verified gains in AI research ability |
| Chips | More efficient computing infrastructure | Measured workload performance per watt and dollar |
| Robotics | Useful automation of physical work | Complete real-world tasks without intervention |
| Materials | Better batteries, semiconductors, and structural materials | Independently synthesized materials meeting property targets |
| Chemistry | Faster discovery of useful reactions and processes | Reproducible improvements in synthesis yield, cost, and waste |
| Biology | Understanding and engineering biological function | Prospective predictions and functional designs validated experimentally |
| Drug Discovery | More effective treatments developed with less wasted effort | Clinical benefit and development success across candidate cohorts |
| Medicine | Better diagnosis and delivery of care | Patient outcomes in prospective comparative studies |
| Energy | More reliable, economical clean energy | Delivered energy and system costs under matched conditions |
| Weather | Better forecasting and adaptation decisions | Prospective forecast skill and verified decision benefit |
| Physics | New explanatory models and faster experiments | Novel predictions confirmed by independent experiments |
| Education | Broad access to effective personalized instruction | Retained learning and transfer without AI assistance |
Each area folder contains a definition.md and a measurable outcomes subfolder. Each measurable feature has its own document, linked from the definition's feature table.
Baseline coverage — November 7, 2026
All 85 measurable-outcome files across 14 areas now contain dated findings, primary-source links, interpretation limits and a next comparable measurement. This includes the original six mathematics baselines and 79 additional baselines. The review records recent evidence where available and explicitly dated older reference results where a newer comparable measurement was not verified. It is a documentary survey, not an exhaustive world-record catalogue or independent experimental replication.
Each file distinguishes a measured outcome from a proxy, demonstration, planned project or missing measurement. An unknown value is not zero; an announced deployment or trial is not a completed impact result. Numerical observations retain their own source dates and evaluation conditions.
All 85 outcomes also have dated judgment-based progress estimates and individual explanations using the requested pre-2022-to-fully-autonomous scale. The shared scoring rubric explains the five components, uncertainty and update rules. Each area's definition contains a summary table; the existing mathematics estimates are unchanged. These judgments are separate from empirical benchmark percentages and must not be averaged into a field-wide completion score.
How to interpret progress
Separate three claims: capability on a controlled task, useful impact in practice, and feedback that improves subsequent AI development. Medicine and education can deliver major benefits without demonstrating recursive AI improvement. AI research is the most direct feedback measure; software and chips are enabling measures. No individual milestone or average across these areas establishes that the singularity has occurred. See Definition.
Shared measurement protocol
Apply these rules to every area document:
Freeze the test before the run. Record task selection, success criteria, permitted tools, budgets, comparator, and stopping rule. Use fresh or private tasks where possible; document possible training exposure. Replace saturated tests and overlap old and new versions for calibration.
Use explicit denominators. Report successes / all assigned eligible attempts, including failures, timeouts, and abandoned attempts. Distinguish first-attempt success from best-of-N results. Report N and the entire search budget.
Compare fairly. Use a competent human or established non-AI baseline on equivalent tasks. Separate human-only, human-plus-AI, and autonomous conditions. Measure both equal-budget performance and total cost to reach equal quality. Record data, compute, equipment, and personnel differences.
Count the complete cost. Record wall-clock time, active human hours, review and repair time, compute, experimental costs, and failed attempts. Cost per accepted result = total campaign cost / accepted results. If none succeed, report no successful result and the full spent budget.
Measure uncertainty. Give sample size, uncertainty intervals, per-domain results, and repeat runs where appropriate. Use power analysis for comparative studies. Small pilot targets in these documents are screening milestones, not sufficient evidence of population-wide superiority. Prespecify primary outcomes and account for multiple comparisons.
Require independent verification. Record whether evidence is a developer report, outside evaluation, peer-reviewed result, independent replication, or field deployment. Peer review alone is not replication. Archive artifacts, model/version, prompts or accessible run records, code, data provenance, and evaluator decisions; use controlled audit access where data cannot be public.
Separate novelty from correctness. Compare with prior work and classify each result as reproduction, incremental improvement, substantially new method, or previously unknown result. Obtain independent domain review for major novelty claims. Count a discovery once even when it appears in multiple area documents.
Track autonomy honestly. Record interventions per task and minutes of human assistance, including setup, task selection, troubleshooting, interpretation, and validation. Do not describe hidden human work as autonomous performance.
Keep thresholds stable. Report relative changes and absolute values with units. For a lower-is-better metric, reduction = 100 × (baseline − AI) / baseline; use only positive baselines. Preregister the acceptable quality, reliability, and harm limits alongside speed or cost targets.
Tracking cadence and record
Review digital benchmarks quarterly; update experiments, clinical trials, and deployments when new results become available. Preserve dated snapshots rather than overwriting historical scores. A simple record for each claim is:
Area | Metric | Test version | Date | Model and tools | Human role | Baseline | Result and units | N | Uncertainty | Full cost | Verification level | Evidence link | Limitations
For discovery campaigns, additionally record the original question, prior-art search, novelty ruling, unsuccessful attempts, and date of independent validation. For AI feedback, record which earlier AI-produced improvement caused which later gain, using ablation or controlled comparison where feasible.
Methodological anchors
METR's long-task research provides a task-duration approach to autonomy. RE-Bench provides human comparisons for research engineering. OpenAI's critique of SWE-bench Verified illustrates why contamination and defective tests must be checked. These motivate parts of this protocol; the project targets and cross-domain rules are proposed here.