A versioned evidence review

Methodology

The research question, evidence rules, and output contract behind the editions. Model stamps record attribution; differences also reflect sources, retrieval, timing, and research configuration.

Current methodology v1.1.0. Edition 2026-09 used v1.1.0.

Methodology

Current prompt version: 1.1.0 · Data schema: 1.1.0

State of Agentic Coding is a structured evidence review for developers and engineering leads choosing coding workflows. It combines research literature, documented product behavior, and first-hand experience, with explicit limits on what each establishes. It does not claim systematic sampling of the literature or original comparative measurements without a separate experimental protocol.

The v1.0.0 protocol remains available for historical editions. The data contract defines exact fields and generated tables; the machine-readable contract is shared by scaffolding, validation, and rendering.

Reproducibility and attribution

A live web review cannot isolate the effect of the researching model. Findings also depend on retrieval, source selection, harness capabilities, timing, and randomness. Model stamps provide attribution. The protocol, evidence ledger, and research log make reasoning inspectable; they do not guarantee identical outputs or prove why two editions disagree.

To study researcher-model effects, use a separately scoped experiment with a frozen corpus, prompt, output rubric, and recorded configuration. Do not describe ordinary live editions as that experiment. Source counts measure workload, not quality. Record unavailable provenance fields honestly as unknown.

Running an edition

Read this document and the data contract together. Copy the prompt below into the research harness; replace {{AS_OF_DATE}} with the agreed YYYY-MM-DD cutoff and {{BASELINE_EDITION}} with an edition slug or none. Record actual run dates separately. A September run covers evidence through its cutoff, not the rest of September. Later verification may confirm an earlier state only when the source supports that effective date; exclude subsequent developments from the snapshot.

The default scope is an evidence review. Original hands-on benchmarking needs a separate agreed task set, access/compute budget, trial plan, and grading protocol. Parallel research is optional when authorized and supported; one researcher can execute every phase serially.

Research prompt v1.1.0 

BEGIN RESEARCH PROMPT v1.1.0

Produce an edition of State of Agentic Coding as of {{AS_OF_DATE}}, using
{{BASELINE_EDITION}} as its baseline. Read methodology/SCHEMA.md and
methodology/contract-v1.1.json as part of this prompt's output contract.
Training knowledge and prior editions are discovery aids, not current evidence.

RESEARCH QUESTION
For which software tasks, users, and operating constraints does each
model-harness-configuration combination provide a defensible advantage,
what evidence supports that judgment, and what changed since the baseline?

DECISION FRAMEWORK
- Identify audience, task, language/architecture, repo maturity, test coverage,
  familiarity, ambiguity, budget, deployment constraints, and acceptable autonomy.
- Compare complete setups: model identifier, harness/version, configuration,
  tools, environment, access route, and availability/tier.
- Consider accepted outcomes, review/rework, elapsed time, cost, interventions,
  portability, and operational controls. Missing measurements are unknown.
- Recommendations need prerequisites, a preferred setup or insufficient-evidence
  finding, an alternative, tradeoffs, confidence with reasons, supporting claim
  IDs, and evidence that would change the decision. Avoid universal winners and
  composite scores that combine incompatible metrics with arbitrary weights.

COVERAGE: retain these 13 report sections, in this order
1. Landscape overview: discover the roster before reading the baseline's winners.
   Identify material changes and distinguish announcements, previews, staged
   rollouts, and generally available products.
2. Frontier model comparison: compare setups on concrete tasks. Separate comparable
   measurements from descriptive results. Explain measurement/experience conflicts.
3. Harness ecosystem: terminal, IDE, cloud, review, and open-source agents; context,
   retrieval, compaction, memory, recovery, testing/browser use, and portability of
   sessions/configuration. Separate documented capability from observed value.
4. Project size and maturity: greenfield/UI, growing projects, large/legacy repos,
   maintenance, and long sessions. Line count is context, not a universal boundary;
   include architecture, ambiguity, familiarity, and test quality.
5. On-prem/self-hosted: useful options across ~8/16/24 GB VRAM and larger systems.
   Distinguish VRAM, system RAM, unified memory, weight quantization, KV-cache
   precision, usable context, serving stack, tool reliability, and measured latency.
   Weights fitting in memory do not establish a usable agent. Scope costs honestly.
6. Pricing/economics: subscriptions, included credits, overages, API input/output,
   cache reads/writes, promotions, regional premiums, and budget controls. Use exact
   units. Workload estimates disclose assumptions and formulas. Measured cost per
   accepted task needs evidence including failures/retries and the denominator.
   Report review effort separately; monetization requires an explicit labor rate.
7. Client compatibility: model, client/version, OS/execution surface, native/BYO-key/
   compatible-endpoint support, account route, MCP, plugins/skills, session transfer,
   and limits. Protocol support alone does not establish interoperability.
8. Privacy/training: exact product/model/tier, client, hosting/intermediary route,
   region, training, retention, human access, residency, deletion, content exclusions,
   and exceptions. No training does not imply no retention. Check plugin/tool paths.
9. Security/authority: prompt injection, tools/supply chain, sandbox/network/
   credential boundaries, permissions, auditability, rollback, and review authority.
   For review agents examine defects, false positives, and review independence.
10. Productivity: distinguish randomized effects, observational associations,
    vendor telemetry, and self-reports. Record population, tasks, tool vintage,
    sample, outcomes, uncertainty, and validity limits. PR/code volume or agent
    runtime alone is not engineering value. Preserve material follow-up evidence.
11. Vendor risk/portability: availability, deprecations, routing/fallback changes,
    migration effort, IP terms, and regulation only where decision-changing.
12. Team workflows: background/parallel work, review queues, ownership, integration,
    human attention, operating costs, and governance. Examine outcomes and failures;
    a demonstration does not establish reliable unattended production work.
13. Recommendations: individual and team scenarios, alternatives, confidence,
    disconfirming conditions, rechecked/unchanged choices, and a watchlist.

PHASES: execute in order, returning to earlier phases when evidence changes a choice
1. SCOPE AND CRITERIA. Freeze cutoff, baseline, discovery/sentiment windows,
   audiences, candidate inclusion/exclusion criteria, and search budgets. Focus on
   changes since the baseline but retain older relevant studies with their vintage.
   Normally use a 90-day first-hand experience window; record exceptions. Discover
   across major providers and the open ecosystem independently. Missing search hits
   do not establish absence. Record research configuration and observability limits.
2. DISCOVER AND LOG. Record queries, date/domain filters, venues, selection and
   exclusion reasons, access failures, budget/stopping decisions. Seek negative
   findings too. Trace reports to original evidence; deduplicate authors, incidents,
   studies, and syndicated reporting.
3. FETCH AND EXTRACT. Open underlying sources; snippets are leads. Record exact
   URLs, publication/update/effective dates where known, retrieval timestamps, and
   supporting locators or short excerpts. Capture scope/evidence type in the ledger.
   Preserve revision IDs/hashes when available without redistributing full copyrighted
   pages. A hash/timestamp documents retrieval, not the truth of an interpretation.
4. VERIFY AND CHALLENGE. Review every recommendation-critical claim and factual
   table entry against its support. Seek counterevidence for comparative claims,
   check independence and follow-up/correction notices, and record conflicts,
   verification status, qualitative confidence/reasons, and limitations. Unresolved
   findings remain disputed/unverified or are omitted, never silently promoted.
5. SYNTHESIZE. Write recommendations, the report, and changes since baseline.
   Separate market changes, corrections, and methodology changes. Make the edition
   standalone. Generate at-a-glance tables from canonical CSVs and shared recommendation
   blocks in summary/report. Preserve decision-changing qualifications in every layer.
   Create a readable source/claim register alongside the machine-readable ledger.
6. VALIDATE AND REVIEW. Check structure, source-to-claim support, recommendation
   logic, arithmetic/comparison groups, generated tables, and condensed caveats; build
   and inspect the site. Record reviewer, time, results, and remaining limits.
   Automated checks verify references/consistency, not truth. Re-fetch volatile
   prices, access and policies in this phase, respecting the frozen effective cutoff.

EVIDENCE RULES
- Official terms govern precisely scoped contractual statements. First-party
  experiments, independent studies, observations, and anecdotes establish different
  things. Source authority and empirical study design are separate axes.
- Every factual table row references claim IDs covering its factual cells. Every
  recommendation-critical claim resolves to exact support in source records.
  Other factual prose still needs inline citations. Preserve material qualifications.
- A vendor's policy may suffice for its own scoped terms. Comparative quality needs
  relevant empirical support with limits. Different domains repeating the same study
  are not independent corroboration. Label vendor measurements and estimates.
- Developer experience is a documented convenience sample, not population polling.
  A recurring pattern normally needs three independent first-hand accounts across
  two venues; otherwise call it a single/limited report. That threshold establishes
  no prevalence. Note affiliations/sponsorship. Do not infer developer consensus
  from selected posts, upvotes, or aggregators. Unopened accounts are not confirmed quotes.
- Benchmark records identify dataset/version/split, evaluator, model/harness/settings,
  tool access, resource/time/token budgets, attempts, metric, sample, uncertainty,
  contamination, and fallback routing. Unknown setups stay outside ranked groups.
  Groups need a reasoned comparison design. Differing harnesses can support a system
  comparison, never model-only attribution. Distinguish pass@1, best-of-N, and repeated
  success. Unknown is not zero. Preserve dataset revisions and reported uncertainty.
- State high/medium/low confidence with reasons. Insufficient evidence is acceptable.
  Do not invent confidence percentages or infer task-cost ratios from token prices.
- Record researcher/harness identity honestly; do not invent backend identifiers,
  usage costs, source counts, completion status, or independent review.

OUTPUT CONTRACT
Produce metadata.yaml; abstract.md (<=200 words including provenance); summary.md
(<=700 words including its final generated recommendation table); at-a-glance.md
(generated); report.md (sections 1-13); sources.md (readable evidence register);
changes.md; research-log.md; evidence/sources.jsonl; evidence/claims.jsonl; and seven
CSVs under data/: setups, benchmarks, recommendations, pricing, privacy-policies,
client-compatibility, onprem-tiers. Follow the exact schemas and table markers in
methodology/SCHEMA.md and contract-v1.1.json. FULL.md and ERRATA.md are optional.
Record incomplete coverage in known_gaps and visible report qualifications.
Do not mark sections or editorial review complete when scaffolding files.

STYLE
Write for a skeptical senior engineer. Lead with decisions/changes, quantify where
supported, label estimates/vendor claims, and give conditions instead of slogans.
Keep prose readable; the ledger holds detailed provenance. New/thinly evidenced
products belong in a watchlist until evidence supports a recommendation.

END RESEARCH PROMPT v1.1.0

September scope

The approved plan is a complete evidence review, updated decision tables, and a July-to-September change summary. Deepen everyday implementation/debugging; large-repo/long-session reliability; unattended execution/review; budget/open-model workflows; and privacy-constrained teams. Include greenfield/UI and local hardware within those scenarios. Retain 13 sections without expanding into a general AI industry report. Original comparative testing is a separately scoped pilot.

Versioning

Record prompt and schema versions in each edition. Retain historical adapters; never reinterpret v1.0 data as satisfying v1.1. Explain methodological differences when comparing editions produced under different protocols.

Changelog