SyncAI.news, a Varaisys broadcasting
Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase
DL

Douglas Leith

· 1 min read

ResearcharXiv cs.AI

Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase

arXiv:2609.29744v1 Announce Type: cross Abstract: We present: (i) a new dataset consisting of the full development history of a 21,000-line Python tool built entirely by Claude AI, with no human-authored code or tests, (ii) two code-provenance tracing tools, (iii) three taxonomies for instruction intent, commit provenance, and response reliability, (iv) application of these to analyse the dataset. We find that: (i) user coding agent CLI instructions differ in kind from IDE-chat instructions, with a greater focus on comprehension, planning and consultation, (ii) code development is mainly proactive, (iii) 14.3% of AI code-generation events contain a real error later caught by the AI-authored test suite, (iv) roughly 1 in 4-5 of the AI's interactive responses contains one or more factual errors.

Original source

This story was published by arXiv cs.AI and written by Douglas Leith. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News