An Empirical Analysis of Test Failures in AI-Generated Pull Requests

Alireza Hoseinpour, Sajjad Rezvani Boroujeni, Jashhvanth Tamilselvan Kunthavai, Kyle Cusimano, Abbas Heydarnoori
April, 2026
DOI

Abstract

Recent advances in large language models (LLMs) and AI agents have enabled automated test generation to become a practical component of modern software development workflows. However, these systems frequently produce tests that fail to compile or execute correctly, raising questions about their robustness in real-world settings. In this study, we conduct an empirical analysis of test failures in the AIDev dataset from the MSR 2026 Mining Challenge. Particularly, we examine 500 merged pull requests containing test-related failures (100 per agent across 5 AI coding agents) and classify each case using a refined taxonomy that distinguishes between compile-time and runtime errors. Our findings reveal that runtime errors dominate (62.6% vs 37.4% compile-time), with assertion failures being the most common issue (28.6%). The analysis exposes distinct failure profiles across agents and provides insights for improving AI-driven testing reliability in large-scale development environments.

Type
1
Publication
Proceedings of the 23rd International Conference on Mining Software Repositories (MSR), Mining Challenge Track
software engineering AI4SE SE4AI AI coding agents Software Testing Empirical Study
Alireza Hoseinpour
Alireza Hoseinpour
Ph.D. Student in Computer Science

My research interests lie at the intersection of agentic AI and software engineering, including LLM agents for software engineering, testing and debugging AI agents, and software maintenance and evolution.