Leakage-Free Early Prediction of Issue Resolution Time in Agile Software Projects: A Comparative Study on Dataset Quality

Abstract

Inaccurate estimation of Issue Resolution Time (IRT) remains a persistent challenge in agile software development, leading to overflow sprints, misallocation of resources, and frequent violations of Service Level Agreement (SLA). Advances in machine learning and predictive analytics offer significant potential to address this challenge by leveraging historical issue-tracking data to generate accurate, data-driven predictions as soon as an issue is created. Early prediction of Issue Resolution Time (IRT) supports sprint planning, risk mitigation, and Service Level Agreement (SLA) compliance in agile software projects. This study proposes a leakage-free prediction framework that restricts features strictly to issue creation time and evaluates Random Forest (RF) and XGBoost under temporal split validation on two datasets: TAWOS (a tracker-native, multi-project dataset) and a Kaggle Jira dataset. A systematic review of 30 papers contextualizes the experimental design by identifying common themes: (i) creation-time metadata strongly predicts resolution duration; (ii) tree ensembles provide dependable baselines; and (iii) temporal and cross-project validation yield more realistic performance estimates than random splits. Experimental results show that XGBoost achieves the best holdout performance on the Kaggle dataset (MAE = 2.278 hours, RMSE = 2.877 hours), while both models perform comparably on TAWOS (MAE ≈ 6.4–6.6 hours). Feature stability analysis across 5-fold temporal cross-validation confirms that creation-time features—particularly Weekday, Hour, Priority, Project_ID, and Assignee_Past_Issues—are consistently the most important predictors, validating the leakage-free design. Residual analysis reveals that XGBoost produces tighter error distributions (Std = 1.93 hours on Kaggle) compared to RF (Std = 5.73 hours), indicating superior prediction consistency.