HonestML — Your 98% Accuracy Is a Lie: A Leakage Audit That Counts Down From Your Reported Score to the Honest One Your Test Set Was Hiding

The proudest I ever felt about a machine learning model was also the most wrong I've ever been. Final year. My project was something I'd stayed up late working on for weeks. After spending hours on feature engineering, I submitted it, and the test accuracy came back at 97.4%. My guide nodded. My teammates celebrated. For about a week, I genuinely believed I'd learned something real. Then the model was tested on actual data, and its performance dropped to somewhere near 60%. Nothing about the model had changed. What changed was that nobody was grading me anymore. I spent the next few days going through the dataset line by line. Three problems had been hiding there from the beginning. The same samples appeared in both training and testing. Not exact copies, but near duplicates with slight formatting differences, rounding changes, or renamed fields. My model recognised rows it had effectively seen before and called that a prediction. The data wasn't independent. Records belonging to the same person, session, or visit appeared on both sides of the split. My model was being evaluated on data connected to examples it had already learned from. One feature was leaking the target. It looked like a brilliant predictor because it was effectively revealing the answer through another column. I didn't have a modelling problem. I had a dataset that was never telling me the truth. The worst part wasn't fixing it. It was realising that I could have taken that misleading accuracy score into a team, a meeting, or a customer presentation and only discovered the problem when it actually mattered. So I built the tool I wish I'd had in that lab: HonestML. Upload a CSV, choose your target and split columns, and HonestML audits your dataset in seconds. • A k-NN baseline scored twice: once using your original split and once using a cleaned split. • Per-row similarity checks to uncover near-duplicate records. • Group overlap detection across columns that matter to your dataset. • Feature-to-target checks to identify potential data leakage. The output isn't just another chart. It's the gap between your reported performance and your audited performance, along with a leakage risk score from 0 to 100. You also get a cleaned CSV with flagged leaky rows and columns removed, helping you retrain your model on more reliable data. I didn't build this because data leakage is an interesting research problem. I built it because a student got a 97% accuracy score that meant nothing, and everyone around him thought it was impressive. If you're about to publish a model's performance, audit it first. If the result holds up, you earned it. If it doesn't, that's one of the most valuable lessons you can learn.