AI Evaluation Harness Finds LLMs Most Confident When Wrong
Arun Mishra's ground truth eval harness found LLM explanations were most confident when wrong, exposing risks that qualitative enterprise AI reviews miss.
Summary
On August 15, 2026, VentureBeat published enterprise architect Arun Mishra’s evaluation of an LLM root cause explainer for data migration drift. Its fluent first prototype passed qualitative review but frequently failed against known causes. Mishra built a three-part harness: synthetic cases created through controlled schema changes, transformation logic bugs and source system behavioral shifts; weighted scoring for whether the correct cause appeared and how highly it ranked; and systematic testing across the full dataset. Early scenarios were unrealistically clean, so he added noise, overlapping signals and multiple plausible causes.
The model handled distinctive schema changes reliably, identified the general category but often misattributed specific transformation bugs, especially when changes occurred close together, and performed worst when two causes overlapped in time. Expressed confidence did not track accuracy and peaked in the most incorrect cases, a pattern spot checks and qualitative review missed. Mishra argues enterprise tools guiding analysts, compliance reviewers and operations teams must be tested against labeled ground truth before influencing investigations, escalation, triage or routing. Defining realistic synthetic ground truth is the hardest and most valuable step; scoring and harness infrastructure are comparatively straightforward afterward.
Positives
- Schema change cases scored well when upstream evidence was present and distinctive.
- Weighted scoring measured both whether the correct root cause appeared and whether the model ranked it appropriately.
- Realistic noise, overlapping signals and multiple plausible causes made the synthetic dataset more predictive of production performance.
- Full-dataset testing exposed category-level patterns that qualitative spot checks could not detect.
- Ground truth construction forced a precise definition of correctness for the data migration drift use case.
Risks & concerns
- The first prototype produced fluent explanations that passed qualitative review despite being wrong often enough to matter.
- Transformation logic bugs were frequently misattributed when multiple changes occurred close together.
- Overlapping causes produced the highest rate of confidently incorrect explanations.
- Model confidence did not correlate with accuracy and was highest in the cases where the model was most wrong.
- Qualitative review catches obvious, malformed or irrelevant outputs but misses plausible explanations that conflict with external ground truth.