Overengineered Underexplanations: Why What-If XAI Systems Fail to Assist Clinicians with Prognostic Tasks Due to Missing Strategic Conformance

Clinicians can struggle to give accurate prognoses to patients and seek help from XAI what-if analysis systems to make more accurate predictions and explore the effects of treatment possibilities. However, many clinicians find XAI systems’ explanations incompatible with their workflows and exclude them from decision-making. We investigated how to design prognostic XAI what-if systems through two studies. The first qualitative study with five neurologists showed that clinicians reason deductively but rejected rule-based explanations due to their algorithmically determined rules. A second study with fourteen neurologists compared a panel-based and conversational interface for what-if analysis. While each interface posed different benefits and challenges, clinicians found the underlying model misaligned with their clinical deductive reasoning. We highlight the hypothesis of strategic conformance as a requirement, arguing that XAI systems must embed users’ domain knowledge into both the model and its explanations before interface design decisions become meaningful.