Break It First: Stakeholder Participation in Quality Assurance for LLM Features

Large language model (LLM) features are difficult to predict upfront. Their useful behavior, failure modes, and acceptable boundaries often become visible only through situated interaction. This Industry Experience paper reports a stakeholder-led quality assurance process used in the development of an LLM-enabled website search assistant. Stakeholders first described expected and unwanted behaviors, then deliberately tested the artefact to find errors before wider deployment. The process produced 44 behavioral specifications, 29 unwanted behaviors, 13 testing targets, and 26 issue reports, including content errors and contextual risks that depended on stakeholder knowledge of the operating environment. We argue that LLM quality assurance should treat stakeholder participation not only as acceptance testers but as a practical adjustment space for discovering situated errors and refining system behavior.