AI-based Analysis and Generation of Nonverbal Behavior: NEUROGES® as a Neuropsychologically Grounded Behavior Taxonomy for Health Applications

Health and therapeutic applications impose stringent requirements on the analysis of human nonverbal behavior and the generation of behavior for socially interactive agents (SIAs): both must be not only natural-looking but also clinically appropriate, explainable, and safe. Current approaches fall short on both sides: representation systems lack functional grounding, and generation prioritizes perceptual naturalness over communicative function. With this position paper, we propose to employ NEUROGES® as a shared taxonomy that addresses this gap across analysis, generation, and evaluation. For analysis, it provides a neuropsychologically grounded, clinically validated categorization of nonverbal behavior, registering movement units linked to cognitive, emotional, and interactive processes. The same categories can be used to guide the generation of SIA behavior, allowing domain experts to specify behavioral constraints at the level of communicative function (e.g., suppressing dominance cues in anxiety-provoking situations). This symmetry allows direct evaluation of SIA behavior against NEUROGES-annotated human references, enabling cross-system comparison on a justified scale. We propose a research agenda, arguing that this approach is essential wherever generated behavior must be appropriate, explainable, and safe, which is most relevant in health applications.