Analyzing Speaker-Role Representations in LLMs for Group Discussions

Speaker roles provide important information for understanding how participants contribute to group discussions. In the machine learning research that aimed at automatically recognizing speaker roles, linguistic information constitutes one of the most important modalities, and previous studies have addressed this task by leveraging pre-trained large language models (LLMs). However, it remains unclear how such concepts related to communication dynamics are captured within LLMs, and in which layers the information for identifying speaker roles is encoded. In this study, using the probing technique, we investigate how speaker roles can be predicted from the layer-wise utterance representations in multiparty group discussions involving four participants. The results of investigating two Japanese LLMs reveal that the layers from which role-related information can be extracted differ across roles. Specifically, the Follower role, which is characterized by short and formulaic utterances, can be predicted accurately even in shallow layers and remains distinguishable in deeper layers, whereas roles that actively drive discussions are best captured in intermediate layers. These results are observed consistently across two LLMs of different parameter sizes, suggesting that identifying representations from layers appropriate to the characteristics of each role is important for designing linguistic features for speaker-role estimation.