Can LLMs Substitute for Humans in Evaluating Conceptual Modeling Language Vocabularies? An Experimental Study

Effective model-driven engineering of software-intensive systems relies on appropriately designed conceptual modeling languages. A well-designed modeling language includes a set of concepts that users of the language find to be relevant for describing phenomena of interest in a domain, as well as a vocabulary of lexical, symbolic, or visual signifiers for representing these concepts. Being the product of a human design process, however, a modeling language may or may not feature the optimal choice of concepts and signifiers. One of the ways that have been proposed to assess the quality of those choices is based on requesting potential users of the language to use it for classifying a sample of chunks of information from an example domain under one or more of the proposed concepts of the language. Characteristics of the classification results, such as agreement among users and between users and authoritative classifications, are revealing of the quality of the chosen set of concepts and their signifiers. In this paper, we report on an experimental investigation on using Large Language Models (LLMs) as substitutes for human input in such classifications to allow rapid, cost-effective acquisition of quality data. Specifically, using an example subject language, we compare the accuracy of human versus LLM classifications with respect to authoritative standards. We find that LLM performance in the specific tasks is substantially higher than human performance, challenging the idea that LLMs can be readily used as a stand-in for humans in such tasks. We discuss the implications of this finding for future research.