DSLReasoning: Evaluating LLM Program Reasoning beyond Pretrained Language Familiarity

Large language models perform strongly on code-related tasks, but pretrained familiarity with existing programming languages can confuse whether models genuinely understand formal semantics. We introduce two synthetic domain specific languages with unfamiliar operators and modified control-flow behavior to evaluate program reasoning under reduced language familiarity. We study program execution and program equivalence checking. Preliminary experiments show strong execution performance but weaker equivalence checking correctness, suggesting that deeper semantic reasoning remains challenging for current LLMs.