Typed Template Fuzzing

Fuzzing finds bugs by testing software behavior with high volumes of input. To create semantically valid inputs for different targets, traditional fuzzers require significant engineering effort. LLM-based fuzzing approaches can be adapted to create inputs for a wide range of software but introduce new issues: every test needs an LLM query, sacrificing the speed of traditional fuzzing. We introduce Typed Template Fuzzing (TTF). It uses LLMs in a preprocessing step to create templates which encode semantic information of seed files. These templates enable semantically-aware mutations without the need for a parser, type-checker, or further LLM queries. We have implemented TTF in our tool Typos. Using Typos we have run a five-month fuzzing campaign, finding 97 previously undiscovered bugs. Typos achieves a 94.7% rate of valid inputs across a wide range of formats including SMT-LIB2, C, C++, Go, Rust, SystemVerilog, Scala, Java, and domain-specific libraries such as PyTorch and Qiskit.