Mixed Subjects Designs

Large language models (LLMs) have prompted proposals to replace human subjects in surveys, social science experiments, and psychometric research with simulated responses. Empirical evaluations suggest that this practice (often called silicon sampling) is not yet interchangeable with human responses. We study an alternative approach: one in which model-based predictions are used not as substitutes for human data, but as auxiliary measurements within other estimation contexts, including randomized experiments and psychometric calibration studies. This setup, which Broska, Howes, and van Loon refer to as The Mixed Subjects Design (2025), requires specific care to guarantee unbiasedness when LLM generated responses are only potentially informative. As such, this work focuses on formalizing appropriate estimands for applied mixed-subjects designs and the related software support required for researchers to implement them. We also extend this work to focus on integrating LLM generated item responses in psychometric calibration and research.

Materials:

• Preprint: Using large language models as a source of human behavioral data in social science experiments
• mixedsubjects: An R package for estimating causal effects in social science experiements with a mixed subjects design
• mixedsubjectsirt: An R package for doing psychometric calibrations with a mixed subjects design