False discovery in the analysis of interactions as a function of the distributional and metric properties of the outcome

Abstract: Lived experience suggests substantial heterogeneity in how people react to a common stimuli. Previous work has considered several potential sources of bias and confusion in studying interactions but relatively less attention has been devoted to the nature of the outcome variable in such studies. Here, we consider power and false discovery associated with estimates of interaction parameters as a function of the distributional and metric properties of the outcome. Focusing on a variety of models for non-continuously distributed outcomes (binary, count, and ordinal outcomes), we show that power analyses need to carefully attend to specific details of a given situation and that attempts to use the linear model for recovery can be catastrophic in some settings. Focusing on transformations of normally distributed variables (i.e., censoring and departures from interval scaling) we show that linear models produce spurious interaction effects when such effects are absent from the generating model. We also provide illustrations offering geometric intuition as to why interactions can be a challenge for these incorrectly specified linear models. In light of these findings, we make two specific suggestions. First, a careful consideration of the distributional properties of the outcome variable should be a standard component in interaction studies. Second, hand-tailored power calculations should be provided; to that end, we have written software to help researchers scrutinize their ability to study interactions in given data contexts