2 - (SYM 44) Variability in Operationalizing the Standardized Mean Difference and Reliable Change in Clinical Trials

Cognitive behavioral therapies (CBT) often aim to promote change in one or more psychosocial outcomes. A key step in developing, innovating on, and ultimately being able to advocate for CBTs is to formally evaluate their effects in clinical trials. Common practice for quantifying change in psychological outcomes is to compute the standardized mean difference (SMD; e.g., Cohen’s d) and reliable change index (RCI). The SMD is a standardized estimate of the average (group-level) treatment effect, whereas the RCI quantifies change at the individual level by evaluating whether each participant reliably improved. These metrics serve complementary purposes and have promise in providing standard benchmarks for quantifying change and evaluating innovations in CBT and other interventions. However, researchers commonly differ in how they operationalize these metrics, even when ostensibly using the same formula. For example, the SMD divides change estimates by the measure’s standard deviation (SD). However, the SD can be operationalized as the baseline SD, pooled SD across time, control group SD, or SD of change scores. The RCI also requires operationalizing the SD as well as a reliability estimate, where at least two distinct operationalizations are commonly used. When researchers use different criteria for quantifying the SMD or the RCI, estimates of change can vary, and these metrics lose their meaning as standardized metrics.

This talk demonstrates how SMD and RCI operationalizations differ across studies, presents results of a systematic review of current operationalizations of these metrics, and empirically demonstrates how different operationalizations of the same metric impacts change estimates. Preliminary results from a pilot review of 39 clinical trials published in the Journal of Consulting and Clinical Psychology found 3 distinct ways the SMD is operationalized and 5 distinct ways the RCI is operationalized, including 4 distinct ways of operationalizing the same RCI formula from Jacobson & Truax (1991). Preliminary estimates of variability in the SMD and RCI found the SMD varied from 1-55% and the RCI varied from 8-63% due to different operationalization of the same formula. The SMD and RCI contain great promise for providing a common meaning for group-level and individual-level change estimates in clinical research, against which to evaluate and ultimately advocate for widespread CBT implementation. We close by discussing paths forward to enable these metrics to match their promise.