5 - (SYM 44) Beyond Jacobson & Truax: Modern Methods for Estimating Clinically Significant Change

Introduction: In intervention trials the focus is most often on differences in the average change over time on outcomes across intervention conditions, with variation in individual trajectories treated as nuisance. In contrast, a primary focus on inferences regarding the improvement (or worsening) of individual participants is best represented clinically significant change (CSC). One of the primary tools in the assessment of CSC is Jacobson and Truax (1991’s) Reliable Change Index (RCI). The original RCI is still very popular, as evidenced by 12,000 total citations and over 800 citations in 2023 alone. However, three specific limitations have been identified with the RCI: a) the RCI estimate is based on a pre-post difference score, b) the scores upon which the RCI estimate is based (typically total scores) often contain both measurement bias and measurement error and c) the RCI standard error of measurement (SEM) is erroneously assumed to be constant across participants and time.

Method: We present an approach that addresses all three limitations simultaneously: a) scale score and SEM estimation using moderated nonlinear factor analysis and b) RCI estimation using a modification of a three-level multilevel model with modeling of observation-specific measurement uncertainty. We present two illustrations comparing the novel RCI methodology to conventional RCI methods: the first from a treatment trial targeting comorbid PTSD/alcohol use disorder among OEF/OIF Veterans and the second from a school-based trial comparing two interventions targeting child externalizing.

Results: In first illustration, the COPE intervention (compared to Relapse Prevention) led to higher percentages of patients with statistically significant improvement (SSI; 75.0% versus 40.7%); using conventional RCI approaches showed no differences in SSI (88.9% for both conditions). The second illustration, from the Coping Power (CP) intervention, showed that the percentages of youth with individual improvements on teacher-reported externalizing greater than d  ≥ .5 were significantly greater in the Individual CP (73%) versus Group CP (45%); however, a) 3 times the number of youth (15% v. 5%) showed significant worsening in Group CP and b) SSI differences were not significant using conventional RCI methods (79.2% versus 73.4% SSI).

Conclusion: Novel forms of CSC analysis give a) more nuanced information regarding clinical outcomes than conventional RCT analysis and b) greater precision in making individual-level inferences than do traditional CSC methodologies.