4 - (SYM 134) Correlation of Machine Learning Derived Estimates of Treatment Mental Health Counseling Content with Patient Outcomes

Background: Use of asynchronous text-based counseling is rapidly growing as an easy-to-access approach to counseling. Similar to in-person treatment, it is challenging to reliably assess quality and content of treatment episodes as measures of process and content do not scale. Accordingly, studies investigating the association between the content of treatment and patient outcomes are often quite small.

Methods: We used machine learning to evaluate clinical content as well as provider use of specific interventions (e.g., empathy, active listening, cognitive behavioral therapist). Participants received text-based counseling between 2014 and 2019 from a licensed mental health clinician via an online and mobile therapy app (Talkspace). Therapists were licensed to provide mental health treatment and were either independent contractors or employees of the product company. The primary outcomes were client engagement in counseling (number of weeks), treatment satisfaction, and changes in client symptoms, measured via the 8-item version of Patient Health Questionnaire (PHQ-8). A previously trained, transformer-based, deep learning model automatically categorized messages into types of therapist interventions and summaries of clinical content.

Results: The total sample included 166,644 clients treated by 4,973 therapists (20,600, 274 messages). There was substantial variability in intervention use and treatment content across therapists. A series of mixed-effects regressions indicated that collectively, interventions and clinical content were associated with key outcomes: engagement (multiple R = 0.43), satisfaction (multiple R = 0.46), and change in PHQ-8 score (multiple R = 0.13).

Conclusion: Consistent with traditional forms of counseling, higher amounts of supportive counseling were associated with improved outcomes. These findings suggest that machine learning–based evaluations of content may increase the scale and specificity of psychotherapy research.