3 - (SYM 134) Enhancing the Quality of Cognitive Behavioral Therapy in Community Mental Health Through Artificial Intelligence Generated Fidelity Feedback (Project AFFECT)

Abstract Body
Significance: Millions of Americans receive evidence-based psychotherapies (EBPs) like cognitive behavioral therapy (CBT) each year, but no scalable methods exist to evaluate their fidelity, leaving EBP quality and effectiveness unmeasured and unknown. We present preliminary data from an NIMH-funded research partnership to develop, implement, and evaluate an artificial intelligence (AI)-based system to automatically estimate CBT fidelity from a session recording.
Approach: In Phase I, we worked from a prototype to develop a LyssnCBT user interface geared to the needs of community mental health (CMH) agencies. Focus groups and interviews with CMH providers and leadership guided design and development of LyssnCBT. Participants then rated usability and implementation readiness of the tool. In Phase II, we are conducting a hybrid implementation-effectiveness randomized trial to evaluate the effectiveness of LyssnCBT to improve therapist CBT skills and client outcomes, and reduce client drop-out.
Data were collected from participants in a large public CMH system, where most clients identify as members of historically marginalized racial and ethnic groups (eg 40% Black, 16% Hispanic/Latine), reflecting efforts to ensure that innovations to increase access EBPs are inclusive of, informed by, and responsive to these groups.

Results: Phase 1 therapists (n=18) and leaders (n=12) reported high acceptability, appropriateness, and feasibility. They noted LyssnCBT’s utility for supervision and skill development, but raised questions about capturing common factors skills. Feedback was used to refine the tool before Phase 2. While system-level shifts led to recruitment challenges, therapists (n=30) and clients (n=85) recorded more than 250 sessions by March 2024, with fidelity coded within minutes of recording. We present preliminary findings related to the association between LyssnCBT use and therapist CBT skills, as well as client symptom change and treatment drop-out.

Discussion: Successful execution will provide the first automated, scalable CBT fidelity feedback, supporting high-quality training, supervision, and quality assurance, and providing a core technology foundation to support quality delivery of other EBPs in the future.