(PS9-22) Evaluating Agreement Between Medical Record Diagnoses and Independent Evaluator Diagnoses in a Community-based Effectiveness Study

Anxiety disorders are prevalent in youth, but undertreated. This could be partially due to underdiagnosis, as there is poor diagnostic agreement between clinicians and researchers. One key source of diagnostic errors is the under-utilization of evidence-based assessment (EBA) tools in community mental health settings. Due to the context and resource differences, there is more EBA use in research settings than community-based settings. This is problematic as EBA use is associated with more accurate diagnostic formulations. Few studies of diagnostic agreement focus specifically on anxiety disorders, which limits our understanding of how anxiety is diagnosed in practice settings and how we might improve it. The current study aims to evaluate diagnostic agreement between clinician diagnoses of anxiety and researcher administered Anxiety Disorders Interview Schedule for DSM-5, Child Version, Child and Parent Forms (ADIS-5-C/P) in a sample of treatment-seeking youth with emotional disorders who participated in a randomized, controlled study examining the effectiveness of Treatment as Usual (TAU), Unified Protocol for Adolescents and TAU plus a measurement and feedback system. As part of the study, an independent evaluator (IE) diagnosed youth using the ADIS-5-C/P after being trained to reliability. IE diagnoses were compared to medical record diagnoses made by community-based clinicians. Agreement between the clinician-generated diagnoses in the youth’s medical record and the IE generated diagnoses was assessed using Cohen’s kappa, and kappa coefficients were interpreted based on McHugh’s (2012) standards. The youth in the current study (N = 103) were participants in one study site from a multi-site effectiveness trial. Youth were on average 14.3 years old (SD = 1.6), mostly cis-gender female (66%, n = 68), heterosexual (57.3%, n = 59), Hispanic/Latine (53.4%, n = 55), and white (53.4%, n = 55). All kappas generated for the four individual anxiety disorder diagnoses indicated “no agreement” (ks < 0.20) except for panic disorder, which indicated “minimal agreement” (k = .32, p < .001). Four additional anxiety disorder diagnoses could not be analyzed due to no diagnoses in one or both settings. To be able to assess diagnostic concordance on a broader level, five diagnostic clusters were created by collapsing individual diagnoses into broader groups: anxiety, depression, trauma and stressor, obsessive compulsive, and behavioral disorders. Kappas computed for all clusters except for the depression cluster were interpreted to have “no agreement” (ks < 0.20). The kappa for the depression cluster (k = 0.26, p < .001) can be interpreted as “minimal agreement”. The anxiety cluster had the second lowest level of agreement (k = .05, p = .095), with the trauma and stressor cluster having the lowest (k = .04, p = .580). The results of this study align with the hypotheses that diagnostic agreement is poor between clinicians and researchers, especially for anxiety disorders. Findings suggest that semi-structured interviews, like the ADIS-5-C/P, can detect anxiety disorder diagnoses that are being missed by clinicians. Future work should seek to identify and address barriers to implementation of EBAs in community mental health clinics.