Analysis of an objective structured clinical examination in dentistry using generalisability theory

Análisis de un examen clínico objetivo estructurado en odontología desde la teoría de la generalizabilidad
Olivia Espinosa-Vázqueza, Adrián Martínez-Gonzálezb, Melchor Sánchez-Mendiolab, Iwin Leenenc

a Facultad de Odontología UNAM, Ciudad de México, México
b Coordinación de Desarrollo Educativo e Innovación Curricular UNAM, Ciudad de México, México
c Instituto Nacional para la Evaluación de la Educación, Ciudad de México, México

Recibido 18 abril 2016, Aceptado 06 septiembre 2016

Palabras Clave

México, Competencia clínica, Teoría de la generalizabilidad, Examen clínico objetivo estructurado (ECOE), Odontología

Keywords

Mexico, Clinical competence, Generalisability theory, Objective structured clinical examinations (OSCE), Dentistry


Abstract

Introduction: Various studies have examined Objective Structured Clinical Examinations (OSCEs) in Dentistry in order to accumulate validity evidence for their use as an assessment tool of clinical competence in students. In this article, a newly designed OSCE in Dentistry (OSCE-D) is introduced and discussion is presented on the results of an analysis from the perspective of generalisability theory using data obtained from an application of the examination.

Method: An observational and cross-sectional study was conducted in the Faculty of Dentistry at UNAM. One hundred and twenty pre-graduate students participated in an OSCE that consisted of 18 stations, with a duration of 6 min each, in the context of a fourth-grade Paediatric Dentistry course. An analysis based on generalisability theory, with raters and stations being considered as facets, identified the main sources of variability in the data.

Results: The overall mean (and standard deviation) of the OSCE score, across participants and stations, was 44% (7%), with the station means varying between 23% and 63%. The generalisability study showed that the facet of the raters explained a significant portion (13%) of the variance in the station results, which was more than the clinical competence of the participants (6%). The decision study produced a generalisability index of 0.63 and a dependability index of 0.55.

Conclusions: In view of the rather low reliability indices from the decision study, it is important to make a further analysis of the OSCE-D so as to minimise the effect of sources that introduce construct-irrelevant variance into the results. In particular, an adjustment of the stations may be required, as well as a better standardising in the use of evaluation criteria by the raters.