Adjustment from a face-to-face OSCE to an on-line modality in a high stakes examination

Abstract

Introduction: In view of the need to remotely assess clinical competencies during the COVID-19 pandemic, an online objective structured clinical examination (OSCE) was developed and applied in 2021 to applicants at the end of their medical degree at the National Autonomous University of Mexico, Faculty of Medicine and of incorporated institutions.
Objective: To assess the OSCE online 2021 psychometric quality and compare it to a face-to-face OSCE modality in 2020 in a high stakes examination.
Method: The OSCE was adjusted from a face-to-face to an online format. The clinical cases and the rubrics used for the online modality were previously designed according to the face-to-face OSCE. In each of the clinical cases, the ability of the physical examination was adjusted and the medical note was replaced to a verbal clinical summary. For this new modality we used the zoom platform; therefore, training courses were given to the assessors and patients to standardize their knowledge and performance on this platform. For the applicants we designed audiovisual resources explaining the process of the examination. We analyzed the applied procedure based on the design, implementation, and grade; also, we used the psychometric analysis to provide the validity evidence related to the internal structure of the test.
Results: Both formats of the OSCE were composed of six-20 minutes stations. We examined the 99 items that constitute the total of the six evaluated competencies with a <0.001 discrimination between the high and the low scores. We identified six factors that explain the 60.7% of the variance in the face-to-face OSCE and six factors with five more indicators in the online format that explained the 56.6% of the variance. We calculated a Chronbach Alfa of 0.96 for the face-to-face format and a 0.93 for the online OSCE.
Conclusions: Both OSCE formats showed psychometric quality evidence based on the content validity, internal structure of the test, and the response process, these characteristics strengthened both tests. This research showed that the online format is a feasible alternative for the evaluation of the practical phase in the professional examination.