Development of validity evidences of a performance instrument on teaching competences: EDOE

Abstract

Introduction: The evaluation of teaching competences is related to the improvement of the educational activity which is oriented towards personal and formative development. There are no high-fidelity evaluations in our specific setting. The aim was to obtain sources of evidence of validity for the Objective & Structured Teaching Examination (OSTE) to evaluate the teaching competencies. 

Method: A quantitative study was carried out in three phases: 1) Planning phase: analysis of the competence model, validation by judges, validation of content by student instructors, and workshop with experts for the creation of the stations. 2) Pilot phase: to obtain the degree of concordance among evaluators, validation and internal structure of the stations (discriminant analysis, Cronbach’s alpha, means and standard deviation), 3) Final phase: Application to a non-random sample formed by 20 professors of the Department of Biomedical Informatics at the Faculty of Medicine of UNAM.

Results: An item discrimination analysis of eight stations was carried out. We observed that two stations have five items that discriminate .05, seven had acceptable factorial loads and explained 88.35% of the variance with a Cronbach’s alpha of .68. The average of teaching performance in the OSTE was 73.23 with a standard deviation of 8.4. The best-evaluated station was learning strategies (85 ± 16.98), and the lowest was discussion guidance (67.48 ± 15.62). Confirmatory evidence was obtained with similar examinations used in the Faculty of Medicine, the objectives set for the creation of this instrument were met, an OSTE pre-evaluation mean was detected, and the teachers got feedback on their performance for training purposes. 

Discussion and Conclusions: The OSTE assessment is proposed as an instrument with sufficient evidence of validity that can be used in the evaluation of the degree of teaching competencies. We aim to create teaching evaluation instruments that follow good practices in the development of instruments based on the chain of evidence of validity.