Validity evidence in the admission process to a medical school in Mexico

Abstract

Introduction: Medical school admission processes need to demonstrate the validity of their scores’ interpretations, since they are high-stakes assessments<<.

Objective: To apply an integrative model using Messick’s and Kane’s frameworks to obtain validity evidence for a medical school admission process.

Method: An analytical observational study was performed at the Autonomous University of San Luis Potosí Faculty of Medicine in Mexico, with the 2013 and 2014 cohorts. The admission process included a psychometric test (PT), a general knowledge test (KT), and EXANI-II. The model was applied in two steps: 1) uses and interpretations argument, and 2) validity argument, which has three stages: instrumental validity, interpretation and decision verification, and action usefulness. Hypotheses were established at each stage using Kane’s inferences, which were supported using Messick’s sources of evidence.

Results: The results of 1,373 applicants from 2013 and 1,554 from 2014 were analyzed, as well as the academic performance of the 145 students admitted in each cohort. A general intelligence factor (g) was identified as influencing the results of admission process. KT explains 15% of the variance in the first-year grades (2013), PT and EXANI-II have no impact on first-year’s academic performance (2013 and 2014). There was a correlation among some subcomponents that measure a similar construct. There was a difference in Biology results (KT) between men and women. Students’ degree of satisfaction with the admission process is adequate.

Discussion and conclusions: The sources of validity evidence provide relevant information to support and improve the admission process, more data and an interdisciplinary team to retrieve more sources of quality evidence are needed. It is recommended to use well-defined frameworks in these processes and plan data collection and retrieval using these models.