06 Dec MILESTONE Ratings of Medical Residents Better Able To Distinguish Competence
MedicalResearch.com Interview with:
Dr. Karen Hauer
Karen E. Hauer, MD, PhD
Associate Dean, Competency Assessment and Professional Standards
Professor of Medicine, UCSF
San Francisco, CA 94143
MedicalResearch.com: What is the background for this study?
Response: As part of the Accreditation Council for Graduate Medical Education (ACGME)’s Next Accreditation System, residency programs are now required to rate residents using the Milestones. Evidence of validity of Milestone ratings is needed to show whether this rating system measures meaningful aspects of residents’ practice.
In the field of internal medicine, we compared ratings of residents using the old evaluation form, the pre-2015 Resident Annual Evaluation Summary (RAES), which has a non-developmental rating scale that rates residents from unsatisfactory to superior on a 9-point scale, with developmental Milestone ratings. This was a cross-sectional study of all United States internal medicine residency programs in 2013-14, including 21,284 internal medicine residents. Milestone ratings are submitted by residency program directors working with Clinical Competency Committees. We correlated RAES and Milestone ratings by training year; correlated ratings of Medical Knowledge milestones using the two systems with American Board of Internal Medicine (ABIM) certification examination scores; and examined ratings of unprofessional behavior using the two systems.
MedicalResearch.com: What are the main findings?
Response: The main findings were that ratings using the Milestones showed progressively higher correlations across training years, meaning that they show a developmental progression, whereas ratings using the pre-2015 RAES were high across all years of training. Of the 6,260 residents who attempted the certification examination, those 618 who failed the examination had lower ratings using both rating systems for Medical Knowledge than those who passed. Low Milestone ratings – ‘deficiencies’ on either of the two Medical Knowledge subcompetencies – indicated residents at risk of failing the certification examination. Ratings for Professionalism using the two ratings systems were correlated, but Milestones identified more residents observed to demonstrate unprofessional behavior.
MedicalResearch.com: What should readers take away from your report?
Response: This study shows evidence of validity of the Milestones for assessing residents in several ways. Among all US internal medicine residents, Milestone ratings demonstrated a greater range of scores across training levels than ratings using the RAES form. This finding means that the Milestones seem to better distinguish residents’ development of competence. For Medical Knowledge, both the Milestone and RAES ratings detected residents who failed or did not take the Medical Knowledge certification examination. However, low Milestone ratings picked up residents who failed or did not take the certification exam despite their RAES ratings indicating they were qualified. Therefore, low Milestone ratings create an opportunity for residents and program directors to provide additional support for residents at risk of failing the certification examination.
Milestones ratings detected more residents felt to have unprofessional behavior than the RAES. Nearly all residents with low Professionalism Milestone ratings had at least satisfactory RAES ratings. We believe that the Milestones detected more unprofessional behavior because the 4 Professionalism subcompetency Milestones give program directors more ways to indicate unprofessional behavior than with the RAES, in which there is only one Professionalism item.
MedicalResearch.com: What recommendations do you have for future research as a result of this study?
Response: Milestone ratings are still a new process in internal medicine training. Future research is needed to follow the same residents longitudinally to confirm whether the Milestones capture a developmental trajectory. Studies are needed to examine Milestones ratings at the program level to determine how ratings change as programs gain experience with the Milestones, and whether programs of certain sizes or characteristics (such as community or university-based) use the Milestones differently. Another question is how Milestone ratings during residency correlate with performance of graduates in practice.
MedicalResearch.com: Is there anything else you would like to add?
Response: There will be a growing body of research over the coming years examining validity evidence for Milestone ratings across specialties. The hope is that this process supports all residents to be on track to become competent physicians who provide high quality, safe patient care.
MedicalResearch.com: Thank you for your contribution to the MedicalResearch.com community.
Citation:
Note: Content is Not intended as medical advice. Please consult your health care provider regarding your specific medical condition and questions.
More Medical Research Interviews on MedicalResearch.com
[wysija_form id=”5″]
Last Updated on December 6, 2016 by Marie Benz MD FAAD