Individuals on the path to becoming a physician in the United States spend a great deal of time in educational settings and must pass many standardized high-stakes examinations. Do adaptive learning board review products like NEJM Knowledge+ help?

An Exploratory Study of an Adaptive Learning Board Review Product

After residency or fellowship training in internal medicine, physicians must pass the American Board of Internal Medicine Certifying Examination (ABIM-CE) to become board certified. Although most physicians use question banks to study, it had been unclear if adaptive learning board review products are truly helpful and which features are most useful. We set out to examine the user ratings of NEJM Knowledge+ and its impact on first-time ABIM-CE test takers.

The results of this study were published in MedEdPublish in a paper titled “An Exploratory Study of a Novel Adaptive e-Learning Board Review Product Helping Candidates Prepare for Certification Examinations” authored by Michael Healy, Emil Petrusa, Carl Gustaf Axelsson, Praelada Wongsirimeteekul, Ole-Petter Hamnvik, Matthew O’Rourke, Roger Feinstein, Ruth Steeves, and me (Roy Phitayakorn).

We anonymously surveyed 4,400 NEJM Knowledge+ users between 2015 and 2017 to determine their level of satisfaction with NEJM Knowledge+ and their board-exam passage rate.

Survey Results

The responding users’ (n=836) average ratings of the content, relevance, components, and features of NEJM Knowledge+ were high.

Similarly, a subgroup of responding users from the United States who primarily used NEJM Knowledge+ to prepare for an initial ABIM-CE between 2014 and 2016 (n=153) felt that the practice exams, adaptive delivery of questions, and reports on progress, performance, and metacognition were the most valuable features.

Significant Pass Rate for a U.S. Respondents Using Adaptive Learning Board Review

Importantly, this subgroup of users passed the ABIM-CE on their first attempt at a significantly higher rate than the national average (95% vs 89%, z=2.6397, p=0.0083).

The results of this study suggest that an examination-preparation product that features adaptive delivery of questions, offers detailed feedback, and analyzes overall metacognition is likely an effective tool for first-time test takers preparing for initial certifying examinations.

Take Home Messages

  • First-time test takers using the NEJM Knowledge + e-learning examination-preparation product, which can be a contributing factor, passed the ABIM-CE at a significantly higher rate than the national average.
  • E-learning examination-preparation products that have features such as practice exams, adaptive delivery of questions, and reports on progress, performance, and metacognition can potentially be effective in the preparation for initial certifying examinations.

Dr. Roy Phitayakorn is a general and endocrine surgeon and the Director of Surgical Education Research at Massachusetts General Hospital and an Assistant Professor of Surgery at Harvard Medical School. He is also the Education R&D consultant to NEJM Group.