The speed and velocity of medical knowledge is increasing,[1] while clinicians’ available discretionary time is decreasing. Traditional learning techniques are no longer adequate. Clinicians, medical schools, residency programs, and hospitals are looking to find and use the most efficient and effective way to acquire, master, and retain medical knowledge
Therefore, it is becoming more and more important to evaluate the effectiveness of the resources being used for learning, teaching, and assessment. To this end, NEJM Knowledge+ is pleased to share the latest research about the effectiveness of the NEJM Knowledge+ adaptive learning platform for improving educational outcomes.
Quick Links
1) Association of Use of the NEJM Knowledge + Product and Performance on the ABIM IM-MOC Exam
Studies
1) Association of Use of the NEJM Knowledge + Product and Performance on the ABIM IM-MOC Exam Read Online
Published In: Teaching and Learning in Medicine
Authors: Michael G. Healy, Bradley M. Gray, Lauren M. Duhigg, Matthew O’Rourke, Leokadia Marchwinski , Melissa Elmore & Roy Phitayakorn
Publication Date: 09/15/2020
Abstract:
Phenomenon: Internal medicine physicians in the United States must pass the American Board of Internal Medicine Maintenance of Certification (ABIM IM-MOC) examination as part of their ABIM IM-MOC requirements. Many of these physicians use an examination product to help them prepare, such as e-Learning products, including the ACP’s MKSAP, UpToDate, and NEJM Knowledge+, yet their effectiveness remains largely unstudied.
Approach: We compared ABIM IM-MOC examination performance among 177 physicians who attempted an ABIM IM-MOC examination between 2014-2017 and completed at least 75% of the NEJM Knowledge+ product prior to the ABIM IM-MOC examination and 177 very similar matched control physicians who did not use NEJM Knowledge+. Our measures of ABIM IM-MOC exam performance for NEJM Knowledge+ users were based on the results of the first attempt immediately following the NEJM Knowledge+ use and for non-users were based on the applicable matched examination performance. The three dichotomous examination performance outcomes measured on the first attempt at the ABIM IM-MOC examination included pass rate, scoring in the upper quartile, and scoring in the lower quartile.
Results: Use of NEJM Knowledge+ was associated with a regression adjusted 10.6% (5.37% to 15.8%) greater likelihood of passing the MOC examination (p < .001), 10.7% (2.61% to 18.7%) greater likelihood of having an examination score in the top quartile (p = .009), and −10.8% (–16.8% to −4.86%) lower likelihood of being in the bottom quartile of the MOC examination (p < .001) as compared to similar physicians who did not use NEJM Knowledge+. Insight: Physicians who used NEJM Knowledge+ had better ABIM IM-MOC exam performance. Further research is needed to determine what aspects of e-Learning products best prepare physicians for MOC examinations.
Figure 1: Passing status on ABIM MOC exam pass rate for NEJM Knowledge+ users compared with non-user controls.
Figure 2: Comparison of the proportion of NEJM Knowledge+ users vs. non-user controls with an examination score in the top quartile of the ABIM MOC exam.
Figure 3: Comparison of the proportion of NEJM Knowledge+ users vs. non-user controls with an examination score in the bottom quartile of the ABIM MOC exam.
Conclusion: We found that physicians who used NEJM Knowledge+ (completed at least 75% of NEJM Knowledge+) had better performance on the ABIM IM-MOC examination than a rigorously matched control group who did not use this product. Further research is needed to determine what product elements best prepare learners for MOC examinations.
Editor’s Highlights:
“While many physicians use an examination product to help them prepare for their MOC examination, the effectiveness of these products remains largely unstudied.”
“We found that NEJM Knowledge+ users were more likely than similar non-users to pass the ABIM IM-MOC examination on the first attempt immediately following NEJM Knowledge+ use and more likely to score in the top quartile, as well as were less likely than similar non-users of being in the bottom quartile.”
2) Design and implementation of an academic enrichment program to improve performance on the internal medicine in-training exam Read Online
Published In: Medical Education Online
Authors: Amr Dokmak, Amr Radwan, Meredith Halpin, Bertrand L. Jaber & Claudia Nader
Publication Date: 11/11/2019
Abstract: The internal medicine In-Training Exam (ITE) is administered at residency training programs to assess medical knowledge. Our internal medicine residency program witnessed a performance decline on the ITE between 2011 and 2014. The goal of this quality improvement project was to improve medical knowledge among residents as measured by an improvement in performance on the ITE, through the design and implementation of an Academic Enrichment Program (AEP). The AEP was designed in 2014–2015, and entailed a multipronged approach, including strengthening and tailoring of the didactic curriculum, establishment of a minimum conference attendance rate, and adoption of the New England Journal of Medicine Knowledge-Plus Internal Medicine Board Review platform. Residents performing below a pre-specified percentile rank cutoff on the previous year’s ITE in any of the 12 content areas were required to complete a pre-specified percentage of the question bank in that specific topic. We examined a total of 164 residents enrolled in our program under the categorical training track. The mean (± SEM) ITE percentile for the 12 content areas increased significantly from calendar years 2011–2014 to 2015–2018, reflecting implementation of the AEP (p < 0.001). In brief, compared to the AEP-unexposed graduating classes of residents, the AEP-exposed graduating classes of residents displayed a significant improvement in the mean ITE percentile rank. This quality improvement project was carried out at a single institution. The implementation of a structured academic enrichment program significantly improves performance on the ITE.
Figure 1: Performance on the internal medicine in-training exam over the course of the implementation of the academic enrichment program. The data represent the mean percentile for all 12 content areas over 8 years.
Figure 2: Three-year rolling pass rate for first-time takers of the ABIM internal medicine certification exam pre- and post- implementation of the academic enrichment program.
Conclusion:
The implementation of an individualized, structured, and tailored self-study plan for medical residents in the form of an AEP significantly improves performance on the ITE and contributes to a higher ABIM certification exam pass rate for first-time takers. It is our hope that this approach can help inform training programs facing similar challenges and provide a platform to develop creative solutions.
Editor’s Highlights:
“Furthermore, our multipronged approach, specifically the adoption of the NEJM Knowledge Plus Internal Medicine Board Review web-based platform, provided a sophisticated and structured delivery of medical knowledge based on artificial intelligence algorithms that can be easily monitored by the chief residents, making the intervention applicable to other residency programs with minimal time commitment from faculty.”
“Knowledge Plus Internal Medicine Board Review web-based platform, provided a sophisticated and structured delivery of medical knowledge based on artificial intelligence algorithms that can be easily monitored by the chief residents, making the intervention applicable to other residency programs with minimal time commitment from faculty.”
“Our experience resulted in a significant improvement in ITE scores in the group of residents exposed to the AEP as compared to those who were not.”
“…residents displayed a significant improvement in their mean ITE percentile rank,”
“In addition, the adoption of the NEJM Knowledge-Plus Internal Medicine Board Review platform enhanced the didactic curriculum,”
3) The use of the NEJM knowledge + online platform to supplement traditional pulmonary didactic: a resident-led educational quality improvement project at a community hospital IM GME program Read Online
Published In: Journal of Community Hospital Internal Medicine Perspectives
Authors: Alan Kaell, Jay Sangwan, Varvara Boryushkina & Greg Haggerty
Publication Date: 06/21/2021
Abstract:
Introduction: Many internal medicine residents struggle to prepare for both the ITE and board test. Most existing resources are simply test question banks that are not linked to existing supporting literature from which they can study. Additionally, program directors are unable to track how much time residents are spending or performing on test preparation. We looked to evaluate the benefit of using this online platform to augment our pulmonary didactics and track time and performance on the pulmonary module and ITE pulmonary section.
Methods: During the month-long live didactic sessions, residents had free access to the pulmonology NEJM K+ platform. A platform-generated post-test was administered with new questions covering the same key elements, including the level of confidence metametric. An anonymous feedback survey was collected to assess the residents’ feelings regarding using the NEJM Knowledge+ platform as compared to other prep resources.
Results: 44 of 52 residents completed the pre-test. 51/52 completed the month-long didactic sessions and the post-test. Residents’ score improvement from % correct pre-test (M = 46.90, SD = 15.31) to % correct post-test (M = 76.29, SD = 18.49) correlated with levels of mastery (t = 9.60, df = 41, p < .001). The % passing improved from 1/44 (2.3%) pre-test to 35/51 (68.6%) post-test, also correlating with levels of mastery. Accurate confidence correlated with improvement from pre to post test score (r = −51, p = .001). Survey feedback was favorable.
Figure 1. Bar graph describing the post-test performance (proportion of residents who passed the post-test with ≥21/30 questions correct) vs. percent range of NEJM Knowledge+ module mastery
Note: The top of each mastery range bar indicates the # of residents who passed/# of residents in each categorical range of mastery. Of 44 who took the pretest, only 1 scored ≥21/30 Q correct. 51/52 took the post-test and 30 of 51 passed the post-test.
Conclusion: The use of the NEJM Knowledge+ platform correlates with improvements in both residents’ Medical Knowledge (MK) domain and metaknowledge metrics. The MK, assessed by comparison of the pretest and posttest performance and passing rate, correlates with the individual resident’s mastery of all 167 key points during their month of traditional didactic session. Since all residents participated, as assessed by attendance, in the traditional didactic sessions to the same degree, this correlation implies that the comparative absolute and relative improvement in performance on the post-test and metametric measures is likely attributable to the degree of mastery itself achieved.
Resident feedback revealed that many felt the platform’s functionality and content was good and would recommend it to colleagues. Many also felt that it was at least as good as any test prep program to which they had previously accessed. Whether such a tool will prove useful to them as a lifelong supplement to the domain of Practice Based Learning (PBL) and how it affects the Patient Care (PC) domain remains to be explored.
Editor’s Highlights:
“The levels of mastery achieved with the tool correlated with both improvements from pre to post-test and improvement in the passing rate for the post-test (Figure 1). The residents’ improvement in their confidence (metaknowledge metric) in answers to the questions from pre to post-test correlated with their change in pre to post-test score (r = −51,p = .001) and passing rate.”
4) An Exploratory Study of a Novel Adaptive e-Learning Board Review Product Helping Candidates Prepare for Certification Examinations Read Online
Published In: MedEdPublish
Authors: Michael Healy Emil Petrusa, Carl Gustaf Axelsson, Praelada Wongsirimeteekul, Ole-Petter Hamnvik, Matthew O’Rourke, Roger Feinstein, Ruth Steeves, Roy Phitayakorn
Publication Date: 06/08/2018
Abstract:
Objectives: Online board-review products are widely used to prepare for high-stakes examinations; however, it is unclear if these products are truly helpful and which features are most useful. This study examined the user ratings of an e-learning examination-preparation product and its impact on first-time American Board of Internal Medicine Certifying Examination (ABIM-CE) test takers.
Methods: 4,400 users were anonymously surveyed between 2015 and 2017 to determine their level of satisfaction with the NEJM Knowledge+ Internal Medicine Board Review (NEJM Knowledge+) product and their board-exam passage rate.
Results: The responding users’ (n=836) average ratings of NEJM Knowledge+’s content, relevance, components, and features were high. Similarly, a subgroup of responding users from the United States who primarily used NEJM Knowledge+ to prepare for an initial ABIM-CE between 2014 and 2016 (n=153) rated the content, relevance, components, and features high, with practice exams, adaptive delivery of questions, and reports on progress, performance, and metacognition as the most valuable features. Importantly, this subgroup of users passed the ABIM-CE on their first attempt at a significantly higher rate than the national average (95% vs 89%, z=2.6397, p=0.0083).
Conclusion: This exploratory study highlighted that an e-learning examination-preparation product featuring adaptive delivery of questions, effective feedback, and analysis of overall metacognition can potentially be an effective tool for first-time test takers preparing for initial certifying examinations. Thus, the ability for residents, fellows, and physicians to utilize products such as NEJM Knowledge+ as learning tools in this high-stakes-examination era can potentially be of high educational value. Finally, further research should be undertaken regarding e-learning examination-preparation products in other medical and allied-health specialties.
Editor’s Highlights:
“First-time test takers using the NEJM Knowledge + e-learning examination-preparation product, which can be a contributing factor, passed the ABIM-CE at a significantly higher rate than the national average.”
“This study highlights that the NEJM Knowledge+ product can potentially be useful for learners and that specific features, including adaptive delivery of questions, practice exams, and reports on progress, performance, and metacognition, could aid in enhancing learning efficiency and knowledge retention.”
“E-learning examination-preparation products that have features such as practice exams, adaptive delivery of questions, and reports on progress, performance, and metacognition can potentially be effective in the preparation for initial certifying examinations.”
Summary
These four studies, conducted independently from each other to demonstrate the efficacy of adaptive learning combined with gold-standard content, provide evidence that using NEJM Knowledge+ improves pass rates and raises scores on board exams as well as improves ITE (In-Training Exam) scores (which are an indicator of future performance on certification exams [2]). Additionally, NEJM Knowledge+ can be useful for learners – and that specific features, including adaptive delivery of questions, practice exams, and reports on progress, performance, and metacognition, could aid in enhancing learning efficiency and knowledge retention.
We are looking forward to future published studies that continue the important work of evaluating the efficacy of various learning methodologies — especially adaptive learning, which is a term applied liberally at times. But what is adaptive learning and what qualifies for the term “adaptive learning?”
What makes NEJM Knowledge+ different
Instead of trying to fit learners into a prescribed mold, we are using an authentic adaptive learning platform that adapts to our users’ knowledge, learning patterns, goals, and needs. In this way, NEJM Knowledge+ does more than just help physicians with board preparation. It delivers a powerful, engaging, personalized learning experience, which promotes the lifelong development of knowledge that is essential in the ever-changing world of medicine.
Adaptive learning has been proven to result in better retention, deeper knowledge, and higher test scores. How does it help? And how do you tell the difference between experiences that provide true adaptive learning and those that just claim to be? Watch the video
Clinicians: If you are interested in learning how NEJM Knowledge+ might help you in your quest to deliver the best possible patient care by acquiring, retaining, and refreshing clinical knowledge, learn more about our solutions for lifelong learning and board review.
Educators: If you are interested in learning how NEJM Knowledge+ might enhance your training program with gold-standard content and proven assessment tools, learn more about our solutions for educators.