On June 13, 2017, a study to determine the relevance of internal medicine board questions was published by JAMA: The Journal of the American Medical Association. The study, entitled “Comparison of Content on the American Board of Internal Medicine Maintenance of Certification Examination With Conditions Seen in Practice by General Internists,” was conducted with financial and material support from the ABIM and was partially in response to physicians’ concerns about ABIM’s Maintenance of Certification (MOC) process. As detailed in our recent blog post about changes to the MOC process, physicians were looking for more relevant exam content that would portray a more reliable standard of physician knowledge, as well as a process that would enable them to keep the focus on their practice rather than on exams. In 2015, ABIM surveyed physicians to better understand the needs of internists and subspecialists and had already begun revising the internal medicine board questions on the MOC exam when this study took place.
Study Purpose: ABIM’s Response to Physicians’ Concerns about Relevance
Relevance to general practice has been a key point of consternation for certain physicians taking the exam. Some argue that the Internal Medicine Board Exam should cover all the possible cases a physician will face in practice, including uncommon diseases, conditions, and treatments, whereas others argue that it should strive to focus on only physicians’ most frequently encountered and highly relevant cases.
Internists participate in the MOC process to demonstrate that they are keeping current in medicine. How much do internal medicine board questions actually reflect the conditions that internists encounter on a daily basis? These questions have been on the minds of some physicians, and the intent of the recent JAMA study was to find the answers.
Study Design: How Information was Collected and Measured
The study compared 3,461 internal medicine board questions with conditions seen in general internal medicine, using the following sources of information:
- 3,600 questions from the 2010–2013 ABIM MOC examinations (180 questions in each examination form)
- These internal medicine board questions covered over 186 categories of medical conditions most often seen in internal medicine practices.
- Of the total exam questions, 96.1% internal medicine board questions could be mapped to study conditions to create a typical exam blueprint on which to base the comparison.
- 2010-2013 National Ambulatory Medical Care Surveys, which represented 13,832 office visits for patients in internal medicine
- 2010 National Hospital Discharge Survey, which covered 108,472 hospital stays for patients in internal medicine
Gathering information from both routine office visits and hospital visits provided authors with the opportunity to study not only common and chronic conditions, but acute presentations of those conditions and medical emergencies.
Study Conclusions: Results and Relevance of Internal Medicine Board Questions
Published study results revealed that among the sampled questions, 69% could be considered relevant, or “concordant with conditions seen in general internal medicine practices.” Areas of discordance, where they appeared, seemed to be due to discrepancies of frequency; i.e., although the condition was present in the surveys, it occurred proportionally less often in real life than it did on the exam.
Here is a breakdown of the results:
- Total concordance: 2,389 internal medicine board questions were categorized as concordant.
- Concordance between questions and office visits only: 2,010 internal medicine board questions involving 145 conditions were categorized as concordant.
- Concordance between questions and hospital stays only: 1,456 internal medicine board questions involving 122 conditions were categorized as concordant.
What’s Next?
With the JAMA study, ABIM aimed to establish a baseline to measure relevance of internal medicine board questions to real-life internal medicine, and gain an understanding of how to improve the exam to better serve physicians (and, by extension, the study and practice of internal medicine and the patients who benefit from medical care). “Physicians asked ABIM to look for ways to make MOC more reflective of clinical practice, so we are studying our program to determine what works well and where we can improve,” study coauthor and ABIM board member Marianne M. Green, MD, said.“ABIM is using real-world data and feedback from a large group of internists to align exam content with conditions seen frequently and the importance of these conditions for patient mortality and morbidity.” Other study authors included Bradley Gray, PhD; Jonathan Vandergrift, MS; and Rebecca S. Lipner, PhD.
ABIM’s exam review and the JAMA study results will both play a role in helping to evolve the ABIM MOC process. In the editorial accompanying the study, “Physician Certification and Recertification: The Role of Empirical Evidence,” Adam B. Schwartz, MD, and J. Sanford Schwartz, MD, noted, “Future studies by independent investigators are needed to replicate and extend the analyses conducted by Gray and colleagues; to address the study limitations and other issues discussed by the authors; and to evaluate other measures of clinical validity, such as physician judgment and decision making, including understanding and application of epidemiology, statistical reasoning, and disease prevention.” Pointing out that all of the study authors are affiliated with or employed by ABIM, the authors of this editorial emphasized the importance of additional independent research to replicate and confirm the results.
Future studies may also pinpoint ways for ABIM to increase the relevance and effectiveness of the MOC exam and process. This study coded 31% of internal medicine board questions as discordant, and although this study did not take recent exam revisions into account, editorialists consider it a starting point for more improvement: “The ABIM must strive for continuous improvement and is most relevant and useful when it asks for help and listens to its members, a process that it has been involved with for the past several years. Open, ongoing evaluation of the recertification examination will be essential, along with frank and productive discussion about how to ensure the continuous improvement, excellence, and relevance of the recertification process.” In this way, ABIM must apply the standards of evidence-based medicine to their own programs to improve their MOC process and present an exam with more relevant internal medicine board questions.
It seems that the true relevance of MOC itself is not being addressed here. The relevance of questions, therefore, is only of secondary importance. As far as I know, a secure test has never been demonstrated to improve quality of patient care or reduce errors. The assertion on this page that “Internists participate in the MOC process to demonstrate that they are keeping current in medicine” is highly questionable. My own survey of nearly 700 physicians a few years ago clearly demonstrated that nearly 80% of respondents participated in MOC because their employment or their insurance and/or hospital credentialing depended on that. Less than 14% of respondents did it for their own professional development.
The true scientific way to separate fact from fiction is to compare MOC participants and nonparticipants over a 10+ year time, with respect to value for patients. Meanwhile, the NEJM should call for an end to linking insurance/hospital credentialing to MOC.
“The study …was conducted with financial and material support from the ABIM”
Ummmmmmmmmm.
If I read the same sentence in a study on a drug, I’d just skip to the next article.
Which I did.
What other result can be forthcoming?
The only useful and believable result from such a study is to find the results were unfriendly to the funders.
Yes! Candidates for Certification might well consider contacting me for Mentoring/Tutoring.
Respectfully submittted…Doctor Josh