Overall ABIM pass rates fell from 90% passing in 2009 to just 78% passing in 2013 for first-time test takers of the Maintenance of Certification (MOC) internal medicine board exam, according to statistics released annually by the American Board of Internal Medicine (ABIM).
The hard numbers from ABIM show that of 5,772 internists attempting the recertification exam in 2013, some 1,270 did not pass on their first attempt, which is nearly triple the number failing on first attempt in 2009.
Year | First-Time Takers | Pass Rate | Passed | Failed |
---|---|---|---|---|
2009 | 4,256 | 90% | 3,830 | 426 |
2010 | 4,961 | 88% | 4,366 | 595 |
2011 | 5,333 | 87% | 4,640 | 693 |
2012 | 5,746 | 84% | 4,827 | 919 |
2013 | 5,772 | 78% | 4,502 | 1,270 |
In their response letter to the anti-MOC petitioners in July 2014, ABIM attempted to clear up the confusion about first-attempt pass rates versus ultimate pass rates (which include retakes), saying first-attempt pass rates have been declining over the past 5 years, but ultimate pass rates have remained fairly constant at 95% to 98%.
Many Theories for Why ABIM Pass Rates Are Falling
As might be expected, the overall decline in ABIM pass rates has engendered much debate, both within medical teaching and learning communities and across the medical blogosphere. What is surprising, perhaps, is the diversity of opinions about what might be driving the decline in ABIM pass rates.
Opponents of new ABIM Maintenance of Certification (MOC) requirements claim that the lower pass rates are proof that the ABIM is surreptitiously making the exam more difficult in order to bring in additional revenue from repeat attempts to pass the exam — despite explicit ABIM claims to the contrary.
Beyond the usual anti-MOC rhetoric, however, a number of additional theories (some might say straw men) have emerged to explain the alarming downtrend in ABIM pass rates. To summarize briefly:
- Technology — namely, the ready ability to search for and find just about any piece of information within a few seconds — has diminished the willingness of physicians (especially younger generations) to study, memorize, and carry around certain bases of medical knowledge inside their brains.
- Work compression — the trend of forcing physicians to accomplish more in less time — is undermining both initial and lifelong learning by providing attending physicians too little time to teach properly and leaving many physicians too fatigued and stressed to focus on consistently reinforcing and updating their medical knowledge.
- Big data — the rate of increase in the sheer quantity of medical information and evidence is outpacing both human capacities for acquiring and retaining knowledge and medical boards’ abilities to evolve appropriate testing methods (while the boards still test knowledge of facts, they should be testing abilities to rapidly find, synthesize and apply facts for successful outcomes).
- Electronic record keeping — proliferating use of and reliance on Health Information Technology (HIT) care-record devices in clinical settings is weakening physicians’ development and ongoing refinement of their cognitive skills in clinical settings.
- Expanding pool of MOC diplomates — greater numbers of physicians are now feeling compelled to participate in MOC (even if they had been grandfathered with board certification originally). Reluctance to participate, long time spans between formal medical training and re-examination, outmoded studying and test-taking skills — or some combination of all three — may also be contributing to declining first-time pass rates.
A 2013 post on The Health Care Blog, by John Schumann, MD — Why Are So Many Younger Doctors Failing Their Boards? — has become ground zero for a heated intergenerational debate, with Schumann suggesting that the study habits of millennials are inferior to those of their boomer and gen X colleagues. Schumann wrote:
One concern that has a ring of truth to it is that young doctors have become great “looker uppers,” and have lost the sense of what it’s like to actually read and study medicine. While doctors enter the profession with a commitment to lifelong learning, some of us fear that the young folk only go far enough to commit to lifelong googling.
Millennials bit right back, however, with Teresa Chan, MD, Assistant Professor, Division of Emergency Medicine, McMaster University writing a post for Boringem.org titled Not Dumber, but Different? Counterpoint from a Millennial. Dr. Chan wrote:
I am a Millennial who will be starting as an attending this year. I still passed my exams and still had to deal with learning the “old fashioned way.” In the end, memorizing factoids and endless lists was the way to beat the exam, but these skills do not always translate well into my daily practice as a physician. I am certainly studying to add to my existing knowledge about cytochrome P450 or the oxidative phosphorylation chain will help someday, but rarely does that problem present itself in my emergency medicine practice. I think the big problem underlying the current examinations systems in most specialties and jurisdictions is that they ask questions that often have not changed with the times. Most important, they value the lower levels of learning (e.g. Bloom’s Taxonomy level = ‘Remember’ and perhaps ‘Apply’) rather than critical reasoning and problem solving.
Meanwhile, over on The Health Care Blog and in direct response to Schumann, David Shaywitz, MD, wrote Are Young Doctors Failing Their Boards — Or Are We Failing Them?:
Let me suggest a third possibility – perhaps today’s doctors are providing better care to patients than their predecessors were a generation ago. Maybe today’s doctors have figured out that, in our information age, your ability to regurgitate information is less important than your ability to access data and intelligently process it. Maybe what makes you a truly effective doctor isn’t your ability to assert dominance by the sheer number of facts you’ve amassed, but rather on how well you are able to lead a care team, and ensure each patient receives the best care possible. In other words, what if the problem isn’t the doctors, who are appropriately adapting, but rather the tests (and the medical establishment), which may not be?
Carving out some middle ground in the debate, a commenter on the Shaywitz rebuttal, identifying herself as Dr. Leora Horwitz, wrote:
Having just (phew!) passed my first recert exam, I read this post and the related one with interest. I think both viewpoints are important. I agree with Dr. Shaywitz that, for complicated hospital inpatients, where you have a lot of time to spend per patient, it is more important that physicians have the skills to look up the latest treatments, differential diagnoses, etc than necessarily trying to memorize a whole ton of facts that change on a regular basis. On the other hand, though, I do most of my clinical work in an outpatient primary care setting. At 20 minutes per patient (generously), I really need a pretty comprehensive and accurate fund of knowledge that I can access without doing a lot of real-time looking up…[At] a minimum I need to know enough to know what to look up. So I think it’s reasonable to test me on the fund of knowledge I should be expected to use on a daily basis and to test my general pattern recognition for common or deadly complaints.
While agreeing with Dr. Shaywitz, another commenter identifying himself as William Hersh, MD, said:
This development of falling board scores is a concern, but before we ascribe blame to health IT, Gen Y/millennial laziness or anything else, can we see some data supporting the assignment of blame? Let’s not let this finding be a Rorschach test for everything each of us does not like about medicine, health care, or society.
ABIM Pass Rates for Initial Certifiers versus Recertifiers
An important point to note is that among physicians seeking initial certification, ABIM pass rates for first-time initial certification exam takers have actually remained fairly stable (and even improved slightly over the past couple of years).
Year | First-Time Takers | Pass Rate | Passed | Failed |
---|---|---|---|---|
2009 | 7,226 | 88% | 6,359 | 867 |
2010 | 7,335 | 87% | 6,381 | 954 |
2011 | 7,337 | 84% | 6,163 | 1,174 |
2012 | 7,303 | 85% | 6,208 | 1,095 |
2013 | 7,482 | 86% | 6,435 | 1,047 |
Millennials may be tempted to take this as a triumph over older generations whose ABIM pass rates for recertification have continued to fall. Higher first-attempt pass rates for initial certification may be a function of residency programs taking more direct actions to improve their first-time board pass rates; they know their graduates’ exam results affect their program’s status with the Accreditation Council for Graduate Medical Education (ACGME) and are reported publicly by ABIM — and, therefore, may be read as general indicators of residency-program quality.
In 2012, researchers at the Cleveland Clinic actually developed a nomogram to predict a resident’s probability of passing the American Board of Internal Medicine’s examination. Interestingly, the strongest predictors for passing the ABIM exam turn out to be scores on other standard exams for residents. This would seem to lend strength to notions that students with the best study habits (and, perhaps, the strongest test-taking skills) are more likely to pass on their first attempts. However, the Cleveland researchers do find that making more time available for dedicated study is also a predictor of success:
An interesting finding is the influence that the number of calls during the [past] 6 months of residency has upon the ABIM probability of success. Although small in comparison to the ITE score relevance, it suggests that easier rotations in months at the end of the residency give residents an additional advantage and increases their chance of passing the board. This could be effectively used by the program directors when scheduling rotations for the residents.
Absent the presence of predictive nomagrams and residency program directors motivated to ensure they pass on their first attempts, physicians practicing outside of active medical teaching environments are on their own when it comes to figuring out precisely what they need to learn and what the best study methods might be for ensuring they pass MOC exams on the first try.
Ultimate Goal: Better Outcomes for Patients
A close reading of various points and counterpoints in the numerous debates surrounding the downtrends in first-time ABIM pass rates actually reveals more underlying agreement than disagreement. Most commentators would appear to agree that physicians do need a certain base of specific and up-to-date medical knowledge — readily recalled under high cognitive stress — so they can at least recognize when their knowledge is insufficient in certain situations, meaning they need to reach out for advice from colleagues or to look up information before making critical patient care decisions.
Some doctors, regardless of age or generation, need to see greater value in learning, retaining, and being able to readily recall current medical knowledge. Others need to become better at knowing and admitting when they don’t know something as well as, perhaps, they should. All of this brings to mind a conversation we reported last spring with Dr. Ulrik Christensen, whose lifetime of work and research into preventing medical errors has evolved into the adaptive learning technology that underpins and powers the new NEJM Knowledge+ medical learning platform. Dr. Christensen said:
We observed through … [medical] simulators that a physician’s inability to recall trivial information would place them under excessive cognitive workload, leading to errors where, for example, they might fail to ask a colleague for assistance. While we could try to train physicians to become better at asking for help, we also became fascinated with the question of “Why are they under high cognitive workload at all?” We started to look at how we might solve the problem from the other end — by improving learning and making it easier for physicians to recall basic medical knowledge so they can keep more cognitive capacity available for addressing the really difficult problems. From there, we started doing research into various tutoring systems. By taking study tools rooted exclusively in repetition and adding even simple intelligence, we discovered that we could make them much more effective. Essentially, we found a way to solve a very fundamental problem that affects learning in every field, including complex ones such as medicine.
While Dr. William Hersh makes a strong point about not allowing the alarming downtrend in ABIM pass rates to become “a Rorschach test for everything each of us does not like about medicine, health care, or society,” the debate inspired by the trend at least seems to be provoking some deep thinking about lifelong medical learning in general and how tools for learning and knowledge assessment must continue to evolve.
The secure exam will remain in place, ABIM confirmed in their July 2014 letter, and will continue to “evolve with time as indeed it has” since 1936 “when ABIM was created by the medical community … as a standard setting organization.”
Whatever changes we make to the exam in the future related to content, format, delivery vehicle, feedback, etc. will need to support the use of the exam as a summative assessment tool that signifies competence in the disciplines of internal medicine. The community has requested a more modular, practice-relevant approach to summative assessment and we are convening a committee to explore how to move those ideas forward.
ABIM suggests that this committee will discuss the “development of the examination, including generation of the exam blueprint and its level of granularity.”
In addition, ABIM has a new initiative called “Assessment 2020,” which aims…
To help us improve and move forward…[W]e seek to engage physicians, the public and other important stakeholders in helping us think through the future of assessment for ABIM Certification and Maintenance of Certification.
We are very interested to hear your thoughts and ideas related to ABIM pass rates. Please share your comments and experience with us.
relevance to what an average md will do is important, and this, clearly described as the content of the exam will tell the person about to take the exam that factual, average knowledge in working memory is essential to pass. The last article or set of related articles about something that can be considered not average and can be found in an smartphone should not be the subject of the questions. If someone is really able to design questions or cases certified designed to measure decision making and team leading capabilities (tongue in cheek) could be used as questions with value related to its true practical use.
Another possibility is that the questions on the secure exam may have incorrect answers. Several months ago, I served as a beta tester for an ABIM knowledge self-assessment module. Many of the questions and their “correct” answers were highly biased in the direction of supporting the current “Choosing Wisely” campaign, which leans heavily toward using the minimum possible diagnostic testing and therapeutic measures. The first time I found a wrong “correct” answer on the answer key, I appealed it to ABIM, and they finally admitted I was right and they agreed to remove the question. Then, I submitted several more equally incorrect answers, and ABIM refused to address my valid scientific criticisms, despite the fact that my objections were documented by recent literature citations, and vetted by top academicians. ABIM refused my offer to volunteer to audit their secure test questions for accuracy. Are they afraid of what I might find?
I propose that, each year, an independent auditing firm conduct a random selection of ABIM’s secure test questions, and that 3% of their questions be released to the public for scrutiny, along with the “correct” answers.
I believe ABIM may be pushing the “Choosing Wisely” ideology too hard, to the point where well-informed internists correctly find ABIM’s minimalist approach to patient care insufficient, and fail the exam – until they purchase the politically-correct study guide and internalize the new party line on how to practice medicine.
The data should be able to give you some more of the answers, despite the multitude of opinions on this issue. You need to do some subgroup analyses. The last hypothesis – the “Expanding Pool” – is testable by breaking down pass rates for first-time MOC takers by year of training or years since initial ABIM certification – i.e., is there a difference in the pass rate among first-time MOC takers 10 years out of training, versus those previously grandfathered physicians? And if so, how much of the overall rate is attributable to this phenomenon?
The other arguments – big data, work compression, technology, etc – all mesh with the “great looker-upper” idea. If that holds true, then how do we fix it? The implication is that younger physicians are somehow lazier or less well-informed than previous generations, but that doesn’t make a lot of sense based on the first time initial certification rates. Rather, is the process of MOC giving society what it needs? It’s easy to blame the pool of young doctors, but no one seems to know how to test whether we get better outcomes from this process.
The whole idea behind maintaining a fund of medical knowledge needs a serious challenge. The future, in technology, is hurtling at us. The capacity for technology in medicine is boundless. We are “smarter than we think” when we use our solid, analytical, trained and informed powers of the mind in tandem with technology. Why not allow technology in testing? Open-book exams have been around forever in other disciplines, why not medicine?
Oh come on… It’s the same all over – if too many people fail, the test needs to be dumbed down. I don’t understand why a 22% fail rate is a problem. An exam that everyone passes is useless. Maybe people should study a little harder (I passed the boards last year, despite being a lazy “millenial”. I don’t think the exam was too difficult).
On a different note, the whole MOC thing is a big scam by the ACP. Shameless milking of doctors, even those who just passed their boards. Keep the exams hard but make them cheaper…
“An exam that everyone passes is useless” – I disagree. The ABIM recertification exam in internal medicine should not be asking esoteric questions about chemotherapy regimens or statistical calculations requiring the prior memorization of formulas (yes, these questions were on my last recert exam in 2007, which I did pass). The exam should test only that knowledge which is required for an internist to practice correctly. We do not need to know details about chemo, or to calculate PPV from sensitivity & specificity. Every question should test knowledge or reasoning ability which is absolutely essential for every practicing internist. Such a test would be a service to us, instead of a chore.
I agree with you 100%. Here is something to ponder…..if ABIM thinks it is critical for me to know what chemi I should use for NHL, and I pass the exam, should I not be allowed to prescribe that chemo? But if I did, then I will be repremanded for “practicing outside the scope” of my practice!
Someone reconcile that for me please?
ABIM recertification exam should be asking questions concerning our general fund of knowledge for day to day practice. Many people have passed the Initial Certification and have a lot of problems with Recertification. There are so many different organizations who think differently concerning patient management. These experts who design questions should not even bring conflicting data into the boards.
Most of us give our patients up to date care and constantly learn new and up to date information. I believe there needs to be some sort of testing situation/even oral exams by our peers on what to do for our patients. The current ABIM is not the answer. We have enough great minds to figure out what is the best answer.
I just passed my third nephrology board and 2 years earlier passed my third IM board. Since 1991, I have taken 6 board exams (3 IMs and 3 Nephro) and passed all 6 on the first try. The solution to the ABIM/MOC situation is the following: The boards only measure one thing: How much you studied for the exam. One cannot even sit for the exam unless they are a competent in IM or a specialty. The board exam should fail only those who did not study for the exam which is about 3-5 % of test-takers. The ABIM already solved their own problem by giving 20 points for just taking the exam. They should follow-up and make the whole sit-down exam 1 large module and assign points on how many question one gets write and go from a pass/fail to a complete/incomplete score. Initial certifiers who do not pass the exam get an incomplete and have to do many modules and get certified 1 year later than someone who passes the board and gets a complete result. Have a 2 tier system where you get board certified after the exam if you pass or certified 1 year later if you fail requiring many modules to COMPLETE your certification. We do not have enough physicians to even have 3-5% of our colleagues be branded as uncertified!
Just passed my 2nd Recertification Exam (take in October 2014).
72% overall pass rate for that exam.
79% pass rate for those taking the MOC exam for the first time.
I left the ABIM some specific feedback about their examination and some recs about how to improve the process of MOC overall.
The pass rates seem quite low to me given the high stakes of the examination and, in my opinion, this reflects on the instrument itself.
I certainly do hope the overall process of MOC by ABIM undergoes some major overhaul.
The very best module that the ABIM MOC has (one that I completed and would do over and over again) is their EBM ‘Point of Care’ module in which you create clinical questions, search out answers in the literature and then answer them with discussion about patient care applicability. This is what docs do every day and reinforces life long learning.
I hope changes are made to make the process better and not a major drag on one’s time and (let’s face it, a reason many of us went into medicine) the joy in learning about our area of study and practice.
I was dual board certified in IM and EM and passed my recert in EM but was reluctant to take my IM recert due to the fact that I have not been practicing IM for a number of years. So much has changed. I have resolved to take my recert this year and though I have passed all my prior certification exam I am concerned that I am fighting an uphill battle. I read regularly and there is a lot of overlap between EM and IM so hopefully this will help. Do most people recommend taking a course or focus on questions?
I LOVE Dr. Tolat’s idea of complete /incomplete tests . I think the test is not that hard but as a nitpicker I went crazy over some of the questions where the “correct” answer is simplistic or politically correct. I would love a complete /incomplete and the the question writers who come up with silly questions couldn’t sink anyone with excessive focus on , say microscopic hematuria workup or “subclinical hypothyroidism” or how to best decide when to biopsy someone with hepatitis C — is he ALT/AST switch, when the fibroscan is positive , or in REAL life when they have insurance coverage tat allows the procedure to be done . .
As follow-up to my initial comments, Pass/Fail Board exams make us red meat for malpractice lawyers who know a jury is more likely to disfavor a physician who has failed even one exam. Only we know that failing a board exam has no relevance to competency and only measures how much time off you got to prepare for an exam. We are facing a shortage of board-certified nephrologists due to the high failure rates of 20-50 % on the last nephrology exams.(Yes there was on 50% failure rate on the Nephrology MOC for those taking it as their second recertification). Stop helping the lawyers and switch to a complete/incomplete format. Allow new fellows to take the exam and receive a score of complete or incomplete and delay their board certification by 1 year if they get an incomplete and have them complete their certification with numerous modules.
lawers cannot ask whether you failed or not. they can ask only whether you are certified or not.
In September 2014 ABIM had dismissed the concerns of the pass rate decline as mere myth and a result of normal variation– that the “pass rates differ from year to year solely because the characteristics of physicians taking the exam change each time it is given” ( https://www.medpagetoday.com/publichealthpolicy/medicaleducation/47601 ).
“Solely because”– remember that.
In an email sent to ACP members in February 2016, the ACP president Steven Weinberger announced that
“The process for determining the passing standard was modified. As part of our feedback to ABIM, we were concerned that the past process for setting the passing standard (using the Angoff method, an established approach for setting a passing standard) typically used the question writers rather than an independent group of physicians who were not involved with writing the questions. ABIM responded to our concerns by changing its process so that the passing standard was set by both those involved in question writing and a group of practicing clinicians who did not write the questions.”
Am I the only one dismayed by the fact that ABIM didn’t recognize that it was a problem having the same group of people who wrote the exam then determine what they felt was the correct passing standard? They seemed to think that the Angoff method corrects for the inherent biases of the individual judges and raters on the panel. I assure you that it does not.
It took 15 years of the MOC exam’s existence and 5 years of a declining pass rate that bottomed out at 78% in 2013 to make any changes– and then they only took action because the ACP pointed out that this fairly obvious bias was concerning.
Considering that after this change on the Oct 2015 exam, the pass rate has gone from 78% in 2013, to 80% in 2014, to 88% in 2015 (where the fall exam ‘passing standard’ was changed due to discussion with the ACP), to 91% in 2016– I’m having a tough time believing that the ABIM is a fair organization, responsive to the concerns of the physician community they claim to be listening to. That their initial reaction to the legitimate concerns about the MOC exam was to blame the test takers for not preparing well enough when in fact their exam instrument was unfairly conceived is unconscionable.
I’d say it’s pretty understandable that there is growing skepticism in the need for or the value of the MOC process at all.
I think there is a deep black hole in the ABIM certification board testing. It’s simple to learn from ones mistakes you need to be able to know your mistakes!! ABIM needs to give the missed questions that one took and the correct answers to all test takers regardless of wheather one passes or fails.
It’s simple and no need for a analytical answer to a wrong question.
I will be 62 years old when it is time for my next recertification exam. I have been practicing successfully for 3 decades with no bad outcomes or lawsuits. I do all my CME. I think the ABIM MOC is a giant money maker with loads of conflicts of interest. Why doesn’t all CME count toward MOC? Why is it only their “sanctioned” CME where money is changing hands? Is there ANY data telling us that there is a correlation between the board certification process and clinical outcomes? DOES THE CERTIFICATION PROCESS CORRELATE WITH COMPETENCY? We need to stop the madness. We are long past the days where memorizing medical minutia creates a good physician. Testing basic knowledge is fine but the whole exam is outmoded. There is tons of data on how physician presence, empathy, and self-care correlates with patient outcomes and adherence.
The abim is a MAFIA!!!
I practiced in emergency rooms 14 yrs and provided outstanding care. The abim should be abolished.
The idea of passing a test and equating it to competence is THE flaw. Who, as a physician, should fail a test? No one! If removing a physician from practice is protecting society from incompetence, that should be done in a court of law, individual by individual, held by a panel of unbiased, non vested panel, and not by a private, money-earning enterprise. I passed medical school. I passed the useless USMLE aptitude tests. That is my bill of right. No more. Who can bank on a career that can through you out (having done nothing wrong, but failing a test) instead of doing something (proven negligent, having passed the test) that throughs you out? No one! How can you practice as a physician, knowing that in 10 Years, or maybe, now, in 2 Years, as the going recertification process has grown, failing a test, you can no longer be paid by insurance companies, and … lose your job as a result? No one!
I work from 4-5 am every day until 5-6 pm every day, sometime more. I am a solo practitioner. A Specialist. I have already repassed my boards once, and stand before the threshold of can I do it again? I have a responsibility to myself, to my family, and as the clock ticksnear, having no time to myself or my family, I ask myself, what will happen if I fail this test. A test, that I am taking, neglecting my patients the whole day taking, as no one can cover for me; At this Point, I have not dedicated any time to this comming test, there simply is none left to devote. Does a test make me competent or incompetent, pass or fail? No!
The test is for profit at my expense, and if I am failed, well, society will pay the costs, and I will lose all I have worked for, probably my family, and certainly my faith in achievement.
Who profits? ABIM. Who pays? Us, physicians. Who benefits. No one of us physicians, just the ABIM profiteers! What other profession is required to stake their payable carriers on a test that is forced on them, with such grave results? No one! Is that discrimination, sounds like it.
As long as we support a test taking system that expels those that do not make the grade, we are supporting professional genocide as an acceptable sacrifice. And who wants that? Not me, maybe no one!
The answer is the same as to why so many students are not passing the bar; our educational system has been dumbed down so significantly and our society is no longer centered around hard work and research. The boards and the bar exam have remained the same. The standards have remained constant over the last 2 decades or more but the quality of the work that leads up to them is seriously lacking. While we may dumb down school and arbitrarily pass kids for the sake of optics, we aren’t about to do that when it comes to a doctor. We aren’t going to dumb down standards for what it takes to be a heart surgeon. As a society, we need to reset the bar high. Kids who make it, make it, those who don’t, can find a line of work congruent with their abilities.