The Journal of the American Medical Association (JAMA) recently published two new research papers (plus a related op-ed piece) investigating statistical relationships between ABIM’s Maintenance of Certification (MOC) requirements and health care quality. On December 15, JAMA convened a follow-up webcast, featuring

  • lead research authors Bradley M. Gray, PhD, and John Hayes, MD,
  • expert commentators Thomas Lee, MD, MSc, and Robert Centor, MD, MACP, and
  • Richard Baron, MD, MACP, President and CEO of the American Board of Internal Medicine (ABIM) and the ABIM Foundation.

The webcast kicked off with Bradley M. Gray, PhD, a health economist at ABIM, presenting major findings of a new paper he coauthored called, “Association Between Imposition of a Maintenance of Certification Requirement and Ambulatory Care-Sensitive Hospitalizations and Health Care Costs.” Dr. Gray described the research as “a natural experiment that occurred when one set of general internists who initially certified in 1991 were required to recertify ten years later in 2001, while another set of internists, who had initially certified just 2 years earlier, were grandfathered out of that requirement.”

The MOC-required group could be seen as the treatment group, while the MOC-grandfathered group could be seen as the control, Gray said. “Applying a difference-in-differences framework to measure whether the MOC requirement was associated with changes in outcomes, we compared differences in outcomes for a group of Medicare beneficiaries treated by the 1991 MOC-required group of physicians both before and after the 2001 requirement.” To control for effects of aging and other factors occurring at the same time that the recertification requirement was introduced, the researchers applied a similar framework for patients treated by MOC-grandfathered physicians.

MOC and Health Care Costs

Dr. Gray and his colleagues found statistically significant reductions in per-patient health care costs after MOC was introduced. Mean health care costs were essentially identical between the two groups of physicians prior to MOC, Gray noted; after 2001, per-patient health care costs for beneficiaries treated by MOC-required physicians were $167 or 2.5% lower, on average. “Digging deeper,” Dr. Gray said, “we also found statistically significant reductions in costs associated with laboratory and imaging testing as well as specialty visits.”

“Our take on these findings,” said Dr. Gray, “is that MOC requirements made internists more knowledgeable, which meant that they needed to do less testing to make medical decisions and did not need to refer as many patients to specialists.” And, while the per-patient magnitude of the cost-savings might seem quite small, it “could easily add up to savings in the billions at the population level, considering [that] general internists represent 45% of adult primary care physicians and treat patients of all ages and that annual health care spending for the Medicare program alone is over $500 billion (and that doesn’t even include the amount covered by Medicare beneficiaries), with total health care spending in the trillions annually.”

Thomas Lee, MD, MSc, the chief medical officer of Press Ganey Associates and a senior physician at Brigham and Women’s Hospital in Boston, added that “the cost differences really are intriguing because most data that look at the relationship [of physician age versus cost] show the opposite. They show that physicians who are older have lower per member/per month spending…The fact that this trend went in the other direction is very intriguing and makes it more likely to be real; 2.5% is bigger than most of the ACOs have been turning in. So, 2.5% would be a very big deal from a health policy perspective.”

ABIM president and CEO Richard Baron, MD, MACP, also commented on Dr. Gray’s cost-related findings:

I firmly believe that a big reason – if not the reason – for the savings resulting from MOC can be traced back to the fundamental knowledge underpinning why we have MOC. Knowledge matters. The more individual physicians know, the less likely they are to take what you could call a ‘shotgun approach’ to care, adding on more referrals and consultations, ordering unnecessary batteries of lab tests and imaging studies, etc. In a world increasingly focused on outcomes that are created by teams, technology, and systems of care, the contributions of individual physicians can be hard to identify. Board certification and MOC focuses on the specific skills of individual physicians.

An alternative explanation for the cost savings — offered by a commenter on the original JAMA article and brought into the webcast conversation by event moderator and JAMA deputy editor Dr. Michael Berkwits — is the notion that, by virtue of having 2 more years (an additional 20%) worth of experience as practicing clinicians, the MOC-grandfathered group might simply have been savvier about administrative billing and obtaining payments from the Medicare system. Dr. Gray responded by noting that they “matched patients in the pre-period…the costs were almost identical before the 2001 MOC requirement. That suggests any cost differences were relatively small due to that difference in experience.”

Other webcast audience members asked for greater insights into whether Dr. Gray and his coauthors had controlled for confounding variables, such as location (for example, “where you train influences cost expenditures”). Dr. Gray explained that they accounted for location in several ways: first, by including control variables for location in their model and second, in matching the two patient populations. “In addition, in one of the sensitivities, we included a model where we accounted for every non-time-varying characteristic of doctors and patients. In that model, as long as the patient moved, any location or practice type would [have been] captured by these panel-fixed effects. The results of that model were slightly higher than those we reported in the base case.”

MOC and Hospitalization

Dr. Gray reported that his study found no significant change to ambulatory-care sensitive hospitalizations (ACHS) per 1000 beneficiaries (patients) after the ABIM MOC requirement went into full effect. In his JAMA editorial reacting to Dr. Gray’s article, Dr. Lee noted that “…the outcome measure (ACHS) was designed to assess access to primary care in populations, not the quality of care delivered by individual physicians.” Dr. Gray made a similar observation during the webcast, noting that the ACHS measure is sensitive to access versus quality of care. “Everyone in our study had access to physicians and had Medicare insurance,” he stated.

Expanding on his JAMA editorial commentary, Dr. Lee said,

For anyone who thinks this is a paper that proves that MOC does nothing, it is important to recognize this was a one-time look at the impact of time-limited certification (the part of the MOC that was introduced in 1990) and the outcome measure was the best they had available, but it was not really designed to assess the quality of care by individual doctors. The outcome measure was used on just about 80 patients per doctor – a very small part of any doctor’s panel – and the doctors being profiled only accounted for a minority of care.

Indeed, Dr. Lee said that, given the rigor with which the researchers approached their MOC study, it would have been “amazing, just amazing” if they had discovered a significant impact on health care quality.

MOC and HEDIS

The second MOC study addressed in the JAMA webcast, “Association Between Physician Time-Unlimited vs Time-Limited Internal Medicine Board Certification and Ambulatory Patient Care Quality,” was conducted by John Hayes, MD, and colleagues within the Great Lakes Department of Veterans Affairs (VA) hospital network.

As the associate chief of staff at the Clement J. Zablocki VA Medical Center in Milwaukee, Wisconsin, an active primary-care physician in the hospital’s outpatient clinics, an inpatient ward attendee, a member of both the hospital’s credentialing and privileging and salary and hiring committees, a peer-review member, and also the hospital’s physician director for quality management and safety, Dr. Hayes began his presentation by noting “a keen interest related to the quality of care we are delivering.” He also emphasized the practical difficulties of evaluating groups of physicians with varying board certification statuses. “In hiring and credentialing, a dichotomy arises,” Dr. Hayes said. “How are we to view a grandfather [who was] certified 25 years ago, who is now certified with an asterisk, versus someone who boarded 11 years ago who is now listed as not certified?”

Dr. Hayes said the VA researchers began their exploration by analyzing peer-review results between Milwaukee VA time-limited and time-unlimited board certification cohorts. “In analyzing peer review in Milwaukee, we found no difference in care between the cohorts, but the numbers were extremely small,” he said, adding that, “the nuanced way in which peer review is run at each facility currently does not lend itself well to pooling this data.”

Standard health care quality measures across VA facilities, however, were more easily pooled, Dr. Hayes noted. “Given our alignment with HEDIS [Healthcare Effectiveness Data and Information Set] measures, we felt the current VA measures would be excellent markers for our objective, which was to determine whether there are differences in primary care quality between physicians holding time-limited or time-unlimited certificates.”

The resulting study included data from the four largest hospitals in the Great Lakes VA network and covered 10 primary care performance measures, Dr. Hayes said. “We felt this would better focus and answer our question of broad primary-care quality as opposed to many other studies that only looked at one or a couple of measures.” The simple result, Dr. Hayes noted, was that, “We found no significant difference in performance between the time-limited and time-unlimited ABIM certificate holders.”

In commenting on the VA study, Dr. Lee said it is important not because it proves or disproves efficacy of MOC but rather because it shines a light on the growing relative importance of team-based care with assiduous attention to quality-performance metrics when compared to impacts of unlimited versus time-limited board certification. “I was struck by how much better the quality data are than the rest of American medicine…it’s a real tribute to Dr. Hayes and colleagues [as to] what the data showed.”

Also joining the JAMA webcast panel was Robert Centor, MD, the chairman of the American College of Physicians (ACP) Board of Regents and the regional dean of UAB Huntsville Regional Medical Campus, who commented on both research papers more conceptually:

Both of the articles…aim to evaluate physician quality as a function of MOC participation. We have to consider a couple of issues when we think about these two articles. Number one, both of the studies look at the original MOC requirement, which was just an exam after ten years. Neither [of the articles] address [MOC] changes that occurred in 2005 and most recently that have caused much angst.

Dr. Centor suggested that both studies relied on “somewhat unidimensional” measures of health care quality and cited Donabedian’s seminal article “Evaluating the Quality of Medical Care” published in the Milbank Quarterly in 1966. “In that classic article, [Donabedian] points out that quality is multidimensional and that you make a mistake if you only measure one piece of quality and try to extrapolate and say that means there is overall quality. It’s a mistake to say we have really measured physician quality; we have measured the performance on a metric that is one of many potential metrics that one might use.”

Dr. Centor expressed concern that standard performance measures may be supplanting such critical measures as diagnostic accuracy and patient-centric measures of physician quality, such as communication skills. “How well do we take a history, do a physical? How well do we explain to a patient what is going on?”

While noting that both studies “are good studies and give us some clues,” Dr. Centor suggested that both studies consider health care quality from narrow perspectives and would also require confirming studies in different settings and across more diverse patient populations before drawing firm conclusions. “We have been cautioned by everybody who looks at research that, when you have one study, you need confirmatory studies to see if [the results] hold up in different settings.”

In addition, Dr. Centor emphasized that neither study addresses the more recent progression of MOC from a once every ten-year exam cycle to a continuous process that is considerably more complicated and burdensome for physicians. Dr. Centor said he believes the MOC exam needs to transition from one that is essentially summative (tests one’s knowledge at a single point in time) to one that is more formative – meaning it incorporates a continuing cycle of questioning, teaching, and requestioning to determine if a concept has been learned – and also more highly focused on what has changed in medical knowledge since a physician’s prior certification cycle.

Taking advantage especially of Dr. Baron’s presence on the JAMA webinar panel, audience members sent in many other questions about MOC not directly related to the research being discussed but, rather, reflecting general concerns regarding MOC relevance, burden, evidence, and so forth. To hear resulting commentary and lively debate in how Dr. Baron and other panelists addressed these questions, you can view the JAMA webinar in its entirety on demand, and we encourage you to add your own thoughts and reactions to it in the comments section below.