The way that physicians see themselves and their own practice patterns is of critical importance. They want to know if their medical knowledge is up-to-date and “how they perform.” The holy grail is assessment that provides results and feedback such that education is enhanced and supported; assessment is most valuable when it drives future learning forward.

A research team led by Dr. John Norcini, president and CEO of the Foundation for Advancement of International Medical Education and Research and a former director of psychometrics at ABIM, backs this up in a paper published in Medical Teacher in 2011:

Multiple methods have been used to assess competence in patient care, including direct observation, simulation-based assessments, objective structured clinical examinations (OSCEs), global faculty evaluations, 360-degree evaluations, portfolios, self-reflection, clinical performance metrics, and procedure logs.…we place particular emphasis on the catalytic effect of the assessment, which is whether the assessment provides results and feedback in a fashion that creates, enhances, and supports education.

Around that same time, and before we developed NEJM Knowledge+, NEJM Group conducted extensive interviews to learn from physicians what they wanted out of education — specifically, what was lacking in their continued striving to become the best doctors they could be.

Physicians said:

  • Make my learning progressive
  • Make the most of my time
  • Make it rewarding and relevant
  • I want to learn in my own way
  • Help me through the process

With all this in mind, we consider summative vs. formative assessment, the strengths and weaknesses of each, and why this is important to physicians.

Formative vs. Summative Assessment

We commonly define “assessment” as a “test”; this is not a bad definition, necessarily, but it is limited. Tests are certainly important, in order to understand whether students, residents, and physicians in practice have learned what they set out to learn — this is summative assessment, or assessment of learning. Experts in higher education and learning have come to conclude that there’s another way to think about assessment — formative assessment, or assessment for learning or even as learning. In this model, the assessment itself is part of the learning continuum.

The Relative Weaknesses of Summative Assessment

In a summative exam, you select a sample number of learning points to test the students on — this is the best you can do, because it is challenging to have people sitting in an exam room for more than a few hours or a day at the most. So you test them every so often on a representative sample of the items they really need to know to demonstrate their competence. The validity of such a sampling test is completely dependent upon the predictive value of the test items.

Summative assessments have the same weaknesses as cross-sectional surveys (prevalence studies): They provide a snapshot of where a learner is at a given point, but they cannot say anything about how the learner got there — or, more important, where the learner is heading. The performance on a summative test can be — and often is — the result of intense cramming leading up to the test. This learning pattern (which is the opposite of efficient spaced repetition) may significantly limit long-term retention.

In addition, the sampling needs to vary significantly over time to avoid test takers focusing on beating the test rather than learning the subject matter. The risk-reward profile of cheating is attractive with summative testing, partly because you only have to cheat for a short period of time.

Formative Assessment Can Transform the Learning Process

Formative assessment, by contrast, assumes physicians are continuously learning through assessment. The assessment can be for learning where physicians learn in increments, and the “test” is merely questions and answers that imbue knowledge and reinforce it — combating memory decay.
bell curve

We can illustrate the difference between these two types of assessment—and the benefits of the formative approach—with the traditional bell curve of test performance. Looking at the traditional bell curve above, we see that around the mean, in the center of the curve, 34% (or 1 standard deviation) fall below the mean and 34% fall above it. There are about 15% that “fail” — those that find themselves below the –1σ line. This percentage corresponds closely with the findings of internal medicine initial certifying examination assessments over the past few years, which show approximately 15% who take this initial examination will fail it (and, correspondingly, 85% will pass).

Assessment for learning has the potential to make everyone better, shifting the bell curve to the right for all learners. So you can see that instead of 15% failing, you only have about 2.5% below the –1σ in the shifted-mean curve. The poor become fair, the fair become good, the good become great, and the great become exceptional. Isn’t that what we all want? The mechanism is simple. Assessment for learning secures that a learner is aware of what still needs to be improved and thus systematically closes the learner’s gaps.

It seems that the regulatory bodies are on the same page. Dr. Lois Margaret Nora, President and CEO of the American Board of Medical Specialties (ABMS) and her coauthors in a 2015 JAMA article, titled, “Of the profession, by the profession, and for patients, families and communities; ABMS board certification and medicine’s professional self-regulation,” writes,

Innovations in examination delivery and alternative assessments, including a greater emphasis on formative assessments, are being implemented.

One example of this move to formative assessment comes from the American Board of Anesthesiology MOCA Minute, which is currently being used in placed of a secure MOC exam.

So Why Haven’t We All Made this Transition?

In short, the reason we haven’t all made this transition is that it isn’t easy. To move from summative to formative assessment, you need both high-quality content and smart technology. You can’t rely on a representative sample of learning points; instead, you need to provide a robust volume of questions, answers, explanations, resources — and couple all that with the learning technology to repeat and reinforce the material over time.

The ultimate method is to use assessment as learning, where the physician is learning as an integral part of the formative assessment process. This requires very short cycles between learning, attempting to use what has been learned, and repeating useful feedback to the learner.
What is the future of assessment in medicine? Do you see changes afoot in your area of expertise?