In Part 2 of our Q&A series with Area9 Learning founder and CEO Ulrik Christensen, MD, we learned what makes the company’s technology stand out in the field of adaptive learning systems. Here, in Part 3, Christensen describes how and why Area9 has modified its technology to build a lifelong medical education system that addresses both maintenance of certification (MOC) and board review.

In Part 1 of our interview, you spoke of your belief that improving basic medical learning and knowledge can lead to reductions in medical errors. Would you say that this conviction makes you a proponent of lifelong learning? Of MOC (Maintenance of Certification)?

We know for a fact that people forget things. We know they forget much more than they think they forget. And we know it is very difficult for people to understand and monitor precisely what it is they have forgotten. So, yes, I believe there needs to be a system for ensuring that our physicians refresh and assess their knowledge on an ongoing basis.

With that said, a potentially valid objection to MOC is that it relies on high-stakes exams that may or may not be perfectly fair or representative of what physicians really know or are capable of accomplishing in practice.

My personal view is that we should be doing much more formative assessment on an ongoing basis; we should be making sure every day that physicians refresh and reinforce their knowledge. This is the ultimate goal of adaptive learning software. We only need the high-stakes exams because we need to be able to measure results and cannot yet do so practically on an ongoing basis. From a learning perspective, however, it is ineffective to cram for high-stakes exams, because that practice results only in superficial knowledge that is quickly forgotten. As a medical education system, NEJM Knowledge+ alleviates this problem because it helps people to monitor and refresh their knowledge systematically over time and is engineered specifically to guard against memory decay.

Are your algorithms essentially the same from one adaptive learning product to the next? Have they been modified for NEJM Knowledge+?

Actually, our algorithms change substantially. The algorithms we have developed for NEJM Knowledge+ are quite different from the ones we typically use in undergraduate university education, because we are addressing much higher levels of abstraction and learning objectives.

Think of the following learning taxonomy in physics: Level 1 would involve knowing the definition of gravity. Level 2 would require understanding the principle of gravity — that if you drop various objects, they will always fall. Level 3 would involve applying gravity — for example, using a heavier object to weigh down a lighter one on a windy day. Level 4 would require analysis — for example, trying one object as a weight, figuring out it’s not heavy enough, and then choosing something heavier. Level 5 would involve monitoring to make sure your solution works consistently over time. Level 6 would require strategy, anticipating the problem of wind and deciding to buy heavier versus lighter items to use outside in the future, and so forth. This is just one of many learning taxonomies, but they all have a similar linear structure.

In NEJM Knowledge+, medical professionals need to be able to analyze complex sets of information to come up with the right answers, so we had to engineer our adaptive medical education system for higher levels of abstraction. That requires quite different adaptive algorithms than systems focused only on Level 1 or Level 2 learning objectives. With NEJM Knowledge+, we assume a certain [high] level of knowledge on our users’ part, so we are not reverse engineering all the way back to Level 1. What’s more, medical learning is not as perfectly linear as the gravity example I have given. To address the higher-level objectives and complexity, the physicians creating content for NEJM Knowledge+ had to be extremely knowledgeable about how the different learning objectives relate to one another. In fact, an enormous amount of effort and intelligence was applied to the back-end proportioning, distribution, and mapping of relationships among various content elements (questions, selectors, answers, citations, and so forth) to the various learning objectives that had been established for NEJM Knowledge+.

Coming soon: Part 4 in our Q&A series, in which we learn more about specific Area9 innovations aimed at more effective learning and long-term retention of information.