When you’re working through question after question for board review, it can be easy to get lost in the weeds. You might look up bleary-eyed during one of your study sessions and wonder how long you’ve been at it, whether you’ve made any progress, and how you’re actually scoring on the questions. Luckily, when you use NEJM Knowledge+ for board review, it tracks all of that information for you and puts it right at your fingertips, so that you always have a sense of how far you’ve come and what lies ahead.

Progress and Proficiency: A Measure of How Far You’ve Come

NEJM Knowledge+ includes more than 1500 case-based questions divided into a series of specialty-specific modules (cardiology, pulmonology, infectious disease, and so on). Each question is based on one or two key learning objectives that we think are important to master in the practice of medicine.

The best way to tell how many learning objectives you’ve been tested on so far is to look at the progress bar that appears at the top of each question and also next to each specialty module name on the dashboard:


The progress bar represents the proportion of learning objectives that you’ve been tested on so far out of all the available learning objectives that are in that module.

In the screenshot above, the user has been tested on about 24% of all the learning objectives in the pulmonology module. (The bar itself gives you a visual estimate, but if you hover over it, a tool tip will appear that tells you the exact percentage.) The number that appears to the right of the progress bar is your proficiency score — it tells you the percentage of learning objectives that you’ve mastered out of all the ones you’ve seen so far in a given module. In this case, the user has been tested on only 24% of all the pulmonology learning objectives, but she has mastered 76% of those.

What about your overall progress and proficiency in NEJM Knowledge+? At the top of the dashboard, you’ll find a progress bar and proficiency score that indicates how you’re doing across all the specialties combined.

overall learning

Time and Effort Remaining: A New Measure of What Lies Ahead

Progress and proficiency are helpful measures of where you’ve been and how much you’ve mastered, but they don’t give you a good sense of how much material you have left to cover. We’re working on a new report now that will do just that: In addition to progress and proficiency, it will tell you how many learning objectives you have left in a given module and how long they’re likely to take you to cover, based on the time you’ve spent so far, the way you’ve answered previous questions, and the way you’ve rated your confidence.

What Can We Do Better?

NEJM Knowledge+ has the ability to serve up multiple questions on each learning objective — and we have multiple formats for each question — so the underlying calculations for progress and proficiency are not as simple as they might appear. Our goal, though, is to make the indicators you see as clear and helpful as possible.

To that end, we’d like to hear from you about the data we’re currently providing on progress and proficiency — and what additional information you might like to see. For instance, our system tracks not only whether you answered a question correctly but how confident you were in your answer. We currently aggregate those data into a comprehensive report (below), but perhaps it would be helpful to see a snapshot on your dashboard — some indication of how often you are consciously competent (answering correctly with high confidence) versus unconsciously incompetent (answering incorrectly with high confidence).

Screen Shot 2014-06-16 at 1.25.08 PM

Another idea is to provide some measure of how much effort it’s taking you to get through each module. For example, you might be answering most of the cardiology questions correctly with high confidence the first time you see them — but consistently answering the rheumatology questions incorrectly with low confidence. Being aware of this pattern might help you decide to where to focus your next round of study efforts. And maybe it would reinforce where can be appropriately confident and where you might need to be more cautious in your clinical decision making. We’d love to hear your thoughts about measuring progress and proficiency in NEJM Knowledge+.

Take our poll (below), drop us a line, or leave us a comment. We’re all ears.