At NEJM Knowledge+, we recognize that our questions and answers aren’t — and shouldn’t be — etched in stone and that our clinical audience is best served by questions that are updated regularly to reflect the evolution of medical knowledge.
Unlike most other online resources for board preparation, we update the content in NEJM Knowledge+ at least once a month. We base those updates on our proactive review of new studies, on changes in guidelines and drug labels, and — most of all — on comments from our learners, which we gather through a feature called “Challenge Us.”
“Challenge Us” allows learners to comment on any question that they think is misleading, incorrect, or unclear — and we take those challenges very seriously. We view our promise to keep our content current and to consider all feedback as a key differentiator for NEJM Knowledge+ that improves both the quality of our questions and the learning experience we offer.
When You Challenge Questions, You Create Change
Since launching NEJM Knowledge+ Internal Medicine Board Review in April 2014, we’ve received about 3000 challenges on our content – and we’ve closely reviewed every single one of them. If the results of a challenge require us to edit a question, we make the changes, and the edited question is typically live within the month. What this means is that clinicians who challenge our questions are constantly improving the experience of NEJM Knowledge+, both for themselves and for one another.
Your Challenges Keep Us on Our Toes
Of the many challenges we’ve received, about 45% have resulted in some degree of edit being made to a question.
A few of those edits have been substantial. For example:
- Just a few weeks after JAMA published a clinical trial showing that entecavir was superior to lamivudine for preventing HBV reactivation, one of our learners pointed us to the new trial and suggested that we reevaluate our question on the topic. We did — and then promptly changed the correct answer to reflect the new evidence.
- In a question about the most appropriate initial medical therapy for highly symptomatic new-onset Graves’ disease, a learner pointed out that standard practice was to prescribe not only an antithyroid drug (as we had noted) but also a beta-blocker. We agreed and changed our correct answer to include both medications.
- In a question about a first recurrence of mild-to-moderate C. difficile infection, several clinicians noted that they’re now using vancomycin in this setting rather than metronidazole (our correct answer). After several rounds of review and debate, we decided that we’ll continue to list metronidazole as the correct answer to this question — to align with current IDSA guidelines — but in recognition of changing practice patterns, we’ve removed vancomycin as a distractor and mentioned it in the feedback as an acceptable treatment option.
The vast majority of our challenge-driven edits are more subtle — but still important. They tend to enhance the learning rather than correct the text. For example:
- In a question about recognizing and managing a suspected tension hemothorax in a hypotensive patient, our correct answer is to place a large-bore thoracostomy tube. A number of clinicians have challenged us, saying that needle decompression would be faster or more appropriate in this scenario. We’ve edited the feedback section of the question to more clearly explain why this is a case of tension hemothorax (and not pneumothorax) and why a large-bore chest tube would indeed be necessary in this case.
- Several learners asked us about the nystagmus typically associated with benign paroxysmal positional vertigo (BPPV). We reviewed all our questions on this topic to ensure consistency and clarity and also added a recent NEJM Clinical Practice review as a reference.
- In a case about managing a 45-year-old woman with lobular carcinoma in situ (LCIS), two challengers questioned the wisdom of prescribing tamoxifen without knowing the hormone receptor status first. We edited the feedback section to explain that LCIS is nearly always estrogen receptor positive.
Our challengers also point out more-minor issues with our questions, calling us out on our preferred word choices and even typographical errors. Our personal favorite of this sort was a unit correction, where a learner wrote to us about a feverish patient: “At 101°C, this woman’s not just ‘hot’ — she’s definitely ‘barbecued.’ Probably you meant 101°F, which is 38.3°C.”
Our Challengers Are Deeply Engaged in Learning
Only a sliver of our challenges are delivered with such humor. Most are much more serious, but either way, they reflect what we already knew about clinicians who use NEJM Knowledge+: they’re actively engaged in learning, whether they’re in residency, mid-career professionals, or nearing retirement. And the challengers are not just a vocal minority: Our 3000 challenges have come from about 1000 learners. While we do have a few super-challengers (logging more than 50 comments each), most users submit only a couple comments here and there, inquiring about a particular point, updating us on a specific reference, or suggesting that we double-check our facts.
By articulating what they find problematic with a question, challengers are engaging with our content at a deeper level that enhances their learning even more. They’re also inviting us to engage with them — and we do so as often as possible.
We respond to many of our challengers directly, thanking them personally for their input. If their challenge led to a change in our content, we let them know when they can expect to see the update in the product. If their challenge was off-base, we provide clarification or perspective from one of our experts.
Some challengers have expressed surprise that we actually respond to their comments:
“Wow, someone listened! I’m impressed!”
Others have said that they like NEJM Knowledge+ all the more for it:
“Now that I see that you guys actually read the comments, I’ll ensure my comments are clearer and hopefully more useful. I’m now more likely to recommend this product to others.”
Many are simply appreciative:
“Thank you. I will change my practice.”
Regardless of whether a clinician submits one challenge or a hundred, we appreciate the feedback and are grateful to all of our learners for keeping us on our toes. For more examples of how our content is constantly evolving, see our ongoing list of the substantial changes we’ve made to our questions.