It is interview season, and I am reviewing candidates for my residency program. It is a very humbling process. Even considering board scores, transcripts, and personal statements, the true person behind the dossier can remain opaque. How do we figure out who might be a good fit? I have come to realize the importance of good letters of recommendation. An effective letter complements the applicant’s grades and board scores and provides a crucial insight: can this student do the job?
Applicants have handpicked the evaluators who will best advocate for them and convey their true qualities — every year I find myself writing several letters. But have we as medical educators failed our charges? There are several problems with our letters, and they are all fixable.
Formulaic and Flat Letters of Recommendation
Many of our letters are so formulaic and flat as to be essentially useless. The applicant “waives their right to review this letter.” This is a good thing, since so many letters are so trite, so dreary, we would be embarrassed if our students saw them. Most of the letters sound as if we downloaded them from the same website. Nearly all of them say a variation of their applicant being, “smart and hardworking…an asset to any residency program.” What do we, on the receiving end, do with this pile of identical bromides?
A survey among program directors, across 21 specialties, evaluated selection criteria for their rank list. Letters of recommendation did not rank in the top 16 criteria. Given the low quality of our letters, this makes sense.
Bias Conveyed Implicitly in Our Letters’ Wording
Last year I wrote about implicit bias in letters of recommendation — and this problem needs to be discussed more widely and addressed. Studies have found that race- and gender-based bias abounds in letters of recommendation: white students are more likely to be described by their evaluators as “exceptional,” “best,” “outstanding,” while black students are more likely to be described as “competent.” Women applicants are more likely than men to be described as “caring,” “compassionate,” and “empathetic.” These differences remained significant after controlling for Step 1 scores.
As we are writing and reading letters of recommendation, we should try to keep these biases top of mind — awareness is at least a first step toward rectifying inequality.
We need to do better. A strong, unbiased endorsement from an attending physician who worked directly with the applicant may override a lower board score or mixed grades in other rotations. A good letter forms an impression of the applicant before the interview takes places — and leaves a reference point after the interview is over. A good letter can nudge an applicant’s place on the rank list.
Alternatives to the Narrative Model Fall Short
Nearly two decades ago, our colleagues in Emergency Medicine rolled out a Standardized Letter of Recommendation (SLOR) — and many program directors have been using it ever since. Instead of the traditional narrative model, the SLOR asks the evaluator to rank the applicant as outstanding (top 10%), excellent (top third), very good (middle third), and good (lower third) in a range of areas from commitment to emergency medicine, work ethic, ability to generate a differential diagnosis, and personal qualities. In a follow up survey, the SLOR was ranked as the most important factor to determine whom to invite for an interview.
More recently, in 2017, the Alliance of Academic Internal Medicine (AAIM) published guidelines for letters of recommendation for residents applying to fellowship that emphasize the six ACGME core competencies — patient care, medical knowledge, interpersonal and communication skills, practice-based learning and improvement, and professionalism. The six ACGME competencies are the rubric by which we evaluate our residents on the wards and in the clinics. It only makes sense that we would convey that in our letters of recommendation.
In addition, the AAIM says that the letters should highlight scholarly contributions, extra skills developed outside the residency, as well as a “deeper insight and clarity about personal characteristics of the resident, such as level of engagement in assigned activities and degree of initiative.” They make no mention about rank within a historic context or their residency cohort, but rather suggest a summary statement about the applicants’ suitability. While this model is intended for residents, it applies to medical students as well. The six ACGME competencies are the framework for resident evaluation, so it stands to reason that we would use them to describe applicants’ skills and knowledge — even as they are still medical students.
However useful the SLOR and the AAIM models are, though, their drawback is that they de-emphasize applicants’ personal characteristics by turning them into grades. We have plenty of objective data. What we really need to know is: What is the person really like?
Recommendations for Your (and My) Letters of Recommendation
Each year, two to three letters really stand out. They tend to flesh out a person’s true characteristics by using a narrative model and have qualities from both emergency medicine’s SLOR and the AAIM’s emphasis on the ACGME core competencies.
My recommendations for writing your letters of recommendation are as follows:
- Tell a story: Share a specific example of how the student connected with a patient, gave a great presentation on hyponatremia, worked with the care coordinator to affect a safe discharge. We have seen their board scores, evaluated their transcripts, and read their personal statement. The core competencies can form an outline, but more importantly, specific examples carry a lot of weight. Anyone can polish up for an interview. You saw them in action. What did they do?
- Endorse: Would you want this applicant at your institution? Where does this applicant stack up in your experience? Top 5%? One of the best this year? One my colleagues believes this is the litmus test for a true endorsement. “If the letter doesn’t say we want the applicant to stay with us,” he says every year. “Then it’s not a strong endorsement.” This might be a tad harsh, but he recognizes the fluff that pads so many letters.
The stakes are high. Our applicants invest their whole selves in their applications. We do them a disservice with half-hearted, half-baked letters. We need to figure this out.
Benjamin R. Doolittle, MD, is an expert in burnout and wellness in residents and physicians. He is an associate professor and program director of internal medicine and pediatrics at the Yale University School of Medicine and the medical director of the Yale Medicine-Pediatrics Practice.
In my experience interviewing candidates for hospital positions, nothing is more important than picking up the phone and speaking to the writer of the letter should the recommendation be positive. It is remarkable what someone wl say over the phone that they are not willing to write.
In Italy, where Medicine doctors can freely bedside apply new diagnostic clinical methods without the fear of having to face the competent health authorities, in the letter of recommendation of one of my disciples, I would write “He is able to recognize with a common stethoscope from birth the Inherited Real Risk of CVD, T2DM and Cancer, eliminating it with inexpensive Reconstructing Mitochondrial Quantum Therapy.