An interview with psychologist Nate Kornell
At the end of a test, the one thing you can say for sure is what you know and what you don’t know — even if you disagree with the test-makers on what was on the test.
You might also find out what you thought you knew, but didn’t.
So, at a minimum, testing (including self-assessment) can inform your practice by allowing you to build up real (not inflated) confidence in your knowledge of diseases, symptoms, drugs, and new or changed guidelines that you ought to know.
An expert on the testing effect on learning and memory as well as self-confidence is Nate Kornell, a research psychologist at Williams College. I asked him a few questions about how his research applies to high-stakes medical examinations like the boards and studying practices.
JAG: You have written, “There’s no point in learning something if you can’t remember it later. The real goal of education is long-term retrieval.” But many physicians remain skeptical. I imagine that they want to know: Can a test assess my aptitude and skills or even my knowledge? Can’t any multiple-choice question be easily answered by “looking it up” — and isn’t this what we all do in real life?
NK: Tests do two things: they measure knowledge and they help you learn. Your question is about assessment, but let me talk about how they help your learning and memory first. The basic comparison, which has been used in hundreds of experiments, is between asking and telling. If I ask you a question and then tell you the answer, you learn more than if I just tell you the answer. The same is true with a skill like using new software or juggling, if I ask you to do it and then help you, you learn more than if I just tell you how to do it. Can people learn more from a multiple-choice test than from just being told the answers? Yes. But multiple-choice tests rarely apply well to skill learning.
Now I’ll talk about the assessment side. First, multiple-choice tests are not a good way to measure skills. A skill is the ability to complete a task that requires a coordinated set of steps. If you want to assess skills, you should be having people perform the skills. But multiple-choice tests (and other written tests) can be effective ways to measure knowledge.
Looking things up is great, but knowledge is always going to be important. If you are a physician, scientists, lawyer, etc., and you have to look things up every 5 minutes, you are not going to be very effective. In fact, when someone makes a discovery or reaches a deeper level of understanding, it is often because they are making connections between facts and concepts that they know. That does not happen if they don’t have the knowledge stored in their head. For example, I was planning a new study yesterday and realized that we should consider the implications of an older article that I read a few years ago. If I had not known about that article it would not have popped into my head and it never would have occurred to me to look it up, either.
Good multiple-choice questions can measure factual knowledge, and they can also help increase it.
JAG: I have seen research showing that high-stakes exams (especially closed-book exams) lead to anxiety and lack of self-confidence. This may dissuade people from using self-testing as a method of learning and memory. What do you make of the connection between the testing effect and self-confidence or self-efficacy?
Tests can be helpful in this regard. Tests keep people’s confidence calibrated with their performance. People who are not being tested are more likely to be overconfident (or, less often, underconfident) than people who are not being tested. Giving fewer tests can actually hurt a student who is not learning the material, because by the time they realize they need help they might be way behind. Tests also increase students’ motivation to study.
However, tests can be stressful and anxiety-provoking. It can help to avoid making the stakes of any single test too high. I find that students like having more tests, but only in retrospect. But to be honest, I’m not an expert in issues of self-esteem or motivation.
JAG: Adaptive learning programs (like ours) are made with the testing effect in mind: the system serves up questions again that the learner struggles with, and skips over questions that the learner initially or subsequently masters after more attempts to answer correctly. The system waits to repeat the questions that the learner found challenging so that it does not rely on short-term memory of the correct answer. Are psychologists and education experts studying adaptive programs, and have they seen results on learning and memory that are similar to earlier studies on retention?
NK: Adaptive learning programs do two things: They focus on questions the student is struggling with most and they are designed to space out repetitions of the same information. Both are important. The idea of focusing is intuitive, but spacing can sometimes be counterintuitive, because the longer you wait before seeing a repetition, the harder the test will be. But that is a very good thing. A century of research have shown that spacing is one of the most robust and valuable effects in the science of learning, so my advice is: trust spacing even if it does not feel right. It hurts in the short term but it’s so, so worth it.
JAG: I repeatedly saw in research papers the finding that the research subjects didn’t believe that active recall or testing would improve their performance, and that they had more confidence in their study habits (reading, taking notes, or concept mapping). When the students saw the results of the study, they were surprised. What do you think about this phenomenon?
NK: It’s natural to assume that doing well while you study is good. Don’t be fooled. A mountain of findings have shown that people learn when they struggle, not when things are easy. The problem is, we judge our learning based on our performance. Making things easy on yourself while you study makes you do better while you study. But only while you study. “Desirable difficulties” like spacing and testing make you do worse while you study, so you think you learned less, but because you are struggling, you actually learn more.
Of course, doing well is a sign that you know a lot, and that is good. But it’s also a sign that you probably aren’t learning much. Learning and memory are different; learning is how much your knowledge is changing. It is not about making the study materials harder, it’s about approaching the materials in a way that makes you think hard and remain an active learner. If you find it easy, try to think of productive ways to make yourself struggle. You’ll feel like you’re learning less but you’ll learn more.
Nate Kornell is an associate professor of cognitive psychology at Williams College. His research focuses on how to learners can increase the efficiency of their learning and how typical learners understand and manage their own learning. He blogs at Everybody Is Stupid Except You. Twitter: @natekornell.