A Brief Update on USMLE Score Creep
HTML-код
- Опубликовано: 19 ноя 2023
- Earlier this fall, the National Board of Medical Examiners released updated USMLE performance data. Naturally, that means it’s time again to talk about USMLE score creep.
In this video, I’ll explain what’s driving the slow and steady increase in USMLE performance; examine what it means for current medical students, residency applicants, and program directors; and predict how the format of the USMLE will change in the very near future.
_
NOTES, BY SECTION:
0:39 - Overview of #scorecreep
USMLE Score Interpretation Guidelines (12-Sept-2023)
www.usmle.org/sites/default/f...
1:56 - USMLE scores, then and now
4:14 - “It’s because Step 1 is pass/fail!”
5:14 - “Everyone is cheating!”
7:18 - “They keep increasing the passing standard”
Sheriff of Sodium: A peek inside the sausage factory - setting the Step 1 minimum passing score (2019)
thesheriffofsodium.com/2019/0...
10:13 - “I’d score 270 if I took Step 2 today”
11:03 - How USMLE scoring works
Sheriff of Sodium: Breaking the magic - how the USMLE three-digit score is calculated (2020)
• Breaking the magic: ho...
14:39 - Thought experiment: Roger Bannister
17:04 - “It’s just because test prep is better”
18:21 - Thought experiment: the electric washing machine
19:49 - “This proves nothing because the test is different”
21:44 - “The exam must be getting easier!”
Case SM, et al. Acad Med 1996; 71(10 Suppl):S238-30.
pubmed.ncbi.nlm.nih.gov/8940926/
Step 2 CK Sample Test Questions
www.usmle.org/prepare-your-ex...
28:00 - “How will programs evaluate applicants when everyone scores a 270?”
Mee J, et al. Adv Health Sci Educ 2023; online ahead of print.
pubmed.ncbi.nlm.nih.gov/37665...
_
MUSIC:
“Clutch,” by Nicolas Major.
This isn't the most relevant comment, but the fact that they continually raise the passing score on a criterion-referenced test just to maintain their fail rate is one of the most mind boggling things about the USMLE, and yet almost no one talks about it, so I appreciate you at least mentioning it. It's so transparently profit driven and totally undermines all their high falutin rhetoric about the stated purpose of the exam.
I'm impressed with myself that I got that all out without using the phrase "fucked up" even once! I'm a professional, damn it.
Thanks for watching - and for your outstanding decorum. I’m gonna do a video about the standard setting procedure later this winter.
It is also important to note that internists licensed before 1990 have grandfathered lifetime certifications without any requirement to show understanding of current practices/literature for the subsequent 30 years. Some chose to recertify on their own accord, many did not. Some choose to continue MOC activities, many do not.
I’m astounded at how many times I’ve been told/reminded by faculty “med school isn’t just about passing your boards.”
Like, wow, that’s so deep. thank you for your wisdom 🙏
27:35 I would even go further as USMLE has changed those types of questions from
Increased, Decreased to actual values that students may need to cross check with pulling up the lab values tab and manually scroll thru values. Many student mitigate this by memorizing the reference values for labs to save time.
Also HPI style q’s that take even longer to read
17:34 This is a good point, but also have to say that medical schools do not cover third party resources ever, which also makes access to resources dependent on financial resources as well as access to recent mentorship to tell them what to do/buy EARLY in medical school.
I think the MCAT will also eventually transition to short answer format. R.I.P. my grandchildren
I think you’re right.
Thank you so much for talking about this! I think a video on how the passing score is set would be beneficial as well.
Thank you for watching! I’ll work on it. Stay tuned.
Thanks for the data
I think this is more complicated than a general trend. Since STEP 1 went P/F, more people have arrived in their clinical years with a poorer knowledge base, yet the STEP 2 average still increased. To me, this means that students are studying harder than ever for STEP 2 as they’re managing to score better than ever despite coming in at a greater knowledge deficit. STEP 2 is no longer an afterthought and is quite literally everything, so I think we can expect an acceleration of this score increase in the coming years until people’s collective cognitive skills max out.
There is use of recalls even inside us. US MD students buy recalls from overseas market thats how scores are rising since scores are what matters to match
I wouldn’t be surprised if some US students were using recalls. But I don’t think that’s behind the rise in USMLE scores. That trend has been longstanding and consistent over time. I discuss more about that here if you’re interested:
An Update on USMLE Score Creep
ruclips.net/video/1gKKAZ5aO8E/видео.htmlsi=-A4UdIRPgSKsPgqZ
This could not be more wrong. Recalls are not nearly as popular as you think they are in America. Certainly we generally don't need to rely on them to score high.
Good video overall, definitely agree that the rise in scores can't just be singled out to one or two blanket excuses, and especially not explained by nonsense explanations like widespread cheating or step 1 going pass/fail. Certainly it's a multifactorial issue that does include better resources and dedicated study time, but also more efficient study strategies, more incentives to score higher, changes in teaching strategies, etc. It's also important to note that the score creep we see in step 2 isn't an isolated event, there are examples of score creep in a number of other academic realms outside of medicine as well. Considering only medical school-related factors likely doesn't cover the whole story either.
Ultimately, it would be interesting to look at the underlying reasons that current physicians have these kinds of objections. Sure, there's a small component of pride involved. But anecdotally, my colleagues that I talk to overwhelmingly agree and admire that the newest crop of students have way better CV's than we did. However, if you asked many of us if these students turn out to be any better residents and physicians than we were, you'd get very mixed answers. And I think it really drives into the central question of: How effective are the scores, letters, publication numbers, etc for determining success as a doctor? I'm not sure myself, would be interesting to hear thoughts and if there's data to that. And it may be that mild to moderate swings in step 2 scores among comparative applicants just doesn't matter that much and certainly may raise questions of why this test isn't pass/fail too.
Lastly, just a minor quibble about that USMLE question example regarding bulimia. It doesn't detract too from your point much I think because yes it's still a more difficult question than just a direct one liner without a vignette. But it does bear noting that large vignettes do offer more opportunities to get to the right answer, even without knowing the diagnosis or the significance of all the items presented. As in this example, the potassium and bicarbonate levels don't actually need you to know what the diagnosis is, just that there is significant loss from vomiting in the stem. There are some ways that students have learned to use these openings in a large vignette to get to the right answer. And maybe a stronger focus by students on learning the underlying pathogenesis rather than just memorizing a diagnosis have also helped them score better? Again, very very minor point I think and your point still stands for sure.
Thank you for such a high-quality comment. And of course, I agree. I do have a video on the Flynn Effect in IQ testing vs. USMLE Score Creep if you’re interested… it’s here:
ruclips.net/video/FHOCuKW8new/видео.htmlsi=OwZ8x2YXgkvf39tN
Agree also with your point about the bulimia question. I probably should have picked a more challenging question as an example - but I decided to go with the first question to avoid the ‘cherry picking’ concern.
Thank you for the explanation! I notice that there will be an update on the question pool. Do you know when they update the question pool?
Do you have any specific thoughts on the increasing focus of ethics, communications, and quality improvement on USMLE? I took step 2 recently and scored very well because these topics happened to be my strengths. Same with the MCAT and psych/soc sections when I was an undergraduate playing to my strengths. Is this by design in order to prioritize/reward a new set of soft skills deemed valuable in future physicians altogether, or just a further push to increase the difficulty of step 2 in an era of score creep. Do you have any specific thoughts in this change in focus away from hard science in general? This has been another topic about standardized testing I was curious about.
Does this apply to the MCAT as well?
I think its the rise in youtube videos. The information is easyer to get to
Students game the system. Tons of ANKI, a small fortune spent on Qbanks. But then the game games the system, and what you have is an utterly toxic hamster wheel of unhealthy toxicity. The profession seems to think that a) you must be hazed with misery, pressure ulcers, and crushing debt and b) ignores the fact that memorizing piles of information is largely an absurd goal when everyone has the internet in their pocket.
Don’t forget the recalls in overseas asian countries ! That plays a big role in increasing the mean of the score.
was the answer A 😂