THE NEXT FEW MONTHS MAY BRING A SEISMIC SHIFT to a medical school milestone, the United States Medical Licensing Exam (USMLE) Step 1. Aspiring doctors typically take it after their second year of medical school, and a good score paves the way for them to become residents and eventually practicing physicians. But critics say the test has become a distraction, an all-or-nothing filter that doesn’t test for many of the qualities that make a physician great.

When the test was introduced in the early 1990s, its purpose was straightforward. The grueling eight-hour, multiple-choice exam provided an objective measure of a student’s command of two years of medical school, which largely focused on lecture and textbook materials. The score gave a clear answer to whether a student had mastered information that was vital to working as a doctor. State licensing boards require medical students to pass four USMLE exams in total.

“It was a national test that any board would accept, so rigorous that anyone who could pass it should be allowed to practice,” says Bryan Carmody, a prominent critic and pediatric nephrologist at the Children’s Hospital of the King’s Daughters in Norfolk, Va.

But Step 1 has now taken on an outsized significance, critics say. Residency program directors use high Step 1 scores to screen for the best applicants, and because a residency can determine the entire career trajectory of a student, getting the highest possible three-digit score on the Step 1 has become a high-stakes goal. Students now invest huge reserves of time and money in test preparation—an annual frenzy that Carmody calls “Step 1 Mania.”

During a recent webinar about the need for changes to the exam, physician Alison Whelan, chief medical education officer of the Association of American Medical Colleges, said: “Students see those Step 1 scores as being the number one thing for getting into residency. It puts much more pressure on them, potentially reducing their focus on other competencies they’re being taught in medical school.”

Critics blast the Step 1 test for rewarding students who can memorize trivia and minutiae, rather than identifying skills that predict how well they’ll practice medicine with human patients. There is a lack of direct evidence connecting Step 1 scores to better practice, they say, and what few studies have been done show little to no correlation.

One proposal, supported by Carmody and a chorus of others, is to make Step 1 pass/fail. “If your goal is to identify people who have a certain amount of knowledge to be licensed, the USMLE performs adequately,” he says. Beyond that, he says he believes the test is a poor predictive tool for medical specialties. “What makes a good neurologist may be different than what makes a good pediatrician,” he says, adding that while ne may need to know how to read an EEG, the other may need to be able to do lumbar punctures on infants. “Yet we assess all those people with this one score.”

That pass/fail plan has gotten widespread pushback, however. Some of it comes from residency directors who review dozens, if not hundreds, of applications for each available spot. In 2018 each U.S. medical student applied to an average of 60 programs, and international medical students applied to an average of nearly 140; and in this landscape of increased competitiveness, administrators find some form of objective ranking invaluable.

Some students are also against the idea of pass/fail. International students and those who study at less prestigious medical schools, for example, say that USMLE Step 1 is an equalizer. Without the score, they fear that the best spots will automatically go to those from the best schools, irrespective of individual talent. But many in this camp also concede that there could be a better way to determine residency than the death race that USMLE Step 1 has become.

In March, representatives of medical schools, medical associations and state licensing, as well as residency program directors, practicing physicians, recent examinees and the two organizations that jointly administer the USMLE—the National Board of Medical Examiners and the Federation of State Medical Boards—met to discuss numeric score reporting in undergraduate and graduate medical education. The conference culminated with a report and preliminary recommendations to the USMLE program, including conducting more research on the link between exam scores and residency and clinical practice performance, as well as reducing the racial demographic disparities in scores. In addition, it was recommended that the USMLE program consider changes to its current numeric score reporting practice, including a shift to pass/fail scoring. The public comment period following the report’s publication closed in July.

Thousands of people responded, according to physician Michael Barone, vice president for licensureof the NBME, and David Johnson, senior vice president of assessment services at the FSMB. Reviewing the feedback, they say, will take weeks, if not months. Any decisions approved by the boards of the NBME and FSMB will likely be announced this winter.

For his part, Carmody says he would like to see a better way to evaluate future residents and a realistic way to implement a new system. But he’s not optimistic that the USMLE will make major changes, and he considers the current review of the test a lost chance. “I think there was more of an opportunity for leadership there than we have seen,” he says.