ONE 46-YEAR-OLD WOMAN WROTE about how grateful she was that her annual mammogram had detected a tiny lesion in her breast, before it had time to spread. To treat the lesion, known as a ductal carcinoma in situ, or DCIS, she had the lump removed and then had dozens of radiation therapy treatments to try to erase all trace of the cancer. And to keep it from coming back, she’ll take the hormonal therapy drug tamoxifen for the next five years.

A man, 64, had his own screening story to tell. His cancer was found through a blood test showing a high level of prostate-specific antigen, or PSA. His doctor followed up with a biopsy of a tissue sample from his prostate that confirmed he had cancer. Then the man did what many people do in that situation: He had the prostate gland and surrounding tissue removed.

These are real patients and real diagnoses, and the websites of cancer advocacy groups and hospitals are full of such survivor stories. But like very many others who have had similar experiences, these two may not have had life-threatening diseases at all. About a quarter of tumors detected in women age 40 to 50 are DCIS—also known as stage 0 cancer—and the vast majority of those lesions won’t cause any harm. Treating them, however, may bring a cascade of negative consequences. While early detection of cancer through mammography might logically reduce the prevalence of mastectomies, studies show that widespread screening has increased the number of women who opt to have their breasts removed to reduce the chance that cancer will spread. Chemotherapy can lead to heart failure, and there’s a risk that radiation or chemotherapy will cause new cancers. And there is huge emotional stress associated with false positive test results or being labeled a cancer patient.

And even though, to judge by autopsy studies, at age 64 perhaps half of all men have prostate cancer, a very small proportion of those cases are serious. In the United States, men have a 3% chance of dying of prostate cancer—but a 16% chance of being diagnosed with the disease. And if they choose to have their prostates removed? About half of men who opt for surgery will become impotent and many become incontinent; radiation may produce those side effects or cause bowel problems.

For decades, screening has been a bedrock principle of preventive medicine, aiming to head off trouble before symptoms appear, and patients are routinely screened not just for cancer but also to detect elevated cholesterol and blood pressure levels and many other potential problems. Yet in recent years, large-scale studies have shown little or no benefit from many tests—and a high potential for overdiagnosis and unnecessary, often dangerous, treatment.

Screening recommendations from the U.S. Preventive Services Task Force (USPSTF) and other groups have been scaled back for breast and prostate cancer, among other diseases. Even the American Cancer Society, long a champion of early detection, now is more cautious. “I’m very concerned that people think screening is better than it actually is for every cancer I can think of,” says Otis Brawley, chief medical officer for the ACS.

And the line between a positive screening test and the next steps of treatment is necessarily thin. When screening leads to a biopsy, “we can make a very rough guesstimate of that cancer’s future behavior by looking at the tumor cells, the size and stage of the tumor and, in the case of prostate cancer, the Gleason score—how angry the cells look,” says Barnett Kramer, director of the division of cancer prevention at the National Cancer Institute. “But we’re not much beyond square one in determining which cancers will never cause any problems. So if you find cancer, most physicians and patients feel pushed to treat it.”

At the same time, not finding that cancer could have its own dire consequences for patients—and their physicians. “Doctors must deal with the crushing, demoralizing impact of realizing that an incurable cancer could have been prevented if it had been detected early with screening,” says James Thrall, chairman emeritus of the department of radiology at Massachusetts General Hospital. “Statistically, it’s true that many screening-detected cancers won’t be those that will kill you. But no one can tell whether a particular cancer in your individual patient will be one of those.”

And that argues for screening and early detection. “That’s critical for curing cancer before it spreads, and it applies to prostate, breast, lung and many other cancers,” says Daniel Haber, director of the Cancer Center at MGH. “There are many men who had a prostatectomy for an early invasive prostate cancer picked up by PSA who rightly feel that it saved their lives. And our clinics are filled with men with advanced prostate cancer who wish that it had been detected earlier.”

That’s the conundrum of disease screening—how to maximize its benefits in helping head off potentially deadly illnesses while minimizing the harm it can cause. And as they consider testing for conditions ranging from cancer and heart disease to thyroid problems and osteoporosis, physicians and patients are weighing research against clinical judgment to find the right balance.

HOW DO YOU SCREEN a screening test? The preferred tool is a randomized controlled trial in which one group of subjects receives the screening and the other does not. Both sets of patients are followed for a number of years, and researchers compare the results. Absent this, researchers turn to population studies that look for a spike in cancer incidence in people who don’t get screened or mathematical models that extrapolate the overdiagnosis and mortality rates of short-term screening studies. Depending on the size of the population being studied and the quality of the research, rates of overdiagnosis and numbers of lives saved through screening can vary widely, making it difficult to assess any individual’s particular risks of being hurt or helped by a specific test.

Despite those difficulties, however, consensus has grown about which patients should receive what tests. Three kinds of screening—for prostate, breast and lung cancer—illustrate the range of those opinions.

Prostate cancer screening has become emblematic of the harm screening tests can do and was one of the first tests to cast doubt on aggressive efforts to find cancer at the earliest possible time. Largely because of PSA testing, diagnoses of prostate cancer doubled between 1986 and 1992, and nine out of 10 men who learn they have cancer opt for either surgery or radiation therapy.

Yet those treatments may do little to decrease the chance of dying from prostate cancer. The Prostate Cancer Intervention Versus Observation Trial compared mortality in men who had surgery for prostate cancers, many detected by PSA testing, to those who did not. After 12 years, the surgery group had just 3% fewer deaths from prostate cancer than the control group, says Timothy Wilt, professor of medicine at the University of Minnesota School of Medicine, who was the trial’s principal investigator. (Wilt notes that men with higher-risk cancers may have a greater benefit from surgery, and that other studies have shown that younger men whose cancers were detected when a doctor did a rectal exam also were helped by surgery.)

In 2012, following publication of two large-scale randomized trials that showed little to no reduction in cancer mortality from PSA testing, the USPSTF concluded that, at most, one man out of 1,000 screened with PSA will avoid dying of prostate cancer. But of those 1,000 men, 30 to 40 will become impotent or incontinent after treatment, two will have a heart attack or another cardiovascular event, and one will develop a serious blood clot. And overall, one-third to two-thirds of men diagnosed with prostate cancer detected through PSA screening will be labeled with a cancer that won’t shorten their life spans.

Because of PSA screening’s frequent harm and at most small benefit, few organizations today recommend the routine use of the procedure, and many say it should only be used when a well-informed man in good health requests it, says Wilt. And while specialty societies representing doctors in the trenches of patient care often support screening, the latest guidelines from the American Urological Association advise against routine PSA screening in men 40 to 54 at average risk and urge “shared decision-making” between doctor and patient for those 55 to 69, when screening for prostate cancer has the biggest benefit.

“What is really needed are better screening tests,” says Haber. “PSA testing is very sensitive, but not specific enough for prostate cancer, because it measures a protein shed by both normal and cancerous cells,” adds Haber, who believes that the next few years may bring major advances in early cancer detection.

sp15_screen_spot1_630x420

WHEN MAMMOGRAPHY was introduced in the 1950s, it appeared to be a lifesaver. A pivotal 1963 study randomly assigned more than 60,000 women to get an annual mammogram and breast exam or no screening, and found that the women in the former group reduced their risk of dying of breast cancer to 30% after 10 years, with no substantial incidence of overdiagnosis.

From those seeds, mammography grew into an annual obligation for millions of women. It seemed an effective defense against a devastating disease. But improvements in the technology meant that it found smaller and smaller lesions, many of which weren’t dangerous.

In 2009, the USPSTF announced that evidence didn’t support the value of annual screening mammograms for women in their forties. That was a shocking change in advice, and the American College of Radiology, American College of Obstetricians and Gynecologists, and the American Medical Association all argued against raising the age for a first mammogram to 50. Yet in making its new recommendation—that most women should begin getting mammograms at 50 and then be tested every other year—the USPSTF committee had ample evidence that earlier, more frequent mammography wasn’t desirable. Since 1963, there had been nine major randomized controlled trials involving some 500,000 women. Based on those studies, the USPSTF estimated that nearly 2,000 women age 39 to 49 would have to get mammograms for 10 years to avert a single death from breast cancer. For women in their fifties, 1,339 would have to be tested to prevent that one death. But for women in their sixties, that number dropped to 377.

Not included in the USPSTF’s 2009 analysis were the results of a landmark Canadian study that followed nearly 90,000 women for 25 years. The trial, published in 2014, was significant for its long follow-up and because it was the only randomized trial of mammography done in the context of modern treatment for breast cancer, which has become increasingly effective even when tumors aren’t found and treated at the earliest stages of disease.

It found that just as many women died of breast cancer in the groups that got the clinical exams and mammography as those who weren’t tested, with or without exams. Over 25 years, 3,250 women who had mammography developed cancer and 500 of those women died. In the control arm, 3,133 were diagnosed with breast cancer and 505 of them died. The researchers concluded that annual mammography in women 40 to 59 doesn’t prevent deaths from breast cancer “when adjuvant therapy [treatment after surgery, such as chemotherapy or radiation] for breast cancer is freely available.” The trial also found that 22% of the mammography-detected breast cancers were overdiagnosed.

The point about adjuvant therapy is important. Medicine has gotten a lot better at treating breast cancer, and as treatments evolve, screening protocols should too. “Adjuvant therapy can blunt the need for earlier and earlier detection because it’s effective for middle-stage breast cancers,” says Barnett Kramer of the National Cancer Institute. “There has been a rapid fall in breast cancer mortality in women under 40, and it’s hard to attribute that to anything but better treatment, because we don’t generally screen these women.”

Still, unlike screening for prostate cancer, mammography does reduce the risk of death from breast cancer, especially for older women who are most likely to have the disease. “If you look at all the randomized trials of mammography aggregated, there is a 10% to 15% reduction in breast cancer mortality for women 50 and above,” says Kramer. Brawley of the American Cancer Society says the reduction may be closer to 35% for women in their sixties, when breast cancer risk is near its highest.

THE DRIVE TO PREVENT deaths from lung cancer could hardly be greater. The disease will kill almost 160,000 Americans this year—more than breast, colon and pancreatic cancers combined. And in this case, using imaging tests to screen those at highest risk does appear to save lives. So in 2013, the USPSTF began recommending that heavy smokers, ages 55 to 80, get screened annually with low-dose computerized tomography, or LDCT.

Other groups followed suit with similar recommendations after the results of the National Lung Screening Trial were published in 2011. That trial enrolled more than 53,000 current and former heavy smokers to receive either LDCT scans or chest X-rays to screen for lung cancer. Participants were tested for three years in both arms of the trial, and seven years from the start the apparent benefit of LDCT screening was so pronounced that researchers halted the study and published the results. Smokers who had gotten the scans saw their risk of dying from lung cancer reduced by 20% compared with smokers receiving chest X-rays.

According to a USPSTF projection, for every 100,000 people screened, 497 lung cancer deaths would be averted—representing a 14% decrease in lung cancer mortality. Two-thirds of those tested would have false-positive results, however, and there would be 910 biopsies or surgeries for benign lesions, and 190 cases of lung cancer would be overdiagnosed.

At first this seemed to represent a clear victory for screening, because it saved a significant number of lives. Yet there are serious caveats. A committee that advises Medicare on what medical procedures and tests it should cover voted overwhelmingly that it wasn’t confident that the benefit of lung cancer screening outweighed the harm. The committee also took issue with basing the USPSTF recommendation largely on the results of one trial. (The USPSTF guideline for screening does, however, mean that private insurance companies must pay for the test, and Medicare subsequently decided to cover it too.)

There are also concerns about how lung cancer screening will be carried out. The trial was conducted at academic medical centers and followed a strict protocol of waiting six months to re-scan small lesions detected on the initial screening before determining whether a biopsy was needed. “Most people will get screened for lung cancer at community hospitals, where doctors may not follow the research protocol from the trial,” says Reid Blackwelder, board chair of the American Academy of Family Physicians. And if physicians don’t wait six months but rather act on the first results, “people who don’t have cancer will get bronchoscopy and surgery, both of which carry significant possible consequences, including death,” Blackwelder says.

And the accuracy of the screening test will depend on the skill of the radiologists who interpret it. A 2014 study of 300 patients evaluated for lung nodules by radiologists at 15 Veterans Affairs hospitals found that of the 197 whose nodules were detected after guidelines were issued by an international society for thoracic radiology, 45% received care inconsistent with the recommendations. Screening of all patients led to 1,044 imaging studies, 76 biopsies, 13 surgeries and 21 hospitalizations—all to diagnose cancer in 27 of the 300 patients.

THE PROBLEM OF overdiagnosis through screening—detecting abnormalities that aren’t serious problems—isn’t limited to cancer. According to Rita Redberg, professor of medicine at the University of California, San Francisco School of Medicine and editor of JAMA Internal Medicine, the recommended thresholds for identifying diabetes, high blood pressure, high cholesterol and osteoporosis have dropped so low that millions of people are being treated for the mildest of disorders. “We’ve invented all sorts of pseudo-diseases like pre-hypertension, prediabetes and metabolic syndrome,” Redberg says. People with mild abnormalities are being treated and may be harmed from side effects of drugs they don’t need to take, she says.

Recently, the American College of Physicians advised doctors not to screen adults at low risk for coronary heart disease with common cardiac screening tests—electrocardiograms, stress tests or myocardial perfusion imaging—citing the tests’ potential harm and lack of benefits. And a new study in the International Journal of Epidemiology by John P. A. Ioannidis, noted epidemiologist and chair of disease prevention at Stanford University, concurs that screening for early disease in asymptomatic people doesn’t save many lives. His analysis of the results of randomized trials on 19 screening tests for 11 major diseases found that few screening tests reduced mortality from the diseases they were designed to prevent, and almost none prolonged life.

Still, the answer to overdiagnosis and excessive treatment isn’t to halt screening but rather to be more selective about who is screened and how often they’re tested, suggests Wilt of the University of Minnesota. People at high risk, such as women with BRCA1 or BRCA2 gene mutations that predispose them to breast cancer or those who have a strong family history of cancer, clearly need to be screened early and often for signs of disease. But for those who are less prone to develop cancer or another malady, “to screen or not to screen” involves carefully weighing the harm associated with a test against the potential benefit of finding a problem.

Too often, suggests Redberg, the pros and cons of screening and its consequences aren’t available or understood. When a blood test 15 years ago led to a thyroid cancer diagnosis for her, there weren’t yet studies showing that to be one of the most frequently overdiagnosed cancers. So she had her thyroid removed. “It didn’t occur to me that I had a choice,” she says. “Today, I’m not sure I would have the surgery.”