Probing Journals

PHYSICIAN MICHAEL ROMAC RUNS A BUSY FAMILY PRACTICE in Hoosick Falls, N.Y., a small village near the Vermont border. Romac sees a wide variety of patients, from newborns to the elderly. “It can be a challenge to keep current in all aspects of medicine,” Romac says. Toward that goal, Romac spends a few hours each week reading American Family Physician and The New England Journal of Medicine. He likes the former for its review articles, which summarize findings from studies on common conditions. And Romac considers NEJM a must-read for the latest, most important research. That Romac has consistently scored among the top 10% of his peers on the American Board of Family Medicine’s certification exam—a routine test of a family doctor’s comprehensive medical knowledge—suggests that his method for keeping tabs on new developments in medicine is working. Yet using medical journals to stay current can be maddening when the messages are inconsistent, he observes. “One day it is ‘Take statins to lower cholesterol,’” Romac says, “but now it is ‘What the hell are these statins doing to our muscles and memory?’”

Many doctors find themselves in the same boat. For them, scientific journals are an indispensable part of the medical landscape, yet also a frequent source of frustration. The researchers who provide these periodicals’ primary content express similar discontent. Many feel that this means of publishing their work is inefficient and often unfair. The process editors use to review submissions can be long and tortuous, and it may favor certain kinds of studies—and suppress negative results or findings that undercut long-established medical wisdom.

Criticism of medical journals is hardly new. When the editors of the Chicago Medical Journal and Examiner surveyed the rapidly expanding universe of biomedical publications available in 1879, they offered a scathing assessment. “Some are absolutely worthless,” they wrote, “and a few are undeniably worse than worthless—they are dangerous and disgusting parasites upon the body medical.”

While it’s unlikely that many doctors or scientists would employ such tart rhetoric today, recent research on the state of journals suggests that the enterprise continues to have inherent problems—ranging from what to publish, and in what kinds of media, to how to fund journals. Those are largely the same challenges that have plagued medical publishing for more than a century, says medical historian Scott H. Podolsky, a primary care physician at Massachusetts General Hospital who wrote about the evolving medical journal earlier this year in NEJM. Podolsky feels it has never been more relevant to ask, “Is the present journal system the ideal one, moving forward?”

A growing chorus of voices is saying maybe not. During the past decade or so, a fresh philosophy and approach to sharing new medical knowledge has gained a foothold. “Open access” publications, presented online, get studies into “print” relatively quickly, publish a broader range of research and eliminate high subscription fees. That model is putting pressure on traditional journals, and its supporters believe it’s the key to producing a better medical journal. But many others aren’t so sure.

THE FIRST ENGLISH-LANGUAGE GENERAL MEDICAL JOURNAL,  Medicina Curiosa, appeared in 1684. It vanished after two issues, yet the idea of disseminating scientific knowledge to doctors in periodical form eventually caught on. Today there are more than 5,600 journals represented in the National Library of Medicine’s database of journal citations, known as MEDLINE/PubMed. Three times a year, a library selection committee considers 180 or so new titles to add to the database.

Until recently, most of these journals operated like consumer magazines, earning revenue by selling advertising space and charging subscription fees ranging from less than $1,000 per year to $20,000 or more. Online journals also sell access to scientific papers and other material à la carte for up to $30 or more per article. But medical journals don’t pay writers. Instead, scientists submit manuscripts for free in hopes of having their research findings distributed to an interested audience.

Journal editors read manuscripts and send the papers that seem promising to peer reviewers, who critique the works’ scientific quality and relevance. Peer reviewers, who are usually unpaid, typically are doctors and scientists from the same fields as the authors of the manuscripts they read. Although journal editors make the final judgment about a manuscript’s fate, the comments of peer reviewers are highly influential.

High-prestige journals publish only a fraction of the manuscripts they receive. The Journal of the American Medical Association (JAMA) receives more than 6,500 submissions each year, but fewer than 5% make the cut. Having a paper accepted by a prestigious journal impresses academic review boards, which can help a scientist obtain a promotion, tenure or a research grant. When a highly selective journal such as  NEJM, JAMA or Cell rejects a paper, a researcher may revise the manuscript and resubmit it. Or the scientist will send the manuscript off to often less august publications until it’s accepted.

Guardians of medical journal publishing believe it has a clear mandate. “Journals are an important conduit for scientific information—from those who generate it, comment on it and review it to those who use it,” says Howard Bauchner, JAMA’s editor in chief. He considers his audience to be not only scientists but also clinicians, policymakers and patients. Journal editors see themselves as gatekeepers. “Our most important job is vetting information,” says Jeffrey Drazen, editor in chief of NEJM. “That’s what you pay us to do.”

But grumbling about journals is a venerable pastime in hospital corridors. One of the oldest complaints is that journals cost too much. “We would need to increase our journal budget 10% annually just to maintain the status quo,” says Elizabeth Schneider, director of Massachusetts General Hospital’s Treadwell Library. And Isaac Kohane, director of Harvard Medical School’s Countway Library of Medicine, says that because of high costs, “we have never subscribed to a smaller percentage of the medical literature than we do today.”

ANOTHER ENDURING COMPLAINT: It takes too long to get an article published. “The review processes have gotten so long and cumbersome that as a policy researcher you may feel that by the time the paper is out there, it’s too late to have the full impact,” says Karen Donelan, a senior scientist at MGH’s Mongan Institute for Health Policy. At some journals, peer review can take months, says Donelan, and it may be a year or more before a paper appears in print.

Peer review’s defenders frequently paraphrase Winston Churchill’s famous quote about democracy, arguing that peer review is the worst system known for identifying high-quality studies—except all the others that have been tried. “The peer review process has lots of flaws, and yet I wouldn’t think of editing a major medical journal without it,” says former JAMA editor George Lundberg. He argues that many mediocre manuscripts have been greatly improved by reviewers’ comments.

Yet others believe that an “old boy network” dominates peer review and can stymie innovation. “Papers that challenge paradigms often go to reviewers who established the paradigms in the first place,” says University of Colorado nephrologist Richard J. Johnson, who has published more than 500 papers in his specialty, “and they’re not eager to have their work challenged.” Relying on a relatively small number of judgments to determine whether a paper is worthy may be problematic too. In a 2008 study, 607 reviewers for the British Medical Journal (BMJ) were sent previously published scientific papers that had been rewritten to include 14 deliberate errors (such as obvious bias in randomization of study subjects and unjustified conclusions). The typical reviewer identified just three errors.

Romac’s complaint about the inconsistency of journals’ conclusions also has scientific support. A 2005 paper in JAMA found that the results of about one-third of highly cited clinical research studies are later contradicted by subsequent studies. Journal editors say that’s simply part of the bumpy road science travels. “It’s rare that an article is published and should change practice overnight,” says Bauchner.

Meanwhile, the number of scientific articles that journals are forced to retract each year has risen tenfold over the past decade, even as the medical literature has increased by only 44%. That has done little to bolster confidence in journal content. “There’s less and less trust,” says Ivan Oransky, a physician, journalist and co-founder of Retraction Watch, a Website that tracks and critiques science journals’ handling of studies that are withdrawn because of errors, fraud or other wrongdoing. High-profile cases such as The Lancet’s publication of a now-discredited paper linking childhood vaccines to autism have left many readers questioning what they read in journals, says Oransky, whose chief complaint is the lack of transparency with which many journals handle retractions. “We’ve had cases where it was completely clear, based on investigation by the institution, that there was fraud or misconduct involved. Yet we see editors finding every reason not to retract. Instead, they call it an ‘erratum’—they downgrade it,’ he says.

Several recent studies have taken journals to task for what they miss. A 2009 analysis found that a study with a positive outcome is nearly four times more likely to be selected for publication than one with a negative or not statistically significant result. A 2008 study of antidepressants found that negative trials were either not published or “spun” to make them look positive, giving the impression that nearly all the trials were positive. Sometimes pharmaceutical companies pressure investigators to suppress the results of negative trials. But “we can’t let journals off the hook entirely,” says Oregon Health & Science University psychiatrist Erick Turner, lead author of the antidepressant study. Editors may favor positive studies because they’re more likely to grab headlines. “They want people saying, ‘Did you see that article in The Lancet?’” says Turner.

Meanwhile, do medical journals accomplish the primary goal of keeping doctors well informed about medicine? “When you ask doctors what they do to keep up, they say they read journals,” says McMaster University professor R. Brian Haynes, who studies the ways medical knowledge is delivered and used. “For that purpose, journals today don’t do a very good job at all. Most information in them is really scientist-to-scientist communication and not ready for clinical consumption.”

IF TRADITIONAL JOURNALS FALL SHORT, could greater openness be the solution? In recent years, hundreds of new-style medical periodicals, known as open access journals, have sprung up online. Instead of collecting fees from subscribers, OA journals charge authors. Fees vary but can run as high as $3,000 or more per paper. The number of OA journals produced around the world has increased by about 15% each year since 2000.

The publishers and editors of these journals insist that the new model is good for medicine. Virginia Barbour, editor of PLOS Medicine, one of several OA journals published by the Public Library of Science, a nonprofit open access pioneer established in 2000, says that free access encourages scientists to read journals outside of their disciplines, which can aid and inspire their research. “One thing we hear time and time again is, ‘I never would have read that article if it was in a subscription journal,’” says Barbour.

OA might also help reduce bias in selecting papers for publication. While some OA journals reject 80% or more of the manuscripts they receive, others are less selective. For example, in 2010 the OA journal PLOS ONE published 6,749 papers, or roughly 34 times more than NEJM publishes in a year. Though that lack of selectivity has been criticized, OA journal proponents contend that it’s valuable to publish a broad spectrum of papers that present a fuller picture of what’s happening in biomedical research. “If an article is scientifically sound, regardless of whether the results are negative or perceived to be exciting or not, there is freedom for the information to be published,” says Iain Hrynaszkiewicz, who oversees many of the OA journals of BioMed Central (BMC).

With the advent of digital publishing, times to publication have shrunk rapidly, and OA journals may stand to benefit. F1000 Research promises to publish studies within a couple of days of submission by skipping the traditional prepublication peer review process. First, it conducts an “initial sanity check” to ensure that a manuscript is scientifically sound. Then several experts are invited to perform post-publication peer review, which will be open—that is, reviewers’ names will be made public. “We think that will mean less bias, more balance,” says Rebecca Lawrence, F1000 Research’s publisher. Registered users on the journal site can add comments, and authors will be encouraged to revise and update papers based on commentary.

Embracing post-publication peer review could promote more accurate, trustworthy science. “Peer review can be useful throughout the life of a paper, not just when it’s submitted and before it’s published,” says Oransky, who speculates that one reason for the recent increase in retractions may be that publishing scientific studies on the Internet promotes closer scrutiny. Oransky notes that the rise of blogs and social networks has ramped up the volume of post-publication commentary on some controversial studies. He cites one example in which scientists and bloggers attacked the credibility of a 2010 paper in Science describing the discovery of a form of bacterium that builds its DNA backbone using arsenic, unlike all other living things, which use phosphorus. The online debate that ensued over this and other controversial recent papers has been good for medicine, says Oransky. “Someone trashed your paper. You responded. It’s all out there for the public to view,” he says. “All of a sudden, it’s a dialogue. That’s how science should work.”

THE NATIONAL INSTITUTES OF HEALTH—the world’s leading supporter of biomedical research—now requires that papers generated with its funding be published in OA journals or subscription journals that allow for free public access to studies within one year of publication. The United Kingdom’s Wellcome Trust, a charitable organization that funds about $1 billion in biomedical research around the world each year, has a similar policy.

Still, it’s not hard to find skepticism about OA. One problem is the “author pays” concept. A review of studies examining scientists’ attitudes published last year in the Journal of the Medical Library Association suggested that “publication fees are perhaps the greatest impediment to broader participation in open access initiatives.” Moreover, some skeptics suggest that charging authors creates a conflict of interest that could result in a lot of poor-quality research getting published. “The more papers a journal accepts, the more revenue it gets,” says Jeffrey Beall, a librarian at the University of Colorado Denver. “A lot of these publishers are basically operating as scholarly vanity presses.” While Beall believes OA publishers such as PLOS and BMC operate ethically, he says many others send e-mail “spam” to solicit articles from researchers, often failing to mention publication fees until an article is accepted.

And while creating a freely accessible, all-inclusive repository of research may be a laudable goal, few OA journals have thus far had a major impact on medical practice. Publishers such as PLOS and BMC have garnered prestige by attracting submissions from top researchers, but studies found in most OA journals are “a much lower class of article” that have often already been rejected by traditional journals, says Haynes of McMaster University. Haynes is editor of ACP Journal Club, a monthly section of Annals of Internal Medicine that provides summaries of important recent scientific findings for clinicians.

Meanwhile, the line between OA and subscription journals has begun to blur. Some subscription journals offer free access to selected articles, while others (such as JAMA) no longer charge to view an article after a set period, usually 6 to 12 months. BMJ has morphed into a kind of hybrid; it has an open access policy for all original research articles, whereas a reader must pay to read reviews, commentary and other types of material. What’s more, some subscription journals allow researchers to pay a fee (usually $2,000 to $3,000) that removes the “pay wall” from a published study, making it freely available. And in 2008, a major subscription-based scientific publisher, Springer, purchased BMC, the largest publisher of OA journals.

Whether this transition of the scientific journal will result in a product that improves medical practice and patient care may not be known for some time. “We’re still in the midst of this revolution,” says JAMA’s Bauchner, “and it’s still uncertain how it will evolve.”