On 5 August, biologists from the Washington Department of Fish and Wildlife ascended in a helicopter to shoot two members of the Profanity Peak wolf pack, the state removed four more members of the 11-wolf pack. Some conservationists were outraged, but the logic behind such lethal control seems airtight: Remove livestock-killing wolves, coyotes, bears, and other predators, and you’ll protect farmers and ranchers from future losses.

A new study, however, claims that much of the research underpinning that common sense notion is flawed—and that the science of predator control needs a methodological overhaul. Adrian Treves, a conservation biologist at the University of Wisconsin, Madison, and his colleagues examined more than 100 peer-reviewed studies, searching for ones that randomized some by removing or deterring predators while leaving others untouched. Not a single experiment in which predators were killed has ever successfully applied this randomized controlled design, they reported 1 September in Frontiers in Ecology and the Environment. “Lethal control methods need to be subjected to the same gold standard of science as anything else,” Treves says. He argues that policymakers should suspend predator management programs that aren’t backed by rigorous evidence.

David Mech, a wolf expert at the University of Minnesota (UM), Twin Cities, isn’t persuaded. He notes that many of the studies Treves scrutinized “met some pretty good scientific standards, but just weren’t quite perfect. … Drawing the conclusion that therefore all these depredation management programs should stop until gold standard studies are done—that’s a very big leap.”

Lethal control has long been a staple of wildlife management. Eurasian lynx have been culled by hunters in Norway, wolves killed in Spain and Sweden, jackals and caracals eliminated in South Africa. In the United States, predator control often falls to the federal APHIS Wildlife Services, a branch of the U.S. Department of Agriculture. In 2015, the agency killed 385 gray wolves, 284 mountain lions, and more than 68,000 coyotes. Unlike the Profanity Peak wolf pack, which wasn’t targeted until it began killing livestock, coyote populations in many states are subject to preemptive thinning.

Treves says he was inspired to look at the science behind predator control by a book unrelated to wild carnivores: The Emperor of All Maladies: A Biography of Cancer, Siddhartha Mukherjee’s epic history of cancer. As Treves paged through it, he says, “a light bulb went off in my head.” He suspected that predator management was plagued by the same methodological problems that had once led cancer researchers to promote ineffective cures—particularly a dearth of randomized controlled trials. Although removing carnivores to ease livestock loss makes intuitive sense, Treves and other scientists were skeptical: For instance, some research suggests that coyote populations subject to culling have higher pup survival rates, and that male cougars expand their ranges in response to hunting.

Lethal control methods need to be subjected to the same gold standard of science as anything else.

What Treves found when he and his co-authors—Miha Krofel, a wildlife researcher at Slovenia’s University of Ljubljana, and Jeannine McManus, a researcher at the Landmark Foundation in Riversdale, South Africa—delved into the literature confirmed his suspicions. Only 12 studies came close to Treves’s gold standard or even a lesser “silver” standard, in which livestock losses before and after predator management were monitored, or analyzed in retrospect. Many other studies had flaws that he says make it impossible to draw reliable inferences. A 1999 experiment purporting to show the effectiveness of shooting coyotes from helicopters, for instance, had a higher density of sheep in its control pastures, which could have made them more attractive to hungry coyotes. Others failed to properly randomize intervention and control sites or described their methods inadequately, making replication impossible.

“There are so many ways that these studies could have been improved,” says Robert Crabtree, a carnivore ecologist and founder of the Yellowstone Ecological Research Center in Bozeman, Montana. “Not by spending more money, but by paying careful attention to standardization protocols, transparency, and replication.”

Some of the authors whose studies Treves critiques object to his analysis. Wildlife Services representative Gail Keirn said via email that Treves’s calculation of sheep densities in the 1999 aerial gunning study was “based on incomplete information.” Treves also discounted a 2008 study suggesting that trapping male wolves reduces livestock loss, for instance, in part because it excluded certain data points. But lead author Elizabeth Harper, then at UM, says that the paper thoroughly explains why each data point was excluded, and that the omissions kept misleading data from contaminating the results. Harper adds that she isn’t convinced Treves’s own study lives up to his standards: The authors assessed the validity of studies themselves, rather than asking independent experts. “That could create their own bias,” Harper says.

Others say that Treves is setting an impractically high bar. The complexity of field biology precludes most gold standard experiments, argues Adrian Wydeven, Timber Wolf Alliance coordinator at Northland College in Ashland, Wisconsin. Scientists face countless variables, including subtle differences in habitat, weather, and the unpredictable movements of animals themselves. “I just don’t see that it’s an attainable standard—it’s not like being in the lab,” Wydeven says. Such research also relies on the cooperation of farmers and ranchers, who may not be eager to take part in a randomized, controlled study. When wolves are at the door, who wants their flock to be one that doesn’t get help?

Treves counters that two of the studies he and his colleagues analyzed did meet the standard. Both evaluated nonlethal predator deterrence techniques: guard dogs and strings of flapping red flags that scare off wolves and coyotes. Performed by Tom Gehring, a biologist at Central Michigan University in Mount Pleasant, they showed that wolves and coyotes both steered clear of cattle farms patrolled by Great Pyrenees dogs, and that the flags deterred wolves, but not coyotes.

To prepare for the studies, done on Michigan’s Upper Peninsula, Gehring combed through data on the ranges of local wolf packs to identify vulnerable farms, then visited each operation to secure its commitment. He paired farms based on attributes like size and location and randomly assigned one to the treatment group and one to the control group. “It was an ordeal,” Gehring acknowledges. “It took months.”

In the end, he claims, the rigorous design was well worth the trouble. Before his experiments, he says, many ranchers and biologists were skeptical that guard dogs could protect stock against wolves. By the end, though, the ranchers who had been assigned to the control group were clamoring for dogs of their own. “You don’t hear anybody question that guard dogs work in Michigan anymore,” Gehring says.