By Dylan Matthews, Published: September 11 at 3:12 pm (Washington Post)

Counterterrorism may be the most significant area of government policy where we still have no idea what the hell we’re doing.

Everywhere else, policymakers are at least trying to know what they’re doing. Development researchers and education wonks have become obsessive about running randomized controlled trials to evaluate interventions. Indeed, the popularity of charter schools is due in part to the fact that their frequent use of lottery-based admission makes them good ways to randomly test different school designs. Criminologists have run experiments on a variety of police tactics, probation designs,anti-gang initiatives, approaches to domestic violence, and more. And While there’s stillplenty we don’t know about what health measures work, the Affordable Care Act isdevoting millions to building up more evidence, and big-deal health policy experiments like the Oregon Medical Study receive the attention they deserve.

But terrorism? We have no idea. The Afghanistan war has cost $657.5 billion so far, we spend $17.2 billion in classified funds a year fighting terrorism through the intelligence community, and the Department of Homeland Security spent another $47.4 billion last year. And we have very little idea whether any of it is preventing terrorist attacks.

Some of this is just that it’s harder to collect good evidence than it is in other policy areas. You can’t randomly select some airports to have security screenings and some to not and measure how many hijacking occur at the ones with or without them — or, at least, you can’t do that and conform to anything remotely resembling research ethics. But merely because true experiments are often impossible doesn’t mean that you can’t evaluate policy interventions using other means.

And people have tried those experiments. It’s just that nothing seems to have any significant effect one way or another. The Campbell Collaboration, an organization that publishes peer-reviewed systematic reviews of the evidence on various policy topics, first released its review of the literature on counterterrorism, written by criminologists Cynthia Lum (George Mason),  Leslie Kennedy and Alison Sherley (both at Rutgers), in 2006 (it’s been updated since).

The first problem the review identifies is that barely any of the terrorism literature even tries to answer questions about effective counterterrorism. “Of the over 20,000 reports regarding terrorism that we located,” the authors write, “only about 1.5 percent of this massive literature even remotely discussed the idea that an evaluation had been conducted of counter-terrorism strategies.”

They found 354 studies that did, however. Further culling left them 80 studies that could be reasonably said to evaluate the effectiveness of counterterrorism measures. Of these, only 21 of those 80 studies “appeared to at least attempt to connect an outcome or effect with a program through a minimally rigorous scientific test.” Of those 21, only 10 met the Campbell review’s methodological standards. Three of those were medical studies dealing with the effects of bioterrorism, leaving seven for the review to consider.

It’s worth dwelling on that number. In 2009, eight years after 9/11, and after decades of work on terrorist groups ranging from the IRA to ETA in Spain to Palestinian groups to the Tamil Tigers, only seven studies, or 0.035 percent of all terrorism studies, evaluated the effectiveness of counterterrorism measures. By comparison, a Campbell Systematic Review of anti-bullying programs in schools found 622 reports “concerned with interventions to prevent school bullying,” of which 89 were rigorous enough to include. Stopping bullying is vitally important and I don’t mean to trivialize that cause, but it’s more than a little concerning that we have almost 13 times as many studies on how to stop bullying as we do on how to stop terrorism.

Anyway, back to the seven measly studies. For one thing, they are mostly done by the same handful of people. Three were coauthored by Walter Enders (at the University of Alabama) and Todd Sandler (at University of Texas – Dallas), two by Enders and Sandler alone and the other one with Jon Cauley (at the University of Hawaii – Hilo). Cauley did another study with Eric Iksoon Im (also at Hilo). So over half of the studies included were coauthored by one of Enders, Sandler, or Cauley. They’re all excellent researchers, and one should not discount their work because of their higher output, but generally we want a range of studies from a range of sources when building a literature like this.

The seven studies include among them 86 findings about the effectiveness of counterterrorism programs, and those findings are startling. Lum, Kennedy and Sherley report that the average effect of the programs examined was negative. That is, the intervention was found to increase terrorist incidents rather than reduce them. The results varied by the type of intervention, but not in a way that should give us any comfort about our strategy:

• Metal detectors reduce hijackings, but terrorist just do other stuff instead.

The studies find that, on average, adding metal detectors and security screenings at airports leads to about 6.3 fewer airplane hijackings in the years examined. But they also find that those policies lead to significant increases in “miscellaneous bombings, armed attacks, hostage taking, and events which included death or wounded individuals (as opposed to non-casualty incidents) in both the short and long run.” In fact, metal detectors and security screenings at airports lead to about 6.9 more of these substitute events. “When calculating the overall weighted mean effect size for all of the findings examining the effectiveness of metal detectors, the positive and harmful effects cancel each other out,” the review’s authors conclude.

• Fortifying embassies and protecting diplomats doesn’t appear to reduce attacks.

Most of the results here are not statistically significant. “In total, the findings do not indicate that the fortification of embassies and efforts to protect diplomats have been effective in reducing terrorist attacks on these targets,” the review authors conclude. More on this issue here.

• There’s no evidence harsher penalties reduce hijackings.

Only one study looked at what increasing penalties for plane hijackers did to hijacking rates, and that one found no effect. That doesn’t mean that it doesn’t work, just that we shouldn’t reject our original assumption that it’s not effective.

• Strongly written letters from the U.N. don’t help much.

One U.N. resolution, which included a recommendation that airports use metal detectors, was associated with a significant reduction in hijackings. But that could just mean that the metal detectors, rather than the U.N. resolution, caused the reduction, and the same substitution issues explained above hold.

• A military reaction can backfire.

The 1986 U.S. bombing of Libya, intended to punish the regime for the bombing of the LaBelle Discotheque in West Berlin, lead to a statistically significant increase in attacks in the short term of around 15.33 incidents. The incidents tended to be less lethal and the effect doesn’t appear to be long-lasting, but still, that’s not the direction you want the dial moving in.

• Changing political regimes can hurt too.

A study of ETA attacks in Spain found that having the Socialist Party — which took a harder line on the Basque separatist group — in power lead to a statistically significant increase in attacks. Meanwhile, the end of the Cold War and fall of Communist bloc governments appears to have led to a significant increase as well. The policy takeaways here aren’t particularly clear (I will go out on a limb and say it’s still a good thing the Cold War ended) but it does rebut the idea that electing hardliners can help fight terrorism.

The Campbell review was last updated in 2009, so it’s worth looking around to see if the literature has produced any good evaluations since it came out. But what new studies we have don’t make our current counterterrorism posture look too promising. The most promising project is the Government Actions in Terror Environments (GATE) databasebeing compiled by the University of Denver’s Erica Chenoweth and the University of Maryland’s Laura Dugan.

The first study out of the project examines Israeli reactions to Palestinian attacks from 1987-2004, and finds that repressive actions are either ineffective or lead to a backlash, and the reconciliatory moves can be effective at preventing future attacks. That’s promising, but it’s just the start of what should be a much larger literature on the question of when reconciliation works. Chenoweth also conducted an evaluation of Spain’s tactics against ETA with Evan Perkoski. They concluded that discriminate, targeted arrests were highly effective, and especially so when combined with expanded security laws, like increased police powers and border agreements.

Case studies can provide some help, but are limited in what they can demonstrate. Evaluations of the U.S. government’s responses to the Earth Liberation Front and the Puerto Rican separatist group Fuerzas Armadas de Liberación Nacional Puertorriqueñaproduced some lessons, but fall short of the rigor of the studies in the Campbell review, and the lessons tend to be general, e.g. “think like a terrorist” or “be creative.”

Perhaps the most studied area since the Campbell review came out has been targeted killings of top terrorists. There, the evidence is somewhat mixed, but leans heavily toward finding that decapitation is ineffective or counterproductive. Matthews Dickenson looked at a dataset of attacks from 1970 to 2008 and found that leadership transition “generally causes a noticeable and statistically significant increase in attacks and casualties for the months immediately afterward.” Similarly, Jenna Jordan at Georgia Tech found that “Organizations that have not had their leaders removed are more likely to fall apart than those that have undergone a loss of leadership.”

Aaron Mannes at the University of Maryland also found decapitation strikes against groups to be ineffective, writing, “The most notable trend from the statistical analysis was that decapitation strikes on religious terrorist groups tended to be followed by sharp increases in fatalities.” Michigan’s Lisa Langdon, Alexander J. Sarapu and Matthew Wells failed to find significant effects of leadership changes, finding that “the arrest of the leader will not significantly alter the ideology or operations of the group in the long term.”

A few other studies found the opposite. Army Major Bryan Price found that decapitations increase the probability that a terrorist group will cease to be active, especially if it’s young and unprepared for leadership transitions. “In the first year of its existence, a terrorist group is 8.757 times more likely to end if its leader is killed or captured,” he writes. RAND’s Patrick Johnston found that decapitation strikes work in a counterinsurgency context, though his finding’s relevance in non-counterinsurgency efforts against militant groups may be limited.

So what do we know?

So the evidence base is getting better. The decapitation research and Dugan and Chenoweth’s work are real additions to the knowledge base on counterterrorism tactics. But there are a whole range of things we don’t know. Does limiting the size of liquid containers you can take on a plane reduce attacks? Does making people take their shoes off during their security screenings? Do drone strikes reduce the number of plots targeting U.S. citizens?

These are real, practical questions that deserve answers that only rigorous research can provide. It’s scandalous that we spend billions every year on counterterrorism but barely spend any effort on evaluating whether what we’re doing works. The federal government is showing slightly more interest than it once did. “We’re lucky because there’s a criminologist in DHS who helps the partnership along a bit,” Lum tells me. But the scale of the efforts pales in comparison the efforts to build evidence on health, education, social welfare, or crime policy. That has to change.

Dylan Matthews covers taxes, poverty, campaign finance, higher education, and all things data. He has also written for The New Republic, Salon, Slate, and The American Prospect.

FRIDE

¿El resurgimiento de al-Qaeda en el Norte de África? Por Anouar Boukhars (27/08/2013) Documento de trabajo