Connecting the Dots

March 22, 2018
Posted by Jay Livingston


Brilliance in science is sometimes a matter of simplifying – paring away complicated scientific techniques and seeing what non-scientists would see if they looked in the right place. That’s what Richard Feynman did when he dropped a rubber ring into a glass of ice water – a flash of brilliance that allowed everyone to understand what caused the space shuttle Challenger disaster.

Andrew Gelman isn’t Richard Feynman, but he did something similar in his blog post about an article that’s been getting much buzz, including at Buzzfeed, since it was posted at SSRN two weeks ago. The article is about Naloxone, the drug administered to people who have overdosed on heroin or opoiods. It keeps them from dying.

The authors of the article, Jennifer Doleac and Anita Mukherjee, argue that while the drug may save lives in the immediate situation, it does not reduce overall drug deaths. Worse, the unintended consequences of the drug outweigh its short-run benefits. Those whose lives are saved go back to using drugs, committing crimes, and winding up in emergency rooms. In addition, a drug that will prevent overdoes death “[makes] riskier opioid use more appealing.” 

The title is “The Moral Hazard of Lifesaving Innovations: Naloxone Access, Opioid Abuse, and Crime.” (A moral hazard is something that encourages people to do bad things by protecting them from negative consequences.)

Naloxone didn’t happen all at once. In 2013 fewer than ten states allowed it; the next year the number had doubled. In 2015, only nine states still did not allow its use. Doleac and Mukherjee used these time differences to look at bad outcomes (theft, death, ER admissions) before and after the introduction Naloxone in the different states.  Here are some of their graphs.

(Click on an image for a larger view.)


They conclude that “broadening Naloxone access led to more opioid-related ER visits.” As for deaths, “in some areas, particularly the Midwest, expanding Naloxone access has increased opioid-related mortality.”

There are reasons to be skeptical of the data, but let’s assume that the numbers – the points in the graph – are accurate. Even so, says Andrew Gelman (here), there’s still the question of how to interpret that array of points. Doleac and Mukherjee add lines and what I assume are confidence bands to clarify the trends. But do these added techniques clarify, or do they create a picture that is different from the underlying reality? Here’s Gelman:

The weird curvy lines are clearly the result of overfitting some sort of non-regularized curves. More to the point, if you take away the lines and the gray bands, I don’t see any patterns at all! Figure 4 just looks like a general positive trend, and figure 8 doesn’t look like anything at all. The discontinuity in the midwest is the big thing—this is the 14% increase mentioned in the abstract to the paper—but, just looking at the dots, I don’t see it.


Are these graphs really an optical illusion, with the lines and shadings getting me to see something that isn’t really there? My powers of visualization are not so acute, so to see what Gelman meant about looking only at the dots, I erased the added lines and bands. Here is what the graphs looked like.


Like Gelman, I can’t see any clear patterns showing the effect of Naloxone. And as I read the reactions to the paper, I sense that its results are ambiguous enough to provide rich material for motivated perception. Conservatives and libertarians often start from the assumption that government attempts to help people only make things worse. The unintended-consequences crowd – Megan McCardle, for example (here) – take the paper at face value. Liberals Richard G. Frank, Keith Humphreys, and Harold A. Pollack (here), who have done their own research on Naloxone – are more skeptical about the accuracy of the data.*

----------------

* This reminded me of a post I did in the first year of this blog.  It was about an editorial in the WSJ that included an utterly dishonest, ideologically motivated connect-the-dots line imposed on an array of points. The post is here.

Le Bron and Grades

March 21, 2018
Posted by Jay Livingston

I’ve never used those percentages in grading – you know, 94% is an A, 82% is a B– and so on. Eighty-two percent of what, I wonder, especially on essays? The percent may be just the same subjective judgment of an essay, but one that makes it seem more precise and objective. You can’t argue with numbers. 

I do use points – a point for each multiple-choice or short answer, so many points for a short essay, more for a long essay – to weight different parts of the exam. But I don’t use a standard formula for converting exam scores to letter grades. Instead, I adjust my own idea of what the grades should be to the actual distribution of scores in the class.

I explain this on the first day of class, and I post a document (“How I Grade”) on Canvas at the start of the semester, but apparently neither of those is very convincing. Students see their scores on Canvas, which also converts these to percents. The student who sees 55% on Canvas comes to class convinced it’s a failure. Even when I give them my own number-to-letters conversion for the exam, some remain skeptical. It’s as though I were singing my own made-up lyrics to some well-known song. They hear, but deep down they know, “Those aren’t the real words.”
                                                   
I came up with a new strategy this week when I gave back the midterms. “Who’s the best basketball player in the NBA?” I asked. Everyone knew: LeBron James.

“What’s LeBron’s field-goal percentage?” The few students who had some idea lowballed it, guessing 40-45%. “Actually, he’s having a good year,” I said. “He’s hitting about 55%.” Then. “So I guess that means LeBron gets an F in basketball.”

That analogy may have done the trick – that, plus a letter grade that was higher than what they had expected.

Drugs and Death Penalties

March 13, 2018
Posted by Jay Livingston


It’s a sure sign of desperation when a politician calls for the death penalty for drug dealing.


Trump is basically admitting that his administration has no idea how to solve the drug problem.

Two hundred fifty years ago, Cesare Beccaria argued that of the three elements of punishment – certainty, swiftness, and severity – severity is the least effective. All the subsequent research has proven him right. But certainty and swiftness are hard to increase; severity is easy. Just pass some laws imposing long sentences, mandatory sentences, life sentences, and of course, death. When politicians call for the death penalty what they are saying is, “We don’t know how to catch very many of these guys, and it takes a long while before they are actually sentenced, so when we do catch one of them, we’re going to show him how pissed off we are.”

Draconian punishments may be very good for expressing the frustration and anger of law-abiding people, and Trump is very good at playing to those emotions. But as for the practical effects, executions are unlikely to have much of an impact on crime. What was true in in the crack crisis of the 1980s is still true: there already is in fact a death penalty for drug dealers. It’s just not administered by the state. It’s administered by rival drug dealers. And compared with any death penalty the state might impose, it is carried out with far greater frequency and swiftness.

President Reagan was fond of saying that in the sixties we fought a war on poverty, and poverty won. He was factually wrong. But if he had made the statement about the war on drugs that the government waged in the following decades, he’d have been closer to the truth. Those years when drugs were winning the war also gave us the spectacle of politicians falling all over themselves to pass harsher and harsher drug laws. Conservative politicians them sounded very much like Trump today. And like Trump today, many of them, perhaps, thought that in this way they were “doing something” about drugs. Of course, what they were more certain of was that they were doing something about getting re-elected.

Ass-Backwards Through the Gateway

March 11, 2018
Posted by Jay Livingston

Imagine that you’re a US Attorney on the drug beat. Your boss is Jeff Sessions, who has announced that he’s going to vigorously enforce laws against marijuana and use the federal law when state laws are more lax. Maybe you also think that weed is a dangerous drug. You do a little “research” and tweet out your findings.



This brief tweet might serve as an example of how not to do real research. The sample, which excludes people who have not gone to treatment centers, is hardly representative of all users. There’s researcher bias since the guy with the ax to grind is the one asking the questions. The respondents too (the drug counselors) no doubt feel some pressure to give the Sessions-politically-correct answer. They may also be selectively remembering their patients. 

But even without the obvious bias, this tweet makes an error that mars research on less contentious issues. It samples on the dependent variable. The use of heavier drugs (opioids, heroin, meth, etc.) is the dependent variable – the outcome you are trying to predict. Marijuana use is the independent variable – the one you use to make that prediction. Taking your sample from confirmed heroin/opioid addicts gets things backwards. To see if weed makes a difference, you have to compare weed users with those who do not use and then see how many in each group take up more serious drugs.

Here’s an analogy – back pain. Suppose that, thanks to advances in imaging (MRIs and the like) doctors find that many of the people who show up with back pain have spinal abnormalities, especially disk bulges and protrusions. These bugles must be the gateway to back pain. So the doctors start doing more surgeries to correct these bulges. These surgeries often fail to improve things.

The doctors were sampling on the dependent variable (back pain), not on the independent variable (disk bulges). The right way to find out if spinal abnormalities cause back pain is to take MRIs of all people, not just those who show up in the doctor’s office. This is pretty much the way it happene in the real world. Eventually, researchers started doing the research the right way and found that lots of people with spinal abnormalities did not pass through the gateway and on to back pain.

The same problem often plagues explanations that try to reverse-engineer success. Find a bunch of highly effective people, then see what habits they share. Or look at some highly successful people (The Beatles, Bill Gates) and discover that early on in their careers they spent 10,000 hours working on their trade.


US Atty. Stuart’s tweet tells a good story, and it’s persuasive. But like other anecdotal evidence and eyewitness testimony, it is frequently misleading or wrong. The systematic research – many studies over many years – shows little or no gateway effect of marijuana. No wonder US Attorney Stuart chose to ignore that research.*

----------------------------
* As Mark Kleiman has argued, even when a marijuana user does add harder drugs to his repertoire, the causes may have less to do with the drug itself than with the marketplace. The dealer you go to for your weed probably also carries heavier drugs and would be only too happy to sell them to you.  Legalizing weed so that it’s sold openly by specialty shops rather than by criminals may break that link to other drugs.