You probably saw the breathless reports suggesting, as CNN did, that “same-sex marriage may decrease teen suicide attempt rates, study says.” Ah yes, a study. A statistical study. That’s supposed to give it gravitas. In fact many, perhaps even most, statistical studies simply can’t be trusted. Certainly not this one.

The study in question is the peer-reviewed paper “Difference-in-Differences Analysis of the Association Between State Same-Sex Marriage Policies and Adolescent Suicide Attempts” by Julia Raifman, Ellen Moscoe, and S. Bryn Austin, in the once-prestigious journal JAMA Pediatrics.

This widely touted work purports to have discovered, using statistical methods, “that same-sex marriage policies were associated with a 7% reduction in the proportion of all high school students reporting a suicide attempt within the past year.” The authors say there is now “empirical evidence for an association between same-sex marriage policies and mental health outcomes.”

Association.

Think about what the authors are implying: that the mere presence of gmarriage — government-defined marriage, as opposed to marriage defined by reality — stops teens from reporting suicide attempts.

Implications

If what these authors are eager to imply is true, it must have been that some teenagers before gmarriage reported trying to kill themselves because there was no such thing as gmarriage. Or it must be that some teenagers after gmarriage became the “law of the land” thought to themselves, “You know, I was going to report trying to kill myself. But now that Bert and Ernie can be gmarried, I won’t report it.” (Both could be true.)

About the number of teenagers who actually killed themselves because of the absence of gmarriage — or because of the presence of gmarriage — nobody knows. The study only relates how many kids self-reported suicide attempts. Since most of the kids giving answers were 15-16, it can’t have been because of actual forbidden gmarriage or marriage ceremonies that caused reporting suicide attempts (of course, there could have been a handful of child brides or grooms in the data).

This is among all teens, mind you, and not just the minority reporting same-sex attraction or other non-biologically oriented sexual desires. The authors claim the effect was greater in the sexual minority.

Weighted Realities

Forgive the dive into the details, but it’s necessary to see what’s really happening. Via a complicated massaging of numbers, the authors say that before gmarriage

a weighted 8.6% of all high school students and 28.5% of 231 413 students who identified as sexual minorities reported suicide attempts before implementation of same-sex marriage policies. Same-sex marriage policies were associated with a 0.6-percentage point…reduction in suicide attempts, representing a 7% relative reduction in the proportion of high school students attempting suicide owing to same-sex marriage implementation.

A weighted 8.6% to a weighted 8%, they say. This is a 7% reduction, all right, but a minor tweak in the actual weighted number. The numbers are weighted averages across several states and the result of a statistical model called a linear regression. The 0.6 drop is not observed, but is the output from a model.

What’s odd is that the authors report the rate for teens reporting non-traditional sexual desires (a modeled 4% drop from 28.5%), and also for all teens (that modeled 0.6% drop), a group which includes the sexual minority. But they don’t report numbers for normal teens (did this number increase?). This omission leads one to suspect the authors are fooling themselves. This is suggested in two ways.

Complication

The first is that these numbers are modeled averages across states. The numbers within states is anything but straightforward (the authors provide graphs). For instance, some states show reported suicide attempts increasing after gmarriage (New York, Maine, New Hampshire and Vermont, for example). The graphs also indicate a general decline in rates before gmarriage and continuing afterwards (see their Fig. 3). What makes this even more curious is that there are only a couple of years of data after gmarriage (there are many before), making comparisons more prone to error.

But those are all quibbles. Forget them if you like. The second reason is more to the point. The year each state imposed gmarriage was put into the authors’ model: this created a before and after period. The attempted suicide rates in the before period were caused by any number of things, but, the authors imply, some of the attempts were because of the lack of gmarriage.

In the after period, there were still myriad causes of suicide attempts, but one cause was removed (lack of gmarriage). Thus some kids who would have tried to unsuccessfully kill themselves did not try because of gmarriage access (but not for themselves, because they were too young).

Correlation is not Causation!

But — and this is a big but — since no questions about why kids tried to kill themselves were asked, the demarcation of before and after is entirely arbitrary. The year most cited for gmarriage was 2014. Thus not only could access to gmarriage by others (but not for themselves) be used, so could the Ebola epidemic becoming a global health crisis, which also happened in 2014, and which was one of the biggest news stories of the year, according to ABC.

Think: putting Ebola in the model works equally well with gmarriage to explain the data. So do the disasters of those crashing Malaysian airliners, or the fighting in Ukraine and Crimea. So does the 2014 Winter Olympics! And the death of sad-funny-man Robin Williams (all mentioned by ABC).

Or any of an uncountable number of events. The truth is the data do not say, and cannot say, what caused the observed changes. It is sloppy statistics — it is bad science, period — to suggest the one thing the authors thought of had to be the one and only cause of the changes.

It is made worse when this cause has so little bearing on the lives of the people purported to be affected. Fifteen and 16-year-olds do not often marry, and nobody has (yet?) heard of any same-sex “weddings” between teenagers.

And it grows worse yet again, when it is implied, as the authors do imply, that the increases after gmarriage in specific states were actually decreases — because the (modeled) mean across states decreased.

If suicide attempts increased in a state after gmarriage, as it did in several states, could it be that the presence of gmarriage is causing more kids to try and kill themselves?

If not, why not?