Schrodinger's pee: Texas tests for steroids

[gdnp]

http://www.google.com/hostednews/ap/article/ALeqM5gx9BSiNT08U9pbZDfbxHOlYCW6ZgD95KB5300

The state of Texas recently introduced random drug testing for student athletes in order to combat the perceived problem of steroid use by high school students.

The results:

According to a University Interscholastic League report released Dec. 1, the first 10,117 tests produced only the four confirmed cases of steroid use.

Hmm...4 cases out of 10,000 tests. 0.4% confirmed positive. At a cost of $6 million per year.

Opponents of the tests claim, not surprisingly, that this shows that they are a waste of money:

Republican state Sen. Dan Patrick has been a vocal critic of the tests, calling them a "colossal waste of taxpayer money" that could be better spent battling recreational drug and alcohol use among teens.

What about the proponents of testing? Well, they claim the program is a rousing success:

The Texas legislator who sponsored the testing bill in 2007 calls it an "incredible success."

The point of testing was to act as a deterrent against steroid use, not catch teens using drugs, said Rep. Dan Flynn, a Republican.

"We don't have a bunch of pelts hanging on the wall," Flynn said. "The success is that we haven't had a lot of positive tests."

So who is right? Well, it's hard to say. We're missing one crucial piece of data: how many athletes would have used steroids if the testing program had not been instituted? Something we can't know...without instituting a testing program...which would invalidate the results of our tests...

There are theoretically ways around this. We could institute a secret testing program, or a program which is announced so close to the testing date that the athletes do not have time to alter their behavior. But this would not be politically feasible in a liberal democracy. People would not stand for "police state" tactics or imposition of such a testing program without public debate.

Similar problems arise much more commonly than we probably realize. It is the problem of the lack of an adequate control group. Has the 700 billion federal bailout been a success or a failure? We don't know: we can only guess what the economy would be like without it. Do gun control laws harm or help? Again, we can only speculate. Even if homicide rates drop, we can only speculate whether they would have dropped more or less without them. Same if they rise.

Social sciences do not lend themselves to rigid scientific control. This leaves the results open to interpretation and spin. Does anyone see a way out of this seeming paradox?

 

[drkitten]

So who is right? Well, it's hard to say. We're missing one crucial piece of data: how many athletes would have used steroids if the testing program had not been instituted? Something we can't know...without instituting a testing program...which would invalidate the results of our tests...

Survey data suggests that somewhere between 1-6% of high school athletes are using steroids. Some surveys sugges the numbers may be as high as 11% among some groups. Texas data is about 2.3%. Of course, this data is somewhat dated and has its own error bars, but it's still substantially higher than the 0.4% that Texas measured.

If the 0.4% number is to be relied upon, then I think the test proponents are correct; there is a substantial deterrent effect. Of course, I'm suspicious and I suspect systematic error in testing as coaches try to protect their star linebackers. But I think there's quite a bit of data out there for us to use for a baseline.

 

[CapelDodger]

There's been an arms-race between steroid-use and steroid-testing for decades, with steroid suppliers generally staying ahead of the game. Every positive result is challenged by lawyers, forcing testers to reveal their methods; successful concealment is not so challenged.

 

[gdnp]

Survey data suggests that somewhere between 1-6% of high school athletes are using steroids. Some surveys sugges the numbers may be as high as 11% among some groups. Texas data is about 2.3%. Of course, this data is somewhat dated and has its own error bars, but it's still substantially higher than the 0.4% that Texas measured.

I guess it comes down to how much you believe survey data. Especially when the surveys say things like "up to 6% have tried steroids". It doesn't tell you how many are actively using. And as has been seen with exit polling, people aren't always honest when surveyed.

If the 0.4% number is to be relied upon, then I think the test proponents are correct; there is a substantial deterrent effect. Of course, I'm suspicious and I suspect systematic error in testing as coaches try to protect their star linebackers. But I think there's quite a bit of data out there for us to use for a baseline.

Well, there is also the problem of the "presumed positives"

Another 22 cases were deemed "positive" results because students broke testing rules. They either refused to provide a urine sample, had unexcused absences the day they were selected, or left the testing area without approval. A positive test brought a 30-day suspension from play for the first offense.

If we assume that all 22 were using, then the overall positive rate is 2.6%, which is in rough agreement with the 2.3% number you report for Texas. Which would imply that there is no deterrent effect at all.

On the other hand, chances are not all 22 would have been positive if tested. So once again, we really don't know.

 

[drkitten]

I guess it comes down to how much you believe survey data.

Shrug. Data never comes with guarantees. What's the error rate on the steroid tests? Why should I believe that the tests Texas used will catch all forms of drugs?

If what you're looking for is iron-clad proof of anything, science can never provide it. If you're looking for evidence against which to evaluate the claim that the test regime has reduced steroid use, then we've got lots of that.

 

[gdnp]

Or alternatively I have evidence that real rate of steroid use was much lower than the testing advocates claimed and the whole exercise has been a waste of time and money. That is why I found the two diametrically opposite interpretations of the same results intriguing. We don't know if Texas is spending 6 million a year on a legitimate public health program or a witch hunt.

 

[neltana]

I guess it comes down to how much you believe survey data. Especially when the surveys say things like "up to 6% have tried steroids". It doesn't tell you how many are actively using. And as has been seen with exit polling, people aren't always honest when surveyed.

Well, Monitoring the Future, which does a pretty high quality annual survey of 8th, 10th and 12th graders reports the following for 2008:

Steroid use in lifetime: 2.2%
Steroid use in past 12 months: 1.5%
Steroid use in past 30 days: 1.0%

These estimates are based on self-reports and they show a good deal of consistency between years. While there is certainly some inaccuracy due to honesty, confirmatory studies suggest that the effects are minor and tend towards under-reporting.

I've also been involved in other data collection efforts in several states that show similar rates of use.