“A good study”
The gatekeepers keep disappearing.
When it cost $500,000 to produce a record album, you could assume that it was going to reach some people and not be completely amateurish. Today, many songs in the iTunes store have had exactly one listen.
When it cost $5,000,000 to make a video or a film, there was a lot of pressure to improve watchability and get an audience. Today, YouTube is filled with videos with no views at all.
And books from major publishing houses used to be assured of at least 20,000 copies in print and perhaps would find some loyal readers. Today, when anyone can write and publish an ebook, there are many that have fewer than ten sales.
While this open marketplace of creativity led to some broken hearts among creators, it also opened the doors for new ideas, new voices and a path to making it as a creative person.
Which brings us to scientific studies.
To get tenure, to spread an important idea, to gain status with colleagues, a scientific paper needed to be published in one of the dozens of journals that existed for this purpose. While there were some studies that were sloppy or even fraudulent, most peer-reviewed journal articles were probably worth taking seriously, with further inquiry appropriate when something important was at stake.
Today, 87.4% of the self-published and popular science articles available contain stats that are made up and methods that can’t hold up to scrutiny. They know that few people will bother to read the footnotes.*
If the goals are speed and clicks, it’s hard to also create a study that’s truly meaningful. Anyone with access to a dozen undergraduate students can publish a ‘breakthrough’ on behavioral economics or even epidemiology. If it gets read, it must be true.
Not so.
Before you get in a cryogenic chamber to help with your eczema, drink ionized water, or take a pill because you saw it mentioned on an opinion-focused cable show, it’s worth thinking hard about what it means for there to be a good study. Did they show their work? Have reputable peers referred to the study? What does the person publishing the study have to gain?
It’s interesting to note that there are very few breakthrough studies in areas like aerodynamics, perpetual motion and bridge design. That’s because it’s really easy to tell when they’re simply making stuff up.
Sometimes, the gates need keeping.
*Did you see what I did there?