"Statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write." - H. G. Wells
Stephen Jay Gould (a really famous biologist, in case you didn't know) wrote the essay The Median Isn't the Message immediately after he was diagnosed with fatal cancer.
Lies, lies, and statistics
Central tendencies are crucial to statistics. The most common is the mean, or each number added up and divided by the number of observations. The median, however, is physically the middle number (just like the median in the road is physically in the middle of the road). Line up all the observations in ascending or descending order and pick the one in the middle, and you have the median.
Mean v. Median
Gould, who was diagnosed with mesothelioma, discovered the statistics surrounding the survival rate for this particular cancer. The median mortality was a mere eight months after discovery.
Gould, however, decided that "Attitude clearly matters in fighting cancer" and he decided to reframe the problem. A median mortality of eight months, after all, simply means that the middle person died within eight months. There could be hundreds of people who lived longer, even years or decades longer. Because the mean wasn't reported, only the median, Gould decided there was hope.
"We still carry the historical baggage of a Platonic heritage that seeks sharp essences and definite boundaries." - Gould
But, Gould argues, "nature often comes to us as irreducible continua" as
This Platonic heritage, with its emphasis in clear distinctions and separated immutable entities, leads us to view statistical measures of central tendency wrongly, indeed opposite to the appropriate interpretation in our actual world of variation, shadings, and continua. In short, we view means and medians as the hard "realities," and the variation that permits their calculation as a set of transient and imperfect measurements of this hidden essence.
"Variation itself is nature's only irreducible essence." - Gould
My own esteemed biostatistics professor in graduate school argued that without variation, there is no statistics. To visualize this, check out these data from this dataset:
NO VARIATION
Pretty boring, right? No variance means there's no standard deviation or standard error BUT there's no information. Variation is the spice of life!
Now let me make up a dataset WITH variation.
VARIATION
As you can see, the MEDIAN didn't change between datasets but the MEAN did. It represents that 13, because it's pulled below 48.
Looking at the descriptive statistics, variance is what drives everything. The median, however, doesn't really reflect variance. As Gould says:
(A)ll evolutionary biologists know that variation itself is nature's only irreducible essence. Variation is the hard reality, not a set of imperfect measures for a central tendency. Means and medians are the abstractions.
Because he understood variation, Gould did not read too much into the median of the mesothelioma statistics, and instead placed himself where he belonged: in the variation in the mesothelioma dataset.
Indeed, upon further examination, Gould realized that the "distribution of variation about the eight-month median would almost surely be... 'right skewed.'"
Let me illustrate a right skewed dataset. A normal distribution looks like a bell (indeed, it also goes by the name "bell-shaped curve.") A right skewed dataset, on the other hand, has more data to the right of the central tendency, as:
A RIGHT-SKEWED DISTRIBUTION
As you can see, the mean is much higher than the median, which gave Gould hope that he fell in that right tail of longer-surviving patients of mesothelioma.
Indeed, in this tightly prescribed set of circumstances ("survival with mesothelioma under conventional modes of treatment") Gould did find a long if narrow right tail. As he cautions, "If circumstances change, the distribution may alter," which is a good caution for those of us who interpret data.
"The reports of my death are greatly exaggerated." - Mark Twain
I highly recommend reading this short essay by Gould. He was a great writer and possessed astonishing intellect. His central thesis about not taking statistics at face value, about digging into the data themselves, about not letting numbers dictate your life... well, those ideas speak for themselves.
Gould was diagnosed with mesothelioma in 1982. He died in 2002 at the age of 60 at the hands of an unrelated cancer (metastatic adenocarcinoma of the lung).
Perhaps I'll write another post about his Spandrels of San Marcos, a much longer paper of his that had a profound influence on me during my graduate studies. I do want to write about his theme of positivity and humor in The Median Isn't the Message, so look for that post later.
Thanks to @lukestokes for posting an article with a Plotly generated graph so I discovered this amazing website!
All images original unless credited here:
76% OF ALL STATISTICS ARE MADE UP ON THE SPOT: Dilbert Cartoon by Scott Adams
Right-skewed Distribution: Schoolbag.info