When it comes to nutrition and weight loss research, the first question everyone always asks is “Were the results significant”.
Unfortunately ‘significant’ isn’t what it used to be.
So what does significant mean anyways?
The use of significant research can often be very misleading, sending you down the wrong path in your quest for ‘research proven’ weight loss.
Some background…
In statistics, a result is called ‘statistically significant’ if it is unlikely to have occurred by chance.
That’s all it means.
So the commonly used significance P value of <0.05 simply means there is less than a 1 in 20 chance that the results found in the study were just a fluke and happened by complete chance.
What it does NOT mean is that the difference is necessarily large, important, or ‘significant’ in the common real-world meaning of the word.
Finding SIGNIFICANCE doesn’t mean the results were SIGNIFICANT.
How confusing is that?
The truth is, having statistical significance says nothing about the practical significance of the findings. Yet, for some reason, we are led to believe that significance is the ‘holy grail’ of scientific research.
Basically, if it is significant, it is right.
I’m not sure how this happened.
I think it has something to do with the use of scientific research as legal evidence in court cases. To me, this is when it no longer mattered if the results were applicable in the real world to real people; it only mattered if they were legally defensible.
The other tricky thing about significance it that significance DOES NOT mean causation.
In other words, just because the results of a study were found to be significant, it does not mean that one thing in the study actually caused the other.
This is Causation and it is actually EXTREMELY difficult to prove.
In fact, a good scientist will point out that causality can ONLY be proven by demonstrating a mechanism. Statistics and significance alone can never prove causality.
To even SUGGEST the idea of causation, you need to have certain evidence…
The relationship has to be strong, consistent, specific, plausible, follow a logical time sequence and shows a dose-response gradient.
It is VERY RARE that one scientific study can prove all of these things, and certainly being ‘significant’ proves NONE of these things.
Often, journalists and bloggers get so geeked up by a finding that is statistically significant that they all but ignore the practical importance or relevance of their findings.
“Scientifically proven to cause X”
If you see a claim like this, it’s probably wrong, or at least greatly oversimplified.
Significance tests are NEVER the whole answer. They are just one single piece of a very large puzzle. Statistical significance is irrelevant if the effect is of no practical or real-world importance.
So why did science ruin the word significance? Well significant used to mean “important; or something of consequence.”
Which I would argue it no longer means. It is an overused, overhyped term that is more science-marketing than it is science.
So we need a new word to describe the findings of research and whether they apply to the real world.
I nominate ‘Remarkable’.
Especially if you define something that is remarkable as “worthy of notice or attention.”
So instead of asking if the results of the study were significant (which they almost always are these days) ask if the results of the study were remarkable.
More importantly whenever you read about research you need to ask yourself did the story give you enough information to determine whether or not the findings were remarkable. If they couldn’t give you enough information to make this decision easy, it probably wasn’t.
Or, if you don’t have the background needed to decide whether the results were remarkable, but feel like they SHOULD be, this is probably just an example of great science-marketing.
Remarkable…it’s the new significant.
BP
