The Word that Science RUINED
When it comes to nutrition and weight loss research, the first question everyone always asks is “Were the results significant”.
Unfortunately ‘significant’ isn’t what it used to be.
So what does significant mean anyways?
The use of significant research can often be very misleading, sending you down the wrong path in your quest for ‘research proven’ weight loss.
Some background…
In statistics, a result is called ‘statistically significant’ if it is unlikely to have occurred by chance.
That’s all it means.
So the commonly used significance P value of <0.05 simply means there is less than a 1 in 20 chance that the results found in the study were just a fluke and happened by complete chance.
What it does NOT mean is that the difference is necessarily large, important, or ‘significant’ in the common real-world meaning of the word.
Finding SIGNIFICANCE doesn’t mean the results were SIGNIFICANT.
How confusing is that?
The truth is, having statistical significance says nothing about the practical significance of the findings. Yet, for some reason, we are led to believe that significance is the ‘holy grail’ of scientific research.
Basically, if it is significant, it is right.
I’m not sure how this happened.
I think it has something to do with the use of scientific research as legal evidence in court cases. To me, this is when it no longer mattered if the results were applicable in the real world to real people; it only mattered if they were legally defensible.
The other tricky thing about significance it that significance DOES NOT mean causation.
In other words, just because the results of a study were found to be significant, it does not mean that one thing in the study actually caused the other.
This is Causation and it is actually EXTREMELY difficult to prove.
In fact, a good scientist will point out that causality can ONLY be proven by demonstrating a mechanism. Statistics and significance alone can never prove causality.
To even SUGGEST the idea of causation, you need to have certain evidence…
The relationship has to be strong, consistent, specific, plausible, follow a logical time sequence and shows a dose-response gradient.
It is VERY RARE that one scientific study can prove all of these things, and certainly being ‘significant’ proves NONE of these things.
Often, journalists and bloggers get so geeked up by a finding that is statistically significant that they all but ignore the practical importance or relevance of their findings.
“Scientifically proven to cause X”
If you see a claim like this, it’s probably wrong, or at least greatly oversimplified.
Significance tests are NEVER the whole answer. They are just one single piece of a very large puzzle. Statistical significance is irrelevant if the effect is of no practical or real-world importance.
So why did science ruin the word significance? Well significant used to mean “important; or something of consequence.”
Which I would argue it no longer means. It is an overused, overhyped term that is more science-marketing than it is science.
So we need a new word to describe the findings of research and whether they apply to the real world.
I nominate ‘Remarkable’.
Especially if you define something that is remarkable as “worthy of notice or attention.”
So instead of asking if the results of the study were significant (which they almost always are these days) ask if the results of the study were remarkable.
More importantly whenever you read about research you need to ask yourself did the story give you enough information to determine whether or not the findings were remarkable. If they couldn’t give you enough information to make this decision easy, it probably wasn’t.
Or, if you don’t have the background needed to decide whether the results were remarkable, but feel like they SHOULD be, this is probably just an example of great science-marketing.
Remarkable…it’s the new significant.
BP
Other posts you may enjoy
-
[...] Dave wrote an interesting post today onHere’s a quick excerptWhen it comes to nutrition and weight loss research, the first question everyone always asks is “Were the results significant”. Unfortunately ‘significant’ isn’t what it used to be. So what does significant mean anyways? … [...]
“It is an overused, overhyped term that is more science-marketing than it is science.”
Excellent summation, as is the suggestion that significant lost its significance (snicker, pun intended) when it became a legal defense.
Brad,
I agree that this has become a confusing term. But you know, it seems to me that a lot of fitness myths are built and perpetuated on research “results” that may show in numbers, but are really NOT of statistical significance. The “chance” numbers can be skewed to marketer’s advantage. Am I reading you right?
Lani Muelrath
Thanks Alyson,
BP
Hi Lani,
The more tests you run the better your chance of finding significance in at least one of the tests..its what we call ‘data mining’.
And really, and number that is found to be significant can then be romanticized by a good marketer.
BP
Statistics gets misused a lot. If anybody’s read Dr. Gregory Ellis’ book Ultimate Diet Secrets, there is a whole section there that explains how statistics can be used to outright LIE about facts.
Good call, BP
-Yavor
I love you article and as everyone has pointed out statistics and scientific proof can be manipulated for your intended outcome.
Now don’t take offense, but I’m having a hard time with “remarkable” as a replacement. I think many things can be remarkable. Good or bad. Just not indifferent. Although when looking it up on line it is defined as “worthy of notice or attention” which is without positive or negative connotations. So perhaps if scientific results are remarkable they are worth taking a look at one way or the other.
All said with good intent. Ellen
@Ellen
Excellent article. Science, like religion, is easily misused as an all-encompassing truth. Only when people disagree with science, they’re labeled “insane” as opposed to when they disagree with religion (usually landing them the label of “heretic” or one of it’s many more-or-less extreme variations.) Tangent.
Remarkable would be a good word, but it can be taken out of context and used to manipulate the less educated public in the same manner which “significant” was. That said, I’m having a hard time seeing how any single word can encompass that goal which people are looking for in any given product (as in the weight loss product you mentioned.)
I suppose a good rule of thumb would be, “If the statements made about a product are not specific, assume that they are vague due to intent to hide details.”
Perhaps easily accessible product reviews (available online, via blog?) would be a good way to educate the wary consumer about the effectiveness of products, as well as perhaps give the manufacturers a better idea of what the consumers would like to have crafted.
Brad, I read your book Eat Stop Eat. I have a unrelated question. I am 6 ft 245 around 17% bf. I want to weigh 210. What do you think about me being in a 500 calorie deficit 6 days out of the week(2500 calories) and then fasting on the 7th. Would i be hurting myself as far as my metabolism or would i be just fine.
@Bobby
Bobby, that sounds fine. Refeeding near or above maintenence really only gets important when you approach a single digit BF% (which is much lower than people think – For example, in Fight Club Brad Pitt was likely about 7-8%, despite people claiming him to be about 5% – and he was ripped). It’s even skeptical at that point.
But if you want to be safe, fast twice per week and eat at maintenence the other 5-days. Again, if there is any impact on your metabolism (which there shouldn’t be), it will be relatively insignificant. Good luck.
Excellent post Brad. Numbers are so damned misleading in food and supplement marketing it’s ridiculous. I believe I remember seeing one advertisement proclaiming their supplement boosted testosterone by some astronomical number “backed by science” – but the study cited was performed on menopausal women…not exactly what young dudes want!
Hi Brad,
I must say this an interesting article and i will start to look at scientific studies from another point of view from now on…
I have an unrelated question that has been pestering me for quite some time now… Is it somehow possible to use the GH spike that occurs during fasting for my advantage during a mass gaining phase?? I used fasting methods similar to what you describe in your blogs/videos (1 fast per week, 24hrs combined w/VERY clean diet)to drop weight for previous MMA fights and it’s worked wonders for me…not to mention allowing for more time training instead of eating.
But lately I’ve fallen in love with bodybuilding/powerlifting, and made a goal for myself to weigh 190lbs by April 1st. I currently managed to get myself up to 175 and i’ve come to realize the food bill is really starting to burn a hole in my pocket
I know there is some speculation whether GH itself actually builds muscle or not. Let’s say for the sake of argument that it doesn’t…but it does for sure stimulate the release of IGF-1 which would undoubtably build muscle. I was thinking to maybe plan my fast around a night time meal, like for example eat at 10pm then not eat again until 10pm the next evening, the reason being that when u sleep u also secrete GH within your body, so it would be like a double dose of GH (during fast + during sleep), and i would have protein/nutrients available for the synthesization of the hormone to the muscle due to my night time meal.
If i were to combine this method of fasting with a caloric surplus diet the rest of the week would it speed up my results due to the increase of GH in my body?? Or would various mechanisms prevent me from making as much of the hormone as i think is possible?
I guess the bottom line is: is it worth it to fast if your goal is to gain a fair amount of muscular weight?
Sorry for the long post, but i love these kinds of topics (there isn’t a day when i don’t think about fitness stuff
)
Hey Bobby,
I’m not Brad, but have feedback for you nonetheless!
If we take the model of calorie intake decrease averaging 20% for a reasonable yet definite weight (fat) loss, then you can test that in your equation – would the total deficit for you be in that range?
Also, recall from Brad’s book that it takes fasting for several days in a row OR extreme undereating for long periods of time to create any kind of metabolism rate problem.
Hope this helps until Brad can stop by!
Lani
Thanks Lani
thanks James
@Yavor Marichkov
Hi Yavor,
Stats really can’t lie, but they can be misused, misinterpreted, poorly explained, or falsely used to support a lie..
The numbers don’t lie, the people using them do.
B
@Ellen
Agreed, probably not the best word, but it is a step in the right direction.
B
@Vadim
A lot of this depends on your age and training status. Simply put, a 40 year old man with 20 years of training experience is not going to gain 25 pounds of muscle. If you are in your 20’s and just starting to train seriously than it is possible.
I think the connection between GH and muscle mass is becoming clearer, however from a hormonal point of view, the strongest relationship is between testosterone and muscle.
That being said, I think it all comes down to the structure of your workouts.
B
@Lani Muelrath
Thanks for the great answers Lani!
B