Typical Characteristics of Less Trustworthy Expert Advice

I’ve certainly ragged on a few specific example of how incredibly stupid and wrong most nutrition “advice” is so here’s a broader statement about “expert” advice in general.

Expert advice with a higher-than-average likelihood of being wrong is often given away by any number of tells. Be extra wary if the advice fits any of these descriptions:

  1. It’s simplistic, universal, and definitive. We’ve discussed the mismatch between the complex, subtle problems that mass experts tackle and the more cut-and-dried conclusions we push them to give us. When advice is of the sort that promises broad benefits and can be described in a sound bit or headline — “Drinking Coffee Extends Life Span!” — chances are good that either it’s coming from an expert who has wandered off track through mismeasurement, bad analysis, or bias, or something has been lost in the translation as the findings made their way through research journals and the mass media. And remember that the conclusions of even the most careful studies are typically based on averages that pretty much ensure the finds won’t neatly apply to most people.
  2. It’s supported by only a single study, or many small or less careful ones, or animal studies. Any advice based on one study should be regarded as highly tentative, no matter how good the study seems. The more studies, the better, as a rough rule, but even a series of big, rigorous studies can occasionally produce wrong conclusions. The risks of error are higher with research conducted with mere dozens of people (though statistical flukes can take over even in studies with thousands of people), or that isn’t controlled and randomized (which means that confounders can take over), or that is limited to certain specific types of subjects such as students or very sick people (which means the results may not be relevant to anyone else). And such errors often can’t be corrected by combining the results of many lower-grade studies, claims to the contrary notwithstanding — putting together several pools of bad data sometimes just creates one really large pool of bad data. I recommend treating as interesting fantasies any claims for human health or behavior that are based entirely on animal studies. And don’t swallow the line that animal evidence has been shown to “translate” to humans because it has been backed up by one or two small or less formal human studies; lousy human evidence is lousy human evidence, regardless of what the animal work has indicated.
  3. It’s groundbreaking. For one thing, most expert insights that seem novel and surprising are based on a small number of less rigorous studies and often on just one small or animal study. That’s because big, rigorous studies are almost never undertaken until several smaller ones pave the way, and if there had already been several studies backing this exciting finding, you probably would have heard about it then and it wouldn’t seem novel now. Or consider the simple, if nonintuitive, logic asserted by Ioannidis and other Bayesians: a novel finding is by definition one that hasn’t previously been clearly observed, and a typical reason why something has never before been clearly observed is that it isn’t real.
  4. It’s pushed by people or organizations that stand to benefit from its acceptance. All experts stand to benefit from their research winning a big audience, of course, and that’s well worth remembering, but in some cases the potential conflict of interest is more likely to be corrosive. That’s especially true when the research is coming out of or being directly funded by individual companies or industry groups whose profits may be impacted by the findings. Corporate sponsorship doesn’t mean a study is wrong, but there’s simply no question it sharply raises the risk of serious bias; tobacco companies, for examples, were always able to find physicians and researchers willing to come up with “evidence” that smoking might be getting a bad rap. And remember that study after study has shown that many potential conflicts of interest are not clearly revealed in expert reports and are sometimes actively obscured, which means that digging or reading between the lines may be required. Government-sponsored research is generally more trustworthy, thought not always — there were widespread charges of research corruption under the George W. Bush administration, which by most (but not all) accounts saw research spending in part as a means to advance an ideological agenda. High-powered consultants, meanwhile, tend to employ mass advice as a way to market their services, leaving the advice approximately as trustworthy as any other form of advertising.
  5. It has that touch of paranoia that characterize people finding conspiracy everywhere. Naturally as soon as any big profit-making entity touches something it turns to poison, according to the experts who want to peddle their agenda while trying to sucker you into believing the contrary point of view is the one with bias. So GMO is a horrible plot to kill us all, fast food is always tainted and toxic, and anything from anti-culture is always good and pure. Unfortunately facts aren’t exclusively available to just one side or support only one ideological point of view. Be wary of anyone who demonizes the alternative views rather than presenting actual facts to support their claims.
  6. You’re reading about the issue in mass media, or worse on Internet media. Real publications have some obligation to at least try to check facts, but this requirement is suspended for anything popular and published by amateurs. Science journals can literally be rated by how carefully they screen articles, how much and how good the peer review is. Don’t be fooled into believing someone with Dr. in front of their name or some letters after their name — science is way more difficult than memorizing some facts at a trade school. But most of all recognize that sensationalist journalism sells, especially to the groupies it’s aimed at, and therefore, unlike real journals, have no burden of proof. Always read the real journal article (plus check that it is really a real journal) and if no link to the real journal article is available immediately put the story you’re reading in the rubbish bin.

There is a reason this prose sounds a little different than my usual, see if you can figure it out as a test of your skeptical reading.


About dmill96

old fat (but now getting trim and fit) guy, who used to create software in Silicon Valley (almost before it was called that), who used to go backpacking and bicycling and cross-country skiing and now geodashes, drives AWD in Wyoming, takes pictures, and writes long blog posts and does xizquvjyk.
This entry was posted in opinion, rant and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s