Highlighted Article: How we fool ourselves. Part II: Scientific consensus building

From: Climate Etc.

By: Judith Curry

Date: April 10, 2021

How we fool ourselves. Part II: Scientific consensus building

“Like a magnetic field that pulls iron filings into alignment, a powerful cultural belief is aligning multiple sources of scientific bias in the same direction. – policy scientist Daniel Sarewitz

Statistician Regina Nuzzo summarizes the problem:

“This is the big problem in science that no one is talking about: even an honest person is a master of self-deception. In today’s environment, our talent for jumping to conclusions makes it all too easy to find false patterns in randomness, to ignore alternative explanations for a result or to accept ‘reasonable’ outcomes without question — that is, to ceaselessly lead ourselves astray without realizing it.”

Psychologists Richard Simmons et al. find that researcher bias can have a profound influence on the outcome of a study. Such ‘researcher degrees of freedom’ include choices about which variables to include, which data to include, which comparisons to make, and which analysis methods to use. Each of these choices may be reasonable, but when added together they allow for researchers to extract statistical significance or other meaningful information out of almost any data set. Researchers making necessary choices about data collection and analysis believe that they are making the correct, or at least reasonable, choices. But their bias will influence those choices in ways that researchers may not be aware of. Further, researchers may simply be using the techniques that work – meaning they give the results the researcher wants.

The objective of scientific research is to find out what is really true, not just verify our biases. If a community of scientists has a diversity of perspectives and different biases, then the checks and balances in the scientific process including peer review will eventually counter the biases of individuals. Sometimes this is true—but often this does not happen quickly or smoothly. Not only can poor data and wrong ideas survive, but good ideas can be suppressed.

How we fool ourselves. Part II: Scientific consensus building