Scientific Truth

If it’s not independently reproducible, it’s not science.

Our society purports to be based on science. As an academic, I saw how the sausage is really made. Science in the abstract is thought to work like this: there's some phenomenon, you study it, you write a paper, it's peer reviewed, it's published. It's then reported on by news outlets, and then it is used as input for regulation. Regulators then cite the article and, if pressed, the scientific paper backing it up. That is “based on science.” That's the assembly line. Seems reasonable.

The problem is when the form and substance of science are mistaken. The form of science today is a peer-reviewed journal paper. But the substance of science is independent replication.

People confuse genuine science (like Maxwell's equations) with “science” (like a paper that came out last week). Maxwell's equations have had countless trillions of independent replications. The study from last week might not even have shared a public data set. Yet we as a society are supposed to be basing large decisions on this one new study?

Many people think “peer review” means “independent replication and confirmation of results.” Peer review usually means months of delay while a few folks in your field write an email-length criticism of the paper and ask for more work. Peer review is not a panacea.

Why is independent replication so important? Because science isn’t about prestigious people at prestigious universities publishing in prestigious journals echoed by prestigious outlets. Science is independent replication.

Some of the people at those prestigious institutions are very intelligent, hard working, and capable of discovering new things and building functional products. But on the whole, the substitution of prestige for independent replication isn’t serving us well.

Civilization has a ripcord people use when centralized institutions get too ossified. It’s called decentralization. Martin Luther used it to argue for the “personal relationship with God,” disintermediating the Church. George Washington used it. So did Satoshi Nakamoto.

Given how many hunches have been marketed to us as science, the emphasis on independent replication as the core of science is just such a ripcord. Only trust as scientific truth what can be independently verified.

The US research establishment set up in the postwar era is now past its prime. It’s an academic jobs program. You see this in things like the h-index, which has the value "h" if you have at least h papers with h citations each. All academics are optimizing their number of citations in high impact journals…but what actually matters in science is not the number of citations. What matters is, once again, independent replication.

And where are the metrics for independent replication? Where are the incentives for it? Not in academic science, where replicating a result isn’t considered novel, publishable, or grant-worthy. In fact, replicating a result is often considered hostile. That’s why we have the academic replication crisis. Scientific truth was gradually redefined to mean "peer review" rather than "independent replication."

Imagine if we optimized for number of independent replications over number of citations.

An alternative to the academic research establishment has been bubbling in open source and crypto over the last few decades. Today the circulation of knowledge in science is restricted by the high prices of journals. Many students and researchers cannot afford academic journals and books locked behind paywalls.

The goal of Sci-Hub is to provide free and unrestricted access to all scientific knowledge ever published in journal or book form. I've thought a lot about what a crypto Sci-Hub could look like. It might align incentives so publishers earn money by making all their papers open online. I'm all about aligning incentives if at all possible. If we can, that attacks the problem at its base, because our entire society is based on "science."

Why this regulation? Because science. Why do this? Because science. What is that science? Well, some of it is "science," meaning some dumb study that appeared last week, and some of it is truly fundamental science. We're equating the two when they're not the same.

The way to understand the difference is the number of independent replications. If our data is on-chain, we can start to do that.

Every statistic is a numerical distillation of a raw data table. Ask for that table.

Science progresses by taking phenomena we think of as non-reproducible (and hence unpredictable), isolating key variables, and turning them into reproducible (and hence predictable) systems. A key example here is understanding that bacteria cause disease.

Science also progresses by improved instrumentation and better recordkeeping. Star charts enabled celestial navigation. Gregor Mendel’s careful counting of pea plants led to modern genetics. Johann Balmer’s documentation of the exact spacing of hydrogen’s emission spectra led to quantum mechanics. Things we believed to be beyond human ken—the stars, the genome, the atom—became things we can comprehend by simply counting them.

I admire Ramanujan. I admire Feynman. These great mathematicians and physicists were able to see things others couldn't. Just by writing down what they observed, they created a huge leap forward.

In entrepreneurship, people often say it's not the idea, it's the execution. But that's for trivial ideas. For great ideas like Maxwell's equations or Newton's laws, the idea itself really does bring us forward.

Sometimes you have the practical phenomenon, but you don't have the theory underneath it. Then that stimulates the advancement of theory to figure out why the thing actually works. For example, people first got steam engines working and only then discovered the thermodynamic theory from that. The practice often leads to the theory rather than vice versa. The limits of our understanding are more of a bug than a feature.

I see a strong correlation between lack of technical ability and naive trust in social authority. The only true authority is raw data.