How to science better

What do exceptional scientists do differently from mediocre ones? Can we train currently-mediocre ones to do better?

Removing unnecessary obstacles and creating better incentives will encourage people to do science, rather than cargo cult pretend-science. Doing actual science better, as well, may yield additional velocity.

We lack specific, accurate knowledge of what scientists do, and why that works. A vague assumption that we do know, based on obsolete philosophical misunderstandings, makes investigation seem needless, and so is a major obstacle to progress.¹

As undergraduates, we learned folk theories of The Scientific Method—derived from long-since disproven early-twentieth-century philosophy—and never noticed they’re wrong. We know in detail why a particular biological lab procedure works, but have no idea why science works. We can fix software bugs, but no one can explain what software engineering is.

It’s assumed that scientists and engineers learn how to do what we do in classrooms and from reading, and therefore our fields are bodies of explicit, codified knowledge. This is false. Key aspects of the work are tacit, and learned only through apprenticeship and personal practice. We know almost nothing about how that works.

We need much more research on what scientists and technologists actually do—not as described in textbooks or even laboratory protocol manuals, but all the parts that aren’t in there. We need to understand how and when and why the shower thoughts and Slack chats and WTF moments in the lab produce breakthroughs.

Excellence involves types of thinking dissimilar to rational problem solving. The work of science is meta-rational, meaning that it is about how to use rationality. Doing tasks like these well is critical to scientific progress:

• Find a general topic that is both important and tractable, and choose to work on it
• Understand the state of the art in the area
• Form opinions about what theories are believable (and why) and what issues are important (and why)
• Choose a specific research question
• Define it precisely enough that it’s amenable to experiment
• Devise an abstract experimental strategy that could, in principle, answer the question
• Turn the general strategy into a nuts-and-bolts procedure that addresses the messy, nebulous details of the phenomenon and the experimental apparatus
• Perform the procedure correctly; this usually involves extensive tacit hands-on skill, observational acuity, and intuitive understanding—none of which ever gets written down²
• Understand the implications of the results
• Explain the experiment and its results accurately and intuitively, so other people in the field understand them too

Experienced technical professionals develop a “feel for” these tasks, which mostly we can’t communicate. We have to do this work without a broader or explicit understanding of what it is or why it works.

Some outstanding researchers develop such understanding through critical reflection on the nature of the field. That makes them better at meta-rationality: better at figuring out what work is worth doing, and at finding strategies for pushing the whole field forward in that direction. Some can teach this too, multiplying their meta-level understanding through a lineage. Not all scientists considered great have this ability, but perhaps the most valuable scientists are those who do.

Exactly why doesn’t cargo cult science work? And what does work? What makes the difference between cargo cult science and the real thing? Institutions can upgrade their norms; but incentives can and always do get gamed.

In “Upgrade your cargo cult for the win” I suggest the antidote is “unflinching lustful curiosity.”³ That means actually wanting to understand what is going on. There is no recipe for finding out, so mindless box-ticking conformity doesn’t work.

Curiosity without a set method implies existential commitment: you are accountable to reality itself, not to any specific incentives or objective function. If that is difficult to automate, then we should expect that Transformative AI in Karnofsky’s sense may be less imminent or likely than Real AI of other sorts.