here are some of my favorite changes that don't measure like "changes":

• "we maintained something against the odds when OTHER things changed." My hometown just flooded. Some places less than others because they had made some choices anticipating this.

• "we showed up in a data collection when the most likely thing was we would become missing data" The learning scores of kids who went to school during a crisis are in this.

• "we intervened on a steep downward trajectory and yeah we couldn't flip it but we made it LESS steep." Climate! Pollution! Poverty! Improving these things are on a continuum and improving that trajectory does tangibly matter.

• "we produced a remarkable impact but only for a particular group too small to move a population avg." If you could completely cure one rare subset of a disease, isn't that be a true huge impact and still matter? Plus "rare" examples often transform science tbh

• "based on all our previous models we had every reason to predict there would be a change, and for some reason there wasn't" boy this is a REALLY important kind of finding. The black hole at the center of the galaxy kind of stuff is also the stuff that transforms science.

• "we did measure a change in a target but there was a big substitution effect because people just swapped their behavior for something else that accomplished the same outcome dammit" human minds. are so. ridiculous clever.

• "we did make a change but shoot this situation that enables the change is both undermeasured and systematically MISmeasured in the standard ways we collect data in this field damn that's a lot of forms of noise" --> I'm looking at you health. DEATH IS NOT THE ONLY THING WE CARE ABOUT and hospitals aren't randoms samples of patients all equally showing up at the same levels of need

by the way there are ways to mislead ourselves less in ALL of these scenarios -- they're just not being taught to most of your data scientists because data science decided it wanted to be a branch of computer science, probably for the $$$, and CS soundly, for decades, has hated social science. 🤷‍♀️

There's no point pretending it's just "we don't know." It is "we systematically, and brutally, have decided not value the expertise is takes to do work on human behavior in the real world."

Cat Hicks