Work Backward from the Choice

The default order is to look at the data, see what it says, and form a view. The order this method argues for is to form a view, write down what would prove it wrong, and then go looking.

The objection is immediate and it deserves a real answer.

“Is that not just confirmation bias with extra steps?”

It would be, if the candidate ranking were protected. It is not. The entire point of writing it down is that it becomes a target.

The mechanism is this. An unstated hypothesis cannot be falsified, because it can quietly deform to fit whatever the data shows. Analysts who claim to approach data without a prior have a prior; they simply cannot be held to it. Writing it down converts a bias you cannot see into a claim you can attack.

The discipline that makes this work is that the claims are written before any query runs, and each one carries, in advance, the result that would falsify it. “If the misses are spread evenly across item classes rather than concentrated, class-based stocking is theater” is a claim with a kill condition attached. You cannot retrofit that after seeing the answer.

The sequence

Write the candidate ranking. What you would pick this afternoon if forced, and why you suspect it. It is allowed to be wrong. It is not allowed to be blank or unspoken.

Write the claims that would kill it. Two to four, of the same kind, each one a statement a result could falsify. Decide now what result falsifies each.

Choose the cheapest test that could change your mind, and run it. Not the most thorough test. Not the most interesting method. A candidate that dies in week two is a successful week two.

Data last

This reads as heresy and it is a claim about ordering, not about importance.

Data gathered before the claims exist is gathered against no standard, which is why exploratory phases expand without bound: there is no criterion for when to stop. Data gathered to test a named claim has a finish line. You stop when the claim is settled.

The practical consequence is usually that you need far less data than planned, and that the extraction work you assumed was a prerequisite turns out to be optional.

The ways you will fool yourself

Three recur often enough to check for by name.

You will find the cause you are equipped to find, because the specialist finds the specialist’s cause. The remedy is to have someone from a different discipline read your claim list before you run anything.

You will keep a candidate alive past its kill condition, because you wrote the condition before you were invested and you are invested now. The remedy is that the condition was written down, which is the only reason you can catch this.

You will treat a surviving candidate as a confirmed one. Surviving two cheap tests is weak evidence. It is enough to proceed and not enough to stop naming what would flip it.