The best model I ever built

I once built a statistical prediction model with $\text{AUC}\approx 0.99$, and in $HOWEVER_MANY years of doing this stuff that remains the most predictive model I have ever built. The story is perhaps instructive. One of the Folks From The Other Functions turned up with a dataset, a binary patient outcome, and A Bunch Of Data from a new assay they were testing out. Being young, stupid, and helpful I of course engaged with this problem, and I was of course clever enough to talk The Stakeholder through the problem and filter the dataset down to the pre-outcome data. And I knew of course that you don’t do clever stuff like oversampling because you’ve got to deal with the patient cohort in front of you rather than the one you want. Go me.

Feeling slightly proud of myself I built the model (ISTR it was gradient boosting). And it worked. It worked almost perfectly. And whilst I was young and stupid I wasn’t quite that young and stupid and fortunately did not go back to colleague-from-other-function with “we can do this perfectly”. When anything works better than it has any right to you first suspect that you have enthusiastically kicked yourself in the anatomy and fouled up your coding. But, no, coding perfect.

Next step. What’s the actual data?¹ Plot it out, take a look, see what’s going on. And it was curious. About half the data seemed to saturate at a lower limit of about $0.1$, the other half went down to $0.01$². Well that’s a bit odd.

And what’s more, the upper half of the data all had $\text{outcome}=\text{yes}$, the lower half all had $\text{outcome}=\text{no}$. So this is now something I need to talk to colleague-from-other-function about. So I did. It went something like this.

Your hero then goes away, artificially censors the v2 data, and refits the model. The phrase “coin toss” is appropriate at this stage of the discussion.

But, I once built a perfect model.