The loan file that rewrites itself

The SBA's public 7(a) file looks like a record of loans as they were made. It isn't. It's a snapshot taken today, and some fields change after the fact.

The clearest case is the loan term. Among loans approved in fiscal 2015 to 2017 that were charged off within 24 months, only 8% show a term that's a whole number of years. Among all the other loans, 77% do. Terms get rewritten when a loan goes bad.

That matters for anyone building a default model. A first version that used the term scored an AUC of 0.98 when tested on later years, far better than credit models usually get. It wasn't predicting default. It was reading a field that had been edited because of the default.

The secondary-market flag has the same problem: it can switch on after origination.

Both fields are now out of the model, and a test fails if any field set after origination comes back in. The honest model ranks loans with an AUC of about 0.68 to 0.80 on later years. That's less impressive, and real.

The general lesson: for every column, ask when it was written. If the answer is "after the thing you're predicting," it can't be a predictor, however good it looks.