A model that predicts churn accurately but can't change what you do is a report, not a tool. Several common errors flatter the accuracy.
Pitfalls
- Leakage — using information only available after the customer decided to leave.
- Fuzzy definitions — 'churn' differs across products and contracts; fix the definition first.
- Wrong evaluation — random splits that ignore time; test on later periods.
- No action — a score with no retention play attached.
- Targeting the customers who would have stayed anyway.
Judge the model by the result of the intervention in a controlled comparison, not only by prediction accuracy.