Decision Trees in Loan Repayment: Stop Using Accuracy as Your “Effectiveness” Metric

When predicting loan repayment, the word “effective” shouldn't be equated with “high accuracy.” For imbalanced problems—where most customers repay and only a few default—a model that always predicts "repay" can easily reach 90%+ accuracy and still be useless.

Why accuracy fails

• Accuracy treats every mistake equally. In lending, a missed defaulter (false negative) and wrongly rejecting a good borrower (false positive) usually have very different business costs.
• Example: If 95% of customers repay, a model that predicts "repay" for everyone scores 95% accuracy but catches zero defaulters.

Metrics you should insist on

Tie the metric to business impact. In interviews and model evaluations, prefer cost-aware and class-sensitive metrics:

• Recall (sensitivity) for defaulters: how many actual defaulters the model catches. Use this to reduce false negatives.
• Precision for defaulters: of those flagged as likely to default, how many actually default. Use this to avoid rejecting too many good customers.
• F1 score: harmonic mean of precision and recall to balance the two when you need a single number.
• AUC-ROC: measures ranking quality across thresholds; useful but can be misleading with heavy class imbalance.
• PR-AUC (Precision-Recall AUC): often more informative than ROC when defaults are rare because it focuses on the positive (default) class.

Also evaluate calibration (do predicted probabilities match actual rates?) so you can set meaningful business thresholds.

Always connect metrics to business loss

Define a cost matrix: specify the loss (in dollars or business impact) for false negatives and false positives. Then:

• Optimize expected business loss (or a profit/loss metric) rather than raw accuracy.
• Choose the threshold that minimizes expected loss or meets a risk policy constraint (e.g., max allowable portfolio default rate).

This turns model evaluation into a business decision, not a modeling vanity contest.

Practical evaluation checklist

1. Compute a confusion matrix on a holdout/test set.
2. Report recall, precision, F1, ROC-AUC and PR-AUC; include calibration plots.
3. Translate false positives/negatives into monetary (or operational) cost and compute expected loss per threshold.
4. Use stratified cross-validation for stable estimates with imbalanced classes.
5. If needed, use class weighting, resampling, or thresholding—not accuracy—to address imbalance.
6. Monitor model performance and business KPIs after deployment; distribution shifts can change the optimal threshold.

Interview tips

• Ask what the business cares about: cost per default, cost to acquire/lose customers, or regulatory limits.
• Show how you'd convert model outputs to a decision policy (thresholds, accept/reject rules) based on expected loss.
• Present ROC and PR curves, plus calibration, and recommend a threshold with its expected business impact.

Bottom line

Accuracy is easy to compute but often meaningless for loan-repayment models with imbalanced outcomes. Use recall, precision, F1, PR-AUC, calibration and—crucially—cost-based metrics that reflect business loss and risk policy. Insist on those in interviews and in production.

#MachineLearning #DataScience #MLOps