📘 Regression Model Evaluation: Credit Limit Assignment

🎯 The Scenario

You're building an XGBoost regression model that predicts:

"How much credit can we safely give this loan applicant?"

Input: Application data, financial history, behavior data Output (y): A dollar amount (e.g., $10,000) — this is continuous, not a yes/no

Since we're predicting a number (not a class), we can't use AUC, KS, or accuracy. We need regression-specific metrics.

---

1️⃣ The Core Metrics: How to Measure "How Wrong" the Model Is

Think of these as different ways to ask: "How far off were our predictions?"

📊 The Five Key Metrics (Simplified)

Code

┌──────────────────────────────────────────────────────────────┐
│                                                              │
│  Predicted: $8,000   Actual: $10,000   Error: -$2,000        │
│                                                              │
│  Different metrics measure this gap differently:             │
│                                                              │
└──────────────────────────────────────────────────────────────┘

Metric	What It Tells You	Easy Example
RMSE	Punishes BIG mistakes more harshly	RMSE = $3,000 → occasional huge errors (like predicting $50K when answer is $10K)
MAE	Average error in dollars	MAE = $2,000 → typical prediction is $2K off
MAPE	Average error as a %	MAPE = 15% → predictions are typically 15% off
R²	% of variation the model explains	R² = 0.65 → model explains 65% of why limits differ
Spearman	How well rankings match (not exact $)	0.8 → applicants ranked high by model usually do get high limits

---

🔍 Quick Visual: RMSE vs MAE

Code

Actual limits:    $10K, $10K, $10K, $10K
Predictions:      $9K,  $9K,  $9K,  $50K   ← One big mistake!

MAE  = average of |errors|       = $11K (smoothed)
RMSE = sqrt(average of errors²)  = $20K (BIG mistake stands out!)

→ Use RMSE when big errors are costly (like over-lending)
→ Use MAE when you want a fair average

---

💡 Why Each Metric Matters in Credit Context

Code

┌────────────────────────────────────────────────────────┐
│                                                        │
│  RMSE → "Did we make any DISASTROUS predictions?"      │
│         (Giving $50K to someone who can repay $10K)    │
│                                                        │
│  MAE  → "What's our TYPICAL miss in dollars?"          │
│         (Off by ~$2K on average)                       │
│                                                        │
│  MAPE → "How proportional is our error?"               │
│         ($1K miss on a $5K limit = bigger problem      │
│          than $1K miss on a $50K limit)                │
│                                                        │
│  R²   → "Is our model better than just guessing        │
│          the average?"                                 │
│                                                        │
│  Spearman → "Do we at least rank people correctly?"    │
│             (Even if dollars are slightly off)         │
│                                                        │
└────────────────────────────────────────────────────────┘

---

2️⃣ What's "Good Enough"? Industry Thresholds

These are typical benchmarks (varies by business, but useful guidelines):

Metric	Acceptable Range	Why It Matters
RMSE	≤ 15–25% of credit range	Prevents catastrophic over-lending
MAE	≤ 10–20% of credit range	Keeps typical errors within tolerance
MAPE	≤ 30–40%	Ensures fairness across small & large limits
R²	≥ 0.5 (≥ 0.4 for noisy data)	Model meaningfully explains differences
Spearman	≥ 0.6–0.7	Critical for ranking applicants into tiers

📌 Example Sanity Check

Code

Credit limits range from $0 to $50,000

✅ RMSE = $3,000   →  6% of range  → GOOD
✅ MAE  = $2,000   →  4% of range  → GREAT
⚠️ MAPE = 35%      →  Borderline   → OK
✅ R²   = 0.65     →  65% explained → GOOD  
✅ Spearman = 0.75 →  Rankings solid → GOOD

Model status: APPROVED ✅

---

3️⃣ Why Rank Ordering Often Matters MORE Than Exact $$

Here's a crucial insight that surprises many people:

In credit risk, the EXACT predicted dollar amount usually gets adjusted by business rules anyway. What matters most is the RANKING — does the model correctly order applicants from low-risk to high-risk?

🎯 Visual Example

Code

┌──────────────────────────────────────────────────────────┐
│                                                          │
│  Applicants and Model Predictions:                       │
│                                                          │
│  Person A: Model predicts $8K, gets approved for $7K    │
│  Person B: Model predicts $15K, gets approved for $14K  │
│  Person C: Model predicts $25K, gets approved for $20K  │
│                                                          │
│  ✅ Exact $ may differ from approvals                    │
│  ✅ BUT the ORDER (A < B < C) is correct                 │
│  ✅ Business policy makes final $ adjustments            │
│                                                          │
│  → Spearman rank correlation captures this perfectly     │
│                                                          │
└──────────────────────────────────────────────────────────┘

Why This Matters for Regulators

• Approval tiers (Tier 1, 2, 3) depend on relative ranking
• Spearman correlation is often a regulatory requirement
• A model with slightly worse RMSE but better Spearman is often preferred ✅

---

4️⃣ Stability Checks: Does the Model Hold Up Over Time?

Accuracy on test data isn't enough. You need to know:

"Will this model still work in 6 months when customer behavior shifts?"

Two Critical Stability Tests

🔄 PSI (Population Stability Index)

What it checks: Are your input features behaving similarly now vs when you trained?

Code

Training time:     Today (6 months later):
                                          
Income distribution:    Income distribution:
$30K-$50K: 40%         $30K-$50K: 25%  ← SHIFTED!
$50K-$80K: 40%         $50K-$80K: 35%
$80K+:     20%         $80K+:     40%

PSI = high → DANGER!
Population has changed → model may fail

PSI Score	Interpretation
< 0.10	✅ Stable — no action needed
0.10 – 0.25	⚠️ Slight shift — monitor
> 0.25	🚨 Major shift — retrain model

For regression: Apply PSI on binned features, not on the target value.

🎯 CSI (Characteristic Stability Index)

What it checks: Is the relationship between a feature and the target stable?

Code

Example: "Does income still predict credit limit the same way?"

Training: Income $50K → avg limit $15K
Today:    Income $50K → avg limit $12K  ← Relationship shifted!

For regression:

1. Bin the feature into groups
2. Calculate mean target per bin (training vs production)
3. Compare distributions

---

🛠️ Plus the Standard Diagnostics

Tool	Purpose
Feature Importance (XGBoost gain/cover)	Which features drive predictions?
SHAP values	Explain individual predictions to regulators

---

5️⃣ The Final Approval Decision

✅ Strong Approval Case

Code

┌─────────────────────────────────────────────┐
│  ✅ RMSE/MAE/MAPE within targets             │
│  ✅ R² ≥ 0.5                                 │
│  ✅ Spearman ≥ 0.65                          │
│  ✅ PSI < 0.10 (stable population)           │
│  ✅ CSI < 0.10 (stable relationships)        │
│                                             │
│  → APPROVED with confidence ✅               │
└─────────────────────────────────────────────┘

⚠️ Borderline Cases (Still Approvable)

A model can be approved even if one metric is weak, IF:

• ✅ It beats the current champion model
• ✅ It's more stable over time
• ✅ It's more explainable for regulators
• ✅ It's policy-compliant

🚫 Likely Rejection

Code

❌ Multiple accuracy metrics fail
❌ AND stability checks fail
→ High risk → REJECT or rebuild

---

🎯 The "Remember Forever" Cheat Sheet

Code

┌──────────────────────────────────────────────────────────┐
│                                                          │
│  📌 ACCURACY METRICS (How wrong are predictions?)        │
│     • RMSE → punishes big errors                         │
│     • MAE  → average error in $                          │
│     • MAPE → average error in %                          │
│     • R²   → % of variance explained                     │
│     • Spearman → ranking accuracy                        │
│                                                          │
│  📌 STABILITY METRICS (Will model still work later?)     │
│     • PSI → are features stable over time?               │
│     • CSI → are feature-target relationships stable?     │
│                                                          │
│  📌 EXPLAINABILITY (Can we justify to regulators?)       │
│     • Feature Importance → which features matter?        │
│     • SHAP → why this specific prediction?               │
│                                                          │
│  📌 KEY INSIGHT                                          │
│     Ranking > Exact $ Value                              │
│     (Policy rules adjust amounts anyway)                 │
│                                                          │
└──────────────────────────────────────────────────────────┘

---

💻 Quick Code Reference

regression_evaluation.py

import numpy as np
from sklearn.metrics import (
    mean_squared_error, 
    mean_absolute_error,
    mean_absolute_percentage_error,
    r2_score

Sample Output:

Code

RMSE:     $3,000
MAE:      $2,000
MAPE:     15%
R²:       0.65
Spearman: 0.75

---