Evaluation techniques for regression

let’s make this concrete with credit engine–style examples so the intuition clicks.

Imagine we’re predicting Loss Given Default (LGD) for loans.
We have actual LGD values from historical defaults and model predictions.

---

1. RMSE – Root Mean Squared Error

Example:

• Actual LGD: [0.10, 0.30, 0.90]

• Predicted LGD: [0.12, 0.50, 0.30]

Errors: [0.02, 0.20, -0.60] → squared: [0.0004, 0.04, 0.36] → mean: 0.1335 → sqrt: 0.365

Interpretation:

• The 0.60 miss on the last loan blows up the RMSE because squaring makes big mistakes shout louder.

• RMSE here says: “Your typical big-mistake-weighted error is 36.5 percentage points.”

---

2. MAE – Mean Absolute Error

Same example:
Absolute errors: [0.02, 0.20, 0.60] → mean: 0.273

Interpretation:

• MAE says: “On average, you’re off by 27.3 percentage points.”

• It treats the 0.60 miss the same way as a smaller one, proportionally.

---

3. MAPE – Mean Absolute Percentage Error

Example:
Absolute % errors:
[0.02/0.10 = 20%, 0.20/0.30 ≈ 66.7%, 0.60/0.90 ≈ 66.7%] → mean ≈ 51.1%

Interpretation:

• “On average, you’re off by 51% of the actual LGD value.”

• If an actual LGD is close to zero (e.g., 0.01) and you predict 0.10, MAPE goes crazy.

---

4. R² – Coefficient of Determination

Example:

• If actual LGDs vary a lot, and your predictions capture that variation well, R² will be high.

• If you just predict the average LGD for everyone, R² might be close to 0 — you didn’t explain any variance.

Interpretation:

• R² answers: “How much of the LGD variability did I explain compared to just guessing the mean?”

---

5. Pearson vs. Spearman Correlation

Example:

• Suppose actual LGD ranking (highest to lowest risk): Loan C, Loan B, Loan A.

• Predictions: Loan C is still ranked highest, then A, then B.

Pearson: Could be low if the exact values are off (linear mismatch).
Spearman: Could still be high because the order is mostly right.

Interpretation:

• Pearson: “Do the numbers line up in a straight-line way?”

• Spearman: “Even if I got magnitudes wrong, did I keep the order right?”

---

Summary in credit engine terms:

• RMSE → penalizes big LGD prediction errors heavily (good if big misses are expensive for the bank).

• MAE → gives equal weight to all misses, good for stable reporting.

• MAPE → interprets error in % terms, useful if LGD has consistent scale across products.

• R² → tells if your model adds value beyond a dumb constant guess.

• Spearman → good for prioritization tasks (e.g., which borrowers to monitor first).

---