┌─────────────────────────────────────────────┐
│ │
│ LOADING: │
│ FP16 weights │
│ ↓ ÷ absmax │
│ Normalized [-1, 1] │
│ ↓ snap to nearest bin │
│ 4-bit index + scale stored in GPU │
│ │
│ RUNNING (every forward pass): │
│ 4-bit index │
│ ↓ look up bin value │
│ NF4 level │
│ ↓ × scale │
│ Approximate FP16 value → used in matmul │
│ │
└─────────────────────────────────────────────┘
Why It Works
Code
Original weights: [0.80, 0.30, -0.50, 1.20, -1.00, 0.10, 0.60, -0.90]
↓ matrix multiply with input
Original output: 42.5
4-bit weights: [0.84, 0.36, -0.48, 1.20, -0.84, 0.12, 0.60, -0.84]
↓ matrix multiply with same input
4-bit output: 42.3
Close enough! The model still picks the same next token.
The small errors in individual weights average out across thousands of multiplications.
Significance of covariance & meaning of high vs low values
Covariance measures how two features vary together:
Positive covariance → When feature is above its mean, feature tends to also be above its mean. (They move in the same direction.)
Negative covariance → When is above its mean, tends to be below its mean. (They move in opposite directions.)
Near zero covariance → No consistent linear relationship — knowing one feature’s deviation from the mean tells you nothing about the other.
Numerically:
Large magnitude (positive or negative) means strong linear relationship.
Small magnitude means weak or no relationship.
Why covariance matters in PCA
The covariance matrix encodes all pairwise relationships between features.
If features are highly correlated (large positive or negative covariance), PCA will combine them into a principal component that captures their shared variation, so you don’t have redundancy.
If covariances are near zero, features are largely independent; PCA will mostly keep them separate unless variances are drastically different.
💡 Analogy
Think of the covariance matrix as a “map” of how all features move together.
The eigenvectors are “routes” through this map that maximize variance.
PCA rotates your view to look along those routes, and you project the original data (not the covariance matrix itself) into that rotated view.
Let’s unpack this step-by-step — because R², RMSE, and MAE all involve “difference between actual and predicted,” but they measure different things and answer different questions.
1. R² (Coefficient of Determination) — Variance Captured
Think of your target values (y_actual) as having some spread (variance) around their mean.
If you didn’t have a model and just guessed the mean for everyone, that’s your baseline.
R² asks:
"How much better is my model compared to just guessing the mean every time?"
Formula
Where:
SSE_model = Σ(Actual − Predicted)²
SSE_mean = Σ(Actual − Mean)²
Intuition
R² = 1.0 → Model perfectly predicts all values (100% of variance explained).
R² = 0.0 → Model is no better than guessing the mean.
R² < 0.0 → Model is worse than guessing the mean (ouch).
Predicted: 9k, 13k, 14k, 21k, 26k →
SSE_model is much smaller than SSE_mean → R² ≈ 0.85.
This means the model explains 85% of the variation in limits between customers.
2. RMSE & MAE — Error Magnitude
These do not compare to a baseline — they tell you how far off predictions are, on average.
RMSE penalizes large mistakes more heavily than MAE (because it squares the errors before averaging).
Both are absolute accuracy metrics, not relative to variance.
You're building an XGBoost regression model that predicts:
"How much credit can we safely give this loan applicant?"
Input: Application data, financial history, behavior data Output (y): A dollar amount (e.g., $10,000) — this is continuous, not a yes/no
Since we're predicting a number (not a class), we can't use AUC, KS, or accuracy. We need regression-specific metrics.
1️⃣ The Core Metrics: How to Measure "How Wrong" the Model Is
Think of these as different ways to ask: "How far off were our predictions?"
📊 The Five Key Metrics (Simplified)
Code
┌──────────────────────────────────────────────────────────────┐
│ │
│ Predicted: $8,000 Actual: $10,000 Error: -$2,000 │
│ │
│ Different metrics measure this gap differently: │
│ │
└──────────────────────────────────────────────────────────────┘
Metric
What It Tells You
Easy Example
RMSE
Punishes BIG mistakes more harshly
RMSE = $3,000 → occasional huge errors (like predicting $50K when answer is $10K)
MAE
Average error in dollars
MAE = $2,000 → typical prediction is $2K off
MAPE
Average error as a %
MAPE = 15% → predictions are typically 15% off
R²
% of variation the model explains
R² = 0.65 → model explains 65% of why limits differ
Spearman
How well rankings match (not exact $)
0.8 → applicants ranked high by model usually do get high limits
🔍 Quick Visual: RMSE vs MAE
Code
Actual limits: $10K, $10K, $10K, $10K
Predictions: $9K, $9K, $9K, $50K ← One big mistake!
MAE = average of |errors| = $11K (smoothed)
RMSE = sqrt(average of errors²) = $20K (BIG mistake stands out!)
→ Use RMSE when big errors are costly (like over-lending)
→ Use MAE when you want a fair average
💡 Why Each Metric Matters in Credit Context
Code
┌────────────────────────────────────────────────────────┐
│ │
│ RMSE → "Did we make any DISASTROUS predictions?" │
│ (Giving $50K to someone who can repay $10K) │
│ │
│ MAE → "What's our TYPICAL miss in dollars?" │
│ (Off by ~$2K on average) │
│ │
│ MAPE → "How proportional is our error?" │
│ ($1K miss on a $5K limit = bigger problem │
│ than $1K miss on a $50K limit) │
│ │
│ R² → "Is our model better than just guessing │
│ the average?" │
│ │
│ Spearman → "Do we at least rank people correctly?" │
│ (Even if dollars are slightly off) │
│ │
└────────────────────────────────────────────────────────┘
2️⃣ What's "Good Enough"? Industry Thresholds
These are typical benchmarks (varies by business, but useful guidelines):
Metric
Acceptable Range
Why It Matters
RMSE
≤ 15–25% of credit range
Prevents catastrophic over-lending
MAE
≤ 10–20% of credit range
Keeps typical errors within tolerance
MAPE
≤ 30–40%
Ensures fairness across small & large limits
R²
≥ 0.5 (≥ 0.4 for noisy data)
Model meaningfully explains differences
Spearman
≥ 0.6–0.7
Critical for ranking applicants into tiers
📌 Example Sanity Check
Code
Credit limits range from $0 to $50,000
✅ RMSE = $3,000 → 6% of range → GOOD
✅ MAE = $2,000 → 4% of range → GREAT
⚠️ MAPE = 35% → Borderline → OK
✅ R² = 0.65 → 65% explained → GOOD
✅ Spearman = 0.75 → Rankings solid → GOOD
Model status: APPROVED ✅
3️⃣ Why Rank Ordering Often Matters MORE Than Exact $$
Here's a crucial insight that surprises many people:
In credit risk, the EXACT predicted dollar amount usually gets adjusted by business rules anyway. What matters most is the RANKING — does the model correctly order applicants from low-risk to high-risk?
🎯 Visual Example
Code
┌──────────────────────────────────────────────────────────┐
│ │
│ Applicants and Model Predictions: │
│ │
│ Person A: Model predicts $8K, gets approved for $7K │
│ Person B: Model predicts $15K, gets approved for $14K │
│ Person C: Model predicts $25K, gets approved for $20K │
│ │
│ ✅ Exact $ may differ from approvals │
│ ✅ BUT the ORDER (A < B < C) is correct │
│ ✅ Business policy makes final $ adjustments │
│ │
│ → Spearman rank correlation captures this perfectly │
│ │
└──────────────────────────────────────────────────────────┘
Why This Matters for Regulators
Approval tiers (Tier 1, 2, 3) depend on relative ranking
Spearman correlation is often a regulatory requirement
A model with slightly worse RMSE but better Spearman is often preferred ✅
4️⃣ Stability Checks: Does the Model Hold Up Over Time?
Accuracy on test data isn't enough. You need to know:
"Will this model still work in 6 months when customer behavior shifts?"
Two Critical Stability Tests
🔄 PSI (Population Stability Index)
What it checks: Are your input features behaving similarly now vs when you trained?
Code
Training time: Today (6 months later):
Income distribution: Income distribution:
$30K-$50K: 40% $30K-$50K: 25% ← SHIFTED!
$50K-$80K: 40% $50K-$80K: 35%
$80K+: 20% $80K+: 40%
PSI = high → DANGER!
Population has changed → model may fail
PSI Score
Interpretation
< 0.10
✅ Stable — no action needed
0.10 – 0.25
⚠️ Slight shift — monitor
> 0.25
🚨 Major shift — retrain model
For regression: Apply PSI on binned features, not on the target value.
🎯 CSI (Characteristic Stability Index)
What it checks: Is the relationship between a feature and the target stable?
Code
Example: "Does income still predict credit limit the same way?"
Training: Income $50K → avg limit $15K
Today: Income $50K → avg limit $12K ← Relationship shifted!
For regression:
Bin the feature into groups
Calculate mean target per bin (training vs production)
let’s frame this in a churn prediction context, because that’s a very common case where people see IV, PSI, and CSI all being used, notice that the formulas look similar, but get confused about why they’re treated differently.
1️⃣ The setting — churn prediction
Target: churn_flag (1 = churned, 0 = stayed).
Feature: avg_monthly_usage (average minutes per month).
Goal: Build a model that predicts churn, and also monitor if the feature is stable over time.
We have:
Train set → Customers from Jan–Mar 2025
OOT1 → Customers from Apr 2025
OOT2 → Customers from May 2025
2️⃣ The same base formula — different contexts
The mathematical core of IV, PSI, and CSI is a weighted log ratio:
The difference is what those “fractions” mean and which datasets are compared.
3️⃣ Information Value (IV)
Question: Does this feature separate churners from non-churners in a single dataset?
Fractions:
= fraction of stayers in that bin (within train set)
= fraction of churners in that bin (within train set)
Data involved: Only one dataset (e.g., Train).
Use: Feature selection — keep features with high IV (e.g., > 0.02).
