Feature Selection techniques

SHAP, IV, VIF, Bivariate Analysis, Correlation & Feature Importance — A Complete Guide

When it comes to feature selection and interpretation in machine learning, there’s no shortage of tools. But knowing which method to use, when, and why can be confusing.

In this guide, we’ll break down six popular techniques — SHAP, Information Value (IV), Variance Inflation Factor (VIF), bivariate analysis, correlation, and feature importance — exploring their purpose, pros, cons, similarities, differences, and when to use them for numerical and categorical features.

---

1. SHAP (SHapley Additive exPlanations)

Purpose:
Explains individual predictions by calculating each feature’s contribution, inspired by cooperative game theory.

Why use it:

• Works for any model — from decision trees to deep learning.

• Offers both local (per observation) and global (overall) explanations.

• Handles feature interactions.

• Works with numerical and categorical features (native for trees, encoding needed for others).

Limitations:

• Computationally heavy for large datasets.

• Needs a fitted model.

• Interpretation can be tricky at first.

Best for: Explaining complex, high-stakes models where transparency is key.

---

2. Information Value (IV)

Purpose:
Measures how well a variable separates two classes — ideal for binary classification problems.

Why use it:

• Simple and easy to interpret.

• Great for initial pre-model feature selection.

• Doesn’t require a model.

Limitations:

• Only works for binary targets.

• Ignores interactions between features.

• Continuous variables need binning.

Best for: Credit scoring, risk modeling, and other binary classification tasks.

---

3. Variance Inflation Factor (VIF)

Purpose:
Detects multicollinearity in regression by showing how much a variable is explained by other variables.

Why use it:

• Highlights redundant predictors.

• Improves regression stability and interpretability.

Limitations:

• Only relevant for linear regression.

• Requires numerical or dummy-encoded categorical variables.

• Not helpful for tree-based models.

Best for: Preprocessing before running regression models.

---

4. Bivariate Analysis

Purpose:
Examines the relationship between one feature and the target — often through visual summaries like group means or bar plots.

Why use it:

• Intuitive and visual.

• Works for any feature type.

Limitations:

• Only looks at one feature at a time.

• Doesn’t provide a formal quantitative score.

Best for: Early exploratory data analysis (EDA) to spot obvious patterns.

---

5. Correlation

Purpose:
Measures linear association between two variables.

Why use it:

• Quick, easy, and interpretable.

• Useful for spotting redundancy.

Limitations:

• Only captures linear relationships.

• Pairwise only — misses more complex multicollinearity.

• Sensitive to outliers.

Best for: Quick checks for related features before modeling.

---

6. Feature Importance

Purpose:
Shows how much each feature contributes to predictions in a trained model.

Why use it:

• Model-driven insights.

• Works for any model type.

• Handles feature interactions.

Limitations:

• Can be biased if features are correlated.

• Requires a trained model.

• May vary depending on algorithm.

Best for: Post-model analysis and refining models.

---

Comparison at a Glance

Method	Purpose	Pros	Cons	Numerical	Categorical
SHAP	Explain predictions	Handles interactions	Slow, complex	Yes	Yes
IV	Pre-model selection	Simple, interpretable	Binary only, binning needed	Yes (bin)	Yes
VIF	Multicollinearity	Regression stability	Linear only	Yes	Yes (encode)
Bivariate Analysis	Relationship check	Visual, simple	No interactions	Yes (bin)	Yes
Correlation	Association check	Simple, fast	Linear only, pairwise	Yes	Yes (encode)
Feature Importance	Model-driven	Handles interactions	Needs model, bias possible	Yes	Yes

---

Similarities & Differences

Similarities:

• All assist with feature selection.

• Most work with both numerical and categorical data (some need encoding).

• Some methods are pre-model (IV, bivariate, correlation), others post-model (SHAP, feature importance).

Differences:

• SHAP and feature importance require a trained model.

• VIF and correlation both assess redundancy, but VIF considers all features together while correlation is pairwise.

• IV works only for binary targets.

---

Key Takeaways

• For early feature selection: Use IV, bivariate analysis, and correlation.

• For redundancy checks in regression: Use VIF.

• For interpreting model predictions: Use SHAP and feature importance.

• Always remember: encoding matters for some methods, especially correlation and VIF.

---