Nabid In Motion Nabid In Motion

Study interactive :: Progress tools open in the Study Hub reader.

All lessons in this module

Previous lesson. Module 20: Handling Imbalanced Data · Next lesson. Advanced Imbalanced Data Topics

Handling Imbalanced Data

This guide covers handling imbalanced datasets in machine learning. Learn to build effective models when classes are not equally represented, with detailed explanations, code examples, and real-world applications.

Table of Contents

Introduction to Imbalanced Data

What is Imbalanced Data?

Imbalanced data occurs when classes in a classification problem are not represented equally. This is extremely common in real-world machine learning problems.

Real-World Examples:

Why is it a Problem?

The Core Issue: Most machine learning algorithms are designed assuming balanced classes. When classes are imbalanced, these algorithms tend to:

  1. Predict the majority class most of the time
  2. Achieve high accuracy by simply predicting the majority class
  3. Ignore the minority class. Which is often the most important

Example:

# Imbalanced dataset: 99% class 0 (normal), 1% class 1 (fraud)
# Naive model: Always predict class 0
# Result:
#   - Accuracy: 99% (looks great!)
#   - Precision for fraud: 0% (catches no fraud)
#   - Recall for fraud: 0% (misses all fraud)
#   - Business impact: Catastrophic! All fraud goes undetected

Why Accuracy is Misleading:

from sklearn.metrics import accuracy_score

# Imbalanced dataset
y_true = [0]*990 + [1]*10  # 990 normal, 10 fraud
y_pred = [0]*1000  # Always predict normal

accuracy = accuracy_score(y_true, y_pred)
print(f"Accuracy: {accuracy:.2%}")  # 99% - looks great!
print("But caught 0 fraud cases!")  # Actually terrible!

When Does Imbalance Matter?

Imbalance is a problem when:

Imbalance might not matter when:

Understanding the Problem

Measuring Imbalance

import pandas as pd
import numpy as np
from collections import Counter

# Check class distribution
def check_imbalance(y):
    """Check class distribution"""
    counter = Counter(y)
    total = len(y)
    
    print("Class Distribution:")
    for class_label, count in counter.items():
        percentage = (count / total) * 100
        print(f"  Class {class_label}: {count} ({percentage:.2f}%)")
    
    # Calculate imbalance ratio
    counts = list(counter.values())
    imbalance_ratio = max(counts) / min(counts)
    print(f"\nImbalance Ratio: {imbalance_ratio:.2f}:1")
    
    if imbalance_ratio > 10:
        print("Warning: Highly imbalanced!")
    elif imbalance_ratio > 5:
        print("Warning: Moderately imbalanced")
    else:
        print("OK: Relatively balanced")

# Example
y = np.array([0]*900 + [1]*100)  # 90% vs 10%
check_imbalance(y)

Why Algorithms Fail

Most ML algorithms assume:

Result:

Resampling Techniques

Oversampling

Increase minority class samples.

Random Oversampling

from imblearn.over_sampling import RandomOverSampler

# Random oversampling
ros = RandomOverSampler(random_state=42)
X_resampled, y_resampled = ros.fit_resample(X_train, y_train)

print(f"Original: {Counter(y_train)}")
print(f"After oversampling: {Counter(y_resampled)}")

SMOTE (Synthetic Minority Oversampling Technique)

Creates synthetic samples.

from imblearn.over_sampling import SMOTE

# SMOTE
smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X_train, y_train)

# SMOTE variants
from imblearn.over_sampling import BorderlineSMOTE, ADASYN, SVMSMOTE

# Borderline SMOTE (focuses on borderline samples)
borderline_smote = BorderlineSMOTE(random_state=42)
X_borderline, y_borderline = borderline_smote.fit_resample(X_train, y_train)

# ADASYN (adaptive synthetic sampling)
adasyn = ADASYN(random_state=42)
X_adasyn, y_adasyn = adasyn.fit_resample(X_train, y_train)

# SVM SMOTE (uses SVM to find support vectors)
svm_smote = SVMSMOTE(random_state=42)
X_svm, y_svm = svm_smote.fit_resample(X_train, y_train)

Undersampling

Reduce majority class samples.

Random Undersampling

from imblearn.under_sampling import RandomUnderSampler

# Random undersampling
rus = RandomUnderSampler(random_state=42)
X_resampled, y_resampled = rus.fit_resample(X_train, y_train)

Remove borderline majority samples.

from imblearn.under_sampling import TomekLinks

# Tomek Links
tomek = TomekLinks()
X_resampled, y_resampled = tomek.fit_resample(X_train, y_train)

Edited Nearest Neighbors

from imblearn.under_sampling import EditedNearestNeighbours

# Edited Nearest Neighbors
enn = EditedNearestNeighbours()
X_resampled, y_resampled = enn.fit_resample(X_train, y_train)

Combined Methods

Combine oversampling and undersampling.

from imblearn.combine import SMOTETomek, SMOTEENN

# SMOTE + Tomek Links
smote_tomek = SMOTETomek(random_state=42)
X_resampled, y_resampled = smote_tomek.fit_resample(X_train, y_train)

# SMOTE + Edited Nearest Neighbors
smote_enn = SMOTEENN(random_state=42)
X_resampled, y_resampled = smote_enn.fit_resample(X_train, y_train)

Algorithm-Level Solutions

Class Weights

Penalize misclassifying minority class more.

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier

# Method 1: Balanced (automatic)
model = LogisticRegression(class_weight='balanced', random_state=42)
model.fit(X_train, y_train)

# Method 2: Custom weights
class_weights = {0: 1, 1: 10}  # Penalize class 1 misclassification 10x
model = LogisticRegression(class_weight=class_weights, random_state=42)
model.fit(X_train, y_train)

# Method 3: Compute from data
from sklearn.utils.class_weight import compute_class_weight

classes = np.unique(y_train)
weights = compute_class_weight('balanced', classes=classes, y=y_train)
class_weight_dict = dict(zip(classes, weights))

model = RandomForestClassifier(class_weight=class_weight_dict, random_state=42)
model.fit(X_train, y_train)

Threshold Tuning

Adjust classification threshold.

from sklearn.metrics import precision_recall_curve

# Get probabilities
y_proba = model.predict_proba(X_test)[:, 1]

# Find optimal threshold
precision, recall, thresholds = precision_recall_curve(y_test, y_proba)

# F1 score for each threshold
f1_scores = 2 * (precision * recall) / (precision + recall)
optimal_idx = np.argmax(f1_scores)
optimal_threshold = thresholds[optimal_idx]

print(f"Optimal threshold: {optimal_threshold:.3f}")

# Predict with optimal threshold
y_pred_optimal = (y_proba >= optimal_threshold).astype(int)

Cost-Sensitive Learning

# Use cost matrix
from sklearn.metrics import make_scorer

def cost_sensitive_score(y_true, y_pred):
    """Custom scoring with different costs"""
    # Cost of false negative (missing fraud) is higher
    cost_matrix = {
        (0, 0): 0,   # True negative
        (0, 1): 1,   # False positive
        (1, 0): 10,  # False negative (expensive!)
        (1, 1): 0    # True positive
    }
    
    total_cost = 0
    for true, pred in zip(y_true, y_pred):
        total_cost += cost_matrix[(true, pred)]
    
    return -total_cost  # Negative because higher is better

cost_scorer = make_scorer(cost_sensitive_score, greater_is_better=True)

Evaluation Metrics

Appropriate Metrics for Imbalanced Data

Don't use:

Use instead:

from sklearn.metrics import (
    classification_report, confusion_matrix,
    precision_recall_curve, roc_curve, roc_auc_score,
    average_precision_score
)

# Classification report
print(classification_report(y_test, y_pred))

# Confusion matrix
cm = confusion_matrix(y_test, y_pred)
print("\nConfusion Matrix:")
print(cm)

# Precision-Recall curve
precision, recall, _ = precision_recall_curve(y_test, y_proba)
pr_auc = average_precision_score(y_test, y_proba)
print(f"\nPrecision-Recall AUC: {pr_auc:.3f}")

# ROC curve
fpr, tpr, _ = roc_curve(y_test, y_proba)
roc_auc = roc_auc_score(y_test, y_proba)
print(f"ROC AUC: {roc_auc:.3f}")

Best Practices

When to Use What

Technique When to Use Pros Cons
SMOTE Moderate imbalance Creates synthetic samples Can create noise
Class Weights Any imbalance No data loss May not work for extreme imbalance
Undersampling Large dataset Faster training Loss of information
Threshold Tuning After training Simple, effective Need validation set
Ensemble Severe imbalance Robust Complex

Workflow

  1. Analyze imbalance: Check class distribution
  2. Choose strategy: Based on dataset size and imbalance ratio
  3. Use appropriate metrics: PR-AUC, F1-score
  4. Validate properly: Stratified cross-validation
  5. Compare methods: Try multiple approaches

Practice Exercises

Exercise 1: Handle Imbalanced Data

Task: Apply different techniques to imbalanced dataset and compare.

Solution:

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from imblearn.over_sampling import SMOTE
from imblearn.under_sampling import RandomUnderSampler
from imblearn.combine import SMOTETomek

# Create imbalanced dataset
X, y = make_classification(
    n_samples=1000,
    n_classes=2,
    weights=[0.9, 0.1],  # 90% vs 10%
    random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Strategy 1: No resampling (baseline)
model1 = RandomForestClassifier(random_state=42)
model1.fit(X_train, y_train)
score1 = model1.score(X_test, y_test)

# Strategy 2: SMOTE
smote = SMOTE(random_state=42)
X_smote, y_smote = smote.fit_resample(X_train, y_train)
model2 = RandomForestClassifier(random_state=42)
model2.fit(X_smote, y_smote)
score2 = model2.score(X_test, y_test)

# Strategy 3: Class weights
model3 = RandomForestClassifier(class_weight='balanced', random_state=42)
model3.fit(X_train, y_train)
score3 = model3.score(X_test, y_test)

print(f"Baseline: {score1:.3f}")
print(f"SMOTE: {score2:.3f}")
print(f"Class Weights: {score3:.3f}")

Complete Workflow Example

Let's walk through a complete example of handling imbalanced data from start to finish.

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score, average_precision_score
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import Pipeline as ImbPipeline
import matplotlib.pyplot as plt
import seaborn as sns

# Step 1: Create imbalanced dataset
X, y = make_classification(
    n_samples=10000,
    n_features=20,
    n_informative=15,
    n_redundant=5,
    n_classes=2,
    weights=[0.95, 0.05],  # 95% vs 5% - highly imbalanced
    random_state=42
)

# Step 2: Check imbalance
print("Class distribution:")
print(f"Class 0: {np.sum(y == 0)} ({np.sum(y == 0)/len(y)*100:.2f}%)")
print(f"Class 1: {np.sum(y == 1)} ({np.sum(y == 1)/len(y)*100:.2f}%)")

# Step 3: Split data (use stratified split!)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Step 4: Baseline model (no handling)
print("\n=== Baseline Model (No Handling) ===")
baseline_model = RandomForestClassifier(n_estimators=100, random_state=42)
baseline_model.fit(X_train, y_train)
y_pred_baseline = baseline_model.predict(X_test)

print("\nClassification Report:")
print(classification_report(y_test, y_pred_baseline))
print(f"\nROC-AUC: {roc_auc_score(y_test, baseline_model.predict_proba(X_test)[:, 1]):.3f}")
print(f"PR-AUC: {average_precision_score(y_test, baseline_model.predict_proba(X_test)[:, 1]):.3f}")

# Step 5: Apply SMOTE
print("\n=== Model with SMOTE ===")
smote = SMOTE(random_state=42)
X_train_smote, y_train_smote = smote.fit_resample(X_train, y_train)

print(f"After SMOTE - Class 0: {np.sum(y_train_smote == 0)}, Class 1: {np.sum(y_train_smote == 1)}")

smote_model = RandomForestClassifier(n_estimators=100, random_state=42)
smote_model.fit(X_train_smote, y_train_smote)
y_pred_smote = smote_model.predict(X_test)

print("\nClassification Report:")
print(classification_report(y_test, y_pred_smote))
print(f"\nROC-AUC: {roc_auc_score(y_test, smote_model.predict_proba(X_test)[:, 1]):.3f}")
print(f"PR-AUC: {average_precision_score(y_test, smote_model.predict_proba(X_test)[:, 1]):.3f}")

# Step 6: Use class weights
print("\n=== Model with Class Weights ===")
weighted_model = RandomForestClassifier(
    n_estimators=100,
    class_weight='balanced',
    random_state=42
)
weighted_model.fit(X_train, y_train)
y_pred_weighted = weighted_model.predict(X_test)

print("\nClassification Report:")
print(classification_report(y_test, y_pred_weighted))
print(f"\nROC-AUC: {roc_auc_score(y_test, weighted_model.predict_proba(X_test)[:, 1]):.3f}")
print(f"PR-AUC: {average_precision_score(y_test, weighted_model.predict_proba(X_test)[:, 1]):.3f}")

# Step 7: Threshold tuning
print("\n=== Model with Threshold Tuning ===")
from sklearn.metrics import precision_recall_curve, f1_score

y_proba = weighted_model.predict_proba(X_test)[:, 1]
precision, recall, thresholds = precision_recall_curve(y_test, y_proba)

# Find optimal threshold (maximize F1)
f1_scores = 2 * (precision * recall) / (precision + recall)
optimal_idx = np.argmax(f1_scores)
optimal_threshold = thresholds[optimal_idx]

print(f"Optimal threshold: {optimal_threshold:.3f}")
y_pred_tuned = (y_proba >= optimal_threshold).astype(int)

print("\nClassification Report:")
print(classification_report(y_test, y_pred_tuned))
print(f"\nF1-Score: {f1_score(y_test, y_pred_tuned):.3f}")

# Step 8: Visualize results
fig, axes = plt.subplots(1, 3, figsize=(18, 5))

# Confusion matrices
models = [
    (y_pred_baseline, "Baseline"),
    (y_pred_smote, "SMOTE"),
    (y_pred_tuned, "Threshold Tuned")
]

for idx, (y_pred, title) in enumerate(models):
    cm = confusion_matrix(y_test, y_pred)
    sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', ax=axes[idx])
    axes[idx].set_title(f'{title}\nF1: {f1_score(y_test, y_pred):.3f}')
    axes[idx].set_xlabel('Predicted')
    axes[idx].set_ylabel('Actual')

plt.tight_layout()
plt.show()

Common Pitfalls

Pitfall 1: Resampling Before Train-Test Split

Wrong:

# Don't do this!
X_resampled, y_resampled = smote.fit_resample(X, y)
X_train, X_test, y_train, y_test = train_test_split(X_resampled, y_resampled)
# Data leakage! Test set information leaked into training

Correct:

# Always split first!
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y)
X_train_resampled, y_train_resampled = smote.fit_resample(X_train, y_train)
# Test set remains untouched

Pitfall 2: Using Accuracy as Metric

Wrong:

accuracy = model.score(X_test, y_test)  # Misleading for imbalanced data!

Correct:

from sklearn.metrics import classification_report, roc_auc_score, average_precision_score

print(classification_report(y_test, y_pred))
roc_auc = roc_auc_score(y_test, y_proba)
pr_auc = average_precision_score(y_test, y_proba)

Pitfall 3: Overfitting to Minority Class

Problem: Oversampling can cause overfitting if done incorrectly.

Solution: Use cross-validation and monitor both classes:

from sklearn.model_selection import cross_validate

cv_results = cross_validate(
    model, X_train_resampled, y_train_resampled,
    cv=StratifiedKFold(5),
    scoring=['precision', 'recall', 'f1', 'roc_auc'],
    return_train_score=True
)

Pitfall 4: Ignoring Business Context

Problem: Choosing technique without considering business costs.

Solution: Incorporate cost-sensitive learning:

# Define cost matrix
cost_matrix = {
    (0, 0): 0,    # True negative: no cost
    (0, 1): 1,    # False positive: small cost
    (1, 0): 100,  # False negative: high cost (missing fraud)
    (1, 1): 0     # True positive: no cost
}

Additional Resources

Online Resources

  1. Imbalanced-Learn Documentation (https://imbalanced-learn.org/): Comprehensive library for handling imbalanced data
  2. SMOTE Paper (Chawla et al., 2002): Original SMOTE algorithm paper
  3. Learning from Imbalanced Data (He & Garcia, 2009): Survey paper on techniques

Books

  1. "Learning from Imbalanced Data Sets" by Alberto Fernández et al. - Comprehensive book on the topic
  2. "Applied Predictive Modeling" by Max Kuhn. Chapter on class imbalance

Datasets for Practice

  1. Credit Card Fraud Detection (Kaggle): Highly imbalanced fraud dataset
  2. Churn Prediction - Customer churn datasets
  3. Medical Diagnosis - Rare disease detection datasets

Key Takeaways

  1. Imbalanced data is common: Especially in real-world problems (fraud, medical, etc.)
  2. Accuracy is misleading: Use precision, recall, F1-score, PR-AUC instead
  3. Multiple solutions exist: Resampling, class weights, threshold tuning, ensemble methods
  4. Choose wisely: Consider dataset size, imbalance ratio, and business context
  5. Validate properly: Always use stratified splits and cross-validation
  6. Avoid data leakage: Never resample before splitting data
  7. Consider costs: Incorporate business costs into your solution
  8. Combine techniques: Often best results come from combining multiple approaches

Try next: Plot precision-recall for your imbalanced task. Pick a threshold from cost, not from accuracy.

Recall ::

Why must you split before SMOTE or other resampling?

Resampling the full set before the split leaks test information into training. Fit resampling on the training split only and leave the test set untouched.

All lessons in this module

Previous lesson. Module 20: Handling Imbalanced Data · Next lesson. Advanced Imbalanced Data Topics