Chapter 32 — Classification & Prediction

Overview

Design supervised models for classification and regression with emphasis on calibration and deployment economics. This chapter covers the complete workflow from problem formulation to production deployment, focusing on practical techniques that maximize business value while maintaining operational efficiency.

Problem Definition Framework

Classification vs Regression Decision Tree

graph TD
    A[Prediction Problem] --> B{Target Variable Type?}
    B -->|Categorical| C{Number of Classes?}
    B -->|Continuous| D[Regression]

    C -->|Two Classes| E[Binary Classification]
    C -->|Multiple Classes| F{Classes Mutually Exclusive?}

    F -->|Yes| G[Multi-class Classification]
    F -->|No| H[Multi-label Classification]

    E --> I[Evaluation: AUC-ROC, Precision, Recall]
    G --> J[Evaluation: Accuracy, Macro F1]
    H --> K[Evaluation: Hamming Loss, Subset Accuracy]
    D --> L[Evaluation: RMSE, MAE, R²]

Problem Formulation Checklist

Data Preparation & Leakage Prevention

Common Leakage Patterns

graph TD
    A[Data Leakage Sources] --> B[Temporal Leakage]
    A --> C[Target Leakage]
    A --> D[Train-Test Contamination]
    A --> E[Aggregation Leakage]

    B --> B1[Using future data in features]
    B --> B2[Incorrect time cutoffs]

    C --> C1[Features containing outcome]
    C --> C2[Proxy variables for target]

    D --> D1[Preprocessing on full dataset]
    D --> D2[Feature selection on test data]

    E --> E1[Group stats include test samples]
    E --> E2[Global normalizations]

Leakage Detection & Prevention

Feature Engineering Quality Matrix

Model Selection Decision Framework

Algorithm Selection Matrix

Model Selection Flowchart

graph TD
    A[Model Selection] --> B{Dataset Size?}
    B -->|Small <10K| C{Interpretability Required?}
    B -->|Medium 10K-1M| D{Feature Type?}
    B -->|Large >1M| E{Deep Features Needed?}

    C -->|Yes| F[Logistic Regression, Decision Trees]
    C -->|No| G[Random Forest, Gradient Boosting]

    D -->|Tabular| H[XGBoost, LightGBM]
    D -->|Text/Images| I[Deep Learning]
    D -->|Mixed| J[Ensemble Methods]

    E -->|Yes| K[Neural Networks, Transformers]
    E -->|No| L[LightGBM with GPU]

    F --> M[Validate & Tune]
    G --> M
    H --> M
    I --> M
    J --> M
    K --> M
    L --> M

Performance vs Complexity Trade-offs

Model Calibration

Why Calibration Matters

graph LR
    A[Uncalibrated Model] --> B[Predicted Prob: 0.7]
    B --> C[Actual Positive Rate: 0.45]
    C --> D[Decision Making Fails]

    E[Calibrated Model] --> F[Predicted Prob: 0.7]
    F --> G[Actual Positive Rate: 0.68]
    G --> H[Reliable Decisions]

Calibration Techniques Comparison

Calibration Evaluation Metrics

Production Deployment Architecture

Deployment Pattern Decision Tree

graph TD
    A[Deployment Decision] --> B{Latency Requirement?}
    B -->|<10ms| C[Model Simplification Required]
    B -->|10-100ms| D{Request Volume?}
    B -->|>100ms| E[Standard Deployment]

    C --> C1[Quantization + Edge]

    D -->|High >1K/sec| D1[Batching + Caching]
    D -->|Medium| D2[Standard API]
    D -->|Low| E

    C1 --> F[Monitor & Optimize]
    D1 --> F
    D2 --> F
    E --> F

Optimization Strategies ROI

Production Monitoring Dashboard

Business Threshold Optimization

Cost-Sensitive Decision Framework

graph TD
    A[Threshold Optimization] --> B[Define Costs/Benefits]
    B --> C[Cost of False Positive]
    B --> D[Cost of False Negative]
    B --> E[Benefit of True Positive]

    C --> F[Calculate Expected Value]
    D --> F
    E --> F

    F --> G[Sweep Thresholds 0 to 1]
    G --> H[Find Optimal Threshold]
    H --> I[Validate on Holdout]
    I --> J{Performance Acceptable?}

    J -->|No| K[Adjust Costs or Model]
    J -->|Yes| L[Deploy with Monitoring]

Example: Churn Prevention Economics

Threshold Selection Matrix

Error Analysis Framework

Systematic Error Analysis Process

graph LR
    A[Model Errors] --> B[Segment by Feature]
    B --> C[Identify Patterns]
    C --> D[Root Cause Analysis]

    D --> E{Fixable?}
    E -->|Data Issue| F[Collect More Data]
    E -->|Feature Issue| G[Engineer Features]
    E -->|Model Issue| H[Try Different Model]
    E -->|Inherent Noise| I[Accept or Set Confidence]

    F --> J[Retrain & Evaluate]
    G --> J
    H --> J
    I --> K[Document Limitations]

Error Analysis Dimensions

Case Study: Churn Prediction System

Business Context

Solution Architecture

graph TB
    A[Customer Data] --> B[Feature Engineering]
    B --> C[90-Day Prediction Window]

    C --> D[XGBoost Model]
    D --> E[Isotonic Calibration]
    E --> F[Threshold Optimization]

    F --> G{Churn Probability}
    G -->|>0.32| H[High Risk: Retention Offer]
    G -->|0.15-0.32| I[Medium Risk: Engagement Campaign]
    G -->|<0.15| J[Low Risk: No Action]

    H --> K[A/B Test Framework]
    I --> K
    J --> K

    K --> L[Measure Retention Impact]
    L --> M[Monthly Model Retraining]

Feature Importance Analysis

Results & Impact

Key Success Factors

1. Calibration was critical: Well-calibrated probabilities enabled optimal threshold (0.32), reducing unnecessary offers by 35%
2. Feature engineering > model complexity: Domain features (payment patterns, usage trends) drove 60% of performance gain
3. A/B testing validated impact: Control group showed 8% higher churn, confirming $8.1M annual value
4. Monitoring caught drift: Detected data shift after competitor pricing changes, triggered timely retraining
5. Tiered interventions: Different actions by risk level (high/medium/low) maximized ROI vs one-size-fits-all

Implementation Roadmap

Phase-by-Phase Checklist

Algorithm-Specific Guidance

When to Choose Each Algorithm

graph TD
    A[Choose Algorithm] --> B{Primary Goal?}

    B -->|Interpretability| C[Logistic Regression or Decision Tree]
    B -->|Performance| D{Data Type?}
    B -->|Speed| E{Training or Inference?}

    D -->|Tabular| F[XGBoost/LightGBM]
    D -->|Text| G[Transformers or TF-IDF + LR]
    D -->|Images| H[CNNs or Vision Transformers]

    E -->|Training| I[Naive Bayes or Logistic Regression]
    E -->|Inference| J[Quantized Models or Linear]

    C --> K[Validate Choice]
    F --> K
    G --> K
    H --> K
    I --> K
    J --> K

Minimal Code Example: Model Comparison

from sklearn.linear_model import LogisticRegression
from xgboost import XGBClassifier
from sklearn.model_selection import cross_val_score

# Quick model comparison (full pipeline omitted for brevity)
models = {
    'Logistic': LogisticRegression(max_iter=1000),
    'XGBoost': XGBClassifier(n_estimators=100, eval_metric='logloss')
}

for name, model in models.items():
    scores = cross_val_score(model, X_train, y_train, cv=5, scoring='roc_auc')
    print(f"{name}: AUC = {scores.mean():.3f} (+/- {scores.std():.3f})")

Common Pitfalls & Solutions

Key Takeaways

Critical Success Factors

1. Start simple, iterate based on data: Baselines reveal if ML is needed; simple models often suffice
2. Calibration enables decisions: Well-calibrated probabilities are essential for threshold-based decision making
3. Production is not an afterthought: Design for deployment from day one (latency, cost, monitoring)
4. Monitor everything: Track performance, drift, and business metrics to catch issues early
5. Business context drives technical choices: The best model delivers business value within operational constraints

Decision Framework Summary

graph LR
    A[Classification Project] --> B[Define Problem & Metrics]
    B --> C[Prevent Data Leakage]
    C --> D[Select & Train Model]
    D --> E[Calibrate Probabilities]
    E --> F[Optimize Threshold]
    F --> G[Deploy with Monitoring]
    G --> H[Iterate Based on Drift]
    H --> D

Recommended Reading Order

For practitioners new to classification:

1. Start with Problem Definition Framework
2. Study Data Leakage Prevention (most critical)
3. Use Model Selection Matrix for algorithm choice
4. Apply Calibration before deployment
5. Implement monitoring from day one