Feature Engineering & Model Tuning

Module Completed: September 2021

---

🎯 Module Overview

Building a model is only part of the job — selecting the right features, handling high-dimensional data, tuning hyperparameters, and validating performance robustly are what separate a prototype from a production-ready model. This module covers the full toolkit: feature selection, regularisation, cross-validation, ROC/threshold tuning, Pipelines, and GridSearchCV.

---

📖 Topics Covered

Missing Value Treatment Strategies

Beyond simple mean/median imputation — a principled approach to large datasets with many missing columns:

• Calculate % null per column before deciding to drop or impute
• Drop columns with > X% null (X chosen based on data distribution of null counts)
• Median imputation for remaining columns (preferred over mean — robust to skewed distributions)
• Zero-variance columns: attributes where all values are (nearly) identical add no predictive signal — identify and drop using variance threshold

Feature Selection

Reducing the feature space to remove noise, speed up training, and improve generalisation:

• Variance Threshold — drop near-constant features (variance ≈ 0)
• Correlation Filter — drop one column from any pair with |r| > 0.8 to remove redundancy
• Recursive Feature Elimination (RFE) — iteratively removes weakest features using model coefficients or feature importances
• Forward / Backward Feature Selection — greedy search adding/removing one feature at a time
• Feature Importance from Random Forest / XGBoost — rank features by mean decrease in impurity

Regularisation

Penalising large model coefficients to reduce overfitting in linear/logistic models:

• Ridge (L2) — adds sum of squared coefficients to loss; shrinks all coefficients but keeps all features; alpha controls strength; closed-form solution
• Lasso (L1) — adds sum of absolute coefficients to loss; drives some coefficients exactly to zero (built-in feature selection); alpha controls sparsity
• ElasticNet — combines L1 + L2; l1_ratio controls the mix
• Choosing alpha via cross-validated grid search (RidgeCV, LassoCV, GridSearchCV)

Cross-Validation

Reliable model evaluation beyond a single train-test split:

• K-Fold CV — splits data into K folds; trains on K-1, validates on 1; repeats K times; average score is the final estimate
• Stratified K-Fold — preserves class proportions in each fold; critical for imbalanced targets
• Leave-One-Out CV (LOOCV) — K = n; maximum use of data; computationally expensive; best for very small datasets
• Bootstrapping — random sampling with replacement; each bootstrap sample trains a model; distribution of scores gives confidence intervals around performance estimates
• Rule: fit scaler/imputer on training fold only, transform test fold — never leak test statistics

ROC Curve & Threshold Management

Beyond default 0.5 threshold — tuning the decision boundary for business requirements:

• ROC Curve — plots True Positive Rate (Recall) vs False Positive Rate at all thresholds
• AUC-ROC — area under the ROC curve; model's discrimination ability; 0.5 = random, 1.0 = perfect
• Threshold Tuning — use predict_proba to get probability scores, then apply custom threshold; raising threshold → higher precision, lower recall; lowering → higher recall, lower precision
• When to lower threshold: when missing a positive (e.g., missing a fail/fraud) is more costly than a false alarm

Hyperparameter Tuning

Systematic search for optimal model configuration:

• GridSearchCV — exhaustive search over all combinations in a parameter grid; evaluates each via cross-validation; best for small grids
• RandomizedSearchCV — samples a fixed number of random combinations; better for large grids; often finds near-optimal faster
• refit=True — automatically retrains the best model on the full training set after search
• cv_results_ — detailed scores for every parameter combination evaluated

Scikit-learn Pipelines

Chaining preprocessing steps + model into a single reusable object:

• Prevents data leakage — scaler/imputer fitted only on training data within each CV fold
• Enables clean fit() / predict() API for the entire workflow
• Compatible with GridSearchCV — hyperparameters addressable via step__param syntax
• Common pattern: Pipeline([('scaler', StandardScaler()), ('clf', RandomForestClassifier())])
• Pickle the entire pipeline for deployment — guarantees identical preprocessing at inference time

Dimensionality Reduction for High-Dimensional Data

When features >> samples:

• PCA to reduce 500+ features to the top K components explaining 90–95% variance
• Apply PCA after scaling, before model training
• Trade-off: slight accuracy loss vs significant speed gain; critical at gigabyte/terabyte scale

Sampling Strategies (Up/Down-sampling)

• Under-sampling — remove majority class records; only viable with very large datasets
• Over-sampling (SMOTE / ADASYN) — generate synthetic minority samples; apply only on training fold, never on test
• ADASYN preferred for complex decision boundaries

---

🏆 Project — Feature Engineering, Model Selection & Tuning

Submitted: 12 September 2021 Domain: Semiconductor manufacturing process

---

Problem Statement

Predict the Pass/Fail yield of a semiconductor manufacturing entity from sensor signal data. Determine whether all 591 features are necessary or whether a reduced feature set can achieve comparable performance.

Dataset

• Shape: 1,567 records × 592 columns (591 features + 1 target)
• Target: Pass/Fail — original encoding: -1 = Pass, 1 = Fail → re-encoded to 1 = Pass, 0 = Fail
• Data: signal-data.csv — continuous float sensor readings; heavy with missing values

---

Data Cleaning Pipeline

Step	Action	Result
Null % analysis	Computed % null per column	538 of 592 columns had NaNs
Drop high-null columns	Dropped columns with > 5% null	Down to 540 columns
Median imputation	Imputed median for remaining nulls	No NaNs remaining
Zero-variance drop	Dropped near-constant columns	Down to 217 columns
Correlation filter	Dropped one from each pair with	r
Final dataset	Ready for modelling	1,567 rows × 116 features

---

EDA

• Wide range of feature scales (0.01–0.1 up to 2,700–3,300) → StandardScaler mandatory
• Target imbalance: significantly more Pass than Fail records
• PCA for visualisation: 90% variance explained by a subset of components (applied for EDA only, not initial modelling)

---

Preprocessing

• Train-Test Split: 70:30 stratified → 1,096 train / 471 test
• ADASYN oversampling applied to training data only to fix class imbalance
• StandardScaler fitted on training data, applied to test data (no leakage)

---

Models Trained

Model 1 — Random Forest (Default Parameters)

• High overall accuracy
• Poor recall for Class=Fail (model missed 33 of 34 actual failures)
• ROC threshold tuning: raised threshold to 0.67 → slight improvement in Fail recall

Model 2 — Random Forest (Hyperparameter Tuned)

• GridSearchCV used for tuning n_estimators, max_depth, min_samples_split, max_features
• Marginal improvement in Fail class recall (+4 correct vs Model 1 with default threshold)
• ROC threshold reapplied → additional uplift

Model 3 — Random Forest + PCA (Dimensionality Reduction)

• PCA applied: retained 87 components explaining target variance
• Comparable accuracy to Models 1 & 2 with reduced feature space
• Slightly better Fail class prediction than baseline; threshold tuning applied

Bootstrap Validation

• Multiple random samples drawn from the dataset; model trained on each
• Score distribution showed RandomForestClassifier is stable across different sample populations — low variance in train score, slightly more variance in test score

---

Model Comparison

Model	Overall Accuracy	Fail Class Recall	Notes
Default RF	High	Very low	Missed most failures
Tuned RF	High	Slightly better	+4 correct fails vs default
PCA + RF	High	Comparable	1/3 fewer features

Final Model: RandomForestClassifier with tuned hyperparameters

---

Pipeline & Deployment

• Pipeline built: ADASYN → StandardScaler → RandomForestClassifier
• Pickled artefacts saved:
  • final_model.pkl — trained RandomForest
  • standard_scaler.pkl — fitted scaler
  • final_pca.pkl — PCA transformer
  • final_attributes.pkl — list of selected feature names
• Future data (Future_predictions.xlsx) loaded, cleaned, scaled, and passed through the pipeline for inference

---

🔑 Key Learnings

1. High feature count demands a cleaning pipeline — blindly using 591 raw features is inefficient; variance threshold + null filter + correlation filter reduced to 116 useful features
2. Scale first, always — features with vastly different ranges (0.01 vs 3000) will break distance-based models and skew linear model coefficients
3. Apply oversampling only to training data — sampling before splitting leaks minority patterns into the test set and gives artificially inflated recall scores
4. High accuracy ≠ good model for imbalanced targets — a model predicting all-Pass for a 90/10 split gets 90% accuracy while failing completely on the minority class
5. ROC threshold tuning is a business decision — choose the threshold based on the cost of false negatives vs false positives; for semiconductor failures, missing a failure (FN) is far worse than a false alarm (FP)
6. Pipelines prevent data leakage — fitting scaler inside a Pipeline ensures it sees only the training fold at each CV step; fitting outside leaks test distribution into the scaler
7. PCA on high-dimensional data is practical — 117 features → 87 PCA components, comparable accuracy, dramatically faster at scale
8. Bootstrap gives confidence intervals — a single train/test split gives one performance number; bootstrapping reveals the distribution of performance across different data samples

---

🛠️ Tools & Libraries

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import pickle

from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import Ridge, Lasso, LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline
from sklearn.model_selection import (train_test_split, GridSearchCV,
                                      RandomizedSearchCV, KFold,
                                      StratifiedKFold, LeaveOneOut,
                                      cross_val_score)
from sklearn.metrics import (classification_report, confusion_matrix,
                              roc_curve, roc_auc_score)
from sklearn.feature_selection import VarianceThreshold, RFE

from imblearn.over_sampling import ADASYN, SMOTE