Big data email classification and model evaluation
End-to-end PySpark machine-learning pipeline for email spam classification.
What I built
- Built an end-to-end PySpark machine-learning pipeline covering data preprocessing, feature assembly and scalable model training.
- Applied Chi-Square feature selection to identify 50 relevant predictive features and reduce unnecessary input dimensions.
- Trained and compared three Naive Bayes classification models using a consistent training and testing workflow.
- Evaluated models with accuracy, precision, recall, F1-score, AUROC, confusion matrices, ROC curves and precision–recall curves.