spark-optimization

Workload-aware ML framework for PySpark performance prediction and configuration recommendation, validated through real benchmark execution and out-of-distribution evaluation.

Research framework for cost-aware Apache Spark self-tuning, combining Bayesian Optimization, Safe Transfer Learning, and hybrid metaheuristics for adaptive performance optimization.

For the GitHub description field: "Hands-on PySpark performance optimization repo — broadcast joins, partitioning, caching, bucketing, AQE, and skew handling, each with runnable before/after benchmarks and correctness tests.

⚡ Intelligent Data Skew Detection & Mitigation in Apache Spark — ML Severity Classifier + Automated Salting/Repartitioning

Apache Spark native computation engine

Demonstrates PySpark broadcast joins vs regular joins in Databricks, with execution plan comparison for performance optimization.

This project demonstrates key PySpark performance optimization techniques using a synthetic banking transactions dataset (~5,000 records). Built using Databricks and Delta Lake.