spark-optimization

Workload-aware ML framework for PySpark performance prediction and configuration recommendation, validated through real benchmark execution and out-of-distribution evaluation.

For the GitHub description field: "Hands-on PySpark performance optimization repo — broadcast joins, partitioning, caching, bucketing, AQE, and skew handling, each with runnable before/after benchmarks and correctness tests.

Apache Spark native computation engine

Demonstrates PySpark broadcast joins vs regular joins in Databricks, with execution plan comparison for performance optimization.

This project demonstrates key PySpark performance optimization techniques using a synthetic banking transactions dataset (~5,000 records). Built using Databricks and Delta Lake.

⚡ Intelligent Data Skew Detection & Mitigation in Apache Spark — ML Severity Classifier + Automated Salting/Repartitioning