English | δΈζ
In the AI era, this project aims to revolutionize Spark performance analysis. Centered around Spark, it leverages LLM to provide deep data insights and uses the Model Context Protocol (MCP) to expose these capabilities, enabling autonomous and natural language-driven tuning. It effectively addresses the core pain points of the native Spark Web UI/History Server by eliminating "slow replay", "information overload", and "lack of comparison" issues through Medallion Architecture, Smart Diagnosis, and Multi-dimensional Benchmarking.
π‘ Core Goal: Transform EventLog into "Trustworthy Actionable Intelligence", providing instant answers instead of raw data.
- Spark Performance Insight: http://demo.fluffyeti.com:18081/
- Spark History Server (Native - For Comparison): http://demo.fluffyeti.com:18080/
While the native Spark Web UI provides basic monitoring, users often face significant hurdles during deep performance analysis:
- Metric Labyrinth: Spark UI presents a massive volume of raw metrics and obscure charts. Beginners struggle to find what matters, and experts spend excessive time digging through pages to correlate metrics.
- Hidden Insights: Critical bottlenecks like GC pressure or data skew are often buried under layers of sub-menus, making it hard to get a quick "health check" of an application.
- Unquantifiable Deviations: When a job slows down compared to yesterday, there is no built-in way to compare the two runs side-by-side to see exactly which stage or task changed.
- Blind Troubleshooting: Hard to identify if performance shifts are due to config changes, resource fluctuations, or hardware issues without manual, error-prone data collection.
- Event Replay Overhead: Replaying raw JSON EventLogs for massive jobs results in extreme CPU/Memory overhead and minute-long wait times.
- Scalability Limits: Without structured storage, the SHS often crashes (OOM) when handling jobs with millions of tasks.
- Deep Bottleneck Analysis: Integrates Zhipu AI (GLM-4.7) and OpenAI to analyze complex performance issues (e.g., Shuffle IO, GC pressure).
- Optimization Advice: Generates expert-level Markdown reports with actionable tuning suggestions tailored to your specific application.
- Autonomous Log Analysis: Exposes Spark Performance Insight as an MCP server. AI Agents (Claude, Gemini) can directly "read" and analyze your local Spark logs.
- Natural Language Tuning: "Analyze the log at
/tmp/spark-logs/app-1" β the AI will automatically trigger the parsing pipeline, wait for completion, and provide tuning advice without you leaving the chat. - Universal Connectivity: Supports Gemini CLI and Claude Code via high-performance HTTP/SSE transport.
- Statistical Precision: Unlike the probabilistic nature of LLMs, the rule engine provides deterministic, stable, and highly accurate analysis based on rigorous statistical thresholds.
- Expert Heuristics: Codifies years of Spark performance tuning expertise into automated rules for detecting Data Skew, Executor GC pressure, Disk Spilling, and Locality issues.
- Instant Root Cause: Provides immediate, quantifiable evidence for performance regressions, serving as the "Gold Standard" for production troubleshooting.
- Cross-App Comparison: Side-by-side comparison of different application instances to identify configuration or resource-induced regressions.
- Stage Benchmarking: Deep dive into two stages to compare statistical distributions (P95, Median) and task execution traces.
- Familiar Interface: Deeply replicates native Spark UI lists (Jobs, Stages, Tasks) and descriptions to ensure a zero-learning-curve transition for developers.
- Enhanced Summary: Provides statistical distributions for all core metrics and high-performance server-side pagination for millions of tasks.
| Application List (Home) | Job List Overview |
|---|---|
 |
 |
| Compare Workspace | SQL / DataFrame Detail |
 |
 |
| Environment Config | |
 |
- Bronze (Raw Ingestion): High-speed streaming ingestion using Jackson, handling TB-sized logs effortlessly.
- Silver (Transformation): Structured parsing that recovers logical relationships and identifies long-tail tasks.
- Gold (Aggregation): Pre-calculated analytical tables stored in DuckDB for instant UI response times.
- OOM Recovery: Automatic DuckDB memory management with
CHECKPOINTand retry logic. - Timing Accuracy: Synchronized epoch milliseconds and monotonic progress tracking for reliable estimates.
- Broad Log Support: Native support for ZSTD compression and Spark V2 log directories.
Run backend unit tests and generate JaCoCo coverage report:
mvn clean test
# Coverage Report: target/site/jacoco/index.htmlRun frontend unit tests and generate Vitest coverage report:
cd frontend && npm run test:coverage
# Coverage Report: frontend/coverage/index.htmlThis project follows a strict Quality First approach, implementing a four-layer guardian system to ensure stability and performance:
- Meaningful Coverage: We target 70% Line and 80% Branch coverage on core logic (Services/Utils), intentionally excluding boilerplate code to ensure our CI acts as a true logic sentinel.
- SQL-Schema Guard: Integration tests run against a real DuckDB instance to ensure MyBatis XML SQLs are always perfectly synchronized with the database schema.
- E2E Parsing Pipeline: Validates the full Medallion pipeline using real-world Spark EventLogs (ZSTD, V2 formats) to prevent parsing regressions.
- Performance Watchdog (JMH): Precise micro-benchmarking using JMH to monitor processing latency across Medallion layers, ensuring no heavy performance degradation during feature updates.
- UI Regression Guard: Structure-based component tests that protect critical UI features (e.g., search, filtering, charts) from accidental breakage.
Detailed Strategy: For a deep dive into our testing philosophy and "Meaningful Coverage" implementation, please refer to the Testing Strategy Guide.
New users can verify their LLM API keys and network connectivity without running the full application by using the manual test tool:
src/test/java/com/fluffyeti/spark/performance/insight/llm/LLMManualConnectionTest.java
For deep dives into the system design and technical specifications, please refer to our structured documentation:
- Frontend: Vue 3 + Vite + ECharts + Material Design.
- Backend: Java 21 (Virtual Threads) + Spring Boot 3.x.
- OLAP Engine: DuckDB (Embedded analytical database).
- ORM: MyBatis Plus (XML-based for optimized SQL).
-
Build and Start:
mvn clean install -Pbuild-frontend mvn spring-boot:run
-
Access the UI: Visit
http://localhost:18081in your browser.
mvn clean install -Pbuild-frontend -PrunStarts both Spark Performance Insight UI and Spark History Server sharing the same log directory.
-
Start Services:
docker compose up -d
-
Access Points:
- Spark Performance Insight UI:
http://localhost:18081 - Spark History Server (Native - For Comparison):
http://localhost:18080
- Spark Performance Insight UI:
We are evolving from a "Performance Dashboard" to a "Trustworthy AI Performance Expert".
- Phase 1: Structured Evidence - P95 distributions, Skew factors, and Context awareness.
- Phase 2: Expert Rule Engine - 10+ bottleneck patterns and Safe Action sets.
- Phase 3: AI-Driven Diagnosis - Evidence-linked multi-agent (Proposer-Challenger) diagnosis.
- Phase 4: Feedback Loop - Human-in-the-loop evaluation and Golden Test Sets.
- Phase 5: Adaptive Tuning - A/B testing and Bayesian-based parameter optimization.
Detailed Planning: Project Evolution Roadmap
- Special thanks to JimLiu/baoyu-skills for the release-skills that streamlines our release workflow.