Apache Spark 3.x · Scala 2.12 / 2.13

Deep Spark
performance analysis

Attach via spark.extraListeners — get a ranked report of performance issues with exact config fixes at every job end, on every cluster.

$ spark-submit \ --packages io.github.nidhal-saadaoui:spark-lens_2.12:LATEST_VERSION \ --conf spark.extraListeners=com.github.saadaouini.sparklens.SparkLensListener \ --conf spark.sparklens.output=text \ myJob.jar
29 analyzers 4 output formats 40+ configurable thresholds Estimated savings per issue Apache 2.0

How it works

Three steps, no infrastructure required.

1

Attach the listener

Add spark.extraListeners=...SparkLensListener and spark.sparklens.output=text to your spark-submit command or spark-defaults.conf. Nothing else needed.

Heads-up: spark.extraListeners= replaces any existing listener list. Use a comma to append: OtherListener,SparkLensListener. For batch applications only — Structured Streaming results will be misleading.

2

Run your job normally

spark-lens listens to every task, stage, job, and SQL execution as they complete — zero overhead on the hot path. It only processes data after the application ends.

3

Read the report

At application end, issues are ranked by estimated savings and printed to driver stdout or written to a file. Every issue includes a concrete recommendation and ready-to-paste config or code fixes — with estimated time saved per run.

Report output

Four formats: text for driver stdout, HTML for humans, JSON for CI parsers and dashboards, log for log aggregation (one line per issue).

text

Driver stdout

Human-readable with priority fixes section, issue list, and quick-wins grouped by config change. Default when a file path is given without a format extension.

html

Interactive dashboard

Metrics summary panel, stage timeline, memory pressure line chart, shuffle metrics breakdown, GC timeline, and issue severity timeline. Collapsible issue cards with severity badges, savings, and fixes. Suitable for sharing as a CI artifact.

json

Machine-readable

Stable schema with health_score, total_estimated_savings_ms, top_actions, and full issue array. Parse in CI pipelines or feed into dashboards.

log

Log aggregation

One log line per issue via the driver Java logger. Compatible with Datadog, Splunk, CloudWatch Logs — filter on SPARKLENS tag. No file path needed.

Text report — priority fixes ranked by estimated savings

spark-submit — driver stdout
====================================================================== spark-lens | daily-revenue-pipeline (app-20241105-0042) Spark 3.5.0 | Duration: 8.4min | Health: 64/100 | 1 critical · 2 warning · 2 info ====================================================================== Priority fixes (estimated savings per run): 1. [CRITICAL] Disk Spill in Stage 3 — 2.1 GB spilled [+2 more stages] ~4.2min (50% of app time) 2. [WARNING] Shuffle Hot-Key Skew in Stage 5 ~1.8min (21% of app time) 3. [WARNING] Java Serializer in Use — Switch to Kryo ~45s [✖ CRITICAL] Disk Spill in Stage 3 — 2.1 GB spilled [+2 more stages] (~4.2min per run) fix: spark.executor.memory=8g Stage 3 spilled 2.1 GB to disk. Disk I/O is 10–100× slower than memory. stages: 3, 7, 11 · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · [⚠ WARNING] Shuffle Hot-Key Skew in Stage 5 (~1.8min per run) fix: spark.sql.adaptive.skewJoin.enabled=true Top-5% of tasks hold 67% of stage time. stages: 5 · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · [⚠ WARNING] Java Serializer in Use — Switch to Kryo (~45s per run) fix: spark.serializer=org.apache.spark.serializer.KryoSerializer Java serialization is 10× slower than Kryo and produces larger shuffle bytes. · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ────────────────────────────────────────────────────────────────── Quick wins spark.executor.memory=8g resolves 1 issue: spill spark.sql.adaptive.enabled=true resolves 2 issues: config-aqe-disabled, config-default-shuffle-partitions ======================================================================

JSON report — stable schema for CI parsers and dashboards

report.json
{ "spark_lens_version": "LATEST_VERSION", "app_id": "app-20241105-0042", "app_name": "daily-revenue-pipeline", "spark_version": "3.5.0", "duration_ms": 504000, "health_score": 64, "issue_count": 5, "total_estimated_savings_ms": 372000, // deduplicated by root-cause cluster, capped at app duration "top_actions": [ { "config": "spark.executor.memory=8g", "resolves": ["spill"], "estimated_savings_ms": 252000 }, { "config": "spark.sql.adaptive.enabled=true", "resolves": ["config-aqe-disabled", "skew-warn"], "estimated_savings_ms": 108000 } ], "issues": [ { "id": "spill", "severity": "critical", "category": "spill", "title": "Disk Spill in Stage 3 — 2.1 GB spilled [+2 more stages]", "description": "Stage 3 spilled 2.1 GB to disk.", "recommendation": "Increase executor memory or reduce concurrent task count.", "fixes": { "config": "spark.executor.memory=8g" }, "affected_stages": [3, 7, 11], "affected_jobs": [], "metrics": { "disk_spill_bytes": "2.1 GB", "stages_affected": "3" }, "estimated_impact": { "summary": "~4.2min saved per run", "saved_time_ms": 252000, "confidence": "medium" } }, { ... } ], "listener_overhead_ms": 42 }

HTML report — interactive, collapsible issues with metrics and stage links

Issues are grouped by type — e.g. Disk Spill in Stage 3 [+4 more stages] — so the report stays readable on large pipelines. Each issue shows its estimated impact badge, metrics table, and config/code fix blocks. Stage pills have callSite tooltips.

Log format — one structured line per issue, inline in the driver log

Use spark.sparklens.output=log. Each line is prefixed with [spark-lens] — grep or filter by this tag in Datadog, Splunk, CloudWatch, or any log aggregator. No separate file path needed.

driver log (YARN / K8s / Databricks)
24/11/05 04:12:33 INFO SparkContext: Successfully stopped SparkContext [spark-lens] SUMMARY app="daily-revenue-pipeline" app_id=app-20241105-0042 spark=3.5.0 health=64 issues=5 critical=1 warning=2 info=2 duration=4.2m savings=6.1m/run [spark-lens] CRITICAL id=spill-3 category=spill title="Disk Spill in Stage 3 — 2.1 GB spilled [+2 more stages]" savings=4.2m/run fix="spark.executor.memory=8g" stages=3,7,11 [spark-lens] WARNING id=skew-warn-5 category=skew title="Shuffle Hot-Key Skew in Stage 5" savings=1.8m/run fix="spark.sql.adaptive.skewJoin.enabled=true" stages=5 [spark-lens] WARNING id=config-java-serializer category=config title="Java Serializer in Use — Switch to Kryo" savings=45.0s/run fix="spark.serializer=org.apache.spark.serializer.KryoSerializer" [spark-lens] INFO id=config-aqe-disabled category=config title="Adaptive Query Execution (AQE) Is Disabled" fix="spark.sql.adaptive.enabled=true" [spark-lens] INFO id=config-default-shuffle-partitions category=config title="Default Shuffle Partitions (200) — May Be Too Few or Too Many" fix="spark.sql.adaptive.enabled=true" 24/11/05 04:12:33 INFO Utils: Shutdown hook called

What it detects

29 analyzers covering skew, memory, I/O, query planning, configuration, reliability, and executor scaling — every issue includes an estimated savings figure.

JobTimelineAnalyzerio
Idle gap > 60 s between jobs (driver is the bottleneck), or > 70% of jobs complete in < 2 s across 50+ total jobs (scheduling fragmentation).
SkewAnalyzerskew
p95/p50 task duration ratio > 3× or top-5% tasks hold > 25% of stage time. Distinguishes shuffle skew from input skew with different recommendations.
TaskOverheadAnalyzerio
Executor deserialize time > 30% of run time — too many small tasks paying serialization overhead on every task launch.
SpillAnalyzerspill
Total disk spill > 100 MB (Warning) or > 1 GB (Critical). Disk I/O is 10–100× slower than RAM — spill is a reliable sign of under-provisioned executor memory.
JoinAnalyzerjoin
Broadcast disabled on SortMergeJoin (missed opportunity), broadcast threshold ≥ 1 GB (driver OOM risk), ≥ 4 shuffle exchanges in one query, or output > 5× input (exploding join).
GcAnalyzergc
GC time > 10% (Warning) or > 20% (Critical) of executor run time. Excessive GC pauses slow tasks and can trigger heartbeat timeouts.
CacheAnalyzercache
Same table or RDD scanned in ≥ 2 jobs without caching — the full upstream lineage is re-executed each time. Savings scale with table size × scan count.
PreemptionAnalyzerpreemption
Executor lost mid-job or task kill rate > 5% per stage — indicates resource contention, YARN preemption, or spot-instance eviction.
PlanAnalyzerplan
CartesianProduct (quadratic cost), Window without PARTITION BY (forces single-partition serial execution), round-robin repartition before a join, missing CBO row-count statistics.
UdfAnalyzerplan
Python UDF (PythonUDF / BatchEvalPython) or Scala UDF detected in the physical plan — breaks Catalyst optimizations and adds row-by-row serialization overhead.
IoClassifierAnalyzerio
Stage throughput ≥ 3 MB/s per core — the stage is storage/network bound, not compute-bound. Predicate pushdown, columnar formats, or caching are the levers.
ConfigAnalyzerconfig
9 checks: AQE disabled, Java serializer, default shuffle partitions (200), low memory overhead, AQE skew-join disabled, small shuffle buffer (< 64 k), CBO histograms disabled, low task.maxFailures, high locality.wait.
ExecutorSizingAnalyzerconfig
Executor memory under-provisioned (p95 task peak × cores > 85% of execution pool) or over-provisioned (< 25% utilized). Also flags driver heap risk and cluster cores vs max stage parallelism.
SmallFilesAnalyzerio
Input avg < 64 MB/task with majority of tasks reading tiny files — too many map tasks, high scheduling and metadata overhead per file open.
OutputSmallFilesAnalyzerio
Output avg < 64 MB/task — each task writes one file, so many tasks create many small files that slow every downstream job that reads this data.
ShuffleLocalityAnalyzerio
> 70% of shuffle bytes read remotely — cross-rack or cross-AZ reads saturate shared network bandwidth and add latency.
DriverBottleneckAnalyzerio
collect() result > 50 MB returned to driver, or CollectLimit / TakeOrderedAndProject in the SQL plan — driver becomes the bottleneck.
CpuEfficiencyAnalyzerio
CPU utilization < 20% of executor run time — most core time is I/O wait, shuffle network, or JVM overhead rather than actual computation.
SpeculationAnalyzerconfig
Speculative tasks actively firing — treating the symptom (slow tasks) instead of fixing the root cause (data skew or undersized executors).
StageFailureAnalyzerreliability
Stage retried (attempt > 0) with a failure reason, or task failure rate > 5% in a stage — surfaces OOM errors, network failures, and data issues.
MemoryPressureAnalyzerreliability
GC > 10% and disk spill > 100 MB co-occurring in the same stage — the executor heap is genuinely undersized for the working set.
StageParallelismAnalyzerio
Stage tasks < 50% of available executor cores on a stage > 10 s (most cores idle), or entire stage runs as a single task — wall time longer than necessary.
LongStageAnalyzerreliability
Stage duration > 5× the median stage duration in its job — one stage serializes all downstream stages and dominates job wall time.
PartitionImbalanceAnalyzerio
Input partition p95/p50 size ratio > 3× (warns at 5×) — a few fat partitions become the bottleneck while most tasks finish quickly.
SchedulerDelayAnalyzerconfig
Median task launch delay > 2 s after stage submission — tasks waiting idle before first execution. Caused by locality wait, busy executors, or driver GC.
CriticalPathAnalyzerplan
DAG critical path (via stage parentIds) ≥ 85% of app wall time across ≥ 3 sequential stages — adding executors will not reduce this; the serial dependency chain must be shortened.

Configuration

All settings are optional. No required configuration.

PropertyDefaultValuesDescription
spark.sparklens.outputoffoff · text · json · html · log — comma-separated for multipleOutput format(s). off is silent unless fail.on is set. Example: text,json
spark.sparklens.report.path(stdout)local path, hdfs://, s3://…Write report to a file. Supports any Hadoop-compatible filesystem. With multiple formats each gets its own extension (.txt, .json, .html).
spark.sparklens.report.path.<fmt>—path per formatFormat-specific path override — highest priority. Available for .text .json .html .log.
spark.sparklens.fail.on(none)critical · warning · infoThrow RuntimeException at app end if issues at this severity or above are found. Exit code is non-zero — CI pipeline fails automatically.

Permanent cluster configuration

Add to spark-defaults.conf on every node — every job gets analyzed automatically, no per-job flags needed.

spark.extraListeners=com.github.saadaouini.sparklens.SparkLensListener spark.sparklens.output=text spark.sparklens.fail.on=critical

CI quality gate

Fail the Spark job itself when critical issues are detected — no external tooling required.

Fail on critical

# CI fails automatically — exit code non-zero spark-submit \ --packages io.github.nidhal-saadaoui:spark-lens_2.12:LATEST_VERSION \ --conf spark.extraListeners=...SparkLensListener \ --conf spark.sparklens.fail.on=critical \ myJob.jar

Save JSON report as CI artifact

# Parse in a subsequent step or upload to S3 spark-submit \ --conf spark.sparklens.output=text,json \ --conf spark.sparklens.report.path=/tmp/report \ myJob.jar # produces /tmp/report.txt and /tmp/report.json # health_score, total_estimated_savings_ms, top_actions…

Health score

Flat deduction per issue, floored at 0. A single CartesianProduct (−30) always scores worse than five config warnings (−50 → but likely −50 is still a 50/100 score showing both matter).

100
Perfect
No issues detected
−30
per Critical
e.g. CartesianProduct, spill > 1 GB
−10
per Warning
e.g. skew, spill > 100 MB, bad join
−2
per Info
e.g. config suggestions, small I/O

Performance contract testing

spark-lens-testing lets you write ScalaTest specs that assert on analysis results — catching performance regressions in CI before they reach production.

Add to your build

// build.sbt libraryDependencies += "io.github.nidhal-saadaoui" %% "spark-lens-testing" % "LATEST_VERSION" % Test

Write performance assertions

// Scala 2.12 / 2.13 · ScalaTest FlatSpec class MyJobSpec extends SparkLensSpec { "aggregation" should "not spill" in { analyse { MyJob.run(spark) } should not(haveIssueOfCategory("spill")) } "health" should "stay above 75" in { analyse { MyJob.run(spark) } .healthScore should be >= 75 } }

Available matchers

haveIssue(id)

Issue with that exact id or id prefix is present. e.g. "plan-cartesian" matches "plan-cartesian-3"

haveIssueOfCategory(cat)

Any issue in that category: spill, skew, join, gc, config, plan, io, reliability

haveIssueOfSeverity(sev)

Any issue at Critical, Warning, or Info

haveNoIssuesOfSeverity(sev)

No issue at that severity — assert a job is clean

haveHealthScoreAbove(n)

Health score > n. e.g. haveHealthScoreAbove(75)

haveHealthScoreBelow(n)

Health score < n — verify a bad job is detected

Failure messages include the full report

When any assertion fails, the complete text report is embedded in the test failure — no guessing which issue fired or what the fix is.

sbt test — assertion failure output
MyJobSpec > aggregation job should not spill to disk *** FAILED *** Did not expect any issue of category 'spill' but one was present. ====================================================================== spark-lens | MyJobSpec (local-xxx) Spark 3.5.0 | Health: 70/100 | 1 critical ====================================================================== [✖ CRITICAL] Disk Spill in Stage 3 — 2.1 GB spilled fix: spark.executor.memory=8g Stage 3 spilled 2.1 GB to disk. Disk I/O is 10–100× slower. stages: 3 ======================================================================

You can also print the report manually during debugging: result.textReport.

Both SparkLensSpec (FlatSpec) and SparkLensSuite (FunSuite) are available. Requires JVM < 23 — build.sbt auto-detects Java 17 when the host JVM is newer. Set JAVA_17_HOME if needed.