spark-lens report

daily-revenue-pipeline · app-20241105-0042 · Spark 3.5.1

46/100

Health Score

Critical

Warning

Info

CRITICAL Disk Spill in Stage 1 — 3.4 GB spilledhigh confidence ~3,2m saved

Stage 1 spilled 3.4 GB to disk across 120 tasks. Disk I/O is 10–100× slower than RAM — this stage is paying a heavy I/O tax on every shuffle write.

→ Raise spark.executor.memory so the execution pool fits the working set, or reduce spark.executor.cores to lower concurrent memory pressure.

config

spark.executor.memory=8g
# or reduce cores to lower concurrent peak demand:
# spark.executor.cores=2

metrics

disk_spill_bytes	3.4 GB
memory_spill_bytes	12.1 GB
stage_duration_ms	87000

stage 1

WARNING Shuffle Hot-Key Skew in Stage 1high confidence ~1,6m saved

Top-5% of tasks hold 71% of stage time. The largest task ran for 52 s while the median was 3 s (p95/p50 ratio: 17×). Shuffle skew detected — the data distribution is uneven across reduce partitions.

→ Enable AQE skew-join handling so Spark automatically splits skewed partitions at runtime.

config

spark.sql.adaptive.enabled=true
spark.sql.adaptive.skewJoin.enabled=true

metrics

p95_task_ms	52000
p50_task_ms	3000
p95_p50_ratio	17.3
top5pct_share	71%

stage 1

WARNING SortMergeJoin With Broadcast Disabled — Stage 2medium confidence ~28,0s saved

Stage 2 contains a SortMergeJoin but autoBroadcastJoinThreshold is set to -1, disabling broadcast entirely. The right side of the join (estimated 280 MB) could be broadcast instead of shuffled.

→ Remove the broadcast disable or raise the threshold so small tables are broadcast automatically.

config

spark.sql.autoBroadcastJoinThreshold=314572800  # 300 MB

code

import org.apache.spark.sql.functions.broadcast
df_large.join(broadcast(df_small), "key")

metrics

right_side_bytes	280 MB
shuffle_written	1.1 GB

stage 2

job 1

INFO Java Serializer in Use — Switch to Kryolow confidence

Java serialization is 10× slower than Kryo and produces 2–10× larger byte arrays. Every JVM shuffle write, broadcast variable, and RDD persist pays this cost.

→ Switch to Kryo for JVM workloads. Register your domain classes for maximum performance.

config

spark.serializer=org.apache.spark.serializer.KryoSerializer

code

spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
// optionally: spark.conf.set("spark.kryo.registrationRequired", "false")

INFO Adaptive Query Execution (AQE) Is Disabledlow confidence

AQE automatically coalesces shuffle partitions, handles skew, and switches join strategies at runtime. Disabling it forces static planning.

→ Enable AQE for all production workloads on Spark 3.x.

config

spark.sql.adaptive.enabled=true