Why Your Spark Job Keeps OOMing Even After You Scale Amazon EMR

Amazon EMR auto-configures spark.executor.cores and spark.executor.memory from the EC2 types in the cluster. AWS documents that explicitly: both values are derived from the core and task instances.

That's convenient. EMR knows the cores and how much memory YARN exposes via yarn.nodemanager.resource.memory-mb, and it will pack executors so you don't leave CPU or RAM idle.

What it doesn't know is whether your joins, aggregations, or skewed partitions actually fit in that heap. AWS's own performance guidance says some workloads want larger executor JVMs and others want smaller, more numerous ones. EMR is solving a packing problem. You still have to size for the work.

Homogeneous fleet: packed machines, tiny heaps

Take a fleet of c5.12xlarge (48 vCPU, 96 GiB). In AWS's troubleshooting example, YARN gets 90,112 MiB and EMR lands on:

spark.executor.cores = 3
spark.executor.memory = 4743 MiB

EMR also applies spark.yarn.executor.memoryOverheadFactor = 0.1875 (since 5.22), so each YARN container is about 5,632 MiB. That's 16 executors by cores and 16 by memory. Perfect pack.

Then a stage with fat joins shows up, and some tasks need more than ~4.7 GiB of heap. The cluster can have terabytes of RAM in aggregate. That doesn't help. A Spark task runs inside one executor JVM. Unused memory on another node does not make this executor bigger. It OOMs.

Adding nodes doesn't enlarge the executor

The errors look like:

ExecutorLostFailure
Container killed by YARN for exceeding memory limits
java.lang.OutOfMemoryError

The instinct is "we need more memory," so the fleet goes from 10 workers to 20. Same instance type, same executor config: 320 executors instead of 160, each still at ~4.7 GiB.

You didn't fix the task that didn't fit. You bought more copies of the same constraint.

Scaling out can change timing and shuffle pressure enough that the job sometimes succeeds. That's not a good price-performance point. You may be paying for twice the EC2 to simply avoid changing two Spark settings.

The question isn't whether the cluster has enough memory. It's whether this executor has enough for the work it got.

Mixed instance families

This gets messier when the fleet mixes CPU-to-memory ratios. From EMR's task config tables:

m5.4xlarge  16 vCPU  57,344 MiB YARN
r5.4xlarge  16 vCPU  122,880 MiB YARN

Same cores, more than 2× the memory. AWS's performance guide pushes it further with c5 / m5 / r5.4xlarge: all 16 vCPU, and 24 / 57 / 123 GiB of YARN memory.

EMR 5.32 and 6.5 turned on heterogeneous executors by default (spark.yarn.heterogeneousExecutors.enabled). Sizes can now follow the instance instead of one fleet-wide profile, and you can cap them with spark.executor.maxMemory and spark.executor.maxCores.

That's better packing. It still doesn't make the workload homogeneous. A fat partition can land on a small executor, OOM, retry, land on another small one, and eventually succeed on an R instance.

The job reports SUCCESS. You still paid for the failed attempts, and for every other node sitting around while Spark recomputes. Workers that accept work they can't finish don't just waste their own time; they stretch the wall clock for the whole cluster.

A cheaper instance that crashes halfway through a task is often more expensive than an R that finishes it once.

Constrain the fleet to what the workload needs

Instance fleets are good for Spot diversity and capacity. They're not a license to admit any instance YARN can schedule on.

AWS's flexibility guidance has the right order: figure out the smallest executor the workload needs, then only allow instance types that can host it. If you need 2 vCPU / 8 GiB, m5.xlarge is the floor for that fleet.

If a machine can't host a useful executor for this job, it isn't spare capacity. It's a place for tasks to die.

Treat Spark config as part of the application

EMR defaults move. Dynamic allocation has been on since 4.4. The 18.75% overhead factor arrived in 5.22. Heterogeneous executors in 5.32 / 6.5. What was fine on one release and instance family can be wrong after you change either.

Executor memory, cores, task size, shuffle, GC, skew, and the EC2 mix belong together. Usually the fix isn't another dozen nodes. It's a few executor and fleet choices. For someone who already knows Spark, YARN, and EMR, finding a better cost-performance point is often hours, not a platform rewrite. That's cheap compared to paying for the same waste on every production run.

Need help?

If Spark on EMR is expensive, flaky, or slower than it should be, I can help. At AWS I spent a lot of time on Spark/EMR deep-dives; they often cut runtime by 30%+ with similar cost savings. On a frequent job that's thousands of dollars a month: fewer EC2 hours, and less compute that never did useful work.

I wrote the ideas and the substance of this article. AI helped with proofreading and formatting. The technical details were accurate at publication.