When Amazon EMR Gives YARN More Memory Than the Machine Has

Amazon EMR publishes yarn.nodemanager.resource.memory-mb per instance type in its task configuration tables. NodeManager treats that as the budget for containers.

On some sizes, that budget is larger than the RAM the guest OS can actually use. I first hit this at AWS. The defaults are still in those public tables.

The numbers do not always add up

Take m5.24xlarge. AWS advertises 384 GiB. EMR configures:

yarn.nodemanager.resource.memory-mb = 385024

That's 376 GiB YARN is allowed to allocate.

Linux does not necessarily see the full 384 GiB. Public NUMA measurements for m5.24xlarge report about 191,415 MiB per NUMA node across two nodes: roughly 374 GiB.

EC2 advertised memory:     384 GiB
Linux-usable memory:      ~374 GiB
YARN configured memory:    376 GiB

YARN can therefore schedule more container memory than the guest has.

The same pattern shows up on m5.16xlarge. EMR sets 253,952 MiB (248 GiB) for YARN. Public NUMA numbers are about 246.5 GiB across the two nodes.

That gap matters. Linux MemTotal is usable RAM after platform and kernel reservations. The OS, NodeManager, Hadoop daemons, and agents still need their own slice of what's left.

Accounting is not RAM

yarn.nodemanager.resource.memory-mb is an accounting limit. It tells YARN how much it may allocate. It does not create physical RAM.

If YARN packs containers up to that limit, the machine can hit memory pressure even when every Spark executor fits its container size.

You can check heap, overhead, and YARN container math, confirm it all fits, and still be looking in the wrong place. The Spark config can be internally consistent while the node-level assumption is not.

Adding nodes can hide the symptom. It does not explain the cause.

Override the EMR defaults

EMR does not lock these values. yarn.nodemanager.resource.memory-mb lives in the yarn-site classification, and application configuration can be overridden. Uniform instance groups can take different configs per group via Configurations on InstanceGroupConfig.

{
  "Classification": "yarn-site",
  "Properties": {
    "yarn.nodemanager.resource.memory-mb": "YOUR_SAFE_VALUE"
  }
}

Don't subtract a few gigabytes and call it fixed. The right number depends on the instance type, EMR release, what's installed on the cluster, and what else runs on the worker.

Treat YARN memory as part of the application

Spark config, YARN accounting, Linux memory, and the EC2 type belong together. Usually the fix isn't another dozen nodes. It's a YARN override that matches what the guest can actually use.

The failure may surface in Spark. The problem can sit a few layers down. Debugging from the application downward can take a team days. For someone who already knows Spark, YARN, and EMR, finding that mismatch is often hours.

Need help?

If EMR jobs fail in ways that don't make sense from the Spark config alone, I can help. At AWS I spent a lot of time on Spark/EMR deep-dives. Sometimes the useful fix isn't more compute. It's why the compute you already paid for is behaving incorrectly.

I wrote the ideas and the substance of this article. AI helped with proofreading and formatting. The technical details were accurate at publication.