Node tuning

Although Nitro is a Go application, it can use significantly more memory than Go's runtime reports. Nitro relies on multiple allocators:

• the Go garbage-collected heap
• CGO (Go's mechanism for calling C code) allocations via calloc
• and direct mmap system calls, each with its own accounting.

Understanding where memory lives, which configuration knobs control it, and how to tune validators is essential for stable operation.

Memory allocators in Nitro

Nitro's total resident memory (RSS) is the sum of four distinct categories:

Allocator	What uses it	Visible in Go memstats?	Controlled by
Go heap	State trie (dirty), transaction processing, goroutine stacks, general application data	Yes	GOMEMLIMIT, trie-dirty-cache
calloc	Pebble block cache, Pebble memtables, Stylus WASM cache	No	database-cache, stylus-lru-cache-capacity
mmap	fastcache (trie-clean and snapshot caches)	No	trie-clean-cache, snapshot-cache
glibc malloc arenas	Per-thread arena overhead for CGO allocations	No	MALLOC_ARENA_MAX

Only the Go heap is subject to Go's garbage collector and GOMEMLIMIT. The CGO and mmap allocations are invisible to Go's runtime. They don't appear in runtime.MemStats or standard Go memory profiles, but they still consume container memory and count toward your memory limit.

Go heap

The Go runtime manages its own heap for all pure-Go allocations. Key consumers include:

• Dirty trie cache (trie-dirty-cache): Modified state trie nodes held in memory before being flushed to disk. Defaults to 1024 MB and is one of the largest bounded caches on the Go heap.
• Contract code cache: An LRU cache of contract bytecode, hardcoded at 256 MB and isn't configurable.
• Activated WASM cache: Compiled Stylus WASM modules cached on the Go heap, hardcoded at 64 MB.
• fastcache index maps: Although fastcache stores its data via mmap, each instance maintains a Go-side index (bucket maps of uint64 to uint64). With two large fastcache instances (trie-clean and snapshot), this index metadata can consume hundreds of MB on the Go heap.
• Snapshot diff layers: Up to 128 diff layers can accumulate, each holding Go maps of modified accounts and storage slots.
• Goroutine stacks, block/receipt caches, and GC overhead: Goroutine stacks, recently accessed blocks/receipts, and Go's own GC metadata collectively add further pressure.

Go reports its total memory usage via runtime.MemStats.Sys, which includes the heap, stack space, and GC metadata. This is the portion of memory that GOMEMLIMIT governs.

CGO allocations (Pebble and Stylus)

Nitro's on-disk database, Pebble, allocates its block cache and memtables through CGO calloc() calls (see pebble/internal/manual/manual.go in the source). These allocations go through the C memory allocator and are out of scope for Go's memory tracking.

Pebble block cache is the largest CGO consumer. It caches frequently read database blocks in memory to avoid disk I/O. Its size is set directly by the database-cache configuration parameter.

Pebble memtables buffer recent writes before they are flushed to disk. Nitro configures four memtables, each sized at database-cache / 8, for a combined maximum of database-cache / 2. For the default database-cache of 2048 MB, this means up to 1024 MB of memtable space (four memtables of 256 MB each).

Stylus WASM cache stores compiled WebAssembly modules for Stylus smart contracts. Rust allocates this cache (invoked through CGO), and stylus-lru-cache-capacity bounds its size.

Raw mmap allocations (fastcache)

Two caches use fastcache, a library that allocates memory via direct mmap system calls, bypassing both Go's allocator and CGO:

• Trie-clean cache (trie-clean-cache): Caches unchanged state trie nodes. Default: 600 MB.
• Snapshot cache (snapshot-cache): Caches state snapshot data for fast reads. Default: 400 MB.

Because fastcache uses raw mmap, this memory doesn't appear in Go's memstats or standard profiling tools. You can only see it by inspecting /proc/<pid>/smaps at the OS level. Each fastcache instance allocates memory in 64 MB chunks, making these regions identifiable when analyzing process memory maps.

glibc malloc arenas

When Nitro makes CGO calls (for Pebble, Stylus, etc.), the resulting C-side allocations go through the system's default C memory allocator: glibc malloc. Unlike Go's garbage-collected heap, malloc manages memory by requesting large regions from the OS and subdividing them to satisfy individual allocation requests. Freed memory is returned to the allocator's internal free lists rather than immediately back to the OS, so the process's RSS can remain elevated even after allocations are freed.

To handle concurrent allocations efficiently, glibc malloc uses arenas, which are independent memory pools, each with its own lock. When a thread allocates memory, it picks an arena, reducing contention compared to a single global lock. By default, glibc creates up to 8 × CPU_count arenas, each reserving a 64 MB region. The worst-case overhead for arenas is:

Arena overhead = 8 × CPU_count × 64 MB

In containerized environments, glibc detects the underlying host CPU count (not the container's CPU requests), which often results in far more arenas than needed. As the process runs and more threads make CGO calls, glibc creates and retains new arenas, causing RSS to drift upward over days or weeks even though no individual allocation is leaking.

This can be controlled with the MALLOC_ARENA_MAX environment variable:

MALLOC_ARENA_MAX=2

Setting MALLOC_ARENA_MAX=2 caps glibc to two arenas, reducing worst-case arena overhead from gigabytes to ~128 MB. In testing, this eliminated the slow memory growth with no measurable performance impact on RPC throughput.

Without MALLOC_ARENA_MAX, a Nitro node on a large host can accumulate gigabytes of arena overhead that appears as a "memory leak" because RSS grows steadily while Go reports stable usage. This is the most common cause of unexplained memory growth in long-running Nitro nodes.

Thread stacks

Nitro spawns native threads for CGO operations (Pebble, compression libraries) and Stylus execution.

CLI flags that control memory usage

These are the flags and environment variables that bound Nitro's memory. All cache flags live under execution.caching.

Flag or variable	Default	Allocator	What it controls
--execution.caching.database-cache	2048 MB	CGO (calloc)	Pebble block cache. Also sets memtable sizes (database-cache / 2).
--execution.caching.trie-dirty-cache	1024 MB	Go heap	Modified trie nodes awaiting flush to disk.
--execution.caching.trie-clean-cache	600 MB	mmap (fastcache)	Unchanged trie nodes cached for read performance.
--execution.caching.snapshot-cache	400 MB	mmap (fastcache)	State snapshot data for fast lookups.
--execution.caching.stylus-lru-cache-capacity	256 MB	Rust (via CGO)	Compiled Stylus WASM modules.
GOMEMLIMIT (env var)	unset	Go heap	Soft memory ceiling for the Go runtime. See below.
MALLOC_ARENA_MAX (env var)	unset	glibc arenas	Caps glibc malloc arenas. Set to 2 on every containerized node.
--node.resource-mgmt.mem-free-limit	empty (off)	—	Declines RPC calls when free memory (excluding page cache) drops below this.
--node.block-validator.memory-free-limit	default (1GB)	—	Pauses block validation when free memory drops below this. Empty string disables.

Tip

All of the execution.caching values are bounded by configuration and won't grow beyond their configured limits, so total non-Go memory is predictable and can be calculated from your configuration.

Calculating GOMEMLIMIT

GOMEMLIMIT is an environment variable that sets a soft memory limit for the Go runtime. When set, Go's garbage collector (GC) runs more aggressively as heap usage approaches the limit, helping to keep total Go memory usage below the target. Without it, the GC relies solely on the GOGC environment variable (which defaults to 100, meaning the GC triggers when the heap doubles in size since the last collection) and has no awareness of an absolute memory ceiling.

For GOMEMLIMIT to work correctly in a containerized environment, you must reserve enough headroom for all the non-Go memory that competes for the container's memory limit.

Non-Go memory budget

Sum all memory that lives outside the Go heap:

Non-Go Memory =
    database-cache            # Pebble block cache (CGO)
  + (database-cache / 2)      # Pebble memtables, max (CGO)
  + trie-clean-cache           # fastcache (mmap)
  + snapshot-cache             # fastcache (mmap)
  + stylus-lru-cache-capacity  # Stylus WASM (Rust)
  + malloc arena overhead      # glibc arenas
  + ~300 MB                    # Thread stacks (varies by workload)

With MALLOC_ARENA_MAX=2, arena overhead is ~128 MB. Without it, arena overhead can grow to several gigabytes depending on host CPU count. See glibc malloc arenas above.

Formula

GOMEMLIMIT = Container_Memory_Limit - Non_Go_Memory - Safety_Margin

You should use a safety margin of 300-500 MB to account for allocator overhead, transient allocations, and kernel page cache.

Example: 16 GB container with defaults

Component	Size	Source
Pebble block cache	2,048 MB	database-cache (CGO)
Pebble memtables (max)	1,024 MB	database-cache / 2 (CGO)
Trie-clean cache	600 MB	trie-clean-cache (fastcache)
Snapshot cache	400 MB	snapshot-cache (fastcache)
Stylus WASM cache	256 MB	stylus-lru-cache-capacity (Rust)
Malloc arenas	128 MB	MALLOC_ARENA_MAX=2
Thread stacks	300 MB*	~2 MB per thread
Total non-Go	4,756 MB

*Thread stack usage depends on the number of active threads, which varies by workload.

GOMEMLIMIT = 16,384 MB - 4,756 MB - 400 MB safety = ~11,228 MB ≈ 11 GB

If GOMEMLIMIT is set too high (not accounting for non-Go memory), the Go garbage collector defers collection, expecting more room than actually exists. The OS then OOM-kills the process when total RSS (Go heap plus all non-Go allocations) exceeds the container limit.

Helm chart shortcut

The community Helm chart calculates GOMEMLIMIT automatically using a multiplier of 0.9 against the container memory limit. It subtracts non-Go memory defaults (database-cache 2048, trie-clean-cache 600, snapshot-cache 400, stylus-lru-cache 256, malloc arena size 64) before applying the multiplier. If you use the Helm chart with default values, GOMEMLIMIT is set for you.

Validator tuning

Validators perform CPU-intensive WASM execution to verify blocks. Their memory and CPU profiles differ from the main Nitro node.

GOMEMLIMIT for validators

Validators use a lower GOMEMLIMIT multiplier of 0.75 (compared to 0.9 for the main node) because WASM execution creates more transient Go heap allocations that need GC headroom. For a 16 GB validator container:

GOMEMLIMIT ≈ 16,384 MB × 0.75 = ~12,288 MB ≈ 12 GB

Memory free limits

Nitro can throttle RPC and pause block validation when system free memory drops below a configurable threshold:

• --node.resource-mgmt.mem-free-limit: Declines incoming RPC calls when free memory (excluding the page cache) falls below this amount. Disabled by default — set it to enable RPC throttling (for example, --node.resource-mgmt.mem-free-limit 1GB).
• --node.block-validator.memory-free-limit: Pauses block validation when free memory falls below this amount. Enabled by default at 1GB; pass an empty string to disable it.

Note

The community Helm chart sets both limits to 0.05 (5%) of the container memory limit. Those multipliers are Helm-chart defaults, not Nitro defaults — if you're not running the chart, set the Nitro flags above explicitly.

GOMAXPROCS

GOMAXPROCS controls how many OS threads Go uses for goroutine scheduling. For validators, the Helm chart defaults to a multiplier of 2 against the container CPU limit (for example, 4 CPUs yields GOMAXPROCS=8). This higher-than-usual setting helps because validators frequently block on CGO calls during WASM execution, and additional Go threads keep non-CGO goroutines progressing.

For the main Nitro node, the Helm chart does not override GOMAXPROCS, letting Go's runtime auto-detect from the container CPU limit.

Tuning recommendations

To learn how to monitor node health and metrics, see Node monitoring.

General rules

1. Set MALLOC_ARENA_MAX=2: This is the single most impactful change for containerized nodes. Without it, glibc can waste gigabytes on arena overhead, causing RSS to drift upward over days. Set this environment variable on every Nitro container.

2. Start from the formula: Calculate GOMEMLIMIT using the formula above with your actual cache configuration values. Do not set it to the container memory limit.

3. Monitor RSS, not just Go heap: Set container memory alerts based on actual RSS (container_memory_rss in Prometheus and cAdvisor), not Go-reported memory.

4. All caches are bounded: Unlike memory leaks, all non-Go memory in Nitro is bounded by configuration. With MALLOC_ARENA_MAX set, if RSS is stable and predictable, the node is behaving correctly. The memory is simply allocated outside Go's visibility.

Example configurations

Full node with 32 GB RAM

Use default cache sizes. Calculate GOMEMLIMIT from the non-Go budget:

# Non-Go: 2048 + 1024 + 600 + 400 + 256 + 128 + 300 = 4,756 MB
# GOMEMLIMIT = 32,768 - 4,756 - 500 safety ≈ 27,512 MB ≈ 27 GB
MALLOC_ARENA_MAX=2 \
GOMEMLIMIT=27512MiB \
nitro \
  --metrics \
  --metrics-server.addr 0.0.0.0

Archive node with 128 GB RAM

Archive nodes benefit from larger caches to reduce disk I/O for historical queries:

# Non-Go: 4096 + 2048 + 1200 + 800 + 256 + 128 + 300 = 8,828 MB
# GOMEMLIMIT = 131,072 - 8,828 - 500 safety ≈ 121,744 MB ≈ 119 GB
MALLOC_ARENA_MAX=2 \
GOMEMLIMIT=121744MiB \
nitro \
  --execution.caching.database-cache 4096 \
  --execution.caching.trie-dirty-cache 2048 \
  --execution.caching.trie-clean-cache 1200 \
  --execution.caching.snapshot-cache 800 \
  --metrics \
  --metrics-server.addr 0.0.0.0

Validator with 16 GB RAM

Validators need less cache but more CPU headroom. Use reduced cache sizes and the lower GOMEMLIMIT multiplier:

# Non-Go: 1024 + 512 + 300 + 200 + 256 + 128 + 300 = 2,720 MB
# GOMEMLIMIT = 16,384 × 0.75 ≈ 12,288 MB
# Alternatively: 16,384 - 2,720 - 500 = 13,164 MB (use the lower value)
MALLOC_ARENA_MAX=2 \
GOMEMLIMIT=12288MiB \
GOMAXPROCS=4 \
nitro \
  --execution.caching.database-cache 1024 \
  --execution.caching.trie-clean-cache 300 \
  --execution.caching.snapshot-cache 200 \
  --metrics \
  --metrics-server.addr 0.0.0.0