NUMA (Non Uniform Memory Access) for MongoDB

October 9, 2014 · by · in MongoDB

From our internal testing we did see some performance boost when NUMA was disabled using -> numactl –interleave=all …..

Below are the details from MongoDB
http://docs.mongodb.org/manual/administration/production-notes/
Running MongoDB on a system with Non-Uniform Access Memory (NUMA) can cause a number of operational problems, including slow performance for periods of time or high system process usage.

When running MongoDB on NUMA hardware, you should disable NUMA for MongoDB and instead set an interleave memory policy.

NOTE
MongoDB version 2.0 and greater checks these settings on start up when deployed on a Linux-based system, and prints a warning if the system is NUMA-based.
To disable NUMA for MongoDB and set an interleave memory policy, use the numactl command and startmongod in the following manner:

numactl –interleave=all /usr/bin/local/mongod

Then, disable zone reclaim in the proc settings using the following command:

echo 0 > /proc/sys/vm/zone_reclaim_mode

To fully disable NUMA, you must perform both operations. For more information, see the Documentation for /proc/sys/vm/*

Sample box configs :-

Current box: –

bash-4.1$ numactl –hardware
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22
node 0 size: 36853 MB
node 0 free: 249 MB
node 1 cpus: 1 3 5 7 9 11 13 15 17 19 21 23
node 1 size: 36863 MB
node 1 free: 109 MB
node distances:
node 0 1
0: 10 20
1: 20 10

> grep -i numa /var/log/dmesg
NUMA: Allocated memnodemap from c000 – c140
NUMA: Using 29 for the hash shift.

> numastat -n

Per-node numastat info (in MBs):
Node 0 Node 1 Total
————— ————— —————
Numa_Hit 1617461.97 1074067.19 2691529.16
Numa_Miss 356851.69 549159.36 906011.05
Numa_Foreign 549159.36 356851.69 906011.05
Interleave_Hit 47199.50 49364.78 96564.28
Local_Node 1597033.36 1042082.41 2639115.77
Other_Node 377280.30 581144.14 958424.44

Changes required:
1. Add to /etc/sysctl.conf: vm.zone_reclaim_mode = 0
2. Call mongo with: numactl –interleave=all /opt/mongodb/bin/mongod ………..

The new MongoDB V2.6 startup script have handle this as it includes the lines below:

# Handle NUMA access to CPUs (SERVER-3574)
# This verifies the existence of numactl as well as testing that the command works
NUMACTL_ARGS=”–interleave=all”
if which numactl >/dev/null 2>/dev/null && numactl $NUMACTL_ARGS ls / >/dev/null 2>/dev/null
then
NUMACTL=”numactl $NUMACTL_ARGS”
else
NUMACTL=””
Fi

  • Ask Question