I'm running my workload on Amazon Elastic Compute Cloud (Amazon EC2) Nitro-based instances. I want to review performance and non-uniform memory access (NUMA) statistics.
Short description
The following factors affect the performance of an application that's hosted on an EC2 instance:
- CPU over utilization
- Memory use
- The EBS volume
- Network statistics
- Not NUMA aware
In NUMA architecture, each CPU has access to its own assigned memory, known as local memory. Each CPU can also access memory that's allocated to other CPUs, known as foreign memory. If applications hosted on your instances aren't NUMA aware, then accessing the foreign memory incurs additional costs and might affect performance. Contact the application vendor to check if your application is NUMA aware.
Resolution
Review NUMA statistics
Note: To review performance components other than NUMA statistics, see How do I troubleshoot slow connections to a website hosted on my EC2 instance?
To review NUMA statistics, complete the following steps:
1. Run the following command to check if the instance type is in a single NUMA or multi-NUMA node.
Note: The instance type is r5.16xlarge has 2 NUMA nodes.
lscpu | grep -i numa
NUMA node(s): 2
2. Run the following command to install the numactl package:
sudo yum install numactl
3. Run the following command to review the NUMA topology:
sudo numactl -H
In the following example output the NUMA topology is divided into two nodes, node 0 and node 1. Node 0 has 32 CPUs and 255,225 MB of memory assigned to it. Node 1 has the other half of the CPUs and 254,924 MB of assigned memory. The node distance indicates the latencies in accessing memory pages from the other node.
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
node 0 size: 255140 MB
node 0 free: 254794 MB
node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
node 1 size: 255225 MB
node 1 free: 254924 MB
node distances:
node 0 1
0: 10 21
1: 21 10
4. Run the following command to review the NUMA statistics:
sudo numastat
The following is an example of the command output:
node node0 node1
numa_hit 314825 288025
numa_miss 0 0
numa_foreign 0 0
interleave_hit 37958 37620
local_node 311752 248476
other_node 3073 39549
For a detailed explanation of each statistic, see A.11. NUMASTAT on the Red Hat Customer Portal.
Increases in the numa_foreign and numa_miss statistics might indicate that the application running on the EC2 instance isn't NUMA aware. This might affect performance. To resolve this issue, turn off NUMA at the grub level. Or, try binding the application to the specific NUMA node. For more information, see Operating system optimizations.
Permanently turn off NUMA
To permanently turn off NUMA on Amazon Linux 2, CentOS7, or RHEL 7, run the following commands:
sudo vi /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="numa=off"
sudo grub2-mkconfig -o /etc/grub2.cfg
sudo reboot
To permanently turn off NUMA on Amazon Linux 2023, run the following command:
# grubby --args "numa=off" —update-kernel DEFAULT