How can I review NUMA statistics for my workload running on EC2 instances?

3 minute read
0

I'm running my workload on Amazon Elastic Compute Cloud (Amazon EC2) Nitro-based instances. I want to review performance and non-uniform memory access (NUMA) statistics.

Short description

The following factors affect the performance of an application that's hosted on an EC2 instance:

  • CPU over utilization
  • Memory use
  • The EBS volume
  • Network statistics
  • Not NUMA aware

In NUMA architecture, each CPU has access to its own assigned memory, known as local memory. Each CPU can also access memory that's allocated to other CPUs, known as foreign memory. If applications hosted on your instances aren't NUMA aware, then accessing the foreign memory incurs additional costs and might affect performance. Contact the application vendor to check if your application is NUMA aware.

Resolution

Review NUMA statistics

Note: To review performance components other than NUMA statistics, see How do I troubleshoot slow connections to a website hosted on my EC2 instance?

To review NUMA statistics, complete the following steps:

1.    Run the following command to check if the instance type is in a single NUMA or multi-NUMA node.

Note: The instance type is r5.16xlarge has 2 NUMA nodes.

lscpu | grep -i numa
NUMA node(s): 2

2.    Run the following command to install the numactl package:

sudo yum install numactl 

3.    Run the following command to review the NUMA topology:

sudo numactl -H

In the following example output the NUMA topology is divided into two nodes, node 0 and node 1. Node 0 has 32 CPUs and 255,225 MB of memory assigned to it. Node 1 has the other half of the CPUs and 254,924 MB of assigned memory. The node distance indicates the latencies in accessing memory pages from the other node.


available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
node 0 size: 255140 MB
node 0 free: 254794 MB
node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
node 1 size: 255225 MB
node 1 free: 254924 MB
node distances:
node   0   1
0:  10  21
1:  21  10

4.    Run the following command to review the NUMA statistics:

sudo numastat

The following is an example of the command output:

node                       node0           node1
numa_hit                  314825          288025
numa_miss                      0               0
numa_foreign                   0               0
interleave_hit             37958           37620
local_node                311752          248476
other_node                  3073           39549

For a detailed explanation of each statistic, see A.11. NUMASTAT on the Red Hat Customer Portal.

Increases in the numa_foreign and numa_miss statistics might indicate that the application running on the EC2 instance isn't NUMA aware. This might affect performance. To resolve this issue, turn off NUMA at the grub level. Or, try binding the application to the specific NUMA node. For more information, see Operating system optimizations.

Permanently turn off NUMA 

To permanently turn off NUMA on Amazon Linux 2, CentOS7, or RHEL 7, run the following commands:

sudo vi /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="numa=off"
sudo grub2-mkconfig -o /etc/grub2.cfg
sudo reboot

To permanently turn off NUMA on Amazon Linux 2023, run the following command:

# grubby --args "numa=off" —update-kernel DEFAULT
AWS OFFICIAL
AWS OFFICIALUpdated a year ago