IBM eX5 Implementation Guide

26 IBM eX5 Implementation Guide

2.3.4 Nonuniform memory architecture (NUMA)

Nonuniform memory architecture (NUMA) is an important consideration when configuring

memory, because a processor can access its own local memory faster than non-local

memory. Not all configurations use 64 DIMMs spread across 32 channels. Certain

configurations might have a more modest capacity and performance requirement. For these

configurations, another principle to consider when configuring memory is that of

balance. A

balanced configuration has all of the memory cards configured with the same

amount of

memory, even if the quantity and size of the DIMMs differ from card to card. This principle

helps to keep remote memory access to a minimum. DIMMs must always be installed in

matched pairs.

A server with a NUMA, such as the servers in the eX5 family, has local and remote memory.

For a given thread running in a processor core,

local memory refers to the DIMMs that are

directly connected to that particular processor.

Remote memory refers to the DIMMs that are

not connected to the processor where the thread is running currently.

Remote memory is attached to another processor in the system and must be accessed

through a QPI link. However, using remote memory adds latency. The more such latencies

add up in a server, the more performance can degrade. Starting with a memory configuration

where each CPU has the same local RAM capacity is a logical step toward keeping remote

memory accesses to a minimum.

For more information about NUMA installation options, see the following sections:

򐂰 IBM System x3850 X5: 3.8.2, “DIMM population sequence” on page 79

򐂰 IBM System x3690 X5: “Two processors with memory mezzanine installed” on page 135

򐂰 IBM BladeCenter HX5: 5.10.2, “DIMM population order” on page 196

2.3.5 Hemisphere Mode

Hemisphere Mode is an important performance optimization of the Xeon 6500 and 7500

processors. Hemisphere Mode is automatically enabled by the system if the memory

configuration allows it. This mode interleaves memory requests between the two memory

controllers within each processor, enabling reduced latency and increased throughput. It also

allows the processor to optimize its internal buffers to maximize memory throughput.

Hemisphere Mode is a global parameter that is set at the system level. This setting means

that if even one processor’s memory is incorrectly configured, the entire system can lose the

performance benefits of this optimization. Stated another way, either all processors in the

system use Hemisphere Mode, or all do not.

Hemisphere Mode is enabled only when the memory configuration behind each memory

controller on a processor is identical. Because the Xeon 7500 memory population rules

dictate that a minimum of two DIMMs are installed on each memory controller at a time (one

Tip: Additional ranks increase the memory bus loading, which is why on Xeon 5500

(Nehalem EP) platforms, the opposite effect can occur: memory slows down if too many

rank loads are attached. The use of scalable memory buffers in the x3850 X5 with Xeon

7500/6500 processors avoids this slowdown.

Two-node configurations: A memory configuration that enables Hemisphere Mode is

required for 2-node configurations on x3850 X5.