Chapter 2. System structure and design 29
When activated, a logical partition can use memory resources located in any book. No matter
in which book the memory resides, a logical partition has access to that memory if so
allocated. Despite the book structure, the z990 is a Symmetric Multi-Processor (SMP).
Each memory card has two memory ports, and each port can access 128 bits (16 bytes).
Actually, the data access path is 140 bits wide, allowing for sophisticated sparing and
error-checking functions. Each port is capable of four fetch and four store operations
concurrently.
Memory upgrade is concurrent when it requires no change of the physical memory card. A
memory card change is disruptive.
Memory sparing
The z990 does not contain spare memory DIMMs. Instead, it has redundant memory
distributed throughout its operational memory and these are used to bypass failing memory.
Replacing memory cards requires the removal of a book and this is disruptive. The extensive
use of redundant elements in the operational memory greatly minimizes the possibility of a
failure that requires memory card replacement.
Memory upgrades
For a model upgrade that results in the addition of a book, a minimum of additional memory is
added to the system. Remember, the minimum physical memory size in a book is 16 GB.
During a model upgrade, the addition of a book is a concurrent operation. The addition of the
physical memory that is in the added book is also concurrent.
If all or part of the additional memory is enabled for installation use, it becomes available to an
active logical partition if this partition has reserved storage defined (see 2.2.9, “Reserved
storage” on page 70 for more detailed information). Or, it may be used by an already defined
logical partition that is activated after the memory addition.
Book replacement and memory
When a book must be replaced, for example, due to an unlikely MCM failure, the memory in
the failing book is removed as well. Until the failing book is replaced, Power-on Reset of the
CPC with the remaining books is not supported.
2.1.4 Ring topology
Concentric loops or rings are constructed such that in a four-book system, each book only is
connected to two others, which means that only data transfers or data transactions to the third
book require passing through one of the other books.
Book-to-book communications are organized as shown in Figure 2-4 on page 30. Book 0
communicates with book 2 and book 3; communication to book 1 must go through another
book (either book 2 or book 3).
30 IBM ^ zSeries 990 Technical Guide
Figure 2-4 Concentric ring structure
A memory-coherent director optimizes ring traffic and filters out cache traffic by not looking on
the ring for cache hits in other books if it is certain that the resources for a given logical
partition exists in the same book.
The Level 2 (L2) cache is implemented on four cache (SD) chips. Each SD chip holds 8 MB,
resulting in a 32 MB L2 cache per book. The L2 cache is shared by all PUs in the book and
has a store-in buffer design. The connection to processor memory is done through four
high-speed memory buses.
There is a ring structure within which the books maintain interbook communication at the L2
cache level. Additional books extend the function of the ring structure for interbook
communication. The ring topology is shown in Figure 2-5 and Figure 2-6 on page 31, and in
Figure 2-7 on page 31.
A book jumper completes the ring in order to be able to insert additional books into the ring
non-disruptively.
Figure 2-5 Two-book system ring structure
Book 0Book 3 Book 1 Book 2
Jumper Book
STI
MBA
STI
STI STI
STI
MBA
STI
STI STI
4 x 2.5 GB/S 4 x 2.5 GB/S
STI
MBA
STI
STI STI
4 x 2.5 GB/S
Ring Structure
PU PUPUPU PU PU PU PU PU PU PU PU
Jumper Book
STI
MBA
STI
STI STI
STI
MBA
STI
STI ST I
4 x 2 GB/S 4 x 2 GB/S
STI
MBA
STI
STI STI
4 x 2 GB/S
MBA Card
PU PUPUPU PU PU PU PU PU PU PU PU
Level 2 Cache 32MB
MCM
MCM
Memory Up to 64GB
Level 2 Cache 32MB
Level 2 Cache 32MB
Memory Up to 64GB
MCM
MCM
Get IBM eServer zSeries 990 Technical Guide now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
