An Introduction to the New IBM eServer pSeries High Performance Switch

54 An Introduction to the New IBM Eserver pSeries High Performance Switch

Figure 2-34 JTAG access to HPS components

The HMC as fabric manager

Currently, fnmd and hrdw_svr are restricted to running on an HMC. However, that may not be

a permanent condition. There has been talk of allowing these packages to be migrated to a

CSM master. In such a situation, there would likely exist conditions in which the HMC itself

was no longer needed. Until such time, client access for CIMOM,low-level commands (llcmd),

and vterm sessions continue to be provided by hdwr_svr through TCP port 9734.

The most important time for the fabric manager is during initialization. Once all of the nodes

are initialized and brought onto the switch network, the fabric manager can be brought down.

Recovery operations will fail while the fabric manager is down. However, reboots should not

leave a node off of the switch.

In other words, once running, the switch can continue should the HMC fail.

2.6 FNM communication

FNM communication is the API used by the Federation Network Manager daemon (fnmd) to

perform all operations required for management of the pSeries HPS. Multiple FNM daemons

will be running on HMC. This is normal and is to facilitate the various functions of FNM

communication. The various commands are issued through hrdw_svr as shown in

Figure 2-35 on page 55.

Note: The fnmd and Hypervisor functions replace the equivalent functions of the fault

service worm from PSSP’s CSS.

Important: Recovery is required for an MP_DOWN or MP_FAIL error. Five such errors

logged against the same link will render that link fenced. Should a link become fenced, the

only recovery will be to logically power cycle the CEC.

R1

R16

Sw1

Sw3

Sw7

Sw5

Sw2

Sw4

Sw6

Sw8

DCA-F

J-Tag

I2c

pSeries HPS

BPA

Server

SIC

SIC

SIC

SIC

L

D

C

L

D

C

L

D

C

Firmware

CSP

SIC

SIC

L

P

A

R

L

P

A

R

L

P

A

R

L

P

A

R

J

-

T

a

g

HMC

hrdw_svr

Chapter 2. Technical overview of the IBM Eserver pSeries High Performance Switch (HPS) 55

Figure 2-35 FNM command distribution

FNM_Comm operates within four threads: FNM_Init, FNM_Recv, FNM_Rtg, and FNM_Diag,

all shown in Figure 2-36 on page 56. There is an external interface as shown. However, very

little information is available about its use. All switch functions are performed through these

interfaces.

HMC

CSP

SNI Related Commands

Gets -

Frame #, Link Info, VPD,

MTMS, State, Path Table,

SCOM, ESW Level,

Svc Pkt, Location IDs

Sets -

Frame#, State, SCOM,

Svc Pkt, Location IDs

Read Tables -

Path, Route, Error

Write Tables -

Path, Route, Error

Alerts -

CkStop, Svc Pkt Rcvd,

Error

Get

MTMs, Ucode Levels,

Frame #, Status,

SRAM

Set

Frame #, PwrOn Sw,

LED, Bias, Status,

SRAM

Alert

Env Error, BPA failure

Passthru

0x70 & 0x71

Get

Riser Info, SCOM,I2C,

LED, IDs

Set

SCOM, I2C, LED,

SwReset

Alerts

Error Attn, Port Timed,

Pwr Complete,

Riser plug/unplug

BPA

DCA-F

FNM

Command

Distribution

56 An Introduction to the New IBM Eserver pSeries High Performance Switch

Figure 2-36 Fabric Network Manager functions

2.6.1 Time of Day (TOD)

Time of Day is crucial to the internal synchronization of switch master and backups. The

master TOD server is equivalent to the switch primary and is chosen by which CEC has been

up the longest. For performance reasons, it may be beneficial to choose a low-load or

higher-speed system, since synchronization is maintained within device drivers and firmware.

All other CECs are Time of Day slaves. This is equivalent to a switch primary backup. Time of

Day synchronization is maintained within the HPS drivers. No user intervention is required

and this will be synchronized on initialization.

2.6.2 Switch data structures

FNM sets up data in flash memory in the CEC as a series of tables. These CSP tables, also

known as LIDs, are used by the CSP to bring up the adapters during basic initial self test

(BIST) prior to LPAR Ready. This data includes the Path and Route tables, and a set of

SCOM instructions in the MP_Regs LID.

Network table

This is a list of the networks that currently exist and relevant information. Presently, this

consists of one entry per switch.

MSS list

This keeps track of the current set of TOD masters and backups.

Device table

This is a table of all devices physically attached to the HPS network. Specifically, this contains

one entry per switch chip and one entry per SNI.

Hardware

Server

FNM_Init FNM_Recovery

FNM_Route

FNM_Diag

FNM_External

Init Queue

Recov Queue

Route Queue

Diag Queue

Extern

Queue

Outbound

Queue

Response

Response

Response

Response

Response Response ResponseResponse

Response

Async

Async

Async

Async

Async

Async

FNM Communications

response/async async

response data out