54 An Introduction to the New IBM Eserver pSeries High Performance Switch
Figure 2-34 JTAG access to HPS components
The HMC as fabric manager
Currently, fnmd and hrdw_svr are restricted to running on an HMC. However, that may not be
a permanent condition. There has been talk of allowing these packages to be migrated to a
CSM master. In such a situation, there would likely exist conditions in which the HMC itself
was no longer needed. Until such time, client access for CIMOM,low-level commands (llcmd),
and vterm sessions continue to be provided by hdwr_svr through TCP port 9734.
The most important time for the fabric manager is during initialization. Once all of the nodes
are initialized and brought onto the switch network, the fabric manager can be brought down.
Recovery operations will fail while the fabric manager is down. However, reboots should not
leave a node off of the switch.
In other words, once running, the switch can continue should the HMC fail.
2.6 FNM communication
FNM communication is the API used by the Federation Network Manager daemon (fnmd) to
perform all operations required for management of the pSeries HPS. Multiple FNM daemons
will be running on HMC. This is normal and is to facilitate the various functions of FNM
communication. The various commands are issued through hrdw_svr as shown in
Figure 2-35 on page 55.
Note: The fnmd and Hypervisor functions replace the equivalent functions of the fault
service worm from PSSP’s CSS.
Important: Recovery is required for an MP_DOWN or MP_FAIL error. Five such errors
logged against the same link will render that link fenced. Should a link become fenced, the
only recovery will be to logically power cycle the CEC.
R1
R16
Sw1
Sw3
Sw7
Sw5
Sw2
Sw4
Sw6
Sw8
DCA-F
J-Tag
I2c
pSeries HPS
BPA
Server
SIC
SIC
SIC
SIC
L
D
C
L
D
C
L
D
C
Firmware
CSP
SIC
SIC
L
P
A
R
L
P
A
R
L
P
A
R
L
P
A
R
J
-
T
a
g
HMC
hrdw_svr
Chapter 2. Technical overview of the IBM Eserver pSeries High Performance Switch (HPS) 55
Figure 2-35 FNM command distribution
FNM_Comm operates within four threads: FNM_Init, FNM_Recv, FNM_Rtg, and FNM_Diag,
all shown in Figure 2-36 on page 56. There is an external interface as shown. However, very
little information is available about its use. All switch functions are performed through these
interfaces.
HMC
CSP
SNI Related Commands
Gets -
Frame #, Link Info, VPD,
MTMS, State, Path Table,
SCOM, ESW Level,
Svc Pkt, Location IDs
Sets -
Frame#, State, SCOM,
Svc Pkt, Location IDs
Read Tables -
Path, Route, Error
Write Tables -
Path, Route, Error
Alerts -
CkStop, Svc Pkt Rcvd,
Error
Get
MTMs, Ucode Levels,
Frame #, Status,
SRAM
Set
Frame #, PwrOn Sw,
LED, Bias, Status,
SRAM
Alert
Env Error, BPA failure
Passthru
0x70 & 0x71
Get
Riser Info, SCOM,I2C,
LED, IDs
Set
SCOM, I2C, LED,
SwReset
Alerts
Error Attn, Port Timed,
Pwr Complete,
Riser plug/unplug
BPA
DCA-F
FNM
Command
Distribution
56 An Introduction to the New IBM Eserver pSeries High Performance Switch
Figure 2-36 Fabric Network Manager functions
2.6.1 Time of Day (TOD)
Time of Day is crucial to the internal synchronization of switch master and backups. The
master TOD server is equivalent to the switch primary and is chosen by which CEC has been
up the longest. For performance reasons, it may be beneficial to choose a low-load or
higher-speed system, since synchronization is maintained within device drivers and firmware.
All other CECs are Time of Day slaves. This is equivalent to a switch primary backup. Time of
Day synchronization is maintained within the HPS drivers. No user intervention is required
and this will be synchronized on initialization.
2.6.2 Switch data structures
FNM sets up data in flash memory in the CEC as a series of tables. These CSP tables, also
known as LIDs, are used by the CSP to bring up the adapters during basic initial self test
(BIST) prior to LPAR Ready. This data includes the Path and Route tables, and a set of
SCOM instructions in the MP_Regs LID.
Network table
This is a list of the networks that currently exist and relevant information. Presently, this
consists of one entry per switch.
MSS list
This keeps track of the current set of TOD masters and backups.
Device table
This is a table of all devices physically attached to the HPS network. Specifically, this contains
one entry per switch chip and one entry per SNI.
Hardware
Server
FNM_Init FNM_Recovery
FNM_Route
FNM_Diag
FNM_External
Init Queue
Recov Queue
Route Queue
Diag Queue
Extern
Queue
Outbound
Queue
Response
Response
Response
Response
Response Response ResponseResponse
Response
Async
Async
Async
Async
Async
Async
FNM Communications
response/async async
response data out

Get An Introduction to the New IBM eServer pSeries High Performance Switch now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.