Rack Sectioning

 The rack has eight same sections called “pools”, 32 node in each, with corresponding InfiniBand and Ethernet switches.
Total costs of whole rack is several million dollars, so it required special decisions to debug and test power and connectivity before assembly the whole rack.
Also, interconnect and control network organization assumes that nodes inside pool should have better (and cheaper) connectivity than nodes in different pools and different racks. It allows using backplane.
Backplane is big board, connected 32 nodes and corresponding InfiniBand and Ethernet switches in 1 whole pool. In addition, which is very important, backplane provides 48V power for all these blocks. One backplane can carry up to 20 kW power.

 RackSectioning

Each section consists of:

Section

From network's point of view, one section is:

NodesNetwork

All connectivity inside one pool is proveded by backplane, both Ethernet and InfiniBand networks. Backplane board, of course, isn't cheap, but at any case, cheaper than 64 InfiniBand cables 300$ each, to connect nodes (2 IB port each) with corresponding switches. Even not speaking that there is no room inside the rack where all these cables would have to be placed.

Here need to notice, that the rack has two upper-level pools, for rack management. These pools contain upper-level both Ethernet and InfiniBand switches, to aggregate and route network connections outside the rack.

©2019 Nikolay Bodunov. All Rights Reserved.