Rack Interconnect Networking

Rack contains 4 independent networks inside:

- FDR 4G InfiniBand Network for interconnect

- FDR 4G InfiniBand network for storage

- 2x1G independent networks for control and PXE booting

Rack interconnect and storage networking seems a bit complicated, but in fact, it's quite simple from organizational point of view (but not simple for engineering implementation, of course!):

- each computational node is connected to 1-st level InfiniBand switches (interconnect and storage) via backplane

- each InfiniBand Interconnect Switch (MPI Fabric) has 28 passive cables FDR 4X that connected with InfiniBand switches on adjacent racks, or with adjacent sections inside the rack

- each InfiniBand Storage Switch (Storage Fabric) has 18 active cables that connected with remote (~100m) storage fabric

About connection media (cables, etc.):

- each network connection (control or storage or interconnect) inside one pool is routed via backplane

- each network connection outside the pool, but inside the rack, or in adjacent racks connected via cheap passive cables

- remote connections made by expensive active optical cables

Interconnect

 

 

 

©2019 Nikolay Bodunov. All Rights Reserved.