Rack Powering

General overview

Let's look on the rack’s power parameters first:

Provides up to 194kW of total power
N+1 power redundancy for each compute section
380 VAC 3-phase input
48 VDC bus with metering function
8 groups x 12 PSUs (3 kW PSU model with up to 97,2% efficiency

Now consider in more details, why we choose that power architecture.

For motherboard in big rack arises many contradictions. Blade servers have its own AC/DC power sources, but it not appropriate for supercomputer rack.
First of all, by safety reasons. We have 256 nodes in the rack, so any short circuit from 380V AC node input power to a ground might lead to personnel's damage with high level of probability.
Second reason is reliability. In case of connection several DC sources in parallel, we have possibility to reserve power sources, making it possible to use N+1 schemes, for uninterruptible, hot-plug power source replacement.
Third reason is a cost, because off-the-shelf AC/DC PSU, being commodities, reduces the rack’s cost quite significant.
Fourth reason is component’s physical volume. We should have nodes as thin as possible, but big coils and high-voltage capacitors don't allow us to decrease node’s thickness.
Therefore, we decided to use external (placed outside the node, but inside the rack) AC/DC power source. Which voltage they should generate? We have some trade-off here: high voltage leads to risk of personnel’s damage, low voltage leads to massive current-carrying buses and heat emission from them. 48V is the best choise for that purposes.

 

Rack level

Let's look at corresponding pictures here.

This is how 380V AC power is distributed inside the rack:

Rackpowering

We can see power lanes on the left side there, that carry 380V AC power. Note that in assembled rack these power lanes are placed behind nodes and PSUs, which prevents access to them during any block replacement, to avoid personell's injuries.

We can see at this in more details:

PowerLanes220V

 

See a picture that shows AC/DC power sources block. One pool consumes about 20 kW in peak (which never happens), so we need 6 PSU 3 kW each plus 3 reserved, if we use 3+1 power scheme for PSU hotplug.

Section

 

Pool level 

Well, let's move on to DC power distribution.

After we get 48V DC from PSUs, we need to distribute the power between nodes and network switches. Please remenber that one node consumes up to 500W, and we have 32 nodes in one pool. Consequently, we need to equally (to avoid addidtional power losses) distribute the power between these 32 nodes via backplane. Backplane has very thick copper planes, to reduce power losses, or, in other words, to reduce voltage drop and heat emission.

However before we start to distribute this power between nodes, we need to carry the power on the backplane from PSUs. It's not so simple as in seems at the first glance. See at the picture how it is implemented:

PoopPower

We see very thick red "+" and blue "-" power lanes, which come from PSU's block and then come along the backplane, because even many thick copper planes on the backplane can't equally distribute the energy along backplane.

©2019 Nikolay Bodunov. All Rights Reserved.