Server Power Data

This page documents GPU and other power data. (figures given in brackets are with 1 redundant power supply (-) removed.)

server power supply Powered Off Peak load during boot idle Full (GPU) load full load plus bonnie++ on all disks
Hannah (Asus 8000 g4 w 8 Geforce 2080-ti) PS1 0.26 (-)(-) 1.19 0.74 (-)(-) 2.74(-)(-) 2.4(-)(-)
PS2 0.26 (0.39)(-) 1.02 0.74 (1.12) 2.53(?)(-) 2.22(?)(-)
PS3 0.26(0.39)(N/A) 1.02 0.74(0.95)(N/A) 2.53(3.89)(N/A) 2.19(?)(N/A)
lennoxtown (gigabyte w 8 Geforce 2080-ti) PS1 083. (-)(-) 1.5 1.36 (-)(-) 3.18(-)(-) 3.15(-)(-)
PS2 0.88(1.27)(n/a) 1.7 1.56(1.68)(n/a) 3.39(4.69)(-) 3.4(?)(-)
PS3 0.87(1.15)(n/a) 1.5 1.40(1.53)(n/a) 3.17(5.02)(N/A) 3.18(?)(n/a)
to be determined (tyan f77d w 8 Geforce 2080-ti) PS1 0. (-)()() 0. 0. (-) 0.(-) 0.
PS2 0.(0.)(-)() 0. 0.(0.)()(-) 0.(0.)(-)() 0.
PS3 0.(0.)()(-) 0. 0.(0.)()(-) 0.(0.)()(-) 0.

Basic procedure (work in progress)

  1. Power off machine.
  2. Replug machine using power meters
  3. Take readings with machine powered off
  4. Remove PSU in sequence taking readings (machine still powered off)
  5. Power On machine
  6. Take readings of meters noting the mac reading on each meter until the login prompt appears.
  7. Poweroff Server
    1. 1 Remove PSU in sequence
    2. 2 Boot server
    3. 3 Take readings of meters noting the mac reading on each meter until the login prompt appears
    4. 4 goto 7 (increment PSU)

Dells

It turns out that the dell bmc will store some power data which is accessble to ipmi through an oem extension:

[glorious]root:  /usr/bin/ipmitool delloem powermonitor
Power Tracking Statistics
Statistic      : Cumulative Energy Consumption
Start Time     : Mon Mar  2 18:05:00 2015
Finish Time    : Mon Dec  9 07:51:16 2019
Reading        : 3143.6 kWh

Statistic      : System Peak Power
Start Time     : Mon Mar  2 18:05:00 2015
Peak Time      : Fri Sep 27 09:59:56 2019
Peak Reading   : 250 W

Statistic      : System Peak Amperage
Start Time     : Mon Mar  2 18:05:00 2015
Peak Time      : Fri Sep 27 09:59:56 2019
Peak Reading   : 1.3 A
[glorious]root: 

Running this on all the Dells give us this interesting Graph.

dells.png

Clearly a power draw of 6.5KA is wrong so if we exclude data points where the current draw is over 60A (pdus in the forum are rated at 32A)

fig_1_all_sensible_dells.png

So what do the GPU numbers look like, if we concentrate on the t630s

fig_2_T630s.png

In this case glorious is not actually a GPU server but the other nodes are showing a fair spread of peak current draw. if we concentrate on one specific GPU (1080 say)

fig_3_t630s_1080-ti.png

Again trying to work out what factors might affect the pwower consumption if we split the graph pabsed on power supplies we get these graphs

t630_1080-ti_1600W.png t630_1080-ti_1100W.png

-- IainRae - 05 Nov 2019

Topic revision: r6 - 09 Dec 2019 - 11:56:46 - IainRae
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies