I’m wondering what folks do to optimise the power efficiency of their Linux servers. I’ve never really got to the bottom of what is the best way to do this and with the current energy crisis its a pertinent topic.
I’m talking about home servers, so the availability requirements are not the same as in a corporate environment. There might be vast chunks of time during the day or night when they sit idle, and home users are more tolerant of a lag when accessing resources if it means lower energy bills.
Specifically I’ve been thinking about:
- allowing lower power states when idle
- spinning-down hdd’s when they’re not in use
- MAYBE letting machines sleep/hibernate
- setting schedules of times where you know demand will be low/zero and efficiency can be managed aggressively
- any other quick wins I’ve missed
It would be amazing if there was one tool or one guide that helps with all of that but thats never the case, is it 😅
Thoughts?
As someone else here mentioned, isknf powertop can significantly reduce consumption at idle, which helps.
In the world of servers, though, unless a device is only accessed once a day, it’s tough to get the HDD to spin down.
I had to start over anyways, so I choose SSD only. Pricier of course, but I don’t need a terrible lot of space anyways.
When I still had HDDs, they were usually used once a day for backups, and I spun them down after that.
Optimizing power profiles and C states makes a little difference, but planning with efficient hardware from the beginning is the most important thing. Don’t use your old gaming PC if you care for power efficiency.
Letting go of older inefficient hardware is no 1.
Why run a 200w server when a 30w mini PC or 10w pi can run a dozen containers
The mini PCs draw up to 30W, mine runs at an 8W average
I have never measured my pi consumption but ilo says my DL380 Gen8 idles at between 120w to 180w
My Pi5 tops out at 15W.
My old Pi4 with SSD averaged around 7W, so only 1W lower than my mini PC, but performance and usability of my mini PC is far greater (and it comes with 1tb NVME and 16GB ram). I generally advice against using most SBCs these days unless you specifically need the pin I/O for something.
The ipmi on my supermicro says it’s running 497w.
But that’s what 38 HDDs will do to ya. I used to spin them down when not in use (it’s mostly a Plex server) but it’s in my garage and over the winter I had some issues with the older HDDs not wanting to spin up in freezing temps. I’ll probably move it back to spinning them down over the summer.
Ooof, your power bill.
Cheaper than netflix, paramount plus, hulu, sling, fubo, Disney+, etc
Maybe not when talking about the hdd cost, though. It’s over 300tb of usable space. But at least a good amount of what I use has been wasted I’ve gotten for free.
Mini PCs are even less usually, mine are around 2W idle which is less than my Pi! (i3-7100u CPUs)
Sadly this. I have a graveyard of nice server boards that I got cheap before realizing how power hungry they are.
For CPUs basically anything older than gen 6 intel is too power hungry (although be careful with Xeon and xeon derived cpus, that are sometimes older gens rebadged as gen 6).
This Jeff Geerling video from a couple of weeks back was an eye-opener.
Just because you might be able to find a cheap G5 Xserve server that runs at 200W when idle, doesn’t mean you should.
Do you have any idea what your hardware is actually pulling from the outlet? Maybe it’s not that bad after all?
Mine is pulling around 55W from the wall in its “normal” state. Meaning two 3.5" HDDs spun up, and a bunch of light services running. Which is squarely in “not great, not terrible” territory.
Apart from flipping the power saver switch on the mainboard I haven’t done anything to save power. I haven’t checked if that’s doing anything either. It’s a 3rd gen core i5 iirc, which isn’t great at idle power consumption, so maybe that switch is doing something…
I also haven’t had any luck with getting the drives to spin down reliably anyways, and afaik it’s better for them to just stay spinning so I haven’t bothered much to change that.
Just as a reference, my NUC with 40+ containers, runs at around 3-4 W, not counting the 2,5" 5400 rpm HDD attached.
Logging power use by my server was one of the motivators to add homeassistant. That also showed me that specific containers use a (relative) ton of background power. Immich and authentik each raised power consumption by 2-3 watts, so I leave them down unless I have specific need.
I use powetop on laptops to recommend config optimizations, it could run on a server too.
hdparm can configure HDDs to powerdown, but I’ve never had any success using it on my router.
In theory I think You could use WoL and have your router wake a device before sending traffic but I haven’t seen any guides for doing this so maybe I’m missing something.
WoL works, but your server will take some time to come back online, but the router probably wont be able to buffer the traffic for that long, and a tcp connection would likely timeout before then anyway. You usually want to send the WoL magic packet, wait for the server to come back online, and only then start sending traffic.
What’s the timeout on a TCP connect?
Could you not wake from suspend in that window? Maybe even a full hibernate?
The TCP connection time out on linux defaults to a bit over two minutes, although individual client programs can use different values, and I’m assuming Windows is similar. Honestly, I was thinking about the time to boot a server, but if you’re just suspending you’d almost certainly be ok, albiet with a slight power draw even when the machine is not in use. Hibernating might also be ok as long as your hardware gets through its POST quickly.
power top is a helpful program to monitor power consumption on some hardware, alsk funny name
I split my loads (gigity) between the power hungry NAS and a passively cooled low power Proxmox host.
For me, most 24/7 activities are low CPU - like Home Assistant, so it needs to be there, but it doesn’t need to do anything.
Other VMs are ansible, uptime kuma, smokeping, etc… the most they use is RAM
Then the (relatively) more power hungry NAS powers up 3 times a day to syncthing everything, maybe upload a backup, and if no-one’s using Immich, etc. then it’ll power back off again.
The only other thing I have yet to downsize is my pfSense box (still a low powered device, but has fans…) and a Raspberry Pi I use for my Zigbee network.
Just yesterday I found this video talking about this very topic: https://www.youtube.com/watch?v=MucGkPUMjNo
Downsizing. If you don’t need to run or keep stuff, then you don’t need so many servers and storage. You may run stuff on cheap mini-pcs.
Sleep: In my experience sleep sucks, I’ve spent long hours planning around sleep in homelab, like: when do I restart, do updates, when do I upload backups. I have Pi and at some point I realised some actions on Pi need my sleeping NAS… so I dropped all the sleep and now it works 24/7.
When I built my NAS I intentionally bought the latest gen cpu, but kept it in to the 65W series with a GPU chip onboard. It’s an AMD Ryzen 5 7600 6-Core @ 3800 MHz. My coral usb does frigate and the integrated graphics chip does jellyfin just fine. I started with ssds, but half of them burned out pretty quick, so I replaced them with spinning rust. But, as-is it can run for an hour on my desktop grade UPS before it shuts down. My proxmox cluster is old laptops that mount an NFS drive from my NAS. So, yes, I took power efficiency into account.
Added solar and a battery. I don’t worry too much about the power use as a result.
Dell Thin Client type machine with a 2TB SSD of the essentials. Backs up once a week to my gaming desktop. Has enough horsepower for Homeassistant, Jellyfin, adblocking. Enough fast USB for two 2.5GBE. Upgraded to 16GB RAM before the crisis, but could make due with 8GB. Cheaper than a raspberry pi for only a few extra watts.
Try testing TLP in battery mode even if you’re not using a laptop. You can configure all kinds of things to your liking with it.
I tried it out a few years ago and none of my server apps showed any noticeable decrease in performance with it running, but my power monitoring plug did show a reduction in power consumption. I ended up leaving it enabled all the time.
Mine shuts down while im asleep and while im at work. I only use it in the afternoon.
My video server runs from 6 pm to 1 am. No need to run always.
I do this via cron. Shuts down the server while I’m asleep. I haven’t gotten around to it, but I want to issue a WOL from my standalone pFsense firewall to the server, and have it power up when I get out of bed.
Unfortunately, wol does not yet work for me.
I use rtcwake
