tl;dr: Proxmox or bare metal? Containers yes/no? What is your use case?
I used a Dell R710 when I first started self-hosting, it ran ESXi with one VM for each service I wanted. Restarting the server required me to first shutdown each VM in order and then the whole host. When setting up a new service to host I had to create a new VM (allocate RAM, disk etc), install the OS (I ran Debian) and then follow instructions for how to setup the service. This mostly was not a problem but one software I never managed to get working was Apache Guacamole.
Nowadays I have a Dell Optiplex I salvaged for parts, got a new case and all my HDDs from my R710. Because it has much less RAM and an old Intel i5 (6th or 7th generation), I decided to get into Docker. With Docker containers you write your compose file and it will just work. No more need to dig through documentation for which version of a dependency to use, how to handle if two services on the same host need different versions (this was part of the reason for one VM/service). With Docker, I can try out a software in seconds and have it configured to my liking in minutes.
Today I have NixOS (bare metal) and it comes with Podman which uses systemd. Hence restarting my OS (albeit not that often, almost never unless I mess up my config) is a no-brainer because Podman via systemd will manage everything. Adding, stopping or removing containers in general is easy. I have a script running as a service which will stop a container, create a BTRFS subvolume snapshot, start the service again and start borg backup to backup from the snapshot.
For me using Proxmox would just an extra layer of complexity I don’t need. I only have one server and I am the only user.
Questions:
- Do you use Proxmox instead of a bare metal installation?
- Do you use containers or do you install manually?
- What is you use case that requires your setup the way it is?
You must log in or # to comment.
Security is a big reason to use VMs. Containers share the kernel with the host. That means that any of the kernel vulnerabilities from this year (like DirtyFrag and CopyFail) could have been used to compromise your host. Once the host is compromised, the only way to be safe is to literally buy a new machine. Seriously. Viruses can bury deep and even infect the motherboard firmware to persist indefinitely.
Kernel vulnerabilities are frequent enough that I find VMs worth it. I still use containers in my VMs though.
Do you have any sources on motherboard firmware viruses? I’m curious to know more about them
Just look up firmware rootkits and UEFI rootkits. For example: https://arstechnica.com/information-technology/2022/07/researchers-unpack-unkillable-uefi-rootkit-that-survives-os-reinstalls/
Having to manually start and stop your VMs is very atypical, proxmox can absolutely handle that itself
I run podman in proxmox vms
I use both. The main benefit of proxmox is having VM snapshots/daily backups, if you mess up, just restore the whole thing. You also have stricter isolation, e.g. put public facing containers in one VM and local only stuff in another.
Also has high availability if you have multiple nodes but that can be achieved with container orchestration as well.
Couldnt you just make daily snapshots on baremetal? Like most configs will support a ZFS pool.
All my stuff is just configs anyway, so my backup is a Git commit. Lighter than a full snapshot. My server can be restarted with like 50mbs of information.
Docker images are images so disposable. All my files and databases are on my NAS which is a ZFS pool mirrored and with a weekly backup to a secondary NAS also mirrored. Plus I have an offsite cloud backup of everything. There isnt really anything on my server that matters other than some configs, scripts and ymls. Which by commiting to my Gitea, are all on my NAS with the rest of my data.
I still feel like Proxmox just invites you to tinker and fiddle with stuff that should be set and forget. Or overcomplexify your setup with too many moving parts.
For public stuff I just use a VPS. Better internet connection than at home, high availability and DDOS protection. Otherwise, an LXC with good firewall rules and a well configured reverse proxy should be sufficient.
Sure, you can probably build something similar with zfs snapshots, but it’ll likely not be as convenient. Have a VM? Create daily snapshots. Break something and can’t figure it out? Just reset to the last backup to a complete working state.
Personally, I have the containers’ mounts in the VMs (ergo on the nodes’ disks) and just using the NAS for the backups. Even if both Proxmox nodes catch fire I can just import the snapshot to a new one and have a working VM in the same state again. That’s a whole different level from re-cloning all the compose stacks, mount NFS shares etc. It’s the ease of use.
That’s apples and oranges.
Anyway, both podman and docker are extremely insecure. Safer to download signed ISOs to run in proxmox
trusting an oci repository is the same as trusting a distro repository, if its a dodgey unknown you got from god knows where and you cant verify, dont use it
Nope. You don’t need to trust a distro Repo’s publishing infrastructure because the packages are cryptographically signed.
Whereas the oci repo had a ton of vectors for delivering maliciously modified code because they include image layers that are totally unsigned.
oci repositories support signing artifacts, container trust policies are a thing. bootc repos are both signed and are distro repos, ostree Fedora and ublue are using it
Just because quay.io or hub.docker can be accessed without signatures or trust doesn’t mean there aren’t repos out there using it.
Proxmox exclusively with unprivileged LXCs, which themselves host other docker containers/stacks managed with Dockhand (a more modern alternative to both Portainer and Dockge). I have a very tiny low power machine so I have to be kind with my resources, therefore no VMs. But LXCs are simpler anyway. I can easily pass devices like
/dev/tunfor tailscale and i forget the name of the iGPU for accelerated workloads (for Immich, Nextcloud).Reasoning: with containers it’s easy to make mistakes without dire consequences (just
docker compose down -vand start over). The LXCs are easy to back up and restore. They hold internal running state of whatever docker containers I run in them.Pro tip: don’t map host paths using the UI (
mp0etc), uselxc.mountinstead. This will let you continue to have snapshots of your LXCs whereas the other approach makes your LXC incompatible with snapshots.Good luck!
Docker has its uses, but it has gone to far.
I already have a webserver with Apache, mariadb and everything. Why does your PHP based website only come as a docker solution? Fuck you.
When you go all in with docker, it quickly becomes a mess with rando NICs and containers named “random 32 character string” and dependencies up the wazoo. Ending up running ancient packages because the developer refuses to apt upgrade their shit.
Proxmox or any other hypervisor platform gives you control. Whereas you have to take it for docker and youre still at the mercy or the developers to keep your shit secure.
all the problems you expressed with docker aren’t real problems.
I think you may not like it because you don’t know how to use it.
don’t like how often a maintainer updates their images? build your own.
don’t like having multiple bridge interfaces on your host? configure and manage networks within docker and assign them to your containers.
don’t like having dozens of containers with random names? use docker compose. bonus, you can set up networking with it even easier.
What? Outdated junk running in containers because the maintainer didnt update their shit is not a real problem?
Seems that it is perhaps you who doesnt know how to use it.
Years ago I’d have one PHP app that required version X of PHP and another that required version Y and my distro often only had one of them (or you couldn’t install both simultaneously), so I’d have to either compile myself (or later use a 3rd party repo). All solvable of course, but then throw in MySQL version requirements and PHP extensions and it’s just extra crap that containers just take care of for you. I’ve had this kind of inter app requirements conflicts with stuff other than PHP, but I’ve had by far more of it with PHP. For a while I used FreeBSD jails to help, but jails was kind of a pain. I enthusiastically embraced docker for PHP stuff early on.
A benefit for application developers, particularly PHP is there’s a lot of differences between distros and how people install and configure PHP. Even like which PHP extensions are available and their settings. Other languages things tend to be more consistent at the system level and customizations are app are usually application level (contrast with PHP extensions and system level configurations). It really helps reduce support issues due to distro/user differences if a developer just provides a working dockerfile as a reference implementation and in my experience this is far more likely with PHP than other languages. I realize this is mostly applicable outside of PHP, but I actually personally see more benefit with PHP based on my past experiences.
Why not have the option for both?
I dont see how this is a defense for jamming everything in to a container.
Ending up running ancient packages because the developer refuses to apt upgrade their shit.
Tell me you never really had to deal with dependency hell without telling me. That’s literally the main problem docker solves.
I have had to deal with dependency hells plenty, i am a Debian user after all.
But with that argument you’re now just running outdated shit in a container. Not a great pro-docker argument tbh.
because sometimes i need a vm (home assisstant has more features thsn home assisstant container), sometimes a container (i mostly use lxc).
its not really an extra lauer because proxmox is an os.you use nixos i use proxmox. everything is out of the box. proxmox has comminntiu scripts to run everythkng and set them up. so want to install radarr? bash -c “$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/radarr.sh)”
sure podman has similar setup but i also need vms and they are all mamaged in one place
so want to install radarr? Paste this totally not sketchy bash script into your terminal and execute it as root!!
@oha @terabyterex @selfhosted
You should checked that script first…ProxmoxVE helper scripts have been around for quite a while and are pretty easy to read, as well as being pretty well supported in the community. You can just go to the GitHub page and read them. I’ve used some of their scripts before by downloading them, customizing the scripts to my needs, and hosting it on a local web server inside my network. Now I can run one command and I’ll have a new debian container on my host with ssh keys and everything.
While I agree with you that they’re well seasoned and established I disagree a lot with them being easy to read. They have a lot of sub-scripts that they include willy-nilly over multiple levels.
Another thing I dislike about them is basically the same as with docker, that they all include everything in it. Like every container wants his own mariadb/postgres/whatever. It should be at least an option to use existing containers, I don’t need to run 5 psql servers… (I know that it had advantages too having everything separate but ehh). And then switching the software back to using the existing psql is more work then just not using the scripts at all…
I used to run bare metal monolith container host and it worked pretty well but eventually network configurations started tripping over each other and causing real issues (complicated by me running rootless Podman and wanting to run K3s cluster etc). I wanted to run Proxmox native containers but it turns out they run full root therefore breakout in one container means full host access, so now I run VMs as container hosts.
@captcha_incorrect Currently I’m running my personal services similar to you using Podman Quadlets and I really like it.
I have also a testing installation of Proxmox on another system, but I’m also not so clear, if it would be a good way for my purposes.I have a mix…
I have a home built NAS running on Arch with local installation of Immich (ie not a container)
I also have a low power, passively cooed box with Proxmox installed to run VMs for Home Assistant, ansible, uptime kuma, smokeping, etc.
I only intended to run Proxmox to test it out (my Home Assistant was running on a Pi3 and getting too slow for Voice), as I’d been using ESXi for years at work.
I now want to migrate to Arch with Incus and move those VMs across, but as I currently only have the 1 host and hardware’s expensive, it’ll stay that way for a while…
(Bonus: I have a few RasPi Zeros scattered around the house too running various things bare metal)
I like Proxmox for it’s first-class treatment of ZFS. You can easily run Docker/Podman on top of it or in a VM. Proxmox has some other advantages like replication and clustering, but I’d say you don’t need that in a home setup (at least I don’t).
If I had to redo my setup today, I’d probably give TrueNAS SCALE a chance though.
I use Proxmox with a VM for each service I provide and I still use docker compose.
This way I have nice VM backups (with Proxmox BS) + dockerized services deployed via GitLab CI pipelines.
Check out the krun runtime for podman, if you want stronger container isolation from the host. It creates a small VM for each container. Gpu passthrough on linux is limited though.
I use Proxmox as a VDI server. I understand that this isn’t homelab territory, but selfhosted VDIs have proven exponentially more reliable and easier to manage than cloud based ones (after the initial PitA setup). Flawless copy/paste, screen resize, and most importantly, file transfers. When you connect to 12 different clients, each with a different set of security requirements, system hardening, monitoring, and VPN access, being able to console amd fully interact with a sandboxed desktop becomes priceless.
