Hi all. I was curious about some of the pros and cons of using Proxmox in a home lab set up. It seems like in most home lab setups it’s overkill. But I feel like there may be something I’m missing. Let’s say I run my home lab on two or three different SBCs. Main server is an x86 i5 machine with 16gigs memory and the others are arm devices with 8 gigs memory. Ample space on all. Wouldn’t Proxmox be overkill here and eat up more system resources than just running base Ubuntu, Debian or other server distro on them all and either running the services needed from binary or docker? Seems like the extra memory needed to run the Proxmox software and then the containers would just kill available memory or CPU availability. Am I wrong in thinking that Proxmox is better suited for when you have a machine with 32gigs or more of memory and some sort of base line powerful cpu?
Proxmox is based on kvm/qemu, and is very resource conservative. There is virtually no impact on performance due to the hypervisor, even on older processors. Scheduling on the cpu and hypervisor makes running multiple VMs at the same time trivial as well. RAM and I/O bandwidth are the two things that can affect performance. Running out of RAM due to too many VMs will grind you to a halt, but so would running too many applications or containers on bare metal. Running everything off of one spinning sata disk will make it impossible, but again, same downfall on bare metal.
Those minimal impacts to performance are a minor nuisance compared to the ability to run experiments and learn on sandboxed VMs. Now that TrueNAS has better virtualization support, it has caught my eye as a better homelab solution, but I will always have a proxmox server running somewhere in my stack just due to the versatility it gives me.
If you want to get things working then never “tinker” with things, maybe it’s not worth it. But if you want to learn and be able to try new things it is really helpful. Having a new VM not breaking existing VMs reduces risk when trying something new.
It’s the same reason I like running things in Docker; you can just wake up and read about something while enjoying your morning shit, then switch the computer on and try and boot it before that thing you’re meant to be doing. If you can’t do it you can just delete it and try again later.
I started Self Hosting with Proxmox 4 months ago and so far my only real snafu has been mapping drives directly to Proxmox with Fstab. If you’re gonna do it, add “nofail” FFS.
I pass my drives through to my NAS VM to handle rather than Proxmox because it’s easier to fix my NAS if it fucks up using Proxmox, than to try and fix a none-booting Proxmox.
Anyway now I’m at a point of stability and dim sat thinking about redoing it bare bones, but I love tinkering so I’m sure this is just the plateau before I discover something new to play with, so I’m keeping Proxmox
I use it on reclaimed hardware … Works great for me. Has all the features you’d want for a home lab, and I run a few production hosts there as well
I’m using a commercial desktop with an i5 Sandy bridge. I maxed out to 32Gb of ram only because I’m running trueNAS, debian with containers, and home assistant. Most RAM goes to trueNAS and trueNAS doesn’t accurately report ram. For CPU, mostly just task limited but I don’t really think thats a proxmox issue. Obviously it’s not going to support an enterprise or even small business but it works for what I need of less than 4 users on my budget.
Proxmox doesn’t really ask for much but I probably would recommend docker for your arm devices.
For me, pros are:
- Fun to learn something new
- Easy to test different systems. For example, I can play with different router or NAS software without having a separate computer around.
- I’ve been able to create different “computers” that serve different needs and require different levels of security.
- Currently, a cluster is probably overkill, it was a fun experiment.
Cons
- Updating all the different systems can be a pain. I could probably automate it, but I haven’t made the time to learn it yet.
- As a beginner, I’m throwing a bunch of parts together and hoping it will work. I should probably be more strategic in my implementation, but I don’t know what to prioritize. I’m sure I’ll have to start over in the future.
- With the previous point, the storage setup doesn’t seem very intuitive. I probably need to set up that better.
- I haven’t quite figured out backups yet. My VM backups all seem too big. I need to figure that out and automate it.
Hope this is helpful.
a simple cron job pointing to an update.sh with an apt update && apt upgrade -y does the trick.
i wouldnt recommend you to completely automate it though
debian has unattended-updates by default and generally takes care of itself
You need to understand what Proxmox gives you, which primarily is ability to run/manage/backup/etc VMs easily. If you don’t care about that, don’t use it. I have a fairly well spec’d desktop I use for homelab and I use proxmox because I often do experiments in VMs where snapshots and ability to jump to snapshots is essential. So is being able to spin up a new VM with new OS (like Windows) for example to do some testing. You can still do VMs without proxmox, but proxmox does make it a lot easier for living with daily.
You need to understand what Proxmox gives you, which primarily is ability to run/manage/backup/etc VMs easily
Yeah and after understanding what it gives you then you move to Incus because while it might be a bit harder to setup it delivers around 80% of what Proxmox does without the overhead, mangled kernel and licensing issues.
https://cockpit-project.org/ also does VMs and can work for people without cluster needs.
Incus is way easier to work with than Proxmox, and it sits on your OS of choice instead of being the OS you must use. For home use it’s way easier to use with the web ui, it even has clustering if you want to go hard.
So you can install Incus when you want a VM/LXC container and not have to commit to a VM/LXC container OS from the start.
Also Proxmox free just had a bad update that björked some stuff if you updated when it was live. Proxmox free is rolling and apparently lacks basic sanity checks for updates.
i played around with Incus yesterday on one of my VPSs that I really don’t care about. I did find it really interesting. But im just wondering if its still a bit too much for what i use my home lab for (running local services like jellyfin, gitea, etc.). I would prefer to containerize all of those, but unless im misunderstanding something somewhere (and I probably am), running Incus to then run another instances of ubuntu 22.04 (or whatever) so i can set up Jellyfin or Gitea inside of that seems like a bit of overkill. However, as im trying to get nextcloud set up and running, having it exposed to the internet would mean spousal factor would go way up. Honestly they are about to kill me for using pihole, so having them have to turn on tailscale to connect to nextcloud, well sometimes it feels like its asking too much. So this is where running something in an isolated container would make me feel a bit more at ease. ah if only my spouse would just learn to turn on tailscale when they need it, but i don’t see that happening any time soon.
I do use it to hold internet-exposed things in LXC containers to sidestep having to figure out how to not run things as Docker root.
You do not need it for everything, but since it’s not an OS that makes it your everything, that’s ok! Run Docker containers as you need, put internet-exposed ones in an LXC container, put home assistant in a VM because it’s special.
I remember updating (maybe a year ago now) and it making all my containers unaccessable.
Incus or Proxmox (e.g., should I shift to Incus LTS or something?)
If incus works for yoy, use it. Proxmox locks you out of the option to choose your base server distros.
Ah, I was wondering which one you updated and it made your containers inaccessible!
Sorry, misunderstood. Proxmox Free broke my containers on updating a while ago.
Now I use Docker-style application containerizing, but I think LXC (the base technology powering Incus/LXD) is useful in a number of situations and perfectly viable for use. I think Incus-containerized applications are easier to upgrade individually (like software updates of your apps, no need to recreate the container image) and gives a closer to native experience of managing. You do lose out on automated deployment of applications from widely available image sources like docker.io, but the convenience-loss is minimal.
Good to know Proxmox’s bad updates are more pervasive than the latest bad update.
I have been able to install Docker in the LXC containers and pull images in with the normal commands. I do that container-in-container to get effectively rootless docker containers for stuff that I couldn’t figure out how to run rootless. So you don’t even lose out on docker if you’re determined! And as you said incus goes on any OS, you can docker just fine on the base OS of your choice and use incus for specific things!
I used Proxmox for awhile, then went to Unraid. I learned a lot using Proxmox but for ease of homelabbing, it’s tough to beat Unraid. It depends on what you’re wanting from your lab.
I use Proxmox/virtualisation because I want to be able to run services within their own OS. I’ve got a VM dedicated to docker both at home and in my colocation, since a lot of services I’m happy to just chuck on there, but there’s others with more complex setups, and other services/systems that just running them in docker isn’t an option.
I run it on a 4GB Fujitsu Futro S920! 😆 All the RAM seems to be used by 3 VMs. Some SWAP is been used, ok, but the Proxmox overhead doesn’t seem that much.
C’mon just move to Incus: https://lemmy.world/comment/10896868 :P
No way! For just 1 reason: I will have to learn another new thing and replace it in about 6 servers. I value my time and for now Proxmox is fine.
P.s. Incus seems nice though! NO, stop tempting me!!! I’m already in the rabbit hole with a gazilion of self hosted services and dozens piling up in the to do list 🙈🙈🙈
Well, I understand your POV… but real software freedom instead of messages asking you to buy a license and a questionable kernel is always a good choice :P
Does Incus support things like Kernel Samepage Merging? How does it handle Windows VMs? Does the WebUI give a nice and easy novnc window that just works?
Yes
ksmtunedis your friend. For VMs it can be managed / enabled like any other Linux Kernel + QEMU/KVM running with KSM enabled.On LXC containers it may be a bit harder as it depends a LOT, best results if you’re using systemd both the host and containers. It may work out all out of the box or you’ll have to resort to
ksm_wrapperin both the Incus executable and the stuff running inside your containers.Don’t forget that:
KSM only operates on those areas of address space which an application has advised to be likely candidates for merging, by using the madvise(2) system call: int madvise(addr, length, MADV_MERGEABLE). https://www.kernel.org/doc/Documentation/vm/ksm.txt
How does it handle Windows VMs
As one would except from QEMU… https://blog.simos.info/how-to-run-a-windows-virtual-machine-on-incus-on-linux/
Does the WebUI give a nice and easy novnc window
Yes it works fine. https://youtu.be/wqEH_d8LC1k?feature=shared&t=508
Just wait until you get a few machines. You can live transfer things and dynamically allocate resources.
Pros:
- you run a home lab
Cons:
- you run a home lab
ahhh thanks, i needed that laugh. too true…
I don’t prefer proxmox, but I will say that when you have even a machine with 8 or 16gb RAM, virtualizing a workload on it just makes sense. At that point the cost is 12% resources, and the benefits IMHO farrr outweight that.
Virtualization has a 1-2% performance penalty
I have tried a couple of Proxmox clusters, one with overkill specs and one with little Mini PCs. Proxmox does eat up a fair amount of memory, but I have used it with Ceph for live migrations. Its really useful to me to be able to power off a machine, work on it, then bring it back up, and have no interruptions in my services. That said, my Mini PCs always seemed to be hurting for RAM. So that’s my pros and cons.
Proxmox doesn’t have a lot of overhead. However, Ceph is a beast and requires very power hardware with at least a dedicated 10g network between hosts for transfers. You also need 5 or more nodes for it to be reliable. I wouldn’t recommend Ceph as there isn’t a lot of point to it. You can get a similar functionality with NFS or ZFS replication.
I have it working with LaCP’d 4gb networking for the transfers. Five nodes. I agree though, It’s a beast on RAM.
I’m currently running proxmox on a 32 gig server running a ryzen 5600 G, it’s going fine the containers don’t actually use all that much RAM and personally I’m actually seeing a better benchmarks than I did when I just ran as a Bare Bones Ubuntu server, my biggest issue has actually been a larger IO strain than anything, because it’s a lot more IO heavy now since everything’s containerized. I think I easily could run it with a lower amount of ram I would just have to turn off some of the more RAM intensive items
As for if I regret changing, no way Jose, I absolutely love the ability of having everything containerized because I can set things up how I want it when I want it and if I end up screwing something up configuration wise or decide that I no longer need that service I can just nuke the container without having to remember well what did I install on this program so I can remove it and do other programs need this dependency to work. Plus while I haven’t tinkered as much in this area, you can hard set what resources you want a lot to each instance, so if you have a program like say a pi hole that you know is never going to use x amount of resources to be able to appropriately work you can restrict what it can do so if something does go wrong with it it doesn’t use all of your system resources
The biggest con out of it is probably having to figure out how to do the networking side because every container is going to have a different IP address, I found using a web dashboard is my friend because I can have heimdel tell me where all my services are and I just have to click the icon to bring me to the right IP address, it took a lot of work to figure out how it’s operational and how to get it working, but the benefits I’ve gotten of having it is amazing. Just make sure you have a spare disk to temporarily clone partitions to because it’s extremly difficult to use existing disks in the machine. I’ve been slowly going one at a time copying it over to an external drive nuking the and then reinitializing the disc as part of the proxmox lvm and then copying the data back over onto their appropriate image file.
You need Proxmox
Seriously though it is nice to have
