Using Proxmox METRICS In Your Homelab
HTML-код
- Опубликовано: 5 авг 2024
- As an engineer, I LOVE looking at DATA! So today, I'm setting up my Proxmox system to push data to InfluxDB, where I can view it using Grafana. With this setup, I can keep track of how many resources all of my homelab services are using, which really helps when trying to size VMs, hosts, containers, etc.
For the installation commands, see the blog post:
www.apalrd.net/posts/2023/pve...
Feel free to chat with me more on my Discord server:
/ discord
If you'd like to support me, feel free to here: ko-fi.com/apalrd
Timestamps:
00:00 - Introduction
00:36 - Software Installation
02:07 - Setup InfluxDB
02:46 - Proxmox Bucket and Token
03:54 - Setup Grafana
05:38 - Grafana Proxmox Dashboard
#datascience #influxdb #homelab #grafana 
Наука
Best tutorials - thorough, no nonsense - just solid information.
Dude, you are slaying it with these video's! Really enjoying the way you make them.
thank you! I will definitely try to implement this in my home lab
Brilliant (as always!) - thank you!
Glad you enjoyed it!
Exactly what I wanted, thank you
Thanks a lot mate! I was struggling to set this up in my Proxmox cluster.
Really good tutorial. Much appreciated
Just wonderful, many thanks. I followed along and seem to have a monitoring CT all set up, so now it's a matter of waiting for some data to see if it actually works 🙂
Thank you for your video!
I love that proxmox dashboard, I customized to be more "LXC focused" since I do not have any VM, but a bunch of containers
Excellent tutorial
Neat! I'm gonna do it!
Thank you for taking the time and energy to create these videos. Your step-by-step is straightforward to follow. Would it be possible to have a video done on the Opnsense dashboard using Grafana
I actually have a setup I've used for awhile now using ntopng + influxdb + grafana, but it's not nearly as easy to setup as the Proxmox ones
@@apalrdsadventures Is that similar to My pfSense System Dashboard
Great video! Grafana is an excellent visual statistics system. Plenty of sample dashboards out there and can easily change them around. I'm also use the dashboard to monitor pfsense.
I'm using telegraf and ntopng for network monitoring and both are excellent, although I'm using OPNsense now.
I am sitting in front of my PC configuring the second Proxmox server with mirrored ZFS boot & five spindle ZFS raid. Soon to start building number 3 - yeah then all my proprietary NAS out the door. Dashy is up and running in a container inside a docker with an Arr stack and Pi-Hole & Traefik & SSL all working.
Thanks so much for all your wonderful videos - still lots of small issues but getting there.
Glad it's working well for you! I have some of those topics in my todo list as well
I've recently done a similar mass consolidation of my proprietary NAS servers as well and migrated all of that to a single, 36-bay, dual Xeon server.
Virtio-FS is AWESOME!
Well you made that easy! Thanks mate!
Glad it helped
@@apalrdsadventures next step...pfSense monitoring.
In my lab Instead of runnning influx and prometheus, because i need both, i chose to run victoriametrics. It can scrape prometheus targets and take in influx metrics and perform alerting using it's tools instead of relying on grafans alerting.
This is the route I went with originally. But then I discovered VictoriaMetrics, which uses several times less resources than InfluxDB, and is very flexible. I deployed prometheus-pve-exporter in a docker container alongside VM and VM scrapes those metrics. Grafana Dashboard #10347 is a good start to read those metrics, since VM can be set up as a Prometheus data source in Grafana. Now since VM can ingest Prometheus, Graphite, InfluxDB and other protocols, I have a one-stop metrics storage for my lab. I also find Prometheus much easier to query in Grafana vs Flux, and tons of dashboards are there using PromQL.
I did basically the same thing yesterday! But on a NixOS vm in proxmox.
I could not find the page of what statistics proxmox makes available, do you know if they also have context switches ? I've found this to be an important indicator of a busy virtualization server.
Hello. Great video.
In case of having Proxmox installed in different colocations, how would be this approach? Have a local instance per node and then replicating that to a central logging/metric server somehow? Or maybe avoiding the local step, and pushing out straight? Using Tailscale (or any other VPN) for instance I could get good connectivity in a secure way between sites.
Enable https in influx and it should be safe to expose on the internet. You can generate a unique api key for each cluster if you’re concerned about revoking them separately. The Proxmox host name is a field in the log, and the dashboard filters on it by default (hence the host drop-down).
Does this also have alerts? like sending an email if CPU goes over certain threshold.
Grafana does support alerts! It's not particularly excellent at it, but it's a feature that does work.
Really simplified it. I was thinking about setting this up but felt it was too complicated just for a dashboard. This video makes it easier.
Also wanted to ask something. I run my homelab on an old gaming pc and shut it down at night when I am done using it. I don't keep it running like an actual server 24x7 just shut it down for the night. I typically do it from mobile, open the Proxmox dashboard and then log in and find the shutdown button in the cluttered mobile webpage UI. Wanted to know if there was an easier way to do this. The Proxmox mobile app is only good for viewing statistics and wanted to know if I could shutdown from mobile in an easier way. I can develop an app for this if needed, but I need a network answer on how to request a Proxmox to shutdown shutdown over the network. I had the idea of running the poweroff command via ssh(with key based authentication setup) but that feels like an unsafe shutdown.
You have any ideas?
Regarding the shutdown via SSH: Proxmox automatically (gracefully) stops all VMs and containers on system shutdown. So the shutdown button and shutdown via command do the same thing. And don't worry about shutdown command "not doing anything" - because the server waits for all VMs and containers to have shut down, it might take longer than expected from a computer to finally go down :)
@@felixe2890 hm. I thought the shutdown command gave the SIGKILL to apps and would force close them. Good to know this. Thanks.
Well what hes says in this video must have some gravitas! Even all his monitors and the water glas is shaking when he speaks. 😁🤪😲
i would be happy if you had a go back to basics episode where you concentrated on the simple mechanics of scaling out using ha load balancer and a few cloned vms/nodes - this is pretty basic but could drive home what most smb needs - they just want to be able to scale out their web server and db - you could analyze what it takes to make a good load balancer - use opnsense or just nginx nodes, you could look at mysql/maria cluster and then you could look at how things really scale when adding a node, lastly you could also look at what the best simple netfs is - such a retro and basic look at the complex internals of clustering would be good content and drive home how great it is particularly for small biz and bigger homelabbers - lastly you could look at what networking speed and reverse proxying - how they may help or hinder performance and help you id the weak links and bottlenecks - it may end up being mostly isp bandwidth but this may be able to be ameliorated with a good proxy or cache tier system - thanks for all the content - it is generally pretty good and a breath of fresh air when compared to other youtube tech channels!
In general I run a wide variety of stuff at home, and none of the services are individually demanding enough to require multiple nodes / load balancing, so it's not something I need myself. I mostly just need to move stuff between my hosts as loads change.
I do however have an HAProxy video coming up, as a layer 4 reverse proxy. I'm using it for IPv4 -> IPv6 without doing TLS termination, but it can do *a lot* of things including a very capable load balancer for any TCP based protocols.
Where'd you get that t-shirt (has proxmox name on it)?
Amazing tool for home-lap and amazing presentation :). Any idea how to add CPU temps (Im-Sensors) with AMD ryzen? That would be really good.
Proxmox's metric collector is more focused on KVM / LXC, but you could install Telegraf (the generic InfluxDB logger) and it can do basic temp and lm-sensors
I was wondering the same thing. What about CPU temperatures?
Because, yes, those metrics are important, but before you could focus on VM and container resources, you should be aware of the CPU temperature and HDD temperatures. If some of those temperatures go to the sky high, then you won't have metrics at all.
So, probably a good suggestion for a new video? Monitor temperatures in proxmox and get an email alert if the temperature goes higher than expected.
I am sure it would be a good addition to this series of videos.
Thanks.
I managed to do it manually and customized the output, it wasnt straight forward for AMD CPUs but I got it done and it works really well. @@EduardoSantanaSeverino
it is possible to do this but with multiple remote servers?
The InfluxDB server can take data from as many Proxmox instances as you want (and in a cluster, Proxmox will automatically push from all of the nodes).
@@apalrdsadventures i'm asking this because i have a lot of proxmox hosts, i have email notification and all but its still a pain to go one by one checking everything. Does this notify by email like storage warning for example ?
Have you used Prometheus? Can you make an instructional video?
There's sorta a trio of time series logging systems. Prometheus, InfluxDB, and Graphite, and software has an almost cult-like allegiance to support a single one of them. Proxmox doesn't support Prometheus, so I use Influx.
For me, they fit an identical use case (storing and analyzing logs of numeric data), even if they are functionally implemented very differently (push-based vs pull-based).
Pros and cons of this kind of setup vs Netdata?
InfluxDB supports pushing arbitrary variables from software which has metrics to log, and Proxmox's metric collector is pulling data from qemu, the qemu agent on the VMs, storage usage, network, ... , so you don't need to install anything (other than the qemu guest agent) on the VMs to collect information.
Grafana supports a lot of data source backends and queries, so it can support fairly arbitrary time series data easily even if it isn't normal system performance data.
Alas, the Grafana template is no more (as of July 2024) and apparently a change in how connections are handled in Influx breaks this set up. Shoot!
I guess I'll have to make a new version soon then
@@apalrdsadventures I'd welcome that.
*promosm* 💪