Thanks for this thorough overview on how to have visibility on my proxmox server, right after i did this i discovered 8.1 includes some native GUI for notifications. Might be worth a new video lol
Just a quick note, the zpool clear command is used to clear the error counts on drives back to zero, it does not clear data off the drive. zfs was asking you to do this so you could see if any new errors occurred on the drive you pulled and reinserted. so you can check if it actually needs to be replaced because errors are increasing or if all is OK and can be left alone. zpool status with high error counts can often be a sign of a drive in a prefail state that SMART has not yet picked upon, but zfs has due to all the checksums zfs uses. hope that helps and saves on future heart attacks 😊
Same story here. Didn't have alerts setup and randomly decided to check the status of my drives and saw that I was one drive away from losing all my data. Thanks for the great tutorial!
1. I love all these recommendations/ alternatives in the comments section. That's one of favorite part about home-labbing! 2. You could also use n8n. You can send the data via curl as a response to a listening webhook on the n8n server. I'm not sure if you could pull the error message from the log (i watched the video but came back for the microcenter link lol). N8n could then send out the alert via any of their options like slack, discord, mattermost, etc.
What perfect timing for this guide! The "old_age" flag on several of the drives in my oldest ZFS array just got triggered a few weeks ago so it will be nice to have notification as soon as one, or more, of them throw a more urgent SMART flag. Thanks for another great guide!
If I had a homelab with as many devices as you have, I would monitor the environment with Icinga, Nagios or CheckMK and use their notifications. Much simpler and easier to maintain than configuring alerts on each device.
you can send metrics to an influxdb or graphite endpoint too but the email alerts are nice and be harder to setup and tune with other monitoring systems
@@justwhyamerica I've build influxdb+Grafana to monitor Proxmox. But, error like that (zfs fail, disk fail or backup fail), can it do that? Where I can improve? Any reference would be appreciated, thank you.
absolutely perfect timing! this is exactly what I'll be spending my last day in the office before the holidays doing. it always irked me that there wasn't more notification control in Prox. I have been relying on the ceph dashboard and cluster health to know when a disk goes down and it is far from ideal. And my Zabbix dashboard is not fully configured yet, so this is a perfect fix in the meantime.
2 года назад
Hey Tim, recently found your channel and this is my first comment. Great fucking work! As a SysAdmin, this video is high quality! All the world's success to you in 2023.
Nice way to get rid of some extra layers of complexity. Some labor involved but for us homelabbers it’s cool - we are eager to spend hours to save money. A very nice video as usually.
Leaving this comment for the algorithm and to reiterate my appreciation for all that you give to the community. This worked flawlessly for me. I have 4 proxmox nodes, so I ran most of the commands with adhoc ansible commands against the nodes. Super happy with having this working now. Before I was sporadically getting emails from proxmox, but gmail would often block them. This is a huge improvement. Thanks, again.
18:50 - Something you may want to mention is the difference between a RAID card and HBA. Most RAID cards won't give you SMART data from the drives even in JBOD mode so that may be a reason SMART may not be working.
Thanks a lot Tim for sharing your knowledge - works like a charm! I was able to setup email notifications on 3 PVE nodes (one Cluster) and my two PBS Servers. Not sure if I am really looking forward to receiving emails now ;-)
I kinda did something similar, except i make a postfix mail relay vm so all my things on the subnet could relay without auth to it, and from there it could forward out to external mailbox. Just makes mail setup on new/additional systems easier/faster.
@TechnoTim Nice how to. I was able to fast forward (thanks for section labels) to the alerts setup. if you or anyone else is interested, I have a mailserver setup that uses mailjet, so 2 factor isnt required, and the emails are not marked as spam thanks to mailjet. All I had to do was configure proxmox postfix as a relay to my mailserver then turn on notifications, alerts, etc So a mailserver is an option especially if you have multiple systems you would like to have alerts setup and not having to setup each one like the how to provided. Setup one mailserver, then configure postfix on the others as a relay. Have a great day and thanks for sharing!
I like this, I've had alerts going since the start but I've had to test them several times because I never get any so I always get worried it's stopped working :D
Question: you chmod the sasl_passwd file to 600, but not the sasl_passwd.db. And when you catted the file, you can see the password is still readable there. Presumably you would want to chmod it too. But does postfix run as root or does it run as its own user that you'd have to chmod the file to?
This video was very interesting. I had the exact same issue recently, a drive bricked itself and I noticed it later. I chose a different route to notifications: a selfhosted ntfy instance. I also ended up touching Zed, but I had to write a script to send the POST messages since ntfy is not one of the preconfigured services. I agree that Proxmox should include better alert management tools, although webhooks would not be my first choice. My ideal system would be similar to Zed: execute command X when event Y is registered. This could be adapted to any notification method and even for automating complex tasks like starting a failover in case of degraded zpool. And finally, if you want to test Zed it is a lot safer to start a scrub or a resilver manually instead of removing a drive, but you probably already knew that.
Oh, probably I missed something, but looks like you have auto dark theme reinstall on each apt install as trigger? Do you have this explained somewhere? 😄
Isn't there a way to forward an email to an webhook? Could be actually made with a pretty simple script. Edit: Forgot to say: Thank you for the tip!!! Something i absolute need either and didn't even thought about and anyway didnt knew how to setup!!! So yes! Amazing Video! Thanks for making such simple but very specific video's! While other RUclipsrs make another 100 videos about setting up failover dual wan with pf or opnsense, where 99% of home peoples don't even have an second isp 😂 So thanks for making really good videos and not boring crap that everyone knows! Thing is, that there are a lot of your videos that are pretty amazing, or at least were at the time when you published them! You know after some time everyone knows something, like "authelia" for example, where 1 year later your video is useless, but at the time you published and not many knew about it, it was amazing.... So however, great Videos man! I mean i know a lot either, but some ideas i got from you and wanted to say thanks once! Cheers 😂👌
Great video Tim. Out of interest (and this may already have been asked below), couldn't you use the likes of Promtail / Grafana Loki to aggregate the logged activities from Proxmox / smartctl (etc.) and push an alert via email / webhook from there?
Awesome video. Followed your steps and got this setup on my PVE box. Everything is working but when I checked my drives as you pointed out at 18:20 it shows "SMART support is: Unavailable - device lacks smart capabilities". This happens on sda1 which is a SATA SSD where the OS is installed and sdb1 & sdc1 which are SAS HDDs used for backups. nvme0n1 does show as being supported; this is where all my VMs are installed.
One additional suggestion would be to check and, if necessary, clear the postfix queue before activating an smtp connection. My email account was besieged by thousands of cron errors that had been building up in the proxmox email queue. 'postqueue -p' will show if anything is in the queue and 'postsuper -d ALL' will purge old emails. Of course you should troubleshoot the source of the error messages but it's easier to do that once your email account is not on fire.
I set up mailjet through my domain on CloudFlare. Way better then using a Google email. Other fun things to do, if you have home assistant, there is a smpt2mqtt unofficial add-on. Then you can do whatever you want with notification
What I find amusing is proxmox has the frontend UI for email alerts but not the backend out of the box, a bit half assed if you ask me. It should be ready to go out of the box.
is it possible to setup something like prometheus, snmp server, zabbix, graphite, influxdb, grafana to aggregate all the alerts from various VM hosts, VM guests, containers... Then configure alert policies and send emails / webhooks / smoke signals and what not? Surely proxmox should have some sort of native methods to report errors right? maybe at least syslog?
It would be nice if the alerts could be posted via xmpp or element protocols instead. With the amount of spam and undelivered mail I get these days, I'm far more likely to spot an IM alert.
Is there a way to get an alert when a VM starts up or shuts down? A Windows VM decided it was done for the day and shut down. Nothing in the event viewer explains why, just that it did it and it was unexpected. I didn't notice until I tried to use a service hosted on the VM and started getting error messages. An e-mail that my VM had shut down would have been extremely useful.
How can I setup smart alerts just for the device, not for a ZFS pool? I am passing my raw disk to a truenas VM and the ZFS is managed within that VM. Truenas can warn me of the degraded pool, but I want to know about smart errors early and also.
Regarding the FROM part, have you seen Datacenter -> Options -> Email from address? That probably could have saved you the additional replace config in postfix :)
@TechnoTim - Just a heads up, Proxmox is actively implementing webhooks, and there is a draft of the functionality already pushed to their mailing list. Sounds like it'll be coming soon!
I think you should look at Zapier to finish this off and send all alerts to Slack. You create "Zaps" which are custom email addresses, and that then is able to send to a webhook, then to Slack. I use this for any alerts that are not webhook enabled and it allows me to send ALL alerts to Slack. And it's free!
I am still little confused .. Need to know if the below scenario will work. I will install proxmox in my server .. install portainer and in each container inside portainer I will install wordpress .. will it work .. Please guide
Curious - Why would this be better than letting Proxmox use its default SMTP relay (built in)? If I enter alert email addresses on a newly-implemented Proxmox server, they send out just fine. Also, it's worth noting that this configuration is very specific to Gmail. If using an alternative port with another service, complexities ensue.
I never set up any email stuff with my install and I get the backup alerts to my email with no issues. not sure what kind of magic proxmox is doing but it does seem to have a built in mail server that works out of the box. Would be nice to get web hooks though!
after switching to spectrum in 2105 Proxmox wold always send n email to the email i set during setup. errors like when my last 2 drives started failing to pass smart testing i would get a email at 5 am every day. back in the day you could send an email from any ip but until it was abused isps block the email out port. i may only get the email as my google account is tied to spectrum. im setting up asterisk again soon as i would prefer a call if i have a drive failure, water flood or carbon monoxide buildup. great video alot of it people still use pagers in my area.
one tip: it's nice that you provide an echo command for simplicity to put the password in the file, however that command will stay with the app password in your bash history file, which I believe it's not the nicest place to have sensitive data, isn't it?
Would say webhook or SNMP trap (I would much prefer webhooks) and should be a cluster-wide setting. I'm guessing the concern is depending on the failure there is risk that the alert message could be lost and that could be even worse. I'm thinking something like a node being offline and the alerts being sent from the cluster master means they aren't sent out till a new master is elected. What if something is stopping the election? I guess simplest way would be each host handles its own system / node level alerts and cluster master sends out cluster level alerts? Should be possible to push that config out via the etcd synced config and they already have the templates to change the message based on node.
Hello!Thanks for everything!I would like to install nextcloud in proxmox for pictures like google photos. Is it ok with 2 hdd of 2 terabytes with zfs storage for data? Can you please tell me how to do it? Thanks in advance.its is ok hdd WD PURPLE SURVEILLANCE HDD?
Is there a story behind the reason why Tim is Sign-Language "Thanks for watching" at the end of every video? Just wondering... Thanks for sharing the video Tim. Followed and implemented the postfix alerts in my homelab pve.
Hey! Thanks! I worked on a team that focused on accessibility and worked with someone who was hearing impaired. So one day I added it to the end and just kept doing it :)
I'd be interested in the pricing of the server and components. The Rack itself is quite expensive IMO, so i figured i get the profiles myself for cheap and build a case just by myself - since it is trivial, unless there is something i dont know about. Also, the servers themselves are very expensive too, so i would wonder how does someone goes about it - i got an Proliant G9 for less than a thousand, relatively happy about it, since it is relatively modern - but asked vendors for "basic but new ones" - and 10k didnt get me anywhere close to the used server i got. Another thing, would be UPS, i am kinda sick of hearing from them., since the upkeep is something you have to buy into as well - batteries turn bad. Thus, i have decided to get a laptop and let run proxmox etc. since it comes with its own UPS, basically - plus peripherals. With 2 drives in raid, it also has redundancy there and is about the same pricepoint as a PSU alone, which could not handle the load of the server(s) anyway for a meaningful time, other to power down. I'd love a video about the concept of how to go about getting a meaningful homelab - maybe it is me, as an European who has to deal with such high prices or just earn shit compared to the US, but spending 15-30k on a single server, really is not in my budget.
maybe its because LTT hacking incident. "To help keep your account secure, from May 30, 2022, Google no longer supports the use of third-party apps or devices which ask you to sign in to your Google Account using only your username and password."
This is helpful. However, I utilize a custom email domain through iCloud. So the email address to authenticate is different than the email I will be sending from. Can't seem to find any Proxmox documentation on this. Any suggestions?
23:25 - Yes, it can resilver that fast given you put the same disk back in. One of the things you do when removing a disk is to clear the partitions / ZFS metadata so it can't recognize the drive. I would guess it recognized the disk and then at worst would have had to write the data that changed since you unplugged it (I would guess it could pick up the outstanding changes via the SLOG but don't quote me on that). That resilvering process only having to touch actual data vs a RAID which operates at the block level of the disk is one of the major optimizations in ZFS. Its supposed to at worst need to scan the tree (ZFS's equivalent of the master file table in windows) looking for differences which is exactly what it does during a scrub just in this case it knows which drive not to trust.
Regarding Webhooks - and to be fair to the Proxmox team - they are just packaging Smartctl and ZED. They do not control these projects. The only one they totally control is their backup notifications. Absolutely yes, they can (and should) build a way to fully configure email via their GUI rather than the pain of fixing up the various config files. That is exactly what they have done for so many other aspects of configuring the tools they build upon. It should do this gracefully across the whole cluster (cluster level mailer configuration, per-host "from" config, etc). And they should push this config into the mail conigs of related tools like ZED and SmartCTL. Bush asking them to do the webhooks for those other projects is probably too big an ask.
The stress caused by deliberately pulling a functional drive!!! Yeah, we all know it should be fine. We all know we should have engineered the system correctly. But we ALSO know, it is only 94% guaranteed to work. If it did, KINGS OF THE UNIVERSE! If it didn't: Well, I guess it was a learning experience.
I am on proxmox 8.2. I had to change compatibility in order to make it work. It is in configuration file. Convert "compatibility_level = 2" => "compatibility_level = 3.6"
While the major point is totally fair - that webhooking should be integrated into Prox itself - an alternative solution could be to webhook from existing monitoring solutions - e.g., Zabbix or Grafana. But of course that comes with a lot of added complexity and configuring to perform something that should be as easy as possible by nature.
Hmm, I had email notifications from the first day of usage of my all proxmox nodes. I just entered my real email while first setup of the node and that's it
I noticed that worked for a while and then stopped. It's not sending through an SMTP server you've designated, so probably not the safest to rely on longterm. Check out the "from" address.
Love watching your videos - although you talk faster than I do.... 😀 I cannot seem to find this APP PASSWORD option. It does not exist on this newly create account following your video.
What's your favorite thing about Proxmox?
Simple
Container Support and low overhead for management.
Scalability
That it is based on Debian and I could install mdadm and samba natively on it and set it up as a file server too. That way all the RAM cash is used :)
As @abdraoufx said it is simple, yet it also is powerful and lets you shape it to suit your needs if you know how to
Thanks for this thorough overview on how to have visibility on my proxmox server, right after i did this i discovered 8.1 includes some native GUI for notifications. Might be worth a new video lol
Just a quick note, the zpool clear command is used to clear the error counts on drives back to zero, it does not clear data off the drive. zfs was asking you to do this so you could see if any new errors occurred on the drive you pulled and reinserted. so you can check if it actually needs to be replaced because errors are increasing or if all is OK and can be left alone. zpool status with high error counts can often be a sign of a drive in a prefail state that SMART has not yet picked upon, but zfs has due to all the checksums zfs uses. hope that helps and saves on future heart attacks 😊
Thank you!!!
Same story here. Didn't have alerts setup and randomly decided to check the status of my drives and saw that I was one drive away from losing all my data. Thanks for the great tutorial!
1. I love all these recommendations/ alternatives in the comments section. That's one of favorite part about home-labbing!
2. You could also use n8n. You can send the data via curl as a response to a listening webhook on the n8n server. I'm not sure if you could pull the error message from the log (i watched the video but came back for the microcenter link lol). N8n could then send out the alert via any of their options like slack, discord, mattermost, etc.
What perfect timing for this guide! The "old_age" flag on several of the drives in my oldest ZFS array just got triggered a few weeks ago so it will be nice to have notification as soon as one, or more, of them throw a more urgent SMART flag.
Thanks for another great guide!
If I had a homelab with as many devices as you have, I would monitor the environment with Icinga, Nagios or CheckMK and use their notifications. Much simpler and easier to maintain than configuring alerts on each device.
There's a really good nagios/icinga plugin called check_pve . They have a gitlab repo easy to find!
you can send metrics to an influxdb or graphite endpoint too but the email alerts are nice and be harder to setup and tune with other monitoring systems
I'd be using something more modern like Prometheus but yes all of the above are better than configuring each host.
@@justwhyamerica I've build influxdb+Grafana to monitor Proxmox.
But, error like that (zfs fail, disk fail or backup fail), can it do that?
Where I can improve? Any reference would be appreciated, thank you.
Ah, I need to mention.
Can we monitor Proxmox temperature?
Please kindly give any suggestion or tutorial online would be nice.
absolutely perfect timing! this is exactly what I'll be spending my last day in the office before the holidays doing. it always irked me that there wasn't more notification control in Prox. I have been relying on the ceph dashboard and cluster health to know when a disk goes down and it is far from ideal. And my Zabbix dashboard is not fully configured yet, so this is a perfect fix in the meantime.
Hey Tim, recently found your channel and this is my first comment. Great fucking work! As a SysAdmin, this video is high quality! All the world's success to you in 2023.
Nice way to get rid of some extra layers of complexity. Some labor involved but for us homelabbers it’s cool - we are eager to spend hours to save money. A very nice video as usually.
Leaving this comment for the algorithm and to reiterate my appreciation for all that you give to the community. This worked flawlessly for me. I have 4 proxmox nodes, so I ran most of the commands with adhoc ansible commands against the nodes. Super happy with having this working now. Before I was sporadically getting emails from proxmox, but gmail would often block them. This is a huge improvement. Thanks, again.
Thanks for the algo boost and the kind words!
18:50 - Something you may want to mention is the difference between a RAID card and HBA. Most RAID cards won't give you SMART data from the drives even in JBOD mode so that may be a reason SMART may not be working.
Thanks for the video, it was very helpful. I just got my proxmox servers set up using this approach, and it worked great!
Thanks a lot Tim for sharing your knowledge - works like a charm! I was able to setup email notifications on 3 PVE nodes (one Cluster) and my two PBS Servers. Not sure if I am really looking forward to receiving emails now ;-)
I kinda did something similar, except i make a postfix mail relay vm so all my things on the subnet could relay without auth to it, and from there it could forward out to external mailbox. Just makes mail setup on new/additional systems easier/faster.
Really handy video guide, and very timely for me. Glad you were balanced in reporting the notification system's shortcomings also.
@TechnoTim Nice how to. I was able to fast forward (thanks for section labels) to the alerts setup. if you or anyone else is interested, I have a mailserver setup that uses mailjet, so 2 factor isnt required, and the emails are not marked as spam thanks to mailjet. All I had to do was configure proxmox postfix as a relay to my mailserver then turn on notifications, alerts, etc So a mailserver is an option especially if you have multiple systems you would like to have alerts setup and not having to setup each one like the how to provided. Setup one mailserver, then configure postfix on the others as a relay. Have a great day and thanks for sharing!
Another great tutorial Tim, thank for sharing!!!
Glad you enjoyed it
Pro-tip: Connect your email alerts to Pushover for even more control! :D
One thing that came to mind is that postfix has a mechanism to retry delivery until it succeeds, a simple webhook script probably won't have that
I like this, I've had alerts going since the start but I've had to test them several times because I never get any so I always get worried it's stopped working :D
New version of ProxMox has a GUI finally! But always nice to know it via CLI.
Question: you chmod the sasl_passwd file to 600, but not the sasl_passwd.db. And when you catted the file, you can see the password is still readable there. Presumably you would want to chmod it too. But does postfix run as root or does it run as its own user that you'd have to chmod the file to?
This video was very interesting. I had the exact same issue recently, a drive bricked itself and I noticed it later.
I chose a different route to notifications: a selfhosted ntfy instance. I also ended up touching Zed, but I had to write a script to send the POST messages since ntfy is not one of the preconfigured services.
I agree that Proxmox should include better alert management tools, although webhooks would not be my first choice. My ideal system would be similar to Zed: execute command X when event Y is registered. This could be adapted to any notification method and even for automating complex tasks like starting a failover in case of degraded zpool.
And finally, if you want to test Zed it is a lot safer to start a scrub or a resilver manually instead of removing a drive, but you probably already knew that.
Oh, probably I missed something, but looks like you have auto dark theme reinstall on each apt install as trigger? Do you have this explained somewhere? 😄
iLude in our discord created it! Hop in and grab it! It's great!
@@TechnoTim Thank you, just found in ilude gist
What a great tutorial. My alerts are now configured, thanks.
Amazing, thank you. 1 quick note in your documentation under the "add the module to our postfix config" you have the word "nano" twice in the command.
Thank you, I can fix it or you can open a PR! It's open source!
postfix setup is horrible on proxmox, there should be an easier way in these days of notifications, or the ability to set it up via the GUI.
Still horrible. Truenas is super easy
They added smtp via gui
Most useful tutorial so far IMO! Thank you Tim! :)
Thanks for that tutorial. I configured it for Proxmox and also for PBS :D
Logging in to set this up now! Once again thank you for the video!!
Isn't there a way to forward an email to an webhook?
Could be actually made with a pretty simple script.
Edit: Forgot to say:
Thank you for the tip!!! Something i absolute need either and didn't even thought about and anyway didnt knew how to setup!!!
So yes! Amazing Video!
Thanks for making such simple but very specific video's!
While other RUclipsrs make another 100 videos about setting up failover dual wan with pf or opnsense, where 99% of home peoples don't even have an second isp 😂
So thanks for making really good videos and not boring crap that everyone knows!
Thing is, that there are a lot of your videos that are pretty amazing, or at least were at the time when you published them!
You know after some time everyone knows something, like "authelia" for example, where 1 year later your video is useless, but at the time you published and not many knew about it, it was amazing....
So however, great Videos man! I mean i know a lot either, but some ideas i got from you and wanted to say thanks once!
Cheers 😂👌
Great video Tim. Out of interest (and this may already have been asked below), couldn't you use the likes of Promtail / Grafana Loki to aggregate the logged activities from Proxmox / smartctl (etc.) and push an alert via email / webhook from there?
I set mine up when I setup the hypervisor. Good vid ,tim
Thank you!
An awesome, informative tutorial. Thank you.
can we get the google mail config lines? your came was blocking a good portion of it.
Thanks!
This is pretty cool. I wondered how this would be set up for email alerts.
Awesome video. Followed your steps and got this setup on my PVE box. Everything is working but when I checked my drives as you pointed out at 18:20 it shows "SMART support is: Unavailable - device lacks smart capabilities". This happens on sda1 which is a SATA SSD where the OS is installed and sdb1 & sdc1 which are SAS HDDs used for backups. nvme0n1 does show as being supported; this is where all my VMs are installed.
One additional suggestion would be to check and, if necessary, clear the postfix queue before activating an smtp connection. My email account was besieged by thousands of cron errors that had been building up in the proxmox email queue. 'postqueue -p' will show if anything is in the queue and 'postsuper -d ALL' will purge old emails. Of course you should troubleshoot the source of the error messages but it's easier to do that once your email account is not on fire.
I set up mailjet through my domain on CloudFlare. Way better then using a Google email.
Other fun things to do, if you have home assistant, there is a smpt2mqtt unofficial add-on. Then you can do whatever you want with notification
What I find amusing is proxmox has the frontend UI for email alerts but not the backend out of the box, a bit half assed if you ask me. It should be ready to go out of the box.
It works fine Tim, thanks for the help!
is it possible to setup something like prometheus, snmp server, zabbix, graphite, influxdb, grafana to aggregate all the alerts from various VM hosts, VM guests, containers...
Then configure alert policies and send emails / webhooks / smoke signals and what not? Surely proxmox should have some sort of native methods to report errors right? maybe at least syslog?
It would be nice if the alerts could be posted via xmpp or element protocols instead. With the amount of spam and undelivered mail I get these days, I'm far more likely to spot an IM alert.
Is there a write up somewhere we can copy and paste the text you are putting in? I don't see the link your description.
Hey Tim, can proxmox send alerts to more than one email address?
Is there a way to get an alert when a VM starts up or shuts down? A Windows VM decided it was done for the day and shut down. Nothing in the event viewer explains why, just that it did it and it was unexpected. I didn't notice until I tried to use a service hosted on the VM and started getting error messages. An e-mail that my VM had shut down would have been extremely useful.
Great video Tim. I wish esxi had alerts like this
how did you find out which cert to use? Mine fails.
How can I setup smart alerts just for the device, not for a ZFS pool? I am passing my raw disk to a truenas VM and the ZFS is managed within that VM. Truenas can warn me of the degraded pool, but I want to know about smart errors early and also.
Regarding the FROM part, have you seen Datacenter -> Options -> Email from address? That probably could have saved you the additional replace config in postfix :)
I have and covered it but that doesn't fix the "name" of root, which I cover :)
@TechnoTim - Just a heads up, Proxmox is actively implementing webhooks, and there is a draft of the functionality already pushed to their mailing list. Sounds like it'll be coming soon!
I had to install postfix-pcre for the header rewrites or I just got an error.
my isp doesnt block port 25 and i get emails form smart monitoring already from the default setup
Thanks got email going!!!!
But cron is emailing on the hour is there a fix?
I think you should look at Zapier to finish this off and send all alerts to Slack. You create "Zaps" which are custom email addresses, and that then is able to send to a webhook, then to Slack. I use this for any alerts that are not webhook enabled and it allows me to send ALL alerts to Slack. And it's free!
I am still little confused ..
Need to know if the below scenario will work.
I will install proxmox in my server .. install portainer and in each container inside portainer I will install wordpress .. will it work ..
Please guide
8:25 what is that hash? the password isn't hashed at all, it's viewable in the database file just fine
Curious - Why would this be better than letting Proxmox use its default SMTP relay (built in)? If I enter alert email addresses on a newly-implemented Proxmox server, they send out just fine. Also, it's worth noting that this configuration is very specific to Gmail. If using an alternative port with another service, complexities ensue.
Great ! Keep going with your Videos - like your way to do it ! Greetings from Germany!
I never set up any email stuff with my install and I get the backup alerts to my email with no issues. not sure what kind of magic proxmox is doing but it does seem to have a built in mail server that works out of the box.
Would be nice to get web hooks though!
Hi, great content! What about proxmox-mail-gateway instead of a third party (that we have to pu our thrust in)?
Awesome tutorial Tim !!!
after switching to spectrum in 2105 Proxmox wold always send n email to the email i set during setup. errors like when my last 2 drives started failing to pass smart testing i would get a email at 5 am every day. back in the day you could send an email from any ip but until it was abused isps block the email out port. i may only get the email as my google account is tied to spectrum. im setting up asterisk again soon as i would prefer a call if i have a drive failure, water flood or carbon monoxide buildup. great video alot of it people still use pagers in my area.
Thanks! That's really helpful.
What do you think about self-hosting a SMTP server for that purpose?
Can you do a tutorial on OVS in proxmox?
Thanks for this very useful tutorial.
one tip: it's nice that you provide an echo command for simplicity to put the password in the file, however that command will stay with the app password in your bash history file, which I believe it's not the nicest place to have sensitive data, isn't it?
Would say webhook or SNMP trap (I would much prefer webhooks) and should be a cluster-wide setting. I'm guessing the concern is depending on the failure there is risk that the alert message could be lost and that could be even worse. I'm thinking something like a node being offline and the alerts being sent from the cluster master means they aren't sent out till a new master is elected. What if something is stopping the election? I guess simplest way would be each host handles its own system / node level alerts and cluster master sends out cluster level alerts? Should be possible to push that config out via the etcd synced config and they already have the templates to change the message based on node.
Super helpful and informative video. Thank you Sir.
Webhooks will definitely make me try Proxmox as I'm still more in Qemu. I'll be using a Discord Server to monitor the services and servers running.
Hello!Thanks for everything!I would like to install nextcloud in proxmox for pictures like google photos. Is it ok with 2 hdd of 2 terabytes with zfs storage for data? Can you please tell me how to do it? Thanks in advance.its is ok hdd WD PURPLE SURVEILLANCE HDD?
Nice work !
Would be good to get this to work with Telegram too 😍
How come your Proxmox UI is much more colorfull than mine ??
Is there a story behind the reason why Tim is Sign-Language "Thanks for watching" at the end of every video? Just wondering...
Thanks for sharing the video Tim. Followed and implemented the postfix alerts in my homelab pve.
Hey! Thanks! I worked on a team that focused on accessibility and worked with someone who was hearing impaired. So one day I added it to the end and just kept doing it :)
This was so helpful! Thank you!
I'd be interested in the pricing of the server and components.
The Rack itself is quite expensive IMO, so i figured i get the profiles myself for cheap and build a case just by myself - since it is trivial, unless there is something i dont know about.
Also, the servers themselves are very expensive too, so i would wonder how does someone goes about it - i got an Proliant G9 for less than a thousand, relatively happy about it, since it is relatively modern - but asked vendors for "basic but new ones" - and 10k didnt get me anywhere close to the used server i got.
Another thing, would be UPS, i am kinda sick of hearing from them., since the upkeep is something you have to buy into as well - batteries turn bad. Thus, i have decided to get a laptop and let run proxmox etc. since it comes with its own UPS, basically - plus peripherals. With 2 drives in raid, it also has redundancy there and is about the same pricepoint as a PSU alone, which could not handle the load of the server(s) anyway for a meaningful time, other to power down.
I'd love a video about the concept of how to go about getting a meaningful homelab - maybe it is me, as an European who has to deal with such high prices or just earn shit compared to the US, but spending 15-30k on a single server, really is not in my budget.
I checked my google account under "Security" --> "Signing into Google" there is no section for "App Password" :(
maybe its because LTT hacking incident.
"To help keep your account secure, from May 30, 2022, Google no longer supports the use of third-party apps or devices which ask you to sign in to your Google Account using only your username and password."
Great tutorial! Do you know a way to use Telegram for the same purpose?
This is helpful. However, I utilize a custom email domain through iCloud. So the email address to authenticate is different than the email I will be sending from.
Can't seem to find any Proxmox documentation on this. Any suggestions?
Does icloud support SMTP?
Yes! @@TechnoTim
23:25 - Yes, it can resilver that fast given you put the same disk back in. One of the things you do when removing a disk is to clear the partitions / ZFS metadata so it can't recognize the drive. I would guess it recognized the disk and then at worst would have had to write the data that changed since you unplugged it (I would guess it could pick up the outstanding changes via the SLOG but don't quote me on that). That resilvering process only having to touch actual data vs a RAID which operates at the block level of the disk is one of the major optimizations in ZFS. Its supposed to at worst need to scan the tree (ZFS's equivalent of the master file table in windows) looking for differences which is exactly what it does during a scrub just in this case it knows which drive not to trust.
Regarding Webhooks - and to be fair to the Proxmox team - they are just packaging Smartctl and ZED. They do not control these projects. The only one they totally control is their backup notifications.
Absolutely yes, they can (and should) build a way to fully configure email via their GUI rather than the pain of fixing up the various config files. That is exactly what they have done for so many other aspects of configuring the tools they build upon. It should do this gracefully across the whole cluster (cluster level mailer configuration, per-host "from" config, etc). And they should push this config into the mail conigs of related tools like ZED and SmartCTL. Bush asking them to do the webhooks for those other projects is probably too big an ask.
The stress caused by deliberately pulling a functional drive!!! Yeah, we all know it should be fine. We all know we should have engineered the system correctly. But we ALSO know, it is only 94% guaranteed to work. If it did, KINGS OF THE UNIVERSE! If it didn't: Well, I guess it was a learning experience.
Haha! Yeah, I was super stressed out doing it if you couldn't tell in my voice!
I am on proxmox 8.2. I had to change compatibility in order to make it work. It is in configuration file. Convert "compatibility_level = 2" => "compatibility_level = 3.6"
It's pretty wild that Proxmox still doesn't have a way to do this through the UI.
Agreed!
Thanks, will give this a go. As others have said, it would be cool if this could be done via the web gui
While the major point is totally fair - that webhooking should be integrated into Prox itself - an alternative solution could be to webhook from existing monitoring solutions - e.g., Zabbix or Grafana. But of course that comes with a lot of added complexity and configuring to perform something that should be as easy as possible by nature.
Hmm, I had email notifications from the first day of usage of my all proxmox nodes. I just entered my real email while first setup of the node and that's it
Same here, alerts working fine
I noticed that worked for a while and then stopped. It's not sending through an SMTP server you've designated, so probably not the safest to rely on longterm. Check out the "from" address.
Weird, how would it know which SMTP server to use?
@@TechnoTim I think it uses local server, it can only send emails. I've seen that in some other linux OS by default too
Last try, I was gonna comment about a script to set up e-mail alerting. But my comments keep getting deleted here.
P - r - o - x - m - o - x T - 0 - 0 - l - B - o - X
Hey! Sorry, it must be getting flagged as spam from RUclips!
@@TechnoTim Yes looks like it. Too bad, I guess their algorithm need some finetuning.
I definitely won't look any further then nearest micro center
It would be just few thousands miles
I love in UK
Thanks Tim
I just came here quickly to like for the thumbnail
Can’t you just change user real name in /etc/passwd?
Grazie TechnoTim
How timely. Just noticed today that a drive in my raid-z1 pool has been dead for a few months. 😓💦
Fantastic! Thnx!
Glad you liked it!
Love watching your videos - although you talk faster than I do.... 😀 I cannot seem to find this APP PASSWORD option. It does not exist on this newly create account following your video.
3:34 Checkpoint
That's so simple and amazing ❤
Spinup your ouw SMTP server and redirect all incoming mails to webhooks