HTML-код
- Опубликовано: 4 июл 2024
- Home automation systems must run all the time. Today we will make sure you get an alarm on your Smartphone if something goes wrong with your system. But how can we create an alarm if the server is down? Let’s have a closer look.
I am a proud Patreon of @GreatScott!, @ElectroBOOM , @Electronoobs , @EEVblog , and others. No Microsoft Teams, Zoom
Links:
Raspberry Pi4 US: amzn.to/3pP32HZ
Raspberry Pi4 DE: amzn.to/3avMgaG
Raspberry Pi4: ebay.to/3n1sPef
ESP-01s Boards: s.click.aliexpress.com/e/_9iUH4Z or amzn.to/2SDaYAx or ebay.to/3wG6I1K
ESP-01 Programmer: s.click.aliexpress.com/e/_AsjoBx or ebay.to/2S0TcY3
ESP-01 Programmer: s.click.aliexpress.com/e/_9w9QoH or amzn.to/3i7BUTQ
Solid Polymer Capacitors 1000uF : s.click.aliexpress.com/e/_A2Ii9X
Power supplies (110-240V - 3.3V) : s.click.aliexpress.com/e/_A6pG3T
Prototype Boards : s.click.aliexpress.com/e/_AtVdgv
Sketch and Flow: github.com/SensorsIot/Heartbe...
Telegram Library: github.com/witnessmenow/Unive...
PubSub Library: github.com/knolleary/pubsubcl...
The links above usually are affiliate links that support the channel (no additional cost for you).
Supporting Material and Blog Page: www.sensorsiot.org
Github: www.github.com/sensorsiot
My Patreon Page: / andreasspiess
Discord: / discord
If you want to support the channel, please use the links below to start your shopping. No additional charges for you, but I get a commission of your purchases to buy new stuff for the channel
My Amazon.com shop: www.amazon.com/shop/andreassp...
For Banggood bit.ly/2jAQEf4
For AliExpress: bit.ly/2B0yTLL
For Amazon.de: amzn.to/2r0ZCYI
For Amazon UK: amzn.to/2mxBaJf
For ebay.com: ebay.to/2DuYXBp
profile.php?...
/ spiessa
www.instructables.com/member/...
Please do not try to email me. This communication channel is reserved for my primary job
As an Amazon Associate, I earn from qualifying purchases
#no#midroll#ads 
Наука
3:48 I think I know what you mean but surely that was 737-max
You are right. My mistake. I will pin your comment that everybody knows...
@@AndreasSpiess 00090⁰0090⁰0000⁰0⁰
And there was in fact no redundancy...
4:20 "in airplanes they invest a lot"... as long as it is not Boeing as we all know :D
@@johanneswerner1140 I came to look for this comment 👍. They had HW for redundancy buy decided not to use it
Once again an excellent, very informative video. The topics of reliability, availability and safety are rarely addressed elsewhere.
I also think that the effort is not very high for such a simple solution...
Thank you, we love your methodic approach and sense of humor, wifes also are pleased. Please never say goodbye again
The last video was probably a little too sarcastic. But I do not like such things happen...
That video reminds me old sysadmin lifehack from 1990s, where was no Corosync or Kubernetes: place watchdog server in front of one which it must monitor, and use its CD tray to push the reset button in case of failure.
Cool hack!
I thought it was a cup holder 😏.
Ah yes, good ol' ITAPPMONROBOT.
Thanks Andreas, your channel has its one level of quality.
If I just knew more about electronics.... but I learn a ton anyway.
Stay save Andreas and keep up your great work.
Thank you for your nice words. And no worries: I am vaccinated ;-)
Huge props for mentioning the possibility to use cloud-based alerting! It was in my mind while I was watching and really glad you had thought of that possibility also. Good job.
Unfortunately, so far I did not find a cloud service which is free and does not require an open port in my firewall.
I have had heartbeats and watchdogs on my mind lately with my soon to start HA project after my lab move to the new property. This video is saved to watch again when I get started. Thanks!
This addition was needed because my Pi recently lost WiFi from time to time. Without this circuit I discovered it always too late. With this watchdog I was able to find the error and correct it.
I bet the notification you received was from your wife! I can imagine how that would go 😂
@@AndreasSpiess Same here, Pi WiFi seems to be unstable. But I fixed it with a simple script. If ping to gateway doesn´t respond, wlan0 on PI will be restarted. Start this bash script via crontab:
gist.github.com/mharizanov/5325450
(you have to change "ifup" and "ifdown" to "ip link set up/down"
I love this small power supply for mains. I plan to build several ESP-01s with temperature sensors to map my house temperature.
As per cloud, I have a Digital Ocean droplet for $5 (+ $1 for backup) that runs docker with one of the containers being a node-red, and I use it to "watchdog" my IoT devices at home.
Great video!
Thank you for the link to "Digital Ocean". I will have a look at it.
I love your configurable box. Used it so many times.
Here I also have many similar looking boxes...
Thank you for a very informative video. I'll look into this for my home systems. The main server uses raid already, but no further redundancy. Thank again.
You are welcome!
As usual quality content by Andreas Spiess!
Thank you!
Excellent Andreas, as usual coming from you.
:-) Thank you!
Very usefull video as usual. After ' getting it working first then optimize' one needs to spend more and more time to optimize indeed. Lots of area's to cover to get things robust. I'll probably dive into clusters in the future. For now I learned about the timeout node. Very effective!
You are right. I was also glad when I found the timeout node. It is extremely simple but effective.
The best watchdog server is the device where you want to receive the notifications.
Your cell phone in this case.
Even if you are outside and the internet or power goes down, you still get the notification.
Great content, Thank you.
I am not sure. Because I do not connect my Smartphone to my home network and use Telegram for the transmission of the messages.
So my watchdog has to be behind my firewall. In addition I have no clue on how to program my smartphone :-(
@@AndreasSpiess I am assuming you use the telegram app on the phone to get the notifications.
You do not have to connect the smartphone to the home network in order for this to work.
I also need something like this for my pi server and other nodes 😕
Definitely (useful AND interesting)!
The 737-max is a text-book example of getting this wrong, with horrific consequences. Boeing did not want to re-train pilots so they added a system (MCAS) to make the plane behave like one the pilots had been trained on. Trouble is they did not regard MCAS as safety-critical and made it depend on a single sensor system. If that sensor failed, it could crash the plane. The pilots, without training, were supposed to spot this failure, disconnect the system and manually fly a different plane they had not been trained to fly. But to fix this Boeing have only added a second sensor system. As you so clearly explained, this can detect a single sensor failure but it cannot decide which sensor to trust - that would require triple redundancy.
So now MCAS can detect its own failure, automatically disconnect itself and raise all kinds of alerts and alarms. But still the pilots have to take over and fly the plane that MCAS was put there to avoid them having to fly. OK they have also added some pilot training so there is a better chance of them handling the situation, but there is still a fundamental logical flaw in the safety case - MCAS is regarded as necessary, but not safety critical! In practical terms, pilots are suddenly being required to fly a different airplane on which they will get little or no practice.
Safety requires clear, logical thinking. That was absent in the original design and regulatory approval. Shockingly, despite 346 deaths, the safety case still does not add up logically. This is one plane that I will never fly!
Thank you for adding all the details. I just had a rough understanding on what happened...
Another excelent and funny (happy wife, happy life) video Andreas! You are really know what you are doing.
I wrote a simple php watchdog with cronjobs and email notifications some time ago, but i also just had a new idea for a heartbeat function. This should work on nearly every hosting service or php server. There are also free cronjob services, if the hosting services does not provide that.
I will improve the code, open source it and post it here later.
Very good! Thank you. It would be great if it would be able to act the same way as my ESP8266 and I could install it on my webhosting! Then I would no more have my internet as a single point of failure.
@@AndreasSpiess The plan is the following:
1. You have a easy to expand list of devices that you want to monitor.
2. The devices can either be pinged via URL / IP (I already had that) or the devices themselves must periodically (heartbeat) call a specified webhook and the php-script stores a simple timestamp and when a definable time has expired, there is an alarm.
3. For the alarm there are different possibilities:
You can call another webhook, or sending an email with phpmailer.
Alternatively I will have a look if there is a telegram library for php. But ff i remember right, telegram also supports simple webhooks. I try to keep it as simple as possible.
Does this sound right for your purposes?
Sounds good. Yes, I also use a simple web call for Telegram in my ESP8266.
Maybe you send me the link also via Messenger because this comment will get out of my focus in a few days
Great video as always.
I often need to use watchdog timers when using an esp32, not so much with a Raspi. But an even cheaper heartbeat for one or two Raspi's would be using some monitor service like Pulseway (Just like you said on your last slide). That way I'm always informed real quick when the WiFi drops again.
After other comments, I just looked into the watchdog of the ESP32. It is much better than the one of the ESP8266 and for sure I will implement it in my future projects.
Pulsway and other services are probably a little too much for me. I still hope that I would get a simple script that does the same task as my ESP8266 but runs in the cloud. That would solve most of my remaining problems, I thing
Thank you Andreas, excellent as always.
My pleasure!
I got excited at the start and then disappointed that you didn't use a second RPi as a redundant wifi router and server.
I still gave you thumbs up for this simple alarm and hope that you will consider and look into redundant router/server/alarm.
Redundancy creates a lot of problems because you have to synchronize all data storage in real-time. This is why I did not do it. It also would be to complex and specific to a video, I think.
Andreas, You have read my mind, this is just what I need for all my ESP8266/01 sensors. My only problem is that everything else runs in Micropython (which I am no expert on) and this is in C, Keep up the good work
If your sensors provide regular updates, you do not need to change them at all. Just add the node-red stuff.
For a cloud enabled solution all you have to create is a "webhook" which is a fancy name for a simple URL address which you call to trigger some action in the cloud. Google Cloud creates a hook for you automatically when you create any Cloud Function. Arduino can now call (GET) this URL via WIFI and done: you've sent data to the cloud. Same thing can be created in AWS by combining AWS Gateway to get an URL with a Lambda Function for Node.js execution. From here on there is only a short step to sending emails, SMS warnings, logging to databases etc...
Very nice idea.
Great video as usual. Just one note: You actually do not need to install a custom timeout node. The built-in Trigger node in Node Red works just fine. Just set it to "Send:nothing, then:wait for x minutes, and 'extend delay if new message arrives'".
You are right. You get the same result using your method. Now I know. Thanks.
Thats perfect timing, because i just wanted to put together a server.
My Raspberry Pi recently died soo… I bought a Mini ITX board from Ricardo
I hope you had great holidays too!
So far I had no holidays :-( My video schedule is 3 videos + 1 week no video...
@@AndreasSpiess Oh, I am kinda dumb. I saw a comment on your last video but only now realized it was a joke.
Thanks for this vid about such an important topic!
I am using Homematic for ca, 10years and ioBroker as an integration platform since its beginnings. Both can send emails ad/or messages via a messanger. So, I implemented a bidirectional watchdog monitoring system. Both systems monitor each other.
I also monitor many of my ESP sensors.
And I implemented power grid monitoring. As a backup power source, in my first units I used supercaps but switched to energizer Li primary cell batteries. They claim 20 years shelf life which the supercaps do not do.
Thank you for sharing your setup. So you use tho different servers? And concerning the primary batteries: Do you properly shut the servers down or keep them running during a power outage? Then I assume the battery would be quickly depleted.
@@AndreasSpiess Homematic is running on an Orange Pi and ioBroker on a refurbished Win notebook. Both are backed with a cyberpower UPS as well as the rest of the infrastructure as switches phone etc. Good solution for some hours.
For different purposes I am using at different places Homematic digital input modules. They are RF bound to Homematic server (called CCU) and low power devices which are designed for battery operation. But I supply them normally by the grid via power adapters. When a power loss happens, these digital input devices will be backed by the Li batteries and using a diode one input detects that the regular supply is down and transmit this inpu change to the server. Having 3 Phases in my house, I also have >= 3 of these units.
Thanks for the great video. I have a custom Python program using Pykka (a Message / Actor model) to do all the automation. I had to make a heartbeat message for the actors for the very rare case where one crashes. I also made one for the MQTT-to-Insteon gateway program I use, which sometimes spontaneously quits working. But very rarely, something else will pop up and show a failure mode I didn't anticipate. You've given me some cool ideas to fill the gaps!
You seem to use different tools, but similar principles. As we say here "many roads lead to Rome"...
Oh my God your home is magnificent!
:-)
Very informative video! I'm so grateful you're out here making such videos, I am a small business owner from Mumbai, India and have implemented a project with NodeMCU ESP8266 and a telegram bot that send me a message everytime a customer checks into of the rooms at my hotel, however since there is a lot of noise on the AC power lines and I am using Mains power to the run the project with a small 220V AC to 5V DC power supply, a lot of noise gets through and gives me endless loops of messages this frustrated me almost to the point of giving up, but after seeing your video about using a 1000uF capacitor between VCC and GND I feel this could really fix my problems! Much thanks and love!
Maybe it will help. Ferrite beads on the USB cable also help prevent noise. But I am not sure if this is the case for your frequent false alarms.
Hi Andreas,
thank you for this video and the watchdog flow. I had the challenge that my raspberry pi with homeassistant (using iotstack) irregularly freezed after 3, 7, 10, 15 ... days. Now I use the heartbeat flow to switch the tasmota plug off and on again to reboot the pi. Thank you for the idea.
Regards from Germany
Klaus
I also had issues with my Pi: It lost the LAN connection from time-to-time. Now I switched WiFi off and it works ok...
Andreas thank you for another very informative video although a little short.
I have the usual Raspberry Pi Server running NodeRed and other programs supporting a selection of 24 ESP 01/8266/32 MC's dotted throughout my home. All ESP's have an MQTT heartbeat which will get a Software Reset if the heartbeat fails for more than 2 minutes then if this fails to reset the device I use a Sonoff Basic to RESET all power for that device and its peripherals via an MQTT Last Will. The Raspberry Pi has its power supported by a bank of batteries. All of this works very well and essentially looks after itself but I have just designed a 555 based external watchdog module that will hopefully replace most of the above. I do not use WiFi on the Pi Server so have never had your problem, I do however have another Pi acting as a WiFi Access Point and is faultless
Much of what I have achieved has been as a result of your and other makers videos for which I thank you.
I like the idea of a 555 timer to create a reset. Simple and efficient if you want "auto-healing". So far I decided against auto-healing because alarming was enough for me. But if you have more critical systems, an external 555 is an excellent and cheap solution (except for the Raspberry which would need a more elaborated shutdown...)
@@AndreasSpiess Thank you for your reply. I have nothing critical to control, I just find human intervention really inconvenient. You have allready made a video relating to using Pi pins 5 and 6 for a controlled Shut Down it would just need power to be disconnected and reapplied. All could be provided by the 555 timer
Great approach. I went with the ESP01 relay module in line with the Pi's power. If the watchdog detects a system down situation, the relay is pulsed by the ESP01to cause the Pi to reboot as well sending a message to my phone. It works flawlessly and keeps my Pi sensor hub up. It has only tripped 3 times in the past year due to Pi crashes and has recovered after one real power fail. Not bad for a $2 relay module.
Good solution. I did not want to automatically reset the Pi because I fear data corruption. That is why I first want to be sure that it really crashed before I switch it off. In my case, most of the times the Pi did not crash but something else happened and I was able to reboot it properly. But for a remote location, you solution is perfect.
Hello there. Can you help me implement your idea? Is there a email where I can send you a message? Thank you
cool idea, thanks again!
You are welcome!
Thanks for the inspiration, this is exacltly what i need right now since my HomeAssistant PI crashes a lot lately.
I think i will use my UnRAID server as a watchdog but the ESP8266 01 is also a great idea. Mabe i use both ^^
Using another running server for sure is the cheapest way to do it...
Another great video! I have used an ESP01 connected to the reset pin (and the UART pins) of an arduino that collects weather + PM data. If the arduino hangs or give erroneous data, it will reset the arduino. If the ESP01 can not connect to WiFi / MQTT server, the ESP01 will reset itself. This is usually what you would be doing yourself if one of the IoT's hangs (or turn off the power momentary) anyway.
A very good solution for a MCU. With the Raspberry it is a little more complex because you can kill data with an unscheduled reboot. This is why I did not automate it. I discovered, that the Pi itself never crashed. Other things happened which stopped it from working. In my case it lost network connectivity. So I was glad it was still running and I could do the analysis. But for a remote location, your proposal would be the way to go.
@@AndreasSpiess Nice to hear from you! I have not connected this reset function to my raspberry pi, because also in my experience, it never crashes. For me the location was not remote, but the time was. These problems often happen when I am not home or at sleep. Now I am using multiple ESP01's and arduino 3.3v pro minis (I like to use the same vcc's and most sensors on the I2C bus are good with 3.3v as well). It is a simpler solution, albeit I use more of those (I used an arduino Mega first with all the sensors connected). I am using Node-Red and Grafana as well (credit to you, thanks so much for that!). Soon this will also control a Swiss Glacier Express (model) train that will be running in our garden. All these (plus more) are powered by 2x 100W solar panels + 8S LiFePo4 batteries. You can tell, I am a big fan of your channel!
Very very Good work. This is basically how i run My home :)
So you are very professional!
@@AndreasSpiess dont know about that but in other hand its part of My daily work as well :)
Fantastic information. Thank you.
Glad it was helpful!
I love the power outage time analogy, measured in beers consumed. 😂
:-))
Well done Andreas!
:-)
I don't really comment on youtube, but I just thought this was a nice opportunity to say two things:
1- I Love your work. It is Very productive and helpful for human kind. Thank you.
2- This video is a perfect usecase for the real power of the cloud. I love my private pi cloud, but this video shows its limitation.
Your explanation of redundancy is excellent, but the solution could be improved to fix the Mains and WiFi failure with the Amazon/GC/Azure cloud option.
I think the free tier option on Amazon will work as a watchdog beyond the 12 month introductory period for free. Setup can be simplified with docker imaging solutions. Similar to the Pi IOT stack. Using a free linux instance on the cloud will come in handy and expand the possibilities for many other automatons in the future.
1. Thank you for your feedback. It is valuable for us RUclipsrs!
2. Another viewer wrote that he tried the free Amazon service and was not able to create outside calls in the free plan. I still hope I will get a small script I can run on my webhosting (where I anyway run my webserver). Another viewer is working on it.
Your video was both useful and interesting.
Thank you!
I/F >>> Important / Fundamental . . . . thanks yet again !
You are welcome!
Another informative video!
Glad you think so!
Prometheus/Alert manager is also a relatively easy system to implement. It also attaches to Grafana and can provide metrics on your system.
You are right. True monitoring systems have these possibilities. So far I always thought that they are an overkill. But maybe I have to revise my opinion...
Excellent exploration of the topic.
I enjoyed the method you used to break down the problem according to points of failure and parameters to deal with relative pain.
I absolutely hate the idea of using a cloud-based watchdog. I see no reason to send any home automation data to an external server.
Instead, how about plugging in an old phone or tablet and have it use either your neighbor's wifi as an "external" watchdog or better yet, use the cell network. It's battery would persist in a power outage easily long enough to message your cell phone or even call it. No fancy UPS required, but potentially an added line to your cell phone plan.
As for the ESP8266 device, it needs a battery and charger plus a status RGB LED(s) to make sure it is working especially if you don't get a lot of messages normally.
The only other critique I might have is regarding the watchdog on the ESP8266.
This could be potentially more functional and easier integrated into Home Assistant, for example, if it were implemented using platform.io or ESPhome which could allow less fidgeting with the device and provide for OTA updates.
Great job with this.
Using a different home instead of a webservice is an interesting idea and for sure works.
Personally I am not too concerned to send just an "I am ok" message to the cloud. I transfer much more important stuff via the internet ;-)
Great, thank you
You are welcome!
4:41 Nice power out procedures 👌
It took many years to develop ;-)
I use Zabbix 5.2 which now supports MQTT. It runs on a raspberry pi and is a dedicated package for network monitoring. It also will tell you if your systems run out of disk space, of ram, if other pingable things go down and so on.
Seems to be a very elaborated system. Probably an overkill for my simple Pi server... But for bigger servers for sure a good addition.
Das war wieder ein cooler Post
Danke!
Hi Andreas, thanks a lot for another amazing video. Can you please create a video on pi-holes? specifically how to backup a pihole? I have a Pi-4B that runs pihole and home assistant and I can live without HA, but pihole is absolutely necessary as it's the DHCP server for my network. So have it redundant would be very useful
Redundancy is extremely complicated and definitively my strength. From what google says you can just run 2 pi-hole devices in parallel without any connection. ( e.g. discourse.pi-hole.net/t/using-a-2nd-rpi-pi-hole-for-redundancy-setup-question/26932 )
Nice topic :) I went for redundancy with 3 pi's where three Node Red instances are updating each other with heartbeats every 500 ms to see who is online and who is master. Only need to buy some extra rflink to also make them redundant.
Here also the WAF was the main driver for the project!
Good project! Because I run an InfluxDB, redundancy is not so simple for me. So I decided to stick with just alarming.
What do you mean with "master", BTW?
@@AndreasSpiess I also run influxdb now on this same fail over cluster. But that is done by docker swarm and the storage by glusterfs.
With the heartbeats the NR instances are electing a master (the highest number is always master if online). Only the master processes the mqtt messages so messages are not double or triple times processed. The slaves are silently waiting till they need to jump in when the master dies.
That sounds like over my head! But very interesting. Thanks for clarification.
Cool project.
Thank you!
Please do yourself a favor and implement some function like "timePassedSince" or "timeDiff". (see your code at 9:50)
inline int32_t timeDiff(const unsigned long prev, const unsigned long next) {
return ((int32_t) (next - prev));
}
inline long timePassedSince(unsigned long timestamp) {
return timeDiff(timestamp, millis());
}
This will save you a lot of headaches when the stuff stops working after 49.7 days.
You now use 2 different ways to compute whether some interval has passed and it does take me a while to figure out whether or not they are correct.
In my optinion, such simple code that cannot be "read" in seconds is very likely incorrect, or at least raises doubts about the correctness.
By using those 2 simple one-liners you can take away a lot of doubt.
I implemented your recommendation. Thanks!
Also, "magic" numbers should be avoided. I guessed the first number as a one-minute "heartbeat" but I did not understand what the second one meant .
@@michaels3003 Looks like a free interpretation of "2.1x" the heartbeat interval. 2x 60 sec + 10 sec.
@@TD-er , Dankjewel (I got it from the WWW; I don't speak Dutch).
Hey Andreas, depending on how customized you have your MQTT messages set up, you can have your clients set a "Last Will and Testament" message on the MQTT broker that will automatically get triggered when the device loses connection.
You are right. That would be another possibility for monitoring sensors.
Yes, monitoring becomes more and more important. In servercenters we are doing it for years, without thinking about it. All the important stuff is monitored, heartbeated, watchdogd and redundant at several levels. It is a set rule. However, at hobby we do not value availability. Even commercial smart home devices do not care about it, it seems. Glad your wife did "force" you do think about it. I was creating a ESP32 to act as a forwarder for BLE beacon sensor data to Home Assistant. However, sometimes the ESP32 dies due the heavy BLE scanning and Wifi use. So I added a ATTiny45, created a serial connection between both microcontrollers, let them play "ping pong". If there is no pong from the ESP32, the ATTiny switches a relay, which cuts the power to both microcontrollers. If there is no ping from the ATTiny, the same action happens triggered by the ESP. Works as expected, and turns the BLE to Wifi forwarder into an almost maintenance free device.
Peter, I like this ATTiny solution. I am going to be setting up HA soon at my new lab to do various activities and had a BLE to WiFi gateway project in mind and your insight here sounds like a necessary solution. Thanks!
Good idea to self-reset the device! This is one step further than what I do and it works for all sensors. With the Pi I had an issue where it lost WiFi connection from time to time. Like that I was able to find it out by looking at the screen (after receiving an alarm). First I always thought, it completely crashed because I was no more able to SSH.
I had similar issues with the ESP32 getting stuck after just a day or two when I followed Andreas' video 176 to build a BLE "presence detector". Being more of a software person than a hardware one I spent some time to rewrite the code and make use of the built-in hardware watchdog timer on ESP32. None of my BLE sensors stuck again since then (several months). I liked doing this because it didn't need anything else, it just used what was available, in the way it was meant to be used. But I agree there are other ways to achieve the same result, to each his own. Too bad though that many times watchdogs are needed just to correct issues with the software - it would be so much better to correct the software instead. But I suppose it's acceptable to use a quick watchdog reset where availability is not as critical that you cannot be offline even for a few seconds.
Thanks Andreas for the video, I really liked the "minimalistic" approach of using just an ESP01. That, coupled with healthchecks.io should be more than enough for "SOHO" usage. Here's to as few alerts as possible! :)
@@dan-nutu I am a software guy myself, and fully agree with what you say. However, I was not able to fix that memory hole that was created by the BLE Scan stuff of that particular esp board. So, the wtahcog hammer is in place now. At work I do the same: try to fix things in software first. But having a hardware watchdog is like a pill to sleep well, and kind of an insurance. if all your measures fail ... the watchdog deals with it.
@@AndreasSpiess I had rhe very same issue last year, I eventually traced it to the metal case for the Pi, and the direction it (the Pi) was facing (WiFi antenna not facing the AP). Since then it has become a non issue but at the time I was so unhappy with the situation, I made a few "adjustments": 1. I used 5 different colour LEDs mounted right on top of the Pi case and connected to 5 GPIO pins to show CPU activity, HDD activity (independent read and write) and network activity (independent send and receive), 2. Wrote a script to test LAN connectivity every minute and store and process the test results (for some reason I can't explain the WiFi connection has more issues during the night) and 3. installed and configured collectd to save plenty of statistics in influxdb and monitor and display them from grafana. I needed some time to do this but it was a very good learning opportunity and I'm also happy with the results!
I use a little watchdog system like this, but it performs double-duty. It has one of those cheap GSM/GPRS radios on it and it acts as an SMS gateway in my basement, allowing sending/receiving messages for interesting projects. (It receives data telemetry from a box 130km away by text message and updates a web site.) But it also can send me a text directly if the power goes out or my main internet goes out, or in certain other cases. "The usual suspects" will sell you a SIM800 based GSM board for $15 to go with your ESP. Adding a SIM to a family plan or getting a super-cheap PAYG SIM will keep you in business for a few bucks a year.
Good concept! I do not want to create GSM systems (systems are or will be switched off) anymore but so far was not able to get a 4G system running. I have to investigate more into the topic.
@@AndreasSpiess The carriers where I live have already "dropped" GSM support, yet all my GSM modems continue to send and receive text messages. I think there's meant to be some level of backward compatibility in the networks. These are very commonly used in alarm systems and perhaps the carriers maintain some compatibility for them. That said, there are 4G and LTE radio boards around too, just a bit more expensive. SIMCOM SIM7600 based boards start in at around €50, for example.
Wow amazing video.
Thank you!
Great video. Have you made any enclosures which do not use a cable, but rather plug directly into the Mains power outlet and are held up by the plug only? For the WAF factor I am looking for a way to power a bunch of ESP32s directly from main plugged in throughout the house. I must look "neat" by hiding all electronics inside a small box, plugged directly into a power outlet.
No, I did not build such a box. But it should be possible with 3D printing.
If you wanted to implement internet monitoring, I would suggest using Oracle's Always Free tier, as they let you use 2 VPS for free (I'm using them for discord bots and they're working great)
Really interesting topic. Thanks for this video 👍
Good to know, but probably a steep learning curve for a beginner like me...
Another option instead of building an AWS service - there are a number of "website monitor services" that can periodically request from a website. Typically this is done by requesting a particular web page and looking for a particular string in the response.
For example if you have an external web address for your home server, the service can request a page you have created in Home Assistant that has information about your critical components. If the response does not have the expected string, or if it has a "warning" string, the service will send you a text, email, or even in some cases call you.
I used to use these services in my work, but I have not used recently so can not recommend any in particular but a quick search shows many of them with reviews etc.
You are right. I got a few links to such services and checked them out. Unfortunately, I did not find a free one which does not require an open port in my firewall. That would be my two requirements.
@@AndreasSpiess that is definitely an issue. I have sent a message to Paulus Schoutsen, founder of Nabu Casa (Home Assistant) to suggest that they add this service to their monthly subscription option. This would increase the value to subscribers who are already allowing this access, at little cost to the company, and would likely increase subscriptions substantially.
Another option, but I don't know if any service supports it, would be a Push notification. One's home server would have an integration that would send a message periodically with status information, while the remote service would notice if the message is missing or has different information. I did not think to suggest that to the Baby Casa team. It would be easy to set that up as a service using AWS or other cloud service but I'm not sure how it would be funded.
In the meantime I was able to create a service on healthcheck.io which does what we need. Unfortunately I am only able to generate emails in the free plan.
I would like to follow your method, and build one myself. Thank you very much.
Good luck!
I really like the tupper for the outdoor sensor!! Does the content of it overheat under sunlight? The problem with watertight systems is the lack of ventilation. A year ago I built a custom navigation system for cycling and a similar tupper but all plastic was also my enclosure of choice. Simple cheap and fully watertight. Cheers
1. It heats up in summer and you have to check if it is acceptable in your area.
2. Completely waterproof is not a good idea, I agree. But I still did not find an ideal solution.
@@AndreasSpiess In my opinion an effective way of cooling a watertight, or waterproof, system would be by using a water cooling circuit. But that would be only beneficial for something more demanding and more critical. For a couple of sensors is a total overkill.
Great video.
I'm thinking of building a power supply like your Tenstar one. But how do I go about connecting the mains power? The AC connectors of my DC converter (from AZ Delivery) only have those flimsy electronic components' legs, that doesn't feel right to solder to a 220V wire.
I did not have problems with it;-) The currents are very low.
Have a look at Nems for devices monitoring system , can be install on a raspberry pi
There were a few mentions of true monitoring systems. I agree they can be used for that purpose (one thing I am not sure are sleeping sensor nodes). For me this always was an overkill for my Pi. But maybe I am wrong...
2:35 why do you tie your bike with your car?
Do you know if it's possible to buy a sun sensor to lower the blinds when the sun shines? I don't know if it makes sense to do it a brightness sensor like the Xiaomi Light Sensor 0~83000 lux Zigbee 3.0 Light Monitor . What do you think? Thanks
1. I was building an antenna with my Harley (see video #212) ;-)
2. I use a home made light sensor, but any light sensor can do. I built my own because I wanted to have it outside and I want to extend the awning only when a lot of sun is shining.
would have been good to see how you soldered the 3.3v transformer to the perf board. is it safe to use mains voltage on those? i would have thought you would need to ensure the traces all around the mains voltage pins were cut away or something in case you get arcing? or can you just solder them to the board then solder wires to them?
I do that, but it is not according to the laws. Still, I became 66 without being killed ;-)
Newer Pis come with the hardware watchdog enabled. If you configure watchdog daemon or systemd to monitor the watchdog you can have your pi automagically reboot when the network fails or CPU load stays above threshold for long.
A good point. I will have a look at this possibility. Thanks for the tip!
At a lower level - don’t forget the use of the watchdog timer built-in to Arduinos, ESPs etc. I think you made a video about it a long time ago! I used it to ‘fix’ the very unreliable software library that came with a 4G shield. If it goes to sleep the hardware timer expires and initiates a restart - fixing the bugs!
Good point. I think, I never used it with the ESP32. Maybe a new project...
thank you
You're welcome!
You are braver than I, sir. If the wife ever asked me to build out automation for her, I'd be using military spec hardware in rad-hard cases, with everything interconnected on a hardline in a buffered loop. 🤣👍
:-)) Sometimes, a tough customer is needed to bring things forward!
Hi, Man thanks for your video. I am interested in learning these things on my own but I only know basics like resistance and voltage. How should I start?
Best is always to start with a question or a problem and solve all the problems. Maybe you start with a simple one. Google and RUclips are your best friends.
You can use a power socket running Tasmota and a few rules which in addition allows to reboot your server. All in one device 😃
I control my central heating with a tasmota-flashed device. It has some rules on it so that if it loses wifi or mqtt conections it turns the heating on. Its my failsafe in case my ha fails when Im away for the weekend in the winter, the central heating would then be controlled by trvs. I have a second pi that monitors my main ha server, and if nothing is heard from the main server in 15 mins it will email me and powercycle the ha server.
Both go much further with the automation as I did. Very well done! I just wanted an alarm, but on my Smartphone screen with a sound. That is why I did not use email.
@@AndreasSpiess You know that on this channel we want more. ;) Thanks for your inspirational videos
He he he you have to respect a man who measures power outages in beers.
As a married man I have to be creative with the excuses ;-)
The BCM2835 System on a Chip that powers the Raspberry Pi has a WDT on board. It has 20 bits and counts down every 16us for a of 16 seconds. This means you have to write to the internal WTD earlier than every 16 seconds, or the WDT will fire. That watchdog can be used to make sure that the Raspberry Pi is operational.
Good idea. Thank you!
02:41 I see what you did there! :D
I know. But you are a good observer!
While I don't have many Raspberry Pi's in my home, I do use several Arduino mega's and ESP8266's. So what I do in my environment is include a heartbeat pulse in my sketches. It changes a pin from High to Low and then back every X number of seconds. Then I program an ATTiny85 with a watchdog sketch. If the tiny doesn't see that change in the defined time period is pulls the reset pin Low for a second on the Host and causes a reboot. But given that all my devices are connected to the network I may incorporate this technique to let me know if something has been force restarted. Then if I get several in a short time frame I'll know I need to look at the device.
Interesting idea for an ATTiny! I just thought if this could not be done with a 555 timer and I found this link: upperbound.com/projects/555-watchdog-timer/
As you write, my setup also checks the network connections (which were the problem in my case)
I am not at all a skilled programmer but this sketch github.com/sktaylortrash/atTiny85-Watchdog in conjunction with this class github.com/sktaylortrash/HeartBeat has proven useful. You can configure the timeout so if you have something running on the host that blocks other processes from running (bad programming I know) you can configure the timeout to be larger than that block so it doesn't reset mid-process
There are services like uptimerobot which can be configured to ping a certain port. The free account works really well!
True and maybe IFTTT can be configured. Also there there might be a telegram app too.
@Aleksandrs: I just tried uptimerobot. Because I do not want to open a port just for that I was happy to see their "heartbeat" option. After registration, I discovered that this is only available in the pro plan :-(
@AppEdTech: I never found something like a timeout on IFTT. This would use resources all the time. Maybe this is why they do not offer it. Or maybe I just did not find it...
Love when this guy talks about Swiss and how fucking awesome it is :DD Of course swiss have many negative things too..but i love how logical everything is in Germany and swiss
Nice quorum system.
Thank you!
You can do monitoring with Google Sheets. But it requires access from the outside.
I use google sheets for another project, but did not think of using it for monitoring and alarming. How is this done?
Excellent topic. Thank you. Beers 2 & 3 == redundancy? :)
Exactly:-))
I get my mains/internet connection watchdog from Flightradar24 feeder outage notification e-mails. It takes a bit more than an hour to trigger, but it's quite foolproof and probably very redundant on their side.
That is a very interesting concept. To use a service which implemented a watchdog for a different purpose. The only thing I would not like is the alarm via email because I do not check my emails regularly. I like telegram because it alarms me with a sound...
@@AndreasSpiess It could just happen that the adsb raspberry pi is just down, but I still have to encounter that situation. For general monitoring, I use timeout and IFTTT notifications.
Another great video Andreas, I would have been very disappointed if you had really stopped.
Thank you! The idea with Helium was probably not a very good idea ;-)
Thankyou Andreas,
In this video at 6:21 you mentioned a transmitter sending signal to a satellite
Perhaps you already has a video about it, Please let me know
Once again ,THANKYOU, I try to watch your very informative videos, a very good public service fir the builder community
It is a Lacuna transmitter and I made a video about it
@@AndreasSpiess Thankyou Sir
Hope the service & reliability has improved now, I am waiting for response from the satellite service provider
THANKYOU
I've created even simpler and cheaper watchdog using digispark plugged in usb port of monitored device. Heartbeat service from system signal states. Digispark monitor atx +5V standby, +5v ON, and heartbeat. I have 3 LEDs - red is signaling +5V standby, yellow - +5 ON, and green have few flashing modes. When system is booting green is gradually lights up, and immediately turn of, after booting green is still on, if few heartbeats are skipped then is blinking 3 times in rapid succession and pauses for 2 seconds. Shutting down system is gradually dimming, and immediately turn on. All leds are extended by UTP from a couple devices to one board next to my desk.
Very nice idea! In my case I wanted to monitor also the network connection as well as all the docker containers like MQTT broker and node-red The network connection was the main problem and not the Pi itself.
"In the worst case, we have to open a second or third bottle" hahaha great man, that plan is not that bad. Is that called "beer redundancy"?
I will call it like that form now on! A very good name.
Great video. And of course Mrs. Swiss accent is always the priority!
Maybe your watchdog can power cycle the Rπ?
well, unless you have some safeguards implemented - soft shutdown or read only filesystem to prevent filesystem corruption - i'd say don't just power-cycle it. but in theory: sure. sonoff sells neat wifi controlled USB power switches, and they can be flashed with tasmota. afterwards you can control them via MQTT. then you just sandwich them between the USB power supply and the PI, and you have power control
sorry, not sonoff but sinilink, the product is here: www.aliexpress.com/item/4000791220450.html
Andreas covered it in this video: ruclips.net/video/lrHhn2AVzSA/видео.html
@@sikedipuuhja7376
I understand your concerns, but if the processor is hung the damage has been done job at as if the power went out. And it will likely require a power down or reset anyway.
I had my fair share of problems (the real reason for this video). The Pi never crashed, but somehow lost its network connection. So I always was able to reboot it from the console. Which I preferred because I rund IOTstack and docker. I was also happy that it did not automatically reboot because it simplified the debugging.
But for a remote Pi, an automatic reboot probably would be a good solution...
@@AndreasSpiess funny thing, this (console access) can be managed by a simple ESP01 wired to the GPIO pins of the PI which is exposed as /dev/ttyAMA0, then you essentially have a wifi-accessible console, where the networking stack is 100% independent from the pi (as it runs in the ESP) so you can connect to it wirelessly, and perform the troubleshooting procedures automatically.
Even Andreas, I am playing with a ESP32-WRoom-32 but it doesn't show up on Win10. It's usb doesn't show up tried the CP210xVCP driver. It is a deve board from China 38 pins mate. Nice vlog time to watch you mate. My front gate needs a watchdog.
For a long time I had no driver issoes with Win10...
The front door watchdog probably would be a little different ;-)
@@AndreasSpiess I wondering if it is the enable to earth capacitor problem. Or maybe I spin up a qubes os instance and play in Linux. with it.
Remote debuggingh is very hard. Particularly around half of the globe ;-)
Hi Andreas,
feedback from (another) old-timer
Great description on the watchdog principle. In the early days of CPU, I used to implement those in hardware, based on a timer/counter that got reset as long as everything is fine. If it missed a heartbeat, then the hardware did reset/reboot the whole thing.
Have you looked at the app IFTTT? That (internet) service can generate alarms if it does not get a signal within a certain time.
BTW... there is also a Telegram bot called Watchdog. It looks like it can do everything... except what we want it to do. Would it not be cool if Telegram could generate an alert if it didn't get a heartbeat from our RPI?
Thank you for your weekly high quality video. It brightens up my Sundays !
Every time
chrisV
I looked at IFTT a few years ago. Unfortunately, they did not have a timeout function. They only di " If this then that" in a positive way. While making this video I only googled and found no result. Do you know more?
concerning @watchdog_robot: I looked at its description, but found that its purpose is different. So we have to search elsewhere.
I was thinking about a hardware-reset to. Both controllers have a reset-pin so let one reset the other. Or, if that does not work let one enable a relay that disables the other one's power for a couple of seconds.
Agreed. Hardware reset would be a good thing for sensors. But not easy because they are everywhere. But I do not want to cut off my Pi if I am not 100% sure it has a severe problem. Mine, for example, recently had an issue with network connectivity. It still ran, but had no more connections. So I was able to properly reboot it and avoid the chance for a loss of data...
Yeah, I learned this the hard way. My Pi-based webserver isn't very reliable, it keeps hanging for whatever reason (power supply? 🤷). I used an Arduino with a relay to let me remotely cycle the Pi's power to reboot it. Unfortunately, the Arduino's Ethernet library didn't work from outside the LAN (🤨), so I ended up replacing it with an ESP with WiFi. The problem is that my ISP's cable-modem is garbage and keeps rebooting and dropping the Internet connection, so I also had to add a relay to cycle its power too. Then I added a relay to cycle the router since it hangs whenever the cable-modem disconnects (wth? 🤨), and finally, I added another relay to cycle my landline's VoIP adapter since it too dies (why can't anything reset when the Internet connection comes back up? 🤦) I also added a DTMF module to let me dial into the landline and punch in a code to reset whatever I need that way instead of thorough a webpage. 😀 Of course, if the Internet connection is down, I'm completely cut off from everything, so I had to add a watchdog to check if the server, router, VoIP adapter, and cable-modem are running, and reset them as needed (unfortunately, not all of them respond to pings even when they are running, so I'm still pulling my hair out and worrying that if I ever have to go out of town, everything will die and I won't have access to my server until I get home. 😒 I'm thinking of attaching a hammer to an arm on a servo to just randomly smash the whole setup. ¬_¬)
Sounds like a very sophisticated system! I am glad my components are a bit more reliable ;-)
Very useful however I'm probably too lazy to implement 😜
I also did not have it when everything went well. But recently, I had stability issues with my Pi...
You can use anything with tasmota as a watchdog, it's really really small, easily configurable, mqtt enabled, rules enabled, and is mains connected... No box required, only 5 dollars... And its easily replaceable.
Good point!
Connect an esp01 directly to the 3v3 line on the pi, and connect a gpio to the pi reset pin. Code the esp01 to reset the pi when needed. Or connect a gpio from the pi to the esp01 and code it to reset if it does not receive a pulse for a while.
Good idea to save the power supply and use the 3.3 volts of the Pi!
For the moment I do not want to reset the Pi automatically because it might not be crashed. Just some service or the network might be corrupted and I can do some root cause analysis. But for a remote location this is perfect.
Many routers can be set up as watchdogs for other devices with a little hactivation.
How would this be done?
@@AndreasSpiess Many routers run some form of Linux. In my opinion the easiest way would be to create a cron job to ping the device. Open-wrt is pretty common on routers. openwrt.org/docs/guide-user/base-system/cron
Aha, I understand. Mine is a "closed shop". So I cannot do it this way...
Hello Herr Spiess,
One piece that has been missing from my IoT system is the ability to remotely monitor without paying for some service. I understand I could port-forward on my router but I think that creates a security risk on my home network. Would appreciate your input. Thanks for the video.
I got some hints about remote access services in my Raspberry Pi videos. But I never tried one.
You could use Blynk. The first couple of widgets are free. And extending only cost some few dollars one time.
It might be a stupid idea, but if you‘d make the watchdog battery-powered, wouldn‘t that also help with monitoring mains? If the battery capacity reaches a treshold value, it sends an alarm as well.
Unfortunately, the internet and WiFi are mains powered. So the Telegram message would not get through… Then I would need a cellular connection in addition.
@@AndreasSpiess That is true. I thought of a LoRa-based or GSM-based solution which might suffice for sending simple alarm codes, but LoRa gateways depend on mains as well.
I implemented watchdog as folloving: the node red pings the nodes (asks for their ip via mqtt) if two responses from one node is missing it send a message with telegram. If connection is restored a new message is sendt.
If node red receives no messages it is set up to execute a reboot.
This has worked flawless. Without any extra components.
A good idea. A reversed heartbeat. I assume you could ask for any data to check if the sensor is alive. But what happens if your node-red goes down?
@@AndreasSpiess if it is the network or other on the pi then node red initiates a reboot. If it is node red itself, then systemd do a restart of the node-red service.
I also have a node-red-contrib-vcgencmd node logging throttling, undervoltage etc to narrow down problems. A good powersuply solved a lot of my network problems...
Have not had any stops on the pi/node red that has not been handeled in this setup.
After a crash/network/powerfailure I get about 45 telegram messages telling me the different sensor /switches is online again.
166 day uptime, installed some new nodes and got 4 restarts (systemd) inn one day. Removed new nodes and is back to normal..
Hi, Andreas Spies please I would like to connect my ESP32 to internet by using Ethernet module to help me to access a remote server but I don't know which Ethernet module I should use
Most of us use Wi-Fi to connect our ESP32s to the internet.
Did you try to use matt heartbeat - sys/# topic could notice that some sensor is off?
I do not think my sensors send anything on the sys topic. But I do not care too much about on which topic I get a message as long as i get one ;-)
@@AndreasSpiess if i know you are using mosquito, if you try to subscribe topic "$SYS/broker/log/N" if some of the iot dev. is disconnected then you see special message "socket error on client ..."
Thanks. I will monitor this topic and see what I get...
I wanted more. I make Google monitor my heart beat. So I know if power/wifi/esp goes down.
Good idea!
You could go even cheaper if you have another devices running permanently by adding that code to it. It could then double up as system watchdog, which is then free (and you don't have to find somewhere out of the way to put it). It has to be a device that doesn't go to sleep when it's not busy, of course.
You are right. But I am glad I do not need a second one. Everything fits on my Pi4
@@AndreasSpiess Ah, yes. Actually I meant you could dispense with the ESP-01 if you have spare capacity on (eg) an ESP-32 device somewhere.
which is the programm that visualize the flow ?
Node-Red. You find other videos where I use it. It is not only visualizing. It is actually the program which runs.