Local Voice Control In Home Assistant Is Next Level!
HTML-код
- Опубликовано: 4 июл 2024
- In this video we are taking a look at how to setup and configure the new local voice control features in Home Assistant so you can start controlling your smart home with your voice and various devices around your home! From push to talk, to analogue phones and voice pipelines, I will take you through step by step to get everything working!
Thank you to PCBWay for sponsoring this video, check them out: www.pcbway.com/
Parts for DIY:
ESP32 Development Board: amzn.to/3MlPH7L
Microphone: amzn.to/3LY3d0c
Alternate mic: amzn.to/41CU1nr
Amp: amzn.to/42QGKZz
Speaker: amzn.to/3I5R6Nl
M5Stack Firmware flash:
www.home-assistant.io/project...
ESPHome example code:
github.com/esphome/media-play...
If you'd like to support me and future projects:
Patreon:
/ everythingsmarthome
Buy Me a Coffee:
www.buymeacoffee.com/EverySma...
Written Article:
everythingsmarthome.co.uk
EVERYWHERE YOU CAN FIND ME:
Everything Smart Home Community Discord:
discord.everythingsmarthome.c...
Twitter: / everysmarthome (@EverySmartHome)
Instagram: / everythingsmarthome
Facebook: / everythingsmarthomeoff...
#homeassistant #smarthome
________________________
Some of the links above are affiliate links, where I earn a small commission if you click on the link and purchase an item. You are not obligated to do so, but it does help fund these videos in hopes of bringing value to you!
0:00 - Intro
1:11 - Configuring Pipelines
7:11 - Debugging pipelines
8:08 - Exposing entities to Assistants
9:13 - Creating an ESP32 ESPHome powered speaker
10:51 - PCBWay
11:24 - Creating an ESP32 ESPHome speaker again
13:23 - Setting up an analogue phone to work with Assist 
Наука
I do like the idea of picking up the phone, and being in contact with the house straight away! It's like sci-fi, but at least 30 year old!
"You can just pick up the phone, and tell the house what to do, and it happens" Seems like something that would have been mildly mindblowing at one point in time
This is mind blowing, hopefully a magic word is introduced like we say "alexa" or "google"
It's definitely planned for the future!
@@EverythingSmartHomeI really hope we can customize the wake word.
@@shadowopal can already do this using Rhasspy, haven't played with Rhasspy 3 yet
The magic word is so gonna be “Jarvis”
I second that. Hey Jarvis 😂
Great Video. Loving the content that you are putting out and the Year of Voice has just made my jaw drop. Would love a video setting up Piper / Whisper on a more robust computer / server while still having all the base home assistant things run on an rpi4 or similar.
This is great, thank you. As soon as there's a similar option without requiring button press - I'm all over this.
this is an awesome video! Would love to see another video showing how you got the DIY build using the ESP32 Development Board. I've been trying to modify the ESPHome example code to work with my own DIY build and not having any luck. It seems like assist just hangs in "listening" mode. Thanks for all the awesome content you create for us!
Great video, as always. Thank you for guiding us to a less painful experience.
Appreciate it!
Lewis you've outdone yourself....again!! Great video!
If only some clever chap had created an ESP based device which could sense my proximity and be used to automatically listen when I was within a configurable range. Such a device might be named Everything Presence Assist or EP2 perhaps? :)
Great video, well done!
Haha! Well to be fair, it does have spare GPIO pins...
Should be doable on the EP1 with a little ESP Home modification. Not something I would do as who knows what it might pickup, but an interesting idea :)
@@EsotericArctos agreed, that's why I'd suggest setting the proximity so that you have to be pretty close to it to start listening, probably also add an LED connected directly to the mic so that you know it's listening. But don't forget, hopefully one day we'll have wakeword detection, then this could be a really cool addition ;)
Cant that be done with an automation? Add a motion sensor and fire a service?
You sir are a gentleman and a scholar. This is great information. Thank you!
This got me up and running. Many thanks!
I am 8 mins into the video and this seems to have fixed the one annoying issue I had, that I would need to repeat my voice command 6-10 times before google assistant would randomly obey. Google would just keep saying, "Sorry, it looks like the home assistant isn't available right now." and on google home app I could watch my devices would grey out and come back over and over. Imagine that one needs to be pretty desperate to ask home assistant 6-10 times for 1 command to work, eg. light off for bedtime. This is G R E A T!
I have a few extra Alexa Echo's, would be nice to gut some of the components and have the ESP32 take control of the microphone/speakers/LED's to get this working!
That was a great video.. it all worked on the first try... Thanks as always!
Appreciate it and glad it helped!
I have had success running the voice pipelines in my docker container install. Once I added the whisper and piper container, I just added the Wyoming protocol for each service giving the address and port. I also used self signed certificates to allow the browser mic feature. Not having the esp device or analog phone that was the only way I could go.
Great video and thank you for sharing.
Good to know, thanks for sharing!
Would you mind posting links to the Whisper and Piper docker containers that you used. I see a few of them on my search, but unsure which might work ---Thanks
@@carltonwbrooks Sorry for the delay. I used the docker run command from the home assistant page thread year of the voice chapter 2. YT will not let me put links. But for both Whisper and Piper there is a link in the article for running in docker. I converted these run commands in my docker compose file and then went to integrations and added the wyoming protocol twice. Once for piper and once for whisper giving the local IP address of my docker stack and the Ports the container was running on. I also used mkcert to create self signed certificates so that my browser and phone could use the mic while on the local network. I hope this helps. If you need more detail, I could make a quick and dirty YT video on my channel.
Joe
@@MyHeap Thank you so much. I will work through it in the next few days. Appreciate the help.
@@MyHeap It looks like those docker images only work on Rhasspy. Am I missing something?
Champion stuff as alwaysss 🎉🎉
Pretty cool. Thanks for the upload.
Thanks for watching!
Guude (Hi) Lewis👋, thanks for this great video. Playing around with that is on my TODO list. Wishing you a great start into the new week 😊.
Thanks!! 🙏🏻
Thanks for the video.
Are whisper and viper available as docker containers? I mean for those who are not using HAS OS
Thanks for another informative and inspiring video. Can I ask if you plan on making a video about making our own esp32 voice operated system soon? I have all the parts ready to go. Thanks very much.
Excellent video, Lewis! The Year of Voice is flying by :)
Thanks Mike and thanks for all your hard work, really looking forward to the next chapter ;)
You'll see I made some comments about it not wanting to understand me, but I believe in the project. Thanks for the hard work!
Hey Lewis, lovin the content. Is there a way to use NS panels for voice input I’ve both original running esp and pro with HA app. Cheers
Hi, Lewis. Great video tutorial, like always. By any chance, can you please share your wiring diagram and code for DIY mic and speaker , which you mention in your video. I have esp-32 and amplifier like yours microphone with 3v,gnd,bclk,dout,lrcl and sel pins. Thank you in advanced.
Thanks again for this very learnfull video Lewis. However I am struggling to install a ssl certificate to get my mic on Android working. Would really appreciatie if you could help out. Assume more less experienced users could also learn from it.
all we are really missing now is a wake word and then we all have a great new ESP32 project to make up 🙂
FYI: Wake word works now, see the official guide (search google for "$13 voice assistant for Home Assistant").
I will be waiting for the option to use this with a voice trigger like, having to press a button kind of defeats the purpose of a voice assistant for me, since I can always use the app on my phone which is close to me most of the time. But it's great to see this progress.
BTW Lewis, can I use a microphone and speakers connect to the my HomeAssistant server (I use a desktop as a server), I saw the AI Thinker Audio Kit on Aliexpress it looks interesting would it work?
I have just started to try this. Using an esp home rigged esp32 board with a mic and amp. Still learning the set up but I have clicked a button and spoke to turn off/on a lamp. Side note... I can also send HA tts notifications to the doohickies amp/speaker. Still much to learn & break/unbreak.
hello, i don't found tutorial for DIY with esp32 esphome, do you write article on your blog for doing DIY with cables ? good video :)
Nice video, as always,please a video to create a diy push to talk with esp32, microphone and speaker Thankyou
Yes, a „real“ alternative to the atom would be appreciated. The phone thing is funny, but I can’t really see anyone walking to a local phone and take a call to switch on some lights.
Thx for the French highlight 😄😁
Thanks for the video. I have 1 problem that on mobile app assist voice control works only if you have https connection. Can you create video how to do https with local network? I don't want to open port on my router and i'm using tailscale for remote access. thank you
I wonder if you can use the Voice over Ip as a intercom as well via HASS
@Everything Smart Home speaking of languages how can i add Albaninan to it....obviously it is not off the shelf but i am willing to put some work into it..... is there somewhere where people add languages and basic phrases etc ....great video as always
Can this processing all be done on the little home assistant boxes, eg yellow or green? Is it possible / necessary to offload processing to a more powerful box?
At this stage is the Grandstream the only ATA supported by Home Assistant, or does it support other adaptors, like Sipura or Linksys ATA's ?
It doesn't seem like something that would be used outside of being a gimmick, but I am a geek and I like gimmicks lol :)
This is so cool! I did the config in my home assistant with a m5stack echo but I can only select english... It doesn't show the others languages. Can someone help me please?
Very interesting update
Do you know if we can wakeup the esp32 by keyword (live Jarvis) instead of a button?
Imagine having it with a battery pack in form of the communicator from StarTrek. But better would be a wake word detection. But im currently waiting for the amazon echo jailbreak. First impression is that there is just a simple android, which acts as base. Hope its fairly simple to put our own app on it and remove the other cloud stuff. That would be a great base for local voice hardware because it is cheap and easy.
You can get an ESP Muse Luxe right now. No wake words yet but eventually will get there.
@@thatsux0rz m5stack echo with latest fw from homeassostant can do wakeword.
I was configuring it together when I came across another OpenAI configuration in the integrations so I added it from its API and now I added a third Option which is probably Cloud but as an OpenAI conversation agent
I will be testing
Is there any video for the creation with esp32 and i2s and amplifier you say at 10nth minute ???. Thanks from Athens
Is Pi4 powerful enough to run whisper tiny 8 model with reasonable response time?
Any chance that using our existing Google Home with a user-defined activation word is theoretically possible later down the track?
I'm wondering if you could leave that analog phone on call and whenever you need to execute a command you just say for example: "turn on the lights", without any wake word like Alexa or Hey Google.
Finally locally run voice control.... I might start looking more into some smarthouse things then.
I was wondering though, do you need to have it push to talk? Cant I set a phrase like with the others, ans say like "Hey house" or something...
When we are the point of it being locally run, I don't mind so much it is listening in. With something like a google assistant, I could see the benefit in it only being allowed to listen when I push a button.
You mention that multiple pipelines can be set up if you want for example multiple languages. However from what I can see both Piper and Whisper only allow you to configure 1 language. How would a second pipeline be made with a different language?
Great video Lewis. Since I can use my iphone to issue Homeassistant voice commands within the HA app and I always seem to have my iphone (and apple watch) with me, it would be great if it could be made easier by initiate a HA command by for eaxample "raising the phone to talk" or some other simple means. Does anyone know if this is possible?
Thanks! I don't think it's possible at the moment
What hardware specs would you recommend for best voice performance?
The m5 stack echo look like it could be converted in to a combadge.. computer tea ..earl grey.. hot 🖖
Definitely going to be linking my asterisk server to my home assistant server this weekend
Enjoy!
Let me know how you configure that. I also have asterisk, but currently configured it the autodial way. I'd like to assign home assistant an extension instead but not sure how to do that.
Custom trunk?
@@mateuszdrab not managed to look at it yet but I think it will be likely a sip peer.
@@marksterling8286 no rush, I'll await your update and hope you figure it out 😉
Van you make a video about that diy esp32 microphone speaker thing? Can I use esp8266 for it?
Sure if enough people want to see it! No esp32 only I believe
Electret Microphone or max446 or MAX9814, since I already have some of these in a bag somewhere. Wonder what it would take to get them working.
Its like is 1999 again with local voice, push button microphones and analog phones.
😂
Great video. I´m missing your promised ESPHome config - also i´m not able to get the replies announced on the M5echo - any idea how to configure that? The speaker does work as audio-player...
By the way: my (german) family loves your accent - it´s scottish, correct?
Links added now sorry.
Yes you need to add a speaker component to the ESPHome config for the replies, check the ESPHome docs. It is indeed Scottish, good catch! :D
I would love to see a true replacement for the google home/alexa speakers, I rarely use them for audio, I think the esp32 could have great potential here. My vision would be esp32's that could simultaneously act as a voice assistant, multi room audio (with your choice of DSP, used with pre existing audio gear), bluetooth receiver for audio, room assistant, bluetooth proxy and maybe presence detection. Maybe pushing the little beast too far though.
Can't see why it won't happen - Amazon/Google have been basically doing it all for years and all those individual features are being improved in the HA world.
Your Assist pipeline processes super quickly. I may have missed it, what hardware do you have?
Thanks for great video. Just a question. Does the Atom Echo have also wake word capability? If not it seems to be pretty limited as if you need to get out from the couch and go to push a button you might as well use wall mounted control panel.
I mentioned during the video they are working on wake word, it's just very difficult and they are taking the time to implement it correctly
FYI: Wake word works now, see the official guide (search google for "$13 voice assistant for Home Assistant").
Hmm now I wonder if I can do this with my google aiy kit somehow. I had it setup with rhasspy home but even as a satellite it doesn't work the greatest. I am going to have to look into this as that is currently collecting dust since the mycroft project was scrapped.
When is the Everything Smart Home mic/speaker assistant board coming out?!
great vid
Thanks!
Thanks for the video do you have links to the stuff so I can look into buying/making a speaker?
Hopefully soon we won't need to to a push to wake and can just use a wake word like Alexa and Google speakers do
Sorry yes added it to the description now!
@@EverythingSmartHome Is that the right config for the esp32 DIY version? Appears to be for the m5-stack echo board, not the esp32 dev board. For example, it looked like you were using the boot pin for the wake which is pin 0 vs that config is pin 39 for us on the m5-stack.
I said on the video that I would leave a link to the M5Stack code so you can use it as an example for creating your own one, since there is no way for me to know which pins you will use 😅
wen two way audio with esp32/esphome, like intercom integration?
Request: How can we get your video's uploaded to some AI db. I find myself returning to them watching over and over after I get more experience in working through projects, but, still struggling to get them to work. This is at least my 3rd time on this one and I found that answer to many questions that came up since I first watched it. You do a GREAT job in the selection of words you choose to be specific and consistent to both the process and/or the item which you are describing. This makes it easier for AI to ID items in searches. Let me know how I can help. BG
Need a touch-free version. I wonder if wake word will be an option... I'm currently using Alexa with Node-RED to command everything in the home and it's the only thing that works for me and the family / visitors. Not perfect every time, but it's so seamless / ubiquitous and over last few years I've become so used to walking into a room with hands full and setting a lighting scene by number using voice, or turning off garden lights whilst locking the door or arming / disarming alarms or presence sensors. I'd love to move away from Amazon to something local though...
FYI: Wake word works now, see the official guide (search google for "$13 voice assistant for Home Assistant").
Anyone know the correct configuration in Opnsense to allow this? Was working fine on my old router.
My phones are "voice over ip" using OOMA service. Any idea what I need to do to use it?
Too geeky for me right now. I won't switch from using Alexa to HA for voice until 1/ Magic word is available 2/ There is a simple push button-less microphone device available that is easy to integrate into HA. But I like the ability to turn on/off each voice assistant option for each entity ..8.15 in...(for security and privacy reasons) Very cool. 😀
I'm sure you can do both of these things with automations.
If this can have a wake word and do Spotify whole home audio I'll set this up. Until then I'm stuck with my Bezos boxes
They said in the livestream that offline wake word detection is super hard.
@EverythingSmartHome
I see you have "Norwegian" in your list of languages. Is that only available for subscribers to Nabu Casa? Would be a bit strange since I though whisper did all the magic locally?
What about using mobile phone for voice and speaker
Is it possible to create how to create Automation using voice?
When you said "analog phone" I was like "oooh a rotary phone, that's cool!". But no I'm just old
I wasn't back at rotary phone - but I was surprised it was a cordless.
I found that older AMD CPUs don't support Whisper which is compiled to require SSE 4.1. My old Wyse 7010 thin client has the AMD G-T56N CPU only supporting SSE4a and can't run Whisper.
[13:56:18] INFO: Service exited with code 256 (by signal 4) 😥
Thanks for the video! Just waiting for the wake word support to ditch Alexa! I also noticed you have the music assistant integration, how did you get it on the newest version of HA?
FYI: Wake word works now, see the official guide (search google for "$13 voice assistant for Home Assistant").
Is it possible to use Android IP Webcam as a voice input ?
FYI: The M5 Stack Echo is no longer being manufactured, according to the M5Stack company.
Really? Got a link for that?
I can’t even find an M5 stack echo reference anywhere at all.
so i just configured 2 M5Stack Atom Echo's just now and i noticed that neither of them show the "controls" box showing the speaker icon but all other boxes are there and it seems to work. am i missing something without that??
I configured mine yesterday and have the same "issue" - like yours it doesn't seem to be affecting anything. Seems to me like it isn't configured as a Media Player - might be fixed in an update
Want to connect cloud voip services like 3cx to be able to make home assistant work as auto attendant
Sorry to be negative but, for a relative beginner to Home Assistant like me, setting up devices like the Atom Echo is a nightmare. After many hours of trying, I finally got mine set up without visible log errors. However, I still can't do anything approaching useful with it. In testing, all the pipelines are present and I can switch the Atom's red light on and off from HA. When I press its button, HA detects this but I see no blue or green light as the config says I should and no security notification. You say "if you want to use the microphone option in your browser then you will need HTTPs, but for everything else you don't". As setting up the secure connection has frustrated a lot of us, can you tell us if there's any *other* way to test the Atom? As far as I can see, HA has no interface for using or testing voice *other* than the browser.
I eventually had to give up for the sake of my mental health (after 5 days of effort!). The Atom Echo never worked, and eventually failed to connect altogether, showing encryption errors that persisted after a complete reinstall - a problem the developers say was fixed a month ago. However, I discovered that for me (and contrary to some reports) my phone *can* connect for voice access via the Companion app. Wish I'd known that before I started...
Was there supposed to be a link for the M5 stack? So many different kinds, not sure which to get. Do we look for an M5 stack echo?
M5Stack Echo is the one yes - didn't include link as out of stock pretty much everywhere
@@EverythingSmartHome thanks.
is there a way to create the pipeline with docker containers? I rul HA in a container on an rpi4 and I want to run wisper and piper on an intel nuc, is it possible?
Add-ons are just containers under the hood so yes 😅
@@EverythingSmartHome thanks for the reply. Sure thing, but I honstly don't know where to start. Any idea where I can find docker images for those?
@@EverythingSmartHome I actually figured that out and just made a quick guide about that on my channel, as it seems to be no tutorial out there so far. I'm sure you or @HomeAutomationGuy would do it better :)
@6:01 - Interesting profile pic for 'Sara' :)
Great info tho!
😂 she picked it I swear
Can a mic and speaker be added to the EP1?
In theory yes since it has 4 extra GPIO's which should be enough just
Can I connect the VOIP to the OpenAI integration instead of Assist?
Yes, just create a pipeline for that :)
Why's the wood chuck looking for weed? lol
Is there an Android app so you can initiate this voice control using your phone? (Like "Hey Google" works now)
I don't understand the HTTPS requirement of voice. HA is running on a local machine that can be accessed remotely via a Wireguard VPN. Why won't voice work when I access my HA instance from within my LAN?
How do I get HTTPS working locally? Can I serve HA via port 443 instead of 8123?
It's a browser limitation; can only access the microphone over HTTPS.
Easiest way to get it running, IMO, is a Cloudflare tunnel via Cloudflared.
self signed certificate with the signer trusted by your browser
Couldn't you wire a Grandstream box to your telephone demarcation box and run all your phones with it? (Does anyone actually own a telephone? How many people had to purchase a landline telephone to do this setup??) Maybe power would be a problem? You could probably run 2 boxes if you rewire the phone jacks or get multi line phones.
The Scottish guy selecting "American English" 😂
Can you use a standard IP phone instead of the Grandstream?
Yes but it needs to support the OPUS codec
I got this working with FreePBX as an extension. I just tested disabling OPUS on the handset I was calling from and FreePBX was able to transcode to OPUS codec.
Did anyone try the VoIP thing with a Yealink VoIP Phone?
I'm thinking we have to get a red rotary dial phone now, in which we install an LED that can be triggered to flash, just like in the campy Batman series from the 60s
I’m on a similar idea, but I want to connect it to OpenAI’s integration instead of Assist, and split out the phone audio to mix in with my mic at my desk. That way I can do a bit on a call and people can hear the response.
The phone I wanted to use for this video would have been so cool but unfortunately didn't arrive in time
Do you know if it's possible to configure it in Spanish?
Just to clarify something in the vid - It is not correct that you can't use the microphone in the browser without a certificate for devices on your local network. There is s setting in Chrome developer settings to enable it for local, insecure, sites.
That sounds great but why not just start with a voip phone and also give it a working local pbx to make call to rooms or outbound
I'm sure it's been said before, but I'm hoping to to get a pipeline from each of the tablet dashboards in my house. That will allow me to complete replace Alexa/Google.
Beyond that, with the rise of open source "AI" models, I'd love to see an AI HAOS add-on or integration, so I can pretend that there's a real intelligence behind my home automation.
There is one already. It is briefed by smart home junkie in his recent video
This is amazing I am 100% gonna make this.
Luckily I have some old speakers laying around. that I could repurpose for this
Press a button to talk to a thing to ask it to turn a device on that you used to have to press a button to turn! Hmm.
not using HTTPS even blocks the android app from being able to use the microphone - I wouldn't mind, but it's not like they make it easy to enable HTTPS (having broken my HA install on 4 separate attempts at doing just that)
:(
Its easy to do but HTTPS and certificates in general can be difficult if you've never done them before
@@EverythingSmartHome i've done them lots (I've worked in IT for 23 years - as a server and infrastructure engineer - so I know my way around a computer) - but not on Linux systems. and when you want to use a cert from your local CA, it becomes almost impenetrable.
I took 6 attempts to work out how to setup my Ubunutu Nextcloud to use an internal SSL cert - as none of the information available is actually reliable. and much of it assumes in-depth linux knowledge.
I will give it another go - but I am not holding out much hope that I will be able to get it working.
Just look at the Let's Encrypt add-on alongside the NGINX Home Assistant SSL proxy add-on. Should take you all of 10 minutes to set up if you've given it a proper URL. I just have mine go over Cloudflare DNS. Haven't had an issue with it since I set it up years ago.
On RPi4, cannot select faster-whisper... It is greyed-out. I'm in french Canadian...
Is the add-on running?
@@EverythingSmartHome yes
I love home assistant, but honestly I was really disappointed when they announced 2023 was going to be the “year of voice” and this update doesn’t seem to change the outlook much
It’s not their fault, but they just do not have the resources to do voice *well*. I’m sure this isn’t the final form, but as it stands:
- It’s a pain to set up.
- The “high quality” voice models still sound incredibly robotic.
- You have to talk to it like a robot to be understood, and detection is spotty
- It doesn’t really support natural speech
- It’s slower
- No wake word (also probably going to be impractical to get it working super well with the slow/spotty speech detection)
- You have to mostly DIY your own hardware to even use it.
They will keep improving things, but not all of those problems can be solved with just more time. They don’t have the massive army of servers to throw at natural language models, or the budget, manufacturing capacity, or volume to put out a smart speaker even remotely competitive with what’s already on the market, especially not at a similar price.
I just don’t see this ever being more than an extremely rough imitation of what’s already out there for only the most die-hard of “no cloud, ever” users.
HA should have called their voice assistant "Home-A", like Homer from the Simpsons. Them we could use a Homer voice for the assistant. It would be epic!
So I am running a Yellow, and I have tried using the local option and it is utter trash. I get it is going to take a while longer to process but the time is way way to long and it never seems to understand. When I switch over and use the cloud option w/ Nabu Casa, using the exact same prompts it is instant. to bad, I really hope the HA guys can get the local option to work better.. :(
My experience pretty much the same. Even using the best model still can't understand basic words like room for example in my language (Hungarian), and it also takes 10+ seconds to process it (4 core VM on an AMD Ryzen 5 3600). Lower precision models are even worst it spits out random letters basically.
Exactly. Watch the processor usage go up when using piper.. followed by..Sorry, I could not understand that.
Are you guys all using languages other than English? Wonder if it's languages that haven't been completed yet or need a little more work. Looking at the list, Hungarian does not appear so it would seem that that one at least needs more support from the community?
@@EverythingSmartHome I am in TX, so I guess you could say I speak a form of English....😆
I tried with german and got a similar bad experience. After ~10 tries I was able to turn on a light. After that I gave up...