How to Code a Machine Learning Lip Reading App with Python Tensorflow and Streamlit
HTML-код
- Опубликовано: 24 фев 2023
- Get notified of the free Python course on the home page at www.coursesfromnick.com
Sign up for the Full Stack course here and use RUclips50 to get 50% off:
www.coursesfromnick.com/bundl...
Hopefully you enjoyed this video.
💼 Find AWESOME ML Jobs: www.jobsfromnick.com
Get the Code: github.com/nicknochnack/LipNet
Links:
CTC Blog Post: distill.pub/2017/ctc
Oh, and don't forget to connect with me!
LinkedIn: bit.ly/324Epgo
Facebook: bit.ly/3mB1sZD
GitHub: bit.ly/3mDJllD
Patreon: bit.ly/2OCn3UW
Join the Discussion on Discord: bit.ly/3dQiZsV
Happy coding!
Nick 
Наука
Thank you so much Nic. This was mind blowing!! 🔥🔥
Phenomenal job and we appreciate the consistency of high quality tutorials. 🙏🏾
Thanks a mil Jabulani!! 🙏
Videos like these, where you transform a model into an app, are 🔥! First, we learn how to build the model, and secondly, we get to take that model and use it to create an application. Please keep those ideas coming!
Do you have running model
I know 0 about coding but man this man is INFECTIOUS. 🎉🎉🎉🎉 I keep coming back for more lol
don't know if u still remember me, but two years ago I got an offer majoring in DS, was very unsure if I should pursue, I asked you, you replied, told me definitely, and gave me some advice. And I have been watching your channel throughout my school time.
Graduated in Dec. Today I just received my degree on paper. Thanks.
CONGRATULATIONS!!!!!! All that hard work paid off 🙌🏼
Maybe the best ML video in RUclips
My guy Nich keeps elevating the level, amazing job. I see the new edits also 😅👌👌
😅 phew, editing took like three days but I dropped so many new things in it!! Thanks a mil @Miguel!!
Well done Nic, we're waiting for another project
🙏🙏🙏
Thanks for the amazing video Nic! I've primarily been using Tensorflow in my work but am curious to learn PyTorch too. Would it be possible for a video implementing a model in both libraries side by side so that we can visualise at a glance the differences between the implementations?
thank you so much! high quality next level teaching and awesome prep for my TensorFlow Certificate exam which I will take. As well for my resume! thx Nic 🤩
hi tobiasm
i want to enqire abt this project
Thanks Nic for creating such a wonderful video. Look forward to the video on the LLama model.
🙏🙏 cheers Priyam!
Awesome Nic! Can't wait to complete this along with you!
Also not sure If you just saw Meta released their LLaMa paper, and releasing that model soon. I know its not feasible to recreate coding that but would it be possible to pull the pretrained model and run it locally (similar to stable diffusion or bloom)? Also apparently the model is small enough to run on a single GPU, well at least that's what the paper claims. I still haven't finished reading through it but it seems like they might open source it...
Got it on my list to look at this week. I’m getting up to speed on LLMs as we speak so should have something in that space soon!!
@@NicholasRenotte Thanks! Yea a LLM project would be pretty fun to follow along with you!!
You're insanely brilliant!
🙏🙏🙏
Waiting for this!!!🔥🔥🔥
Ayyy, thanks so much for checking it out!
@@NicholasRenottecould you make a tutorial for live feed from camera. I'm waiting for it.
Hi again Nicholas. Thank you so much for the split payment method. Is it on purpose that the price of the 3 months split is going to at up at 800$ instead of the original one-time 697$ Price?
Hi Nicholas! I really love your videos, you have something in you that makes me feel comfortable and friendly. Could you please do a video of an AI detecting floor and changing it to different products or can you just give me some advice on this. BTW I got a client for this so if you could help me I would really appreciate it.
Fantastic as ever.
Are you using the streamlit defaults for styling?
Cheers Jon! Yep, all defaults are the same bar changing the layout to wide.
Hi Nicholas, Is there a way this can be changed so it can work on live videos? if you can do a video that will be great.
Thank you so much sir.
Can you tell me how did you calculate the alignments for the model?
More so, if we want to have our own video, what should we do ?
Can you do a video about the laptop you use? What are your laptop's specifications?
This is great , can this work on random video or webcam ?
Thanks Nic wonderful tutorial again ! 🎉🎉🎉
Is there a way doing this with our own videos input by extracting the face with mediapipe for example?
Yep!! Could use mediapipe to mask out the mouth and make detections from that. Once I get some time I want to get that done!
I have a challenge for you Nic my man (Btw, thanks a ton for your great videos, I'm learning a lot from you) :)
Make a CNN classifier that specializes in 2 classes of grayscale images. But not just any grayscale images, they have to be a perfect circle of white with black around (good objects), and the other class is the same, but it has small imperfections in the white circle, like a small line of gray, or maybe the spraycan tool from paint.
Can you please upload a tutorial on that like how to fine tune existing model with some random dataset so that it performs well irrespective of the video we use for prediction. Maybe real time prediction from the web camera. That would be helpful, Thanks.
hey man i wanted to know that can i use this model to make prediction on other videos which are around 60 seconds ?
will it output if the video has no sound but only movement of lips?
Hey Nick, can you consider to make a real-time lip reading model ?
Thank you very much very much 🥰
Anytime!! Glad you liked it!
I want to make a face image dataset for my Final year research. And I downloaded faces from google images by using an image downloading chrome extension. I want to know weather is it ok to use those dataset or else do I need ethical clearance on this. Can you please share your thoughts?
Hey nick, I was wondering, how can the distance of an object be measured simply with the use of object detection ?
Do we have to use the rescaling of bounding box around that object to measure the distance? I don't want to use any sensors for distance measurement. Simply camera,
Any other suggestions would be great too
it is difficult brother, idk exactly if any high level libraries are available to approximate the distance. But sensors are the way to go for accurate measurements. LIDAR or Ultrasonic sensors works perfect. But have to mention, Elon actually worked on this. He negated the entire sensor based approach and built image based measurement systems from scratch. Do look into it for any ideas...
You are a legend!!!!!
Nan Nic! Awesome!
I just want to know something, like I'm 14 and finding project ideas is like super hard, and seeing such amazing projects are amazing.
And 2nd, how do you work though a project...meaning from idea to deployment? Would highly appreciate your response.
Damn you’re 14 and already getting into this. Amazing work!! Ideas sort of come from everywhere; colleagues, RUclips comments, research papers and clients.
For the second q: one step at a time! The CRISP-DM framework helps clarify the steps required :)
@@NicholasRenotte PS. Looks good. Sent you a DM on Linkedin. Do have a look!
Hi Nick can we have some projects related to NLP too?
hi sir , actually i would like to add an option based pop windows ,like you have window so that should show the option of " i want to eat" or "i want to go out for walk" like this ,so is that possible to add it in this , if possible please help me out
Hey Nick,
Could you put custom videos to check the model's accuracy?
Also, these are the errors when I put a custom video :
1. For one of the videos I got this -> ValueError: Attempt to convert a value (None) with an unsupported type () to a Tensor.
2. For the other one, this -> in assert_input_compatibility raise ValueError( ValueError: Input 0 of layer "sequential" is incompatible with the layer: expected shape=(None, 75, 46, 140, 1), found shape=(None, 61, 46, 140, 1)
it means that the video had only 61 frames, while the model was expecting 75
Can someone pls give an idea on how to take inputs from our webcam and feed that to the model to get the lip reading output?
Hi, great video!
Can you build a SaaS Startup with Streamlit? Or do you need JS? Thanks!
Would need to get into Js eventually for things like auth, teams and payments.
@@NicholasRenotte Thanks! Keep up the great content 💪
Hey nick how do you think we can use the webcam for taking video input?
and also can you make a tutorial on multiple activities recognition. Thank you
you could use opencv for webcam
Can you please make a video for multiple objects detection (like SSD or yolo) but from scratch not transfer learning (you've made a similar video but with transfer learning using tensorflow model zoo). There is not much out there on how to build it except for a book (dive into Deep learning) which is very difficult to follow
Hello Nic! I am 16 years old, And I had a question that I was not able to find anywhere. The questions is that I have trained MNIST model as well as fashion mnist model. But when I am working with yolov5, i was curious on how to create models like yolov5 from scratch in PyTorch. They have bounding box, classification as well.
I did a basic implementation with Tensorflow, I think it was around Face Detection. Should be up on the channel!!
I have try to run the code, i want try with a video. but this isn't working. Isn"t compatible .... a problem with format of image or something like this... And I don't understand how to have alignements (??)
Bro where can I contact you
If there is no sound can it'll predict the sentence?
Can we try this with any other video without having video alignment file???
Dose tenserflow certificate helps getting a job to newbies?
Outstanding! But what happened with custom videos? Can we at least record a video saying equal sentences from the trainning videos and get a good result? I tried but the code ask me for the align file... Thanks!
Next project idea ! Noise cancellation??
while accessing dataset Iam getting this error : Cannot retrieve the public link of the file. You may need to change
the permission to 'Anyone with the link', or have had many accesses
can some one give me any idea on which tool to be used to align the video to get the same data like video.align like in the video
I am sure that at least a big tech will watch this ....
👀
can u make a tutorial on how to give user input directly from web cam????
great and it works fine 😇
Nick love u man how do u even do this please tell us the way u understand the paper and implement it ❤️❤️
Love from india ❤️❤️
Love that you enjoyed it 🙏
How to create checkpoint , or where we will get this checkpoint file
Nice video bro, Can you develop a ML model that take input as a voice and determine the person is under pressure or not under pressure and give suggestions also😊
Iam unable to access the dataset , can you help me with it.
hey nick can you tell me its algorithm and architecture
Can you mix all lip reading and object detection at one for deaf and blind people
Data set?
Do software engineers report to product managers?
How did you find the mouth coordintes? Can anyone explain?
I need to add a option of translating it to 2 different languages, how can i do that, i need help
does it work on other languages than english?
Hey can you make a video on realtime or change data set with our own video it will be very grateful
can we make it realtime with webcam
can you please make a project for that and video
Nic! can I use this model to predict a video without alignments?
i tried that and it didnt work.
Please work on the topic of MTCNN, FaceNet
For some videos i got decompression bomb error some video getting crt predictions anyone help
Hey how the videos is getting change by the chossing another video, isn't that hard coded for test_video4.mp4 only???? Please tell me
We pick up the video from the s1 folder then convert it to test_video.mp4. So we're actually converting on the fly and only reading the converted video!!
@@NicholasRenotte thanks Nic, I don't know why it's not working for me, may be the reason is of ffmpeg is not recognized as internal or external command....every thing is working fine just the video is not updating.
@@SaurabhSingh-oi5ev might need to install ffmpeg. Check this out: phoenixnap.com/kb/ffmpeg-windows
would it be better to retrain the model on the other datasets?
Yep! I didn't bother because I eventually want to create my own dataset and train it on other words and people!
What's your pc mate?
How can be grep real world object by his image with robotics arm in tensorflow
hey nicholas please made video on kaggle
Can you make it for realtime lip reading tutorials
I have a doubt, like all the videos that you are using to test the model are already used to train the model so thats why u get good predictions. What if we use a video to test the model that the model has never seen before? Will we still get good accuracy?
You are spot on. It makes sense that they would perform well on data similar to mine as that's the sample that the model has seen during training. For this to perform well in another use case you would need to fine tune on that data.
@@NicholasRenotte Can you please upload a tutorial on that like how to fine tune it so that it performs well irrespective of the video we use for prediction. Maybe real time prediction from the web camera. That would be helpful, Thanks.
can we upload our own video to get the text?pls someone can help me??
Create a video about fine tunning chat neo plzzz
It's a complete pain in the ass loading the 'latest' tensor flow GPU version on windows 11.
And as a solution if we use containers opencv starts throwing tantrums for video files.
So yes, please to everything PYTORCH
Oh, I didn't know. Don't know why they keep making it so hard. Anyway, I think that'll be the gameplan soon!
yo bro but what if i want it to be in real time by using my webcam
Will this model work for a custom video ?
Have you tired model for custom video?
Can it do lip reading like Spanish?
Guys, anyone tried running locally ? How is it performing in other videos ??
wowowooww
Can i upload my own video? Will it read my lip?
coooooooool
Can I do this when I know absolutely nothing about coding
Please finish Forza
Dave, I’m afraid I can’t do that
🤖🤖🤖
plz do it for webcam too..
Is the project working for you?
Great Now I have the same abilities as a Russian spy who can read lips
LOL, skillset enhanced!
Android App Tensorflow Object Detection Pleaseeee 😭😭😭😭😭😭😭
But is this more accurate and efficient than speech-to-text 🙃
raise ValueError(
ValueError: File format not supported: filepath=E:/BackUp/Desktop/LipNet_Final/LipNet/models - checkpoint 50/models/checkpoint. Keras 3 only supports V3 `.keras` and `.weights.h5` files, or legacy V1/V2 `.h5` files.
thats what I get when I load the checkpoints
Do we have checkpoints in .h5 format uploaded?
Any alternative to this?
Hey did you find solution to this?
Anyone found the solution?
SOLUTION