Real Time AI Face Landmark Detection in 20 Minutes with Tensorflow.JS and React
HTML-код
- Опубликовано: 19 сен 2020
- Ever wondered how snapchat filters work?
Three words...facial landmark recognition!
Facial landmark recognition allows you to detect a number of different points on your face that together make up your eyes, mouth, ears, nose and so on. From there you're able to apply overlays aka filters to get your snap! But there's so much more that it can be used for like emotion analysis and face tracking.
Sounds a little difficult though, right? Well, in this video you'll learn how to build your own face landmark detector app in 20 minutes using Javascript and Tensorflow.JS.
In this video you'll learn how to:
1. Install the Tensorflow.JS facemesh model
2. Build a Javascript React.JS face landmark detection app
3. Detecting landmarks on your face in real time using your webcam
Resources:
Tensorflow.Js models: www.tensorflow.org/js/models
HTML Canvas: developer.mozilla.org/en-US/d...
Create React App: reactjs.org/docs/create-a-new...
GitHub: github.com/nicknochnack/Facia...
Oh, and don't forget to connect with me!
LinkedIn: / nicholasrenotte
Facebook: / nickrenotte
GitHub: github.com/nicknochnack
Happy coding!
Nick
P.s. Let me know how you go and drop a comment if you need a hand!
Music by Epidemic Sounds 
Наука
One of the best Facemesh tutorials I've seen so far online. Keep up the good work.
YESSS! Thanks so much @Thang!
why dont you have more subs/views? youre amazing at explaining your code and actually seem passionate unlike most coding tutorial creators. keep going!
Thanks soo much, I've only been going at it since May so we're only just getting started! But plenty more to come!!!
I haven't even started watching and can already tell you deserve more views.
🙏 thanks so much @Rav Ada! Getting there!
Oh yes!
Best 30 minutes spent in my life
🙏🙏🙏 thanks so much @Alfredo!
You deserve more subs dude. I'm on your team now
Hell yeah!! Welcome to the team @Nico Leiva!
WHOA YOU WILL RECEIVED ONE MILLIONS OF VIEWS THAT AMAZING CONTENT RIGHT HERE
Hell yeah, that's the goal, thanks so much 😊!!
Absolutely great tutorial...And believe me, you will get dozens of subscriber soon..
All the Best!
Heya @Sufyan Khan, thanks a million! More tutorials to come 👨💻!
Thanks a lot man! looking for more quality content like these
Ayyy, cheers! Thanks so much!
best tutorial ever
Thanks so much @mustafa! Definitely, in the works!
You are AWESOME!!! Your tutorials are easy to follow and I love how you break down the concepts as well.
As a point of feedback, I would love if you could mention some possible applications for noobs.
Thanks again,
Subscriber!
Solid suggestion, thanks @Tony!
Great content man, thank you !!!
As a backend dev, it wasn't difficult to understand the front-end part.
I like your teaching style. 👍
Thanks a bunch! So glad you enjoyed it.
Bro thanks very much for all these tutorials.
Best Face Landmark Detection + Coding Tutorial that I've seen!
Thanks so much @Timothy! So glad you enjoyed it. Same Timothy from LinkedIn?
@@NicholasRenotte Yes! :) It's me. I just sent you another epic video idea.
@@TagenigmaLLC yesss, alrighty will check it out!
This was super helpful thank you very much
Dope tutorial, keep it going :)
Thanks so much @Diego Caumont! A bunch more coming. 😄
This is an amazing video, what could make it even better, is possible applications for the landmarks, for example, detecting certain dots, and based on it's movements, detect micro-expressions, that could be interesting
Thank you friend for sharing the teachings learned here will be of great help for my projects
great stuff man!
Thanks so much @scott!
Love from India.... Hoping one day I can also figure it out by myself...... Ur videos are awesome..... Explained well..
Awesome! Let me know if you need anything along the way :)!
Thanks so much for your effort!
Plenty more to come @Panda! 🙏
Amazing tutorials!
Thanks so much @Jerry LEE 🙌 !
Really awesome video.
Thanks so much @Vijay!
Thank you to make me gradueated
Found your channel today! Great content!
🙏 Thanks so much @Yona, anything you'd like to see?
@@NicholasRenotte Yeah! I would love to be able to work with the detected landmark points and trigger certain areas (mouth, eyes). Something like gaze recognition or emotion detection.
@@yonakislinger6301 awesome stuff, was just planning iris detection today!
@@NicholasRenotte Amazing man! Really cool what and especially how you create!
@@yonakislinger6301 thanks so much, honestly appreciate it a ton 🙏!
Thanks for sharing!
I try your GitHub repo and build in 1 command " npm i " and it's working so amazing :)
Awesome!! Glad you got it running so easily, YESSS!
love your videos
Thanks so much @Mantas!
Looking forward to converting this to Angular!
Yess, let me know when you do, would love to take a look at the code @Libby!
wow sir, i love the Beginning ||||
Good tutorial! ❤️👍🏻
Thanks so much @Tennyson 🙏🙏!
You're the bestt!!!!
nice, thanks for your videos, i will learn
i from indonesia
The key points for me was to switch to video.videoWidth instead of video.offsetWidth & second one is resetting canvas width and height for each loop. Thanks for the simple learning
Anytime! Awesom suggestions.
I would love to see a continuation of this project. Maybe create a filters of your own? Rabbit ears, whiskers, etc...
Yah, working on it as we speak. Trying to get the hang of 3d rendering for it!
@@NicholasRenotte If you have a patreon, buy me coffee.. or any sort of donation links I will personally donate 100 dollars if you cover how to make a snapchat like camera experience in terms of putting filters on ones face.
Heya @@haibertbarfian6343, I have a Patreon but no need to donate for this particular use case. Got it on the list already, what types of filters are you thinking of? Background removal, adding sunglasses etc?
@@NicholasRenotte You are a SAINT MY FRIEND!!!! I was thinking of a snapchat like experience where you can put sunglasses, and all sorts of fun filters on a face using react native. Many people have joined the RN community and we need to show the prowess that RN has obtained over the years.. if snapchat filters are possible in RN.. almost anything is! I know that currently there are SDKs that allow this functionality on WEB.. but no one has done it on RN for mobile apps... this is going to be bananas.
@@haibertbarfian6343 hell yesss! Definitely, I'm going to get to work on it soon!
Love it!
🙏 thanks so much @Laura Estupiñan!
Amazing thanks for such great videos, Can you please tell me how to just extract eyes coordinates
Great !
Thank you so mush Nicholas, this is a great content, so much valuable. I have just one confusion that why the Triangulation matrix is used and how did it formed ?. Again thank you so much.
Canva has a delay in relation to the video component. Hide the video and add the camera image to the canva, along with the points on the face. Congratulations on the video!
Awesome suggestion @Diego!
Awesome! Great tutorial. Are you planning for more tutorials using Posenet and React JS? (Using Teachable Machine)
Thanks so much @Haesun Shim, I sure am! I've done a Posenet basic video but have a few others planned with Tensorflow.JS as well: ruclips.net/video/PyxsziqatFE/видео.html
I encounter error
Hello! I loved your video, it was very helpful and easy to understand. Sorry for the inconvenience but would there be some way to recognize the face of a specific person and allow only this person to log in? Like in the phone?? Thank you very much for everything
Excelent tutorial as usual. Do you have a tip about how to implement liveness in face detection?
are you able to implement
I was wondering if you can make a separate video on who to map the detected face landmarks on a 2D/ 3D avatar
Working on it @Sarina, definitely planned!
Your tutorial is very good and I understand it completely. Thank you so much for sharing.
How to use landmark coordinate point to generate animation avatar?
In other words, how to use landmark coordinates to compare with virtual components such as eyebrows, and then select a matching eyebrow for image assembly;
Working on this atm :) will share a tutorial once I'm done!
@@NicholasRenotte Is the tutorial out yet?
hey Nicholas! this is so cool! do you think I can use this on a tiktok clone I already built using dart?
Great work Nicholas. But, there is a typo on line 2673 of utilities.js: 'TRIANGULATION[i * 3 + 2],u' doesn't need the 'u', so guessing a typo. Really enjoyed though - thank you.
Thanks so much @David, ah i'll update the repo, I think I might have thrown it in just before committing to GH.
Thanks for sharing bro, btw how to get blink eye, node, turn head etc (gesture face)
thanks before
This is amazing! There is no video out there showing an actual implementation of face filters using react native. Do you think you would be able to pull this off? I believe that is a highly coveted topic and there are absolutely 0 videos on it out there.
😉 stay tuned!
@@NicholasRenotte github.com/haibert/graphics-project I want to share my attempt at this with you incase it helps or gives you any ideas.. I have used the expo-face-detector to get a 2D X Y coordinate of the nose and map that to the rotation of this 3D box I have generated on top of the camera using the libraries mentioned below. It is by no means near what it needs to be.. because I only have 2D data using expo face detector... but with the face landmarks you should be able to do much more interesting things. I'M SO STOKED TO SEE YOUR VIDEO!
"expo-gl": "~10.2.0",
"expo-graphics": "^2.0.0",
"expo-three": "5.4.0", (the package versions REALLY matter)
github.com/haibert/graphics-project
🥴
Greate stuff :-)
Thanks so much!
Great video! You’re really great at explaining explaining everything but is it possible you could help me out? Because I’m getting stuck at the Load Facemesh part. After coding everything It’s throwing an error saying “load” is not a function…
Is there any way to export this data into 3D software like blender?
Not too sure about pushing out to blender but you could definitely export the x,y and z coords for use elsewhere? I've seen it used alongside three.js for 3d viz @KriGeta.
@@NicholasRenotte sounds intresting
Great Tutorial! Can you please come up with react-native example of same tensor flow model using expo-gl?
Ya, in the works @Yash, will let you know once it's out!
@@NicholasRenotte Thanks! Appreaciated
@@yashthakkar3453 anytime!!
Awesome work honestly you deserve much more recognition :D One question though when I cloned it and tried to run it it keeps crashing before the detection with an Unhandled Rejection (Error): method must be bilinear or nearest, but was undefined error :) thanks for your help
Ah, baby steps, the channel is growing thanks to all the love from you guys 😍!! Can you share your code? Where exactly is the error happening in the trace?
@@NicholasRenotte I just cloned the code from your repo and tried to start the react app :) (I installed all the prerequisites). Thank you fro your help
@@alaeddineyacoub6225 you cloned it directly and it's still crashing? Just checking is your webcam activating when you start the app?
this is awesome. i came here coz i have a digital image processing project and i was wondering how to do the same thing in python (_Coz i'm weak in python). now i'm wondering why tf use python when u can do the same in js.(coz my strong point is javaScript. xD).
Hahahah, the beauty of those libraries is that you can do it in either right! Thanks for checking it out @Vijay!
Loved the video it was awesome - do you know how you might remove the flickering of the mask? I'm trying to expand on this project to make a snapchat-like filter application and the flickering gets in the way.
Heya @Jacob, I haven't had a chance to fix it yet, I believe it's because the drawing is staged in two parts the dots and the lines. I'm working on a fix in the coming weeks though!
@@NicholasRenotte Thank you for the help!
@@jacobpiirsalu2862 anytime, will let you know once it's updated!!
awesome tutorial👍🏼 btw, do we have to use react webcam? how do we make it for phone user? I'm a junior student in IT department. I just figured out how to make phone app with unity (using C#).. and wonder how can I combine it with ai face detection
Heya @Rhena thanks so much! Nope you don't! I just found it easy to work with. Oooooh Unity is something I'm super interested in. Have you checked out Unity ML?
Hi Nick! I just completed your Media Pipe face mesh video and now i am here... one question is coming into my mind is ... can i build the same react app and integrate it with the MediaPipe one ?? please let me know if this is possible then i will try that integrating media pipe with this react app (I don't have much depth knowledge in react.js ). Also I feel media pipe is better than tf face mesh. Please let me know what are your answers/thoughts on this :)
Heya @Abhishek, yes you can definitely use medipipe on React in place of TF.js. I believe they're actually based on the same model so I think it's all about using what's easiest. I haven't tested out the js version yet but it's definitely on the list!
like video filter,is it possible to change image instead of drawing line ? in here only to show us how to detect face! to change line to image ! what is best way ? if y know , plz tell me know ㅠㅠ
Hi, first of all, great job explaining. But i have a question, where do you get the triangulation data? Keep it up!
The landmark map? I got it from the official TF Model repository. Need a link?
@@NicholasRenotte Thanks for this video. I'm new the space but I also don't understand the triangulation data. It's hardcoded data. Why is there hard coded triangle data in there?
nice video
Thanks so much @Ekonaldi Hutabarat, glad you enjoyed it!
Is there a way to choose either to display or not the predictions based on a "confidence" level of the predicition? I mean, when I get off camera the model keeps trying to predict if there are any faces and draws over the objects in front of the camera.
Hi this project was very interesting I only have one doubt as to how are u getting the TRIANGULATION metric data since I can't wrap my head around it if you can help or explain
Amazing Content man, spent a day trying to figure this out till I came across your video! Lastly, i was trying to use this model to use pre recorded videos. Do u have any resource/tips I could look into to change the input from webcam to pre recorded videos?
Heya @Tejas, does it have to be with .Js or are you comfy with Python as well? TBH it'll probs be easier to do with Python if you're wanting to work with pre-recorded vids. LMK I can chase it up for you!
Hey @@NicholasRenotte! Thank you for your response! I’m comfortable with Python but not sure how I would use it to implement this for pre recorded videos! And Yes! Can you please look into this/guide me?? Thanks a lot!!
@@tejaschoudhary3882 will add it to the list of videos. You should be able to do it with Media Pipe: google.github.io/mediapipe/solutions/face_mesh.html
@@NicholasRenotte Just had a look at it! The facemesh is implementable on Python but Iris tracking (separately on Media Pipe but included in the latest version of tensor flow) is unavailable on Python yet.
@@tejaschoudhary3882 ah got it!
Can you try with external usb connected camera plz?
nice
Very good trainer. I wanna make an easy reconigtion app, first I would load an image with the name on (text) from phone camera to the database. The name input as watermask or another way the image's file name. . then at another location, this app could detect this item (from camera) and output the name on the live image. React Native + Mongobase.
Can you make something about body detection? I wanna make a virtual try on
Hi Nicolas, you are great, I have a big ask for your, can you help me to create a good app by React and Face Detection functionality in your video for my graduated work in university
Hi, Thank you so much for this tutorial! Can you make a video explaining how to do this in Java and Android studio? thank you so much!
Ooooh, not really my expertise @Daniel but I'll throw it on the list!
@@NicholasRenotte Thanks any way!
@@danielrotnemer2564 anytime!!
Hey, great video! couldn't stop myself from implementing this! React-webcam is not working on localhost, do you have a way out? 🤔
Hmmm, any errors?
Wow
Hello! Thanks for the tutorial! What is your recommendation for the next steps to create a filter; such as distortion/displacement of the face or particles that attach to the mesh? I'm familiar with doing these things with a Kinect camera in Touchdesigner, but working in the browser is new territory for me. Thanks!
You could apply an viz overlay from the JS canvas inside of utilities.js! Where and how you adjust the canvas would be dependent on what points you want to attach the filter onto or from. Want me to make a video on this in the future?
@@NicholasRenotte yes!
@@nixcampbell5399 done, added to the list!
@@NicholasRenotte kindly,,,
@@NicholasRenotte Can you please share the link
how can we use this to make a blink detector/ face verification/ yawn detector
Great video Nicholas!
I want to link this model to a react native based app. ( Hoping to determine whether a driver is drowsy or not) Can you refer me a direction to look for on how to do it? I'm kinda new for these environments. Thanks.
Heya @Chris, awesome use case. I'm actually working on a RN solution as we speak, will let you know as soon as it's out!
@@NicholasRenotte Okay. looking forward to it. Cheers!
this is an amazing tutorial i just had to open my laptop and try it for myself! thank you so much. i have a question, i followed your tutorial, however the package @tensorflow-models/facemesh is no longer available. do you have any suggestions on how to resolve this issue? once again, thank you!
Heya @Putri, looks like the model name has changed from facemesh to face-landmarks-detection (github.com/tensorflow/tfjs-models/tree/master/face-landmarks-detection). Was looking into it over the last couple of days. There's three quick changes required to get it to work (plus I've updated the code here: github.com/nicknochnack/FacialLandmarkDetection):
1. Install the new package
// OLD Installation
npm install @tensorflow-models/facemesh
// NEW Installation
npm install @tensorflow-models/face-landmarks-detection
2. Change the import
// OLD Import
import * as facemesh from "@tensorflow-models/facemesh";
// NEW Import
import * as facemesh from "@tensorflow-models/face-landmarks-detection";
3. Change the load
// OLD Neural Network Load
const net = await facemesh.load({
inputResolution: { width: 640, height: 480 },
scale: 0.8,
});
// NEW Neural Network Load
const net = await facemesh.load(facemesh.SupportedPackages.mediapipeFacemesh);
4. Last but not least update the detection function
// OLD Detection
const face = await net.estimateFaces(video);
// NEW Detection
const face = await net.estimateFaces({input:video});
@@NicholasRenotte Thanks a lot for this update!
I can detect which face it is based on these points? Like save them in a database an use for face detection
You're awesome man!👍👍👍
Instead of using create-react-app, can I use react native instead for this code to work? And how do I add my face mask image on the detected area of my face?
Sure can, but I think the webcam module is slightly different if you're doing it on RN. Working on the overlay code now, will share once I've got it up and running @Benjamin!
@@NicholasRenotte Yes, I think instead of webcam, you'll use RNCamera. i will be waiting for the overlay video. Thanks for your kind response.
@@benjaminikwuagwu6469 brilliant! Will let you know as soon as it's out!
@@NicholasRenotte I really appreciate :)
@@benjaminikwuagwu6469 anytime!
thank you for your tutorial its help me for my task, but i get some error on const face = await estimateFaces(video); and this error like this estimateFaces is not defined. Can you help me to fix it?
i am curious if you could make a facial recognition using this face mesh?? hmm just like loading or storing dataset as 3d???
Which one is better py or javascript I am from the earth observation domain and I want to work on spatial using images for advanced applications in the Earth observation domain
I asked a question about this in your other video - but now that I looked at your latest push to your repo. Does the use of useEffect and RequestAnimationFrame prevent some of "lag" the previous implementation had? I used that with blaze pose in my code and it seems to have fixed any sense of lag which is great. I would love to know why this works if this is what fixed that issue. Thanks again for the content.
I was wondering the same thing
Can u help me for make tutorial liveness detection with face gesture like blink, nod, turn head etc?
We need more ML and AI
Hell yeah!!
@@NicholasRenotte haha
Awesome tutorial! btw, can you use the landmarks to get the actual measurement of your face? I'm trying to get the actual measurements of the eyes and nose something like that. its for my school project. TIA
Ya, I believe you could, will add it to the upcoming tutorials @Randy!
@@NicholasRenotte alright I will watch out for that, thanks for the response! thanks in advance for the tutorial
@@randycook6802 anytime, let me know if there's anything else you'd like to see. I'm always updating the video list!
@@NicholasRenotte i think you need a special webcam for that with 2 cameras
Is it possible to use this for facial expression recognition, with data point used for data science /machine learning?
can you make a face recognition program, please?
Yep! Siamese networks for recognition is coming up!
why you use react and not react-native for mobile ?
Great video Nicholas. However, I want to plot only the landmarks of the eyes and nose. Do you know their landmark id?
Oooh, I don't have them fully mapped out unfortunately. I think best way to do it is just try mapping out a few at a time.
Hi Nick, any idea how to implement custom Face Landmark Detection. example: only for lips, so i didn't need to track other landmark.
maybe can u make video for make this custom models :D :thumsup:
Heya @Andi, you can filter out what you show by isolating the keypoints. Although it might be easier to train a custom segmentation model for this!
Okai. Thank you Nick. Could you refer me to video how to make custom model for lips segmentation?🙏
Hey ,using this can i able to detect which is live face which is image or photo by using only react or java script
How To Convert Your Photo To Avatar Though Tensorflow Face Landmark Detector?
how did you get those triangulation metrics? did you calculate them or are those standard? I am completely new to this, I was able to follow all along till this point...
No problems, great question!! They're standard triangulation metrics for that specific facemesh model.
Amazing tutorial. Appreciate it. How can I use it for user login? I want to grant access to a certain page after positive face id. Any resources I can look up? I didn't find anything on that on RUclips, nor was I able to google it. Appreciate it.
Heya @Feruz Astanov, the technique that would allow you to do that is called Face Recognition. You can read more about it here: paperswithcode.com/task/face-recognition. Would you like me to do a video on it?
@@NicholasRenotte oh definitely ! Lol. I have a project to finish in 2 weeks, hope you can drop a video before then. Thank you so so much !
Hi, can you help me with text detection in an image using tensorflow js or opencv js
How can i training model , can you explain using python and open CV
how to fix the face mesh on the face as it moving too much
I'm trying to make an app where the user can try different shades of lipstick virtually.. I've found the annotations where it separates the lips out into 4 arrays, 3 of 11 and one of 10. Where I'm getting confused is how did you find the triangulation to draw the whole face mask? How would I map out drawing a texture for just the lips? Any advice or links to other tutorials would be greatly appreciated. It just doesn't seem to be well documented or I'm missing a keyword in my searches.
Yeah, I'm with you here, there isn't much out there on it. This isn't quite what you're looking for but it's a start: livecodestream.dev/post/detecting-face-features-and-applying-filters-with-javascript/ I've got some stuff planned for filters and textures planned!
Error:
export 'load' (imported as 'facemesh') was not found in '@tensorflow-models/face-landmarks-detection' (possible exports: SupportedModels, createDetector, util)
What should I do now ?