Last year, I saw your augmented reality lecture and found out that you only make very useful videos! I'm currently a RUclipsr who makes Android classes, and I'm still interested in your videos! Thank you for always making good videos.
You can improve using enhance image technique, and transfer learning to improve accuracy. The EIT will just stretch, rotate, drift and flip the images, and the transfer learning will help you to go at 95% accuracy
Very amazing work. You can also rig or animate the character to generate a larger dataset. For instance, while animating change the camera views and collect the data
Lol I had this exact problem with the bounds! I used basically the exact* same approach you did with the mesh sweep. Also, I think that's sweet how you tuck your laptop in.
You can use colab which has a GPU better than 2080Ti, just put all files in google drive. Colab comes with TensorFlow installed by default. Synthetic data on machine learning is definitely doable, there are some papers using blender to generate these training datasets. To address the poor results, by default all convolution-based networks tend to learn the texture of the image instead of the shape, maybe unity's texture is not that photo-realistic. It's very difficult to train a model from scratch, the common approach is to do a transfer learning, where you take a pre-trained model of a large dataset like imagenet, then unfreeze the top and bottom layer, tuning on your data. With 5000 images to train from scratch, you won't get good results unless it's a super simple classification, like 28x28 handwritten digits. Since it's synthetic data, might be better to just do a pixel-wise label instead of a bounding box label.
Amazing Amazing project. With outstanding informative video, I can't hardly thank you enough but still. Thank you very much for the helpful video. Would love to try this.
Idea: maybe try the 3d scan in a completely diffuse lighting setup. So you do not have hard shadows on the textures of your 3d scans. I think that could mess up the recognition of the shapes. Or did you try that already? Anyway , apart from the bulbosaur(?) (not a pokemon expert here) recognition seems quite good already.
I did try that I just didn’t film that part I still did get some shadows though it might have helped if I had a box with the correct lighting setup and then I could spin the object inside.
i use vuforia app to scan the 3d object and use it to create augmented reality app just like this one... i think it provide better experience and faster workflow..
Hey Matthew I had a question is there any possibility for hand detection in unity ? (without that paid plugin of OpenCV) I had a project in my mind where you could interact with models shown by vuforia's image target using bounding box coordinates and 3d model coordinates ... i have done some workaround by scanning the hand through openCv in an external script and then sending the coordinates as packets over UDP to unity every frame .... but the main problem is it is not feasible for mobile phones :(
You could have used Google colab, they give 12gb ram and GPU/TPU for free. You can keep it running even if you close the browser or shut down the computer
should have used a bunch of different pikachu and bulbasaur images instead of the same ones, then your algorithm would learn that they come in different forms, shapes and sizes
How do you draw the bounding box around the object it is detecting in-house object? I tried using renderer.bound but it does not work. Do you have samples?
Was recently asked to estimate the work need to make a mobile app that could detect specific (known) objects. I remembered watching this video a while back. Can some of this be combined with your previous ML video? Would more computer power be a partial solution to this? Would more lighting variations enrich your data set? Thanks so much for this video. You're my go-to guy for all those off the wall projects.
@@MatthewHallberg Good :D I thought for a while should I even ask... But for the geometry randomizing, you should definitely check Houdini. I've been using that for all kinds of synthetic ML data stuff.
Hi Matthew, I have been following your tutorials for quite a while. I’m trying to do object detection of real objects with an iPhone. I know iPhones, Unity and window don’t always work well but still want to give it a try. I can use photogrammetry for real objects but I’m not sure on how to do machine training once I have these images. Any suggestions would be appreciated!
prototype with 2k pictures... make sure the item you want to identify is 100% in your frame in your training pics. Don't put two items in your picture - the AI will potentially try to think there's supposed to be two; like lenses in glasses. check the the type of tensor flow model is right for what your trying to do.. some are very quick but have a lower success rate.
Would reccomend to just use some cloud instance instead of a laptop GPU as it will usually start to throttle after a while. Also, most of those frameworks works better on Linux due to the fact that virtually all the server do run on whatever linux distro was chosen by the user(usually Ubuntu or Debian like distros, sometimes Red-Hat based etc)
great stuff bro, i was searching to make something like wanna kicks app. Do you have any idea how to track the feet and put the shoes on legs like that? i know it's AR but i don't have any insights about how to do it :(
Am so confused when can I start learning about ML ... Do I need like probability theory and statistics class first after that intro to AI and read few books after that I can jump to ML What introductory courses would you recommend Side note : I’m coming from 2+ years with java and about few months with python and am learning flask now (just for the heck of it ) Do you think am ready to start doing stuff with tenser-flow or scikit maybe ??? And which one to start with ??
I literally have no idea on that one I never took any courses just started playing with tensorflow and following tutorials which is probably why I have no idea what I am doing in this video.
You have a very good machine learning course on youube by Andrew Ng. It's the ultimate starting point. You don't have to know a lot of probability stuff but know the basics like false positive, false negative etc...
I am new to the package of perception from Unity and my experience it's just for game designing, please if anyone has experience in such thing like this, I need help
Your model was overfit because you only had a few thousand images. Next time, print the summary from your model and take notice of the total number of parameters in the model. Then try to get 2 to 3 times that many image samples. You should get decent results then.
@@MatthewHallberg Look at the loss function via tensorboard. If your training data loss curve jumps to a high accuracy while your validation curve is way behind, you're probably overfitting
@@MatthewHallberg Really depends on your setup. But I think with tensorflow there is a callback in the keras library for early stopping and you just tell it to stop if its not seeing inprovement after so many steps and that can help avoid over fitting.
You don't neet 20000 epochs to train the model with such less data,.. it's over fits the model and the model can not predict all the labels with equally accuracy!!!, You have to train untill the loss is stable,.. thats it!! When your detecting every pichachu in the world,.. you need different types of pichachu with different background images,..... Don't worry about the huge data.., all you need is 1000 images each!!,.. and epochs not more than 500
@@MatthewHallberg yeah haha I watched it. I was sugesting that you use a pretrain model and retrain the last or last few layers to save many hours on training. thats how the watson and google object detection APIs work
That’s interesting never thought of that, I guess the only problem would be getting multiple objects in the same image cause that helped the model a lot.
Last year, I saw your augmented reality lecture and found out that you only make very useful videos! I'm currently a RUclipsr who makes Android classes, and I'm still interested in your videos! Thank you for always making good videos.
Thanks for reminding me why I dropped ML and focused 100% on gamedev (:
omg its the worst haha
You can improve using enhance image technique, and transfer learning to improve accuracy. The EIT will just stretch, rotate, drift and flip the images, and the transfer learning will help you to go at 95% accuracy
EIT?
Fact: there is not another RUclipsr out there as innovative as Hallberg.
Love you bro
sick man ! i was wondering about this process for a very long time. You just created the gate of the metaverse !
Very amazing work. You can also rig or animate the character to generate a larger dataset. For instance, while animating change the camera views and collect the data
Well done bro! Great work!
Thanks man!! It finally worked...somewhat lol.
Amazing!!!
Never seen your tutorials before and I usually do not Subscribe on RUclips but you just got a new Subscribe (and a fan)
Lol I had this exact problem with the bounds! I used basically the exact* same approach you did with the mesh sweep.
Also, I think that's sweet how you tuck your laptop in.
You can use colab which has a GPU better than 2080Ti, just put all files in google drive. Colab comes with TensorFlow installed by default.
Synthetic data on machine learning is definitely doable, there are some papers using blender to generate these training datasets.
To address the poor results, by default all convolution-based networks tend to learn the texture of the image instead of the shape, maybe unity's texture is not that photo-realistic. It's very difficult to train a model from scratch, the common approach is to do a transfer learning, where you take a pre-trained model of a large dataset like imagenet, then unfreeze the top and bottom layer, tuning on your data.
With 5000 images to train from scratch, you won't get good results unless it's a super simple classification, like 28x28 handwritten digits.
Since it's synthetic data, might be better to just do a pixel-wise label instead of a bounding box label.
Amazing Amazing project. With outstanding informative video, I can't hardly thank you enough but still. Thank you very much for the helpful video. Would love to try this.
" i have no idea what that is, And i don't even want to know " -
MatthewHallberg 2:30
.
.
.
.
Much love and support for you bro , such a great content
Great video ! What apps did you use to scan object to 3d models?!
Idea: maybe try the 3d scan in a completely diffuse lighting setup. So you do not have hard shadows on the textures of your 3d scans. I think that could mess up the recognition of the shapes. Or did you try that already? Anyway , apart from the bulbosaur(?) (not a pokemon expert here) recognition seems quite good already.
I did try that I just didn’t film that part I still did get some shadows though it might have helped if I had a box with the correct lighting setup and then I could spin the object inside.
Super!!
that kickflip
i use vuforia app to scan the 3d object and use it to create augmented reality app just like this one... i think it provide better experience and faster workflow..
It definitely will but my goal here was to create a generalized tracker that recognized more than one style of the same object... I just failed lol.
Hey Matthew I had a question is there any possibility for hand detection in unity ? (without that paid plugin of OpenCV) I had a project in my mind where you could interact with models shown by vuforia's image target using bounding box coordinates and 3d model coordinates ... i have done some workaround by scanning the hand through openCv in an external script and then sending the coordinates as packets over UDP to unity every frame .... but the main problem is it is not feasible for mobile phones :(
That is commitment. A+ for effort.
Always loved watching your videos❤
thanks!!
Your hard drive must be massive to hold all that data!
Haha MASSIVE. And you know what they say about guys with big hard drives...
You could have used Google colab, they give 12gb ram and GPU/TPU for free. You can keep it running even if you close the browser or shut down the computer
should have used a bunch of different pikachu and bulbasaur images instead of the same ones, then your algorithm would learn that they come in different forms, shapes and sizes
Impressed. Keep up the good work. Appreciate your insight.
Great video as usual!
I'm also planning to start ML sooner :)
thank you! Yeah do it so you can teach me lol.
@@MatthewHallberg guess what? I haven't started it yet lol.
Hamza Shafi haha me either
@@MatthewHallberg ayy bro
@@MatthewHallberg Hey what app u used for photogammetry(scanning 3d model)
What is the name of mobile 3d images app
@MathewHallberg Hey nice project.. what app did you use for photogrammetry? To make the model...please reply.
I also have this doubt
@@tharaniv6267 Have you even watched the full video? 11:16
@@abhishekgoyaldev thanks
How do you draw the bounding box around the object it is detecting in-house object? I tried using renderer.bound but it does not work. Do you have samples?
Was recently asked to estimate the work need to make a mobile app that could detect specific (known) objects. I remembered watching this video a while back.
Can some of this be combined with your previous ML video?
Would more computer power be a partial solution to this?
Would more lighting variations enrich your data set?
Thanks so much for this video. You're my go-to guy for all those off the wall projects.
sooo coool
very cool vid
You need to remove the green bbox outline from images and then it will work.
you should join the Two Minute Papers Discord!
Wondering how hard this would be using 2D Data instead of 3D Data
Kudos to ur efforts man! 💯💯
Thank you I tried lol, almost didn't make this video for sure.
Great video
First like and first comment broo ❤️😘
MY DUDE!
is it possible to feed ReferenceImageLibrary remotely ? like download the images from remote server with the 3D prefabs to place
Literally loved this .....😍😍
hard fan from sri lanka you AI man
Did you have the green borders in all your rendered training images (7:57)?
That was just for the video I didn’t actually train like that lol. I am dumb so that’s a valid question but not that dumb haha.
@@MatthewHallberg Good :D I thought for a while should I even ask... But for the geometry randomizing, you should definitely check Houdini. I've been using that for all kinds of synthetic ML data stuff.
@@maknien INTERESTING, checking that out now thanks.
Hi Matthew, I have been following your tutorials for quite a while. I’m trying to do object detection of real objects with an iPhone. I know iPhones, Unity and window don’t always work well but still want to give it a try. I can use photogrammetry for real objects but I’m not sure on how to do machine training once I have these images. Any suggestions would be appreciated!
Awesome!!
prototype with 2k pictures... make sure the item you want to identify is 100% in your frame in your training pics. Don't put two items in your picture - the AI will potentially try to think there's supposed to be two; like lenses in glasses. check the the type of tensor flow model is right for what your trying to do.. some are very quick but have a lower success rate.
Have you heard of fractional factorial testing? Might allow you to change multiple variables at once and run far fewer tests.
nice video with details.. thanks for showing.. whats ur laptop specs?
Would reccomend to just use some cloud instance instead of a laptop GPU as it will usually start to throttle after a while. Also, most of those frameworks works better on Linux due to the fact that virtually all the server do run on whatever linux distro was chosen by the user(usually Ubuntu or Debian like distros, sometimes Red-Hat based etc)
great video
Good job!
you should use for loop inside for loop inside for loop... instead of random
:D idk anything about ml too
great stuff bro, i was searching to make something like wanna kicks app. Do you have any idea how to track the feet and put the shoes on legs like that? i know it's AR but i don't have any insights about how to do it :(
Just FYI, Google has a new API named AutoML. It is much easier than their old Vision API which is hard to use. But it cost $$$$.
Hi Mattehw, How can I do to enter a augmented reality house but first put the house in a specific place and then scale it?
you are so great
Am so confused when can I start learning about ML ...
Do I need like probability theory and statistics class first after that intro to AI and read few books after that I can jump to ML
What introductory courses would you recommend
Side note : I’m coming from 2+ years with java and about few months with python and am learning flask now (just for the heck of it )
Do you think am ready to start doing stuff with tenser-flow or scikit maybe ???
And which one to start with ??
I literally have no idea on that one I never took any courses just started playing with tensorflow and following tutorials which is probably why I have no idea what I am doing in this video.
You have a very good machine learning course on youube by Andrew Ng. It's the ultimate starting point. You don't have to know a lot of probability stuff but know the basics like false positive, false negative etc...
I am new to the package of perception from Unity and my experience it's just for game designing, please if anyone has experience in such thing like this, I need help
How to place 3d model on detected object in unity3d with tensorflow.
dude, use Unity Perception package..........
Can someone please explain what type of deep learning method this process uses? is it CNN?
Yes
I wanna do keypoints annotations, anyone who could help me on that please???
one like just for ur Efforts! I can feel the pain.
I need help with the coronavirus ar app
Your model was overfit because you only had a few thousand images. Next time, print the summary from your model and take notice of the total number of parameters in the model. Then try to get 2 to 3 times that many image samples. You should get decent results then.
I think you overfitted to your training data which may be why the first model didn't work. Cant be sure though.
I was definitely getting that vibe but I don’t know how to tell for sure.
@@MatthewHallberg Look at the loss function via tensorboard. If your training data loss curve jumps to a high accuracy while your validation curve is way behind, you're probably overfitting
@@MatthewHallberg Really depends on your setup. But I think with tensorflow there is a callback in the keras library for early stopping and you just tell it to stop if its not seeing inprovement after so many steps and that can help avoid over fitting.
someone knows of any tutorial
You don't neet 20000 epochs to train the model with such less data,.. it's over fits the model and the model can not predict all the labels with equally accuracy!!!, You have to train untill the loss is stable,.. thats it!! When your detecting every pichachu in the world,.. you need different types of pichachu with different background images,..... Don't worry about the huge data.., all you need is 1000 images each!!,.. and epochs not more than 500
Add jarvis to the play store!!!!!
two words: transfer learning :0
Yeah this is transfer learning lol did you watch the video?
@@MatthewHallberg yeah haha I watched it. I was sugesting that you use a pretrain model and retrain the last or last few layers to save many hours on training. thats how the watson and google object detection APIs work
Why... 😑
Not what you wanted??
@@MatthewHallberg I hate machine learning 😅
Hahah same
You have to use a GPU to speed it up. Try training on Google Colab. Also, tensorflow sucks, use PyTorch :)
Great vid, too bad you have no real computer :-)
Haha yeah
Lol "It was so hard to figure out how to convert bounding boxes from world space to screen space...". Really?
Joseph Moulton I wish it was that simple lol Unity has a function for that.
@@MatthewHallberg Instead why don't you manipulate the camera to look at the model and then window the frame? Then take the snapshot
That’s interesting never thought of that, I guess the only problem would be getting multiple objects in the same image cause that helped the model a lot.
Hope some loaded dude gives u a quantum computer... You might even find a way to prevent future AI to wipe out humanity...