Loading in your own data - Deep Learning basics with Python, TensorFlow and Keras p.2
HTML-код
- Опубликовано: 22 май 2024
- Welcome to a tutorial where we'll be discussing how to load in our own outside datasets, which comes with all sorts of challenges!
First, we need a dataset. Let's grab the Dogs vs Cats dataset from Microsoft: www.microsoft.com/en-us/downl...
Text tutorials and sample code: pythonprogramming.net/loading...
Discord: / discord
Support the content: pythonprogramming.net/support...
Twitter: / sentdex
Facebook: / pythonprogramming.net
Twitch: / sentdex
G+: plus.google.com/+sentdex
Just wow , didn't wanted to watch the whole video , but your are a magnet !! Excellent style of teaching
Another amazing set of tutorials. You truly are helping me understand Python and Deep Learning at a whole different level. Thank you for your time and expertise, Sentdex.
5:04 I don't know why, but the way you casually said "ha, bluedog", and then continue on, was hilarious to me xD
Works like a charm. For those(beginners like me) who had issue in layers like "kernel size not defined". Just replace "(3,3 ) by kernel_size=3 " in layer 1 and 2 and it will be good to go.
I love the way how you personified the NN. The part with shuffling makes me laugh!
I keep Getting:
error: (-215:Assertion failed) !ssize.empty() in function 'cv::resize'
What does it mean, could not find anything on the web, help!
Here's a course you'll need.
Face Mask Detection Using Deep Learning . It's paid but it's worth it.
khadymschool.thinkific.com/courses/data-science-hands-on-covid-19-face-mask-detection-cnn-open-cv
These were videos that I requested. Please make more Project videos in Machine learning and deep learning videos and real-world machine learning projects in PYTHON because You Are The Best to learn from
This is so true. I've been so frustrated trying to learn this topic and a lot of videos are just people explaining what neural networks are, like the begging of the previous video in this series, but nobody actually gets into the code and explains how to set up functions. Like there is A HUGE DIFFERENCE BETWEEN DRAWING A NEURAL NETWORK AND ACTUALLY CODING ONE!!! ANYONE CAN DRAW ONE
So true ! Make more such project videos. They prove to be of great help.
Can someone tell me what X=[] and Y=[] is used for?
@@a.n.7338 Yes I do want to know
@@a.n.7338 That's to initialize it to an empty array. But at 15:30, he learned that it did not initialize the type. and that's something I especially like about his videos: he doesn't just edit out the mistakes, any talks about things like the need to reshape being kind of stupid.
Thank you so much for this video. As a programmer who just wants to start prototyping a simple model without a great DL background this video gave me the tools to get on with my work.
i started my python journey with you back in the university days, thanks for being there boss.
Hi Harrison.
You've been doing an absolutely amazing list of implementating Deep learning videos with Python, Tensorflow, Keras, etc.
This is the most useful job you've ever done. I've learned the Machine learning, Deep learning theory easily but implementation and application is something difficult to me. Keep doing this please.
Here's a course you'll need.
Face Mask Detection Using Deep Learning . It's paid but it's worth it.
khadymschool.thinkific.com/courses/data-science-hands-on-covid-19-face-mask-detection-cnn-open-cv
This series is so thorough and easy to understand (for me at least :D)!
I can't wait for the next part!
Great to hear!
THANK YOU SO MUCH!!! I just started with Machine Learning and Neural Networks and this video helped me a lot!!!
I like the teaching style, it's simple to understand
This is currently the best TensorFlow tutorial on youtube. Can't express how thankful I am, after wasting much time on crappy videos named TensorFlow in 10 mins, 5 mins, 1 min and etc...
i enjoyed this so much, i was from CS degree. But not have quite good moment with programming. So i decided to get job that not programming. But, since i was try to learn about pyautogui and selenium from your video, i was so exited to learn ML, and now here am i ... following your keras tutorial :D
You actually saved my life!!! Thank you so much for this amazing tutorial.
These vids always cheer me up :) You are by far my most favourite instructor. :) When I feel depressed i just watch your videos.
Nice to hear :)
amazing videos, great AI tutorials, honestly one of the best programming channels on RUclips. thank you for making these videos
Yeah, I also really enjoy his videos. He inspires me to make my own AI videos.
Heay sentdex pls keep up with your videos. They are really helpful in so many ways. Im just starting to get into ML and started studying Computer-Science just because of ML and your videos are so helpful. Thumbs up to you
A nice alternative to pickle is
np.save('features.npy',X) #saving
X=np.load('features.npy')#loading
Thanks for sharing!
thanks
Thanks
very helpful
thanks
You are very good in teaching and the world need you, Sir.
Epic moment: "haa, blue dog" @ 5:05
hahahahaa
i laugh so hard
Sentdex these are the best videos I have ever seen in Deep Learning. Amazing tutorials. You are the best at what you do. Why did it take me long to find this channel.
Welcome here :D
@@sentdex can I have your email if you don't mind?
@@sentdex Hello. I got stuck when instructing my directory on the file.
Kindly advise. Thank you.
Amazing video man... Looking forward to the next one
The value of these videos is fucking incredible. After some setup with anaconda to get tensorflow and python 3.6 to work in pycharm, i was able to reproduce all of this with my own data. Your explanations are absolutely on point and i have no questions left after this part.
Thanks, that's awesome to hear!
this video is amazing, I am so glad I found your channel. I have tried learning this stuff for quite a while now through other RUclips videos but nobody could explain it that well.
@Ruben can u share the code
@RubenUribe through via email if possible
Thanks @sentdex, just what I was looking for :)
This was the most helpful video I've found. thank you!
11:54 that's a great imitation of a model trying to learn ^^
Here's a course you'll need.
Face Mask Detection Using Deep Learning . It's paid but it's worth it.
khadymschool.thinkific.com/courses/data-science-hands-on-covid-19-face-mask-detection-cnn-open-cv
Brother, you're amazing. This video has been a huge help. Thanks.
Thanks you! Awesome video. Looking forward to the next one 👍👍👍
6:50 "the hand of a dog" - it is called a paw! hahaha
Great video, looking forward to see the next one.
For those who want to use RGB/color images, modify these lines!
Change these:
img_array = cv2.imread(os.path.join(path,img) ,cv2.IMREAD_GRAYSCALE)
plt.imshow(img_array, cmap='gray')
X = np.array(X).reshape(-1, IMG_SIZE, IMG_SIZE, 1)
To:
img_array = cv2.imread(os.path.join(path,img))
plt.imshow(img_array)
X = np.array(X).reshape(-1, IMG_SIZE, IMG_SIZE, 3)
And this should work, good luck!
you are the man
thanks
So far, so good! The first dog grayscale image was successfully displayed. I was getting nervous there for a minute! I got confused when you added all that space at the bottom. It threw off my Jupyter notebook. I followed you thereafter, but my output did not print. We'll on to the other videos from other channels. I got to keep moving on. It was good while it lasted.
Exactly the video I needed! Thx bro!
Thanks alot for the videos! Great way to start my Data Science Journey. Really grateful to you for posting such content for free.
Thank you for the super thanks, best wishes to you on your programming journey!
one of the best programming channels on RUclips. Subscribed and hit the bell ; )
Thanks!
Thanks Snowden, nice tutorial.
Sir your videos are epic! You are an excellent teacher
Cant wait to the next video!! congratulations!!
This video was 2 years ago! Hey sentdex! THANKS :D
Really interesting and objective explanation of the topic! lol'd hard bc of the sudden blue dog
1:52 I was taking it serious till the mug appears
I like your way of teaching.
What is your opinion on setting an aspect ratio and adding padding during resizing? I just feel like forcing an n x n dimension distorts images too much when we have the varied original resolutions.
Guys, the kaggle dataset that he is referring to no longer has folders named as cats and dogs... there are 2 folders, one is for training and another one is for test. You've gotta loop through images in the training folder and assign the labels using the image name
It was really a very useful video...Thaaank u very much for ur timely help
Great tutorial. Looking forwart to next part!
Expect it tomorrow!
Thank you so much this video helped out so much with an up coming video of mine
"the hand of a dog", so wise sentdex. forever indebted
Man , you are the best !! Keep up doing like this !! :D
seriously you teaches better than my professors
Thanks for teaching us. :)
If your using keras you should use the flow_from_directory function ,it's really the same thing without the hassle of running out of memory trying to load the entire dataset.
Hey Sentdex like always awesome tut but one suggestion: Can you mention your library versions you're using at the beginning? I had version compatibility issues which I realized really late and I had to reinstall everything and make sure I also got the correct python version
Thanks have been looking forward to this tutorial will help with my thesis.
For windows, if you have anaconda installed and cannot find module cv2, you may simply have to do:
pip install opencv-python
if you are on linux you can do :
pip install opencv-python
If you are having trouble here use anaconda prompt. This is in the Anaconda Manager where you start jupyter. Then simply type in pip install opencv-python and mine at least worked great
opencv-python is installed, but it cannot find the module cv2
That's an informative video. thank you so much
*sentdex* and *DeepLizard* have both been _VERY_ helpful with teaching me how to program.
Thanks.
holy shit these videos have helped me, thank you so much dude!
I've been looking for ways to upload 40k images to my Drive for 3 days. You in one word: you are perfect
you can download google drive on ur machine and sync it with your drive
Sentdex: understands neural nets
Also Sentdex: doesn't know what to call a dogs paw
More like neural pets lol
Very detailed and helpful and cute. Thank you
Waiting for the next video...thanx man you are an amazing teacher
Next one just released :)
Amazing! going through it now
This helps me a lot.
Excellent work
Thanks for the awesome tutorials. How do you perform image augmentation in this? You sir are a great man
Great Video !!
Great tutorial! However, I got a question. What if you have an image in multiple categories. So you could be sorting images based on size and colour and you stumble on an image that is red and big.
I suggest using context managers for file opening. Cleaner and is better for beginners as you don't have to remember to close the file
Oh thank you! I've looking for the way to load my own dataset and here you go! :З
Awesome video.. Waiting for more uploads...
finally, I got through this video without any error
10:22 what kinda cpu you have that could iterate over all of those images in few seconds :D
I am loving it I am loving it, that's what I want
thanks for showing how to save the weights for future training. also, how did your self driving program from GTA turn out?
Great tutorial, but the way you load the data is not very memory efficient and this will cause problems with large datasets. First the training_data list is written into RAM and afterwards the same amount of memory is reserved when converting into a numpy array. So this approach is only good for datasets < RAM size/2.
Another option would be to create the numpy array at the beginning using np.empty and then write the data as entries into the array. This way the dataset can be as large as your RAM.
If the dataset is larger than the RAM size it is suggested to use a generator that loads and yields the data during training. This way your dataset can be as large as your SSD, but training speed is most likely limited by the read speed of the drive.
Just something I had to deal with during my thesis in the last couple of months. Maybe you could make a tutorial on the generator one, not a lot of people know about this.
Anyways, keep up the good work!
This looks very interesting and I'm experiencing some errors with this as well on my thesis. Can I contact you via email about this?
Sure, can you contact me via youtube? Or post you email and I will contact you
@@will1337 delete the comment with your email
@@will1337 Did you fixed your problem? my python returns "MemoryError" when doing np.array(X).reshape(-1 , IMG_SIZE, IMG_SIZE, 3) step
I'm doing the colored version (so I have about 3 channels of colors which is causing me trouble)
@@stewie055 I did fix it with changing my sampling rate of my data. Maybe resize your images? I am not sure how to fix it with image data, sorry.
so good and so clear
I found for the Y labels you have to make it a numpy array if not the model will not take them. Other than that this is an amazing tutorial
Thank you so much SentDex
the plylist is amazing, however i came across this issues after running part 2 and part 3 back to back..
the y also needs to be an array, so the model.fit in part 3 can run...
thank you once more :)
Great tutorial as always im in love with this channel i learn a lot, i'm trying to build OCR to identify a low resolution documents , do you recommended any source to help me out on this , and i wish some day you create a vedio about this ..Regards
Thanks. On Windows, I had problems with your DATADIR="X:/Datasets/PetImages" @2:10. I had my own version of course, but the code said (in effect) my path was not valid, even tho it was. I discovered Colab runs Linux-style OS. There are several methods of doing it; I used the Google Drive mount and zipfile to extract (instead of the Windows File Explorer Extract), ending with DATADIR=''/content/drive/MyDrive/datasets/training_set'. I finally got to see the gray dogs!
I'm not sure which ends up being better....the videos or the random (read: dope) coffee mugs you keep pulling out in them ;)
Do you have any resources/videos going into detail the os library? Thanks. your vids are awesome!
I did convert my x to a no.array without reshaping it as you did and I got the exact same shape as you did! So I guess you no longer need to reshape it to (-1,50,50,1). plz tell me if I've done something wrong
awesome tutorial
The first video was great, looking forward to watching this one through as well. Can make a video about using CPU vs GPU for some of these training processes? I would like to learn more about forcing the script to use the GPU for running instead of the CPU. For instance some of your older videos (like the Monte Carlo Simulation series) could benefit from this. Thanks!
To use the GPU, you just install the GPU version of TensorFlow. Depending on your OS this is slightly different, but:
Windows: ruclips.net/video/r7-WPbx8VuY/видео.html
Ubuntu: ruclips.net/video/io6Ajf5XkaM/видео.html
Obviously now you do the later version of TF and the correct matching CuDNN and cuda toolkit. Currently Cuda Toolkit 9.0 and CuDNN 7.0.
Great tutorial. What would be the method of saving the collected data to put on a USB? Just a text file?
Super nice tutorial...
you are simply amazing
Great tutorial as always. Very easy to listen to and follow. I was wondering, are you planning to cover things like TFRecords for handling very large datasets sometime in the future? There are other tutorials, but I think the topic would really benefit from your style.
Yep I'd like to. Might wait for TF 2.0 tho
Makes sense. I saw they were prioritizing usability, so hopefully it's more straightforward soon. Unfortunately my project is due in 2 weeks, so a little late to avoid wading through 1.10 docs trying not to cry :P.
Thanks a lott🙏🏽🙏🏽 u just saved me
you saved me, thank you
Great video! How could I modify this to use multiple categories for classification instead of just single category label?
Well done video
Great tutorial, but if the images with multi label ,that way is same to load the data with binary classification or multi label classification
Hi, great work!
I have a question, though, upon the "homework challenge" !
reshape(-1, IMG_SIZE, IMG_SIZE, 3) pops a ValueError: cannot reshape array of size 239640576 into shape (224,244,3).
What's your opinion and solution ??
Thank you
Thank you so much 💙
Love getting tought ML by Snowden!
Hi Sir,
I found you posted X, y empty lists to store the features and labels. However, I have no idea how you distinguish the corrected features and labels from training datasets. Please kindly advise me. Thanks in advance!
How to load the image data set from my computer and split into two data set for training and testing?