👉 Check out the blog post and other resources for this video: 🔗 deeplizard.com/learn/video/LhEMXbjGV_4 👀 Come say hey to us on OUR VLOG: 🔗 ruclips.net/user/deeplizardvlog
Ina addition to watching your videos, I visited your website. Your ability to explain Deep Learning fundamentals (both here and in your comprehensive website courses) without shying away from confounding details, is a strong reflection of your immense talent and domain expertise. It's rare. You've earned a supporter. I'm excited to watch this channel grow into something big which I know it will. Keep it up!
thank you, I really regret not knowing about this channel before, I had some difficulties on my university because it was hard to get the information, really thank you :D
Thank You Mandy for the hard work . I regularly visit you website and I can say you are have one of the best ways to make things simpler for NN. Can I request you something in RNN/LSTM/ NLP space? Something of this kind of series . I would be grateful to you .
This is something that you do only for toy datasets or small sample sets. In the reality using image data generator is not recommended when you have huge volumes of images or video frames (which is usually the case). In those cases is a good practice to load and split the data across multiple tfrecords, then loading it progressively with tf.data and perform multithreaded augmentations dynamically during training. The majority of the augmentations can be computed using low level tf or common python functions.
Mher Arabian you can see tf.data documentation for more concrete examples. In general try to manage everything related to input pipelines with tf.data www.tensorflow.org/guide/data
Hey Sanni - Yes you're right, we could use either type of label. Just a matter of arbitrary choice for this example since we would get the same results either way. If we use integer labels, we need to use sparse categorical crossentropy loss. If we use one-hot labels, we need to use categorical crossentropy loss.
@@deeplizard Hi I have followed the code in this video and it get executed without error and the respective folders of test, valid, train are created in the directory but they don't contain the images. Could jus tell what's cozing that error?
There are no images in the train, valid and test files after running all the for loops...., and when I run the train_batch, valid_batch, test_batch cell, the output was all 0 images found under the 2 classes. Help me out with this error.
Thank you for the video. Where does should the jupyter notebook lie relative to the data folder? I am able to create the train, val, and test directories but can't seem to get the images moved to them. Any help would be appreciated. Thanks!
I want to fine-tune vgg-16 model for age detection which is a regression modal. how can I use this image date generator function for regression model (u used for classes classification )
When we are randomly picking 500 dogs for train and 100 dogs for test and 50 dogs for validation, there is a possiblity that same image may be picked up(may be in all three cases).
The move function we use actually moves the files to a new location, not just copies them. Therefore, there is no overlap or duplication between the three sets.
After deleting the csv files with labels, how did the training batches with preprocessing function recognize the cats and dogs images by using classes=[ 'cat' , 'dog' ] 11:30 ? How did the model know which is which and then correctly label at 14:20 ? Can anyone help me understand? Im super confused.
instead of deleting the files by hand you can use this script: path = r"path/to/your/dogs-vs-cats/dataset/" train_folder = "train/" allfolders = os.listdir(path) for folder in allfolders: source = os.path.join(path,folder,train_folder) destination = os.path.join(path,folder+'/')
for img in os.listdir(source): shutil.move(source+img, destination+img) os.rmdir(source) that will do the exact thing :) thank you for the video!
Hi. Thanks for this amazing video. I am getting some errors while compiling line PlotImages(imgs). The error shows "invalid shape (3,224,224) for image data".
Hey! Thank you for uploading such amazing content. I have a question about this video. I am a bit confused as to why you apply the function below to both train, validation and test sets: preprocessing_function=tf.keras.applications.vgg16.preprocess_input My understanding is that this line of code will apply the same type of pre-processing that was applied in vgg16, instead of we manually specifying our own parameters. But I thought this was only important for the train set, and not for validation or test since those directories will be helping us understand the model performance with unseen data...thus, no need to pre-process. Can you clarify why you also run that line in validation and test as well? Thank you so much!
Whats the difference between using flow_from_directory and preprocessing.image_dataset_from_directory? Is there a specific reason to use one before the other?
Hi. I am using Anaconda - Jupyter notebook. May I know how to read the file ?? As it could only allow zip file, I cannot do the splitting of photos like you did starting from 5:14 after unzip. Hope there is a full code tutorial on this. Thanks !!
my code won't load, the first part when I import all the libraries I'm gonna need is all ok, but then when I try to divide the images between train, test and validation the code won't load, it show this symbol * and have stayed like that for hours.. can someone tell me what to do?
Hello Mandy, I would like to thank you first for the awesome work you have done; your tutorials have really been helpful to me, from data augmentation to image preparation. However, I am encountering some problems whereby I have been getting this "Found 0 images belonging to 2 classes". Albeit, I place the directory in jupyter notebook as you have advised, I still am getting the same result. I am utilising my own data which I augmented and the original data I place then into the validation folder. I beseech you, would you advise me to what to do to overcome this problem? I would like to hear from you very soon.
The way you specify your paths to the data depends on your OS. Instead of using forward slashes / in the path, try using escaped backslash characters like this: 'data\\dogs-vs-cats\\train'
@@deeplizard Good day, dear Mandy, tranks a million for reapplying to my request; I will strongly apply this new approach you have come to advise me. I will come back to you for feedback. Thanks in advance, stay safe.
First of all, may the Lord God bless you and your team enormously. You are a lifesaver; now it is perfectly working. In fact, I made a mistake on data labeling, instead of a small letter, I utilised a capital letter. for instance, instead of using dog, I used Dog. Thanks a million.
the if sentences could be at the same indentation as the for sentences, also os.chdir('../../'). Like this (excuse my basic english): os.chdir('M:/Descargas F en M/Varios archivos/Python/Libros/Python/IA/Datasets/Curso Keras Tensorflow/trainer') # acá va el direct relativo a mi workplace if os.path.isdir('train/dog') is False: # chequea si dentro de ese direct están creados los subdirect os.makedirs('train/dog') os.makedirs('train/cat') os.makedirs('valid/dog') os.makedirs('valid/cat') os.makedirs('test/dog') os.makedirs('test/cat') for i in random.sample(glob.glob('cat*'), 500): shutil.move(i, 'M:/Descargas F en M/Varios archivos/Python/Libros/Python/IA/Datasets/Curso Keras Tensorflow/trainer/train/cat') for i in random.sample(glob.glob('dog*'), 500): shutil.move(i, 'train/dog') for i in random.sample(glob.glob('cat*'), 100): shutil.move(i, 'valid/cat') for i in random.sample(glob.glob('dog*'), 100): shutil.move(i, 'valid/dog') for i in random.sample(glob.glob('cat*'), 50): shutil.move(i, 'test/cat') for i in random.sample(glob.glob('dog*'), 50): shutil.move(i, 'test/dog') os.chdir('../../')
Hay errores: *Donde dice tensorflow.keras. ....., borrar tensorflow. , que comience desde keras. ..., porque keras trabaja ahora como módulo independiente *Esto no es error pero puede confundir: en os.chdir('data/dogs-vs-cats'), puede llevar dentro de las comillas el dir relativo o el absoluto, por ej: 'M:/Descargas F en M/Varios archivos/dogs-vs-cats' , respetando la barra hacia la derecha, y una sola barra después de la letra de unidad y los 2 puntos *El indentado está mal. Los for i in random. ... , van a la altura del if os.path.isdir ... y el os.chdir('../../') también a la misma altura de indentado. Si lo ejecutamos así, crea los directorios, pero no les carga las imágenes, puede cargar una sola la primer pasada, porque los for con esa indentación son una condición del if, que está chequeando la creación de los subdirectorios, condición que se cumple en la primer pasada del código
i can not fix this. i followed exactly the same steps and even created all the folders using the given code. however i think it cannot fine the path directory train_path = 'C:\data\dogs-vs-cats\train' .please help. Found 0 images belonging to 2 classes. Found 0 images belonging to 2 classes. Found 0 images belonging to 2 classes.
@@deeplizard I have a similar problem here. Instead of Found 0 images belonging to 2 classes. Found 0 images belonging to 2 classes. Found 0 images belonging to 2 classes. It shows Found 2000 images belonging to 2 classes. Found 400 images belonging to 2 classes. Found 200 images belonging to 2 classes. See each number here should actually be half because there are 1000 images in train, 200 in validate and 100 in test. I even checked the number of images from the folders. They really are 1000 in train and so on. Any help?
I got same error, to fix this, first, I print the length of glob.glob('cat*') function, and I got 0 value. My conclusion is glob.glob didn't find the right directory of the cat images. So what I did is specify the directory for base cat and dog images url., and that's work for me. Maybe you can solve your problem using this solution..
This is the path I mentioned - os.chdir('data\d-vs-c') But I am getting this error - The system cannot find the path specified: 'data\\d-vs-c' Could you please help me !!
How do I fix this? ----> 1 for i in random.sample(glob.glob('cat.*'), 500): 2 shutil.move(i, 'train/cat') ValueError: Sample larger than population or is negative Great video btw! But please help me fix this error, thank you.
Since we have been living in different areas of South East over the last year and a half, we don't have a dedicated filming studio. This episode was filmed from our Airbnb in Vietnam, which is why you see a bed.
👉 Check out the blog post and other resources for this video:
🔗 deeplizard.com/learn/video/LhEMXbjGV_4
👀 Come say hey to us on OUR VLOG:
🔗 ruclips.net/user/deeplizardvlog
Instablaster
Ina addition to watching your videos, I visited your website. Your ability to explain Deep Learning fundamentals (both here and in your comprehensive website courses) without shying away from confounding details, is a strong reflection of your immense talent and domain expertise. It's rare.
You've earned a supporter. I'm excited to watch this channel grow into something big which I know it will. Keep it up!
Thank you, Srujan!
thank you, I really regret not knowing about this channel before, I had some difficulties on my university because it was hard to get the information, really thank you :D
One of the best keras dataset preprocessing videos on youtube. I fully watched this, thank you
Thank You Mandy for the hard work . I regularly visit you website and I can say you are have one of the best ways to make things simpler for NN.
Can I request you something in RNN/LSTM/ NLP space? Something of this kind of series . I would be grateful to you .
Thanks Mandy!!
best video for image data processing. Many many thanks for you efforts.
I've been looking for your channel for a long time, thanks for the work!
The pre processing is always the most annoying part
Thanks for the clear explanation
Thank you very much for this video! It's an excellent video with very beginner-friendly handles on how to get started with CNN models!
Awesome tutorials. These tutorials have really helped me in my path towards learning machine learning. Great stuff 👍
Thank you very much for this teacher ❤
thanks for such great and easiest way to learn preprocessing
Great video. Runs perfectly in my VSCode. Thanks.
your tutorial was very useful and helpful for me, thank you very much
Second time I'm going through this topic and learning more about the os.listdir functions. Great Video. Thanks.
This is something that you do only for toy datasets or small sample sets. In the reality using image data generator is not recommended when you have huge volumes of images or video frames (which is usually the case). In those cases is a good practice to load and split the data across multiple tfrecords, then loading it progressively with tf.data and perform multithreaded augmentations dynamically during training. The majority of the augmentations can be computed using low level tf or common python functions.
Mher Arabian you can see tf.data documentation for more concrete examples. In general try to manage everything related to input pipelines with tf.data www.tensorflow.org/guide/data
BUenas tardes Miguel. Me dirías donde puedo ver ejemplos de lo que comentás?. Muchas gracias
Hi, awesome explanation. Hope this series covers the deployment part too.
You are just amazing! Exploring and Explaining every single little detail is extremely awesome and helpful, Thankyou!
Hey great work,you made it very simple to understand.If possible,can you please make a video on face recognition using cnn it would be very helpful.
Neural Network is amazing ...
7:56 Im going to make these into a desktop wallpaper that changes to a different cat image 10 times per second.
I love thank you I've been waiting for this.
gj..you are making deep learning easy..
Awesome explanation and great content.
Nice explanation
thanks
good stuff! amazing clarity! Hi, professor!
I am getting this ,"ValueError: Sample larger than population or is negative" when running the second cell. Means, the change directory cell.
same here idk what could be the problem
okay so the fix is that you have to put all the images into the 'dogs-vs-cats' dir but not the 'Train' dir
@@Faisalamin01 yeah i realized after lol
can you make something on OpenCV ...that will be great help.
wow!
Very nice explanation 👌👌.make videos on opencv and object tracking and detection
Great tutorial, May I ask why you used one-hot encoding? We could have used 0 for cat, 1 for dog for instance right?
Hey Sanni - Yes you're right, we could use either type of label. Just a matter of arbitrary choice for this example since we would get the same results either way. If we use integer labels, we need to use sparse categorical crossentropy loss. If we use one-hot labels, we need to use categorical crossentropy loss.
@@deeplizard Hi I have followed the code in this video and it get executed without error and the respective folders of test, valid, train are created in the directory but they don't contain the images. Could jus tell what's cozing that error?
This is awesome.
There are no images in the train, valid and test files after running all the for loops...., and when I run the train_batch, valid_batch, test_batch cell, the output was all 0 images found under the 2 classes.
Help me out with this error.
Change the directory to C: drive
@@saloniagarwal4646 it's already in c drive only, I'm sure. There should be something else....
Ok, so far i learned that we have a dog directory, and a cat directory ...
Cool vid, thank you !
I am getting this error module 'tensorflow' has no attribute 'preprocessing_function'. It will be great if anyone could help with this.
Thank you for the video. Where does should the jupyter notebook lie relative to the data folder? I am able to create the train, val, and test directories but can't seem to get the images moved to them. Any help would be appreciated. Thanks!
Choose a directory. Within that directory, place your Jupyter notebook. Within the same directory, place the train, val, and test directories.
@@deeplizard i was stick in this problem from the last 4 hours then i read this comment and done
Thankyou :)
you are the real gal gadot ;)
I want to fine-tune vgg-16 model for age detection which is a regression modal. how can I use this image date generator function for regression model (u used for classes classification )
subscribed to your patreon!!!!
Thank you Hoa!
If you have an image preprocessing problem, like super resolution. How would you do the image generator part????
Can't we pass other features along with directory and classes in the generator to train our model?
Mam can you make some videos on projects on computer vision plz
When we are randomly picking 500 dogs for train and 100 dogs for test and 50 dogs for validation, there is a possiblity that same image may be picked up(may be in all three cases).
The move function we use actually moves the files to a new location, not just copies them. Therefore, there is no overlap or duplication between the three sets.
deeplizard Excellent point and a good question asked.
After deleting the csv files with labels, how did the training batches with preprocessing function recognize the cats and dogs images by using classes=[ 'cat' , 'dog' ] 11:30 ? How did the model know which is which and then correctly label at 14:20 ? Can anyone help me understand? Im super confused.
flow_from_directory() understands labels based on the names of the directories for which the images are stored.
@@deeplizard Great! Thanks for the clarification.
Hello,deeplizard! Mandy i like your job, great and impressive. Please, can you help your community with image segmentation? It should be great! Thanks
Thanks! May add in the future :)
Thank you so much😎!
instead of deleting the files by hand you can use this script:
path = r"path/to/your/dogs-vs-cats/dataset/"
train_folder = "train/"
allfolders = os.listdir(path)
for folder in allfolders:
source = os.path.join(path,folder,train_folder)
destination = os.path.join(path,folder+'/')
for img in os.listdir(source):
shutil.move(source+img, destination+img)
os.rmdir(source)
that will do the exact thing :)
thank you for the video!
Hi. Thanks for this amazing video. I am getting some errors while compiling line PlotImages(imgs). The error shows "invalid shape (3,224,224) for image data".
Hey! Thank you for uploading such amazing content. I have a question about this video. I am a bit confused as to why you apply the function below to both train, validation and test sets:
preprocessing_function=tf.keras.applications.vgg16.preprocess_input
My understanding is that this line of code will apply the same type of pre-processing that was applied in vgg16, instead of we manually specifying our own parameters. But I thought this was only important for the train set, and not for validation or test since those directories will be helping us understand the model performance with unseen data...thus, no need to pre-process.
Can you clarify why you also run that line in validation and test as well?
Thank you so much!
Whats the difference between using flow_from_directory and preprocessing.image_dataset_from_directory? Is there a specific reason to use one before the other?
if our image dataset was gray, can we specify (224, 224,3) instead?
Hello. I am getting "TypeError: 'str' object is not callable" at
img, labels=next(train_batches)
line.
Hi. I am using Anaconda - Jupyter notebook. May I know how to read the file ?? As it could only allow zip file, I cannot do the splitting of photos like you did starting from 5:14 after unzip. Hope there is a full code tutorial on this. Thanks !!
my code won't load, the first part when I import all the libraries I'm gonna need is all ok, but then when I try to divide the images between train, test and validation the code won't load, it show this symbol * and have stayed like that for hours.. can someone tell me what to do?
when i run the code of 6:24 it throws "Sample larger than population or is negative" value error
please help me
Hello Mandy, I would like to thank you first for the awesome work you have done; your tutorials have really been helpful to me, from data augmentation to image preparation. However, I am encountering some problems whereby I have been getting this "Found 0 images belonging to 2 classes". Albeit, I place the directory in jupyter notebook as you have advised, I still am getting the same result. I am utilising my own data which I augmented and the original data I place then into the validation folder. I beseech you, would you advise me to what to do to overcome this problem? I would like to hear from you very soon.
The way you specify your paths to the data depends on your OS. Instead of using forward slashes / in the path, try using escaped backslash characters like this: 'data\\dogs-vs-cats\\train'
@@deeplizard Good day, dear Mandy, tranks a million for reapplying to my request; I will strongly apply this new approach you have come to advise me. I will come back to you for feedback. Thanks in advance, stay safe.
First of all, may the Lord God bless you and your team enormously. You are a lifesaver; now it is perfectly working. In fact, I made a mistake on data labeling, instead of a small letter, I utilised a capital letter. for instance, instead of using dog, I used Dog.
Thanks a million.
@@lukokimpova26 many thanks
@@deeplizard yeah it wouldn't work until i moved all the images from train into dogvscats
Finally can do something useful with the 25000 cat photos on my phone
i dave one problem. Why train, valid and test directory are empty when i try creating them by meanf of running the code?
the if sentences could be at the same indentation as the for sentences, also os.chdir('../../'). Like this (excuse my basic english):
os.chdir('M:/Descargas F en M/Varios archivos/Python/Libros/Python/IA/Datasets/Curso Keras Tensorflow/trainer') # acá va el direct relativo a mi workplace
if os.path.isdir('train/dog') is False: # chequea si dentro de ese direct están creados los subdirect
os.makedirs('train/dog')
os.makedirs('train/cat')
os.makedirs('valid/dog')
os.makedirs('valid/cat')
os.makedirs('test/dog')
os.makedirs('test/cat')
for i in random.sample(glob.glob('cat*'), 500):
shutil.move(i, 'M:/Descargas F en M/Varios archivos/Python/Libros/Python/IA/Datasets/Curso Keras Tensorflow/trainer/train/cat')
for i in random.sample(glob.glob('dog*'), 500):
shutil.move(i, 'train/dog')
for i in random.sample(glob.glob('cat*'), 100):
shutil.move(i, 'valid/cat')
for i in random.sample(glob.glob('dog*'), 100):
shutil.move(i, 'valid/dog')
for i in random.sample(glob.glob('cat*'), 50):
shutil.move(i, 'test/cat')
for i in random.sample(glob.glob('dog*'), 50):
shutil.move(i, 'test/dog')
os.chdir('../../')
I have problem when I organize the data. My output is error, it said sample larger than population or is negative. How to fix it ?Thankyou
i got the same problem any solution?
Hi Madam, how to HOW TO load from npZ file . please let me know
Can you upload the invoice image model
Is there a way to get the dataset without kaggle? I dont own a phone and cant download it.
EDIT: I got it anyway
can i build a mobile application using this data model?
Hay errores:
*Donde dice tensorflow.keras. ....., borrar tensorflow. , que comience desde keras. ..., porque keras trabaja ahora como módulo independiente
*Esto no es error pero puede confundir: en os.chdir('data/dogs-vs-cats'), puede llevar dentro de las comillas el dir relativo o el absoluto, por ej: 'M:/Descargas F en M/Varios archivos/dogs-vs-cats' , respetando la barra hacia la derecha, y una sola barra después de la letra de unidad y los 2 puntos
*El indentado está mal. Los for i in random. ... , van a la altura del if os.path.isdir ... y el os.chdir('../../') también a la misma altura de indentado. Si lo ejecutamos así, crea los directorios, pero no les carga las imágenes, puede cargar una sola la primer pasada, porque los for con esa indentación son una condición del if, que está chequeando la creación de los subdirectorios, condición que se cumple en la primer pasada del código
How to choose the image_size ?
where are labels?
i can not fix this. i followed exactly the same steps and even created all the folders using the given code. however i think it cannot fine the path directory train_path = 'C:\data\dogs-vs-cats\train' .please help.
Found 0 images belonging to 2 classes.
Found 0 images belonging to 2 classes.
Found 0 images belonging to 2 classes.
i fixed it.
it must be like this
train_path = 'C:/data/dogs-vs-cats/train'
not
train_path = 'C:\data\dogs-vs-cats\train'
Thanks for sharing your solution. They way in which you specify file paths (with forward or back slashes) depends on your OS.
@@deeplizard I have a similar problem here. Instead of
Found 0 images belonging to 2 classes.
Found 0 images belonging to 2 classes.
Found 0 images belonging to 2 classes.
It shows
Found 2000 images belonging to 2 classes.
Found 400 images belonging to 2 classes.
Found 200 images belonging to 2 classes.
See each number here should actually be half because there are 1000 images in train, 200 in validate and 100 in test.
I even checked the number of images from the folders. They really are 1000 in train and so on.
Any help?
ValueError: Sample larger than population or is negative
getting this error while running code for organize data
I got same error, to fix this,
first, I print the length of glob.glob('cat*') function, and I got 0 value. My conclusion is glob.glob didn't find the right directory of the cat images. So what I did is specify the directory for base cat and dog images url., and that's work for me.
Maybe you can solve your problem using this solution..
Is there this code on github or somewhere i can download ?
website
I don't know why I'm getting this Error ValueError: Sample larger than population or is negative !
i had the same issue it means that the pics arent in the correct location
hi, i faced the same error, try moving the pics from train folder to the dogs-vs-cats folder. (with the train folder being empty). that should work
@@AyushGupta-kp9xf Thanks dear..it works. I just rename os.chdir('data/dogs-vs-cats') to os.chdir('data/dogs-vs-cats/train')...
This is the path I mentioned - os.chdir('data\d-vs-c')
But I am getting this error - The system cannot find the path specified: 'data\\d-vs-c'
Could you please help me !!
hi, i faced the same error, try moving the pics from train folder to the dogs-vs-cats folder. (with the train folder being empty). that should work
cat.6.jpg: Cat or Human? Yes
How do I fix this?
----> 1 for i in random.sample(glob.glob('cat.*'), 500):
2 shutil.move(i, 'train/cat')
ValueError: Sample larger than population or is negative
Great video btw! But please help me fix this error, thank you.
hi, i faced the same error, try moving the pics from train folder to the dogs-vs-cats folder. (with the train folder being empty). that should work
Why there is a bed in the background ? :>
Since we have been living in different areas of South East over the last year and a half, we don't have a dedicated filming studio. This episode was filmed from our Airbnb in Vietnam, which is why you see a bed.
Lockdown time it doesn't matter. Look at the video content and enhance your learning.
You wanna marry me?