Speed up your TensorFlow code using TFRecords and dataset pipelines

In this video you'll learn how to use TFRecords and dataset pipelines to increase your TensorFlow Training code.
A link to my website:
aiexpedition.com/
A link to the github repo:
github.com/jla...
Documentation:
tf.data.TFRecordDataset
www.tensorflow...
tf.train.Example
www.tensorflow...
tf.train.Features
www.tensorflow...
tf.train.Feature
www.tensorflow...
TFRecord and tf.train.Example
www.tensorflow...

[@chloehe3889]

I am happy to finally find out your video with still low view counts. It is so helpful. Thank you!

[@sanjaynt7434]

Thank you❤🙏, no one explained tf record this clearly.

[@erlendllandgundersen5446]

This video really helped me on my master's thesis! Thanks a lot!!

[@eduardojreis]

Such a helpful video! Thanks a lot for taking time and effort doing it!!!

[@helloworld9478]

Exactly what I am looking for

[@DiziNoer]

Very helpful video, thanks !

[@aiexpedition5314]

Glad it helped!

[@fununterhaltung6556]

best video! thanks u so much

[@aiexpedition5314]

Thanks so much for the kind words. glad it helped :)

[@blackburn9085]

Great Video! Very Informative! Keep it UP

[@aiexpedition5314]

Thank you :)

[@SS-cz2de]

Very well explained.

[@aiexpedition5314]

Thank you Samit :) I appreciate the kind words.

[@virajjadhav3601]

The video is really helpful, but please work on the audio quality. Thanks for the tutorial

[@aiexpedition5314]

Thanks Viraj, I will work on it :)

[@ccuuttww]

liked very helpful

[@ccuuttww]

And I think not even TFrecord can really speed up the process also the memory fitting U may not fit it inside to 8GB ram with that large dataset

[@kameranjacobsohn7561]

👏👏👏

[@ginanottenkaemper7960]

thank you very much.
How long did it take to create the TFRecord?
I am loading 600,000 images of size 2500x3000 pixels and am looking for a way to reduce the image size before saving as TFRecord.

[@aiexpedition5314]

Hi Gina, sorry for the late reply. I can't remember exactly how long it took to create the TFRecord. It was a few minutes. Your dataset seems large so I would recommend creating the TFRecord on a fast machine and then uploading the TFRecord somewhere that colab can download it from. (The upload file functionality for colab was quite slow last time I checked).

[@ginanottenkaemper7960]

@@aiexpedition5314 Thank you for your answer.
My current work-around is to create multiple TFRecord files so that I can run it in parallel on multiple machines.
I will do the upload for Colab with the link to Google Drive, which I have only had good experiences with so far.
You seem to have trained CNNs quite often. Can you estimate how long the training will take? (600.000 Images from 2 Classes, Transfer learning with a ResNet-50)
Also, I'm going to start your Udemy course next week and I'm very excited about it!

[@aiexpedition5314]

@@ginanottenkaemper7960 Downloading from Google Drive works well.
600,000 images of size 2500x3000. That's quite a lot and will probably take days or even weeks to train if you perform a couple of training iterations and use all of the data. However, with transfer learning and only using 2 classes you should start to get good training accuracy even after your network has only seen a couple hundred of examples, so you could stop the training early even after a few minutes.
Thanks for considering my course! I hope you learn a lot and have a good learning experience. Feel free to ask if you have any questions. Good luck

[@ginanottenkaemper7960]

​@@aiexpedition5314 Hey Jeremy,
Thanks for your quick reply. Hope you are well.
Training the network worked perfectly thanks to you.
now i'm having trouble evaluating my model. I would like to call the function model.predict() or model.evaluate(). I tried to create a batch of the test dataset (dataset.batch(128)) and pass it to the methods. The code has been running for over an hour with no result and I don't know if I should have done something differently or if I need to wait for longer.

[@aiexpedition5314]

@@ginanottenkaemper7960 glad your training is working.
It could be that you just have many images in your testing dataset and it will take a while. Is there any indication of progress while predicting or evaluating? How many images are there in your test set? Something you could try is calling predict() or evaluate() on dataset.take(256).batch(128). This will just run prediction on the first 256 images over 2 batches and should be a lot quicker than running over the entire dataset. Then you can use that to give you an indication of how long the evaluation will take over the entire dataset.

[@JamesBond-ux1uo]

video quality is very poor

[@aiexpedition5314]

Hi James, thanks for letting me know. The video is playing on HD on my side but the audio seems a bit poor near the end of the video. Is that what you meant?

[@PhiloMath1412]

I don't usually comment, but more ppl need to subscribe and share this video, or channel, haven't looked at channel yet, but I sub'ed 👍

[@aiexpedition5314]

Thank you :). I appreciate the sub!