FASTER Inference with Torch TensorRT Deep Learning for Beginners - CPU vs CUDA
HTML-код
- Опубликовано: 11 июн 2024
- Hi everyone! 😀 In the last video we've seen how to accelerate the speed of our programs with Pytorch and CUDA - today we will take it another step further with Torch-TensorRT!
We will focus on a Machine Learning process called Inference (which is when the model is trained, perfected and ready to make a prediction).
For this we will load a state-of-the-art artificial neural network and we will use it to classify a picture of my cat! 🙀🙀🙀
Specifically - we will borrow ResNet50 for our little Pytorch experiment! 😉
We will also run a speed test comparing Pytorch models running on CPU, on CUDA and on Torch-TensorRT - which of these do you think is faster??
⏲️ TIMESTAMPS ⏲️
-----------------------------------
00:00 - intro
01:05 - clone Torch-TensorRT
01:40 - install and setup Docker
03:52 - install Nvidia Container Toolkit & Nvidia Docker 2
05:02 - Torch-TensorRT container (option #1)
07:22 - Torch-TensorRT Nvidia NGC container (option #2)
09:00 - import Pytorch
09:16 - load ResNet50
10:25 - load sample image
11:45 - sample image transforms
14:48 - batch size
16:19 - prediction with ResNet50
17:12 - softmax function
18:07 - ImageNet class number to name mapping
20:10 - predict top 5 classes of sample image (topk)
23:33 - speed test benchmark function
27:33 - CPU benchmarks
28:13 - CUDA benchmarks
30:09 - trace model
31:20 - convert traced model into a Torch-TensorRT model
33:02 - TensorRT benchmarks
34:32 - download Jupyter Notebook
34:50 - HOW DID I MISS THIS???
35:31 - thanks for watching!
🛑 REFERENCED TUTORIALS 🛑
----------------------------------------------------------------------
⭐ CUDA Parallel Computing for beginners:
• CUDA Simply Explained ...
⭐ Neural Networks for beginners:
• Neural Network Simply ...
⭐ Machine Learning Databases:
• Machine Learning Datab...
⭐ Gradient Descent:
• Gradient Descent - Sim...
⭐ INSTALLATION LINKS AND BASH COMMANDS ⭐
--------------------------------------------------------------------------------------
1. Clone Torch-TensorRT and change directory:
$ git clone github.com/NVIDIA/Torch-TensorRT
$ cd Torch-TensorRT
2. Docker installation guide:
docs.nvidia.com/datacenter/cl...
$ curl get.docker.com | sh \ && sudo systemctl --now enable docker
$ sudo groupadd docker
$ sudo usermod -aG docker $USER
$ newgrp docker
$ docker run hello-world
3. Nvidia Docker 2 installation:
$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \ && curl -s -L nvidia.github.io/nvidia-docke... | sudo apt-key add - \ && curl -s -L nvidia.github.io/nvidia-docke... er.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
$ sudo apt-get update
$ sudo apt-get install -y nvidia-docker2
$ sudo systemctl restart docker
$ sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
4. Official Torch TensorRT Container:
$ docker build -t torch_tensorrt -f ./docker/Dockerfile .
$ docker run --gpus=all --rm -it -v $PWD:/Torch-TensorRT --net=host --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 torch_tensorrt:latest bash
$ cd /Torch-TensorRT/notebooks
$ jupyter notebook --allow-root --ip 0.0.0.0 --port 8888
5. Nvidia NGC Container (alternative to 4):
catalog.ngc.nvidia.com/orgs/n...
$ docker pull nvcr.io/nvidia/pytorch:21.12-py3
$ docker run --net=host --gpus all -it --rm -v local_dir:/container_dir nvcr.io/nvidia/pytorch:21.12-py3
$ jupyter notebook --allow-root --ip 0.0.0.0 --port 8888
6. My cats picture:
github.com/MariyaSha/Inferenc...
7. Transforms Normalize - Pytorch Documentation:
pytorch.org/vision/stable/tra...
8. ImageNet class to name mapping:
github.com/pytorch/hub/blob/m...
9. Complete Notebook on Github:
github.com/MariyaSha/Inferenc...
---------------------------------------------------------------------------
💗 THANK YOU SO MUCH FOR WATCHING! 💗
- Sound effect by: www.zapsplat.com
- Icons by: www.flaticon.com/
- Resnet50 image by: commons.wikimedia.org/wiki/Fi... 
Наука
Great video as always. One advice: in the newer versions of Ubuntu apt-get is replaced with apt because apt-get is deprecated and apt has better performance. Everything stay the same while installing things but instead of apt-get you will use apt.
Thank you so much for the tip and for the lovely comment Luka! 😀😀😀
I love Python Simplified. All the tutorials are very well explained and the content is super interesting. Excellent work. Thank you very much.
Thank you so much for the beautiful comment! 😀😀😀
I'm super happy you enjoy my tutorials! Will keep em' coming! 😊
This approach is much more simple than development in Anaconda environments. Thanks!
Thank you so much! Your video helped me to manage with TensorRT!
I have a graduation project in which we will use Pytorch and NLP, so I want your support because your videos are so amazing and simple, we want more
Thank you so much Abdallah! 😁
NLP is absolutely fascinating! I think I have a repository on Github that might help you with your project:
github.com/MariyaSha/StoryTeller
It's a Pytorch Word Embeddings model that was trained on children books/fairytales and it predicts an entire story once you select the first 5 words 😉
It's a fairly stupid network, I must admit hahahaha, but if you're having issues with some of the processes - you might find a solution there.
I also believe that's the network I cover in this tutorial: ruclips.net/video/mzbJd0NhW2A/видео.html (It's one of the first videos I've filmed so my apologies in advance hahaha 😅)
Best of luck with your graduation project! let me know how it turned out! 😀😀😀
@@PythonSimplifiedThank you very much, I really watched the whole playlist and benefited a lot and we want more of your wonderful videos and it is my pleasure to tell you about my project
Every time, I hear you say "Holy Smoke", I have a little laugh (the only one I ever heard say that expression was my wonderful father and he was born in 1924). Your videos are great! ... very comprehensive and entertaining. Thanks so much!
Love you very much! Like your content! Its great!
Thank you so much for the lovely comment Andrei! 😁
Very nice content well explained and perfect for beginners
Thanks for providing best contents.
Thank you so much! 😊
Another Fantastic Preesentaion ~
Thank you so much dear! 😁
Thank you so much for these videos. Is it possible for you to make another video on how to run the opener Whisper model in TensorRT for near real-time transcription of voice-to-text?
Thanks for making this video. I was looking for this information. Excellent presentation.. keep posting new videos. please make one video on jetson with torch-tensor RT. Do we need to follow different commands or simply this tutorial will work with jetson.
Thanks!! Clear!
You have my subscribe!!
Nice Fantastic import video ..thanks again :) you have the best channel for learning
Thank you so much! I've put some extra work on the installation portion in case some folks run into Docker issues 😉
How did you find Docker compares to Anaconda? I'm so used to Anaconda that I rarely try new stuff hahahaha 😊
@@PythonSimplified me too I much prefer anaconda ...sometime you have to try new things ....it could be better :)
I'm in Love 😂❤️
Thank you for this excellent video. Do you have time to make another video on how to convert WhisperAI models to TensorRT for faster (i.e., near real-time) conversion of voice to text?
Thank you for great video. Really appreciate this. What is process you follow for shooting/editing videos? Also please suggest the gear you use for shooting videos.
Thank you so much Sushruti! 😀
If you're just starting with making videos - I recommend to only get a nice microphone and some lights. You can start with your phone camera and connect it to your computer with Android's IpWebcam (I actually used it in a tutorial a while back: ruclips.net/video/UdCSiZR8xYY/видео.html )
Then I use OBS Studio to combine the screen capture with the IpWebcam stream. You don't really need a green screen at first - but it's definitely nice to have one somewhere along the way 😉
To use a green screen, just add a "chroma key" filter in OBS and set it to green. And of course as you move along you can invest in a fancy camera and go from there - but the key note is to start with the tools you have and keep expanding 😜
In terms of video editing, I use the Adobe Creative Suite (a combination of Premiere, Illustrator and Photoshop).
And you can find open source alternatives as well (Kdenlive instead of Premiere, Inkscape instead of Illustrator and Gimp instead of Photoshop) which may be a better starting point.
My first few videos were edited with Windows Movie Maker (which was unfortunately discontinued when Windows10 has arrived)
Best of luck on your new video making journey, let me know how it turned out! 😁
I don't use computer and laptop because I don't have xDDD
But no longer it will happen I will purchase one! :)
But yeah I'm liking it now coding from phone!
Great video 💪🙌🤠
Thank you so much Udbhav! 😃 I did mention your supreme phone coding skills during the livechat! 😉
Good luck with your PC/laptop search, let me know if you need an advice with hardware 😊
@pythonSimplified, Thanks for your great videos! would it be possible to make a video about openvino? what it is, and what benefits it brings?
Lol! I missed yesterday 😅 🔥❤️❤️
Hahaha no worries, attendence is only recommended 😉
You nailed "Habibi"😂
Excelente conteúdo. I love you
Thank you so much Jose! 😁
Perfect
Hi there, gr8 video! can i ask what gpu server you are using please? because i have a project that grabs a frame from a video, then sends the image as base64 to a ubuntu server with tensorflow serving server using flask. The problem i have is that it takes 30 seconds to inference the image and send the data back. The server is just a CPU server, but looking at renting a GPU server seems to be way to expensive. Do you have any thoughts how I can speed this up?
Very Helpful
Thank you so much Collins! Super happy to help! 😃
@@PythonSimplified absolutely your lessons have saved me a lot of time and I have learned so much, very grateful
You so good at this. How long have you been coding for ?
Is it applicable for NVIDIA Jetson Nano? What machine and GPU are you using?
Não entendo nada do que fala mas adoro ver você falando.
hahaha obrigado! Eu gostaria de poder falar português... por enquanto tenho que confiar no Google Translate 😅
@@PythonSimplified Parte do que disse foi brincadeira, apesar de não entender o que fala, eu sou programador (Python inclusive) e aproveito bem seus exemplos. Obrigado.
One important thing is that simply tracing also improves inference speed by 2x often. So I am unsure if the TensorRT is actually adding much additional speed for how much more of a nuisance it is to install.
what a big monitor you have
Can you share a video on inference of large transformer model from huggingface on Multi-GPU with TensorRT
very nice, but maria what version is your ubuntu?
👍😉
Hi. I tried doing second approch of installing tensorrt. Jupyer notebook is not launching though i changed from hostname to localhost. Its still saying this site cant be reached. Please can you guide me what to do here????
This first video about trt tutorial. Could you try convert yolov5 models, will be good
I haven't had a chance to work with yolov5 before, but if I see a few more requests on it - will definitely film a tutorial on the subject! 😉
Thank you so much for your suggestion! 😀
Do you think you can make a crash video on how to from scratch install and use docker on a deep learning simple project and then how to provide it for the user to use it? I never got it how to set and use and deliver docker....
Sounds like a great idea! 😀 If I see a bit more requests on it - will definitely film a Docker dedicated video (similar to the Anaconda one 😉)
Thank you so much for suggesting! 😊
Can you please make a tutorial on DOCKER? Thankyou
Could you do a tutorial using Jetson Nano?
Mariya thanks for everything, you are great!! please if you can do the same but with Windows I would appreciate it very much!!! :p
Better to get wsl on windows and do the same
Can you make video on how to create Windows Application with Python?
It is already out.
I already have a bunch of videos for that, Kesav 😊
1. A Simple Greeting App with Kivy: ruclips.net/video/YDp73WjNISc/видео.html
2. A trivia game with PyQT5: ruclips.net/video/9iZLDnW_vwU/видео.html
3. A simple SMS Spam filter with DearPyGUI: ruclips.net/video/2RocXKPPx4o/видео.html
4. A PDF Reader with Tkinter: ruclips.net/video/itRLRfuL_PQ/видео.html
All these are Python Desktop Application tutorials, so you have lots of things to investigate now! 😉
Good luck and have fun! :)
Please and please I request you make video on how to use multiple databases in flask
Make a tutorial on hand written Character recognition
Why zero predictions for Mafia cat?
3% - carton because your image has brown color subject like box and a lot of straight lines. it's 3%🤗
we need Transformers
when will u work on django
Facing cuda out of memory issue when I call the model which is loaded in GPU memory. The GPU doesn't free memory
Try restarting the kernel and then give it another go. Otherwise - are you trying to load the entire dataset at once? or are you loading it in batches? if batches - decrease the batch size. If not - load the data batch by batch instead! 🙂
@@PythonSimplified thanks for ur help. The issue was there was no context manager torch.no_grad() used when doing foward passes to the model. With this change it released memory.
yeah Im just starting machine learning if your beauty let me to concentrate
Hey ! Thank you for video ! I really appreciate your tutorials.
I'm just thinking that maybe the title of the video doesn't sound totally correct in fact that TensorRT is based on CUDA and it runs only with NVIDIA GPU, so for comparaison you can say Torch CPU vs TensorRT GPU
Both of PyTorch and TensorRT are deep learning framework, CUDA is a parallel programming language for NVIDIA GPU's and ofc CPU and GPU represents hardware for training or inference
Can you make a Rust tutorial? 🙏🏽
I haven't had a chance to work with Rust yet, so I'm probably not the best person for the task... 😅
But I've just stumbled upon a Rust tutorial recommendation on Discord, you might find it helpful: ruclips.net/video/zF34dRivLOw/видео.html
А что там в Канаде случилось ?_?
Ludi hotiat svobodu - provitelstvo ho4et rabstvo 😢
Neskolko dnei nazat oni zaiavili voenoie polozeniie iza tovo 4to dalnoboishiki bostuiut v Ottawe... oni sei4as v turme. takovo ne kogda v kanadskoi istorii nebilo, voenoie polozenie eto tolko dla voini protiv drugih stran - ne protev nashih ludei... tak 4to vse v shoke!
Interesnie vremina... eto to4no! 😅
Mariya~,
(?) RU caught up near thoze Trucker 'eventz'; &,
(?) what R y'r thought'z on that...
ShoutOut from @ Michigan
You know I can't talk about it on YT... 🙃
@@PythonSimplified Ya, I get that...
who'da thought 'in the West'
we'd loose 'FREE Speech' & have to bight our tounge'z!
Good always wins evil, even in the darkest of times!
Stay positive, determined and don't let them drag you into hatred. We're all on the same side - it's just that not all of us realize it yet 😉
@@PythonSimplified Deep,
@@PythonSimplified unReleated,
could U do a video show'ng how to covert between number bases, w/Lg numbers, & w/'fractional-number-parts'; in my prj I waz us'ng Excel { base() }, but it iz limit'd to integers & to a certain range of values { I think (2^53)-1 }, I was just think'ng perhaps Py may be a better choice; specifically, I want to go from base 10 to base 24.
& of course, y'r ML video waz great, az alwayz
I'm late back, looks like you've been busy with the Ukraine crisis, I hope you're fine
There's actually an exponentially bigger crisis in Canada... 😅 (and I wouldn't trust the news for reliable information in any shape or form, ask people who you personally know and live there - you'll get a very different angle, trust me)
@@PythonSimplified We always wish the two countries well, however welcome back
Привет у тебя Вк есть ?_? Ну в контакте ?_?💖
Nea... u menja bil kogdato dovno, no uze ne pomnu parol i "username" 😊
hi my sister.
please I want to learn how to scrape full data from Facebook like email, phone number and convert code to user interface
please............. please......... please............
thank you
Mam love your content, but I have one question. You are already so cute and beautiful then why do you put that extra makeup
why not...? does it really matter?😜
You would very rarely see people who go in front of the camera without makeup... even guys! (especially when filming in 4k quality... every little flaw is captured by the lenses)
@@PythonSimplified yeah that's true even we boys also put a lot of makeup but I must say that I have never seen a teacher as beautiful as you are, God bless you and may you accomplish all your goals 😇♥️
What is your name dear??
how the hell she could remember all the command :v
The attendance program through facial recognition was working without any problem, but suddenly it started giving the following error
how can i fix ?
module' win32 com has no attribute 'CLSIDToClassMap
AttributeError: module 'win32com.gen_py.00020813-0000-0000-C000-000000000046x0x1x9' has no attribute 'CLSIDToPackageMap