This is second video Ive watched from this channel after "quantization". And frankly wanted to express my gratitude towards your work as it is very easy to follow and the level of abstractions is tenable to understand concepts holistically.
Great video, thanks for creating this. I have use DDP quite a lot but seeing the visualizations for communication overlap helped me build a very good mental model. Would love to see more content around distributed training - Deepspeed ZeRO, Megatron DP + TP + PP
That's an amazing resource! It's great to see you sharing such detailed information on a complex topic. Your effort to explain everything clearly will really help others understand and apply these concepts. Keep up the great work!
Starting to watch my 3rd video on this channel, after transformer from scratch and quantization. Thank you for the great content and also for the code and notes to look back again. Thank you.
The video was very interesting and useful. Please make a similar video on DeepSpeed functionality. And in general, how to train large models (for example LLaMa SFT) on distributed systems (Multi-Server) when GPUs are located on different PCs.
well explained each and every detail, Great work Great Explanation👍 can you make this type of detailed video on distributed training through tensor parallelism? it would be very helpful. Thank you!
If time permits for you, Please make an video for entire GPU and TPU and how to them effectively and most of us donno . please create a playlist for pytorch for beginners and intermediates. Thanks for reading.
Hi Umar, Great video and enjoyed thorughly but i have one question.why are we using the approach of sum(grad1+grad2+....+gradN), why cant we use Avg of Gradients.
Of course you can (but you don't have to) use the average of the gradients. Actually, people usually take the average of the gradients. The reason we use the average is because we want the loss to be (more of less) the same as the non-distributed model, so you can compare the plots of the two. I don't know if PyTorch internally automatically takes the average of the gradients, I'd have to check the documentation/source.
In broadcast , if we are sending the copy of file from rank 0 and rank 4 node to other node. How is the total time still 10 second. Because still I am having same internet speed of 1MB/s. Could anyone explain? I am bit confused. Also what happens if I am having odd numbers of nodes
@@umarjamilai not sure if it’s in your plans, but if you are open to suggestions, I would love to watch a video on multimodal models. Again, awesome work!
Wouldn't the accumulated gradient need to be divided by the total number of individual gradients summed (or the learning rate needs to be divided by this value) to make it equivalent?
Yes, if you want to treat the "cumulative gradient" as a big batch, then you'd usually divide it by the number of items to keep it equivalent to the single-item setup. But it's not mandatory: as a matter of fact, loss functions on PyTorch have a "reduction" parameter, which is usually set to "mean" (so dividing the loss by the number of items) but can also be set to "sum". One reason we usually calculate the "mean" loss is because we want to make comparisons between models with different hyperparameters (batch size), so the loss should not depend on the batch size. But remember that mathematically you don't have to
In my understanding, yes the loss is accumulated for one batch theoretically, and the gradients are computed based on this accumulated loss too. But in the parallel implementation, both the loss calculated in the feedforward process, and the gradients calculated in the back propagation process executed in a parallel way. Here @umarjamilai use a for loop to illustrate the de facto parallel mechanism.
This is the best video about Torch distributed I have ever seen. Thanks for making this video!
This is second video Ive watched from this channel after "quantization". And frankly wanted to express my gratitude towards your work as it is very easy to follow and the level of abstractions is tenable to understand concepts holistically.
I really love your vidoes. you have a natural talent on simplifying logic and code. in same capacity as Andrej
Great video, thanks for creating this. I have use DDP quite a lot but seeing the visualizations for communication overlap helped me build a very good mental model.
Would love to see more content around distributed training - Deepspeed ZeRO, Megatron DP + TP + PP
Great introduction. Love the pace of the class and the balance of breadth vs depth
Umar hits the sweet spot (Goldilocks zone) by balancing theory and practical😄😄😄😄😄
That's an amazing resource! It's great to see you sharing such detailed information on a complex topic. Your effort to explain everything clearly will really help others understand and apply these concepts. Keep up the great work!
Starting to watch my 3rd video on this channel, after transformer from scratch and quantization. Thank you for the great content and also for the code and notes to look back again. Thank you.
Amazing video. Ideal video of how a lecture on a video should be
非常清楚的解释~
Dang. Never thought learning DDP would be this easy. Another great content from Umar. Looking forward for FSDP
Super high quality lecture. You have a gift of teaching, man. Thank you!
absolutely amazing! You made these concepts so accessible!
Incredible content, Umar! Well done! 🎉
Great work! thank you !
Another great video, Umar. Nice work
The video was very interesting and useful. Please make a similar video on DeepSpeed functionality. And in general, how to train large models (for example LLaMa SFT) on distributed systems (Multi-Server) when GPUs are located on different PCs.
well explained each and every detail, Great work Great Explanation👍
can you make this type of detailed video on distributed training through tensor parallelism? it would be very helpful. Thank you!
this channel is hidden gem
Amazing content! Thanks for your sharing
thanks for the video!! can you cover the Tensor, Sequence and Pipeline Parallel and D using Dtensors in Pytorch next?
Great intro video. Do you have any plans to also cover other parallelism: Model, Pipeline, Tensor, etc.
Thank you very much for your wonderful video. Can you teach a video on how to use the accelerate library with dpp?
Really impressive!
You deserve many more likes and subscribers!
Thankyou so much for this amazing video. It is really informative.
very nice and informative video. Thanks
Awesome video. Please make tutorial on FSDP as well
Amazing learning stuff ! Very Thanks !~ 🥰🥰🥰
Thank you so much for this
If time permits for you, Please make an video for entire GPU and TPU and how to them effectively and most of us donno .
please create a playlist for pytorch for beginners and intermediates.
Thanks for reading.
thanks
Great video
Thanks!
Hi Umar, Great video and enjoyed thorughly but i have one question.why are we using the approach of sum(grad1+grad2+....+gradN), why cant we use Avg of Gradients.
Of course you can (but you don't have to) use the average of the gradients. Actually, people usually take the average of the gradients. The reason we use the average is because we want the loss to be (more of less) the same as the non-distributed model, so you can compare the plots of the two. I don't know if PyTorch internally automatically takes the average of the gradients, I'd have to check the documentation/source.
@@umarjamilaithanks for the info.
In broadcast , if we are sending the copy of file from rank 0 and rank 4 node to other node. How is the total time still 10 second. Because still I am having same internet speed of 1MB/s.
Could anyone explain? I am bit confused.
Also what happens if I am having odd numbers of nodes
you teach soooooooo good
Please create a video on model parallelism and FSDP.
You rock always 😂
fantastic
I'm always confused with DP and DDP. Can you please tell me the difference between them? While both of them belong to data parallelism method.
DP only works on a single machine, while DDP can work on multiple machines. However, PyTorch now recommends using DDP also for single-machine setup.
@@umarjamilai thank you for your reply
Working with fsdp and megatron now and I really want to figure this out from scratch haha, it sounds fun but a big headache
Federated learning basics please.❤
please do some thing related to audio large models like conformers,quartznet ,etc
could provide another videos with respect to model parallel and pipeline parallel ? thanks..
Great!
I wish i could like it twice
You can share it on social media. That's the best way to thank me 😇
@@umarjamilai not sure if it’s in your plans, but if you are open to suggestions, I would love to watch a video on multimodal models. Again, awesome work!
Check my latest video!
Valeu!
great video
SUUUPERRRR
another banger
How to do in Kubernetes? Please explain it.
Wouldn't the accumulated gradient need to be divided by the total number of individual gradients summed (or the learning rate needs to be divided by this value) to make it equivalent?
Yes, if you want to treat the "cumulative gradient" as a big batch, then you'd usually divide it by the number of items to keep it equivalent to the single-item setup. But it's not mandatory: as a matter of fact, loss functions on PyTorch have a "reduction" parameter, which is usually set to "mean" (so dividing the loss by the number of items) but can also be set to "sum".
One reason we usually calculate the "mean" loss is because we want to make comparisons between models with different hyperparameters (batch size), so the loss should not depend on the batch size.
But remember that mathematically you don't have to
shouldnt loss be accumulated ? loss += (y_pred - y_actual)^0.5
In my understanding, yes the loss is accumulated for one batch theoretically, and the gradients are computed based on this accumulated loss too. But in the parallel implementation, both the loss calculated in the feedforward process, and the gradients calculated in the back propagation process executed in a parallel way. Here @umarjamilai use a for loop to illustrate the de facto parallel mechanism.
do you have a discord channel?
21:01
Always great to watch your video, excellent work
Thanks!
谢谢你！我们在领英connect吧
Valeu!