After I saw your next video "Cross Attention | method and math explained", I would like to see ControlNet's openpose in PyTorch Implementation which control posing on image of a dogs. Or if it is too complicate, you may simplify it to control 2 - 3 branches shape of a tree.
This videos is crazy! I don't get tired of recommend it to anyone interesting in diffusion models. I have recently started to research with these type of models and I think your video as huge source of information and guidance in this topic. I find myself recurrently re-watching your video to revise some information. Incredible work, we need more people like you!
We chose Diffusion Model as part of our course project, and your videos do save much of my time to understand the concepts and have more focus on implementing the main part. I am really grateful for your contribution.
This is my first few days of trying to understand diffusion models. Coding was kinda fun on this one. I will take a break for 1-2 months and study something related like GANs or VAE, or even energy-based models. Then comeback with more general understanding :) Thanks !
Hey! I am start my CompSci Masters program in the Fall, and just wanted to say that I love this video. I've never really had time to sit down and learn PyTorch, so the brevity of this video is greatly appreciated! It gives me a fantastic starting point that I can tinker around with, and I have an idea on how I can apply this in a non-conventional way that I haven't seen much research on... Thanks again!
thank you so much for your detailed explaination of the code. It helped me a lot on my way of learning diffusion model. Wish there are more youtubers like you!
Hi, @Outlier , thank you for the awesome explanation ! Just one observation, I believe in line 50 of your code (at 19:10) it should be: uncond_predicted_noise = model(x,t,None) 😁
Could you please explain the paper "High Resolution Image Synthesis With Latent Diffusion Models" and its implementations? Your explanations are exceptionally crystal.
Hello, thanks for your a lot contribution ! But a bit confused, At 06:04, just sampling from N(0, 1) totally randomly would not have any "trace" of an image. How come the model infer the image from the totally random noise ?
Hey there, that is sort of the "magic" of diffusion models which is hard to grasp your mind around. But since the model is trained to always see noise between 0% and 100% it sees full noise during training for which it is then trained to denoise it. And usually when you provide conditioning to the model such as class labels or text information, the model has more information than just random noise. But still, unconditional training still works.
Wonderful video! I notice that at 18:50, the equation for the new noise seems to differ from Eq. 6 in the CFG paper, as if the unconditioned and conditioned epsilons are reversed. Can you comment on that?
Great video I faced a question at 19:10 line 50 of the code. why do we call ```model(x,label,None)``` what happened to t? shouldn't we instead call it like ```model(x,t,None)``` ?? also line 17 in ema (20:31) ```retrun old * self.beta +(1+self.beta) * new``` why 1+self.beta? shouldnt it be 1-self.beta?
great walkthrough. but where would i implement dynamic or static thresholding as described in the imagen paper? the static thresholding clips all values larger then 1 but my model regularly outputs numbers as high as 5. but it creates images and loss decreases to 0.016 with SmoothL1Loss.
Roughly how long does an Epoch take for you? I am using rtx3060 mobile and achieving an epoch every 24 minutes. Also i cannot work with a batch size greater than 8 and a img size greater than 64 because it overfills my GPUs 6gb memory. I thought this was excessive for such small batch and img size?
8:38 in the UNet section, how do you decide on the number of channels to set in both input and output to the Down and Up classes. Why just 64,128, etc. ?
@@outliier oh okay got it. Thank you so much for clearing that and for the video! I had seen so many videos / read articles for diffusion but yours were the best and explained every thing which others considered prerequisites!! Separating the paper explanation and implementation was really helpful.
Thank you so much for this amazing video! In mention that the first DDPM paper show no necessary of lower bound formulation, could you tell me the specific place in the paper? thanks!
Sorry if I am misunderstanding, but at 19:10, shouldn't the code be: "uncond_predicted_noise = model(x, t, None)" instead of "uncond_predicted_noise = model(x, labels, None)" Also, according to the CFG paper's formula, shouldn't the next line be: "predicted_noise = torch.lerp(predicted_noise, uncond_predicted_noise, -cfg_scale)" under the definition of lerp? One last question: have you tried using L1Loss instead of MSELoss? On my implementation, L1 Loss performs much better (although my implementation is different than yours). I know the ELBO term expands to essentially an MSE term wrt predicted noise, so I am confused as to why L1 Loss performs better for my model. Thank you for your time.
There are from Yannick Kilcher (on the right side), the one in the lower left is from AICoffeeBreak, the one in the top right corner is the first video that comes when you google „diffusion models explained“ and I forgot the middle one sorry. But shouldnt be hard to find
having hard time to understand the mathematical and code aspect of diffusion model although i have a good high level understanding...any good resource i can go through? id appreciate it
Great videos on diffusion models, very understandable explanations! For how many hours did you train it? I tried adjusting your conditional model and train with a different dataset, but it seems to take forever :D
Thank you so much for this amazing video! You mention that changing the original DDPM to a conditional model should be as simple as adding in the condition at some point during training. I was just wondering if you had any experience with using DDPM to denoise images? I was planning on conditioning the model on the input noisy data by concatenating it to yt during training. I am going to try and play around with your github code and see if I can get something to work with denoising. Wish me luck!
Really nice video! I also enjoyed your explanation video - great work in general :) However, I noticed at around 5:38, you are defining sample_timesteps with low=1. I am pretty sure that this is wrong, as Python indexes at 0 meaning you skip the first noising step every time you access alpha, alpha_cumprod etc. Correct me if I am wrong but all the other implementations also utilise zero-indexing.
Which paper are you referring to? In the first paper, you would just set the variance to beta and since you add the std * noise you take the sqrt(beta)
With this training method, wouldn't there be a possibility of some timesteps not being trained in an epoch? wouldn't it be better to shuffle the whole list of timesteps and then sample sequentially with every batch?
@@outliier There are Diffusion implementations out there that are a lot longer but it also makes it harder to understand what is happening. You removing all the unnecessary parts and just focusing on the absolute minimum is much better in my opinion. Well done.
I think you would just sample 10-50k images from the trained model and then take 10-50k images from the original dataset and then calculate the FID and IS
Your videos are a blessing. Thank you very much!!! Have you tried using DDIM to accelerate predictions? Or any other idea to decrease the number of steps needed?
I have not tried any speedups in any way. But feel free to try it out and tell me / us what works best. In the repo I do linked a fork which implements a couple additions which make the training etc. faster. You can check that out too here: github.com/tcapelle/Diffusion-Models-pytorch
Very great topic. Could you please make another video of generating text to text images? Example: cat with “hello world “, so the model could generate the picture. Thanks ❤
Great video, thanks for making it. I started working with diffusion models very recently and I used you implementation as base for my model. I am currently facing a problem that the MSE loss starts very close to 1 and continues like that but varying between 1.0002 and 1.0004, for this reason the model is not training properly. Did you face any issue like this one? I am using the MNIST dataset to train the network, I wanted to first test it with some less complex dataset.
I am facing similar problems. I did the experiment on CIFAR10 dataset. The mse loss starts descresing normally but at some points the loss increse to 1 and never descrese again.
` x = torch.randn((n, 3, self.img_size, self.img_size)).to(self.device) predicted_noise = model(x, t) ` in the deffusion class why you create an noise and pass that noise into the model to predict noise ... please explain
Thank you very much for this very easy-to-understand implementation. I have one question: I don't understand the function def noise_images. Assume that we have img_{0}, img_{1}, ..., img_{T}, which are obtained from adding the noise iteratively. I understand that img{t} is given by the formula "sqrt_alpha_hat * img_{0} + sqrt_one_minus_alpha_hat * Ɛ". However, I don't understand the function "def noise_images(self, x, t)" in [ddpm.py]. It return Ɛ, where Ɛ = torch.randn_like(x). So, this is just a noise signal draw directly from the normal distribution. I suppose this random noise is not related to the input image? It is becasue randn_like() returns a tensor with the same size as input x that is filled with random numbers from a normal distribution with mean 0 and variance 1 In training, the predicted noise is compared to this Ɛ (line 80 in [ddpm.py]). Why we are predicting this random noise? Shouldn't we predict the noise added at time t, i.e. "img_{t} - img_{t-1}"?
I had the same misconception before. It was actually explained by "AI Coffee Break with Letitia" channel in a video titled "How does Stable Diffusion work? - Latent Diffusion Models EXPLAINED". Basically, the model tries to predict the WHOLE noise added to the image to go from noised image to a fully denoised image in ONE STEP. Because it's a hard task to do, the model does not excel at that so at inference we denoise it iteratively, each time subtracting only a small fraction of the noise predicted by the model. In this way, the model produces much better quality samples. At least that's how I understood it :P
@@Laszer271 While I understand it predicts the "whole noise", this "whole noise" is newly generated and I suppose the ground truth is (img_{t} - img_{0)).. still can't wrap my head around it.
You do not use any LR scheduler. Is this intentional? My understanding is that EMA is a functional equivalent of LR scheduler, but then I do not see any comparison between EMA vs e.g. cosine LR scheduler. Can you elaborate more on that?
Random person 6 months later, but you could try decreasing the batch size during training. Your results may not look like what he got in the video though!
Link to the code: github.com/dome272/Diffusion-Models-pytorch
21:56 The way you starred your own repo makes my day bro 🤣🤣 really appreciate your work, just keep going!!
@@bao-dai xd
@@outliier Thanks for sharing but how do you not get bored or tired of doing the same thing for so long and deal with all the math?
@@leif1075 I love to do it. I don’t get bored
After I saw your next video "Cross Attention | method and math explained", I would like to see ControlNet's openpose in PyTorch Implementation which control posing on image of a dogs. Or if it is too complicate, you may simplify it to control 2 - 3 branches shape of a tree.
This videos is crazy! I don't get tired of recommend it to anyone interesting in diffusion models. I have recently started to research with these type of models and I think your video as huge source of information and guidance in this topic. I find myself recurrently re-watching your video to revise some information. Incredible work, we need more people like you!
Thank you so much for the kind words!
Hello, this has become a great video once again. We didn't understand much, but it's still nice to watch. Greetings from home say Mam & Dad. ;-))))
Great, this video is finally out! Awesome coding explanation! 👏
These implementation videos are marvelous. You really should do more of them. Big fan of your channel!
We chose Diffusion Model as part of our course project, and your videos do save much of my time to understand the concepts and have more focus on implementing the main part. I am really grateful for your contribution.
This is my first few days of trying to understand diffusion models. Coding was kinda fun on this one. I will take a break for 1-2 months and study something related like GANs or VAE, or even energy-based models. Then comeback with more general understanding :) Thanks !
And transformers for the attention mechanisms + positional encoding
I got that snatched in the past 2 months. Gotta learn the math, what is actually a distribution etc.@@zenchiassassin283
Hey! I am start my CompSci Masters program in the Fall, and just wanted to say that I love this video.
I've never really had time to sit down and learn PyTorch, so the brevity of this video is greatly appreciated! It gives me a fantastic starting point that I can tinker around with, and I have an idea on how I can apply this in a non-conventional way that I haven't seen much research on...
Thanks again!
Love to hear that
Good luck on your journey!
After my midterm week i wanna study diffusion models with your videos im so exited .thanks a lot for good explanation
Incredible. Very thorough and clear. Very, very well done.
I was wating for so long i learnd about condicional difusion models
thank you so much for your detailed explaination of the code. It helped me a lot on my way of learning diffusion model. Wish there are more youtubers like you!
Hi, @Outlier , thank you for the awesome explanation !
Just one observation, I believe in line 50 of your code (at 19:10) it should be:
uncond_predicted_noise = model(x,t,None)
😁
good catch thank you. (It's correct in the github code tho :))
Haha, thanks. I also saw the same error and wanted to point it out!
this is the most underrated channel i've ever seen, amazing explanation !
thank you so much!
This channel seems to be growing very fast. Thanks for this amazing tutorial.🤩
most informative and easy to understand video on diffusion models on youtube, Thanks Man
This video is really timely and needed. Thanks for the implementation and keep up the good work!
Sincere gratitude for this tutorial, this has really helped me with my project. Please continue with such videos.
The best video for diffusion! Very Clear
Thank you for sharing the implementation since authentic resources are rare
Congrats, This is a great channel!! hope to see more of these videos in the future.
Dude, you're amazing! Thanks for uploading this!
great tutorial! looking to seeing more of this! keep it up!
The Under rated OG channel
Amazing tutorial, very informative and clear, nice work!
Thank you. Best explanation with good DNN models
Very helpful walk-through. Thank you!
Thanks, this implementation really helped clear things up.
Incredible explanation, thanks a lot!
Thank you so much for this sharing, that was perfect!
nice demonstration, thanks for sharing
Very well done! Keep the great content!!
Could you please explain the paper "High Resolution Image Synthesis With Latent Diffusion Models" and its implementations? Your explanations are exceptionally crystal.
Thank you very much, it has solved my urgent need
It's definitely cool and helpful! Thanks!!!
Hello, thanks for your a lot contribution ! But a bit confused, At 06:04, just sampling from N(0, 1) totally randomly would not have any "trace" of an image. How come the model infer the image from the totally random noise ?
Hey there, that is sort of the "magic" of diffusion models which is hard to grasp your mind around. But since the model is trained to always see noise between 0% and 100% it sees full noise during training for which it is then trained to denoise it. And usually when you provide conditioning to the model such as class labels or text information, the model has more information than just random noise. But still, unconditional training still works.
Very nicely explained. Thanks.
Wonderful video! I notice that at 18:50, the equation for the new noise seems to differ from Eq. 6 in the CFG paper, as if the unconditioned and conditioned epsilons are reversed. Can you comment on that?
Great video
I faced a question at 19:10 line 50 of the code. why do we call
```model(x,label,None)```
what happened to t? shouldn't we instead call it like ```model(x,t,None)``` ??
also line 17 in ema (20:31) ```retrun old * self.beta +(1+self.beta) * new``` why 1+self.beta? shouldnt it be 1-self.beta?
great walkthrough. but where would i implement dynamic or static thresholding as described in the imagen paper? the static thresholding clips all values larger then 1 but my model regularly outputs numbers as high as 5. but it creates images and loss decreases to 0.016 with SmoothL1Loss.
Roughly how long does an Epoch take for you? I am using rtx3060 mobile and achieving an epoch every 24 minutes. Also i cannot work with a batch size greater than 8 and a img size greater than 64 because it overfills my GPUs 6gb memory. I thought this was excessive for such small batch and img size?
thanks for your amazing efforts!
Looking forward for some video on Classifier Guidance as well. Thanks.
Could you also make a video on how to implement DDIM? Or make a GitHub repository about it?
best diffusion youtube
Thank you for sharing!
Fantastic video!
awesome implementation!
8:38 in the UNet section, how do you decide on the number of channels to set in both input and output to the Down and Up classes. Why just 64,128, etc. ?
People just go with powers of 2 usually. And usually you go to more channels in the deeper layers of the network.
@@outliier oh okay got it. Thank you so much for clearing that and for the video! I had seen so many videos / read articles for diffusion but yours were the best and explained every thing which others considered prerequisites!! Separating the paper explanation and implementation was really helpful.
it is very helpful!! You are a genius.. :) thank you!!
Thank you so much for this amazing video! In mention that the first DDPM paper show no necessary of lower bound formulation, could you tell me the specific place in the paper? thanks!
Thank you for the review. So, what is the key to make a step from text description to image? Can you please pinpoint where it is explained?
Sorry if I am misunderstanding, but at 19:10, shouldn't the code be:
"uncond_predicted_noise = model(x, t, None)" instead of "uncond_predicted_noise = model(x, labels, None)"
Also, according to the CFG paper's formula, shouldn't the next line be: "predicted_noise = torch.lerp(predicted_noise, uncond_predicted_noise, -cfg_scale)" under the definition of lerp?
One last question: have you tried using L1Loss instead of MSELoss? On my implementation, L1 Loss performs much better (although my implementation is different than yours). I know the ELBO term expands to essentially an MSE term wrt predicted noise, so I am confused as to why L1 Loss performs better for my model.
Thank you for your time.
Great videos by the way
Ah, I see you already fixed the first question in the codebase
one CRAZY thing to take from this code (and video)
GREEK LETTERS ARE CAN BE USED AS VARIABLE NAME IN PYTHON
Awesome video.
great video. please can you list the creators of the other helpful videos at 00:52? thanks
There are from Yannick Kilcher (on the right side), the one in the lower left is from AICoffeeBreak, the one in the top right corner is the first video that comes when you google „diffusion models explained“ and I forgot the middle one sorry. But shouldnt be hard to find
Can you do one for tensorflow too btw very good explaination
How did you learn do much?
I read a lot of papers and watched a lot of tutorials
Awesome! How did you type Ɛ in code?
Why is the bias off in the initial convolutional block?
Great video!! You make coding seem like playing super mario 😂😂
Can you please explain how to use Woodfisher technique to approximate second-order gradients? Thanks
Thank you!!
having hard time to understand the mathematical and code aspect of diffusion model although i have a good high level understanding...any good resource i can go through? id appreciate it
This is GOLD
Hi , I want to use a single underwater image dataset what changes do i have to implement on the code?
Very cool. How would DDIM models be different? Do they use a deterministic denoising sampler?
yes indeed
Great videos on diffusion models, very understandable explanations! For how many hours did you train it? I tried adjusting your conditional model and train with a different dataset, but it seems to take forever :D
Yea it took quite long. On the 3090 it trained a couple days (2-4 days I believe)
@@outliier Thanks for the feedback. Ok seems like I didn't do a mistake, but only need more patience!
@@maybritt-sch Yea. Let me know how it goes or if you need help
Thank you so much for this amazing video! You mention that changing the original DDPM to a conditional model should be as simple as adding in the condition at some point during training. I was just wondering if you had any experience with using DDPM to denoise images? I was planning on conditioning the model on the input noisy data by concatenating it to yt during training. I am going to try and play around with your github code and see if I can get something to work with denoising. Wish me luck!
This video is priceless.
Amazing stuff!
Really nice video! I also enjoyed your explanation video - great work in general :)
However, I noticed at around 5:38, you are defining sample_timesteps with low=1. I am pretty sure that this is wrong, as Python indexes at 0 meaning you skip the first noising step every time you access alpha, alpha_cumprod etc. Correct me if I am wrong but all the other implementations also utilise zero-indexing.
this function sample the timesteps of the denoising step. selecting time=0 is the original image itself. there is no point in taking 0 timestep.
6:57 Why the formula is ... + torch.sqrt(beta) instead of calculated posterior variance like in paper?
Which paper are you referring to? In the first paper, you would just set the variance to beta and since you add the std * noise you take the sqrt(beta)
With this training method, wouldn't there be a possibility of some timesteps not being trained in an epoch? wouldn't it be better to shuffle the whole list of timesteps and then sample sequentially with every batch?
Great video!
Great Video,
On what Data did you train your model again?
Thank you for the video.
How can we use diffusion model for inpainting?
Where is the link to the code?
sorry I totally forgot to put the link in the description. I updated it now, but here is the link too: github.com/dome272/Diffusion-Models-pytorch
@@outliier There are Diffusion implementations out there that are a lot longer but it also makes it harder to understand what is happening. You removing all the unnecessary parts and just focusing on the absolute minimum is much better in my opinion. Well done.
I think your code bugs when adjust image_size?
Hi! Can you please explain why the output is getting two stitched images?
What do you mean with two stitched images?
thanks for the easiest implementation. could you plz tell us how to find FID and IS score for these images?
I think you would just sample 10-50k images from the trained model and then take 10-50k images from the original dataset and then calculate the FID and IS
@@outliier thanks
How can i increase the size of the generated image here?
Thanks alot :)
There is a slight bug at 19:11
it should be
uncond_predicted_noise = model(x, t, None)
and not
uncond_predicted_noise = model(x, labels, None)
Yes correct. Good catch
Super cool
great video, you got one new subscriber
Your videos are a blessing. Thank you very much!!! Have you tried using DDIM to accelerate predictions? Or any other idea to decrease the number of steps needed?
I have not tried any speedups in any way. But feel free to try it out and tell me / us what works best. In the repo I do linked a fork which implements a couple additions which make the training etc. faster. You can check that out too here: github.com/tcapelle/Diffusion-Models-pytorch
@@outliier Thank you! I will try it for sure.
Can you please tell me how much time was need to train this 3000 image for 500 Epoch?
hey can we use an image as a condition
So the process of adding noise and removing it happens in a loop
Very great topic. Could you please make another video of generating text to text images? Example: cat with “hello world “, so the model could generate the picture. Thanks ❤
@@decode168 usually this is learnt when you train a big model on a lot of data automatically. So there is no specific technique for this
terrific！
Great video, thanks for making it. I started working with diffusion models very recently and I used you implementation as base for my model. I am currently facing a problem that the MSE loss starts very close to 1 and continues like that but varying between 1.0002 and 1.0004, for this reason the model is not training properly. Did you face any issue like this one? I am using the MNIST dataset to train the network, I wanted to first test it with some less complex dataset.
I am facing similar problems. I did the experiment on CIFAR10 dataset. The mse loss starts descresing normally but at some points the loss increse to 1 and never descrese again.
Great video! How long did it take to train the models?
About 3-4 days on an rtx 3090.
`
x = torch.randn((n, 3, self.img_size, self.img_size)).to(self.device)
predicted_noise = model(x, t)
`
in the deffusion class why you create an noise and pass that noise into the model to predict noise ... please explain
Thank you very much for this very easy-to-understand implementation. I have one question: I don't understand the function def noise_images.
Assume that we have img_{0}, img_{1}, ..., img_{T}, which are obtained from adding the noise iteratively. I understand that img{t} is given by the formula "sqrt_alpha_hat * img_{0} + sqrt_one_minus_alpha_hat * Ɛ".
However, I don't understand the function "def noise_images(self, x, t)" in [ddpm.py].
It return Ɛ, where Ɛ = torch.randn_like(x). So, this is just a noise signal draw directly from the normal distribution. I suppose this random noise is not related to the input image? It is becasue randn_like() returns a tensor with the same size as input x that is filled with random numbers from a normal distribution with mean 0 and variance 1
In training, the predicted noise is compared to this Ɛ (line 80 in [ddpm.py]).
Why we are predicting this random noise? Shouldn't we predict the noise added at time t, i.e. "img_{t} - img_{t-1}"?
I had the same misconception before. It was actually explained by "AI Coffee Break with Letitia" channel in a video titled "How does Stable Diffusion work? - Latent Diffusion Models EXPLAINED".
Basically, the model tries to predict the WHOLE noise added to the image to go from noised image to a fully denoised image in ONE STEP. Because it's a hard task to do, the model does not excel at that so at inference we denoise it iteratively, each time subtracting only a small fraction of the noise predicted by the model. In this way, the model produces much better quality samples. At least that's how I understood it :P
@@Laszer271 While I understand it predicts the "whole noise", this "whole noise" is newly generated and I suppose the ground truth is (img_{t} - img_{0)).. still can't wrap my head around it.
You do not use any LR scheduler. Is this intentional? My understanding is that EMA is a functional equivalent of LR scheduler, but then I do not see any comparison between EMA vs e.g. cosine LR scheduler. Can you elaborate more on that?
last self attention layer (64, 64) changes my training type from 5 minutes to hours per epoch, do you know why?
training on a single rtx 3060 TI gpu
Nice tutorial
@outliier Do you think there is a way to run the code with a 3060 GPU on personal desktop? I get the error message: CUDA out of memory.
Random person 6 months later, but you could try decreasing the batch size during training. Your results may not look like what he got in the video though!