3:44 Intro, Gaussian Distribution, Probability Density Function (PDF) 7:38 GMM Intro 9:08 Covariance matrix 10:15 GMM Definition, K Gaussians 11:30 How to apply GMM for classification 12:30 Problem statement, Fitting a GMM model, Maximum Likelihood Estimate (MLE) 13:58 Similarity to Kmeans clustering algorithm 16:13 Expectation maximization (EM) algorithm and difference to Gradient Descent 18:15 When to apply GMM, anomaly detection, clustering, object tracking 19:30 Coding example with Python 25:10 EM algorithm workflow in practice, Log Likelihood 27:54 EM algorithm visual / walkthrough 36:30 Summary great video, many Thanks :)
warning: when he finger styles his hair, get ready for hardcore info dump. PS: 3blue1brown series on linear algebra has THE BEST vid on eigen vectors/value pairs, no joking.
Siraj never fails to inspire, and I agree with his point strongly - we are the most important community in the world today. We all have a common goal, of making the world better with the best tech we have to offer. I for one am working on a universal translator not just for spoken languages, but for sign, braille and more. ML and NNs has moved my research forward by at least a decade.
In case you have bad results using Gaussian mixtures, keep in mind the EM optimization only has local convergence properties, just like gradient descent: it can get stuck. Restarting the the density estimation with other initial parameters might solve it ! :)
Siraj, I think it would of been helpful if you showed the resulting clusters that you get from the gaussian mixture model approach in your data. You showed how to model your data using the gaussian mixture, but I am unclear on how we get the specific clusters (say 2 clusters) from that?
When my friends ask me how to start with machine learning and AI, I tell them Siraj is the way to go! Thanks for making the AI community so cool! Yes we are the COOL GUYS!
suggestion at time 6:45 minutes, the y values aren't the probabilities of the x values, intuitively the probability for a single point on the gaussian will be 0.
So... 38 minutes to predict something and he just forgets about the prediction part? I'm sorry, but the justification on 36:02 is not enough for my satisfaction.
Great video, I tried running your code on my terminal and it's giving the error that 'GaussianMixture' object has no attribute 'loglike', would you happen to know why an error like would occur, or anyone by that matter. Thank you so much
I got pretty confused around 33:33 with the E step. You've computed wp1 and wp2, which is cool, and then normalised them so their sum is 1 [wp1/(wp1+wp2) + wp2/(wp1+wp2) = (wp1+wp2)/(wp1+wp2) = 1], which makes sense. You then add the log of this sum to self.loglike. But the log of 1 is 0... Which is where you lost me.
He makes mistakes... If only that was the only one... Referring to Variance as Variation... Doesn't know how a Standard Deviation is calculated... omg.
I watch 4-5 vídeos of you per day. I'm Learning generative models for drug Design Siraj. Watch your videos not only motivates me, also makes my life & study fun and cool.
Another great tutorial, Siraj. By the way, if anyone gets an error seaborn deprecation and being unable to plot histograms, I resolved it by going into the linked 3rd party lib seaborn/distrubutions.py and changing "normed" to "density"
5:45 +siraj "whether it's a car or roller coaster that's increasing in velocity reaches a peak then decreases or a soundwave... very likely a Gaussian distribution would be a great model"...????? Isn't the bell curve representative of the frequency of the data, not the data itself??
It's always great and informative to watch and learn from your video. But my question is a non technical, but do provide a solution plz... Question : I saw your github profile, and I'm curious what filters you applied on your profile pic(dp) ?? :p ps: I already told you this question is going to be a non-technical one and Yes !!! you have been on my youtube's subscription list from the very beginning. Cheers !!!
Loved the explanation. If I have to model 6 features instead of 2, and use a sliding windows approach on my dataframe (I need to find the anomalous windows), how can I modify the weights and the rest of the code? Just looking for direction.
I have the problem with the gaussian mixture models, I don't know how generate outliers uniformly in the p-parallelotope defined by the coordinate-wise maxima and minima of the ‘regular’ observations in R?
Hi all! I've fixed some bugs from the original .ipynb so you are welcome to try my one: github.com/ivanpastukhov/Gaussian_Mixture_Models/blob/master/intro_to_gmm_%26_em.ipynb Siraj, thanks a lot for your brilliant videos!
Thank you! Your videos helped me a lot... I was so lost and confused about this topic that I was on the verge of giving up. Checked out your tutorials that gave a lot of useful information and insights. Thanks a tonne! :) :D Keep up the good stuff
I keep getting this error : AttributeError Traceback (most recent call last) in 10 try: 11 mix.iterate() ---> 12 if mix.loglike > best_loglike: 13 best_loglike = mix.loglike 14 best_mix = mix AttributeError: 'GaussianMixture' object has no attribute 'loglike' I am not sure what to do in this case. Any ideas? Thank you
Hi Siraj, I appreciate your videos and I love your content. I' am working on a project on cross-matching using active learning, what advice would you have for me? I' am trying to build something scalable but not so computationally intense.
Hey Siraj, I have vectors with 10 components, thus 10 features. I labeled the vectors by 4 classes. I wanna use GMMs to calculate the probabilities for a new incoming vector belonging to each one of the classes. What do I use? Do I have to create a GMM for every class? If yes, how to model a GMM to a 10 feature vector? Or could or even should I use Multivariate Gaussian Distributions instead?
hey siraj ! EM is a heuristic with no guarantees for global convergence. there have been recent algorithms based on method of moments, random projections etc. which provably recover the gmm under some assumptions
You guess a theta ( model params) , then that gives you a probability distribution of the hidden variables. With that known, you maximize the joint probability distribution of X and the hidden variables. That gives you a new theta. Repeat the 2 steps above: use the new theta model params instead of your guess.
Hey Siraj! Just found your channel and it doesn't cease to amaze. I am learning a lot about AI and ML with your vibrant and enthusiastic expression. My 2 cents would be to talk a tiny bit slower but it is up to you. Congrats and Keep up the Good Work!
I am not being mean. But I think you should do more practice example and teach why you do what you do in that practice example rather than jamming up with multiple pointless equations which I have been following word by word. I think application is more important then theories you've been talking.
I am trying to use your notebook and getting this error -- any ideas?? I am getting an error for #checking the fitting process AttributeError: 'GaussianMixture' object has no attribute 'loglike'
Just five cents off real life. Get two dices. Get 10 glasses in a row and a bag of rice. Throw dices check the sum and put one rice in the glass with same number. Repeat it a lot. Get your real life Gausian/Normal.)
Hello Siraj, I am working on a project to extract the total bill from restaurant receipts. Is there any way that I could use CNN or any other deep learning techniques to achieve this. I am new to Ml and would greatly appreciate your suggestions.
Would be nice with timestamps, since it is quite impossible to find the bit of information about Gaussian mixture models that I was actually looking for...
Siraj I have a question/problem. I have two data inputs which is to be comparatively trained by a learning model. It's not a multiple set of data but only one. It's a set of pair of inputs. I have been reading pairwise svm. How do I do that? Is there a better model.
At 4:35, it appears that the score is nonnegative. Although a Gaussian distribution is a close approximation in this case, could a log-normal distribution also be used in a Gaussian Mixture Model? Are there advantages to selecting a Gaussian distribution instead?
3:44 Intro, Gaussian Distribution, Probability Density Function (PDF)
7:38 GMM Intro
9:08 Covariance matrix
10:15 GMM Definition, K Gaussians
11:30 How to apply GMM for classification
12:30 Problem statement, Fitting a GMM model, Maximum Likelihood Estimate (MLE)
13:58 Similarity to Kmeans clustering algorithm
16:13 Expectation maximization (EM) algorithm and difference to Gradient Descent
18:15 When to apply GMM, anomaly detection, clustering, object tracking
19:30 Coding example with Python
25:10 EM algorithm workflow in practice, Log Likelihood
27:54 EM algorithm visual / walkthrough
36:30 Summary
great video, many Thanks :)
The butt kissing ends at 3:40
Thanks. Haha
warning: when he finger styles his hair, get ready for hardcore info dump.
PS: 3blue1brown series on linear algebra has THE BEST vid on eigen vectors/value pairs, no joking.
Love the motivation at the start, preach!
Siraj never fails to inspire, and I agree with his point strongly - we are the most important community in the world today. We all have a common goal, of making the world better with the best tech we have to offer. I for one am working on a universal translator not just for spoken languages, but for sign, braille and more. ML and NNs has moved my research forward by at least a decade.
awesome thanks Adam!
In case you have bad results using Gaussian mixtures, keep in mind the EM optimization only has local convergence properties, just like gradient descent: it can get stuck. Restarting the the density estimation with other initial parameters might solve it ! :)
thanks Jason!
Your accent reminds me of Mitchell from Modern Family(fav character) :')
Also great video thanks!!
Siraj, I think it would of been helpful if you showed the resulting clusters that you get from the gaussian mixture model approach in your data. You showed how to model your data using the gaussian mixture, but I am unclear on how we get the specific clusters (say 2 clusters) from that?
the iteration function is empty, which makes the current code completely random, it should be "mix.Mstep(mix.Estep())" inside that function
Like he understands that
you are the best source of ML... thanks for your attention(s) and love to AI!!!!!
Thank you very much! Your explication is very good and educative! I'm recommending your channel to my friends too.
Clearly explained the concept!!! Great presentation
Those are really strong motivating words in the beginning :). Thanks.
Love this video. It presents so clear.
When my friends ask me how to start with machine learning and AI, I tell them Siraj is the way to go! Thanks for making the AI community so cool! Yes we are the COOL GUYS!
hell yeah! thanks
Made no sence. Better try comming up with more and more examples rather than just reading slides.
Great presentation about GMM !! Thanks
From a muddy blur to crystal clear in 30 min, thank you very much for this video Siraj
Super tutorial! Thank you so much!
Can you please control your moving hands data points? too much distraction.
Where do I get the dataset? It is not mentioned anywhere and is not in Github repository either
Dataset can be found at: raw.githubusercontent.com/brianspiering/gaussian_mixture_models/master/bimodal_example.csv
Hi Siraj Raval, we love you from Tunisia
suggestion at time 6:45 minutes, the y values aren't the probabilities of the x values, intuitively the probability for a single point on the gaussian will be 0.
pdf does not give the probability. It gives the probability density at that x
i just loved the energy :D
Awesome work Siraj
So... 38 minutes to predict something and he just forgets about the prediction part?
I'm sorry, but the justification on 36:02 is not enough for my satisfaction.
Great video, I tried running your code on my terminal and it's giving the error that 'GaussianMixture' object has no attribute 'loglike', would you happen to know why an error like would occur, or anyone by that matter. Thank you so much
I got pretty confused around 33:33 with the E step. You've computed wp1 and wp2, which is cool, and then normalised them so their sum is 1 [wp1/(wp1+wp2) + wp2/(wp1+wp2) = (wp1+wp2)/(wp1+wp2) = 1], which makes sense. You then add the log of this sum to self.loglike. But the log of 1 is 0... Which is where you lost me.
You are right! Siraj should check and fix that with RUclips annotations.
Agree
6:45 y is not the probability. y is the "likelihood" because the probability function is a pdf.
yeah probability is the area under the curve
1:30 to 3:40 - talk trash
I keep getting the following error though:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
in
8 #train!
9 mix.iterate(verbose=True)
---> 10 if mix.loglike > best_loglike:
11 best_loglike = mix.loglike
12 best_mix = mix
AttributeError: 'GaussianMixture' object has no attribute 'loglike'
Hey Siraj thank you. If you ever come to México, you'll have a room, a meal, a beer and a friend :)
whether wp1 + wp2 = 1 always...so self.loglike += log(wp1 + wp2) will be zero ????
Is it true ?? whether my assumption is wrong ??
Kindly explain...
He makes mistakes... If only that was the only one... Referring to Variance as Variation... Doesn't know how a Standard Deviation is calculated... omg.
Thanks for reading theory to me. Couldn't do that by myself
I know you're being sarcastic, but honestly, I'm looking for people to do just that for me, I HATE reading technical material.
I watch 4-5 vídeos of you per day. I'm Learning generative models for drug Design Siraj. Watch your videos not only motivates me, also makes my life & study fun and cool.
Thank you for this great lecture and video...
Great series!!!! even helps me in my AI learning curve at Udacity. Thanks for it. rgds tibor
where is the mathematical explanation?
Apple sends their hinge prototypes to this guy for testing. If this guy won't wear out hinges, who will?
Another great tutorial, Siraj. By the way, if anyone gets an error seaborn deprecation and being unable to plot histograms, I resolved it by going into the linked 3rd party lib seaborn/distrubutions.py and changing "normed" to "density"
You're great! Finally I understood GMM.
5:45 +siraj "whether it's a car or roller coaster that's increasing in velocity reaches a peak then decreases or a soundwave... very likely a Gaussian distribution would be a great model"...????? Isn't the bell curve representative of the frequency of the data, not the data itself??
It's always great and informative to watch and learn from your video.
But my question is a non technical, but do provide a solution plz...
Question : I saw your github profile, and I'm curious what filters you applied on your profile pic(dp) ?? :p
ps: I already told you this question is going to be a non-technical one and Yes !!! you have been on my youtube's subscription list from the very beginning.
Cheers !!!
siraj, my guy.. this is so 🔥. will you be in Amsterdam sept 4-16 ?
6:45 "y values are the probabilities for the x values." NO!! y values are decidedly NOT probabilities.
Loved the explanation. If I have to model 6 features instead of 2, and use a sliding windows approach on my dataframe (I need to find the anomalous windows), how can I modify the weights and the rest of the code? Just looking for direction.
I have the problem with the gaussian mixture models, I don't know how generate outliers uniformly in the p-parallelotope defined by the
coordinate-wise maxima and minima of the ‘regular’ observations in R?
Hi all! I've fixed some bugs from the original .ipynb so you are welcome to try my one: github.com/ivanpastukhov/Gaussian_Mixture_Models/blob/master/intro_to_gmm_%26_em.ipynb
Siraj, thanks a lot for your brilliant videos!
Thank you so much Ivan!
Relating EM to K-means set off an epiphany in my mind. Thanks for that, it really helped clarify EM like it it never did in school.
8:30 "x is the number of data points"? What are you talking about?!
You're the best! You've helped turn this 19 year old from a lazy kid into an inspired workaholic
so amazing! Keep it up
same! although I am 15 though
Thank you! Your videos helped me a lot... I was so lost and confused about this topic that I was on the verge of giving up. Checked out your tutorials that gave a lot of useful information and insights. Thanks a tonne! :) :D Keep up the good stuff
I keep getting this error :
AttributeError Traceback (most recent call last)
in
10 try:
11 mix.iterate()
---> 12 if mix.loglike > best_loglike:
13 best_loglike = mix.loglike
14 best_mix = mix
AttributeError: 'GaussianMixture' object has no attribute 'loglike'
I am not sure what to do in this case. Any ideas?
Thank you
7:39 "Sometimes the data has multiple distributions". WTF?! This video is deeply misleading. Please take it down.
Hi Siraj, wonderful video! I am wandering what is the difference between Gaussian mixture model and least square method in the data fitting' view?
Here, x1, x2... are the vecors or are the data points of a vector x?
25:22 EM model
Hi Siraj, I appreciate your videos and I love your content. I' am working on a project on cross-matching using active learning, what advice would you have for me? I' am trying to build something scalable but not so computationally intense.
Hey Siraj, I have vectors with 10 components, thus 10 features. I labeled the vectors by 4 classes. I wanna use GMMs to calculate the probabilities for a new incoming vector belonging to each one of the classes. What do I use? Do I have to create a GMM for every class? If yes, how to model a GMM to a 10 feature vector? Or could or even should I use Multivariate Gaussian Distributions instead?
Hi, your videos are great!. Please cover VGG, Alexnet, and others sometime.
thanks Aamir!
Code doesn't work, There is a problem in GaussianMixture class.
hey siraj ! EM is a heuristic with no guarantees for global convergence. there have been recent algorithms based on method of moments, random projections etc. which provably recover the gmm under some assumptions
good job!
Hey Siraj! Come down to Mumbai for some beers and nerding out!
Hey Siraj, Where will you be meeting folks in India?
when I do your codes couldnot find data file error? Why? how can find it?
33:30
wp1/(wp1+wp2) + wp2/(wp1+wp2) = 1
log(wp1 + wp2) = log(1) = 0
How is his model being trained?
You guess a theta ( model params) , then that gives you a probability distribution of the hidden variables. With that known, you maximize the joint probability distribution of X and the hidden variables. That gives you a new theta. Repeat the 2 steps above: use the new theta model params instead of your guess.
We actually try to get the value of log(wp1 + wp2) =1 not (wp1 + wp2) to be 1.
Great Video! Really helpful for Data scence students..
The hand gestures as always are a little bit distracting to me by the way..
Wow! Finally I got my head around this subject. Well done and amazing teaching skills 👏🏻
Andre
thank you for Gaussian Mixture
Handsome. I like you teaching
We love you Siraj
R u a Gay... Bcz u r reacting like that... Plzz... Stop overacting and try to speak normally.. Like a civilian not like karan johar
hahah
Thanks for the vid. Owe ya one big broth
Where I can get the blog he is following?
I love how passionate you are about this
Does anyone know what the mix value is ?
Really thanks man, your video helped me a lot in my Hyperspectral Images classification project's
Hey Siraj!
Just found your channel and it doesn't cease to amaze. I am learning a lot about AI and ML with your vibrant and enthusiastic expression. My 2 cents would be to talk a tiny bit slower but it is up to you. Congrats and Keep up the Good Work!
thanks Kashyap!
Video starts at 03:40
This guy does not know what he is saying. K-means does not have the EM algorithm but only the E part.
That was a lot longer than 30 seconds....
Siraj's desktop background has the Sierra mountains, but doesn't OS Sierra not work with Tensorflow and OpenAI and other machine learning stuff?
Hi! How can we select the number of components?
I am not being mean. But I think you should do more practice example and teach why you do what you do in that practice example rather than jamming up with multiple pointless equations which I have been following word by word. I think application is more important then theories you've been talking.
I am trying to use your notebook and getting this error -- any ideas??
I am getting an error for #checking the fitting process
AttributeError: 'GaussianMixture' object has no attribute 'loglike'
Just five cents off real life. Get two dices. Get 10 glasses in a row and a bag of rice. Throw dices check the sum and put one rice in the glass with same number. Repeat it a lot. Get your real life Gausian/Normal.)
Hello Siraj, I am working on a project to extract the total bill from restaurant receipts. Is there any way that I could use CNN or any other deep learning techniques to achieve this. I am new to Ml and would greatly appreciate your suggestions.
Awesome !!!!!!
You can visit us in Uruguay! Everyone is welcome in Uruguay and especially, people who motivate the world to be better, like you @siraj!
How does he manage to have brains, AND a personality? Next question, does that mean he's a psychopath? It doesn't help; only time will tell :)
Ridiculous
If
we add to the covariance matrix the gradient decent of the covariance matrix will the result stay positive definite?
Would be nice with timestamps, since it is quite impossible to find the bit of information about Gaussian mixture models that I was actually looking for...
lolll I read about this model and though, Jeez, maybe I should send a message to Siraj to explain this to me. Well, nvm, he already did. Thanks man!
Siraj I have a question/problem. I have two data inputs which is to be comparatively trained by a learning model. It's not a multiple set of data but only one. It's a set of pair of inputs. I have been reading pairwise svm. How do I do that? Is there a better model.
At 4:35, it appears that the score is nonnegative. Although a Gaussian distribution is a close approximation in this case, could a log-normal distribution also be used in a Gaussian Mixture Model? Are there advantages to selecting a Gaussian distribution instead?
@Siraj
, why do you change the formula at 29:54? instead of sigma^2 you are using abs(sigma).