The expression for the expectation and variance in terms of log(Z) shows up again in statistical mechanics. There the energy and energy fluctuations are derivatives of log(partition function), precisely because the distribution over microstates is in the exponential family with t(x)=E and theta=beta. Great stuff!
The party is just getting started! This topic is one of my favorites, but it's quite technical, so there were never any expectations of getting a lot of attention. But those familiar with stats definitely appreciate it
Lol I don't like my old videos because they have this awk-as-hell hand shit. I was listening to some garbage 'talk with your hands' advice for RUclipsrs.. ugh!
I think you can technically form uniform distributions within the family, it is just uninteresting since you have to choose your set beforehand and the distribution would be effectively parameterless: Choose h to be an indicator function of some set A and t to map everything to 0 or, even simpler, be zero-dimensional. This gives you probability density 1/nu(A) 1_A.
This one matters a lot! And I’m not familiar with any other classes of distributions that come close. A lot of theorems/tools are designed for the exponential family, just b/c they have such convenient properties
Does it make sense to analogize the exponential family to overloaded functions in computer science? Something like: “given some inputs that determine the search space and sufficient statistics, return the appropriate distribution.” This a very lucid explanation, by the way - I’ve bookmarked the video for future reference!
Thank you! And to answer your question, I don't quite see the analogy your referring to in the first sentence, but the second sentence seems fair to me. If the inputs are the decisions for t(x), h(x) and v(x) and the function your referring to searches for the theta* according to the data, then returning theta* would amount to returning "the appropriate distribution". So I think you're right
Really good video that help me understand the concept of exponential family. However I just have one small confusion. In a normal distribution case, why t(x) needs to be x and x^2? Because to me having an x will tell us the x^2 value and thus knowing x should give us the probability distribution via theta. I know to form the PDF for normal distribution you need x and x^2. I just can’t find a good logic behind this.
I see you're point. The way I would think about x and x^2 is they are measures of the same thing.. such that their probability depends (basically) **linearly** on these measures, using a fixed parameter vector. Let's say you want to only use x. Could you write the normal density as a linear function of *only* x?? That's what you can't do... and that's why x^2 is needed as well.
Is that a Gaussian process regression model on your desktop background? I appreciate the high-quality content and the visuals. I just wouldn't call this p(x|\theta) = 1/Z(\theta) h(x) exp(t(x)\theta) an equation, because asking for a solution of this doesn't make any sense.
Yep, that’s a GP and in fact that’s my next video coming out (sometime near the end of this week). And yea, you’re right. It’s an expression, not an equation. I’ll keep that in mind.
Love your content, but it would be nice if you could sloooooow it down a bit. You feed us a shit-ton of information in 15 minutes. I find that noob creators tend to unnecessarily zip through their material. I personally like the pace of 3b1b & Trefor Bazett.
I love your content and explanations but i find you being side-by-side with the blackboard explanations very distracting, I think it’s gotta do with the hand gestures combined with intonations. Maybe thats just me. All love though, your content is superb.
I agree in fact. My new format (see my most recent video) makes the latex more front and center, and that's how things we'll be going forward. It's all a work in progress
Can't wait for part 2!
This video is so amazing. Great visuals, really nicely explained.
This explained the concept and inner workings of the exponential family so much better than my lecturer did. Keep up the amazing work!
The pace for the bullet points was perfect, it's clear the effort to get it to flow so well!
Ha yea getting the timing right is a challenge. Still a work in progress, but glad people are noticing g
Here because of mCoding and honestly I’m baffled. Amazing content, amazing lecture, your channel is going to blow up. All the best!
You mCodings folks are likely half my followers! What a bump :)
The expression for the expectation and variance in terms of log(Z) shows up again in statistical mechanics. There the energy and energy fluctuations are derivatives of log(partition function), precisely because the distribution over microstates is in the exponential family with t(x)=E and theta=beta. Great stuff!
Thanks, DJ! I was coincidentally trying to learn about these in Bishop but my eyes just glazed over. These visualizations are great!
Thanks! Glad it helped and more coming!
Thank you for delivering the concept so clearly!!!
Happy to! Nice to see this very technical topic getting some love
I am late to the party. But you deserve so much more credit and attention. This answered all the questions I had and then some.
The party is just getting started!
This topic is one of my favorites, but it's quite technical, so there were never any expectations of getting a lot of attention. But those familiar with stats definitely appreciate it
This is amazing. So well explained
This is amazingly done, thank you so much!
Thank you
eager to see part 2.
Lovely video! Thanks a lot. Waiting for part 2..
Your explanation is the best on internet
That is quite a compliment - thank you very much
Thank you for your help
Your content is gold!
Thanks a lot!!!
Is DJ practicing boxing? He naturally guards up constantly.
Lol I don't like my old videos because they have this awk-as-hell hand shit. I was listening to some garbage 'talk with your hands' advice for RUclipsrs.. ugh!
Glad I found this channel
thank you so much for making these videos
mannn this is phenomenal
Thank you Yi! The exponential family is a beautiful thing
Crazy good video, really! Thank you so much
I love it when someone likes my harder videos!
Wow this was crystal clear! looking forward to watching your other videos thank you
And this is an old one! Glad you like it. I'm actually thinking about reshooting it..
@@Mutual_Information I'll be looking out for it if you do, keep up the great work!
You are a cool guy for this explanation, mr. Mutual Information!
can't wait for part 2 :D
Thank you very much - please do!
If math stats books had added intuitive explanations like this we would have been able to solve climate change by now.
That's quite a compliment - thanks!
Keep up the great work man!
The details are crazy! But i guess the exponential family is not completed yet. I might come back here a few months later
Brother, love from India!
U r my HERO!!!!
lol should I wear a cape next time??
bruh how do you have so few views???? amazing videos!!
lol RUclips just can't handle me
Can you please also do a video on deep exponential families?
I think you can technically form uniform distributions within the family, it is just uninteresting since you have to choose your set beforehand and the distribution would be effectively parameterless: Choose h to be an indicator function of some set A and t to map everything to 0 or, even simpler, be zero-dimensional. This gives you probability density 1/nu(A) 1_A.
HOLY SHIT UR CHANNEL IS AMAZING BROTHER, just a CS grad interested in the secrets of AI. P=NP tho
Wow, never would have guessed that they were somehow related. Is this the main family that matters or are many more?
This one matters a lot! And I’m not familiar with any other classes of distributions that come close. A lot of theorems/tools are designed for the exponential family, just b/c they have such convenient properties
Does it make sense to analogize the exponential family to overloaded functions in computer science? Something like: “given some inputs that determine the search space and sufficient statistics, return the appropriate distribution.” This a very lucid explanation, by the way - I’ve bookmarked the video for future reference!
Thank you! And to answer your question, I don't quite see the analogy your referring to in the first sentence, but the second sentence seems fair to me. If the inputs are the decisions for t(x), h(x) and v(x) and the function your referring to searches for the theta* according to the data, then returning theta* would amount to returning "the appropriate distribution". So I think you're right
woah, this was awesome..
It's nice when someone can appreciate the harder videos.
Nice lesson, but 4K please! Even if your camera is 1080p, the rendered math would look sharper.
Yes, that was a more recent change. I was less familiar with camera best practices when I shot this.
Sometimes it takes only one video to make people subscribe 😂.... thank you
I was actually thinking of re-shooting this one. I'm glad it still works for some!
Really good video that help me understand the concept of exponential family. However I just have one small confusion. In a normal distribution case, why t(x) needs to be x and x^2? Because to me having an x will tell us the x^2 value and thus knowing x should give us the probability distribution via theta. I know to form the PDF for normal distribution you need x and x^2. I just can’t find a good logic behind this.
I see you're point. The way I would think about x and x^2 is they are measures of the same thing.. such that their probability depends (basically) **linearly** on these measures, using a fixed parameter vector.
Let's say you want to only use x. Could you write the normal density as a linear function of *only* x?? That's what you can't do... and that's why x^2 is needed as well.
That gestures xD
haha Old terrible habit.. Rest assured, I feel shame
Is that a Gaussian process regression model on your desktop background?
I appreciate the high-quality content and the visuals. I just wouldn't call this p(x|\theta) = 1/Z(\theta) h(x) exp(t(x)\theta) an equation, because asking for a solution of this doesn't make any sense.
Yep, that’s a GP and in fact that’s my next video coming out (sometime near the end of this week).
And yea, you’re right. It’s an expression, not an equation. I’ll keep that in mind.
Suggestion: Set your playback speed to 0:75 if you don't like to feel like a dumb idiot
ha yea these older videos are too fast. Amateur mistake on my part, but the new stuff is better paced!
@@Mutual_Information I was feeling frustrated and commented a bit too harshly. It was a great explanation overall.
3Blue1Brown's little brother ;)
Love your content, but it would be nice if you could sloooooow it down a bit. You feed us a shit-ton of information in 15 minutes. I find that noob creators tend to unnecessarily zip through their material. I personally like the pace of 3b1b & Trefor Bazett.
I love your content and explanations but i find you being side-by-side with the blackboard explanations very distracting, I think it’s gotta do with the hand gestures combined with intonations. Maybe thats just me. All love though, your content is superb.
I agree in fact. My new format (see my most recent video) makes the latex more front and center, and that's how things we'll be going forward. It's all a work in progress
Cool video and all, but why are you wasting precious energy by having two monitors and a computer on idle while filming?
Ha never thought of that. If it makes any difference to you, I turn the AC and fridge off since they make noise. So maybe that nets out :)
@@Mutual_Information I don’t mind, perhaps your wallet and/ or the planet do 😅
how to map this form
en.wikipedia.org/wiki/Exponential_family#:~:text=This%20yields%20the%20canonical%20form
with your equation?
eta = theta and log Z(theta) = A(eta)
@@Mutual_Information why not subtract A(eta)? How you get 1/A(eta)?? exp(x-a) = exp(x)/exp(a)