Inverse Transform Sampling : Data Science Concepts
HTML-код
- Опубликовано: 8 фев 2025
- Let's take a look at how to transform one distribution into another in data science!
Note: I should have included a lambda in front of the exponential PDF. I mistakenly forgot it. I appreciate the comments which helped me realize this mistake.
---
Like, Subscribe, and Hit that Bell to get all the latest videos from ritvikmath ~
---
Check out my Medium:
/ ritvikmathematics
This video just transformed my life
pdf of exponential is (lambda)*e^(-lambda x)
correct
Great video. It made the underlying concept crystal clear. Thanks a lot, Ritvik.
Just saw two 20 minutes long videos before this, none made me understand this at all. Then, I saw this 10 minutes long video of yours and it made this subject so much clearer than before. Amazing professor, congratulations!
Great to hear!
Thank you so much for doing these videos.
I can't agree with you more.
That was a great intuitive explanation of inverse Transform Sampling. It seems so easy to me after watching this video,. Thanks a lot.
It only seems easy. Inverting the cdf is difficult. The exponential distribution is kind enough to let itself invert, but many other ones are mean.
omg you saved my stats degree, much thanks
I really want to thank you because your clear explanation helped me get an A in my statistical programming exam. You are a hero.
I can't thank you enough. You have been of help in many subjects from time series analysis to this. I would like to see EM algorithm, latent class models, and hidden Markov models in the future.
You solved a big curiosity I had. I learned about the power of MonteCarlo analysis and how easy it is to get a uniform distribution from Excel, but knew I would always need more specific distributions. So the question was how to get any distribution from a set of randomly generated numbers from the usual Excel Rand() generator. Thanks for the brilliant and easy demonstration! Congrats for your terrific work!
Great to hear!
You have brilliantly and simply explained a topic that I have been struggling with for a whole semester. Thank you so much! :)
Glad it was helpful!
Excellent video, super clear! One thing on the maths side, when you do the final derivation step at around 7:28. I think you should do the following, since you have F_X(x) = T^-1(x) = u, you then apply the inverse of F_X to the left and right hand side to get x = F_X^-1(u), i.e. we take the CDF inverse of the uniform sample.
Love how calm you are. I'm shitting myself when I have to explain topics like these to someone.
Trying to wrap my head around this in class but to no avail. Thank you so much for your amazing explanation
Thanks for the video! I was struggling to understand the motivation behind it, but your explanation has made it much easier for me :)
Glad it helped!
João, veja meu comentário acima para um exemplo de aplicação.
Thanks a lot! your videos on DS and Stats is the best!
Thanks for the video! I'm still struggling with this, but your explanation definitely helped!
Thank you!
He is going to be a fabulous professor
Haha I appreciate the kind words :)
Just fantastic ! keep it up man great videos and great explanation
Brilliant teacher! I guess it is a sort of gift.
Brilliant!!!!
thanks!
Super helpful, thank you very much!
nice! Thanks for the work. Like the way you explained concepts in a straightforward and smooth way. Please keep it up ! :)
Great video! Your exponential density is missing it's normalizing constant, though. Since your CDF is correct, no harm , no foul, but it might confuse some people.
Yep, I'm looking for the lambda*e^ -(lambda*x).
Well explained .. thanks. One minor suggestion .. if there is a way you can make the video screen capture friendly or leave a screen capture slides to the video, that would be super helpful. Thanks for the clear presentation.
This was super pedagogical, thank you very much.
Captivating!
u r best teacher ever
very clear explanation, thanks for sharing!
Quite an amazing explanation. Well done!!
thanks for good explanation about Inverse Transform sampling !
Glad it was helpful!
finally i understood, thank you
I love your explanation always produce the best please don’t stop
Excellent explanation! Keep ut the good work!
Greatly explained! Thanks!
Thank you. Our instructor did not explain it and just gave the theorem. I was confused like I have three heads.
Yo Ritvik not sure if you still remember me we talked during orientation (I was the guy work with Tasty). We had a class last week about MCMC and I was confused about certain parts and RUclips directed me to this video lol. Great job man keep it up. Hope we can catch up when things get back to normal after the pandemic
That was awesome. Thank you !!!!!
no problem!
Great explanation! Thanks.
Glad you enjoyed it!
Great video, it really helps me out a lot. One thing I still dont really understand is why we might do this. As in, why would we use the inverse transformation method to find the exponential random variable instead of just using the exponential PDF directly if we have lamda?
Hey - great video! I think you might have forgotten the lambda in front of the exponential for the exponential PDF. If you calculate the CDF from what you have written you will get a 1/lambda factor.
Yup you’re definitely right !
Thank you so much. You help me a lot with a homework I have.
Brilliant, thank you so much.
you saved my life
That subtle pen flip at 5:49.. Damnn
LOL
@@EubenM You replied :DD Great videos man! Your channel is awesome :DD
Hi Ritvik, great videos! would be interested to have a set of videos explaining variational bayes, ELBO etc. in order to perform bayesian optimisation on hyper-parameters
very helpful, thanks for this video!
Thank you, this helped a ton! :)
Glad it helped!
Excellent. !
Great Explaination! Thanks so much :)
Thank you for this, it has helped me a lot! :)
Thank you for this
great explanation, thanks a lot!
Can you please do a video about Copulas? For example in a (credit) risk management context
Thanks a lot bro, so helpful
great explanation
great video! thanks!
hi, is there an error with the PDF function? f(x) = lambda * exp^(-lambda)(x)? thank you for this video!
Hi! I loved the video, but I've got a question. What are the cases when the CDF is not invertible? And what are the strategies then? Should we try to make the CDF invertable by interpolating it or should we use another random variate generation technique? Thank you in advance! Happy New Year.
Happy new year! And great question, indeed this technique is good only if you can find the inverse of the CDF, so if that is not possible, interpolation is a great idea as long as the fit is "good enough"
You can use the generalized inverse of the function. This is a function g such that g(y) is the infimum of the x such that F_X(x) >= y. Since F_X is a continuous function from the right this is always a minimum. So this function is such that F(g(y)) =y, it works like the inverse and the difference is that if there are other values with the same image you take the least of them and you can always do that. This is the same function to calculate quaintiles, so Q_{0.5} = g(0.5). Take in account that g(0) = -infinity and g(1) = infinity, to get the values right. More information here en.wikipedia.org/wiki/Probability_integral_transform
Thank you 🙏
could you make more advance time series tutorial? really like your videos and i'm struggling in grad level time series course
More time series vids coming up soon!
Well explained. Thanks a mil
What is the role of lambda? I've seen other videos that don't include it, so now I'm curious. Amazing explanation btw!
Very helpful thank you !
thanks, this is a great video!
Thanks for sharing. Just one small comment....pdf of the exponential is lambda*e^(-lambda*x).
Nice lecture!
thank you so much sir, I would like to know which probability distributions commonly used that we use inverse method with.
Great video. Quick question: at the end of the video, you said we could swap 1- u for u. That means 1 - u = u, which translates into u = 1/2. Yet u is a random variable, it is not necessarily 1/2, right? What am I missing?
Good question! We are not swapping 1-u for u in an algebraic sense (in which case you would be absolutely correct). Rather, we note that u is a uniform random variable between 0 and 1. Therefore 1-u is also a uniform random variable between 0 and 1. Thus, it does not matter (in terms of probability) whether we use 1-u or u. And using just u makes the formula look a bit nicer.
@@ritvikmath Oh. I understand. RVs are not really variables. When it comes to RVs, what matters is not the specific value of the RV, yet it is the distribution of the RV that matters. Since u and 1-u are both RVs with the same distribution, they are interchangeable.
Gem level bhau
Lets say u = 0.25. Then 1 - u = 0.75, right? Could someone explain how 1- u = u in the uniform distribution?
Up
Actually the magic for this inverse transform to work is the equation P(T(U)
Thank you!!!!!!!!!!!
this guy is epic!!!
perfect!
excellent
Thank you man
Very nice video! thank you!
The uniform should be (0,1] without 0 right? so the ln will be defined.
Thanks!
Great video. Could you do one on copulas, building on this one?
Then, why is important the uniform pdf?. I mean you could sample directly from one distribution to another just by putting the value returned from the CDF of the first pdf as input to the inverse CDF of pdf you want to arrive at. Am I wrong?
I have question about the math, on how to derive other inverse transformations especially for datasets that predict number of clicks on a web page for example. Some of them are tricky and might even need estimation by iterative numerical methods or ML, because the Poisson is simple to find the inverse function for. And then how do you put the inverse transform into an sklearn pipeline exactly? Here's why I ask this: Sometimes I am using a Generalized Linear Model which provides a convenient link function already built-in, but we are not always going to just use a linear model as we might need to use for example the large feature vectors that an NLM model is producing to describe some text. GLM is not necessarily the only tool to consider. Besides for random sampling, Transforms are also good for ML preprocessing and postprocessing pipelines to help your model learn easier. The log(Y) and e(Y) are the Poisson distributions transformations when your response Y is a count. Quasipoisson and Negative Binomial are good for count data when the mean and variance are not staying equal as the Poisson requires, but instead are showing some overdispersion or underdispersion. There's also zero inflation model which combines a logistic model and a Poisson model together in sort of an ensemble to help pre-predict the count = 0 case when 0 appears a lot more often than plain old Poisson can account for alone.
Thx body u the best
I don't get it why the exponential distribution is called memoryless? Yes, I know that that lambda or hazard rate is constant but isn't that just the speed or rate of the probability (not the actual probability because the lambda can be more than 1). From the exponential PDF, you can clearly see that the chances in the early phase are bigger than in the later phases so why is it called memoryless? If I sampled time to failures, should I get more numbers early than later because of that decreasing curve?
Are there any resources I can look at to understand why it's valid to assume that p(T(U)
TLDR: The distribution of the CDF of _any_ PDF is uniform, so if you want to sample from a PDF that has an invertible CDF, you can sample from the uniform distribution and convert it to the desired distribution with the inverse of the CDF.
In data science can we transform weibull distribution into Gamma or poison distribution?
Not very proficient in statistics, but in sum, if I do the transformation and have the final function, given a number u that is randomly generated from a uniform distribution, I will get an equivalent randomly generated number that falls under an exponential distribution?
great video, I will subscribe and continue to watch them!
this is super clear and I can go to bed very happy
If I understand correctly, the reason why uniform distribution is used because its output range from 0 to 1. Just out of curiosity, can we use beta distribution to replace uniform distribution?
Was hoping to hear more about motivations for WHY i need to know this method for DS. "that's how computer gives you random samples from a distribution" is not enough to care about it. What about cases where maybe I don't have a pdf or it cannot be integrated or I get only proportionality of pdf (like in Bayesian model) so I can't just plug in the variable into the proportional pdf and get accurate samples..... maybe that's when I need to use this method.....
I appreciate the feedback!
Having a hard time understanding EWMA and GARCH model ,can you make some videos introducing them?thx
GARCH is coming up soon!
Can you do (or recommend) a video on Granger Causality?
Thanks for the suggestion! I'll look into it
I probably missed this moment: why transformation to CDF actually gives you desired distribution?
in order to find the inverse of CDF, we just find the value of x..why? in other word, how come x is the inverse of CDF?
if the distribution we want is not the exponential distribution, are the steps are still the same？
So if I had some other distributions apart from exponential one, I just need to derive its inverse, and set the number of simulations I will like to do with a U that is unif from 0 to 1? I just need clarification in that part.
I have a question if we graph the inverse function of that exponential function how it will looks like? whether it looks similar to graph of uniform distribution? otherwise how this can be equal?
its not a ''datascience'' method (which sounds like it comes from modern era). it is known as smirnov method who lived around 1900 and likely known before
there is an error for the pdf of the exponential distribution, the lambda is missing.
Question : Hi, i have rainfall data as a 2d matix/frame of the UK every 5 minutes so the data is spatially and temporarily correlated. The data has severely positive skewness. Around 90% of pixels or points are less than 10 and 10% between 10-128. When i train a cnn, it is only predict rainfall of low values because of the data imbalance. I would like to transform to uniform distribution. I tried log transformation which compressed the data but still there is imbalance. Do you know how to convert to a uniform distribution so all of the values have the same chance to be predicted? It is a regressio task to predict the next 12 frames of rainfall. The data is represented by only one continuous variable, rainfall intensity. Many thanks