The distributional parameters are not preserved. If X and Z are iid Normal (10,2), and Y=rX + SQRT(1-r^2)Z where r is the target correlation, then Y is effectively a weighted average of X and Z, each of which have a mean of 10. But the weights sum to more than 1, so the mean of Y will always be more than 10. For example, suppose that r=1/2. Then SQRT (1-r^2) is the square root of 1/2, which is SQRT(2)/2, which is about 0.707. Then the sum of the "weights" is 0.5+0.707 = 1.207, and Y will "overshoot" by 20.7%, on average. To make this work, all of the Y values would have to be multiplied by 1/(1+SQRT(1-r^2)).
Can you elaborate on why the correction factor is needed and the motive? I found through a simulation that even with your correction factor, the correlation coefficient still over(under)shoots by +/-35%-50%. Basically, I simulated 2 scenarios and ran 50 trials whereby the 1st is the correlation equation as presented in the video and the 2nd is by correcting each and all Y values with your correction factor. And in my view, I don't think you should modify the correlation equation Y by your 'corrected' factor since it relates to matrices at the end of the day.
Dear Dr. Verschuuren, great video, it's very useful for my work. Two questions: 1- I used the first formula with X=(lognormal random numbers) and Z=(normal random numbers) and after some attempts I found the new Y distributions in agreement with my requirment (mean,std,r), is it a good approach? 2- Regarding the first formula, could the numbers generated of new variables (Y) be considered as random generated and used to stochastic models by Monte Carlo method? 3- Do you have some references (for my thesis bibliography) about that formula (Y=r*X+sqrt(1-r^2)*Z)? Thank you in advance
Hi Anacoda, Hope you are doing well after 5 years! Just wondering if you have solved your problem. TBH, I am facing kind of the same problem as you have described. Could you give me some suggestions?
I am not sure if the matrix multiplication formula in column J,K,L ( at 7.28 minutes in the video) should use "TRANSPOSE" against the Cholesky decommpositn matrix, as the Cholesky Decmposition is alreday transformed in the VBA code. I may wrong though. Please correct me.
I am not sure myself, but I guess as the decomposition ist transformed in the VBA code u havet to transpose again in order to get the not transposed decomposition part. Watch this video it describes the mathematical background of the Cholesky decomposition. ruclips.net/video/QCqsJVS8p5A/видео.html
This approach generates the desired correlations but the variables no longer follow the initial marginal distributions. What if we need the random variables to have the same mean and std dev?
In the last part of the video, I believe the transpose on the Cholesky decomposition is not needed and would provide you with better results that more closely resemble the original correlation matrix. Can someone please check?
0':35'', how the formula : Y = r X + sqrt (1- r^2) *Z is determined? If only want Pearson correlation (X,Y) = 0.9 , why need Z? why not just set Y=0.9 X ? Thank you in advance!
This appears to be an incorrect solution. I copied the formulas exactly and set the correlation to .6. I ended up results above .6 3/36 times. In short, it appears to skew to the left. When I used .9 as the correlation with a large data set, it never came close to approaching .9 (hovering around .7). I am uncertain why this is incorrect, but I have a suspicion that it is due to two factors. First, correlations are bounded at 1. Assuming a normal distribution around a set out for .9, 1 will be the limit to be over the correlation, whereas two items can be far below .8. Moreover if things go "over" one they are instead at a lower correlation than 1. This brings me to the more important point: I think the fundamental logic behind trying to create correlated variables through multiplication is incorrect. Instead one would need to use Z scores, std deviations etc. I am not sure precisely how to do this mathematically or in Excel, but will post results if I can find the answer.
I found the error. You start with two standard normal distributions X1 and Y1. Use the formula above so that Y2 = r*X1 + SQRT(1-r^2)*Y1. Now X1 and Y2 will be correlated by r. To give each their own mean and variance simply multiple X1 by expected mean and std, and same with Y2 to generate X2 and Y3 that are correlated by r and with their mean and std.
Could X, Y, and Z be three different independent distribution, such as X= normal, Y = beta and Z=uniform? Or would they have to be a random set from the same distribution? (x,y,z=norm)
They can be different distributions if you like... but be aware that using the above approach results in correlations that are still coming from a normal or Gaussian copula (copulas are just functions that join marginal distributions together to form a joint distribution). If you like you can join your marginal distributions using other correlation structures (copulas) that give you greater control over the joint behavior of the variables (of course only go down this route if you have expert knowledge about the joint behavior).
Hello Dr. Geard Verschuuren, Thank you for the helpful video. It really helped me a lot. But there's one thing I can't get on with. What about the 'correlation aimed for' matrix? How do I get them? Thank you
Thank you very much! Among all the videos I watched on this topic this is very best one about Cholesky decomp and its application in Monte Carlo !
Brilliant, thankyou so much.
No video on youtube I could find on this topic!👍🎉
Thanks, I was just looking for the way of implementing this in Excel for my own Multivar Monte Carlo ! 👍
The distributional parameters are not preserved. If X and Z are iid Normal (10,2), and Y=rX + SQRT(1-r^2)Z where r is the target correlation, then Y is effectively a weighted average of X and Z, each of which have a mean of 10. But the weights sum to more than 1, so the mean of Y will always be more than 10. For example, suppose that r=1/2. Then SQRT (1-r^2) is the square root of 1/2, which is SQRT(2)/2, which is about 0.707. Then the sum of the "weights" is 0.5+0.707 = 1.207, and Y will "overshoot" by 20.7%, on average. To make this work, all of the Y values would have to be multiplied by 1/(1+SQRT(1-r^2)).
Can you elaborate on why the correction factor is needed and the motive?
I found through a simulation that even with your correction factor, the correlation coefficient still over(under)shoots by +/-35%-50%. Basically, I simulated 2 scenarios and ran 50 trials whereby the 1st is the correlation equation as presented in the video and the 2nd is by correcting each and all Y values with your correction factor.
And in my view, I don't think you should modify the correlation equation Y by your 'corrected' factor since it relates to matrices at the end of the day.
The name of the method is Cholesky decomposition by the way!
en.wikipedia.org/wiki/Cholesky_decomposition
Dear Dr. Verschuuren, great video, it's very useful for my work. Two questions: 1- I used the first formula with X=(lognormal random numbers) and Z=(normal random numbers) and after some attempts I found the new Y distributions in agreement with my requirment (mean,std,r), is it a good approach? 2- Regarding the first formula, could the numbers generated of new variables (Y) be considered as random generated and used to stochastic models by Monte Carlo method? 3- Do you have some references (for my thesis bibliography) about that formula (Y=r*X+sqrt(1-r^2)*Z)? Thank you in advance
Hi Anacoda, Hope you are doing well after 5 years! Just wondering if you have solved your problem. TBH, I am facing kind of the same problem as you have described. Could you give me some suggestions?
@Liu Fengyuan Hi Liu, unfortunately it is long time ago, so I don't remember very well how I solved the situation. I am sorry. Good luck for your work
I am not sure if the matrix multiplication formula in column J,K,L ( at 7.28 minutes in the video) should use "TRANSPOSE" against the Cholesky decommpositn matrix, as the Cholesky Decmposition is alreday transformed in the VBA code. I may wrong though. Please correct me.
I am not sure myself, but I guess as the decomposition ist transformed in the VBA code u havet to transpose again in order to get the not transposed decomposition part. Watch this video it describes the mathematical background of the Cholesky decomposition. ruclips.net/video/QCqsJVS8p5A/видео.html
I think you are right. I had the same doubt when I saw an upper triangular matrix as the result of the Cholesky decomposition.
This approach generates the desired correlations but the variables no longer follow the initial marginal distributions. What if we need the random variables to have the same mean and std dev?
Copula is the way then
In the last part of the video, I believe the transpose on the Cholesky decomposition is not needed and would provide you with better results that more closely resemble the original correlation matrix. Can someone please check?
0':35'', how the formula : Y = r X + sqrt (1- r^2) *Z is determined? If only want Pearson correlation (X,Y) = 0.9 , why need Z? why not just set Y=0.9 X ?
Thank you in advance!
This appears to be an incorrect solution. I copied the formulas exactly and set the correlation to .6. I ended up results above .6 3/36 times. In short, it appears to skew to the left. When I used .9 as the correlation with a large data set, it never came close to approaching .9 (hovering around .7).
I am uncertain why this is incorrect, but I have a suspicion that it is due to two factors. First, correlations are bounded at 1. Assuming a normal distribution around a set out for .9, 1 will be the limit to be over the correlation, whereas two items can be far below .8. Moreover if things go "over" one they are instead at a lower correlation than 1. This brings me to the more important point: I think the fundamental logic behind trying to create correlated variables through multiplication is incorrect. Instead one would need to use Z scores, std deviations etc. I am not sure precisely how to do this mathematically or in Excel, but will post results if I can find the answer.
I found the error. You start with two standard normal distributions X1 and Y1. Use the formula above so that Y2 = r*X1 + SQRT(1-r^2)*Y1. Now X1 and Y2 will be correlated by r. To give each their own mean and variance simply multiple X1 by expected mean and std, and same with Y2 to generate X2 and Y3 that are correlated by r and with their mean and std.
@@SpicyDijon You multiply by the standard deviation and add the mean.
@@rodolfooviedo6542 yes sorry thats right
@@rodolfooviedo6542 Could you please explain further? This approach didn't work in my test example.
Thank you very much sir.
Could X, Y, and Z be three different independent distribution, such as X= normal, Y = beta and Z=uniform? Or would they have to be a random set from the same distribution? (x,y,z=norm)
They can be different distributions if you like... but be aware that using the above approach results in correlations that are still coming from a normal or Gaussian copula (copulas are just functions that join marginal distributions together to form a joint distribution). If you like you can join your marginal distributions using other correlation structures (copulas) that give you greater control over the joint behavior of the variables (of course only go down this route if you have expert knowledge about the joint behavior).
Hello Dr. Geard Verschuuren,
Thank you for the helpful video.
It really helped me a lot. But there's one thing I can't get on with. What about the 'correlation aimed for' matrix? How do I get them?
Thank you
Great video just want to ask, why is it i try Cholensky() cell function. in my excel it does not work?
He created it. You have to create it as a custom module yourself within vba
How did you highlight your mouse pointer?
1:14 Can you tell more about the formula? Where does it come from? Can u provide your excel file via a download link?
it is just a cholesky decomposition of the covariance matrix. If you want to know more, www.numericalexpert.com/blog/correlated_random_variables/
if you found any reference you can provide it to me, please
This guy has an annoying voice, but the content is fantastic. Well done. Very helpful.