L12.5 Covariance

MIT RES.6-012 Introduction to Probability, Spring 2018
View the complete course: ocw.mit.edu/RES-6-012S18
Instructor: John Tsitsiklis
License: Creative Commons BY-NC-SA
More information at ocw.mit.edu/terms
More courses at ocw.mit.edu

[@jeffreyzhuang4395]

Covariance shares conceptual similarities with the vector dot product. The dot product helps us quantify how similar two vectors are. Likewise, covariance provides a measure of how two variables change together.
To illustrate, consider a set of N samples, where each sample has coordinates (x_i, y_i). Let's form two feature vectors:
a = [x_1, x_2, ..., x_N]
b = [y_1, y_2, ..., y_N]
To calculate the covariance, we first center the vectors by subtracting their means:
a' = a - mean(a)
b' = b - mean(b)
The covariance is then calculated as:
dot(a', b') / N

[@brandomiranda6703]

summary notes:
- since in the scatter plot most of the time positive values tend to be with negative the product x*y is positive, so the expectation E[X*Y] is positive
- similarity in opposite
- so Cov shows if the centered r.v.s tend to have the same sign on average
- fact:
- indepedence => Cov(X,Y) = E[X*Y] = 0
- converse not true i.e.
- if E[X*Y] =!=> indepdence
- example provided:
- since the values are in the axis the product X*Y is zero no matter the distribution
- but knowing the value of say X=-1 tells you deterministically that Y=0 or any value of X!=0 tells you Y is not zero deterministically, so X & Y are not independent

[@thomasstoubos3029]

nice job!

[@YouAreTheRaidBoss]

What is the definition of zero-mean in this case? Trying to google it doesn’t lead to the desired answer

[@rezafarmani5858]

Nice

[@Econ_Macro]

Nice video. At the end, you should add that it is the equation of a circle: (x)^2 +(y)^2 = 1. The covariance is equal to zero, but there is a relationship between x and y: (y)^2 = 1-(x)^2.

[@gallenlesleifelicen8176]

haha nc vid

[@nileshkharat236]

4:33 why is E[X - E[X]] = 0?

[@camiloernestodiazvasquez4825]

Try to do it with a set of two numbers or three. You'll see it right away ;)

[@korggg123]

1) E(X-E(X)) = E(X-u) [where u is mu, a notation for average]
2) E(X-u) = E(X)-E(u) [by linearity of expectation]
3) E(X)-E(u) = u - u = 0 [E(X)=u by definition and E(u) is just an average of a constant mu, which is also just mu = u]
On a side note, if 2 R.V weren't independent, then we wouldn't be able to separate covariance into the multiplication of 2 expectation terms of different R.V.s [E(X-E(X))*E(Y-E(Y))]. We would have to expand the original definition of covariance using linearity of expectations.
E((X-E(X))*(Y-E(Y))) = E(XY-XE(Y)-E(X)Y+E(X)E(Y)) = E(XY)-E(X)mu(y)-mu(x)E(Y)+E(X)E(Y) = E(XY)-mu(x)mu(y)-mu(x)mu(y)+mu(x)mu(y) = E(XY) - mu(x)mu(y) = E(XY) - E(X)E(Y), which is non-zero.

[@nileshkharat236]

@@korggg123 thanks

[@moritzwolff9383]

@@korggg123 How do we know that if X and Y are independent then also X-E(X) and Y-E(Y) are independent?

[@nipunkothari5518]

​@@moritzwolff9383 because we are just shifting X and Y by a constant, it doesn't have any value on the overall distribution, so they remain independent