How autocorrelation works
HTML-код
- Опубликовано: 29 июн 2024
- Part of the End-to-End Machine Learning School Course 212, Time-series Analysis at e2eml.school/212
To use autocorrelation in a weather prediction model, check out the Time-Series Analysis project course:
end-to-end-machine-learning.t...
Slides are available: docs.google.com/presentation/...
Follow me for announcements and update: / _brohrer_ 
Наука
Well explained! Very natural transition from topic to topic until I accidentally learned autocorrelation.
Woah, thank you, Sir. This short tutorial gets me to understand this entire auto-correlation thing than 10 page pdf explaining with all kinds of formulas .
Thank you very much. This give the insight for ACF, not just mindlessly calculating them
Came to revise Autocorrelation. Ended up revising regression too. Great video.
The kind of explanation I was looking for! Thanks!!!
One of the best videos about acf
Autocorrelation 7:02
Partial Autocorrelation 9:45
Brandon's videos deserve a lot more likes and views. Very underrated.
Cool description, very detailed, with lots of graphs
Great explanation with such a good example.
Easy to understand too.
Thanks!
Best video on this topic. Thank you!
I watched this 2 years ago. Didn't understand the concept. I watch again in quarantine days.
Another great and super clear explanation! Thanks.
This was explained excellently!
Very good video. In Numerical Methods it always helps to visualise what you are doing
Very nice and clear video!
Great video! I liked how you presented the python code that would perform what you're explaining.
best explanation found till now
I really liked the autocorrelation part!
Really helpfull
And straight forward
U r genius man
this is best available in you tube thanks it was very helpful
fantastic explanation
Nice and clear explanation. I will add at 4:30. That the points can fit the line very well but there is no correlation. It is the case when the fitted line is parallel to the x-axis or y-axis. It is the case of near zero variance variables.
nice and concise video!!
Thanks a lot, view the code helpful to understand quickly
Good job on this video!
I wish it would have showed up when I searched for information on ARIMA, all the ones that did were pretty poor.
Auto correlation sounds useful for finding data cycles. Like every year temperature correlation, around ~365 days shift~, there is probably a max again. You can find cycles in shifts where correlations peak.
Also thanks for showing how it can be refined the same way as tailor series. Didn't cross-concept that correlation can give differentials weight for interpolation.
Instead of using list slicing it is a bit easier to use the "shift" method from the pandas library. The video in general is great :)
This is great. Thanks
Very nice and explanatory video! Thanks! One thing that you could also do is mention real-world applications. A toy model is always very helpful for me to *understand* a topic but real-world application helps me to also *motivate* it.
Besides that keep it up your videos are super nice! :)
I am also currently learning so I'm not sure if my idea is entirely correct, you could use this for any data set, to see how it changes. You could use the auto-correlation on gas prices to see how it changes with time, on flight tickets to see how it changes on a day to day basis or even on the price of a big mac to see how it has changed over the years.
Thank you!
wonderful!
Thanks a lot!
Thank you! :D
this was super helpful
THANK YOU
Sounds like George Clooney explains it
Thanks !
Close your eyes and you can hear Matthew McConaughey explaining autocorrelation. The only thing missing in here "Alright, alright, alright"
he literally sounds nothing like him
I think you might be conflating 2 different concepts: 1. de-trending the timeseries before forecasting with AR, 2. partial auto-correlation.
For example the first de-trending step may not be just fitting a straight line, it could be non-linear or have seasonality etc. This is unrelated to PACF which teases out what marginally new information are gained with each auto-correlation step
do you know how to do a white noise time series to test for seasonality?
nice and clear explanation for this useful concept! for me its just not clear how to determine whether or not a value is still whithin the 95% confidence interval? R is giving some limits but there it remains unclear as well how to calculate those boundaries..
Thanks darmastadvideo! This is important, but unfortunately didn't make it into this video. I'll have to save it for a future one.
I think you have small error on screens, please 11:02 , on left graph should be "temperature on day i-1"
How to compare two voice signal similarity in python?
Intuitively, how does one gather the data for the temperature scatter plot for autocorrelation? This is my only stumbling block.
To see a detailed example in practice (including step-by-step data collection and organization), check out e2eML course 212: e2eml.school/212
Give this video a like
can i have the data sample sir on the basis of which you are calculating thanks
Is the autocorrelation with zero lag always 1? Because I've read that Gauss-Markov processes have the following equation for autocorrelation:
autocorrelation(lag) = variance*exp(-constant*lag)
In the case of this formula, autocorrelation(0)=variance (!!!) why is it different in this case?
no question about this. with no lag, you're basically computing the correlation of a variable with itself. Any algorithm that gives you something different than 1 in this case simply fallacious.
im confused! So when you talk about correlating the shifted (lagged) dataset. Lets say in an AR(1) process you shift the Y values by 1 day, does that mean the Y-1 dataset will have 1 less data point than the original Y dataset? And by extension if you specify an AR(10) process are you then knocking off 10 data points and hence correlating two datasets where one dataset has 10 less data points than the other?
Sorry if i explained that like a moron, maybe this will help:
let's say
y = [1,2,3,4,5,6,7,8,9,10]
y-1 = [1,2,3,4,5,6,7,8,9]
y-2 = [1,2,3,4,5,6,7,8]
y-3 = [1,2,3,4,5,6,7]
y-n = [.........]
so for example if i'm interested in specifying an AR(3) process, to determine the autocorrelation would i just find the correlation coefficient between y & y-3:
y = [1,2,3,4,5,6,7,8,9,10]
y-3 = [1,2,3,4,5,6,7]
i am confused as to whether the fact you are correlating a [1,10) with a [1,7] is correct? or an issue? or what!
sorry this was really long!
yes .
Think of it like this , when you are using a lag of 1 , you are comparing every day value to its previous day value therefore automatically for day 1 you won't have anything to compare it to. But it doesn't matter , the data points left are enough to give an accurate correlation value.
Dhirendra Singh amazing, thanks so much for the response. I have since experimented with some data and it makes much more sense now! Cheers mate
Great video! Except:
Traceback (most recent call last):
NameError: name 'temp_change' is not defined
;)
At 4:20 in the video a solution is provided.
Hi! Great video but there is an error at 5:24 - The correlation is not negative if just the slope of the line is negative. Correlation is negative if the data points and the line have converse slopes.
%matplotlib inline # JuPyter Notebooks
import matplotlib.pyplot as plt
import numpy as np
n = 10
x = -np.linspace(0, 1, n)
y = x + (0.3 * np.random.rand(n)) - 0.3/2
neg = np.corrcoef(x, y)[0, 1]
plt.plot(y, marker='o', markersize=4, ls='None', c='k')
plt.plot(x, label='negative ({:.2f})'.format(neg), c='r')
plt.legend(); plt.show()
No error there, video is correct. you are plotting x and y independently on the same plot, not y against x. correct code should be:
plt.plot(x,y)
good but i think the way you represent the days shift in the code using [1:] and [:-1] is wrong (i do not get the meaning) or please explain what you meant. tks
It isn't wrong, its just python notation to slice a array. Just google it :)
Bad
Nice video, the best one I've seen yet about the concept of partial autocorrelation!
"degrees" is not a unit
a degree is a dimensionless unit of measurement
Very very nice explain .. many thanks
🙏
Wow thank you
Great explanation, Thank you.
Nice
thanks for the great video, @4:57 should the greater than and smaller than signs be flipped in the moderate correlation plot ?
Thanks mohammed, you're exactly right :)
🤓
temp_change is not defined in your code is it? Something is missing there.
The graph at 10:46 and the graph at 10:52 are the same but the x-axis is labeled differently. Which is accurate?
And does the correlation coefficient, r, from the regression performed on the graph at 11:02 give you the point on the PACF (2, r)? And the graph at the right I'm confused by. If you are regressing the series with lag 2 against itself it would be a perfect fit. Why are there residuals at all?
@@zagwask78 doesn’t answer questions, sad :(
Very nice demonstration!
You covered regression better than videos on regression 👏
6:48 you'll pull off??
Thanks for the video!
In partial autocorrelation, the graph for shift 2 is incorrect, it has to show the points tilted along the 1st fit.
👏
Thank you for existing.
Amazing video, but the autocorrelation loop contains a bug. for shift in range(10) will start from zero, and when you temp[ : -0] you will get an empty array, and your code won't work, because you will do a corrcoef of an empty matrix with another that contains data, thus, the correct loop will be something like: for shift in range(1,10)...thanks a lot for the vid
Great Video
This is simply excellent. I haven't seen a better explanation of this concept.
very good video!! Thank you
Hello Brandon,
thank you for the explanation. Especially the code helped to understand whats happening.
I'm glad to hear it Bartosz. Thanks.
Excellent
And this is where the info on autocorrelation starts: 6:57
You're an hero, bro.
cool !!!
Cool! You get a coordial thanks from me.
You are most welcome
I have just now seen a simple and easy video to check auto correlation assumption with Durbin Watson test ruclips.net/video/ZDMQo6ID0sg/видео.html. It was very useful for my research work
This is amazing, thank you
Thank you :)
Appreciated
Stellar explanation!
Poor Explanation
Amazing explanation, subscribed!
Thanks Carlos :)
Very clearly explained!
Great video
Many thanks!
great video man
really helpful video, thank you for this
Damn. So mf nice!!! 10 stars
Aw, thanks Marlon. I'm really happy it clicked.
Amazing video! Very useful. Thanks!
The definition of autocorrelation is not correct. For temps=first 100 natural numbers, I get autocorrelation value as 1 for all different values of 'shift'. And that's not true. Autocorrelation is given by the formula: www.itl.nist.gov/div898/handbook/eda/section3/eda35c.htm
The relevant Python function for autocorrelation is np.correlate.
great video
If in the time series , the residuals on temperatures day i after fiting temp(i-1) are correlated with actual temperatures of day i , then can it be said the there is partial auto correlation with lag beyond (i-1). If that method holds good ,then it could be much easier method of detecting PACF.
But I have liked your video also.
It is not clear in the video how much residual of the previous periods to be added to the time series equation of the present period.
Where you use the word "shift", when describing autocorrelations, wouldn't it be better to use the word "lag", as I've seen this appear in literature more often and am now wondering whether they're the same thing
they are used interchangeably, as well as the word "delay"
Excellent explanation, thank you!
Thanks yung!
This is the best video ever on autocorrelation! Thank You!
Thanks Ani!
Brendon, I would so love to learn ML from you. teaching code along with concept would definitely decrease the MSE between ignorance and knowledge. Thanks
Thanks Siddarth! In case you're interested, there's a whole collection of concept-with-code courses at e2eml.school/catalog
I was looking for good video to understand the concepts and this video has done that. I have the following questions:
- Can we do correlation to compare stock market bottoms or tops? If we compare the market price by its 1 day 1 week prior price then will this help in prediction of top or bottom?
- How can we compare two asset prices? I am thinking of below methods
- Do a ratio of two asset prices and run the n-1 vs n comparison of the ratio
- Do a ratio of two asset prices and compare the ratio vs one of the asset prices
- Compare the % return of each assets against each other on X and Y axis
Let me know what do you think