Multivariate Time Series Forecasting Using LSTM, GRU & 1d CNNs
HTML-код
- Опубликовано: 6 окт 2021
- Thank you for watching the video! Here is the Colab Notebook: colab.research.google.com/dri...
I offer 1 on 1 tutoring for Data Structures & Algos, and Analytics / ML! Book a free consultation here: calendly.com/greghogg/30min
Learn Python, SQL, & Data Science for free at mlnow.ai/ :)
Subscribe if you enjoyed the video!
Best Courses for Analytics:
---------------------------------------------------------------------------------------------------------
+ IBM Data Science (Python): bit.ly/3Rn00ZA
+ Google Analytics (R): bit.ly/3cPikLQ
+ SQL Basics: bit.ly/3Bd9nFu
Best Courses for Programming:
---------------------------------------------------------------------------------------------------------
+ Data Science in R: bit.ly/3RhvfFp
+ Python for Everybody: bit.ly/3ARQ1Ei
+ Data Structures & Algorithms: bit.ly/3CYR6wR
Best Courses for Machine Learning:
---------------------------------------------------------------------------------------------------------
+ Math Prerequisites: bit.ly/3ASUtTi
+ Machine Learning: bit.ly/3d1QATT
+ Deep Learning: bit.ly/3KPfint
+ ML Ops: bit.ly/3AWRrxE
Best Courses for Statistics:
---------------------------------------------------------------------------------------------------------
+ Introduction to Statistics: bit.ly/3QkEgvM
+ Statistics with Python: bit.ly/3BfwejF
+ Statistics with R: bit.ly/3QkicBJ
Best Courses for Big Data:
---------------------------------------------------------------------------------------------------------
+ Google Cloud Data Engineering: bit.ly/3RjHJw6
+ AWS Data Science: bit.ly/3TKnoBS
+ Big Data Specialization: bit.ly/3ANqSut
More Courses:
---------------------------------------------------------------------------------------------------------
+ Tableau: bit.ly/3q966AN
+ Excel: bit.ly/3RBxind
+ Computer Vision: bit.ly/3esxVS5
+ Natural Language Processing: bit.ly/3edXAgW
+ IBM Dev Ops: bit.ly/3RlVKt2
+ IBM Full Stack Cloud: bit.ly/3x0pOm6
+ Object Oriented Programming (Java): bit.ly/3Bfjn0K
+ TensorFlow Advanced Techniques: bit.ly/3BePQV2
+ TensorFlow Data and Deployment: bit.ly/3BbC5Xb
+ Generative Adversarial Networks / GANs (PyTorch): bit.ly/3RHQiRj
I offer 1 on 1 tutoring for Data Structures & Algos, and Analytics / ML! Book a free consultation here: calendly.com/greghogg/30min
I just wanted to let you know that I went through many LSTM tutorials over the Internet and yours is the one that got me understanding it all. Thank you! And let me give you a compliment you probably didn't get yet: the way you talk is very easy to understand for a non-native english speaker like me!
I'm really happy to hear that Felipe! Thanks for such a nice comment :) :)
After scourging through the entire RUclips, I finally landed on a great video that helps me get the idea of how to implement RNNs. I am trying to tackle a problem in my side project that involves RNNs and this video is exactly what I needed. Thanks for the lovely explanation, Greg!
Oh I'm so glad to hear that!! I hope your project is going okay!!
@@GregHogg Yes, still to get the data in a sequential format. But man, you helped out a TON!!
Greg you really explain so nice. Finally I landed up to a solution to problems in my project. Thank you so much for providing this knowledge for free. God bless you, your family and your RUclips channel. Lots of love from India.
Thank you so much Sanjeev, I appreciate that a ton and am really glad to hear I've been helpful to you 😊
Thank you so much! This video is by far the best one to show how to fit multivariate LSTM.
You're very welcome, and I'm super glad to hear that!
Very simple yet comprehensive. Thank you!
Glad to hear it.
Best dl time series tutorial by far.. well done Greg!
Great to hear, thanks a ton!
It was very helpful. Thank you Greg
That is awesome video! Thanks for sharing this wonderful material.
Thanks Rafael, I really appreciate that!
Please continue the series , amazing explanation
I learn a lot just from this video, thanks mentioning all the details as they are always the confusing parts!
Glad to hear that, you're very welcome!
Greg thanks for sticking with this even to the point of how tired you were getting at the end of it. You are a gifted lecturer.
If I were to add anything it would be more about the need for and the technique of pre-processing the data.
Hahaha thanks so much! Yeah some of these ML tutorials are kinda crazy lol. And yes great point :)
Great video Greg! could you please share some ideas on how to adapt the models to forecast several values (hours) ahead, not only just one? thank you.
Interesting, fascinating detailed explanation. Keep it up!
Thanks so much, I really appreciate that :)
marvellous. learnt a lot!
Thank you Greg for this awesome video. To think that i have watched this video severally and i just got my "A ha moment". I am currently working on a project on Forex prediction using LSTM and this will help me alot. Thank you again.
It's definitely a confusing topic for sure. Well done and great job!! You're super welcome, and good luck on your project!!
Hello @@GregHogg, I trust you are doing great? I am working on a multivariate time series forecasting and i am getting the same values for my predictions, do u know why this is please and how I can resolve that? Thank you
Thank you so much i was very confused about the multivariate ( multiple features ) to use as input. You cleared my doubts. Thank you
Glad to hear it! :)
@@GregHogg one question, i have a client request. Let's talk in your case my client wants me to use temp column as 1 variable input and another column will be scaled temp column as 2nd variable. And use both of them to predict temp. I don't think using another column of same data but scaled will do any better than just using one of those.
Any suggestions?
Great video. Very informative. You mentioned doing one predicting stock prices. It would be good to see if the human emotional influence of stock prices could be modeled. Seems very difficult. Have you already done one? Thanks
Love it!! Thank you!!
You're very welcome :)
Just coming back to reference some stuff. great vid!
btw if you're dealing with large datasets your df_to_X_y2 function might be a bit slow. This should speed it up significantly and give the same result.
"""
def split_data(data, n_steps):
x, y = list(), list()
for i in range(len(data)):
end_ix = i + n_steps
if end_ix > len(data)-1:
break
seq_x, seq_y = data[i:end_ix, :-1], data[end_ix-1, -1]
x.append(seq_x)
y.append(seq_y)
return np.array(x), np.array(y)
"""
takes in a numpy array as data. if you're dealing with a dataframe just input df.values, n_steps is the window size. this is more for a classification task as it uses the last column of the input as the target (y=target) and thus wont include the target in the output x array. can tweak as you'd need ofc.
Invaluable information thanks man!
Thank you and you're super welcome!
Thank you so much for your this tutorial! God bless you! Regards!
Thank you!!
excelent tutorial, thanks you very much
No problem!
@Greg Hogg Very interesting video!Super helpful for my current project!Just wanted to understand why aren't we standardizing y_train,y_val and y_test?
Not necessary. The model doesn't know that y_train is one value in the future of X_train. It's just trying to discover the system behavior. As long as the inputs are on the same scale, that's really all the model cares about, and will produce outputs that are on the same scale as the target.
Thanks Greg... Wonderful explanation. Really helped in understanding the Forecasting logic. Just one question. What is in the last model, we need to forecast Temp and Pressure together for next 10 observations may b? What changes we need to do in order to get, Multi-step Multivariable Forecast problem??
Thanks Once Again !!!! ✨✨✨
Brilliant video Greg!
How would you treat months in cyclic terms as days in a month varies over time?
Amazing video! Thank you so much for this awesome guide
Thank you and you're very welcome 😁
@@GregHogg Thank you so much for replying to my comment, These guides were extremely helpful, You're awesome! ❤️
Hi Greg, great video! But i've got a question about your model. Are you predicting temperature based on historical values of temperature alone? Or are u using extra variables to predict the temp, like for example pressure. Because i understand that you only predict both of them only. But do the predictions have any correlation? Is the pressure variable correlated with the temperatuur model prediction?
Thank you!
Same question in my mind. Like to know how to do the LSTM with 2 or more features (variables) to get the label (result). Anyone here can help?
thanks for the great videos man
You're very welcome!
This is the best video I’ve seen on this topic - well done. One question: why didn’t you also standardize your validation data in the example of temp and day/year sun/cos?
Thank you very much for your good video
It was very, very useful
Great to hear! You're very welcome 🙂
Hello, ty for your content! :-)
I have a question: you used the X2_train for the mean and std at 38:56 is it because you want the predictions later on to have the same basis when preprocessing?
This video is insane!
thank you for the video!
Thanks for the kind words Ihsan!! You're very welcome!!
Thanks for your videos, massive work!! I have a question : Whats happened if instead of having just one time serie of temperatures, you have many temperature time series for many countries ? How you do the split? do you put 'country' and 'date' as indexes ? and what's happened if you dont have the same length for each country ?
AWESOME VIDEO !!!!!!🤩🤩🤩
Hi Greg. Thanks for the useful video. Let me ask you a question. Can we use a Transformer model (Attention All you Need) for this dataset? as you know Transformers receive all data sequences together!
Thank you first for the great video, very helpful
I have a question :
I developed a temperature control system with a specific set point using arduino in a small greenhouse and I logged data in a SD card. Now I wanted to develop an LSTM model that I can implement it as a predictor controller in my greenhouse.
my question is to know after training the model and get the prediction accurate, what to do in order to convert that model and use it as a controller?
and how to proceed for the implementation?
Looking forward to hear from
Thank you
great. thank you.
I love u brother. U are a great teacher
I really appreciate that 😍
Great and understandable video. You have a beautiful voice btw
Why thank you
Hi Gregg,
nice video and explanation of your code.
One question: Why do you standardize your train, validation and test set with the same mean and std from the trainings set?
Thank you! The same preprocessing must be done for every input. It's just something that ML folk usually agree upon.
Great video Greg! For the preprocessing and postprocessing, I was wondering why did not we do it at the start on temp_df itself before we split it up in [train, val, test]. In that case, it would be much easier to keep track of it, right?
I think that's because this would represent a form of data leakage, i.e. you would be using data from your validation and test sets (mean and std) to train your model.
Thanks!
This has been so helpful and maybe to ask what did you consider to choose the input and output layers
Input has to match your features, which has some flexibility. Output has to be whatever you feel like predicting
Thanks for the awesome video. A fucntion preprocess_output3 will take mean and std of the y's, isn't it?
Merci a lot 😊
I am going through a hiring process and a test project related to time series forecasting. Happy that I found your video series on this topic. Thanks!
Great to hear. Best of luck! You're very welcome :)
thanks, very useful
You're very welcome!
This is cool, but I'm not really that impressed. While the agreement seems impressive at first glance when looking at the comparison plot, what we are doing here is plotting our estimate of the *next* temperature based on a recent historical window of data. That means that at the next time step, our model is getting the values from the previous step. In other words, the model is always making a prediction only one time-step in the future. A more helpful evaluation of its performance would be to plot and compare the difference between the last and current temperature data between the actual and estimation. Another way to understand why I'm skeptical is to consider doing a control experiment using a much simpler estimate of the temperature: a linear interpolation of the last two temperatures. If we plotted the results the same way, I suspect my linear interpolation method would also look extremely good. Really, the valuable kinds of questions aren't just whether the estimate at the next time is close to the actual value, but rather if the change in temperature from the previous time-step to the current time-step is predicted accurately -- does this difference have the same sign between the prediction and actual values? Does it have a similar magnitude?
Hello Greg,
Hope you are doing well.
Question: How we can design a machine learning/NN model for time series forecasting in case its a multivariate forecasting and also at the same time if some of our independent variables are regressor's.
Thanks
you are a legend 👊
No you are
Thank you
Great video!!
Thanks so much!!
Great video - this is exactly the level of detail I’ve been looking for!
Quick question- why do you set the window size to be 1 + number of variables? Is it bad to have a smaller window size then the number of variables? Thank you
Great to hear!
Did I say I wanted it as 1 + number of variables? Thinking right now, I can't see why these things are related
@@GregHogg just to not confuse between window size and parameters numbers in the jumpy shape
This was awesome. I got through both the previous video on LSTM time series and this one using data from Hugging Face. Thanks for the great content
Great, great content. This is my personal introduction to tensorflow, and I really want to thank you because the explanation is crystal clear. I really appreciated the detail and also how you are gradually putting more stuff in modeling, so it's easy to follow along.
I have just one question: I can't understand why (about at 45:00) when going through the df_to_X_y3(df, window_size=7) you are expecting two output values for the target variable y. Shouldn't we have expected a single output from the 6 variables (now that df['p (mbar) was added into our p_temp_df dataframe), with window_size=7 (the size of the sliding window for creating sequences)?
Thank you very much!
42:30 - Now I see where the second output comes from: also the pressure is included as the output ;)
Hello, Greg! I've watched several videos from you, they're all great and easy to understand!
I am doing a similar project now and there is a question (maybe a silly one): how do you prepare X to really predict the future values (let's say to predict temp and pressure from today to the next 7 days, and there is no real data of temp and pressure in X set (I am asking because I think both pressure and temperature are in X now). Let's imagine that the model is already trained and fitted from the train and the validation sets and there is no "test set" for today
In my project, I am trying to predict demand, still very similar :)
I too have same doubt how to predict future
Great vid !
Thanks!
Hey great video. Curious why you didn't use StandardScalar before splitting the data into train/test. Thanks.
I believe it will lead to data leakage. The data should be separated before, so that test data have no connection to train data whatsoever :)
hi Greg. Why do you stardardization temperature in this video?
In part 1, why you didn't do that?
Thank you so much
Is this standardization process help us increase the accuracy? or this is the rule that we have to follow?
Why are we processing the output when using the pressure column too? Why not when we were just using the temperature?
Thanks for the great explanation Greg.
I was wondering why for model4 we scaled only the X_test and X_train data (and did not scale the y_test y_train data) before we fed it into the model?
We only scale the data that will be used to train the model. The y value is the output we want to predict therefore we do not scale y.
Thank you. This is very informative and clearly explained.
Very nice trick with the periodic sin/cos feature construction.
Do you possibly plan to explain more about the different feature construction tricks?
I'm using a fit_transform from sklearn for data preprocessing (which is virtually the same as your preprocessing functions, I guess).
The question is - couldn't it be easier to apply preprocessing just once to the entire dataset instead of test/train/validation subsets each time for ease of inverse transformation for prediction?
You're very welcome. In the future, this is planned. Fit transform will be fine for temperature and pressure for sure. You could probably preprocess the whole dataset, yes.
No, the point of the validation and test set are supposed to be an unseen dataset. Therefore when preprocessing the data you should only have access to the training data. Therefore you should always FIT the scaler (Min/Max or Standardization or etc) on the Training data and then only use TRANSFORM on the validation/test datasets.
Never fit your scaler on the validation or test set, as this will result in the validation and test set ("unseen") data leaking into your training data and having a bias in your model.
@@cassiusvlok6916 You may be correct as i have read this in a few papers as well. Now guys pls i need help. I am working on a multivariate timeseries with LSTM, i have 28 variables as input. Input to LSTM is a 3D as expected, i am however having trouble doing inverse_transform on my predictions.
Also liked the sin/cos feature construction trick, but I have a question. Why convert to a sine wave as opposed to having a value between 0 and 1 where 0 would be the start of day/year and 1 woul be the end?
Wow this is exactly what I was looking for!
Can i ask something, how to predict for the next timestep? let say 24 hour (for hours timeseries) or 30 days? and can you give 95% prediction interval for the prediction.. thank you
Great videos. But I have been waiting for the continuation of future prediction of multivariate LSTM. Will you continue this video to run future predictions?
can we also use the other variables that are in the dataset instead of day sin, day cos, etc. ? which one is the better approach
When using 1D CNN layers for time series you have to use the parameter padding=“causal”, otherwise you will train a model on target t, with data from t+1. This is called leakage
Fantastic video, Greg! I found your video to be highly educational. However, I would like to seek clarification on a particular point. It appears that you're utilizing the present temperature as an input in order to predict the current temperature. Is my understanding accurate? If this is indeed the case, I wonder if such an approach might be considered less sophisticated. Perhaps the model's performance could be enhanced if the input values were temporally shifted and not directly included as features.
Additionally, it seems to me that you might be utilizing the prior prediction outcome as an input to forecast subsequent outcomes. This is just my interpretation, and I'm open to correction if I've misunderstood your methodology.
Amazing video! thanks helps a lot. A request to make a video on temporal fusion transformer model for time series analysis! Thankyou
Hi Greg, thanks a lot! I learn a lot. Can you please help me to understand why did you use Standardization for preprocessing of temp data instead of minmaxscaler ?
Great to hear! There's a lot of flexibility and sometimes arbitrary choices on preprocessing, I wouldn't worry too much about it :)
Nice Video! Question, after i have my model trained, do I have to normalize any new data i am using the model to predict on? So model.predict(new_data) is normalized as well?
I forget what we did to transform the training data, but whatever transformation we performed, that same transformation should be applied to any new data.
25:08 I understand why you turn the Seconds column (every increasing values) into something periodical, since the weather is also periodical BUT will this realy be beneficial since it´s probably the case that the periodic pattern of the temp missmatches the periodic pattern of sin/cos values ?
edit: Am I right in answering my own question with: "the temp_df[seconds] * 2pi/year input in sin/cos takes care of this concern" ?
Hi, useful and clear video for multivariate time series. Could u plz clarify me following usecase is feasible using time series data. Predicting student performance using past semester data for multiple students in a single model. We need to convert multiple student time series history to supervised for binary classification. Is this possible?Thanks in advance.
Hey Greg,
Good job, great service to the community!
What is your experience wrt prediction ability if you combine two outputs (make the neural network predict temp and pressure). I am thinking that during the learning process the weights will try to minimize the loss of both outputs. Then for example some weights may be better just for temperature, but not for pressure, what the algorithm would do?
I appreciate inisghts or a reference in this context, as in the past I have tried to predict two outputs from a system, but I trained two different networks.
Cheers,
You need to normalize the output so that it doesn't care. And thanks so much Carlos!
@@GregHogg Thank you for your prompt response, normalizing would help to make an even influence of both output. Still I think during the learning process not every weight adjustment would optimize the fitting of both outputs. Will try to find a paper discussing this topic.
Cheers,
@@betokrt I see no reason why it wouldn't try and equally balance both, but I might be missing something. No problem!
how to do future prediction sir. (should we append the predicted value to the window and the send that data to predict next value) should we repeat the same till time frame we want to predict in future?
Hi Greg! Thanks for this informative video. It helped a lot to understand. However, I have a question. Did you try any customized data loader for your technique? I try to use one with batch size of 32 which leads the numpy array to become (32,15000(length of my dataset),10(window),18(features). However, I am having problems, if you have any example available, I would be glad to see it. Thank you !
Brilliant tutorial. Thanks so much for putting it together. Any plans for implementing this in PyTorch?
Thank you! I recently posted something very similar in pytorch. Text follows this pattern, so I'll probably do something similar with nlp.
Very interesting , how to view this dataframe from plot_predictions , this dataframe will be a part from another dashboard on powerBI and start , end will be dynamic
Hello Greg, Thanks a lot for your videos and sharing your knowledge. One question if you allow me. Once we have the model trained, how to add new data (same kind of data) and train just the new data without have to train the whole history and predict using the already trained model + new data? does it make sense to do? thanks a lot!!
You have a couple options that make sense to me. 1. You don't, which is probably the easiest. 2. You train on all or some of the old data and the new data. 3. You train on just the new data using a very low learning rate
Nice work 😊
I love your work!! Let me know if you'd like to get on a call together to chat some time:)
@@GregHogg Definitely! You can hit me up by email or LinkedIn under the name “Ajay Halthor”
(Sorry I thought I typed this out a few days ago. Looks like I hadn’t :) )
Awesome will do! And np haha
Hello, can I split datasets like this? For example, window=5, one row of training data=[[t1], [t2], [t3], [t4],[t5]], target=[t7]. skip T6
A bit confused on 2 things.
1) We are trying to predict the temperature for a row of data. In our features, we include that row's temperature too. Isn't that data leakage/cheating since the label is in our features?
2) Let's say that it isn't data leakage, and what we are predicting is the next hour's temperature. Wouldn't our labels then be shifted one row back so the actual is in line with predicted?
Shouldn't the mean of all temperatures be like np.mean( X2_train[0, :, 0] + X2_train[1:, -1, 0]) ?
So we don't repeat any number in the mean?
How can you standardise data sets which uses the entire data set to calculate the Z value and yet, you are thrn scrolling through the data where future values used to make the Z value is not yet available??? Those values are not available to make out of sample set Z values as well. How can you then inverse out of sample results for causal prediction? What your Mu and Sigma be for a 1 step ahead prefiction with real data? You woild have to train youf model on a sliding Z value using only past data?
Great video Greg!! 1 question: i want to do classification on a numerical data set.. what changes do I make in your code? It is for a pattern recognition kind of project where I have to identify if the same data pattern is repeating itself
thanks in advance
You just have to make the output a sigmoid neuron :)
hey hi, its great and thanks for the video ...actually i have done same project as following your way...i have some doubts ...can you show me how to forecast for next three days form today?it will be helpful for me
Hi Greg, awesome video! I noticed you're using the data from the test set to make predictions on the test set. I believe this will be impractical when it comes to real world deployment of such model, because the assessment would have been wrong.
In the real world, we would want our model to make predictions using data from the past (train set), not from the future (val or test set) as we would have have had the future values then.
Can you provide insights on how that would play out when calling the model.predict() function? Thanks Greg, and I look forward to your response.
Great video! :)
Notes for me:
multivariate models at 17:16
multivariate sliding window data structure at 29:00
Shows CNN at 14:00
CNN multivariate 1:05:50
Thank you so much. May I ask a question about the 21:48 of the video?
I'm just don't understand why we ''need" the (2* np.pi / day) part in the day sin. Can it be just np.sin(Timestamp)?
or maybe just
np.sin((temp_df['sec'])/seconds_in_day)
Pretty sure the trigonometry functions work in radians - not degrees. So there are 2 * pi radians in a circle; he's just scaling the values against that maximum value. So if you're thinking in degrees, you would (erroneously) be trying to do something like sin(400 degrees) which doesn't make sense as there are only 360 degrees in a circle
Thank you so much
Thank you for your video, it is very expressive. Please, as the LSTM model is being trained, how can we predict data for the furtur years (e.g 2018-2030).
I am looking foward to your answer
best regards,
Clint
Glad to hear it! You can call it recursively and use its output as new inputs :)
Thanks for sharing this video!
I've tried to followed this tutorial to build a model to predict the closing price in electricity market but got poor result, perhaps the amount of data (
Hi, i am getting nan after using preprocess code of yours in my output variable.
Can you use Stock Prices?
Hi Greg! I wanna ask about the coding, so I copy and paste your code and run the function "plot_predictions1(model4, X2_test, y2_test)" but why I got the normalized values for X2 but I got the real values for the y2? please tell me if i missed something in your great video! thanks!
and i wanna ask about the preprocessing function too, why we only use the mean and the std from the x train when we use the function for x val and x test? we should be using the mean and std from x val for x val and from x test for x test right? or I missed something again? thanks!
If you run your cell with Shift + Enter, it automatically selects the next cell. If you are at the end, it automatically creates one. Took me months of using Colab before I finally figured that out! Hope this helps. :-)
Haha yeah I think when I made this I also didn't know, thank you :)
i have facing the problem ,how do i predict the value for future in multivariate lsrm plzz help?
Just as an info, ALT+return gives you next blank cell in colab.
Haha thank you!
does lstm work in under 100 monthly data point with 4 variables?
wait if these didn't add 60001, 65001 does it cause data leakage?
X_train1, y_train1 = X1[:60000], y1[:60000]
X_val1, y_val1 = X1[60000:65000], y1[60000:65000]
X_test1, y_test1 = X1[65000:], y1[65000:]