181 - Multivariate time series forecasting using LSTM
HTML-код
- Опубликовано: 7 дек 2020
- For a dataset just search online for 'yahoo finance GE' or any other stock of your interest. Then select history and download csv for the dates you are interested.
Code generated in the video can be downloaded from here:
github.com/bnsreenu/python_fo... 
Наука
I should apologize for misspeaking towards the end when it comes to forecasting. I said you can forecast the future 90 days when you do "model.predict(trainX[-n_future])". This is not true; you'd be predicting the past and not the future. This is a multivariate time series which means you need multiple variables as input to predict a single output. The input size shall be (n, 14, 5) where 14 was the number of days we look back and 5 was the number of variables. We only have multiple variables available until the last day in our input. So we can only predict one day into the future. In a single variate time series, you can predict multiple days into the future as you can use your predicted values as input to the model. You cannot do that here as we are not predicting multiple variables. Sorry again for making a rookie mistake. The code on github will be updated accordingly. Thanks to all who alerted me about the mistake.
I have a question to ask... You trained the model on windows of 14 days so (14 * 5) was each window and the window was slid several times predicting one "open" price in the future (w.r.t the window). The model described only knows how to process 14 days on inputs.. i.e. (14*5) and give out 1 output. Given this context can you please explain why you are passing the last 90 days to the model when forecasting to predict one value? I would imagine that training and testing/forecasting instance shapes should be the same while training and inferring. I am unable to grasp the concept at the end. :)
yes exactly, you need predictions of all features to predict the next bar. But I would think in theory this would be possible. So not only predict the Open, but also the other 5 features. I did not do this yet myself but it should be possible, right?
Why isn't it splited into test and train data as in the case of single variate time series? Wouldn't it be the same exercise of predicting a 1 day?
Thanks for the great video! My question is how can I do multi-time-step predictions with multivariate time series using LSTM? E.g, How can I predict 10 days into the future with the LSTM model
Looks like the code wasn't update yet, right?
Figuring out the data input is indeed a crutial part of the whole training pipeline. Thank you very much for such a clear explanation and a great tutorial !
I have been struggling with the input-shape for multivariate time series for such a long time. And finally I found a great explanation for how to prepare my data. Thanks for this.
Many thanks, this is the first time I understand multivariate time series forecasting using lstm.
You cannt imagine how much I need it, appreciated..
The BEST explanation as I saw!!
I am french and I understand this video better than any papers or videos that are actually in my own language. Thanks for this amazing explanation !
J'espère que tes etudes avancent bien ou que tu es déja en fonction
@@ivoriankoua3916 @Wysumay j'ai besoin d'aide j'essaie de faire la m^me chose mais j'ia toujours une sortie exponentielle
Thanks for this great video. It has been extremely helpful.
At the end of the video you asked for ideas for future videos that would help people get through a difficult subject.
I have been struggling with implementing Feature Importance for Multivariate Time Series prediction using LSTM.
I believe this is very valuable for better understanding the predicting model and helping give it credibility. Please consider for a future video.
Thanks again!
Thank you for the detailed explanation on multivariable input
This has been a huge help for a new project i am working on. I still need to optimize a lot of inputs to the model and implement feature importance methods but this basic framework has been an excellent start!
I love the "I'm done with Covid" tone... again great content! Keep it up!
You explained it really well. I was stuck in the datashape things. Now it's crystal clear. Thanks
same for me
You're simply the best. This is torn into pieces
The scaling explanation was worth it, great video.
Thank you for taking the time to explain the input shape! You explained it in a way that helps us develop intuition.
After doodling for past two weeks, finally got some relevant info. Thank you so much Sir
Good day, Sir.! I'm here to express my gratitude. This video has really saved my life. I can't express how grateful I am! I wish you the best of luck in the future.
Thank you so much for taking the time to put this together. This has been very helpful.
You are welcome.
Plainly explained. Enjoyed. Thank you
After searching many places, finally I got it.
Very well explained by you sir.
Please make a video on Object localization of medical x-ray of chest and brain.
Thank you so much for great explanation! I have couple of questions: 1. If I have a monthly data and I would like to forecast next 5 years, then what would be the best size for n_past. Would it be 60 (5 years) or just 12 (1 year). How do we gauge this size. 2. How do I compare validation loss with training loss and what is the good range to consider the validation loss is considerable. Any help in understanding these 2 questions would be appreciated. Thanks!
The video was very clear and easy to follow, certainly in comparison to your peers. I like how you zoomed in to focus on the important part that you were currently talking about.
The concept I am looking for was clearly explained in this video. Thank you so much for sharing your knowledge. even small small things you explain in your videos even though you had already explained in previous videos. thanks a lot.
You're most welcome
Thank you very much for the trick about reshape predicted data to inverse scale. Lot of videos I watched before just plot the scaled data with predicted data without de-scaling, I was wondering how actually use the model instead of comparing performance. this video is very useful to me.
This was excellent! Great work with explaining the input shapes! Your predictions look pretty good also… and we got through Covid… 👍
Thank you for the explanation, really help me for complete my thesis
Fantastic content! The part that you believed you dwelled on was actually the best.
Thank you :)
Thanks a lot for this video. Super helpful to understand how the conversion of timeseries data to a supervised ML problem takes place
Thank you very much.
Few times I comment. But I do when really deserved. This is exactly what I am looking for ;easy and simple on any level .Thank you so much
I have one question: do we use forward walking validation or random validation trsting
This video really helps me a lot! Thank you so much!🎉🎉
Thanks for that amazing video. Help me a lot with my current work.
Remember in this example, the first column is the predictable variable.
Thank you very much for the knowledge you share, greetings from Colombia.
Thank you :)
I just started a project similar to this, and every single second in this video is an answer of of my questions during this project. Thank you and GBU.
Great to hear!
I really appreciate the method of your teaching, I've learned a lot from you.
Glad to hear that!
Amazing content, thank you. It helped me a lot.
Really nice explained! Thanks for this video
Thank you for very clear tutorial, I deeply appreciated your support.
Glad it was helpful!
Thanks for the video.
This is what I looking for
Hi , I hope you are doing well. I am new to Deep Learning. This video was very helpful to understand the LSTM implementation.
Thank you, has been very useful and clearly explained.
Hey, we survived even after Jan 26, 2021. Cheers to that!!! And this video was really thorough and helpful. Thank you so much for this and for all other videos on your channel.
Very nice explanation Sir, I always get confused with shape of the data which you explained very well. I had one question, when we predict future values, lets suppose daily prediction then the value of future day 1 should be one input for future day 2 and so on. Please let me know if I am wrong. If my thinking is correct then how this prediction is working. If possible please explain.
Very well explained! thank you so much for sharing your knowledge!
Glad it was helpful!
great! awesome for your explain about LSTM , thank you sir
you teaching very well, tanks man
Great video. Great explanation. Very clear and very well done!
Thank you!
Thanks so much Sreeni. It helps me a lot !
Hi Sir , i am big fan of you . Learnt lot of stuff from your videos . My sincere thanks for preparing such a awesome practical content .
You are most welcome
I'm glad I found your channel, that's exactly what I was looking for. It's important to spend time on the details, and the video was perfect!
Awesome, thank you!
Thanks man! Excellent explained!
Absolutely loved it. Thanks
Thank you. A great tutorial!
You made it. As usual, awesome !!
Yes! Thank you!
Your explanations are absolutely spot on! Please don't think "dwelling" on a topic is a waste; it really helps with understanding! I would have preferred if you could also explain (or trace) the somewhat head scratching logic in those for loops for trainX and trainY but after starring at it for quite some time I was able to understand. Nonetheless, 11/10!! Thank you for these videos.
I am glad you find these videos useful. Keep watching :)
@@DigitalSreeni I have one question Mr. Sreeni. I was looking for your video #164b becuase in your GitHub I found something I was very interested in, which is the same LSTM forecasting but using the Timeseriesgenerator(). If you can link me the video or any resources that helped you I would be very grateful. Thank you.
Thanks for the great video 👍
Perfect. Thank you for perfect presentation
You're very welcome!
Nice class! Thank you very much
Great explanation - very thorough & clear!
Thank you!
Glad it was helpful!
Amazing explanation sir
Hi Sreeni,
In this lesson you covered the most important concepts in time series forecasting I can think of. I would very much appreciate if you would expand some of the ideas you mentioned, such as including volume in the forecast, decouple the forecast from pd.to_datetime (use time of day or index as well), further coding ideas, etc.
Once again, this is by far the most useful, applicable and compelling explanation I've attended. Thanks a lot.
Noted
thank you so much, you are amazing 🙌
Xin cảm ơn tác giả. Một bài hướng dẫn rất hay.
You are welcome.
Great Video! Thanks for that.
Amazing video !!! Pure gold :)
Thank you very much. You are a very good teacher.
Glad you think so!
Neat & Clear! Thank you!
Great tutorial, thanks a lot.
Thanks for the detailed explanation.
You are welcome!
Sir, I am PhD student - I have seen many videos but I love to see your video. When I was doing image processing part, I learnt the most from your videos. Now I am learning multi-step forecast horizon. Again, I am learning the data reshaping part from you. I have not met you but apart from my supervisors and co-supervisors you are one of my guru. Thank you so much sir.
Thank you so much for sharing such a wonderful video! I have a question...In this case, each observation in the dataset has included the past 14 days' stock data, does the sequential time order between the observations still matter when training the classification model? Can random shuffling be used when splitting training and testing data?
Thank you for your vedio. That is so helpful. I have a question about forcast. If I assume that if we have just one Y value at initial time step and have other variables for forcast period, can I use the predicted values as my new input?
Thank you so much Sir..really you made my day!!
It's my pleasure
sir love your videos
first thank you for the video,
im new to machine learning and im a little bit confused, why is the target label (Open column) included in the input data (trainX) while training the model?
should we not split the data into features (input) and target labels?
Great explanation and great technical skills.
Congratulations !
Could you please be as kind as to share the code that you used in a git repository or a gist ?
Please look at the description under the video for the link to my github repo.
Therefore, your video is complete, professor @@DigitalSreeni !
I've seen so many videos that don't share their code that I forgot to find your sharing information in the video description.
Thanks a lot for that one too, kind sir.
great video, thank you very much sir
great content and explanation, thank you!!
Glad it was helpful!
Awesome channel! Keep up the good work!
Gracias. Please keep watching and learning.
thanks this was very helpful and you are a cool dude!
Glad I could help!
This video is pretty amazing. The approach you have used is when we want to predict using the existing lag feature values (let's say 'close' in your example). What if I want to use some future scraped feature values for prediction? What will be the input shape then?
Just to make draw the attention that your channel deserves, if you add ...LSTM ( Stock Price Prediction ) in title, people will come rushing to this video :P
GREAT VIDEO..THANKS ALOT
Dear sir, thank you very much for your efforts. Anomalies detect by using SOM (Self-organizing maps by using Keras ). I did not see it in your series of lectures. because that is also one of the best Data anomalies methods. pls, do a video on that sir.
Hi DigitalSreeni
I appreciate your work and it helped me a lot. As I'm new, I could not understand the data flow of how the input variables predicted the output. I'm clear about the time series and the cell, but not sure about how the input variables give the output. does the model used the inputs to predict 'open' in the training set and used the weights to make prediction of 'open' from the forecasted time series of inputs?
Please let me know the data flow and operations through the model.
very informative video
Hi,
Finally found your channel many of concept is now clearing with the help of your videos.
Thanks a lot..
Plz, can u make a video on video classification based on CNN lstm.
well explained sir
I am an indian. Thank you very much for such a clear explanation and a great tutorial !
Glad it was helpful!
Great Explanation.. Very useful
Glad you liked it
Thank you very much
Thank you so much for your effort. I want to know if this method and the code are the same if I have 5 variables as inputs and one output.
Best explanation sir please make tutorial GAN based anomaly detection for multivariate dataset
Very nice tutorial...
Cool, this video is easy to understand lstm things. And I should using spyder tool for some project it look better than notebook to monitor the parameters
thank you very much! your code and teach very useful for my project
You are welcome!
my question is that: why I run the prediction for 180 and 90 days, both cases gave diffirences prediction curves and values , those are correct?
You have done wonderful job of explaining tech things in simple manner. Although not needed, but apologising frequently shows your humbleness. But believe me, many ppl got their concepts/doubts cleared when you explained them in between, you are great mentor, take a bow !!
Thank you very much for your kind comments :)
Hi, Thank you for this amazing video. Could you kindly make a video on multivariate time series anomaly detection using LSTM autoencoder ?
Thank you !
Great Always
Thank you so much for this
You're so welcome!
Seriously Great work
Thanks