I have a LSTM Model that I think is good but I want it to run multiple stock tickers at the same time to save time and give me all their individual results. I am having a tough time figuring out how to do this and am constantly getting errors can you help?
the value at the end is constant because theres a big logic error in your foor loop. as you always set it back to x_test[-1] before the prediction. s.t. the LSTM will output a constant value. Non the less if you do it correctly (i just tried it) the prediction will ofc diverge from the stock price anyways (it ll keep going in the direction of first prediction
I believe your final recurrent forecasting is wrong. last_window should be initialized outside the loop, and then for each loop do np.roll(last_window,-1) and update last_window(-1) such that you have a continuously-moving window. In your case you only update the last value, but the rest remain fixed and that is why your prediction remains flat.
@@shengchuangfeng227 you can do even better just by doing last_window = np.concatenate([last_window[-2:], np.array(next_prediction)]). You won’t need anymore to specify last_window[-3:] in your model.predict() function.
We don’t need a np.roll() function. He should just change the last_window expression in the loop by : last_window = np.concatenate([last_window[-2:], np.array(next_prediction)]).
Hi Greg, I believe that you made a mistake at the recursive prediction. If you print your windows, you'll see that every time you predict with the same 3 values, instead of changing it at every iteration. That is why your plot is completely flat, which it shouldn't. Here is my correction (just for last 14 days), I think it's OK: recursive_predictions = [] recursive_dates = dates_test[-14:] last_window = X_test[-14] for target_date in recursive_dates: print(last_window) next_prediction = model.predict(np.array([last_window])).flatten() recursive_predictions.append(next_prediction) new_window = list(last_window[1:]) new_window.append(next_prediction) new_window = np.array(new_window) last_window = new_window If you try to predict with it still doesn't fit to the actual values, but at least is not a straight line hehehe
@@DevilErnest this one almost worked, the last line needs to be last_window = np.concatenate([last_window[1:], [next_prediction]]) otherwise it errored out on me. But yeah this is the way how last_window actually gets updated with the last three predictions of each iteration Still the result is almost a straight line...too bad :]
I'm gonne make a "course" about machine learning but I actually don't know how it works so I'm gonna copy-paste this function and voila, working like a charm. Way to go
Hi, great video! I have written a more efficient function to window the data using the pandas "shift" method. Feel free to use it! def window_data(data, n=3): windowed_data = pd.DataFrame() for i in range(n, 0, -1): windowed_data[f'Target-{i}'] = data['Close'].shift(i) windowed_data['Target'] = data['Close'] return windowed_data.dropna()
Hi, I might be wrong here but I think one of your issues about recursive prediction is - that you always replace the last element, meaning that it tried to predict from the same values with the last one changed, and that might be why it predicts the same. I would change the part to be (append prediction to the end) and pop the first one out,
Which is pretty much the same copy paste mistake all these stock predictors wannabe do Appreciate the effort in confusing thousands of people though...
I know little about machine learning but I think you could fix the extrapolation probelm by making predictions on the stock's log return rathern than price. with a pandas dataframe it should be a line like np.log( 1 + df.pct_change() ). The goal is to get the data down to a stationary time series, so modeling returns should get you closer but the data is not quite symmetric, which is why you also take the log. of course, you outout then becomes a series of logreturns, which you will have to apply this line to fix, np.cumprod( np.exp(df) - 1 ).
00:00 Forecasting Microsoft stock using LSTM neural networks. 03:56 Convert date column to datetime objects and make it the index of the dataframe 07:44 Convert the given data frame into numpy arrays for input and output 11:28 Performing univariate forecasting on closing value over time 15:06 Create and train a sequential model with LSTM and dense layers 18:35 The LSTM model for predicting data shows poor performance on validation and test sets. 22:00 Training LSTM model on a subset of data to improve prediction accuracy. 25:19 The model recursively predicts future values based on the available data.
Hi Greg, thank you for another awesome video. I am working on a multivariate LSTM and i have scaled my data (containing both the input and output variables) as in the code below. scaler = StandardScaler() scaler = scaler.fit(df_train) df_train_scaled = scaler.transform(df_train) Now, doing inverse_transform as below in order to be able to compare my predicted values with the actual, i am getting an error. prediction_copies = np.repeat(prediction, df_train.shape[1], axis=-1) #This will repeat the prediction the number of times columns it was originally scaled with y_pred = scaler.inverse_transform(prediction_copies)[:,0] #Here i am taking all the rows and just the first column, since the other columns are replications of the first. ERROR Found array with dim 3. StandardScaler expected
The problem is with experienced Ai developers they always wanted to predict the future price, but with real traders or investors they all wanted to take a profitable trades, no one wants to exactly know the price and nothing can do that
awesome video 👍 the flat line from recursive prediction is most likely caused by an glitch in the code. "last_window = deepcopy..." should be moved up and outside of the loop. After that's fixed, my guess is that the recursive predictions will converge going forward into an almost flat line
Yeah, I noticed that too. Glad I'm not the only one! The loop is overwriting the last_window at the start so predicting on the same data in every loop should reasonably result in the same value. Also setting last_window[-1] = next_prediction every loop seems wrong. Realistically, you would want to shift the first two values before doing that otherwise, you'd be stuck with the first two values in the list the whole time. It should probably be something like this last_window = last_window[:-1] last_window.append(next_prediction) (I'm not very good at python)
@@idrisseahamadiabdallah7669I have and haven’t been able to get anything over 60% on shorter timeframes. I’m using 15 min candles with about 80k candles of data. It has been harder than I anticipated. I’ve tried 3 variations of LSTM’s,, 3 different random forests, a perception, and I’m not yet at the point where im pleased with the results.
Great video, it's really well explained. I was expecting the recursive predictions to work a little bit better since I will apply this to a school project more oriented to failure detection but I understand that's a hard task. But still is a great base to start with. Thanks so much.
Nice, very detailed. I just have a few questions: 1. I suppose Sequential means Time Series. If that's the case, is there still a need to create 3 columns for the last 3 closing prices? My understanding of Time Series is it automatically looks on the order of values (output of last record is input of current record). 2. Believe LSTM is an optimization technique so that if there are many layers, the vanishing gradient problem will not be encountered. I'm also confused since you mentioned Adam (I'm not familiar with it) as optimizer so it might be in conflict with LSTM? 3. You mentioned LSTM(64) means 64 nodes and Dense(32) means 32 layers .. so its just 2 nodes per layer? Thanks.
@@lorryzou9367 Yes that's the intention. However, the neural network used is time series so the last 3 closing prices is redundant. If a feed forward neural network is used then the input of last 3 closing prices is logical.
I'm trying to adapt your code to hourly and minute time series. I think I need to modify the complicated section that you mention @6:40 can you help me find that tutorial to understand this section? I don't see that in the video description above
I have a concern. It looks like the predicted vs observed values are offset by a day. In other words, when zoomed in the predicted values do seem close to the observed, but they're delayed by a day. From the perspective of actual trading this is very detrimental.
I watched another video on neural training and it said that the model is not good for this purpose as it just predicts the last day - or + on the close value that’s why it’s real close. Their reasoning is that if you take the difference each day it can’t predict and the error rate is huge.They suggested that it’s better to use other techniques for this reason.
Hi Greg, its a nice video to learn. Can u explain what changes we should made to forecast the values after the dates_test, mean after the end date (mentioned in the 2022-03-23) for few days or weeks.
Greg if I understood correctly I can include more variables (multivariate) to make predictions (e.g. Predict the sales for Icecream using the Units per Day + Temperature) and for that I would need to include these 2 variables into the dataset and reshare the feature (X) accordingly, but I got confuded on how to windows the data ?
Hi thanks a lot for this video! How would you move from forecast to a Signal 1/0/-1 column to be applied to the price variation? I have issues with potential future leakage and the condition to use to create the signal. 😊
Hi, Greg! In the MLP network, data from independent variables from date t are used to predict a future value t+n. In the LSTM network, instead of using only data from time t of the independent variables, it uses data from time t, t-1, t-2, ..., t-n as desired by the programmer, and after that, generates the prediction for a future time t+n? Is this reasoning correct? Thank you very much!
hey! love the way you deliver your content sir! learnt a lot from you. just a quick question, can we use a direct type conversion to convert the type of date (pd.to_datetime or astype(datetime64)). if yes why didnt you use it and create a function for the conversion, how is it different from doing it directly as i mentioned above. thanks!! good day!!
The lines are close to each other becuase you are always feeding "future" data, into prediction. The predictions are actualy 1 day shifted to the right, which means the so called prediction is actually delayed. This won't be helpful to stock price prediction at all.
There is a huge mistake in your loop which predicts the test & validation closing prices according to the last 3 values of the training data. You should just change the last_window expression by : last_window = np.concatenate([last_window[-2:], np.array(next_prediction)]) And see what you will get.
My question is , how will one model fit all stock, every stock will have different behaviour based on news and companies end results, so will it fit all??
Great video! I've just one doubt: if I want to put the model in 'production' and predict prices for the next month, I won't have the previous 3 prices for most of those dates (in that production dataset). So, is it right to include those features in training,validation and test set?
I have the exact same doubt. Shame nobody aswered. I this ideia we could only predict the next day, but what you can do is try to make the same he did in the validation, you must try to predict a larger range of days and the performance will drop quickly, but problably enough for a few days ahead. Which I find interesting besides being limited.
Guys, at the end of the video, it should not be a constant value. Just a little mistake. You guys can use my code for reference. In the video, the constant trend happened due to the last_window always keep constant. # Predict future from copy import deepcopy recursive_predictions = [] recursive_dates = np.concatenate([dates_val, dates_test]) last_window_new = deepcopy(X_train[-1]) for target_date in recursive_dates: next_prediction = model.predict(np.array([last_window_new])).flatten() recursive_predictions.append(next_prediction) last_window_new[0] = last_window_new[1] last_window_new[1] = last_window_new[2] last_window_new[-1] = next_prediction print(X_train[-2:]) # Print the last 2 elements of X_train print(np.array([last_window_new])) # Print the value of np.array([last_window])
@GregHogg, If i want 'Close' and 'Open' both (last 3 days data) as input for single row. What code I need to change. I tried to update middle_matrix.reshape and the InputLayer with 2D. The predictions went awful. Can you please guide?
Can someone explain why the prediction is so flat when the train data is big? as the video show , it should be using the 3 previous days' price to predict the next day close price. that's mean it always predict a dramatic drop of few hundred dollars? Is that the cause of LSTM? would it be better to use the price differences to make this model?
hello I am new to this so I was wondering if this exact model would work on say VTSAX stocks and not just microsoft? if I were to just import VTSAX data instead of microsoft data as a csv in the beginning
Greg, Thanks for this. Two questions 1. Will adding in more target values (e.g. go to Target-10) help? 2. What would shuffling the windowed data set do? At least some of the data in the training set would be after 2016 then. Thanks
@@GregHoggsklearn.utils.shuffle i think is what i mean. In your windowed dataframe, the 2nd row might end up as the 100th row, the 3rd as the 250th, the 1000th might end up as the second and so on. Obviously I do not have enough of the vocabulary to truly express myself and typing on an iPad is a bit clunky!
@@GregHogg After shuffling, the only time related data would be within each sample. The shuffling would destroy any time related dependency between samples. This might not be desirable with the LSTM layer? Also, what might be the effect of just using % change on a daily basis, rather than absolute value of the stock. And this is where you get to say “Why don’t you go ttrry it out and let me know what you find”?
I've done it before. It depends on future forecasting and some other parameters. There is no clear relationship between past data and the accuracy of results. Generally, you should test different scenarios based on your project and your data.
Hello Greg, great video again as usual, can advice ,whether can we have a LSTM neural network for predicting stock price of multiple companies say 1000 companies ?
Yes compile each stocks data into a key value pair of a dictionary, the key will hold the stocks name, and the values will hold the data frames that contain the dates and prices/more indicators etc., then turn each into sequence, then train your lstm on each stock in your dictionary
@@jarrodmautz159 thats very impressive explanation , do you have any sample notebook( if you free to share ) on similar use case or any article link ? iam really looking for one sample notebook, thanks for the reply ,it helps :-)
Hi Greg, great video and I learnt a lot here. May I ask whether detrending and deseasonalisation (if they exist) are needed during time series modelling. There are resources out there which suggests these should be done and the series must be stable before feeding time series to the model, while others (like yours) don't really care about trends and seasonality within the model and let the model does its work. My guess is there's no right or wrong answer to it, but what's the reasoning behind these decisions in time series modeling using machine learning algos? Thanks.
Thanks a lot Greg, very interesting. One question, once you have your model, how to add new data, without re-train the whole model, but train just the new data and predict under the whole dataset? does it make sense? thanks a lot!
I think at some point you have to retrain your model. like how Greg threw out all the old data because microsoft went parabolic in the more recent years. the model didn't know how to cope with that. it's like budgeting gas money for a trip, but your're using gas prices from 5 years ago. You're just not gonna get where you want to go.
Hi Gred, I had a doubt, for an unseen data (which contains only the date), how can I send that data into our LSTM model and predict for the next 5 days, when I have no input data apart from the DATE? Please help.
Hi Greg thanks for this great video. I tried this on SP500 futures market and have three questions: 1- what did you do to Handel missing values for weekends and days that the market is closed? 2- what you did not do any normalization on your data? 3- I can see that my prediction and observation charts got pretty similar trend but looks like prediction has a lag meaning prediction is behind the actual which is not good at all. How can I fix that?
Hi Greg, Thank you very much for the video! It's excelent! I have a doubt regarding the windowed_df function. What is the meaning of the if statement about df_subset? I'm talking about this part: while True: df_subset = dataframe.loc[:target_date].tail(n+1)
if len(df_subset) != n+1: print(f'Error: Window of size {n} is too large for date {target_date}') return Because I'm getting this error when I pass to the function a period higher that 2 years (+/-) . For example from 2020-12-01 to 2022-12-4 is retrieving me the correct array (with the 3 timesteps), but if I pass 2020-05-01 to 2022-12-4 I'm getting the error "Window of size 3 is too large for date 2020-05-01 00:00:00" So how can I solve it ? Why this statement is present? What is trying to avoid? Many thanks! Regards Marcos
thankyou for the video, but i have a problem it seems like the prediction that i have is just a straight line, does that mean i need normalization for the data? cause the stock price in my currency is so big it reaches thousand
Hi Greg, my MAE is getting pretty huge, do you have any idea how to fix it? And wondering why you can achieve the low MAE without standardization, thanks!
Another great video! I have a doubt: My date column is in the format day/month/year hour:minutes. What should I change in the str_to_datetime def function to show hours and minutes?
Hi 👋👋 I am looking to create a model that predicts token bonding curves. In order to do that, moreover to predict token price, I need to predict the supply. The majority of youtube videos are talking about the price. I have some trouble finding tutorials explaining how to predict token supply and which features would be better used. Any help, please !!
Where do I find windowing the data frame video the size doesn't support my dataframe and how do I change the dates according to the size of the window?
idk if someone else had trouble converting the date column, but there is a solution: df = pd.read_csv('TSLA34.csv') df['Date'] = pd.to_datetime(df['Date']) print(df['Date'].dtypes)
Would be interesting to see how inclusion of other features like trading volume and maybe some common technical analysis indicators could affect prediction!
I have backtested many of the TI and found they are broken clocks. Not that they are not useful to help filter data, but as for predictions they ideal. The oscillators help with sideways markets, but not trending. Detecting trends before they happen is tough. As for inputs into predictive models I do not think it will help the model. But definitely try. TALIB is best library for that.
great video, but it never pass following part running. import numpy as np def df_to_windowed_df(dataframe, first_date_str, last_date_str, n=3): first_date = str_to_datetime(first_date_str) last_date = str_to_datetime(last_date_str) ...... ...... # Start day second time around: '2021-03-25' windowed_df = df_to_windowed_df(df, '2021-03-25', '2022-03-23', n=3) windowed_df
Hello Sir, Thank you for the tutorial. I am getting the error when we start using the function: df_to_windowed_df() I get error:- Error: Window of size 3 is too large for date 1986-03-12 00:00:00 None Thus, need your help to understand about this error and how to resolve it. Thank you.
I don't get the part on 20:25, how is it possible that it gets the beginning of the prediction so right? I just don't see how this is possible, am I missing something?
Can't you do all the numpy window creation stuff by simply using the pandas .shift() method? smth like this: for i in range(1, n+1): df[f'Target-{i}'] = df['Close'].shift(i)? and then dropna()?
My dataset has only Years and the Total. can someone help me on how df_to_window code which comes after the very 1st matplot graph will look like? i have tried changing the code but it is very complicated.
Very good video!!! I have a question, when we separate the validation data, because of the the size of the window is three, the first row of the validation tdata will carry out the last 3 values that are in the target of of the three last rows of train data, and the second row will carry two values of the the target of last two rows of train data and so forth. Is this considered a data leakage? I mean, I have the target of train set as inputs of the validation set, is this ok?
@@GregHogg Thanks for the answer. I really appreciate your job here on RUclips and on LinkedIn. So I have to be careful when choosing the size of the window, right? Because, if I choose a larger window, the amount of data that will "leak" will grow.
@@ismaelbastos4097 you're very welcome! And yeah, I guess I agree with that. I really wouldn't worry too much though, when you're in a company these things will sort themselves out.
I offer 1 on 1 tutoring for Data Structures & Algos, and Analytics / ML! Book a free consultation here: calendly.com/greghogg/30min
please provide link for that big Window code tutorial. Thanks
go through his notebook which is the first description@@RishiKahtri
Thank you so much for this video. IT taught me LSTM and how to use Python
I have a LSTM Model that I think is good but I want it to run multiple stock tickers at the same time to save time and give me all their individual results. I am having a tough time figuring out how to do this and am constantly getting errors can you help?
HELLO I AM INTERESTED CAN I CONTACT U?
the value at the end is constant because theres a big logic error in your foor loop. as you always set it back to x_test[-1] before the prediction. s.t. the LSTM will output a constant value.
Non the less if you do it correctly (i just tried it) the prediction will ofc diverge from the stock price anyways (it ll keep going in the direction of first prediction
Can you show the example on how you change it? I also noticed that it is incorrect but have no idea how to correct it.
I believe your final recurrent forecasting is wrong. last_window should be initialized outside the loop, and then for each loop do np.roll(last_window,-1) and update last_window(-1) such that you have a continuously-moving window. In your case you only update the last value, but the rest remain fixed and that is why your prediction remains flat.
@TheRafler can you please paste the code, I tried using np.roll as you suggested but the prediction is still flat.
@@shengchuangfeng227 you can do even better just by doing last_window = np.concatenate([last_window[-2:], np.array(next_prediction)]). You won’t need anymore to specify last_window[-3:] in your model.predict() function.
We don’t need a np.roll() function. He should just change the last_window expression in the loop by :
last_window = np.concatenate([last_window[-2:], np.array(next_prediction)]).
Using the price alone won't be a good idea. How about other variables like MA
Hi Greg, I believe that you made a mistake at the recursive prediction. If you print your windows, you'll see that every time you predict with the same 3 values, instead of changing it at every iteration. That is why your plot is completely flat, which it shouldn't. Here is my correction (just for last 14 days), I think it's OK:
recursive_predictions = []
recursive_dates = dates_test[-14:]
last_window = X_test[-14]
for target_date in recursive_dates:
print(last_window)
next_prediction = model.predict(np.array([last_window])).flatten()
recursive_predictions.append(next_prediction)
new_window = list(last_window[1:])
new_window.append(next_prediction)
new_window = np.array(new_window)
last_window = new_window
If you try to predict with it still doesn't fit to the actual values, but at least is not a straight line hehehe
from copy import deepcopy
recursive_predictions = []
recursive_dates = np.concatenate([dates_val, dates_test])
last_window = deepcopy(X_train[-1])
for target_date in recursive_dates:
next_prediction = model.predict(np.array([last_window])).flatten()
recursive_predictions.append(next_prediction)
last_window = np.append(last_window[1:], next_prediction)
@@DevilErnest this one almost worked, the last line needs to be
last_window = np.concatenate([last_window[1:], [next_prediction]])
otherwise it errored out on me.
But yeah this is the way how last_window actually gets updated with the last three predictions of each iteration
Still the result is almost a straight line...too bad :]
I'm gonne make a "course" about machine learning but I actually don't know how it works so I'm gonna copy-paste this function and voila, working like a charm. Way to go
Hi, great video!
I have written a more efficient function to window the data using the pandas "shift" method.
Feel free to use it!
def window_data(data, n=3):
windowed_data = pd.DataFrame()
for i in range(n, 0, -1):
windowed_data[f'Target-{i}'] = data['Close'].shift(i)
windowed_data['Target'] = data['Close']
return windowed_data.dropna()
Thanks so much, well done!
This doesn't have an integer index so the resultant shape of X is wrong
@@GregHogg Pin this!
@liornisimov9367 You should start a channel. Thanks:)
Hi, I might be wrong here but I think one of your issues about recursive prediction is - that you always replace the last element, meaning that it tried to predict from the same values with the last one changed, and that might be why it predicts the same. I would change the part to be (append prediction to the end) and pop the first one out,
Which is pretty much the same copy paste mistake all these stock predictors wannabe do
Appreciate the effort in confusing thousands of people though...
I know little about machine learning but I think you could fix the extrapolation probelm by making predictions on the stock's log return rathern than price. with a pandas dataframe it should be a line like np.log( 1 + df.pct_change() ). The goal is to get the data down to a stationary time series, so modeling returns should get you closer but the data is not quite symmetric, which is why you also take the log. of course, you outout then becomes a series of logreturns, which you will have to apply this line to fix, np.cumprod( np.exp(df) - 1 ).
I thought the same
Great video, Gregg. Best one I’ve on how to implement LSTM on stock price prediction. I’m going to see if I can replicate this LSTM. Thank you 👍🙏
Great to hear! Perfect!!
00:00 Forecasting Microsoft stock using LSTM neural networks.
03:56 Convert date column to datetime objects and make it the index of the dataframe
07:44 Convert the given data frame into numpy arrays for input and output
11:28 Performing univariate forecasting on closing value over time
15:06 Create and train a sequential model with LSTM and dense layers
18:35 The LSTM model for predicting data shows poor performance on validation and test sets.
22:00 Training LSTM model on a subset of data to improve prediction accuracy.
25:19 The model recursively predicts future values based on the available data.
Hi Greg, thank you for another awesome video. I am working on a multivariate LSTM and i have scaled my data (containing both the input and output variables) as in the code below.
scaler = StandardScaler()
scaler = scaler.fit(df_train)
df_train_scaled = scaler.transform(df_train)
Now, doing inverse_transform as below in order to be able to compare my predicted values with the actual, i am getting an error.
prediction_copies = np.repeat(prediction, df_train.shape[1], axis=-1)
#This will repeat the prediction the number of times columns it was originally scaled with
y_pred = scaler.inverse_transform(prediction_copies)[:,0]
#Here i am taking all the rows and just the first column, since the other columns are replications of the first.
ERROR
Found array with dim 3. StandardScaler expected
I am also working on LSTM
@@gauravnarkhede3622 Nice one Gaurav, how is it going with u?
@@folashadeolaitan6222 Good, need some company to work
Hi @@folashadeolaitan6222Did you found the error?
Its an extremely basic one, so it should be resolved by now.
The problem is with experienced Ai developers they always wanted to predict the future price, but with real traders or investors they all wanted to take a profitable trades, no one wants to exactly know the price and nothing can do that
awesome video 👍 the flat line from recursive prediction is most likely caused by an glitch in the code. "last_window = deepcopy..." should be moved up and outside of the loop. After that's fixed, my guess is that the recursive predictions will converge going forward into an almost flat line
Yeah, I noticed that too. Glad I'm not the only one! The loop is overwriting the last_window at the start so predicting on the same data in every loop should reasonably result in the same value.
Also setting last_window[-1] = next_prediction every loop seems wrong. Realistically, you would want to shift the first two values before doing that otherwise, you'd be stuck with the first two values in the list the whole time. It should probably be something like this
last_window = last_window[:-1]
last_window.append(next_prediction)
(I'm not very good at python)
Glad someone else caught this.
@@YoungMoneyInvestments , have you tried this ? Because i want do forecasting, like predict after the test_dates that we already have ?
@@idrisseahamadiabdallah7669I have and haven’t been able to get anything over 60% on shorter timeframes. I’m using 15 min candles with about 80k candles of data. It has been harder than I anticipated. I’ve tried 3 variations of LSTM’s,, 3 different random forests, a perception, and I’m not yet at the point where im pleased with the results.
You're a legend I was looking for ages for a explanation like this one
Glad I could help!
wow amazing. for a tiny mortal without a PHD in maths i actually understood all what you said. amazing work
Glad to hear it :)
One of the best explanations on the subject - I would do some of the pandas stuff slightly different - but overall 10/10
Yeah, I would too looking back. But thanks - great to hear!
this actually explained things that I didn't understand by other people's explaination
Glad to hear it!
Great video, it's really well explained. I was expecting the recursive predictions to work a little bit better since I will apply this to a school project more oriented to failure detection but I understand that's a hard task. But still is a great base to start with. Thanks so much.
Nice, very detailed. I just have a few questions:
1. I suppose Sequential means Time Series. If that's the case, is there still a need to create 3 columns for the last 3 closing prices? My understanding of Time Series is it automatically looks on the order of values (output of last record is input of current record).
2. Believe LSTM is an optimization technique so that if there are many layers, the vanishing gradient problem will not be encountered. I'm also confused since you mentioned Adam (I'm not familiar with it) as optimizer so it might be in conflict with LSTM?
3. You mentioned LSTM(64) means 64 nodes and Dense(32) means 32 layers .. so its just 2 nodes per layer?
Thanks.
the last 3 closing prices are our input features, and the closing price for today is the output/prediction value.
@@lorryzou9367 Yes that's the intention. However, the neural network used is time series so the last 3 closing prices is redundant. If a feed forward neural network is used then the input of last 3 closing prices is logical.
Yay, 2nd comment!! My Saturday night is now completed with this video! Thanks Gregg.
You're very welcome and have a great night!
Was waiting for this one!!
Great to hear!
The final catch was fantastic, thanks for the lesson!
Yeah I feel like adding that in there is very useful to know haha
At first I thought "I do not believe that he will be able to predict the market only with this information", a good lesson, congratulations!
@@senna_william yeah! :)
I'm trying to adapt your code to hourly and minute time series. I think I need to modify the complicated section that you mention @6:40 can you help me find that tutorial to understand this section? I don't see that in the video description above
I have a concern. It looks like the predicted vs observed values are offset by a day. In other words, when zoomed in the predicted values do seem close to the observed, but they're delayed by a day. From the perspective of actual trading this is very detrimental.
Hmm I'll have to check this out.
I watched another video on neural training and it said that the model is not good for this purpose as it just predicts the last day - or + on the close value that’s why it’s real close. Their reasoning is that if you take the difference each day it can’t predict and the error rate is huge.They suggested that it’s better to use other techniques for this reason.
For me the jury is still out I would do lots of testing and your own research
6:40 "If you need to know what that code is, check out the tutorial"...
Can you please add the tutorial to the description? Thank you!!
still hate how you left us on a cliffhanger greg! please make a part two!
The first 6 minutes could be shortened to:
df['Date'] = pd.to_datetime(df['Date'])
df.set_index('Date', inplace=True)
Hi Greg, its a nice video to learn. Can u explain what changes we should made to forecast the values after the dates_test, mean after the end date (mentioned in the 2022-03-23) for few days or weeks.
Greg, you are the man! greetings from Spain :)
Thanks so much! I'd love to visit Spain some day
Hey Greg, thanks for sharing your knowledge. Amazing video and very well explained. Following you onwards
Great content solved one of my major problems. many thanks❣
Super glad to hear that!
Greg if I understood correctly I can include more variables (multivariate) to make predictions (e.g. Predict the sales for Icecream using the Units per Day + Temperature) and for that I would need to include these 2 variables into the dataset and reshare the feature (X) accordingly, but I got confuded on how to windows the data ?
Amazing video. You were so clear to explain! Thanks!
Great to hear!
at 11:41 why we have to convert it to 3-D matrix, can't we train the lstm on 2-d X matrix?
Hi thanks a lot for this video! How would you move from forecast to a Signal 1/0/-1 column to be applied to the price variation? I have issues with potential future leakage and the condition to use to create the signal. 😊
Hi, Greg! In the MLP network, data from independent variables from date t are used to predict a future value t+n. In the LSTM network, instead of using only data from time t of the independent variables, it uses data from time t, t-1, t-2, ..., t-n as desired by the programmer, and after that, generates the prediction for a future time t+n? Is this reasoning correct? Thank you very much!
Man, at 6:40 you refer to a tutorial in the video description that explains the function df_to_windowed_df. What link is that, exactly? Can't find it
hey!
love the way you deliver your content sir! learnt a lot from you. just a quick question, can we use a direct type conversion to convert the type of date (pd.to_datetime or astype(datetime64)). if yes why didnt you use it and create a function for the conversion, how is it different from doing it directly as i mentioned above.
thanks!! good day!!
Congrats on Hitting 22k Subs.
Aw thanks Anmol, and nice dab!!
Thank god I found this video, I tried the same and really poor performance over long term
The lines are close to each other becuase you are always feeding "future" data, into prediction. The predictions are actualy 1 day shifted to the right, which means the so called prediction is actually delayed. This won't be helpful to stock price prediction at all.
There is a huge mistake in your loop which predicts the test & validation closing prices according to the last 3 values of the training data. You should just change the last_window expression by :
last_window = np.concatenate([last_window[-2:], np.array(next_prediction)])
And see what you will get.
Awesome video by the way ! 👌🏽
My question is , how will one model fit all stock, every stock will have different behaviour based on news and companies end results, so will it fit all??
What is the value for test_val?
Great video! I've just one doubt: if I want to put the model in 'production' and predict prices for the next month, I won't have the previous 3 prices for most of those dates (in that production dataset). So, is it right to include those features in training,validation and test set?
I have the exact same doubt. Shame nobody aswered. I this ideia we could only predict the next day, but what you can do is try to make the same he did in the validation, you must try to predict a larger range of days and the performance will drop quickly, but problably enough for a few days ahead. Which I find interesting besides being limited.
Just Awesome! very helpful😀😀😀
really nice tutorial and thank you so much for making it brother
Guys, at the end of the video, it should not be a constant value. Just a little mistake. You guys can use my code for reference.
In the video, the constant trend happened due to the last_window always keep constant.
# Predict future
from copy import deepcopy
recursive_predictions = []
recursive_dates = np.concatenate([dates_val, dates_test])
last_window_new = deepcopy(X_train[-1])
for target_date in recursive_dates:
next_prediction = model.predict(np.array([last_window_new])).flatten()
recursive_predictions.append(next_prediction)
last_window_new[0] = last_window_new[1]
last_window_new[1] = last_window_new[2]
last_window_new[-1] = next_prediction
print(X_train[-2:]) # Print the last 2 elements of X_train
print(np.array([last_window_new])) # Print the value of np.array([last_window])
hi, why did you not include the other columns open, volume, low, high in the training dataset?
because he actually doesn't know what he's doing
@GregHogg, If i want 'Close' and 'Open' both (last 3 days data) as input for single row. What code I need to change.
I tried to update middle_matrix.reshape and the InputLayer with 2D. The predictions went awful.
Can you please guide?
Can someone explain why the prediction is so flat when the train data is big?
as the video show , it should be using the 3 previous days' price to predict the next day close price.
that's mean it always predict a dramatic drop of few hundred dollars?
Is that the cause of LSTM? would it be better to use the price differences to make this model?
Thank ， so much ! Very good project .
You're very welcome
Error: Window of size 3 is too large for date 1986-03-18 00:00:00
How to Resolve?
I guess the correct recursive prediction should be :
from copy import deepcopy
recursive_predictions = []
recursive_dates = np.concatenate([dates_val, dates_test])
last_window = deepcopy(X_train[-1])
for target_date in recursive_dates:
next_prediction = model.predict(np.array([last_window]), verbose=False).flatten()
print(target_date, last_window[:, 0], next_prediction)
recursive_predictions.append(next_prediction)
last_window[0] = last_window[1]
last_window[1] = last_window[2]
last_window[2] = next_prediction
However, it not working as well.
it does not work because you get an array of single arrays
and the code is good, thank you
I have a doubt asked by my professor "Why are we using only one attribute "close price" when the stock market can depend on volume too????
You absolutely could
Im getting a syntax error on the def df_to_windowed_df(dataframe, first_date_str, last_date_str, n=3): where its telling me its incomplete.
hello I am new to this so I was wondering if this exact model would work on say VTSAX stocks and not just microsoft? if I were to just import VTSAX data instead of microsoft data as a csv in the beginning
Greg, Thanks for this. Two questions
1. Will adding in more target values (e.g. go to Target-10) help?
2. What would shuffling the windowed data set do? At least some of the data in the training set would be after 2016 then.
Thanks
Changing the window may help. Not sure what you mean by shuffling.
@@GregHoggsklearn.utils.shuffle i think is what i mean. In your windowed dataframe, the 2nd row might end up as the 100th row, the 3rd as the 250th, the 1000th might end up as the second and so on. Obviously I do not have enough of the vocabulary to truly express myself and typing on an iPad is a bit clunky!
@@GregHogg After shuffling, the only time related data would be within each sample. The shuffling would destroy any time related dependency between samples. This might not be desirable with the LSTM layer? Also, what might be the effect of just using % change on a daily basis, rather than absolute value of the stock. And this is where you get to say “Why don’t you go ttrry it out and let me know what you find”?
I've done it before. It depends on future forecasting and some other parameters. There is no clear relationship between past data and the accuracy of results. Generally, you should test different scenarios based on your project and your data.
Hello Greg, great video again as usual, can advice ,whether can we have a LSTM neural network for predicting stock price of multiple companies say 1000 companies ?
Yes compile each stocks data into a key value pair of a dictionary, the key will hold the stocks name, and the values will hold the data frames that contain the dates and prices/more indicators etc., then turn each into sequence, then train your lstm on each stock in your dictionary
@@jarrodmautz159 thats very impressive explanation , do you have any sample notebook( if you free to share ) on similar use case or any article link ? iam really looking for one sample notebook, thanks for the reply ,it helps :-)
Hi Greg, great video and I learnt a lot here. May I ask whether detrending and deseasonalisation (if they exist) are needed during time series modelling. There are resources out there which suggests these should be done and the series must be stable before feeding time series to the model, while others (like yours) don't really care about trends and seasonality within the model and let the model does its work. My guess is there's no right or wrong answer to it, but what's the reasoning behind these decisions in time series modeling using machine learning algos? Thanks.
There's always multiple ways of doing things. In theory you try everything and do what works best
majjjggic😂
Thanks a lot Greg, very interesting. One question, once you have your model, how to add new data, without re-train the whole model, but train just the new data and predict under the whole dataset? does it make sense? thanks a lot!
I think at some point you have to retrain your model. like how Greg threw out all the old data because microsoft went parabolic in the more recent years. the model didn't know how to cope with that. it's like budgeting gas money for a trip, but your're using gas prices from 5 years ago. You're just not gonna get where you want to go.
Hi Gred, I had a doubt, for an unseen data (which contains only the date), how can I send that data into our LSTM model and predict for the next 5 days, when I have no input data apart from the DATE? Please help.
Your input for LSTM is not a date, but prices for previous N days before now.
i have the same question what can i do?
if i want to know values for next days
Hi Greg thanks for this great video.
I tried this on SP500 futures market and have three questions:
1- what did you do to Handel missing values for weekends and days that the market is closed?
2- what you did not do any normalization on your data?
3- I can see that my prediction and observation charts got pretty similar trend but looks like prediction has a lag meaning prediction is behind the actual which is not good at all. How can I fix that?
Please don't attempt to actually predict the stock market with this :)
Hi Greg,
Thank you very much for the video! It's excelent!
I have a doubt regarding the windowed_df function.
What is the meaning of the if statement about df_subset? I'm talking about this part:
while True:
df_subset = dataframe.loc[:target_date].tail(n+1)
if len(df_subset) != n+1:
print(f'Error: Window of size {n} is too large for date {target_date}')
return
Because I'm getting this error when I pass to the function a period higher that 2 years (+/-) . For example from 2020-12-01 to 2022-12-4 is retrieving me the correct array (with the 3 timesteps), but if I pass 2020-05-01 to 2022-12-4 I'm getting the error "Window of size 3 is too large for date 2020-05-01 00:00:00"
So how can I solve it ? Why this statement is present? What is trying to avoid?
Many thanks!
Regards
Marcos
Same here :(
Awesome video , but how do we know our window size in this video case 3 is there a way to know that. I think it's most crucial while working with LSTM
You choose
@@GregHogg Based on what? or by seeing the data visualization to see some pattern that repeats
thankyou for the video, but i have a problem it seems like the prediction that i have is just a straight line, does that mean i need normalization for the data? cause the stock price in my currency is so big it reaches thousand
Thanks for the brilliant work!
Thanks so much for the kind words Ivan!! :)
Will there be any advantage if we scale the closing stock price between 0-1
You could definitely try that :)
Hi Greg, my MAE is getting pretty huge, do you have any idea how to fix it? And wondering why you can achieve the low MAE without standardization, thanks!
You just made my project for bitcoin prediction in college, thanks man
Sounds great!
Great stuff! Why did you set date as an df.index in min ~5, was it necessary/what's a rationale behind it?
See the other comments haha
@Greg Hogg Please provide vedio link for window function
Another great video!
I have a doubt:
My date column is in the format day/month/year hour:minutes.
What should I change in the str_to_datetime def function to show hours and minutes?
Hi 👋👋
I am looking to create a model that predicts token bonding curves. In order to do that, moreover to predict token price, I need to predict the supply.
The majority of youtube videos are talking about the price. I have some trouble finding tutorials explaining how to predict token supply and which features would be better used.
Any help, please !!
Can you implement this in a streamlit interface
thank you , where is the windowed_df code explanation ? i could not find it in the description ?
Super tutorial 💪 great code💪💪🤝
Thanks so much and great to hear!!
Amazing work Greg!!👏 Do you know any good community to share and talk about projects like this one? discord or something...🧐
Thank you!! LinkedIn and Facebook have some very large groups :)
Where do I find windowing the data frame video the size doesn't support my dataframe and how do I change the dates according to the size of the window?
what is the meaning of prediction for every single day, nothing, your prediction give back the value with big standard error
idk if someone else had trouble converting the date column, but there is a solution:
df = pd.read_csv('TSLA34.csv')
df['Date'] = pd.to_datetime(df['Date'])
print(df['Date'].dtypes)
I don't understand why he didn't use the built in function of panda to_datetime to convert
Would be interesting to see how inclusion of other features like trading volume and maybe some common technical analysis indicators could affect prediction!
I'll do this and keep you posted
I have backtested many of the TI and found they are broken clocks. Not that they are not useful to help filter data, but as for predictions they ideal. The oscillators help with sideways markets, but not trending. Detecting trends before they happen is tough. As for inputs into predictive models I do not think it will help the model. But definitely try. TALIB is best library for that.
Why is that windowed_df function taking so long to run, it isnt stopping...
great video, but it never pass following part running.
import numpy as np
def df_to_windowed_df(dataframe, first_date_str, last_date_str, n=3):
first_date = str_to_datetime(first_date_str)
last_date = str_to_datetime(last_date_str)
......
......
# Start day second time around: '2021-03-25'
windowed_df = df_to_windowed_df(df,
'2021-03-25',
'2022-03-23',
n=3)
windowed_df
Have u got the solution?
Hi
the reason that it couldn't predict correctly was because there was never a rise like the test set in train set right ?
Hello Sir,
Thank you for the tutorial.
I am getting the error when we start using the function: df_to_windowed_df()
I get error:-
Error: Window of size 3 is too large for date 1986-03-12 00:00:00
None
Thus, need your help to understand about this error and how to resolve it.
Thank you.
If you want more previous days, then take at least two years worth of data or more, that should work
Which mathematical model we you have used for prediction?
LSTM
that is one excellent video to setup a basic LSTM model! great work and thank you
Awesome! Thank you, and you're very welcome!
sometimes in the big datas my epochs value be nan i mean loss : nan, i haved normalize the data but it didn't change anyting, what should i do ?
I don't get the part on 20:25, how is it possible that it gets the beginning of the prediction so right? I just don't see how this is possible, am I missing something?
how to determine the parameter values in LSTM model? such as LSTM(64) and Dense(4）
Generally through trial and error, but you could automate a search
@@GregHogg 😳thank you for reply
amazing video, why does your model on this video perform differently from the actual code.
Thank you! Probably just randomisation
Can't you do all the numpy window creation stuff by simply using the pandas .shift() method? smth like this: for i in range(1, n+1):
df[f'Target-{i}'] = df['Close'].shift(i)? and then dropna()?
how can we check the rmse score?
My dataset has only Years and the Total. can someone help me on how df_to_window code which comes after the very 1st matplot graph will look like?
i have tried changing the code but it is very complicated.
Very good video!!! I have a question, when we separate the validation data, because of the the size of the window is three, the first row of the validation tdata will carry out the last 3 values that are in the target of of the three last rows of train data, and the second row will carry two values of the the target of last two rows of train data and so forth. Is this considered a data leakage? I mean, I have the target of train set as inputs of the validation set, is this ok?
I see what you're saying, and thanks for the kind words. I wouldn't really consider this is a leakage. Formally? Maybe. Practically? Not really
@@GregHogg Thanks for the answer. I really appreciate your job here on RUclips and on LinkedIn. So I have to be careful when choosing the size of the window, right? Because, if I choose a larger window, the amount of data that will "leak" will grow.
@@ismaelbastos4097 you're very welcome! And yeah, I guess I agree with that. I really wouldn't worry too much though, when you're in a company these things will sort themselves out.
i copy your code in description, why my data is error? "window of size 3 is too large" what thats mean? thnks sir i hope you answer soon 🙏
I got the same error, but I fix it by downgliding the entire max dataset
if your data is not linear, you can use the log to make it linear
BRAND NEW TO CODING. Is there a way to make the chart interactive to see exact predicted prices?
Yes use plotly instead of matplot
What type of LSTM it is ? Vanilla , stacked or CNN ?
Good ol' regular lstm