ive got a 6 gb csv file with the last 5000 days of price info for around 4800 companies merged with quarterly financial data like assets revenue etc merged with macro indicators like interest rates and all of that interpolated and cleaned up before passing to an lstm with many many layers with tuned hyper params. instead of using a standard loss value you should use mean absolute percentage error because if a stock is trading at 1$ and you have a loss of 1$ you would be off by 100% but if its trading at 1000$ and yoru loss is 10 then you would be off by 1%. also try to use time distributed lstm layers at the input and to use leaky relu not just relu so that parts of the model that arnt use often dont just become dead weight . Also use a deeper model to capture the complex relationship between the columns in the training data and MOST IMPORTANTLY..... Kill any over fitting with lots of dropout layers and batch norm
Thanks a lot for your input I will pin your comment for viewers building a LSTM system, will consider these ideas as well in the future. Thank you again.
hey man, I'm doing a university project comparing deep learning methods of stock prediction vs traditional methods such as regression. Can I ask how accurate your model is and where you sourced the data from?
The biggest pitfall of anyone starting with time series predictions (of any kind)! Predicting actual values instead of predicting the difference between values. Great video explaining why!
@@CodeTradingCafevery interesting video. I was wondering whether there is another one proposing ways to come up with a model that may exhibit some predictive value
Using Bayesian Optimization I tried optimize hiperparameter such as layers, neurons, dropout, learning rate, backcandles another else but until now I can't predict the next close price, I use 1 minute candle but I can't get a good prediction, indeed I add unsupervised model before supervised model (lstm) and I can't improve the final model... any suggest about what I could do?
@@Mammel248I read a paper about time series forecasting recently and they combined technical analysis with a neutral network and sentiment analysis. I think it's a vital part because you can never predict the price with only technical analysis because those indicators are not what is driving the market (yes to an extent) it's a lot of sentiment as well. Name of the paper: "EvoTrader: Automated Bitcoin Trading Using Neuroevolutionary Algorithms on Technical Analysis and Social Sentiment Data" They beat buy and hold strategy of Bitcoin by over 200% so really good I would say.
Hi, just want to say I've watched many videos about this topic and that this is the best one because you let me see the mistake in other's results that look good superficially
This is a valuable lesson about uncertainty and how much 'magic' one can squeeze from AI. I think you are asking too much from the model. Using the highest w/l strategies, not even the best traders can predict what the price will do at the *next* candle, for *every* candle. They are content with making predictions about the near, unspecified future, at times when many concepts converge on a bias. In essence, *nobody* can predict exact price movements at all times. Apart from making this demand, the worst 'crime' you ended up committing was throwing away volume information. Your model is tasked with price-action prediction, and you took away *its most critical* input. I don't know how one would train a model with the fuzziness of "tell me when prices will rise/fall, by approximately how much", which is what traders do. Probably best, however, would be to skip this approach entirely and task the machine to predict order signals, SL, and TP prices, then measure strategy performance, instead. And please, don't throw away volume.
I agree, it's too much to ask from an AI model or even a trader. I just don't believe in Volume when it comes to Forex (not centralized info), but yes volume is crucial when trading stocks and even Gold. I will get back to some strategies using volume and maybe backtest, had this on my mind for a while now. Thank you for your input.
Great video, excellent observation about the good predictions phenomena, almost 90% of articles and papers i came across online show high accuracy, that is not applicable for the future only for test data, which is useless!
Great video! You nicely explained why we encounter a lagged graph. I think what basically happens is that the model begins to "cheat" because the data is more or less useless. So the model just picks the current value and uses it as the future value (since it can't really learn anything from the data). I would argue that when you have useful data, you can theoretically predict the future price. How would one check if this works? Just train your model and calculate the lagged correlation with a range from 0 to the number of days you want to predict into the future. If your argmax is 0 or close enough to 0, you're in good shape and can continue with the additional tests. Removing the current price from the data the model can train on and predicting the rate of change over a time span will prevent the model from simply creating a lagged graph, but this also assumes that the current price is not useful for making predictions. I'm not sure if this is always a valid assumption.
Hi, thank you for your detailed message. In my opinion (but I might be wrong) providing the current price is not an indicator for future price, somehow I am copying the thinking of a trader, if I as a trader look at the current price value I wouldn't be able to guess the future, but if I look at a series of recent prices aka the slope of the price.
@@CodeTradingCafe In theory, an LSTM should have no problem learning this implicitly, so feature engineering likely won't have a significant impact on the model's predictive power. If the current price is unnecessary, the model would simply disregard that feature. However, when it relies solely on the current price for its predictions, it suggests that the model is either unable to learn from the data or is stuck in a local minimum. In general, I don't think the issue is with feature engineering (although min-max scaling or standardization can certainly be useful), but rather with finding the right data that the model can actually learn from. To prevent the model from simply predicting the next price as the future price, one could add a penalty for this behavior in the cost function (though, again, having good data is key). I believe the main challenge in predicting future prices is essentially a search problem-identifying and leveraging the most relevant economic and stock data.
I totally agree it's a search problem, sometimes you might get lucky and the search time is shortened (getting the right info, from the right colleague, or webpage or book ...), but sometimes not, in the latter I guess it's consistent searching.. trial and error.
Nice ! But I sincerely think that data scaling is also a source of problems as most of the features are continuous (ATL, +inf). Therefore, usual scaling methods assume that the previous ATH will never be reached in the future in which case the model, no matter how sophisticated it is, will never predict a breakthrough (new ATHs).
@@kadourkadouri3505 To calculate the IQR, follow these steps: Arrange the data in ascending order. Find the median, or the middle value, of the dataset. If there is an even number of data points, the median is the average of the two middle values. Divide the dataset into two halves: the lower half (below the median) and the upper half (above the median). Find the first quartile (Q1), which is the median of the lower half of the data. Find the third quartile (Q3), which is the median of the upper half of the data. Calculate the IQR by subtracting Q1 from Q3: IQR = Q3 - Q1.
1- If you calculate power of predicted results few times looks like you will get barely enough results for price change. 2- Instead of predicting how many pips price gonna change, predict how much percent price gonna change rational to the last candle. In other words don't confuse our little brained neural net with exact numbers or pips, instead teach it to predict relatively change of price as daily percent.
Hey, thank you. Your proposition makes perfect sense, I did something similar in another video it's still challenging but needs more time to make it work.
You are certainly right. I have traded cryptocurrency in practice for 3 months. I do the swing trading and clearly realize that it is hardly to calculate or predict the oscillate range of any coin. But I think trend prediction would be a better direction. Transferring the problem from regression to classification may be more efficient🧐
The new output is stil not conclusive. Data analysis is an art. It doesn't at all what values the AI is predicting, the yestion is more if it is giving the right direction most of the time. Is the price going up when price is predicted to go up, even if it has averaged out to be a very small number.
The problem in shifted values is in fact that regression model predicts value of Close very similar to close of current bar, not future ,even if you have targeted NextClose column with future close values. That’s why you have shifted values. So, they are very similar to previous closes … So I can’t understand why it works like this. Why people invented regression ML models if they work like this
Your description makes sense, but here we are only trying to predict one bar ahead, ideally we would try 5 or even 10 bars in the future, but this only works in different fields (climate forecasting) because there is a pattern followed by nature and the we do understand the pattern well enough to make some forecasting, even there it's also challenging. The trading market is so noisy these models are not able to see a specific pattern.
@@CodeTradingCafe Almost agree. However, you just need to figure it out. I tried various regression options by collecting last candle data from my broker. Knowing that the model can only predict the next value based on the last row (OHLC..), I also created the Features/Target column with a shift up one row. However, every time the values of the predictions of this column come out close to the current Close and not the next one. While the next prediction is close in value (~0.997 r2) to what should have been in the previous line. That is, the predictions of the random number have a close (~0.997 r2) value to what should follow next, but they are somehow shifted. And on your chart, on the video, you also demonstrated this. This breaks my brain. I can't understand why
"the one on the right is predicting the closing price of the next candle and the one on the left is trying to predict the price movement between the two candles" you have said. so meaning at the close of the candle lets say we are on 4h chart . if we use the model on the right to predict the market at the close of the 4 hour chart .literally basing on your explanation we are predicting the close of the next candle. so if we have a almost right predictions for the next close every time a 4h candle closes or daily candle closes using the model in the right cant we use that info to open trades
Definitely not! visually it looks appealing but it has no prediction power. To make it simple most of the times the algorithm is predicting the same close price as the previous candle and we would have this perfect look of predictions only with a slight translation. Again it looks so nice but in reality it's just copying previous closing prices. The problem is that blogs online show this to attract audience but it doesn't provide value.
Would love to see this built getting beginning to end and then modified and back tested. I was thinking, couldn't you see how high the green line was above the black line on average and add a line of code that was predicted close minus 20 pips equals new target?
i guess different parts in the video, but its related to the two line graph lines you display and how they are not exactly in sync with each other. If you knew how far the one was off from the other, you could just add a calculation to add X-amount of pips to the calculation to adjust the line relative to the other line. @@CodeTradingCafe
Ah ok now I see what you mean, actually no it doesn't work because the translation is on the x-axis and what we see on the y-axis is just the result of it, if we force it let's say 20 pips down it would only look nice for the eye but tricky because the difference is not always 20 pips it's just not seen from the scale of the presented figure.
You mean using the actual values instead of the difference it's a mistake because for price prediction we are looking for accurate results. But if we simply want a forecast to understand the direction of the time series, wouldn't we be happy with the results ? In some use cases we might not need an exact number. Do you think in this cases, the model with actual value prediction is still okay ?
If the model can do such exact predictions then yes it's ok, the issue is it's impossible. and only differentiation would inform about the price current and probable future trend.
Could it be that you spent too long thinking about how the neural network works and too little time learning what is useful information for a trading strategy ? The graph your model predicted is nowhere near realated to a moving average, but I agree that it follows the same trajectory as a moving average configured to the smallest time interval. Now why do I say that they are nowhere near related ? Because you forgot to account for WHEN you are receiving the data from the moving average compared to your AI prediction. The strategy itself should not be based on AI. Because the AI makes immesurable decisions. Its the SIGNUM of your positions that the AI should handle. This here is perfect for Trend forecasting. If you implement a solid strategy (iteratively, no AI needed) and just let the AI feed the Strategy the Trend Data. You get an absolute monster of a trading bot.
Hi, thank you, you can add a new category into the target column, but be aware that more categories might make it more difficult for the model to forecast all the possibilities... but that's the beauty of the challenge :)
Does that mean that predicting price fluctuations on a line that is not too attractive like that is not necessarily wrong? Because I was also studying and working on the same problem, and the results were similar!
why to limit LSTM. It can use any size of memory. I think no need to set any number of candles to limit... If you use that limit then just use usual ANN instead of LSTM
Definitely need to limit the backcandles even for LSTM otherwise we would be fitting on the whole set of data meaning also data from very long in the past which will not influence current behavior in reality, so past data becomes noise and hinders the model.
@@connectrRomania exactly I mean we could easily scrap few articles per day and check the sentiment but when the effect will take place is another story... I would say weekly timeframe in this case is a good candidate but drowning in open trades fees is a problem.
As a first answer no lstm can deal with non stationary data, however I am now wondering if they would perform better on stationary data types... It's a never ending story with these stuff 🙂
Not exactly, the idea I wanted to highlight is that if I gave you absolute open close values there is now way we can guess current trend or future trend. Only by considering relative values (differentials) that you get information on price movement. Edit: However, using stationary data is also a relevant point and shouldn't be neglected, in this video I didn't transform the data into stationary state.
In the MLP network, data from independent variables from date t are used to predict a future value t+n. In the LSTM network, instead of using only data from time t of the independent variables, it uses data from time t, t-1, t-2, ..., t-n as desired by the programmer, and after that, generates the prediction for a future time t+n? Is this reasoning correct? Thank you very much!
Yes exactly, LSTM fits over a window of time and window width is defined by the programmer. The length of the window is however hard to figure, too long vs too short... among other issues.
Finally someone said it. The LSTM prediction is as useless as doing df['pred'] = df.close.shift(1). It thinks tomorrow the close price would be the same as today.
Thank you for your comment, I found better results with normal ML classifiers (still not enough for a descent trading model), and even better results with simple NON-ML models using classic price action detection! ironically :)
@@CodeTradingCafe yes same here. I tried a bunch of stuff including using the openai api (gpt-3.5 model) but the backtest shows I actually would have lost about 40% of the initial sum...keep trying...there's bound to be a good ML strategy...I was actually thinking to get historical financial news data, get a sentiment analysis and clump that in the LSTM somehow...I've also looked up the gym-trading lib and reinforcement learning. Here's something I've just watched about it. ruclips.net/video/D9sU1hLT0QY/видео.html. If you get some good results it would be amazing. I'll let you know if I get any good results too
@@CodeTradingCafe no problem. another thing I was trying was using fbprophet (now just prophet) to get weekly and daily seasonality for the price movement...found some interesting results but not so much for signal classification but rather for deciding how much to invest as a % if a signal does appear.
I am sorry but I don't understand the whole point. Normally we use differencing ( what the video author suggests to use as a target) to remove trends. But aren't trends something we want to capture in our model? And why would differencing make a difference? Sry for my noobeness
Hi, no worries about questions. Actually if I gave you the closing price of an asset would you be able to predict the future price direction ? I guess not simply because a value of a price has nothing related to the trend, but if I give you the difference between today's and yesterday's prices then you would guess the current trend and its strength and you might be able to extrapolate for future price trends as well.
@@CodeTradingCafe Models like SARIMA (python, statsmodels) allow to indicate the order of differencing. Wouldn't this solve the issue ? Since the prediction will be made on the difference between today's and yesterday's prices (differenced series). The output though is automatically undifferenced for interpretation reasons. (I am not debating whether SARIMA is a good model for stock price prediction)
you are right SARIMA can make predictions based only on differentiation, ML models can take more than one input feature and from different types, this is why it hyped hopes at some point, and many still believe it can be a powerful solution, although I disagree on this for trading.
I could give you a beautiful shiny (but fake) answer, or I can tell you the honest truth :) predicting values 10 days ahead relying only on technical analysis is IMPOSSIBLE. But predicting a simple trend (which is more than enough for trading) is maybe possible only for the next day or 2!
hm based upon avaraged indicators predict..>> Something slow moving, stears something wildly moving.. this might not be ideal engineering for LSTM use. LSTM's need patterns, not noice. I'll analyse it deeper tomorow, as it helps me thinking about my tensorflow projects too.
i think i have failed to understand you, but my question is if i run my model on the right at the closing of the trading day to predict the nextclose for the next trading day .cant i use that generated prediction to open trades since it predicts the market close for the next closing day. i mean if its on thursday 12:00 midnight or friday 12:15 am and i run my model on the right and i see that the candle on friday will close below the thursday candle ,can't i use those predictions to trade the market
Hi again, (check my answer to your other comment). In brief no because the predictions are really bad they only look good when we "zoom out" showing the full time slice of predictions, if we zoom in and check the details of the results the error is huge and not good for trading.
@@CodeTradingCafe thanks for your feedback actually i had created a trading bot that was using the model that's similar to the one in the right and it was making many losses , would adding more indicators make our model more accurate
Hi, no I don't think adding more indicators would solve the problem, it's more related to the model itself. Steer away from ML for trading, classic programming works better.
@@CodeTradingCafe when i use that code for currency pairs, in the target column i get zero for almost all rows as the change in price movement , what's causing that?
There is no best book, straight answer is you have to master basics in python or any language, and then research strategies and try them out couple of years later you'll find that you can guess if a strategy is worth it before trying it. However I do have my plan that I follow when putting a new strategy together.
I have a question: to calculate EMAF or RSI you need some historical Data. How do you calculate the first row where there is no historical data? i don´t understand that
It's not calculated for the first few rows, if we choose RSI length 14 for example we start our first values from row 14, when all indicators are calculated we discard the first few rows from our dataframe to keep only clean rows. I hope this answers your question.
@@CodeTradingCafe thanks for your fast respond and explanation! I understand that, but where exactly do the values in the first rows come from? Are they calculated oder just random values? Thanks again!
Sorry, I'm quite dumb. I'm trying to understand why for a simple classification model using the next closing price wouldn't work If I'm trying to catch a trend (and really don't care about how much will go up or down), looking at the predicted close isn't enough? So for example if I do: prev_actual_close < actual_close && prev_actual_close < predicted_close and gather all results for the testing data, with (just for the example) 90% success rate, doesn't that mean I *could* use it as a "going go go up, not sure how much" type of flag? I know there are other (maybe) better classification models, but I'm just getting started and have much to learn still, your videos are helping a lot so thank you!
Hi not dumb at all, I did a video previously using machine learning just for classification trend up and trend down ruclips.net/video/q_0rvwPbIqg/видео.html, it did work to an extent but not very robust in the sense that any change in the parameters or different asset the results go back to 50% win rate which is a random indicator baseline. I hope this helps.
Yes I did automated bots previously some are on this channel, although I don't recommend it, I prefer to receive phone alerts and I log in and execute trades myself instead of the bot, so the algorithm is only my assistant.
@@CodeTradingCafe okay i understand, if i should ask, why don't you recommend it? I was thinking of learning pinescript for the tradingview platform, have been considering it but haven't reach a conclusion yet, are you familiar with it? It is not a full fledged programming language but it get most of the job done, bad thing it doesn't allow external libraries or analyzing more than 30 symbols in one script.
@@Kay-qg1vn I don't recommend it because algorithms can be very powerful at some point and mess up at others (like existing a trade at the correct moment for example). I am not familiar with other tools only python, c++/metatrader.
Excellent explanation. Does your model predict different numbers after each running? Isn't it worth to make range of predictions like in Monte Carlo simulation?
Hi thank you, I don't think any neural net approach is worth it for trading predictions, at this point neural nets perform very poorly on this type of problems.
Hi, yes and no, yes the noise is reduced but the moving average adds a lag to the related information, it might work better on high timeframes... but I don't think it's going to be any better (only a feeling opinion).
@@CodeTradingCafe thank you for your reply, that makes sense. With LSTM May I know what is the best accuracy you can get for the next day stock prediction? (Just in terms of trend, either up or down. I.e if random there should be 50% chance to go up or down)
This has better chances in my opinion, it might be a good video idea as well, ... Hourly timeframe and predict if the average future price is higher than the current price.
thanks for the clarification - maybe worth removing the OHLC and add a lower EMA and then feed to the model? then the result may not be the same previous price?
@@CodeTradingCafe thanks for your reply. I think the difference between close [i] and close [i-1] or return would be a better option compared to closing price? I guess if we have to then result can be converted to closing price if required. Another issue if there is a difference between the adj close and other OHLC price - i would suggest to convert the OHLC to adjusted values and run the model. it may not be the case for Forex but it is definitely the for stocks.
thanks for this excellent video - so basically the LSTM RNNs just dont work for predicting future price action. Is that correct? I did try and rewatch why training it to predict pip differentials vs future spot price results in wildly different predictive value but dont understand your reasoning, Can you rephrase it for me please?
Hi, the way it's done in this video no it doesn't work. Basically the model is estimating the future by copying the current price which is its best guess.
@@CodeTradingCafe As a proficient trader with significant knowledge of TA and fundamentals, may I make a suggestion that could increase the predictive accuracy substantially? Essentially there is a significant temporal issue here and the very premise of trying to predict next days close price based on previous 30 day data is flawed for a number of reasons (fundamentally, technically etc). Happy to also take this 'offline' and communicate via email if you prefer.
@@CodeTradingCafe thank you for reply, still confused, so if i test It with real time data, to predict next 10 candles for example i still get mistakes i guess.
@@CodeTradingCafe I think the only real sucess i have heard of in this area is using deep languge models on news. The fact u can use a pretrained one REALLY helps with the small data problem
@@CodeTradingCafe honestly anns have not showen good results on time seiries data compared to arima. I have tried it myself and i gota say the results r awful
I Just got started with algo trading and I was wondering if using a LSTM model is really worth exploring due to this issue that has been brought up by you in this video. If not, do you have any suggestions on what model works best for forex trading in small timeframes such as 1min or 5 min charts? Great video by the way!
Thank you for your support! Actually I noticed that the more advanced/complicated methods work less in trading, surprisingly simple methods are still the most efficient, for example support resistance and candle patterns, what people tend to ignore is the trade management it's more important than your indicator/forecast.
Hi, you say its not working to predict the difference between the value. But what about predict whether positive/negative/zero Delta Predicting? Because I thinkt thats what matters to make money :D
What kind of loss function are you using? Remember one of the most common loss functions, MAE, will yield the same error regardless of whether your prediction is above or below the true value. For example: todays closing value is 6.0 and tomorrow the closing value will increase to 7.0. If using the MAE loss function, then, say a prediction of either 5.0 or 9.0 will yield the same error (they are equally far away from 7.0). MAE doesn't care that we want to know the direction of the movement, and is equally happy giving us the answer of either 5.0 or 9.0. Although they are the same in terms of absolute error, one if them is a prediction in the true direction, and the other is a prediction in the false direction. But in a real life scenario predicting the direction of movement (up or down) is just as (or even more) important than arriving at a low error. Well, that is my two cents at least. You have to get creative constructing a custom, asymmetric loss function that punishes the model not just for the absolute error but also for predicting in a faulty direction of movement. If you figure this out, please make a video about it and I will perhaps "borrow" your loss function! xD
I am not ignoring your comment I just didn't have a moment to review the code... will get back to you from the mess of the endless files on my desktop 🙂
@@CodeTradingCafe Hello friend. I wanted to ask: what is the best amount of video memory for LSTM? And what graphics card do you use for LSTM machine learning?
@@AliBaba__ Hi, for this type of strategies like in the video we don't need a graphic card so far it's running on CPU power and it's enough, However if you intend to train larger datasets and more neural layers with high complexity models you can go for a simple graphic card 2GB memory should be enough. I hope this helps.
Hello. I’m really new in this world of programming, however I really want to start learning machine learning and coding in trading sphere, may you give some tips of the better way to start?
Hi 3 months python basics with exercises (see my codility playlist for few examples) then numpy pandas matplotlib for 1-2 months. At this point you can already build trading strategies without machine learning. When you feel ready start machine learning but this is a life long journey because there are new tools released almost every year. Good luck it gets fun with time.
Python basics (including classes and inheritance), pandas and numpy, backtesting library (any, this is where you'll need classes). If you want to dive in machine learning it's a long way but for algo trading not worth it. And last but the most important... experience which can only be acquired through coding hours. Good luck!
I'm having trouble getting data using yfinance on intervals less than 1d. It is impossible to get more than two months of any asset. If you have any ideas I would appreciate it. Very good video!
Yes it's limited to 60 days but you can make a loop to download many 60 days batches, it's annoying I know but they probably have a reason for limitations
@@CodeTradingCafe If you have a csv collection of forex majors on low timeframes, I'd appreciate it if you'd share them with me. I can't get them at all. Thanks a lot
@@anderwork7571 I usually download most recent data using dukascopy, I don't have like a proper database, go for dukascopy it provides a lot of historical data you just need to create an account it's free.
Super material. Did you take into account to take additional data, i.e. I am working with BTC and to this I want to add data from the traditional market DXY and additionally sentiment with values for example from -10 to 10. How do you think it makes sense? I will treat DXY and sentiment as an additional indicator.
Thank you! I think your idea makes sense, I haven't included other prices in as indicators input, but this seems interesting to try. keep us posted, and good luck!
Friend I am a trader many years but only now wanting to learn python so found your channel. Is it possible to exclude news from python backbacktests, I think that is a major difference between live performance and backtested performance.
So far news are excluded in my videos, the trading bots usually can probe the news calendar and stop trading during news hours, the best way to test this is live on a paper account.
Great video! Have you ever thought about using a bot to make buy and sell decisions instead of predicting whether the ticket will go up or down? Using deep Q leaning techniques like DQN? Just a suggestion!
Thank you, I did think of reinforcement learning but it involves more parameters than expected sl tp trend trade volume ... Might be challenging but I admit I bam eager to try it 😂
1st of all you need much more candles to train such model maybe a few or several thousands, 2nd features are not correct. You predict return, so features should be e.g. lags of prices not absolute values of EMAS which are not stationary so perhaps lags of emas plus lags of prices plus more candles will help. To compare just calculate RMSE with/without those changes to see if there is any improvement.
Hi thank you for your input, I think we have couple thousands of candles I believe it should be fine since it's not the priority looking at the results. For the lags of prices you might be right I could try this with lags of emas as well and see what it gives but I don't expect any different results, lstms are very tricky to handle.
I spent 4 months trying to predict if bitcoin would rise or fall based on previous candles. I used many indicators and dynamic range of backcandle values. The result was %48 success at best. So flipping a coin has better chance.
Good point bro, instead of predicting some precise values, we can build some models to predict for example trends or convergence based on the right indicators and still alot of preprocessing techniques must be applied to reach lets say some reasonable results.
@@CodeTradingCafe The most challenging part in trading in general! And if you're a smart trader you know trying to predict prices is the worse thing you can do, but I am going to add this to my strategy in the hopes it will give me an edge anyway. It won't be profitable overnight but I'm sure it'll become more accurate after a few consistent months of fine tuning the model
I would be careful with this model as so far it's not really working, but other indicators we have seen on this channel can be really good mainly candle patterns and candle wick absence are the best I have found so far.
but we just need the trends not the actual price right. maybe the profit or loss would be less or more but we can surely get if it would be a profit or loss and trade only when profit is predicted
I was thinking just predicting the future average price for the coming let's say 5 candles... Only predict if it's above or below current candles close so basically it's a binary classification it should be easier this way.
Professional bank trader here. You can make money on market ONLY by eliminating market inefficiencies, not by predicting prices, it's impossible. There is no way to find those only with the chart data. You need full order flow in real time. If people can trade with positive expected value, than algorithms will too. But you have to be a trader yourself, to build a working one. Imagine being a profitable trader and a machine learning engineer at the same time, a unicorn.
Hi thank you for your input, I agree if a trader does it than a an algorithm should as well in theory, the only issue put all your skills as a trader into an algorithm... it's possible only it takes some time. The unicorn... you can craft it in 2 years time with a good teacher :)
do this and make a series pls: # Data Acquisition function gather_data(): collect_stock_data() collect_news_data() collect_social_media_data() save_data() # Preprocessing function preprocess_data(): clean_stock_data() clean_news_data() clean_social_media_data() process_text_data() # Tokenization, stemming, etc. save_preprocessed_data() # Feature Extraction function extract_features(): compute_technical_indicators() compute_sentiment_scores() create_feature_matrix() save_feature_matrix() # Time Series Analysis function train_time_series_model(): fit_ARIMA_or_LSTM_model() save_trained_model() # Ensemble Modeling function train_ensemble_model(): fit_GBM_model() fit_SVM_model() fit_Random_Forest_model() combine_models() # Bagging, boosting, or stacking save_ensemble_model() # Reinforcement Learning function train_trading_strategy(): implement_Q_Learning_or_PPO() optimize_trading_strategy() save_trading_strategy() # Model Evaluation and Selection function evaluate_models(): compute_MAE_or_MSE() compute_Sharpe_Ratio() compare_model_performance() update_ensemble_model() update_trading_strategy() # Continuous Learning function continuous_learning(): while True: gather_data() preprocess_data() extract_features() train_time_series_model() train_ensemble_model() train_trading_strategy() evaluate_models() # Monitoring and Alerts function monitor_performance(): track_model_performance() track_trading_strategy_performance() send_alerts() # Main Execution function main(): gather_data() preprocess_data() extract_features() train_time_series_model() train_ensemble_model() train_trading_strategy() evaluate_models() continuous_learning() monitor_performance() main()
Hi thank you for taking the time to present all the details, this is a lot of work though and I am not sure a video with all of this is something most people can follow, I might however take some parts of your proposition. Thanks again I'll see you around :)
Hi, input data is stationary here, e.g., RSI values are always between 0-100, the price differences as well between 2 limits, we didn't use absolute values for any of the features, even with these considerations it didn't work, the randomness of the market can't be adjusted into a model I think the whole approach here is not optimal.
@@CodeTradingCafe I used to play with keras on EURUSD at tick lvl. I have attained 73% precision with a wavelet compression, for the next 1 minute movement. So at high resolution, market is predictible. I think guessing the next move is a wrong problem since most candles are random, and some others are predictible. the core problem is filtering the noise, aka trying to predict the predictible move, and throwing the non-predictible one. For instance there are moves according to liquidity pockets on the market. But keep on the good work bro.
@@sgrouge Can you explain me a bit about noise filtering? I know that there are some predictable patterns that would give a good signal, however there are just too many indicators out there that I'm just lost while searching for the effective ones
It’s all pure random walk It’s the reason way hedge funds use arbitrages and hedging instead of spending time to beat the market The idea is to see some errors or some extra situations that give certainty that the price level will join a normal values That how professionnel traders works No one of them try to predict the future, it’s just impossible
I presume this whole quant thing is just a waste of time. Looking at results i guess i am not going to make money with data science skills in this life.
No, don't get me wrong, the neural networks thing doesn't work for trading (out of experience and discussions with quants), but algorithmic trading there are plenty of ways to assist in making money, it works, just don't imagine it can turn 1000$ into 1000000$ in couple of months, it's not gambling it doesn't work this way and sometimes it's really slow.
A why i never see in those type of videos people using something other than Japan candals. I mean, come on! Why aren't you using data from order book or volume or eitherscan or some other information that have a significant impact on the price? The price chart is a SCAM. Those fancy technology does absolutely nothing to improve our chances. It's still no better than flip of the coin.
Yes and no... in forex volume is meaningless because it's broker related, same for market depth level 2 data, these only work well for stocks and futures. The price chart is not a scam it's smoothed data for me, the raw tick data is too noisy, just smoothing it can make things easier, and works really well for high timeframes for example 4H and daily these are really easy to trade, albeit slow so you have to be patient and just wait for opportunities, this is where algorithms come into play.
@@CodeTradingCafe Sorry how to implement this code: y_pred = model.predict(X_test) #y_pred=np.where(y_pred > 0.43, 1,0) for i in range(10): print(y_pred[i], y_test[i])
ive got a 6 gb csv file with the last 5000 days of price info for around 4800 companies merged with quarterly financial data like assets revenue etc merged with macro indicators like interest rates and all of that interpolated and cleaned up before passing to an lstm with many many layers with tuned hyper params. instead of using a standard loss value you should use mean absolute percentage error because if a stock is trading at 1$ and you have a loss of 1$ you would be off by 100% but if its trading at 1000$ and yoru loss is 10 then you would be off by 1%.
also try to use time distributed lstm layers at the input and to use leaky relu not just relu so that parts of the model that arnt use often dont just become dead weight . Also use a deeper model to capture the complex relationship between the columns in the training data and MOST IMPORTANTLY..... Kill any over fitting with lots of dropout layers and batch norm
Thanks a lot for your input I will pin your comment for viewers building a LSTM system, will consider these ideas as well in the future. Thank you again.
hey man, I'm doing a university project comparing deep learning methods of stock prediction vs traditional methods such as regression. Can I ask how accurate your model is and where you sourced the data from?
can you give link to your code
Bump, how accurate the model is ?
Can I buy the CSV file ?
The biggest pitfall of anyone starting with time series predictions (of any kind)! Predicting actual values instead of predicting the difference between values. Great video explaining why!
Thank you for your support, it actually is a widespread error all over the internet.
@@CodeTradingCafevery interesting video. I was wondering whether there is another one proposing ways to come up with a model that may exhibit some predictive value
Using Bayesian Optimization I tried optimize hiperparameter such as layers, neurons, dropout, learning rate, backcandles another else but until now I can't predict the next close price, I use 1 minute candle but I can't get a good prediction, indeed I add unsupervised model before supervised model (lstm) and I can't improve the final model... any suggest about what I could do?
@@andresponce2497 the only way I have found to make my prediction better (not good, just a bit better) is to include sentiment analysis
@@Mammel248I read a paper about time series forecasting recently and they combined technical analysis with a neutral network and sentiment analysis. I think it's a vital part because you can never predict the price with only technical analysis because those indicators are not what is driving the market (yes to an extent) it's a lot of sentiment as well.
Name of the paper:
"EvoTrader: Automated Bitcoin Trading Using Neuroevolutionary Algorithms on Technical Analysis and Social Sentiment Data"
They beat buy and hold strategy of Bitcoin by over 200% so really good I would say.
Hi, just want to say I've watched many videos about this topic and that this is the best one because you let me see the mistake in other's results that look good superficially
Glad it was helpful! Thank you for your comment.
This is a valuable lesson about uncertainty and how much 'magic' one can squeeze from AI.
I think you are asking too much from the model. Using the highest w/l strategies, not even the best traders can predict what the price will do at the *next* candle, for *every* candle. They are content with making predictions about the near, unspecified future, at times when many concepts converge on a bias. In essence, *nobody* can predict exact price movements at all times.
Apart from making this demand, the worst 'crime' you ended up committing was throwing away volume information. Your model is tasked with price-action prediction, and you took away *its most critical* input.
I don't know how one would train a model with the fuzziness of "tell me when prices will rise/fall, by approximately how much", which is what traders do.
Probably best, however, would be to skip this approach entirely and task the machine to predict order signals, SL, and TP prices, then measure strategy performance, instead. And please, don't throw away volume.
I agree, it's too much to ask from an AI model or even a trader. I just don't believe in Volume when it comes to Forex (not centralized info), but yes volume is crucial when trading stocks and even Gold. I will get back to some strategies using volume and maybe backtest, had this on my mind for a while now.
Thank you for your input.
Great video, excellent observation about the good predictions phenomena, almost 90% of articles and papers i came across online show high accuracy, that is not applicable for the future only for test data, which is useless!
Thank you! yes actually even academic papers are biased!
I love you , i was stuck for the previous month on this mistake i didn't understand why my model gave predictions near 0 all the time
Thank you for your support. Yes I understand how frustrating this can be sometimes.
I tried to do exactly the same 2 years ago and I gave up. Congrats for the achievement!!
Thank you for your support, although it's not really working but at least we know :)
this is great. Were you able to repurpose LSTm to predict the trend?
Great video! You nicely explained why we encounter a lagged graph. I think what basically happens is that the model begins to "cheat" because the data is more or less useless. So the model just picks the current value and uses it as the future value (since it can't really learn anything from the data). I would argue that when you have useful data, you can theoretically predict the future price. How would one check if this works? Just train your model and calculate the lagged correlation with a range from 0 to the number of days you want to predict into the future. If your argmax is 0 or close enough to 0, you're in good shape and can continue with the additional tests.
Removing the current price from the data the model can train on and predicting the rate of change over a time span will prevent the model from simply creating a lagged graph, but this also assumes that the current price is not useful for making predictions. I'm not sure if this is always a valid assumption.
Hi, thank you for your detailed message. In my opinion (but I might be wrong) providing the current price is not an indicator for future price, somehow I am copying the thinking of a trader, if I as a trader look at the current price value I wouldn't be able to guess the future, but if I look at a series of recent prices aka the slope of the price.
@@CodeTradingCafe In theory, an LSTM should have no problem learning this implicitly, so feature engineering likely won't have a significant impact on the model's predictive power. If the current price is unnecessary, the model would simply disregard that feature. However, when it relies solely on the current price for its predictions, it suggests that the model is either unable to learn from the data or is stuck in a local minimum.
In general, I don't think the issue is with feature engineering (although min-max scaling or standardization can certainly be useful), but rather with finding the right data that the model can actually learn from. To prevent the model from simply predicting the next price as the future price, one could add a penalty for this behavior in the cost function (though, again, having good data is key).
I believe the main challenge in predicting future prices is essentially a search problem-identifying and leveraging the most relevant economic and stock data.
I totally agree it's a search problem, sometimes you might get lucky and the search time is shortened (getting the right info, from the right colleague, or webpage or book ...), but sometimes not, in the latter I guess it's consistent searching.. trial and error.
Nice ! But I sincerely think that data scaling is also a source of problems as most of the features are continuous (ATL, +inf). Therefore, usual scaling methods assume that the previous ATH will never be reached in the future in which case the model, no matter how sophisticated it is, will never predict a breakthrough (new ATHs).
Good observation ! I think these models can only be trained to read the current trend and they can be combined with other models/indicators
@@CodeTradingCafe Precisely! for your information ATH stands for All time high (the maximum historical price). Thanks for the content
scaled_value = (value - median) / IQR
@@domenicperito4635 more precisely please without abbreviations
@@kadourkadouri3505 To calculate the IQR, follow these steps:
Arrange the data in ascending order.
Find the median, or the middle value, of the dataset. If there is an even number of data points, the median is the average of the two middle values.
Divide the dataset into two halves: the lower half (below the median) and the upper half (above the median).
Find the first quartile (Q1), which is the median of the lower half of the data.
Find the third quartile (Q3), which is the median of the upper half of the data.
Calculate the IQR by subtracting Q1 from Q3: IQR = Q3 - Q1.
1- If you calculate power of predicted results few times looks like you will get barely enough results for price change.
2- Instead of predicting how many pips price gonna change, predict how much percent price gonna change rational to the last candle. In other words don't confuse our little brained neural net with exact numbers or pips, instead teach it to predict relatively change of price as daily percent.
Hey, thank you. Your proposition makes perfect sense, I did something similar in another video it's still challenging but needs more time to make it work.
You are certainly right. I have traded cryptocurrency in practice for 3 months. I do the swing trading and clearly realize that it is hardly to calculate or predict the oscillate range of any coin. But I think trend prediction would be a better direction. Transferring the problem from regression to classification may be more efficient🧐
Definitely much easier and more understandable, still challenging though once you start the code and the tuning.
The new output is stil not conclusive. Data analysis is an art. It doesn't at all what values the AI is predicting, the yestion is more if it is giving the right direction most of the time. Is the price going up when price is predicted to go up, even if it has averaged out to be a very small number.
I totally agree, in this shape it's not useful
The problem in shifted values is in fact that regression model predicts value of Close very similar to close of current bar, not future ,even if you have targeted NextClose column with future close values. That’s why you have shifted values. So, they are very similar to previous closes … So I can’t understand why it works like this. Why people invented regression ML models if they work like this
Your description makes sense, but here we are only trying to predict one bar ahead, ideally we would try 5 or even 10 bars in the future, but this only works in different fields (climate forecasting) because there is a pattern followed by nature and the we do understand the pattern well enough to make some forecasting, even there it's also challenging. The trading market is so noisy these models are not able to see a specific pattern.
@@CodeTradingCafe Almost agree. However, you just need to figure it out. I tried various regression options by collecting last candle data from my broker. Knowing that the model can only predict the next value based on the last row (OHLC..), I also created the Features/Target column with a shift up one row. However, every time the values of the predictions of this column come out close to the current Close and not the next one. While the next prediction is close in value (~0.997 r2) to what should have been in the previous line. That is, the predictions of the random number have a close (~0.997 r2) value to what should follow next, but they are somehow shifted. And on your chart, on the video, you also demonstrated this. This breaks my brain. I can't understand why
"the one on the right is predicting the closing price of the next candle and the one on the left is trying to predict the price movement between the two candles" you have said. so meaning at the close of the candle lets say we are on 4h chart . if we use the model on the right to predict the market at the close of the 4 hour chart .literally basing on your explanation we are predicting the close of the next candle. so if we have a almost right predictions for the next close every time a 4h candle closes or daily candle closes using the model in the right cant we use that info to open trades
Definitely not! visually it looks appealing but it has no prediction power. To make it simple most of the times the algorithm is predicting the same close price as the previous candle and we would have this perfect look of predictions only with a slight translation. Again it looks so nice but in reality it's just copying previous closing prices. The problem is that blogs online show this to attract audience but it doesn't provide value.
Would love to see this built getting beginning to end and then modified and back tested.
I was thinking, couldn't you see how high the green line was above the black line on average and add a line of code that was predicted close minus 20 pips equals new target?
Hi, thank you for your input. regarding your question can you please point out the video time related to this?
i guess different parts in the video, but its related to the two line graph lines you display and how they are not exactly in sync with each other. If you knew how far the one was off from the other, you could just add a calculation to add X-amount of pips to the calculation to adjust the line relative to the other line.
@@CodeTradingCafe
Ah ok now I see what you mean, actually no it doesn't work because the translation is on the x-axis and what we see on the y-axis is just the result of it, if we force it let's say 20 pips down it would only look nice for the eye but tricky because the difference is not always 20 pips it's just not seen from the scale of the presented figure.
For the bad predictions graph, I think it can be used as a golden crossover strategy between test and predicted prices
You might as well use a simple moving average probably the same outcome
@@CodeTradingCafe No, moving average has a time delay.
You mean using the actual values instead of the difference it's a mistake because for price prediction we are looking for accurate results. But if we simply want a forecast to understand the direction of the time series, wouldn't we be happy with the results ? In some use cases we might not need an exact number. Do you think in this cases, the model with actual value prediction is still okay ?
If the model can do such exact predictions then yes it's ok, the issue is it's impossible. and only differentiation would inform about the price current and probable future trend.
Could it be that you spent too long thinking about how the neural network works and too little time learning what is useful information for a trading strategy ? The graph your model predicted is nowhere near realated to a moving average, but I agree that it follows the same trajectory as a moving average configured to the smallest time interval. Now why do I say that they are nowhere near related ? Because you forgot to account for WHEN you are receiving the data from the moving average compared to your AI prediction. The strategy itself should not be based on AI. Because the AI makes immesurable decisions. Its the SIGNUM of your positions that the AI should handle. This here is perfect for Trend forecasting. If you implement a solid strategy (iteratively, no AI needed) and just let the AI feed the Strategy the Trend Data. You get an absolute monster of a trading bot.
Maybe... could it be... who knows :)
@@CodeTradingCafe thanks for the python file tho, im just learning how to implement forecasting like this.
I believe you are correct
Dude it was the same problem that made mu furious 2years ago, and I couldn't understand it.
THANKS YOU! This class was a blessing!
Glad it helped! thank you for your support!
Great video and explanation! Thanks!
Nice emojis lol thank you
Great video! how would we modify this to predict multiple targets instead of one?
Hi, thank you, you can add a new category into the target column, but be aware that more categories might make it more difficult for the model to forecast all the possibilities... but that's the beauty of the challenge :)
Does that mean that predicting price fluctuations on a line that is not too attractive like that is not necessarily wrong? Because I was also studying and working on the same problem, and the results were similar!
Well if it's not too attractive it might be right :)
why to limit LSTM. It can use any size of memory. I think no need to set any number of candles to limit... If you use that limit then just use usual ANN instead of LSTM
Definitely need to limit the backcandles even for LSTM otherwise we would be fitting on the whole set of data meaning also data from very long in the past which will not influence current behavior in reality, so past data becomes noise and hinders the model.
Have you trained with a large amount of data, in the code it seems to be only a few thousand data?
The problem is where to get more data, and clean data without noise, it might simply be the bad application case for ML.
yes delayed prediction still a problem when using LSTM model, maybe prediction based on market sentiment is more useful
Also not useful cause you don't know the market sentiment in which period could affect the prices
@@connectrRomania exactly I mean we could easily scrap few articles per day and check the sentiment but when the effect will take place is another story... I would say weekly timeframe in this case is a good candidate but drowning in open trades fees is a problem.
A question on lstm models. Do they care about data stationarity like regression models (eg ARIMA)?
As a first answer no lstm can deal with non stationary data, however I am now wondering if they would perform better on stationary data types... It's a never ending story with these stuff 🙂
@@CodeTradingCafe haha! Yep! Forecasting from value difference as you have shown may be stationary naturally is what comes to mind 😊
Isn't this the principle of using stationary series for predictions instead of wrongly using non-stationary series ?
Not exactly, the idea I wanted to highlight is that if I gave you absolute open close values there is now way we can guess current trend or future trend. Only by considering relative values (differentials) that you get information on price movement.
Edit: However, using stationary data is also a relevant point and shouldn't be neglected, in this video I didn't transform the data into stationary state.
In the MLP network, data from independent variables from date t are used to predict a future value t+n. In the LSTM network, instead of using only data from time t of the independent variables, it uses data from time t, t-1, t-2, ..., t-n as desired by the programmer, and after that, generates the prediction for a future time t+n? Is this reasoning correct? Thank you very much!
Yes exactly, LSTM fits over a window of time and window width is defined by the programmer. The length of the window is however hard to figure, too long vs too short... among other issues.
What was the acuracy of prediction, in porcentage, aproximatelly?
Hi, depends on the limit we use for setting the target category, but in straight answer to your question around 52% if TP/SL = 1
Finally someone said it. The LSTM prediction is as useless as doing df['pred'] = df.close.shift(1). It thinks tomorrow the close price would be the same as today.
Thank you for your comment, I found better results with normal ML classifiers (still not enough for a descent trading model), and even better results with simple NON-ML models using classic price action detection! ironically :)
@@CodeTradingCafe yes same here. I tried a bunch of stuff including using the openai api (gpt-3.5 model) but the backtest shows I actually would have lost about 40% of the initial sum...keep trying...there's bound to be a good ML strategy...I was actually thinking to get historical financial news data, get a sentiment analysis and clump that in the LSTM somehow...I've also looked up the gym-trading lib and reinforcement learning. Here's something I've just watched about it. ruclips.net/video/D9sU1hLT0QY/видео.html. If you get some good results it would be amazing. I'll let you know if I get any good results too
Thank you for sharing this, I haven't seen this video! will take a look at it, just for the fun of RL.
@@CodeTradingCafe no problem. another thing I was trying was using fbprophet (now just prophet) to get weekly and daily seasonality for the price movement...found some interesting results but not so much for signal classification but rather for deciding how much to invest as a % if a signal does appear.
@@andreibudaes3966 that's interesting, someone else did mention fbprophet in here I have to add it on my todo list.
I am sorry but I don't understand the whole point. Normally we use differencing ( what the video author suggests to use as a target) to remove trends. But aren't trends something we want to capture in our model?
And why would differencing make a difference?
Sry for my noobeness
Hi, no worries about questions. Actually if I gave you the closing price of an asset would you be able to predict the future price direction ? I guess not simply because a value of a price has nothing related to the trend, but if I give you the difference between today's and yesterday's prices then you would guess the current trend and its strength and you might be able to extrapolate for future price trends as well.
@@CodeTradingCafe Models like SARIMA (python, statsmodels) allow to indicate the order of differencing. Wouldn't this solve the issue ? Since the prediction will be made on the difference between today's and yesterday's prices (differenced series). The output though is automatically undifferenced for interpretation reasons.
(I am not debating whether SARIMA is a good model for stock price prediction)
you are right SARIMA can make predictions based only on differentiation, ML models can take more than one input feature and from different types, this is why it hyped hopes at some point, and many still believe it can be a powerful solution, although I disagree on this for trading.
when are you going to make next video to this series
Do you mean revisiting neural networks?
@@CodeTradingCafe evaluating the model in order to obtain the better predictions results
@@CodeTradingCafe where are you going to make video on improving on above mistakes?
how to predict values for the next 10 days?
I could give you a beautiful shiny (but fake) answer, or I can tell you the honest truth :) predicting values 10 days ahead relying only on technical analysis is IMPOSSIBLE. But predicting a simple trend (which is more than enough for trading) is maybe possible only for the next day or 2!
What about gbt algorithms like xgboost & catboost for price prediction?
Price prediction is a long shot, I did try xgboost for trend category prediction it's also complicated, simple methods work better!
hm based upon avaraged indicators predict..>> Something slow moving, stears something wildly moving.. this might not be ideal engineering for LSTM use.
LSTM's need patterns, not noice. I'll analyse it deeper tomorow, as it helps me thinking about my tensorflow projects too.
Let us know how it goes, good luck!
i think i have failed to understand you, but my question is if i run my model on the right at the closing of the trading day to predict the nextclose for the next trading day .cant i use that generated prediction to open trades since it predicts the market close for the next closing day. i mean if its on thursday 12:00 midnight or friday 12:15 am and i run my model on the right and i see that the candle on friday will close below the thursday candle ,can't i use those predictions to trade the market
Hi again, (check my answer to your other comment). In brief no because the predictions are really bad they only look good when we "zoom out" showing the full time slice of predictions, if we zoom in and check the details of the results the error is huge and not good for trading.
@@CodeTradingCafe thanks for your feedback actually i had created a trading bot that was using the model that's similar to the one in the right and it was making many losses , would adding more indicators make our model more accurate
Hi, no I don't think adding more indicators would solve the problem, it's more related to the model itself. Steer away from ML for trading, classic programming works better.
@@CodeTradingCafe when i use that code for currency pairs, in the target column i get zero for almost all rows as the change in price movement , what's causing that?
I am suspecting a scaling line somewhere, but need to go through the code to be confirmed.
how can we implement market structure such as wyckoff and elliot waves, how do we draw the lines?
Stay tuned 🙂
@@CodeTradingCafe please and thank you, i have been beating my head about this.
Might be a while though, to be honest I keep most of requests on a list but the list is faster than my video production...
@@CodeTradingCafe That's understandable honestly. The list is too long, nevertheless i am looking forward to it.
What the best books or course to get better in algo trading ?
There is no best book, straight answer is you have to master basics in python or any language, and then research strategies and try them out couple of years later you'll find that you can guess if a strategy is worth it before trying it. However I do have my plan that I follow when putting a new strategy together.
I have a question: to calculate EMAF or RSI you need some historical Data. How do you calculate the first row where there is no historical data? i don´t understand that
It's not calculated for the first few rows, if we choose RSI length 14 for example we start our first values from row 14, when all indicators are calculated we discard the first few rows from our dataframe to keep only clean rows. I hope this answers your question.
@@CodeTradingCafe thanks for your fast respond and explanation! I understand that, but where exactly do the values in the first rows come from? Are they calculated oder just random values? Thanks again!
@@CodeTradingCafe everything is clear now. I just commented the line "dropna..." out and know i get all this NaN values.
Sorry, I'm quite dumb. I'm trying to understand why for a simple classification model using the next closing price wouldn't work
If I'm trying to catch a trend (and really don't care about how much will go up or down), looking at the predicted close isn't enough? So for example if I do:
prev_actual_close < actual_close && prev_actual_close < predicted_close
and gather all results for the testing data, with (just for the example) 90% success rate, doesn't that mean I *could* use it as a "going go go up, not sure how much" type of flag?
I know there are other (maybe) better classification models, but I'm just getting started and have much to learn still, your videos are helping a lot so thank you!
Hi not dumb at all, I did a video previously using machine learning just for classification trend up and trend down ruclips.net/video/q_0rvwPbIqg/видео.html, it did work to an extent but not very robust in the sense that any change in the parameters or different asset the results go back to 50% win rate which is a random indicator baseline. I hope this helps.
@@CodeTradingCafe thanks man! Will check it out! Love your videos btw
I wanted to ask, do these work in real time? For example let's say i have a market structure script, it will work in real time?
Yes I did automated bots previously some are on this channel, although I don't recommend it, I prefer to receive phone alerts and I log in and execute trades myself instead of the bot, so the algorithm is only my assistant.
@@CodeTradingCafe okay i understand, if i should ask, why don't you recommend it?
I was thinking of learning pinescript for the tradingview platform, have been considering it but haven't reach a conclusion yet, are you familiar with it? It is not a full fledged programming language but it get most of the job done, bad thing it doesn't allow external libraries or analyzing more than 30 symbols in one script.
@@Kay-qg1vn I don't recommend it because algorithms can be very powerful at some point and mess up at others (like existing a trade at the correct moment for example).
I am not familiar with other tools only python, c++/metatrader.
Excellent explanation.
Does your model predict different numbers after each running?
Isn't it worth to make range of predictions like in Monte Carlo simulation?
Hi thank you, I don't think any neural net approach is worth it for trading predictions, at this point neural nets perform very poorly on this type of problems.
Is it better to use LSTM on moving average curve? As there will be reduce noise in the data. Thanks a lot for your video!
Hi, yes and no, yes the noise is reduced but the moving average adds a lag to the related information, it might work better on high timeframes... but I don't think it's going to be any better (only a feeling opinion).
@@CodeTradingCafe thank you for your reply, that makes sense. With LSTM May I know what is the best accuracy you can get for the next day stock prediction? (Just in terms of trend, either up or down. I.e if random there should be 50% chance to go up or down)
This has better chances in my opinion, it might be a good video idea as well, ... Hourly timeframe and predict if the average future price is higher than the current price.
thanks for the clarification - maybe worth removing the OHLC and add a lower EMA and then feed to the model? then the result may not be the same previous price?
Good idea actually using the ema instead of the candles here can be good. But we would still need the closing price data for comparison.
@@CodeTradingCafe thanks for your reply. I think the difference between close [i] and close [i-1] or return would be a better option compared to closing price? I guess if we have to then result can be converted to closing price if required. Another issue if there is a difference between the adj close and other OHLC price - i would suggest to convert the OHLC to adjusted values and run the model. it may not be the case for Forex but it is definitely the for stocks.
@@CodeTradingCafe also have you thought about removing the tgt variable from the "independent" set?
Hi, I just watch this video, is there a new update about that?
Hi, thank you for your comment, I haven't done neural networks lately, I was thinking reinforcement learning deserves a try... maybe.
@@CodeTradingCafe great video! You teach excellent
thanks for this excellent video - so basically the LSTM RNNs just dont work for predicting future price action. Is that correct?
I did try and rewatch why training it to predict pip differentials vs future spot price results in wildly different predictive value but dont understand your reasoning, Can you rephrase it for me please?
Hi, the way it's done in this video no it doesn't work. Basically the model is estimating the future by copying the current price which is its best guess.
@@CodeTradingCafe As a proficient trader with significant knowledge of TA and fundamentals, may I make a suggestion that could increase the predictive accuracy substantially? Essentially there is a significant temporal issue here and the very premise of trying to predict next days close price based on previous 30 day data is flawed for a number of reasons (fundamentally, technically etc).
Happy to also take this 'offline' and communicate via email if you prefer.
Sure! that would be great if you have time just either email or even here in the comments, let me know your thoughts now I am curious :)
@@CodeTradingCafe sure what is a good email to reach you at? I will email you from my work email
Hi, codingntrading at gmail dot com it's in the about section of this channel, in case.
Sorry but i don't understand why first pred are wrong, someone could explain me?
Because the model is choosing the same values as previous candles close with a small shift.
@@CodeTradingCafe thank you for reply, still confused, so if i test It with real time data, to predict next 10 candles for example i still get mistakes i guess.
@@williamlacerra1835 yes definitely, this was a demo showing how tricky machine learning can be
Please continue with this series very interesting and useful!
@@williamlacerra1835 more is coming but it takes time to make videos with concise content
Tried lstms on market things and it was terible.
I dont think there is much to do on it from just price data.
Yes my observation as well it doesn't work for this type of data, too much random noise, the model is lost.
@@CodeTradingCafe
I think the only real sucess i have heard of in this area is using deep languge models on news.
The fact u can use a pretrained one REALLY helps with the small data problem
Ah yes language processing might do the trick
@@CodeTradingCafe honestly anns have not showen good results on time seiries data compared to arima.
I have tried it myself and i gota say the results r awful
Excellent video.
Thank you!
I Just got started with algo trading and I was wondering if using a LSTM model is really worth exploring due to this issue that has been brought up by you in this video. If not, do you have any suggestions on what model works best for forex trading in small timeframes such as 1min or 5 min charts? Great video by the way!
Thank you for your support! Actually I noticed that the more advanced/complicated methods work less in trading, surprisingly simple methods are still the most efficient, for example support resistance and candle patterns, what people tend to ignore is the trade management it's more important than your indicator/forecast.
I tried your code for a few stocks and the results are amazing. Thank you
I am glad you liked it. Just warning if you are using this videos code it's NOT supposed to work well for real trading.
How do you pull data for stocks?
Hi, you say its not working to predict the difference between the value. But what about predict whether positive/negative/zero Delta Predicting? Because I thinkt thats what matters to make money :D
Yes we are talking about a classifier which is a good idea to try, it might not work but it's good to see the results
How did you get the data?
Yfinance, my broker, or Dukascopy...
What kind of loss function are you using? Remember one of the most common loss functions, MAE, will yield the same error regardless of whether your prediction is above or below the true value.
For example: todays closing value is 6.0 and tomorrow the closing value will increase to 7.0. If using the MAE loss function, then, say a prediction of either 5.0 or 9.0 will yield the same error (they are equally far away from 7.0). MAE doesn't care that we want to know the direction of the movement, and is equally happy giving us the answer of either 5.0 or 9.0. Although they are the same in terms of absolute error, one if them is a prediction in the true direction, and the other is a prediction in the false direction.
But in a real life scenario predicting the direction of movement (up or down) is just as (or even more) important than arriving at a low error.
Well, that is my two cents at least. You have to get creative constructing a custom, asymmetric loss function that punishes the model not just for the absolute error but also for predicting in a faulty direction of movement.
If you figure this out, please make a video about it and I will perhaps "borrow" your loss function! xD
I am not ignoring your comment I just didn't have a moment to review the code... will get back to you from the mess of the endless files on my desktop 🙂
... using the mse, I see your point no turn around at this point, unless a simple difference function. And to be honest I don't think it changes much.
That has really helped me! Straight to my notebooks for some experiments!!
Glad it's of use! Good luck! ... long coding hours ahead 😂
Какой объем видеопамяти лучше для lstm ? Вы на какой видеокарте выполняете вычисления ?
Hi I am really sorry Russian is one of the few languages I don't understand 🙂
@@CodeTradingCafe Hello friend. I wanted to ask: what is the best amount of video memory for LSTM? And what graphics card do you use for LSTM machine learning?
@@AliBaba__ Hi, for this type of strategies like in the video we don't need a graphic card so far it's running on CPU power and it's enough, However if you intend to train larger datasets and more neural layers with high complexity models you can go for a simple graphic card 2GB memory should be enough. I hope this helps.
Hello. I’m really new in this world of programming, however I really want to start learning machine learning and coding in trading sphere, may you give some tips of the better way to start?
Hi 3 months python basics with exercises (see my codility playlist for few examples) then numpy pandas matplotlib for 1-2 months. At this point you can already build trading strategies without machine learning. When you feel ready start machine learning but this is a life long journey because there are new tools released almost every year. Good luck it gets fun with time.
Why dosnt it soft the sa though I've watched it over and over so many tis?
I am not sure I understand your question sa tis?
Hi brother, can you please share with me a roadmap to becoming an algorithmic trader like you. I have already learnt python.
Python basics (including classes and inheritance), pandas and numpy, backtesting library (any, this is where you'll need classes). If you want to dive in machine learning it's a long way but for algo trading not worth it. And last but the most important... experience which can only be acquired through coding hours. Good luck!
You just saved me a LOT of time, thank you 🙏🏼❤️
I am glad these videos are of help, thank you for your comment! Good luck!
can i use the trained model to trade on the pair that it wasnt trained on ?, ie trading on usdjpy using a model that was trained on usdchf
Hi, I guess not, models barely work on the same currency and mostly they fail, I am not sure switching currencies would work.
i tested this way, it does not work.you can divide close price to open price, then use it for target and also features. but it does not work too.
Lstm has a lot of struggles with this type of random data
Great video. Do you teach somewhere online? Thanks.
Thanks, I don't teach online at the moment, I admit it's something that came to my mind, maybe providing full recorded courses, when time permits.
When are you going make next video while improving on this ??
Left it for a while and the list is getting longer, might not be anytime soon unless I discover an interesting idea I will share it.
I'm having trouble getting data using yfinance on intervals less than 1d. It is impossible to get more than two months of any asset. If you have any ideas I would appreciate it. Very good video!
Yes it's limited to 60 days but you can make a loop to download many 60 days batches, it's annoying I know but they probably have a reason for limitations
@@CodeTradingCafe can you make a video explaining how to correctly join the dataframes? Thanks a lot
@@CodeTradingCafe If you have a csv collection of forex majors on low timeframes, I'd appreciate it if you'd share them with me. I can't get them at all. Thanks a lot
@@anderwork7571 I usually download most recent data using dukascopy, I don't have like a proper database, go for dukascopy it provides a lot of historical data you just need to create an account it's free.
@@CodeTradingCafecan you download sets of 5 min data for 60 days from y finance from over 60 days ago?
Super material. Did you take into account to take additional data, i.e. I am working with BTC and to this I want to add data from the traditional market DXY and additionally sentiment with values for example from -10 to 10. How do you think it makes sense? I will treat DXY and sentiment as an additional indicator.
Thank you! I think your idea makes sense, I haven't included other prices in as indicators input, but this seems interesting to try. keep us posted, and good luck!
Как офигенно поднимает тебя этот сайт. Глянул твой видос и теперь тоже хочу попробовать!
Thank you for your kind support!
Do you ever show any positive results?
Only if I am in good mood :)
Friend I am a trader many years but only now wanting to learn python so found your channel. Is it possible to exclude news from python backbacktests, I think that is a major difference between live performance and backtested performance.
So far news are excluded in my videos, the trading bots usually can probe the news calendar and stop trading during news hours, the best way to test this is live on a paper account.
its an eye opening video for the algo traders . People should stop publishing fake articles on the ML
Yes ML techniques are still very weak for trading maybe just for trend detection.
Great video! Have you ever thought about using a bot to make buy and sell decisions instead of predicting whether the ticket will go up or down? Using deep Q leaning techniques like DQN? Just a suggestion!
Thank you, I did think of reinforcement learning but it involves more parameters than expected sl tp trend trade volume ... Might be challenging but I admit I bam eager to try it 😂
@@CodeTradingCafe , Can you share a video on reinforcement learning or DQN? if you try it. It will really helpful :)
1st of all you need much more candles to train such model maybe a few or several thousands, 2nd features are not correct. You predict return, so features should be e.g. lags of prices not absolute values of EMAS which are not stationary so perhaps lags of emas plus lags of prices plus more candles will help. To compare just calculate RMSE with/without those changes to see if there is any improvement.
Hi thank you for your input, I think we have couple thousands of candles I believe it should be fine since it's not the priority looking at the results. For the lags of prices you might be right I could try this with lags of emas as well and see what it gives but I don't expect any different results, lstms are very tricky to handle.
Thank you a lot for your effort and the valuable content of this channel :)
Thank you for your support, it's really great that the content is useful!
Just compare all model avec naive forecasting : price tomorrow = price yesterday and the results will be more interesting then any other model
Interesting idea! didn't occur to me to compare with previous day price.
It is impossible to predict a value, but predicting a probability of up and down move has a chance.
True, it's easier to guess a trend, we will see how to sort it out... But the lstm was appealing to try, neural networks sound fancy 🙂
I spent 4 months trying to predict if bitcoin would rise or fall based on previous candles. I used many indicators and dynamic range of backcandle values. The result was %48 success at best. So flipping a coin has better chance.
Just reverse the signal😂
@@JonCianci12 Hahaha, so if I make a %5 successful model, I can just reverse the signal and win %95 of the time 😀😀
@@eto38581 Of course it is. All indicators are calculated from the price/volume data. Thus, they don't add any new information.
Good point bro, instead of predicting some precise values, we can build some models to predict for example trends or convergence based on the right indicators and still alot of preprocessing techniques must be applied to reach lets say some reasonable results.
Actually I think the only info we can quickly get from lstm might be the current trend... I will see if I can make it for next week, lots of work 🙂
@@CodeTradingCafe which is still very beneficial toward price action strategies 👌🏽
Yes, I think it's the most challenging part of the algo ... Guessing the overall trend.
@@CodeTradingCafe The most challenging part in trading in general! And if you're a smart trader you know trying to predict prices is the worse thing you can do, but I am going to add this to my strategy in the hopes it will give me an edge anyway. It won't be profitable overnight but I'm sure it'll become more accurate after a few consistent months of fine tuning the model
I would be careful with this model as so far it's not really working, but other indicators we have seen on this channel can be really good mainly candle patterns and candle wick absence are the best I have found so far.
Thanks for the links bro 😊
Glad they're useful, good luck!
but we just need the trends not the actual price right. maybe the profit or loss would be less or more but we can surely get if it would be a profit or loss and trade only when profit is predicted
Well if we can't get the price I will go for the trend, so we need to modify the code for this maybe I will think about something related.
@@CodeTradingCafe maybe i can help
I was thinking just predicting the future average price for the coming let's say 5 candles... Only predict if it's above or below current candles close so basically it's a binary classification it should be easier this way.
mb after reconstruction ts lose patterns
So far neural networks are not performing well on market patterns, simply because these are not periodical and have a lot of randomness.
Professional bank trader here. You can make money on market ONLY by eliminating market inefficiencies, not by predicting prices, it's impossible. There is no way to find those only with the chart data. You need full order flow in real time. If people can trade with positive expected value, than algorithms will too. But you have to be a trader yourself, to build a working one.
Imagine being a profitable trader and a machine learning engineer at the same time, a unicorn.
Hi thank you for your input, I agree if a trader does it than a an algorithm should as well in theory, the only issue put all your skills as a trader into an algorithm... it's possible only it takes some time. The unicorn... you can craft it in 2 years time with a good teacher :)
do this and make a series pls:
# Data Acquisition
function gather_data():
collect_stock_data()
collect_news_data()
collect_social_media_data()
save_data()
# Preprocessing
function preprocess_data():
clean_stock_data()
clean_news_data()
clean_social_media_data()
process_text_data() # Tokenization, stemming, etc.
save_preprocessed_data()
# Feature Extraction
function extract_features():
compute_technical_indicators()
compute_sentiment_scores()
create_feature_matrix()
save_feature_matrix()
# Time Series Analysis
function train_time_series_model():
fit_ARIMA_or_LSTM_model()
save_trained_model()
# Ensemble Modeling
function train_ensemble_model():
fit_GBM_model()
fit_SVM_model()
fit_Random_Forest_model()
combine_models() # Bagging, boosting, or stacking
save_ensemble_model()
# Reinforcement Learning
function train_trading_strategy():
implement_Q_Learning_or_PPO()
optimize_trading_strategy()
save_trading_strategy()
# Model Evaluation and Selection
function evaluate_models():
compute_MAE_or_MSE()
compute_Sharpe_Ratio()
compare_model_performance()
update_ensemble_model()
update_trading_strategy()
# Continuous Learning
function continuous_learning():
while True:
gather_data()
preprocess_data()
extract_features()
train_time_series_model()
train_ensemble_model()
train_trading_strategy()
evaluate_models()
# Monitoring and Alerts
function monitor_performance():
track_model_performance()
track_trading_strategy_performance()
send_alerts()
# Main Execution
function main():
gather_data()
preprocess_data()
extract_features()
train_time_series_model()
train_ensemble_model()
train_trading_strategy()
evaluate_models()
continuous_learning()
monitor_performance()
main()
Hi thank you for taking the time to present all the details, this is a lot of work though and I am not sure a video with all of this is something most people can follow, I might however take some parts of your proposition. Thanks again I'll see you around :)
@@CodeTradingCafe cool
THANCK YOU SO MUTCH IT WORKINR FIN
Thank you for your support.
First, input data must be *stationary*
Second rule, it must be *stationay*
Hi, input data is stationary here, e.g., RSI values are always between 0-100, the price differences as well between 2 limits, we didn't use absolute values for any of the features, even with these considerations it didn't work, the randomness of the market can't be adjusted into a model I think the whole approach here is not optimal.
@@CodeTradingCafe I used to play with keras on EURUSD at tick lvl.
I have attained 73% precision with a wavelet compression, for the next 1 minute movement.
So at high resolution, market is predictible.
I think guessing the next move is a wrong problem since most candles are random, and some others are predictible.
the core problem is filtering the noise, aka trying to predict the predictible move, and throwing the non-predictible one.
For instance there are moves according to liquidity pockets on the market.
But keep on the good work bro.
@@sgrouge 73% is quite impressive using NN. I like your idea on filtering only what is predictable, makes things better.
@@sgrouge Can you explain me a bit about noise filtering? I know that there are some predictable patterns that would give a good signal, however there are just too many indicators out there that I'm just lost while searching for the effective ones
It’s all pure random walk
It’s the reason way hedge funds use arbitrages and hedging instead of spending time to beat the market
The idea is to see some errors or some extra situations that give certainty that the price level will join a normal values
That how professionnel traders works
No one of them try to predict the future, it’s just impossible
Deep down we know it's true but curiosity drives us differently sometimes. However you can always join an ongoing trend for a short time.
Thank you 👍
Welcome! and thank you for your support.
I presume this whole quant thing is just a waste of time. Looking at results i guess i am not going to make money with data science skills in this life.
No, don't get me wrong, the neural networks thing doesn't work for trading (out of experience and discussions with quants), but algorithmic trading there are plenty of ways to assist in making money, it works, just don't imagine it can turn 1000$ into 1000000$ in couple of months, it's not gambling it doesn't work this way and sometimes it's really slow.
Good
Thanks
Sir I need the source file please
Hi check the description of this video ruclips.net/video/hpfQE0bTeA4/видео.html
A why i never see in those type of videos people using something other than Japan candals. I mean, come on! Why aren't you using data from order book or volume or eitherscan or some other information that have a significant impact on the price? The price chart is a SCAM. Those fancy technology does absolutely nothing to improve our chances. It's still no better than flip of the coin.
Yes and no... in forex volume is meaningless because it's broker related, same for market depth level 2 data, these only work well for stocks and futures. The price chart is not a scam it's smoothed data for me, the raw tick data is too noisy, just smoothing it can make things easier, and works really well for high timeframes for example 4H and daily these are really easy to trade, albeit slow so you have to be patient and just wait for opportunities, this is where algorithms come into play.
the problem is overfitting
One of many 😂
If the person posted this video was successful at what he’s talking about he wouldn’t have posted the video
exactly :) because successful trading would simply pile money overnight and then all is good.
Please, god transformers better lstm bad
It might be worth trying transformers but maybe they are data hungry.
@@CodeTradingCafe feature engineering works well for that, add financial indicators to the features.
Both models are crap...
Hi, I don't which models exactly, this video doesn't present any specific models?
Even your tip is bad :) on the right you got lag
Pointless data.
Your prediction must be before real data or on the same index.
Now I am curious which tip from the video :) but yes a look ahead bias is the main pitfall
Xgboost
doesn't really work... tried it.
Please 1 video on cci+obv@@CodeTradingCafe
@@CodeTradingCafeI found than cci+obv did somewhat better than rsi so please 1 video on that too
CCI+OBV on the way :) (I meant on the list)
how to get real prices in this code, how to use (sc.inverse_transform())
the inverse_trasform takes the scaled values as argument and returns the unscaled real prices.
@@CodeTradingCafe Sorry how to implement this code: y_pred = model.predict(X_test)
#y_pred=np.where(y_pred > 0.43, 1,0)
for i in range(10):
print(y_pred[i], y_test[i])