This was just regression. One can't just use regression to "predict" stock prices as everyday the parameters involved are changing. Stock prices prediction involves a lot of different statistical concepts like Brownian Motion, Random Walk etc. There are different models in finance like Black-Scholes equation, Monte Carlo Simulation etc. which take into account these parameters. You have to know a decent amount of math to build a stock prediction algorithm
lmfao, even if you know a 'decent amount of math' you still can't predict stock prices. If you can't predict human behaviour, you can't predict stock prices. Period. What you can use math for, is to capture very short-term trends and buy/sell in very high frequencies using a computer.
this is a video on learning code, not making money. the choice of the stock market as a subject is just a tool to create interest in the real subject of the video: coding ideas. I don't understand how people are so irate in this comments section about this. If the poster, or anybody, made real money with 40 lines of code... none of you would be posting here arguing about it, you'd be making money with 40 lines of code you found on a YT video. the stock market is just the frame for the idea.
@@michaelheery7427 How is being a consistently profitable trader classed as gambling? Its rather contradictory? I suppose it depends on how you define the term gambling but atleast separate the consistently profitable traders from the haphazard emotionally driven buffoons by appending the prefix "professional" and rightfully referring to the former as "Professional Gamblers."
It is really blissful to share great opportunities like this. I never thought i would make it this far in just weeks with just trading and i give accolades to Iqd momentum Forex strategy. I was referred to the author Lukasz Wilhelm only in December of last year after watching series of Siraj Raval wonderful videos on youtube and my life has changed so much since then. Thank you Man .
That's so freaky. I literally just decided to get into stock market prediction yesterday and was having trouble getting started. Perfect timing Siraj. Thank you!!
in fact yesterday, i also wrote a convenient function for downloading stocks from yahoo finance given a range of dates and an interval of time (by days, months, or years). it returns a list of dicts. i used csv.DictReader instead. if anyone's interested in using or tweaking it, i can post it on github.
I know this is an old video, I just wanted to compliment you on your ability to take a very complex topic, and explain it like I'm five. I've heard it said in the past that if you can't Eli 5 a topic then you don't really know it as well as think you do. However, you've displayed that you know this topic very very well. Fantastic job.
Isn't this just overfitting? at 6:24 you can see that you've trained on the variables dates and prices and then also use them as your testing data...or is there something I'm missing?
the graph fits to the training data, i don't plot predicted points but you can print predicted data points to command line using the 2nd helper function.
Seems like Siraj was trying to apply this model for different set for test and could not, after decided to just overfit. Anyway, find his videos very useful. Thanks, Siraj!
This is a toy example. To build a system that works you would need to use Deep Reinforcement learning. And approach the stock market similar to playing a game. The system should be able to identify Good Risk to Reward setups and continuously monitor price action to ascertain whether the odds have changed. It is not enough to just predict the price action. The system should provide a good entry price a stop loss price and a profit exit price.
lol.... this model is all bulls hit.... common sense... make money..... the idea is very simple dear..you don't need 180 IQ to make make money in stock market nor you need any fancy financial model... these are all look Gud and seems interesting. The truth is that any dumb guy can make money in stock market... timing is impossible... it is time in the market and holding period that matters... quarter to quarter earning and reporting don't works... rest is the history...
A good trader can just look at a naked chart and if he sees recurring patterns he knows he has a high probability of winning. This talent alone requires no math and often I see bad traders get bogged down in math, algorithms and indicators when in reality you your brain does it all for you without even thinking about it. Just look at a chart and watch the patterns appear before your eyes.
Very true. Price action trading is the way to go: watch out for candlestick patterns and make your trade based on a risk:reward ratio with stopLoss and takeProfit
I've seen a team from a top uni using something based on rnn to do this and the project last for a year. Their best effort was around 60%~70% in next day accuracy in first 5 months and 40% or less in the rest months. Overall it was not better than tossing a coin or even worse. Because it was "garbage in garbage out", and no one is able to ( or it is not worth to, compared to other profiting methods) get the genuine features of stocking market. However, there is a company using semantic analysis to read news to get a firm's status and ultimately making a bigger profit. But in this story the determiner is still human.
Exactly. Any intro statistics course will tell you regression models help you make predictions for values within the range of your independent variables. Predicting outside this range is extrapolating the model and it has significantly higher risk. I wouldn't necessarily go for it to stack 'dem benjamins.
@@sescalaster nobody gonna be stacking benjamins with a simple regression model with none of the underlying variables taken into account hahaha it's ludicrous
@@sescalaster Hey sergio Quick question Im intersted in math & using data to predict the future. Do you think a data analyst is a good carrer choice? Thank you anyone who can give me clarity
A great video. Reading the reviews it seems a lot of poeple are missing the point. The value of your output is the enthusiasm and inspirations to seek further knowledge. Those hoping to watch a 6 minute video then make a fortune the following day will always just be dreamers.
Great appreciation and praise for his enthusiasm and clear presentation Been through the cycles and have a little bit knowledge about the stocks, trading, investment...etc 1) Long term investment is not the short term speculation which most people engage 2) Modeling is useful only to the extend that you have all the needed data (Such as the over the counter trading, falsified data and behind curtain operations like what happened before the 2008 financial crisis) 3) The best time to make the highest return? Unfortunately, the crisis time...
Cheers for this, I've been looking for "stock market outlook tomorrow" for a while now, and I think this has helped. Have you ever come across - Sanames Stockify Scripophily - (just google it ) ? Ive heard some unbelievable things about it and my friend got excellent results with it.
@@d7eemo270 It is possible to predict asset and also portfolio optimization. but I only trust those who are making living on those predictions, not RUclipsrs.
Exactly, the field and title are nontheless interesting but show it works by testing in the real world. It won't since top investment banks, hedge funds and research firms employ the cream of the crop STEM researchers to model statistical arbitrage of stock markets. Also be careful not to fall into the trap of "over-optimisation" whereby you essentially begin "curve-fitting" your model to known historical data. All falls apart when you "walk" the model forward and test out-of-sample data. There must be a balance for the behaviour of your model.
Really love your videos! I started watching them last night, and can't stop. Although you're a bit fast, I think it actually helps us focus on the things you say/ catch our own mistakes as we code. Great job Siraj! :) Please keep uploading more videos haha
Great tutorial. Had to comment out the poly model as it just hung there. I noticed with a larger data set of 365 days, it took about 10x longer to show the graph compared with the 30 days in this tutorial, but the predict function for predicting on the 366 day for example, never worked. Why? Is it just too much processing for my humble mac?
Having worked in the finance industry, the answer is "no, you can't predict stock prices." The signal to noise ratio makes it so any model you train won't work for long. It's the reason why High Frequency Trading is so dominant today. You don't need to predict the price, if you just skim a few cents per share and execute hundreds of millions of transactions. Before HFT became popular, algorithmic trading made up a few percent of the trading activity. Today HFT is responsible for more than half the activity on the major exchanges. It's neat to use deep learning, but predicting stock prices isn't going to work :)
No, you can't @techn0vids. You are missing a very important variable and that is how the market is going to react, that is something you cannot predict and at best you'll be taking a shot in the dark.
What's missing there in prediction (which are ground of trading ...) .. is the use of some (yep not so many !) patterns to describe the trends of the trade. Also first exercise is to train WITH the patterns useful to add some corrections to the prediction.
woolfel as a trader i totally agree with you, predicting pricing is nonsense, your much better off by doing things like making sure your asset allocations are diverse but not spread to thinly, and diversify across asset classes, and strategy, one trade plan or strategy is not the solution for every trade....one must learn to trade in all market conditions...Markets are Dynamic and conditions can change rapidly...anyone who is new to the market and reading this should check out tastytrade.com and get educated in "Options Trading" God luck out there
Amazing video! Going to use this example in a workshop I'm conducting for business school students this week! Will remember to cite you during the workshop and ask them to check out your channel. Cheers!
Great video series! Technically, what you said about making predictions and visualizing them is not incorrect, but it would be a good idea to note that the predictions are in-sample. We should really make out-of-sample predictions to evaluate model performance!
**Very wrong way** of doing time series prediction. I am sure a dummy model that predict t+1 being t would give a better R2 score. Things is that with market stock time series, the data is heavily correlated and you need to "de-correlate" (for instance x_t = x_t - x_t-1) before doing any regression work, otherwise your are getting a systemic error, x_t-x_t-1. The whole problem is to predict the variation of the "de-correlated" data set (ie the absolute difference between t and t+1 for instance). At least there is one absolute truth with your model, if you use it to trade you are sure to lose money :)
So energetic! and so oversimplified. I love your videos, but it's true. Keep up the good work. you teach people the basics, You're not the only click-baiter on younube
what parameters are used to predict the next movements of a person given his current movement .Siraj could you make a video and teach us this with machine learning and neural networks .
I suspect accurate market prediction would change it's outcome, making it inaccurate in the end) One would have to consider consequences of selling or buying in anticipation of predicted dive/hike, possibly causing rapid bubble growth with inevitable burst, or just a total crash right away. If anything, guys down at "Wall St." would use tools to manipulate trading, actually shaping the market outcome rather than predicting it. Such software would also have to counter anything unusual, especially if it looks like trade anticipates its own activity (someone or something discovered a pattern)
Also, one cool way to get stock data. symbol = 'MSFT' df = web.DataReader(symbol, data_source='yahoo', start='2012-01-01', end='2020-01-01') data = df.filter(items = ['Open'])
Thanks for this video. I found it to be simple to follow and easy to duplicate. The only question/issue is that you are doing the calculation based on the day and if you do it from the middle of one month to the next the graph loops back. I assume the date can be looked at using month and day or maybe month day and year correct?
Syntax issue in line 38 6:30 , predicted(price -> predicted_price & predict.price is not defined since, def predict.prices is in line 14 predict.prices(etc.. etc...) should be there in line 38.
I had to include month and day. Probably best to include year also. Unfortunately, this doesn't work when spanning a month since days from 32 to 99 are added. I modified the get_data routine as follows: def get_data(filename): with open(filename, 'r') as csvfile: csvFileReader = csv.reader(csvfile) next(csvFileReader) day = 1 for row in csvFileReader: print("".join(row[0].split('-')[1:])) dates.append(day) prices.append(float(row[1])) day += 1 return
Yep, it also could exist without Apple. (Don't forget about factors, that affect stocks. One cannot simply predict a result without all necessary data, but you could have very precise expectations, which is better than nothing)
Would it enhance the AI analysis of the price data, if another data field was added to indicate time / date of calendar events that typically effect the stock market. Would adding another AI mechanism that analyzes past significant calendar events, to provide weighted factors for future calendar events. And combine the price line based results with weighted calendar events, in hopes of providing more accurate price predictions?
Have you tried validating your models. If so, within what certainty are they able to predict prices? Also, I wouldn’t bother with Snapchat or Spotify. Snapchat is mainly used for thot communication and Spotify will follow a similar route to Napster and Pandora
one question... are we not essentially doing the data fitting here, rather than actual prediction? I mean we used the (x,y) data, we still do not know yt for unknown xt from the future yet? Btw thanks for all your videos, you teach things really well!!! To the point, covering most important bits, concise, and making it interesting all the way!!!! Many many thanks!!!
thanks a lot! the graph fits the existing data, we can print the predicted price to command line using the helper function. i didn't plot the predicted price good point will do better next time
I have ever seen, such a great tutorial !!>. You are simply superb.. You turned out the traditional teaching method to be fun & fact based all the time!.. Kudos to you!!... I have one question here, how do we plot the predicted price?
Hey, so I tried running this script on my mac and execution starts, but then nothing happens. Tried switching the backend for matplotlib and that didn't help. Any thoughts as to what I'm doing wrong?
Traceback (most recent call last): File "stockprice.py", line 42, in get_data('aapl.csv') File "stockprice.py", line 12, in get_data with open(filename, 'r') as csvfile: IOError: [Errno 2] No such file or directory: 'aapl.csv' Help pls i'm stuck : (
If you are trying it with python 3.6 on windows probably need to change your imports 1. csv is already there no need to import it 2. use numpy+mkl from lfd.uci.edu instead of numpy 3. have to install scipy to make skikit-learn to work also available on above site. 4. matplotlib is fine. other surprise is that google finance is no longer provide link to download the historic prices however you can type the url in address bar to access the historic data www.google.com/finance/historical?q=aapl happy coding :)
OK Folks. Don't try this at home. The code is full of minor misspelling bugs which you will spend hours trying to figure out. Also Google stopped exporting the date value of stock prices in the format this example needs in order to work 'out of the box' so to speak, so you will have to correct for that issue as well. Finally, like all sane persons, if you use python in virtualenv you will need to use the entire directory to tell the program where the csv file is, on linux in my case: get_data('/home/user/Desktop/pystock/aapl.csv').
Nice simple example. A bit simplistic though. Suggestions: Using dates to predict stock price is clearly overfitting, add lagged features. Also, why not compute some error metric in addition of looking at the graph?
Hi Siraj! Great tutorial this! However if we just try prediction with one feature, it is bound to get erroneous values like in this case, the predicted price is off by a good 25 points if we consider the day number as the only parameter. Hence I decided to include month and weekday as well and performed one-hot encoding on it as it gives better results this way. So basically my data set now has 17 columns excluding the price column now. The function that we have made here accepts only 2 parameters, so how should we go about tackling this problem? I was thinking we could maybe store the entire test data as an array of arrays and then pass it to the function. If yes, how should I modify the function to adapt to this change. Please let me know if I'm in the right direction and also suggest a way to implement this part in the predict_price function. Thanks!
So by doing this you check if the existing prices are predicted correctly, how would you get the new ones? you don't show that in this video, for example: how can you predict the prices for the next day, for each hour or something
I like your video. However, I have some questions, so according to your video, you will have a predicted price of over $660 for a certain date. Do you think that's correct? please correct me of I am wrong. I am just learning
Hi everyone, when I ran this line : get_data('AAPL.csv') it shows a error message": NameError: name 'get_data' is not defined" Can Anyone help me. I copied the same code from the video, but that line doesn't work.
my code is producing a graph that does not pass the vertical line test (basically circles around). any suggestions for fixing this? its driving me crazy
I see a lot of people trying to buy and sell stocks based on price data and common statistics used by traders, but they do so without doing any analysis of the business whose stock they’re buying. Is this such a good idea?
Dear Siraj! Thanks for a great tutorial! Could you please explain what does the number 29 mean here: predicted_price = predict_prices(dates, prices, 29) Is that how far the prediction goes? Best, Eugene
Great try. I think these linear regression models are valuable and better than MA(Moving Average) Models which are widely used in daily life. But the cost of calculation is much more than MA.
Something seems wrong but I don't know what. I have a ANN and I fed it GSPC data (Standard and Poors 500) from Jan 2000 to date. The ANN loved it, converged in like 80 iterations, 100%. Ok, I suspect overfitting so I do the cross-validation thing, run the data again for each data vector against the others. Result: 100%. I'm asking it to predict one of five categories, down moderately, down a little, flat, up a little or up moderately. I simply do not believe 100% accuracy is possible but IDK what I'm doing wrong.
The price of the stock is not tied to the profits of a company -- it's an open market where it's worth whatever a buyer is willing to pay. Profit might be an input but it's only one of many.
@Sirajology With the CSV I got the following message: python -m pip install csv Collecting csv Could not find a version that satisfies the requirement csv (from versions: ) No matching distribution found for csv
I don't know if google changed the way they format their CSVs, but I'm getting: File "./stockmarket.py", line 46, in get_data('aapl.csv') # calling get_data method by passing the csv file to it File "./stockmarket.py", line 21, in get_data prices.append(float(row[1])) ValueError: could not convert string to float: -
I tried running dow jones data and it only comes up with a graph with 2 lines. Green and Blue. The blue line goes down diagonally, and the green line is across the bottom. No curves or deviation. Used data from 2015 to 2018. The actual numbers seem to fluctuate fine. Any ideas?
Fantastic!!! I've been searching and searching and searching, yearning for such a vid, and finally youtube recommended me you. ;) Thank you! do your parameters work only on daily timeframes of US stocks? cuz i tried it on Brent Crude Oil of the commodities market with an 8-hour tick dataset, 'SLIGHTLY BETTER THAN RAMDOM' is truly the case, whether tick, 1 min, 5 min, 10 min, 15 min or 30 min. but the market wasn't trending but ranging. does it affect the outcome? thanks
i then check apple vs other investment instruments, and to me, apple doesn't do anything but keep going up only with some normal retracements. meanwhile other instruments always go down when the price is too high. i think that should be considered too. for apple stock, sky's the limit. so, that's too easy for machine learning :P
I have ever seen, such a great tutorial !!>. You are simply superb.. You turned out the traditional teaching method to be fun & fact based all the time!.. Kudos to you!!... I have one question here, how do we plot the predicted price?
I have trouble finding the CSV file on the google finance site. They should information on the stock but no place to click on "historic prices". Is there another way I can access Facebooks csv? I hope Yahoo doesn't have the same problem. :(
Dude, the way you explain the things... easily and with fun. The best teacher ever I know. Thank you for these amazing video lessons.
This was just regression.
One can't just use regression to "predict" stock prices as everyday the parameters involved are changing.
Stock prices prediction involves a lot of different statistical concepts like Brownian Motion, Random Walk etc.
There are different models in finance like Black-Scholes equation, Monte Carlo Simulation etc. which take into account these parameters.
You have to know a decent amount of math to build a stock prediction algorithm
lmfao, even if you know a 'decent amount of math' you still can't predict stock prices. If you can't predict human behaviour, you can't predict stock prices. Period. What you can use math for, is to capture very short-term trends and buy/sell in very high frequencies using a computer.
this is a video on learning code, not making money. the choice of the stock market as a subject is just a tool to create interest in the real subject of the video: coding ideas. I don't understand how people are so irate in this comments section about this. If the poster, or anybody, made real money with 40 lines of code... none of you would be posting here arguing about it, you'd be making money with 40 lines of code you found on a YT video. the stock market is just the frame for the idea.
Aleays remember ITS GAMBLING...
@@michaelheery7427 How is being a consistently profitable trader classed as gambling? Its rather contradictory? I suppose it depends on how you define the term gambling but atleast separate the consistently profitable traders from the haphazard emotionally driven buffoons by appending the prefix "professional" and rightfully referring to the former as "Professional Gamblers."
It is really blissful to share great opportunities like this. I never thought i would make it this far in just weeks with just trading and i give accolades to Iqd momentum Forex strategy. I was referred to the author Lukasz Wilhelm only in December of last year after watching series of Siraj Raval wonderful videos on youtube and my life has changed so much since then. Thank you Man
.
Fuckoff
That's so freaky. I literally just decided to get into stock market prediction yesterday and was having trouble getting started. Perfect timing Siraj. Thank you!!
in fact yesterday, i also wrote a convenient function for downloading stocks from yahoo finance given a range of dates and an interval of time (by days, months, or years). it returns a list of dicts. i used csv.DictReader instead. if anyone's interested in using or tweaking it, i can post it on github.
haha, life is stranger than fiction
I know this is an old video, I just wanted to compliment you on your ability to take a very complex topic, and explain it like I'm five. I've heard it said in the past that if you can't Eli 5 a topic then you don't really know it as well as think you do. However, you've displayed that you know this topic very very well. Fantastic job.
Isn't this just overfitting? at 6:24 you can see that you've trained on the variables dates and prices and then also use them as your testing data...or is there something I'm missing?
You are absolutely right
the graph fits to the training data, i don't plot predicted points but you can print predicted data points to command line using the 2nd helper function.
Thanks! I don't mean to nitpick, but what is your performance on data points outside of the date range upon which you trained?
Doesn't that depends on the time and resources associated with the training?
Seems like Siraj was trying to apply this model for different set for test and could not, after decided to just overfit. Anyway, find his videos very useful. Thanks, Siraj!
This is a toy example. To build a system that works you would need to use Deep Reinforcement learning. And approach the stock market similar to playing a game. The system should be able to identify Good Risk to Reward setups and continuously monitor price action to ascertain whether the odds have changed. It is not enough to just predict the price action. The system should provide a good entry price a stop loss price and a profit exit price.
good point thanks Ramesh
lol.... this model is all bulls hit.... common sense... make money..... the idea is very simple dear..you don't need 180 IQ to make make money in stock market nor you need any fancy financial model... these are all look Gud and seems interesting. The truth is that any dumb guy can make money in stock market... timing is impossible... it is time in the market and holding period that matters... quarter to quarter earning and reporting don't works... rest is the history...
A good trader can just look at a naked chart and if he sees recurring patterns he knows he has a high probability of winning. This talent alone requires no math and often I see bad traders get bogged down in math, algorithms and indicators when in reality you your brain does it all for you without even thinking about it. Just look at a chart and watch the patterns appear before your eyes.
Very true. Price action trading is the way to go: watch out for candlestick patterns and make your trade based on a risk:reward ratio with stopLoss and takeProfit
I've seen a team from a top uni using something based on rnn to do this and the project last for a year. Their best effort was around 60%~70% in next day accuracy in first 5 months and 40% or less in the rest months. Overall it was not better than tossing a coin or even worse. Because it was "garbage in garbage out", and no one is able to ( or it is not worth to, compared to other profiting methods) get the genuine features of stocking market. However, there is a company using semantic analysis to read news to get a firm's status and ultimately making a bigger profit. But in this story the determiner is still human.
It's just regression, not prediction
shhh don't tell the masses
Exactly. Any intro statistics course will tell you regression models help you make predictions for values within the range of your independent variables. Predicting outside this range is extrapolating the model and it has significantly higher risk. I wouldn't necessarily go for it to stack 'dem benjamins.
@@sescalaster nobody gonna be stacking benjamins with a simple regression model with none of the underlying variables taken into account hahaha it's ludicrous
This guy is a poser if you look into his code base o. Github his coding is garbage and most of his stuff is incomplete or doesn't work
@@sescalaster
Hey sergio
Quick question
Im intersted in math & using data to predict the future.
Do you think a data analyst is a good carrer choice?
Thank you anyone who can give me clarity
A great video. Reading the reviews it seems a lot of poeple are missing the point. The value of your output is the enthusiasm and inspirations to seek further knowledge. Those hoping to watch a 6 minute video then make a fortune the following day will always just be dreamers.
Man, you break this down soo much better than other machine learning videos :)
Great appreciation and praise for his enthusiasm and clear presentation
Been through the cycles and have a little bit knowledge about the stocks, trading, investment...etc
1) Long term investment is not the short term speculation which most people engage
2) Modeling is useful only to the extend that you have all the needed data (Such as the over the counter trading, falsified data and behind curtain operations like what happened before the 2008 financial crisis)
3) The best time to make the highest return? Unfortunately, the crisis time...
Just want to thank you, Siraj. Your videos are amazing...They're changing lives, for sure. Best of luck to you, and thanks again.
pandas, pd.read_csv would probably make reading the csv alot faster/ easier.
tru tru
@The Math Student How do you write this in a program with pandas?
df = pd.read_csv(filename)
dates = df.index.values
price = df['price'].values
The Math Student how prediction is done for 2018 year?? I mean what are values for input node ??
Cheers for this, I've been looking for "stock market outlook tomorrow" for a while now, and I think this has helped. Have you ever come across - Sanames Stockify Scripophily - (just google it ) ? Ive heard some unbelievable things about it and my friend got excellent results with it.
If this guy can predict stock price, he won’t doing RUclips
haha that is so very true xD
@@d7eemo270 It is possible to predict asset and also portfolio optimization. but I only trust those who are making living on those predictions, not RUclipsrs.
But that is totally my opinion :D
Spoken like a true ass
Exactly, the field and title are nontheless interesting but show it works by testing in the real world. It won't since top investment banks, hedge funds and research firms employ the cream of the crop STEM researchers to model statistical arbitrage of stock markets.
Also be careful not to fall into the trap of "over-optimisation" whereby you essentially begin "curve-fitting" your model to known historical data. All falls apart when you "walk" the model forward and test out-of-sample data. There must be a balance for the behaviour of your model.
Fantastic Man! You are quick in teaching..like long story short. Subscribed...!!!!
thx
Really love your videos! I started watching them last night, and can't stop. Although you're a bit fast, I think it actually helps us focus on the things you say/ catch our own mistakes as we code. Great job Siraj! :) Please keep uploading more videos haha
Great tutorial. Had to comment out the poly model as it just hung there. I noticed with a larger data set of 365 days, it took about 10x longer to show the graph compared with the 30 days in this tutorial, but the predict function for predicting on the 366 day for example, never worked. Why? Is it just too much processing for my humble mac?
This is a great channel, just what I was looking for. You have a good way of bringing someone up to speed fast.
thanks Scott
it's not a prediction, it's just a fit!
Dude why cant all the CS guys be like you. They are always full of themselves and socially awkward af. Big fan :)
I thought these short length videos wont explain much, but am amazed this is as effective as a 20min. video tutorial. you got one more subscriber! ;)
Having worked in the finance industry, the answer is "no, you can't predict stock prices." The signal to noise ratio makes it so any model you train won't work for long. It's the reason why High Frequency Trading is so dominant today. You don't need to predict the price, if you just skim a few cents per share and execute hundreds of millions of transactions. Before HFT became popular, algorithmic trading made up a few percent of the trading activity. Today HFT is responsible for more than half the activity on the major exchanges. It's neat to use deep learning, but predicting stock prices isn't going to work :)
Tell that to James Simons ^^
No, you can't @techn0vids. You are missing a very important variable and that is how the market is going to react, that is something you cannot predict and at best you'll be taking a shot in the dark.
What's missing there in prediction (which are ground of trading ...) .. is the use of some (yep not so many !) patterns to describe the trends of the trade.
Also first exercise is to train WITH the patterns useful to add some corrections to the prediction.
woolfel as a trader i totally agree with you, predicting pricing is nonsense, your much better off by doing things like making sure your asset allocations are diverse but not spread to thinly, and diversify across asset classes, and strategy, one trade plan or strategy is not the solution for every trade....one must learn to trade in all market conditions...Markets are Dynamic and conditions can change rapidly...anyone who is new to the market and reading this should check out tastytrade.com and get educated in "Options Trading" God luck out there
woolfel please read "the market wizards"
you have a funky channel, I'm definitely subscribing.
this channel is gold. Thx bro
You are amazing!!!! Thanks so much for taking the time to do this
thanks so much! more to come
My gawd man you are like prometheus that bring light to the mankind
LOL thank you
Siraj you're so freaking awesome! Thanks for all the wonderful videos you've taught me so much!
thanks Dalton
Amazing video! Going to use this example in a workshop I'm conducting for business school students this week! Will remember to cite you during the workshop and ask them to check out your channel. Cheers!
'ascii' codec can't decode byte 0xef
Great video series! Technically, what you said about making predictions and visualizing them is not incorrect, but it would be a good idea to note that the predictions are in-sample. We should really make out-of-sample predictions to evaluate model performance!
true, in-sample predictions i'll make a note thanks
True.Also train sample to check whether model is overfitting or not
**Very wrong way** of doing time series prediction. I am sure a dummy model that predict t+1 being t would give a better R2 score. Things is that with market stock time series, the data is heavily correlated and you need to "de-correlate" (for instance x_t = x_t - x_t-1) before doing any regression work, otherwise your are getting a systemic error, x_t-x_t-1. The whole problem is to predict the variation of the "de-correlated" data set (ie the absolute difference between t and t+1 for instance).
At least there is one absolute truth with your model, if you use it to trade you are sure to lose money :)
So energetic! and so oversimplified. I love your videos, but it's true. Keep up the good work. you teach people the basics, You're not the only click-baiter on younube
Thanks for putting this together - will continue to follow your work
Wow, you are making complicated so simple!! Thank you!
anytime!
You can use pandas for simplifying the CSV reading process
Do you know how to implement it for years of dataset instead of just 30 days?
How big is your dataset?
If it is very huge you can use chunk by chunk import using pandas
As in the above video, the dataset downloaded was for only a month...I want to implement it for data of 5 years.
what parameters are used to predict the next movements of a person given his current movement .Siraj could you make a video and teach us this with machine learning and neural networks .
I suspect accurate market prediction would change it's outcome, making it inaccurate in the end) One would have to consider consequences of selling or buying in anticipation of predicted dive/hike, possibly causing rapid bubble growth with inevitable burst, or just a total crash right away. If anything, guys down at "Wall St." would use tools to manipulate trading, actually shaping the market outcome rather than predicting it. Such software would also have to counter anything unusual, especially if it looks like trade anticipates its own activity (someone or something discovered a pattern)
Also, one cool way to get stock data.
symbol = 'MSFT'
df = web.DataReader(symbol, data_source='yahoo', start='2012-01-01', end='2020-01-01')
data = df.filter(items = ['Open'])
Thanks for this video. I found it to be simple to follow and easy to duplicate. The only question/issue is that you are doing the calculation based on the day and if you do it from the middle of one month to the next the graph loops back. I assume the date can be looked at using month and day or maybe month day and year correct?
good point. this is a very simple example and you're right. it can definitely be improved
Siraj you are a python guru!
thanks Mark!
1:50 to 2:07 this is an invaluable addition that makes the whole project easier
well im pretty sure you cant do it without installing those things lol, not a case of easy or not
Why not fully automate by using APIs? :)
Syntax issue in line 38 6:30 , predicted(price -> predicted_price & predict.price is not defined since, def predict.prices is in line 14 predict.prices(etc.. etc...) should be there in line 38.
I just love this guy.
Love back thanks!
I am getting . vertical line of black dots instead of the graph (on python 3)
I had to include month and day. Probably best to include year also. Unfortunately, this doesn't work when spanning a month since days from 32 to 99 are added.
I modified the get_data routine as follows:
def get_data(filename):
with open(filename, 'r') as csvfile:
csvFileReader = csv.reader(csvfile)
next(csvFileReader)
day = 1
for row in csvFileReader:
print("".join(row[0].split('-')[1:]))
dates.append(day)
prices.append(float(row[1]))
day += 1
return
Perfect! You go at a really good pace. Thanks a lot for the knowledge dump.
thanks Faiyaad!
You are awesome man! I watched every video and it has motivated me to learn these programs. I might even do a blog about it! Keep it up!
Robert thats awesome, if you do a blog definitely send me the link
In real life chaos, Apple could not even exist by tomorrow, but awesome topic =)
lol tomorrow will always exist as long we're around.
Yep, it also could exist without Apple. (Don't forget about factors, that affect stocks. One cannot simply predict a result without all necessary data, but you could have very precise expectations, which is better than nothing)
Would it enhance the AI analysis of the price data, if another data field was added to indicate time / date of calendar events that typically effect the stock market. Would adding another AI mechanism that analyzes past significant calendar events, to provide weighted factors for future calendar events. And combine the price line based results with weighted calendar events, in hopes of providing more accurate price predictions?
Brilliant use of WoWS clip! Bravo
Hmmm, but the dataset is backwards. This shows you how to predict prices in the past! Unless I missed something that is.
That's what I don't get it either? So we find the model that fits the data the best, but how can I input a date returning the predicted price?
Have you tried validating your models. If so, within what certainty are they able to predict prices? Also, I wouldn’t bother with Snapchat or Spotify. Snapchat is mainly used for thot communication and Spotify will follow a similar route to Napster and Pandora
one question... are we not essentially doing the data fitting here, rather than actual prediction? I mean we used the (x,y) data, we still do not know yt for unknown xt from the future yet?
Btw thanks for all your videos, you teach things really well!!! To the point, covering most important bits, concise, and making it interesting all the way!!!! Many many thanks!!!
thanks a lot! the graph fits the existing data, we can print the predicted price to command line using the helper function. i didn't plot the predicted price good point will do better next time
I have ever seen, such a great tutorial !!>. You are simply superb.. You turned out the traditional teaching method to be fun & fact based all the time!.. Kudos to you!!... I have one question here, how do we plot the predicted price?
Correct. This is completely worthless for prediction.
Hey could you tell which helper function could we use to predict the future price? I was trying SVR.predict, but i don't think if it is the right one
Hey, so I tried running this script on my mac and execution starts, but then nothing happens. Tried switching the backend for matplotlib and that didn't help. Any thoughts as to what I'm doing wrong?
The game of monkey throwing darts, now in python!
I can't believe all this is coming out of his head. So smart. Ugh
what is the python IDE that used at 2:23 ? It looks really smooth.
sublime text
thanks !
How did you decide that we must use SVM/SVR, no-free-lunch, rbf and parameters like value of C, gamma etc. ?
+Hitesh Vaidya decided to use SVM because its easiest to demo, other points were necessary to explain svm
+Hitesh Vaidya decided to use SVM because its easiest to demo, other points were necessary to explain svm
Traceback (most recent call last):
File "stockprice.py", line 42, in
get_data('aapl.csv')
File "stockprice.py", line 12, in get_data
with open(filename, 'r') as csvfile:
IOError: [Errno 2] No such file or directory: 'aapl.csv'
Help pls i'm stuck : (
If you are trying it with python 3.6 on windows probably need to change your imports
1. csv is already there no need to import it
2. use numpy+mkl from lfd.uci.edu instead of numpy
3. have to install scipy to make skikit-learn to work also available on above site.
4. matplotlib is fine.
other surprise is that google finance is no longer provide link to download the historic prices however you can type the url in address bar to access the historic data www.google.com/finance/historical?q=aapl
happy coding :)
OK Folks. Don't try this at home.
The code is full of minor misspelling bugs which you will spend hours trying to figure out.
Also Google stopped exporting the date value of stock prices in the format this example needs in order to work 'out of the box' so to speak, so you will have to correct for that issue as well. Finally, like all sane persons, if you use python in virtualenv you will need to use the entire directory to tell the program where the csv file is, on linux in my case: get_data('/home/user/Desktop/pystock/aapl.csv').
great videos! would you try an end to end model for this type of problem?
thanks John! i'm sure that has been done, it would be really interesting to see the results.
Nice simple example. A bit simplistic though. Suggestions: Using dates to predict stock price is clearly overfitting, add lagged features. Also, why not compute some error metric in addition of looking at the graph?
Great channel, I subbed. Where did you learn this stuff? University or Internet?
internet
Sirajology Thank you. Are there good Tensorflow moocs you can advise?
Can we have techno fund to predict rather than only on price
I laughed at the machine learning whisper. So funny. I'm pulling the trigger. Siraj, can you do a Google Hangout with us? Let's change the world.
lmao thanks. hmm email me your idea sirajraval1@gmail.com
Hi Siraj! Great tutorial this! However if we just try prediction with one feature, it is bound to get erroneous values like in this case, the predicted price is off by a good 25 points if we consider the day number as the only parameter. Hence I decided to include month and weekday as well and performed one-hot encoding on it as it gives better results this way. So basically my data set now has 17 columns excluding the price column now. The function that we have made here accepts only 2 parameters, so how should we go about tackling this problem? I was thinking we could maybe store the entire test data as an array of arrays and then pass it to the function. If yes, how should I modify the function to adapt to this change. Please let me know if I'm in the right direction and also suggest a way to implement this part in the predict_price function. Thanks!
on line 43 I got a bug
what is it? predicted)price = .... predicted_price = ..... or something else?
So by doing this you check if the existing prices are predicted correctly, how would you get the new ones? you don't show that in this video, for example: how can you predict the prices for the next day, for each hour or something
I like your video. However, I have some questions, so according to your video, you will have a predicted price of over $660 for a certain date. Do you think that's correct? please correct me of I am wrong. I am just learning
Interesting video, like always.
thanks Floretta!
Hi everyone, when I ran this line : get_data('AAPL.csv') it shows a error message": NameError: name 'get_data' is not defined" Can Anyone help me. I copied the same code from the video, but that line doesn't work.
Hey Siraj, we don't have to install csv library with pip. It is already installed.
This video is of 2016 bro...may be it was like that at that time..!!
2016 version of Python didn't have it pre-installed
Apparently it can be predicted, by using progressive A.I. However, currently it's only merchandises and currency exchange rate
How did u predict snpachat ipo price when there is no historical stock data?
Nice video man. By the way did u make any money with ur software?
i couldn't get it to display the graph. don't have any errors. video might be outdated now. or it may be the data csv file.?
If you are working for google you are not getting a free lunch from them. You are working for your lunch.
Siraj, thanks for bringing this info to the masses!
How can I use a API from my broker to collect data?
Don't you think the stock is a non-stationary series, even if we try to make it stationary, don't you think it's too random to predict.
Where is the prediction? All you did was do the regression of the data. Overfiting the model completly. What I am. Missing?
my code is producing a graph that does not pass the vertical line test (basically circles around). any suggestions for fixing this? its driving me crazy
Error module object is not callable plz help me to solve this
I see a lot of people trying to buy and sell stocks based on price data and common statistics used by traders, but they do so without doing any analysis of the business whose stock they’re buying. Is this such a good idea?
Dear Siraj! Thanks for a great tutorial! Could you please explain what does the number 29 mean here: predicted_price = predict_prices(dates, prices, 29)
Is that how far the prediction goes?
Best, Eugene
Great try. I think these linear regression models are valuable and better than MA(Moving Average) Models which are widely used in daily life. But the cost of calculation is much more than MA.
I'm very confused, how do you print forecast data points?
I loved the wolf's reference LMAO
Something seems wrong but I don't know what. I have a ANN and I fed it GSPC data (Standard and Poors 500) from Jan 2000 to date. The ANN loved it, converged in like 80 iterations, 100%. Ok, I suspect overfitting so I do the cross-validation thing, run the data again for each data vector against the others. Result: 100%. I'm asking it to predict one of five categories, down moderately, down a little, flat, up a little or up moderately. I simply do not believe 100% accuracy is possible but IDK what I'm doing wrong.
The price of the stock is not tied to the profits of a company -- it's an open market where it's worth whatever a buyer is willing to pay. Profit might be an input but it's only one of many.
it is one of many inputs yup
I admit ignorance of python. Around 6:31 I get an invalid syntax at line 43. I am not sure what I am doing wrong lol
Awesome dude...great work
Is there an online collection of dataset where we can just call it, then get the real time latest data ?
@Sirajology
With the CSV I got the following message:
python -m pip install csv
Collecting csv
Could not find a version that satisfies the requirement csv (from versions: ) No matching distribution found for csv
csv should come pre installed with any distro of python
yea it should come preinstalled don't worry about it
I don't know if google changed the way they format their CSVs, but I'm getting:
File "./stockmarket.py", line 46, in
get_data('aapl.csv') # calling get_data method by passing the csv file to it
File "./stockmarket.py", line 21, in get_data
prices.append(float(row[1]))
ValueError: could not convert string to float: -
Because some lines of your CSV file have a '-' instead of your number value, just remove the line
*****
Isn't that what row[0].split('-')[0] is for?
my bad, this code is to get the day from the date, it should work anyway
glad you got it
I tried running dow jones data and it only comes up with a graph with 2 lines. Green and Blue. The blue line goes down diagonally, and the green line is across the bottom. No curves or deviation. Used data from 2015 to 2018. The actual numbers seem to fluctuate fine. Any ideas?
Fantastic!!! I've been searching and searching and searching, yearning for such a vid, and finally youtube recommended me you. ;) Thank you!
do your parameters work only on daily timeframes of US stocks?
cuz i tried it on Brent Crude Oil of the commodities market with an 8-hour tick dataset, 'SLIGHTLY BETTER THAN RAMDOM' is truly the case, whether tick, 1 min, 5 min, 10 min, 15 min or 30 min. but the market wasn't trending but ranging. does it affect the outcome? thanks
thanks Oliver!
my parameters are set for daily but you can modify for other intervals
i then check apple vs other investment instruments, and to me, apple doesn't do anything but keep going up only with some normal retracements. meanwhile other instruments always go down when the price is too high. i think that should be considered too. for apple stock, sky's the limit. so, that's too easy for machine learning :P
tweaked it a bit, tested it a few rounds, and found that it's very powerful if you predict 12 periods into the future instead of just 1 period
I have ever seen, such a great tutorial !!>. You are simply superb.. You turned out the traditional teaching method to be fun & fact based all the time!.. Kudos to you!!... I have one question here, how do we plot the predicted price?
exactly. ive the same doubt
Where can we get the file with the data (the .csv file)? Because in GitHub the link points to wikipedia for support Vector Machine.
I have trouble finding the CSV file on the google finance site. They should information on the stock but no place to click on "historic prices". Is there another way I can access Facebooks csv? I hope Yahoo doesn't have the same problem. :(