Autonomous Trading System using Reinforcement Learning by Melissa Tan

The idea of predicting financial instruments has been the goal of many due in part to the expectation that predicting these instruments can prove lucrative. Whilst the accurate rediction of price seemed reasonable, they do not necessarily guarantee positive returns due to commissions, large profit draw-downs and excessive switching behaviours. Reinforcement Learning (RL) is an autonomous approach to decision making process through repetitive self-learning and evaluation. The idea is to train an agent to learn to execute an order by acting on a suitable strategy that maximizes profit. In this capstone project, we first conduct a systematic review of 50 literature that applies RL in trading, in particular, to uncover the common theme to maximizing the chance of a successful model. We then prototyped a trading system that applies Proximal Policy Optimization (PPO) which is the brainchild of Schulman et al. (Schulman, Wolski, Dhariwal, Radford, & Klimov, 2017). Thismodel achieved an annualised return of 34.06%and outperformed thestudies by Xiong et al. (Xiong, Liu, Zhong, Yang, & Walid, 2018)whose DDPG model produced an annualised return of 25.87%. We also found that adding technical indicators altered the agent’s trading activities significantly.With the added information, the model achieved a lower annualised return of 27.47% but the result was more consistent to the training performance. In summary, we conclude that RL can be successfully applied to trading, however the models are highly dependent on the characteristics of the underlyingdata, training regimeand the RL model itself, thus a rigorous hyperparameters tuning is required to achieve good result.

[@ai4finance_byfintech]

FinRL: A Deep Reinforcement Learning Library for Quantitative Finance.

[@snivesz32]

I think if you are trying to produce a RL based solution, you should compare to a benchmark of a random decision agent. Then when they get approximately the same results you will realize that the model has no statistically significant edge. Another test would be to compare the RL agent on artificial random walk data instead of real data and see if there is a difference. If they both perform similarly well then you know the agent is not learning anything beyond memorizing the data it was fed. This is the age old bias-variance tradeoff problem at its root. Once you reduce the bias to a level that generalizes well, you no longer find any information gain.

[@kevinalejandro3121]

If You have the binnacle of a consistent trader and You feed the reinforcement learning with that data , it can learn how to trade as the trader??

[@redcabinstudios7248]

Very good study. Appreciate it. I am testing algos in real small tradings, also interested to implement RL. If you want to share give a buzz.

[@meltjl]

The code is available here if you are interested to explore
github.com/meltjl/RL-Trading/blob/master/README.md

[@rakhasaputra6985]

Thank you @mel tjl

[@MatloobKhushi]

@@meltjl Thanks Melissa.

[@hannann6416]

How well does your model generalize to new unseen data?

[@JuanRamirez-di9bl]

This is the most important question!

[@AIstepbystep366]

Would it be possible to share the source code for this algorithm?

[@guregori_san1188]

i checked the code on github, i didnt see any transformation on the data, do you normalize the data at some point, or do you use the raw data as it is outputed from the indicators directly?

[@kadourkadouri3505]

you're probably referring to those dump tutorials where values are normalized (or standardized). It doesn't work that way. Those people are coming from computer engineering backgrounds. Those people are more likely, statistically speaking, python users. Therefore, if you want to gain some knowledge on quantitative finance I strongly suggest you to search for R tutorials in stead.

[@monanica7331]

BTC for $75K by end of this year& Control
of The Currency is already Decentralised And now the China disruption would simply
Decentralise the Mining setup for the better

[@phongdang2874]

I think the non-indicator results are real. If you think about it, if yesterday’s price action is similar to today’s, then why should the AI be forced to make a decision? If yesterday was a red candle and today is a red candle; the AI would probably revert to a “how many red candles in a row” (heinkin ashi) strategy. In Heikin Ashi, if price changes direction 2 times in a month, then a trader is only going to make 2 trades. This explains why the non-indicator AI made minimal amount of decisions.

[@MatloobKhushi]

Thanks for your insight.

[@NIKDEFAULT]

I am a student of USYD, is it possible for me to get the files of this project?

[@MatloobKhushi]

The code is available here github.com/meltjl/RL-Trading/blob/master/README.md

[@ihebbibani7122]

How can you make a presentation of an OVERFITTED model . Worse than that , How does the professor let you even stand in front of people and make the presentation....Incredible....
However ,
Good to know that technical indicators changes the behaviour of the agent
Not sure about the fact the your model (PPO2) performs better than DDPG as It's is overfitting . Actually , I'm sure it will be worse than DDPG as when integrating a commission fee , it already performs worse then DDPG.
Hope you'll be more serious next time..

[@juhanbae7231]

Why would you say it is overfitted?

[@meltjl]

@Iheb
Perhaps in your haste to make a quick judgment, you have skipped through the presentation and have completely missed the point.
The presentation shows the differences between with and without the use of Technical Indicators and how the latter reduced overfitting.
- In 11:21s, there is a comparison between DDPG & PPO2 using the same test date range, in which, PPO2 shows a marginal improvement.
- The DDPG study was performed under the assumption of zero commission. In 12.24s, the table compares the test result of PPO2 with Technical Indicators with various commission rates.
The presentation is part of the course curriculum at the university to allow students to learn and present in front of their fellow classmates. You have the right to comment on the topic, but please exercise some respect and do not attack the professor and the presenter.

[@ihebbibani7122]

@@meltjl
-- If I do remember since the last time that I have seen this video , reinforcement learning performs better without technical in the training set but not wasn't good in the test set however without technical indicator it was a way more stable then the previous one in both train set and test which is according to me less profitable indeed but more stable thus BETTER.
-- I know that your goal was to compare two models and if yours improved without caring about overfitting but you SHOULD for te sake of the presentation and for doing things professionally.
--
masquerade. Also ,you feel attacked because you know deep inside that you acted in a bad faith. That's it.