Hi Martyn, I love this series and the apprentice the effort you have put into sharing your knowledge. One thing I must point out here is that of course if you have a working strategy and you only optimize to have 30 odd permutations then they will fit closer the liner regression line. If you add another 500 permutations naturally you are going to get a much wider variance from the line both better and worse making the R squared value less. I’d be keen to hear how this is cherry picking because of such a small sample size?
Thanks for the amazing content! But I really didn't get this one. You're saying to choose "sensible" values instead of optimizing them because you have a risk of getting a small sample and overfitting. I have two points regarding this statement. Firstly, how do you choose "sensible" values? You can pick some numbers that make sense to you and see that resulting sample size is low. So you choose another set of values which has a bigger sample size. Isn't this an optimization already? Secondly, it looks to me that what you showed in this video is not an example of why you shouldn't use a lot of variables, but an example of poor metric which doesn't take sample size into account. From what you said I can conclude that using a performance metric that takes sample size into account can work great with optimizing a lot of parameters, because you will be able to find those parameter combinations that yield both good profits AND statistical significance.
That was the eye opener like you said it would be! I do have some questions: 1. How do you choose indicators for your system? Do you firstly test them individually with their 3 parameters? I mean I don't think you randomly put different indicators with each other and optimize. 2. I never went beyond just filtering the data in excel - Plotting in and out of sample results in a chart and calculating its correlation seems very interesting. The question is, how do you have both in and out of sample data for each parameter sets? As always thank you!
Hello Martyn, perfect as always! Really enjoying the series. I have some question though. Do you think optimization of SL/TP values only can turn a losing system to winning? As for my current ideas about my EAs I think that optimizing money management could be sufficient. What do you think?
It's all in the circle, I've learned and I'm convinced Risk Management is a golden law, unspoken law which doesn't need to be mentioned, is always good to understand and properly manage your capital when investing, after all, is part of the strategy too!
Clear and concise explanation for the danger of optimizing too many parameters at the same time! I love the examples and the charts! Thank you for your hard work and dedication of putting this series together for the Algo Trading community! Does this still apply with optimizing with money management parameters such as how much deposit to use or how much equity of an account the EA is allowed to use, risk per trade, etc.? Basically parameters that are different from the indicator and strategy rules parameters; are these considered different in meaning and can be optimized with indicator/strategy parameters? So, you could optimize 1 strategy rule parameter, 2 indicator parameters, and 2 MM parameters? Thanks!
Thanks for the great feedback Michael. I would never attempt to optimize things such as starting equity or position size. I would simply select sensible values for these. And of course some performance metrics need a static pos size to give you a true measure, and some require a scaled pos size in relation to equity, to give an accurate measure. More on this in a future episode.
Hi Martyn, you are suggesting to optimize maximum of 3 parameters at one time. If I have say 3 indicators and each of them have 5 parameters, which mean I could only optimize max 3 parameters for each indicator and in my case will be total 9 parameters?
Hi. It's seems for me, that your analysis is spoiled by unreliable results with low number of trades. I guess it will be better to filter results putting aside all that don't have enough trades and then repeat all the analysis you did.
Is it me or was that line chart beautifully drawn?
Really powerful demonstration on a fascinating topic! Thanks for sharing this, it's so easy to steer yourself wrong when optimising
Hi Martyn, I love this series and the apprentice the effort you have put into sharing your knowledge. One thing I must point out here is that of course if you have a working strategy and you only optimize to have 30 odd permutations then they will fit closer the liner regression line. If you add another 500 permutations naturally you are going to get a much wider variance from the line both better and worse making the R squared value less. I’d be keen to hear how this is cherry picking because of such a small sample size?
Thanks for the amazing content! But I really didn't get this one.
You're saying to choose "sensible" values instead of optimizing them because you have a risk of getting a small sample and overfitting. I have two points regarding this statement.
Firstly, how do you choose "sensible" values? You can pick some numbers that make sense to you and see that resulting sample size is low. So you choose another set of values which has a bigger sample size. Isn't this an optimization already?
Secondly, it looks to me that what you showed in this video is not an example of why you shouldn't use a lot of variables, but an example of poor metric which doesn't take sample size into account.
From what you said I can conclude that using a performance metric that takes sample size into account can work great with optimizing a lot of parameters, because you will be able to find those parameter combinations that yield both good profits AND statistical significance.
Beautiful Video, congratulations!!!! Please one question: how do you exactly calculate your final score for each combination of parameters?
That was the eye opener like you said it would be! I do have some questions:
1. How do you choose indicators for your system? Do you firstly test them individually with their 3 parameters? I mean I don't think you randomly put different indicators with each other and optimize.
2. I never went beyond just filtering the data in excel - Plotting in and out of sample results in a chart and calculating its correlation seems very interesting. The question is, how do you have both in and out of sample data for each parameter sets?
As always thank you!
Hello Martyn, perfect as always! Really enjoying the series.
I have some question though. Do you think optimization of SL/TP values only can turn a losing system to winning? As for my current ideas about my EAs I think that optimizing money management could be sufficient. What do you think?
It's all in the circle, I've learned and I'm convinced Risk Management is a golden law, unspoken law which doesn't need to be mentioned, is always good to understand and properly manage your capital when investing, after all, is part of the strategy too!
Very useful and clearly explained - thank-you!!
Clear and concise explanation for the danger of optimizing too many parameters at the same time! I love the examples and the charts! Thank you for your hard work and dedication of putting this series together for the Algo Trading community! Does this still apply with optimizing with money management parameters such as how much deposit to use or how much equity of an account the EA is allowed to use, risk per trade, etc.? Basically parameters that are different from the indicator and strategy rules parameters; are these considered different in meaning and can be optimized with indicator/strategy parameters? So, you could optimize 1 strategy rule parameter, 2 indicator parameters, and 2 MM parameters? Thanks!
Thanks for the great feedback Michael. I would never attempt to optimize things such as starting equity or position size. I would simply select sensible values for these. And of course some performance metrics need a static pos size to give you a true measure, and some require a scaled pos size in relation to equity, to give an accurate measure. More on this in a future episode.
Hi Martyn, you are suggesting to optimize maximum of 3 parameters at one time. If I have say 3 indicators and each of them have 5 parameters, which mean I could only optimize max 3 parameters for each indicator and in my case will be total 9 parameters?
Hi Martyin, I think you forget to mention about the 6th technique or I missed it somehow?
Hi. It's seems for me, that your analysis is spoiled by unreliable results with low number of trades. I guess it will be better to filter results putting aside all that don't have enough trades and then repeat all the analysis you did.