Data professional here (I've worn DBA, data architect, and db architect hats at different times of my professional career) and love the "inside your head" angle of this series of videos as I appreciate the organization of your thoughts and how you express them in your analysis.
Thanks, validating to hear that from someone who actually works with data! I did some Statistics in college but most of what I do is just learned from practical application analyzing data for poker or other games.
Assuming all this data is just lying around, I'm trying to think through how someone might use it. 1. Of players with at least 200 runs, take only the top 100 players by win percentage over the last 100 runs. This will discount the earlier runs, in which we can reasonably assume the players were less skilled. Presumably, these top players have pretty solid fight tactics, so their wins/losses will be more due to relic/card choices and maybe shuffle luck. 2. Look at card choice and relic possession over the last 100 runs (again to discount early, abandoned strategies). We would want to know both how often a particular relic/card is chosen when offered alongside others and how they perform in terms of win percentage. I think this data would not give us a solution to the game, but it would give some insights into the current prevailing strategies, establishing something like a "meta" for the game. It also would be not too different from early, pre-chess-engine analysis of chess games by statistics, such as comparative opening win rates.
Awesome. A great approach to evaluate the elites. Sure, all those contain a ton of things that aren't 100% reliable, but at least it does provide an estimate, and if someone else has different stats or assumptions, the model still works.
Nice video Steve, I like it when you talk mathy to me >:). One comment about your view of heuristics in this video. At one point you mention that any heuristics used for solving this puzzle are almost certainly wrong, because they are made up. Then not long after you note that everything you are doing in this video is using heuristics drawn from the hard mechanics of the game and your experience playing it. This reminded me of another point you made about the possibility that your feelings on what is the best way to play might be wrong. And THAT reminded me of another video you made (I think the one on Corruption?) in which you described the decision space of the game as a topographical surface with local peaks and valleys around certain cards/relics/strategies/etc representing how making choices around those elements affects your chances of winning or losing a run, and how you have not fully explored all of that surface. So, to bring this back to wrong heuristics. You use a certain set of heuristics based how you feel is the right way to play the game. Presumably these heuristics are wrong, but you think they are less wrong than others. However, you are clearly setting up a situation where, if some other player comes along with strategies significantly better than yours, it will be because they used different heuristics that you didn't even give a chance, because all heuristics are probably wrong, but you simply picked some you felt were less wrong. Then, if your goal is to exhaustively search for the winningest strategies, would it not be productive to take almost any vaguely reasonable heuristic (even one you have strong doubts about) that comes along and see where it leads? I feel this would be the best way to unearth strategies that are counterintuitive or obscure, but nevertheless effective. Do you think this is too inefficient? Are you waiting until your current approach doesn't seem to progress any further before exploring other paths? Or are you simply doing what is most enjoyable for you?
This helps a lot but it also makes things a lot more difficult. What about when you're not that high on HP? How good does your deck need to be to actually beat the elites? I have noticed on higher ascension levels that avoiding elites can leave you at an okay life total but can really hinder your deck. I'll start taking a bit more risk for now and see if I can go for elites a bit more often.
We need a relic that gives you 2 block, .3 card choices, .24 health, .24 card draws, .5 max HP and .14 energy at the start of each fight. Call it the mean mug
I'm a bit confused on how you converted the estimated values during a fight to card upgrades. Doesn't it depend on how many cards you have in your deck?
This is a good point. The way he described that section made it sound like he was assuming an upgraded card is getting used once per fight. That might not be the case, though, he might have actually estimated some average number of times the card is played per fight in the same way he estimated lengths of fights for the relics. It's an important factor to consider, but you can't really do it any more or less cleanly than anything else in this video.
Does this game have some kind of API functionality for gathering in-game statistics? This sounds like a problem that Power BI or some other big data tool could solve.
Wow I literally just lost my asc 10 run because time eater ended my turn inbetween my necro skewers, then used his large single attack (which was the worst he could have done because I malaised him for 12).
Generally, Elites are really good. That much is true. However, they also pose a significant risk. On very high ascensions, after elites got their last buff, I find that taking elite fights becomes much more difficult and should only be done sparingly. Especially in act 3, elites are.. iffy. The elites there can randomly end a run that would have had a good chance at beating the heart even. This changes the risk/reward factor of the elites by quite a lot and honestly? If you pick around that, upgrades are extremely potent. It's a bit character dependant and in general I find upgrades to make the most difference on Defect
The big problem with your spreadsheet, and I wish you would have really talked about this, is that your relic "win rate" is not independent for each relic. Your 79% win rate with Orichalcum could be because 30% of the time you had a dead tree branch with that orichalcum, 30% you had a smooth stone with it, 40% of the time you had pantograph with it etc. and those other relics are really why you won; the Orichalcum didn't really do anything but was just along for the ride with other powerful relics. This whole spreadsheet is pretty meaningless and I posit that a similarly-skilled player would have +/- 15% win rate compared to you across the board. The win rates are almost entirely random under this analysis. Furthermore, the best you could hope with a huge sample size are win rates that are somewhat close to your overall win rate. A "good" relic might have a 70% win rate after 10,000 games when your overall win rate is 67%. A "not-so-good" relic might have a 64% win rate. So, on top of not being able to de-correlate the win rates, there's no reliable metric as to what makes a good relic and you'll find that the average impact to your win rate is close to zero. If this is the case, then it's suggesting that the average relic does not improve your success chance at all which is at odds with your initial point about elite fights providing 2 campfires worth of value. I think what you need to do with relics is note when they make a significant impact in your play through the game at the time you're playing. For example, you can note when you lift with Girya and how you feel about that improving your chances to win. Or, you can track how much damage anchor blocks when you wouldn't have been able to make the block otherwise. Then you can grade each relic on some scale (maybe 1 to 10) on how it improved the overall success of a given run. On this scale, a 5 suggests that the relic did nothing, a 1 suggests that the relic lost you the game (i.e. snecko eye in the wrong deck), and a 10 suggests that you would have lost without the relic. You can then more accurately analyze these scores over multiple runs because you've taken the time to de-correlate the impact of a given relic within each run.
My only issue with this video is that there weren't enough spreadsheets. More spreadsheets next time please. You're getting there though.
same
xD
Data professional here (I've worn DBA, data architect, and db architect hats at different times of my professional career) and love the "inside your head" angle of this series of videos as I appreciate the organization of your thoughts and how you express them in your analysis.
Thanks, validating to hear that from someone who actually works with data! I did some Statistics in college but most of what I do is just learned from practical application analyzing data for poker or other games.
This stuff is a rabbit hole. I didn't even realize how interesting spreadsheats could be.
xD
I love the beautiful two-tone lighting
as a statistician I do love my spreadsheets.
Assuming all this data is just lying around, I'm trying to think through how someone might use it.
1. Of players with at least 200 runs, take only the top 100 players by win percentage over the last 100 runs. This will discount the earlier runs, in which we can reasonably assume the players were less skilled. Presumably, these top players have pretty solid fight tactics, so their wins/losses will be more due to relic/card choices and maybe shuffle luck.
2. Look at card choice and relic possession over the last 100 runs (again to discount early, abandoned strategies). We would want to know both how often a particular relic/card is chosen when offered alongside others and how they perform in terms of win percentage.
I think this data would not give us a solution to the game, but it would give some insights into the current prevailing strategies, establishing something like a "meta" for the game. It also would be not too different from early, pre-chess-engine analysis of chess games by statistics, such as comparative opening win rates.
That would certainly be valuable information to look at. The Devs actually have access to information like this that they use to help balance things.
Me before watching this video:
Me after watching: Mind blown
This was fascinating, thank you.
I really want to have a neural network learn how to play this game now...
Awesome. A great approach to evaluate the elites. Sure, all those contain a ton of things that aren't 100% reliable, but at least it does provide an estimate, and if someone else has different stats or assumptions, the model still works.
Nice video Steve, I like it when you talk mathy to me >:). One comment about your view of heuristics in this video. At one point you mention that any heuristics used for solving this puzzle are almost certainly wrong, because they are made up. Then not long after you note that everything you are doing in this video is using heuristics drawn from the hard mechanics of the game and your experience playing it. This reminded me of another point you made about the possibility that your feelings on what is the best way to play might be wrong. And THAT reminded me of another video you made (I think the one on Corruption?) in which you described the decision space of the game as a topographical surface with local peaks and valleys around certain cards/relics/strategies/etc representing how making choices around those elements affects your chances of winning or losing a run, and how you have not fully explored all of that surface.
So, to bring this back to wrong heuristics. You use a certain set of heuristics based how you feel is the right way to play the game. Presumably these heuristics are wrong, but you think they are less wrong than others. However, you are clearly setting up a situation where, if some other player comes along with strategies significantly better than yours, it will be because they used different heuristics that you didn't even give a chance, because all heuristics are probably wrong, but you simply picked some you felt were less wrong. Then, if your goal is to exhaustively search for the winningest strategies, would it not be productive to take almost any vaguely reasonable heuristic (even one you have strong doubts about) that comes along and see where it leads? I feel this would be the best way to unearth strategies that are counterintuitive or obscure, but nevertheless effective. Do you think this is too inefficient? Are you waiting until your current approach doesn't seem to progress any further before exploring other paths? Or are you simply doing what is most enjoyable for you?
This helps a lot but it also makes things a lot more difficult. What about when you're not that high on HP? How good does your deck need to be to actually beat the elites?
I have noticed on higher ascension levels that avoiding elites can leave you at an okay life total but can really hinder your deck. I'll start taking a bit more risk for now and see if I can go for elites a bit more often.
Love this analysis methodology!
We need a relic that gives you 2 block, .3 card choices, .24 health, .24 card draws, .5 max HP and .14 energy at the start of each fight. Call it the mean mug
Pessimitically
All I got from this video is that I’m never taking art of war again. 45% winrate?!? I’m not gonna make my deck worse with a relic.
u can't be serious
Lol
nice meme
Excellent analysis!
I'm a bit confused on how you converted the estimated values during a fight to card upgrades. Doesn't it depend on how many cards you have in your deck?
Yup, my decks don't generally deviate in size too much, but the smaller the deck was the more valuable an upgrade would become.
This is a good point. The way he described that section made it sound like he was assuming an upgraded card is getting used once per fight. That might not be the case, though, he might have actually estimated some average number of times the card is played per fight in the same way he estimated lengths of fights for the relics. It's an important factor to consider, but you can't really do it any more or less cleanly than anything else in this video.
Relics are worth more with Silent and card upgrades are worth less. It's a fair heuristic, but I prefer campfires way more as ironclad.
That's a good point.
3.5k views, 0 dislikes.
This was great
nice spreadsheets , have you heard of eve online?
Does this game have some kind of API functionality for gathering in-game statistics? This sounds like a problem that Power BI or some other big data tool could solve.
ca caw!
Wow I literally just lost my asc 10 run because time eater ended my turn inbetween my necro skewers, then used his large single attack (which was the worst he could have done because I malaised him for 12).
69 relics. Nice
Generally, Elites are really good. That much is true. However, they also pose a significant risk. On very high ascensions, after elites got their last buff, I find that taking elite fights becomes much more difficult and should only be done sparingly. Especially in act 3, elites are.. iffy. The elites there can randomly end a run that would have had a good chance at beating the heart even. This changes the risk/reward factor of the elites by quite a lot and honestly? If you pick around that, upgrades are extremely potent. It's a bit character dependant and in general I find upgrades to make the most difference on Defect
The data “are”. The datum “is”.
how are you watching youtube from the 17th century
@@Jorbs With a tablet. Duh.
The big problem with your spreadsheet, and I wish you would have really talked about this, is that your relic "win rate" is not independent for each relic. Your 79% win rate with Orichalcum could be because 30% of the time you had a dead tree branch with that orichalcum, 30% you had a smooth stone with it, 40% of the time you had pantograph with it etc. and those other relics are really why you won; the Orichalcum didn't really do anything but was just along for the ride with other powerful relics. This whole spreadsheet is pretty meaningless and I posit that a similarly-skilled player would have +/- 15% win rate compared to you across the board. The win rates are almost entirely random under this analysis.
Furthermore, the best you could hope with a huge sample size are win rates that are somewhat close to your overall win rate. A "good" relic might have a 70% win rate after 10,000 games when your overall win rate is 67%. A "not-so-good" relic might have a 64% win rate. So, on top of not being able to de-correlate the win rates, there's no reliable metric as to what makes a good relic and you'll find that the average impact to your win rate is close to zero. If this is the case, then it's suggesting that the average relic does not improve your success chance at all which is at odds with your initial point about elite fights providing 2 campfires worth of value.
I think what you need to do with relics is note when they make a significant impact in your play through the game at the time you're playing. For example, you can note when you lift with Girya and how you feel about that improving your chances to win. Or, you can track how much damage anchor blocks when you wouldn't have been able to make the block otherwise. Then you can grade each relic on some scale (maybe 1 to 10) on how it improved the overall success of a given run. On this scale, a 5 suggests that the relic did nothing, a 1 suggests that the relic lost you the game (i.e. snecko eye in the wrong deck), and a 10 suggests that you would have lost without the relic. You can then more accurately analyze these scores over multiple runs because you've taken the time to de-correlate the impact of a given relic within each run.
15:50?
I know you wrote this 3 years ago, but it is not possible “de-correlate” a relic’s impact through surface-level factors.