I feel instead of 30 minutes this videos should be at least 3 hours long. I love how you condensed all the important information but I stell want to learn more about this. At least I want to believe sony devs and researchers watch your video and feel proud. Anyway: your channel is such a hughe well of information. Even as a low level service desk employee I feel like I learn something new with every video you publish!
Ah that's great to hear. The goal of every video is for viewers to hopefully learn something, and encourage their curiosity to explore it more if they want. So yeah, I call this a win! 🙌
One conclussion to draw from Sophy is how *the lack of a time dimension and higher context can limit neural networks* like this, as it has trouble 'waiting' for the conditions to be right for a move, or 'being cautious' to avoid ending up in a tricky situation, like car colissions or very close moves. The AI absolutely aces the efficiency tests and min-maxes the hell out of the challenges due to how it works (best input set, for maximum reward), but feels like highlyskilled, but shortshighted player. Of the four goals described at 6:00, Sophy's AI solution would clearly lack strategy, as passing an oponent is sometimes more complex than just driving faster. Some corners don't lend themselves to passes, the oponent can sacrifice speed to block you and keep you behind. Its the goal where time and a longer, 'slower' tactic tempo often becomes very relevant. Something that's out of the scope of a neural network, I'd say. I wonder if the Track Progress it keeps an eye of (12:48) could be of help there. Its not exacly time, but can be used for some tempo.
There exist techniques out of recent research that attempt to solve planning with deep learning. Problem is lack of context, but it was a problem in language models before and see where we got now with Attention. What Sophy is doing had been attempted before on the same premise; but the resulting prevailing wisdom was to let machine learning figure out how to operate the steering wheel and the pedals, while strategic decisions are made by handwritten code. So the results were previously more on the catastrophic side, or at least considered useless.
My 'first blush' response to the 'still too aggressive' problem would be to add a bad sportmainship penalty for when ANYONE gets a pentaly and have it scaled to how far that car was from the 'offender'. Making it a bad thing for allowing situations where something you might have done, forced someone else to incur a penalty. Probably an overreach, but it was my kneejerk response to the issue.
Amazing video, thanks so much for all the work you put in! I'm not even a Grand Turismo player, but this was incredibly intriguing nonetheless. (And as others have said, your pronunciations of non-English names have been really impressive all around, too!)
One issue for some racing games is that when you play single-player they do not want to spend the computing resources fully modelling the physics of each car. So the AI controlled are not really controlling the same car as the player, they are instead just driving on a curve that seems correct to human players.
Having played with Uniy's Proxy Policy Optimalisation i understand a bit about how this works, and that it's not easy setting up a good reward structure for RL. Especially rewards for strategy seems really hard, since the rewards are a lot of steps in the future. I do wonder if they could have used some imitation learning to start the learning process and make it a bit human like, and then continue training it. On the other hand that might not be necessary in this case.
I think they should have multiple Ai's to choose from.. Like Sophie being the gentle AI, and having an optional more aggressive Ai and being able to race them both. It would give personality to the game like how Pac Man had each ghost have different behaviors.
When they ran the Sophy playtest earlier this year I noticed that there were different 'personalities' or versions of the drivers. I made a video about it over on my second channel
Those micro rewards are still hand crafted. Ideally the algorithm would let the AI figure out what the rewards should be. The ultimate regression should be the final place on the race (or even better: final position in a league). And that's when the AI will be able to figure out completely new strategies by itself. It could figure out that it might be a good idea to crash into an opponent because that opponent is currently #2 in the league, meta stuff like that.
I'm thinking that the "overtaking bonus" and it's reward loop problem could have been simply mitigated by making the "track progression" reward scale with the car's position in the race (lower position -> higher progression). That way an AI actor is incentivized to always overtake to increases its bonus race reward progression and try to stop others from overtaking so that it does not lose its bonus reward. If the penalty for collisions and sportsmanship is tweaked high enough then the penalty risk associated to overtaking would also be high (big loss of reward if a collision vs small rate bonus on race progression) - ideally the AI car should overtake and attempt to stop others from overtaking only when collision chance is very low to minimise the chance of being punished thus also decreasing the aggressiveness? I think this would be a better approach than instant rewards.
Tommy if you feel like it I would like to see a piece on project cars 1 and 2 AI. What I found interesting is that in first game the ai made a huge crash (10+ cars) on a chicane so on a replay I was watching how it acted and while it look clumsy the cars were trying to get out of the situation they were in, some doing better (finding a good route between opponents to get on track) but some other was just backing up (hitting other car) and moving forward to hit the barrier, but at the same time they were turning wheels so they managed to get back on track but it took them more time. So this seems to me that the AI had at least a predefined set of rules to try, and trying to adopt to the situation. The GT Sophy is interesting but up till now not present in gt7 with maybe a patch with it later this year.
In-game microtransactions so agressive it's cheaper to buy the cars IRL Good to see such a massive flaw isn't being ignored, shame how it ruins an otherwise great game, making the gameplay Grindy and awful just to push players to spend on it.
@@MansMan42069 he called it "fine" and ignored that if an update can remove it an update will bring it back, the second Sony feel like they won't get a backlash for putting it back in, you bet they will.
@@LillyP-xs5qe "fine" came after "after" so he acknowledges that it wasn't fine before. Try reading. The doomsaying is making you out to be worse than calling someone a lapdog. My condolences.
Thanks a lot for another great video. I feel the AI is kind of cheating as we humans don't have access to the velocity and acceleration vectors and other crucial data. We have to infer this data based on screen motion and rudimentary force feedback on a controller or wheel. When you drive a car you can feel the G forces which helps you understand but that's not the case when playing video games. It would be much fairer if the AI used exactly the same video output and controller information that we human players get.
I believe this was discussed on this channel as well as there was a GDC panel about it. Creating an omnipotent AI is one thing but creating an AI which will be fun and fit the game is w completely different thing. This was for Doom 2016 where AI had a set of rules that gave players a chance. I recently played GRID Legends, arcade but quite fun game. The AI is far from perfect but it was programmed the way to fit the game. Once in w while you have AI made stupid mistake leading to a big crash, you have the nemesis system in which when you hit another car hard enough the AI will try to return the favour and push you out of the track. It leads to some unpredictability for each race. Sure sometimes it does not work at all like in one case the leader of the race first lost its rear bumper without any contact, next lost the left door on a slight curve to finally go of track with no reason whatsoever on a long straight on Brands Hatch GP long straight. The problem I have with GT (I played 5, 6 and Sport) is that AI gets very small improvements but still is lacking in my opinion. If I have to do the chase the rabbit race when the AI just has predefined range of time in which it completes the race all the cars on track are just obstacles, a moving cons, and what player does is basically a time trial.
I never thought about breaking down an entire track into segments. Very smart. Human drivers don't think like that. Humans often just think about how much deceleration is needed to turn a corner and when to release the brake. No wonder Sophy is so strong. Perhaps I should copy from this strategy and start thinking about segments of the track. The problem is track information is not presented exactly in the game. Should GT7 start presenting more details about each track?? LOL
No. Sophy can teach itself, because its effectiveness can be objectively measured by a computer. As far as etiquette goes, it has only learned a few hard coded rules. Training an AI judge would need a different approach, you would have a big library of incidents with human rulings and an AI would learn to mimic them.
@@AIandGames you totally deserve to be there, join the video co-operative. Your videos are definitely high quality enough to join. Maybe ask Sam from HAI about it. BTW my partner teach AI and deep learning at a high ranking England University and I sent him this video to look at cause it's so interesting.
Gawd... if Sophy is anything like Milestone's ANNA, then it's doomed. I have ALL the motoGP games since year 1 and there's no discernible difference in AI from old games to new. I had high hopes for ANNA to improve the racing but in truth, it seems limited, artificial, and pretty shit really. Where these systems fall down for me so far, is that they can only react ot what IS happening and cannot predict - and therefore plan for - what MIGHT happen given what IS happening in the moment. AI cannot predict potential outcomes except in very limited closed loop systems, as it does not understand human fallibility and unpredictability. So far only adult human perception can do that. It's why AI driven cars can hit pedestrians - because it doesn't consider a man who seems drunk and texting on his phone walking towards a roadway, might become a pedestrian hazard... until he does. In games like this they no doubt will get AI systems to get quite good, possibly even unbeatable, because thats a closed environment. It will sooner or later learn all the parameters. But to then be let AI systems loose in the wildly unpredictable human world, gives me chills. AI implementation is the most fascinating and potentially most devastating impact of technology ever devised. Thanks for a great video.
Great video, Tommy. I'm getting Deep Blue vs. Kasparov and AlphaGo vs. Lee Sedol vibes from Sophy. Any thoughts on whether this would be included in the game? Shouldn't be too hard to run the AI after it is trained, but I suppose that depends on how big the model is. I'm also thing of the "developer acceptance factor". Current in-game AIs might be rubbish and/or cheating bastards, but at least they are understood and in the control of the devs. I'd also like to see Sophy put into a real car and see how much of the training is transferable. Oooh! So many questions!
It might sound creepy, but 'reward' is really just the term given for 'score' to a reinforcement learning system. It's how these things learn, they're really just drawn to the positive numbers.
Soo...when will this be implemented into gt7? The AI back markers block you.. During the race they never give space, they run you off the road, tbone you..They act like you're not their basically.
Hopefully you see this comment, but one thing I wonder in all these machine learning approaches to AI is how they tweak it for different difficulty levels. Is it a matter of changing its rewards? Or maybe changing the weights to make it less "aggressive"?
I believe the primary difficulty scaling mechanism is handicap. A car where the accelerator input and brake input only go half way and no further, for example, and you can limit the steering inputs similarly as well. If a maneuvre is unlikely to succeed because your car is a little too shitty to make it, it's also potentially less likely to be attempted, so the agent may seem less aggressive. In reality if it could, it would rapid fire cover you in obceneties top to bottom, 8 times over, you'd have the middle finger permanently burned into your retina. A fair assumption is that the difference between a proficient driver and a lower skill one is that a proficient driver will make optimal inputs on time to make the track as quickly as possible. While a lower skill driver will brake too late missing the apex or too early/too hard losing fractions of a second here and there, accelerate carefully so they don't spin out since they don't have that great a grip on the vehicle dynamics, will make more erroneous steering inputs which will take longer to correct, and will have a delayed reaction time. So you can model a less proficient driver using vehicle handicap.
My problem with GT in single player in comparison to other games is that most (if not all) races are “chase the rabbit” style. It is not even a proper rolling start with two columns of car going in formation. This gets really tiresome. In GT sport AI feels brain dead and player needs to catch up and win starting from far in the end of the grid. And maybe the AI in gt7 was improved but what is the point of it if the game does not feel like racing with training, qualifications and the race. What is beyond me is that this is in online since gt5 so why not add this to single player. Sorry but I lost my heart for GT long time ago. I bought GT Sport from bargain bin and sure for that price it gave me some fun in licenses, challenges and track learning. What is more in patches PD even added campaign with 300 races (including endurance). But after doing like 150 chase the rabbit I got a bit tired. When I learned that GT7 (and I confirmed this information on gt planet) has less single player content (in terms of number of races, and didn’t even include any endurance races day one) I was disappointed. I might pick it up 3 years from now from a bargain bin.
Forzas "drivatars" is just a salesman pitch. In reality they just follow racelines as closely as possible without hitting other cars, relying entirely on their extra grip and TCS/SC when it comes to cornering. You'll never see Forza AI using technique that intentionally lower/break traction, but you'll see them cornering at impossible speeds. In FH5 they even added very obvious rubberbanding to "pressure" players, which has more effect on AIs speed than what vehicles they use. I have little hope for AI in FM8.
Heya! I'm someone who studies AI! And I gotta say that I am very pleasantly surprised by this video. I am new to your channel and honestly expected an open-door kind of story about the AI but you really went out of your way to make this video accurate, high-quality and entertaining! I have one very, very small tiny nitpick though x3. The part about Deep Reinforcement Learning you got indeed correct, but there's a very small mistake being made in the visualization of the network. A deep learning algorithm, or deep reinforcement learning algorithm technically has 3 or more hidden layers to it, which are the layers between the first and and last layer. Given your visualization of the network only has two hidden layers, it technically is not deep learning. Now I'm done with writing this I feel like such an ass... This is the biggest "uhh, actually"-moment I ever experienced and probably will experience. I'm sorry. Thank you for your incredible video though!
A better way to put this would be drivatars except instead of emulating 1 person it's trained to be perfect. Also my drivatar doesn't divebomb, even in horizon. If I had to choose which to race against in a single player mode it would be drivatars. I'd rather ai that drives like other players instead of something trained to be as perfect as possible. Sophie is simple by comparison and literally the same as the origin of drivatar except that was for the 1st Forza and they determined it wasn't as fun to play against.
that's not really how it works. drivatars don't really work the same way, they only try to emulate some patterns the players present. They don't exactly learn to drive like you, they still work exactly the same as any other AI would. Meaning they're still terrible drivers and need to cheat to keep up with you. Sophy in the other hand is not trained to be perfect but rather to actually learn how to drive properly, without cheating and being dirty. Acting in a more human manner than the drivatars ever could. But of course there's also the problem of not being as adaptable to different skill levels as other AIs (since the AI is not allowed to cheat, like the drivatars or the normal GT AI)
And for Gran Turismo the AI has the same problem as the drivatars for Forza. They are absolut shit and no valid rival. As interesting as AI in gerneral is, today it still sucks in all racing games. I do not know who the "professional players" are that raced the AI in development, but either they are rather famous than professional or they didn't race the AI that is now in game.
I failed to mention that the main in-game AI is not the same as Sophy. But that speaks to a larger issue, in that right now it's as easy to train these systems to scale up/down to different skill levels. Hence right now, they focus entirely on top-tier skill levels. This is still an ongoing issue to solve.
If Sophy has already been added to GT7 it still needs driving lessons if you ask me. The game is way too easy in single player mode and the AI just cannot compete. I would like to know how Sophy was able beat the top drivers though because my 10 year old son can run rings around the Ai in this game and he does it without bashing them..
Saying there's racing in gran turismo is abit optimistic😂😂😂😂 its more like you start at the back of a line of cars and try and work ur way to the lead. Instead of creating a dumb AI get real racing drivers to give there input on how an AI to act
I still think "game AI trained by playing the real game" should be seen as a bigger red flag, rather than something to brag about. In practice, it's just a sign that game developers have absolutely no idea how to refactor the relevant code into a library that can run standalone without the whole game engine because games are all spaghetti. Without rendering and audio and whatnot happening like a normal game, they probably could have trained the AI at 100x+ the speed with just the relevant bits.
@@noctum2854 That's not how it works. Physics is generally complete, regardless of whether you can see it, it would be too fragile otherwise. Some distant car can be nonphysical, following static race lies, but the transfer of it into physical and back out will be purely track-distance based. You can cull some sound and logic based on distance and even frustum but you don't need real rendering for it. You can always remove most of the rendering logic and have the game still operational
Nah i have see spaghetti game codebases and turning them headless is a little filthy looking but not a real problem - especially if it's a one off of a previous title and not something you have to upkeep, it's really a basic hatchet job. They won't be a library but not a problem, you get TCP telemetry and inputs and time manipulation to fast forward it, which is just as well, you can cluster your simulation, AI and training on commodity hardware, separate processes work well and no tighter coupling is desirable. Seems like a bit of a management stunt there, where this was decided without engineers while managers drank champaigne and shouted "SYNERGY!" After all anyone can make a headless game, but it takes Sony to deploy a cluster of obsolete Playstations running an instrumented version of a full game.
![Felix "Unfair" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/Oswujxm2Ag0/mqdefault.jpg)
I feel instead of 30 minutes this videos should be at least 3 hours long. I love how you condensed all the important information but I stell want to learn more about this. At least I want to believe sony devs and researchers watch your video and feel proud. Anyway: your channel is such a hughe well of information. Even as a low level service desk employee I feel like I learn something new with every video you publish!
Ah that's great to hear. The goal of every video is for viewers to hopefully learn something, and encourage their curiosity to explore it more if they want. So yeah, I call this a win! 🙌
One conclussion to draw from Sophy is how *the lack of a time dimension and higher context can limit neural networks* like this, as it has trouble 'waiting' for the conditions to be right for a move, or 'being cautious' to avoid ending up in a tricky situation, like car colissions or very close moves. The AI absolutely aces the efficiency tests and min-maxes the hell out of the challenges due to how it works (best input set, for maximum reward), but feels like highlyskilled, but shortshighted player.
Of the four goals described at 6:00, Sophy's AI solution would clearly lack strategy, as passing an oponent is sometimes more complex than just driving faster. Some corners don't lend themselves to passes, the oponent can sacrifice speed to block you and keep you behind. Its the goal where time and a longer, 'slower' tactic tempo often becomes very relevant.
Something that's out of the scope of a neural network, I'd say.
I wonder if the Track Progress it keeps an eye of (12:48) could be of help there. Its not exacly time, but can be used for some tempo.
There exist techniques out of recent research that attempt to solve planning with deep learning. Problem is lack of context, but it was a problem in language models before and see where we got now with Attention.
What Sophy is doing had been attempted before on the same premise; but the resulting prevailing wisdom was to let machine learning figure out how to operate the steering wheel and the pedals, while strategic decisions are made by handwritten code. So the results were previously more on the catastrophic side, or at least considered useless.
11:10 - Please tell me I'm not the only one who hears "Noodle Network" here :D
My 'first blush' response to the 'still too aggressive' problem would be to add a bad sportmainship penalty for when ANYONE gets a pentaly and have it scaled to how far that car was from the 'offender'. Making it a bad thing for allowing situations where something you might have done, forced someone else to incur a penalty.
Probably an overreach, but it was my kneejerk response to the issue.
Your french prononciation is top tier 👌
Amazing video, thanks so much for all the work you put in! I'm not even a Grand Turismo player, but this was incredibly intriguing nonetheless. (And as others have said, your pronunciations of non-English names have been really impressive all around, too!)
One issue for some racing games is that when you play single-player they do not want to spend the computing resources fully modelling the physics of each car. So the AI controlled are not really controlling the same car as the player, they are instead just driving on a curve that seems correct to human players.
Great new video! Thank you for all your research and work.
Great work 🥳🥳🥳 Thank you 💜💜💜
Having played with Uniy's Proxy Policy Optimalisation i understand a bit about how this works, and that it's not easy setting up a good reward structure for RL. Especially rewards for strategy seems really hard, since the rewards are a lot of steps in the future.
I do wonder if they could have used some imitation learning to start the learning process and make it a bit human like, and then continue training it. On the other hand that might not be necessary in this case.
I think they should have multiple Ai's to choose from..
Like Sophie being the gentle AI, and having an optional more aggressive Ai and being able to race them both.
It would give personality to the game like how Pac Man had each ghost have different behaviors.
When they ran the Sophy playtest earlier this year I noticed that there were different 'personalities' or versions of the drivers. I made a video about it over on my second channel
Those micro rewards are still hand crafted. Ideally the algorithm would let the AI figure out what the rewards should be. The ultimate regression should be the final place on the race (or even better: final position in a league). And that's when the AI will be able to figure out completely new strategies by itself. It could figure out that it might be a good idea to crash into an opponent because that opponent is currently #2 in the league, meta stuff like that.
Imagine the capabilities of this kind of AI not only in racing games but in ALL games
I'm thinking that the "overtaking bonus" and it's reward loop problem could have been simply mitigated by making the "track progression" reward scale with the car's position in the race (lower position -> higher progression). That way an AI actor is incentivized to always overtake to increases its bonus race reward progression and try to stop others from overtaking so that it does not lose its bonus reward. If the penalty for collisions and sportsmanship is tweaked high enough then the penalty risk associated to overtaking would also be high (big loss of reward if a collision vs small rate bonus on race progression) - ideally the AI car should overtake and attempt to stop others from overtaking only when collision chance is very low to minimise the chance of being punished thus also decreasing the aggressiveness? I think this would be a better approach than instant rewards.
Tommy if you feel like it I would like to see a piece on project cars 1 and 2 AI.
What I found interesting is that in first game the ai made a huge crash (10+ cars) on a chicane so on a replay I was watching how it acted and while it look clumsy the cars were trying to get out of the situation they were in, some doing better (finding a good route between opponents to get on track) but some other was just backing up (hitting other car) and moving forward to hit the barrier, but at the same time they were turning wheels so they managed to get back on track but it took them more time.
So this seems to me that the AI had at least a predefined set of rules to try, and trying to adopt to the situation.
The GT Sophy is interesting but up till now not present in gt7 with maybe a patch with it later this year.
In-game microtransactions so agressive it's cheaper to buy the cars IRL
Good to see such a massive flaw isn't being ignored, shame how it ruins an otherwise great game, making the gameplay Grindy and awful just to push players to spend on it.
@@piotrekf943 oh look it's the Sony lapdog, you do know they don't pay you to excuse their awful practices right?
@@LillyP-xs5qe He didn't excuse it, he said it was fixed. You got your panties in a twist there, dear.
@@MansMan42069 he called it "fine" and ignored that if an update can remove it an update will bring it back, the second Sony feel like they won't get a backlash for putting it back in, you bet they will.
@@LillyP-xs5qe "fine" came after "after" so he acknowledges that it wasn't fine before. Try reading.
The doomsaying is making you out to be worse than calling someone a lapdog.
My condolences.
Thanks a lot for another great video.
I feel the AI is kind of cheating as we humans don't have access to the velocity and acceleration vectors and other crucial data. We have to infer this data based on screen motion and rudimentary force feedback on a controller or wheel. When you drive a car you can feel the G forces which helps you understand but that's not the case when playing video games. It would be much fairer if the AI used exactly the same video output and controller information that we human players get.
We humans can see, and judge if we need to correct if we take a corner
The Ai only have data, that data is it's eyes basicly
To be fair, we cheat by having a biological brain which is in many ways more capable then AI
I believe this was discussed on this channel as well as there was a GDC panel about it. Creating an omnipotent AI is one thing but creating an AI which will be fun and fit the game is w completely different thing. This was for Doom 2016 where AI had a set of rules that gave players a chance.
I recently played GRID Legends, arcade but quite fun game. The AI is far from perfect but it was programmed the way to fit the game. Once in w while you have AI made stupid mistake leading to a big crash, you have the nemesis system in which when you hit another car hard enough the AI will try to return the favour and push you out of the track. It leads to some unpredictability for each race. Sure sometimes it does not work at all like in one case the leader of the race first lost its rear bumper without any contact, next lost the left door on a slight curve to finally go of track with no reason whatsoever on a long straight on Brands Hatch GP long straight.
The problem I have with GT (I played 5, 6 and Sport) is that AI gets very small improvements but still is lacking in my opinion. If I have to do the chase the rabbit race when the AI just has predefined range of time in which it completes the race all the cars on track are just obstacles, a moving cons, and what player does is basically a time trial.
I never thought about breaking down an entire track into segments. Very smart. Human drivers don't think like that. Humans often just think about how much deceleration is needed to turn a corner and when to release the brake. No wonder Sophy is so strong. Perhaps I should copy from this strategy and start thinking about segments of the track. The problem is track information is not presented exactly in the game. Should GT7 start presenting more details about each track?? LOL
Of course, one relatively simple application for AI automation lies in autonomous buses for public transit.
I wonder if any of this would be applicable to the penalty system? Judging player vs player interactions against the Sophy model.
No. Sophy can teach itself, because its effectiveness can be objectively measured by a computer. As far as etiquette goes, it has only learned a few hard coded rules. Training an AI judge would need a different approach, you would have a big library of incidents with human rulings and an AI would learn to mimic them.
did jeremy clarkson write this intro 😄
0:33 "in game monitization so aggressive it is cheaper to buy the cars in real life"
Ouch F bombs got dropped real hard there
Btw when you gonna join nebula?
AFAIK you need to be invited on Nebula. So yeah, no sign of that anytime soon.
@@AIandGames you totally deserve to be there, join the video co-operative.
Your videos are definitely high quality enough to join.
Maybe ask Sam from HAI about it.
BTW my partner teach AI and deep learning at a high ranking England University and I sent him this video to look at cause it's so interesting.
@@LillyP-xs5qe Funnily enough I too teach AI at an English University.
@@AIandGames I'm not surprised based on the quality of your videos!
Gawd... if Sophy is anything like Milestone's ANNA, then it's doomed. I have ALL the motoGP games since year 1 and there's no discernible difference in AI from old games to new. I had high hopes for ANNA to improve the racing but in truth, it seems limited, artificial, and pretty shit really. Where these systems fall down for me so far, is that they can only react ot what IS happening and cannot predict - and therefore plan for - what MIGHT happen given what IS happening in the moment. AI cannot predict potential outcomes except in very limited closed loop systems, as it does not understand human fallibility and unpredictability. So far only adult human perception can do that. It's why AI driven cars can hit pedestrians - because it doesn't consider a man who seems drunk and texting on his phone walking towards a roadway, might become a pedestrian hazard... until he does. In games like this they no doubt will get AI systems to get quite good, possibly even unbeatable, because thats a closed environment. It will sooner or later learn all the parameters. But to then be let AI systems loose in the wildly unpredictable human world, gives me chills. AI implementation is the most fascinating and potentially most devastating impact of technology ever devised. Thanks for a great video.
Great video. Very interesting.
Great video, Tommy. I'm getting Deep Blue vs. Kasparov and AlphaGo vs. Lee Sedol vibes from Sophy. Any thoughts on whether this would be included in the game? Shouldn't be too hard to run the AI after it is trained, but I suppose that depends on how big the model is. I'm also thing of the "developer acceptance factor". Current in-game AIs might be rubbish and/or cheating bastards, but at least they are understood and in the control of the devs. I'd also like to see Sophy put into a real car and see how much of the training is transferable. Oooh! So many questions!
Deep Blue! It's always intrigued me so much that I use it as my username!
0:57 Japanese Bill Gates
The whole Reward section was kinda creepy. Sounds almost sentient.
It might sound creepy, but 'reward' is really just the term given for 'score' to a reinforcement learning system. It's how these things learn, they're really just drawn to the positive numbers.
@@AIandGames ain't we all
Soo...when will this be implemented into gt7?
The AI back markers block you..
During the race they never give space, they run you off the road, tbone you..They act like you're not their basically.
Hopefully you see this comment, but one thing I wonder in all these machine learning approaches to AI is how they tweak it for different difficulty levels. Is it a matter of changing its rewards? Or maybe changing the weights to make it less "aggressive"?
I have the same question, i dont know how they do that
I believe the primary difficulty scaling mechanism is handicap. A car where the accelerator input and brake input only go half way and no further, for example, and you can limit the steering inputs similarly as well. If a maneuvre is unlikely to succeed because your car is a little too shitty to make it, it's also potentially less likely to be attempted, so the agent may seem less aggressive. In reality if it could, it would rapid fire cover you in obceneties top to bottom, 8 times over, you'd have the middle finger permanently burned into your retina.
A fair assumption is that the difference between a proficient driver and a lower skill one is that a proficient driver will make optimal inputs on time to make the track as quickly as possible. While a lower skill driver will brake too late missing the apex or too early/too hard losing fractions of a second here and there, accelerate carefully so they don't spin out since they don't have that great a grip on the vehicle dynamics, will make more erroneous steering inputs which will take longer to correct, and will have a delayed reaction time. So you can model a less proficient driver using vehicle handicap.
My problem with GT in single player in comparison to other games is that most (if not all) races are “chase the rabbit” style. It is not even a proper rolling start with two columns of car going in formation. This gets really tiresome.
In GT sport AI feels brain dead and player needs to catch up and win starting from far in the end of the grid.
And maybe the AI in gt7 was improved but what is the point of it if the game does not feel like racing with training, qualifications and the race.
What is beyond me is that this is in online since gt5 so why not add this to single player.
Sorry but I lost my heart for GT long time ago. I bought GT Sport from bargain bin and sure for that price it gave me some fun in licenses, challenges and track learning. What is more in patches PD even added campaign with 300 races (including endurance). But after doing like 150 chase the rabbit I got a bit tired.
When I learned that GT7 (and I confirmed this information on gt planet) has less single player content (in terms of number of races, and didn’t even include any endurance races day one) I was disappointed.
I might pick it up 3 years from now from a bargain bin.
Seems the next phase of Sophie's development should be based around the 'Drivatar' system employed in Forza.
So what is different with drivatar from forza?
Forzas "drivatars" is just a salesman pitch. In reality they just follow racelines as closely as possible without hitting other cars, relying entirely on their extra grip and TCS/SC when it comes to cornering. You'll never see Forza AI using technique that intentionally lower/break traction, but you'll see them cornering at impossible speeds.
In FH5 they even added very obvious rubberbanding to "pressure" players, which has more effect on AIs speed than what vehicles they use. I have little hope for AI in FM8.
FWIW, FIA hasn’t yet blessed off on Gran Turismo 7 yet.
Heya! I'm someone who studies AI! And I gotta say that I am very pleasantly surprised by this video. I am new to your channel and honestly expected an open-door kind of story about the AI but you really went out of your way to make this video accurate, high-quality and entertaining! I have one very, very small tiny nitpick though x3. The part about Deep Reinforcement Learning you got indeed correct, but there's a very small mistake being made in the visualization of the network. A deep learning algorithm, or deep reinforcement learning algorithm technically has 3 or more hidden layers to it, which are the layers between the first and and last layer. Given your visualization of the network only has two hidden layers, it technically is not deep learning. Now I'm done with writing this I feel like such an ass... This is the biggest "uhh, actually"-moment I ever experienced and probably will experience. I'm sorry. Thank you for your incredible video though!
Still waiting on that offline update before I buy this... I got time.
I believe you might not get one.
What about NPCs AI in GTA games(especially in GTA V)
A better way to put this would be drivatars except instead of emulating 1 person it's trained to be perfect.
Also my drivatar doesn't divebomb, even in horizon.
If I had to choose which to race against in a single player mode it would be drivatars. I'd rather ai that drives like other players instead of something trained to be as perfect as possible. Sophie is simple by comparison and literally the same as the origin of drivatar except that was for the 1st Forza and they determined it wasn't as fun to play against.
that's not really how it works.
drivatars don't really work the same way, they only try to emulate some patterns the players present.
They don't exactly learn to drive like you, they still work exactly the same as any other AI would. Meaning they're still terrible drivers and need to cheat to keep up with you.
Sophy in the other hand is not trained to be perfect but rather to actually learn how to drive properly, without cheating and being dirty. Acting in a more human manner than the drivatars ever could.
But of course there's also the problem of not being as adaptable to different skill levels as other AIs (since the AI is not allowed to cheat, like the drivatars or the normal GT AI)
And for Gran Turismo the AI has the same problem as the drivatars for Forza. They are absolut shit and no valid rival. As interesting as AI in gerneral is, today it still sucks in all racing games. I do not know who the "professional players" are that raced the AI in development, but either they are rather famous than professional or they didn't race the AI that is now in game.
I failed to mention that the main in-game AI is not the same as Sophy. But that speaks to a larger issue, in that right now it's as easy to train these systems to scale up/down to different skill levels. Hence right now, they focus entirely on top-tier skill levels. This is still an ongoing issue to solve.
@@AIandGames Good to know. Now I want to race Sophy to test it myself how good it is. :D
Spends years developing an AI and the AI can't leave their driving lines :(
SOPHY isn't even in the game, why did you say it shipped already?
If Sophy has already been added to GT7 it still needs driving lessons if you ask me. The game is way too easy in single player mode and the AI just cannot compete. I would like to know how Sophy was able beat the top drivers though because my 10 year old son can run rings around the Ai in this game and he does it without bashing them..
Saying there's racing in gran turismo is abit optimistic😂😂😂😂 its more like you start at the back of a line of cars and try and work ur way to the lead. Instead of creating a dumb AI get real racing drivers to give there input on how an AI to act
Appreciate that you do mention how shitty gt7's monetisation is. Nobody should support this shit.
Lol it's just lip service. Cry more.
I still think "game AI trained by playing the real game" should be seen as a bigger red flag, rather than something to brag about. In practice, it's just a sign that game developers have absolutely no idea how to refactor the relevant code into a library that can run standalone without the whole game engine because games are all spaghetti. Without rendering and audio and whatnot happening like a normal game, they probably could have trained the AI at 100x+ the speed with just the relevant bits.
Until I got to that part, I assumed they weren't using the entire game but just the physics and controls part of it.
@@noctum2854 That's not how it works. Physics is generally complete, regardless of whether you can see it, it would be too fragile otherwise. Some distant car can be nonphysical, following static race lies, but the transfer of it into physical and back out will be purely track-distance based. You can cull some sound and logic based on distance and even frustum but you don't need real rendering for it. You can always remove most of the rendering logic and have the game still operational
Nah i have see spaghetti game codebases and turning them headless is a little filthy looking but not a real problem - especially if it's a one off of a previous title and not something you have to upkeep, it's really a basic hatchet job. They won't be a library but not a problem, you get TCP telemetry and inputs and time manipulation to fast forward it, which is just as well, you can cluster your simulation, AI and training on commodity hardware, separate processes work well and no tighter coupling is desirable. Seems like a bit of a management stunt there, where this was decided without engineers while managers drank champaigne and shouted "SYNERGY!" After all anyone can make a headless game, but it takes Sony to deploy a cluster of obsolete Playstations running an instrumented version of a full game.
I wish they would spend time and resources improving the physics of the game and the AI instead of trying to make it more like Forza
GT7’s AI is inconsistent and gets a thumbs down from me.
First!
Too bad GT7 is dogshit and this AI would have been better utilized in a better, finished game.