check out HubSpot's Free AI Task Delegation Playbook here! clickhubspot.com/s6ef As for the new paper gamegen-o I've shown at 12:00, not actual interactive demo is shown. Looks like the demo is heavily cherry-picked. So yes, gamengen is by far the closest example to an actual AI game engine. edit: Looks like they deleted their gamegen-o project, with no reason specified. Odd for sure.
the AI model is still "deterministic" its just represented in latent neural space than specific measurements, and can behave unpredictably from our perspective
I know from the footage that it forgets enemies you killed in a matter of seconds, same as well with barrels and pickups.. changes the gibs sprite even while in full view,
There is no level, it just tries to guess where the walls are supposed to be lol The game is constantly being procedurally generated, which could be a novel concept
Why are chronically online people so opposed to AI unless it's for gooning to character ai Software engineers are needed to... make the ai... you know... xD
@@px8 At least some people recognize the boring and monotonous aspects of software work that AI can improve such as boilerplate code generation, faster bug fixes, and documentation rewording/simplification. Congrats for seeing the bright side. I'm glad this will actually make more engineers to debug the prompts in addition to the codebase :)
I had a lucid dream once where I was noclipping through the "map" of a dream earlier from that night. There were even missing areas over the hills and stuff that weren't visible from where I had been in the previous dream. It was all as smooth as an actual video game. Indistinguishable experience.
I had a similar dream, but instead of no clipping, I just climbed a rock face up to where you aren't supposed to go and could see outside of the map. It was a lot like that part in Half Life 2 speedruns where they get on top of that narrow valley. Except it was more of a desert environment at night. Then when I was up there, for some reason, Frank Herbert (author of Dune) was there and I had a deep conversation with him. One of the coolest dreams I've ever had.
@@JustinBA007 Badass, man! Apparently very common in lucid dreaming to find wise characters hanging out in the subconscious "behind the scenes." I have a friend who met Will Farrell as a game-show host congratulating him when he stepped backward through a dream wall.
There were earlier AI writers & image generators, which used different methods, often mixing & matching pieces if data based on rules set up by their programmers.
Yeah but it won't see the same change. There's mathematically a point after which you can't optimise stuff. A point where you are dealing with raw information in pure form. Also, mores law is kinda dead. We won't have todays server room fit in a computer, for hundreeds of years, as it would need a complete change in computer. QC won't solve it. They are not faster to do serial task. They just allow some problem that are //able to be shrunk to fewer steps needed to do it.
So, in order to have an IA simulate Doom, you need the following : 1) A functional Doom copy 2) Have it played for 10,000h while recording inputs 3) Have your IA try to simulate Doom for 10,000h while simultaneously feeding it the result of actually playing 10,000 identical hours of normal Doom on a regular computer Sounds like MacGyver quickly assembling a revolver from a shoelace, a bubblegum wrap, and a revolver.
These "generative AI" systems aren't intelligent in the slightest, they just regurgitate slop until they randomly get it right. Practically the definition of monkeys at typewriters.
Very interesting! I imagine they were trying to make a point by not providing any game state, but you might achieve better results if you provided some basic game state (eg location history, health, time in level) as inputs and outputs.
@float32 that could be quite helpful. In general my concern is that if you went AFK for a minute staring at a wall it could lose all context. You could have a low res "rear view" camera which is used to avoid that problem, and not displayed onscreen during play.
@Survivalist_Redo That's fine, but two problems. 1. It can't reason about information it doesn't have within its window. 2. So far, the strong suit of deep learning is not long term temporal consistency. Don't rely on a feature that is a weakness of your technology choice.
@@davidcummins8125The game state doesn't have to be stored in the model itself. There have been successful attempts at giving AI models something akin to a "notebook", where it can "write down" information and later recall it.
I guess it's important here that we understand that GameNGen isn't modeling the functionality of a game; it's modeling the generic game experience - i.e. what it looks and feels like to play; sort of like having a dream. Actually, likening AI to a dream is pretty good analogy to how the results manifest: cause and effect do not correlate properly and images only moderately represent what objects or events really mean.
Neural networks are inspired by our understanding of how biological nervous systems and brains work in the first place, hence the name "neural network". So yes, this is essentially a model that is lucid dreaming gameplay of Doom.
This AI is literally just a framegen with inputs as context, it'a not really capable of anything, and can not be added to. Super inflexible. A cute tech demo, but nothing more. Calling it a "game engine" is just bs to build hype over nothing. Thanks for helping to dispell the misinformation regarding this cool project.
I think any attempts on world simulation with AI would look like human dreams: inconsistent due to short memory and hallucinating crazy things. It might actually be interesting to see, but it will be not comparable with deterministic physical worls simulations in our games.
Which is why its only an expirement, and not any "groundbreaking discovery". We can do it, its pretty trippy and interesting, but not much else, this approach is simply not worth it.
@@alexturnbackthearmy1907 I do think it could be useful for giving robots "imagination" in the sense of the ability to predict what its actions will do to the world around it
The movie "The City of Lost Children" made in 1995 tried hard to simulate in CGI (likely a kind of photo distortion) what a morphing and mutating dream world looks like. Now we begin to have computers do dreams by their own.
Isn't the biggest problem less the GPU power and Ram but the training data? Imagine the huge amount of complexity to alone get that data for most games we have, and to then recreate / retrain the model each time you want a new type of game genre. You would still have to completly code a game first, create textures, logic etc. yourself.. and that with tons of games of the same genre.. just for the AI Game Engine to be able to train what this new genre is about and what you want from it. If you need to do that, you could just code the games yourself instead and save a lot of ressources you use to code the game in first place & then retrain the whole network etc.. i don't think it's really possible to make a good game creation engine from that concept alone if we don't add a aspect to the AI which actually is "aware" and understands what you ask it to do.
I guess the idea would be to train it on thousands of video games together with descriptions of those games and then hope that it would gain the ability to extrapolate to video games from novel descriptions. but it would require a ton of manual labor to add support for each of those games, and even then, thousands of games is not that big of a number in machine learning context, so it might still be way too little
@@asdfghyter An actually realistic AI solution to game dev would rather be to have the AI generate the code needed to have a fully functional game engine, and then get it to make a bunch of graphics for the engine afterwards. At least these problems have generalised AI built for them (even if the code you get can be somewhat sketchy at times)
@@apotato4873 How to let an AI make doom 1. Train an llm on early 90's engines (Doom, Build) 2. Train an image generator on sprites and textures from some WAD files 3. Train some other model on map data (these are essentially graphs)
This technology could probably be tweaked to work a lot more effectively by buttressing the AI some deterministic elements.The domain-specific generation would probably be much more stable if had a proverbial ariadne thread to refer back to.
Any game using randomness would like to disagree: Most board games Monopoly Card games Games involving dice Rogue-like video games Social games, like Among Us
@@Hlebuw3kWhile I don't necessarily agree with the comment you're responding to, this is a still a really weak argument. Relying on some randomness as part of a game isn't the same thing as being completely nondeterministic. All of those games still have very well-defined rules that can't just change on the fly by random chance; only within those rules there's specific points of randomness. More comparable to this would be a version of monopoly where the order and ownership of the properties randomly change or a version of Among Us where the vents might just make you glitch out of the map on occasion, lol.
@@gayusschwulius8490 in the future I'm sure deterministic features will be possible to implement. things that will break a game if not anchored in only one way. hey AI make a racing game where only first place wins and gets a gold medal, any other racer will not be first, but they can compete for first. the biggest challenge is interactability and continuity honestly.
@@Airbigbawls indeed, that's why I don't agree with the original comment saying determinism is 100 % necessary for a game. With a sufficiently advanced AI model that is capable of maintaining consistent gameplay, a nondeterministic game is perfectly possible and might have some massive advantages. Technology still needs to improve a lot for that to happen, though, and I also think that it'd probably make more sense for the rendering of the 3D scene itself being done by a deterministic engine and only the gameplay and 3D scene generation being handled by the AI.
Not only that, but neural networks ARE supposed to be deterministic anyway. Also randomnes in videogames and ANY software running in a computer will never be trully random, everything is pseudo-random
I don't think anyone would want an AI only game. Not only would you most likely never experience the same game twice, your experience would be wildly different from someone else's, which in theory sounds cool, but in practice, not really. It's like if I watched the Two Towers and the kill count of Legolas and Gimli changed each time I watched it.
@@brandongregori995 Still governed by a defined set of deterministic algorithms that will always produce the same output given the same input as opposed to a black box that’s the complete opposite My argument is the lack of application for this for anything other than (pseudo-)randomness, which is already covered by more efficient means anyways
@@chadyways8750 All algorithms are deterministic. LLMs are deterministic, but they are configured to use CPU and GPU temperatures to facilitate random outcomes. You can do this with procedural generation too. There is nothing stopping a seed from having current temps, datetime, and other things appended to it to give you seemingly non-deterministic outcomes.
@@brandongregori995 determinism is largely irrelevant anyways, my main argument is that a whole game predicated on black box weights and biases will almost never produce the same result twice this isn't about random elements, generation, everything that's been done, use AI to enhance that if you really want, although there are cheaper ways to achieve what you want nine times out of ten this is about the fact that any games purely predicated on AI in their entire loop, like this "doom" example, would be non-deterministic enough to (almost) never produce the same "game" for two different people, limiting it's actual application for anything story-driven for example is this entire demo and the fact that they were able to do this cool? yeah, kinda is this actually ever going to be useful on this level? doubtful
John Carmack asked on Twitter if anyone had actually got this running on a consumer GPU. Has anyone actually got this running at home? It should be theoretically possible, but so far, I've seen no evidence.
@@SierraSierraFoxtrot Yeah, but everyone here seems to understand what he means, and that is the point of language after all. So, I suppose it's fine... We're just not his target audience apparently.
@@flameofthephoenix8395 you have to remember the Gell-Mann effect. When I see sych a mistake I assume there will be other mistakes that I won't catch. I don't need junk information.
I wonder why no one mentions Game Gan and Gan Theft Auto that did basically the same (frame prediction with input and previous frames conditionning). This was like 4 years ago!
Yeah, the attention the paper has been getting feals like just clickbait. It’s just a fairly normal RL system that they are letting you use the controls while visualizing internal state representation and prediction instead of running the game. Which is kind of cool for interpratibily and figuring out “why is my agent going insane”. But definitely not an engine…
Predicting an extra frame here and there between conventional rendering with conventional game state and everything else is a vastly different challenge than predicting the next everything entirely with a single neural network.
@@georgesmith4768 Uhh, previous RL internal world states are not only vastly lower-quality, but also not anywhere close to as stable over long periods of time. And I'm not sure what definition of "game engine" you're using if a program that runs a game doesn't meet it.
@@somdudewillson gan theft auto does not use any game state, it is entirely simulated inside the neural network. The image quality is not the best, but GTA5 graphics are also vastly more complex than Doom's
Having an engine that generates an entire game automatically would be impractical, to be honest. There needs to be human oversight in the process. While using AI to simulate physics, provide instructions for NPC behavior, or create area descriptions could be useful and make development easier, relying on AI to fully design a game is asking for trouble. People who think it's a great idea often overlook basic principles of human creativity and integrity.
it just also seems really really inefficient to use AI to try and model these on the fly when we have decades of research on how to, you know, just use an algorithm to do it rather than hope the AI wont hallucinate something wrong, and take 1000x the computing power to do so.
@nimrag659 Absolutely agree. AI should be used in areas like NPC development, dynamic systems, and interaction detection in game engines, where it can truly enhance the experience. It’s important to see AI as a tool to assist, not as a one-click solution for everything. We still need to rely on proven algorithms for specific tasks, and AI should complement that, not replace it.
Layman question: I know neural networks don’t *feel* deterministic, but aren’t they? Don’t they give the same output given the same input/seed? Genuinely curious if you use the term as a reference to the black box nature, or if I have more basics to learn lol
Yes, indeed neural networks are 100% deterministic. The only thing that makes them feel random is the different behaviour they display when changing the seed values, the weights, and the temperature. Since there are so many parameters that one can tweak, it can lead to having the feeling that it is not deterministic because the same inputs will produce different outputs, but if you pick any neural network, and feed it the same inputs, including the temperature and seed, which btw they are also inputs, then you will get the exact same result. It is this change in behaviour over time that makes it seem like it is non deterministic because it does have an element of randomness to it, but it's still pseudo randomness.
I think it's worth pointing out that in this particular case of running doom on ai, you get a bit of the butterfly effect. Since the user is controlling it in real-time, they're not gonna be able to reproduce the exact same inputs every time, so it becomes essentially non-deterministic. But, not truly non-deterministic, for all the reasons tiranito mentioned. I'm not sure if this butterfly effect was what the video was referring to, but it's the closest thing to non-determinism happening in this ai that I'm aware of.
@@tiranito2834 not true, some transfers may include random components in their kernels which influence the result ever so slightly with the same weights used. It really depends on network topology if its deterministic
If you don't add randomness many AI models are prone to get stuck repeating patterns. If the AI is 51% confident that next to a floor there should be a floor you'd risk having a never ending plane of floor for an eternity.
So all you need to do if you are trapped in a nondeterministic nightmare is find something that looks similar to the exit, look really close at it and wait 3 seconds to find your way out
I think people are looking at this from the wrong angle. This won't be about making an entire game engine where you can generate games with text to speech. This will be more about creating a game with incredible unfeasible graphics with amazing lighting and whatnot, and then deploying it to machines that can use AI hardware to play the game rather than a traditional GPU to render stuff. At least, that's how I see it. You make a game that runs on a supercomputer then train an AI model from it. This is how you get to movie level graphics in games without having to brute force render physics and lighting calculations. Or you could have a traditional game engine running under the hood that handles logic and whatnot, but it's just data, and that data is used by AI to create what you "see" with a pure AI rendering pipeline. Very exciting stuff.
Yeah though the latter approach doesn’t necessarily need this kind of model. Something like Runway’s newer video to video model can do improvement of more limited graphics and I’m sure more can be done in that direction.
I am still waiting when anybody can do a visually and physically correct behaving mudracing simulator (a sort of rally sport of mostertrucks sploshing through pits of mud and water). Ar least in driving simulator games it is still the final frontier of things where PC and game consoles fail.
5:52 Actually, there's TONS of data that already has the inputs built in. Gameplay for DOOM is usually recorded as a demo (a series of inputs tied to the framerate). Gathering your training data is no more difficult than recording video to pair with the demo file.
I think its important to point out that you have to make the game in its entirety BEFORE you can train an ai on it to make the inferior AI version where enemies pop in and out of existance (while burning up ur gpu) call me crazy but this isnt exactly useful.
@@BitTheByte I'm saying this to all the people who think THIS is how we will create games in the future. It's important to note that this MIMICKS games, it does not create new ones. Almost every comment in this comment section is about how this is the future of gaming.
What I love about this project is the potential for AI to "imagine" a future scenario and how it's actions would effect it. Self driving is an example where I see it being pretty cool to se some work here to me. Predict the next three seconds of driving and a few different courses of actions, feed that back into the real time model to adjust based on what COULD happen soon. All assuming you can predict faster than real time. if it takes 6 seconds it's maybe just using data generation.
Honestly a very obvious use case for this would be DLSS-like frame generation, but with the ability to move the camera around and it not being locked strictly to source framerate and being able to generate a ton of intermediate frames.
We could use AI as a post-process shader that has access to Vertex data from the scene to the promp of the AI to generate, lets say, the flames and ray-trace of a fire-pit, since those are very dynamic in reality.
Ugh, fake news. This isn't Doom, and it's also only misunderstood to be Doom if shown in extremely brief snippets. It's just generating frames that look like Doom, but it doesn't behave like Doom.
I can imagine this kind of thing being used in conjunction with a deterministic system to generate infinite experiences, like imagine Skyrim but NPCs don't repeat conversations every time you talk to them or quest objects are different every time
Processing power increases to such insane rates, that having such a brute force solution might be viable before we figure out how to use AI to make deterministic game engine. What we have now with various AI, could likely be done with 100 time less processing power, if we had few decades of research done on it. But processing power and AI improvements vastly outpace the speed at which we can do research now, so a brute force approach like in here is not a dead end.
I like to think this is more of an addon much like how upscaling is currently. For example, this could look at the game state both in visual (for upscaling) but also game state in terms of code and from there, make frames that would better fit a game style while still having the game engine/code as the back bone. With that, you could get all the benefits of a fake frame + real frames without out the negatives like input latency that a fake frame couldnt deal with. In that case, your AI would only need to understand the code, what is current being displayed, and what the ai + code thinks will happen next. More or less it becomes a fancy branch predictor that could hold a few states until they're ready to be used. Once on refresh of the next frame, it looks at current input from user or game and then picks the best "what if" branch that can help speed stuff up for a frame as the real frame gets made in the background.
As for what you’ve said at 5:50 - we have thousands upon thousands of recorded demos that we can use for that, and thanks to its format it’s 100% lossless
It blew me away when Ha and Schmidhuber published the first World Models in 2018. You could even play them in the browser. (Of course it was doom too).
this is literally just like the ai is just *thinking* about a game of doom being played. if you just think about doom in action, you're doing the same thing, you're guessing what's gonna happen based on your knowledge as a player. hell, you do it subconsciously while playing any game. but the fact that a computer is capable of that is mind melting.
Wouldn't a non deterministic game engine make the base for a true open world game where you can literally do whatever without restrictions? Like blow up a building, and the AI can just generate the collapsing pieces based on physics.
I doubt if AI-generated games will ever get as good as hand-crafted games but being able to play and explore AI-generated games honestly does sound extremely cool and I'm excited to see more of it.
i genuinely cannot think of a reason why you would want a game developed in a non deterministic way. Even if you ignore the fact that it just straight up makes shit up and forgets stuff you were just looking at, it is unbelievably inefficent, and requires an insane amount of computing power to approximate something that could be done lightning fast for essentially free and works every time by just coding the darn thing
I was gonna say that TASing a fully AI generated game would become impossible, but then I remembered that TASers have dealt with RNG for as long as it has existed. So as long as they can get the same inputs for the AI, it should be possible for a TASer to play through an AI generated game, and even manipulate the values so that they can beat the game much quicker.
This is a really neat tech demo showing a novel implementation of realtime inference, but I wonder if it would be more effective to have an AI generating memory values in realtime than just guessing the next frame of visuals with no real concept of game state. Kind of like that somewhat famous bot that monitored all the memory values in an emulator and could play Super Mario Bros. surprisingly well after enough reinforcement learning.
There's no reason we can't run a "game" of this kind with real life graphics if we produce enough training data (gopro footage) with specific on screen patterns (eg. lives and # of enemies remaining or cash)
In theory, if you could encode the game’s state outside of the game’s frame as binary data (e.g., the border of the game representing the long term state of the game), then there could be less errors when it comes to how the states are estimated by the model.
Next step is making an AI imagine an AAA graphics, add some frames, and you are where computer game graphics will be in near future. It is inevitable, video game graphics will be eventually mostly generated or imagined by an AI.
I once had a dream I was playing the original Super Mario for the NES. Not on a computer or tv or anything, the dream itself was the medium on which it was running. It wasn’t 3d or anything, just the same side view 2D of the actual game. The level layout wasn’ accurate, but the spritework was close. This would’ve been like a decade ago by now, so I don’t really remember it all that well, but the sheer oddity of this experience compared to other dreams made it stick with me. My brain, my *neural net* was running/simulating Mario gameplay.
There has to be a decent way to combine the two methods. Even we humans have 'hard drives', we are not just a stream of random conciousness but our conciousness is molded by our memories (hard drive)
I was about to ask about Nvidia & Tencent research, there's other Individuals who tried to create this neural game engine, but claiming "the first" is something need to be evaluated again.
An AGI would probably understand it has to build the level in advance and incorporate level design. As we would if we were tasked to do it. We would use pen and paper to offload, AGI could use memory by changing its weights. So, that's feasible, we are just not there yet. We know about the passage of time, because we have internal clocks. Even when the wall clock is not available, we do realize the passage of time. Maybe that's one evolution for AI.
a fully AI generated VR version of Squad sounds like FIRE. Wait like, 20-30 years, once these technologies have been better figured out. The Playstation 12 bout to be lit
Would be interesting to find a mathematical way to get the AI 'back on track' when it hallucinates too much. Then, when you train it on more diverse data sets, you basically get "Dreaming - The Game".
If you wanted to use an algorithm to make a random Doom experience, the Obsidian Level Generator is already a thing and SLIGE existed in some form since 1998.
This is really cool if your in to image generation models, but doesn’t seem like it comes much closer to being a “game engine”. A conversational interface for making a game is almost certainly possible, but it would need to be doing a lot more than guessing at an appropriate next frame based on past data (since, that would require the game to already exist). The more likely path there is probably some combination of llms and some of the 3d asset generation models and some interesting prompt engineering.
training the AI purely off of frames i think is not sustainable at all. but i think what could work is if the AI was trained off of additional, raw information not encoded in the frame. most of this information can probably even be static, like which enemies there are, how many of them there are, their starting positions, the player's starting positions, positions of items on the ground, doors, what keys are mapped to what doors, and so on. note that im not strictly mentioning things like weapons the player is using, the damage those weapons do, weapon switching time, player health and armor, ammo, and so on. while some of that may be static (such as weapon switching time, starting positions, weapon damage), i think _these_ are probably things the AI can reasonably infer from the frame data and it seems to already be doing a good job on that.
I'm surprised you didn't mention multiplayer when talking about the downsides. I think a lot of people overlook the fact that determinism is almost a must-have for anything networked, since otherwise any clientside prediction becomes impossible. The only real applications for NNs I can see in gamedev would be for upscaling (as we already see), framegen (as we also already see), procedural animations (which we also already see in some very limited examples), and physics (which we see in use for softbodies/cloth in some games/engines already, see Unity 6 cloth physics as an example). Actual game logic? No.
It is at least promising for further offloading hardware rendering aspects to AI models, a power density shortcut. A full model of a game should also be lighter storage wise than many AAA titles, but I think pursuing this avenue does represent some particularly interesting avenues for new game engine architectures. Such a model would require offloading the training parts to a training farm off the "golden" game, which does also bring up unusual aspects of it being almost like DRM. By that I mean now it'll be a new problem all together to recover or data mine from such a game, since the actual original code can't even run on your computer anymore.
I think the way forward will be using multiple AI engines in the same "product". With director AI trained on how the final experience should look like, and trained to look for inconsistencies in image, text and sound. And being used to train other more specific AI's for image, text and sound, constantly giving each other prompts and reacting to player input. I think with such solution you eventually would only need to train the director AI for each game, and have a reliable structure for AI's to cooperate like that, feeding each other off a given rough script with some time stamps for example. Maybe best if AI's would be trained on custom image and text base and that particular training material being given more weight. But yea I think in far future it could work. You buy game director UI instead of game with all presets and base mechanics hard-coded, and a rough storyline which indicates what happens when in game time, characteristics of locations ect. And everything is generated, you can go anywhere, do anything, talk to anyone. Story and scenery might get a bit autistic as AI fills in the blanks, but who cares... you buy DLC pieces with new "pre-made" instructions for director, or you can write them yourself if you want, instructions being open to edit, let's say - after finishing the base game first. Hey, you wanted to see how the story would be if that evil female dragon that you normally have to kill fell in love with you instead - go ahead. I think it would still need to have hard coded instructions and some script/description to follow and a way to communicate with AI's responsible for text and image. Yeah, maybe in next 40 years. Maybe I will live long enough to see it one day...
imagine single player games with non deterministic storylines where each player has a personally crafter story and ending, with each new game being unique!
I wonder if it could get better long-term robustness if instead of just training of predicting the next frame based on a noisier version of a correct previous frame, it would predict based on its own output like it would in the real life application? You would start by only letting it drift from a few frames ago and then gradually increase the time since the last known valid frame as it gets better at remaining robust against its own drift. The maximum time probably won't be that long, since a chaotic system means that even an almost perfect simulation would start drifting pretty quickly. In order to compensate for this increased difficulty, it might also be worth to add a bit of memory that it can write to in order to remember data outside of the pixel data, though the question there is what to set that data to when we start from training data instead of from AI output? Maybe you could run just the part of the network that generates the extra memory first? You would still only give it the same 3 seconds (or shorter) of history in this case, even though the time since the last known valid frame may be more than 3 seconds.
Basically people have to understand that the impressive part doesn't come from when it's a copy machine, but when the new stuff it makes actually is useful, which is not the case for that DOOM model. One good way of understanding the generative AI wave that I figured out recently is roughly comparing it to a lossy zip compression, your prompt is mapped to a "hash key" (embedding to be more true to the theory) that points to what to decompress, but since it was a lossy compression, the result will not be 1:1 with the original data, but a weird blur that might happen to be good new stuff and that's the part we should aim to improve.
There actually is data on how players play Doom. It's perhaps the first game to allow players to share gameplay videos by storing player input data. I don't know how much of this has been preserved, but there should at least be some speedrun and other interesting lmps that could be used for training purposes.
Well this is at least getting close to Ross's game dungeon's dream for game preservation, he dreamed of one day being able to feed in game footage of any existing game and having it spit out a replica that didn't rely on old hardware or a companies remote server that will eventually shutdown. Though more ideal would be it spitting out the source code to a deterministic engine written in some like C that could be modified and recompiled on other hardware if needed.
There would be a necessary middle step in game development where we make single game capable engines, but we do it on a general model. That way, the game is actually a file of weights. This would make more commercial sense than a truly general AI game engine.
Are neural nets really non deterministic? I don't think so, if you train 2 neural nets with the same inputs and use a PRNG with the same seed for the magic sauce, it will give the exact same result, it's just a computer program like any other
I think the "non deterministic" was more about game behavior than neural net output - so running away from an enemy for a few seconds might cause them to disappear, or move to a different spot, or still be there when you turn around.
@@travissmith5994 maybe... but "deterministic" is a well defined term, it just means that the same inputs will give the same output and everything a computer does is deterministic Neural nets (an other things like video encoders and CSPRNGs) only seem non deterministic when you are dealing with noisy real world data
@@olhoTron not all programs are deterministic. There are sources of true randomness connected to your computer. It starts with the input devices, goes on with the time of day and finally cryptographic algorithms use special hardware RNGs based on physical devices like crystal oscillators, integrated into the hardware security module. In the case of this AI model the user input is probably enough to cause it to show chaotic behaviour (that is, small changes in the input give big changes in the output)
@@MrFaerine no, all programs *are* deterministic, the change in behavior comes from the inputs taken from the real world, but if for some magic reason the noise captured by the hardware RNG repeats, the values it will give will also repeat
There are TAS of videogames that would be perfect, not only do they record button inputs which you can relate to the game content, it records the very best of it.
I doubt that these things will ever practically contribute to streamlining workload. As someone who trained old school natural network to predict water quality, I can say that AI right now is a worse search engine, and it will stay that way until some new method emerged. But it is great how it is possible. The product quality seems to be exaggerated, but they did train AI to ignore or correct its own mistake to crash the game in 3 minutes instead of 3 seconds. It is like realizing laying down train track as the train is running is now possible, even if it is inferior to doing it the own fashion way in anyway.
![YoungBoy Never Broke Again - Missing Everything [Official Video]](http://i.ytimg.com/vi/dTYeMZGzOzw/mqdefault.jpg)
check out HubSpot's Free AI Task Delegation Playbook here! clickhubspot.com/s6ef
As for the new paper gamegen-o I've shown at 12:00, not actual interactive demo is shown. Looks like the demo is heavily cherry-picked. So yes, gamengen is by far the closest example to an actual AI game engine.
edit: Looks like they deleted their gamegen-o project, with no reason specified. Odd for sure.
Gamegen-o looks like another "reflection" kind of move. More scammy people trying to ride the wave of hype.
the AI model is still "deterministic" its just represented in latent neural space than specific measurements, and can behave unpredictably from our perspective
if we think about it, we can run doom with our thought if we think about it....
NPCs can't
If we think about it
@@ghaith2580 If we think about it
You are genius
@@Getenari and then we can run software by making notepad for mac !
3 seconds of memory? So if you hug a wall for 3 seconds and turn around, is it still the same level?
I really wanna play this to find out!
I know from the footage that it forgets enemies you killed in a matter of seconds, same as well with barrels and pickups.. changes the gibs sprite even while in full view,
Yeah when it showed the blue door I was expecting the player to walk right up to it so you can't see the blue border, wait a few seconds, and open it.
There is no level, it just tries to guess where the walls are supposed to be lol
The game is constantly being procedurally generated, which could be a novel concept
@@stephen-torrenceyou can't play it. It's a video.
It is a brilliant demonstration of fuzzy memory and human recall rather than saying "we can make vidyajamez on AI. No more software engineers!"
So would fuzzy memory for Generative gaming be useful for logic that self reinforces within a certain time window?
Why are chronically online people so opposed to AI unless it's for gooning to character ai
Software engineers are needed to... make the ai... you know... xD
@@px8 At least some people recognize the boring and monotonous aspects of software work that AI can improve such as boilerplate code generation, faster bug fixes, and documentation rewording/simplification.
Congrats for seeing the bright side. I'm glad this will actually make more engineers to debug the prompts in addition to the codebase :)
@@SimGunther Lol ty, sorry for (kinda) insulting a little you at first :(
@@px8 'chronically online' has really become completely meaningless as a phrase now huh
I had a lucid dream once where I was noclipping through the "map" of a dream earlier from that night. There were even missing areas over the hills and stuff that weren't visible from where I had been in the previous dream. It was all as smooth as an actual video game. Indistinguishable experience.
You beat the dream and unlocked Debug Mode
I had a similar dream, but instead of no clipping, I just climbed a rock face up to where you aren't supposed to go and could see outside of the map. It was a lot like that part in Half Life 2 speedruns where they get on top of that narrow valley. Except it was more of a desert environment at night.
Then when I was up there, for some reason, Frank Herbert (author of Dune) was there and I had a deep conversation with him. One of the coolest dreams I've ever had.
@@JustinBA007 Badass, man! Apparently very common in lucid dreaming to find wise characters hanging out in the subconscious "behind the scenes." I have a friend who met Will Farrell as a game-show host congratulating him when he stepped backward through a dream wall.
It was not a dream you just go in the backrooms
I have something similar... Dreams take place within some strange huge island, within which 90% of my dreams take place
I played so much Transport Tycoon that even when I closed my eyes, I could still see the game.
The local authority refuses to allow this
Happens sometimes
I believe this is called Tetris syndrome
I'm playing Ultrakill like that when i don't have access to my phone and pc
same thing happened to me with Helldivers 2, i kept hallucinating bots when i closed my eyes
ai art started as image upscalers
llms started as autocomplete
ai games are starting with whatever tf this is
There were earlier AI writers & image generators, which used different methods, often mixing & matching pieces if data based on rules set up by their programmers.
It's basically just a simulated neural network daydreaming a run of Doom, when you think about it...
Llms are still autocomplete. They just got really, really, good at it
Yeah I don't thing this week replace game engines, but I'm eager to play actual games generated like this.
Yeah but it won't see the same change.
There's mathematically a point after which you can't optimise stuff. A point where you are dealing with raw information in pure form.
Also, mores law is kinda dead. We won't have todays server room fit in a computer, for hundreeds of years, as it would need a complete change in computer.
QC won't solve it. They are not faster to do serial task. They just allow some problem that are //able to be shrunk to fewer steps needed to do it.
Okay. So now we almost had anything. Can we now run doom on brains?
Lmao it's already under development checked out the thought emporium
@@ludologianI think he is trying to play DOOM with grown rat neurons, but I wouldn't be surprised if he tries to simulate the game on them as well
@@fluffsquirrel thx for correcting me, yep but still amazing that he training rat brain cells to play/ control doom
I'm imagining playing doom right now
@@ludologian Np, and I agree. I can't wait to see the results!
So, in order to have an IA simulate Doom, you need the following :
1) A functional Doom copy
2) Have it played for 10,000h while recording inputs
3) Have your IA try to simulate Doom for 10,000h while simultaneously feeding it the result of actually playing 10,000 identical hours of normal Doom on a regular computer
Sounds like MacGyver quickly assembling a revolver from a shoelace, a bubblegum wrap, and a revolver.
Wonder if this would work with demo recordings of, say, glitchless speedruns and such
These "generative AI" systems aren't intelligent in the slightest, they just regurgitate slop until they randomly get it right. Practically the definition of monkeys at typewriters.
I don't think that's the point
Very interesting! I imagine they were trying to make a point by not providing any game state, but you might achieve better results if you provided some basic game state (eg location history, health, time in level) as inputs and outputs.
Imagine what a mini map could do
@float32 that could be quite helpful. In general my concern is that if you went AFK for a minute staring at a wall it could lose all context. You could have a low res "rear view" camera which is used to avoid that problem, and not displayed onscreen during play.
I'd prefer that the model gets to determine what gets stored in the game state
@Survivalist_Redo That's fine, but two problems. 1. It can't reason about information it doesn't have within its window. 2. So far, the strong suit of deep learning is not long term temporal consistency. Don't rely on a feature that is a weakness of your technology choice.
@@davidcummins8125The game state doesn't have to be stored in the model itself. There have been successful attempts at giving AI models something akin to a "notebook", where it can "write down" information and later recall it.
I guess it's important here that we understand that GameNGen isn't modeling the functionality of a game; it's modeling the generic game experience - i.e. what it looks and feels like to play; sort of like having a dream. Actually, likening AI to a dream is pretty good analogy to how the results manifest: cause and effect do not correlate properly and images only moderately represent what objects or events really mean.
Neural networks are inspired by our understanding of how biological nervous systems and brains work in the first place, hence the name "neural network". So yes, this is essentially a model that is lucid dreaming gameplay of Doom.
But that only works well when you're asleep and are complaint to cause and effect not working the way you're used to.
This AI is literally just a framegen with inputs as context, it'a not really capable of anything, and can not be added to. Super inflexible. A cute tech demo, but nothing more. Calling it a "game engine" is just bs to build hype over nothing.
Thanks for helping to dispell the misinformation regarding this cool project.
I think any attempts on world simulation with AI would look like human dreams: inconsistent due to short memory and hallucinating crazy things. It might actually be interesting to see, but it will be not comparable with deterministic physical worls simulations in our games.
Which is why its only an expirement, and not any "groundbreaking discovery". We can do it, its pretty trippy and interesting, but not much else, this approach is simply not worth it.
@@alexturnbackthearmy1907 I do think it could be useful for giving robots "imagination" in the sense of the ability to predict what its actions will do to the world around it
I could see later Rougelite games taking advantage of this to make wildly different runs
The movie "The City of Lost Children" made in 1995 tried hard to simulate in CGI (likely a kind of photo distortion) what a morphing and mutating dream world looks like. Now we begin to have computers do dreams by their own.
It has to start somewhere.
Honestly I can run any game I play often in my head I'm surprised we didn't have this sooner
We didn't because it's completely useless.
@@endlessstrata6988 not to me
@@David-lp3qy are you using it?
@@user-sl6gn1ss8p maybe
Isn't the biggest problem less the GPU power and Ram but the training data? Imagine the huge amount of complexity to alone get that data for most games we have, and to then recreate / retrain the model each time you want a new type of game genre. You would still have to completly code a game first, create textures, logic etc. yourself.. and that with tons of games of the same genre.. just for the AI Game Engine to be able to train what this new genre is about and what you want from it. If you need to do that, you could just code the games yourself instead and save a lot of ressources you use to code the game in first place & then retrain the whole network etc.. i don't think it's really possible to make a good game creation engine from that concept alone if we don't add a aspect to the AI which actually is "aware" and understands what you ask it to do.
I guess the idea would be to train it on thousands of video games together with descriptions of those games and then hope that it would gain the ability to extrapolate to video games from novel descriptions. but it would require a ton of manual labor to add support for each of those games, and even then, thousands of games is not that big of a number in machine learning context, so it might still be way too little
@@asdfghyter An actually realistic AI solution to game dev would rather be to have the AI generate the code needed to have a fully functional game engine, and then get it to make a bunch of graphics for the engine afterwards. At least these problems have generalised AI built for them (even if the code you get can be somewhat sketchy at times)
@@apotato4873 How to let an AI make doom
1. Train an llm on early 90's engines (Doom, Build)
2. Train an image generator on sprites and textures from some WAD files
3. Train some other model on map data (these are essentially graphs)
This technology could probably be tweaked to work a lot more effectively by buttressing the AI some deterministic elements.The domain-specific generation would probably be much more stable if had a proverbial ariadne thread to refer back to.
shut. up.
EVERY COPY OF DOOM IS PERSONALIZED!
Doom running on ai is crazy
The peak of over engineering
We will probably run Doom on a single atom, and we will find out that atom is a whole universe...
Wait until someone runs Doom on a computer built in minecraft...
Doom being "generated" by ai.
omg it's that guy from the gdmod server
I love how shooting demons in this version instead of turning them into pixels, it is turning them into gloobs
I'm sorry but determinism is 100% needed for a game to work
Any game using randomness would like to disagree:
Most board games
Monopoly
Card games
Games involving dice
Rogue-like video games
Social games, like Among Us
@@Hlebuw3kWhile I don't necessarily agree with the comment you're responding to, this is a still a really weak argument. Relying on some randomness as part of a game isn't the same thing as being completely nondeterministic. All of those games still have very well-defined rules that can't just change on the fly by random chance; only within those rules there's specific points of randomness.
More comparable to this would be a version of monopoly where the order and ownership of the properties randomly change or a version of Among Us where the vents might just make you glitch out of the map on occasion, lol.
@@gayusschwulius8490 in the future I'm sure deterministic features will be possible to implement. things that will break a game if not anchored in only one way. hey AI make a racing game where only first place wins and gets a gold medal, any other racer will not be first, but they can compete for first.
the biggest challenge is interactability and continuity honestly.
@@Airbigbawls indeed, that's why I don't agree with the original comment saying determinism is 100 % necessary for a game. With a sufficiently advanced AI model that is capable of maintaining consistent gameplay, a nondeterministic game is perfectly possible and might have some massive advantages. Technology still needs to improve a lot for that to happen, though, and I also think that it'd probably make more sense for the rendering of the 3D scene itself being done by a deterministic engine and only the gameplay and 3D scene generation being handled by the AI.
Not only that, but neural networks ARE supposed to be deterministic anyway.
Also randomnes in videogames and ANY software running in a computer will never be trully random, everything is pseudo-random
everyone gangsta until game mastermind hallucinates you into zero hp and negative ammo
P(DOOM) is 100%
Can't believe we got doom on neural network before GTA 6
We got gta 5 on Gan before doom on nural network
If the AI sees what we are seeing, what would it do if you move backwards?
I don't think anyone would want an AI only game. Not only would you most likely never experience the same game twice, your experience would be wildly different from someone else's, which in theory sounds cool, but in practice, not really. It's like if I watched the Two Towers and the kill count of Legolas and Gimli changed each time I watched it.
Procedurally generated games have been a popular thing for a long time
@@brandongregori995 Still governed by a defined set of deterministic algorithms that will always produce the same output given the same input as opposed to a black box that’s the complete opposite
My argument is the lack of application for this for anything other than (pseudo-)randomness, which is already covered by more efficient means anyways
Short sided and misses the point
@@chadyways8750 All algorithms are deterministic. LLMs are deterministic, but they are configured to use CPU and GPU temperatures to facilitate random outcomes. You can do this with procedural generation too. There is nothing stopping a seed from having current temps, datetime, and other things appended to it to give you seemingly non-deterministic outcomes.
@@brandongregori995 determinism is largely irrelevant anyways, my main argument is that a whole game predicated on black box weights and biases will almost never produce the same result twice
this isn't about random elements, generation, everything that's been done, use AI to enhance that if you really want, although there are cheaper ways to achieve what you want nine times out of ten
this is about the fact that any games purely predicated on AI in their entire loop, like this "doom" example, would be non-deterministic enough to (almost) never produce the same "game" for two different people, limiting it's actual application for anything story-driven for example
is this entire demo and the fact that they were able to do this cool? yeah, kinda
is this actually ever going to be useful on this level? doubtful
John Carmack asked on Twitter if anyone had actually got this running on a consumer GPU. Has anyone actually got this running at home? It should be theoretically possible, but so far, I've seen no evidence.
I think what's preventing that is they haven't apparently published the model weights or code.
@@tad2021yeah. We don’t event know if there aren’t „ifs” and other normal code in this fever dream clickbait AI project.
In pretty sure it runs on a warehouse full of GPUs.
@@JohanFruend If they parallelized SD to run across multiple GPUs for faster generation then that would have been a paper on its own.
I wonder if it’s storing game state in the pixels of the frame or the hud in a way that isn’t visible to the eye but is still viable for game state
0:48 That's still deterministic...
Yes, really bad mistake to make in the first minute.
@@SierraSierraFoxtrot Yeah, but everyone here seems to understand what he means, and that is the point of language after all. So, I suppose it's fine... We're just not his target audience apparently.
@@flameofthephoenix8395 i took the hint and stopped watching.
@@SierraSierraFoxtrot Good idea, I'm a sucker though and watched it anyway.
@@flameofthephoenix8395 you have to remember the Gell-Mann effect.
When I see sych a mistake I assume there will be other mistakes that I won't catch.
I don't need junk information.
I wonder why no one mentions Game Gan and Gan Theft Auto that did basically the same (frame prediction with input and previous frames conditionning). This was like 4 years ago!
Yeah, the attention the paper has been getting feals like just clickbait. It’s just a fairly normal RL system that they are letting you use the controls while visualizing internal state representation and prediction instead of running the game.
Which is kind of cool for interpratibily and figuring out “why is my agent going insane”.
But definitely not an engine…
Yeah but 4 years ago AI wasn't the buzzword so it didn't matter then 😅
Predicting an extra frame here and there between conventional rendering with conventional game state and everything else is a vastly different challenge than predicting the next everything entirely with a single neural network.
@@georgesmith4768 Uhh, previous RL internal world states are not only vastly lower-quality, but also not anywhere close to as stable over long periods of time.
And I'm not sure what definition of "game engine" you're using if a program that runs a game doesn't meet it.
@@somdudewillson gan theft auto does not use any game state, it is entirely simulated inside the neural network. The image quality is not the best, but GTA5 graphics are also vastly more complex than Doom's
Randomly remixing every existing game in the world to make a "unique experience" feel like a scam
Having an engine that generates an entire game automatically would be impractical, to be honest. There needs to be human oversight in the process. While using AI to simulate physics, provide instructions for NPC behavior, or create area descriptions could be useful and make development easier, relying on AI to fully design a game is asking for trouble. People who think it's a great idea often overlook basic principles of human creativity and integrity.
Only dumb ignorants think this is „it’s over for game developers”.
it just also seems really really inefficient to use AI to try and model these on the fly when we have decades of research on how to, you know, just use an algorithm to do it rather than hope the AI wont hallucinate something wrong, and take 1000x the computing power to do so.
@nimrag659 Absolutely agree. AI should be used in areas like NPC development, dynamic systems, and interaction detection in game engines, where it can truly enhance the experience. It’s important to see AI as a tool to assist, not as a one-click solution for everything. We still need to rely on proven algorithms for specific tasks, and AI should complement that, not replace it.
I mean, it's not even an engine, deterministic or not. We're not any closer to it because of this project.
Layman question: I know neural networks don’t *feel* deterministic, but aren’t they? Don’t they give the same output given the same input/seed? Genuinely curious if you use the term as a reference to the black box nature, or if I have more basics to learn lol
Yes, indeed neural networks are 100% deterministic. The only thing that makes them feel random is the different behaviour they display when changing the seed values, the weights, and the temperature. Since there are so many parameters that one can tweak, it can lead to having the feeling that it is not deterministic because the same inputs will produce different outputs, but if you pick any neural network, and feed it the same inputs, including the temperature and seed, which btw they are also inputs, then you will get the exact same result.
It is this change in behaviour over time that makes it seem like it is non deterministic because it does have an element of randomness to it, but it's still pseudo randomness.
I think it's worth pointing out that in this particular case of running doom on ai, you get a bit of the butterfly effect. Since the user is controlling it in real-time, they're not gonna be able to reproduce the exact same inputs every time, so it becomes essentially non-deterministic. But, not truly non-deterministic, for all the reasons tiranito mentioned. I'm not sure if this butterfly effect was what the video was referring to, but it's the closest thing to non-determinism happening in this ai that I'm aware of.
@@tiranito2834 not true, some transfers may include random components in their kernels which influence the result ever so slightly with the same weights used. It really depends on network topology if its deterministic
@@unityasteroids1562 the "random components" are obviously pseudo-random, so same seed = same result.
If you don't add randomness many AI models are prone to get stuck repeating patterns. If the AI is 51% confident that next to a floor there should be a floor you'd risk having a never ending plane of floor for an eternity.
So all you need to do if you are trapped in a nondeterministic nightmare is find something that looks similar to the exit, look really close at it and wait 3 seconds to find your way out
So the eternal question is asked again in our age....will it run doom?
I think people are looking at this from the wrong angle. This won't be about making an entire game engine where you can generate games with text to speech. This will be more about creating a game with incredible unfeasible graphics with amazing lighting and whatnot, and then deploying it to machines that can use AI hardware to play the game rather than a traditional GPU to render stuff. At least, that's how I see it. You make a game that runs on a supercomputer then train an AI model from it. This is how you get to movie level graphics in games without having to brute force render physics and lighting calculations. Or you could have a traditional game engine running under the hood that handles logic and whatnot, but it's just data, and that data is used by AI to create what you "see" with a pure AI rendering pipeline. Very exciting stuff.
Yeah though the latter approach doesn’t necessarily need this kind of model. Something like Runway’s newer video to video model can do improvement of more limited graphics and I’m sure more can be done in that direction.
But it will not be the same each time. Maybe that's a good thing
the end of polygons and textures
I am still waiting when anybody can do a visually and physically correct behaving mudracing simulator (a sort of rally sport of mostertrucks sploshing through pits of mud and water). Ar least in driving simulator games it is still the final frontier of things where PC and game consoles fail.
1:23 Litteraly a definition of Lucid Dreams
"Is it Doom?"
the new "can it run Doom"
Now it might be a fun idea to try to display doom using ai which would be distinctly slightly a little less stupid.
5:52 Actually, there's TONS of data that already has the inputs built in. Gameplay for DOOM is usually recorded as a demo (a series of inputs tied to the framerate).
Gathering your training data is no more difficult than recording video to pair with the demo file.
I think its important to point out that you have to make the game in its entirety BEFORE you can train an ai on it to make the inferior AI version where enemies pop in and out of existance (while burning up ur gpu)
call me crazy but this isnt exactly useful.
Not everything needs to have a use. They can just be cool
@@BitTheByte I'm saying this to all the people who think THIS is how we will create games in the future. It's important to note that this MIMICKS games, it does not create new ones. Almost every comment in this comment section is about how this is the future of gaming.
AI researchers just created a lucid dream of Doom 😂
What I love about this project is the potential for AI to "imagine" a future scenario and how it's actions would effect it. Self driving is an example where I see it being pretty cool to se some work here to me. Predict the next three seconds of driving and a few different courses of actions, feed that back into the real time model to adjust based on what COULD happen soon.
All assuming you can predict faster than real time. if it takes 6 seconds it's maybe just using data generation.
Honestly a very obvious use case for this would be DLSS-like frame generation, but with the ability to move the camera around and it not being locked strictly to source framerate and being able to generate a ton of intermediate frames.
You just described DLSS/FSR frame gen...
We could use AI as a post-process shader that has access to Vertex data from the scene to the promp of the AI to generate, lets say, the flames and ray-trace of a fire-pit, since those are very dynamic in reality.
That was really informative, great work at making it both in depth and yet being easily digestible.
In each frame feed it the game state + player input and have it return a new game state + image
Ugh, fake news. This isn't Doom, and it's also only misunderstood to be Doom if shown in extremely brief snippets. It's just generating frames that look like Doom, but it doesn't behave like Doom.
its mimick simulation that cannot be played properly . pointless
people see this wrong, the intention was never to make something that can build games or whtv
Congrats. You gave doom a dungeon master.
I can imagine this kind of thing being used in conjunction with a deterministic system to generate infinite experiences, like imagine Skyrim but NPCs don't repeat conversations every time you talk to them or quest objects are different every time
Me turning around before I get shot so that the enemy shrimply disappears from reality
The George Berkeley school of threat mitigation!
Processing power increases to such insane rates, that having such a brute force solution might be viable before we figure out how to use AI to make deterministic game engine. What we have now with various AI, could likely be done with 100 time less processing power, if we had few decades of research done on it. But processing power and AI improvements vastly outpace the speed at which we can do research now, so a brute force approach like in here is not a dead end.
I like to think this is more of an addon much like how upscaling is currently. For example, this could look at the game state both in visual (for upscaling) but also game state in terms of code and from there, make frames that would better fit a game style while still having the game engine/code as the back bone. With that, you could get all the benefits of a fake frame + real frames without out the negatives like input latency that a fake frame couldnt deal with. In that case, your AI would only need to understand the code, what is current being displayed, and what the ai + code thinks will happen next. More or less it becomes a fancy branch predictor that could hold a few states until they're ready to be used. Once on refresh of the next frame, it looks at current input from user or game and then picks the best "what if" branch that can help speed stuff up for a frame as the real frame gets made in the background.
As for what you’ve said at 5:50 - we have thousands upon thousands of recorded demos that we can use for that, and thanks to its format it’s 100% lossless
It blew me away when Ha and Schmidhuber published the first World Models in 2018. You could even play them in the browser. (Of course it was doom too).
this is literally just like the ai is just *thinking* about a game of doom being played. if you just think about doom in action, you're doing the same thing, you're guessing what's gonna happen based on your knowledge as a player. hell, you do it subconsciously while playing any game. but the fact that a computer is capable of that is mind melting.
Wouldn't a non deterministic game engine make the base for a true open world game where you can literally do whatever without restrictions? Like blow up a building, and the AI can just generate the collapsing pieces based on physics.
I doubt if AI-generated games will ever get as good as hand-crafted games but being able to play and explore AI-generated games honestly does sound extremely cool and I'm excited to see more of it.
i genuinely cannot think of a reason why you would want a game developed in a non deterministic way. Even if you ignore the fact that it just straight up makes shit up and forgets stuff you were just looking at, it is unbelievably inefficent, and requires an insane amount of computing power to approximate something that could be done lightning fast for essentially free and works every time by just coding the darn thing
I knew they would eventually create this, the potential is infinite
I was gonna say that TASing a fully AI generated game would become impossible, but then I remembered that TASers have dealt with RNG for as long as it has existed. So as long as they can get the same inputs for the AI, it should be possible for a TASer to play through an AI generated game, and even manipulate the values so that they can beat the game much quicker.
This is a really neat tech demo showing a novel implementation of realtime inference, but I wonder if it would be more effective to have an AI generating memory values in realtime than just guessing the next frame of visuals with no real concept of game state.
Kind of like that somewhat famous bot that monitored all the memory values in an emulator and could play Super Mario Bros. surprisingly well after enough reinforcement learning.
There's no reason we can't run a "game" of this kind with real life graphics if we produce enough training data (gopro footage) with specific on screen patterns (eg. lives and # of enemies remaining or cash)
I would like to play it, man finding the hallucinated parts would probably the most interesting aspect.
In theory, if you could encode the game’s state outside of the game’s frame as binary data (e.g., the border of the game representing the long term state of the game), then there could be less errors when it comes to how the states are estimated by the model.
Next step is making an AI imagine an AAA graphics, add some frames, and you are where computer game graphics will be in near future. It is inevitable, video game graphics will be eventually mostly generated or imagined by an AI.
I once had a dream I was playing the original Super Mario for the NES. Not on a computer or tv or anything, the dream itself was the medium on which it was running. It wasn’t 3d or anything, just the same side view 2D of the actual game. The level layout wasn’ accurate, but the spritework was close. This would’ve been like a decade ago by now, so I don’t really remember it all that well, but the sheer oddity of this experience compared to other dreams made it stick with me. My brain, my *neural net* was running/simulating Mario gameplay.
so doesnt this mean, we are playing doom from an AI's mind?... from an A 's imagination?
There has to be a decent way to combine the two methods. Even we humans have 'hard drives', we are not just a stream of random conciousness but our conciousness is molded by our memories (hard drive)
I was about to ask about Nvidia & Tencent research, there's other Individuals who tried to create this neural game engine, but claiming "the first" is something need to be evaluated again.
An AGI would probably understand it has to build the level in advance and incorporate level design. As we would if we were tasked to do it. We would use pen and paper to offload, AGI could use memory by changing its weights. So, that's feasible, we are just not there yet.
We know about the passage of time, because we have internal clocks. Even when the wall clock is not available, we do realize the passage of time. Maybe that's one evolution for AI.
This is the best video explaination on this topic!!
a fully AI generated VR version of Squad sounds like FIRE. Wait like, 20-30 years, once these technologies have been better figured out. The Playstation 12 bout to be lit
This is revolutionary, it’s just not going to change game dev world forever. Tho it’s really good experiment nonetheless
How many times more compute power do you need to run doom within AI than normal game?
You know that it takes a long time to create fully AI video on normal computer? Well...that thing is real time AI video.
Would be interesting to find a mathematical way to get the AI 'back on track' when it hallucinates too much. Then, when you train it on more diverse data sets, you basically get "Dreaming - The Game".
Great video. Disappointed by current tech hurdles i thought i will be able create my own gta 6
The output of a neural network is still deterministic, lol.
If you wanted to use an algorithm to make a random Doom experience, the Obsidian Level Generator is already a thing and SLIGE existed in some form since 1998.
This is really cool if your in to image generation models, but doesn’t seem like it comes much closer to being a “game engine”.
A conversational interface for making a game is almost certainly possible, but it would need to be doing a lot more than guessing at an appropriate next frame based on past data (since, that would require the game to already exist).
The more likely path there is probably some combination of llms and some of the 3d asset generation models and some interesting prompt engineering.
training the AI purely off of frames i think is not sustainable at all. but i think what could work is if the AI was trained off of additional, raw information not encoded in the frame. most of this information can probably even be static, like which enemies there are, how many of them there are, their starting positions, the player's starting positions, positions of items on the ground, doors, what keys are mapped to what doors, and so on. note that im not strictly mentioning things like weapons the player is using, the damage those weapons do, weapon switching time, player health and armor, ammo, and so on. while some of that may be static (such as weapon switching time, starting positions, weapon damage), i think _these_ are probably things the AI can reasonably infer from the frame data and it seems to already be doing a good job on that.
I'm surprised you didn't mention multiplayer when talking about the downsides. I think a lot of people overlook the fact that determinism is almost a must-have for anything networked, since otherwise any clientside prediction becomes impossible. The only real applications for NNs I can see in gamedev would be for upscaling (as we already see), framegen (as we also already see), procedural animations (which we also already see in some very limited examples), and physics (which we see in use for softbodies/cloth in some games/engines already, see Unity 6 cloth physics as an example).
Actual game logic? No.
It is at least promising for further offloading hardware rendering aspects to AI models, a power density shortcut. A full model of a game should also be lighter storage wise than many AAA titles, but I think pursuing this avenue does represent some particularly interesting avenues for new game engine architectures. Such a model would require offloading the training parts to a training farm off the "golden" game, which does also bring up unusual aspects of it being almost like DRM. By that I mean now it'll be a new problem all together to recover or data mine from such a game, since the actual original code can't even run on your computer anymore.
I think the way forward will be using multiple AI engines in the same "product". With director AI trained on how the final experience should look like, and trained to look for inconsistencies in image, text and sound. And being used to train other more specific AI's for image, text and sound, constantly giving each other prompts and reacting to player input.
I think with such solution you eventually would only need to train the director AI for each game, and have a reliable structure for AI's to cooperate like that, feeding each other off a given rough script with some time stamps for example.
Maybe best if AI's would be trained on custom image and text base and that particular training material being given more weight.
But yea I think in far future it could work. You buy game director UI instead of game with all presets and base mechanics hard-coded, and a rough storyline which indicates what happens when in game time, characteristics of locations ect. And everything is generated, you can go anywhere, do anything, talk to anyone. Story and scenery might get a bit autistic as AI fills in the blanks, but who cares... you buy DLC pieces with new "pre-made" instructions for director, or you can write them yourself if you want, instructions being open to edit, let's say - after finishing the base game first. Hey, you wanted to see how the story would be if that evil female dragon that you normally have to kill fell in love with you instead - go ahead.
I think it would still need to have hard coded instructions and some script/description to follow and a way to communicate with AI's responsible for text and image.
Yeah, maybe in next 40 years. Maybe I will live long enough to see it one day...
imagine single player games with non deterministic storylines where each player has a personally crafter story and ending, with each new game being unique!
TLDR: While it is POSSIBLE to use AI to generate a game, it is not VIABLE to do so, thanks to the all-important QA
I wonder if it could get better long-term robustness if instead of just training of predicting the next frame based on a noisier version of a correct previous frame, it would predict based on its own output like it would in the real life application? You would start by only letting it drift from a few frames ago and then gradually increase the time since the last known valid frame as it gets better at remaining robust against its own drift. The maximum time probably won't be that long, since a chaotic system means that even an almost perfect simulation would start drifting pretty quickly.
In order to compensate for this increased difficulty, it might also be worth to add a bit of memory that it can write to in order to remember data outside of the pixel data, though the question there is what to set that data to when we start from training data instead of from AI output? Maybe you could run just the part of the network that generates the extra memory first?
You would still only give it the same 3 seconds (or shorter) of history in this case, even though the time since the last known valid frame may be more than 3 seconds.
Basically people have to understand that the impressive part doesn't come from when it's a copy machine, but when the new stuff it makes actually is useful, which is not the case for that DOOM model. One good way of understanding the generative AI wave that I figured out recently is roughly comparing it to a lossy zip compression, your prompt is mapped to a "hash key" (embedding to be more true to the theory) that points to what to decompress, but since it was a lossy compression, the result will not be 1:1 with the original data, but a weird blur that might happen to be good new stuff and that's the part we should aim to improve.
great content keep it up!
There actually is data on how players play Doom. It's perhaps the first game to allow players to share gameplay videos by storing player input data. I don't know how much of this has been preserved, but there should at least be some speedrun and other interesting lmps that could be used for training purposes.
I don't know about you, but I wanna play through those edge cases
Seems like it could be nightmare fuel
Hey, we made a way for a computer to play DOOM!
Yeah, so did I...
* Clicks on DOOM game on desktop. *
Well this is at least getting close to Ross's game dungeon's dream for game preservation, he dreamed of one day being able to feed in game footage of any existing game and having it spit out a replica that didn't rely on old hardware or a companies remote server that will eventually shutdown. Though more ideal would be it spitting out the source code to a deterministic engine written in some like C that could be modified and recompiled on other hardware if needed.
There would be a necessary middle step in game development where we make single game capable engines, but we do it on a general model. That way, the game is actually a file of weights. This would make more commercial sense than a truly general AI game engine.
I think those blobs are bullet decals. It was trained on footage where some had decals and some didn't
Are neural nets really non deterministic? I don't think so, if you train 2 neural nets with the same inputs and use a PRNG with the same seed for the magic sauce, it will give the exact same result, it's just a computer program like any other
I think the "non deterministic" was more about game behavior than neural net output - so running away from an enemy for a few seconds might cause them to disappear, or move to a different spot, or still be there when you turn around.
@@travissmith5994 maybe... but "deterministic" is a well defined term, it just means that the same inputs will give the same output and everything a computer does is deterministic
Neural nets (an other things like video encoders and CSPRNGs) only seem non deterministic when you are dealing with noisy real world data
@@olhoTron not all programs are deterministic. There are sources of true randomness connected to your computer. It starts with the input devices, goes on with the time of day and finally cryptographic algorithms use special hardware RNGs based on physical devices like crystal oscillators, integrated into the hardware security module. In the case of this AI model the user input is probably enough to cause it to show chaotic behaviour (that is, small changes in the input give big changes in the output)
@@MrFaerine no, all programs *are* deterministic, the change in behavior comes from the inputs taken from the real world, but if for some magic reason the noise captured by the hardware RNG repeats, the values it will give will also repeat
@@olhoTronwell by this definition everything is deterministic I.e no such thing as true randomness in nature yadaydadayada
There are TAS of videogames that would be perfect, not only do they record button inputs which you can relate to the game content, it records the very best of it.
Maybe the real best way to run Doom is on the friends we made along the way.
I doubt that these things will ever practically contribute to streamlining workload. As someone who trained old school natural network to predict water quality, I can say that AI right now is a worse search engine, and it will stay that way until some new method emerged.
But it is great how it is possible. The product quality seems to be exaggerated, but they did train AI to ignore or correct its own mistake to crash the game in 3 minutes instead of 3 seconds. It is like realizing laying down train track as the train is running is now possible, even if it is inferior to doing it the own fashion way in anyway.
It literally looks identical to the time when I was playing a Doom wad in my dream