Google's DreamFusion AI: Text to 3D
HTML-код
- Опубликовано: 27 сен 2024
- DreamFusion is Google research (arxiv.org/pdf/...) into generating 3D objects from text prompts, and is based off 2D diffusion models.
Stable DreamFusion github: github.com/ash...
NeRF (Neural Radiance Fields): www.matthewtan...
Meta AI make-a-video: makeavideo.stu...
Neural Networks from Scratch book: nnfs.io
Channel membership: / @sentdex
Discord: / discord
Reddit: / sentdex
Support the content: pythonprogramm...
Twitter: / sentdex
Instagram: / sentdex
Facebook: / pythonprogramming.net
Twitch: / sentdex
Me: Watching you talking about DreamFusion on 144p
Me too bro me too
144p Bros
Resolution not available please come back soon
you missed at zero at the end right?
Lucky you. This model employs just 64 x 64 to do its thing.
welcome to turkey
A decade in AI development is like a century for me. There is, and will be more progress with AI and all technologies combined, than anybody can think of.
4D model generator
Holy crap, Imagine instead of modelling an entire player character you just fill out a template file and it generates the entire model, along with automatic AI rigging would be incredible and massively speed up current game development/video production tasks.
I really wonder where this leads to in maybe 10 to 15 years.. are we gonna be able to design games and every other content on the fly.. so much potential and risk.. super exciting imo
@@nulled7888 30 years
@@onlycorner5565 At the current rate of advancement, 10-15 years seems very realistic, probably less.
@@acorgiwithacrown467 well yea i think so too, but the exact number isnt the point here.. even if it took 50 years its so incredible that humanities engineers can build something so insanely complex.. it almost feel like the level of crazyness i felt the first time opening a webbrowser and realizising the possibilities.. its insane and its only getting better over time... :D
@@nulled7888 Yeah. I was just thinking that. You could have boilerplate code, without the assets, and then the assets get generated randomly at gameplay. Every time you play it could be like playing a different game (at least in appearance).
Like you said in a previous video, the rate at which these innovations are coming out is crazy. It seems like every month we're taking a massive step towards a fully generated AI world. Awesome video keeping us updated!
After all billions have been poured into this , imagine it the same billions were poured into space research or cancer/medical research
@@shukrantpatil billions ARE poured into that research, are you living under a rock? (however, i think most of the funds are pocketed 😢) (so i dunno what i’m saying 😢)
I never thought artists could become obsolete.. Until now
I am starting to believe it will plateau. Given it is being trained on human art, it will forever approach that without ever arriving, there. (Rather than ever doing anything truly transformative or eclipsing.)
Lavamar- And then one day, you realize that you are the AI. You are in the simulation.
I would like to see some videos on the open variants. You pick and I'll watch. You are one of the core influences that got me into AI/ML. Thanks for your work.
Nerfs are cool.. but actual 3d models are truly exciting as a 3d artist. I can't wait to see where this goes.
For game modders and 3d printing enthusiasts this is going to revolutionize everything. Imagine being able to just type in some words and then you have a free model that you didn't have to pay for. Now you can focus on animation or effects.
@@The_Foolish_Fool Even if it just spits out a model thats 90 percent of the way there, that saves me a ton of time.
@@The_Foolish_Fool It would be more useful to me if it used images from google to generate an accurate 3D model of something, like a car. I can see why this AI is gonna be really useful, though.
@@incription I guarantee an AI could be written to do so.
@@The_Foolish_Fool the wait wil continue for 18 years
in few decades we will just ask for a film, AI will write the scenario, create characters (make you the hero) and render everything from 0 for a complete film
I am sure you are editing a video on OpenAI's Point-E - image2pointcloud tech. Can't wait to get your take and how you can tweak the code to improve results. Your insights are always valuable! :)
Can the technology just slow it down for a notch, like 1 month at least so we can absorb what have been done by now. I got raised with the technologies for the past 25 years and im 30, starting with dial-up internet and everything in-between until now. The tempo for the last 5 years is bananas dude. :D
and for what? nothing can replace nature, emotions, love.
@@dompeca Yes emotions, people felt bad and fixed a lot of problems to be happy, take a look around and the convenience you have in your life, that's what we do, emotions, love, hate and everything in-between. That's our nature... :)
What happened to the neutral networks from scratch series? 😭😭
You seem to completely give up on one of the best projects on RUclips in this genre ever
he literally wrote a book about it :D
@@tiefkluehlfeuer There was going to be a free RUclips series accompanying the book
These are indeed exciting times. I’d love to see more videos on text to video generators.
What a time to be alive!
saw your video for 3 seconds and i had to sub. you're great, keep it up
To take it to another level, I can imagine this real time, VR environment generated using input like your heart rate, galvanic skin response, and other physical body reactions, so the environment changes, according to how much you are enjoying it, or feeling, scared or feeling bored.
The future is AI. So glad to follow the best channel that gives a technical know how of machine learning and deep learning. Wonderful work as always.
As an AI lover since childhood, I am excited and proud to be living in this era with so much advancement in AI.
A real Open world Game without any Limits would be generate endless 3D models buildings flowers weapons and stuff like that maybe there a possibility to simulate and Eco System automatically via AI
can it create me a videogame? can it create a body for itself with a 3d printer?
Would be interesting to see an NN, to generate models from reference images. For example, front, side, top and back references images.
This is a game changer for accessibility in games for the handicap
Have you ever heard about the simulation theory? Fascinating to see it in action. ♥ ✨
The double faced objects and the green hollow cube still look super cool!
I remember watching you when I was 15, I’m 20 now that’s crazy
Hope you are keeping up with your Math and general coding skills.
Great advances. However, high polygon 3D overlays for AR are very hard and will be for a while. And we are still terrible at modeling dynamic fluids. (The real world has wind, water, dust, and plain ol' second law of thermodynamics. When someone codes rotting wood and growing mold routines into my favorite VR experience... I shall be impressed ;) Get working on it folks!
Insane work with the book, *congrats*
I have a cnc machine, being able to 3d model and turn them into physical objects would be insanely cool.
AI generated nightmares
In a few years, we will probably say: ‘A Minecraft-like game but more realistic’ and their will come out a fully designed video game with 3D models included 🤯
Newbie question: why is all the code work being done in Python? I’d like to buy your book in hardcover, but the language I’m learning is Swift because I want to develop exclusively for Mac
Developing for mac means running your AI model inference on a mac I assume which is totally separate from learning how to actually create and train that model. So you can build your model with python like in the book and use swift to run it on mac. hope that makes sense
How do I install this on Windows?
Seems like videos, on RUclips for example, can provide endless supply of training data for traning.
Lovely that was something I was thinking
oh, as soon as you showed the 3D renderings from text I was already thinking about creating fully virtual worlds in real time, as in open world and generate the world as you go, kinda like minecraft, except that the environment could be... unbounded? unique? idk how to describe it...
that's gonna need some serious GPUs tho
The future is fully procedurally generated games. You could simply enter a type of game you want (as specific or general as you would like) and have the AI direct the flow of the game, generate assets, have realistic NPCs , and game styles/abilities generated that cater to your likes. Imagine a game where every part is directed by an AI that KNOWS what you like. You will never play a bad game ever again
@@lack_of_awareness I'm extremely excited for a future like this. It seems unreal and I can't wait haha
"unplanned" maybe
So, by 2030 Google will finally be able to get rid of the pesky RUclipsrs? Is this the final goal?
hello!
would you like to make a serie of how to make a CNN from scratch(only numpy)?
why we cant use it?
The code in the end was lit.
7:00 my favourite part
great overview - thank you.
Still waiting on stable diffusion for audio
Great reporting, keep us up on the latest devs pls!👍✨
What would be interesting here is if you can use this to create training data for more efficient/higher quality text-to-3D models that doesn't solely need to rely on 2D images. But with the current speed of 30 minutes per 3D model that seem unlikely.
I'm confused. Should I proceed with me 3d modelling lessons or it's better to jump toward the unreal engine?
Whatever you do, take a pottery class or (woodworking, stone carving) once a week... very useful for disconnecting from screens and naturally understanding textures and intuitive physics. ;)
*text to videogame, other than text based adventure ofcourse
"yo Mr white we need more vram"
Image to 3d is going to be a game changer whan this gets good!!!! 😊😊
I did a test a few weeks ago and made a video twith the open colab stabke drealfusion. The results aren't there yet but soon they will be.
This is crazy my thought is in the future they can and will put whole books in the generator and get a whole different film each time. Imagine books like Harry Potter reinterpreted by AI 🤯🤯🤯. The only real question is how long
Mash up your twenty favorite books starring your personal favorite actors and a soundtrack by your favorite music groups and instead of watching it on a screen its in your head and you can't tell the difference between your mashup multimedia experience (interactive as well...u r a character in it) and a dream.
When is the nnfs video series are coming back?
Basically Computers on LSD
Wait....can you guide these videos in RT? Sounds scary....I see what you're thinking.....wow.
Look around . . . . we live in a very complex simulation.
Add time to make 4D diffusion models.
Wow… Right to call it a rabbit hole. Putting on a VR headset and walking into a world being generated in realtime based on your feedback. It will be just like Alice in Wonderland. Inclusion of biofeedback as model features could be super interesting. I’m curious what should be used as training sets though. Users playing video games? Watching movies? Moving through real life?
book to movie could be the outcome in 10 years
6:24 - Yep, that sums up the whole video. Very exciting!
We've finally hit the point where "can it play crysis" isn't a thing. The 4090 is super buff even without AI frame addition like DLSS 3.0. PS5 games are still feeling pretty PS4, which also feels kind of PS3. Things are prettier, but even games like the Final Fantasy Crisis Core on the PSP had better face animations than most present day content. We're on the cusp of a new thing. Like that first time we saw a PS1 or N64 game. Something that really makes ya go OH S H I T. It's coming.
Player input effecting the AI generated world. It's making my brain fizzle just contemplating the possibilities. GPT-4 integrated into VR at 240HZ+ with eye tracking, DLSS 3.4, and foveated rendering, just to name the tippy top.
Web 3 is looking like AI stuff and block chain stuff
Please, could you define your assertion "hugely impactful" at 5:23. My point of view is that Diffusive models (2d or 3d or video) are impressive from a technical point of view but their impact is still to be demonstrated. For generating content.... ok why not but .... why? Do you really think that millions of people will rush in theaters to watch an AI-generated movie? Or to read an AI generated novel? I doubt about it. Now, in the industrial world, we are still at the dark age of using business valuable AI. Mainly because of the lack of massive datasets needed to train these architectures on industrial related domains. Ask any diffusive model to generate something related to a industrial technical topic and you will cry. Even for synthetic dataset, as you said, the limitations are enormous today. So for creating an image of a "Panda wearing short and surfing in Hawai" just bravo, impressive. But to make something real valuable I doubt. A better way to spend time and money is to develop GNN in my opinion :)
After writing nearly an essay in response, I think instead I'll respond to this in a video since I've seen this question a few times.
I think it's incredibly short-sighted to just look at surfing cat pictures and declare the tech overall pointless.
2D and 3D generation can be used for art/games, sure, but it goes beyond that obviously. I mentioned simulators in the video, sim2real is still a very hard problem in robotics due to generalization being hard. Solving this problem alone will be massive for humanity.
Beyond this, looking at things like the NNFS series for transmission of knowlege/education, people love these videos largely due to the animations. These animations take unbelievably long to develop, but they help convey knowledge super well. Imagine if we could 10X this process like copilot 10X development. The impacts just in these areas alone would be world-changing, and these are just 2 small examples.
To add, you are also missing that a new workflow is being created here. Our reality is surprisingly simple and if you push this far enough. Not only do you have 3D style transfer, which is one of the harder aspects of organizing a team of artists for a project: constancy.
You have the ability to take images of a product and endlessly generate variations of such. Meaning with these models we move towards the next industrial revolution. One that is hyper personalized and as needed versus mass produced generalized. Accounting for entertainment and products.
@@sentdex Thanks for the reply and comments. I will try to make it short too :). In my experience, the usage of synthetic data is often disappointing for AI training. You mention robotics training and I agree that it is impossible to put a robot prototype in every environment that it (he/she/they?) can encounter in real life. 3D Synthetic image data is a way to create labeled datasets but it assumes that the generator is unbias and does not produce "anchors" not visible by humans but well discovered by the AI model during the training process. The detection of bias and anchors is a very difficult problem that could occur in both natural and synthetic datasets but is more susceptible to damage to synthetic ones due to their size and their nature. Regarding the NNFS videos, it is a great piece of work for sure and I doubt that any AI can do as good as you did. Cheers.
How long before Google NightmareFusion: Text to DNA...
Thanks now I have to get a new job
what was your old job?
Interesting talk. Hey, Sentdex, have you read Summa Technologiae, by Stanislaw Lem? 🌌
This ride will *accelerate*
Hey man, what kind of thumbnail you added for video, what kind of images are you showing here ? This aint right.
Any way I can get my hands on a copy of DreamFusion?
Please make a video on open source models.. Will be so helpful
This is an open source model, as have the previous videos been too
So this is how it begins….
Thank you daddy google! Remember! Don’t be evil !
High fidelity + unique asset = infini digi avatars = diluted real humans = ubi
this is my shit, thank you
Working on it! 🤣
Jessie Pinkman talking about ai
Problem is, these AI generated models fits simple indie type games with simple graphics. What if you want to create AAA quality games. This wont cut it (for now)
Spaces tank
another powerful tool to amplify the ocean of ignorance... unless one studies wisdom
I liked this lesson a lot, thank you
Interesting times
Nice
Watch out Spielberg! lol
nft generator
Such a good video! Keep it up
DreamFusion is a dead end since this is completely unusable in any current 3D processes in movies or games. I suppose the easiest way to explain this is to understand the way 3D is currently constructed. The current 3D construction is discrete math-based and basically folding 2D planes to form a 3D object. From the basic calculus, you will know that it takes an infinite number of vertex and edges to form a perfect circle because it is a continuous and non-divisible curved line. To form a curved surface discretely from a 2D plane, you will need to subdivide the 2d plane a lot. That is the reason a 3D object is referred to as mesh. As the shape gets more curved and complex, the number of vertex and edges required to create that form will exponentially increase needing more and more computing resource to manipulate and render. There is something called retopology in 3D. It is basically a tedious and laborious manual work to reduce the number of vertex and edges while maintaining the shape of high polygon model so that the game assets can be run on a consumer PC.
To be honest, I found this rather annoying because there are so much AI can do to revolutionize the current 3D industry. I mean there is a reason why each new AAA game takes longer and longer to make and hundreds of millions of dollars to create. As a result, the game industry is moving more and more toward easy money from Dopamine addition generated from modern gocha games. Yet, there seems to a gap of between what AI community is doing and the people who understand what AI need to do to make a difference.
How is DreamFusion's process different from the one you're describing? Genuinely curious. Do you assume automatic repotology to be impossible?
@@KutluKanyilmaz DreamFusion uses 2D to 3D AI and the way it works is by projecting or simulating 3D and rendering directly to a 2D raster image. So, there is no real 3D object, which can be used in the current 3D workflows, created in the process.
I believe Ai retopology is not only possible but most desirable. But there seems to be no development currently going on in that direction.
@@mrdragonrider Thanks for the answer. If that's the case, I guess you're right. I was a bit misled when the host of the video said that the 3d objects can be "exported to other software" (like a .fbx file for example).
@@KutluKanyilmaz Technically, any simulated mesh can be converted into a real mesh such as converting a displacement map into a mesh. But as far as I know, NeRF research is going into the direction of skipping 3D modeling, texturing and rendering altogether and go straight to 2D raster image rendering. And if you think about it, it only makes sense because it will only make the process far more complicated due to the finicky nature of 3D meshes.
10:20 - "undiscovered prompts" - Man... such a weird thing. We're literally learning what the magic words are to invoke the wishes we want from the genie. And being articulate is considered one of the highest spiritual abilities.
Speaking creation into being.
LOL , if u twist it enough you can kinda think of it as figuring out the spell phrase to cast
Your book is awesome 🎉
>steal google dreamfusion
>type "half life 3"
>????
>profit
I would love to see a brain to computer interface fed into stable diffusion training.
oo. Imagine training an AI to generate images but the input is a person’s brain signals. We could generate images catered to a person and have it display images of what they feel. Emotion/feeling/thought to image lol
Now imagine brain getting the input from the ai too lol
Love the book!
Of course it's cherry picked.
It's early tech and they're trying to show you what it's capable of in the best cases, the promise.
If this is true then writer becomes director
The biggest annoyance with the entirety of AI everything is that we still live in a world where depending on where you live, it can still be challenging to get stable internet, climate change is a thing, corruption is rampent, and millions of other problems exist. Could these brilliant technicians maybe focus on solving these major concerns before solving the non-existent problem of art? Killing off hundreds of dream jobs. Maybe? This stuff is cool, but seems like a gross misuse of talent and brilliant minds. Just a thought.
Climate change is not a "thing". Climates have been changing since Gondwana was no more ;)
Stop being a complete idiot please. It's a different skillset entirely. Starvation and climate problems are political issues.
@@MrNote-lz7lh Climate is not a "problem", and certainly not a political "issue". More like coordinated propaganda to force the unwitting to pay more taxes!
@@MrNote-lz7lh Clearly I was implying that people intelligent and skilled enough to create machine learning could have applied all of that ability towards something the people of the world are in dire need of, especially since political issues will never get resolved by worthless politicians. It would be infinitely faster if these problems were handled by scientists, engineers, and the like. In the future, try thinking before you respond.
this is amazing
Yes please. Text to video. The thought alone is like magic.
Useless until we democratise AI, until Google, NVDIA and Meta will hold the key and processing power to collect and train those immense dataset AI will forever have an impossible barrier to entry for _real_ progress.
That's why NeRF is probably the most interesting idea came out in the last 10 years or so, you do not need a massive train dataset that belongs to those huge corporation.
That alone makes a huge difference.
Welp, I learned modeling for nothing 😭
This is TRUE ai. technology is finally catching up to all my assumptions.
They may not understand that this will define sentience. MARK MY WORDS. If they continue this technology, this will make robots become human like.
this AI thing is getting out of hands
If you condition on the previous frame and user input, you could have text2game 🙂
Love me some automated Photogammetry
This is sooo awesome
Dear Harison,
I wonder if you could use AI to create some meaningful art to show your support to our revolutionary movement in Iran. Women and teenagers here are sacrificing their lives for freedom. We are facing massive Internet blackouts. Please be our voice.
Not so much exciting as depressing. There goes my CGI career. Humans will be out of the creative picture when the AI goes from mere technical prowess to conceptualization. There will be a plethora (glut) of artificially generated art -- visual, musical, performance, literary, etc. -- that consumers will become numbed to novelty. Very sad for humanity, imo.
so what are you gonna do then?
@@danny.golcman6846 I'm retired now. My career spanned the 40+ years of the computer graphics industry. There were no computers for graphics when I got my BFA. Now, how do you assure an art student there will be a career for them after school? Why even pursue an art career if you'll have to compete against entrenched AI replacing your job?
@@srb20012001 hm yea good point. I was thinking of doing an art career but seeing all this ai stuff makes me reconsider. Imma probably do something in tech and coding instead.