A 3D foundation model is very hard due to the scarce availability of 3D / spatial data which means brute forcing with a neural net by throwing billions and billions of data points does not work. Furthermore, transformers may not be the most suitable architecture and likely a new model will need to be build and trained from the ground-up to handle this new data paradigm. There is a lot of buzz around NeRF as a viable path forward (Nvidia is certainly pushing hard), what some people may not realize is that NeRF is not built with mesh optimization in mind which is what you need for 3D models, it is good for rendering and presenting volume in 3D space but at its current state is not meant to displace photogrammetry. In the near to mid term, the best chance to a performant 3D engine is by leveraging the combinations of neural nets, geometric priors and curated spatial data (to the extend available).
It's an excited time to be in tech yet again! Tools are evolving and ML is getting closer to being in the hands of indie developers.. imo where the magic happens! Standing on the shoulders of giants and I'm looking forward to see what comes of it.
A 3D foundation model is very hard due to the scarce availability of 3D / spatial data which means brute forcing with a neural net by throwing billions and billions of data points does not work. Furthermore, transformers may not be the most suitable architecture and likely a new model will need to be build and trained from the ground-up to handle this new data paradigm. There is a lot of buzz around NeRF as a viable path forward (Nvidia is certainly pushing hard), what some people may not realize is that NeRF is not built with mesh optimization in mind which is what you need for 3D models, it is good for rendering and presenting volume in 3D space but at its current state is not meant to displace photogrammetry. In the near to mid term, the best chance to a performant 3D engine is by leveraging the combinations of neural nets, geometric priors and curated spatial data (to the extend available).
Very hard...but not impossible!
@@A16ZGAMES any startups working on this?
@@pinghe3605 genielabs tech close the gap between current state of the art 3D generative models to 3D game-ready assets
Exactly!
It's an excited time to be in tech yet again! Tools are evolving and ML is getting closer to being in the hands of indie developers.. imo where the magic happens! Standing on the shoulders of giants and I'm looking forward to see what comes of it.
More interviews ❤❤🎉