Wow, NVIDIA’s Rendering, But 10X Faster!
HTML-код
- Опубликовано: 1 сен 2023
- ❤️ Check out Fully Connected by Weights & Biases: wandb.me/papers
📝 The paper "3D Gaussian Splatting for Real-Time Radiance Field Rendering" is available here:
repo-sam.inria...
Unofficial implementation: jatentaki.gith...
Community showcase links:
/ 1693755522636206494
/ divesh-naidoo-48809934...
/ huguesbruyere_gaussian...
jo...
/ 1694322101744738706
/ 1692346451668636100
My latest paper on simulations that look almost like reality is available for free here:
rdcu.be/cWPfD
Or this is the orig. Nature Physics link with clickable citations:
www.nature.com...
🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Aleksandr Mashrabov, Alex Balfanz, Alex Haro, Andrew Melnychuk, Benji Rabhan, Bret Brizzee, Bryan Learn, B Shang, Christian Ahlin, Geronimo Moralez, Gordon Child, Jace O'Brien, Jack Lukic, John Le, Kenneth Davis, Klaus Busse, Kyle Davis, Lukas Biewald, Martin, Matthew Valle, Michael Albrecht, Michael Tedder, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Rajarshi Nigam, Ramsey Elbasheer, Richard Sundvall, Steef, Taras Bobrovytsky, Ted Johnson, Thomas Krcmar, Timothy Sum Hon Mun, Torsten Reil, Tybie Fitzhugh, Ueli Gallizzi.
If you wish to appear here or pick up other perks, click here: / twominutepapers
Thumbnail background design: Felícia Zsolnai-Fehér - felicia.hu
Károly Zsolnai-Fehér's links:
Twitter: / twominutepapers
Web: cg.tuwien.ac.a... 
Наука
This paper is the actual goat not just the goat
My man is the "-est" in goat not enough
This paper is the new tupac
'Goat' is such poor English, it should be 'GAT' but of course that would be to say there will never be better so its better to say its the 'Greatest Algorithm Yet!' Yeah, 'GAY', that'd suit millennials better.
This is the paper of all time
What's the difference?
I really love when the most amazing papers don't use neural networks
For as cool and magical as neural networks are, there's just so much more you can do when you actually understand what the algorithm is doing
yes, because people are tired of inconsistency, daydreaming result, gaslighted, or just blatantly wrong.
this on the other hand produce much consistent result and could be improved further
Yeah, neural networks/ai are great but are being used for everything now… even when it’s unnecessary or a good ol’ algorithm would be enough.
@@jensenraylight8011 .. Not saying that people are not tired of the things You enumerate. But I would like to point out that neither Neural networks or other similar "methods/algorithms", nor their creators, are "gaslighting" or "lying" to anyone by creating their "methods/algorithms".
Rather it's the people using the results of these "methods/algorithms" that might be using them for that purpose. But that doesn't, tie this "negative activity" to any inherently quality or property of the "methods/algorithms" used.
Rather the "method/algorithms" used are only chosen due to their perceived efficacy and/or quality of their results. And that being the case if we create better algorithms "by hand" then those will be used instead, for all the "negative activities" You just mentioned.
E.g. If I choose use a "program" to create a realistic picture of a well known politician in the act of a "heinous act". And I then use the resulting image to spread misinformation. It makes NO DIFFERENCE WHAT SO EVER whether the "program" I used is built on an "A.I.-based" or a "completely hand crafted" algorithm. Imho the "underlying origin" of the "method/algorithm" is a total non sequitur in this context. It's simply the "perceived quality" of results that will dictate the choice, and if the "hand crafted algorithms" are "better" then these will used, for good and ill purposes alike.
Best regards.
Wtf stop this scam shit in my comment section
This is why explainability in AI is so important. Let AI do the tedious exploration of solution space, it explains what it's doing, and then we can work with what it found and modify the solution however we want.
I love your use of the phrase "just one more paper down the line". So true.
But this one side-steps and leapfrogs all the AI ones?
Can't believe it has no neural components! Just proof that we humans still have the advantage, but one day that may change 😅
It's human-made tools either way.
How does that prove it?
@@AlvaroALorite Because it's superior to the neural net models.
I mean, it's still an evolution of NERF. They just figured out what was in the black box reasoning that NERF was doing. This is the real strength of neural networks: not offloading reasoning onto machines, but creating alternative paths toward conclusions that can then be studied and developed into theory.
@@michaelleue7594great view
Would love to experience this level of detail in VR.
give it some years more^^
It'll happen in your lifetime guaranteed. The technology improvement are still exponential which is crazy to me.
i give it 5 years@@AdiusOmega
@@minheritancethere's a gaussian plugin for unreal now, so this could be done today
This is even more amazing than the average mind blowing break throughs on this chanbel.
Those thin structures were impossible to even scan a few years ago. What a time to be alive and be able to witness the advances that we achieve just by pushing further.
Similar to removing noise from astral photography, we've been able to recover fine details/structures from videos that didn't appear to have them based on just the raw resolution. Just because you or the researchers in these fields are ignorant to something doesn't mean it hasn't existed for many years. I've improved open NeRF models by adding in steps to include gaussian/eulerian amplification/magnification. It's really a simple fix and should have been picked up years ago. It baffles me that it's been such a problem for so long that they just can't seem to figure out.
this paper is just lit. imagine it being used in future games. crazy times ahead everyone.
BOOM!
2 decades at least
@@lawsen3719 2 years you mean whenever it can be rendered real time.
@@Andytlp Realtime is 30-60fps. They showed here 135 fps..
@@jenkem4464 Would be one way to get hyper realistic graphics. Fps seems good and runs real time huh. What are the requirements though?
It’s pretty impressive that we get better speed without sacrificing quality. Amazing work!
Sublime! The amount of progress that's been made in just a year or two feels _unreal._ Looks like the Singularity is right on-schedule!
This is actually really impressive, also glad to see how much potential there still is to be gained from non-AI research. I feel like most recent papers were largely based on machine learning.
"quite a bit of memory" is a nice and understandable value
At least It seems to be very compressible.
5:10 can't that be fixed somehow by combining it with ray tracing as like the next step of rendering a scene?
It's incredible. I never thought things would advance so quickly, I'm starting to believe we might see actual AR and VR games before the end of the decade even.
@@Faizan29353keyword "actual" - stable immersive media that looks (practically) indistinguishable from reality
No longer "It's NeRF or nothing" 😮💨
It has very very minor issues I can see like blurriness and bad aliasing at corners, but it looks 100x better than traditional render 10 years ago. And it took 10 years so, I can imagine using anti aliasing, anistrophic, and multisampling can make this WAY better than it already is. The gta clip was so impressive.
Love the endless advancement, hope this new technique can output 3D mesh data
Currently it can not but maybe in the near future - it will be a game changer.
Does it have to?
I hope videogames start using this tech instead of meshes, or a mix of both 🤔
It’s an interesting question whether this replaces mesh objects. I could see it going in a hybrid direction.
The regular NeRF can be converted to meshes via methods such as the marching cubes, so I guess the same can be done with this one. This method is terrible for anything that isn't a solid smooth object but there are probably others.
This is ground breaking! This is what I subbed to this channel for.
🎯 Key Takeaways for quick navigation:
00:53 🏞️ New technique for real-time rendering of virtual worlds promises over 10x faster rendering than previous methods, addressing thin structure challenges.
01:52 🎮 The new technique offers both faster rendering and higher quality results compared to NVIDIA's Instant NERF technique, surprising with its superior performance.
02:49 🖥️ This breakthrough method is not a NERF variant and doesn't rely on neural networks; it's a handcrafted computer graphics technique with innovative ideas.
03:14 🌊 The technique uses a 3D Gaussian splatting approach to represent objects as a sum of waves, enabling efficient rendering on 2D screens while conserving computation around solid objects.
04:40 🎨 The algorithm focuses on scene primitives rather than pixels, drawing from a long-standing concept in computer graphics, resulting in a fast yet high-quality rendering solution.
Thank you 🎉
People sometime think neural networks are just brains, and to an extent, you can call them that. But what they really are, are function replicators, and depending on the function you are replicating, it may or may not be faster to just write the function.
Honestly their approach is so mind blowingly creative its insane
5:41 - My mind is blown....this is like the most incredible graphics tech I've ever seen and the most photorealistic and convincing fully realistic virtual dog I've ever seen by far ! what the....it's so incredible to the point of having hard time to believe such tech can even exist..
I was impressed a year ago and now wow
3dgs is so intuitive when looking at it, I'm surprised we didn't think about this in the 90s
Somebody needs to use this in VR asap, and make us a nice RUclips video.
Does anybody else think that the Professor used an AI voice clone to do the sponsor segment? To me it seemed less express or and I think there were some audible artifacts.
NVIDIA is popping off!
So incredible to see more and more incrdible papers from NVIDIA, Google's Deepmind AI, Open AI, Adobe and more!
What a time to be alive!
My jaw dropped when you said the word "algorithm". Insanity.
Fascinating work.
Imagine integrating both High Resolution Photography (for perfect details), Video (for motion, 3D perspectives) & Lidar (for comprehensive depth perception) with this kind of neural network processing to render real life spaces for games/movies/etc., as well as to teach AIs about 3D spaces.
Incredible detail. Almost indistinguishable from real images at points.
This proves that you don't need over engineer techniques for certain tasks... also LEARN FROM GAME DEVS some are specialized in hacks e.g demoscenes where 3D implemented in systems that don't have gpu !
Mixing this with photogramitry is going to be insane, I've been using an AI that practically does that already using images generated through MidJourney
Thank you. I approve of this. Much gratitude. Now add a simple transform vector as we swirl and jerk it around so it looks like it has mass.
Ever since the corridor video I've been waiting for more news and coverage on nrfs. I'm happy to see it coming from you.
"Two" Minute Papers' video is longer than the original author's presentation.
Whenever I see these results I go, great. But how do I use it in a workflow to add stuff to it or add these volumes to a normal mesh and shader render so I can modify the result?
Most exciting report I've seen in ages. Dawn of the next level.
A feedback: can you use a more objective metric than FPS? If it's FPS on a supercomputer GPU cluster, I don't know how realtime that is. But if it's a stock CPU's onboard graphics chip, that's much different.
On the paper - stunning results... can't wait to see the impact of this.
I think it popped up on screen that it was on an A600 which as I understand it, is kinda supercomputer levels.
5:15 _"These first, unoptimized versions, consume quite a bit of memory"_
Like Ram? Exactly how much memory are we talking?
Its almost never mentioned in his videos unfortunately. Love the channel, but wish these factors were not just glossed over as an afterthought so often.
That dog looked exactly like my dog that passed a few years back.
I just love how there are still smart people who don't use AI and instead optimize things
Not to be rude or anything, but did you use an AI to generate the voiceover for this video?
That settles it. We do actually live in a simulation.
What a day to be alive, humanity crafty coding beats out a neural network
I have shown this channel for some people and they are 100% sure the narrator voice is generated by an AI 🤖😅😅
Or he's just Austrian and has a different cadence when speaking English... lol
Can someone pleae explain this to me like I'm an idiot audience member watching a scifi movie?
What is the source they're using to create the 3D scenes? A collection of photos? Or is it a 3D model created with this technology applied over the top? What sort of hardware does it rely on for rendering?
Your voice sounds really robotic on this video, did you try a voice generated by AI or is it just the jump cuts?
Very nice video btw :)
I think it's just his voice lol
Yes, it does sound like TTS AI, which has been used on the channel for a while
Given the speed and clarity of these, I wonder: can language embeddings work with these, the way LERF handles it for NeRFs? And how much overhead, in terms of time, would that add? Because if we can do 135 fps real-time radiance fields with full language embedding, we can make robots that fully understand everything about their surroundings from just real-time camera information... which is a huge step forward in AI-brained robots.
Need to create a large database of scenes and objects along with detailed descriptions, that way we can train models to generate new scenes based on user input.
Can these techniques output a geometric mesh like photogrammetry or do they only output 2D images?
What a time to be alive! It is absolutely amazing! 😉 👍
This is actually really incredible.
Mr 2 minute papers' voice has an unnatural repetitive rhythm about it in parts....
This is just so beyond impressive and I can’t wait to see this tech used in video games
Am I the only one here who can't comprehend how the 3D Gaussian thing works? How is it rendering a 3D world using "simple computer vision techniques"?
I haven't read the paper so this is just speculation but the gaussian is a good building block, like a voxel but fundamentally smooth. They have well understood properties and are highly suitable for linearization (eg. like in a Kalman filter) so you can combine and project them easily.
Please talk normally. You're ruining your own channel. 10/10 content. 2/10 syncopated narration. Finish...your....sentences....and phrases.....according to......everyday speech.... This also....applies.....to inflection. Thank you. I hope you will sort it out.
Mind blowing development!
What a time to be alive!
Curious how animations would be handled.
Man, imagine this in real time streaming!!!
just another excuse where cards costing over 1500$+ cant render games at stable 60Hz on max settings, because of those supposedly "faster working" technologies.
Is the voice over generated too😆?
I keep coming back to your videos for the groundbreaking scientific discoveries presented.
What really puts me off tho is the pacing, pronunciation, and rhythm of your narration.
Hope you'll be able to train your speaking flow to make these otherwise great videos more enjoyable.
That's...
That's incredible.
This is the next step of the evolutionary path of technology.
what a time to be alive!!!
That's my dog!
Well, not actually my dog, but it is identical to mine!
Maybe she's got a secret hobby. 😊
Soo..... Use Google maps as your input, this creates you a 3D world to play in ?
Incredible stuff!
Creating a virtual world maybe solve some problems but make a huge number of new problems at the same time. I really don't understand why you are so excited?
so if i get that right 🤔
they "just" ignore the 3d part & only "smooth out" the output picture before the screen shows it?!
cool idea indeed!
Fellow scholars... I did not hold onto my papers for this one
Hey, was the voice fr this episode done by AI?
Saw the video of the games looking "realistic" and got the reminder again thar realism today is not what we see in reality, bur rather what we see in the movies or through a cheap camera that is struggling wit the white balance.
videogames in 6 years are gonna look amazing
Impressive! Amazing! What a time to be alive!
Hey guys, I have a question. I am a land surveyor and I want to utilize this technology. Can I simply upload a video into the "software" and get an output as 3d model ? Would this be a potential use case for this study? (Sorry If it's a dumb question) If so, is this years away for end user or I can actually make it happen today with a graphics card and some codes complied? Sorry if it's an obvious question. I'd appreciate if one of you can send me in the right direction. Because this would be a game changer in my profession! Thanks.
This is the type of tech that both fascinates me and also makes me depressed as it is clearly going to replace my dream job
I would like to see a paper on 3Blue1Brown's video on the topic of the light through a barber pole, that would be impressive.
Now this one gets me excited
this is gonna make for some insane horror games
Was relatively easy to create Gaussian splat model from my own video.
I'm not understanding what this is about, like what are the inputs and outputs to the algorithms discussed?
His voice just gave up 💀
Hey, just so we know, some fancy new vr headsets with depth sensors are coming soon...
Building virtual worlds with just a glance? I bet money on it
wow! Amazing advancements. Thank you very much for sharing.
i mean these were thoughts which theorised would take longer time but here we are.....WOW....it is finally here.
Soon this channel will be called Two Minutes Philosophy, since I would question my reality soon.
Are there any practical use-cases for radiance field rendering? Seems like there are so many papers on it, but they are all there for the sake of just being there? I'd get it if it made 3D models (meshes), but as I understand, it doesn't.
How practical is a photograph? Well, now we have 3D photographs. With the modern developments in virtual reality in particular I think we will find ourselves able to experience "photos" much more immersively in the near future. It could change how films are shot. It could even be used in making truly photorealistic videogames. It's really up to the creative people of the world how far we can take advancements in media technology like this.
It makes a full 3D rendering of the scene, which gives computers the ability to understand 3D geometry from 2D inputs. Off the top of my head, this gives a few immediate use cases:
1. It allows you to take a few photos of a location, then set up and experiment with virtual cameras in the scene in post, which could be invaluable for video/film creation. Especially to get shots that would be impractical or even impossible to do with physical cameras.
2. LERFs embed semantic information into a NeRF scene, but I think a similar technique may work for Splatting as well. This allows natural-language queries to return segmentation within the 3D scene. To put that into perspective, if we could do this in real time (currently, LERFs take 45 minutes to generate a single scene, but I don't know how much faster it'd be applied to a Splat scene), a robot could use a single camera, take a brief look around its environment, and in real-time it could understand not only the full 3D geometry around it (for things like motion planning, etc.) but also the semantic nature of *everything around it.* So imagine telling a robot that's never seen your living room before "please clean my silver-framed movie poster", and after just a few seconds of looking around the room, it immediately is able to navigate all your furniture and obstacles, make its way to the poster (which it also autonomously recognized the nature and location of), and start cleaning it. It's Rosie from the Jetsons levels of robotic comprehension and autonomy.
"What a time to be alive!"
these renders look so real
If the model creates a 3D structure from a few images or a video, surely it won't have enough information about all the details in the scene? What happens then? Does it make up details?
Are you saying you're making real life, faster than real life?!
@@rst7190 No I live in my head with the voices
@@beri4138relatable
@@beri4138 they won't stop
More videos on papers like this please doc. ✍🏽
I think this will be great for things like using VR in Google Maps Street View but I don't think it will be applied to things like gaming or movies.
Is the voice AI generated in this video? It sounds really unnatural, I'll use subtitles instead.
Is the dude who talks an ai cause his voice has a pattern
This is AI voice
This is a game changer
Now that (!) is one of the times to be alive.
San Francisco and LA are so dangerouns now... you'd be far safe driving though the streets virtually than irl.