Oh whoa! Curtis! Dang, that's a hefty shout-out, thank you so much! Also hot damn this is cool. With the LIDAR and facial mocap tracking/scanning and all that wild stuff, it seems like an iPhone is totally the way to go if you're into a virtual-productiony workflow. Also had no idea about the clone tool in texture paint! That's going to help cleanup an absurd amount. Personally I can't wait to see what happens once machine learning gets more involved in the LIDAR/photogrammetry workflow (is it already??)- it seems like stuff'll just get insane at that point.
Totally! I'm really looking forward to seeing where this will go. And honestly, I'm starting to feel like a kid in a candy store having all this stuff to play with. There are so many cool things to make, but so little time 😊
There are some papers floating around demonstrating AI generated photogrammetry, albeit mostly for a lot more niche purposes than virtual production, though it is very promising. Especially if they were to start using stuff like LIDAR for training data. It just takes someone with the right project idea in mind. You should check out Two Minute Papers if you haven't already, they cover a lot of awesome stuff relating to AI use in CG as well as other fields and technology.
Underrated? Ian literally has about 4.5 times more subscribers than this channel. Like blender gru is probably only blender tutorial channel who has more subscribers than Ian.
Plus super handy for vfx work, rough scan of room + 3d camera track saves a lot of time, placing reference walls etc. The scanned room could even be used for better reflections than a hdri in the middel of the room.
"Light + Radar" and "Light Detection and Ranging" are basically the same, since Radar stands for "radio detection and ranging". But the second is more accurate, since there are no radiowaves involved in Lidar
This technology could've been and was implemented over ten years ago (Kinects, Realsense). What it needs is AI denoising and aligning, current algorithms are probably the best at what they do, so unless that happens you won't see it much better in the future, the point clouds will be just more dense but only 2cm accurate at 1m.
@@HerbaMachina What is meant by wavelength they're using. Doesn't this just virtualise the geometry within the device and call it a day, how come by increasing the speed of which the device can capture said scans as well as by increasing the subdivisions of the geometry not simply improve the tech to the point its just the norm to use it for real world objects (obviously clean up would likely be needed, but how come this isn't a big thing yet they're trying to advance?)
@@HerbaMachina No, this is not limited by the wavelength. This uses LIDAR, not RADAR, and LIDAR uses a laser in either red wavelength or infrared. Either way, we're talking a wavelength in the hundreds of nanometers, so something like 2 to 5 micrometers. The wavelength is not what's limiting this technique.
I've always wanted to combine high quality lidar with high quality photogrammetry, best of both worlds. That would be for film/game close up work though
I tried doing this and it worked in limited scenarios, most of the time the scanned mesh is too different from photogrammetry pointclod that the textures look warped.
This is awesome. I'd digitize my whole town. Imagine technology like this in 10 years, being able to simply 3d map a building like a grocery store inside and out by simply filming it like a video. and then you drop the file into any game creation program and it automatically makes it so you can play it like you're playing a first person shooter game.
Damn this is cool. Iphone seems to have a lot of interesting features for 3D, also for facial capture in combination with mocap suits... great video man!
I’m a gearhead, and I’m really enthusiastic about LIDAR. Imagine seeing a cool car and being able to make a 3D model of it, instead of making a 2D picture!
This seems really important. I want to play around with point clouds and I am keeping my eye out for a second hand xbox kinect - which a quick Google search suggests would work. Now, I would think that LIDAR would work for this. The issues holding me back would be the resolution (your point about small items) and that I have an android phone. Thanks!
I've tried it, results can be a little underwhelming when using an original Xbox 360 kinect. Maybe try the Xbox One kinect. Using software like Skanect gave me some useful results, but little details get ruined like LIDAR
Curtis, regarding the UV issue, that's one example where PTEX could be of large benefit. Perhaps you could touch on what potentially PTEX can bring to the table, seeing as how it has moved to the background. It has the potential of being a real gamechanger as it doesn't need UVs to map and paint.
Honestly a major reason I bought an iPhone 12 Pro over another phone was for the LiDAR Scanner, you did a great job of highlighting just how useful these kinds of tools can be for environment design
That’s a great video. LIDAR looks awesome and you can quickly get a ton of background environment material quickly. I’ve seen this on $100K cameras, but seeing it on a f🤘🤘🧟♀️ing iPhone... dayummmm!
Imagine using swarms of small drones with lidars and cameras to scan the entire Earth from every angle! It could become the most interesting open world game/driving/flight simulator ever! Or even just crowdsourcing 3d-scans from enthusiasts with the latest smartphones. Then developers and AI can adjust things to make it super realistic.
thanks for your great review! I'm also planning to test iPhone 12's "lidar" myself so this is a great reference. But I don't think that at this point "lidar" can replace photogrammetry yet, judging from the results in your video. The big advantage of photogrammetry is the way we can manipulate (to a certain degree) the end resolution to our needs depending on the camera-to-object distance, and yeah costs matter too. PS. I put "lidar" in quotes because as far as I can see despite the name Apple's system does not seem to be a real lidar/laser scanning per se, more like a depth-camera system. An upgraded Kinect if you will. CMIIW PPS. another great alternative is videogrammetry/photogrammetry from videos!
The biggest benefit I see for this, in filmmaking anyway, is getting a live accurate model of your set or location for both accurate compositing with live action footage, and reflection maps. Like, holy crap, this is invaluable.
@@avnzx5177 I did see a video back in 2015 about a camera that did that. It could do live, depth accurate effects like post-production refocusing, Z-depth based color correction, add fog I think, and was able to composite backgrounds and other elements without the use of green screen because it knew where the actors or subjects were vs their environment. Is that what you're referring to? Cause I actually haven't heard much about that technology since that video.
Just imagine the applications in vr. You could make photorealistic videogames with explorable environments just by sending a few drones around to film stuff. Microsoft flight sim, this - the only problem is its not perfect but the framework is there for Cyberspace
For the latest Huawei devices, there is an app, called "3d Scanner". The free version works on AR core enabled phones, but without the lidar sensor. Latest Huawei and Honor phones with lidar make really nice scans with the pro version, which is updated every few months. Checkout the developers name: Lubosh and match that with 3d and you can see his clips on youtube
This is indeed very interesting! We have even launched a business based on lidar. Considering the huge price drop and size reduction of lidar hardware we this it’s actually not expensive anymore and very accessible. Thanks for sharing🙏🏻
Thanks for this! Just got this phone and I’m well into a project making maps for a game/app and this is (potentially) gonna make building custom assets so much easier. Thanks for saving me some time with failed experiments lol
You missed a key difference between LASER-based LIDAR (see the top of google maps cars ) and image processing based on stereoscopy and variants (leveraging multiple views of the same scene). Apart from that, thanks for the video, nice sharing your work!
been 3d camera tracking scenes. and on site use the llidar. then after tracking it, I use the models in the camera tracked scene for particle systems with the lidar scans as guides for effectors or collision objects. works beautifully
So cool! Would love to see a meshroom photogrammetry video from you sometime... And What that pipeline looks like compared to the Lidar pipeline.. How you decimate and retopo in meshroom etc. ... lidar seems interesting because it looks like a continuous stream of collection whereas photogrametry is multiple photos that create a point cloud, I would love to find out if mushroom calculates GPS data as well?... in which case it would make sense to use a camera that has GPS in metadata for mushroom to incorporate that? Lots more to learn 😂 thank you so much for your contributions. Keep up the great work.
I have seen these similar devices on jobsites years ago. Last time, a company was getting 3D measurements of a framed staircase, so they can use that information to cut the marble pieces to be installed at a later time.
The mesh quality limitations probably stem from it's real-time roots, if you were able to capture pictures with LIDAR data, which can have higher resolution and detail than video, and then instead process that on a desktop application, it'd open up the opportunity to use the mesh data with the parallax data generated from standard photogrammetry. The LIDAR sensor probably has a much higher resolution than the phone can process in real time as well.
if you want the LiDAR scanner make sure it’s the 12 Pro or Pro Max. There’s also Android phones with ToF sensors if want the tech-ish but don’t want iOs, but there seems to be nothing around in terms of phones that beats Apple’s implementation of the technology. I may be wrong, of course, but I genuinely doubt that.
well yes but no you still can do this without a proper lidar but the process will be far more difficult and the result won't be as clean it's called photogrammetry, and there is free apps you can use to do it there is also a lot of tutorial for it, so I don't need you to go into too much detail but a fair warning, it takes much more time and effort
@@aronseptianto8142 with photogrammetry you will find out where your holes are way too late. realtime scanning is a big step forward. seeing what you get has tremendous value
Our geology teacher (studying Civil Engineering degree) used a LIDAR scanner (proper stationary tool rather than a smartphone toy) to scan an entire cliff (about 15m high). Really incredible stuff to be able to manimulate something from real life on your desktop. Perhaps using a dolly or some stabilisation handle would give better end results with a phone as well. Hopefully LIDAR will become more common, though bet people rather would have something dumb like yet another camera ¬_¬
@@sircher943 this. I was shocked that a pc is equally fast at making a scene on the lowest settings, than arcore apps. And so much more accuracy, control I'm using metashape
Had terrible results with lidar apps, apparently it’s software locked, I get overlapped textures and the application locks up often.. but you say it’s usable?
@@hugoantunesartwithblender not sure if there was a tutorial with the app.. ..people on 3d scan group called this a toy really :/ .. but I’m having hopes of software improving.. I was really asking for best apps - apparently all have same issues.. But we can see from the video, you can use some of it.. And mine reasoning is the same - I can do photogrammetry - but requires so many steps - just copy pasting a bunch of photos is enough work..
Lidar isn't a toy, it's well-proven technology across all kinds of industries, from industrial design and geological study to archaeology and, damn, even NASA have been using it. Software design seems to be the most limiting factor at the moment. Using just photo or lidar data to generate a mesh is a mistake I think, perhaps using data from both to inform more accurate decisions about surfaces would be better. Having an effective/efficient way of doing that live is tricky.
@@CurtisHolt the iphone lidar is a toy, not even lidar - that’s what the 3d scan people who used the expensive lidar claimed, accuracy is very low apparently - not my claims - but quality is pretty low - and apparently apple locked the resolution so that might be it, there might be better solutions to come
@@robob3ar Like I said, software is still a limiting factor. Realtime photogrammetry is possible, using Lidar data even at a low resolution should make it better, but it seems choices have been made to exclusively use low resolution Lidar data to construct meshes. Why not both? People think that just because something isn't a hyper-expensive top of the range specialised tool that it's useless. It's not.
Very cool to see the 'adaptive refinement' of the mesh if you move around and cover an previously 'undiscovered' area with the camera And yes @IanHubert is great 👍🏼
This thing will be next most wanted feature for every phone. Precision is acceptable and fast. Main problem for photogrammetry is precision. Precision require more processing power which implies more time. In many situation we have not time. When phones become enough powerful then we will combine both to achieve better quality.
This sounds incredibly promising. However recent automotive developments in Autonomous Vehicle (AV) driving has resulted in AV effective LIDAR units that cost about $100USD a piece. One could see such a Handheld LIDAR Scanner that can use almost any Smart phone, or as an accessory to a compatible Video Camera, with much more accuracy and denser scans just as easily as the system you are using now.
I can see the use of Lidar for static scenes like what is shown here but I don't see a whole lot of use cases for it outside of that such as in games. The cleanup work seems like it would be much more complicated, and the texture resolution seems to be lacking from what I've seen as well as the accuracy of its simplistic geometry generation. It is however very fast and easy to do from what I can tell. I look forward to seeing how it improves over time.
Years from now there will be a LiDAR sensor in every iPhone model not just the “Pros” it has the same effect as the HoloLens lots of applications but lots of development is still necessary. However anyone jumping into this subject matter early will surely benefit. Imagine starting a virtual real estate agency where you have a bunch of agents out there scanning homes and selling them around the world giving virtual tours all from your phone!
I would totally make copies of things and put them right next to the object. My coworkers would discover replicants of their action figures and plants one day.
I can't wait to see what happens once this stuff takes off in the hobby community. If we could get lidar, photogrammetry, high resolution mesh and texture generating, as well as somehow generating an hdri for the world space(s), this would be an amazing tool. Though, I'm curious what the hardware limitation of various phones with lidar are, compared to a dedicated tool with higher quality components.
amazing technology, only going to get more impressive VERY quickly. at the moment, just looking at some of those wonky textures, I'll stick with photoscans where this doesn't really happen. can't wait for *this* way to improve though
I wonder if they've improved the Lidar this year. I've watched a number of videos showing the iPhone 12 Lidar and quite a few are waiting for it to improve.
Perhaps it would be good to capture and clean-up scene elements like the path and the fence and the grass separately and then compose them in a blender scene. That way you can really concentrate on capturing each part well, shooting from the best angles. It would also let you rescan a part later if you found a problem you couldn't fix in clean-up.
The tools are getting insane already... but I'm not sure photogrammetry will disappear, outside of its commercial applications. The advantage of an SLR/DSLR/mirrorless camera, if you already have one along with the strobe lighting gear, is that you can light the scene/object in more flexible ways. I guess you can't beat the convenience of a mobile phone you already carry with you though - it's made me think about upgrading my iPhone 7 Plus (with a bit of man maths thrown in)... thanks a bunch!
As smartphones get smarter, DSLR will need to capture up. As human beings, we tend to go for easy, and when it counts, our phone is always with us and ready to use. What was once a select profession is now an average Joe can create or exceed someone with a “professional” camera/equipment.
@@Mangolite I don't think a smartphone is better than a dedicated camera. The thing is that a new camera costs much more than the gain in quality is worth it when an average Joe tries to use it. Still it is quite possible to get better results with a proper camera even by a Joe.
@@Squee7e As someone who sells stock photography I agree, you will never get the same results with a smart phone. The control you have over aperture, ISO, shutter speed and the vast selection of lens that are available. Also DSLR lens are usually made of glass which is vastly superior to any plastic lens on a smart phone. Plastic tends to distort images.
@@Squee7e As I stated, as a smartphone gets smarter, the line between a high-end camera and a smartphone is blurring. Technology constantly changes. We came from film to digital, and digital at first was not welcomed, but just look how far digital has surpassed even the best film stock available. A smartphone is smaller and can go where a DLSR may not be able to, especially when it counts.
i do get it for a very quick result to use in background or whatever, but the topology, artefacts, stretching, warping and overall resolution looks so bad that I don't think it's usable for anything with quality. maybe use the lidar for a very dirty base or 3d reference, but I don't think the technology is very useful for now. In a few years this might be totally different though :)
This is actually quite interesting really. Could use this to add so much to a game. Probably how they got some of the items into prop hunt in garry's mod.
I wouldn't even begin to theorize how one would patch holes with proper textures. UV wouldn't be much easier. For this reason, I'll wait another 5 years and check this tech out then.
Making a scene with custom objects in Blender is time consuming, so now i scan the scene with Lidar and spend hours fixing the scene i have no control on.
On another note oculus quest 2 has some sort od 3d scan built in, it can detect objects blocking your enviro.. I'm certain this all could be combined with just software into something very usable, this is sort of 1st gen, still needs refinement, but at some point people will take 3d photos like we do ordinary ones
Such a cool idea. I guess that's technically Augmented Reality, but you can imagine walking around your house with a headset on and the rooms and objects are all digitally viewable and interactable but maybe with different textures or filters on them, or even highlights or graphics overlaid onto the image. Arrows pointing you around your house to where you left your keys, or virtually instructing you on a recipe you're cooking.
Isn't this photogrammetry in a way though? As "LiDAR Scans" create point cloud not meshes. This seem to me to be a mesh created from a mix between camera and LiDAR tech. AKA likely relying on the camera to create surfaces and the LiDAR to determine distance to vertices. Moreover I don't know what I think about dynamic scans (video in favor of photos) as the result a lot of the time seem to be warped or in other ways disfigured more then the traditional slower method. With that said I'm not an expert in the area and it is an interesting emerging mix, the fusion between moblie photogrammetry and LiDAR.
This looks interesting, I know nothing about 3D modeling or blender but I have heard how blender can help with comics. Which is something I'm interested in.
Any app recommended for Android? This looks awesome. Can't wait for this tech to mature enough to give good result on budget phones and not needing to buy expensive lidar scanners.
I forgot I had a black moustache lol.
great vid
Lol
I bet it was cgi
He & his mustachio were being CGMatter, I was being me.
lmfao..
Now buy a iphone cg
Oh whoa! Curtis! Dang, that's a hefty shout-out, thank you so much! Also hot damn this is cool. With the LIDAR and facial mocap tracking/scanning and all that wild stuff, it seems like an iPhone is totally the way to go if you're into a virtual-productiony workflow. Also had no idea about the clone tool in texture paint! That's going to help cleanup an absurd amount.
Personally I can't wait to see what happens once machine learning gets more involved in the LIDAR/photogrammetry workflow (is it already??)- it seems like stuff'll just get insane at that point.
Totally! I'm really looking forward to seeing where this will go. And honestly, I'm starting to feel like a kid in a candy store having all this stuff to play with. There are so many cool things to make, but so little time 😊
There are some papers floating around demonstrating AI generated photogrammetry, albeit mostly for a lot more niche purposes than virtual production, though it is very promising. Especially if they were to start using stuff like LIDAR for training data. It just takes someone with the right project idea in mind.
You should check out Two Minute Papers if you haven't already, they cover a lot of awesome stuff relating to AI use in CG as well as other fields and technology.
Give it a couple of months and all newer android phones even the budget ones will have it
Way to go? Android has had the same shit for years.
I also wanted to say that the clone tool for texture painting was an incredible knowledge drop! Brilliant
“Export an OBJ and E-mail that to yourself...” lol I feel your pain, iPhone and Windows computer user!
your poor soul
Props to you for giving Ian Hubert a shout out, he's a seriously underrated blender genius
Underrated? Ian literally has about 4.5 times more subscribers than this channel. Like blender gru is probably only blender tutorial channel who has more subscribers than Ian.
Highly recommend his Ted Talk. Very entertaining and informative at the same time.
(Edit) lol he JUST mentioned it as I was watching. . . Touche'
Ian Hubert is among the most applauded vfx artists today lol
@@dirtiestharry6551 lmao facts and Ian's a god compared to that self claimed guru
Oh look, it's the "seriously underrated", meme comment.
Plus super handy for vfx work, rough scan of room + 3d camera track saves a lot of time, placing reference walls etc. The scanned room could even be used for better reflections than a hdri in the middel of the room.
Yes!^
Hadn’t considered hdri alternative!
Friendship with photogrammetry ended, now lidar is my best friend
Dying...
You'll have to wait for another 5 years...
FRIENDSHIP END WITH PHOTOGRAMMETRY, NOW LIDAR IS MY BESTFRIEND
Careful now with ending friendships! I’m sure you want textures to go with your geometry :p
LiDAR is one f the tools used in photogrametry...
I've been making sure to grab photos as I grab LIDAR models, which allows you to project them back on to really messed up areas, texture-wise.
"Light + Radar" and "Light Detection and Ranging" are basically the same, since Radar stands for "radio detection and ranging". But the second is more accurate, since there are no radiowaves involved in Lidar
Measuring the architecture and composing an environment from photos is in the past! Man this evolves so fast!
God this is insane, this technology is just going to keep advancing and advancing, imagine this in 10 years
This technology could've been and was implemented over ten years ago (Kinects, Realsense). What it needs is AI denoising and aligning, current algorithms are probably the best at what they do, so unless that happens you won't see it much better in the future, the point clouds will be just more dense but only 2cm accurate at 1m.
@@sircher943 yeah this, as the image is ultimately limited by the wavelength they're using.
yeah, and it's all going to be used for advertising :D
@@HerbaMachina What is meant by wavelength they're using. Doesn't this just virtualise the geometry within the device and call it a day, how come by increasing the speed of which the device can capture said scans as well as by increasing the subdivisions of the geometry not simply improve the tech to the point its just the norm to use it for real world objects (obviously clean up would likely be needed, but how come this isn't a big thing yet they're trying to advance?)
@@HerbaMachina No, this is not limited by the wavelength. This uses LIDAR, not RADAR, and LIDAR uses a laser in either red wavelength or infrared. Either way, we're talking a wavelength in the hundreds of nanometers, so something like 2 to 5 micrometers. The wavelength is not what's limiting this technique.
More like the beginning of photogrammetry. Photogrammetry neural nets trained with lidar data will be scary good in the future.
I've always wanted to combine high quality lidar with high quality photogrammetry, best of both worlds. That would be for film/game close up work though
I tried doing this and it worked in limited scenarios, most of the time the scanned mesh is too different from photogrammetry pointclod that the textures look warped.
I would also love an AI solver that approximates materials beyond albedo, with roughness, and specular, maybe even metalness.
Thank you for this video, incredibly helpful, the LIDAR is much better than I expected on the iPhone 12 Pro. Thanks!
In a couple of years a bunch of Indie filmmakers will be able to take on the blockbuster market. I can’t wait to see that!
This is awesome. I'd digitize my whole town. Imagine technology like this in 10 years, being able to simply 3d map a building like a grocery store inside and out by simply filming it like a video. and then you drop the file into any game creation program and it automatically makes it so you can play it like you're playing a first person shooter game.
Damn this is cool. Iphone seems to have a lot of interesting features for 3D, also for facial capture in combination with mocap suits... great video man!
I’m a gearhead, and I’m really enthusiastic about LIDAR. Imagine seeing a cool car and being able to make a 3D model of it, instead of making a 2D picture!
Totally.
Or walk around it while recording a video for 10 secs? 😂
Thank you for introducing this. By the way Happy New Year! Keep Blending.
Great video and new information about working with my iPhone 12Pro and Blender.
This seems really important.
I want to play around with point clouds and I am keeping my eye out for a second hand xbox kinect - which a quick Google search suggests would work.
Now, I would think that LIDAR would work for this. The issues holding me back would be the resolution (your point about small items) and that I have an android phone.
Thanks!
I've tried it, results can be a little underwhelming when using an original Xbox 360 kinect. Maybe try the Xbox One kinect. Using software like Skanect gave me some useful results, but little details get ruined like LIDAR
Interesting!!
Looking forward to your next works involving Lidar 👀👀
@Curtis Holt -- Thanks for sharing Ian's Blender Talk. I loved it.
"I need your clothes, your boots and your motorcycle."
You forgot to say please
the app needs a terminator vision mode
*scans them with phone then politely gives items back*
THANK YOU
Modozycle!
@@flavienvolken3733 😂😂
Just got the iPhone 12 pro and forgot it had Lidar capabilities this is really useful to me and I didn’t even know about it
4:15 “because datar is data” lol. Love those ransom R’s in words
Curtis, regarding the UV issue, that's one example where PTEX could be of large benefit. Perhaps you could touch on what potentially PTEX can bring to the table, seeing as how it has moved to the background. It has the potential of being a real gamechanger as it doesn't need UVs to map and paint.
Honestly a major reason I bought an iPhone 12 Pro over another phone was for the LiDAR Scanner, you did a great job of highlighting just how useful these kinds of tools can be for environment design
That’s a great video. LIDAR looks awesome and you can quickly get a ton of background environment material quickly. I’ve seen this on $100K cameras, but seeing it on a f🤘🤘🧟♀️ing iPhone... dayummmm!
Imagine using swarms of small drones with lidars and cameras to scan the
entire Earth from every angle! It could become the most interesting
open world game/driving/flight simulator ever! Or even just
crowdsourcing 3d-scans from enthusiasts with the latest smartphones.
Then developers and AI can adjust things to make it super realistic.
Its coming
Have you ever tryed Microsoft flight simulator 2020?
It might blow your mind.
I literally got the iPhone 12 pro Max because of this. Plus other good things that come with this.
been using polycam a lot with blender and its awesome. learnt some cool tips here so thank you :)
thanks for your great review! I'm also planning to test iPhone 12's "lidar" myself so this is a great reference. But I don't think that at this point "lidar" can replace photogrammetry yet, judging from the results in your video. The big advantage of photogrammetry is the way we can manipulate (to a certain degree) the end resolution to our needs depending on the camera-to-object distance, and yeah costs matter too.
PS. I put "lidar" in quotes because as far as I can see despite the name Apple's system does not seem to be a real lidar/laser scanning per se, more like a depth-camera system. An upgraded Kinect if you will. CMIIW
PPS. another great alternative is videogrammetry/photogrammetry from videos!
The biggest benefit I see for this, in filmmaking anyway, is getting a live accurate model of your set or location for both accurate compositing with live action footage, and reflection maps. Like, holy crap, this is invaluable.
RGBD cameras have existed for a good while.
The D being depth
@@avnzx5177 I did see a video back in 2015 about a camera that did that. It could do live, depth accurate effects like post-production refocusing, Z-depth based color correction, add fog I think, and was able to composite backgrounds and other elements without the use of green screen because it knew where the actors or subjects were vs their environment.
Is that what you're referring to? Cause I actually haven't heard much about that technology since that video.
that UFO looks so great!
Just imagine the applications in vr. You could make photorealistic videogames with explorable environments just by sending a few drones around to film stuff. Microsoft flight sim, this - the only problem is its not perfect but the framework is there for Cyberspace
For the latest Huawei devices, there is an app, called "3d Scanner". The free version works on AR core enabled phones, but without the lidar sensor. Latest Huawei and Honor phones with lidar make really nice scans with the pro version, which is updated every few months. Checkout the developers name: Lubosh and match that with 3d and you can see his clips on youtube
This is indeed very interesting! We have even launched a business based on lidar.
Considering the huge price drop and size reduction of lidar hardware we this it’s actually not expensive anymore and very accessible. Thanks for sharing🙏🏻
Thanks for this! Just got this phone and I’m well into a project making maps for a game/app and this is (potentially) gonna make building custom assets so much easier. Thanks for saving me some time with failed experiments lol
How is it going?
You missed a key difference between LASER-based LIDAR (see the top of google maps cars ) and image processing based on stereoscopy and variants (leveraging multiple views of the same scene). Apart from that, thanks for the video, nice sharing your work!
I have been hoping for this tech for 20 years ! Can’t wait for it to evolve so I can do hyper accurate scans for engineering purposes
Laser scanning is already a thing
You can already do this for years it’s just the technology being integrated into smaller and cheaper devices
It's not that expensive anymore, for 1k usd you can get a HP SLS-2 which will let you scan any mechanical parts with great accuracy and resolution.
@@sircher943 I knew the tech has been around but didnt realize the price had dropped so far. thanks for the advice
@@jittertn no duh..... the comment is related to the lidar in the phone genius.
been 3d camera tracking scenes. and on site use the llidar. then after tracking it, I use the models in the camera tracked scene for particle systems with the lidar scans as guides for effectors or collision objects. works beautifully
So cool! Would love to see a meshroom photogrammetry video from you sometime... And What that pipeline looks like compared to the Lidar pipeline.. How you decimate and retopo in meshroom etc. ... lidar seems interesting because it looks like a continuous stream of collection whereas photogrametry is multiple photos that create a point cloud, I would love to find out if mushroom calculates GPS data as well?... in which case it would make sense to use a camera that has GPS in metadata for mushroom to incorporate that? Lots more to learn 😂 thank you so much for your contributions. Keep up the great work.
I have seen these similar devices on jobsites years ago. Last time, a company was getting 3D measurements of a framed staircase, so they can use that information to cut the marble pieces to be installed at a later time.
This could be a game changer in the sim racing industry for the laser scanned racin tracks.... may be I dont know
This will all be obsolete when someone feeds all these scans into a neural network.
The mesh quality limitations probably stem from it's real-time roots, if you were able to capture pictures with LIDAR data, which can have higher resolution and detail than video, and then instead process that on a desktop application, it'd open up the opportunity to use the mesh data with the parallax data generated from standard photogrammetry. The LIDAR sensor probably has a much higher resolution than the phone can process in real time as well.
The main issue is that the lidar sensor's resolution is extremely low at least on iphones, and the dots are very spaced apart
wow this is amazing! this makes me wanna buy an iPhone 12 :D
it's pretty cool but you don't need an iPhone to use Lidar
if you want the LiDAR scanner make sure it’s the 12 Pro or Pro Max. There’s also Android phones with ToF sensors if want the tech-ish but don’t want iOs, but there seems to be nothing around in terms of phones that beats Apple’s implementation of the technology. I may be wrong, of course, but I genuinely doubt that.
Exciting. One more thing to look for the next time I buy a phone.
Me: no way, after display.land got shut down, I have needed something like this.
Sees that an IPhone 12 is required...
:(
:(
I got my 12 pro max yesterday 🤭
@@manavgala2361 nice
well yes but no
you still can do this without a proper lidar
but the process will be far more difficult and the result won't be as clean
it's called photogrammetry, and there is free apps you can use to do it
there is also a lot of tutorial for it, so I don't need you to go into too much detail
but a fair warning, it takes much more time and effort
@@aronseptianto8142 with photogrammetry you will find out where your holes are way too late. realtime scanning is a big step forward. seeing what you get has tremendous value
god damn it i just got into photoscanning
Our geology teacher (studying Civil Engineering degree) used a LIDAR scanner (proper stationary tool rather than a smartphone toy) to scan an entire cliff (about 15m high). Really incredible stuff to be able to manimulate something from real life on your desktop. Perhaps using a dolly or some stabilisation handle would give better end results with a phone as well.
Hopefully LIDAR will become more common, though bet people rather would have something dumb like yet another camera ¬_¬
I chose to switch to an iPhone at the right time, this is one of my favorite features of any phone I've ever owned.
i don't even use blender but man this is fascinating to watch
I'm still sad display.land is gone. That thing was very useful.
trnio is a pretty good alternative for objects (not so much for scenes)
Just use some pc software, they give you more control and the results are much better.
@@sircher943 this. I was shocked that a pc is equally fast at making a scene on the lowest settings, than arcore apps.
And so much more accuracy, control
I'm using metashape
@@sircher943 I've used pc software before. Display land was much much faster for stuff that doesn't need to be perfect. They both have their use.
Wait what?
Had terrible results with lidar apps, apparently it’s software locked, I get overlapped textures and the application locks up often..
but you say it’s usable?
Remember of filming really slow
@@hugoantunesartwithblender not sure if there was a tutorial with the app..
..people on 3d scan group called this a toy really :/ .. but I’m having hopes of software improving.. I was really asking for best apps - apparently all have same issues..
But we can see from the video, you can use some of it..
And mine reasoning is the same - I can do photogrammetry - but requires so many steps - just copy pasting a bunch of photos is enough work..
Lidar isn't a toy, it's well-proven technology across all kinds of industries, from industrial design and geological study to archaeology and, damn, even NASA have been using it. Software design seems to be the most limiting factor at the moment. Using just photo or lidar data to generate a mesh is a mistake I think, perhaps using data from both to inform more accurate decisions about surfaces would be better. Having an effective/efficient way of doing that live is tricky.
@@CurtisHolt the iphone lidar is a toy, not even lidar - that’s what the 3d scan people who used the expensive lidar claimed, accuracy is very low apparently - not my claims - but quality is pretty low - and apparently apple locked the resolution so that might be it, there might be better solutions to come
@@robob3ar Like I said, software is still a limiting factor. Realtime photogrammetry is possible, using Lidar data even at a low resolution should make it better, but it seems choices have been made to exclusively use low resolution Lidar data to construct meshes. Why not both? People think that just because something isn't a hyper-expensive top of the range specialised tool that it's useless.
It's not.
Very cool to see the 'adaptive refinement' of the mesh if you move around and cover an previously 'undiscovered' area with the camera
And yes @IanHubert is great 👍🏼
Hi There
Love to know more about scanning the outside and inside of houses for renovation and extension plans please
This thing will be next most wanted feature for every phone.
Precision is acceptable and fast.
Main problem for photogrammetry is precision.
Precision require more processing power which implies more time.
In many situation we have not time.
When phones become enough powerful then we will combine both to achieve better quality.
Good thing this somehow develops. I can relate this one to 3D capture on Sony Xperia XZ1. I guess XZ Premium was also updated to capture 3D.
This sounds incredibly promising. However recent automotive developments in Autonomous Vehicle (AV) driving has resulted in AV effective LIDAR units that cost about $100USD a piece. One could see such a Handheld LIDAR Scanner that can use almost any Smart phone, or as an accessory to a compatible Video Camera, with much more accuracy and denser scans just as easily as the system you are using now.
I can see the use of Lidar for static scenes like what is shown here but I don't see a whole lot of use cases for it outside of that such as in games. The cleanup work seems like it would be much more complicated, and the texture resolution seems to be lacking from what I've seen as well as the accuracy of its simplistic geometry generation. It is however very fast and easy to do from what I can tell.
I look forward to seeing how it improves over time.
Years from now there will be a LiDAR sensor in every iPhone model not just the “Pros” it has the same effect as the HoloLens lots of applications but lots of development is still necessary. However anyone jumping into this subject matter early will surely benefit. Imagine starting a virtual real estate agency where you have a bunch of agents out there scanning homes and selling them around the world giving virtual tours all from your phone!
This algo has a lot of room for improvement, it should just resample textures once it resolved a smoother version of the mesh somewhere
Imagine using this to scan objects irl objects and exporting them to a CAD software to be cleaned up and 3D printed as a replacement backup
I would totally make copies of things and put them right next to the object. My coworkers would discover replicants of their action figures and plants one day.
Really cool stuff. I looked up the particular phone you have and it's a little eye watering for me haha
I can't wait to see what happens once this stuff takes off in the hobby community. If we could get lidar, photogrammetry, high resolution mesh and texture generating, as well as somehow generating an hdri for the world space(s), this would be an amazing tool. Though, I'm curious what the hardware limitation of various phones with lidar are, compared to a dedicated tool with higher quality components.
This creates a lot of new possible scenarios! Interesting video!
This is one of the most impressive things I've seen in a long time
You haven't seen much, have you?
this is amazing but the meshes look wonky in places that can be a problem with further modelling / texturing... thoughts?
Just iphone? Is there some special about iPhone or
Wow! !!
I wasn't interested in iphone12 at all, but I started to want to buy it a little...
Thanks for sharing
amazing technology, only going to get more impressive VERY quickly. at the moment, just looking at some of those wonky textures, I'll stick with photoscans where this doesn't really happen. can't wait for *this* way to improve though
Is there an alternative that is Android compatible? Great video
I wonder if they've improved the Lidar this year. I've watched a number of videos showing the iPhone 12 Lidar and quite a few are waiting for it to improve.
Blender + LIDAR + AI + Cerewave Natural speech engine = Unbelievable explosion in creativity at the individual level.
Perhaps it would be good to capture and clean-up scene elements like the path and the fence and the grass separately and then compose them in a blender scene. That way you can really concentrate on capturing each part well, shooting from the best angles. It would also let you rescan a part later if you found a problem you couldn't fix in clean-up.
Wow that's amazing, didn't know this kind of tech was out there. Am so going to try this
The tools are getting insane already... but I'm not sure photogrammetry will disappear, outside of its commercial applications. The advantage of an SLR/DSLR/mirrorless camera, if you already have one along with the strobe lighting gear, is that you can light the scene/object in more flexible ways. I guess you can't beat the convenience of a mobile phone you already carry with you though - it's made me think about upgrading my iPhone 7 Plus (with a bit of man maths thrown in)... thanks a bunch!
As a photographer I would prefer to use one of my Nikon DSLR cameras over a phone.
I'd love for my dslr to be able to do this, but I'd prefer lidar over photogrammetry until I can do something realtime with my dslr
As smartphones get smarter, DSLR will need to capture up. As human beings, we tend to go for easy, and when it counts, our phone is always with us and ready to use. What was once a select profession is now an average Joe can create or exceed someone with a “professional” camera/equipment.
@@Mangolite I don't think a smartphone is better than a dedicated camera. The thing is that a new camera costs much more than the gain in quality is worth it when an average Joe tries to use it. Still it is quite possible to get better results with a proper camera even by a Joe.
@@Squee7e As someone who sells stock photography I agree, you will never get the same results with a smart phone. The control you have over aperture, ISO, shutter speed and the vast selection of lens that are available. Also DSLR lens are usually made of glass which is vastly superior to any plastic lens on a smart phone. Plastic tends to distort images.
@@Squee7e As I stated, as a smartphone gets smarter, the line between a high-end camera and a smartphone is blurring. Technology constantly changes. We came from film to digital, and digital at first was not welcomed, but just look how far digital has surpassed even the best film stock available. A smartphone is smaller and can go where a DLSR may not be able to, especially when it counts.
i do get it for a very quick result to use in background or whatever, but the topology, artefacts, stretching, warping and overall resolution looks so bad that I don't think it's usable for anything with quality.
maybe use the lidar for a very dirty base or 3d reference, but I don't think the technology is very useful for now. In a few years this might be totally different though :)
Mind blown, really enjoyed this. Do some more videos on how lidar can be used
Maybe it is pretty useful to make not important background environment.
This kind of technology has been making great strides.
Oh this would save me SO much work!! Im in!
This is actually quite interesting really. Could use this to add so much to a game. Probably how they got some of the items into prop hunt in garry's mod.
Nice test thanks.
Scan resolution is very bad.
Unusable for professional purpose, but don't worry good cheap lidar scanner are coming this year.
I wouldn't even begin to theorize how one would patch holes with proper textures. UV wouldn't be much easier.
For this reason, I'll wait another 5 years and check this tech out then.
Didn't know this technology existed. Great Info
thanks for sharing man
Making a scene with custom objects in Blender is time consuming, so now i scan the scene with Lidar and spend hours fixing the scene i have no control on.
the thing is lidar is only in iphones and the latest ipad, I will stay with photogrammetry, well I don’t have a choice...
On another note oculus quest 2 has some sort od 3d scan built in, it can detect objects blocking your enviro.. I'm certain this all could be combined with just software into something very usable, this is sort of 1st gen, still needs refinement, but at some point people will take 3d photos like we do ordinary ones
My -graphic tablet- I use to watch RUclips can do what?
Thanks for showing something amazing!
Have a good day!
The music has a constant door bell interval that just drove me mad - thanks for sharing! :D
How long until vr headsets have this for mapping your environment and converting to map
Such a cool idea. I guess that's technically Augmented Reality, but you can imagine walking around your house with a headset on and the rooms and objects are all digitally viewable and interactable but maybe with different textures or filters on them, or even highlights or graphics overlaid onto the image. Arrows pointing you around your house to where you left your keys, or virtually instructing you on a recipe you're cooking.
Isn't this photogrammetry in a way though? As "LiDAR Scans" create point cloud not meshes. This seem to me to be a mesh created from a mix between camera and LiDAR tech. AKA likely relying on the camera to create surfaces and the LiDAR to determine distance to vertices. Moreover I don't know what I think about dynamic scans (video in favor of photos) as the result a lot of the time seem to be warped or in other ways disfigured more then the traditional slower method. With that said I'm not an expert in the area and it is an interesting emerging mix, the fusion between moblie photogrammetry and LiDAR.
Thumbs up for apple engineer. The crazy idea that soon every smartphone will follow. 👍
This looks interesting, I know nothing about 3D modeling or blender but I have heard how blender can help with comics. Which is something I'm interested in.
if youre wondering, yes , you can build or buy a lidar and make this work.
Any app recommended for Android? This looks awesome. Can't wait for this tech to mature enough to give good result on budget phones and not needing to buy expensive lidar scanners.
Not possible as there is need for special sensor on your device
@@pranaykumar6648 i would say its 100% confirmed as there is a big push for AR content on smartphones from companies like Apple and Google.
@@shayan-gg true but but sensors are expensive so wouldn't be available on budget smartphones.
I think we can see already that reconstruction from a camera will be possible. If we're lucky it will happen for the public soon.
@@danielloo1255 it is already possible, just not realtime, as curtis mentioned, google photogrammetry
I wonder if you could slap a LIDAR + camera onto a Raspberri Pi and download the app onto the Raspi.