💻 You can also find the demo shown on today’s video via GitHub from github.com/dilmerv/VisionOSObjectTrackingDemo I also recommend going over all the resources available from the video description.
Cool Video! I just tried to use a CAD model to train the reference object. However, it didn’t work. I am curious about how to config the usdz material for a CAD model if we do not want to use the scanned version which is not as accurate as the CAD😅😅
I don't believe CAD files are supported, Apple normally only supports USDZ files only. Take a look at this suggestion as it seems to may be an option: forums.developer.apple.com/forums/thread/745911
Very cool feature. I have noticed that there is another newer session called SpatialTrackingSession which can provide tracked object's world position, can the one you used also output some similar data?
i'd love to see a detailed video on Unity Polyspatial and how to use Dynamic Lights and Shadows on AVP builds. Its been a super headache, with no clear direction. Especially for bounded volume experiences.
@@dilmerv We finally got shadows to work, but litterally had to create an empty project, and use all the defaul settings, and the sample lighting scene, just to do a build test and see if it was even possible. From there, were were able to make small adjustments. We found we liked Hard shadows better than soft shadows.
Thank you, Dilmer. It's very easy to follow. Is it possible to combine this object-tracking feature with the Unity AR project? (Like Ar foundation etc.)
That’s a great question, I don’t believe it is available yet but I know the Unity team always stays up to date with moving native functionality to AR Foundation which in this case I agree it should be available with Unity PolySpatial. I will keep an eye on it and once they add support then will make a new video. Thanks for your comment!
@dilmerv thank you for your great job. I have a problem. Could you please help me? I want to attach a reality view with a usdz & an attachment(SwiftUI view) to the object tracking anchor how to do this?
Hello Dilmer! Thank you for the video :) I'm studying for VisionOS object tracking these days, and your video is being so much helpful. I already finished ML for an object and put the 'referanceobject' in the 'Referance Objcts' folder. But when I ran the project on Vision Pro, it was saying that, "Start Tracking 0 Object(s)". Am I missing some steps?
I remember having such problems when the app didn’t have the correct permissions. Did you get a popup when you launch the app to give the app permission for spatial understanding? Also thanks for your feedback on the video!
@@dilmerv Yes, I did! I solved the problem I guess. Thanks for your answer! Ah, one more questions please, is there ways making the tracking speed faster? Or need we wait for next visionOS, ARKit version updates?
I believe Apple updated it, here is the new link: developer.apple.com/documentation/realitykit/scanning-objects-using-object-capture but to be honest I need to test it, however feel free to check it out. Thanks for your feedback on the video!
I don’t believe there is a limit imposed but a device limitation in terms of performance. The files are quite large so if you were to add 10 or more I could see the binary be more than 1gb so that should give you an estimate of what’s possible.
Hey thanks for your comment. As of now yes, but I know Unity always tries to stay in parity with platforms features, so perhaps it will be added in the near future. I recommend posting in the Unity forums to see if that’s in the roadmap. Also, another way could be to add a native plugin, here is an example of one docs.unity3d.com/Packages/com.unity.polyspatial.visionos@2.0/manual/InteropWithSwiftUI.html
Does the tracking generalize well or is it specific to the physical object you scanned? Specifically I'm wondering if you can scan 1 car (compact sedan) but have the app work for all cars (compact sedans)?
That’s a great question, currently the reason it takes so long is mainly because the machine learning algorithm tries to track the exact object in question, and it is pretty good about it. I don’t believe you could do what you are asking unless Apple provides additional data, such as how similar it could be on some type of percentage through their SDK.
Do you happen to know if I update to the Beta if there are errors, is it possible to downgrade by factory resetting the device? Or will it factory reset back to the same beta?
Factory reset will reset it back to Beta with a clean version, you could download the current prod version from Apple dev portal and use Apple Configurator app to restore it back to the prod version. Great question!
I don’t believe that’s supported right now from the Unity side of things. But I could see that being integrated in the future within AR Foundation / object tracking features. For now, I would stick with native Xcode.
If the model is very very high quality & photorealistic then yes it should work. Also, right now I believe you can only train with USDZ files so if you convert it to that format then things should work.
Hi Dilmer First of all, thanks for your content ! Do you think that The Object Tracking will be accessible from Unity side similar to Vuforia model targets
Thanks for your feedback, and yes I am sure Unity will be making this available with PolySpatial just like they do add support for all ARKit features available for visionOS today.
Does it way more battery doing that as it constantly looks for the objects and needs to know their positions? I assume if you have to make it search for more than 2, it will drop the FPS heavily and be unusable then the app crashes?
Normally when I run CV processes on mobile phones I can tell that it is pushing its capabilities, but with this, the device performs so well, I didn’t feel like it was causing any kind of performance degradation, but I only did it with 2 objects. I will be testing this more to find out its limits, thanks for your feedback.
@@dilmerv yea, need to look for memory leaks too Being able to store that stuff on a cloud to use would be good. Would be cool if it activated certain things if within a geozone.
Based on what I read and saw in the code this is only for visionOS. ARKit for iOS provides object tracking and it came out a few years back, here is some info: developer.apple.com/documentation/arkit/arkit_in_ios/content_anchors/scanning_and_detecting_3d_objects Thanks and great question!
@@dilmerv Yeah i remember :) have used it for a few different projects as well. But the scanning process is extremely limiting and adds a lot of randomness to the result so am really hoping this new way become the iOS way as well :) thanks for your answer!
It seems like it should be possible to reduce tracking latency on stationary objects if you integrate the tracking box with input from the gyros and accelerometers. Just a thought. Anyways, thanks for the upload!
@@dilmervyour 3D models are pretty big in scale compared to the PS controller and the little figure or is the UI showing it wrong? ruclips.net/video/NaB_6hLzYN0/видео.html
They're about the same size and accurate, if you look at the measurements within Create ML, it shows the figure as 22 cm ish, that's the real size. The UI shows them big mainly because I zoomed in.
Is this anything even close to this in the Meta Quest 3 universe? I can't find a way to track real life objects. Do you know any solutions? Even with external hardware..
Hey thanks for your feedback, but this won't work with Meta devices. Meta recently released an open source project for computer vision with Meta devices called "ocean". I recommend taking a look at: facebookresearch.github.io/ocean/docs/overview/ and facebookresearch.github.io/ocean/docs/demoapps/questapps
Yes you could use polycam that’s correct, looks like they also support usdz files. I didn’t use them mainly to keep the workflow with Apple tools but you are more than right to have that as an option. Thanks for your comment.
Currently, this is mainly for visionOS but I remember that recently Meta implemented Apple’s spatial video, so perhaps they may leverage this tech in the near future. Great question!
The best thing this can do for the market is convince Meta and Valve to remove the bullshit privacy restrictions on their headsets’ forward facing cameras. That is the only reason indie devs didn’t beat Apple to this feature.
You can identify the PS5 controller during the anchor added event and swap the models yes, the code currently tracks the visualizations in a dictionary so you could easily tweak the code to do so.
So instead of Apple building human humanoid robots. They’ve just created a training platform for humanoid robots. And all they have to do is plug in the OS and electric motors.
I know 😅 crazy how what may look so simple could be so complex. This type of thing is very easy with Unity and other game engines, but with Apple is somehow so complex.
I was very surprised how long it took, honestly this is still in beta and I wouldn’t be surprised if it was drastically improved when it comes to prod or in a future version, but I agree, it would be very interesting to be able to run it against an NVIDIA GPU. Thanks for your comment!
@@ChiltonWebb I am very curious to see how this improves once it gets out of beta. But, I agree it is definitely capable of a lot more! Thanks for your comment!
@@dilmerv Yeah I think you did an earlier video about ARKit with object tracking a few years ago, and it was butter smooth. But hey, this is Apple, they don't really half ass this stuff. So I expect it will only get better before it's finalized.
![Felix "Unfair" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/Oswujxm2Ag0/mqdefault.jpg)
💻 You can also find the demo shown on today’s video via GitHub from github.com/dilmerv/VisionOSObjectTrackingDemo
I also recommend going over all the resources available from the video description.
This is a very cool idea, it was interesting to make something like the face masks that we applied. In real time.
Sweet, I can’t wait to see what you create! Thanks for your comment.
This is incredible! Can't wait for 2.0 to drop more formally :)
Thanks for your feedback, I agree this is going to be huge and super helpful!
Thanks for the great video. That was exactly what I was looking for.🤩
I am very glad you found it helpful, thanks man!
Thanks for these detailed instructions! 😀
You are very welcome! Thanks for your feedback.
Cool Video! I just tried to use a CAD model to train the reference object. However, it didn’t work. I am curious about how to config the usdz material for a CAD model if we do not want to use the scanned version which is not as accurate as the CAD😅😅
I don't believe CAD files are supported, Apple normally only supports USDZ files only. Take a look at this suggestion as it seems to may be an option: forums.developer.apple.com/forums/thread/745911
Very cool feature. I have noticed that there is another newer session called SpatialTrackingSession which can provide tracked object's world position, can the one you used also output some similar data?
i'd love to see a detailed video on Unity Polyspatial and how to use Dynamic Lights and Shadows on AVP builds. Its been a super headache, with no clear direction. Especially for bounded volume experiences.
Thank you for your feedback! I am about to make a new video and this will be very useful.
@@dilmerv We finally got shadows to work, but litterally had to create an empty project, and use all the defaul settings, and the sample lighting scene, just to do a build test and see if it was even possible. From there, were were able to make small adjustments. We found we liked Hard shadows better than soft shadows.
Thank you, Dilmer. It's very easy to follow. Is it possible to combine this object-tracking feature with the Unity AR project? (Like Ar foundation etc.)
That’s a great question, I don’t believe it is available yet but I know the Unity team always stays up to date with moving native functionality to AR Foundation which in this case I agree it should be available with Unity PolySpatial.
I will keep an eye on it and once they add support then will make a new video.
Thanks for your comment!
@dilmerv thank you for your great job. I have a problem. Could you please help me? I want to attach a reality view with a usdz & an attachment(SwiftUI view) to the object tracking anchor how to do this?
That's a pretty good question! let me look into it and see if I can put something together.
@@dilmerv This would be awesome
Hello Dilmer! Thank you for the video :) I'm studying for VisionOS object tracking these days, and your video is being so much helpful. I already finished ML for an object and put the 'referanceobject' in the 'Referance Objcts' folder. But when I ran the project on Vision Pro, it was saying that, "Start Tracking 0 Object(s)". Am I missing some steps?
I remember having such problems when the app didn’t have the correct permissions. Did you get a popup when you launch the app to give the app permission for spatial understanding? Also thanks for your feedback on the video!
@@dilmerv Yes, I did! I solved the problem I guess. Thanks for your answer! Ah, one more questions please, is there ways making the tracking speed faster? Or need we wait for next visionOS, ARKit version updates?
Great video, you never disappoint. I have a question tho. How do you access Guided Capture iOS on your phone? The link in description is unavailable
I believe Apple updated it, here is the new link: developer.apple.com/documentation/realitykit/scanning-objects-using-object-capture but to be honest I need to test it, however feel free to check it out. Thanks for your feedback on the video!
@@dilmerv Thank you for answering me so quickly. You seem like a very kind person 😄
What's the limit on the number of reference objects that you can track?
Couldn't find that anywhere in the docs.
I don’t believe there is a limit imposed but a device limitation in terms of performance. The files are quite large so if you were to add 10 or more I could see the binary be more than 1gb so that should give you an estimate of what’s possible.
Hey Dilmer, is the object tracking feature only supported in apple native development, and not supported by PolySpatial?
Hey thanks for your comment. As of now yes, but I know Unity always tries to stay in parity with platforms features, so perhaps it will be added in the near future. I recommend posting in the Unity forums to see if that’s in the roadmap. Also, another way could be to add a native plugin, here is an example of one docs.unity3d.com/Packages/com.unity.polyspatial.visionos@2.0/manual/InteropWithSwiftUI.html
@@dilmerv Thanks for your reply !
Does the tracking generalize well or is it specific to the physical object you scanned?
Specifically I'm wondering if you can scan 1 car (compact sedan) but have the app work for all cars (compact sedans)?
That’s a great question, currently the reason it takes so long is mainly because the machine learning algorithm tries to track the exact object in question, and it is pretty good about it. I don’t believe you could do what you are asking unless Apple provides additional data, such as how similar it could be on some type of percentage through their SDK.
Do you happen to know if I update to the Beta if there are errors, is it possible to downgrade by factory resetting the device? Or will it factory reset back to the same beta?
Factory reset will reset it back to Beta with a clean version, you could download the current prod version from Apple dev portal and use Apple Configurator app to restore it back to the prod version. Great question!
Can we use this as native library and plug it with Unity? to send the bounding box pose?
I don’t believe that’s supported right now from the Unity side of things. But I could see that being integrated in the future within AR Foundation / object tracking features. For now, I would stick with native Xcode.
What if I already have the 3D model of the object I want to track as an fbx or cad model. Can I use that to track the real world object?
If the model is very very high quality & photorealistic then yes it should work. Also, right now I believe you can only train with USDZ files so if you convert it to that format then things should work.
Hi Dilmer
First of all, thanks for your content !
Do you think that The Object Tracking will be accessible from Unity side similar to Vuforia model targets
Thanks for your feedback, and yes I am sure Unity will be making this available with PolySpatial just like they do add support for all ARKit features available for visionOS today.
Does it way more battery doing that as it constantly looks for the objects and needs to know their positions?
I assume if you have to make it search for more than 2, it will drop the FPS heavily and be unusable then the app crashes?
Normally when I run CV processes on mobile phones I can tell that it is pushing its capabilities, but with this, the device performs so well, I didn’t feel like it was causing any kind of performance degradation, but I only did it with 2 objects. I will be testing this more to find out its limits, thanks for your feedback.
@@dilmerv yea, need to look for memory leaks too
Being able to store that stuff on a cloud to use would be good.
Would be cool if it activated certain things if within a geozone.
Have you tested if this works and tracks as well in an iOS app? Or is it VisionPro only?
Based on what I read and saw in the code this is only for visionOS. ARKit for iOS provides object tracking and it came out a few years back, here is some info: developer.apple.com/documentation/arkit/arkit_in_ios/content_anchors/scanning_and_detecting_3d_objects
Thanks and great question!
@@dilmerv Yeah i remember :) have used it for a few different projects as well. But the scanning process is extremely limiting and adds a lot of randomness to the result so am really hoping this new way become the iOS way as well :) thanks for your answer!
@@NikolajStausbl on iOS is it just 3d object detecting or also 6DoF tracking? is it fast?
@@0GRANATE0 It is definitely not realtime and it definitely doesn’t like moving objects. Static, non-glossy, high detail objects are best.
@@NikolajStausbl I was hopping for something like this: ruclips.net/video/0ORZvDDbDjA/видео.html (demo is allegedly with monocular 2d camera)
It seems like it should be possible to reduce tracking latency on stationary objects if you integrate the tracking box with input from the gyros and accelerometers. Just a thought. Anyways, thanks for the upload!
Hey thanks for your feedback, and I agree that we may have a few ways to improve it, but knowing Apple I know they will improve this very soon.
@@dilmerv Turning on Apple Enterprise API can improve the tracking rate.
@@dilmervyour 3D models are pretty big in scale compared to the PS controller and the little figure or is the UI showing it wrong? ruclips.net/video/NaB_6hLzYN0/видео.html
Interesting, could you send me a link to a document where I can read more about it?
They're about the same size and accurate, if you look at the measurements within Create ML, it shows the figure as 22 cm ish, that's the real size. The UI shows them big mainly because I zoomed in.
what's the lag? How about accuracy?
Good questions, it is slow and not instant as perhaps you've seen on iOS. For Accuracy, yes it is very very accurate.
By accurate, I mean actual metrics. Do you know if Apple provides that?
Is this anything even close to this in the Meta Quest 3 universe? I can't find a way to track real life objects. Do you know any solutions? Even with external hardware..
Hey thanks for your feedback, but this won't work with Meta devices. Meta recently released an open source project for computer vision with Meta devices called "ocean". I recommend taking a look at: facebookresearch.github.io/ocean/docs/overview/ and facebookresearch.github.io/ocean/docs/demoapps/questapps
@@dilmerv Thank you so much! What about Pico 4 or Pico 4 Entreprise? Any chance I can access the RGB sensor feed? THXX
Doesn’t Polycam scan objects easily?
Yes you could use polycam that’s correct, looks like they also support usdz files. I didn’t use them mainly to keep the workflow with Apple tools but you are more than right to have that as an option.
Thanks for your comment.
This program will work on quest-3 or it is supported only on vision os?
Currently, this is mainly for visionOS but I remember that recently Meta implemented Apple’s spatial video, so perhaps they may leverage this tech in the near future. Great question!
The best thing this can do for the market is convince Meta and Valve to remove the bullshit privacy restrictions on their headsets’ forward facing cameras. That is the only reason indie devs didn’t beat Apple to this feature.
Yeap, I just can’t wait for Apple to remove the enterprise limitation on camera access, and having other manufactures also allow it would be amazing.
Someone actually did, and it’s a lot better than this. ruclips.net/video/kiSOmFVfNpc/видео.htmlsi=IsBPWxaI5FPrHdvK
Can you identify the PS5handle and turn it into a robot with one click?
You can identify the PS5 controller during the anchor added event and swap the models yes, the code currently tracks the visualizations in a dictionary so you could easily tweak the code to do so.
@@dilmerv cool！
So instead of Apple building human humanoid robots. They’ve just created a training platform for humanoid robots.
And all they have to do is plug in the OS and electric motors.
Yeap, that's what they do. They create the foundation and then go big on building the top layers later on.
Thanks for your comment.
Bro really spent hours changing a bunch of lines of code, just to put some red lines around his real world objects😂
I know 😅 crazy how what may look so simple could be so complex. This type of thing is very easy with Unity and other game engines, but with Apple is somehow so complex.
11hrs to train a medium quality model? LOL so much for M processors, would love to be able to use this with Nvidia cards
I was very surprised how long it took, honestly this is still in beta and I wouldn’t be surprised if it was drastically improved when it comes to prod or in a future version, but I agree, it would be very interesting to be able to run it against an NVIDIA GPU. Thanks for your comment!
You’re comparing an SoC to a dedicated GPU….
The M1 Pro is a tiny chip...
That's disappointingly slow. ARKit is capable of much faster speeds. I wonder what's up with that.
@@ChiltonWebb I am very curious to see how this improves once it gets out of beta. But, I agree it is definitely capable of a lot more!
Thanks for your comment!
@@dilmerv Yeah I think you did an earlier video about ARKit with object tracking a few years ago, and it was butter smooth. But hey, this is Apple, they don't really half ass this stuff. So I expect it will only get better before it's finalized.
18fucking hrss hell noo
Looking forward to drastic time cuts in the near future!