Using an iPhone as a tracker for Virtual Production
HTML-код
- Опубликовано: 6 авг 2024
- Can you use an iPhone to track your live-action camera within Unreal Engine? This is a step-by-step experimental guide to getting it to work. The setup works to some degree, but still needs some tweaking and experimenting to make it viable for production.
Some tutorials that I learned from to make this.
FattyBull's Composure tutorial
• Unreal UE4 Composure V...
Andy Blondin's Composure tutorial
• Unreal Engine Composur...
Unreal Virtual Camera information
docs.unrealengine.com/en-US/E...
Matt Workman's channel
/ cinematographydb
Possible Android version plugin:
www.unrealengine.com/marketpl... - Кино
some guys use word "indie virtual prodution" while it's still costs thousands of dollars...
but this video is the real indie... thank you Bro !!
really help for someone from 3rd world country where the resources are very limited
Very cool. Definitely not an easy setup. Hope it gets more fluid the more people get used to filming this way. Thanks for making the video!
Not arf!!!
It's quite unvalivable because many people will leave this knowledge for themselves and You decide to share it. Big BIG THANKS MAN
Thanks for illustrating this overall concept, and in such good detail. Very inspiring!
Can confirm that this also works with an Iphone 7. Thanks for making this tutorial!
Green bits are caused by lighting on the green screen. Fix it by altering the vignette value in the colour keyer transform pass.
Thanks this helped with the edges!!!
Virtual hand claps for you sir! Bravo, well done, and thanks x1million.
Great video man. Helped me out a lot. I am doing similar work with my Canon T5I and T6S and getting decent results.
Great tutorial! This is basically all the kit I got at home, I will give this a try :) thanks!
I bought hitfilm pro yesterday and am getting and iphone 12 today. I have been working on a large scale unreal scene and have been looking into getting in virtual production. Thanks for posting this. Its motivating that others are shooting for same or similar goals.
Be sure to check out CamTrackAR from Hit Film.
This was awesome! Thanks for the detail!
Thanks for this very helpful introduction. Answered a lot of my questions!
Great to hear! If you experiment with this, I'd love to hear your findings.
You just opened my mind... THANK YOU!
this is a nice tutorial !!!! thank you... I will try to follow the steps
This is so valuable. Many Thanks.
Can't wait to get home from work and watch this
Thanks for sharing this video. Much appretiated!
Wow this was such epic tutorial, completely revolutionary!!!
As I’ve said in some other comments here, this method might not be the best approach going forward. It seems like there’s at least one app being developed to handle the tracking of the camera with a phone much better (filmengineers dot com had something in beta which seems like it could be the perfect solution).
@@hyperbolicfilms there is no video or something about these "filmengineer . com " solution in this field as far as I search there is nothing else but only their page
@@hyperbolicfilms Thanks a lot for this, its been one year now, do you know if new latest tracking apps have released for this yet?
@@anirudhsays1534 I have not been following closely. There were a few promising apps like byOwls and Filmengineers, but I have not seen anyone’s result using those.
I have been thinking of trying this again a year later to see what’s possible now. Hopefully I’ll have time before Christmas.
Thanks for the excellent explanation
thanks for this! definitely wanna give this a try.
Great tutorial! Thank you!
This is awesome, I will try this!
Thanks so much for the video tutorial, its really helps me as a entry level.
A lot of this information may not apply in more recent versions of Unreal Engine, as a lot has changed in almost 2 years.
Very helpful video, thanks for this!
very nice and quick info, thanks
Very nice tutorial
Wow I have never seen this before. Pretty awesome
I'm from Germany and I must to say thank you for your help now I will make Livestream with this
Blind Gamer Good luck!
Finally. After watching all kinds of virtual setups with trackers and vr kits for tracking and making a virtual cam with the iphone, I kept thinking "can't we just ... use an iphone to do this" but no one ever did, but I was sure there was a way. Sure enough your video finally shows up in recommendations after watching all these other ones (the algorithm finally pays off) and you deliver what I assumed could already be done. Thanks
You should consider that this wasn’t far from the limit of where I got with this approach, and I don’t know that it is really viable for production of any kind. In some of the comments, I’ve listed some of the companies making plugins to make the phone tracking approach a little more reliable, but I don’t know that any of those are production-ready over a year later.
@@hyperbolicfilms yeah, no doubt, I was just looking for simple stuff around the house - mainly to have the knowledge of it. I also didn't feel like spending $600+ on sensors and such just to mess with it a bit when I have an iphone and an ipad and a dslr laying around. It's a good start to bounce off of.
Esto es todo lo que busco !!!! Oh porfavor esto es oro para filmakers
probablemente no sea bueno para la producción. solo un experimento
Incredible work!
Thanks a lot!
Great tutorial ! 👍👍
thanks! got it working with my iphone 11 pro
Amazing tutorial
love this content!
thanks for your share actually not easy setup
Brilliant!!! Thank you!
This is such a great video and has saved me probably about a year or mores worth of research! Please try outputting the unreal feed to a data projector to skip the green screen? And in the unreal vid today, Matt Workman said he had to genlock the camera to the video feed. Didn't quite follow why then you tube suggested this video!
I'm not sure I'll get to the projection or LED wall stage of things, just based on the limitation of the space I have in my apartment. I just moved to a larger space, but still there's no real good space for a projection based on the layout. If I can get into someone's studio for a while, I'd definitely be interested in playing with it, but I've moved more towards animation in Unreal rather than live action/CG background integration because it's doable for less money.
I'd suggest looking into Ian Fursa's work (ruclips.net/video/_FC1lvhoPRc/видео.html). He posts a lot of his experiments on Matt Workman's Facebook group and he's definitely shown that you can have some success with a low cost projector and a Vive tracker, so probably around $1000 for the hardware.
@@hyperbolicfilms Thank you for the link and not sure either way will be easy on our pockets!
I require a good camera and a good green background. but THANK YOU SO MUCH SIR !
Thank you!
this is blowing my mind
Awesome . This is so far the most clearly explained video I have watched about virtual production . I only have one question .. I was wondering if its possible to connect live video from my camera to Unreal engine wirelessly using 7D Mark 11 . Because 7D mk 11 has an inbuilt Wi-Fi feature which works when you purchase a card for it that you place in an SD slot of the 7d mk 11 camera ? Would that work without having to connect the camera via Camlink device ?
I think the green flickering is due to the auto vignetting down to the chroma keying module. And the synching issue is something to do with genlock on camera. Please correct me if i am wrong.
I followed this video for virtual production, thank u faculty at GBC
How would you go about syncing the tracking and video? At the moment my tracking data seems to come a little after the video feed and I would like to somehow delay the video feed to get them to sync
This is a great tutorial. Thanks! Any chance you have an HD version so we can see the drop down selections better?
I uploaded a 4K file, so I'm not sure if RUclips is still crunching that or what.
Thanks alot for the great tutorial, I've been looking everywhere for this. Do you have any updates tweaking this?
Improved lighting, 25 fps, RTX 2080, and a better camera will fix this.
thank you
Great video man! i have a GH4, my question is what cable should i use ? should be the same as GH5? Could you please tell more info? Thanks a lot!!🎥📸
very Excellent for beginners, one question How did you connect iphone cam with unreal?
hello, I already set the video camera feed and the iphone as a tracker. I havent done the compusure yet, but I was wondering ,after setting all these, then what? I mean do you know how to record this and export a video? Can the camera movements tracked by the iphone be recorded and possibly exported as an fbx? I know you will tell me yes with sequencer, but how? by the way for now im talking about just the camera tracked with the iphone, no video feed necesary. Im doing lot of test focusing mainly in tracking the camera.
thank you for this tut but i want to ask you if i want to make out put from this to ndi
Looks like the offset of the phone and the camera? At this close range the angle is changed a lot from the field of view?
I think there is a frame rate offset...and if you type how many frames you want to offset it by 4 - 6 it usually fixes the drift. Don't quote me but it's something along those lines.
Thank! I haven’t been able to find that, but i have heard that there should be something to compensate.
My gh5s can connect to stream on pc via usb cable. So how to connect gh5s with unreal 4 via usb ? Thank you !
Hello. So with this method can the compositing(iphone and video camera feed) be recorded? I have tried everything and it cant just record anything. When trying with sequencer and take recorder it creates another cam that doesnt record anything cause it´s a "ghost" cam that disappears as soon as you stop playing. Can you please tell me how to record sth after do all this set up?
Great video! One thing though... my camera shows up in media player, but now video shows... but when i open camera in windows, the video works fine... any solution? Thank you!
What’s the app that you use for tracking on iPhone?
Legend 👍🏼
You can now do this much easier than I was trying in Aximmetry with their new Eye app
Thanks
You are amazing… technology is amazing… this is amazing…. UE5 is amazing… RUclips is amazing thank you man. Please do more green screen UE5 tutorial
Great tutorial!!! Any idea how to turn your TV into a makeshift LED Wall? Not for like an official production but for leaning and practicing without 1000s of dollars
I'm trying to figure that out myself, if I get it going in the next few weeks I'll post a step-by-step as well. I assume it has something to do with nDisplay still. I was thinking that using a green screen would be good for wide shots, but maybe getting an 80 inch TV would work for close-ups and like toy commercial/product shot type things.
When i export the sequence, the rendered video or image sequence is 4 times faster than it should be.
great thank you
By the way, you might get different delays between the tracker (phone in this case) and the render. Remember that there's a delay in capturing the image and this might well be different than the framerate of the transform captured from the phone. So usually with a vive tracker, we add a buffer of N values, where N is some tweaked values that are eyeballed to give the approximate correct delay between tracker update and video display update. Anyway, keep it it, might be a useful thing to be able to do!
zoombapup I think there are too many limitations to this approach. Shortly after I recorded this, I realized that doing any sort of severe Dutch angle on the camera would cause the iPhone to switch into portrait mode, which makes the traffic unreliable.
My best results were in capturing the camera movement data, recording a take of the live action plate in camera, and combining the live action and CG element in After Effects. It’s definitely not nearly as accurate as a Vive tracker would be, but I think it can work for some shots.
@@hyperbolicfilms Good point. No way to lock the camera in landscape mode? I'm using a vive tracker and a single base station and that approach works pretty well. I think these virtual cameras are probably best for doing entirely virtual shots though. I've hooked up a vive tracker based shoulder rig with a virtual camera setup and that works well (made some custom controls for an xbox PC controller). I'm also doing some work on controlling take recorder via an app on my surface pro so I can do different virtual setups and pull focus and play with the virtual cameras.
@@zoombapup I've mostly moved on to playing with mocap and animated characters for my own work. I think Epic could make the iPhone do what I was hoping it could with some adjustments to the Unreal Remote app, but considering the situation now with Apple, I doubt it would happen. It's frustrating because it is close to being a good solution for mixed live/virtual productions, but just not enough in the app.
hyperbolic films the virtual camera spawns a new camera instead of using yours.
How should I go about adding the buffer? Just using an iphone but the tracking is coming later than my video feed
Is there any way to use EOS webcam, or an OBS virtual camera as the Video Capture Device?
Also whenever I try to capture video it also captures the virtual camera control hud (even though I otherwise don´t see it even on the phone) and the quality is really terrible compared to the previews. Any ideas anyone?
Brilliant
At 8:18 my video tab isn’t showing my cam link. It’s just empty. My cam link is installed in my computer correctly since it’s working on other apps. Any help?
Is this need any Vive Tracker or HTC? Or only Plug-ins and Iphone to connect and camera track?
this video is old now but still impressive ! I've tried to do the same but outdoor and offline using BlendARtrack (i'm on android, i didnt tried on iphone yet)
I was close to a good result but even if i turn off the stabilization of my differents cameras, it's not perfect :(
I think the problem is even with IBIS Off the sensor still move a bit and make the tracking a little bit wrong... Did you tried to do an offline/outdoor tracking ? on iphone there is also CamtrackAR app.
See you !
and when you click this and tada, we're losing our comp... hahahahah but really cool setup!
ARKit didnt work to respone the location and rotation in UE4.2.6,it any bro know what s wrong?
Especially inspiring because I just bought a new computer, a lumix g7, a zhiyun crane, and a new phone because my moto z2 had lost it's mic from sanitizing liquid.
Hi, very nice Tutorial.
How did you get access to the "VirtualCameraGameMode" in the World Settings? :)
i can't find it in Unreal 4.27 or in 5.1.
same issue.
The sync issue is a common problem. most people i know they delay the trackers because its frame rate is far higher than the video source
I am looking to combine live action from Camlink 4K (DLSR) and a matching virtual camera background from Unreal using the built in chromakeyer in OBS. Skipping the composure elements all together. I want to use ipad or iphone for the tracking, but I can't tell from the video if you get world translation with the iphone or just rotational movement of the camera. I.E can I move freely in the virtual environment?
Will Huff You do get both translation and rotation with this technique. There are a number of problems with the accuracy though that I haven’t sorted out. The Unreal Remote app seems to be adding some smoothing that cant be removed, so it’s not a perfect match the way a Vive tracker can be. That is definitely a more tried and true method with far more documentation on it than this approach (which only a handful of people have ever told me they’ve tried).
hyperbolic films ahh ok I see. Thanks for that. I’ll prolly do a test then get to ordering the vive trackers and go thru those videos on setting up. Thanks!
Is there a way to stabilize footage in post if I record video on an iPhone while shooting video on my camera in a rig similar to what is shown in the video? I guess a better question would be is there a way to utilize the iPhone sensor suite to stabilize footage in the same way go pros or iPhone work? Im thinking Sony released a post processing software that can stabilize footage in post using sensor data so i feel like this is possible.
No idea. I think this would take Epic making a big update to their app, which seems unlikely.
Hey there, great video. It has been massively helpful. Can I ask can you still use the uremote camera control fully this way? I have found I can control the camera with the uremote, a keyboard or a controller all the way up to the point I change the composite output to player viewport then I seem to lose control of the touch controls and mouse/keyboard inputs once it plays the comp. Any idea how to keep the inputs and the comp?
I haven’t had much of a chance to revisit this workflow since the summer as I’ve been back at work the last few months l, so I don’t have a good answer for you. The URemote does seem to pick its starting point and there’s no way to change much. It’s a very blunt tool, and I hope they add in more control at some point.
@@hyperbolicfilms thanks. No worries. If I work it out I’ll let u know. Cheers
Does anyone know if it's possible to track the position of your IPhone without a vive tracker? It seems to be only using the rotation
The Unreal Remote app tracks both the position and rotation of the iPhone, but it isn't as precise as using a Vive tracker. You can move around and do pans/trucks with just an iPhone.
If you are just using the iPhone for a virtual camera within Unreal, Unreal remote offers a lot of power.
my camera is red Komodo and latest update has gyroflow data is this enough for unreal? or does it need the lidar from iPhone? as the gyroflow data should contain all the camera position data
I am not sure, but I imagine that you would still need a tracking device of some kind, as I haven't heard of anyone using a Red without having some other tracker involved.
My method with the iPhones kind of hit a dead end not long after I made this video in 2020. People have tried it recently and there's still some issues with delay between the iPhone tracking info and the Unreal scene that haven't been resolved.
There are a few lower cost solutions to doing tracking, such as Retracker or the VIVE Mars that are more viable. but way more expenisve than a phone.
if we use sensor on iphone for camera movement in unreal engine , is it only for rotation movement ??? or we can use it also for position movement ?? please answer , sorry for my bad english.
That’s been answered a few times already, it does both 👌🏼
Does it matter if the capture card is 1080p or 4k?
thanks for such a detailed video, but still a question.. does it track dolly movement too, or just a fix trapped movement? please answer.
It should record any movement, except Dutch angles. Because of the iPhone’s inability to lock into landscape mode, tilting into a Dutch angle causes the Unreal Remote app some problems.
@@hyperbolicfilms thanks for this valuable information.
interest.. My phone is Android (samsung s20) or Logitech webcam but Media player in UE4 screen black? how? let me know thanks.
me too,i hope he can answer this
Is the phone sending the gyroscopic motion ? Can we go forward, backward and sidewise in this setup !! Also with tilt and pan
Yes, you can move in all directions and the virtual camera follows.
Awesome video! Did you actually move the video camera itself while rendering the 15sec video? I didn't understand that part. If i had a greenscreen that is infinite around, i could possibly orbit around the subject with my camera and the sequencer would record my movement? Thank you!
In theory yes, but this method has broken since Epic changed things in more recent versions of UE. Their new mobile app doesn't work the same way and doesn't seem to work with many iPhones. I've tested it on my iPhone 11 and a new iPad and they won't connect with Unreal.
@@hyperbolicfilms that's sad. Thank you for your reply!
Did anyone figure out why the camera feed keeps disconnecting?
What are the details on the iphone mount? Looks like you're using an adjustable mount and an adapter to the camera body? Links or details about both would be appreciated!
John Gabriel The mount is a simple tension-based phone holder attached to the camera by a cold shoe to 1/4 20 mount. This is a Manfrotto phone holder, so was about $20, but I have a few $2 ones that would do just as well.
Thanks, I was thinking about doing something like this, this will greatly help me. I hope I can add iphone camera to the UE instead of a second camera, as only second camera I have is a 720p webcam 😅
Update: Couldn't make ARKit work on 4.26.1, had to download version 4.24.3. On 4.24.3 with iPhone 7 Plus ARKit works. But we had issues with camera, we bought a workstation with Aja Kona 4, but "Open URL" blueprint thing didn't work with Kona 4. If we press play, it does not recognize the camera. So we had to reopen the media player and select capture card again while engine still on play mode on the background, that way it manages to capture the camera. We'll try to find a cheap Blackmagic Decklink card that is compatible with Unreal Engine to test this at a later time. I'll try to update this comment if we can find a cheap Decklink card.
It’s not real time, but CamTrackAR may be a better choice for you. It tracks from the iPhone camera and they are adding Unreal Engine functionality.
hi! I have a question, how can i connect an intel realsense tracker camera in unreal engine? to make something like the step two. I should do that magic that you did with the Command Prompt? or what is the ideal way to do it? thanks for your help and have a nice day.
Hi, I’m not sure how that would be done, I haven’t used realsense.
Bro, there was no CamLink in my city and I bought an alternative to AverMedia Live Gamer Portable 2 Plus
When I create a player in UE4 and select a view, it does not give it out
Although in other programs the signal receives
Maybe I did not turn on something in the settings
If possible, please tell me
I'm not sure, it could be that AverMedia doesn't work well with UE4. I know that the Elgato Camlink wouldn't work with Aximmetry's Unreal edition even though it works in regular UE4. Without official support from the companies it's all a guessing game.
Hey man, thanks for this amazing tutorial. Just one question, I don't know why the virtualcameragamemode option doesn't appear in the gamemode override options, even after have installed all the pluggins. Thanks very much C:
I think things may have changed in more recent versions of Unreal. This was made in 4.23 I believe.
There is a new version of the virtual camera app, but I have not been able to get Unreal LiveLink to see my phone with it.
Thanks very much again for this tutorial!
when I connect unreal remote 2 and connect I can see the scene in the phone but get no camera control?
I haven’t tested the new version of unreal remote, so I can’t say for sure.
Spoiler alert. Deckard is alive! :) Thank you for a great tutorial!
Glad you like it!
Hello, i have an iPhone XR and doesn't track the camera movement, anyone has the same issue?
Sir i don't have iphone camera how can i do with my android phone please tell me
Hi, im currently working on trying to implement a setup like this at my university, is there an updated video I should use as reference instead?
This does seem like something that would work, but it’s also 2 years old :)
Thanks for the great video!
Hi, a teacher at the college where I took a virtual production course this summer tried to recreate it with the latest version of the software, and he determined that there was no good way to get sync.
I still haven't found a good, reliable, affordable way to do virtual production. I had tried Assimilate LiveFX, which is a very good live compositing program that says it supports tracking from phones or the affordable Intel RealSense camera, but even using two fairly powerful computers with multiple SDI I/O cards and timecode, it was still out of sync. I could record the Unreal background and the video in the camera and sync the two and use LiveFX for a video preview though.
There is a new very affordable tracker that hit the market now (has similar packaging/marketing images to the ReTracker Bliss, but I can't recall its name). It might be the game changing device, but still doesn't have a lot of software support.
The fastest/most affordable way to do any of this is using the CamTrackAR app on iOs, but it has its limitations and isn't real time.
@@hyperbolicfilms thanks for the very detailed answer! I really appreciate it. 🙏🏼
You can get rid of the green corners if you change the material to "M_SinglePassDiffColorKeyer". Question: Do you know better methods or plugins for a very good greenscreen keying? Or would you stick just to the in-built tools? Thanks
Thanks for the info. I have not had the opportunity to dive into this much since I made this tutorial. I’m going to start using Aximmetry in the near future to see how well that does, partially because of the keyer and because it already has the input and output side of things handled.
@@hyperbolicfilms Would be great to share your experience again :)
I was thinking of doing this with VHS combined