Great work! Would really like to know how you dial the delay in in smaller increments than frames. Or is it the trick with camera delay also in the media bundle?
For the moment I think using the delay in the media bundle to delay the video by a bit extra (less than a frame) is the easiest way to do it. However just this morning I saw some new stuff about timecode that I need to try out...we'll see how that works. I was looking at interpolating between tracker samples too, but if LiveLink support for VIVE comes out anytime soon, it should handle this better.
Greg! ......... great stuff!......... could you pleeeeeeease make a video showing how to build this setup from a blank template?......just an unedited and long video is fine, would learn so much from that. I know you have the project on Github but i'd learn a lot more seeing how it's built from scratch.
I'll consider it, but to be honest this is the only request I've had to build the whole thing from scratch on camera. Most people don't seem to like watching videos that long. The unreal learning portal already has some good starter documents and the Unreal website documents on composure and video in/out cover all the basics pretty well.
@@GregCorson Hi Greg. Would love to watch something from the very basic, right from setting up the vive pro. Long video is perfectly fine, cause we do learn a lot from your video. Though there are other Virtual Production tutorial on YT, It's difficult to find good one really.
Thanks for the tutorial ! I just wondering how to switch between the composure result and the original image from the camera during game. In other words, a way to toggle visibility of each layers, but during game. Have you got some ideas on how to do that ?
If you click on one of the composure layers in the world outliner, you should get a preview image of it in the editor screen. If you want to see the raw image (without chroma key) you need to turn off chroma key in the composure pass. I'll have to mess around with it a bit to see if there is an easier way.
Hi Greg, We are looking to add a bit of stabilization to the tracker, just to smooth it a bit as its extremely sensitive. I wonder how would you recommend approaching this? Would you add a 'tinterp to' node after the time delay in the BP?
I have messed with it a bit but haven't found a perfect method yet. A lot will depend on how much you are moving the camera. Most people find that the jitter is not noticeable if the camera is moving or handheld but it can show up when it's on a tripod. The biggest issue with smoothing is to find a method that won't cause a jerk when the camera transitions from stationary to moving.
I don't have a video about this but it is not too hard to do. The video comes in through a "media player" object. To use a webcam you need to set one up for it, then connect one of my composure "media plates" to use the video texture it outputs. Unreal has a short tutorial at the address below that should give you enough info to get your webcam working. Feel free to come back and ask more questions if this doesn't work for you. docs.unrealengine.com/en-US/WorkingWithMedia/MediaFramework/HowTo/UsingWebCams/index.html
@@GregCorson Hi, thank u for the answer, I did figure out that it through the media player but couldn't make it work. And also saw that page on ue4 from your readme notes. didn't help.
Sorry for the delay, were you able to get that tutorial on the Unreal site to work at all? Just wondering if it doesn't work at all or if you just had trouble connecting it to composure? If you got a working media player you just need to go into the Media Plate in composure and set it to use the media player's texture. Let me know if you are still stuck
Hi Greg? How do i assign pucks to specific trackers? right now my tracker is moving plum line instead of camera. Also if u could go in detail how to output the composed output to a stream that would be awesome.
There are two parts to this, in steam you have to assign a "role" to each tracker and then map each tracker to Special_1, 2, 3...etc There are a couple of tutorials on the channel that show how to do this. Whenever you drag a tracker object into the scene you need to go into the actor's "motion controller component" and set the tracker to Special_1, 2...etc. Once you've done this the trackers should stay assigned to the right actors in between runs of Unreal, no matter what order you turn them on. Regarding "output to a stream" do you mean just recording it or do you want to live stream the output to something like RUclips? For recording you can use OBS to record the window or you can send full screen output to your graphics card or out an HDMI output on your video card if you have one, then record it with something like an Atomos Ninja V recorder. I haven't worked out a solution for directly live streaming to RUclips yet.
@@GregCorson In The project file VivePuck 1 and VivePack2&3 are sharing same VivePuck blueprint and consequently same Motion Controller component. Is that where i supposed to change motion Controller setting? it didn't work when i did. Can u please let me know if i am looking at the wrong blue print?
@@GregCorson Just found the second part of your reply)). Thanks. Regarding Output stream. i want to send it to graphics card output full screen. Lets say to my second monitor. Please advise.
Also in VivePuck it asks to set owner or motion controller will not work - maybe that's where my lack of understanding is. Can you please let me know what should i set it to and how?
Thank you very much, Great work! I have a little question. I use an APSC camera with sigma 30 mm so with the crop factor the focal length of my video feed is 45mm. Why when I set it to the Cine Camera the background is not matching? And I'm getting a lot of parallax...
You need to set the specs of your camera and lens in the VPCamera actors in the level. If these aren't right you will have the problems you describe. Select one and look in the details panel, there is a preset in "filmback" for APS-C. I'd suggest you try it and then look at the sensor width and height, if they don't match the specs of your camera, set them manually. Having the correct sensor size will automatically figure the crop factor for your lens. In the lens settings, set the focal length of the lens to the ACTUAL focal length marked on the lens. In my earlier tutorials I posted some links for how to find the entrance pupil (aka no paralax point or nodal point). There are a lot of tutorials on RUclips that tell how to do this for panoramic photography. A correctly measured camera rig with a correct Entrance Pupil estimate should solve this for you. If it doesn't, please feel free to ask more questions either here or in the Discord group linked on the right side of my channel banner.
hi Greg, I'm a big fan, and I just start to learn VP, one question: how to creat the garbage matte, I mean I don't know how to get this"FlagGreenSgreen" thing, many thanks
I'm not sure what you mean by this? In VPStudio there is already a garbage matte setup. You just have to position it in unreal so it lines up with your real world screen. You have to start by making sure you know where your room center is in the real world, usually by setting the HMD on the floor there, facing the green screen and doing a "room setup" for standing position. Then you can measure the distance from your HMD to the green screen and use that as a starting point for where you put the green screen in unreal. Then just go to the green screen object, make it a bit too big (so you can see parts of the room where there is no green screen). From there you should be able to line up the edges and reduce the height/width till you don't see the real world anymore. This is made a lot easier if you are using release 7 of VPStudio and Unreal 4.26 with the VIVE trackers running through live link because with LiveLink even when you are in the editor, all your cameras are tracked and showing up in the right places. Hope this helps, if I haven't answered your question please feel free to ask more.
Oh, forgot to mention, since Release 6 I changed the mattes a little, you might want to watch that tutorial. Instead of one "FlatGreenScreen" object with both a wall and a floor there are now separate "MatteWall" and "matteFloor" objects. This was because a lot of people don't need the floor matte. Also it's easier to setup and align the mattes one at a time. The new mattes have the origin in the lower right, so you just use "location" to line them up with that edge of your green screen, then use the height and width controls in "details" to adjust the matte to the right size.
Hello Greg, I wanted to ask: When your actor walks into the scene, is there a way to get shadows onto the 3D objects aswell when you move around? Does that require motion capture to be on and tracking the actor to get shadows?
Shadows are a difficult problem. Whether you want the actor to cast shadows on the floor or onto something like a desk it is really tough to get them to look completely realistic. Many productions have actual physical objects painted green (boxes, tables...etc) to catch the shadows. Then an alpha channel is extracted based on the areas of darker green where the shadow is. Other people use normal physical props like chairs, desks...etc close to the actor they can interact with and cast shadows on, using the virtual production stuff for backgrounds. Sometimes the actor is tracked in some way and a fake 3d object is used to cast shadows. Basically, there are a lot of techniques that only work in specific circumstances, there is no one way that works with all setups.
@@GregCorson Thank you for the reply! I suspected as much that it would help having the fake 3d object tracked on and cast a shadow, physical objects greatly help. Thanks! I look forward to figuring out which method works best for unique situations.
The simplest solution is to have an actual real floor (not greenscreen) to cast shadows on. This eliminates any need for special effects work. A lot of productions use this trick, they will lay down a strip of carpet or floor tiles on a green floor just so people can walk on it and cast real reflections. Sometimes when you see people walking down a sidewalk in a video, everything but the sidewalk will be greenscreen! To find new techniques, I'd recommend looking at a lot of the VFX breakdown videos from various effects companies on RUclips. Stargate Studios is a real interesting one because they do both big expensive stuff as well as cheaper things for TV.
You can easily set your actor (ie after green screen removal) to cast a dynamic shadow onto the virtual scene using the scene light. It’s definitely not perfect but works pretty well. I couldn’t get it to work with a translucent material but works fine with a masked one.
@@GregCorson Fantastic. Now I got to figure out how to do all this so I can comp my real robots into a sci fi set! Thanks Greg keep these videos coming.
Cool robots you have there! I'll have to go through all your videos to see more! I have messed around a bit making 2 wheel balancing robots (segway style) to experiment with. Biggest one is about a foot tall but the design scales up.
Thanks! I have another one coming up soon where I add the arms on to the drink dispensing robot. I’m thinking about putting him against green screen and set up a sci fi background to shoot some fun shots after he is finished. I’m hoping to spend sometime to set up the Unreal VP studio. I’m sure I’ll have a few questions for ya as I learn.
Sounds cool! Of course there is always the risk people will think your robot is just CG from unreal too. ;-) Be sure to check in the Unreal marketplace at the start of every month for "free for the month" content. They have had some good stuff like the science lab. There's some good stuff in the permanently free collection too.
Hi Greg, Great work on the video man! this stuff has been super helpful, Thanks! My buddy and I are running in to a problem we thought you might be able to help with. We're good on all of the compositing elements but whenever we launch the game, our CG camera linked to the tracker doesn't move. the CG plate doesn't seem to update when the game is running. All of the trackers have been tested and work but only when editing. Any idea why the CG layer image wouldn't be updating during game? The only way we've been able to get it to work so far is to select the CG layer and pin it to another monitor, then select the live cam layer and pin it to a separate monitor and use OBS to Key... Thanks in advance for the help!
Select one of the vive tracker objects while the game is running, go into "details" find the motion controller component and select it. Are the coordinates updating? If they are not then something is messed up in your setup of the trackers. If the tracker is updating I'm not sure why the view in the CG camera is not. Did you build your own rig in unreal to attach the camera to the tracker? I don't have my setup running right now, I'll try to get it going in the morning and see if I can think of anything else that would break things.
@@GregCorson Thanks for the reply Greg! I think we figured it out. We didn't have the CG layer directed to take input from the Cinecam on the details tab.
You mean how to setup the greenscreen? I'm going to try and do a tips video soon to cover things I may have skipped over too fast. There is an older video on my channel about setting up the greenscreen that might help in the meantime. Or do you need more information on how to setup the actual floor and other parts of your virtual set? Please help me out a bit here and tell me what I've missed and I will try to cover it.
Check out the AJA video quickstart on this page docs.unrealengine.com/en-US/Engine/ProVideoIO/index.html Be sure you have the right video format set in the plugin or you will get nothing. If you are not sure you can run the AJA control room program and it shows the format/framerate of the incoming video.
Greg, i cannot find in the blueprints how to make the tracking activate. I seem to have trouble having the tracking to activate, although if i make my own project with motion controller-special 1 I am able to. But with your VPStudio i’m not sure which blueprint makes it track
In my project, all the tracker stuff is in the "tracker" objects. One for each tracking puck. Set Special_1...etc in there as well as the tracker delay for that puck. If you find the trackers are not working, check the STEAM tracker roles to make sure they are setup right to send their info to the right trackers (there are two videos about this on the channel). Sometimes you have to set this up again when you change the name of the project. If one project works and another does not, this mapping stuff is probably why.
Dude; your videos are so useful. Thanks for continue to putting these out
Glad you like them!
Thank you very much for your time and for sharing your excellent technical work.
Great work! Would really like to know how you dial the delay in in smaller increments than frames. Or is it the trick with camera delay also in the media bundle?
For the moment I think using the delay in the media bundle to delay the video by a bit extra (less than a frame) is the easiest way to do it. However just this morning I saw some new stuff about timecode that I need to try out...we'll see how that works. I was looking at interpolating between tracker samples too, but if LiveLink support for VIVE comes out anytime soon, it should handle this better.
@@GregCorson Yes LiveLink would be sweet!
Greg! ......... great stuff!......... could you pleeeeeeease make a video showing how to build this setup from a blank template?......just an unedited and long video is fine, would learn so much from that. I know you have the project on Github but i'd learn a lot more seeing how it's built from scratch.
I'll consider it, but to be honest this is the only request I've had to build the whole thing from scratch on camera. Most people don't seem to like watching videos that long. The unreal learning portal already has some good starter documents and the Unreal website documents on composure and video in/out cover all the basics pretty well.
@@GregCorson Hi Greg. Would love to watch something from the very basic, right from setting up the vive pro. Long video is perfectly fine, cause we do learn a lot from your video. Though there are other Virtual Production tutorial on YT, It's difficult to find good one really.
HI Greg do you have Video about using T265 Tracker for Unreal Engine?
Unfortunately I have not had time to do one yet.
Thanks for the tutorial !
I just wondering how to switch between the composure result and the original image from the camera during game. In other words, a way to toggle visibility of each layers, but during game.
Have you got some ideas on how to do that ?
If you click on one of the composure layers in the world outliner, you should get a preview image of it in the editor screen. If you want to see the raw image (without chroma key) you need to turn off chroma key in the composure pass. I'll have to mess around with it a bit to see if there is an easier way.
Hi Greg, We are looking to add a bit of stabilization to the tracker, just to smooth it a bit as its extremely sensitive. I wonder how would you recommend approaching this? Would you add a 'tinterp to' node after the time delay in the BP?
I have messed with it a bit but haven't found a perfect method yet. A lot will depend on how much you are moving the camera. Most people find that the jitter is not noticeable if the camera is moving or handheld but it can show up when it's on a tripod. The biggest issue with smoothing is to find a method that won't cause a jerk when the camera transitions from stationary to moving.
Search youtube for "Virtual Camera Usability Improvements in UE4 (Tutorial)" and go to 9:25
Hi Greg.
Do u have a video explaining how and where you connect the real-life camera? and how can I change it for a web camera? thank u
I don't have a video about this but it is not too hard to do. The video comes in through a "media player" object. To use a webcam you need to set one up for it, then connect one of my composure "media plates" to use the video texture it outputs. Unreal has a short tutorial at the address below that should give you enough info to get your webcam working. Feel free to come back and ask more questions if this doesn't work for you. docs.unrealengine.com/en-US/WorkingWithMedia/MediaFramework/HowTo/UsingWebCams/index.html
@@GregCorson Hi, thank u for the answer,
I did figure out that it through the media player but couldn't make it work. And also saw that page on ue4 from your readme notes. didn't help.
Sorry for the delay, were you able to get that tutorial on the Unreal site to work at all? Just wondering if it doesn't work at all or if you just had trouble connecting it to composure? If you got a working media player you just need to go into the Media Plate in composure and set it to use the media player's texture. Let me know if you are still stuck
Hi Greg? How do i assign pucks to specific trackers? right now my tracker is moving plum line instead of camera. Also if u could go in detail how to output the composed output to a stream that would be awesome.
There are two parts to this, in steam you have to assign a "role" to each tracker and then map each tracker to Special_1, 2, 3...etc There are a couple of tutorials on the channel that show how to do this. Whenever you drag a tracker object into the scene you need to go into the actor's "motion controller component" and set the tracker to Special_1, 2...etc. Once you've done this the trackers should stay assigned to the right actors in between runs of Unreal, no matter what order you turn them on.
Regarding "output to a stream" do you mean just recording it or do you want to live stream the output to something like RUclips? For recording you can use OBS to record the window or you can send full screen output to your graphics card or out an HDMI output on your video card if you have one, then record it with something like an Atomos Ninja V recorder. I haven't worked out a solution for directly live streaming to RUclips yet.
@@GregCorson In The project file VivePuck 1 and VivePack2&3 are sharing same VivePuck blueprint and consequently same Motion Controller component. Is that where i supposed to change motion Controller setting? it didn't work when i did. Can u please let me know if i am looking at the wrong blue print?
Thank you for replying.
@@GregCorson Just found the second part of your reply)). Thanks. Regarding Output stream. i want to send it to graphics card output full screen. Lets say to my second monitor. Please advise.
Also in VivePuck it asks to set owner or motion controller will not work - maybe that's where my lack of understanding is. Can you please let me know what should i set it to and how?
Thank you very much, Great work! I have a little question. I use an APSC camera with sigma 30 mm so with the crop factor the focal length of my video feed is 45mm. Why when I set it to the Cine Camera the background is not matching? And I'm getting a lot of parallax...
You need to set the specs of your camera and lens in the VPCamera actors in the level. If these aren't right you will have the problems you describe. Select one and look in the details panel, there is a preset in "filmback" for APS-C. I'd suggest you try it and then look at the sensor width and height, if they don't match the specs of your camera, set them manually. Having the correct sensor size will automatically figure the crop factor for your lens. In the lens settings, set the focal length of the lens to the ACTUAL focal length marked on the lens. In my earlier tutorials I posted some links for how to find the entrance pupil (aka no paralax point or nodal point). There are a lot of tutorials on RUclips that tell how to do this for panoramic photography. A correctly measured camera rig with a correct Entrance Pupil estimate should solve this for you. If it doesn't, please feel free to ask more questions either here or in the Discord group linked on the right side of my channel banner.
@@GregCorson Thanks 🙏🙏🙏
hi Greg, I'm a big fan, and I just start to learn VP, one question: how to creat the garbage matte, I mean I don't know how to get this"FlagGreenSgreen" thing, many thanks
I'm not sure what you mean by this? In VPStudio there is already a garbage matte setup. You just have to position it in unreal so it lines up with your real world screen. You have to start by making sure you know where your room center is in the real world, usually by setting the HMD on the floor there, facing the green screen and doing a "room setup" for standing position. Then you can measure the distance from your HMD to the green screen and use that as a starting point for where you put the green screen in unreal. Then just go to the green screen object, make it a bit too big (so you can see parts of the room where there is no green screen). From there you should be able to line up the edges and reduce the height/width till you don't see the real world anymore. This is made a lot easier if you are using release 7 of VPStudio and Unreal 4.26 with the VIVE trackers running through live link because with LiveLink even when you are in the editor, all your cameras are tracked and showing up in the right places.
Hope this helps, if I haven't answered your question please feel free to ask more.
Oh, forgot to mention, since Release 6 I changed the mattes a little, you might want to watch that tutorial. Instead of one "FlatGreenScreen" object with both a wall and a floor there are now separate "MatteWall" and "matteFloor" objects. This was because a lot of people don't need the floor matte. Also it's easier to setup and align the mattes one at a time. The new mattes have the origin in the lower right, so you just use "location" to line them up with that edge of your green screen, then use the height and width controls in "details" to adjust the matte to the right size.
@@GregCorson OMG，it all makes sense to me after your explanation, thanks!
Hello Greg, I wanted to ask: When your actor walks into the scene, is there a way to get shadows onto the 3D objects aswell when you move around? Does that require motion capture to be on and tracking the actor to get shadows?
Shadows are a difficult problem. Whether you want the actor to cast shadows on the floor or onto something like a desk it is really tough to get them to look completely realistic. Many productions have actual physical objects painted green (boxes, tables...etc) to catch the shadows. Then an alpha channel is extracted based on the areas of darker green where the shadow is. Other people use normal physical props like chairs, desks...etc close to the actor they can interact with and cast shadows on, using the virtual production stuff for backgrounds. Sometimes the actor is tracked in some way and a fake 3d object is used to cast shadows. Basically, there are a lot of techniques that only work in specific circumstances, there is no one way that works with all setups.
@@GregCorson Thank you for the reply! I suspected as much that it would help having the fake 3d object tracked on and cast a shadow, physical objects greatly help. Thanks! I look forward to figuring out which method works best for unique situations.
The simplest solution is to have an actual real floor (not greenscreen) to cast shadows on. This eliminates any need for special effects work. A lot of productions use this trick, they will lay down a strip of carpet or floor tiles on a green floor just so people can walk on it and cast real reflections. Sometimes when you see people walking down a sidewalk in a video, everything but the sidewalk will be greenscreen! To find new techniques, I'd recommend looking at a lot of the VFX breakdown videos from various effects companies on RUclips. Stargate Studios is a real interesting one because they do both big expensive stuff as well as cheaper things for TV.
You can easily set your actor (ie after green screen removal) to cast a dynamic shadow onto the virtual scene using the scene light. It’s definitely not perfect but works pretty well. I couldn’t get it to work with a translucent material but works fine with a masked one.
Greg, can this be used with a free moving camera or must it be on a tripod for pan and tilt only?
This uses a Vive tracker to track the camera, so you can move the camera however you like including handheld.
@@GregCorson Fantastic. Now I got to figure out how to do all this so I can comp my real robots into a sci fi set! Thanks Greg keep these videos coming.
Cool robots you have there! I'll have to go through all your videos to see more! I have messed around a bit making 2 wheel balancing robots (segway style) to experiment with. Biggest one is about a foot tall but the design scales up.
Thanks! I have another one coming up soon where I add the arms on to the drink dispensing robot. I’m thinking about putting him against green screen and set up a sci fi background to shoot some fun shots after he is finished. I’m hoping to spend sometime to set up the Unreal VP studio. I’m sure I’ll have a few questions for ya as I learn.
Sounds cool! Of course there is always the risk people will think your robot is just CG from unreal too. ;-) Be sure to check in the Unreal marketplace at the start of every month for "free for the month" content. They have had some good stuff like the science lab. There's some good stuff in the permanently free collection too.
Hi Greg, Great work on the video man! this stuff has been super helpful, Thanks! My buddy and I are running in to a problem we thought you might be able to help with. We're good on all of the compositing elements but whenever we launch the game, our CG camera linked to the tracker doesn't move. the CG plate doesn't seem to update when the game is running. All of the trackers have been tested and work but only when editing. Any idea why the CG layer image wouldn't be updating during game? The only way we've been able to get it to work so far is to select the CG layer and pin it to another monitor, then select the live cam layer and pin it to a separate monitor and use OBS to Key... Thanks in advance for the help!
Select one of the vive tracker objects while the game is running, go into "details" find the motion controller component and select it. Are the coordinates updating? If they are not then something is messed up in your setup of the trackers. If the tracker is updating I'm not sure why the view in the CG camera is not. Did you build your own rig in unreal to attach the camera to the tracker? I don't have my setup running right now, I'll try to get it going in the morning and see if I can think of anything else that would break things.
@@GregCorson Thanks for the reply Greg! I think we figured it out. We didn't have the CG layer directed to take input from the Cinecam on the details tab.
Please take a step back on another video a few things I wanted to understand how you even set those plates w wall floor etc
You mean how to setup the greenscreen? I'm going to try and do a tips video soon to cover things I may have skipped over too fast. There is an older video on my channel about setting up the greenscreen that might help in the meantime. Or do you need more information on how to setup the actual floor and other parts of your virtual set? Please help me out a bit here and tell me what I've missed and I will try to cover it.
@@GregCorson Hi Greg, Is it possible to get a tutorial on getting the camera input via the AJA card (the set up process)?
Check out the AJA video quickstart on this page docs.unrealengine.com/en-US/Engine/ProVideoIO/index.html Be sure you have the right video format set in the plugin or you will get nothing. If you are not sure you can run the AJA control room program and it shows the format/framerate of the incoming video.
@@GregCorson Great, thank you!
@@GregCorson I stuck in the same problems here, is there a new video about how to setup greenscreen? thanks a ot
thanks a lot!
Greg, i cannot find in the blueprints how to make the tracking activate. I seem to have trouble having the tracking to activate, although if i make my own project with motion controller-special 1 I am able to. But with your VPStudio i’m not sure which blueprint makes it track
In my project, all the tracker stuff is in the "tracker" objects. One for each tracking puck. Set Special_1...etc in there as well as the tracker delay for that puck. If you find the trackers are not working, check the STEAM tracker roles to make sure they are setup right to send their info to the right trackers (there are two videos about this on the channel). Sometimes you have to set this up again when you change the name of the project. If one project works and another does not, this mapping stuff is probably why.