Can AI kill the greenscreen?
HTML-код
- Опубликовано: 17 апр 2023
- Can a color really beat the AI revolution? For now, it looks like it can.
Subscribe and turn on notifications 🔔 so you don't miss any videos: goo.gl/0bsAjO
The greenscreen is a staple of visual effects - and it may stick around even in the age of AI “magic.” The video above explains why.
It turns out that greenscreens, while imperfect, provide certain background separation benefits that are tough for AI to replicate due to the way it’s been trained and the limitations of available data. Preparation can help improve results, but this video shows why, ultimately, AI tools will remain one in a suite of options rather than a greenscreen killer.
Further reading
www.cs.unc.edu/~ronisen/
You can find more of Soumyadip (Roni) Sengupta’s papers here, including links to his various greenscreen work.
segment-anything.com/
If you want to try the latest breakthrough in image segmentation, Meta’s demo lets you upload your own images.
runwayml.com/
research.runwayml.com/publica...
Runway ML has a variety of tools that sit on top of AI infrastructure, allowing you to play with all sorts of applications. They’ve documented a bit of their process at the link above.
openaccess.thecvf.com/content...
If you want to really learn more, you can read this paper that explains image matting on an important dataset.
Make sure you never miss behind-the-scenes content in the Vox Video newsletter, sign up here: vox.com/video-newsletter
Vox is an explanatory newsroom on a mission to help everyone understand our weird, wonderful, complicated world, so that we can all help shape it. Part of that mission is keeping our work free. You can help us do that by making a gift: www.vox.com/contribute-now
Watch our full video catalog: goo.gl/IZONyE
Follow Vox on TikTok: / voxdotcom
Check out our articles: www.vox.com/
Listen to our podcasts: www.vox.com/podcasts
Thanks for watching! Join us every Tuesday in April when Phil takes a look at what’s going on with AI.
To watch more of our robot revolution coverage, check out our AI playlist here: ruclips.net/p/PLJ8cMiYb3G5ek1Ux66aJ_qWf6CfBaAkGG
@DavidLimReportyeah, as in next week. Commenting on what 'AI' cannot do is just about today, in the moment discussed. That's it. The public and media hasn't yet grasped the evolution rate of AI now. And still call it a pattern machine. They are talking about what the system was generations ago, when it started.
I think greenscreened footage will always be possible to spot until the technology can also match the lighting
AI relighting is in the works! I think Two Minute Papers has a video about it.
Clipdrop has an AI relight that’s pretty decent tbf
Ray tracing is doing just that
That's why they created The Volume
@@MatthewHoworko Hold on to your papers.
The Filmmakers who are choosing to still replace backgrounds in The Volume and other LED rooms are part of the Hollywood VFX problem. They aren't making enough of their decisions before filming, instead throwing more and more of the responsibility on VFX teams to solve their issues.
Yup, and they give the VFX team zero time to do it. They just scream "Fix it in post!" without having any idea what the words *really* mean. If all directors, producers and cinematographers tried to do what the VFX houses do for a month _(or six months would be perfect),_ I think movies would get better again instead of this decline of quality we've seen this last decade or so.
VFX are very overworked. It’s a very stressful job sitting in front of an editing screen for hours everyday.
@@angelgjr1999 Honestly as a VFX worker, sitting in front of the screen is the easiest part. The hard part is being able to turn around something high quality in short time frames. Some productions are good and allot enough time for VFX, but others are nuts and often exploitative or don't have necessary assets prepared. VFX artists can do anything 2d and 3d as long as they have enough time and money.
Exactly!
Change hair, replace hand, replace character with digital double, add weapon, etc.
All requests coming up in the editing bay, en masse.
just spent a day filming with a ton of rear projection... like Hitchcock would do in the 40s. But it looked amazing BECAUSE the lighting was accurate and the background image was meticulously crafted by an artist. My director made a huge point to make as many decisions before filming as possible and because of that the frames turned out so well. Greenscreen, AI, or even simple rear projection is a creative choice and it must be made in coordination with the story, the themes, the lighting, the sound, the cinematography in order to work well as an element of media art.
Why did they stop asking people to step out of the frame at the beginning of the call, like iChat used to do back in the day? Easy way to tell the AI what the background is
UX geeks thought users want filtering to happen instantly and this extra step was too disruptive?
That kind of approach doesn't work if someone moves their phone or laptop, either on purpose or accident. Then everything breaks down.
Thank you Phil! You always bring the most interesting videos on subjects I didn't even know I *needed* to know! And, _of course,_ a *big* thanks to Vox, *all* of you are amazing
how do i like this comment 100 times
In davinci resolve you can basically use a combination of magic mask and Keying colors and skin to get a near perfect key without a greenscreen. It takes some practice buy I can remove most backgrounds fairly quickly using this technique.
Solid tip. Resolve is moving in leaps and bounds with these features. It's what prompted me to switch from Adobe Ae - They rested on their laurels and are now being left in the dust.
Any videos or guides on this?
The lighting is the things that make it stand out to me
I recently discovered AI background removal in Filmora and I couldn’t believe the results, helped me with a poorly lit background that was obvious during jump cuts and I was able to replace it with a still image
This was great. Rip to Sheila Jr.
Sheila Junior had it coming.
Phil's videos are the best, the mix of information and humor makes them really stand out.
Who else here is amazed/shocked how fast AI is coming together in our world 😲
it's been developed for a long time
The real unappreciated change is the ability to store, share and evaluate millions, even billions, of gigabytes of data.
RUclips alone is something like 2 500 000 000 gigabytes of data.
Its always been there, the youtube algorithm, the insta feed, the advertisements on Facebook, it's just more visible
@@BarnabyTheEpicDoggo you think those are AI?
I’m terrified
There is a key difference that makes the comparison insufficient: traditional green screen has a permanent but static potential, while ai could be constantly trained until the perfect result will be achieved, but then you can apply even more features green screen can’t produce like changing the painted out parts to different things.
Why AI hasn't replaced GoT S8 already?
When I started in commercial art in 1976, we literally did 'cut and paste' photographic text.
wow you’re ancient
@@triniteemoore43 Ancient, yet still learning as 10 years ago I went to school to learn Adobe software including PS. Vote BLUE 💙
@@TheKing-qu8cm University of Minnesota in 1980, let me see what school could be like when teachers have earned their respect.
Mind blown 🤯
The Sheila definitely knows what it did!
3:46 It IS a hard problem. I bet sometimes even Ronnie has trouble separating what is Ronnie and what is office, causing him untold problems at home.
Man, this is really interesting to see honestly about this whole thing in the whole process of the green screen effect in a different AI green screen. The whole differences in that.
Great video! I cant wait to see how much these ai technologies improve over the next few years
Loved this video series! Phil is a gem
Green screen work has its challenges but if some simple rules are followed - things like making sure your key is evenly lit, working with your stage lighting to match your digital environments/ set extensions, proper tracking markings if your doing a 3D solve so “fixing” green screen is a lot lower on my priority list than having to roto when chroma key is not available. Tools like Mocha and Silhouette make things easier but it’s still very labor intensive to get good results, especially when manually tracking and doing roto work on things like hair and edges of objects that are highly detailed or have semi opaque boundary areas of transition. If AI could aid with those problems I’d be very happy…
Amazing editing and explanatory powers!
Loving this series!
Timestamps & key points by Tammy AI 🔥🔥🔥Enjoy!
0:51: 🤖 AI's ability to separate foreground from background is improving through segmentation and learning from manually created datasets.
3:46: 🎥 AI green screens have limitations and are optimized for speed rather than Hollywood-level quality, but can be improved with the use of masks.
7:56: 🎥 Visual effects artists often replace LED walls with meticulously traced backgrounds for better quality.
Why do you need timestamps on a fairly short video?
@@tomaccino playing with new technology. 😄😄
@@lindadawson902 Thanks for sharing! Tammy AI is very useful!
There is another detail to consider, and that is that the color palette used in the cuts must match the background lighting.
Phil is such a great host. Always love his segments!
Loved the little fact at the end - that was really interesting!
So no one is commenting on: “Must destroy Sheila Junior”?
I am curious about the shadow and reflection effect of a green screen. I recalled seeing a reflection of the people on the floor in a talk show while actually, the background studio is all green including the floor. How did they do it?
If the floor is reflecting the green, it can be replaced just like the actual green screen
@@DanielRieger I see. Thanks 😁
Very interesting video as an AI enthusiast, thank you!
"She knows what she did." (We used to lose our allowance. Now Sheila Jr. gets deleted from the simulation!)
I missed this episode of vox until now
Phil's line "we can get rid of Shelia Jr, she knows what she did" made me laugh so hard I woke my wife from her sleep thinking I was having some kind of deadly problem
I'm so glad nurses run on love of their job and *mostly* caffeine, because that joke was worth buying her an extra energy drink in 3 hours
The GOD box, unreal, and led walls are nice. But there is a difference in picture quality of items recorded, vs the quality of the walls image. They seem to be what’s in and the future.
Apple just released new AI tech called "FaceLit: Neural 3D Relightable Faces" which add one more level of WOW to a green screen effect.
I love all the recent videos 🤗💛
Love me a good Phil Edwards video, great as always!
Funny how the arts were once the riskiest fields to get into and now they’re the safest
*if you're a specialist **and know how to readily and frequently get work :\
that very last sentence about artists having a lot of time in their hands has gotta be wishful thinking. as any VFX artist will tell you, more often than not the deadlines they're given are borderline unfeasible.
Loving all the AI videos, Vox!
Saw this on my school board and it looked cool so I’m gonna watch it later
So where does things like Google Pictures "Magic Eraser" feature fall on that graph? Does it work well?
Loved this video. ❤
I've used RunwayML. As good as it is with basic mattes, it's terrible with hair edges
If I used runway for mattes, how could I go in afterwards and manually pick out the hairs? For me, I think in photoshop terms where I would take a tiny brush over a mask as well as slight feathering. Don’t know how it works in video.
Sweet Jesus, that background with the Sobeys is in New Minas, Nova Scotia 😳. How on earth did you come to select that!?
Address?
this is the kind of stuff AI should be doing. Not making entirely finished works
Why shouldn't it do both?
Burning a lot of processor for zero value add? Why bother?
@@switchdeck9164 one is trying to help an artist finish their work quicker and easier, the other is trying to replace the artist altogether. Try and figure out which is which
Man, people can do whatever they want with technology as long as it's not murdering people.
@cverse why should artists be exempt from their jobs being replaced by computers when no other profession is?
yup, rotoing of things like the mandalorian, but oftentimes you have really good info on what is in the background because that is a seprate video that can be reconstructed from the game engine itself! (worked on a few of these volumes myself)
Best solution.
Does any of the TikTok green screen filters allow for a difference matte, as in, it learns the background before the subject steps into the frame? That solves the laptop issue.
Need this for VIDEO!
I've been doing motion graphics and VFX for 20 years and I have not seen a seamless quick solution yet for high quality keying and compositing. As mentioned in other comments, the variables of lighting, cameras, files, and color grades will always throw this off.
Magic Mask in DaVinci Resolve is really really good.
Rotoscoping with the rear-screen for tweaking is still a win as you will be less likely to have "fringes", the composition is in the can, reflections, speculation and lighting are a done deal.
I feel like the best method would be to have an LED wall with a really fast refresh rate and a global shutter camera recording at high speed, so then you can have some frames be the subject in silhoutte. Of course this will require interpolating the frames to perfectly line up with the non silhoutte images, but ideally it'll be so quick that it's only visible to the camera so that epilepsy is ruled out.
I know they've done similar things with lighting on the Matrix and recently on Thor 4 but to be able to dynamically keyframe changes in lighting in post.
Great video!
Because it just started getting good and will only become better. It's already being used and will see more use in the future.
What is the editor shown in the video?
I remember the green screen tech in the early days had a similar problem where you'll see a green or white outline around the subject. I don't know what had improved to make the outline disappear.
Back then, in the early ages of digital cameras, the colour information was sampled in less pixels than luminance, basically it was a sharp black and white image overlaying a slightly blurry colour image. But since the greenscreen effect depended on the blurry colour information, not the sharp luminance values, so it was less precise. Modern (cinema) cameras sample as much colour as luminance, so that problem is gone. A good/bad example of these greenscreen fringes is seen in the StarWars prequels, since they were shot with some of the first professional digital cinema cameras
@elfrjzanti aliasing isn’t a property of real footage, sometimes it’s better to have a basic grasp of the tech before trying to educate others
@elfrjz I’m a high end VFX supervisor with 15 years of experience, I’ve used many softwares and I’m well known in the industry. Don’t be stubborn and get your terms right. AA has nothing to do with alpha channels. It’s a reformat filter.
Sheila Jr. will be missed.
Great video. As someone who has been mulling using a green screen or ai versus a natural background it was both amusing and enlightening to note that the experts featured in this video seemed comfortable using their laptop cameras with just their natural messy backgrounds and poor lighting 😂
You should look up the Lytro cinema camera that was abandoned when their employees went to work for Google. They made light field cameras that could calculate an object’s distance from the lens and remove background by thresholding distance. No green screen needed. I hope google does something with this technology but it seems to have disappeared.
Awesome video! Why is a Verge video on Vox though?
No, there is so much QC work we do just to greenscreen . Also don’t forget, how the shot is filmed too, having a perfect green screen background is rare
That last part, even tokusatsu had started using it these days, most noticeably in KingOhger
One thing people forget is the lensing around an objects outline.
They should try using switchable polarized lenses with those LCD screen walls.
Good job Phil!
OMG! The Sobeys & Tim Horton’s at County Fair Mall in New Minas, Nova Scotia! Go Nova Scotia! 🎉
I remember a ps3 game called movie scene or something like that that allowed it to remove the background without a green screen.
would xbox kinect tech possibly be related to this til tok green screen tech? or is it entirely different? 😅
What a great example to look at in order to learn about how machine learning works!
Exactly as you mentioned the future for roto/alpha depends on learning.
The process is soft key, hard key.. These are what can preform on industry Ai based already on nuke, silhouette.. Which have tons of data already.
What surely need to keep in mind some finaling always need to happen
Now what did Shiela Jr. do? 😰
I see a possible solution to the problem in the possibility of integrating the metahuman technology from unreal engine 5 into a real person in the frame. For example, to cut the outline of a person at a certain distance and replace this outline with the outline from the 3D model of metahuman, created by the person in the video. The same goes for hair. Replace the edges of the head with 3D hair that rubs against the movement of the person's head in the video. And the most interesting thing is the question of lighting. If you recognize the light from the backspace and illuminate the 3D model of the person, you can take just a layer of light and overlay it on the 2d person from the video to make it appear as if they are so illuminated. In short, if you create a 3D layer of the person, you can solve the green background issue perfectly.
AI has grown so much, it's hard to keep track of them.
and it's gonna grow even more and faster
Lollll “she knows what she did” 💯 😂
Real smart video, detective Gordon.
I need to know what Sheila Jr. did 😂
AI would love my shaved head. 😂😂😂
Lost it at the robot shooting the kid with lasers
Been wondering this myself
Well written Abhishek!
I'm sorry but I can't watch this video until you tell me where I can find the Minions/Guernica image from 0:24
The same algorithm to get the matte applied to enlarged portions and limited by clarity-resolution DPI of target video-output (not master) for real time performance.
Focal-length is a clue
Zoom and movements of previous frames are vectors
Stereoscopic "view" from other LCD-panel technology as a quick analog guide
After what people experienced from news and actors, they will be even more sceptical and cynical about the usual-suspects and will demand proof for every sentence. The pathological-liars are toast
The point at the end about the LED screens, even the “accurate lighting” isn’t always correct. The screens can not push a brightness equivalent to natural sunlight meaning if the suns the light source, it’s probably far dimmer than is required. They not only have to roto to clean up the background but to create mattes for colourists to boost brightness especially on the reflective surfaces like mando’s armour.
The screens also come with massive visual issues like moire patterns that are extremely hard to avoid. An awesome concept but still years away from what it is currently being marketed as!
You can normally tell when a green screen is used by looking closely at the hair as it's one of the hardest parts to get right.
The subtle display of what a "normal happy family" means. The art of training the human brain by continuous reinforcement is displayed.
Working on a hybrid green screen/projection technique. The flexibility in post of a green screen with the lighting and look of an LED wall. 1/100th the cost.
Oh, and it's 3D.
SHEILA jr™HAD IT COMING!!!
Low budget films can become top tier with the right amount of effort at this rate.
“She knows what she did” 💀😂
I like how to make this video 8 min long you guys added that last part lol
Sheila Jr. catching strays 💀
what about combining the two
What did Sheila jr do?
Loving these AI videos
Not there yet
Hey it’s New Minas Nova Scotia!!
Yeah didn’t expect to see that in this video haha
0:24 This is a masterpiece?!
Sheila hits 😂😂😂
0:25 Masterpices. Mhm, horrifying.
Depth sensors in cameras can also solve this problem. For example the Xbox Kinect
Green Screen: AI sits on my matte at my doorstep