Stable Video AI Just Got Supercharged! - For Free!
HTML-код
- Опубликовано: 17 фев 2024
- ❤️ Check out Lambda here and sign up for their GPU Cloud: lambdalabs.com/papers
📝 The paper "MotionCtrl: A Unified and Flexible Motion Controller for Video Generation" is available here:
wzhouxiff.github.io/projects/...
Try it out: huggingface.co/spaces/Tencent...
huggingface.co/spaces/Tencent...
It is also open source - run it locally:
github.com/TencentARC/MotionCtrl
📝 My latest paper on simulations that look almost like reality is available for free here:
rdcu.be/cWPfD
Or this is the orig. Nature Physics link with clickable citations:
www.nature.com/articles/s4156...
🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Alex Balfanz, Alex Haro, B Shang, Benji Rabhan, Bret Brizzee, Gaston Ingaramo, Gordon Child, Jace O'Brien, John Le, Kyle Davis, Lukas Biewald, Martin, Michael Albrecht, Michael Tedder, Owen Skarpness, Richard Putra Iskandar, Richard Sundvall, Taras Bobrovytsky, Ted Johnson, Thomas Krcmar, Tybie Fitzhugh, Ueli Gallizzi.
If you wish to appear here or pick up other perks, click here: / twominutepapers
Thumbnail background design: Felícia Zsolnai-Fehér - felicia.hu
Károly Zsolnai-Fehér's research works: cg.tuwien.ac.at/~zsolnai/
Twitter: / twominutepapers 
Наука
![Dimension 20: Never Stop Blowing Up Trailer [Dropout Exclusive Series]](http://i.ytimg.com/vi/xYeXpbHnGEM/mqdefault.jpg)
Nothing is as good as Sora, however, this is something that we can all try right now. So cool!
Well we dont realy know how good Sora is because its not public.
Sora has tons of power behind, is not that much ahead of the rest.
@@nox5555, nonsense. The live prompting examples that Sam gave on Twitter provided all the proof one needs to conclude that it is state of the art by far.
The examples demonstrated leading temporal consistency, sequence length, resolution and preservation of fine details as well as more accurate physical modeling (less hallucinated fingers & hands, better physics between light & material) as a researcher in the space is demonstrated clear state of art in several dimensions without needing to be available to the public.
How do we try this stuff? Does Stability AI have any code samples, or something we can download?
@@nox5555 on the other hand, it's not public because it's almost identical to real video. They need to put restrictions in.
Sora jumped about 4 papers. Everyone else is going to have to release anything they've been holding back and triple their efforts for further breakthroughs.
Yeah, but Sora is owned by private company. Stable Diffusion on the other hand is open source, which means it contributes everyone not only one corporation and its clients.
@@chanpasadopolskaand if it can't keep up it will be left behind.
@@chanpasadopolska how can i get educated about Stable Diffusion? is there have any beginner friendly courses?
@@chanpasadopolska I only care about results. This is good I guess to make things that are censored by companies, but otherwise, I only care about what gives me what I want, not whether it's open-source.
@@chanpasadopolska yet it's way ahead.
I love that the AI thinks shutterstock watermarks are part of our world
those content copyright games and stock images. My god, that is exactly why I love AI.
It's hands-down proof of Elon Musk's claim that these companies have violated copyright laws to train their models.
i dont care@@mvmlego1212
@@mvmlego1212 Who cares
@@user-my3sp4oi4r -- ...presumably, the visual artists who will never work again because a company stole their artwork to create a contraption that will put them out of business.
Its very encouraging to see other AI models being improved despite the Sora breakthrough.
Sora works but is being kept to a limited customer base to protect against mis use. Open source will get there eventually but better networking stacks for training needs to be adopted to scale to this level of problem.
@@Iswimandrunthey’re not only limiting to prevent misuse , it’s for $ too
@@Iswimandrun
In history, open-source products rarely achieve the same level of success as commercial, closed-source products. This is because open-source often means it's created by people who don't make money from it. And in this world, you need money to live. It's money that unlocks your full potential.
@@ykwtfgo tbf the government would shut them down if they didnt do their will
dont worry, open source takes a while to catch up but it will.
I'm all for supporting the competition. We need a vibrant range of companies all competing. No one wants a Sora monopoly and the dire consequences that come with that over time. Come on little A.I's, you can do it!
Mf do something with yr life, try to create art yourself
I don't think you comprehend what Sora is. It is racks upon racks of H100's, if you had that horsepower you could have Sora at home rn. But noone does, so only ClosedAI has it for now.
Sora Is boring, look at all those restrictions. When they release it, well, dumb people will be excited, but what kind of interesting things will they do with a so restricted model? 🥱
Stability AI has lots of money, tons. They're already figuring out their next move. As Emad said, they're cooking something...
Where is my comment?
@@OhioNPC911RUclips and everyone you know is out to get you
This paper is more about the amount of control that is able to be expressed in the output rather than simply the "quality" of the output (which Sora clearly exceeds). It's a great demonstration of the new types of things we'll be able to do with this technology as it develops.
Excatly, in SVD1 you had no control about any movements and it was trial and error. Now there is much more control about what should happen in the video. I'm sure quality will also improve significantly in the future.
Yes. Sora has different capabilities.
@@adrianfiedler3520can you imagine what we’ll be able to control just two more papers down the line
people forget that sora isnt available right now and was revealed early to counter gemini 1.5, and that stable video is completely open-source. people also forget that openAI has microsoft's financial backing and that stability AI is a start-up company. this is incredibly promising news because we dont need to run prompts/queries through openAI's API to get something like what is in the video.
All these smaller AI companies will probably be eventually gobbled up by the bigger ones. The smaller ones won't be able to keep up and the bigger ones will want more talent.
Well said. It’s also amazing to see promising competition because we do not want a microsoft monopoly on AI. Also worth noting, as Károly mentioned in another comment, what’s new here is something, that sora can’t do, which is fine controllability of the objects and the camera in a video
@@jacobnunya808 unlikely for stability AI, as their vision directly clashes with openAI's. together they would probably make something fantastic, but i dont think current leadership at stability would stand for it
@@moritz584 exactly. it flew under the radar, but 6 days ago stable diffusion cascade was released. it brings it up to par to midjourney, which costs USD per month pre-tax if you dont want people to see what you are doing, with multiple features not found in openAI's DALL-E. stable video and cascade are both MASSIVE for people who do not want a monopoly from corporations that want to endear ONLY to the mass public for profits
It’s very impressive what they’ve done but it’s also the shamelessness of the shutter stock watermark is insane 😂
seeing the shutterstock logo is funny because I remember when older video AI's also did that like that one where will smith ate spaghetii
I remember when ai interpolation for higher fps was groundbreaking
I mean 3x higher fps was pretty cool. Made ray tracing practical.
It kinda is groundbreaking. But it also kinda sucks. You can notice the latency, especially in some games. I don’t use it in The Finals, because it’s awful
I remember when AI had to cheat to beat you in games.
All this shit is about to fuck us all up.
Will save companies a lot of money with special effects.
This channel is the Nostradamus of the tech world. Ive been watching for years and everything that has been ahow always come to fruition. In games, video, and ai. Obviously these arent predictions but current research that gets used eventually. i just dont know any other channel that accuretly shows the future of tech like two minute papers.
What if I told you there is no doctor Károly Zsolnai-Fehér. It was a clever name AI chose.
4:07 funny how the generated image has the shutterstock watermark :P
I guess the people who made the ai didn't want to pay for a bunch of stock images
Whose gonna pay? LAION?
THANK You for making this video. I was hopeless in trying to find an freee updated version of ai text to video.
This may not be as technically impressive as "Open"AI Sora, but for one, it's still early days, and two (more importantly) it's freely accessible and here now! I am unimpressed with Sora solely because it's going to be "cool kids club only" material where we uneducated peasant classes will never get access to tech like that. It might as well be movie hologram technology in my mind. Sora gives me a "Huh. Cool. I guess..." feeling, quite honestly.
I'm glad the open source community is making leaps and bounds ahead for this technology. :) I wish them nothing but success in bringing technology to the everyman! Thank you for covering this!
actually very valuable video. We all scrolled through Sora and forgot about the other projects. Thanks bro ❤
Open-source will inevitably be equally good as private. As AI steps into chatgpt 7-8 it will be used to develope opens open-source clones and open-source video models.
That is not necessarily true if enormous "open-source" computing power is not available.
This is great. I think having direct camera and object control is more important than trying to understand the same command in text. For people seriously trying to get a video scene they need, knowing the model will nearly deterministically follow your order is much better than "suggesting" the model to do the same and hoping it works.
It's so good that it's open source. Big companies having a monopoly on these incredible world changing technology would be an even bigger problem than it already is
When we gonna be able to revive old cancelled animated shows with this tech?
2025.
"Hey machine, please generate Firefly season 2. Thank you."
Never, the corps will pay wall it
@@soulsmith4787
I’m with you …
no idea why it didn’t last longer …
@@JohnKerrashVirgo how they gonna paywall open source? lmao
1:40 actually I believe the correct term for some of those camera motions are pedestal up/down and truck left/right.
It feels like the singularity got significantly closer.
we have mere seconds
Can't wait 🎉
🤣 I also believe a mini - AI - singularity is taking place now, so crazy that just a few hours later Stable Video AI releasing this more advanced model of theirs, it feels like this AI revolution is getting out of control and getting faster and faster and more and more crazy and advanced by each day/hour that passes.
not even close.
Singularity: The single infinitesimal point of mass of a black hole where not even light can escape where time is effectively frozen by the sheer force of gravity (also would kill people who touched it).
AI bros: this sounds like the word I want to use.
The accent and enthusiasm of the Dr Feher. makes the videos 3 times better! I held on to my papers!
The speed at which this is developing is scary
It is awesome!
Such an incredibly fast race between all the AI projects! IIt's exciting and scary. Wow.
The text to video at the moment is when we were with Dalle-2 and Midjourney v1-2. Wait a year or two...
It's all about accurate and highly granular segmentation, which luckily is one of those things that can use its own output to improve itself
As I've said many a time, this'll all be just a forgotten about part of a Holodeck one day, but how cool is it that we get to see this technology emerge *today* 😊
Noooooo, your video is only 21 minutes old and the huggingfasce webpage is already too busy 😭
Is this video recorded before sora? you don't mention sora at all, doctor.
You are indeed right, my apologies! Right as I was done making this one, Sora appeared and I could not believe my eyes. Luckily, this innovates in a different direction (controllability) and is free so I think it is a fantastic value proposition to show it to you Fellow Scholars now.
@@TwoMinutePapers That's completely fine. I just wondering why you doesn't mention Sora at all. You're right, this paper have its different inovation. Thank you for delivering this paper to us fellow scholars 😁
Europe likes to have their own A-Bomb security. "What a time to be alive" @@TwoMinutePapers
What a time to be alive
How to have it locally on Mac? Is there something like DiffusionBee for image generating?
To be far, finetuning 1.5 (and now SDXL) by the community led to insane improvements in image diffusion, so I assume the same can be said for video diffusion.
Yes! What a time to be alive
If you showed me this last week I'd be so impressed. But after seeing Sora...
Well, how many ppl have currently access to Sora? And how many people have access to Stable Diffusion?
The walls are a massive upgrade. Samwise was wise to hire you for a redecorating!
I love lore stuff like the reflective pool. If only we had half slabs of glass that you could walk on but put stuff like end robs and skulk under it. We need more blocks Minecraft!
Maybe the ender city needs a temple to give chorus fruit sacrifices to their moon godess?
The cats and zebras walk by cloning their legs tho :D
From my understanding Sora generates Unreal Engine scripts that then generate the images and video hence know weird warping etc
Thanks Ren
Imagine giving a whole comic book to this ai and having it animate it. Cool
AI is advancing so fast this technology got surpassed before this video was even posted
It's funny to think of how valuable this channel will be as a historic document of AI
What is your linguistic origin? Such a unique and consistent rhythm and cadence
Soon you'll be able to feed a novel into a prompt and ask it to make a feature movie out of it. Imagine that.
put the bible as prompt! ...will be pegi18 for sure ^^
Are you a kid?
So no one gets to enjoy the process of actually making the film? Sounds incredibly boring and depressing to me... I really hope that's not our future...
What I really want is to have brand new episodes of a TV series that is no longer made
@@aylameridian that's not true. Who ever wants to film in traditional ways can still do so. It's just an additional way of creation.
The Shutterstock logo on the outputs kills me lol
What a time to be alive..👍
Sora is in another level, the way it’s able to create simulations of the real world is 🤯 I think open AI is using a totally different approach on training their new models, I wouldn’t be surprise if they are using unreal engine to teach the model to have an understanding of 3D and light, they just need to teach the model cause and effect and it will be perfect 😅
you can read their paper on Sora
The difference between Sora and Tencent SVD is that Tencent SVD actually exists and is not a scam of dying startup
Two (or) more papers down the line we will have video output from generated high-detailed 3D worlds
I think the trick to SORA is that it has an autonomous GPT agent governing the diffusion process on a minute scale.
Now one thing came to mind. In the old times, whenever something new came to the market, it would cost a liver to have at home. But now, i just can download it nd use it the way i want to. I just need to wait a few seconds. If that's not amazing, nothing else can be.
We just need to be afraid of technology bombardment. Cause the more we see thing, less strange and less amazing it might get. And i don't wanna feel this way. It's like if we had teletransport machine. If we had it, after a few months or years, it wouldn't bother you or make you be amazed the same way anymore.
it's an interesting question if AI should generate into a traditional euclidian 3D cad space and render or if it should stay in a pure 'live' neural space. I think I have the answer actually.
We enter the era of quantic pepars both the last one and 2 papers behind the last one ! This is going so fast !
Controlability engineer = cameramanof the future
Two Minute Papers bringing back Slashdoting into fashion.
I think there's a huge need for open models, or at the very least "open weight" self-hostable models.
While it's incredibly cool what OpenAI (and Midjourney, Google, etc) are doing -- We need products which work in a datacenter of your own choosing, or even locally on your own consumer graphics cards. Especially when you want to combine image, video and LLMs with potentially sensitive customer data, it is essential that we can take security measures appropriate for the use case.
Are you even following along with what's going on with local models
@@jerbear7952what’s going on?
I can imagine a future where we put in a prompt, and out comes a complete world that we can explore using the camera.
That didn't take your entire imagination did it?
The application I am looking for is to enhance the realism of 3D scenes that I make myself without creating anything new but only giving a perfect color grading and perfect shades. Could that work?
"Pan Up, Pan Down" kills me as a camera man.
What is it then?
@@Tyrone-Wardpanning would be changing the angle I think, what this is doing is moving linearly on one axis. I guess you’d call that move up/down
@@Tyrone-Ward In film lingo you "pan" left/right and "tilt" up/down
@@Tyrone-Ward Tilt up, Tilt down
But you're a camera man. They can't die.
I can't wait to see this used to make a Bad Apple video.
This is what i want. Free models i can run on my computer.
That shutterstock watermark is an easy loss in court
SVD is for research only so it is not same as free. Since you cannot monetize, it's use is fairly limited. Hopefully Stable AI will bring us commercial license soon for video.
It's great these tools exist, but there needs to be a new file format specifically for AI generated video which forms the entire video using a new type of encoded pixel called AI pixels or an AI based vector file format for video or images. That would make it clear when any piece of video content is created or derived from AI generated content as these tools become widely adopted. Since each pixel is an AI generated pixel type, it would not be possible to remove the indicator that this was an AI generated file because each pixel is indicated as computer generated.
Well, doc, it appears we’re too late already; the demo is definitely functioning like the ticket sales portal for a David Bowie resurrection tour.
Seems you made this video before the Sora one :D
Geralmente fico impressionado com o que é apresentado aqui, mas como já vi Sora, o interessante desses papers é que são acessíveis
From my point of view, it's just insane what's happening right now. It's happening so quickly that people who thought they had found a new income and could build a business with software for a feature find themselves a month later in a situation where everything is obsolete. Not to mention the large number of people who haven't even got rid of the habits from the Stone Age. It's fascinating, yes, but I think we're heading into a time that's even darker than we previously thought. I almost wish that a independent super AGI would take over as soon as possible...🙈
Damn, imagine being an artist
Shutterstock won’t like it
This acceleration looks like the proximity of the singularity.
With SORA out there this seems like news from a year ago.
This technology is working out like VR, miles of potential but unrealised or inaccessible.
To be honest all of these videos about ai video on this channel feel the same. Can't even tell the difference any more.
First Sora and then this. I really want to become a professional animator someday. I hope AI doesn't take away the career of my dreams from me, otherwise I don't know what I would do with my life. It's so impressive and depressing at the same time.
What this tells me is just how far advance OpenAI SORA trully is. This video basically shows a new paper state of the art, but here comes SORA on a literal nother level than this.
At this rate, Unsplash, Shutterstock and other stock image, video sites are gonna go bankrupt 😂😂😂😂
I'm waiting for the day I can upload an entire book of my choice and then generate an entire movie.
Do you speak like that because you are the firts ai that made a RUclips channel on its own?
Or because you translate and then read?
(Im 100% respectful and serious)
It's his accent, he's not an AI.
@@NutrejaSFD LMAO
Hes been on the platform long before chatgpt got announced, hes legit
@@NutrejaSFD no it's not just the accent it's also the way he speaks like he says every sentence the first time in his life, smartass
@@makeitraindom1634Hungarian accent is like that, and his is quite strong. I am hungarian myself so i know how it sounds. He is not an AI.
What a time to be alive! 😀
I suspect you won't be saying that 3 years from now and nobody has any work. The entire film industry is about to lay off everybody, the entire gaming industry is already in the process of laying off everybody as is tech sector laying off hundreds of thousands of people. Can anyone explain to me why this is a good thing??
@@aegisgfx I was just humourously referencing the uploader's catchphrase, it wasn't meant to be a treatise on the future human civilisation.
@@jayaybe1 I'm aware of that. Regardless, we will all be starving in a few years while openai will be worth 80 trillion dollars. All of this makes no sense
@@aegisgfx I think you are being somewhat over the top, but yes, there are some serious problems coming. Software engineers will be first to go as the code output is already fantastic (I am using GPT-4 on a project right now - sure, it makes mistakes, but it is correct 8 times out of 10). Then it will be the turn of the creative industries - first writers and photographers, then film makers, followed by actors and singers. There is going to have to be a massive shift in how we think of work and whether we need a universal basic income implemented. So, not quite the disaster you seem to suggest, but we are in for a rough and scary ride.
@@aegisgfx Seriously, I do share your concerns. Governments cannot and will not let that happen. They'd be strung up from lampposts. I'm no communist but the wealth will have to come from somewhere to pay for the 80% unemployed or whatever it is.
Maybe governments will seize control of AI companies citing national security and redistribute the wealth through universal basic income. Who knows?
I don't know if you follow David Shapiro's channel but he is excellent. Not just bringing AI news but also looking at the philosophical and practical outcomes for civilisation. Please check him out.
All the best 🙂
That shutterstock watermark is gonna get this killed.
I just yesterday watched the video about Sora and it just underlines how much better the Sora is right now. However, the difference is that Sora is locked into a lab and we can actually use this one.
lol all those shutterstock watermarks in there. :D
Commence the scholarly stampede!
holy s**t that escalating fast
Did anyone get this to work? The local is not working cause of dependency errors [some modules not even available] and with the API I get code errors
Just a few more problems and puzzle pieces to fit together and we have an AI rendering engine
Good progress in there paper. However Sora seems superior! Are there any benefits, using this technique over Sora?
It is available now for free.
In the comments it says he produced this video before Sora was announced, so that's why there's no comparison
hello 👋 i just saw Amazon Fire Stick has some kind of Ai ambient background generator
But when can I give it a video and have it give it's version of it. That's the real mind blower. Sora, showed it can do that.
It’s with no small irony that these are not Zoom-ins, they are “Dolly In” or “dolly out” in camera movement speak
Woop!
What is a paper???
You know technology is moving fast, when current models are already way better than the last video you produced on the subject.
Sora's leap forward means we're all stepping up our game - time to unleash our potential and push even harder for groundbreaking discoveries!
Anyone got a link to run this kind of stuff?
Now more stock images and videos in company presentations.
This'll be my biggest use. My corporate blogs are gonna start looking awesome lol
cool
At this point I am convinced AGI will happen in about 5yrs
"scholary stampede" 😁
Was it all trained on Shutterstock images? 😄