Stable Diffusion IMG2IMG Settings Pt. 2 (Consistent Animations!!)
HTML-код
- Опубликовано: 12 май 2024
- By popular demand here is how I was able to get smooth and consistent animations using Stable Diffusion.
HOW TO SUPPORT MY CHANNEL
-Support me by joining my Patreon: / enigmatic_e
_________________________________________________________________________
SOCIAL MEDIA
-Join my discord: / discord
-Instagram: / enigmatic_e
-Tik Tok: / enigmatic_e
-Twitter: / 8bit_e
_________________________________________________________________________
NerdyRodent:
/ nerdyrodent
Aitrepreneur
/ aitrepreneur
Midjourney Artstyle
mega.nz/folder/Z0xS1BpI#S40xU...
Timecodes:
00:00 Intro
01:03 Preparing video frames
01:40 Stable Diffusion settings
03:39 Working Script menu
06:31 Experimenting with results
08:20 Exporting and renaming frames
08:50 Importing into AE
Watching the beginnings of a new art form emerge these last few weeks. We're at the cutting edge, thanks so much for putting the effort in to share your techniques!
🙏🏽
Basically it's a rotoscoping setup for much cheaper and quicker than the traditional methods.
Seriously, this is an artistic revolution, the likes of which we haven't seen... ever!
Great tips! I hadn't thought about using the frame blending method to smooth it out, but that makes sense. It also makes me wonder whether using optical flow might help sometimes, too. But in your example, when you hit that frame blending tab, it really made a difference. Thanks again!
Thank you so much for taking the time to help us. You are a blessing to humanity brother. I mean it. ❤
The img2img alternative script runs the model in reverse to find the noise pattern that would generate an approximation of that image and then uses it as seed for the actual generation. So it is a way to do fine adjustments to an image. Also it is only precise in the euler sampler.
Thank you for sharing this info. It’s good to know.
I love you for making this video, thank you so much
Thanks! I've been searching how to get it and this is brilliant :D
Dude really appreciate your videos. Awesome work and thanks for sharing your knowledge on this. This tech is so ground breaking ...
Thank you! I agree is ground breaking and…… it kinda scares me 😂
This is the first video I stumbled on for this channel. I can already tell it's always top quality 👌
👍🏽
You can control the background by reducing the "denoising strength", but you will lose the changes in the character. I watched some other videos about video generation via img2img but yeah, the "frame blending" is the catch for smoother non-stuttering AI transition videos. This video totally dope!
Thanks for sharing it 👍
Thanks for the tips!
You're awesome and friendly. Thanks for the video
well done video
honestly I love your "how to delete every second file" technique. it's such smooth little hack.
also never new about mass renaming.
the stable diffusion stuff is cool too tough :P
THANKS DUDE
Dope 🙏🏽✨
A few other tricks that could help;
1- reducing the number of frames without deleting every other frame; you can do that in adobe premiere, after effects or whatever software you use to export into single frames. You just reduce the frame rate before exporting.
2- Use DAIN ai frame interpolator or similar, so you can use with even less frames per second. Then input into DAIN and it will create the in-between frames even smoother.
3- 2D animations are usually made with 12 or 15 frame per second. So you can reduce the jittery-ness even more by just reducing the frame per second to 12 or 15 fps, and it won't look weird at all. Because animation audience are used to reduced frame per second videos. For example Rick and Morty characteres move at about 12 fps.
I am an animator and thanks for your videos. I am really excited to try this technique out and make some content.
Thank you for the tricks! I’m definitely interested in reducing frame rate to something 12 fps and testing that with frame interpolation! I will definitely try that next time!
awesome video Mate 🙂
Great video!
Bruh that was the most entertaining n original intro 👍👍
Great video ❤
Very useful ! Thanks !
Thank you for sharing all of these informations.... God bless you
Nice Tutorial. Subscribed. Good luck with the channel!
Thank you!
pretty cool!!!
lmao look whoz here
maybe next video is ai based triology
Good job man, thank you for putting that out !
Thank you! I appreciate you checking out the video.
great video bro
works, chock-full thanks!
Deleting half of frames really game changer . Nice and wonderful trick you `ve been share in this video . Super thanks !!!
Thank you!
This is so cool
Looks pretty good
I love your channel !
🙏🙏
you rock!!!
Keep making the content bro
Thanks! 🙏🏽🙏🏽🙏🏽
cool stuff, thanks for sharing! quick tip with the first part of the video: you can set different frame rates in your AE comp on the original footage (8, 12 etc) in Composition Settings, then export picture sequences before you plug them into SD. this way it's easier to play around with lower framerates instead of having to delete them manually. Instead of frame blending, you can try the Time Warp effect in AE, which is pretty much the same just with more settings. But what I would try next, is dropping the video frame rate to something like 10, then using an AI frame interpolation algorythim like TVP instead of the built in AE one. Those algorythms are trained for cartoons and will give a lot better results than the pixel morphing AE effects. this is just theory though, I'm yet to try this next week.
Great job my broër°°°
Thanks bro!!
Thank you
thanks ma a brotha
thank youuuuuuuuuuu. i found your channel. been looking for this style, cant find. i saw this style from WebUigirls, but they dont have a tutorials. though they used LoRA model. but hey thnk this is very helpfulllll! more power!
I've been waiting for this one. I don't have nearly as dramatic subject matter, so my first attempt will be turning my hopping bunny into a hopping potato.
😂 nice
thanks!
So interesting dude, well explained too. At least for an AI beginner like me it was easy to follow.🤙
thanks ^^
Thanks!
Thank you!!!!
gracias!
Dude! your 0:37 animation just got shown live on stage at the Stable Diffusion presentation today. Look up "Stability Diffusion announcements" from Robert Scoble. It appears at 50:27. Congrats!
Wow!!! Thats so cool! I’m so happy that people are showcasing what I’ve work on as an example. Thank you for letting me know about this, definitely made my day!
@@enigmatic_e When I saw them show it on stage, I was like, OMG I just saw a video about how this was made. I had to tell you.
Here is a tip: create 12FPS composition in AE when you are exporting your video as PNG sequence and it will make your life easier to import it back again and you don't have to rename files.
Note for people who don't work with video, "12FPS" is only true for 23.976 FPS footage. Essentially half your normal frame rate in the composition.
Maybe also worth checking out Pixelmotion blur in AE. Double the length to get original time-frame back (from deleting every 2nd frame) keep frame blending and also add pixelmotion blur. It interpolates the expected blur on moving images to bring back natural motion.
Nice, is that a plug in already in AE?
@@enigmatic_e Yes it's default in AE.
Great video! Very useful tips! Could you tell us what is the framerate of your original video, and once you import your custom png's what's the "assumed framerate" in after effects? Thanks!
If you open your original clip in premiere,create sequence from clip, then go to sequence settings, you can drop the frame rate from what you have it to less, say 24fps to 12fps, then export as png img sequence without having to delete the pictures...
so im curious about your framerates on this. Is the initial video at 30 fps for your png sequence, then you delete half, then take the you ai rendered sequence back into AE, but is it now faster than the original video? or does the frame blending change that? Going to try some of this out now, I like seeing this difference between the more jumpy part, frame blending is a cool trick, somehow never noticed that box was there
in after effects you can export your original frames at a lower frame rate - which should save a bit of messing around. Also, try using timewarp instead of frame blending. This uses a softer morphing technique between frames and you'll be able to get away with doing maybe 1 frame in 5 rather than 1 in 2 for a much smoother animation.
also, how about separating the background and foreground so you can give them different settings. Maybe you could even motion track the background, export and run SD on just one frame of it (with some outpainting to extend the edges) then use the tracking data to animate that background. That way you'll have a rock-steady background painting.
Finally, experiment with a "difference matte". This is a great way to key out only those parts of an image that stay the same between frames. Use a difference matte (slightly blurred) to create a b/w mask. then have after effects only "repaint" the sections of the image that change from one frame to the next.
That way, if you've got (say) a closeup of a face, speaking, only the mouth will be redrawn for different positions, leaving the rest of the face untouched.
Thanks for advice. Will try this!
So the whole video has to already be in a stylized frame? My video has a lot of different people and scene changes. Because what if I only styled the first 10 secs will eybsnth now to stylize the rest of the video into like a drawing. Or will it only stylize the frame I gave it. I have a 5 minute clip I want to stylize but I only have a stylized frame for the first 10 secs? Will eybsnth know to stylize the rest of my clip because it has a lot of moving parts it?
que hermoso video se me enciende la imaginación gracias por las clases... pero el programa donde esta?
Thanks for that. Also try flowframes for smoothing the animation
Hey, this is fantastic! Thanks for putting this together. You should drop a link in for aitrepeneur's video about training styles in your description, he'd probably appreciate it since you reference his midjourney training. Would anybody by chance happen to know if that frame smoothing option or something similar is available in Apple Motion? I'd like to give it a go, but I refuse to give Adobe a lead cent, screw those guys.
you can use optical flow in fcp
Could “frame blending” be another name for “image interpolation “?
Cause that would make perfect sense why it becomes so smooth and retains the consistency longer. image interpolation is an amazing tool that I rarely see used. Basically predicts the frames in between two separate stills. It can easily turn a 30 fps vid into a 60fps. It works wonders on old (and new) claymation videos!!
the original prompt is for what the original image is supposed to be. And the prompt at the top should be what you want it to be.
Very helpful! Im rendering a long video now and Im going to try this. I do have a question. Ive been shooting my videos in 60fps in the hopes of having more smooth end results, but if this video is the case, would you suggest shooting in 24 or 30fps and still deleting every other frame?
You can shoot in 60 fps and test results. You can always remove frames after you want to.
thanks for the tutorial~ by the way, what device do you use for lighting the background and yourself it looks cool~
Thanks! The lights are called CAME-TV which aren’t sold anymore. But there are plenty of RBG saber type lights out there.
@@enigmatic_e thanks a lot~
this is funny because I was wondering if I should remove every other frame and then use Topaz Video Enhance AI to do a slow mo version of my video, which would then use AI to fill in the frame gaps and then use Premier Pro to speed the video back up ... but watching your video, you've found an easier method
Thank you for this video! I have an error using img2img alternative test "OutOfMemoryError: CUDA out of memory". What can i do with it? I can't buy a new videocard as for now
You can also use SD to style animations you render in Blender. SD is more creative and diverse than EbSynth.
thank you , also I wonder if you can do the same on google colab.. maybe slow but maybe not .. most of people use colab
Yes you can technically use this version of SD using Colab. I show how to use it in this video ruclips.net/video/qmnXBx3PcuM/видео.html check the second half of the video.
AI Researchers: Let's have a model with temporal consistency
Creators: Blend the frames 😂
You can also do another thing, increase the fps to like 120 and then slow down video, smoother video with overall less jittery video + spamming de-flicker node on davinci resolve
do you have the disco diffusion style for this? I see that in the description there is only the one from midjourney.
Thanks for the trick on deleting every other frame on windows. I wasted so much time the other day doing it manually. 😪 Have you tried flowframes for the frame interpolation?
Glad i could help!
No i havent but i heard a lot about. I will try it out.
Maybe image morphing could help? There is a stable-diffusion-morphing repo on Github...
When I delete the odd frames and then bring the even into After Effects or even Media encoder it is showing color bars or media off line for the deleted frames. Can anyone remedy?
I have been using SD v1.5 and it is SIGNIFICANTLY better than 1.4. In fact I now prefer the overall quality over MJ. But since MJ uses SD the next release of MJ v4 will incorporate SD v1.5. Things are moving so fast.
ouch, any way to blend frames in blender?
Could you edit video like Second gear RUclips channel?
Hi please tell me there must be some way to install im2img into the MAC iOS system 😢
Otherwise I need to buy a new windows computer….no…
what frame rate are your videos set to ? say if you have it set to 60fps and do every other frame like you suggest, would changing the frame rate to 30fps and doing every frame result in the same smoothness?
Mmm I think 30 fps works slightly better. 60 might be too many frames in a short amount of time. Test it and see how you like the results.
since you have used both, what do you think is better, img2img SD like this video, or using EbSynth? pros and cons to both?
I would say using them together creates great results when there are subtle movements. SD img2img is getting better though now with ControlNet. Have you heard of it?
@@enigmatic_e no i haven't. imma check that out
nice one! where/how did you get that art style exactly?
Artstyles can be made by yourself but it take a lot of steps. AItreprenuer has some tutorials on how to do that. ruclips.net/video/tgRiZzwSdXg/видео.html
@@enigmatic_e thanks alot mate!
was there a reason you didn't post your negative prompts into the Script settings?
I wonder how well the blending would be if you ran the images through a RIFE or DAIN
I was thinking this too. I doubt the results would be perfect, but I think there's a strong chance they'd look better than AE's frame blending. I'm also curious how Adobe's other frame interpolation methods would perform. I'm guessing worse (in this situation). I don't think it'll be all that long before this is an actual feature in SD, though. I don't know the ideal path to achieving that, but given that many AI algorithms have focused on compensating for change over time in the last couple of years, I wouldn't be surprised if it happens. Image generation is already progressing at an insanely fast rate. I wish I had more time to play around with these techniques right now.
Frame blending works (was using it) but still don't have stability in SD (even with seed & proper setup). And it's difficult to track video properly. (At this stage)
I know its a bit tricky to get it right. Theres a lot of variables that go into it and now that i updated to new ui things seems to be different. I will def be making another video on the topic
@@enigmatic_e Thank you, great work! It's fun learning & growing together with this technology.
Please if someone gave an answer, is there a way to blend images in premier pro ?Thank you very much
If you mean frame blending, yes. Use optical flow.
do you have to drop in each frame manually .... or can you set it to automatic ?
Sorry not sure what you mean exactly.
If you are animating a video with audio like singing, deleting every other frame will shorten the final animation and cause the audio to not be in sync correct?
Just make the sequence same fps and should look fine.
how can i get to this site? I use mac os and I can't find it at all.
I followed what you were doing but when I turn on Img2Img Alternative my image completely changes and doesn't resemble my original image, any idea why this might be happening?
Try clicking on sigma adjustment right the bottom of script. Sometimes that fixes it.
if you delete half of the the frames, will you still be able to sync audio you may have ?
Yes, when you import sequence into editing program, just interpret frame rate to original clip. Most programs will automatically make it 30 fps, so change it if you have to. Then just change speed to double the length and it should sync with audio
a young woman looking towards camera with green hair holding wearing gloves with a blue shirt, anime, midjoumeyart style
blotches, blown-out, maximalist, teeth, mouth open
Try Flowframes + EbSynth. I bet that could be something cool
I heard of flowframes and even downloaded it but never tried it! I will try it soon!
@@enigmatic_eYou'll love it. NMKD made GUIs like Flowframes for easy Frame Interpolation, Cupscale for Image and Video Enhancement and Stable Diffusion GUI.
@@enigmatic_e anf IF it helps you, PLEEEASE give me a credit in a Video it would mean so much to me 💜🥹
@@enigmatic_e Flowframes is the best. I've thrashed it pretty hard with 4k MB3D renders. Interpolating crisp fractals is hard, and this will easily do 4 frames out of 5 and you won't notice a thing.
It also only takes 20 minutes to do a vid clip that Topaz wants to spend all night on.
It's interesting, I've tried this technique shown here, on nerdy rodents page, and on max novak's channel and the alternative test scripts dont seem to function the same way at all. Turning on the script immediately breaks any style tokens you add in the prompt, override denoising strength doesn't seem to have the same function as you guys display, and the sigma checkbox locks it down even further. Are you guys testing this on the most recent git pull? All the videos showing this off are from a couple days ago and i dont think the updated version runs the same. I've tested this on brand new clean installs on 2 machines and it just doesn't function the way you guys are showcasing.
Hey, i can tell you that its just tweaking things. Ive had my image look good and sudden i change the a word in the prompt or change a value by 1 unit and it becomes a total messed.
@@enigmatic_e appreciate the reply and im sure that plays a part, but using the same source images settings and prompts yield completely different results. Particularly with the override check boxes that were introduced that do not exist in any of the videos on this topic including yours since they were made before the update. Something has changed recently guess anyone trying this now will have to wait on clarification as people experiment.
Ahh ok. I wasn’t aware of an update. I will def look into it.
How about using some ai interpolation instead of frame blending ?
I will try that. Maybe even do a short video showing results. Thank you!
hey hey, so the new UI version sucks (not even sure it comes from the 1.5 tho cause I tested a lot of différents settings and still the same aweful results) so I wanted to downgrade my UI and going back to this exact same version of this video! It seems that I understood how to downgrade to this version but I need the “commit hash” numbers (which seems to be the “version”) but I can’t find it on the automatic1111 git hub (there’s a lot…) so I was wondering if by any chance you still have and feel to share this hash so I can try to downgrade it! thanks!
Yea i feel you. I cant seem to get as good results with the new UI either. Unfortunately i dont have that hash. If you do find it, send it my way too!
that moment when you want stable diffusion into your iphone, it's already someone have make an app to replace sd filters in camera on the go?
That would be nice!
can i use this in 4gb vram?
Doesn't using "-a" samplers hurt the consistency due to the added randomness?
It could but I‘ve actually gotten pretty good results using it. You can check out some of my shorts and see if you like the results.
Please do one using google colab, I currently do not have enough gpu
I'm sorry to be a layman, but how can you access this site, when I see your screen and try to access only numbers appear in the browser
All good. I went over that in part one but basically you need to install some stuff in order to access Stable Diffusion. Theres a tutorial on how to do that ruclips.net/video/vg8-NSbaWZI/видео.html
I feel like frame interpolation could be cool instead of frame blending
The generated image is similar to the original image, I want to generate a woman on the moon, in this case the result is no longer stable
More people don't use this often? I'm using it all time since two weeks. I should start posting more lol
Batch Img2Img
AttributeError: 'NoneType' object has no attribute 'ema_scope'
Error Fixed :
www.reddit.com/r/StableDiffusion/comments/yjbc2d/batch_img2img_help/?context=3
Good thing I got 10gbs of VRAM