NEW Image2Video for ComfyUI. How to use CogVideoX.
HTML-код
- Опубликовано: 23 сен 2024
- CogVideoX-5b Image2Video is released for ComfyUI and it's fantastic!
Workflow and guide here / 112417217
Chat with me in our community discord: / discord
Stable Diffusion for Beginners Playlist • Stable Diffusion Begin...
My Weekly AI Art Challenges • Let's AI Paint - Weekl...
My Stable diffusion workflow to Perfect Images • Revealing my Workflow ...
ControlNet tutorial and install guide • NEW ControlNet for Sta...
Famous Scenes Remade by ControlNet AI • Famous Scenes Remade b...
Workflow and guide here www.patreon.com/posts/112417217
👋 hi
thanks, went ahead and subscribed on patreon. your worflow works well and you packaged everything up nicely.
Thank you for the video and workflow! Can't wait to try it out!
This is taking forever on my 3080ti. 12gb VRAM. It's stuck on cogvideosampler. 2% for 15 mins, I don't think 12gb VRAM is enough to run this unless you're willing to wait whole day.
In the "(Down)load CogVideo Model" Node, change the "enable_sequential_cpu_offload" to true and try again. That helped me get it to run on my 12GB RTX 4070Ti.
@@97BuckeyeGuy I'll try that.. Thank you.
Definitely needs optimization. It says 10 hours on my 12 GB 3060.
In my case, I had no error but every time output was weird - the picture consisted of tiles or mosaics... Tried to change some settings - doesn't help. So.. I discovered that there is another node - CogVideo GGUF Model and it can use I2V gguf. So I gave it a try, and it's worked.
Nice find!
Where was this? I can't seem to identify it. I got something that says (Down)load CogVideo GGUF Model as a node. Is that it? Just a node to force a download? Can't find the file independently?
@@royjones5790 yes, this node. At first run it will automatically download all files.
Cool concept and well explained! Not sure what I would use it for, but good to know. Also, I always learn some other thing when I watch your videos, so thanks for explaining things so clearly! 😀
How and where can I download this manual model?
CogVideoX-5-i2v
My internet is automatically weak for me
Thank you for this video, very helpful!
This is amazing! Thank you so much~
everything work fine but with img2video mode only appears me the "envato" font with grains, any suggestions? very strange
Worked on my Titan 24gvram. How to proceed is yet to come as it still needs to reduce to low vramm
Thanks
Is anyone else experiencing an issue with the blocky video and it turns into blue/purple color? I just waited an hour for this thing to render just to see the blocky mess it turned into. And it doesn't work with any prompt, any image, nothing. I just followed the entire video, used the given workload with the darth vader picture and it's always the same with every picture and every prompt, blocky colored mess!
What am i doing wrong?
same
@@knot-head So far i've only experienced the issue with this workload, but theres another one with cogvideo fun 5b i2v, different workload, different model but i had no issues apart from unsatisfying results which was a mess on every prompt.
I got it working, so it seems it does have to be num_frames 49 to avoid the bad result, But, I have to enable_sequential_cpu_offload and enable_vae_tiling to avoid other errors. This makes it use a fraction of my vram unfortunately and takes forever. It seems to barely overrun tho, maybe if the 4060ti 16GB was headless it would make it, idk
@@gateopssss got it working but very slow, I am going to try some other workflows and or nodes, I'm especially interested in getting the temporal tiling working for endless loops .
@@knot-head The num of frames was the issue? Damn thats so stupid..
I had a situation with the cpu offloading to be put on false, and i had it generate a blocky mess (24 frames) for 4 hours! When i enabled cpu offloading, it took only 20 minutes.
If you find another workflow please hit me up!
Is it lenght limited or can you extend it after the initial generation?
You could extract the last frame and run it again, haven’t tried it but might work.
Is there any way to expand on this to be able to stitch multiple clips together and pass multiple prompts in (1 per clip) so we can have tons of small clips that we can make a longer video out of?
Not a good one, you can run it twice and then play one in reverse so they go together seamlessly. I'm about to try prompting for panning the camera left, and then one panning the camera right to try to get a decent result from that. With the one in reverse it should create one long pan in the same direction, hopefully it would work to create one long zoom in also.
would be nice to do looping videos...
I have an assignment for you buddy! do a tutorial on " Importing & Combining Workflows" from multiple .json files!
Failed due to trying to allocate 56GB of cuda memory. I must be missing something? my 12gb card should be enough for your default settings?
I too get a failed error do to not enough vram and I'm using a rtx 3080 12gb model.
I tried a fresh install and no change. I think 12 gig cards just aint enough unfortunately.
@johnfraser21 maybe you need to change to mid vram in the config
Yeah, I'm not sure how 6GB cards are supposed to work. Maybe once the 2b-I2V comes out. In the meantime I ticked the "enable_sequential_cpu_offload" to get things rolling on my 16GB 4060ti. 25 minutes to run the job but that's fine...
You can also run it with Pinokio and everything will be installed automatically for you. (CogStudio)
Where does the Thudm cogvideox 5b 12v file save?
noticed u using Thinkdiffusionn would that help if my computers complet rubbish ?
Will it handle 3:2 and 2:3 or can it only do landscape?
unfortunately ; cant change resolution
the 5b-I2V only does 720x480 but 5b and 2b will run at other resolutions but the results were not as good. Also notice that the width and height are reversed on the sampler node for some reason, but need to match the resize image node,
hello, just wanna ask, why my 5b-i2v not appear, there is only 2b and 5b?
tyvm
Update to latest
Anyone know why I cannot see the manager in my UI?
The AI Martin Shkreli.
That's kind of rude
@@sebastiankamph Yeah I don't know what this troll is talking about. Keep on doing what you do my guy, you're great!
@@sebastiankamph Haha, I love ya dude, been following for ages, you'll be right with my jokes
I get stoked about these tittles to watch until realize ComfyUI is those steps and layout that throws my brain way off
Getting about 5-6 mins on my 4090. However, I cannot increase the num_frames passed 49 or I get an output that show a bunch of tiny tiles, bright chroma, and one frame of it is almost the original image.
I tried increasing tile sample height/width size... decreasing frame rate... decreasing steps... everything exhibits the same colorful tiny tiles...
It seems the only thing causing the error is setting the num_frames above 52
you can't change the number of frames because it runs at 49. while waiting for it to get faster you can lower the step to 10
@@octopuss3893 Well I was able to raise the num_frames to 52... after that it gets the tiling. Was hoping to adjust the frame rate to like 15, but increase the num_frames to 60 or 75, then use Davinci Resolve to boost the frame rate with interpolated frames to 24 or 30. At 740x480 it's already NTSC resolution from 25 years ago... but to only have 2 second clip... well that really limits its uses. I mean choosy beggars and all aside...
git clone and install missing nodes fails to import for me.
must clone all folders ; tolenizer ; transformer ; vae and ...
outputs black ...
Thanks!
Very kind, thank you for your support, you absolute mvp you ☺️💫
@@sebastiankamph your video is perfectly done. Thank you! I am encountering a couple of errors. on my Mac (m3 max pro - 120 gb unified ram) I get the following error "CogVideoSampler
User specified an unsupported autocast device_type 'mps'". Thoughts on that? I am double checking the error on my Windows machine (4090 with 16gb vram, gaming laptop)
Any guidance much appreciated.
@@vincentnestler1805 I'm sorry but I don't own a mac so can't test. Do let me know if it ran on your pc. Check the requirements.txt in the custom node. For windows you can: pip install -r requirements.txt
Maybe something similar for Mac.
@@sebastiankamph update - On my desktop gaming rig with 4090 - ran great. On my lapton... It hangs at about 8%. Might not meet the min spec. On the mac... I will keep digging. Thanks again for a great video. Very helpful.
it's really slow. I'm only 3060 6G😂.I don't expect it to work out🤣
I want to see what type of video he's making out of this thumb nail...
first
You got it! 🌟
Not a single person on youtube speaks about how to install this, it's all about how to use it. At this point I feel like they just bait you into watching their videos to get their watch time. I am fed up of going through tons and tons of videos just to find the same content. Even here, apparently if you don't sign up for their patreon you can't get the content !!! This is absolute bullfkingshit
The workflow is not behind the paywall, just scroll down.
@@knot-head thanks
These are no "videos", only fotos with a moving background! Who needs this?
Why don't you prompt it for what you want then? Like a person walking for example.
?? you can get a lot of movement, just depends on your prompt
@@joethompson9124 Have a link to some examples? (The tutorial should provide this!)
@@sebastiankamph What I saw up to now, did not convince me! Do you have a link to some examples?
you need to prompt
that install is way too complicated for me. why does everybody seem to prefer incredibly complex installations?
I personally like that there's that barrier to entry, it means those of us willing to put in the work to use a lot of this cutting edge stuff have an advantage over those who aren't. Anyway, I think people at the forefront of open source AI are interested in advancing the technology, not in creating frictionless user experiences for laypeople so as to have mass appeal. That'll come in 2/3 years when this is an iPhone app
its actually not that hard. this stuff is bleeding edge technology it is not beginner friendly. if you want in on this, you need to put in some work and spend time learning. you can do it.
Comfy Ui is not that complicated if you put in the hours to get used to it. The amazing part of comfy is it allows different technologies to be incorporated into one single platform without much modifications.
If you want easier installation, there are plenty of video generators that only require a credit card number to run.
Got this issue "Given groups=1, weight of size [1920, 16, 2, 2], expected input[26, 32, 60, 90] to have 16 channels, but got 32 channels instead" any fix?
Same here, did you find a fix ?
@@stevietee3878 I have the same problem!
must make new folder in model and clone all files and folders ; like tolenizer ; transformer ; vae and ...
@@SemorezX hmm interesting
I had this error, because I had chosen the wrong model. I forgot to pick the I2V one.
If I want to download the model manually, then in which folder should I put it?
google_t5-v1_1-xxl_encoderonly
models/clip
Thanks