"The size of tensor a (64) must match the size of tensor b (96) at non-singleton dimension 4" So at first I was getting the above error but after some tinkering I found that if I set resolution_x to 768 the video will then render. But at 512 I get the above error. Is that something you can easily fix with an update to your node? Or maybe there is something I can do? The problem now is that when I was using your older node, I could get about 120 frames at 512. This was too short for my project. I become very excited to see you make a node that could render longer videos. But now at 768, that takes up a lot more VRAM so I can only get 100 frames (23.4GB of VRAM). Can't go much higher without OOM errors. So, at least for me, your new LONG node is producing shorter videos than your older node ;) I really hope there is an easy fix :) Anything you can think of to reduce the amount of VRAM needed would be VERY helpful.
I haven't come across this error. Please can you show the full error let me know the part of the code that threw it, and also the sizes of the picture and video so that I can try to reproduce the error.
@@Isi-uT Sure. I've used a variety of resolution videos as the input. All portrait orientation. I started with 720 X 1080 and I resized it down to 660x512 to see if that would help, but it didn't. I also tried different sized images, so that's not it. Here's what the CLI says... 2024-09-14 19:33:07,823 - dw pose extraction - INFO - All frames have been processed. 32 Ready for inference. Running UniAnimate inference on gpu (0) Loaded ViT-H-14 model config. Loading pretrained ViT-H-14 weights (X:\AI\ComfyUI3\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-UniAnimate-W\checkpoints/open_clip_pytorch_model.bin). X:\AI\ComfyUI3\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch n\functional.py:5504: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen ative\transformers\cuda\sdp_utils.cpp:455.) attn_output = scaled_dot_product_attention(q, k, v, attn_mask, dropout_p, is_causal) Restored from X:\AI\ComfyUI3\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-UniAnimate-W\checkpoints/v2-1_512-ema-pruned.ckpt Load model from (X:\AI\ComfyUI3\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-UniAnimate-W\checkpoints/unianimate_16f_32f_non_ema_223000.pth) with status () Avoiding DistributedDataParallel to reduce memory usage Seed: 30 end_frame is (32) Number of frames to denoise: 32 0%| | 0/25 [00:00
The UNET model was getting the default resolution from the config file which could sometimes be different from the resolution used by the noise. I have updated the code to prevent the error. All you need to do is to add: cfg.resolution = resolution at line 240 in tools/inferences/inference_unianimate_long_entrance.py at line 234 in tools/inferences/inference_unianimate_entrance.py Please let me know how it goes. As for the VRAM requirement, I can't think of any other way to reduce it. The inference initially required at least 22GB of VRAM to run, but was reduced to around 10GB by transferring the clip_embedder and autoencoder computations to CPU. The only advantage I know for using the long version is to maintain consistency of appearance of the output.
@@Isi-uT YES! That worked perfectly and the results are awesome lol! Am now able to create MUCH longer videos. Just made one with 370 frames, and maybe I can do longer! Thanks so much for your nodes, your help and your hard work :)
The new workflows are : image2VidLong.json & reposeImgNew.json You can find both workflows in the newWorkflows folder in this github repository: github.com/Isi-dev/ComfyUI-UniAnimate-W
Brother, I watched your channel and I have two questions. The first question is, can I upload any video of mine that has a certain movement and will put it on a picture in a very professional manner? The second question is, give me a way to install it, please, brother. By the way, I am a new subscriber. Please, brother, take care of me 💯
Yes, you can upload any video and the movement will be transferred to the picture, but I cannot guarantee that it will be very professional. Sometimes, extra editing might be needed. Please note that this implementation is for the Windows OS. You can watch a video on the installation here: ruclips.net/video/NFnhELV4bG0/видео.html Or you can install the custom nodes with the ComfyUI Manager by searching for: ComfyUI-UniAnimate-W You can download the required models (about 14GB) from huggingface.co/camenduru/unianimate/tree/main and place them in '\custom_nodes\ComfyUI-UniAnimate-W-main\checkpoints' folder. In case you haven't done so, you can download comfyUI from this link: www.comfy.org/
when i installed the custom node,IT MESSED UP MY PYTORCH VERSION.... Can you plz help me how to get it working on my existing comfy UI ,This is my system now, Total VRAM 4096 MB, total RAM 23903 MB pytorch version: 2.3.1+cu121 xformers version: 0.0.27
@@Isi-uT Actually when i installed unianimate Custom node it reindstalled Pytorch to other version, and my comfy ui was not working,then i deleted the Unianimate Custom node and i reinsatlled the pytorch version: 2.3.1+cu121 ,then comfy started working BUT now i am Not sure to install the custom node again as i fear it will mess up my pytorch again,can you help plz..Thanks for replying.
The pytorch version in the requirements.txt file in the Unianimate custom node is 2.3.1 which is the same as what you currently have, so I am quite surprised that it would install another pytorch version. The only other thing is to ensure that the xformers version is 0.0.27.
An alternative is to have another comfyUI installed for unianimate to avoid dependency conflicts with other custom nodes. That's what I usually do for new custom nodes with requirements that conflict with the ones I already have.
I have been having issues getting any of the UniAnimate nodes to work at all. Any help would be appreciated. One of my errors reads as follows: ReposeImage Failed to init class , with PytorchStreamReader failed reading zip archive: failed finding central directory
This looks either like a corrupted file issue or the models were not properly extracted from the zip file download into the ComfyUI-UniAnimate-W/checkpoints folder. You can check your ComfyUI-UniAnimate-W/checkpoints folder and see if you can find the 'open_clip_pytorch_model.bin' model. It is around 3.85GB in size. If the size is less, then you will have to download it again. Also ensure other models are present and fully downloaded: unianimate_16f_32f_non_ema_223000.pth ~ 5.5GB v2-1_512-ema-pruned.ckpt ~ 5.1GB yolox_l.onnx ~ 211.6MB dw-ll_ucoco_384.onnx ~ 131.2MB If you downloaded all the models in a zip file, then ensure the models are first extracted before placing them in the ComfyUI-UniAnimate-W/checkpoints folder. Please let me know the outcome.
@@Isi-uT You were correct. It was the 'open_clip_pytorch_model.bin' file. I downloaded it manually and it got past that step. Working on the next error now.
You should be able to render a video up to 20 sec if you have a high VRAM. The highest I have done is 4 sec. Someone mentioned rendering up to 370 frames which is a little above 12 sec for a 30fps video. The video quality depends on the inputs and the seed. The team behind the original project suggested using a seed of 7 or 11 in their project page. You have to keep experimenting with different seeds, and upscaling vids and images to find out what works best.
heyy, got this error while reposing an image "Failed to init class , with /usr/local/lib/python3.10/dist-packages/torchaudio/lib/libtorchaudio.so: undefined symbol: _ZNK3c105Error4whatEv". PLease help!!
It seems the torch & torchaudio in your comfyui environment are not compatible. You can check the versions of both libraries and confirm if they are compatible by visiting this pytorch site: pytorch.org/get-started/previous-versions/ Also note that this project was tested successfully with pytorch versions 2.0.1 & 2.3.1 with compatible torchvision and torchaudio libraries. I don't know if other versions work well.
![Seungmin "그렇게, 천천히, 우리(As we are)" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/kAzmhLHePqU/mqdefault.jpg)
"The size of tensor a (64) must match the size of tensor b (96) at non-singleton dimension 4"
So at first I was getting the above error but after some tinkering I found that if I set resolution_x to 768 the video will then render. But at 512 I get the above error. Is that something you can easily fix with an update to your node? Or maybe there is something I can do?
The problem now is that when I was using your older node, I could get about 120 frames at 512. This was too short for my project. I become very excited to see you make a node that could render longer videos. But now at 768, that takes up a lot more VRAM so I can only get 100 frames (23.4GB of VRAM). Can't go much higher without OOM errors.
So, at least for me, your new LONG node is producing shorter videos than your older node ;)
I really hope there is an easy fix :) Anything you can think of to reduce the amount of VRAM needed would be VERY helpful.
I haven't come across this error. Please can you show the full error let me know the part of the code that threw it, and also the sizes of the picture and video so that I can try to reproduce the error.
@@Isi-uT Sure. I've used a variety of resolution videos as the input. All portrait orientation. I started with 720 X 1080 and I resized it down to 660x512 to see if that would help, but it didn't. I also tried different sized images, so that's not it.
Here's what the CLI says...
2024-09-14 19:33:07,823 - dw pose extraction - INFO - All frames have been processed.
32
Ready for inference.
Running UniAnimate inference on gpu (0)
Loaded ViT-H-14 model config.
Loading pretrained ViT-H-14 weights (X:\AI\ComfyUI3\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-UniAnimate-W\checkpoints/open_clip_pytorch_model.bin).
X:\AI\ComfyUI3\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch
n\functional.py:5504: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen
ative\transformers\cuda\sdp_utils.cpp:455.)
attn_output = scaled_dot_product_attention(q, k, v, attn_mask, dropout_p, is_causal)
Restored from X:\AI\ComfyUI3\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-UniAnimate-W\checkpoints/v2-1_512-ema-pruned.ckpt
Load model from (X:\AI\ComfyUI3\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-UniAnimate-W\checkpoints/unianimate_16f_32f_non_ema_223000.pth) with status ()
Avoiding DistributedDataParallel to reduce memory usage
Seed: 30
end_frame is (32)
Number of frames to denoise: 32
0%| | 0/25 [00:00
I see, thanks. I will look into it.
The UNET model was getting the default resolution from the config file which could sometimes be different from the resolution used by the noise. I have updated the code to prevent the error. All you need to do is to add:
cfg.resolution = resolution
at line 240 in tools/inferences/inference_unianimate_long_entrance.py
at line 234 in tools/inferences/inference_unianimate_entrance.py
Please let me know how it goes.
As for the VRAM requirement, I can't think of any other way to reduce it. The inference initially required at least 22GB of VRAM to run, but was reduced to around 10GB by transferring the clip_embedder and autoencoder computations to CPU. The only advantage I know for using the long version is to maintain consistency of appearance of the output.
@@Isi-uT YES! That worked perfectly and the results are awesome lol! Am now able to create MUCH longer videos. Just made one with 370 frames, and maybe I can do longer! Thanks so much for your nodes, your help and your hard work :)
Give me the name of the form or workflow please.
The new workflows are : image2VidLong.json & reposeImgNew.json
You can find both workflows in the newWorkflows folder in this github repository: github.com/Isi-dev/ComfyUI-UniAnimate-W
Brother, I watched your channel and I have two questions. The first question is, can I upload any video of mine that has a certain movement and will put it on a picture in a very professional manner? The second question is, give me a way to install it, please, brother. By the way, I am a new subscriber. Please, brother, take care of me 💯
Yes, you can upload any video and the movement will be transferred to the picture, but I cannot guarantee that it will be very professional. Sometimes, extra editing might be needed.
Please note that this implementation is for the Windows OS.
You can watch a video on the installation here: ruclips.net/video/NFnhELV4bG0/видео.html
Or you can install the custom nodes with the ComfyUI Manager by searching for: ComfyUI-UniAnimate-W
You can download the required models (about 14GB) from huggingface.co/camenduru/unianimate/tree/main
and place them in '\custom_nodes\ComfyUI-UniAnimate-W-main\checkpoints' folder.
In case you haven't done so, you can download comfyUI from this link: www.comfy.org/
when i installed the custom node,IT MESSED UP MY PYTORCH VERSION....
Can you plz help me how to get it working on my existing comfy UI ,This is my system now, Total VRAM 4096 MB, total RAM 23903 MB
pytorch version: 2.3.1+cu121
xformers version: 0.0.27
The xformers requirement makes the installation quite difficult and it took me sometime to get it working. Can you check for any error in your CLI?
@@Isi-uT Actually when i installed unianimate Custom node it reindstalled Pytorch to other version, and my comfy ui was not working,then i deleted the Unianimate Custom node and i reinsatlled the pytorch version: 2.3.1+cu121 ,then comfy started working BUT now i am Not sure to install the custom node again as i fear it will mess up my pytorch again,can you help plz..Thanks for replying.
@@ParthKakarwar-b7j it`s my problem too
The pytorch version in the requirements.txt file in the Unianimate custom node is 2.3.1 which is the same as what you currently have, so I am quite surprised that it would install another pytorch version. The only other thing is to ensure that the xformers version is 0.0.27.
An alternative is to have another comfyUI installed for unianimate to avoid dependency conflicts with other custom nodes. That's what I usually do for new custom nodes with requirements that conflict with the ones I already have.
I have been having issues getting any of the UniAnimate nodes to work at all. Any help would be appreciated. One of my errors reads as follows: ReposeImage
Failed to init class , with PytorchStreamReader failed reading zip archive: failed finding central directory
This looks either like a corrupted file issue or the models were not properly extracted from the zip file download into the ComfyUI-UniAnimate-W/checkpoints folder.
You can check your ComfyUI-UniAnimate-W/checkpoints folder and see if you can find the 'open_clip_pytorch_model.bin' model. It is around 3.85GB in size. If the size is less, then you will have to download it again. Also ensure other models are present and fully downloaded:
unianimate_16f_32f_non_ema_223000.pth ~ 5.5GB
v2-1_512-ema-pruned.ckpt ~ 5.1GB
yolox_l.onnx ~ 211.6MB
dw-ll_ucoco_384.onnx ~ 131.2MB
If you downloaded all the models in a zip file, then ensure the models are first extracted before placing them in the ComfyUI-UniAnimate-W/checkpoints folder. Please let me know the outcome.
@@Isi-uT Yes. I suspected that as well. I have downloaded them from the link you provided several times now. Could the source files be corrupted?
@@Isi-uT You were correct. It was the 'open_clip_pytorch_model.bin' file. I downloaded it manually and it got past that step. Working on the next error now.
Thanks for the feedback.
Can you provide me setting for rendering 20 sec video in HD quality
You should be able to render a video up to 20 sec if you have a high VRAM. The highest I have done is 4 sec. Someone mentioned rendering up to 370 frames which is a little above 12 sec for a 30fps video. The video quality depends on the inputs and the seed. The team behind the original project suggested using a seed of 7 or 11 in their project page. You have to keep experimenting with different seeds, and upscaling vids and images to find out what works best.
Thanks It was helpful. I'll try and response you when I get desire result
@Isi-uT , i have this error(((( UniAnimateImageLong
Unknown error
Please can you post the full error message let's see if we can resolve it.
heyy, got this error while reposing an image "Failed to init class , with /usr/local/lib/python3.10/dist-packages/torchaudio/lib/libtorchaudio.so: undefined symbol: _ZNK3c105Error4whatEv". PLease help!!
It seems the torch & torchaudio in your comfyui environment are not compatible. You can check the versions of both libraries and confirm if they are compatible by visiting this pytorch site: pytorch.org/get-started/previous-versions/
Also note that this project was tested successfully with pytorch versions 2.0.1 & 2.3.1 with compatible torchvision and torchaudio libraries. I don't know if other versions work well.