LOKI - Mimic Motion - Pose Animation with SVD XT 1.1
HTML-код
- Опубликовано: 27 дек 2024
- UPDATE: • RotoMaker V2 | Animate...
we further refined this into a new workflow pack, check it out
MimicMotion allows a target image to be animated following a driver video, similar to LivePortrait, only this time it's for character movements, such as dancing. The pipeline requires Stable Video Diffusion (SVD XT 1.1) in diffusers in combination with the MimicMotion FP16 checkpoint.
While settings can improve the outputs, driving SVD with openpose from a reference, and quickly is a huge step forward and another impressive tool in the belt of artists looking to experiment with these new abilities.
Workflow Pack: civitai.com/po...
Loki-FaceSwap: • LOKI FASTEST FACE SWAP...
Loki-LivePortrait: • Loki - Live Portrait -...
Full install instructions are written up on the workflow page and also given in this video.
Join this channel to get access to perks:
/ @fivebelowfiveuk
discord: / discord
www.fivebelowfive.uk
Workflow Packs:
Hyper SUPIR civitai.com/mo...
Merge Models civitai.com/mo...
cosXL Convertor civitai.com/mo...
Looped Motion civitai.com/mo...
Trio Triple Latents civitai.com/mo...
Ananke Hi-Red civitai.com/mo...
SDXL Lora's
civitai.com/mo...
civitai.com/mo...
civitai.com/mo...
civitai.com/mo...
civitai.com/mo...
civitai.com/mo...
civitai.com/mo...
civitai.com/mo...
civitai.com/mo...
civitai.com/mo...
Introducing series (music/video)
Noisee.ai • Introducing Noisee.ai ...
Udio.com • Introducing Udio.com [...
suno.com • Introducing Suno V3 Music
haiper.ai • Introducing Video Gene...
Checkpoint Merging
• Create The Best Model ...
cosXL / cosXL-edit conversion
• Convert any SDXL model...
• Unlock CosXL with any ...
3D Generation
• DJZ 3D Collection
New Diffusion Models (April '24)
Stable Cascade:
• Stable Cascade Comfy C...
• Stable Cascade in Comf...
SDXS-512:
• SDXS - New Image Gener...
cosXL & cosXL-edit:
• CosXL & CosXL-Edit - N...
Stable Cascade series:
• Stable Cascade Workflo...
Image Model Training
datasets • Datasets in detail - M...
colab • Updated Lora Training ...
local • Updated Lora Training ...
civitai • Stable Cascade LORA tr...
civitai • SDXL Lora Training wit...
Music with Audacity
• Make Music with Audaci...
• Make Music with Audaci...
DJZ custom nodes (aspectsize node)
• AspectSize (djz-nodes)...
stable diffusion cascade
stable diffusion lora training
comfyui nodes explained
comfyui video generation
comfyui tutorial 2024
best comfyui workflows
comfyui image to image
comfyui checkpoints
civitai stable diffusion tutorial
I hope to come back here once I figure out wth is going on. I can't figure out how to even do the first 4 minutes. lol
It might be that a new video is arriving to cover mimic motion again, but i have to work in reallly big circles due to the amount of new stuff I want to show.
@FiveBelowFiveUK thanks. I actually figured everything out. I had to use Google Gemini to figure out all the Git stuff. But it all worked. Video results were iffy but I'll keep learning. Thanks for the help.
Kijai is killing it! Thanks for the video!
Thank you for your effort and credits.
Thanks !!
Hi! Is this tool only capable of working in the resolution of 576x1024? Is there a way to have any other resolution and aspect ratio? Maybe some workaround for that?..
I cannot think of a reason why it would not, try it and see
Dimensions limits are usualy determined by your hardware specs
Hello, you mention you use a Mac in this tutorial. How did you manage to make mimicmotion work on a mac. when I install and run, I get this error "he operator 'aten::_upsample_bicubic2d_aa.out' is not currently implemented for the MPS device."
um, if i did say that i'm not sure quite why - i don't own any mac's, perhaps somebody in my discord told me that they used this with one and i relayed that info, but honestly i cannot confirm that myself - I cannot test it to find out - sorry.
After 2 months still i can't install Mimic motion. always get a " Import Failed" error. with 2-3 ways i'v tried, from inside comfy manager or install with git clone command. i've a RTX 4070Ti GPU.
Could you please share some tips or solutions ?
it's usually the requirements that cause these types of problems, ignoring major incompatibility issues, you might try my script which forced the installation of the requirements.txt. It must be run inside the custom node folder, where there is a requirements.txt
it's worth trying to eliminate the possibility, that you may have installed them to system python, not the portable python comfyui regularly used,
github.com/MushroomFleet/x-flux-comfyui/blob/main/setup-py.bat
this file can be used in any custom node folder, place in same folder as the requirements and let it run.
great vid, thanks
Glad you enjoyed it
Error occurred when executing MimicMotionGetPoses:
An error happened while trying to locate the files on the Hub and we cannot find the appropriate snapshot folder for the specified revision on the local disk. Please check your internet connection and try again.
Please tell me where is the error in this question?
check the description for this workflow pack, there may be some backend models you need.
also the author has updated his repo, so i may need to come back and update for new workflows, this can happen after some weeks :) thanks for the comment - it helps to alert me to this. Here is the authors project link:
github.com/kijai/ComfyUI-MimicMotionWrapper?tab=readme-ov-file
I'm getting close to the results. There is a lot of warping even with high resolution image. Can you share a screen shot of your parameters in the MimicMotion node? I'll follow up on X as well
my settings are baked into the workflow i have shared, I did a follow up to this in later videos, RotoMaker Studio builds on this. V2 is all-in-one. There may be other settings there, i may have used the same, worth a look.
@@FiveBelowFiveUK I will definitely check it out. Thanks!
Getting the following error:
Error occurred when executing MimicMotionSampler: "compute_index_ranges_weights" not implemented for 'Half'
Anyone have a solution?
unless something was broken since i last used this, which is possible, check your nodes are up to date, if it is still broken, the workflow may need to be rewritten. This can happen when custom nodes used in the workflows are updates or reworked by their respective authors.
do you know how to use same picture in comfyui , i lost the seed of the person i used
you should be able to drag the image you made into comfy, it should give you the seed it was about to run, to make that image. check your output folders, any image created in a generation should load the workflow used to make the image.
getting this error every time OSError: Error no file named config.json found in directory W:\ComfyUI_windows_portable\ComfyUI\models\diffusers\stable-video-diffusion-img2vid-xt-1-1.
i am getting this because all my config.json are renamed while downloaded like that feature_extracto_preprocessor_config.json
make sure your all files are saved with correct name
─feature_extractor
│ preprocessor_config.json
│
├───image_encoder
│ config.json
│ model.fp16.safetensors
│
├───scheduler
│ scheduler_config.json
│
├───unet
│ config.json
│ diffusion_pytorch_model.fp16.safetensors
│
└───vae
config.json
diffusion_pytorch_model.fp16.safetensors
yes another person had this too, it's hugging face being silly and appending prefix_ to the .json. very annoying. I can see your other comment, Glad you worked it out !
Nice! 좋아 고마워 🎉
nice
Its stuck at the node MimicMotion GetPose. It tries to download a model but it can't: ComfyUI\\custom_nodes\\ComfyUI-MimicMotionWrapper\\models\\DWPose\\.cache\\huggingface\\download\\dw-ll_ucoco_384_bs5.torchscript.pt.d86a0b2b59fddc0901a7076e9f59c9f8602602133ed72511c693fd11eea23d91.incomplete'. The problem continues even if i download it myself and put it in the folder.
the filename definitely does not contain all the numbers etc with .incomplete extension, you were right to try re-downloading the files. "dw-ll_ucoco_384_bs5.torchscript.pt" is the full filename.
It's using the huggingface cache, so you might need to delete that file and have it refetch.
I worked with a mobile internet for years so i feel your pain, i've seen this before.
Often the huggingface cache is not in the comfyui path but somewhere else in windows (if you are even on windows).
direct download might not work because of how the HF cache works.
Good Luck !
@@FiveBelowFiveUK I just used another node and It worked. AdvancedLivePortrait. I doesn't like It because It needs to have two animations nodes after the image. I'll try to fix this problem another time. Don't you know if there's some nodes for facefusion 3? They have liveportrait behind It now. I need something to generate audio through Text and lypsync it with some image, do you have any tutorial for comfu ui?
Amazing but too advanced for this lamin.