Live Portrait in Comfy UI similar to Hedra AI but its FREE - Talking Avatar in Local PC

Install on your local PC/laptop: LINKS
Github (for Comfy UI) - github.com/kijai/ComfyUI-Live...
My workflow with assets -openart.ai/workflows/seal_har...
Made with Stable Diffusion and Comfy UI
#hedraai #ai #comfyui
More details about Live Portrait:
LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control is an innovative approach for creating lifelike video animations from a single source image. Here are the key details:
Objective: LivePortrait synthesizes realistic videos by using a single source image as an appearance reference and deriving motion (facial expressions and head pose) from a driving video, audio, text, or other sources.
Framework: Unlike mainstream diffusion-based methods, LivePortrait leverages an implicit-keypoint-based framework. This approach balances computational efficiency and controllability.
Training Data: The model is trained on an extensive dataset of approximately 69 million high-quality frames. A mixed image-video training strategy enhances generalization.
Enhancements:
Compact Implicit Keypoints: These keypoints effectively represent blendshapes, contributing to better generation quality.
Stitching and Retargeting Modules: These modules, implemented with a small MLP, enhance controllability without significant computational overhead.
Performance: LivePortrait achieves remarkable generation speed (12.8ms on an RTX 4090 GPU with PyTorch) even when compared to diffusion-based methods.

[@sir_wesley86]

this would be so cool for our NPC portraits in game

[@OctoberSurprise11]

Looking for Leonardo to add this asap

[@Avalon19511]

Unfortunately unlike Hedra it does not really animate the entire head, characters with short hair look better but anyone with long hair it looks odd.

[@Omsip123]

Yeah, as soon as the head rotates, it gets weird. But who knows, soon this might be solved

[@heysoygiovanni6616]

good

[@AlejandroGuerrero]

Is there any tool (like this to run locally) to upload an mp3 voiceover and generate the mouth and eye movements to later use in this process? Thanks!

[@AI_Made_Simple.]

Capcut

[@leandrogoethals6599]

How much vram do u require?

[@user-ex6pk6fh4p]

I have (IMPORT FAILED) ComfyUI-LivePortraitKJ everytime. Tried reinstall but it's still failed.

[@AI_Made_Simple.]

What is the error in console?normally import failed is related to dependencies

[@user-ex6pk6fh4p]

@@AI_Made_Simple. I found out that it's because I didn't install insightfacem but still i have an issue when installing it
"C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -Iinsightface/thirdparty/face3d/mesh/cython -IS:\ComfyUI_windows_portable\ComfyUI_windows_portable\python_embeded\Lib\site-packages
umpy\core\include -IS:\ComfyUI_windows_portable\ComfyUI_windows_portable\python_embeded\include -IS:\ComfyUI_windows_portable\ComfyUI_windows_portable\python_embeded\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt" /EHsc /Tpinsightface/thirdparty/face3d/mesh/cython/mesh_core.cpp /Fobuild\temp.win-amd64-cpython-311\Release\insightface/thirdparty/face3d/mesh/cython/mesh_core.obj
mesh_core.cpp
insightface/thirdparty/face3d/mesh/cython/mesh_core.cpp(147): warning C4244: =: преобразование "int" в "float", возможна потеря данных
insightface/thirdparty/face3d/mesh/cython/mesh_core.cpp(147): warning C4244: =: преобразование "int" в "float", возможна потеря данных
insightface/thirdparty/face3d/mesh/cython/mesh_core.cpp(210): warning C4244: =: преобразование "int" в "float", возможна потеря данных
insightface/thirdparty/face3d/mesh/cython/mesh_core.cpp(210): warning C4244: =: преобразование "int" в "float", возможна потеря данных
insightface/thirdparty/face3d/mesh/cython/mesh_core.cpp(294): warning C4244: =: преобразование "int" в "float", возможна потеря данных
insightface/thirdparty/face3d/mesh/cython/mesh_core.cpp(294): warning C4244: =: преобразование "int" в "float", возможна потеря данных
"C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -Iinsightface/thirdparty/face3d/mesh/cython -IS:\ComfyUI_windows_portable\ComfyUI_windows_portable\python_embeded\Lib\site-packages
umpy\core\include -IS:\ComfyUI_windows_portable\ComfyUI_windows_portable\python_embeded\include -IS:\ComfyUI_windows_portable\ComfyUI_windows_portable\python_embeded\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt" /EHsc /Tpinsightface/thirdparty/face3d/mesh/cython/mesh_core_cython.cpp /Fobuild\temp.win-amd64-cpython-311\Release\insightface/thirdparty/face3d/mesh/cython/mesh_core_cython.obj
mesh_core_cython.cpp
insightface/thirdparty/face3d/mesh/cython/mesh_core_cython.cpp(36): fatal error C1083: Не удается открыть файл включение: Python.h: No such file or directory,
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.40.33807\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for insightface
Failed to build insightface
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (insightface)

[@colinfransch6054]

Thank you. Everything works great except only on the cpu. I have tried to reinstall onx-gpu within the ComfyUI env but still resorts to cpu.

[@AI_Made_Simple.]

Did you try installing prebuilt version of insightface... github.com/Gourieff/comfyui-reactor-node#i-for-windows-users-if-you-still-cannot-build-insightface-for-some-reasons-or-just-dont-want-to-install-visual-studio-or-vs-c-build-tools---do-the-following

[@user-ex6pk6fh4p]

@@AI_Made_Simple. thanks, i'll try it