Convert single image to 3D model with AI in ComfyUI with CRM, for GPU & CPU
HTML-код
- Опубликовано: 15 окт 2024
- In this tutorial I walk you through a basic CRM workflow in ComfyUI. CRM stands for Convolutional Reconstruction Models and is now usable with ComfyUI. With CRM you can create simple 3D models from single images. It is usable with GPU and CPU. It is quite hungry on the RAM.
Be sure to update your ComfyUI to the newest version and install the needed models and custom nodes which are linked below.
The installation steps for the CRM nodes are described on their Github page.
Please comment if you have questions or tell me your suggestions for future videos.
Get the workflow on Patreon:
www.patreon.co...
CRM Models:
huggingface.co...
Custom Nodes:
github.com/flo...
github.com/cub...
More infos on CRM:
ml.cs.tsinghua...
Connect with me:
/ _neuron_ai
/ _neuron_ai
The equipment that I use (affiliate links):
GPU: ASUS ROG STRIX RTX 4090 amzn.to/42ptUTi
#comfyui #stablecascade #stablediffusion #tutorial #animatediff #automatic1111 #aiart #ai #inpaint
Hey everyone, it seams that many people have problems installing and using the crm nodes. TripoSR might be a better alternative. After a recent reinstallation of my ComfyUI I also failed to reinstall the module. Since there is no update on the Flowty crm github for month. It seems we have to find something else to convert images to 3D. I will not make a new video on this module. As soon as I have a better way to transform images to 3D I will let you know. Since then check out the TripoSR way and my video on it: ruclips.net/video/Eb3YPBk71ok/видео.html
Incredible. I mean forget the Modeler, but the pose sampler is out of this world
I have an error, when the CRMPoseSampler is executed : "Error occurred when executing CRMPoseSampler:
The shape of the 2D attn_mask is torch.Size([77, 77]), but should be (1, 1)." Have you got an idea to resolve it ?
I am getting the same error
did anyone manage to resolve this error?
Same, weird issue so far along into the video, its quite hard to find a concrete answer at the moment, still looking.
I am done with this custom nodes for now. So many problems. Last time I installed it I gave up. Did you try TripoSR? ruclips.net/video/Eb3YPBk71ok/видео.html
I got the same problem, I think it might be something with matrices, I thought it was because the machine I was running it in CPU only, but today I will try running a gpu, let's see if the error persists.
Have you run into the error about tensors? "Sizes of tensors must match except in dimension 3. Expected size 256 but got size 64 for tensor number 1 in the list." Have seen this particular issue across several different models and unsure how to resolve it. I have input a 256x256 image but unsure if that is related.
Unfortunatly I ran into lots of errors when I tried to reinstall this custom nodes. I hope for a better solution. this is not really practical.
Hey, I have RTX 3070, 32GB of RAM and 8GB of VRAM, and when I run the node « CRM modeler » I have error message « CUDA : out of memory » is this normal. When I put 6 step the error is always here
Sometimes other apps use lots of vram too. so be sure not using other apps at the same time. sometimes this happens to me with workflows that I had no problems with before. sometimes restarting comfy or even the pc can help as well.
There's a lot of info packed in your videos and have watched a few. A quick tip, I find that I have to listen on average 1m to 2m to get an idea what the video is about, sometimes I am well into a video before I know if I have to download any custom nodes, models etc. Would you be able to front load that in your videos, just so that we can follow the video?
Thanks for watching. And thanks for your question. Great idea. I will make clear what is needed in the beginning of every video from now on.
Is there a way to export?
The readme mentions obj, but it's unclear on the method.
I am not able to check right now. But it should put a mesh file into your output folder.
Looking about how ti increase texture resolution. It's only 256x256
I am not aware if this is possible at this early stage of development. But it would be great and there might be a way. But I am not sure how ATM.
Hi,good sir,do you mind send me a link of your comfyui installation guide? thank you
There is no video like this so far. I plan to make a step by step series in the future.
Getting this error when trying with CPU node
Error occurred when executing CRMModeler:
Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument grid in method wrapper_CUDA__grid_sampler_2d)
Thats odd! Do you use the CRM Modeler node and not the CRM Modeler (Cuda only) node? And in the RemBG session node choose CPU as Provider!
@@neuron_ai NOT using the cuda only node. And I was using an already made PNG cutout of a cat, was not removing backgrounds at that moment.
What keybind did you use at 2:20 to pull up that search menu?
doubleclick on the background brings up the search menu. or pull the port out schould also give a context menu where you can click search.
@@neuron_ai Can't believe I didn't try that, thank you!
Can you please help me fix this error? When my ComfyUI loads the CRM Pose Sample nodes, it gives the following error "Error occurred when executing CRMPoseSampler:
module 'open_clip' has no attribute 'create_model_and_transforms'
File "C:\Users\Admin\pinokio\api\comfyui.git\app\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "C:\Users\Admin\pinokio\api\comfyui.git\app\execution.py", line 81, print get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "C:\Users\Admin\pinokio\api\comfyui.git\app\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "C:\Users\Admin\pinokio\api\comfyui.git\app\custom_nodes\ComfyUI-Flowty-CRM\__init__.py", line 77, print sample
stage1_model = instantiate_from_config(OmegaConf.load(stage1_model_config.config).model)
File "C:\Users\Admin\pinokio\api\comfyui.git\app\custom_nodes\ComfyUI-Flowty-CRM\crmlib\imagedream\ldm\util.py", line 90, in instantiate_from_config
return get_obj_from_str(config["target"])(**config.get("params", dict()))
File "C:\Users\Admin\pinokio\api\comfyui.git\app\custom_nodes\ComfyUI-Flowty-CRM\crmlib\imagedream\ldm\interface.py", line 51, prints __init__
self.clip_model = instantiate_from_config(clip_config)
File "C:\Users\Admin\pinokio\api\comfyui.git\app\custom_nodes\ComfyUI-Flowty-CRM\crmlib\imagedream\ldm\util.py", line 90, in instantiate_from_config
return get_obj_from_str(config["target"])(**config.get("params", dict()))
File "C:\Users\Admin\pinokio\api\comfyui.git\app\custom_nodes\ComfyUI-Flowty-CRM\crmlib\imagedream\ldm\modules\encoders\modules.py", line 190, prints __init__
model, _, preprocess = open_clip.create_model_and_transforms"
It could be lots of things. did yoiu try to use my worklfow which is linked in the description? how did you install the crm custom node?
I installed and followed your instructions. I also use your process but when running to nodes "CRMPoseSampler"
"Error occurred when executing CRMPoseSampler:
module 'open_clip' has no attribute 'create_model_and_transforms'
Or is it because my pixel-diffusion.th model is corrupted?
thanks for the excellent video. please consider making a video on how to use CRM without comfyui, as the CRM nodes available are being unreliable to some people. i for one had success with following your tutorial but then the next day the workflow was giving me the nvdiffrast not found error which many people face and there is no consistent solution for.
Great idea. Will look into this. Its a pitty that CRM is not nicely implemented so far.
hello, what is the diffirence from triposr?
The main difference is that triposr uses LRM. This is a different approach then CRM. yiconghong.me/LRM/ will do a video on triposr as well. regarding the quality I am not yet sure which one is best. WIll compare them all soon.
@@neuron_ai thanks.
Does this work w AMD GPUs or NVDA only?
I dont know. Seems the github is only talking about cuda it might only be usable on nvidia gpu and cpu.
Has anyone had the this error? ModuleNotFoundError: No module named 'kiui'
did you try to install it with pip install kiui
@neuron_ai yes, first thing I tried!
Is there a text-to-3d model workflow tutorial?
Not yet. But this is a good idea. you can exchange the image loader with a default workflow.
check the recent video. I did a video on this
@@neuron_ai what do you mean check the recent video? Can you put a link right here to what you're referring to?
@@neuron_ai this? ruclips.net/video/c73zYOjOYYo/видео.html
@@Rachelcenter1 it was my most recent video. ruclips.net/video/c73zYOjOYYo/видео.htmlsi=LJVKTeJeIwScGBE5
Hey! thanks a lot for your video. I have followed it very carefully, but I have encountered an error just in the CRM modeler step. The Comfyui gives me this error:
"Error occurred when executing CRMModelerCuda:
No module named 'nvdiffrast'"
I have searched the way to install that missing module but I have not found any clear way. Could you give me some clue? thanks in advance
Thanks! Did you try to install nvdiffrast with "pip install nvdiffrast" ?
@@neuron_ai Hey! thanks for your quick response! :D
I have tried just now with "pip install nvdiffras" in a cmd window, but that gives me this other error "ERROR: Could not find a version that satisfies the requirement nvdiffrast (from versions: none)
ERROR: No matching distribution found for nvdiffrast"
...I`m using Python 3.11.8. Maybe I need some other pack or component?
Did you manage to get it working? I'm getting the same error with a 4080 on Win 10. I also installed the CUDA Toolkit, but still nothing
@@TheValkire Nop, sorry. I have left it for later because no luck in my reseach by the moment
i somehow alleviated this error but at the same time encountered "No module named nvdiffrast.torch" which is a different error i am still trying to fix