Stable Diffusion ComfyUI Workflow - Using Multimodal Pipeline To Create AI Video

Stable Diffusion ComfyUI Workflow - Using Multimodal Pipeline To Create AI Video
We'll be exploring how to create stunning AI videos with the help of a multimodal pipeline.
As we all know, AI video models have been evolving rapidly, and now we have companies like Kling AI, Luma AI, and the latest Gen-3 AI video models from RunwayML. Although Runway Gen-3 is currently available only for text-to-videos, image-to-video capabilities are just around the corner.
If you need to setup Local Ollama to host your local LLM with ComfyUI.
Here's previous tutorials :
1 - • Stable Diffusion Comfy...
2 - • ComfyUI Node Use Llama...
For Freebies : / stable-diffusion-10731...
Goodies For Patreon Supporters: / create-story-llm-10731...
In this workflow, I'll show you how to transform natural language content into Stable Diffusion Text Prompts, which can be used to generate images for each scene. These images serve as the initial keyframes for the AI video generator. We'll be using powerful models like LLaMA 3 fine-tuned SD Prompt large language models to transform the storyline into text prompts for image generation in Stable Diffusion.
Here's the Things Will Be In To-Do List:
Connect Database for query story background setting , SD prompt
Character setting , SD prompt
Story Contents in Database Table structure, and it will be able to process each scenes automatically.
Connect ComfyUI as Client App to AI Video provide API.(If they open up for connect)
Don't forget to hit that subscribe button and turn on the notification bell, so you won't miss any of my upcoming tutorials on AI video generation and more.
If You Like tutorial like this, You Can Support Our Work In Patreon:
/ aifuturetech
Discord : / discord

[@TheFutureThinker]

For Freebies : www.patreon.com/posts/stable-diffusion-107319629
Goodies For Patreon Supporters: www.patreon.com/posts/create-story-llm-107317948
If you need to setup Local Ollama to host your local LLM with ComfyUI.
Here's previous tutorials :
1 - ruclips.net/video/EQZWyn9eCFE/видео.html
2 - ruclips.net/video/yR2Y9G71w6E/видео.html

[@DOCTOR-FLEX]

this workflow was absolutely amazing. Thank you for this information.

[@TheFutureThinker]

Glad it was helpful!

[@sebastiopoulos92]

brooooo i have been looking to do that for weeks but im a cinematographer not a developper thanks a lot :DDD

[@Djonsing]

*Hello, will RTX 2060 SUPER (8GB) handle it?*

[@crazyleafdesignweb]

Pretty cool idea. Looking forward to it. And AutoStudio you talked about, do similar image generation.
But yours are build on workflow and run all of this.
What backend SQL and code you will be use for the next update?

[@TheFutureThinker]

yup , something like that for the next update. As previous video we talked about Comfy API, so things are doable.

[@leandrojareno]

Hey @Benji any chance to have the complete .json workflow?
Also, have you ever tried Open-Sora? I've tried to install but my GTX 1070 didn't run.
Thanks for the videos!

[@TheFutureThinker]

Not sure if it will be work with the graphic card you mentioned. But my opinion, I will upgrade the hardware before test this. But thats only my opinion.

[@DOCTOR-FLEX]

Is this workflow available somewhere?

[@TheFutureThinker]

Link In the description

[@mordokai597]

i made a custom gpt for gpt4 that converts prompts/sentences to wd 1.4 tag prompts. it's called '(BooruKai_Prompter: 1.4)'
"### Example Prompt Conversion:
- **Original Sentence**: "A sci-fi cyberpunk heroine in an industrial area, wielding futuristic weapons."
- **Token-Sparse Prompt**: "1girl, cyberpunk, weapon, industrial_area, futuristic, short_hair, blue_eyes, armor, neon, machinery""

[@TheFutureThinker]

Oh great , that works too

[@anuresearch]

[@선비Seonbi]

[@TheFutureThinker]

😉👍

[@ProcXelA]

understand nothing where's tutorial?

[@kalakala4803]

it can be integrated like LBX Studio, but for better AI Video Model, not SVD. 🤭

[@TheFutureThinker]

Hehe.. of course not use SVD. You know 😎