SegMoE - The Stable Diffusion Mixture of Experts for Image Generation!

Mixture of experts. Seems hot for AI text generation... but what if you had a mixture of experts for IMAGE generation? Oh. Segmind just did that. Welcome to SegMoE - the mixture of experts for SDXL, SDXL Turbo and Stable Diffusion 1.5.
Want to support the channel?
/ nerdyrodent
== Links ==
huggingface.co/segmind
github.com/segmind/segmoe
github.com/segmind/segmoe/blo...
== More Stable Diffusion Stuff! ==
* Faster Stable Diffusions with the LCM LoRA - • LCM LoRA = Speedy Stab...
* How do I create an animated SD avatar? - • Create your own animat...
* Installing Anaconda for MS Windows Beginners - • Anaconda - Python Inst...
* Add anything to your AI art in seconds - • 3 Amazing and Fun Upda...
* Video-to-Video AI using AnimateDiff - • How To Use AnimateDiff...
* One image Gets You a Consistent Character in ANY pose - • Reposer = Consistent S...

[@Mediiiicc]

Need one of those experts to specialize in "hands" lol

[@sherpya]

one expert per finger 😂

[@greengoblin9567]

Per cohk

[@MarcSpctr]

Finally a finetuned model for hands and legs can be used as expert, and maybe some model which can understand stuff like ON, ABOVE, UNDER, INSIDE, etc.

[@MrGTAmodsgerman]

What you wanna generate with "inside" in relation to body parts?

[@elihusolano5993]

Hope you have a speedy recovery. Thanks for the great content.

[@ritpop]

I don't comment a lot but your content is great. Hope you get better soon.

[@worthstream]

This will be a game changer as soon as it's somewhat optimized. Expecially if they do manage to release a finetuning framework. Using prompts to compute gating functions is an ok starting point, but a (relatively) quick fine tune of that can make the difference.

[@paulpardee]

early days, as you say... I don't think this really gives the concept a fair shake. You have models that are better at one thing than others, but all the models currently out today are generalists who just happen to be slightly better at text or prompt adherence, or counting... An expert model would be focused on just text or just counting and those don't exist as far as I know.
I'd love to see models built for this that have markup built in to tell Moe what they specialize in so it could direct that work to them... It'd be even better if you could have a standard library of models and Moe would dynamically load the best ones based on your prompt.

[@aimademerich]

This is phenomenal!!

[@c0nsumption]

Fn love that you’re always willing to get dirty when there no community support bud. Thanks for the hard work 🙏🏽

[@stephantual]

Thanks that was fun :) 🤠

[@ImAlecPonce]

looks so cool!! I only have 16 gig vram though

[@kariannecrysler640]

So few comments! I’m not used to that lol. Hope you’re good my nerdy friend ✌️💕🤘🥰 🐭

[@NerdyRodent]

Will be soon!

[@kariannecrysler640]

@@NerdyRodent very happy to hear that 😁

[@Sandy5of5]

hoping you're feeling back to full rodent normal soon

[@AC-zv3fx]

I wonder if it is possible to create MOE of Pony diffusion, AnimagineXL 3, realistic model and a model that is based on illustrations or traditional paintings.

[@blacksage81]

I feel like these researchers skipped a whole breakthrough by skipping Qlora, and the myriad of quantization flavors we could have played with and went straight to moe, when nearly all the models are just finetunes of the sd base. Its odd.

[@yahiiia9269]

Could you theoretically use multiple LCM Turbo models?

[@nickolaygr3371]

its like computer processors evolution

[@elihusolano5993]

can this new MoE be applied to Loras?

[@fast_harmonic_psychedelic]

i guess its a little better but CLIP training with partiprompts would be just as good

[@AC-zv3fx]

I thought those experts must be trained with the model, so it can know what model to choose

[@poipoi300]

Wonder if we could truly consider this MoE. Haven't read the code, but I suspect all this does is amplify bias, probably akin to LCM but instead it's distributed.

[@LouisGedo]

👋

[@fast_harmonic_psychedelic]

the whole MOE paradigm seems to me to be theoretically dubious lol

[@Alice_Fumo]

Ok, but why?

[@sadshed4585]

what cuda do you have? my torch is not saying cuda is available

[@NerdyRodent]

I use 12.3 locally

[@oquletz]

i don't realy understand what is this. this is a tool to merge models? does it work for sd 1.5?

[@mattkupka1702]

How was this much different than a checkpoint merge

[@fast_harmonic_psychedelic]

theyre all general models, none of the constituents are experts on any particular thing lol

[@kallamamran]

Isn't this just the same as merged models?

[@DoorknobHead]

___m_/ o o \_m___
0:46 Can someone take the Segmoe Ferret to the vet and get that ringworm removed from it's neck? Thanx, in advance.

[@JavierGarcia-td8ut]

in the SDXL I think you are using too low CFG setting... maybe?

[@aimademerich]

Wow this whole time I thought your voice was AI, get well soon

[@renovacio5847]

By by Chat GPT4 😂.. i was using it because the image generation.. but know..

[@AliasArketer]

I boggle at what has been done, I boggle at what may yet BE done. We're in territory that we can't show grandparents and convince them it isn't magic anymore than other silly daftards can be convinced it isn't copy-paste.

[@Guytron95]

groovy. Too bad they didn't include image-to-image but still groovy.

[@erics7004]

Me, with 4gb vram GPU 😢😢

[@raymond_luxury_yacht]

24gb humblebrag