Dreambooth and Lora Tutorial - Checkpoints, the basics and walkthrough
HTML-код
- Опубликовано: 4 июн 2024
- Hey Everyone! This tutorial builds off of the previous training tutorial for Textual Inversion, and this one shows you the power of LoRA and Dreambooth custom checkpoints!
Training allows us to push our models toward a very specific look, and here's where we can learn how to give that push.
00:00 Intro
00:45 Understanding the difference between LoRA and Dreambooth
02:40 Processing Images
04:40 Extension Information
05:15 Create Model Checkpoint and Settings
07:00 Concept Settings
08:50 Choosing LoRA (or not)
linktr.ee/FrankTankGaming
Extensions:
Locon: github.com/KohakuBlueleaf/a11...
Dreambooth: github.com/d8ahazard/sd_dream... 
Кино
I'm glad i found your channel, very well explained!
Thank you!
Thank you for the tutorial!
Thanks for checking it out! Glad it helped : 😃
Lots of good info scattered around in this tutorial. It was quite confusing that you talked about different subjects though. A focus on just the rabbits from start to finish while showing the steps in detail would have helped tremendously. Still, I got enough out of it to get started and I figured the missing steps out (like having to resize the images and clicking the Save Settings button before starting the training). Thanks for this!
UPDATE: after creating the Lora and copying it to the Lora folder, I can use it, but it doesn't seem to do anything. I get the same image with or without the lora.The checkpoint it created does have a huge effect, but I preferred a smal lora file. Any ideas?
UPDATE 2: I needed to check "Generate lora weights for extra networks" in the Saving tab and install the extra extension you mentioned. I'm now retraining the model, hopefully a working lora file pops out. :)
UPDATE 3: It create a .safetensors lora file and placed it in the loras directory, but if I try to use it I get "AttributeError: 'str' object has no attribute 'filename'". Still no luck.
I really want to create consistent characters based upon images I have hand selected, so much to the point that I just purchased 128Gb of ram, a new power supply, and an RTX 4090 Gpu. my long term goal is to create assets for low/no budget games for any engine, from Renpy to Unity. so far every single content creator on the subject of AI has done amazing work, but as you said, Ai is changing so rapidly it seem like every tutorial is outdated from day one. every time I've tried to train a model with Dreambooth I've had to completely wipe stability and all of its models/extensions, and re-download everything.
Can you do a tutorial for model training on SDXL @x1024 in windows 10. for some reason no training method seem to work in dreambooth for me.
How would I go about it, if I want to create couple illustrations (boy+girl)? Do I train a model or lora?
ca you train any check point. .ckpt or do they have to be "trainable" . there are more realistic models now then 1.5 base. so say i want to train one with images of me or jon doe can i do it or are some .ckpt files "locked" and untrainable.
Hmm. I don't believe it's possible to lock one, but only because I've never seen it or ever seen an option for it. I've never had any trouble loading a checkpoint or merging it to train it off of. :)
Stumbled upon your channel. Interesting video. Will definitely try them out.
Tiny suggestion - see if you could add a pop filter on your microphone. The high frequencies are too ‘sharp’. You could also do a quick fix in Audacity or something. It’s not a deal breaker but given that sound is such an integral part, I fthink you’d greatly benefit from fixing your audio.
Your content is certainly top notch. I love the hands-on approach. Keep em’ coming!
Thanks! Appreciate the feedback as well :)
how much vram is required while training?
This was the most useful of dreambooth tutorials I've seen so far. It was really confusing to me what was the difference between LORA and Dreambooth models and also I didn't understand the instant tokens and prompts.
However, last time I trained a model using LORA checked and I still received a cpkt file. How is that possible? Is the LORA baked into the new model then?
Dreambooth has been updated, it can generate .ckpt files after training. Not sure how well it works though
I have photos of my dad (passed away last year and his dog, who died 3 days later. They weee inseparable.
Out of all the different options & methods that exist, in your experience, what would be your recommendation for generating (realistic) pictures of the two of them (in the same image)?
Thanks in advance. ❤
A dreambooth model based on your father would be the first move I would recommend or a hyper network (or both, you can use the same material).
You probably want to do the process completely separately for the dog, and then you can either combine the models or maybe use tricks like inpainting to generate each of them one at a time. It'll be hard to get good results trying to recreate them both at the same time in a single image rather than a few processes stacked together. Good luck to you!
to be honest, take a picture of your dad and then either train an AI on pictures of his dog, and try and use prompts and inpainting to refine to your liking, or just give a bunch of reference to a skilled photoshop artist.
If "man" is the class token. Would you caption your training images with or without the classification token. For Example "sks man standing in front of a wall" or just "sks standing in front of a wall". What would be better?
You can experiment, but I've had the best results with just having the prompt be simpler like "man standing in front of wall." The training is going to make connections with each word in your prompt, but your dream booth settings should indicate your main instance keyword that you're going to be calling upon, so it doesn't necessarily have to be in each of the text files.
Don't know why but when I activate "LORA" it just gives me random noise. When I uncheck "LORA" it works fine. I use the default settings with xformers and fp16. Can you help me?
I'd attempt first to bring down the strength of the LORA - start with a 0.1 or 0.2 to see if that helps, and if so, the issue is just the training strength.
You could retrain it differently, or you could just use it at lesser strength if this helps.
One way to make sure that your LoRA is marketable is to make sure your incoming dataset doesn't interfere with general prompt words. For instance, tying "Morgan Freeman" to class "man" is going to cause huge problems vs. tying "Morgan Freeman" to "actor" still provides the class/type relationship but won't effect general requests. "Actor" is used significantly less often than "man". The prompt "three unique individuals morgan_freeman and two men waiting at an elevator" is now going to pump out three Morgan Freemen. Conversely, no one is typing in "...and two actors...".
This is the link to the first video: ruclips.net/video/Ep4T8fyy2LE/видео.html
It is called: "Textual Inversion Tutorial - Embeddings and Hypernetwork basics and walkthrough"
Thanks!
Thank you! Glad it helped ☺️
i am not familure with sanity prompt and had to replay what you were saying a few times to understand it would be nice to have seen a sample sanity output when you went over that. and how to finetune for better results using sanity prompts. i actually stopped training models because the process was so hit or miss for me
In case it wasn't clear, the sanity prompt is a kind of regular prompt. If you are changing what 'man' means, for example, you can use that output to judge if you are going maybe too far.
That said, using default settings and the performance wizard should give you an idea of what is 'normal', etc. Hope that helps!
@@FrankTheTankGaming yeah thanks.great to know
'RuntimeError: Exception training model: 'No executable batch size found, reached zero.'.'
I'm sure there are a billion reasons this may have happened, but just finished my first attempt at this and after an hour, I received this message. I was training a Lora using your settings. I don't suppose you know if there's a simple fix?
Hmm. Id be curious if it may be related to the associations you are trying to add or build onto. If the training material seems good, I'd check your sample output to see what's going on, and maybe try different words to see if you get different results
@@FrankTheTankGaming Genuinely apprecite your reply! I'm painfully new to this and not sure I understand some of your terminology. The "associations" that I'm trying to add - by that do you mean "class token" or the BLIP words? Also, by "sample output", I'm not sure what that means. I found the folder where the model folder is located inside the DreamBooth folder, but there isn't anything labeled "sample" in there, just "backups", "cache", "logging", and "working". I'm not sure it finished? I did have "photo of [my model name]" in the "Sample Image Pomrpt". For context, I'm attempting to train myself using 37 images.
@@mrzackcole I would try to start with a checkpoint first instead of a lora, just to make sure the process is functional for you with all of the steps. Can also check the boxes in the settings so that if you cancel early, it will also save a checkpoint. You don't have to wait until it's complete to see if it worked.
To give you an example of what I was talking about, if you are training a lot of pictures of a man, and your class token is something common like the word man, you should be okay. If you used an uncommon word that isn't in the model for example that might cause issues.
@@FrankTheTankGaming Thank you kindly. I'll give those suggestions a go! Cheers!
I have no idea. I set class path, but dreambooth do not recognize pictures I collected. How can you fix it please, if it is a bug with my webui😂
I'm not sure - I'd first try a different set of pictures and try to see which step is screwing up for you. Start with an embedding or a hypernetwork first maybe to see if the material itself is the issue?
@@FrankTheTankGamingthank you, I will try
To fix it
No one caught that the .pt extension is used for Textual Inversions, while LoRAs are typically .safetensors files and NOT .pt?
.safetensors is actually a replacement for .ckpt files typically, and forces them to be 'unpickled' in a safe way, from my understanding.
I caught that. I thought it was strange as well
dreambooth is saving my loras as .pt now and i keep getting an error. did you guys find a fix ?
can i merge a Lora with a dreambooth
Well, you couldn't use the lora ile directly, but you could make images from it and then train them into a dreambooth. :)
Is the tutorial for training lora not dreambooth?
I'm not sure if you saw the whole video, but they use the same process. You use the check boxes to determine what type of output you want based on what you're trying to create
I still prefer Textual Inversion. Fast training and good results. Btw, images don't need to be 512x512. You can throw in any aspect ratios and model still works just fine. Training can be a bit slower.
but ít's my understanding hat Textual inversion is trained based on others' styles that already common on the internet, if you are an artist like comic creator, you must be having your own style, own characters, Dreambooth is much better for serving your work, have to make your own model. in that case, LoRA is good for creating characters, backgrounds, dress...; and Dreambooth is to train the model for the general style (colors, lineart...). am i correct???
@@Dinhlong20 TI training is proof that pretty much nobody has "own" style. TI is just a guide/map to the model. Example: You can do TI from SamDoesArt images that are NOT in the dataset alreay and get 1:1 results. It proofs that SDA is just a copycat. Funny thing is that you can get quite good results with human face proofing that most faces are so generic that they already have similar data in the dataset.
plese make a colab version
I'm not opposed to it but I haven't messed with collab at all yet actually. Once I do I'll see what the differences are. Thanks for your support in the meantime! :)
@@FrankTheTankGaming callab?
Ther are already a few colab versions of Dreambooth. There is the one by ShivamShrirao, TheLastBest, etc.
I wish I could find a decent tutorial on how to do this. You skip around too much and your example between cats, rabbits, and freeman confuse the process.
If you have questions, feel free to ask 🙂
I agree.
You should've mentioned at the very start to just watch the entire thing before doing anything. This jumping around bit is annoying.
Was going to say that I got nothing for it, but as has often been the case: I end up having to restart the SD instance when "Reload UI" doesn't work. I see it working now. Thanks.
BUT, I still say that a warning at the beginning would be nice. The jumping around can be really frustrating to the uninitiated.
@@faddlewaddle2615 I appreciate the feedback and I'm glad you got it working
Yet another tutorial that doesn't work or correspond to what I see when I follow the outlined steps.
Sorry bud -they keep changing the layout, the versions, and the requirements. I promise it worked at the time this was made, and I'll likely have to do another update, but this is moving super fast.