Видео

Really good and useful talk.

I’m in healthcare, my data isn’t publicly available. Fine tuning has pretty decent results because most things are single shot. Agents/rags should be fun but again our data being segregated is a hassle. I think Bloomberg has to be similar.

Around 46:00 - context size is mostly a vanity metric AFAICT. I'd like to see data about how accuracy varies with the percentage of total nominal context that is actually used by the prompts. In fact, this could be one of the most beneficial uses of fine-tuning, for avoiding filling up the context with very long instructions.

Around 24:00 Another insight here wrt the question of the life science guy, is that when we say "RAG", we tend to assume "out of the box", embeddings match RAG. But RAG is in many special cases best implemented with dedicated software parts that take the LLM query output and use other domain-specific NLP and business rules software to actually do the retrieval of what you care about. In other words, build LLM workflows that are not only using LLMs. Get the LLM to do a task, then use that output to get your advanced semantic retrieval stuff that you know works and embeds a lot of your subject matter expertise to take care of the next step of your workflow, then use that output, which will typically be much more precise than a vanilla embedding, to build your next LLM prompt. I would have given the advice to the life sciences guy that he's very likely not going to get much benefit from fine tuning. You can't train a knowledge representation into an LLM using fine tuning. Fine tuning helps with task-specific input and output simplification and formatting, pruning, compliance, that sort of thing, not with the actual "logical" inference that the model does.

Around 19:10 absolutely fine tuning works better for some tasks, and certainly doesn't for knowledge injection, for which you must use RAG. But you have to take into account also the economics and logistics of it. Fine-tuning is a task-specific thing that you have to do, then maintain all your FT models and so on, which costs money and introduces complexity. RAG and prompting is far more nimble. It's not often in life that the easier solution is in fact better than the complex one.

Around 8:30 to 10:10 - The RAG picture absolutely turns the problem into a search problem that is at least as important as the prompting problem. This is a far less trivial problem than most people realize. Using RAG requires deep thinking about the retrieval part, and this is notoriously difficult using embeddings only, at least if you want to optimize your token consumption, and overall inference time of your prompt chain. You'd greatly boost your RAG-based workflow by not only using embeddings but considering sticking a real search index behind it that is configured for the retrieval that you care about. That's a kind of LLM workflow optimization that I feel is not being talked about.

For some use cases, having a better prompt will add more tokens hence more cost for the same inference. Any thoughts?

I love how all midwits think RAG is some kind of miracle without basic understanding of freshman level math. RAG is literally the most low eye-queue idea one can come up with.

When you say we should be spending days working on prompts, how so? I'm assuming you have a high level overview of how these prompts should be structured in mind, but can you break it down and simplify it for me?

I was expecting more from this talk, the tldr was do RAG first, maybe fine tune later, but it also very depends on a case by case and what you want to do with model.

Thanks for sharing this course! Could you please post the discord link? Thanks!

The problem is if you don't do the cool thing, you help your company continue to generate revenue, but you don't get promoted and you spend the rest of your life writing SQL queries for shit pay

I have some experience about fine-tuning, I mean fine-tuning and transfer learning is still a great technique but you have to had in depth knowledge about the math to know how to apply it. LLM is really hard to fine-tune, some of the time it will break the parameters learn from previous context (just my opinion tho). Some CNN application on the other hand have good fine-tune result since it can capture the shapes and patterns of the image then after CNN layer it still maintain and capture the key features.

Hamel, I just cant thank you enough. for making this public.

The questions that constantly keep interrupting the talk are super distracting.

Another place I think this is echoed is in image diffusion models. IP Adapter has become some powerful in stable diffusion that we're able to basically create instant models using 3/4 reference images and normalizing/averaging the tokens they create. By conditioning a prompt with those tokens via IP adapter, fine tuning base models is in all but the most niche cases, pointless now.

it's a shame the host didn't actually let the guest answer the last question and instead proposed a different question, otherwise I found this very interesting

This paper(Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs) only does continued pre-training(ie. next word prediction) not supervised finetuning so the comparison is not entirely accurate.

I have a question regarding sharegpt style datasets for function calling (tool use) like the glaiveai/glaive-v2 dataset. I am trying to get that work with axolotl but with the latest checkouts from GitHub this seems to be broken despite having been added in a MR to main earlier this year. To my understanding it needs the specific type: sharegpt.load_glaive which has the problem that the tool role never gets actually added and recognized. Now I have seen just a few days ago, Wing Lian has added a commit where exactly this issue was solved by modifying the code in those functions and adding another field “tool_field” in the config to allow for setting the tool role correctly. I see that this commit has not been merged and has been deleted and closed for some reason. Why is that the case and will the use of sharegpt tool datasets be fixed somehow? P.S. I also created a bug request for this on GitHub. It would be nice if you could come back to me about this issue as I desperately need it. Thanks

I like the questions but really wished they were asked at the end of the presentation rather than break the flow and keep on having the answer being a few slides or talking points down the way.

Thanks for sharing! One question I had about `input_output` format discussed at the beginning when specifying your own prompt templates. If you are fine-tuning a base model, I assume you need to come up with your own start/stop tokens (in the example, that would be <s> and </s>) and explicitly include them in the training pairs? If so, how do you know if the tokens you pick for that purpose are already in the model's vocabulary, or if you should add them to the embedding table?

Great Course. Thanks for putting this out. Is the slides shared anywhere? Didn't find it on the website.

46:53 context window line is incorrect. For the models you mentioned we are are around 100K average

35:41 sorry but I don’t get that. Isn’t the hard part creating the fine tune data set? The actual execution isn’t that much cost, is it?

11:16 I think I get the idea of where to not finetune. But for the 5% where it is needed, I always thought that it’s like when you want to carve out a specific part of the general knowledge and behavior LLMs have. So, to give concise fitting financial advice LLMs have to ignore much (bad) knowledge and also stop a lot a expansive responses. Thats bot something good prompting can achieve, right?

This guest has quite a condescending attitude, I don't buy what he's saying as a result of it. On top he came across as rude (his "This is not an Anthropic presentation" etc, you can say that in a less aggressive way. Anyway, interesting topic, but it's too much founded on fluffy lines and reasoning that serves ... suppliers of big foundation models like Anthropic. As closely guarded models like those of Anthropic are the opposite of being easy to fine tune. Anyway, Meta will prove open weights model are the future, and yes finetuning has it's place not everything is an LLM-problem built in to the foundation model.

Could finetuning make the model overall less capable overall, i.e. forget general knowledge, reasoning, instruction following, and other abilities that help you answer questions even in the domain that you are finetuning for. And this type of pervasive capability loss maybe hard to measure.

In the context of diffusion-based image models, fine-tuning is infinitely more important than prompting, so I don't think your talk applies to all machine learning models.

The conceptual analogy that I like to use comes from cartography. The LLM is a map of regions called "concepts" that are projected into the multidimensional tensor space of tokens. Fine-tuning is a conformal map projection of this tensor space to create a "view" appropriate to a user's domain. Prompts are tokens that adjust the zoom level of the conformal map to view greater detail and narrow the possible output responses from the tensor space. RAG is like "street-view" images or satellite data that adjust the temporal window of the map beyond its training cutoff date. Prompts can be optimized for either the LLM or the fine-tuned map. If the prompt tokens are optimized for the LLM, fine-tuning is superfluous. If the prompt tokens are domain-specific for a conformal "view," the fine-tuned map should perform somewhat better.

In LLM, fine-tuning just makes more complicated the problem of grounding: where the output comes from? the base model, the learned data or from nothing (hallucination) ? that's why embedding-based search is great: you know what data you're generating your output from. In some industries (e.g., medical), being able to justify ("ground") an answer is mandatory. There are a few use-cases for fine-tuning, if the cost for transforming the data in pair (prompt -> completion) is not too expensive.

I am fairly convinced that we are doing LLM wrong. We should have language models that generate complete nonsense, and puppet them with knowledge models. So if RAG is how you do this, you want your non-RAG model to score essentially zero on all benchmarking except language structure, and inject everything with RAG or some other knowledge or skill injection.

ONLY FOR LLM!!!

great and informative conversation! I wonder if there are more channels like yours!

Nice session. Thank you. I would love to see a reliable stable base model understanding the languages. But the domain knowledge is questionable always in my opinion, as most/some? of the books used for base model training (technical books, advanced papers) are quite complex and I am still not truly convinced that text + pictures + math was captured with very high precision.

Are you comparing Fine tuning with Prompting ????? Who made this chart?

hey man , kindly check your work email !! 😊

Fabric AI summary: SUMMARY: HL, Dan, and guests discussed evaluation methods for large language models, including unit tests, using LMs as judges, human evaluation, and various metrics. IDEAS: - Unit tests are a first line of defense for catching obvious failures. - Look at your data rigorously to find failure modes to test for. - LM as judge can help scale evaluations but requires human alignment periodically. - Human evaluation is important but doesn't scale well for large datasets. - Metrics like recall, ranking, and ability to return zero results are important. - Evaluations should evolve as you learn more about failure modes. - Code-based and LM-based evaluations have different use cases. - Iterative grading of outputs can help refine evaluation criteria over time. - Evaluation criteria may drift as you see more outputs from the LM. - Avoiding contamination of test data in base model training is challenging. INSIGHTS: - Evaluations enable fast iteration and feedback for improving LLM applications. - Different evaluation methods suit different use cases and stages of development. - Evaluations are an iterative process of discovering and codifying desired behavior. - Human judgment is crucial for aligning evaluations with true goals. - Evaluation criteria and implementations should evolve with increased understanding. - Logging outputs and revisiting evaluations is important for production systems. - A combination of methods is often needed for comprehensive evaluation. - Evaluation frameworks can help but the hard part is understanding requirements. QUOTES: "If you don't have really dumb failure modes like things that can trigger an assertion often times like is it's natural to think that hey like I can't write any unit tests for my AI because it's spitting out natural language and it's kind of fuzzy." "We want to make sure that the more relevant ones are closer to top personally for me what I find to be quite important uh for rag is this metric that I've never had to considered before." "I don't think an evaluation interface we learned no evaluation assistant can just be a One-Stop uh thing where you grade your examples come up with evals and then push it to your Ci or push it to your production workflow no you've got to always be looking." "Grading has to be continual you've always got to be looking at your production data you've always got to be learning from that." HABITS: - Look at data rigorously to find failure modes to test for. - Use LM as judge but periodically check human alignment. - Conduct human evaluation regularly, especially for evolving criteria. - Log outputs and revisit evaluations for production systems. - Iterate on evaluation criteria as understanding of requirements increases. - Grade outputs continually to refine evaluation criteria and implementations. - Check for contamination of test data in base model training. - Use a combination of unit tests, LM judges, metrics for comprehensive evaluation. FACTS: - Unit tests are limited for open-ended language model outputs. - LM as judge can provide directional signal but requires human alignment. - Human evaluation doesn't scale well for large datasets. - Metrics like recall, ranking, zero-result ability are important for retrievers. - Evaluation criteria may drift as more outputs are seen. - Code-based and LM-based evaluations suit different use cases. - Avoiding test data contamination in base models is challenging. REFERENCES: - Hamil's blog post on the iteration cycle - Spade paper on generating assertion criteria - Shrea Shankar's work on systematic LM judging - Langs Smith for logging, testing, datasets - BrainTrust, Weights & Biases tools mentioned - Instruct library for honeycomb example - Eugene's writeups on LM evals, hallucination detection, domain fine-tuning ONE-SENTENCE TAKEAWAY: Comprehensive evaluation of large language models requires an iterative process combining multiple methods like unit tests, LM judges, metrics, and human evaluation to continuously align with evolving goals. RECOMMENDATIONS: - Write unit tests to catch obvious failures as a first line of defense. - Look at data rigorously to find and test for different failure modes. - Use LM as judge but periodically check alignment with human judgments. - Conduct regular human evaluation, especially when criteria are evolving. - Log outputs and revisit evaluations for production systems to refine criteria. - Iterate on evaluation criteria as understanding of requirements increases through grading. - Use a combination of methods like unit tests, LM judges, metrics. - Consider evaluation frameworks but focus on understanding requirements first. - Check for contamination of test data in base model training data. - Evaluate agents by breaking down into steps and evaluating each component.

Extracted Wisdom from Fabric Summary: This is a discussion about fine-tuning large language models using tools like Axolotl and Hugging Face Accelerate. The key points covered include: Ideas: - Fine-tuning involves adapting a pre-trained language model to a specific task by further training on relevant data. - Choosing the right base model (size, family) and using LoRA vs full fine-tuning are key decisions. - LoRA (Low-Rank Adaptation) is recommended over full fine-tuning as it requires less GPU memory. - Quantization like Q8BERT can further reduce memory requirements with some performance tradeoff. Insights: - Focus more on curating high-quality training data than obsessing over model details. - Write evaluations and assertions to validate outputs and filter bad training examples. - Use techniques like LLM-as-a-judge to encode human preferences into the fine-tuning process. - Iterate between training, evaluating, and improving the dataset - it's not a linear process. Habits: - Always sanity check the fine-tuned model's outputs before deployment. - Inspect preprocessed data to catch tokenization issues early. - Use tools like Weights & Biases to track training metrics. - Start with working example configs and make incremental changes. Facts: - 7B-13B parameter models are a popular sweet spot for fine-tuning. - Larger models require model/data parallelism techniques like DeepSpeed and FSDP. - Mixed precision training can reduce memory requirements with little accuracy loss. Recommendations: - Use Axolotl as it bundles best practices and rapidly integrates new techniques. - Try Modal for cloud-based parallel hyperparameter tuning of Axolotl jobs. - Leverage tools like sample packing, offloading, and efficient memory loading in Accelerate. - Explore deployment-time techniques like greedy/topk sampling for deterministic outputs. One-Sentence Takeaway: Focus on curating high-quality data and iterating between training/evaluation, using tools like Axolotl to rapidly leverage cutting-edge model parallelism techniques.

LLM hype is causing severe brain rot.

Is there any details about this video like the first one? and Chapters?

When fine-tuning LLM with techniques like LORA, what is the model actually doing? I know it is weight updating, but what does the model learn from it? Can anyone explain in an intuitive way.

I think what is misleading here is conflating machine learning with LLMs. The scope of LLMs is not the same as machine learning overall. Fine tuning of foundation models is not dead. Furthermore, aren’t the big LLM companies like Anthropic already doing extensive fine tuning on their own base models before releasing to the public? How does that fit with this idea?

great presentation

This was great, thank you.

I saw that you are looking for a thumbnail designer, plz let me know with yes or no.

share your email please

Nice discussion, thanks for sharing. I am 70% into it and still didn’t hear examples or justification why fine tuning should be avoided. Lots of evaluation results, but that does not make sense if you are fine tuning. You are doing that to work on your custom data mostly and therefore generic evaluation models may not apply nor portray the real performance of the fine tuned model. I fine tune for example to do better detection of service requests into categories and potential solutions.

no description is given notes and all

Thank You 😊

Notes link is not working :)