Title: Aligning Language Models with LESS Data and a Simple (SimPO) Objective
Abstract:
Aligning pre-trained language models ensures they follow human instructions reliably to produce helpful and harmless outputs. Supervised fine-tuning and preference optimization are two key approaches for achieving this goal. In this talk, I will introduce two novel algorithms designed to enhance these two stages. First, I introduce LESS, a model- and optimizer-aware algorithm for data selection. LESS leverages a few curated examples to identify instruction-tuning data that fosters specific capabilities in the model. It avoids relying on surface-form cues by framing data selection as an optimization prob...