Dimitris Papailiopoulos - "Self-Improving Transformers: Overcoming Length Generalization Challenges"

Time: Wednesday, Jan 29th, 12:30-1:30 pm
Speaker: Dimitris Papailiopoulos (UW-Madison)
Title: Self-Improving Transformers: Overcoming Length Generalization Challenges
Abstract: Large language models can perform algorithmic tasks through test-time computation but struggle to generalize far beyond the task difficulty of the training distribution. These limitations manifest across even simple tasks like arithmetic, string manipulation, and maze solving, where transformers learn shortcuts rather than the underlying algorithms. While prior solutions modify transformer architectures with task-specific engineering, we overcome these limitations with a general-purpose, self-improvement approach using standard transformers. Our method starts with models trained on simple problems, then iteratively uses them to generate training data for progressively harder tasks. Scaling this weak-to-strong training approach yields (seemingly) unbounded improvements in both length and hardness generalization, allowing models to solve problem instances far exceeding the difficulty of those in the training data distribution. We find that “controlled sampling” of problem difficulty is key and also the ability to filter out “negative” self labeled examples; without it, generalization performance plateaus. Our results show that careful self-supervision allows small transformers to transcend superficial pattern matching failures and learn multi step algorithms.

[@sosson97]

great work!!

[@pelayocf4558]

I wonder if the techniques shown in the paper can be applied to things other than arithmetic operations. The way language models multiply numbers reflects a general problem in how they function, and if we are able to teach them to operate in the way suggested in the paper for anything without having to repeat the techniques with each specific type of task, it would represent a major advance in the utility of language models.