E07 | Fast LLM Serving with vLLM and PagedAttention

Fast LLM Serving with vLLM and PagedAttention (SOSP'23)
Abstract: LLMs promise to fundamentally change how we use AI across all industries. However, actually serving these models is challenging and can be surprisingly slow even on expensive hardware. To address this problem, we are developing vLLM, an open-source library for fast LLM inference and serving. vLLM utilizes PagedAttention, our new attention algorithm that effectively manages attention keys and values. vLLM equipped with PagedAttention achieves up to 24x higher throughput than HuggingFace Transformers, without requiring any model architecture changes. vLLM has been developed at UC Berkeley and deployed for Chatbot Arena and Vicuna Demo for the past 5 months. In this talk, we will discuss the motivation, features, and implementation of vLLM in depth, and present our future plan.
Bio: Zhuohan Li is a CS PhD student at UC Berkeley, where he is advised by Professor Ion Stoica. He is interested in designing and building efficient machine-learning systems. Recently, he has been focusing on the training and serving of large models, specifically LLMs. His works include Alpa, AlpaServe, Vicuna, and vLLM (PagedAttention). He completed his BS at Peking University and has interned at Microsoft Research, Anyscale, and Google Brain.

[@pingkeeng7305]

Thank you for sharing!!👍

[@ethanhe42]

thanks for sharing!

[@aron8500]

Is there a way to get the powerpoint?

[@ginsongsong]

Thanks for the sharing. It’s educational for me.
One question, is the block size(16/32) related to the warp size(half-warp/warp)? Wondering the theory that you define the black size in kv cache.

[@stevenshi8687]

According to my own understanding, the block size is not related to warp size (which depends on the computing unit). The block size is determined by experiments based on the trade-off of cache locality (of using larger block size) and internal fragmentation (as result of large blocks). Feel free to correct me if I am wrong!

[@shabdanbatyrkulov2791]

Thanks for sharing!
Is it possible to turn on an automatic subtitles (with translation)?

[@MLSysSingapore]

Thank you for the suggestion! We wanted to, but RUclips is not giving us the option😭 Sorry for the inconvenience!

[@chenghao0825]

Any implementation that work with Azure?

[@maciejgawinecki1270]

Is there a version with English speaking?

[@MLSysSingapore]

Hi! Sorry that we only have a Chinese version, and RUclips currently does not allow for auto generation of subtitles in Chinese. We will take it into considerations and upload English-speaking videos in the near future!

[@njulijianguo]

maybe i can translate it for you？

[@MLSysSingapore]

@@njulijianguo Thanks for volunteering!