Overview of an Example LLM Inference Setup

[@FarhadOmid]

Great work, Jordan! Gonna start scraping the parts together...

[@rodrimora]

I feel jalous of that 8xH100 server. Currently using a 4x3090 at home. I actually use a pretty similar setup with vLLM for the full precision models and exllama or llama.cpp for quantized models + OpenwebUI as a frontend.

[@MadeInJack]

Why would you need more than that? Be glad for what you already have or you won't find happiness :)

[@ricardocosta9336]

Bitch i have a p40 and im over the moon. Being poor in ml is hard.

[@fakebizPrez]

Sweet rig. Is that your daily driver? 😀😀

[@niceshotapps1233]

- what are you using it for?
- .... stuff

[@0101-s7v]

AI, apparently. (LLM = Large Language Model)

[@KCM25NJL]

The cost of such a setup is circa $500,000........ amma get me 2 :)

[@ZiggyDaZigster]

30k GC? 8 of them?

[@nesdi6653]

Word

[@nesdi6653]

Why not podman tho