Видео

Hi man, Should I get a used 3090 or a new 4070 ti? Sadly 4070 ti has the half Vram so I leaning towards 3090 but what you think?

Just a note: sitting here with a 32" 1440P screen, and I can barely read the text you are showing

I liked every comment on this video!

work on parallel every thing multi lan multi optical fibres theır not that expensive but ı feel ai and lights are better fit! multi nmve on different pcies you need to map every single pcie lines to that cpu loaded up to the point that oems have use fans too cool motherboards multi rams that u do even training the main mode to use small swarmed agent models in smaller vrams something like treadreaper even the cheapest oldest has 4 time lines like 4 16 lane pcie 4090!

im working hard to run it with amds i feel youve been a miner, a miner can run gpus no mater how many. if gpu miner could. sell ai tokens! also multi agent approach can us small devices

Two questions sir. Have you tried other inference backends? There's a project called Cortex cpp - a kind of alternative to ollama made by guys, who developed Jan AI. The main pro for it is that it supports TensorRT - Nvidia inference engine (and specific model format, different from ggml). These guys are claiming it's 40-60% faster, than llama cpp. They made a big comparison between post about it, running different GPUs and everywhere TensorRT was a huge win. Second question is - did you try to equip tensor parallelism?

Looks like it'd be better to buy ram and offload the models there since it has little to no difference in speed between cpu only or 4x 4060ti's. Also would save a lot of money from expensive gpu's. Perhaps just buying one or 2 4060ti's for smaller 8b models would make more sense, but for larger 70b models it's useless. If one is super desperate and already have one 4060ti and want to load a 70b model it'd make more sense to simply use the one 4060 with the rest of the model offloaded to Ram (if you have enough ram) since you won't be missing out on anything in terms of speed compared to 3 or 4 4060ti's.

nice. but then there is price too if its just a test lab. considering in australia a 3090 is USD $1500-$2000 in australia. and 4090 $2500 USD. so tempted to get an old tesla. but the 3090 just works in simple motherboard.

Superb! Could you make a tutorial on how to setup and implement everything needed (SOFTWARE WISE) to achieve what you did here?

Very interesting.

Can someone explain in a nutshell what this is? Is it an Ai language model like chatgpt that runs entirely offline on my own computer?

Noobie question, can you use the same 6x 4060TI 16GB for model training (as in, would it make a total of 96GB of available memory) in order to overcome memory limitations on a single GPU?

405b

I have two Tesla P40s here but I unsuccessfull in my trys on making use of both for my AI workloads. especially my stable diffusion trainings are taking very long. do you know how i could make them appear as one large gpu?

good stuff, deserve more subs.

Could you try the amd radeon instinct series? Those are cheap and have a lot of vram

How many shrouds and fan sizes have you tried on Tesla GPU's? I want to get a quieter run which a larger fan could theoretically do but the shroud funneling might be a source of noise so I don't know what to get for best silence.

running optiplex 7040 sff with 24 gig ddr4. i5 6700 .3.4 gig 4 cores. no gpu. i get 5 tokens per sec in ollama run llama3.1 8b --verbose, 9 tps on the new 3.2 3b. on the single test "write a 4000 word lesson on the basics of python". its usable. ollama run codstral. 22b pulled a 12 gig file. same test : results use 99%cpu 0%gpu 13 gig ram. it crawled 7 min.. 1.8 tps. it ran.

Your content is what I define as Gold, much appreciate the work

I was waiting big models to be ran on these :) How about 6x a770 16GB GPUS?

thx

Are you going to test Intel A770 performance?

I think where this gets really interesting is not necessarily as a chatbot, but as an agent. Agents capable of running in the backround over a large quantity of time without user intervention don't really depend on latency, but depend a lot on bandwidth. In that case, it's not uncommon to have a Raspberry Pi running in the backround somewhere, and it sounds really stupid to try and run a huge model on it at first, but in a distributed setting it gets really interesting because you could imagine a swarm of other people with a similar device, and you could queue up requests such that everyone "gets their turn" on the model. It's also potentially useful for large batch requests (ie: synthetic data generation, database labeling, dataset labeling, potentially coding agents, etc) where you kind of just want it done, and as cheaply as possible, but don't really care exactly when it gets done. I could imagine some player in the space (Kobold swarm, etc) offering some measure of contribution (credits), and you could trade in slowly generated passive credits (ie: lending your SBC to the swarm) for a smaller number of fast tokens run on a GPU cluster, etc.

I was watching the GPU utilization percentage and it seems that the average is around 25% for each GPU or am I wrong? Is it expected to be so or is there any configuration to utilize more GPU % during the inference?

Excellent explanation of this feature.

PNY 3090 Revel response tps: Q8: 88.3 tps; FP16: 52.4 tps.

Asus Strix 4090 response tps: Q8: 99.2 tps; FP16: 60.4 tps.

I want to run big models cheaply, I use a 1080 TI now on 8b llama, fast enough but would like a reliable code assistant with bigger model. Suggestions? Can you test multiple 3060s in parallel on big model?

Just to be clear, can vLLM host llms like LLaMA or Mistral and distribute queries across multiple server racks, such as three racks, each with four GPUs?

Excellent video. Definitely feel better now about not getting a 4090. Now it's between a 3090 and a 7900 XTX

5090 is going to be amazing , with that gddr7 memory modules.

If you don't need speed, the fact that the M40 24gb can run the same models as the 4090 though is impressive for the value.

The price to performance ratio is not convincing at all. Why would I want this over say a dual 3090 setup? Or am I reading the data wrong?

Love my 4090, bought it at launch. Watercooled version was $2k, now selling for $2400 💀 Can't wait for the 5090. 32gb 👍. Wish it was 64.

Keep cranking this out. love it.

4090 is the Godzilla of LLM and gaming GPU, it’s 30% faster than my 24Gb AMD Radeon 7950 XTX 🧐.

If you get the chance could you run tests on intel a770 and 7900xtx . I want to know how far behind are them compared to nvidia.

My new phone runs rocket 3b llm (~3gb) on my phone ams gives answers in under 2 seconds. I have 3.3ghz based on 4nm and with ai hardware support + 16gb ddr5 ram. I can use an offline picture generation ai which finishes a 512x512 picture with 20 steps in around 2 minutes. Thats absoluteley INSANE IMO

mates I have a question. currently I have a 3060 ti 8gb and I want to pair it with a 4060 ti 16gb for deep learning. do you thinks it is a good chois to go with a 3060ti 8gb and a 4060 ti 12gb for deep learning?

a reminder - you could get a gpu with lots of memory banks and upgrade the memory chips in to have huge capacity cards. you could for instance mod a 12gb rtx 3080 to have 24gbs

this is due to extreme pci-e bandwidth bottlenecking, probably. putting a ton of gpus together without the bandwidth to push the data to them and back yields no improvement.

Mistral is the only AI I like and trust. These AI's eat a lot of power, yet they expect us to go green ??? What do you mean by fully offloaded ?

you can undervolt the 3090 and will use around 280 watts with basically the same speed :) (only like 3% slower) (And if you use tensor cores, the core can run at 1500mhz while the tensors run the same speed and around 280 watts total)

Excellent video. I've been thinking about getting a AMD 7900XTX since it has 24GB but it is less than $900 new since ROC supports interface on LM Studio. I checked out the website and didn't find any AMD GPU testing. Any thoughts?

I'm trying to do this locally. I have two separate systems, with 3090's connected via NVlinks. It's on a 2.5gbps network, not connected to the internet. Do I need to get a 10Gbps to make this work better, or should I run them all docked in the same system, and drop the NVlinks. Eventhough the NVlinks give me 48gb Vram. My hope was to get them to network, to push 96. Sorry for the technical errors in my question, but I'm pretty noob. It's the first time I have setup a network, and first time I've used SLI/NVlink. So this might not make any sence, but I have to try. Thank you for the great video.

I know it’s be SLOW… but very curious what 80B FP16 (Q16) would do? As that’s what we need even if we have to wait all night for an answer (but couldn’t wait 4 nights!). 🙃

around 4 Times faster than cpu only... But around 100x more expensive...

I'm interested in more tests on how two or four 4060ti comparing against a single more expensive consumer card like 3090 or 4090. The benefits of larger total VRAM 2x16GB or 4x16GB against 1x24GB. Also, the graphs not showing CPU usage, is there any bottleneck from CPU? Are there benefits from more cores, or single threaded performance is crucial?

The graphs are showing ~20-25% GPU usage each on all four GPUs. That also explaining 180W power draw in total, and not like 400W or so (100W per GPU). Could you please explain why it is not consuming all GPU power?