I haven't played around with Oobabooga - but looks like similar functionalities (although I didn't see a .exe installation of Oobabooga) - in my experience with LMStudio vs other similar offerings, LM studio was the best by far: book.premai.io/state-of-open-source-ai/desktop-apps/
It depends on the type of quantization. A rule of thumb is for 8-bit quantization it is the same i.e. 30b parameter model 8-bit would need 30 GB of ram (preferably GPU)
@@Derick99 You would probably need to figure out if LMStudio could communicate with multiple GPUs. I know packages like huggingface accelerate can handle multiple GPU configurations quite seamlessly
thanks a lot for sharing this with us
great information! glad you are uploading this! but i would really appreciate if you upgraded your mic :3
sorry i did not understand which file needs to be modified for the external server and where i should look in the Lm Studio folder. thank you :-)
thanks ! Do you know how to connect it with your own data ?
Privategpt
Please share your laptop specifications. Mine works so slow...
how to use lm studio server as a drop-in replacement to OpenAI API?
please make a video on this ASAP
openai.api_key = 'your-actual-api-key'
openai.api_base = 'localhost:1234/v1'
mistral_response = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
messages=[
{"role": "user", "content": PROMPT_TEMPLATE},
],
stop=["[/INST]"],
temperature=0.75,
max_tokens=-1,
stream=False,
) Here ya go
when i tried to download model and then tried to run it's not loading model on studio can you please help me with that?
what's the difference between this and oobabooga?
I also would like to know
I haven't played around with Oobabooga - but looks like similar functionalities (although I didn't see a .exe installation of Oobabooga) - in my experience with LMStudio vs other similar offerings, LM studio was the best by far: book.premai.io/state-of-open-source-ai/desktop-apps/
can you run 7b model on normal i5 16gb ddr5 laptop without gpu ?
Try LMStudio - combined with quantization
how do i change the chatgpt url with this?
what's the requirements for the 30b and larger models when quantized? how much vram or system ram is needed?
It depends on the type of quantization. A rule of thumb is for 8-bit quantization it is the same i.e. 30b parameter model 8-bit would need 30 GB of ram (preferably GPU)
@scienceineverydaylife3596 what would something like 128gb ram but an 8gb gpu 3070 do compared to having multiple graphics cards
@@Derick99 You would probably need to figure out if LMStudio could communicate with multiple GPUs. I know packages like huggingface accelerate can handle multiple GPU configurations quite seamlessly
Your microphone is terrible, but the video is great. Thanks. 🎉🎉❤
whats the memory requirement for windows 10? can it run in cpu mode?
Yes CPU is fine - although slower inference than GPU
Is the SAAS 100% free?
LMStudio is free!