Getting Started with Groq API | Making Near Real Time Chatting with LLMs Possible

Let's have a look at the Groq API that offers access to their Language Processing Units (LPUs) for free (for a limited time). The API can provide close to 500 tokens per second with Mixtral-8x7B. Enjoy :)
🦾 Discord: / discord
☕ Buy me a Coffee: ko-fi.com/promptengineering
|🔴 Patreon: / promptengineering
💼Consulting: calendly.com/engineerprompt/c...
📧 Business Contact: [e-mail]
Become Member: tinyurl.com/y5h28s6h
💻 Pre-configured localGPT VM: bit.ly/localGPT (use Code: PromptEngineering for 50% off).
LINKS:
Signup for API Access: groq.com/
Colab Notebook: tinyurl.com/2nxdv2m8
Streamlit Chatbot: tinyurl.com/3f485knj
TIMESTAMPS:
[00:00] Introduction
[00:34] How to Access the Groq API?
[01:06] API Playground
[03:15] Getting Started with Groq API
[05:33] Near real-time speed
[07:24] Advanced API Features: Streaming and Stop Sequences
[11:02] Building a Chatbot with Streamlit and Grok API
All Interesting Videos:
Everything LangChain: • LangChain
Everything LLM: • Large Language Models
Everything Midjourney: • MidJourney Tutorials
AI Image Generation: • AI Image Generation Tu...

[@martg0]

Thanks for the video! I will start testing this API with a POC I am working now to learn.

[@dhruvpatel2554]

Awesome stuff !!!!

[@osamaa.h.altameemi5592]

this is next level. OpenAI got some serious competition.

[@thierry-le-frippon]

They should sell their LPUs instead and compete with Nvidia. They would surely get lots of backup and investments. They will probably be copied instead othetwise and fade away quickly.

[@KOTAGIRISIVAKUMAR]

why you cant use the conversational retrieval chain instead of the conversation chain Because it can handle the memory by default no need maintain externally?
@prompt Engineering

[@DestanBegu]

Thanks for your content! I´m using Streamlit as well and want to give Content as the System role. For Example "answer me in short sentences in italian" so it will do this for each prompt i do. Where can i do this in the code? I used the Streamlit Chatbot Repo.
Thanks in advance

[@shaheerabdullah6738]

Very Helpful.

[@jonoburcham4059]

Great video! Can ou make a voice chatbot using groq in one of your next videos please? I would also love to see if you do this on streamlit or if it's too slow and you use something else. Thanks so much for your videos

[@engineerprompt]

Planning on making that. For voice chatbot, might just do cli though

[@benben2846]

tu est fort man ^^👍

[@user-yz8tf3uv4y]

Please Create a step-by-step video guide on using the Groq API with Streamlit.

[@engineerprompt]

That's coming soon

[@vishnuprabhaviswanathan546]

How to control the output of LLM for a single input?

[@mickelodiansurname9578]

Heres the question, can Groq cards also work on inference for art and audio and voice models? or is it just LLM inference specific? It is like, well superfast... the only worry is literally the latency from you to the endpoint... so if its say, a streaming interruptible feed you are giving the model then the use cases for TTS and Speech applications just went through the damn roof!

[@engineerprompt]

I am not sure but I was listening to Chamath (who is an investor in Groq) and he was talking about the initial use cases of the hardware. Seems like they were focused on vision so it might have the ability

[@engineerprompt]

I am trying to put together an example for end to end speech conversation, let's see how that goes

[@jmay3230]

If temp can adjust to minus what is impact on generation ( consider it as hypothetical if case don't exist )

[@engineerprompt]

It will be same as setting it zero :) basically if you set it zero, it will pick the next most probable token. If you set a higher value, it can to sample among the most probable tokens

[@ramimithalouni6592]

what is the time to receive the first chunk in streaming?

[@easy-dashboard]

Depends on the amount of input tokens. With a one line instructions it's below 1 second. If you include context of a RAG-System it will go up to 3 seconds to start the first token (30k tokens of context)

[@jesusleguizamon6566]

Hi, this api have function calling? regards

[@bobsmithy3103]

Can it run other models?

[@sausage4mash]

did someone say free

[@engineerprompt]

For the time being :)

[@siriyakcr]

Wow

[@hmsfaceface8925]

how can the groq fpga use mixtral 8x7b with just 250gigs of vram?

[@coyoteq]

Bcoz of groq tpu...

[@Francotujk]

What are the rate limits of the free api? Is it necessary to provide credit card?

[@engineerprompt]

It's free at the moment and there is a rate limit as well. Seems to keep changing. Last time I checked, it was around 20 messages per minute

[@prestonmccauley43]

I tried a few things with this and it is incredibly fast.

[@engineerprompt]

I agree!

[@ranaayushmansingh2368]

can we fine tune this and use it?

[@engineerprompt]

You can't fine-tune via their api yet.

[@CharlesDonboscoA]

Hi whether it's free or paid ?

[@engineerprompt]

Free at the moment

[@ZombieJig]

Fuck all these cloud only AI services, release the cards!

[@thierry-le-frippon]

Yes, otherwise they will fade away quickly. Their window of opportunity is small. Money is looking at eating in the nvidia cake now not tomorrow.

[@conciousaizielia]

Grok is not a llm it can run a llm

[@TheJscriptor09]

YALLM ... it is almost becoming daily news ... Yet Another LLM.

[@savire.ergheiz]

Fast but useless. These oss models still way far behind cgpt4.

[@manishadeshmukh2519]

Bro groq outsmarts GPT-4 in 70B model

[@manishadeshmukh2519]

It is super faster than gpt 4

[@geo4design]

This is an AD