Thank you for creating this demo in python! As I am building a startup, I have come to realize how essential the Twilio platform is. 1. You need Stripe to collect payments from users 2. You need user authentication such as Oauth 3. You need to be able to communicate with your users and collect data about your users. That is where Twilio is essential with its communications apis and segment to collect user data and build profiles. Keep up the great work, and keep investing in the core communications and data products. 😊
Thank you for watching and for the kind words! Happy to be in your customer engagement toolbelt! Let us know if there's any other videos we could make that would be helpful.
Interesting. Thanks, Bart. What's the best RAG solution for using and storing vector bindings in the Twilio / OpenAI ecosystem? Looking specifically for code examples.
If you haven't checked it out yet though, take a look at Twilio AI Assistants. It won't work for this use case but worth exploring nonetheless: ruclips.net/video/TDv0O20lC3M/видео.html
This was cool! My requirement is to build the voice feature on web based chat system in addition to Twilio phone call. I hope I will do it with this video
Can you give me suggestions on how I improve my voice to text problem of twilio , I used gather function but they don't take right input or it cannot correct transcription?
Take a look at the function calling docs for the OpenAI Realtime API: platform.openai.com/docs/guides/realtime/overview?text-generation-quickstart-example=audio#function-calls
This is awesome! I keep getting a busy signal when call the twilio number in my setup. I'm using free Ngrok, twilio (purchased a number). I can see Ngrok, and Uvicorn running in terminal and I'm getting "Twilio Media Stream Server is running!"
Thanks for the guide. Can you tell me how you called the same phone you use as an call receiver? When I setup "Dev phone" to test my app it reset the configuration of the phone number
Yeah you would need a second Twilio phone number for the Dev Phone. Probably easier to just call from your cellphone. I used the Dev Phone because it was easier to demo for the video.
You can make outbound calls as well. Haven't finished a video for it but here's a blog post: www.twilio.com/en-us/blog/outbound-calls-python-openai-realtime-api-voice
I should have called this out properly in the video but for dev phone you need a second Twilio number. You can test by just calling from your phone. I used the dev phone because it was the easiest way to demo in the video (and is a cool tool to test with).
On an upgraded business account you can have unlimited concurrent calls: help.twilio.com/articles/223180028-How-fast-can-I-place-or-receive-phone-calls-with-Twilio- If you did want to do a waiting room style concept you could use Twilio Queue: www.twilio.com/docs/voice/queue-calls
I am deploying this thing in gcloud and trying to call the webhook through public url and calling the webhooks through call initiation . Up here it's saying the initial things but call is getting disconnected and saying websocket refusal error. Is there anybody in your team who could help me with this?
@@TwilioDevs This relates to my use case - I would like it to make outbound calls. I also want to put controls on the responses, since it needs to perform specific actions for me.
@@TwilioDevs Hi, thanks for putting all of this together. I am using this as a starting point for an elderly companion app. For the outbound call tutorial, will it be an add-on to this code base, or will it be separate?
Is this the same model as the one in chatgpt voice mode. Seems pretty dumb and lots of times doesnt understand what im saying and talks about something else completely irrelevant
This is a little bit different since you have some ways to control it. It has been pretty accurate and you can guide it with the system prompt as well as control the "temperature" which will control how much randomness it uses.
Thank you for creating this demo in python! As I am building a startup, I have come to realize how essential the Twilio platform is. 1. You need Stripe to collect payments from users 2. You need user authentication such as Oauth 3. You need to be able to communicate with your users and collect data about your users. That is where Twilio is essential with its communications apis and segment to collect user data and build profiles. Keep up the great work, and keep investing in the core communications and data products. 😊
Thank you for watching and for the kind words! Happy to be in your customer engagement toolbelt!
Let us know if there's any other videos we could make that would be helpful.
Great video! Can one interrupt the AI assistant while is talking?
Yes! If you check the repo there is an update for that: github.com/twilio-samples/speech-assistant-openai-realtime-api-python
How can we integrate the knowledge base to this so that our voice assistant can answer referring to a given data source?
Interesting. Thanks, Bart. What's the best RAG solution for using and storing vector bindings in the Twilio / OpenAI ecosystem? Looking specifically for code examples.
RAG implementation is in Part II. Got it. Liked. Subscribed. Commented. 😀
Hey @PhoneGenieAI! Thanks for watching.
We don't have a standalone RAG solution that you can use with the OpenAI Real-time API.
If you haven't checked it out yet though, take a look at Twilio AI Assistants. It won't work for this use case but worth exploring nonetheless:
ruclips.net/video/TDv0O20lC3M/видео.html
This was cool!
My requirement is to build the voice feature on web based chat system in addition to Twilio phone call. I hope I will do it with this video
Thanks for watching!
great!
is it possible to make a call on my given number with this setup ?
Can you give me suggestions on how I improve my voice to text problem of twilio , I used gather function but they don't take right input or it cannot correct transcription?
Try whisper model . It works fine on transcription part but be ready to face latency issues if you don't deploy it locally
What is the use case you are trying to build?
@@TwilioDevs Ai call support agent
@@rohitmane2578 I have the same requirements, Are you done with it or still doing?
Is it possible to do rag in the middle of the conversation with this setup or is it only speach-to-speach?
Take a look at the function calling docs for the OpenAI Realtime API: platform.openai.com/docs/guides/realtime/overview?text-generation-quickstart-example=audio#function-calls
Is there a way to trigger the first response without needing to say something first?
Check the GitHub repo. It has a "assistant speaks first" option in it that got added after this video was made.
The repo in the description and the code in the video does not match
Hi! Yes, there have been improvements in the repo since the video was made.
This is awesome! I keep getting a busy signal when call the twilio number in my setup. I'm using free Ngrok, twilio (purchased a number). I can see Ngrok, and Uvicorn running in terminal and I'm getting "Twilio Media Stream Server is running!"
Thanks for the guide. Can you tell me how you called the same phone you use as an call receiver? When I setup "Dev phone" to test my app it reset the configuration of the phone number
Yeah you would need a second Twilio phone number for the Dev Phone. Probably easier to just call from your cellphone. I used the Dev Phone because it was easier to demo for the video.
does it makes inbound and outbound calls both or only outbound?
You can make outbound calls as well. Haven't finished a video for it but here's a blog post: www.twilio.com/en-us/blog/outbound-calls-python-openai-realtime-api-voice
On doing twilio dev-phone it overrides the webhook configuration. How will I test it then ?
I should have called this out properly in the video but for dev phone you need a second Twilio number. You can test by just calling from your phone. I used the dev phone because it was the easiest way to demo in the video (and is a cool tool to test with).
Nice!Could u make it so that it can be in a website by like using cradio and python along with openai?
If u do make this video,thank you in advance
I'll see if I can work something in. Thanks for watching!
How would you personally set up a wait room if multiple users called?
On an upgraded business account you can have unlimited concurrent calls: help.twilio.com/articles/223180028-How-fast-can-I-place-or-receive-phone-calls-with-Twilio-
If you did want to do a waiting room style concept you could use Twilio Queue: www.twilio.com/docs/voice/queue-calls
I am deploying this thing in gcloud and trying to call the webhook through public url and calling the webhooks through call initiation . Up here it's saying the initial things but call is getting disconnected and saying websocket refusal error. Is there anybody in your team who could help me with this?
yep, I get the same
@ivant_true try ngrok approach . It solves
Any way we get the number to call us and still use the webhook?
So you want to be able to make the agent place a call?
@@TwilioDevs This relates to my use case - I would like it to make outbound calls. I also want to put controls on the responses, since it needs to perform specific actions for me.
@@josefsieber6226 I'm working on an outbound tutorial right now actually!
Yes, we are working on an outbound call tutorial right now.
@@TwilioDevs Hi, thanks for putting all of this together. I am using this as a starting point for an elderly companion app.
For the outbound call tutorial, will it be an add-on to this code base, or will it be separate?
Is this the same model as the one in chatgpt voice mode. Seems pretty dumb and lots of times doesnt understand what im saying and talks about something else completely irrelevant
This is a little bit different since you have some ways to control it. It has been pretty accurate and you can guide it with the system prompt as well as control the "temperature" which will control how much randomness it uses.
@@TwilioDevs I see. But I pretty much followed everything you did and its quite unusable for me. Will try again
Can you share any modification that would be needed to have the AI assistant collaborate on an outgoing call?
Yes, we've been working on an outbound tutorial. Should have something for you soon.