Open AI Embeddings in Azure Vector Database of Cognitive Search
HTML-код
- Опубликовано: 7 фев 2025
- Connect Open AI Models to your Data using the new Vector database of Azure Cognitive search for having hybrid search indexing ( based on both word embeddings and semantic search) for Retrieval Augmented Generation (RAG) or chatGPT on your own data.
☎️ Do you need any career or technical help? Book a call with me: calendly.com/m...
Use ChatGPT on your own large data video:
• Use ChatGPT On Your Ow...
Reference code used in this video in the discord channel under the reference section:
/ discord
*******************
LET'S CONNECT!
*******************
Join Discord Channel: / discord
✅ You can contact me at:
LinkedIn: / mohammad-ghodratigohar
Email: mo.ghodrati95@gmail.com
Twitter: / mg_cafe01
🔔 Subscribe for more cloud computing, data, and AI analytics videos
by clicking on the subscribe button so you don't miss anything.
#ChatGPT #AzureCognitiveSearch #wordembeddings #vectordatabase #OpenAI #AI #artificialintelligence #machinelearning #cloudcomputing #chatbot #virtualassistant #customerservice #developers #technews #innovation #Microsoft #CognitiveServices #BotService #naturalanguageprocessing #NLP #conversationalAI #videotutorial #tutorial #learningAI #AIchatbot #AIassistant #AzureAPI #AzureTools #MG #AzureMG
🎯 Key Takeaways for quick navigation:
00:06 🤷♂️ Struggling with deciding between using Azure Cognitive Search and creating a vector database
- Discusses the struggle of deciding between using Azure Cognitive Search for indexing words retrieval or creating a vector database with word embeddings.
- Mentions the possibility of taking a hybrid approach.
01:16 🤝 Introduction to the hybrid approach
- Discusses that Azure Cognitive Search now has the capability to store word embeddings as a vector database and index them, enabling both semantic search and word embedding based retrieval.
- Suggests using this approach for chat with your data scenarios.
03:44 🔎 Deep Dive into Vector Search
- Highlights the details and the best practices of using Vector Search.
- Explains that Vector search uses Azure Cognitive Search as a vector database to store generated word embeddings for text, images, and videos.
07:16 🌐 Multilingual Search Capability with Vector Search
- Talks about the ability to perform searches using Vector Search regardless of language, thanks to the vector representation of the context.
- Briefly mentions the use of vector search to support semantic search and word embedding based search.
08:54 💾 Creating an Index for Word Embeddings in Azure Cognitive Search
- Demonstrates a Python code showing how you can create an index for word embeddings using Azure Cognitive Search.
- Specifies required credentials and configurations to connect to Azure Open AI and Azure Cognitive search services.
14:25 🗣️ Conversation with Data using Lang Chain and Cognitive Search
- Uses LangChain for managing the conversation with the data, using Cognitive Search for retrieving the information based on word embeddings.
- Shows how questions are answered based on the closest word embeddings.
16:34 🔍 Digging into the Backend of Cognitive Search
- Looks into the backend of Cognitive Search, showing how word embeddings and data are stored and indexed in Azure Cognitive Search.
- Asserts the value and efficiency of using Vector search in the retrieval process.
18:48 🌟 Highlighting the Value of Vector Search
- Emphasizes the value of Vector search and its advantages in making retrieval of information simpler and more efficient.
- Encourages followers to use the hybrid approach leveraging semantic search and word embeddings based search.
21:06 💡 Final thoughts and Philosophical ending
- Discusses the concept of imperfection and the power of forgiveness.
- Encourages viewers to dream big, believe in themselves, and take action.
Made with HARPA AI
Excellent cog search tutorial as usual, but honestly I needed to hear the last minute and half of this video more than anything else. Thank you.
One of the best content, and the end spiritual teaching was awesome too. Thanks 🙏
Was wondering what the azure equivalent was to vector databases like pinecone , thank you so much MG 🎉 appreciate your hard work and dedication 🥳
Redis Cache
Excellent tutorial.Please make One video for Azure open AI and azure AI with practical explanation .Also need tutorial work on google cloud for Generative AI
Thank you MG, for the clear explanation.
keep up the good work! great tutorials!
Excellent Video.
Say, if our document repository grow in size on daily basis. Creating the embedding of the newly added documents is not a problem but the time taken to index the whole vectors again is time consuming. How can we reduce the time of indexing?
Interesting MG. Thanks 👍
how to add custom metadata to azure ai search
Hello I have a doubt if I am using RBAC in my azure AI search how can I Create a connection? below code we have to pass Key but if I am using RBAC how can I create Connection
acs = AzureSearch(
azure_search_endpoint=endpoint,
azure_search_key=azureaikey,
index_name=index_name,
embedding_function=embeddings.embed_query,
)
What about data storage? Where is original data and data embeddings stored?
I found where the documents are stored, which are stored in a blob container, but I cannot figure out where the actual embeddings are stored. Any luck? This UI sucks!
Can you point me where is the code you are using?
Thank you so much for this wonderful video. Do we have any trade off for hybrid search? Let’s say we have 1000 of pdfs ingested in azure congnitive index and also use same index for embedding vectors and then we do a hybrid search, so seach will take more time in hybrid search compare to only vector or semantic seach ?
Thank youuu
How and where do i push this code to azure and make it run remotely?
Does small chunk size matters ? What happen if my question is between 2 chunk ?
do you have this done in nodejs by any chance?
I am getting errror in Azure Search while connecting with cognitive search,it says resource not found,although i am grabbing correct resource as cognitive search url and key.i have models and cognitive search in same region,East US
Hi
I am getting the same error. Were you able to resolve the issue?
I am facing the same issue as well
@@sayan3023 facing the same, if you are able to solve please do share
I will be sharing a code base shortly.i have resolved the issue.
Even i got same resource not found error, its due to OpenAIEmbeddings.
embeddings = OpenAIEmbeddings(model="model name",
deployment="deployment name",
openai_api_key = OPENAI_API_KEY,
openai_api_base = OPENAI_API_BASE,
openai_api_version = OPENAI_API_VERSION,
chunk_size=1,
openai_api_type="azure",)
use this for embeddings it should work
This intro 😂😂😂😂
Really Interesting MG, Thank you, Your accent is transforming from persian English into Indian English, maybe because your colleagues in MS are mostly Indian :) kidding boro :D.
I have a similar question @shamaldesilva9533 as to if it is similar to pinecone and also, as a first time viewer of yours, I wanted to say the spiritual message at the end was unexpected but AWESOME!