Wow! This is amazing work you did here. Your work is very clear, the process well well explained 👏. I was looking into setting up vector embeding search with Elastic and you've clarified the whole process and fundamental understanding of such a solution.
This was a great video! Thank you so much. I enjoyed the format (very much to the point, with a nice clarification at the beginning) and you presented it really well. I would love to see a video in the future that maybe explains how picking a custom model could help being charged a lot by ES's ML nodes. In other words, how do you think we could use custom models to be able to use smaller ES ML nodes (e.g. 1GB or 2GB, vs the 4GB ESRE) and be charged less? Thanks again
Glad it was helpful. I'll need to look into the smaller node usgae stuff. Never used it before. Will try to make video about it future for sure. Thanks for such detailed comment. Really Appreciate it!
I have watched the video, need advice from you on a similar use case. So I have API log data and for that I want to create a similar search application. Is it possible for the search application to answer questions like the ones below? - "List all failed transactions between Merchant_B and any bank.", "Find transactions between Merchant_A and Bank_X." etc.
We just need to change the similarity parameter in indexMapping file. I have used l2_norm here. Just change it to "cosine". Here is reference link: www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html#dense-vector-similarity:~:text=dimensions%20per%20vector.-,cosine,in%20this%0Acase.,-max_inner_product
Thank you for sharing, it really was a pleasure to follow that tutorial. I just had a video recommendation about the latest improvements of ElasticSearch and now I am wondering if that video is still relevant as of today considering the improvements that have been made (ELSER) ?
Glad you find the video helpful Valentin. Thanks for the suggestion. I'll surely look at ELSER. Today I'll be posting a video where I have used OpenAI Embedding in Elasticsearch 8.11 as they have updated the vector dimension limit to 4096.
@@AbidSaudagar Hey Abid, thanks for coming back to me. Regarding ELSER, it requires a ml node so one has to have a licence that most likely costs thousands of dollars so I will stick to vector embedding. Bell activated, waiting for the video!
Hello abid Is it possible to convert a specific field in elasticsearch that already contains data into vector and then use semantic search in it. Great Content
great tutorial, wanted to ask if elastic search is the best option for basic l2 or cosine vector similarity insted of some online or offline vector db based methods as it uses approximate nearest neighbours ( ANN ), also this approach cannot be used in cases where false positives are not acceptable such as cache implementation or medical data semantics, can you suggest something that resolves these problem statements 😢
You can make all field you wanna search for as dense_vector field, then search on all of them separately and then combine the results manually. This is the simplest way.
you didnot provide the repository link to work with, or the dataset path . please share the repository in git. it would be quite helpful coz i am currently working on it
hi @abid, thanks for the video, can you help me in combining multiple column data for vector search instead of just one as somtimes the description does not contain everything and the customer may look for things from other column. i hope, you got it. please help.
This can be done with multiple approaches. 1. We can aggregate the text data from multiple field into one and then create the vector. 2. If for some reason you do not want to aggregate the data in one field. We can use LTR plugin: chat.openai.com/share/01ef9614-af6e-47e4-be1d-c0697eee6bd7
Appreciate your effort Abid, but when i run the search query code at 28:42 as shown in the video it shows this error : "BadRequestError(400, 'search_phase_execution_exception', 'failed to create query: [knn] queries are only supported on [dense_vector] fields')" I'm using the same dataset you're using with the same mappings, Could you please assist, thank you in advance!
@@AbidSaudagar Appreciate your response Abid, i managed to solve the snippet error and prepared the index properly but however the code runs without any trouble in the vs code and the Ai model is working but when i launch it using streamlit it pops up the same error regarding the [dense_vector fields] and i've made sure the field of the query is a type of dense vector,.
You can checkout New Advanced Semantic Search Video by using GPT 3.5 Turbo and ElasticSearch: ruclips.net/video/frnKiugyFRI/видео.html
I appreciate your will and dedication to rerecord the video second time
Thank you for sharing! Very clear, thorough, well-paced and learner centered. What an amazing educator!
Thank you!! Glad you found it valuable!
I think you should consistently upload videos .
More video like this very helpful ✅
❤❤❤🙌🏻
That's very nice .. glad I searched and got your video
Incredible video, you are amazing! Thank you for the concise outline of the video and explanations of all the key concepts. You have a new sub :)
Wow! This is amazing work you did here. Your work is very clear, the process well well explained 👏. I was looking into setting up vector embeding search with Elastic and you've clarified the whole process and fundamental understanding of such a solution.
Thank You. I'm glad it was helpful!
Very good tutorial
Like a semantics search ,as chatting with your Json data privately?. I love ur tutorials
Nicely done. Thanks for the walkthrough
This was a great video! Thank you so much. I enjoyed the format (very much to the point, with a nice clarification at the beginning) and you presented it really well. I would love to see a video in the future that maybe explains how picking a custom model could help being charged a lot by ES's ML nodes. In other words, how do you think we could use custom models to be able to use smaller ES ML nodes (e.g. 1GB or 2GB, vs the 4GB ESRE) and be charged less? Thanks again
Glad it was helpful.
I'll need to look into the smaller node usgae stuff. Never used it before. Will try to make video about it future for sure.
Thanks for such detailed comment. Really Appreciate it!
GREAT ❤ WELCOME BACK ... BURHANIS...
Hey man great video. it would be amazing if you could teach us how to do the same thing using Opensearch and docker
How to run this as different services for es and streamlit via docker compose? Also if I need to use any other vectordb is there any way to do it?
Great Video 👏Nice to see you back
Thanks Jasleen😄
I have watched the video, need advice from you on a similar use case. So I have API log data and for that I want to create a similar search application. Is it possible for the search application to answer questions like the ones below? -
"List all failed transactions between Merchant_B and any bank.",
"Find transactions between Merchant_A and Bank_X." etc.
Superb 😊
Great video brother👍👌👌
Thanks 👍
Any chance you would show us how to implement this but using cosine similarity? :), Amazing video, helped me learn a lot!!
We just need to change the similarity parameter in indexMapping file.
I have used l2_norm here. Just change it to "cosine".
Here is reference link: www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html#dense-vector-similarity:~:text=dimensions%20per%20vector.-,cosine,in%20this%0Acase.,-max_inner_product
Great content
At final part of code i have this error BadRequestError(400, 'illegal_argument_exception', 'Invalid type: expecting [_doc] but got [_knn_search]')
same error in my side even i am using 7.17.17 version of es which support knn_search
Nice to see you sir
Thank you Parvez ✌️
Thank you for sharing, it really was a pleasure to follow that tutorial. I just had a video recommendation about the latest improvements of ElasticSearch and now I am wondering if that video is still relevant as of today considering the improvements that have been made (ELSER) ?
Glad you find the video helpful Valentin.
Thanks for the suggestion. I'll surely look at ELSER.
Today I'll be posting a video where I have used OpenAI Embedding in Elasticsearch 8.11 as they have updated the vector dimension limit to 4096.
@@AbidSaudagar Hey Abid, thanks for coming back to me. Regarding ELSER, it requires a ml node so one has to have a licence that most likely costs thousands of dollars so I will stick to vector embedding.
Bell activated, waiting for the video!
Hello abid Is it possible to convert a specific field in elasticsearch that already contains data into vector and then use semantic search in it. Great Content
Phenomenal tutorial! Can you up the audio? It's low.
Thanks for kind words and also feedback regarding audio. I have increased audio levels in later videos.
great video
Glad you found it helpful.
Hi, thanks for the tutorial. Could you also make one to compare elasticSearch with other vector database solution in terms of semantic/vector search?
Great suggestion! I'll surely put it in my next videos list.
great tutorial, wanted to ask if elastic search is the best option for basic l2 or cosine vector similarity insted of some online or offline vector db based methods as it uses approximate nearest neighbours ( ANN ), also this approach cannot be used in cases where false positives are not acceptable such as cache implementation or medical data semantics, can you suggest something that resolves these problem statements 😢
awesome
great work amigo
i just had one question please :
what should i do in order to search by all the fileds (price , brand , gender ...ect ) ???
You can make all field you wanna search for as dense_vector field, then search on all of them separately and then combine the results manually. This is the simplest way.
ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': False, 'index': 'series120'}) I am getting this error
How can I solve this?
thanks bro!
great video. I was also wondering what vs code plugin you used that color coded your python indentations?
Thank You Austin.
Plugin name is indent-rainbow.
Coding a privateGPT using LANGCHAIN, HuggingFace Embeddings and FREE LLM
ruclips.net/video/kUxfr2i2zn8/видео.html
you didnot provide the repository link to work with, or the dataset path . please share the repository in git. it would be quite helpful coz i am currently working on it
Sorry I forgot to add the link. Here is the link.
github.com/abidsaudagar/semantic-search-elastic-search-and-BERT-vector-embedding
Totally loved it! Code access link for referral please?
github.com/abidsaudagar/semantic-search-elastic-search-and-BERT-vector-embedding
hi @abid, thanks for the video, can you help me in combining multiple column data for vector search instead of just one as somtimes the description does not contain everything and the customer may look for things from other column. i hope, you got it. please help.
This can be done with multiple approaches.
1. We can aggregate the text data from multiple field into one and then create the vector.
2. If for some reason you do not want to aggregate the data in one field. We can use LTR plugin: chat.openai.com/share/01ef9614-af6e-47e4-be1d-c0697eee6bd7
👏
where is the Kaggle file link in description ??
github.com/abidsaudagar/semantic-search-elastic-search-and-BERT-vector-embedding
You can download from my Github.
Sir how to run multiple nodes
You can use docker to easily run multiple containers of ElasticSearch.
Without docker is also possible.
I'm gonna make video about it soon.
@@AbidSaudagar thank you sir ....plz upload that video
Appreciate your effort Abid, but when i run the search query code at 28:42 as shown in the video it shows this error :
"BadRequestError(400, 'search_phase_execution_exception', 'failed to create query: [knn] queries are only supported on [dense_vector] fields')"
I'm using the same dataset you're using with the same mappings,
Could you please assist, thank you in advance!
Can you confirm that the field you are searching on, has field type as dense vector.
@@AbidSaudagar Appreciate your response Abid, i managed to solve the snippet error and prepared the index properly but however the code runs without any trouble in the vs code and the Ai model is working but when i launch it using streamlit it pops up the same error regarding the [dense_vector fields] and i've made sure the field of the query is a type of dense vector,.