*Correction to the skip-list logic, if we hit the end block or overshoot the target, we move down a layer to the previous value directly (not the start), they'll be an updated visual in the article **www.pinecone.io/learn/hnsw/** as soon as possible*
I was trying to read and understand the paper as I have to use HNSW KNN in one of my project. Demystifying topics through videos help me understand the paper more easily. Thanks for this video!
Good video overall, but at 3:07 you don't go to the start block again (that would yield O(N) average search time complexity). You instead go down the last column that wasn't higher than the search key (which in this case is the 5 column), and this gives you O(log(N)) average search time complexity.
Looking at the C++ code in the video, the defaults are set to efSearch=16 and efConstruction=40 which is in line with what is shown in the graphs. M between 32 and 50 seems to be the sweet spot to maximize recall, minimize search time and have a memory usage below 1GB. Maybe efSearch can be pushed to 50-60 to increase recall with a small hit on search time. Thanks for the great video.
Sir, I have a question. Why nodes in the top layer has higher degree? "During the search, we enter the top layer, where we find the longest links. These vertices will tend to be higher-degree vertices (with links separated across multiple layers), meaning that we, by default, start in the zoom-in phase described for NSW." I found this in the doc, don't understand why
“if you need it to be faster,, you can use an ivf index on top of HNSW to improve search time”， I don't understand this statement. Any article that can refer to ?
is it not the case that when we move from layer 3 to layer 2, we do not need to go to the very beginning, it goes to the node 5 in layer 2. and from node 19, when it jumps to layer 1, it should go to node 11 instead of again going back to the start node. otherwise, it search efficiency again goes back the O(n)
I would like to draw contour lines based on the degree of anomaly when representing the plot on a two-dimensional graph. If there is a method, please instruct me.
If I understood correctly, the size of the graph does not depend on the number of dimensions, i.e. 128-dimensional vectors would result in the same space cost as 1024-dimensional vectors. For reasonable M, this space should probably be smaller than the space used by the vectors themselves. However, I saw threads (unfortunately, I didn't keep the links) where they complain that using HNSW results in using several times more space than the space used by the original vectors and hence requires a lot of RAM and disk space. How could that be explained?
Can this method be used for searching non-vector data in a non-metric space? I'm thinking of variable length timeseries with dynamic time warping as a distance measure.
Hi Faiss is like a vector index without data management, infrastructure, optimisation etc - Pinecone is a vector index with data management, managed infrastructure, custom (and very good) search algorithms, metadata filtering, etc :)
With BM25 you are doing sparse search and so would not use faiss, if you wanted to use faiss you need to do dense search and that requires sentence transformer models
@@jamesbriggs Thank you for your answer, do you have any video where you explain how to do hybrid indexing? using BM25 for ranking, index BM25 then using Cross Encoder for the result?
@@Data_scientist_t3rmi this wouldn't be hybrid indexing, it would be sparse search followed by reranking -- hybrid inxexing would require something like sentence transformer with Faiss and BM25 as two separate indexes, then you could rerank the results based on some "merge" of the results at the end My most recent video talks about hybrid search in Pinecone: ruclips.net/video/0cKtkaR883c/видео.html Other than this I don't have anything else but I will focus on hybrid methods and reranking over the next few weeks, so hopefully that will help :)
@@jamesbriggs I have one more question, if i have 8 books for example and i want to conduct research, how can i do it ? do you have article or key words to search that type of search ? i konw when we have one text file or 8 text file we simply concatinate these files then transform them into embedding matrix then conduct a neural search, however, when we have 5000 documents, i do not think so that it would be the same. for now i built a dictionary per document, in wich you can find , document name, page number,, text and embedding, but i found it quit difficult to conduct my search on lists of dictionary rather then on simple text. the idea is when we find the document id, we can say that this id belongs to that odcument wich is belong to that book.
You can use ‘add_with_ids’ rather than ‘add’ when adding vectors to the index, and you pass both ids and vectors, although it is a lot of effort to get it working when you’re adding, deleting, overwriting etc - so be prepared 😅
@@wilfredomartel7781 if you're using Faiss you need to build a framework around it to handle addition/deletion/overwrites, definitely use add_with_ids rather than add because that will be useful later on when needing to overwrite vectors, however add_with_ids doesn't allow any alphanumeric combo for IDs, I don't remember what it requires but in the past I resorted to something like '0', '1', '2', etc and then mapping this 'Faiss IDs' to the IDs in a database with an external mapping file. If you want to get something good up and running quick, use Pinecone www.pinecone.io - it's also free up to 1M vectors
Thanks for the presentation. Totally new subject for me. Can you direct me to the docs? Also, "You can use 'add with ids' rather than 'add ' when adding vectors to the index, and you pass both ids and vectors, although it is a lot of effort to get it working when you're adding, deleting, overwriting etc - so be prepared " If you have a Self fulfilling ID that would land on a default layer for all vectors and then be archived within it's own file like ID.sqlite you could then recall the data in a decentralized way and with 0 latency. These are my thoughts. Appriciate your feedback.
*Correction to the skip-list logic, if we hit the end block or overshoot the target, we move down a layer to the previous value directly (not the start), they'll be an updated visual in the article **www.pinecone.io/learn/hnsw/** as soon as possible*
Yes I was about to point out the same!
Men, I have been sleeping on this channel, your stuff is awesome!
Tried to read the paper, was super perplexed, this was awesome!!!
I was trying to read and understand the paper as I have to use HNSW KNN in one of my project. Demystifying topics through videos help me understand the paper more easily. Thanks for this video!
21:19 Why in the lowest level there aren't 1.000.000 values since the dataset is 1 million? What happened to the rest?
Good video overall, but at 3:07 you don't go to the start block again (that would yield O(N) average search time complexity). You instead go down the last column that wasn't higher than the search key (which in this case is the 5 column), and this gives you O(log(N)) average search time complexity.
lol, came here for the exact comment , stopped the video at 3.09
Looking at the C++ code in the video, the defaults are set to efSearch=16 and efConstruction=40 which is in line with what is shown in the graphs. M between 32 and 50 seems to be the sweet spot to maximize recall, minimize search time and have a memory usage below 1GB. Maybe efSearch can be pushed to 50-60 to increase recall with a small hit on search time. Thanks for the great video.
So it's basically skip list but in vector space?
Sir, I have a question. Why nodes in the top layer has higher degree? "During the search, we enter the top layer, where we find the longest links. These vertices will tend to be higher-degree vertices (with links separated across multiple layers), meaning that we, by default, start in the zoom-in phase described for NSW." I found this in the doc, don't understand why
Is it possible to identify which combination of elements caused the outlier? I am using the HNSW algorithm on Github.
“if you need it to be faster,, you can use an ivf index on top of HNSW to improve search time”， I don't understand this statement. Any article that can refer to ?
thank you yet again !!!!
is it not the case that when we move from layer 3 to layer 2, we do not need to go to the very beginning, it goes to the node 5 in layer 2. and from node 19, when it jumps to layer 1, it should go to node 11 instead of again going back to the start node. otherwise, it search efficiency again goes back the O(n)
I would like to draw contour lines based on the degree of anomaly when representing the plot on a two-dimensional graph. If there is a method, please instruct me.
If I understood correctly, the size of the graph does not depend on the number of dimensions, i.e. 128-dimensional vectors would result in the same space cost as 1024-dimensional vectors. For reasonable M, this space should probably be smaller than the space used by the vectors themselves. However, I saw threads (unfortunately, I didn't keep the links) where they complain that using HNSW results in using several times more space than the space used by the original vectors and hence requires a lot of RAM and disk space. How could that be explained?
Um, how to build the index. Is there any explanation step by step ?
what is the python editor you use
VS Code with Python and Jupyter plugins + the Night Owl theme :)
@@jamesbriggs thanks !
Great Video. Thank you.
Can this method be used for searching non-vector data in a non-metric space? I'm thinking of variable length timeseries with dynamic time warping as a distance measure.
I don't believe so, but could be wrong
@@jamesbriggs Thanks for your reply. Btw are you aware of any other search method that can be used in this scenario?
Hello, what is faiss to pinecone? is pinecone a cloud indexing method that can be based on faiss?
Hi Faiss is like a vector index without data management, infrastructure, optimisation etc - Pinecone is a vector index with data management, managed infrastructure, custom (and very good) search algorithms, metadata filtering, etc :)
@@jamesbriggs Thank you James
I have a question, if i have BM25 + Cross Encoder architecture for information retrieval, how can i do my indexing ?
With BM25 you are doing sparse search and so would not use faiss, if you wanted to use faiss you need to do dense search and that requires sentence transformer models
@@jamesbriggs Thank you for your answer, do you have any video where you explain how to do hybrid indexing? using BM25 for ranking, index BM25 then using Cross Encoder for the result?
@@Data_scientist_t3rmi this wouldn't be hybrid indexing, it would be sparse search followed by reranking -- hybrid inxexing would require something like sentence transformer with Faiss and BM25 as two separate indexes, then you could rerank the results based on some "merge" of the results at the end
My most recent video talks about hybrid search in Pinecone:
ruclips.net/video/0cKtkaR883c/видео.html
Other than this I don't have anything else but I will focus on hybrid methods and reranking over the next few weeks, so hopefully that will help :)
@@jamesbriggs thank you
@@jamesbriggs I have one more question, if i have 8 books for example and i want to conduct research, how can i do it ? do you have article or key words to search that type of search ? i konw when we have one text file or 8 text file we simply concatinate these files then transform them into embedding matrix then conduct a neural search, however, when we have 5000 documents, i do not think so that it would be the same. for now i built a dictionary per document, in wich you can find , document name, page number,, text and embedding, but i found it quit difficult to conduct my search on lists of dictionary rather then on simple text. the idea is when we find the document id, we can say that this id belongs to that odcument wich is belong to that book.
Excellent thanks !
this is brilliant!
Please teach me omost simplest knn c# code with HNSW if you like.
Any idea of how to add ids to the vector for a posterior integration with database? Excellent explanation!
You can use ‘add_with_ids’ rather than ‘add’ when adding vectors to the index, and you pass both ids and vectors, although it is a lot of effort to get it working when you’re adding, deleting, overwriting etc - so be prepared 😅
@@jamesbriggs so what do you propose ?
@@wilfredomartel7781 if you're using Faiss you need to build a framework around it to handle addition/deletion/overwrites, definitely use add_with_ids rather than add because that will be useful later on when needing to overwrite vectors, however add_with_ids doesn't allow any alphanumeric combo for IDs, I don't remember what it requires but in the past I resorted to something like '0', '1', '2', etc and then mapping this 'Faiss IDs' to the IDs in a database with an external mapping file.
If you want to get something good up and running quick, use Pinecone www.pinecone.io - it's also free up to 1M vectors
@@jamesbriggs thank you so much for your tipos , i really appreciate them.
Thanks for the presentation. Totally new subject for me. Can you direct me to the docs? Also, "You can use 'add with ids' rather
than 'add ' when adding vectors to
the index, and you pass both ids and
vectors, although it is a lot of effort
to get it working when you're adding,
deleting, overwriting etc - so be
prepared " If you have a Self fulfilling ID that would land on a default layer for all vectors and then be archived within it's own file like ID.sqlite you could then recall the data in a decentralized way and with 0 latency. These are my thoughts. Appriciate your feedback.
Requires lot of improvement on how topic is covered. Skip list and HNSW is better explained by showing running examples.
Reminds me of a skip ratchet