Vector Database Search - Hierarchical Navigable Small Worlds (HNSW) Explained

[@datamlistic]

Text tokenization is one of the most overlooked topics in LLMs, although it plays a key role in how they work. Take a look at the following video to see how the most popular tokenization methods work: ruclips.net/video/hL4ZnAWSyuU/видео.html

[@poppop101010]

best explanation on youtube atm

[@datamlistic]

Thanks! Glad you think so! :)

[@mehdimohsenimahani4150]

beeest

[@Andies450]

You have explained a complex topic in very simple terms. Keep up the good work.

[@datamlistic]

Thanks! Will do! :)

[@ZoinkDoink]

great explanation, massively underrated video

[@datamlistic]

Thanks! Glad you liked the explanation! :)

[@parth191079]

Very good explanation with a good use of animations!

[@datamlistic]

Thanks! Glad you think so! :D

[@himanikumar7979]

Perfect explanation, exactly what I was looking for!

[@datamlistic]

Thanks! Glad you found it helpful! :)

[@AkshayKadamIN]

Good Explanation Provided. Thank you vey much for this.

[@datamlistic]

Thanks! Glad it was helpful! :)

[@Lukas-il6xg]

In 6:37 there is an mistake because the "closest to query"-point was in Level 2 already there and not selected. Do you understand?

[@rohann1310]

Exactly!

[@emanuelgerber]

Very nicely explained! Thank you for making this video

[@datamlistic]

Glad it was helpful! :)

[@Omarsayan]

When initially building the small world we need to iteratively look for k nearest neighbors while inserting the new documents. How do we find those neighbors

[@andrefu4166]

great explanation

[@datamlistic]

Thanks! Happy to hear that you liked the explanation! :)

[@maheswaranparameswaran8532]

Hey, have a question, isnt there a risk of getting stuck in a local optima when comparing similarity between query node and db nodes in the graphs?

[@datamlistic]

Good question! Of course there's always a chance of getting stuck in a local optima because you are basically using a greedy algorithm here, and that's why you usually perform the search algorithm multiple times, so you reduce the chance of that happening.

[@lynnwilliam]

How did you go from a group of random vectors to a skip linked list structure?

[@datamlistic]

The nodes between levels represent the same vectors. Basically on the top level you have you have a sparse graph of vectors and on the lowest level you have the entire graph. Similar to a skip linked list, you move to another node in the same level if that's closer to the query, or move down if no such nodes exists. This allows you to travel higher distances since you start at a higher level. Hope this makes sense and please let me know if you need further clarification! :)

[@alicetang8009]

If the K equals to the total number of documents, will this approach also be like brute force? Because it needs to go through each linked document.

[@datamlistic]

If k equals to the number of documents, why not just simply return all documents? :)

[@saisaigraph1631]

could u please come up with nlp bascis course please. also basics of ML course please.

[@datamlistic]

First of all, thanks for becoming a member of this channel!❤️ I thought of making an introductory course for either NLP or ML (although the later is a bit saturated). Stay tuned for updates!