Pinecone vs FAISS vs pgvector + OpenAI Embeddings
HTML-код
- Опубликовано: 19 июл 2023
- Comparing 3 vector databases - Pinecone, FAISS and pgvector in combination with OpenAI Embeddings for the semantic search.
Please find the corresponding Google Colabs:
- Pinecone - colab.research.google.com/dri...
- FAISS - colab.research.google.com/dri...
- pgvector - colab.research.google.com/dri...
or in github.com/IuriiD/pinecone-fa...
MongoDB and Qdrant would be great additions to this test :) Great video! Thanks
Thanks for a very concise and informative video!
Very useful. Thank you.
Great video thanks!
Very helpful
Yes I was specifically thinking about chatbot response speed
Quite useful and good comparison. It would have been easier to see using English sentences though. Well done!
Thanks, @christofferbjorkwall1441, and agree. I just had some local telegram chatbot communities in mind when working on this comparison, thus wanted in parallel to test embeddings on Ukrainian
Thank you for the great explanation. How does MongoDB fare in this regard?
You probably mean MongoDB Atlas Vector Search 🤔Sorry, but I didn't have a chance to try this one yet. Interesting to know, thank you for the hint!
When using a hosted pgvector postgre service, would the cost be more like 2.5 stars?
Hi. If by a "hosted Postgresql service" we mean e.g. AWS RDS for Postgresql and choose an instance like db.t3.small, and pay monthly ($0.036/h), it will be ~$26/month (aws.amazon.com/rds/postgresql/pricing/) which is 2.8 times less than the minimum paid plan on Pinecone ("Standard", $70/m)
Добрий день! Дякую за дуже корисне та своєчасне відео. Маю питання, чи доцільно зберігати ембедінги в pgvector, а для similarity пошуку використовувати faiss?
Пан гурман 😎Чесно кажучи, я не думав про такий варіант і маю сумніви на рахунок того, чи це технічно можливо. А навіщо так?
@@defaultfallback Суть в тому що faiss треба постійно переіндексувати коли зʼявляться нові ембедінги, а в pgvector їх можна просто поступово додавати плюс зберігати усю додаткову інформацію в postgress. Але коли доходить до search similarity, faiss наче це робить швидше та точніше.
Поправлю свою першу відповідь - так, Postgresql цілком підходить як постійне сховище для векторів (і метаданих якщо треба), звідки вони можуть загружатись в оперативну пам*ять для пошуку за допомогою faiss
@@defaultfallback sir could you recommend vector db for images?
🎯 Key Takeaways for quick navigation:
00:00 🌐 *Overview of Semantic Search with OpenAI Embeddings*
- Introduction to three setups for semantic search using OpenAI embeddings: Pinecone, FAISS, and pgvector.
- Overview of the project "dry" aimed at solving repetitive questions in chatbots using semantic search.
- Explanation of the schema, involving generating vectors, searching the vector database, and storing metadata for incoming messages.
02:26 🧠 *Understanding Semantic Search and Embeddings*
- Definition of embeddings models, with a focus on OpenAI's text embeddings ada002.
- Explanation of how embeddings models generate dense vectors representing words, sentences, or paragraphs.
- Clarification of how the multi-dimensional space in the model reflects the semantic meaning of different strings.
05:24 🍍 *Pinecone Integration*
- Introduction to Pinecone as a fully managed Vector database.
- Demonstration of setting up a project, specifying environment variables, and creating a Pinecone index.
- Implementation of Pinecone integration in a Google Colab notebook, including generating vectors, saving them, and performing similarity searches.
16:24 💲 *Evaluation of Pinecone*
- Efficiency assessment of Pinecone as a managed service for vector storage.
- Low effort needed for integration with Pinecone due to available Python and TypeScript examples.
- Consideration of costs, with the expectation of a $70/month plan being sufficient for a pet project but potentially expensive for larger-scale commercial use.
18:23 📚 *FAISS Integration*
- Introduction to FAISS as a library for efficient similarity search of dense vectors.
- Noting nuances of FAISS, including the need to manage large indexes and separate metadata storage.
- Implementation of FAISS integration in a Google Colab notebook, demonstrating vector generation, saving, and similarity searches.
28:23 💲 *Evaluation of FAISS*
- Recognition of FAISS's efficiency in similarity search with proper usage and fine-tuning.
- Intermediate effort level due to additional considerations for metadata storage and potential index management.
- Acknowledgment of potentially lower costs as an open-source package but with considerations for high RAM requirements.
29:18 🐘 *pgvector Integration*
- Introduction to pgvector as an extension for PostgreSQL, allowing searching by vectors.
- Setting environment variables and connecting to a PostgreSQL database using Superbase.
- Creation of a table with columns for vector ID, text, embeddings, and metadata, including dimensionality.
34:00 💲 *Evaluation of pgvector*
- Recognition of pgvector's efficiency with potential for fine-tuning in usage.
- Intermediate effort level, considering the need for an additional database for metadata storage.
- Acknowledgment of potential costs, including high RAM requirements and potential additional expenses for hosting metadata.
34:15 📊 *pgvector Database Operations*
- Introduction to functions for counting rows and deleting rows by ID in pgvector.
- Demonstration of adding test messages to the table, generating embeddings, and saving vectors with metadata.
- Usage of functions to search for top relevant messages based on cosine distance and manual filtering.
35:58 🧾 *Testing Similarity Search with pgvector*
- Implementation and explanation of the get top relevant messages function in pgvector.
- Testing similarity search on three test messages with varying relevance scores.
- Display of results, showcasing the closest match with a similarity score of 0.106.
36:41 🔄 *Searching and Storing Similar Messages in pgvector*
- Introduction to the gets similar and store function, aiming to search for similar messages and store new messages.
- Execution of the function on a list of 91 messages from a Telegram chat.
- Confirmation of matches found and the addition of new messages to the pgvector database.
38:13 💼 *Evaluation of pgvector*
- Efficiency rating of pgvector with three stars, indicating its capability for the intended purpose.
- Effort assessment, noting a moderate complexity in integration, especially for users familiar with SQL queries.
- Consideration of potential costs, suggesting minimal impact if an existing PostgreSQL database is already part of the architecture.
Made with HARPA AI
Can you do a video with FAISS and Any DB (Postgres)?
Sorry, do you mean using FAISS as a search engine to query entries in Postgresql?
Otherwise, this video contains a quick comparison of FAISS vs pgvector (Postgresql addon for vector search)
5:14 Pinecone
18:24 FIASS
29:17 pgvector
Thanks a lot, @migunovich
🎯 Key Takeaways for quick navigation:
00:00 📋 *Introduction to Semantic Search and Comparison of Vector Stores*
- Introduction to the video's topic: Semantic search and comparison of vector stores.
- Explanation of the project "dry" aimed at creating a chatbot for repetitive questions.
- Overview of how semantic search can solve the problem of repetitive queries.
02:00 🤖 *Understanding Semantic Search and Embeddings Models*
- Explanation of semantic search and embeddings models.
- Description of the OpenAI embeddings model (text embeddings ada002).
- How embeddings models generate dense vectors for text strings.
05:24 🌲 *Pinecone: Fully Managed Vector Database*
- Introduction to Pinecone as a fully managed vector database.
- Setting up a Pinecone project and environment variables.
- Demonstrating the integration with Pinecone for semantic search.
09:18 🌟 *Pinecone Efficiency, Effort, and Cost Evaluation*
- Evaluation of Pinecone's efficiency for semantic search.
- Discussion of the low effort required for integration with Pinecone.
- Consideration of Pinecone's cost, especially for commercial projects.
20:06 🧊 *Facebook Faiss: Efficient Similarity Search*
- Introduction to Facebook Faiss as an efficient similarity search library.
- Setting up environment variables for Faiss and pidgey vector.
- Integration of Faiss with the ability to store metadata.
26:30 ⚙️ *Faiss Efficiency, Effort, and Cost Evaluation*
- Evaluation of Faiss's efficiency for semantic search.
- Discussion of the initial integration effort and the need for metadata storage.
- Consideration of costs associated with Faiss, including RAM requirements.
29:32 🐘 *PGVector: PostgreSQL Extension for Vector Search*
- Introduction to PGVector as a PostgreSQL extension for vector search.
- Overview of using PGVector to search by vectors in PostgreSQL.
- Setting up PostgreSQL for PGVector integration.
29:44 🧩 *Setting Up Environment Variables and Database Connection*
- Setting environment variables for API keys, connection string, similarity limit, and top K records.
- Explanation of options to obtain a PostgreSQL database.
- Introduction to using Superbase as an online service for PostgreSQL.
31:16 🧬 *Integration with OpenAI API and PostgreSQL Database*
- Installation of the OpenAI package and confirmation of interaction with the OpenAI API.
- Establishing a connection to the PostgreSQL database and creating a test table.
- Installing the PGVector extension for PostgreSQL.
34:28 🛠️ *Generating Embeddings, Storing Data, and Testing Similarity Search*
- Explanation of functions for generating embeddings, creating vector IDs, and storing data in the table.
- Testing similarity search using cosine distance and filtering results.
- Applying the "get similar and store" function to a list of test messages.
38:13 💼 *PGVector Efficiency, Effort, and Cost Evaluation*
- Evaluation of PGVector's efficiency in the context of the project.
- Discussion of the complexity of integration compared to Pinecone.
- Consideration of cost implications, including using existing PostgreSQL databases and managed services.
Made with HARPA AI
Which is fastest?
Hi, @PizzaLord. Sorry, but speed comparison wasn't in the scope of this (mini) project. The main question I tried to answer was: "Is it possible to replace Pinecone (which is a good and convenient but paid and "external" service) with something that is cheaper and can be fully in-house (e.g. pgVector)?" And the answer seems to be positive (at least based on such a limited comparison). I'm not sure about the speed - on small amounts of data I'd assume that the difference between the 3 may be negligible. On big volumes of vectors FAISS possibly could be the fastest (at least I saw such statements). Though if a vector DB is to be used as a part of some chatbot backend, where the bot is not expected to respond immediately, then even big differences (e.g. in 1 second) may still be acceptable IMHO.
I am using FAISS and it runs perfectly. It is very accurate and blazing fast. The only drawback is the databse, which has to be generated in advance, if adding new documents, it has to be created again (for my usecase no problem). I would recommend Faiss
@@GeigenAkademieThanks for sharing!
What's your use case - you don't need to update the vectors?
@@GeigenAkademie I tried it for the vector data for a chatbot and it was really slow. why is that? using a .json file was way faster
@@PizzaLord My data is a 15MB pdf. The Faiss lookup in the database takes around 15ms for multiple results including score (however generating the vectorstore was around 45min) on a Laptop i5, 16GB memory. I'm using a local GPT4all embedding
I wish you did not use langchain!
why
@@bbrother92 a 1000 pounds heavy framework that hides all the details from the user. I hate that framework.
@@0xeb- could you recommend something for image search, and labeling?
Nice one mate. Just wish you did not use the russian text. As I do not understand russian, the results do not make any sense to me. But I am just believing... rather than knowing and seeing with my own eyes.
Hi, @Chuukwudi . Sorry for that. The reason why I used a non-English language because in parallel I wanted to check how good/bad the Embeddings model and similarity search will work with Ukrainian (that's not Russian, though also uses Cyrillic letters) as I might need to use pgvector instead of Pinecone for one local project (whose audience mainly uses this language).
@@defaultfallback
Thank you very much. Sorry for the mistake, did not realise it was Ukrainian until you said it in the video after I had already commented. After watching your videos, I finally decided to use pgvectors as well. I am doing something similar and will find out for myself. Thank you very much once again.
@@defaultfallback greeting from Kazakhstan! I just wanted to say that It was very useful for me!
Kazakh language is also written in cyrillic and your content was perfect for me