Thanks for the great work demystifying pgvector, I was very worry as I had to work on a new project with pgvector but now dependencies seems quite clear as the business scenarios you exposed. Thanks!
00:01 Learn how to use PG Vector extension in practice 02:17 Using PGVector extension with Docker and loading sample dataset 04:44 Generating movie embeddings and connecting to PostgreSQL instance for analysis. 07:20 Using pgvector for similarity search and movie recommendations 09:38 Data pre-filtering is essential for optimizing vector similarity search in PostgreSQL. 11:52 PostgreSQL pgvector supports various index types for vectorized data and performs similarity search. 14:18 Introducing Yab DB as a distributed SQL database for horizontal scaling and high availability. 16:36 Running PostgreSQL pgvector for Python developers on a local laptop Crafted by Merlin AI.00:01 Learn how to use PG Vector extension in practice 02:17 Using PGVector extension with Docker and loading sample dataset 04:44 Generating movie embeddings and connecting to PostgreSQL instance for analysis. 07:20 Using pgvector for similarity search and movie recommendations 09:38 Data pre-filtering is essential for optimizing vector similarity search in PostgreSQL. 11:52 PostgreSQL pgvector supports various index types for vectorized data and performs similarity search. 14:18 Introducing Yab DB as a distributed SQL database for horizontal scaling and high availability. 16:36 Running PostgreSQL pgvector for Python developers on a local laptop Crafted by Merlin AI.
quite interesting topic! it would be interesting to see dynamic query rewriter for SQL queries + PgVector search, specifically for RAG applications. The downside of using traditional filtering/search seems to be automatic query construction that is aware of the existing database options to filter on. To scale to different types of databases (different clients) we might need to customize our code to match domain specific knowledge which ideally we wouldn't need to since this is labor expensive. Automatic schema creation for optimal querying seems to be a topic that needs more research .
Use the following command to add a column of the vector type to your existing table: ALTER TABLE my_table ADD COLUMN my_new_vector_column vector(1536); where 1536 is the dimension of the vectors generated by the OpenAI text-ada-2 model. You can change the dimension to the value supported by your model. Next, to generate embedding and store them back into the `my_new_vector_column` column: 1. Suppose you want to generate embedding for the `description` column of the text type. Read all the `description` for all the rows. 2. For every description generate an embedding using your model 3. Use the UPDATE statement to write the generated embedding back into the `my_new_vector_column` column
Im getting this error while running the cell where we are establishing a connnection with psycopg. OperationalError: connection to server at "localhost" (::1), port 5432 failed: FATAL: password authentication failed for user "postgres" - i havent changed the password, db or host in the code. Any idea why this is happening or how to fix this? Sorry Im a beginner and doesnt have much ideas to troubleshoot.
Could you confirm the database container started successfully in Docker? Check the preceding "Start PostgreSQL With pgvector" section that starts the database using a docker compose command and then tries to connect to it.
@@DevMastersDb i searched a bit online had tried docker ps in the terminal, which shows that the container is running, its also showing as running in docker desktop, then i thought maybe its because i have pgadmin installed on my device that this happens and tried changing the port to 5433 in dockerfile, but still the error persisted.
@@AI.ML.DataScientistWork does this command work for you? docker exec -it postgres-pgvector psql -U postgres Just want to make sure you can connect from within the Postgres container using the psql tool.
@@DevMastersDb I tried a lot of things but the issue was never resolved. Finally I uninstalled the local installation postgresql and then the docker instance connected without any hiccups both on pgadmin as well as in the code. Any Idea what could have been the reason?
@@AI.ML.DataScientistWork I have no idea, buddy, what was happening. Do you run on Windows? Might it have been something Widows specific? stackoverflow.com/questions/71984505/postgres-local-installed-instance-interfered-with-docker-instance
Thanks for the great work demystifying pgvector, I was very worry as I had to work on a new project with pgvector but now dependencies seems quite clear as the business scenarios you exposed. Thanks!
Glad you found it helpful! Have fun building gen AI apps with pgvector!
00:01 Learn how to use PG Vector extension in practice
02:17 Using PGVector extension with Docker and loading sample dataset
04:44 Generating movie embeddings and connecting to PostgreSQL instance for analysis.
07:20 Using pgvector for similarity search and movie recommendations
09:38 Data pre-filtering is essential for optimizing vector similarity search in PostgreSQL.
11:52 PostgreSQL pgvector supports various index types for vectorized data and performs similarity search.
14:18 Introducing Yab DB as a distributed SQL database for horizontal scaling and high availability.
16:36 Running PostgreSQL pgvector for Python developers on a local laptop
Crafted by Merlin AI.00:01 Learn how to use PG Vector extension in practice
02:17 Using PGVector extension with Docker and loading sample dataset
04:44 Generating movie embeddings and connecting to PostgreSQL instance for analysis.
07:20 Using pgvector for similarity search and movie recommendations
09:38 Data pre-filtering is essential for optimizing vector similarity search in PostgreSQL.
11:52 PostgreSQL pgvector supports various index types for vectorized data and performs similarity search.
14:18 Introducing Yab DB as a distributed SQL database for horizontal scaling and high availability.
16:36 Running PostgreSQL pgvector for Python developers on a local laptop
Crafted by Merlin AI.
quite interesting topic! it would be interesting to see dynamic query rewriter for SQL queries + PgVector search, specifically for RAG applications. The downside of using traditional filtering/search seems to be automatic query construction that is aware of the existing database options to filter on. To scale to different types of databases (different clients) we might need to customize our code to match domain specific knowledge which ideally we wouldn't need to since this is labor expensive. Automatic schema creation for optimal querying seems to be a topic that needs more research .
Very good.
Thanks, glad you liked it!
Nice can we run the entire demo in a CPU laptop? Please confirm
Thanks! Yes, you can run the Jupiter notebook on any machine
I am using pgvector to insert data in postgres table, how to add additional columns like your movie schema?
Use the following command to add a column of the vector type to your existing table:
ALTER TABLE my_table ADD COLUMN my_new_vector_column vector(1536);
where 1536 is the dimension of the vectors generated by the OpenAI text-ada-2 model. You can change the dimension to the value supported by your model.
Next, to generate embedding and store them back into the `my_new_vector_column` column:
1. Suppose you want to generate embedding for the `description` column of the text type. Read all the `description` for all the rows.
2. For every description generate an embedding using your model
3. Use the UPDATE statement to write the generated embedding back into the `my_new_vector_column` column
Im getting this error while running the cell where we are establishing a connnection with psycopg. OperationalError: connection to server at "localhost" (::1), port 5432 failed: FATAL: password authentication failed for user "postgres" - i havent changed the password, db or host in the code. Any idea why this is happening or how to fix this? Sorry Im a beginner and doesnt have much ideas to troubleshoot.
Could you confirm the database container started successfully in Docker? Check the preceding "Start PostgreSQL With pgvector" section that starts the database using a docker compose command and then tries to connect to it.
@@DevMastersDb i searched a bit online had tried docker ps in the terminal, which shows that the container is running, its also showing as running in docker desktop, then i thought maybe its because i have pgadmin installed on my device that this happens and tried changing the port to 5433 in dockerfile, but still the error persisted.
@@AI.ML.DataScientistWork does this command work for you?
docker exec -it postgres-pgvector psql -U postgres
Just want to make sure you can connect from within the Postgres container using the psql tool.
@@DevMastersDb I tried a lot of things but the issue was never resolved. Finally I uninstalled the local installation postgresql and then the docker instance connected without any hiccups both on pgadmin as well as in the code. Any Idea what could have been the reason?
@@AI.ML.DataScientistWork I have no idea, buddy, what was happening. Do you run on Windows? Might it have been something Widows specific? stackoverflow.com/questions/71984505/postgres-local-installed-instance-interfered-with-docker-instance