AWS re:Invent 2023 - Amazon DynamoDB zero-ETL integration with Amazon OpenSearch Service (DAT339)
HTML-код
- Опубликовано: 11 сен 2024
- Making the most of your data often means using multiple AWS services, including Amazon OpenSearch Service for full-text, geospatial, or vector search. Amazon DynamoDB offers many different ways to build integrations, including export functionality and Amazon DynamoDB Streams. In this session, learn about best practices by seeing how Amazon DynamoDB zero-ETL integration with Amazon OpenSearch Service works within different architecture patterns and sample implementations, including an end-to-end vector search example.
Learn more about AWS re:Invent at go.aws/46iuzGv.
Subscribe:
More AWS videos: bit.ly/2O3zS75
More AWS events videos: bit.ly/316g9t4
ABOUT AWS
Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts.
AWS is the world's most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers-including the fastest-growing startups, largest enterprises, and leading government agencies-are using AWS to lower costs, become more agile, and innovate faster.
#AWSreInvent #AWSreInvent2023
Thanks for this, very interesting. So do I not need to do an initial snapshot of the database. I thought the process was to do an initial snapshot of the db which is then stored in s3, and any changes to do db are then streamed to the opensearch domain?
Hi, great question! I've found a guide that may help clarify things regarding index snapshots in our Amazon OpenSearch Service. You can check it out, here: go.aws/46Fdh7X. 🔎 For any additional questions, feel free to ask our community of experts on re:Post: go.aws/aws-repost. 👈 ^TH
Will there be CDK support to manage the pipeline configuration?
We appreciate this question! 😀 I've passed this along directly to our team to take a look at. ^AK
Hi there! I'd like to check back in to share the answer that our Service team provided. 💭 Yes, you can use CDK to set up the integration. You can find the GitHub project showing this as described at: go.aws/47cLjPN. ✅ ^AK
36:17 The query has a filter on doc_type : question. However, the results are of type answer. I also see item_name in thr results which is not part of the answer schema shown initially (it was part of ASIN type). Does opensearch dynamoDb integration kind of automatically join related data given they share the same partitionKey?
That's a great question! Our community of experts on AWS re:Post might be able to help you troubleshoot this with details on your current configuration. Feel free to share more info, here: go.aws/aws-repost. ^RM
I'm considering switching over from streams and lambda approach to opensearch ingestion.
Will there be expected improvements in latency? What is the expected latency for opensearch ingestion from dynamodb?
You can find more information on using an OpenSearch Ingestion pipeline with Amazon DynamoDB here: go.aws/3w7V70H. In addition, you can also review best practices for Amazon OpenSearch Ingestion, which may prove to be helpful: go.aws/3UtdJ5e. Feel free to reach out to our re:Post community for further guidance: go.aws/aws-repost. ^AM
So if my opensearch index consists of attributes from multiple dynamoDb tables. Earlier I used Lambda to create an aggregated document from multiple DDB tables. How can I switch to DDB Integrations?
Hi there, great question! You can check out our blog on building aggregations for DynamoDB tables using DynamoDB Streams and Lambda for more insight, here: go.aws/4cZDtM4. 🔗 Additionally, feel free to reach out to our community of experts on re:Post: go.aws/aws-repost. 👈 ^TH
@@awssupport Appreciate for getting back to me. But that’s how I do it now. My question is can this new DDB integration with Opensearch cover this aspect as well?
Hi, thanks for the follow-up! This document on DynamoDB zero-ETL integration with Amazon OpenSearch Service may help: go.aws/47keYIq. 🔎 If you have any further questions, you can check out our guide on how to get technical support, here: go.aws/tech-support. 💬 ^TH
At 14:24 you show a generic way of creating the index and document_id, but in the following examples you have ingested from dynamodb using a single-table design. I suppose (but can't get it to work) that OpenSearch will need each unique type in its own index, but using a dynamic field for creating the index like: ${getMetadata(\"item_type\")} results in an error: Failed to setup dynamic index with an exception
Could you provide a link to the exact setup used in the presentation?
Edit: I just re-wathced that part and can see that you load all items to the same index which makes sense in this example. But is it also possible to ingest each item to a dynamic index like suggested above?
Hi there, thanks for reaching out. I have sent your feedback off to our service team for review. ^AM
Hi! 👋 I'd like to circle back around with the feedback our Service team has shared with me regarding your question. With STD, it's possible, but not necessary, to put each unique type into its own index. It depends on your search query pattern. 🔎 If your table holds data about restaurants (say the restaurant metadata, the menu text, the various reviews) then you probably want to search all of those at the same time so thus one index. But if you decide you want separate search indexes, you can use "route." ^AK
@@awssupport Great, thanks for getting back to me :)