OpenSearch
OpenSearch
  • Видео 253
  • Просмотров 131 669
OpenSearch Deep Dive - Search (part 2)
In this session, we pick up from where we left off in TransportSearchAction and dive into the ‘execute’ request method. The talk covers how listeners work, especially for transforming requests and responses using OpenSearch’s search pipeline feature from 2023. We also explore query rewriting and cross-cluster search logic. There’s a look at point-in-time searches, slicing, and shard routing. We wrap up by setting the stage for the next talk about executing async search actions.
(part 1) ruclips.net/video/MJPqYT7QPXA/видео.html
Просмотров: 123

Видео

OpenSearch Indexing Deep Dive (Part 4)
Просмотров 18514 дней назад
In this fourth part of the OpenSearch indexing deep dive series, we explore shard-level operations, primary terms, and mapping updates. Michael critiques the current distributed system code and suggests improvements. The session wraps up just before we get into the InternalEngine that finally hands indexing operations to Lucene (along with managing the transaction log and version map). Part 1: ...
OpenSearch Deep Dive - Search (part 1)
Просмотров 25014 дней назад
The video marks the beginning of a new series focusing on the search code path in OpenSearch. This first episode introduces key concepts such as REST actions, the prepareRequest method, and how URL parameters and the request body are parsed and handled. The session touches on SearchSourceBuilder, the use of Jackson for parsing, handling XContent, and possible future changes in OpenSearch 3.0. T...
OpenSearch Deep Dive - Indexing (part 3)
Просмотров 12421 день назад
In this third episode of our deep dive into OpenSearch indexing, we continue from where we left off, focusing on the TransportBulkAction, specifically the doRun method of the BulkOperation class. We delve into the process of grouping bulk item requests by shard ID and creating bulk shard requests. Moving forward, we explore the TransportShardBulk action, handling shard-level requests, and trans...
OpenSearch Deep Dive - Indexing (part 2)
Просмотров 15928 дней назад
Continuing the journey started in part 1 (ruclips.net/video/Nc5Za3KJzYk/видео.html), we get into the BulkOperation, which is responsible for dispatching parts of the indexing bulk request to individual shards. Along the way, we look at how aliases get resolved into “concrete indices”. We also dive into document routing and how documents are routed to shards by default or based on a custom value...
OpenSearch Deep Dive - Indexing (part 1)
Просмотров 1,1 тыс.Месяц назад
In the first of a series of videos diving into OpenSearch’s expansive codebase, we look at how OpenSearch processes single-document indexing requests. We start from the indexing REST handler and see how that gets mapped to the underlying transport action. All index/delete/bulk actions get processed by the same underlying “bulk” code, eliminating duplication. This first dive finishes partway thr...
Akash Shankaran - Improvements to OpenSearch using Intel Sapphire Rapids and hardware accelerators
Просмотров 1762 месяца назад
In this session, we will share performance improvements in vector search scenarios due to the use of Intel Sapphire Rapids on AWS. The value of hardware accelerators such as AVX512 in Lucene and Faiss libraries will be discussed, along with performance results. We will further dive into the topic of compression, and some innovations in that space. This will focus on using Intel QAT (Quick Assis...
Laysa Uchoa & Yuliia Barabash - Search Results with Amazon Personalize and OpenSearch
Просмотров 682 месяца назад
Full title: Search Results with Amazon Personalize and OpenSearch: Making Searches Smarter OpenSearch is a powerful open-source search service, but its traditional ranking system doesn’t take into account user behavior. That’s where Amazon Personalize comes in! By combining OpenSearch with Amazon Personalize, we can re-rank search results based on what users have done in the past and various me...
Lucian Precup & Amine Gani - Leveraging OpenSearch for enhanced e-commerce search experiences
Просмотров 1202 месяца назад
Dive into the transformative capabilities of OpenSearch in revolutionizing e-commerce search experiences. This presentation will showcase how OpenSearch’s robust architecture supports advanced features like auto-completion, targeted merchandising, and more. Discover strategies to optimize search functionalities with practical, real-world examples and innovative tools. Speaker: Lucian Precup - o...
Mohit Godwani - Optimizing Storage Performance & Efficient Index Configuration
Просмотров 1842 месяца назад
Full title: Optimizing Storage Performance and Efficient Index Configuration with Compression Codecs in OpenSearch In the realm of distributed search and analytics engines, efficient storage management and index configuration are paramount to ensuring optimal performance and scalability. As data volumes continue to grow exponentially, optimizing storage utilization and reducing disk footprint h...
Jonah Kowall - Implementing OpenTelemetry and Jaeger with OpenSearch
Просмотров 2452 месяца назад
Observability is top of mind today, we’ll start with the basics and then jump into why OpenSearch should be part of your observability story. Today, there are so many great open source options, but the complexity of running your open source stack can be daunting. There are so many options out there today, in this talk you’ll learn why OpenSearch is the most flexible option to be a key part of y...
Charles Christian & Jay Wilks - Managing the SIGMA detection rule lifecycle
Просмотров 2112 месяца назад
Full title: Managing the SIGMA detection rule lifecycle using a python, git repository and the OpenSearch API How to take control over the default SIGMA detection rules in OpenSearch Security Analytics and deploy a custom set of detectors and sigma rules using python, a Git repository, and the OpenSearch API. Speaker: Charles Christian - opensearch.org/community/members/charles-christian.html J...
Samuel Herman - Lucene And Beyond - Core Storage Extension In OpenSearch
Просмотров 1872 месяца назад
OpenSearch is tightly bound to the Lucene core APIs that facilitate the following functionalities: - Encoding - Transactions - Merges - Search - And more… In this presentation I will discuss how the OpenSearch storage encoding can be extended to popular formats (e.g. Parquet, Avro) that are readable by public big data systems such as Apache spark. This provides a strategic long term benefit for...
Kassian Wren - OpenSearch Notebooks: Monitor, Report, Learn, Explore and more!
Просмотров 852 месяца назад
OpenSearch notebooks are a powerful tool within the OpenSearch platform. They allow you to combine text, code, and visualizations of your data in one place. In this talk, I’ll walk you through the concept of the OpenSearch notebook, create one using some real-world Open API data, and talk about how this tool can be used for so much more than basic reporting. Speaker: Kassian Wren - opensearch.o...
Laysa Uchoa & Yuliia Barabash - Exploring OpenSearch, Python, and Serverless
Просмотров 392 месяца назад
Full title: Exploring OpenSearch, Python, and Serverless: Crafting Efficient and Modern Search Applications Facing challenges with search capabilities in your production environment? Discover how the combination of OpenSearch, Python, and serverless architecture can be your solution. This talk provides hands-on examples, from building efficient queries to implementing production-ready practices...
Aruna Govindaraju, Eric Pugh & Stavros Macracis - You’ve Deployed User Behavior Insights. Now What?
Просмотров 482 месяца назад
Aruna Govindaraju, Eric Pugh & Stavros Macracis - You’ve Deployed User Behavior Insights. Now What?
Moutupsi Paul, Xiaotang Wang & Amulya Sharma - Airbnb Embedding Platform
Просмотров 2442 месяца назад
Moutupsi Paul, Xiaotang Wang & Amulya Sharma - Airbnb Embedding Platform
Ramesh Kumar Manickam - Scaling Operational Intelligence into Business Intelligence
Просмотров 682 месяца назад
Ramesh Kumar Manickam - Scaling Operational Intelligence into Business Intelligence
Gustavo Sanchez - Future-proof AI applications with OpenSearch as vector database
Просмотров 1082 месяца назад
Gustavo Sanchez - Future-proof AI applications with OpenSearch as vector database
Trey Grainger - Building AI Models with OpenSearch's User Behavior Insights
Просмотров 582 месяца назад
Trey Grainger - Building AI Models with OpenSearch's User Behavior Insights
Migrating from Self managed Elasticsearch to OpenSearch & How Indias Fantasy Sports Giant Leveled Up
Просмотров 3002 месяца назад
Migrating from Self managed Elasticsearch to OpenSearch & How Indias Fantasy Sports Giant Leveled Up
Unconference Talk: Fernando Rejon Barrera - How to finetune state of the art vector embeddings
Просмотров 592 месяца назад
Unconference Talk: Fernando Rejon Barrera - How to finetune state of the art vector embeddings
Anandhi Bumstead - OpenSearchCon North America 2024 Opening Remarks
Просмотров 1142 месяца назад
Anandhi Bumstead - OpenSearchCon North America 2024 Opening Remarks
Charlie Yang - Toward Semantic Search, what we have now and what it will be in the future
Просмотров 2772 месяца назад
Charlie Yang - Toward Semantic Search, what we have now and what it will be in the future
Pallavi Priyadarshini & Samuel Herman - OpenSearch Project Roadmap 2024 and Beyond
Просмотров 3042 месяца назад
Pallavi Priyadarshini & Samuel Herman - OpenSearch Project Roadmap 2024 and Beyond
Derek Ho - Building Your First OpenSearch Dashboards Plugin: A Hands-On Demo
Просмотров 3782 месяца назад
Derek Ho - Building Your First OpenSearch Dashboards Plugin: A Hands-On Demo
Mehul A. Shah - Luna: LLM-powered Unstructured Analytics (aka “RAG-supercharged”)
Просмотров 3232 месяца назад
Mehul A. Shah - Luna: LLM-powered Unstructured Analytics (aka “RAG-supercharged”)
Prabhat Chaturvedi & Shikhar Jain - Attribute based access control (ABAC) & the Future of FGAC
Просмотров 3542 месяца назад
Prabhat Chaturvedi & Shikhar Jain - Attribute based access control (ABAC) & the Future of FGAC
Community Meeting October 1, 2024
Просмотров 982 месяца назад
Community Meeting October 1, 2024
Unconference Talk: Infino - Prometheus / OTel integration added natively to Lucene
Просмотров 852 месяца назад
Unconference Talk: Infino - Prometheus / OTel integration added natively to Lucene

Комментарии

  • @naveenkumarreddy3073
    @naveenkumarreddy3073 7 дней назад

    Make one video for creating network traffic map visualisation in an open search dashboard.

  • @CanaryJudge
    @CanaryJudge 8 дней назад

    I know we can use mustache variables in the action template but can we just those in the subject as well or are those variables only available in the body?

  • @roharbaconmoo
    @roharbaconmoo Месяц назад

    There is nothing unethical about measuring employee performance.

  • @Spartans_Of_Stick_War
    @Spartans_Of_Stick_War Месяц назад

    nice efforts by amazon

  • @charleschristian2898
    @charleschristian2898 Месяц назад

    Thank you for this. I tried looking for the first video you mentioned a year ago but could not find it. Any chance you could link it.

    • @OpenSearchProject
      @OpenSearchProject 28 дней назад

      Here you go: ruclips.net/video/9nD0vinQ7Gs/видео.html

  • @ddd2283
    @ddd2283 Месяц назад

    Иссуси

  • @fspezzano
    @fspezzano Месяц назад

    Hello, I followed the guide and waited some time after getting the link for the dashboard in the console but I still get the red error. I tried both on windows wsl and macos. Please can anyone help?

  • @manki6472
    @manki6472 Месяц назад

    intersting!!

  • @nishantgoel769
    @nishantgoel769 2 месяца назад

    Kudos!! Great talk Mohit 😃

  • @prinzuchoudhury6920
    @prinzuchoudhury6920 2 месяца назад

    Great insights. Very informative session 🔥

  • @nishantgoel769
    @nishantgoel769 2 месяца назад

    This is great guys!!

  • @rishavkumar7715
    @rishavkumar7715 2 месяца назад

    🤩🤩

  • @ganeshpai8095
    @ganeshpai8095 2 месяца назад

    Hi, Why only Kafka and In Memory Buffer. Can we not have file based buffer like in Fluentd and like Persisted Queue in logstash. Also can we introduce painless like script for advanced processing in processor

  • @_alvintang
    @_alvintang 3 месяца назад

    Awesome content!

  • @anonymousxy7330
    @anonymousxy7330 3 месяца назад

    Nice video, thx for uploading.

  • @jamesmadlangtuta3156
    @jamesmadlangtuta3156 3 месяца назад

    where can I see references/demo code on the otel-collector, fluentbit, and data-prepper config files?

  • @JossOrtan
    @JossOrtan 4 месяца назад

    Great insights on integrating threat intelligence with security analytics! How do you prioritize which threats to focus on with so much data coming in?

  • @ChristophLünswilken
    @ChristophLünswilken 4 месяца назад

    Heyho! Why can't i use an index-pattern in the correlation engine? Would i have to work with an index-alias instead? My index would be named like "logs-aws-cloudtrail-yyyy-MM-dd" by the ingestion pipeline. Would it be better to name it "logs-aws-cloudtrail" and do a rollover via ISM policies instead, to have a better usage with e.g. the correlation engine?

  • @sg0luc
    @sg0luc 4 месяца назад

    Hello! Is it possible to specify the required number of events to generate an alert? Example: I want to detect a brute force scenario followed by a successful login on a Windows computer. To do this, I need there to be at least 3 failed logins and, after these failed logins, a successful login event. Thanks!

  • @sg0luc
    @sg0luc 4 месяца назад

    Hello! Is it possible to specify the required number of events to generate an alert? Example: I want to detect a brute force scenario followed by a successful login on a Windows computer. To do this, I need there to be at least 3 failed logins and, after these failed logins, a successful login event. Thanks!

  • @YeonghyeonKo
    @YeonghyeonKo 4 месяца назад

    Is there any update about Management Console?

    • @BeforeAndAfterScience
      @BeforeAndAfterScience 13 дней назад

      The Management Console was developed, and it was nice. It was, however, developed under a proprietary license. So when Elastic acquired Opster at the end of 2023, Elastic pulled it out of public view.

  • @ganeshpai8095
    @ganeshpai8095 4 месяца назад

    Sync issues with the video

  • @sarthaksoni986
    @sarthaksoni986 4 месяца назад

    बहुत खूब

  • @VenkataPrasad-n4l
    @VenkataPrasad-n4l 5 месяцев назад

    how to take backup of the data from the DB's persistent volume?

  • @siddhesh-shinde-trellix
    @siddhesh-shinde-trellix 6 месяцев назад

    No explanation of how to create docker image. Not present in this video and not present anywhere in the docs as well except the README present in docker folder of opensearch-build repository. It advises to run a script which will use the Dockerfile present in the repository. But the script itself fails, even after passing appropriate parameters to the script.

  • @qato62
    @qato62 6 месяцев назад

    This person doesnt know where Voskopoja is, and writes history about Voskopoja You can trust this person about databases etc.

  • @Guerilla333
    @Guerilla333 6 месяцев назад

    the dimensional explanaton of vector space, (3d) is excellent at 12:50

  • @KiplangatPeter
    @KiplangatPeter 7 месяцев назад

    Great presentation

  • @johndevan3505
    @johndevan3505 7 месяцев назад

    The presentation is disorganized. Just like random stuff being talked about by a bunch of people. The dashboard and visualization terms in opensearch are not obvious. I wish you can present more clear ways on what to accomplish. The sample data is also nothing useful.

  • @dep4b
    @dep4b 7 месяцев назад

    Thanks for highlighting the evaluation piece of the puzzle, and the fact that most don't have good ways to validate that your semantic solution is working!

  • @dep4b
    @dep4b 7 месяцев назад

    This was fun Charlie! You are much more natural than I!

  • @andyssaid5512
    @andyssaid5512 7 месяцев назад

    Fantastic job! The video was beautifully executed and incredibly insightful. Keep up the great work!

  • @manojprabhakar7951
    @manojprabhakar7951 7 месяцев назад

    Hi , How to convert seconds or millisecond's to HH:MM:SS FORMAT, our data is contact center data, we are working on an real time contact center dashboard

  • @roharbaconmoo
    @roharbaconmoo 7 месяцев назад

    Bears. Beats. Battlestar Galactica.

  • @MillionairesClub2023
    @MillionairesClub2023 8 месяцев назад

    Threat intelligence feature missing in on priem version

  • @MillionairesClub2023
    @MillionairesClub2023 8 месяцев назад

    How you created index ?

  • @alexandr0id
    @alexandr0id 8 месяцев назад

    Sound like these "challenges" and tricks are really the result of poor design. It's poor design to leave everything customizable because you have no idea and no guarantees about system behaves under load. You just offload engineering work to your customers...

  • @isalla3180
    @isalla3180 9 месяцев назад

    ScreenCapture fail starting at around 16:00

  • @aljabitech3131
    @aljabitech3131 9 месяцев назад

    Hi everyone, I want all OSD packs because I am having problem downloading them via link "Yarn OSD Boot Strap". I would like to receive packages in "Zip" format from you Note: I am using Windows 11.

  • @aljabitech3131
    @aljabitech3131 9 месяцев назад

    Hi everyone, I want all OSD packs because I am having problem downloading them via link "Yarn OSD Boot Strap". I would like to receive packages in "Zip" format from you Note: I am using Windows 11.

  • @aljabitech3131
    @aljabitech3131 9 месяцев назад

    I want all osd packages because I have problem downloading them via link "Yarn OSD Boot Strap". I would like the packages in "Zip" format please.

  • @Luis-jl6oh
    @Luis-jl6oh 10 месяцев назад

    Thanks

  • @wangchi562
    @wangchi562 10 месяцев назад

    Thank so much bro

  • @jianghancn
    @jianghancn 11 месяцев назад

    Good to see your presentation, David!

  • @JohnJosephKennedy
    @JohnJosephKennedy 11 месяцев назад

    Great to see this Mehul and Jon. Thanks for going into this level of details.

  • @turan_sultan
    @turan_sultan Год назад

    Great presentation!

  • @dailywisdomquotes518
    @dailywisdomquotes518 Год назад

    could you please open source the code, react components.

    • @OpenSearchProject
      @OpenSearchProject Год назад

      Hello. You should be able to find the code you are looking for here: github.com/opensearch-project or under Aryn.ai's org: github.com/Aryn-AI Hope this helps.

  • @siddhantgupta7831
    @siddhantgupta7831 Год назад

    Jon is best

  • @phongb2b
    @phongb2b Год назад

    Amazing ;-)

  • @ravi4team
    @ravi4team Год назад

    Thanks for the presentation. Like mike, I'm also wondering how can I enhance our keyword search to semantic search without major changes. Please provide a reference or code samples to use SPLADE function to get the expanded text data.