Видео

Glad you liked it

Thanks for commenting

In my video around the 2:20 mark, I explained the process of republishing events from the holding table back to a retry topic, not the main event topic. This approach is crucial for maintaining order and avoiding duplicate processing, especially since multiple consumers might be consuming from the same topic. Deletion of Records: The strategy for deleting records depends on your specific needs for tracking and managing data. You have the option to permanently delete the record or employ a soft delete method, where the record is marked as inactive or flagged without being physically removed from the database. Event Processing Awareness: It is essential for the consumer to perform checks as it processes events to determine if there are related failed or holding events. This step ensures that event handling is informed and accurate. Handling High Traffic and Failed Events: High traffic and the occurrence of a single failed event can indeed result in multiple events being redirected to the holding table, potentially slowing down overall event processing. To address this challenge, implementing a system that maintains strict ordering while still enabling event retries-without outright discarding them-is crucial. However, this setup introduces additional latency due to the need for extra lookups, presenting a trade-off between reliability and processing speed. Alternative Approaches: One possible alternative is to discard failing events and not attempt retries. This approach simplifies processing but risks losing important event data. Another strategy involves designing a more robust system that allows for retries of specific events multiple times by the same consumer. This could be achieved by partitioning topics in such a way that events related to each other are published to the same partitioned commit log, while unrelated events are directed to other partitions. This method helps in isolating problem events to prevent them from affecting the processing of unrelated events, thereby maintaining processing efficiency even under high traffic conditions. By adopting these strategies, it's possible to balance the need for event ordering, processing efficiency, and the capability to handle retries effectively.

I recommend incorporating these tables directly within your existing application database. This approach enables you to fully leverage ACID transaction capabilities, ensuring that operations involving multiple tables and records are executed atomically. This means that all parts of a transaction are completed successfully together, or none are applied at all, enhancing data integrity and consistency across your system.

Thanks

More to come! And glad you enjoyed it

You're welcome!

Noted!

Glad you liked it

I am glad you found it helpful

I am currently scripting new videos, any particular topics/subjects you wish for me to delve deeper into?

Thanks you

Thanks for commenting In answer to your question Based on what you’ve described, here are a few strategies to manage such scenarios effectively: 1. Event Ordering and Timestamps: Ensure that every event in your system is timestamped or has a sequence number to maintain the order of events. This allows you to process events in the correct sequence, even if they are received out of order. 2. State Management with Versioning: Implement versioning in your state management. When processing an event, check that the event’s version matches the current state version or the expected next version. This approach helps in avoiding situations where the state is incorrectly overwritten by an out-of-order event. 3. Event Handling Logic: When processing the holding events, you can check if a newer event has been received that affects the current state. If so, you may need to re-evaluate the holding events in the context of this new state. This might involve discarding some holding events, updating them, or processing them differently based on the new information.

Main Consumer Workflow: 1. Incoming Event Handling: When an event arrives, the main consumer checks if there are any existing events for the same customer ID in the failing event table. • If Yes: The incoming event is moved to the holding table, to be processed after the failing events are resolved. • If No: The event is processed. • If Processed Successfully: Acknowledge the offset. • If Processing Fails: The event is moved to the failing event table. Retry Producer Workflow: • Polls the Failing Table: Regularly checks the failing table for events. • Event Reprocessing: For each event found, it creates a new event and publishes it to the retry topic. Retry Consumer Workflow: • Processing Retry Events: When an event is present in the retry topic, the retry consumer attempts to process it. • If Successful: It retrieves all related messages for the same customer ID from the holding table and publishes them to the retry topic to be processed in sequence. • If Fails: The event should ideally be moved back into the failing table or a similar mechanism, possibly with an incremented retry count or a delay before the next attempt. Handling Failures in the Retry Topic: • Failed Retry Events: If an event from the retry topic (e.g., with consumer index 4) fails: • This event should be placed back into the failing event table. Optionally, include a retry count to avoid infinite retries. • Regarding consumer index 5 and others: If they depend on the successful processing of the previous event, they should also be managed to preserve the processing order. This might mean delaying their reprocessing or placing them in a sequence that respects their dependencies. Special Considerations: • Multiple Messages for the Same Customer ID: If multiple messages for the same customer ID are in the holding table and are pushed to the retry topic, but the first message fails again: • The retry consumer should handle this by either pausing the processing of subsequent messages for this customer ID until the failing message is successfully processed or by re-evaluating the order and dependencies of these messages. The goal is to ensure that messages are processed in a manner that respects their logical sequence and dependencies. Summary of Corrections/Clarifications: • Failing Events Handling: Your understanding is mostly correct. It’s crucial to ensure that failing events, especially in a retry scenario, are handled in a way that respects the order of events and their dependencies. Failed retry events should be carefully managed, possibly with incremented retry attempts, and considering the impact on related events. • Dependency and Order Management: For events related to the same customer ID or having logical dependencies, ensure the processing order is maintained, particularly when dealing with failures and retries. This may involve sophisticated logic in the retry consumer to handle dependencies correctly.

Thanks Here are some common reasons for what might lead to a message might get retired: 1. Transient Failures: If an event fails due to a transient issue (e.g., a temporary network failure, a dependent service being momentarily unavailable), retrying the event after a delay might result in successful processing. Moving the event to a retry topic allows the system to handle it separately without blocking the processing of new events. 2. Rate Limiting and Backpressure: External systems or APIs might enforce rate limits, and surpassing these limits can result in failed event processing. Publishing failed events to a retry topic enables you to implement backoff strategies and control the rate at which you attempt to reprocess these events. 3. Resource Contention: If processing fails due to resource contention (e.g., database locks, high CPU utilization), moving events to a retry topic allows the system to alleviate immediate pressure and retry processing later, possibly under more favorable conditions. 4. Error Isolation and Analysis: Moving failed events to a separate topic makes it easier to isolate and analyze errors without disrupting the flow of successfully processed events. This separation facilitates monitoring, debugging, and fixing issues specific to the failed events. 5. Prioritization of Events: In some scenarios, certain events might be more critical than others. If an event fails but does not immediately need to be retried (due to lower priority), it can be moved to a retry topic, allowing higher-priority events to be processed without delay. 6. Maintaining Event Order: If the order of events is crucial, and a failed event needs to be processed before subsequent events, retrying the event while continuing to process others might violate the order. By using a retry topic, you can control the order of reprocessing to ensure that events are handled in the intended sequence. 7. Handling Poison Messages: Some events might repeatedly fail processing due to being malformed or due to an issue that cannot be resolved immediately (poison messages). Moving these events to a separate topic prevents them from repeatedly causing failures in the main processing flow and allows for special handling or manual intervention.

Thanks for taking the time to comment

I’ll be checking my videos soon to ensure they are have subtitles

@@Danieltammadge thanks

Thank you Marco for taking the time to watch and for leaving a comment. Glad you found the video useful

No I do not. Everything I have published so far is free on RUclips or my blog

Excellent

Thanks again Javis for taking your time to comment.

Hopefully I have understood your question. But here we go What is not recommended is to consume an event, and perfrom a database transaction which updates/inserts records and then after that database activity has finished/committed, to, within the same consuming process to then publish another event. Because what happens if the event publishing fails? You have lost Data and data/systems will be inconsistent. So it’s recommended to only perform one network operation within a process, unless you can ensure state management, retry ability and rollbacks/reconciliation processes. So use the outbox table pattern and use CDC on that table, or CDC on an entity table (i.e. employees table). Let’s use a different example If I needed to trigger a process when an object is uploaded to an AWS S3 bucket. I could have a process which puts the object to the S3 Bucket and after it is uploaded I could have the process publish an event. But what do I do if the event publishing fails? Do I delete the object? What do I do if the object deletion fails? I am going to have objects which are not going to be processed. I could develop a housekeeping process to clean up and find orphaned objects. I could. Or I enable CDC on the S3 bucket and whenever an object is written to S3 an event is published. By a solution which is fully tested, stable, and highly performant. From my experience not pain for CDC ends up being more costly in the long run.

@@Danieltammadge that's correct, I mean you can also use another technique to publish and consume (in order to persist) your own event, to ensure you publish and store the data, so you do not perfom two operation at the same time or any kind of transaction protocol such as phase commits(2pc). great comment, thanks

Thank you for watching and leaving your comment

Good question! The consumer should ensure that the data being consumed in consistent with the schema. Zero Trust, always put guards in place and never trust upstream services. How do you know if an upstream service validated the event in the first place?

Thank you

Thanks for asking your question My approach principle 8 as if it is describing breaking systems down into microservices or cells which can be scaled independently using load balancers or even better in my view loosely coupled communication like message queues or events streams. And how do you enable services or functions to scales horizontally, well you ensure they are stateless in the approach that the input contains what a process requires, or it can use a networked service and fetch/lookup what it requires.

Using this definition of a partition. In Apache Kafka, a partition is a unit of parallelism for managing and storing data. Each topic in Kafka is split into one or more partitions, which are essentially ordered, immutable sequences of records. Each partition can be thought of as an independent "mini" Kafka topic, with its own set of messages, offsets, and metadata. Partitions enable Kafka to provide high throughput and scalability, as they allow multiple consumers to read from a topic in parallel. And the offset is the position of a event/message in a partition. By storing the two. You know what partition the event was from and it’s position therefore as you consume later events you are able to validate which events are required to be processed after the current failed event is processed.

Ta

Thanks for taking the time to comment. Glad you liked it

Yes. It follows the same technique used as retry topic consumers. I.e if circuit is open then wait a predefined time before trying again.

Thank you. Event-carried-state-transfer is a pattern that uses events to transfer the state of an object from one service to another. Whenever an event occurs in one service, a message is sent to all other services that need to know about the event. This message contains the state of the object, which allows other services to update their own state accordingly or to process the event without requiring fetching the state from an external source, often the publishing system. Compared to CQRS, which is a pattern that separates the read and write operations of a system into two different services, the write service is responsible for handling commands and updating the state of the system, while the read service is responsible for handling queries and retrieving data from the system. This segregation allows for better scalability and performance optimization. For example, you may be writing to a relational database where data is normalized. However, performing complex queries on relational databases can be slow and may not fit your requirements, whereas using an OpenSearch database might be better for querying and fetching data. So you implement CQRS and stream the changes, the change events, from the relational database to Opensearch denormalising and tuning the data for fetching.

@@Danieltammadge wow..sir.. thanks a lot for taking time to respond in detail. 1 follow up question if you do not mind. If I have a restaurant-service, payment-service, delivery-service, can I have them as restaurant-service (command), payment-service (command), delivery-service (command) as 3 command handling services and 1 query-service (which collects data from all other services to build materialized view to query)?

### Overview In terms of performance, it's possible to have three command services and one query service. However, it's important to consider the tradeoffs and whether this is the most efficient solution. ### Network Operations If the client queries three different endpoints, there will be multiple network operations. This can result in longer latency and more compute power needed. Additionally, if one of the endpoints fails, the client may not be able to receive all the data. ### Materialized Views An alternative to querying multiple endpoints is using a materialized view. This denormalizes the data and stores it in a way that allows for quicker queries with less compute power required. However, materialized views may not always be consistent and may have lag in distributed services. It's important to consider whether this is acceptable for the use case. ### Considerations When deciding whether to use multiple command services and one query service or a materialized view, consider the ordering of events, the need for consistency, and the potential tradeoffs between performance and efficiency. Overall, while it's possible to have three command services and one query service, it's important to carefully consider the best solution for the specific use case. However, if you are asking, is this an acceptable approach to certain use cases, yes, yes it is.

@@Danieltammadge I have never seen anyone like you who answers the question very seriously with much details. I really really appreciate it. God bless you. Please keep releasing new videos. 🙏

I am glad you found it helpful and taken the time to comment

Thank you

In event-driven systems, you encounter two types of issues: transient faults and errors due to poison events. A transient fault occurs when your system is attempting to call an external component, external service, or third-party service. In most cases, these faults are self-recovering and intermittent, and retrying the call to the external service is usually sufficient. However, you should design for long-term outages and implement the circuit breaker pattern to ensure that your processors wait for the downstream service to recover before resuming processing if a transient issues arise. On the other hand, poison events are a type of error that can be caused by the publisher or the processor/consumer. For these errors occurring due to the event then you should publish a retry event to a retry stream or to dead letter queue to be picked up by another process. This approach ensures that processing continues for other events and is not halted by a single poison event.

Thanks Daniel..this makes complete sense.. i started with EDA systems with wrong understanding that its simple but the more i read i understand its a wider topic.. thanks for sharing your knowledge on this.

I use Lucidcharts (lucid.app/) for my diagrams, using actions to hide and show layers. By using layers and actions together, I record my iPad screen and then scale the screen recording into the videos. I recommend the following tools for architects: - Lucidcharts (lucid.app/): A good diagramming tool for creating diagrams for documents and presentations. But also the free [draw.io](draw.io) is also good, when working with teams - Archi (www.archimatetool.com/): An open source modelling toolkit for creating ArchiMate models and sketches. It is good for maintaining the current and future architecture state for both technology and the business. - [Notion.io](notion.io/): Better than Excel and really good for maintaining notes, dependencies (as well as component/interface lists), and relationships between notes, requirements, and other artifacts. - [Stoplight.io](stoplight.io/): I have not found a better API design and documentation tool for HTTP/REST APIs than Stoplight. - Jira, Confluence, and Bitbucket. - sequencediagram.org/: An amazing tool for creating sequence diagrams. - jsoneditoronline.org/: An awesome tool for designing JSON objects.

Really appreciate that you took the time to comment and ask your question. Thank you Sergey

Glad you liked it!

As we want to use Azure Service Bus in our case which does not support schema registries I need to create an Event Hub Namespace while only using the schema registry feature?

Yes in your case it looks like this is the way forward for you learn.microsoft.com/en-us/azure/event-hubs/create-schema-registry?source=recommendations I am not too familiar with Azure as I spend the majority of my time with AWS

Thank you for taking the time to watch and share your experience.

@@Danieltammadge Thank you for the confirmation. 👍🏼

Hopefully the following will help danieltammadge.com/2023/02/delaying-apache-kafka-retry-consuming/ Thanks for watching and taking the time to comment

Thank you, I will

I have not used Debezium myself. As I run my systems in AWS for now an I prefer managed, or as a service over self-run. So my systems use AWS Data Migration Service to stream to AWS Managed Kafka Service Topics. Thank you for taking the time to comment and share your experience.

@@Danieltammadge I also run in AWS using AWS Aurora (MySQL flavour) and MKS (Kafka) and Debezium is a Kafka connect "app" which is also run as a managed service by AWS. The "only" trick I had to do was to implement a Kafka config plugin so that Debezium could pick up passwords for the DB from AWS Parameter store. And you have to enable the bin log in Aurora which normally you don't enable.

Glad it was helpful! And thank you for watching

No. I was referring that a simple queuing service like SQS or a a message broker like activemq or RabbitMq (running under Amazon MQ, if on AWS), has the concept of a queue which are designed to be consumed by one type of consumer. When multiple consumers are subscribing or pulling from the same queue. It acts as a load balancer and can distribute messages between them. Amazon Kinesis and Apache Kafka full under streaming or logs. A stream or a log maintains the immutable sequence of events for a retention period, this allows consumers to consume the stream or log independently from each other

@@Danieltammadge Thanks , this does clear my confusion.

Perfect. Thanks for watching Paras

ChessMaster thank you for taking the time to comment. Quick question is your comment a question?

@@Danieltammadge yes. Can you provide some. Ode configuration examples a out how many error scenarios need to be handled in a messes queue.

Thanks for taking the time to comment. Glad you enjoyed it.