The best way I've come to describe this concept is: "Don't store the value. Instead, store the things you do to the value." From there the questions start coming organically that I can answer as needed. The two main questions I hear are: * What about the values needed for the operation? - I create classes for the various operations (usually record types), serialize them, and store a timestamp of when it happened, the serialized version of the object, and the name of the operation. * So what do I do with that information? - The naive approach is to start at the beginning, iterate through each operation, and call the function associated with that operation. By the time you are done your service class or whatever it is will be in the same state you left it in when you closed the app. From there I usually end up talking about snapshots and explain that. Like Nick said...it really isn't that hard and isn't as complicated as people think it is. Starting with the first thing I mentioned at the beginning helps the conversation a lot.
As someone that's worked with accounting systems this is completely understandable. You never go back and change a value, even if it was in error. You can only add new transactions that modify the value . Everything is just a sequence of transactions that takes you from a starting balance to the current balance. (The starting balance is typically arrived at by closing the previous financial period where that prior set of transactions are either moved to history or, if the GAAP allows, discarded). We'd go about determining the value as of some point in time in a different way, but it's the same concept. An in-memory similar concept is a Hughes list. You never modify the starting empty list, you only build up a sequence of lambdas or similar that each take an input list and produce a new list that is modified in some way. Chained together they hold the sequence of modifications you want to make to that initial empty list. When those lambdas are executed they produce the final list. But the original list and all the intermediate lists are immutable, so you could branch from any of those lists, using it as a starting point for an alternate list.
That's a very nice video you had put together sir! Would be also nice to have a future video exploring some gotchas of the Event Sourcing world, specially the trade-offs one needs to consider when working with it, since is very easy to shoot yourself in the foot when attempting to use ES.
I'd love a video on how to handle versioning with the events. It's one of those things that I don't think I've ever really done well and would like alternative approaches to consider.
Having built a smaller cloud-native app in event sourcing, I can confirm this video captures the concept pretty well and makes it easy to understand. Also, having built the aforementioned app in event sourcing, the architecture is quite annoying to work with. There are limitations you have to respect, issues you have to deal with and extra infrastructure to implement. It does go well with CQRS and together they make nice looking tests though.
This video was really interesting! I'd love to see more videos like this. I always enjoy your videos because I can always follow along with your explanations, even on concepts that are typically difficult to follow.
That's a great introduction. Would be cool in oncoming videos to go a little bit deeper down the rabbit hole of Event Sourcing. You have really great content! Thank you!
Just wanna say thanks for the great video. I know you have dometrain that you want to grow, so I'm just happy that this channel is still a place you put out valuable educational videos
Awesome video nick! would love to see more about event sourcing and the outbox pattern. Would be interesting to see how bugs are handled and what should be done when the requirements and domain model evolve.
Great vid, Event sourcing is quite a paradigm shift for a lot of people, rethinking of objects as a first derivative of an event stream is less familiar than the classic CRUD / SQL. It's definitely powerful in the right context. I'd definitely be hesistant to spear head event sourcing proposals in the workplace though, I sense it can be less forgiving if you approach it with the same mindsets to most crud systems of "oh we'll just do a migration / run some sql to solve it".
What database should I use for a small project expecting a low usage volume and with two different event domains? Should I use MongoDb, Cosmos or DynamoDb, or a database made for events such as EventStoreDb, or even use PostgreSql using Marten library?
I have some question. I'm implementing a system that have a object Tasks, this tasks need to have an historial of changes of user asignade, status, comments and modfications occurred to show at FrontEnd. But too I need to filter this task by actual status, user asigned, priority, created date. How filter the all registers with Event Sourcing? it s not be to heavy for the system? Event Sourcing is ideal to get the all events but i have dubts about his performance when i filter the tasks
Just want to mention the EventFlow C# library, which provides nice flexible DDD and CQRS structures handling most concurrency issues for an event sourcing system. Thank you for covering this topic Nick, would like to see more on the topic, most videos on the topic only touch the surface.
I needed some time to grasp what Eventsourcing is, but you introduction made it so easy to understand: It's same as mssql replication. Every transaction is replicated even the data that gets transferred is no more...
This is an absolutely fantastic video. One thing I want to know, how would you do something like Get all Students with a Birthday in January, or all students enrolled in Maths. What student is enrolled in the most cores? Also, how would deleting work?
Nice and simple explanation. I do however think you should use a number for the version. When running projections you don’t know if you have missed an event or not using datetime. If you were using integer you would directly see that something was wrong if you received version 6 before 5 for example.. using timestamp you can’t know if some event for a timestamp is missed.
Events are published and consumed given the code at their time. If I try to process an old event with the today's code, it may fail. How does event sourcing handle this issue? Please, don't tell me that we have to version events AND handlers, that would end up in a ball of mud
@@PhantomPhobos there's a huge difference: ef migrations are applied before the deployment of the application itself, within a transaction. If applied to prod, they are correct; on the contrary, we know code will always evolve
This seems like your data would be better stored in a Temporal table in SQL (if using SQL). You basically have a single "add" and then just updates to the row. The underlying DB creates a new row for each update. You can always get the "latest" record, but also the historical data. .NET EF Core handles it perfectly.
Sure, then you are creating a table for each object, if you want to add something to your aggregate you have to modify the schema, etc. Temporal tables aren't really event sourcing, more a method for auditing, and could possibly be a better use case in some places.
Very nice, there are scenarios I don't know how to manage: 1 - what if you need to wait to one entity is created to create another entity, 2 - If I have to create two related entities, and one have db generated autonumeric id that I need for the other entity?, 3 - If I have an API that creates an entity how to return a success or failed operation?
Fantastic topic and highly relevent to a current project of mine. Would love to see more please.
7 месяцев назад
Is DateTime/Timestamp used for ordering of events in real-world applications? In case of updates that are atomic (take from one account and place on another) are then timestamp synchronized in some way?
You can reject an append if you try to write something that had something else written already during the time you were processing if that's what you mean
A system designed like this may lead to unexpected results in highly contested scenarios. Just because events order in a particular way after the fact, it does not mean that they were originally processed in that order. Think of multiple instances of the application running simultaneously. -- some may be a little quicker, some may be a little slower, clocks drift, etc. Using Conflict Free Resolution Data Types (CRDTs) would be one way to mitigate the issue. Putting an actual Event Sourcing database or a persistent que as the source of truth in front of the system, instead of a general-purpose database, would be another way.
Is there a reason at the 24:00 minute mark that you don't use interfaces, and have the classes implement the interfaces so there is no polymorphism, and thus fewer serialization issues?
I tried once to develop an app that would be a game server from a Unity MMORPG could or should the event sourcing be implemented in such project, what is your opinion?
Great video (just as your presentation at Techorama Belgium ;-) ) One important note out of my own experience: the projection you're building is an single entity projection. If you're in need of an "overview" (typical "list" views in UIs e.g. a list of bookings, list of students, ... ) you're still in need of another projection with a different partition key since other document stores e.g. Azure Cosmos, ... don't like to query accross multiple partitions (high query cost). This can be done in various ways of course (e.g. listening to event feed as you showed to update the "overview" projection table). Or do you see still other approaches for the scenarion over having "overviews"?
That looks like it is destined to be used with the newtonsoft json serializer. Imagine all the polymorphic json attributes in a bigger project for that abstract event class... I would be in shambles! EDIT: Yes, please more Event Sourcing!
Suggestion: It would be good to mention Discrete Event Systems and their fundament[s] Automata Theory. Those are different from Continious Systems described by differential equations and Discrete Systems described by difference equations. Applications: Compilers (lexycal and semantical anaylsis), Manufacturing systems, Traffic/Transportation systems, Networking, Banking, Crypto, Supply Chain... edit: added Crypto
Awesome. I would really like to hear more about how to handle versioning if the model changes. For example, if a field is added or removed, how should should earlier events be processed?
I once designed such a system. What I did is that I added to each event a version property and a dictionary of arbitrary objects used to store data fields of the event that corresponded to its API at the time the event was recorded. When reading an event from the data layer corresponding to an earlier version of the event, instead of mapping the data fields to API members of the current version, I would instead store the data into the object map, using the original property names as keys. After that, I would send the event through a chain of transformations meant to project the event into the current version. These are essentially a collection of strategy objects designed to change the state of the event from version T to T+1, moving around the contents of the object map to reflect what the event would have been like if it had been recorded at version T+1. This keeps up until the event transitions from the version at the time it was recorded until the current version, at which point the actual payload properties of the event would be set for the same event captured now. Nowadays, instead of the object map, we might consider using serialized JSON, but the idea is the same. That being said, this adds a lot of overhead in terms of read processing, which is why this solution was also supplemented with optimized read models (see CQRS) to decouple the concerns. The idea is to playback past events to allow a subscriber to create a view optimized for its own needs. These read models were entirely disposable as they could easily be recreated from the event history at application start.
Great Video as usual Nick. I appreciate the idea was to keep it as simple as possible, however, I would contest your use of StudentUpdatedEvent, I would have expected a StudetNameChangedEvent, StudentEmailChangedEvent, StudentDoBChangedEvent, that way you are only storing the actual change not the whole object each time. If there were a lot of fields, storing the whole object each time could use a lot of space (something EV is often criticised for anyway). I appreciate that, ordinarily, the student isn't going to change that often so I'm probably being a little pedantic, but there could be scenarios where an instance (student in this case) may change more often. Would this not then lose the benefit of the audit trail that EV is supposed to give us?
Seems great, but I have a question: what happens if you have millions of users, and business comes with an idea that we should send a birthday greetings email to everybody who has a birthday in the current day. How would you approach this to even find the students meeting the criteria in a given day? Normally I would probably just search for them using some sql query, but here you don't store the current state anywhere, we have to build every student from the ground up which is going to be massive. Am I missing something?
I wonder if RavenDB could recreate the projection using stored procedure. It would make the saving into a trivial insertion. The only downside I can see is that Apply() should not depend on the object state (e.g. validating number of products based on storage capacity), should not contain business logic and might require full indepotency...
Nice video! Can you elaborate more about the uses cases of Event sourcing. I have never seen event sourcing used in the real world. What types of applications are suitable for this approach ?
I tried to create a lightweight event sourcing framework where aggregates are also their own inline projection. What I wanted to do is, use an sql db for storing aggregate as the traditional way, but adding its events to the events table with intercepting saveChanges of EfCore. The result? I can query the aggregate(actually its projection also, because its its own projection) using SQL and have an append only log of its state changes. Don't know if its applicable in real life scenarios
I our GDPR world, Is event sourcing even legal as a concept? Using a immutable log where someone has the right to be forgotten and data deleted seems a risky choice.
That is a strange concept of rebuilding the state from scratch. In an accounting software I worked on there was a state calculated from some point of time. In simple case you just get the last state from the db, apply one event and save it back to the view instead of applying thousands of events from the beginnning
There are obviously a range of optimizations to improve state management, also called snapshot patterns. Nick only touches on the surface of event sourcing, and it’s a good introduction imo but you’d want to skill up significantly before moving to a more complete implementation.
Yes, long living entities with frequent updates are a performance nightmare. One solution would be to sort of archive the older parts of poor-performant streams.
Short lived entities are a key concept that is usually not stressed enough in entry level introductions to event sourcing. Or else... a lot of things can go off, performance-wise, even if you implemented it by the book.
Question about naming - Nick used GetStudent and GetStudentView for projected Student. Why GetStudent is used for the entity that is created based on the events and not vice versa? When I see View name reminds me SQL views that materialise data from other source (because view does not have data itself) and in this case I would rather use GetStudentView for Student created from events instead. I believe it is event-source terminology but can someone explain this to me or refer me to a proper definition? Thanks!
I think I found answer, but terminology is quite confusing. Events are source-of-truth and this is main point of interest in Event Sourcing. The second approach to keep the most recent state in a projection is called view - to just access this information from a faster route (straight from db rather than going through all events). That means this projection is just a view (not the main source-of-truth) of the actual state we have in our events regardless if it is stored in database or not.
@@furrman18 I consider the projection part of the read model if you are using CQRS. The nice thing is you can add projections without changing any of the events to make your query side super fast, ie, one projection/view to display a certain thing in your UI.
Nice! Just curious, how would you deal with concurrency using DynamoDB in case two things load the same Student and both try to update something? I would assume that by default the last one to update would win.
7 месяцев назад+1
It's a bit complex but possible, in a transaction, persist the record + the latest "version" of it somewhere conditioned to the version matching the expected
That's where the version thing comes in. You can use optimistic concurrency version checks to fail a write transactionally and re-read and re-write the state. There will be a video on that at some point
I really don't like having to list all derived types in the derived type. Sounds like it goes against the point. Similar to the "permits" keyword they added to I want to say Java, to allow exhaustive switches on interfaces by sealing that interface and only allowing specific classes to implement it.
Highly interesting video. What drawbacks would you say there are to event sourcing (besides the obvious increase in complexity)? Is it easy to make mistakes with? Do things that were previously ergonomic become cumbersome? Etc.
Hardest thing, IMO, is when a concept changes form (e.g. first name and last name need to be 2 separate fields, a new mandatory field is introduced,...) and older events (already persisted in production) are no longer "compatible" with the newer events. Introducing versioning from the start MIGHT help, but the problem then becomes, what if we change the concept's form regurarly. Do we really want to have 5-10 different versions of an event?
Great content as always. I wonde if the following scenario could be implemented with event sourcing. For some reason the requirements change in a way where changes could be submitted with effective date (in the past or future). Let's say in a HR software someone has salary 40k untill 1 June then it changes to 45k onward. Now the HR manager figures out they missed to do a salary increase on 1 March to 42k. Could this be done with event sourcing?
If HR failed to do a salary increase in March, then whatever system pays out salaries has already paid 40k instead of 42k to the employee for March, April and May. When we realize the error, the primary outcome that we want in the real world is to pay the balance of 2k for these months. I see one domain aggregate here: `Employee` with a field (among others) called `CurrentSalary` which can get updated when an event like `EmployeeSalaryIncreased` occurs. Another relevant event is `EmployeeSalaryPaid` with date and amount. Ideally, this event would originate from an automatic salary payment process that uses the `CurrentSalary` value of the Employee aggregate. As such, for a backdated payment, one could add a discriminator to the `EmployeeSalaryPaid` event like "Type": "Automatic", or "Manual". The events representing the extra 2k payments for March, April, and May will have "Manual" while a typical payment would be "Automatic". However, since the salary was never changed to 42k, the event stream for this employee would not reflect that error.
@@nickchapsas fair enough, but I had no idea what Event Sourcing was, so it felt like you just went straight into describing how to do make an inefficient database before eventually I figured out why it might be useful in some circumstances
It really doesn't. You can stream it asynchronously without affecting performance at all if you can afford some eventual consistency and even if you don't there are many ways to keep it efficient
Literally any persistence / materializing mechanism "may cost too much" for a "high load" scenario. Your assertion makes no sense since these things are relative to their specific use cases. Event soucing is not for everything, but it's certainly well established especially in the highest of demands such as banking and high frequence trading. If your takeaway from Event sourcing is that building an object from an event stream is slower than your EF Core SqlLite GetTodoList query, then I am doubtful you've really understood what event sourcing brings to the table.
@@WolfieVenturi in high frequence trading, e. g. instrument price change event contains full object state. More commonly the rates table will have a key of instrument id, stock id (trade board id) and datetime, that's all you'll need to set to get the rate. You won't need to examine several events to know the state at a given time.
First, thanks for this informative video (and many others). Maybe it was a little bit too much with all the additional DynamoDB content. What I hardly can imagine is how this works in a real world context where you have many different parent child relationships. For example a Salesorder with many SalesorderDetails with many Products and a Customer and CustomerAccout etc.
Good content. Have you looked at the Object Persistence Mode with the .NET DynamoDB API/SDK? It’d simplify much of the DynamoDB plumbing in this example.
I love this video, “event sourcing” has just been this buzzword I’ve heard of but not something I’ve encountered in my day job. I do prefer this over the sql server temporal tables as they don’t show actions that were performed, although you could write a complicated sproc to infer actions taken, but your example is definitely much simpler. You kind of lost me when you went into the DynamoDB part though. It looks sweet but I really struggled in your quick RUclips video approach where it’s a brain dump vs the slow and well presented way your Dometrain courses are done. Would love to see some more details around using these object databases and why you’d use dynamo over mongo or redis, even why you wouldn’t just use a sql database.
One question at ruclips.net/video/n_o-xuuVtmw/видео.html is the stream id when you have a student enrolled to a class. I would think that in a more complete example, Class would be an object of itself, and thus have an ID, what if I want to find all events related to a particular class? My question is basically, how do you handle such situations, where an event actually relates to more than one object, and thus ... maybe? ... should exist in more than one stream?
StudentUpdatedEvent is CRUD sourcing. As an example or introduction video i think that was a poor choice. Besides that nice video. Please cover EventStoreDB and Marten at some point.
EventStoreDB is terrible. It can't scale to save its life and I'm saying this as someone who's used it to its absolute limits and had to migrate away from it. Marten is ok but I prefer NoSQL dbs for event sourcing.
@nickchapsas What about a video of Event Sourcing with CosmosDb? The tricky part of event sourcing for me is to have a global ordering like EventStoreDB or not to, like CosmosDb per-partition-ordering guantantees
"You shouldn't be able to update your DOB" ... this doesn't work in the real world. So a student enrolls and the clerk enters their DOB wrong. Maybe their hand written DOB was misread. Now they have to create a whole new record to correct it?
This is append only nature of event sourcing. If you allow in place mutation of records then you cannot reconstitute the past if say a shutdown of the application happens. You keep the entire history of changes.
This is where a "Compensation action / event" comes in. Something to apply to correct a mistake or revert a downstream action that could not complete. Another scenario is booking a flight. One upstream service might handle seat allocation, and let you reserve a seat whilst you enter credit card details. This seat could be reserved for that time and it's last event is- SeatReserved, effectively holding the seat for you. But downstream the payment service, separate to booking, might reject your card, in which case that seat will have to be released again, so through some compensating action, the payment service would let the booking service know, hey this payment failed. Remember none of these things should be synchronous as a rule, unless you have a VERY good reason to make these things communicate synchronously. Now the booking service releases the seat again under an event, perhaps - SeatAllocationRevoked or SeatReservationDeallocated, something that captures the fact that the seat is available again maybe capturing the idea that it was due to a failed payment, depends on what's needed.
@shivanshusharma422 practice, if you are a developer, see if you're allowed to use it while working per your employer/customer. If so, use it at work, understand what it is good at and what it is not, learn to leverage that. If you're not allowed to use it while working or on their code base, experiment with it on personal projects, or where work allows. Just learn how to use it as efficiently as possible. That should give you an edge over those ho refuse to use/learn ai tools. A company I contract for has an on-site openai instance that we can use. It'd be great if thy exposed an api for us to use ai tools directly in visual studio or vscode, but alas, they have not YET.
I really dislike the PK/SK column/property names. Even Nick got confused and said "primary key" instead of "partition key" multiple times. PartitionKey/SortKey property names might be better.
I didn't want to go too in depth on the why, but pk and sk is pretty standard because of the size. Dynamo charges by the size of the document you read or write and being efficient with document size really makes a different at scale
@@nickchapsas Makes sense. Then I'd at least call the C# properties something else as they'll get converted to pk/sk thanks to the JsonPropertyNameAttribute.
Nick, love your content and I am almost always impressed with your decisions on coding style but I have to admit to cringing on this video. This is obviously my own personal opinion but using a reserved word as a class or variable name is too likely to lead to code readability issues especially with newer coders. Is "@event" really clearer than something like "studentEvent"? But great video!
The best way I've come to describe this concept is:
"Don't store the value. Instead, store the things you do to the value."
From there the questions start coming organically that I can answer as needed. The two main questions I hear are:
* What about the values needed for the operation?
- I create classes for the various operations (usually record types), serialize them, and store a timestamp of when it happened, the serialized version of the object, and the name of the operation.
* So what do I do with that information?
- The naive approach is to start at the beginning, iterate through each operation, and call the function associated with that operation. By the time you are done your service class or whatever it is will be in the same state you left it in when you closed the app.
From there I usually end up talking about snapshots and explain that. Like Nick said...it really isn't that hard and isn't as complicated as people think it is. Starting with the first thing I mentioned at the beginning helps the conversation a lot.
This is the content level I expect from Nick
As someone that's worked with accounting systems this is completely understandable. You never go back and change a value, even if it was in error. You can only add new transactions that modify the value . Everything is just a sequence of transactions that takes you from a starting balance to the current balance. (The starting balance is typically arrived at by closing the previous financial period where that prior set of transactions are either moved to history or, if the GAAP allows, discarded). We'd go about determining the value as of some point in time in a different way, but it's the same concept.
An in-memory similar concept is a Hughes list. You never modify the starting empty list, you only build up a sequence of lambdas or similar that each take an input list and produce a new list that is modified in some way. Chained together they hold the sequence of modifications you want to make to that initial empty list. When those lambdas are executed they produce the final list. But the original list and all the intermediate lists are immutable, so you could branch from any of those lists, using it as a starting point for an alternate list.
That's a very nice video you had put together sir!
Would be also nice to have a future video exploring some gotchas of the Event Sourcing world, specially the trade-offs one needs to consider when working with it, since is very easy to shoot yourself in the foot when attempting to use ES.
I'd love a video on how to handle versioning with the events. It's one of those things that I don't think I've ever really done well and would like alternative approaches to consider.
Having built a smaller cloud-native app in event sourcing, I can confirm this video captures the concept pretty well and makes it easy to understand.
Also, having built the aforementioned app in event sourcing, the architecture is quite annoying to work with. There are limitations you have to respect, issues you have to deal with and extra infrastructure to implement. It does go well with CQRS and together they make nice looking tests though.
This video was really interesting! I'd love to see more videos like this. I always enjoy your videos because I can always follow along with your explanations, even on concepts that are typically difficult to follow.
That's a great introduction. Would be cool in oncoming videos to go a little bit deeper down the rabbit hole of Event Sourcing.
You have really great content! Thank you!
You explained it so clearly! 🤩
It'd be awesome if you continued this as a series with more concepts.
Nice introductory video for Event Sourcing! This topic deserves a course on dometrain 😛
Just wanna say thanks for the great video. I know you have dometrain that you want to grow, so I'm just happy that this channel is still a place you put out valuable educational videos
Awesome video nick! would love to see more about event sourcing and the outbox pattern.
Would be interesting to see how bugs are handled and what should be done when the requirements and domain model evolve.
Wow how much effort he put into this video. Excellent job
Great vid, Event sourcing is quite a paradigm shift for a lot of people, rethinking of objects as a first derivative of an event stream is less familiar than the classic CRUD / SQL. It's definitely powerful in the right context. I'd definitely be hesistant to spear head event sourcing proposals in the workplace though, I sense it can be less forgiving if you approach it with the same mindsets to most crud systems of "oh we'll just do a migration / run some sql to solve it".
I watched this stoned so I'll have to watch again, but this was really great content Nick. I'd love to see more on this. Thank you.
Great Video Nick!! You were able to provide a vast glipmse of the matter with a simple examples!
Very nice video Nick, thanks for the effort that you put on it, looking forward for more content like this. Thanks
What database should I use for a small project expecting a low usage volume and with two different event domains? Should I use MongoDb, Cosmos or DynamoDb, or a database made for events such as EventStoreDb, or even use PostgreSql using Marten library?
I have some question. I'm implementing a system that have a object Tasks, this tasks need to have an historial of changes of user asignade, status, comments and modfications occurred to show at FrontEnd.
But too I need to filter this task by actual status, user asigned, priority, created date. How filter the all registers with Event Sourcing? it s not be to heavy for the system?
Event Sourcing is ideal to get the all events but i have dubts about his performance when i filter the tasks
Just want to mention the EventFlow C# library, which provides nice flexible DDD and CQRS structures handling most concurrency issues for an event sourcing system. Thank you for covering this topic Nick, would like to see more on the topic, most videos on the topic only touch the surface.
I recommend go take a look on the Microsoft Orleans actor framework, that is able to persist its actors, as well as it has event sourcing implemented.
Awesome video Nick, would be nice with more content like this.
Simple, Elegant, To the point video. 👍🏻
I needed some time to grasp what Eventsourcing is, but you introduction made it so easy to understand: It's same as mssql replication. Every transaction is replicated even the data that gets transferred is no more...
I needed this 3 months ago. Thanks Nick.
This is an absolutely fantastic video.
One thing I want to know, how would you do something like Get all Students with a Birthday in January, or all students enrolled in Maths.
What student is enrolled in the most cores?
Also, how would deleting work?
Same questions :)
Nice and simple explanation. I do however think you should use a number for the version. When running projections you don’t know if you have missed an event or not using datetime. If you were using integer you would directly see that something was wrong if you received version 6 before 5 for example.. using timestamp you can’t know if some event for a timestamp is missed.
Events are published and consumed given the code at their time.
If I try to process an old event with the today's code, it may fail.
How does event sourcing handle this issue? Please, don't tell me that we have to version events AND handlers, that would end up in a ball of mud
Yep, however i don't see this as a problem, is this any different from writing code-first EF migrations?
@@PhantomPhobos there's a huge difference: ef migrations are applied before the deployment of the application itself, within a transaction. If applied to prod, they are correct; on the contrary, we know code will always evolve
This seems like your data would be better stored in a Temporal table in SQL (if using SQL). You basically have a single "add" and then just updates to the row. The underlying DB creates a new row for each update. You can always get the "latest" record, but also the historical data. .NET EF Core handles it perfectly.
Sure, then you are creating a table for each object, if you want to add something to your aggregate you have to modify the schema, etc. Temporal tables aren't really event sourcing, more a method for auditing, and could possibly be a better use case in some places.
Very nice, there are scenarios I don't know how to manage:
1 - what if you need to wait to one entity is created to create another entity,
2 - If I have to create two related entities, and one have db generated autonumeric id that I need for the other entity?,
3 - If I have an API that creates an entity how to return a success or failed operation?
Fantastic topic and highly relevent to a current project of mine. Would love to see more please.
Is DateTime/Timestamp used for ordering of events in real-world applications? In case of updates that are atomic (take from one account and place on another) are then timestamp synchronized in some way?
You can reject an append if you try to write something that had something else written already during the time you were processing if that's what you mean
A system designed like this may lead to unexpected results in highly contested scenarios. Just because events order in a particular way after the fact, it does not mean that they were originally processed in that order.
Think of multiple instances of the application running simultaneously. -- some may be a little quicker, some may be a little slower, clocks drift, etc.
Using Conflict Free Resolution Data Types (CRDTs) would be one way to mitigate the issue.
Putting an actual Event Sourcing database or a persistent que as the source of truth in front of the system, instead of a general-purpose database, would be another way.
Is there a good place to read up on the problems and potential solutions?
@@SlackwareNVM You could start by reading the Leslie Lamport’s 1978 paper titled “Time, Clocks, and the Ordering of Events…”
Is there a reason at the 24:00 minute mark that you don't use interfaces, and have the classes implement the interfaces so there is no polymorphism, and thus fewer serialization issues?
Because with interfaces in System.JSON there would be a serialization issue by property omission
I tried once to develop an app that would be a game server from a Unity MMORPG could or should the event sourcing be implemented in such project, what is your opinion?
Extremely cool video! Looking forward for part 2
Great video (just as your presentation at Techorama Belgium ;-) )
One important note out of my own experience: the projection you're building is an single entity projection. If you're in need of an "overview" (typical "list" views in UIs e.g. a list of bookings, list of students, ... ) you're still in need of another projection with a different partition key since other document stores e.g. Azure Cosmos, ... don't like to query accross multiple partitions (high query cost). This can be done in various ways of course (e.g. listening to event feed as you showed to update the "overview" projection table). Or do you see still other approaches for the scenarion over having "overviews"?
That looks like it is destined to be used with the newtonsoft json serializer. Imagine all the polymorphic json attributes in a bigger project for that abstract event class... I would be in shambles!
EDIT: Yes, please more Event Sourcing!
I assume you could scan the assembly and do it automatically in some way.
Fantastic video. You should make a course on this topic. I'd be interested to learn more about it.
Why using "@" when defining Event @event as a param. Is it a convention ?
Because event is a C# keyword, so the @ tells the compiler to ignore that and use it as a parameter name.
Suggestion: It would be good to mention Discrete Event Systems and their fundament[s] Automata Theory. Those are different from Continious Systems described by differential equations and Discrete Systems described by difference equations.
Applications: Compilers (lexycal and semantical anaylsis), Manufacturing systems, Traffic/Transportation systems, Networking, Banking, Crypto, Supply Chain...
edit: added Crypto
Awesome. I would really like to hear more about how to handle versioning if the model changes. For example, if a field is added or removed, how should should earlier events be processed?
I once designed such a system. What I did is that I added to each event a version property and a dictionary of arbitrary objects used to store data fields of the event that corresponded to its API at the time the event was recorded.
When reading an event from the data layer corresponding to an earlier version of the event, instead of mapping the data fields to API members of the current version, I would instead store the data into the object map, using the original property names as keys.
After that, I would send the event through a chain of transformations meant to project the event into the current version. These are essentially a collection of strategy objects designed to change the state of the event from version T to T+1, moving around the contents of the object map to reflect what the event would have been like if it had been recorded at version T+1. This keeps up until the event transitions from the version at the time it was recorded until the current version, at which point the actual payload properties of the event would be set for the same event captured now. Nowadays, instead of the object map, we might consider using serialized JSON, but the idea is the same.
That being said, this adds a lot of overhead in terms of read processing, which is why this solution was also supplemented with optimized read models (see CQRS) to decouple the concerns. The idea is to playback past events to allow a subscriber to create a view optimized for its own needs. These read models were entirely disposable as they could easily be recreated from the event history at application start.
Great Video as usual Nick. I appreciate the idea was to keep it as simple as possible, however, I would contest your use of StudentUpdatedEvent, I would have expected a StudetNameChangedEvent, StudentEmailChangedEvent, StudentDoBChangedEvent, that way you are only storing the actual change not the whole object each time. If there were a lot of fields, storing the whole object each time could use a lot of space (something EV is often criticised for anyway). I appreciate that, ordinarily, the student isn't going to change that often so I'm probably being a little pedantic, but there could be scenarios where an instance (student in this case) may change more often. Would this not then lose the benefit of the audit trail that EV is supposed to give us?
Seems great, but I have a question: what happens if you have millions of users, and business comes with an idea that we should send a birthday greetings email to everybody who has a birthday in the current day. How would you approach this to even find the students meeting the criteria in a given day? Normally I would probably just search for them using some sql query, but here you don't store the current state anywhere, we have to build every student from the ground up which is going to be massive. Am I missing something?
I wonder if RavenDB could recreate the projection using stored procedure. It would make the saving into a trivial insertion. The only downside I can see is that Apply() should not depend on the object state (e.g. validating number of products based on storage capacity), should not contain business logic and might require full indepotency...
please continué with this topic ❤
This is the content I've been waiting for. How would you handle versioning of events and views - adding and removing properties, etc.?
Thanks for the video. I'd like to see how you handle complex events that modify multiple entities
Nice video! Can you elaborate more about the uses cases of Event sourcing. I have never seen event sourcing used in the real world. What types of applications are suitable for this approach ?
Finance, banking, e-commerce, ERPs, basically anything
I tried to create a lightweight event sourcing framework where aggregates are also their own inline projection. What I wanted to do is, use an sql db for storing aggregate as the traditional way, but adding its events to the events table with intercepting saveChanges of EfCore. The result? I can query the aggregate(actually its projection also, because its its own projection) using SQL and have an append only log of its state changes. Don't know if its applicable in real life scenarios
You can use CQRS to update "Read Database" every time you write events to "Write Database"
@@shahrukhkhan3967 thanks for your comment. Can you describe the update process more?
I our GDPR world, Is event sourcing even legal as a concept? Using a immutable log where someone has the right to be forgotten and data deleted seems a risky choice.
Yeah it is. There are solutions around GDPR
Encrypt each customer’s data with own key, store keys separately, when the customer asks to be forgotten - delete the key. HTH
At from 17:36 Please provide the video of eventual/consistency async/sync read/write heavy option please
That is a strange concept of rebuilding the state from scratch. In an accounting software I worked on there was a state calculated from some point of time. In simple case you just get the last state from the db, apply one event and save it back to the view instead of applying thousands of events from the beginnning
Audit trails is good use case for this pattern. For finances, you have a balance state which gets updated with every transaction.
There are obviously a range of optimizations to improve state management, also called snapshot patterns.
Nick only touches on the surface of event sourcing, and it’s a good introduction imo but you’d want to skill up significantly before moving to a more complete implementation.
Yes, long living entities with frequent updates are a performance nightmare. One solution would be to sort of archive the older parts of poor-performant streams.
Event sourcing is one of those things that sounds good on paper but in practice it rarely makes sense to go through the pain of implementing it.
Short lived entities are a key concept that is usually not stressed enough in entry level introductions to event sourcing. Or else... a lot of things can go off, performance-wise, even if you implemented it by the book.
Awesome Video!!
Keep pumping them out :)
any plans for your training courses to expand on similar topics as the AWS ones you've covered but in Azure equivalents?
Great video - amazing content. Thank you !
This is really helpful and to the point
thanks a lot
Question about naming - Nick used GetStudent and GetStudentView for projected Student. Why GetStudent is used for the entity that is created based on the events and not vice versa? When I see View name reminds me SQL views that materialise data from other source (because view does not have data itself) and in this case I would rather use GetStudentView for Student created from events instead. I believe it is event-source terminology but can someone explain this to me or refer me to a proper definition? Thanks!
I think I found answer, but terminology is quite confusing. Events are source-of-truth and this is main point of interest in Event Sourcing. The second approach to keep the most recent state in a projection is called view - to just access this information from a faster route (straight from db rather than going through all events). That means this projection is just a view (not the main source-of-truth) of the actual state we have in our events regardless if it is stored in database or not.
@@furrman18 I consider the projection part of the read model if you are using CQRS. The nice thing is you can add projections without changing any of the events to make your query side super fast, ie, one projection/view to display a certain thing in your UI.
Nice! Just curious, how would you deal with concurrency using DynamoDB in case two things load the same Student and both try to update something? I would assume that by default the last one to update would win.
It's a bit complex but possible, in a transaction, persist the record + the latest "version" of it somewhere conditioned to the version matching the expected
That's where the version thing comes in. You can use optimistic concurrency version checks to fail a write transactionally and re-read and re-write the state. There will be a video on that at some point
Versioning would solve it indeed, unfortunatelly at the expense of higher throughput.
how does one domain affect another ?
I really don't like having to list all derived types in the derived type. Sounds like it goes against the point.
Similar to the "permits" keyword they added to I want to say Java, to allow exhaustive switches on interfaces by sealing that interface and only allowing specific classes to implement it.
Thanks for the simple and intuitive video about the theme.
YES to more Event Sourcing!
Highly interesting video. What drawbacks would you say there are to event sourcing (besides the obvious increase in complexity)? Is it easy to make mistakes with? Do things that were previously ergonomic become cumbersome? Etc.
Hardest thing, IMO, is when a concept changes form (e.g. first name and last name need to be 2 separate fields, a new mandatory field is introduced,...) and older events (already persisted in production) are no longer "compatible" with the newer events.
Introducing versioning from the start MIGHT help, but the problem then becomes, what if we change the concept's form regurarly. Do we really want to have 5-10 different versions of an event?
Great content as always. I wonde if the following scenario could be implemented with event sourcing. For some reason the requirements change in a way where changes could be submitted with effective date (in the past or future). Let's say in a HR software someone has salary 40k untill 1 June then it changes to 45k onward. Now the HR manager figures out they missed to do a salary increase on 1 March to 42k. Could this be done with event sourcing?
If HR failed to do a salary increase in March, then whatever system pays out salaries has already paid 40k instead of 42k to the employee for March, April and May. When we realize the error, the primary outcome that we want in the real world is to pay the balance of 2k for these months.
I see one domain aggregate here: `Employee` with a field (among others) called `CurrentSalary` which can get updated when an event like `EmployeeSalaryIncreased` occurs.
Another relevant event is `EmployeeSalaryPaid` with date and amount. Ideally, this event would originate from an automatic salary payment process that uses the `CurrentSalary` value of the Employee aggregate.
As such, for a backdated payment, one could add a discriminator to the `EmployeeSalaryPaid` event like "Type": "Automatic", or "Manual". The events representing the extra 2k payments for March, April, and May will have "Manual" while a typical payment would be "Automatic".
However, since the salary was never changed to 42k, the event stream for this employee would not reflect that error.
So event sourcing is just command pattern?
Nop
I think it would have been good to start with why you would want to do this...
There are many videos that talk about that. I am not interested in teaching theory
@@nickchapsas fair enough, but I had no idea what Event Sourcing was, so it felt like you just went straight into describing how to do make an inefficient database before eventually I figured out why it might be useful in some circumstances
@@ghosttie There is nothing inefficient about the database in event sourcing. Storage is cheap
Wow. Great video. Many thanks.
Great video! Thank you
event sourcing is doubtful, but okay =) repro of an object state through events may cost too much in high loaded systems
It really doesn't. You can stream it asynchronously without affecting performance at all if you can afford some eventual consistency and even if you don't there are many ways to keep it efficient
Literally any persistence / materializing mechanism "may cost too much" for a "high load" scenario. Your assertion makes no sense since these things are relative to their specific use cases.
Event soucing is not for everything, but it's certainly well established especially in the highest of demands such as banking and high frequence trading. If your takeaway from Event sourcing is that building an object from an event stream is slower than your EF Core SqlLite GetTodoList query, then I am doubtful you've really understood what event sourcing brings to the table.
@@WolfieVenturi in high frequence trading, e. g. instrument price change event contains full object state. More commonly the rates table will have a key of instrument id, stock id (trade board id) and datetime, that's all you'll need to set to get the rate. You won't need to examine several events to know the state at a given time.
First, thanks for this informative video (and many others). Maybe it was a little bit too much with all the additional DynamoDB content. What I hardly can imagine is how this works in a real world context where you have many different parent child relationships. For example a Salesorder with many SalesorderDetails with many Products and a Customer and CustomerAccout etc.
Hi, first of great video.
Wondering how it will work with entities that have birectional relationships.
Direct to the point 👌
So, can we expect a course on DomeTrain for this? Would be awesome!
There are two event sourcing courses coming this summer
@@nickchapsas love it!
Thanks for that topic, but Nick... Single Responsibility Principle?
Good content. Have you looked at the Object Persistence Mode with the .NET DynamoDB API/SDK? It’d simplify much of the DynamoDB plumbing in this example.
I love this video, “event sourcing” has just been this buzzword I’ve heard of but not something I’ve encountered in my day job. I do prefer this over the sql server temporal tables as they don’t show actions that were performed, although you could write a complicated sproc to infer actions taken, but your example is definitely much simpler. You kind of lost me when you went into the DynamoDB part though. It looks sweet but I really struggled in your quick RUclips video approach where it’s a brain dump vs the slow and well presented way your Dometrain courses are done. Would love to see some more details around using these object databases and why you’d use dynamo over mongo or redis, even why you wouldn’t just use a sql database.
One question at ruclips.net/video/n_o-xuuVtmw/видео.html is the stream id when you have a student enrolled to a class. I would think that in a more complete example, Class would be an object of itself, and thus have an ID, what if I want to find all events related to a particular class?
My question is basically, how do you handle such situations, where an event actually relates to more than one object, and thus ... maybe? ... should exist in more than one stream?
StudentUpdatedEvent is CRUD sourcing. As an example or introduction video i think that was a poor choice. Besides that nice video. Please cover EventStoreDB and Marten at some point.
EventStoreDB is terrible. It can't scale to save its life and I'm saying this as someone who's used it to its absolute limits and had to migrate away from it. Marten is ok but I prefer NoSQL dbs for event sourcing.
@@nickchapsas that is actually valuable information. Thanks.
@@nickchapsas To be fair, wouldn't Postgres qualify already as NoSql database?
@nickchapsas What about a video of Event Sourcing with CosmosDb? The tricky part of event sourcing for me is to have a global ordering like EventStoreDB or not to, like CosmosDb per-partition-ordering guantantees
@@nickchapsas Hey Nick, would you mind extending a bit on how EventStoreDB wasn't able to scale for usage? Thanks for your time
What a great topic!
"You shouldn't be able to update your DOB" ... this doesn't work in the real world.
So a student enrolls and the clerk enters their DOB wrong. Maybe their hand written DOB was misread.
Now they have to create a whole new record to correct it?
its easy, make an event `DateOfBirthRedacted` and go on
This is append only nature of event sourcing. If you allow in place mutation of records then you cannot reconstitute the past if say a shutdown of the application happens. You keep the entire history of changes.
Yeap. That's how event sourcing works. you never update or delete. It works very well in the real world
This is where a "Compensation action / event" comes in. Something to apply to correct a mistake or revert a downstream action that could not complete. Another scenario is booking a flight. One upstream service might handle seat allocation, and let you reserve a seat whilst you enter credit card details. This seat could be reserved for that time and it's last event is- SeatReserved, effectively holding the seat for you. But downstream the payment service, separate to booking, might reject your card, in which case that seat will have to be released again, so through some compensating action, the payment service would let the booking service know, hey this payment failed. Remember none of these things should be synchronous as a rule, unless you have a VERY good reason to make these things communicate synchronously.
Now the booking service releases the seat again under an event, perhaps - SeatAllocationRevoked or SeatReservationDeallocated, something that captures the fact that the seat is available again maybe capturing the idea that it was due to a failed payment, depends on what's needed.
DateTimeOffset instead of DateTime no?
Hi, I'm a dotnet developer and would you advice me how can i cope up with this AI wave and also how should shape up my future to prepared for it?
Use it, learn it, stay up to date with it
@@ferd1775 thanks and i wanna know and learn how to use it best as developer. What's the key?
@shivanshusharma422 practice, if you are a developer, see if you're allowed to use it while working per your employer/customer. If so, use it at work, understand what it is good at and what it is not, learn to leverage that.
If you're not allowed to use it while working or on their code base, experiment with it on personal projects, or where work allows. Just learn how to use it as efficiently as possible. That should give you an edge over those ho refuse to use/learn ai tools.
A company I contract for has an on-site openai instance that we can use. It'd be great if thy exposed an api for us to use ai tools directly in visual studio or vscode, but alas, they have not YET.
wo Nick! you play guitar! cool! do you have any video you playing?
Awesome content.
+1 for more of this
Nice content bro, following you since 2015, you abandoned Java xD
You should do a video on Microsoft SQL temporal tables.
Sweet!!..Thanks
How easy is that? :)
Great topic, try to have some advanced content
I really dislike the PK/SK column/property names. Even Nick got confused and said "primary key" instead of "partition key" multiple times. PartitionKey/SortKey property names might be better.
I didn't want to go too in depth on the why, but pk and sk is pretty standard because of the size. Dynamo charges by the size of the document you read or write and being efficient with document size really makes a different at scale
@@nickchapsas Makes sense. Then I'd at least call the C# properties something else as they'll get converted to pk/sk thanks to the JsonPropertyNameAttribute.
Try Eventuous framework
Hello EviBaaadee!
I seem to have come away from this video with a lot more of the what and less of the why. Something something auditing ... never delete anything.
There are many videos talking about the why. There are very few that break down how in simple terms
Did someone say Event Sourcing? 👀
Nice
Nick, love your content and I am almost always impressed with your decisions on coding style but I have to admit to cringing on this video. This is obviously my own personal opinion but using a reserved word as a class or variable name is too likely to lead to code readability issues especially with newer coders. Is "@event" really clearer than something like "studentEvent"?
But great video!
Give me a better name for type Event that represents an event in its abstract form
please sir, can I have more? :D
First nakh!
Tumben bang bahas beginian
Visitor pattern would be a great way to remove the need of a switch for every event.
Or strategy or a million other approaches. The video doesn't need this because it would take away from the main focus which is the event sourcing part
Switches are a clean replacement for the visitor pattern.
@@nickchapsas true was the part of my brain that likes to over engineer stuff.