Have you ever tried doing some code review? I always find your videos so insightful. I believe it would be great to see your thought process on some real world examples.
Nice video! Fully agreed. My takeaways and thoughts: - use DTOs to reduce coupling - if there's no coupling you can reduce (e.g. 1-1 db to model) then get rid of unnecessary DTOs - think about team's/module's (DDD bounded context) responsibilities: a DTO within your boundaries might be useless (pure overhead). The coupling that does make you suffer is cross-module/cross-team/etc. coupling. - think about the higher-level dependencies that you might want to loosen - and not your internal code dependencies.
This is a typical example of why CQRS is a good idea! For the read side, I sometimes even just run database queries straight inside the API endpoint. No fuss needed. Just grab the data you need and map it to a format that you want.
Another very important reason is the performance. I've seen many projects and companies who read the whole entity (a Product for example) from the database, and when you go forward you see they only wanted the Description from that entity. So, yes, DTO's are very important. To make it clear for everybody here, there are 2 types of APIs: 1) Microservice API (API-to-API call) 2) BFF API (BackEnd for FrontEnd) You may not need a DTO for the first type, but for the second type it's 100% important.
Yes, 130 database models modified directly from the frontend. It is the horror project I joined last time. It was not a problem until new requirements appeared to integrate automatic synchronisation from the external 3rd party services. All protections were implemented as validators for the HTTP layer, so not existing domain layer couldn't protect consistency of data coming from external services launched by queues or CRON, because it does't use HTTP validators. It is real nightmare now. Adding domain layer is extremely time-consuming and there is required some strategy to apply it correctly, small piece after another. Work for a few next years. More over implicit syntax sugars minimised HTTP controllers to zero lines of the code giving automatic access to the database-model directly from the HTTP request. So there is no domain and controller layer you can explicitly customise.
I Like Immutable DTOs to act like diodes in software, making it much harder to create cycles in systems. Like oh I have 10000 records, and oh now I'll iterate over 10,000 of them to change one field, and then oh I'll just do it again to change another field. If the data is immutable and sufficiently hard to change, then as well as an outward (not necesarrily public) contract, they also enforce directionality, and encapsulate behaviour. Past this point you can't just save this record without coming back in from a less privileged state. Great video!
I think the problems stem from developers being bad at abstraction and separating the contexts. A words might mean different things depending on the context and use. That is why these videos are so useful, because they, you teach, teach developers that it is not just about the code itself, but what the code represents, what it means, and how to express that. Not just for yourself but for others through code. A program is more than its sum of methods/functions.
I agree for the most part, but wanted to add that good/bad abstractions and context separation is not just a 'skill issue' from developers or communication. As the product and its requirements change over time, something that was 'good' may become 'bad'. The difficulty lies in identifying this early, dealing with biases/opinions (we always did it this way), prioritizing a refactor, sharing knowledge, etc. For most codebases I've worked in, the developers knew an abstraction was bad, but had to deal with it for a reason.
Worst mapping experience I ever came across was a REST API mapping requests and payloads to service layer DTO's, which then were passed along to the domain, which was then mapped to ORM objects, which were used to store into a RDBMS. And of course upon successful persisting, that "saved ORM" was then mapped back into the domain, which was then mapped back to a DTO broadcasted from the service layer, which was then mapped into a serializable form (JSON) to return to the frontend client. We had driving adapter (the rest api) and one driven adapter (the jpa layer). But you know, "what if" the business wanted to also accept queue or drive an integration event stream? At least all that indirection was nice to test the domain in isolation. And it was a great project for learning. "But at what cost?"
@@TimSchraepen Question is if you want to have fun or if you want to produce resilient software that serves business value and doesnt break or gets unmaintainable.
I just return the model, that runs through a filter and removes sensitive data :D I know its bad, but man, I am not willing to create hundreds of dtos.
I don't think it's "bad" per se. At all. It's one of the many trade-offs you'll have to evaluate and - if it provides more benefits than drawbacks - accept into a typical application.
@@Kasper-mu4ov depends, mostly yes, but in somecases i delete them entirely. using reflection to generate the graphql types, which are the dtos, and mostly they are the same
I've only really found DTOs useful once, and it was when the particular nosql storage I was using was annoying to work with so we wanted to keep all of that logic hidden from the rest of the application. For example writing one record actually required writing it in many different ways by different keys, because there were no secondary indexes, but the logic in the app just wants to write it once, not have to worry about partition keys and row keys or denormalization or table scans vs point reads or etags etc. It also allowed us to update the client library we were using without causing breaking changes in the rest of the application, and eventually we got sick of it and switched databases entirely, and this made that a bit easier to do.
I love these talks, but often find them hard to put into practice. Applying these context-tailored practices requires understanding the 'why pattern', not just 'what pattern'. The difficulty is to stimulate that mentality with concerns like time, money, and personal interest from my position as developer. If only we could do away with the DTOs of a youtube video and my attempts to convey the contents, and absorb the concepts directly 😅
I haven’t had much problems with a 1:1 mapping for DTOs It’s a very thin mapping function anyway. Updating the mapping doesn’t take much work. Accidentally leaking data sounds more scary.
I mostly share your opinion. Converting to and from Dto isn't too complex, provided you don't use automapper. And they allow a stricter typing system imo. Let's take for example EF Core here: I return an entity from my service layer instead of a Dto. That entity has a navigation property to allow us to fetch FK entities. Whatever's receiving that entity needs to be mindful of what's been loaded in context through .Include() in the service method, otherwise it might try to query the navigational property and get erroneous results or exceptions. Or I can provide a well-defined api to my service that returns exactly what I need, and if I need more data, then I add a new dto. It even helps me fine-tune my queries a little better by limiting over-querying
With EF and contexts it's much more important. Ask that change teaching being shared throughout layers is expensive and not necessary. 1-1 making solves this by turning your entity into a dto. Context.Data.Select() Couldn't be much easier
Hey Derek, great video, love it! I have one question. I’ve been having an argument with some developer folks: does CQRS really need to be physically two different services, or can I have one service with two handlers separating the command and query handling? There’s a lot of cohesiveness, and the data models are mostly identical for command and query. Also, within the same service, rather than having communication through events, can we avoid event consistency? In my opinion, we should avoid it most of the time rather than embrace it. What do you think? Could this be a topic for your next video? Haha.
1) There’s no strict requirement for CQRS to involve physically separate services. 2) This is why many developers treat command and query handlers as "interactors" rather than rigidly following CQRS as a separate architectural concept. It's primarily about separating responsibilities and documentation. One scenario where command interfaces might differ significantly is in orchestrational sagas, where commands may include compensational logic. 3) When working within a single service, there’s no need to introduce eventual consistency. Reads and writes should typically operate on the same database (the "write DB"), and query handlers are usually designed to serve external consumers, such as users.
They don't need to be two different "services". It's just two different code paths for handling a command or a query. It's a step farther than CQS, which is a method/function should perform an action or return data. Not both. CQRS is just taking it higher up.
What about the fact that entities are generally mutable vs. dtos are not? So with that thinking, you would almost always want to pass a dto to your view instead of the actual entity.
6:13 "when you dont have that problem yet", right.. but DTOs are part of a layered architecture that you use because you're trying to think ahead, the bigger picture. They're also about creating a visible layer and pattern that your colleagues easily can follow. You never need to debate if "something should be a DTO or not" because the layered architecture dictates that everything should be a DTO.
Do you suggest actually combining information from a contact table to an entity inside repository code? If it's a single sql or ORM request, that's probably fine, but when it involves multiple requests, this would require more integration tests on the repository and sometimes greatly complicate repository code (maybe even with logic -- which is probably a no-no...) also just having extra fields in the data class to "fill in" in other layers would make it very hard to track, who's adding what and where... all this would probably usually be a candidate for DTOs...
I am doing this data model wrapping mainly because my data model is auto generated and is entirely tied down to the SQL library I'm using (JOOQ in my case). And fields that are typed as UUID for example, or JSONB gets properly exposed using a typed Id and so on. I am also in some cases combining multiple database records to a single "entity". I'm also not serializing or deserializing, I'm just wrapping the database record with a class that implements my interfaces. It has benefits when it comes to enforcing audit fields or multi tenancy and more. This might be unnecessary with some frameworks but I would hate to have all my code depend on the structure of the auto generated code directly.
Sorry mate but I think this is bad advice. For example the object you POST doesn’t want to contain the properties - Id, CreatedBy, CreatedDate as these want to be populated by the service. But you do want these on the GET response object.
@@coloresfelices7 my comments are brief for readability and are valid. The advice in the video is generalised, but I feel there’s only really a subset of use cases where they would constitute good advice. Almost always you have an identifier, right?
I get the part of using DTO to composing data and managing coupling. But in your sample code the Shipment logical boundary seems to have data for the Client, which means I can expose the data together with the Shipment. Otherwise, if I just have the reference ID inside Shipment boundary I can't do this composition, am I right? Therefore we end up with almost the same model as the beggining (the CRUD one you showed in the video). But additionally we could put the actions and so on (HATEOAS) and we can only do the composition in the API gateway or even in the Client side
The query and command objects are just DTOs. You can't change my mind. 1 to 1 mapping DTOs still serve a function. When you add that credentials field to the entity you don't accidentally return it to the outside world without realizing
I’ll be honest every time I write a DTO these days it’s to not refactor some other code. Either some api or entity is structured in someone that was perfectly fine for one use case but now maybe needs to be reused but in a situation it wasn’t designed for. Probably an anti pattern but I’ve used dtos a lot because of it and honestly I’d even say it’s the wrong way
First reason i use a dto is when i need to return a complex object and i want to get rid of all the child parent relations you usually have on a dataentity model. they have backreferences so they get circular. that does not work well with serialization as an object you can return in an api. An other horror story is automapper. i needs complex code (that is hard to follow using f12, you need to search) for basically saying target.a=source.a. in my project i have 2 simple ways of mapping to a dto. when they have the same properties i just serialize and deserialize using a helper methodof type T. for all specials i just code them out like target.a= source.b.
Exactly. I'm having a project with a similar situation. Overneglecting over tech debt went to the situation, where the data in the database are making changes in the UI. You need to put spaces/comas in the database, to have a correct dropdown with values. Just because someone cut corners in the beginning.
First what is your "API". I'm going to assume you mean your HTTP API? If so, if your doing some type of composition and building an output, how would it "just show up"? I never suggested exposing your database to your HTTP API, I'm suggesting not using a DTO throughout layers if you don't NEED to and you can couple to them. I also stated to be aware if its public or private. Your HTTP API is public (as in contract).
When i want to pass data from HTTP request to the handler ( controller, usecase) , i am using a DTO to encapsulate those parameters with a class. Is that a good use of DTO ?
I am not going to answer your question but just want to point something out. There are always just pros and cons of every decision. I don’t like to call it good or bad because sometimes bad can be good depending on the context and vice versa. In general I think it is a problem within development world that we try to label things as good or bad instead of knowing pros and cons.
@@Bosslogq Here i am talking about sepcific case and clear one i think, you have a REST Api and you are getting data from HTTP world and you want to make your system aware about the data he is dealing with, as easy as that !!!
It's good if you have other clients that talk to the same controller/usecase. By using DTO's you'd be making these part of the API of your controller/usecase. These could be Query or Command objects if you're also using CQRS. If you only have one client (the http endpoints), then maybe those DTO's can be replaced with either just function calls, or in the other direction, domain interfaces. You gotta watch out though that your domain doesn't take on the shape of your http endpoints.
I generally convert an HTTP Request to a DTO which i really term as an application level request. I'm removing any HTTP specifics from it. I also do the exact same for messages on a queue. That way what processes my application requests is agnostic of the top level I/O.
Have you ever tried doing some code review? I always find your videos so insightful. I believe it would be great to see your thought process on some real world examples.
I've done a few various types of reviews and they do seem to get a good response. Thanks for the suggestion, I'll do some more.
Nice video! Fully agreed.
My takeaways and thoughts:
- use DTOs to reduce coupling
- if there's no coupling you can reduce (e.g. 1-1 db to model) then get rid of unnecessary DTOs
- think about team's/module's (DDD bounded context) responsibilities: a DTO within your boundaries might be useless (pure overhead). The coupling that does make you suffer is cross-module/cross-team/etc. coupling.
- think about the higher-level dependencies that you might want to loosen - and not your internal code dependencies.
This is a typical example of why CQRS is a good idea! For the read side, I sometimes even just run database queries straight inside the API endpoint. No fuss needed. Just grab the data you need and map it to a format that you want.
Oops. Should've watched it all. CQRS comes at 6:50 😅
It was my first thought a minute into the video haha
I really enjoy your no-nonsense and undogmatic approach to programming.
Thanks, I try!
Another very important reason is the performance. I've seen many projects and companies who read the whole entity (a Product for example) from the database, and when you go forward you see they only wanted the Description from that entity. So, yes, DTO's are very important.
To make it clear for everybody here, there are 2 types of APIs:
1) Microservice API (API-to-API call)
2) BFF API (BackEnd for FrontEnd)
You may not need a DTO for the first type, but for the second type it's 100% important.
Yes, 130 database models modified directly from the frontend. It is the horror project I joined last time. It was not a problem until new requirements appeared to integrate automatic synchronisation from the external 3rd party services. All protections were implemented as validators for the HTTP layer, so not existing domain layer couldn't protect consistency of data coming from external services launched by queues or CRON, because it does't use HTTP validators. It is real nightmare now. Adding domain layer is extremely time-consuming and there is required some strategy to apply it correctly, small piece after another. Work for a few next years. More over implicit syntax sugars minimised HTTP controllers to zero lines of the code giving automatic access to the database-model directly from the HTTP request. So there is no domain and controller layer you can explicitly customise.
I Like Immutable DTOs to act like diodes in software, making it much harder to create cycles in systems. Like oh I have 10000 records, and oh now I'll iterate over 10,000 of them to change one field, and then oh I'll just do it again to change another field. If the data is immutable and sufficiently hard to change, then as well as an outward (not necesarrily public) contract, they also enforce directionality, and encapsulate behaviour. Past this point you can't just save this record without coming back in from a less privileged state.
Great video!
I think the problems stem from developers being bad at abstraction and separating the contexts. A words might mean different things depending on the context and use. That is why these videos are so useful, because they, you teach, teach developers that it is not just about the code itself, but what the code represents, what it means, and how to express that. Not just for yourself but for others through code.
A program is more than its sum of methods/functions.
I agree for the most part, but wanted to add that good/bad abstractions and context separation is not just a 'skill issue' from developers or communication. As the product and its requirements change over time, something that was 'good' may become 'bad'.
The difficulty lies in identifying this early, dealing with biases/opinions (we always did it this way), prioritizing a refactor, sharing knowledge, etc.
For most codebases I've worked in, the developers knew an abstraction was bad, but had to deal with it for a reason.
Worst mapping experience I ever came across was a REST API mapping requests and payloads to service layer DTO's, which then were passed along to the domain, which was then mapped to ORM objects, which were used to store into a RDBMS. And of course upon successful persisting, that "saved ORM" was then mapped back into the domain, which was then mapped back to a DTO broadcasted from the service layer, which was then mapped into a serializable form (JSON) to return to the frontend client.
We had driving adapter (the rest api) and one driven adapter (the jpa layer). But you know, "what if" the business wanted to also accept queue or drive an integration event stream?
At least all that indirection was nice to test the domain in isolation. And it was a great project for learning. "But at what cost?"
It did also help provide our "junior developers' with guard rails because of its consistency. But it wasn't very fun to work in.
@@TimSchraepen Question is if you want to have fun or if you want to produce resilient software that serves business value and doesnt break or gets unmaintainable.
@ those are not mutually exclusive
I just return the model, that runs through a filter and removes sensitive data :D I know its bad, but man, I am not willing to create hundreds of dtos.
I don't think it's "bad" per se. At all. It's one of the many trade-offs you'll have to evaluate and - if it provides more benefits than drawbacks - accept into a typical application.
In a static language? You mean you set the fields you dont want to null?
@@Kasper-mu4ov depends, mostly yes, but in somecases i delete them entirely. using reflection to generate the graphql types, which are the dtos, and mostly they are the same
I've only really found DTOs useful once, and it was when the particular nosql storage I was using was annoying to work with so we wanted to keep all of that logic hidden from the rest of the application.
For example writing one record actually required writing it in many different ways by different keys, because there were no secondary indexes, but the logic in the app just wants to write it once, not have to worry about partition keys and row keys or denormalization or table scans vs point reads or etags etc.
It also allowed us to update the client library we were using without causing breaking changes in the rest of the application, and eventually we got sick of it and switched databases entirely, and this made that a bit easier to do.
I love these talks, but often find them hard to put into practice. Applying these context-tailored practices requires understanding the 'why pattern', not just 'what pattern'. The difficulty is to stimulate that mentality with concerns like time, money, and personal interest from my position as developer.
If only we could do away with the DTOs of a youtube video and my attempts to convey the contents, and absorb the concepts directly 😅
I haven’t had much problems with a 1:1 mapping for DTOs
It’s a very thin mapping function anyway. Updating the mapping doesn’t take much work. Accidentally leaking data sounds more scary.
I mostly share your opinion. Converting to and from Dto isn't too complex, provided you don't use automapper. And they allow a stricter typing system imo. Let's take for example EF Core here:
I return an entity from my service layer instead of a Dto. That entity has a navigation property to allow us to fetch FK entities. Whatever's receiving that entity needs to be mindful of what's been loaded in context through .Include() in the service method, otherwise it might try to query the navigational property and get erroneous results or exceptions.
Or I can provide a well-defined api to my service that returns exactly what I need, and if I need more data, then I add a new dto. It even helps me fine-tune my queries a little better by limiting over-querying
With EF and contexts it's much more important. Ask that change teaching being shared throughout layers is expensive and not necessary. 1-1 making solves this by turning your entity into a dto.
Context.Data.Select()
Couldn't be much easier
add to that, that passing ORM entity can result in unwanted modification to database if not done and configured probably.
@@desmaraisp2894 Haven't used EF Core before, but in CQRS you would just query directly from the db for the DTO. Why return any entity?
This DTO response looks like HATEOAS
It is!
Most de-coupling architectures are built on the principles of just-in-case and scalability
Hey Derek, great video, love it!
I have one question. I’ve been having an argument with some developer folks: does CQRS really need to be physically two different services, or can I have one service with two handlers separating the command and query handling? There’s a lot of cohesiveness, and the data models are mostly identical for command and query.
Also, within the same service, rather than having communication through events, can we avoid event consistency? In my opinion, we should avoid it most of the time rather than embrace it.
What do you think? Could this be a topic for your next video? Haha.
1) There’s no strict requirement for CQRS to involve physically separate services.
2) This is why many developers treat command and query handlers as "interactors" rather than rigidly following CQRS as a separate architectural concept. It's primarily about separating responsibilities and documentation. One scenario where command interfaces might differ significantly is in orchestrational sagas, where commands may include compensational logic.
3) When working within a single service, there’s no need to introduce eventual consistency. Reads and writes should typically operate on the same database (the "write DB"), and query handlers are usually designed to serve external consumers, such as users.
CQRS is a logical construct. It does not care how it's deployed.
They don't need to be two different "services". It's just two different code paths for handling a command or a query. It's a step farther than CQS, which is a method/function should perform an action or return data. Not both. CQRS is just taking it higher up.
What about the fact that entities are generally mutable vs. dtos are not? So with that thinking, you would almost always want to pass a dto to your view instead of the actual entity.
6:13 "when you dont have that problem yet", right.. but DTOs are part of a layered architecture that you use because you're trying to think ahead, the bigger picture. They're also about creating a visible layer and pattern that your colleagues easily can follow. You never need to debate if "something should be a DTO or not" because the layered architecture dictates that everything should be a DTO.
Do you suggest actually combining information from a contact table to an entity inside repository code? If it's a single sql or ORM request, that's probably fine, but when it involves multiple requests, this would require more integration tests on the repository and sometimes greatly complicate repository code (maybe even with logic -- which is probably a no-no...) also just having extra fields in the data class to "fill in" in other layers would make it very hard to track, who's adding what and where... all this would probably usually be a candidate for DTOs...
I am doing this data model wrapping mainly because my data model is auto generated and is entirely tied down to the SQL library I'm using (JOOQ in my case). And fields that are typed as UUID for example, or JSONB gets properly exposed using a typed Id and so on. I am also in some cases combining multiple database records to a single "entity". I'm also not serializing or deserializing, I'm just wrapping the database record with a class that implements my interfaces. It has benefits when it comes to enforcing audit fields or multi tenancy and more. This might be unnecessary with some frameworks but I would hate to have all my code depend on the structure of the auto generated code directly.
Sorry mate but I think this is bad advice. For example the object you POST doesn’t want to contain the properties - Id, CreatedBy, CreatedDate as these want to be populated by the service. But you do want these on the GET response object.
Not including those properties is just an implementation detail of your application, so don't generalize.
@@coloresfelices7 my comments are brief for readability and are valid. The advice in the video is generalised, but I feel there’s only really a subset of use cases where they would constitute good advice.
Almost always you have an identifier, right?
I get the part of using DTO to composing data and managing coupling.
But in your sample code the Shipment logical boundary seems to have data for the Client, which means I can expose the data together with the Shipment. Otherwise, if I just have the reference ID inside Shipment boundary I can't do this composition, am I right?
Therefore we end up with almost the same model as the beggining (the CRUD one you showed in the video). But additionally we could put the actions and so on (HATEOAS) and we can only do the composition in the API gateway or even in the Client side
The query and command objects are just DTOs. You can't change my mind. 1 to 1 mapping DTOs still serve a function. When you add that credentials field to the entity you don't accidentally return it to the outside world without realizing
I’ll be honest every time I write a DTO these days it’s to not refactor some other code. Either some api or entity is structured in someone that was perfectly fine for one use case but now maybe needs to be reused but in a situation it wasn’t designed for. Probably an anti pattern but I’ve used dtos a lot because of it and honestly I’d even say it’s the wrong way
First reason i use a dto is when i need to return a complex object and i want to get rid of all the child parent relations you usually have on a dataentity model. they have backreferences so they get circular. that does not work well with serialization as an object you can return in an api. An other horror story is automapper. i needs complex code (that is hard to follow using f12, you need to search) for basically saying target.a=source.a. in my project i have 2 simple ways of mapping to a dto. when they have the same properties i just serialize and deserialize using a helper methodof type T. for all specials i just code them out like target.a= source.b.
9:23 Yes you kinda expose your DB model through GraphQL... At least the project I'm currently working on does that.
Exactly. I'm having a project with a similar situation. Overneglecting over tech debt went to the situation, where the data in the database are making changes in the UI. You need to put spaces/comas in the database, to have a correct dropdown with values. Just because someone cut corners in the beginning.
Thank you, very interesting.
very interesting thanks
Uh no. So I add a new column to my db it should just show up in my api? No man. And mapping is not an issue anymore with AI or other tools.
I had a customer that wanted exactly that. The DB admin wanted to add a col that should immediately show up in the frontend.
@@GnomeEU json or nosql would be much better
First what is your "API". I'm going to assume you mean your HTTP API? If so, if your doing some type of composition and building an output, how would it "just show up"? I never suggested exposing your database to your HTTP API, I'm suggesting not using a DTO throughout layers if you don't NEED to and you can couple to them. I also stated to be aware if its public or private. Your HTTP API is public (as in contract).
When i want to pass data from HTTP request to the handler ( controller, usecase) , i am using a DTO to encapsulate those parameters with a class.
Is that a good use of DTO ?
I am not going to answer your question but just want to point something out. There are always just pros and cons of every decision. I don’t like to call it good or bad because sometimes bad can be good depending on the context and vice versa. In general I think it is a problem within development world that we try to label things as good or bad instead of knowing pros and cons.
@@Bosslogq Here i am talking about sepcific case and clear one i think, you have a REST Api and you are getting data from HTTP world and you want to make your system aware about the data he is dealing with, as easy as that !!!
It's good if you have other clients that talk to the same controller/usecase. By using DTO's you'd be making these part of the API of your controller/usecase. These could be Query or Command objects if you're also using CQRS.
If you only have one client (the http endpoints), then maybe those DTO's can be replaced with either just function calls, or in the other direction, domain interfaces. You gotta watch out though that your domain doesn't take on the shape of your http endpoints.
@Bosslogq Bingo!
I generally convert an HTTP Request to a DTO which i really term as an application level request. I'm removing any HTTP specifics from it. I also do the exact same for messages on a queue. That way what processes my application requests is agnostic of the top level I/O.