"My general rule of thumb: don’t violate DRY within a microservice, but be relaxed about violating DRY across all services. The evils of too much coupling between services are far worse than the problems caused by code duplication." -- Sam Newman in "Building Microservices"
Sometimes, when a function is very general and will most likely not need module specific changes, it would make sense to share it between multiple modules.
@@porky1118 I guess you mean general == technical. Of course, you o not want to duplicate technical framework/support code. But if you happen to duplicate business code in several services, you most likely screwed-up you overall microservices design.
@@zerettino No, general is not technical. It means it's not special purpose. It has many use cases. It can be used in many circumstances. If you recognize some pattern in many of your modules, which could be generalized, and will still work if the modules change, it makes sense to put them into a module used by both. Even if it will cause a lot of coupling, that's not a huge problem. You can still refactor by just copying the affected function(s). So it shouldn't be too bad.
Until you have to change the same code in hundreds of microservices. But I like the idea of creating a library independently and each microservice upgrading or not based on the version number. In my first IT job (in the noughties) I had to copy a lot of code between SQL2000 DTSs and it was a pain to change the same code and remember which package used it between hundreds of DTSs packages.
The subject is very important. In our company we've resolved the problem with templates. When we need a new microservice that needs common stuff like logging, metrics etc. we create it from a template. So when we change something common we change the template, and all next services will use the new one, with new features (and bugs), but all existing services aren't forced to be changed.
we solve it by putting common stuff in libraries and name the explicit version. So if something new in the library is necessary for a service, there is at least a conscious decision.
@@ichduersiees8374 yep I think libraries with specific versions is the correct approach, templates are good for defining the skeleton of the Mircoservice. We've found the libraries don't change that often anyway. We put a step in the CI with a warning (Slack) if any library is not the latest version.
@@2k10clarky ye. Though for internal libraries we do not care too much if an old version is used. So if there is no reason to update, it won't be updated. Anyways, I disagree a little bit with Dave that libraries are breaking DRY as well. I mean the whole software world is built from libraries and they are one of the strongest tools to follow DRY. Repeating code in microservices can be a lot of pain as well. What does it help me if I formally avoid an interdependence but then have to copy paste code into 18 different services.
This is exactly what I’m doing now. One servicehelper library does all the background work, threads, logging etc. the service ends up being a few lines of code to configure and the actual working methods for that service
I liked this phrase so much that I had to register in the comments: "If I find my tests difficult to write, for whatever reason, I know I have a problem with the design of my code"
That's honestly the biggest benefit I saw by starting using TDD. Not even the other major benefit which is feeling secure about deploying, but how writing the tests forces me to write much better and less coupled code.
Very good treatment of this challenge. I’m the lead architect of a SW platform for deeply embedded systems and, if I translate the terminology of your examples from Web Microsystems to embedded platform modules, you’ve just described one of the fundamental challenges of my career. And, it has often been a bigger challenge to explain the trade offs made than it has actually been to make them. Many intelligent and skilled engineers wield design principles like DRY as a weapon, using it to attack any trade off (and it’s proponents) whenever it is inconvenient, without consideration for longer term effects. So, thank you for having the courage and intellectual honesty to challenge a design principles that is often taken as sacred. I may send people to his video when I’ve hit the limits of my own ability to explain it. :-)
Everytime I watch these videos I realise I'm a senior dev. It's been really good for overcoming my imposter syndrome. Which I think is largely due to how quickly trends change, and how aggressively people dwell on specific technology choices.
An absolute must watch is Ben Christensen's "Don't Build a Distributed Monolith" talk. He addresses this exact issue which gives you all the downsides of a distributed system AND a monolith but none of the benefits. 1. Don't put shared *business* logic in a library that is then pulled across your services, rather create a service that contains that logic. 2. Don't *force* consumers to use an "official" client to connect to your service. Expose contracts (not a lib) that can be consumed by any stack.
These types of videos are there because people have already had the problem and some kind of solution has been found. The only purpose of this type of video is to share knowledge already found and possibly published in an article. Such videos are also good for simplifying complicated facts so that a wide range of people can understand them.
@@Banefane Oh, I'm aware. I'm rather highlighting how useful this video is in that I remember a recent project that this video reminded me of in many ways.
Keep shared code in an external library and simply follow the open/closed principle. The library is open for extensions, but closed to modifications. Updating a library should theoretically never break existing code, although this does happen entirely too often in the real world, which is why I am always hesitant to using third party libraries, but that is a different topic.
While debugWarning() is a good *specific* example of when to discard DRY, it's a *bad* example because that type of function is a good case for DRY. Shared libraries are helpful when they provide common functionality without increasing coupling, where two services using different versions won't cause problems. Think of a JSON library. JSON has a fixed spec, and all implementations should implement that spec faithfully. If one service uses a buggy version that doesn't affect its corectness, that's fine. The key is that one service's JSON output must be parseable by other services. There's no reason to duplicate that code across services because it's stable. While DRY is binary, DRYness is a spectrum. Choose your level of DRYness based on the context. Within a service, keep DRY as close to 1 as possible, but slide closer to 0.5 or 0 if the costs outweigh the benefits between services. Also, the importance of DRYness decreases with the DRY code size. A shared library consisting of "const double = x => 2 * x" isn't going to win you any favors. 😆 Great talk!
I treat each microservice as if it were written by another company, on another planet, in another language and the only way to communicate is over the network.
Same. If i happen to have two microservices that is using some common functionality, then i'll just copy paste that shit right into the other microservice one way or another. At some point, i might change the functionality in one of them to fit a criteria, and thats fine. I might not have to alter any of them, and thats fine aswell.
you can do it like this, but it can lead to a very inefficient development process. depends on lots of things. i've seen "everything should be a microservice" lead to ">50% of the code is just for communication"
Excellent video, searched for something like this and ended up with a Dave Farley video of course. It is why I always say "the code may seem the same, but it might not always mean the same".
It's a fascinating problem. I often think about the data that travels between services, through those contracts, from one service to another. In this case, we're not talking about sharing implementation but a common language. It suffers from similar tradeoffs: how much is shared, how much is service-specific and hidden, who owns those contracts? It's another shared dependency. But with everything, it's finding that tradeoff.
Good advice and essential for successful agile delivery of multi-team efforts. There is another adjacent problem with DRY; more a result of practice than concept. Dry is a concept that is so deeply ingrained into the psyche of developers that any whiff of similarity gets packed away into a DRY refactor. Sadly, this often leads to maintenance issues when DRY is the leading construct as the code must introduce more and more special cases to address differences, or forces awkwardness and, possibly, type safety issues as we torture the code into calling the DRY, refactored, code. Like so many skills, mastery is difficult. Knowing when to use DRY as a best practice and when to break the rules and repeat yourself is a key quality to remaining most agile while delivering quality, maintainable code.
Sidecars as a pattern are an important tool to have in your toolbox for DRY, since they are independently deployable. I would have liked for you to mention it and compare with the shared library approach. Everything you said is spot on, though. Keep up the good work.
I dont recall Dave using sidecar as an example in any of his videos that I have watched. Though in researching sidecars, I dont think it fits with his philosophy as a rule as he prefers more loosely coupled code. Though I have to admit, the example I saw was from MS so it may not have been the best example.
What a nice challenge you have! If I may add my 2 cents, always keep the business natural divisions in mind and draw the microservices boundaries taking Conway's law into account.
@@antoruby Absolutely, and thanks for that; I hadn't read about Conway's Law yet. Indeed, our client-facing sites and their CMSs are designed around the various different departments (we've been planning & designing the new 'way' over the last couple of years in fact), and it's the more back-end stuff I can see scope for abstraction and micro-servicing. - Very early days on that front though!
Well in the context of microservices, you can still DRY by putting the "duplicated" behavior in a library that is linked by the various microservices that need to duplicate. So the coupling is not between the microservices, but only a compilation-time dependency.
Don't Repeat Yourself: For Team A the term yourself means Service A and so on. If service A and service B implement debugWarning, neither of them repeats itself. I quite like your lessons, Mr Farley!
I think the biggest problem with DRY is that people don't understand what "repeating" means in context. It's _not_ about the crass mechanics of text looking the same in more than one place. It is about what should happen when you change the code: Do you need to do it in more than one place, then you are repeating yourself. If each place should change independently, then you aren't repeating yourself, even if you have the same piece of code repeated four hundred times. Taking your "debugWarning" example: are two identical implementations of "debugWarning" in different services "repeating yourself"? The anwer is the same as to the question: "Should adding logging to debugWarning in service A automatically add logging to service B?".
Wow, the bit about design and seams being where real skill in software engineering shows was great 😱 Thanks for another great video, Dave! I’ll make sure to keep these topics in mind on my projects 😤
One point not completely made is this: If you're extracting similar code, the question is: are they similar functionality, or similar implementation. If it's the first, then, extraction would make sense, and putting them in a library. If they're 'just' similar implementation, but the overall functionality is separate and different (or not related), then you're adding invalid coupling, since a change in functionality would then break the (other) coupled service. This will also show up if you want to give it a descriptive name. That name will either be fine in both (and then the functionality is similar), or clash since the functional name does not match up to the expectations of one of the components.
8:00 That's basically what I do with my projects. If I see some functionality, that might be useful for other programs as well, or when I need it for another program and don't want to rewrite it, I create a new repo, add this submodule of the existing program into that repo and let the other program use it. When I find a bug in my new repo or have ideas for improvement, or if I need to extend it to work with one of my programs depending on it, I have to upgrade the repo, but not all programs, which depend on it. Sometimes I do, sometimes I don't, most of the time I at least plan to upgrade to a newer version some day. Especially when using Rust, it's easy to have different programs using different versions of the same library.
Another great video, and personally feeling super lucky as this came out just when I was struggling with exactly the same issue in a project. Thank you so much for sharing your wisdom. 🚀🚀🚀
I dont see the problem with making "debugWarning" its own repo and using fixed versions of it in different apps/services, especially if "debugWarning" was designed to be autonomous. We all do this on a daily basis by installing Npm and other packages... we use a specific version of a package, and as our application ages we might choose to lock it to an old version of the package or upgrade our app/service to use a new major version of the package. IMHO the technical debt, code quality and developer frustration of duplication and maintaining those instances is a much bigger problem and should be avoided at all cost... I would argue that if you are forced into anti-DRY then there are fundamental problems with your design.
Dave I'm not sure I agree that treating the shared function as a versioned library makes the code not DRY, at least not in the sense of the problems that are typically created when code is not DRY. As you said, the big issue with having copies of the same behavior scattered around the code is that small changes to each copy add up fractally over time, but that is completely mitigated if any changes to that shared code happen in a linear, version-controlled fashion that can be adopted by the calling code at the latter's own pace. Of course it does create some overhead on the calling services to keep up with those changes over time, but that cost is already a given with proper external libraries (and the underlying technology) anyway so it still seems like the best of all worlds - especially since the shared code can use semantic versioning so that non-breaking changes can be adopted automatically by the calling services via configuration.
I like the part of the video with the logo that goes after the introduction (I don't know what that part is called in English). It's short and the music is melodic and calming. I also like your logo
I was just thinking about it today, how the whole world has gone literally mad about software and system design, DDD, patterns ... They keep adding up layers and layers of complexity on top of software that should be something simple and straightforward ... And being honest, I don't think software being developed today is that much better than software developed 20 years ago ...
I have recently discovered this channel, and I really enjoy the show. However, respectfully, I have another take on this subject, and I would like to put it out there for your consideration. Caveat: I have not worked too much on microservices myself. This makes sense if you have 2 microservices. But what if you have 100 microservices, and debugWarning() is 50 lines of code? If debugWarning never changes, then copy/paste is ok, but its a lot of work to install. Maybe you copy/paste some of them wrong if you get hasty. Its easy to make a mistake on tedious tasks. On the other hand, if debugWarning() changes in all of the microservices, now you have to change 100 microservices. If only 25% of the debugWarnings() change, thats still 25 microservices to handle. And any time you make changes to debugWarning(), you have to regression test the changes in all the affected microservices. Even if you only make a change to one debugWarning(), you have to ask yourself, why did only one instance of the method change? Did I make a mistake somewhere? Why didn't the other instances change? Good luck figuring that out. On the other hand, if you create a single microservice to encapsulate what varies, debugWarning(), yes there is coupling, but all the microservices are intercommunicating anyway across apis, so adding one more microservice doesnt add to the design complexity anymore than any other new microservice does. Make one microservice call to debugWarning() be the same for all other microservices. If microservice B needs a different version of debugWarning(), put a new version in your platform microservice and call it debugError(). debugError() will likely depend on code in debugWarning(), so you get code reuse locally and the benefits of that. And no one is making you use the platform version of the method. If one microservice needs a debugWarning() that is the same in name only to the platform version, then its free to write up a method locally. Versions of methods is handled, and so is minimizing coupling. It might be said that a downside of this is that a bug in the platform will cascade to all of the dependent microservices, due to coupling, but to be fair, so does the bug fix, saving you effort. It might also be said that having one microservice call per platform method call could have a lot of framework code to make it work, but its still better than having all the copied code in all the many microservices that are out there. But its a judgement call. If you have 2 microservices, copy/paste, but for 100, platform microservice it. At the end of the day, you want to make your microservices independent of other microservices, but according to my argument, you wouldn't achieve that completely. But realistically, if you did achieve this, you would never make a call to any microservice ever. EDIT: After a bit more thought, it occurs to me that the key issue with bad coupling caused by this approach would be that if the platform microservice goes down, then it brings down any other microservice that is using it, in a cascade effect. The solution is to spin up enough instances of the platform microservice to avoid this issue. Also, if the dependent microservices are composed well, then they will be fault-tolerant enough to carry on even if there is a platform outage.
Isn't the solution here to make sure that the dependencies are well-specified before they're used across services, and can thus be kept relatively backwards compatible? If you want to change the specification, then you can duplicate the code and change what you need to match the new specification.
In my case, between a service and a client (C# web-api and Blazor) I reduce coupling and repetition by using Interfaces (Contract between producer and consumer) also with default implementations of the intrinsic validations. Both sides share the Interace , and this forces both parts to have the same Entities/DTOs properties and comply with basic integrity.
So right about interfaces of library vs microservice. I hear decoupling sometimes as reason behind using separate service. But in reality it might be a bit far from achieving that.
Even doing data analysis with MATLAB I recently decided to forego DRY for copies of the same functions in different folders for the very reasons mentioned herein
So few people are willing to talk about the cons of the divine DRY. 99% of devs think DRY is a non-negotiable and is a must in every situation. I would argue that DRY is NOT about code duplication but the underlying intent or knowledge of what the code does. So if two methods have the same lines of code in it but the intent is very different then you do NOT want DRY because you are coupling them together. If intent is different then you want the code to be separate so that they can evolve separately. The code may be the same now but wont be in the near future. This is where devs try to make complex solutions that can accommodate a million different possibilities that might happen in the future. Rather just leave it separate and simple and change the code when the future comes.
One good example I've had is when you see two things that look the same but really aren't the same I tend to air on the side of caution and not make them the same. Trying to unpick it later is an absolute nightmare and likely one will break at some point
"Discarding DRY for micro services" might be extreme. There are common services that you have to use. If your micro service offers a RESTful API and uses Node.js, it likely uses Express. If the service's state is stored in a database, it probably uses a vendor's product. Granted, these are external dependencies, but still are dependencies. No micro service lives in a vacuum. There is a fuzzy zone where DRY benefits equates costs, and navigating this is part of design indeed. The "best" choice may change with time, as products (internal or external) appear, change and die. Micro service approach moves the boundary significantly away from DRY, but it does not necessarily eliminate it. Sometime you can handle change by maintaining API compatibility, even if just adapter stubs. That's a design skill too, not easy but the more abstract the API is the easier it is. I agree with other comments: versioning is not full duplication. Versioning loosens the dependency. The rule is to not need all services to use the same version. That gives you temporary autonomy and buys you time. Eventually the old version goes away, unlike most real duplication. It is a tool in the box.. I have a love/hate relationship with the Node.js ecosystem. It is an extraordinary tool, but change is the name of the game. I reluctantly gave it up for personal projects due to the upgrade overhead. I don't consider external dependencies as different than internal ones: to "buy" or make is a design choice too, one that comes with a cost later.
In the context of microservices, for me, DRY has a different meaning than in the context of monolithic code. In think having a library and defining the dependencies in the microservices is actually DRY in terms of code, because the code is technically in one place at design time and at development time. The code is only duplicated at runtime, and DRY, as I see it, applies to design and development times, not at runtime.
Thank you for an excellent and insightful video. Thanks for also noting the lack of a simple solution or easy rule for some areas where you need skilled developers to create good designs.
Good video. I've gone through dependency hell but have also heard about things like sidecar pattern to deal with reusable functions in microservices. Do you have a video that talks about that?
What determines of two pieces of code are duplicate code or not is not if the code looks the same to the naked eye. What determines duplication is potential sources of change. In other words if Service A could have a reason to change the code that would not apply to Service B then it is not duplicate code, it is merely code that looks similar.
Another great video. I address DRY vs coupling by means of a nuget “subscriber” model with strong adherence to the Open/Closed principle. I create on-prem Nuget packages. Consumers of libraries can upgrade to new versions as they wish. “Breaking changes” are very vary seldom. DDD is also closely adhered to - especially with models and contracts. Again, great video!
The question I asked myself in this case is who is responsible for changing this piece of code if both A and B are responsible for changing debugwarning I m not repeating myself because even if it's the same code, it's not the same reason. it's just the same code by coincidence and those requirements will change at their own pace. I feel like it's a mistake to see code duplication issue behind dry I think the idea of dry is to prevent duplication of responsibility. What do you think ?
I prefer packaging apps with a package manager that uses semver with automatic updates for minor and patch releases which in semver should only be non-breaking feature additions, performance improvements, bug fixes. Major version releases need someone to check over and address possible breaking changes and push a button (by committing a major version bump and trigger a deploy) when it's ready.
Fine as long as the "guaranteed backward compatible" statement is true. Hard to do for a long period, but that is essentially the game you buy into with Microservices. Define APIs that are either loosely-coupled enough to be true even when stuff changes, or so stable that they never really change. The latter has been done for years by OS and language vendors. Big deal if the String library changes or how you save a file!
An excellent explanation once again. Links to where to get the t-shirts would be a nice addition. ;-). I have a few but can always make space for more.
I think this is one of your best videos thus far. This topic is can be hard to break down and described in such a concise terms but you did it. Would you consider a video where you sit down and deal with example of real world issue where DRY and close coupling clashes?
In my opinion DRY can ruin mental abilities. Its much easier to reason about code when you know its isolated. Having repeat code is not a problem in itself, tje colpiler cetainly doesnt care, the prpbkem is human memory. I actually propose toolsets that can track duplicate code across your version control system so if you DID need to make a change you could easily find everything anyway. In fact i have designed a visual studio plugin that can do this, but in have not implemented it yet Also, another thing people forget about DRY is many times you will eventually come up with a "special case" well now you have to always remember who has the special case...and burden the code with it.p or give the special case its own copy. Lets say you give the special case its own copy instead. Well now everyone shares it but the special case. Good luck rememnering that. Its much easiet to remember by just having everyone own their own copy.
The problem is that developers too often start with "everything has to be a microservice". There are use cases where microservices make sense, but my observation is that the pattern is overused. I would rather do the following: * Focus on well defined interfaces that tests can be written against. This allows the development team to change the insides of functions without breaking other functionality. You get the benefits of services without the overhead. * Use dependency management and semantic versioning. You can declare that one part of the system is using version 2.X of your function. All upgrades to the function can stay at version 2 as long as no breaking changes are introduced. Having multiple versions does not mean that you have broken DRY. * Automated testing is essential. You can test that the function still works as specified and can test that all the consumers of the function still works. Imaging if you implement the above example as a microservice and you do a really good job so that the response time is 10 milliseconds. Now you call it 30 times in the code and you have just added 300 milliseconds to your function execution time.
I, and most Microservice experts, agree with this. My preference is to design my systems mourned services, but store them all together in a single repo. That gives you a sensible scope for evaluation (a deployment pipeline) that is definitive. Use this time to learn what a good level of abstraction is for the interfaces between the services, and as you start to nail that, you can then start to think about migrating services out to separate repos and treating them as independently deployable microservices.
Multiplying technical mechanisms is just a source of defects and complexity. Multiplying business rules representation is is all of the above plus a reckless harm to our clients' business. Multiplying something is not decoupling, is always a (horrible) logical coupling. Is not independent deployed if you need to change all the copies (and for business rules is mandatory). We can multiply with less problems only what is easy to identify and produce (easy to identify and replace copies). Recap: Having different versions/copies for the technical mechanism or business access forms is not as bad as multiplying the representation of business rules. We may have discretion over our design, but we cannot deliberately affect the representation of the business. I have seen "dead" products due to the destruction of the business through multiplication (multiple versions = a lot of wrong versions). Resuscitation was done by DRY applied on the part of the business representation
Grandmaster, what is your feeling on a shared library of utilities that do simple things to reduce boilerplate? Things like clamping a number to a range + log a warning. Or defining a RGB color that can convert to HSV? The team is only 4 people in the mobile app space, but it has the potential to touch every part of the code, often in unexpected places in modules that have nothing to do with each other. i.e. the UI module, or the login service, etc etc.
In general I am a subscriber to the UNIX doctrine, of many small tools that work well, rather than big frameworks. So if each of your examples are separate libraries, that is better than bundling them altogether, if they aren't really related. The next thing is to recognise that ANY SHARED CODE adds to the coupling. There are 2 ways to manage coupling, make interfaces that don't change, or make interfaces that are loose-coupled, and so hides any change. For low-level libraries, "interfaces that don't change" is best, which is one reason to separate the pieces out from one another, so that you don't force a change for a reason the doesn't matter to the consumer of the code.
@@ContinuousDelivery Thank you Grandmaster. I will separate out the shared library into smaller subsets like data structures, algorithms, and form validators and such. I will also hide from callers that these are being used. Also, thank you for making a video about YAGNI. I am the YAGNI master (in the negative sense...)
DRY converted to us to configurable and customizable microservices. A video about difference of configuration and customization will be helpful. Regarding this video - debug warning has to be a microservice as well.
note that if you would use a service for debugWarning you would have to develop, probably, a client, which would become a dependency, kind of a library, to both projects A and B. Thus, you would have more to manage for what could be done using a linkage. In the end there is no definitive answer to this question.
I use to duplicate code and declare global variables when developing a certain f(). It's a bit faster. But once done, I immediately get rid of those issues. 4:00, or even a macro. 11:40, I think it's kind hard to write tests when you have classes holding data global to the tested f(). Those data are read/written inside the f(), but not come as parameters.
@@antoruby The macro I mentioned was in place of an alternative f(). Sometimes, only a macro can DRY properly. Global variables and duplicated code I keep only in development of a bit of code. It's temporarily, for a short time. PS: my comment was about 3 independent moments of the video, btw. 3 comments in 1.
The presentor missing the point Deploy-Service does not equal Deploy-Code. Nothing wrong if a service is not aware (and so never touches) of some parts of the code. debugWarning function should exist in preceding shared-code (back reference Consumable) to make it callable from any service comes after. Well constructed (or bootstrapped) code is DRY, otherwise the cow is mad.
I'm happy my two verticals I'm working on doesn't use DRY but I can't say for the other verticals my services communicates with. The maintenance is a pain and fixing. bugs Frontend wise we try to use public components so we don't DRY.
When he says it is no longer dry. That is wrong. Should debug warning be the same implementation he is right. But then whomever updated version 7 to do the precise same thing as version 6 fucked up. If you have two different behaviours that are required, either by extension in your repo, or your service version. It is the same thing. You are loading two different behaviours. And in both cases you are following dry. It's the main argument of composition over inheritance
If you have 100+ micro services to use that `debugWarning`, do you really want to have 100+ copies of `debugWarning`, and then if there is a change required in `debugWarning` to be applied to all micro services, do you really want to update all those 100+ copies to apply the same change? What's bad about versioning the common-service-lib which accommodates the `debugWarning`? Each micro service gets to decide whether it wants to get the latest version of common-service-lib or stay with the currently installed version, depending on the major and minor version updates. When versioning done properly, the benefits of not having to update 100+ copies in 100+ micro service repos, outweighs the overhead of managing versioning of common-service-lib.
Having non-dry code isn't necessarily a bad thing - I have the same error handler function copied 4 times in my game engine. Bad? Perhaps, but I know there won't be coupling between different parts of the codebase. BTW, the solution would be "closed for modification, open for extension".
Good design is tough, copy-paste is easy. If you're a big tech company, you can hire dozens of low-skilled off-the-shelf copy-pasters in place of one properly educated programmer (who uses that word any more?) because the copy-pasters are readily replaceable and if you still end up spending 5 times more to develop a product, that doesn't matter because money is easily printed today and the so-called "market" has long been replaced by politics and other criminal means in determining the success of big business.
i see no problem with putting many microservices into the same repository. they can still be independently deployable. but more importantly, "everything in the codebase" means i do not sabotage my IDE support and make code sharing easier. there can still be an internal structure ("submodules") to prevent cross-service code contamination. also: whoever changes "debugWarning" is forced to change all the calls, so they will not ruin other services.
Dave I think to use concept in barinless way is a bad habbit. I recall the time some 30 eyars ago... Take the requiremet text. Mark all the werbs and subjects. Now You have all the object with methods of the code. Okay back to CI/CD. There was a company where jenkins pipelines have been built with the use of ctrl-c and ctrl-v. I did an outrage with introducing shared jenkins libraries. Yes, writing a pipeline became harder but the quality of the whole is got better.
The point is this: Use dry when it makes sense, which is most of the times. But as usual there are exceptions. I started working in microservices embedded system where they duplicated fucking common enums...
I understand your point of view, but myself as a Software Architect, and ex software engineer, I must say that I disagree with you on DRY being bad for Micro services, or that using an external library containing a DRY codebase causing coupling is bad. Every software product uses external libraries, and yes, you're coupled to those, nothing new there. I'd say that repeating code across any application just to avoid coupling is a mistake. I think DRY should change to NRY: NEVER REPEAT YOURSELF! Life's to short to continuously reinvent the wheel. 🙂 Just saying.
I still don't see how treating infra as 3rd party libraries could cause any harmful coupling. Should I not use Entity Framework in two of my services and implement my own ORM each time? I think the problems here would not come from coupling but poor API design and maintenance.
Imagine your service and mine are both using some infra service, a generic data-store (DB) of some kind perhaps. If I decide to upgrade to the new version of the data-store, and that forces you to upgrade because our code is using the same dependency management then my decision forces change on you. If I can upgrade my stuff, without you upgrading yours, there is repetition, we are not DRY.
why not make "base" debugWarning(), then debugWarningA() and debugWarningB() to "decouple" services and keep it DRY? i mean, services are coupled by os kernel anyway and call the same copy of read() and write() when doing i/o. i think it's not very nice to run each service on a different kernel even if you can, probably not a good omen when you see something like that. perhaps DRY is generally always better than not DRY, or maybe service should be agnostic of such things.
I kind of hate these acronyms for programming principles. I always think "Oh, another weird business technique, I'm not interested in". And most of the time, when I find out, what one means, it's either common stuff, I do anyway, because not doing it that way would be stupid (for example DRY), or it does not apply to the languages I'm using (like SOLID mostly).
It seems like a great reason to not use microservices unless absolutely forced because of organization scaling issues. A small company with 6 developers shouldn't be doing this to themselves.
Yes, much easier to keep everything in a single repo, build, test & deploy all together is the simplest approach. Still keep the code modular, still design as services, if you like, but deployment independence is difficult to do well.
Take the reasoning of two different versions of debugWarning to not be called DRY. Does that mean that using different versions of libraries from npm or from Apache within your system result in it being WET? Or should that code always be forked into Microservices? Using software repositories to run multiple versions of the same utilitarian library in a complex system is practically speaking DRY (enough.) The Wikipedia article on DRY mixes two different approaches. It starts with the very pragmatic "reducing repetition of software patterns" and then follows up with the aspirational "Every piece of knowledge...". The former describes DRY's effects and the latter describes the ideal state of being when writing code; These should not necessarily be coupled. The words "Every piece of knowledge must have a single, unambiguous, authoritative representation within a system" seem to allow the discarding of the evolution of the very "knowledge". I'd say something like "Tomorrow's knowledge may be more refined and efforts will be taken to maintain backwards compatibility. Upon braking BC mitigation measures will be undertaken." If your conceptual approach is to accept that versions of the same API represent the evolution of the correct knowledge, then different versions of the same utilitarian library may be considered DRY (enough.) Minor remark: Not the VCS but a software repository should hold released code to be linked into the applications thereof. One should remember that applications (that use APIs) gain the most by DRY-ness.
![Ice On My Teeth - ATEEZ エイティーズ 에이티즈 [Music Bank] | KBS WORLD TV 241115](http://i.ytimg.com/vi/c-n92OgbbwA/mqdefault.jpg)
"My general rule of thumb: don’t violate DRY within a microservice, but be relaxed about violating DRY across all services. The evils of too much coupling between services are far worse than the problems caused by code duplication." -- Sam Newman in "Building Microservices"
Sometimes, when a function is very general and will most likely not need module specific changes, it would make sense to share it between multiple modules.
@@porky1118 I guess you mean general == technical. Of course, you o not want to duplicate technical framework/support code. But if you happen to duplicate business code in several services, you most likely screwed-up you overall microservices design.
@@zerettino No, general is not technical.
It means it's not special purpose. It has many use cases. It can be used in many circumstances.
If you recognize some pattern in many of your modules, which could be generalized, and will still work if the modules change, it makes sense to put them into a module used by both.
Even if it will cause a lot of coupling, that's not a huge problem. You can still refactor by just copying the affected function(s). So it shouldn't be too bad.
Until you have to change the same code in hundreds of microservices. But I like the idea of creating a library independently and each microservice upgrading or not based on the version number.
In my first IT job (in the noughties) I had to copy a lot of code between SQL2000 DTSs and it was a pain to change the same code and remember which package used it between hundreds of DTSs packages.
The subject is very important. In our company we've resolved the problem with templates. When we need a new microservice that needs common stuff like logging, metrics etc. we create it from a template. So when we change something common we change the template, and all next services will use the new one, with new features (and bugs), but all existing services aren't forced to be changed.
we solve it by putting common stuff in libraries and name the explicit version. So if something new in the library is necessary for a service, there is at least a conscious decision.
@@ichduersiees8374 yep I think libraries with specific versions is the correct approach, templates are good for defining the skeleton of the Mircoservice. We've found the libraries don't change that often anyway. We put a step in the CI with a warning (Slack) if any library is not the latest version.
@@2k10clarky ye. Though for internal libraries we do not care too much if an old version is used. So if there is no reason to update, it won't be updated.
Anyways, I disagree a little bit with Dave that libraries are breaking DRY as well. I mean the whole software world is built from libraries and they are one of the strongest tools to follow DRY.
Repeating code in microservices can be a lot of pain as well. What does it help me if I formally avoid an interdependence but then have to copy paste code into 18 different services.
@@2k10clarky "We put a step in the CI with a warning (Slack) if any library is not the latest version."
This is exactly what I’m doing now. One servicehelper library does all the background work, threads, logging etc. the service ends up being a few lines of code to configure and the actual working methods for that service
I liked this phrase so much that I had to register in the comments: "If I find my tests difficult to write, for whatever reason, I know I have a problem with the design of my code"
That's honestly the biggest benefit I saw by starting using TDD. Not even the other major benefit which is feeling secure about deploying, but how writing the tests forces me to write much better and less coupled code.
"DRY is a useful guideline and a rotten rule"
You just put in a sentence all my feelings about DRY
Thanks
"Just because you write it in C it doesn't mean that you have to write crap code" -- my thought for this week
@@twigsagan3857 Not my words, watch the video
"Just because you're writing in C it doesn't mean you have to write crap code" it's this kind of aside that makes me such a huge fan of Dave
Very good treatment of this challenge. I’m the lead architect of a SW platform for deeply embedded systems and, if I translate the terminology of your examples from Web Microsystems to embedded platform modules, you’ve just described one of the fundamental challenges of my career. And, it has often been a bigger challenge to explain the trade offs made than it has actually been to make them. Many intelligent and skilled engineers wield design principles like DRY as a weapon, using it to attack any trade off (and it’s proponents) whenever it is inconvenient, without consideration for longer term effects. So, thank you for having the courage and intellectual honesty to challenge a design principles that is often taken as sacred. I may send people to his video when I’ve hit the limits of my own ability to explain it. :-)
Everytime I watch these videos I realise I'm a senior dev. It's been really good for overcoming my imposter syndrome. Which I think is largely due to how quickly trends change, and how aggressively people dwell on specific technology choices.
An absolute must watch is Ben Christensen's "Don't Build a Distributed Monolith" talk. He addresses this exact issue which gives you all the downsides of a distributed system AND a monolith but none of the benefits.
1. Don't put shared *business* logic in a library that is then pulled across your services, rather create a service that contains that logic.
2. Don't *force* consumers to use an "official" client to connect to your service. Expose contracts (not a lib) that can be consumed by any stack.
I have been thinking a lot about this problem the last days and I am a little proud to have come up with the same conclusion
I wish this video existed 1.5 years ago. Spot on with the problems we had then. :)
These types of videos are there because people have already had the problem and some kind of solution has been found.
The only purpose of this type of video is to share knowledge already found and possibly published in an article. Such videos are also good for simplifying complicated facts so that a wide range of people can understand them.
@@Banefane Oh, I'm aware. I'm rather highlighting how useful this video is in that I remember a recent project that this video reminded me of in many ways.
I think DRY is so entrenched in our minds we wont listen unless we have experienced the major problems it can cause.
Keep shared code in an external library and simply follow the open/closed principle. The library is open for extensions, but closed to modifications. Updating a library should theoretically never break existing code, although this does happen entirely too often in the real world, which is why I am always hesitant to using third party libraries, but that is a different topic.
Planning for monotonicity of API helps with the breakage thing.
Business specifications change all the time so if you do that you are just wasting your time
Dave's t-shirts are always the icing on the cake
While debugWarning() is a good *specific* example of when to discard DRY, it's a *bad* example because that type of function is a good case for DRY.
Shared libraries are helpful when they provide common functionality without increasing coupling, where two services using different versions won't cause problems. Think of a JSON library. JSON has a fixed spec, and all implementations should implement that spec faithfully. If one service uses a buggy version that doesn't affect its corectness, that's fine. The key is that one service's JSON output must be parseable by other services. There's no reason to duplicate that code across services because it's stable.
While DRY is binary, DRYness is a spectrum. Choose your level of DRYness based on the context. Within a service, keep DRY as close to 1 as possible, but slide closer to 0.5 or 0 if the costs outweigh the benefits between services. Also, the importance of DRYness decreases with the DRY code size. A shared library consisting of "const double = x => 2 * x" isn't going to win you any favors. 😆
Great talk!
I treat each microservice as if it were written by another company, on another planet, in another language and the only way to communicate is over the network.
Bingo. This opens the door to polyglot systems.
Same. If i happen to have two microservices that is using some common functionality, then i'll just copy paste that shit right into the other microservice one way or another. At some point, i might change the functionality in one of them to fit a criteria, and thats fine. I might not have to alter any of them, and thats fine aswell.
you can do it like this, but it can lead to a very inefficient development process. depends on lots of things. i've seen "everything should be a microservice" lead to ">50% of the code is just for communication"
@@HoD999x I didn't say everything has to be a microservice
@@HoD999x as well as 50%+ of the performance and traffic.
The clarity and quality of these videos is over the top!
Thank you! We’ve been working on it 👍
Excellent video, searched for something like this and ended up with a Dave Farley video of course.
It is why I always say "the code may seem the same, but it might not always mean the same".
Great stuff! Just came into this DRY vs coupling in my project.
Yeah me too.
Excellent video. Everyone who thinks they have deployed Microservices should understand this.
It's a fascinating problem. I often think about the data that travels between services, through those contracts, from one service to another. In this case, we're not talking about sharing implementation but a common language. It suffers from similar tradeoffs: how much is shared, how much is service-specific and hidden, who owns those contracts? It's another shared dependency. But with everything, it's finding that tradeoff.
Good advice and essential for successful agile delivery of multi-team efforts.
There is another adjacent problem with DRY; more a result of practice than concept. Dry is a concept that is so deeply ingrained into the psyche of developers that any whiff of similarity gets packed away into a DRY refactor. Sadly, this often leads to maintenance issues when DRY is the leading construct as the code must introduce more and more special cases to address differences, or forces awkwardness and, possibly, type safety issues as we torture the code into calling the DRY, refactored, code.
Like so many skills, mastery is difficult. Knowing when to use DRY as a best practice and when to break the rules and repeat yourself is a key quality to remaining most agile while delivering quality, maintainable code.
Sidecars as a pattern are an important tool to have in your toolbox for DRY, since they are independently deployable. I would have liked for you to mention it and compare with the shared library approach. Everything you said is spot on, though. Keep up the good work.
I dont recall Dave using sidecar as an example in any of his videos that I have watched. Though in researching sidecars, I dont think it fits with his philosophy as a rule as he prefers more loosely coupled code. Though I have to admit, the example I saw was from MS so it may not have been the best example.
This was real food for thought, especially as I'm beginning to plan moving some of our online systems into microservices.
What a nice challenge you have! If I may add my 2 cents, always keep the business natural divisions in mind and draw the microservices boundaries taking Conway's law into account.
@@antoruby Absolutely, and thanks for that; I hadn't read about Conway's Law yet. Indeed, our client-facing sites and their CMSs are designed around the various different departments (we've been planning & designing the new 'way' over the last couple of years in fact), and it's the more back-end stuff I can see scope for abstraction and micro-servicing. - Very early days on that front though!
Well in the context of microservices, you can still DRY by putting the "duplicated" behavior in a library that is linked by the various microservices that need to duplicate. So the coupling is not between the microservices, but only a compilation-time dependency.
Exactly! The whole idea is based on not knowing the standard design idiom of outlining a common dependency.
What if a change in the common library breaks behavior in some microservices?
Don't Repeat Yourself: For Team A the term yourself means Service A and so on. If service A and service B implement debugWarning, neither of them repeats itself.
I quite like your lessons, Mr Farley!
I think the biggest problem with DRY is that people don't understand what "repeating" means in context. It's _not_ about the crass mechanics of text looking the same in more than one place.
It is about what should happen when you change the code: Do you need to do it in more than one place, then you are repeating yourself.
If each place should change independently, then you aren't repeating yourself, even if you have the same piece of code repeated four hundred times.
Taking your "debugWarning" example: are two identical implementations of "debugWarning" in different services "repeating yourself"?
The anwer is the same as to the question: "Should adding logging to debugWarning in service A automatically add logging to service B?".
Wow, the bit about design and seams being where real skill in software engineering shows was great 😱
Thanks for another great video, Dave! I’ll make sure to keep these topics in mind on my projects 😤
One point not completely made is this: If you're extracting similar code, the question is: are they similar functionality, or similar implementation. If it's the first, then, extraction would make sense, and putting them in a library. If they're 'just' similar implementation, but the overall functionality is separate and different (or not related), then you're adding invalid coupling, since a change in functionality would then break the (other) coupled service. This will also show up if you want to give it a descriptive name. That name will either be fine in both (and then the functionality is similar), or clash since the functional name does not match up to the expectations of one of the components.
8:00 That's basically what I do with my projects.
If I see some functionality, that might be useful for other programs as well, or when I need it for another program and don't want to rewrite it, I create a new repo, add this submodule of the existing program into that repo and let the other program use it.
When I find a bug in my new repo or have ideas for improvement, or if I need to extend it to work with one of my programs depending on it, I have to upgrade the repo, but not all programs, which depend on it. Sometimes I do, sometimes I don't, most of the time I at least plan to upgrade to a newer version some day.
Especially when using Rust, it's easy to have different programs using different versions of the same library.
You are one of the best software youtubers. Thank you for your hard work.
Appreciate that!
Awesome content! You really discuss the most important subjects everytime
Another great video, and personally feeling super lucky as this came out just when I was struggling with exactly the same issue in a project. Thank you so much for sharing your wisdom.
🚀🚀🚀
Glad it helped!
I dont see the problem with making "debugWarning" its own repo and using fixed versions of it in different apps/services, especially if "debugWarning" was designed to be autonomous.
We all do this on a daily basis by installing Npm and other packages... we use a specific version of a package, and as our application ages we might choose to lock it to an old version of the package or upgrade our app/service to use a new major version of the package.
IMHO the technical debt, code quality and developer frustration of duplication and maintaining those instances is a much bigger problem and should be avoided at all cost... I would argue that if you are forced into anti-DRY then there are fundamental problems with your design.
Dave I'm not sure I agree that treating the shared function as a versioned library makes the code not DRY, at least not in the sense of the problems that are typically created when code is not DRY. As you said, the big issue with having copies of the same behavior scattered around the code is that small changes to each copy add up fractally over time, but that is completely mitigated if any changes to that shared code happen in a linear, version-controlled fashion that can be adopted by the calling code at the latter's own pace.
Of course it does create some overhead on the calling services to keep up with those changes over time, but that cost is already a given with proper external libraries (and the underlying technology) anyway so it still seems like the best of all worlds - especially since the shared code can use semantic versioning so that non-breaking changes can be adopted automatically by the calling services via configuration.
I like the part of the video with the logo that goes after the introduction (I don't know what that part is called in English). It's short and the music is melodic and calming. I also like your logo
Thank you for the great video! It would be fascinating to watch some talk about technology obsession and overcomplicated design of modern software.
Yeah this is the biggest trap developers fall into nowadays.
I was just thinking about it today, how the whole world has gone literally mad about software and system design, DDD, patterns ... They keep adding up layers and layers of complexity on top of software that should be something simple and straightforward ... And being honest, I don't think software being developed today is that much better than software developed 20 years ago ...
I have recently discovered this channel, and I really enjoy the show. However, respectfully, I have another take on this subject, and I would like to put it out there for your consideration. Caveat: I have not worked too much on microservices myself. This makes sense if you have 2 microservices. But what if you have 100 microservices, and debugWarning() is 50 lines of code? If debugWarning never changes, then copy/paste is ok, but its a lot of work to install. Maybe you copy/paste some of them wrong if you get hasty. Its easy to make a mistake on tedious tasks. On the other hand, if debugWarning() changes in all of the microservices, now you have to change 100 microservices. If only 25% of the debugWarnings() change, thats still 25 microservices to handle. And any time you make changes to debugWarning(), you have to regression test the changes in all the affected microservices. Even if you only make a change to one debugWarning(), you have to ask yourself, why did only one instance of the method change? Did I make a mistake somewhere? Why didn't the other instances change? Good luck figuring that out. On the other hand, if you create a single microservice to encapsulate what varies, debugWarning(), yes there is coupling, but all the microservices are intercommunicating anyway across apis, so adding one more microservice doesnt add to the design complexity anymore than any other new microservice does. Make one microservice call to debugWarning() be the same for all other microservices. If microservice B needs a different version of debugWarning(), put a new version in your platform microservice and call it debugError(). debugError() will likely depend on code in debugWarning(), so you get code reuse locally and the benefits of that. And no one is making you use the platform version of the method. If one microservice needs a debugWarning() that is the same in name only to the platform version, then its free to write up a method locally. Versions of methods is handled, and so is minimizing coupling.
It might be said that a downside of this is that a bug in the platform will cascade to all of the dependent microservices, due to coupling, but to be fair, so does the bug fix, saving you effort. It might also be said that having one microservice call per platform method call could have a lot of framework code to make it work, but its still better than having all the copied code in all the many microservices that are out there. But its a judgement call. If you have 2 microservices, copy/paste, but for 100, platform microservice it. At the end of the day, you want to make your microservices independent of other microservices, but according to my argument, you wouldn't achieve that completely. But realistically, if you did achieve this, you would never make a call to any microservice ever.
EDIT: After a bit more thought, it occurs to me that the key issue with bad coupling caused by this approach would be that if the platform microservice goes down, then it brings down any other microservice that is using it, in a cascade effect. The solution is to spin up enough instances of the platform microservice to avoid this issue. Also, if the dependent microservices are composed well, then they will be fault-tolerant enough to carry on even if there is a platform outage.
Isn't the solution here to make sure that the dependencies are well-specified before they're used across services, and can thus be kept relatively backwards compatible?
If you want to change the specification, then you can duplicate the code and change what you need to match the new specification.
Great video, really good reminder that context is key not just applying principles blindly.
Clear and valuable presentation. Thank you!
In my case, between a service and a client (C# web-api and Blazor) I reduce coupling and repetition by using Interfaces (Contract between producer and consumer) also with default implementations of the intrinsic validations. Both sides share the Interace , and this forces both parts to have the same Entities/DTOs properties and comply with basic integrity.
So right about interfaces of library vs microservice. I hear decoupling sometimes as reason behind using separate service. But in reality it might be a bit far from achieving that.
Even doing data analysis with MATLAB I recently decided to forego DRY for copies of the same functions in different folders for the very reasons mentioned herein
So few people are willing to talk about the cons of the divine DRY. 99% of devs think DRY is a non-negotiable and is a must in every situation. I would argue that DRY is NOT about code duplication but the underlying intent or knowledge of what the code does. So if two methods have the same lines of code in it but the intent is very different then you do NOT want DRY because you are coupling them together. If intent is different then you want the code to be separate so that they can evolve separately. The code may be the same now but wont be in the near future. This is where devs try to make complex solutions that can accommodate a million different possibilities that might happen in the future. Rather just leave it separate and simple and change the code when the future comes.
One good example I've had is when you see two things that look the same but really aren't the same
I tend to air on the side of caution and not make them the same. Trying to unpick it later is an absolute nightmare and likely one will break at some point
"Discarding DRY for micro services" might be extreme. There are common services that you have to use. If your micro service offers a RESTful API and uses Node.js, it likely uses Express. If the service's state is stored in a database, it probably uses a vendor's product. Granted, these are external dependencies, but still are dependencies. No micro service lives in a vacuum.
There is a fuzzy zone where DRY benefits equates costs, and navigating this is part of design indeed. The "best" choice may change with time, as products (internal or external) appear, change and die. Micro service approach moves the boundary significantly away from DRY, but it does not necessarily eliminate it.
Sometime you can handle change by maintaining API compatibility, even if just adapter stubs. That's a design skill too, not easy but the more abstract the API is the easier it is.
I agree with other comments: versioning is not full duplication. Versioning loosens the dependency. The rule is to not need all services to use the same version. That gives you temporary autonomy and buys you time. Eventually the old version goes away, unlike most real duplication. It is a tool in the box..
I have a love/hate relationship with the Node.js ecosystem. It is an extraordinary tool, but change is the name of the game. I reluctantly gave it up for personal projects due to the upgrade overhead. I don't consider external dependencies as different than internal ones: to "buy" or make is a design choice too, one that comes with a cost later.
Was literally just thinking on this. Great video!
In the context of microservices, for me, DRY has a different meaning than in the context of monolithic code. In think having a library and defining the dependencies in the microservices is actually DRY in terms of code, because the code is technically in one place at design time and at development time. The code is only duplicated at runtime, and DRY, as I see it, applies to design and development times, not at runtime.
Thank you for an excellent and insightful video. Thanks for also noting the lack of a simple solution or easy rule for some areas where you need skilled developers to create good designs.
Good video. I've gone through dependency hell but have also heard about things like sidecar pattern to deal with reusable functions in microservices. Do you have a video that talks about that?
What determines of two pieces of code are duplicate code or not is not if the code looks the same to the naked eye. What determines duplication is potential sources of change. In other words if Service A could have a reason to change the code that would not apply to Service B then it is not duplicate code, it is merely code that looks similar.
Another great video. I address DRY vs coupling by means of a nuget “subscriber” model with strong adherence to the Open/Closed principle. I create on-prem Nuget packages. Consumers of libraries can upgrade to new versions as they wish. “Breaking changes” are very vary seldom. DDD is also closely adhered to - especially with models and contracts.
Again, great video!
Does anyone know when the State of DevOps Report 2022 is going to come out?
One of the best channel for me...many thanks
Isn't the sidecar pattern meant to address this issue?
Thanks for this. Thought provoking!
Pure and enlightened. Just dont write a manifesto to pester us for decades.
The question I asked myself in this case is who is responsible for changing this piece of code if both A and B are responsible for changing debugwarning I m not repeating myself because even if it's the same code, it's not the same reason. it's just the same code by coincidence and those requirements will change at their own pace.
I feel like it's a mistake to see code duplication issue behind dry I think the idea of dry is to prevent duplication of responsibility. What do you think ?
I prefer packaging apps with a package manager that uses semver with automatic updates for minor and patch releases which in semver should only be non-breaking feature additions, performance improvements, bug fixes. Major version releases need someone to check over and address possible breaking changes and push a button (by committing a major version bump and trigger a deploy) when it's ready.
I insisted on DRYing microservices and it was not fun. Great video
Thanks 👍
Excellent video! thanks for sharing!
What do you think about architecturally DRY platforms with guaranteed backwards-ocmpatible µService APIs? Changes would only apply to newer releases.
Fine as long as the "guaranteed backward compatible" statement is true. Hard to do for a long period, but that is essentially the game you buy into with Microservices. Define APIs that are either loosely-coupled enough to be true even when stuff changes, or so stable that they never really change. The latter has been done for years by OS and language vendors. Big deal if the String library changes or how you save a file!
An excellent explanation once again. Links to where to get the t-shirts would be a nice addition. ;-). I have a few but can always make space for more.
I think this is one of your best videos thus far. This topic is can be hard to break down and described in such a concise terms but you did it. Would you consider a video where you sit down and deal with example of real world issue where DRY and close coupling clashes?
Thanks for the suggestion, I will think about it.
Good piece of advice
Awesome as always... Thank you very much Dave
In my opinion DRY can ruin mental abilities. Its much easier to reason about code when you know its isolated. Having repeat code is not a problem in itself, tje colpiler cetainly doesnt care, the prpbkem is human memory. I actually propose toolsets that can track duplicate code across your version control system so if you DID need to make a change you could easily find everything anyway. In fact i have designed a visual studio plugin that can do this, but in have not implemented it yet
Also, another thing people forget about DRY is many times you will eventually come up with a "special case" well now you have to always remember who has the special case...and burden the code with it.p or give the special case its own copy. Lets say you give the special case its own copy instead. Well now everyone shares it but the special case. Good luck rememnering that. Its much easiet to remember by just having everyone own their own copy.
What's the problem? Use a package manager, NuGet for instance, a package is both DRY and versioned
The problem is that developers too often start with "everything has to be a microservice". There are use cases where microservices make sense, but my observation is that the pattern is overused.
I would rather do the following:
* Focus on well defined interfaces that tests can be written against. This allows the development team to change the insides of functions without breaking other functionality. You get the benefits of services without the overhead.
* Use dependency management and semantic versioning. You can declare that one part of the system is using version 2.X of your function. All upgrades to the function can stay at version 2 as long as no breaking changes are introduced. Having multiple versions does not mean that you have broken DRY.
* Automated testing is essential. You can test that the function still works as specified and can test that all the consumers of the function still works.
Imaging if you implement the above example as a microservice and you do a really good job so that the response time is 10 milliseconds. Now you call it 30 times in the code and you have just added 300 milliseconds to your function execution time.
I, and most Microservice experts, agree with this. My preference is to design my systems mourned services, but store them all together in a single repo. That gives you a sensible scope for evaluation (a deployment pipeline) that is definitive. Use this time to learn what a good level of abstraction is for the interfaces between the services, and as you start to nail that, you can then start to think about migrating services out to separate repos and treating them as independently deployable microservices.
Multiplying technical mechanisms is just a source of defects and complexity. Multiplying business rules representation is is all of the above plus a reckless harm to our clients' business. Multiplying something is not decoupling, is always a (horrible) logical coupling. Is not independent deployed if you need to change all the copies (and for business rules is mandatory).
We can multiply with less problems only what is easy to identify and produce (easy to identify and replace copies).
Recap: Having different versions/copies for the technical mechanism or business access forms is not as bad as multiplying the representation of business rules. We may have discretion over our design, but we cannot deliberately affect the representation of the business.
I have seen "dead" products due to the destruction of the business through multiplication (multiple versions = a lot of wrong versions). Resuscitation was done by DRY applied on the part of the business representation
Grandmaster, what is your feeling on a shared library of utilities that do simple things to reduce boilerplate? Things like clamping a number to a range + log a warning. Or defining a RGB color that can convert to HSV? The team is only 4 people in the mobile app space, but it has the potential to touch every part of the code, often in unexpected places in modules that have nothing to do with each other. i.e. the UI module, or the login service, etc etc.
In general I am a subscriber to the UNIX doctrine, of many small tools that work well, rather than big frameworks. So if each of your examples are separate libraries, that is better than bundling them altogether, if they aren't really related.
The next thing is to recognise that ANY SHARED CODE adds to the coupling. There are 2 ways to manage coupling, make interfaces that don't change, or make interfaces that are loose-coupled, and so hides any change. For low-level libraries, "interfaces that don't change" is best, which is one reason to separate the pieces out from one another, so that you don't force a change for a reason the doesn't matter to the consumer of the code.
@@ContinuousDelivery Thank you Grandmaster. I will separate out the shared library into smaller subsets like data structures, algorithms, and form validators and such. I will also hide from callers that these are being used. Also, thank you for making a video about YAGNI. I am the YAGNI master (in the negative sense...)
DRY converted to us to configurable and customizable microservices. A video about difference of configuration and customization will be helpful. Regarding this video - debug warning has to be a microservice as well.
note that if you would use a service for debugWarning you would have to develop, probably, a client, which would become a dependency, kind of a library, to both projects A and B. Thus, you would have more to manage for what could be done using a linkage. In the end there is no definitive answer to this question.
Jinx man!
I use to duplicate code and declare global variables when developing a certain f(). It's a bit faster. But once done, I immediately get rid of those issues.
4:00, or even a macro.
11:40, I think it's kind hard to write tests when you have classes holding data global to the tested f(). Those data are read/written inside the f(), but not come as parameters.
I hope the conclusion is that you don't use global variables behind macros anymore =p
@@antoruby The macro I mentioned was in place of an alternative f(). Sometimes, only a macro can DRY properly.
Global variables and duplicated code I keep only in development of a bit of code. It's temporarily, for a short time.
PS: my comment was about 3 independent moments of the video, btw. 3 comments in 1.
The presentor missing the point Deploy-Service does not equal Deploy-Code. Nothing wrong if a service is not aware (and so never touches) of some parts of the code.
debugWarning function should exist in preceding shared-code (back reference Consumable) to make it callable from any service comes after.
Well constructed (or bootstrapped) code is DRY, otherwise the cow is mad.
I'm happy my two verticals I'm working on doesn't use DRY but I can't say for the other verticals my services communicates with.
The maintenance is a pain and fixing. bugs Frontend wise we try to use public components so we don't DRY.
When he says it is no longer dry. That is wrong.
Should debug warning be the same implementation he is right.
But then whomever updated version 7 to do the precise same thing as version 6 fucked up.
If you have two different behaviours that are required, either by extension in your repo, or your service version. It is the same thing. You are loading two different behaviours. And in both cases you are following dry.
It's the main argument of composition over inheritance
Maybe the microservice pattern is wrong and not DRY? Haven't anyone thought of that?
If you have 100+ micro services to use that `debugWarning`, do you really want to have 100+ copies of `debugWarning`, and then if there is a change required in `debugWarning` to be applied to all micro services, do you really want to update all those 100+ copies to apply the same change? What's bad about versioning the common-service-lib which accommodates the `debugWarning`? Each micro service gets to decide whether it wants to get the latest version of common-service-lib or stay with the currently installed version, depending on the major and minor version updates. When versioning done properly, the benefits of not having to update 100+ copies in 100+ micro service repos, outweighs the overhead of managing versioning of common-service-lib.
Having non-dry code isn't necessarily a bad thing - I have the same error handler function copied 4 times in my game engine. Bad? Perhaps, but I know there won't be coupling between different parts of the codebase. BTW, the solution would be "closed for modification, open for extension".
Good design is tough, copy-paste is easy. If you're a big tech company, you can hire dozens of low-skilled off-the-shelf copy-pasters in place of one properly educated programmer (who uses that word any more?) because the copy-pasters are readily replaceable and if you still end up spending 5 times more to develop a product, that doesn't matter because money is easily printed today and the so-called "market" has long been replaced by politics and other criminal means in determining the success of big business.
i see no problem with putting many microservices into the same repository. they can still be independently deployable. but more importantly, "everything in the codebase" means i do not sabotage my IDE support and make code sharing easier. there can still be an internal structure ("submodules") to prevent cross-service code contamination.
also: whoever changes "debugWarning" is forced to change all the calls, so they will not ruin other services.
Dave I think to use concept in barinless way is a bad habbit. I recall the time some 30 eyars ago... Take the requiremet text. Mark all the werbs and subjects. Now You have all the object with methods of the code. Okay back to CI/CD. There was a company where jenkins pipelines have been built with the use of ctrl-c and ctrl-v. I did an outrage with introducing shared jenkins libraries. Yes, writing a pipeline became harder but the quality of the whole is got better.
The point is this: Use dry when it makes sense, which is most of the times. But as usual there are exceptions.
I started working in microservices embedded system where they duplicated fucking common enums...
I understand your point of view, but myself as a Software Architect, and ex software engineer, I must say that I disagree with you on DRY being bad for Micro services, or that using an external library containing a DRY codebase causing coupling is bad. Every software product uses external libraries, and yes, you're coupled to those, nothing new there. I'd say that repeating code across any application just to avoid coupling is a mistake. I think DRY should change to NRY: NEVER REPEAT YOURSELF! Life's to short to continuously reinvent the wheel. 🙂 Just saying.
Lambda Layers can be used as template in serverless lambda architecture
Why is having versions of a library violation of DRY?
I still don't see how treating infra as 3rd party libraries could cause any harmful coupling. Should I not use Entity Framework in two of my services and implement my own ORM each time? I think the problems here would not come from coupling but poor API design and maintenance.
Imagine your service and mine are both using some infra service, a generic data-store (DB) of some kind perhaps. If I decide to upgrade to the new version of the data-store, and that forces you to upgrade because our code is using the same dependency management then my decision forces change on you. If I can upgrade my stuff, without you upgrading yours, there is repetition, we are not DRY.
"I don't know how to agree with the type system so I just copy-paste code."
Very good video.
Amazing, as always 💯
why not make "base" debugWarning(), then debugWarningA() and debugWarningB() to "decouple" services and keep it DRY? i mean, services are coupled by os kernel anyway and call the same copy of read() and write() when doing i/o. i think it's not very nice to run each service on a different kernel even if you can, probably not a good omen when you see something like that. perhaps DRY is generally always better than not DRY, or maybe service should be agnostic of such things.
I kind of hate these acronyms for programming principles.
I always think "Oh, another weird business technique, I'm not interested in".
And most of the time, when I find out, what one means, it's either common stuff, I do anyway, because not doing it that way would be stupid (for example DRY), or it does not apply to the languages I'm using (like SOLID mostly).
"DRY enough" is what I'd say is THE pragmatic strategy.
3:50 Only three of the lines are repeated. The last one is always different. It might be useful to only put the three same lines into a new function.
In these cases the ancronym becomes WET we enjoy typing
DRY can be not violated even in a microservices architecture and very naturally.
It seems like a great reason to not use microservices unless absolutely forced because of organization scaling issues. A small company with 6 developers shouldn't be doing this to themselves.
Yes, much easier to keep everything in a single repo, build, test & deploy all together is the simplest approach. Still keep the code modular, still design as services, if you like, but deployment independence is difficult to do well.
Yeah
Take the reasoning of two different versions of debugWarning to not be called DRY. Does that mean that using different versions of libraries from npm or from Apache within your system result in it being WET? Or should that code always be forked into Microservices?
Using software repositories to run multiple versions of the same utilitarian library in a complex system is practically speaking DRY (enough.)
The Wikipedia article on DRY mixes two different approaches. It starts with the very pragmatic "reducing repetition of software patterns" and then follows up with the aspirational "Every piece of knowledge...". The former describes DRY's effects and the latter describes the ideal state of being when writing code; These should not necessarily be coupled.
The words "Every piece of knowledge must have a single, unambiguous, authoritative representation within a system" seem to allow the discarding of the evolution of the very "knowledge". I'd say something like "Tomorrow's knowledge may be more refined and efforts will be taken to maintain backwards compatibility. Upon braking BC mitigation measures will be undertaken."
If your conceptual approach is to accept that versions of the same API represent the evolution of the correct knowledge, then different versions of the same utilitarian library may be considered DRY (enough.)
Minor remark: Not the VCS but a software repository should hold released code to be linked into the applications thereof.
One should remember that applications (that use APIs) gain the most by DRY-ness.