Focus on the right parts of the problem when you are creating a new microservices system. Here's my FREE 'how-to-guide-book' offering advice on the Microservices basics to help you get started ➡www.subscribepage.com/microservices-guide
I’m a 30+ years Software Engineer, with clearly a similar path. This is extremely well articulated and should be on the ‘reading’ list of all modern software developers who want to understand concepts from a big-picture level. I don’t often comment on RUclips content but this is worth expressing thanks for a good, down to Earth presentation.
Every developer out there should watch this video. Orchestration and monitoring tools vendors made a lot of damage by trying to get everybody adopting microservices so they could sell their shiny tools. Then developers found themselves producing distributed entangled spaghetti aberrations just because nobody taught them what Dave excellently does here.
Developers are usually very aware of the problems and find themselves producing distributed spaghetti aberrations because they are not the ones who makes such decisions. Typical victim blaming.
@@linuxgaminginfullhd60fps10 It seems to me that @Fran Dayz was saying tools vendors luring developers to purchase their product and then not providing proper support was the issue he had experience with. But you are saying that decision makers forcing those tools onto developers is the issue you are facing? I'm sorry, I do not understand how those situations are the same in a way that is offensive. How does any of that change the point made that Dave's explanation here can help avoid aberration creation in the first place?
Thanks a lot for your videos. Besides the content I really appreciate the way you speak. Somehow it feels that are using the words of the English language, optimized for simplicity and information throughput.
Wow! With almost 25 years of professional software development under my belt, this was the first video/document on Microservices, which made sense to me. Everything else, and that includes most of the big software conferences, was just a variation of marketing bullshit. It is really fascinating to see that it always comes back to how we deal with "complexity". And that is not technical stuff, but the organizational and domain side of things. Thanks a ton!
I have been interviewing with a lot of companies lately that want candidates with micro services experience. Unfortunately most of the companies don’t really understand micro services. They think it’s the NEW way to do development. What they don’t realize is that micro-service’s is just one of many ways to implement your project. Great video!
It made me want to read& understand how NASA/ESA build the ISS modules, to see if these two different contexts share the same "Bounded Context" concept. The "Fire-Breaks" idea led me there.
This channel is amazing, thank you sir! I am arguably a seasoned engineer at this point but listening to you keeps me motivated to better myself constantly. I think will watch every single video of yours. Keep up the good work!
Great to hear! One of my enduring principles is that we all need to keep learning! and to use software development techniques that help us learn. Thank you for the feedback.
I'm a 25 year Master Journeyman plumber...but somehow have become fascinated by microservices over the last few weeks. I absolutely never plan on coding a single thing in my entire life...but really enjoyed this video. Gives me a little perspective when I update some software and wonder how they introduced so many bugs with a simple small update...
It's remarkable how similar are system architecture and plumbing. Or, at least how I imagine plumbing works. I've come to believe that civil engineering falls in that group too. Basically anything that applies patterns and practices to the management of flow and complexity.
My development is mostly a private affair. I write code that I use, web interfaces that I am the only customer for. This content is still helpful, even for my hobby. This video got me thinking of problems that should be decoupled that aren't, and conversely convinced me not to decouple things just to make them "micro." There is no shame in coupling my front end with my back end code, for example in my latest node.js project.
Starting monolithic and transitioning to a microservice architecture seems to be a common route when the team GROWS- Because in an ideal world, each microservice deserves it's own team.
As a relatively new developer (2+ years) I really enjoy your video. Helps me to think at a scale larger than the code immediately in front of me. Thank you for this!
Your experience shed light on exactly what I felt was off about people/companies claiming to be doing microservices but most definitely are not. This is an enormous problem for semantics and execution. Not everyone is on the same page. I really think of big monolith companies that want to decompose their design into "microservices", but in the end they will spend a ton of money for little to no benefit (ROI). So much time is often wasted chasing after 'trends' before thinking things through...
I don't think their chasing trends, their chasing ways to bring "complexity" somehow under control (probably because they did not pay much attention to that previously at an organizational level, which resulted in a lot legacy code/systems) without understanding all the implications of their decisions and approaches, and as Mr. Farley said, there are many other ways to do that besides microservices. But the change is not because it's trendy, it is because the people involved feel the pain in developing and maintaining said legacy system. The company big wigs don't care that much about fancy code or architecture they care about money and they were ill advised that this new shiny architecture will save the system from crumbling from within (and also solve all world problems from global warming to the inevitable demise of human kind), and in the process they sometimes accelerate the decay of the system by over-engineering stuff (that also get polluted by the old way of doing things).
@@nerrierr I think you missed my point entirely by focusing on a single word. Microservices DO NOT attempt to bring complexity under control, it introduces more of it. The core purpose is for scalability and team environments to work on separate code bases in isolation but will work with one another when connected. Monolith and Microservice design are essentially antithetical to one another when done properly. But the point I was making was that companies using monolith design are decomposing their code into microservices for the wrong reasons and doing so in a hybrid way which gives little to no benefit for all the increased difficulty. Doing so is following a trend: see technology adoption lifecycle.
@@destroyonload3444 fair enough, but to be honest that still is directly related to the complexity and requirement of the project. You don't scale and parallelize if you don't need to (ex: for a simple console application that just prints a simple message you will not do it in a microservices architecture no matter how misguided you are). As for the fact that it adds more complexity it is like saying that abstraction/encapsulation just adds more classes/methods without a benefit (instead of helping you in the grand scheme of things, nothing is free in life and in this case you pay some extra code "locally" to save your a*s some place else, and you do it because of the complexity of today's systems). In the end I think we agree that you need to pick the right tools for the job and for the right reasons, the difference is why we like/dislike microservices (in that sense it's probably relevant to what we are comparing it to). And that is perfectly fine, no team or project is the same in every aspect and people have different opinions based on their previous experiences (there's a reason why giant companies recommend it, and in fact only they are, but most companies aren't Netflix nor is their product of the same scope/context) (well one of the reasons is obliviously practical and the other is probably more marketing focused but that's another topic, I guess we all have to earn our bread somehow :) )
@@nerrierr right. You still have abstraction in microservices, but the real complexity comes with multiple middlewares and dealing with events. Statistically, I have no idea what the most used implementation is, but the most I hear about is the use of an event server and eventually consistent design. I think we're more or less on the same page. BTW, I watch this channel often because his methodology seems to make the most practical sense in implementation, and he's been through several major eras of programming milestones. I find it funny that people are still trying to reinvent what he already had working in the 90's! I watch a lot of Computerphile, too. The down-to-earth "old timers" have my utmost respect.
The only software engineering processes and practices related content that makes 100% sense. I've been a part of multiple discussions around such topics like software architectures, design patterns, agile development, behavior/test driven development etc...none of them make as much as sense as much as your content does. Dave, please continue posting such videos. I want you to know that your experience, talks, and discussions make lives easier for many software engineers worldwide! 🙂
My key takeaway here is this: the API (as in the names of the calls, the expected arguments, the expected results, etc) should be the first thing to be designed and agreed upon when the analysis phase is over. On top of that it must be cast in stone, immutable and stable during a development cycle. Finally it must be the first thing that will be mocked during implementation so the service consumers have something to work upon. The mentality to adhere to all of this in an agile environment where there are constant releases every e.g. month or so, this is the real fight, this is the real issue, the fact that people must put down some communication constraints, draw a box and remain in there, no "well, you see, the customer also decided they want this and this and this so we are mutating the API definition". A mini lecture on how to keep people in check (both on customer side and on the PM/TechLeads side), now, that would be a treasure!
I think having any design that is inmutable and cannot adapt to customer demands should be a definite no-go for any serious software project. Instead, having stable APIs means that once you create an API, you have to keep it running. To add new features, you introduce new APIs without breaking the old API. Some of the old APIs may be implemented as wrappers for the new API but the consumers of the microservice API don't need to mind about that.
@@MikkoRantalainen Introducing a new API for every change isn't practical either. It leads to growing legacy code base, as it's problematic to find out when it is safe to delete some old API version. That's why big companies end up with monorepos, as in a monorepo there's more control over the consumer's code, as far as the API does not escape the monorepo.
@@НиколайТарбаев-к1к I'd argue that if you continously modify the API in a way that requires consumers to be modified, too, you don't actually have an API at all. You just have fully custom protocol instead.
I've watched a number of your videos now and they are brilliant. Straight talking and oozing knowledge and wisdom. From now on, part of my decision process is going to be 'what would Dave Farley do'
As a person, who made microservices before people started calling it that way I am happy for this video. Separately deployable is something that I see so many people out there do not grasp. What I do not grasp however is sometimes how heavyweight the infrastructure is around real microservices! In my team I did not use any kind of infrastructure just plain REST and pretty much plain javascript (not even js frameworks) to implement these ideas and yet there was a clean way to achieve copy-paste kind of integration even on the frontend side - and by that I literally mean the code was just copied into its place and was magically working. Never saw a lightweight solution like that later, but only heavy solutions where they used some tool to "help" create microservices, but only then I felt the cost of it - not when we did the stuff manually without framework. I think some movement like "lightweight micro services" should arise. Maybe it is already arisen, but not what I saw. The idea is that if you cannot set up your microservice infrastructure in a week with everyone on the team understanding all the details - then you are doing it wrong. This is similar to what you say about the microservices themselves and their sizes, but here I am talking about the support infrastructure that make the deployment and everything work - both backends and frontends of the microservices and the ability to publish them even when working together! I hope such a thing exists, but as I said I only saw two things: we doing without any specific infrastructure OR some teams using heavy infrastructure that incurs more cost that simple projects can afford. This is a big deal because original company where we worked like that was less than 50 person and team around projects where we employed this was less than 6 person and still we did not feel at all "too big work" to develop in this way. Any yes again: it was completely shippable separately - even the frontend of these services were separately shipped and built up a complex web app. Anyways: I think everyone who ever use the word by their mouth in any circumstances should watch this video. Would help a lot if people would all do so...
The idea of "bounded context" is so important. Whenever i discuss architecture with people i talk about my idea of a "module boundary" but i always struggled to explain what i mean. This description of a bounded context is exactly what I was looking for.
One problem that I’ve experienced with microservices is that it is really hard to keep everyone up on what are the boundaries of each of you services… sometimes a lot of requirements come from the managers and they want it do be done in service A when it should be done in service B and not all employees know all of the microservices that well to be able to set those boundaries… I’ve experienced features being done in a service and then it came up that the same feature was already being done in another service that is called way up in the architecture, way after it was already shipped to production. I’ve thought about it if that’s a problem of not having so many people understanding that well the architecture or if that’s a leadership or management issue… I feel like there’s a small inner system that developers are capable of understanding, it’s almost impossible to master the design/domain of so many of the microservices. I haven’t seen anyone talking about this, I’m always wondering how companies like Netflix deal with that, like, how many other mircroservices do your engineers need to grasp besides the one/ones they are really contributing code to?
Netflix originally was a huge monolith. They then became microservices! This is the biggest mistake system designers make. You need to have a fully functioning monolith in production with very clear documentation and perfomance benchmarks before you even think about converting it into microservices!
I almost never comment. But I must. This is a must watch by ANYONE involved in solutions development in a modern organization. 10 out of 10. This should even be watched by the non-technical stakeholders. Well done.
Really great stuff! Thanks so much! Working as a management consultant I see my projects 'moving closer' to the field of technology every year. Thus, usually I can't deliver valuable results for my clients without understanding related technological practices and principles. Your videos are amazing in the way you are able to explain those technological issues! 10/10
Thanks Dave. As you mention, there is very much more to cover on this subject, but from a QA point of view the subject of contract testing needs to be raised early on in any discussion about microservices. Also there are some severe gotchas with the approach too. As with many things we need to do our upfront research or risk project failure! Sam Newman's books contain a wealth of info.
Yes I am thinking of doing more stuff on design and architecture on the channel. I don't think of it as "doing up front research" as much as understanding what the things that we do are, and how they work, we often we seem to jump on ideas based on sound-bytes and fashion rather than understanding. I link to Sam's book in the description and meant to reference it in the video, but didn't want to appear to put words into his mouth. Sam's book is a very good book on the topic.
Microservices is the most convenient model for cloud providers profitability. You’ll end up buying much more api gateways, load balancers, monitoring, databases, virtual machines etc than normal services
I really appreciate all of the work you do to provide clarity and context to the field of technology! As of recently, I landed my first full-time job as a software developer, so this is providing excellent information as to how I can pursue my career with a strong foundation. Due to being early on amidst my CIS education, several of these topic areas are new to me which provides its own set of difficulties regarding comprehension, yet I always find a plethora of solid resources from these videos to help elaborate on any confusion. So, again, thank you for setting me in the right direction.
Thankyou, and congratulations on your new job. I have a new video coming out in a couple of weeks that is intended to offer some advice for new developers, so I hope that you enjoy it.
Although I am a software developer with 22 years of experience I am new to microservices and this video really helped to improve my understanding, so thank you very much. My immediate conclusion, taking Melvin Conway's wisdom into consideration, is that an organization that wants to adapt to a microservices architecture should be willing to reorganize accordingly. If the management of the organization is not aware of this, the campaign of migrating towards microservices will fail (at least initially). So business consultancy (of good quality) should be involved to make this a success. What I have often seen, however, is that new paradigms or technologies are presented to managers as a panacea of all their current problems without any big effort on their part (just buy our product or pay our consultancy fee). Do you have any thoughts on that? How can a senior software developer advice or influence the management of an organization to do the right thing? (either endeavour a reorganization or recognize that microservices are not for them, yet)
Over the years I've seen many people making pretty abstract pictures of systems that are build up from loosely coupled independent modules, but I have never seen it actually work in the real world. In the real world, users want complex screens that gets data from all over the place with complicated rules and calculations, living in different modules and all of this needs to deliver results in a short time frame. And for all the talk of keeping your architecture clean, there is often just too much pressure and too little time to consider these factors. In the end, you always end up with a tightly coupled spaghetti mess that is twice as hard to maintain than if you just did everything in one project. Maybe there are people out there that somehow managed to do it right, I just haven't ever met them.
That bell sound is so loud and ridiculous, and it always makes me laugh when I hear it. I could just listen to these videos on a loop all day and never get bored.
Graduating college and entering the workforce. University, by no means is useless, but I feel has not prepared me for a lot of the ‘ecosystem’ around development. Courses focus on languages, algorithms, data structures, operating systems, but never any courses on real-world problems, and insight from dealing with those problems. These videos are invaluable to me as I start the path of actual software engineering! Not to mention being so much more digestible. I just saw a video called “Why I don’t use else clauses”. I respect the hustle, but it’s refreshing to get real advice in a legible way from someone who isn’t a year older than me.
Don’t feel the college courses are useless at all. I am a 20 year veteran and I my time languages and frameworks have come and gone, but at the end of the day the base that I got in my CS degree with algorithms, data structures, compiler theory etc has not changed. The change is in how you do things. So that college edu is super useful over the long term.
They have their place, but "loosely coupled" is often a panacea in an enterprise. The difference between poster-boy microservices and those typically found in an enterprise, is persistent data, and the fact that the organisation wants to treat services (and their underlying data models) as autonomous and loosely coupled one day, and as one cohesive data model the next. When someone demands a report that spans service data models, they don't want to hear about artificial boundaries. When the domain changes, it often requires API changes across multiple services, similar to adding a new field on a screen, which ripples through all layers of the architecture, all the way down to the database. Microservices, just like "components" of old, are typically best split along the seams between different levels of abstraction in the architecture.
Totally agree: The most important part in using micro service architecture is organizational. Specially in context of independent agile teams these days. 😅
Sharpen your modern software engineering skillset with Continuous Delivery's online courses, presented by Dave. Whether you're a developer, development team leader, or an entire organisation striving to provide quality software... There is no better place to focus your efforts inl earning to build better software faster. TAKE A LOOK HERE ➡ courses.cd.training/
Thanks for the video. The idea of a ‘distributed monolith’ is what i find intriguing and I think it is what we are building where I work although we ‘think’ and ‘say’ that we are building micro services. The benefits however are still those of a distributed system: scalability, resilience etc, and maybe that is all we really wanted. :)
I really love this sentiment that the reason for using microservices should be driven more by a people/organization strategy rather than technical reason.
The way I have thought about micro services (granted without actually diving into it and only using intuition) was that it was small programs with a very distinct responsibility. Then several of these were connected together to get more elaborate behavior. Although it doesn't have the "distributed part" (even though it actually could now that I think about it) I felt the Unix or Linux is actually built using parts of that philosophy. Instead of monolithic programs the core is made up of small programs, with defined responsibility which you can then chain together (pipe) and have them transform data. But I might be way of and have misunderstood it completely... or at the very least way of from what people mean by it today. Ok, everything above this I wrote before watching the video. After having watched it, I still feel the philosophy of for instance Linux fits pretty well. Most applications (especially core ones) are small, they tend to focus on one task, they seem to fall within alignment of a bounded context, they are indeed autonomous, independently deployable and loosely-coupled. For instance, let us say we want to know the number of files in a directory recursively find /mydirectory -type f | wc -l We run the find command and tell it to find files in the given directory, we specify no limit in depth so it will return a list of all found files. But this is not what we are after, so we pipe the resulting data into the wc program (word count) and asks it to return the number of lines, based on the data we passed to it. This will now give us the number of files. But wc could just as well have given us the number of lines in a document, or the number of characters returned in a stream of bytes. Each little program has a defined responsibility and strung together they can make seemingly new functionality appear. A bit like functions in a library or something.
Yes, that is a good way to think about them IMO. The Unix model *is* not just an analogy, but an application of the same thinking. The key idea that I see so many teams miss completely is, again rather like Unix, you don't get to test every program with every other to see if 'piping' works! You test 'piping' in abstract, but you don't do integration testing with every Unix command ever. Microservices is defined by the idea of "independent deployability". That isn't my addition. What most orgs do, that claim to be building microservices is build a distributed monolith. Which is fine, if you recognise that is what you are doing, but the way to optimise the approach for a distributed monolith is different to how you optimise for microservices. So most orgs that I see are gain none of the benefits of either microservices or distributed monoliths.
This video added more value than anything else I found up to now. The motivation brought with the referenced quotes gives a soul to any application developed with this mindset. Great work and thank you!
Essentially he is advocating contract testing to ensure interfaces aren't broken. It would also be good to abstract out discovery. security, monitoring tasks into a common service mesh layer and let microservices focus on business rules implementation.
Hey Dave, great video. I've often heard the "2 week rule of thumb" for rebuilding a microservice, but one thing that was never clear, was whether that was an individual doing it alone, or a team? Be great to understand that. Thanks for all the videos and contributions you've made!
Philip, I think the timeframe is 3-10 business days for a small team (3 people). I’m just guessing on that but this is just meant as a rule of thumb. The idea is that you dont want more than say 150-200 hrs of work to go into a microservice. After that you are really talking about medium-services 😂. Smaller is better here. Another rule of thumb I have heard is that you want no more than 6-10 methods/endpoints.
The one big problem with microservices within business is the fact that whatever small autonomous program you are making probably has an open source solution out there already.
Why are we constantly trying to reinvent the wheel in programming? Its impossible to keep up with everything! We never master one thing before we are pulled to another.
I think that constant experimentation is a good thing. Our paymasters, the business people are constantly looking for new a better ways of capturing value (here Amazon is a good example). We need to be able find new ways of building, expanding and maintaining the systems that support them. However, also believe there is too little time is put into evaluating and learning when a a new tool comes along with a slick sales pitch and the promise of easy wins.
Mostly because the infrastructure reality constantly changed. Most of the principles of programming were already "set" in the 60s and 70s: functional programming, structured/procedural programming; object-oriented programming; modularity; etc; they were all figured out before the 80s. However, the technological reality then was only that of a single mainframe or a single personal computer; a single operating system; etc. There was no internet, there were no web browsers, there were no smartphones. The organizational/business realities were different too, you had a specific team in IBM/Xerox/Bell Labs/etc with a small team of developers and that's it. Now you are expected to have the scalability David mentions to become a big company in the tech market. The organizational reality, as well as the infrastructure reality changed, even if the principles are the same. So this necessitates "reinventing the wheel" in some form.
@@gonzalowaszczuk638 Hypertext was demoed in **1968** when Douglas Englebert gave _The Mother of All Demos_ before that hack Tim re-invented it for the THIRD time. Most concepts in Computer Science get re-invented every 20 years.
The biggest problem I see with microservices is how they deal with the underlying data layer. The problem with data is that as your software gets bigger and bigger - so are the relationships between the various data entities. Since each microservice is deployed and versioned separately - this means that you will not be able to utilize these data relationships effectively as each microservice needs to has its own small "data island" that is separate from all other data parts. This usually reduces your DB queries to the most primitive single table SELECTs - which means you will not be able to enjoy all the progress done with relational databases in the last 40 years or so.. this means no advanced JOINs, statistics, stored procedure etc. Unless I'm missing something here - in which case I'll be happy to see how this problem is being resolved in big microservice installations..
Part of the definition of "Microservice" is that they DON"T SHARE DATA STORES with other services. So this shouldn't be a problem. Microservices are responsible for their own storage needs. They only share information through their public APIs, whatever form those take. DBs, of any form, are specifically ruled out in all of the definitions that I have ever seen, as a means of data-sharing. See Martin Fowler & James Lewis' description, in which they describe this as "Decentralised Data Management" martinfowler.com/articles/microservices.html
@@ContinuousDelivery from my experience - using compensating actions and avoiding database transactions can become extremely complex very quickly. Definitely not something i would use as my normal design methodology. Im surprised they are so casual about something so problematic as maintaining data consistency.
@@gppsoftware These things though are both part of, literally, the definition of a microservice. You don't share databases and each service is aligned with a bounded-context. So in your example, it is fine for you order processing service to have a reference to a customer, but order processing is a different bounded context to customer management, so the nature of "customer" is different in each of these contexts, My guess is that all you need to represent a "Customer" in order processing, is a customer id or similar, enough to register the idea of "This Specific Customer" so that you can ask the Customer Service for any more detailed Customer information that may be required, this de-coupling is really the hole point. So what you are really saying when you say "the vast majority of microservices I have seen actually share a common database and data model behind the scenes" is "the vast majority of microservices I have seen WEREN'T microservices" which is true for me too. Most teams get this wrong!
@@gppsoftware The problem here is what do the names that we apply to things mean if we ignore the definitions that go with them? What is the use of calling something a microservice if it doesn't meet ANY of the criteria of a microservice, at that point it is merely a "service", and a poorly designed one at that. I agree with you that the practice is to not build microservices, but I don't think it helps to accept calling them microservices if they are not all of these things: independently deployable, aligned with a bounded context, Decentralised data management, Loose coupled.
@@gppsoftwareWell that is part of the definition of "bounded context" too, Eric Evans says that you should ALWAYS TRANSLATE data that crosses between bounded contexts so that you can 'adapt' (as in ports and adaptors) the information that flows across these critical boundaries.
So simply put forward, I guess for a start-up projects, the loosely coupled systems will cost more in the terms of integrity. Agreed, microservices must ne created as separate micro - independent modules to the same organisation. You made it so simple ❤❤❤
I think a couple other points that destroy microservice implementations are a tendency to break services at business unit boundaries and a tendency to delineate horizontally almost exclusively. I think you touched on the first a bit in the video indirectly when talking about how systems mirror the communication channels of the organization, but maybe not as explicitly. On the second point, organizations don't tend to build services that, as you said, do things like storage, but rather each business domain service implements it's own (and often multiple) storage subsystems.
Great content Dave and well laid out. Your trajectory of your career is basically the same as mine, I could steal that slide to explain my journey. One difference is in the early to mid nineties I spent time on shrink wrapped software and then operating systems. The one caveat I would say to your whole thesis is just that there is no silver bullet architectural pattern. Micro services are great for the reasons expressed but ONLY if it solves the problem domain in an elegant and sustainable way. Every pattern has its use and I’ve seen teams try to use micro services to “evolve” legacy processes like batch jobs to “modernize”. Guess what? The Batch job, even in this century is still a legit pattern in some instances. There is no one size fits all and in my experience blending or cherry picking the past of different patterns is often the right thing to do. Religious adherence to a pattern can make your system more complex if you first don’t explore multiple options and aren’t predisposed to have to design with the currently fashionable pattern or technology. My favorite example of this faux pas is XML. I’ve had to fix many systems because everyone assumed XML as a data interchange was a requirement for a good system. It only slowed things down and caused tighter coupling with dtds, parsing code, and made interchange with non xml systems a nightmare. Nobody likes to write XSLT let alone debug it. New subscriber here.
Long-time software engineer - I think my main issue with "micro-services" as a concept is that it tends to encourage OVER-abstraction. The distributed nature - as you pointed-out - is MORE complex by its very nature - couple [sic] that with over-abstraction, and it can get ridiculously complex to do the simplest thing. Further, if you have a user-interface that actual humans have to use - then you have at least ONE customer of your microservice that DOES have to know all the innards, effectively - or certainly the data models. As you might surmise, I'm more of a data-driven engineer at this point. Data, in the REAL world is NOT an abstract object. In the REAL world, an "object" is composed of many other "objects" in particular patterns, and is damned hard to work with in an over-abstracted form. Basically, the simplest order form uses both composition, aggregation, association, etc., and basically - has to know about it. It can then interact as necessary with whoever supplies the pieces - BUT IT HAS TO KNOW at some point. This over-emphasis of object-oriented design gives rise to the popularity of no-sql databases because it's a LOT simpler for engineers to encapsulate a complex object that way, than to fragment-and-reassemble them using a relational provider. Lastly, when testing, each ultimate consumer of various microservices MUST be called-on before anybody would consider a service to be fully tested. Some would say that a microservice is a well-defined black box, and if it passes those tests, it's fine. But in the REAL world, that isn't my experience.
Ye right, the real reason of splitting services into seperate rest projects is, gaining an ability to run a separate service in a seperate processor and cache/ram in a distributed way which allows you to speed up your processing time to respond to large scale requests. Thanks for reminding us about SOLID principles and applying it to Microservices.
Very good video that captured a lot of the discussions I've had surrounding misused microservice terminology very well. Just because something isn't a pure microservice doesn't mean it's bad, but it does mean you should find a different term for it or people will get the wrong idea.
What a brilliant video! The decoupling between services brings so many questions to me (that really likes modularized monoliths and haven’t worked a lot with MS’s). How to handle relations in between services, if at all? Are people storing IDs and just hope for the best? How to tie the pieces together? Via API GW? or independent calls to each service? Wouldn’t that be considered coupling since they’re no longer independently deployable? Would it not only be really decoupled by letting each service being complete (with UI and everything)? So many questions!
Thank you. Yes it does raise a lot of questions. I think that one of the secrets is to think very clearly of the "protocol" of information exchange, being separate from the service implementation.
I’d love to see a video like this in regards to web development.. it’s getting crazy with all these JavaScript frameworks, package managers, node, yarn, etc.. they compile and create a project folder filled with so much code and very little organization or explanation of what it all does, you basically have to just hope all the dependencies do something of value..
For real! I'm currently trying to marry a quasar Vue 3 composition api project with a vuetify Vue 2 class based components api project. They share some fundamental logic and architecture that I'm trying to refactor out. Wish me luck!
@@emaayan I get the idea, but in that case the real question is: why do you need a complex framework for a simple "hello world" app, you should try to think about mvp and yagni to avoid getting lost in insignificant details (and following the videos mr. Farley and other software engineering gurus put out is a really good direction for that).
Indeed, Domain-Driven Design is a fantastic book. For me, the alignment of microservices to Bounded Contexts is a prerequisite for that microservice being "independently" manageable. The translation of concepts between microservices (specifically those belonging to different Bounded Contexts) is no small task, and either requires close coordination with those familiar with the microservice being integrated, or for the other microservices' API to have outstanding documentation. An organization pursuing microservices for the right reasons would presumably be happy to ensure such communication were adequately invested in. One other potential benefit with microservices, which Eric Evans suggested 5 or so years back, is protecting model integrity with hard boundaries. With a monolithic system, there is little to stop a developer adding a new feature from reaching into another module's database and creating some new point of coupling between the modules. While in theory a monolithic system can be modular, as Evans noted, decades of experience suggests that that isn't how they tend to work out over time. Here the key thing for me is that these software components (not necessarily "microservices" as defined in this video) have physical boundaries to help keep the module's conceptual integrity protected. At a minimum, protecting the module's database access credentials as if they were akin to ssh private keys would perhaps go a long way to achieving such aims.
Great video! One issue that can be tough to resist with these things is when scope creep by changing needs gets forced onto the developers and breaks the autonomy aspect. Would love to hear your thoughts on how to address scope creep when trying to keep your projects clean and modular.
I think that you can't fix things forever, over time, the ideal division of responsibilities in code, and in teams, will morph. A thoughtful organisation will take this into account and re-assess team structure, and software architecture over time. These are deeply related ideas, and I have a video that gives one take on the org challenges here: "How to build big software with small teams" ruclips.net/video/cpVLzcjCB-s/видео.html At the more practical, tech level, I think that as you discover new features, you decide if this fits within the current scope of your services or modules or if you need to add new ones. If you add new services, then at some point you need to decide if you should spawn-off a new team to look after the new services.
YAGNI is the biggest problem I'm seeing with microservices. You describe (very rightly) how it should reflect the organization, that's a very good test to tell if it is applicable. I've seen a single team of 6 people try to develop a system consisting of 40+ microservices. Because 'it's what Netflix does', never considering the fact that they where building a niche line-of-business application and not a million users online platform. So they took all the costs, but the benefits didn't apply to them. (And on top of that they got all the boundaries wrong, because they never took the time to properly understand the domain.) Like so many things in software development microservices are a perfectly valid solution for a problem you may not have. But it's often touted as 'the one true way' for everything. There is no silver bullet...
I have a few videos that you may find interesting: Reactive Systems - ruclips.net/video/eRxLfUIMJwk/видео.html How Fast is Your Computer? - ruclips.net/video/0reMVgn6kRo/видео.html The Next Big Thing in SW Architecture - ruclips.net/video/-IZlOPciSl0/видео.html
Great video. I’ve found one of the biggest problems with the actual implementation is modelling transactional consistency boundaries and sharing data. This is where I’ve found teams can really get tripped up. Do you have any videos or advice on these?
Indeed, this is a tricky problem. The SAGA pattern is designed to solve it. I have not actually seen it implemented within my organization, but I'm sure there are plenty of companies that are effectively utilizing it.
Working on something and just found this great video. Thanks for it. I was already using his advice on 12:41 and will continue to keep it that way until the need grows. If this is not late, maybe advising on how to handle cases on communication between modular designed services will be awesome.
You need to start with the business process and identify the tasks and events that trigger the processes. Micro service should map to the discrete tasks in a business process . An example would be the booking service for a hotel reservation system
Great explanation of what it is and what it is not a microservice, would be nice to add or to discuss the different approaches to actually make good decitions in the design of the bounded contexts and each microservice in more depth.
Great explanation - like! One comment though. You say that the APIs should be considered "public" - "be defended and rarely and with great caution changed". While this is definitely true and is a very important point - we should maintain backwards compatibility in a distributed environment, and give our consumers time to migrate, there's still a spectrum in the "publicness" of the API's in regards of changes. A microservice API can be internal to a team, and therefore retain the ability to evolve very quickly and with very little communication or compatibility overhead. Next, an API can be a dedicated (non-generic) communication line to a selected and curated group of consumers, which again decreases the costs - especially if those consumers are agile enough. Finally, an API can be considered public - like what you described - with the maximum costs of maintenance and thought up-front. Wouldn't jumping into defining and building the last - most expensive - type of the API's before the architecture has matured and the organization evolved to deal with microservices - wouldn't it be a bit of a risky proposition?
Thanks man that's really a confusing idea I've struggled with. It's not easy to differentiate between SOA and microservices. I think it would be very interesting if you give examples of implementing microservices wrongly and explaining what is wrong with it. Wish you all the best thank you again
Eric Evans, who wrote "Domain Driven Design" is a friend of mine, he says that if he was writing the book today, he would put Bounded Contexts earlier in the book so that they were more prominent. DDD is not an easy book, but one of the most important IMO.
@@ContinuousDelivery I completely agree with you and his assessment. Also, I dont like that DDD is so often only applied to cloud solutions. It’s merit to cloud implementations is without doubt, but its overall contribution to technical solutions implementations is far more vast than cloud.
I certainly consider myself to be lucky to call Eric a friend, is is a very nice person, and smart enough to always have interesting and sometimes surprising ideas. I have been speaking on the international conference circuit for many years now, you get to meet a lot of interesting people that way.
I think that DDD is regaining popularity, it is quite an old idea, but Eric captured it in more detail than before in his book, which significantly pre-dated the birth of the cloud. I think though that the cloud has given it new focus, because if you are commoditising infrastructural concerns, which to some extent the cloud does, then it allows you to staty thinking more clearly in terms of separating "essential" from "accidental" complexity. Which is some of what DDD talks about, in a slightly different way. I think that there is a next step beyond that too. I have done some work on async, event-based systems, I was one of the authors of something called "The Reactive Manifesto" www.reactivemanifesto.org/which describes them, and then you can achieve an almost pure separation of accidental and essential, your services become almost pure domain model. A lovely way to work. I think that moves towards something called "stateful serverless" may be one to watch in that space.
@@ContinuousDelivery The main problem I have with microservices is that I don't generally have a bounded context that's small enough to implement in a few weeks. I'm usually dealing with dozens of entities, each with a rich set of constraints and that just takes time to do right. Does that mean that these domains simply aren't amenable to microservice approach, or do we flex a bit and say that if it's one bounded context, it's still a microservice, even though it might take 3 months to do a rewrite?
One problem I can see (non developer here) is that the service will bloat quickly in order to maintain backward compatibility. I.e. You call service with params a,b,c and get back value X. You update the service to be able receive param d to return value Y, but you will still have to maintain the old code to ensure compatibility.
Very well explained, as an absolute beginner in software development, I find your videos easy to understand and thought provoking! Look forward to many more videos from you :)
A great video & at least every architect, who thinks that they design microservices applications, must watch it. As I believe many such good thoughts are talked about in many sessions in the organisations but many times the same ethics/thoughts go down the drain in their projects, as money is involved. If profits are flowing in & quick delieveries are made then no-one cares if these concepts are being followed or not.
- Microservice is NOT all about small services. - Microservice is NOT just a rest api. - NOT all services are microservices. - Microservice is a distributed system architecture. - Microservice is small: - fits in my head. - can be rewritten from scratch in a week or two. - Microservice is a single responsibility and does it well. - Microservice is aligned with bounded context: - A bounded-context is a defined part of software where particular terms, definitions, and rules apply in a consistent way. - A bounded-context is a cohesive unit. - when crossing a bounded-context, you need to translate the data. - Microservice is autonomous - progress can be made without waiting or coordinating with other teams. - this is the real value of microservices. - Microservice is a single deployable unit - you should not have to test another service when you deploy a service. - Microservice is loosely coupled. - make external & internal dependencies explicit. - interface to a microservice should be public. - when consuming api of another microservice: - you should not have to know the implementation details. - use minimal data. so that you can reduce the coupling.
I've been using Rails and never been happier. Yeah I have a monolith, but it's not a problem. I have 2 micro services for mail and message queuing. I don't need anything else. I was a node developer, and used microservices heavily. They're not a bad thing, but they're misused today. Martin Fowlers enterprise architecture is an excellent read.
I don't quite agree with you on the assertion about what "small" or "micro-" means, i.e. that a microservice should be fairly easy/cheap to throw away. As you pointed out, microservices have meaning in the context of distributed systems. If it were enough for a microservice to be easily disposable, then any cheap-to-develop module in a monolith would be a candidate for becoming a microservice, if we were to turn a centralized monolith into a distributed system based on microservices. Modules too are supposed to be autonomous with low coupling and high cohesion (i.e. bounded by domain context). But that's not the case. A microservice should be small, not necessarily from a development effort/cost perspective, but from a deployment and scaling perspective. A microservice MUST be easy to scale up and down. It must be easy and cheap to deploy and that naturally translates into favoring them being geared towards a single task.
The problem with Microservices, is caused because the people think Microservices are an architecture, Microservices not are an software architecture it’s part of it, but Microservices are more like a deploy strategy, the architecture of the software are clases, layers, and your deploy strategy like Microservices, the people who thinks Microservices are an architecture is because they doesn’t know anything about clean architecture. For example I have a project in my GitHub that can be deployed as a monolithic or as a Microservices applications, it’s because the real thing that we use to decouple layer, not is the deploy strategy, are the interfaces and how your system interact with that interfaces and clases.
Recently come across your channel and I already feel like you've been living in my head for the last 10 years! Great videos. I haven't heard or seen (yet) you talk about consumer driven contract testing (like PACT) when working with micro services, do you have a video on it? I feel it adds a lot of value in the space!
I talk a little about that in a few videos, I talk about the principles here: ruclips.net/video/cpVLzcjCB-s/видео.html I probably should do a video more specifically focused in this topic though, thanks for the suggestion.
@@ContinuousDelivery Thanks for the reply, Ill check that out and look forward to the next. On another note, would love to see some Q/A live streams on the channel..
thank you. in terms of things I'd like to hear about - you mentioned how software design ends up modelling the organisations communications - I have encountered strong coupling between org culture and development culture - especially as a road-block to change. It's bleedin obvious to me that say moving from monolithic software to distributed software takes cultural change within your development team (eg. moving from test/deploy to CI/CD, or agile, or DevOps) - and communication/cooperation/coordination between teams needs to increase significantly... I'd love to hear about the sorts of approaches that make this change possible, success and failures, and so on - not so much a tech question I know but damn if it isn't the core of the problem...
Did I hear this correctly Dave? Your favorite architectural style is the distributed monolith??? (13:00) How can you do CD with a distributed monolith? You can‘t IMHO. I‘m confused 😅. Please clarify. Thanks! Beside that: AWESOME episode! 🤩
Thanks 🙂 Yes, I have been building service-oriented systems for a few decades now, it is much my preferred approach in terms of architectural style, though naturally I try to fit the solution to the requirements before picking an architecture - there is no "one-size-fits-all" option. When you start out on any complex system, you don't know where you will end, you will grow your system as your understanding grows. At least that is how I work, and think is the best way to create complex systems. In the early days you will be exploring more and making more mistakes, it takes a while for your design, at the level of services, consolidate. You will find that some of the service interfaces, even if you got the service boundaries in about the right place to start with, change a lot as you refine the responsibilities of the service and evolve the cleanest interfaces. During this period there is no benefit to a microservice approach IMO. A much better strategy is to bung everything in one repo and build and test it all together. Th HUGE advantage of this approach is that you can "build and test it all together" there is NO DEPENDENCY MANAGEMENT of any kind. I can change the interface to my service, and update all of its consumers in the same commit that I make that change! The main downside of this approach is that you have to be efficient enough in your building and testing to get answers back on CD timescales, so under 1 hour for everything. It also means that the pieces aren't really "independently deployable". This approach doesn't stop you having separate, reasonably independent, small teams though. This how we built our financial exchange at LMAX, and it is how Google and Facebook organise too. It is surprisingly scalable! As I describe in the video, microservices is an organisational scaling strategy, nothing more. It limits the options for optimisation, because each service is discrete. It means that you have to extra work to do to facilitate the "independent deployability" and so on. It also demands a higher level of design sophistication to keep the services separate and "independently deployable". My advice, and Sam Newman's advice who wrote the most popular book on microservices, is to begin with a distributed monolith, and only, once the interfaces have stabilized, move to microservices.
@@ContinuousDelivery Thanks for the answer Dave! But don't we usually start with a "single process monolith" instead of a "distributed monolith"? (Sam Newman "Monolith to Microservices" p. 12 - 14)
@@PetiKoch Ah, I see what you mean. Sure, maybe, depends on the application. The way that I build my services, that is more of a deployment-time decision than an architectural one, which is why I don't think of it that way. The danger of starting with your monolith as a "single-process monolith" is that the other description of that is just a bunch of code! Unless we are writing something that we KNOW will be throw-away, then I think that the guiding principle for good design, even for small systems, is to manage complexity. So I want my design to be modular and cohesive, with good separation of concerns and as loosely-coupled and well abstracted as seems to make sense at the point that I write it. So I like the idea of "services" as a modular unit. Now, what I mean by a "distributed service architecture" is that I don't care if the services are running on the same machine or on machines on different parts of the planet. That means that the comms mechanism, and my design, needs to allow for the case where I want to make the services non-local. My favourite way to do that is to make the interfaces to my services Async. Now it is a deploy-tome decision wether I optimise for simple, local, fast, comms or distributed, non-local, comms. I have separated that comms concern. But this only works if I assume that the services are non-local. If I assume they are local, I may become addicted to the local performance, and then when I try to distribute them later, when my system needs to scale, my architecture is no longer fit for purpose. You could argue YAGNI, the trick here is to do this in a way that makes the overhead of this "thinking ahead" low-enough that its not really a problem. That is a matter of experience and design-taste I suppose, but we got that right when we built our exchange and I have used that approach a few times since. There is some more stuff on the architectural style that I am hinting at here: martinfowler.com/articles/lmax.html and here: I was involved in creating something called the "Reactive Manifesto" to describe some of the properties of this async architectural approach. www.reactivemanifesto.org/
I find one of the bug bears with Microservices in my experience is different teams using different tech to build their services, creating a huge range of different implementations across the estate which confuses the hell out of other teams who sometimes have to support these services. It's important to decide what to use from a curated tool box of tech that the teams agree on, so at least working with each service isn't a complete nightmare.
That whole bit at the end about defending interfaces and changing them rarely. My experience is that going heavy handed early on in a project with pact for example takes an inordinately heavy toll on development speed.
I feel like the word "micro" is what's causing misunderstanding for many. Because, in fact, you're building very real services. Then you build services that use your existing services. It's a very simple idea, but people look at it like it's something magical and that's only thanks to over hype and marketing. Exactly same thing happened with dockers/containers/"cloud" as well. I am personally not at all amused how we always go this same route with every technology. "This thing is so cool and you're just outdated"
Great video. Unfortunately, in my experience (so far), Microservices have been used basically to let the backend developers say “I can’t do this obviously simple thing because "microservices" ”. Meanwhile we are stuck waiting for something that used to be quick, and now it is not. The level of misuse of this concept is literally weighing entire companies down. I’m not saying microservices are not useful, but since developers themselves are doing it wrong, you get none of the benefit, and all of the cost.
Focus on the right parts of the problem when you are creating a new microservices system. Here's my FREE 'how-to-guide-book' offering advice on the Microservices basics to help you get started ➡www.subscribepage.com/microservices-guide
I’m a 30+ years Software Engineer, with clearly a similar path. This is extremely well articulated and should be on the ‘reading’ list of all modern software developers who want to understand concepts from a big-picture level. I don’t often comment on RUclips content but this is worth expressing thanks for a good, down to Earth presentation.
Every developer out there should watch this video. Orchestration and monitoring tools vendors made a lot of damage by trying to get everybody adopting microservices so they could sell their shiny tools. Then developers found themselves producing distributed entangled spaghetti aberrations just because nobody taught them what Dave excellently does here.
Thanks😎
Developers are usually very aware of the problems and find themselves producing distributed spaghetti aberrations because they are not the ones who makes such decisions. Typical victim blaming.
@@linuxgaminginfullhd60fps10 It seems to me that @Fran Dayz was saying tools vendors luring developers to purchase their product and then not providing proper support was the issue he had experience with. But you are saying that decision makers forcing those tools onto developers is the issue you are facing? I'm sorry, I do not understand how those situations are the same in a way that is offensive. How does any of that change the point made that Dave's explanation here can help avoid aberration creation in the first place?
As always: Don’t solve Problems you don’t have 😀.
Thanks a lot for your videos.
Besides the content I really appreciate the way you speak.
Somehow it feels that are using the words of the English language, optimized for simplicity and information throughput.
Expectations: Rant about microservices
Reality: Misuses of microservices and advices how to avoid mistakes
10/10 quality content!
Yes i was expecting the same but found gold 👌
Reality: Clarity on the essence of MS (mistakes and misuses come from that lack of clarity). Still 10/10
The same thing happened to me. 100% quality content indeed!
I agree. The title is misleading. It is very good content!
Copy that
Wow! With almost 25 years of professional software development under my belt, this was the first video/document on Microservices, which made sense to me. Everything else, and that includes most of the big software conferences, was just a variation of marketing bullshit. It is really fascinating to see that it always comes back to how we deal with "complexity". And that is not technical stuff, but the organizational and domain side of things. Thanks a ton!
Thanks, I am pleased that you liked it
YES! Microsservices is a solution to a problem that the majority of companies never had and probably never will.
@@rafaelrosa3841 it absolutly is, otherwise why would every big tech company use it ?
agree
Would have been even better if presented by Elon Musk
I have been interviewing with a lot of companies lately that want candidates with micro services experience. Unfortunately most of the companies don’t really understand micro services. They think it’s the NEW way to do development. What they don’t realize is that micro-service’s is just one of many ways to implement your project. Great video!
The explanation of the bounded context is better than any other I've heard.
It made me want to read& understand how NASA/ESA build the ISS modules, to see if these two different contexts share the same "Bounded Context" concept. The "Fire-Breaks" idea led me there.
This channel is amazing, thank you sir! I am arguably a seasoned engineer at this point but listening to you keeps me motivated to better myself constantly. I think will watch every single video of yours. Keep up the good work!
The way he explains things is amazing
Great to hear! One of my enduring principles is that we all need to keep learning! and to use software development techniques that help us learn. Thank you for the feedback.
I'm a 25 year Master Journeyman plumber...but somehow have become fascinated by microservices over the last few weeks.
I absolutely never plan on coding a single thing in my entire life...but really enjoyed this video.
Gives me a little perspective when I update some software and wonder how they introduced so many bugs with a simple small update...
It's remarkable how similar are system architecture and plumbing. Or, at least how I imagine plumbing works. I've come to believe that civil engineering falls in that group too. Basically anything that applies patterns and practices to the management of flow and complexity.
My development is mostly a private affair. I write code that I use, web interfaces that I am the only customer for. This content is still helpful, even for my hobby. This video got me thinking of problems that should be decoupled that aren't, and conversely convinced me not to decouple things just to make them "micro." There is no shame in coupling my front end with my back end code, for example in my latest node.js project.
Starting monolithic and transitioning to a microservice architecture seems to be a common route when the team GROWS- Because in an ideal world, each microservice deserves it's own team.
As a relatively new developer (2+ years) I really enjoy your video. Helps me to think at a scale larger than the code immediately in front of me. Thank you for this!
It's a pleasure, and thank you.
As a new dev, have you checked out my video on "Career advice"? ruclips.net/video/hjIlTaAMsbI/видео.html
Your experience shed light on exactly what I felt was off about people/companies claiming to be doing microservices but most definitely are not. This is an enormous problem for semantics and execution. Not everyone is on the same page. I really think of big monolith companies that want to decompose their design into "microservices", but in the end they will spend a ton of money for little to no benefit (ROI). So much time is often wasted chasing after 'trends' before thinking things through...
Yes I see a lot of orgs investing a lot for little in return too.
I don't think their chasing trends, their chasing ways to bring "complexity" somehow under control (probably because they did not pay much attention to that previously at an organizational level, which resulted in a lot legacy code/systems) without understanding all the implications of their decisions and approaches, and as Mr. Farley said, there are many other ways to do that besides microservices.
But the change is not because it's trendy, it is because the people involved feel the pain in developing and maintaining said legacy system. The company big wigs don't care that much about fancy code or architecture they care about money and they were ill advised that this new shiny architecture will save the system from crumbling from within (and also solve all world problems from global warming to the inevitable demise of human kind), and in the process they sometimes accelerate the decay of the system by over-engineering stuff (that also get polluted by the old way of doing things).
@@nerrierr I think you missed my point entirely by focusing on a single word. Microservices DO NOT attempt to bring complexity under control, it introduces more of it. The core purpose is for scalability and team environments to work on separate code bases in isolation but will work with one another when connected. Monolith and Microservice design are essentially antithetical to one another when done properly. But the point I was making was that companies using monolith design are decomposing their code into microservices for the wrong reasons and doing so in a hybrid way which gives little to no benefit for all the increased difficulty. Doing so is following a trend: see technology adoption lifecycle.
@@destroyonload3444 fair enough, but to be honest that still is directly related to the complexity and requirement of the project. You don't scale and parallelize if you don't need to (ex: for a simple console application that just prints a simple message you will not do it in a microservices architecture no matter how misguided you are). As for the fact that it adds more complexity it is like saying that abstraction/encapsulation just adds more classes/methods without a benefit (instead of helping you in the grand scheme of things, nothing is free in life and in this case you pay some extra code "locally" to save your a*s some place else, and you do it because of the complexity of today's systems). In the end I think we agree that you need to pick the right tools for the job and for the right reasons, the difference is why we like/dislike microservices (in that sense it's probably relevant to what we are comparing it to). And that is perfectly fine, no team or project is the same in every aspect and people have different opinions based on their previous experiences (there's a reason why giant companies recommend it, and in fact only they are, but most companies aren't Netflix nor is their product of the same scope/context) (well one of the reasons is obliviously practical and the other is probably more marketing focused but that's another topic, I guess we all have to earn our bread somehow :) )
@@nerrierr right. You still have abstraction in microservices, but the real complexity comes with multiple middlewares and dealing with events. Statistically, I have no idea what the most used implementation is, but the most I hear about is the use of an event server and eventually consistent design. I think we're more or less on the same page. BTW, I watch this channel often because his methodology seems to make the most practical sense in implementation, and he's been through several major eras of programming milestones. I find it funny that people are still trying to reinvent what he already had working in the 90's! I watch a lot of Computerphile, too. The down-to-earth "old timers" have my utmost respect.
The only software engineering processes and practices related content that makes 100% sense.
I've been a part of multiple discussions around such topics like software architectures, design patterns, agile development, behavior/test driven development etc...none of them make as much as sense as much as your content does.
Dave, please continue posting such videos. I want you to know that your experience, talks, and discussions make lives easier for many software engineers worldwide! 🙂
My key takeaway here is this: the API (as in the names of the calls, the expected arguments, the expected results, etc) should be the first thing to be designed and agreed upon when the analysis phase is over. On top of that it must be cast in stone, immutable and stable during a development cycle. Finally it must be the first thing that will be mocked during implementation so the service consumers have something to work upon.
The mentality to adhere to all of this in an agile environment where there are constant releases every e.g. month or so, this is the real fight, this is the real issue, the fact that people must put down some communication constraints, draw a box and remain in there, no "well, you see, the customer also decided they want this and this and this so we are mutating the API definition".
A mini lecture on how to keep people in check (both on customer side and on the PM/TechLeads side), now, that would be a treasure!
I think having any design that is inmutable and cannot adapt to customer demands should be a definite no-go for any serious software project.
Instead, having stable APIs means that once you create an API, you have to keep it running. To add new features, you introduce new APIs without breaking the old API.
Some of the old APIs may be implemented as wrappers for the new API but the consumers of the microservice API don't need to mind about that.
@@MikkoRantalainen Introducing a new API for every change isn't practical either. It leads to growing legacy code base, as it's problematic to find out when it is safe to delete some old API version. That's why big companies end up with monorepos, as in a monorepo there's more control over the consumer's code, as far as the API does not escape the monorepo.
@@НиколайТарбаев-к1к I'd argue that if you continously modify the API in a way that requires consumers to be modified, too, you don't actually have an API at all. You just have fully custom protocol instead.
I've watched a number of your videos now and they are brilliant. Straight talking and oozing knowledge and wisdom. From now on, part of my decision process is going to be 'what would Dave Farley do'
Thank you 😁😎
As a person, who made microservices before people started calling it that way I am happy for this video. Separately deployable is something that I see so many people out there do not grasp. What I do not grasp however is sometimes how heavyweight the infrastructure is around real microservices! In my team I did not use any kind of infrastructure just plain REST and pretty much plain javascript (not even js frameworks) to implement these ideas and yet there was a clean way to achieve copy-paste kind of integration even on the frontend side - and by that I literally mean the code was just copied into its place and was magically working. Never saw a lightweight solution like that later, but only heavy solutions where they used some tool to "help" create microservices, but only then I felt the cost of it - not when we did the stuff manually without framework.
I think some movement like "lightweight micro services" should arise. Maybe it is already arisen, but not what I saw. The idea is that if you cannot set up your microservice infrastructure in a week with everyone on the team understanding all the details - then you are doing it wrong. This is similar to what you say about the microservices themselves and their sizes, but here I am talking about the support infrastructure that make the deployment and everything work - both backends and frontends of the microservices and the ability to publish them even when working together!
I hope such a thing exists, but as I said I only saw two things: we doing without any specific infrastructure OR some teams using heavy infrastructure that incurs more cost that simple projects can afford. This is a big deal because original company where we worked like that was less than 50 person and team around projects where we employed this was less than 6 person and still we did not feel at all "too big work" to develop in this way. Any yes again: it was completely shippable separately - even the frontend of these services were separately shipped and built up a complex web app.
Anyways: I think everyone who ever use the word by their mouth in any circumstances should watch this video. Would help a lot if people would all do so...
I deal with large deployments and this contents is spot on 100% accurate.
The idea of "bounded context" is so important. Whenever i discuss architecture with people i talk about my idea of a "module boundary" but i always struggled to explain what i mean. This description of a bounded context is exactly what I was looking for.
I clicked because I saw Captain Dissillion's logo.
One problem that I’ve experienced with microservices is that it is really hard to keep everyone up on what are the boundaries of each of you services… sometimes a lot of requirements come from the managers and they want it do be done in service A when it should be done in service B and not all employees know all of the microservices that well to be able to set those boundaries…
I’ve experienced features being done in a service and then it came up that the same feature was already being done in another service that is called way up in the architecture, way after it was already shipped to production.
I’ve thought about it if that’s a problem of not having so many people understanding that well the architecture or if that’s a leadership or management issue…
I feel like there’s a small inner system that developers are capable of understanding, it’s almost impossible to master the design/domain of so many of the microservices.
I haven’t seen anyone talking about this, I’m always wondering how companies like Netflix deal with that, like, how many other mircroservices do your engineers need to grasp besides the one/ones they are really contributing code to?
Netflix originally was a huge monolith. They then became microservices! This is the biggest mistake system designers make. You need to have a fully functioning monolith in production with very clear documentation and perfomance benchmarks before you even think about converting it into microservices!
I almost never comment. But I must. This is a must watch by ANYONE involved in solutions development in a modern organization. 10 out of 10. This should even be watched by the non-technical stakeholders. Well done.
Really great stuff! Thanks so much! Working as a management consultant I see my projects 'moving closer' to the field of technology every year. Thus, usually I can't deliver valuable results for my clients without understanding related technological practices and principles. Your videos are amazing in the way you are able to explain those technological issues! 10/10
Thanks Dave. As you mention, there is very much more to cover on this subject, but from a QA point of view the subject of contract testing needs to be raised early on in any discussion about microservices. Also there are some severe gotchas with the approach too. As with many things we need to do our upfront research or risk project failure!
Sam Newman's books contain a wealth of info.
Yes I am thinking of doing more stuff on design and architecture on the channel.
I don't think of it as "doing up front research" as much as understanding what the things that we do are, and how they work, we often we seem to jump on ideas based on sound-bytes and fashion rather than understanding.
I link to Sam's book in the description and meant to reference it in the video, but didn't want to appear to put words into his mouth. Sam's book is a very good book on the topic.
Great video! What about duplicated code?
Microservices is the most convenient model for cloud providers profitability. You’ll end up buying much more api gateways, load balancers, monitoring, databases, virtual machines etc than normal services
You can have your own cloud in-house on your own servers.
I can't with how valuable this content is
Thank you! I'm glad you find it useful
It is best explanation found so far. Thank you very much
You can't what?
I feel that the font used for the presentation points is the same font used in the Star Wars opening text crawls, and it is immensely appealing.
A long time ago, in a galaxy far, far away.
I really appreciate all of the work you do to provide clarity and context to the field of technology! As of recently, I landed my first full-time job as a software developer, so this is providing excellent information as to how I can pursue my career with a strong foundation.
Due to being early on amidst my CIS education, several of these topic areas are new to me which provides its own set of difficulties regarding comprehension, yet I always find a plethora of solid resources from these videos to help elaborate on any confusion. So, again, thank you for setting me in the right direction.
Thankyou, and congratulations on your new job. I have a new video coming out in a couple of weeks that is intended to offer some advice for new developers, so I hope that you enjoy it.
@@ContinuousDelivery Looking forward to it!
Although I am a software developer with 22 years of experience I am new to microservices and this video really helped to improve my understanding, so thank you very much.
My immediate conclusion, taking Melvin Conway's wisdom into consideration, is that an organization that wants to adapt to a microservices architecture should be willing to reorganize accordingly. If the management of the organization is not aware of this, the campaign of migrating towards microservices will fail (at least initially). So business consultancy (of good quality) should be involved to make this a success.
What I have often seen, however, is that new paradigms or technologies are presented to managers as a panacea of all their current problems without any big effort on their part (just buy our product or pay our consultancy fee). Do you have any thoughts on that? How can a senior software developer advice or influence the management of an organization to do the right thing? (either endeavour a reorganization or recognize that microservices are not for them, yet)
Don't work for pointy haired bosses.
Over the years I've seen many people making pretty abstract pictures of systems that are build up from loosely coupled independent modules, but I have never seen it actually work in the real world. In the real world, users want complex screens that gets data from all over the place with complicated rules and calculations, living in different modules and all of this needs to deliver results in a short time frame. And for all the talk of keeping your architecture clean, there is often just too much pressure and too little time to consider these factors. In the end, you always end up with a tightly coupled spaghetti mess that is twice as hard to maintain than if you just did everything in one project. Maybe there are people out there that somehow managed to do it right, I just haven't ever met them.
That bell sound is so loud and ridiculous, and it always makes me laugh when I hear it. I could just listen to these videos on a loop all day and never get bored.
Thanks, sorry about the bell, I am making it quieter in future videos 😉
Graduating college and entering the workforce. University, by no means is useless, but I feel has not prepared me for a lot of the ‘ecosystem’ around development. Courses focus on languages, algorithms, data structures, operating systems, but never any courses on real-world problems, and insight from dealing with those problems. These videos are invaluable to me as I start the path of actual software engineering! Not to mention being so much more digestible. I just saw a video called “Why I don’t use else clauses”. I respect the hustle, but it’s refreshing to get real advice in a legible way from someone who isn’t a year older than me.
Don’t feel the college courses are useless at all. I am a 20 year veteran and I my time languages and frameworks have come and gone, but at the end of the day the base that I got in my CS degree with algorithms, data structures, compiler theory etc has not changed. The change is in how you do things. So that college edu is super useful over the long term.
What a fantastic experience is to hear explaining theses concepts from someone who already implemented microservices in many contexts!!!!
I am glad that introduced microservices back to 2000. And after 21 years finally I found your video. Thank you.
You're channel is a gold mine for CS students, thank you so much for sharing your knowledge!
This channel deserves more subs and views ❤️❤️ Finally: A professional software content for everyone .... Greetings from Colombia.
Thank you very much!
Traigan el cafe berracos, que sea del weno.
They have their place, but "loosely coupled" is often a panacea in an enterprise. The difference between poster-boy microservices and those typically found in an enterprise, is persistent data, and the fact that the organisation wants to treat services (and their underlying data models) as autonomous and loosely coupled one day, and as one cohesive data model the next. When someone demands a report that spans service data models, they don't want to hear about artificial boundaries. When the domain changes, it often requires API changes across multiple services, similar to adding a new field on a screen, which ripples through all layers of the architecture, all the way down to the database. Microservices, just like "components" of old, are typically best split along the seams between different levels of abstraction in the architecture.
Totally agree:
The most important part in using micro service architecture is organizational.
Specially in context of independent agile teams these days. 😅
Sharpen your modern software engineering skillset with Continuous Delivery's online courses, presented by Dave. Whether you're a developer, development team leader, or an entire organisation striving to provide quality software... There is no better place to focus your efforts inl earning to build better software faster. TAKE A LOOK HERE ➡ courses.cd.training/
Thanks for the video. The idea of a ‘distributed monolith’ is what i find intriguing and I think it is what we are building where I work although we ‘think’ and ‘say’ that we are building micro services. The benefits however are still those of a distributed system: scalability, resilience etc, and maybe that is all we really wanted. :)
Man, this video has so much more quality content crammed on it's 17 min than a lot of books and courses.
I really love this sentiment that the reason for using microservices should be driven more by a people/organization strategy rather than technical reason.
The way I have thought about micro services (granted without actually diving into it and only using intuition) was that it was small programs with a very distinct responsibility. Then several of these were connected together to get more elaborate behavior. Although it doesn't have the "distributed part" (even though it actually could now that I think about it) I felt the Unix or Linux is actually built using parts of that philosophy.
Instead of monolithic programs the core is made up of small programs, with defined responsibility which you can then chain together (pipe) and have them transform data.
But I might be way of and have misunderstood it completely... or at the very least way of from what people mean by it today.
Ok, everything above this I wrote before watching the video. After having watched it, I still feel the philosophy of for instance Linux fits pretty well.
Most applications (especially core ones) are small, they tend to focus on one task, they seem to fall within alignment of a bounded context, they are indeed autonomous, independently deployable and loosely-coupled.
For instance, let us say we want to know the number of files in a directory recursively
find /mydirectory -type f | wc -l
We run the find command and tell it to find files in the given directory, we specify no limit in depth so it will return a list of all found files. But this is not what we are after, so we pipe the resulting data into the wc program (word count) and asks it to return the number of lines, based on the data we passed to it.
This will now give us the number of files. But wc could just as well have given us the number of lines in a document, or the number of characters returned in a stream of bytes. Each little program has a defined responsibility and strung together they can make seemingly new functionality appear. A bit like functions in a library or something.
Yes, that is a good way to think about them IMO. The Unix model *is* not just an analogy, but an application of the same thinking. The key idea that I see so many teams miss completely is, again rather like Unix, you don't get to test every program with every other to see if 'piping' works! You test 'piping' in abstract, but you don't do integration testing with every Unix command ever.
Microservices is defined by the idea of "independent deployability". That isn't my addition. What most orgs do, that claim to be building microservices is build a distributed monolith. Which is fine, if you recognise that is what you are doing, but the way to optimise the approach for a distributed monolith is different to how you optimise for microservices. So most orgs that I see are gain none of the benefits of either microservices or distributed monoliths.
This video added more value than anything else I found up to now. The motivation brought with the referenced quotes gives a soul to any application developed with this mindset. Great work and thank you!
Essentially he is advocating contract testing to ensure interfaces aren't broken. It would also be good to abstract out discovery. security, monitoring tasks into a common service mesh layer and let microservices focus on business rules implementation.
Hey Dave, great video. I've often heard the "2 week rule of thumb" for rebuilding a microservice, but one thing that was never clear, was whether that was an individual doing it alone, or a team? Be great to understand that. Thanks for all the videos and contributions you've made!
Philip, I think the timeframe is 3-10 business days for a small team (3 people). I’m just guessing on that but this is just meant as a rule of thumb. The idea is that you dont want more than say 150-200 hrs of work to go into a microservice. After that you are really talking about medium-services 😂. Smaller is better here. Another rule of thumb I have heard is that you want no more than 6-10 methods/endpoints.
The one big problem with microservices within business is the fact that whatever small autonomous program you are making probably has an open source solution out there already.
Why are we constantly trying to reinvent the wheel in programming? Its impossible to keep up with everything! We never master one thing before we are pulled to another.
If you learn techniques and not technology then you can bring 80% of it with you to the next thing.
I think that constant experimentation is a good thing. Our paymasters, the business people are constantly looking for new a better ways of capturing value (here Amazon is a good example). We need to be able find new ways of building, expanding and maintaining the systems that support them. However, also believe there is too little time is put into evaluating and learning when a a new tool comes along with a slick sales pitch and the promise of easy wins.
Not all wheels work well on certain cars and/or certain terrains.
Mostly because the infrastructure reality constantly changed. Most of the principles of programming were already "set" in the 60s and 70s: functional programming, structured/procedural programming; object-oriented programming; modularity; etc; they were all figured out before the 80s. However, the technological reality then was only that of a single mainframe or a single personal computer; a single operating system; etc. There was no internet, there were no web browsers, there were no smartphones.
The organizational/business realities were different too, you had a specific team in IBM/Xerox/Bell Labs/etc with a small team of developers and that's it. Now you are expected to have the scalability David mentions to become a big company in the tech market.
The organizational reality, as well as the infrastructure reality changed, even if the principles are the same. So this necessitates "reinventing the wheel" in some form.
@@gonzalowaszczuk638 Hypertext was demoed in **1968** when Douglas Englebert gave _The Mother of All Demos_ before that hack Tim re-invented it for the THIRD time.
Most concepts in Computer Science get re-invented every 20 years.
The biggest problem I see with microservices is how they deal with the underlying data layer. The problem with data is that as your software gets bigger and bigger - so are the relationships between the various data entities. Since each microservice is deployed and versioned separately - this means that you will not be able to utilize these data relationships effectively as each microservice needs to has its own small "data island" that is separate from all other data parts. This usually reduces your DB queries to the most primitive single table SELECTs - which means you will not be able to enjoy all the progress done with relational databases in the last 40 years or so.. this means no advanced JOINs, statistics, stored procedure etc. Unless I'm missing something here - in which case I'll be happy to see how this problem is being resolved in big microservice installations..
Part of the definition of "Microservice" is that they DON"T SHARE DATA STORES with other services. So this shouldn't be a problem. Microservices are responsible for their own storage needs. They only share information through their public APIs, whatever form those take. DBs, of any form, are specifically ruled out in all of the definitions that I have ever seen, as a means of data-sharing.
See Martin Fowler & James Lewis' description, in which they describe this as "Decentralised Data Management" martinfowler.com/articles/microservices.html
@@ContinuousDelivery from my experience - using compensating actions and avoiding database transactions can become extremely complex very quickly. Definitely not something i would use as my normal design methodology. Im surprised they are so casual about something so problematic as maintaining data consistency.
@@gppsoftware These things though are both part of, literally, the definition of a microservice. You don't share databases and each service is aligned with a bounded-context.
So in your example, it is fine for you order processing service to have a reference to a customer, but order processing is a different bounded context to customer management, so the nature of "customer" is different in each of these contexts, My guess is that all you need to represent a "Customer" in order processing, is a customer id or similar, enough to register the idea of "This Specific Customer" so that you can ask the Customer Service for any more detailed Customer information that may be required, this de-coupling is really the hole point. So what you are really saying when you say "the vast majority of microservices I have seen actually share a common database and data model behind the scenes" is "the vast majority of microservices I have seen WEREN'T microservices" which is true for me too. Most teams get this wrong!
@@gppsoftware The problem here is what do the names that we apply to things mean if we ignore the definitions that go with them? What is the use of calling something a microservice if it doesn't meet ANY of the criteria of a microservice, at that point it is merely a "service", and a poorly designed one at that. I agree with you that the practice is to not build microservices, but I don't think it helps to accept calling them microservices if they are not all of these things: independently deployable, aligned with a bounded context, Decentralised data management, Loose coupled.
@@gppsoftwareWell that is part of the definition of "bounded context" too, Eric Evans says that you should ALWAYS TRANSLATE data that crosses between bounded contexts so that you can 'adapt' (as in ports and adaptors) the information that flows across these critical boundaries.
Thank you, you are very knowledgeable.
Decoupling
Autonomous
Independently deployable
Easily removed without coordinating with others
Thank you
Think the discussion about bounded context is relevant when implementing an MS.
Excellent video
This reminds me a bit of agile, people parroting a new buzz word but failing to implement it spectacularly
So simply put forward, I guess for a start-up projects, the loosely coupled systems will cost more in the terms of integrity.
Agreed, microservices must ne created as separate micro - independent modules to the same organisation.
You made it so simple ❤❤❤
I think a couple other points that destroy microservice implementations are a tendency to break services at business unit boundaries and a tendency to delineate horizontally almost exclusively. I think you touched on the first a bit in the video indirectly when talking about how systems mirror the communication channels of the organization, but maybe not as explicitly. On the second point, organizations don't tend to build services that, as you said, do things like storage, but rather each business domain service implements it's own (and often multiple) storage subsystems.
The size issue is easy to get right. Describe in a sentence what the service does. If that sentence contains “and”, you’re doing it wrong
Great content Dave and well laid out. Your trajectory of your career is basically the same as mine, I could steal that slide to explain my journey. One difference is in the early to mid nineties I spent time on shrink wrapped software and then operating systems. The one caveat I would say to your whole thesis is just that there is no silver bullet architectural pattern. Micro services are great for the reasons expressed but ONLY if it solves the problem domain in an elegant and sustainable way. Every pattern has its use and I’ve seen teams try to use micro services to “evolve” legacy processes like batch jobs to “modernize”. Guess what? The Batch job, even in this century is still a legit pattern in some instances. There is no one size fits all and in my experience blending or cherry picking the past of different patterns is often the right thing to do. Religious adherence to a pattern can make your system more complex if you first don’t explore multiple options and aren’t predisposed to have to design with the currently fashionable pattern or technology. My favorite example of this faux pas is XML. I’ve had to fix many systems because everyone assumed XML as a data interchange was a requirement for a good system. It only slowed things down and caused tighter coupling with dtds, parsing code, and made interchange with non xml systems a nightmare. Nobody likes to write XSLT let alone debug it. New subscriber here.
Long-time software engineer - I think my main issue with "micro-services" as a concept is that it tends to encourage OVER-abstraction. The distributed nature - as you pointed-out - is MORE complex by its very nature - couple [sic] that with over-abstraction, and it can get ridiculously complex to do the simplest thing.
Further, if you have a user-interface that actual humans have to use - then you have at least ONE customer of your microservice that DOES have to know all the innards, effectively - or certainly the data models. As you might surmise, I'm more of a data-driven engineer at this point.
Data, in the REAL world is NOT an abstract object. In the REAL world, an "object" is composed of many other "objects" in particular patterns, and is damned hard to work with in an over-abstracted form. Basically, the simplest order form uses both composition, aggregation, association, etc., and basically - has to know about it. It can then interact as necessary with whoever supplies the pieces - BUT IT HAS TO KNOW at some point.
This over-emphasis of object-oriented design gives rise to the popularity of no-sql databases because it's a LOT simpler for engineers to encapsulate a complex object that way, than to fragment-and-reassemble them using a relational provider.
Lastly, when testing, each ultimate consumer of various microservices MUST be called-on before anybody would consider a service to be fully tested. Some would say that a microservice is a well-defined black box, and if it passes those tests, it's fine. But in the REAL world, that isn't my experience.
Solid video. My compliments to you for simplifying a misunderstood and misused term amongst sales and marketing people .
I think micro-services for most organizations just turns into a versioning nightmare
I watched his cyberpunk video and now I'm here learning more
Thanks, I am pleased that you are enjoying the channel
Ye right, the real reason of splitting services into seperate rest projects is, gaining an ability to run a separate service in a seperate processor and cache/ram in a distributed way which allows you to speed up your processing time to respond to large scale requests. Thanks for reminding us about SOLID principles and applying it to Microservices.
"focused on a task" could be being an backend of a web app or converting an integer to a string
Solid presentation. Everything is spot on! Congratulations on tackling this misunderstood topic very elegantly.
Very good video that captured a lot of the discussions I've had surrounding misused microservice terminology very well.
Just because something isn't a pure microservice doesn't mean it's bad, but it does mean you should find a different term for it or people will get the wrong idea.
Starting my DevOps / DevSecOps journey.....this is helpful thank you.
Glad it was helpful!
What a brilliant video!
The decoupling between services brings so many questions to me (that really likes modularized monoliths and haven’t worked a lot with MS’s).
How to handle relations in between services, if at all? Are people storing IDs and just hope for the best?
How to tie the pieces together? Via API GW? or independent calls to each service?
Wouldn’t that be considered coupling since they’re no longer independently deployable? Would it not only be really decoupled by letting each service being complete (with UI and everything)?
So many questions!
Thank you. Yes it does raise a lot of questions. I think that one of the secrets is to think very clearly of the "protocol" of information exchange, being separate from the service implementation.
I’d love to see a video like this in regards to web development.. it’s getting crazy with all these JavaScript frameworks, package managers, node, yarn, etc.. they compile and create a project folder filled with so much code and very little organization or explanation of what it all does, you basically have to just hope all the dependencies do something of value..
For real! I'm currently trying to marry a quasar Vue 3 composition api project with a vuetify Vue 2 class based components api project. They share some fundamental logic and architecture that I'm trying to refactor out. Wish me luck!
YES , this! , a single hello world, can have frameworks for lints, unit tests, css webpacks, ci cd, package management and more . So much noise!
@@emaayan I get the idea, but in that case the real question is: why do you need a complex framework for a simple "hello world" app, you should try to think about mvp and yagni to avoid getting lost in insignificant details (and following the videos mr. Farley and other software engineering gurus put out is a really good direction for that).
Indeed, Domain-Driven Design is a fantastic book. For me, the alignment of microservices to Bounded Contexts is a prerequisite for that microservice being "independently" manageable.
The translation of concepts between microservices (specifically those belonging to different Bounded Contexts) is no small task, and either requires close coordination with those familiar with the microservice being integrated, or for the other microservices' API to have outstanding documentation. An organization pursuing microservices for the right reasons would presumably be happy to ensure such communication were adequately invested in.
One other potential benefit with microservices, which Eric Evans suggested 5 or so years back, is protecting model integrity with hard boundaries. With a monolithic system, there is little to stop a developer adding a new feature from reaching into another module's database and creating some new point of coupling between the modules. While in theory a monolithic system can be modular, as Evans noted, decades of experience suggests that that isn't how they tend to work out over time.
Here the key thing for me is that these software components (not necessarily "microservices" as defined in this video) have physical boundaries to help keep the module's conceptual integrity protected. At a minimum, protecting the module's database access credentials as if they were akin to ssh private keys would perhaps go a long way to achieving such aims.
No, it isn't small, but essential to retain the loose-coupling, otherwise do something else, not microservices :)
i’ve been working with a large microservice-based system for years and this is exactly right
Great video! One issue that can be tough to resist with these things is when scope creep by changing needs gets forced onto the developers and breaks the autonomy aspect. Would love to hear your thoughts on how to address scope creep when trying to keep your projects clean and modular.
I think that you can't fix things forever, over time, the ideal division of responsibilities in code, and in teams, will morph. A thoughtful organisation will take this into account and re-assess team structure, and software architecture over time.
These are deeply related ideas, and I have a video that gives one take on the org challenges here: "How to build big software with small teams" ruclips.net/video/cpVLzcjCB-s/видео.html
At the more practical, tech level, I think that as you discover new features, you decide if this fits within the current scope of your services or modules or if you need to add new ones. If you add new services, then at some point you need to decide if you should spawn-off a new team to look after the new services.
I've always considered them synonymous with the 'S' in Solid: Single Responsibility Principal.
YAGNI is the biggest problem I'm seeing with microservices. You describe (very rightly) how it should reflect the organization, that's a very good test to tell if it is applicable. I've seen a single team of 6 people try to develop a system consisting of 40+ microservices. Because 'it's what Netflix does', never considering the fact that they where building a niche line-of-business application and not a million users online platform. So they took all the costs, but the benefits didn't apply to them. (And on top of that they got all the boundaries wrong, because they never took the time to properly understand the domain.)
Like so many things in software development microservices are a perfectly valid solution for a problem you may not have. But it's often touted as 'the one true way' for everything. There is no silver bullet...
Yes, its always good to understand the trade-offs with any technical choice - and there are always trade-offs.
Would love to learn more about high performance computing with event driven architecture! Sounds really engaging.
I have a few videos that you may find interesting:
Reactive Systems - ruclips.net/video/eRxLfUIMJwk/видео.html
How Fast is Your Computer? - ruclips.net/video/0reMVgn6kRo/видео.html
The Next Big Thing in SW Architecture - ruclips.net/video/-IZlOPciSl0/видео.html
Great video.
I’ve found one of the biggest problems with the actual implementation is modelling transactional consistency boundaries and sharing data. This is where I’ve found teams can really get tripped up.
Do you have any videos or advice on these?
Indeed, this is a tricky problem. The SAGA pattern is designed to solve it. I have not actually seen it implemented within my organization, but I'm sure there are plenty of companies that are effectively utilizing it.
Working on something and just found this great video. Thanks for it. I was already using his advice on 12:41 and will continue to keep it that way until the need grows. If this is not late, maybe advising on how to handle cases on communication between modular designed services will be awesome.
Your videos are extremely valuable to me especially as I'm going down the software/systems developer route. Thank you for sharing knowledge.
You need to start with the business process and identify the tasks and events that trigger the processes. Micro service should map to the discrete tasks in a business process . An example would be the booking service for a hotel reservation system
Brilliant video. I see micro-services tossed around like a buzzword with few people articulating the why. This was great.
Great explanation of what it is and what it is not a microservice, would be nice to add or to discuss the different approaches to actually make good decitions in the design of the bounded contexts and each microservice in more depth.
Great explanation - like! One comment though.
You say that the APIs should be considered "public" - "be defended and rarely and with great caution changed".
While this is definitely true and is a very important point - we should maintain backwards compatibility in a distributed environment, and give our consumers time to migrate, there's still a spectrum in the "publicness" of the API's in regards of changes.
A microservice API can be internal to a team, and therefore retain the ability to evolve very quickly and with very little communication or compatibility overhead.
Next, an API can be a dedicated (non-generic) communication line to a selected and curated group of consumers, which again decreases the costs - especially if those consumers are agile enough.
Finally, an API can be considered public - like what you described - with the maximum costs of maintenance and thought up-front.
Wouldn't jumping into defining and building the last - most expensive - type of the API's before the architecture has matured and the organization evolved to deal with microservices - wouldn't it be a bit of a risky proposition?
Really appreciated this video, should be a must watch for on-boarding developers in an environment with microservices implementations
Thanks man that's really a confusing idea I've struggled with. It's not easy to differentiate between SOA and microservices. I think it would be very interesting if you give examples of implementing microservices wrongly and explaining what is wrong with it. Wish you all the best thank you again
I have the same confusion.
Bounded Context is the MOST important, BY FAR!
Eric Evans, who wrote "Domain Driven Design" is a friend of mine, he says that if he was writing the book today, he would put Bounded Contexts earlier in the book so that they were more prominent. DDD is not an easy book, but one of the most important IMO.
@@ContinuousDelivery I completely agree with you and his assessment. Also, I dont like that DDD is so often only applied to cloud solutions. It’s merit to cloud implementations is without doubt, but its overall contribution to technical solutions implementations is far more vast than cloud.
I certainly consider myself to be lucky to call Eric a friend, is is a very nice person, and smart enough to always have interesting and sometimes surprising ideas.
I have been speaking on the international conference circuit for many years now, you get to meet a lot of interesting people that way.
I think that DDD is regaining popularity, it is quite an old idea, but Eric captured it in more detail than before in his book, which significantly pre-dated the birth of the cloud.
I think though that the cloud has given it new focus, because if you are commoditising infrastructural concerns, which to some extent the cloud does, then it allows you to staty thinking more clearly in terms of separating "essential" from "accidental" complexity. Which is some of what DDD talks about, in a slightly different way.
I think that there is a next step beyond that too. I have done some work on async, event-based systems, I was one of the authors of something called "The Reactive Manifesto" www.reactivemanifesto.org/which describes them, and then you can achieve an almost pure separation of accidental and essential, your services become almost pure domain model. A lovely way to work. I think that moves towards something called "stateful serverless" may be one to watch in that space.
@@ContinuousDelivery The main problem I have with microservices is that I don't generally have a bounded context that's small enough to implement in a few weeks. I'm usually dealing with dozens of entities, each with a rich set of constraints and that just takes time to do right. Does that mean that these domains simply aren't amenable to microservice approach, or do we flex a bit and say that if it's one bounded context, it's still a microservice, even though it might take 3 months to do a rewrite?
One problem I can see (non developer here) is that the service will bloat quickly in order to maintain backward compatibility. I.e. You call service with params a,b,c and get back value X. You update the service to be able receive param d to return value Y, but you will still have to maintain the old code to ensure compatibility.
Very well explained, as an absolute beginner in software development, I find your videos easy to understand and thought provoking! Look forward to many more videos from you :)
A great video & at least every architect, who thinks that they design microservices applications, must watch it.
As I believe many such good thoughts are talked about in many sessions in the organisations but many times the
same ethics/thoughts go down the drain in their projects, as money is involved.
If profits are flowing in & quick delieveries are made then no-one cares if these concepts are being followed or not.
- Microservice is NOT all about small services.
- Microservice is NOT just a rest api.
- NOT all services are microservices.
- Microservice is a distributed system architecture.
- Microservice is small:
- fits in my head.
- can be rewritten from scratch in a week or two.
- Microservice is a single responsibility and does it well.
- Microservice is aligned with bounded context:
- A bounded-context is a defined part of software where particular terms, definitions, and rules apply in a consistent way.
- A bounded-context is a cohesive unit.
- when crossing a bounded-context, you need to translate the data.
- Microservice is autonomous
- progress can be made without waiting or coordinating with other teams.
- this is the real value of microservices.
- Microservice is a single deployable unit
- you should not have to test another service when you deploy a service.
- Microservice is loosely coupled.
- make external & internal dependencies explicit.
- interface to a microservice should be public.
- when consuming api of another microservice:
- you should not have to know the implementation details.
- use minimal data. so that you can reduce the coupling.
I Love your format - slides make your content easy to follow and review, while you’re delivering in earnest alongside
I've been using Rails and never been happier. Yeah I have a monolith, but it's not a problem. I have 2 micro services for mail and message queuing. I don't need anything else. I was a node developer, and used microservices heavily. They're not a bad thing, but they're misused today. Martin Fowlers enterprise architecture is an excellent read.
I don't quite agree with you on the assertion about what "small" or "micro-" means, i.e. that a microservice should be fairly easy/cheap to throw away. As you pointed out, microservices have meaning in the context of distributed systems. If it were enough for a microservice to be easily disposable, then any cheap-to-develop module in a monolith would be a candidate for becoming a microservice, if we were to turn a centralized monolith into a distributed system based on microservices. Modules too are supposed to be autonomous with low coupling and high cohesion (i.e. bounded by domain context). But that's not the case. A microservice should be small, not necessarily from a development effort/cost perspective, but from a deployment and scaling perspective. A microservice MUST be easy to scale up and down. It must be easy and cheap to deploy and that naturally translates into favoring them being geared towards a single task.
The problem with Microservices, is caused because the people think Microservices are an architecture, Microservices not are an software architecture it’s part of it, but Microservices are more like a deploy strategy, the architecture of the software are clases, layers, and your deploy strategy like Microservices, the people who thinks Microservices are an architecture is because they doesn’t know anything about clean architecture. For example I have a project in my GitHub that can be deployed as a monolithic or as a Microservices applications, it’s because the real thing that we use to decouple layer, not is the deploy strategy, are the interfaces and how your system interact with that interfaces and clases.
Is stateless other angle of decoupling and independent deployment of microservices ?.. I appreciate your comment on this regard.
Recently come across your channel and I already feel like you've been living in my head for the last 10 years! Great videos. I haven't heard or seen (yet) you talk about consumer driven contract testing (like PACT) when working with micro services, do you have a video on it? I feel it adds a lot of value in the space!
I talk a little about that in a few videos, I talk about the principles here: ruclips.net/video/cpVLzcjCB-s/видео.html
I probably should do a video more specifically focused in this topic though, thanks for the suggestion.
@@ContinuousDelivery Thanks for the reply, Ill check that out and look forward to the next. On another note, would love to see some Q/A live streams on the channel..
thank you.
in terms of things I'd like to hear about - you mentioned how software design ends up modelling the organisations communications - I have encountered strong coupling between org culture and development culture - especially as a road-block to change. It's bleedin obvious to me that say moving from monolithic software to distributed software takes cultural change within your development team (eg. moving from test/deploy to CI/CD, or agile, or DevOps) - and communication/cooperation/coordination between teams needs to increase significantly...
I'd love to hear about the sorts of approaches that make this change possible, success and failures, and so on - not so much a tech question I know but damn if it isn't the core of the problem...
Did I hear this correctly Dave?
Your favorite architectural style is the distributed monolith??? (13:00)
How can you do CD with a distributed monolith? You can‘t IMHO. I‘m confused 😅.
Please clarify. Thanks!
Beside that: AWESOME episode! 🤩
Thanks 🙂
Yes, I have been building service-oriented systems for a few decades now, it is much my preferred approach in terms of architectural style, though naturally I try to fit the solution to the requirements before picking an architecture - there is no "one-size-fits-all" option.
When you start out on any complex system, you don't know where you will end, you will grow your system as your understanding grows. At least that is how I work, and think is the best way to create complex systems. In the early days you will be exploring more and making more mistakes, it takes a while for your design, at the level of services, consolidate. You will find that some of the service interfaces, even if you got the service boundaries in about the right place to start with, change a lot as you refine the responsibilities of the service and evolve the cleanest interfaces.
During this period there is no benefit to a microservice approach IMO. A much better strategy is to bung everything in one repo and build and test it all together. Th HUGE advantage of this approach is that you can "build and test it all together" there is NO DEPENDENCY MANAGEMENT of any kind. I can change the interface to my service, and update all of its consumers in the same commit that I make that change!
The main downside of this approach is that you have to be efficient enough in your building and testing to get answers back on CD timescales, so under 1 hour for everything. It also means that the pieces aren't really "independently deployable".
This approach doesn't stop you having separate, reasonably independent, small teams though. This how we built our financial exchange at LMAX, and it is how Google and Facebook organise too. It is surprisingly scalable!
As I describe in the video, microservices is an organisational scaling strategy, nothing more. It limits the options for optimisation, because each service is discrete. It means that you have to extra work to do to facilitate the "independent deployability" and so on. It also demands a higher level of design sophistication to keep the services separate and "independently deployable". My advice, and Sam Newman's advice who wrote the most popular book on microservices, is to begin with a distributed monolith, and only, once the interfaces have stabilized, move to microservices.
@@ContinuousDelivery Thanks for the answer Dave!
But don't we usually start with a "single process monolith" instead of a "distributed monolith"?
(Sam Newman "Monolith to Microservices" p. 12 - 14)
@@PetiKoch Ah, I see what you mean. Sure, maybe, depends on the application. The way that I build my services, that is more of a deployment-time decision than an architectural one, which is why I don't think of it that way.
The danger of starting with your monolith as a "single-process monolith" is that the other description of that is just a bunch of code! Unless we are writing something that we KNOW will be throw-away, then I think that the guiding principle for good design, even for small systems, is to manage complexity. So I want my design to be modular and cohesive, with good separation of concerns and as loosely-coupled and well abstracted as seems to make sense at the point that I write it. So I like the idea of "services" as a modular unit.
Now, what I mean by a "distributed service architecture" is that I don't care if the services are running on the same machine or on machines on different parts of the planet. That means that the comms mechanism, and my design, needs to allow for the case where I want to make the services non-local. My favourite way to do that is to make the interfaces to my services Async. Now it is a deploy-tome decision wether I optimise for simple, local, fast, comms or distributed, non-local, comms. I have separated that comms concern.
But this only works if I assume that the services are non-local. If I assume they are local, I may become addicted to the local performance, and then when I try to distribute them later, when my system needs to scale, my architecture is no longer fit for purpose.
You could argue YAGNI, the trick here is to do this in a way that makes the overhead of this "thinking ahead" low-enough that its not really a problem. That is a matter of experience and design-taste I suppose, but we got that right when we built our exchange and I have used that approach a few times since. There is some more stuff on the architectural style that I am hinting at here:
martinfowler.com/articles/lmax.html
and here:
I was involved in creating something called the "Reactive Manifesto" to describe some of the properties of this async architectural approach.
www.reactivemanifesto.org/
I find one of the bug bears with Microservices in my experience is different teams using different tech to build their services, creating a huge range of different implementations across the estate which confuses the hell out of other teams who sometimes have to support these services. It's important to decide what to use from a curated tool box of tech that the teams agree on, so at least working with each service isn't a complete nightmare.
So hard to setup contract testing with MSA, this video highlit (to me) why I've been struggling lol, thanks
That whole bit at the end about defending interfaces and changing them rarely. My experience is that going heavy handed early on in a project with pact for example takes an inordinately heavy toll on development speed.
great explanation, any project you go you will be working with API so this is essentials things to know.
This is fantastic stuff, as someone who's changing career trajectory having this high level overview is very helpful. Thank you
I feel like the word "micro" is what's causing misunderstanding for many. Because, in fact, you're building very real services. Then you build services that use your existing services. It's a very simple idea, but people look at it like it's something magical and that's only thanks to over hype and marketing. Exactly same thing happened with dockers/containers/"cloud" as well. I am personally not at all amused how we always go this same route with every technology. "This thing is so cool and you're just outdated"
Great video. Unfortunately, in my experience (so far), Microservices have been used basically to let the backend developers say “I can’t do this obviously simple thing because "microservices" ”. Meanwhile we are stuck waiting for something that used to be quick, and now it is not. The level of misuse of this concept is literally weighing entire companies down. I’m not saying microservices are not useful, but since developers themselves are doing it wrong, you get none of the benefit, and all of the cost.
yes, generally, microservices is a terrbile idea, is more costly, slower and takes more time to develop.