Design Microservice Architectures the Right Way
HTML-код
- Опубликовано: 30 июн 2024
- InfoQ Dev Summit Boston, a two-day conference of actionable advice from senior software developers hosted by InfoQ, will take place on June 24-25, 2024 Boston, Massachusetts.
Deep-dive into 20+ talks from senior software developers over 2 days with parallel breakout sessions. Clarify your immediate dev priorities and get practical advice to make development decisions easier and less risky.
Register now: bit.ly/47tNEWv
----------------------------------------------------------------------------------------------------------------
Download the slides & audio at InfoQ: bit.ly/2ypNF04
Michael Bryzek highlights specific key decisions that very directly impact the quality and maintainability of a microservice architecture, covering infrastructure, continuous deployment, communication, event streaming, language choice and more, all to ensure that teams and systems remain productive and scale.
This presentation was recorded at QCon London 2018: bit.ly/2hxsoN1
The next QCon is in San Francisco, Nov 5-7, 2018. Check out the tracks and speakers: bit.ly/2uYyHLb. Save $100 by using the discount code “INFOQSF18”
For more awesome presentations on innovator and early adopter topics check InfoQ’s selection of talks from conferences worldwide bit.ly/2tm9loz
Interested in Software Architecture? Follow the topic on InfoQ: bit.ly/2NmIOpa
#Microservices #ContinuousDeployment # ContinuousDelivery #InfoQ #QCon #QConNewYork 
Наука
A personal story (change API URL) [0:13]
How does this happen? [0:38]
Great Architecture [1:10]
Not So Great Architecture [1:55]
Design Microservice Architectures the Right Way [2:29]
About Me [2:45]
Let's Start With a few Misconceptions [3:37]
M#1 teams to choose language [3:40]
(expensive)
M#2 code generation is evil [4:32]
(just a tech)
M#3 event log must be the source of truth [4:55]
(ok to record service own record)
M#4
holy shit
you're a god
This is awesome
Of course it's not a RUclips comment section without heroes who put timestamp for sections
One of the things I learnt in this video: Code generation can be used to set strict standards across the systems
This the best presentation I have watched my entire professional life of 22 years. And the best of the 85 videos I have watched on microservices.
Thank you! Really appreciate your note
Thanks for your comment. I was only 3 min into the presentation and about to leave, as I already had seen so many Microservice presentations. But I thought first lets look at the comments. And you are so right, thanks very much!
@@mbryzek I have to second eJay's comment. This answered so many decision questions on utilizing microservices along with event driven architecture in examples of real world success. There are too many talks with buzzwords and not enough practical information on implementation. A great question you answered was using individual databases per microservice that is subscribing/publishing to an event. The "source of truth" narrative has been a bit ambiguous with regards to the rules of using other DBs.
@@jamesmcmurtry5351 Thanks James - Really appreciate your comment!
@@mbryzek Great talk. A good followup would be on the optimal microservice network for various use cases.
I know this comment is 4 years too late, but the sheer volume of things I've learnt from this talk tops all the videos/tuts I've watched regarding microservices without debate.
I loved this talk, not because it is the gold standard but because Michael knows what he is speaking and has taken the efforts to setup the standardisation in every step of the process.
I've watched a lot of videos on microservices. This is by far the most sensible, brilliant and cohesive explanation of all! Should be made into a textbook tbh
Great software design material! It not only covers microservices, but also the devops operations associated to deliver quality software.
this guy either was really prepared or a savant with his incredible train of thought
In the entire presentation most of time, it not slides with some block diagrams but real code examples. I really enjoy that. Creating/ investing into such discipline to standardise stuff, you got my respect man.And indeed the code was really beautiful.
Excellent talk - in particular, note the advice @24:25 - adopting a microservices strategy requires Continuous Deployment
Their DevOps game is on point
I could like this a million times over. The best presentation on Microservices.
I am blown away! Great video! Thank you!
So much content in such a short time, great presentation.
This was an excellent talk (thank you Michael) and he is the living example of how to execute on the principle "if you need to do something more than once, you need to automate." Automation is so much more predictable then humans ;-)
Great presentation! Must to view for all practitioners!
One of the most amazing talks about Microservices Architecture - You guys are geniuses and thanks for sharing all that knowledge!
Rewatching for 4th time. Will continue. 👍👍👍
Amazing architecture story. One of the best tech presentations I've ever seen.
Lots of useful information here, i've always been into defining metadata around data structures - i had written the same kind of thing about 7 years ago where classes were automatically generated from the schema, which was defined in the database with standard types for text, email, textareas etc. but what i find most interesting in your talk is grpc; its able to not only generate data structure templates but also has client and server libraries for creating a service out of the box.
This talk was absolute best and absolute gold. Thank you.
This is the most practical presentation on the quote "Talk is cheap. Show me the code". Finally found something practical otherwise those theoretical rhetoric.
Really a great presentation. Michael is talking what he has done in his real life. I was surprised he is not using Kafka for streaming and adopted a different approach. Thank you a lot!
Kafka is way more efficient and fast, but can be fussy in production. Kinesis is easier to manage on a global scale because it is so well supported by AWS.
One of the greatest presentation I saw in the tech world. Really a lot to learn from Michael. I have hugely invested my decade in frontend world and now when I am digging in backend worlds, I see, I can invest next decade in it.
absolutely love this. thank your for posting this. Keep doing good work and keep sharing.
Wow, we've basically built this exact thing over the last few years. It's nice to see this kind of validation.
Great talk! thanks
i'm really impressed by that rds dev cli
i want one
Thank You
i watch this video again every year, and every time i learn something new. such a treasure trove. Would love to work with Michael someday, he seems an awesome engineer and an emphatic learder.
The way I view this talk is as a representation of the “perfect workflow” for microservices. In my small startup, most of it is “good to know” but I can see adopting things more and more as we grow. Particularly helpful for me was understanding the journaling event publishing aspect.
The code generation stuff is mind blowing - I would not have thought to automate that much. I wonder how long it took to set that stuff up before they even shipped any product. Yes, now they probably develop lightning fast, but this would be a huge distraction from finding product market fit.
most of our code gen started with a 2-4 hour "hack" that was sufficient to solve the use case at hand; then we built on that to add features over time. We did this as a startup - definitely doable!
I haven't used it, but you can take a look into Atomist: docs.atomist.com/ they're doing something similar
@Robin Chan I was thinking the same. What if the generated code needs to change? Are all your manual changes and implementations lost?
@@Hex72 I would say there should be no manual editing in the generated code. When I build a code generator it always produces interfaces or abstract classes where actual manual work is required. Everything else like Parsers etc. Should not be touched by anyone except the code generator itself.
@@hibas1235 That would make the most sense, yeah.
Wow! Thank you so much. This talk is going to drive a lot of decisions in my upcoming microservice architecture.
Love this talk! Loads of insights
Realworld Microservice example, amazing!
It's funny, I keep coming back to this video after studying up on microservices and understanding more and more of why these are good design principles. Great video
This is SO interesting !
these people can rebuild the internet if it goes down
Great presentation. Very informative and great design lessons. It also represents great team collaborations and good culture at Flow. I wish more companies could adopt such innovative ways of working.
Great presentation, but i have to admit, reading your comment that its great, i started to think its a very good guideline to quite specific practices when developing a microservice system. Not as an microservice architecture as a whole. Not complaining for sure, i learnt a lot from it, but it seemed like it focues on the micro elements of it XD Not the whole.
really amazed by - "6 lines of yaml" great effort by folks who automated the default setup.
Great speaker and even better content.
I would say, these are Gold Standards of MicroService Architecture From code generation till deployment. Every stage is well designed!!
Thanks a lot Michael Bryzek, you made my day!!
Thanks!
I think I'm going to have to have to borrow heavily from these ideas. This information is pure gold. Thank you so much! Just what I was looking for.
thanks!
Fantastic video, really inspiring!
Fantastic!!! Packed with info and best practices! Thank you!
Brilliant talk, very informative
I hope everyone understands how brilliant this is
Agree, The Right Way. Isn't that much and difficult. Microservices rule!
This is one of the best talks on microservices
45:08 Haha! Love this part. How much time, you ask? A lot! And a lot of anxiety too :D
Cool guy. Great speech!
great presentation. Thank you
thanks!
Awesome presentation sir! Well done.
thanks!
My life feels like a lie now when I think about the fact that I know how to write microservices. Absolute gold standard!
Very cool presentation! Lots of practical tips to use.
omg dev cli is an AWSOME idea, thanks for that :)
agreed
thanks!
WOW! Amazing talk.
thanks!
I'd probably watch this several times over
I love this talk. So useful and so practical.
Great talk. Most tech talks nowadays are passing time with things like "show me your hands, who is using kafka". But on this one, I actually had to pause a few times to digest what's been said.
Same for me. He seems to be a clever guy. Especially when he corrected the guy asking a question about feature development. CDeployment vs CDelivery are two different things. This little examples shows that he knows what he is talking about.
So... onboarding a new developer to all of that custom built stuff takes time to pick up and is very specific for their build / devops case. Its a fun talk and I like his delivery style and self-confidence in it. I am amazed they built stuff on their own and did not use existing stuff. Also those services are super thin from what I got the idea but then yet they have databases. Being unexperienced in this much code gen I can not say it is that worth it but again, hey they deliver and auto gen those e2e tests. Cool talk especially as I never did java or play professionally.
very well presented especially focus on tooling
I love the way this guy approaches developing: schema first, strong types, end to end tests, strong tooling, expressive language (scala). All of it together creates a virtuos cycle where each iteration carries you upward. I do wonder about schema changes and the event bus though. What happens when Service A puts a changed User onto the queue when the consumers haven't been yet updated. I also wonder about transactionality because at my job we sure seem spend a lot of time trying to tack on stuff that looks a lot like commit / rollback to the event bus.
Great moment to watch this presentation :)
impressively simple, well done
Awesome talk! Thanks Michael.
Can you perhaps elaborate a bit on the different main microservices that you have? What does the users microservice is in charge of for example? How do you ensure data consistency between multiple services that each save their own local copy of the same data (a user for example)?
What a great talk!!
wrote tests for production and ran in cron job. omg never thought of that.
Excellent
First of all, Nice presentation, thanks for sharing.
IMHO, The services he described are plain CRUD operations, this is why this approach works well. I've done similar solution in the past where I would design the database schema first and generate the rest of the service, it worked like a charm for CRUD services.
DDD systems on the other hand are much more complex, will be much more complex to write business logic just using DSL or schemas like he demoed.
The code generation would help a lot though.
Thanks, perhaps would you mind nudging us in the direction of what to look for when solving similar issues in DDD systems?
@@gonkula DDD systems are designed around business rules, which most of the times are checks and validations happening before an operation is applied to an entity, these are generally designed as workflows with multiple stages that are updated gradually on each user/system interaction.
I've seen solutions in the past trying to auto generate code based on business rules, but the amount of code generates and the complexity for maintenance doesn't worth the investment.
If you are just planning to do DDD, a good example I can share is this:
docs.microsoft.com/en-us/dotnet/architecture/microservices/microservice-ddd-cqrs-patterns/net-core-microservice-domain-model
I came here to learn microservices. He is teaching me how to code, and I never felt dumber.
May this encourage your faith
facebook.com/esther.chester77/videos/1029971633787995/
ruclips.net/video/mIxKl9KYwNU/видео.html
m.facebook.com/story.php?story_fbid=2582272895211292&id=490097534428849
m.ruclips.net/video/EE2sJ-WIfts/видео.html
ruclips.net/video/Qw2hwcocVvM/видео.html
m.facebook.com/story.php?story_fbid=777934336005747&id=100013675634259
How you deal with states and stubs in integration tests: you create user and you validate that user been created? web stub should be simple but state make it complex...
what are the best resources for learning software architecture?
Code must be generated from a well defined model of the in context of the service. Almost anything can be derived from the model especially SOLID code. The question is how best to define a validated conceptual model up front in a data platform and language agnostic way that conforms to some first order predicate logic that proves the correctness of the facts expressed in the domain.
... but maybe the developers themselves are generated clones...
Brilliant one @mbryzek. Absolutely loved watching it. Lot to learn from this.
BTW, anyone here knows a similar implementation already available for Golang/java?
Great
Man, this guy is a monster.
Nice talk, thanks!
Even if it's old, I have a question: how do you/they automatically create git tags. I guess they are following semantic versioning or something similar. How do you/they know automatically the version to increment?
It could be done by standardized commits. Given that most things are standardized I wouldn’t be surprised if that’s the case.
Hey experts, Please advise where to start and what software are needed to practice some projects from start to end with intention to know to design APIs, Test APIs please. Can I accomplish this with single laptop I have ?
How are complex queries implemented in microservices?
WOW
the approach is limited to the situation when you actually CAN devote some time (like 9-12 months) to bring up the infra and the start coding aggressively. in most cases, you cannot.
Rightly said. It takes years to get there. In most cases you're better off managing stuff on your own. Automation comes with its own headache.
Manual work vs automated work is still just work. It's just stuff that needs to get done. Either you spend the time to make the computer do the work for you, or you're stuck doing it forever. The real benefit to automation is consistency across teams. That's the main takeaway from this talk: the only way to have consistency is with automated tooling.
If you don't have very many developers and can't spare the time to maintain that automated tooling, because it does take maintenance, then it's not worth it to you. The larger your team and product, the more important the tooling gets.
There are several things wrong with the second half of the presentation:
- how did they decompose the GRPC messages to be analogous to the API with it's personal DBs?
- what happens when one microservice is dependent on data in a different API or event?
- if there is some reporting that needs all the DBs then how is that performed?
- if a transaction requires more than one API and there is an upgrade/deploy mid transaction how is repeatability supported (see monkey patching and erlang live code replacement)
... good enough for now.
have you found answers to those questions? those are exact things I was wondering
- Not sure I understand, but if I do, everything is described in the API schemas, even the form of the objects used in the APIs
- Local copies. The rule of thumb if I've understood is that *all* the data you need is in your own db. If you need to add new data for a new service, and you don't own that data, then you subscribe to the events from the owner, and you duplicate that data into your own db.
- Events. If every single service emits upsert events, then you can have a reporting system that consume them all and aggregate them in a big db (at least I'd do it that way)
- There is no transaction in micro services. First, the probability of that happening is (and needs) to be extremly low, with parallel and hot deployment. Then if it does there is two situations for me :
- The event that failed can be replayed because no break in interface was made : he talked about that, you can just replay it, automatically or not I didn't understand that part. I'd say automatically.
- The event that failed *cannot* be replayed because of a breaking change in the interface : then you'll need some way to create the new version of all the failed events in the system, and then I don't really have an answer because he didn"t...
Note that breaking changes in event is the biggest challenge (by far) of an Event Driven System, so I guess it must be tough for them as well :/
I hope I helped and that my anwers were correct :)
What are you using for your schema definitions? Your format looks similar to Apache Avro but it's not identical. Also what code generating/templating engine are you using?
@Chris ter Beke You can host apibuilder in AWS it seems....
Wow, who ever would have guessed that Topher Grace would have gone on to be so successful in the software industry after That 70s Show?!
Are there any open source tools to work with these APIs?
Microservices are required to extend the DB business logic in some way that can not be easily implemented in the DB language.
bravo
I might be wrong (and also a bit inexperienced) but this talk seems highly focused on Designing a Microservice Architecture **for the cloud**. Most of it seems to be great advice but still requires some deep though on "how do I adapt this to my own environments?" for example their Delta project would not work without a cloud environment (unless?).
crack is wack
Cloud is just a deployment detail. It could work *exactly* the same with private cloud or even Virtual Machines.
Especially since he mentioned docker images. It is after all a standardized way of starting an app.
So deploying it to the cloud, private cloud, VM, or your desktop is just a matter of wich command to run, that's all.
Excellent. But I think this just highlights why you shouldn't build a microservice architecture if you don't need it. Especially if you are doing a small to medium size project, with few developer resources. Until the day when you have some standardized tools that allows you to pump out services/components in your tech stack, that just works out of the box.
This is 100% true. Please, to all devs scrolling through comments trying to find reviews of smaller companies who tried using microservices : don't. Just build a monolith, make it clean, make it work. Have tests. Have a clean workflow. Once you're there and you start to feel the need to extract a part of your app into another project, extract just that part, nothing more, nothing less.
It takes a crazy amount of work to succeed with microservices on the long term, and it also requires a great deal of skill. Let's face it, all developers aren't equally competent, and most have more experience working with monoliths (for now).
So just write good monolith code, ship on time, and you'll still be doing better than most.
What is the API documentation standard that he is using?
What is the fastest way to deploy 'Galactic the all knowing user service provider'?
Great presentation but doing this in established companies with layers of beaucracy is incredibly hard. If your building a platform from scratch, then this is a nice way to go about it. The big takeaways here is the power of code generation and schema management. The way its described here makes so much sense. Continous delivery on the other hand only pays off if your testing is exceptional, else its just a quicker way to break things and lose money. The idea of pushing code to production on a Tuesday afternoon before you leave for the day would give me the jitters. That PR can definetely wait 😁
KUDOS!
For product owners some misconceptions discussed in first 7 min starting at 3:30
Regarding Misconception No 3...are you saying that to create a resource in database you set up an event via a microservice and then that event was processed by another microservice and you were waiting for the response, which naturally you found not feasible. After that you got the solution that you directly saved the resource in your database via your microserver and also put in an event stating that this resource got saved via your microservice. Is my understanding right?
Yes
this is the employee handbook
how does one make sure automatic dependancy upgrade doesn't bring breaking changes!?
You write tests, to verify if everything still works
Somewhere around @45:10 he talks about testing and how much they rely on it.. Apparently they have a lot of tests which guarantee such an action to be truthful. And such they deploy with confidence.
check out app.apibuilder.io/history?org=apicollective - you can see explicitly every API change that might be breaking
Did he not just mention that every change to master is automatically tagged and tested before deploying into production?
I did not understand his point about creating indexes at 22:54 . He mentions that they would catch the developer from trying to create an index before they do, but what is the advantage and why should we appreciate whatever he is trying to say. If someone can clarify, please?
What he means is that the developer shouldn't have to think about something like an index on a column, when they could have just worked on better tools so that the dev can focus on more important things, mainly the application itself. His point is that developers shouldn't have to waste time on trivial tasks or tasks that could have been solved with proper automation and code generation.
Gupta, plz shut up and go cold call ppl. GUPTA!
Great question and thanks for asking. What I have seen happen all too often is a situation where the "application becomes slow" and then is "fixed" by creating an index. This ideally should never happen and be instead part of the development process. By using code generation that provides methods or parameters based on the fields that are indexed, you can significantly reduce the probability of this error occurring because the indexes will be there from the start.
If we have the API, why favor the events? He must be talking about internal clients and not public ones, right?