My bad. I should have done that. It's a front-end of your developer portal. It's a graphical user interface similar to, for example, AWS/GCP/Azure console and often obtained through Backstage, Port, or similar solutions.
Great video Victor, I 💯 agree with you that having an UI to build an IDP should be very very optional and not required. In CNOE we have some helper tools to discover XRDs and build the forms in Backstage then Backstage creates the k8s YAML in git for ArgoCD
That's how it should be. It does not necessarily have to be about XRDs. The same logic could, for example, discover that knative is in the cluster and paint fields based on its schema as well.
Thank you for this video! As usual you Inspire us for new ideas. One time you inspired us to use Charm for our CLI tools. And know where we use it already? We are making UI for create K8S resources by filling forms generated by API schema. Thank you for confirming this great idea! btw: Crossplane is on background ;)
I agree in principle, but often we make UIs smart so that people can be dumb. For example, at Netflix we had a UI to create a DB which had smart defaults based on the workload, with the associated costs computed at the bottom. Need a low-risk DB for internal only apps? Or maybe a business-critical DB serving millions of RPS across 3 regions? No problem, pick your profile, tweak your settings as needed, and acknowledge the cloud costs. UIs can provide a high-context interface for people who don't want to think too hard, they just have a job to do and don't want easily-avoidable emergencies or surprises. These sorts of smarts should be incrementally added, but as you rightly point out, most people go straight to over-engineering! How do we get the best of both?!
Those defaults/smarts can all be baked into services instead of being hard coded into the frontend. After all, there is no guarantee that those services will be consumed only through the frontend.
@@DevOpsToolkit For sure, and we did source the data from other backends. It wasn't as simple though as just a UI generated from an API -- getting the UX right in certain situations required tweaks specific to the view in which it was presented. The devs using UIs were generally a different population from those VIMing YAMLs, with different needs and expectations.
@LeakMemory that is one of the main reason why I'm proposing creation of APIs. I do not mean usage of existing APIs but creation of custom ones that meet the needs of your users.
The CLI tools are just like the portals you talk about in relation to being "dumb". Each CLI tool you mentioned is versioned whether it is terraform, gcloud, aws, az, tf etc. The CLI tools don't go out and discover the API capabilities either. It would be a great world if CLI, and webUI tools/consoles went out and discovered API capabilities, but none do. The versioning of each tool is for stability and known support. Instead of making the tools "dumb", the developers try to match up known capabilities and often times backwards compatible, depending on versions. For example, in gcloud there are the beta and alpha channels, and in aws there there are API versions to choose from.
Great video as always, do wish you had emphasized GitOps more. In the ideal world, the panel would only need readonly access to Kubernetes to pull in information and write to the git-reop that the gitops tool pulls from and it applies Kubernetes. But that probably would just have made the video more complicated/to long.
I did not want to go towards GitOps since that would make the video much longer. Part of the reason is that GitOps is moving away from interacting with repos from pipelines to processes that listen to registries and update repos. I'll explore that in one of the upcoming videos.
@@DevOpsToolkit ah, yes, an other good topic. But if they disconnect GitOps tool from repo, ... the panel can't look in Kubernetes to figure out the correct repo.
Hey viktor i really liked this video, One question still: do you mean than that the frontend UI should talk directly to the kubernetes server or use a backend proxy that will talk to kubernetes? Thanks!
i love your videos and would like to understand further, we built IDP that talks to our gitops ecosystem, we built API that wraps our gitops and the IDP talks to gitops api and you are saying simply create tons of CRDS (for example we have openapidocs in gitops service and we have over 1000) wont that in the end crash the k8s cluster.. i mean that what helm2 releases saved as CM did we saved huge files and alot of them in the end causing problems to the k8s cluster itself.. moreover we have etcd size limit for single object we have to take in account here... and also the solution of crossplane is awsome!! but it brings tons of CRD creating a strain on the cluster. I would like to hear your opinion, but yet another simple but brilliant idea!
If I’d go that way I would probably need something like 6-9 different CRDs to describe my gitops data and in total I would have close to 5000instances of those CRDs
@matankeler2761 instances (resources or custom resources) are rarely the issue. If you have 5k instances, you'll have tens of thousands resources anyways (e.g. deployments, services, ingresses, etc.). What might be an issue is if you have thousands of CRDs but that is not your case.
I did my best not to make that video about Crossplane but about a potential path forward. It does not have to be Crossplane. There are many tools that allow us to create APIs (CRDs) and controllers. Crossplane abstractions (Compositions) can easily become complex but, as far as I know, there is nothing simpler that allows us to combine resources and expose them as a CRD. I tend to (partially) solve that complexity by not writing YAML but genering it with CUE or pkl.
Again, another great video. We've been discussing this for a few months, and I'm implementing something like reflection for custom entities in backstage. Somehow I struggle with a generative approach, but this leads me to the question: Are you aware of any tool or framework that might be able to render react (backstage) UI from let's say JSON schema or the like?
Let's assume I don't need UI but CLI would be sufficient for the developers using the platform. Does it change anything regarding what you were trying to prove? Are you suggesting CLI should be merely autogenerated from an API and a "dumb" piece of software, that enables operations, discoverability, and basic observability?
That's what I'm suggesting. That's why we have APIs. For example, if you execute 'kubectl get something', something is not hard-coded in kubectl. There is no need to have it inside it since it will go to kube API and ask for "something" and the API will return it or throw an error saying that something does not exist.
Thanks for another great video! This is an interesting topic indeed. I agree that Developer Portals tools should be dumb and if I had to build one on top of Kubernetes I would architect it the way you described it too. You have CRDs, schemas, events and everything you need right there.
Assuming that you're doing it with an account that has only read permissions, I don't see a problem with that, at least when discoverability is concerned. For operations, you can push to Git directly, through the kube API endpoint dedicated to that, or anything else. It's all about permissions you give and policies you have. Also, I am not necessarily talking about a cluster whether you run production but a control plane cluster. In that scenario, it is actually easier to secure a single control plane cluster than providing credentials to different tools that would perform operations directly. A single point of entry is always easier to secure than multiple points of entry.
Interesting But I am thinking about IaC as mandatory. Most company need you write HCL, YAML or whatever as part of this part of the git ops solution. If we developer a API to do this, where should I store the IaC?
I don't think internal APIs and IaC are mutually exclusive. You can write custom a Terraform provider to access internal APIs, or, if you use Kubernetes, use many existing GitOps tools that apply YAML code to Kubernetes API.
A couple weeks ago, something clicked in my brain. I think you just summarized it pretty well in this video. Crossplane planted a seed in my mind. Up to this point, i was only seeing K8s as infrastructure and basic application management. You define the node pools and all the hardware you want and let the control plane schedule that for you. You use helm for packaging your app and create your services. But we can push that much further. Eg, you are building a multi-tenant system, why not create a Tenant CRD ? Have an operator scheduling everything that you need for that tenant. You need a stripe billing account ? It's own Sevice A, B, C ? It's using a shared Mongo cluster, but you want database isolation ? Same thing, CRDs & Operators. It sounds good, i just don't know if it really is.
The control plane in k8s does not spin up new nodes when you define node pools, but rather the cloud provider abstracts this away from you, behind the scenes. Try deploying a kubeadm cluster, and you'll see there is no concept of node pools that k8s spins up for you. K8s would have to be aware of physical/virtual nodes in order to then deploy a physical/virtual server to join the cluster, and this is what a cloud provider should do for you There is much, much more to provisioning physical, and virtual servers to a cluster. Consider the following steps for provisioning a new worker node to an existing k8s cluster. 1. Procure new physical hardware OR provision VM (assuming a hypervisor already exists for the VM) 2. Cable new physical hardware OR configure virtual switch/network for a VM 3. Install an OS 4. Configure server/VM for k8s 5. Join that server/VM to a running cluster as a worker node via kubeadm
@@AlienResidents Oh i know that, i just never thought about using a control loop system to go beyond what we generally see like a basic application deployment. The idea of having a desired state and an observed state can be applied to much much more than just deploying a nodejs service for example. With CRDs and custom Operators, you can create and manage ressources that makes sense for your application domain and with the control loop you continuosly watch that the system is in the desired state and take actions if not. You don't need to build some internal tools to do that, you can just take advantage of k8s.
There are many ways to create APIs. One of the many is with Crossplane. If that might be interesting for you, you'll find a course in this channel. I've been public shibg videos for it every Thursday.
We're embarking on building internal cloud APIs, and we basically made the same decision to use Kubernetes as an extensible API server. We dismissed Crossplane due to some of its limitations (for example, Crossplane compositions cannot include arbitrary Kubernetes manifests, only Crossplane's Managed Resources), and generally not a very good experience during PoC (providers failing randomly). But we actually found out that writing our own operators using Operator SDK or kubebuilder is not that scary as we thought. And currently we are looking for a "dumb" UI through which our engineers could observe our cloud's resources and do some operational tasks (for this we're thinking about using Argo Workflows as an execution engine). We still haven't found any good UI. So, to reiterate - I totally agree with all you're saying in this video. Do you have any good "dumb" UI recommendations?
Unfortinately, I do not have a dumb UI recomendation just yet. I am advising a project in that space to take that direction but I cannot guarantee that they will. I do not want to reveal which project that is right now. I'll do that if they move in that direction. Apart from that one, something in that direction is "cooking" in Upbound, but that will be limited to Crossplane so it might not be a good choice for you.
That depends on whether the question is what I'm proposing should be done or which existing tool I propose. If it's the former, I propose what I said in the video (make a UI that talks to APIs and not much more). If it's the latter, there is no solution that I am aware of but I did advice two projects to go in that direction (that was the inspiration for this video), so, hopefully, there will be a ready-to-go solution for those who do not want to build it (even though building it would be relatively easy.
@@DevOpsToolkit Been following you for years - really seems like this could be a capstone of all the work you've shared in this space. Here's hoping you and your team go the FOSS/CNCF route with this, or at least for the Crossplane service API component.
Amazing video. We are trying to build something you are taking about and bring super simple settings to developers to create entire environments simplifying input making it… dump. I’d be happy to get your feedback if possible
Not even finished the video yet, but man I want to say. I envy people who even are at the "ticket"-ops stage. Cause where I work, I have to do all the infrastructure shit myself as a developer. Grab VM image, install the image on to an esxi host, configure windows (yes, thats correct sadly), then I manually have to copy my application over via network share. CI/CD is not possible, because the IT deparment wont accept our lab machines in the environment. Of course, there is no cloud, only on prem, because air gapped envs. Sometimes I just want to hunch down in fetal position and cry like a baby.
your channel went down really fast, it used to be really good few months ago but you started to get greedy and sell us a usless product every other video and now this stupid idea. its a shame
I am not sure with this statement. Viktor is presenting tools with the pros and cons and it is on us to decide if they are delivering value for our organization.
Is your developer console (portal) dumb?
Perhaps you should explain what a "Developer Platform Console" is before you proceed?
My bad. I should have done that.
It's a front-end of your developer portal. It's a graphical user interface similar to, for example, AWS/GCP/Azure console and often obtained through Backstage, Port, or similar solutions.
You have explained it in your other videos.
Great video Victor, I 💯 agree with you that having an UI to build an IDP should be very very optional and not required.
In CNOE we have some helper tools to discover XRDs and build the forms in Backstage then Backstage creates the k8s YAML in git for ArgoCD
That's how it should be. It does not necessarily have to be about XRDs. The same logic could, for example, discover that knative is in the cluster and paint fields based on its schema as well.
Thank you for this video! As usual you Inspire us for new ideas. One time you inspired us to use Charm for our CLI tools. And know where we use it already? We are making UI for create K8S resources by filling forms generated by API schema. Thank you for confirming this great idea! btw: Crossplane is on background ;)
I agree in principle, but often we make UIs smart so that people can be dumb. For example, at Netflix we had a UI to create a DB which had smart defaults based on the workload, with the associated costs computed at the bottom. Need a low-risk DB for internal only apps? Or maybe a business-critical DB serving millions of RPS across 3 regions? No problem, pick your profile, tweak your settings as needed, and acknowledge the cloud costs. UIs can provide a high-context interface for people who don't want to think too hard, they just have a job to do and don't want easily-avoidable emergencies or surprises. These sorts of smarts should be incrementally added, but as you rightly point out, most people go straight to over-engineering! How do we get the best of both?!
Those defaults/smarts can all be baked into services instead of being hard coded into the frontend. After all, there is no guarantee that those services will be consumed only through the frontend.
@@DevOpsToolkit For sure, and we did source the data from other backends. It wasn't as simple though as just a UI generated from an API -- getting the UX right in certain situations required tweaks specific to the view in which it was presented. The devs using UIs were generally a different population from those VIMing YAMLs, with different needs and expectations.
@LeakMemory that is one of the main reason why I'm proposing creation of APIs. I do not mean usage of existing APIs but creation of custom ones that meet the needs of your users.
The CLI tools are just like the portals you talk about in relation to being "dumb". Each CLI tool you mentioned is versioned whether it is terraform, gcloud, aws, az, tf etc. The CLI tools don't go out and discover the API capabilities either. It would be a great world if CLI, and webUI tools/consoles went out and discovered API capabilities, but none do. The versioning of each tool is for stability and known support. Instead of making the tools "dumb", the developers try to match up known capabilities and often times backwards compatible, depending on versions. For example, in gcloud there are the beta and alpha channels, and in aws there there are API versions to choose from.
Great video as always, do wish you had emphasized GitOps more. In the ideal world, the panel would only need readonly access to Kubernetes to pull in information and write to the git-reop that the gitops tool pulls from and it applies Kubernetes. But that probably would just have made the video more complicated/to long.
I did not want to go towards GitOps since that would make the video much longer. Part of the reason is that GitOps is moving away from interacting with repos from pipelines to processes that listen to registries and update repos. I'll explore that in one of the upcoming videos.
@@DevOpsToolkit ah, yes, an other good topic. But if they disconnect GitOps tool from repo, ... the panel can't look in Kubernetes to figure out the correct repo.
Hey viktor i really liked this video,
One question still: do you mean than that the frontend UI should talk directly to the kubernetes server or use a backend proxy that will talk to kubernetes? Thanks!
Directly to kubernetes which you extended with CRDs.
i love your videos and would like to understand further, we built IDP that talks to our gitops ecosystem, we built API that wraps our gitops and the IDP talks to gitops api
and you are saying simply create tons of CRDS (for example we have openapidocs in gitops service and we have over 1000) wont that in the end crash the k8s cluster.. i mean that what helm2 releases saved as CM did we saved huge files and alot of them in the end causing problems to the k8s cluster itself..
moreover we have etcd size limit for single object we have to take in account here...
and also the solution of crossplane is awsome!! but it brings tons of CRD creating a strain on the cluster.
I would like to hear your opinion, but yet another simple but brilliant idea!
Do you have 1000 truly different services (not 1000 instances of services)?
I'm very curious how you built API that wraps gitops.
If I’d go that way I would probably need something like 6-9 different CRDs to describe my gitops data and in total I would have close to 5000instances of those CRDs
@matankeler2761 instances (resources or custom resources) are rarely the issue. If you have 5k instances, you'll have tens of thousands resources anyways (e.g. deployments, services, ingresses, etc.). What might be an issue is if you have thousands of CRDs but that is not your case.
Would agree with the gist of what you're saying, but then you mention the god awful complex abstraction of crossplane? :facepalm:
I did my best not to make that video about Crossplane but about a potential path forward. It does not have to be Crossplane. There are many tools that allow us to create APIs (CRDs) and controllers.
Crossplane abstractions (Compositions) can easily become complex but, as far as I know, there is nothing simpler that allows us to combine resources and expose them as a CRD.
I tend to (partially) solve that complexity by not writing YAML but genering it with CUE or pkl.
Again, another great video. We've been discussing this for a few months, and I'm implementing something like reflection for custom entities in backstage. Somehow I struggle with a generative approach, but this leads me to the question: Are you aware of any tool or framework that might be able to render react (backstage) UI from let's say JSON schema or the like?
Unfortinately, I haven't been using JS for a while now so I cannot recommend anything 😔
No problem. Thanks for answering.@@DevOpsToolkit
Let's assume I don't need UI but CLI would be sufficient for the developers using the platform. Does it change anything regarding what you were trying to prove? Are you suggesting CLI should be merely autogenerated from an API and a "dumb" piece of software, that enables operations, discoverability, and basic observability?
That's what I'm suggesting. That's why we have APIs.
For example, if you execute 'kubectl get something', something is not hard-coded in kubectl. There is no need to have it inside it since it will go to kube API and ask for "something" and the API will return it or throw an error saying that something does not exist.
Thanks for another great video! This is an interesting topic indeed. I agree that Developer Portals tools should be dumb and if I had to build one on top of Kubernetes I would architect it the way you described it too. You have CRDs, schemas, events and everything you need right there.
But in this case backstage would need access to the cluster to call the apis which might not be advisable.
Assuming that you're doing it with an account that has only read permissions, I don't see a problem with that, at least when discoverability is concerned. For operations, you can push to Git directly, through the kube API endpoint dedicated to that, or anything else. It's all about permissions you give and policies you have. Also, I am not necessarily talking about a cluster whether you run production but a control plane cluster. In that scenario, it is actually easier to secure a single control plane cluster than providing credentials to different tools that would perform operations directly. A single point of entry is always easier to secure than multiple points of entry.
Interesting
But I am thinking about IaC as mandatory. Most company need you write HCL, YAML or whatever as part of this part of the git ops solution.
If we developer a API to do this, where should I store the IaC?
Store IaC behind an API.
I don't think internal APIs and IaC are mutually exclusive. You can write custom a Terraform provider to access internal APIs, or, if you use Kubernetes, use many existing GitOps tools that apply YAML code to Kubernetes API.
A couple weeks ago, something clicked in my brain. I think you just summarized it pretty well in this video. Crossplane planted a seed in my mind.
Up to this point, i was only seeing K8s as infrastructure and basic application management. You define the node pools and all the hardware you want and let the control plane schedule that for you. You use helm for packaging your app and create your services.
But we can push that much further. Eg, you are building a multi-tenant system, why not create a Tenant CRD ? Have an operator scheduling everything that you need for that tenant. You need a stripe billing account ? It's own Sevice A, B, C ? It's using a shared Mongo cluster, but you want database isolation ? Same thing, CRDs & Operators.
It sounds good, i just don't know if it really is.
The control plane in k8s does not spin up new nodes when you define node pools, but rather the cloud provider abstracts this away from you, behind the scenes.
Try deploying a kubeadm cluster, and you'll see there is no concept of node pools that k8s spins up for you. K8s would have to be aware of physical/virtual nodes in order to then deploy a physical/virtual server to join the cluster, and this is what a cloud provider should do for you There is much, much more to provisioning physical, and virtual servers to a cluster.
Consider the following steps for provisioning a new worker node to an existing k8s cluster.
1. Procure new physical hardware OR provision VM (assuming a hypervisor already exists for the VM)
2. Cable new physical hardware OR configure virtual switch/network for a VM
3. Install an OS
4. Configure server/VM for k8s
5. Join that server/VM to a running cluster as a worker node via kubeadm
@@AlienResidents Oh i know that, i just never thought about using a control loop system to go beyond what we generally see like a basic application deployment.
The idea of having a desired state and an observed state can be applied to much much more than just deploying a nodejs service for example.
With CRDs and custom Operators, you can create and manage ressources that makes sense for your application domain and with the control loop you continuosly watch that the system is in the desired state and take actions if not.
You don't need to build some internal tools to do that, you can just take advantage of k8s.
I would love a API course following that kubernetes parttern. I took your udemy course and was very nice.
There are many ways to create APIs. One of the many is with Crossplane. If that might be interesting for you, you'll find a course in this channel. I've been public shibg videos for it every Thursday.
@@DevOpsToolkit awesome thanks
Great insights on IDPs.
We're embarking on building internal cloud APIs, and we basically made the same decision to use Kubernetes as an extensible API server. We dismissed Crossplane due to some of its limitations (for example, Crossplane compositions cannot include arbitrary Kubernetes manifests, only Crossplane's Managed Resources), and generally not a very good experience during PoC (providers failing randomly). But we actually found out that writing our own operators using Operator SDK or kubebuilder is not that scary as we thought.
And currently we are looking for a "dumb" UI through which our engineers could observe our cloud's resources and do some operational tasks (for this we're thinking about using Argo Workflows as an execution engine). We still haven't found any good UI.
So, to reiterate - I totally agree with all you're saying in this video. Do you have any good "dumb" UI recommendations?
Unfortinately, I do not have a dumb UI recomendation just yet. I am advising a project in that space to take that direction but I cannot guarantee that they will. I do not want to reveal which project that is right now. I'll do that if they move in that direction.
Apart from that one, something in that direction is "cooking" in Upbound, but that will be limited to Crossplane so it might not be a good choice for you.
Crossplane compositions can create any k8s resources you need thanks to kubernetes provider
I would to see more cloudbee and Jenkins content
I don't work with Jenkins and/or CloudBees much these days so I might not be the best person to do such content.
what solutions do you propose?
That depends on whether the question is what I'm proposing should be done or which existing tool I propose. If it's the former, I propose what I said in the video (make a UI that talks to APIs and not much more). If it's the latter, there is no solution that I am aware of but I did advice two projects to go in that direction (that was the inspiration for this video), so, hopefully, there will be a ready-to-go solution for those who do not want to build it (even though building it would be relatively easy.
may be you can start a product like that ? ;)
@@DevOpsToolkit
@adrien-barret maybe the team working with me already started working on something like that.
@@DevOpsToolkit Been following you for years - really seems like this could be a capstone of all the work you've shared in this space. Here's hoping you and your team go the FOSS/CNCF route with this, or at least for the Crossplane service API component.
Amazing video. We are trying to build something you are taking about and bring super simple settings to developers to create entire environments simplifying input making it… dump. I’d be happy to get your feedback if possible
That sounds great. Send me a DM on LinkedIn or Twitter and we can schedule a chat.
Not even finished the video yet, but man I want to say. I envy people who even are at the "ticket"-ops stage. Cause where I work, I have to do all the infrastructure shit myself as a developer. Grab VM image, install the image on to an esxi host, configure windows (yes, thats correct sadly), then I manually have to copy my application over via network share. CI/CD is not possible, because the IT deparment wont accept our lab machines in the environment. Of course, there is no cloud, only on prem, because air gapped envs. Sometimes I just want to hunch down in fetal position and cry like a baby.
I feel for you. I was in, more or less, the same position a while ago.
your channel went down really fast, it used to be really good few months ago but you started to get greedy and sell us a usless product every other video and now this stupid idea. its a shame
I am not sure with this statement. Viktor is presenting tools with the pros and cons and it is on us to decide if they are delivering value for our organization.