Holy smokes, I have been struggling with grasping the stable/prometheus-operator helm deployment. 40K lines of yaml for us. This made it all clear, thank you!
I would stay clear of stable helm upstream charts. They're great to study implementation details, however they are super over-engineered, hide things they install on your cluster and also make it nearly impossible to upgrade components later down the track. Prometheus eco-system changes too often and doing diff's between charts a few months apart is a nightmare. My DevOps top tip = "Keep it Simple" :)
@Marcel If i may request - i am currently evaluating `Integrating Thanos with prometheus operator" to solve known problems associated with prometheus. > Global Query > Longer Retention > HA If you could explore that option and extend this prometheus operator series. Thank you :)
Nice video man - What do you mean on 9:35: "Now you can go ahead and create as many prometheus instances as we wanted" - "This is completely up to you how you want to design this" - Did you install the Operator right? Now I need to install a new prometheus deployment in every namespace where I want to monitor? Is that is? How can I install the Prometheus in the k8s cluster where I can monitoring everything that's running in there? Thanks again!
There are a number of ways you can achieve this. a) Expose Prometheus using an ingress over TLS and something like basic authentication with IP restriction can help. b) Use Prometheus federation to move data to a local Prometheus grafana can connect to. Federate endpoints can also be protected by an ingress controller. c) Another method ive come to learn about is Remote write. Prometheus instances can write data out to an external endpoint in your grafana cluster. This may help reduce number of exposed endpoints you have to protect. Hope this helps 💪🏽
Is there any document for Prometheus to start from scratch? i mean i have knowledge on Kubernetes now since i have CKA exam completed. I need to know how to install Prometheus on nodes exactly do we need to install on master node or worker node? or we need separate node to get Prometheus installed? from this videos i understood that Prometheus is a tool which i will pull the metrics logs from the kubernetes objects and then we need some visualization tools like Graphana to display the logs pulled by Prometheus do we need lots of storage space to get these logs stored ?
Great material! How to aggregate the metrics if we have multiple Prometheus instances running into cluster? And can we aggregate metrics from k8s multi-cluster may be deployed in Gardner? Thanks in Advance.
Thank you for the kind words 🤓 There are a few approaches you could take. If you are running multiple prometheus instances you can either replicate the data to a central prometheus instance by using the /federate API. Prometheus instances can scrape each other to move data out to a central one for dashboarding. Another approach is have every prometheus instance push its metrics out to a central storage using the "remote-write" feature. You can then use that data store as a data source for dashboarding. Hope that helps 💪🏽
Like your style! If we have multiple replicas of prometheus in a deployment, for resilience.. they will double-up the same stats and network iops? Do you recommend only 1 prom pod per deployent / stateful set? Have you tried the stable/prom-operator helm that bundles grafana, kube-state-metrics, etc..? it looks good but seems to run 1 prometheus that pulls in everything in that infrastructure scope. I guess you extend it per your example here .. and add that new prom service DB / datasource to grafana? A lot is not transparent in the helm deployments... quick but you lose control.
One thing to note is that Prometheus architecture has high availability in mind. You dont need many instances scraping the same targets. What I would recommend is perhaps slicing it up to where you have 1 instance per subset of targets. Like 1 that looks after node-exporter. Perhaps 1 that is dedicated to kube-state-metrics, 1 for API server, something along those lines. By distributing the responsibilities of Prometheus, you get improved resilience. Same goes for 1 prometheus per group of micro services. Another thing to note is that if prometheus is down, you persist data on something like a persistent volume (statefulset). You won't lose metrics. As long as Prometheus comes up in time it will rescrape and continue to collect stats that may have been missed during downtime. You can also increase availability by federating all the instances data across to other Prometheus instances with the /federate endpoint. Dashboards can point to a federated copy too, increasing resilience even more. Prometheus is complex, but extremely flexible and the above statements make it very resilient, when setup with availability in mind.
@@MarcelDempers Thanks for the reply!! I agree. I was working with the helm stable/prometheus-operator which comes with a load of useful stuff to get you up and running (Alertmanager, Grafana with Dashboards, all the K8 infrastructure ServiceMonitors, etc). I was worried it wouldn't be modifiable - but you can add a new Prom instances the way you show here for the application layer etc. You can even move off kube-state-metrics, etc to dedicated promethei too if that 1 instance gets too big. Persistence works fine and new prom's all get added as stateful sets. I just create a new grafana data source as they get added. So it's just a helm release with a few patches applied to customise it... very cool!! Looks like I've done a month of work 😆👍
Thanks for the great video. can i ask a quick question. Does prometheus pulls the metrics directly from the containers/pods ? it doesnot need cadvisor ? Also, if I understood correctly, prometheus pulls the metrics using http . Does it mean every container should listen to HTTP requests from prometheus ?
For container level performance metrics, they are exposed by the kubelet's cAdvisor endpoint and scraped by Prometheus. Similar metrics come from the kube-state-metrics server as well. You only need to expose HTTP endpoint on your service if you have custom application metrics that you want to grab and then you'd use a ServiceMonitor to scrape those
Hello sir , i got stuck in helm chart of prometheus-operator. In my project i deployed my php application under kube-system namespace. And install the prometheus operator helm chart under the monitoring namespace. Now i want to monitor my application (not the nodes and kube-system just the applications). What will be the direction path for it?
executed the operator directory... had errors, did a tear down, is there a order in which i can execute the yaml files manually, after having created the namespace.
Hi, thanks for the nice video. i have a question. if after setting up Prometheus for my cluster, i later on added decided to scale my cluster by adding new VMs, will the existing node exporter pick the metrics for the new VMs? or will i have to configure the node exporter for the new VMs? Thank you
This works for me to get things installed, but how do I add my targets (my custom Apps)? I know with prometheus VM I populate my static_configs in /etc/prometheus/prometheus/prometheus.yml file. but I have no clue how to accomplish this with the helm approach. Also, I have not been able to locate any documentation on how to do this.
For grafana, the pods do not come up and I see the error below for the pod events Normal Scheduled 2m29s default-scheduler Successfully assigned monitoring/grafana-f667ddbdc-2pk2p to ip-10-2-100-235.ec2.internal Warning FailedMount 26s kubelet, ip-10-2-100-235.ec2.internal Unable to attach or mount volumes: unmounted volumes=[grafana-dashboard-nodeexporter], unattached volumes=[grafana-dashboard-node-cluster-rsrc-use grafana-dashboards grafana-dashboard-k8s-resources-workloads-namespace grafana-dashboard-nodeexporter grafana-dashboard-proxy grafana-dashboard-namespace-by-pod grafana-dashboard-namespace-by-workload grafana-dashboard-node-rsrc-use grafana-dashboard-statefulset grafana-dashboard-k8s-resources-node grafana-datasources grafana-dashboard-scheduler grafana-dashboard-k8s-resources-workload grafana-dashboard-k8s-resources-pod grafana-dashboard-pods grafana-dashboard-workload-total grafana-dashboard-nodes grafana-dashboard-prometheus-remote-write grafana-storage grafana-dashboard-cluster-total grafana-dashboard-controller-manager grafana-dashboard-kubelet grafana-dashboard-prometheus grafana-dashboard-apiserver grafana-dashboard-k8s-resources-namespace grafana-dashboard-pod-total grafana-dashboard-persistentvolumesusage grafana-dashboard-k8s-resources-cluster grafana-token-xs4fs]: timed out waiting for the condition What am I missing here? I am simply applying everything from your repo.
This does seem strange. Grafana mounts a ton of configmaps where the dashboard definitions are stored. Make sure the configmaps are created. You have to kubectl create not apply on the configmaps because the files are very large. Also make sure you're creating configmaps and the deployments in the right namespace as the tutorial suggests. Check ruclips.net/video/abq-6FBn0fo/видео.html too for more info
@@MarcelDempers Yes thanks, using create instead of apply works but what is the rationale behind that, you mentioned to use create instead of apply for large config map files, I am wondering if I were to run this in a CI pipeline like argo, it will do an apply as part of sync and will fail, any thoughts on this?
I have a question, can we edit the prometheus.yaml file, I mean the configuration file. not the one that is in the repository. If yes how can we do that..?
The Prometheus CRD provides the interface to configure prometheus so you dont have to touch and manage the underlying configuration. ServiceMonitors are also an interface to this. You can check the docs here github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#prometheusspec
Really Nice explanation, I am getting this error ---> error: unable to recognize "service-monitor.yaml": no matches for kind "servicemonitor.monitoring.coreos.com" in version "monitoring.coreos.com/v1"
Thank you 💪🏽 Older version of the operator will create the CRDs for service monitors automatically when starting up. Once the operator is running you can retry adding service monitors. OR - You can deploy the CRD's separately like I do in this video ruclips.net/video/abq-6FBn0fo/видео.html
Hello Marcel, great videos in your channel and great you have a series of prometheus and grafanna !!! One question, i try to apply your yamls and i get this error... error: unable to recognize "service-monitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1" I guess it will be the version of my k8s cluster (18.0)... !! Greetings from argentina :D ! I deploy it again after all deploy and it deploy it well... depends of the cluster speed to create all the instances :D !
Thanks mate💪🏽 Just a notice, the prometheus operator needs to be running first. When it runs, it creates the servicemonitor CRDs so that you can apply the service monitor. You can only apply the service monitors once the operator has initialized properly 🤓If you apply all the YAML files too fast, the ServiceMonitor CRD will not have been created and you will face that error. hope that helps
@@MarcelDempers Thanks for the advice Marcel ! another question... i have problems to scrape kubelet if i deploy all your yamls... and i cant found where the error is, i think because your yamls are for minikube ? i deploy a k8s HA cluster on prem... cheers !!
Sure you can. You can run it as a deployment and simply keep data in memory or If the data is important to keep you can use a statefulset and appropriate volume to persist data to.
@@MarcelDempers Can you please make a video where we monitor the ingress controller for nginx and then get the required matrix. In one of the video you have made is only to use ingress controller and there I did not understand how to get the domain name?
An application needs to expose a "/metrics" endpoint to a Prometheus instance (not the operator) for this to work. A service monitor needs to select the application service correctly too
Your source code links do not work. On this page there is a lot of broken links associated with you series ( github.com/marcel-dempers/docker-development-youtube-series) click on almost any Source link. (Found it)
This is soo helpful!!! I've been struggling for a while now trying to understand operators and how they work... Thank you so so much! 😊😊
Holy smokes, I have been struggling with grasping the stable/prometheus-operator helm deployment. 40K lines of yaml for us. This made it all clear, thank you!
I would stay clear of stable helm upstream charts. They're great to study implementation details, however they are super over-engineered, hide things they install on your cluster and also make it nearly impossible to upgrade components later down the track. Prometheus eco-system changes too often and doing diff's between charts a few months apart is a nightmare. My DevOps top tip = "Keep it Simple" :)
@@MarcelDempers true true ... been there ...
I love your videos because a learn to use tools with a real-world approach. Thanks to sharing you Knology.
Great work man! Just keep releasing new videos and your channel will be getting more popular. Marry Christmas and good luck!
@Marcel If i may request - i am currently evaluating `Integrating Thanos with prometheus operator" to solve known problems associated with prometheus.
> Global Query
> Longer Retention
> HA
If you could explore that option and extend this prometheus operator series. Thank you :)
Nice video man - What do you mean on 9:35: "Now you can go ahead and create as many prometheus instances as we wanted" - "This is completely up to you how you want to design this" - Did you install the Operator right? Now I need to install a new prometheus deployment in every namespace where I want to monitor? Is that is? How can I install the Prometheus in the k8s cluster where I can monitoring everything that's running in there? Thanks again!
My How to monitor Kubernetes in 2022 video covers this 👍🏼💪🏽
Yes, you use the operator.
Excellent explanation....was struggling a alot now cleared all my doubts....pls make a video for the same using helm chart pls make it soon
Merry Christmas Marcel, nice video, such Prometheus helps DevOps have a better holiday, lol.
Thanks a lot! Your videos are amazing. Liked all of them.
I haven't found any good video about RBAC. Please make one!
Thanks for making these videos. Very helpful! :)
very clearly explianed, thank you!
Hi
Nice explanation. Do you share deployment of prometheus operator using dynamic provision storage.
Regards
Zaheer
How do you recommend exposing this service so that it can be added as a data source to Grafana in a different k8s cluster?
There are a number of ways you can achieve this. a) Expose Prometheus using an ingress over TLS and something like basic authentication with IP restriction can help. b) Use Prometheus federation to move data to a local Prometheus grafana can connect to. Federate endpoints can also be protected by an ingress controller. c) Another method ive come to learn about is Remote write. Prometheus instances can write data out to an external endpoint in your grafana cluster. This may help reduce number of exposed endpoints you have to protect. Hope this helps 💪🏽
Can you do a video on the crunchy Postgres operator or Zalando operator?
Thx for the explanations ! Congratz !
Is there any document for Prometheus to start from scratch?
i mean i have knowledge on Kubernetes now since i have CKA exam completed.
I need to know how to install Prometheus on nodes exactly do we need to install on master node or worker node?
or we need separate node to get Prometheus installed?
from this videos i understood that Prometheus is a tool which i will pull the metrics logs from the kubernetes objects and then we need some visualization tools like Graphana to display the logs pulled by Prometheus
do we need lots of storage space to get these logs stored ?
Very helpful, thanks!
Can you speak of the Thanos sidecar and other Thanos services offered with the Prometheus operator?
Great material! How to aggregate the metrics if we have multiple Prometheus instances running into cluster? And can we aggregate metrics from k8s multi-cluster may be deployed in Gardner?
Thanks in Advance.
Thank you for the kind words 🤓 There are a few approaches you could take. If you are running multiple prometheus instances you can either replicate the data to a central prometheus instance by using the /federate API. Prometheus instances can scrape each other to move data out to a central one for dashboarding.
Another approach is have every prometheus instance push its metrics out to a central storage using the "remote-write" feature. You can then use that data store as a data source for dashboarding. Hope that helps 💪🏽
Thanos querier component is exactly designed to do - merge and deduplicate similar metrics. Visit thanos.io to know more
Like your style! If we have multiple replicas of prometheus in a deployment, for resilience.. they will double-up the same stats and network iops? Do you recommend only 1 prom pod per deployent / stateful set? Have you tried the stable/prom-operator helm that bundles grafana, kube-state-metrics, etc..? it looks good but seems to run 1 prometheus that pulls in everything in that infrastructure scope. I guess you extend it per your example here .. and add that new prom service DB / datasource to grafana? A lot is not transparent in the helm deployments... quick but you lose control.
One thing to note is that Prometheus architecture has high availability in mind.
You dont need many instances scraping the same targets. What I would recommend is perhaps
slicing it up to where you have 1 instance per subset of targets. Like 1 that looks after node-exporter.
Perhaps 1 that is dedicated to kube-state-metrics, 1 for API server, something along those lines.
By distributing the responsibilities of Prometheus, you get improved resilience.
Same goes for 1 prometheus per group of micro services.
Another thing to note is that if prometheus is down, you persist data on something like a persistent volume (statefulset). You won't lose metrics. As long as Prometheus comes up in time it will rescrape and continue to collect stats that may have been missed during downtime.
You can also increase availability by federating all the instances data across to other Prometheus instances with the /federate endpoint.
Dashboards can point to a federated copy too, increasing resilience even more.
Prometheus is complex, but extremely flexible and the above statements make it very resilient, when setup with availability in mind.
@@MarcelDempers Thanks for the reply!! I agree. I was working with the helm stable/prometheus-operator which comes with a load of useful stuff to get you up and running (Alertmanager, Grafana with Dashboards, all the K8 infrastructure ServiceMonitors, etc). I was worried it wouldn't be modifiable - but you can add a new Prom instances the way you show here for the application layer etc. You can even move off kube-state-metrics, etc to dedicated promethei too if that 1 instance gets too big. Persistence works fine and new prom's all get added as stateful sets. I just create a new grafana data source as they get added. So it's just a helm release with a few patches applied to customise it... very cool!! Looks like I've done a month of work 😆👍
Thanks for the great video. can i ask a quick question. Does prometheus pulls the metrics directly from the containers/pods ? it doesnot need cadvisor ? Also, if I understood correctly, prometheus pulls the metrics using http . Does it mean every container should listen to HTTP requests from prometheus ?
For container level performance metrics, they are exposed by the kubelet's cAdvisor endpoint and scraped by Prometheus. Similar metrics come from the kube-state-metrics server as well.
You only need to expose HTTP endpoint on your service if you have custom application metrics that you want to grab and then you'd use a ServiceMonitor to scrape those
Hello sir , i got stuck in helm chart of prometheus-operator. In my project i deployed my php application under kube-system namespace. And install the prometheus operator helm chart under the monitoring namespace. Now i want to monitor my application (not the nodes and kube-system just the applications). What will be the direction path for it?
Can you use the Prometheus operator to monitor for example barebones machines running node exporter on them?
Very interesting content, thanks for this lecture! 5* feedback.
Do you have anything on getting spark application metrics using prometheus?
really useful video, helps a lot! Thanks you!
Great content. Keep it up! Thanks!
executed the operator directory... had errors, did a tear down, is there a order in which i can execute the yaml files manually, after having created the namespace.
legendary video
Hi
Could you plz guide us to install snmp exporter in kubernetes
How do you add Kube state metrics to Prometheus with this set up?
video coming soon 👍
Hi, thanks for the nice video. i have a question. if after setting up Prometheus for my cluster, i later on added decided to scale my cluster by adding new VMs, will the existing node exporter pick the metrics for the new VMs? or will i have to configure the node exporter for the new VMs? Thank you
node-exporter should be running as a daemonset, so it will scale to the new machines when they start up
@@MarcelDempers okay, thank you
This works for me to get things installed, but how do I add my targets (my custom Apps)? I know with prometheus VM I populate my static_configs in /etc/prometheus/prometheus/prometheus.yml file. but I have no clue how to accomplish this with the helm approach. Also, I have not been able to locate any documentation on how to do this.
Got two videos coming soon that address this. You want to learn about Service monitors
@@MarcelDempers awesome! Thanks
i am a devops fresher and i have just come across prometheus and grafana. is there any step by step learning video on these technologies?
Nice! Welcome, here is a playlist with step by step content and source code ruclips.net/p/PLHq1uqvAteVuEXCrRkPFWLXRKWNLOVUHn
💪🏽
@@MarcelDempers thanks 😊..also can you make some videos on managed services offered by cloud providers like EKS and AKS?
Great video. But wearing an Arch Linux T-shirt whilst recording on Windows? :D
I wonder how much Marcel can bench press
... the prometheus-operator directory does not seem to be part of your directory structure anymore ?
All k8s prometheus related items are under👉🏽 ./monitoring/prometheus/kubernetes/README.md
For grafana, the pods do not come up and I see the error below for the pod events
Normal Scheduled 2m29s default-scheduler Successfully assigned monitoring/grafana-f667ddbdc-2pk2p to ip-10-2-100-235.ec2.internal
Warning FailedMount 26s kubelet, ip-10-2-100-235.ec2.internal Unable to attach or mount volumes: unmounted volumes=[grafana-dashboard-nodeexporter], unattached volumes=[grafana-dashboard-node-cluster-rsrc-use grafana-dashboards grafana-dashboard-k8s-resources-workloads-namespace grafana-dashboard-nodeexporter grafana-dashboard-proxy grafana-dashboard-namespace-by-pod grafana-dashboard-namespace-by-workload grafana-dashboard-node-rsrc-use grafana-dashboard-statefulset grafana-dashboard-k8s-resources-node grafana-datasources grafana-dashboard-scheduler grafana-dashboard-k8s-resources-workload grafana-dashboard-k8s-resources-pod grafana-dashboard-pods grafana-dashboard-workload-total grafana-dashboard-nodes grafana-dashboard-prometheus-remote-write grafana-storage grafana-dashboard-cluster-total grafana-dashboard-controller-manager grafana-dashboard-kubelet grafana-dashboard-prometheus grafana-dashboard-apiserver grafana-dashboard-k8s-resources-namespace grafana-dashboard-pod-total grafana-dashboard-persistentvolumesusage grafana-dashboard-k8s-resources-cluster grafana-token-xs4fs]: timed out waiting for the condition
What am I missing here? I am simply applying everything from your repo.
This does seem strange. Grafana mounts a ton of configmaps where the dashboard definitions are stored. Make sure the configmaps are created. You have to kubectl create not apply on the configmaps because the files are very large.
Also make sure you're creating configmaps and the deployments in the right namespace as the tutorial suggests.
Check ruclips.net/video/abq-6FBn0fo/видео.html too for more info
@@MarcelDempers Yes thanks, using create instead of apply works but what is the rationale behind that, you mentioned to use create instead of apply for large config map files, I am wondering if I were to run this in a CI pipeline like argo, it will do an apply as part of sync and will fail, any thoughts on this?
how to run 1 prometheus instance to monitor 2 seperate services in 2 seperate namespace
Take a look at ServiceMonitors , you can select a service
amespace to scrape and tell Prometheus which ServiceMonitors to select
I think background musics between chapters are little bit loud? Almost died of heart attack.
I have a question, can we edit the prometheus.yaml file, I mean the configuration file. not the one that is in the repository. If yes how can we do that..?
The Prometheus CRD provides the interface to configure prometheus so you dont have to touch and manage the underlying configuration. ServiceMonitors are also an interface to this.
You can check the docs here github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#prometheusspec
@@MarcelDempers So how do I edit the present configuration of the current scrape configuration. ?
Hello,
I think, now is the time to do Thanos-Operator.. or just thanos.
Really Nice explanation, I am getting this error ---> error: unable to recognize "service-monitor.yaml": no matches for kind "servicemonitor.monitoring.coreos.com" in version "monitoring.coreos.com/v1"
Thank you 💪🏽
Older version of the operator will create the CRDs for service monitors automatically when starting up. Once the operator is running you can retry adding service monitors.
OR - You can deploy the CRD's separately like I do in this video ruclips.net/video/abq-6FBn0fo/видео.html
Hello Marcel, great videos in your channel and great you have a series of prometheus and grafanna !!! One question, i try to apply your yamls and i get this error... error: unable to recognize "service-monitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
I guess it will be the version of my k8s cluster (18.0)... !! Greetings from argentina :D !
I deploy it again after all deploy and it deploy it well... depends of the cluster speed to create all the instances :D !
Thanks mate💪🏽 Just a notice, the prometheus operator needs to be running first. When it runs, it creates the servicemonitor CRDs so that you can apply the service monitor. You can only apply the service monitors once the operator has initialized properly 🤓If you apply all the YAML files too fast, the ServiceMonitor CRD will not have been created and you will face that error. hope that helps
@@MarcelDempers Thanks for the advice Marcel ! another question... i have problems to scrape kubelet if i deploy all your yamls... and i cant found where the error is, i think because your yamls are for minikube ? i deploy a k8s HA cluster on prem... cheers !!
Can I use Prometheus without installing or deploying Prometheus Operator?
Sure you can. You can run it as a deployment and simply keep data in memory or If the data is important to keep you can use a statefulset and appropriate volume to persist data to.
@@MarcelDempers Can you please make a video where we monitor the ingress controller for nginx and then get the required matrix. In one of the video you have made is only to use ingress controller and there I did not understand how to get the domain name?
Does the python application have to be exposed to Prometheus metrics for this to work?
An application needs to expose a "/metrics" endpoint to a Prometheus instance (not the operator) for this to work. A service monitor needs to select the application service correctly too
Marcel Dempers -DevOpsGuy thank you for the quick response!
Would like to see thanos on kubernetes.
Your source code links do not work. On this page there is a lot of broken links associated with you series ( github.com/marcel-dempers/docker-development-youtube-series) click on almost any Source link. (Found it)