We use Dynatrace today for tracing.. I will investigate openTelemetry and see how it can add value and prevent vendor lock-ing. Thanks for another great vid!.
Thanks for the content, Viktor. Would love to see a trilogy of videos on 1. the otel instrumentation needed to get red metrics, such that one can answer health questions 2. instrument more rich spans around the places where metrics are instrumented and unlock answering questions about change. 3. wrap it up with trying to answer health and change questions in various backends (garafana, datadog, honeycomb).
Huge topic for just one video, a serie for this, digging in the different parts (tracing, monitoring, alerting and logging) and the integration with other tools, grafana, Jaeger, Prometheus etc..., would be terrific 😊. Anyway thanks for your work, keep it up!
Thanks for these series! Open tracing and Open Census were good project and with Open Telemetry it also comes with auto-instrumetation. For Java there are Java agents based that propagates the headers, traces, spams, etc.. to upstream services.
Excellent and concise overview. Thank you! Teams must also be mindful of how this data is used and queried. From what you said in this video, I think you'll get into that in more detail in the next video.
as you asked, in term of Otel Receiver, you might consider Grafana Tempo as Receiver and Visualization, as you have all three types of Otel in one Visualization, Traces, Logs, metrics, and cold even correlate them (implemented right)
@@DevOpsToolkit BTW, do you know Crossplane (runtime) will get OTEL Tracing one day, I think it's a great fit for such a tool interacting with so. many components (Compositions, Providers) down and upstream...
Prometheus will fetch metrics from places where metrics exist (e.g. kubernetes API). Your app needs to be instrumented to provide those metrics so you do need to instrument it besides having those tools if you want app-specific metrics to be in Prometheus.
Often, I wait to see whether people are interested in a topic I propose and, if they are, I start working on it some time in the future. That often results in it coming weeks or months later. Jaeger is different. Many already asked for it and I'm already on it. It will be released next week.
You mentioned several times that we should use OpenTelemetry to be able to switch one observability solution to another. I remember a time, when people had the same argument for ORMs, which allegedly allow to switch database, but give you least common API. So you kinda exchange part of features to possibility of changing database/observability solution. Maybe in some cases a Datadog agent is not that bad, especially if we consider that observability consist of several parts (agents, collectors, storage backend, UI and so on, and one vendor could integrate these parts better, than common solution).
Many other attempts to standardize something failed because of the lack of support. Having a standard that is not implemented by many is not very useful. OpenTelemetry, however, is one of the few "standards" that managed to rally everyone. Almost every vendor or a project in the observability space supports it or is working on the support (including Datadog). Something similar happened to Kubernetes API a bit earlier (almost everyone gathered behind it). So, OpenTelemetry is different than others due to the size of its adoption and contributions coming from vendors and projects. As a counter example, we have OAM (Open Application Model) for which I had high hopes but which failed to convince a significant percentage of the industry to adopt it.
One more thing... I don't care much about agents. They can be OpenTelemetry or anything else since there is almost no work involved in switching from one to the other. What I do care is what we use to instrument our code since that represents invesment (mainly in time).
Datadog supports otel pretty well but it works much better with the agent. Same with New Relic, Instana, splunk and most of the hosted services out there. It's how the softly push you in to vendor lock in to make it less appealing to switch away.
@@davehoffman515 That follows typical stages of adoption of standards. 1. We will not support this. Our stuff is better and we have the competitive edge. 2. We have to support this because others are supporting it and prospects are asking for it, but we will not make it as good as our own stuff. 3. We have to support it well because our customers do not want to use our own API. They are leaving because they do not want to be locked in. Most vendors are in the phase 2 right now, and will be moving towards the phase 3 soon. P.S. Most attempts to standardize something never pass the phase 1 and eventually die. OpenTelemetry managed to get to the phase 2 thus (probably) securing its future.
OpenTelemetry does not replace any specific tool or, to be more precise, does not have to replace anything. It's mostly focused on standardizing the API that other tools might use and instrumenting your code with that API.
@@DevOpsToolkit When I access the webapp (not node.js) which is a java based webapp, can I get the js files and the js functions that gets invoked from the browser side?
I must NOT ask my app vendors to do additional coding (plus maintaining it) that will cost for "emit" when that application consumed unless consumer is really needed that if and when. Or stick with git-driven config change enforcement for enabling those information elements that will steer app to ship that that to a remote server via pull or push rather, some stuff we inherit from legacy not necessarily bad but optimized over ages of operational experience and cost for it.
The spec for logs has been done a while ago (but might change over time). As for implementations, it depends on the language you're using. Some are stable and others are in beta. Both are stable and can be used. There are a few that are in development and you might want to avoid those.
Robusta is two dudes spamming Linkedin constantly with their vision about "Limits make no sense, dont put limits in your deployments"... After short discussion with "CEO" stopped wasting my time with them.
@@DevOpsToolkit ok cool, so means I install only OTEL and Grafana is enough for me, no need for Promothous and loki ?? can you help me understand this more ?
OTEL Is, first and foremost a protocol that most vendors (both open source and commercial) are adopting. On top of that there are OTEL agents. They are optional and are used to aggregate, filter, and transport data. You often do not need those. There are at least three types of data. There are metrics, traces, and logs. You can collect and store metrics with Prometheus, logs with Loki, and traced with Jaeger. There are many other options available, some are open source while others are commercial. Grafana does not store and data. It is used to visualize data from other places (like those i mentioned).
I have conducted research on enabling OpenTelemetry for multi-services written in Go, and I have chosen to utilize Signoz as a visualization tool that combines traces, metrics, and logs. Signoz is also an open-source project, and I highly recommend that you check it out.
But there is a big problem with OTel itself. It tries to do tooo much and is unjustifiably complex and bloated. Classical stupidity of OTel designers called intellectual masturbation: observability, metrics, logs, traces, spans, layers, collectors, context propagators, receiver, connector, backend, etc.....other $4it.
I agree. It is complex and bloated, mostly as a result of too many companies contributing and adding "stuff" that matters to them. As far as I know, they already started working on reducing the spec. In spite of that, it is a good thing that we do not have to instrument code using vendor-specific libraries/specs and being forever locked.
@@DevOpsToolkit Totally agree with you. I am not fan of AWS, GC.etc, however this is what happens in the end: > Here is the kicker: Despite all the bloat and inefficiency, OpenTelemetry still feels incomplete. This is because large enterprises are more interested in creating an open-source product that is ‘good enough’ to get us in the door but not so good that it satisfies all our needs. If you want the full observability package - optimized performance, advanced analytics, smooth scaling - you would be nudged toward their proprietary tools and services. It is a classic bait-and-switch. In other words, OpenTelemetry is functional enough for basic observability, but when things get complex, you are often pushed into premium enterprise add-ons. It is not a coincidence - this is part of the corporate strategy. They contribute just enough to the open-source project to make it widely adopted but hold back the best features for their paid offerings. (excerpt)
Do you instrument your code?
Hi Viktor, you type it "Robuta" in description, hope you find this usefull
Thanks for the note. Corrected :)
We use Dynatrace today for tracing.. I will investigate openTelemetry and see how it can add value and prevent vendor lock-ing. Thanks for another great vid!.
@@fpvclub7256 Dynatrace support OpenTelemetry (not sure whether they support all all of or only a subset), so that part should not be a problem.
Thanks for the content, Viktor. Would love to see a trilogy of videos on 1. the otel instrumentation needed to get red metrics, such that one can answer health questions 2. instrument more rich spans around the places where metrics are instrumented and unlock answering questions about change. 3. wrap it up with trying to answer health and change questions in various backends (garafana, datadog, honeycomb).
Man, this is a best, most clear and concise answer to the question on what is OT. You are simply the best!
Dear, you have one of the best channels about DevOps. I always enjoy your content. Please keep it on.
Wow, thank you!
This channel is my daily podcast ❤
Huge topic for just one video, a serie for this, digging in the different parts (tracing, monitoring, alerting and logging) and the integration with other tools, grafana, Jaeger, Prometheus etc..., would be terrific 😊. Anyway thanks for your work, keep it up!
Jaeger video was just published and is available at ruclips.net/video/FK0uh-7nDSg/видео.html.
Great video on OpenTelemetry! It's definitely a game-changer in the world of observability. Can't wait for the future videos.
I love your overviews so much! Thank you and good luck!
Need to do a full devops training course. You are a very good teacher
My new favorite channel 😃❤ thank you Viktor
Great insight! Great video as always!
Thanks for these series! Open tracing and Open Census were good project and with Open Telemetry it also comes with auto-instrumetation. For Java there are Java agents based that propagates the headers, traces, spams, etc.. to upstream services.
Excellent and concise overview. Thank you! Teams must also be mindful of how this data is used and queried. From what you said in this video, I think you'll get into that in more detail in the next video.
It's a very big subject so it'll take more than one more video. The next one related to OTel is coming next Monday.
Looking forward to the tracing video!!
Waiting for next video.... very excited :)
It's coming next Monday :)
Yeah awesome video! I need now to build in the tracing library!
A video about Jaeger and Tempo would be nice, Viktor!
One of those two requests is coming next Monday :)
Jaeger was just published (ruclips.net/video/FK0uh-7nDSg/видео.html).
as you asked, in term of Otel Receiver, you might consider Grafana Tempo as Receiver and Visualization, as you have all three types of Otel in one Visualization, Traces, Logs, metrics, and cold even correlate them (implemented right)
Tempo is amazing
@@DevOpsToolkit BTW, do you know Crossplane (runtime) will get OTEL Tracing one day, I think it's a great fit for such a tool interacting with so. many components (Compositions, Providers) down and upstream...
@holgerwinkelmann6219 it's on the roadmap.
Just found your videos, amazing thanks
Thank you for explaining it.
I'm really curious to see how the instrumentation work with jaeger tracing components. Looking forward to it!
Already scheduled for next week :)
@@DevOpsToolkit It would be also interesting to see comparison with other tracing tools like Zipkin
It would. I’ll do my best to do it soon.
Done!
ruclips.net/video/FK0uh-7nDSg/видео.html
Does using the otel-collector makes sense if you already instrumented your app with Prometheus, Loki and Tempo ?
Prometheus will fetch metrics from places where metrics exist (e.g. kubernetes API). Your app needs to be instrumented to provide those metrics so you do need to instrument it besides having those tools if you want app-specific metrics to be in Prometheus.
Excellent content!
What do you use for centralized logs?
I prefer Grafana Loki.
Video with Jaeger would be nice
Often, I wait to see whether people are interested in a topic I propose and, if they are, I start working on it some time in the future. That often results in it coming weeks or months later. Jaeger is different. Many already asked for it and I'm already on it. It will be released next week.
Done!
ruclips.net/video/FK0uh-7nDSg/видео.html
You mentioned several times that we should use OpenTelemetry to be able to switch one observability solution to another. I remember a time, when people had the same argument for ORMs, which allegedly allow to switch database, but give you least common API. So you kinda exchange part of features to possibility of changing database/observability solution.
Maybe in some cases a Datadog agent is not that bad, especially if we consider that observability consist of several parts (agents, collectors, storage backend, UI and so on, and one vendor could integrate these parts better, than common solution).
Many other attempts to standardize something failed because of the lack of support. Having a standard that is not implemented by many is not very useful. OpenTelemetry, however, is one of the few "standards" that managed to rally everyone. Almost every vendor or a project in the observability space supports it or is working on the support (including Datadog). Something similar happened to Kubernetes API a bit earlier (almost everyone gathered behind it).
So, OpenTelemetry is different than others due to the size of its adoption and contributions coming from vendors and projects.
As a counter example, we have OAM (Open Application Model) for which I had high hopes but which failed to convince a significant percentage of the industry to adopt it.
One more thing... I don't care much about agents. They can be OpenTelemetry or anything else since there is almost no work involved in switching from one to the other. What I do care is what we use to instrument our code since that represents invesment (mainly in time).
Datadog supports otel pretty well but it works much better with the agent. Same with New Relic, Instana, splunk and most of the hosted services out there. It's how the softly push you in to vendor lock in to make it less appealing to switch away.
@@davehoffman515 That follows typical stages of adoption of standards.
1. We will not support this. Our stuff is better and we have the competitive edge.
2. We have to support this because others are supporting it and prospects are asking for it, but we will not make it as good as our own stuff.
3. We have to support it well because our customers do not want to use our own API. They are leaving because they do not want to be locked in.
Most vendors are in the phase 2 right now, and will be moving towards the phase 3 soon.
P.S. Most attempts to standardize something never pass the phase 1 and eventually die. OpenTelemetry managed to get to the phase 2 thus (probably) securing its future.
Is the logging part intended to replace promtail + loki or does it do something differently?
OpenTelemetry does not replace any specific tool or, to be more precise, does not have to replace anything. It's mostly focused on standardizing the API that other tools might use and instrumenting your code with that API.
You should do a video on OpenFeature!
Adding it to my to-do list... 🙂
Excellent fwd to the twitter
Any thoughts on Grafana Tempo vs Jaeger? Does it make sense to stick to Grafana tools if you already use them for most other things?
Grafana tools are awesome and they are my default recommendation. Tempo works with OpenTelemetry so no problems on that side.
Title: MUST!!!!
Actual video: maybe should 🥺👉👈
Hi, have you seen any way how to document Otel metrics based on some kind of Schema, basically like OpenAPI or MIB for Cloud Native Metrics?
I haven’t seen something like that :(
Hi, Can we get the javascript files and method name details in tracing in a non node.js app?
I'm not sure I understood the question. Are you referring to traces in other languages or tracing with JavaScript without nodejs?
@@DevOpsToolkit When I access the webapp (not node.js) which is a java based webapp, can I get the js files and the js functions that gets invoked from the browser side?
@cristineees not that I know. I believe it is server side only. I don't work much with frontend so I might be wrong though.
Jaegermaister!!!
I must NOT ask my app vendors to do additional coding (plus maintaining it) that will cost for "emit" when that application consumed unless consumer is really needed that if and when. Or stick with git-driven config change enforcement for enabling those information elements that will steer app to ship that that to a remote server via pull or push rather, some stuff we inherit from legacy not necessarily bad but optimized over ages of operational experience and cost for it.
log still not ready? have they given up
The spec for logs has been done a while ago (but might change over time). As for implementations, it depends on the language you're using. Some are stable and others are in beta. Both are stable and can be used. There are a few that are in development and you might want to avoid those.
And here I thought Robusta is a coffee species.
Robusta is two dudes spamming Linkedin constantly with their vision about "Limits make no sense, dont put limits in your deployments"... After short discussion with "CEO" stopped wasting my time with them.
what the pricing ?
OTEL Is open source and owned by cncf so there's nothing to pay.
@@DevOpsToolkit ok cool, so means I install only OTEL and Grafana is enough for me,
no need for Promothous and loki ??
can you help me understand this more ?
OTEL Is, first and foremost a protocol that most vendors (both open source and commercial) are adopting. On top of that there are OTEL agents. They are optional and are used to aggregate, filter, and transport data. You often do not need those.
There are at least three types of data. There are metrics, traces, and logs. You can collect and store metrics with Prometheus, logs with Loki, and traced with Jaeger. There are many other options available, some are open source while others are commercial.
Grafana does not store and data. It is used to visualize data from other places (like those i mentioned).
thanks
Otel collector + tempo pls 😆
Tempo is already on my to-do list :)
OT logging is still to this day inexistent or experimental.
Yes. Logging is behind other types of telemetry.
Interrupting the content with your ads makes the content confusing and makes me dislike the ad product
I understand that. However, they are means for me to cover the production cost of the videos and I'm not sure which alternative i could employ.
Thomas Amy Thomas Michael Moore Mark
I have conducted research on enabling OpenTelemetry for multi-services written in Go, and I have chosen to utilize Signoz as a visualization tool that combines traces, metrics, and logs. Signoz is also an open-source project, and I highly recommend that you check it out.
Adding it to my TODO list... :)
But there is a big problem with OTel itself. It tries to do tooo much and is unjustifiably complex and bloated. Classical stupidity of OTel designers called intellectual masturbation: observability, metrics, logs, traces, spans, layers, collectors, context propagators, receiver, connector, backend, etc.....other $4it.
I agree. It is complex and bloated, mostly as a result of too many companies contributing and adding "stuff" that matters to them. As far as I know, they already started working on reducing the spec.
In spite of that, it is a good thing that we do not have to instrument code using vendor-specific libraries/specs and being forever locked.
@@DevOpsToolkit Totally agree with you. I am not fan of AWS, GC.etc, however this is what happens in the end: > Here is the kicker: Despite all the bloat and inefficiency, OpenTelemetry still feels incomplete. This is because large enterprises are more interested in creating an open-source product that is ‘good enough’ to get us in the door but not so good that it satisfies all our needs. If you want the full observability package - optimized performance, advanced analytics, smooth scaling - you would be nudged toward their proprietary tools and services. It is a classic bait-and-switch.
In other words, OpenTelemetry is functional enough for basic observability, but when things get complex, you are often pushed into premium enterprise add-ons. It is not a coincidence - this is part of the corporate strategy. They contribute just enough to the open-source project to make it widely adopted but hold back the best features for their paid offerings. (excerpt)