The Logging Everyone Should Be Using in .NET
HTML-код
- Опубликовано: 17 окт 2024
- The first 100 get 20% off our new OpenTelemetry course on Dometrain with code TELEMETRY20: bit.ly/3y8GTOv
Subscribe to my weekly newsletter: nickchapsas.com
Get the source code: mailchi.mp/dom...
Become a Patreon and get special perks: / nickchapsas
Hello, everybody. I'm Nick, and in this video, I will show you how you can get started with OpenTelemetry logging in .NET
Workshops: bit.ly/nickwor...
Don't forget to comment, like and subscribe :)
Social Media:
Follow me on GitHub: github.com/Elf...
Follow me on Twitter: / nickchapsas
Connect on LinkedIn: / nick-chapsas
Keep coding merch: keepcoding.shop
#csharp #dotnet
Lidl swag makes you a better developer
I have the lidl shorts hahaha
OTEL and/or Log Shipping can be overwhelming when you have used only traditional console loggers for many years, not to mention they all want you to ship to cloud. This is the simplest and clearest 'get started' walkthrough I've seen to date.VERY appreciated.
Great video. Howerver, albeit not the scope of this video. When configuring the endpoints for OTEL. How should we go about doing that in production? I mean, there is an API key, that we want to keep secret, so perhaps fetching that from i.e HCP vault. Then I suppose we can use an environment.variable for the endpoint URI.
It's a very interesting topic. I think it would be very useful to a video about how to integrate like open telemetry + some scoped logging context/metadata (e.g. if we need to log a tenant ID in a multi-tenant app) + for example, Azure app insights as a logging storage, but in a case of using ILogger from DI. I don't think that in a real-world app, devs will create a new logger manually each time.
serilog implements the ILogger, and you can use the serilog sink package
Nick's drip goes so hard
Just a lidl drip
This video is invaluable for my team and myself and arrives with perfect timing for our necessities. Thank you, thank you, thank you.
Thank you so much for this. I've been investigating integrating OTLP into some new development currently on my plate and this answers a LOT of the questions that I had been having. Appreciate all you do!
Hey Nick thx for a great video. How do you feel about Application Insights and Log Analytics?
Had the same question
@@atziazas me too. We pretty much use ILogger to app insights and all our stuff is running in apps service is Azure. and I'm wondering now if we should be changing that?
Seq is not free, you need to pay for a license to deploy to a server.
Serilog (Loki Sink) + Loki + Grafana is the easiest and fastest way to get a free, simple, queryable, structured logging solution up and running. No magic instrumentation, no security headaches with certificates (like with ELK, just let the reverse proxy handle it), no pantheon of collectors and exporters and services. It doesn't have metrics and tracing (though they can be added) but sometimes you just want something simple to start off with and logging is the most crucial of the 3 observability components when starting out.
Do these anything to do with OpenTelemetry, or is this a completely separate stack?
@@g3ff01 They do work with OTel. Loki can infact also ingest OTel logs, but also has its own sink for Serilog using its own API. Grafana is what is used to build the Dashboards and bring it all together. They also have services to ingest other data like traces into Tempo, or metrics into Prometheus. Seq is more or less and all-in-one solution and I like it for lokal testing on my machine because its smaller. Grafana + Loki + Tempo etc. is a stack of multiple services that you can piece together as you need them. Thats what we run in Prod.
Been using SEQ with Serilog for years but this highlighted how easy it is to include open telemetry. Thanks Big Fella. Another great video.
Will Dometrain courses be available from the Lidl middle aisle any time soon Nick ?
I'm trying to get them to sponsor but they just want me to advertise their "Greek Week" instead
First time I’ve ever been able to watch one of Nick’s videos and think ‘we already do that!’ Result!
3:10 Do you have a video about the structered Logging Naming Convention for attributes? - This is a new "problem" when doing the structured logging as the name you provide in the template will be name in the logging provider like ElasticSearch. So lets say you were to log the execption ex and you write the structered log like ("There was an error handling the request. The error was {Ex}", ex.Message); - The naming in template could easily be {Exception} or {ExceptionMessage} or just {Message}. If you're the only developer on a project that's not a problem but if you're a team you need to have a standard for this attributes but I haven't found a "standard" for this and might have missed if you're ever mentioned this in any of your videos or talks.
I'd like to have some tips re that too. Even in my own project, I find myself I am not consistent enough - because I don't know what's good and simple vs too short vs too bloated. I catch myself on that often.
If you use mediatr then you can catch exceptions or log the request in one place and define your template there.
Alternatively if you use source generated high performance logging you can define templates in a resource file for example
@@Wfmike I don't think that's the concern. We know that. The concern is good old variable naming, but with logging in mind in this case.
@@TehGM I'm not sure if you are going to find a specific naming convention for that. Every team needs to agree to a format that suits them.(And build guard rails to adhere to the convention)
Just post it on whatever chat app your team uses and have your team vote for options. There’s not always going to be a convention for everything.
Hi Nick, Thanks for this amazing and knowledgeable video. I have a question though.
Let’s say I want to log to a file or maybe create my own log viewer within the application which I can host with the application and push my logs there using open telemetry. Is that doable?
nice, serilog and seq are the best combination we're using since years!
Hey nick -
I was wandering, as a Unity developer, a former validation developer at intel, what courses whould i take to make my c# knowledge better?
i mean, i do have 3 years of experience in a varaity of c# useages (not .Net in particular), and i am a computer science major, so taking the "SOLID" course or those kind seem obsolete.
I am in "big tech" company, OTel is still struggling internally. There are as many ways to emit telemetry as there are organizations in my company. Some services emit almost fully denormalized data - one table with something like 500 columns. My point is - don't do what big tech does, instead do what a very serious tech startup would do
So what does a very serious startup do?
.
Serious startups spend millions of investor dollars buying storage to store 10GB of log per hour in the cloud provider 😅
Amen.. I've seen some "Big Tech" companies that don't have any structured logging in place. Customer queries have to be resolved by engineers remoting into various servers and inspecting local log files. I 💩you not.
Great content as usual. One question: how can we use open telemetry in production, where could the logs be exported to. Any ideas or suggestions. Thanks
You can use the ELK stack or even Seq itself since it is OSS, as well as any other paid offering. I’ve used DataDog in the past
LIDL 💛💙
Is there any video on how to build a custom solution to send and view those logs?
Nick do you have an example with Azure App Insights?
I've always assumed it was pronounced "seek" as in "sequence".
Ah that's smart actually
Same here
You are correct, I spoke to Nicholas Blumhardt (the guy behind Seq) at NDC Oslo and he did pronounce it "seek" :)
I've been pronouncing it like Nick for many years already. But 'seek' makes so much sense...
Hi there, how would you be able to NUnit Test specfically NSubstitute to test OpenTelemetry logging, thanks
Id already added Open Telemetry but it can get complicated when configuring a boot logger, Serilog, Open Telemetry, and Application insights (or Grafana etc) - then configuring Tracing and Metrics, adding custom Meters. I managed it but I'm not convinced it's all correct. Maybe a new course idea for Dometrain on Observality...?
With Serilog having so many different ways to do the same thing, it's becoming confusing...
I wonder if the exporter runs asynchronously as a background service at the lowest priority, unless a critical exception must not be lost and should be logged immediately before the app crashes.
How would I log transactions or "sagas" from multiple services, that might collaborate to fulfill a task. So that I have logs for all steps and can combine logs from multiple services for exactly one business process?
the sweater looks total dripp
What if the external logger service goes down? Is there a possibility to also save the logs to the file in a host or something? In case of an event where the logger service is down and we need to investigate something how can we achieve that?
You can log to a file and have an agent that reads that file and sends the content to your log server in otel format, for example.
Or you write your logs to different sources (otel cloud, file,…) at the same time.
Apologies, if it has been asked before, what font are you using in your video?
Which is better?
- use serilog sink to push logs
- use fluentbit to collect logs from the app
Great video, I was not aware of otlp
Great vid btw. This is what I wish I had a year ago when I first decided to look into OTEL lol.
Hey Nick, thank you for this great video. Do you perhaps know of a way how OpenTelemetry can log to files? (This is a requirement from our client we do work for)
I'm not quite sure - how is it different to Application Insights?
AWESOME!!!!! Thanks for sharing!
Of course I know seqs, I do the seqs every day
It feels like you are about starting new trend for .NET developers clothes... with your nice Lidl shirt 😀
... ok but how to add OTEL traceability for microservices subscribed to a service bus?
if the default logger gets the job done, i don't see value of using the additional package , serilog in this case.
Do you know if the .NET built-in logging provider can deconstruct the objects and log its composing properties using the OpenTelemetry exporter?
Yes it can
Question: what is the name of the app you use to test the API ?
It is called Kong Insomnia.
I just install the Seq sink. I've also coded a nuget package that enhances the log properties with the user information, payloads, etc.
Can I get a link?
I always assumed it was pronounced SEEK (as is seek and you shall find). :)
Great video. One question though. What is the benefit of adding Serilog to the picture? Never used it, so maybe that's a dumb question with obvious answer ;-)
Serilog, in general, brings many benefits, but in this specific context you can get a more useful formatting as well as the ability to serialize objects as part of the structured logged message. Please, keep in mind this is a very superficial answer. Serilog’s true benefits become more evident as more intricate scenarios come into play.
@@josefromspace ah. Thanks. Just always assumed that Serilog is basically predecessor of Microsoft.Extensions.Abstraction.Logging with (mostly) similar functionalities and just small deviations in interfaces or some obscure functionality. Will take a closer look than.
@@woocaschnowakabsolutely, Serilog’s claim of fame was that by using structured logging, it allocated less memory on the heap than .Net’s counterpart at the moment. As time passed both have grown in functionality, definitely worth looking into it.
Nice PJs Nick! :D
Lidl is the new Gucci 💖
Logging από τα LIDL? :D
Nice shirt ❣
So you're saying that I shouldn't be logging the console to text files and pushing them to a database. 😅
why string interpolation in logs not accepted as standard?
Structured logging makes it easier to analyze logs. For example, if you wanted to see the 95th percentile response time grouped by day.
"Getting weather for {CityName}" is the same string every time, so you can easily search for this in your logs and go from there. If you used string interpolation, you'd have to search for the common substring which would be "Getting weather for " and that's a bit weird
I'm adding logging into a bunch of apps at the moment, and to be honest this level of logging just isn't needed. Normal structured logging is enough for a lot of smaller applications. It comes down to only logging what you actually need and will look at, all the rest is useless fluff.
Can we use the log instead of events? Can I subscribe on the log and then act on them. For let's say save data persistent?
You shouldn't use logs for anything they are not meant for.
No.
The beaty of using Seq is you can also visualise in it your OTEL traces and spans. Without having to use a seperate software, like Jaeger
That was exactly the thing I expected to see in the video, but didn't see. Thanks for confirming Seq can do that!
Hello guys how can i have access to Nick's discord server? (I have the 5 dollar subscription)
Have you linked your RUclips channel to Discord?
@@nickchapsas yes nick, still dont have access! (update: i subscribed to patreon and got access)
Nick looking different, cool!
I hate that you cannot export Request collections in insomnia
Is that a Lidl tshirt? What the heck?
a friend of mine has a full swaggy lidl outfit guys!!
Why not string interpolated logging?
there's a few reasons, but structured logging allows you to preserve the parameters in such a way that you could query them later like he's doing in the video. Otherwise, you'd have to use complicated patterns (regex) to extract those parameters again after being joined into one string. Also, each unique log line would allocate a new string further increase in garbage collection pressure and slowing down the app.
god damn lidl !
Is it only me who whenever hears the "Telemetry" word thinks *VERY INTRUDING PRIVACY INVASION* ?
Is that a LIDL tshirt? 😅🤣😂
Big tech uses Java 🤔
SEQ is pronounced SEEK.
First comment
Why English thow?
After watching this video I feel shame because of logging we are doing in our company. 😢
Long winded
Congrats, now every log entry filled with tons of useless bullshit.
It's only useless until you need it. Then you'll wish you had it
You never will need this default *additional* fields, like corellation id / other stuff, unless you really have linked services with such tracing. But you example free from this. It never useful in given example. This did not meant what them useless, but by default they have little no sense, but you pay for this. I know enough about tracing and how get 4gib trace log in few seconds, so, no, this is reverse of what software should do by default.
Also logs is not tracing - is even splitted in open telemetry, but they mix all things down poorly. Really poorly. Unfortunately .net have no better alternatve.
@@NotACat20 Why do you mean they mix them poorly? There's logging, tracing and metrics. They are all separate things. You can use OpenTelemetry for everything or you can use it for something specific like Nick did in this video.
There's also an alternative approach: just write logs to console output and capture it with something like Filebeat and send them to Elasticsearch. Elasticsearch is also more widely used than Seq.