Python Logging: How to Write Logs Like a Pro!
HTML-код
- Опубликовано: 8 июн 2024
- Logging can make all the difference in commercial software products, and it's essential to have a proper understanding of how to use it effectively. In this video, I dive into the details of Python’s built-in logging package, along with tips on how to make your logs easier to manage.
Git repository: github.com/ArjanCodes/2023-lo...
✍🏻 Take a quiz on this topic: www.learntail.com/quiz/yzrdwa
🚀 Next-Level Python Skillshare Class: skl.sh/3ZQkUEN
👷 Join the FREE Code Diagnosis Workshop to help you review code more effectively using my 3-Factor Diagnosis Framework: www.arjancodes.com/diagnosis
💻 ArjanCodes Blog: www.arjancodes.com/blog
🎓 Courses:
The Software Designer Mindset: www.arjancodes.com/mindset
The Software Designer Mindset Team Packages: www.arjancodes.com/sas
The Software Architect Mindset: Pre-register now! www.arjancodes.com/architect
Next Level Python: Become a Python Expert: www.arjancodes.com/next-level...
The 30-Day Design Challenge: www.arjancodes.com/30ddc
🛒 GEAR & RECOMMENDED BOOKS: kit.co/arjancodes.
👍 If you enjoyed this content, give this video a like. If you want to watch more of my upcoming videos, consider subscribing to my channel!
💬 Discord: discord.arjan.codes
🐦Twitter: / arjancodes
🌍LinkedIn: / arjancodes
🕵Facebook: / arjancodes
📱Instagram: / arjancodes
👀 Code reviewers:
- Yoriz
- Ryan Laursen
- James Dooley
- Dale Hagglund
🎥 Video edited by Mark Bacskai: / bacskaimark
🔖 Chapters:
0:00 Intro
1:29 Logging basics
3:12 Configuring logging
5:07 Integration with logging services
8:48 Big mistake with logging
10:42 Outro
#arjancodes #softwaredesign #python
DISCLAIMER - The links in this description might be affiliate links. If you purchase a product or service through one of those links, I may receive a small commission. There is no additional charge to you. Thanks for supporting my channel so I can continue to provide you with free content each week!
👷 Join the FREE Code Diagnosis Workshop to help you review code more effectively using my 3-Factor Diagnosis Framework: www.arjancodes.com/diagnosis
I would have loved to see your approach to logging from multiple processes in the same files. For example logging from a FastAPI application that uses uvicorn or gunicorn with multiple workers. Perhaps in a future video. Looking forward to that :)
I really love watching your videos. They're full of useful information.
+50
Or from multiple files/modules. I've written a function that sets up logging and call that one in every file, but have no idea if that's the way to go.
@@thomasjohanns7661 You typically only run the logger in your main function. Then in the config you can set up how it should behave for different python modules (including any imported modules. say if you want to silence, or actually increase, logging for anything in particular).
@@_DRMR_ While this is a common practice, it does not always get implemented this way. For example, if I use a custom library that might have its own logging module. Also, multiple processes could imply two or more handlers on the same logger.
We use initialized loggers. Do not forget to check official python build in: QueueHandlera and QueueListener for separation of handler (OUT end of queue) from logger(INPUT side pf queue)
You're videos are absolutely brilliant but what I really love is the code being available. It saves me a lot of time when I come back to one of your videos I've already watched a couple times.
Fantastic as always, would love a deep dive on the structlog library for your next video! really takes logging to the next level
Great suggestion!
Logging throughout a module is way more complicated than you’ve shown though. Can you show how you set that up?
Was about to type this
Yes please!
100 % agreed
i just put "logger = logging.getLogger()" at the top of each file and call logger as usual. Works great
@@dirtdart81 but then you’re always grabbing the root logger, which is bad practice as far as I can tell
I’m a terrible coder, but only do it for personal projects, and I never thought about using logging to replace the print statements I use when developing, this would save me having to find all the random print statements all over my code 😊
Great video as always!
Interesting to see use of rotating logs as well as multiple package log implementation.
I know I ran into issues with those when I first started writing in python.
Great content as always Arjan! I think it would be also a good idea to show ways of not overly littering your code with logger calls (ie. using decorators, as in one of your recent videos, or perhaps using other methods) and perhaps demonstrating a real-world example of crucial logging within an app. All the best!
Yessss I’ve been waiting for this! Thanks Arjan!
You are doing great videos, but IMHO this was a bit too basic, would love to see an even more deep dive into this module. Like setting it up for a bigger project, best practices, etc. Anyway, thanks a lot and have a nice weekend! 🙂
I think its good he started off basic so those of us who dont get a chance to use this stuff can have a basic setup. I hope he makes an advanced one too!!!
I am also looking for setting it up for bigger practices and best practices. i.e. I also would like to know how you can pipeline the logs to may be ELK or anything where it can be consumable
You gotta start somewhere. I loved it!
+ 100 on this idea. Most logging ends up being a overflowed garbage can of rotten log salad and parsing it to find confirmation of correct application, business logic performance, and the mixed bag of errors is just difficult to retrofit after integrated into production. Once it rots in production awhile, logs are definitely an 8th "olfactory offense".
@@OhsoLosoo Title should NOT be misleading. This is amateur/beginner logging, pretty far from Pro.
As it is named deep-dive I was a bit hoping the focus of the logging levels and what are the daily praxis where to log, and how to handle the multiple levels in the code. This is something after years of developing software I still struggle with. Sometimes it feels like every second line is a log, and sometimes there are 3 log lines in the full project (hyperbolic). It never found a good regular way / a principle to follow ... :)
Whatever! :) Keep going, I learned a lot from you the last year I have now in my daily praxis. It helped me a lot :)
I've found talks by Kevlin Henney incredibly insightful and interesting. Here is a part from a talk (with a clickbaity title) where he addresses logging: ruclips.net/video/FyCYva9DhsI/видео.html. No concrete advice, but hope it's helpful nonetheless.
My personal idea of logging is pretty minimalistic: YAGNI. I make extensive use of logging in the debugging phase, as regular debuggers are often of little help with kinds of software I'm dealing with. But after I have figured out some part of the code and it is covered with tests, logging goes out of the window, unless I have very specific scenarios in mind where I would need it for a post-mortem. For example, if I need to log user actions, I have a set of requirements for that. Otherwise, I do not need to log user actions.
Would love to see more advanced logging concepts especially unified logging in a big library
Thanks Arjan! Could you do a follow up on structured logging?
waiting for this!!!! Thanks Arjen!!
I feel as though you just glossed over some of the basics and added the SysLog as a bonus. What about getLogger(__name__) combined with propagation? What about different handlers at different levels? What about custom formatters? What about using decorators to do your logging for you? If nothing else, there are some ideas for a future show :)
I love your videos about theory and architecture....once you have been coding for a while and you are no longer a begginer this kind knowledge feels really escential.
This was a lot more basic than I was expecting. Python's logging has a lot of pitfalls that I would have liked addressed.
I also find logging super important! Thanks a lot for this video with lots of good advice :)
Great job! Thank you so much
*Thanks Arjan!* That is super useful 👍
Glad it was helpful!
As always, excellent video. Making a simple topic so robust, I want to build some apps just to log now.
Arjan, thanks for uploading this :) the date/time formatting of logs was very helpful. I've been using INFO:ROOT this whole time like a n00b.
Nice. I would have loved if you had shown what's best practice to actually log and at which places in the code loog statements are supposed to be.
That's very difficult to explain and is up to the project. It's almost artistic. The best way I can describe it is log anywhere you think will help you understand what the software did without tracing the code. You want to be able to diagnose what a problem is almost from the logs alone. That expands to log any information you need to diagnose. Like user names or values that were switched on or errors encountered
Thanks, Arjan! As always, amazing video!
Glad you liked it!
Again an excellent video ! I'm actually pointing all my new devs to some of your videos so that I'm sure they pick up good practices (even old ones like me). I didn't know papertrail, and I'm now starting using it. it's perfect, though I'm a bit worried about anybody being able to send logs to my account (no real auth)
Amazing content Arjan, keep it up. Cool intro to Paper Trail too :)
Been playing with log2d that makes logging a bit easier.
I am glad to have found this video. Thank you!
Glad it was helpful, William!
Yeahhhh thank you so much!! Really appreciate a logging tutorial!
You're very welcome!
best for someone who just started to learn logging.
long-awaited topic. thanks!
as always, great content Arjan.
Thank you so much!
I've used grafana + loki for very long time. Its very efficient with little to no resources. It can be setup in multiple ways and there is also a package for python called python-logging-loki. Loki has a really nice and powerful logql language with which you can make awesome grafana dashboards.
Great Job bro !! Keep it up.. Thanks for this ❤🙏🏻
Thank you for the kind words! Will do.
Super useful, thanks!
Glad it was helpful!
Always good videos
DUDE.. I was searching for Logging tips TODAY.... lol
Thank you for the video. I didn't know SysLogHandler and Papertrail before.
In our company, we are using Graylog instead of Papertrail. Graylog isn't only for Python apps but also for other apps creaded via Java etc. Graylog has a REST API Server. Thus, we are accessing logs in Graylog via requests library in Python. Then, we perform Fraud Detection based on logs data.
There are also some libraries like loguru and structlog for logging in Python.
I really appreciate your efforts in making this video and it was enjoyable to watch.
Informationally however it doesn't quite live up to the expectation set by the title...
I would agree with other commenters, it's nice to get a basic intro to logging, but I'd really like to know WHAT I should be logging.
In theory I could put a debug log for every line of code I write stating would should happen, but I imagine this is overkill. Or is it? I don't know where to draw the line, it would be good to get some examples of what kinds of things you log, and what level is appropriate for them.
But good video, and I'm looking forward to a part 2 😉
another important use is audit logs... we have some critical operations that the customers have to be able to see when they ran, how many times, with what parameters etc
Some logging services like Splunk provide a custom query language, and it's worth to learn it because it will save a lot of time. Also another tip is to avoid root logger and prefer to create a logger for each module, it will reduce the searching time as you can do more specific searches.
Thanks 🙏🙏🙏
you are best my friend.😍😍😍😍😍😍😍
I enjoyed the video. I am currently working on doing a presentation on logging. During that process, I spent a lot of time understanding handlers and formaters. I become most unclear when looking at the best way to implement logging in a package compared to how to implement it in an application. Do you have the link to the second video you mentioned on logging?
Thanks Arjan, good start on logging.
I recently struggled a lot to configure getLogger to get logging object from other module. Python documentation is not that clear on this. I even tried to use thirdparty pkg like logguru.
Try doing a video on configuring multi file/module logging and best practices
Great video logs simple and insight full. But this content is available like everywhere, can you make a video on logging with different modules in the project or when using a design patter in projects and we have tons of files and folders in our project.
Hi Arjan, thanks for this video! I was hoping to learn how I should set up logging for a package, so that people that might install my package via pip get access to the logs when executing the code if they wish so, but maybe not automatically. Do you think you could make a second video about more advanced nuances about logging?
Also I saw another package (requests) set a NullHandler in __init.py__ and I was wondering what the reason behind that is.
I like you content very much. I'd like to know when to use logging & when to use print statement. Currently, I tend to checking "logging" via print statement.
Thanks for the video. Looks like papertrail is now solarwinds, any idea how to do the port stuff with the new set up? Would be great if you have any suggestions
Hi there, from Portugal,
Any recommendation on using python for google cloud function using google cloud logging?
Obrigado(Thanks)
We have struggled with logging quite a bit in our Company and I guess the best solution if to use a central logging config object (dict or even better yaml file). There u set the Formatter, Handler and if you need to set custom attributes, you can do that pretty nice by using a filter.
Yesss. Log as JSON, ingest those logs into some central service (you could use Filebeat for ingesting the log files and use Elastic as search engine with Kibana for dashboarding and searching through your centralized logs).
You can also use `jq` (a terminal application) to sort, filter, format your files, etc. It's sooooo nice.
I use the structlog lib, which can be a pain to configure, but it's incredibly powerful and flexible.
@@NostraDavid2 Yep, we also use JSON format for the logs and a central instance where u can see all logs. This is luckily done automatically in our Kubernetes Cluster
I agree that this was too basic. Mentioning Paertrail was very helpful
Nowadays one of the main reasons for a logging server is separation of concerns in the light of information security. If you get attacked, you may want to have a log server separated from your operational data proccessing - with no possibility to remotely delete your logs. Forensics will be very thankful.
Some tools (which can be a bit overkill) like splunk and elastic stack can really allow you to search and filter log files with ease. But this however is coming from a digital forensics background.
Hi, I am using this functionallity, but module "requests" messes up my log, as it also throws its requests, and for example I see updates of virus signatures and configuration downloads from firefox, is there any way to prevent it?, I could upscale my messages as warning, as it seems it only happens in the INFO configuration, but there has to be another way. Thanks for your great videos!
With normal file logging you can search with grep, less and all the other standard tools, what could be easier than that?
Logging to splunk is useful too.
I expected you would mention log data has an interface. Changing your log statements can have serious impact for tooling that analyzes the log files for statistics, problems, and improvement opportunities.
Im working in how to implemet logs with fastapi and just I realised logs are printed in stderr by default wich will be a problem if a cicd pipeline relies on stderr in order to continue the right flow. I hope you will explain it here :(
Why did papertrail mark the log message as coming from "logger" when the logger you configured was named 'arjancodes'?
nice
Good introductory video about logging in general. However not a deep dive on Python's logging package, as announced at the outset. In fact there is surprisingly little about Python in this video.
A great video and explanation.. Do you have courses in Python from intermediate level?
Yes I do! You can see all of my courses here: arjancodes.com/courses/
how do I log to system.log on macos ventura? none of what i've found on SE or what has been provided by chatGPT works (no errors though)
I used nbdev in VSCode to write python modules. Where does the logging module write it's output when working with notebook cells?
# import reload
from importlib import reload
reload(logging)
logging.basicConfig(format='%(asctime)s %(levelname)s: %(message)s', level=logging.DEBUG, datefmt='%I:%M:%S')
Valeu!
Thank you so much for the support!
Logging service, you just blowed up my mind :) I had not idea it existed. As usual thanks for the great content.
Happy to help, Daniel!
I wonder how often do large companies use an in-house logging solution. I’d imagine that would prevent some of those data leaks and would be perfectly feasible to implement given their size.
IMHO the logging module seems really over-complicated and easy to get lost in (handlers FTW). But now that it’s in the std lib, nothing more Pythonic gets much traction. See at all the comments asking for complex configuration examples, across multiple files. Video is a good intro to its existence but there’s only so much that can be conveyed this way. And in blog posts as well, most of them also barely scratch the surface of Python logging.
👏🏾👏🏾👏🏾
those unequal length cords on your hoodie is driving me crazy
Thanks! I am trying to force people to use logging framework on a company internal project, it packed full of prints and I am so sick of getting screenshots of a command prompt.
I'm liking the glass of water 🤓
My question is if something goes wrong in production.. how can we back trace the problem from logs?
Here's a great example... You can have python logging framework output to a syslog server... It's AMAZING. Imagine a few hundred users using your stuff and you get all the usage info, who is crashing and so on and so forth.
I like the sweater, what brand is that?
👍👍
what is the name of the compiler?
What are the pro's to have an explicit main() function vs having the contents of main() under __name__ == "__main__"?
I tend to do this because it increases the consistency of the code I write, especially as this also translates well to concurrent code. In that case, you would simply change the main function to async, and use asyncio.run under the if statement.
what is the purpose of the "if __name__ == "__main__":" block
Really important! Why is getLogger etc from the logging package camel case? It hurts my pythonic eyes
It'd be cool if you showed logging config from different files, like ini and yaml. thanks!
/dev/null as a Service!
Real talk where is that sweater from?
Why at the first place you need a separate logging service.??
All your logs are in one place, so you can more quickly find problems, better log visualization features such as colored logging and various types of charts, better log search, better export and backup capabilities, better automation features. Just to name a few things off the top of my head.
This dive was into this module was really shallow
Me, an intellectual: print(x) scattered throughout my spaghetti code
Anybody here uses loguru?
This type of logging is insufficient. Where's the line number? Where is the thread ID if your app is multi-threaded? Where is the task ID if it's asynchronous? How will you know in which thread/task the error occurred.
Logging into a File with a Rotating FileHandler from multiple processes is not so nice tho xD
How do you configure your emacs this good? Can you make a video about this as well?
nothing compares to print("here")
Log to a database ..sqlite3 baby
remove all of this pain with just `pip install loguru` and `from loguru import logger`... yw
Logging is hella important. If you think logging and not well organized and built logging systems is not important, you're still very junior.
I get a bit depressed when I watch videos where I already knew everything that was explained. It's nice to watch it accurately and well explained but like following a football team you're hoping for a goal, that piece of knowledge you didn't have, but that never comes.
After a while you reach the end with this programming thing and you learn that it was the learning itself you loved, not the programming.
It is good，but i choose loguru😂
Big mistakes with logging: Do not over-log information, just what needs to be logged. You kind of said this with the security part, but still.
Auto translation of the video title to german totally butchered it.
Yes, we switched that feature off for more recent videos. It doesn’t work that well.
this was hardly a deep dive. this was VERY surface level