I would have loved to see your approach to logging from multiple processes in the same files. For example logging from a FastAPI application that uses uvicorn or gunicorn with multiple workers. Perhaps in a future video. Looking forward to that :) I really love watching your videos. They're full of useful information.
Or from multiple files/modules. I've written a function that sets up logging and call that one in every file, but have no idea if that's the way to go.
@@thomasjohanns7661 You typically only run the logger in your main function. Then in the config you can set up how it should behave for different python modules (including any imported modules. say if you want to silence, or actually increase, logging for anything in particular).
@@_DRMR_ While this is a common practice, it does not always get implemented this way. For example, if I use a custom library that might have its own logging module. Also, multiple processes could imply two or more handlers on the same logger.
We use initialized loggers. Do not forget to check official python build in: QueueHandlera and QueueListener for separation of handler (OUT end of queue) from logger(INPUT side pf queue)
I’m a terrible coder, but only do it for personal projects, and I never thought about using logging to replace the print statements I use when developing, this would save me having to find all the random print statements all over my code 😊
You're videos are absolutely brilliant but what I really love is the code being available. It saves me a lot of time when I come back to one of your videos I've already watched a couple times.
As it is named deep-dive I was a bit hoping the focus of the logging levels and what are the daily praxis where to log, and how to handle the multiple levels in the code. This is something after years of developing software I still struggle with. Sometimes it feels like every second line is a log, and sometimes there are 3 log lines in the full project (hyperbolic). It never found a good regular way / a principle to follow ... :) Whatever! :) Keep going, I learned a lot from you the last year I have now in my daily praxis. It helped me a lot :)
I've found talks by Kevlin Henney incredibly insightful and interesting. Here is a part from a talk (with a clickbaity title) where he addresses logging: ruclips.net/video/FyCYva9DhsI/видео.html. No concrete advice, but hope it's helpful nonetheless. My personal idea of logging is pretty minimalistic: YAGNI. I make extensive use of logging in the debugging phase, as regular debuggers are often of little help with kinds of software I'm dealing with. But after I have figured out some part of the code and it is covered with tests, logging goes out of the window, unless I have very specific scenarios in mind where I would need it for a post-mortem. For example, if I need to log user actions, I have a set of requirements for that. Otherwise, I do not need to log user actions.
You are doing great videos, but IMHO this was a bit too basic, would love to see an even more deep dive into this module. Like setting it up for a bigger project, best practices, etc. Anyway, thanks a lot and have a nice weekend! 🙂
I think its good he started off basic so those of us who dont get a chance to use this stuff can have a basic setup. I hope he makes an advanced one too!!!
I am also looking for setting it up for bigger practices and best practices. i.e. I also would like to know how you can pipeline the logs to may be ELK or anything where it can be consumable
+ 100 on this idea. Most logging ends up being a overflowed garbage can of rotten log salad and parsing it to find confirmation of correct application, business logic performance, and the mixed bag of errors is just difficult to retrofit after integrated into production. Once it rots in production awhile, logs are definitely an 8th "olfactory offense".
I used to work in tech risk for a FI. Regulators require all logs to be maintained and those logs once collected gets streamed to some security vendor to do help look for security issues at runtime. They ingest records at around 1K per second. That exposure changed my perception on logging
Great content as always Arjan! I think it would be also a good idea to show ways of not overly littering your code with logger calls (ie. using decorators, as in one of your recent videos, or perhaps using other methods) and perhaps demonstrating a real-world example of crucial logging within an app. All the best!
That's very difficult to explain and is up to the project. It's almost artistic. The best way I can describe it is log anywhere you think will help you understand what the software did without tracing the code. You want to be able to diagnose what a problem is almost from the logs alone. That expands to log any information you need to diagnose. Like user names or values that were switched on or errors encountered
Great video as always! Interesting to see use of rotating logs as well as multiple package log implementation. I know I ran into issues with those when I first started writing in python.
I feel as though you just glossed over some of the basics and added the SysLog as a bonus. What about getLogger(__name__) combined with propagation? What about different handlers at different levels? What about custom formatters? What about using decorators to do your logging for you? If nothing else, there are some ideas for a future show :)
I've used grafana + loki for very long time. Its very efficient with little to no resources. It can be setup in multiple ways and there is also a package for python called python-logging-loki. Loki has a really nice and powerful logql language with which you can make awesome grafana dashboards.
I love your videos about theory and architecture....once you have been coding for a while and you are no longer a begginer this kind knowledge feels really escential.
another important use is audit logs... we have some critical operations that the customers have to be able to see when they ran, how many times, with what parameters etc
I would agree with other commenters, it's nice to get a basic intro to logging, but I'd really like to know WHAT I should be logging. In theory I could put a debug log for every line of code I write stating would should happen, but I imagine this is overkill. Or is it? I don't know where to draw the line, it would be good to get some examples of what kinds of things you log, and what level is appropriate for them. But good video, and I'm looking forward to a part 2 😉
Again an excellent video ! I'm actually pointing all my new devs to some of your videos so that I'm sure they pick up good practices (even old ones like me). I didn't know papertrail, and I'm now starting using it. it's perfect, though I'm a bit worried about anybody being able to send logs to my account (no real auth)
I see that a lot of ppl here are asking for a more complex level of logging explanation, but let me tell you that as being totally new to logging, this video was perfect! straight to the point and very very useful for we ppl that are looking for an starting point. Thanks a lot.
Thank you for the video. I didn't know SysLogHandler and Papertrail before. In our company, we are using Graylog instead of Papertrail. Graylog isn't only for Python apps but also for other apps creaded via Java etc. Graylog has a REST API Server. Thus, we are accessing logs in Graylog via requests library in Python. Then, we perform Fraud Detection based on logs data. There are also some libraries like loguru and structlog for logging in Python.
Some logging services like Splunk provide a custom query language, and it's worth to learn it because it will save a lot of time. Also another tip is to avoid root logger and prefer to create a logger for each module, it will reduce the searching time as you can do more specific searches.
We have struggled with logging quite a bit in our Company and I guess the best solution if to use a central logging config object (dict or even better yaml file). There u set the Formatter, Handler and if you need to set custom attributes, you can do that pretty nice by using a filter.
Yesss. Log as JSON, ingest those logs into some central service (you could use Filebeat for ingesting the log files and use Elastic as search engine with Kibana for dashboarding and searching through your centralized logs). You can also use `jq` (a terminal application) to sort, filter, format your files, etc. It's sooooo nice.
@@NostraDavid2 Yep, we also use JSON format for the logs and a central instance where u can see all logs. This is luckily done automatically in our Kubernetes Cluster
I really appreciate your efforts in making this video and it was enjoyable to watch. Informationally however it doesn't quite live up to the expectation set by the title...
I like you content very much. I'd like to know when to use logging & when to use print statement. Currently, I tend to checking "logging" via print statement.
Thanks Arjan, good start on logging. I recently struggled a lot to configure getLogger to get logging object from other module. Python documentation is not that clear on this. I even tried to use thirdparty pkg like logguru. Try doing a video on configuring multi file/module logging and best practices
I expected you would mention log data has an interface. Changing your log statements can have serious impact for tooling that analyzes the log files for statistics, problems, and improvement opportunities.
Great video logs simple and insight full. But this content is available like everywhere, can you make a video on logging with different modules in the project or when using a design patter in projects and we have tons of files and folders in our project.
I enjoyed the video. I am currently working on doing a presentation on logging. During that process, I spent a lot of time understanding handlers and formaters. I become most unclear when looking at the best way to implement logging in a package compared to how to implement it in an application. Do you have the link to the second video you mentioned on logging?
Good introductory video about logging in general. However not a deep dive on Python's logging package, as announced at the outset. In fact there is surprisingly little about Python in this video.
Hi, I am using this functionallity, but module "requests" messes up my log, as it also throws its requests, and for example I see updates of virus signatures and configuration downloads from firefox, is there any way to prevent it?, I could upscale my messages as warning, as it seems it only happens in the INFO configuration, but there has to be another way. Thanks for your great videos!
Nowadays one of the main reasons for a logging server is separation of concerns in the light of information security. If you get attacked, you may want to have a log server separated from your operational data proccessing - with no possibility to remotely delete your logs. Forensics will be very thankful.
Some tools (which can be a bit overkill) like splunk and elastic stack can really allow you to search and filter log files with ease. But this however is coming from a digital forensics background.
Hi Arjan, thanks for this video! I was hoping to learn how I should set up logging for a package, so that people that might install my package via pip get access to the logs when executing the code if they wish so, but maybe not automatically. Do you think you could make a second video about more advanced nuances about logging? Also I saw another package (requests) set a NullHandler in __init.py__ and I was wondering what the reason behind that is.
Im working in how to implemet logs with fastapi and just I realised logs are printed in stderr by default wich will be a problem if a cicd pipeline relies on stderr in order to continue the right flow. I hope you will explain it here :(
I wonder how often do large companies use an in-house logging solution. I’d imagine that would prevent some of those data leaks and would be perfectly feasible to implement given their size.
All your logs are in one place, so you can more quickly find problems, better log visualization features such as colored logging and various types of charts, better log search, better export and backup capabilities, better automation features. Just to name a few things off the top of my head.
I tend to do this because it increases the consistency of the code I write, especially as this also translates well to concurrent code. In that case, you would simply change the main function to async, and use asyncio.run under the if statement.
IMHO the logging module seems really over-complicated and easy to get lost in (handlers FTW). But now that it’s in the std lib, nothing more Pythonic gets much traction. See at all the comments asking for complex configuration examples, across multiple files. Video is a good intro to its existence but there’s only so much that can be conveyed this way. And in blog posts as well, most of them also barely scratch the surface of Python logging.
Thanks! I am trying to force people to use logging framework on a company internal project, it packed full of prints and I am so sick of getting screenshots of a command prompt.
Here's a great example... You can have python logging framework output to a syslog server... It's AMAZING. Imagine a few hundred users using your stuff and you get all the usage info, who is crashing and so on and so forth.
This type of logging is insufficient. Where's the line number? Where is the thread ID if your app is multi-threaded? Where is the task ID if it's asynchronous? How will you know in which thread/task the error occurred.
👷 Join the FREE Code Diagnosis Workshop to help you review code more effectively using my 3-Factor Diagnosis Framework: www.arjancodes.com/diagnosis
I would have loved to see your approach to logging from multiple processes in the same files. For example logging from a FastAPI application that uses uvicorn or gunicorn with multiple workers. Perhaps in a future video. Looking forward to that :)
I really love watching your videos. They're full of useful information.
+50
Or from multiple files/modules. I've written a function that sets up logging and call that one in every file, but have no idea if that's the way to go.
@@thomasjohanns7661 You typically only run the logger in your main function. Then in the config you can set up how it should behave for different python modules (including any imported modules. say if you want to silence, or actually increase, logging for anything in particular).
@@_DRMR_ While this is a common practice, it does not always get implemented this way. For example, if I use a custom library that might have its own logging module. Also, multiple processes could imply two or more handlers on the same logger.
We use initialized loggers. Do not forget to check official python build in: QueueHandlera and QueueListener for separation of handler (OUT end of queue) from logger(INPUT side pf queue)
I’m a terrible coder, but only do it for personal projects, and I never thought about using logging to replace the print statements I use when developing, this would save me having to find all the random print statements all over my code 😊
Fantastic as always, would love a deep dive on the structlog library for your next video! really takes logging to the next level
Great suggestion!
*Thanks Arjan!* That is super useful 👍
Glad it was helpful!
You're videos are absolutely brilliant but what I really love is the code being available. It saves me a lot of time when I come back to one of your videos I've already watched a couple times.
As it is named deep-dive I was a bit hoping the focus of the logging levels and what are the daily praxis where to log, and how to handle the multiple levels in the code. This is something after years of developing software I still struggle with. Sometimes it feels like every second line is a log, and sometimes there are 3 log lines in the full project (hyperbolic). It never found a good regular way / a principle to follow ... :)
Whatever! :) Keep going, I learned a lot from you the last year I have now in my daily praxis. It helped me a lot :)
I've found talks by Kevlin Henney incredibly insightful and interesting. Here is a part from a talk (with a clickbaity title) where he addresses logging: ruclips.net/video/FyCYva9DhsI/видео.html. No concrete advice, but hope it's helpful nonetheless.
My personal idea of logging is pretty minimalistic: YAGNI. I make extensive use of logging in the debugging phase, as regular debuggers are often of little help with kinds of software I'm dealing with. But after I have figured out some part of the code and it is covered with tests, logging goes out of the window, unless I have very specific scenarios in mind where I would need it for a post-mortem. For example, if I need to log user actions, I have a set of requirements for that. Otherwise, I do not need to log user actions.
Logging throughout a module is way more complicated than you’ve shown though. Can you show how you set that up?
Was about to type this
100 % agreed
i just put "logger = logging.getLogger()" at the top of each file and call logger as usual. Works great
@@dirtdart81 but then you’re always grabbing the root logger, which is bad practice as far as I can tell
You are doing great videos, but IMHO this was a bit too basic, would love to see an even more deep dive into this module. Like setting it up for a bigger project, best practices, etc. Anyway, thanks a lot and have a nice weekend! 🙂
I think its good he started off basic so those of us who dont get a chance to use this stuff can have a basic setup. I hope he makes an advanced one too!!!
I am also looking for setting it up for bigger practices and best practices. i.e. I also would like to know how you can pipeline the logs to may be ELK or anything where it can be consumable
You gotta start somewhere. I loved it!
+ 100 on this idea. Most logging ends up being a overflowed garbage can of rotten log salad and parsing it to find confirmation of correct application, business logic performance, and the mixed bag of errors is just difficult to retrofit after integrated into production. Once it rots in production awhile, logs are definitely an 8th "olfactory offense".
@@OhsoLosoo Title should NOT be misleading. This is amateur/beginner logging, pretty far from Pro.
I used to work in tech risk for a FI. Regulators require all logs to be maintained and those logs once collected gets streamed to some security vendor to do help look for security issues at runtime. They ingest records at around 1K per second. That exposure changed my perception on logging
Great content as always Arjan! I think it would be also a good idea to show ways of not overly littering your code with logger calls (ie. using decorators, as in one of your recent videos, or perhaps using other methods) and perhaps demonstrating a real-world example of crucial logging within an app. All the best!
Would love to see more advanced logging concepts especially unified logging in a big library
Thanks Arjan! Could you do a follow up on structured logging?
Nice. I would have loved if you had shown what's best practice to actually log and at which places in the code loog statements are supposed to be.
That's very difficult to explain and is up to the project. It's almost artistic. The best way I can describe it is log anywhere you think will help you understand what the software did without tracing the code. You want to be able to diagnose what a problem is almost from the logs alone. That expands to log any information you need to diagnose. Like user names or values that were switched on or errors encountered
Great video as always!
Interesting to see use of rotating logs as well as multiple package log implementation.
I know I ran into issues with those when I first started writing in python.
I feel as though you just glossed over some of the basics and added the SysLog as a bonus. What about getLogger(__name__) combined with propagation? What about different handlers at different levels? What about custom formatters? What about using decorators to do your logging for you? If nothing else, there are some ideas for a future show :)
as always, great content Arjan.
Thank you so much!
I've used grafana + loki for very long time. Its very efficient with little to no resources. It can be setup in multiple ways and there is also a package for python called python-logging-loki. Loki has a really nice and powerful logql language with which you can make awesome grafana dashboards.
Arjan, thanks for uploading this :) the date/time formatting of logs was very helpful. I've been using INFO:ROOT this whole time like a n00b.
best for someone who just started to learn logging.
I love your videos about theory and architecture....once you have been coding for a while and you are no longer a begginer this kind knowledge feels really escential.
Yeahhhh thank you so much!! Really appreciate a logging tutorial!
You're very welcome!
That was very informative. I was avoiding logging like a fire but it's not as difficult as it initially seems.
Yessss I’ve been waiting for this! Thanks Arjan!
another important use is audit logs... we have some critical operations that the customers have to be able to see when they ran, how many times, with what parameters etc
I would agree with other commenters, it's nice to get a basic intro to logging, but I'd really like to know WHAT I should be logging.
In theory I could put a debug log for every line of code I write stating would should happen, but I imagine this is overkill. Or is it? I don't know where to draw the line, it would be good to get some examples of what kinds of things you log, and what level is appropriate for them.
But good video, and I'm looking forward to a part 2 😉
I am glad to have found this video. Thank you!
Glad it was helpful, William!
Again an excellent video ! I'm actually pointing all my new devs to some of your videos so that I'm sure they pick up good practices (even old ones like me). I didn't know papertrail, and I'm now starting using it. it's perfect, though I'm a bit worried about anybody being able to send logs to my account (no real auth)
I also find logging super important! Thanks a lot for this video with lots of good advice :)
I see that a lot of ppl here are asking for a more complex level of logging explanation, but let me tell you that as being totally new to logging, this video was perfect! straight to the point and very very useful for we ppl that are looking for an starting point. Thanks a lot.
Great Job bro !! Keep it up.. Thanks for this ❤🙏🏻
Thank you for the kind words! Will do.
Thanks, Arjan! As always, amazing video!
Glad you liked it!
Great job! Thank you so much
waiting for this!!!! Thanks Arjen!!
As always, excellent video. Making a simple topic so robust, I want to build some apps just to log now.
Thank you for the video. I didn't know SysLogHandler and Papertrail before.
In our company, we are using Graylog instead of Papertrail. Graylog isn't only for Python apps but also for other apps creaded via Java etc. Graylog has a REST API Server. Thus, we are accessing logs in Graylog via requests library in Python. Then, we perform Fraud Detection based on logs data.
There are also some libraries like loguru and structlog for logging in Python.
This was a lot more basic than I was expecting. Python's logging has a lot of pitfalls that I would have liked addressed.
Thank you for this!
DUDE.. I was searching for Logging tips TODAY.... lol
Amazing content Arjan, keep it up. Cool intro to Paper Trail too :)
Been playing with log2d that makes logging a bit easier.
Some logging services like Splunk provide a custom query language, and it's worth to learn it because it will save a lot of time. Also another tip is to avoid root logger and prefer to create a logger for each module, it will reduce the searching time as you can do more specific searches.
We have struggled with logging quite a bit in our Company and I guess the best solution if to use a central logging config object (dict or even better yaml file). There u set the Formatter, Handler and if you need to set custom attributes, you can do that pretty nice by using a filter.
Yesss. Log as JSON, ingest those logs into some central service (you could use Filebeat for ingesting the log files and use Elastic as search engine with Kibana for dashboarding and searching through your centralized logs).
You can also use `jq` (a terminal application) to sort, filter, format your files, etc. It's sooooo nice.
I use the structlog lib, which can be a pain to configure, but it's incredibly powerful and flexible.
@@NostraDavid2 Yep, we also use JSON format for the logs and a central instance where u can see all logs. This is luckily done automatically in our Kubernetes Cluster
I really appreciate your efforts in making this video and it was enjoyable to watch.
Informationally however it doesn't quite live up to the expectation set by the title...
you are best my friend.😍😍😍😍😍😍😍
Valeu!
Thank you so much for the support!
Thank you Arjan I like your videoes
Super useful, thanks!
Glad it was helpful!
Always good videos
With normal file logging you can search with grep, less and all the other standard tools, what could be easier than that?
I like you content very much. I'd like to know when to use logging & when to use print statement. Currently, I tend to checking "logging" via print statement.
Logging service, you just blowed up my mind :) I had not idea it existed. As usual thanks for the great content.
Happy to help, Daniel!
Thanks Arjan, good start on logging.
I recently struggled a lot to configure getLogger to get logging object from other module. Python documentation is not that clear on this. I even tried to use thirdparty pkg like logguru.
Try doing a video on configuring multi file/module logging and best practices
I expected you would mention log data has an interface. Changing your log statements can have serious impact for tooling that analyzes the log files for statistics, problems, and improvement opportunities.
Great video logs simple and insight full. But this content is available like everywhere, can you make a video on logging with different modules in the project or when using a design patter in projects and we have tons of files and folders in our project.
Hi there, from Portugal,
Any recommendation on using python for google cloud function using google cloud logging?
Obrigado(Thanks)
I agree that this was too basic. Mentioning Paertrail was very helpful
Thanks 🙏🙏🙏
those unequal length cords on your hoodie is driving me crazy
A great video and explanation.. Do you have courses in Python from intermediate level?
Yes I do! You can see all of my courses here: arjancodes.com/courses/
I enjoyed the video. I am currently working on doing a presentation on logging. During that process, I spent a lot of time understanding handlers and formaters. I become most unclear when looking at the best way to implement logging in a package compared to how to implement it in an application. Do you have the link to the second video you mentioned on logging?
Logging to splunk is useful too.
Good introductory video about logging in general. However not a deep dive on Python's logging package, as announced at the outset. In fact there is surprisingly little about Python in this video.
I used nbdev in VSCode to write python modules. Where does the logging module write it's output when working with notebook cells?
# import reload
from importlib import reload
reload(logging)
logging.basicConfig(format='%(asctime)s %(levelname)s: %(message)s', level=logging.DEBUG, datefmt='%I:%M:%S')
Hi, I am using this functionallity, but module "requests" messes up my log, as it also throws its requests, and for example I see updates of virus signatures and configuration downloads from firefox, is there any way to prevent it?, I could upscale my messages as warning, as it seems it only happens in the INFO configuration, but there has to be another way. Thanks for your great videos!
Nowadays one of the main reasons for a logging server is separation of concerns in the light of information security. If you get attacked, you may want to have a log server separated from your operational data proccessing - with no possibility to remotely delete your logs. Forensics will be very thankful.
Some tools (which can be a bit overkill) like splunk and elastic stack can really allow you to search and filter log files with ease. But this however is coming from a digital forensics background.
Hi Arjan, thanks for this video! I was hoping to learn how I should set up logging for a package, so that people that might install my package via pip get access to the logs when executing the code if they wish so, but maybe not automatically. Do you think you could make a second video about more advanced nuances about logging?
Also I saw another package (requests) set a NullHandler in __init.py__ and I was wondering what the reason behind that is.
Im working in how to implemet logs with fastapi and just I realised logs are printed in stderr by default wich will be a problem if a cicd pipeline relies on stderr in order to continue the right flow. I hope you will explain it here :(
how do I log to system.log on macos ventura? none of what i've found on SE or what has been provided by chatGPT works (no errors though)
I'm liking the glass of water 🤓
Why did papertrail mark the log message as coming from "logger" when the logger you configured was named 'arjancodes'?
what is the name of the compiler?
I wonder how often do large companies use an in-house logging solution. I’d imagine that would prevent some of those data leaks and would be perfectly feasible to implement given their size.
I like the sweater, what brand is that?
thanks:)
Why at the first place you need a separate logging service.??
All your logs are in one place, so you can more quickly find problems, better log visualization features such as colored logging and various types of charts, better log search, better export and backup capabilities, better automation features. Just to name a few things off the top of my head.
What are the pro's to have an explicit main() function vs having the contents of main() under __name__ == "__main__"?
I tend to do this because it increases the consistency of the code I write, especially as this also translates well to concurrent code. In that case, you would simply change the main function to async, and use asyncio.run under the if statement.
Can I write some code to like all your videos?
Haha, thank you, your comment made my day ☺️.
IMHO the logging module seems really over-complicated and easy to get lost in (handlers FTW). But now that it’s in the std lib, nothing more Pythonic gets much traction. See at all the comments asking for complex configuration examples, across multiple files. Video is a good intro to its existence but there’s only so much that can be conveyed this way. And in blog posts as well, most of them also barely scratch the surface of Python logging.
Thanks! I am trying to force people to use logging framework on a company internal project, it packed full of prints and I am so sick of getting screenshots of a command prompt.
Here's a great example... You can have python logging framework output to a syslog server... It's AMAZING. Imagine a few hundred users using your stuff and you get all the usage info, who is crashing and so on and so forth.
My question is if something goes wrong in production.. how can we back trace the problem from logs?
what is the purpose of the "if __name__ == "__main__":" block
Really important! Why is getLogger etc from the logging package camel case? It hurts my pythonic eyes
Real talk where is that sweater from?
Me, an intellectual: print(x) scattered throughout my spaghetti code
It'd be cool if you showed logging config from different files, like ini and yaml. thanks!
/dev/null as a Service!
This dive was into this module was really shallow
👏🏾👏🏾👏🏾
How do you configure your emacs this good? Can you make a video about this as well?
Anybody here uses loguru?
nice
Logging into a File with a Rotating FileHandler from multiple processes is not so nice tho xD
👍👍
This type of logging is insufficient. Where's the line number? Where is the thread ID if your app is multi-threaded? Where is the task ID if it's asynchronous? How will you know in which thread/task the error occurred.
Logging is hella important. If you think logging and not well organized and built logging systems is not important, you're still very junior.
remove all of this pain with just `pip install loguru` and `from loguru import logger`... yw
nothing compares to print("here")
Log to a database ..sqlite3 baby