I wanted to tell you I pressed like at 00:42, when I realised I'm watching one of the best summaries ever. The cotent as well was great. Very good job, mate.
argh would've saved me sooooo much time had I known about it sooner! Definitely improves code structure right from the onset and I hadn't appreciated how messy your code can get when working with argparse.
i believe if you add "Beginning 0:00" to the beginning of the timestamps in the description, youtube will divide the video bar to the partitions of it. edit:typo and grammar mistakes
Your channel and videos are really amazing. Great to continue with AI/ML/Reinforcement and other CS subjects. I am convinced that your are going to have a lot of subscribers. Good luck
Not sure if you’ve become aware of it since making this video, but Python 3 has the built-in `functools` library which has both the `cache` and `lru_cache` functions that can be used as decorators. `lru_cache` takes an argument `maxsize` to specify how many combinations of unique input arguments should have their results cached, and the `cache` decorator is the same but assumes an infinite number of catchable results. This has the benefit of not requiring redis nor an external plugin, but the downside is the cache only exists in that runtime of python.
I made a separate video on this. These in memory caching decorators have been around forever but the benefit of Redis Simple Cache is that it can survive a runtime restart.
@@JackofSome oh awesome! I just found your channel (through your Numba JIT video) and have been going through all the videos I'm interested from oldest to newest, so I guess I just haven't gotten there yet. Love what I've been seeing so far!
Hi Jari and thanks for the great content. Apparently you're using a tiling window manager. Have you considered doing an "abs (min (...))" video on that? By the way, your videos really improved my dev workflow. You introduced me to org-mode, spacemacs, fzf, autojump and vspacecode and, from now on, I could not code without them. You should set up a patreon, I will support you with great pleasure.
Hi. I get a lot of requests for my setup so I guess I should do a video soon. I used to have a patreon but my output is really sporadic so I took it down. If you want to donate you can do that here www.buymeacoffee.com/Ej8qUm5 Or alternatively you can donate to a charity like Humanity First or the Red Cross
One point about floating points getting truncated, the meaningful part of a double is the 16 numbers, the numbers that are truncated are just artifact of the format of floating-point and would not be exactly as given, you can test it with entering this snippet:"a = float('0.1111111111111111111'); print("{:.20f}".format(a)) " Which on my machine give the following output: 0.11111111111111110494 For more information about the issues rising working with floating points, check here: www.topcoder.com/community/competitive-programming/tutorials/representation-of-integers-and-reals-section-1/
Rich is really cool for building beautiful CLI apps, and if I am not mistaken it has it's own progress bars. Also looked for Typer and Loguru, they seem super easy to use
Thanks, a really cool video! I will definitely try some of these libraries out soon. As for the redis cache, do you think there's an advantage to using it over sth like lru_cache?
I tried to use tqdm to implement a file loading progress bar but somehow it just wasn't working out. I could figure out how to pass the total size of the file and see how much has been accessed or read to the package. I'm not even sure what the file type was anymore but if anyone could give an example or some insight, that'd be great. btw @jack, your videos really ARE great. I just came across them today randomly and I love them!
Nope. Doesn't persist which is what's different here. I seem to have done a bad job of explaining this because I get this comment a lot. Maybe I should make a video
I'm trying to figure out a good time. With the shelter in place order right now I can't go to the office to do the stream which is what I used to do. Maybe I'll do tutorials first and then do the livestreams for Q&A
Has something changed in msgpack? I downloaded the file from the git repo and ran it in python3 and python3.9 and I get a traceback as Traceback (most recent call last): File "c:\temp\ytvideo_python_lib_favs\five_amazing_python_msgpack.py", line 62, in msgpack_example_1() File "c:\temp\ytvideo_python_lib_favs\five_amazing_python_msgpack.py", line 43, in msgpack_example_1 back_from_msgpack = msgpack.unpackb(ftmp) File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb ValueError: int is not allowed for map key
I got the same error with msgpack-1.0.2 and patched it like this : back_from_msgpack = msgpack.unpackb(f.read(), strict_map_key=False) Hope this helps!
Argh is not really maintained, and I don't see the advantage of redis simple cache over functools' lru_cache. The msgpack homepage has some issues with showing the available implementations as well. Other than that it's pretty interesting! I can recommend APScheduler for a more flexible and powerful scheduler for Python.
Not maintained yes but still perfectly functional, but yes it's probably better to use click or something. As far as I know lru_cache gets scrubbed as soon as the process exits. Is that not the case? Redis cache is persisted to disk. I'd be interested in reviewing the issues with msgpack. I know in the early days they had some problems but haven't encountered any myself. It was a huge boon to a project where I needed a python server to send lots of data over websockets to a webpage.
@@JackofSome Cheers for the reply. Yes click does indeed seem more on top of things! You are right that lru_cache is not persisted across runs, neither is 3.9's unlimited "cache" afaik. But on the other hand it doesn't require any external dependencies, so it's worth a mention I think. Regarding msgpack, I'm just referring to their homepage not displaying anything in the bottom section titled "Languages" and "API". Other than that I think it's a solid option for serializing data that doesn't need human reading.
i have sed lru_cache from functools for cache memory, but you use redis simple cache, i understand that the recently used value it save in redis database, is it true? but is this the case its more powerfull for big data!
Aren't these all in-memory though? I've been meaning to make a python 3 compatible fork of redis cache and maintaining it. Maybe now is the time edit: I reviewed the codebase for cachetools and it's a very nice library. I really like it. But it _is_ still in memory. Even the TTL cache does not persist to disk in any way.
That's a very complicated and situation specific question. For my production environment, for example, I use a combination of in memory caching, disk based caching, and redis based caching, all hand written.
Hi can someone explain when did we realistically need argparse and tqdm becaue from my perspective it seems to not be that useful because python is mainly for backend development right?
Pickle is specific to python whereas msgpack is more like json. Many languages implement the spec and so the data can be easily shared (e.g. I have a 3d visualizer that runs on the web and can communicate with a local process over websockets and send data using msgpack)
Included the libraries and what they do at the start of the video to not waste your time. Instant like for me.
Holy shit the last one (Redis Cache Library) that's my fork of the main library lmao
Ayoooo. Thank you so much for making the fork! Did you also upload one of the pypi packages?
Nah, I haven't uploaded any pypi package ever, lol.
Timestamps for mobile users:
Argh - Painless CLI Generation (00:43)
tqdm - Easy Progresbars (05:10)
msgpack - Like Binary JSON (08:44)
schedule - Python's cron (12:43)
Redis Simple Cache - Memoization (14:11)
Very insightful advice about the potential dangers of redis simple cache from a Reddit user: www.reddit.com/r/programming/comments/flww79/intro_to_5_amazing_python_libraries_argh_tqdm/fl18qpp?
Alas the insightful advice has been deleted.
I wanted to tell you I pressed like at 00:42, when I realised I'm watching one of the best summaries ever.
The cotent as well was great. Very good job, mate.
Came for the suggestion, stayed for the insightful software engineering commentary. ❤️
This is the nicest comment I've ever gotten.
the name tqdm derives from the Arabic word "تقدم: taqaddum" which translates to progress.
Wonderful
Nice video!! I have just made use of tqdm in one of my university projects :D. Thanks for discovering it to me!!!
My pleasure :)
argh would've saved me sooooo much time had I known about it sooner! Definitely improves code structure right from the onset and I hadn't appreciated how messy your code can get when working with argparse.
i believe if you add "Beginning 0:00" to the beginning of the timestamps in the description, youtube will divide the video bar to the partitions of it.
edit:typo and grammar mistakes
Thanks! I finally did that and the chapters are there now.
Your videos are just awesome, I truly enjoy them. Please don't ever stop making them.
My last serious video was a year ago 😅
But I'm making more :)
Your channel and videos are really amazing. Great to continue with AI/ML/Reinforcement and other CS subjects. I am convinced that your are going to have a lot of subscribers. Good luck
Thank you for your kind words. Please consider sharing the videos with others that might be interested.
No worries. As lover to open source SW I always share the state-of-the-art work. Thanks for your effort!
Everything you need to know about coding new string methods is perfect for you, the person who is reading right now will have a luck tomorrow!
I use tqdm on a daily basis. It’s my canary when crunching number or looping.
It also works in CL and Jupyter NBs.
Not sure if you’ve become aware of it since making this video, but Python 3 has the built-in `functools` library which has both the `cache` and `lru_cache` functions that can be used as decorators. `lru_cache` takes an argument `maxsize` to specify how many combinations of unique input arguments should have their results cached, and the `cache` decorator is the same but assumes an infinite number of catchable results.
This has the benefit of not requiring redis nor an external plugin, but the downside is the cache only exists in that runtime of python.
I made a separate video on this. These in memory caching decorators have been around forever but the benefit of Redis Simple Cache is that it can survive a runtime restart.
@@JackofSome oh awesome! I just found your channel (through your Numba JIT video) and have been going through all the videos I'm interested from oldest to newest, so I guess I just haven't gotten there yet. Love what I've been seeing so far!
Awesome content, some python gems I didn't know. Thank you! Subscribed.
Hi Jari and thanks for the great content.
Apparently you're using a tiling window manager. Have you considered doing an "abs (min (...))" video on that?
By the way, your videos really improved my dev workflow. You introduced me to org-mode, spacemacs, fzf, autojump and vspacecode and, from now on, I could not code without them.
You should set up a patreon, I will support you with great pleasure.
Hi. I get a lot of requests for my setup so I guess I should do a video soon. I used to have a patreon but my output is really sporadic so I took it down. If you want to donate you can do that here www.buymeacoffee.com/Ej8qUm5
Or alternatively you can donate to a charity like Humanity First or the Red Cross
4:44 nobody can keep me from passing my settings dict through 50 functions!
_Oh_ lol
0:54 - I feel a Korra reference
Freaking amazing
Great mentor, please keep keep on, although found a year later.
I love this, this is going to revolutionise my code! However, for msgpack, there is the native pickle library. How do these compare?
pickle is great but it's python specific. msgpack allows you to use the data in other languages (e.g. nodejs, something I do alot).
One point about floating points getting truncated, the meaningful part of a double is the 16 numbers, the numbers that are truncated are just artifact of the format of floating-point and would not be exactly as given, you can test it with entering this snippet:"a = float('0.1111111111111111111'); print("{:.20f}".format(a)) "
Which on my machine give the following output:
0.11111111111111110494
For more information about the issues rising working with floating points, check here:
www.topcoder.com/community/competitive-programming/tutorials/representation-of-integers-and-reals-section-1/
Ever heard of APscheduler? It's similar to the scheduler library but you add jobs to a scheduler object instead. I find it pretty reliable
Heard it from another commenter here. Will try it out
tqdm is fantastic.
Other cool random libraries: Typer, Rich, Loguru, Diskcache, pyzmq, Datasette, FastAPI .... just continue the list :-)
Rich is really cool for building beautiful CLI apps, and if I am not mistaken it has it's own progress bars.
Also looked for Typer and Loguru, they seem super easy to use
Thanks, a really cool video! I will definitely try some of these libraries out soon.
As for the redis cache, do you think there's an advantage to using it over sth like lru_cache?
As far as I know lru_cache is gone once your process exits. redis cache doesn't have that limitation.
@@JackofSome Oh yeah, thanks, that makes sense.
WHOA
You are a great guy!
You forgot keyboard and mouse for python macro manipulation.
Gotta leave some things for future videos :D
Instead of redis you can Just use cache from functools (works the Same but without extra Installation)
I just put out a video showing why that's not equivalent
One library i like a lot is pretty_downloader by deadsec-security
I tried to use tqdm to implement a file loading progress bar but somehow it just wasn't working out. I could figure out how to pass the total size of the file and see how much has been accessed or read to the package. I'm not even sure what the file type was anymore but if anyone could give an example or some insight, that'd be great.
btw @jack, your videos really ARE great. I just came across them today randomly and I love them!
What's the lib/command you're using to see that nice history menu on 2:50? :o
Nice content, btw :)
Fzf. I have a video on that.
I really like the font you used! Could I know the name of it?
Mononoki
So you moved to doom now? Funny how I didnt like spacemacs but I am actually using doom emacs rn :)
Nope. Still Spacemacs, theme and modeline are Doom.
tqdm is awesome
Amen to that
For cache use lru_cache from functools which should already come with your python installation, that one does the exact same thing.
Nope. Doesn't persist which is what's different here. I seem to have done a bad job of explaining this because I get this comment a lot. Maybe I should make a video
When are you doing your policy optimization lviestream! I am looking forward to it
I'm trying to figure out a good time. With the shelter in place order right now I can't go to the office to do the stream which is what I used to do. Maybe I'll do tutorials first and then do the livestreams for Q&A
Whaaat. I haven’t really done a cli because it just seemed to cumbersome for me to even bother.
I guess there is no reason not to now...
The inverse for me, I guess. The vast majority of my programs are CLI-only because I just didn't want to deal with a GUI.
Your using xubuntu, right? What desktop manager are you using? That makes your windows split like that?
i3
Has something changed in msgpack? I downloaded the file from the git repo and ran it in python3 and python3.9 and I get a traceback as
Traceback (most recent call last):
File "c:\temp\ytvideo_python_lib_favs\five_amazing_python_msgpack.py", line 62, in
msgpack_example_1()
File "c:\temp\ytvideo_python_lib_favs\five_amazing_python_msgpack.py", line 43, in msgpack_example_1
back_from_msgpack = msgpack.unpackb(ftmp)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: int is not allowed for map key
Interesting. I haven't tried msgpack with newer versions. Lemme check
I got the same error with msgpack-1.0.2 and patched it like this :
back_from_msgpack = msgpack.unpackb(f.read(), strict_map_key=False)
Hope this helps!
Argh is not really maintained, and I don't see the advantage of redis simple cache over functools' lru_cache. The msgpack homepage has some issues with showing the available implementations as well. Other than that it's pretty interesting! I can recommend APScheduler for a more flexible and powerful scheduler for Python.
Not maintained yes but still perfectly functional, but yes it's probably better to use click or something.
As far as I know lru_cache gets scrubbed as soon as the process exits. Is that not the case? Redis cache is persisted to disk.
I'd be interested in reviewing the issues with msgpack. I know in the early days they had some problems but haven't encountered any myself. It was a huge boon to a project where I needed a python server to send lots of data over websockets to a webpage.
@@JackofSome Cheers for the reply. Yes click does indeed seem more on top of things! You are right that lru_cache is not persisted across runs, neither is 3.9's unlimited "cache" afaik. But on the other hand it doesn't require any external dependencies, so it's worth a mention I think. Regarding msgpack, I'm just referring to their homepage not displaying anything in the bottom section titled "Languages" and "API". Other than that I think it's a solid option for serializing data that doesn't need human reading.
Is there any gain from using APScheduler over schedule?
i have sed lru_cache from functools for cache memory, but you use redis simple cache, i understand that the recently used value it save in redis database, is it true? but is this the case its more powerfull for big data!
click is better than argh. And typer adds support for type annotations.
I've used Click + setuptools for CLI functions before. How does argh compare?
Click is just as good (or possibly better) than argh
What about pypi.org/project/cachetools/ instead of redis-cache since it looks abandoned?
Aren't these all in-memory though?
I've been meaning to make a python 3 compatible fork of redis cache and maintaining it. Maybe now is the time
edit: I reviewed the codebase for cachetools and it's a very nice library. I really like it. But it _is_ still in memory. Even the TTL cache does not persist to disk in any way.
Which desktop-environment is he using here? 4:38
What would u recommend for prod in case of redis cache it?
That's a very complicated and situation specific question. For my production environment, for example, I use a combination of in memory caching, disk based caching, and redis based caching, all hand written.
Hi can someone explain when did we realistically need argparse and tqdm becaue from my perspective it seems to not be that useful because python is mainly for backend development right?
If you're working on web, then probably yes python is probably only useful for backend work. To the rest of the world it's used for everything.
why would one use msgpack
over json+ gzip?
Why not use pickle as data serialization?
Pickle is python specific and some times stores more than what you need. Very useful in some cases, but not as a general format in my experience.
@@JackofSome Thanks for the fast response! I actually didn't know that.
What desktop environment are you using?
xfce with i3 as the window manager
Would you be able to do an example of msgpack with sockets?
I actually use msg pack for communication between a tornado server and a webapp written using typescript. Check out safijari/3viz if you're curious
@@JackofSome Okay, thanks for the quick reply! Much appreciated
isnt msgpack just NBT
What about lru_cache from functools? Is it so much better?
I made another video about why this isn't the same thing and has a different use case. I recommend giving it a watch
@@JackofSome thank you very much!
Argh vs click? Who will win?
Click since argh is kind of dead.
Unless...
Whats the difference between msgpack and pickle?
Pickle is specific to python whereas msgpack is more like json. Many languages implement the spec and so the data can be easily shared (e.g. I have a 3d visualizer that runs on the web and can communicate with a local process over websockets and send data using msgpack)
How does msgpack differ from protobuf?
Protobuf needs a defined schema. Msgpack does not. Msgpack is just a more storage efficient json.
Cant u just import cache instead of using redis cache?
Cache is in memory for the current process no? Or does it persist to disk?
Argh has been abandoned for years, I think Click is a better option
Fair point. Though it's not like there's anything particular that needs maintenance. I'll check out click
what is you font?
Mononoki
@@JackofSome thanx, also can you make video about secure data serialization, like pickings or Json over socket.