i will literally never have a practical use case for this but it still makes me happy it exists edit: ok within a year I have used it several times I retract my previous statement
The answer is basically proper encapsulation. You can shift the complexity to the caller (outside the class) or you can keep it in the class where its manageable while keeping the caller (outside the class) simple. Creativity is everything and this is just another tool to support that.
I actually used Metaclass to modify the __iter__ method of my Enum class to skip some values. Basically like this: from enum import Enum, EnumMeta, auto class MyEnumMeta(EnumMeta): def ignore(Cls, e): return False def __iter__(Cls): return (e for e in super().__iter__() if not Cls.ignore(e)) class MyState(Enum, metaclass=MyEnumMeta): Empty = auto() A = auto() B = auto() C = auto() @classmethod def ignore(Cls, e): return e is Cls.Empty
I used it all the time when I was a Java developer when I needed similar blocks of code to handle single item of type X or a list of items of type X. 99% of the time it was two functions with one function handling single item and the "overloaded" 2nd function handling the List of that item
dunno, i use overloads all the time in Nim and C++, its rather handy tool for segmenting code otherwise you would need to create a single function in python that has all the implementations depending on args* kwargs** signature which is rather clunky
Man the information density in your videos is mindboggling. Every 20 seconds there's something I'd need to sit an hour for to fully absorb the concept and the code. Well done, thanks
Since I've -lot's of work- nothing to do right now, I counted how often you've said the word 'class' and 'metaclass' in this video: *·* class ⟶ *62 times* *·* metaclass ⟶ *33 times* Oh my god, I watched the video on 2x speed and at 1:37, my brain just started to hurt. I really like your content by the way. There are so many useful python features I've hever heard of before. 👍
@@mCoding No, I counted them seperately, I counted things like "class body" though. Still 70 times seems like a very high number. EDIT: I counted again and updated the numbers, though not much changed.
Metaclasses looks very powerfull, unfortunately most people using "powerfull" features in a big codebase usually leaves the most powerfull mess to be maintained. Great video! #pycharm
My mind is blown. So many possibilities once you know what Python does under the hood. I would find it super interesting to learn more about how C-bindings like numpy uses them are built and how one could add them too. #pycharm
My mind is not blown as these features have been available in other OOP languages for ages. However I still like Python for its simplicity and would dread going into typing library, generics etc. I prefer simplicity.
All programming languages seem to converge towards each other. Combine this with Cython and static typing, and you just made a slower C++, but with more libraries. #pycharm
@@chrisvinciguerra4128 Yeah that beautiful manual memory management when you're designing a high-level user-facing application was always the right way all along.
one of my favorite things about python is how modular the language itself is. You can just overload anything you want, and having the syntax be so malleable is so fun!!
"the type of big A is a type but the type of small a is big A, whereby the type of type is type" That's exactly the feature you will probably not use for a long time, but years from now you will remember and it will save your life! Best thing about python, it's not like other languages where many base features are hard-coded, but you can customize EVERYTHING #pycharm
@@georgplaz yeah, too much customisation just gets everyone confused, you cant replace every descriptive function name with some __magic__ and have people (and yourself) still understand it.
This is actually that rare gem content found under the depths of the internet. Been trying to wrap around meta-classes for a week now, with so little luck. The docs/content revolving around such concepts are so scarce, But you made it so clear as day. Thank you :) -don't need pycharm cause VSC is nicer lol-
I've not used python in years, and primarily use TypeScript myself currently. I had an alright time using Python when I did, but didn't think too much of the language, but your videos including this one really make me appreciate python more! This whole system shown in the video is very elegant imo, and very powerful.
I feel like your videos have given me a really deep understanding and appreciation of all the internal magic that goes on behind the scenes on Python. Thanks for the content!
David Beazley, as far as I remember, advocates the use of class decorators (where possible) instead of metaclasses. The decorators are easier to handle, easily composable and easier to understand. But sometimes... yes, one has to take the bull by the horns and dive into metaclasses (not for the faint of heart). There's no wizardry at play here but one has to go through a lot of theory and practical examples to really start to see (appreciate?) the structure and intent of the creators of the language. Thank you for the video and spreading the knowledge.
It's interesting to see overloading implemented in Python, I saw a stackoverflow comment suggesting to use multiple dispatch instead and now I might see why considering the slowdowns. Great video as always :) #pycharm
I taught Python for 2 years at the university level. I thought I knew more deep wizardry than any person should know. After seeing this, it's clear that I know nothing. #pycharm
Same here, my friends call me when they have a question about Python because they know I love it and know it quite well, but when they call me a "Python master" I humble myself by remembering this channel exists :')
Very interesting information there. I would think, though, that most people have gotten around not having overloads built-into standard Python for so long, that normal Python paradigms suffice for all their coding. Overloading is nice to have, but I think it's more of syntactic sugar to other possible syntaxes. Somehow I can't help but think about the Zen of "Explicit is better than Implicit" might apply here as well, especially when usage is coupled with non-keywords args. Thank you for the wonderful videos, and I must admit this is my first comment on your channel, but I follow your videos constantly. #pycharm
Just some feedback... the constant cuts saves viewing time but also impedes comprehension. It takes you time to talk about these ideas and likewise it takes time to comprehend them.
I've definitely learned more from your channel than any of the other python channels, and courses I've taken combined. Please keep making such great content. #clion
Well. That was... mind-bending indeed! Not so much the usage (that was pretty straightforward) but all the details and digressions in the implementation. I always follow along with these videos in my python interpreter, so I can pause the video whenever I'm not certain about something, and play around a bit until I'm sure I've got it, then resume the video. Normally that only adds a few minutes here and there, but this video took me -- I kid you not -- OVER THREE HOURS to get through due to all the times I had to pause it and work out what was going on, not to mention all the scrubbing back and forth to refer to an earlier part of the video, unwinding the code as I went, to understand exactly what was being passed around in a later part. ANYWAY. I think I mostly have it straight now, with one glaring exception: At around 11:23 on line 64, you throw 𝚔𝚎𝚢: 𝙾𝚟𝚎𝚛𝚕𝚘𝚊𝚍(𝚟𝚊𝚕) into the 𝘰𝘷𝘦𝘳𝘭𝘰𝘢𝘥𝘦𝘥_𝘯𝘢𝘮𝘦𝘴𝘱𝘢𝘤𝘦 dictionary. But I don't understand how the instance of Overload gets its __𝘨𝘦𝘵__() method called when it's an element of the dictionary? I would understand if it were an attribute of some class object 𝘢. Which is to say: let's assume we have an instance 𝘰𝘷 of 𝘖𝘷𝘦𝘳𝘭𝘰𝘢𝘥. Then if 𝘰𝘷 were an element of 𝘢, accessing 𝚊.𝚘𝚟 would for sure trigger a call to 𝘢.𝘰𝘷.__𝘨𝘦𝘵__(). But if you have 𝘰𝘷 as an element of a dictionary (let's call it 𝘥) which is in 𝘢, then accessing 𝚊.𝚍['𝚘𝚟'] would presumably only ever trigger a call to 𝘢.𝘥.__𝘨𝘦𝘵__()? So how does 𝘢.𝘥['𝘰𝘷'].__𝘨𝘦𝘵__() ever get called? I'm guessing that it's some magic behind the implementation of 𝘵𝘺𝘱𝘦.__𝘯𝘦𝘸__() ? Do elements of the namespace being passed to that function get transformed into attributes of the class being created? Or am I missing some other intuition here? Other than that, well... I am... enlightened!
Wow! I'm so happy to hear that you follow my vids to closely, this is surely the best way to learn! Indeed it is some magic behind type's new method. type.__new__(mcs, name, bases, namespace) basically eats the namespace argument and turns a wrapped copy of it into the new class's __dict__. So when I pass the overloaded_namespace in that spot, the overloaded_namespace essentially becomes the class dictionary for the newly created class! So you guessed exactly right!
James, every video I watch you blow my mind with what is possible in Python, I don’t know yet what I’ll do with all this new knowledge but dang is it fascinating! Thanks for the quality content #pycharm
Now, that we've learnt about this from this fantastic video, probably it's time for the designers of the language to learn, from the very same video, that the most basic feature of am OO language should not be paid in blood.
Haha you're welcome to suggest but I think the designers are mostly against checking types at runtime. Also they will tell you python is a multiparadigm language, not just OO!
I'm really happy that you are making speed comparisons. This is something I often do with python, to figure out what works best, but I don't always understand why. #clion
Thanks sir, your content is always enlightening. Frankly, it sometimes stupefies me how much there is to learn about a language that many of my friends, and my computer science teacher, dismiss as simplistic... great content as always! #pycharm
Seriously, if this language had tricks up its sleeves to do a lot of this at compile-time (like Zig), it would've been the most powerful language across the domains.
@@comradepeter87 You can look into JIT compiling solutions for Python like numba (library) or PyPI (interpreter) for example, it lets the code be compiled at runtime (which slows down the first function call, but generally makes the next ones way faster, catching up to the realm of fully compiled languages)
This is really fascinating stuff James! I thought I was pretty experienced with Python, but you are helping me take my understanding to another level. Many thanks for sharing your wisdom.
"a class defn is actually just syntactic sugar for calling a type constructor" There's a tiny caveat here, like being able to pass kwargs to the type definition's bases, which can't be done with `type`. To be able to do that you'd have to use `__build_class__()` instead, so in python3 (i believe starting from py3.1) it's actually sugar for the build class top level function instead. It's nothing major though, but I think if you're making an absolute statement it should be correct.
I've faced with problems trying to implement code at 14:30. The key is pylance can't recognize arguments for dynamically generated `__init__()` method and keep warning me. Are there any ways to fix this? I implemented this in order to get autocomplete on __init__ arguments but I'm wondering now if it's even possible.
In every large code base that I've worked in, a few people have tried out obscure-but-extremely-cool language tricks. Most, but not all, of those cases resulted in long discussions with the people who have to maintain the code, test the code, see the code and want to know about it just out of curiosity, or worst of all, were tracking down a difficult bug and wondered if this obscure code they don't understand could somehow be the culprit. If you take into account the cost of those discussions, it's very rare for the trick to provide a clear net benefit. It's happened, but it's very rare. In a large code base, the highest priority is to make the code readable by others. Please remember that before you use a feature that most others do not know.
I totally agree, I'm often that person tracking down why a framework needed to change everything to save a few characters. But I do love exploring the limits of what is possible.
Yet another banger, I didn't even know this existed to be honest! Really interesting watch. As for the giveaway I 'm a student so I get their stuff for free, but it's actually incredible you got that opportunity from Jetbrains! Congrats on that.
Your videos are very useful. Short and to the point and ofter cover subjects you may not encounter when using the language even professionally. Thank you!
Another little suggestion: you generally shouldn't try to subclass `dict` because it has special cases in the interpreter, you should extend `collections.UserDict` instead.
I think the good part is that there are so many concepts you touch that I didn't even know in Python. I had to stop several times to look the documentation and learned a lot! Thanks! #pycharm #clion
Excellent video. It took me some time to understand it, but it is pretty genius what you do. I saved this video because of it being so valuable... I see that your example implementation that allows function overloading, indeed causes a big amount of overhead. Maybe its an idea to raise a PEP for getting a real solution that allows us to overload functions in future version of python. Because honestly, I think it is a very worthwile addition to python.
Being a beginner with Python I'm quite happy I managed to follow what was going on here. Even though the approach discussed here has various negatives the content as a whole was quite informative. Thank you. #pycharm
a perhaps more efficient way to do overloading is via monomorphisation instead of having one key having a list of possible values that you then need to search through, you instead create a new name for each method based on its argument list and create a method called something like "f_int" and "f_string" and "f_any_any" as the names doing this lets you take advantage of the dictionary's fast lookup without being also limited by the list's slow lookup
No, function overloading improves readability and avoids remembering too much function. Instead of having multiple "f_int" and "f_string" and "f_any_any". It is better to only have 1 "f" function to handle all arguments.
Really neat! Just a question: why not use a dict of signature hashes resolve the best_match? I feel like it would be faster than iterating over the list of signatures.
Yeah, metaclasses are super powerful but also something you should think hard about as to whether they're the right tool for the job. For the purpose of function overloads, I'd probably even say they're not the correct choice, and would just be better just as a property-like decorator where you do @overload for the first definition and then @f.overload for the subsequent definitions. And of course there's the question of whether having overloads in python code is even good
This is called polymorphism in OOP and also I’ve seen a technique of overloading in Python before, can’t remember for certain now, but I think it’s in one of Corey M Schafer videos
I'd love to see a short followup to this concept using the new Python Match syntax, I don't imagine it drastically changing performance, but I could imagine it allowing for a much cleaner code base.
@mCoding would a set be a better structure than a list to keep "seen" parameters? In a list you'd run the risk to overload more than once with the same arguments. Unless you want to combine overload and duck typing? 🤯
I was quite lost in this video. What are the pre-requisites to understanding it? Can you point me to other videos which would build my knowledge up to understanding the concepts in the video? Thanks.
Is method overloading something that should be done in practice in Python? It seems like this is a way to do it but because it can't be done directly, I wonder if the intention was to not allow for overloading methods.
Definitely not something that should be done in practice. It essentially involves decision-making based off the type of an object, which goes against Python's duck typing philosophy.
Oh wow I didn't know you could do actual-for-real code generation with python metaclasses. I do a lot of multi-language development, and that often means needing to use a common configuration file between all the different languages and platforms to perform language-specific code generation, just to make sure that all of the implementations stay in sync and we don't introduce bugs where one language treats some byte in a packet different from another language. If I can just read the config and do the code generation at _import time_ in python, that'll cut out the need to add a whole separate build step for every change and make iterating so much faster.
Okay, okay. I’ll bite: #pycharm #clion Thanks for another great video. My brain did a double-take at the start, I was questioning myself “wait no, Python doesn’t have function overloading…” mCoding: “Python doesn’t have function overloading.” Me: “oh thank god” mCoding: “Except when it does.”
At least a clear and also deep explanation of metaclass mechanism and some uses. Not sure I will ever use it since changing much class behaviour makes code difficult to maintain. Thanks man. #pycharm which is my favorite IDE for python.
Amazing. I’ll definitely be needing to watch this vid again a few times to understand it, but it’s truly mind blowing what you can do with metaclasses. #pycharm
WoW !!! Metaclass are overwhelming. But they are quite amazing. And really like your efforts in teaching these concepts to us. This is the most interesting RUclips channel out of all my subscribers #pycharm
I'm sure you are in good company, metaclasses are probably the most difficult thing to master in Python. Master metaclasses and you can master the whole language!
Yes, pass it used where syntactically something is required but you want to say "do nothing". It is typically used in functions, classes, or if clauses that aren't supposed to do anything.
Glad you noted the performance hit there; runtime reflection can cause slowness in other languages too (e.g. C#). Things like Rust macros or C# source generators can move this work to compile time (which has it's limits too), but I'm not sure there's any way you could do that in Python.
`eval` and `exec` are there to do such things. (Even though, technically, the compilation will be at runtime for it, but at least this is relatively quick, much faster than introspection, because Python is good and fast at parsing and compiling Python). If you need it quicker (bypassing the needed compilation at runtime), you can also generate/template the compiled bytecode, that's possible, too and if the templating is fast (it should be usually), then you have close to no overhead. But of course, you need to be relatively good in understand Python Bytecode and working with the disassembler to get there. (Tbh, I only know theoretical that it works from other author's articles, but never implemented it on my own). And of course, you still can take the same approach as C/C++ and write a code replacement and generation engine with all the same caveats as in C/C++. In some areas, this is even regarded as a good practice, e.g. when working with protobuf. The biggest advantage is, that it allows to have an easy IDE integration, as the IDE can inspect statically what's going on. Otherwise, the IDE needs internal support (that's what's happen in PyCharm for Django as an example where also a lot of code is generated dynamically - there also mainly for performance reasons over any introspection/reflection injection)
Is it old-fashioned to implement the multimethods module to invoke function overloading in Python? Which is the more convenient method of overloading functions in Python?
Your videos are so impressive, very good explanation and really clear examples - for real you are a cut above the rest in this space and I'm glad your channel is growing so quickly! #pycharm
If this concept were incorporated into the python core language (and thus implemented in C), would it be fast enough to be worth it, or still too slow? #pycharm #clion
It definitely could be made fast enough if it was implemented natively, but the idea of a feature that does runtime type checking would probably never be voted into the core part of the language.
I remember stumbling upon this concept once and wrote some magical shit that even I couldn't parse after the fact, but it worked like a charm so we went with it... If I could track down the code I bet I could understand it better now, thanks for the deep dive! On a side note, abstract base classes are one of my faves, love making use of them #pycharm
#Python Question This is getting very close to what I am attempting to do using a Groupby with a Lambda Series for multiple function calls and appending the results to a growing List if it rings True from the Function. Would you suggest this is doing essentially the same thing ? Do you have a deep dive on Groupby with Custom Lambda Function with a Series ? Thanks for any references that supports this effort.....
This is really cool, I use function/method overloading all the time in Java. But as with everything, if the others reading your code don't know what's going on, you're probably going to have issues down the road :/ But your videos are excellent, really delving into what's going on in the internals.
learning these bits of lower-level mechanisms in Python lets me understand how Python magic works, especially for those built-in decorators like property. #pycharm #clion.
6:58 "__call__ is called whenever you try to make an instance of the class" Isn't __call__ usually called when an instance of the class is called, not the class itself? Is it different for classes that inherit from type?
Dunder call is called when you put parens after the object. If your object is a type, "calling" the class object semantically means making an instance of that type.
Metaprogramming and Reflection might be a little over my head, but this is way easier to read, than other methods for python overloading even if it does cost some time. #pycharm
This was amazing! I was just wondering if there was a way to do this in python. I didn’t even know metaclasses existed before now. Thank you for the very detailed explanation! Keep it up! #pycharm
i will literally never have a practical use case for this but it still makes me happy it exists
edit: ok within a year I have used it several times I retract my previous statement
The answer is basically proper encapsulation. You can shift the complexity to the caller (outside the class) or you can keep it in the class where its manageable while keeping the caller (outside the class) simple.
Creativity is everything and this is just another tool to support that.
I actually used Metaclass to modify the __iter__ method of my Enum class to skip some values.
Basically like this:
from enum import Enum, EnumMeta, auto
class MyEnumMeta(EnumMeta):
def ignore(Cls, e):
return False
def __iter__(Cls):
return (e for e in super().__iter__() if not Cls.ignore(e))
class MyState(Enum, metaclass=MyEnumMeta):
Empty = auto()
A = auto()
B = auto()
C = auto()
@classmethod
def ignore(Cls, e):
return e is Cls.Empty
I used it all the time when I was a Java developer when I needed similar blocks of code to handle single item of type X or a list of items of type X. 99% of the time it was two functions with one function handling single item and the "overloaded" 2nd function handling the List of that item
dunno, i use overloads all the time in Nim and C++, its rather handy tool for segmenting code
otherwise you would need to create a single function in python that has all the implementations depending on args* kwargs** signature which is rather clunky
@@Jason-b9t crazy
This video was so magical I'm tempted to call it a ___video___
Or a video of a __video__
5 letters are too long. use __vid__
Ah... *dunder video*
magic, like a magic ___methods___ in Python )))
@@zoulei881027 yeah and the __str__ will return “video”
Man the information density in your videos is mindboggling. Every 20 seconds there's something I'd need to sit an hour for to fully absorb the concept and the code. Well done, thanks
Since I've -lot's of work- nothing to do right now, I counted how often you've said the word 'class' and 'metaclass' in this video:
*·* class ⟶ *62 times*
*·* metaclass ⟶ *33 times*
Oh my god, I watched the video on 2x speed and at 1:37, my brain just started to hurt.
I really like your content by the way. There are so many useful python features I've hever heard of before. 👍
Do occurrences of metaclass also count towards class?
Don't worry, if you watch plenty of videos at 2x, you'll get used to it
@@mCoding
No, I counted them seperately, I counted things like "class body" though. Still 70 times seems like a very high number.
EDIT: I counted again and updated the numbers, though not much changed.
@@foxcirc was that "edit" a form of youtube commit message? 😁
7:55: _"No need to be confused"_
Me: Most confused I've ever been in an mCoding video
#pycharm
Metaclasses looks very powerfull, unfortunately most people using "powerfull" features in a big codebase usually leaves the most powerfull mess to be maintained. Great video! #pycharm
Hello! "Powerful" is spelt with 1 'l' not 2 :)
Well said
@@5h5hz Thanks. At least I was consistent.
@@pericofalcor hahaha such a programmer response, I love it!
He meant powerfull of poo poo
My mind is blown. So many possibilities once you know what Python does under the hood. I would find it super interesting to learn more about how C-bindings like numpy uses them are built and how one could add them too. #pycharm
How do you go about finding what python does under the hood, in a relevant manner?
My mind is not blown as these features have been available in other OOP languages for ages. However I still like Python for its simplicity and would dread going into typing library, generics etc. I prefer simplicity.
All programming languages seem to converge towards each other. Combine this with Cython and static typing, and you just made a slower C++, but with more libraries. #pycharm
As the world progresses more we all figure out the best way to do things has been C++ all along.
@@chrisvinciguerra4128 c++ is not a good choice for embedded systems c ftw
@@duality4y May aswell just hammer the opcodes into RAM yourself
@@chrisvinciguerra4128 Yeah that beautiful manual memory management when you're designing a high-level user-facing application was always the right way all along.
Gratz! You (Yannick) get a PyCharm license! Contact me to claim your prize! mcoding.io/contact
This is the most complicated subject of Python I’ve ever seen. Thank you for your work. #pycharm #clion
one of my favorite things about python is how modular the language itself is. You can just overload anything you want, and having the syntax be so malleable is so fun!!
"the type of big A is a type but the type of small a is big A, whereby the type of type is type"
That's exactly the feature you will probably not use for a long time, but years from now you will remember and it will save your life!
Best thing about python, it's not like other languages where many base features are hard-coded, but you can customize EVERYTHING
#pycharm
its not always a good feature
@@georgplaz yeah, too much customisation just gets everyone confused, you cant replace every descriptive function name with some __magic__ and have people (and yourself) still understand it.
This is actually that rare gem content found under the depths of the internet. Been trying to wrap around meta-classes for a week now, with so little luck. The docs/content revolving around such concepts are so scarce, But you made it so clear as day. Thank you :) -don't need pycharm cause VSC is nicer lol-
Very clear explanation. After over 10 years of python programming I still learn new tricks from all of your videos. Thanks!
Great to hear!
So you were trying to overload a function with the wrong syntax and didn't work, the you used the correct way to do it and worked.
Mind bending.
This is a terrible idea, super slow, and very very dangerous.
I absolutely love it.
You had me in the first half
I've not used python in years, and primarily use TypeScript myself currently.
I had an alright time using Python when I did, but didn't think too much of the language, but your videos including this one really make me appreciate python more!
This whole system shown in the video is very elegant imo, and very powerful.
I feel like your videos have given me a really deep understanding and appreciation of all the internal magic that goes on behind the scenes on Python. Thanks for the content!
David Beazley, as far as I remember, advocates the use of class decorators (where possible) instead of metaclasses. The decorators are easier to handle, easily composable and easier to understand. But sometimes... yes, one has to take the bull by the horns and dive into metaclasses (not for the faint of heart). There's no wizardry at play here but one has to go through a lot of theory and practical examples to really start to see (appreciate?) the structure and intent of the creators of the language. Thank you for the video and spreading the knowledge.
It's interesting to see overloading implemented in Python, I saw a stackoverflow comment suggesting to use multiple dispatch instead and now I might see why considering the slowdowns. Great video as always :) #pycharm
7:33 This is how the enum module is implemented -- you create enum classes by subclassing base classes that have a special metaclass attached.
I taught Python for 2 years at the university level. I thought I knew more deep wizardry than any person should know. After seeing this, it's clear that I know nothing.
#pycharm
Same here, my friends call me when they have a question about Python because they know I love it and know it quite well, but when they call me a "Python master" I humble myself by remembering this channel exists :')
Very interesting information there. I would think, though, that most people have gotten around not having overloads built-into standard Python for so long, that normal Python paradigms suffice for all their coding. Overloading is nice to have, but I think it's more of syntactic sugar to other possible syntaxes. Somehow I can't help but think about the Zen of "Explicit is better than Implicit" might apply here as well, especially when usage is coupled with non-keywords args.
Thank you for the wonderful videos, and I must admit this is my first comment on your channel, but I follow your videos constantly. #pycharm
Just some feedback... the constant cuts saves viewing time but also impedes comprehension. It takes you time to talk about these ideas and likewise it takes time to comprehend them.
I've definitely learned more from your channel than any of the other python channels, and courses I've taken combined. Please keep making such great content. #clion
Well. That was... mind-bending indeed! Not so much the usage (that was pretty straightforward) but all the details and digressions in the implementation.
I always follow along with these videos in my python interpreter, so I can pause the video whenever I'm not certain about something, and play around a bit until I'm sure I've got it, then resume the video. Normally that only adds a few minutes here and there, but this video took me -- I kid you not -- OVER THREE HOURS to get through due to all the times I had to pause it and work out what was going on, not to mention all the scrubbing back and forth to refer to an earlier part of the video, unwinding the code as I went, to understand exactly what was being passed around in a later part.
ANYWAY.
I think I mostly have it straight now, with one glaring exception:
At around 11:23 on line 64, you throw 𝚔𝚎𝚢: 𝙾𝚟𝚎𝚛𝚕𝚘𝚊𝚍(𝚟𝚊𝚕) into the 𝘰𝘷𝘦𝘳𝘭𝘰𝘢𝘥𝘦𝘥_𝘯𝘢𝘮𝘦𝘴𝘱𝘢𝘤𝘦 dictionary. But I don't understand how the instance of Overload gets its __𝘨𝘦𝘵__() method called when it's an element of the dictionary? I would understand if it were an attribute of some class object 𝘢. Which is to say: let's assume we have an instance 𝘰𝘷 of 𝘖𝘷𝘦𝘳𝘭𝘰𝘢𝘥. Then if 𝘰𝘷 were an element of 𝘢, accessing 𝚊.𝚘𝚟 would for sure trigger a call to 𝘢.𝘰𝘷.__𝘨𝘦𝘵__(). But if you have 𝘰𝘷 as an element of a dictionary (let's call it 𝘥) which is in 𝘢, then accessing 𝚊.𝚍['𝚘𝚟'] would presumably only ever trigger a call to 𝘢.𝘥.__𝘨𝘦𝘵__()? So how does 𝘢.𝘥['𝘰𝘷'].__𝘨𝘦𝘵__() ever get called?
I'm guessing that it's some magic behind the implementation of 𝘵𝘺𝘱𝘦.__𝘯𝘦𝘸__() ? Do elements of the namespace being passed to that function get transformed into attributes of the class being created? Or am I missing some other intuition here?
Other than that, well... I am... enlightened!
Wow! I'm so happy to hear that you follow my vids to closely, this is surely the best way to learn! Indeed it is some magic behind type's new method. type.__new__(mcs, name, bases, namespace) basically eats the namespace argument and turns a wrapped copy of it into the new class's __dict__. So when I pass the overloaded_namespace in that spot, the overloaded_namespace essentially becomes the class dictionary for the newly created class! So you guessed exactly right!
@@mCoding Aha! Awesome. Thanks for the confirmation!
James, every video I watch you blow my mind with what is possible in Python, I don’t know yet what I’ll do with all this new knowledge but dang is it fascinating! Thanks for the quality content #pycharm
Awesome video! It's a bit more advanced than usual, but it's good to hear something like that once in a while!
Damn, I've got to appreciate how flexible Python is. This is face meltingly awesome.
Thanks for the kind words!
Every time I think I am quite knowledgeable in python a new video is posted and I realize how few features of python I actually use
Now, that we've learnt about this from this fantastic video, probably it's time for the designers of the language to learn, from the very same video, that the most basic feature of am OO language should not be paid in blood.
Haha you're welcome to suggest but I think the designers are mostly against checking types at runtime. Also they will tell you python is a multiparadigm language, not just OO!
I'm really happy that you are making speed comparisons. This is something I often do with python, to figure out what works best, but I don't always understand why. #clion
For once Python is making me appreciate the simplicity of C++ and not the other way around #clion
don't worry, we'll get reflections in c++ soon(tm)
@@kurtmayer2041 i need that
This is so much simpler than anything C++ reflection ever will be. At least this is consistent lol
simplicity... of C++? in what world is it true lol
You must've just started learning C++.
Thanks sir, your content is always enlightening. Frankly, it sometimes stupefies me how much there is to learn about a language that many of my friends, and my computer science teacher, dismiss as simplistic... great content as always!
#pycharm
Glad to hear that! It never ends, I learn many new things every day! Sometimes in the process of recording :)
Seriously, if this language had tricks up its sleeves to do a lot of this at compile-time (like Zig), it would've been the most powerful language across the domains.
I mean seriously, so easy to pick up and get started, yet so accessible to dive deep and get lost into the wild wild technicalities.
@@comradepeter87 You can look into JIT compiling solutions for Python like numba (library) or PyPI (interpreter) for example, it lets the code be compiled at runtime (which slows down the first function call, but generally makes the next ones way faster, catching up to the realm of fully compiled languages)
@@matthieurochette Yeah numba is awesome!! Relatively simple to use as well
This is really fascinating stuff James! I thought I was pretty experienced with Python, but you are helping me take my understanding to another level. Many thanks for sharing your wisdom.
Hi Wizard, thank you very much for your kind words! I'm glad you enjoyed and glad to have your viewership!
"a class defn is actually just syntactic sugar for calling a type constructor"
There's a tiny caveat here, like being able to pass kwargs to the type definition's bases, which can't be done with `type`. To be able to do that you'd have to use `__build_class__()` instead, so in python3 (i believe starting from py3.1) it's actually sugar for the build class top level function instead.
It's nothing major though, but I think if you're making an absolute statement it should be correct.
I've faced with problems trying to implement code at 14:30. The key is pylance can't recognize arguments for dynamically generated `__init__()` method and keep warning me. Are there any ways to fix this? I implemented this in order to get autocomplete on __init__ arguments but I'm wondering now if it's even possible.
In every large code base that I've worked in, a few people have tried out obscure-but-extremely-cool language tricks. Most, but not all, of those cases resulted in long discussions with the people who have to maintain the code, test the code, see the code and want to know about it just out of curiosity, or worst of all, were tracking down a difficult bug and wondered if this obscure code they don't understand could somehow be the culprit. If you take into account the cost of those discussions, it's very rare for the trick to provide a clear net benefit. It's happened, but it's very rare. In a large code base, the highest priority is to make the code readable by others. Please remember that before you use a feature that most others do not know.
I totally agree, I'm often that person tracking down why a framework needed to change everything to save a few characters. But I do love exploring the limits of what is possible.
Yet another banger, I didn't even know this existed to be honest! Really interesting watch.
As for the giveaway I 'm a student so I get their stuff for free, but it's actually incredible you got that opportunity from Jetbrains! Congrats on that.
this is heresies, black wizardry and demonology combined
Every use of this forbidden knowledge needs to be marked with
# Here be dragons!
Your videos are very useful. Short and to the point and ofter cover subjects you may not encounter when using the language even professionally. Thank you!
Another little suggestion: you generally shouldn't try to subclass `dict` because it has special cases in the interpreter, you should extend `collections.UserDict` instead.
Damn. The type of video to really smack you in the face, when you think you know programming.
Do more, James, your content is fantastic!
#pycharm
I think the good part is that there are so many concepts you touch that I didn't even know in Python. I had to stop several times to look the documentation and learned a lot! Thanks! #pycharm #clion
This is extremely interesting. I tried to learn about metaclasses about two years ago, but I didn't understand it - until now. Excellent explanation.
Great to hear!
Great video! It was very insightful. We definitely need more videos like this with such deep-level insights.
Excellent video. It took me some time to understand it, but it is pretty genius what you do. I saved this video because of it being so valuable...
I see that your example implementation that allows function overloading, indeed causes a big amount of overhead. Maybe its an idea to raise a PEP for getting a real solution that allows us to overload functions in future version of python. Because honestly, I think it is a very worthwile addition to python.
Python gets more intricate and beautiful the more I use it. (and learn from your videos)
This was actually quite interesting. I didn't know function overloading was even possible in python! Very cool.
#python
Being a beginner with Python I'm quite happy I managed to follow what was going on here. Even though the approach discussed here has various negatives the content as a whole was quite informative. Thank you. #pycharm
a perhaps more efficient way to do overloading is via monomorphisation
instead of having one key having a list of possible values that you then need to search through, you instead create a new name for each method based on its argument list and create a method called something like "f_int" and "f_string" and "f_any_any" as the names
doing this lets you take advantage of the dictionary's fast lookup without being also limited by the list's slow lookup
No, function overloading improves readability and avoids remembering too much function. Instead of having multiple "f_int" and "f_string" and "f_any_any". It is better to only have 1 "f" function to handle all arguments.
Fabulous! Just what I needed to understand this shiny yet murky realm of metaclasses.
hold on, slow down a bit.. what do i type again? class?
and yet again, I feel like my python programming skill has leveled up. #pycharm
Some more C/C++ content would be great.
Really neat! Just a question: why not use a dict of signature hashes resolve the best_match? I feel like it would be faster than iterating over the list of signatures.
I probably will never use Metaclasses personally, but it's great to know that they're there and what they do!
#pycharm #clion
Imagine forcing the interpreter to do all of that on the fly
Drinking game: you take a shot every time James says "class"
I feel like this channel both teaches you advance python and the foundations you didn't know you don't know. Thanks! #pycharn
Yeah, metaclasses are super powerful but also something you should think hard about as to whether they're the right tool for the job. For the purpose of function overloads, I'd probably even say they're not the correct choice, and would just be better just as a property-like decorator where you do @overload for the first definition and then @f.overload for the subsequent definitions. And of course there's the question of whether having overloads in python code is even good
This is called polymorphism in OOP and also I’ve seen a technique of overloading in Python before, can’t remember for certain now, but I think it’s in one of Corey M Schafer videos
Thank you for your dedicated work, James Murphy. This video proves it once again. #pycharm
My pleasure!
I'd love to see a short followup to this concept using the new Python Match syntax, I don't imagine it drastically changing performance, but I could imagine it allowing for a much cleaner code base.
James Powell had a great talk at some PyData which included metaclasses though your introduction is much more practical. Nice one! #pycharm
@mCoding would a set be a better structure than a list to keep "seen" parameters? In a list you'd run the risk to overload more than once with the same arguments.
Unless you want to combine overload and duck typing? 🤯
I was quite lost in this video. What are the pre-requisites to understanding it? Can you point me to other videos which would build my knowledge up to understanding the concepts in the video? Thanks.
Never thought we could define two functions with same name but receive differents parameters, like I was used in C++. That's very nice! #pycharm
just saying, you can (and i have) implement run-time overloads with just function decorators (which makes it also work on module-level functions)
Would love to see your implementation!
Is method overloading something that should be done in practice in Python? It seems like this is a way to do it but because it can't be done directly, I wonder if the intention was to not allow for overloading methods.
Definitely not something that should be done in practice. It essentially involves decision-making based off the type of an object, which goes against Python's duck typing philosophy.
Oh wow I didn't know you could do actual-for-real code generation with python metaclasses. I do a lot of multi-language development, and that often means needing to use a common configuration file between all the different languages and platforms to perform language-specific code generation, just to make sure that all of the implementations stay in sync and we don't introduce bugs where one language treats some byte in a packet different from another language. If I can just read the config and do the code generation at _import time_ in python, that'll cut out the need to add a whole separate build step for every change and make iterating so much faster.
Okay, okay. I’ll bite: #pycharm #clion
Thanks for another great video. My brain did a double-take at the start, I was questioning myself “wait no, Python doesn’t have function overloading…”
mCoding: “Python doesn’t have function overloading.”
Me: “oh thank god”
mCoding: “Except when it does.”
At least a clear and also deep explanation of metaclass mechanism and some uses. Not sure I will ever use it since changing much class behaviour makes code difficult to maintain. Thanks man. #pycharm which is my favorite IDE for python.
Amazing. I’ll definitely be needing to watch this vid again a few times to understand it, but it’s truly mind blowing what you can do with metaclasses. #pycharm
Even after watching this video, metaclasses still seem magical. There's no end to what you can do with them. #pycharm
I have trouble understanding descriptors and metaclasses - perhaps the two constructs underlying Python's various features.
WoW !!! Metaclass are overwhelming. But they are quite amazing. And really like your efforts in teaching these concepts to us. This is the most interesting RUclips channel out of all my subscribers #pycharm
Great video, i had no idea these aspects where so customizable!
how can i understand pthon the way you do?
So... meta-modules when?
Well that's the first video that I watch on your channel that left me puzzled yet excited to watch it again #pycharm
I'm sure you are in good company, metaclasses are probably the most difficult thing to master in Python. Master metaclasses and you can master the whole language!
Loved the part where you showed what's behind the curtain of some standard libraries. You should make a whole series of this. 👏🏻
#pycharm
What's the "pass"? is that because white space/indentation is significant in Python?
Yes, pass it used where syntactically something is required but you want to say "do nothing". It is typically used in functions, classes, or if clauses that aren't supposed to do anything.
Glad you noted the performance hit there; runtime reflection can cause slowness in other languages too (e.g. C#). Things like Rust macros or C# source generators can move this work to compile time (which has it's limits too), but I'm not sure there's any way you could do that in Python.
`eval` and `exec` are there to do such things. (Even though, technically, the compilation will be at runtime for it, but at least this is relatively quick, much faster than introspection, because Python is good and fast at parsing and compiling Python).
If you need it quicker (bypassing the needed compilation at runtime), you can also generate/template the compiled bytecode, that's possible, too and if the templating is fast (it should be usually), then you have close to no overhead. But of course, you need to be relatively good in understand Python Bytecode and working with the disassembler to get there. (Tbh, I only know theoretical that it works from other author's articles, but never implemented it on my own).
And of course, you still can take the same approach as C/C++ and write a code replacement and generation engine with all the same caveats as in C/C++. In some areas, this is even regarded as a good practice, e.g. when working with protobuf. The biggest advantage is, that it allows to have an easy IDE integration, as the IDE can inspect statically what's going on. Otherwise, the IDE needs internal support (that's what's happen in PyCharm for Django as an example where also a lot of code is generated dynamically - there also mainly for performance reasons over any introspection/reflection injection)
Is it old-fashioned to implement the multimethods module to invoke function overloading in Python? Which is the more convenient method of overloading functions in Python?
Your videos are so impressive, very good explanation and really clear examples - for real you are a cut above the rest in this space and I'm glad your channel is growing so quickly!
#pycharm
10:48 „owning the dot“ - I consider descriptors rather a way of owning the *equals sign*
If this concept were incorporated into the python core language (and thus implemented in C), would it be fast enough to be worth it, or still too slow?
#pycharm #clion
It definitely could be made fast enough if it was implemented natively, but the idea of a feature that does runtime type checking would probably never be voted into the core part of the language.
I remember stumbling upon this concept once and wrote some magical shit that even I couldn't parse after the fact, but it worked like a charm so we went with it... If I could track down the code I bet I could understand it better now, thanks for the deep dive! On a side note, abstract base classes are one of my faves, love making use of them #pycharm
4:20 I think the documentation suggests that you can create metaclasses that do not inherit from type. But I haven’t been able to get that to work.
#Python
Question
This is getting very close to what I am attempting to do using a Groupby with a Lambda Series for multiple function calls and appending the results to a growing List if it rings True from the Function. Would you suggest this is doing essentially the same thing ?
Do you have a deep dive on Groupby with Custom Lambda Function with a Series ?
Thanks for any references that supports this effort.....
This is really cool, I use function/method overloading all the time in Java. But as with everything, if the others reading your code don't know what's going on, you're probably going to have issues down the road :/
But your videos are excellent, really delving into what's going on in the internals.
learning these bits of lower-level mechanisms in Python lets me understand how Python magic works, especially for those built-in decorators like property. #pycharm #clion.
Seems like the "new" structural pattern matching would be the perfect match (pun unintended) to implement python function overloading
Been using abc for awhile, and it's super interesting to know how it operates under the hood. #clion
This stuff is super cool. That function overloading still looks like black magic after you explained it.
#pycharm
now everytime someone says, learning python is easy, I will forward this video 🙂
6:58 "__call__ is called whenever you try to make an instance of the class"
Isn't __call__ usually called when an instance of the class is called, not the class itself? Is it different for classes that inherit from type?
Dunder call is called when you put parens after the object. If your object is a type, "calling" the class object semantically means making an instance of that type.
Metaprogramming and Reflection might be a little over my head, but this is way easier to read, than other methods for python overloading even if it does cost some time.
#pycharm
But what about singledispatch overload?
This was amazing! I was just wondering if there was a way to do this in python. I didn’t even know metaclasses existed before now. Thank you for the very detailed explanation! Keep it up! #pycharm
I just come to know about overloading 😮... Btw what could be the usecase of overload?
#pycharm