In this case it was the same as a BFS, but in general it does not have to be! See the L(Z) example in en.wikipedia.org/wiki/C3_linearization for a case where it is not a BFS.
Technically, it does a depth-first, left-right search and can contain duplicates. Then, it removes all but one of the duplicates. The last duplicate is preserved, all else are removed. Eg: class A: pass class B(A): pass class C(A): pass class D(B, C): pass so depth-first, left-right search list for 'D' would be something like this: ['D', 'B', 'A', 'object', 'C', 'A', 'object'] Now, all earlier duplicates will be removed for the __mro__, i.e the first 'A', and the first 'object' will be removed from list. ['D', 'B', 'A'(first 'A', hence removed), 'object'(first 'object', hence removed), 'C', 'A', 'object'] the final list is this: ['D', 'B', 'C', 'A', 'object']
i spent days treading in convoluted documentation to finally understand MRO, multiple inheritance, and super (had to also get through descriptors to understand the bounding mechanics)... and here it is, a 20 minute video that explains everything. this channel is way too underrated.
Thanks for the kind words! And I also recommend "Super is considered super"! It's linked and Hettinger was (I believe) the one who coined the "next in line" phrase.
@@mCoding If anything, it's worth watching for the volunteer part (explaining cooperative inheritance and MRO) at very least. Visuals like that make things a lot easier to comprehend/memorize. I'm not a fan of some of Raymond's teaching techniques but this specific part was brilliant.
With that experience maybe you could make the documentation better? I’m not trying to be one of the “then open a pull request!” guys but it sounds like you have special insight that might help future users.
The *level* of this channel is 👌. I've been coding in Python for quite awhile and I always learn stuff here. It's really refreshing when a lot of what is out there is like "how to use dicts in Python!"
So true, it's high level, but it doesn't use an overwhelming amount of jargon. Learning a new programming language is a pain, because 90% of the material and tutorials are "This is the syntax for a for loop", while the remaining 10% is an argument on stack exchange about the best boosting the efficiency of a highly specific package from 1994 with some compiler meta-optimization. It's either teaching a complete beginner how to code for the first time, or a set of code-words that will remind an expert of how to do something. When I almost always want is help on turning my ok-ish code, into good code. I wish he did material on, say, C# or Julia, so I could go from a basic understanding of the surface feature of the language to having an idea of how the structure of it works.
This is why multiple inheritance, especially "diamond" multiple inheritance is evil. Even if you're one of the 0.01% who knows how it works, it will still trip you up when you dont expect it (that is, 99% of the time). Having recently come across C#'s solution to this (a class can only inherit from 1 class, but can inherit from as many Interfaces as it wants), I now think python's approach is too liberal. Of course that's the whole python ethos, but sometimes it creates far more traps than necessary. #pycharm
Here the trap isn't that problematic imo, once you inherit from multiple classes there is an obvious problem with super. Best practise I think is to specify which parent function you want to call
So you've discovered interfaces. My favorite C# feature. Initially they seem hard but once you get the hang of it you'll never go back to MI. When you get a bit deeper you'll hit covariance / contravariance with interfaces, and the slightly ugly type parameter constraints that all kind of tie together. C#'s type system is not quite perfection.
James: your expert knowledge of not just Python, but C, and CPython implementation - along with dense and info-rich presentation - make for a quite rare resource of deep, under-the-hood understanding of Python. Thanks for putting in the work!
Thanks for being so clean and straight-to-the-point in how you present your topics. Your videos are quite like 3Blue1Brown but for programming. Keep up the dry humor, stellar stuff. #pycharm
The only time I used multiple inheritance it was a big mistake, it became impossible to have the classes cooperate nicely. A slightly more verbose solution based on composition solved the same problem without all the subtle bugs #pycharm
@@mCoding Could you please make a video where you show with a few different examples to replace a solution with inheritance with functional composition?
That's what always happens with multiple inheritance in python. It's a magical feature: the instant you implement it, the code becomes unmaintainable. Not at some future date, not after some drift, etc, no: the very next thing you try to change will be extremely difficult. It's truly impressive how bad it is.
Even though I am aware about using super and cooperative multiple inheritance when I watched the iconic talk of Mr. Raymond Hettinger on super()... But even then, my mind was simply in awe when you dived into the details... I never even imagined that super class is sort of implementing a Proxy pattern...
In current example it is the right form of using dunder getattr method. Since you don't want to get the attribute of an instance of this class but the self.obj, that will make the code work excatly as it should. Changing dunder getattr to getattribute will cause the RecusrionError because it will invoke itself infenetily long, as you're trying to get the attribute not of self.obj but of self - an instance of current class.
A good addition for cooperative inheritance is that the root class' sink method verifies that both *args and **kwargs are empty at that point and throws an error if either isn't. This helps prevent unnecessary arguments being passed somewhere in the chain that stick around after changes/refactoring as the *args and **kwargs in the signatures of children mean that static code analysis often won't see anything wrong.
Very nice! Your example code at the end is the part that was missing in PEP 3135 back when it was written, it would have cleared up how super works for most people. For a good example, look at PEP 343 - with Statement, where the "Specification" section shows exactly how it works in a very clear fashion.
IIRC, you can rebind super to have the compiler magic keep working while also allowing you to change its behaviour (it's definitely the case that using super bound to another name causes the compiler magic to fail) EDIT: To be clear, if you had written `class super:` instead of `class Super:` it would be a drop-in replacement
Another great video, the first half was satisfyingly confirming some practices I have picked up by using python for a while, and the latter half was part mind-boggling, part "oh please don't do that". Learned a lot as always, thank you James! #pycharm
I remember that I watched Raymond Hettinger's "Super considered super!", understood a little more about the MRO but could not understand how `super()` works. Now I have watched this video and I have learnt something new today! Thank you! I still don't fully understand the `super(...)` with argument(s) form, but I guess/hope I don't need it since I don't need to consider my job security. 😂
Those theoretical questions were bait-and-switch tricks. It would have been less confusing if you had shown all the information before asking the question rather than hiding the most important part that makes it possible to accurately assess the question.
To be fair, it's python, which is about convenience over speed. If you want method and property lookups to be fast, you should probably be using a different kind of language
@@berylliosis5250 Considering that attribute/method lookups are one of the most common things you do in a programming language, it's quite frightening that what takes one machine instruction in a compiled language takes upwards of 100 *bytecode* instructions, including both branches and memory fetches. I hope (and think, but am too lazy now to check) that Python does some optimizations here to bypass this cost because I've never really felt that Python is that slow in comparison. I do expect a certain level of speed from any language.
Side effect of this video: seeing a weird / operator in the __init__ of your custom Super implementation, looking it up and learning about positional only arguments. Loved this video!
I would love to see an explanation of the Walrus (:=) operator. How it works,tips and tricks, and use case scenarios. Mostly I've missunderstood when i should/can use it. My code often feels non-reable when using it. Loved this video (and ofc your previous ones)! #pycharm
Had to deal with a MRO nightmare I was building Custom Operators for an Apache Airflow application. Custom Operators basically inherit from other default Operators (to build custom solutions for your specific use-case), which in turn can inherit other operators and extra classes and so on until it gets to the BaseOperator class. To make matters worse, I wanted to merge 2 Operators into one. Took me a long time to figure out what super() was actually calling and made me have to dig into a lot of stuff to find out. Your video is really straightforward and had it been released 1 year ago would have saved a couple of days of frustration. Awesome content as always! #pycharm
@@drygordspellweaver8761 I needed a simple way for the Data Analysts and Data Scientists to have their own self-serving Airflow instance. Since they already knew how to use an Operator, I built that way so I could "hide the wiring" behind that complicated process behind an interface in which was more familiar to them.
@@josephlyons3393 At least we don't have a whiny compiler. :) Just kidding, I like the rust's approach. Cppckeck for cpp is sometimes all what you need tho.
@@embeddor3023Rustc certainly is soul crushing, lol. C++ was my main language for many years, but I mostly dev in either Rust or Python these days, depending on the task.
I find it quite enlightening when you dive deeper into how these features work rather than working with the superficial layer of when to use/ how to use these features. I always come away with some useful knowledge I can apply to my code. #pycharm
Finally, a video that covers the inner details of super and how it works, also the pure python implementation of super is awesome, learnt a lot from this video, thank you so much Murphy. #pycharm
Well, tbh. in most cases it's not a big deal because you can compose functions pretty easily by explicitly calling them. You don't need "object oriented" programming (more like inheritance) in these situations.
It seems like it, but it's actually far less complicated _in practice_ than in many other languages, like C++ (public vs private inheritance, recursive inheritance [CRTP], pure abstract classes, constness concerns, public/protected/private members, etc. -- and that's before we add templating!). In both cases, though, you should really avoid inheritance for code reuse, anyway. Inheritance should be used for polymorphism. Only in some rare cases when following certain design patterns (or idioms) should you inherit anything but the interface, and in those cases, you should just follow the pattern. Generic programming is usually closer to what you want with code reuse; Python has this with duck typing, while languages like C++ do this with templates, and some others like Java fake it (somewhat poorly) with type erasure.
There is one valid use case for the two and one argument form of super. I am a bit surprised you did not mention them. As you pointed out the first argument tells you where to look for the next in line in the MRO. If you pass in one of the parents explicitly you can ignore part of the MRO and look for methods or attributes starting from a fixed parent / sibling. Not that you need it all the time but it can be handy.
Note to self: never use inheritance in Python if I want my code to run fast. Seeing what goes on behind the scenes is amazing, and a bit scary... the VM must be doing *so* much work every time a super() gets called XD
after like 8 years of programming I've come to the conclusion to use inheritance as little as possible it just creates a mess in places you never think of
This video again shows how Python's compiler does magic things. And again emphasizes how it's very important to understand when things happen (in the compiler or in the interpreter).
It is impressive, how Python implements all these functionality _with Python_ - from a technological point of view. Python, as an interpreter language, implements itself by using Python... fascinating! But from the perspective of a user of the language ... all this black magic everywhere ... meh. :/ In other languages, using "clever coding" is regarded as something bad ...
I'm assuming that super is implemented in C (at least for cpython) and not in pure python since it's a built-in. But it could be implemented in python, aside from the class injection hack.
Compared to most other languages, python has very little black magic. Other languages also have things like implicit `this` variable, which feels more like black magic to me
I think this is a case of "practicality beats purity" :D I remember programming in Python 2, which didn't have this black magic. This meant that every. single. super call needed to be the two-argument form... Believe me, it gets old very quickly, apart from being error-prone, as you now have to mention the class you're in everywhere.
It's always interesting to see in-depth explanations of how a language works under the hood, like this or the ",=" python operator you explained in a past video. Keep up the quality content :) #pycharm
I started using super() due to the structure of a project (needed a parent class with some general methods and configs, and child classes made to inherit from it). I was exposed to it from a previous video from you, where you used to create a subclass dictionary to be able to call the subclass given the parameters ( this is something i ended up needing in said project). I already knew some of the stuff you expose here due to researching, but as always, you go the extra mile. Thanks for your content. #pycharm
Multiple inheritance needs a carefully designed class hierarchy. As much as possible, structure your Python classes to emulate the functionality of the single inheritance + interfaces/mix-ins model. In particular, only the principal parent class of a class should descend from anything other than object, thus mitigating the diamond problem. But also consider composition may be a better option for secondary parents.
I'm a data scientist/engineer/Web developer and I've made use of your videos in my work many times. Your channel is one of my training links I give out to new staff. Definitely would appreciate a Prof pycharm license for personal use #pycharm
I love both PyCharm and CLion (the only editor I can not only stand for C++ but that I actually enjoy). I seldom use super in my projects (personal and work), but this was a good explanation.
I am glad I came across your channel. Just realised I've been more focused on the 'how' things work in python but didn't give much thought to the 'why'...good job mate. #pycharm
hey, i just came up to that video an i really love it!! although, i think super and mro are far from the best python feature, and i made an alternative to super for fun. I think it's weird to assume it's possible to order multiple inheritence trees, it's weird the target of super is so implicit, and depends on context, it's weird that child class affect the super behavior in its parent, it feels like a side effect, and it's weird that mro can fail, and makes some class definition impossible. the alternative i produced for super (i called it __as_parent__) follow those rules : 1) it should reliably target the parent class it targets, no matter what other child class you decide to create later on 2) in case of multiple inheritence, it should be passed the parent it targets as argument (you can then make one call per parent method you wanna refer) 3) it should be able to stay implicit when it doesn't need to be explicit, no need to specify the parent when there's only one 4) it should be able to target ancestors, direct or not (it's needed to make it work with the alternative to mro i produce, in some weird cases) the alternative i produced for mro (i call it explicit method resolution) follow those rules : 1) any method / attribute present in the class definition should be the one resolved to (other rules assume the method / attribute is no present in the class definition) 2) in case a method / attribute is not present in the class definition and in any ancestors definition, it should raise an AttributeError 3) in case only one parent can resolve it, it should be the one resolved to 4) in case multiple parents can resolve it, it should raise an ExplicitResolutionRequired error (i'll explain how to solve it later) 5) in case one parent raises an ExplicitResolutionResquired error, this error is transmitted to the child 6) in case multiple parent can source it from on single source (diamond shaped inheritence), the ExplicitResolutionRequired error is not needed To solve an ExplicitResolutionRequired error, you can just define the method / attribute in the child class definition. (method can then refer to parents method through the use of super or my __as_parent__ alternative) Do you think it make sense? here's the github repository : github.com/malmiteria/super-alternative-to-super
I love the idea and huge props for implementing it! That's the blessing and the curse of such a dynamic language. Don't like something? You can completely write your own.
I predicted that super uses __locals__ and type() but the Proxy concept was the missing piece. Very enlightening. I learned about proxy objects last yearish but haven't intentionally used the pattern in my own software yet, so it doesn't come to mind immediately as a solution. Gotta find an excuse to apply the concept in my own code sometime so the idea sticks
Yeah proxy objects are one of those things that make a lot of python magic happen. They can be useful, but they can also be confusing for users sometimes.
absolute knowledge.... , though most of these i knew olredy, but you are actually going gr8 sir. how to get which class is being run by super , i did not know...thanks for that and really appreciable work sir, all hands down.
I looked for this A LOT through last week You’re a mind reader and your content is amazing! Thank you so much for helping the community to understand these concepts better! #pycharm
Your videos are always very informative and straight to the point, unlike other tutorials, where every character of code is analyzed on it's own. Keep up with these good videos #pycharm
This channel has really helped me level up my Python skills. Videos like this are really helpful for people like me who want an “under the hood” understanding of the language. #pycharm
the comments at 3:00 seems wrong, it seems that it does go to A's f but A's f IS Root's f. So it does go to A's f in that case, which is the same f as Root's f.
Oh this is perfect, i was just struggling with this like an hour ago. I had some dataclass inheritance structure (frozen) and i had some classmethod as a factory which i wanted to inherit but it all fell apart.
I did use it once, it frelled up my code... but what it did for me was to prove that I should not have used classes for that problem in the first place. The need for "super" is a super-sized warning flag that the design is absolutely flawed.
Personally, for those very few times it _appears_ ambiguous which defining class the `super` will go to, I either use the super call with arguments, or just outright call `superclass_name.function_name(self)` explicitly. That way I don't have to require the reader to untangle the MRO algorithm. The downside of course is that now the super classes have to be named multiple places. The class's definition and any place I need to use this trick. But I consider that worth not requiring knowledge of the "deep magics" of Python
10:05 I think there's a potential problem here with using a mix of explicit and packed arguments. The explicit arguments will be absent from the packed arguments, so then would also be absent from the super call. Eg if the next sibling class also needs the argument `validators`, it won't get it because here `kwargs` doesn't contain `validators`. If, however, you preserve all the arguments in the super call, then there may instead be a problem if the base class doesn't accept the extra arguments.
super in python is weird, but thanks for this explanation. When I saw the super before I opened the video I thought that the video would be some kind OOP bullshit that people doesn't get right, but boy, oh boy, how wrong I was. Thanks the amazing video. And lesson learned, never assume anything and never underestimate stuff.
Bit of a different video idea suggestion! I understand you prefer these in-depth videos for important python features, but how about a video on: All the python class dunders? Not in detail, but as a quick: 'here's a use for them' Even after 8 years of python, I still only just learnt the __setitem__ and __getitem__ dunders last week, yet there's still quite a few more I have never used, and don't know when to use.
Interesting and insightful as usual. What's really surprised me the part where you unraveled the mystery behind how zero argument super works, which also explains the older form of super. Thanks for your content! #clion
Nice video, though you didn't fully specify the MRO search algorithm (C3 linearization). I fully agree with Mark Lutz (Learning Python) on the concepts of "Cooperative Inheritance": best not to do it, since any programmer using a library you write using cooperative inheritance can break your code by simply adding a new mis-behaving class in the wrong level in your hierarchy.
In my opinion, if you need to think about MRO, then it's time to refactor your code. I use inheritance only when I create interface. Composition is the answer most of the time...
In your own super implementation in the init method I see you do: `if cls is None and obj_or_cls is None` I ilke to use: `if foo is bar is None` or `if None is foo is bar` or `if foo is None is bar`. Personally, I think it's less bulky to read.
@@mCoding Oh okay, just watched that video and in it you actually present a `if x is y is z` and you put in in the ok-questionable range, but you also said that the only reason you didn't put it in good use cases was because "most people probably aren't aware that `is` [...] is a comparison operator". However I very much agree about the bad examples you showed in that video. That's hella confusing. Also at the end of your video you are saying that you don't want to get rid of chained comparisons, only those that inherently don't make sense. In the end you're even saying that you'd like "to see an expansion of chained comparisons to other cases that do make sense", after which you'll talk about how on numpy `xs[3 < xs < 5]` didn't work and that you'd like to see it there. All in all from the video it isn't entirely clear that you dislike chained comparisons, only that you dislike confusing chained comparisons (i.e. comparisons with opposing operators, f.e. ``). EDIT: Actually you do say that one probably shouldn't use this except if one does a very simple comparison, however as a simple comparison you name `x
![Lethal Company Music Video 🎵 "Follow The Orders" [VERSION A]](http://i.ytimg.com/vi/QeooZB4Z7RA/mqdefault.jpg)
Isn't the search that the MRO is doing the breadth-first search?
In this case it was the same as a BFS, but in general it does not have to be! See the L(Z) example in en.wikipedia.org/wiki/C3_linearization for a case where it is not a BFS.
@@mCoding my assumptions keep breaking and I love it.
Technically, it does a depth-first, left-right search and can contain duplicates. Then, it removes all but one of the duplicates. The last duplicate is preserved, all else are removed.
Eg:
class A: pass
class B(A): pass
class C(A): pass
class D(B, C): pass
so depth-first, left-right search list for 'D' would be something like this:
['D', 'B', 'A', 'object', 'C', 'A', 'object']
Now, all earlier duplicates will be removed for the __mro__, i.e the first 'A', and the first 'object' will be removed from list.
['D', 'B', 'A'(first 'A', hence removed), 'object'(first 'object', hence removed), 'C', 'A', 'object']
the final list is this:
['D', 'B', 'C', 'A', 'object']
@@mCodingthat algorithm looks like an O(n^3) mess lol
i spent days treading in convoluted documentation to finally understand MRO, multiple inheritance, and super (had to also get through descriptors to understand the bounding mechanics)... and here it is, a 20 minute video that explains everything. this channel is way too underrated.
Raymond Hettinger's old talk "Super is considered super" is fairly good, too
Thanks for the kind words! And I also recommend "Super is considered super"! It's linked and Hettinger was (I believe) the one who coined the "next in line" phrase.
@@mCoding If anything, it's worth watching for the volunteer part (explaining cooperative inheritance and MRO) at very least. Visuals like that make things a lot easier to comprehend/memorize. I'm not a fan of some of Raymond's teaching techniques but this specific part was brilliant.
With that experience maybe you could make the documentation better? I’m not trying to be one of the “then open a pull request!” guys but it sounds like you have special insight that might help future users.
I've been programming in Python professionally for almost 10 years and I learned something in this video.
Glad to hear!
Same!
does that mean these things are not going to be used in industry?
@@TemporaryForstudy not if you want to keep working in that industry ;-)
@@Ratimus_ Thanks for the information. Are you a python developer?
The *level* of this channel is 👌. I've been coding in Python for quite awhile and I always learn stuff here. It's really refreshing when a lot of what is out there is like "how to use dicts in Python!"
yeah that the problem, he is teaching super , to advanced people.
So true, it's high level, but it doesn't use an overwhelming amount of jargon. Learning a new programming language is a pain, because 90% of the material and tutorials are "This is the syntax for a for loop", while the remaining 10% is an argument on stack exchange about the best boosting the efficiency of a highly specific package from 1994 with some compiler meta-optimization. It's either teaching a complete beginner how to code for the first time, or a set of code-words that will remind an expert of how to do something. When I almost always want is help on turning my ok-ish code, into good code.
I wish he did material on, say, C# or Julia, so I could go from a basic understanding of the surface feature of the language to having an idea of how the structure of it works.
Exactly! And it's always some of the most error-prone structures that you can't understand!
Love your videos!
This is why multiple inheritance, especially "diamond" multiple inheritance is evil. Even if you're one of the 0.01% who knows how it works, it will still trip you up when you dont expect it (that is, 99% of the time). Having recently come across C#'s solution to this (a class can only inherit from 1 class, but can inherit from as many Interfaces as it wants), I now think python's approach is too liberal. Of course that's the whole python ethos, but sometimes it creates far more traps than necessary. #pycharm
Too many people take my job security comment a bit too seriously!
Here the trap isn't that problematic imo, once you inherit from multiple classes there is an obvious problem with super.
Best practise I think is to specify which parent function you want to call
So you've discovered interfaces. My favorite C# feature. Initially they seem hard but once you get the hang of it you'll never go back to MI. When you get a bit deeper you'll hit covariance / contravariance with interfaces, and the slightly ugly type parameter constraints that all kind of tie together. C#'s type system is not quite perfection.
That's precisely the case with Java as well.
Don't all classes derive from object, making any multiple inheritance diamond shaped?
James: your expert knowledge of not just Python, but C, and CPython implementation - along with dense and info-rich presentation - make for a quite rare resource of deep, under-the-hood understanding of Python. Thanks for putting in the work!
Thanks for being so clean and straight-to-the-point in how you present your topics. Your videos are quite like 3Blue1Brown but for programming. Keep up the dry humor, stellar stuff. #pycharm
The only time I used multiple inheritance it was a big mistake, it became impossible to have the classes cooperate nicely. A slightly more verbose solution based on composition solved the same problem without all the subtle bugs
#pycharm
As is always a question you should ask, just because you *can* solve the problem with multiple inheritance, *should* you?
@@mCoding I believe 99,99999% of the time the ansewer is sounding NO.
Rule 2 of OOP: Favor Object composition over inheritance
@@mCoding Could you please make a video where you show with a few different examples to replace a solution with inheritance with functional composition?
That's what always happens with multiple inheritance in python. It's a magical feature: the instant you implement it, the code becomes unmaintainable. Not at some future date, not after some drift, etc, no: the very next thing you try to change will be extremely difficult. It's truly impressive how bad it is.
Even though I am aware about using super and cooperative multiple inheritance when I watched the iconic talk of Mr. Raymond Hettinger on super()...
But even then, my mind was simply in awe when you dived into the details... I never even imagined that super class is sort of implementing a Proxy pattern...
Amazing insights into the super() function implementation in Python! Thank you for your great work, I always learn something new!
Glad it was helpful!
The 2 argument super used to be the required way of calling super in python2. So glad we don't need that anymore
May Python 2 rest in peace, forever!
13:18 little mistake. __getattr__ is called whenever getting attribute is failed, __getattribute__ is called everytime you trying to get an attribute.
In current example it is the right form of using dunder getattr method. Since you don't want to get the attribute of an instance of this class but the self.obj, that will make the code work excatly as it should. Changing dunder getattr to getattribute will cause the RecusrionError because it will invoke itself infenetily long, as you're trying to get the attribute not of self.obj but of self - an instance of current class.
A good addition for cooperative inheritance is that the root class' sink method verifies that both *args and **kwargs are empty at that point and throws an error if either isn't. This helps prevent unnecessary arguments being passed somewhere in the chain that stick around after changes/refactoring as the *args and **kwargs in the signatures of children mean that static code analysis often won't see anything wrong.
Very nice! Your example code at the end is the part that was missing in PEP 3135 back when it was written, it would have cleared up how super works for most people. For a good example, look at PEP 343 - with Statement, where the "Specification" section shows exactly how it works in a very clear fashion.
IIRC, you can rebind super to have the compiler magic keep working while also allowing you to change its behaviour (it's definitely the case that using super bound to another name causes the compiler magic to fail)
EDIT: To be clear, if you had written `class super:` instead of `class Super:` it would be a drop-in replacement
Another great video, the first half was satisfyingly confirming some practices I have picked up by using python for a while, and the latter half was part mind-boggling, part "oh please don't do that". Learned a lot as always, thank you James! #pycharm
Glad you enjoyed it!
I remember that I watched Raymond Hettinger's "Super considered super!", understood a little more about the MRO but could not understand how `super()` works.
Now I have watched this video and I have learnt something new today! Thank you!
I still don't fully understand the `super(...)` with argument(s) form, but I guess/hope I don't need it since I don't need to consider my job security. 😂
Fantastic introduction to super() and MRO, thanks.
My pleasure! Have a super day!
Those theoretical questions were bait-and-switch tricks. It would have been less confusing if you had shown all the information before asking the question rather than hiding the most important part that makes it possible to accurately assess the question.
It's a bit scary how much work at runtime goes behind a simple attribute/method lookup.
To be fair, it's python, which is about convenience over speed. If you want method and property lookups to be fast, you should probably be using a different kind of language
@@berylliosis5250 Considering that attribute/method lookups are one of the most common things you do in a programming language, it's quite frightening that what takes one machine instruction in a compiled language takes upwards of 100 *bytecode* instructions, including both branches and memory fetches.
I hope (and think, but am too lazy now to check) that Python does some optimizations here to bypass this cost because I've never really felt that Python is that slow in comparison. I do expect a certain level of speed from any language.
I love these metaprogramming videos. The best way to explain a complex language features is to show the implementation.
Side effect of this video: seeing a weird / operator in the __init__ of your custom Super implementation, looking it up and learning about positional only arguments. Loved this video!
Intermediate programming channel are so rare.
It's always a pleasure watching your videos !
#clion
This was the most I have learned about python in any 20 min ever.
Subscribed
I would love to see an explanation of the Walrus (:=) operator. How it works,tips and tricks, and use case scenarios. Mostly I've missunderstood when i should/can use it. My code often feels non-reable when using it.
Loved this video (and ofc your previous ones)!
#pycharm
Had to deal with a MRO nightmare I was building Custom Operators for an Apache Airflow application. Custom Operators basically inherit from other default Operators (to build custom solutions for your specific use-case), which in turn can inherit other operators and extra classes and so on until it gets to the BaseOperator class. To make matters worse, I wanted to merge 2 Operators into one.
Took me a long time to figure out what super() was actually calling and made me have to dig into a lot of stuff to find out. Your video is really straightforward and had it been released 1 year ago would have saved a couple of days of frustration. Awesome content as always! #pycharm
Can you explain why you needed custom operators as opposed to just Methods?
@@drygordspellweaver8761 I needed a simple way for the Data Analysts and Data Scientists to have their own self-serving Airflow instance. Since they already knew how to use an Operator, I built that way so I could "hide the wiring" behind that complicated process behind an interface in which was more familiar to them.
As a C/C++ dev, all this runtime stuff python is doing just to do basic static things is horrifying.
As a Rust dev, all this unsafe UB in C/C++ is horrifying! :D
@@josephlyons3393 At least we don't have a whiny compiler. :)
Just kidding, I like the rust's approach. Cppckeck for cpp is sometimes all what you need tho.
@@embeddor3023Rustc certainly is soul crushing, lol. C++ was my main language for many years, but I mostly dev in either Rust or Python these days, depending on the task.
I find it quite enlightening when you dive deeper into how these features work rather than working with the superficial layer of when to use/ how to use these features. I always come away with some useful knowledge I can apply to my code. #pycharm
Finally, a video that covers the inner details of super and how it works, also the pure python implementation of super is awesome, learnt a lot from this video, thank you so much Murphy. #pycharm
Very welcome and glad you enjoyed!
And here I thought “How complicated can super() really be?” It’s a joy to learn from you. Keep up the great content! #pycharm
Object oriented programming in Python is complicated. Thank you for this video!
Well, tbh. in most cases it's not a big deal because you can compose functions pretty easily by explicitly calling them. You don't need "object oriented" programming (more like inheritance) in these situations.
It seems like it, but it's actually far less complicated _in practice_ than in many other languages, like C++ (public vs private inheritance, recursive inheritance [CRTP], pure abstract classes, constness concerns, public/protected/private members, etc. -- and that's before we add templating!).
In both cases, though, you should really avoid inheritance for code reuse, anyway. Inheritance should be used for polymorphism. Only in some rare cases when following certain design patterns (or idioms) should you inherit anything but the interface, and in those cases, you should just follow the pattern. Generic programming is usually closer to what you want with code reuse; Python has this with duck typing, while languages like C++ do this with templates, and some others like Java fake it (somewhat poorly) with type erasure.
There is one valid use case for the two and one argument form of super. I am a bit surprised you did not mention them. As you pointed out the first argument tells you where to look for the next in line in the MRO. If you pass in one of the parents explicitly you can ignore part of the MRO and look for methods or attributes starting from a fixed parent / sibling. Not that you need it all the time but it can be handy.
Wanted to get into open source dev. One of the main projects I wanted to get into is pretty #clion oriented would love a key!
Your python knowledge is truly frightening... I like the video, but man this is a deep serious stuff. I'm overwhelmed.
Note to self: never use inheritance in Python if I want my code to run fast. Seeing what goes on behind the scenes is amazing, and a bit scary... the VM must be doing *so* much work every time a super() gets called XD
after like 8 years of programming I've come to the conclusion to use inheritance as little as possible it just creates a mess in places you never think of
This video again shows how Python's compiler does magic things. And again emphasizes how it's very important to understand when things happen (in the compiler or in the interpreter).
It is impressive, how Python implements all these functionality _with Python_ - from a technological point of view. Python, as an interpreter language, implements itself by using Python... fascinating!
But from the perspective of a user of the language ... all this black magic everywhere ... meh. :/ In other languages, using "clever coding" is regarded as something bad ...
I'm assuming that super is implemented in C (at least for cpython) and not in pure python since it's a built-in. But it could be implemented in python, aside from the class injection hack.
Compared to most other languages, python has very little black magic. Other languages also have things like implicit `this` variable, which feels more like black magic to me
I think this is a case of "practicality beats purity" :D
I remember programming in Python 2, which didn't have this black magic. This meant that every. single. super call needed to be the two-argument form... Believe me, it gets old very quickly, apart from being error-prone, as you now have to mention the class you're in everywhere.
Love the 1-letter variables in your functions. Right after coming from one of your "nooby habits" videos too.
Discord gang 🤙🤙
Always appreciated!
I have been wondering whether a function like this exists for a long time. Thank you!
Very welcome!
I finally understood why multiple inheritance is dangerous. I didn't understood it in University. Thanks!
Awesome advanced Python stuff. Already knew about how super works and the MRO but didn’t know about the implementation details. #pycharm
I love the level of depth you manage to go into in such a short time. Few guys on RUclips deliver this stuff as well as you do. Also #pycharm :)
The more I watch this channel the more I know that I don't know Python at all.
Excellent video once again!
Wow, thanks!
Awesome explanation, especially the part on how super works underneath the hood!! Thanks a lot.
It's always interesting to see in-depth explanations of how a language works under the hood, like this or the ",=" python operator you explained in a past video. Keep up the quality content :) #pycharm
That's an old one! You must be a long time viewer! Or a dedicated one at least!
Coming from Ruby's super, it's nice seeing how Python's works exactly for those more complex cases. #pycharm
I started using super() due to the structure of a project (needed a parent class with some general methods and configs, and child classes made to inherit from it). I was exposed to it from a previous video from you, where you used to create a subclass dictionary to be able to call the subclass given the parameters ( this is something i ended up needing in said project). I already knew some of the stuff you expose here due to researching, but as always, you go the extra mile. Thanks for your content.
#pycharm
Very welcome and glad to hear you're doing your own experiments! That's the best way to learn it!
Multiple inheritance needs a carefully designed class hierarchy. As much as possible, structure your Python classes to emulate the functionality of the single inheritance + interfaces/mix-ins model. In particular, only the principal parent class of a class should descend from anything other than object, thus mitigating the diamond problem. But also consider composition may be a better option for secondary parents.
I'm a data scientist/engineer/Web developer and I've made use of your videos in my work many times. Your channel is one of my training links I give out to new staff. Definitely would appreciate a Prof pycharm license for personal use #pycharm
You are really good at teaching. I like how you build up the complexity. Nice!
I've been asked many times about a clear explanation of super(), and here it is!
After so many years using python, I still learn a lot from this channel.
#Pycharm #CLion
Wow. You put in a LOT of detective work on this. Thanks.
Your videos always make me regret I ever got into Python. They're great!
You just simply have to be one of the most informative channels on programming out there. #pycharm
Thank you very much I appreciate the praise
I love both PyCharm and CLion (the only editor I can not only stand for C++ but that I actually enjoy).
I seldom use super in my projects (personal and work), but this was a good explanation.
Wow - that was complicated but enlightening. Your videos are great!
Oh my God thank you. Having a video this concise and clear is really really helpful.
I am glad I came across your channel. Just realised I've been more focused on the 'how' things work in python but didn't give much thought to the 'why'...good job mate. #pycharm
The way the topics are choosen and presented is astonishing 🔥😇
#pycharm
Loved it! Really really nice and descriptive video. Thanks for making this @mCoding.
You are the best in class!!! Salute!
hey, i just came up to that video an i really love it!!
although, i think super and mro are far from the best python feature, and i made an alternative to super for fun.
I think it's weird to assume it's possible to order multiple inheritence trees, it's weird the target of super is so implicit, and depends on context, it's weird that child class affect the super behavior in its parent, it feels like a side effect, and it's weird that mro can fail, and makes some class definition impossible.
the alternative i produced for super (i called it __as_parent__) follow those rules :
1) it should reliably target the parent class it targets, no matter what other child class you decide to create later on
2) in case of multiple inheritence, it should be passed the parent it targets as argument (you can then make one call per parent method you wanna refer)
3) it should be able to stay implicit when it doesn't need to be explicit, no need to specify the parent when there's only one
4) it should be able to target ancestors, direct or not (it's needed to make it work with the alternative to mro i produce, in some weird cases)
the alternative i produced for mro (i call it explicit method resolution) follow those rules :
1) any method / attribute present in the class definition should be the one resolved to (other rules assume the method / attribute is no present in the class definition)
2) in case a method / attribute is not present in the class definition and in any ancestors definition, it should raise an AttributeError
3) in case only one parent can resolve it, it should be the one resolved to
4) in case multiple parents can resolve it, it should raise an ExplicitResolutionRequired error (i'll explain how to solve it later)
5) in case one parent raises an ExplicitResolutionResquired error, this error is transmitted to the child
6) in case multiple parent can source it from on single source (diamond shaped inheritence), the ExplicitResolutionRequired error is not needed
To solve an ExplicitResolutionRequired error, you can just define the method / attribute in the child class definition. (method can then refer to parents method through the use of super or my __as_parent__ alternative)
Do you think it make sense?
here's the github repository : github.com/malmiteria/super-alternative-to-super
I love the idea and huge props for implementing it! That's the blessing and the curse of such a dynamic language. Don't like something? You can completely write your own.
I predicted that super uses __locals__ and type() but the Proxy concept was the missing piece. Very enlightening. I learned about proxy objects last yearish but haven't intentionally used the pattern in my own software yet, so it doesn't come to mind immediately as a solution. Gotta find an excuse to apply the concept in my own code sometime so the idea sticks
Yeah proxy objects are one of those things that make a lot of python magic happen. They can be useful, but they can also be confusing for users sometimes.
Using super with the two arguments to call any object's function is blowing my mind!
This channel is pure gem!
This is actually insane, the cursed local var situation is just shocking.
absolute knowledge.... , though most of these i knew olredy, but you are actually going gr8 sir.
how to get which class is being run by super , i did not know...thanks for that and really appreciable work sir, all hands down.
I looked for this A LOT through last week
You’re a mind reader and your content is amazing!
Thank you so much for helping the community to understand these concepts better! #pycharm
Your videos are always very informative and straight to the point, unlike other tutorials, where every character of code is analyzed on it's own. Keep up with these good videos
#pycharm
Time to learn something I never needed to know about Python at this level of depth. #pycharm
Please write a book James. I would 100% buy it and study it. I know I would be a better engineer for it.
This channel has really helped me level up my Python skills. Videos like this are really helpful for people like me who want an “under the hood” understanding of the language. #pycharm
As a next step, try reading a bit of the CPython source code!
the comments at 3:00 seems wrong, it seems that it does go to A's f but A's f IS Root's f.
So it does go to A's f in that case, which is the same f as Root's f.
good Lord, so much depth inall of your videos. Keep up the great work
Amazing. Your videos are so consistently great. Good work!
Oh this is perfect, i was just struggling with this like an hour ago.
I had some dataclass inheritance structure (frozen) and i had some classmethod as a factory which i wanted to inherit but it all fell apart.
I did use it once, it frelled up my code... but what it did for me was to prove that I should not have used classes for that problem in the first place. The need for "super" is a super-sized warning flag that the design is absolutely flawed.
I just hope to be as insightful as this guy one day
Personally, for those very few times it _appears_ ambiguous which defining class the `super` will go to, I either use the super call with arguments, or just outright call `superclass_name.function_name(self)` explicitly.
That way I don't have to require the reader to untangle the MRO algorithm.
The downside of course is that now the super classes have to be named multiple places. The class's definition and any place I need to use this trick.
But I consider that worth not requiring knowledge of the "deep magics" of Python
wow. it's amazing that this language ever computes anything
02:04 Jim Harper feelings
Always wanted too see info about super but never thought about it directly. Thanks! #pycharm
#PyCharm Could have never have guessed that MRO was being used in resolving super calls.
10:05 I think there's a potential problem here with using a mix of explicit and packed arguments.
The explicit arguments will be absent from the packed arguments, so then would also be absent from the super call. Eg if the next sibling class also needs the argument `validators`, it won't get it because here `kwargs` doesn't contain `validators`.
If, however, you preserve all the arguments in the super call, then there may instead be a problem if the base class doesn't accept the extra arguments.
super in python is weird, but thanks for this explanation. When I saw the super before I opened the video I thought that the video would be some kind OOP bullshit that people doesn't get right, but boy, oh boy, how wrong I was.
Thanks the amazing video. And lesson learned, never assume anything and never underestimate stuff.
I am using this frequently but this is far the best explanation .
#pycharm
I was just reading reading an article from Real Python about super() to clear some doubts and you posted a video about the same thing! #pycharm
Thannks, You make difficult topics interesting and easy to learn
Thanks for this video, I will use this in my current project #pycharm
Bit of a different video idea suggestion! I understand you prefer these in-depth videos for important python features, but how about a video on:
All the python class dunders? Not in detail, but as a quick: 'here's a use for them'
Even after 8 years of python, I still only just learnt the __setitem__ and __getitem__ dunders last week, yet there's still quite a few more I have never used, and don't know when to use.
This is already on my short list!
@@mCoding Ah thank you! ❤️
Also this is a wonderful video too if I do say so 😁. Keep up the good work!
Interesting and insightful as usual. What's really surprised me the part where you unraveled the mystery behind how zero argument super works, which also explains the older form of super. Thanks for your content! #clion
Nice video, though you didn't fully specify the MRO search algorithm (C3 linearization). I fully agree with Mark Lutz (Learning Python) on the concepts of "Cooperative Inheritance": best not to do it, since any programmer using a library you write using cooperative inheritance can break your code by simply adding a new mis-behaving class in the wrong level in your hierarchy.
#pycharm
Thanks for the explanation. Currently in class and the instructor glossed over the SUPER keyword.
Nice video, this video is a good example of why we should avoid multi inheritance in most cases
#pycharm
very cool. even just the 99% bit - the rest was just gravy! thanks man!
Never thought about super as proxy, rather evident in hindsight :D Anyways, well covered, thanks! #pycharm would sure be nice!
In my opinion, if you need to think about MRO, then it's time to refactor your code. I use inheritance only when I create interface. Composition is the answer most of the time...
#pycharm ive been using python for a while now and have never known what super() does so its cool to learn about it
In your own super implementation in the init method I see you do:
`if cls is None and obj_or_cls is None`
I ilke to use:
`if foo is bar is None` or `if None is foo is bar` or `if foo is None is bar`.
Personally, I think it's less bulky to read.
I made a whole video about how I don't like these chained comparisons, but it seems like many people do prefer them! It's just a preference to me.
@@mCoding Oh okay, just watched that video and in it you actually present a `if x is y is z` and you put in in the ok-questionable range, but you also said that the only reason you didn't put it in good use cases was because "most people probably aren't aware that `is` [...] is a comparison operator".
However I very much agree about the bad examples you showed in that video. That's hella confusing.
Also at the end of your video you are saying that you don't want to get rid of chained comparisons, only those that inherently don't make sense. In the end you're even saying that you'd like "to see an expansion of chained comparisons to other cases that do make sense", after which you'll talk about how on numpy `xs[3 < xs < 5]` didn't work and that you'd like to see it there.
All in all from the video it isn't entirely clear that you dislike chained comparisons, only that you dislike confusing chained comparisons (i.e. comparisons with opposing operators, f.e. ``).
EDIT: Actually you do say that one probably shouldn't use this except if one does a very simple comparison, however as a simple comparison you name `x