I've used generics a couple of times, but this explanation just made it 'click' for me. Thanks for such a good video, love seeing creators put out videos for a more intermediate/advanced coding skillset.
I love generics!!! I personally come from Swift which is statically-types so I’m forced to use generics all the time. Hearing about them in Python was a surprise though 👍
Great video! I was really surprised when I first saw generics in a library and it took me a while to wrap my head around them, especially because it's just not documented very nicely. But you did a great job explaining it!
Yeah that's why I couldn't get it to work haha. It's also a shame Pyright isn't the defacto type checker, it's definitely far more on the ball than Mypy is.
C is the return type of the converter specifically. In the example, type T becomes type C, meaning that our value will be the return type of the converter. If we used T directly as you've got there, I believe you wouldn't be passing a generic type through and the type hints wouldn't work. I could be wrong though, try it! These types are difficult to follow sometimes, it's possible you've found a shortcut I missed.
@@Carberra He is using a converter Class and the __call__ - Method, which defines a "own" T. By the way, you should use from __future__ import annotations. Then you can get rid of the single quotation marks in def __init__(self: 'EnvVar[str|None]', name: str) -> None: ... anyway, i a am not a pythong typing buff. Like your videos man!
Agreed. When Mypy supports them fully I probably will come back and do that, it's a shame it's taken them this long to support it, and even now it's incomplete support. For now, if you're curious about it, the old and new syntaxes should both be in the GitHub repo in the description. I think I remembered to do all that.
Thanks for giving a more advanced example. I did not feel like it was too difficult to follow but i do feel like i need to play around more with overloads, cause they do look cool.. Although it can make the code less readable and harder to maintain.
I guess it can go both ways -- if you have an awkward type signature and need to explicitly state what's valid, they help you get a better idea of what the code can actually do.
This may be a stupid question...but, why the callable has to be type C, isn't it more straightforward for it to have type T? Is there some edge case where return type of converted has to be different than return type of .value()?
Python's @overload is just an annoying way to implement "real" overloads like OO languages such as Java, C# and C++ have. To write the same EnvVar class in any of those languages would be much, much easier since you can actually just write different versions of the constructor, but since Python doesn't have runtime overloads, we need to go through all that nonsense to ensure that the typing comes out right. Because of this, I've wanted to use @overload many times because I felt like it suited my case, but ended up doing something else because it wasn't worth it. In the EnvVar scenario, I'd probably just use different classmethods to initialise the class instead, and probably have an `identity` converter instead of a None one.
If your using types in python maybe it's time to just use GO or something. I'm learning it now and it feels a lot like python. It's clunky to try and add a bunch functionality on top of a language that was never meant for it, look at js. You wind up with some grotesque monstrosity of a codebase lol
Just a reminder: Python doesn't have a static type system. Python has type annotations and runtime value types. The semantics of this "static type system" is created by external non-core tools like mypy and their interpretations of the core typing module. And the typing module is second fiddle to the actual runtime value type system of the interpreter implementation. Also, since type annotations have no runtime semantics (unless you enforce it), anything you do with such type annotations serves either your own satisfaction, or serves as documentation. In the latter case is it really relevant to communicate that "kwargs: dict[str, Any]"? Since Python is interpreted, can you really achieve performance optimizations via type annotations? No you can't, because in runtime type annotations are noops. Since type annotations are noops during runtime, can you really claim correctness of code via static type checking? No you can't. So the only things Python's type annotations are useful for, is making sure your IDE developers don't have to work to hard to infer types because you're doing that job for them so diligently, and making your type documentation so specific and complex, that everyone else has to suffer through the n-th definition and alias of dict[str, any] and list[any]. Generics isn't even worth talking about, because it's just the pinnacle of the type masturbation evolution. It's literally for people that only care about the "technically correct" static representation. So if you encounter people who are deadset on going full ham with Python type annotations and static type checking, just know they're out to waste your time.
I'll start by acknowledging that you're absolutely right about the noop nature of the annotations. And also agree that zealotry is a waste of time. Where we diverge is in that static code analysis is useless. I have caught dozens of silly bugs before and after going to production when adding well-targeted type hints. The duck typing nature of python invites Devs to be liberal with variables and eventually they forget that two calls upstream a parameter can be `T | None`, which invariably leads to unhandled exceptions when people try to access methods that `None` doesn't have. That's just a common example off the top of my head. Others say, "then you need Go" or other statically typed language, but in enterprise we may have production systems written in python and it's non-trivial and *expensive* to rewrite, so there's definitely value in annotations, however clunky they are.
@@virtualraider My comment was already so long, that I didn't but should have added: You should write unit tests, probably in any language, but definitely in Python. For the amount of work you input, nothing else gives you quite the same result like unit tests do. Unit tests are runtime tests, they most definitely actually run your code. Any form of static code analysis can produce results, but produces results about the static nature of programs. The static nature of programs is largely irrelevant, because you must run programs to actually get results. In static time, when writing all those types, feeling good about mypy not giving warnings, you're still only dealing with an idealized fantasy of a running program. Static typing and static analysis is ASSUMPTIONS about a program. Unit tests also run in an idealized fantasy setting, but at least it runs the code. Even with the sometimes excessive mocking required in unit testing, you're still at least running the code. So unit testing, with practice around achieving branch and line coverage, will test and verify many assumptions about your code. And because unit testing tests runtime, it also tests the runtime value types. So unit testing or runtime testing will verify pretty much the same assumptions static typing makes and even a bit more. In all my 20+ years of programming professionally in several languages, unit tests where always the best cost-vs-benefit investment. In all my 20+ years of programming I have also learned, most developers don't actually want to run or test their programs. They like writing code but they don't like running code. Those that did run their programs, they used unit tests or other runtime tests.
@@FrederikSchumacher yeah fair enough re: unit tests, I'm a huge advocate. However they serve different use cases. As you well pointed out, they come to play at runtime, which means you either have to be disciplined enough to run and update your tests regularly (Ideally do TDD!) or your pipeline needs to do it. That's late in the Dev cycle, the code is already written, possibly you're pushing a commit. OR you're new, learning, or the thing doesn't grant the effort. On the other hand, annotations are immediate feedback. You read it with the code, and if you use a good IDE it will tell you in real time about that code over there. Neither _change_ the way code works. You're exactly right it's just fantasy and idealised code. There's something else in the real world that behaves exactly like annotations: Traffic signs A stop sign doesn't actually stop your vehicle. You can drive at 80Kmh on a 40Kmh school zone and the signs are powerless against you. BUT that's not what they're there for. Signs are there to tell you what to do and it's up to you to follow them or face the consequences of not doing it. Same with annotations, they are there to help and prevent accidents.
@@Carberra It's a phrase that comes from the realization how limited the usefulness of static typing in languages like Python and also TypeScript is. The most real-world practical benefit of investing into deep typing systems - like generics - is documentation. "From me the author, to you the reader and user, this is the structure of data my code is capable of handling". For readers/users, there's a threshold where the complexity of specificity surpasses the usefulness. At that point, you're no longer writing and designing the type hierarchy for anyone but yourself, you're quite literally only satisfying yourself. And satisfying only yourself and the imaginary people you think will appreciate your intense and complex type hierarchy without regard for real other people...well, that just sounds a lot like "masturbation" except without the sexual aspect of it.
Just not clear how one would use this in an actual project and how it’s preferable to any other particular method…. Without that it’s just very abstract and of unclear utility.
At this point, why use the type system at all...? In java and csharp type parameters exist to work around the impracticalities and anoying limitations inherent to a strict type system. This completely feels like the world upside down. (Also why is the syntax in python soooooo extremely ugly and convoluted???)
This. Just use GO; it feels a lot like python, and it's the right tool. Stop trying to squeeze python into something it isn't, you just end up with something ugly. Look at js lol
Well, it's mostly relevant for library developers. Just makes it easier to write good code if your editor gives you the early warning. Absolutely not needed for your own apps and scripts that are not published or used by large teams.
@@Jaranvir yes I know and I admit my original comment was a bit of a clickbait comment to be honest. I think in the context of writing a library they make a lot of sense and serve also as kind of additional documentation. My personal beef is more with the syntax of the type hints which I find unnecessarily convoluted. But I also don't have a better idea how to do it.
It's good to see advanced coding methods from time to time.
Yeah definitely, the advanced stuff is the most fun!
Generics are fundamentals
I've used generics a couple of times, but this explanation just made it 'click' for me. Thanks for such a good video, love seeing creators put out videos for a more intermediate/advanced coding skillset.
It's funny to hear about generic types in duct-typed language like Python or JavaScript
I love generics!!! I personally come from Swift which is statically-types so I’m forced to use generics all the time. Hearing about them in Python was a surprise though 👍
Great video! I was really surprised when I first saw generics in a library and it took me a while to wrap my head around them, especially because it's just not documented very nicely. But you did a great job explaining it!
I'd love to see a video on ParamSpecs. I.e. how to typehint decorators
I showed how to do that in my decorators video: ruclips.net/video/cG451ZXWYC4/видео.html
Holy boilerplate, Batman
That isn't boilerplate.
Do you know what boilerplate actually is?
I haven't seen the video yet, but it seems like just what I've been looking for. Very excited for this!
Hope you enjoy it!
About `int`, there is in builtin.pyi
```python
class int:
@overload
def __new__(cls, x: ConvertibleToInt = ..., /) -> Self: ...
@overload
def __new__(cls, x: str | bytes | bytearray, /, base: SupportsIndex) -> Self: ...
class bool(int):
def __new__(cls, o: object = ..., /) -> Self: ...
```
`bool` could be built from any object (including `None` → `False`), `int` doesn't suport `None` (raises `TypeError`).
BTW. pyright support `Box[T]`
Yeah that's why I couldn't get it to work haha. It's also a shame Pyright isn't the defacto type checker, it's definitely far more on the ball than Mypy is.
@@Carberra yeah, you can use variable params length converter, eg. via Protocol.
BTW I have a question, why do you use `C` in converter, not just `T`?
```python
class Converter[T](Protocol):
@overload
def __call__(self) -> T: ...
@overload
def __call__(self, value: str, /) -> T: ...
class EnvVar[T]:
@overload
def __init__(self: 'EnvVar[str|None]', name: str) -> None: ...
@overload
def __init__(self: 'EnvVar[str]', name: str, *, default: str) -> None: ...
@overload
def __init__(self, name: str, *, default: str | None = None, converter: Converter[T]) -> None: ...
def __init__(self, name: str, *, default: str | None = None, converter: Converter[T] | None = None) -> None:
self.name = name
self.default = default
self.converter = converter
@property
def value(self) -> T:
value = os.environ.get(self.name, self.default)
if self.converter is not None:
if value is None:
return self.converter()
return self.converter(value)
return cast(T, value)
if __name__ == '__main__':
EnvVar('FOO').value
EnvVar('FOO', default='bar').value
EnvVar('FOO', converter=int).value
```
C is the return type of the converter specifically. In the example, type T becomes type C, meaning that our value will be the return type of the converter. If we used T directly as you've got there, I believe you wouldn't be passing a generic type through and the type hints wouldn't work.
I could be wrong though, try it! These types are difficult to follow sometimes, it's possible you've found a shortcut I missed.
Can you also elaborate on the "variable params length converter"? I'm not sure I follow.
@@Carberra He is using a converter Class and the __call__ - Method, which defines a "own" T. By the way, you should use from __future__ import annotations. Then you can get rid of the single quotation marks in def __init__(self: 'EnvVar[str|None]', name: str) -> None: ... anyway, i a am not a pythong typing buff. Like your videos man!
Very good example
I would love a video about the 3.12 generic syntax. Much better than subclassing Generic and using TypeVar.
Agreed. When Mypy supports them fully I probably will come back and do that, it's a shame it's taken them this long to support it, and even now it's incomplete support.
For now, if you're curious about it, the old and new syntaxes should both be in the GitHub repo in the description. I think I remembered to do all that.
Thanks for giving a more advanced example. I did not feel like it was too difficult to follow but i do feel like i need to play around more with overloads, cause they do look cool.. Although it can make the code less readable and harder to maintain.
I guess it can go both ways -- if you have an awkward type signature and need to explicitly state what's valid, they help you get a better idea of what the code can actually do.
Nice vid
It was easily understandable, not like the one with descriptors... Could you make an other video on that? Thanks :)
You can either have strong typing or strong tests. The trick is finding the right balance
This may be a stupid question...but, why the callable has to be type C, isn't it more straightforward for it to have type T? Is there some edge case where return type of converted has to be different than return type of .value()?
I think it might because `int` do not accept `None` as input, it raise TypeError.
So `int` conflict with `Callable[[str | None], C]`
Thank you 😊
Would the final environ code work just as well if you were to replace the `| None`s with `Optional`?
Yes, it'll work fine.
Which theme are you using for vscode? I really like that contrast
I've got a video in the description going through everything! The theme is Ayu though.
Could we specify the generic type explicitly when we call a function? Like `result = my_func[T](...)`?
From 3.12 yes, but Mypy doesn't support it yet. Pyright does, I believe.
@@Carberra You're right I just tried it. It works in Python3.12 and VSCode Pylance!
Great video.
Though I had a bad case of semantic satiation the nth time you said “BananaBox”.
Huh, that's what that phenomenon is called! And thanks!
This just feel like C++ templates without the rigorous type system and more boilerplate.
Python's @overload is just an annoying way to implement "real" overloads like OO languages such as Java, C# and C++ have. To write the same EnvVar class in any of those languages would be much, much easier since you can actually just write different versions of the constructor, but since Python doesn't have runtime overloads, we need to go through all that nonsense to ensure that the typing comes out right.
Because of this, I've wanted to use @overload many times because I felt like it suited my case, but ended up doing something else because it wasn't worth it. In the EnvVar scenario, I'd probably just use different classmethods to initialise the class instead, and probably have an `identity` converter instead of a None one.
What kind of color scheme are you using? I’ve searched far and wide and haven’t got a clue. Is it a custom theme?
A video talking about my setup is in the description.
"vital" is probably an exaggeration!
dang ı was gonna make a video about them as well ı guess another time
You still can if you want! You have my blessing haha.
@@Carberra thanks
If your using types in python maybe it's time to just use GO or something. I'm learning it now and it feels a lot like python.
It's clunky to try and add a bunch functionality on top of a language that was never meant for it, look at js. You wind up with some grotesque monstrosity of a codebase lol
Just a reminder: Python doesn't have a static type system. Python has type annotations and runtime value types. The semantics of this "static type system" is created by external non-core tools like mypy and their interpretations of the core typing module. And the typing module is second fiddle to the actual runtime value type system of the interpreter implementation. Also, since type annotations have no runtime semantics (unless you enforce it), anything you do with such type annotations serves either your own satisfaction, or serves as documentation. In the latter case is it really relevant to communicate that "kwargs: dict[str, Any]"? Since Python is interpreted, can you really achieve performance optimizations via type annotations? No you can't, because in runtime type annotations are noops. Since type annotations are noops during runtime, can you really claim correctness of code via static type checking? No you can't. So the only things Python's type annotations are useful for, is making sure your IDE developers don't have to work to hard to infer types because you're doing that job for them so diligently, and making your type documentation so specific and complex, that everyone else has to suffer through the n-th definition and alias of dict[str, any] and list[any]. Generics isn't even worth talking about, because it's just the pinnacle of the type masturbation evolution. It's literally for people that only care about the "technically correct" static representation. So if you encounter people who are deadset on going full ham with Python type annotations and static type checking, just know they're out to waste your time.
I'll start by acknowledging that you're absolutely right about the noop nature of the annotations. And also agree that zealotry is a waste of time. Where we diverge is in that static code analysis is useless.
I have caught dozens of silly bugs before and after going to production when adding well-targeted type hints. The duck typing nature of python invites Devs to be liberal with variables and eventually they forget that two calls upstream a parameter can be `T | None`, which invariably leads to unhandled exceptions when people try to access methods that `None` doesn't have. That's just a common example off the top of my head.
Others say, "then you need Go" or other statically typed language, but in enterprise we may have production systems written in python and it's non-trivial and *expensive* to rewrite, so there's definitely value in annotations, however clunky they are.
@@virtualraider My comment was already so long, that I didn't but should have added: You should write unit tests, probably in any language, but definitely in Python. For the amount of work you input, nothing else gives you quite the same result like unit tests do. Unit tests are runtime tests, they most definitely actually run your code.
Any form of static code analysis can produce results, but produces results about the static nature of programs. The static nature of programs is largely irrelevant, because you must run programs to actually get results. In static time, when writing all those types, feeling good about mypy not giving warnings, you're still only dealing with an idealized fantasy of a running program.
Static typing and static analysis is ASSUMPTIONS about a program.
Unit tests also run in an idealized fantasy setting, but at least it runs the code. Even with the sometimes excessive mocking required in unit testing, you're still at least running the code. So unit testing, with practice around achieving branch and line coverage, will test and verify many assumptions about your code. And because unit testing tests runtime, it also tests the runtime value types. So unit testing or runtime testing will verify pretty much the same assumptions static typing makes and even a bit more.
In all my 20+ years of programming professionally in several languages, unit tests where always the best cost-vs-benefit investment. In all my 20+ years of programming I have also learned, most developers don't actually want to run or test their programs. They like writing code but they don't like running code. Those that did run their programs, they used unit tests or other runtime tests.
Why is everyone against type annotations obsessed with comparing it to masturbation? Just euck, stop it.
@@FrederikSchumacher yeah fair enough re: unit tests, I'm a huge advocate. However they serve different use cases.
As you well pointed out, they come to play at runtime, which means you either have to be disciplined enough to run and update your tests regularly (Ideally do TDD!) or your pipeline needs to do it.
That's late in the Dev cycle, the code is already written, possibly you're pushing a commit. OR you're new, learning, or the thing doesn't grant the effort.
On the other hand, annotations are immediate feedback. You read it with the code, and if you use a good IDE it will tell you in real time about that code over there.
Neither _change_ the way code works. You're exactly right it's just fantasy and idealised code. There's something else in the real world that behaves exactly like annotations:
Traffic signs
A stop sign doesn't actually stop your vehicle. You can drive at 80Kmh on a 40Kmh school zone and the signs are powerless against you.
BUT that's not what they're there for. Signs are there to tell you what to do and it's up to you to follow them or face the consequences of not doing it. Same with annotations, they are there to help and prevent accidents.
@@Carberra It's a phrase that comes from the realization how limited the usefulness of static typing in languages like Python and also TypeScript is. The most real-world practical benefit of investing into deep typing systems - like generics - is documentation. "From me the author, to you the reader and user, this is the structure of data my code is capable of handling". For readers/users, there's a threshold where the complexity of specificity surpasses the usefulness. At that point, you're no longer writing and designing the type hierarchy for anyone but yourself, you're quite literally only satisfying yourself. And satisfying only yourself and the imaginary people you think will appreciate your intense and complex type hierarchy without regard for real other people...well, that just sounds a lot like "masturbation" except without the sexual aspect of it.
Just not clear how one would use this in an actual project and how it’s preferable to any other particular method…. Without that it’s just very abstract and of unclear utility.
Stop polluting Python!
At this point, why use the type system at all...? In java and csharp type parameters exist to work around the impracticalities and anoying limitations inherent to a strict type system. This completely feels like the world upside down.
(Also why is the syntax in python soooooo extremely ugly and convoluted???)
Having flashbacks to C++ templates.
This is turning Python into an abomination of a language. Worse than Java.
Typing is still optional in Python, therefore the potential abomination is optional
This. Just use GO; it feels a lot like python, and it's the right tool. Stop trying to squeeze python into something it isn't, you just end up with something ugly.
Look at js lol
Well, it's mostly relevant for library developers. Just makes it easier to write good code if your editor gives you the early warning. Absolutely not needed for your own apps and scripts that are not published or used by large teams.
@@Jaranvir yes I know and I admit my original comment was a bit of a clickbait comment to be honest. I think in the context of writing a library they make a lot of sense and serve also as kind of additional documentation. My personal beef is more with the syntax of the type hints which I find unnecessarily convoluted. But I also don't have a better idea how to do it.
@@mudi2000a have you seen Rust’s Types ?