Functions within functions, closures, and variable scopes in Python
HTML-код
- Опубликовано: 31 май 2024
- Functions within functions explained!
― mCoding with James Murphy (mcoding.io)
Source code: github.com/mCodingLLC/VideosS...
SUPPORT ME ⭐
---------------------------------------------------
Sign up on Patreon to get your donor role and early access to videos!
/ mcoding
Feeling generous but don't have a Patreon? Donate via PayPal! (No sign up needed.)
www.paypal.com/donate/?hosted...
Want to donate crypto? Check out the rest of my supported donations on my website!
mcoding.io/donate
Top patrons and donors: Jameson, Laura M, Dragos C, Vahnekie, Neel R, Matt R, Johan A, Casey G, John Martin, Mutual Information
BE ACTIVE IN MY COMMUNITY 😄
---------------------------------------------------
Discord: / discord
Github: github.com/mCodingLLC/
Reddit: / mcoding
Facebook: / james.mcoding
CHAPTERS
---------------------------------------------------
0:00 Intro
0:18 Motivating example
2:54 The One Rule
3:13 Level 1, return global
3:38 Level 2, conditional local
4:50 Level 3, nonlocal z
6:48 Level 4, cells and closures
10:35 Level 5, functions are objects, distinct closures
11:39 Level 6, donky chonky
12:52 Lambdas and comprehensions
14:32 Nonlocal and global
16:27 Level 7, please dont do this
18:18 Thanks 
Наука
Ah, your explanation on "scope determined at compile time" and "value determined at run time" really helps! Thanks!
I really should start thinking of compile time / run time behaviour differences...
I seriously appreciate the pace and length of this video. It really helps a lot.
Nice video as always james, understanding how decorators and closures work can be a tad bit complex, but you explained it with crystal clear perfection.
Video Request:
1.) Implementing builtin mathematical functions by yourself or creating your own complex ones efficiently.
2.) A full series on python typing, typehinting, linting and how python typecheckers like mypy, pyright work, explaining python type system and how to get good at it and not make nooby mistakes.
( Prefferably this one, because there aren't any other good guides on it on youtube, plus this fits as a idea for your channel and python typing is considered advanced level python.)
ball
@@trag1czny ball
I'd love to see a series on typing too.
Your first question is answered on stackoverflow /q/2284860
Long story short. Don't do any of that crazy stuff in your python code. If you're nesting function definitions within function definitions and reusing global scope names at every stage, refactor your code back to sanity.
Fun fact: even noddy old PHP has added “use”-clauses so it can do this kind of thing. Maybe the concept is more useful than you realize?
@@lawrencedoliveiro9104 Maybe. Can you give me an example where nested functions that shadow variable scope is better than linearising your functions and having distinct variable names? I don't think I've ever seen any, and I struggle to dream one up. Maybe in something like decorators? But even then messing around with scope seems unnecessary...
@@QuantumHistorian Shadowing is incidental, rather than being something you deliberately do. But lexical binding is certainly extremely useful in conjunction with functions as first-class objects.
If you look up my HarfPy binding for HarfBuzz, for example, the way I define the wrappers for defining callbacks makes heavy use of closures. I estimated that doing it this way saved me about 200 lines of repetitive code.
@@lawrencedoliveiro9104 Thanks for actually giving an example, I'll look into it!
@@QuantumHistorian That same module also includes a def_struct_class function, which creates, not just a function closure, but a whole class closure.
This really clears up the confusion I was having about closures and loops
That last example was masterful! I can't wait to obfuscate all code in production to be the only one that understands it, and keep my job!
14:42 “global” and “nonlocal” are actual declarations in Python. “def” and “class” are not declarations, since they construct function and class objects every time they are executed.
You are awesome at teaching complex subjects in a digestible manner.
This is deep! Great technicality and delivery. This is a gem.
Great video!
I have to rewatch the video for completely understanding this concept but it is really fascinating!
James, you keep surprising me with very interesting stuff! Love it. I've been programming Python for years and I really thought I understood variable scope etc. But this is a new insight to me and actually really helpful!
I think the reason you DON'T know about this though, is that if you encounter anything like this in real life i doubt the immediate reaction would be "hmm, interesting". Likely closer to "Hmmm, this needs to be fixed" 😅
But for the sake of demonstration, it certainly is cool to see how the inner workings actually pan out
Wow, it took me quite a while to fully grasp some of the examples. Thanks for the video.
This is very specific, but are you planning a video on Python wheels and possibly how to build them for different platforms, such as with Github actions?
I was litterally sitting in pycharm yesterday trying out various configurations of some testcode i wrote JUST to better understand this concept. Crazy good timing.
Great video, have a exam next week on closures and static/dynamic scope.
Bro you got the BEST examples and demonstrations!
I followed until the end. That last thing breaks my head, but I'll get there once I get more experience. Though luckily I'm aware enough to know not to write this in the first place.
A double edged sword in python is really the amount of 'this thing exists within python for a reason, but that reason isn't to use it.' these things can be great for debugging, but can never be in finished code.
That generator stuff at the end was well and truly cursed. Thank you
Would love a video on decorators and another one on how to incorporate custom C code into python
As extension modules or via ctypes?
@@lawrencedoliveiro9104 I'm really not sure, I think that's something that could be addressed. Some type of overview maybe.
Extension modules are where you write special C code to wrap the functionality of other C code as Python objects.
ctypes is a standard Python module which lets you write the wrapper in pure Python. This usually makes for much less code than writing extension modules.
Crystal clear, very good video
You're really good at explaining!
I always look forward to your videos. Thanks!
Thanks James for this video which helps to assimilate the subtleties of nested functions. Can you make a video on the most important design patterns? thank you in advance.
Your channel is like heaven
Wow good to know that WHERE value lookups are determined at compile time! That could save a bit of headaches during debugging! Excuse my English please
Great Video! to the point👌
Happy to report that I seem to have all of this recorded in my spine already, seeing as I answered correctly to everything. I think videos like these are a good way to do some self-valuation even if you think you understand the concepts already.
I don't think I've ever used the nonlocal statement in actual code, either.
Awesome, content, as always!
What about one use function:
def f():
global f
f = lambda: False
return True
also, it's fun that you can make function return itself
This is both wonderful and horrifying.
I wish I learned closures from you, this programming concept is frikkin hell and is mostly the cause of "magic" not only in python but in every other different languages as well
My guesses: ('chonky x', 'y arg', 'donky z').. I thought it was either.. most inner scopes (my guess) or most outer scopes ('global x', 'y arg', 'outer z'). Lol yea real surprised it was a mix of the two.. Good link we got that rule.
Also didn't realize closure was an attribute of the function (but ah, of course it has to be)! Knowing that's available to inspect would have helped my closure confusion in the past.
As a newbie I understand that for a function to run, it must be called by name. None of the functions shown in the initial example are externally called. If the first (outer) function is called, that function does not call the functions nested within it, so from my understanding, those functions would just sit there and do nothing, i.e., not operate.
Functions are first-class objects, same as any other variable. You can define a function called F, then bind it to G, and use it through G as you normally would with F.
Try creating a function that takes a function as an argument and runs it on the other passed arguments - you'll get the hang of it easily.
Scheme suddenly looks good to me with its sane lexical scoping, and JavaScript. And then I remembered that JavaScript was heavily inspired by Scheme.
At 14:00 or so you mention semantic equivalence of different list comprehension forms. Could you explain what semantic equivalence is? Do you mean to say that given the same input it will produce the same output but may differ in memory usage and performance? Or do you mean exact equivalence, where A and B compile to the same bytecode?
I mean the first one, they will produce equal lists, throw exceptions in the same situations, create the objects in the same order, etc, as long as you aren't doing crazy introspection stuff. They are not exactly equivalent, and they produce different bytecode.
i once wrote a reddit post asking for useful mnemonics in python. the responses were 'durrr there are none it's all simple'... but now i have one: "VARLAC" = Values at runtime, Locations at compiletime.
Some other languages I've programmed in ( **cough** Java **cough** ) would complain that *a = None* following *if False:* is unreachable. So Python doesn't complain about such things? What about Python linters / style checkers?
The compiler doesnt care, but I did have to turn my linter off for the video so it didn't highlight that section.
when I see the thumbnail , I was instantly having the JS-phobia :)
Donky chonky were great. :D
Dude I’ve been stuck trying to understand scope for weeks now, I’m working on a project to help me get the hang of python but oh my GOD, scope is kicking my butt because nothing I want to do seems to work. Might have to watch this video 20+ times to finally FULLY understand, even then maybe not cause my brain is mush lmao
16:40
What I expect to be printed
print(fun())
> 1
next(gen)
print(fun())
> 2
next(gen)
print(fun())
> 3
Video Request:
Can you do a video on memoization and caching generators?
What if we replace line `z = "donkey z"` with `locals()["z"] = "donkey z"`? Will it still works the same, or will the compiler ignore this definition? 🧐
Are there multiple non local scopes for deeply nested functions? like
x= "global"
def f1():
x = "non local 1"
def f2():
x="non local 2"
def f3():
x="local"
f3()
f2()
print(x)
f1()
are non local 1 and 2 assigned to the same cell? If so this should print non local 2.
The code prints non local 1 so there are multiple non_local scopes?
You have misunderstood. Since x is defined in function f1 it will look for x there, or if it wasn't defined there it will look for global, it will only peel outwards. This has to be the case or you would be afraid calling a function anywhere because it might rename global or local variables. This would be a nightmare. It can happen that code affects variables in outer functions, but that is either when specifying it or when you have something like an array where it is refrencing a memory location. Here is an example of the latter:
import numpy as np
y = np.ones(3)
def f1():
# y+=1 gives UnboundLocalError: local variable 'y' referenced before assignment
data = y # Reference to array from global variable y
print(y) [1,1,1]
def f2():
data2 = data
data2 +=1 # Changes the global variable y and nonlocal variable data
data2 = 1 # Doesn't change the global variable y
data +=1
print(y) # [2,2,2]!
f2()
print(y) # [3,3,3]!!!
f1()
print(y) # [3,3,3]
################
Now in f2 it doesn't matter what you call data2, as long at isn't the same as the variable your referencing (you get an error). So no y=y or data=data, but data=y or y=data will reference global variable or the non local variables.
Sorry, I'm moderately new to python so I might be asking a silly question here but at 4:40:
- You said that the local x gives a run time error simply because you didn't assign a value
My Question --> Isn't the string itself called "local x" the value assigned to that local x?
Maybe I missed something. thanks !
thaanks
How did you learn Python to this level?
Python has the key word global which allows you to look up global variables while reusing the name locally or not? Is there something similar for the outer scope and if not why not?
Are you looking for the keyword: nonlocal ?
Did you watch the video? Nonlocal is explained in detail with examples.
in level 2 isn’t the x assignment supposed to be in the if statement’s scope rather than the function scope?
As a developer with 20 years professional experience I can guarantee a 100% getting bitchslapped by some of your colleagues if you ever write code with these crossovers within and everyone around will give him 100% support. You CAN do that in any language, but ONLY for fun and giggles. NEVER as a production code.
I don't understand why you ever want to use Python for production code whatsoever...
Au contraire, lexical binding is one of the fundamental concepts that every programming language should have.
Yes, I have taken major advantage of it in production code. Check out
DBussy, my pure-Python binding for libdbus.
It constructs proxy-interface classes for D-Bus RPC interfaces on the fly, making heavy use of lexical binding and closures to define the proxy methods.
If you think you can come up with some simpler way of doing it, by all means post your code as well. Then we can compare.
@@lawrencedoliveiro9104 bro, probably no one told you that it is bad to be proud of writing a spaghetti code.
Inner functions are great for currying multi-parameter functions into single-parameter functions.
You should've mention this also:
>> [(lambda: x)() for x in range(10)]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> [f() for f in [lambda: x for x in range(10)]]
[9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
>>>
I mean, as with many of the examples in the video, please don't do this 😉 buuuut yes even these examples follow the same rules!
@@mCoding yes, I love to surprise my friends by this little piece of code and then explain them closures and variable lookup rules :)
@zohnannor Thanks for your example! It may be a short example, but I have stared at it for 5 minutes to understand it 😂
For those who cannot understand the 2nd line in short time, here is the explanation (Thanks James for his video explanation):
- [lambda: x for x in range(10)] can be slightly rewritten for clarity by [(lambda: x) for x in range(10)].
- The list comprehension can then be rewritten as:
def gen():
for x in range(10):
yield lambda: x
list(gen())
- The list comprehension can further be rewritten as:
def gen():
for x in range(10):
def inner():
return x
yield inner
list(gen())
- x in inner() refers to the x in the enclosing for-loop, as decided in compile time, but the value lookup is at runtime.
- x is first 0, and then 1, and then ..., and finally 9, because of range(10).
- list(gen()) is a list of inner's, each with a x in their closure, pointing to the same x in the enclosing for-loop, but the value has not been looked up.
- When the code runs f() for f in list(gen()), it assigns inner to f, and runs inner().
- The value of x is now looked up, and the latest value of x is 9. And this apply to all 9 inner's.
- Thus, [9, 9, 9, 9, 9, 9, 9, 9, 9].
I hope I did not miss anything. Thanks again for your example, zohnannor! And thanks to James for his video!
@@NicolasChanCSY yes, pretty much like that! I also show this code after, as an explanation (notice `id(x)`):
>>> [(lambda: id(x))() for x in range(10)]
[2117705466064, 2117705466096, 2117705466128, 2117705466160, 2117705466192, 2117705466224, 2117705466256, 2117705466288, 2117705466320, 2117705466352]
>>> [f() for f in [lambda: id(x) for x in range(10)]]
[2117705466352, 2117705466352, 2117705466352, 2117705466352, 2117705466352, 2117705466352, 2117705466352, 2117705466352, 2117705466352, 2117705466352]
and then, this:
>>> l = [lambda: x for x in range(10)]
>>> l[0].__closure__[0] # 0 index
>>> l[1].__closure__[0] # 1 index, same address
>>> l[0].__closure__[0].cell_contents
9
>>>
And also, I think it's a good thing to not only show some confusing, "broken" code as it may seem, but also a way to circumvent the shown behavior, in this case, it's this:
>>> [(lambda x=x: x)() for x in range(10)] # notice `x=x`, `x` becomes local variable
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> [(lambda x: x)(x) for x in range(10)] # notice `x` being passed as an argument
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>
🙂
@@zohnannor Excellent follow-up code to show the inner working (pun intended)!
Can you talk about how objects work in Python 3.11? According to Pablo Galindo object instances don't have a dictionary anymore. This leads me to wonder: do slots save memory compared to a regular class?
__slots__ does save memory. Not much, like 100-ish bytes per instantiated object. But if you create LOTS of those objects, the differences add up, sometimes significantly.
That said, if you handle thousands or millions of objects, those objects likely hold immutable datapoints to be processed. Consider not writing your own custom classes, but use NamedTuple or Dataclasses with slots=True
got x, y right, but z was a bit confusing why it gets sent with `inner`
can someone explain me what does the "return inner " stament do in minute 0:32 ?I do not get it
Inner is a function. In python, functions are objects just like ints and strings. Just like I can return my_int_varable I can also return my_function. In this case, the name of my function variable is "inner".
@@mCoding thanks,I got it!
4:06 so if you get the value of a variable, it refers to the global variable, but when you use an assignment, it suddenly refers to a newly created local variable instead of changing the value of the global one? How does that make sense; it seems really unintuitive that the variable name itself refers to completely different things depending on whether you are assigning or just getting a value. I really wish python just used a different syntax for declaring a new variable than for assigning to an existing one.
Edit: 14:33 ok so it _is_ actually still possible to assign to a nonlocal/global variable, I was wondering if that's even possible. What I said before still applies.
It's nice to know what happens behind the scenes, but as an experienced python programmer with some common sense and generalization abilities I could understand what level 6 does even without knowing about __closure__s. The x is taken from the globals and not from chonky because it makes no sense that a variable defined inside one function will affect another funcction it only calls, and the z is from after the inner function definition in exactly the same way that a function can use functions defined after it, just with a variable instead of a function and in a local and not a global scope.
What would be a legitimate reason to have nested functions like this?
Function factories.
What about
Comments within comments?
Nice
It’s functions all the way down baby
There’s a name for that in mathematics: “λ-calculus”.
So apparently I've seen this video before. And yet I still added it to my watch later playlist and had it there for weeks now
Thank you very much. This is an excellent video!
At first, when I saw x = "global x" I thought you will do something like exec(x).
Man python sometimes get wild because of how scopes works
Afaik windows 11 actually got rid of the vertical taskbar for the "centered" startbar.
The setting to move the taskbar to vertical on the left I think was in the display settings of Win10, if someone still has win10 please reply.
In recent videos i have seen a lot of people defending functional languages instead of oo.
But the idea of using closures to store values for me is so much more complex.
It might have since powerful features but for me it is a complete nonstarter.
In Python, functions and classes are first-class objects. That means you can operate with them at run-time just like you can with other types like numbers, strings etc.
This idea did not originate with Python, but with certain advanced languages from the early days of computing, like LISP. It was recognized to be very useful, which is why Python is designed with it as a core idea.
@@lawrencedoliveiro9104 i know. I program in python but never needed to use closures for anything.
I am sure it is probably powerfull ... but i cannot get my head around it when there are easier more straighforward ways of doing things... at least for me
@@DuarteMolha Function factory for code reuse.
@@lawrencedoliveiro9104 factory pattern?
Not sure i undertand. The factory parttern is usualy given as an example of oo programing.
If you mean more exotic code than maybe. I have never seen a reason to use a closure. But sure ... I am talking about my own experience.
I think the examples on this video do show how it can be a complex beast
@@DuarteMolha Look at my HarfPy module. It makes heavy use of both function factories and class factories.
If I saw that first example in a code review, I'd reject it merely based on the shadowing or the names. Sure, it's all logical, but readable is important in code. If a refactor with good naming isn't possible, I'd demand at the very least a comment or docstring explaining the developer intent and function. In Python, your tools may be powerful, but [uncle ben quote].
What if I wanted to do something like this to print out non local 1?
x= "global"
def f1():
x = "non local 1"
def f2():
x="non local 2"
def f3():
nonlocal nonlocal x #???????????
return x
return f3()
return f2()
print(f1())
There is no syntax for this in Python, you would need to pick another variable name besides x 😀
8:37 The CS term is “environment”, but perhaps the Python folks wanted to minimize confusion with the POSIX concept of “environment variables”.
That is, assuming they had a non-capricious reason for using the nonstandard term ...
Plans within the plans...
Yes but why would you want to define a function within a function?
Great question! The most common reason is to define some kind of higher order function. A common example would be decorators like @dataclasses.dataclass or @functools.wraps
Am i the only one here more surprised that variables in python apparently have function scope instead of block scope?
Oh shit divide by zero error.
That's why I feel like Python, such great language, is broken.
I am dreaming of Python 4 with pointers and fixed scopes.
You pass arguments like in C++, without thinking about mutable, scopes etc. If you don't want to work on the copy, then pass it by reference - ALL PROBLEMS SOLVED in the most elegant, readable way. That's the point of python, isn't it?
this made me more confused lol
Thirst
Discord gang.
Yes.
Sup anas
Im a member of the "I just opened RUclips, accidently" Gang
Yeah pythons way of doing closures is counter-intuetive to me. I would lock in any variables to their values at runtime when the it is defined. Changing variables later would not change the defined function.
Yeah well, even tough you put the two rules next to each other there are still two of them :D
SECURITY
The problem in Python is that assignment implicitly declares a local variable, so one has to declare (by nonlocal or global) if this is not wanted. Other languages like Scheme or Lua do this better, local Variables have to be declared explicitly.
luckily my brain does not function like that
I was going to learn Python but now I am not so sure. Scopes should be clear, simple, and clearly defined by the programmer. The fact that you need this video telling us it is very confusing and you need to learn a lot of rules, tells me something is structurally wrong with scopes in Python.
Ok, I will NEVER do this.
Sounds cool, but your pull request is not approved 👎
It's ethically illegal to use global variables inside a nested function.
This should be a PEP 😅
So basically, dont make your code as unnecessarily complicated and contradictory like this
Lvl 7 has not really a point to it:
1. You use "if False:" to explicitly create an "empty" state of that variable, but then you later might complain about it or get bitten by it!? UNLIKELY
2. You would usually go with a default value of "None" with no need of "If False" and then be pretty safe.
So, I don't understand why you would fear someone could unintentionally create a local variable as an empty cell, instead of a simple local variable initialized to "None".
It's like shooting yourself in the foot by knowing the internals of a language!?
The cell would still be emtpy if False was replaced by any condition that happened to be falsey at runtime. This is a very common bug in practice, though the rest of example 7 is of course just an exercise to test your understanding of the rules.
if you ever think have you do this: grow up
at 4: 24 you are passing something false but i see no change in code so what are you passing that is false?
Off screen I have a main function calling the code with True then with False.
cc @corridorcrew
LOAD_GLOBAL 0 (x)