Python OPTIMIZATION Trick!!
HTML-код
- Опубликовано: 11 ноя 2023
- ⭐ Join the Byte Club to practice your Python skills! ($2.99/mo): / @b001
🐦 Follow me on Twitter: / b001io
Background Music:
Rain, Book And Cup Of Tea by | e s c p | escp-music.bandcamp.com
Music promoted by www.free-stock-music.com
Creative Commons / Attribution 4.0 International (CC BY 4.0)
creativecommons.org/licenses/...
Another alternative is to create a default parameter for the function.
I was just about to ask this. Thanks for pointing this out.
Can you give an example
@@shreehari2589def func_name(local_var):
when using function in main code
func_name(global_var)
@@shreehari2589def func(local_var = 10) You can call the function with a custom local var or use 10 as default if none specified
@@shreehari2589I think they mean pass the global_var as a parameter like def function(local_var): and then pass global_var when you call the function
It’s common practice to also avoid using global variables as much as possible. Only use them as constants.
can you explain why we should avoid global variable? I am new to python and been seeing alot of this information, but quite unsure how to workaround on that.
@@novianandrian global variables sow the seeds of terribly bad practices in learners, and experts never use them anyway.
the reason why is because new learners dont fully grasp what a function is. it isnt code you just put under a `def` block because thats what you think you are supposed to do. its not even just "repeatable" code. a function is essentially a "sub"program.
suppose you are making a library, and someone imports your functions. that function is a black box to them, and they cant see that it wants to use "x" from the global scope. if the function tried to, and they actually did have an "x" in their global scope, even though they didnt know that was part of the use case, they would almost certainly run into inaccurate outputs with absolutely zero clue as to where said output was coming from, and literally no possible way to debug that.
but thats not just to say that globals are bad only in the context of libraries, using global variables like this is genuinely harmful no matter what. they direct your code into a very specific architecture which is completely unmaintainable, unreadable and harder to debug. they promote misconceptions on what functions are and how they are meant to interact with a program. and you will never, ever be able to advance beyond basic scripting while you still maintain this habit, because projects that are better formatted in multiple files are practically a nightmare to debug or expand when even a few globals are being thrown around between scopes and files.
in short, one of the biggest things i would tell anyone at this point in their journey to learn programming is to make sure that in every circumstance your functions are able to be run on their own. all of the information they need to function fully should be passed in through parameters. if you feel as though you cant do this for your specific case, it may be a sign that your code architecture is wrong, as I have never had that actually be the case in over 6 years, nor have I heard of anyone who has.
@@novianandrianbecause it creates a lot of dependencies for your function, and makes the code less organized
either pass it as a paramater, or create a class and make the function a member of the class
using it for constants is fine
not using global variables is a good practice for all programming languages
@@the_cheese_cultist when you have code that might need constant change or update don't global variables make everything faster? Im doing my capstone college class and im working on a website that deals with different aspects of CSV data surveys. and i approached it with the mindset of "hey making everything global makes it easier to edit later. (side not it was inherited from last year and the whole approach the other team started with was a mess so we could only focus on refactoring.)(we worked with ruby on rails )
@@stilyandimitrov39 I'm gonna need more info to give you an opinion, but there are many sources online about good coding practices, avoiding globals included, that can explain everything better than a RUclips comment.
best python optimization trick: switch to another language
thanks to this trick I managed to get a 50000% performance increase!
I switched to NodeJS and then bun.sh!
@@sanchogodinho lmaooo, yeah great... optimization I guess
Thank you❤I started learning binary code😍😍😍🥰🥰😍
@@firstnamelastname2775 nice, congratulations :)
... or just pass it to the function as an argument, not only more efficient but also way easier to read
This doesn't have to semantically make sense at all
kinda annoying to keep track of the variables and has to unnecessarily place arguments everytime u call the function
This is common in Pure Functional Programming
there's the `global` keyword, or `nonlocal` for nested functions
instead of creating local_variable, you can just declare `global global_variable` at the top of the function
exactly this video is super wrong
At least someone knows!
This can make the code very complex when using nested function, see mcoding's video
You have completely missed the point of the video. Hint: adding "global global_variable" at the top of the function doesn't change the bytecode at all. The whole (bad as it might be) point of the video is that accessing a global variable from the inner loop takes more time than loading the global variable into a local variable first.
@@joergsonnenberger6836 we can declare the variable as global inside the function before the for loop
I prefer passing the required values into the function as arguments. If I need to reuse a function in a different script, I know by the function signature exactly the function requires to run.
That's called dependency injection
Yep, DI is the way to go. Reuse your code and it makes it easier to read.
oh my god that is not dependency injection. that's just having a pure function
@@l3gacyb3ta21 Realized after two weeks but who cares i got two likes thanks to misinformation
If you need enough performance that something like this becomes worth it, you now are in need of a higher performance language.
best reply imo
I mean having faster python is still better than slow python.
@@l3gacyb3ta21 I wouldn't say it's strictly better, it's a tradeoff - because to get faster python you have to decrease the readability by making your code appear a little more complex. You should choose Python when code readability/understandability is important and performance isn't important, because that is what the language is designed for. That's why it's a really good language for one-off quick scripts. But if you are in a problem space where performance is valuable, it's nice to use a language that's really designed for that from the get-go. Where you can actually use the language's nice features without surprise performance problems.
That being said, sometimes you inherit a codebase that's already in Python and performance may become an issue so sometimes you have to choose lower code readability for better performance.
Right
If you care about improvement to 0.1 miliseconds from 0.11 miliseconds you should probably not be programming in python ...
I believe this is incorrect, any recent (python 3 and up) will emit a LOAD_FAST for locals (like ans and i) but a LOAD_GLOBAL for nonlocal, non-cell variables (like global_var). The speed difference between LOAD_FAST and LOAD_GLOBAL is pretty much negligible, especially with the adaptive opcodes in newer versions where most of these have faster, adaptive versions.
That's what we call a premature optimization.
Except its done after the mvp and before it's too big to optimise?
@@revimfadli4666yeah, it’s probably a fine optimization if your program is just a counter multiplying by a global, but it is woefully insignificant in the scale of any program larger. Or, if your program is just a big loop, use PyPy. I remember hearing that it’s better at this stuff.
no, that's not
it’s just an example of
He literally profiled it. How's that premature??
Definitely use this trick to run out of variable names
This is not true. Variable scoping is done at compile time (yes, Python has one), not at runtime. It's actually the only complex thing Python compiler does.
You can look through the related data structures using the symtable module.
It's also visible if you use dis.dis on each version of the function - one will have LOAD_FAST op, the other will use LOAD_GLOBAL. It's predetermined.
It's predetermined but it doesn't mean it has the same cost. Lookup in local scope is faster for many reasons - it can be stored more efficiently, you can have more faith in what you are dealing with etc. For example, if you are dealing with something in outer scope, there are no guarantees about the value assigned to the variable as it may be changed in some other part of the code and global lookup has to be used to ensure that you are using proper value at the time you want to access it. If you copy the global value to a locally scope one, you can avoid that lookup as it is "detached" from its old variable. I don't know how lookups are managed for collections but if they are not explicitly copied, then I would assume they are stilled handled through global lookup rather than a local one based on the same problems with it being stored at a global level.
@@viCoN24 Your reasons are also wrong.
The object under reference is stored the same way no matter if it's local or global. "Outer scope" and "global" are two different things, but whether you have "guarantees" about the referenced value or not is irrelevant - cPython has no JIT.
It has global interpreter lock, so the dereference doesn't have to do nich Apart from finding the actual memory address.
I would invite you to profile this in an actual code, where the local scope has just as many items as the global one.
I think I missed your point. Thanks for the explanation!
You focused on the underlying mechanism where it's still one operation rather than two lookups. I don't know how these operations are performed by CPython but one is clearly faster than the other so I tried to provide some reasoning behind it but it's true that GIL simplifies the situation.
In principle the lookup resolution behaves as author explained even though the interpreter is able to scope it internally to decide on the proper operation rather performing dynamic lookup and failing at runtime when variable is not found. You are right that it's only one operation but whether it's assigned that one operation or the other is decided in the fashion described by the video maker.
@@KennyWlr Wow! Thank you so much for this detailed explanation. It's good to be wrong sometimes to learn this much. Cheers! :)
Are you sure? I've done a "benchmark" in Google Colab trying it, and the second case gains some performance indeed (Not too much)
Bro I can't even here you over this beat. It's slaps more than I could have imagined 😂
Sorry this is just very bad. Really, really, really bad. NEVER teach people how to optimize the use of global variables. Teach them how to avoid them alltogether.
Right? I thought he was going to say to pass it into the function as a parameter, since he said “new to python” but I guess not lol
look it's bad practice but people use globals and sometimes they provide value too
Especially his changes also changed the whole program's behaviour as Global_var is no longer updated.
Question, how is he even getting the global variable to work in his function without write
global (var_name)
at the top of the function?
so you never use a global variable / constant in your code?
The same point applies to attribute lookup on imported modules. For example, you can assign imported function to a local variable to avoid an expensive lookup.
Let's say you use "math.sin" in a loop. Every time you call it, you perform a lookup on "math" module for "sin" attribute before calling that function. If you assign "sin = math.sin", you get rid of that unwanted lookup from "math" module and your loop with local "sin" will be faster.
If you have a function that creates a lot of objects, you might also want to assign the class locally to also avoid global lookup for the same reason.
If you find it interesting, you are dealing with some problem that Python might not be the best language to solve it in. Still, if you have to resort to those optimizations, it's at least nice to know about them.
That is literally what I needed for a school project I'm working on. How unimaginably lucky. Thank you!
Were yall doing AST analysis for a school project 0_0
This problem is the same in every interpreted programming language ( it doesn't affect performance on compiled langs cuz this only happends while compiling, not when running )
Is this why pointers are used in C and C++?
This is one of the reasons, yes. C and C++ are, by default, pass by value. That means when you pass arguments to a function the values are copied for the function to use.
Passing a pointer is still technically pass by value, but the value being copied is the *address* of the data; rather than the data itself. This allows you to save memory for larger variables (e.g., a big struct or a class). It also allows you to modify the original data, since you're being given a pointer to it rather than a copy of it.
Note that C++ has references, which are often used over pointers because they are safer (but are very very similar). If you're in C++, default to using references unless you have a reason to use a pointer (e.g., so you can call C functions).
For functions which are taking a pointer/reference to save time on copying data, you'll often see `const MyLargeDataType *mldt` or `const std:: string &str`. The `const` part signifies that the data being pointed to will never be modified by the function. Something to keep in mind also is that, for pointers, the following two are not the same: `const int *a`, `int *const b`. Here, `a` can be re-assigned to a new address of another `const int`. However, the contents at the address cannot be changed. For `b`, the contents at the address can be changed, but the address it actually points to cannot be changed. Here are some examples:
```
int c = 123, d = 456;
const int *a = &c;
int *const b = &c;
a = &d; /* okay, we can change where it points */
*a = 789; /* not okay, we can't change the data it ponts to */
b = &d; /* not okay, we can't change where it points */
*b = 1011; /* okay, we change the data it points to */
```
Note that in C++, all references are `const` (the address they point to cannot be changed), but you can still modify the data they point to so long as the type is not marked as `const`.
@@whirvis Thanks for the explanation. I'm trying to learn C, and pointers are a bit confusing to me, particularly when and why to use them. As I understand it:
int a; //declares a variable containing a value.
int *b; //essentially declares two variables, one contains a value, the other, an address.
*b = 123; //contains a value
b = &a; //contains an address
void Function(int *c){} //expects an address. Call like this:
Function(&a);
@@MortonMcCastle
int a; // designates enough memory to store an int but doesn’t say what int to store
int *b; // designates enough memory to store an address which, when dereferenced, should point to memory that stores an int (should not be thought of as declaring two variables because it’s only declaring memory for the pointer, not for any actual int value)
*b = 123; // is a very insidious bug because you’re modifying data at a random memory location because you never initialized b with a valid pointer so it’s interpreting whatever bits happened to be at that spot in memory before as a memory address. This is the kind of bug that hackers exploit to gain escalated privileges to systems.
b = &a; // b is now a pointer to a, so now it is safe to do *b = 123 which is equivalent to a = 123
One alternative I really like, is just using C.
Interesting. Didn't know it impacts performance that much.
Thanks for that, im learning Python coding, so it will help me in improving coding skills
In other langs, this fixes race conditions, as the code no longer assumes the variable won't be mutated.
The program no longer has to load from heap every iteration, since the value is already on the stack
This is actually useful, I'll try to keep it in mind.
Can you make a video on how you did the performance test at the end?
I dont know which he used, but he most likely used a profiler
Maybe cprofile or mprof or else lineprofiler I guess.
That was super insightful
I do this a lot when building a data-cleaning function. It also avoids mistakingly corrupting the global variable.
If you try writing to global_var it will create a local instance precisely to avoid corrupting it. You have to use the global keyword to specify global_var is global and be able to write to it. But really in the example in the video global_var should just be a parameter that is passed in. 99% of the time globals issues can be solved by just not using them and refactoring your code appropriately or starting with best practices from the beginning.
If you had an (inexplicable) reason to make the variable global, you wanted it updated within the function.
What I understood is to make that variable in local scope so as to prevent checking for the golabal variable each and every time. This reduces the time complexity
this optimization doesn't change time complexity, for that function it is O(n). time complexity refers to how well an algorithm scales when the amount of input data is increased. this optimization just reduces cpu cycles required to access the variable.
@@topzozzle6319 Thanks for the info sir
The synthwave theme extension though. My man 👌
Man being new to Python and new to coding just sounds like an awful time. Coming into python with years of statically typed OOP these are just things I do automatically lol
Use parameters or the global
Keyword. This solution is questionable at best, and also difficult to read since it’s not clear that you’re referencing a global variable.
Using the global keyword doesn't change the generated bytecode at all. For good or for bad, it doesn't avoid the problem this video is supposedly about.
@@joergsonnenberger6836 there’s no problem, the video pretends to do an optimization which, besides potentially not working, makes a bad practice worse and hard to understand. Adding the global variable makes the code more readable. And I doubt it doesn’t change the bytecode, since it literally allows you to edit global variables inside a function.
@@molohktegg2462 You are completely missing the actual point of the video: local object references are faster than non-local object references. Modify the example slightly and it becomes a lot more practical: create an empty function f in a subpackage a.b.c.d. Now compare the time of a module-level "import a.b.c.d" and calling "a.b.c.d.f" in a loop in the test function with doing a "from a.b.c.d import f" in that test function. You will notice that the second version takes about half the time.
It's also really annoying to see comment likes "I doubt it doesn't change the bytecode" since it is very easy to test that statement beforehand.
@@joergsonnenberger6836 I’m not missing the point, I’m disagreeing with the recommendation. First, if you need write access to the global variable, the solution won allow that, you should use the global keyword instead; and if you only need read access, then there are much cleaner ways to do it, like passing a parameter, which would eliminate the need to look for global scope variables, and the performance cost, while also eliminating the risk of secondary effects. Also if you’re using Python, most likely performance is not your first priority, and readability/maintainability are just as important if not more; and this recommendation is exactly the reason why global variables are discouraged, poor readability and leads to unpredictable code.
No, Python does not check locally then globally. Python knows what is local and what is global at compile time. The difference is that globals is a dict, locals is an array, so local access is direct indexing vs dict look up. You can have a look at disassembly, when python determines the variable is local it uses LOAD/STORE_FAST, vs LOAD/STORE_NAME. The name is quite explicit which is more efficient.
def func(size, mult):
return mult * ((size**2 - size) / 2)
print(func(1000, 10))
then add some math :p
Thanks for the tip
I've been playing around with global variables lately and your tip is really helpful!
Using Global variables is not good, unless they are constants. If you need a value [especially a variable one] it should be passed as a parameter to the function, so that the function always have the same exact behavior when it is passed the exact same parameters. makes them predictable and individually testable.
@@olivierdulac Use of global variables is fine in some contexts esp when writing smaller scripts or module level variables
You're better off using the global keyword if worrying about scope
Fun fact: a new scope is not created in a for loop
If you’re good at python, you should do the multiplication after the summation, since they’re separable.
Never mention "performance" and "Python" in one item!
the main i see here is the for loop, when at that scale of iterations i'd look at a different way to implement that section of code first, for instance doing that specific part in a more optimized language
The best optimization trick for Python is to write the performance-heavy bits in C++ and use Python as a frontend
This increase your program's speed while accupying more space for another variable.
Alternative: use C
the best optimization is:
for i in range(1000):
ans += 10 * i
This is the why you must use memory managed languages such as c/cpp or go
Thanks!
Python for CPU intensive tasks...
Can you do a video on how you test performance like that?
I think it is much better to make function with this 'global varible' as argument. Using global variables in functions is bad practise, because it makes functions less portable (you can't just take this function and place it in another program or file, because it may doesn't have this variable in it)
Better - when writing non trivial programs, it's often hard to avoid globals - unless you want to make the call graph a complete mess. If you're determined to kill all globals, you might want to malloc() a singleton (a struct) containing all of them (so you can re-enter the function without side effects) and pass it down all the way - even if you don't use it. That way if you change a function so it requires the use of such singleton, it's there. You might get a few warnings, but it works - and it makes maintenance less of a pain in the neck. BTW, always initialize your global vars in a separate function. You might be in for a surprise if you call it multiple times.
What theme do u use?
Or, use a global keyword
here and from Brazil a question what is your theme I thought it was very good for viewing the code
nice video!
Could you give details about how to test the performance of a code?
cant wait for the livestream
A person would think that python would ready pointers for all the available variables inside the scope like a decent compiled programming language
Replace the loop with something non-trivial and it can't guarantee that the module scope object binding doesn't change. Similar to how a C compiler could have to load the same pointer again and again as if it can't proof that it wasn't changed.
hoping for more python optimization trick.
amazing trick
Another optimization technique is to write code in assembler
Actually it resolves variable locations at compile time, yes python has a compile time. He fos the for loop, it resolves it once and then knows where to get it for each loop.
But only for locals. Globals are always a dictionary lookup.
Right that way it already has the variable to access
Another W for compilers
How did you build that graphic of the performance?
I was using this for convenience but it is optimization aswell 😅
Hey, how can I make my Terminal look like yours? Or can you someday do an general showcase of your visual stuff? Love your videos.
This looks like VScode with maybe the Dracula theme?
Wait, what about the global declaration that you can add at the beginning of your function that tells Python it's a global variable?
Where’d this guy go I like these videos
I was wondering about the `global` keyword.
Parameters✨💪
In this exact specific case, there is something even more simplier, no action are being done on the global_var, so why creating one, just use directly the value of it
My only issue is you didn't have your global variable in all caps to signalify it as a constant.
helpful 💯😌
But in order to assign the new local variable with the value of the global variable, wouldn’t the function still look within itself for local first, then global?
Yes, but this only happens once before the for-loop. It prevents it from searching globally in each of the 1000 for-loop iterations.
This does not reassign the local variable to the global correct, like it doesn’t change the global from what it was originally to what it is after the local function is done? Sorry super new here to python and coding.
@@thesleepingforest8929 It does, assigning the global_var to local_var creates kinda like a pointer so every time you reference the local_var it points to the global one and edits the global_var
Unless I'm wrong.
@@nyther❤
@@nytherit depends on the data type. If it is a string, int, double it creates a copy. If it's a list, set, ect then changing the local will affect the global.
Python has 'nonlocal' keyword for the lookup in global scope
Good bro😮
As a js dev I never use grobal variables. Always pass the variabes as parameters.
Thanks. Yes and no about this method, Python is about writing understandable code, only big picture optimizations are needed (general algorithm). If you value even 10% loop speed increase, beter switch to C, Rust or Cython
that prt of code or
you should pass the variable into the function.
I taught im the only one who using synthwawe 84 material theme
Have y’all ever heard of the global expression? It’s made exactly for this problem
Can you create a playlist on python
I feel like the locality of reference for variable scopes should be handled by the compiler, but I don't really know python does this given its an interpreted language.
It's compiled to bytecode. Many still think of it as an interpreted language but that's basically never been the case.
If performance is an issue in your application and it’s written in python, python is the problem, and no matter how many micro optimisations like this you add, your app is still written in a very non-performant language.
Another optimization trick is not to use global variables
well but actually you don't use the global variable in the first place
What is your vs code theme?
Bro , I want to take input from the user . But the input should be a number b/w 1 and 62 , what should I do to remove errors and take the input
Either regex or try-except int(input()) or .isdigit string method, then if/assert converted_to_int in range(1, 63)
@@Nip403 will definitely try
@@Nip403 advising the use of regex for someone new to programming is mean.
Python definitely has a built-in function to check if a str is a valid number before converting.
what theme are you using?
Remember friends: Premature optimization is the root of all evil. Benchmark before you optimize.
Another alternative: implement caching in python interpreter? This should not be an issue.
Counterintuitive if you are a C++ programmer.
I prefer use a parameter in that case
Could you make a video how I could make a text document where there is a list and in my python program I want a list that is the list of the text document. I would really appreciate it. Thanks for the helpful videos.
Video style reminds me of a certain ship that is on fire.
When nanoseconds make a difference.
Good luck training and hosting a large ML model on that
I always try to pass global variables in through parameters because it makes the function easier to test. I hate seeing global scopes combined with local scopes of functions. Literally zero reason not to pass it unless you need to change the value. Even then just return the new value and set it after. Even in react I don’t set state inside a function unless I pass the setState through params cause I think it just feels better. Or if the whole code base is doing it one way I follow that
doesnt that also make a copy of the global var? like any changes to the global var wont be kept outside of the function's scope
What if I want to make changes to the global variable? I'll have to use global or globals() right?
Right. If you want to make changes to the global_var using the for-loop, you could still use this method and after the for-loop is finished, re-assign the global_var with the new local_var. This would prevent searching globally each for-loop iteration.
How'd you customize your terminal to not show file path and so on?
Thought you were going to return ans*global_var instead of multiplying by global_var inside the loop.
Please can you show us how you check the performance fir both options
you can run the function 10k times and store the start and finish time with time.perf_counter()