Python OPTIMIZATION Trick!!
HTML-код
- Опубликовано: 5 окт 2024
- ⭐ Join the Byte Club to practice your Python skills! ($2.99/mo): / @b001
🐦 Follow me on Twitter: / b001io
Background Music:
Rain, Book And Cup Of Tea by | e s c p | escp-music.ban...
Music promoted by www.free-stock...
Creative Commons / Attribution 4.0 International (CC BY 4.0)
creativecommon...
![KSI - Thick Of It (feat. Trippie Redd) [Official Music Video]](http://i.ytimg.com/vi/At8v_Yc044Y/mqdefault.jpg)
Another alternative is to create a default parameter for the function.
I was just about to ask this. Thanks for pointing this out.
Can you give an example
@@shreehari2589def func_name(local_var):
when using function in main code
func_name(global_var)
@@shreehari2589def func(local_var = 10) You can call the function with a custom local var or use 10 as default if none specified
@@shreehari2589I think they mean pass the global_var as a parameter like def function(local_var): and then pass global_var when you call the function
It’s common practice to also avoid using global variables as much as possible. Only use them as constants.
can you explain why we should avoid global variable? I am new to python and been seeing alot of this information, but quite unsure how to workaround on that.
@@novianandrian global variables sow the seeds of terribly bad practices in learners, and experts never use them anyway.
the reason why is because new learners dont fully grasp what a function is. it isnt code you just put under a `def` block because thats what you think you are supposed to do. its not even just "repeatable" code. a function is essentially a "sub"program.
suppose you are making a library, and someone imports your functions. that function is a black box to them, and they cant see that it wants to use "x" from the global scope. if the function tried to, and they actually did have an "x" in their global scope, even though they didnt know that was part of the use case, they would almost certainly run into inaccurate outputs with absolutely zero clue as to where said output was coming from, and literally no possible way to debug that.
but thats not just to say that globals are bad only in the context of libraries, using global variables like this is genuinely harmful no matter what. they direct your code into a very specific architecture which is completely unmaintainable, unreadable and harder to debug. they promote misconceptions on what functions are and how they are meant to interact with a program. and you will never, ever be able to advance beyond basic scripting while you still maintain this habit, because projects that are better formatted in multiple files are practically a nightmare to debug or expand when even a few globals are being thrown around between scopes and files.
in short, one of the biggest things i would tell anyone at this point in their journey to learn programming is to make sure that in every circumstance your functions are able to be run on their own. all of the information they need to function fully should be passed in through parameters. if you feel as though you cant do this for your specific case, it may be a sign that your code architecture is wrong, as I have never had that actually be the case in over 6 years, nor have I heard of anyone who has.
@@novianandrianbecause it creates a lot of dependencies for your function, and makes the code less organized
either pass it as a paramater, or create a class and make the function a member of the class
using it for constants is fine
not using global variables is a good practice for all programming languages
@@the_cheese_cultist when you have code that might need constant change or update don't global variables make everything faster? Im doing my capstone college class and im working on a website that deals with different aspects of CSV data surveys. and i approached it with the mindset of "hey making everything global makes it easier to edit later. (side not it was inherited from last year and the whole approach the other team started with was a mess so we could only focus on refactoring.)(we worked with ruby on rails )
@@stilyandimitrov39 I'm gonna need more info to give you an opinion, but there are many sources online about good coding practices, avoiding globals included, that can explain everything better than a RUclips comment.
... or just pass it to the function as an argument, not only more efficient but also way easier to read
This doesn't have to semantically make sense at all
kinda annoying to keep track of the variables and has to unnecessarily place arguments everytime u call the function
This is common in Pure Functional Programming
@@vulpritproozeSo, you can set default parameters
I always get annoyed when looking at others' Python code and see something like: "object1 = object2".
I've even seen inside a function something like:
c_time = time.time()
current_time = c_time
Please explain to me why anyone would do something that seems so redundant?
best python optimization trick: switch to another language
thanks to this trick I managed to get a 50000% performance increase!
I switched to NodeJS and then bun.sh!
@@sanchogodinho lmaooo, yeah great... optimization I guess
Thank you❤I started learning binary code😍😍😍🥰🥰😍
@@firstnamelastname2775 nice, congratulations :)
"All that for a drop of blood?" - Thanos.
If you need enough performance that something like this becomes worth it, you now are in need of a higher performance language.
best reply imo
I mean having faster python is still better than slow python.
@@l3gacyb3ta21 I wouldn't say it's strictly better, it's a tradeoff - because to get faster python you have to decrease the readability by making your code appear a little more complex. You should choose Python when code readability/understandability is important and performance isn't important, because that is what the language is designed for. That's why it's a really good language for one-off quick scripts. But if you are in a problem space where performance is valuable, it's nice to use a language that's really designed for that from the get-go. Where you can actually use the language's nice features without surprise performance problems.
That being said, sometimes you inherit a codebase that's already in Python and performance may become an issue so sometimes you have to choose lower code readability for better performance.
Right
If you care about improvement to 0.1 miliseconds from 0.11 miliseconds you should probably not be programming in python ...
there's the `global` keyword, or `nonlocal` for nested functions
instead of creating local_variable, you can just declare `global global_variable` at the top of the function
exactly this video is super wrong
At least someone knows!
This can make the code very complex when using nested function, see mcoding's video
You have completely missed the point of the video. Hint: adding "global global_variable" at the top of the function doesn't change the bytecode at all. The whole (bad as it might be) point of the video is that accessing a global variable from the inner loop takes more time than loading the global variable into a local variable first.
@@joergsonnenberger6836 we can declare the variable as global inside the function before the for loop
I prefer passing the required values into the function as arguments. If I need to reuse a function in a different script, I know by the function signature exactly the function requires to run.
That's called dependency injection
Yep, DI is the way to go. Reuse your code and it makes it easier to read.
oh my god that is not dependency injection. that's just having a pure function
@@l3gacyb3ta21 Realized after two weeks but who cares i got two likes thanks to misinformation
I believe this is incorrect, any recent (python 3 and up) will emit a LOAD_FAST for locals (like ans and i) but a LOAD_GLOBAL for nonlocal, non-cell variables (like global_var). The speed difference between LOAD_FAST and LOAD_GLOBAL is pretty much negligible, especially with the adaptive opcodes in newer versions where most of these have faster, adaptive versions.
Yes this is correct. Also corroborated by the fact that the variable scope (aka what namespace python will search for a given variable) is determined at compile time before the interpreter will execute the byte code. Mcoding has a great video about this
No, Python does not check locally then globally. Python knows what is local and what is global at compile time. The difference is that globals is a dict, locals is an array, so local access is direct indexing vs dict look up. You can have a look at disassembly, when python determines the variable is local it uses LOAD/STORE_FAST, vs LOAD/STORE_NAME. The name is quite explicit which is more efficient.
That's what we call a premature optimization.
Except its done after the mvp and before it's too big to optimise?
@@revimfadli4666yeah, it’s probably a fine optimization if your program is just a counter multiplying by a global, but it is woefully insignificant in the scale of any program larger. Or, if your program is just a big loop, use PyPy. I remember hearing that it’s better at this stuff.
no, that's not
it’s just an example of
He literally profiled it. How's that premature??
def func(size, mult):
return mult * ((size**2 - size) / 2)
print(func(1000, 10))
then add some math :p
The best optimization trick for Python is to write the performance-heavy bits in C++ and use Python as a frontend
Amazing than I will try it tomorrow 👍
Nice! Thanks for sharing that optimization trick :)
If you want fast code, don't use python. No, don't optimize your python code, write anything that has to be fast in C and use pyhton for things that don't have to be fast.
This is not true. Variable scoping is done at compile time (yes, Python has one), not at runtime. It's actually the only complex thing Python compiler does.
You can look through the related data structures using the symtable module.
It's also visible if you use dis.dis on each version of the function - one will have LOAD_FAST op, the other will use LOAD_GLOBAL. It's predetermined.
It's predetermined but it doesn't mean it has the same cost. Lookup in local scope is faster for many reasons - it can be stored more efficiently, you can have more faith in what you are dealing with etc. For example, if you are dealing with something in outer scope, there are no guarantees about the value assigned to the variable as it may be changed in some other part of the code and global lookup has to be used to ensure that you are using proper value at the time you want to access it. If you copy the global value to a locally scope one, you can avoid that lookup as it is "detached" from its old variable. I don't know how lookups are managed for collections but if they are not explicitly copied, then I would assume they are stilled handled through global lookup rather than a local one based on the same problems with it being stored at a global level.
@@viCoN24 Your reasons are also wrong.
The object under reference is stored the same way no matter if it's local or global. "Outer scope" and "global" are two different things, but whether you have "guarantees" about the referenced value or not is irrelevant - cPython has no JIT.
It has global interpreter lock, so the dereference doesn't have to do nich Apart from finding the actual memory address.
I would invite you to profile this in an actual code, where the local scope has just as many items as the global one.
I think I missed your point. Thanks for the explanation!
You focused on the underlying mechanism where it's still one operation rather than two lookups. I don't know how these operations are performed by CPython but one is clearly faster than the other so I tried to provide some reasoning behind it but it's true that GIL simplifies the situation.
In principle the lookup resolution behaves as author explained even though the interpreter is able to scope it internally to decide on the proper operation rather performing dynamic lookup and failing at runtime when variable is not found. You are right that it's only one operation but whether it's assigned that one operation or the other is decided in the fashion described by the video maker.
@@KennyWlr Wow! Thank you so much for this detailed explanation. It's good to be wrong sometimes to learn this much. Cheers! :)
Are you sure? I've done a "benchmark" in Google Colab trying it, and the second case gains some performance indeed (Not too much)
A rule of thumb I have is to maximize clean encapsulation like in pure functional programming. Everytime I want to use a global state, I pass that state by reference.
This is so that you know which function is modifying what at any given time via the function calls.
It's really hard to spot any modification or references to global state if you directly refer to it within a method.
Bro I can't even here you over this beat. It's slaps more than I could have imagined 😂
Definitely use this trick to run out of variable names
Thanks. Yes and no about this method, Python is about writing understandable code, only big picture optimizations are needed (general algorithm). If you value even 10% loop speed increase, beter switch to C, Rust or Cython
that prt of code or
That is literally what I needed for a school project I'm working on. How unimaginably lucky. Thank you!
Were yall doing AST analysis for a school project 0_0
The same point applies to attribute lookup on imported modules. For example, you can assign imported function to a local variable to avoid an expensive lookup.
Let's say you use "math.sin" in a loop. Every time you call it, you perform a lookup on "math" module for "sin" attribute before calling that function. If you assign "sin = math.sin", you get rid of that unwanted lookup from "math" module and your loop with local "sin" will be faster.
If you have a function that creates a lot of objects, you might also want to assign the class locally to also avoid global lookup for the same reason.
If you find it interesting, you are dealing with some problem that Python might not be the best language to solve it in. Still, if you have to resort to those optimizations, it's at least nice to know about them.
In other langs, this fixes race conditions, as the code no longer assumes the variable won't be mutated.
The program no longer has to load from heap every iteration, since the value is already on the stack
This problem is the same in every interpreted programming language ( it doesn't affect performance on compiled langs cuz this only happends while compiling, not when running )
Thanks for that, im learning Python coding, so it will help me in improving coding skills
This is actually useful, I'll try to keep it in mind.
I think it is much better to make function with this 'global varible' as argument. Using global variables in functions is bad practise, because it makes functions less portable (you can't just take this function and place it in another program or file, because it may doesn't have this variable in it)
Better - when writing non trivial programs, it's often hard to avoid globals - unless you want to make the call graph a complete mess. If you're determined to kill all globals, you might want to malloc() a singleton (a struct) containing all of them (so you can re-enter the function without side effects) and pass it down all the way - even if you don't use it. That way if you change a function so it requires the use of such singleton, it's there. You might get a few warnings, but it works - and it makes maintenance less of a pain in the neck. BTW, always initialize your global vars in a separate function. You might be in for a surprise if you call it multiple times.
well actually, you should pass it as an argument
Is this why pointers are used in C and C++?
This is one of the reasons, yes. C and C++ are, by default, pass by value. That means when you pass arguments to a function the values are copied for the function to use.
Passing a pointer is still technically pass by value, but the value being copied is the *address* of the data; rather than the data itself. This allows you to save memory for larger variables (e.g., a big struct or a class). It also allows you to modify the original data, since you're being given a pointer to it rather than a copy of it.
Note that C++ has references, which are often used over pointers because they are safer (but are very very similar). If you're in C++, default to using references unless you have a reason to use a pointer (e.g., so you can call C functions).
For functions which are taking a pointer/reference to save time on copying data, you'll often see `const MyLargeDataType *mldt` or `const std:: string &str`. The `const` part signifies that the data being pointed to will never be modified by the function. Something to keep in mind also is that, for pointers, the following two are not the same: `const int *a`, `int *const b`. Here, `a` can be re-assigned to a new address of another `const int`. However, the contents at the address cannot be changed. For `b`, the contents at the address can be changed, but the address it actually points to cannot be changed. Here are some examples:
```
int c = 123, d = 456;
const int *a = &c;
int *const b = &c;
a = &d; /* okay, we can change where it points */
*a = 789; /* not okay, we can't change the data it ponts to */
b = &d; /* not okay, we can't change where it points */
*b = 1011; /* okay, we change the data it points to */
```
Note that in C++, all references are `const` (the address they point to cannot be changed), but you can still modify the data they point to so long as the type is not marked as `const`.
@@whirvis Thanks for the explanation. I'm trying to learn C, and pointers are a bit confusing to me, particularly when and why to use them. As I understand it:
int a; //declares a variable containing a value.
int *b; //essentially declares two variables, one contains a value, the other, an address.
*b = 123; //contains a value
b = &a; //contains an address
void Function(int *c){} //expects an address. Call like this:
Function(&a);
@@MortonMcCastle
int a; // designates enough memory to store an int but doesn’t say what int to store
int *b; // designates enough memory to store an address which, when dereferenced, should point to memory that stores an int (should not be thought of as declaring two variables because it’s only declaring memory for the pointer, not for any actual int value)
*b = 123; // is a very insidious bug because you’re modifying data at a random memory location because you never initialized b with a valid pointer so it’s interpreting whatever bits happened to be at that spot in memory before as a memory address. This is the kind of bug that hackers exploit to gain escalated privileges to systems.
b = &a; // b is now a pointer to a, so now it is safe to do *b = 123 which is equivalent to a = 123
Can you make a video on how you did the performance test at the end?
I dont know which he used, but he most likely used a profiler
Maybe cprofile or mprof or else lineprofiler I guess.
What I understood is to make that variable in local scope so as to prevent checking for the golabal variable each and every time. This reduces the time complexity
this optimization doesn't change time complexity, for that function it is O(n). time complexity refers to how well an algorithm scales when the amount of input data is increased. this optimization just reduces cpu cycles required to access the variable.
@@topzozzle6319 Thanks for the info sir
Interesting. Didn't know it impacts performance that much.
better optimization: redefine your function using integration as the increase is constant and determinant
Thank you for this short ^^
If you had an (inexplicable) reason to make the variable global, you wanted it updated within the function.
If you really need performance, do it in C. If you need it to fly, use assembly.
The best optimization trick in this case is not bringing the global_var into the local scope as local_var. It is going back to 5th grade and learning that you can simplify this function to
def func():
return global_var * 1000 * 1001 / 2
(**exceptionally hard to do**)
That makes two of us who recognized this was a summation problem so far.
I've been playing around with global variables lately and your tip is really helpful!
Using Global variables is not good, unless they are constants. If you need a value [especially a variable one] it should be passed as a parameter to the function, so that the function always have the same exact behavior when it is passed the exact same parameters. makes them predictable and individually testable.
@@olivierdulac Use of global variables is fine in some contexts esp when writing smaller scripts or module level variables
You're better off using the global keyword if worrying about scope
May I know how you have produced the graph of performance.
I would highly appreciate if you can create shorts or a detailed video on this.
I would like to test few other functions and Optimizations for personal use.
Thanks in advance.
Your first mistake was using python if you're worried about performance
It would be more optimal to calculate the answer rather than iterate. The sum from 1 to N is N*(N+1)/2. So your code could be changed to
global_var * N * (N+1)/2.
That was super insightful
Another optimization trick is not to use global variables
If you’re good at python, you should do the multiplication after the summation, since they’re separable.
The synthwave theme extension though. My man 👌
the best optimization is:
for i in range(1000):
ans += 10 * i
Great tip! I love your font. What is it?
This increase your program's speed while accupying more space for another variable.
Thanks!
I got enough performance with cpp thanks
amazing trick
I do this a lot when building a data-cleaning function. It also avoids mistakingly corrupting the global variable.
If you try writing to global_var it will create a local instance precisely to avoid corrupting it. You have to use the global keyword to specify global_var is global and be able to write to it. But really in the example in the video global_var should just be a parameter that is passed in. 99% of the time globals issues can be solved by just not using them and refactoring your code appropriately or starting with best practices from the beginning.
also lookup the phrase 'premature optimization'
make code harder to read for dubious gains
well but actually you don't use the global variable in the first place
Where’d this guy go I like these videos
Thanks for the tip
Never mention "performance" and "Python" in one item!
the main i see here is the for loop, when at that scale of iterations i'd look at a different way to implement that section of code first, for instance doing that specific part in a more optimized language
Parameters✨💪
Another W for compilers
Fun fact: a new scope is not created in a for loop
A) Calling it in the first place isn't unoptimal, the extra instruction to look outside the scope is completely irrelevant. B) you can use the global parameter. C) you should be passing it in instead if you're really worried about optimization.
Because you want to avoid an instruction to look outside the local scope you force it to declare another variable in the existing stack, how is that more optimal?
I didn't code in python for quite some time but my guess is, that it only allocated memory once and then keeps a close reference to that value, similar to how the register keyword in C works
Before giving tips to others, please make sure you are correct first. It is very obvious that you only have a surface level understanding of what is going on.
The reason this is faster is not because it only looks up the variable in one scope: That happens either way, since python's compiler emits a "LOAD_GLOBAL" if the variable isn't found within the local scope. However, global variable accesses are always\* a slow name-based dictionary access instead of a fast index based array access.
\* In Python 3.11, global dictionaries actually got optimized a bit, which is why the difference is only 10%. In older versions it should be a bit more IIRC.
hoping for more python optimization trick.
Can you do a video on how you test performance like that?
I thought that after first iteration, python would be clever enough to figure it out fir the next iteration.
cant wait for the livestream
Good bro😮
Insted of doing that just assign a variable inside the loop.😊
Could you give details about how to test the performance of a code?
Man being new to Python and new to coding just sounds like an awful time. Coming into python with years of statically typed OOP these are just things I do automatically lol
you should pass the variable into the function.
Sorry this is just very bad. Really, really, really bad. NEVER teach people how to optimize the use of global variables. Teach them how to avoid them alltogether.
Right? I thought he was going to say to pass it into the function as a parameter, since he said “new to python” but I guess not lol
look it's bad practice but people use globals and sometimes they provide value too
Especially his changes also changed the whole program's behaviour as Global_var is no longer updated.
Question, how is he even getting the global variable to work in his function without write
global (var_name)
at the top of the function?
so you never use a global variable / constant in your code?
When nanoseconds make a difference.
Hey, how can I make my Terminal look like yours? Or can you someday do an general showcase of your visual stuff? Love your videos.
This looks like VScode with maybe the Dracula theme?
Workflow optimization trick: switch the programming language to brainfuck as you only need 8 different symbols resulting ina a much easier coding experience.
Actually it resolves variable locations at compile time, yes python has a compile time. He fos the for loop, it resolves it once and then knows where to get it for each loop.
But only for locals. Globals are always a dictionary lookup.
Another optimization technique is to write code in assembler
We need more tutorials where are you???
Or just don’t use globals, only if they’re constant. Python doesn’t specifically implement constants but you can denote them with capitalization.
In this exact specific case, there is something even more simplier, no action are being done on the global_var, so why creating one, just use directly the value of it
This is the why you must use memory managed languages such as c/cpp or go
Good luck training and hosting a large ML model on that
Alternative: use C
Cool but is there EVER a justification for a global variable?
here and from Brazil a question what is your theme I thought it was very good for viewing the code
Another alternative: implement caching in python interpreter? This should not be an issue.
This is the exact type of thing a compiler would optimize for.... oh wait
Did you also test the effect of declaring global_var as as a “global variable” by placing “global global_var” at the beginning of the given function?
Havenʼt done any tests, but I think this could also give a speedup (without introducing a second name for the same thing). Hmm :-\
My only issue is you didn't have your global variable in all caps to signalify it as a constant.
I prefer use a parameter in that case
Could you make a video how I could make a text document where there is a list and in my python program I want a list that is the list of the text document. I would really appreciate it. Thanks for the helpful videos.
Python has 'nonlocal' keyword for the lookup in global scope
or just use the global keyword at the start of the function
A person would think that python would ready pointers for all the available variables inside the scope like a decent compiled programming language
Replace the loop with something non-trivial and it can't guarantee that the module scope object binding doesn't change. Similar to how a C compiler could have to load the same pointer again and again as if it can't proof that it wasn't changed.
wow love it
I'm surprised it even allows you to use out of scope variables.
btw for that function couldn't you do some math trickery
If performance is an issue in your application and it’s written in python, python is the problem, and no matter how many micro optimisations like this you add, your app is still written in a very non-performant language.