Multiprocessing in Python: Locks
HTML-код
- Опубликовано: 9 окт 2018
- In this video, we will be continuing our treatment of the multiprocessing module in Python. Specifically, we will be making use of the "lock", or equivalently, "mutex" object in the multiprocessing module. We will see an example as to why you would want to make use of locks in your program.
According to the official documentation (docs.python.org/2/library/mul...
"multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows."
Software from this video:
github.com/vprusso/youtube_tu...
For more videos on multiprocessing:
bit.ly/lp_multiprocessing
Do you like the development environment I'm using in this video? It's a customized version of vim that's enhanced for Python development. If you want to see how I set up my vim, I have a series on this here:
bit.ly/lp_vim
If you've found this video helpful and want to stay up-to-date with the latest videos posted on this channel, please subscribe:
bit.ly/lp_subscribe
![CENTRAL CEE - GEN Z LUV [VERTICAL VIDEO]](http://i.ytimg.com/vi/VIbidJm9q34/mqdefault.jpg)
The reason for this "weird" behavior is that the += or -= operations are actually 3 operations, and not 1 (once it gets translated into machine ops). The three operations are: read from memory, add/sub, save to memory. Now suppose process A reads the value 500 and then gets suspended, and then process B reads 500, subtracts 5, and saves 495, then process A gets control again and add's 5 to it's 500, and saves the 505 in the place where 495 was, then you just "lost" a subtraction operation. This is a race condition, and you need synchronization/locks to solve it.
good clarification
+= or -= are not not atomic
Now it all make sense, thanks man!
Aha! I was wondering why the different answers are showing up since join() is used and mathematically a series of additions and subtractions should not affect the final answer no matter in what order they are performed BUT thanks to your explanation I get it. It all makes sense now. Thank you.
@@dacao9240 thanks. “Atomic” is a key term to remember in multiprocessing when shared data is involved. Thanks for the reminder.
Nice video! Just to note, that it's extremely important for you to release the lock.
To prevent forgetting (which is natural in lengthier functions), you should consider using a context manager on the lock; example:
def add_500_lock(total, lock):
for in range(100):
time.sleep(0.01)
with lock:
total.value += 5
Another great point. Thanks for the tip, and thanks for continuing to provide great and helpful comments! Cheers.
Many thanks to @@LucidProgramming and JOATTOAJ v2. You guys are a great combo. Love you for your contribution.
@@sudarsandm Thanks!
@@sudarsandm Thanks!
I am newbie when it comes to Python. Could you please explain what the “with” does here? Do I still need to use acquire() and release() or does the use of “with” ensure that functionality? Thank you.
Thanks for the multiprocessing series of videos. You make it easy to understand a complicated subject. Thank you.
Cheers! If you enjoyed and benefited from my content, please consider liking the video and subscribing to the channel for more content like this. If you would like to support the content creation on this channel please consider unblocking ads when watching my videos as this is how I support my time to make content. I hope to be putting out more similar videos soon!
I just wanted to say thank you for the great tutorials. They help me a lot and I've learned a lot from you.
Thank you!
Thank you! If you like my content, I've been working on some projects during the past couple of months. If you would like to stay up-to-date, please consider subscribing to my mail list. Also, if you haven't already, please consider subscribing!
I really appreciate the support, and if you want to support the channel, I do have a PayPal link (www.paypal.me/VincentRusso1) for donations that go directly to the creation of content on this channel.
I hope that the content I provide there will enhance the videos on my RUclips page.
bit.ly/lp_email
Great teacher 👨🏫 !!! Thank you very much
Thank you! If you like my content, I've been working on some projects during the past couple of months. If you would like to stay up-to-date, please consider subscribing to my mail list. I hope that the content I provide there will enhance the videos on my RUclips page.
bit.ly/lp_email
And, if you'd like to help support my channel, I've put together a Patreon page. If there are certain topics you'd like to see covered, would like to chat with me one-on-one, or would simply like to support the creation of content, I sincerely appreciate whatever you are able to contribute:
www.patreon.com/lucidprogramming
Thanks again for your kind comment, and I hope to see you around! Cheers :)
Wow - a million likes here - what a lovely teacher you are!
Cheers! If you enjoyed and benefited from my content, please consider liking the video and subscribing to the channel for more content like this. If you would like to support the content creation on this channel please consider unblocking ads when watching my videos as this is how I support my time to make content. I hope to be putting out more similar videos soon!
Very helpful sir😊
good video and explanation
Great video! I just have a pro-tip: You can use '%' to reference the file name while executing a command in VI, so you could just put '!python ./%' and it would run the script.
Awesome tip! Thank you for sharing that :)
Nice video sir
Thank you, Subhash! Thanks for watching, and have a nice day!
Great video, thanks. I think your explanation might be slightly off though. From what I understand, add and subtract are happening concurrently. Lock, locks the "total" value in memory, specifically for while the function is mutating the value. Lock will queue the operations that need to be performed on the same variable by making the function have to wait for lock.release() from the other function, before it can complete it's own lock.acquire().
Thanks, Benjamin. I appreciate the clarification and the kind words. Thank you for watching, and I will put this to use to continually improve the content on this channel. Thanks again!
@@LucidProgramming Actually, this is incorrect. The lock is an entirely separate object that knows nothing of the total value or any of the statements following an acquire() call. It simply manages its own state and ensures that when one process acquires a lock that another process trying to acquire it will block until the current holder has released it.
If you want to verify this just remove the lock calls in your subtract function while leaving them in the add function. If you run your code enough times you'll get errors for the final result. The locking only works because you have put the acquire and release calls surrounding the access to the shared total value in ALL of the code that is accessing it which ensures that it doesn't get corrupted.
@@dontworrybehappy5139 Thanks for the clarification and for the comment! Cheers!
It's a nice video, well done. However, I feel that you should have talk more about the Value object. And as wrote before, you should also talk about locking with a context manager. Other than that, I really like your videos.
Thanks, and I appreciate the feedback!
Awesome video!!!
Why add the locks only around the changes to total and not the for loops within the functions? Would it yield the same thing?
If you put the lock around the entire for loop, then the entire for loop will run to completion before letting anything else have the lock; in other words, it would do all of the adding operations before it can do any of the subtraction operations
It seems that one thing you pointed out about the locks in your program happens to be incorrect. You stated that you put the lock statements around both "total.value += 5" and "total.value -= 5" so that all the addition operations happen prior to all of the subtractions. I did some tests and this does not appear to be the case. What actually happens is that you are still alternating between addition and subtraction operations, but the lock prevents both the processes from accessing that variable at the exact same time causing the weird behavior we were experiencing before. If you put the lock statements right in the beginning and end of both the add_500_lock function and sub_500_lock function, then what you said would be correct.
Hi Brandon. That is an excellent observation and an oversight on my part that I could have certainly explained better. Thank you for pointing that out, I'm always trying to improve the content on this channel, and I appreciate comments like yours that help in that goal. Cheers!
@@LucidProgramming Glad to have helped. Thanks for making such great tutorials.
@@brandonlavigne3670 Cheers, thanks very much!
Ah I was wondering what caused these unreliable outcomes other than the order they come in, thanks for the explanation!
I have a doubt at 12:27 . You are locking and unlocking each time you add or subtract number.
So as soon as the lock is released during the add function the process of subtract function might acquire lock and subtract 1.
The statement you said at the timestamp is true only when you put lock above the for loop and release after executing for loop, for both functions. I might be wrong though.
The statement that all the adds are all done first is incorrect. It would be true if the join() call for the add process was made before the start() call of the subtract process, but then you wouldn't need the lock at all.
If we apply locks here, is it not gonna be sequential execution rather than multiprocessing
I think you made a mistake. The way you put the lock does NOT make all those operations execute sequentially, i.e., execute all the addition before subtraction. The lock only makes the add or sub atom, which means nothing else interrupts them. This is my understanding coz if you ever add a print at the end like this:
for i in range(100):
time.sleep(0.01)
lock.acquire()
total.value += 5
lock.release()
print(total.value)
You will get numbers alternating between 500 and 505. This is determined by your contrived way. Then if ever change the time it sleep, say one sleep for 0.01 and another 0.05, then you get something different
Hmm, I think I see your point. Thank you for mentioning that. If you want to make a pull request on the code on my Github, by all means, I would really thank you for that! Cheers, and thanks again for watching.
It doesn't even mean that you couldn't get a value of 495 at some point since as was demonstrated in previous videos in this series on multiprocessing, once you start the separate processes they don't necessarily get execution time in the same order that they were started. As you mentioned, the lock only guarantees that no other process can acquire the lock at the same time. If the other processes were to use the total Value variable without a lock around it, you could still get a final incorrect value.
@@dontworrybehappy5139 Interesting. Let me see if I understand your comment. What you are saying is if, let’s say, the sub function does not use acquire() and release() then it would be possible for it to modify “total” even when “total” is locked by the acquire() call in the add function. So every time a process needs to modify “total” it should acquire() first otherwise even if “total” is locked by process A some other process B can change it if B doesn’t call the acquire() function first but instead just goes ahead and assign it a new value. Did I get your point right?
@@paulah1639 Yes, it is the acquire() call which blocks when someone else has the lock. It will keep that process blocked until the process that holds the lock releases it with the release() call at which time any process which was blocking on acquire() has a chance at gaining the lock. Different languages and operating systems work differently on who might get the lock if several are blocking at the same time. Search for "thread starvation" for a good discussion on the topic.
@@dontworrybehappy5139 ok I will look it up. Thanks.
Thanks for the great video! If we use lock to make sure 'add' operation happens before 'sub', Do we still need to use join?
I don't believe so, but you might want to check to ensure it works with your code. Cheers!
@@LucidProgramming This is not the way join() works. The join() method indicates that the worker process/thread needs to "join" back with the main process/thread or its creator after it completes. This means that the main process/thread which calls join() needs to block and wait until the worker process/thread has completed before it can continue. So when you call join() on the add worker, the main process will stop after that and wait until the add worker is totally completed and exits before it continues and does the join() call on the subtract worker which will then wait until the subtract worker completes and exits before it continues its own execution.
If you want to verify this, just add additional delay in your sleep calls for the subtract worker and add print statements before both the join() calls.
Without the join() calls in your program, the final print from your main function would get called right away and the main function and process would exit causing the Lock and Value objects to be destroyed and causing errors in the workers the next time they tried to use them.
@@dontworrybehappy5139 thanks for the note. That is an interesting point. So it is always better to call join() before the main process ends or proceeds to other tasks to avoid unexpected errors caused from unfinished business from child processes.
12:27 says all adds are completed before all subs.
That is not true. Add a print after each function like print(f"{total.value} after add/sub") to see that it's actually interweaved and fluctuating around 490-510, but eventually ending on 500
Hi Nice, Videos. I have question. I have written a multiprocessing code, for some reason Parent process gets detached from child process. five process runs simultaneously. Can I know why parent process gets detached?
Hmm, not sure I quite understand. You might want to read the docs for more info on your question.
When at the portion of the video introducing the shared variable "total", I get the error: "line 7, in getvalue OSError: [WinError 6] The handle is invalid". When running the lock example of the code, the program never finishes, like it's stuck in an infinite loop. I tried your code on github and it produced the same results.
[ Python 3.7.2 ]
Are you by chance running via the terminal or via an IDE?
@@LucidProgramming Thanks for the reply LucidProgramming. I'm running via PyCharm and the code worked as intended with version 3.7.0.
I was also running in pycharm. If I run it from the command line it works. I wonder why I can't run it in the pycharm VM? This makes me second guess errors in pycharm.
Nice video.You should do series on concurrent programming in python using aysncio
Hi Shaheer. That's an excellent idea. For the time being, I'm going to continue with multiprocessing and multithreading, but asyncio would be nice to include as well. Thanks for the suggestion!
Hi. This is an excellent series. Adding asyncio to this series would make it comprehensive since asynchronous programming is the hot new thing made popular because of javascript. In python documentation asyncio is not very clear.
Thanks
Gotcha. Thanks for the tip, and I'll add it to the list! Cheers :)
I get the way programs run... Basically they run from top to bottom unless it's a function then same idea the function must be defined before it's called so that the program can then know where to jump to but when I look at any of this code it doesn't really make much sense.
Is there something I'm missing?
I though the first video made sense but I think I have to watch that again because at some point in the second video I feel like I got lost because from the beginning of this video I felt like I was on shaky ground..
Like the words you say sound like English and even after my brain processes those words the ideas and concepts seem to make sense but... If I were to write a program right now I wouldn't be sure that it would do what I wanted.
Hi AnteConfig. It sounds like you have an understanding of the way in which a serial program executes but are having an issue with multiprocessing. Without knowing precisely where your disconnect is, it's hard for me to know how to help. I do offer consulting and would be happy to set up a remote session to help you work through this subject. Feel free to reach out to me if this is something you'd like to explore. Outside of that, I would continue either rewatch my videos or check out some other multiprocessing resources. Hope that helps. Cheers!
@@LucidProgramming No I'll be ok. I've seen people using threading and multiprocessing before. My problem is that I just can't see how the program will flow. I haven't tried to write anything yet so I think that's what I have to do. I'll try and fail until I start seeing a pattern. Thanks again.
@@anteconfig5391 It's a bit counterintuitive at first. Stick with it, and I'm sure it will start to make sense. Best of luck, and thanks for the comment!
the code is working fine , but i noticed as pointed in 7:22 if I switch the two lines making the join after start so in that case the result is consistent of total.value as 500, hence no need to use the lock, am I missing something important about Lock?
Perhaps run it a few times and you will see a discrepancy?
@@LucidProgramming thanks for the prompt reply, keep making good videos :)
@@mohamedhindam1793 Thanks, cheers!
9:21 - you kind of missing the point. We don't care if "add" happens first, we just care that whichever operation happens first - completes the transaction before another operation starts - they can be totally out of order. Race condition happens when one function reads, then another function reads before the first function had a chance to write it's output. A more clear example would've been:
tmp = total.value
tmp = tmp+1
total.value = tmp
Ah, I appreciate the clarity of your example. Thanks for pointing that out, and thank you for bringing that to my attention. I'm always trying to improve the content and delivery of what I do on this channel, so thank you for the valuable information!
@@LucidProgramming Please keep making great videos! Online tutorials are extremely useful! :)
@@not_a_human_being Cheers, thank you! :)
I got the same question too. I think in the video the purpose of the lock is to make sure one process finishes addition before another process uses the value for subtraction. The two processes are running concurrently still
@@xh3992 yeah, you're right.
At the no_lock stage of this video, it always prints 500 as the final value on my machine. I do not output the 505 or the 520 or any other number. Using the join method, does this not imply that ALL +5 sums and ALL -5 sums must be completed before printing the resultant total, which is then 500? So is the lock irrelevant here? Different results to your output (Thx, superb vids!) (I should add that I am using Python 3.6.5, Pycharm and Pycharm terminal)
Hi Mark. Hmm, I'm not quite sure I understand. Are you saying that you're seeing different output than what I'm seeing here? Did you make sure that you ran the same code? The code that I use is on Github if you'd like to verify. Sorry if I don't quite follow the question.
@@LucidProgramming No problem - my code runs fine, is exactly as yours, but only ever outputs the correct value (500), not any other values due to an unlocked shared resource, as you demonstrated. I assumed that to get any other value than 500, as your code did, the add/subtract functions cannot both have completed, which means that the join statements could not be executed, which means that the print value statement could not execute...but my assumptions are obviously wrong somewhere...Really superb series here, am trying to implement MP now having viewed all 6 vids - sincere thanks...m
@@markd964 Hmm, as a shot in the dark, what OS are you running?
Thanks as well for your comment. I'm glad to hear that this series has served as a benefit to you. If you like my content, I've been working on some projects during the past couple of months. If you would like to stay up-to-date, please consider subscribing to my mail list. Also, if you haven't already, please consider subscribing! I really appreciate the support. I hope that the content I provide there will enhance the videos on my RUclips page.
bit.ly/lp_email
@@markd964 Hmm, as a shot in the dark, what OS are you on?
Thanks as well for your comment. I'm glad to know that this series has served to benefit you. If you like my content, I've been working on some projects during the past couple of months. If you would like to stay up-to-date, please consider subscribing to my mail list. Also, if you haven't already, please consider subscribing! I really appreciate the support. I hope that the content I provide there will enhance the videos on my RUclips page.
bit.ly/lp_email
@@LucidProgramming Python 3.6.5, Pycharm and Pycharm terminal on Windows7.64bit
addition/subtraction and multiplication/division would have been a nice example. (100+5)*2 != (100*2)+5
Sure, that would be nice as well. Thanks for the comment.
So lock() turns multiprocess to multithread..........
Nope. Two different things which have nothing to do with locking.