I'm aware that this video is kinda old, but people might still stumble across this series. I wanted to correct a slight misunderstanding concerning parallelism of threads. The behavior might not be what you expect. This addresses the slide at 20:25. I'll try to keep this short and just scratch the surface of things. The Python interpreter is using something called GIL, or Global Interpreter Lock. Google that up if you are interested in the details (you might stumble across some in depth discussions), but in the end it results in the issue that only one thread can be executed at any given time. Or in different terms: While Thread T1 is running, all the rest of your threads is fast asleep. So this differs from the behavior you might be used to from languages such as C# and the likes. Let's say T1 (your main thread) is your UI, and T2 does some background stuff, say writing a file, as in the example of this video. There might be no issue: While T1 is idle and waiting for you to input something, another thread (in our case T2) can work in the background and use existing processing power. At the same time, your scheduler can still switch back to the UI when it is needed, so the UI thread is not blocked by the file-writing action, and the User Interface stays responsive. Note: While the UI is processing your inputs, the other thread is paused. But maybe you have a different scenario: Maybe there is an expensive deep learning based function running in the background. All the while, another thread manages input from a camera (for example). And we have our main UI thread. So: We might have several entities which we want to (actually) run in parallel (on different cores of your CPU). This is not working with (python-)Threads. While the UI is active, both the camera and the DL stuff cannot work. Or while the "camera" is digesting data, the classification framework has to be idle (as is true for the UI main thread). Coming from another programming language this might be a huge deal for you! If you want "true" multi-threading, you might want to have a look at what is called "multi-processing". These processes will then in turn all have their seperate GIL and thus can actually run in parallel. The issue is, sharing data gets more complicated. Tools such as Queues and Pipes have been introduced for that purpose. Processes need a certain amount of overhead - so be sure what you do when using processes, since they might not bring the kind of speedup you want. If it is only to use processing power while your UI is idle - threads might be great. But when you have several heavyweight actions running in parallel, processes might be what you want to go for. Not sure if there will ever be a better alternative, but that is what we have as of now. Cheers!
great job. I've spent days on the internet to understand the basics of threading and this is seriously the only way I could approach the subject. it was simple and clear. seriously, you have made the internet a better place.
Very nice tutorial! I'm not exactly a Python newbie, but I'm new to threading in Python and this concise tutorial helped me quickly understand how it works. Easy as Py, if you'll excuse the bad pun. Thanks!
+Letmecode Front-end These are the advanced tutorials. So I would hope you're not a newbie! :D I'm glad this was easy to follow! Thanks for taking the time to let me know you enjoyed it!
Thanks a lot for making this video. Very helpful. Have a interview for a python developer and this video helped me in learning the concepts.Thank you once again
You are Great I watched a lot of video about threading, but i really didn't understand. Your English is excellent!!! don't give up making videos! i am your fan! Thanks a lot!!!
I thought that Python doesn't support using more than a single core? You can create more than a single thread, but they won't be executed in parallel due to Global interpreter lock. Instead they would be executed one after another. Could someone confirm it? Nevertheless, using multiple threads is still useful.
You're right about that! I've seen your comment only after I've added my thoughts about that, otherwise I would have added this to your comment. (see ruclips.net/video/6eqC1WTlIqc/видео.html&lc=UgyySjRv7NWttJrpWhd4AaABAg)
Thank you so much for you valuable videos! I already watched your class and method tutorial vids, so I know that __init__() method is like a constructor in other languages like C#. When a user create an object of AsyncWrite class, they can initialize the object by just passing the parameters to (). I got that. However, why would we need 'threading.Thread.__init__(self)'? Does Thread class has __init__() method? why do we need this part inside __init__()? Thank you always!
+Busy Learner I'm glad you like them! We use init, to use what they have already made, but add out functionality on top. So we use it for taking a file to asyncWrite too. :)
+Busy Learner Everything in python is an Object. So every object of Python as access to __init__ method. You can simply write >>> a=2 >>> a.__init__ #This will give the __init__ method address #You can also invoke the method >>> a.__init__()
I do understand what you were trying to do when explaining locks, but it sorta defeats the purpose of the lock and it is a bit difficult for beginners to see how this is going to be implemented to get the benefit of threading. The basic function of the lock is to block variables from being accessed by other threads while one thread is using this variable (variables in this case also includes functions). The code in this example includes the time.sleep() function within the lock, so all other threads have to wait to use this function, eliminating the benefit of threading. To solve this, put the lock around the prints and the repeat variable update (repteat -=1) like below (within the while loop). # remove the tLock.acquire from outside the while loop while repeat > 0: time.sleep(delay) tLock.acquire() print(name+ " har acquired the lock") print(name+": "+str(time.ctime(time.time()))) repeat -= 1 print(name+" is releasing the lock") tLock.release() This will allow all the threads to access the time.sleep() function, but they won't print stuff on top of each other, except maybe the print("Timer: "+name+" completed") thing. In general, put locks around variable updates, printing and function calls that may cause collisions when done at the same time.
Thanks but here's my thought: In this example demonstrating lock, you actually nullified the whole advantage of threading and made your example simply sequential in effect, which first the first thread thread starts and ends and then a second thread starts and end. This is not good. although you can use locks to guarantee a specific order for a set of tasks at hand, I highly doubt this was your initial intention here, it seems just an unfortunate outcome of a simplistic example you chose to demonstrate locks with. Locks are especially needed during simultaneous writes, so if all you do is printing/reading stuff, you should be fine, unless your sequence of operations requires some form of order as I said before.
DrapsTV I'm not sure to be honest, haha, I was readng how it's useful in ethical hacking? And that it's the best first programming language to learn, i'm sure there's more, it just seems like a fun language to learn :), btw your videos are awesome.
A doubt...Doesn't a lock defeat the whole purpose of a multi threading if say thread 2 has to wait for thread1 to release the lock? I thought threading was used to run code concurrently.. Correct me if wrong Thanks
Definitely doesn't defeat the purpose. In the example I use it's more to show how the lock works in action. There are often times when you need threads to sync up.Accessing shared data is a common one. writing is slower than reading and it's quite possible for one thread to be mid-write when another thread reads. (This becomes very apparent when writing to disk!) Making it possible to read jumbled data. Eliminating how often you need to do this is always a plus. If a thread doesn't need to ever interact with another thread. Then most of the time you won't need to use locks.
hey, I hope that someone will answer me here, but in this lock example, with the way he used the lock, the program behaves like it's single threaded. so what is the best practice to use a lock and still use the advantages of multithreading?
The thing about multithreaded software is that shared data can be accessed at anytime. So order can not be guaranteed. Let's say we have 5 concurrent threads and they need to process their data and then append their result to an output file. file handles are not thread safe. to ensure that two threads don't try write to the stream at the same time we would use a lock. Here is an article on synchronizing access if you're interested: effbot.org/zone/thread-synchronization.htm
DrapsTV I understand the reason we need to lock the threads, and thanks to you tutorial and that article you sent I also understand how to use locks, the thing is that when you use it the way you did, or any other way by my understanding, you ruin the purpose of having several threads because they now are dependent on each other
DrapsTV if we stay with your example from the video, in my comprehension of multithreading you would want both the threads to run in a parallel fashion
hi you seem pretty good at python I'm trying to make a tcp socket server with a counter that counts how many connections are connected but I cant make the counter go down when the connection disconnects please help while True: conn, addr = sock.accept() clients = clients + 1 start_new_thread(client_thread, (conn,)) sock.close()
most robust way would be to read using a select. but for your purposes you could probably just get away with a timeout try/catch. eg. try: data = conn.recv(1024) except socket.timeout: clients = clients - 1
you should check out the - HACKING SECRET CIPHERS WITH PYTHON by Al Sweigart, it has an excellent explanation for your doubt. (page 140 of the book to be precise)
This shit ain't working bro lol Exception in thread Thread-1: Traceback (most recent call last): File "C:\Users\Austin\AppData\Local\Programs\Python\Python36-32\lib\threading.py", line 916, in _bootstrap_inner self.run() File "C:\Users\Austin\AppData\Local\Programs\Python\Python36-32\lib\threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "C:/Users/Austin/PycharmProjects/test2/timer.py", line 8, in timer tLock.aquire() AttributeError: '_thread.lock' object has no attribute 'aquire'
I'm aware that this video is kinda old, but people might still stumble across this series.
I wanted to correct a slight misunderstanding concerning parallelism of threads. The behavior might not be what you expect.
This addresses the slide at 20:25.
I'll try to keep this short and just scratch the surface of things.
The Python interpreter is using something called GIL, or Global Interpreter Lock.
Google that up if you are interested in the details (you might stumble across some in depth discussions), but in the end it results in the issue that only one thread can be executed at any given time.
Or in different terms: While Thread T1 is running, all the rest of your threads is fast asleep.
So this differs from the behavior you might be used to from languages such as C# and the likes.
Let's say T1 (your main thread) is your UI, and T2 does some background stuff, say writing a file, as in the example of this video. There might be no issue: While T1 is idle and waiting for you to input something, another thread (in our case T2) can work in the background and use existing processing power. At the same time, your scheduler can still switch back to the UI when it is needed, so the UI thread is not blocked by the file-writing action, and the User Interface stays responsive.
Note: While the UI is processing your inputs, the other thread is paused.
But maybe you have a different scenario: Maybe there is an expensive deep learning based function running in the background. All the while, another thread manages input from a camera (for example).
And we have our main UI thread.
So: We might have several entities which we want to (actually) run in parallel (on different cores of your CPU). This is not working with (python-)Threads. While the UI is active, both the camera and the DL stuff cannot work. Or while the "camera" is digesting data, the classification framework has to be idle (as is true for the UI main thread).
Coming from another programming language this might be a huge deal for you!
If you want "true" multi-threading, you might want to have a look at what is called "multi-processing".
These processes will then in turn all have their seperate GIL and thus can actually run in parallel.
The issue is, sharing data gets more complicated. Tools such as Queues and Pipes have been introduced for that purpose.
Processes need a certain amount of overhead - so be sure what you do when using processes, since they might not bring the kind of speedup you want. If it is only to use processing power while your UI is idle - threads might be great. But when you have several heavyweight actions running in parallel, processes might be what you want to go for.
Not sure if there will ever be a better alternative, but that is what we have as of now.
Cheers!
great job. I've spent days on the internet to understand the basics of threading and this is seriously the only way I could approach the subject. it was simple and clear. seriously, you have made the internet a better place.
Very nice tutorial! I'm not exactly a Python newbie, but I'm new to threading in Python and this concise tutorial helped me quickly understand how it works. Easy as Py, if you'll excuse the bad pun. Thanks!
+Letmecode Front-end These are the advanced tutorials. So I would hope you're not a newbie! :D I'm glad this was easy to follow! Thanks for taking the time to let me know you enjoyed it!
Thanks a lot for making this video. Very helpful. Have a interview for a python developer and this video helped me in learning the concepts.Thank you once again
Thanks! Good Luck in your interview!
You are Great
I watched a lot of video about threading, but i really didn't understand.
Your English is excellent!!!
don't give up making videos!
i am your fan!
Thanks a lot!!!
"You're English is excellent!!!". That's because he's from Australia (? I think), and English speaking country
Great video
I feel that I learned a lot in this video
+Sami Kanafani That's awesome to hear mate!
It was a clean tutorial. Really enjoyed it!
I just found you on RUclips on my iPad anyways I am really enjoying your videos. They are very useful and easy to understand. Stay awesome. -A new sub
Thank you so much for this video! It just helped me solve a problem I've been working on for a few days!
Thanks for the vid, could you explain why you have to do threading.Thread.__init__(self) ? Wouldnt super.init(self) also work ?
great video dude:)
don't give up
I thought that Python doesn't support using more than a single core? You can create more than a single thread, but they won't be executed in parallel due to Global interpreter lock. Instead they would be executed one after another. Could someone confirm it? Nevertheless, using multiple threads is still useful.
You're right about that! I've seen your comment only after I've added my thoughts about that, otherwise I would have added this to your comment. (see ruclips.net/video/6eqC1WTlIqc/видео.html&lc=UgyySjRv7NWttJrpWhd4AaABAg)
Hey! man...
Come Back and start Uploading Videos.
You are going great Man.
Wonderful tutorial, Thank you very much! But why did you use a while loop instead of a "for i in range(x):"?
Probably felt like writing a while that day :) too long ago to remember
I think it's nice if you already have the variable initialised because it just feels right.
Can you simply explain the "self"? I have read that it's a convention but i don't understand it. Great videos btw
Thank you so much for you valuable videos!
I already watched your class and method tutorial vids, so I know that __init__() method is like a constructor in other languages like C#. When a user create an object of AsyncWrite class, they can initialize the object by just passing the parameters to ().
I got that. However, why would we need 'threading.Thread.__init__(self)'?
Does Thread class has __init__() method? why do we need this part inside __init__()?
Thank you always!
+Busy Learner I'm glad you like them! We use init, to use what they have already made, but add out functionality on top. So we use it for taking a file to asyncWrite too. :)
+Busy Learner Everything in python is an Object. So every object of Python as access to __init__ method. You can simply write
>>> a=2
>>> a.__init__ #This will give the __init__ method address
#You can also invoke the method
>>> a.__init__()
I think lock is useful when im gonna run multiple processes at the same time while each process has micro processes that are dependent to themselves.
best python tutorial ever ...... great job :)
I hope u make a video about how to set up vim for python3
+Fadel Berakdar sudo apt-get install vim
I do understand what you were trying to do when explaining locks, but it sorta defeats the purpose of the lock and it is a bit difficult for beginners to see how this is going to be implemented to get the benefit of threading.
The basic function of the lock is to block variables from being accessed by other threads while one thread is using this variable (variables in this case also includes functions). The code in this example includes the time.sleep() function within the lock, so all other threads have to wait to use this function, eliminating the benefit of threading.
To solve this, put the lock around the prints and the repeat variable update (repteat -=1) like below (within the while loop).
# remove the tLock.acquire from outside the while loop
while repeat > 0:
time.sleep(delay)
tLock.acquire()
print(name+ " har acquired the lock")
print(name+": "+str(time.ctime(time.time())))
repeat -= 1
print(name+" is releasing the lock")
tLock.release()
This will allow all the threads to access the time.sleep() function, but they won't print stuff on top of each other, except maybe the print("Timer: "+name+" completed") thing. In general, put locks around variable updates, printing and function calls that may cause collisions when done at the same time.
@DrapsTV do you agree with this, or is there some reason you put the lock around the time.sleep(delay)?
Thx,Great tutorial .expect the next
Thanks but here's my thought:
In this example demonstrating lock, you actually nullified the whole advantage of threading and made your example simply sequential in effect, which first the first thread thread starts and ends and then a second thread starts and end.
This is not good. although you can use locks to guarantee a specific order for a set of tasks at hand, I highly doubt this was your initial intention here, it seems just an unfortunate outcome of a simplistic example you chose to demonstrate locks with.
Locks are especially needed during simultaneous writes, so if all you do is printing/reading stuff, you should be fine, unless your sequence of operations requires some form of order as I said before.
Great video!
Love this video. DrapsTV keep em coming :D
Hi so you can't use lock in dos? am i right?
i know it's lame, but Im learning python on codeacademy, i learned the basics so far, but my question is, what book(s) would you recommend afterwards?
JDuke98 Learn what ever way is easiest for you! :) I'm not sure what are you interested in doing with your new found programming skills?
DrapsTV I'm not sure to be honest, haha, I was readng how it's useful in ethical hacking? And that it's the best first programming language to learn, i'm sure there's more, it just seems like a fun language to learn :), btw your videos are awesome.
JDuke98 Thanks mate! Well If you are interested in hacking then that "Black hat python" and "grey hat python" are great books.
You are great
If we added t1.end(). What would this do?
A doubt...Doesn't a lock defeat the whole purpose of a multi threading if say thread 2 has to wait for thread1 to release the lock? I thought threading was used to run code concurrently.. Correct me if wrong
Thanks
Definitely doesn't defeat the purpose. In the example I use it's more to show how the lock works in action. There are often times when you need threads to sync up.Accessing shared data is a common one. writing is slower than reading and it's quite possible for one thread to be mid-write when another thread reads. (This becomes very apparent when writing to disk!) Making it possible to read jumbled data. Eliminating how often you need to do this is always a plus. If a thread doesn't need to ever interact with another thread. Then most of the time you won't need to use locks.
Oh okay..That makes sense..Thanks.
No Problem :)
hey, I hope that someone will answer me here, but in this lock example, with the way he used the lock, the program behaves like it's single threaded. so what is the best practice to use a lock and still use the advantages of multithreading?
The thing about multithreaded software is that shared data can be accessed at anytime. So order can not be guaranteed. Let's say we have 5 concurrent threads and they need to process their data and then append their result to an output file. file handles are not thread safe. to ensure that two threads don't try write to the stream at the same time we would use a lock. Here is an article on synchronizing access if you're interested: effbot.org/zone/thread-synchronization.htm
DrapsTV I understand the reason we need to lock the threads, and thanks to you tutorial and that article you sent I also understand how to use locks, the thing is that when you use it the way you did, or any other way by my understanding, you ruin the purpose of having several threads because they now are dependent on each other
DrapsTV if we stay with your example from the video, in my comprehension of multithreading you would want both the threads to run in a parallel fashion
The example is for demonstration purposes only. When learning about locks, you need to see what it's doing first hand.
Cool, but PEP8 please.
hi you seem pretty good at python I'm trying to make a tcp socket server with a counter that counts how many connections are connected but I cant make the counter go down when the connection disconnects please help
while True:
conn, addr = sock.accept()
clients = clients + 1
start_new_thread(client_thread, (conn,))
sock.close()
most robust way would be to read using a select. but for your purposes you could probably just get away with a timeout try/catch. eg.
try:
data = conn.recv(1024)
except socket.timeout:
clients = clients - 1
Thanks, is there any tutorial in Pycharm (windows)
its same on pycharm
finally!
ImportError: cannot import name 'Thread'
Don't call the .py file threading lol
import _thread
thx
I'm Getting This Error.., AttributeError: 'module' object has no attribute 'Thread' Please Help
Phani Raj make sure you imported/spelled the threading module correctly!
Why can't I just do Main() without that stuff at the end it works the same.
you should check out the - HACKING SECRET CIPHERS WITH PYTHON by Al Sweigart, it has an excellent explanation for your doubt.
(page 140 of the book to be precise)
uderscoreUndersocre I Love This video , thank you :) uderscoreUndersocre
The only way to learn this stuff would be to go through an entire PhD in computer science.
This shit ain't working bro lol
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Users\Austin\AppData\Local\Programs\Python\Python36-32\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "C:\Users\Austin\AppData\Local\Programs\Python\Python36-32\lib\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "C:/Users/Austin/PycharmProjects/test2/timer.py", line 8, in timer
tLock.aquire()
AttributeError: '_thread.lock' object has no attribute 'aquire'