Nice vid and very much liked the accuracy of using __main__ to stop the spawned processes recursively generating infinite processes. It's a subtlety that is easily overlooked.
Great Video Sir … very well explained …. I wish I had seen this video a few weeks back. Now I understand that if __name__ == “__main__” is vital to multiprocessing. Now I understand. Thanks a ton Sir ❤❤❤
@8:23 my time "Finished running after seconds:" is always above 2 seconds, the first two prints are almost identical, just the last print from multiprocessing is always above 2 seconds, any ideas why?
Great content!!! I have a question that needs ur kind help. I am using python multiprocessing module to write data out to multiple files at the same time to speed up the writing process. But the output files end up having mixed data. For example, I want to write dict[1] to file1.txt, dict[2] to file2.txt, ......etc, But the problem is file2.txt has some data that was supposed to be in file1.txt. Any knowledge on why this could happen? i have been googling for a while but the only potential evidence is that someone mentioned the file writting is not thread safe, so some data could be flying between different process of writting. Any knowledge on this? I would really appreciate ur kind help!
I thought this would work for my script, but unfortunately I'm having trouble returning the values I need from my functions which run in parallel. My functions don't return integers or floats, but pandas dataframes (2 dataframes in fact). I unfortunately have not found a way to return these dataframes. Would you know how to return complex objects using the multiprocessing method? UpdatE: nvm your second video fixed my problem. Thank you so much!!! Subscribed and liked for your help
Awesome explanation bro, I have a question... If u create 10 different scripts(without using multiprocessing) and run them at the same time, will they run on a single core or be distributed on all the cores available?
@@johangodinho btw Please make video about making GUI with PyQt (I watched all of your Tkinter tutorials but i think PyQt is better than Tkinter in most cases).
Hi there, Great video. I have a little question. I have a program that has a while loop and a for loop. I want to run them simultaneously. Do you how I can do that
AttributeError: partially initialized module 'multiprocessing' has no attribute 'Process' (most likely due to a circular import) i'm using pycharm & python 3.8
Make sure that you have not called your python file multiprocessing.py or anything similar, since it will confuse it to be the actual module otherwise.
I have a process that opens a file, writes to it, then blocks until serial data comes in. However, if I try to Cntl-C on it, I get an "QObject" timer error?! import serial import signal #TERM (kill) INT (Ctl-C) from time import time, sleep #Or just import time, and call time.sleep(x) import sys from multiprocessing import Process from multiprocessing import Queue as mpQueue import matplotlib.pyplot as plt import matplotlib.animation as animation # File Handle sysStdFile = None #Dummy Process1 opens/writes to file, waits for serial data def Process1(qdata): with serial.Serial("COM29", 57600, 8, "N", 1, timeout=None) as ser: filename = "Hi.csv" sysStdFile = open(filename, 'w') sysStdFile.write ( "file data!" ) i = 0 while True: #block until a byte comes in c = ser.read(1) # ** Uses QTimer?? AND PREVENTS FILE FROM CLOSING!! ** i = i+1 qdata.put ( i ) sleep( 1 ) #Dummy Process2 that opens a matplotlib figure def Process2 (qdata): plt.ioff() fig = plt.figure(figsize=(1,1)) ax = fig.add_subplot(111) plt.show() while True: qdata.get () sleep( 1 )
if __name__=='__main__': # Know when to quit t = Terminator() q = mpQueue() # Start 1st Process p1 = Process(target=Process1, args=(q,)) p1.daemon = True #Must be done BEFORE .start in order to kill child when main process dies p1.start() # Start 2nd Process p2 = Process(target=Process2, args=(q,)) p2.daemon = True #Must be done BEFORE .start in order to kill child when main process dies p2.start() # Dummy super-loop while not t.kill_now: sleep( 1 ) print (".") # Exit Gracefully print ("Keyboard Interrupt!") sysStdFile.close ( ) q.close() q.join_thread() #Every queue creates a thread processReadPort.terminate() processReadPort.join() # "wait for this [thread/process] to complete processPlotter.terminate() processPlotter.join() # "wait for this [thread/process] to complete sys.exit(0)
Thia Video is wrong. It only shows, that you have no understaning of multiprocessing. You have a terrible code style too. A sleep is showing nothing ....you can start as many sleeps as you want. Only cpu tasks can Show you Performance gains!
Absolutely amazing, cannot wait to try it!
Nice vid and very much liked the accuracy of using __main__ to stop the spawned processes recursively generating infinite processes. It's a subtlety that is easily overlooked.
Awesome video, I was looking for a tutorial for this.
Hello Johan. How about you want to store the value you produced when you run the functions. How do I code that one?
Nicely explained bro thanks 🙌
This is an absurdly good tutorial.
Great video. Thanks!
Great Video Sir … very well explained …. I wish I had seen this video a few weeks back. Now I understand that if __name__ == “__main__” is vital to multiprocessing. Now I understand. Thanks a ton Sir ❤❤❤
thank you very much!
thank you, you just saved me a two day headache.
hey! multiprocessing don't work when compiled to an .exe file so if you want multiprocessing in an executable file you should use threading library
👋🔥🙏🏿🙌🏾👏🏿🙌🏾LIFE SAVER👌🏻🙌🏼✌🏿THANK YOU
thanks bro
How many processes can you have running at the same time? Also, what exactly is the purpose of having the __main__ thing?
@8:23 my time "Finished running after seconds:" is always above 2 seconds, the first two prints are almost identical, just the last print from multiprocessing is always above 2 seconds, any ideas why?
Hi sir it was useful, thanks a lot. Can I ask a question, how to write a parallel program for spell checking in python
Will upload soon
What's the difference between multiprocessing and multithreading?
Is it possible to take 2 say 200 line codes into one, because when I just try to copy and paste it into one on visual studies, one of them darkens
Great content!!! I have a question that needs ur kind help. I am using python multiprocessing module to write data out to multiple files at the same time to speed up the writing process. But the output files end up having mixed data. For example, I want to write dict[1] to file1.txt, dict[2] to file2.txt, ......etc, But the problem is file2.txt has some data that was supposed to be in file1.txt. Any knowledge on why this could happen?
i have been googling for a while but the only potential evidence is that someone mentioned the file writting is not thread safe, so some data could be flying between different process of writting. Any knowledge on this? I would really appreciate ur kind help!
I am also stuck on this same thing
I thought this would work for my script, but unfortunately I'm having trouble returning the values I need from my functions which run in parallel. My functions don't return integers or floats, but pandas dataframes (2 dataframes in fact). I unfortunately have not found a way to return these dataframes. Would you know how to return complex objects using the multiprocessing method?
UpdatE: nvm your second video fixed my problem. Thank you so much!!! Subscribed and liked for your help
Awesome explanation bro, I have a question... If u create 10 different scripts(without using multiprocessing) and run them at the same time, will they run on a single core or be distributed on all the cores available?
Anyone??
@@AlgoCoder I have been wondering about this too.
Maybe you can check on task manager(win) or activity monitor(mac)
I think it will use single core, because python has 1 core by default
why i copy whole code and 'finish' returns 64734.0125234 seconds instead of 0.5 or 1 or just regular values?
This is cool, but I can’t find a good way to be able to actually get the return value of the function, just execute it.
I'm waiting it. I like the video even before it's published. Because i know Who made it, there's no way for it to be bad.
haha. That's nice of you to say :))
@@johangodinho btw Please make video about making GUI with PyQt (I watched all of your Tkinter tutorials but i think PyQt is better than Tkinter in most cases).
@@weeg6361 Alright, will surely take a look into it and post something on that topic very soon. Thanks for the suggestion though :))
@@johangodinho You are so polite, Thank you 🙏🏻
That keyboard tho
Bruh how me get heart? This video is old like the bread at the back of the fridge
Hi there, Great video. I have a little question. I have a program that has a while loop and a for loop. I want to run them simultaneously. Do you how I can do that
hi Sir I want to know the answer too :)
AttributeError: partially initialized module 'multiprocessing' has no attribute 'Process' (most likely due to a circular import)
i'm using pycharm & python 3.8
Make sure that you have not called your python file multiprocessing.py or anything similar, since it will confuse it to be the actual module otherwise.
@@johangodinho your source file name was multprocessing.py. i got it.
@@user-kl5zi8qx4b no I named it multprocessing it doesn't have an "I" I misspelt it on purpose :))
Nice video
Thanks
thanks
No problem :), do consider joining the discord server its a small community of programmers who are willing to help :)
discord.gg/UVdVBtR
F A B U L O U S
I have a process that opens a file, writes to it, then blocks until serial data comes in. However, if I try to Cntl-C on it, I get an "QObject" timer error?!
import serial
import signal #TERM (kill) INT (Ctl-C)
from time import time, sleep #Or just import time, and call time.sleep(x)
import sys
from multiprocessing import Process
from multiprocessing import Queue as mpQueue
import matplotlib.pyplot as plt
import matplotlib.animation as animation
# File Handle
sysStdFile = None
#Dummy Process1 opens/writes to file, waits for serial data
def Process1(qdata):
with serial.Serial("COM29", 57600, 8, "N", 1, timeout=None) as ser:
filename = "Hi.csv"
sysStdFile = open(filename, 'w')
sysStdFile.write ( "file data!" )
i = 0
while True:
#block until a byte comes in
c = ser.read(1) # ** Uses QTimer?? AND PREVENTS FILE FROM CLOSING!! **
i = i+1
qdata.put ( i )
sleep( 1 )
#Dummy Process2 that opens a matplotlib figure
def Process2 (qdata):
plt.ioff()
fig = plt.figure(figsize=(1,1))
ax = fig.add_subplot(111)
plt.show()
while True:
qdata.get ()
sleep( 1 )
# Exit Gracefully
class Terminator:
kill_now = False
def __init__(self):
signal.signal(signal.SIGINT, self.Terminate)
signal.signal(signal.SIGTERM, self.Terminate)
def Terminate(self,signum, frame):
self.kill_now = True
if __name__=='__main__':
# Know when to quit
t = Terminator()
q = mpQueue()
# Start 1st Process
p1 = Process(target=Process1, args=(q,))
p1.daemon = True #Must be done BEFORE .start in order to kill child when main process dies
p1.start()
# Start 2nd Process
p2 = Process(target=Process2, args=(q,))
p2.daemon = True #Must be done BEFORE .start in order to kill child when main process dies
p2.start()
# Dummy super-loop
while not t.kill_now:
sleep( 1 )
print (".")
# Exit Gracefully
print ("Keyboard Interrupt!")
sysStdFile.close ( )
q.close()
q.join_thread() #Every queue creates a thread
processReadPort.terminate()
processReadPort.join() # "wait for this [thread/process] to complete
processPlotter.terminate()
processPlotter.join() # "wait for this [thread/process] to complete
sys.exit(0)
😘😘😘😘😘😘😘😘
what happen i no understans
Why do you say “cool”?
Thia Video is wrong. It only shows, that you have no understaning of multiprocessing. You have a terrible code style too. A sleep is showing nothing ....you can start as many sleeps as you want. Only cpu tasks can Show you Performance gains!