Multiprocessing in Python: Introduction (Part 1)

Поделиться
HTML-код
  • Опубликовано: 18 окт 2024

Комментарии • 118

  • @joe_of_all_things
    @joe_of_all_things 5 лет назад +54

    The reason you define a single value tuple in the following manner (value, ) is due to the fact that if you did not finish it off with a comma, it would be considered brackets for the purposes of grouping such as used in BEDMAS for arithmetic. By finishing it off with a comma, you're explicitly telling Python that what you're defining is a tuple.

    • @LucidProgramming
      @LucidProgramming  5 лет назад +12

      That's a fantastic response, and something I was really quite curious about. Thanks for clearing that up! I love making videos like this because I am just as susceptible to learning as anyone who is viewing! Thank you, and thanks again for clearing that up for me!

    • @GelsYT
      @GelsYT 4 года назад

      That's great!

    • @GelsYT
      @GelsYT 4 года назад

      @@LucidProgramming Cool!

    • @alexismandelias
      @alexismandelias 4 года назад +1

      Honestly if you don't know that you define a tuple with one object like this (a, ) you are in no position to talk about processes

    • @GelsYT
      @GelsYT 4 года назад

      LOOOOOOOOOOOOOOOOLLLLLLL

  • @mushkamusic
    @mushkamusic 5 лет назад +3

    Just wanted to say thanks for this. I managed to get my script working with multiprocessing after watching your video. Cheers.

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      Thank you! Really glad to hear you got it working :)
      If you like my content, I've been working on some projects during the past couple of months. If you would like to stay up-to-date, please consider subscribing to my mail list. Also, if you haven't already, please consider subscribing! I really appreciate the support. I hope that the content I provide there will enhance the videos on my RUclips page.
      bit.ly/lp_email

  • @gouthamsreekarrayaprolu5194
    @gouthamsreekarrayaprolu5194 Год назад +1

    I enjoyed watching this video. Just as your name goes, it is lucid. Thank you!

  • @SCARSIDNEY
    @SCARSIDNEY 3 года назад +1

    one of the simplest and best explanation of code online I've ever seen. bless ur soult

    • @LucidProgramming
      @LucidProgramming  3 года назад

      Cheers! If you enjoyed and benefited from my content, please consider liking the video and subscribing to the channel for more content like this. If you would like to support the content creation on this channel please consider unblocking ads when watching my videos as this is how I support my time to make content. I hope to be putting out more similar videos soon!

  • @daniellee6254
    @daniellee6254 5 лет назад +1

    Much clearer than the blogs who just displayed the code you showed and expected the reader to infer what a process class was. Thanks for the explanation.

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      Thank you! If you like my content, I've been working on some projects during the past couple of months. If you would like to stay up-to-date, please consider subscribing to my mail list. Also, if you haven't already, please consider subscribing! I really appreciate the support. I hope that the content I provide there will enhance the videos on my RUclips page.
      bit.ly/lp_email

    • @daniellee6254
      @daniellee6254 5 лет назад +1

      @@LucidProgramming I got you

  • @MrMijuskovic
    @MrMijuskovic 5 лет назад +3

    Great video! One remark: I think it is safe to assume that people who look into Mulitihreading and Multiprocessing know how print functions work. ;)

    • @LucidProgramming
      @LucidProgramming  5 лет назад +3

      Hi Milos. True, but they might not be familiar with the way in which the print statements were updated in Python 3+, so I wanted to mention that just in case. Thanks for the comment!

    • @MrMijuskovic
      @MrMijuskovic 5 лет назад +1

      LucidProgramming thanks for the video!

  • @lockborethdevid3533
    @lockborethdevid3533 2 года назад +1

    Best video ever. Amazing explanation. Thank you so much for this valuable video.

    • @LucidProgramming
      @LucidProgramming  2 года назад

      Cheers! If you enjoyed and benefited from my content, please consider liking the video and subscribing to the channel for more content like this. If you would like to support the content creation on this channel please consider unblocking ads when watching my videos as this is how I support my time to make content. I hope to be putting out more similar videos soon!

  • @mungwanmumbad8254
    @mungwanmumbad8254 4 года назад +1

    this guy is an excellent teacher, i have not understood anyone who teaches python the way i understood you.
    please can you also make videos for beginners ?
    i am a beginner and i want to learn everything from scratch from you I wish

    • @LucidProgramming
      @LucidProgramming  4 года назад

      Cheers! If you enjoyed and benefited from my content, please consider liking the video and subscribing to the channel for more content like this. If you would like to support the content creation on this channel please consider unblocking ads when watching my videos as this is how I support my time to make content. I hope to be putting out more similar videos soon!
      Regarding the content for beginner material, you can feel free to make suggestions on my Patreon page. This would streamline what you want to see and also help to support my channel.
      www.patreon.com/lucidprogramming

  • @IM1THE2KING3
    @IM1THE2KING3 6 лет назад +4

    Almost 4K Subscribers. Before you know it you will be at 400K. Keep up the great work.

  • @HKwak
    @HKwak 3 года назад +1

    Man, your explanation is clear and amazing. I always wanted to learn about threading and multiprocessing in a more organized manner like this. Clicked subscribe already! I'll keep watching your videos.

    • @LucidProgramming
      @LucidProgramming  3 года назад

      Thank you for the support! Very much appreciated!

  • @anteconfig5391
    @anteconfig5391 5 лет назад +1

    3:00 I just wanna say that in python3.7 you can still format like so:
    "this number %d squares to this number %d" % (number, result)
    or
    "this number {0} squares to this number {1}".format(number, result)
    Thanks for the video. I needed it.

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      Hey AnteConfig. Great to know that the video helped you, it's always nice to hear that. You can indeed format content in the way you described. Personally, I prefer "f" strings, but if you prefer to use the format function, that's supported as well. Thanks for the comment, and cheers for watching!

  • @manojsitapara
    @manojsitapara 3 года назад +1

    Perfect example for beginner !! Thank you so much for sharing knowledge. Keep it up !! liked your video & subscribed your channel ..

    • @LucidProgramming
      @LucidProgramming  3 года назад

      Cheers! If you enjoyed and benefited from my content, please consider liking the video and subscribing to the channel for more content like this. If you would like to support the content creation on this channel please consider unblocking ads when watching my videos as this is how I support my time to make content. I hope to be putting out more similar videos soon!

  • @rafaelgpontes
    @rafaelgpontes 5 лет назад +6

    The comma is to differentiate a tuple from an expression.

  • @sudarsandm
    @sudarsandm 5 лет назад

    in Python2 it also supports statements like print("The number is {} and the result is {}".format(number, result)). May be we need to do "from __future__ import print_function". I find this quiet readable. But Fstrings are even more simple and straight forward. Thanks for your videos. You speak at a speed which is neither slow nor too fast and your content is real good. Keep up the good work and all my best for your future.

    • @sudarsandm
      @sudarsandm 5 лет назад +1

      I subscribed after looking at content in your channel. Appreciate your hard work and commitment.

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      Thanks, I really appreciate that!

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      Yep, that is supported. Personally, I like the functional strings in Python3, but that's a fine way to print the output as well. Cheers, and thanks for the comment!

  • @breakdancerQ
    @breakdancerQ 4 года назад +1

    Very nice introduction

    • @LucidProgramming
      @LucidProgramming  4 года назад

      Cheers! If you enjoyed and benefited from my content, please consider liking the video and subscribing to the channel for more content like this. If you would like to support the content creation on this channel please consider unblocking ads when watching my videos as this is how I support my time to make content. I hope to be putting out more similar videos soon!

  • @Alan-qb9qt
    @Alan-qb9qt 5 лет назад

    Pretty straightforward thx a lot

  • @Optisoins
    @Optisoins 5 лет назад +1

    Wonderful Video !! Thank you so much 😊

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      Really appreciate the kind words and thank you so much for watching! Cheers.

  • @dinohsu1019
    @dinohsu1019 7 месяцев назад

    It seems multiprocessing doesn't directly work with Jupyterlab (or IPython, not sure), one workaround is to have the worker function in a separate .py file and imported. Also, I am not sure about the high level concurrent module, but I think it's important to let new multiprocessing learners know about this at the beginning, thanks.

  • @GelsYT
    @GelsYT 4 года назад +1

    THANK YOU! SO MUCH!

    • @LucidProgramming
      @LucidProgramming  4 года назад

      Cheers! If you enjoyed and benefited from my content, please consider liking the video and subscribing to the channel for more content like this. If you would like to support the content creation on this channel please consider unblocking ads when watching my videos as this is how I support my time to make content. I hope to be putting out more similar videos soon!

    • @GelsYT
      @GelsYT 4 года назад

      @@LucidProgramming Thank you sir! I have already liked and subscribed to you a long time ago :D Your tutorials are great!

    • @GelsYT
      @GelsYT 4 года назад

      @@LucidProgramming May I ask hmmmm... is there a way to somehow stop or kill the process and check if it is still doing its job or process? Thank you!

    • @LucidProgramming
      @LucidProgramming  4 года назад

      @@GelsYT htop

  • @mikecripps2011
    @mikecripps2011 2 года назад

    Hey wondering if this can be applied to data extractions. For instance take data from a database, do what is needed and load it to a local database. The data is well in the billions and I am being pinched by the fact the machine and network are a bit weak. Can I have all 4 processors extracting the database at the same time and loading it to the same local db or is there some better method?

  • @lakeguy65616
    @lakeguy65616 Год назад

    I watched your video and followed your code in my own application. briefly, I have 38 files that have to be processed. Each file takes one or two minutes to process. It runs on my core i5 laptop with 4 cores. (eventually, I have over 500 files to process which explains the appeal of using all the cores of my pc...)
    It runs dramatically faster but it doesn't work the way I expected. I thought It would loop through the files starting and completing each file before starting the next (one file per cpu core...) Instead, it seems to start all 38 files and then processes each a few lines at a time. It maxes out my CPU, which runs at 100% until finished. I can process all 38 files in about 15 minutes. But the pc is unusable during that time. The CPU runs at 100% and its uses 65% of the available RAM. Is this normal?
    Is there a way to make it finish a file before starting a new file to process (using all the available cores)? Have I done something wrong? Is there a better way to use multiprocessing? My thinking is I could code in a pause between files which would give the CPU a chance to catch up on other tasks running..
    thank you for reading and responding...

  • @ShukyPersky
    @ShukyPersky 2 года назад

    I don't understand your mentioning of GIL and multiprocessing. multiprocessing creates independent processes. I think GIL's scope of function is a single application.
    so why mix 2 topics which do not have relations? unclear .... In case the python implementation is different please make your comment on that ... thanks

  • @shahid28285
    @shahid28285 5 лет назад +2

    The Processes array where you appended the process objects are like unused. May I know why?

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      Hi Mohammed. I believe I was setting the stage for the next video in this series. At least, that's what I think I was doing :P. Thanks for the question!

  • @alexpolymath
    @alexpolymath 5 лет назад +1

    really good video

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      Thank you! If you like my content, I've been working on some projects during the past couple of months. If you would like to stay up-to-date, please consider subscribing to my mail list. Also, if you haven't already, please consider subscribing! I really appreciate the support. I hope that the content I provide there will enhance the videos on my RUclips page.
      bit.ly/lp_email

  • @zeta0001
    @zeta0001 Год назад

    What is the code editor you are using? I've only used the Python IDLE and Nano so far.

    • @LucidProgramming
      @LucidProgramming  Год назад

      All of that info is in the description. Thanks for watching!

    • @zeta0001
      @zeta0001 Год назад +1

      Thank you so much. I looked it up and it looks like a pain in the butt to learn honestly. Ur a freaking champ.

    • @LucidProgramming
      @LucidProgramming  Год назад

      @@zeta0001 Glad to hear. Cheers and thanks for watching!

  • @bastiandiazsaez7027
    @bastiandiazsaez7027 4 года назад

    Hey! Thank you for your explanation, it is very clear. However, I have a problem. I have exactly the same program than you in 9.33 (I copy it) , but the program continues giving the result as a serial way: number1->number2->number3->number4, no matter of how many times I compile it :( . Do you know what could be the problem with this?. Thank you!

    • @bastiandiazsaez7027
      @bastiandiazsaez7027 4 года назад

      by the way, I am using Python 3.8.0, and I tried with two text editors: Mousepad and Visual Studio Code. My PC is an Intel CORE i7. Thank you!

    • @LucidProgramming
      @LucidProgramming  4 года назад

      @@bastiandiazsaez7027 Hmm, that is interesting. You might want to try and Google around for an answer here.

  • @BeefLoverMan
    @BeefLoverMan 6 лет назад +1

    Is it mandatory to use this __name__ == "__main__" trick? What if I want to try this for only a couple of tasks in a larger piece of code?

    • @LucidProgramming
      @LucidProgramming  6 лет назад +2

      According to Ned Batchelder from StackOverflow:
      Windows doesn't have fork, so there's no way to make a new process just like the existing one. So the child process has to run your code again, but now you need a way to distinguish between the parent process and the child process, and __main__ is it.

    • @kronusplayz8184
      @kronusplayz8184 5 лет назад +1

      @@LucidProgramming thanks now i know why it wasn't working.

  • @KickAxxes
    @KickAxxes 4 года назад

    I understood your tutorial, and it makes sense. But when I tried it on my compiler, all I get on the console is blank. I've tried to google online for help but I really can't find the solution to my dilemma.
    I tried writing your code step by step, I tried just taking it from the link you've provided, but it just keeps giving me the same results. This might be because I'm a complete beginner, but I really want to learn, and I'm just not sure what is wrong. Please do help if you are available.

    • @LucidProgramming
      @LucidProgramming  4 года назад

      PM me and I can provide you my rates for consulting / tutoring. Cheers!

    • @ArnavPushkar
      @ArnavPushkar 3 года назад

      Hi, I am having the same problem as you. Was your query resolved?

    • @LucidProgramming
      @LucidProgramming  3 года назад

      @@ArnavPushkar This is a non-specific problem. You'll need to provide more details.

    • @victormanuelvictormroro5290
      @victormanuelvictormroro5290 2 года назад

      I have the same problem but after review I notice that write
      if __name__=='__main___'
      I had written three '_' instead two
      I hope it had was your problem also

  • @harshalgosavi5659
    @harshalgosavi5659 5 лет назад +1

    Is there couchdb support to run two or more process parallel using python

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      Hi Harshal. There is not, however, using the videos in the multiprocessing video series it would be easy enough to add in that functionality

  • @atreyhazelhispanic
    @atreyhazelhispanic 5 лет назад

    I'm not positive, but the args style is exactly the same as Pandas series type. So when the series (Tuple) is printed to console that data type from Pandas shows single listed items in this same format.

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      Hi Angelo. Thanks for the response. I'm not sure that answers my question though. Just because Pandas uses the same convention, I have the same question for why they happened to go with that as well :).

    • @atreyhazelhispanic
      @atreyhazelhispanic 5 лет назад +1

      @@LucidProgramming Sorry...
      It is only required for single-item tuples (lists and dict as well) to disambiguate defining a tuple or an expression surrounded by parentheses.
      For more than one item, it is no longer necessary since it is perfectly clear it is a tuple. However, the trailing comma is allowed to make defining them using multiple lines easier. You could add to the end or rearrange items without breaking the syntax because you left out a comma on accident.
      I did enjoy your video though sir.

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      @@atreyhazelhispanic Ah, gotcha. That makes a bit more sense to me and adds some justification as to why they went with that decision. Thanks for unpacking that, and thanks for your comments about the video as well! Cheers!

  • @novianindy887
    @novianindy887 2 года назад +1

    is it still valid in 2022?

  • @billma8064
    @billma8064 4 года назад

    What is the purpose of the processes list in this example? You are just appending a process to it but nothing else?

    • @LucidProgramming
      @LucidProgramming  4 года назад

      It's a list of processes, so why would we append anything else to it?

    • @billma8064
      @billma8064 4 года назад

      LucidProgramming er, I mean in this specific program it seems like you created a list but don’t use it. But I see how you use it in the next video

    • @LucidProgramming
      @LucidProgramming  4 года назад

      @@billma8064 Ah okay, I misunderstood your question. In subsequent videos, I use this list.

  • @kaustavchakraborty1406
    @kaustavchakraborty1406 4 года назад

    I can't understand how is this really reducing the real run time? If it is not, then what's the use of it?

    • @LucidProgramming
      @LucidProgramming  4 года назад

      It divides the time for the task by the number of processors you are able to farm out to.

  • @scienceblossom6197
    @scienceblossom6197 5 лет назад

    when I don't use a list (processes) and just call the start() method straightly, the processes run completely sequentially in a way that I feel it's not doing multi-processing. Can anyone explain why this is happening? Here's my code:
    from multiprocessing import Process
    import os
    def func_to_distribute(num1,num2):
    print(os.getpid())
    print(f"{num1} times {num2} is:", num1*num2)
    if __name__ == "__main__":
    for a in range(10):
    for b in range(10):
    p = Process(target=func_to_distribute,args=(a,b))
    p.start()

  • @talha_anwar
    @talha_anwar 5 лет назад

    How many number of process we can use? Can i use 50 process?

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      If you have 50 processors, then yes, but I sincerely doubt that you do. You can use the cpu_count function to see how many cores your machine has.

    • @talha_anwar
      @talha_anwar 5 лет назад

      @@LucidProgramming I have 12 processor, I tried running with 21 process and every thing went well. How this is possible?

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      @@talha_anwar It may have automatically capped it so as not to crash your computer?

    • @talha_anwar
      @talha_anwar 5 лет назад

      @@LucidProgramming no, my avg processing time was reduced from 20 min to 11 min

    • @LucidProgramming
      @LucidProgramming  5 лет назад +1

      @@talha_anwar No, I'm saying that it capped it at the max processors. Your statement isn't a contradiction of what I said.

  • @dimitrisbokaris6192
    @dimitrisbokaris6192 5 лет назад

    I am executing exactly the same code (9:45) , but my pc doesn't print the results, nor can I save the results through return, (main function doesn't identify the results), I would appreciate if you can help. I am using python 3.7.2 64 bit

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      Are you running the code via terminal or in a IDE like PyCharm?

    • @dimitrisbokaris6192
      @dimitrisbokaris6192 5 лет назад

      @@LucidProgramming im running it in idle , it only works in terminal? I open the source code and press f5 to run it

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      @@dimitrisbokaris6192 It doesn't only work in terminal, no. However, I assume that the reason you might be not seeing output is because you're running this within the IDLE scripter. You can run a Python file in many ways. I would recommend running from the terminal as I do to ensure you have consistency with what I am doing in this video. Does that make sense?

    • @dimitrisbokaris6192
      @dimitrisbokaris6192 5 лет назад

      @@LucidProgramming I am not really that good in programming . didn't know that running the same code somewhere else makes a difference. the way i run codes is by going in idle, then new file, then save and then execute. how can i run it through terminal?

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      @@dimitrisbokaris6192 The reason comes down to the fact that when you run the program in the terminal, you're executing a process and when you're running in IDLE, you are kind of running within that process. It's a bit hand-wavy, but yes, there are going to be differences. For how to run via terminal, you can just follow what I do in my video.

  • @juhichoubey7114
    @juhichoubey7114 5 лет назад

    How can i perform an operation in parallel to multiple groups of pandas dataframe?

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      What do you mean by a "group" of pandas dataframes? I offer consulting and tutoring services for problems like these. I encourage you to reach out to my email listed in the "About" section to set up a remote consulting session. Cheers!

  • @DavidTangye
    @DavidTangye 4 года назад

    Nice video except please do not digress off into red Herrings like different ways to use the print statement! And also you have explained why you write each line of code so you don't have to waste all that time writing comments into there as well.

    • @LucidProgramming
      @LucidProgramming  4 года назад

      Thanks, and noted! I feel like that was a bit tangential as well. Thanks for the feedback!

  • @Didanihaaaa
    @Didanihaaaa 5 лет назад

    Thanks for your video. I do have a question about multiprocessing. Here I asked it on StackOverflow.
    stackoverflow.com/questions/57529308/solving-a-problem-utilizing-multiprocessing-and-value-in-python
    I appreciate any help.
    Solving the same problem using the new approach provided in your video, still, I get no benefits in term of speed.

  • @alexismandelias
    @alexismandelias 4 года назад

    So. Much. Useless. Talking.

    • @LucidProgramming
      @LucidProgramming  4 года назад

      If you can suggest a way for me to improve, I'm happy to ear it.

  • @abdurrezzakefe5308
    @abdurrezzakefe5308 5 лет назад +1

    Amazing video, thanks a lot.

    • @LucidProgramming
      @LucidProgramming  5 лет назад

      Thank you! If you like my content, I've been working on some projects during the past couple of months. If you would like to stay up-to-date, please consider subscribing to my mail list. Also, if you haven't already, please consider subscribing!
      I really appreciate the support, and if you want to support the channel, I do have a PayPal link (www.paypal.me/VincentRusso1) for donations that go directly to the creation of content on this channel.
      I hope that the content I provide there will enhance the videos on my RUclips page.
      bit.ly/lp_email

  • @VinsysVlogs
    @VinsysVlogs 6 лет назад +1

    What is the text editor you are using?

    • @LucidProgramming
      @LucidProgramming  6 лет назад +1

      Hi Vinay. The text editor is Vim, and if you want to see how to use it and how I set it up for Python development, you can check out this playlist on my channel: ruclips.net/p/PL5tcWHG-UPH2HrF5M7-IIXK6JSRG0obYo. Cheers!

    • @VinsysVlogs
      @VinsysVlogs 6 лет назад +1

      LucidProgramming thanks a lot

    • @LucidProgramming
      @LucidProgramming  6 лет назад

      @@VinsysVlogs No problem! :)