Python Threading Examples - Issues and Caveats

In this video I show a couple of examples implementing some Python multithreading techniques, such as creating, starting and joining threads, daemon threads, locks, and threadsafe queues (this is not really a tutorial on threading - that would be an entire course, but more of a demo of what can go wrong and a common misconception).
My main aim in this video is to show that multithreading (in Python) does *nothing* to improve performance in CPU bound workloads, and hopefully convince you that writing multithreaded code is not something you should just do casually as it is very easy to introduce (often invisible) bugs and is not some kind of performance improving magic bullet.
#mathbyteacademy #python
Code for this Video
================
Available in GitHub blog repo: github.com/fba...
Direct link: tinyurl.com/u6...
My Python Courses
=================
- Python 3 Fundamentals (introduction to Python)
www.udemy.com/...
- Python 3 Deep Dive (Part 1 - Functional)
www.udemy.com/...
- Python 3 Deep Dive (Part 2 - Iteration, Generators)
www.udemy.com/...
- Python 3 Deep Dive (Part 3 - Hash Maps)
www.udemy.com/...
- Python 3 Deep Dive (Part 4 - OOP)
www.udemy.com/...

[@MohamedAhmed-rf5bk]

I seriously love this man and how generous he is with his knowledge.
Thanks Fred for your work

[@mathbyteacademy]

Thanks!

[@ceebz10]

Fred’s wisdom and knowledge on Python surpasses an advanced deep learning AI trained for billions of hours on how to use python most effectively.

[@mathbyteacademy]

Thanks!

[@cudanakiju0]

The best Python-related content in the web.
Cant wait for async and multiprocessing examples!

[@mathbyteacademy]

Thanks!

[@chrisyoung7704]

This is really a great presentation with clear logic and well-organised examples. Thanks!

[@mathbyteacademy]

Glad you enjoyed it!

[@iChrisBirch]

The examples were perfect for explaining the usage of multithreading and their limitations. Fantastic!

[@mathbyteacademy]

Glad you found it useful!

[@abdellahiaioun8430]

I just finished the video. Thank you , thank you.
Went to your udemy courses too. Thank you
Now, I don't like multithreading that much :)

[@mathbyteacademy]

Thanks, glad you liked the video. Nothing wrong with threading, just that it does not have the impact many people think when working with Python.

[@valentinplechuc5710]

I think that many of us thinks that, in some way, multi threading will speed up the app. But as you explain here, it is not quite simple. Anyway, thanks for sharing!!

[@ApneDharamSeJudo]

Dear Sir. You are awesome. I was terrified of coding. My friend suggested your Udemy course. I am going through it. It made coding easy for me. Sir, please make a course on Data Science and Machine Learning.

[@mathbyteacademy]

Thanks, glad you find the videos useful!

[@bennykopilov2819]

Thanks Fred as always best courses / explained ! I am waiting for a design pattern course !!!

[@mathbyteacademy]

Thanks Benny, and thanks for the suggestion. I can do some RUclips videos on the subject, but not sure I would do a whole course.

[@Emma-ox4ki]

Great video as always!

[@mathbyteacademy]

Thanks - glad you liked it!

[@selvamm1234]

Thanks fred. I was waiting for this course

[@mathbyteacademy]

You are welcome!

[@prashlovessamosa]

Thank you Sir

[@mathbyteacademy]

You're welcome!

[@arturkabitcher]

a long-awaited topic. thanks!

[@mathbyteacademy]

You're welcome!

[@fkdosilovic]

Good explanations (the first example seems a bit contrived, as you mentioned, but ok). But, the explanation you provided at 48:12 on why locking is not necessary is a bit misleading in my opinion.
The main reason why locking isn't required is because append on a list is atomic. (More can be read at Python's docs FAQ under What kinds of global value mutation are thread-safe?)

[@mathbyteacademy]

I probably should have mentioned that appending to a list is threadsafe, but what i was trying to point out is that you do not need locks here because the only operation being done is an append - there is nothing that first has to read something in the list, or read the length of the list, and then append - it's a pure append, so no lock needed (and indeed, no lock needed because append is threadsafe). If you had a series of operations that have to read something off the list, then even if append is threadsafe, you might still require a lock.

[@redDuck757]

You are the best. Thank you for this video. Can you, please, make an some kind of deep understanding for the Async? I've bought all your courses in udemy and still in the process of working with them, but in topics I haven't seen such topic. So it would be great if you will make video of additional course for the async programming, understanding of it and best practises.

[@mathbyteacademy]

Thanks!

[@kosmonautofficial296]

Great video! That was a lot of great information. I wasn't aware of the daemon thread type and that is pretty cool. I am a network engineer and I find myself using threading a lot for speeding up my code that sends commands to many different network devices over ssh. I use concurrent.futures ThreadPoolExecutor with the map method and I find it to be very helpful. I use a thread safe logger in the code and return the result I need from the function sent to threads. The function used in threads doesn't use any global variables and doesn't modify any of the arguments passed to it. Then outside of the threads I put the results together into a larger data structure. To test some of this I would keep a threaded version and non-threaded version to compare the results to make sure they were always the same.

[@mathbyteacademy]

Thanks! I tend to use async over threading for I/O ops (I think I'm less intimated by async than threading 😀) - in my typical use cases I rely heavily on async queues (I have one or more async workers that publish items to the queue, and then have multiple async workers that process the queue items). But then I also use a daemon thread for readyness/liveness probes (since typically I run all my stuff in k8s). I never thought of automated testing comparing results between multi-threaded (or async) and single-threaded - that's a really neat idea!

[@kosmonautofficial296]

@@mathbyteacademy ok interesting. I haven't used async in production yet but I am leaning that way because I have a few REST APIs I need to use and speed up those transactions. A common library for ssh to network devices is netmiko but recently switched to scrapli because it supports async. So I think I could speed up things even more in that not only is each device being used concurrently, but also when multiple commands are sent to a device that can be concurrent.
Really love your videos man! They really go in depth in a way no one else is.

[@sarthak-salunke]

Does this included in any of the part in your's udemy course? Please let me know sir ,i am looking to buy your all part it would be better if i find one spot source which covers all python topics

[@mathbyteacademy]

None of my courses cover concurrency.

[@deepakchaube9576]

Very useful

[@mathbyteacademy]

Glad you think so!

[@emmajennnings5920]

i got stuck

[@reneeliu1648]

Personally, I always thought multi-threading is the way to go to make a program faster but I also looked into approaches like asyncio, concurrent.futures. But they all are fuzzy topics to me.

[@mathbyteacademy]

Multi-threading in Python doesn't make code run faster in most CPU bound scenarios. (Python is inherently single-threaded). It certainly can help with I/O bound processing, but asyncio is probably better/easier to use. Multi-threading however has its uses, but not necessarily for speeding things up - just getting things to run concurrently.

[@mathbyteacademy]

I'm going to do a video next month to explain these concepts (multithreading, multiprocessing, async) and related topics - hopefully that will clarify things!!