The best explanation about tqdm in the internet. Direct to the point and with easy explanations! Please continue sharing your knowledge. I found your channel looking for data analysis in Python, and really enjoy your content. Cheer's from Brazil!!!!
Hi Rob, warm greetings from India I got to know about your channel when you did a live stream on Abhishek Thakur's channel I've been following you since then and I really like your content, especially how you make it so easy to understand and yet it's still in-depth and cover most of the concepts about that topic I just have one little request please make some tutorials on advanced topics as well like Transformers, Auto encoders etc
Thanks for the lesson, sir. I have one question. I defined a function and used the tqdm in it. When I call this function from a .py file, the tqdm does not get recognized, even importing as you shown it here. Do you have any clue how to solve this issue? When I define the function in a cell and then I call it, it works! But the functions are too big and, thus, I wanted it to keep on the .py file I'm using to call some functions when need it. Thanks for your time.
Hi. Thanks for the comment. Have you made sure to import tqdm at the top of your .py file? It needs to be imported in the file where it is called. Hope that helps.
Is there a way i can make a progress bar that tracks sub loops? I have a function that loops through Members, and for each member loops through assignments and then writes to a file. Is there a way to have it track every time it writes to the file without having to be in the bottom loop? As it is if i put it on the members it just updates x% after each member and the time remaining is way off because some have a lot more assignments than others.
Hi First of all thank you for sharing this. I have some doubt. When I use sleep(0.00001) like you under for dogs loop with 50000 iterations, it's taking more than 5 minutes to run but in your case I see it ran under 2 minutes. May I know why is it faster in your case but slower in mine? I am using VS code for running this. I am talking about the part at 02:50 Please clarify.
Hello! I don't know why but I have an error: out = fake_df.groupby('dog').progress_apply(lambda row: int(row['smell'])**2) my bar: 67%|██████▋ | 2/3 [00:00
@@robmulla Setting axis=1 doesn't help, the problem was that numpy.random.randint returns a string in my case (I really don't know why), whereas from documentation it has to be int. Interesting, but this is how a local program is executed (I used to do it in Colab). I simply converted 'smell' data to int. Anyway, thanks for you quick reply!
Can you show some examples where in place of logging to console it’s logging in production setup and for example let’s use Loki as scrapper. So now we need the logger setup but because of buffering it only print when entire thing is done. Let me know if you have a solution for this
Can you please show an example of how you would implement this for scikit-learn when doing a RandomForest or XGBoost model and you're hyper-parameter tuning?
If you are using something like random grid search or grid search in sklearn - it essentially provides a generator object which you iterate over. You simply need to wrap that generator with tqdm and have a total= the number of iterations you will run. Check out this walkthrough looks to have a good example: www.jpytr.com/post/lazy_tuning/
@@robmulla Thank you... that looks surprisingly simple, though they did not have the "total=" portion that you mention. By the way, what is your favorite way to tune hyper-parameters since you linked to this "lazy tuning" article? I read about Halving Grid Search recently. Wondering if there's better ways you came across in your experience.
@@robmulla i meant traking progress of calculations spread between multiple processes by means id multiprocessing package. I ended up manually updating progress bar after checking number of items in shared queue.
Hi Rob, greetings from Singapore! I am coding following along with the video, but the section on TQDM and Pandas, doesn't seem to work for me. Both scenarios are a no go for me. Please enlighten. Thank you very much in advance. PS: I find your shorts and videos very helpful, helping me fall asleep, or rather, keeping me away, thinking how I can integrate into my code portfolio!
Glad you've found my videos helpful. I'm not sure what exactly your issue is. Sometimes you need to provide the total=len(df) to get tqdm to work correctly on dataframes. Hope that helps.
Thanks for the feedback. I'm using the Nord theme which can be installed from here: github.com/arbennett/jupyterlab-themes there are a lot of other good ones too.
Love your helpful videos. Could you show me the way to set up the underline Separator (_) for a number when typing in code cell, for example 50_000. Thank you.
Basically in the newer versions of python you can add _ to any integer and not have an issue. I like to use them to make large numbers easier to read. 1000000 becomes 1_000_000
Hey, thanks for watching. Not sure what you mean by current tasks. If you wrap your current loop in tqdm it will display a progress bar. Hope that helps.
Could you make a video teaching how to generate combination number and saving in Excel? and at each end of the 1048576 line it generates a new excel. I have an excel generator but it creates in folders on the side, inside the same file so it is very heavy, but if you make a file in each folder, it would be ideal. like the mega sena, and instead of generating all the possibilities, leave a field to generate combinations only with the numbers that we would prefer. here's the tip. it would be great, thanks.
Hey Telmo. Thanks for writing this comment. This sounds like a very specific case and I’m not sure I follow exactly what you are looking to do. It might be something you could have better luck asking on my discord? Hope you figure out a good solution.
@@robmulla hmm thanks for that comment about chunking. Found some interesting ideas regarding it. I assume it possible to use chunking with other types of libraries that load files? I have one program that uses pythonOCC library to load model .STL files, would love to get a progress bar for reading those files into the program.
@@robmulla my humble request to you would be to expand the bert sentiment analysis video to include bert keyword extraction from the Amazon dataset - this would be phenomenal.
The best explanation about tqdm in the internet. Direct to the point and with easy explanations! Please continue sharing your knowledge. I found your channel looking for data analysis in Python, and really enjoy your content. Cheer's from Brazil!!!!
Thanks so much for the feedback. That really means a lot to me.😊
Rob, thank you so much for your hard work and contribution to the development of the data analyst community! From Russia with love!
Hi Rob, warm greetings from India
I got to know about your channel when you did a live stream on Abhishek Thakur's channel
I've been following you since then and I really like your content, especially how you make it so easy to understand and yet it's still in-depth and cover most of the concepts about that topic
I just have one little request please make some tutorials on advanced topics as well like Transformers, Auto encoders etc
Thank you, sir.. learning from a Pro is a much better option than reading through the help manual.
Thanks for the video!! Hope you have a great week.
Thanks for watching!
Great explanation with practical examples. Thanks!!
Excelent video! Thank you, Rob.
Appreciate the kind words. Thanks for watching.
Here is a gist with the code from the tutorial if you would like to try it yourself: gist.github.com/RobMulla/0fe4acf99001d4657df4e73c93cf3218
Thanks for the lesson, sir. I have one question. I defined a function and used the tqdm in it. When I call this function from a .py file, the tqdm does not get recognized, even importing as you shown it here. Do you have any clue how to solve this issue? When I define the function in a cell and then I call it, it works! But the functions are too big and, thus, I wanted it to keep on the .py file I'm using to call some functions when need it. Thanks for your time.
Hi. Thanks for the comment. Have you made sure to import tqdm at the top of your .py file? It needs to be imported in the file where it is called. Hope that helps.
@@robmulla That was it! Thanks! Now I have a new problem to solve, but little by little I'll get there lol! Thanks again!
@@RafaelCataldo Glad you figured it out!
Very smooth explanation...
Thanks! Glad you found it helpful.
Is there a way i can make a progress bar that tracks sub loops? I have a function that loops through Members, and for each member loops through assignments and then writes to a file. Is there a way to have it track every time it writes to the file without having to be in the bottom loop? As it is if i put it on the members it just updates x% after each member and the time remaining is way off because some have a lot more assignments than others.
Hi
First of all thank you for sharing this.
I have some doubt.
When I use sleep(0.00001) like you under for dogs loop with 50000 iterations, it's taking more than 5 minutes to run but in your case I see it ran under 2 minutes.
May I know why is it faster in your case but slower in mine?
I am using VS code for running this.
I am talking about the part at 02:50
Please clarify.
Thanks so much for this insightful video.🙏
This tutorial is very good.
Thanks Vy!
Hello! I don't know why but I have an error:
out = fake_df.groupby('dog').progress_apply(lambda row: int(row['smell'])**2)
my bar: 67%|██████▋ | 2/3 [00:00
Sorry I don’t think I can fully debug this for you. But maybe try setting axis=1 in the apply?
@@robmulla Setting axis=1 doesn't help, the problem was that numpy.random.randint returns a string in my case (I really don't know why), whereas from documentation it has to be int. Interesting, but this is how a local program is executed (I used to do it in Colab). I simply converted 'smell' data to int. Anyway, thanks for you quick reply!
Great tutorial! Btw how do I update the progress bar if im using multiprocessing?
Can you show some examples where in place of logging to console it’s logging in production setup and for example let’s use Loki as scrapper. So now we need the logger setup but because of buffering it only print when entire thing is done. Let me know if you have a solution for this
Can tqdm be initialized _inside_ a loop?
so without specifying the iterable's length first
@Rob Thanks a tonn for this video. Its really helpful.
Glad it was helpful for you. Like & share with all your friends!
Nice video . From Vietnam
Thanks for watching!
Can you please show an example of how you would implement this for scikit-learn when doing a RandomForest or XGBoost model and you're hyper-parameter tuning?
If you are using something like random grid search or grid search in sklearn - it essentially provides a generator object which you iterate over. You simply need to wrap that generator with tqdm and have a total= the number of iterations you will run. Check out this walkthrough looks to have a good example: www.jpytr.com/post/lazy_tuning/
@@robmulla Thank you... that looks surprisingly simple, though they did not have the "total=" portion that you mention. By the way, what is your favorite way to tune hyper-parameters since you linked to this "lazy tuning" article? I read about Halving Grid Search recently. Wondering if there's better ways you came across in your experience.
It's great, thanks bro!
Thanks for watching!
It would be great to hear some of your thoughts on progress bar for multiprocessing stuff. For me it was a real pain in the back to make it work xD
Yea, nested tqdm bars can be annoying. Sometimes it's best to test on a subset of the data and then disable them for the actual run.
@@robmulla i meant traking progress of calculations spread between multiple processes by means id multiprocessing package. I ended up manually updating progress bar after checking number of items in shared queue.
Hi Rob, greetings from Singapore!
I am coding following along with the video, but the section on TQDM and Pandas, doesn't seem to work for me. Both scenarios are a no go for me. Please enlighten.
Thank you very much in advance.
PS: I find your shorts and videos very helpful, helping me fall asleep, or rather, keeping me away, thinking how I can integrate into my code portfolio!
Glad you've found my videos helpful. I'm not sure what exactly your issue is. Sometimes you need to provide the total=len(df) to get tqdm to work correctly on dataframes. Hope that helps.
Nice video, do you mind sharing what theme are you using for jupyter notebook?
Thanks for the feedback. I'm using the Nord theme which can be installed from here: github.com/arbennett/jupyterlab-themes there are a lot of other good ones too.
Love your helpful videos. Could you show me the way to set up the underline Separator (_) for a number when typing in code cell, for example 50_000. Thank you.
Basically in the newer versions of python you can add _ to any integer and not have an issue. I like to use them to make large numbers easier to read. 1000000 becomes 1_000_000
Hey, I'm Brazil, How to implement for current tasks?
Hey, thanks for watching. Not sure what you mean by current tasks. If you wrap your current loop in tqdm it will display a progress bar. Hope that helps.
Could you make a video teaching how to generate combination number and saving in Excel? and at each end of the 1048576 line it generates a new excel. I have an excel generator but it creates in folders on the side, inside the same file so it is very heavy, but if you make a file in each folder, it would be ideal. like the mega sena, and instead of generating all the possibilities, leave a field to generate combinations only with the numbers that we would prefer. here's the tip. it would be great, thanks.
Hey Telmo. Thanks for writing this comment. This sounds like a very specific case and I’m not sure I follow exactly what you are looking to do. It might be something you could have better luck asking on my discord? Hope you figure out a good solution.
very usefull
I am trying to use
"with torch.no_grad():
for idx, data in enumerate(tbar):"
on my own code, but keep getting a " 0%| | 0/1 [00:00
tbar = tqdm(val_loader, file=sys.stdout)
Hey Jason. I’m not aware of any reason why it would work on Kaggle but not vscode. Are you using tqdm.auto? Maybe try forcing it to use the base tqdm?
@@robmulla how do you do that?
@@dingaroo2003 from tqdm.auto import tqdm
I would be impressed if tqdm could give a progress bar when reading a excel or csv file as a dataframe.
I think it can if you use chunking when reading. Pretty sure I’ve seen it before.
@@robmulla hmm thanks for that comment about chunking. Found some interesting ideas regarding it. I assume it possible to use chunking with other types of libraries that load files? I have one program that uses pythonOCC library to load model .STL files, would love to get a progress bar for reading those files into the program.
Please sir, do a KeyWord Extraction video
Thanks for the feedback. I’ll see what I can do.
@@robmulla my humble request to you would be to expand the bert sentiment analysis video to include bert keyword extraction from the Amazon dataset - this would be phenomenal.
Carolyn Spring
Maggio Walks
Greyson Shoals
Flatley Pines
Haley Knolls
McLaughlin Vista
Alvina Cliffs
McClure Crest
Jones Radial
Miller Forge
Rippin Groves
Royal Stravenue
Blick Radial
Durward Prairie
Abshire Haven
Schultz Cliffs
Leone Vista
Conner Circles
Sherwood Dale
Runolfsson Divide
Gerlach Avenue
Alexis Circles
Jast Haven
Judd Route
McClure Bridge
Erna Corner
Bartoletti Crest
Stanford Crossing
Torrey Shores
Goodwin Passage
Adolphus Oval
Clifton Isle
Ritchie Circle
Cornell Plaza
Witting Harbor
how about `df.isnull().sum()` how can i use tqdm here
Not sure if you can because it's vectorized already.
Treutel Glen
Luna Harbors
Rhoda Expressway
Jacobs Place
Annabel Shore
Sawayn Mountain
Goyette Ridge
Roob Hollow
O'Keefe Motorway
Veum Harbor
Sandrine Squares
Josefina Mission
Shad Prairie
Rosenbaum Greens
Oberbrunner Mills
Bernhard Crescent
Maggie Circle
Leopold Vista
Claudine Rapid
Boyle Rapids
Daniel Mountain
Bins Haven
Jillian Summit
Hintz Ridges
Brekke Branch
Schmidt Garden
Hane Summit
Matilda Heights
Jerde Ridge
Myrtle Glens
Friesen Mount
Emilia Plains