How to Make 2500 HTTP Requests in 2 Seconds with Async & Await
HTML-код
- Опубликовано: 26 фев 2022
- DISCORD (NEW): / discord
This is a comparison about how to use Async and Asynio with AIOHttp and Python vs using threads and concurrent futures to best understand how we could make several thousand http requests in just a few seconds. Learning how to do this and understanding how it works will help you when it comes to running your own servers and web services, and stress testing any API environments you offer.
github.com/jhnwr/non-blocking...
text articles from: realpython.com/
Support Me:
Patreon: / johnwatsonrooney (NEW)
Scraper API: www.scrapingbee.com/?fpr=jhnwr
Proxies: proxyscrape.com/?ref=jhnwr
Amazon UK: amzn.to/2OYuMwo
Hosting: Digital Ocean: m.do.co/c/c7c90f161ff6
Gear Used: jhnwr.com/gear/ (NEW)
-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
------------------------------------- 
Наука
You do not need to explicitly create task on line 18
why so ?
Gather already accepts coroutines, so any async function can be supplied directly
I can't believe how fast you got to the point. thank you for reading the room
John, your channel is absolutely fantastic! congratulations!
Thank you very kind
@@JohnWatsonRooney I don't say this lightly, best educational content I have seen in years. Straight to the point, rich and very well explained! (ok, and I now I stop :-)
Thanks for your video!! Tried to send multiple async request for some api but my previous code was unefficient and your example helped me!!
Great stuff glad it was useful for you thanks!
Awesome man , I was working with these things Today , I'll try this one too ♥️♥️♥️🙏
Thanks for the video, makes me appreciate go routines and their simplicity even more :)
I couldn’t believe how easy it was when I started learning go
@@JohnWatsonRooney preaching to the choir. Even with its rudimentary error handling
hey sir👋
as a self taught dev You are one of my inspiration 🙌🙌🙌
i started learning web scraping recently. i downloaded some videos about scrapy and BeautifulSoup tutorials of yours and
i followed along them and found pretty comprehensive and clear i did learn better by your videos.
i do hope more and more tutorial videos!!!
thanks million times !🙏
Thank you that is very kind!
I am scraping a lot of things on web, this is just a pretty thing to do, I like this too much, it's so beautiful to see all the thing beeing "gathered" ahaha, are you still on this thing?
Hi good sir thanks for the vid and most importantly, being actively engaging with people in the comment section Cheers
Thanks I’m glad you enjoyed the vid 👍
Thank you good sir, you are a master at this!! You have helped me land and keep my data scraping job!! Thank you so much, truly an inspiration :))
Thank you very kind!
ThreadpoolExecutor is slower than Thread but saves memory. You don't need to use asyncio for waiting. You can use joins, barriers, wait groups and mutex locks with Thread to achieve the same. It comes down to preference, although asyncio is more streamlined, and thread is better suited for manual optimization that requires utmost speed and care.
Thanks for the info. I still have a lot to learn about threads!
"how to build a ddos attacker..."
Right 😂😂
Very well explained
Thanks, I'm gonna implement this in my "yfrake" package (PyPI).
Very good vid!! I just finished making use of concurrent futures (based on your previous vid) and it speed up my code considerably! Looks like I have the potential to speed up further 😀. Will making a lot of requests at the same time slow down the source server, thus passing the waiting time to the server?
It should be fine, obviously you can overload it (ddos attacks work this way) but as we are just trying to maximise our efficiency we can stay within any rate limits or server limitations set. It’s much more useful for getting data from many places at the same time rather than just one
Thank you John
very instructive thank you John, I think that scrapy uses async requests, that's why some scraping jobs can be impressive quick with scrapy
Yes scrapy uses twisted, which is asynchronous
hey thanks, good info.
how to run 'subprocess' Asynchronously, while I dont want to use 'requests.content' to download
subprocess.run(["yt-dlp", download_link, "-o", f"{output_dir}/{episode_name}.%(ext)s"], shell=True, stdout=PIPE)
Also, What would you prefer Multi-Threading or Async in this case ?
Awesome vid bro!
Thanks!
I was looking for a good explaination of the difference between async and multithreading...so threading is for doing things in parellel and async is for waiting for future tasks to complete but dosn't stall the current program?
your videos are as efficient as your code
While using executors, if the capacity of tomcat to process is set to 200 only. What happens after 200 request? Will we have to wait till 200 request are processed by the server? Or, it will pick up some request when call passes from server to db.
Thanks for your helpful videos.
Please, have you any idea how to avoid dat*ad*ome protection ?
Why does the threaded version take so much longer than the async one, when it should be sending all the requests simultaneously as well? The synchronous one on the other hand waits for each response before it sends the next request. You could have a version that doesn't use any parallel programming facilities in Python but still works in parallel by having the async / parallel stuff handled by a Python package, or handled in an underlying C library. Eg. if you used the (much lower level) socket module and just repeatedly called send (), your requests would all go into the send buffer and the OS would send them while your Python program is waiting or doing something else.
Nicely done, a Proxy or socks5 "which i believe works with requests" will do the trick bypassing the traffic limit. But how can i implement it in this scenario? Thanks John.
proxies=proxies :)
Dear Ron,
I am going to send unlimited fetch requests by javascript code from console to the server.
In backend or frontend there is policy for each user to send request with 300 mili second time distance.
If I want to send multiple requests in a second, I am blocked.
Is there a solution for this issue from your point of view?
How can we apply same thing in PHP? Currently I am using CURL Multi but it boost up my server utilization. So is there any alternative way to do same in PHP?
hi john, or anyone who knows, I keep getting the error "UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 28855-28856: invalid continuation byte" , do you have a work around for this? the ones that I found on google I can't seem to implement correctly.. much thanks!
That's so great! ow to do this with dynamic payload in a Post request?
We use asyncio to scrap around 300+ web pages of a site. Because of res = await asyncio.gather(*tasks) the res gets heavier and script stops there. We changed the logic to process 50 pages at a time using asyncio
Also, I see you have not used loop = asyncio.get_event_loop(), does this affect the performance?
Can you please help on how the asyncio would work in AWS Lambda?
Thank you for the video. What theme of VS Code are you using please let me know
Hey thanks, this is PyCharm but the theme is gruvbox which is available for vs code too
THIS IS FREAKING AWESOME, THANKS MAN, NOW I CAN DDOS FBI SERVERS WITH BILLIONS OF REQUESTS
is the normal request library a blocking library?
Hey John!!
Could you make a video about handling cookies with aiohttp?
With the ClientSession I can send cookies but they are shared with all instances. I just dont get it how to pass a cookie which is only shared with one website and how to retrieve them. Cookie handling with requests is way much easier, but i really want to use aiohttp because it is SO FAST! :D
Hi John, I want to use Asynio and threading in combination. I want to use Asynio for making requests and threading for making calculations. What do you think?
You could make a thread that runs a main loop (Inside of which async stuff is happening) while other threads are doing your calculations, so yeah, sure, they can both work
But why tho? If you are already making a bunch of threads, why not also use threads for your network request? You are adding complexity to your code base by using both techniques when it seems that your use case would be perfectly fine with just one of them. Making your code slightly easier to maintain
So yeah, you can use both of them, but do think about what you are trying to achieve, and if this will actually achieve that
Thanks!
Thank you, that’s very kind
Hey I really like your color scheme! Could you please share the theme and color shemes used? Thanks
Sure this is gruvbox material and I’m using PyCharm community edition
hello john can you make a video on scrapping world population data website please i tried but failed beacuse the span tag is constandly changing
can i do the same things with Flask?
On Yahoo finance, a company's income statement has table rows which need to be clicked on to expand and show the data. Do you know of a way to scrape such rows that doesn't involve using selenium to click on the button to expand the row?
I haven’t looked for a while but I believe there was a Python package for working with yahoo finiance, yfinance I think? That made it very easy to get all the data
What about running async bots that in turn run threads? 🤔
In this example when we send 2500 async requests , does each request have the same session or each api call had its own session ?
no `aiohttp` maintains pool of connection !
u can speed up threading by using maxworkers = x
with concurrent.futures.ThreadPoolExecutor(max_workers=10_000) as executor:
executor.map(scraper_sub, links)
max workers = 10_000 means 10k workers work at same time OR a single worker work 10k work at a time
Can you please make a video on how to run a webdriver with headers on google colab? or if you have any tips
Thank you
I love your vids
they really help a lot
i wish this used a real networked example as i have a real networked API example i'm trying to speed up. i'm already using async and it's literally not any faster. so i don't know if aiohttp will be any better.
for me, thread pool executer is best to go. using it from last few years and always gives best results. easy to handle in python standalone scripts or even in python websites with celery.
Hello
I get this error
Event loop is closed
Hi Ron! I’m a business engineering student from Belgium currently writing my own thesis. As a part of my research I would need to make my own dataset by scraping Zalando. Would you be open to assisting me with this 1-to-1, ofcourse when publishing my work I would properly reference you and your help. I have tried some things using your very helpful videos, but I’m still mostly very stuck. I look forward to your answer
still trying to do? it's possible to scrape this website in a very easy way. I'll be glad to help =]
That's a very neat trick! However is it safe to send so many requests in such a short period of time? I'm learning web scraping and from what I've read your IP can be banned by the website if it detects that all those requests are scripted. That's why in my web scraper I use sleep function to wait from 0 to 2 seconds (randomly) between each request so it resembles more human behavior. But I guess it's not the best solution since it takes about 15 minutes to scrape roughly 1000 webpages.
Absolutely, if you are working on something that needs to spread out the requests the do so. This is about sending as many requests a possible- if you have 1000 different urls you wanted some data from then having to wait 1 by 1 would be painful, so we can use async to make them quicker. Also learning async technology is definitely worth it as you progress as a developer
@@JohnWatsonRooney For sure. While I might not use async in this particular case it's good to know about it for the future
Also, I'm curious whats your day job? Is it related to web scraping?
It’s not directly no but I use Python everyday for data extraction over our APIs
Hey, wanted to point something out: I've often timed my own code too, and I find that print() consistently causes a large delay. So if you're analysing these delays, I would suggest not including a print statement there.
Love your content though!! Helped me with a lot of stuff.
Great tip thank you!
What if you need to render the content? What would be the best approach in that case?
Requests-html I believe can do that but I’d have to check
@@JohnWatsonRooney well that would be a nice topic for a new vid wouldn't i? :D
@@oparpax He already did a video on that topic! its called Slow Web Scraper? Try this with ASYNC and Requests-html
This method puts a lot of pressure on the machine, and to my use case, I want to send more than 170 thousands requests, and when I use ThreadpoolExecutor, I can divide requests to lesser group using max workers' argument. Can that be done using Async & Await?
It sounds like maybe Python isn’t going to be the right language, something like Go or rust might be better for you as they much better built in concurrency models
Which is easier to learn?@@JohnWatsonRooney
How to login in "jumia"
Every devops developers nightmare.
How to send million request?
async for loop would have been more readable…
More polite title: How to do 2500 handshakes in 2 seconds and disappear.
Next video: How to make 2500 people shake hands with each other in 2 seconds, for fun!
require 'popcorn' // btw
Here is a good tutorial on how to destroy someone's backend in seconds :)
work in parallel with python threads? nope, GIL.
503 incoming
First!!
Nice 👍
Really appreciate the videos! Cheers from Australia 🇦🇺
that's 2499 requests