Thank you so much ! I literally spent so much time putting time.sleep(x) everywhere between my calls but it really didn't helped me. indeed, using semaphores fixed my problem ! I needed to use some games api to retrieve matches from more than 250 players (20 matches approx, by player) and to write them inside a csv :]
also another question, i tried creating an async function in flask to handle api requests, but it didnt work on flask, but worked as an independent package, help please if you can
Thank you so much ! I literally spent so much time putting time.sleep(x) everywhere between my calls but it really didn't helped me. indeed, using semaphores fixed my problem ! I needed to use some games api to retrieve matches from more than 250 players (20 matches approx, by player) and to write them inside a csv :]
pog
THANK YOU! WHAT AN ABSOLUTE LEGEND DESERVE A MILLION SUBS!
Update. Httpx has a asyncclient
could i get a souce code to this?
Could we have the code please. thanks for the vid
This works in Python env but not in databricks
Do you have a link to the code @Ethan Lyon ?
# Import the libraries needed to run the code.
import asyncio
import aiohttp
import attr
@attr.s
class Fetch:
limit = attr.ib()
rate = attr.ib(default=5, converter=int)
async def make_request(self, url, limit):
async with self.limit:
async with aiohttp.ClientSession() as session:
async with session.request(method="GET", url=url) as response:
json = await response.json()
status = response.status
print(f"Made request: {url}, Status: {status}")
await asyncio.sleep(self.rate)
async def main(urls, rate, limit):
limit = asyncio.Semaphore(limit)
f = Fetch(rate=rate, limit=limit)
tasks = []
for url in urls:
tasks.append(f.make_request(url=url, limit=limit))
results = await asyncio.gather(*tasks)
def test_run():
URL_LIMIT = 20
urls = [f"httpbin.org/anything/{n}" for n in range(URL_LIMIT)]
LIMIT = 6
RATE = 1
asyncio.run(main(urls=urls, rate=RATE, limit=LIMIT))
test_run()
also another question, i tried creating an async function in flask to handle api requests, but it didnt work on flask, but worked as an independent package, help please if you can
FastAPI
please share a github page with the source-code.
2 req every call? Where have you mentioned 2 in the code, i don't see it at all
i think it's the semaphore which is called as the limit and the asyncio.sleep that is 2 seconds
He's passing the values from the pytest script that he has written on the left.