10x Data Scraping with AsyncIO / Concurrency

We use Asyncio to speed up the download of financial data from a REST API. In this video I use Polygon to download 1 minute OHLC candles, but the same technique will work for any provider.
We use a Queue to concurrently download multiple chunks of data at the same time, asynchronously using asyncio.
⭐ Code:
github.com/ChadThackray/async...
00:00 - Introduction
02:06 - Polygon scraping code overview
05:27 - Optimizing code using Asyncio
23:35 - Conclusion

[@neonlinneonlin]

I love your channel. It helps me a lot for building my own tools.

[@totesthegoats8935]

Been Killing it with the films recently Chad!

[@raffaelegirodoangelin6649]

Thank you so much Chad for sharing your knowledge !!!👏👏👏
I am a beginner in algorithmic trading.
How can I transform data ticks or pips into Range Bars Chart?
How can I transform 1 minutes chart into a Range Bars Chart?
It's possible?
Regards
Raffaele

[@dataml-trading4085]

I think Python is not reliable and robust for processing high-volume & High speed realtime data. I occasionally get unknown Errors when dealing with Binance websocket. That is why I am studying Rust !

[@ChadThackray]

Good luck! I hear Rust has a steep learning curve at first

[@dataml-trading4085]

@@ChadThackray Thanks, Indeed.. It is pretty steep ! lol But I've already got a super stable/high-performance comparing to Python websocket-based server. It never fails with C++ like memory consumption! It's worth to learn it man~ The only downside is I cannot do DataAnalysis/ML stuff with Rust bc it has a poor ecosystem in terms of it so Python would be the #1 choice for ML thing

[@dataml-trading4085]

I think it just took a month to learn basic Rust and make a simple high-performance websocket server with Database system like TimescaleDB. so only for realtime ingest dataset consumption, learning Rust is definitely worth it.I can keep doing ML thing with Python communicating with Rust via Websocket.