Web Scraping with Python - How to handle pagination
HTML-код
- Опубликовано: 28 июн 2024
- Join the Discord to discuss all things Python and Web with our growing community! / discord
This is the second video in the series of scraping data for beginners. We're gonna to really clean up our code by adding functions and adding support for pagination, including how to break out of loops.
This is a series so make sure you subscribe to get the remaining episodes as they are released!
If you are new, welcome! I am John, a self taught Python (and Go, kinda..) developer working in the web and data space. I specialize in data extraction and JSON web API's both server and client. If you like programming and web content as much as I do, you can subscribe for weekly content.
:: Links ::
My Patrons Really keep the channel alive, and get extra content / johnwatsonrooney (NEW free tier)
I Host almost all my stuff on Digital Ocean m.do.co/c/c7c90f161ff6
I rundown of the gear I use to create videos www.amazon.co.uk/shop/johnwat...
Recommender Scraper API www.scrapingbee.com/?fpr=jhnwr
:: Disclaimer ::
Some/all of the links above are affiliate links. By clicking on these links I receive a small commission should you chose to purchase any services or items. 
Наука
Love this series man. I really hope for the advanced series too. Thank you for sharing!!!
thank you very much, very kind!
same here
videos everyday??
oh man, thank you for your time!!!
thanks!
The next step is async scraping now 👀
Great series! Thank you tons, man.
Really valuable for all. Especially when new to this topic.
Absolute legend. Amazing content. Learning a tonne, thanks dude 🙏.
Great presentation! Neat use of Python’s yield.
THANK YOU!
thank you for this series, I think you should structure your future videos like this too. so maybe complex ideas/projects will be displayed better.
you got a course or something on udemy? I'd love to buy it both to learn from you and to support you a bit to show my gradtitude. I don't have a visa or credit card so I can't thank you on youtube!
Your videos are always amazing love your way of teaching can you please make video one google map contects scraping
Maybe if you could show how to pull all pages by handling how to bypass rate limit & ip blocking using rotating ip/ user-agent etc, then it becomes interesting! Hope you add such videos to this playlist.
Great video as usual John.
Per your other videos, would finding out max page count be alternate way for pagination stop condition versus checking for not 200 OK http response code? I like how you showed this option though. Thanks.
Yes I have done it that way before, sometimes there's justa "next page" button so you don't always know but certainly an option!
Can you make a video to scrap the data from tripadvisor restuarant ?Like a big website
Would love to see advanced web scraping with puppetier
John, could you tell me why httpx works with some urls and not with others?
genius!!😁😁
How would ou go about a webscraper that monitors changes to a website and notifies you ?
Hello John. Thanks for your videos. I’m learning scraping and recently saw one order on freelance, I decided to complete it for myself (to test my knowledge). The problem with this task is that when there is more than one page in a category, the site only returns data from the first page. 72 products are posted on two pages, and when you collect information from two pages, you get 36 products that are duplicated. I think the site has parsing protection. but how to get around it? I use a random proxy and user agent. What do you think about this? Can you give me your hint, what is the matter here and how to solve this problem.
Thaaaank you so much, veeeeery helpful 🙏🏾🙏🏾. You're the best.
Do you have any recommandation where we can host like this script as an api with fastapi framework or flask?
There are free places but I generally use digital ocean - they have an app deployment service which i use. I also heard good things about railway
@@JohnWatsonRooney Thaaaank you so much for your help and time 🙏🏾🙏🏾
If we scrape a website let's say every hour generally speaking is there a chance that we'll get our IP blocked ?
Depends on a lot but if it’s smaller amounts of requests you should be ok
I am learning alot John But i would recommend to make it more advanced level i had texted u at twitter too.. it would be of a great help if you pleaseeeee make a complete series related to scraping explain each n every aspect used in modern day scraping
I had seen alot of tutorials but you are just beyong someone can even imagine how good you...i really want to appreciate you but i would say please make a complete series/playlist where you can start spreading knowledge from basic 1st step to the highest last step scraping diff sites n all more power to uh John ❤
REI Shop: "Why is our conversion rate 100,000x lower on Camping and Hike Deals??"
can you make a video on how to use neovim
Sure I can
Thanks for lesson. I keep getting "NoneType" error -- "AttributeError: 'NoneType' object has no attribute 'text'" - on 22 in video - but it still runs through with the code. if I just keep hitting continue. 😄
Following this tutorial and creating new scraping projects based on new knowledge. Can't figure out yet, how to get the actual html links. Say there's a code, that looks something like: ... txt. How to get this "/art/7/" part? I can get the 'txt' part from the a-link tag, but not the actual link, that I would like to follow later. Please, help.
instead of calling ".text()" call ".attributes["href"]" and it will get it
Thank you!@@JohnWatsonRooney
Can we scrape Java ScriptED sites with HTTPX and SELECTOLAX?
no you'll need something to render the JS, a browser, or you can look to find the sites API and see if you can use that
@@JohnWatsonRooney does Playwright seem like a good choice?
@@rohitlekhrajani6217 yes it is, i've used it a lot and rate it highly
.......🎉... .
Pretty cool, can you do a video on scraping openapi specs from a website, if you have already can you post the link to the video?
Thank you for sharing your knowledge.
How the fuck can I do this to extract text like business titles, addresses and phone numbers?