Hidden APIs with Scrapy - easy JSON data extraction
HTML-код
- Опубликовано: 10 сен 2024
- I've shown this web scraping method before but never using Scrapy, and given that the Scrapy framework gives us some reaslly good features I thought it was about time I demo'd this. This is it in its most basic form.
This Scrapy project will should you the basic methods for scraping API like data from a website, be it a proper API or the API endpoint you find when scraping a web site.
Support Me:
Patreon: / johnwatsonrooney (NEW)
Amazon UK: amzn.to/2OYuMwo
Hosting: Digital Ocean: m.do.co/c/c7c9...
Gear Used: jhnwr.com/gear/ (NEW)
-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
-------------------------------------
Good to see it in Scrapy. Your channel need more Scrapy tutorials. 👍
More coming, I have a few planned!
Awesome videos John. I wish I had found you before I paid money to learn everything you're explaining here more succinctly and free 👏
i requested for your scrapy x api video and voila it's right here, thank you!
Ha yeah I hope you enjoy it!
Tremendous help - thank you so much.
I try using this same method but I did not get an object attribute. The object values were blank eg, d[]. But if I check the response tab I see the values in the Json format. Any idea what could cause this?
I'm getting a 429 unkown error. What type of method should I use to slow down my scraper calls?
As always, detailed & clear explanation. Threw me off when Pycharm was fired up though 😅
Haha snuck it in there!
Love your videos, really helped me get a project off the ground. Could you do a video on not overwhelming an API server with requests? What is the best way to slow the requests to the server down? I would like to hear your thoughts, process, etc. keep up the great work!
Great suggestion! I'll add it to my list of video ideas
@@JohnWatsonRooney thank you, and again really wonderful job you're doing with your videos. Be blessed.
There are several ways to slow down the request, the most commons are:
1) Sleep after each run of a for loop
2) Limiting the quantity of requests made, for example: each 10 requests, sleep more 30seconds
Often I go with the first one that is the easiest to implement:
from time import sleep
for x in my_links.json()['data']:
sleep(0.5)
print(x['id'])
each loop the code will sleep for 0.5 seconds, decreasing the flood of requests..
@@JohnWatsonRooney Hi, can make a video or share guide on how to do the same process as in this video with POST Method? cuz, the website i'm trying scrape has this data in POST request.. can you help me pls?
Thanks allot 💞💞, amazing video 😍
seems to me every website has its own custom API and blocks access to these type of request to even HTTP GET data
Could you make a tutorial where you deploy the scraper on a VPS? I've seen many options like using scrapyd or running a cron job. I'd be helpful to see examples.
Hey john! Many guitars in the back. So any plans for a music youtuber soon?
congrats with 10K
Thank you!
helped me, thanks!
Thanks for this video.
hey sir if the scrapy.Request(-) is not showing any thing what should i do
is this works for privat company network??coz i failed just like need to login..or something
Hi John,
I tried scraping a website using hidden API like this, I succeed in parsing the first page.
When I tried to loop the next page, it returned 403 error.
Now, when I tried going back to parsing one page only, it also returned 403 error
I have tried changing the user agent in the settings.py, but still no luck
I can open the API endpoint link just fine in browser. So I think it's not an IP ban
Can you suggest something ?
I think you need to copy the cookie over from your browser and put it in the headers you are sending. Copy the request as curl and see what the headers are and try putting them into your code
@@JohnWatsonRooney it turns out I was actually blocked by Perimeter X when I manually visit the website. So assume this is IP ban right ?
@@DittoRahmat Yes sounds like it, bot protection. Assuming you don't have a static IP from your ISP you can restart your router for a new IP, or just wait they don't usually last for that long
@@JohnWatsonRooney how do you put the copied cookie (and other headers) into the scrapy spider code can you cover this please?
@@JohnWatsonRooney because I'm having the same isssue where I'm able to get it to work following your earlier video using requests and insomnia but not in scrapy
Is there a good reason to use Scrapy for this instead of the requests library? Isn't it bringing a gun to a knife fight?
I try to show lots of examples and I like the easy expanding use case of Scrapy but yes here it’s not needed as such
@@JohnWatsonRooney Good to know, thanks! Your videos are very useful.
Do you have a video or can you create a video on how to schedule recurring scraping? Ie, say having the scrape run every hour?
I’ve covered it in my cronjobs video- and am going to cover it again soon
super cool very well explained thanks so much. subscribed :)
Hello John, could you compare Python Requests and Python Scrapy? I just found out about scrapy but want to know the caveats between the two.
Thank you man!
thanks for watching!
Awesome 👍
Awesome! But what if the website doesn't make any xhr requests? Is headless browser the only way (by clicking and pretending to be a user)?
That is a way yes, but more last resort - if we can render the page with the headless browser and grab the html to parse that way it’s a bit better
You did with GET method.
How to do this task if you have POST request/mthod?
I am trying to scrape foreclosure .com and response.body will print just fine but will throw errors when I try to load it as json.loads
Great video! The hidden api I'm trying to work around is a fetch type though. And it's response is really not as clean as this one. I don't know how to work with it really
Try saving the response to a file and opening it up in a separate Python script and work out how to extract the data you need then reimplement it into your main project. Notebooks are good for this too
@@JohnWatsonRooney how about I just use selenium? This seems like a lot of work, the text is so jumbled
@@JohnWatsonRooney though I never worked with selenium before
Bravo
Hi John
Can you please guide about instagram scraping
Thanks
Great Video. I am wondering if scrapy can get that long URL by itself instead of copy and paste by ourself?
Loved the video!
Any plans for a web scraping course soon?
Yeah I am planning one, just not sure where or how to release it. Or just make it free on yt
@@JohnWatsonRooney You can try to upload on Udemy. As i have checked there is not much content available on Web scraping. And thanks for making short tutorial on Yt like always
@@JohnWatsonRooney would be great to learn how to do price comparison from website to website and how to track that data / store it.
Fine
That's a nice video. Like always. Hi, John, Can you post a video to show how to scrapy home depot product reviews? Thank you.
Thanks for video!!!!!!
I did this but unfortunately the API is salted 😢
do you have example for POST request?
What is software where was GET editor?
God bless 😎👍🏻
John I came up with your youtube channel and it is an amazing resource! Right now I am working in scrapper project but I have issues with ng class elements in the website I have done my research but without luck. Can you recommend something or a video in your channel? (I coding in Python)
Regards!
Hi John, My comment is unrelated to this video. I've been trying to scrap music data from streaming platforms like soundcloud for the data only not the actual music. to create a analytics platform for independent musicians. and these websites are loaded dynamically so its been giving me a problem. I tried everything from selenium to request_html but it is just not happening. Could you please do a video on it. So I can learn.
what's the website and fields your trying to scrape?
@@brothermalcolm spotify, amazon music, tidal & youtube music. the fields are name, song name, streams, date uploaded and so forth.
Can you do a video showing how to scrape wikidata info on people like famous programmers?
hi, i follow your content, and your content awesome at all! but, can i get tutorial scrapy scraping graphql & follow the link after first request? thank you!😉
Hey! Thanks. I’ll be covering some of those topics coming up, I’ll see if I can drop that in too!
This is really cool but also kinda illegal, I guess it depends on what data you are fetching
The legality is a bit grey but we are only getting data that is publicly available online, it’s not behind a login nor are we abusing the website with 1000s of requests. I think if you use the data for personal consumption ie don’t try to sell it it’s ok
@@JohnWatsonRooney thank you for the response!
I need help scraping data from a website that has a firewall. Will pay
Can you make a video about android app scraping?
this are not hidden apis -.-
Hi John, I sent you an email. I'm having trouble with something I hope you can help me.
hey man, i will get to it as soon as i can