Request Headers for Web Scraping
HTML-код
- Опубликовано: 29 сен 2024
- With every HTTP request there are headers that contain information about that request. We can maipulate these with requests or which ever web scraping tool we are using with Python to change how the server reacts to us. In this video i'll show you the basics of how they work and what they look like, and then demo how to change the most important ones in your code.
-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
-------------------------------------
Digital Ocean (Cloud Servers, Affiliate Link) - m.do.co/c/c7c9...
Sound like me:
microphone amzn.to/36TbaAW
mic arm amzn.to/33NJI5v
audio interface amzn.to/2FlnfU0
-------------------------------------
Video like me:
webcam amzn.to/2SJHopS
camera amzn.to/3iVIJol
studio lights amzn.to/3aBpKik
small lights amzn.to/2GN7INg
-------------------------------------
PC Stuff:
case: amzn.to/3dEz6Jw
psu: amzn.to/3kc7SfB
cpu: amzn.to/2ILxGSh
mobo: amzn.to/3lWmxw4
ram: amzn.to/31muxPc
gfx card amzn.to/2SKYraW
27" monitor amzn.to/2GAH4r9
24" monitor (vertical) amzn.to/3jIFamt
dual monitor arm amzn.to/3lyFS6s
mouse amzn.to/2SH1ssK
keyboard amzn.to/2SKrjQA
You are the Scrapy GOAT . keep up the content!!
Best content I found so far
You read my mind! This was the exact video I needed for today! Thank you for making all these videos!
Happy to help!
I hope you plan to make a discord channel or a Telegram group so the scraper community has a place to exchange ideas :) Awesome content btw!
Thank you dude! Interesting and useful content! Keep it up and good luck ;)
Thanks!
You make great content bro
Nice, thanks, pks talk about handling cookies that on every request changes
8:28 I like you man :D Sorry for the many comments. I am just very interested in this and you have the best content I have found so far!
Can you please make a in-depth video about cookies and how to use them with "request"?:D
Great tutorial. I follow your every video. Would you like to give us some tips to prevent blocking when scraping websites?
Did you find some solution?
Great one
Man anyway I can email you, I am stuck scraping a certain website for my business and the cookie is being set by the website under set cookie and I can’t for the life of me get to change. Pleaseee
Hey my email is on my main yt page - I’ll do my best to help if I can
Hi there! can you make a video on a chat bot?
// Standard for general requests
Accept: "*/*"
// Standard for navigation-requests in browsers
Accept: text/html, application/xhtml+xml, application/xml;q=0.9, */*;q=0.8
Just want to take the time to thank you for the high quality, easy to understand information in such a compressed amount of time in this video
Hey mate, look into selenium-wire to grab cookies for using to scrape APIs.
Will do thanks for the suggestion
Boy, you have great content. I am studying at the International Internet Academy. I'm learning Python. Your videos help me well in my studies. Thank you very much! Keep making good videos, don't stop. With best wishes from Russia!
Is it also a good idea to employ custom headers when sending requests to a REST API?
Thanks for the solution, just started with python and scraping 🤞
Thank you John for the great explanation of this important topic in web scraping! Also, the comparison between requests and requests-html is nice.
John congratulations for excelent video and explanation. Do you can a video using the method with playwright?
Thanks for the explanation. Didn't stump on this topic thanks to this video
❤🧡💛💚💙💜🤎🖤🤍 Starts 4:00
do u intentionally fade out of the images to annoy us?
Nice Change 🙄 Informative Dear👍
+ like for the guitars 🎸
BOOM! Great share John :)
i have a question , i get a response code 200 , so its good to go but , when i start scrapping it result to none, ??
I think maybe you are being sent a captcha page or similar, try printing the whole soup text and see what it is
This was a great video
Great.. John Bro, You are the Man... GBU
Why do requests headers let us make our programs more human-like? Thanks!
We can send headers that a browser would normally sent automatically to make it look more like we are a normal browser
Thank you so muchh John!!!!
How do I scrape info from the payload after its submitted in the code? Im making a account creator that needs to print the token from the account, how can i scrape this from the headers/payload?
🙏 Thank you
Hey Man,
is there any way to extract the cookie value into microsoft excel automatically? if yes, please share something about it. thanks in advance
Once I've written the code in python, does that mean I can go to my browser and the details will transfer to the relevant website?
Not sure what you mean but the code does everything itself is won’t make anything in your browser change
@@JohnWatsonRooney no worries mate, still new to all of this. I solved it in the end
Nice hoody bro
Hi can we add underscore in request header? by default underscore is converting to Hyphen, can we restrict this? please suggest
Hi John, did you receive my email?
Yes sorry, reply coming!
DNT is not depricated? What can be an alternative?
the best like before watching
rock on!
Can u please explain how we keep both Selenium and Requests?
Great video man, Thank you
Glad you liked it!
Thank you ,helpful and fruitful
hi, john thank for best tutor, I happen to discuss about headers, I want to ask how to rotate headers so I can avoid captcha here I am facing a case where I have to change the headers when making requests, where the form data parameter changes how to get the form data automatically so that when running the scraper it does not replace it manually?
Rotating headers doesn't defeat recaptchas - you should read this: www.blackhat.com/docs/asia-16/materials/asia-16-Sivakorn-Im-Not-a-Human-Breaking-the-Google-reCAPTCHA-wp.pdf
To rotate headers, have a list of them in a txt file, the:
with open('your file name.txt', 'r') as file:
useragent_list = [item.strip('
') for item in file]
Then you can use random.choice(useragent_list) in the relevant place for your header...
Hi,John .thank you for this nice video (as usually) .Please i need a help : can we store scraped data of images into a csv file ? ,Thanks again .
Definitely.
The channel is growing. Keep it up John!