Thanks for the video! My first Python experience. I did get 'No module named 'openpyxl'' error when trying to export to excel, even though CSV worked. Although I tried to install via pip in VS. I ended up having to run 'py -m pip install --user xlsxwriter' in my windows terminal and then it worked. Thanks!
Hy, Thomas you've done a great job, I learned a lot of imp things from you by watching only this video. Plz make a playlist on web scraping from basic to advance so that a new person can understand better. Once again amazing video
Hello, very good. What about products that have variants that need click for show price? How are different prices handled, whether by color or other different features? called without ajax, in detail page
If you only scrape 1 page and only do it one time (so not for example every hour) I would say the risk of your IP being banned is pretty low.. In case you have any doubt, just google "free proxy server" - there're a lot of proxy servers you can use for free. The challenge with free proxy servers is that they aren't reliable - but if you only scrape one page that's not a big deal.. Good luck with your project :)
Great video and flow. I did this for another website but when i run the code at the end the scraping of pages should be 235 but it doesn't stop it keeps scraping pages? How do i stop it at the amount of pages i would like to scrape from the website.
Hi, there’s a link to my Discord server in the description of this video. If you share your script and the website you want to scrape there, I’m happy to help you with your script 😀
I followed the same code for a different website except I left out the if and else codes because I wanted to scrape only the first page. Everything worked out except my output which came back in the same column, and it is very disorganized. How can I fix this in my code?
Hi, best way to help you is if you share the website + your code on my Discord, then I can help you right away :) Join my Discord here: discord.gg/WHJWFNDXXX
I can't tell you whether this method works for specific websites. It should work for most websites, although in case you need to log in or the website uses lazy loading, you need a bit more work to get it up and running..
Hi! I'm practiced in C#, but new to Python. I watched your video all the way through before watching again to try myself. My requests.get() works off and on without changing any of the code. I keep getting 'getaddrinfo failed' error even though it just worked with no changes made. I didn't add a proxy yet since that's later in the tutorial. Would that cause this?
Hi, this shouldn't be about the proxy. It could be either an issue with the website or the connection. Could you try: - Running the same script on a different network / PC - Try scraping from a different website (preferably a very simple one like the one I use in the demo - definitely not something behind Cloudflare) This will help you to figure out whether this issue is network/pc or website related. Good luck!
Thank you for your comment, glad my video is useful for you 😀 you can ask any questions in the comments, this way others who have the same question can also benefit from the answer.
Hi Thomas, thanks so much for sharing it! i was looking all day trying to find some good material and this by far saved the day!
Thank you for your comment! Glad to hear my video has been helpful for you :) happy coding!
Thank you so much for your valuable information and the easy-to-follow explanation style.👏👏
You're very welcome Abdullah 😃 is there anything else you would like to learn?
Thank you so much man. I'm totally beginner and it's very easy to follow.
Thank you for the kind words! Happy coding 😀
Very good explaination.. Tyx.. Hope you make some more videos on data scraping using BeautifulSoup
Thank you for the kind words, there’re more data scraping videos in the pipeline 😀
ty please consider covering other frameworks or websites
Thanks for the video! My first Python experience. I did get 'No module named 'openpyxl'' error when trying to export to excel, even though CSV worked. Although I tried to install via pip in VS. I ended up having to run 'py -m pip install --user xlsxwriter' in my windows terminal and then it worked. Thanks!
Congrats with your first Python script! Here's to many more scripts ;)
Hy, Thomas you've done a great job, I learned a lot of imp things from you by watching only this video.
Plz make a playlist on web scraping from basic to advance so that a new person can understand better.
Once again amazing video
Thank you for your comment, expect more videos about web scraping soon! Cheers, Thomas
@@TomsTechAcademy Thomas, that's good for all new web scrapers bcz your style of conveying useful content is very simple thanks ☺☺☺
hello, I can teach you web scraping from the basics to advanced
Hello, very good. What about products that have variants that need click for show price? How are different prices handled, whether by color or other different features? called without ajax, in detail page
Great video, really very helpful, thanks!
Thank you Ebin! 😀
There could be f used instead of plus
Thanks for your suggestion :)
Awesome tutorial 👍
Thank you for the kind words! Happy coding :)
Thank you for this video. What is the risk of web scraping a Google scholar page (only one page) with an unhidden IP address?
If you only scrape 1 page and only do it one time (so not for example every hour) I would say the risk of your IP being banned is pretty low.. In case you have any doubt, just google "free proxy server" - there're a lot of proxy servers you can use for free. The challenge with free proxy servers is that they aren't reliable - but if you only scrape one page that's not a big deal.. Good luck with your project :)
@@TomsTechAcademyThank you 🙏🏽❤️
Great video and flow. I did this for another website but when i run the code at the end the scraping of pages should be 235 but it doesn't stop it keeps scraping pages? How do i stop it at the amount of pages i would like to scrape from the website.
Hi, there’s a link to my Discord server in the description of this video. If you share your script and the website you want to scrape there, I’m happy to help you with your script 😀
Great Video , Thanks
Thank you, glad to hear my video is useful for you 😀
Since the items are added to a set, how do you ensure that the columns always come back in the same order?
Hi, the items in the dictionary (e.g. Title, Link, Price) will always be in the same order
Good video. Many thanks.
Thank you for your comment Code Cobber! Glad my video was useful for you 😀
Can i scrape the reviews from amazon through this same method? Cuz I'm unable to access the page through my requests
are you new to web scraping?
Thanks for the video! The only issue I have is that it will not allow me to put the results into an excell file. It does for a csv, but not excell
hello i can help you with this
Hi, you mention "it will not allow" - which error are you getting exactly?
Hey I also had this problem. I fixed it by adding the openpyxl inside my python interpreter
I followed the same code for a different website except I left out the if and else codes because I wanted to scrape only the first page. Everything worked out except my output which came back in the same column, and it is very disorganized. How can I fix this in my code?
Hi, best way to help you is if you share the website + your code on my Discord, then I can help you right away :)
Join my Discord here: discord.gg/WHJWFNDXXX
Good tutorial.
Thank you for your comment, glad my video is useful for you 😀
very nice
Thank you for your comment, glad the video was valuable for you 😀
Nece tutorial, thanks
You are welcome, glad my video was helpful for you!
thank you so much
Thank you for your comment, glad my video is helpful for you :)
does it work in similarweb
I can't tell you whether this method works for specific websites. It should work for most websites, although in case you need to log in or the website uses lazy loading, you need a bit more work to get it up and running..
Ich rieche deutsch
You’re almost right, my native tone is Dutch :)
Brilliant. Thank you.
You're very welcome!
Hi! I'm practiced in C#, but new to Python. I watched your video all the way through before watching again to try myself. My requests.get() works off and on without changing any of the code. I keep getting 'getaddrinfo failed' error even though it just worked with no changes made. I didn't add a proxy yet since that's later in the tutorial. Would that cause this?
Hi, this shouldn't be about the proxy. It could be either an issue with the website or the connection. Could you try:
- Running the same script on a different network / PC
- Try scraping from a different website (preferably a very simple one like the one I use in the demo - definitely not something behind Cloudflare)
This will help you to figure out whether this issue is network/pc or website related. Good luck!
Nice explanation, Very Good, I have a doubt , Kindly provide your email pls....
Thank you for your comment, glad my video is useful for you 😀 you can ask any questions in the comments, this way others who have the same question can also benefit from the answer.