Subscribed. The last 4 minutes of the video is exactly what I needed. The Soup tree structure part, specifically dissecting the price out of the HTML code. I could get the price on my own web scrap script but it also came with a bunch of other "junk" that was apart of the "tree". Thanks for pointing me in the right direction and explaining how it works!!!!!
I was puhsing learning web scraping for some time now and finally jumped in today and watched my first video on this topic. I like Tim's videos because they are simple and easy to underatsnd, so I decided to go with his video on this topc. Thank you.
@@IanWeingardt It appears that the content might be dynamically loaded by JavaScript in amazon website, which means it might not be present in the initial HTML response
SO I am writing some software to start a business. I am scraping 25 web pages. I hacked through the first 20. The last 5 were difficult so I tried using BS4 with this video. Using BS4 made the last 5 easier than any of the first 20. Thank you Tim!!
Hi, l love all your videos boss. Thank you very much. I do not have an IT background but fell in love with your videos and started studying Software development in school right now.
Unfortunately, the code mentioned in the video at 13:15 no longer seems to work, probably because NewEgg no longer allows a Python script to download the htlm from web site pages. It seems to me that most web sites are being "bot protected" today, a problem that is specifically mentioned by Tim in the video at 11:25. This points to a very significant problem when you consider using a tool like Python to web scrape, because using standard Python code is not ever going to work. Finally, when you do get lucky and get your Python code to web scrape, that code that works perfectly today will probably not work very long.
your videos on web scraping motivated me to create my own video-series about this topic(s)! as always, great content! thanks for sharing your knowledge.
Amazing stuff man. You got yourself a new sub. Thank you for all of the content and hard work. I've been benging all of your videos like I was watching Netflix lol
This was a really useful tutorial and it was clear to understand, unlike some of the other videos I found. Thank you! I'm sure there are many people out there like me that find themselves trying to slightly improve their code, resulting in learning how to use some new massive python library like this. Back to the video: I think it would have been good to replace the URL at the end of the video with another NewEgg listing to show the same code extracting a different price (assuming the tags are the same). Also, looks like you forgot to edit out the part at 8:24.
Great video and great timing to put it out! I had to use BeautifulSoup for the first time just last week and this was great at solidifying everything I learned!
Anybody know why I would be getting "IndexError: list index out of range" on line 10 - "parent = prices[0].parent" at the 15:29 minute mark in the video? I've copied the whole code exactly.
Do you have an availability if I paid for you a Zoom call? I watch your videos all the time and I'd really appreciate if I could hop on a Zoom at your hourly rate to answer a few specific questions. Thanks for everything!
Thank you Tim. I always wanted to learn Beautiful soup for personal projects but never did. Today is the first time I watched a tutorial on this topic. Anyway, how long will this series be? Can't wait for the next part.
I'm trying in a different website. I can get the HTML, but when I try to look for specific texts it doesn't work, I always get an empty list, even though I can see that there is that text in the page.
I have the same problem but if you noticed in the video he said that some websites may block you when you try to script their html page so maybe the is the reason why you get an empty list
@@abdulrahmanal-saadani8769 I have the same problem.. but I think that if the html can be seen in the console in the previous steps, that means the robots haven't done any blocking? I would think if you can see the data that's gameover once you learn how to manipulate it.
For me it only work if you look for the hall test in the element. For ejemaple The full text" will not work for "full" or "The full". It only work if you search "The full test". You can test it with a local HTML file. Is not the web filtering it.
1 quick point to add about BS4 not installing. Sometimes being connected to a VPN can stop modules being installed. Try dropping off the VPN and running Tims install commands
8:09 Isn't nesting tags in HTML impossible? This part got me confused Also at 16:12, couldn't you just use parent.strong instead of parent.find("strong") ? Great video by the way
So when I run the script looking for the "$" on the site I'm parsing, I get an empty list [ ]. Does that mean that the website is preventing me from seeing that particular item/price?
This reminds me of how some nights Grandpa and I would eat melty cheese in the mudroom. We laughed so much as cheese dripped down his face. I can still remember his laugh. It sounded like a hundred murders of crows filtered through a ring modulator. RIPO Grandpa please stop haunting my dreams.
I was following along with this video and couldn't get it to work. Actually I was following along but with my public "all games" steam library page. I couldn't figure out why it wasn't work. I was losing my mind. Then I finally saw in the source this JavaScript block with formatted data for all my games. It's "DB Query" and also the JS appends the data to the DOM programmatically. So indirectly this video taught me why Beautiful Soup couldn't find the tags I kept searching for on the steam library page. Side note, anyone want to scrape their steam library for some reason (instead of using steam db or whatever) it's all there on that page as some kind of JSON. Good video btw.
Tried 3 BS4 tutorials, on 2 completely fresh environments (one native py3 the other one a jupyterlab environment) and find_all() always returns empty. Any ideas why this happens?
@@ivanyosifov2629 For me, the .find("strong") returns NONE even though the print statement before it clearly shows strong tags, any idea what could be causing that?
I am using the find_all(text="something"), and it also returns and empty array... I know that what I am looking for it actually is in the document. Could the problem be something of how the html file is structured?
Amazing tutorial, but one question - in 9:09 what does the [0] do? I tried running the code without that and there was a bug, but with it it worked perfectly fine. Im just wondering what the function of that is
He accessed the first tag in the html file using index [0]. Supposed he used index [1] he would've accessed the second tag in the html file. He did something similar again @15:25 when he was locating the first parent tag of the price value. Hope that helps !
I tried your code but on a different webstore website. Definitely has "$" signs on the page and i was, with modification able to print the whole HTML code out via prettify, but I get this error : "IndexError: list index out of range" on the line "parent = prices[0].parent". Any ideas ?
great video , hit a bunch of roadblocks with the imports and environment %PATH% changes i had to make, and then the openssl issue, but yeah took me a day to get thru this and finish, i appreciate the hardwork.
Glad that you finally solved your problem! Me too, had a hard time with the %PATH% thingy!@$$. Time to time %PATH% problem appears from no where and eats up lots of my time. I have worked a little with R and I found it relatively simple and easy in this aspect, package installation and management etc...
Absolutely great videos, I’m new to python and coding in general. Your content is really great and easy to follow, would this web scraping method work for finding stuff like meta data for songs ?
How do you use BeautifulSoup to find an element with a specific attribute? For example, There is a div whose data-label = "Lifetime access to course content" and doc.find_all() did not work for it. I had to use doc.select('div[data-label="Lifetime access to course Content"]'). However, I was then not able to do another find_all or select afterwards. It would pass an error. I'm finding scrapy and xpaths to be much simpler than Beautiful Soup, and I'm confused at how to chain in B.S.
Hi, This is very interesting, Thanks for sharing. I have one question, Can we scrape website industry? Like Any site which belonging to any category like, Automotive or healthcare etc.
What if for some reason the price is not inside a predefined tag in let’s say 2 out of the 500 pages that I scrape, that’s going to cause an error right? Or do all pages follow the same format?
I started learning python few months ago and chose web scraping as my specialization. Your selenium playlist is fascinating. Thanks Tim
Hey how did you progress?
r u currently freelancing webscraping ?
@@AliAhmed63708 i am
@@alex59292 So where can I reach you for some webscraping freelancing.
@@hjvela1907 hello do you still need a web scraping freelancer?
Subscribed. The last 4 minutes of the video is exactly what I needed. The Soup tree structure part, specifically dissecting the price out of the HTML code. I could get the price on my own web scrap script but it also came with a bunch of other "junk" that was apart of the "tree". Thanks for pointing me in the right direction and explaining how it works!!!!!
I was puhsing learning web scraping for some time now and finally jumped in today and watched my first video on this topic. I like Tim's videos because they are simple and easy to underatsnd, so I decided to go with his video on this topc. Thank you.
The 'text' argument is changed into 'string' in Beautiful Soup 4.4.0.
yes thank you
thank you so much, I was very lost when I got the "DepacrationWarning"
@@IanWeingardt It appears that the content might be dynamically loaded by JavaScript in amazon website, which means it might not be present in the initial HTML response
SO I am writing some software to start a business. I am scraping 25 web pages. I hacked through the first 20. The last 5 were difficult so I tried using BS4 with this video. Using BS4 made the last 5 easier than any of the first 20. Thank you Tim!!
Man I was literally looking for a good tutorial on Bs4 and guess what Tim read my mind. Thank you very much Tim :)
And I started with web scraping like 2 days ago
@@BB-si6cz Oh that's cool !
Deffo RUclips AI reading your mind maybe it was Alexa
@@Damientrades I was already subscribed to Tim so I got the notification :)
I'm pretty sure that Corey Schafer also has a good tutorial about beautiful soup.
def amazing():
while 1:
print("Thanks Tim")
amazing()
Such a great tutorial! I usually don't comment but this was absolutely spectacular. Thank you so much!
Hi, l love all your videos boss.
Thank you very much.
I do not have an IT background but fell in love with your videos and started studying Software development in school right now.
I'm using Anaconda, so had to use different import information. You explained this better then any video I've watched.
Straight forward, simple explanations , easy to follow. Thanks Tim! You're a excellent teacher, keep up the great work you're doing here on youtube.
Unfortunately, the code mentioned in the video at 13:15 no longer seems to work, probably because NewEgg no longer allows a Python script to download the htlm from web site pages. It seems to me that most web sites are being "bot protected" today, a problem that is specifically mentioned by Tim in the video at 11:25. This points to a very significant problem when you consider using a tool like Python to web scrape, because using standard Python code is not ever going to work.
Finally, when you do get lucky and get your Python code to web scrape, that code that works perfectly today will probably not work very long.
real, its a struggle to learn when you're trying to debug and its just privacy policies in your html files xD
THANK YOU, I've been waiting for you to make a tutorial on this for some time now, thanks again.
8:26 Oh nooo,... you're in the matrix. You glitched!!!
Top tutorial!!!
Tim I just started scraping with BS4 THANK YOU!
your videos on web scraping motivated me to create my own video-series about this topic(s)! as always, great content! thanks for sharing your knowledge.
Amazing stuff man. You got yourself a new sub. Thank you for all of the content and hard work. I've been benging all of your videos like I was watching Netflix lol
Finaly the best timing ever i was waiting it plz speedeun that series
This was a really useful tutorial and it was clear to understand, unlike some of the other videos I found. Thank you! I'm sure there are many people out there like me that find themselves trying to slightly improve their code, resulting in learning how to use some new massive python library like this.
Back to the video: I think it would have been good to replace the URL at the end of the video with another NewEgg listing to show the same code extracting a different price (assuming the tags are the same). Also, looks like you forgot to edit out the part at 8:24.
Beautiful video man! Just realised how pedagogical and well dispositioned you videos are.
Many thanks. I was struggling with scraping some links from a page but couldn't until I watched this video.
Great video and great timing to put it out! I had to use BeautifulSoup for the first time just last week and this was great at solidifying everything I learned!
As always your tutorials are super educational and also thanks teaching it for free, it really helps.
"Dummy html file"
The html file who is trying his best: 😿👍
god that 3080 price gave me stress of when I was hunting down my own. Great tutorial btw it's been a great help!
I am good in Bs4 But I just came to give you like .For your work
you should have more than "just" 1,18m subscribers. thank you Sir!
Anybody know why I would be getting "IndexError: list index out of range" on line 10 - "parent = prices[0].parent" at the 15:29 minute mark in the video? I've copied the whole code exactly.
make sure your url is correct
Congrats on your 1 million,
All the best
Love your channel Tim, the best tutorial ever..
need this to access my uni's timetable more easily, thanks man !
Glad I could help!
The best tutorial ever! You're saving my life!
Great video. Thanks to you, I'm starting a new business in the tyre industry.
Do you have an availability if I paid for you a Zoom call? I watch your videos all the time and I'd really appreciate if I could hop on a Zoom at your hourly rate to answer a few specific questions. Thanks for everything!
8:23 did you forget to edit 😂😂
love your videos BTW ❤️
Dude, I think you should do more C++ tutorials. They're really good!
I second this. Those tuts are good!
Excellent done for a first example! Amazing explained!
8:25 It's always fun seeing bloopers mid video lol
i show this video after two years thanks pro❤❤
Thank you this video is so straightforward I think I finally got the hang of this
Thank you Tim. I always wanted to learn Beautiful soup for personal projects but never did. Today is the first time I watched a tutorial on this topic. Anyway, how long will this series be? Can't wait for the next part.
This serial is my need. Thank Tim!
I'm trying in a different website. I can get the HTML, but when I try to look for specific texts it doesn't work, I always get an empty list, even though I can see that there is that text in the page.
Same problem. Did you find a solution?
@@labscience8271 did u find one?
I have the same problem but if you noticed in the video he said that some websites may block you when you try to script their html page so maybe the is the reason why you get an empty list
@@abdulrahmanal-saadani8769 I have the same problem.. but I think that if the html can be seen in the console in the previous steps, that means the robots haven't done any blocking? I would think if you can see the data that's gameover once you learn how to manipulate it.
For me it only work if you look for the hall test in the element. For ejemaple The full text" will not work for "full" or "The full". It only work if you search "The full test". You can test it with a local HTML file. Is not the web filtering it.
Great tutorial Tim! I appreciate the clear and concise explanations you provided.
Awesome tutorial. New to this channel, but I like your style Tim. Thanks a lot, very well explained!
Quite a very concise course and informative. Thanks for this guide.
outstanding walkthru, as usual, ty... I like the chapter divisions, concise talking, maximized screen, text size :)
Anybody else having problems with bot protected sites? I finally got it to read on Wikipedia, but other websites return an empty string.
Web scraping is very useful trust me and of course beautifulsoup is the best library for this
Wonderful to be learning Beautiful Soup with Kermit
1 quick point to add about BS4 not installing. Sometimes being connected to a VPN can stop modules being installed. Try dropping off the VPN and running Tims install commands
this is great, good work!
8:09 Isn't nesting tags in HTML impossible? This part got me confused
Also at 16:12, couldn't you just use parent.strong instead of parent.find("strong") ?
Great video by the way
I am running into this same exact issue. Not sure what to do and can't find a stack overflow forum that's helpful
super helpful for a project I am working on, thanks!
I wanted to create a BS4 project 😀 and you made a tutorial
ur explanations are very clear,thank for ur effort.
As always, this was super clear
How do we input the html or the website we want to scrape? Great video, thank you.
So useful! Thank you so much!
hey, just updating.
find_all(text=) is deprecated
switch to string= method instead;)
thanks, but it still doesnt work for me
So when I run the script looking for the "$" on the site I'm parsing, I get an empty list [ ]. Does that mean that the website is preventing me from seeing that particular item/price?
im getting a index error when i put prices[0]
I think it is the website. I tried another website and it worked.
This reminds me of how some nights Grandpa and I would eat melty cheese in the mudroom. We laughed so much as cheese dripped down his face. I can still remember his laugh. It sounded like a hundred murders of crows filtered through a ring modulator. RIPO Grandpa please stop haunting my dreams.
best guy on yt
Hey Tim, can you make a video on how to get a programming job? 👋
I was following along with this video and couldn't get it to work. Actually I was following along but with my public "all games" steam library page. I couldn't figure out why it wasn't work. I was losing my mind. Then I finally saw in the source this JavaScript block with formatted data for all my games. It's "DB Query" and also the JS appends the data to the DOM programmatically. So indirectly this video taught me why Beautiful Soup couldn't find the tags I kept searching for on the steam library page. Side note, anyone want to scrape their steam library for some reason (instead of using steam db or whatever) it's all there on that page as some kind of JSON.
Good video btw.
Thanks Tim! :) awesome and helpful
Tried 3 BS4 tutorials, on 2 completely fresh environments (one native py3 the other one a jupyterlab environment) and find_all() always returns empty. Any ideas why this happens?
If find_all returns empty array that means what you're looking for is not in the document
@@ivanyosifov2629 I have the same issue and what I'm searching for is in the document. Even when I test with a simple html.
@@Xero_Wolf It might depend on the editor you are using. For some editors you need to give the file path as */index.html* or *./index.html*
@@ivanyosifov2629 For me, the .find("strong") returns NONE even though the print statement before it clearly shows strong tags, any idea what could be causing that?
I am using the find_all(text="something"), and it also returns and empty array... I know that what I am looking for it actually is in the document. Could the problem be something of how the html file is structured?
exactly needed this
Amazing tutorial, but one question - in 9:09 what does the [0] do? I tried running the code without that and there was a bug, but with it it worked perfectly fine. Im just wondering what the function of that is
He accessed the first tag in the html file using index [0]. Supposed he used index [1] he would've accessed the second tag in the html file. He did something similar again @15:25 when he was locating the first parent tag of the price value. Hope that helps !
@@Mmmkay.. Thank you so much!
I think Tim can read our minds 👀
So sick thanks for the video!
Thanks, this was easier than I thought
Thank you so much for such valuable content
I liked this video, good content.
Keep on your beautiful contents
Thank you! Now I can grab dad jokes from 20yo websites and send them to my friends!
Fellow man of culture
wooo wooo thanks TIM huge help!
Interesting video, as always
hey tim great work , i need to learn how to do using python columns and boxes in visual studio code thanks very much .
8:58 i tried bur it gives SyntaxError, what could happened?
Nice, good video, learned a lot
well played sir...well played.
Thanks Tim 🎉
I tried your code but on a different webstore website. Definitely has "$" signs on the page and i was, with modification able to print the whole HTML code out via prettify, but I get this error : "IndexError: list index out of range" on the line "parent = prices[0].parent". Any ideas ?
I am having the same problem, I even inspect element copied the $ sign from site incase there was unicode problem but it still didn't work
maybe website have bot protection?
great video , hit a bunch of roadblocks with the imports and environment %PATH% changes i had to make, and then the openssl issue, but yeah took me a day to get thru this and finish, i appreciate the hardwork.
Glad that you finally solved your problem! Me too, had a hard time with the %PATH% thingy!@$$. Time to time %PATH% problem appears from no where and eats up lots of my time. I have worked a little with R and I found it relatively simple and easy in this aspect, package installation and management etc...
@@unpatel1 another workaround is to use pycharm
@@andrews9168 Thank you for your suggestion. I do have pycharm but not using it, just use VS Code all the time. I will definitely try pycharm.
Thanks you a lot, you're the best.
the camera angle alone is increasing my intellect
nice editing job and content ofc thx
Tim you wont believe i was working on a meteo app and needed a parser thx
Absolutely great videos, I’m new to python and coding in general. Your content is really great and easy to follow, would this web scraping method work for finding stuff like meta data for songs ?
Perfect ! Thank you
Superb...🤩
Great video, but the most impressive thing is when you did this video the video cards price was $2613, and now it's $1549 in less than a year xD
Tim, please make a series or video fo the "requests" library.
How do you use BeautifulSoup to find an element with a specific attribute? For example, There is a div whose data-label = "Lifetime access to course content" and doc.find_all() did not work for it. I had to use doc.select('div[data-label="Lifetime access to course Content"]'). However, I was then not able to do another find_all or select afterwards. It would pass an error. I'm finding scrapy and xpaths to be much simpler than Beautiful Soup, and I'm confused at how to chain in B.S.
Hi, This is very interesting, Thanks for sharing. I have one question, Can we scrape website industry? Like Any site which belonging to any category like, Automotive or healthcare etc.
Hi Tim, nice video, need stuff. have you any links to a tutorial of you or else someone, who has scraped websites that block scrapers?
What if for some reason the price is not inside a predefined tag in let’s say 2 out of the 500 pages that I scrape, that’s going to cause an error right? Or do all pages follow the same format?
How come doing 'parent' on the $ sign returned so many layers of tags? Why was it not just the tags directly surrounding?
Can you make tutorial how I can scrape a dynamic web page? Like built with react.?
it would be same
That would only be possible with Selenium, which Tim has a series on, just search it