Python Tutorial: Web Scraping with BeautifulSoup and Requests

Поделиться
HTML-код
  • Опубликовано: 26 сен 2024

Комментарии • 1,1 тыс.

  • @sng8240
    @sng8240 4 года назад +1188

    *Great video! Even the IBM staff suggested watching this video to complete a part of a course offered by them. Did you know? Congratulations!*

    • @coreyms
      @coreyms  4 года назад +270

      I didn't know that. Very cool! Thanks for letting me know!

    • @joaquinrequena3494
      @joaquinrequena3494 4 года назад +13

      Damn course, I'm getting whipped there

    • @JoaoVitor-dq9sf
      @JoaoVitor-dq9sf 4 года назад +13

      Yes, it was Alex Aklson from IBM

    • @vaibhavkrkm
      @vaibhavkrkm 4 года назад +2

      He deserves that 😀

    • @amazing-graceolutomilayo5041
      @amazing-graceolutomilayo5041 4 года назад +11

      I don't know why I'm really happy about this 😂. Thank you Mr Corey

  • @Masonx42
    @Masonx42 6 лет назад +737

    Hello,I'm a viewer from China, comparing with other youtube python tutors, your English speaking is very clear for me to understand. I just registered as a one dollar patron for a little support.If I can find a job as programer later, I will contribute more to help you continue making these high quality videos. Thank you Mr Corey!

    • @coreyms
      @coreyms  6 лет назад +69

      Thanks!

    • @abby4922
      @abby4922 5 лет назад +3

      @@coreyms hey you still using this acc for youtube?

    • @abby4922
      @abby4922 5 лет назад +5

      hey Einsteiincy , are you still using this youtube acc, im also from China, would like some help with python

    • @kyefondeme136
      @kyefondeme136 4 года назад

      @@abby4922 i think we in China knowledge seeker lol

    • @abby4922
      @abby4922 4 года назад

      @@kyefondeme136 hahaha broah lets link up,

  • @samueltukua3061
    @samueltukua3061 4 года назад +7

    I started watching your videos years ago and I was afraid you wouldn't be around when I returned from my programming hiatus. Thank you for still being here!

  • @imtiazshahed944
    @imtiazshahed944 5 лет назад +57

    Corey Schafer Thanks a million. best tutorial I have ever seen on Web Scrapping in Python, No one explain like you in a very single point of code. That's how we learn. great job and keep it up.

  • @failuregreat9424
    @failuregreat9424 6 лет назад +198

    you never fail to impress me with your teachings! I don't watch videos at all once, because I feel it's easier to learn watching by 'parts' - but again and again, I'm watching the videos and I feel 'wow.. this is really well done, I have to like this, it's the least I can do' and I realized I already liked before lol

    • @cosmicdarkmatter1128
      @cosmicdarkmatter1128 5 лет назад +5

      @Failer Great...Agreed. Corey Schafer is very good at explaining things. A lot of professors can't do that....

  • @proxyscrape
    @proxyscrape Год назад +6

    5 years ago, this video was already a great tutorial. Now, it's even more valuable. Thanks for the great content!

  • @samdavepollard
    @samdavepollard 6 лет назад +267

    45 minutes of pure, unadulterated, hardcore geek-porn.
    Absolutely loved it.
    Thanks, Corey. Even by your standards this one was pretty special.

    • @coreyms
      @coreyms  6 лет назад +17

      Thanks!

    • @pauloesperon7697
      @pauloesperon7697 4 года назад +2

      I felt the same.

    • @mateusribeiro9384
      @mateusribeiro9384 4 года назад +9

      I actually enjoy much more watching his videos than watching normal movies, series...

    • @dope8452
      @dope8452 4 года назад +2

      @@mateusribeiro9384 Nerrrrrd

    • @shivanishimpi7018
      @shivanishimpi7018 3 года назад

      @@coreyms at the time 26.50, how do I scrape out both tags instead of just one?

  • @leslievanelsie7545
    @leslievanelsie7545 2 года назад +1

    I'm not really a video type of person, I learn faster with books but I always find myself coming to your channel to clarify things. You are the best Corey

  • @user-lh4hv3tx8b
    @user-lh4hv3tx8b 4 года назад +3

    Every time I find myself struggling with python as a beginner and going through so many random sources, you always seem to be the clearest.

  • @sairamsubramaniam8316
    @sairamsubramaniam8316 4 года назад +17

    You are the best programming teacher I ever had.

  • @varunabhi
    @varunabhi 4 года назад +4

    No where I have seen advanced topics explained in an easy way like this. Even in paid courses
    THANKS Mr.Corey. THANKS A LOT!
    I'd learnt much from your videos in a easy way.

  • @mohanlpuri1
    @mohanlpuri1 4 года назад +5

    I have watched couple of videos on web scrapping with Python. This video is very clear, and pace of delivery and the example teaching of the author is excellent.

  • @tntcaptain9
    @tntcaptain9 4 года назад +4

    This is how to teach in a proper way, in some videos a code is already given and they just read it out which makes it difficult to understand.
    Thanks Corey, keep up the great work.

    • @Reese_414
      @Reese_414 3 года назад

      Tech with Tim is horrible at that... I'll click to watch a tutorial on how to do something and he pulls up a game that's already half way complete... And on top of using pre-written code, he also talks very fast... Salute to Corey for live coding while speaking clearly with good explanations!.!.!

  • @Reese_414
    @Reese_414 3 года назад

    Just wanted to share something I do to practice web scraping that others may find useful... Instead of running your request to a live url while working out your code and possibly overwhelming a website, I right click on the browser and select the "view source" option. Then select all and copy the source code and paste as an HTML file in the same local folder then scrape through it until I get it working the way I want... May not work with all sites, but good for simple sites that aren't JavaScript heavy...
    P.S.
    Excellent tutorial, thank you for being so clear and so thorough!.!.!

  • @gwanghyeongim768
    @gwanghyeongim768 4 года назад +6

    Thank you Corey. I can personally assure that your work doesn't go for nothing. I hope I can donate more to your patreon in the future.

  • @SableWinters
    @SableWinters 2 года назад +1

    The best and quickest tutorial for small to medium sized website data scraping I’ve encountered. You are clear, concise, and you voice is easy to listen to and follow void of irritation. I actually have confidence now to achieve my tasks after years of placing them on the back burner.👍🏼🤗

  • @geekyprogrammer4831
    @geekyprogrammer4831 6 лет назад +85

    Best Python WebScraping video on RUclips! Seriously brother you deserve money for this masterpiece! Keep it up :')

    • @jw2965
      @jw2965 4 года назад +3

      pls no charge. me no have money

    • @rift8997
      @rift8997 4 года назад

      @@jw2965 what the heck

    • @jw2965
      @jw2965 4 года назад

      @@rift8997 was trying to be funny, but jesss i failed hard.

    • @rift8997
      @rift8997 4 года назад +1

      @@jw2965 ye ok i was just confused. lmao

  • @daringetae5136
    @daringetae5136 3 года назад +1

    Best explanation I've found so far on how to scrape websites using soup. Worth watching yours.

  • @RadioactiveNinja
    @RadioactiveNinja 4 года назад +19

    One of the best tutorials Ive ever followed. Hopefully you help kickstart my career in data science!

  • @mohamedabdo-xg6tw
    @mohamedabdo-xg6tw 4 года назад +1

    I don't know why some people disliked this video? I paid a money to learn this and the content of the class was way bellow this level, thank you for the great Tutorial!

  • @manojuppala3941
    @manojuppala3941 4 года назад +3

    Corey is the world's best python teacher.

  • @oluwadamilaretijani1777
    @oluwadamilaretijani1777 2 года назад

    Mr Corey, please wherever you are, keep safe. I really miss your content on RUclips. You're a legend when it comes to teaching.

  • @eyoba6027
    @eyoba6027 5 лет назад +7

    My first video from your channel and it didn't took me long to press the subscribe button. You are just perfect, no second wasted. Every word is important and worth mentioning. Cheers!

  • @BoxingPills
    @BoxingPills 3 года назад +2

    Im doing IBM data science certification. Understood nothing. After 2 minutes of this video I m already hooked! you re phenomenal!

  • @adityatripathi9865
    @adityatripathi9865 6 лет назад +3

    your understanding of python is very deep and your teaching method is excellent

  • @balasubramanian4314
    @balasubramanian4314 5 лет назад +2

    ONE OF THE BEST DEVELOPER I HAVE EVER SEEN in this planet! Hats Off to your skills!

  • @OrientalMelodies2010
    @OrientalMelodies2010 6 лет назад +5

    An awesome video. Your way of moving from one idea to another is amazing. I have been searching RUclips for years for a good tutorial video explaining web scrapping with BeautifulSoup and finally found it. Thanks a million!

  • @amitanand9125
    @amitanand9125 5 лет назад +2

    Corey Schafer : I don't know if Python is easy or you made the Python Learning very easy.I did watch almost all your videos till date and refer my friend to do so.
    Thank you for making our live easier with such a ease of learning

  • @alexpaes9161
    @alexpaes9161 5 лет назад +19

    It could be possible to hit "like" twice for ur videos! Mate, ur a legend! Congrats!

  • @prabav86
    @prabav86 5 лет назад +1

    Hello from Canada! I've been impressed with the way you teach Python and related subjects. So I was referring/recommending your youtube videos to my friends. Recently, I'm finishing up online course on Data Science from IBM, I was pleasantly surprised that course recommended your video on webscrapping! All your hard works helping a lot of folks out there. Simply outstanding! take care

  • @KenPryor
    @KenPryor 5 лет назад +18

    This was the first of your tutorials I've seen. I definitely subscribed after this great video. I've struggled with learning Python and this was very helpful.

  • @FredMny
    @FredMny 3 года назад +1

    Thank you!!! I love it how you are able to not over explain, but also without leaving out anything important.

  • @qianlima2196
    @qianlima2196 5 лет назад +19

    I spent 1 hour to figure out why my outcome is different from yours, then I finally realized it's because your website content has changed!

    • @iumair_mehmood
      @iumair_mehmood 3 года назад

      I am glad you figured it out 😂lolx

    • @i8nff225
      @i8nff225 3 года назад +1

      @@iumair_mehmood 😂😂

  • @shariqahmed7319
    @shariqahmed7319 4 года назад

    One of the finest web scraping with BS4 tutorials on RUclips! Thank you so much sir!

  • @allahakbar9580
    @allahakbar9580 5 лет назад +5

    Such a beautiful and comprehensive explanation! I liked the idea that you started from the simplest things, like scraping a small HTML file, not an entire website! Because that what I could not find in the many videos I watched about BeautifulSoup, they jump into the deep straight ahead!

  • @allistergraham9448
    @allistergraham9448 4 года назад +1

    Very informative video, you're an excellent teacher! While other RUclipsrs ramble off topic, you get straight to the point, and I really appreciate that.

  • @MrKashyani
    @MrKashyani 6 лет назад +3

    Man..The way you explains !!! It works like a charm. Be it Regex or web scarping, it's just awesome ! Love your posts.

  • @dhssb999
    @dhssb999 2 года назад

    This is definitely the best web scraping tutorial I've ever seen!

  • @hasanfares5774
    @hasanfares5774 6 лет назад +17

    thanks was just taking an online class about it and couldn't understand until I got the notification you just uploaded wow you really are in sync with your followers

  • @hamzakhalid8668
    @hamzakhalid8668 3 года назад

    This guy is the best. Hands down. For python tutorials. My man slayin'

  • @Nearco1000
    @Nearco1000 5 лет назад +11

    Just curious about those 34 slackers that did Thumbs down on this vid, seriously?.
    It's very clear and straight forward, plain English, highly understandable .......................Thumb up....
    Very appreciated #Corey, I was looking for this, great vid, again and again tku for your contribution.

  • @md.bodruzzamansifat7357
    @md.bodruzzamansifat7357 4 года назад

    Brother, your videos are always great. I follow them and learnt python very easily. This course worth money but you are serving free. Thanks a lot.

  • @tarunsukhpalani9514
    @tarunsukhpalani9514 4 года назад +3

    Master Corey, you are a magician. I just mesmerized.

  • @rajnigoyal6653
    @rajnigoyal6653 4 года назад +1

    Thank you Corey Schafer.. I really loved your teaching style and got every part of this. The best part is pointing out about the problem and give the solution.

  • @MarkJay
    @MarkJay 6 лет назад +27

    Another great video Corey! Keep them coming.

  • @iliveforthevibez6216
    @iliveforthevibez6216 3 года назад

    I try to get into Data Science and I did a coursera online course to learn python which also contained lessons about html and web scraping with BeatifulSoup4. I did not understand everything that was explained in coursera but your tutorial right here summed everything up in 40 minutes and could have saved me some hours if I had watched it earlier. Instant subscription man, the way you teach (and also speak, I am from Germany so it is easy to understand) is amazing , keep blessing us with your videos !

  • @dpoznyak
    @dpoznyak 6 лет назад +13

    Corey your Python tutorials are great! You explain everything very well. It would be great if you could make a series of Django tutorials for beginners! Keep up the good work!

  • @X19-x5f
    @X19-x5f 4 года назад +1

    Excellent video Corey, thanks. I finally got to see it because we're all quarantined to our houses due to the virus. Videos like this make it bearable.

  • @kanika724
    @kanika724 5 лет назад +3

    Corey a Big Bow from India..... really you are too amazing

  • @pydocs-pro
    @pydocs-pro 4 года назад +1

    Corey, this is just THE BEST explanation about web scrapping with python. Just notice that you could explore a little bit more about the page response with the get method. Sometimes I have trouble trying to access one specific site, and I have to manage it in different forms. And some more detailed explanations about when it's possible and not possible to scrape data from websites.
    Only this detail that I've missed. All the rest it's just perfect.

  • @18nikhilchauhan80
    @18nikhilchauhan80 4 года назад +3

    thank you so much. i got my first client just because of you.

  • @ikki411
    @ikki411 3 года назад

    This tutorial was incredible. I've done 2 Python courses that touched the 'Web Scraping' subject, but I wasn't able to fully understand it. This video was one of the two videos that made me fully understand it, and I couldn't be more happy about it.

  • @amysandstrum531
    @amysandstrum531 5 лет назад +3

    Thanks! I have very limited knowledge of html but was able to scrape price information off of websites. Just for funsies :D
    Really, all i did was tweak it and instead of div they use span class, so i switched 'div' with 'span' and got the same result.

  • @abdallahalaa492
    @abdallahalaa492 2 года назад

    What a great teacher, I never wanted to watch any tutorials about scraping as it was explained in then most boring way by others, but when it comes to you , you did a great job (y)

  • @saritaprabhu
    @saritaprabhu 5 лет назад +3

    You're a legend! There's a lot to learn from you. 🙇

  • @leosimon1746
    @leosimon1746 5 лет назад +2

    I have to say you did a great job! You covered pretty much everything that a novice programmer needs to know about scraping. I immediately developed my own solution after watching the video. Thank you so much!!!

  • @hashcoder
    @hashcoder 6 лет назад +5

    I've tried today! Successfully. Thank you, Corey!

  • @elliottcarter7700
    @elliottcarter7700 5 лет назад +1

    I can always count on clear and concise explanations from Corey Schafer! Thanks!

  • @lawrencekrukrubo2640
    @lawrencekrukrubo2640 5 лет назад +3

    This is just amazing, your knowledge, no your practical knowledge is amazing

  • @ayoadeomoniyi1538
    @ayoadeomoniyi1538 Год назад

    i love this tutorial because i feel confident to write even write python with this tutorial .God bless you.

  • @khrenovina
    @khrenovina 5 лет назад +3

    Thanks, man. You're great! Everything is clear for the beginner like me.

  • @clairein1433
    @clairein1433 4 года назад

    Thank you so much for making this video! I'm from China and I've searched many courses to learn Web Scraping. But it is you who make me successfully run my code.

  • @maicomcoelholopes9032
    @maicomcoelholopes9032 4 года назад +3

    Corey muito obrigado eu sou brasileiro, seus tutoriais são excelentes e me ajudam muito, principalmente porque não tenho dinheiro para comprar cursos, muito obrigado

  • @techknowbabble
    @techknowbabble 3 года назад

    Thanks for the video, using actual HTML language to explain helped immensely. Most tutorials just seem to be reading from a manual

  • @jakesnell7707
    @jakesnell7707 3 года назад +3

    Corey you have the coolest videos, you've taught me much my friend.

  • @timfulton6920
    @timfulton6920 2 года назад

    I love this training. I am going to subscribe for sure. I pasted with a questions about an error I was getting. Then I clicked play to continue and realized you answered that question next. Great video. Thanks for the help this is my first endeavor into Python. Thanks

  • @johnjacobs6234
    @johnjacobs6234 4 года назад +4

    Just a note, BeautifulSoup doesn't work with dynamic content.
    If you see something like:
    ::before
    (html content)
    ::after
    That content is dynamically loaded by your browser from some sort of redirect or Javascript. The html inside it doesn't exist when BeautifulSoup parses everything.

  • @oracle11iappsdba
    @oracle11iappsdba 3 года назад

    I watched this video just once but moved back the time line several times. Result I learned more than what I got in a stupid 1 day paid workshop. Thanks man.

  • @silverzero9524
    @silverzero9524 6 лет назад +8

    awesome was just surfing the web about bs4 and u uploaded this
    awesome
    awesome awesome thx

  • @mangaart3366
    @mangaart3366 Год назад +1

    Love how you explain things so clearly. Corey you are the best!

  • @wolfisraging
    @wolfisraging 6 лет назад +4

    Holy maaaaan, you are truly most amazing....
    Big fan😊

  • @saisrikanthdvn6901
    @saisrikanthdvn6901 2 года назад

    Excellent video on web scraping. I watched few videos in my National Language and local language. But didn't understand what they are doing. Might, the way you explain the things and use case of particular points in the right place in your video makes all understand better.Thank you so much.

  • @jackbird5839
    @jackbird5839 3 года назад +4

    great solution and tips bro!! thank you for your video., I will definitely try to improve my scraping skills with your tutor. but as non-tech user Also I have found good alternative to scrape amazon reviews e-scraper maybe it helps to somebody too.

    • @willjohn6807
      @willjohn6807 3 года назад +2

      Thank you Jack, e-Scraper really help me with my web scraping needs for my store.

    • @vskiy26
      @vskiy26 3 года назад +1

      Awesome tip Jack

    • @jonatanlast3849
      @jonatanlast3849 2 года назад

      @@willjohn6807 En este tutorial de programación de Python

    • @jonatanlast3849
      @jonatanlast3849 2 года назад

      @@willjohn6807 nasıl yapılacağını öğrenmek

    • @jonatanlast3849
      @jonatanlast3849 2 года назад

      @@vskiy26 BeautifulSoup o se meafaigaluega sili ona lelei mo le faʻavasegaina

  • @poorvikjain5323
    @poorvikjain5323 3 года назад

    why i didn't find you before sir
    i just wasted my time by seeing unhelpful tutorial of others
    you are one of the beat teacher of my journey in python 💓💓💓💓💓💓

  • @salah2fois
    @salah2fois 3 года назад +10

    Using Pyton 3, I added this part: csv_file = open('web_scraped.csv','w', newline = '') in order to avoid empty lines in the csv file.
    stackoverflow.com/questions/3348460/csv-file-written-with-python-has-blank-lines-between-each-row

  • @GeoffGroves
    @GeoffGroves 6 лет назад +1

    Great Corey! Something of note for new MAC / Py 3 users: If you have Python 3 installed, then you have to specify pip3 (pip3 install...) in your terminal commands for all library installs (soup, requests, html5lib etc) as MAC OSx ships with PY 2.7. Also, if you are building in Sublime as Corey does, you need to install a python 3 build package, otherwise Sublime will try to build it with Py 2.

  • @brucemurdock5358
    @brucemurdock5358 3 года назад +29

    00:00 What is webscraping
    00:30 Our end-goal
    02:23 Installation of modules/libraries/packages
    05:05 Primer on HTML and CSS
    08:32 Example
    19:27 Scraping the webpage

  • @pravinbagul5305
    @pravinbagul5305 4 года назад +2

    Thanks Corey for making my python journey very simpler with your tutorials!!!

  • @baconsledge
    @baconsledge 6 лет назад +23

    As always, Corey’s videos are top-notch. Thank you, Corey!

  • @DJ-ct6so
    @DJ-ct6so Год назад

    Excellent Corey. Very comprehensive, well explained, step by step, easy to follow. You're a very talented teacher imho.

  • @venkysmoments9155
    @venkysmoments9155 6 лет назад +22

    Corey please add tutorials related with multithreading and multiprocessing in python.

    • @ApplySkills
      @ApplySkills 6 лет назад +2

      Venkatesh K this gets my nipples hard^

    • @warofwars7608
      @warofwars7608 6 лет назад

      did u have nipple alert app installed

  • @Muzahid19dec
    @Muzahid19dec 4 года назад

    Thank you. Can't believe I have been building scrapers for months still got so much deep understanding from a single video.

  • @pkavenger9990
    @pkavenger9990 3 года назад

    Seriously your teaching style and explaining is amazing, you even explained what split does, most people just assume that you already know about it but even then you explained it and it refreshes everyone memories and helped those who forgot due to some reason.

  • @Edwin-ri8xu
    @Edwin-ri8xu 2 года назад

    Wow....didn't even notice that the video was 45 minutes long. Thanks, Corey for the wonderful explanation!

  • @_boris
    @_boris 4 года назад

    Whoever gets weird symbols when opening the .csv in *Excel*, change this line:
    csv_file = open('cms_scrape.csv', 'w', *encoding='utf-8-sig'*)
    Thanks once again Corey for brilliant content 💎

  • @efqanmustafayev5189
    @efqanmustafayev5189 5 лет назад +3

    Thanks for video, sir.
    How we can scrape login required websites?

    • @lucasmiguel2633
      @lucasmiguel2633 4 года назад

      Have you found out yet? I'm having a rough time trying to do this

  • @excessreactant9045
    @excessreactant9045 4 года назад

    Thanks for allowing users to use your website as a tool to learn

  • @MidjourneyandB
    @MidjourneyandB 3 года назад

    I just found your videos. I'm 5 months in to learning coding (toward data science). Your examples and explanations are extremely clear and helpful. I love how large the text is and how clear the narration is as well. Thanks. I will be watching tons more.

  • @lefebvre4852
    @lefebvre4852 2 года назад

    Thank you very much! It's well understandable for a beginner like me (also non-native english speaker) and 45 minutes have flown by quickly.

  • @appliedstatistics2043
    @appliedstatistics2043 5 лет назад +2

    simple and fast, best tutorial ever. Now I'm trying to scrape the IMDb review for text mining

    • @azmylle
      @azmylle 5 лет назад +1

      please teach me if you manage to do that ^_^

  • @dnyaneshwar53
    @dnyaneshwar53 Год назад

    Corey you're a blessing for guys like myself who wish to learn Python. 😃😃😃

  • @tirthajotisinha9088
    @tirthajotisinha9088 4 года назад

    And this would be the best video for web scrapping using python.Thanks a lot for making this a lot simpler.

  • @adarshtiwari7395
    @adarshtiwari7395 4 года назад +2

    I truly appreciate your compassionate teaching and dedication to the tutorials. Thank you for all the teaching!!

  • @abdullahsarker3595
    @abdullahsarker3595 3 года назад +1

    Subscribers += 1.
    Great video! I've never watched such a helpful video about web scraping. Thank you.

  • @yaboi2043
    @yaboi2043 4 года назад +1

    This is the only video I could find which explains this topic in a simple, understandable way. Thank you!

  • @TimFromPerth
    @TimFromPerth 4 месяца назад +1

    Outstanding. Clear and simple. Thank you.

  • @neighbourhoodcoder5988
    @neighbourhoodcoder5988 4 года назад

    this 45_M & 47_S tutorial= 10 hours of other tutorials+plus some books+some paid courses.
    keep it up...respect & love

  • @lucastoccheton989
    @lucastoccheton989 4 года назад +1

    Absolutely the best. Didn't need anything else for my first web crawler!!! Thanks you!

  • @codewithkolhar3131
    @codewithkolhar3131 5 лет назад +1

    Corey you made my career from an absolute IT nerd to hardcore Python developer. Thanks to you. You are gr8. God bless you.
    looking forward to learn Machine learning. It would be great if you make some videos on it. Thanks.

  • @centralscrutinizer5116
    @centralscrutinizer5116 Год назад

    You sir, by far, have the best structured videos for learning from. Great job there young feller!