A Guide to Web Scraping with Node.js
HTML-код
- Опубликовано: 6 сен 2024
- Learn how to build a web scraper ⛏️ with NodeJS using two distinct strategies, including (1) a metatag link preview generator and (2) a fully-interactive bot for Instagram. fireship.io/le...
1. Build a simple link preview scraper with Cheerio.
github.com/che...
2. Build an interactive Instagram scraper with puppeteer.
github.com/Goo...
#javascript #nodejs #webdev
Take the quizzes 🤓
iOS itunes.apple.c...
Android play.google.co...
Upgrade to Fireship PRO at fireship.io/pro
Use code lORhwXd2 for 25% off your first payment.
Me yesterday: Hmm...there is no API available, well, need to learn how to scrap with node.
Fireship today: A Guide to Web Scraping with NodeJS
Well done.
No one like this comment, it has achieved greatness
@Killian Malik Tried plenty of methods, this one seems to be working!
L
when you say as hackers i feel so cool
Haha, web scraping always feels very hacky.
@@Fireship Which font do you use?
@@anonwithamnesia Fira code
Haha yeah its hacky and cool. Once you start to make money off scrapped pages you will sued though
😊yes
Hi Jeff, thank you so much for your videos! We are building a startup and without your tutorials and explanations, I don't know if we could have ever gotten such a great insight how the mechanims work, especially for Ionic/Firebase.
Much Love from Germany
Wow, that is awesome! Keep me posted on your progress :)
@@Fireship If I didnt think you were an AI, I'd want to buy you a beer
So many things could be built around puppeteer, and the await syntax makes it so easy to reason about.
Also seems like this is a perfect use-case for cloud functions vs having a dedicated API server. Great video.
As a long time subscriber, you need to make tutorials every day. Fantastic work.
I wish I could, my idea list is massive.
I love the fact your using Insomnia for your rest requests, when I started working with API, I was told to try Postman, and found it overly complicated everything, then I found Insomnia, so much cleaner and easier to use.
I agree
I love it! It’s the most useful video on RUclips for me, ever! Today all day I was making app using puppeteer, I didn’t know it’s so easy, thank you as always for great video. Now I’ll be saving over 40 min every day at work 😂👍
When it comes to Web Scraping, most will talk about Python.
Felt very happy knowing Web Scraping can also be done easily using JavaScript.
Thanks a lot.
Javascript is the native environment for web scraping. Not only it is asynchronous, but also Javascript was created to be used on the web. To see people using interpreted, slower python for that is just sad. Not that there is something wrong with python, but it has other purposes. JS is much better for scraping pages generated with React, for example. On python, you can, at best, use a bot to simulate a user control. With JS, you can evaluate the DOM and then retrieve your data without the need of emulating a real user, and that is very powerful.
This video is literally like you listened to me today what I was wondering about. Thanks for the puppeteer part! :)
Awesome! The topic came from a suggestion in the slack channel last week :)
🔥ship is AWESOME!
Every time I have a project on my mind I find something useful to steal from this channel.
In an open source world there is no such thing as stealing.
This is one of the best programming channels on RUclips. Love the production on your videos!
Link preview I wonder if I will ever need this.
Said jokingly knowing they talked about it just 1 week ago.
Awesome content !!
I get my best ideas from Slack :)
@@Fireship not for nothing we love/hang around slack ;)
I only start understanding things better after watching your videos. It's such a great explanation and never boring.
Thanks for this upload. I got into a little bit of web scraping a couple years ago but ran into a lot more complexity than I was expecting for a tiny little side project. Now that I know a lot more, and JS, hoping to add this to the toolbelt.
Just as I'm building a news-scraper you make this amazing video! You're using the exact same packages I am :^) Great video as always!
00:43 use-case 1 02:19 cherrio 06:53 puppeteer
Thanks for those amazing videos... u r covering the full full stack 👍
You must be a mind reader. I was struggling with my link preview generator all week! Thank you soooo much.
I've been doing this for that past 2 days at work and you upload a video today haha , fortunately I got it working but its always nice to see you upload !
I love it! :D such well-made videos. I appreciate how you explain the problems that packages solve instead of just trying to peddling hype
Your content is top notch and so well done!! Easily my favorite programming channel
the puppeteer support for firefox is awesome!
An outstanding video that has been a valuable reference for our newcomers. We sincerely thank you for sharing!
You saved me a lot of time to find the name of a tool
It has been more than 3 days searching for that tool
For anybody wondering how to get firebase:
npm install -g firebase-tools
Thank you!
HUGE fan of your Node videos. Thank you so much! Best channel on RUclips.
Been here for a long time, your content just gets better and better!
Love how much more squeaky Jeff's voice was!
Love your content! Well done, my favorite youtube resource. Love that you get straight to the point.
Great content! I have previously made a scraper for a discord bot with cheerio before, but never really fiddled around with puppeteer or anything similar to it earlier. Thanks for the ideas :)
Hey one of the thing I have been searching from last 4 months..
Thanks again for the video Jeff. Last year, I was using a library called HtmlAgilityPack to achieve some of this with C#. You've made it quite simpler. YOU DESERVE THE FIRST FLYING CAR MAN👍🏾
OMG I’ve been looking for all the web for resources on WebScraping and suddenly your video appears! Well done!
Man you are awesome, all your videos are latest, useful and very informative. I very much happy that I have subscribed to your channel.
Dude...this is exactly what I needed... why are you so helpful
Perfect timing, I just started a project that will involve scraping 👌🏻
Fantastic video! I’m excited to test this out. Thank you for sharing this information.🙏🏾
This channel's videos are all so cool. You rock!
Ooohhhhwwww.. More Ideas about cloud functions please... 🔥❤️
As a programmer there is a lot of stuff that I kinda want to look in to but I never really get the chance to. These videos are helping me hard with that
This is some quality content yo! Thanks for this!
Very insightful. Appreciate all the work that's put into the videos
Never new about puppeteer. Excellent one
Interesting topic and it was also helpful seeing a firebase cloud function being set up.
Thanks, understood what I came for from the first two minutes.
I have been waiting for this video for so long thank you so much
scraping’s also useful for a cms allowing users to maintain easy html, but then transform page into a better ui
Cant you just do const urls = images.map(v => v.src) ? 09:00
Or is Array.from added for readability ?
That callback function returns a node list. So we are converting it into array.
Thank you for this video .I was looking for guidance for web scraping .This video prooved to be a great source.
That a cool detailed video in a very short time about scrapping...
Just getting into Node, and Json. Having a lot of fun! Definitely a challenge! 😎💻
Thanks Jeff this can be extended for ssr and seo 😁
Always such interesting and well produce videos. Thanks, Fireship.
Im using puppeteer in my work to make short videos testing the app automatically (feels awesome).
Also really cool video keep the good work !
All your videos are amazing!! Keep up the good work and we will do our best to support your channel. Cheers!! :-)
This video feels illegal...
I love it.
That puppeteer library looks neat, never knew about it
Thats pretty cool. Don't have a use case yet to use this yet but I can imagine how useful it could be. Great video as usual!
It is. Imagine you could scrape the entire net and let people browse your scraped pages using keywords. Oh wait...
That Tshirt Looks SICK!
Love your videos man, best quality!
you just got yourself a subscriber, great content my man
This is a really good video. Very clear explanation and simple
Your videos are just so good
Very interesting video. Learned so much from this video. Didn't know you can do such idea using Instagram.
A great video for adding a tool to our web Swiss Army knife.
I like this kinda demo so much!
Your tutorials are top quality !!
Thanks for your video, I have all of your videos. Loving it.
Good explanation with example, Jeff.
Very interesting video and very well presented. Thank you.
this is awesome man. Thank a lot!
"Awesome tutorial thank you. but for a non-tech user it is quiet hard to do a workable scraper for my WooCommerce store. As a side solution i am using eCommerce "ESCRAPER" maybe it helps somebody too.
But I am not giving up))) Thank you for your input!!!"
I really love watching your videos ❤️
Wish youtube had a heart react button also for such videos.
You could bypass the cors in the front-end using cors-anywhere
Great Video! Would love a tshirt!
This video is great! Just subscribed. Fantastic channel, amazing content. 👍
I've always used Cheerio for scraping. Seems to work pretty well.
Luke Pighetti are you able to scrape sites that use JavaScript to populate the DOM? I’ve always had issues scraping with it.
@@jaimegonzalez3956 Not with Cheerio, need something like Puppeteer for that
I don't get what was said about CORS: IMO The main reason why cross origin requests are blocked from reading inside the browser is because an attacker could otherwise easily gain access to private data of a user or read CSRF Tokens of remote sites (which rely on the SOP and CORS). Spectre has nothing to do with that and only became known to public in 2018. CORS on the other hand is in existence much longer and dates back to 2009.
As I understand CORB is different from that and specifically targets remote resources that are fetched by means of HTML elements (e.g. img element) as opposed to XHR or fetch requests which would probably be used to implement a Website preview for embeded links in the frontend.
^
Web scraping is a great thing to know, thanks!
Thanks for the insights!
Everytime I share some url in my LinkedIn posts..it loads the preview so beautifully...and I was always thinking how do they do that... U made it so easy.. now I can do it too... Thanks a ton
Exactly what i was looking for 👨💻
Thanks for the great video... I always wanted to do something in this domain...I was just shifting to python scrapy tool... Thanks again for awesome video..😃😃
I'll be having such a scrape time with this!
wish i had this vid a few years ago.
you just gained a new sub :)
Very interesting topic! Thank you!
Was looking forward to this, thanks a ton!
In some cases, poorly built web services leave out the internal API which can be found in client side JavaScript. So. its nice to do a hunt for that api.
Great Video, informative as always
Thanks so much for this video
Thanks for watching!
Amazing video! Love your content!
Thanks to you I got into Angular
Nice, loved it
For web-scraping the best approch for me is PHP + simplehtmldom class. Simplehtmldom have a nice sytax it's like jquery for scraper :-D
Another great video. Thank you.
Great Video! I love your videos. Keep it up bro!!
This is great! I want to try make like an app that uses face recognition to try find pictures of you where you were tagged or not, but I was too late to start using the API. This now gives me hope that it's possible
Kevin Darrent.. I have used puppeteer and did a meetup with it. Thanks!
Amazing video as usual!