🎯 Key Takeaways for quick navigation: 00:00 🌐 Introduction to web scraping with ChatGPT Code Interpreter 01:07 📂 Saving a webpage as an HTML file 03:38 🧩 Extracting data from HTML using ChatGPT 04:08 📄 Exporting extracted data to CSV 05:03 🔁 Repeating the process for multiple pages 06:51 🌐 Scraping data from Glassdoor 08:13 🧩 Scraping specific data elements from Glassdoor 11:56 📄 Exporting Glassdoor data to CSV 12:37 🚀 Conclusion and call to action Made with HARPA AI
You can tell Chat GPT code interpreter to do the same thing for any amazon page, just upload the html and run the same process. Instant data scraper is clunky and the interface looks old, I'm sick of clicking an dragging with all these data scrapping app extensions. Also certain websites don't work well with data scraper apps to many blockers. try extracting data from harbor freight with instant data scraper. its a nightmare! no thank you.
Is it not possible to provide the url and then get it to loop through the pages till it gets to the last page and then get it to export the results to csv
@@anuroop345 I tried this but it’s not as easy, Chatgpt keeps having trouble with the looping side, plus a few websites require a login to access their site
This was pretty good, but I notice that none of these GPT methods for data extraction deal with pagination. How would we do this if we've got multiple pages of jobs on Glassdoor and you wanted to let chatGPT run through all of them?
I've had a huge increase in scrapers across my entire domain portfolio. Given the power of AI, I've managed to poison most of the scraped data and cause major headaches going forward.
But not the automation I am looking for - will keep up with the options choose and one click. Maybe a long way to complete it but will do everything I need. Really python and is with some chat chatbot content writing - might use ChatGPT - but the content never passes ai detection like clause on it's first time.
When I checked the code I could see it's using Beautiful Soup. I'm not quite sure what would happen if you specify to use Selenium. Just remember that code interpreter is mostly parsing the HTML file
Hi Frank ty for the great walkthrough! I am trying to scrape google to deterniine companies near me that do not have a website. it appears to be a simple task to determine on the search results which ones to choose but for some reason gpt says the html structure of nesting elements is not allowing it to do so. Any ideas?
I love your contents especially this one.. Sir how can we offer this as a service. Who would need this service ? Can you help out maybe with a resources thank your
@@ThePyCoachNo i didn’t. I just uploaded direct link with scraper plugin on. And prompted the code interpreter to generate a new programming language that is more advanced.
![Felix "Unfair" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/Oswujxm2Ag0/mqdefault.jpg)
Download my FREE ChatGPT Cheat Sheet: frankandrade.ck.page/08c94cf1c1
It will be nice if we can apply this approach on Facebook posts and websites?
HI , please can we still assess the the free chatGPT cheat sheet. please do share again
Mind-blowing stuff! Wonder if combining it with Proxy-Store's proxies would level up the scraping game even more. Any ChatGPT fans here?
A terrific video that has been the cornerstone for our new team members. Thank you for sharing it!
Great stuff @ThePyCoach! Question now is, how do we automate this scraping for multi-pages instead of manually saving each page as a new html file?
Very practical and helpful video with very detailed explanation!
Hey PyCoach, learning a lot from you. Thanks man!
You're welcome!!
Top tier content bro. Easy sub.
how would you do it for the products page. So the tv plus the product pages details of the tv then so on.
Thank you so much The PyCoach!
🎯 Key Takeaways for quick navigation:
00:00 🌐 Introduction to web scraping with ChatGPT Code Interpreter
01:07 📂 Saving a webpage as an HTML file
03:38 🧩 Extracting data from HTML using ChatGPT
04:08 📄 Exporting extracted data to CSV
05:03 🔁 Repeating the process for multiple pages
06:51 🌐 Scraping data from Glassdoor
08:13 🧩 Scraping specific data elements from Glassdoor
11:56 📄 Exporting Glassdoor data to CSV
12:37 🚀 Conclusion and call to action
Made with HARPA AI
Cool process. But what is the advantage of doing this process, instead of using pre-built scrapers already available like "Instant Data Scraper".
You can tell Chat GPT code interpreter to do the same thing for any amazon page, just upload the html and run the same process. Instant data scraper is clunky and the interface looks old, I'm sick of clicking an dragging with all these data scrapping app extensions. Also certain websites don't work well with data scraper apps to many blockers. try extracting data from harbor freight with instant data scraper. its a nightmare! no thank you.
Can you make a video like this that's scrapes the SEO keywords that a website is using?
You are the best, subscribed :)
is it same work as like other language website like Hindi , Japanese , Chinese and many more language ?
the interpreter code does not appear in the beta feautures to activate it just i have only plugin why? do you have a solution please
Thanks, that's really helpful advice - and indeed pretty mind-blowing!
thank you!!! never thought about the save page as html and then uploading
this is cool ! although It is probably not able to scrape any IMG/Links ?
Awesome!!!! and extremely useful!
Hi, is there any way now to scrape all posts and comments of a subreddit? I need them for my research.
Thank you very helpful.
Try saving all the pages at once to reduce manual repetitive work
I guess the manual download part is intended for demonstration. You'd do this programmatically.
can this be used to scrape multiple pages like 100 pages on a domain? i guess so, because one page would be silly
This is really cool❤
very nice! thumbs up.
Is it not possible to provide the url and then get it to loop through the pages till it gets to the last page and then get it to export the results to csv
If you pass the URL, ChatGPT will probably tell you that it's not able to perform web scraping or that is against the ToS
you can ask chatgpt for python script to scrap that page. and then you can run that script on your computer to loop and scrap.
@@ThePyCoach I would have thought someone would have made a plug-in for this now
@@anuroop345 I tried this but it’s not as easy, Chatgpt keeps having trouble with the looping side, plus a few websites require a login to access their site
This was pretty good, but I notice that none of these GPT methods for data extraction deal with pagination. How would we do this if we've got multiple pages of jobs on Glassdoor and you wanted to let chatGPT run through all of them?
Thank you for pointing this out before I went too far in this video, as I am trying to scrape dynamic content loaded with JS
Thanks for the video! I tried both methods to scrape Udemy and it didn't work, since the content is in a container. Do you have a solution for this?
code interpreter isnt even an option was it removed? Maybe explain how to get it thatd be helpful
listening to this video gives me "Two Minute Papers" vibes. I was waiting for ""Alright everyone, hold onto your papers"
what a time to be alive
I've had a huge increase in scrapers across my entire domain portfolio.
Given the power of AI, I've managed to poison most of the scraped data and cause major headaches going forward.
What do you mean?
@@tryderrick upon detection, I feed a load of nonsense back in between legit data, and it totally ruins the dataset.
@@ronin4518 depends what you're scraping. Not my data
So how do I scrape data from the product pages as well?
Just upload its HTML file and follow the same steps
So basically you made a tutorial about saving a file and loading it into chatgpt?
But not the automation I am looking for - will keep up with the options choose and one click. Maybe a long way to complete it but will do everything I need. Really python and is with some chat chatbot content writing - might use ChatGPT - but the content never passes ai detection like clause on it's first time.
You really have to do the most part of the writing for SEO, based on the 'draft' or outline of the content.
Did they remove the Code Interpreter plugin? How do you get it? I have a paid account. Thanks
Try enabling it in your settings. Open Settings > Click on "Beta Features" > Press "Code interpreter" toggle button
Hi, is Code Interpreter also available through the API?
Hi. I'm not quite sure
Why would I do this? What is the use case for this
But it appears this only works with the paid version of chatGBT
Thanks, but what if we ask it to use link directly with help of beautiful soup and selenium? What would happen
When I checked the code I could see it's using Beautiful Soup. I'm not quite sure what would happen if you specify to use Selenium. Just remember that code interpreter is mostly parsing the HTML file
Could have also just Did View Source, then copy paste, or maybe even use plugins with link reader and paste that viewed source URL in.
View source won’t work if the page is rendered dynamically
true that@@egauci1
If you used harpa or perplexity it will get the job done with no complications 😅
Hi Frank ty for the great walkthrough! I am trying to scrape google to deterniine companies near me that do not have a website. it appears to be a simple task to determine on the search results which ones to choose but for some reason gpt says the html structure of nesting elements is not allowing it to do so. Any ideas?
So have you done it?
Just make a list of all the businesses near you. Then identify among those which could benefit from having a website. BOOMSHAKALAKA!
I love your contents especially this one..
Sir how can we offer this as a service. Who would need this service ?
Can you help out maybe with a resources thank your
we need to all this thing for scrap? And scrap for what?
I used it to scrap some websites. Last week my account was banned, said my activity was suspicions.
How could that happen? Did you upload too many HTML files?
@@ThePyCoachNo i didn’t. I just uploaded direct link with scraper plugin on. And prompted the code interpreter to generate a new programming language that is more advanced.
that is why you need to use langchain and make your own code agent.
Also, only use interpreter to write code once, then handle it yourself.
@@curious_one1156how
Good. We don't like dodgy scrapers.
I don't wanna be that guy, but you can do this with a free tool called Harpa in a split of a second, it is nice to see the procedure but... 🤷♀
cool
What is the purpose of doing this? What was the problem that makes your work its solution?
This scraping is very manual. There are much better ways.
Next version of code interpreter
Could you please share them
@@izzyal-zyoud7563 webpilot plugin, can scrape any number of pages with one prompt and by just providing the page 1 link
@@mishos.2228 Won't that get you banned anytime soon using just links?
Yea this aint web scraping. this is html extracting data, where is the web scraping at?? you are doing it manual…
This was actually slower than classic python scraping.
Hey would you be able to assist in python scraping script and looping the scape for another search ?
this video should have been a short.
What will happen if you want to scrape 1 million pages? 😀
1 million page downloads, and 1 million ChatGPT uploads.
This is more of a chatgpt-aided manual scraping. The processes aren't automated.
I really don’t like this title. It feels dishonest. There’s are much better ways to use chatgpt to scrape
If you know web scraping this is really nothing :D
Dude 😂
Watching your video made me angry
Not too useful. Stick to node puppeteer or perl/python command line scripts.
Bro how to train a own AI 😅 and make like chatgpt
I think you need to use Langchain for that
@@ThePyCoach you can create a chatbot using Langchain how to make a chatgpt using Langchain
Clickbait 😅
Please remove this video, you wasted my 15 minutes 😢