Thank you John for the continuous value added, I would advise to debug json data in debug mode to be presented hierarchically and to be clear for the audience to see the structure of it.
Hey John! Great video as always. Do you have a video on scraping aspx websites and submitting form data? The table data I want does not appear in json and I can't find it anywhere except document.aspx
I just found your channel but have 1 question. Is it possible to scarp the price of for example a pen from amazon and compare that to the same item on ebay? If yes, would it also be possible to add filters for example (Shipping cost < 5$. and everything above 6$ will be ignored from for example amazon with 4 sites with the hundreds of different listings of that pen)? WOuld that be with the if then true command?
very good indeed sir. I'm sure that asking is a long shot -but, can you do the same video using the actual API? -it is quite different. Also (from a non expert view) why use functions inside a class? -the functions are not really dynamic and without the class you would simplify the whole thing considerably for one that is starting to learn. Then, the juicy stuff is around the sales by product...
Either view source and search for the work shopify or type products.json at the end. When you see enough shopify stores you learn the templates and how they look too
Great video. Is the method still valid? Please update your links here! And point out where to find the code. Github does not seem to show it (any longer?). Thanks!
Curious. How do you go about finding access to these "hidden json-data" on a certain website? I mean here you just wrote "products.json" after the base url, but that is not going to work on any website, I believe. So possible you may tell a little about your search methods for finding json data behind a certain website -what to look for and what makes you suspect easy access json data? Like in your example. Thanks.
I work with Shopify every day! So this one I just found out by being around the platform a lot. Once you realise these things exist too you can search the relevant user forums etc to see what’s available
Thanks for this really helpful and greatly explained project, John! Coincidentally, just last week I developed a code with the Shopify API for a client to get their orders and build a sales dashboard in Data Studio. Two questions: - Should I always use classes in my code? - I'm using Google Sheets as the source for the Data Studio dashboard, but I'd like to use a SQL database instead and run my code periodically and automatically in the cloud, so it doesn't depend on me. Do you have any suggestions on how to do that? Thanks!
Hey! Sounds like a cool project. Yes I would always use classes where necessary- if it makes sense as it will help you keep your code clean and readable. As for the database, I think you will definitely need to use one for that project, you can run your scraper on a crown to add to the dab then have your app pull from it
I'm stuck at printing the data it says moduleNotFoundError: no module named 'dataset' and i tired to install it and nada still getting the error, I'm on python 3.8.5
The totals is not calculating the sum of products, it is still showing 9. Also, the results is only getting the first item from each page. Can you help please?
Hi I Love the content and Thanks! for making Videos on scraping Can you make video on scraper which has GUI interface and hosted on server so simple visit the link and run it and check the scrape data on GUI interface from database In simple words web interface where we can extract data and see the scrape data.
At the end you have the totals list, with it you can iterarte through each item and check for your searchpattern on for example the title or id or whatever. Is it the Fight for your purposes, add this Special Item to your "Input List" and finally add thus Input List to the DB
Dam did you get a new haircut? Looks sharp
Great video John!
More videos where you combine Python and SQL would be awesome. Keep ut the great work!
Hi John, Thank you allot for this channel.
learnt allot especially for my data sourcing projects
John, I've just discovered your channel and I'm amazed by it! Your explanations are top notch. Thanks a million for your efforts and transparency
Thank you I appreciate the kind words!
can you do a live data web scraping end-to-end project?
Thank you John for the continuous value added, I would advise to debug json data in debug mode to be presented hierarchically and to be clear for the audience to see the structure of it.
Thanks yes I think that would have been better too
Great video, it helped me a lot, thank you for your work 👍
The 🐐!!! Thank you so much for this video
Ohhh gonna watch this one tomorrow. Keep them coming John! :D
Thanks Kyo will do!
Thanks. Would be great to see a video with bulk inserts/merges and transactions. What was the db browser at end please?
Hey John! Great video as always. Do you have a video on scraping aspx websites and submitting form data? The table data I want does not appear in json and I can't find it anywhere except document.aspx
Really cool video! Keep it going...love your content!
Thanks!
Thank you for your amazing videos. Would you concider making a video about Scrapy-Playwright ? Just a thought
Sure! I keep meaning to look into playwright but never have- I’ll add it to my list
@@JohnWatsonRooney 😍
Fantastic video!!!!! Fantastic channel !!! Keep it up !!!!
Thank you!
Fantastic video John. Please, how do I open my SQLite on the browers
I use DB Browser which is a free windows app that you can open and view/edit sql databases with
I just found your channel but have 1 question.
Is it possible to scarp the price of for example a pen from amazon and compare that to the same item on ebay? If yes, would it also be possible to add filters for example (Shipping cost < 5$. and everything above 6$ will be ignored from for example amazon with 4 sites with the hundreds of different listings of that pen)? WOuld that be with the if then true command?
very good indeed sir. I'm sure that asking is a long shot -but, can you do the same video using the actual API? -it is quite different.
Also (from a non expert view) why use functions inside a class? -the functions are not really dynamic and without the class you would simplify the whole thing considerably for one that is starting to learn.
Then, the juicy stuff is around the sales by product...
Hi Eric, do you mean as if you had admin access to the actual stores API?
@@JohnWatsonRooney that’s correct. Yes. Thanks!
Hi John Great video again. 1 question: how do you get to the SQLite in the video?
Really, really amazing!
Thanks!
John do you know of budget friendly ways of adding an efficient proxy??
Great, keep going on..
How to know if this store is Shopify or not??
Either view source and search for the work shopify or type products.json at the end. When you see enough shopify stores you learn the templates and how they look too
Hi John! I'm following your channel. It's really very helpful to improve scrape website skills. Let me know a better way to scrape bet365 odds.
Great video. Is the method still valid?
Please update your links here! And point out where to find the code. Github does not seem to show it (any longer?).
Thanks!
Curious. How do you go about finding access to these "hidden json-data" on a certain website? I mean here you just wrote "products.json" after the base url, but that is not going to work on any website, I believe. So possible you may tell a little about your search methods for finding json data behind a certain website -what to look for and what makes you suspect easy access json data? Like in your example. Thanks.
I work with Shopify every day! So this one I just found out by being around the platform a lot. Once you realise these things exist too you can search the relevant user forums etc to see what’s available
A little detective work i guess... 😉. No real shortcut. Thanks.
Thanks for this really helpful and greatly explained project, John!
Coincidentally, just last week I developed a code with the Shopify API for a client to get their orders and build a sales dashboard in Data Studio.
Two questions:
- Should I always use classes in my code?
- I'm using Google Sheets as the source for the Data Studio dashboard, but I'd like to use a SQL database instead and run my code periodically and automatically in the cloud, so it doesn't depend on me. Do you have any suggestions on how to do that?
Thanks!
Hey! Sounds like a cool project. Yes I would always use classes where necessary- if it makes sense as it will help you keep your code clean and readable. As for the database, I think you will definitely need to use one for that project, you can run your scraper on a crown to add to the dab then have your app pull from it
@@JohnWatsonRooney awesome, thanks for the tips, John! Looking forward for more of your content!
hey @aaakel, I am working on very similar projects now, let's catch up if you are still in that area.
how would i go about accessing the info from the products.db to maybe display to a discord bot? i.e checking if item is available or not
I'm stuck at printing the data it says moduleNotFoundError: no module named 'dataset' and i tired to install it and nada still getting the error, I'm on python 3.8.5
code link
I keep getting "line 63, in products = main() NameError name 'main' is not defined" Any thoughts?
Where did you even find the products.db file?
The totals is not calculating the sum of products, it is still showing 9. Also, the results is only getting the first item from each page. Can you help please?
A web app that visualizes the scraped data?
help
Import "dataset" could not be resolved
lol, have been also doing python from sublime to vsc
thx for this nice video, where can we download the source code from?
super!
Hi I Love the content and Thanks! for making Videos on scraping
Can you make video on scraper which has GUI interface and hosted on server
so simple visit the link and run it and check the scrape data on GUI interface from database
In simple words web interface where we can extract data and see the scrape data.
I’ve been thinking about something like this for a while - I have some ideas but it could be a big project!
is there a way to take only items that include what keyword did i choose not all of them
At the end you have the totals list, with it you can iterarte through each item and check for your searchpattern on for example the title or id or whatever. Is it the Fight for your purposes, add this Special Item to your "Input List" and finally add thus Input List to the DB