Right now I'm learning Bootstrap from your udemy bootstrap course. Man, it's amazing! it's not just videos or slides, it has a very comprehensive code examples which accompanies whatever Brad says in the video. Brad did a hell of a job in this course! It's wonderful! I highly recommend it. I've not tried other brad's udemy courses, but if they are as half good as his bootstrap course, I'm sold! This man is a god among us! Love you brad, the great instructor and the awesome family man! God bless you and your family.
Let's scrape ... the scraping blog! I had a good laugh. Your courses are amazing and every now and then we get a good laugh. Keep up the excellent work.
I was just looking what a crawler is some hours ago. Now logging in to see this uploaded an hour ago! Are you reading the minds of your subscribers? :)
Bro ur the best, I hate my life but these vids help make it better. I do IT and dev because I like it and because I don't have anything/anyone else for me. Thenk u for helping me learn, Been into python and C# lately, as I re visit JS, it only strengthens my skills, after thinking in New paradigms
for code 'page = response.url.split('/')[-1]'. I thought it should be page = response.url.split('/')[-2] and it works for me. But I donot know why it works with 'page = response.url.split('/')[-1]' in the vedio.
great tutorial, but Im having trouble following along. filename='posts-%s.html' % page fails to number the pages so i just get post-.html and it overwrites itself for page 2, i assume. also tried filename = 'posts-{}.html'.format(page) with no joy.
@@bentraje I have the same issue, were you able to solve it? is it related to kite? EDIT: Found the problem, you need to replace this line OLD: "def parse(self, response):" with this one NEW: "def parse(self, response, **kwargs)"
@@TraversyMedia Looking forward to them both. Have a phone screen with Amazon coming up and was really worried about my lack of experience with vanilla stuff. What great timing!
Nice and I know you've said there's lots more you could do with this, but one obvious improvement you could make to this is to collect an array or a set of URLs as you go to ensure you don't crawl the same page more than once - as I think that's what this code might end up doing, as it is right now. Right?
This is great content @Brad, would it be possible for you to explain some basic topics of SEO, I feel as engineers we often lack those skills, right now I am going through the pain. Again very grateful for every piece of content you put out there.
Booyamakashi I hadn’t watched the video by the time I commented, I honestly would’ve loved it more if it was a JavaScript, I still like the video as long as brad made it.
I'm confused regarding why sometimes we specify the attribute name (div) and sometimes we don't while selecting by class. For example: 13:54: No 'div' keyword 18:16: There is 'div' keyword
The o ly difference is that when we don't select with div, all elements with class will be selected, when we write with div, we gonna select the elements with class, that are divs only. With sayi g with div, you're more specified.
so weird that i just started looking at scrapy this morning and boom... this vid drops. question - i cant seem to get vs studio to launch the debugger for a scrapy file. any secrets? thx
Love your tutorial man. Thank you. With scrapy can we scrape millions of data with sequenced/scheduled interval to not get blacklisted and keep updating out file?
Can you please explain why you used yield on lines 13 and 21 for final version of code? Does this mean parse is generator function in this case? How does this work under the hood?
I cant seem to make this code work on Python IDLE. upto 22:24 and it gives me the output on scrapy shell but cant make it work in Python IDLE 3.8.2 please help.
I may be mistaken but I believe there is already a default method named "parse" that is overwritten here. Nothing wrong with overwriting it but it could cause unexpected behavior for someone that doesn't know.
Can Someone Help me I am getting two exception errors while putting the command Scrapy crawl posts: 1)KeyError :posts 2)Spider not Found in Posts Thank you in Advance! (Any Help Appreciated)
@@rishabhkothari1763 sounds like you are making it search the wrong place. are you sure you are set rightfully up with your virtual environment? try going to 'debug configuration' and change the 'source path', e.g. make the last path equal to PostsSpider.py. then it should be able to find the spider. Hope it helps :)
I have 2 dozens of sites with jobs in europe. I would like to crawl and scrapp several data sets from it. Is there a way to do this in a generic matter to get it all at once ?
Hello! I want to do some web scraping but to find info on a certain thing. So normally, I would use a search engine to find the urls then from there, find the data I need. How would I automate the process of obtaining the URLS? The websites are pretty much the same ( I only really end up using 4 or 5 five websites with the data being a specific spot on the site). I would really appreciate any suggestions! Web scraping is such a good tool, but I need to automate the URL gathering process to accompany the Web scraping
it is used to tell the command "scrapy crawl posts" where to get the data from. It's like the variable name that you don't used in the code but you use it in the command line.
Quick question, using xpath insead of css when generating with yeild creates in json file differently, I mean it puts all the titles first, then the dates and so on. It's a there a different sintax that I need to use?
I did a little web scraping a while back-- this video is very timely because I was going to get back to it!! I needed a refresher, thank you!!!
@@Tolrias Good to know, thank you!!
@@Tolrias thanks. I wanted to know this. Also could you link me to python scraping with headless chrome tutorial? A blog is also fine
Awesome video. I never thought I'd learn this much in 30 mins. Every second of video is full of useful information. Thank you so much
Right now I'm learning Bootstrap from your udemy bootstrap course. Man, it's amazing! it's not just videos or slides, it has a very comprehensive code examples which accompanies whatever Brad says in the video. Brad did a hell of a job in this course! It's wonderful! I highly recommend it. I've not tried other brad's udemy courses, but if they are as half good as his bootstrap course, I'm sold! This man is a god among us! Love you brad, the great instructor and the awesome family man! God bless you and your family.
Thank you so much brad. I purchased the Django course on udemy. Awesome content. Congratulations. U will soon reach 1M subscribers. Wow.
Let's scrape ... the scraping blog! I had a good laugh. Your courses are amazing and every now and then we get a good laugh. Keep up the excellent work.
man after watching this video and executing this video in just one morning I managed to crawl an entire website in seconds. Thank you!!!!
I was just looking what a crawler is some hours ago.
Now logging in to see this uploaded an hour ago!
Are you reading the minds of your subscribers? :)
YEAH, ME TOO. What a coincidence!
A great content creator that knows what people want. Brads a legend :)
No no, he's projecting his thoughts into your mind.
abj freakin brad get out of our heads lol.
😊 maybe, i do hear that a lot
I was following along with a different site I needed to scap all made sense still always grateful Brad
I have use scrapy for many web crawling and web scraping projects. However, I still found this tutorial very handy.
This video couldn't have come at a better time... Thanks a bunch Brad... God bless
Thank you for the instructions, I like how the last minutes made things clear for me...
great lesson. After doing some webscraping with selenium, this finally made a lot of sense because I was lost a month ago
It's a little out of my league since I am only a beginner coder but it was utterly fascinating! Thank you very much!
thank you so much. My first time with Scrapy and you've been really clear. Great video. Tranks mate :)
Your videos always great.
A lot of other coding vids built on python talk about simple math for 8 hours and I learn nothing.
Bro ur the best, I hate my life but these vids help make it better.
I do IT and dev because I like it
and because I don't have anything/anyone else for me.
Thenk u for helping me learn,
Been into python and C# lately, as I re visit JS, it only strengthens my skills, after thinking in New paradigms
Love your series! Thank you always!
Ridiculously awesome video! Def an amazing teaching and great start to web scraping with scrapy. Dope Stuff!
thanks for your comprehensive description. i think this is good as start point.
Thank you very much for this tutorial! It's nice, short and crisp!
Such an awesome tutorial, sir!
for code 'page = response.url.split('/')[-1]'. I thought it should be page = response.url.split('/')[-2] and it works for me. But I donot know why it works with 'page = response.url.split('/')[-1]' in the vedio.
Great video, both compressive and concise!
Like => Add to Watch later => Thanks, Brad. :)
Great tutorial, the copy XPath from the browser was very handy
I see Brad's Video I click it even though I don't know what's going on :P . Like it anyways.
MeGaZ haha, thanks I appreciate that ❤️
Excellent tutorial, as usual. Kudos!
Very good, please keep doing this tutorial series :)
Glad to be here
I dont search for this but i am kind of like to watch this so thank you
This is sooooo cool! Thanks a lot Brad!
This is a great Tutorial for crawling data
great tutorial, but Im having trouble following along. filename='posts-%s.html' % page fails to number the pages so i just get post-.html and it overwrites itself for page 2, i assume. also tried filename = 'posts-{}.html'.format(page) with no joy.
I have the same issue. Did you managed to solve it?
@@bentraje I have the same issue, were you able to solve it? is it related to kite?
EDIT:
Found the problem, you need to replace this line
OLD: "def parse(self, response):" with this one
NEW: "def parse(self, response, **kwargs)"
@@AdamEfrati ah gotcha. didn't solve it. thanks for the reply!
Very clear explanations :-) Thanks a lot !
very useful video super educative and clear
Great Video simple explanation Thank you
You are literally the best
In 8:04, why not use an f-string instead of the old percent sign way?
Awesome! Thank you Brad!
thx for this one, helped me a lot
Hello brad ! Please could you tell me when would yould you share the front end course for the devBootcamp backend on udemy?
After my next course (20 vanilla Projects) which will be released within 25 days or so. I will start working on it
@@TraversyMedia Looking forward to them both. Have a phone screen with Amazon coming up and was really worried about my lack of experience with vanilla stuff. What great timing!
Thank you for sharing the knowledge.
Your video is best, thank you help me a lot!
how are you know my thought. i looking for web scrapper and you make a tutorial with this ? are you an alien brad
Hey brad m still waiting for your new vanilla javascript course can you tell us when it will available in udemy???
Nice and I know you've said there's lots more you could do with this, but one obvious improvement you could make to this is to collect an array or a set of URLs as you go to ensure you don't crawl the same page more than once - as I think that's what this code might end up doing, as it is right now. Right?
you're really doing good job ... keep it up buddy... joey says
I love me some scraping but I did it with puppeteer and something else for work. My custom API did get blocked a few months later though...
This is great content @Brad, would it be possible for you to explain some basic topics of SEO, I feel as engineers we often lack those skills, right now I am going through the pain. Again very grateful for every piece of content you put out there.
Scrapping with js, just in the perfect time for me
Thank you brad
You clearly watched the video if you think its in js.
Booyamakashi I hadn’t watched the video by the time I commented, I honestly would’ve loved it more if it was a JavaScript, I still like the video as long as brad made it.
scrapy is python lib lol
neesyler you’re right, scrapy and beautiful soup are python, puppter is JS
Hey man, pls do a course on setting up a bespoke MVC system from scratch with express server , node etc.. going over the MVC fundamentals etc.
What about pages that are secured with middleware can you scrap them aswell?
I haven't found any better videos for data structure & algorithm. If you know something please make a vid about it.
Wow! It would be great if you make JIRA course and Agile development, Love all your courses here in udemy, keep going sir.
sujal khatiwada you don’t need a course in Jira. If you don’t know it you are in some ways lucky.
@@nathanlewis42 but why, JIRA is used in industry
thank you
one question
why is necessary to create a virtual env?
I'm confused regarding why sometimes we specify the attribute name (div) and sometimes we don't while selecting by class. For example:
13:54: No 'div' keyword
18:16: There is 'div' keyword
The o ly difference is that when we don't select with div, all elements with class will be selected, when we write with div, we gonna select the elements with class, that are divs only. With sayi g with div, you're more specified.
@@nowieszco868 oh oh I see. Thank you for explaining :)
so weird that i just started looking at scrapy this morning and boom... this vid drops. question - i cant seem to get vs studio to launch the debugger for a scrapy file. any secrets? thx
have you tried turning it off and on again?
Love your tutorial man. Thank you. With scrapy can we scrape millions of data with sequenced/scheduled interval to not get blacklisted and keep updating out file?
I'm getting an Unknown command: crawl error which is at 8:49 into the video. I can;t seem to find the error.
Any help here?
wish you did a whole series on this
Can we also take the user input for the url to scrape in scrapy?
Next video: How to overcome captcha with Scrapy :)
Can you please explain why you used yield on lines 13 and 21 for final version of code? Does this mean parse is generator function in this case? How does this work under the hood?
I cant seem to make this code work on Python IDLE. upto 22:24 and it gives me the output on scrapy shell but cant make it work in Python IDLE 3.8.2 please help.
Hi Brad, thanks for the video. Is Scrapy also able to handle SPAs and specifically with content that is dynamically generated with Javascript?
Very helpful, thank you.
Hi,
Please advice me on how to improve / speed up the scrapy process
I'm stuck at 24:50. I run the program and no data returns, and no errors, either.
I may be mistaken but I believe there is already a default method named "parse" that is overwritten here. Nothing wrong with overwriting it but it could cause unexpected behavior for someone that doesn't know.
Can Someone Help me I am getting two exception errors while putting the command Scrapy crawl posts:
1)KeyError :posts
2)Spider not Found in Posts
Thank you in Advance! (Any Help Appreciated)
Sorry I have the same problem
@@Vincent.Esders Hey! If you get any Solution, please notify me here in the comment box. Would Really Appreciate the help!
@@rishabhkothari1763 sounds like you are making it search the wrong place. are you sure you are set rightfully up with your virtual environment? try going to 'debug configuration' and change the 'source path', e.g. make the last path equal to PostsSpider.py. then it should be able to find the spider. Hope it helps :)
4:38 When I type 'import scrapy' I get the message 'unresolved import 'scrapy'Python(unresolved-import)' I am using vscode
This solved the issue:
www.reddit.com/r/learnpython/comments/a97p09/unresolved_import_warning_vscode/
Is it possible to code this normally like in pycharm or sublime without using a virtual environment?
I think PyCharm automatically creates a venv for your projects..
@@aarongonzales3765 whenever i try to code this in pycharm i run into issues
@@trinimafia001 Something like package not found? If so, that is easy to fix.
Can you make a scrapping tutorial in js? There, maybe, so many persons who are looking for web scrapping tutorials in javascript.
I have 2 dozens of sites with jobs in europe. I would like to crawl and scrapp several data sets from it. Is there a way to do this in a generic matter to get it all at once ?
How about if I have multiple keywords, for instance, “123”, “apple”, orange” or even with date time, can I use these before crawling it?
Lovely project!!
Hello! I want to do some web scraping but to find info on a certain thing. So normally, I would use a search engine to find the urls then from there, find the data I need. How would I automate the process of obtaining the URLS? The websites are pretty much the same ( I only really end up using 4 or 5 five websites with the data being a specific spot on the site).
I would really appreciate any suggestions! Web scraping is such a good tool, but I need to automate the URL gathering process to accompany the Web scraping
Is there any upcoming course for Vue with TypeScript?
ruclips.net/video/TGW-z1bIWyg/видео.html
ruclips.net/video/Ww57lUS9dF4/видео.html
Nice video man !
which extension do you use to see scrapy help in vscode ?
it's called "Kite"
Sir I have watched the while series but I got one question how to bypass 423 status code as the user agent and proxy pool isn't working
This is great. Any plans for a Python video that calls an external API and fills models?
do u have its course ?? or playlist where are other scrapy videos
you're the man!!
Can you do a video about *unit testing* ? Please
Great video
8:41 I don't understand how that works. He declared a starts_url array and then doesn't use it?
it is used to tell the command "scrapy crawl posts" where to get the data from. It's like the variable name that you don't used in the code but you use it in the command line.
So it ends the text as it bumps into an apostrophe in regex? Congrats to the almost 1m subscribers!
Quick question, using xpath insead of css when generating with yeild creates in json file differently, I mean it puts all the titles first, then the dates and so on. It's a there a different sintax that I need to use?
For me yield doesn't do the same when is generating, is just putting all the text under a single tag for each section.
Hi, Is it a vscode extension to see a document at 4:57 How can I use that of 'docs'?
I found the answer myself;marketplace.visualstudio.com/items?itemName=kiteco.kite
cant we create a spider using genspider? or we need to do it manually.
I want to do scrape using scrapy in jupyter lab. How can i do that?
i am getting error 'str' object has no attribute 'css' 19:00
Hello Brad, When 20 vanilla Projects course will release.. Waiting for that.
To be safe, I will say within a month. Most likely sooner though
My VS doesn't show any of the scrapy folders. But when I cd into my project name and do Tree it shows the folders.
genius my friend.
In vs code how do you execute the python code in the terminal. Like when he starts the for loop?
Thank you.. Great video.
Nice video!
Hah I was just watching Scrappy Coco.
What is the setup on your developer tools on chrome?
Also tried a scraping with a node app. I don't know why but the performance was really different from this Scrapy.