Just started learning Python last week. Stumbled onto your channel today and it’s a breath of fresh air to provide some excitement halfway through my coursework! You explain everything in easy terms and it’s a lot more exciting than basic programs. I’m looking forward to finishing my course now!!!
I really like the fact that you explain things so clearly, and I regret the fact that I haven't been able to see your videos in the past few months Great work!
hey mate, i recently found your channel when looking for some projects to busy myself with in lockdown and i love how simple you explain things and i hope you get many more subs because your amazing 😀
@@MakeDataUseful 🙂 Also, if you need a video idea Id love to see a notepad / MS word kind of thing in python. It doesn't have to have a GUI but I've been trying to make one and I'd love to see how someone else would try this. I'm using as few external librarys as possible so it could be a kind of plugandplay solution for quick edits?
Many thanks to you! I found an Upwork job after watching your tutorial and creating portfolio project. Keep it up, your tutorial are indeed practical. Thanks agian!
Hey there, absolutely loving this series. It is quite hard for new comers to get their first job on upwork, if you had any tips regarding how to land the first job, smooth and easy it would be a real help ! Anyways, keep the series going. Loving these videos
Thank you for the feedback and yes totally agree about landing those first few jobs! I will do some more client communication and landing the job in upcoming videos :)
This is the second tutorial I am following with you and I like this one way better than the first (the website in the 1st one didn't work), it s good to see the process of webscraping working, thank you!
I was able to accomplish the first video, if you need help ask away. The xml sitemaps page changed in this video's (#2) and now has images instead of the text in 4:38 , but I don't think it is important, but the speed at which this site has changed over 3 years is amazing.
masterful tutorial dude by far the best explanations of everything even though im still trying to understand some for loops and exception handling also what packages to use but with practice i know i can do it i just stumbled on your videos so i hope i can start learning to do jobs like this so i can do this on the side
@@MakeDataUseful wow... Really excited... U know it's really useful... I was struck at a point thinking what will i do after learning python. How will i convert my skills to a profession... Now this tutorials really helped me... I am thinking of learning web scraping with python and if u teach any webscraping courses with python.. please let us know.... Will be happy to be your student...
when you do work with python yourself do you usually only target web scrapping or do you other jobs as well. by the way i really like web scrapping with some practice i think i can become good at this
I do a lot of web scraping when I am building my own databases/APIs. A lot of my work is data cleaning and remediation prep for the visualization of data. The viz component is usually 10-20% of the effort, most of it is extracting and transforming data. Keep practicing an let me know if you need any help!
Thanks for the video. I have a python experience as a backend developer for 1 year. I have used python flask framework. Due to covid ,the start i was working has stopped its operation and i lost my job. Can i know how can i start earning as a freelancer with the python skills.
There are several no-code/low-code tools available that allow you to perform machine learning tasks on images without having to write code. I imagine they may be utilising one of those. Also might be easier to give the task to someone else to do to save time and focus on other aspects of their project
I tried to follow along but couldnt get the xml to work. FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml. Do you need to install a parser library?
In Aug 2023 the page format is changed and BeautifulSoup has changed too. Fixes that worked for me: 1) install lxml (letter L in front) with pip install lxml 2) Then use this at about line 20: soup = BeautifulSoup(response.text, features="xml") The web page structure has changed so that the id number is on one line and the PNG on the line following it. I used the following code (may not be the best way but it works): for image_url in master_list[0:5]: # get a sample of some images. (requires two lines per image) this 0:5 downloads 2 images if 'image' in image_url: # the id number line has the word 'image' in it id_number = image_url.split('/')[-1] # get the last string following the backslash print (id_number) elif 'files' in image_url: # the png line follows the id line and has the word 'files' in it print(image_url) image = requests.get(image_url) # download the PNG image image_title = image_url.split('/')[-1] # get the file name in the last string after the final backslash file_name = id_number + "-" + image_title # add the id number + image name together # write the PNG file # example: 755899-Poker-Chip-PNG-Photos-80x80.png with open(file_name, 'wb') as file: file.write(image.content)
Did RUclips suggested me this video because it has Eren Jeager's images in this video or is it because i am learning python lol. every where i go my favorite anime follows me.
In 8:29 and 14:04 you defined "soup" and "loc" on different lines. Does that mean we can use the same name for variables? Sorry im still new to python. Also i noticed you mispelled "response" as "repsonse" in 11:50, was that intentional? Thank you for a real practical use of python though :) Edit: nevermind, you said its okay for repeating the "soup", my bad :)
Python is both a strongly typed and a dynamically typed language. Strong typing means that variables do have a type and that the type matters when performing operations on a variable. Dynamic typing means that the type of the variable is determined only during runtime. So yes, you can use the same variable
Please create a video in which you interact with the employer? I'm looking for my first job at the platform, but I'm not sure how to proceed when I finally get "selected" and have to interact with the employer. Thanks!
@@MakeDataUseful HTTPConnectionPool(host='www.pngmart.com', port=80): Max retries exceeded with url: /image/246105 (Caused by NewConnectionError(': Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))
@@rahalmehdiabdelaziz8121 looks like a timeout, a couple of things to test. Try visiting the url with your browser and I'd you are successful they connection may be getting refused due to incorrect headers. Another option you could try is adding timeout=X where X is the number of seconds you want to wait. More details available here requests.readthedocs.io/en/master/user/quickstart/
@@MakeDataUseful I tried those type of solution but there is no efficient result ... however I found a " SOLUTION" : the error doesn't occur everytime (I suspected the weaknesss of my internet connection) so I've done a WHILE loop to repeat the request until it works ... thank you very much for your response ... I wish you do series of real world tasks other than webscraping ... Thank you and good luck
Hey Art, Thanks for the feedback! At around 20:10 I source the image URL then use a response = request.get(image_url) to source the image then use a with open... and access the response.content to obtain the binary image file. Best of luck! Adam
dude this was clear as fuuuck, thanks for just saying what's neccesary, and being so clear ... LLLLLLOVE ITT
Thanks for the feedback! Feeling motivated to put out more content!
Just started learning Python last week. Stumbled onto your channel today and it’s a breath of fresh air to provide some excitement halfway through my coursework! You explain everything in easy terms and it’s a lot more exciting than basic programs. I’m looking forward to finishing my course now!!!
Woohoo! That's awesome to hear. I really struggled learning Python years ago and I think it helps to have a project to work through.
Same!
How am I not exhausted after watching a 30 min video. Man you are awesome.
Thank you!!
I really like the fact that you explain things so clearly, and I regret the fact that I haven't been able to see your videos in the past few months
Great work!
Man you are pure gold. Your examples are very useful and clear. Definitely worth every second of it. Keep the great job man!
hey mate, i recently found your channel when looking for some projects to busy myself with in lockdown and i love how simple you explain things and i hope you get many more subs because your amazing 😀
Thank you!!
@@MakeDataUseful 🙂
Also, if you need a video idea Id love to see a notepad / MS word kind of thing in python. It doesn't have to have a GUI but I've been trying to make one and I'd love to see how someone else would try this. I'm using as few external librarys as possible so it could be a kind of plugandplay solution for quick edits?
You're*@@pw5687
Many thanks to you! I found an Upwork job after watching your tutorial and creating portfolio project. Keep it up, your tutorial are indeed practical. Thanks agian!
hello, have you made money?
Hey there, absolutely loving this series. It is quite hard for new comers to get their first job on upwork, if you had any tips regarding how to land the first job, smooth and easy it would be a real help ! Anyways, keep the series going. Loving these videos
Thank you for the feedback and yes totally agree about landing those first few jobs! I will do some more client communication and landing the job in upcoming videos :)
@@MakeDataUseful hello, thanks to you I got my first contract ! Keep inspiring, keep building !
@@MakeDataUsefulhi
Great content 👍. It's very explanatory and easy to follow along
I'm happy I discovered your channel. Just what I was looking for!
You'll have 1k subscribers way before the end of the year if you continue, i'm sure of it
Thanks Carl! Hopefully!
Really helpful bro. Your content is unique. I just found your channel accidently. It gives a lot of information in easy way. Thanks a lot
Superb video. 🤯🤯Loving the real-world examples for scripts!!! Great work!! 👏🏻👏🏻
Thank you! More to come soon :)
This work is really great. It is a real problem example.thanks for this video.
Glad it was helpful!
This is the second tutorial I am following with you and I like this one way better than the first (the website in the 1st one didn't work), it s good to see the process of webscraping working, thank you!
I was able to accomplish the first video, if you need help ask away. The xml sitemaps page changed in this video's (#2) and now has images instead of the text in 4:38 , but I don't think it is important, but the speed at which this site has changed over 3 years is amazing.
Thank you for this wonderful work, it is very useful, easy and amazing
You are welcome!
Great job, just found your chanel and I already love your videos! Keep the good work.
Thank you very much! These videos are very useful. It would be nice if you continue publish them.
You are a wonderful teacher!!! I just had to say.
fantastic tutorial !!!!!
your approach to try to solve real life problems is incredibly useful !
Thanks Oscar! Much appreciated!!
Even better than episode one. Thank you
Thanks for the feedback Jeff, appreciate it! :)
This is a good video for me as i am also a Attack on Titan fan as well.
Keep posting! Now it’s my wish to learn and solve problems like you do :)
Thank you!! Lots more to come 😎
masterful tutorial dude by far the best explanations of everything even though im still trying to understand some for loops and exception handling also what packages to use but with practice i know i can do it i just stumbled on your videos so i hope i can start learning to do jobs like this so i can do this on the side
dude ! you are awesome thank you very much for the content! respect!
Thank you Sir, for your tutorials
AsyncIO or Multiprocessing can speed this up a lot and request sessions.
Love the videos. I have learned a ton! Towards the end you mentioned the virtual machine. Could you do a video on how that works?
That's really great to hear Ryan! And yep, virtual machine video is on the cards.
Excellent, nice video. Keep doing this work please, man.
Thanks! New ep coming out soon :)
love that shirt, just like my high school teachers wearing their batiks 😄
Really good videos! Thanks a lot. I’d be interested in the server deployment / infrastructure process one day
Hey great idea! Thanks for the idea, I'll share some hosting examples in upcoming vids
Which IDE are you using for coding Python? I've never seen this.
Enjoying your videos!
Thank you
who tf disliked this amazing guy!!!!
Very awesome video
Great video mate, easy to follow
Thank you Danny Rock, really means a lot to me :)
Please do more videos like this .... U will have many subscribers
You got! Another video coming up!
@@MakeDataUseful wow... Really excited... U know it's really useful... I was struck at a point thinking what will i do after learning python. How will i convert my skills to a profession... Now this tutorials really helped me... I am thinking of learning web scraping with python and if u teach any webscraping courses with python.. please let us know.... Will be happy to be your student...
I am a new subscriber and I am happy to join this family.
Welcome to the family! Lots of great people here all helping each other learn and grow
Very nice video, keep up the good work.
Great video, well explained. how about a video showing how to use headers
Superb video
You will have 20 k subs by the end of 2021 don't worry.
Challenge accepted!
Ripper content (did I do that right?). Subscribed and looking forward to more. Thank you!
Great content and inspiring ❤
Bro I just love this serie
Wonderful!
Absolute legend. Thanks a lot man
You got a new subscriber. This was great thank you
Awesome, thank you!
Thanks brother, just subscribed.
Thanks brother! Appreciate it
Im wondering if this days you can still make money on that
Great video❤, Can you teach us about the ide you use?
You deserve a million subs
can i do this to google my business scraping?
❤
Please do you have a community for web scrapers especially for beginners like us that hardly gets the job and feels they are not good enough
when you do work with python yourself do you usually only target web scrapping or do you other jobs as well. by the way i really like web scrapping with some practice i think i can become good at this
I do a lot of web scraping when I am building my own databases/APIs. A lot of my work is data cleaning and remediation prep for the visualization of data. The viz component is usually 10-20% of the effort, most of it is extracting and transforming data. Keep practicing an let me know if you need any help!
Thanks for the video.
I have a python experience as a backend developer for 1 year. I have used python flask framework.
Due to covid ,the start i was working has stopped its operation and i lost my job.
Can i know how can i start earning as a freelancer with the python skills.
thanks man I really appreciate your efforts!
Arigato Kosaimas Sensei!
Douitashimashite!
Hey there, great video. Can you post the consolidated code I´m stucked trying to put it all together. Thanks
best regards
Here's a question. Why would someone who's going to utilize 50,000 images in a machine learning environment need someone to scrape the web for them?
There are several no-code/low-code tools available that allow you to perform machine learning tasks on images without having to write code. I imagine they may be utilising one of those. Also might be easier to give the task to someone else to do to save time and focus on other aspects of their project
I tried to follow along but couldnt get the xml to work.
FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml. Do you need to install a parser library?
yeah i failed too i thought site changed its structure but im new in web scraping so i dunno
In Aug 2023 the page format is changed and BeautifulSoup has changed too.
Fixes that worked for me:
1) install lxml (letter L in front) with pip install lxml
2) Then use this at about line 20: soup = BeautifulSoup(response.text, features="xml")
The web page structure has changed so that the id number is on one line and the PNG on the line following it.
I used the following code (may not be the best way but it works):
for image_url in master_list[0:5]: # get a sample of some images. (requires two lines per image) this 0:5 downloads 2 images
if 'image' in image_url: # the id number line has the word 'image' in it
id_number = image_url.split('/')[-1] # get the last string following the backslash
print (id_number)
elif 'files' in image_url: # the png line follows the id line and has the word 'files' in it
print(image_url)
image = requests.get(image_url) # download the PNG image
image_title = image_url.split('/')[-1] # get the file name in the last string after the final backslash
file_name = id_number + "-" + image_title # add the id number + image name together
# write the PNG file
# example: 755899-Poker-Chip-PNG-Photos-80x80.png
with open(file_name, 'wb') as file:
file.write(image.content)
great tutorial!
i understood all but i have confusion. there were 5k link so how did we download 50k images??
Did RUclips suggested me this video because it has Eren Jeager's images in this video or is it because i am learning python lol. every where i go my favorite anime follows me.
I applied to. numerous projects in upwork, no one is contacting. Competitions are too much.
thanks so much for the video, but unfortunately the website didn't works for me , (page not found). but i'll keep learning
Dang, about time I did a new web scraping tutorial! Watch this space!
In 8:29 and 14:04 you defined "soup" and "loc" on different lines. Does that mean we can use the same name for variables? Sorry im still new to python. Also i noticed you mispelled "response" as "repsonse" in 11:50, was that intentional? Thank you for a real practical use of python though :)
Edit: nevermind, you said its okay for repeating the "soup", my bad :)
Python is both a strongly typed and a dynamically typed language. Strong typing means that variables do have a type and that the type matters when performing operations on a variable. Dynamic typing means that the type of the variable is determined only during runtime.
So yes, you can use the same variable
@@Gasterus69 thanks!
hi body.
you forgot the insert loop for part xml link.(part1,part2,.....
)
thank you so so much for this ❤❤
great tutorial my friend , i want to ask you how you send this script to the client ? as .exe file or just the script?
Depends on the client, a lot of the time they just care about the data and solving the problem.
@@MakeDataUseful ok, thank you
You're a champ!
Please create a video in which you interact with the employer? I'm looking for my first job at the platform, but I'm not sure how to proceed when I finally get "selected" and have to interact with the employer. Thanks!
Will do!
Very helpful
Do you have a video about how to deal with clients?
Hello, Do you make money from upwork?
Awesome tutorial, Can you post here the code for incremental downloading up to 50k to not stress down the web server.
If you run it in the cloud, just use something like time.sleep(time_in_sec). There is other ways to do it, but that one is quite easy.
Could u do a vid using urllib and beautiful soup please
That's great, but I have a ConnectionError problem, even StackOverflow solution doesn't work perfectly
Hi Rahal what is your error?
@@MakeDataUseful HTTPConnectionPool(host='www.pngmart.com', port=80): Max retries exceeded with url: /image/246105 (Caused by NewConnectionError(': Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))
@@rahalmehdiabdelaziz8121 looks like a timeout, a couple of things to test. Try visiting the url with your browser and I'd you are successful they connection may be getting refused due to incorrect headers. Another option you could try is adding timeout=X where X is the number of seconds you want to wait. More details available here requests.readthedocs.io/en/master/user/quickstart/
@@MakeDataUseful I tried those type of solution but there is no efficient result ... however I found a " SOLUTION" : the error doesn't occur everytime (I suspected the weaknesss of my internet connection) so I've done a WHILE loop to repeat the request until it works ... thank you very much for your response ... I wish you do series of real world tasks other than webscraping ... Thank you and good luck
I didn't know Matt Walsh was a programmer.
😂 He wishes!
this is great but how the extracted html file become a picture
Hey Art,
Thanks for the feedback! At around 20:10 I source the image URL then use a response = request.get(image_url) to source the image then use a with open... and access the response.content to obtain the binary image file.
Best of luck!
Adam
Do you get any money
You have a skinny seth rogen look going on here
Webscraping mission on upwork for 20$..you're never gonna be rich doing that. I hate those fakes titles
It says making money with web scraping on upwork, not getting rich lol
Agree! A lot has changed over the last couple of years. Great opportunity to learn Python