GO for Beginners - Web Scraping with Golang Tutorial
HTML-код
- Опубликовано: 13 авг 2024
- In this golang video tutorial I will guide you through an excellent example of a beginner project for those who have started learning GO. This basic HTML web scraper will introduce you to some of the GO fundamentals, and also hopefully make a slight more interesting project that you can use to expand your GO skills.
We will learn how to setup our project and use a web scraping framework called Colly - go-colly.org/ to extract some data from a test site.
Patreon: / johnwatsonrooney
Donations: www.paypal.com/donate/?hosted...
Proxies: iproyal.club/JWR50
Hosting: Digital Ocean: m.do.co/c/c7c90f161ff6
Gear I use: www.amazon.co.uk/shop/johnwat...
-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
------------------------------------- 
Наука
Hey great work,
Actually there is zero values for fields in the struct so you don't have to specify all the fields values if you don't have it.
for example :
type Item struct {
Name string
Price string
]
func main() {
item := Item{Name: "John"}
fmt.Printf("%+v
", item)
}
try it and let me know what you have learned.
Ahh ok thanks that makes sense!
@@JohnWatsonRooney Keep going John Go is so great I love it
Best guide for web scraping on the internet!!! God bless you mate. Because of you I've been able to start so many projects that I was scared of starting because I thought web scraping was too complex. But you simplified so many things for me
Great video, John, thank you! You strike a perfect balance of moving at a good pace but clearly explaining what you're doing.
Glad it was helpful!
This is the way to learn a new language. Simple, concise programs. Great video!
I've already been familiar with Golang syntax but coded interesting with until today, wow, the speed and simplicity of Golang plus this library that comes packed with all the features I need for scrapping, amazing discovery, I don't think I'm going back to python for this kind of projects. Nice video mate as always 👌
thanks! it's very different but i really like it, and am learning more and more GO in my free time
I really like how you present the information and structure your videos. Would you ever consider doing a video on how you prep to make videos?
Thanks! Sure, I could look at doing something like that
Great content as always! I’m pleasantly surprised that you are covering new languages, as I really like your succinct style of presentation. I have always heard of the good things about Golang but had an impression that it’s mostly for web backend stuffs. Your video sparks my interest to actually pick it up now!
Great walk-through! I used this to get a web-scraper going. I had done some web scraping with Python using selenium to get product info for a local supermarket chain, the code was much longer and getting it working was a lot more involved. It would probably have been neater with Scrapy. Anyway, this makes for a nice Part 1 of a mini-project. Part 2 would be to get your data into a database, sqlite as a start. Finally, Part three would create a RESTful API to expose your scraped data.
Thanks! Part 2 is in the works
Please do a comparrison of speed between Python and Golang scraping/frameworks/libraries. Thank you. Love your content. I hit the like button first and watch later, top quality content as always.
Thank you very much! Yes I am going to do a comparison video!
I'm Python programmer and I'm changing to Golang for these cases. The way each one access hardware resources is huge different. Golang is faster than Python by far.
Thanks for the video
At 11:30 it start to get really interesting when the Colly Framework starts auto-navigating to different pages. However, the details for doing so might not be that obvious for beginners like me (tooke me some minutes) and could have been explored in a bit more detail and length. All in all, great vid. Have learned a lot in no time!!
Sure that's a fair comment. I'm glad you enjoyed the video - I will do some more on GO and go into it a bit deeper!
Great content, John. Seems Go/colly will make things easier for the personal project I've been working since I first thought about trying scraping a couple of months ago. Just as I'd managed to get what i needed from python following your vids and adapting the process for the site I'm wanting data from, and others code i'd found online, to work for the most part. Certainly getting the url's i need to be able dive in further was a lot easier with Go.
Thanks. I really like Go and colly is great for these things. But Python is still a simpler language and easier to pick up - but give Go a chance and see how you feel
Great video. Need more Go content please
awesome tutorial, really enjoyable to follow along! thanks, you earned a sub!
Awesome, thank you!
Thank you brother.
As always great content from great mind.
Keep going brother 🤗.
Hi John,
This was great content! Go with Go ;)
Thanks much appreciated!
good content man, please do more scraping with go especially on dynamic website, ty so much
Thank you appreciate it!
simple and provement code.... 👍👍👍
👍
This is an excellent tutorial, John,
At the end of your demonstration you mentioned about using it in a Proxy. I found that intriguing, could you please elaborate on this, or if you know any tutorial online that I can read on would be great,
Cheers.
Much easier to get started than Scrapy used to be . General comment - such FWs are good for structured sites but what do you think about where there are no next links and no sitemap?
it makes it a bit more tricky, but you can either use a range loop for pages, or generate page numbers to add to URLs and use c.Visit(). It really depends on how the page works. but yes you are correct scraping frameworks like colly are really best when its structured HTML sites
I am learning your python videos,I need to keep the pace, haha!
Ha thanks I just fancied a change, back to more Python next week!
great video !
Oh my god, I need to learn Go...
I'm learning more and more of it now and really like it so far
Really good work, exactly what I was searching. Also, what color schema do you use?
Cool video and setup mate! What font are you using, and is that the Gruvbox theme?
thanks! i change my setup a lot but yes this is Gruvbox, or "Gruvbox Material" i used to use a lot. as for the font i'm not sure but I think its "m plus 2m" from adobe
Top man! I see that you moved over to Neovim if I am not mistaken. That's my goal eventually, currently using Vim motions in VScode as an easier transition :)@@JohnWatsonRooney
@@sebwylleman I have, I've been using it for a while now - sometimes I still use vs code for demos, but with vim motions!
awesome!
Thanks for the video i really liked it
But I have a question can we do web scraping with c/c++? If yes how ? And Sorry for my bad English
Hey! I’m sure you can, however I don’t know anything about c or c++ so I’m afraid I can’t really help
Is possible to scrape a page that need login first ?
Yes it is, however you will need to find out how the login form works and make a POST request to login first, saving that in a session object. Possible but more complex than what I covered in this video
A lot of people are learning go But I keep googling rust libraries LOL. My history is full of "rust rest api framework", "rust web scraping library", "rust machine learning library".
I also looked at rust for a second language but after searching a bit I chose Go and am liking it so far!
I heard that go is really fast. does that apply in case of web scraping too? I have very big scrapping projects with millions of pages. Even python multi threading, async isn't enough. I was wondering which other language might work instead.
It absolutely can be, it adds complexity but yes. Colly states 1k plus pages a minute - I’m sure this can be achieved with Scrapy too which is on the twisted framework. Interesting though I will do some testing
@@JohnWatsonRooney would be a great video benchmarking stuff!
I guess having native concurrency with Goroutines doesn't hurt either
@@mondesinxi9397 sure, all stuff I am still learning about but so far so good
Which is more flexible if compare to Python?
is go scriper is faster then python scriper😐
I’m working on a comparison!
Hope you're still enjoying go :)
Yes very much so!
@@JohnWatsonRooney any remarkable limitations / drawbacks on Windows? I've only heard it once on some talk
@@wtfdoiputhere to be honest i;m not sure, I don't use windows so much anymore
Hey, please what is ur VScode theme name ?
Hey it’s one of the GitHub themes, dark medium or dark soft I think
Xpath + regex are much better ❤️
do we have to do an os.Exit anywhere?