- Видео 148
- Просмотров 185 704
Web Scraping Ninja
Пакистан
Добавлен 17 сен 2011
This channel is about web scraping techniques python web scrapers, scrapy framework, web crawling and selenium web automation and web scraping.
You can buy my services here:
www.upwork.com/workwith/assad360
Or contact me through my website by submitting a form request
dataguru.cc
Read blogs here
dataguru.cc/category/blog/
You can buy my services here:
www.upwork.com/workwith/assad360
Or contact me through my website by submitting a form request
dataguru.cc
Read blogs here
dataguru.cc/category/blog/
Selenium in Google Colab; Simple Selenium Project in Python | Create Graph in google sheets
Want to take your Google Sheets visualizations to the next level?
This video tutorial dives deep into creating dynamic charts using Python's Selenium library and Google Colab.
We'll guide you through the process step-by-step, from:
Setting up your Google Colab environment
Importing essential Python libraries like Selenium
Interacting with Google Sheets using Selenium commands
Building interactive charts that update with your data
By the end of this video, you'll be able to:
Automate chart creation in Google Sheets using Python
Generate dynamic charts that react to changes in your data
Impress your audience with visually compelling presentations
This video is perfect for:
Data analysts who want to str...
This video tutorial dives deep into creating dynamic charts using Python's Selenium library and Google Colab.
We'll guide you through the process step-by-step, from:
Setting up your Google Colab environment
Importing essential Python libraries like Selenium
Interacting with Google Sheets using Selenium commands
Building interactive charts that update with your data
By the end of this video, you'll be able to:
Automate chart creation in Google Sheets using Python
Generate dynamic charts that react to changes in your data
Impress your audience with visually compelling presentations
This video is perfect for:
Data analysts who want to str...
Просмотров: 648
Видео
Selenium in Google Colab Tutorial For Beginners: Web scraping To Google Sheets
Просмотров 5 тыс.8 месяцев назад
In this tutorial you will learn about selenium automation and web scraping in google colab, the data collected will be save in google sheets by authenticating the google credentials in a very simple and easy steps. 00:00 Introduction 01:00 Install Selenium on Google Colab 06:08 Scrapie Website Using Selenium in Colab 14:40 Write Google sheets from Google Colab 22:44 Summary
Write Dictionary and List to CSV and Json Files | Python Jupyter Notebook
Просмотров 1399 месяцев назад
If you are struggling to save your scraped data to csv and json files, in this video i will walk you through each steps, In First part of the video you will learn how you can use csv and pandas libraries to save a csv file First you will learn how to save simple string with comma separated values to a csv file using context manager, after this you will learn how to import csv library and pandas...
These Web Scraping Proxy APIs will BLOW YOUR MIND 🤯
Просмотров 30710 месяцев назад
These Web Scraping Proxy APIs will BLOW YOUR MIND 🤯
How to Run Scrapy on Jupyter Notebook and Crawl a Website to CSV
Просмотров 2,1 тыс.Год назад
How to Run Scrapy on Jupyter Notebook and Crawl a Website to CSV
Python scrapy on google colab real quick.. part-1
Просмотров 130Год назад
Python scrapy on google colab real quick.. part-1
Web Scraping 2.0: Scrapy & Google Colab - The Future of Web Scraping
Просмотров 4,4 тыс.Год назад
Web Scraping 2.0: Scrapy & Google Colab - The Future of Web Scraping
Scrapy Tutorial: Python Web Scraping
Просмотров 1,8 тыс.Год назад
Scrapy Tutorial: Python Web Scraping
Roadmap for Python Web Scraping: Beginner to Pro
Просмотров 402Год назад
Roadmap for Python Web Scraping: Beginner to Pro
Selenium Automation in Jupyter Notebook: Extracting Google Search Results
Просмотров 2,7 тыс.Год назад
Selenium Automation in Jupyter Notebook: Extracting Google Search Results
Get started with Playwright in Google Colab
Просмотров 1,5 тыс.Год назад
Get started with Playwright in Google Colab
Complete Python Web Scraping Roadmap | 2023
Просмотров 938Год назад
Complete Python Web Scraping Roadmap | 2023
Beginners Tutorial: Selenium and Python on Your Mobile Phone
Просмотров 6 тыс.Год назад
Beginners Tutorial: Selenium and Python on Your Mobile Phone
Introduction to Proxies for Web Scraping and Automation
Просмотров 350Год назад
Introduction to Proxies for Web Scraping and Automation
Create Python API with Scraped Data in 30 minutes
Просмотров 1,2 тыс.Год назад
Create Python API with Scraped Data in 30 minutes
Upload Your Prize Bond Numbers & Get Notified of Wins with Our Website
Просмотров 268Год назад
Upload Your Prize Bond Numbers & Get Notified of Wins with Our Website
How to Run Selenium on Google Colab Notebook: Fixing the "Unexpectedly Exited" Error (status code 1)
Просмотров 12 тыс.Год назад
How to Run Selenium on Google Colab Notebook: Fixing the "Unexpectedly Exited" Error (status code 1)
Python Selenium Tutorial in Hindi: Dynamic Websites Scraped
Просмотров 6 тыс.Год назад
Python Selenium Tutorial in Hindi: Dynamic Websites Scraped
Tutorial: Selenium Automation on Google Colab Notebook
Просмотров 28 тыс.Год назад
Tutorial: Selenium Automation on Google Colab Notebook
Automate Web Scraping with Selenium & Jupyter Notebook | Tutorial & Examples
Просмотров 4,8 тыс.Год назад
Automate Web Scraping with Selenium & Jupyter Notebook | Tutorial & Examples
Complete Python Web Scraping Tutorial: Scrapy, Selenium, Requests, BeautifulSoup and lxml
Просмотров 6 тыс.2 года назад
Complete Python Web Scraping Tutorial: Scrapy, Selenium, Requests, BeautifulSoup and lxml
Automate Your Excel Tasks With Python: Simple steps
Просмотров 2432 года назад
Automate Your Excel Tasks With Python: Simple steps
Scrapy Tutorial: Learn How to Build a Web Crawler with Examples
Просмотров 1,3 тыс.2 года назад
Scrapy Tutorial: Learn How to Build a Web Crawler with Examples
Selenium Automation In Jupyter Notebook: Step By Step Guide
Просмотров 12 тыс.2 года назад
Selenium Automation In Jupyter Notebook: Step By Step Guide
Plz reply
Sir aap python expert ho to plz kuch help kr dijiye muje Main web automation ki script bna rha hu to ek jagah pr atka hua hu 2 din se plz help 🙏 Ek Web field me date nhi dal pa rha hu us field se interact to ho rha hai python code but calendar ka pop open hi rh raha hai date select nhi horhi
Popup is probably in so try switching to , look for examples online and your problem will be resolved
@@ScrapingNinja thankyou sir main try krta hu
@@ScrapingNinja sir help no there there are open a calendar 🗓️ widget
@@ScrapingNinja kya main code beju sir
Assalamualaikum sir Maine apki bhi videos dekhein or bht sary or RUclipsrs ki bhi Kuch samjh nhi arha or Jo main videos dekh ky code perform kar rha hu woh work nhi kr rhy to kindly mujhy Thora guide krein mujhy ek month hogea mujhy samjh nhi arha mujhsy contact karlein please
Ws, some websites change over time so the exact code doest always work, you need to use the techniques and terms and apply them in your context instead of just copy pasting the code…
@ScrapingNinja nhi Matlab bilkul copy paste nhi kar rha hu lekin jesa main bs4 ko use kr rha hu code run nhi kr rha nah response code nah hi request.get url main Kuch run horaha hai terminal main error arha hai
Agr possible hoto kia hum connect hoskhtay hain via phone or meeting mujhy concepts clear krnay hain thoray to please?
Requests pn first really?? What about dynamic content loading then. U need some headless browser scrapping
Most of the dynamic content is retrieved by requests using their ajex request
I get an error while running !apt-get install chromium chromium-driver The error message is Reading package lists... Done Building dependency tree... Done Reading state information... Done Note, selecting 'chromium-chromedriver' instead of 'chromium-driver' Package chromium is not available, but is referred to by another package. This may mean that the package is missing, has been obsoleted, or is only available from another source However the following packages replace it: chromium-bsu E: Package 'chromium' has no installation I am running Colab in Chrome browser on an Android phone.
chromium-driver is installed and it installs chromium as well so you can start using it, ignore the error
Web development or scraping wich best ?? Best sallary packege
Both are interrelated learn web development and you can use that knowledge for web scraping..
i did everything same as you but my csv file is empty, why ?
Can you see scraped data in your out logs?
WebDriver.init got an unexpected keyword argument 'executable_path' help pls
What do You think about brightdata?
Brightdata is proxy service, they are very reliable and one of the best in business.
I am from Korea. I love you, Guy
Thanks.. 🤩
I want to do free lancing
Get a skill, no matter major or minor and get started
Appreciate you video, but Unfortunately i couldnt learn anything as you were just writing the code and didnt explain what all those code mean and when to write what
I think it has audio as well..
Thanks a lot Mate, literally spent hours with ChatGPT trying to resolve the error in Selenium. It is always better to start off with a working piece of code rather trying to generate entire code from scratch.
I got an error.ReactorNotRestartable(). The Crawler Process doesn't seem to allow rerunning a spider in the same notebook multiple times. Have you tried any workaround?
In async you need to add task to existing task group i haven’t tested it yet but you can try adding task in the loop.
@@ScrapingNinja I tried !scrapy runspider spidername. It works and allows rerunning it.
What is filter function ?
Filter function is to filter your existing prizebonds if you want to search for specific prozebond in the list to delete or see etc.
GitHub account nahi hai
I dont see many channels with the dame quality of content. Good work!
Appreciate that
debir haber visto este tutorial de primeras muchas gracias thank you so much
I am glad you find this helpful.
cant find the second code
Follow this video.. Tutorial: Selenium Automation on Google Colab Notebook ruclips.net/video/mOJiWrjFVKY/видео.html
Commands were missing I found it and it worked. Thanks to God.
Follow this video.. Tutorial: Selenium Automation on Google Colab Notebook ruclips.net/video/mOJiWrjFVKY/видео.html
I can't find the second command on the link you provided, thank you!
Follow this video.. Tutorial: Selenium Automation on Google Colab Notebook ruclips.net/video/mOJiWrjFVKY/видео.html
me too
i use this to scrape google maps reviews, but it return in chinese 😵💫, also does Google Colab really not display pop ups?
Open whatsmyip website in colab and check where your colab ip is located, colab runs on servers and yours probably located in china.
Can I scrape sites using this that use captchas
You will need to use proxies to avoid captcha.
@@ScrapingNinja I’m new to code. Can I do this in Google colab? Does it cost money to implement?
SessionNotCreatedException: Message: session not created exception: Missing or invalid capabilities (Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 6.1.85+ x86_64)
Check your code to make sure you have set compatibilities correctly
Options
Bro make a video on chrome webdrivers
Chromium and chrome are same..
Playwright
sir I wanted to scrap shopping website like amazon product can you pls suggest how to take there lxml code because they have given it in the form of div id,script data and mostly reactjs code finding it very difficult to scrape products
You need to adapt, first try to parse with lxml if its not present try looking for something in script tag, sometimes data is loaded as json in script if its not their either try something else like selenium/playwright.
I wait for your answer friend
Can you please repeat the question?
Thank you so much. You tutorial helped me a lot!!
I am glad 😌
SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 114 Current browser version is 126.0.6478.126 with binary path /usr/bin/google-chrome error
You need to update chromedriver using apt-get
Unfortunately couldn’t pass from step on 5:18, the command driver = web_driver() didn’t work. Just giving up on trying to use selenium on colab. Visual Code has been better to me…
Can you share the error?
It works perfectly fine on Colab. I'm using the selenium on colab.
It doesn't work without shell command maybe😢
It does work without shell commands now, follow this Tutorial: Selenium Automation on Google Colab Notebook ruclips.net/video/mOJiWrjFVKY/видео.html
how to stop scraping until certain page? like until 25 page to stop? please reply me
you can try different logics here for example: if you are looping over pages you can break, If you are clicking on next page, check if page=25 in next_page_link
bhai kha se ho aap?
Planet 🌍
music background me mat lagao disturb hota hai
Ok boss
no me funcilno
Make sure to include options, it wont work without options.
This is still relevant in 2024. Definitely a helpful vid.; Btw what to do if the same is to be done locally? I am on MacOS
Thanks man, and I think you can, i will check
Thank you so much it legit worked you are soo underrated god bless you❤❤
Thanks ☺️
Nice video Nailed the point
Thanks 🙏
very low voice
Thanks for feedback 👍👍
bro, try using this method again to scrape all the product review components on the Lazada website, when I tried it like in the video and just adjusted what I wanted to scrape by copying the inspection results and then running the code, the results didn't appear like at minute 08:21
make sure you are using correct xpath and also try to print page source to see if the correct page is being loaded
i am getting this error "'str' object has no attribute 'capabilities'" please help
Check to make sure you have correctly imported options and its not a string
Hey Bro! I am a beginner and I am getting different errors. Nobody taught me to handle those errors (particularly IP blocking) and I am getting different responses. Can you help me?
what error you are getting? share error message here
@@ScrapingNinja Bro I am talking about general problems which every person encounters such as captchas, IP blocking etc. Kindly make a video regarding how to prevent IP blocking for beginners using tools such as selenium and bs4 and show bypassing in a easy way! Also please make a video on how to bypass bot detection and captchas. I have searched for it everywhere, I am getting the related tutorials but of no use because I am a beginner and they aren't teaching for beginners. So waiting for your kind response.
@@realitywords-17398 Use a proxy server to avoid it.
I am from Thailand. I love you, Guy
Thank you so much dear.. ❤️
Perfect, you saved my day!
Glad I could help! don't forget to subscribe.
@@ScrapingNinja You earned it, for sure
Thank you so much for this video. I wasted 24 hours in futility trying to find a way to get Selenium to work on Colab; TILL i found your video, and bam! It worked. I had errors with the executable path, so i had to remove it completely and passed only the options argument. ❤
You are welcome, 🤗 I am glad you find this video useful, don’t forget to subscribe.
AssalamuAlaikum buhat acha sikha rahe aap, Sir saath he notebook ko github main push karna bhi btadia krain n main VS code pr Selenium se chrome k latest version jo 124 ya 126 hai tu usk saath web scrapping karna chahri tu kindly uska bhi btadain
ws, thanks you cand ownload github desktop to clone with github and push you files and as far as web scraping using selenium in concerned method is same regardless of version. you can watch video on my channel related to web scraping using selenium
@@ScrapingNinja Sir chrome k latest version k drivers available nahin hain wahan jo aapne website btayi hai n secondly main iss video main 48:00 pr aakr stuck hogai hun q k main VS Code use krti hun n aap pycharm use karre hain, I am a beginner tu VS Code ka btadain k kese hoga
You are struggling with scrapy shell part?
Good 👍 information
So nice of you
Hi friend, I can't find more videos of this tutorial selenium bot. It will be more easy to learn if you make set of tutorial by numbers like class1 class 2 etc. Thanks and reply to where i can find more videos about this tutorial
I have some videos i will make playlist for you.
Thanks, brother. I will be thankful if you make it