Introduction to Selenium Using R (RSelenium)
HTML-код
- Опубликовано: 11 сен 2024
- Thank you for watching! Comment for any questions/suggestions.
Let's connect: / samer-hijjazi
To install Java on Windows 10:
• How to Install Java JD...
Check out the code from my Github repository:
github.com/ggS...
To learn more about RSelenium:
github.com/rop...
To learn more about XPath:
www.w3schools....
DISCLAIMER: If you're having trouble running RSelenium, I would suggest you watch my installation guide here:
ruclips.net/video/GnpJujF9dBw/видео.html
Wow, I spent 6 hours failing at what this video showed me how to do in 16 minutes. 45 if you count when I paused it.
Great video Samer, the interactive nature of RSelenium in controlling the web browser is really neat.
Another great tutorial Samer, thanks a lot! I have started learning RSelenium some time ago, and it would have been so great to have this video at that time.
Thanks a ton for your support, Joachim! The lack of videos on RSelenium is what encouraged me to make this video haha :)
Wow this is by far the best tutorial I've ever seen on RSelenium! Everything is so helpful and making it a video really helps me follow along. Fantastic!
Excellent Video on RSelenium Samer. Thanks for benefitting the R community.
Hi Samer. You are such a great teacher. I learnt a lot from this video. Thank you for your time and effort.
My pleasure. Thank you for the kind comment!
Thanks, Samer! My Selenium-Python bot stopped working, but R (and your tutorial) saved me!
I have never coded in my life and was able to follow this tutorial to start what I am looking to do. I can't tell you how excited I am. Thank you so much!! Made some things a legitimate option for me to figure out. I was able to use this and apply it to a specific webpage I am looking to navigate
Absolute Goat. This was awesome. Had some hiccups setting up chrome browser but you even had videos for that thank you.
You are way better at teaching this than any resource on the web, and even better than my teacher! love the videos. Could you make a video scraping multiple pages and putting it into a data frame or xlsx?
Thank you for the awesome feedback! That's a pretty good idea. I'd like to make a video like that in the future :)
Samer: This is great. Amazing that it is your first video. You are a natural. I look forward to many more and am subscribed.
Thank you, Peter! I really appreciate it :)
You rock Samer! Thanks so much. This was exactly the get going information I needed to begin a work related Selenium webscrape. 🙌🙌
Thank you for the wonderful feedback. I'm glad it helped!
This video helped me a lot. I'm already using RSelenium at my job. Thank you so much
You're welcome! Glad to hear it helped you out
Very helpful, I think that’s exactly what I needed to perform a task at my job, thanks a lot!!!
Just as an FYI, it was just a tiny typo at the end, you typed jave.exe (with an e) and not java.exe
Thanks again!
Thank you for the great feedback. Glad to hear it's helped you out at your job. Grrrrr, it's always the tiny typos that get me. Great catch!
Really helpful tutorial for RSelenium. Thanks a lot!
Hello Samer, thanks a million for the detailed explanation given to each line of code. Its been really helpful.
Please at your spare time could you make a video about scrapping data from RUclips using R (using either RSelenium or RVest) or you could refer me to any tutorial about such if you know any
Thanks. Please treat as urgent.
I'm glad to hear this, Uche! Although there's lots of videos on scraping RUclips data with Python, I am not sure about any R tutorials for that. Perhaps look into the RUclips API for this. Anyhow, I would like to visit this topic and see if I can make a video on it. Thanks!
Thx a lot Samer Hijjazi. Very nice explanation.
what do I do when my chrome version (120.0.6099.109) isn't in binman::listversions? I am using RSelenium ver 1.7.9
thank so much, It's the first time this works for me, great video
This video was great, thanks!
Thank you very much, Tivan!
thank you. It was very helpful. you're a great teacher!
kind Sir. I can't thank you enough for this !
Great tutorial! I was wondering if you could provide advice on how to run RSelenium just like this while using a proxy? I'm unable to find any good tutorials out there on how to do this
Thank you for the kind comment! I have not tried that before, but maybe I can look into it.
Hi. When I give command: R.4.2.0 > binman::list_versions('chromedriver'), I get: Error: C:\Users\eduardo\AppData\Local/binman/binman_chromedriver app directory not found. Am I missing something? Thanks
Oh, I figured out: We must first run a successful RSelenium::rsDriver in order to create the missing directories. Agter that the binman command works ok
@@tugastone How do you do that? I'm getting the same error. Thanks!
@@flordemaria127 To create the needed directories, You must run for the first time: selenium = RSelenium::rsDriver( browser = "chrome" ) After that you can now run binman::list_versions('chromedriver') you can then stop selenium server selenium[["server"]]$stop() and then run the RSelenium::rsDriver command again, this time fine-tuning all the parameters you could need. I hope it helps
@@tugastone Thank you very much!
Thanks for helping out! :)
Hi Samar, How do we select a drop down menu in Rselenium. I tried using findElement drop down name and clickElement() but not sure how to select a value from that. Also the drop down is not expanding to show the options
Make sure to inspect the elements in your dropdown after you click on it. Then use those attributes to help click on the elements.
Great video
Great tutorial. Thank you so much.
Hey Samer, thanks for your video, it helped me get started with RSelenium.
Do you happen to know some way to make the steps taken by RSelenium longer? I'm using it to scrape data from multiple pages, using a for loop (enter link -> click button -> get data -> start over with another link). However, because everything happens so fast the pages can't even fully load and it can't get the data before going to the next loop. Any tips?
My pleasure! Try using the Sys.sleep function between your lines of code. You can specify how long it should wait before moving to the second line
Great video, thanks for sharing
Great video! Thanks so much.
I have a question and is what if we want a specific element (like prices) for multiple pages with r selenium and no just the first page, ¿how we can get this?
Thanks Juan! You'll have to click on the next button and move to the next page. You can do so using the clickElement() function. Then put that in a loop so it loops over all the pages.
Great tutorial!
not sure if youll respond but I get this error:
Could not open chrome browser.
Client error message:
Undefined error in httr call. httr output: Failed to connect to localhost port 14415: Connection refused
Check server log for further details.
Hi! Watch my RSelenium installation guide. It should help out: ruclips.net/video/GnpJujF9dBw/видео.html
Thank you very much @Samer Hijjazi Is me who asked you the question about the possibility to use RSelenium from Rstudio cloud during the meetup @R-ladies Tunis
Thank you for the feedback! I remember you from that workshop. Glad you liked the video :)
Hi Samer thanks for the helpful video. I am getting an error httr output: Failed to connect to localhost port. Can you please guide
Hi Sukalyan. Do you have the Java Development Kit installed? You may need to install and set it up first to solve the issue you're running into.
Excellent!
@Samer Hiijazi Great tutorial, as always! Would be super helpful to also have some instructions on how to set up a VNC so that remDr$()open actually opens a Browser window!
I get this: "PATH to JAVA not found. Please check JAVA is installed.". So, I can not continue. ¡Help!
You have to install JAVA for RSelenium to work. Please refer to my RSelenium Installation Guide tutorial on my channel.
Great video. Can you do a video on how to bypass Cloudflare and other anti-bot tactics?
Thank you! It might be a little challenging, but I'm willing to explore this idea :)
Amazing tutorial Samer. However, I kept getting this error:
Could not open chrome browser.
Client error message:
Undefined error in httr call. httr output: Failed to connect to localhost port 14415: Connection refused
Check server log for further details.
Warning message:
In rsDriver(browser = "chrome", chromever = "114.0.5735.90", verbose = FALSE, :
Could not determine server status.
Any suggestions will be really appreciated.
Great! Subscribed!
Do you know if RSelenium has stopped being maintained? It doesn't seem to support the most recent version of Chrome.
RSelenium has stopped being maintained a long time ago, although it still works. Refer to my latest video on setting it up, as the setup has changed
Hello Samer, thank you a perfect tutorial! Could you please help me with this error I'm getting when trying to set up the server (JAVA is installed, chrome version is all right):
Selenium message:Unable to create new service: ChromeDriverService
Could not open chrome browser.
Client error message:
Summary: SessionNotCreatedException
Detail: A new session could not be created.
Further Details: run errorDetails method
Check server log for further details.
Nevermind, I managed to solve the issue. If anyone else has the same problem, I'm happy to help.
@@adamruschka Thank you for the great feedback! I was looking for some solutions via stack overflow, but then I saw you figured it out. What was the problem?
@@SamerHijjazi Hello. The problem was in the AppData\...\binman. The folders containing the .exe drivers needed to be erased and downloaded again. I'm not exactly sure why, but making the script to download them once again made it running.
Thank you for this tutorial. Well done! However, after trying two different computers (both Win10 64-bit), different internet connection(s), also with antivirus offf... and most of the proposed solutions below, I still get the message "could not open chrome [or firefox...] browser. Client error message: Undefined error in httr call. httr output: Failed to connect to localhost port 4567: Connection refused Check server log for further details. Warning message: In RSelenium::rsdriver(browser = "chrome") : Could not determine server status." Can you, please, review, if the tutorial (which is otherwise great) is still working with the current version of chrome/r/rstudio, the packages in question? I have certain preoccupation (based also on quick search elsewhere on the internet) that for many users, the suggested approach does not work. My apologies for this stupid question.
Hello! This issue has been brought to my attention recently. I have found a solution to your error and I am hoping to make an update video soon. Thank you for the feedback :)
@@SamerHijjazi I have the same, is it related to using Mac OS?
Many thanks for putting this tutorial together! however, when I use the binman::list_versions("chromedriver"), I get the following error: binman::list_versions("chromedriver") - any suggestions?
Thank you for the feedback. You may need to install the binman package first. See if that solves it.
Hi @SamerHijjazi, Thanks for the great work. I was wondering if there's a way to use RSelenium within a ShinyApp? Any help would be greatly appreciated.
Hi! Thank you for the feedback. I personally do not have much experience with Shiny, so I wouldn't know.
Magnificent tutorials.
Could you please teach us how to scrape Facebook, the Marketplace?
Or public comments from any public group from Facebook?
We would greatly appreciate it.
Regards and keep going crack
Thank you! Social media is usually a little trickier to scrape, so I probably won't cover it on this channel
my chrome version is 124.0.6367.92,
But the only options i have in chromever is 113.0.5672.63" "114.0.5735.16" "114.0.5735.90"
do i have to install an older version of chrome?
I have struggled with this issue for the last 3 months. After doing all your steps, i get: BEGIN: POSTDOWNLOAD
Error in if (file.access(phantompath, 1) < 0) { :
argument is of length zero
Hi there! Within the rsDriver function, you'll need to set the phantomver = NULL. That should make it work.
Hi Samer, nice video. I keep getting Geckodriver timeout when I try to load driver. Do you know why everytime R runs that predownload process for different things like phantom, geckodriver ete?
Hi there! Hmmm, I wouldn't have a solution at the moment. Perhaps try to reinstall the RSelenium package. Also, make sure you have Java installed on your computer.
Hi samer.
why getting this error in rsDriver ?
rs_obj
Please refer to my installation tutorial: ruclips.net/video/GnpJujF9dBw/видео.html
Thanks!
Thank you for the super thanks!! Really appreciate it 😊
Hello sir. I got an issue message when using the function 'findElement'. that said :
Selenium message:no such element: Unable to locate element: {"method":"css selector","selector":"#pagination\-table\-container"}
(Session info: chrome=107.0.5304.107)
Make sure you're referring to the right syntax when trying to locate by either css or xpath. That's what's causing the error.
Hi Samer. Fantastic job. I'm trying to follow along your tutorial, but I'm getting an error in rsDriver command. Could you help me? I need to install the chrome webdriver for selenium? Thank you very much for your help.
Thanks Jose! Please refer to my RSelenium Installation Guide tutorial.
@@SamerHijjazi, thank you very much. Delete the license file it's all that was needed.
Thank you very much for the great instructions. It helps me a lot. I works well within RStudio IDE. However, when I tried to put the script into cron job to automate it everyday, it did not work and an error message occurred "sh: netstat: command not found". I checked and the netstat is already installed from Terminal. Not sure what happened. Could you please help with this? Much appreciated! I'm using MacOS
Thank you for the amazing feedback. Try removing the free_port argument from the rsDriver function. That could help
Hi Samer, Thanks for the amazing video. I have question. When i run rsDriver(), i get this error: "Error in if (file.access(phantompath, 1) < 0) { :
argument is of length zero". I am new to RSelenium and i am not able to figure out how to fix this issue. Any suggestion would be really helpful. Thanks
Hi pyarasid, thank you for feedback! Have you installed the java development kit first?
@@SamerHijjazi Hi Samer thanks for the video. I received the same error, and I do have jdk installed. I'm stumped, any thoughts?
@@MCshaneization Hi. Try this .
binman::rm_platform("phantomjs")
wdman::selenium(retcommand = TRUE)
For me, worked.
When I create the selenium server the log says there is some problem, I dont know how to solve it.
What could be happening?
Please refer to my RSelenium Installation Guide video. That should help! :)
@@SamerHijjazi I solve it few days ago with that video.
Thx!
Hello! Thanks for the excellent contribution! I used this guide on my personal computer, and everything worked fine. However, on my job computer, I'm encountering an error with the free_port function (it seems to be a language issue, the operating system must be in spanish on my laptop). When creating the "rs_driver_object," it throws the following error:
"Error in gsub("\s+", ",", ActiveConnections) : input string 1 is invalid
In addition: Warning message:
In gsub("\s+", ",", ActiveConnections) :
unable to translate ' Proto Direccin local Direccin remota Estado' to a wide string"
Do you know of any other way to find a free port?
Greetings from Chile!
Hi there! Thank you for the great feedback. To find a list of free ports to use, use the free_ports() function from the net stat package. That will give you a list of all the free ports you can use. Then manually enter the number in place of the port argument followed by an L. So for example, if 21483 is a free port, you want to set the port = 21483L.
I hope this helps!
Salam Samer, Hope to explain how to deal with frames and alerts, etc
Salam moh. Great question. There are several functions that help you deal with alerts. The most common being the acceptAlert() function, which is the same as clicking "OK" on the alert. You can also use the dismissAlert() function to ignore the alert.
As for the frames, if you'd like to switch to the frame, use the switchToFrame() function. I recommend you check out the documentation for both of these. However, I plan to make deeper dive videos into RSelenium.
How can I use a specific chrome profile in selenium? Thanks
I don't believe you can do that in Selenium. At least not from my understanding
Why didn't you explain how to set up selenium?
I made an updated video on that. ruclips.net/video/GnpJujF9dBw/видео.html
I'm having issues getting the Chrome session to start. Every time I run the code, I get the following:
Could not open chrome browser.
Client error message:
Summary: UnknownError
Detail: An unknown server-side error occurred while processing the command.
Further Details: run errorDetails method
Check server log for further details.
I've installed the most recent version of Java jdk but I still get this.
Oh well, guess I won't be using Selenium then lol. Such a shame since this is a great video
Try using Firefox. It still works the same way as Chrome. Also thank you for the wonderful feedback!
@@SamerHijjazi thanks so much for your reply! Your suggestion worked, so glad I don’t need to give up on RSelenium. Thanks so much again!
can you make R selenium for automate youtube?
Possibly! If you're trying to obtain data from RUclips, you can use the RUclips API
what do i do if the chrome version i have does not match up with the versions listed?
this is what it shows me:
> ?rsDriver
> binman::list_versions("chromedriver")
$win32
[1] "113.0.5672.63" "114.0.5735.16"
[3] "114.0.5735.90"
but the version i have of chrome is 124.0.6367.92
thanks in advance!
Could you please help me resolve the following error, I am unable to connect to the server.
I typed the following code:
rD
Hello! This issue has been brought to my attention recently. I have found a solution to your error and I am hoping to make an update video soon. Thank you for the feedback :)
@@SamerHijjazi Hi I am getting the same error too and cannot solve it. An updated video soon would be awesome!