is web login & password and google Authenticator for selenium ? is python create from input for login website page ... result can't load from a selenium
It wouldn't matter, we replace old scraped values with old + new ones each time. Is there a reason you want to start specifically where you left of? (Performance wise it doesn't matter)
Thanks for a great video! Could you tell please, I just dont get it. Why should we update the items list every time instead of appending to it? Because I've tried to see how instagram behaves and it seems like everytime it scrolls down it loads an exact set of items and deletes the previous ones out of the code. Or am i being mistaken?
How about appending element.text directly to items list instead of updating items list with textElements list? Or is it each time Selenium scroll the page, it will scrape all over again all of the previous element.text? If that's the case, what if we use set instead of list to contain the result, so it will be only the unique result we keep?
@@MichaelKitas Ah, I see. I assume they will just scraping the current page after scrolling, but it seems it's not work like that. Thanks for the info.
@@yafethtbThat’s a bad practice, as some pages like Facebook Marketplace never have an ending and by the time they do you ram will overload and you will never get any data
In my use case the first items disappear as new items are loaded, which makes sense for an application to not crash the RAM. In these cases unfortunately this wouldn`t be a solution.
is web login & password and google Authenticator for selenium ? is python create from input for login website page ... result can't load from a selenium
Not sure what you are talking about
I can't find the code for this video in your github link.
Is it possible to start parsing right away?
with the fiftieth element and not start parsing everything again?
It wouldn't matter, we replace old scraped values with old + new ones each time. Is there a reason you want to start specifically where you left of? (Performance wise it doesn't matter)
Hi I am having a doubt! You code works very well, but when I scrap, the data gets scraped from the start after some time. Is there any way for it?
Yeah, you should put an if statement to check if the amount you scraped is the same amount you currently saved, if so then stop the script
@@MichaelKitas Actually it didn't scrap everything. It just scrapes everything from the start again. But I got it solved it. Thanks
Thanks for a great video! Could you tell please, I just dont get it. Why should we update the items list every time instead of appending to it? Because I've tried to see how instagram behaves and it seems like everytime it scrolls down it loads an exact set of items and deletes the previous ones out of the code. Or am i being mistaken?
Because we would have duplicates each time we append since when new items are loaded we also get the old items in there.
How about appending element.text directly to items list instead of updating items list with textElements list? Or is it each time Selenium scroll the page, it will scrape all over again all of the previous element.text? If that's the case, what if we use set instead of list to contain the result, so it will be only the unique result we keep?
It scrapes all over again, correct. You can try set, I am not sure what the difference is 👍
@@MichaelKitas Ah, I see. I assume they will just scraping the current page after scrolling, but it seems it's not work like that. Thanks for the info.
Then it might be better to scroll the page till the end of page and then scraping all the content? By doing this we don't have to updating the list.
@@yafethtbThat’s a bad practice, as some pages like Facebook Marketplace never have an ending and by the time they do you ram will overload and you will never get any data
For some reason my website can't load from a selenium scroll, it just stucks there.
What do you mean? It doesn't scroll?
great video thanks for the help
nah, nothing work. browser just closing before scroll to page 2
It’s not that the method doesn’t work, you either have an error and the browser is crashing or you are closing browser too soon
@@MichaelKitas solved with albums?page=* . Infiniti scrolling have pages
In my use case the first items disappear as new items are loaded, which makes sense for an application to not crash the RAM. In these cases unfortunately this wouldn`t be a solution.
Why not? Just save the items and every time you scrape new items just append them to an array or json file
Did you ever solve this? I have the same problem
thank you