This really shows that RUclips live streaming was an afterthought that was slapped on top of RUclips infrastructure. We already knew this from the user experience of live streaming, but it's aggravated when using their API. What baffles me is the fact that although it's not their money maker, they still do a bunch of deals with 3rd parties, either by pulling streamers exclusivity from other platforms or paying to be the sole platform of an event.
This seems like a recurring theme for Google products. I tried to pull some simple stats from the Google Analytics API to show in our company's dashboard, but kept running into poor or outdated documentation. I understand Google is a big company, and maintaining documentation is a lot of work that doesn't provide as much value as a nice new feature implementation. But like you said in the video, it's the entry point for us devs (and customers) to their service. For a company with that many employees they should be able to do better.
Hi Adam :) I've worked with the RUclips API for many many years. I used to have the old quota of 1 million, which they reduced to 100k and then to 10k once my projects were abandoned. Anything related to the chat or live broadcasting is a sh*thole to work with. And from many years of experience I do not think they will ever change this. It's a very exclusive area for the big companies only running with unlimited quota.
It’s absolutely astonishing how little RUclips seems to care for their live streaming market share. It feels almost like they have already given up on trying to compete with Twitch.
I think a possible workaround would be to manually open a RUclips tab in your browser with the chat showing and then injecting some Javascript code that observes the chat's DOM element for changes. On every change you could then forward that to your own bot and the bot could use the RUclips API to send a response. This would neither fall under scraping as no additional requests are send by the injected Javascript code to RUclips nor does this call any internal APIs.
This is actually the definition of scraping, which is gathering data from a site by parsing it as opposed to via an officially supported API. I think that people tend to read a policy and ascribe a potential reason for it, then gauge whether they're violating that reason rather than the policy itself. In this case, we may think that Google is preventing scraping in an attempt to cut down on API calls, so fetching a page once is okay, right? But maybe Google is trying to make sure that clients can trust that the API won't change overnight, which scraping would violate because Google isn't going to ensure that you can always find DOM elements the same way. I can only guess about Google's full reasoning, but rarely do websites want people scraping data for a myriad of reasons, traffic being just one of them. Also, I believe RUclips actually makes efforts to prevent reading the DOM (e.g. imagine changing the DOM structure or CSS classes occasionally to break unmaintained scraping clients). Of course, the only thing upholding the terms of service is a company's legal team, so the barrier here is legal rather than technical, but I myself try to avoid breaking a ToS whenever possible so that I don't run into issues down the road.
@@AdamLearns Just to clarify, I was not proposing the intermediate script to interact in any way with the RUclips API or any of their servers directly. I was thinking about injecting a purely client-side script that would forward the chat messages directly from your normal RUclips chat window (which you would probably have open anyway) to your bot. This would be 100% invisible from RUclips's point of view as this approach does not generate any network traffic between your computer and RUclips's servers. In summary, the way I understand "web scraping" is the process of "fetching and extracting data from the fetched data". Since this approach does not include the fetching part (as it only processes the data that you receive from *normal* usage of RUclips's services, e.g. having a chat window open), I would not really classify it as "web scraping", thus I would argue it does not break any ToS. Of course, the point that this approach would certainly not be very robust regarding potential DOM changes remains.
I do find it very strange that they are not putting any love into improving this when it's clear that they are spending money elsewhere on exclusivity deals etc to get livestream viewers to their platform.
You should apply at google's documentation team, get hired. Figure out the live streaming rates, update the documentation and hand in your 2 weeks Wowee
Someone give this man a raise!
This really shows that RUclips live streaming was an afterthought that was slapped on top of RUclips infrastructure. We already knew this from the user experience of live streaming, but it's aggravated when using their API.
What baffles me is the fact that although it's not their money maker, they still do a bunch of deals with 3rd parties, either by pulling streamers exclusivity from other platforms or paying to be the sole platform of an event.
This seems like a recurring theme for Google products. I tried to pull some simple stats from the Google Analytics API to show in our company's dashboard, but kept running into poor or outdated documentation. I understand Google is a big company, and maintaining documentation is a lot of work that doesn't provide as much value as a nice new feature implementation. But like you said in the video, it's the entry point for us devs (and customers) to their service. For a company with that many employees they should be able to do better.
Hi Adam :)
I've worked with the RUclips API for many many years. I used to have the old quota of 1 million, which they reduced to 100k and then to 10k once my projects were abandoned. Anything related to the chat or live broadcasting is a sh*thole to work with. And from many years of experience I do not think they will ever change this. It's a very exclusive area for the big companies only running with unlimited quota.
It’s absolutely astonishing how little RUclips seems to care for their live streaming market share. It feels almost like they have already given up on trying to compete with Twitch.
I think a possible workaround would be to manually open a RUclips tab in your browser with the chat showing and then injecting some Javascript code that observes the chat's DOM element for changes. On every change you could then forward that to your own bot and the bot could use the RUclips API to send a response. This would neither fall under scraping as no additional requests are send by the injected Javascript code to RUclips nor does this call any internal APIs.
This is actually the definition of scraping, which is gathering data from a site by parsing it as opposed to via an officially supported API. I think that people tend to read a policy and ascribe a potential reason for it, then gauge whether they're violating that reason rather than the policy itself. In this case, we may think that Google is preventing scraping in an attempt to cut down on API calls, so fetching a page once is okay, right? But maybe Google is trying to make sure that clients can trust that the API won't change overnight, which scraping would violate because Google isn't going to ensure that you can always find DOM elements the same way.
I can only guess about Google's full reasoning, but rarely do websites want people scraping data for a myriad of reasons, traffic being just one of them. Also, I believe RUclips actually makes efforts to prevent reading the DOM (e.g. imagine changing the DOM structure or CSS classes occasionally to break unmaintained scraping clients).
Of course, the only thing upholding the terms of service is a company's legal team, so the barrier here is legal rather than technical, but I myself try to avoid breaking a ToS whenever possible so that I don't run into issues down the road.
@@AdamLearns Just to clarify, I was not proposing the intermediate script to interact in any way with the RUclips API or any of their servers directly. I was thinking about injecting a purely client-side script that would forward the chat messages directly from your normal RUclips chat window (which you would probably have open anyway) to your bot. This would be 100% invisible from RUclips's point of view as this approach does not generate any network traffic between your computer and RUclips's servers.
In summary, the way I understand "web scraping" is the process of "fetching and extracting data from the fetched data". Since this approach does not include the fetching part (as it only processes the data that you receive from *normal* usage of RUclips's services, e.g. having a chat window open), I would not really classify it as "web scraping", thus I would argue it does not break any ToS.
Of course, the point that this approach would certainly not be very robust regarding potential DOM changes remains.
I do find it very strange that they are not putting any love into improving this when it's clear that they are spending money elsewhere on exclusivity deals etc to get livestream viewers to their platform.
You should apply at google's documentation team, get hired. Figure out the live streaming rates, update the documentation and hand in your 2 weeks Wowee