Hussein: Oh gee why are preflights options not cached! Everyone: It helps if you don’t disable the cache Hussein! Thanks everyone for the good catch. No bug here just user error
Hello Hussein, Engineer from Spotify here currently trying to improve our web player performance. Good video, you have some good points that I can share around here. BTW, you can disable the service worker by going to Application > Service Workers > Bypass for network. That way, any asset that should be served by it won't and you'll see in the network tab the full request for the asset.
Cool, how does it feel to work on something that has so large user base? Work like every other work or does it feel special? PS when improving, please keep end users in mind, not just sales people. If you let sales people and stakeholders have everything they want then it gonna suck for end users, if you ignore them to much you get replaced of course so yea, probably not a easy task.
@@test-rj2vl On the firstd part, it is very special for me cause I'm a user first. Have been using the app since the early days. On the second, the more years of development you have the more you understand that is a lot more nuanced than you first thought and most of the time, in the end, it is all about the ammount of compromises and tradeoffs both sides can have to make the best for the company in general. You as a developer always want to ship the best version out there, but companies are a more complex beast.
I wanted to ask about how many users do actually use the web player? Cause I think most traffic is b/w mobile app and desktop app. Would be great if you could answer this
@@charan775 Why would I use app if web version works? If I were to download app for everything that has app then my computer would soon be full of bloatware to the point where clean install of windows is the only thing that can make my computer fast again. If they did not have web interface it would either be deal breaker for me or the app would run in some kind of sandbox that would isolate it from the rest of system - specially after seeing from this video how big data miner it is.
Are you under the illusion that big tech company have amazing architected apps ? Their stuff are so big it's impossible for them to be perfect. Also they must have to deal with a lot of legacy code.
Do netflix next please, I live in an area where my internet speed is super low but I get HD videos without any buffering and the thing thats really interesting is that no other website can give the type of quality they do (For contrast the standard minimum bandwidth need for streaming an SD video is 3 Mbs and I have 2Mbs). They probably have some black magic going on in the back lol
Just saw clicked on this video, watched 2 mins and looked up the difference between http1.1,2,3 on youtube (because you mentioned it, and I was unsure and wanted to learn more), and found another one of your videos explaining it, in that video you mentioned tcp vs. udp, so I ended up also watching your video on how to choose tcp vs. udp when making an app, which explained the difference very well and different applications and uses, and then I ended up watching more videos on your channel (log4j vid and some incident reports) and can say that I am now a lot more interested in networking and how backend engineers interact with it! I love your way of explaining things, it makes me inspired to explain the way you do.
The "graphql-y" request they're making is actually just regular graphql. Graphql servers allow you to send request parameters that would otherwise be a POST body in the url itself with a GET request to allow caching layers in between the user and the origin server to kick in. This is usually done per-query depending on which query is deemed cache-safe according to the operationName.
Great episode. I learned a lot. As a firefox user, I noticed some difference in the request chain on my dev tools. The amount of tracking is mindblowing. It will be interesting to see a comparison to based on different user-agent or different browser. I know few years ago, that youtube on firefox ( which is slow because of shadow-dom polyfil ) was faster if I changed my user-agent to mimic chrome.
I'm building a working Spotify clone but without the tracking. Learning as I go 😉 Holy moly the amount of requests 😱🤣 I just can't get the device syncing to work properly yet.
@@trsh404 You're right. I figured something was wrong, it made me feel a bit like a failure even though I'm objectively in a better financial-situation than most of those in my age group. But hey, all power to 23 year old Hussein 😅
@@abdelrahmanhafez990 Hah yea np, I replied because I even researched it myself in disbelief that he would be one month younger than me. Not sure what his age is, but he definitely has the spirit of a 23 year-old :)
@@trsh404 On his Linkedin it says he went to college from 2001-2005 so if we assume that he went at 16 (youngest possible age usually), then he has to be born in 1985, so he has to be 36/37 today at least. Realistically i'm guessing he is late thirties/early forties.
I loved this video series! Can I make a suggestion? You should make a video about alternative front-ends, more precisely about Nitter, which is alternative/better front-end to Twitter than the default one, way lighter and faster to load. I think it would be interesting a comparison between both.
What is the name of the procedure where he addresses 24:30. I would like to know more about the fixed "offset" procedure. Is there a specific term for it?
10:29 307 “Internal Redirect” means that the browser is saying “I’m not even going to issue that request, instead I’m going to change it to HTTPS then try again”. This is due to the Strict-Transport-Security response header or as we know it, HSTS (HTTP Strict Transport Security).
This video is a proof that we need some kind of privacy related browser addons. Sadly for average spotfy user telling to facebook, google and everyone else what you do on spotify is rather feature than privacy issue.
FB and Google, no positive outcome of tracking. For Spotify though it helps me discover new music and it's actually my favourite part. I'm very privacy-conscious, but for Spotify it's an actual feature
CORS doesn't block GET and POST requests (with some headers, it's too complicated to explain in a youtube message section), it only blocks the response. If your request has a side effect, it will happen on the server but your browser will not return a response.
Question about third party cookies, specifically when it comes to mobile apps. If someone is logged into the Snapchat app and the Spotify app, when Spotify sends the request to Snapchat with the third party cookies, does that go through? I understand if you were logged into both via the same web browser it would work. I guess this depends on how separated apps are on mobile. Does anyone know the answer?
Good question, I don’t know the answer to that, hopefully a iOS/Andriod mobile developer can help shed light. My guess if cookies are not sent on the mobile apps there must be another form of tracking between apps (unless you ask the app to not track new to IOS) .its just too valuable to miss on.
Every phone has a unique device ID think of it like all the apps are sharing the same cookie data, so if spotify is doing the same thing they did on the web app in thier mobile app they are probably sending the unique ID + some data to snapchat or other 3rd party trackers
If you use Spotify as your music service on desktop, but think the official client is too bloated, you may want to try ncspot, available on GitHub. It's a minimal TUI program with a very small memory footprint, ideal for listening music while you work.
I could understand that if you're using a Raspberry Pi or a 1996 desktop with Windows B.C. on it, but in todays age the amount of RAM spotify uses (like 200MB) really isn't something you need to download some random ish of GitHub to address. It's low enough it really doesn't matter...
@@AggroAceOfficial some people dispise Electron apps out of principle, which I don't fully agree with but can understand. Personally I find the programmable shortcuts of ncspot and its lightning-fast launch times very appealing. It's the same reason why some users prefer to do simple tasks in the terminal as opposed to the GUI. Why would I need a large feature-rich client, when all I do is play some random song in the background?
@@紺野-純子 yes it does. I believe it's technically possible (I'd say even trivial), but it's against Spotify's TOS which is why you can't actually find any.
Yea, we could use wireshark other interceptors to chanage the response right? So we can chnage the playable attribute 😯 . Maybe they have some kinda checks on server before playing. Btw, another great episode... ❤️
Update: They fixed the plain http redirect. While watching the video I was wondering the purpose of having the same exact redict(http first and then https) 🤔
How could you tell whether or not they sent your cookie over HTTP? We don't actually send cookies right, we just send String data as JSON or XML which is then placed into browser's cookie once received
It looks to me like it didn't actually get sent, the 307 internal redirect is likely because of HSTS. I think the browser correctly internally redirected the url to use https, and we can see a 0ms time at 11:00 . Still a very bad practise of redirecting something coming from https to http on the same domain, if it were my system any redirect to the same domain will keep the original protocol intact (unless deliberately modifying it for a specific reason). We can see that the request method is GET, which does not (should not) have a request body so all the data sent can be seen in the headers
Do one on Deezer and how their api was available for anyone with a free account to download music for free for so many years. Even in lossless quality. Now they have fixed that
by the way, i noticed spotify has already disabled access dev tools using command and right clicking on the web player. but there's another another way to access it
Unlikely this will ever happen but if a certain ISP charges servers instead of the customer for the amount of data transferred, and will not let you visit a website unless the server's company agrees to the deal, then we may see these tech companies trying to minify their impact on the internet traffic.
This dude doesn't know what he is doing. So unprofessional, unknowledged. Why not use adblockers and privacy blockers (to filter out all the ad/tracking stuff?) Why not use react, angular, redux etc. dev tools extensions? Why not filter by type (fetch/xhr)? Why not show "payload" tab of the requests? CORS are url specific, and since theyre passing parameters of the operation in the URL, the options are not "cached" like you said they would be.
The structure of large complex things often has more to do with the structure of the company than the quality of the engineers. Spotify engineers are pretty rock solid
Hussein: Oh gee why are preflights options not cached!
Everyone: It helps if you don’t disable the cache Hussein!
Thanks everyone for the good catch. No bug here just user error
Hello Hussein, Engineer from Spotify here currently trying to improve our web player performance. Good video, you have some good points that I can share around here. BTW, you can disable the service worker by going to Application > Service Workers > Bypass for network. That way, any asset that should be served by it won't and you'll see in the network tab the full request for the asset.
Cool, how does it feel to work on something that has so large user base? Work like every other work or does it feel special?
PS when improving, please keep end users in mind, not just sales people. If you let sales people and stakeholders have everything they want then it gonna suck for end users, if you ignore them to much you get replaced of course so yea, probably not a easy task.
@@test-rj2vl On the firstd part, it is very special for me cause I'm a user first. Have been using the app since the early days. On the second, the more years of development you have the more you understand that is a lot more nuanced than you first thought and most of the time, in the end, it is all about the ammount of compromises and tradeoffs both sides can have to make the best for the company in general. You as a developer always want to ship the best version out there, but companies are a more complex beast.
I wanted to ask about how many users do actually use the web player? Cause I think most traffic is b/w mobile app and desktop app. Would be great if you could answer this
@@charan775 Why would I use app if web version works? If I were to download app for everything that has app then my computer would soon be full of bloatware to the point where clean install of windows is the only thing that can make my computer fast again. If they did not have web interface it would either be deal breaker for me or the app would run in some kind of sandbox that would isolate it from the rest of system - specially after seeing from this video how big data miner it is.
@@test-rj2vl I always felt that the app is really smooth and better than the web player
After seeing how Spotify web is designed, i feel great about my architectural skills
😂
Me too it's a great confidence booster 😂😂😂
LOL yeah so true, this whole series, other than being super interesting it also gives me a confidence boost
it's actually designed really good compared to other modern websites
Are you under the illusion that big tech company have amazing architected apps ? Their stuff are so big it's impossible for them to be perfect. Also they must have to deal with a lot of legacy code.
Do netflix next please, I live in an area where my internet speed is super low but I get HD videos without any buffering and the thing thats really interesting is that no other website can give the type of quality they do (For contrast the standard minimum bandwidth need for streaming an SD video is 3 Mbs and I have 2Mbs). They probably have some black magic going on in the back lol
That would be interesting
Isn't that the purpose of CDN?
@@RamyTamer At their scale it might be more interesting than that?
@@RamyTamer more fficient streaming protocols
If I remember correctly, Netflix playbacks videos from your ISP. They have like boxes installed or something
Just saw clicked on this video, watched 2 mins and looked up the difference between http1.1,2,3 on youtube (because you mentioned it, and I was unsure and wanted to learn more), and found another one of your videos explaining it, in that video you mentioned tcp vs. udp, so I ended up also watching your video on how to choose tcp vs. udp when making an app, which explained the difference very well and different applications and uses, and then I ended up watching more videos on your channel (log4j vid and some incident reports) and can say that I am now a lot more interested in networking and how backend engineers interact with it! I love your way of explaining things, it makes me inspired to explain the way you do.
The "graphql-y" request they're making is actually just regular graphql. Graphql servers allow you to send request parameters that would otherwise be a POST body in the url itself with a GET request to allow caching layers in between the user and the origin server to kick in. This is usually done per-query depending on which query is deemed cache-safe according to the operationName.
does graphql accept POST with urlencoded body ?
Great episode. I learned a lot. As a firefox user, I noticed some difference in the request chain on my dev tools. The amount of tracking is mindblowing. It will be interesting to see a comparison to based on different user-agent or different browser. I know few years ago, that youtube on firefox ( which is slow because of shadow-dom polyfil ) was faster if I changed my user-agent to mimic chrome.
Thoroughly enjoyed my day break with popcorn and this movie. Thank you for keeping it free.
After 3 hours of coding. There is no better break than this. Lets go
I love that you’re only 23 and as passionate about proper execution under the hood as someone nearly twice your age.
I fell for it when you said you were 23 and got shocked about it ahaha
i really enjoyed this , cant believe i watched all 30mins 🙌🏾
It helps if you set video playback speed to 2x
you can use Ctrl+Shift+R to reload the page and miss the browser's cache
I'm building a working Spotify clone but without the tracking.
Learning as I go 😉
Holy moly the amount of requests 😱🤣
I just can't get the device syncing to work properly yet.
Hussein is 23
Way to go, man! Your knowledge is what I'm hoping to have by the time I'm 33.
Also, one-month-early happy birthday 🎉🎉
On his website it says he received his first book on software development in 1998. He's not 23, he just used a fake birthday and was making a joke lol
@@trsh404
You're right. I figured something was wrong, it made me feel a bit like a failure even though I'm objectively in a better financial-situation than most of those in my age group.
But hey, all power to 23 year old Hussein 😅
@@abdelrahmanhafez990 Hah yea np, I replied because I even researched it myself in disbelief that he would be one month younger than me. Not sure what his age is, but he definitely has the spirit of a 23 year-old :)
@@trsh404 On his Linkedin it says he went to college from 2001-2005 so if we assume that he went at 16 (youngest possible age usually), then he has to be born in 1985, so he has to be 36/37 today at least. Realistically i'm guessing he is late thirties/early forties.
I loved this video series! Can I make a suggestion? You should make a video about alternative front-ends, more precisely about Nitter, which is alternative/better front-end to Twitter than the default one, way lighter and faster to load. I think it would be interesting a comparison between both.
What is the name of the procedure where he addresses 24:30. I would like to know more about the fixed "offset" procedure. Is there a specific term for it?
You could use an autoincrementing id, then select all rows that are larger than that id.
Options Issue: You disabled the cache 😅
😱 user error, forgot I had that box checked. Good catch
@@hnasr I have done the same before. 😂
10:29 307 “Internal Redirect” means that the browser is saying “I’m not even going to issue that request, instead I’m going to change it to HTTPS then try again”. This is due to the Strict-Transport-Security response header or as we know it, HSTS (HTTP Strict Transport Security).
This video is a proof that we need some kind of privacy related browser addons. Sadly for average spotfy user telling to facebook, google and everyone else what you do on spotify is rather feature than privacy issue.
FB and Google, no positive outcome of tracking. For Spotify though it helps me discover new music and it's actually my favourite part. I'm very privacy-conscious, but for Spotify it's an actual feature
At 3:50, shouldn't this be blocked by CORS? Did all these websites disable CORS, or gave Spotify special permission?
CORS doesn't block GET and POST requests (with some headers, it's too complicated to explain in a youtube message section), it only blocks the response. If your request has a side effect, it will happen on the server but your browser will not return a response.
Cors max age is ignored when disable cache is on. It's not a bug.
Good catch! I forgot I had that box checked
Third party cookies doesn’t work for years anymore.
It’s blocked by default for a long time
Question about third party cookies, specifically when it comes to mobile apps. If someone is logged into the Snapchat app and the Spotify app, when Spotify sends the request to Snapchat with the third party cookies, does that go through? I understand if you were logged into both via the same web browser it would work. I guess this depends on how separated apps are on mobile. Does anyone know the answer?
+1
On the same browser it would depend on CORS configuration. Regarding mobile apps: i don't know how they work
Good question, I don’t know the answer to that, hopefully a iOS/Andriod mobile developer can help shed light.
My guess if cookies are not sent on the mobile apps there must be another form of tracking between apps (unless you ask the app to not track new to IOS) .its just too valuable to miss on.
Every phone has a unique device ID think of it like all the apps are sharing the same cookie data, so if spotify is doing the same thing they did on the web app in thier mobile app they are probably sending the unique ID + some data to snapchat or other 3rd party trackers
what program do you use to zoom like that?
died laughing at “this is my birthdate, as u can see i’m excited to finish college soon”
7:40 They aren't using the IP even. Just the OS or Browser Language apparently.
Could you do Office online and/or Teams please. I've always felt they are very bloated, especially Office Online
yeah, office online felt bloated compared to google doc
Teams is an electron app, Chromium but watered down
Bruh how are you so good at this at 23 god dam goals
This is such a great learning resource
Just a tip u can reset cache and browser data by right clicking on refresh button when developer console is opened
Hussein if you click on the request and go to the timing tab you will get the waterfall
If you use Spotify as your music service on desktop, but think the official client is too bloated, you may want to try ncspot, available on GitHub. It's a minimal TUI program with a very small memory footprint, ideal for listening music while you work.
I could understand that if you're using a Raspberry Pi or a 1996 desktop with Windows B.C. on it, but in todays age the amount of RAM spotify uses (like 200MB) really isn't something you need to download some random ish of GitHub to address. It's low enough it really doesn't matter...
@@AggroAceOfficial not even 200mb xD
@@AggroAceOfficial some people dispise Electron apps out of principle, which I don't fully agree with but can understand.
Personally I find the programmable shortcuts of ncspot and its lightning-fast launch times very appealing. It's the same reason why some users prefer to do simple tasks in the terminal as opposed to the GUI. Why would I need a large feature-rich client, when all I do is play some random song in the background?
does it need a premium account?
I have yet to see a custom spotify client that doesn't need a premium account.
@@紺野-純子 yes it does. I believe it's technically possible (I'd say even trivial), but it's against Spotify's TOS which is why you can't actually find any.
Love this series learn a lot of cool stuff
What extension do you use for nice coloring of text in devtools?
I work on WhatsApp Web and would love if you did a video on that!
Yea, we could use wireshark other interceptors to chanage the response right? So we can chnage the playable attribute 😯 .
Maybe they have some kinda checks on server before playing.
Btw, another great episode... ❤️
Using wireshark won't help, since it's encrypted.
Is it possible to do this kind of discovery with apps (android/ios)?
I'm quite curious about how spotify mobile works
National Grid, for some reason, has requests to facebook and google ads.
what you are 23?
He meant that he became 23 in 1999
of course it's a joke
Update: They fixed the plain http redirect. While watching the video I was wondering the purpose of having the same exact redict(http first and then https) 🤔
Man you are awesome, you appreciate the good stuff and don't even flinch while criticizing 🤟
Netflix?
9:22 Checked and maybe after your video they made redirect to HTTPS :)
How could you tell whether or not they sent your cookie over HTTP? We don't actually send cookies right, we just send String data as JSON or XML which is then placed into browser's cookie once received
It looks to me like it didn't actually get sent, the 307 internal redirect is likely because of HSTS. I think the browser correctly internally redirected the url to use https, and we can see a 0ms time at 11:00 . Still a very bad practise of redirecting something coming from https to http on the same domain, if it were my system any redirect to the same domain will keep the original protocol intact (unless deliberately modifying it for a specific reason). We can see that the request method is GET, which does not (should not) have a request body so all the data sent can be seen in the headers
congratz on finishing college
This was great to learn 🙌 thanks
Do one on Deezer and how their api was available for anyone with a free account to download music for free for so many years. Even in lossless quality. Now they have fixed that
Can you do Onshape next? This devtooling series is really cool.
I have never thought that you were younger than me. Already having a family.
would be really awesome if this can be compared with youtube music and other services......
شكرا على هذه الفيديو
really good content thanks man
the spotify desktop app is chromium based so there is a fair chance that it also has a lot of these problems
Spotify prob uses S3 for audio storage and serves it via akami cdn
Yet I have to use the web app because the windows app is literally crashing my entire PC...
by the way, i noticed spotify has already disabled access dev tools using command and right clicking on the web player. but there's another another way to access it
This is very nicee!
Amazing content!
can you devtool google meet sometime?
Please do IMDbs website it’s so so very bad
Spotify has a poggers api tho
Love it
Man you are 23🙀 I am also 23 and still a beginner in all these things. Wish to be as knowledgeable like you some day
I think it was a joke..
wow this guy is just 23, damn looks so mature....
he is not
Unless there's some way that the company is taxed for its laziness and not its consumers, such lazy engineering will continue to flourish.
Unlikely this will ever happen but if a certain ISP charges servers instead of the customer for the amount of data transferred, and will not let you visit a website unless the server's company agrees to the deal, then we may see these tech companies trying to minify their impact on the internet traffic.
I would watch this but it is way to long for me to care
dude, they make us put our card details on the website and they are using http.... holy.... shit
Most sites are very bloated....
What business does that Spyware TikTok have in Spotify web? Or Snapchat? Oh right, more data to grab
So many people are fooled about his age haha
analytics facebook, tiktok and google 💀
*Facebook, TikTok, Google, Twitter, Snapchat
@@itsTyrion TikTok*
@@joopie46614 ffs I can’t spell
@@itsTyrion 😉
This dude doesn't know what he is doing. So unprofessional, unknowledged. Why not use adblockers and privacy blockers (to filter out all the ad/tracking stuff?) Why not use react, angular, redux etc. dev tools extensions? Why not filter by type (fetch/xhr)? Why not show "payload" tab of the requests? CORS are url specific, and since theyre passing parameters of the operation in the URL, the options are not "cached" like you said they would be.
Spotify -web- is bloated
Great video as usual :) Thank you!
Hussein, is this how OData is implemented here ruclips.net/video/NzKcLB0_uAY/видео.html?
Thanks for showing how incompetent the engineers at Spotify are.
The structure of large complex things often has more to do with the structure of the company than the quality of the engineers. Spotify engineers are pretty rock solid
dude not chill, im sure the engineers do their best within all the messiness of the company
sweet 1st
Spotify overrall is trash
The web player absolutely horrible
eww a mac user