Thanks for posting this. Its important to learn from what goes wrong as well as what goes right. As a dev its really frustrating to have stuff fail when paid for infrastructure breaks from underneath you.
Lol 😂😂😂 just... couple of days ago you made a video on how to scale backend with AWS and today this video pops out can't believe. But very informative video 👍👍.
I enjoy your content about architecture & the back-end so much. Even though I am a front-end dev, I sit down grab coffee & snacks and watch your videos. Gives me a bigger picture of what I am doing. Knowledgeable & fun at the same time.
I watched till the End ... Keep posting more Backend Videos I like them and in a Process to build my Own Startup ... So it helps me a lot .. I am in 12th but Aspiration of having more knowledge drives me to watch every video of yours till the end ... Thanks Mehul Bhaiya it's amazing to learn from you ...
We have a multi region active-active setup. It seemed like our west users were unaffected. I’m not sure if saying cloudfront went down because our users could still view the Ui, but yeah maybe deploying new changes during this outage was not working as expected which brought down your ui
you're right, cloudfront was not fully down. It would have kept working fine for us too if only a deployment change would have completely crashed/not propagated. But for some reason, a new deployment did not propagate properly and at the same time the old deployment was invalidated successfully. After this, cloudfront stopped incremental static regeneration for some unknown reason (lambda@edge failing with NoSuchKey errors), and no further deployment was possible for a very long time. I do believe this would mean certain parts of cloudfront were down.
@@codedamn curious, did you happen to have multiple origins setup? I’m assuming you had an s3 bucket containing you UI hosted on east, but did you also have another failover origin coiffured?
There are tons of videos over the internet about how to learn frontend and how to learn backend. But there is only fews resources that shows how actually to put projects on production and which types of issues will be there etc. Thanks for sharing this type of real problems.
Once you move to a more “diy” architecture, you gain flexibility but definitely open yourself up to risk. I would suggest looking at infrastructure as code (terraform) etc for easily migrating from region to region
I watched this video till the end. It's good to know what's happening in the backend and what are the options we have to solve this. Yeah, it's interesting and helpful.
I would suggest to use terraform to setup the infrastructure items, to allow setting up a completely new system in a matter of minutes - in a different region
@@codedamn In that case, I'd suggest having IaaC (using Terraform) setup in a way that allows switching to GCP (that's probably the most underrated cloud provider ATM) - playing around with that from some time and it is pretty promising when it comes to global outages like this. Automating more than that would probably come under "paranoid-SRE" (in a good way though)
@@JagdeepSinghKalsi i haven’t done a multi provider setup using terraform. Is it pretty one-to-one or would it require a lot of refactoring to get your aws code deployed to GcP? It sounds like a lot of work for a smaller business with limited resources.
Hii sir, I have created two separate next js project one for desktop and another for mobile. Now I want to load specific website based on user-agent, can you please guide me how to configure this on server like vercel. Thanks sir
I'm begining to think that I ever think up a good saas idea, that I would need to have my softwarwe running on multiple platforms simultaneiously to make sure it doesn't go down and customers don't get pissed and leave. Like have one copy running on vercel and another on netlify or something like that.
Wow , who would have thought that you would ever need a backup (Plan). It's an Optimistic World out there , a very very trusting one , and therein opens up a world of potential (difficulty)
This a great video and your learnings are put together in a nice way. Just a thought what about using a simple http server for hosting the static files as a fallback in any of the existing backend servers to avoid additional infrastructure cost. Of course need to compromise on performance during the fallback as it is going to be temporary.
The thing is that if you are not following the 5 pillars of AWS which literally is telling you to deploy to multiple regions/zones/edges whatever you can, depending on the scenario ( if that's not your plan, then you are better off using some VPS services like DO, Vultr, Linode etc..). The main idea of AWS is to go global in minutes and automatically recover from failure and you should've setup a disaster recovery plan. The thing you mentioned that you don't want to create a EC2 instance in some other Region is no-brainier, since Vercel actually is doing the same to you, they are hosting on AWS so us-east-1 is down for them as well, so they are hosting your website on some other region as well, the only difference is that they have that automated, and it's pretty easy to set it up. Why would you not want to create a new instance in another region? It could be on-demand instance so you will not be billed if it's not used at the moment. Maybe your next video should be a lesson learned and how to setup AWS the correct way :) Btw. no offense, I enjoy watching your videos and have learned a ton of stuff, but this is just a recommendation! :)
I think most people missed the point in the video. We had the capacity to recover the backend very quickly. The problem was that cloudfront deployment went down. Cloudfront is a global service by AWS having something critical in us-east-1. Therefore it doesn’t matter if you were multi region because if you triggered a cloudfront deployment update yesterday your website would go down. To avoid total failure for yesterday, the ONLY solution was multi-cloud. Not multi region
@@codedamn That's true, because they need to instantly purge the cache for incremental static generation. Which in this case saved them from being down. My point was, that if you didn't started the rebuild of the website, but leave it as it is, since it was not part impact, but instead just rebuild the backend on different region, you would've had the issue that you had.
Hello Codedamn- I see there is a video of yours on 5 Dec 2021 (Bye Vercel , Hello AWS-ruclips.net/video/Do92vKNr3Og/видео.html). And after 3 days there is another "Bye AWS, Hello Vercel". I am bit confused, please clarify or summarize both videos. I really appreciate your tutorials
I did, and our company account on vercel is still frozen due to billing issues - I moved the codedamn deployment for 12 hours on my hobby account while AWS fixed the cloudfront issues.
![Highly Scalable Backend with AWS + MongoDB [Architecture video]](http://i.ytimg.com/vi/Phg2ezNp-v0/mqdefault.jpg)
![Highly Scalable Backend with AWS + MongoDB [Architecture video]](/img/tr.png)
![Seungmin "그렇게, 천천히, 우리(As we are)" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/kAzmhLHePqU/mqdefault.jpg)
You deserve a very big appreciation for making this video for us even after so much frustration.
Thanks for posting this. Its important to learn from what goes wrong as well as what goes right. As a dev its really frustrating to have stuff fail when paid for infrastructure breaks from underneath you.
Lol 😂😂😂 just... couple of days ago you made a video on how to scale backend with AWS and today this video pops out can't believe. But very informative video 👍👍.
🤣🤣🤣🤣 that was epic
spain without s
Thanks for the video. These are the struggles that come with building a business. Lessons learned. Thanks for sincerity and transparency.
You just told what I wanted to convey. Seriously the transparency is highly appreciated!!
I enjoy your content about architecture & the back-end so much. Even though I am a front-end dev, I sit down grab coffee & snacks and watch your videos. Gives me a bigger picture of what I am doing. Knowledgeable & fun at the same time.
how things have gone in just 3 days
I watched till the End ... Keep posting more Backend Videos I like them and in a Process to build my Own Startup ... So it helps me a lot .. I am in 12th but Aspiration of having more knowledge drives me to watch every video of yours till the end ... Thanks Mehul Bhaiya it's amazing to learn from you ...
We have a multi region active-active setup. It seemed like our west users were unaffected. I’m not sure if saying cloudfront went down because our users could still view the Ui, but yeah maybe deploying new changes during this outage was not working as expected which brought down your ui
you're right, cloudfront was not fully down. It would have kept working fine for us too if only a deployment change would have completely crashed/not propagated. But for some reason, a new deployment did not propagate properly and at the same time the old deployment was invalidated successfully. After this, cloudfront stopped incremental static regeneration for some unknown reason (lambda@edge failing with NoSuchKey errors), and no further deployment was possible for a very long time. I do believe this would mean certain parts of cloudfront were down.
@@codedamn curious, did you happen to have multiple origins setup? I’m assuming you had an s3 bucket containing you UI hosted on east, but did you also have another failover origin coiffured?
I was thinking about codedamn last night when us-east-1 went down 😂. Thanks for posting it though.
Thank you for your transparency Mehul Much respect for you and the team
AWS has been having too many outages in the past 3-4 years. I have shifted many loads to GCP
When Mehul says, that "Something happend, I am still not sure what this reason was."
You know it's a big issue.
Bye Vercel, Hello AWS
bye AWS, Hello Vercel... OP 😂😂
and now back to AWS 😂
There are tons of videos over the internet about how to learn frontend and how to learn backend. But there is only fews resources that shows how actually to put projects on production and which types of issues will be there etc. Thanks for sharing this type of real problems.
Was it Cloud Front? CLoud front needs to get data from Route 53 as you changed domains.
Once you move to a more “diy” architecture, you gain flexibility but definitely open yourself up to risk. I would suggest looking at infrastructure as code (terraform) etc for easily migrating from region to region
What about creating cloud formation templatates of your infra to make the switches easier? Wont that work?
I watched this video till the end. It's good to know what's happening in the backend and what are the options we have to solve this.
Yeah, it's interesting and helpful.
Why don't you use azure or gcp??
They can and have failed as well - multi region would have been helpful
Why not netlify ??
Thanks Mehul for making this video
I'll got to know about lots of things
🤗🤗🤗
Watched till the end. Thanks for all the info.
I would suggest to use terraform to setup the infrastructure items, to allow setting up a completely new system in a matter of minutes - in a different region
It’s not about IaaC. We already do that. But when global services like cloudfront are facing problems your multi region effort wouldn’t work
@@codedamn In that case, I'd suggest having IaaC (using Terraform) setup in a way that allows switching to GCP (that's probably the most underrated cloud provider ATM) - playing around with that from some time and it is pretty promising when it comes to global outages like this.
Automating more than that would probably come under "paranoid-SRE" (in a good way though)
@@JagdeepSinghKalsi i haven’t done a multi provider setup using terraform. Is it pretty one-to-one or would it require a lot of refactoring to get your aws code deployed to GcP? It sounds like a lot of work for a smaller business with limited resources.
I am loving your recent videos
Hii sir, I have created two separate next js project one for desktop and another for mobile. Now I want to load specific website based on user-agent, can you please guide me how to configure this on server like vercel. Thanks sir
Hey Mehul, can you make a video about how you guys setup lambda functions for nextjs without using amplify?
we use serverless-nextjs, I'll explain the architecture as a video soon here
Great Video!
All your videos are very informative. 💯
Man the thumbnail is 🥵
Your videos are really highly practical .
You are superb sir
Great. I am impressed. I am buying full stack course from codedamn
Thanks for posting this when you are frustrated, relay like your videos\
Your'e such a gem big fan sir💯🔥🙏
Vercel: 'look who came crawling back...'
I'm begining to think that I ever think up a good saas idea, that I would need to have my softwarwe running on multiple platforms simultaneiously to make sure it doesn't go down and customers don't get pissed and leave. Like have one copy running on vercel and another on netlify or something like that.
Why did you choose US data centers over others
Wow , who would have thought that you would ever need a backup (Plan). It's an Optimistic World out there , a very very trusting one , and therein opens up a world of potential (difficulty)
This a great video and your learnings are put together in a nice way. Just a thought what about using a simple http server for hosting the static files as a fallback in any of the existing backend servers to avoid additional infrastructure cost. Of course need to compromise on performance during the fallback as it is going to be temporary.
Just 3 days ago you uploaded Bye Vercel, Hello AWS 🥺🙂
curious to know why aren't you using aws Bombay zone? most of your customers are from India right
no
Thanks for sharing the experience.
Video on what is AWS CloudFront, please?
Mutli-Cloud DNS is the way to go. Check the recent HN discussion regarding AWS downtime, it's a goldmine
Link?
Next Video, Migrating To GCP :)
Very Informative Video ❤
Why have you hosted it on US based servers aren't there servers in Asia closer to India?
we get global traffic, not just from India. And US servers seems to have the best bandwidth rates/network capacities
@@codedamn oh that's good. I thought you were an Indian startup so you'd get most of the traffic from India.
What a turn of events.
The thing is that if you are not following the 5 pillars of AWS which literally is telling you to deploy to multiple regions/zones/edges whatever you can, depending on the scenario ( if that's not your plan, then you are better off using some VPS services like DO, Vultr, Linode etc..). The main idea of AWS is to go global in minutes and automatically recover from failure and you should've setup a disaster recovery plan. The thing you mentioned that you don't want to create a EC2 instance in some other Region is no-brainier, since Vercel actually is doing the same to you, they are hosting on AWS so us-east-1 is down for them as well, so they are hosting your website on some other region as well, the only difference is that they have that automated, and it's pretty easy to set it up. Why would you not want to create a new instance in another region? It could be on-demand instance so you will not be billed if it's not used at the moment.
Maybe your next video should be a lesson learned and how to setup AWS the correct way :)
Btw. no offense, I enjoy watching your videos and have learned a ton of stuff, but this is just a recommendation! :)
I think most people missed the point in the video. We had the capacity to recover the backend very quickly. The problem was that cloudfront deployment went down. Cloudfront is a global service by AWS having something critical in us-east-1. Therefore it doesn’t matter if you were multi region because if you triggered a cloudfront deployment update yesterday your website would go down. To avoid total failure for yesterday, the ONLY solution was multi-cloud. Not multi region
Vercel is NOT using AWS Cloudfront - the entry point for a Nextjs website. They have their own CDNs. We don’t use vercel for Backends (then and now)
@@codedamn That's true, because they need to instantly purge the cache for incremental static generation. Which in this case saved them from being down.
My point was, that if you didn't started the rebuild of the website, but leave it as it is, since it was not part impact, but instead just rebuild the backend on different region, you would've had the issue that you had.
😂😂😂great chain of contents from you
Till the end as always
Does your company has sponsors?
Thanks for sharing. Bro..awesome
Thanks for this video!
Well well well, how the turn tables.
Respect for you and your team mehul.
Lol, request to make this a series now Mehul 😄
8:00 lol moment🤣 can't stop laughing, jokes apart, great insights from every bit of video 🔥
I WATCHED THIS VIDEO TILL THE END!
I watched this video till the end 👍
thank you for the info
When US cloud services rain
Indian businesses get floods
Lol, I just saw your video about moving from vercel to aws and now you’re moving back to vercel? So confusing lol
Is this what they call irony ?
Appreciated
Seems like you are having a love-hate relationship with vercel
love the platform hate the pricing beyond 1TB bandwidth
😂😂👌 main twist after mid of the video he himself laughs 😅
Watched till end
I watched this video till the end.
It was at this moment that he knew he f*cked up
People get night sweats, you on the other hand, sweat the nights off trying to keep an orchestra alive. 🤣
Love your content
I watched this video till the end
lol I saw your last video on bye vercel/aws. If you end up staying with vercel let us know haha
no we're back on aws 😂
Things change fast apparently
GCP + Cloudflare 👑
bollywood script right here codedamn x vercel
Brother can u please make video in Gatsby vs next am totally confused please will be waiting for your video brother
welcome to Vercel again😁
2lakh subs🥁
Lol 😂. Welcome back
nah 😂 we're back on aws
@@codedamn Farewell : )
You are giving your best to the yt that's inspires me... Great Content.....
i watched this video till the end
I saw one of your video, stating why you swiched to aws from varcel...... Please settle this for once
I am stunned by the ignorance when people comment without watching the video
We did not move permanently back to vercel, only for the downtime associated with deployments with cloudfront. Please watch the video :(
🙌
I watched till the End
I watched till end
lol 😂😂 I love these videos
Hello Codedamn- I see there is a video of yours on 5 Dec 2021 (Bye Vercel , Hello AWS-ruclips.net/video/Do92vKNr3Og/видео.html).
And after 3 days there is another "Bye AWS, Hello Vercel". I am bit confused, please clarify or summarize both videos.
I really appreciate your tutorials
Didn't you just make a video about leaving Vercel due to their billing and bandwidth prices?
I did, and our company account on vercel is still frozen due to billing issues - I moved the codedamn deployment for 12 hours on my hobby account while AWS fixed the cloudfront issues.
Another fact why being in the community.
When you trust corruption you get what you deserve
Bro, you jinxed AWS nothing else. AWS must be blaming you for that as well. 😂😂😂
One word: Varnish.
Cloudflare is the way
Omg
Lol that was fast
😂😂😂😂
lol... to fro
Anything to keep the site live
You can use flux instead, a decentralized AWS ❤❤
vercel itsel uses aws ig
they don't use cloudfront CDN. They have their own/custom CDN implementation. That's why vercel new deployments were unaffected yesterday
Bruh, make up your mind xD
It IS AWS. Can’t help if you’d not watch the full video :)
@@codedamn Chill man. Just messing around. Love your videos
I don't Know BUT I Think, Your Video's Title is Going To be Negative Day By Day. Please Put Positive Titles Man. This is a very bad way to get views.