Platform for Scalable Web Apps | How I built my website with Kubernetes
HTML-код
- Опубликовано: 23 июл 2024
- I've spent months building a platform to support my development of scalable web applications. In this video, I try to share the learning process.
My personal website has a lot behind the scenes that most people cannot interact with. I've written up a few microservices and decided to deploy them on kubernetes. Some may say that what I've done is overkill - but it really does simplify the workflow once its setup (and I don't have to rewrite my codebase in the future!)
The end result:
- Kubernetes cluster on digital ocean.. 3 nodes, a load balancer, and persistent volume
- nginx-ingress controller exposes services to the outside world (all other services are internal)
Microservices:
- jwt authentication server (security)
- hangfire background processing server
- MySQL database
- mailer
- front end
There is still a lot of work to be done. While most services are scalable - the database is not. Currently, our database is a single instance. I think someday I will need to deploy a MySQL cluster to support replication, etc. This was a massive topic to cover in a single video so I could not explain everything, my hope is that people can take what they see and research further.
Technologies Used:
- CPU Intensive Backend: C# with ASP .NET Core
- Simple I/O Intensive Backend: Node js
- Database: MySQL
- Scheduler: Hangfire open source background processing library
- Authentication: JWT backed by Dotnet Identity
- Mailing Service: Sendgrid
- Integrations: RUclips, Github
- Deployment: Docker images w/ Kubernetes
- Front end: Next js / React
Hangfire (background processing): • Learning a new Program...
RUclips Descriptions Updater: • Coding a RUclips Descr...
Next Video: • Hacking the RUclips Al...
Previous Video: • Why I'm not in School
Social
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Website: devoncrawford.io
Twitter: / devoncrawford13
Snapchat: / devoncrawfordyt
Instagram: / devoncrawford_
Github: github.com/devoncrawford
Discord: / discord
Gear
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
- Video Editing Software (Premiere Pro CC): prf.hn/l/BOomWo3
- All Adobe Apps (I use this plan): prf.hn/l/Xv5qk5Q
- Keyboard (Velocifire TKL01): amzn.to/2AQTjJQ
- Monitors (Dell U2518D) : amzn.to/2AQQmco
- Webcam (Logitech C920): amzn.to/2Cw588c
- Desk Microphone (Blue Yeti): amzn.to/2FPFmzi
- DSLR (Canon T7i): amzn.to/2T54mpZ
- Wide Lens (Tokina 11-16mm F2.8): amzn.to/2MlpvcO
Music
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Puar - GEMS W/ sin
Link: / gems-w-sin
bob le head - gapintheclouds [ep] @ 6:52
Link: / gapintheclouds-ep
dis.joint - knock
Link: / knock-3
A L E X - Easy Track (part of Growing Up, Vol.2)
Link: beatsupply.fanlink.to/growing... 
Наука
![Sean Rii, Karyon, Sharzkii - Taungule [My Love] (Official Music Video)](http://i.ytimg.com/vi/eJoMuypbSzQ/mqdefault.jpg)
"It never ends".... Looks like your videos ended :( . You were one of the first tech channels I started watching when I barely knew anything about tech. Hope you are alright man.
check out his twitter, you will be surprised
He is now back
@@skyprogramming2177 "back"
Good job.
Although i personally am not intrested in web development i find it nice to watch these videos. Keep up the videos they are intresting and enjoyable
Probably when we are seeing videos like this one, watching the simple "thinking" guy and sometimes watching things we doesnt even understand, this blowing our minds too, for example this gave me a emotional and mind burst to learn new things, and be happy of that, that even when it is very very hard to me to get up to work, and learn difficult things, but when I see this guy smiling when the main problem is solved it reminds me that learning can be so rewarding! Its all about motivation.
"it never ends"...lol Welcome to a developer's world, my friend!
😅
😅
I have no idea what you are talking about but I like watching these videos, crazy. Testament to your editing, knowledge, and persona.
I love these videos where you blow up OUR minds !
You give us some clarity and show us potential paths to achieve specific learning goals.
I am waaaay behind you but because of the high quality and clarity of your videos, I understand the concepts and can see where i'm going. It probably the same for most of the people watching.
So thank you Devon and keep 'em coming please !
I'm honestly amazed that you can learn so much and achieve so much using only tutorials on the internet, keep it up man you have a great future ahead
lushu That’s how I learned just about everything I know about programming. Definitely possible!!
@@bruhdabones not alot of people can process information like that especially at the speed that he does
stop flattering him like he's a run-of-the-mill... everyone can do it ok. U, reading this comment, u can do it too!
@@marcusaurelius8540 You should say no one instead of 'not a lot of people'
@@gadgetboyplaysmc Of course but in decades not in months
In a weird way, It feels good to see someone else go through the problems of learning everything on their own and create a production-ready set of services. I honestly enjoy watching your videos and keep picking up new things every time!
This is some great content Devon. It's very impressive to see the progress you have been making. Your setup there is similar to how some of the leading web organisations in the world are organising their services. Great job.
Videos once a week? I can't believe that's happening
IKR
ikr its awesome
more like once a month lol
@@neil9151 now like almost one per two month :/
@@MasteAlex95 No, more like once 1/2 Year...
You are a motivator! Thank you for your videos. Now, let's watch!
replying to your last question. I like these styles of videos. IT shows some of the thought process involved in what you're doing.
seeing someone talking about coding with such "passion" really motivates me to learn more
Shoutout to all y'all who don't really know much about computers and watched this just being blown out of your mind
This is the truth
So true
You don't need to know about computers to know about web development.
@@aleaallee that's like saying you don't have to know about cooking to know about making a steak
@@kreglfromworld That makes no sense. I can build computers and know about them because i'm an I.T Tech and I got taught how to do it but you don't need to know how to build a pc or how to crimp a rj-45 cable to be a web developer. You need to know about programming and related stuff to be a web developer.
Love the videos, I understand about 5% of what’s happening, even being a cyber security major myself, but I’m so inspired to start self teaching myself things. Awesome!
This is probably the your first video in which I have not understood anything but I still love watching you and listening to you desceibing what you are doing!
It's cool to see you transitioning from theory (school) to real world projects. I'm even debating starting a RUclips channel to help out our fellow devs-to-be after seeing how helpful people find these kinds of videos.
watching over and over again until next video! 2021 team here
If your mind explodes while you're programming, imagine mine watching your videos.
brain goo
Gabriel Oliveira brasileiro?
To answer your question, yes. It's fun to see your thought process, struggles, and breakthroughs. Thanks for the uploads!
I'm literally so amazed by how much this guy gets done. Honestly it's a bit overwhelming for me, especially when I try to make a website with a very limited schedule and end up falling far short of being comparable to his front end.
Everytime u upload a video and I see how far ahead of me you are in terms of web development inspires me to learn hardcore for like a week. It's kind of unhealthy how long I code for.
I don't understand what's happening but I like it.
Ricardo Rodriguez same
@@TINTUHD same
tru
Hes bascially describing what a devops does.
The overwhelming joy and motivation I get from seeing you in these videos is high.
Yes, I do like these videos. I watch them while I work (well listen)
I enjoy hearing what you’ve learned and how the progress is going
OMG you can't be true another video in the same year?????? no way
Victor Mendonça Nogueira hahaha Its unbelievable
Mind = Blown.
hi devon, i find it motivational watching ur videos.
i just started front-end. lol
Lets go!!!!
ayyy!!!
you have a long road ahead ! good luck
Wow. That was actually super enjoyable to watch! Cool to see your process and how you came to the decisions you made for your website.
You have got such a huge positive attitude to what you are doing. You are inspiring me everytime I come to your channel (I'm an IT student also interested mainly in dev and electronics who sometimes gets bored/frustrated too easily or doesn't belive he makes something useful).
Please keep doing this great job !!!
"Do you like these videos where I sit down to blow up my mind?"
About which are you talking? We are here to see minds blowing up from both sides!
Near the end of 2019 and still waiting for his next vid
Ivan Grey yeah whatever let’s hope he is alive
@@groultaria6732 i hope he's still alive
@@greycruz2672 Yeah he is. Last I heard he was working on something big according to his mods in Discord and he doesn't wanna show us for some reason. I guess so no one copies his idea while he is building it. He inspired me to make my own channel. I hope he comes back!
yeah he hasn't uploaded since last decade
@@kataiho4405 Phew! That's good to hear. I just started his videos and checked a lot of his stuff and everything seemed to have just died off mid last year and I began to think the worst. Glad to hear he's okay!
You are a huge inspiration to any one who wants to become a programmer or already is. Thank you
Watching this video again, can't believe it's 3 years now, glad to see you back dude.
You do so much hardwork .
Thanks for this video to motivate me.
I hope the number of concurrent users is worth the effort and cost of all this work. I get the feeling your doing all this because your just so curious about it
"what I'm saying is that... It NEVER ends !" This is so relatable lmaoo
ikrrrrrrrrrrrrr
Loved this video! Nice that you went a bit into backend architecture, very interesting to see how this all worked and how your learning process was.
Your videos always get me in the mood to work on my own projects so I'm personally a big fan of content like this. Keep it up!
It’s funny cuz he said “do you like these videos where I just sit down and blow up my mind” and my minds getting blown up just watching his videos 😭😭😭
Devon you're my idol!
Honestly, this kind of channel is exactly what I'm on RUclips to see. Well produced, different and about development. Massive thumbs up dude 👍
Blowing up you mind simply means your learning at a higher level and really applying yourself. A++ for effort.
Teaching yourself kubernetes wow, myself trip up myself everytime i want to setup some docker image with docker compose for my work lol.
Have you ever count how many time you put your hand at your head lol? In the timelapse i saw you do that so many times.
And i like the video, keep doing it.
While everyone is stocked to see you upload more, it's safe to leave this advice here: Don't let this schedule lead you to a burnout, follow your pace, and be sure that you have your own time, Like Casey Neistat does, I think you're a bright guy who has a bright future ahead of him, here in RUclips or anywhere
Nadir Belmokhtar he didn’t even start yet to post weekly video and you talk about burnout? You crazy?
@@Dan997 every experienced programmer knows he is telling the truth. Stay in your lane.
AnonymousHobbit I guess OP’s advice came too late :p
The "IT NEVER ENDS" at the end of the video hits close to me lol. I started a project back in July with the intention of having it be a super simple inventory tracker, but I just keep wanting to build it out and better. I came across this video while looking up Kubernetes tutorials for example. I love the passion that comes across. I'm excited to see where you'll go.
Yes i love these videos so much. The production quality is great but also it just keeps me motivated to keep learning programming. Its nice to see what it takes and how long it takes to build cool stuff.
i'm that kid who literally only knows front end and understand close to nothing here but that doesn't stop me from watching bc devon's video still make sense?
thank you bro 🙏
Kinda the other way around for me
stardustaed. It does man. Keep your interest on complex and soon it won't be complex at all.
You live in a completely different universe!
I like this channel a lot. It's pretty unique in the realm of Software Engineer youtubers. I'm a developer and it's nice to see someone actually show their life and process along with the technical aspects of development
As someone who is going down the path of backend development and what's to eventually get a job out of it, these videos are really cool and inspirational. Can't wait to see more!
A bit stupid question, but how do you host your C# server and on what platform?
Use digital ocean with a ubuntu droplet and install dot net core.
@@menuly Can I host a TCP server on DO?
#CSArmy where yat?
(OMG MY BIGGEST INSPIRATION NOTICED ME :O)
AmazingJonnie Mine too XD
I hate to see sharp
@@devtech4661 You seem a bit RUSTy. GO get yourself an ELIXIR to see better
This was great, I'm planning on using Kubernetes for a personal project but found surprisingly free resources for it. It was helpful to see you walk through it
I’m form Mexico and I started watching your videos like four months ago when I started my first semester in the university and seeing your videos that you are coding here coding there, well, I feel inspired
For Authentification you should try OIDC. This is a common method in a Microservice Architecture.
cool I'll look into it!
@Z3U5 I´m not an expert on that topic right now. So im trying to setup keycloak and github api for my authentication needs.
@Z3U5 As far as i know yes, Apache License 2.0.
Yep. I recommend Azure B2C.
Has a nice integration with Dot Net Core of course.
try IdentityServer4
It would be really really helpful if you made a list of websites or sources you learned these things from and also the order in which you learned them ( although this might not be important ). This will actually help a lot of people understand in detail and follow the learning path.
ASH Studios When you do a project, do you save and bookmark everything? Probably not with five to ten browser windows open at a time
@@Anony584 yeah but at least I keep track of websites which were helpful so that I can refer them later. So maybe, just maybe he remembers some or he could make a list of websites which he thinks is best to learn things from
Bro, the way you explain complex architecture is AMAZING!! KEEP UP THE GOOD WORK!!!
These videos are super inspiring and insightful. Its really helpful to see how other expand their projects with out any guidance or prior experience in certain topic. Thankyou and please keep putting these types of videos out!
Pay attention and take notes people, this is how you become successful
Wow! I'm going to start learning Kubernetes
Been a front-end dev my whole career so far. After hearing your excitement about working through these problems, I'm setting a goal to learn some of this Docker and Kubernetes stuff for myself. Great vid!
Even though I dont understand anything, I find it somewhat satisfying looking at you explaining all that stuff.
You should give Istio a try if you're heading down the micro-services route. It gives you tonnes of control over how your services interact with each other. It let's you split traffic between different service versions depending on certain conditions, and you can control which services can talk to each other securely, using mTLS and JWTs. Your apps don't even need to be concerned with authentication because it's all done by istio.
@Z3U5 Istio 1.0.0 was released back in august. It's sitting at 1.0.5 right now. I'd argue it's more than ready for production.
I got confused by this video.... Oh wait I get confused by all of his videos.
He is very fast in his videos as in his work
Your videos where you sit down and blow up your mind are the biggest inspiration:D
Yo been a few months since I've checked out one of your vids, Your progress is amazing can't wait to see more.
you dont only just blow up your own mind, but ours too lol
2:18 "Micro-services... AAAAH!"
Yo these are a actually my favourite videos, man. I'm going through the same process as you and its sick knowing I'm on a good path. You're an inspiration bro!
Great video. I came across this same video a year ago and didn't understand what he was on about. But now I do after an year of learning, thanks Devon.
I can't understand shit LOL
But you are a good example for selfteaching for me thanks man
"What I'm trying to say is that ... it never ends!" #relatable
To answer your last question, yes.
Been looking at an architecture for some big ass database that runs live data in a cloud and have just come across kubernetes 5 days before watching this. Your journey is relevant to me and I appreciate your explanation. Your words make sense. Keep going x
3 minutes in and this may very well be my favorite channel on RUclips
"this is really basic"
*i'm never going to be able to pass computer science in college*
Is it normal that I'm on my second Computer science year and understood close to nothing?
you have to learn by yourself. School cannot give you everything. You need play with technologies in your free time.
I have computer science degree. College didn't teach me anything that I can apply in production environment. But college gave some crucial life lessons and made me stronger. Teach yourself buddy
yes, this is not what is taught in academia just an fyi kid
Yes this is normal. You need to teach yourself.
Your vids are AMAZING bro, I love your content, the style, everything. I’d love longer vids 🙌🏻🙌🏻🙌🏻🙌🏻
I'm in my second year of applied computer sciences in Belgium, And I understand what hes doing but I think that I would never be able to learn all hes doing myself. I have a lot of admiration for your perseverance and keep this channel updated the content is amazing.
The reason half the comment section is confused is because you guys don't actually code, you just talk about it. Start applying yourselves and you'll realize this stuff isn't too difficult.
Word!
Except for backend... Backend is always confusing ._. or just too much work .____.
@@CZTachyonsVN Why do you think "backend" is confusing. Sounds to me like you just haven't done enough backend work. I smell a preference for frontend work :) i guess we are talking about web.
@@stylishskater92 That's why I said that it's too much work. I mostly do software dev (mostly games and apps). I rarely do web and too lazy to get into backend.
CZTachyonsVN Isn’t software dev similar to the backend of websites?
i love how screen recording on linux always sucks
I think it could be Wayland's fault
Working with deploying a research project for Iowa State University using kubernetes and docker containers. Some very powerful stuff. Great videos that are entertaining, aesthetic, and also very informative. Personally I enjoy when you go into depth on a topic--and I feel that you could go even more in detail than you did here. But this is a great overview of how you applied the technology.
Awesome, Deployment Planning and Executing. Goodluck on your lab and future projects.
You don't need any of this. 😆 You want all of this.
Finally
I love your stuff man! inspiring me to get back into web dev and rediscover programming. Also the way you document your work is great to watch and super educational! Makes me want to document my digital media/art projects. Cheers from Ottawa!
Please keep doing these, I'm more of a networking guy but I can tell that you are pretty passionate about programming and I love that. Keep up the good work man
Learning new things is cool and often a lot of fun, but I don't think over-engineering things is necessarily the best way to understand WHY we do certain things. I think it might have been much more useful to have actually built something that could benefit from a Kubernetes setup, and then learn how Microservices/Kubernetes/Docker/etc. work. Personally I think it's much more useful to experience a problem yourself first, and then learn how to solve it instead of just learning a piece of technology just because it's popular. Now you're running (essentially) a static site on top of Kubernetes. That's like the equivalent of getting rid of an annoying mosquito in your bedroom with a nuke.
Not trying to dissuade you from anything, I'm also self-taught. It's just my impression that you're learning things in the wrong order. Just my two cents and an attempt to be helpful. Of course, in the end, if it works for you it's all good.
Kind of difficult to come up with a home project that benefits from kubernetes. I think it’s fine to go completely overboard and over-engineer personal projects. ☺️
Now configure the GitLab CI to perform your versioning after successful unit test. Embrace the CI/CD pipelines ma dude. No need for all that crazy kubectl
how do we do that? where can I learn that?
@@fadilnatakusumah195 do the proper Google searches, first learn docker, then kubernetes (do not use the easy way from the pipelines)
@@gabrielmelnik6796 thanks man
seeing you coding blows my mind
i've started programming with html/css some months ago and im now learning js and rails
Doing a great job! Please keep making this sort of videos once a week! I really like them
You could have a paid subscription service on your website (To make some extra paper) that gives access to your videos 1 week early. Basically like a poor man's Floatplane.
PCI liability, streaming servers, robust authentication systems and a payment gate seems like a lot of infrastructure to build and pay for especially considering Devon is college student #1
RE your authentication mechanism, it is ok to write them from scratch for educational/intellectual curiosity. If you are ever working with a client, you should not do that. Look at established products like #Okta, #Ping, #IdentityServer (from LeastPrivillege) or #ADFS.
Indeed. Security is hard, and clients don't really care if you're making it from scratch and would prefer you using something that has been battle tested. However, using something popular doesn't necessarily make everything safe, in fact it automatically makes your site a target. The ideal authentication server would be the one that never receives passwords nor should it provide fully working session keys, and this can be achieved by using zero knowledge proof and commitment schemes techniques. I'm not yet aware if there are such implementations available, but OPAQUE protocol is a good way to start. What I'm trying to say is that there is always room for improvement, and we learn the hard way xD I myself got hacked 8 or so years ago when learning sql/php, I used same password everywhere, I stored hashed passwords as MD5 and not even used SSL.
This is dope. Please keep uploading! Your progress is insane, I would love to do something like this.
Channel is getting amazing! Keep going, cheers from Brazil
This seems like total overkill for your website.
Obviously it is. He is doing it for the learning experience, and as a side product he gets some nice references. It's better to be able to link to that than to a simple 3 tier website.
Isn’t he still in college? Although it’s overkill, his website can prove to employers that he can build a scalable website right out of the gate. He may have bigger future plans for it.
@Matthew Jones An important part of being a software engineer is understanding the overhead of the architecture you're building. Not every monolithic webapp needs to be aggregated into microservices. If you intend to add some additional heavy lifting algorithms (i.e. machine learning) then microservices aren't a bad way to go. Don't reinvent the wheel.
@ Sure but this is about learning. This is literally so he can understand how it's done. What's not to get?
It's not about the website. It's about having a platform to host/deploy applications. which is why he is focusing on scalability.
Keep confusing me, so as I can get motivated, not proud with my programming skills
Man, I was like checking every day for a video. You inspire me man.
I love it, this is extremely helpful cuz its all real world ideas. We've all got project we wanna do and when we see someone else brainstorming, its smart like I never even thought of doing a router type of app with what you did in node.js and it makes sense to use a docker vs all these separate VMs. Keep it up bro
Overengineering x 100
Yep. His site is nothing more than a static site with cron jobs running periodically.
@@8Trails50 But it's the learning experience that counts. But must only pay attention to the finished product.
Agreed. He needs to learn the tools and go through the thinking process. His project is irrelevant if learning is the main goal here.
@@armanmirk Sure, learning is great, but you can learn all this things on your local machine without spending all this cash and putting stuff up online. Maybe he has cash to spare and doesn't care which is fine, but as someone who doesn't have a lot of extra money, this is really odd. I mean, considering what his setup looks like, I guess he really DOES have too much money.
Learning to identify overengineering is the turning point in your career.
Children: This is a classic example of overthinking architecture. Something as simple as Netlify would have done. Or even Amazon Lightsail.
I think he picked those solutions to learn them rather than pick the best simplest tool for the job. Resume driven development
Awesome videos! It is really good to find these kinds of contents on architecture. Thank you
Keep up the excellent work. I love watching & learning from you.