How Instagram Stores BILLIONS of Videos
HTML-код
- Опубликовано: 11 янв 2023
- How does Instagram handle BILLIONS of images and videos per month?!🤯
In this video, we take a deep dive into the technology behind Instagram's video storage infrastructure. Learn about the various techniques and tools used to store and manage billions of videos, including distributed storage systems, cloud computing, and data compression. 📀
See how Instagram ensures the availability and reliability of its video content, even under high traffic conditions. Discover the challenges of scaling such a massive video library and the solutions that Instagram has implemented to overcome them. 📈
Join me as we explore the behind-the-scenes of Instagram's video storage and learn how they are able to handle such a large volume of content. 🎥
Don't forget to like and subscribe for more tech-related content like this. 😀
My name is Lewis, I am a professional and self-employed Software Developer that likes to help inspire developers. 👩💻
Source: engineering. 2022/11/04...
How Meta load balances globally: engineering. 2022/09/06...
Leo's video on compression: • How Video Compression ...
MY NEW NEWSLETTER 💌
thebetter.dev
------
CONNECT WITH ME ON SOCIAL
📸 Instagram:
/ lewismenelaws
🎚TikTok:
/ lewismenelaws
🐣 Twitter:
/ lewismenelaws
--
My gear 💻
liinks.co/lewismenelaws
----- 
Наука
What company do you want to see next? 🤔
RUclips
Google Maps, how it is working? who it is automatically suggesting even though we changed route? i would love to see you with that video in depth content
Spotify!
you tube
Google(their search system)
"How Instagram Stores BILLIONS of Videos"
RUclips: 🗿
About 80 years of content per day 🗿🗿🗿
Is it just me or the voiceover is too quiet? Especially when the sound effects are played, the voiceover is barely hearable. Great video tho
My bad! I’ll fix in the next video 😅
Sounds normal to me
It's not too bad.
Yeah, music and sound effects are a bit loud
I actually like it.
Speaking of encoding, Jpeg XL has the potential to revolutionize images on the web as it allows for lossless encoding of jpeg, png, bitmap, and gif files. It also allows for progressive loading. Google's Chromium team left it dead in the water in favor of their own webp and AVIF formats despite JXL having similar compression and superior backward compatibility with the older formats.
Webp and AVIF are also limited in terms of supporting a range of bit-depths or arbitrary channel configurations. JPEG2000 could actually be really useful too, particularly the high-throughput version. It’s frustrating these things don’t get better adoption.
I HATE WEBP I HATE WEBP I HATE WEBP I HATE WEBP I HATE WEBP I HATE WEBP I HATE WEBP I HATE WEBP I HATE WEBP
agreed@@Vivi-xn9iz
@@Vivi-xn9iz I think we all do
I really enjoy seeing you discuss real world examples. It really helps to understand what the field is like.
Just started watching your channel and I’m really enjoying being a regular viewer of your channel. I’m thinking about getting into programming and overall finding a career with a great mix of being analytical and creative. Keep up the great work!
Well done, this was a great watch, even though I know nothing about programming you made the video feel very casual and easy to understand
You make these videos really interesting and thank you for that :)
I have no clue half what you talked about, but this was entertaining AF!
Thanks for linking the article!! What a great read
Great vid, I wonder, besides performance issues; how many security issues(from the lowest levels on) remain hidden in pieces of software left that way under the principle of "if it ain't broken don't fix it". Specially on those software components which are the foundational pillars of our modern technologies. Not long ago I saw one meme/picture which stated that some essential long-lived pieces of software are maintained by a group having just one person.Scenerios like that are terrifying considering how flawed a single person can be as to left unnoticedly one vulnerability open, or how many backdoors a wicked programmer could build and left hidden in such software components.
This is one of the verryy few channels for which i have the bell icon to all
Man you're great, its so good to see your videos with all that subtle humour, the animations and the story telling.
Where is the link to the meta load balancing article?
My bad! Added it in
You just got a new sub because of a video topic like this keep on
The links you said will be in the description are not there.
video suggestion: "How tick tock stores billions of Videos " and youtube also
I love how netflix was just stapled on as an after thought. They are basically an amazon product anyway.
Compression technology is fascinating!
The amount of braincells of mine is *spent* today with this video
Thanks dear ❤❤
"I know talking about video compression is kinda boring" bro that's literally why I'm here
man that was amazing
I inmediately noticed the SMITE gameplay on the video example, do you play it?
Good video Lewis 👌🏻
Very very very informative, these type of videos will get you to the moon inchallah
Great video, thanks!
Make a video on flask vs django
At least we know size matters
@CodingWithLewis what do you use to create simple animations like this in your video? Can you tell about the process?
Grate video lewis 🥰🥰🥰🥰
Thank you! :D
It would be very cool to get straight to point(s)
I always assumed they did a lot of the encoding in the app itself while uploading
loved this
Most modern phones can encode and decode videos locally.
I think they should just make the user's device encode it when posting.
Although, this will increase the amount of data transmitted.
No but it can in many cases increase upload time... sometimes a lot
This also increases processing time.. a lot and with it battery life suffers a lot. Phones lasting forever depends on them not doing any real processing at any point.
The ROTMG video on the TV. I see you. 👀
Someone tell the airline industry about this thing called "technical debt"
Great video 👍
Thanks for your sharing
I wish you could also talk about how Instagram shadowban users as well...
Always very interesting videos señor
Thanks for your sharing
Omg! Was that smite ? 😅😅 We should q together 😂😂 love your videos 👏👏
Make a video on how the Internet works. Why cannot we make our own web page with Google name explain this also.
10:03 .mov is container, it can contain h264 video
Bro, your video is really good and I enjoyed it except the sound of your video is really low! I was using my Macbook Pro 14" and even with full sound in both system and youtube I struggle to hear some sentences. I had to turn off my ceiling fan in order to hear it properly. I would appreciate it if you could fix that in your later uploads, thanks.
How does Google store all of these RUclips videos?
Looks like I came early 😁
lewis taking shots at ltt
I'm really disturbed by how he read my mind at 3:57. Was literally mousing over to the next video x.x
First time on your channel, i just watched your discord video too. The technical content was great and well delivered. Keep that up and you'll have every success!
The bits in-between the actual content where you apologise for the content you're about to share wasn't great for me. I wouldn't be watching if i didn't care about video encoding formats for example. It's probably an attempt to be light-hearted and add personality, but it makes it seem like you're not confident about the quality of the content or material, or confident in yourself and how you made it. It's also time where I as the viewer am waiting for you to just get on with it and am just waiting for it to be over.
If I may, I suggest you don't include those bits and instead keep the flow of the video focussed on the topic you are exploring and the narrative you are building. Trust that your audience want to see what you have to share with them, and share it instead of apologising. You got this.
Totally agree! The technical content is great. more focus on the main topic would be even better.
A clear lack of views compared to the quality you are providing. I hope your channel will blow up soon !
Thanks
@Lewis can you make a video about QAs? Feel like it’s an underrated overshadowed position
One day we'll outgrow centralized systems and it'll be required to distribute like IPFS.
how about instagram database work? query image/video?
It has been around a year and yesterday they went down
Meanwhile youtube: hold my beer
I wonder who was the guy that came up with that, and most importantly, why is he not getting the well deserved credit?
Instagram still has no volume slider
why did you think they would store these videos as mp4 ?
please make a video about chat gpt
can you set a better tone for your voice? it always fluctuate from high pitch to low pitch, hard to hear =(
You forgot to talk about VideoLAN
3:28 how long was the video they did the tests on? We just hear the function took 86 seconds to run but we don't know how long the video was. What was the original video resolution?
In the article they mention that that was a 23-second video, which they transcoded into 720p, unfortunately I don't see a mention of the original video's resolution.
Most likely, we are discussing a one-to-many encoding scenario where a single input stream is utilized to produce multiple output streams. This approach offers significant potential for reusing several steps across the various outputs, potentially resulting in CPU savings of about 20-60% depending on the codecs used. Additionally, it can enhance scalability since many transcoding tasks for different outputs can be performed in parallel. For instance, if you're working with a video in QuickTime format and need to create H264 outputs, there are several stages involved, such as reading, decoding, and frame preparation. Without reuse, these steps must be repeated for each different output, like AV1. However, it's feasible to read the QuickTime format once, decode and prepare it, and then simultaneously generate both H264 and AV1 outputs.
By the way, this is an educated guess on my part.
I still didn't understand after watching full video 🤧. Seems like I am dumb or it's very complicated
Its complicated! You aren't dumb :)
Sarnia?? You doing good boy.
🙏
So they basilly made a reusable class
How about h265?
where is h265 ?
And what about redundancy? Do they compress those too… idk how it would work. Welp, this is too chaotic and scary.
I love you my bro am first
My social media launching soon
Make sure you share!!
First🎉
Second
Could we use machine learning to create a new video format? 😅
Can u lower the music or completly remove it please
So basically threw a hash map on the problem
Next video, How TikTok Stores BILLIONS of Videos.
Are you kidding??? I love talking about video compression!
your videos deserves more likes and suscribers
Arweave?
And that's why instagram have so many dropped frames last months
I am Sparkly to
Next do RUclips
hey lewis could you do a video about how we can code better with chatgpt?
Yes! That sounds like a great video!
@@CodingWithLewis Thanks for considering it!
Сколько интересно вообще весят все видео на RUclips???
Do you have an discord server?
Link in description :)
Oh
0:40
Please tell me why I cannot sell a method to compress any data like from 1TB to 1MB? I can't contact anyone who would like to invite me to show them how it works..
Guy has the intonation of a valley girl
Feels really condescending for you as a creator to decide what is and isn't boring or difficult for me (the viewer) to comprehend. Let me decide for myself.
Also half the video is tangential to the point of the video.
😉
Do you Wana colab: anyone reading comment can colab.
Information:
I am creating a AI for it to write programming languages automatically from English
Goal:
To create a better place for debugging and creating code for new programers like it can explain code.
not much details given about actual process
This is a kind of channel which has good host, good topic, good editing... still i feel no value added! not sure why, seem lewis himself has a very limited knowledge for the content!
Middle out,imagine you j two guys at one time....
RUclips
3:50 boring? video compression is anything but boring!
That’s not that many
@coodingwithLewis how do i convert a 10gb vid mp4 to H264 without having to pay subscriptions
tldr: They use middle out compression.
It's hard to enjoy ur videos:")