At least with Erlang the edge cases won't take down the whole system... Erlangs paradigm was always 'Let it crash', in fact the whole language was designed with Fault tolerance in mind, they knew in critical systems like telecommunications the system needs to always remain up and available, so rather than attempt to defensively program, they viewed there will always be unforeseen bugs. hardware may fail, but they asked how can we ensure those problems never break the wider system. Every user action is isolated in its own PID, so when an edge case arises in that PID it just crashes, and a supervising process just restarts it.
They found all the mythical 10x engineers, captured them, and locked them in a basement with only laptops with Linux terminals and a mountain of anime pron to keep them company Those engineers have never been more happy
For context about FreeBSD, because the article skims over it: For years Erlang developers preferred FreeBSD because IO multiplexing is done in FreeBSD with something called kqueue, which requires one less syscall than the Linux equivalent (epoll). I believe Linux has iouring now. Let us just appreciate for a moment that the Erlang VM handles IO so well that the OPERATING SYSTEM ITSELF BECOMES THE BOTTLENECK.
Just a tiny technicality; Erlang uses "green" processes rather than green threads. Difference is that threads can share state, memory and address spaces, processes are isolated and share nothing (other than by copying values). Erlang implements its processes in its runtime system and is managed in its VM, rather than using OS threads or processes.
@@SeanPoulter Its amazing how easy is to create massively scalable things with Erlang, like most of the heavy lifting is really the programming paradigm. I was reading the Ejjaberd source code and I was impressed how little code in relation to the runtime is necessary to create the core of what is Whatsapp.
Try doing that with C you might save a lot of memory, but you lose all scaling as well. Its almost like C is cheating by being unsafe with its memory model full of aliasing and sharing via the use of pointers. You are also going to need 35.000 engineers because C is a shit programming language that requires a lot of stupid work to manage the memory. Its ironic that computer were invented to automate industrial process, yet, the very own engineers working with computers decided to not industrialize the most boring and stupid programming task : managing memory. All because in 1970, the PDP11 only have like 8Kb of RAM or something... But Erlang coming from the telephony world don't use memory to process data, only to control process that control the data flowing in actual wires, that's the magic. Its the control-plane/data-plane separation. Stored program with mutable shared memory and big linear address space was a mistake, and its not even how hardware works anymore anyways, the so called "C" programmers don't even program bare-metal anymore, its all a fad, its only there for their ego or something, that a modern compiler has to make a modern computer look like a 1970's era PDP11. And we are forever going to be stuck in this memory model thanks to C and Unix and Intel.
A manager went to the Master Programmer and showed him the requirements document for a new application. The manager asked the Master: "How long will it take to design this system if I assign five programmers to it?" "It will take one year," said the Master promptly. "But we need this system immediately or even sooner! How long will it take if I assign ten programmers to it?" The Master Programmer frowned. "In that case, it will take two years." "And what if I assign a hundred programmers to it?" The Master Programmer shrugged. "Then the design will never be completed," he said.
I was expecting more from the article tbh. Yeah, it lists some things, but it's just a lists of concepts that by now I think it's just standard good practices. (Aside from the lang choice and fBSD OS choice). I was expecting the article to go a little deeper. It's nice to know that they located and fix bottlenecks, but I wanted to know, how they identified, what was the issue and how that got fixed.
Fear of telling too much and then having the information used against you later; fear of clones; industrial secret; or it just takes too much effort to ask the techs how shit is actually working.
Yea it’s not like this one big villain bro These are different people Then this L mentality when you’re seeking revenge because you were “rejected”…. Duh I’m pretty sure such intelligent and successful people are above that
I worked in an IoT platform, and we reached a peak that we were struggling to manage. We started talking to some people who managed a lot of messages a day, and they were shocked by our volume at the time I worked there. We were hitting 500 million a day, and passing that mark was a challenge. I cannot imagine how it is to handle 50 billion.
Agreed. I learned nothing from this video. Glad I skipped 90% of it. Dev youtubers really should focus more on providing value with their article-based videos. Most of the time it's empty phrases like "if you've never worked on x problem....it's HARD! Like...actually hard" and other trivial statements. Feel like it's more about engaging the dev community by addressing obvious common concerns, struggles and frustrating experiences as a dev more than providing new insights
This respond from you is that you have not being born before Internet was a thing, I worked with Video Editing/Rendering to built those system in the 1994, the hard drives needed a special software because normal hard drives where "calibrating" and in those moments you where losing frames when you rendering hard core stuff, now days, just put a SSD or NVMe drive and you think problem solve. That is what programming is today, put just more Framework, and in the end you have just built a system even with the newest hard drives/SAN, and your performance will be like if i remember 100 IOPS per hard drive, so in a stripe configuration for performance, you should know that would be a cascading effect, . But you got to love that bill from any of the Cloud providers. If you ever built a successful app. LOL and BTW this was the time when Silicon Graphics had there computers running the HIPPI protocol. To much knowledge have been lost in time. But one thing remains, we that have struggle during the early days, really know how to build system from ground up today.
Having worked at Ericsson, I can tell you that they mostly dont use Erlang these days. Where I worked there was a lot of Scala and Akka, which is funny to me. One really cool feature of Erlang is that they could predict the amount of hardware needed. X amount of messages means Y hardware requirements, 2X messages 2Y hardware. But the coolest feature has to be the suicide pacts of processes
Akka is cool, if only wasn't running under JVM, and if I'm not wrong is just replicating what Erlang already does. Can't figure out why would they do that.
@7:40 No, not really green threads. That article misrepresents what is going on in Erlang. Erlang is using actor processes and has practically a full OS like process scheduler. And because it is built into the language and platform it offers features not found elsewhere and often can't be found elsewhere when the feature is an afterthought.
I think perhaps another advantage that WhatsApp has (or had) was lack or server side storage? Generally messages were just sent and lived on client devices. When you went to a new device, if you didn't do a transfer yourself, you'd start fresh. I'm sure there was temporary storage but when you can cut out those huge database challenges you can save more engineering resources
Thank you for showing where that meme came from, and the potential rapidity of meme generation and sharing. Love it! Am familiar with GIMP but your demo (about halfway through) produced much faster results.
okay I think I still am a javascript advocate but @ThePrimeTime is like the Red pill I needed because "You just pushed back 591 messages because you were off by a half millisecond"
One important thing the author has missed: FreeBSD. it was very important for them to run FreeBSD for two reasons. 1. The co-founders were very familiar with the internals of FreeBSD. 2. The FreeBSD network stack was (and still is) way better than other open-source operating systems (e.g. Linux). They were able to support 2-3 million concurrent connections on a single commodity server.
They have life and people who care how about you ? You need to comment here to express yourself ? No one left to listen you ? Oh you guys work for groceries worth $600 per month ,$2000 rent and and flex of earning $4000 per month? Thinking elite indians manage all this for $100 and still saves for life. So live a life
@@riwajghimire I was stating a fact. WhatsApp is the most common messaging service in the subcontinent. Maybe you should drink a chai and chill, Riwaj.
While Erlang is functional it is perhaps the least important fact about the language and given the syntax and style of Erlang it is very procedural feeling. I find C/C++/Go/Java devs have little issue moving to it once getting used to replacing loops with maps, folds, and tail recursive functions.
Maps as in applying functions to iterable objects? I learned that Go call key/val pairs for maps after i learned that - So i'm wondering what the actual norm of maps actually is
@@GOTHICforLIFE1 map as in the high order function. Like fold. Check wikipedia for "map (higher-order function)". It is a function / behavior. Not a data structure.
@@GOTHICforLIFE1 Both are map, same word used for different things. But I would say more people would use map for applying a function to every element of a collection and call an associative array a dictionary.
@@disguysnReally? I like its relative simplicity. It might be inspired by a rather unpopular and odd language (Prolog) I really never minded it. I certainly prefer it over a number of other functional languages... including OCaml. I think the pattern matching syntax is the nicest I've seen in any language.
I have a theory that pretty much any scale of software system can be produced if you stick just a handful of engineers in a couple of rooms. If it's a hard problem, give them £300k plus and hire the very best. It will still work out cheaper than 200 drones grinding away in their corporate cubicles. The effort of coordinating all those low skilled hacks and the quality control bureaucracy you'll need will cost far more than simply hiring good people in the first place.
@@bos9824 According to the stats, £300k is around the median for a top-tier engineer in the US and £250k in the EU. If you want the best of the best, you'd have to pay a bit more than the median.
Ok, I'm aware of the US paying that kind of money but where did you get the statistics for the EU??? Median would mean that 50% of software engineers in the EU make more than 250 British Pounds that is absolutely laughable and absolutely unrealistic.
Joe Armstrong is such an inspirational talker and person. In many ways whatsapp was the perfect showcase of the core ideas of what Erlang was designed for. Instant messaging needs to be low latency, highly reliable and capable of handling partial failures. And it was pretty well defined: Do SMS but over IP, with few new capabilities such as user-defined profile pictures.
Prime on CI/CD: "Everyone does that." (2008 btw) On the other end, here I am, trying to convince the company to let me implement pipelines in the repos while waiting for my boss to "deploy" with one-week delays using a copy-paste bash script, saying that it's good enough: 🤡
This title sounds funny. Imagine 32 engineers sitting here and manually sending all those messages. And also it's sad that Telegram now goes on this path of "feature creep". It was a wonderful messenger and now it's full of social networking shit.
3:33 - True, with all due respect, at the end of the day it is a messaging app with not a whole lot of features. By its very nature it is easy to parallelize and scale. Most messages are sent between 2 people and the group limit is 1024 (compared to other apps such as Telegram which support 200.000 or Discord which supports 250.000 members or more). The app existed for over 14 years now, so plenty of development time went into it already. 32 engineers (especially talented ones) is a lot for a problem as narrow as this. Past a certain point you simply can't split responsibilities anymore and you don't wanna end up having 3 senior engineers solely responsible for styling the Send button.
@@joaovmlsilva3509 I didn't assume there were 32 people just for the client, 32 engineers for the whole operation is a lot. And pretty sure they didn't require infrastructure to support 50 billion messages / day from launch day. Back in 2009 very few had smart phones world-wide. Now even people in starving 3rd world countries use smartphones and Whatsapp.
I think that most shops should probably have between 3 and 40 devs depending on the depth of their app ecosystem. It doesn't make sense to have 600 or 40000 developers. That makes communication impossible...
@@ThePrimeTimeagen Netflix does contribute to the source code of FreeBSD, they got even more performance out of the network stack ! Thanks for your reply, I’m honoured 😊
I expected it to say "Kafka" but given how far back it was, ejabberd makes perfect sense! It must have been amazing to work at such a high-growth, engineering-focused company building such a large public-facing service, at that time in internet history
A part of how scalable the backend system is, we miss an important point here: the offline-first app. It's very hard to build an offline-first app even today, where we have the majority of problems already solved by third-party SDKs and packages. Building a seamless offline app in 2008 is some level of engineering to marvel at.
The "-agen" bit seriously never gets old. Also thank you for inspiring me to get back into coding a bit. Starting back with python and became enraptured by using python for AI stuff. Huggingface is awesome lol
SMS in Ukraine is still expensive. We pay for every 100-200 messages, depending on provider. And 100 messages is equivalent in price to 50Gb of internet, on my provider.
@@catto-from-heavenFortunately I am not paying for them. But there are not that many plans, that do not have SMS included. There is an option to make your own plan, but its base cost is higher, than my whole plan. So, we just look for what is closest to what we need. And even then, we still get some things, that we do not use, nor need, but are still forced to pay for. My plan, for example, has 250 minutes per 4 weeks, to call other providers' numbers. Even if I use 10-12 of them at most, I still pay for them, as there is just no better plan, that suits my needs. And there are plans, that have 400 and even 600 SMS, that cost 8-12 times of my plan. There's a lot of them actually. If you have good internet - it's 99% that you'll get a ton of useless SMS with it. I literally never even once heard of anyone, who would use SMS, even if their plan provides them (and it usually does). Though, I get 3-4 daily spam ads via SMS. So yeah, people do not use it. The ad companies do. But we are still paying for them.
everyone has green threads because they were copying Erlang, even the Go guys knew about Erlang but Erlang is even better than Go, Rob Pike also recommends Erlang. Erlang code always lives in a green thread, and all of their green threads are individual erlang runtimes with their own heap so when you kill one of them you are automatically collecting garbage, giving Erlang "arenas" by default. Erlang's hot reloading works due to always running on a green thread, because they can load v1 and v1.1 in different threads, last time I checked it had only support for 2 simultaneous versions.
1:40 I still don't understand why YUI was not maintained. I mean NodeJS and React and all that are good, but nothing beats a well designed, single-source comprehensive JS framework. It was the JDK of Javascript.
another thing about yahoo engeneers , of all the big name .commers, they were the only ones toe also register bitflip domain names , where the rest only rigistered the typo domain names ... they foresaw and closed that vulnerability long before anyone ever came up with abusing it
Turns out (as somebody mentioned in the chat) that Jan Koum is actually Ukrainian. It's the English transliteration that makes it look absolutely non-Ukrainian (I thought he's Chinese, lol) P.S. You read the name correctly, and the surname -is more like Koom ("oo" as in "door")-
Feature creep can be harder than people think. One of the reasons why Linux is a thing, was Linus understanding that there has to be someone with power to say "No" to anything proposed.
i love the impending feeling of i'm doom i get from watching your videos. because i have no clue what 90% of the tings are your speaking about and will looking for a job in like a year
It's important to build a decoupling system with fault tolerance.. that if one part is missing it's not a big deal.. the system can go on in full efficiency.
I love that he got rejected by Facebook, then they acquired his company --7 years later for billions... I've heard it described before as the "most expensive hire ever" 😅
4:02 Until you pointed it out I didn't even recognize that you tried to imitate any accent. And I'm German, so maybe I'm more (or less?) sensitive to a German accent in other languages. Although I funnily enough start to develop a slight Swedish sccent now that I speak Swedish all day XD
We have a very serious communication problem in our "team" right now. I put "team" in quotes because it's actually about 50 people in six work streams that are identical. It's so bad that I can't make any breaking changes to our API at all since it is practically IMPOSSIBLE to communicate with everyone involved in a certain feature since there is no central person who knows who is working on what since the six work streams actually all do the same thing and work on the same code base.
Simple: make a stable application on stable server platforms. You really don’t need thousands of engineers for applications that are that simple. I struggle to understand why even Netflix requires hundreds of engineers full time.
My dude....they may handle 50 billion messages.... but for over a year, the wrong avatar picture of the wrong person showed up in a convo using the desktop app.
That article felt very superficial, just listing the tech stack and common good practices. Does 32 engineers include mobile devs though? Really impressive if so, cause at first they even had Symbian and Blackberry support. To be sure, they've had some embarrassing problems with encryption at first, and Zuck seems to have found a selling price for Jan's idealism. Nowadays, their desktop app is total crap. I only have a few groups with old friends, not even any large or active channels, and it on an older macbook I sometimes use, after a couple weeks it takes couple of minutes to start up, with 160% cpu. Goodness knows what it's doing, it's maybe 500 messages at most, and it's not even rendering anything apart from a progress bar. Seriously, just doing math wth cores and cpu cycles per message hints at some monumental idiocy happening.
Whatsapp should be using the Actor Model with high concurrency, similar to Akka, I participated in an HR interview for a Reactive position and they told me they use Erlang in most of their stack services. The Actor model is so interesting as a design pattern an I think should be spread more among us mortals.
SMS were so expensive in Brazil back then that even now that it is cheap people only use WhatsApp. And some of less tech ones savvy don’t event know what an sms is
I'm surprised this article didn't mention aggregators, since WhatsApp pays Infobip for monitoring, filtering, and managing the loads on their traffic through special routes made for WhatsApp which are configured according to their requirements. WhatsApp doesn't do all of the job with 32 engineers, they're smart about outsourcing things when possible.
Yahoo were pioneers in scaling early day web setvices, so the statement about how early day engineers were among the greatest of their time doesn't surprise me
Maximum of 10,000 Clients times 60 times per second = 600,000 UDP messages per second with my game server. Admittedly, it's just a multiplayer dungeon crawler, so very small packets, and no wasting time with TCP. I just want to know why it took them 32 engineers ;)
@@ThePrimeTimeagen And yes, you were totally right it is [Swedish chef voice] ”Jaan”. Ironically, have also worked with actual SMS protocols, Erlang/RabbitMQ stack back in the day. And these theoughputs are damn impressive. No similar B$ exit story tho.. 😅
50 billion per day means all of the one-in-a-million edge cases will hit you 50 thousand times a day.
truly terrifying
nice way to put it
yet shit happens and things doesn't work so it's accurate lol
no but maybe
At least with Erlang the edge cases won't take down the whole system... Erlangs paradigm was always 'Let it crash', in fact the whole language was designed with Fault tolerance in mind, they knew in critical systems like telecommunications the system needs to always remain up and available, so rather than attempt to defensively program, they viewed there will always be unforeseen bugs. hardware may fail, but they asked how can we ensure those problems never break the wider system.
Every user action is isolated in its own PID, so when an edge case arises in that PID it just crashes, and a supervising process just restarts it.
They found all the mythical 10x engineers, captured them, and locked them in a basement with only laptops with Linux terminals and a mountain of anime pron to keep them company
Those engineers have never been more happy
They are running FreeBSD. Not Linux.
lmao
... with monochrome screens and Vi as editor...
😂😂😂
@@julienmarie_personaltheir servers run FreeBSD, surely they didn't use it to write WhatsApp
For context about FreeBSD, because the article skims over it:
For years Erlang developers preferred FreeBSD because IO multiplexing is done in FreeBSD with something called kqueue, which requires one less syscall than the Linux equivalent (epoll). I believe Linux has iouring now.
Let us just appreciate for a moment that the Erlang VM handles IO so well that the OPERATING SYSTEM ITSELF BECOMES THE BOTTLENECK.
BSD also allows running programs in the kernel memory space.
@@stevenhe3462so can Linux en.wikipedia.org/wiki/EBPF
@@stevenhe3462but shouldn't that also be a security issue?
@@ummijaan1448 not if you know what a computer is
@@qwoolrat sorry bro i thought I knew what a computer is but turns out I don't. Sorry for wasting your time.
Just a tiny technicality; Erlang uses "green" processes rather than green threads. Difference is that threads can share state, memory and address spaces, processes are isolated and share nothing (other than by copying values). Erlang implements its processes in its runtime system and is managed in its VM, rather than using OS threads or processes.
So are you saying that I should start to use Erlang?
Basically Erlang FORBIDS mutable state (stored in RAM)
@@SeanPoulter Its amazing how easy is to create massively scalable things with Erlang, like most of the heavy lifting is really the programming paradigm. I was reading the Ejjaberd source code and I was impressed how little code in relation to the runtime is necessary to create the core of what is Whatsapp.
Try doing that with C you might save a lot of memory, but you lose all scaling as well. Its almost like C is cheating by being unsafe with its memory model full of aliasing and sharing via the use of pointers.
You are also going to need 35.000 engineers because C is a shit programming language that requires a lot of stupid work to manage the memory.
Its ironic that computer were invented to automate industrial process, yet, the very own engineers working with computers decided to not industrialize the most boring and stupid programming task : managing memory. All because in 1970, the PDP11 only have like 8Kb of RAM or something...
But Erlang coming from the telephony world don't use memory to process data, only to control process that control the data flowing in actual wires, that's the magic. Its the control-plane/data-plane separation.
Stored program with mutable shared memory and big linear address space was a mistake, and its not even how hardware works anymore anyways, the so called "C" programmers don't even program bare-metal anymore, its all a fad, its only there for their ego or something, that a modern compiler has to make a modern computer look like a 1970's era PDP11.
And we are forever going to be stuck in this memory model thanks to C and Unix and Intel.
@@sck3570if you want a more readable language you can use Elixir, which basically runs on Erlang.
A manager went to the Master Programmer and showed him the requirements document for a new application. The manager asked the Master: "How long will it take to design this system if I assign five programmers to it?"
"It will take one year," said the Master promptly.
"But we need this system immediately or even sooner! How long will it take if I assign ten programmers to it?"
The Master Programmer frowned. "In that case, it will take two years."
"And what if I assign a hundred programmers to it?"
The Master Programmer shrugged. "Then the design will never be completed," he said.
the tao of programming is a masterpiece
...and one month if you put one good engineer on it.
I was expecting more from the article tbh. Yeah, it lists some things, but it's just a lists of concepts that by now I think it's just standard good practices. (Aside from the lang choice and fBSD OS choice). I was expecting the article to go a little deeper. It's nice to know that they located and fix bottlenecks, but I wanted to know, how they identified, what was the issue and how that got fixed.
yeah, like, do they store the messages in a txt? how do they query that volume of data? that would be interesting to know
Exactly it was so generic, it was like someone had an assignment to write a 500 words essay
It's been a while but I'm pretty sure there are much more thorough articles on the topic of WhatsApp and their Erlang + FreeBSD setup.
Fear of telling too much and then having the information used against you later; fear of clones; industrial secret; or it just takes too much effort to ask the techs how shit is actually working.
If ADHD was a streamer
He indeed has ADHD
Why does he switch context so much?
@@souls.7033 proof?
Being rejected by a company, just to be bought out by the same company, is so vindicating
he probably didn't know how to reverse a linked-list
lol @@mrlectus
Yea it’s not like this one big villain bro
These are different people
Then this L mentality when you’re seeking revenge because you were “rejected”…. Duh
I’m pretty sure such intelligent and successful people are above that
You’re clearly not
Probably wasn't a "Teanworking hardworker and go-getter" that the HR wants@@mrlectus
Signal is fantastic, I absolutely love it. I wish more people were using it though
Just do a sponsored ad bro 😂
agreed
Yeah I want to use it as well. But tell that to my fam, my friends and co-workers lol
@@kevyyarjust hack them, add signal to their phone, and then tell them they wouldn’t have been hacked if they had used signal.
They lost me when they added Cryptocurrencies as In-app Payments
Erlang is all about creating robust distributed systems as seamlessly as possible. The fact that it is functional and you can hot reload is secondary.
LOL, it is distributed because of the immuatabilty that functional programming provides, nothing is secondary
When you don't have useless managers in between engineers can work like cake.
That is true for good and smart engineers. The average engineer needs a manager to tell him what to do. :D
but in reality its the other way around. Even the best manager needs maximum a worse programmer to tell him what to do 😂
I worked in an IoT platform, and we reached a peak that we were struggling to manage. We started talking to some people who managed a lot of messages a day, and they were shocked by our volume at the time I worked there. We were hitting 500 million a day, and passing that mark was a challenge. I cannot imagine how it is to handle 50 billion.
That article screams AI generated.
It describes a bunch of generic best practice things that you would expect from any succesful company.
I was thinking the same thing
Was about to write the same comment. Such a waste of time. Such a shallow and boring article I do not see how it deserved the youtube upload.
actually it didn't explain why and how WhatsApp could handle all that with just 30 engineers
Agreed. I learned nothing from this video. Glad I skipped 90% of it. Dev youtubers really should focus more on providing value with their article-based videos. Most of the time it's empty phrases like "if you've never worked on x problem....it's HARD! Like...actually hard" and other trivial statements. Feel like it's more about engaging the dev community by addressing obvious common concerns, struggles and frustrating experiences as a dev more than providing new insights
This respond from you is that you have not being born before Internet was a thing, I worked with Video Editing/Rendering to built those system in the 1994, the hard drives needed a special software because normal hard drives where "calibrating" and in those moments you where losing frames when you rendering hard core stuff, now days, just put a SSD or NVMe drive and you think problem solve. That is what programming is today, put just more Framework, and in the end you have just built a system even with the newest hard drives/SAN, and your performance will be like if i remember 100 IOPS per hard drive, so in a stripe configuration for performance, you should know that would be a cascading effect, . But you got to love that bill from any of the Cloud providers. If you ever built a successful app. LOL and BTW this was the time when Silicon Graphics had there computers running the HIPPI protocol. To much knowledge have been lost in time. But one thing remains, we that have struggle during the early days, really know how to build system from ground up today.
Having worked at Ericsson, I can tell you that they mostly dont use Erlang these days. Where I worked there was a lot of Scala and Akka, which is funny to me. One really cool feature of Erlang is that they could predict the amount of hardware needed. X amount of messages means Y hardware requirements, 2X messages 2Y hardware.
But the coolest feature has to be the suicide pacts of processes
Akka is cool, if only wasn't running under JVM, and if I'm not wrong is just replicating what Erlang already does. Can't figure out why would they do that.
oh yeah I remember the suicide pacts -- 'links'.
And where are you working now? Or are you retiredd
@@daumienebi I work at Sinch
@@bacon-SG It's mainly funny to me because the Scala/Akka people often talk about Erlangs influence on them. But I agree, Akka is cool
@7:40 No, not really green threads. That article misrepresents what is going on in Erlang. Erlang is using actor processes and has practically a full OS like process scheduler. And because it is built into the language and platform it offers features not found elsewhere and often can't be found elsewhere when the feature is an afterthought.
I think perhaps another advantage that WhatsApp has (or had) was lack or server side storage? Generally messages were just sent and lived on client devices. When you went to a new device, if you didn't do a transfer yourself, you'd start fresh. I'm sure there was temporary storage but when you can cut out those huge database challenges you can save more engineering resources
Thank you for showing where that meme came from, and the potential rapidity of meme generation and sharing. Love it! Am familiar with GIMP but your demo (about halfway through) produced much faster results.
okay I think I still am a javascript advocate but @ThePrimeTime is like the Red pill I needed because "You just pushed back 591 messages because you were off by a half millisecond"
That’s a cool blog post because it shows a text book development of a product.
One important thing the author has missed: FreeBSD. it was very important for them to run FreeBSD for two reasons. 1. The co-founders were very familiar with the internals of FreeBSD. 2. The FreeBSD network stack was (and still is) way better than other open-source operating systems (e.g. Linux). They were able to support 2-3 million concurrent connections on a single commodity server.
hah, turns out the author DID mention FreeBSD :D should've watched the video completely before commenting :D
Most of those 50 billion messages are "Good Mornings!" and replies from India.
They have life and people who care how about you ? You need to comment here to express yourself ? No one left to listen you ? Oh you guys work for groceries worth $600 per month ,$2000 rent and and flex of earning $4000 per month? Thinking elite indians manage all this for $100 and still saves for life. So live a life
@@riwajghimire I was stating a fact. WhatsApp is the most common messaging service in the subcontinent. Maybe you should drink a chai and chill, Riwaj.
Jesus you are confused.@@riwajghimire
@@riwajghimire mad cause you have to shit on the street?
Yahoo brickhouse (and the groups right around there) had the most influential people ever. I wanted to work there SO badly in the early 2000s!
While Erlang is functional it is perhaps the least important fact about the language and given the syntax and style of Erlang it is very procedural feeling. I find C/C++/Go/Java devs have little issue moving to it once getting used to replacing loops with maps, folds, and tail recursive functions.
Maps as in applying functions to iterable objects? I learned that Go call key/val pairs for maps after i learned that - So i'm wondering what the actual norm of maps actually is
@@GOTHICforLIFE1 map as in the high order function. Like fold. Check wikipedia for "map (higher-order function)". It is a function / behavior. Not a data structure.
@@GOTHICforLIFE1 Both are map, same word used for different things. But I would say more people would use map for applying a function to every element of a collection and call an associative array a dictionary.
It's the features around concurrency and distributing work between multiple machines that makes Erlang so great. I absolutely hate the syntax though
@@disguysnReally? I like its relative simplicity. It might be inspired by a rather unpopular and odd language (Prolog) I really never minded it. I certainly prefer it over a number of other functional languages... including OCaml. I think the pattern matching syntax is the nicest I've seen in any language.
4:02 i love that comment.. The difference between 32 and 2000 is about 2000
I have a theory that pretty much any scale of software system can be produced if you stick just a handful of engineers in a couple of rooms. If it's a hard problem, give them £300k plus and hire the very best. It will still work out cheaper than 200 drones grinding away in their corporate cubicles. The effort of coordinating all those low skilled hacks and the quality control bureaucracy you'll need will cost far more than simply hiring good people in the first place.
I will follow this advice for my company, seems sensible and feasible,
where are they paying that kind of money in europe? lol
@@bos9824 According to the stats, £300k is around the median for a top-tier engineer in the US and £250k in the EU. If you want the best of the best, you'd have to pay a bit more than the median.
Ok, I'm aware of the US paying that kind of money but where did you get the statistics for the EU??? Median would mean that 50% of software engineers in the EU make more than 250 British Pounds that is absolutely laughable and absolutely unrealistic.
@@JohnSmith-ir1fo He's not talking about average engineers - he's talking about elite performers.
"classic 2008, everyone was there" 1:13 - I don't know why this cracked me up so much
Behold the power of Erlang and Elixir. Functional programming is the future.
You know what’s really funny about that? That’s how programming kind of started in the first place.
Yes, the irony of it all. The way things are now is way over complicated than it should be. @@airman122469
Joe Armstrong is such an inspirational talker and person. In many ways whatsapp was the perfect showcase of the core ideas of what Erlang was designed for. Instant messaging needs to be low latency, highly reliable and capable of handling partial failures. And it was pretty well defined: Do SMS but over IP, with few new capabilities such as user-defined profile pictures.
RIP
Prime on CI/CD: "Everyone does that." (2008 btw)
On the other end, here I am, trying to convince the company to let me implement pipelines in the repos while waiting for my boss to "deploy" with one-week delays using a copy-paste bash script, saying that it's good enough: 🤡
😂😂
These are some crazy ass busy engineers to work through 50 billion messages per day.
This title sounds funny.
Imagine 32 engineers sitting here and manually sending all those messages.
And also it's sad that Telegram now goes on this path of "feature creep". It was a wonderful messenger and now it's full of social networking shit.
signal is the way
Telegram never was a wonderful messenger, it's spyware
Telegram was always apooky
I like Signal's simplicity, but I wouldn't replace Telegram for it by any means. It's still really useful for communities, way better than Discord.
3:33 - True, with all due respect, at the end of the day it is a messaging app with not a whole lot of features. By its very nature it is easy to parallelize and scale. Most messages are sent between 2 people and the group limit is 1024 (compared to other apps such as Telegram which support 200.000 or Discord which supports 250.000 members or more). The app existed for over 14 years now, so plenty of development time went into it already. 32 engineers (especially talented ones) is a lot for a problem as narrow as this. Past a certain point you simply can't split responsibilities anymore and you don't wanna end up having 3 senior engineers solely responsible for styling the Send button.
There wasn't 32 people just for the client, aws didn't exist, and docker/kubernetes wasn't a thing
@@joaovmlsilva3509 I didn't assume there were 32 people just for the client, 32 engineers for the whole operation is a lot. And pretty sure they didn't require infrastructure to support 50 billion messages / day from launch day. Back in 2009 very few had smart phones world-wide. Now even people in starving 3rd world countries use smartphones and Whatsapp.
You know, old Yahoo was really ahead of its time. They essentially did everything Google and Microsoft/MSN would end up doing, but in the age of AOL.
I think that most shops should probably have between 3 and 40 devs depending on the depth of their app ecosystem. It doesn't make sense to have 600 or 40000 developers. That makes communication impossible...
Some projects have such a large scope that you need a team for every stream. And thus, you get 20 teams and can't figure out who owns which feature
@@dejangegic I disagree. I’ve been to big and small firms. You don’t need that many devs. You need better leadership.
I watched the video just to see when FreeBSD is mentioned in the article. It’s one big reason of the performance !
yeah, i have heard many a good things about it. i also know we use it extensively on our own boxes at netflix
@@ThePrimeTimeagen Netflix does contribute to the source code of FreeBSD, they got even more performance out of the network stack !
Thanks for your reply, I’m honoured 😊
Yeah the network stack is blazing
Normally I have to pay for it. Thanks for letting me know about FreeBDSM
@@jayshartzer844
I expected it to say "Kafka" but given how far back it was, ejabberd makes perfect sense! It must have been amazing to work at such a high-growth, engineering-focused company building such a large public-facing service, at that time in internet history
A part of how scalable the backend system is, we miss an important point here: the offline-first app. It's very hard to build an offline-first app even today, where we have the majority of problems already solved by third-party SDKs and packages.
Building a seamless offline app in 2008 is some level of engineering to marvel at.
BOE THAT WAS FUNNY kkkkkkkkkkkkkk thanks for the great article, as always
This is one of your best videos, and Flip as well.
The "-agen" bit seriously never gets old.
Also thank you for inspiring me to get back into coding a bit. Starting back with python and became enraptured by using python for AI stuff. Huggingface is awesome lol
SMS in Ukraine is still expensive. We pay for every 100-200 messages, depending on provider. And 100 messages is equivalent in price to 50Gb of internet, on my provider.
Why are you still paying for them?
@@catto-from-heavenFortunately I am not paying for them. But there are not that many plans, that do not have SMS included. There is an option to make your own plan, but its base cost is higher, than my whole plan. So, we just look for what is closest to what we need. And even then, we still get some things, that we do not use, nor need, but are still forced to pay for. My plan, for example, has 250 minutes per 4 weeks, to call other providers' numbers. Even if I use 10-12 of them at most, I still pay for them, as there is just no better plan, that suits my needs. And there are plans, that have 400 and even 600 SMS, that cost 8-12 times of my plan. There's a lot of them actually. If you have good internet - it's 99% that you'll get a ton of useless SMS with it. I literally never even once heard of anyone, who would use SMS, even if their plan provides them (and it usually does). Though, I get 3-4 daily spam ads via SMS. So yeah, people do not use it. The ad companies do. But we are still paying for them.
But no one uses them
YES!!!! Love the shout out to Signal at the end. It is the best messenger there is, light-years beyond everything else. ❤❤❤❤❤
They missed DB
Any idea about db that they were using?
everyone has green threads because they were copying Erlang, even the Go guys knew about Erlang but Erlang is even better than Go, Rob Pike also recommends Erlang.
Erlang code always lives in a green thread, and all of their green threads are individual erlang runtimes with their own heap so when you kill one of them you are automatically collecting garbage, giving Erlang "arenas" by default.
Erlang's hot reloading works due to always running on a green thread, because they can load v1 and v1.1 in different threads, last time I checked it had only support for 2 simultaneous versions.
Erlang code lives in Erlang processes (or "green processes") since they are isolated and cannot share state, unlike threads.
At netflix you had less than 2000 engineers. That's not a comparison, you had 2 orders of magnitude more engineers
1:40 I still don't understand why YUI was not maintained. I mean NodeJS and React and all that are good, but nothing beats a well designed, single-source comprehensive JS framework. It was the JDK of Javascript.
another thing about yahoo engeneers , of all the big name .commers, they were the only ones toe also register bitflip domain names , where the rest only rigistered the typo domain names ... they foresaw and closed that vulnerability long before anyone ever came up with abusing it
Turns out (as somebody mentioned in the chat) that Jan Koum is actually Ukrainian. It's the English transliteration that makes it look absolutely non-Ukrainian (I thought he's Chinese, lol)
P.S. You read the name correctly, and the surname -is more like Koom ("oo" as in "door")-
well lets go!
@@ThePrimeTimeagen oops, I meant Kum as in "could" (and that's probably why it's Koum). My English breaks at evenings...
Feature creep can be harder than people think. One of the reasons why Linux is a thing, was Linus understanding that there has to be someone with power to say "No" to anything proposed.
Erlang elixir baby 😂
LFE
First time coming across your channel. Loved the video. Subscribed 👍
i love the impending feeling of i'm doom i get from watching your videos. because i have no clue what 90% of the tings are your speaking about and will looking for a job in like a year
And without kubernetes or serverless crap.... actually languages like erlang are VM's so they have their own docker baked in...
It's important to build a decoupling system with fault tolerance.. that if one part is missing it's not a big deal.. the system can go on in full efficiency.
I know it, Elixir is the way to go!, long live to Erlang!
I find Erlang syntax SO much better, is Elixir really he future? I have to make this choice.
is it faster than Erlang?
Instructions unclear accidentally learnt Go
@@sck3570 No. It compiles to BEAM byte code.
@@thepedrorriva Elixir is an inferior copy of Erlang, claiming to be superior.
1:20 "He double hockey sticks" is the most Flanders statement I've ever heard 😂
Would have been cool to see some specifics on bottlenecks and stuff.
I love that he got rejected by Facebook, then they acquired his company --7 years later for billions... I've heard it described before as the "most expensive hire ever" 😅
The article was like ughhh nothing great just textbook definitions slapped upon.
4:02 Until you pointed it out I didn't even recognize that you tried to imitate any accent. And I'm German, so maybe I'm more (or less?) sensitive to a German accent in other languages. Although I funnily enough start to develop a slight Swedish sccent now that I speak Swedish all day XD
I’ve been learning Erlang but it’s HARD
There was Douglas Crockford gutting javascript and scavenging the good parts.
We have a very serious communication problem in our "team" right now. I put "team" in quotes because it's actually about 50 people in six work streams that are identical. It's so bad that I can't make any breaking changes to our API at all since it is practically IMPOSSIBLE to communicate with everyone involved in a certain feature since there is no central person who knows who is working on what since the six work streams actually all do the same thing and work on the same code base.
Simple: make a stable application on stable server platforms.
You really don’t need thousands of engineers for applications that are that simple. I struggle to understand why even Netflix requires hundreds of engineers full time.
netflix has many applications that arent public facing thats why
i dont know how much i get offended when ever out of nowhere primeagen calls out javascript
Rare Signal W, which is not what I was expecting in a video about WhatsApp
Jan Koum is Ukrainian. The first developer was a Russian. Oh those times...
My dude....they may handle 50 billion messages.... but for over a year, the wrong avatar picture of the wrong person showed up in a convo using the desktop app.
8:34 “Edge lord with a beard.” Got em. Rekt.
Wish Telegram shared more insight into their operation. Apparently they too have 30 engineers and no HR team. Mind blowing
Telegram is great too without group size limitations like WhatsApp. Wonder how they engineered it.
i love this guy fr
Erlang was specifically designed to ensure reliable operation in network and telecom applications, so yeah... makes sense.
The reason the are able to do that is precisely because they only have 32 engineers.
1:18 bro respect for correctly pronunciation his polish name, it such a rare american thing.
That article felt very superficial, just listing the tech stack and common good practices. Does 32 engineers include mobile devs though? Really impressive if so, cause at first they even had Symbian and Blackberry support.
To be sure, they've had some embarrassing problems with encryption at first, and Zuck seems to have found a selling price for Jan's idealism.
Nowadays, their desktop app is total crap. I only have a few groups with old friends, not even any large or active channels, and it on an older macbook I sometimes use, after a couple weeks it takes couple of minutes to start up, with 160% cpu. Goodness knows what it's doing, it's maybe 500 messages at most, and it's not even rendering anything apart from a progress bar. Seriously, just doing math wth cores and cpu cycles per message hints at some monumental idiocy happening.
I cried at the Jan impression
Whatsapp should be using the Actor Model with high concurrency, similar to Akka, I participated in an HR interview for a Reactive position and they told me they use Erlang in most of their stack services. The Actor model is so interesting as a design pattern an I think should be spread more among us mortals.
He clearly didn't do enough leet puzzles to get into facebook. That's probably what happened.
SMS were so expensive in Brazil back then that even now that it is cheap people only use WhatsApp. And some of less tech ones savvy don’t event know what an sms is
huehue brbr
I'm surprised this article didn't mention aggregators, since WhatsApp pays Infobip for monitoring, filtering, and managing the loads on their traffic through special routes made for WhatsApp which are configured according to their requirements. WhatsApp doesn't do all of the job with 32 engineers, they're smart about outsourcing things when possible.
There’s a difference between public data and content moderation, compared to a stored db of data.
Wow, that means each engineer handled 1.5 billion messages just by himself.
Must have taken him ALL DAY!
post makes it sound like the engineers hand delivered each message
Thanks for your video and thanks to Facebook.
Love that ending.
JavaScript catching the stray 😂
Yahoo were pioneers in scaling early day web setvices, so the statement about how early day engineers were among the greatest of their time doesn't surprise me
that’s e-jabbered. Like e-mail.
And jabbered as in jibber-jabber.
Great video though!
I just realized you could do an amazing Jordan Peterson impression
Maximum of 10,000 Clients times 60 times per second = 600,000 UDP messages per second with my game server. Admittedly, it's just a multiplayer dungeon crawler, so very small packets, and no wasting time with TCP. I just want to know why it took them 32 engineers ;)
But yeah good video. And yes, FP shines through baby ❤
How they handle load balancing during peak bours?
Whatsapp is the result of when you put curious and smart people in room together
+1 For name butchering 😅
Excellent points and great vid
// A fellow Jan
hello jan
@@ThePrimeTimeagen And yes, you were totally right it is [Swedish chef voice] ”Jaan”.
Ironically, have also worked with actual SMS protocols, Erlang/RabbitMQ stack back in the day. And these theoughputs are damn impressive.
No similar B$ exit story tho.. 😅
Ty for Signal shoutout.
When you put it that way (how many per millisecond), … damn
that's a lot of engineers for such tiny load
Meanwhile, I can't get my leadership to add a third engineer.
4:46 this part is wild man 🤣🤣