We need to talk about this benchmark
HTML-код
- Опубликовано: 16 апр 2024
- These next.js server side benchmarks are pretty brutal. I don't think they're being discussed fairly though. Wanted to do my best to balance things out, I know y'all LOVE that 🙃
RYAN STREAM • Wiz: Behind the Curtai...
ORIGINAL TWEET
/ 1778442925061394755
github.com/eknkc/ssr-benchmar...
ADDITIONAL SOURCES
github.com/tinylibs/tinybench/
x.com/youyuxi/status/17786877...
x.com/davedbase/status/177916...
github.com/vercel/next.js/pul...
x.com/feedthejim/status/17794...
Check out my Twitch, Twitter, Discord more at t3.gg
S/O Ph4se0n3 for the awesome edit 🙏 
Наука
![ChatGPT Can Now Talk Like a Human [Latest Updates]](/img/1.gif)
As a long time backend guy I can tell you, don’t compare next to half done server frameworks. Compare it to the batteries-included ones - Rails, Laravel, Phoenix, Simfony. You’ll quickly learn that rendering time is both irrelevant and close to them, because what truly matters is parallelization and caching. Next exells in both of those.
You might not know this, but other backend ecosystems also get their “look this memcached C hello world with half a http protocol is 200x faster than this production ready framework with a process manager and server” becnhmarks every couple of months. But most people in those ecosystems already know that this is not why or how you choose a framework. If it was, we’d all be sendfile-serving static html.
I find myself siding with the "backend guys" more and more nowadays...
The performance is all in how you make it. I built a system with Laravel that fetches and caches text to speech audio. It also encodes them to mp3 before saving. Every request starts php process and they each run their own ffmpeg process. This is true multithreading and if one process fails it does not affect the others. The system has been running unchanged for 7 years and I don't think it has ever gone down. I did some performance benchmarks at the time and it was able to really utilize the CPU and give good throughput.
@@DefenderPumatruly respect this. Not because I have ever touched PHP, the technology is irrelevant, but the fact that good design is 99% of the equation to success.
Doesn't server rendering time correlate with server costs? Perhaps not end-all number, but something to look into, as well as CPU/memory/energy requirements?
But I've done years of ASM/SSE programming, so perhaps I'm more sensitive that others.
why do you trade so much performance for that? I don't see any direct correlation besides "it doesn't matter. ship it"
You know what is even faster? Sending plain old HTML
direct from the database
Which needs to be rendered
@@ThectI just send it as a fax. You only need to render it once.
@@Thectwhich is instant with plain html
HTMX mentioned
Benchmarks like this remind me of Microsoft touting how fast Windows 10 booted vs Windows 7, but it was mostly because things were moved out of the the boot process to later, so the time to actually get from pressing the power button to starting to actually use the computer was actually longer. Something something metrics as goals, etc
Ryan from Solid js is a genius.
A personal hero of mine, I'm lucky to call him a friend 🫡
Tom is also a genius.
His room looks like shit
@@t3dotgg I think all the guys name that start with Ryan is a genius
@@everyhandletaken is his name Tom, I just call him King.
Although I cannot explain why this is the content that reignites my spark for development after months of struggling with severe burnout... I am incredibly grateful for your impact.
Thank you, Theo! 👊🙌
You got this, you are not the only one strugging with development burnout
At a super basic level of understanding that NextJS is a heavy tooling framework built on top of ReactJS, people are this surprised? Like people really tripping over this?
This is the is first thing I expect from NextJS being obviously slower.
Programming benchmarks are almost always people testing things that they'd never do in real life because deep down inside they don't actually care enough about what's really faster to ship a benchmark that's meaningful. You'll know someone cares about real world performance when they're profiling their own project trying to get more performance into the features they're ACTUALLY shipping. Isn't it funny no one cares about the custom parsing algorithm you wrote because the string parser in the standard library of your favorite language requires an extra iteration because of 2 extra steps your app doesn't need that costs you 81% performance on an operation you do a thousand times a day but they'll raise hell over a benchmark someone else made, that they don't even understand, which doesn't do anything that resembles a feature that could be put in an application and shipped to users?
Benchmarks are best for comparing different versions of the SAME framework over time to see if new features are introducing slowness or if optimizations are improving performance.
Disabling 90% of a framework in a happy path to win a benchmark is essentially equal to what VW did with Dieselgate. We should think twice before even entertaining such ideas.
modern webdev people would be blown away when they find out the performance of HTML files + AJAX, and we have had that since 2000. Angular appear in 2010, and then the gates of hell were opened. It has been a complete downhill slide from there. People seem to completely forget that a web server is simply a thing that receives a GET whatever and responds some text back. The level of overcomplication that people continue to add in the middle of that is completely insane
this argument of: it doesnt matter if it's slow, because we have other things that are also slow, is an argument from low quality developers. It's also not true, if your request is waiting for a DB request, it can do other things because of async io. Which isnt the case for rendering. Constantly saying it doesnt matter is just a bad message for new developers. Some of them will have to make an app that will need to handle alot of traffic, in that case, it will absolutely matter.
if you're trying to optimize a tenth of a millisecond away while wasting tens of milliseconds elsewhere, maybe reconsider your priorities
I think we as a dev community need to come to terms that selling our projects on benchmarks is purely toxic because it always sets bad expectation when benchmarks almost never focus on real world scenario, but cherry picked or purely synthetic use-cases. There is value to benchmarking. But I am under the belief that after seeing time after time, these solutions that come out that advertise _only_ on the benchmarks, fail to give a real win to end users for the performance targets they're shooting for.
Hey there! Just want to let you know that I appreciate your unbiased and very thorough analysis of these frameworks. Currently, I am loving the Vuejs ecosystem but I love to see what is out there. Keep the good work!
Number of views a subject gets on RUclips isn’t a very good heuristic for how much people “care” about a particular topic.
A great explanation, definitely won't help with the "I didn't watch it but-" zombies calling you a shill though
Look at the top comment 😂
Somebody needs to do a real-world benchmark. Like, an actual app, classify requests into CPU-heavy, memory-heavy and IO-heavy, then run it in the cloud for at least an hour. Then make a report with flamegraphs, p-90s, memory usage over time, etc.
Yeah, I may agree, but I'm afraid not every framework supports the same functionalities that the classification requires, or differs too much architecturally to reasonably compare results. Also, now the performance comes down to the implementation of said classification, which creates another dimension to compare results with.
looking at 28:00 I feel like Remix is the true winner here.
the slower nextjs is, the more money vercel can charge for function exections/cpu hours etc
Vercel needs to make that bucket sir, we could only send the javascript and host it in a CDN for free, but no, we need to overcomplicate everything with Next.js
How far have we fallen that backend routines take more than 100ms per request, I guess if you're using a 3rd party service for every little thing and an ORM that never heard of joins you'll get to that number.
Yeah, this is the part that seems really suspect to me. Lots of apps just display a little bit of data from a simple DB for many pages. Not everything is doing intense data aggregations and graph searches and a hundred remote procedure calls. A use case in which the "business logic" of your request could be sub-10ms seems totally reasonable and worth benchmarking to me. Hell, a handful of Postgres queries could complete in under 1ms. Am I working on toy apps, or have I just successfully avoided a deep minefield of inefficient development practices and abstractions?
I'm a strictly backend for backend dev (was fullstack in the past) I don't use next or react or anything related to it.
But I really enjoy these videos from both an entertainment perspective and an educational perspective.
Because even though it won't benefit me directly, I do care about servers performance.
2:12 "Server-side rendering pages on requests is difficult"
PHP has been doing it since the 90s
33:24 "if you care a lot about how many times your server can generate an HTML page, don't write it in Go or Rust, maybe C or C++"
Not sure about Go, but Rust has mostly/almost identical performance to C++, which is in turn roughly identical to C. C++ wastes 1 register runtime and inflates code size a bit due to RTTI and exception handling, compared to C, but this is rarely a measurable perf difference.
I think you've misheard, he said "go write it in Go or Rust, maybe C or C++", not "don't write it".
Also for your first point, we're talking in the context of a single page architecture, PHP has SSR sure. But it also has page reloads.
I think Theo means in the context of hydration, PHP had no hydration, mainly because browsers don't understand PHP, making PHP hydrate with JS would be a nightmare.. :)
Node, is fine for speed as long as you don't do things that are CPU heavy, and this is the part I think Next / SSR is a problem. I personally think there trying to use Node JS backend for something it's not good at (speed wise). And it's so ironic when Theo blasts PHP for been slow, and it's the exact same reason SSR is slow (server end).
@@_y7ya Yeah, I may have misheard it.
@_y7ya and @keithjohnson6510 Yes, PHP has SSR and reloads, so no hydration is necessary. 🙃 Reloading a simple page can still be faster than a data refresh in most web apps out there today.
PHP can be quite fast, but it's terrible language-design-wise, similarly to JS. Heavyweight frameworks like Symfony tend to be slow, similarly to Next.
@@keithjohnson6510 React SSR will always be slower, best part of React in the client side(SPA) is sending the rendering process to the client device(phone, PC) making the backend light and simple, If you want to do SSR, go back Rails views, Laravel, Django, any of these tools would be better than React SSR.
Next.js is just a Vercel product to make easy money, honestly, I don't see a use case for React SSR.
Awesome, will use this info in my next-python-electron desktop app
Man went from theres no way, to testing it and finding out that it really do be like that, to so what it doesnt matter 😂
I mean, doesn't matter how fast your server reply when you get unhydratable thing.
More like: We need to yap about this benchmark
Edit: Watched the video. Good video.
Still watching bro. You're vids are amazing
Theo: "There's no way Evan You would have left a 2x performance win on the table."
A few moments later:
Evan You: "Anyways, by tweaking a few things in Vue, I made Vue and Next 2x faster in this benchmark."
Optimization is minimizing or maximizing some metric or metrics relative to some other metric or metrics. Everything is always equally "optimized". I can always find a set of metrics and conditions but which anything is optimal. What matters is what things are optimized "to".
All that benchmarking to be held back by database queries.
Well... I've been a next guy for a while and I finally had a project with Remix. I'm not going back to next. Pages does loads faster than Next andddddd it's just a better dev experience .
The best framework is the one you enjoy working with. All these performance worries are gone when the server takes 10s to send a response.
This is the issue I have with 98% of all the hype trains in anything related to code and pulled back from reddit, twitter, etc. - there's too much whiplash to any data without context, without discussion, without really thinking. I appreciate this video more than many of yours. Funny that all the videos you mentioned, Theo, were the ones I've enjoyed the most!
Besides: PHP got a lot faster with PHP 7 and 8.
eew php..
@@backupmemories897 still holds the world together. The large favor Facebook did to the web world with Hack was to prove that PHP can be fast. While Hack didn’t get much traction, PHP optimization did.
eeeewww php
@@backupmemories897eww JS!
I have an aneurysm hearing PHP
Totally agree that these benchmarks are generally not of much use, apart from potential dev optimisations.
In the same way our recent fascination on FCP ( and the whole SSR debate. Yes, first page render matters, but is it really that hard to tell if your website first page render is slow or not, out of curiosity I have a very heavy dynamic dashboard, cache disabled none build, FCP = 339 ms, oddly warm start is slightly slower (surprising), 390ms, of course running locally so warm start will of course be faster in production. In both cases doing a refresh I can't really see a delay. Throttling fast 3G does of course show this delay, but warm start still feels snappy. But even then on a Client side render there are ways to make this feel fast again, eg. embed some minimal HTML, doesn't have to be a none hydrated version either as long as the user has some feedback your good. And of course the bonus here, less work for the server, more requests per second, (not rocket science that bit).
Saying the above SSR still has it's uses, SEO been the main one, and if your website has very little dynamic content then the simplicity of SSR is also a bonus.
As someone who wrote an RSC framework that outperforms Next in this benchmark I can confidently say: Meh, it probably doesn't matter much, but a fun thing to play with
I do care about these videos. Seeing stuff like this on twitter is distressing as it implies clear mistakes and no nuance in decision making. Seeing this framed and explained in a reasonable way is exactly what I like about you and your channel.
42 minutes of theo coping
real
How did you even watch the video that fast, it was published 42mins ago
@@javierflores09he didn't. He just hates Theo
@@javierflores09probably at 1.5-2.0 speed. Usually what I do.
@@javierflores09Ever heard of 2x playback
you can make a fair benchmark. use cypress and check when required text from database apear on page. and repeat this in cycle. this way you can track time to load pages completely. also measure stress on servers and interpolate it to estimate amount of requests that can be handled and users served. so you will have time to load page and amount of users served on some configuration. these numbers are meaningfull and can be used to chose best option. I made this test to compare nuxt with server rendering vs client rendering and for me it was faster to render on client and amount of users to serve was many times larger.
It's interesting timing -- I just (like, 2 days ago) got my home fiber upgraded to 2gb/s, and although I doubled my throughput, almost NONE of the services I use sped up /at all/ -- and literally zero of them are what I would consider "fiber fast". I would love us to have some kind of standardized end-state tests, as it feels like DX has overtaken UX as the important benchmark.
Stats can be followed. Zach star made a very nice video about it.
Also one of the reasons i prefer pnpm over bun
There's some wild opinions in these comments. Y'all know you don't have to pick an extreme side on this, right? Most people in the world just use what works.
I find it hilarious how they represent the "Average Time (ns)" with 9 decimals 😂 Some people just have never learned how error/precision calculation works.
I like your color scheme in VS Code. What is it called?
@Theo: bias of confirmation ;) That's a trap we all fall into one time or another ;)
I have implemented the same backend in axum rust and in go std lib. In the beginning Rust was faster. But as I added logging, database trips and caching on disk, Rust was insignificantly faster at serving static files from disk at 21000req/s vs go's 19000req/s, while being slower than Go at SSR than Go, 4000 vs 3500req/s, connected to the exact same DB etc. So, hello world performance does not matter. You need to build a small POC and test it rigorously and then decide on the tech.
People still don't understand frameworks aren't built specifically for them, it's built for all of us.
im sorry guys i've just starting nextjs and i see people recommended not to use "use client " directive in every component in nextjs so if i want to make a fully animated web with nextjs with say library like framer motion and i want animation on each component i shouldn't use next ?? , why ?? plz help me
if you're using animations in every page, just use regular react. imo for a `fully animated web` nextjs isn't really the right tool
@@prawnstars3160 Next still makes a lot of stuff easy for you out of the box rather than if you were to use react by itself. Just because you aren't partial pre-rendering and using use server on every single page doesn't make it not useful. There are still pages + ways to use both framer and the server
Dude just use React, forget this whole SSR, it's just a waste of time.
Well now I wanna make an actually good End to End Benchmarking system
while coding in nextjs my vscode get slower 100x saving file take 1min its hard to code
I don't JS but i felt kinda aggravated, because this breaks a "Liskov substitution principle of sorts". Sending "all new content"!="sending how to update the existing content", that's the obvious one. But even if they do produce the same result, you can't compare them if they don't do the exact same thing in the back, like for example one having a request log and the other not so. Don't get me wrong, benchmarks are great, but there's the pitfall of comparing apples to oranges and people not even realizing they aren't the same fruit...
Benchmarks are like Test cases: we make assumptions about things that are not real in both cases.
Yes there can be greeat Benchmarks and Test cases, but most do not get them close to right.
I'll always watch and Ryan or Theo have something important to say.
A bit of STATISTICS would actually show the real expectations. Beyond just averages and total time, we should see standard deviation, so that you know what the overall shape of the results looks like. A wide std dev means the results vary significantly between executions. A tight one shows the results closely align. And 2 std dev show where 95% of the expected results will be. Factoring out those outliers from an overall understanding of what's expected is critically important.
It's like comparing walking straight back home, with doing your shopping on the way back. One will take a bit longer, but you end up with fresh groceries.
The Next.js community should create a beautiful benchmark UI template with Shad UI to own these people back.
The underlying problem from people picking frameworks from a benchmark is from a trend of restricting themselves to using popular software instead of working from their problem itself. If you are making a app that is tested by the benchmark, then that benchmark is a perfect reason to pick a framework. I really blame the job market. Example, recruiters think because your a great Next.js developer, its not worth investing into you learning Remix.
So TLDR; Use Svelte (always the tldr)
Theo can we get the source code just to compare what it takes to initialize each of these technologies.
The original poster definitely was trying to mislead people with those statistics. Having something render in 100 nanoseconds as opposed to 1 nanosecond means that it is 100x slower, but 100 nanoseconds is still unnoticeable to the user. The other benchmarks showcased in this video were still (if I recall) sub 10ms which is still plenty fast for what was tested.
25:20 ish - Is this why I see some of the behavior I do in Svelte/Kit? Sometimes it seems like 'things happen twice' not sure how to put it.. I'm a newbie with little experience learning dev with Svelte/Kit/mongoose. ALways love the vids. Thank you.
Theo has Linus Tech Tips’ voice and looks like Artosis’ son
Yup still here at the point you ask
I only watched the first 5 minutes, but here's the perspective from someone who just took a computer organization class in his second year of college:
speedup formula, with p being the portion of the program being sped up, and s being the amount:
1-p + s*p
from the formula we see that even when the speedup approaches infinity, the program is still at most p times faster.
if you speed up 1% of the program, no matter how much, it can only be up to 1% faster.
The people writing apps that make 10 concurrent requests on page load, with some of them taking over 3s, are complaining that the rendering lib takes too many milliseconds. Right.
42 minutes??
I've said it before and I'll say it again, the only benchmark that matters is your actual use-case.
100% this.... i ported my soon coming SaaS from gatsby to astro and reduced "scripting" more than 2x down (SSR shaves at least 80ms; +more like 3x quicker if it's SSG of course). sure rendering is 2x slower, still
some thing run so fast that it will be done before the the need is there . Cinema . You arrive , the screen say The end
I'm one of the devs that do care. And I do care because even if yes the database query will take more time than the rendering, if a benchmark tells me the framework I use will only ever be able to render 10 pages per second per thread, it means I will never be able to serve more than 10 pages/s anyway (and here I'm no longer talking about round trip time for the end user, but CPU load!), which means I'll have to scale way sooner than I would have with another, more optimized framework. So I see it purely as "how much compute it will require to serve XXX concurrent users, and will it costs me significantly more ?". At least that's the fear I would have. And I think your bias is thinking in terms of total request time, but those benchmark instead give us an idea of compute-required-per-request. This is indeed negligible in terms of total request time, but it is very much relevant when it comes to the compute you'll need to do to serve a certain amount of requests.
Granted, if that's a concern, I guess JS on server isn't a good idea and you should write your backend in Go, Rust, or whatever floats your boat. But I feel like answering this misses the point. If the solution to perf problems is just "use a lower level language", we would never care about performance at all. I think performance needs to be kept in mind at any time, independently of the language you're using. Let's say they can make the Next renderer 2x faster, which also means 2x less compute (not quite, but let's make this equivalence) : as a Next user, I'd love it.
then if you do care about render time why are you using one of the slowest modern popular languages there is
@@bigpod Compared to what, Rust / Go / Zig?, these languages are great, apart from Rust (ugliest code ever developed), sorry (Prime). :) Node will easy handle 50K request/sec, yeah, maybe Zig/Rust 80K on the same hardware, I think were speed is a problem with Node is when making it do more than just serving requests, and this is were SSR / Hydration is were things start to fail. But taking speed out of the equation, having FrontEnd / BackEnd using the same language is a massive bonus, especially if using TS as remote calls are really simple.
@@keithjohnson6510 as a backend guy i will always say JS on backend is a mistake and will always be a mistake, chose anything else, my personal(when doing things for myself) choice is between go and c# depending on what i want (C# is when i care about dev side more then perf, go other way around), what you choose for professional usage should be mainly mandated by people you have and long term maintenance.
@@bigpod As someone who does both backend / frontend, I'll be sticking with Node. If I was only backend, then yeah I would likely change (maybe Zig), but losing the integration between backend / frontend is not something I would give up for a little gain in speed. Doing some C# code atm in Visual Studio, not a fan, but still better than Rust.. :)
@@bigpod So because you don't use the faster language, you shouldn't care about render time ? 🤔
This sounds like Python VS C benchmarking. Sure, C is waaaaaay faster. But python gets you walking in a day, while you're still coding C for three weeks, before you can run. Both have their value and I like and use 'em both. The speed benchmark means very little without contexts like use case and place in the entire stack.
6:14 you are right I actually don’t care at all about benchmarking. And I actually did not click on the video the first time I saw it 😅. Listening it now while doing the dishes.. I assume I’m not the only one who’s not into benchmarking. Nevertheless, you still made a boring subject decently interesting 😊
ms likely still have something that are still paced at 400 milisecond because it's ideal .
Thanks for staying Solid on your videos Theo, and I appreciate article videos like this not React videos make sure those don't come Next
Considering the caching and prefetching all these frameworks are almost identical. Next.js with RSC, server actions and streaming has a really good DX that these benchmarks don't take into consideration.
Feedback:
I’m watching because nextJS for me developing is slow and I want a fix.
I won’t watch vendor lock in as I don’t care. If I want to lock into a specific vendor I know what I’m up for and I’m willing to make that choice if it be if it’s me.
High bills I also don’t care, you need to use the vendor correctly and actually consider the billing for your solution.
Specifically Netlify I also don’t care as I am locked into AWS and Vercel.
It's easy to say "But look it's faster !", just disable all the heavy lifting.
Noticed your cool terminal Warp.... make a video about it?
Another thing to mention. Next SSR just got 18% faster from node stream optimizations. Thats not in this benchmark I think
Yeah next is cool its dx wise the best to manage and deploy on vercel but i dont like the costs of vercel too much for a developer from a third world country
it's really irresponsible for devs to make dx their top priority, when companies lead with dx i get suspicious
@@pcap8810this is an idiotic take. DX means work comfortability. Show me an employee who thrives in a job that makes them uncomfortable and I will stop thinking you're an idiot
I’m still watching 🤪
server is likely waiting . Next what ?
41 vs 241 op/s is important for scaling
Isn't this a common trope for benchmarks?
They do not necessarily test real world performance, be it framework testing or hardware testing. Look at all the smartphone and desktop testing, the whole benchmark testing phenomenon has made that its sometimes is made to make them look good in benchmarks
So, what I've learned is that Vue will soon be the best framework on the planet, proven by this benchmark.
HTML + JS frontend making AJAX (these days JSON instead of XML) to the backend for updated data. JS frameworks just overcomplicate everything. Sure I could make my system for work use a JS framework, but why? There's no reason, mine is already extremely fast for what it needs to do, and less complicated. React, NextJS, etc... are all just fads as far as I am concerned.
Useful benchmarks are really hard to write. Even if you try to do everything right and leave the details to a dedicated benchmarking framework things can go wrong. In my thesis I tried to benchmark some parallel sorting algorithms with Google Benchmark and some of them displayed a 15x speedup when going from 4 to 8 threads. Absolute nonsense. Had to measure all times manually to get results that made sense. Don't trust benchmarks you haven't verified yourself
Surprised you got through all of that without saying "premature optimization"
I like looking at benchmarks, but not all benchmarks are made equal.
Weaponizing benchmarks is weird and framework tribalism is weird. We're all frontend developers working towards the same goals. Playing favourites with frameworks is how you pigeonhole yourself. If you've only ever used one framework, if you've ever referred to yourself as a "react/angular/vue/etc developer" instead of a "frontend developer" then consider branching out a bit.
I tried out the Nextjs tutorial and deployed it on Vercel, did everything as Vercel told. It is SLOW AF both during development and deployed. It's nowhere as fast as the old school PHP.
24 million nanoseconds are 24 microseconds, the choice of such a small unit for such large values is a bit deceptive
One thing I never see being discussed when people talk about SSR is overall memory footprint and memory/session... 🤔🤔🤔
Key takeaway: use Remix
npm run dev on next 14 is painfully slow (even with -turbo) on m1 mac, even with small projects
Why isnt there a comparison to vanilla webcomponents? This is perpetuating reacts mith.
Ppl are mad because their new shiny favorite framework is slower than all other xD ouin ouin ouin so funny
One day there will be an episode where you finally say "fk all this crap use Laravel", soon.
I don't like next.js, but I don't like bs numbers more.
I hate Twitter so much. You can just write some nonsense without any context and everyone will take it as truth and start screaming about it.
where angular at bruh
Parallel on edge used to be 8