► What should I test next? ► AWS is expensive - Infra Support Fund: buymeacoffee.com/antonputra ► Benchmarks: ruclips.net/p/PLiMWaCMwGJXmcDLvMQeORJ-j_jayKaLVn&si=p-UOaVM_6_SFx52H
It would be cool to see what the Ntex framework (Rust) can do. It’s what the original creator of Actix moved-on to after giving Actix over to the community. It seems like it performs really well on the benchmarks I’ve seen it in, but it would be cool to see it in this format here against the other rust frameworks!
Honestly the fact that you took feedback and went back to make further comparisons that incorporated the feedback from different communities may make it it one of the best videos of this kind around.
Thank you for correcting your mistake by rerunning the tests with optimisations! Mistakes are completely normal, but seeing corrections is surprisingly rare...
As a Rust dev, its really cool to see Zig and Rust trade blows. considering picking up Zig where i would normally reach for C due to the dev environment.
@@tbird81 honestly, I think zig would’ve been a better choice for kernel development rather than rust. It’s more like a modern C and I think the veteran C devs are more likely to accept zig.
@@CoolestPossibleName there is some truth to that though. not that it doesn't have benefits over c, but i don't know if the benefits are strong enough for most companies to justify it over c. odin is kind of in that same place. in my opinion, c3 has more compelling arguments going for it.
Note on the previous benchmark: Zig was in 'debug' mode, which is the default, and I didn’t know that. I found a "production-ready" Dockerfile and thought it was properly compiled, so I’m not the only one who made that mistake.
Zig has only a safe mode to detect faults at runtime, in real word application you would release optimised for space or speed without the runtime failsafe
Release Safe does not ensure (full) memory safety. It adds stuff like UB checks (some of it is missing in Rust), but no "full" memory safety like in rust.
even after those optimization performance and compile command change, rust still on the top. its so satisfying how the developer and man behind rust and actix can achive such performance.... they are the GOAT
Python/Nodejs for fast prototyping. Golang for production 98% of the time. Rust less than 2% of the time for certain super low latency requirements like HFT.
@@Z3U5.0g Yep. Its cool that its so fast, but I would expect a systems language to have a worse DX than Go. But I think choosing feature complete framework in any language (e.g. Laravel) is also a very good solution. No "stack" hopping
@@NabekenProG87DX comes down to preference. I found golang panicked too easily and the error handling was pretty bad. Rust's pattern matching, union types (enums) and macros make it a joy to work with if you can push through the steep learning curve. But it comes down to preference in the end
Its really cool to see this sort of "live"/iterative benchmarking developments with feedback. When you feel like it's all as good as it can get, I think it would be really cool to see a blog post with everything you learned, mistakes and all, all in one place!
Rust can actually be FASTER than Zig, C or C++ in some cases because their ownership and aliasing rules for &T and &mut T allow for more aggressive optimization techniques. Although the downside to Rust lifetimes is that it restricts which programs the compiler will mark as semantically correct, that is also one of its benefits. Since Rust programs are restricted to a smaller subset, the compiler can also make more aggressive assumptions/optimizations of those valid programs.
Unfortunately there are a lot of cases where the assembly generated by LLVM is suboptimal, like additional memory/register copies. It would be very interesting to see a Rust compiler that is built from the ground up to take advantages of all available guarantees.
You are talking about avoiding unnecessary copy operations? Does Rust actually have an advantage over Zig here? Zig does seem to do rather aggressive optimizations in this department already. Since Zig is going to have its own backend, it might be able to benefit more from Zig specific optimizations in the future. For the time being, most effort in Zig seems to go into optimizing the compiler performance (incremental compilation, etc.). I'm curious to see what happens when the focus shifts to code generation and optimization later on.
@@michaelutech4786 If Zig uses pointers, it should be the exact same. I suspect iterators are the main reason why. The Zig/C implementation probably does bounds and null checks every iteration of a loop. Rust's iterators don't have to.
@@ionutale1950Depends wholly on the application. In-mem caching can help a lot with db hits. If you’re using a global state you can batch appends and read state once per server on ticks and it’d stay pretty light. If you’re displaying tables too big for cache for numerous reports and clients then yeah you’ll have some DB pains. But you likely wouldn’t get that kind of volume for that kind of thing.
rust is a low level lang that doesn't feel low level at all, also it will not allow you to use pointers like crazy, and forces you to write everything thread safe, u can only make shitty code if u use "unsafe" keyword. also "thread_local" macro feels like magic, its wonderful.
@@Cuca-hn3md I wouldn't call Rust a low level language. It is a compiled language that allows you to write extremely efficient code and provides memory safety despite manual memory management. Those are indeed two highly desirable traits in system software, but that doesn't make Rust a low level language. Go, with its relatively low expressiveness, is a low level language - most of what you can write in other languages will take at least one and a half as many lines of code in Go. Rust, however, doesn't have this problem. With Rust, you can write a database driver or a Web application equally well, without writing much more code than if you'd use languages traditionally used for each job. In fact, if you compete against plain C in the database driver, chances are the Rust code will be smaller. I'd venture to say Rust be about on par with Python for the Web application.
@@a0flj0 "level" could also mean how much control you have over what will actually happen in the computer. The fact that you can write inline assembly in Rust makes it IMO as low level as C.
150k with 2 cores that is a lot! i did some similar tests, but on my local machine with NestJs and Go Fibber, not even close: - NestJs 400 req/s - Go Fibber around 10k dam, rust is fast. i am wondering if there is anything faster than Rust, like a server is C or C++
I mean they would roughly be the same performance, since they are "manage your own memory" languages. what differs is the different HTTP framework implementation which actix seems to have the edge
Zig is something like 0.6 version lang m8. Id say that was pretty fucking impressive how zig ran that C library. Only small margin behind fully fledged rust. Id say that Zig will literally eat C
@@ionutale1950I would normally argue for it just being a bad zig implementation, but with the weird behavior we see, I suspect that most Zig server frameworks are just C copies, and a lot of C frameworks are pretty stable and reliable, but outdated when it comes to algorithms and implementations.
It's interesting that, from an ideological/approach perspective, it seems like toward the upper bound of performance limits, Actix is interested in conserving device resources, while Zap ensures nearly 0 request failures
I'd like to see a comparison between the same web app on Rust Actix-Web and Elixir Phoenix. It might look like apples and oranges but it'll be interesting to see the memory utilization of the two apps. Elixir "processes" are light threads. Actix implements its own light threads but I understand those still use more memory than Elixir (BEAM) processes
Actix-web stopped using the Actix actors, tho. And yes, Rust actors cannot be as efficient as BEAM actors, because BEAMs are preemptible (need assembly for that) and all state in BEAM is immutable.
@@LtdJorge Thank you. I wasn't aware of the Actix-web change. I still think the comparison would be interesting to see though. It'll put specific numbers to the memory usage
I was thinking of running this benchmark on a standalone VM, and I even have everything ready, including the systemctl service files. However, the problem is that when you run something without limits, it starts to affect all other services on that VM. For example, I use node-exporter to collect basic metrics, so without limits, those exporters and agents begin to degrade as well, causing gaps in your monitoring system. You can set cgroup limits using systemctl, but it would be similar to how Kubernetes handles it.
Hello Anton, I saw that a PR for setting the zap and zig stdlib implementation to 2 workers. Is that PR already included in this benchmark video? I really appreciate the effort you have put in these benchmark video. Thank you!
@@jiinyeongoh7458 no actually i just cloned that lesson including this PR which uses only 1 worker and run with it - github.com/antonputra/tutorials/pull/271/files i got pr to increase the number of workers later.
Hi @AntonPutra I've been following your channel for a while now and I absolutely love your content on comparing tests and benchmarks of different programming languages and frameworks. I was wondering if you could consider doing a similar comparison between in-memory databases like Redis and RabbitMQ. I'm particularly interested in seeing how they perform under various workloads and use cases. Thanks for all the great content!
Thank you! Actually, the second benchmark should have been Redis vs. Memcache, but every time I get distracted by reading all the comments. I'll get there eventually 😂
You also have to remember that Rust has hardened a lot of small optimization into std and some core libraries, this test would be more fair result in 10 years to give Zig more time to mature. But I'm team rust, even if it's slower I'll have less bugs with the safety features.
What was the request output? Could you make it more "real world". Like an json response of 1kb, 10kb, 100kb. And then another test that compiles some zig to html or rust to html (using thr same 1kb, 10kb, and 100kb html outputs). Could you also add golang to these ?
@sighupcmd A proposal for a more practical benchmark: add two dimensions, JSON/API and HTML/Templating. This moves away from pure theory towards real-world use cases. Research into optimal engines for each will be needed, trading some purity for more relevant metrics.
@@Matty0187 again, you propose to bench libs responsible for json/html (where are a dozens of them, btw). What's the point of this here? Here author used "minimal" required stack to make service respond. That's the correct way. Also, there's only one real use case: yours (your service's).
@@sighupcmd errors, rps, and latency while great metrics, a response of "hello world" from a server is a naive. We should measure bytes or packets out as an additional dimension. I still wouls love to see how common template and JSON responses effect the rps, latency, and error rates cross the servers
Thank you for this comparison, can you take into account the content of the queries? Actix sends more Data ( Http headers ) so you need Bytes/sec metrics. Also it would be really interesting to change Serde by Sonic-rs which is more performant, which will make Rust win even more points. And why not use Hyper directly instead of Actix? Thanks again 🙏🏻
It's the difference between caching a lot and no caching at all. Caching means availability remains constant as long as you can keep caching requests at the cost of using more memory. Actix is almost completely functional; no caching.
I don't agree with the "only use Rust if you need the performance" since I prefer the DX of Rust over Golang. Once I was used to the ownership concept and wrote some async code, I don't feel like Rust is slowing me down at all. Although the problem that if one function becomes async and the async keyword has to propagate to all caller functions is pretty annoying. Learning Rust was probably easier for me since I come from a C/C++ background, I guess it's much harder to learn Rust if you only used garbage collected languages before.
I would love to see a odin language. I'm just learning it and I know it's made primarly for games and simulations, however I would love to see how it could perform for a game server applications. I write much less code when I use odin comparing to javascript or C++ (I use the same C library in all cases - raylib), I would like to try it as a game server but I'm not sure about what performance I can expect. I really like this video, great point of view on those languages.
Hi love your videos. Can you do some more main stream comparison like php/laravel or JS/Node vs Rust/Actix vs Go/Fiber, as most of the web is php. Can we can see how big the performance difference is?
the pure zig implementation is another league on memory usage, it's heavily optimized to avoid heap allocations as much as possible at the expense of using the heap for parallelism with that memory usage it can scale an order of magnitude better than both zap and actix actix is an overall winner here, it has a mature and more featured api, memory usage & req/s better than zap zap is little immature as a project, it will benefit from both code optimization & from zig future versions
Zap is interop with Node to begin with. I use Zig for low level stuff, they dropped async out of stdlib as of 0.11.0. The fact Zap runs that well is still pretty amazing. The amount of code tweaking. I would expect Rust to handle volume, and even beat C if there were a framework. I'm suggesting allocation schemes a hot spot. On a smaller level, evaluate the defers. It would be an exquisite test. No js. Just system cranking. I like both languages, but I wouldn't think Zig would be ready for that. LLVM differences? Great test though. We're talking framework differences also. That's why I say isolate, and create a system grinder.
Would be cool to see this in comparison to older languages like C(++) and java, that might not be using the newer techniques, but do have years of optimizations
Honestly with how much more knowledge we have now, it would be very surprising if you can "optimize away" the difference. It will still be C at the top unless there is a paradigm shift (like the functional style of Actix).
Probably a fair result then. The “std” zig implementation there is using only 1 thread, so 80k shows us where the hardware is getting saturated The flatline from there on is (likely) kubernetes throttling that maxed out single CPU Not surprisingly- all 3 implementations that are using the same llvm optimizer here perform very close up to that point. @anton - send me some links plz to how to duplicate your test setup so we can properly tune this for the environment it’s running on. Need to iterate the config to find a suitable balance here Fun project! Working with low level code (be it C / Rust / Zig whatever) is like tuning a car to match the course it’s racing on :) At least the audience is enjoying the show :)
haha, sure I'm using EKS which is a managed Kubernetes cluster in AWS. If you have never used it before it can be problematic and expensive. You can use minikube locally but you would never be able to reproduce the benchmark locally. But at least you can test it and compare it. So minikube - minikube.sigs.k8s.io/docs/start/?arch=%2Fmacos%2Farm64%2Fstable%2Fbinary+download Kubernetes deployments - github.com/antonputra/tutorials/tree/main/lessons/208/deploy you can apply with kubectl apply -f ... also i have clients as well - github.com/antonputra/tutorials/tree/main/lessons/208/1-test also kubectl apply -f ... all images including clients are compiled for both arm and amd, so you can run them on mac as well if you need any help please let me know
@@AntonPutra Cheers thanks. Yeah, I will go back and review some of your playlists, and try out those deploy scripts. Managed EKS is probs the way to go then. I dont mind spending a $bit to get it right .. just have to remember to turn it off when im done :) Interested in finding the code for your load-tester container - is that in a lesson somewhere ?
@@steveoc64 i have older version written in go with prometheus metrics - github.com/antonputra/tutorials/tree/main/lessons/201/client but it is really outdated but can give you an idea. i recently rewrote that client in rust but it is not ready for release yet. with go client i used m7a.8xlarge instances with new rust client i only need 4xlarge
@@steveoc64 Also, make sure when you run those tests in the cloud that your application and client are deployed in the same availability zone. AWS has data transfer charges between regions and between availability zones, but inside the same zone, it is free. If you start running load tests between zones, it could cost thousands of dollars in data transfer fees, so be careful.
@@AntonPutra 208-tuning pr done :) Added a third zig implemention that uses another new network lib See what happens! Really appreciate the large scale test you are offering here - it’s a non trivial test, done right Kudos
Cool to see that actix performance is basically the same as what is essentially a C implementation. Also you managed to show clearly what the effect is of setting cpu limits in kubernetes. When it starts throttling the stdlib one you see the latency jump up and the amount of requests it can handle plummet. Would be interesting to see what would happen if you run the exact same test but on a m7a.large without cpu limits. I would expect it to then not have such a sudden breaking point but just gradually increase latency with sustained max requests per second
I was thinking of running this benchmark on a standalone VM, and I even have everything ready, including the systemctl service files. However, the problem is that when you run something without limits, it starts to affect all other services on that VM. For example, I use node-exporter to collect basic metrics, so without limits, those exporters and agents begin to degrade as well, causing gaps in your monitoring system. You can set cgroup limits using systemctl, but it would be similar to how Kubernetes handles it.
@@AntonPutra Ah I see what you mean, cgroups have cpu shares which is what requests map to (after calculations). Just running in k8s without limits should already give the insights since all nodes will have the same kubelet overhead
Exactly what sort of hardware does m7a.large provide? Are they quoting real cores, or are they including hyperthreaded cores ? It does make a bit of a difference depending on what the app does Would be interesting as well to compare x86 vs Arm offerings on AWS
really you're comparing the libraries used rather than the languages themselves. Hypothetically every compiled language would be compiled to the same most-efficient machine code with safety checks where necessary.
As a Rust dev, I was expecting more from zig on all fronts in this test, as I keep hearing that zig is more performant and has better memory control, while being somewhat memory safe. For it to not beat rust in any of these fields really showed me that any low system programming will accumulate a die hard, cult like, core, that will try to make a sell for things the language cannot even still do. In this case, zig is still far too young to compete with a powerhouse like Actix, and once mature, probably will match it more than beat it. Thats fine. The true selling point of the language is to be c like to ease dev in it and to improve over c. Rust is at the extreme opposite and you might fit in its targeted audience or not, but regardless, it delivers on the core premises of memory safety at no performance cost. In this regard, taking sides seems pointless. As a dev, you are not your tool, you just use the tools that you have at your disposal and are the most expert with even if its not the greatest one for the task (I look at you nodeJs and Flask). So learn both, become an expert on the one you prefer and respect those who prefered the other option.
this hit rock bottom. golang for fast prototyping and changing requirement. rust for rewriting core business or legacy system which cost is too expensive
Python/Nodejs for fast prototyping. Golang for production 98% of the time. Rust less than 2% of the time for certain super low latency requirements like HFT.
@@Z3U5.0g sadly, i am currently working in a company with very little care of prototyping. once it's implemented in python / node, it will stay that way. make sure your company can afford rewrite. mine is not that friendly toward this. that's why prototyping in golang is better, consider that we are only talking about web server in this video. python is a trap for complex web server. nodejs is better but we know "javascript"
How do you time-lapse thru the results so it looks like they are going really fast? At first I thought it was real time and then I realized you are replaying them somehow on grafana
yes, zig was in 'debug' mode, which is the default, and i didn’t know that. i found a production-ready dockerfile, and i thought it was properly compiled, so i'm not the only one who made that mistake.
Zig is a pleasant language that is a joy to program in. It feels proper language, its approach to memory safety is more correct than Rust making it way simpler. I still not a big fan shorthand syntax but I can live with it. Zigg feels like a sensible refinement of C language for system programming. I like it, but clearly work is still needed on its stdlib for better implementations, being native compiled they should not see differences unless there is some inefficiencies in the library implementation .
well this is the old one in Go - github.com/antonputra/tutorials/tree/main/lessons/201/client but i just rewrote it in rust and probablly will release it soon
Well, it's a little bit more complicated, then 100%. 1 CPU = 1000 millicores = 100% of a 100ms cgroup interval. So 2 CPUs = 200ms, or 100ms for each core. Here's the link for more details: ruclips.net/video/h2pCxj_Fkdc/видео.html.
@@AntonPutra Kubernetes solves the problem of inefficient and complex server infrastructure by creating a replacement system of equally inefficient and complex container infrastructure.
@@illegalsmirf agree, most of the time you are much better running your apps on VMs especially in the cloud when you have target groups/autoscaling for vms etc
► What should I test next?
► AWS is expensive - Infra Support Fund: buymeacoffee.com/antonputra
► Benchmarks: ruclips.net/p/PLiMWaCMwGJXmcDLvMQeORJ-j_jayKaLVn&si=p-UOaVM_6_SFx52H
node vs bun vs deno please
kotlin native vs kotlin jvm vs go vs expressjs vs hono
Express vs Laravel vs Django vs Fastapi vs Fastify
c vs c++ vs rust vs go
It would be cool to see what the Ntex framework (Rust) can do. It’s what the original creator of Actix moved-on to after giving Actix over to the community. It seems like it performs really well on the benchmarks I’ve seen it in, but it would be cool to see it in this format here against the other rust frameworks!
Honestly the fact that you took feedback and went back to make further comparisons that incorporated the feedback from different communities may make it it one of the best videos of this kind around.
thanks, i always read every single comment and improve with each new video and fix mistakes
Thank you for correcting your mistake by rerunning the tests with optimisations! Mistakes are completely normal, but seeing corrections is surprisingly rare...
yeahh, Anton is cool
always!
As a Rust dev, its really cool to see Zig and Rust trade blows. considering picking up Zig where i would normally reach for C due to the dev environment.
🤭
It could be useful in the Linux kernel.
@@tbird81 honestly, I think zig would’ve been a better choice for kernel development rather than rust. It’s more like a modern C and I think the veteran C devs are more likely to accept zig.
That and it cross compiles with C/uses LLVM. Just seems to make a lot of sense.
How fucking good that not even 1.0 zig took the C library and ran it like a fucking champ. Zig has great future ahead
Hardly. It has no benefits over C.
@@KushLemon You clearly haven't used zig enough
@@KushLemon*It has an ever-so-slight benefit over C23. It has significant benefits over C17/C99/ANSI
Not sure if this is serious or bait lmao
@@CoolestPossibleName there is some truth to that though. not that it doesn't have benefits over c, but i don't know if the benefits are strong enough for most companies to justify it over c. odin is kind of in that same place. in my opinion, c3 has more compelling arguments going for it.
FYI: Zap pre-allocates memory during startup, expecting higher loads… so memory tests don’t mean much.
true
Is that called arena allocation?
@@StingSting844 that is correct
@@StingSting844no, its just preallocating so it doesnt need to do any malloc or systemcalls later
Note on the previous benchmark: Zig was in 'debug' mode, which is the default, and I didn’t know that. I found a "production-ready" Dockerfile and thought it was properly compiled, so I’m not the only one who made that mistake.
I'm a rust developer and all of my applications are written in Rust using Actix-web. These results are so stress relieving to see. Thank you.
My pleasure, but I have yet to test Ntex. They say it's the best :)
I would like to see this compiled with Zig safe considering that is the biggest selling point of rust is memory safety.
Zig has only a safe mode to detect faults at runtime, in real word application you would release optimised for space or speed without the runtime failsafe
well i did compiled zig in safe mode in the previous benchmark 😊
Release Safe does not ensure (full) memory safety. It adds stuff like UB checks (some of it is missing in Rust), but no "full" memory safety like in rust.
@@AntonPutra Ah, I see! Thanks for letting me know 😅
Is it that relevant if your using actix which uses unsafe all over the place?
even after those optimization performance and compile command change, rust still on the top. its so satisfying how the developer and man behind rust and actix can achive such performance.... they are the GOAT
fr
Python/Nodejs for fast prototyping. Golang for production 98% of the time. Rust less than 2% of the time for certain super low latency requirements like HFT.
@@Z3U5.0g Yep. Its cool that its so fast, but I would expect a systems language to have a worse DX than Go. But I think choosing feature complete framework in any language (e.g. Laravel) is also a very good solution. No "stack" hopping
@@Z3U5.0g imagine Go with memory arenas (like zig) ❤
@@NabekenProG87DX comes down to preference. I found golang panicked too easily and the error handling was pretty bad. Rust's pattern matching, union types (enums) and macros make it a joy to work with if you can push through the steep learning curve. But it comes down to preference in the end
Its really cool to see this sort of "live"/iterative benchmarking developments with feedback. When you feel like it's all as good as it can get, I think it would be really cool to see a blog post with everything you learned, mistakes and all, all in one place!
yes, I'll keep collecting PRs with optimizations and eventually release a video
I really appreciate these videos. Thanks for putting in the effort to make them and thanks everyone who submits PRs.
thank you!
Rust can actually be FASTER than Zig, C or C++ in some cases because their ownership and aliasing rules for &T and &mut T allow for more aggressive optimization techniques.
Although the downside to Rust lifetimes is that it restricts which programs the compiler will mark as semantically correct, that is also one of its benefits. Since Rust programs are restricted to a smaller subset, the compiler can also make more aggressive assumptions/optimizations of those valid programs.
Unfortunately there are a lot of cases where the assembly generated by LLVM is suboptimal, like additional memory/register copies. It would be very interesting to see a Rust compiler that is built from the ground up to take advantages of all available guarantees.
I think the most optimizations come from iterators in this case. There probably is a whole lot of bounds and null checking in the C/Zig libs.
@@dark0sv i think zig has a plan to remove dependency from llvm
You are talking about avoiding unnecessary copy operations? Does Rust actually have an advantage over Zig here? Zig does seem to do rather aggressive optimizations in this department already. Since Zig is going to have its own backend, it might be able to benefit more from Zig specific optimizations in the future.
For the time being, most effort in Zig seems to go into optimizing the compiler performance (incremental compilation, etc.). I'm curious to see what happens when the focus shifts to code generation and optimization later on.
@@michaelutech4786 If Zig uses pointers, it should be the exact same. I suspect iterators are the main reason why. The Zig/C implementation probably does bounds and null checks every iteration of a loop. Rust's iterators don't have to.
100k req/s - like a few small towns clicking in app at the same time
Me when I forget to turn off my autoclicker on that one shitty Browser game ... on 20 machines
exacly, but with that load, you need like 16 vCPU for the database, and i may not be even close
@@ionutale1950Depends wholly on the application. In-mem caching can help a lot with db hits. If you’re using a global state you can batch appends and read state once per server on ticks and it’d stay pretty light. If you’re displaying tables too big for cache for numerous reports and clients then yeah you’ll have some DB pains. But you likely wouldn’t get that kind of volume for that kind of thing.
@@houstonbova3136 really? cache can do that?
wow, this must be 2024, where cache is being used
that's possible when it comes to a trading system.
Rust vs C++ is that one thing we all want from a very long time! 😄
ok, i'll get there 😊 just a couple more js frameworks
@@AntonPutra Thanks a lot buddy! You can also try Elysia vs Express on both Bun and Node.
Nonetheless thanks for all the benchmarks 😄
Rust is such a beast, will learn it asap for side projects kkkkk
Try out Loco (based on rust) for a Ruby on Rails like setup that helps you get started, or Jeremy Chone on youtube with Rust 10x :)
rust is a low level lang that doesn't feel low level at all, also it will not allow you to use pointers like crazy, and forces you to write everything thread safe, u can only make shitty code if u use "unsafe" keyword.
also "thread_local" macro feels like magic, its wonderful.
@@Cuca-hn3md I wouldn't call Rust a low level language. It is a compiled language that allows you to write extremely efficient code and provides memory safety despite manual memory management. Those are indeed two highly desirable traits in system software, but that doesn't make Rust a low level language. Go, with its relatively low expressiveness, is a low level language - most of what you can write in other languages will take at least one and a half as many lines of code in Go. Rust, however, doesn't have this problem. With Rust, you can write a database driver or a Web application equally well, without writing much more code than if you'd use languages traditionally used for each job. In fact, if you compete against plain C in the database driver, chances are the Rust code will be smaller. I'd venture to say Rust be about on par with Python for the Web application.
It's nice, but it's quite unique and takes time to learn
@@a0flj0 "level" could also mean how much control you have over what will actually happen in the computer.
The fact that you can write inline assembly in Rust makes it IMO as low level as C.
Thanks for running and sharing test results!
my pleasure!
150k with 2 cores that is a lot!
i did some similar tests, but on my local machine with NestJs and Go Fibber, not even close:
- NestJs 400 req/s
- Go Fibber around 10k
dam, rust is fast.
i am wondering if there is anything faster than Rust, like a server is C or C++
I mean they would roughly be the same performance, since they are "manage your own memory" languages. what differs is the different HTTP framework implementation which actix seems to have the edge
@@OrtinFargo zig sopposed to "roughly be the same" or even better, but is not. we should test and let the data decide
Zig is something like 0.6 version lang m8. Id say that was pretty fucking impressive how zig ran that C library. Only small margin behind fully fledged rust. Id say that Zig will literally eat C
@@rasvatissi580 i think you are right. zig is great
@@ionutale1950I would normally argue for it just being a bad zig implementation, but with the weird behavior we see, I suspect that most Zig server frameworks are just C copies, and a lot of C frameworks are pretty stable and reliable, but outdated when it comes to algorithms and implementations.
It's interesting that, from an ideological/approach perspective, it seems like toward the upper bound of performance limits, Actix is interested in conserving device resources, while Zap ensures nearly 0 request failures
interesting
@AntonPutra This is just an observation though; it could just be a design *_consequence_* , and not a decision lol
I'd like to see a comparison between the same web app on Rust Actix-Web and Elixir Phoenix. It might look like apples and oranges but it'll be interesting to see the memory utilization of the two apps. Elixir "processes" are light threads. Actix implements its own light threads but I understand those still use more memory than Elixir (BEAM) processes
Actix-web stopped using the Actix actors, tho. And yes, Rust actors cannot be as efficient as BEAM actors, because BEAMs are preemptible (need assembly for that) and all state in BEAM is immutable.
@@LtdJorge Thank you. I wasn't aware of the Actix-web change. I still think the comparison would be interesting to see though. It'll put specific numbers to the memory usage
ok noted!
Rust proved it's top, i don't care about zig, elixir, go or anything else.
@@iulikdev ok
Would love to see more golang stuff! Thanks for all the content anyway :)
thanks! django vs go next :)
Interesting memory vs cpu trade-off. Depending on throughput it could mean that the zig implementation would be better suited for embedded devices.
i have a new pr with zig optimization, i'll test it soon - github.com/antonputra/tutorials/pull/280
I was thinking of running this benchmark on a standalone VM, and I even have everything ready, including the systemctl service files. However, the problem is that when you run something without limits, it starts to affect all other services on that VM. For example, I use node-exporter to collect basic metrics, so without limits, those exporters and agents begin to degrade as well, causing gaps in your monitoring system. You can set cgroup limits using systemctl, but it would be similar to how Kubernetes handles it.
Hello Anton, I saw that a PR for setting the zap and zig stdlib implementation to 2 workers. Is that PR already included in this benchmark video?
I really appreciate the effort you have put in these benchmark video. Thank you!
@@jiinyeongoh7458 no actually i just cloned that lesson including this PR which uses only 1 worker and run with it - github.com/antonputra/tutorials/pull/271/files
i got pr to increase the number of workers later.
thanks for nicely formatted and informative videos, subscribed
thank you!
Hi @AntonPutra
I've been following your channel for a while now and I absolutely love your content on comparing tests and benchmarks of different programming languages and frameworks.
I was wondering if you could consider doing a similar comparison between in-memory databases like Redis and RabbitMQ. I'm particularly interested in seeing how they perform under various workloads and use cases.
Thanks for all the great content!
Thank you! Actually, the second benchmark should have been Redis vs. Memcache, but every time I get distracted by reading all the comments. I'll get there eventually 😂
You also have to remember that Rust has hardened a lot of small optimization into std and some core libraries, this test would be more fair result in 10 years to give Zig more time to mature. But I'm team rust, even if it's slower I'll have less bugs with the safety features.
Sure, I'm mostly testing frameworks, so as they mature, I'll rerun my tests again.
@@AntonPutrayeah, I guess a yearly checkup would be great
What was the request output? Could you make it more "real world". Like an json response of 1kb, 10kb, 100kb. And then another test that compiles some zig to html or rust to html (using thr same 1kb, 10kb, and 100kb html outputs). Could you also add golang to these ?
adding json parsing: that will additionally bench json parser lib performance, lowering the quality of "pure" stack bench
interesting, I'll see what i can do to make it short
@sighupcmd A proposal for a more practical benchmark: add two dimensions, JSON/API and HTML/Templating. This moves away from pure theory towards real-world use cases. Research into optimal engines for each will be needed, trading some purity for more relevant metrics.
@@Matty0187 again, you propose to bench libs responsible for json/html (where are a dozens of them, btw). What's the point of this here? Here author used "minimal" required stack to make service respond. That's the correct way.
Also, there's only one real use case: yours (your service's).
@@sighupcmd errors, rps, and latency while great metrics, a response of "hello world" from a server is a naive. We should measure bytes or packets out as an additional dimension. I still wouls love to see how common template and JSON responses effect the rps, latency, and error rates cross the servers
Actix / Rust still rock hard !
Zig / Stdlib seem to have great memory usage.
it was optimized that way specifically for the test
It's time for c vs c++ vs rust vs go
Do people actually use C for web servers?
@@justsomeguy8385 use case might be there for low latency applications not sure still would like to see a comparison if possible
just couple more and i get to c++ i promise 😊
well i may use facil.io directly in c without zig wrapper
@@AntonPutra ... subscribed 🙂
Thank you for this comparison, can you take into account the content of the queries? Actix sends more Data ( Http headers ) so you need Bytes/sec metrics.
Also it would be really interesting to change Serde by Sonic-rs which is more performant, which will make Rust win even more points.
And why not use Hyper directly instead of Actix?
Thanks again 🙏🏻
@@DevRJPro make the pr
ok noted, yes it really sends more data and i actually have network usage metrics, i may show it next time
Good to see Rust remains competitive. I wonder what causes rust to drop out on the latency test though. Sobering weird with an allocator?
The memory usage gives it away. Actix is very functional. The processor overhead difference is due to caching.
yes it's caching
Looks like an opportunity for improvement!
Thank you. Great job
thank you!
Some things are weird here.
The availability drops of the rust app and the memory use of the native zig app.
It's the difference between caching a lot and no caching at all. Caching means availability remains constant as long as you can keep caching requests at the cost of using more memory. Actix is almost completely functional; no caching.
The native Zig app is 1Mb which isn't much different from the Rust app.
correct
love this series Anton!
thank you!
I don't agree with the "only use Rust if you need the performance" since I prefer the DX of Rust over Golang. Once I was used to the ownership concept and wrote some async code, I don't feel like Rust is slowing me down at all. Although the problem that if one function becomes async and the async keyword has to propagate to all caller functions is pretty annoying.
Learning Rust was probably easier for me since I come from a C/C++ background, I guess it's much harder to learn Rust if you only used garbage collected languages before.
thanks for the feedback
Previous Benchmark - ruclips.net/video/VxW0ijXAfOs/видео.html
Do a laravel vs express vs django benchmark. Battle of the interpreted
I love these followups!
thanks i do my best
I would love to see a odin language. I'm just learning it and I know it's made primarly for games and simulations, however I would love to see how it could perform for a game server applications.
I write much less code when I use odin comparing to javascript or C++ (I use the same C library in all cases - raylib), I would like to try it as a game server but I'm not sure about what performance I can expect.
I really like this video, great point of view on those languages.
thanks will try! someone already suggested odin in the past
Dear Anton, please benchmark Rust vs C, thank you!
i was thinking about it, but do you know any other C web framework besides facil.io?
Hi love your videos. Can you do some more main stream comparison like php/laravel or JS/Node vs Rust/Actix vs Go/Fiber, as most of the web is php. Can we can see how big the performance difference is?
yes about to start in a day or two
do bun vs go comparison
ok will do soon
nice work
thank you!
Nodejs vs php. I am interested in learning new insight on how to optimize both of them.
Exactly. I have been requesting this too. But he seems too focused on Rust :(
i imagine v8 would outperform php
thank you! will do!
Who use php in 2024?
Also i do want to see the Ntex vs Actix!!!!
haha, ok soon!
@@AntonPutra yes please!
As usual great content :)
thank you!
пожалуйста сравни go с node js, gin/fiber vs nest.js/next.js + простые запросы к бд, может пулл соединений, очень интересно!
horosho, obyazatelno sdelau! zapisal
the pure zig implementation is another league on memory usage, it's heavily optimized to avoid heap allocations as much as possible at the expense of using the heap for parallelism
with that memory usage it can scale an order of magnitude better than both zap and actix
actix is an overall winner here, it has a mature and more featured api, memory usage & req/s better than zap
zap is little immature as a project, it will benefit from both code optimization & from zig future versions
thanks for the feedback
Zap is interop with Node to begin with. I use Zig for low level stuff, they dropped async out of stdlib as of 0.11.0. The fact Zap runs that well is still pretty amazing. The amount of code tweaking. I would expect Rust to handle volume, and even beat C if there were a framework. I'm suggesting allocation schemes a hot spot. On a smaller level, evaluate the defers. It would be an exquisite test. No js. Just system cranking. I like both languages, but I wouldn't think Zig would be ready for that. LLVM differences? Great test though. We're talking framework differences also. That's why I say isolate, and create a system grinder.
thanks for the feedback, yesterday i got a new pr with optimizations so will test soon - github.com/antonputra/tutorials/pull/280
Amazing. 🎉
thanks!
Spring boot with virtual threads please
yes soon
Would love to see these three against .NET, js and Java/kotlin.
noted!
can you do a actix vs django rest?
django is next
Compare C++ too . I want to see something
up
soon!
A comparison of zap ReleaseSafe vs ReleaseFast would be interesting.
maybe, thanks
php (laravel) vs other.. why.. because in my country .. we have cult that php is best web app ...
haha, ok will do!
This just makes me more in awe of Actix.
i was told that Ntex is even better
@@AntonPutra Making an Ntex vs Actix video would be interesting
@@almuhajer6760 will do, i'll add axum as well cause last time i didn't generate enough load
Could you do a comparative with Actix and Spring or some Java framework?
ok noted!
Next is nim vs c++
noted!
Can you also do Python? would be cool to see how much slower it is as compared to these compiled languages
yes django is next
Would be cool to see this in comparison to older languages like C(++) and java, that might not be using the newer techniques, but do have years of optimizations
Honestly with how much more knowledge we have now, it would be very surprising if you can "optimize away" the difference. It will still be C at the top unless there is a paradigm shift (like the functional style of Actix).
I'll try. Any language is only as good as the DevOps engineer running it in production😂
Probably a fair result then. The “std” zig implementation there is using only 1 thread, so 80k shows us where the hardware is getting saturated
The flatline from there on is (likely) kubernetes throttling that maxed out single CPU
Not surprisingly- all 3 implementations that are using the same llvm optimizer here perform very close up to that point.
@anton - send me some links plz to how to duplicate your test setup so we can properly tune this for the environment it’s running on. Need to iterate the config to find a suitable balance here
Fun project!
Working with low level code (be it C / Rust / Zig whatever) is like tuning a car to match the course it’s racing on :)
At least the audience is enjoying the show :)
haha, sure I'm using EKS which is a managed Kubernetes cluster in AWS. If you have never used it before it can be problematic and expensive. You can use minikube locally but you would never be able to reproduce the benchmark locally. But at least you can test it and compare it.
So minikube - minikube.sigs.k8s.io/docs/start/?arch=%2Fmacos%2Farm64%2Fstable%2Fbinary+download
Kubernetes deployments - github.com/antonputra/tutorials/tree/main/lessons/208/deploy
you can apply with kubectl apply -f ...
also i have clients as well - github.com/antonputra/tutorials/tree/main/lessons/208/1-test
also kubectl apply -f ...
all images including clients are compiled for both arm and amd, so you can run them on mac as well
if you need any help please let me know
@@AntonPutra Cheers thanks. Yeah, I will go back and review some of your playlists, and try out those deploy scripts.
Managed EKS is probs the way to go then. I dont mind spending a $bit to get it right .. just have to remember to turn it off when im done :)
Interested in finding the code for your load-tester container - is that in a lesson somewhere ?
@@steveoc64 i have older version written in go with prometheus metrics - github.com/antonputra/tutorials/tree/main/lessons/201/client
but it is really outdated but can give you an idea. i recently rewrote that client in rust but it is not ready for release yet. with go client i used m7a.8xlarge instances with new rust client i only need 4xlarge
@@steveoc64 Also, make sure when you run those tests in the cloud that your application and client are deployed in the same availability zone. AWS has data transfer charges between regions and between availability zones, but inside the same zone, it is free. If you start running load tests between zones, it could cost thousands of dollars in data transfer fees, so be careful.
@@AntonPutra 208-tuning pr done :)
Added a third zig implemention that uses another new network lib
See what happens!
Really appreciate the large scale test you are offering here - it’s a non trivial test, done right
Kudos
Not today Zig 🤣, but we know you're younger.
wdym by "we".
Soon, accepting new PRs😊
try Fast API vs node js Vs php laravel
ok noted!
Can you expand on the 50% CPU usage autoscaling? I think it's an interesting detail.
sure, here is an example - github.com/antonputra/tutorials/blob/main/lessons/181/1-hpa/cpu-memory/k8s/2-hpa.yaml#L20
Cool to see that actix performance is basically the same as what is essentially a C implementation.
Also you managed to show clearly what the effect is of setting cpu limits in kubernetes. When it starts throttling the stdlib one you see the latency jump up and the amount of requests it can handle plummet. Would be interesting to see what would happen if you run the exact same test but on a m7a.large without cpu limits. I would expect it to then not have such a sudden breaking point but just gradually increase latency with sustained max requests per second
I was thinking of running this benchmark on a standalone VM, and I even have everything ready, including the systemctl service files. However, the problem is that when you run something without limits, it starts to affect all other services on that VM. For example, I use node-exporter to collect basic metrics, so without limits, those exporters and agents begin to degrade as well, causing gaps in your monitoring system. You can set cgroup limits using systemctl, but it would be similar to how Kubernetes handles it.
@@AntonPutra do you have cpu requests set for the other services? That should also give them their share of cpu
@@Kavantix in k8s yes (for example cadvisor/kube state metrics etc), on VM usually you don't set limits
@@AntonPutra Ah I see what you mean, cgroups have cpu shares which is what requests map to (after calculations). Just running in k8s without limits should already give the insights since all nodes will have the same kubelet overhead
Exactly what sort of hardware does m7a.large provide? Are they quoting real cores, or are they including hyperthreaded cores ?
It does make a bit of a difference depending on what the app does
Would be interesting as well to compare x86 vs Arm offerings on AWS
AWESOME, i think axum would perform a little better here. And could you compare JAVA framework vs node ones?
well based on the previous benchmark Axum is a bit slower, but i'll do java soon
Go vs NodeJS / NodeJS vs Rust!
next, go vs node
Interesting that rust seems to perform better but is a lot more unstable.
i'll do some more tests including data pipelines in the near future
really you're comparing the libraries used rather than the languages themselves.
Hypothetically every compiled language would be compiled to the same most-efficient machine code with safety checks where necessary.
yes, i'm mostly focused on frameworks and libraries used in everyday development
maybe benchmark python vs go vs rust? To showcase the "tiers of performance"?
yes in a couple of days
As a Rust dev, I was expecting more from zig on all fronts in this test, as I keep hearing that zig is more performant and has better memory control, while being somewhat memory safe.
For it to not beat rust in any of these fields really showed me that any low system programming will accumulate a die hard, cult like, core, that will try to make a sell for things the language cannot even still do. In this case, zig is still far too young to compete with a powerhouse like Actix, and once mature, probably will match it more than beat it.
Thats fine. The true selling point of the language is to be c like to ease dev in it and to improve over c.
Rust is at the extreme opposite and you might fit in its targeted audience or not, but regardless, it delivers on the core premises of memory safety at no performance cost.
In this regard, taking sides seems pointless. As a dev, you are not your tool, you just use the tools that you have at your disposal and are the most expert with even if its not the greatest one for the task (I look at you nodeJs and Flask).
So learn both, become an expert on the one you prefer and respect those who prefered the other option.
Unfortunately seems like most comments are about "winning'. This is the first time I've ever seen this phenomenon in programming
thanks for the feedback. as a DevOps, I try to stay neutral and fair to all the languages I deal with on a day-to-day basis
Cool!!!! i will try it myself
👍
Cool! may be you can examine the MOJO language?
Interesting, I'll take a look. It's a very new language."
Can you please make performance tests for kotlin vs other languages?
sure, in the future!
Really nice video! Could you please compare actix with ntex framework on rust?
yes, soon! i was told ntex is the best
@@AntonPutra yeah, I read that it is from the creator of actix. Will be waiting!
Could you add axum in the benchmark? Would make a lot of sense to compare it to actix.
i'll make another one actix vs axum vs ntex
this hit rock bottom. golang for fast prototyping and changing requirement. rust for rewriting core business or legacy system which cost is too expensive
Python/Nodejs for fast prototyping. Golang for production 98% of the time. Rust less than 2% of the time for certain super low latency requirements like HFT.
@@Z3U5.0gYeah. Golang is the 98% use case language. Rust is only if you absolutely need it or have to because of concerns where golang can't be used
@@Z3U5.0g sadly, i am currently working in a company with very little care of prototyping. once it's implemented in python / node, it will stay that way. make sure your company can afford rewrite. mine is not that friendly toward this.
that's why prototyping in golang is better, consider that we are only talking about web server in this video.
python is a trap for complex web server. nodejs is better but we know "javascript"
yes, I just rewrote my clients in Rust to switch from 8xlarge to 4xlarge
Can we please please please finally have Kotlin Ktor vs Java vs Dotnet
ok soon!
How do you time-lapse thru the results so it looks like they are going really fast? At first I thought it was real time and then I realized you are replaying them somehow on grafana
oh just using editing software to speed up
At least we all can agree in something, CPP sucks...
@@AntonPutra C++
haha thanks, long day :)
Common Rust W
yes for now
LOL, so the last benchmark was optimized rust vs zig with NO OPTIMIZATIONS? xDD
yes, zig was in 'debug' mode, which is the default, and i didn’t know that. i found a production-ready dockerfile, and i thought it was properly compiled, so i'm not the only one who made that mistake.
Now let's do the same using C and C++, no wrappers, instead of the amateurish languages shown here 😃
do you know any other library except facil.io for C?
⬇ Team Rust button
🤣
⬇️ Team Rust Sub-Button
Common W for Rust
Hell Yeah
Team rust bottom
I need rust to fail
sure, please send me PR
Why?
@@fabiopetrillo because some people liked to see the world burn (though i dont like rust but had to use on tauri applications)
Keep dreaming. This won't happen! 🦀❤️
@@RustIsWinning the higher you fly the harder you fall
Rust literally destroys the competitors
well, as soon as I can, I'll prove otherwise😂
@@AntonPutra Waiting 😄
Actix is wild
it is :)
Rust is awesome
it is
Zig bros malding in the chat
i tried 😔
Zig is a pleasant language that is a joy to program in. It feels proper language, its approach to memory safety is more correct than Rust making it way simpler. I still not a big fan shorthand syntax but I can live with it. Zigg feels like a sensible refinement of C language for system programming. I like it, but clearly work is still needed on its stdlib for better implementations, being native compiled they should not see differences unless there is some inefficiencies in the library implementation .
take the L
Honestly, skill issue. I find rust more appealing.
@@u-k I simply love working with Rust. Took me like 3 years but I'm there
Hell no. I will rather write C++ over zig anyday.
Can't age with you. I find Rust programming more joy.
I don't like the way how Zig trying to be secure.
Would you consider sharing the source code of the benchmarking client? At least I did not find after a short search in your repositories.
well this is the old one in Go - github.com/antonputra/tutorials/tree/main/lessons/201/client
but i just rewrote it in rust and probablly will release it soon
Ничего себе акцентище. Даже без перевода все понятно )
staraus :)
Crab 🦀 still wins 😭
@@ibrahimmohammed3484 Always! 🦀❤️
i tried
Zig in debug mode is compared to Rust optimised. Embarrassing, delete the video.
Run in Docker! What a troll.
debug mode???
github.com/antonputra/tutorials/blob/main/lessons/208/zap-app/Dockerfile#L15
Does 100% CPU means 2 cores from your limit: 2 or it means 1 core ?
Well, it's a little bit more complicated, then 100%. 1 CPU = 1000 millicores = 100% of a 100ms cgroup interval. So 2 CPUs = 200ms, or 100ms for each core. Here's the link for more details: ruclips.net/video/h2pCxj_Fkdc/видео.html.
Summon C++ magic developers to handle twice as many requests as Rust. 😄
will see :)
Go (Fiber) is it faster than Rust (Actix)?
Possibly, I'll compare them soon.
please do php swoole
ok noted!
People who use things like Rust and Kubernetes are being different for the sake of being different. It's just the latest fashionable thing.
Sure, but Rust isn't much older than Kubernetes :)
@@AntonPutra Kubernetes solves the problem of inefficient and complex server infrastructure by creating a replacement system of equally inefficient and complex container infrastructure.
@@illegalsmirf agree, most of the time you are much better running your apps on VMs especially in the cloud when you have target groups/autoscaling for vms etc
They tried so hard😂😂😂😂😂 they even bringed a "specialist"😂
I tried as well, maybe I should have tried Graviton 🤔