On modern solid state storage all the olde go faster tweaks are counterproductive. I can entirely understand Canonical’s decision to switch to NONE scheduler. Thanks for showing us some examples in support.
@@VioletGiraffe Yes, especially ensuring actual writes to storage match the block size of flash media to reduce the number of times a block has to be rewritten (this speeds things up significantly and also reduces wear on flash storage).
Modern ssd have controllers that operate on information not available to the o/s. I/O schedulers are a problem programmers love to solve, but sadly, it's mostly obsolete today.
Always better to let the hardware do it...but my experience with early SSD's even hardware controllers can have bugs in one case so bad the SSD self destructed just one day after the warranty expired.
Deep content like this is some of my favorite YT content. This is the stuff you’d only be able to dog out of lwn forums back in the day, now I can listen while driving. Thanks DJ!
Just watching your videos i think you have enough knowledge to build And break system.. you have very in-depth knowledge in what you're talking about.. salute you DJ
Fedora also compresses btrfs by default in level zstd:1 - so that could also make a difference - especially on low performance hardware - however, actual throughput shouldn't be affected.
@@CyberGizmo 🤔 hm. this is interesting. I've heard of memory/disk lag when hammering btrfs, and the same thing not being a case when using xfs or ext4. could this be the case? does this mean more memory usage as well? i love the btrfs compression (because i use nixos) but perhaps xfs would be a better choice.
I noticed this the other day when I updated my laptop. I backed up 35GB from Ubuntu 22.10 onto a USB HDD and it took 20~25 minutes and when I copied it back to 23:04 it took 10~15 minutes
Doesn't that also depend on the read write speeds of your USB HDD vs internal SSD? Also how & what (what parts of system, apps, user data, etc) did you back up?
@@therealb888 Just documents. I have done this transfer at every Ubuntu upgrade for 2~3 years , same external HDD and same internal SSD . Usually the time each way is not noticeably different. With 23.04 there was a definite change.
@@PaulG.x Thanks, I guess it's worth the upgrade. In my case I want to do a full system backup, all repos, apps, libs, configs & app data like browser history, bookmarks etc without using cloud sync or use an e2ee cloud sync. On a side note do ensure you're checking for drive health & try to have a third backup so you don't suffer from sudden drive failure.
@@therealb888 Over years of Linux use , my preference on my main desktop has evolved to be a fresh install of the OS (no separate /home ) and manually install the apps I use - that only takes an hour now. I back up the browser and email client profiles and restore them to the freshly installed instances. All my data is on dedicated storage devices , not on the system drive.
I have aarch64 build of fedora 38 running in the UTM virtual environment. And it has "none" io scheduler on block devices by default. So... there should be some glitch in kernel device type detection code or in dev rules. Maybe.
I have got my dbus rules set up so that: * If it is a spinny platter hard drive, it will assign it the BFQ io scheduler * Else if it is using NVMe, then assigns it the "none" scheduler * Otherwise, it uses mq-deadline I feel that in the vast majority of cases, this will give me a pretty good choice for IO scheduler on that device.
I noticed Fedora would almost hang sometimes with large file copies so maybe they are trying to fix it... I am afraid to switch from 36, it is at least mostly working properly.
I am on 37, I feel that. My device is also encrypted so if I have to backup to my external for this computer on its last breath, it will probably be turned into a server when I update. And yes, as a current workstation, this all works. I think I will wait for 39 or even 40.
Sooo, I'm a bit confused. In the end it didn't really mattered the scheduler used, but the file system. Why then the discussion about the schedulers ? Is it because that's what you initially thought was the culprit ? Or you simply wanted to talk about them ? Or something else ?
TL:DR Below 1. Ubuntu 23.04 has faster file I/O than Fedora 38 in benchmarks. 2. The difference in performance is due to different I/O schedulers used by the two systems. 3. Linux has two categories of I/O schedulers: single cue and multi-q. 4. The choice of I/O scheduler depends on the device being used and the type of application. 5. High-performance SSDs are recommended to use "none" or "kyber", while desktop/workstation applications are recommended to use "bfq".
This explains why Ubuntu on a SATA SSD booting and loading programs on my 2012 PC with an AMD APU is *faster* than Fedora on the same brand/model of SSD on a PC with a Ryzen 7 5800X! I just thought I had screwed something up on my newer PC 😂
BTRFS with read/write compression and copy-on-write, vs the unsafe ext4, is not a fair comparison. The benefits of BTRFS are in reliability, snapshots, compression, etc. We know that it is slower than ext4. What would you recommend that Fedora changes, if anything?
@@CyberGizmo Yeah according to ext3's and ext4's creator, Theodore T'so, at the Linux Collaboration Summit 2009: "Despite the fact that Ext4 adds a number of compelling features to the filesystem, T'so doesn't see it as a major step forward. He dismisses it as a rehash of outdated "1970s technology" and describes it as a conservative short-term solution. He believes that the way forward is the Btrfs filesystem, which is designed to deliver significant improvements in scalability, reliability, and ease of management." I think that he refers to the way ext4 is a very basic filesystem without file checksumming, and not even metadata checksumming at all by default, and which is hindered by the journal where all data is written before it's then written to its final place on disk, which leads to write amplification on SSDs and performance degradation on HDDs. Either way, it seems weird to compare vastly different filesystems, one of which(Btrfs on Fedora) is configured to checksum all file blocks on both read and write, do copy-on-write, compress and decompress all file blocks on the fly, support snapshots, etc. Those reliability improvements have a performance impact for sure. Theodore actually had something to say about Btrfs performance and filesystems benchmarks in 2012 or so, I don't remember the exact quote but it was along the lines of saying that benchmarks don't emulate real-world workflows or what happens when a FS is dirty or nearly full and that synthetic benchmarks are misleading, and even if they do get close to it sometimes, you are likely to prefer the extra features of BTRFS, even though they come with some performance impact Oh yeah, did you hear that BTRFS is working on a new metadata format which has better concurrency, thus better performance? I am not sure where that is at now but it's coming.
@@MyAmazingUsername Oh that Internet thread, you don't need to quote it I read it when it was posted 14 years ago now. so just so you know that thread are comments regarding ext4 which had been newly released a year before in 2008. That is not relevant today, second most of the comments were from people who were clueless about filesystem design. Thomas did not write the that article he was quoted in it, and it was referring to a bug in ext4. Today ext4 is widely deployed throughout the world in environments which require 5 9's in uptime or better, it is stable, reliable and safe. I am not going to debate that article all of those issues were fixed long ago. As for choice of filesystem, it has been my experience in HPC, Cloud, large server farms, with higher performance high transaction counts, there is no single filesystem which offers the perfect solution to every environment. I want a filesystem to manage my files, I don't need to want it to do anything else for me on the operating system level. I content BTRFS is not a good choice for a Linux Desktop or Workstation, that is not an opinion, I can back it up with tests. For a more recent assessment of BTRFS vs EXT4 this article from 2022 is a pretty interesting read and a fair one I might add: linuxopsys.com/topics/btrfs-vs-ext4
Hi ! Is BTRFS generally slower than ext4 or *just* Fedora's implementation of BTRFS ? I'm on Fedora 38 right now, with ext4, but I'm wondering if I would change for openSuse Tumbleweed with BTRFS.
by design yes. somewhere in the range of 2-5 times less iops (depending on the used feature set - see mount options). you get the features instead. you can not have both
Hey I really like this type of info. I've always been curious. Maybe you can also talk about the differences between network queues like pfifo_fast vs fq_codel vs fq (and are there others?), and tcp congestion control protocols like illinois, cubic, yeah (my favorite), bbr, westwood, htcp and so on.
My Root Fedora 38 BTRFS is also compressed. That would make more sense for the 3-to-1 ratio in the reads. I don't know if it would do the same if I didn't have separate /home and /var, but if it's enabled everywhere it would explain part of the difference
Very interesting video. I would love to see a comparison between Fedora + Ext4 and Ubuntu + Ext4, just to make this a more normalized comparison since Ext4 was the default from Fedora 11 to Fedora 32 on workstation. Fedora 22 started shipping XFS for server edition, but I would wager there are still lots of Fedora workstation installations out there that are running Ext4. Another one to throw in the mix would be OpenSUSE, both Tumbleweed and Leap 15.4 flavors. The reason I suggest that is because our green lizard friends were shipping Btrfs way before Fedora started doing it. Anyway, back to work, thanks for a nice video 😸
The comparison with EXT4 on Fedora is up now, dunno if I will get to OpenSUSE or not, have a different set of videos I want to do this week but will add it to the list
If there are old Fedora systems (older than one year, at this time that's 36 or older), there is no LTS version of Fedora. They're vulnerable, and should be upgraded, or if they don't want frequent annual or biannual upgrades, they should switch to Red Hat Enterprise Linux or a RHEL clone like Alma or Rocky.
@@MobyTurbo If someone has upgraded continually from earlier Fedora versions without doing a reinstall (this is common), then they would still be using Ext4.
Heh, very interesting... I've been running Linux since the early 90s and I prefer Redhat's package management utilities (no special technical reason, other than there's less of them and I can remember a lot of the commands better, especially querying.). But I run Ubuntu because... well, this is a desktop and I run goofy games and stuff. More eyes on Ubuntu means more goofy desktop stuff targets it. But I guess if that's what I was worried about, I'd run Arch (just can't force myself to do it.) Should be fairly easy to switch between these schedulers at will for end-users, no? (Unless some distro is including I/O sched patches another isn't.) Thanks DJ.
Welcome glad you enjoyed it. Yeah its easy to switch between them I should do a video on how to make it permanent there is a writeup in the red hat guide about using udev configurations to do it. Otherwise you can just echo which scheduler you want into /sys/blocks//queue/scheduler
the thing wirh BTRFS is that you do not need all features but more shockingly on a desktop/laptop I do not need or miss the iops also. compiles and all my workloads are no longer disk bottlenecked anyway. and on a server in a lot of cases snapshots are also great to have.
Always good to know sometimes defaults are not always the best (meaning Fedora's choice of btrfs). Xfs + Kyber looked really good on some of your follow-on re-benchmarks. And that's impressive, but as you point out it's a kind of "niche" combo, emphaszing reads, etc.
I installed just the root filesystem (40GB) in btrfs for the snapshotting and then a /var of 130 GB with XFS and a home with XFS just because no matter what I do I don't want to run into issues with fixed inodes with EXT4
How were these benchmarks produced? What are the chances each successive test produced higher results because the files were already in the Disk Cache? Separate installations? Recreated partitions?
I wonder if changes to btrfs is why rpm-ostree has been chugging so hard on my PC? Fedora Silverblue 38. In fact, I don't trust the state of rpm-ostree in the current release as far as my machine goes, a generic mini PC from Beelink. I'm switching to MX Linux until this is sorted out. I've considered an LTS kernel again since I'm not wanting to cross my fingers anyways.
Are you using it in virtual machine? and what file system do you use for test, all this will impact in the result. Here the fedora using brfs is more fast than ubuntu using ext4. I am using nvme for the test.
That title brings up a joke in my head - the dissonance created by the extra "is" is perfectly emblematic of the Fedora way. I don't get the Fedora love affair in vogue at this time. It's a wobbly wheel if you're looking for a stable ship. You get what you get - and down the road you'll get a lot of good stuff.... but not now.
"Fedora installations now enable transparent compression on the root Btrfs file system, including all its subvolumes." Is that compression the bottleneck?
Totally interesting video, DJ. Thank you. I keep hearing what a dog Btrfs is in terms of R/W performance compared to just about any other modern fs, and am flabbergasted why distros such as Fedora and manufacturers such as Synology insist on making it the default fs. Yeah, I get there are some great advantages such as snapshoting and file robustness, but hell the performance issues are just a non starter for me. I've been watching a lot of Synology videos lately as I've been mulling over installing a NAS cloud server here in my home, but man oh man they really insist and push Btrfs over Ext4 even though the latter is so much faster.
Welcome Appaling glad you enjoyed it. Interesting point on Synology I forgot about them, does the fact they run btrfs on top of LVM make any difference in performance? I am a do it my selfer so yeah I build my own NAS
I've been using btrfs for a couple of years. Never noticed a slowed down. There is probably situations where every second counts but on an average day or daily use for me, it's never been a problem. On the other hand, trying to use timeshift with ext4 is a slowwwwwwwwwwww nightmare. Making snapshots is slow. Reverting changes? Takes an eternity. Just saying, i think the benefits of btrfs outweigh the drawbacks for average users. Having said, i thought I'd heard that btrfs is less reliable? I don't know much about file systems so all I know is what I've encountered while daily driving a Linux system. 🧂👌
There was a problem with Btrfs slowing down after a few days to a week of usage for me, it happened whenever I tried Fedora or anything using Btrfs for a long period of time, that issue is gone ever since kernel 6.0/6.1 though. So now I just have a decent experience with it, especially on space savings, for example my /nix since I'm on NixOS, is compressed to 5.3GB from 13GB with force-zstd:1.
btrfs isn't about performance as it is recovery with snapshots and also having subvolumes you can partition out your /, /home/ etc while still having them within the same partition thus letting them dynamically resize as needed. If you're using btrfs but aren't using snapshots or subvolumes, you're just wasting performance yes, but like I said, the point of btrfs isn't max speed, it's additional baked-in features that make ordinary use easier and more reliable. If you want to manage snapshots or volume sizes on your own, then by all means do that, but I think it's worth the ding to performance to not have to worry about that stuff.
I'm trying to figure out how to make the change to None permanent in Fedora 38 by adding it to the GRUB config file but when I go to update the GRUB it seems as though I haven't done the modification to the right one cause it doesn't do the initramfs. All it says is it added an entry to the GRUB menu and that's it. I don't see the changes being reflected on all the kernels. Weird.
I am wondering why you chose to compare the different file systems like that, but make it sound like a distro issue... You benchmarked Ubuntu in ext4, but didn't try Fedora the same way. I run Fedora with ext4 as its always seemed to run the quickest.
I did test it, but didnt publish the results because Fedora's default previous to BTRFS was XFS not Ext4, hope that helps, and nothing wrong with ext4 its a good filesystem as you say.
Great content. The HAL9000 on your wall is reason enough to subscribe though 😂 Btw, have you ever asked chat gpt to open the pod bay doors? You should try.
4:20 we reached a point in computer history where our "sacred" Caches are not useful anymore... in fact, the Caches are giving us more problems than solutions... basically, in all the 16GB 32GB RAM we use these days, Caches are not necessary, and all that happens in the end is that we are doing the same procedure TWICE (check cache, proceed... check cache, proceed...) Significant I/O latency on that small verification
Hi DJ Ware, found your video when searching for reported problems when updating from fedora 37 to 38, just noticed you have a speededitor on your desk, does resolve work on fedora 38? People are complaining about it not working on 38, it took me an afternoon to get it working on 37 with dependecies issues etc etc, what has been your experience so far?
I used to use Fedora for Davinci Resolve, but its a pain to keep Fedora kernel changes in sync with BMD's kernel drivers, so I finally gave up on it, got tired of it breaking every kernel upgrade. I still use Resolve for editing but moved it to my audio machine which is a Mac Mini
Things get even more bonkers on VMs, because the hypervisor and often the enterprise storage system have their own I/O schedulers, and different VMs might be trying to do I/O with different philosophies. I've found that it's often best to have VMs use the 'none' scheduler, which will take advantage of multiple cores a VM might get in a hypervisor timeslice, but let the hypervisor or storage system handle the actual I/O scheduling. Definitely use the multipath scheduler options in VMs, you don't want eight physical CPUs to be consumed by a VM that's just using a portion of one to do I/O.
Also... if you're managing VMs on systems with battery backup... you CAN consider turning write barriers off and boosting the commit time. This will dramatically improve I/O. Also, give your databases a maximum memory limit that leaves enough room in physical or virtual memory for the kernel's write buffers to get big. That should give the I/O queues room to do their thing properly.
Three comments 1) I use Proxmox and it uses the "none" I/O scheduler, 2) I run all my tests on the same platform and 3) I recheck on hardware to make sure I am seeing similar results, I realize they won't be the same unless the hardware, software, configuration is exactly the same. Also I have heard the phase of the moon comes into play occasionally :)
@@CyberGizmo Oh yeah, this isn't a criticism of your testing setup, but a note to folks running lots of VMs on big metal in the enterprise or cloud. Tuning a busy VM environment properly can have huge consequences that are hard to measure; you can have a bunch of VMs bogging each other down while the CPU usage looks 'low' if you are using the 'noop' or other single-path schedulers instead of 'none' or the multipath ones.
i didn't notice that, i am using fedora 38, so i can actually address this i/o issue right? though i don't know my way well around linux just yet, do you have any resources that can help me learn how to do this myself
I would recommend doing this in a virtual machine first, Keep in mind if you do this directly on your hardware, you will lose what you have on your drive, so back it up first, this is part of the fedora install process: You can read about how to partitioning your storage here: jfearn.fedorapeople.org/fdocs/en-US/Fedora/20/html/Installation_Guide/s1-diskpartitioning-x86.html
Fedora is the slowest base install across the board. You want none or mq-deadline for platters/NVME platters. Ubuntu and Manjaro will default mq-deadline if you have platters attached. Manjaro adds Kyber and a few others. I thought it was just from running services or Gnome Cubes or something. Single queue makes sense for the slow-down. Also, if you have a modern cpu, like a Ryzen, check the kernel for x86_64v4 support. There are people still running v2 or v3, not even utilizing the entire cpu feature set. I'm using btrfs. It made me turn away, but I totally forgot about the scheduler, and none on platters is bad. I want to try out Alma.
My system runs much faster in opening and closing apps... even when copying files from internal to my usb drives on Fedora ... ubuntu feels laggy on my systems so will keep on using fedora thx
I'm still frustrated by the fact that people moved away from deadline to noop per latency reasons, when, in reality, they just ran into the difference that noop doesn't io merges and deadline per default does. deadline + no merges worked as fast just preserving a fair scheduler. and this misunderstanding was pushed upwards until most vm disk schedulers used noop. so now we're back to issues like swap stalls just because they hack around instead of writing/reading specs for their own stuff. "sorting" and "merges" are NOT NOT NOT NOT a mandatory component of deadline. the effects are disgusting on large applications but i suppose people deserve that.
@@CyberGizmo If you mean making a list of hardware to build a PC, ChatGPT already can do it. But it makes mistakes and the builds usually are not great
You see, I don't really care all that much about I/O, as today's NVMe disks are fast enough for any speed difference to be really noticeable. What I care for is the ability to make snapshots with BTRFS and revert to them when I need. Can Ubuntu do that with its outdated ext4 FS? no.
It's a totally different subject. And dj ware has talked about this subject. You can make an argument about "outdated ext4" but it's a total nonsense argument.
@@benign4823 it's actually very logical. They are saying even if "slow" the features btrfs has make it more valuable to them. Snapshots are highly important to me as well so I fully understand.
@@SinisterSpatula It is, in fact, very illogical. They're comparing Ubuntu + ext4 with Fedora + Btrfs and asking why Ubuntu doesn't have Btrfs features like it's Ubuntu's fault that you picked ext4. You can literally use Btrfs on anything, including Ubuntu, and their point of "well I can have Btrfs snapshots on Fedora so Fedora's better" is even more senseless considering that Fedora doesn't even have snapshots setup by default, and so you'd have to set it up, same way you'd set it up on any other distro minus OpenSUSE that has it by default, some immutable distros that use it and NixOS that has it's own rollback system that's not dependent on a fs.
after I came back to Windows, I notice it also has problems with severe (unnecessary) Disk Caching for process and unusually high Pagefile (Swap) usage on Windows too. But here comes the deal: on Windows the Disk Paging doesn't bring my entire system to a halt like Linux did... something is deeply wrong in the core foundation of Linux kernel, and we've been using pretty much the same foundation from 2 GB spinning disk days (obsolete)
also I noticed this after going back to Windows: it doesn't do that thing anymore that Linux did, where things stayed in Swap even after I closed and reopened programs... Windows is more smart about flushing the Swap... Linux does not Flush swap ever and it becomes full (system enters kernal panic mode)
and it's not only dangeous when it comes to performance alone... we're talking Linux performing too much Disk Caching to a point where it can severely reduce SSD lifespans... we would have to do some kind of research Poll in the linux community in order to know how much in average people's SSD drives last before dying, SSD's became prevalent 2012 onwards so we don't have that data yet
let's give a real life usage example here: let's say you are trying to Livestream a game using OBS broadcasting software, running a moderately heavy game on the side and having a browser open (because you need to check your stream page, your live chat, etc etc etc) What you notice is that after 1 hour or so your memory in Linux maxes out and intense Disk paging occurs, bringing the whole thing to a chaos and even dropping frames on the broadcast (terrible for someone who needs a working system in order to present video to the public)
uhhh, I think you missed the boat here entirely - how does using a different scheduler have anything to do with security? You sound very confused (or very ill-informed)
This is a lot of noise about nothing. Applying the correct scheduler to the current hardware and situation is and was always the responsibility of the admin. Simple as that. If the defaults of the distribution are not optimal out of the box for a certain situation it does not matter very much for the non-professional user and a professional user will take care of it anyways. So it does not really matter in practice. So I think this is just a storm in a teacup and klick-bait - or just a waste of time at best. Nothing against a video about scheduler variants and learning about it - but this has nothing to do with any specific distribution. This is simply not an intelligent angle of approach to the topic.
Not entirely true. Sure, lots of noobs use it but even professionals at Google, Meta use it extensively. Even for ML it is widely used, NVIDIA too! It's great so is gentoo, opensuse, Fedora etc
Personally, I think Fedora is a CROCK anyway--- every time I've tried it-- and I try it with every new version--- it's been SLOW_ clunky- and caused issues with my harddrive.. EVERY TIME.. it's a CROCK.. looked "cartoony" looking last issue too- and I hated that-- all the pastel colors and cartoon looking art.. like something a 3 year old made. I have Sparky linux- based on Debian- MUCH faster-- beautiful and kde highly customizable.. so why would I mess with slow, clunky, ugly- waste of space like Fedora???
On modern solid state storage all the olde go faster tweaks are counterproductive. I can entirely understand Canonical’s decision to switch to NONE scheduler. Thanks for showing us some examples in support.
Welcome, Chris
Not ALL. Write buffering is still a good idea, for example.
@@VioletGiraffe Yes, especially ensuring actual writes to storage match the block size of flash media to reduce the number of times a block has to be rewritten (this speeds things up significantly and also reduces wear on flash storage).
A good SSD controller basically replaces the scheduler. Doing all the reordering, batching, etc.
@@ChristianBrugger Fair point, I should hope it does.
Found this your channel and this video on a search for why my fedora kept hanging.. good info, definitely subbed. Love your style man!!
zfs has its own internal scheduler. So, it is best to set the schedulers in /sys/block/sd?/queue/scheduler to none.
Modern ssd have controllers that operate on information not available to the o/s. I/O schedulers are a problem programmers love to solve, but sadly, it's mostly obsolete today.
Always better to let the hardware do it...but my experience with early SSD's even hardware controllers can have bugs in one case so bad the SSD self destructed just one day after the warranty expired.
Thank you for sharing your knowledge!
Deep content like this is some of my favorite YT content. This is the stuff you’d only be able to dog out of lwn forums back in the day, now I can listen while driving. Thanks DJ!
Hey DJ can you do more deep dives on the internals of Linux/Unix? These have been my favourite videos of yours.
Just watching your videos i think you have enough knowledge to build And break system.. you have very in-depth knowledge in what you're talking about.. salute you DJ
Thanks, Panther
@@CyberGizmo you're welcome Sir
Great content thanks for sharing.
Thank you for your service.
Fedora also compresses btrfs by default in level zstd:1 - so that could also make a difference - especially on low performance hardware - however, actual throughput shouldn't be affected.
I thought that might be the case also, however I found the problem with btrfs...its overwhelming the kernel with Memory Copies
@@CyberGizmo 🤔 hm. this is interesting. I've heard of memory/disk lag when hammering btrfs, and the same thing not being a case when using xfs or ext4. could this be the case? does this mean more memory usage as well?
i love the btrfs compression (because i use nixos) but perhaps xfs would be a better choice.
I noticed this the other day when I updated my laptop.
I backed up 35GB from Ubuntu 22.10 onto a USB HDD and it took 20~25 minutes and when I copied it back to 23:04 it took 10~15 minutes
Doesn't that also depend on the read write speeds of your USB HDD vs internal SSD?
Also how & what (what parts of system, apps, user data, etc) did you back up?
@@therealb888 Just documents.
I have done this transfer at every Ubuntu upgrade for 2~3 years , same external HDD and same internal SSD . Usually the time each way is not noticeably different.
With 23.04 there was a definite change.
@@PaulG.x Thanks, I guess it's worth the upgrade. In my case I want to do a full system backup, all repos, apps, libs, configs & app data like browser history, bookmarks etc without using cloud sync or use an e2ee cloud sync.
On a side note do ensure you're checking for drive health & try to have a third backup so you don't suffer from sudden drive failure.
@@therealb888 Over years of Linux use , my preference on my main desktop has evolved to be a fresh install of the OS (no separate /home ) and manually install the apps I use - that only takes an hour now.
I back up the browser and email client profiles and restore them to the freshly installed instances.
All my data is on dedicated storage devices , not on the system drive.
I have aarch64 build of fedora 38 running in the UTM virtual environment. And it has "none" io scheduler on block devices by default. So... there should be some glitch in kernel device type detection code or in dev rules. Maybe.
I think they patched it shortly after making this video
I have got my dbus rules set up so that:
* If it is a spinny platter hard drive, it will assign it the BFQ io scheduler
* Else if it is using NVMe, then assigns it the "none" scheduler
* Otherwise, it uses mq-deadline
I feel that in the vast majority of cases, this will give me a pretty good choice for IO scheduler on that device.
nice
Oh wait, I mean udev, not dbus. Oops
Interesting, thank you.
My openSUSE Tumbleweed desktop defaulted to none with Btrfs.
Tumbleweed defaults to Btrfs because it also comes packaged with snapper installed so you can roll back your system after a bad update
@@tumescent understood -- its use of [none] was the new part for me.
This was very informative video, thank you.
Welcome NeoBTK
I noticed Fedora would almost hang sometimes with large file copies so maybe they are trying to fix it... I am afraid to switch from 36, it is at least mostly working properly.
I am on 37, I feel that. My device is also encrypted so if I have to backup to my external for this computer on its last breath, it will probably be turned into a server when I update. And yes, as a current workstation, this all works.
I think I will wait for 39 or even 40.
I am on 38 feels good no problem faced in my usage..
stick with 36 or 37 and wait some..
@@DawidKellerman already upgraded bro...
Sooo, I'm a bit confused. In the end it didn't really mattered the scheduler used, but the file system. Why then the discussion about the schedulers ? Is it because that's what you initially thought was the culprit ? Or you simply wanted to talk about them ? Or something else ?
I recommend using "none" unless the file system is a spinning rust, in that case go with what redhat recommends mq-deadline or bfq
@@CyberGizmodid you answer the wrong comment?
TL:DR Below
1. Ubuntu 23.04 has faster file I/O than Fedora 38 in benchmarks.
2. The difference in performance is due to different I/O schedulers used by the two systems.
3. Linux has two categories of I/O schedulers: single cue and multi-q.
4. The choice of I/O scheduler depends on the device being used and the type of application.
5. High-performance SSDs are recommended to use "none" or "kyber", while desktop/workstation applications are recommended to use "bfq".
Except that in the end it was shown that it was not the schedulers but the file system used.
This explains why Ubuntu on a SATA SSD booting and loading programs on my 2012 PC with an AMD APU is *faster* than Fedora on the same brand/model of SSD on a PC with a Ryzen 7 5800X! I just thought I had screwed something up on my newer PC 😂
BTRFS with read/write compression and copy-on-write, vs the unsafe ext4, is not a fair comparison. The benefits of BTRFS are in reliability, snapshots, compression, etc. We know that it is slower than ext4. What would you recommend that Fedora changes, if anything?
ext4 is unsafe?
@@CyberGizmo Yeah according to ext3's and ext4's creator, Theodore T'so, at the Linux Collaboration Summit 2009:
"Despite the fact that Ext4 adds a number of compelling features to the filesystem, T'so doesn't see it as a major step forward. He dismisses it as a rehash of outdated "1970s technology" and describes it as a conservative short-term solution. He believes that the way forward is the Btrfs filesystem, which is designed to deliver significant improvements in scalability, reliability, and ease of management."
I think that he refers to the way ext4 is a very basic filesystem without file checksumming, and not even metadata checksumming at all by default, and which is hindered by the journal where all data is written before it's then written to its final place on disk, which leads to write amplification on SSDs and performance degradation on HDDs.
Either way, it seems weird to compare vastly different filesystems, one of which(Btrfs on Fedora) is configured to checksum all file blocks on both read and write, do copy-on-write, compress and decompress all file blocks on the fly, support snapshots, etc. Those reliability improvements have a performance impact for sure.
Theodore actually had something to say about Btrfs performance and filesystems benchmarks in 2012 or so, I don't remember the exact quote but it was along the lines of saying that benchmarks don't emulate real-world workflows or what happens when a FS is dirty or nearly full and that synthetic benchmarks are misleading, and even if they do get close to it sometimes, you are likely to prefer the extra features of BTRFS, even though they come with some performance impact
Oh yeah, did you hear that BTRFS is working on a new metadata format which has better concurrency, thus better performance? I am not sure where that is at now but it's coming.
@@MyAmazingUsername Oh that Internet thread, you don't need to quote it I read it when it was posted 14 years ago now.
so just so you know that thread are comments regarding ext4 which had been newly released a year before in 2008. That is not relevant today, second most of the comments were from people who were clueless about filesystem design.
Thomas did not write the that article he was quoted in it, and it was referring to a bug in ext4. Today ext4 is widely deployed throughout the world in environments which require 5 9's in uptime or better, it is stable, reliable and safe.
I am not going to debate that article all of those issues were fixed long ago. As for choice of filesystem, it has been my experience in HPC, Cloud, large server farms, with higher performance high transaction counts, there is no single filesystem which offers the perfect solution to every environment.
I want a filesystem to manage my files, I don't need to want it to do anything else for me on the operating system level.
I content BTRFS is not a good choice for a Linux Desktop or Workstation, that is not an opinion, I can back it up with tests.
For a more recent assessment of BTRFS vs EXT4 this article from 2022 is a pretty interesting read and a fair one I might add: linuxopsys.com/topics/btrfs-vs-ext4
Hi ! Is BTRFS generally slower than ext4 or *just* Fedora's implementation of BTRFS ?
I'm on Fedora 38 right now, with ext4, but I'm wondering if I would change for openSuse Tumbleweed with BTRFS.
Good question, I do not know, I have OpenSUSE on my list to review
@@CyberGizmo thanks for the reply !
(Well, I actually just switched/distro-hopp'd to Elementary OS 7 :) )
by design yes. somewhere in the range of 2-5 times less iops (depending on the used feature set - see mount options). you get the features instead. you can not have both
Cpu schedulers is one thing, your talking about like IO scheduling here..Must tell you I had no idea, about them. Interesting video!
Glad you enjoyed it
Is this super recent? I switched from Ubuntu to Red Hat on the NAS and have noticed slower transfer speeds. Could be the reason?
I don't know if its recent, I can say I recently noticed it heh
Thank you this helped me a lot, very good video just like all the others.
Thanks again. This is valuable info.
Hey I really like this type of info. I've always been curious. Maybe you can also talk about the differences between network queues like pfifo_fast vs fq_codel vs fq (and are there others?), and tcp congestion control protocols like illinois, cubic, yeah (my favorite), bbr, westwood, htcp and so on.
Very good idea!
My Root Fedora 38 BTRFS is also compressed. That would make more sense for the 3-to-1 ratio in the reads. I don't know if it would do the same if I didn't have separate /home and /var, but if it's enabled everywhere it would explain part of the difference
Very interesting video. I would love to see a comparison between Fedora + Ext4 and Ubuntu + Ext4, just to make this a more normalized comparison since Ext4 was the default from Fedora 11 to Fedora 32 on workstation. Fedora 22 started shipping XFS for server edition, but I would wager there are still lots of Fedora workstation installations out there that are running Ext4.
Another one to throw in the mix would be OpenSUSE, both Tumbleweed and Leap 15.4 flavors. The reason I suggest that is because our green lizard friends were shipping Btrfs way before Fedora started doing it. Anyway, back to work, thanks for a nice video 😸
The comparison with EXT4 on Fedora is up now, dunno if I will get to OpenSUSE or not, have a different set of videos I want to do this week but will add it to the list
If there are old Fedora systems (older than one year, at this time that's 36 or older), there is no LTS version of Fedora. They're vulnerable, and should be upgraded, or if they don't want frequent annual or biannual upgrades, they should switch to Red Hat Enterprise Linux or a RHEL clone like Alma or Rocky.
@@MobyTurbo If someone has upgraded continually from earlier Fedora versions without doing a reinstall (this is common), then they would still be using Ext4.
Heh, very interesting... I've been running Linux since the early 90s and I prefer Redhat's package management utilities (no special technical reason, other than there's less of them and I can remember a lot of the commands better, especially querying.). But I run Ubuntu because... well, this is a desktop and I run goofy games and stuff. More eyes on Ubuntu means more goofy desktop stuff targets it. But I guess if that's what I was worried about, I'd run Arch (just can't force myself to do it.)
Should be fairly easy to switch between these schedulers at will for end-users, no? (Unless some distro is including I/O sched patches another isn't.) Thanks DJ.
Very good point about not getting in the weeds about benchmarks... it's about the workload and the resources at hand.
Welcome glad you enjoyed it. Yeah its easy to switch between them I should do a video on how to make it permanent there is a writeup in the red hat guide about using udev configurations to do it. Otherwise you can just echo which scheduler you want into /sys/blocks//queue/scheduler
@@CyberGizmo Nod, yup. That would be a good video, seems a lot of people distro-hop for unnecessary reasons sometimes. Cheers.
the thing wirh BTRFS is that you do not need all features but more shockingly on a desktop/laptop I do not need or miss the iops also. compiles and all my workloads are no longer disk bottlenecked anyway. and on a server in a lot of cases snapshots are also great to have.
Is your music anywhere? or what kind of DJing do you do?
Always good to know sometimes defaults are not always the best (meaning Fedora's choice of btrfs). Xfs + Kyber looked really good on some of your follow-on re-benchmarks. And that's impressive, but as you point out it's a kind of "niche" combo, emphaszing reads, etc.
I installed just the root filesystem (40GB) in btrfs for the snapshotting and then a /var of 130 GB with XFS and a home with XFS just because no matter what I do I don't want to run into issues with fixed inodes with EXT4
2:46 --- Never place your beverage close to a keyboard (said someone with a $150 -- but not waterproof -- keyboard)!
I keep in a cup holder, and yeah good advice, I too have wiped out keyboards with coffee
How were these benchmarks produced? What are the chances each successive test produced higher results because the files were already in the Disk Cache? Separate installations? Recreated partitions?
I have a full break down of the methodology I use on my benchmarks elsewhere in the video stack
This is strange. My Fedora 37 system already uses none for the NVME drive. I didn't have to change it.
I wonder if changes to btrfs is why rpm-ostree has been chugging so hard on my PC? Fedora Silverblue 38. In fact, I don't trust the state of rpm-ostree in the current release as far as my machine goes, a generic mini PC from Beelink. I'm switching to MX Linux until this is sorted out. I've considered an LTS kernel again since I'm not wanting to cross my fingers anyways.
Are you using it in virtual machine? and what file system do you use for test, all this will impact in the result. Here the fedora using brfs is more fast than ubuntu using ext4. I am using nvme for the test.
I’m curious which is the best scheduler if you are running Optane drives.
I'd try the "none" scheduler
I like the AI art for your thumbnail. The monitor was throwing me off for a bit. And then I realized why.
its amazing the things AI gets right and as that photo showed the things it gets wrong :)
That title brings up a joke in my head - the dissonance created by the extra "is" is perfectly emblematic of the Fedora way. I don't get the Fedora love affair in vogue at this time. It's a wobbly wheel if you're looking for a stable ship. You get what you get - and down the road you'll get a lot of good stuff.... but not now.
As long as we can change it during install, I'm not complaining
"Fedora installations now enable transparent compression on the root Btrfs file system, including all its subvolumes." Is that compression the bottleneck?
Totally interesting video, DJ. Thank you. I keep hearing what a dog Btrfs is in terms of R/W performance compared to just about any other modern fs, and am flabbergasted why distros such as Fedora and manufacturers such as Synology insist on making it the default fs. Yeah, I get there are some great advantages such as snapshoting and file robustness, but hell the performance issues are just a non starter for me. I've been watching a lot of Synology videos lately as I've been mulling over installing a NAS cloud server here in my home, but man oh man they really insist and push Btrfs over Ext4 even though the latter is so much faster.
Welcome Appaling glad you enjoyed it. Interesting point on Synology I forgot about them, does the fact they run btrfs on top of LVM make any difference in performance? I am a do it my selfer so yeah I build my own NAS
Pretty sure ext4 doesn't support dynamically allocated inodes 😦🌝
I've been using btrfs for a couple of years. Never noticed a slowed down. There is probably situations where every second counts but on an average day or daily use for me, it's never been a problem.
On the other hand, trying to use timeshift with ext4 is a slowwwwwwwwwwww nightmare. Making snapshots is slow. Reverting changes? Takes an eternity.
Just saying, i think the benefits of btrfs outweigh the drawbacks for average users. Having said, i thought I'd heard that btrfs is less reliable? I don't know much about file systems so all I know is what I've encountered while daily driving a Linux system. 🧂👌
There was a problem with Btrfs slowing down after a few days to a week of usage for me, it happened whenever I tried Fedora or anything using Btrfs for a long period of time, that issue is gone ever since kernel 6.0/6.1 though.
So now I just have a decent experience with it, especially on space savings, for example my /nix since I'm on NixOS, is compressed to 5.3GB from 13GB with force-zstd:1.
btrfs isn't about performance as it is recovery with snapshots and also having subvolumes you can partition out your /, /home/ etc while still having them within the same partition thus letting them dynamically resize as needed. If you're using btrfs but aren't using snapshots or subvolumes, you're just wasting performance yes, but like I said, the point of btrfs isn't max speed, it's additional baked-in features that make ordinary use easier and more reliable. If you want to manage snapshots or volume sizes on your own, then by all means do that, but I think it's worth the ding to performance to not have to worry about that stuff.
I'm trying to figure out how to make the change to None permanent in Fedora 38 by adding it to the GRUB config file but when I go to update the GRUB it seems as though I haven't done the modification to the right one cause it doesn't do the initramfs. All it says is it added an entry to the GRUB menu and that's it. I don't see the changes being reflected on all the kernels. Weird.
mine is using "none" also under Fedora (Silverblue) using an nvme device
On Fedora 38 with btrfs I notice that the rate of page scanning is high. I wonder if having more RAM helps and/or turning for memory requirements?
I am wondering why you chose to compare the different file systems like that, but make it sound like a distro issue... You benchmarked Ubuntu in ext4, but didn't try Fedora the same way. I run Fedora with ext4 as its always seemed to run the quickest.
he is comparing distrios, btrfs is standard on fedora, if ram disk or whatever is the quickest, it doesn't mean we should use it on all distros
I did test it, but didnt publish the results because Fedora's default previous to BTRFS was XFS not Ext4, hope that helps, and nothing wrong with ext4 its a good filesystem as you say.
@@CyberGizmo wasnt xfs default on fedora server only not desktop/workstation
@@CyberGizmo That makes sense... thanks. I guess I never paid attention to defaults since I just set it the way I want.
@@alexstone691 Both as I recall
Great content. The HAL9000 on your wall is reason enough to subscribe though 😂
Btw, have you ever asked chat gpt to open the pod bay doors? You should try.
4:20 we reached a point in computer history where our "sacred" Caches are not useful anymore... in fact, the Caches are giving us more problems than solutions... basically, in all the 16GB 32GB RAM we use these days, Caches are not necessary, and all that happens in the end is that we are doing the same procedure TWICE (check cache, proceed... check cache, proceed...) Significant I/O latency on that small verification
9:02 Don't you mean "4 ms and even 3"?
lower is always better, but if it gets above 7ms, the delay in my experience will mess up your playing
Hi DJ Ware, found your video when searching for reported problems when updating from fedora 37 to 38, just noticed you have a speededitor on your desk, does resolve work on fedora 38? People are complaining about it not working on 38, it took me an afternoon to get it working on 37 with dependecies issues etc etc, what has been your experience so far?
I used to use Fedora for Davinci Resolve, but its a pain to keep Fedora kernel changes in sync with BMD's kernel drivers, so I finally gave up on it, got tired of it breaking every kernel upgrade. I still use Resolve for editing but moved it to my audio machine which is a Mac Mini
I just 'nerded out' as my wife says 😆 Thanks so much for this!!
Welcome
Things get even more bonkers on VMs, because the hypervisor and often the enterprise storage system have their own I/O schedulers, and different VMs might be trying to do I/O with different philosophies.
I've found that it's often best to have VMs use the 'none' scheduler, which will take advantage of multiple cores a VM might get in a hypervisor timeslice, but let the hypervisor or storage system handle the actual I/O scheduling.
Definitely use the multipath scheduler options in VMs, you don't want eight physical CPUs to be consumed by a VM that's just using a portion of one to do I/O.
Also... if you're managing VMs on systems with battery backup... you CAN consider turning write barriers off and boosting the commit time. This will dramatically improve I/O.
Also, give your databases a maximum memory limit that leaves enough room in physical or virtual memory for the kernel's write buffers to get big. That should give the I/O queues room to do their thing properly.
Three comments 1) I use Proxmox and it uses the "none" I/O scheduler, 2) I run all my tests on the same platform and 3) I recheck on hardware to make sure I am seeing similar results, I realize they won't be the same unless the hardware, software, configuration is exactly the same. Also I have heard the phase of the moon comes into play occasionally :)
@@CyberGizmo Oh yeah, this isn't a criticism of your testing setup, but a note to folks running lots of VMs on big metal in the enterprise or cloud. Tuning a busy VM environment properly can have huge consequences that are hard to measure; you can have a bunch of VMs bogging each other down while the CPU usage looks 'low' if you are using the 'noop' or other single-path schedulers instead of 'none' or the multipath ones.
How sbt xfs?
i didn't notice that, i am using fedora 38, so i can actually address this i/o issue right? though i don't know my way well around linux just yet, do you have any resources that can help me learn how to do this myself
I would recommend doing this in a virtual machine first, Keep in mind if you do this directly on your hardware, you will lose what you have on your drive, so back it up first, this is part of the fedora install process: You can read about how to partitioning your storage here: jfearn.fedorapeople.org/fdocs/en-US/Fedora/20/html/Installation_Guide/s1-diskpartitioning-x86.html
I liked the video. How about going over the commands to change the scheduler. That would be really helpful.
Will add it to the list
Fedora is the slowest base install across the board. You want none or mq-deadline for platters/NVME platters. Ubuntu and Manjaro will default mq-deadline if you have platters attached. Manjaro adds Kyber and a few others. I thought it was just from running services or Gnome Cubes or something. Single queue makes sense for the slow-down. Also, if you have a modern cpu, like a Ryzen, check the kernel for x86_64v4 support. There are people still running v2 or v3, not even utilizing the entire cpu feature set. I'm using btrfs. It made me turn away, but I totally forgot about the scheduler, and none on platters is bad. I want to try out Alma.
I used fedora ssd on btrfs and system was basically unusable. Now i use ex4 it is a lot better, but atill not as good as ubuntu
cant really get good tests off vms
and why not?
Grammatical error in title as a heads up.
Thanks, fixed it
My system runs much faster in opening and closing apps... even when copying files from internal to my usb drives on Fedora ... ubuntu feels laggy on my systems so will keep on using fedora thx
I'm still frustrated by the fact that people moved away from deadline to noop per latency reasons, when, in reality, they just ran into the difference that noop doesn't io merges and deadline per default does. deadline + no merges worked as fast just preserving a fair scheduler. and this misunderstanding was pushed upwards until most vm disk schedulers used noop. so now we're back to issues like swap stalls just because they hack around instead of writing/reading specs for their own stuff.
"sorting" and "merges" are NOT NOT NOT NOT a mandatory component of deadline.
the effects are disgusting on large applications but i suppose people deserve that.
Why do so many people love BTRFS and why does Fedora default to it when it looks horrible in these charts? Why is it so popular?
The preview is definitely AI-generated
yep it is
What, you can’t read the back of things without turning them around?
@@AWriterWandering they got them Nickelodeon displays
I can't wait for AI to design a computer, it will be an interesting design no doubt
@@CyberGizmo If you mean making a list of hardware to build a PC, ChatGPT already can do it. But it makes mistakes and the builds usually are not great
You see, I don't really care all that much about I/O, as today's NVMe disks are fast enough for any speed difference to be really noticeable. What I care for is the ability to make snapshots with BTRFS and revert to them when I need. Can Ubuntu do that with its outdated ext4 FS? no.
It's a totally different subject. And dj ware has talked about this subject. You can make an argument about "outdated ext4" but it's a total nonsense argument.
No you cannot use a feature developed for a file system on another one, quite illogical what you're saying.
@@benign4823 it's actually very logical. They are saying even if "slow" the features btrfs has make it more valuable to them. Snapshots are highly important to me as well so I fully understand.
@@SinisterSpatula It is, in fact, very illogical. They're comparing Ubuntu + ext4 with Fedora + Btrfs and asking why Ubuntu doesn't have Btrfs features like it's Ubuntu's fault that you picked ext4.
You can literally use Btrfs on anything, including Ubuntu, and their point of "well I can have Btrfs snapshots on Fedora so Fedora's better" is even more senseless considering that Fedora doesn't even have snapshots setup by default, and so you'd have to set it up, same way you'd set it up on any other distro minus OpenSUSE that has it by default, some immutable distros that use it and NixOS that has it's own rollback system that's not dependent on a fs.
You're not wrong but you're taking things too literally.
Looks like the bottom line is that BTRFS performance still sucks.
I use Xero Linux and CachyOS...
No problems...
They are both fast, responsive and crisp...
after I came back to Windows, I notice it also has problems with severe (unnecessary) Disk Caching for process and unusually high Pagefile (Swap) usage on Windows too. But here comes the deal: on Windows the Disk Paging doesn't bring my entire system to a halt like Linux did... something is deeply wrong in the core foundation of Linux kernel, and we've been using pretty much the same foundation from 2 GB spinning disk days (obsolete)
also I noticed this after going back to Windows: it doesn't do that thing anymore that Linux did, where things stayed in Swap even after I closed and reopened programs... Windows is more smart about flushing the Swap... Linux does not Flush swap ever and it becomes full (system enters kernal panic mode)
and it's not only dangeous when it comes to performance alone... we're talking Linux performing too much Disk Caching to a point where it can severely reduce SSD lifespans... we would have to do some kind of research Poll in the linux community in order to know how much in average people's SSD drives last before dying, SSD's became prevalent 2012 onwards so we don't have that data yet
let's give a real life usage example here: let's say you are trying to Livestream a game using OBS broadcasting software, running a moderately heavy game on the side and having a browser open (because you need to check your stream page, your live chat, etc etc etc) What you notice is that after 1 hour or so your memory in Linux maxes out and intense Disk paging occurs, bringing the whole thing to a chaos and even dropping frames on the broadcast (terrible for someone who needs a working system in order to present video to the public)
👍
ubunt-has no security be default.
'ooh-boon-two'
Security costs performance. :)
uhhh, I think you missed the boat here entirely - how does using a different scheduler have anything to do with security? You sound very confused (or very ill-informed)
@@gorak9000 BTRFS vs EXT4 vs XFS.
@@gorak9000 I think you're the confused one. It wasn't the scheduler, it was the filesystem. And btrfs, the slow one, has more security features.
@@gorak9000 I think you're confused. BTRFS has way more security features
Ooh-boon-too
Another conclusion would be, no matter what you do, BTRFS will suck eggs :P
The title should be "What's wrong with Fedora 38?"
This is a lot of noise about nothing. Applying the correct scheduler to the current hardware and situation is and was always the responsibility of the admin. Simple as that.
If the defaults of the distribution are not optimal out of the box for a certain situation it does not matter very much for the non-professional user and a professional user will take care of it anyways. So it does not really matter in practice.
So I think this is just a storm in a teacup and klick-bait - or just a waste of time at best.
Nothing against a video about scheduler variants and learning about it - but this has nothing to do with any specific distribution.
This is simply not an intelligent angle of approach to the topic.
Ubuntu is for noobs not for pro's.
Not entirely true. Sure, lots of noobs use it but even professionals at Google, Meta use it extensively. Even for ML it is widely used, NVIDIA too! It's great so is gentoo, opensuse, Fedora etc
I laughed hard at all those pushing Fedora 38 as a valid, everyday OS. Fedora has always been flaky at best. It's a test bed ONLY!
Personally, I think Fedora is a CROCK anyway--- every time I've tried it-- and I try it with every new version--- it's been SLOW_ clunky- and caused issues with my harddrive.. EVERY TIME.. it's a CROCK.. looked "cartoony" looking last issue too- and I hated that-- all the pastel colors and cartoon looking art.. like something a 3 year old made. I have Sparky linux- based on Debian- MUCH faster-- beautiful and kde highly customizable.. so why would I mess with slow, clunky, ugly- waste of space like Fedora???