idd , as a dj you dont wanna run into buffer underruns (wich are more likely to happen if you use 96khz samplerates - to reduce latency in the first place - and a 16 samples buffer ) if systemd-timers decide its time to start that btrfs maintenance task that you forgot to disable ) been using an rt-kernel and wine asio for that reason for a long time now allows me to play 96khz 16samples buffer using pipewire+wineasio and ableton in wine
the final revisions to enable RT code after 30 years gold wrapped with a purple bow is the most bureaucratic thing I have seen in a long time, for some reason it reminds me of that futurerama quote "Don't quote regulation to me! I co-chaired the committee that reviewed the recommendation to revise the color of the book that regulation is in. We kept it gray."
It's unfortunate that bureaucracy slows things down so much. But without it, crazy radicals would constantly change things back and forth their own ways. I'd rather a bureaucracy if it's what it takes to have stability in matters that affect a large number of people. I wish the bureaus were entirely staffed with people who know the subjects they govern. And that we could have them with ability to ensure they work for the people, not selfish power mongering oligarchs.
I had a tradition when I was much younger. Every year on December 31st I did build my custom "slimmest" kernel build. On a 466MHz Celeron it was a real paint.
@@jdrabCrazy how now I sometimes rebuild my kernel multiple times in a row in less than 10 minutes, because I forgot to enable something and Portage is telling me it needs that for a package. 466MHz Celeron -> 3950x (not that new anymore) Edit: I forgot, without LTO it would even take much less. Also, I never rebuild, I have a script that mrpropers, cleans and builds.
@@elpelicanojiji Well there started to appear distros like Ubuntu Studio and such which had RT kernels. I've had Mac for quite long now so I don't know the situation, but like the video says, RT is now part of the mainline kernel so I reckon you don't have to compile it yourself.
@@Winnetou17 You understand the actual market share is drastically under reported? And Windows market share is drastically over reported, which has the effect of further reducing the overall percentages for Linux? Linux actually has desktop share on par with MacOS. The Year of Linux desktop came and went years ago.
Real time makes you not ingest data and digest data in chunks and choke. For Video processing and other places where "buffering" is disastrous (you thought it was bad watching ;) ) this is one helluva god thing for Linux. A person first looking into Linux for media like audio or video processing and design no longer has to be told to go straight into "how to recompile or change kernel" just in the very quick end to say "Windows and $500 bucks for this & That it has to be... ". This is a good thing.
I was just about to "ackshually..." when you said "20 years ago". So yes, very momentous - i remember those realtime special kernel builds from 20 years ago.
Real time kernel in nutshell: sacrifice some throughput all the time to avoid bad worst case behavior. Generic kernel is about maximum throughput while ignoring rare edge cases with high latency (e.g. it's acceptable that system handles some event with 500 ms extra delay randomly a couple of times a day if that allows the system to get more work done per second on average). In gaming terms, generic kernel is about average FPS or 1% low FPS, but real time kernel is purely about 0.0000000001% low FPS.
As long as things keep improving, and if a 'micro controller' with Linux with PREEMPT_RT and busybox and your application are reliable then I'm all for it.
@@autohmaeMost of the time, it would be better to have a fully RT MCU controlling the hardware, and use a Linux board with a PREEMPT_RT kernel and a fast interface to the MCU as the control plane. It’s the separation of concerns, as in other fields like dataflow, where you have your predictable, low latency data plane which is just a dumb worker, and the control plane which might have more long running and complicated tasks (branching) driving the decisions for the data plane.
@@LtdJorge I don't know much about Linus or computers, because I have a Macbook Pro and iPhone, all I know is that I prefer graphics over text (that's why I've never tried Linus). But what is the reason you call the workers dumb? Is it because they are doing manual labour instead of typing on a computer in an office? That is very narrow minded of you in that case, and you should reconsider saying things like that. Remember cooking food is a manual task, and that writing text or watching streaming on a computer will not produce any food - not even my iPhone can do that.
Preempt ==> low latency Rt kernel will be optimized for response time over throughput. Want high throughput kernel for servers. Want rt kernel for, e.g., audio, etc. Want a normal kernel for desktops
You may want RT kernel even for servers if you're experienced in worst case performance instead of average performance. Depending on the use case, the clients may require known timeouts.
Most of the benefits of the RT work are already applied in the regular kernel and this has been the case for a few years now. This is why Linux desktops running modern kernels are more responsive than ever. The battle for desktops is on improving the system scheduler to squeeze better user input responsiveness while remaining as fast as possible. 6.12 and beyond should see benefits regarding that.
@@SterileNeutrino I can't point you at anything specific, but the way X11 works makes it especially difficult. Wayland is likely to further improve things there.
PREEMPT RT , "Real Time Patch" etc have always been misnomers. Linux is not a Real Time OS , with or wirhout the RT patch. It should be called the "lower latency" patch or something like that. Linux will be better for multimedia streaming and responsive applications, but this does not really open the door for actual real time domain, where real time OSes dominate, in automotive, aviation, and other deeply embedded domains.
@@YellowCable Lower latency is the entire point of the preempt code. But Linux does in fact have an RT scheduler. Making it an RT kernel. One can split hairs on soft vs hard, but Linux is factually an RT kernel so long as it's appropriately compiled.
See a lot of misconceptions... RT doesnt make latency lower. It makes the time between an event and the code be predictible. That;s all. If you have something sampling 100's of channels of adio or any measurement, all of them will have the same latency with the signal. Even if the latency is 30 seconds. Thats what real time means. In audio that means that if you are recording a guitar, a drummer, a bassist a singer and a whole orchestra, all of the mics will receive the audio in sync and be mixed in sync. The latency can be 1ms for a super duper multicore machine, or 10secons for a raspberry pi. It doesn't matter. If the buffers are big enough, the recording will be exactly the same.
Will be great for LinuxCNC which has required pre-empt RT for a long time but always suffered the need to carefully choose the distribution on which to install.
I would argue that real-time might be a huge difference for linux gaming. One of the most long-standing issues is audio skips and stutters which are happening even when overall system have plenty of resources to run the game.
These are more related to pulse audio and related problems with Linux audio. The fairly recent move to Pipewire as the preferred audio subsystem will iron out a lot of these issues. For games, you can get away with latencies in the 50-100 ms range and no one will notice. For professional audio production, especially recording or creating music, you need sub 50ms reliability, and ideally lower than 20ms. It also has to be sample accurate at high sample rates.
@@Intermernet In my current experience pipewire does not fully resolve clicking and skips without asking it to be not nice. I agree, that professional sound work has much higher requirement for latency and it is more important there, but it's still something that will be very noticeable for gamers.
Again, RT linux will not help in that. It doesn't magically solves anything. Needs to be coded for it. It just ensures a minimum time between interruption and code. It has no idea if the code is audio or game or physics or what.The code needs to take than in account. A Game, particularly, will need to be structured completelly different to be able to use it. A RT linux will be probably slower for Gaming. It the case of a machine entirelly dedicated to sampling stuff (A audio station, a scientific data recorder, etc...) may help generically because ensures all interrupts are equals and all channels have the same time to do their thing. That said, most of those programs have special flags for RT kernels.
@@framegrace1 Yes, I understand, but since majority on linux gaming right now are not native games, it only needs to be accounted in the wine/proton layer which is pretty realistic to expect once real-time core become commonplace. Also, since issue is usually fixed by setting nice -1 to audio server, it might be affected on the level of how it can interact with the core without additional changes. Also, gaming performance is not a singular thing, games I had sound issues where running at 60 fps with not visual stutter and spare cpu, only sound is an issue. Getting +2% cpu until or even losing five frames out of the hundred is not a meaningful penalty on the performance in exchange for fixing sound stutters. Basically that is what I do, setting negative nice anyway. IDK, why do you not want me to be glad for something which might improve user experience down the line? I don't expect that it will solve something straight tomorrow and I expect that some developers will still find a way to fck up performance, I played in Arma 3, but it still likely to gradually improve general situation in some time, 2 to 4 years i guess with next wave of LTS releases it might be common enough?
We have been using patches to make Linux pre-emptive and real time for ages. I see some stupid comments below. Just listen to what the nice man in the video is saying. Linux RT is great for industrial applications like drives or PLC’s using communications like ProfiNet etc where you set up things to execute in determined time slots and communicate etc at set intervals. So like he says, not necessarily fast, but necessarily predictable. You don’t need that in general computing. But things like signal processing etc can do with it. And no, it’s not for safety applications. But it can be part of a safety type application. RTOS etc are for much simpler applications where you don’t need a full on operating system like Linux. So you code according to the need. Either bare bones, or some scheduling scheme, or RTOS etc, or a full on OS.
ASIO has nothing to do with RT. ASIO is just hardware comunicating with hardware directly, very fast. No guarantees of latency and only works for audio sampling. RT doesn't try to make anything faster. Just ensures that when you are mixing 100 audio channels for example, all of them will have the same latency with the signal. Thats what real time means. All inputs are processed together at the same time at a certain latency (The drummer and the bassist will be always on time) the latency will only depend on the power of the machine, it can me ms or seconds. But everything will be on time.
Anyone else remember when "pre-emptive multitasking" was one of the huge buzzwords for Windows 95? (Win 3.1 was based on voluntary multitasking, meaning that it could ask nicely but it was ultimately up to the process to give back control of the CPU). Linux, being awesome right out of the gate, had pre-emptive multitasking right out of the gate. Unfortunately both Linux and Windows from that era were still susceptible to processes hogging the processor if they were running in kernelspace.
@@JPs-q1o I believe Windows 95 / 98 did NOT have pre-emptive multitasking. Windows NT did, but was often used only in business environments - (notably many games did not run on NT). It was Windows XP that merged those two projects, and brought pre-emptive multitasking to the masses.
Oh, actually, I suppose the innovation in NT and Windows XP was more about using an MMU that truly prevented processes from crashing the whole machine. Maybe 95 did in fact introduce preemption.
I have been using linuxcnc with real time kernels for my cnc router since 2011. One big hurdle to good performance is that the system management interface runs whenever it wants regardless of the operating system kernel leading to a lot of jitter in the timing of control loops. Apparently it is more important to check fan speeds than running my router. Solution is to disable acpi / smi.
The kernel scheduler is a rather complex thing (far beyond setting niceness values of processes), but it will allow you to manually override that sort of behavior even within the normal kernel.
@@mytech6779system management mode runs below the operating system and any hypervisors and has absolute control over the processor. You can't patch the code that it's running because it's signed in firmware and its segment of memory is very heavily protected against reading/writing from other processes.
@@arlobubble3748 I didn't realize the OP was referring to ring -1 . Yeah, that is just the result of poor hardware for the task. aka non-RT hardware. Though that level can still be altered it generally isn't worth the effort when the end result will still be mediocre in the best case.
@@mytech6779 I mean there may be instances where you could but it's not exactly realistic for hobbyist level stuff. Motherboard manufacturers have access to development kits that let them alter SMM code that used to be used for all sorts of things like USB keyboard support on old BIOS based systems, and I'm sure many more features even though it wasn't really made for that purpose. If you were a manufacturer making custom boards to run the x86 processor on you could probably pay for access to a development kit and modify it to suit your needs.
About 15 years ago, I was learning about CNC machining -- and I had even gone so far as to compile a real time kernel to check out an app on Linux that needed it -- and although I never went much farther than that, it gave me the experience I needed to be *very* excited by this news when I heard it a couple of weeks ago! I have since thought about a different goal: four or five years ago, I saw the suggestion that a computer should always be responsive to the human -- and how I'd get frustrated when I'd try something, and it would slow my computer to a crawl, to the point where I can't move my mouse pointer, and it would take a few minutes for ^C to kill a process -- oh, how I wished for responsive computing! To this end, I'd like a real-time UI that gives the app in focus enough real-time access to be responsive too. I'm not in a position (I don't think) to create such a UI, but I'm glad that this opens the door to that!
The ability to do what you describe depends on what you want the app in focus to do in response. You can't guarantee that it will be able to do anything you want.
Maybe not games per se, but their physics engines or their collision detection might give this some use. Specially with faster action RPGs sword swinging games like the souls games where you literally only have a few frames to try your best at low latency detection of what enemies and the player are doing. It's so hit and miss that they have to resort to huge sausage capsule collider (and sphere colliders) for hitboxes to make it look good, and even then sometimes people rightly complain about dodgy hitboxes.
Realtime means that you can run very dangerous software that needs realtime reaction to inputs and predictable latency, like medical equipment, factory robots, self driving cars. But hardware that needs this uses other, smaller operating systems (RTOS), so I doubt the bloated Linux kernel will be used for that.
Been used for this stuff for ages. Real time simply means guaranteed worst case latency. Many people believe it means fast. Real time usually means lower bandwidth in exchange for increased reliability to maintain the worst case latency. Common example of where real time make sense is audio processing. Thus the birth of Jack audio server. Heck, I've even seen RT Linux used with CORBA for real time flight control avionics. As well as radar processing. Real time doesn't have to mean embedded nor specifically embedded processors. It simply means one has a worst case latency requirement.
Pretty much agree with all of this, although in the embedded world, in many applications you don't even need an RTOS. Totally agree that Linux bloat in many embedded applications is a PITA, not least boot time, the energy required, and cost of hardware to support it. I'll add one more factor... time deterministic behaviour means everything in low level high speed applications, such as motor control or power supply feedback loops.
@@rezah336this is not my expertise however there are many safety regulations in industries like medical equipment or assembly line equipment etc. and thus certifications that you meet these safety requirements. So to make a device for these industries you must use a certified kernel that meets safety regulations and these certifications give those kernels high pricing.
@@rezah336 Sure. Liability. If your medical pump misses an interrupt which halts the pump, and creates an overdose, who pays for that? This is just one example. QNX and VxWorks are very popular and their pricing reflects the reality of liabilities and support for their products. Not to mention, someone has to port to new hardware and new peripherals. I've played with QNX. Thought it was pretty cool. Used VxWorks for P95 radio, including the entire init process. Something I did was used at NASA for rover testing. And I also do embedded development, in addition to some phone stuff on the application processor side. The RT world heavily overlaps with the embedded world but it's not exclusively so. As nez rightly points out, many applications within the embedded world (most?) use no OS at all. Even still, for many non mission critical apps, there are many RTOS solutions available (example, FreeRTOS). Cheap hardware (couple bucks) is available to play with much of this.
Latency is actually quite relevant in gaming, in particular the delay from last input to the next frame flip with updates based on that input. Then, though, availability of variable refresh without excessive buffering is sufficient: Latency needs to be low, yes, but timing doesn't need to be evenly spaced humans are *very* bad at distinguishing 75fps from 74fps so it's no biggie if you're a bit late. Audio is of course also involved but we're quite tolerant of audio latency because sound is naturally slow so as long as you can fill that ring buffer you're generally fine. Haptics probably needs RT, but then it also needs 1000fps so that's a whole different kind of beast anyway.
Very well explained for someone like me that is new to RT 👍 Would love to have a video dedicated the history, theory and implementation if RT in different architectures (like VAX for ATC, etc) Maybe too much to ask for in one video 😀
This real-time thing sound real cool, really good example with the Tesla, the metaphor is very clear. Kind of scary how a task can take over the whole system, maybe I don't understand that properly but the concept when said like that sound a bit scary.
Nice for labs, where you need to poll your sensors, synchronize the data. Especially when your polling speed is fast and you need to organize a feedback
RT and the OS cannot do miracles. It just promises lower latency. This basically just gives you *even higher priority* and ensures that the kernel itself will not itself run on that high priority and interfere with your stuff. But there is no way to prevent several tasks requesting RT priority and fighting for the CPU resources and exhausting the system.
The realtime patch has been available for anyone to put in the kernel when they want. And the people who need realtime already had the technical competence to do it.
Not really, for the past at least 5 years all major distros have had a Preempt-RT kernel right in their package manager. I've been using Preempt-RT for like 10 years but I haven't patched one myself in at least 5.
Lions have gotten 4ms audio latency forever but it's always been getting incrementally more reliable. It'll never be able to bit bang a servo, bit bang LED brightness, or run clockcycle accurate emulators from user space.
honestly, for a desktop computer, I usually care more about maximum latency than throughput. i of course don't care about microsecond or even millisecond latencies, but if getting a more responsive system and fewer skipped frames means that background tasks take twice as much time, so be it. the main case where performance is important is when the soft-realtime tasks are often close to their limit, so that decreasing their throughput for the sake of determinism would mean regularly missing deadlines, but I'd argue that that's a design flaw for that application. Video games are an excellent example of this, where missed/delayed frames sucks and you're often close to the limit, but you should aim to use settings where you have some margin on performance to avoid that. however, if i am doing data processing of some kind (video processing, compilation etc.), then performance can actually be more important than latency, since those things can take many minutes that you don't want to wait unnecessarily long for, but even then, I might want to keep using the system while I'm waiting for the task, so I'm still ready to sacrifice some throughput for latency
It is not true that this means a reliable latency limit, because modern CPUs have a security kernel built in that can interrupt anything CPU is doing and cause unacceptable latency. I think this is not likely but it can happen. This is why important realtime things are and will be done on specialty hardware. Which may use the Linux kernel. In that case it would be reliable.
Yup. Industrial machines I work on have relays and long loops of wire for E-stops and door interlocks. Opening a door or smashing the big red button will open relays to make things safe. If it doesn't completely shut off the power a spare relay contact will tell the Real-Time server what happened when it can spare some cycles. Even programmable logic components aren't reliable enough, I guess. Several hundred feet of wire jumping to dozens of relays around the machine "just works."
Even for desktop gaming-/workstations, when it’s less than 1% performance penalty e.g. FPS in Counter-Strike, I would prefer having e.g. pipewire running in RT mode. Reliability is better than up to 1% performance penalty. I mean some processes can need it, right? Better making it possible that they - can - have it when needed, than not having the possibility, and that just enabled for those processes who need it, by default, that would probably be nice in my limited view. I have no idea, when I‘d install e.g. future Arch Linux´ kernel 6.12+, will that possibility be enabled there? No kernel parameters or whatever needed, just process priority change from whatever to rt? Will it be possible in Linux-zen which I use? As mentioned, I‘d want to use pipewire with rt, I‘d test FPS in CS. But as seen in the text, I am still not sure how it will or won’t work; I am in for the mentioned idea. More consistent latencies are nicer than little bit performance. It would require a lot of testing, but imagine, as mentioned potentially less than 1% penalty, but less stuttering as well. Because that often comes because of inconsistency in e.g. frame times, maybe rt improves those a little bit, especially when not closing the billion chromium tabs in the background. Game and pipewire having high priority and rt mode could help I guess in that situation, but needs testing, and I just speculate, I have never used anything RT. 🤷
Basically, RT just ensures all interrupts take a minum time. That's all. TO ensure that ALL kernel must be RT. There's no other option. You can do what you are explaining on normal kernels, just give pipewire a higher priority with `nice`. But most probably you will only achieve that the game will stutter instead of the audio. Which I guess is much worse.
guys, tell me the latency or jitter that you get and how you achieve it i get 40 micro seconds sample jitter with 'sudo chrt -f -p 99 PID' standard xubuntu 18.04
IMO games could in principle benefit from RT support, especially with high refresh rates, to actually ensure consistent, guaranteed time to next frame... but it seems like everyone gave up on consistent frame times in games. This should be especially useful for VR
nice, can i mark this on a running program from htop without needing to have the program know about it? it's basically letting one process have prioritized compute right?
I've tried the RT version before for music production, in hopes of getting the lowest possible latency when record monitoring, but it never seemed to make much difference.
Used correctly with software designed to make use of it (e.g. Jack or PipeWire with Reaper or Ardour), it can reduce your minimum buffer/frame size without underruns, but you still have to select that smaller frame to get the benefit.
That's a lesson in perseverance. If someone can be so stubborn they can make Linus Torvalds relent control of one of his babies, nothing is impossible if you put enough effort.
I'm imagining a room full of industrial equpment designers perking up at this news like a bunch of cats when you announce they're getting fish for dinner.
Low latency is not a given! You can even have more latency as the RTOS scheduler decides to give your process a lower precedence because other time critical processes are set to have a high time response. The only thing RT guarantees is that specific task will start in a specific time frame. And if it will need to stop for example user input or data acquisition then it will do just that.
Does it allow for reserving processing and etc? Without being able to tell whether other processes are too heavy to actually respond within a deadline, I'm not sure this is actually "real time."
I believe QSC Q-SYS is built on a realtime variant of linux. It's pretty great but the prices are... well, they're great too, but a different kind of great :/ perhaps this will foment something great in a good way :D
I still don't understand: Does this generally allow for lower latency, or just for more relability in exceptional cases (like other programs running in the background trying to do something)?
Do you think a real-time linux (and some changes in other areas) might improve a desktop linux's responsiveness to mouse and keyboard when working under stress?
Gaming is absolutely a relevant application. It will let you make a game that has no tearing or judders in animation, and consistent latency from user input to the results onscreen. Right now if you want that you have to brute force it with extremely high framerates.
That was too much waffle. The best way to explain what it means is to provide a real-life demo example. What you did was reword the same metaphor about priority multiple different ways.
I'm running PREEMPT kernel even on desktop system because it seems to reduce stutter, jitter and random stalls enough to be noticeable for me. However, it causes lower average throughput (for the same hardware) so it's not without side-effects either. If your brain is happy to use e.g. Bluetooth audio which has built-in delay around 40-200 ms then you probably don't need RT or PREEMPT for anything. I'm unfortunate enough to have brain that can detect latency around 6 ms and I cannot stand to use bluetooth audio because of its latency, and I'm also very sensitive to jitter in the display. For me, it's better to sacrifice some average performance to get rid of those latency spikes that become too noticeable for my taste. I'm looking forward to see if I can see the difference between RT and PREEMPT kernels. I'd hope not because otherwise it would mean I'm probably going to have to sacrifice even more throughput to run RT kernel at all times. If the kernel is controlling something external such as CNC machinery, then RT kernel is obviously mandatory because it is not acceptable for kernel to randomly fail to stop the moving parts of the machines at the correct time. The traditional solution without RT kernels has been more hardware: instead of kernel controlling the CNC machine directly, it controls an extra piece of hardware that does nothing but runs the machine constantly. If you don't have multitasking kernel nor use hardware that can randomly run SMI at will, you have 100% known latency at all times. RT kernel is about allowing hardware to be used for true realtime processing at the same time as using extra CPU cycles for non-RT tasks such as user visible GUI interface.
so they're saying they'll be crowstrike BSOD'ing linux now?
2 месяца назад
Funnily enough, your self-driving example is actually quite close to reality: openpilot runs several processes on linux, some of them not being RT-critical (UI), others are. But I don't know if openpilot uses RT kernel etc.
Well no akshually. A realtime kernel has nothing to with process priority levels and starving processes. You may be thinking of Windows' realtime priority class, which is its highest priority class, everything >= priority 16. The priority class is called realtime but Windows is not a realtime operating system. A kernel, realtime or not, can perfectly well limit a high priority process so it doesn't starve lower priority processes, if it's made to.
@@xlerb1637 By definition, a higher priority task should starve lower priority tasks. That's why they are prioritized accordingly. If one shouldn't starve the other they should have the same priority. Priorities basically says I want the higher priority to run even if a lower priority task wants to. That's why it's higher priority. It's more important it runs than the lower priority task.
@@justanothercomment416 Linux has built into it, I don't remember the details, the ability to limit CPU time to a high priority process so an out-of-control process doesn't kill the system.
Fun things are fun, but this was misleading. printk() was dear not just to Linus, but to all developers dealing with kernel crashes. To ensure that last debug messages and dumps appear on serial console or screen, it was given shortcuts to deal with logging directly, with minimal involvement of the rest of the system code (which might already be garbage or point to nowhere in case of memory corruption). Special case included bypassing the scheduler (because any context switch could be the last on a broken system). Because that parallel prioritisation system was not playing by the rules, calls to printk() in kernel code could take control of the system for unpredictable amount of time. Not that important on a general purpose system (for a code that doesn't interact with printk() and its special handling), but breaking all guarantees if you need predictable low latency.
Interesting, I use arch and I think I use an RT kernel. What did they change about printk that allowed this to be mainline?
2 месяца назад
If only he had delivered the scroll on a pillow 😅 Anyway, no need to wait for a new machine. Just build an extra initramfs and pick it in GRUB and you're done 😁
Well, if you ever tried to implement PWM on Raspberry Pi (versus Arduino) and ask yourself why it's so sporadic, well, RT will make it possible not to be sporadic.
It's scary that the absurd scenario of the self-driving logic running on the same computer as the AC and the onboard entertainment system, is actually a plausible thing...
Wow so it’s true real time. Too often, real time has just been used to mean ‘fast response’, which is not what it means. The classical real-time system is the industrial PLC, which can give a guaranteed maximum response time to react to control inputs.
Your explanation seems to beg one important question: What IS the maximum guaranteed latency for the RT Linux kernel? I assume it's dependent on the processor's clock speed, but if the key takeaway is that there is a known maximum latency, how is that calculated?
A companion page to a 2020 paper stated: "The blocking (or interference-free latency in the paper) is defined in Lemma 7 as: LIF ≤ max(DST, DPOID) + DPAIE + DPSD"
@@XGD5layer Since you can't post links, can you give me the exact name of the paper? I have no idea what any of those acronyms are (or what Lemma 7 is for that matter), but I assume they are defined in the paper somewhere.
If someone can find that video of Linux Plumbers Conference Where the developers handle him the gold code changes? It would be funny to watch, could not find by searching it
Real-Time is huge for processing live audio where minimum possible latency is desirable.
These patches decrease the highest latency peaks. Real-time != low latency
Drones.
Pretty much anything a drone is doing can be improved with a real-time kernel.
real-time = predictable, but maybe not fast.
idd , as a dj you dont wanna run into buffer underruns (wich are more likely to happen if you use 96khz samplerates - to reduce latency in the first place - and a 16 samples buffer ) if systemd-timers decide its time to start that btrfs maintenance task that you forgot to disable ) been using an rt-kernel and wine asio for that reason for a long time now allows me to play 96khz 16samples buffer using pipewire+wineasio and ableton in wine
Please DM me if you're looking for someone to talk to about RT-DSP for live audio...
the final revisions to enable RT code after 30 years gold wrapped with a purple bow is the most bureaucratic thing I have seen in a long time, for some reason it reminds me of that futurerama quote "Don't quote regulation to me! I co-chaired the committee that reviewed the recommendation to revise the color of the book that regulation is in. We kept it gray."
It's unfortunate that bureaucracy slows things down so much. But without it, crazy radicals would constantly change things back and forth their own ways. I'd rather a bureaucracy if it's what it takes to have stability in matters that affect a large number of people. I wish the bureaus were entirely staffed with people who know the subjects they govern. And that we could have them with ability to ensure they work for the people, not selfish power mongering oligarchs.
Recognizing that PREEMPT_RT is mature enough to be included in the mainline is bureaucratic?
A quarter century of RTLinux means morning to this channel, it seems.
real time support will be a blessing for midi clocked applications
I remember when you had to build your own RT kernels for audio Linuxes back in the day.
I had a tradition when I was much younger. Every year on December 31st I did build my custom "slimmest" kernel build. On a 466MHz Celeron it was a real paint.
@@jdrabCrazy how now I sometimes rebuild my kernel multiple times in a row in less than 10 minutes, because I forgot to enable something and Portage is telling me it needs that for a package. 466MHz Celeron -> 3950x (not that new anymore)
Edit: I forgot, without LTO it would even take much less. Also, I never rebuild, I have a script that mrpropers, cleans and builds.
@@jdrab Had the exact same CPU, and an even more extreme habit of building kernels with every stable release.
How is it now? There is no need to compile a custom kernel anymore?
@@elpelicanojiji Well there started to appear distros like Ubuntu Studio and such which had RT kernels. I've had Mac for quite long now so I don't know the situation, but like the video says, RT is now part of the mainline kernel so I reckon you don't have to compile it yourself.
We'll be so screwed once Linus retires.
systemd will become the kernel
@@BillinSD is this a joke?
@@BillinSD Three letters just got aroused.
@@MM-do5yx There is no fking way
No hurries. He’s only 54
This and the recent FreeCAD 1.0 announcement bode well for CNC tinkerers using Linux.
Yep, that was my first thought... CNC/3D printer controllers using an SOC with a full UI built in would be nice.
Now we only need GIMP 3.0 to launch to wrap up this year. With a proper 5% Linux market share :)
@@Winnetou17 You understand the actual market share is drastically under reported? And Windows market share is drastically over reported, which has the effect of further reducing the overall percentages for Linux?
Linux actually has desktop share on par with MacOS. The Year of Linux desktop came and went years ago.
Which flavors FreeCAD will run on?
@@justanothercomment416 Any sources on that ?
Real time makes you not ingest data and digest data in chunks and choke. For Video processing and other places where "buffering" is disastrous (you thought it was bad watching ;) ) this is one helluva god thing for Linux. A person first looking into Linux for media like audio or video processing and design no longer has to be told to go straight into "how to recompile or change kernel" just in the very quick end to say "Windows and $500 bucks for this & That it has to be... ". This is a good thing.
I was just about to "ackshually..." when you said "20 years ago". So yes, very momentous - i remember those realtime special kernel builds from 20 years ago.
Real time kernel in nutshell: sacrifice some throughput all the time to avoid bad worst case behavior. Generic kernel is about maximum throughput while ignoring rare edge cases with high latency (e.g. it's acceptable that system handles some event with 500 ms extra delay randomly a couple of times a day if that allows the system to get more work done per second on average).
In gaming terms, generic kernel is about average FPS or 1% low FPS, but real time kernel is purely about 0.0000000001% low FPS.
I second the idea of having a scroll with the commits but also a ceremony for major merges.
This RT merge should become an official holiday for Linux devs.
Just a reminder that OS using Linux with PREEMPT_RT is still *not* true RTOS.
As long as things keep improving, and if a 'micro controller' with Linux with PREEMPT_RT and busybox and your application are reliable then I'm all for it.
@@autohmaeMost of the time, it would be better to have a fully RT MCU controlling the hardware, and use a Linux board with a PREEMPT_RT kernel and a fast interface to the MCU as the control plane.
It’s the separation of concerns, as in other fields like dataflow, where you have your predictable, low latency data plane which is just a dumb worker, and the control plane which might have more long running and complicated tasks (branching) driving the decisions for the data plane.
RTAI is a lot closer to RTOS. Too bad they lack funding and developers.
@@LtdJorge I don't know much about Linus or computers, because I have a Macbook Pro and iPhone, all I know is that I prefer graphics over text (that's why I've never tried Linus). But what is the reason you call the workers dumb? Is it because they are doing manual labour instead of typing on a computer in an office? That is very narrow minded of you in that case, and you should reconsider saying things like that. Remember cooking food is a manual task, and that writing text or watching streaming on a computer will not produce any food - not even my iPhone can do that.
@@oonmm the term "dumb worker" in this sense refers to part of the software running on the computer. It's not referring to a person.
Preempt ==> low latency
Rt kernel will be optimized for response time over throughput.
Want high throughput kernel for servers.
Want rt kernel for, e.g., audio, etc.
Want a normal kernel for desktops
You may want RT kernel even for servers if you're experienced in worst case performance instead of average performance. Depending on the use case, the clients may require known timeouts.
Most of the benefits of the RT work are already applied in the regular kernel and this has been the case for a few years now. This is why Linux desktops running modern kernels are more responsive than ever. The battle for desktops is on improving the system scheduler to squeeze better user input responsiveness while remaining as fast as possible. 6.12 and beyond should see benefits regarding that.
Is there anywhere a writeup on how this is done practically? (An article in ACM Queue maybe)
@@SterileNeutrino I can't point you at anything specific, but the way X11 works makes it especially difficult. Wayland is likely to further improve things there.
battle for desktops is about hardware and software support
PREEMPT RT , "Real Time Patch" etc have always been misnomers. Linux is not a Real Time OS , with or wirhout the RT patch.
It should be called the "lower latency" patch or something like that. Linux will be better for multimedia streaming and responsive applications, but this does not really open the door for actual real time domain, where real time OSes dominate, in automotive, aviation, and other deeply embedded domains.
@@YellowCable Lower latency is the entire point of the preempt code. But Linux does in fact have an RT scheduler. Making it an RT kernel. One can split hairs on soft vs hard, but Linux is factually an RT kernel so long as it's appropriately compiled.
See a lot of misconceptions... RT doesnt make latency lower. It makes the time between an event and the code be predictible. That;s all.
If you have something sampling 100's of channels of adio or any measurement, all of them will have the same latency with the signal. Even if the latency is 30 seconds.
Thats what real time means.
In audio that means that if you are recording a guitar, a drummer, a bassist a singer and a whole orchestra, all of the mics will receive the audio in sync and be mixed in sync. The latency can be 1ms for a super duper multicore machine, or 10secons for a raspberry pi. It doesn't matter. If the buffers are big enough, the recording will be exactly the same.
Will be great for LinuxCNC which has required pre-empt RT for a long time but always suffered the need to carefully choose the distribution on which to install.
I would argue that real-time might be a huge difference for linux gaming.
One of the most long-standing issues is audio skips and stutters which are happening even when overall system have plenty of resources to run the game.
These are more related to pulse audio and related problems with Linux audio. The fairly recent move to Pipewire as the preferred audio subsystem will iron out a lot of these issues. For games, you can get away with latencies in the 50-100 ms range and no one will notice. For professional audio production, especially recording or creating music, you need sub 50ms reliability, and ideally lower than 20ms. It also has to be sample accurate at high sample rates.
@@Intermernet In my current experience pipewire does not fully resolve clicking and skips without asking it to be not nice.
I agree, that professional sound work has much higher requirement for latency and it is more important there, but it's still something that will be very noticeable for gamers.
Again, RT linux will not help in that. It doesn't magically solves anything. Needs to be coded for it. It just ensures a minimum time between interruption and code. It has no idea if the code is audio or game or physics or what.The code needs to take than in account.
A Game, particularly, will need to be structured completelly different to be able to use it.
A RT linux will be probably slower for Gaming.
It the case of a machine entirelly dedicated to sampling stuff (A audio station, a scientific data recorder, etc...) may help generically because ensures all interrupts are equals and all channels have the same time to do their thing. That said, most of those programs have special flags for RT kernels.
@@framegrace1 Yes, I understand, but since majority on linux gaming right now are not native games, it only needs to be accounted in the wine/proton layer which is pretty realistic to expect once real-time core become commonplace. Also, since issue is usually fixed by setting nice -1 to audio server, it might be affected on the level of how it can interact with the core without additional changes.
Also, gaming performance is not a singular thing, games I had sound issues where running at 60 fps with not visual stutter and spare cpu, only sound is an issue. Getting +2% cpu until or even losing five frames out of the hundred is not a meaningful penalty on the performance in exchange for fixing sound stutters. Basically that is what I do, setting negative nice anyway.
IDK, why do you not want me to be glad for something which might improve user experience down the line? I don't expect that it will solve something straight tomorrow and I expect that some developers will still find a way to fck up performance, I played in Arma 3, but it still likely to gradually improve general situation in some time, 2 to 4 years i guess with next wave of LTS releases it might be common enough?
I noticed that Starfield on PC runs at an elevated priority (i.e., a negative "nice") when you start it up, just for that reason.
We have been using patches to make Linux pre-emptive and real time for ages. I see some stupid comments below. Just listen to what the nice man in the video is saying. Linux RT is great for industrial applications like drives or PLC’s using communications like ProfiNet etc where you set up things to execute in determined time slots and communicate etc at set intervals. So like he says, not necessarily fast, but necessarily predictable. You don’t need that in general computing. But things like signal processing etc can do with it. And no, it’s not for safety applications. But it can be part of a safety type application. RTOS etc are for much simpler applications where you don’t need a full on operating system like Linux. So you code according to the need. Either bare bones, or some scheduling scheme, or RTOS etc, or a full on OS.
The first thing i thought of when i heard the news was audio production similar to the asio in Windows. Im glad that hunch was somewhat correct!
ASIO has nothing to do with RT. ASIO is just hardware comunicating with hardware directly, very fast. No guarantees of latency and only works for audio sampling.
RT doesn't try to make anything faster. Just ensures that when you are mixing 100 audio channels for example, all of them will have the same latency with the signal.
Thats what real time means. All inputs are processed together at the same time at a certain latency (The drummer and the bassist will be always on time) the latency will only depend on the power of the machine, it can me ms or seconds. But everything will be on time.
Best update since kernel 3.11 for workgroups.
Anyone else remember when "pre-emptive multitasking" was one of the huge buzzwords for Windows 95? (Win 3.1 was based on voluntary multitasking, meaning that it could ask nicely but it was ultimately up to the process to give back control of the CPU).
Linux, being awesome right out of the gate, had pre-emptive multitasking right out of the gate.
Unfortunately both Linux and Windows from that era were still susceptible to processes hogging the processor if they were running in kernelspace.
@@JPs-q1o I still remember when SMP was new to Linux and it had the Big Kernel Lock.
@@JPs-q1o I believe Windows 95 / 98 did NOT have pre-emptive multitasking. Windows NT did, but was often used only in business environments - (notably many games did not run on NT). It was Windows XP that merged those two projects, and brought pre-emptive multitasking to the masses.
Oh, actually, I suppose the innovation in NT and Windows XP was more about using an MMU that truly prevented processes from crashing the whole machine. Maybe 95 did in fact introduce preemption.
@@carterthaxton I believe they did for the most part as did 3.x in 386 mode. No memory protection though, NT had that.
Thanks dude! Great video! Very informative and interesting
Really helpful for a low cost, single CPU panel with a high priority controller application but a lower priority UI.
I've been following QNX, RTLinux, etc. for decades. This is huge.
We can finally throw QNX and it's horrible development environment into the trash bin.
I have been using linuxcnc with real time kernels for my cnc router since 2011. One big hurdle to good performance is that the system management interface runs whenever it wants regardless of the operating system kernel leading to a lot of jitter in the timing of control loops. Apparently it is more important to check fan speeds than running my router. Solution is to disable acpi / smi.
The kernel scheduler is a rather complex thing (far beyond setting niceness values of processes), but it will allow you to manually override that sort of behavior even within the normal kernel.
@@mytech6779system management mode runs below the operating system and any hypervisors and has absolute control over the processor. You can't patch the code that it's running because it's signed in firmware and its segment of memory is very heavily protected against reading/writing from other processes.
@@arlobubble3748 I didn't realize the OP was referring to ring -1 .
Yeah, that is just the result of poor hardware for the task. aka non-RT hardware.
Though that level can still be altered it generally isn't worth the effort when the end result will still be mediocre in the best case.
@@mytech6779 I mean there may be instances where you could but it's not exactly realistic for hobbyist level stuff. Motherboard manufacturers have access to development kits that let them alter SMM code that used to be used for all sorts of things like USB keyboard support on old BIOS based systems, and I'm sure many more features even though it wasn't really made for that purpose. If you were a manufacturer making custom boards to run the x86 processor on you could probably pay for access to a development kit and modify it to suit your needs.
Yeah, one of these days maybe Oxide Computer Corp will save us from ACPI on the PC.
Linux has potential to go ballistic now 😅
Literally, it will be used in drones. It might already be.
ICBM 😓
This will be most important for CNCLinux as it should help with stability. Thanks for sharing.
About 15 years ago, I was learning about CNC machining -- and I had even gone so far as to compile a real time kernel to check out an app on Linux that needed it -- and although I never went much farther than that, it gave me the experience I needed to be *very* excited by this news when I heard it a couple of weeks ago!
I have since thought about a different goal: four or five years ago, I saw the suggestion that a computer should always be responsive to the human -- and how I'd get frustrated when I'd try something, and it would slow my computer to a crawl, to the point where I can't move my mouse pointer, and it would take a few minutes for ^C to kill a process -- oh, how I wished for responsive computing!
To this end, I'd like a real-time UI that gives the app in focus enough real-time access to be responsive too. I'm not in a position (I don't think) to create such a UI, but I'm glad that this opens the door to that!
The ability to do what you describe depends on what you want the app in focus to do in response. You can't guarantee that it will be able to do anything you want.
Maybe not games per se, but their physics engines or their collision detection might give this some use. Specially with faster action RPGs sword swinging games like the souls games where you literally only have a few frames to try your best at low latency detection of what enemies and the player are doing. It's so hit and miss that they have to resort to huge sausage capsule collider (and sphere colliders) for hitboxes to make it look good, and even then sometimes people rightly complain about dodgy hitboxes.
How come this is the first time I see them being referred to as sausages.
@@XDarkGreyX It's a damn corndog hitbox change my mind lol
Realtime means that you can run very dangerous software that needs realtime reaction to inputs and predictable latency, like medical equipment, factory robots, self driving cars. But hardware that needs this uses other, smaller operating systems (RTOS), so I doubt the bloated Linux kernel will be used for that.
Been used for this stuff for ages.
Real time simply means guaranteed worst case latency. Many people believe it means fast. Real time usually means lower bandwidth in exchange for increased reliability to maintain the worst case latency.
Common example of where real time make sense is audio processing. Thus the birth of Jack audio server. Heck, I've even seen RT Linux used with CORBA for real time flight control avionics. As well as radar processing.
Real time doesn't have to mean embedded nor specifically embedded processors. It simply means one has a worst case latency requirement.
Pretty much agree with all of this, although in the embedded world, in many applications you don't even need an RTOS. Totally agree that Linux bloat in many embedded applications is a PITA, not least boot time, the energy required, and cost of hardware to support it.
I'll add one more factor... time deterministic behaviour means everything in low level high speed applications, such as motor control or power supply feedback loops.
@@justanothercomment416 if it has been used like this, do you know why there exists real time OS that are very expensive?
@@rezah336this is not my expertise however there are many safety regulations in industries like medical equipment or assembly line equipment etc. and thus certifications that you meet these safety requirements. So to make a device for these industries you must use a certified kernel that meets safety regulations and these certifications give those kernels high pricing.
@@rezah336 Sure. Liability. If your medical pump misses an interrupt which halts the pump, and creates an overdose, who pays for that? This is just one example.
QNX and VxWorks are very popular and their pricing reflects the reality of liabilities and support for their products. Not to mention, someone has to port to new hardware and new peripherals.
I've played with QNX. Thought it was pretty cool. Used VxWorks for P95 radio, including the entire init process. Something I did was used at NASA for rover testing. And I also do embedded development, in addition to some phone stuff on the application processor side.
The RT world heavily overlaps with the embedded world but it's not exclusively so. As nez rightly points out, many applications within the embedded world (most?) use no OS at all. Even still, for many non mission critical apps, there are many RTOS solutions available (example, FreeRTOS).
Cheap hardware (couple bucks) is available to play with much of this.
Latency is actually quite relevant in gaming, in particular the delay from last input to the next frame flip with updates based on that input. Then, though, availability of variable refresh without excessive buffering is sufficient: Latency needs to be low, yes, but timing doesn't need to be evenly spaced humans are *very* bad at distinguishing 75fps from 74fps so it's no biggie if you're a bit late. Audio is of course also involved but we're quite tolerant of audio latency because sound is naturally slow so as long as you can fill that ring buffer you're generally fine. Haptics probably needs RT, but then it also needs 1000fps so that's a whole different kind of beast anyway.
Very well explained for someone like me that is new to RT 👍 Would love to have a video dedicated the history, theory and implementation if RT in different architectures (like VAX for ATC, etc) Maybe too much to ask for in one video 😀
I remember merging that to our own build back in 2003 and then merging all the mainline changes back into that
This real-time thing sound real cool, really good example with the Tesla, the metaphor is very clear.
Kind of scary how a task can take over the whole system, maybe I don't understand that properly but the concept when said like that sound a bit scary.
Nice for labs, where you need to poll your sensors, synchronize the data. Especially when your polling speed is fast and you need to organize a feedback
RT and the OS cannot do miracles. It just promises lower latency. This basically just gives you *even higher priority* and ensures that the kernel itself will not itself run on that high priority and interfere with your stuff. But there is no way to prevent several tasks requesting RT priority and fighting for the CPU resources and exhausting the system.
This will be fantastic for embedded Linux. Makes it a much more viable alternative to RTOSes such as Zephyr and FreeRTOS
LinuxCNC has been using the RT patch that existed for a long time
The realtime patch has been available for anyone to put in the kernel when they want. And the people who need realtime already had the technical competence to do it.
pressing X to doubt
@@bullpup1337 What's this, the middle ages? You don't have to travel to a secret library to confirm what I said is true. Fucking idiocracy.
Not really, for the past at least 5 years all major distros have had a Preempt-RT kernel right in their package manager. I've been using Preempt-RT for like 10 years but I haven't patched one myself in at least 5.
Lions have gotten 4ms audio latency forever but it's always been getting incrementally more reliable. It'll never be able to bit bang a servo, bit bang LED brightness, or run clockcycle accurate emulators from user space.
honestly, for a desktop computer, I usually care more about maximum latency than throughput. i of course don't care about microsecond or even millisecond latencies, but if getting a more responsive system and fewer skipped frames means that background tasks take twice as much time, so be it.
the main case where performance is important is when the soft-realtime tasks are often close to their limit, so that decreasing their throughput for the sake of determinism would mean regularly missing deadlines, but I'd argue that that's a design flaw for that application. Video games are an excellent example of this, where missed/delayed frames sucks and you're often close to the limit, but you should aim to use settings where you have some margin on performance to avoid that.
however, if i am doing data processing of some kind (video processing, compilation etc.), then performance can actually be more important than latency, since those things can take many minutes that you don't want to wait unnecessarily long for, but even then, I might want to keep using the system while I'm waiting for the task, so I'm still ready to sacrifice some throughput for latency
It is not true that this means a reliable latency limit, because modern CPUs have a security kernel built in that can interrupt anything CPU is doing and cause unacceptable latency. I think this is not likely but it can happen.
This is why important realtime things are and will be done on specialty hardware. Which may use the Linux kernel. In that case it would be reliable.
You can run Linux on 'microcontrollers', my guess is those do the correct thing.
Yup. Industrial machines I work on have relays and long loops of wire for E-stops and door interlocks. Opening a door or smashing the big red button will open relays to make things safe. If it doesn't completely shut off the power a spare relay contact will tell the Real-Time server what happened when it can spare some cycles. Even programmable logic components aren't reliable enough, I guess. Several hundred feet of wire jumping to dozens of relays around the machine "just works."
Even for desktop gaming-/workstations, when it’s less than 1% performance penalty e.g. FPS in Counter-Strike, I would prefer having e.g. pipewire running in RT mode. Reliability is better than up to 1% performance penalty. I mean some processes can need it, right? Better making it possible that they - can - have it when needed, than not having the possibility, and that just enabled for those processes who need it, by default, that would probably be nice in my limited view. I have no idea, when I‘d install e.g. future Arch Linux´ kernel 6.12+, will that possibility be enabled there? No kernel parameters or whatever needed, just process priority change from whatever to rt? Will it be possible in Linux-zen which I use? As mentioned, I‘d want to use pipewire with rt, I‘d test FPS in CS. But as seen in the text, I am still not sure how it will or won’t work; I am in for the mentioned idea. More consistent latencies are nicer than little bit performance.
It would require a lot of testing, but imagine, as mentioned potentially less than 1% penalty, but less stuttering as well. Because that often comes because of inconsistency in e.g. frame times, maybe rt improves those a little bit, especially when not closing the billion chromium tabs in the background.
Game and pipewire having high priority and rt mode could help I guess in that situation, but needs testing, and I just speculate, I have never used anything RT. 🤷
Basically, RT just ensures all interrupts take a minum time. That's all. TO ensure that ALL kernel must be RT. There's no other option.
You can do what you are explaining on normal kernels, just give pipewire a higher priority with `nice`.
But most probably you will only achieve that the game will stutter instead of the audio. Which I guess is much worse.
guys, tell me the latency or jitter that you get and how you achieve it
i get 40 micro seconds sample jitter with 'sudo chrt -f -p 99 PID'
standard xubuntu 18.04
There's been (terribly outdated) RTOS Linux forks for aeons
cooool man , tnks for the update...
Linus himsef is our insurance that Linux never pulls a google and quietly removes the "don't be evil" philosophy statement and.... Becomes evil.
IMO games could in principle benefit from RT support, especially with high refresh rates, to actually ensure consistent, guaranteed time to next frame... but it seems like everyone gave up on consistent frame times in games. This should be especially useful for VR
nice, can i mark this on a running program from htop without needing to have the program know about it? it's basically letting one process have prioritized compute right?
This is great, I wonder if raspberry pi OS would update to RT kernel, the IO would benefit dramatically
This will be great for dedicated audio and networking devices.
I've ran the RT patch on systems before, but this will be nice not having to switch over to it every time.
I always love Linus Torvalds' Tech Tips.
I've tried the RT version before for music production, in hopes of getting the lowest possible latency when record monitoring, but it never seemed to make much difference.
Used correctly with software designed to make use of it (e.g. Jack or PipeWire with Reaper or Ardour), it can reduce your minimum buffer/frame size without underruns, but you still have to select that smaller frame to get the benefit.
@@microcolonel Yea, I know. It's just my latency is already reported at
This sounds like what we need to move gaming from windows to Linux aswell as some other things like audio and video etc.
That's a lesson in perseverance. If someone can be so stubborn they can make Linus Torvalds relent control of one of his babies, nothing is impossible if you put enough effort.
For me with all my studio gear realtime matters. Latency is a killer.
Hi, good news for audio processing.
Real time is important for simulating and controlling engines and transmissions in dynamometers.
Seems like it would actually be great for the lag spikes i've been experiencing lately on games
@BryanLunduke
You can eliminate audio stutter on your gaming machine. There you go.
I'm imagining a room full of industrial equpment designers perking up at this news like a bunch of cats when you announce they're getting fish for dinner.
Now if Universal Audio will just give me some Linux drivers for my Apollo Twin...
My take is that average people will see difference in OBS if their CPU is near 100% used, it will not have audio stutters.
it doesnt change the cycles per second, so its really just enabling the PREEMPT_RT system
This is gonna be great for freecad
if you have a gaming machine will this make the "boss key" more responsive
Low latency is not a given! You can even have more latency as the RTOS scheduler decides to give your process a lower precedence because other time critical processes are set to have a high time response.
The only thing RT guarantees is that specific task will start in a specific time frame. And if it will need to stop for example user input or data acquisition then it will do just that.
Does it allow for reserving processing and etc? Without being able to tell whether other processes are too heavy to actually respond within a deadline, I'm not sure this is actually "real time."
I think this means that the Linux community wants to expand into business verticals where Linux was largely absent until now, like automotive etc.
I believe QSC Q-SYS is built on a realtime variant of linux. It's pretty great but the prices are... well, they're great too, but a different kind of great :/
perhaps this will foment something great in a good way :D
I think if it will unfortunately help HFT companies
I still don't understand: Does this generally allow for lower latency, or just for more relability in exceptional cases (like other programs running in the background trying to do something)?
Do you think a real-time linux (and some changes in other areas) might improve a desktop linux's responsiveness to mouse and keyboard when working under stress?
Gaming is absolutely a relevant application. It will let you make a game that has no tearing or judders in animation, and consistent latency from user input to the results onscreen. Right now if you want that you have to brute force it with extremely high framerates.
That was too much waffle. The best way to explain what it means is to provide a real-life demo example. What you did was reword the same metaphor about priority multiple different ways.
Very cool.
Anyone got a link to the video of the code wrapped on gold being delivered to Linus?
This is amazing
Does this mean im going to have a sound driver that wont break at random?
I'm running PREEMPT kernel even on desktop system because it seems to reduce stutter, jitter and random stalls enough to be noticeable for me. However, it causes lower average throughput (for the same hardware) so it's not without side-effects either.
If your brain is happy to use e.g. Bluetooth audio which has built-in delay around 40-200 ms then you probably don't need RT or PREEMPT for anything. I'm unfortunate enough to have brain that can detect latency around 6 ms and I cannot stand to use bluetooth audio because of its latency, and I'm also very sensitive to jitter in the display. For me, it's better to sacrifice some average performance to get rid of those latency spikes that become too noticeable for my taste.
I'm looking forward to see if I can see the difference between RT and PREEMPT kernels. I'd hope not because otherwise it would mean I'm probably going to have to sacrifice even more throughput to run RT kernel at all times.
If the kernel is controlling something external such as CNC machinery, then RT kernel is obviously mandatory because it is not acceptable for kernel to randomly fail to stop the moving parts of the machines at the correct time.
The traditional solution without RT kernels has been more hardware: instead of kernel controlling the CNC machine directly, it controls an extra piece of hardware that does nothing but runs the machine constantly. If you don't have multitasking kernel nor use hardware that can randomly run SMI at will, you have 100% known latency at all times.
RT kernel is about allowing hardware to be used for true realtime processing at the same time as using extra CPU cycles for non-RT tasks such as user visible GUI interface.
so they're saying they'll be crowstrike BSOD'ing linux now?
Funnily enough, your self-driving example is actually quite close to reality: openpilot runs several processes on linux, some of them not being RT-critical (UI), others are. But I don't know if openpilot uses RT kernel etc.
For me realtime OS doesn't mean faster, just higher priority can starve out lower priorities, and that's what desktop OS trying to avoid.
Well no akshually.
A realtime kernel has nothing to with process priority levels and starving processes. You may be thinking of Windows' realtime priority class, which is its highest priority class, everything >= priority 16. The priority class is called realtime but Windows is not a realtime operating system.
A kernel, realtime or not, can perfectly well limit a high priority process so it doesn't starve lower priority processes, if it's made to.
@@xlerb1637 By definition, a higher priority task should starve lower priority tasks. That's why they are prioritized accordingly. If one shouldn't starve the other they should have the same priority.
Priorities basically says I want the higher priority to run even if a lower priority task wants to. That's why it's higher priority. It's more important it runs than the lower priority task.
@@justanothercomment416 Linux has built into it, I don't remember the details, the ability to limit CPU time to a high priority process so an out-of-control process doesn't kill the system.
Which are the distros that feature it by default (i.e., without the need to recompile the kernel)?
Fun things are fun, but this was misleading. printk() was dear not just to Linus, but to all developers dealing with kernel crashes. To ensure that last debug messages and dumps appear on serial console or screen, it was given shortcuts to deal with logging directly, with minimal involvement of the rest of the system code (which might already be garbage or point to nowhere in case of memory corruption). Special case included bypassing the scheduler (because any context switch could be the last on a broken system). Because that parallel prioritisation system was not playing by the rules, calls to printk() in kernel code could take control of the system for unpredictable amount of time. Not that important on a general purpose system (for a code that doesn't interact with printk() and its special handling), but breaking all guarantees if you need predictable low latency.
Interesting, I use arch and I think I use an RT kernel. What did they change about printk that allowed this to be mainline?
If only he had delivered the scroll on a pillow 😅
Anyway, no need to wait for a new machine. Just build an extra initramfs and pick it in GRUB and you're done 😁
Well, if you ever tried to implement PWM on Raspberry Pi (versus Arduino) and ask yourself why it's so sporadic, well, RT will make it possible not to be sporadic.
Before watching: Are we guitar pedal yet?
Wobbly-Windows gets top priority on my machine! 😉🐧❤
Most underrated comment on this video!
It's scary that the absurd scenario of the self-driving logic running on the same computer as the AC and the onboard entertainment system, is actually a plausible thing...
Thanks
So will this make Linux on par with VxWorks? And what is the difference between Windriver linux and the common kernel now then?
How would it effect the virtualization and kvm machines.
Liked and subscribed. Just because you know what you are talking about and you are not some automated crapface.
Wow so it’s true real time. Too often, real time has just been used to mean ‘fast response’, which is not what it means. The classical real-time system is the industrial PLC, which can give a guaranteed maximum response time to react to control inputs.
Hi Brian, is this the same as the Realtime process priority in Windows?
Does that mean the real time mode does not need the real time version of the kernel any more.
Your explanation seems to beg one important question: What IS the maximum guaranteed latency for the RT Linux kernel? I assume it's dependent on the processor's clock speed, but if the key takeaway is that there is a known maximum latency, how is that calculated?
A companion page to a 2020 paper stated:
"The blocking (or interference-free latency in the paper) is defined in Lemma 7 as: LIF ≤ max(DST, DPOID) + DPAIE + DPSD"
@@XGD5layer Since you can't post links, can you give me the exact name of the paper? I have no idea what any of those acronyms are (or what Lemma 7 is for that matter), but I assume they are defined in the paper somewhere.
@@XGD5layer NVM, a quick Google search of that quote revealed the paper: "Demystifying the Real-Time Linux Scheduling Latency
"
Is it like high and low CAN bus ?
If someone can find that video of Linux Plumbers Conference Where the developers handle him the gold code changes? It would be funny to watch, could not find by searching it