Linux Crash Course - Understanding Memory and Swap Usage
HTML-код
- Опубликовано: 14 июн 2024
- The "Linux Crash Course" series (formerly known as the "Linux Essentials" series) tackles important concepts around Linux, one video at a time. In this episode, we'll take a look at understanding memory and swap usage.
This video was sponsored by Linode:
- Set up your own cloud server with Akamai Connected Cloud ➜ learnlinux.link/akamai
Support Learn Linux TV
Note: Commission may be earned for any and all links presented here.
- Support me on Patreon and get early access to new content! ➜ learnlinux.link/patron
- Check out jay's latest book, Mastering Ubuntu Server 3rd Edition ➜ ubuntuserverbook.com
- Affiliate store for Linux compatible hardware/accessories ➜ learnlinux.link/amazon
- Check out the Tiny Pilot KVM for your Homelab ➜ learnlinux.link/tinypilot
Time Codes
00:00 - Intro
03:40 - The free command
12:11 - Understanding swap
14:49 - Should you disable swap and not use it?
16:57 - Adjusting how often swap is used (Swappiness)
Full Courses from Learn Linux TV
• Linux Essentials Certification Workshop ➜ learnlinux.link/lpi-course
• Linux Crash Course series ➜ linux.video/cc
• Learn how to use tmux ➜ linux.video/tmux
• Learn how to use vim ➜ linux.video/vim
• Bash Scripting Series ➜ linux.video/bash
• Proxmox VE Cluster Full Course ➜ linux.video/pve
• Learn Ansible ➜ linux.video/ansible
Linux-related Podcasts
• Enterprise Linux Security ➜ enterpriselinuxsecurity.show
• The Homelab Show ➜ thehomelab.show
Learn Linux TV on the Web
• Main site ➜ www.learnlinux.tv
• Community ➜ community.learnlinux.tv
• Enterprise Linux Security Podcast ➜ enterpriselinuxsecurity.show
• The Homelab Show Podcast ➜ thehomelab.show
• Content Ethics ➜ www.learnlinux.tv/content-ethics
• Request Assistance ➜ www.learnlinux.tv/request-ass...
Disclaimer
Learn Linux TV provides technical content that will hopefully be helpful to you and teach you something new. However, this content is provided without any warranty (expressed or implied). Learn Linux TV is not responsible for any damages that may arise from any use of this content. The person viewing Learn Linux TV's content is expected to follow their best judgement and to make their best decisions while working with any related technology. Always make sure you have written permission before working with any infrastructure. Also, be sure that you're compliant with all company rules, change control procedures, and local laws.
#Linux #LearnLinux #LinuxTutorial 
Наука
I LOVE the way you and DJ Ware cover these kinds of topics- a great blend of more advanced topics that are useful for home desktop users as well as server and enterprise admins. I'm a compulsive tinkerer and videos like these help me learn ways to satisfy my tinkering/ optimization addiction.
Can't thank you enough for explaining the memory swap subject! Over 3 years using Linux as my daily driver I still found myself closing browsers, shutting down my virtual machines, even rebooting to address it (even with 32gb of ram). I now have the confidence to tackle this and really make use of my rig!
Hi Jay! Your videos are a godsend. As a career changer and a sort of a late-starter in IT, your content is pure gold to me. This video helped me fit some pieces better in place and I feel more confident with this topic now. Thank you so much!
it's never late to learn! keep on learning.
i've always wanted to learn how this works. thanks so much for uploading!
This is the best vid I've seen on the science of memory and use of swap!
I like this format where you go through tools and features individually with their own videos. It works better for me than longer more wide-ranging videos.
Thanks for the helpful content!
Thanks. Nice clear explanation. Simple enough to understand.
Was looking for this. Thanks 👍🏼
hey Jay, nice episode as ever - you've amassed a great body of work already. What are the plaques with Japanese (I'm assuming) characters behind you?
Once again you have helped me to better understand a Linux topic, esp swap. Thank you so much!
Keep doing what you're doing. Much appreciated
Very helpful video. Thank You
Great intro to the topic, but would love a deep dive into memory.
Thanks as always for the great content. Much appreciated.
My pleasure!
All about data handling. The whole IT field rest on that. Thanks Jay for readjusting this in my brain. Your content is a brain saver.
Any content you put out for System 76 is valuable to me, I own one & it's my first Linux machine.
I've always included a swap file on distributions I've installed. Now, there is a different type called a page file. In the Windows world, the difference is a page file is dynamicly allocated as needed up to the limit you set, whereas a swap file is a static size. Page files are more disk management efficient because it only takes up as much space as it needs. But, contiguous space is not assured, which can really hurt performance.
My first experience with paging and swapping was in 1985 on a VAX 785, where idle procecess would be swapped out to allow higher priority and active interactive processes access to memory (all 2 megabytes of it).
We were running VMS 3.4, and the system administrator made the swapper a lower priority than he should have. The VAX (which serviced the 28,000 undergrad and graduate student population - even if you weren't a student that had a reason for an account, you had an account) was freezing. After about 3 days of troubleshooting, it was found that the swapper was swapping itself out to the page file. When something is paged out, it is put in a sleep state until the swapper swaps it back into memory. Well, if the swapper is put to sleep, how is the swapper going to swap itself back in? The system admin pit the swapper back at the hiest priority so it would never be swapped out.
Thank you for reading my war stories of my leading a time sharing system.
I always benefit from LLTV. My knowledge has expanded ❤️
Great work 🥳🥳🥳 Thanks 💜💜💜
Thanks my dude
Linux Crash Course helped me in improving Linux knowledge! Great thanks for your contribution and outstanding tutorials!
On the other hand, I wanted to ask your opinion on WSL2. Did you have a chance to interact with it? Any thoughts from professional community on it's usage in day-to-day tasks?
I use swappiness on my machines to limit the swapping to my SSD. I'd love to hear your take on swappiness, and recommendations.
Excellent content, thanks. In regard to allocating storage for swap-space, maybe you can elaborate on this: When a laptop is put to sleep/suspended I thought that the OS writes the contents of RAM to swap-space and back the other way when the laptop is awakened. So, in that use-case wouldn't a sufficient amount of swap be necessary? Sleep/suspend isn't relevant to servers/workstations but maybe it's worth mentioning.
@Reptilicus Rex yes that's true. if you wish to put your machine to sleep or suspend, you need swap enabled, generally 2gb bigger than your ram.
I come back to this video every time I need a swap refresher. One thing I will say is that by the end of the video, I hate the word "swap".
Thank you
Tnx , this is very helpfull and informative for people migrating to linux like me :)
1y old, but still be a great video.
I have an old original Raspberry Pi with 256 MB ram and 64 MB allocated to GPU. It runs headless and has Apache server installed along with ssh, dlna server, network scanner server and a vnc server. A desktop is installed. I very rarely need the desktop so both it and vnc server are disabled and only started if required. I did enable 512MB of swap as I think this may be needed when I do run the desktop and vnc server.
I do have more modern Pi devices but old faithful has server me well and continues to do so.
In the scenario of hosts behind a load balancer, I would rather avoid swap slowness all together, and quickly remove that host as it stops taking requests reaching memory limitations, that I would see request latency increase, especially if your application is set to backoff/retry behind the scenes. This is the best argument against it imo, its not just about disk space being cheap.
That sounds like a good strategy, thanks for sharing
Now days I use zram swap that uses ram block instead of drive and decrease swapiness to 10 to desktop and 30 to a server.
What's your opinion on compcache / zram like it is used by default on fedora instead of a swap partition/file.
How do you calculate the total memory from the available memory, free and used, etc? It used to be you could do that in the past, but in this example there seems to be megabytes of missing ram.
Wow thank so much.
I just set up a Raspberry pi 4 8GB RAM, booting off a 1TB SSD for Ubuntu Server 22.04
I was worried about having enough RAM
Excellent explanation of the /proc/meminfo Cached parameter. My new slogan is "Unused RAM is wasted RAM". Now, where's the T-Shirt?
This feeling when you understand how it works😊
Thanks for this Video and what is the shell used to get this kind of arrow prompt ?
It's just bash with a custom config, nothing special.
Thanks for the video. Unfortunutely, my server uses swap because of the analysis dependencies. I have more than enough space, ram and storage but the analysis sometimes oversteps to swap.
Then you have enough memory demand to write dirty anonymous pages to the swap file, there's no harm in having idle unused daemons end up with saved state in swap rather than retain memory wasting it.
It's when you have a high rate, due to insufficient memory causing thrashing and have the OOM killer terminate processes, possibly due to lack of swap allocation that you have a real issue.
This swap is bad mantra is over simplistic, it's there to use memory better.
Can you make a video on audit daemon since its running in your htop?
good video
The issue with swap memory is also about how much to spare. Now is 1 GB good enough or should it 50% of the RAM ? like I have a 16 GM RAM and partitioned 4 GB for swap in my SSD
It all depends on whether or not you plan to use hibernate, or "suspend then hibernate". If you do, swap should be slightly larger than how much RAM you have. If you're not using hibernate (and never plan to) then I usually just standardize on 2GB (4GB at the most).
Please make a detailed overview of the pros and cons of your system76 laptop and not a big comparison with the MacBook Air m1 13 in terms of work?
This guy is awesome 😎
I know! He's just so amazing. How does he do it? He's just soooo cool 😃
Hii please reply. I need help. I kinda set my swapsize to 2092 and now i can't remotely desktop connect to my Jetson Nano with Linux system. And I checked in it's SSH that the "system has been minimized". Is it because i set up it high? Did i broke my device???
A m.2 at a good clock only moves 5000 mb/s at best and ram does 35500 mb/s. A sata ssd is around 500 mb/s. It's better to use the ram for video than hard drive. That's my theory is try not to use swap at all. But you need swap if you freeze your computer. The swap will slowly untangle it. So that is why the older thinking of twice the ram.
Is there a way, driver or something that would allow one to use storage as system memory? Say use a flash drive as part of system memory and a hard drive for swap.
If I have 2 GB of ram and need 32+gb, setting a 64gb swap isn't going to help.
How about a distributed shared ram?
When I installed a 2TB SSD 16G mem in an old Toshiba laptop running Ubuntu I got 15sec boot time I set up 16G swap file, everything starts in seconds I can run multiple applications and I never ever need to use Htop or other system monitors.
Nice video,, but the swap space is used even I have free and available memory,, what would be the cause?
Even though it may cause slowing down the server, any swap is better than no swap, especially running a database aplication. 30+ years of experience suggests that. Cheers,.
Agreed!
I use the -h on things like free or df, h is for human readable...
Same for me.
So now I will learn whether reducing swap to 10 makes Linux faster or not.
It is a compromise: It will make your system more "snappy", applications that didn't run will stay in RAM for longer and therefore be available more quickly.
On the other hand, data of running applications will not be cached as well, so accessing data from disc will take longer and therefore your running applications will be slower.
If you want to know more learn to use vmstat and how to interpret the values. This gives you the tool to fine-tune your applications.
Someone please tell if it's possible to install GCC 12.1 stable version on Ubuntu 22.04 or not. I've been struggling to update mine from 12.0.1 which is experimental. I just can't find a way to install 12.1 which is the latest version.
I’ve got 15.5 GB RAM and the max thats being used is 3.8GB ( 2.1GB AVG). Swap is always used up to 3 GB ( 1.7 AVG). The performance of the applications I am hosting is not very good and I feel that is because it is not using enought memory even thou I’ve set 2.5GB for each ( total 7.5GB).
For example right now in bpytop it shows the ram as 19% used ( 2.94 GB) ,81% free (12.6GB) , 79% cached (12.3GB) , 1% Free ( 163 MB) and the swap 328 MB used.
So at this point I don’t understand swap since I am not running out of memory.
Can you do a Video on fio command
"free -h" is the best readable for noobies like me :)
Why not make the swap file very large ( for ex 20G) to have a lot of ram
test it out - assign 20GB of your disk for swap space, set the swappiness to 100 (use swap aggressively) and you will get the answer to your question very quickly :)
basically you would think you have a machine with a lot of memory, while in fact everything will be extremely slow
How about making a video about open source BIOS. If I want to install an open source BIOS in my computer replacing previous closed source BIOS what kind of challenges do I meet? Or, is that even possible?
It’s very possible to do, but depends on what model you have. Unfortunately, that particular topic wouldn’t translate well into a video because a small number of people would have the tools and appropriate hardware.
@@LearnLinuxTV Yup, that's true. Well, in near future, if it is possible explain these concepts with your amazing video.
I'll just give you this real example: you might have an app which is memory leaking, it will first consume all the ram, then it will make other processes memory to be pushed into swap. And then it's a matter of chance, what kind of error handler for memory allocation exceptions that buggy app will have. In vast majority of cases it will have something like "okay, we failed to malloc, we wait for a while and try again". Which in practical term means your system will become very unresponsive, because whatever memory will be freed, it will be immediately occupied by that single buster. And your chances to quickly ssh to that machine and kill the nasty app will be getting thinner and thinner, as now SSH daemon, and login/sh processess it spawns will be 100 times slower, as they'll be swapped to. So swap is bad not because of religious holywar. And to be 100% safe, have some resource manager set up on your system. And test how your system behaves, when some shitty process eats all the ram. It can be coded with just few lines of C code. And it will open your own eyes what a disaster it can be, if any of apps your server might be running will get some memory leak.
I would love to see also tutorials that touch this harder topics (why its show less mem than actually physical mem, what are shared and cache plus free values differ from available, etc.). This concerns also other your videos/topics. As an Linux admin/user for years I already know basics, please consider to make videos for more advanced Linux user.
Why doesn't "disks" set up the linux file system when you are partitioning a external HDD
"How do we monitor memory usage without having to install htop?" … use top instead. It shows you mem & swap stats as numbers sans the fancy graphs.
-m is show output in mebibytes; while --mega will show output in megabytes
I love you
I f I reboot the system . How I am always keep the swap memory on
How to like twice a video? 😉 great content
Thanks!
I am naughty, I haven't used swap for years and it goes fine. :) 32 GB DDR4,
Excellent video, easy to understand explanation of the essentials. However I only use desktops, I trust the defaults chosen by the kernel designers, so I never touch swapinness anymore :) I moved my applications to VMs and my Host OS is a minimal install of Ubuntu 22.04 LTS, my VMs run off OpenZFS 2.1.2.
My Host OS has 2 swap partitions; first priority a 2GB Swap on a nvme-SSD (3400/2300MB/s) and second priority a 16GB Swap on HDD as an overflow swap, but intended more for future hibernation. I often see swap usages between 100MB up to 1500MB and I don't care, since I don't notice any negative effect on the responsiveness. Remember many programs have large parts, that are never used or only once during e.g. initialization, closing the program or in case of errors or exceptions. I see it as "unused and wasted" RAM too :) Those parts of the program will be swapped first (least frequently used), so you will not see any effect with respect to responsiveness.
I limit the amount of RAM I assign to VMs, because I want to be able to run 3 to 4 VMs in the 16GB of my Ryzen 3 2200G. The VMs share the RAM with a 0.5GB to 3GB ZFS memory cache (L1ARC). So for Windows 11 Pro I use 4GB and for Linux VMs I use 1.5GB to 2.5GB. I have no complaints about responsiveness. Also there I see the SWAP is used extensively, but that swap storage is part of the virtual disk and that virtual disk is cached by the ZFS memory cache (L1ARC) of the Host OS. The L1ARC is lz4 compressed, so currently it stores 6.12GB of compressed stuff in 3GB physical space. Looking at the size of Linux after booting say 0.6GB to 1GB of uncompressed storage, the Host OS stores most of the used disk content of the 4 VMs in the L1ARC, including any used swap space.
Consequently my VMs run from memory :) :) and even on the 14nm Ryzen 3 2200G (4C4T, OC 3.7GHz) it is fast.
The boot time and initial program load times depend on the speed of the storage, so Xubuntu 22.04 LTS boots from nvme-SSD in 9 seconds and Xubuntu 20.04 LTS boots from two striped HDDs supported by a 90GB sata-SSD Cache (L2ARC) in 30 seconds. Booting Xubuntu 20.04 LTS for the second time takes 11 seconds, since now it largely boots from the L1ARC memory cache.
Note that e.g. ext4 installations also extensively uses caching, but by default the cache is uncompressed and the algorithm is LRU. ZFS uses a more sophisticated method of caching, mixing LRU and MFU, while dynamically adapting the boundary between the two based on the dynamic results of both methods. My VMs use ext4, so I have some caching there too, but the available memory is relatively small, so caching is relatively often deferred to the Host level and the superior L1ARC.
Could you make the explanation of free and available more simplified than it is on the video
For me free shows 32087 MB total RAM, just over 32 GB of what I have installed.
Agreed, I have said myself "why argue about putting aside a 'tiny scratch' on a huge disk", I think some individuals just attach to ideas and 'get comfortable'. Swapping may be slow but it could be the best place to be when needed.
swap drinking game confirmed! 😂
Ig im the first XD, btw love your vids!
Yay! Thank you!
OK, I'm laughing my butt off. It looked to me like you used your mouse to quit htop, so I went to a terminal and opened it and clicked quit and it quit. I've used htop for I don't even know how many years and didn't know you could use a pointing device. I don't know that I'll do it again, but it sure made me laugh.
I use zram so my swap is in my ram
I wish every distro, whether it be for desktop or server, came with vim and htop. These 2 programs are my first downloads for any distro I install. Great content as always, keep up the great work!
i've set swap = 0 but my proxmox dont care abt it ((((
I really like your videos but this one is so complicated when you are trying to explain differences between free and available
This is probably the same amount of water in your book.
No swap is bad. You want the kernel to kill process that eats memory, not swapping.
Hey friend
I'm Jiejie from KAMRUI. We sell all kinds of MINI PC on Amazon. We are looking forward to presenting it to your audience through your professional presentation. Can I have a chance to work with you?
You should upload your videos to odysee.
Too streched out to cover very little information.
With 20 min, you could've went a lot deeper. The video has way too many filler sentences
You waste a lot of time saying the obvious and when it comes down to understanding the differences you just say "I'm going to stop my explanation right here". What's the point of making this video? Engagement only I guess
learn to focus your thoughts, stop jumping around, one thought at a time. unwatchable
No program is made to use swap. You are wrong. And you are wrong about swapiness