Simple Trick to protect your Raspberry SD card from Crashing

Поделиться
HTML-код
  • Опубликовано: 7 сен 2024
  • This is a replay of the video 403 (during my summer break)
    SD cards in Raspberry Pi computers wear out and die after a while because they are not made for computer usage, and many write cycles. In addition, the SD cards can become corrupted if the Pi loses power without a proper shutdown. We have several possibilities to avoid this disaster. Only one is simple and cheap. But, unfortunately, it has its caveats. So, let’s get an overview and use a nearly unknown Raspbian feature to help us solve the problem.
    My second channel: / hb9blawireless
    Links:
    Raspberry Pi 4: amzn.to/4ckpLUF
    Raspberry Pi 5: amzn.to/4cD8rKp
    Barefoot method 1: / make-your-raspberry-pi...
    Barefoot Method 2: www.raspberry-...
    Adafruit's writeup: learn.adafruit...
    Patreon supporter companies:
    www.welectron....
    The links above are usually affiliate links that support the channel (at no additional cost to you).
    Supporting Material and Blog Page: www.sensorsiot.org
    GitHub: www.github.com...
    My Patreon Page: / andreasspiess
    Discord: / discord
    If you want to support the channel, please use the links below to start your shopping. No additional charges for you, but I get a commission on your purchases to buy new stuff for the channel
    My Amazon.com shop: www.amazon.com...
    For Banggood bit.ly/2jAQEf4
    For AliExpress: For AliExpress: bit.ly/3MtXUY8 (just go on from here to your product)
    For Amazon US: www.amazon.com...
    For Amazon.de: amzn.to/2r0ZCYI
    For Amazon UK: amzn.to/2mxBaJf
    For ebay.com: ebay.to/2DuYXBp
    / spiessa
    Please do not try to email me. This communication channel is reserved for my primary job
    As an Amazon Associate, I earn from qualifying purchases
    #no#midroll#ads

Комментарии • 307

  • @joels7605
    @joels7605 Месяц назад +70

    In /etc/fstab:
    add commit=300 to the mounts options for the SD card.
    Linux will normally write out changes to the filesystem every 5 seconds. That's a lot of writes. commit=300 caches writes for 5 minutes (when possible) and does one write. Very easy change, greatly increases SD card life. ...and the whole operating system works like normal.

    • @pjohnson21211
      @pjohnson21211 Месяц назад +4

      I like this idea

    • @joels7605
      @joels7605 Месяц назад

      @@pjohnson21211 It works really well. And it speed up your system (at the cost of potentially losing 5 minutes of data in the event of a power loss). I have about 20 SBCs doing things at my house, and SD cards would die once in a while. I started using the commit trick about 5 years ago and haven't lost a single one since.

    • @mattpujol4787
      @mattpujol4787 Месяц назад +9

      Yeah. Flash media likes large block writes. This in essence is write coalescing, which is what the SD driver in Raspian doesn't do. Changing the cache flush timing forces write coalescing. That's what I thought Andreas was up to.

    • @SomeGuyInSandy
      @SomeGuyInSandy Месяц назад +3

      Great suggestion, thanks.

    • @satoshimanabe2493
      @satoshimanabe2493 Месяц назад +2

      Wow, I haven't heard of this trick. Very nice!

  • @onecircuit-as
    @onecircuit-as Месяц назад +23

    This is why I lean towards Orange Pi - the EMMC option means that I can run the whole show without the need for an SD card (after installation). Plus - cheap! Another advantage - I don't have to partition any RAM to avoid log files etc. Currently I have an Orange Pi 3 LTS doing "Pi-hole" and NAS duties. The OPi 3B has removable EMMC plus it also has a port for an NVME solid state drive.

    • @hesspet
      @hesspet Месяц назад

      EMMC or SSD work flawless. For newer installations of MagicMirror (It's a friend and familly thing) I also switched to the Orange Pi. No SD Card chaos anymore.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад +1

      A good idea!

    • @kevinmalec4977
      @kevinmalec4977 Месяц назад +1

      EMMC has the exact same problem unless you use expensive parts with one or two levels. and you can do the same by getting an slc sd card.

    • @onecircuit-as
      @onecircuit-as Месяц назад

      @@kevinmalec4977 there’s no utopia out there for data (well, maybe this voyager.jpl.nasa.gov/golden-record/), but I’ve had no emmc failures so far 🤷🏻

  • @ericks1979
    @ericks1979 Месяц назад +4

    Awesome! I did a bunch of writing and was worried about this. I started with just using a separate sacrificial SD card, then using a RAM disk, but never changed all the OS writing to it.
    This makes it so easy.

  • @MaxWithTheSax
    @MaxWithTheSax Месяц назад +15

    Another benefit of overlay fs is that you don't need to shut down the raspberry pi, you can simply unplug it without loosing any data.

    • @docwhogr
      @docwhogr Месяц назад +6

      actually you are loosing all the data

    • @MaxWithTheSax
      @MaxWithTheSax Месяц назад +5

      @@docwhogr Well yes, but non of the data on the SD card :)

    • @hesspet
      @hesspet Месяц назад

      But this hard shutdown can also kill the SD Card. The small spikes from the power supply are enough to kill the card.

    • @davidgari3240
      @davidgari3240 Месяц назад +2

      *losing
      We should just exchange the definitions of 'lose' and 'loose' in the dictionary.

  • @rGunti
    @rGunti Месяц назад +17

    I built a (now wallmounted) LED clock display, which cycles through various different displays (like weather, sunrise/sunset, etc.). After I noticed that things broke due to an SD failure, I went to search for a solution and found one which was like No. 2. (That was before OverlayFS was a thing I think). It boots the OS and then switched the root to Read Only. With a simple command, I can remount it as writable to run updates. Either by running a command or rebooting, I can switch it back to read only mode. This thing is running for what I believe are 3-4 years on the same SD card at this point and survived multiple power outages without issue.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад +2

      Thank you for sharing your experiences!

  • @koskos758
    @koskos758 Месяц назад +1

    Nice video. If Rpi is connected to your network, you can export a NFS share from your server and mount it in Rpi. With that you can have a data folder or /home or even /var mounted on the server and all you RPI data will be on the server without additional transfers.

  • @pmsrodrigues
    @pmsrodrigues Месяц назад +24

    log2ram is also an option if you use the Raspberry Pi as a headless server, as most frequent writes will be logs and go in /var/log .

    • @hoggif
      @hoggif Месяц назад +3

      I like using that too. In my use logs are the most source of writes (and software updates but they are not constant)

  • @SightsToKeepInSight
    @SightsToKeepInSight Месяц назад +11

    With a bit more knowledge on Linux, you can also mount your logs folder in a remote filesystem (like a NAS or similar) and then even your logs are permanent after a reboot.

  • @bartjsmit
    @bartjsmit Месяц назад +5

    Hi Andreas, I solved the issue with a half-way house between external SSD and SD card - the RaspiKey, which is a EMMC module with an SD card interface. I used it for PiKVM, which uses Arch Linux and admittedly is read-only out of the box. To use its ISO storage though would involve writes and I have just been scarred too many times by SD failures. You can make your own with an adapter and modules from your favourite middle kingdom electronics bazaar, but I did not have time for that. BTW PiKVM is an awesome project, well worth a look and perhaps a video ;-)

  • @c128stuff
    @c128stuff Месяц назад +6

    Do all your logging to some 'centralized' syslog server instead of to your local filesystem, this solves more than just avoiding writes to sd cards. I do this for all the RPIs I use, which all run from a read-only filesystem with just a ram overlay for tmp files.
    The only 'issue' with that is having to make the filesystem writable temporarily when wanting to save persistent config changes.

    • @hesspet
      @hesspet Месяц назад +1

      That works if you have an infrastructure with network and server around you. But for isolated systems that are operated alone in the middle of nowhere and usually have somewhat, let's say, unstable operating conditions, an SDD or EMMC solution is generally the less stressful option.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад +1

      A good possibility for many applications.

  • @asiw
    @asiw Месяц назад +1

    So useful. I wondered what the overlay file system was for. Now I know I can use this method with confidence. Thank you.

  • @jeffpkamp
    @jeffpkamp Месяц назад

    Good to see I was finally ahead of my time on something. I did the ram disk on my raspberry pi 3 after cooking two SD cards in a matter of days logging things.
    SSD is much better but it was a pain in the butt to get it to boot from with the 3.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      Indeed, the Pi 3 was made before SSDs became popular.

  • @Hfil66
    @Hfil66 25 дней назад

    I have had a Raspberry Pi with an SD card running flawlessly for 3 1/2 years.
    First point is that some SD cards have built in wear levelling, although it is difficult to say which ones since they often do not specify whether they have build in wear levelling or not.
    Secondly, I bought a card that was significantly larger than I needed so there was plenty of spare capacity to write to if it was needed without having to overwrite what was already there.
    Thirdly, I have (in response to this video) decided to move my /var directory (the directory where most writes happen on production machines) onto a separate partition, and format that partition with BTRFS, since BTRFS is a copy-on-write filesystem one would hope it has less probability of rewriting to the same part of the disk over and over again.
    As for the log file itself, the logs are configured to forward their messages over the network to another machine, so even if for some reason the log file on the Raspberry got corrupted it would hopefully have an almost up-to-date copy on another machine.

  • @PhG1961
    @PhG1961 Месяц назад

    Indeed a nice recap and wonderful summer Sundays entertainment. These days there's a lot of work that needs to be done in the garden/back yard. Can't wait for the fall... long evenings in the workshop!

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      I would enjoy summer as long as it lsts. At least this is what I do (this is the reason for my late answer ;-)

  • @kylek29
    @kylek29 Месяц назад +1

    One additional thing you can do is add a USB drive to be your RW script/file holder, that way you do have something writeable and it won't get hit with the card-killing small writes of log files. Good for if your software has saveable configs or periodic write needs where you need persistence.
    Another way is you can configure the Pi to run a specific script file after it does a RSync from a network share -- this RSync and script would let you do some reconfiguration and/or change things as needed without having to unlock the RPi SDCard since it'd be syncing your folder of files into the RAM FS overlay. Works similar in concept to enterprise config-on-boot setups.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      Thank you for adding other possibilities!

  • @JanicekTrnecka
    @JanicekTrnecka Месяц назад +3

    I once forgot that I have overlay fs enabled, and did lot of tweaking ...took me pretty long...did a reset and it took me several seconds to realise what I did :D

    • @matambale
      @matambale Месяц назад +2

      put a warning in your build script (that would catch a few instances of this, if you're building c/c++ code for instance)

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      That is bad luck!

  • @zvpunry1971
    @zvpunry1971 Месяц назад +12

    This is actually implemented using the debian package overlayroot. After it is installed, it adds a few scripts to the initramfs and is controlled using a boot parameter.

    • @zvpunry1971
      @zvpunry1971 Месяц назад +5

      There is also the possibility to see which files have been changed, just look at /media/root-rw/overlay. There is even the possibility to make /media/root-ro temporarily writable and copy some data to it, so it survives a planned reboot.

    • @zvpunry1971
      @zvpunry1971 Месяц назад +2

      All this is based on overlayfs, it has a lowerdir= and an upperdir= mount option (see the output of the mount command). When you access a file, it first looks at the upperdir and if it finds the file there, it uses this version. If it doesn't find it there, it looks at the lowerdir and so on. So if something exists in the upperdir, the file from the lowerdir is ignored.

    • @pofjiosgjsoges
      @pofjiosgjsoges Месяц назад

      I was running rPi 3B with overlayroot and no writes at all. SanDisk SD card died after 5-6 years. Only writes every 6-12 months were updates.
      Probably due to rare low voltage. I was using 5V 1A charger to run it.

  • @JohnBurgessMusic
    @JohnBurgessMusic Месяц назад +3

    On my last project, I used an RPi5 with an NVME drive attached to the PCIe flat flex slot. Even running at the standard gen 2 speed, the OS is so fast and responsive that I wouldn't go back even to a USB drive. I had a few issues using the USB method, I think there was some power delivery issues which caused the drive to disconnect occasionally, which obviously causes everything in the OS to crash. The NVME way works without any special setup needed, I then also set swappiness to 0 persistent on reboots.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      Indeed, the Pi5 with SSD is a different thing. This video is for small projects using SD cards.

  • @arva1kes
    @arva1kes Месяц назад +4

    My two cents on todays sdcards- we have been using samsung sd cards for around 3 years now. We have around 30 units working and they log around 100-200mb of data daily. They work to this day still. Kingston sdcards died after 3 months on same workload ( 2 sites). Samsung sdcards and usb sticks get slow if their lifespan is depleted, so they kinda protect you and they let you know that you should replace them. So far I have not seend single one sdcard failure with the samsung ones.

  • @Bus2000
    @Bus2000 Месяц назад +2

    Thanks was looking for this a long time. Had a lot of SD card crashes.

  • @JeffFlowersgoogle
    @JeffFlowersgoogle Месяц назад

    Whoa - thank you for this. This is a great tool, and will help a lot. I had no idea this was an option until your video.

  • @fixsomebits
    @fixsomebits Месяц назад +2

    Thanks Andreas, good summary!

  • @randomguy3784
    @randomguy3784 Месяц назад +2

    Great tip!
    Thank you Andreas! ♥

  • @magoostus
    @magoostus Месяц назад +1

    there are industrial grade SLC and pSLC microSD cards available that have many more write cycles. MTBF over 2 million hours, etc. also using a filesystem like BTRFS and setting ZSTD compression can help reduce the amount of data written with transparent filesystem compression.

    • @TheFrewah
      @TheFrewah Месяц назад +1

      How interesting! Please tell me the brand and models.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      Thank you for the info!

  • @greg4367
    @greg4367 Месяц назад

    Always educational, this week very appreciated. Thank you.

  • @avejst
    @avejst Месяц назад

    Great walk-through video of the process ❤

  • @RichardT2112
    @RichardT2112 Месяц назад

    That is just slick! Thanks for posting it!

  • @stefanzapf
    @stefanzapf Месяц назад

    I use WD purple SD-cards. They are made for video purpose and can be used for intense writing. I have 5 Raspi‘s with a lot of write cycles. They are working since 3 years without any issues. For important systems I change the cards all 2 years.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      Thank you for sharing your experience!

  • @danielrodding8522
    @danielrodding8522 Месяц назад

    If you really want to permanently store data on your SD card, but are interested in reducing unnecessary write cycles, use mount option "noatime" in /etc/fstab. That's especially useful for systems where tons of shell scripts and other tools are to be used.

  • @piotrk4608
    @piotrk4608 Месяц назад

    Good video and very informative. There is one more thing, it is named Alpine Linux which have a slightly different approach, the whole filesystem is stored on archives on a simple FAT partition. They get decompressed at boot time. Your changes reside also in a file which You can very easy update. I use it often for small tasks and homelab. It works brilliant, is very clean and lightweight. Maybe it helps someone. Thx.

  • @PATRIK67KALLBACK
    @PATRIK67KALLBACK Месяц назад +1

    Thank you for the valuable information!

  • @eskibrew
    @eskibrew Месяц назад

    Excellent, that's really useful, thanks for highlighting.

  • @matambale
    @matambale Месяц назад

    Excellent, Andreas - as always interesting and very useful.

  • @johnsonlam
    @johnsonlam Месяц назад +2

    Thank you for your tips!

  • @gigaherz_
    @gigaherz_ Месяц назад

    I 3d printed my own case, and used panel mount extension cables to bring the extra usb3 port and ethernet port to the back, and a usb2 hub in th front for connecting wired controllers or kb+mouse. I have a usb sata adapter inside, with an old SSD I wasn't using. now I have a small tv-connected game console for running emulators.

  • @eduardogodoy7621
    @eduardogodoy7621 Месяц назад +1

    There are some "extreme" and "industrial" SD cards for cases when you need frequent writing. Maybe a video about and testing then would be a good complimentary for this video. Tks

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      Thanks for mentioning them. I do not know how good they are. I hardly use SD cards for critical applications anymore (I use SSDs). And for small projects I use this method.

    • @hesspet
      @hesspet Месяц назад

      We tried those cards. They live a little bit longer, but the PI breaks those card also when you need extenive writing to the card. Those cards are really expensive. And they also die on hard power failures and such things like the cheaper standard cards. See my text above.
      BTW: A small SSD is cheaper than those cards.

  • @sigmata0
    @sigmata0 Месяц назад +2

    I'm guessing you could also redirect logging using something like rsyslog to a central location if the logs are important enough to survive a reboot.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад +1

      For sure. I would call this the "barefoot" approach.

  • @jmr
    @jmr Месяц назад +1

    Except for PI Zero I almost always use SSD. I found you can physically cut a lot of 2.5" SSD in half to make them smaller.

    • @hesspet
      @hesspet Месяц назад +1

      Or use M2 SDDs with an adapter :-)

    • @jmr
      @jmr Месяц назад

      ​@@hesspet Anyone can buy a smaller SSD but it's way more fun to take a hacksaw and cut the whole thing in half. If I completely deconstruct the cheap SSD and adapter it's even smaller then an M2 plus adapter off the shelf. I've been thinking about designing a 3D printable case for all the guts to make it super compact.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад +1

      Interesting concept😉

  • @tonisee2
    @tonisee2 Месяц назад

    Very good overview, thank you a alot!
    Two weeks ago I had to replace SD-card of a Weewx weather station RPi, which was working for last 6 years. I was really surprised it did that long with once in a minute logging etc. I'll try out if overlay FS can help me to forget it till I retire! 🙂

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      A good concept! So you can focus on more important things😉

  • @Megavoltamper
    @Megavoltamper Месяц назад

    I use endurance SD cards with my RPis, one of my Rpi based allsky camera is operating 24/365 since 5 years with a Kingston High Endurance SD card and that's a lot of writing.
    I also added some of your scripts to safely turn the Rpi off with a push button and things like that.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      Thank you for sharing your experience!

  • @peppigue
    @peppigue Месяц назад

    looking into configurations for kiosk pcs at the moment, so interesting that you covered overlayfs. windows has a similar thing, unified write filter, requires enterprise or education edition i think.

    • @hesspet
      @hesspet Месяц назад +1

      We build a lot of systems similar to kiosk systems (sort of interactive display for a industrial solution) based on a raspi style system. Don't use SD Cards! Wie tried every thing. (Sie my text above). More than 100 Systems. And 20-30 died per year with SD Card problems. We changed to SSDs. All problems are gone. Using a less or more out fo the box Linux with only small tweaks.

  • @georgezimmer5622
    @georgezimmer5622 Месяц назад +10

    I had issue with my Raspberry and SD Cards for months but I finally found an easy solution. First make sure your Rapsberry isn't powered on, if it is turn it off and gently disconnect the power cord. Then remove everything attached to it, grab it and toss it into a trashcan. Then I went on Amazon and bought an Arduino and a NUC for each use cases. Since then I've had zero issue with SD Cards.

    • @LanceThumping
      @LanceThumping Месяц назад +2

      I've become more and more convinced that it's better for most people to split their work into MCU and Mini-PC instead of using an Raspberry Pi.
      If you use a mini-PC you can get x86-64 processors and all the extra performance and compatibility that comes with it. Plus the iGPU will be like 100x better.
      If the minis that can do hardware decoding continue getting cheaper then they'll completely kill the RPi in my heart.

    • @rabindrasharma
      @rabindrasharma Месяц назад

      man you roasted whole 8 mins of my time I watched this video !!

    • @hesspet
      @hesspet Месяц назад

      Same here: I use some of those 200€ Mini PCs. Hard case, power supply, ready to run and "endless" SSD for HA running with DietPi. Never looked back to Raspis for newer Projects.

  • @gregebert5544
    @gregebert5544 Месяц назад

    I've had multiple RasPi's (mostly Zero W's) running 24/7 for years, and the only SD card problem was my fault when I attempted to re-image it using dd (instead of the PC-based imager).
    There isn't a lot of specific info about wear-leveling, but a few searches I've done suggests that present-day micro-SD cards have internal wear-leveling, so unless you are doing a lot of writes, the progression of info into /var/log shouldn't cause your SD card to "wrap-around" thousands of times and exhaust all of the blocks. I see logfiles being gzip'd weekly (to reduce disk usage). According to timestamps, files get touched when I reconnect via VNC, but otherwise seem to be static.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      I agree, wear leveling helps. But as you said, not a lot of info is available.

  • @neildotwilliams
    @neildotwilliams Месяц назад

    Hi Andreas, as always a great video! There is a combination that i use which is the overlay file system AND High or Max Endurance SD cards.

  • @polymorphism1966
    @polymorphism1966 Месяц назад +1

    Thank Andreas 😊

  • @NiHaoMike64
    @NiHaoMike64 Месяц назад +1

    Nowadays, there are "dashcam" SD cards that are specifically designed for a lot of writes.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      Let’s hope they are also made for the patterns of computers (lots of small random writes)

  • @maxmn5821
    @maxmn5821 Месяц назад

    Once I made a kiosk-like raspi with tiny core linux. They have a raspi distro. I turned on the card‘s write protection (with a software tool). I can pull the card out of the slot any time - it will be needed no sooner than the next boot.

  • @ugetridofit
    @ugetridofit Месяц назад +1

    The simplest solution is to avoid RPI all together. They have had more than enough time to be smart about it and stick emmc on all their products. There are much lower cost solutions with emmc out there.

  • @teebeeonline
    @teebeeonline Месяц назад

    thanks! I never know that before your video.

  • @jrioublanc
    @jrioublanc Месяц назад

    Good to know. Thanks!

  • @linux_gamer
    @linux_gamer Месяц назад

    I like the SSD drive at the end.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      Me too. But an overkill for small projects.

  • @John_Smith__
    @John_Smith__ Месяц назад +1

    This is all very good but there is not Persistence between reboots. Data will be gone if the systems powers down. And worse, the user programs can not write to the SSD. This can be a severe limitation on applications that Require data persistence between reboots.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад +1

      You are right. But keep in mind: This is a solution for SD cards. SSDs do not need it. They do not wear out on a Pi

  • @lolplayfelix-_-2713
    @lolplayfelix-_-2713 Месяц назад

    if it's possible, the recommended filesystem for flash storage sensitive to writing is f2fs, if possible as your rootfs, while this is complicated i could also recommend alpine linux, as it has built in support for an overlay filesystem, that you can then write using the lbu utility, but that would also need a bit of getting used to

  • @roysigurdkarlsbakk3842
    @roysigurdkarlsbakk3842 Месяц назад

    Another way is to install armbian on the pi, since it uses tmpfs for the usual temporary stuff, zram (compressed tmpfs) for /var/log and zswap (compressed swap) for the paging file.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      Thanks for the info. I stick with Raspberry OS because I still am a Linux noob.

    • @roysigurdkarlsbakk3842
      @roysigurdkarlsbakk3842 Месяц назад

      @@AndreasSpiess really, armbian isn't that hard ;)

  • @Mrmicpolock
    @Mrmicpolock Месяц назад +1

    100% agree, SD card are hot trash. I used many but the event that solidified that I'll never use one again was in a pi 3b with solar Assistant. a year + in and then the card corrupted. Lost all the data points from it. After that I upgraded to a pi 5 witha m.2 adapter. never looked back

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      Thank you for sharing your experience!

  • @isolance1
    @isolance1 Месяц назад

    Personally, I use industrial SD cards. When I bought them, they were only about 5-10% more expensive than regular ones, but the regular cards sometimes failed within weeks, while the ones marked with an 'I' have been working for over three years and not a single one has failed yet.

    • @TheFrewah
      @TheFrewah Месяц назад

      Interesting! Please tell me what they are called, I’m not sure I can make the distinction

  • @Alan_UK
    @Alan_UK Месяц назад

    Hi Andreas. Another option I am using is Log2Ram pimylifeup.com/raspberry-pi-log2ram/
    I'm running it on a number of Pi including Pi Zeros. Log2Ram is aimed at Raspbian Logs but via the configuration one can specify other directories as well. Log2Ram will write out the files to the SD card once a day (time configurable) and also on a normal shutdown (but not a crash or power failure). The amount of memory used is also configurable.
    I also use MQTT to send important data to a Pi 4 that has a SSD and also just after the end of the day FTP critical files to the same Pi 4. At the end of the month on the Pi Zeros etc I create new log files and delete the old ones so as to keep the file storage within available memory allocated to Log2Ram.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      Thank you for sharing your solution. Interresting!

  • @alx8439
    @alx8439 Месяц назад

    I did it hard way with my Debian on Orange Pi 4 LTS. Equipped myself with all kind of tools like iostat, fsstat, iotop to find the most aggressively writing processes. Surprisingly there were not so many of them and none of them was required to be running for me. Things like SAR and systat, various cronned unmaintained os updates and tracker-miners. But the most aggressive thing from RW perspective was ... Mozilla Firefox! But it can be fixed as well quite easily

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      It is always good to understand Linux! Thanks for the info.

  • @ronm6585
    @ronm6585 Месяц назад

    Thank you Andreas.

  • @SomeGuyInSandy
    @SomeGuyInSandy Месяц назад

    I have three pi based digital signs that download and then display a number of slides. They have been running for several years (so far) without doing this. When I have some time I'll try switching one of them and see how it goes. Thanks :)

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      So you were lucky. Or your raspberries did something to avoid logging.

  • @dr_jaymz
    @dr_jaymz Месяц назад

    I'm surprised you didnt mention you can redirect your syslog to a remote syslog server. This means that you'll have the logs after a reboot. You can use another pi or set up remote syslog server on something like a nas.

  • @IMBlakeley
    @IMBlakeley Месяц назад

    I like log2ram for this, maps the log folder to memory and writes it out on a schedule.

  • @NiklasAndersson7
    @NiklasAndersson7 Месяц назад +1

    Are the filesystems mounted with 'atime=no' per default in Raspberry OS? That is a huge one to avoid unnecessary writes.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      I do not know. Maybe somebody else can help?

    • @AS-oq7io
      @AS-oq7io Месяц назад

      The Linux kernel since 2.6.30 uses relatime by default, so noatime is no longer needed. relatime is almost as good as noatime but without the pitfalls.

  • @paulstubbs7678
    @paulstubbs7678 Месяц назад

    Definitly a loss on the micro SD cards, it would be nice if the 'pi had a link on the board for this. It would also alleviate the problem of the 'Pi being hacked into and turned into a bot.
    On the last one, (hacking) this current software solution is only partially effective as a hacker can always set the switches that allow writes

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      Keep in mind: You still can write, just not to the SD card. So you still could create a bot. But only till a reboot.

  • @xDR1TeK
    @xDR1TeK Месяц назад

    In puppy Linux they had a better solution. You can save the changes to a file or you can choose to load the previous changes next time you boot. At least if you had changes you don't want to lose, you can still choose to save them rather than fall victim to choosing an overlay file structure.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад +1

      That seems to be the best of both worlds!

    • @xDR1TeK
      @xDR1TeK Месяц назад

      @@AndreasSpiess with enough research I believe one can make this available for raspi. My issue would be in the bootloader. I was doing some research back in the first raspi model, and i found that the source code for the bootloader wasn't maintained since 2010. However, it keeps finding its way into new releases for raspi. When one is taking on this challenge of building a new image with ramfs and saving to flash, I have no idea what challenges exist knowing the bootloader is just too antiquated and not openly available for making changes. Just my thoughts, hoping my memory didn't fail me.

  • @jamesmichener7526
    @jamesmichener7526 Месяц назад

    Interesting video. I wonder what the MTBF is of SSD in a relatively static Pi is. To date I haven't had any fail, and it has been well over five years.

    • @hesspet
      @hesspet Месяц назад

      If you have a good stable power supply until the universe dies :-) OK If you google you will find e.g. this: he average MTBF for SSDs is reported to be 2.5 million hours. So it should run longer than you. The the MTTD (Mean Time To Die) for average people ist only 650000 hours. :-) :-) :-) SCNR

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      This video is about SD cards. SSDs do not suffer those problems.

    • @jamesmichener7526
      @jamesmichener7526 Месяц назад

      Dang.... I mean to say micro SD card

  • @GinnyGlider
    @GinnyGlider Месяц назад

    Not, entirely sure If I can do this on a pi-zero-2w running pi-hole/VPN, (I would need to save the storage DB off-site,
    but there could be other writes I'm missing, which I would need to determine by tracking the writes, I guess).
    However, I do like the idea in terms of speed... My idea is to have writes go into memory first before being written to the SD card.
    This would occur at set time intervals or when a certain memory limit is reached. The idea is in theory, since I'm not sure how I would go about it.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад +1

      You have to try if your setup survives a reboot using this configuration.

  • @joachimkeinert3202
    @joachimkeinert3202 Месяц назад

    My method is to use an SD card that is at least 2 times as big as needed. So it has enough spare tracks to be used when the card detects bad tracks.

    • @hesspet
      @hesspet Месяц назад

      None of the SD cards that were wrecked over the past few years had issues with overuse of the memory cells. They simply died electrically their internal system was brocken. If the bad sectors occur in the wrong place, you can have as much storage as you want, but the card is then destined for the trash.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      AFAIK there is not a lot of information on how good the SD cards are with such algorithms. But if it works for you that is good!

  • @AllToDevNull
    @AllToDevNull Месяц назад

    That is why I use NixOS. That would then be one flake somebody builds and you just import the flake in your configuration. nix store is ro by default and is only remounted when new derivates are installed. Only thing to do is to move some /var /home etc into overlay and do a regular writeback. Declarative systems are the future, I will never go back to copy pasta systems together :)

  • @belenhedderich3330
    @belenhedderich3330 Месяц назад

    Hi thanks again for your valuable videos. I wonder if this will work with klipper

  • @alanmon2690
    @alanmon2690 Месяц назад

    My understanding of the write protect switch on full size cards is that it tells the OS not to write to the SD card, which the software mods achieve...

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      I for a long time did not use such big cards anymore...

  • @eldorado3523
    @eldorado3523 Месяц назад +1

    Funny how this video shows up exactly when I find out my rpi sd card is dead haha

  • @AgozieAni
    @AgozieAni Месяц назад

    This feature saved my kiosk project in 2020.

  • @tinygriffy
    @tinygriffy Месяц назад

    Thanks for the Video !!
    if you have a server somewhere in your home network, would PXE booting be an option ? ;)

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад +1

      I am not sure it would solve the issue as PXE boot is read-only.

    • @tinygriffy
      @tinygriffy Месяц назад

      @@AndreasSpiess Oh, I am sorry.. that's a very old idea and probably quite off topic ;) Remote booting an OS.. have system FS mount via NFS.

  •  Месяц назад

    Zram and log2ram are not that hard to install. Use a light version like dietpi or armbian does help to. There is a posibillity to boot from network but I haven't tested that yet.
    A fast usb device or a portable ssd unit is most convinient but the user case matters too.
    I noticed that my pi runs cooler and use less energy with zram and log2ram. Something I use on my other linux systems with more available ram, like 64GB in my laptop.

  • @alx8439
    @alx8439 Месяц назад

    And one more thing. The more free space you have on SD card, the longer it survive over time if writes are still inevitable. It's just the way how all the SSD work - aka copy-on-write

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      I am not sure if standard SD card do wear leveling like the SSDs because they are made for a specific purpose.

    • @alx8439
      @alx8439 Месяц назад

      @@AndreasSpiess I read from multiple sources that internally, when it comes to new block writes, or block updates, the card controller marks the old block as empty and writes everything to a new one instead. Practically it means the more free space (unused blocks) you have, the less often each block will be overridden. Which means more lifespan to them

    • @alx8439
      @alx8439 Месяц назад

      @@AndreasSpiess the major difference to SSD disks is that SD card controllers (usually) do not implement any batching, not have any memory to cache. Which makes writes to happen more often thus more impactful

  • @Darkk6969
    @Darkk6969 Месяц назад

    If you use DietPi it automatically make use of log to ram feature by default.

  • @CanadianYewTube
    @CanadianYewTube Месяц назад

    I had corruption problems with several SD cards when I used unofficial power supplies. Since I switched to the official power supply, I have had no problems with continuous running for about 5 years (with no modifications to limit writing). At the time I made the change, I read that the official power supplies deliberately provide 5.1V or higher rather than 5V. I can't find the page now, but I remember reading that a reason for data corruption was low voltages. Can anyone confirm this?

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад +1

      I also select a voltage above 5V for my Raspberries if I can.

  • @eminbe312wqx
    @eminbe312wqx Месяц назад

    Bilgilendirme için teşekkürler efendim huzur mutluluk dilerim efendim güzel video

  • @nyeleskettes
    @nyeleskettes Месяц назад

    The only good thing that raspi gave to me is the DietPi OS. This works flawless on my passively cooled PC. The raspberry ended in the trashcan. This gadget WAS good for education since the price was well placed. Nowadays it is just an overhyped/priced electronic waste.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      I still use it for small projects, also when I need a few GPIOs. For the rest, I run Proxmox on X86...

  • @DataDashy
    @DataDashy Месяц назад

    There are distros like Voyager linux which was tailormade to protect flash devices from wearing off by ro mounting most of the cards partitions and even the volatile data is stored in ram. Raspbian and other malwared based distros trashing the storage like hell, expect crowdstrike integration in systemd soon 😂

  • @Suchtzocker
    @Suchtzocker Месяц назад +3

    i never use raspberrys for applications that could cause any damage when crashing or becoming corrupt for me its more like a testing platform. GUI maybe for multimedia applications where i can just redo it if it crashes and i dont loose any media because its on a proper server

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      Thank you for sharing your experience. I use a few Raspberries in productive environments. But only very dedicated systems...

  • @botialoach1
    @botialoach1 Месяц назад

    Very useful info, can you use the overlay file system on HA raspberry pi ?

    • @DumahBrazorf
      @DumahBrazorf Месяц назад

      You should use an ssd for HA, not only for data safety but for speeeeed

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      This is not useful for HA.

  • @bepstein111
    @bepstein111 Месяц назад

    Did adafruit update their writeup? Because adafruit's writeup (as of right now) is exactly the same thing you do in this video, the two "barefoot" methods are not from adafruit like you suggest.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      Maybe they adapted their method in the meantime.

    • @bepstein111
      @bepstein111 Месяц назад

      @@AndreasSpiess Ah , I'm just now seeing that this is a rerun, so yeah, it's probably been long enough. Glad they updated their writeup!

  • @klauszinser
    @klauszinser Месяц назад

    Thats what i would have wanted to know maybe 7 years ago. There were 2 oder maybe 3 RasPII failures.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      The first appearance of this video was a few years ago ( now a replay during my summer break). But better late than never 😉

    • @klauszinser
      @klauszinser Месяц назад

      @@AndreasSpiess Not your fault. It was before there was the more complicated SSD solution.
      I stopped using them.
      As you say, better late than never.

  • @TheFrewah
    @TheFrewah Месяц назад

    How funny, this was the first thing I thought of. I recently bought software that creates a ram disk and found things work differently on linux. You van redirect browser data to this disk and it will be gone when you reboot. It will also make your ssd last longer.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      I usually do not care too much about the wearing of SSDs in raspberries because they are made for „big computers„. But it helps for sure.

  • @JJ-fr2ki
    @JJ-fr2ki 22 дня назад

    Clever. Never disappoints. But I think this would be better as a very short video with a web link to a step by step directions .pdf or separate video. I don’t have the IQ to remember it but had to scrub through video to see your solution.

    • @AndreasSpiess
      @AndreasSpiess  16 дней назад

      The video is only 8 minutes long. I always try to keep it short. But obviously not short enough for you ;-)

  • @hankhulator5007
    @hankhulator5007 Месяц назад +1

    Hi Andreas, you have another solution :
    * reconfigure shytstemd journal to redirect output to a socket (1 line to modify, 1 to enable),
    * install rsyslog and configure it to sent all logs to a (rsys)log server.
    The main advantage is the log server only has a few open ports, so hacking the logs is much more tedious.
    You can achieve this only with shytstemd, but as you might have guess, for me it is just an insane POS whose only goal is to crack open all Linux machines like a can of coke on a summer day when the dictatorship time will come.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад +1

      Thank you for adding another possibility!

    • @hankhulator5007
      @hankhulator5007 Месяц назад

      @@AndreasSpiess I wonder if logs could be sent to the log sever through LoraWan.

  • @Electronieks
    @Electronieks Месяц назад

    Youre my favoriete yt channel ever

  • @docwhogr
    @docwhogr Месяц назад

    new sd cards seem to die very easily so i bought an kingston industrial last month... we'll see..... 8gb for 9€ isn't bad price and they are very fast and stable on the writing speed'

    • @hankhulator5007
      @hankhulator5007 Месяц назад

      That's a terrible price, €1.5/GB !

    • @docwhogr
      @docwhogr Месяц назад

      @@hankhulator5007 no, my bad it was 9€ for the 8gb and 25€ for a 32gb but the 8gb is sufficient for my IOT. if you consider that the one i replaced was a normal kingston 16gb i bought for 5-6 euros and after ~8 months it crash my iot rasberry i think it totally worth the money... before that i was using an 8gb local store brand name sd for 5 years...

  • @bastiannenke9613
    @bastiannenke9613 Месяц назад

    I still believe the issue was partially with the way those cards managed the writes/ wear leveling.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад +1

      You are right. Most of them do not have a good wear levelling.

    • @bastiannenke9613
      @bastiannenke9613 Месяц назад

      @@AndreasSpiess I believe that changed at some point, just grab a random brand with industrial grade cards and look at the max tb written.

  • @tabeschektabeschek1852
    @tabeschektabeschek1852 Месяц назад

    I use an USB Stick AS memory. Runs fine.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      Good idea if they have wear levelling implemented...

  • @ovalwingnut
    @ovalwingnut Месяц назад

    U. R. the "Pi Whisperer"...

  • @Piotr_T
    @Piotr_T Месяц назад

    You can boot RPi from network

  • @idcrafter-cgi
    @idcrafter-cgi Месяц назад

    i have had a linux laptop with a almost dead ssd which i almost never noticed due to linux loading everything into the 32GB ram, i noticed it when kde alerted me that the smartdata notification said the ssd is failing.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      Good that the SSDs have such an alarm system!

  • @TimoNoko
    @TimoNoko Месяц назад

    Strange adventure with overlay I am just having. "sudo apt upgrade" crashes because 1.9 G overlay is too small. Only solution seems to be to turn the overlay off, use 32G SDcard as "/", and then turn the overlay back on.

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      I mostly turned the overlay off for upgrades as I wanted them permanent.

    • @TimoNoko
      @TimoNoko Месяц назад

      So. That is why the upgrade grew so big. Previous upgrades were just temporary.

  • @sshah876
    @sshah876 Месяц назад

    Nice, will it work when docker applications/home assistant is installed in Raspbian os. I remember a couple of years ago my docker apps were not working with overlay file system enabled, can anyone kindly confirm,. Many thanks and cheers

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад +1

      Such applications store a ton of info. You do not want to loose it during reboot. So this is not a good solution.

  • @Ed19601
    @Ed19601 Месяц назад +1

    TBH, I have given up on raspberries for anything reliable. Had too many crashes, and before anybody comes with 'yeah but yo have to use good quality sd cards', I have. Anyway, current prices make the raspberry a total joke anyway

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад +1

      I still use the raspberries for small projects like gateways, or weather balloon receivers because they are small and easy accessible GPIOS. For server type applications I replaced them, too.

  • @Monochram
    @Monochram Месяц назад

    SD card issues are soooooo exaggerated with proper configuration . I have a rpi4 and a decent ssd, and it's running for 4 years almost. the same sd. one just need to make sure to lower the amount of writes, or use a framework that does this for u, such as dietpi (which I'm currently using). and not that I'm running just HA, something around 12~13 different services
    using ssd for home server that have sufficient ram for the job is an overkill

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад

      You sometimes use SD and sometimes SSD in your comment. SSDs are made for heavy writing, standard SD cards not. But I agree, there are other possibilities to get around that issue.

  • @steinmar2
    @steinmar2 Месяц назад

    I already had this problem with the SD card and Home Assistant OS.. one time after updating one timer after rebooting

    • @AndreasSpiess
      @AndreasSpiess  Месяц назад +1

      This will not work with HA because you loose all your data during reboot.

    • @steinmar2
      @steinmar2 Месяц назад

      @@AndreasSpiess yes i found out the hard way ^^