MergerFS + SnapRAID : Forget about RAID 5 in your Home Server !

Поделиться
HTML-код
  • Опубликовано: 4 фев 2025

Комментарии • 106

  • @trapexit
    @trapexit Год назад +112

    mergerfs author here. Thanks for the coverage. Very thorough.

    • @thenomadcode
      @thenomadcode  Год назад +8

      You are very welcome !

    • @rohannatarajan6048
      @rohannatarajan6048 Год назад +8

      Thank you, your creation is MOST appreciated and exactly what i was looking for! :)

    • @OsX86H3AvY
      @OsX86H3AvY Год назад

      Hey i wrote an article about your work in Linux Magazine a little while back - been using it for a NAS for years now, i LOVE it, keep it up!!!!!

    • @mikewickenden8754
      @mikewickenden8754 4 месяца назад

      I've been looking to do what he shows. Are you planning to make a version for Windows like Drivepool?

    • @trapexit
      @trapexit 4 месяца назад

      @@mikewickenden8754 No. I'm really not interested in working on Windows and Drivepool fills the general need.

  • @dm1978
    @dm1978 10 месяцев назад +2

    Great content!
    Professional video, music selection and volume kept it as a pleasant "fill", great B-roll and you're very easy to listen to.
    This makes your helpful knowledge and content pleasant to listen to over someone who has a grating voice, loud/inappropriate music etc.
    Love your work.

  • @bvd_whiteytighties7349
    @bvd_whiteytighties7349 Год назад +12

    After watching this video through completely, I noticed the small number of subscribers and views. I was pretty surprised! High quality video, audio, graphics, cuts, and demos. Keep it up!

  • @angelo9176
    @angelo9176 Год назад +7

    Wow, it really surprised me when you said this is your first video. This was super helpful and well produced. I think im sold on doing mergerfs and snapraid.

  • @thespencerowen
    @thespencerowen Год назад +4

    How do you only have 121 subscribers? Your video quality is fantastic. Great lighting, Great B-Roll. Great Ambient music.

    • @thenomadcode
      @thenomadcode  Год назад +2

      Man this is so amazing to have this kind of comment ! My video is my phone (Pixel 6), the lightning are the few colored light bulbs I found, and the ambient music was free.
      I have only a few subscribers because it is my only video yet. I have tons of topics to share about, from servers, to development, project management etc... But free time to make them has been spread thin recently with the future arrival of my first baby. I had to move to another flat and all. But promise soon I will make some more on regular basis.

  • @ipbody
    @ipbody Год назад +7

    What a great video It was really clear, the editing is on point, the music at the right volume and so on. I'll wait for new videos!

    • @thenomadcode
      @thenomadcode  Год назад +2

      Thank you so much for your comment ! It really motivates me to produce good quality content !

  • @miricostanti
    @miricostanti Год назад +1

    Very nice video! I hope to see more tutorials from you, your channel will explode in no time with this production quality!

  • @TomAtkinson
    @TomAtkinson 8 месяцев назад +2

    I like how it uses vanilla filesystem and JBOD (just a bunch of disks) ethos. I plan to use mergerfs (plus snapraid) for my primary music library, with syncthing updating a mirror one to a second computer / jukebox. Syncthing is good on this second machine as it also propagates deletes to help me maintain and prune the library.

  • @Konstantin_Kon
    @Konstantin_Kon Год назад +1

    Great level of production! I'm finally made my choise omv + mergerfs + snapraid

  • @robinclark5945
    @robinclark5945 7 месяцев назад

    great video, gave me confidence to go with this solution for my NAS.
    You’re great and breaking down complex topics and making them approachable and understandable. Thank you 🙏

  • @swampcat4901
    @swampcat4901 Год назад +1

    Thanks! Great alternative for Unraid. Best for situations where data is quite static.

  • @soffizatyon
    @soffizatyon Год назад

    Very nice video guide! The idea to use Slack for the notifications is quite cool :)

  • @deadlightning4532
    @deadlightning4532 Год назад

    Great video! Very informative and straightforward with the explanation and setup. Forked the script and got it working in a discord channel rather than slack.

  • @slablife
    @slablife 5 месяцев назад

    Love the quality of your content, good work.

  • @bobclarke5913
    @bobclarke5913 Год назад

    Perfect guide, thank you so much.
    Since I'm picky about what data goes on which drive I was going to skip MergerFS, but now I see it'd be handy to have it running as a read only drive to share out to media servers and such.

    • @thenomadcode
      @thenomadcode  Год назад +1

      Glad it helped ! Merger Fs still let you decide where to put what somehow. I am not sure but I think you can put a rule that say that if a folder is in a drive, all the subfolders and files has also to be in this drive. Have a look on the doc !

  • @Bob-Fields
    @Bob-Fields 21 день назад

    Great video! Medium level difficulty for noobs but still accessible. Bravo!

  • @atilla4352
    @atilla4352 6 месяцев назад

    subscribed! I'm getting into linux and servers. it was a great explanation.

  • @veitsandfort2212
    @veitsandfort2212 8 месяцев назад

    Cool trick with the slack channel. Thanks!

  • @mynameiszoro
    @mynameiszoro Год назад

    Great video man!
    I am successfully up and running exactly like I wanted.
    Thank you!

  • @andyvoyto
    @andyvoyto Год назад

    Great video - you covered the content perfectly and it was really easy to understand. Look forward to more of your content 👍🏼

  • @juanherediamayer
    @juanherediamayer 4 месяца назад

    Great content! Keep it going. Thanks

  • @Erelyes
    @Erelyes 6 месяцев назад +1

    Great video. Hope you make more at some point. A minor point not covered is Snapraid supports as many parity levels as one likes. There's also split parity although the documentation for it is poor.
    Any reason you chose xfs over ext4 or another filesystem?

    • @thenomadcode
      @thenomadcode  3 месяца назад +1

      Yes. XFS is supposed to be better for large files than ext4. But honnestly the difference is very small. ext4 would have been perfect as well.
      And yes I will make more videos, just got a baby that came in the middle :D

    • @Erelyes
      @Erelyes 3 месяца назад

      @@thenomadcode Cool, thanks. And congrats!

  • @camilojose3147
    @camilojose3147 2 месяца назад

    Excelent video !

  • @old-barn
    @old-barn Год назад +1

    Thanks for the video, very informative. Was there a reason you installed the packages on the cmd line instead of the omv plugins? both snapraid and mergerfs are available and presnt UI options to manage? (asking as a relatively new user, not a pro!) - only thing I think you missed, when you stated what mergerfs policy you chose, you did not show how you set that? Many thanks!

    • @thenomadcode
      @thenomadcode  Год назад

      Hello ! I didn't installed OMV plugins because I don't use Open Media Vault ! I run ubuntu server LTS ! For the policies, I invite you to have a look here as it can depend of your usecase : github.com/trapexit/mergerfs#policies. I used the pfrd policy on creation, for this you need to add category.create=pfrd to your options in your /etc/fstab file.
      Mine is now looking like :
      #MergerFS
      /mnt/data* /mnt/pool fuse.mergerfs allow_other,use_ino,cache.files=partial,dropcacheonclose=true,category.create=pfrd 0 0

    • @old-barn
      @old-barn Год назад +1

      @@thenomadcode oops I hadn’t even noticed as i was researching omv and came across your vid, my apologies! Still a good vid though, many thanks for the reply!

  • @AlbertVanOpstal
    @AlbertVanOpstal 2 месяца назад

    Interesting Video. And I did it a one year ago. Bu how to recover from a situation where now the Rpi crashed and need to be replaced by a complete new fresh installation? (I don't have a copy of the old OS SSD.) What is the fasted and correct procedure tin that case ?

    • @thenomadcode
      @thenomadcode  2 дня назад

      Hey !
      You might have notice that my data and OS are on separated drives. I have a OS SSD, for the OS and the rest of my data, is on the HDD drives. What I do is very basic, I just regularly save the configuration files present in /etc, in my data disks with a cron task. All the software running on this server are docker images running on a stack, and I have the definition of this stack hosted on a github repo.
      So long story short, the OS drive can be erased, reset with a fresh os, and then in like 30 minutes, I am able to put back the previous configuration and spin up the software that I use.
      But nowadays, you can do way better than this, with Ansible you are able to reconfigure all your machine automatically. Immutable distros like NixOS can be also very interesting in order to re-establish your system quickly.

  • @andrewparsons1041
    @andrewparsons1041 5 месяцев назад

    How are you achieving the SSD caching? Using bcache?

    • @thenomadcode
      @thenomadcode  3 месяца назад

      Personnaly I did not add SSD Caching. Later maybe when I will have som extra SSD

  • @rohannatarajan6048
    @rohannatarajan6048 Год назад +1

    Really nice video! Could you explain why you chose XFS instead of ZFS as your file system?

    • @thenomadcode
      @thenomadcode  Год назад +3

      Yes of course ! ZFS is made basically to be able to create logical volume RAID. As I don't need such functionality due to the use of SnapRAID and MergerFS. The 2 choices left where the classical ext4 and XFS. ext4 is very good for general usage purposes It works amazingly well for small files. XFS is better for large file transfers, or I/O operations, and is famous not perform amazingly with small files. This is why I choose ext4 for my system SSD drive, and XFS for my storage drives.

  • @Mrbibi38
    @Mrbibi38 Год назад

    Awesome video this is going to help me a lot.
    Question that has nothing to do, what do you use to make your diagrams?
    Thanks for this content, continue that ways!

    • @thenomadcode
      @thenomadcode  Год назад

      Hi ! Thank you, I am glad to hear it helped you.
      For my diagrams I just used Obsidian canvas feature. I take all my notes on Obsidian and their quite lite canvas feature was enough to do it those diagrams :) Otherwise, if I need something more complex I use draw.io or Figma's FigJam.
      I will try to make another video soon about how to backup your NAS for super cheap ;)

  • @ColinUniverse
    @ColinUniverse Год назад +1

    Great video! One question, do you use ECC memory in your NAS?

    • @thenomadcode
      @thenomadcode  Год назад +2

      Absolutely not :D My server is here to store my medias, photos and video footage. If one day I can use ECC I will of course, but my current build was made with spare parts from my previous PC, and I was on a budget :D

    • @Bixmy
      @Bixmy Год назад +1

      On homelab level ECC is probably overkill for most people 😂

  • @ProCoder777
    @ProCoder777 Год назад

    Thanks for the informative video. But one aspect remains unclear - how to use SnapRAID to recover data when disks are damaged? I would be grateful for an explanation.

    • @thenomadcode
      @thenomadcode  10 месяцев назад

      Hello ! It is quite easy with the "snapraid fix" command. I invite you to have a look on snapraid documentation for more details : www.snapraid.it/manual

  • @PVPFTL
    @PVPFTL Год назад

    Thanks for the video it was great. @thenomadcode can you make a video showing the steps to remove and replace a failing drive?

    • @thenomadcode
      @thenomadcode  3 месяца назад

      Thank you ! Indeed I should. But as it is not just a test machine and I have a tons of data on it, it could be a bit complicated. Snapraid documentation is very exhaustive about it. You could for exemple, just delete files on a disk, and use snapraid to recover them.

  • @AgentMoler
    @AgentMoler 9 месяцев назад

    Would this be a good solution for combining an array of 14TB, 16TB, and 18TB? I’m assuming the parity drive would be one of the 18TB.

  • @cjgrandjean
    @cjgrandjean Год назад

    Great informative video. And nice to have an alternative to the raid standard. However, even if this should prove to be more robust than raid, you must still not consider your critical data safe.
    It sounds like your real problem was lack of backup.
    Raid (even this version) is NOT backup.
    Please, Google 3-2-1 backup, to keep your critical data safe.
    It is not a software, or brand name. It is a mindset

    • @Bixmy
      @Bixmy Год назад

      True. Even thou i myself dont follow that rule to a T. i still at least have one off site that get sync to wasabi s3every night.

    • @Bixmy
      @Bixmy Год назад

      Raid zraid or any kind of raid is not a backup it just redundancy for local storage not backup

    • @thenomadcode
      @thenomadcode  Год назад +1

      I glad you say it. My next video will be about backups. This was indeed the real problem here. A complete lack of professionalism from me.

  • @tristanbautista9705
    @tristanbautista9705 8 месяцев назад

    Hey man! I followed your tutorial and it worked but when I reboot my server the mergerfs pool basically unmounts, the hdd is still detected and partitioned, but the pool is gone.

    • @thenomadcode
      @thenomadcode  3 месяца назад

      Did you correctly added a mount line in your /etc/fstab ? (sorry for the late answer !)

  • @ryanrocksize5
    @ryanrocksize5 9 месяцев назад

    I've particularly dumped raid 5 for mergerfs , Raid 5 gave far more headaches when the drive failed and replacement became a logistical nightmare at times
    Then i realized the integrity of all the data was no longer important since its just a media server , and only store important data on SSDs in the server instead

  • @loboaureo6661
    @loboaureo6661 7 месяцев назад

    But you can have the same problem you had with your previous raid. if you lost one disk, you dont know it and make the sync, the parity will delete the content of that disk, making it unrecovereable, isn't it?

    • @thenomadcode
      @thenomadcode  7 месяцев назад

      This is a good point, this could be avoid easily by checking that your disks are correctly mounted and a test file accessible before making the parity. Example : On disk 1 you create a test1 file, on disk 2 you create a test 2 file. You check that these files are here before runing the parity and raise an alert if they don't !

  • @luizrezende9329
    @luizrezende9329 Год назад

    Parabéns por sua explicação!

  • @elitekt6
    @elitekt6 Год назад

    Is it possible to use encryption with this solution?

    • @thenomadcode
      @thenomadcode  Год назад

      This is a very good question ! Technically, I don't see any issue with encrypting on the file level. As long as the file list is visible by MergerFS. As this one is not checking any content, it should not be any issue. For SnapRaid, as it is a parity software, it will treat the files the same way, encrypted or not.
      As encrypting on the drive level with LUKS or other, this should not be any issue neither !
      If you try it, keep me in touch, I'm curious !

  • @leandrocosta4283
    @leandrocosta4283 Год назад


    174 / 5.000
    Hello,
    I really liked your article about MergerFS + SnapRAID, I would like to know if it is possible to place some cache using an m2 mvme disk?
    Could you do a little about that?

    • @thenomadcode
      @thenomadcode  10 месяцев назад

      Hello ! I know that MergerFS is allowing some form of caching. But not sure it corresponds to your needs. I invite you to have a look on MergerFS documentation about caching here github.com/trapexit/mergerfs?tab=readme-ov-file#caching
      Specificaly the section about tiered caching.

  • @battmarn
    @battmarn Год назад

    installing fuse tried to uninstall half my essential ubuntu programs

    • @thenomadcode
      @thenomadcode  3 месяца назад

      Seems you went to depedencies hell

  • @OsX86H3AvY
    @OsX86H3AvY Год назад

    oooooo man - WD Green spinning drives - I assumed they were all dead by now - I had one go as well, they tried to make it green but they just made it blue (sad I mean cause its dead, get it?) but that said, I ONLY use WD for spinning drives...I have only ever seen a hadnful of WD fail but have seen literally ALL of the seagate ones i've used fail, EVERY ONE...anyways nice content!

    • @thenomadcode
      @thenomadcode  Год назад

      Hahah indeed. Last time ever I put a WD green somewhere :D

  • @alexbright7735
    @alexbright7735 4 месяца назад

    WD Green strikes again

  • @espressomatic
    @espressomatic 7 месяцев назад

    I've watched 75-100 videos on every conceivable filesystem and non-RAID NAS/drive setup out there. And every single time the answer seems to be "just use Unraid instead."

    • @thenomadcode
      @thenomadcode  3 месяца назад

      Yes, but no way that I am paying for propriatery sofware when there are open source tools available ::D

  • @alexbright7735
    @alexbright7735 4 месяца назад

    File level parity sounds slow. Block level much faster.

    • @thenomadcode
      @thenomadcode  3 месяца назад

      . I don't know how fast it is to be honest, because it does not really matter. As I do the parity only once every 4 hours, it does not have any impact on system writes speed.

  • @devoxer6549
    @devoxer6549 Год назад

    That is basically what Unraid do

    • @thenomadcode
      @thenomadcode  Год назад +2

      More or less, the main difference to me is that unraid is not free, and not open source. This is important to me.

    • @mrpops2ko
      @mrpops2ko Год назад +1

      snapraid is not live parity it is snapshot, whereas unraid is live. unraid has major performance issues in terms of their SHFS implementation which cause significant IO wait times and bring your computer to a crawl. Not an issue if you go all flash but if you go all flash, why even use unraid in the first place.
      Unraid also has major data integrity issues in that it has no method of checksumming. So you can end up in the scenario where a file silently corrupts and the parity implementation they use will never know. checksumming is important because the parity could be wrong or the filesystem could be wrong and the checksummed file acts as the arbiter between the two. When you run a 'check' with unraid if the parity has had any flipped bits / bitrot then it'll report back that its found errors but you have no means of assessing whether the bitrot occurred in the parity, or in the file - with checksumming you could do that. (which snapraid provides)

    • @trapexit
      @trapexit Год назад

      ​@@mrpops2ko The IO wait issue can come up with mergerfs as well. More investigation is needed but may be a disconnect between the storage performance and rest of the system. Too much concurrency. In mergerfs I've added thread pinning and ability to change general priority as a work around for now.

  • @hamisgulyas
    @hamisgulyas Год назад

    why not use hardware RAID instead of software bodged workarounds?

    • @thenomadcode
      @thenomadcode  Год назад

      Because it is expensive ! You need the mother board that support it ( my case is Mini ITX form factor). Merger fs and snapraid are enough for me ! But as I said, in production server environment, go for raid !

    • @franciscooteiza
      @franciscooteiza Год назад

      Hardware Raid is dead and is a bad Idea in 2023

  • @FlorinStancu
    @FlorinStancu Год назад

    Did you tried unRAID?

    • @thenomadcode
      @thenomadcode  10 месяцев назад

      No because it is a paid software, and I like to run the OS I want without having to pay for it :D

    • @FlorinStancu
      @FlorinStancu 10 месяцев назад

      i said the same thing until I tried unraid

  • @Bixmy
    @Bixmy Год назад

    In short dont use hardware raid. Depend on the usecase I still prefer zfs more than this setup.

    • @hamisgulyas
      @hamisgulyas Год назад

      ZFS require ECC memory, which requires server grade tin. so why bother over-consuming your memory for storing files when you have HW RAID?

    • @Bixmy
      @Bixmy Год назад

      ​@@hamisgulyas I mean if you want a bitrot from a raid that does not actually check parity, you do you. I'll stick with BTRFS or ZFS. this is not even counting the other advantages like a filesystem that actually knows what space is in use it just scans the in-use block, not the whole storage.
      Also, it doesn't require server-grade stuff, and server-grade stuff is quite cheap anyway these days. 256 GB DDR4 RAM is quite cheap now. just get the last-generation server equipment and run software raid on it.
      TLDR; In my opinion, HW raid is a bad idea for storage in 2023

    • @hamisgulyas
      @hamisgulyas Год назад

      @@Bixmy let's agree to disagree..
      Why would I need parity check? What do you store on your missing critical NAS server? I'm absolutely not afraid of "bit rot" as you say, if something deteriorating data wise than you have a bigger problem than your RAID controller.
      When you buy the latest server grade stuff is not cheap WHEN it's a server. The power consumption is over the roof and keep in mind: most servers come with hardware RAID, so why double the hassle? Software RAID is also using memory & CPU, so why add yet another level of complexity just for the sake of file storage? The more complexity you introduce to your storage stack you have a better chance you screw up something. Seen that, wouldn't do that.
      TLDR; It's dull to say that HW RAID is a bad solution because you are a ZFS aficionado. RAID technology exists since 1987, ZFS does since 2001. Also, I trust more HW than SW.

    • @thenomadcode
      @thenomadcode  Год назад

      Personally, I store photos and videos, but also private files ( Nextcloud), could be text files or other. I like to have the parity check, I just don't need it real time.
      My homemade Nas has been made by scrapping some parts here and there and was not having HW raid.

  • @franciscooteiza
    @franciscooteiza Год назад

    you look very, very sad, are you ok?

    • @thenomadcode
      @thenomadcode  10 месяцев назад

      I am perfectly fine, thank you for asking :D

  • @BrianThomas
    @BrianThomas 11 месяцев назад

    If you're losing data with a RAID. That means you're not backing up. I always try to go with the idea of "Two is One and One is none". If I'm building a Raid 5 with 5 drives. Rest assure the plan is to acquire 10 drives in another Raid 5 configuration or a Raid 10 with 4 drives for backup.

    • @thenomadcode
      @thenomadcode  10 месяцев назад

      Indeed, I was really stupid !

  • @goffs24
    @goffs24 Год назад

    Everytime I run the script, I get [WARNING] SCRUB job ran but did not complete successfully. Here is the output in the terminal. Can anyone give some insight?
    ###SnapRAID SCRUB [Fri Dec 1 10:09:39 PM UTC 2023]
    Self test...
    Loading state from /home/xxx/snapraid.content...
    Using 0 MiB of memory for the file-system.
    Initializing...
    The array appears to be empty.
    SCRUB finished [Fri Dec 1 10:09:40 PM UTC 2023]

    • @goffs24
      @goffs24 Год назад

      Tried adding more files than my two test text files and I guess it's working now. Must "fail" when the files are only on one drive or something.

    • @thenomadcode
      @thenomadcode  Год назад

      Sorry for the late answer ! Yeah weird issue ! I hope everything is working now !