Microsoft (and others) implements Data Deduplication at a much lower level - not the file level. Each 4K chunk of data (regardless of what file it belongs to) is compared to other 4K chunks and if a duplicate is found, a pointer is created and a counter is increased for that chunk. This video describes what Microsoft calls "Single Instance Storage" - a less efficient form of deduplication.
Eli, the technology you describe in this Introduction is was used to be called single-instance storage (it's not the current definition). It's what Microsoft Exchange and other products use when storing messages and attached files. Only a single instance of a message or file is saved with pointers to each user in the distribution list. Modern data deduplication is performed at the block level so if, say, a single bit of a store file is change, only a single block with the new bit is saved.
Data deduplication on SAN and NAS is performed automatically or on a scheduled basis (weekly, monthly) to recover space already used. There is a modern form of storage called single instance storage (the new definition), which performs in-line deduplication in real time, never saving blocks in a datastore which already exist in the store. This is perfect for things like virtual desktops, where individual VMs are 95% identical to all other VMs. Single instance storage is currently very expensive.
After many years of using data deduplication, I can honestly say that I have never lost more data to any other computing technology. I've lost entire RAID 5 arrays (RAID 5 is a scam) but I haven't lost any more data there than I have using data deduplication. It sounds good on paper but in practice, it always fails. Wendell from Level 1 Techs says this: "Friends don't let friends do data deduplication."
Nice video! Thanks for the explanation! Does all vendors like NetApp, Dell EMC achieve the same performance in terms of data duplication, or could varies based on how they use this technology?
Does the DL160 have an iLO built in? If so and it's licensed, you can use virtual media to install Server 2012. Put the CD in your PC's DVD-ROM drive and mount the drive virtually through iLO. Reboot the DL160 and if it's set to boot from DVD, it should boot from the DVD in your PC.
Eli, that was super interesting, thank you. One question though: As far as you said, data deduplication works on file level (you talk about folders etc). Would this also be of use if for instance i store a lot of backups from different users / machines eg in disk images? Like - if i have 10 or 20 backups from different windows 7 machines would it recoginze the "similarities" within the disk images or does it strictly work on a file system level?
If they have a pointer to an identical file, just their individual pointer goes away. If they have some identical file that other people have a pointer to, the original will be moved to another user, and the pointers will be redefined.
i brought a hp prolaint DL160 Gen 8 it doesn't come with cd I brought windows server 2012 cd how can i transfer it to a usb Do a video how to do it Then do video how to install it on the hp prolaint DL160 Gen 8
Eli, your drawing looks like someone jump and split his legs & arms, am not saying that is not good I am normal user who learn alof of things from you, personally it's just the most and easiest explanation this way. BUT if you have add icon's like 4 computers and use the mouse as link then we'll know exactly and more specifically what you talking about to be clean and clear ,,, circles are not like squares. Anyway it's just a suggestion & still the best of the best, thanks alot man
I have never seen such a clear and plain explanation. Simply put it was supercalifragilisticexpialidocious elaboration.
Microsoft (and others) implements Data Deduplication at a much lower level - not the file level. Each 4K chunk of data (regardless of what file it belongs to) is compared to other 4K chunks and if a duplicate is found, a pointer is created and a counter is increased for that chunk. This video describes what Microsoft calls "Single Instance Storage" - a less efficient form of deduplication.
Other companies also have their own name for such a technology.
You're the best, monsieur! I'm not the youngest, but new to this. There's so much to learn, and you make it simpler. Thanks!
Eli, the technology you describe in this Introduction is was used to be called single-instance storage (it's not the current definition). It's what Microsoft Exchange and other products use when storing messages and attached files. Only a single instance of a message or file is saved with pointers to each user in the distribution list.
Modern data deduplication is performed at the block level so if, say, a single bit of a store file is change, only a single block with the new bit is saved.
Gotta LOVE Eli!!! Now this guy knows his stuff!!!
Data deduplication on SAN and NAS is performed automatically or on a scheduled basis (weekly, monthly) to recover space already used. There is a modern form of storage called single instance storage (the new definition), which performs in-line deduplication in real time, never saving blocks in a datastore which already exist in the store. This is perfect for things like virtual desktops, where individual VMs are 95% identical to all other VMs. Single instance storage is currently very expensive.
not only exact, variations also possible
I like how says exact :) 3:23
After many years of using data deduplication, I can honestly say that I have never lost more data to any other computing technology. I've lost entire RAID 5 arrays (RAID 5 is a scam) but I haven't lost any more data there than I have using data deduplication. It sounds good on paper but in practice, it always fails.
Wendell from Level 1 Techs says this: "Friends don't let friends do data deduplication."
Eli, you simply rock man !! Great stuff
great job nice and easy explanation
Its called SketchBook Express from Autodesk, its an app for Windows 8.
Nice video! Thanks for the explanation!
Does all vendors like NetApp, Dell EMC achieve the same performance in terms of data duplication, or could varies based on how they use this technology?
Thanks Eli
Sketchbook!!! Thank you so much. :-)
Perfect and thorough explanation.. lots of thanks from Calgary! Sorry for the smoke on your visit to my backyard Eli lol
great Eli!
is this concept same as single instance storage? so far it sounds like a exact same thing. Please can someone explain?
Could you talk about Global deduplication across the entire server environment? Thank you...
Lot of duplication in Explaining the De-duplication concept. Just for Fun.
Does the DL160 have an iLO built in? If so and it's licensed, you can use virtual media to install Server 2012. Put the CD in your PC's DVD-ROM drive and mount the drive virtually through iLO. Reboot the DL160 and if it's set to boot from DVD, it should boot from the DVD in your PC.
easy busy :) thanks mate.
Eli, that was super interesting, thank you. One question though: As far as you said, data deduplication works on file level (you talk about folders etc). Would this also be of use if for instance i store a lot of backups from different users / machines eg in disk images? Like - if i have 10 or 20 backups from different windows 7 machines would it recoginze the "similarities" within the disk images or does it strictly work on a file system level?
i'd say dedust man. also why are your books upside down? great introduction though
What are you using for your whiteboard (hardware/software)?
So what if any user deletes the data from the file server ?...
If they have a pointer to an identical file, just their individual pointer goes away. If they have some identical file that other people have a pointer to, the original will be moved to another user, and the pointers will be redefined.
i brought a hp prolaint DL160 Gen 8 it doesn't come with cd
I brought windows server 2012 cd how can i transfer it to a usb
Do a video how to do it
Then do video how to install it on the hp prolaint DL160 Gen 8
I need this for my 200GB's of music..... cuz i know theres so many copies!!
Eli, your drawing looks like someone jump and split his legs & arms, am not saying that is not good I am normal user who learn alof of things from you, personally it's just the most and easiest explanation this way.
BUT if you have add icon's like 4 computers and use the mouse as link then we'll know exactly and more specifically what you talking about to be clean and clear ,,, circles are not like squares.
Anyway it's just a suggestion & still the best of the best, thanks alot man
1080p for me
u look like jason statham
incorrect.... dedup does not work at file level.. it works at chunk level.
+chander prakash Jaiswal No, there is block-level & file level deduplication.
why daaata I thought you were American