LSM trees (Log Structured Merge Trees) - Detailed video

Introduction to LSM trees, their implementation and the concepts involved.
Please drop down any questions that you may have in the comment box :)
References :
paperhub.s3.ama...
en.wikipedia.o...

[@elliotnixon8138]

Wow..never thought about the database optimisation from write and read perspective...always thought about query optimsation
Thanks a lot for providing such a useful information!!

[@gltechtutorials3337]

Thanks a lot :)

[@m113able1]

Have been trying to find a decent explanation of LSM trees and stumbled across this channel purely by luck! Channels providing CRAP content are the reason that channels providing authentic content such as above cannot do better and remain hidden from the reach of folks seeking knowledge.

[@rchintha]

Explanation is beyond crystal clear. I highly encourage the presenter to author a book to demystify and simplify such tough concepts.

[@gltechtutorials3337]

Thank you!

[@vijayrajasekar3351]

Good Video, it is realy a video of a full nosql architecture.
You mentioned you are working on lsm tree, may I know the project name?

[@哈吉-d2d]

I would love to have your email please

[@logicboard7746]

genius! very clear...please consider making more videos...subscribed

[@voila920]

Very good explaination .. thank you

[@ujjaldutta1716]

I think, in the compaction slide the worst case read time should be NlogM where N is the num of SSTables and M is the size of max SSTable length in your e.g. N=2 and M=5

[@kovidasurampudi1411]

Thanks a lot for simple explanation of complex topic. This helps me understand a research paper.

[@AbhijitRanalkar]

Good Video there.. explains the things very clearly.

[@shivanshjuyal9463]

Excellent. Really helped me understand a paper regarding LSM based storage techniques

[@DaviBrg]

Excellent tutorial! Thanks for uploading it!

[@chirag_soni_07]

really a great video, was troubling understanding LSM tree, and there's not much stuff available online too. But thanks ma'am, you got me.

[@whoispankaj]

but why would you read both SSTables instead of just the SSTable with the latest timestamp

[@gltechtutorials3337]

There is no definitive way of saying a key will be present in a SSTable. Sure, we can start searching from SSTable having the latest timestamp but the key may not be present in that and we would then be required to search the other SSTables

[@shivagarg9458]

Thanks for crystal clear explanation. This has cleared my HBASE architecture concepts.

[@gltechtutorials3337]

Glad it helped!

[@pkuLoki]

very helpful. thanks

[@AbhishekVaid]

WAL mentioned at 16:30 means write-ahead-logs. Another name for it is journaling and it's present in most DB and filesystems.

[@ameynaik2743]

ruclips.net/video/oUNjDHYFES8/видео.html Shouldn't it be O(K*logN) where K is number of SSTABLES and N is size of SS table?

[@naveedhassan7399]

Excellent video. Only one point , Cassandra has a cache layer(Row cache and key cache) before reaching out to Memtables, which considerably improves read performance.

[@ameynaik2743]

Here ruclips.net/video/oUNjDHYFES8/видео.html don’t we look in lastest SSTable first and not both the SSTables? We shouldn’t be looking into all SSTables. Could you please confirm this?

[@ameynaik2743]

Can you please throw some light on what does in-memory index hold (shown here - ruclips.net/video/oUNjDHYFES8/видео.html). Thanks

[@haaarshiiiit]

Excellent content!
Thanks for taking the time to make this and share it with everyone ! 👍👍👍

[@gltechtutorials3337]

Glad that it helped :)

[@ramamoorthysrinath5500]

Good , Uncomplicated explanation !! Perhaps add some examples of realworld applications that require Low write latency and why etc.

[@gltechtutorials3337]

Sure! More topics incoming on LSM trees soon. Will add more examples there :)

[@NikhilSharma-gk4ge]

Great explanation concise and accurate.

[@aditya789101]

extremely good. Thanks a lot!!

[@dheemanthmallikarjun514]

Kudos! Amazing explanation 😀.

[@sunilmadan]

The concept explained very well and in a simplest possible way. Thank you, nicely done 👏🏼

[@glsruthi6522]

Nice explanation, God bless you

[@rithisharithisha4210]

It's very easy to understand and it's very clear. Thank you it helped me a lot to understand this topic.

[@gltechtutorials3337]

Glad it helped you :)

[@sirimeghana6640]

Very clearly explained 👏👏👏

[@gltechtutorials3337]

Thanks 😁

[@remram44]

The only way "2 * log 5 = 6" and "log 6 = 3" is if you use base 1.7 :-P

[@blumki]

Nice explanation

[@gltechtutorials3337]

Thanks !!

[@mankritsingh4058]

Well explained tutorial🎉

[@sunilkumar-dq3fo]

Awesome explanation

[@gltechtutorials3337]

Thanks :)

[@ranhu2312]

is mongo based on b tree (not b+)

[@dubeyankur83]

Nice !!! Thanks.

[@groundtouche]

Why we need both compaction and Bloom filter? IF compaction is already reducing state to single non redundant set, our read would search only one SS Table..isn't it? Or is it that we want to avoid I/O's on disk and perform better using Memory?

[@gltechtutorials3337]

The compactions don't always keep on running. There are generally 2 types of compactions - minor which clubs only some SSTables per CF into one SSTable and major which clubs all the SSTables per CF into one CF. Minor compactions run frequently while major runs only once in a couple of days. So we need to bloom filters to avoid disk I/os for faster reads.

[@manishsakariya4595]

beautiful

[@sajaybondadi78]

Nice akka

[@gltechtutorials3337]

Thanks Dheeru 😁

[@upadajanki8447]

Explained clearly. Can you also explain how update works in LSM tree based databases ?

[@gltechtutorials3337]

Sure, will make a video on it next :)

[@chiranjibghorai6950]

Excellent

[@wennwenn1422]

Perfect. More content pls. Thank you

[@gltechtutorials3337]

Thanks, tune in for more content this weekend on LSM trees :)

[@BinaryIgor]

The main drawback I see here is that the more data you have in your database the more time and resources will be required for the compactor to merge/compact SSTables. If you go above certain size it may not be feasible anymore. I wonder if there is some kind of optimization/trick for that ;)

[@sasisharan9743]

Great video :)

[@abhinavsharma-gy9mi]

Very well explained ma'am

[@lkri7951]

Nice video, would like to develop a timeseries db with these concepts, whts your 1 year background on using lsm tree?

[@gltechtutorials3337]

oh cool, all the best! I worked on Cassandra internals :)

[@harjeetkajal]

Thanks very helpful !

[@官军兄]

Thanks for helping understand LSM-Tree.

[@ameyapatil1139]

Wonderful explanation ! ( note - mongodb does not use lsm tree )

[@gltechtutorials3337]

My bad!! Meant RocksDB not MongoDb. Thanks for pointing it out :)

[@MrStile97]

Thanks for explaining!

[@gltechtutorials3337]

No problem!

[@kavithanjali123]

This video saved me. Very good explanation :) Could anyone please how to perform delete (key) ?

[@gltechtutorials3337]

Glad that this helped you Kavitha. Let me explain what happens in case of a key deletion.
When a request comes for key K1 to be deleted, it is written with a special marker called tombstone into the memtable which may later be flushed into a SSTable. So, now if a read comes for key K1, we see that the latest value written for K1 has a tombstone marker set indicating that the key was deleted. Hence null is returned for that read. Eventually, as the compaction kicks in, all the keys having tombstone markers are deleted by the compacting thread.

[@kavithanjali123]

@@gltechtutorials3337 Thank you so much for the explanation :) So, It will flush to the SStable from Memtable immediately or it will wait in memtable till Memtable reaches its desired size?

[@gltechtutorials3337]

@@kavithanjali123 It will wait in memtable till Memtable reaches its threshold. This process is as usual.

[@maksimpashkov712]

Thank you so much for this video. I like it. It really helped me to understand LSM trees.

[@gltechtutorials3337]

Glad that it helped you :)

[@SatishKumar-jb9qm]

Very well explained. Thanks for sharing.

[@gltechtutorials3337]

Thanks :)

[@gabrieldjebbar7098]

Awesome explanation !!