A very insightful video. Just had one question around repair. Once we find a corrupted log wouldn't the data be insconsistent even after the repair? I am coming from the place where I have to use repair for a DB recovery
The repair is essentially discarding all data that it cannot read. It preserves entries up till the last row of data that the system is able to read successfully. We work on the assumption that all data which we can read is correctly stored in the WAL (Reinforced by the checksum). The recovery mechanism is on best effort basis, and attempts to retrieve all data that it can successfully read.
Great video and git repo. I had one question though, before writting the entry to the segment file we are checking if rotation is needed right? But we are calling the rotateIfNeeded() before adding anyting to the writer buffer. So when it checks for a overflow of file size then the wal.bufWriter.Buffered() return 0 as the buffer is empty because entry is being filled after the rotateIfNeeded() call?
For eg if we have a log segment file lets say "f" filled with "x" bytes and assume that f is the last segment file written to. We have the file limit of x+1 bytes. Which makes 1 bytes yet to be filled. Lets say we boot up the system and write a log entry of say 3 bytes. Now, the buffer is currently empty so the rotation check of f.Size()+int(bufwriter.Buffered()) would still be x bytes which would be less than file size limit. But we wouldnt have space in the file would we?
Nice walkthrough Jyotinder! Thank you for the video!
Great video please make more videos on low level stuff
Great video on a topic I was curious about. Thanks!
What a great video! thanks for the clear explanation!
Thank you!
Great stuff! Thanks a lot! Please keep posting such videos with such clear explanation. 👌
Thanks! More videos and articles coming!
A very insightful video. Just had one question around repair. Once we find a corrupted log wouldn't the data be insconsistent even after the repair? I am coming from the place where I have to use repair for a DB recovery
The repair is essentially discarding all data that it cannot read. It preserves entries up till the last row of data that the system is able to read successfully. We work on the assumption that all data which we can read is correctly stored in the WAL (Reinforced by the checksum).
The recovery mechanism is on best effort basis, and attempts to retrieve all data that it can successfully read.
nice explanation
Great video and git repo. I had one question though, before writting the entry to the segment file we are checking if rotation is needed right? But we are calling the rotateIfNeeded() before adding anyting to the writer buffer. So when it checks for a overflow of file size then the wal.bufWriter.Buffered() return 0 as the buffer is empty because entry is being filled after the rotateIfNeeded() call?
Like shouldnt we first fill the buffer then check if the file size would exceed? Or maybe im wrong about the Buffered() function's working?
For eg if we have a log segment file lets say "f" filled with "x" bytes and assume that f is the last segment file written to. We have the file limit of x+1 bytes. Which makes 1 bytes yet to be filled. Lets say we boot up the system and write a log entry of say 3 bytes. Now, the buffer is currently empty so the rotation check of f.Size()+int(bufwriter.Buffered()) would still be x bytes which would be less than file size limit. But we wouldnt have space in the file would we?