How processes get more memory. (mmap, brk)
HTML-код
- Опубликовано: 15 апр 2018
- Patreon ➤ / jacobsorber
Courses ➤ jacobsorber.thinkific.com
Website ➤ www.jacobsorber.com
---
We've all called malloc to get more memory for our programs, but what is REALLY happening when we call malloc? This video goes under the hood and shows you how a computer process (running computer program) gets more memory from the operating system.
***
Welcome! I post videos that help you learn to program and become a more confident software developer. I cover beginner-to-advanced systems topics ranging from network programming, threads, processes, operating systems, embedded systems and others. My goal is to help you get under-the-hood and better understand how computers work and how you can use them to become stronger students and more capable professional developers.
About me: I'm a computer scientist, electrical engineer, researcher, and teacher. I specialize in embedded systems, mobile computing, sensor networks, and the Internet of Things. I teach systems and networking courses at Clemson University, where I also lead the PERSIST research lab.
More about me and what I do:
people.cs.clemson.edu/~jsorber/
persist.cs.clemson.edu/ Note that any Amazon links in my video descriptions are generated by Amazon. If you click one of them and then buy something it helps support this channel. Thanks.
***
Welcome! I post videos that help you learn to program and become a more confident software developer. I cover beginner-to-advanced systems topics ranging from network programming, threads, processes, operating systems, embedded systems and others. My goal is to help you get under-the-hood and better understand how computers work and how you can use them to become stronger students and more capable professional developers.
About me: I'm a computer scientist, electrical engineer, researcher, and teacher. I specialize in embedded systems, mobile computing, sensor networks, and the Internet of Things. I teach systems and networking courses at Clemson University, where I also lead the PERSIST research lab.
More about me and what I do:
www.jacobsorber.com
people.cs.clemson.edu/~jsorber/
persist.cs.clemson.edu/
To Support the Channel:
+ like, subscribe, spread the word
+ contribute via Patreon --- [ / jacobsorber ]
+ rep the channel with nerdy merch --- [teespring.com/stores/jacob-so...]
Source code is also available to Patreon supporters. --- [jsorber-youtube-source.heroku...]
Want me to review your code?
Email the code to js.reviews.code@gmail.com. Code should be simple and in one of the following languages: C, C++, python, java, ruby. You must be the author of the code and have rights to post it. Please include the following statement in your email: "I attest that this is my code, and I hereby give Jacob Sorber the right to use, review, post, comment on, and modify this code on his videos."
You can also find more info about code reviews here.
• I want to review your ...
You should have also mentioned what malloc actually writes into the memory, which is why free doesn't need a size.
Ah, good point. I've been thinking that it would be good to do a more in-depth dive into how allocators work. I'll see what I can do. Thanks.
Your way of explaining things is one of the best that i've found on youtube! Thumbs Up!
Thanks.
Your way of presentation Awesome. Keep doing more videos. I subscribed.
Awesome! I've been looking for a good presentation of this topic for days. Subscribed!
I just stumbled upon one of your videos, and man, I went down a rabbit hole with your excellent guidance! Your explanations are so clear and thorough; Thanks for breaking down complex concepts into digestible bits. Your passion for teaching really shines through, and it's making a huge difference in my understanding. Keep up the fantastic work!
What a crisp and clean explanation! I am impressed man. Thanks a ton.
Holy dang dude, you’re literally the most knowledgeable person about C libraries I’ve seen on RUclips! You should be a teacher or something. Great videos, please keep em coming!
Actually if you click into his profile link, it says he is an associate professor of an univ in the states.
Found this helpful and enjoyable, thank you!
You're welcome. Glad it helped.
amazing information , please more videos on mmap
Clean presentation!
You are my life savior......your every teaching helps me move a step ahead....you came late on RUclips.....but at least you came......love you sir.....
Glad I could help.
@@JacobSorberSharing knowledge is a noble job......and you are pulling this off beyond good....thanks for everything.......
Great brother. Keep on doing more..
please don't stop.
I'm not quite good in English but the way you present the INFO is something new and good.
PS: please don't stop.
Thanks. I'll try to keep things going.
This is a fantastic video. One of the best you made yet. Strangely, it is one of the videos I watched last.
There are two things that I feel have been left out a bit.
First, when you showed that increasing the size of the memory being allocated dramatically made malloc use mmap instead of brk, you did not really tell us why that happened :) (I assume it happened because in the first case, there was enough left from the last page allocation, so all that was necessary was to raise the break?)
Second, when allocating small chunks (in the order of bytes), I sometimes see multiple mmaps (with strace), but it always ends with brk, which means mmap is a higher level command and cannot function completely independently of brk. A bit of explanation would be super useful :)
But regardless of all that, great video!
hello, been really enjoying your videos and learning a lot. i've got a question though: should we use munmap once we finished? kinda like free() is used after malloc(). and if so, is it the same for brk/sbrk (like can sbrk called to shrink or does it have a shrinking counterpart). I've done a bit of strace and found than munmap is in fact called by free() but haven't seen a counterpart with brk sbrk (or something like calling sbrk with a negative number perhaps as two's complement of the initially allocated memory)
I once tried to write an "infinite storage glitch" before I learned that the address limit is not actually the physical limit.
And of course, I tried to write it in python.
I think you will like this video: ruclips.net/video/JcJSW7Rprio/видео.html
Great video
Thanks for the video
You're welcome. Glad you liked it.
Great content, as always. If you can answer me: could you explain how actual RAM dump actually works? I always wondered that bc of the Program Break.
This video is perfect.
Yes, more of these please !!
Will do. Anything specific you would like to hear more about?
@@JacobSorber maybe more about back end programming, processes, strace, signals, buffer overflows and alike.
@@hamamsofiane7098 Ok. thanks. I'll see what I can do.
your profile is awesome
Great video.
Thanks!
Hi Jacob, Thanks for your videos.
I have one doubt here. In first program, void *second = sbrk(4096) ; here your are trying to increment to 4k bytes. But at 3:38 you are saying that "trying to increase 10byes but page size(4Kbytes) is rounded up. I couldn't able to relate.
Can the stack size limit be modified at runtime or compile time ?
I get exact amount of bytes from sbrk. Not 4k aligned.
Upd: I see it's aligned none the less: when incrementing less then a page and then writing into Third+1 memory, it doesn't segfault; but it does segfault when the increment is full page and thus Third is the real program break.
Hello Jacob I hope you are fine .. I just had a question .. how can I get the first and last addresses of data, heap and stack and dimensions for C language in my machine? (Linux)
2:10 According to the official brk manual, the program break is the first location after the end of the process's data segment and the uninitialized data segment. So if we look at the memory layout of processes we see program break isn't the top of the heap. Am I right ?
Thanks for the Precise introduction. As I know current computer has two size of page. Is that true?And how do I know my page size with what command or C code?
You can get the size of standard pages for your machine with getpagesize() on UNIX-style OSes. Not sure how it's done on Windows. As for the other size, I'm guessing you're talking about HugePages, which are usually 4-256MB and are handled a bit differently. Might be a fun topic for a future video.
Can you elaborate when and why this mmap is used?
If one can't read/write data to the space above the Program Break, that means, that he can't access Stack or can he?
I using layered fascia uncle lenny& anty Ingrid
I am full scaled
I'm not understanding the point of the program break, I've implemented a memory allocator before using mmap under the hood to get whole pages at a time and just linked them together with some pointers inside them, I never once used sbrk
Hello Jacob. Thanks for your wonderful videos. I need clarification on the statement where you said adding 10 bytes further using sbrk would round it up to 4k as page size is 4k. I was playing with sbrk and see that adding 10 bytes and then checking the pointer address just shows it incremented by 10 bytes and not 4K, as given in the man page of brk. Is there a better way/example to understand this. Thanks
CODE:
void *s1 = sbrk(0); //to find the current brk
void *s2 = sbrk(4096); //move the brk by 4K i.e. one page
void *s3 = sbrk(0); //find the new brk
void *s4 = sbrk(10); //move the brk by 10 bytes
void *s5 = sbrk(0); //find the new brk
printf("First: %p
", s1);
printf("Second: %p
", s2);
printf("Third: %p
", s3);
printf("Fourth: %p
", s4);
printf("Fifth: %p
", s5);
OUTPUT IS :
First: 0xbe5000
Second: 0xbe5000
Third: 0xbe6000
Fourth: 0xbe6000
Fifth: 0xbe600a
Hi Neetish. What operating system (and version) are you using? Some may handle this differently than others.
And, regardless of your answer. My comment refers to how modern virtual memory works. When you move the program break, it's effectively mapping more pages to the process's memory space. And, since pages are fixed size (usually 4KB), that means that the boundary between Ok and SegFault always increases by a multiple of the page size. Even if the OS implementation tells you it moved the program break by 10 bytes, the real boundary either didn't increase at all (because the needed page was already mapped) or it increased by 1 page (4096 bytes).
I liked this channel and have subscribed it..i have a question can you generate page fault error using mmap??
Thanks, rajan. Welcome.
Can you be more specific? A page fault happens when you access a page in your page table that isn't resident in memory. You pay a penalty because the OS has to move the data into RAM from disk. You can use mmap to request pages of memory. How you use them will determine whether or not you have page faults.
@jacob sorber Thanks for the prompt reply.. i mean to say "major page fault" its an error which occur when page is not found in disk also .
@@rajanjha5797 I still think I'm misunderstanding. A Major Page Fault is when the requested page is not in RAM and the data must be read in from disk (it's slow but it's not an error). A Minor Page Fault is when the data is already in memory, but the page table needs to be updated (much faster, since no disk access is required). If the program tries to access memory that isn't mapped in the page tables, then you get a Segmentation Fault (sometimes called an Invalid Page Fault). You don't need mmap to create Invalid Page Faults. Just dereference NULL or any other unmapped address. I guess a rare case would be when you try to access a page that the OS thinks is out on disk, but then when it goes to the disk it can't find it. That would mean that either the OS messed up or maybe you had a hardware error (disk went bad or RAM failed). That shouldn't happen during normal operation-even with buggy programs.
@@JacobSorber You are right about it ...its a kind of segmentation fault but of different kind i recently came across of debugging such a situation but now i am not able to generate it ...maybe this link will able to help you understand that which i am talking about : smackerelofopinion.blogspot.com/2010/02/kernel-oops-page-fault-error-codes.html?m=1
Debugging page fault error is one of the popular question for interviews also would truly appreciate if you can try to explain such a topic as not much is talked about this over net ... :)
3:40 0xDEAD is my new favorite hex number :)
Yep!
u r just awesome..i found u late..
Thanks. Better late than never. :)
What compiler did you use? Thank you
he uses clang
I have been curious of the Thai books you have on your book shelf, what's that about :)
They're Khmer, but yeah, I lived in Cambodia for a few years and try to keep up with the language as best I can.
@@JacobSorber Your videos have really helped me crack some coding interviews. Thank you
Hi Jacob, I'm experimenting with the same code as yours (using sbrk) to see what will happen if I access an address beyond the program break. But there is no segmentation fault. I'm using Ubuntu 18.04.
Did you use sbrk(0) ? Also, virtual addresses do exist beyond there - the globals and the code. But there is a MASSIVE gap - to allow expansion of the heap. But if you added a REALLY big amount you could end up in that area of the Virtual memory!
Also if you allocated a local var AFTER reading the sbrk(0) - then you may be accessing that local var - which is valid in VM.
So, if I `mmap` some memory as writable and executable, and then put some machine code in it, can I then jump into that code and execute it? :q
Of course, it would be better to change that memory to read-only after I prepared the executable code in there. How can I change permissions on a pre-existing memory mapping?
And how can I actually jump to that code afterwards? Is it enough if I make a fake function pointer type, and then use the address returned by `mmap` like this?:
typedef int (*Func)(int x);
Func *code = mmap(NULL, 4096, PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
/* populate it with some code */
/* change its permissions to PROT_READ | PROT_EXEC - how? */
int retval = code(123);
Answering my own question again :J
The protection flags can be changed this way:
mprotect(addr, size, PROT_READ | PROT_EXEC);
Didn't understand the part where you said sbrk will (ey?) a page even if supplied an argument smaller than page size.
Memory is allocated to processes in pages (typically 4KB). If you use sbrk to request that the program break move up 10 bytes, the kernel can't give you just 10 bytes. If it gives you anything, it will give you a multiple of 4KB. That's all I was saying.
Jacob Sorber so you are implying the sbrk return value will always be a pointer to the beginning of a new page?
I'm not sure about that. Effectively, yes, the kernel will only hand out memory in full pages. But, I suppose an OS could allow the program break to be different from what is actually mapped. What I mean here is this. Let's say you request the program break be increased by 5B. One option would be to just increment it by 5B and then check to see if another page needs to be mapped to ensure you won't seg fault when you access those 5 new bytes. Alternatively, it would just round up to the nearest 4096B and move the break appropriately. I'm not sure what that would benefit you, though. Try it out and see.
what is first and second ? to what data type they belong to ?
They are (void*), so just pointers (memory addresses) with no implied type information about what they point to.
@@JacobSorber what is uint8_t ? can u explain what is fd and offset ?
@@shruthiabirami232 uint8_t is an unsigned 8-bit int. I have a rather old video about these types (ruclips.net/video/iX1uGr6Si0E/видео.html). As for, fd and offset, they relate to memory mapped files. More about that here (ruclips.net/video/m7E9piHcfr4/видео.html)
#include
#include
int
main(void) {
void *first = sbrk(0) ; // will tell where the memory break currently is
void *second = sbrk(2) ; // move the break above/more by 2 bytes
void *third = sbrk(0) ; // again tell where the memory break currently is
printf("the memory break currently is in : %p
", first) ;
printf("moved the break above/more by 2 bytes, new position : %p
", third) ;
printf("trying to use beyond this region will lead to segfault !
") ;
int *ptr = (int *) third + 1 ;
*ptr = 2 ;
printf("%p
", ptr) ;
return 0 ;
}
I wrote this, but it did not segfault, why ?
every tutorial is better with explosions
couldn't agree more.
Holy moly, that's why python and JS programmers are afraid of C!
C is awesome! And totally gives you freedom
FEED BEEF
666