Это видео недоступно.
Сожалеем об этом.
Race Condition Trick /proc/self/fd
HTML-код
- Опубликовано: 20 дек 2019
- In the suidbash CTF challenge there was an inconspicuous line of code, but it was actually there to prevent a race condition. Let's talk about it this video!
Source code: gist.github.com/LiveOverflow/...
=[ 🔴 Stuff I use ]=
→ Microphone:* amzn.to/2LW6ldx
→ Graphics tablet:* amzn.to/2C8djYj
→ Camera#1 for streaming:* amzn.to/2SJ66VM
→ Lens for streaming:* amzn.to/2CdG31I
→ Connect Camera#1 to PC:* amzn.to/2VDRhWj
→ Camera#2 for electronics:* amzn.to/2LWxehv
→ Lens for macro shots:* amzn.to/2C5tXrw
→ Keyboard:* amzn.to/2LZgCFD
→ Headphones:* amzn.to/2M2KhxW
=[ ❤️ Support ]=
→ per Video: / liveoverflow
→ per Month: / @liveoverflow
=[ 🐕 Social ]=
→ Twitter: / liveoverflow
→ Website: liveoverflow.com/
→ Subreddit: / liveoverflow
→ Facebook: / liveoverflow
=[ 📄 P.S. ]=
All links with "*" are affiliate links.
LiveOverflow / Security Flag GmbH is part of the Amazon Affiliate Partner Programm.
Love the upload schedule, but make sure you're comfortable with it. We don't want you to burn out. Keep up the good work!
For those who would wonder how come recreating the file does not change the status into the /proc/pid/fd directory after removing the file, it is because it gets another inode. Something that people should realise is that the kernel is not aware about file name (or path). It only knows about file inode. Since file descriptors are allocated by the kernel, it actually links the file descriptor of that process to a specific inode number. When you delete the file, you delete the inode. When you re-create the file, the a new inode is allocated and so the already opened file descriptors and their links are not impacted.
lcarlier how long does it have to keep track of old inodes once they're deleted to make sure that it doesn't allow collisions if it accidentally assigned an inode to a file that had actually been used before and deleted?
@@RyanLynch1 The way inode are managed is actually file system dependent. In EXT like FS, it will keep the inode in used as long as there is one process using it. So you can't have the inode re-assigned before all the processes have finished to use that specific inode. Once the last process has finished to use that inode, the inode is deleted and then can be reused. That can lead to situation where you can have a bug in application that would create temporary files, delete them, but never close the related file descriptor. In that case, you'll see some "no space left" on the device error coming on that partition because all the inodes will be in used. This can be very tricky to detect because commands like "df" will show that there is still plenty of space available.
Loving all of the new content! Thanks for all the hard work.
This was surprisingly helpful. Binging your videos lately
Another trick for using proc file system that saved me before is /proc/self/exe. A user by accident deleted /bin/bash on the server and a lot of things started to fail because of that, I had a shell still open on the server so I could execute cat /proc/self/exe > /bin/bash and then fix the permissions on the recovered /bin/bash. This was many years ago, on the age of the kernel 2.0.X, I wonder if it still works actually.
Yes, this is still possible with current kernel versions. Why would they remove such an useful feature? :P
Yea, quite useful, you can dump binaries this way easily.
@@HA7DN I wonder if I could get access to "unaccessible" parts of nvidia kernel driver. It would be neat to "steal" data this way.
Daniel I agree that I don’t see reason for it to be removed, but I just didn’t want to think that because something worked on the 90’s it still works today without confirming as I didn’t had a computer available at the moment to not have people “correcting me”
@@WizardNumberNext, since it’s a kernel driver, it does not appear in /proc and all ‘inaccessible’ parts of it are accessible if you write a kernel module to inspect them. The only inaccessible thing about them is the source code.
Love your vids even tho i am still a newbie and don’t understand everything. I will continue to learn
Damn video after video. Awesome work!
I don't know what you are talking about but I'm glad that you uploaded a video
Wow, this was a great video! Very informative, thanks!
Very educative stuff. I love it.
Mind. Blown.
Thanks for sharing.
Very interesting! Thank you!
I like this upload schedule!
Very good explanation of file descriptor. And this is the reason that if you want to remove the large log file, better to use echo > logfile instead of rm logfile :)
Very cool content, you always make my day. BTW, get well soon if you are sick( heard from last video), look forwards to your next good video
Awesome 👍. Thanks man.
This is fascinating! I was always annoyed when I tried to rename or delete a file that is being opened and Windows does not allowed me to do so (it will throw an error such as the file is in use). Linux solves this problem much more elegantly and I don’t know how it works until I watched this video.
nice trick, didn't know that the deleted files are visible in /proc. This knowledge would have been useful when I had a problem with PHP's APC which stores cache info in files in a deleted state. I had no space left on a file system which was virtually empty and had 100% available.
Awesome video! Make one on zombie processes please.
I remember back in the day, letting movies buffer all the way in the flash player browser plugin, and then copying the file from the deleted proc fd that the plugin was still holding open. Speaking theoretically of course.
Awesome stuff !
Still waiting for the nx and aslr bypass videos 🙏
It’s already covered in the binary exploitation playlist and through other ctf video writeups
@@LiveOverflow i completed the binary exploitation for now. Its just awesome. But i didnt see much of bypassing some security mechanism ( nx, canary, aslr..)
I will check the ctf writeup videos.
Thanks mate
Cool to teach the basics to people. The secure temporary files script shell tools were using these tools by creating a file with random name, keeping the fd then removing the file. People that don’t know the basics of filesystems don’t understand that when you remove a file, you just remove it from the directory structure, the file is still there until replaced by other data. The file descriptor is the true reference to a file though. As long as one reference exists, the filesystem won’t overwrite the file. When you understand how a filesystem works, you then can understand the purpose of the file shredding tools used in security, and why they can take so much time. Though for that, you need to understand how data are stored on storage media and know how each storage media works.
What surprised me is that the filename both
a) is reflecting the new name after the rename - which suggests that it's doing a reverse lookup from inode to path (which in turn I assumed was only possible through a full filesystem traversal because filename -> inode is a one way link)
b) stays visible after deletion - which suggests it's not doing a reverse lookup but instead the original path used in the open call is stored as string.
That makes me wonder what happens if you have a hardlinked file that you open through filename A, then unlink that and run ls on the file descriptor. will it then show "A (deleted)" or will it swap over to "B"?
Or even more tricky: If it also has a 3rd hardlink "C", will it then choose B or C?
ls -i
Will show the inodes and provide some tips.
And also you can check lsof
With the content of both videos you could build lsof from scratch
nice!
Greate video i always learn new thing from you . keep the good work fabian
appolgize if i misspeled your name!
I'm guessing that open(path, flags) treats symlinks in /proc/{pid}/fd/{fd} specially - instead of doing normal symlink resolution, it is equivalent to newfd = dup(fd); fcntl(newfd, F_SETFL, flags); except where {fd} belongs to the process {pid}
Hello, amazing content.
Could you do a video about this but instead of file descriptors for files you could use a socket?
Like, could we share sockets between processes?. Or can a web server spawn several processes reading from the same socket?
If what I'm suggesting goes beyond your channel scope, sorry.
Thank you
Yes, sockets are just file descriptors.
In case of a web-server though, you wouldn’t worry about /proc/self/fd since children of the process inherit all file descriptors of their parent. If file descriptor isn’t market O_CLOEXEC such file descriptors also survives exec.
@@mina86 Thanks!
It would also be awesome if we had a video about it.
That is funny. I was showing how handle works back in 2003 and I was not even aware of what is handle (I even didn't know of its existence). It is a bit shame I didn't know about file descriptors in proc as I could have saved a lot of files back then
very interesting. I can't help but think that opening /proc//fd/0 via an LFI could yield valid PIDs and then a loop through FD's could yield some cool stuff
First of, why brute force PIDs if you can just ls /proc? Second of, you don’t have access to /proc/$pid/fd for processes you don’t own so coolness of things you might find is limited.
@@mina86 Because you cant execute commands from a LFI. Its a Local File Inclusion from something like PHP
@@harvuk7729 still, the apache/webserver user shouldn't have access to /proc, and generally will be chrooted, but you may still try, sometimes a webserver is run as root for whatever reason.
Harv UK : it could be possible before kernel 5.1, but it will be too late once everyone has switched to these kernels that have pidfd which remove this possibility. One of the things we can thank systemd for.
I was under the assumption that the fd files where handled as hardlinks, and the limits on hardlinks were ignored by the kernel because its the kernel.
What is the purpose of the swap syscall?
It is interesting that in procfs the files are presented as symbolic links while acting almost exactly like hard links (i.e. as if using reference-counted inodes). The difference is merely that the links cross the FS boundary between the virtual procfs and the target FS. While the latter most likely is an extN variant under Linux this now makes me curious as to how many other kinds of more or less obscure file systems this works with ... ntfs, fat12, cifs?
I don’t know if you can say it like this, but is an open file descriptor similar to the windows files which have that ~ in front of them ?
I wonder if I can use Swift REPL as Python replacement?
Will this work on Windows? since windows doesn't have the concept of file descriptors. This is interesting.
Damn boy why you always sick, take care of yourself! We still want a shit load of videos outcha.
I didn't know about /proc/self/fd, very neat.
What about Windows or OSX thought?
MacOS does not have /proc
@@happygimp0 yes, indeed. many unix-like OSes do not have /proc at all or with vastly differing implementations/features. The mentioned solution is not cross-platform or even cross unix by any means.
What would happen if you read the deleted file via the file descriptor, AFTER overwriting that sector on your hard drive with some other data? Presumably if it's marked deleted, the file system would mark it as free and allow for overwrites. Unless the file descriptor being held open prevents the corresponding physical memory on the hard drive from being marked as "free".
The file is deleted when there are no pointers to the inode. Which means while the process holds it, the space isn't free.
But if you were to copy the data of the inode and use that, you'll read the file, with parts of it which may be overwritten.
How would you solve it on Unix operating systems without procfs? I'm thinking about forking, closing stdin, then reassigning the fd 0 to the file of interest and then `cat -`. Would it work?
With POSIX you might be able to open a Unix socket and "sendmsg" the fd to the target process.
man, u do look like Michael Cera!
Can you make files invulnerable with this though?
so what happens if we change the symlink in proc fd?
You can’t. Procfs is a weird place an those are not real “files”
When the file is deleted from the file system but the handle still exists, where is that still existing file stored? In memory or is it stored as a copy on the disk somewhere (tmp)?
In linux there are what are called inodes, which are basically pointers to "real files" and then whatever path you use is linked to it.
We can assume the block device is a string of bits (and not fragmented).
the inode has a few things:
- which device is it on
- how far along the device is it (aka pointer to the file)
- how long is the file
- how many pointers to it exist.
Deleting a file means removing the pointer to that inode, if there are no pointers to it, the inode is deleted and the space between the start and the end of the file are marked as free
Basically, when you delete a file, it isn't really deleted, even once the inode is deleted, the data is still there. (So if you copy the pointer to the file you can still read it, but some part may be overwritten.)
Note that this is very simplified and innacurate, if you wanna learn more search for "inode" "[linux] file descriptor" and "inode pointer structure"
Basically, there are multiple pointers to the file with various sizes and they are stringed together to form the file, which is why fragmentation is a thing.
wow.
Does that mean, if you open a file it is loaded into memory and the file descriptor points to that memory region? How about very big files?
Thx for your videos @liveoverflow. I really appreciate it!
Think of an encyclopedia, you load the line at which the file is stored in the index, and then when you want to read the file, you look in the index which points you to which page it is in different volumes.
If someone "forgets" their line number to the index, you still have yours.
if no one has a number to that index node, then the page numbers are moved to a "free space" section in the index, which basically means the file is deleted, but you could still search the whole encyclopedia for parts of that file, though you might find some parts have been overwritten
Ok thanks. I think I got it. The filedescriptor is a copy of the file's position in the filesystem. By deleting the file, you delete that information in the file tree but still have the copy. Guess the os is preventing that region to be overwritten as long the filedescriptor exists?
@@llGr4Yll yup, basically, the kernel won't overwrite the file till it's marked as free, and it is marked as free when the inode is removed. Note that it doesn't prevent stuff that accesses the raw data from overwriting (like the classic dd -if /dev/urandom -of /dev/sda1, which is rm -rf / on steroids.)
Also directories are simply files with a bunch of inodes with a name, and as they have their own inode, (on older kernels) you could have a directory be it s own parent.
So after removing the file, it was still in RAM? Or was it still read from the hard drive somehow?
it was reading from the hard drive. the space a file used isn't made available for reuse until the last file descriptor referring to it is closed.
It doesn't matter if the file is still buffered or read from disk. The important thing to know is that unix filesystems separate paths from inodes. As long as you have a reference to an inode, you can still access the file. And /proc/self/fd/* references the inodes. Somehow the symbolic link is misleading, similar to kernel namespaces: these also appear as symbolic links, but are similar to hard links, as they control the lifetime of Linux kernel namespaces.
Cool videos about security... BTW in c you try to avoid to call exec* calls..
Why? If you need exec you need exec
@@LiveOverflow yeah that's why I said "you try to avoid" ... For example more easy is to make a read_file().. but this kind of videos are excellent.. I don't see any channel with this content..thanks.im subscribing right now.
@@matiasm.3124 wtf are you talking about?
@@matiasm.3124, as often is the case with examples, this was a contrived example. Yes, in this particular case making a simple loop to read the file might be better solution, but it’s not possible in general.
While the file is deleted from the filesystem, but the file-descriptor still exists: the data is still physically present on the drive, right? Can it happen, that by creating new files, these new files will be created at the same physical location, therefore overriding the first file? Or is this space still somewhat "protected"?
The inode for the file still exists and the file system treats it as an existing file. There are just no links to it. As such, creating a new file will never overwrite such open-but-deleted files.
@@mina86 Ah, good. Thanks for the answer!
dup2(fd, 0);
;)
I'm so scared by race conditions...
Nice trick, but there is one thing I don't understand: if we call cat with "/proc/self/fd/xxx" won't cat look at the wrong process? Like, won't it access its own /proc/ dir instead of ours?
Hey liveoverflow..
For your next video try to reverse destiny 2, if you want
well that's illegal and way too much content for one video
@@halbgefressen9768 it may be illegal, but it is a pretty interesting challenge if you ask me, even just acessing assembly would be considered a success
Which Os and kernel version you are using for this exploit.
Does Windows have anything even remotely similar to this?
Windows does this the exact opposite way: While a program has a fd for a file, that file cannot be deleted or renamed.
And you can thank that backwards method for the reason why you are forced to reboot Windows so many times after simple updates! Woo hoo....
Dreamagine1 how so? I don't think I fully get it the other way around (or even entirely understand the way Linux does it)
@@RyanLynch1 Let's take an example:
You want to update the command line processor (/bin/sh on Linux, cmd.exe on Windows). That file is in use by all running command processes, including the one that wants to update it. On Linux that is no issue at all, you delete the file, copy over the new file and are done. Every new process will see the new file, every process that's already running will still have the old file. On Windows you cannot delete the file because it is in use. So you make a special system call "replace file at next boot" and give it the new file. Then you reboot the system and Windows will replace the file during startup.
(Disclaimer: simplified, duh...)
astyle --style=allman --indent=spaces=4 --convert-tabs --indent-switches --add-braces ./*.{c,h,cpp,hpp}
Sorry, but the code formatting really bugs me :(
(The format would still not be perfect with that command, but much better)
Nothing wrong with 2 space indents. Saves horizontal space.
@almightyhydra
I do not mind 2 spaces, the things that bugs me is that the style is not the same everywhere and that open and closing brackets are not on the same vertical or horizontal line everywhere.
astyle --style=allman --indent=spaces=2 --convert-tabs --indent-switches --add-braces ./*.{c,h,cpp,hpp}
or
astyle --style=gnu --indent=spaces=2 --convert-tabs --indent-switches --add-braces ./*.{c,h,cpp,hpp}
would also be fine.
First comment
all im hearing is racecar
Erster
gz
no