I Made a FAST File Explorer (in Rust)
HTML-код
- Опубликовано: 25 июн 2024
- Project Repository: github.com/conaticus/fileexpl...
Join this channel to get access to perks:
/ @conaticus
Discord: / discord
Github: github.com/conaticus
Twitter: / conaticus
0:00 Basic File Explorer
0:56 Simple Search Bar
1:20 Search Optimisation
3:26 Result
I Made a FAST File Explorer in Rust 
Развлечения
Now write a whole OS
thank you so much
Yea, Would love to see that
Yes please!
And make it Android, GNU/Linux, MacOS, Windows, FreeBSD, IOS apps compatible
This comment gives hard ‘Do a flip!’ vibes.
You attained such speed because your "cache" is just reading a file of paths, the real and realistic time is ~50 sec like you showed since there is no cache invalidation. But hey, enough to fool most apparently
this is tik tok content stretched out to 4 minutes for no reason, every video this guy has made are superficial projects or stuff that he just gives up half way through
gpt "programmers" are easily impressed
In practice this is more useful if you're looking for multiple files though. I wish windows would keep a cache.
Sure it's a cheat, but it doesn't invalidate it being useful. It just changes the acceptable use cases
@@CatMeowMeow as I'm modifying and removing files too often it's useless for me, luckily visual studio code has a pretty powerful search system
@@gonzalologwouldn't you be able then to use windows file events to update the cache?
Yeah... But what if your file isn't cached? Does it just not appear?
You should do an analysis of the Windows search, indexing and all. Then see where it leaps ahead.
If the file isn't found in the cache, it must be in a modified or new folder, so all you need to do is not check where you know it isn't, based on the modified folder time and the cache update time, and whether it was in the cache or not before the search.
Or something like that. Comparing this to Windows search isn't really fair, as it searches file contents as well, like txt files, etc., idk.
@@johnmarston2474 this is exactly why I'm saying you should research the system you're saying you're improving. It's not fair to say you made a fast Explorer if you implemented it and missed a massive part of how it works.
Windows search might be slower but it checks file contents and thus they didn't actually make the same functionality but fast.
They could do a fast search and then do a file content search, a sort of two stage search. That would make more sense for this title.
@@johnmarston2474oh so it's like that meme explaining how the missile algorithm works.
@@StiekemeHenk linux file explorers are still faster, and they also index content
A file explorer is a pretty good project imo because it's easy enough on a basic level but it also can be made significantly more complicated if you want to torture yourself. Optimizing stuff like this is always very satisfying as it's what us - programmers - are paid for. I would love to see this project with a better UI, i think it's worth it.
And finally,
"wOw" - conaticus, 2023
I started playing with similar project several years ago. Now it is almost done, approaching 1.0 release... Just google for HiFile file manager... And I can confirm that it is really a lot of work if you want to have your file manager feature rich, crossplatform and supporting archiving! :)
@@vladimirkraus1438 I just searched for it and didn't find the repo. Link it, please?
@@vladimirkraus1438 but is it fast?
@@vladimirkraus1438 That file manager looks really good!! Congratulations!
yeah, the next level for search is to implement it in a way that it also knows about updates to the disk and you can have multiple file explorers open at the same time
afterwards you want to add support for finding stuff in the file contents
The explorer search does more than look at file names. It also does some content reading.
I'm not sure that's completely true, but it doesn't really matter since the search barely works.
@@IshCaudron It's true, look at the search settings: there's an option to enable content searches, which explicitly warns about performance degradation.
@@IshCaudron it's slow indeed, because the indexing feature is not applied to the entire drive. In your implementation you're basically caching the entire tree. This can also be done for the windows search, but you have to manually set it that way, because indexing comes at a cost. If you're on Windows 10 or 11, you can look at the indexing settings in the start menu.
@@minciNashu not slow, I'm saying it doesn't work. It doesn't find things on the drive even if I have an explorer window showing them. It's just broken.
@@IshCaudron file explorer search based on indexing. if your file doesn't get indexed, it won't get searched. regardless whether the file exists or not.
The speed is honestly insane compared to the usual search! Great job man!
@@RadinTheFroggta 6?
Does it also beat Everything? I am asking this because if I am right that application does raw NTFS searches, without indirectness API of the Windows operating system in between, making it practically unbeatable, unless the application in this video is doing the same thing.
@@jongeduard Everything caches the data too so the result should be ~the same.
@@jongeduardit's just cache and there is no invalidation
find / | grep "your search" is faster lol
If you enable windows search indexing (enhanced mode), searching via explorer or start menu works amazingly fast.
Wanted to say the same. Like querying your database without an index, just going from beginning of storage to the end of storage looking for whatever you're searching for lmao.
What a lovely thumbnail, the group that designed it have to be quite tallented :O
100%
It's such a shame that they'll never see our appreciation
Yo super cool project. BTW there's something made just for this called as "Everything" which also searches your entire file system faster than windows could ever imagine.
i use this every day also power toys has this feature too
Yeah, he could have just written a explorer with everything integration. But his results where pretty impressive as well while utilizing a similar approach.
But I would say it's not possible to search faster than everything as it literally adapts the result by the keystroke.
I actually installed it on computers of all my friends when I just wanted to search 1 file as it was faster to install everything than search it in any other way. Those who understood it's power continue to use it for everything 😉
SSD users can just use dir.
@@TariqSajid what? did you mean powertoys run? or is there something like everything inside powertoys
@@BarraIhsan yes i was talking about powertoys run
I should work on a file explorer, seems like a fun project to do. Great job on how fast it was
Such great content. This is so cool and interesting. I need to come up with smaller project ideas that can be tackled in a few days like this one.
This is awesome! I've thought about doing this for a while but never got around to it haha
that's pretty awesome lol, makes me wanna do something similar
This is awesome. Great stuff!
This search is not fast because of the Rust programming language. Rather, it is fast because of the data file cache data structure you are using and the algorithm you are using to access it. Searching for strings inside a single json file will be way faster than running file search queries across a file system on a hardware disk for sure. Windows should be able to achieve similar or better results. I have no idea if the search is using the windows index or not as it can ve turned off
He just cached everything and searched there, but there could be a case when cached data is old and not up-to-date (some files/folders could be already deleted or renamed, new files could be already created at that time). So there must be some syncing that runs every n minutes/seconds. Am I right?
@@ulanaitbay afaik there are "watchers" that allow your program to know when a file/folder is modified, one could use that to add/remove cache entries dynamically
@@ulanaitbay You are indeed correct.
@@m.projects watching every single file on a drive is very inefficient.
Thanks!)
This really inspires me to try Rust ! Thanks for sharing and good trail and error!
Good job lad, you got a subscriber. Hope this goes viral.
Great work!
what happens if a file is created after the cache? does it know to recache?
also i think windows search is understandably slower than this since it searches for non exact matches (like if i search for eclipse it will show me eclipse-workspace too), and it doesnt have a cache since it needs to find recent files.
so basically yours was like a bit faster but then you added caching, which in my mind is kind of pointless since it constantly changes, but i guess it helps.
As was shown, substrings are also searchable. The cache doesn't change much. I believe the hybrid approach is the best in this case: first load the cache and ask the filesystem if the file still exists. If it does, congrats, you found a match. If it doesn't, delete it from the cache.
And in parallel the slow search should happen which will re-evaluate the cache
@@phoenix-tt true i guess the hybrid solution is best
sorry i missed the substrings being searchable
this video is pretty cool though its inspired me to make my own in rust
@@lythd Yeah, it's a cool task, good luck mate
@@phoenix-tt thank you marcel
Nice work!
Rust developers in nutshell: I don't get how turn on Windows indexation, and it's a reason to write it by myself from "scratch" and bunch of libraries...
Windows indexing is shit. It takes loads of performance in the background and can't handle large amounts of file changes. If you try indexing an entire disc you can literally trash the pc.
Use "Everything" there is literally no discussion on what the best file search is. It's Everything period.
@@redcrafterlppa303 Yep. Windows indexation a lot worse in way of metafile size per file, and content indexation by default as i know. But as "From a box solution" it's enough, especially on "modern" hardware
@@redcrafterlppa303most pc's today can handle a bit of indexing
this is only practical if you are planning on searching stuff in bulk...
its undeniable that the windows search is slow af but comparing it to your caching aproach is pretty like comparing apples to oranges.
windows search is used after an unknown ammount of changes were made to the file system meanwhile in your approach you need to re cache everything whenever you make any change so the boot time of the program is similar to a normal windows search (please correct me if i m wrong, i have no clue what your caching times are since you didnt really show them) so yes your code is faster id you need to make 2+ searches but for what people usually use it for its pretry useles
Windows search is still bad, Everything (by voidtools) caches things and updates immediately and can update the results as fast as you type
Probably worse than windows search, as that is designed to index the files in the background when you're not active so as to be as unnoticeable as possible - enabling indexing for your entire drive is probably a much better solution
@@CorrosiveCitrus Even with search indexing on, explorer takes at least a minute to find a file even when I give it the exact name and the start menu takes 10-30 seconds, and much more if the file is in multiple sub folders.
I don't know how the explorer search manages to be so bad, Everything can index my entire computer faster than explorer can find a file.
@@daviddube9215 yeah, explorer somehow manages to be slower than just brute force searching it for every new term
give the kid a break. He is learning and this is great initiative. He is prob gonna be a better problem solver than 99% of engineers.
Forgot to add that the Rust Project did not review and approve your work and does not support it.
lmfao he better get rid of the logo on the thumbnail xD
This is cool, but can it be set to just update the cache every now and then? Perhaps on the creation or movement of a file? If a file gets deleted while the explorer is running, will it notice the file no longer exists when you run a new search in the same instance of the explorer?
I would like you to note that the windows file explorer's search function is a little more complicated than that, as it searches through the contents of the file as well. It looks for keywords in the files, or uses certain search terms to match with the file name. It's not just a simple .includes/.contains match.
Great video, and I found it really interesting :)) Also, you became a victim of a premature optimization with your BK-tree :P
As far as I know, Windows' search is significantly slower as it also searches file metadata for supported file types. There are apps that implement pretty fast searching using indexing as well though, like voidtools' "Everything" tool is blazing fast if you give it the time to index all your drives, and it's pretty good at picking up changes in your FS almost immediately and indexing those changes.
yes!!! more rep for the best search tool
and the reason Everything can pick up changes so fast is because it reads the changes journal of the filesystem directly, which you can see by doing Ctrl+` (backtick) (closing the debug window will close Everything, do the keybind on the search window again to hide the debug window)
Amazing 🤩🤩🤩
You're absolutely inspiring and smart
Pls do more videos like this
Linux Users (like me): "Lol, we just use the terminal! And our GUI explorers are fast anyway! XD"
love dolphin
@@xvnexus8814 Dolphin is good.
Next: I made whole GNOME Desktop Environment but faster (in Rust)
great project
thanks for uploading videos
Glad to know that the file search is not slow only on my computer. Honestly, how come Microsoft never fixed that? Windows after windows and it's all the same, you have to install third party software if you want to find files in your computer.
Because there's nothing to fix. It's a raw-dog search, not an indexed search. Nobody fails to understand that indexed searching is faster than unindexed searching, but indexing operations are expensive and you don't want to be running them on your system nonstop for ages.
If your files are constantly changing, indexing can easily be out of date, and you then lose the benefit of indexing because you have to re-index before every search, which, surprise, is basically what windows search does.
There is a "fix", just go to your start menu and search for "indexing options" and turn on indexing for whatever folders that you want faster search in.
And as @@Sammysapphira mentioned, indexing is expensive, so it will take a while for windows to build a full index in the background. But when it is done, searching in that folder is usually under a second.
the fix is to install "everything" and search using that instead
@@SammysapphiraYeah if not for the literal fact that Windows already has indexing (at least partial) for other features and a boatload of "maintenance" or otherwise labeled services that spin all your drives at random, even when not needed at all for the current operation.
Enabling full disk indexing will be answer
It's not enabled by default because it's taxing to the system and is supposed to I my run while you're not active, so needs to complete in a short time, but with a decent enough pc, or one you leave on, it should work fine
This looks like a really fun project to put together, but as others have said there are some comparisons to be made. Windows explorer works without extra disk space, works on even the slowest of drives, and takes no extra boot time even after thousands of file changes. For this program to work at full speed, you'd have to have an insanely fast and empty disk to begin a cache and be ok with it loading/flushing that cache every time. Most people who use the search feature dont have such hardware which is why Windows had to sacrifice performance for full combability with all hardware.
that sounds fun, I wanna do something similar
This is absolutely beautiful
Quick question: how do you handle cache invalidation? What happens when the filesystem changes?
Good question! I was trying to handle this with the notify crate. Sadly doing that asyncronously went past my Rust competency and I didn't have time to implement it 😅 It would look something like watching for a new file change, updating the cache in memory and then updating the cache on disk every so often
that moment you use your whole cpu/gpu and ram to just find that 1 file on your harddrive.... i love it
How do you detect when the cache becomes outdated without scanning the whole filesystem whenever the program opens?
ur my fav youtuber now.
This is actually so sick. I need this so much in my life. Please keep updating this
Look up Voidtools Everything :)
It's not really a file explorer but it definitely finds files quickly
you don't even know how happy I was when I saw that you're using CLion, not a lot of creators use it, but it's such an amazing IDE
Love your vids
These videos are pure QUALITY!
Microsoft, please hire this guy.
Thank you for making this file explorer! Now I wouldn't have to wait 7 minutes just to find one file anymore.
everything is a... thing and it does seraches by reading the ntfs data directly.
much faster then this.
@@v0xl just heard about it
Try enabling full disk caching for windows search, they have it disabled by default even if you have a powerful pc
How much time caching is taking? What if you make caching in different thread?
Does the cache alter it's data when a new file is added or file locations are changed?
And tomorrow, Spacedrive, a file manager that also uses Tauri, should finally be released. Great video and nice timing xD
I see the Tauri logo! Actual walking W
I use this app called everything
I require this searching speed
Nice that you used Tauri! How was the DX for you?
Hi i love your videos love it
cant wait the official release from you and i only need to execute the installation
What microphone do you use? Also, what post processing are you doing?
But how do you update the cache to catch up with filesystem updates?
One improvement you could do for NTFS file system is use NTFS USN Journal. Everything already uses that and it's insanely fast!
Other than that, great project, good work!
I think putting "Everything by Voidtools" would help provide context here
What about cache invalidation. What if we edit file delete or create from another application like Visual Studio, will the data be updated in the cache
no
A good example that windows can be optimized WAY more than it currently is, but M$ being M$ hardly gives two shits about it
Honestly a clean FM using the Everything SDK for searching in the background would be my preference.
How do you have that A: drive? I thought A and B are assigned only to floppy disks and now are just a part of the windows history
Something you could do to account for changes in the file tree is have it redo the caching when it finds nothing. Only thing you should keep in mind is to show the file that's found during the caching immediately, not after the caching is done.
Wise Jetsearch is an amazing tool to search multiple drives at the same time under 10 seconds
wait till he finds "everything"
rust programmers be like: "so I found this library that does everything for me" and posts it as a video
This guy is officially a chad
Alright underrated channel I just found!
are you storing the cache in plaintext?
I will show this video every person which will say that rust is over hyped. Great work.
have u tried turning on indexing
Finally a project which makes use of tauri on RUclips. We need more tauri fr
if the file is deleted xD tho and ur cache still has it?
There's a really good program called "everything" that I use to search for files normally. It's been my go-to over default windows search for years now.
"so I just spent the entire afternoon looking at fancy searching libraries just to find that I can just do this with a hashmap" hashmaps are in fact so overkill sometimes it isn't even fun to use them
**unsatisfied trait bounds** ahh the true rust experience
I was always using vscode files search (Find in files), instead of windows, as it does support globs (I always using them). Would be interested to compare these (Hope vscode as fast as possible). You would better know how often I need to do a glob search while browsing directories in explorer
There is those two things fzf/bat but i guess you're looking for mouse click solution.
This is really good but you need a background thread that’s constantly updating the cache
how much time was it to cache?
Does it manage cache invalidation properly?
was the windows search indexing turned on?
Kind of funny that you basically recreated the unix utility “locate” (more easily googleable by it’s indexing command “updatedb”)
Best music!
It appears you also have a mapped network drive which would increase the search time also
How are you like "connecting" the NodeJs and Rust?
That's interesting. Is this filesystem dependent? I.e. would this work with common Linux filesystems as well? I'd assume this isn't faster than ripgrep.
But I'm interested in building cross platform desktop apps with Tauri, specifically something that can search through a lot of text very fast.
ripgrep isn't intended for this. If you're talking about GNU find or sharkdp's fd, possibly, possibly not. I'd wager that fd's far faster if there isn't a cache entry though. There's also locate (and plocate, mlocate, etc.) that should be fastest of all because they use a cache for this and are specifically made this, some of which are always (assuming you only use the accessed mount points with this one system) up to date by virtue of registering for relevant filesystem changes with the Linux kernel. And if by ripgrep you meant searching some other program's filesystem cache rather than just using that cache, I'd wager that the more specialized (and presumably non-regex) parsing they'd use for the cache would beat out ripgrep, especially since much of ripgrep's remarkable speed comes from the sheer amount of parallelism it chucks around, which it can't benefit from nearly as much when it's only scanning a single file (and furthermore biases any parallelism benefits heavily towards the creating program since it'd likely have more information about where it can send stuff to be done with other threads according to the cache file's structure, unlike ripgrep which obviously can't).
The Linux equivalent to what he built is mlocate. And I've never seen it take any longer than "instant" to find a file, so his program is still way slower if sometimes it takes 1s.
I don't see why this wouldn't work on a Linux filesystem however the program sadly doesn't support Linux just yet. Would be achievable with Tauri though!
The problem with this implementation (as far as i know) is that if you want to keep all files findable, it is impossible to rely on the file cache alone. If you download a file and haven't cached the filesystem after that, you won't be able to find it, so you will have to cache very, very often, which would slow down the system.
Another option would be to use the cache to show those results first (which would work most of the time) and then keep looking for files that match the search that are not cached, so at first it would quickly show you the files that are already cached, and after about 50 seconds show the rest (it would be an improvement compared to Windows Explorer, but in the base, in certain cases it would be the same).
were you running in release?
Yay. Fix what Microsoft cant
It is really funny that windows explorer is literally the slowest file explorer possible. Try deleting the js 'node_modules' folder in it vs in terminal. The difference is sometimes in minutes.
@@phoenix-tt I just use wiztree and wizfile. Literally the best file searchers/drive indexers out their. Wiztree is especially good at showing huge folders of junk that you should delete and wizfile can find any file in less than a second. Its fucking insane how a multibillion dollar company cant at least improve one of the main tools of windows instead of piling and piling more shit ui changes that can be pulled off to reveal windows xp. Im so frustrated that i rely on windows for most of the apps that i use. Linux will always be king.
@@phoenix-tt If you add the cache indexing time to every search then the speeds are almost identical. There's no change here. Caching is just a "cheat".
more ctrl-z for notepad ?
you also could just use "Everything" and "Explorer" together (Everything is like a super fast search engine for your files)
I was about to say this lol
That’s a good idea for a project, the gui was done using Tauri right ?
there are excellent file search softwares like everything from voidtools
Okay so what happens if a new file comes in or gets removed?
and if you delete a file? does the cache update? ...
You don't understand, Windows cares so much about user experience that they know the average user is gonna want to go to the kitchen and grab a cup of coffee while they are using file search so they make it slow on purpose.
You've taken all the fun out of searching for a file.
When is the cash invalidated? Because after changes on the drive it would need to be updated.
I have 12TB, does it mean it will cache into a big, very big file and memory usage? I'm using voidtools' Everything tool and it consumes 700MB for the caching, of course, ignoring node_modules folders, .git, windows, program files, etc. and I guess Everything uses a special way to list files in the NTFS (file system) USN Journal, which speeds up the caching process.
Does this do anything other than search for a file??
1:03 -- The... ...A: drive? What, you have a floppy disk drive on your computer?
Yesterday I also starting write a directory file indexer library for indexing all file from sveltekit svelte build and gonna map each file content with path and serve them using actix web.... I know i could have used some library for this but still to learn rust better I started implementing this myself......
cool, now make a beauty ui