Many Rust content creators are making introductory content or covering application topics. I enjoy that, but at my current stage on my Rust journey I also really like this kind of well explained more advanced technical content. Thank you for sharing your experience and understanding with us. You are unique, effective and appreciated.
It doesn't even seem like that much of an advanced technical video - it is informative, for sure, but the way it explains things makes it - it think - useful for anyone who is new to the language and is a bit confused by all the different type operators (mut/refs and enum types) and why they matter
I think the following analogy is instructive. If you think of `Option` as a container, it makes sense to compare: `&Option` vs `Option`, with: `&Vec` vs `&[T]`. I think the analogy works quite well. `&[T]` is all you'd ever need to immutably access the contained items. If you had `&Vec` you couldn't really call `into_iter()` on it since you don't want to move out of the underlying `Vec`. You can't really do anything useful with it that you couldn't do with a `&[T]`. In the mutable case `&mut Vec` vs `&mut [T]` the distinctions are also similar to `&mut Option` vs `Option`. A `&mut Vec` would allow you to change the underlying `Vec` by for example: `push`, `pop`, changing capacity or even replacing it with an entirely new Vec. That seems like something you would almost never want to do. `&mut [T]` on the other hand makes much more sense in the majority of cases, since that will allow you to mutate any number of the items in place, but would never mess with the integrity of the `Vec`. That would almost always be what you'd want. The considerations about changing the underlying storage seem to match logically as well. I think the analogy mostly checks out. If anyone spots somewhere it fails, please let me know =).
If you are doing performance optimizations, it is sometimes useful to have some persistent buffer (Vec) to use, when building strings for example, and you ussually want them to grow, so you get a `&mut` to them, but that is ussually relegated to an internal API as it is clunky and if you have invariants, wildly unsafe.
@@Jplaysterraria That sounds like a valid usage. My guess is that in such cases you'll be mostly interested in pushing/extending the underlying Vec and won't be interested in changing the details of the contained elements them themselves. I think that's the crux of the matter. There might of course be cases where such a simple dichotomy doesn't hold exactly either. Thanks, for the example thought!
@@Galakyllz There are a few different ways to do the same things in rust and sometimes it takes a while to figure out what to use when. Glad I could help. I might add that another analogy one might pursue is to think of Option as an iter of Option. After all, Option implements map for example. So, why not. So another for constructing APIs is to expose iterators over references or the mutable counterpart. It can be very enlightening to sit down and just twist and turn these abstractions around to get new insights.
Option is pretty much just a pointer - the pointer arithmetics + a bunch of convenience functions. Even outside of rust it is the prefered type for nullable references (unless you want to use raw pointers which usually you shouldn't)
Another big reason related to the Box issue why to use Option is if you use an alternate type internally like Result, or even a raw pointer (for some reason). It's trivial within a function to construct an owned Option that has a &T, but it isn't trivial to return a &Option from a function.
Again, a good insight gained from your videos. Had the realization after watching that this also applies to (&K, &V) vs &(K, V) in many situations, for the same reason of not exposing how you store the data. To return a ref of tuple, you are saying "I have a tuple which I can lend to you" whereas a tuple of refs can be freely constructed from any data source that allows you to get refs to both sides of the tuple i.e. "I have a K and V somewhere, and I'll lend them both to you as a pair."
The only case I return &mut Option is in traits when I want to directly expose the underlying storage for it to be modified. In this case, I can't just make it public (as said at 3:15) because it's a trait. I never encountered another use case.
Arguably, an Option still makes more sense here. It's probably rare that you need to take ownership of the inner T instead of just mutating it and it probably makes more sense to return a mutable reference. Edit: I think in this use-case it only makes sense if you otherwise *need* to transfer ownership of T of the underlying data.
Another reason: You're implementing a trait method that returns &Option but you don't actually have an Option in your implementation and just want to return None. You can't just return None; you now need a redundant empty Option in your type to return a reference to with the correct lifetime. Had it been an Option return type you could return None no problem.
I really like the constant all black background. Perfect for bed time watching! Also this content is extremely useful to me as a beginner rustacean! Great explanations keep it up!
It's nice to hear opinions of experienced rust programmers. It really helps a lot to understand more about rust and have a better foundation when programming I'd really like in depth explanations on more advanced stuff too like niche optimizations and static code analysis
I'm the developer of Hurl, an Open Source cli tool in Rust: I've one year of Rust experience, and still on my learning journey. Your video are crystal clear and invaluable, on essential concepts. Please keep up the good work !
15:12 “Option of a reference has no space overhead over a regular reference. It’s essentially just a reference that can also be null.” Wow! Really cool that Rust has safe and ergonomic null pointers without any overhead.
Well no space overhead. Safe rust will force you to check if the Option is None before dereferencing it which may have a time overhead. If you want to live dangerously than unsafe rust has unwrap_unchecked which does basically what it says on the tin.
Another thing is that using the ? operator on &Option only works if T is Copy. You need to do .as_ref() to turn it into Option anyway if you want to use it in that case 🙂
What a perfect explanation, thank you so much! Yesterday just found out that sizes of Option and &T are equal and thought that it's some fancy compiler optimization with null pointers. And you confirmed it
The all-zeroes Option None is the closest thing I've seen to a NULL so far in Rust. I'm a pretty seasoned developer, but still dipping my toes in the crab-infested waters. My initial instinct was to say "of course you just want a real Option" but I had no idea there were so many reasons to reinforce my gut.
The Rust community has flocked to your channel, and now you are doomed(?) to produce only Rust content for the rest of your life, lest your followers ask "WHY U NO RUST!?" in every non-Rust video!!!!! j/k But seriously, you have a knack for explaining Rust concepts. This channel is going places. Thanks for the great video!
Every time I watch a video of yours I feel there's still so much left to learn about Rust. The quality of your videos and the explanations you're giving really help others like me understand the nuances of Rust. Thanks for the video. Keep up the good work!
I've been learning Rust for a while (coming from 17 years of .NET and C#), and when I saw &Option, the first thing that came to my mind was: what the heck? It doesn't make sense. Although it was very clear to me that the 'correct' way would be Option, it was worth the time spent watching your video. Thanks for your time and effort in making this great content, man.
Thank you soo much for such content, rust is hard as it is, but getting the philosophy and consequences of minute choices we take while writing rust gives a lot of perspective! Thanks for making me a better dev. ❤
10:05 - I'm pretty sure the only "reasonable" way to do that is to construct a new Option and leak it (e.g. with Box::leak(), called on a second box enclosing the whole mess). Which is obviously terrible and bad and wrong, but it is at least "safe."
There's some other advantages to Option which I don't think you covered. 1. It should be slightly more performant to check whether the Option is None, because there isn't a reference that has to be looked up. 2. (This one isn't relevant to discussions of API design because it's about how the data is stored and not how it is exposed) If you do end up storing your data in Option, that should use only 8 bytes of storage to store the pointer, or null. But Box will be 8 bytes to store the pointer to the Option, however many bytes to store the option state (None / Some) plus padding, and the size of T, which has to be reserved so that the option can still be turned into a valid Some. That can be WAY more memory used. Especially if you have a lot of Nones.
This was so helpful! I've banged my head against the issues you discussed many times but never took a step back to think about _why_ before. I stopped to think when you said to at 1:48 and I figured out the basic issues around intent and ownership. You also explained some things I would not have thought of (especially the niche optimizations).
Excellent deep dive into option. It answered questions I had long time ago about designing API when Option is needed. Huge thank you. Please make more.
I'm quite new to Rust and really enjoying how you explain (to me) more advanced topics - it's enticing me to spend some real time with the language. Thank you - and sub earned!
Thank you, that was a very clear explanation! As a relatively new rustacean, even though I understood the difference, I hadn't trained my intuition about which way is superior.
Man, I just discovered your videos and I can’t stop binge-watching them. Keep them up! Are you aware of any tools that I can run on my codebase that would surface some of these ‘bad smells’? That would be very useful for making a conscious decision.
Before watching, my take in one sentence is: One converts easily into the other. &Option converts into Option, so the latter should be used in function signatures so that they're less restrictive. This is the same idea as why &[T] should be preferred over &Vec, or &str over &String. Just as with Vec and String, and for essentially the same reasons again, the argument does not transfer to mutable references. &mut Option can be useful, just as &mut Vec and &mut String can be. These can be used to insert something into the Option, or push something into the Vec/String, respectively, whereas Option, and similarly &mut [T] and &mut str, don't allow the same. This observation is also closely related to the fact that &T is covariant in T, but &mut T is invariant in T.
Partially into the video, I’m noticing you start out with function return types, not argument types. Of course, in that case the function with Option is not less but more restrictive on the caller; but I suppose the restriction on the callee is the relevant point then, likely winning in a trade-off against how little extra usefulness &Option provides to the caller.
0:20 I totally disagree already. If you have a struct containing an option, it normally should not contain lifetimes. You want it to be static and the option should own the value. And if you need Option, you can always use as_ref on the option. You can't convert Option into Option though.
We need more videos like this. I’d suggest something related to parsing Real life crazy JSON documents with a few possible different types for the same fields
Truely master piece of an educational video!! 😊 You probably come from the C++ side, actually that reasoning directly applies to std::optional as well.
I do indeed have a fairly strong C++ background... I'm curious what tipped you off! I'm a huge fan of std::optional in C++, although it's a tragedy that std::optional is illegal. Until pigs fly and that's addressed, sadly the best alternative (IMHO) remains the nullable T*.
@@_noisecode Jeah me too. Hm... I am still learning Rust, but now thinking again, its abit vague what I said, I mean in C++ its totally different: Returning `std::optionalconst &` (or just T* const with the same semantics "contains something" or "doesnt") is totally ok in code where you know that you own the `std::optional` (class member function e.g.). Returning a reference in C++ is always wrong if a temporary is involved etc. So as C++ programmers we know, that member functions returning refs is ok as long as the encapsulated object is alive etc... But in Rust this thinking is not really needed as lifetime tracking is done for you by the compiler etc. So you can do stuff in Rust which the compiler guards for you but in C++ it would be insane. Maybe you have a better analogon how std::optional compares to Option in Rust? Maybe for C++ folks its more clear that a ref to a `Option(al)` : you refer to the object `Optional` itself, so the question is " do you want this??" mostly not... because C++ folks know that std::optional is a ligthweight wrapper around something... which you can always recreate/wrap etc (likewise you would not return a ref to a std::string_view)
Is it possible to have a _future_ c++ compiler that can traces all the lifetimes like in rust compiler, given all allocations are done via the allocator in constructor and the pointer trickery is able to be avoided and replaced by some safer abstractions?
@@JohnWilliams-gy5yc I think the hardest part of trying to apply rust's safety model to C++ is addressing the issue of sharing and mutability. In rust you can safely derive a &[T] from a &Vec, or a &str from a &String or an Option from an &Option because rust's sharing/mutability rules mean that you can be confident that not only will the Vec/String itself continue to exist, but also that it will remain in a stable state. In C++ on the other hand, shared mutability is considered totally normal.
even if you dont think you need to offer a mut accessor, making the field pub is often still the best option. the way i look at it, its not about whether you expect people to modify the field, its about whether modifying that field will violate some invariant. in many cases, your struct will still make perfect sense if someone decides to replace that field without going through your code, so you may as well let them and save yourself some pointless accessor methods
I don't have strong opinions on the matter. I'd just caution that if you let a field be pub, then removing that later will be a breaking change. So giving pub-access to a field might prevent you from restricting the invariants your code is supposed to uphold in a later stage. Sometimes that's fine, sometimes you'll wish you'd been more careful initially. There are many different use cases for structs since they are so fundamental, so your suggestion probably makes sense for a bunch of cases you have in mind. But it might also not make sense in may other cases. Putting pub on something really means "Hey anyone, please mess with this field!".
about the last point you made on "being honest about what function needs" kinda contradicts an idea of "abstract in, concrete out" - making function parameter generic instead would result in more flexible API anyway, thanks a lot for this content. I'd have never figured this out myself!
Please correct me if I'm wrong. What I learn from the video is that we always want to take the reference to a Data if we can. So the point of this video is not about &Option and Option. but in fact about &T and T so writing something like &Option should be okay because it can coerce into Option
There's no reason to take &Option as a parameter because it doesn't provide any more power or anything than Option. You also shouldn't return &Option because it runs into the same encapsulation issues mentioned in the video. It requires you to actually store your data in an Option rather than maybe some other structure, like a Result.
3:10 You're contradicting your later point about encapsulation. Making a field public is the least forward-compatible approach you can think of. If you instead return a &mut Option, it become possible to move this optional field to a sub-field structure later (e.g. a Box).
I saw the thumbnail and the reasons seem immediately clear, I had just never thought about it. I have not the need to sit through this entire video, but i'll at least comment on it.
I hacked together a tiny game and ended up using Option in a place where it's probably not intended and as a result the "correct" reference type in some places of the code ended up being &Option. It was a tiny tile based game, a clone of 2048, in the terminal, using a fixed size array of tiles as a game model and just using stdin/stdout for input. Instead of going for something like Box with a Tile trait, I just made a Tile enum with all the possible tile states. (For such a simple game one could easily just have gone with some integer type, but enums allow me to make bad states unrepresentable so I went with that.) It ended up being the case that all tiles except the case of an empty tile shared some common simple behaviour like being able to be incremented to the next higher state. Also in the game most tiles can slide until they hit other tiles. Empty tiles shouldn't really slide along and they don't count as hits. So again the empty tile is a bit special. Given that I'm already in enum land, the design to go for would probably be to create a two state wrapper enum like `Tile{Empty,Occupied(SpecificTile)}`. And then the SpecificTile enum would containing the details of the specific non empty tile. The benefits of that is that code becomes super easy to read and I can define the semantics however I want. This is definitely a legitimate option. But, since this was just a hacky project to get up and running that I don't intend to ever touch again, I decided to just use an option instead. This is not as semantically clear. In fact, it can end up being quite semantically confusing. Later down the line in a serious project I might be in a state where I maybe have an empty tile and maybe not. That would be represented with Some(None) and None respectively, whose meaning isn't exactly self-evident. I'd advice against using this for anything half serious. There are a few upsides though when iterating quickly. You get a bunch of convenience methods for free. You can use stuff like flatten to ignore all the Nones, and you can use `if let` constructs and more, without writing any boilerplate. It lets you do some things in a quickly and dirty way. Don't do it ^^. But alright. Given that I went with this *bad* design choice, the corollary was that &Option actually is the type that should be used in certain places, since, this particular Option actually was part of the data model and I would sometimes want to mess with the Option itself. My overall conclusion after thinking about it is that it's just a bad decision. The semantics become confusing. I didn't shoot myself in the foot or anything and it does technically work. But if I want to make it useful I'll probably have to end up rewriting that bit eventually. Even when you've been programming in rust for a while, sometimes it can be a bit tricky and take some work to remember how to reproduce the behaviour you want of some of the standard library types. You have to implement a bunch of traits. Understanding what to do and how to do it is a lot trickier than using the existing types. But semantically, it's probably better to give up some of the convenience for clarity. There is some benefit to use a standard library type to quickly iterate and that's exactly the convenience. But it's good to make types and states have a clear unambiguous meaning and that is what the type system allow you to do if you use it right.
This is a super interesting dive into some important design choices. I really appreciate you sharing these thoughts! I agree that when you end up using a "raw" Option for something very core to your design like a Tile in a 2048 game, you can end up in gross situations (Option definitely being one, as you identified). I wonder how much luck you would have had with a design like this: struct Tile(Option); // use an option, but wrap it up impl Tile { pub fn specific(&self) -> Option { self.0.as_ref() } } That way you have a stronger notion of the fact that a Tile can exist-but-be-empty, and you also let callers use all of Option's powerful APIs on the tile's content if they want to. My favorite part of writing code in Rust, especially when I'm starting on a new module / library / project, is "getting the types right", writing out structs and enums that robustly match my mental model of the problem, before writing a single line of procedural code. But I definitely don't always get it perfect. I appreciate you sharing this war story on your own "getting the types right" experience where you won some and lost some. :)
I kinda just always used `Option` without thinking about it most of the time; I always just assumed that if you can get away with returning an owned type for something you always should
Liked it a lot. Around 16:00 i think the arguments aren't perfect, since right in the beginning you talked about your data does not have clone or copy, so this would be impossible here. That's why you propably chose i32. But i think your 2nd point makes more sense here, you should indeed chose Option as signature instead and let the caller figure out how he can provide you with the value in that case. Regarding your initial setup with your Data struct you should have put more emphasis on this point i guess. But in any case great video!
Correct, that last section uses i32 because I needed a Copy type for those examples. I wouldn't have been able to illustrate my points at all with Data, since it doesn't implement Copy or Clone. But yeah, if you need ownership, say so in the function signature, and let the caller figure out how to get you ownership! That'll let call sites move stuff into your function when they can, whereas if you clone inside the function, you clone every time, even from call sites that could have moved.
Option memory representation not just simply "niche" optimization, but also a bridge into void* for FFI. None is actually null pointer and Some pointer to actual data when passed into languages like C/CPP.
I rarely comment under videos but here i have to!! Great content, i need to go check my code later for this. Gives me the same vibes as code aesthetics, just for rust. Magnificent🎉 Edit: please make more videos! Its sad to see that your latest video is already 8 month old
5:05 I'm new, so maybe wrong, but suppose you have an x = &Option, then call some function on widget that e.g. loads a Data, would x then reflect the fact you now have loaded data? As opposed to Option, where you'd need to query again? (I'm not entirely sure you WANT the former behavior, but it's a thing you can't do with Option. And assuming you CAN do it with &Option.)
That situation wouldn't be allowed in Rust; it would mean you mutated the Option while holding an immutable reference to it. The borrow checker would catch it cold, but even if you hacked around it with unsafe code, it would be undefined behavior.
That would make sense if it was &mut Option; In other words a write access to a place that may or may not contain Data (ie. the Data can be set, reset or removed). It's best to think of & references and &mut references as two entirely separate things.
Wait... Option is just a reference that can also be null? So it's just a pointer? Wasn't pointers that can be null the billion-dollar mistake, the evil to end all code, the Bane to Ref's Batman? I take this to mean that explicitly nullable refs are fine, and that the problem is: a. being unable to trust any pointer b. having no mechanism to enforce explicit error handling If f() returns an option, I can't forget to check if it's null. I either use a match on it, or use the '?' shorthand, or my code doesn't compile. Right?
Yes you have to use match or even better "if let Some(v)" or ? if you want to propagate result up to call stack. You don't work with null pointers ever in safe Rust this is just internal optimization that compiler will make for you.
@@maniacZesci Thanks. I'm poking around Options and Results and Maybes because I'm working in JavaScript and I'm not happy with the error handling. It's _mostly_ fine, but it all goes to hell when I have an action that can fail with an exception, and a fallback action that can fail with an exception, and a second fallback that can fail... So I end up with code like this: try { return store.getThing(key); } catch (err) { try { return store.createThing(key); } catch (err2) { try { return store.enqueueCreationOfThing(key); } catch (err3) { // ... } } } Which is horrific. I'm aware of some potential answers. Result caught my eye, but on reflection it doesn't seem to be what I want.
You can also use "if let" which will only enter the body of the if if the option is not None, use ".unwrap()" which will panic if it's None, use unwrap_or() or which will replace it with a default value if it was None or if you really want to throw caution to the wind you can use ".unwrap_unchecked()" in an unsafe block which will invoke undefined behaviour if it is None. But you have to explicitly make the descision, rust won't let you just forget..
Wow, that memory layout thing kinda reminds me of of NULL pointers in C execpt abstracted away such that you don't have to thing about null pointers! great!
I've thought a lot about sharing the source code for these animations, and yes--I would love to, in particular the little library I've developed for doing the data structure visualizations. I'm planning to hold off until it matures a little more though; it's very scrappy and practical and not the most user-friendly at the moment (mostly it's been get-it-done code rather than make-it-pretty code), and I'd be apprehensive about harming the Manim community by publishing it and people viewing it as a model of how things should be done. TL;DR yes, but not yet. ;) Thank you for the support!
Hahah, I literally paused the video after the first second and thought "That first variant looks completely useless and misguided". Then I hit play and you say exactly my thought. :D
Very beautiful video! Regarding your example of a method with argument type &Option that is uncallable with the reference to Data, it may be beneficial to restrict the calling of a function. Artificial casting &Data to &Option would be a code smell, so it's a way to not make it impossible, but not recommended.
That's undefined behavior, you should **never** do that. There's absolutely no guarantee that Option has the same memory layout as &T. The only repr(Rust) types that you're allowed to transmute are types that are marked transparent, with their only field.
@@appelnonsurtaxe Can't you do it like let casted = &Some(*data)? It messes up with the ownership, that's why it naturally prevents from calling a function taking argument of type &Option using &Data.
Many Rust content creators are making introductory content or covering application topics. I enjoy that, but at my current stage on my Rust journey I also really like this kind of well explained more advanced technical content. Thank you for sharing your experience and understanding with us. You are unique, effective and appreciated.
It doesn't even seem like that much of an advanced technical video - it is informative, for sure, but the way it explains things makes it - it think - useful for anyone who is new to the language and is a bit confused by all the different type operators (mut/refs and enum types) and why they matter
This is the Rust content we need!
13:08 Rust actually *guarantees* memory layout of Option to be optimized. Check option documentation, paragraph about representation
Brilliant mid-level content, perfect for folks with some experience and looking to level up. I feel like I just leveled up.
yeah, me too, definitly learned something here :-)
Man, I went into this having no opinion on the matter and came out convinced that you'd have to be crazy to use &Option
lmao same
I think the following analogy is instructive.
If you think of `Option` as a container, it makes sense to compare: `&Option` vs `Option`, with: `&Vec` vs `&[T]`. I think the analogy works quite well. `&[T]` is all you'd ever need to immutably access the contained items. If you had `&Vec` you couldn't really call `into_iter()` on it since you don't want to move out of the underlying `Vec`. You can't really do anything useful with it that you couldn't do with a `&[T]`. In the mutable case `&mut Vec` vs `&mut [T]` the distinctions are also similar to `&mut Option` vs `Option`. A `&mut Vec` would allow you to change the underlying `Vec` by for example: `push`, `pop`, changing capacity or even replacing it with an entirely new Vec. That seems like something you would almost never want to do. `&mut [T]` on the other hand makes much more sense in the majority of cases, since that will allow you to mutate any number of the items in place, but would never mess with the integrity of the `Vec`. That would almost always be what you'd want. The considerations about changing the underlying storage seem to match logically as well.
I think the analogy mostly checks out. If anyone spots somewhere it fails, please let me know =).
If you are doing performance optimizations, it is sometimes useful to have some persistent buffer (Vec) to use, when building strings for example, and you ussually want them to grow, so you get a `&mut` to them, but that is ussually relegated to an internal API as it is clunky and if you have invariants, wildly unsafe.
And now I cannot believe that I've been passing around/requiring `&Vec` in my code. Thanks for pointing this out.
@@Jplaysterraria That sounds like a valid usage. My guess is that in such cases you'll be mostly interested in pushing/extending the underlying Vec and won't be interested in changing the details of the contained elements them themselves. I think that's the crux of the matter. There might of course be cases where such a simple dichotomy doesn't hold exactly either. Thanks, for the example thought!
@@Galakyllz There are a few different ways to do the same things in rust and sometimes it takes a while to figure out what to use when. Glad I could help.
I might add that another analogy one might pursue is to think of Option as an iter of Option. After all, Option implements map for example. So, why not. So another for constructing APIs is to expose iterators over references or the mutable counterpart. It can be very enlightening to sit down and just twist and turn these abstractions around to get new insights.
&Vec is sometimes useful over &[T], like when you want to check the capacity of the Vec
Option is pretty much just a pointer - the pointer arithmetics + a bunch of convenience functions. Even outside of rust it is the prefered type for nullable references (unless you want to use raw pointers which usually you shouldn't)
Another big reason related to the Box issue why to use Option is if you use an alternate type internally like Result, or even a raw pointer (for some reason). It's trivial within a function to construct an owned Option that has a &T, but it isn't trivial to return a &Option from a function.
Yep, the `ok` function is very nice for this
Again, a good insight gained from your videos. Had the realization after watching that this also applies to (&K, &V) vs &(K, V) in many situations, for the same reason of not exposing how you store the data. To return a ref of tuple, you are saying "I have a tuple which I can lend to you" whereas a tuple of refs can be freely constructed from any data source that allows you to get refs to both sides of the tuple i.e. "I have a K and V somewhere, and I'll lend them both to you as a pair."
The only case I return &mut Option is in traits when I want to directly expose the underlying storage for it to be modified. In this case, I can't just make it public (as said at 3:15) because it's a trait. I never encountered another use case.
Arguably, an Option still makes more sense here. It's probably rare that you need to take ownership of the inner T instead of just mutating it and it probably makes more sense to return a mutable reference.
Edit: I think in this use-case it only makes sense if you otherwise *need* to transfer ownership of T of the underlying data.
@@T0MMYGUNZ911 another use-case for &mut Option is if you want the caller to be able to set your option to None
Another reason: You're implementing a trait method that returns &Option but you don't actually have an Option in your implementation and just want to return None. You can't just return None; you now need a redundant empty Option in your type to return a reference to with the correct lifetime. Had it been an Option return type you could return None no problem.
I believe in that case you can use a static to avoid having the redundant field in your data type. Still ugly though.
This is my new fav channel
The explanations in your videos are honestly the best i've found on youtube. I wish I could find more content just like this.
I really like the constant all black background. Perfect for bed time watching! Also this content is extremely useful to me as a beginner rustacean! Great explanations keep it up!
Please keep making these. They're wonderful for an intermediate Rust dev.
It's nice to hear opinions of experienced rust programmers. It really helps a lot to understand more about rust and have a better foundation when programming
I'd really like in depth explanations on more advanced stuff too like niche optimizations and static code analysis
I'm the developer of Hurl, an Open Source cli tool in Rust: I've one year of Rust experience, and still on my learning journey. Your video are crystal clear and invaluable, on essential concepts. Please keep up the good work !
15:12 “Option of a reference has no space overhead over a regular reference. It’s essentially just a reference that can also be null.”
Wow! Really cool that Rust has safe and ergonomic null pointers without any overhead.
Well no space overhead. Safe rust will force you to check if the Option is None before dereferencing it which may have a time overhead.
If you want to live dangerously than unsafe rust has unwrap_unchecked which does basically what it says on the tin.
Thanx, great stuff! Real world problem for a Rust beginner like me - not just repeating stuff from the Rust book. Looking forward to more!
Another thing is that using the ? operator on &Option only works if T is Copy. You need to do .as_ref() to turn it into Option anyway if you want to use it in that case 🙂
What a perfect explanation, thank you so much! Yesterday just found out that sizes of Option and &T are equal and thought that it's some fancy compiler optimization with null pointers. And you confirmed it
The all-zeroes Option None is the closest thing I've seen to a NULL so far in Rust.
I'm a pretty seasoned developer, but still dipping my toes in the crab-infested waters. My initial instinct was to say "of course you just want a real Option" but I had no idea there were so many reasons to reinforce my gut.
The Rust community has flocked to your channel, and now you are doomed(?) to produce only Rust content for the rest of your life, lest your followers ask "WHY U NO RUST!?" in every non-Rust video!!!!!
j/k
But seriously, you have a knack for explaining Rust concepts. This channel is going places. Thanks for the great video!
B-but my poor Unreal videos....
I ended up here because I'm still learning Rust and felt aggrieved when dealing with linked lists on Leetcode. That's great content! Thank you!
First Arc, now Option, what else is there to come? I want a trilogy!
Been doing rust full time for 2 years. This is a *great* video, keep the amazing work
Every time I watch a video of yours I feel there's still so much left to learn about Rust. The quality of your videos and the explanations you're giving really help others like me understand the nuances of Rust. Thanks for the video. Keep up the good work!
Been thinking this for years, glad someone finally said it
This is the first RUclips video I'd be watching on coding and I'm so glad to come across it. So expository! Thanks for sharing. You're amazing.
I've been learning Rust for a while (coming from 17 years of .NET and C#), and when I saw &Option, the first thing that came to my mind was: what the heck? It doesn't make sense. Although it was very clear to me that the 'correct' way would be Option, it was worth the time spent watching your video. Thanks for your time and effort in making this great content, man.
A good video for people who started learning Rust but are a bit out of the baby steps stage already!
Thank you soo much for such content, rust is hard as it is, but getting the philosophy and consequences of minute choices we take while writing rust gives a lot of perspective! Thanks for making me a better dev. ❤
this reminds me of scott meyer's books, but for rust and in video form!! very well thought out and more attentive than most articles
Your videos are extremely high quality! Please make more! I'd happily to pay for an in depth course.
10:05 - I'm pretty sure the only "reasonable" way to do that is to construct a new Option and leak it (e.g. with Box::leak(), called on a second box enclosing the whole mess). Which is obviously terrible and bad and wrong, but it is at least "safe."
I'm a beginner Rustacean, with tons of background in other languages. These videos are very helpful.
Two and a half minutes in, and you got yourself a new subscriber
Absolutely golden. Loved every second of it! I knew Rust was blazingly fast, but the niche optimization part blew me away
I love FEET 😍
There's some other advantages to Option which I don't think you covered.
1. It should be slightly more performant to check whether the Option is None, because there isn't a reference that has to be looked up.
2. (This one isn't relevant to discussions of API design because it's about how the data is stored and not how it is exposed)
If you do end up storing your data in Option, that should use only 8 bytes of storage to store the pointer, or null. But Box will be 8 bytes to store the pointer to the Option, however many bytes to store the option state (None / Some) plus padding, and the size of T, which has to be reserved so that the option can still be turned into a valid Some. That can be WAY more memory used. Especially if you have a lot of Nones.
This was so helpful! I've banged my head against the issues you discussed many times but never took a step back to think about _why_ before. I stopped to think when you said to at 1:48 and I figured out the basic issues around intent and ownership. You also explained some things I would not have thought of (especially the niche optimizations).
This is one of the best explanations I saw about Rust, keep up with great videos like this, your way of speaking is amazing
Finally, some advanced Rust advice! Thank you!
Excellent deep dive into option. It answered questions I had long time ago about designing API when Option is needed. Huge thank you. Please make more.
I'm quite new to Rust and really enjoying how you explain (to me) more advanced topics - it's enticing me to spend some real time with the language. Thank you - and sub earned!
Thank you, that was a very clear explanation! As a relatively new rustacean, even though I understood the difference, I hadn't trained my intuition about which way is superior.
Im not ur audience but just for fun, i never touched a single like of Rust. Still i understood almost everything. Great video, very well explained
Man, I just discovered your videos and I can’t stop binge-watching them. Keep them up!
Are you aware of any tools that I can run on my codebase that would surface some of these ‘bad smells’? That would be very useful for making a conscious decision.
Glad to find someone who likes null pointer optimisation as cool as I do x)
Before watching, my take in one sentence is: One converts easily into the other. &Option converts into Option, so the latter should be used in function signatures so that they're less restrictive. This is the same idea as why &[T] should be preferred over &Vec, or &str over &String.
Just as with Vec and String, and for essentially the same reasons again, the argument does not transfer to mutable references. &mut Option can be useful, just as &mut Vec and &mut String can be. These can be used to insert something into the Option, or push something into the Vec/String, respectively, whereas Option, and similarly &mut [T] and &mut str, don't allow the same. This observation is also closely related to the fact that &T is covariant in T, but &mut T is invariant in T.
Partially into the video, I’m noticing you start out with function return types, not argument types. Of course, in that case the function with Option is not less but more restrictive on the caller; but I suppose the restriction on the callee is the relevant point then, likely winning in a trade-off against how little extra usefulness &Option provides to the caller.
0:20 I totally disagree already.
If you have a struct containing an option, it normally should not contain lifetimes. You want it to be static and the option should own the value.
And if you need Option, you can always use as_ref on the option. You can't convert Option into Option though.
2:00 Now it gets clearer. It's not about never using &Option, but about converting it to Option rather sooner than later.
Wow, great follow up to your previous Rust video! Loving the super focused in-depth style you have
Nice video... I 99.9% agree with you and that .1% is filled with "A long enough timeline will provide at minimum one counterexample to any rule".
We need more videos like this. I’d suggest something related to parsing Real life crazy JSON documents with a few possible different types for the same fields
Truely master piece of an educational video!! 😊 You probably come from the C++ side, actually that reasoning directly applies to std::optional as well.
I do indeed have a fairly strong C++ background... I'm curious what tipped you off!
I'm a huge fan of std::optional in C++, although it's a tragedy that std::optional is illegal. Until pigs fly and that's addressed, sadly the best alternative (IMHO) remains the nullable T*.
@@_noisecode Jeah me too. Hm... I am still learning Rust, but now thinking again, its abit vague what I said, I mean in C++ its totally different: Returning `std::optionalconst &` (or just T* const with the same semantics "contains something" or "doesnt") is totally ok in code where you know that you own the `std::optional` (class member function e.g.). Returning a reference in C++ is always wrong if a temporary is involved etc. So as C++ programmers we know, that member functions returning refs is ok as long as the encapsulated object is alive etc... But in Rust this thinking is not really needed as lifetime tracking is done for you by the compiler etc. So you can do stuff in Rust which the compiler guards for you but in C++ it would be insane. Maybe you have a better analogon how std::optional compares to Option in Rust?
Maybe for C++ folks its more clear that a ref to a `Option(al)` : you refer to the object `Optional` itself, so the question is " do you want this??" mostly not... because C++ folks know that std::optional is a ligthweight wrapper around something... which you can always recreate/wrap etc (likewise you would not return a ref to a std::string_view)
Is it possible to have a _future_ c++ compiler that can traces all the lifetimes like in rust compiler, given all allocations are done via the allocator in constructor and the pointer trickery is able to be avoided and replaced by some safer abstractions?
@@JohnWilliams-gy5yc I think the hardest part of trying to apply rust's safety model to C++ is addressing the issue of sharing and mutability.
In rust you can safely derive a &[T] from a &Vec, or a &str from a &String or an Option from an &Option because rust's sharing/mutability rules mean that you can be confident that not only will the Vec/String itself continue to exist, but also that it will remain in a stable state.
In C++ on the other hand, shared mutability is considered totally normal.
even if you dont think you need to offer a mut accessor, making the field pub is often still the best option. the way i look at it, its not about whether you expect people to modify the field, its about whether modifying that field will violate some invariant. in many cases, your struct will still make perfect sense if someone decides to replace that field without going through your code, so you may as well let them and save yourself some pointless accessor methods
I don't have strong opinions on the matter. I'd just caution that if you let a field be pub, then removing that later will be a breaking change. So giving pub-access to a field might prevent you from restricting the invariants your code is supposed to uphold in a later stage. Sometimes that's fine, sometimes you'll wish you'd been more careful initially. There are many different use cases for structs since they are so fundamental, so your suggestion probably makes sense for a bunch of cases you have in mind. But it might also not make sense in may other cases. Putting pub on something really means "Hey anyone, please mess with this field!".
about the last point you made on "being honest about what function needs" kinda contradicts an idea of "abstract in, concrete out" - making function parameter generic instead would result in more flexible API
anyway, thanks a lot for this content. I'd have never figured this out myself!
I appreciate having the answer in the thumbnail.
Please correct me if I'm wrong.
What I learn from the video is that we always want to take the reference to a Data if we can.
So the point of this video is not about &Option and Option.
but in fact about &T and T
so writing something like &Option should be okay because it can coerce into Option
There's no reason to take &Option as a parameter because it doesn't provide any more power or anything than Option. You also shouldn't return &Option because it runs into the same encapsulation issues mentioned in the video. It requires you to actually store your data in an Option rather than maybe some other structure, like a Result.
3:10 You're contradicting your later point about encapsulation. Making a field public is the least forward-compatible approach you can think of. If you instead return a &mut Option, it become possible to move this optional field to a sub-field structure later (e.g. a Box).
I saw the thumbnail and the reasons seem immediately clear, I had just never thought about it. I have not the need to sit through this entire video, but i'll at least comment on it.
I hacked together a tiny game and ended up using Option in a place where it's probably not intended and as a result the "correct" reference type in some places of the code ended up being &Option.
It was a tiny tile based game, a clone of 2048, in the terminal, using a fixed size array of tiles as a game model and just using stdin/stdout for input.
Instead of going for something like Box with a Tile trait, I just made a Tile enum with all the possible tile states. (For such a simple game one could easily just have gone with some integer type, but enums allow me to make bad states unrepresentable so I went with that.) It ended up being the case that all tiles except the case of an empty tile shared some common simple behaviour like being able to be incremented to the next higher state. Also in the game most tiles can slide until they hit other tiles. Empty tiles shouldn't really slide along and they don't count as hits. So again the empty tile is a bit special.
Given that I'm already in enum land, the design to go for would probably be to create a two state wrapper enum like `Tile{Empty,Occupied(SpecificTile)}`. And then the SpecificTile enum would containing the details of the specific non empty tile. The benefits of that is that code becomes super easy to read and I can define the semantics however I want. This is definitely a legitimate option.
But, since this was just a hacky project to get up and running that I don't intend to ever touch again, I decided to just use an option instead. This is not as semantically clear. In fact, it can end up being quite semantically confusing. Later down the line in a serious project I might be in a state where I maybe have an empty tile and maybe not. That would be represented with Some(None) and None respectively, whose meaning isn't exactly self-evident. I'd advice against using this for anything half serious.
There are a few upsides though when iterating quickly. You get a bunch of convenience methods for free. You can use stuff like flatten to ignore all the Nones, and you can use `if let` constructs and more, without writing any boilerplate. It lets you do some things in a quickly and dirty way. Don't do it ^^.
But alright. Given that I went with this *bad* design choice, the corollary was that &Option actually is the type that should be used in certain places, since, this particular Option actually was part of the data model and I would sometimes want to mess with the Option itself.
My overall conclusion after thinking about it is that it's just a bad decision. The semantics become confusing. I didn't shoot myself in the foot or anything and it does technically work. But if I want to make it useful I'll probably have to end up rewriting that bit eventually.
Even when you've been programming in rust for a while, sometimes it can be a bit tricky and take some work to remember how to reproduce the behaviour you want of some of the standard library types. You have to implement a bunch of traits. Understanding what to do and how to do it is a lot trickier than using the existing types. But semantically, it's probably better to give up some of the convenience for clarity. There is some benefit to use a standard library type to quickly iterate and that's exactly the convenience. But it's good to make types and states have a clear unambiguous meaning and that is what the type system allow you to do if you use it right.
This is a super interesting dive into some important design choices. I really appreciate you sharing these thoughts! I agree that when you end up using a "raw" Option for something very core to your design like a Tile in a 2048 game, you can end up in gross situations (Option definitely being one, as you identified). I wonder how much luck you would have had with a design like this:
struct Tile(Option); // use an option, but wrap it up
impl Tile {
pub fn specific(&self) -> Option { self.0.as_ref() }
}
That way you have a stronger notion of the fact that a Tile can exist-but-be-empty, and you also let callers use all of Option's powerful APIs on the tile's content if they want to.
My favorite part of writing code in Rust, especially when I'm starting on a new module / library / project, is "getting the types right", writing out structs and enums that robustly match my mental model of the problem, before writing a single line of procedural code. But I definitely don't always get it perfect. I appreciate you sharing this war story on your own "getting the types right" experience where you won some and lost some. :)
I kinda just always used `Option` without thinking about it most of the time; I always just assumed that if you can get away with returning an owned type for something you always should
That’s hot! Well done, it’s really nice to see beautiful explanations and animations to make information structured, reliable and easy, thanks!
Liked it a lot.
Around 16:00 i think the arguments aren't perfect, since right in the beginning you talked about your data does not have clone or copy, so this would be impossible here.
That's why you propably chose i32. But i think your 2nd point makes more sense here, you should indeed chose Option as signature instead and let the caller figure out how he can provide you with the value in that case. Regarding your initial setup with your Data struct you should have put more emphasis on this point i guess.
But in any case great video!
Correct, that last section uses i32 because I needed a Copy type for those examples. I wouldn't have been able to illustrate my points at all with Data, since it doesn't implement Copy or Clone.
But yeah, if you need ownership, say so in the function signature, and let the caller figure out how to get you ownership! That'll let call sites move stuff into your function when they can, whereas if you clone inside the function, you clone every time, even from call sites that could have moved.
Option memory representation not just simply "niche" optimization, but also a bridge into void* for FFI. None is actually null pointer and Some pointer to actual data when passed into languages like C/CPP.
U r the guy, whose gonna help me through rust
I rarely comment under videos but here i have to!! Great content, i need to go check my code later for this. Gives me the same vibes as code aesthetics, just for rust. Magnificent🎉
Edit: please make more videos! Its sad to see that your latest video is already 8 month old
Very cool detailed explanation. And also, I tend to agree with this pattern.
5:05 I'm new, so maybe wrong, but suppose you have an x = &Option, then call some function on widget that e.g. loads a Data, would x then reflect the fact you now have loaded data? As opposed to Option, where you'd need to query again? (I'm not entirely sure you WANT the former behavior, but it's a thing you can't do with Option. And assuming you CAN do it with &Option.)
That situation wouldn't be allowed in Rust; it would mean you mutated the Option while holding an immutable reference to it. The borrow checker would catch it cold, but even if you hacked around it with unsafe code, it would be undefined behavior.
@@_noisecode Oh, I see. That makes sense, thanks.
That would make sense if it was &mut Option; In other words a write access to a place that may or may not contain Data (ie. the Data can be set, reset or removed). It's best to think of & references and &mut references as two entirely separate things.
Your visualisations are really clean!
Another point wrt parameters would be that you can bind a generic type to Into and allow the caller to pass a &T directly without having to wrap it!
Great video, it was interesting to see how rust stores option reference None too.
If I need ownership, I like to use `impl Into`. That way, the caller can simply `call(42)` or `call(None)`.
Wait... Option is just a reference that can also be null? So it's just a pointer? Wasn't pointers that can be null the billion-dollar mistake, the evil to end all code, the Bane to Ref's Batman?
I take this to mean that explicitly nullable refs are fine, and that the problem is:
a. being unable to trust any pointer
b. having no mechanism to enforce explicit error handling
If f() returns an option, I can't forget to check if it's null. I either use a match on it, or use the '?' shorthand, or my code doesn't compile. Right?
Yes you have to use match or even better "if let Some(v)" or ? if you want to propagate result up to call stack. You don't work with null pointers ever in safe Rust this is just internal optimization that compiler will make for you.
@@maniacZesci Thanks. I'm poking around Options and Results and Maybes because I'm working in JavaScript and I'm not happy with the error handling.
It's _mostly_ fine, but it all goes to hell when I have an action that can fail with an exception, and a fallback action that can fail with an exception, and a second fallback that can fail...
So I end up with code like this:
try {
return store.getThing(key);
} catch (err) {
try {
return store.createThing(key);
} catch (err2) {
try {
return store.enqueueCreationOfThing(key);
} catch (err3) {
// ...
}
}
}
Which is horrific. I'm aware of some potential answers. Result caught my eye, but on reflection it doesn't seem to be what I want.
You can also use "if let" which will only enter the body of the if if the option is not None, use ".unwrap()" which will panic if it's None, use unwrap_or() or which will replace it with a default value if it was None or if you really want to throw caution to the wind you can use ".unwrap_unchecked()" in an unsafe block which will invoke undefined behaviour if it is None.
But you have to explicitly make the descision, rust won't let you just forget..
This video was something I definitely needed to see. Thank you for making my code better. I really appreciate it.
I love those topics. It improves my intuition about design choices!
You videos are amazing really
They are very depth and help a lot in understanding
Thank you for all your videos
You’re welcome and I’m so glad they’re helpful!
I just discovered your channel today, great stuff by far. Subscribed!
God I love your videos. Please let me support your content with a patreon or through RUclips so it continues to flow.
Wow, that memory layout thing kinda reminds me of of NULL pointers in C execpt abstracted away such that you don't have to thing about null pointers! great!
Awesome explanation! I also enjoy the visuals you use with Manim. Would you be willing to share your Manim scripts?
I've thought a lot about sharing the source code for these animations, and yes--I would love to, in particular the little library I've developed for doing the data structure visualizations. I'm planning to hold off until it matures a little more though; it's very scrappy and practical and not the most user-friendly at the moment (mostly it's been get-it-done code rather than make-it-pretty code), and I'd be apprehensive about harming the Manim community by publishing it and people viewing it as a model of how things should be done. TL;DR yes, but not yet. ;) Thank you for the support!
This is good and honest content, keep it up!
another banger video 🔥
Never seen a programming video like this one. Super good 😮
this video is useful and valuable! I enjoyed learning about this and will employ it personally from now on
Thx for this video! I already knew about the Null Pointer Optimization stuff but I'll now know which to use!
There was a pretty good opportunity for a Sum 41 reference in this video
I KNOW, I was kicking myself. I even mentioned my regrets in the description
This video was amazing and incredibly informative
Great content. Extremely useful overview. Hope you will keep on releasing such great videos. Thank you
Hahah, I literally paused the video after the first second and thought "That first variant looks completely useless and misguided". Then I hit play and you say exactly my thought. :D
I'm instantly subscribed. Thank you for this video !
You explanation is absolutely clear and you can make a video shorter definitely. Thanks for Rust content, btw.
loving you rust videos. instantly subscribed
16:28 and Clone?
I'm starting out with teaching myself Rust. Is there a linter that can point out such best practices for me inline?
Really good video as always, thanks for your detailed explanations, I'm looking forward to the next video :)
i love your channel, keep up with those type of videos, we need more of them :)
Great video! I will definitely be putting this to use in the workplace.
Where are you working with Rust?
Another great Rust video!! Thanks for making this!
Very beautiful video! Regarding your example of a method with argument type &Option that is uncallable with the reference to Data, it may be beneficial to restrict the calling of a function. Artificial casting &Data to &Option would be a code smell, so it's a way to not make it impossible, but not recommended.
That's undefined behavior, you should **never** do that. There's absolutely no guarantee that Option has the same memory layout as &T.
The only repr(Rust) types that you're allowed to transmute are types that are marked transparent, with their only field.
@@appelnonsurtaxe Can't you do it like
let casted = &Some(*data)? It messes up with the ownership, that's why it naturally prevents from calling a function taking argument of type &Option using &Data.
@@mzg147 if you do that, it would work, but it would create a copy of Data, and require Data to implement Copy in the first place