- Видео 16
- Просмотров 10 573
Self-Directed Research
Германия
Добавлен 9 авг 2024
The hosts: Amos Wenger & James Munns
Amos (they/them) is known online as @fasterthanlime. After years working as a software engineer at companies in music, indie gaming and web infrastructure, they've now made "being nerd-sniped" their full-time job, writing long-form technial blog posts and videos.
James (he/him) is an independent researcher and full-time consultant. He founded OneVariable, where he works on Rust, embedded systems, and convincing computers to talk to each other.
We’re doing this because we spend a lot of time figuring things out, and it’s frustrating to do that solo a lot of the time. Researching is fun, but not having anyone to talk to about it is demoralizing.
Sometimes you spend way too much time learning about something, and have to share it.
Amos (they/them) is known online as @fasterthanlime. After years working as a software engineer at companies in music, indie gaming and web infrastructure, they've now made "being nerd-sniped" their full-time job, writing long-form technial blog posts and videos.
James (he/him) is an independent researcher and full-time consultant. He founded OneVariable, where he works on Rust, embedded systems, and convincing computers to talk to each other.
We’re doing this because we spend a lot of time figuring things out, and it’s frustrating to do that solo a lot of the time. Researching is fun, but not having anyone to talk to about it is demoralizing.
Sometimes you spend way too much time learning about something, and have to share it.
Compile Time Crimes
*We shouldn't have to, but it turns out we can*
James explains how to combine macros and const-fns to work around limitations of what is possible at compile time, and how to do extremely wasteful calculations at compile time to deduplicate lists of things to make embedded systems go brrr
Visit sdr-podcast.com/episodes/compile-time-crimes to see the show notes and transcript!
Sponsor: CodeCrafters is a service for learning programming skills by doing. Visit our referral link at
app.codecrafters.io/join?via=sdr-pod to start your free trial. If you upgrade, you'll get a discount and a portion of the sale will support this podcast.
## [00:00:00] Intro
## [00:01:35] Postcard RPC and needed trickery
#...
James explains how to combine macros and const-fns to work around limitations of what is possible at compile time, and how to do extremely wasteful calculations at compile time to deduplicate lists of things to make embedded systems go brrr
Visit sdr-podcast.com/episodes/compile-time-crimes to see the show notes and transcript!
Sponsor: CodeCrafters is a service for learning programming skills by doing. Visit our referral link at
app.codecrafters.io/join?via=sdr-pod to start your free trial. If you upgrade, you'll get a discount and a portion of the sale will support this podcast.
## [00:00:00] Intro
## [00:01:35] Postcard RPC and needed trickery
#...
Просмотров: 2 304
Видео
Target triples - lies, damned lies
Просмотров 53919 часов назад
The Wonderful Story of Target Triples and Six More - Visit sdr-podcast.com/episodes/target-triples to see the show notes and transcript! - CodeCrafters is a service for learning programming skills by doing. Visit our referral link at app.codecrafters.io/join?via=sdr-pod to start your free trial. If you upgrade, you'll get a discount and a portion of the sale will support this podcast. [00:00:00...
What good is partial understanding?
Просмотров 46414 дней назад
*Self-describing formats, but at what cost?* An exploration of self-describing vs non-self-describing formats, and how it changes the shape of your programs more than you might think Visit sdr-podcast.com/episodes/partial-understanding to see the show notes and transcript! Descript is the fully featured, end-to-end video editor that you already know how to use. Check out our referral link (get....
Things you might not have known about clipboards (and some things I still don't understand)
Просмотров 55921 день назад
Amos and James reminisce about how weird clipboards have always been. Or is it pasteboards? Or buffers? Oh boy. Visit sdr-podcast.com/episodes/clipboards/ to see the show notes and transcript! CodeCrafters is a service for learning programming skills by _doing_. Visit our referral link (app.codecrafters.io/join?via=sdr-pod) to start your free trial. If you decide to upgrade, you'll get a discou...
Async Allocators
Просмотров 4,6 тыс.28 дней назад
A deep dive into the potential benefits, and awkward drawbacks, by making all allocations async and fallible Visit sdr-podcast.com/episodes/async-allocators/sdr-podcast.com/episodes/async-allocators/ to see the show notes and transcript! CodeCrafters is a service for learning programming skills by _doing_. Visit our referral link at app.codecrafters.io/join?via=sdr-pod CodeCrafters to start you...
Frame Synchronization
Просмотров 416Месяц назад
An overview of how devices decide how to split streams of bits and bytes into frames, and the things that can go wrong. Visit sdr-podcast.com/episodes/frame-sync/ to see the show notes and transcript! The Self-Directed Research podcast is made possible by our sponsors. We offer 30 second host-read ads at the end of every episode. Not sure how to get your message out, or what to say? Let us help...
Fixing build times with rubicon
Просмотров 322Месяц назад
Amos presents rubicon, which through terrible dynamic linking crimes, brought joy again into developing their website Visit sdr-podcast.com/episodes/dynamic-linking/ to see the show notes and transcript! The Self-Directed Research podcast is made possible by our sponsors. We offer 30 second host-read ads at the end of every episode. Not sure how to get your message out, or what to say? Let us h...
What Are You Syncing About?
Просмотров 492Месяц назад
An introduction to how the 'plumbing' of async tasks and drivers wait, sleep, and are notified for efficient cooperative action; and a glowing overview of the maitake-sync crate's main primitives Visit sdr-podcast.com/episodes/syncing/ to see the show notes and transcript! Today's episode is sponsored by the postcard crate (crates.io/crates/postcard), which is looking for sponsors for the 2.0 r...
You might not need Arc
Просмотров 207Месяц назад
How global is your context? And does it really need atomic reference counting? Visit ladybird.org (ladybird.org) for more information and to join the mailing list. Check out sdr-podcast.com/episodes/arc/ for more information on this episode (like show notes, transcript, more information on the sponsor) *Show Notes* - American Psycho business card scene (ruclips.net/video/T4UQf7zb3Pk/видео.html)...
Talking to Microcontrollers with Postcard-RPC
Просмотров 1232 месяца назад
A conceptual introduction to structured communication protocols, and the design decisions behind the postcard-rpc crate Visit ladybird.org (ladybird.org) for more information and to join the mailing list. Check out sdr-podcast.com/episodes/postcard-rpc/ for more information on this episode (like show notes, transcript, more information on the sponsor) [00:00:00] Start [00:00:56] You can get a l...
Thread-locals galore
Просмотров 1452 месяца назад
An overview of Thread Local Variables, and the challenges they pose for experimental dynamic linking support. Visit ladybird.org (ladybird.org) for more information and to join the mailing list. Check out sdr-podcast.com/episodes/thread-locals/ for more information on this episode (like show notes, transcript, more information on the sponsor) [00:00:00] Start [00:00:43] Pre-presentation chat [0...
Direct Memory Access for the Uninitiated
Просмотров 1522 месяца назад
An introduction to DMA, including what it is commonly used for, and a model of how to think about what the hardware does. You can read the Inside Rust Survey Announcement (blog.rust-lang.org/inside-rust/2024/08/22/embedded-wg-micro-survey.html) for information about the Embedded Rust Community Micro Survey, or you can take the survey now by clicking here (www.surveyhero.com/c/uenp3ydt). Check o...
How Usable is the Rust Cranelift Backend Today
Просмотров 1672 месяца назад
An experience report of attempting and MOSTLY succeeding in using the Cranelift backend for real MacOS applications Thank you to fasterthanlime for sponsoring this episode. Check out fasterthanli.me/ to see how you can support them! Check out sdr-podcast.com/episodes/cranelift/ for more information on this episode (like show notes, transcript, more information on the sponsor) [00:00:00] Start [...
BBQueue - Going Just Far Enough with Generics
Просмотров 1193 месяца назад
A dive into abstracting unusual behavior differences using generics to manage different storage and async usage styles Thank you to OneVariable (onevariable.com/) for sponsoring this episode - Get in touch by emailing contact@onevariable.com. Check out sdr-podcast.com/episodes/bbqueue/ for more information on this episode (like show notes, transcript, more information on the sponsor) [00:00:00]...
I Was Wrong About Rust Build Times
Просмотров 1653 месяца назад
An update to previous research about speeding build times, informed by unexpected increased cost of maintenance Thank you to Tweede golf (tweedegolf.nl/en) for sponsoring this episode! Check out sdr-podcast.com/episodes/wrong-build-times/ for more information on this episode (like show notes, transcript, more information on the sponsor) [00:00:00] Start [00:00:39] Slow Rust build? I was wrong [...
23:59 this is exactly what happened to me when i started trying to use compile time rust. I really like the language, i like a lot of the features, and I think learning it has made be better, but I switched back to C++ for my day-to-day because of garbage like this. It felt like the Rust STL and compile time stuff is extremely not-fit for production quite yet, and I'll be following the changes, but I'm still not switching over to it for large projects just yet.
Important clarification: zig currently does not allow allocations in comptime
Oh, good to know! Neither of us are Zig programmers, so we were definitely guessing.
Why using the O(n^2) loop if you could use sort instead. Implementing merge sort is easy. After you sorted your array, it is easy to extract unique values into result array. Easy in C++14 constexpr.
What about the dead simple approach Ethernet takes: 8b10 or 64b66 for newest faster variants
Interesting. I made the Serial Chinese Room, Interproces, and Telemetry Script, AKA Script2 and the SCRIPT Protocol for RPC, which is hybrid byte encoded and human readable based on contiguous data types.
16:25 I know this is not very rust-like, but do you even need the maybe-uninit? Just initialize everything with zero, you already unwrap everything that is below the current output index, so you don't even use the fact that it's an Option (you also return the number of `Some` values, so it doesn't even help on the consumer side)
For this example, no! For the slightly more complex data types in the real code (or for the general case), it would be nice to have!
@@sdr_pod can we get a link to the real code put in the show notes?
Sure! We'll add it there soon. For now, this is most of the code that inspired this episode: github.com/jamesmunns/postcard-rpc/blob/main/source/postcard-rpc/src/uniques.rs
"It's O(n^2) but that's the compiler's problem" its not even a problem if your n is small anyway.
20:32 oh no typo on the slides, second loop needs an increment... very good episode tho! I'm in the midst of some compile-time crimes myself, w/ a proc_macro tho, neat to see const fn crimes too :)
Wow this is just like C++ constexpr but more painful. I thought C++ is already too restrictive for when you use vectors in constexpr code but not being able to even use for loops seems annoying as hell to me. Props to you for getting it working.
It is helpfully getting better
C++ 🤮
@@oserodal2702 such insightful critique, you clown
@@stefanalecu9532 C++ 🥰
I don't have this problem. For my C++ API Script2 I replaced the C++ std lib and unicode engine with modern data driven design and I found that it was easiest for me to remember and avoid abstract argument collisions if I prefix the function with T if it's a template, and C if it's constexpr. constexpr actually works on non-const data too, it's an optional compile implementation. Search for constexpr in Script2 and you'll see where it comes in useful. For my API I don't define the print code until the top layer using operator overloads, which allows me to print to any destination like ring buffers, blocks of RAM, streams, etc. It might not be the nicest to look at but it doesn't take much thought.
I dont think most people in my life would und erstand why someone would watch this for and reason but I enjoeyed it nonetheless :)
I'd say, while having async allocator for general purpose might be questionable(since if you plan to schedule other work.while waiting for the allocation, the work might need an allocation too and it all turns onto itself perpetually), you can have a niche applIcation where you want to use Huge Pages but say implement your own allocation on top of them with whatever guarantees yoh like, since bigger page means lesser virtual page table(trivially speaking) and while Linux kernel for example can us free huge pages for general purpose also, if you really want you can and then the global allocator can still do most of the general purpose lifting while you have a clear interface to cooperatively manage "the new thing"
The same, more for kernel level, actially goes for stuff like CMA(continious memory allocation) reserved of predetermined size of memory at the system startup, which you could request as a big chunk at once and then cooperatively navigate arounds its usage for the data structures you require, as well as care for address alignmemt & all
The reason aaui for the file uri is the os-x capability sandbox. Sandboxed applications can't just enumerate the filesystem, instead, to gain access to a file at the user's request they need to perform an NSLookup on the request. This prevents applications from gaining access to files not explicitly shared with the app by the user.
This was super fun! On FP registers: besides the fact that these are sometimes different sizes to gp registers, another reason they are often separate is that adding additional ports to a register file is very costly in space. There's additional circuitry per-register-per-port. Add to that, many implementations duplicate register files to get more fast read ports, but you still need to write to all of the register files whenever you update. Separating out the FP path, then, makes sense for FP heavy workloads. Also, I think POWER predates PowerPC, but I could be misremembering. The thinking was that when IBM, Apple, and Motarola came together to design the RISC they could all standardise on, they ended up starting from IBMs ISA and pruning out COBOL-specific instructions.
This episode had all so many little details that give great perspective on what I was already familiar with. Thanks!
There are a bunch of Power architecture CPU cores with a variable length encoding extension (VLE) in the automotive embedded space. These are like the thumb-only ARM cores, except based on PPC rather than Arm. When I started using them, you couldn't even get an upstream GCC release that supported them. There's also an arch named Coldfire which is the name under which the 68k arch lives on.
6:18 - Saying Siri turned on all of the lights in my house for some reason. LOL. Thanks Amos!
That's quite a lot of advertisement in the first minutes for such a small channel. 👋
Seems brilliant
eliza mentioned, hell yeah
Only 280 subscribers? This guy's underrated, and I bet he's gonna blow up in popularity in a few months.
I don't know how it works on other platforms, (I imagine it is similar though), but at least on wayland, clipboards are implemented as IPC. For example if you run `wl-copy`, it actually forks into the background and *hosts* the specified data under a mime type. (Which is per default determined by xdg-mime.) The `wl-copy` source code is actually very readable and is great for figuring out such details. So any export to a specific mime type can be done at the time of the request of the actual content, instead of at time out the copy.
Recently I discovered how the clipboard works on gnome, no matter if you run wayland or x11. It is really similar to Mac OS It only has those two "clipboards", clipboard and primary, since wayland copies that from x11 and gnome just uses that same format. But it also has different MIME types, it doesn't have the ability to store multiple items, but some mime types like "text/uri-list" just say that URIs are seperated by " ", so in the end you have all files you e.g. copy stored in that format. It also behaves like Mac OS, if you e.g. copy an image, it has a "image/png" mime type, and if it has an associates name also the "text/plain" of the name. the URIs are actual files in case of copied files. So after I heard that what you said about Mac OS, I am not sure if mac invented that, or linux / gnome 🤔 P.S: I use linux since 5 years now and never encontered any clipboard bug, the clipboard is just awesome and if you are a programmer or copy / paste often, you should get a application that serves as a clipboard history, windows has thateveb built in.
Surely, SURELY someone you guys know works at apple and is close to or on the Safari team. I used to be able to just twitter DM a few people, but they all left twitter.
each episode brings something interesting! 😉
So using a u8 pointer or slice has always confused me a bit. You can only free the exact u8 pointer that was allocated, right? Not part of it? So should it not be something like Allocation(NonNull([u8]))?
Maybe it's for implementations that want a safe free-after-free?
I read allocators as alligators and was VERY curious
Solving the wrong problem most likely. Would be better if the code could be restructured such that arena allocators could be used and freed all at once, rather than worrying about the lifecycle of every object individually.
My unpopular opinion is that we shouldn't have async anything. Just use a proper thread and stop dealing with ambiguity. It would also eliminate the viral nature of async functions. This is really just another N-cost abstraction where you have to know the internals of how it works to know the actual cost.
Async is literally just the compiler writing a state machine for you. If that is something you need, then having to do it by hand is *normally* way more confusing and annoying. Reification in general is the best way to deal with confusing concepts: push the complexity onto the type system so at least if you have the wrong idea of what's going on you find out about it. Async language support is a pretty good option for doing that for parallelism, in comparison to other options like, say, observables.
@@SimonBuchanNz The implication inherent in what you're saying is both insulting and wrong. I'll forgive your word choice here as you being ignorant to its etymology and not that you're evil, but your attempt to describe how the mechanism works is factually incorrect. Saving state between calls doesn't make a state machine and they're not equivalent just because they both use the word state. The whole point of asynchronous functions was to act as coroutines. If you'd ever manually implemented your own coroutines, you'd know that you don't even need threading to implement them, which is sadly how modern implementations handle them, with threads.
@@anon_y_mousse firstly, that's really annoying and dumb to call a counterargument "wrong and insulting". You don't have to say I'm wrong if you're arguing, that's implicit in the arguing, so you're just pointlessly being a jerk; and it's very silly to feel insulted that somebody dares to disagree with you. Regarding state machines, the way every language lowers async involves introducing a state variable with a value for each await point, and the continuation will transition to a new state, this is consistently referred to as a state machine in implementation literature. And if you do manually implement an async function, you do end up building an essentially equivalent implementation. Literally the only practical difference to the traditional definitions from CompSci courses is those normally have explicitly enumerated concrete inputs that transitions are defined in terms of; in the real world it's quite rare for things explicitly named state machines implementations to bother to separate collecting the input from selecting a transition. Perhaps you shouldn't be assuming the other person knows less than you, and back up your arguments with actual reasons *why* it's wrong rather than simply claiming you know better. And yes, you don't need to use threads for async, and most languages and runtimes will anyway (notably, *not* with what's possibly the most common usage in JavaScript!). There's lots of reasons, both more obvious (it's faster, spawn ergonomics are far nicer) and not as obvious (async FS is garbage on many platforms, so you need a blocking thread anyway) for that, but I don't really know why you're bringing this up as a reason to not use async when your best alternative is to spin up threads yourself. Now you both have handling real concurrency *and* annoying expensive thread management to deal with and the best fix for the latter is to just use a thread pool and dispatch jobs to it and... now you have the main runtime cost and blocking hazards of an async runtime but with worse ergonomics. There's a *reason* things are the way they are. If you want to use threads, sure, go nuts. But you sure are nuts if you think that lets you "stop dealing with ambiguity", that's just an inherent property of both parallelism and concurrency.
Async isn’t about threading. Single-threaded, IO-limited processes benefit immensely from async. How would you use a “proper thread” to make slow, concurrent network calls?
@@funkdefied1 It's implemented with threads in the backend. Learn how things work then get back to me.
9:15 And now i want to allocate SIZE_MAX (or rust equivalent) bytes of contiguous memory.
allocations need to be atomic right. So if async alloc is called twice you might have cases where one of the allocations gets invalidated. eg. freeBlock->next gets set twice. If you're using mutexes for async, might as well block the whole time which is the current interface
How are you allocating a future when you get a allocation error that means oom.
If you are out of memory, then the executor is likely unable to allocate the memory needed to track the future returned from the async allocator. That kind of thing is the big challenge with failable allocation: the fail code path needs to be allocation free and (like with panic mentioned in the video) we don't have a good way to ensure a given code path is allocation free in Rust.
At least in Rust, futures don't necessarily HAVE to allocate, and in the experimental allocator we were discussing, the allocation future doesn't. The future is stored in the task (so you would have allocated the task when you spawned it), meaning that the allocation path has no allocations of its own.
> we don't have a good way to ensure a given code path is allocation free in Rust. I encountered this when I needed to ensure a code path is async-signal-safe :). Almost all wakers potentially allocate.
Perhaps you could preallocate a small pool you maintain? And then if you fail to allocate another spare the next allocation fails.
My personal opinion is that async alloc and/or drop should be an optional library thing rather than the default behavior of the language. Async drop for example would complicate RAII a lot since that would mean that now the lifetime of objects must be async, and since the async state machines arent strictly guaranteed to ever complete, any time you drop something you have an issue of "destructor running somewhere on an undetermined scheduler at an undetermined point in time". Which kinda defeats the point of RAII that says that objects are dropped when they exit scope. It would also mean that you have to implicitly schedule a drop, otherwise you cant drop things once they exit a scope. And for a zero-overhead language where scheduling is an explicit action its a definite no-no.
async drop gets implicitly awaited by the implicit drop at end of scope, so RAII is actually no problem at all
Also just to clarify, nobody is proposing changing any default behaviour. Async drop is something a type has to opt into by implementing AsyncDrop, and it actually _extends_ the ability to use RAII to cases where the cleanup needs to await.
not async, but Zig's allocators are fallible
But what if overcommit is turned on. Also what is the program really supposed to do on OOM? If I would write an OOM safe program it would be one that allocates everything upfront so it cannot fail during execution, one could even punt that job to the loader and allocate out of a static char array (with strict aliasing turned off),
@@theevilcottonball i'm kinda new to this, but i'd guess the following: if overcommit is on, you're out of luck. as for what to do on OOM, it'd depend on the program and how critical the allocation is, most programs probably can't do much with it, but, for example, it could enable implementing allocation retries. about allocating everything upfront, if your program allows for that then sure, not usually the case though.
17:38 basically, C/C++ programming
this is a cool video
Ah, ETX my old friend. Never heard of COBS tho, and that's pretty awesome!
13:46 Polyglot files?
That's the term we were looking for, thanks!
I liked the visual gag :)
Don't know much about rust but this was very relaxing, thanks!
Glad you enjoyed! We have new episodes every week :)