Function Iterators might just change the way we write loops in Go
HTML-код
- Опубликовано: 28 май 2024
- The rangefunc experimental feature is now available with Go 1.22.
Over the last couple of weeks I have been playing with this new feature to see how far I can push it, and I think it may just change the way we write some loops in Go.
Sign up for my Golang course at: dreamsofcode.io/courses/comma...
(Website is written in Go, btw)
Video links:
- RangeFunc wiki: go.dev/wiki/RangefuncExperiment
- Loop package: github.com/dreamsofcode-io/loop
Join this channel to get access to perks:
/ @dreamsofcode
Join Discord: / discord
Join Twitter: / dreamsofcode_io
00:00:00 Intro
00:00:23 rangefunc
00:01:55 Parallel iteration in Go
00:03:46 Convert to Parallel Iterator
00:05:57 Handling Break
00:07:47 Making Generic
00:08:28 Footguns
00:09:08 iter package
00:10:01 Loop package
00:10:38 Conclusion
00:10:59 Good news! 
Наука
The waitgroup verison of the example is MUCH easier to reason about. I would hate to have to debug that function.
me too
If the function is implemented by a random CoWorker, with a silly name, obscur pattern etc.
Yeah obviously, I prefer the WaitGroup version. But, If Parallel becomes an actual thing in std, I'll be happy to use it.
It's like Generic, random package using Generic for nothing or container that I never needed, is a redflag for me. But, Generic helpers like `cmp.Or` or the `slices` package is actually a great thing which remove a lot of boilerplate.
I think a lot of the potential harm that could be caused by this feature would be almost entirely mitigated with a strong standard library implementation of many/most of the most common use cases. I think this approach is fine, and generally needed when writing a lot of iterator-heavy code. This seems fairly reasonable.
What I get scared of is every sophomore programmer thinking they need their own implementation of exotic iterators, writing their own package, and it just creating a bunch more headache that would've been prevented with a simpler or more explicit approach. Not everyone should get equal access to parallel iterators! That comes with some very real risks!😆
This feature sounds more and more like templates in c++
also couldn't he just put the code in the for loop in its own function to make the logic more readable? seems like so much work to create all that code to have an iteratable function
This is really powerful. But it feels very un-Go-y……eg it’s too much magic. Obfuscation isn’t good for Go, I know verbosity gives it a bad repo, but I also think this forces better code
That's why it's experimental, and will improve over time :)
The syntax is not the most readable, but overall, I'm in favour of the proposal. It will be a very useful tool for the Go devs and for library devs like myself. Endusers don't have to deal with the inner complexity as much and can have a more homegeneous experience dealing with ranges. It's up to the library devs to write self-documenting and readable iterators.
Elliott, pls, make a step by step guide on how to use pprof professionally.
THIS! cpu and memory profiling.
yesss
Nice feature to extend loops, but such an implementation feels difficult to handle and violates the value of simplification. So, whatever the Go Community prefers :)
Go is such a cool language, I just started learning it so this was very helpful. Thanks Dreams!
Python and C++ (modern C++, anyway) make fairly heavy use of generators (in C++ up to 20 they were implemented with input iterators; C++23 has an explicit std::generator class). It's a very powerful pattern for parsing data streams, for example. Node has had something similar since basically forever.
very interesting! great quality content as always. thanks
Just what I needed! TY TY TY
5:40 that moment when I'm finally stupid enough for Go.
This will be very nice to work with when using remote data sources - just treating them as arrays, imstead of something like 'for val, err := source.Next(); err == nil { dostuff(val) }
Nice to see Go finally getting generators. I use this pattern a lot when working in PHP, so good to see it's going to be possible to do it in Go.
Every time I'm surprised by the fact that go hasn't all the things I take for granted now in Rust and Kotlin.
I need to make my own stuff too much, it feels like left-pad all over again.
This video is great. The explanation and animation is great. One nit, the Parallel implementation is wrong. It works because it is unlikely that that race condition will be met once you've canceled the context, but another goroutine could be yielding in as you cancel. This is a classic case where you need a mutex. A potential solution could look like this:
func Parallel[T any](list []T) func(func(int, T) bool) {
return func(yield func(int, T) bool) {
var (
wg sync.WaitGroup
m sync.Mutex
cancel = make(chan struct{})
isCanceled bool
)
wg.Add(len(list))
for i, e := range list {
go func() {
defer wg.Done()
select {
case
but it will no longer run in parallel right?
@@what1heh3ck well this example is trivial. Technically you would want to be doing doing some work to generate the items you yield. In this case we are doing no work, and since we need to synchronize at the moment of yielding as per the spec or risk a panic, then in this case there is no advantage to using concurrency and only serious drawbacks.
But as an example of how concurrency could be built into function iterators it’s great!
TLDR: we don’t want concurrent yielding, but concurrent generation/work.
I’m not super familiar with Go, but isn’t your code holding the lock while it calls yield? Yield being the function that runs the body of the for loop, right?
The entire point of this wrapper is to run the body of the for loop multiple times in parallel. Your function doesn’t do that, as far as I can tell. All your version seems to do is run a single-threaded for loop split across N threads.
I’m also not clear on what race condition you’re trying to avoid. The entire purpose of this wrapper is to call yield from multiple threads simultaneously, so that can’t be it.
I suppose there is the case where one thread calls cancel while another thread is in the middle of calling yield. However, that’s an unavoidable consequence of the task at hand. If you run a for loop in parallel, you don’t get any guarantees about iteration order. Break could never work like it normally does.
@@TheArtikae you are correct.
However the code shown in the video is not safe. The context does not definitely block two goroutines from yield racing whilst one returns false.
I am not sure this parrallel can be safe in Go using function iterators. Mine basically just made it serial
@@daviddesmarais-michaud4345 Oh dear, I assumed that yield was safe to call from multiple threads, seeing as that was the entire point of the Parallel wrapper shown in the video. I was wrong. You're absolutely right about the issue, and the data race. The wrapper could never possibly be correct with function iterators.
Maybe it's the specific example, but this implementation made the code far more complicated and less obvious than it was when using the async and go func directly. It can be bad when abused.
I mean, the video specifically intended to show a footgun example.
There's other examples that swing the other way like iterating over the results of postgres cursor without leaking it. This makes it possible to write code that will not leak resources needed by an iterator (unless you use iter.pull)
The syntax of "functions returning functions accepting functions that you can pull values from" is too complicated and magic-y. It seems like they're trying to stretch the extra-simplistic syntax of Go1, when it might be better to learn from its shortcomings to come up with Go2
But that's my humble opinion
They try a little *too* hard to never add any new keywords or core types to the language, and always reuse what already exists, in this case "for..range". Sometimes it would be better to just expand the language a bit for the sake of clarity. Reading the developer discussions on various language proposals is a frustrating experience: practically everything is rejected if would require anything new whatsoever to the syntax of current lexicon/vocabulary of the language. I appreciate that they err on the side of simplicity, it is one of Go's greatest strengths, but then there is examples such as this, where it actually makes the language more complicated by being so strict.
This experimental feature challenges boilerplate oriented programming.
Jokes aside. It's more convenient for sure. In my opinion it's bad for go's simplicity.
yeah, it does feel like a little out-of-place for go. This would perhaps be a good candidate for a dedicated keyword (i.e. "iter" instead of reusing "range") to make it abundantly clear what is going on, and/or some refinement on the syntax, which feels a bit clunky. While i understand it is not a requirement to write it as such, realistically the syntax encourages that this is most often going to be written as layers of anonymous nested closures, which feels "off" as a core language feature.
@@ForeverZer0 Dedicated keyword could be too confusing. It's feels weird that range for an array or a slice yields index and value, but just value for functions. Take python, for example, it has "for x in iterable". It doesn't matter if iterable is a generator or an array. But definitely less confusing because of consistency.
Parallel processing can definitely get interesting when hidden somewhere down in code you don't know about there is a shared resource. Something like a memory allocator perhaps.
yaaa.... it think go should bench this one. it just seems like the plot has been lost in this feature. i know go wants to be this gc lang that has first class support for concurrency to make up for that but it also promises to stay simple like c. It feels like too big of a sacrifice of the former in favor for the latter....
So long as the standard library gets a good set of iterators covering most use cases this could be quite good without introducing too much magic you need to lookup to the language. My favorite thing about go is it’s dead simple and you can at a glance see exactly what the code is doing.
So it seems like function integrators are, generally, a good feature. The concern seems to come if an iterator implements concurrency because the concurrent logic is obfuscated behind the function.
I’d love to see some high level iterators implemented by the std lib but default (or only) be synchronous. Maybe there could be a pattern to “enable” or “opt in” to concurrency on the new iterators. That way it would be obvious that they’re concurrent without having to read how exactly that iterator you’re using works
what do you use to animate the code in your videos?
Please create a premium course on Go. I am all ready to subscribe to it. 🎉🎉🎉🎉🎉You are probably the best teacher who teaches golang perfectly.
A callback closure is basically the same thing as the range loop body so I don't think this is really needed. I wouldn't mind it being there though, and I'd very much like if stdlib ranging was standardized so we'll see I guess.
One of the ways I've been identifying goroutine usages obfuscated by a function call is to require a context passed in as the first parameter.
func func func func func func
it seems rust is becoming easier and clear than go. The only good thing i like about go is compile time.
As someone who stills prefers rust... go has way, way, way easier concurrency. Doing async stuff in rust is a huge headache if you go beyond trivial examples.
@@omgnifty5957 In Go you will get panic in production. In Rust compilation error.
This is just try_for_each tho, isn’t it? What’s the big deal?
@@omgnifty5957why can't you use tokio select and mpsc channels to achieve the same concurrency approach?
Finally Go has gotten a feature that existed in other languages like Python or C# 20 years ago.
I highly prefer the boiler plate "heavy" way of doing this. No reason to abstract away easy patterns everyone knows
if go manages to polish typing system or move to traits (which is unlikely), it'll be hands down the best language
Yes, this video made me appreciate Rust even more. Thanks!
Rust is trash
L
I know nothing about neither Rust nor Go, but this code seemed complex to me, maybe because I have zero contact with the language.
My question is, when a person gets to the point where they need to use such functionalities as shown in the video, is Rust a more practical language when it comes to achieving the same results?
@@heldim92 I'm not saying it's _practical_, I believe that would just be untrue (depending on your definition of "practical"). But what I came to enjoy about Rust is that code is explicit: you know at compile time that it is race condition free (at least in your own usage, i.e. multithreaded code) , mutation-free, or explicitly signaled as a mutation. Is it more practical? Most of the time, no. But it's usually correct. And iterators are already baked into the language and a joy to use
this is huge
Can you make a neovim for elixir tutorial? I keep getting stuck on it myself.
I think this could be really powerful but it's kind of complicated, I would use it sparingly and wouldn't introduce custom iterators to a project if I didn't have to. Old-school for loops and indexing can be cumbersome at times but even a monkey understands what it's doing.
3:24 what happens if we just call wg.Done() after calling process? Ie without defer?
The only difference is that defer will execute even if `process` panics. Without defer, it would possibly cause a deadlock
defer is a little safer, especially if the logic changes later on. However, in this example, it'll also work fine.
I love how youtube decides i should see these replies only after 6 days
the less boilerplate the better IMO. makes it easier to understand and to learn
So it took them just 15 years to discover iterator infrastructure... Now we wait for local-scope level for defer (that is calling on end of the current scope, not the whole function), mutual module inclusion, configurable ciphersuites in TLS1.3, modern error handling and a few others.
Another 15 years for null safety!
@@dreamsofcode is null safety really needed? I mean, c++ doesn't have it and also you have to deal with null pointers. I think Go was designed with that kept in mind
The thing I'm waiting for the most is sane enums. Just do what Rust is doing in this context
@@dreamsofcode Null safety? I'm merely asking for const ptrs... func (hs *const HugeStruct) CantDamageHS() {}
@@tomas120 C++ has references that guarantee null-safety (if not abusing some UBs), if you fear pointers.
MANN I WAS JUS ABOUT TO COMMENT IF U COULD MAKE MORE GO ADVACNED CONTENT SOO GOOOD
need this shit.
I need a slice keyword because i’m slow and confuse a lot of things 😂. This feature is great but i need something more declarative where im just calling a keyword & it just works! Yes im a bad developer but swiftui’s declarative nature has ruined me and made me lazy 😂. No more imperative button code. You just call button and it’s there. I need this in a package or more preferably in the language. Ok it started off minimal but when i first used rust is was basic, now you barely see lifetimes because they have slowly abstracted them into keywords where you can literally build your own state manager with miner lifetime usage to get it done. Pack the language with declarative stuff! rant over. 😂
I don't know much about Go, so forgive this potentially stupid question: are these basically generators as defined in Python?
Looks a lot like what is done in C# / Dotnet. I like the comment you mention, it's hard to tell how the underlying code is run. I would say good naming could help with this. But for love of god, please don't do like dotnet, by making everything a Task and have async / await all the way up the stack.
that's a lot of steps to run a par...
I think Go is only fun when you learn about the syntax. When I use it, I don't get it, it's like a primitive thing. Most Go developers won't update Go to a new version because they don't need it.
Still waiting for arena in go
ah yes, features that exist in other languages for years
As it stands, I'm not sure I like it. Go's power comes from the verbosity, and obfuscating that behind a convenience interface seems more like other languages for little benefit
Yeah agreed. I'm in two minds about it.
I think for the standard library to tidy up the many iterators patterns we currently have is a good thing, but for concurrency it might hide too much.
Have you ever used them? There is not much hidden. All the logic behind these functions are obvious.
Did they hire a JavaScript programmers in the core team?
Personally, I prefer generators in the style of python for something like this (e.g. the nicety of just using "yield" instead of this strange function stuff), but I guess this'll do.
I think the best part that will come out of this is watching all the gophers complaining for years about how this too complex... "What's next???? Usable Enums?!?!? Preposterous I say"
wow, Go has finally got yet another feature that exists in other languages for like forever
Nice video, but I'm too stupid to understand it.
oh... they finally discovered operations on iterators...?
I hope you start making Rust content
WE DON'T WANT TO CHANGE THE WAY TO WRITE LOOPS! THAT'S THE ENTIRE POINT OF GO!
Go lang and programmers slowly realizing have some more features may actually be good 😆 Watch it basically doing what other languages do but worse because they didnt accept complexity from the start and think "simplicity" is cutting corners
Even "they didnt accept complexity from the start" is debatable.
if err != nil all over the place.
Ah yes, another one screaming "quit having fun"
in the end it’ll have to add the other stuff anyway. So i like how it started but they need to rev-up some new declarative functionality. We are in the age of AI. This function should just work. I shouldn’t need to know the details of it unless i click through the different function signature docs in the ide.
This is not the go way.
One of the key distinguishing features of go is its relentless simplicity.
We don't need new syntactic sugar.
What we need is more speed and less memory consumption.
While I've always been a critic of Go for being anemic in features, this one feels out of touch with the rest of the language. It shies away from Go's main idea of simplicity.
Govascript 😅
the syntax is terrible
I dont like it
uu can this change it so that we don't write fucking loops no more? :D
Congrats on making Go look just as shitty and unreadable as Scala. Generics was such a mistake, dear lord.
Rust will be easier to use than this.
They could have simply added generator rather than this ugly shit. A simple yield statement is simpler and far more readable
There is no point in using Golang, when there is Rust. Go is just worse at everything
I like Rust, but I think they both have their own strengths and weaknesses.
There's no equivalent yet for Go+Templ+HTMX when it comes to Rust. Additionally, Concurrency in Go is better than async in Rust imho.
@@dreamsofcode Agreed. Async and lifetimes are great ideas, but in practice they make everything exponentially more complex and boring to read and write as a dev. Rust did simplify lifetimes as much as they could, though... maybe it's the same with async, but the end result is still far from enjoyable or productively readable.
May the Almighty God bless your steadfast faith in Holy Rust ✝🛐
I hate golang, for very deep reasons. From the fact that it’s shitty by design, and the fact they cover it up with “simplicity” (see clojure for REAL simplicity, and not just easiness), all the way to the fact that this language forces programmers to be primitive code monkeys that cant do anything besides write lines of unexpressive gibberish. verbosity language has the repetitiveness and verboseness of assembly, but without the benefits that verbosity gives to other languages. It has the shortsighted sloppy design of C, but without the ability to do what C is capable of.
I hope they keep piling features on top of Go so that all these new features will eventually reveal more and more of the fundamental problems with the language, just like once happened with C++. Trying to evolve C is a bad idea, it’s proven by all the garbage latguages we have today. The only ones that are designed relatively well are Erlang, Clojure and Common Lisp.
Calm it down..
I think the languages you mentioned as having good design, has horrible syntax and is tough to read.
We all have different opinions, but nobody agrees that C is sloppy or shortsighted. That's simply not correct especially given the history of things.
Take a chill pill mate
What a salty way to praise your personal taste.
The beauty of Haskell, Erlang, Clojure... comes from the fact that they tolerate the problem of copying data everywhere. And eat up as much memory as possible.
If these languages try to solve this problem...then they will fall into what you call: bad design.
@@baxiry. Clojure solved this problem without falling into bad design. Please don't show your lack of knowledge. Clojure data structures don't copy data everywhere thanks to what they call „persistent data structures”. Erlang employs this technique too, where it's important. I never mentioned Haskell, you did. I don't like Haskell.
Do your research before talking. Study efficient persistent data structures, hash-array mapped trie, rbb trees and others. There even are implementations in C++ (see „immer” library), where these exact data structures allow for massively efficient code (See the talk „Postmodern immutable data structures”).
What you described is false and your opinion is based on misinformation. Have a good day.