Attention Approximates Sparse Distributed Memory

[@konradhudson4610]

Trenton is clearly someone to keep an eye on within this field; I look forward to seeing how his research develops

[@maxreeder]

🤯 blew my mind even more than “Neural Ordinary Differential Equations”

[@trentonbricken2731]

thanks and glad you found it interesting :)

[@JayPatel-ne6jf]

This is beautiful! I wonder why memory models keep falling out of use after a few years in the ML community. First it was hopfield networks/boltzmann machines and later SDM.

[@trentonbricken2731]

Thanks for the kind words Jay. As I said during this talk (it might have been cut out at the end though) I am still unsure why SDM disappeared with the sands of time to the extent it did and Hopfield networks took so much more of the limelight.

[@AA-gl1dr]

brilliant presentation. Thanks for uploading

[@trentonbricken2731]

thanks for the kind words!

[@pisoiorfan]

I see a next paper titled "Pattern searching is all you need"

[@trentonbricken2731]

👀

[@444haluk]

I could swear I am hearing Jeff and Subutai asking questions

[@randomgamingstuff1]

Yeah this stuff seems to line up nicely with what I've been hearing Numenta talking about recently

[@trentonbricken2731]

funnily enough a friend who works at Numenta first got me to read Kanerva's Sparse Distributed Memory

[@CharlesVanNoland]

@@randomgamingstuff1 Numenta has been talking about sparse distributed representations for 20 years, which I guess is "recently" in the grand scheme of artificial intelligence.

[@trentonbricken2731]

Slides for the talk available here! docs.google.com/presentation/d/1gGErBUIDM5SQCroovgTGmwgaz61EZ6RdN-eg8niCb0w/edit?usp=sharing

[@faster-than-light-memes]

That's cool!

[@StochasticCockatoo]

Is this the same as maximum entropy reinforcement learning? AND SOME DEFINITIONS OF INTERESTINGNESS?

[@StochasticCockatoo]

FYI Trenton is edgy friendly I think this is an important detail

[@StochasticCockatoo]

ADHD IS ALL YOU NEED