It's gonna take a lot of doing to make that feasible but I'm really curious what could happen with this attentional type of processing for multimodal data. Like, imagine you could scrub the web like they did for GPT-3 but include not just text but also images. Entire illustrated books. Embedded videos with spoken language. Language is fundamentally dependent on the real world. It's crazy how far we can get with *just* text but I'd imagine a lot of things could be easily disambiguated if words aren't just typed but also heard or in the context of other stuff. So making attention more efficient for images is a solid step towards something like this and I'm really looking forward to what'll come of it.
Google actually has a trained NN that accepts all sorts of inputs (images, text, etc). The idea was to have a single model for everything. I cant remember the name of it though.
I think we are very close to something like this but positional embedding should become something general then. Like context embedding. Something like an Image caption should be associated with both the image and the text referring to the image. Maybe after that, this will be possible.
27:00 I think a better interpretation would be. "When I am at this position, I am more important. Or less important." Also attention based models are inherently interpretable compared to convolution based models. So, I think these will win out in the long run. Perhaps we can have a hybrid of CNN and attention.
To an extent convolutional models can also be analysed to see what parts were the most excited (and contributed to the final prediction). The other main advantage - and the reason I think we will at least have some hybrid of conv + attention - is that convolutions are much more parameter-efficient than FC or self-attention layers.
When convolutions go deep, they seem not only to be more efficient but also condense information in various abstract and profound ways. Certainly the Attention layers need more efficiency.
As the subscript implies, there seems to be a positional embedding r_p for every output position o. Then I'm not sure if that would be memory friendly.. Having relative positional embeddings for every pixel seems intense.
In times when SOTA is unfortunately king to young reviewers, I can appreciate the authors' need to perform well at least within some class of models. Imagine the frustration when all you offer to the community is a competitive alternative, only for the reviewer to retort it's not the best tool by an arbitrary margin. Great video.
Yep. A lot of approaches scale very poorly requiring exponentially more resources the more data you have. So there's a lot of experimenting to try to get around that major limitation.
![[ECCV 2020 Spotlight] Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation](http://i.ytimg.com/vi/-iAXF-vibdE/mqdefault.jpg)
It's gonna take a lot of doing to make that feasible but I'm really curious what could happen with this attentional type of processing for multimodal data.
Like, imagine you could scrub the web like they did for GPT-3 but include not just text but also images. Entire illustrated books. Embedded videos with spoken language.
Language is fundamentally dependent on the real world. It's crazy how far we can get with *just* text but I'd imagine a lot of things could be easily disambiguated if words aren't just typed but also heard or in the context of other stuff.
So making attention more efficient for images is a solid step towards something like this and I'm really looking forward to what'll come of it.
Google actually has a trained NN that accepts all sorts of inputs (images, text, etc). The idea was to have a single model for everything. I cant remember the name of it though.
@@felipemello1151 Google Brain I think, but I'd imagine there was quite some progress since
I think we are very close to something like this but positional embedding should become something general then. Like context embedding. Something like an Image caption should be associated with both the image and the text referring to the image. Maybe after that, this will be possible.
27:00 I think a better interpretation would be. "When I am at this position, I am more important. Or less important."
Also attention based models are inherently interpretable compared to convolution based models. So, I think these will win out in the long run. Perhaps we can have a hybrid of CNN and attention.
To an extent convolutional models can also be analysed to see what parts were the most excited (and contributed to the final prediction). The other main advantage - and the reason I think we will at least have some hybrid of conv + attention - is that convolutions are much more parameter-efficient than FC or self-attention layers.
@@socratic-programmer Also local connectivity at low level visual features make perfect sense.
Are we a couple of million dollars of Neural Architecture Search away from the end of convolutions???
When convolutions go deep, they seem not only to be more efficient but also condense information in various abstract and profound ways. Certainly the Attention layers need more efficiency.
Thanks for the video Yannic. Amazing explanation!
Beautiful. Thank you Yannic
Check out Tabnet... Attention is coming for tabular data as well...
7:55 Yannic just predicted the Perceiver architecture. Madman.
Nicely explained! 👍
As the subscript implies, there seems to be a positional embedding r_p for every output position o. Then I'm not sure if that would be memory friendly.. Having relative positional embeddings for every pixel seems intense.
are there any code using this for segmentation?
I don't think so
Did you checked out MaX-DeepLab paper? It will be nice if you can make a video on that.
thanks!
You should do a video on TabNet for tabular data using neural nets. I feel like there's a lot there and the explanations online kind of suck.
can someone explain to me what does axial means? :) have a hard time getting it
In times when SOTA is unfortunately king to young reviewers, I can appreciate the authors' need to perform well at least within some class of models. Imagine the frustration when all you offer to the community is a competitive alternative, only for the reviewer to retort it's not the best tool by an arbitrary margin.
Great video.
I enjoyed a great video let's be friends 😊😊😊😊
우리 친구 해요.!!!~~^^
Does anyone know what Yannic uses to be able to draw and edit over the PDFs?
OneNote
Attention is the new MLP when you are rich
"What's a good number?"
So many tricks for reducing computational powers lately. Intuitive but also qustionable
Yep. A lot of approaches scale very poorly requiring exponentially more resources the more data you have. So there's a lot of experimenting to try to get around that major limitation.
These names are getting out of hand
with transformer dominating ranking indexes, light weight neural network might be a thing of the past.
what do you mean lightweight neural network?