It's fascinating to revisit this after three years. Must say, by throwing in more parameters, we indeed have made incredible progress in truly understand languages.
The device that establishes interconnection between GPUs within a node is called NVLink. InfiniBand is used to establish connection between different nodes.
Can we get a videos series on the graveyard of machine learning why ideas like synthetic gradients and capsule-nets have gone dormant the deep learning space.
My bet is those ideas never worked in the first place. Yes, they "work" in their papers, but you can get anything to work for a paper. They will probably keep re-appearing every couple of months because someone combines them with something else and wants a bit of name recognition. In very seldom cases, someone will figure out how to make one of these actually work, like it happened with neural networks (alexnet) or GANs (goodfellow). These ideas were around long before, looking like crap. But so were 1000 others that actually are crap.
Interesting to note, that when compared to pegasus-large, this didn't even perform better on all the tasks. A bit misleading to have a column in the performance evaluation called "Previous SOTA" while there's another column in the same chart that beats the "Previous SOTA". And this has ~30x more parameters than the pegasus large model. If this isn't evidence that better training beats raw size, I don't know what is. However, this is a cool way of training that allows for very big models with low overhead. On top of that, with this technique, size should scale approximately linearly with the number of GPUs+machines (if you use multicast routers). It makes techniques that use self-supervised local loss to avoid full-model backpropagation seem a lot less necessary. All things considered, I think the next important step is learning to distill such large models into smaller models with similar performance. Primarily, this is needed for inference, because 17 billion parameters has a hard time fitting on GPUs. But also, this might be able to extend training beyond it's initial bound - perhaps distilling a model and adding more layers over again in a loop.
Hi, in the first approach of Data Parallelism without ZeRO, is it sequential? Meaning is Data0 first fed to the network, then backwards propagations is done, then all new parameters are sent to the other GPUS (GPU_1 to GPU_n-1) ? And then Data1 is fed to the network stored in GPU_1 sending its own updated parameters to GPU_2 etc. I feel like that would be not be optimal. But I do not see the intuition behind the parameters sharing across GPUs. Any idea how that would work? :)
My understanding from the video is that the parameters are shared across GPUs because you are trying to train the same model with multiple GPUs. Without sharing/synchronizing parameters, you would be effectively training N separate models, which would become difficult-to-impossible to reconcile quite quickly.
It's a little bit better...but maybe it's sentient (joke). But seriously, I think the memory footprint of things like BERT is already pushing it. So much so that the things I'm most interested in in the last year are things like DistilBert. But even that is too much for the CPU peasants out there of which there are many. Having said that I do trust that leaders in the field are aware that the "No replacement for displacement" parameter dick measuring contest is not the endgame solution.
![Seungmin "그렇게, 천천히, 우리(As we are)" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/kAzmhLHePqU/mqdefault.jpg)
watching now and hearing 17b is _huge_ really makes you feel the time passing
It's fascinating to revisit this after three years. Must say, by throwing in more parameters, we indeed have made incredible progress in truly understand languages.
Thanks for such an interesting overview of the concept. Probably the only video with decent quality I've managed to find on the youtube. Thank you!
The device that establishes interconnection between GPUs within a node is called NVLink. InfiniBand is used to establish connection between different nodes.
Very interesting. Thanks for the great explanation! Ill read more on this
9:01 on board network connecting gpus is called nvlink
Amazing video, with amazing achievements in AI. Great work Yannic. Thanks a lot !!
Pretty clear explanation on ZeRO. Appreciate it.
the zero optimizer is described at 11:37
thanks for making this so simple to understand
Can we get a videos series on the graveyard of machine learning why ideas like synthetic gradients and capsule-nets have gone dormant the deep learning space.
My bet is those ideas never worked in the first place. Yes, they "work" in their papers, but you can get anything to work for a paper. They will probably keep re-appearing every couple of months because someone combines them with something else and wants a bit of name recognition.
In very seldom cases, someone will figure out how to make one of these actually work, like it happened with neural networks (alexnet) or GANs (goodfellow). These ideas were around long before, looking like crap. But so were 1000 others that actually are crap.
Capsules were the main topic of Geoffrey Hinton at the recent 2020 Turing Awards vimeo.com/390347111 at about 1:30:00
Very clear explanation. Thanks!
Thank you Yannic 💕💕
Interesting to note, that when compared to pegasus-large, this didn't even perform better on all the tasks. A bit misleading to have a column in the performance evaluation called "Previous SOTA" while there's another column in the same chart that beats the "Previous SOTA". And this has ~30x more parameters than the pegasus large model. If this isn't evidence that better training beats raw size, I don't know what is.
However, this is a cool way of training that allows for very big models with low overhead. On top of that, with this technique, size should scale approximately linearly with the number of GPUs+machines (if you use multicast routers). It makes techniques that use self-supervised local loss to avoid full-model backpropagation seem a lot less necessary.
All things considered, I think the next important step is learning to distill such large models into smaller models with similar performance. Primarily, this is needed for inference, because 17 billion parameters has a hard time fitting on GPUs. But also, this might be able to extend training beyond it's initial bound - perhaps distilling a model and adding more layers over again in a loop.
Hi, in the first approach of Data Parallelism without ZeRO, is it sequential? Meaning is Data0 first fed to the network, then backwards propagations is done, then all new parameters are sent to the other GPUS (GPU_1 to GPU_n-1) ? And then Data1 is fed to the network stored in GPU_1 sending its own updated parameters to GPU_2 etc.
I feel like that would be not be optimal. But I do not see the intuition behind the parameters sharing across GPUs. Any idea how that would work? :)
My understanding from the video is that the parameters are shared across GPUs because you are trying to train the same model with multiple GPUs. Without sharing/synchronizing parameters, you would be effectively training N separate models, which would become difficult-to-impossible to reconcile quite quickly.
"AND.... it is a bit better!"
NLP is slowly going the way of imagenet.
@@YannicKilcher Absolutely. Single GPU plebs like me will have to stick with toy problems for now!
great video!
as epistemological perspective: commonly known (objective history) facts can be manifactured by total possibility language space
It's a little bit better...but maybe it's sentient (joke).
But seriously, I think the memory footprint of things like BERT is already pushing it. So much so that the things I'm most interested in in the last year are things like DistilBert. But even that is too much for the CPU peasants out there of which there are many. Having said that I do trust that leaders in the field are aware that the "No replacement for displacement" parameter dick measuring contest is not the endgame solution.
I think to the big companies, it's mostly a recruitment advertising platform.
Awesome!
It's somewhat similar to microprocessor pipelines. Probably there are more pipelining tricks to adapt.
I thought the same thing. The classic laundry machine example is exactly what I thought of during the naive-ish model splitting section.
First one on TNLG
"I don't know"'
🤷