- Видео 2
- Просмотров 51 732
Zico Kolter
Добавлен 8 сен 2013
NeurIPS 2020 Tutorial: Deep Implicit Layers
This is a video recording of our NeurIPS 2020 Tutorial - Deep Implicit Layers: Neural ODEs, Deep Equilibrium Models, and Beyond - by David Duvenaud, Zico Kolter, and Matt Johnson. Full information about the tutorial, including extensive notes in Colab notebook form, are available on our website: implicit-layers-tutorial.org
Просмотров: 46 982
Видео
Beyond "provable" robustness: new directions in adversarial robustness
Просмотров 4,8 тыс.4 года назад
ICLR 2020 Towards Trustworthy ML Workshop Talk
Thank you for the presentation was really useful.
I wonder how much can be done here with stochastic continuous evaluations in the spirit of MCMC or recent "Walk on Stars" style evaluations, where you don't have any discretization error at all, but trade that off with some noise...
I am a graduate student in Physics. This video is clear, easy to follow and highly informative. Many thanks for making this video public! This is very helpful for me
Thanks for making this video public. The explanations are very intuitive and clear.
Great video!
Thank you for a very informative video. I have a very limited mathematics background and was wondering if there are any good resources to better understand the differentiation in ODE. Please let me know if have such resources if you see my comment. Cheers!
I like this tutorial very much!
earned like + sub at min. 1.47
like
Great tutorial and notes!
shouldn't the last partial differentiation at 54:00 in backward pass be d1(z*,x,theta) ? its written d2(z*,x,theta)
Thank you very much for sharing this amazing tutorial!
50:56
Thank you for sharing this.
Thank you guys! Very solid video, and good tempo. You present the material with a smile in a very user-friendly manner, that's a rare delicacy :) I wish new successes for your trio in the coming year. Separate thank you for the website and the code! I think I will try to apply DEQ to image denoising.
Great Idea.
Hi Zico Kolter, great work! ....What about the inference time of DEQs w.r.t DNNs? Are they similar? ...Another question Do you recommend to use JAX instead PyTorch or Tensorflow2?
Hi, I'm just getting started with DILs/DEQs but from what I can tell, their inference time tends to be x2 slower when compared to DNNs. Still, depending on your application it might not be important at all; e.g. in my case we are interested in processing requests on the minute, while a feed-forward DNN takes milliseconds to do inference, so doubling the milliseconds is not going to be a problem. In fact, our hope is that solving the optimization problem directly via this method will save time overall (compared to DNN + optimization algorithm).
Very cool idea!! Congratulations! and thanks for the tutorial.
Thanks for the tutorial! I have a question about the representations created by DEQs, in normal Deep Networks depth means you can compose features and deeper layers are supposed to have higher level representations, does the same story apply for DEQs or is there a similar way to understand its computation?
This work is amazing! When I saw GPT-3 use 175 billion parameters to build a language model, just feel hopeless. It's more fair to compete state-of-the-art performance based on model complexity.
I learn a lot. Thank you very much. There are 2 questions about DEQ. 1. Why does equilibrium point z* matter? How is z* better representation than any intermediate representation z_t? 2. ALBERT is BERT but share the weight by all transformer layers. How DEQ save memory sounds like ALBERT computes the gradient of only last layer and update the "shared" weight. ALBERT actually computes all gradients of all layers and update the "shared" weight by average of gradients. Why does DEQ work even though it doesn't care of gradient of intermediate layer?
Thanks for the questions. For 1) this is mainly just an empirical issue, but in practice we do see that "deeper" networks (even in the weight-tied setting) do appear to work better, and thus the equilibrium point works best as the final representation (plus allowing efficient differentiation). 2) Yes, ALBERT would store all the intermediate activations, and compute gradients through the whole unrolled network. The idea of the DEQ model is that this is actually unnecessary, though, precisely via the implicit differentiation method we discuss in the tutorial.
Awesome, but closed caption is little bit out sync. Could you sync it?
Thanks for pointing this out! We've re-uploaded them to properly sync. They should work correctly now.
Really cool. One question though? What is the fuss about neural ODEs? Honestly, I think I am missing something. They look just as taking a fireing rate model as an RNN... What is the difference?
Awesome! Really well presented!
Nice talk, thank you!
Ah, the old single pixel attack...
Awesome. Can I get slides for this video?
Just found out that videos on RUclips is much more stable than ICLR streaming.