Hi Zico, got to know you on 2024's NeurIPS MLNCP workshop, and wasn't familiar with DEQ until then. I'm very glad to have found this material. It's got everything: theory, simple examples, code implementation, supplementary material. Besides, I'm a big fan of the way the three of you convey your points, it's always very clear and adequate. Thank you!
I am a graduate student in Physics. This video is clear, easy to follow and highly informative. Many thanks for making this video public! This is very helpful for me
Thank you guys! Very solid video, and good tempo. You present the material with a smile in a very user-friendly manner, that's a rare delicacy :) I wish new successes for your trio in the coming year. Separate thank you for the website and the code! I think I will try to apply DEQ to image denoising.
I wonder how much can be done here with stochastic continuous evaluations in the spirit of MCMC or recent "Walk on Stars" style evaluations, where you don't have any discretization error at all, but trade that off with some noise...
I learn a lot. Thank you very much. There are 2 questions about DEQ. 1. Why does equilibrium point z* matter? How is z* better representation than any intermediate representation z_t? 2. ALBERT is BERT but share the weight by all transformer layers. How DEQ save memory sounds like ALBERT computes the gradient of only last layer and update the "shared" weight. ALBERT actually computes all gradients of all layers and update the "shared" weight by average of gradients. Why does DEQ work even though it doesn't care of gradient of intermediate layer?
Thanks for the questions. For 1) this is mainly just an empirical issue, but in practice we do see that "deeper" networks (even in the weight-tied setting) do appear to work better, and thus the equilibrium point works best as the final representation (plus allowing efficient differentiation). 2) Yes, ALBERT would store all the intermediate activations, and compute gradients through the whole unrolled network. The idea of the DEQ model is that this is actually unnecessary, though, precisely via the implicit differentiation method we discuss in the tutorial.
This work is amazing! When I saw GPT-3 use 175 billion parameters to build a language model, just feel hopeless. It's more fair to compete state-of-the-art performance based on model complexity.
Thank you for a very informative video. I have a very limited mathematics background and was wondering if there are any good resources to better understand the differentiation in ODE. Please let me know if have such resources if you see my comment. Cheers!
Thanks for the tutorial! I have a question about the representations created by DEQs, in normal Deep Networks depth means you can compose features and deeper layers are supposed to have higher level representations, does the same story apply for DEQs or is there a similar way to understand its computation?
Hi Zico Kolter, great work! ....What about the inference time of DEQs w.r.t DNNs? Are they similar? ...Another question Do you recommend to use JAX instead PyTorch or Tensorflow2?
Hi, I'm just getting started with DILs/DEQs but from what I can tell, their inference time tends to be x2 slower when compared to DNNs. Still, depending on your application it might not be important at all; e.g. in my case we are interested in processing requests on the minute, while a feed-forward DNN takes milliseconds to do inference, so doubling the milliseconds is not going to be a problem. In fact, our hope is that solving the optimization problem directly via this method will save time overall (compared to DNN + optimization algorithm).
Really cool. One question though? What is the fuss about neural ODEs? Honestly, I think I am missing something. They look just as taking a fireing rate model as an RNN... What is the difference?
![BLACK BAG - Official Trailer [HD] - Only in Theaters March 14](http://i.ytimg.com/vi/Du0Xp8WX_7I/mqdefault.jpg)
Hi Zico, got to know you on 2024's NeurIPS MLNCP workshop, and wasn't familiar with DEQ until then. I'm very glad to have found this material. It's got everything: theory, simple examples, code implementation, supplementary material. Besides, I'm a big fan of the way the three of you convey your points, it's always very clear and adequate. Thank you!
I am a graduate student in Physics. This video is clear, easy to follow and highly informative. Many thanks for making this video public! This is very helpful for me
Thanks for making this video public. The explanations are very intuitive and clear.
Thank you guys! Very solid video, and good tempo. You present the material with a smile in a very user-friendly manner, that's a rare delicacy :) I wish new successes for your trio in the coming year. Separate thank you for the website and the code! I think I will try to apply DEQ to image denoising.
Thank you for the presentation was really useful.
I like this tutorial very much!
I wonder how much can be done here with stochastic continuous evaluations in the spirit of MCMC or recent "Walk on Stars" style evaluations, where you don't have any discretization error at all, but trade that off with some noise...
I learn a lot. Thank you very much. There are 2 questions about DEQ.
1. Why does equilibrium point z* matter? How is z* better representation than any intermediate representation z_t?
2. ALBERT is BERT but share the weight by all transformer layers. How DEQ save memory sounds like ALBERT computes the gradient of only last layer and update the "shared" weight. ALBERT actually computes all gradients of all layers and update the "shared" weight by average of gradients. Why does DEQ work even though it doesn't care of gradient of intermediate layer?
Thanks for the questions. For 1) this is mainly just an empirical issue, but in practice we do see that "deeper" networks (even in the weight-tied setting) do appear to work better, and thus the equilibrium point works best as the final representation (plus allowing efficient differentiation). 2) Yes, ALBERT would store all the intermediate activations, and compute gradients through the whole unrolled network. The idea of the DEQ model is that this is actually unnecessary, though, precisely via the implicit differentiation method we discuss in the tutorial.
This work is amazing! When I saw GPT-3 use 175 billion parameters to build a language model, just feel hopeless. It's more fair to compete state-of-the-art performance based on model complexity.
shouldn't the last partial differentiation at 54:00 in backward pass be d1(z*,x,theta) ? its written d2(z*,x,theta)
Awesome! Really well presented!
Great tutorial and notes!
Thank you for a very informative video. I have a very limited mathematics background and was wondering if there are any good resources to better understand the differentiation in ODE. Please let me know if have such resources if you see my comment.
Cheers!
Thank you very much for sharing this amazing tutorial!
Very cool idea!! Congratulations! and thanks for the tutorial.
Awesome, but closed caption is little bit out sync. Could you sync it?
Thanks for pointing this out! We've re-uploaded them to properly sync. They should work correctly now.
Thanks for the tutorial!
I have a question about the representations created by DEQs, in normal Deep Networks depth means you can compose features and deeper layers are supposed to have higher level representations, does the same story apply for DEQs or is there a similar way to understand its computation?
Thank you for sharing this.
Hi Zico Kolter, great work! ....What about the inference time of DEQs w.r.t DNNs? Are they similar? ...Another question Do you recommend to use JAX instead PyTorch or Tensorflow2?
Hi, I'm just getting started with DILs/DEQs but from what I can tell, their inference time tends to be x2 slower when compared to DNNs. Still, depending on your application it might not be important at all; e.g. in my case we are interested in processing requests on the minute, while a feed-forward DNN takes milliseconds to do inference, so doubling the milliseconds is not going to be a problem. In fact, our hope is that solving the optimization problem directly via this method will save time overall (compared to DNN + optimization algorithm).
Great Idea.
Really cool. One question though? What is the fuss about neural ODEs? Honestly, I think I am missing something. They look just as taking a fireing rate model as an RNN... What is the difference?
50:56
earned like + sub at min. 1.47
like