Neural ODE - Pullback/vJp/adjoint rule

How do you backpropagate through the integration of a Ordinary Differentiational Equation? For instance, to train Neural ODEs to fit data. This requires the solution of an adjoint ODE running backward in time. Here are the notes: github.com/Ceyron/machine-lea...
-------
👉 This educational series is supported by the world-leaders in integrating machine learning and artificial intelligence with simulation and scientific computing, Pasteur Labs and Institute for Simulation Intelligence. Check out simulation.science/ for more on their pursuit of 'Nobel-Turing' technologies (arxiv.org/abs/2112.03235 ), and for partnership or career opportunities.
-------
📝 : Check out the GitHub Repository of the channel, where I upload all the handwritten notes and source-code files (contributions are very welcome): github.com/Ceyron/machine-lea...
📢 : Follow me on LinkedIn or Twitter for updates on the channel and other cool Machine Learning & Simulation stuff: / felix-koehler and / felix_m_koehler
💸 : If you want to support my work on the channel, you can become a Patreon here: / mlsim
🪙: Or you can make a one-time donation via PayPal: www.paypal.com/paypalme/Felix...
-------
⚙️ My Gear:
(Below are affiliate links to Amazon. If you decide to purchase the product or something else on Amazon through this link, I earn a small commission.)
- 🎙️ Microphone: Blue Yeti: amzn.to/3NU7OAs
- ⌨️ Logitech TKL Mechanical Keyboard: amzn.to/3JhEtwp
- 🎨 Gaomon Drawing Tablet (similar to a WACOM Tablet, but cheaper, works flawlessly under Linux): amzn.to/37katmf
- 🔌 Laptop Charger: amzn.to/3ja0imP
- 💻 My Laptop (generally I like the Dell XPS series): amzn.to/38xrABL
- 📱 My Phone: Fairphone 4 (I love the sustainability and repairability aspect of it): amzn.to/3Jr4ZmV
If I had to purchase these items again, I would probably change the following:
- 🎙️ Rode NT: amzn.to/3NUIGtw
- 💻 Framework Laptop (I do not get a commission here, but I love the vision of Framework. It will definitely be my next Ultrabook): frame.work
As an Amazon Associate I earn from qualifying purchases.
-------
Timestamps:
00:00:00 Neural ODE integration in a wrapper function
00:01:15 Scientific Computing Interpretation
00:01:30 Only interested in the final time value
00:02:22 Task: Backpropagation of cotangent information
00:03:53 Interpretation of the input cotangents
00:05:17 Without unrolling the ODE integrator (we want OtD instead of DtO)
00:07:59 General Pullback or vJp definition
00:13:39 (1a) Parameter Cotangent: Starting with ODE constraint
00:14:18 (1b) Total derivative wrt parameter vector
00:16:31 (1c) Inner product with adjoint variable
00:21:57 (1d) Integration by Parts
00:24:13 (1e) Move right-hand-side Jacobian
00:26:09 (1f) Investigating the limit evaluation
00:28:13 (1g) Adding an artificial zero
00:30:38 (1h) Identify the adjoint problem
00:39:39 (1i) Discussing the adjoint problem
00:45:59 (2a) IC cotangent: Starting with ODE constraint
00:47:33 (2b) Total derivative wrt initial condition
00:49:04 (2c) Inner product with adjoint variable
00:50:15 (2d) Integration by Parts and moving the Jacobian
00:53:27 (2e) Add artificial zero
00:55:31 (2f) Identify the adjoint problem
00:59:30 (2g) Discussing the new adjoint problem
01:02:47 (3a) Final Time Cotangent: Starting with general solution to an ODE
01:03:51 (3b) Total derivative wrt "T"
01:05:21 (3c) Build vector-Jacobian product
01:07:13 Full Pullback rule
01:11:35 No adjoint problem needed if only interested in final time cotangent
01:12:22 Adjoint Problem can be stepped through if only interested in IC cotangent
01:13:53 Parameter cotangent needs full adjoint trajectory
01:14:29 How to evaluate the vJp of the ODE dynamics
01:17:08 (1) Save primal trajectory and interpolate
01:21:27 (2) Run primal problem reversely alongside adjoint ODE
01:26:51 How to evaluate the functional inner product for the parameter cotangent?
01:28:51 Introduce another auxiliary ODE problem to accumulate the quadrature reversely in time
01:34:18 One large ODE running reversely in time
01:39:23 Summary
01:41:57 Outro