Error Embraced: Making Trustworthy Scientific Decisions with Imperfect Predictions

Clara Wong-fannjiang (Genentech)
simons.berkeley.edu/talks/cla...
AI≡Science: Strengthening the Bond Between the Sciences and Artificial Intelligence
Machine learning has demonstrated great promise in helping to accelerate scientific discovery and decision-making. However, machine-learning predictions contain errors. How can we make trustworthy scientific discoveries and decisions in spite of prediction error? This talk will share a line of work motivated by this question. The first part of the talk will focus on prediction-powered inference, a novel framework for performing valid statistical inference when a gold-standard data set is supplemented with predictions from a machine-learning system, without making any assumptions about the system. Prediction-powered inference may enable scientists to draw valid conclusions in a more data-efficient way, as we demonstrate with applications in proteomics, genomics, and astronomy. In the second part of the talk, I will touch on our recent efforts building upon these ideas to make reliable machine learning-guided decisions in biological sequence design, and beyond.

[@drdca8263]

So far I’m only 24 minutes into watching this, but, this seems like a really important idea?
In addition to the clear practical application, it seems to me like this should also imply something about like, the theory of the development of scientific theories ?
If we replace the ML model with, e.g. Newton’s law of universal gravitation (regarding “observations Isaac Newton knew about before publishing anything about the universal gravitation” as the “training set” used to produce that theory, when we consider the requirement that the gold-standard data be independent of the training set...
uhh...
I guess this should give a...
hm, maybe this isn’t as applicable to this case as I thought. Still, seems very important!)
Edit: I suppose the points at 33:00 - 35:15 should temper my uh, somewhat wild imaginings for how widely this could be applied
.. and also she goes on to point out connections to previous literature dealing with somewhat similar things that I hadn’t at all heard of,
I guess one can tell that I haven’t really studied statistics in much depth..
Nonetheless, I continue to be of the opinion that this is *very* cool.