Видео

Nicely organized and neat introduction. Thanks!

Hi, My question is why it makes sense to assume that X and Y come from a joint distribution P, which to me at least would imply that X is random. In the frequentist setting like regression I am familiar with, one assumes that the predictors X are deterministic and only y is random, i.e. the conditional expectation of y given x is modelled. Can anyone explain the intuition behind this assumption? Is it just a more general assumption that doesn't contradict the frequentist view?

Thank you

Your videos are such great resources, thank you! I am just wondering, how do we know the theoretical distribution that the coverage should follow? Is there any resource that explains this in more detail? Thanks!

The hierarchical clustering and its relation to the Learn Then Test framework (LTT) was slightly unclear to me. Am I right in thinking the general procedure looks something like adaptive predictive sets for classification, but instead of moving through individual labels ordered in terms of probability mass to build your quantile you consider a sequence of large and larger subtrees? Or do you consider a threshold such that if a subtree has total mass greater than the threshold value you accept it and then employ the LTT framework directly to select a risk controlled threshold value?

Is the small diagram Anastasio's draws at ~30:00 slightly incorrect? Shouldn't the quantile line be horizontal?

Hi! In minute 13:00 with the first algorithm, in cases with no clear class detected it could be the case that no class has score greater than q_hat, so the output is an empty set. But in case of uncertainty I would expect bigger set, is this the expected behavior?

Truly a video of education. Thank you for taking the time to explain this concept clearly.

Hey there! Nice video. I recently ran some experiments on conformal prediction distillation and I was curious if you had any ideas or suggestions. In a nutshell, the goal was to see if problem-specific conformal prediction guarantees (e.g. coverage in classification) could be instilled into a model without the calibration step. I tested some knowledge distillation techniques (e.g. KL divergence for "soft" distillation loss) in training a smaller model trained on a dataset of (X,Y) pairs, where X is the initial input and Y is the a multiclass binary vector representing the prediction set from a calibrated prediction-set generator f on X. This approach seems to somewhat (?) work and is better than just a naive model without any calibration but does not nearly preserve the same amount of coverage guarantees as actually performing the calibration step. Any suggestions or thoughts on how to expand on this?

I'm a little confused in 18:11 step 3 is taking the scores < q_hat. It should be > q_hat according to 13:35 ?

You can explain difficult concepts more clearly than most of my professors from undergrad!

Wonderful and concise tutorial! Could you please elaborate on what 'finite sample validity' means? Is it something like this: 'Given that the training data is finite, and a portion of it is calibration data, which is also finite but smaller, one can create a conformal framework around that 'small' sample and still achieve coverage of (1-alpha)?'

Thanks for the video, it is very accessible despite the complexity of the topic. Indeed, I have a question: what about the application of conformal prediction to the case of binary classification? My doubt regards the size of the prediction set since in the case of binary classification we only have two labels and I guess having a coverage on a prediction set greater than one would be useless.

Thank you for this lecture.

Thanks a lot for this super -simple, elegant expansion on a topic that appears daunting. Hats off to you guys.

Hey great video! One portion I’m stuck at, now that I have a prediction set - now what? How do I get this into a single prediction? Also - I am using an ensemble approach to my modeling using multiple classification techniques, then averaging the softmax scores across the models to get my “final” prediction. Is this something I would do for each model, or just once the softmax scores are combined/averaged? Thanks!

Absolutely brilliant lecture series! Subscribed.

Beautiful indeed.

haha technically you guys are doctors, just not the kind cutting people open 😂

Thanks for the video and great work guys Anastasios and Stephen. This video really helped me to understand the concept behind conformal prediction quickly. I do have one question regarding CP for regression model. In the video, you have mentioned about training quantile regression models for (alpha/2) and (1-alpha/2). While this is possible by re-training the model based on pinball loss (I assume that the process remains the same for non-NN models as well), is there a way to get this quantile regression without re-training the model? I am particularly concerned with the cases where (1.) re-training is either very costly, or (2.) the model training is performed in an automated fashion (where changing the loss function is not possible).

Very well done! I read your "Gentle Introduction.." report as well, equally well done. Unfortunately I am left behind the academic paper wall when it comes to your ref[41] V. Vovk, “Cross-conformal predictors”. Hence I figured I'll post my question here: I am considering using Cross Validation (or nested) to obtain out of fold predictions for a train set, then use these as calibration data to calculate nonconformity_scores. I am withholding a test set outside the CV, for which I aim to evaluate final model and coverage. If all good, I retrain the model on all train data, but I am tempted to keep my nonconformity_scores.. Furthermore I could then get additional scores from the "unseen" test set using the final model. Combining these test scores with previous out of fold scores, I obtain scores for all data, betting on that my outer fold models produce comparable errors to the final version. Does this make sense? If not, how do you recommend making use of CV procedures in relation to conformal predictions?

what do you guys use for generating these slides? I really like the font and layout

Very interesting and informative!

Excellent lecture!

Awesome! I have been working on clinical trial outcome prediction recently. Do you think it is sensible or necessary to apply conformal prediction on a binary classification problem? I asked because sometimes I doubt the value of doing this. Well, a binary classification has only two classes and intuitively, it's kind of "small". And if it is worth doing, is the method in the vedio suitable, or do you think there're some other ways of quantifying the uncertainties for a two-class classification?

Fantastic video, thank you!

Great tutorials on Conformal Prediction, thanks a lot! I have a question regarding the algorithm to evaluate the performance. There, in line 4, you compute 1-D_cal.max(axis=1). Shouldn't the score be defined as in your first tutorial as 1-D_cal[np.arange(len(y_cal),) y_cal], i.e., the estimated probabilities of the true labels? Let's say we have a classifier which classifies the wrong class with a probability of almost 1. Then the scores are close to zero and hence our quantile. In line 6, the rhs would be close to one and hence only false prediction would be added to the set. So 'covered' computed in line 7 will mostly be false. Or am I missing something here?

how is it gentle introduction?

First, thank you for the clear explanation. Second, I have a remark/question. I have been reading about statistical analysis recently, and how there is a reccuring problem surrounding the confidence interval interpretation. One should not say: 'There is a 90% chance that our interval holds the true value of our statistic of interest', but instead: '90% of the intervals built with our method will hold the true value'. The mathematical reason for this is detailled in "The fallacy of placing confidence in confidence intervals". In practice, this mistake can lead to understimating uncertainty for important decisions (e.g. cancer diagnosis). So here is my question: have you considered whether there is the same problem for conformal prediction intervals/sets (e.g. is it truly okay to say that our prediction interval has a 90% chance of containing the true value?)? Have a nice day 😁

Great presentation! I'm curious, when applied to a multi-label problem, is the quantile function just setting a new decision threshold? The quantile is basically a decision threshold for which labels are predicted. And given an alpha, the quantile is a constant for all new predictions regardless of their matrix of dependent variables. So isn't this process basically just updating your decision threshold to account for a desired coverage probability?

So I know to get the upper and lower quantiles for regression, you use the pinball loss function. But does the loss for the point/mean prediction in regression have to be a specific kind of loss? Or can we use mse, mae, CE?

Extremely helpful, thanks so much!

Thank you so much for this informative and nice presentation. I am looking for resources to guide me how the end user benefits from a prediction set and make a decision upon this set in a typical classification problem?

Very informative video. Thanks

Great video! Anyone recognize the app they are using?

does this conformal prediction method for regression take covariance into account?

is there conformal prediction for regression problems?

Excuse me for being naive, but how do you get the prediction intervals/sets on unknown data with no labels? I understand that you calibrate the probabilities with the calibration set, but how do I know the prediction interval for model prediction yhat(X_i) where the ground truth y_i is not known?

Hi, really amazing explanation. I had a question, How do you do it for binary classification?

Very good video. I did enjoy it, thanks. Wrt to adaptive prediction sets and in the case of binary classification problems, Ei will be either <= 1 (the value of the first score in the list of sorted scores) or = 1. Is my intuition correct? I am also trying to figure out what would be the p values for a test example from a Mondrian conformal predictor. An explanation on that would be very appreciated.

Please what writing tool is he using?

📣Code for conformal prediction on real data! github.com/aangelopoulos/conformal-prediction The new codebase is part of a huge update to the gentle intro document: arxiv.org/abs/2107.07511 . It includes Imagenet classification, MS-COCO multilabel classification, time-series regression, conformalized quantile regression on medical data, and much more! Leave a ⭐if you enjoy it :)

I'm ill right now so I'm missing Volodymyr Vovk's lecture on this and I'm watching this video instead, thanks for making it!

These videos are a fabulous addition to the tutorial document - thank you! Will you add a example Colab notebook to you git repo for the object detection case?

Hi @Anastasios, I am going through the video and am quite eager to try it out. First question: does this method handle scenarios where a CNN predicts incorrectly but with high softmax scores

Just what I needed for my master's degree proposal! Full of ideas thanks to you guys! :)

First of all, Thank you for the great presentation! I have some questions concerning your examples for the classification tasks. 1. Did I understand it correctly, that it is necessary for the heuristic output (in your case softmax output) to add up to the same value (1 in your case) for all data points when all classes of the label space are considered? 2. I tried to adapt your method with the adaptive prediction sets on my binary classification problem at hand (more precisely it is a anomaly detection problem). I observed the q threshold value to become very high resulting in the situation that in almost all cases the prediction set will contain both classes. Obviously this doesn't have much added value for me. I'd like to outline my approach. I used a Variational Autoencoder and trained it with normal data only. I have a mixed test set and computed the reconstruction probability (RecProb) for every data point. I was able to find a optimal decision threshold for anomaly detection by maximizing the Matthews Correlation Coefficient, but unfortunately the RecProbs of anomalous data points are only marginally greater than those of normal data points, so there is a significant amount of overlap. That's why I am looking for a (ideally not very complicated) approach to indicate the uncertainty of my model in some way. Now I thought about doing some kind of comparison of the relative frequencies for normal and anormal data for a given RecProb looking at the corresponding histogram bin. It would be very helpful for me, if you could maybe give me some advice. Many thanks in advance and keep up the great work! Regards, Tobias.

Thanks for this video. Very informational and to the point.

Fantastic presentation! I was captivated from start to finish

Really wonderful complement to your paper "Gentle Intro to Conformal Prediction..". One question! - for each of the example algorithms, you create a three-box summary of what the algorithm does. In the first of these three boxes (for each of the algorithms), you have "score" on the y-axis of the histogram. However, I feel as though this should be named "heuristic-output" or, for example, "softmax output" (for multi-class classification). To me, "score" means the "E" in that first box, i.e., it is defined differently for each algorithm, and encodes the properties we want the prediction set function to have. Correct me if I'm wrong! I may be misunderstanding. Thanks again!