Видео

Would you be able to provide any guidance on videos, papers, blogs etc. that are using conformal prediction for unsupervised algorithms? Thanks!

1:00

really interesting 👏👏👏👏

Thank you for the explanation, this is of high quality

Amazing!

This is great!! Thanks!

Thank you for the video. However I didn’t understand the ladder about the conformity scores. You say that it “Shows the ranking of the points in the sorted non-conformity array and not the non-conformity values themselves.” But how do you sort them if not according to their non-conformity value?

Amazing videos! I am wondering when the next one is coming out? How do I properly check for coverage when applying conformal predictors?

I was curious if heteroskedasticity would be a problem. I looked it up and someone had the same question and found a solution. See "Distribution-Free Predictive Inference For Regression" by Lei, G'Sell, Rinaldo, Tibshirani and Wasserman (2017).

After watching this series a few times, the thing that (still) confuses me is the fact that for (100 % of training data + each new point) we have to fit a new model each time for each plausible label, but for (percentage of training data or calibration data + each new point) we don't. It looks to me that in both cases test point could be out of bag, be it full training data or e.g. 20% of training data (calibration dataset). I see that everything is the same in terms of implementing CP, but the difference is only in the number of points (training vs calibration).

Very interesting, thank you for the video!

Awesome video series. Thanks!

Absolutely brilliant presentation!

if the square is a test point why the model need to be fit accounting for it? Thanks for the video

God I have sinned, of the 70th like. Pls forgive me. 🛐 Amen.

How is this different to a quantile approach with X% confidence intervals? I guess the quantile approach would only meet some but not all of the requirements mentioned😅. Interesting stuff.

Very good video. Thanks for making this!

Excellent 👌 explanation

also hope to cover this ??? one thing its important is that its probably is not dynamic for point, for i.e. if the loss of a point in the dataset is .098 is in 90% interval, so the other points with the same loss are also in 90% interval, but in more dynamic quantifier, a point with this loss may have 60% interval or 93% interval, I mean `conformal predictor` doesnt take to account the Uncertainty Quantification of input space, so model agnostic and distribution free are not good criteria, instead `model and distro adaptive` are better

@ 8:40 ??? why these sub intervals have equal probabilities of 20%? isnt it better to assign prob with their MAE range which they take(ofc I know for last subInterval we would have a problem that it would get infinite)? what if we had another point with .16 MAE loss(note we have .15 MAE also), so it would have created another subInterval with same prob as others?

so I hope in next vids we would get review on how practically the `conformal predictors` respect these criteria like 1. `coverage validity` and `efficency` are respected in conformal because the data itself is used in making intervals? 2. how we know its model agnostic? is not involving any model params, enough. also the same thing for distribution free?

thanks for the explanation!

Great lecture. Thank you very much. I subscribed this channel. 🙏

Good work

This video deserves more views and likes!

At [8:42] I'm trying to see the bigger picture. I get that there are such edge cases where MAPE could be misleading as a function for minimizing. But, if minimizing the MAPE involves minimizing the absolute percentage error, which accounts for both under-forecasting and over-forecasting, then why it doesn't make sense to use it in business case like forecasting inventory items you've explained? It does not necessarily imply that there will be a bias towards under-forecasting or over-forecasting. That is, minimizing the MAPE does not necessarily imply that there will be a lot of under-forecasting or a lot of over-forecasting. Maybe we should get a step further and whilst minimizing MAPE we explicitly state a bias towards over-forecasting using some kind of "weighted MAPE", and thus reducing the number of under-forecasting values. Something like that?

amazing!

My reply to a question raised in an earlier comment: Question: “3) Why are you talking about how to 'cheat' such metrics in this (in my opinion) unrealistic situation (where you have access to the underlying distribution). What if an average of all 3 metrics is used? Do these assumptions still hold if the interval on which the evaluation is done increases? What if the evaluation is done on a test set twice as big as the known train set?” The discussion is important even when one does not have access to the underlying distribution because, even then, one typically needs to evaluate and rank different and competing forecasting methods (ex. tree-based vs. neural networks), which requires deciding on which evaluation metric to use (ex. MSE vs. MAE). For example, imagine a scenario where we have two competing models A and B. The MSE of model A is lower than that of model B but the MAE of model A is higher. So if we use MSE as an evaluation metric, then model A is the better model, but if we use MAE then model B will be the better one. So, the question is which metric should we use for evaluation? It is important to base our evaluation on a metric that properly targets the point from the underlying and unknown distribution that is more important for us. For example, if, for the business case at hand, predicting the mean is more important than the median, then, model A is a better one for that business case. Different evaluation metrics can be combined but one should be aware that by doing so a non-conventional point from the underlying distribution is then being targeted. In the example above if predicting the median and mean are both important, then it would make sense to use some combination of MSE and MAE as the evaluation metric. And yes the arguments stated in the video (ex. mean is optimal under SE) will hold even if you increase the evaluation interval. But if you significantly shorten the interval they may not hold.

My reply to a question raised in an earlier comment: Question: “2) Once such an expert has access to the underlying data distribution, why aren't all these error metrics set to zero since they can perfectly predict the future data point? -> I assume that this is because it's an random process and you don't have a y(t+1) = f(y(t)) deterministic relationship, with 'f' being the distribution.” Yes, and let’s also keep in mind that these metrics are point ones, meaning that to evaluate each one has to pick a point from the distribution.

Reply to the question raised in an earlier comment: “1) How would an 'expert' given only the data point correctly infer the data distribution?” That will require the expert to build an error-free model. For complicated cases such as the one discussed in the episode, I am not sure that would be possible in practice. However, the point of this series of episodes is to highlight that different evaluation metrics (MAE, MSE, MAPE, etc.) get their minimum values with different points from the distribution. When one performs point forecasting for a phenomenon that is inherently probabilistic, which is the most common real-life forecasting setting, even after one builds the model, so much care still needs to be given to select the right point. I believe the former (building model) typically receives enough attention but the latter does not. One may reasonably argue that, ok if the task is point forecasting, why should we build a probabilistic model? And the answer is that when you build a point-forecasting model, the point from the underlying, unknown distribution you are training for (i.e., the training loss) should be consistent with the evaluation metric. I will discuss this point in more detail in the upcoming episodes. Hope this answers your first question. Looking forward to follow-up comments if any. Thanks for reading!

I'm trying to follow these videos, but they're a bit out of my depth. Could you explain a few things? 1) How would an 'expert' given only the data point correctly infer the data distribution? 2) Once such an expert has access to the underlying data distribution, why aren't all these error metrics set to zero since they can perfectly predict the future data point? -> I assume that this is because it's an random process and you don't have a y(t+1) = f(y(t)) deterministic relationship, with 'f' being the distribution. 3) Why are you talking about how to 'cheat' such metrics in this (in my opinion) unrealistic situation (where you have access to the underlying distribution). What if an average of all 3 metrics is used? Do these assumptions still hold if the interval on which the evaluation is done increases? What if the evaluation is done on a test set twice as big as the known train set?

How fortunate we are that center of mass exists in a Hookean universe.

very informative! I like the questions presented around 3:43.

Hi Mahdi, I like your videos very much! They are very well structured and well referenced by academic papers. What I would like to discuss here, is related to the material presented after 8:39 in your video. Two tables are presented on when to use MAPE vs. adjusted MAPE. Can you please elaborate on the "Outlier Impact on Cost" case? I would like to know more about cases where that impact is limited vs. unlimited. What I am trying to understand deeper is how one should decipher whether the impact of an outlier on cost is limited or unlimited in his/her case. Thank you in advance!

Great and clear explanation! Helped me further, thanks! 👏🏻

This is a great video, keep it up, please! I have a question: Around 7:54 in the video while you are explaining under-forecasting and over-forecasting examples in the table, why are we swapping the values of actuals and forecasts? I was thinking why are not we considering this scenario in which under-forecasting is (Actual 150, Forecast 100) and over-forecasting is (Actual 150, Forecast 200). What I am trying to say is, if we go with the scenario mentioned here, MAPE doesn't change. So, does the definition of a counterpart in this context mean the actual and forecasted values should be swapped in order to compare under and over-forecasting in a given time point?

I hope you cover distributional prediction

very helpful content. keep the good work!

Great video! I found you through LinkedIn and found the video really informative. Although I knew MAPE is weighted based on actual value, I hadn’t ever considered the fact that this leads to a heavier penalty on positive errors. As far as feedback, I think your graphics and the sound are all really good. I would say the structure of the video could use improvement. For example, I would prefer you start the video with an explanation of when MAPE is useful and discuss its use cases. I felt the video jumped into the limitations before we fully got to see why it’s useful. It sort of reminded me of a linear algebra proof where the professor only discusses the preconditions required for the proof that by then end of the class It’s hard to remember what the actual proof was for. All in all, a really quality video and I’ve subscribed. Also a small other point, but when first looking at the thumbnail it gave me the impression that this is one of those ads stating that MLBoost is a package (pretty cool name for a package though imo) that you can calculate time series with. I typically avoid anything like that and since this video is informative I would consider changing the title to something more explanatory.

Question 2: How much of the content presented in the video was new to you, and how much did you already know?

Question 1: Did the video provide you with new insights on MAPE, or did you already know about it? If you did gain new insights, what were they?