I always wanted to be a Credit risk Quant. But as time progresses it seems I'll end up in HFT as a quant dev. I worked for a medium sized bank in Denmark doing data engineering for them. They had 90 day default, AIRB method. They decided to do backtesting on previous defaulted loans, where their criteria was that the model should be able to catch 90% of failing loans.
Not that it can't be useful but I have some issues with AUC and would like to hear your thoughts. 1. It treats false positives and negatives the same way when in reality one might be worse than the other meaning more careful threshold tunning would be preferred over separation through a range of values. 2. It's not a probabilistic metric. 3. It doesn't care about model calibration. For that reason I tend to prefer the Brier Score, the Brier Skill Score also makes it very easy to compare models to a benchmark.
Hey Dimitri, I just want to say I'm a huge fan of your videos. I'm trying to follow a similar route and get into the quant world. I've headed your advice and instead of going from a business studies bachelors to a msc in quantitative finance my current plan is to spend a year studying mathematics as a higher diploma (encapsulates all core modules of a mathematics bachelors) then spend the following year studying financial maths as a masters programme. I also have the opportunity to study statistics and data science msc part time for 3 years so at the end of the the three years I should hopefully have the hdip in math msc in financial math and msc in stats and data science. All of these should be from Trinity College Dublin and University College Dublin. They are the two top schools in Ireland but my dream would be to work eventually similar to yourself in America. I would love to know your opinion on whether you think this academic route would suffice and is the most efficient for broadening my opportunities and becoming a quantitative researcher.
@eoghandoheny4144 in general sounds like a good educational path however, remember it is hard to find a job in a country where you didn't go to school. I would consider the US for a masters degree if possible.
18:55 it seems like AUC is most useful if you intend to use a variety of thresholds on probabilistic models (like logistic regression). Do you usually convert decision trees / svm / other models to predict probabilities for classification problems, and then set custom thresholds for those models as well?
so sick of bad modelers bragging about 90%+ AUC when the industry standard is way lower and somehow they've magically found the holy grail that outperforms every bank in history
@sentralorigin I've seen that as well with decision tree models but they don't rank order. They also fail quickly. I now feel like the old man explaining the good enough range lol.
Hi Dimitri, I have a couple of questions: 1. In the case of an imbalanced classes, a model biased towards the majority class could lead to an increase in TN, a lowered FP rate, and thus a high AUC even if the model fails to classify the Bads correctly. For this reason, I have read some articles recommending using alternative metrics such as average precision from the Precision-Recall Curve. What do you think about it? 2. How do you include economic factors in your model? Sampling data from the entire economic cycle or including economic variables as independent variables? Thank you :)
When you are modelling the 1-year-ahead default, what would be considered a good recall value for the default class? Also, could you, please, recommend me some papers to help me understand this part of credit risk modelling ?
Hi Dimitri! I love your technical videos, we need more 😃😃😃
I always wanted to be a Credit risk Quant. But as time progresses it seems I'll end up in HFT as a quant dev.
I worked for a medium sized bank in Denmark doing data engineering for them. They had 90 day default, AIRB method. They decided to do backtesting on previous defaulted loans, where their criteria was that the model should be able to catch 90% of failing loans.
Thank you, great video
Not that it can't be useful but I have some issues with AUC and would like to hear your thoughts.
1. It treats false positives and negatives the same way when in reality one might be worse than the other meaning more careful threshold tunning would be preferred over separation through a range of values.
2. It's not a probabilistic metric.
3. It doesn't care about model calibration.
For that reason I tend to prefer the Brier Score, the Brier Skill Score also makes it very easy to compare models to a benchmark.
Hey Dimitri, I just want to say I'm a huge fan of your videos. I'm trying to follow a similar route and get into the quant world. I've headed your advice and instead of going from a business studies bachelors to a msc in quantitative finance my current plan is to spend a year studying mathematics as a higher diploma (encapsulates all core modules of a mathematics bachelors) then spend the following year studying financial maths as a masters programme. I also have the opportunity to study statistics and data science msc part time for 3 years so at the end of the the three years I should hopefully have the hdip in math msc in financial math and msc in stats and data science. All of these should be from Trinity College Dublin and University College Dublin. They are the two top schools in Ireland but my dream would be to work eventually similar to yourself in America. I would love to know your opinion on whether you think this academic route would suffice and is the most efficient for broadening my opportunities and becoming a quantitative researcher.
@eoghandoheny4144 in general sounds like a good educational path however, remember it is hard to find a job in a country where you didn't go to school. I would consider the US for a masters degree if possible.
@@DimitriBianco Thanks so much for your advice!
18:55 it seems like AUC is most useful if you intend to use a variety of thresholds on probabilistic models (like logistic regression). Do you usually convert decision trees / svm / other models to predict probabilities for classification problems, and then set custom thresholds for those models as well?
Thank you for the interesting video. Can you explain VaR and CVaR modelling please?
so sick of bad modelers bragging about 90%+ AUC when the industry standard is way lower and somehow they've magically found the holy grail that outperforms every bank in history
@sentralorigin I've seen that as well with decision tree models but they don't rank order. They also fail quickly. I now feel like the old man explaining the good enough range lol.
Hi Dimitri, I have a couple of questions:
1. In the case of an imbalanced classes, a model biased towards the majority class could lead to an increase in TN, a lowered FP rate, and thus a high AUC even if the model fails to classify the Bads correctly. For this reason, I have read some articles recommending using alternative metrics such as average precision from the Precision-Recall Curve. What do you think about it?
2. How do you include economic factors in your model? Sampling data from the entire economic cycle or including economic variables as independent variables?
Thank you :)
When you are modelling the 1-year-ahead default, what would be considered a good recall value for the default class? Also, could you, please, recommend me some papers to help me understand this part of credit risk modelling ?