NOTE: At 31:25 we should use the mean and standard deviation from the training dataset to center and scale the testing data. The updated jupyter notebook reflects this change. ALSO NOTE: You can support StatQuest by purchasing the Jupyter Notebook and Python code seen in this video here: statquest.gumroad.com/l/iulnea Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
After eight years of employment after graduation, I got laid off in 2020. I went back to school to pursue my second master in Data Science. I was still confused after machine learning classes, but after I watched your videos which were the same topics as the ones in my classes, you led me into a totally different world. Same concepts were taught by you in much easier way. BAM!!!
I love this kind of webinar where you teach in real time and go through concrete examples. Just purchased the material package and can't wait to go through them with you. I hope you'll make more content like this in the future 😊(I love the short and sweet vids too but I learn by doing so this helps solidify all the theory stuff!)
Really amazing video, I've been in and around data science and ML for a while but this is the first time I feel like I've gone the full way from mathematical concept -> working program (using medium complexity ML methods) -> insight/ question answered.
Again great work Josh, thanks so much. I actually worked at UNC-Chapel Hill, but I discovered you after moving to another University. Hope will meet you one day to thank you in person for the amazing content you are creating.
I am not an expert but a small help for everyone here ^_^ , if you want to find the missing values very easily, you can type dataframe.isnull().sum() ; dataframe is the name of the object containing the data. And thank you Josh for the amazing webinar ♥
You already get a lot of love, but I have to add to it and tell you how great these are. No joke, I've had nights when I plan on watching some TV or some movies and I decide to check out some 'Quests instead!
Those who are facing error can update the code as follow. clf = SVC(random_state=0) clf.fit(X_train_scaled, y_train) predictions = clf.predict(X_test_scaled) cm = confusion_matrix(y_test, predictions, labels=clf.classes_) disp = ConfusionMatrixDisplay(confusion_matrix=cm,display_labels=clf.classes_) disp.plot()
So, although the publishing company is elsevier, they are not the ones who did the research. If you ever want to read a paper, you can send an email to the primary investigator (the last author of the paper) or any of the first authors really, and they will freely give you the article to read
Thanks for the brilliant tutorial Josh! You are truly an inspiration. I just had two questions here :- 1) You applied a regularization technique here by finding the right value for C. What kind of regularization is this? L1, L2 or L1&L2? 2) Is it possible to apply L1, L2, and elastic net regularization on SVMs? If yes, how should I do it?
The most perfect guide for SVM in RUclips. Will donate after I get my first job! Thank you so much. Btw, I have question, why don't you use PCA before doing the modelling part? Are PCA only been use for visualization?
@@statquest I see but, so far what I know it will reduce the accuracy, but will help to avoid multicollinearity. But because of we have done OneHotEncoder, multicollinearity will be not occur. Am I right?
@@joxa6119 Using PCA first would definitely reduce multicollinearity if that was something we thought we needed to deal with. Multicollinearity usually means that we have 2 or more highly correlated features (also called variables), and thus, they are somewhat redundant. One-hot-encoding will not change the fact that those variables are redundant.
I decided it was interesting to draw two different PCA versions: 1) of the training data - so we can see the classifier with respect to the data it was trained on and 2) of the testing data - so we can see the classifier with respect to the data it was tested with. So the code has both versions, however, one of them (the latter) is commented out. However, you can swap which line is commented out and draw the latter.
I have a request. You explain brilliantly (also with your background info in other videos) how to create and optimize your SVM. Could you also make a video about how to actually use your svm in a target system? That would make sense I think. Because I think that this would necessitate saving the scaler during creation of the SVM and loading it at runtime. Regards.
Your videos are amazing !!!! I am soo happy u clearly explain many of the the topics I need!! :) (p.s. do u receive requests? I would really love a StatQuest on AR,MA,ARIMA,SARIMA models)
Your videos are so informative as always. The way you explain the topics are on another level. But I see a Tabla(twin hand drums) behind you. Do you play that? I also loves to play Tabla. Double BAM!!!! :D
I used to play Tabla a lot. I spent a lot of time in Chennai when I was a kid because my dad taught at the IIT there. When I was there I took lessons on tabla and veena.
Great tutorial Josh! You must truly have one of the highest thumbs up to thumbs down ratios on youtube. Just two questions. 1) Right now you are using standarscaler on all of your variables, including the ones you have encoded. What is your reasoning for this instead of just scaling the continous variables, or maybe it doesn't affect the result? 2) What are your thoughts on onehotencoding before vs after splitting the data? Obviously right now, when your doing get_dummies your are doing it before splitting the data. From what I have understood, whether to do it before or after splitting is a pretty heated topic and I have found several questions on stack exchange where half the people say do it before and the other half say that doing it before is absolutely wrong and that it instead should be done after. In this dataset it will have an effect, because using your random states will produce a train test that on some variables have fewer categories than the test data does, which would mean that those observations should be dropped if onehotenconding is done after splitting. If I instead used onehotencoding before splitting, they would not be dropped. Would love to hear your thoughts on that topic, because I have found no real consenus on what is the right approach. Thanks again Josh!
1) For support vector machines, I'm pretty sure it does not effect the result. However, I have not tried it both ways. 2) I think there is a fear that if you one-hot-encode before splitting the data, then there will be data leakage. With most transformations, this is a problem, but for one-hot-encoding this is not the case. If a value in one dataset does not occur in the other dataset, then the column representing that value will be full of zeros and not have an effect on classification. In fact, the preferred method for industrial pipelines is "ColumnTransformer()", which keeps track of the values during the initial one-hot-encoding and when a testing set has new values, it throws an error.
Josh - Thanks for the video and it is super helpful!! A couple of questions though: 1. Under "Transform the test dataset with the PCA...", should we use the code that you commented out - i.e. X_test_pca=pca.transform(X_test_scaled), instead of X_test_pca=pca.transform(X_train_scaled)? didn't get why we applied the PCA transformation on train dataset to derive testing data. 2. Noticed that 1,000 defaults and 1,000 non-defaults were selected to construct the training sample. Do the numbers of two classes have to be equal for SVM? If not, would this cause any bias as the ratio seems a lot different from the original data? Thank you!
1) Because the SVM was fit to the training data, I wanted to show how it "looked" relative to the training data. However, you can also "see" how the boundary applies to the testing data. It's up to you. 2) Typically it's a good idea to have "balanced" data - data with an equal number of both classes. However, this is not a requirement for SVM - and, whether or not you need it depends on how you want the SVM to perform. For more details, see: ruclips.net/video/iTxzRVLoTQ0/видео.html
Hi Josh, Thank you very much for your lessons ! you explain very well unlike many teachers. I just have one doubt, when you scale(X_train) and scale(X_test) you're actually scaling the encoded 'categorical' variables. Thus the sparse encoded matrix of 0 and 1 encoded by the features ['SEX','MARRIAGE',....] will be scaled as well, is that correct ? Shouldn't be only the numerical features to get scaled ? Thanks a lot for your lessons
Josh, this video has helped me out a lot in my studies, but I have a question. When we scale the data, we should also include the categorical variables? Shouldn't we just scale all the data excluding the categorical ones?
Because the categorical variables are one-hot-encoded, we can scale them. All of the 0s will stay the same and the 1s will all turn into another constant value. In other words, when one-hot-encoding, 1 is arbitrarily chosen to begin with, so it doesn't hurt to turn it into another arbitrary number.
@StatQuest with Josh Starmer First before all, I really like _all_ your videos especially this one! I have a comment on your comment on your scaling of the X data. You are scaling the train and test data with two sperate scalers. However this is not optimal since the mean/std of the train and test data can differ. For example, let's say the mean of your train data is 5 and the mean of your test data is 6 and that both sets have a std of 1. Then a observation x with a value of 4 is normalized differently depending on when it is observed. If it is observed in the train set then it is normalized to -2 and if it is observed in the test set then it is normalized to -1. This should not be happening since the information of the observation should not depend on the scaling. Further since the model is trained on the train set a value of -2 could have a different meaning to the model and ultimately for the prediction than a value of -1 (if the unscaled trainset contains 3 for example). So you should scale the train and test data using a scaler which is trained on the train data.
Dear Josh, My understanding, n_components hyperparameter in PCA() is the number of dimensions that we want to reduce down to. Therefore, I make some confusion. 1. If we use PCA() with no reference any n_components, so what exactly is the number of components in this case? 2. In other tutorials, n_components can set in floating (0.0 to 1.0), it is not make sense if we understand as a dimension number. Thanks, have a nice week!
@@statquest Thank for your recommend video. I understanding in this way. when we use PCA() with no n_components hyperparameter, the program will calculate all PCs of data. n_components in this situation is equal to all dimensions of data (pca.explained_variance_ratio_.shape[0]) and when we use PCA(n_components=2) that we only take care 2 first PCs. Sorrry if this question make inconvenience from you. I am only want to sure that my understanding is correct.
Another great video, I wish I had found this channel years ago! I am assuming the way you have coded for the optimising of Parameters could be used as the basis code for other models like Random Forest and it will just be the parameters changing dependent on the model that is being optimised?
Thank you very much.. In the radial basis function video, only hyperparameter gamma was involved.. regularization parameter C was not there in the radial kernel function.. Are we using different radial kernel function here or the same one which was shown in radial kernel video? Thanks again.. your videos are great help ..
We are using the same kernel - so the only kernel parameter that we are optimizing is gamma. However, most, if not all, machine learning implementations also include regularization in one form or another. So we'll be talking about that as well.
Hi i have a question, aren't we supposed to split the data even more, and then use the validation dataset for hyperparameter tuning, we can pass it to grid_search, e.g. grid_search(x_validation,y_validation) instead of using the training dataset again?
Hello Josh Starmer, Can you explain more about some hyperparameter in resample? replace=False --> we will not change any data in original data (df_default) and if True mean original df_default will be changed? random_state --> help others can get the same result with you? So how many people can get same result to you? 42??? Thanks
1) Yes 2) We are setting the seed for the random generator to the number 42, this ensures that everyone will get the same results. In other words, the random number generator generates a sequence of random numbers based on a starting value. If we all set the starting value to the same number (in this case, 42) then we will all get the same sequence of random numbers.
Hi Josh, really great content, learning a lot. Out of curiosity when doing One Hot Encoding, is there a reason why you did not say drop-first=True to avoid Multi-collinearity?
Dear Josh, I fell not logic when you set the C hyperparameter in 2-time(when apply to X_train_scaled and pca_train_scaled) you define the param_grid. The first, C= 1000 is not in your list, the second C = 1000 is adding and it is becoming the best parameter in grid-search. Any ideal in this step? Thanks and have a nice week!
Why are you fitting your GridSearchCV on only the training data? Shouldn't you pass your entire X dataset when doing GridSearchCV with cross validation?
Correct me if im wrong but when people think of machine learning, they think of ai self improvement but SVM is just finding correlations between data sets?
Hi Josh! What if our dataset has 【continuous columns】 & 【"categorical number" columns】 at the same time, should we start with getting dummies first to convert our categorical columns to continuous columns AND Standardscaler the rest continuous columns in order to give the data 0 mean? Is there any correlation between "get_dummies" & "encoder" ? I really appreciate any answers you would share with US, cheers!
XGBoost works well with sparse data (data with lots of zeros), so it is probably a good idea to only one-hot-encode the categorical data. Do not standardize them as well.
Dear Josh, I do not understanding your decision in Confusion Matrix. Do not default : 79% and defaulted: 61% --> not awesome. Why? This number is small, isnt it? or do you have a threshold in your decision?
Great tutorial! In fact, all your videos are great. I have just on question: When looking for the best value for C, the algorithm went for the upper limit. Shouldn't we try again with higher values as suggestions? I haven't tried myself so I really don't know what would happen.
Hey Josh, thanks for the video. One question: you drop the ID column right from the start. In real life, once you made sure your model is valid and accurate, you would actually need to match those IDs to the probabilities of default. How would you do that? Put the ID in a list before dropping and then adding the list as a column to the predict proba?
Hi Josh, I have a question. in 32:30 ,we scale the X_test and X_train, but i think that they didnt scaled same way. Bc They are not in same sample and their standard dev and means are different from eachother. I tried with this tiny sets to check if i think correct, and looks like scaling process little wrong? xxx = [1, 4, 400, 10000, 100000] yyy = [1,4,400,10000,11] scale(xxx) scale(yyy) Can u check and write me, did i think wrong?
In a pinned comment I wrote: At 31:25 we should use the mean and standard deviation from the training dataset to center and scale the testing data. The updated jupyter notebook reflects this change.
@@statquest I didnt realized, sorry. Thank you for the reply. I wanna thank you so much. There could be too much informative people on internet but you are the best. Thank you for having fun while teaching!!
Hello Josh, from a statistical perspective, how do you deal with "weird data"? As an illustration, for this dataset, EDUCATION, Category 1 = graduate school 2 = university 3 = high school 4 = others However, df['EDUCATION'].unique() array([2, 1, 3, 5, 4, 6, 0], dtype=int64) How do you deal with "5 and 6"? They are not in the category. Do you treat them as "missing values'? Also, how about some data values which are out of range? They are definitely wrong.
NOTE: At 31:25 we should use the mean and standard deviation from the training dataset to center and scale the testing data. The updated jupyter notebook reflects this change.
ALSO NOTE: You can support StatQuest by purchasing the Jupyter Notebook and Python code seen in this video here: statquest.gumroad.com/l/iulnea
Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
By scaling do you mean data normalization?
@@Dani-hh3qd Normalization is a specific type of scaling.
After eight years of employment after graduation, I got laid off in 2020. I went back to school to pursue my second master in Data Science. I was still confused after machine learning classes, but after I watched your videos which were the same topics as the ones in my classes, you led me into a totally different world. Same concepts were taught by you in much easier way. BAM!!!
I'm glad my videos are helpful! :)
A 2nd master?
How much has the curriculum changed in the past 8 years?
I will definitely donate to this channel as soon as I got a job! Thanks.
Thank you very much! :)
Really appreciate for your slow speaking speed ,which makes it possible for not a English speaker ,like me ,a Chinese,to learn.
Thank you!
I'm so happy to find out that saying BAM + DOUBLE BAM comes naturally to you (and was not just for the videos). Amazing walkthrough as usual, Josh!
Triple bam! :)
455 likes and 0 dislikes.... that's a double BAM!
Thanks!
i will be the first one
I purchased the notebook and I also watched the whole ad so you can make more money.
Thank you so much for your support! It means a lot to me. BAM! :)
I love this kind of webinar where you teach in real time and go through concrete examples. Just purchased the material package and can't wait to go through them with you. I hope you'll make more content like this in the future 😊(I love the short and sweet vids too but I learn by doing so this helps solidify all the theory stuff!)
Thank you, and thank you for your support!
You're a pretty amazing nerd, I love it. This is an amazing tutorial.
Thanks! 😃
Really amazing video, I've been in and around data science and ML for a while but this is the first time I feel like I've gone the full way from mathematical concept -> working program (using medium complexity ML methods) -> insight/ question answered.
Glad you enjoyed it!
Again great work Josh, thanks so much. I actually worked at UNC-Chapel Hill, but I discovered you after moving to another University. Hope will meet you one day to thank you in person for the amazing content you are creating.
Wow! Thank you very much! :)
Hi from Argentina.
Great video! It really was from start to finish, it covers every step with dedication.
Thanks for sharing your knowledge!
Thank you very much! :)
This guy doesn't deserve the dislike button to be in his videos...what a clear explanation!!!
Awesome! Thank you very much! :)
Thank you so much for making this amazing code-walkthrough for SVM. Looking forward for more code walkthroughs like this.
You're very welcome!
you video deserves to be translated into more languages so people don't speak English can also learn from your amazing content
Thanks! :)
I am not an expert but a small help for everyone here ^_^ , if you want to find the missing values very easily, you can type
dataframe.isnull().sum() ; dataframe is the name of the object containing the data.
And thank you Josh for the amazing webinar ♥
Nice tip!
Triple BAM!! Guess What?? You are the best teacher I've ever come across. My life is saved. Good to know you play Tabla too.
Thank you very much!!! :)
svm are kinda my favourite thing in ML. very simple and mathematically concise yet highly usable.
Nice!
This is amazing! Am in love with your approach of handling these stuff. Very clear and concise.
Thank you! :)
Really detailed and nice lesson! I liked how detailed the explanations were, It is definitely DOUBLE BAM worthy!
Thank you.
Glad you enjoyed it!
Josh, you are Phenomenal! Love and Respect from Madras !
Respect from kerala too
@@anjalivijay9577 adhaan💥💪
Hooray!!! Thanks! :)
@@statquest 🤩🤩🤩🤩
@@statquest BAAAAM
I've reread the "hitchhikers guide to galaxy" again (first time I read I was 12) and now it makes a lot more sense why the random state is 42 :)))
Yes!
Thank you Josh, this taught me a good lesson on both PCA and SVM. Great work!
Bam! :)
i always love those musical intros
Bam!
You already get a lot of love, but I have to add to it and tell you how great these are. No joke, I've had nights when I plan on watching some TV or some movies and I decide to check out some 'Quests instead!
BAM! Thank you very much! :)
You Sir are an outstanding educator.
Thank you!
Your video is so awesome. Everything related to SVM in one video, BAM.
Glad you liked it!
Josh, you are wonderful! Thanks a million form Italy!
Thank you very much!!!
This video came out the same week I decided to learn this. Get out of my head!
BAM! :)
You are amazing. Keep posting. Best wishes from India.
Thank you very much! :)
You are awesome. I hope you do something on NLP (tf idf, word2vec, etc.), for some reason your style was made for my brain
Thanks! :)
Precise and to the point. Luv this and I am def going to extend my support to you
Thank you! :)
Hope to listen to the Tabla's behind you at the start of your training one day.
Maybe one day!
@@statquest Amen to that ✌️
I learned a lot from your channel. I am a big fan of you. Looking forward for your Deep learning and NLP tutorial with python
Awesome, thank you!
Josh u r an inspiration in teaching...Plz keep it up
Thank you! :)
Thanks a bunc. Helping me a lot getting started with my SVM. Regards
Happy to help!
When Josh said 'OH NO!!', I was waiting for the line 'Terminology Alert!!!'.
:)
Haha the double bam at 31:22had me dying lol. Great content! And love your channel!
Thank you so much! :)
I love StatQuest. please continue to make video with python =)
Thank you! :)
Those who are facing error can update the code as follow.
clf = SVC(random_state=0)
clf.fit(X_train_scaled, y_train)
predictions = clf.predict(X_test_scaled)
cm = confusion_matrix(y_test, predictions, labels=clf.classes_)
disp = ConfusionMatrixDisplay(confusion_matrix=cm,display_labels=clf.classes_)
disp.plot()
Yep. The notebook has been updated.
Wow. Thanks Josh . Your videos are always a go to resource
Thanks! :)
Good to see no haters for the saintly man.
Thanks!
Thanks! I really like the way you explain things: calm and simple :)
Thank you! :)
what a lovable smart man, thanks for the great work!
Thank you!
So, although the publishing company is elsevier, they are not the ones who did the research. If you ever want to read a paper, you can send an email to the primary investigator (the last author of the paper) or any of the first authors really, and they will freely give you the article to read
That's a great idea! :)
Thanks for the brilliant tutorial Josh! You are truly an inspiration.
I just had two questions here :-
1) You applied a regularization technique here by finding the right value for C. What kind of regularization is this? L1, L2 or L1&L2?
2) Is it possible to apply L1, L2, and elastic net regularization on SVMs? If yes, how should I do it?
C controls L2 penalty. I think that might be the only regularization you can use with scikit-learn svm.
@@statquest Yes I read the documentation of scikit-learn svm and the only other penalty allowed is L1.
The most perfect guide for SVM in RUclips. Will donate after I get my first job! Thank you so much.
Btw, I have question, why don't you use PCA before doing the modelling part? Are PCA only been use for visualization?
In this case, we only use PCA for visualization.
@@statquest I see but, so far what I know it will reduce the accuracy, but will help to avoid multicollinearity. But because of we have done OneHotEncoder, multicollinearity will be not occur. Am I right?
@@joxa6119 Using PCA first would definitely reduce multicollinearity if that was something we thought we needed to deal with. Multicollinearity usually means that we have 2 or more highly correlated features (also called variables), and thus, they are somewhat redundant. One-hot-encoding will not change the fact that those variables are redundant.
Love python! Been using R much lately! Would love to have some of R videos
Yes, I'm going to cover all of these topics (and more) in R. For example, R does a much better job with Random Forests than Python.
StatQuest with Josh Starmer I totally agree! Expect videos to come~
Can you explain why you used 'x_test_pca =pca.transform( x_train_scaled) when you wanted to transform test data with PCA?
I decided it was interesting to draw two different PCA versions: 1) of the training data - so we can see the classifier with respect to the data it was trained on and 2) of the testing data - so we can see the classifier with respect to the data it was tested with. So the code has both versions, however, one of them (the latter) is commented out. However, you can swap which line is commented out and draw the latter.
@@statquest Thank you so much
I have a request. You explain brilliantly (also with your background info in other videos) how to create and optimize your SVM.
Could you also make a video about how to actually use your svm in a target system? That would make sense I think.
Because I think that this would necessitate saving the scaler during creation of the SVM and loading it at runtime. Regards.
Good idea!
I feel a bit starstruck finally seeing your face... :p Love your videos as always!
😊 thank you
Hello Josh, Do you have any lecture about support vector data description (SVDD) as well. Actually, your way of describing problems is amazing.
Not yet! :(
It helped a lot! Thank You on shared time and knowladge.
Thank you! :)
Thank you very much for the video!
I have a question, in SVM should the variables only be numeric or does it also support text?
Thank you!
Only numeric
Hooray! :)
Your videos are amazing !!!! I am soo happy u clearly explain many of the the topics I need!! :)
(p.s. do u receive requests? I would really love a StatQuest on AR,MA,ARIMA,SARIMA models)
I'll keep those topics in mind.
I always ser your videos! Please continue this series of videos and surely I will purchase a notebook soon.
Thank you very much! :)
Your videos are so informative as always. The way you explain the topics are on another level. But I see a Tabla(twin hand drums) behind you. Do you play that?
I also loves to play Tabla. Double BAM!!!! :D
I used to play Tabla a lot. I spent a lot of time in Chennai when I was a kid because my dad taught at the IIT there. When I was there I took lessons on tabla and veena.
929 likes and 0 dislikes ... that's a triple BAM !!
Hooray! :)
You are just awesome. I just love your videos as they are really amazing. Stay safe .
Thank you! You too!
Great tutorial Josh! You must truly have one of the highest thumbs up to thumbs down ratios on youtube. Just two questions.
1) Right now you are using standarscaler on all of your variables, including the ones you have encoded. What is your reasoning for this instead of just scaling the continous variables, or maybe it doesn't affect the result?
2) What are your thoughts on onehotencoding before vs after splitting the data? Obviously right now, when your doing get_dummies your are doing it before splitting the data. From what I have understood, whether to do it before or after splitting is a pretty heated topic and I have found several questions on stack exchange where half the people say do it before and the other half say that doing it before is absolutely wrong and that it instead should be done after. In this dataset it will have an effect, because using your random states will produce a train test that on some variables have fewer categories than the test data does, which would mean that those observations should be dropped if onehotenconding is done after splitting. If I instead used onehotencoding before splitting, they would not be dropped. Would love to hear your thoughts on that topic, because I have found no real consenus on what is the right approach.
Thanks again Josh!
1) For support vector machines, I'm pretty sure it does not effect the result. However, I have not tried it both ways.
2) I think there is a fear that if you one-hot-encode before splitting the data, then there will be data leakage. With most transformations, this is a problem, but for one-hot-encoding this is not the case. If a value in one dataset does not occur in the other dataset, then the column representing that value will be full of zeros and not have an effect on classification. In fact, the preferred method for industrial pipelines is "ColumnTransformer()", which keeps track of the values during the initial one-hot-encoding and when a testing set has new values, it throws an error.
@@statquest Thanks for your insights Josh! Really appreciate it
@@statquest is it the same for K-means cluster analysis also ?
@@causticmonster Presumably if you use ColumnTransformer().
How it started:
df
How it is going:
df_23_without_missingdata_scaled_with_magic_powers
Bam! :)
The man behind the voice
:)
Thank you so much, it was a wonderful video!!!
Glad you enjoyed it!
This is really great. Thank You Sir for this great effort!!
Glad you liked it!
Josh - Thanks for the video and it is super helpful!! A couple of questions though:
1. Under "Transform the test dataset with the PCA...", should we use the code that you commented out - i.e. X_test_pca=pca.transform(X_test_scaled), instead of X_test_pca=pca.transform(X_train_scaled)? didn't get why we applied the PCA transformation on train dataset to derive testing data.
2. Noticed that 1,000 defaults and 1,000 non-defaults were selected to construct the training sample. Do the numbers of two classes have to be equal for SVM? If not, would this cause any bias as the ratio seems a lot different from the original data?
Thank you!
1) Because the SVM was fit to the training data, I wanted to show how it "looked" relative to the training data. However, you can also "see" how the boundary applies to the testing data. It's up to you.
2) Typically it's a good idea to have "balanced" data - data with an equal number of both classes. However, this is not a requirement for SVM - and, whether or not you need it depends on how you want the SVM to perform. For more details, see: ruclips.net/video/iTxzRVLoTQ0/видео.html
Hi Josh,
Thank you very much for your lessons ! you explain very well unlike many teachers. I just have one doubt, when you scale(X_train) and scale(X_test) you're actually scaling the encoded 'categorical' variables. Thus the sparse encoded matrix of 0 and 1 encoded by the features ['SEX','MARRIAGE',....] will be scaled as well, is that correct ? Shouldn't be only the numerical features to get scaled ? Thanks a lot for your lessons
It doesn't really matter if you scale binary variables or not: stats.stackexchange.com/questions/59392/should-you-ever-standardise-binary-variables
@@statquest thanks for the reply! BAM
I instantly liked the video just for the Ukulele
Hooray!
One final question (I swear!): At the final code segment, you type
X_test_pca = pca.transform(X_train_scaled)
Isn't that supposed to be X_test_scaled?
Hmm....I'm actually on vacation right now and can't dig through this code. Can you re-ask this question in a few weeks?
Josh, this video has helped me out a lot in my studies, but I have a question. When we scale the data, we should also include the categorical variables? Shouldn't we just scale all the data excluding the categorical ones?
Because the categorical variables are one-hot-encoded, we can scale them. All of the 0s will stay the same and the 1s will all turn into another constant value. In other words, when one-hot-encoding, 1 is arbitrarily chosen to begin with, so it doesn't hurt to turn it into another arbitrary number.
@@statquest Got it Josh, thanks for responding
@StatQuest with Josh Starmer First before all, I really like _all_ your videos especially this one!
I have a comment on your comment on your scaling of the X data. You are scaling the train and test data with two sperate scalers. However this is not optimal since the mean/std of the train and test data can differ. For example, let's say the mean of your train data is 5 and the mean of your test data is 6 and that both sets have a std of 1. Then a observation x with a value of 4 is normalized differently depending on when it is observed. If it is observed in the train set then it is normalized to -2 and if it is observed in the test set then it is normalized to -1. This should not be happening since the information of the observation should not depend on the scaling. Further since the model is trained on the train set a value of -2 could have a different meaning to the model and ultimately for the prediction than a value of -1 (if the unscaled trainset contains 3 for example).
So you should scale the train and test data using a scaler which is trained on the train data.
That is correct. Sorry for the mistake.
I'll make a note about this in a pinned comment and update the notebook asap.
Dear Josh,
My understanding, n_components hyperparameter in PCA() is the number of dimensions that we want to reduce down to. Therefore, I make some confusion.
1. If we use PCA() with no reference any n_components, so what exactly is the number of components in this case?
2. In other tutorials, n_components can set in floating (0.0 to 1.0), it is not make sense if we understand as a dimension number.
Thanks, have a nice week!
The number of components is explained here: ruclips.net/video/oRvgq966yZg/видео.html
@@statquest Thank for your recommend video. I understanding in this way.
when we use PCA() with no n_components hyperparameter, the program will calculate all PCs of data. n_components in this situation is equal to all dimensions of data (pca.explained_variance_ratio_.shape[0])
and when we use PCA(n_components=2) that we only take care 2 first PCs.
Sorrry if this question make inconvenience from you. I am only want to sure that my understanding is correct.
@@tanphan3970 Yes, that is correct.
This is amazing. Thank you, Josh!
Thank you!
Another great video, I wish I had found this channel years ago!
I am assuming the way you have coded for the optimising of Parameters could be used as the basis code for other models like Random Forest and it will just be the parameters changing dependent on the model that is being optimised?
Yes. However, the scikit-learn implementation of random forests is terrible...
Very approachable!
Thanks!
Awesome teaching! Very interesting lectures.
Thank you! :)
Aahhhh....Something that I was stuck with...thanks a lot❣
Hooray! :)
I bought your machine learning book but I can not find the pdf version right now , what should I do ? Would you like to help me to find a copy?
Did you buy the physical book or the PDF? If you bought the PDF, you should be able to download another copy.
where are the predicted y_values? Only the graph is shown in the end. where are the numerical results? What's the code for obtaining that?
Just call clf.predict(). This is the same for all sklearn machine learning modules.
Thank you very much.. In the radial basis function video, only hyperparameter gamma was involved.. regularization parameter C was not there in the radial kernel function.. Are we using different radial kernel function here or the same one which was shown in radial kernel video? Thanks again.. your videos are great help ..
We are using the same kernel - so the only kernel parameter that we are optimizing is gamma. However, most, if not all, machine learning implementations also include regularization in one form or another. So we'll be talking about that as well.
that tabla behind you tho!😵
I used to play and took lessons when I lived in Chennai.
Are you supposed to scale the one-hot encoded variables as well?
I think it can go either way.
Amazing content! How do I know when you have a webinar planned? and where do you stream it? Thanks!!
If you subscribe, you can find out about webinars.
@@statquest excellent, will do!
Hi i have a question, aren't we supposed to split the data even more, and then use the validation dataset for hyperparameter tuning, we can pass it to grid_search, e.g. grid_search(x_validation,y_validation) instead of using the training dataset again?
You can definitely do that.
Hello Josh Starmer,
Can you explain more about some hyperparameter in resample?
replace=False --> we will not change any data in original data (df_default) and if True mean original df_default will be changed?
random_state --> help others can get the same result with you? So how many people can get same result to you? 42???
Thanks
1) Yes 2) We are setting the seed for the random generator to the number 42, this ensures that everyone will get the same results. In other words, the random number generator generates a sequence of random numbers based on a starting value. If we all set the starting value to the same number (in this case, 42) then we will all get the same sequence of random numbers.
Thank you for great tutorial!!!
Thanks!
Hi Josh, really great content, learning a lot.
Out of curiosity when doing One Hot Encoding, is there a reason why you did not say drop-first=True to avoid Multi-collinearity?
Yes, this is different from a linear model.
sorry. but thanks. from where can i get these two jupytor notebooks ? i need to practice once.
You can purchase the jupyter notebooks (and support StatQuest) here: statquest.org/product/jupyter-notebook-support-vector-machines-in-python/
Dear Josh,
I fell not logic when you set the C hyperparameter in 2-time(when apply to X_train_scaled and pca_train_scaled) you define the param_grid. The first, C= 1000 is not in your list, the second C = 1000 is adding and it is becoming the best parameter in grid-search.
Any ideal in this step?
Thanks and have a nice week!
It might have been originally but I forgot to add it back in.
Is there a reason why you're not using LDA(multi-class) rather than PCA?
What time point, minutes and seconds, are you asking about?
gr8 experience, looking for ANN.
Thanks!
why do you use 1:1 resampling instead of stratified resampling? The dataset contains 3.5 no_default:1 default. Does this affect SVM results?
What time point, minutes and seconds, are you asking about?
Why are you fitting your GridSearchCV on only the training data? Shouldn't you pass your entire X dataset when doing GridSearchCV with cross validation?
That would overfit the model
Correct me if im wrong but when people think of machine learning, they think of ai self improvement but SVM is just finding correlations between data sets?
When I think of machine learning, I think of classifying things and making predictions. SVM can be used to classify things.
Hi Josh!
What if our dataset has 【continuous columns】 & 【"categorical number" columns】 at the same time, should we start with getting dummies first to convert our categorical columns to continuous columns AND Standardscaler the rest continuous columns in order to give the data 0 mean? Is there any correlation between "get_dummies" & "encoder" ?
I really appreciate any answers you would share with US, cheers!
XGBoost works well with sparse data (data with lots of zeros), so it is probably a good idea to only one-hot-encode the categorical data. Do not standardize them as well.
Dear Josh,
I do not understanding your decision in Confusion Matrix.
Do not default : 79% and defaulted: 61% --> not awesome.
Why?
This number is small, isnt it? or do you have a threshold in your decision?
It's subjective.
Great tutorial! In fact, all your videos are great. I have just on question: When looking for the best value for C, the algorithm went for the upper limit. Shouldn't we try again with higher values as suggestions? I haven't tried myself so I really don't know what would happen.
Yes, we should probably try higher values.
Hey Josh, thanks for the video.
One question: you drop the ID column right from the start. In real life, once you made sure your model is valid and accurate, you would actually need to match those IDs to the probabilities of default. How would you do that? Put the ID in a list before dropping and then adding the list as a column to the predict proba?
We don't really need to re-add the IDs for the training data. However, when we get new data, we can just keep track of the ID by hand.
Hi Josh,
I have a question. in 32:30 ,we scale the X_test and X_train, but i think that they didnt scaled same way. Bc They are not in same sample and their standard dev and means are different from eachother.
I tried with this tiny sets to check if i think correct, and looks like scaling process little wrong?
xxx = [1, 4, 400, 10000, 100000]
yyy = [1,4,400,10000,11]
scale(xxx)
scale(yyy)
Can u check and write me, did i think wrong?
In a pinned comment I wrote: At 31:25 we should use the mean and standard deviation from the training dataset to center and scale the testing data. The updated jupyter notebook reflects this change.
@@statquest I didnt realized, sorry. Thank you for the reply.
I wanna thank you so much. There could be too much informative people on internet but you are the best. Thank you for having fun while teaching!!
How do we set values for C and gamma especially the penalty Parameter C .. is it only by Hit and trial?
Pretty much. You just test a bunch with values with cross validation and see which is best.
Hello Josh, from a statistical perspective, how do you deal with "weird data"? As an illustration, for this dataset,
EDUCATION, Category
1 = graduate school
2 = university
3 = high school
4 = others
However, df['EDUCATION'].unique()
array([2, 1, 3, 5, 4, 6, 0], dtype=int64)
How do you deal with "5 and 6"? They are not in the category. Do you treat them as "missing values'?
Also, how about some data values which are out of range? They are definitely wrong.
I would treat them as missing data.