Outlier detection and removal using percentile | Feature engineering tutorial python # 2

Поделиться
HTML-код
  • Опубликовано: 7 авг 2024
  • Outliers are unusual data points that differ significantly from rest of the samples. They can occur due to an error in data collection process or they are just legitimate data points and represent natural variation. In this video we are going to use percentile to detect and remove outliers from a dataset. We will use python pandas for this.
    Code & Exercise: github.com/codebasics/py/blob...
    Topics
    00:00 Outliers introduction
    01:44 What is percentile
    03:41 Remove outliers from simple dataset
    08:10 Remove outliers from complex dataset
    14:59 Exercise
    Do you want to learn technology from me? Check codebasics.io/ for my affordable video courses.
    Feature Engineering Tutorial Python Playlist: • What is feature engine...
    Website: codebasics.io/
    Facebook: / codebasicshub
    Twitter: / codebasicshub

Комментарии • 133

  • @codebasics
    @codebasics  2 года назад +4

    Check out our premium machine learning course with 2 Industry projects: codebasics.io/courses/machine-learning-for-data-science-beginners-to-advanced

  • @chivalrousforlan238
    @chivalrousforlan238 4 года назад +48

    Most of RUclips tutorials I've gone through are top notch, but yours is loads of miles away. What makes yours different is you start from the basics with a slow pace, which makes it easier to understand.

    • @sumit121285
      @sumit121285 3 года назад

      a true comment for a true teacher......

  • @onlyguitars
    @onlyguitars 7 месяцев назад +1

    Best channel by far. You have become my DS - DA youtuber by far. You tackle one of the most important things that is understanding the concepts and translating it into how to do it. I recently realized that is far better to understand well the concepts than knowing perfectly each pandas function and how to code them fast. Also makes things much more interesting and fun because it clicks and makes sense.

  • @arhataria
    @arhataria 4 года назад +5

    This is amazing, Thanks a lot for posting this valuable video!

  • @BarongoCalvine
    @BarongoCalvine 4 года назад +21

    Thank you, I just had a DS interview and used the age example for the outlier question, thanks for your useful lessons

  • @suyash7450
    @suyash7450 Год назад +1

    You Teach from zero till the the end your pace is perfect and the best part is you provide exercises and resources , Thanks for helping us and teaching us

  • @KenJee_ds
    @KenJee_ds 4 года назад +11

    Great video as always!

  • @vidhanmaheshwari2082
    @vidhanmaheshwari2082 3 года назад

    i was stuck in middle of a project due to messy datasets and then i came acoss your video..it helped a lot..Thanks a lot

  • @AbhishekBalsara
    @AbhishekBalsara 3 года назад +1

    This is exactly what I was looking for and particularly for outliers, Thanks a lot for this 👍

  • @kenny87ification
    @kenny87ification 4 года назад

    Thank you for this video.. Got to learn a great way of how to detect Outliers and remove them

  • @Ankurkumar14680
    @Ankurkumar14680 4 года назад +2

    It is an amazing explanation, eager to watch more videos on this topic....thanks a lot for sharing your knowledge and skills :)

    • @codebasics
      @codebasics  4 года назад +1

      I am glad it was helpful

  • @kirandeepmarala5541
    @kirandeepmarala5541 4 года назад +1

    max_threshold = df['price'].quantile(0.90)
    print(max_threshold)
    min_threshold = df['price'].quantile(0.05)
    print(min_threshold)
    df1 = df[(df['price'] > min_threshold) & (df['price'] < max_threshold)]
    Really, I am Learning a Lot from Your Channel..Waiting everyday for your videos..Your way of Explaining Concepts, Giving Exercises and genuine Talks makes your channel Different from Others..Thank you once again

  • @r7918
    @r7918 3 года назад

    Thanks a lot Sir!!! This video was very helpful for me. I was doing things wrong till now. I was under impression that outliers are meant to deleted. Instead of wasting time on irrelevant data points.

  • @atunraseayomide2504
    @atunraseayomide2504 2 года назад

    You never go wrong watching codebasics, I soo much love your work sir.

    • @codebasics
      @codebasics  2 года назад +2

      👍👍 thanks for you kind words of appreciation Atunrase 🙏

  • @leelavathigarigipati3887
    @leelavathigarigipati3887 4 года назад +2

    Thanks a lot for sharing your knowledge and skills.

  • @manojkumar-hf6vk
    @manojkumar-hf6vk 2 года назад

    you have created amazing series Sir.

  • @pallavikharbanda1372
    @pallavikharbanda1372 3 года назад

    Great video!!Please upload more methods to detect and remove outlier detection

  • @r21061991
    @r21061991 4 года назад +1

    Thank god u again started making videos😀😀

  • @stuttzzzi
    @stuttzzzi 2 года назад

    thank you..'if a person cant explain simply,,means he hasnt understood/knows it properly'
    following ue entire playlists

  • @OceanAlves23
    @OceanAlves23 4 года назад +5

    👏👏👏✔ from Brazil - Teresina - Piauí

  • @saisridatta844
    @saisridatta844 2 года назад

    Awesome sirr

  • @naveenkalhan95
    @naveenkalhan95 4 года назад +2

    would request you , if possible please add the link to the playlist of this series in the description of the video... though i have bookmarked it.. but it becomes really easy :) thank you very much for your good work again

  • @fahadreda3060
    @fahadreda3060 4 года назад

    Thanks for the video .. keep up the good work.. wish you all the best

  • @yogeshbharadwaj6200
    @yogeshbharadwaj6200 3 года назад

    Tks for the great video...

  • @temurochilov
    @temurochilov 2 года назад

    thank you a lot great content.

  • @TheShubham743
    @TheShubham743 4 года назад

    Great video

  • @Deepsim
    @Deepsim 2 года назад

    Very clear explaination! Thank you.

  • @saswatleo
    @saswatleo 3 года назад

    Amazing Video... Superb Dhaval Sir🙏

  • @santoshkumarmishra441
    @santoshkumarmishra441 3 года назад

    nice explanation sir like always

  • @AlonAvramson
    @AlonAvramson 3 года назад

    Thank you!

  • @immanuelsuleiman7550
    @immanuelsuleiman7550 4 года назад

    great content
    you just gained a new subscriber
    thank you for your efforts sir

    • @codebasics
      @codebasics  4 года назад

      I am glad it was helpful

  • @NikitaSharma-bs4gg
    @NikitaSharma-bs4gg 2 года назад

    thank you- really helpful

  • @nafisehyazdi9876
    @nafisehyazdi9876 3 года назад

    this tutorial was really good, Thank you

  • @AjayKumar-id7mb
    @AjayKumar-id7mb 3 года назад

    Really you explain it in a very easy way.

  • @PassionateVideography
    @PassionateVideography 10 месяцев назад

    Thank you Sir

  • @abhimistry9226
    @abhimistry9226 2 года назад

    Thank you dhaval bhai

  • @javedj5338
    @javedj5338 2 года назад

    nice explanation. clear

  • @ishandaar
    @ishandaar 4 года назад

    thanks, this video is really helpful

  • @saramoeini4286
    @saramoeini4286 4 года назад

    thanks alot. i have question . should we do this procedure for all features one by one for detecting outliers and then remove it?

  • @soniadubey4773
    @soniadubey4773 3 года назад

    thanks. very clearly explained.

  • @sumit121285
    @sumit121285 3 года назад

    you are a real teacher ..... thanks....thanks a lot....

  • @Ashokkumar-ds1nq
    @Ashokkumar-ds1nq 4 года назад

    The Tutorial is just amazing, can you please magnify your screen a bit for a clearer view?

  • @sakshamverma3114
    @sakshamverma3114 4 года назад

    Sir pls upload all the videos for feature engineering.......
    And you are teaching methods are great

  • @YO-in2ij
    @YO-in2ij 2 года назад

    thank you

  • @chrschra
    @chrschra 3 года назад

    Nice explanation, thx! But what do to if my data points following are following an exponential distribution?

  • @amc8437
    @amc8437 2 года назад

    How about using log transformation to remove the skewness, doesn't it do a similar job with min, max thresold?

  • @sudhanshusingh8508
    @sudhanshusingh8508 4 года назад

    Hello Sir, I have been through your video and its nice. Since I am new to data analytics I have fair theoretical knowledge about quantiles
    but don't know on what basis you choose the quantile level, I mean what do you look for in the describe command of any table.
    please help me with this.

  • @mohlagare3417
    @mohlagare3417 3 года назад

    thanks a lot for sharing your knowledge and skills , can you please give us dataset. Thanks in advance

  • @yashikasorathia8639
    @yashikasorathia8639 3 года назад +2

    Amazing tutorial sir, but I have one question. How would you decide what quantile value to keep since dataset would be from different domain apart from retail price such as weather report or sales report or any other?

    • @covelus
      @covelus 7 месяцев назад

      If I properly understood your question (good one, BTW), I would say domain knowledge mixed with an initial dataset statistical analysis.... Right?

  • @rajasekharboppasamudram4463
    @rajasekharboppasamudram4463 4 года назад

    Your videos are very easy to understand and thanks for the content.
    one request from my side, volume of your videos are less compared with other sources. If possible , please increase volume

    • @codebasics
      @codebasics  4 года назад

      Can you check volume of your computer. I played this video on my computer and it's quite good in terms of volume

  • @raghuram6382
    @raghuram6382 3 года назад

    Is it possible to filter out the outliers of multiple columns in a single program? please do let me know...

  • @sagar8460830871
    @sagar8460830871 4 года назад +1

    please make video how to setup gpu laptop. for deep learning project. i have gpu laptop but when i am starting training gpu is not process my task my haul task done on cpu. i have 4gb nvidia gtx 1650 graphics card

  • @trashantrathore4995
    @trashantrathore4995 2 года назад

    Hi, anyone did the Exercise? Actually, the main quantile outlier removal is done using the process but what to do with 10k NAN values in "last_review" and 'reviews_per_month" column? Apart from this exercise if we encounter that big number as NAN what should we do? Any suggestions...

  • @naveenkalhan95
    @naveenkalhan95 4 года назад +9

    @10:24 when you take min and max quantile as .001 and .999... I am confused with 0.001... Should it not be 0.01? (meaning 1 percentile). Thank you again.

    • @iketanbhalerao
      @iketanbhalerao 4 года назад +1

      Yes, I am also. bcz 0.001 means 0.1% ?

    • @viveksingh881
      @viveksingh881 3 года назад

      no need to get confused he just went too low with quantile value.... happy learning.....

    • @aldot1532
      @aldot1532 3 года назад +1

      no, basically he excluded 0.1 % of samples in both ends (upper and lower).
      the lower end is 0.1 %, which means he excluded 0.1% of samples below
      the upper end is 99.9%, which means he excluded 0.1 % of samples above

  • @mallickamu
    @mallickamu 3 года назад

    Correct me if I am wrong.... We can use percentile based outlier detection only if we know that the variation is normal distribution. For the height example taken by you, we know that it follows normal distribution, so the percentile based outlier detection can be applied. What if the variable distribution is Weibull type. Can we use percentile based outlier detection?

  • @robertaraujo347
    @robertaraujo347 2 года назад

    I'd like to know why you've used the variable price per area (which is just the quotient between price and area) to do the outlier treatment instead of using the mahalanobis distance (that infact, take into account the correlation between the variables) since you have 4 numerical columns.
    I hope someone can answer to me. I'm new in this and I have so many questions about it :)

  • @_Raz_
    @_Raz_ Год назад

    This is good only for small data sets . But if we have big datasets with multiple of col so it's very hard to apply

  • @shaikhkashif9973
    @shaikhkashif9973 Год назад +1

    8:10 After this if u check outliers in box plot u will find again because of q1 q2 q3 again assigning so that's reason u can't go for remove better go for replacement

  • @manish17788
    @manish17788 2 года назад

    what if data has no outlier. In that case we will loose tiny data? how to know if not outlier removal is needed in big dataset?

  • @mitalipatle4993
    @mitalipatle4993 Год назад

    Can we use this method with larger Data Set?

  • @suryanshsingh2873
    @suryanshsingh2873 Год назад

    How to find that where is the outlier present, because there are so many variable presents in the data set

  • @mithilanavishka4531
    @mithilanavishka4531 2 года назад

    Sir,
    How do we decide based on, which column to remove the outliers what is the logic for finding that column? I mean why cant we use other fields like total_sqft
    after I got describe of data, I saw 75% of samples have total_sqft less than 1672, and max total_sqft is 52272, I wanted remove row have maximum total_sqft, is it wrong ? the way I thought

  • @haintuvn
    @haintuvn 4 года назад +3

    "quantile(0.001 , 0.999)". When we choose 0.001 or why dot not we choose 0.005 or other? Are there any regulation/ suggestions to choose these numbers? Thank you teacher!

  • @matlabtutorials2986
    @matlabtutorials2986 8 месяцев назад

    Sir , hdbscan outlier detection pe bhe video banaye please

  • @akashgaddam6320
    @akashgaddam6320 3 года назад

    how can we fix the thresole value

  • @malinibhattacharyya304
    @malinibhattacharyya304 2 года назад

    Sir, how to get jupiter note book?

  • @MindyBrockdesigns
    @MindyBrockdesigns 2 года назад

    What if you have several variables to fix outliers for in one data set. For example What if you wanted to remove outliers in the ‘price’ and ‘price-per-square’ variables.

  • @preeethan
    @preeethan 4 года назад

    What if we want to treat the outliers rather than removing them.? Which is a better practise.?

    • @codebasics
      @codebasics  4 года назад

      You can treat them based on a situation either of them is good

  • @ankitac4994
    @ankitac4994 2 года назад

    Are there outliers present in categorical data?

  • @dhruvshah3394
    @dhruvshah3394 5 месяцев назад

    Just one correction @12:15 you mentioned minimum threshold is 1% but isn't it actually 0.1% ?

  • @khushpatelmd
    @khushpatelmd 4 года назад +3

    Shouldn’t it be 97.5 and 2.5 as 95% of values are within 2SD

  • @ismailkaracakaya260
    @ismailkaracakaya260 Год назад

    But how do you know if the outlier value is above 0.95th percentile?

  • @jaitiwari241
    @jaitiwari241 2 года назад

    where i can find 'bhp.csv file

  • @tauhidanwar3512
    @tauhidanwar3512 4 года назад +1

    I am confused between the percentage you have taken in min and max thersold. You said that you used 1% as min but you used 0.001 which 0.1 % and 99 % as max but you used 0.999 which is 99.9%. Plz clarify this issue sir. Thanks for this hard work for us.

    • @codebasics
      @codebasics  4 года назад +2

      I actually meant 0.1% and 99.9%, exact percentage or quartile can really vary on case to case bases. You basically use your sense of judgement to come up with these threshold values

  • @panduenglishacademy7856
    @panduenglishacademy7856 4 года назад +1

    I downloaded data from.kagle to.do hand on but when I import csv file in jupyter notebook by its name , it warn an error( name error, file not found
    Pls help me in solving this issue.)

    • @panduenglishacademy7856
      @panduenglishacademy7856 4 года назад

      Attribute error :- model panda has no attribute 'read'

    • @swapnshah3234
      @swapnshah3234 4 года назад +1

      @@panduenglishacademy7856 df = pd.read_csv('yourfilename.csv') . it should be like this format.

  • @sa89879
    @sa89879 4 года назад

    can yo do an example of removing outliers using box plot

    • @codebasics
      @codebasics  4 года назад +1

      Yes that is coming up

    • @sa89879
      @sa89879 4 года назад

      @@codebasics thank you very much

  • @ashfakurrahman79
    @ashfakurrahman79 Год назад

    Why quantile? and how quantile works?

  • @bhaskartripathi
    @bhaskartripathi 2 года назад

    Good explanation mate ! However you can apply fix percentile removal only on toy datasets.
    For real world data you would Hampel filters, Isolation forest, rolling window based MAD etc for outlier removal.

  • @haneulkim4902
    @haneulkim4902 2 года назад

    There are different ways to remove outliers, when to use what??

  • @mayanktripathi4u
    @mayanktripathi4u 4 года назад

    Is Outlier and Imbalanced are same concept or different?
    if different could you please share some information...
    i tried to find based on Definition both seems to be same, but both have different methods to detection and removal. So bit confused.

    • @codebasics
      @codebasics  4 года назад +1

      They are different concepts. By imbalanced most likely you are referring to imbalanced data sets in terms of machine learning where one class label have very less samples compared to another class label

    • @mayanktripathi4u
      @mayanktripathi4u 4 года назад

      @@codebasics - Does it means that Imbalanced data is mainly for Target / Class label and Outlier is for other features from the dataset?

  • @muditsrivastava7719
    @muditsrivastava7719 4 года назад

    Sir i am not able to download this csv file, it says that data is very large, what to do?

    • @codebasics
      @codebasics  4 года назад

      Can you just git clone it?

    • @muditsrivastava7719
      @muditsrivastava7719 4 года назад

      @@codebasics yes sir i did, but it says we can't make it to csv ( only raw data is available)

  • @mihirit7137
    @mihirit7137 11 месяцев назад

    the outliers in this video are the mean prices today 😅

  • @AnushkaSingh-YearBTechChemical
    @AnushkaSingh-YearBTechChemical 2 года назад

    This wont work in case if the data contains na values

  • @JatinSharma-tu2zg
    @JatinSharma-tu2zg 3 года назад

    Sir secand wala dataset chahiye

  • @tharallaanil4115
    @tharallaanil4115 3 года назад

    Data set please sir

    • @codebasics
      @codebasics  3 года назад

      You can find it on GitHub page

  • @zainnaveed267
    @zainnaveed267 2 года назад

    min_thresold = df['price'].quantile(0.01)
    _thresold = df['price'].quantile(0.9999)

  • @shubhamtyagi4962
    @shubhamtyagi4962 4 года назад

    exercise done
    github.com/styagi9817/oulier-detection-using-quantile-funtion

    • @codebasics
      @codebasics  4 года назад

      That’s the way to go Shubham, good job working on that exercise

  • @jeevan999able
    @jeevan999able 3 года назад

    binod

  • @borgir6368
    @borgir6368 3 года назад

    who else spot binod at 4:09 lol ;()

  • @krutkanjiya2434
    @krutkanjiya2434 3 года назад

    4:00
    Binod xD

  • @allasrinivasulu1464
    @allasrinivasulu1464 4 года назад

    Thank you so much sir