Amazing video you have put together here. I enjoyed how clear you were as well as the pace you took to go through the steps and explain everything. I am new to this kind of thing so does anyone have resources on where I can learn how to interpret cluster graphs
I will be jumping between some Python topics and machine learning topics over the future episodes. Is there any particular algorithms you would like to see covered?
Thanks! I have been doing this on resistivity and seismic values on different profiles in a catchment. However, everytime I get same trend but clusters change in their places. Would like to know about this issue...
pretty cool. I have used K-means and DBSCAN to identify electrofacies, but I am still working on a way to optimize this task. It would be grade to see the Well Plots (depth Vs logs) with each point identified by its own cluster.
Thanks Jose. I did have a section of code for displaying the facies data on a log plot but I did not include it in the video. The full plotting code can be found here: towardsdatascience.com/how-to-use-unsupervised-learning-to-cluster-well-log-data-using-python-a552713748b5
Hi Andy, this was a great tutorial as it's something I would like to try on a csv file with various metrics in the design of a pharmaceutical. I have one question though: I will be wanting to use 5-7 columns on the csv file for clustering - how do you go about visually representing this? I can't think of a good way to do it. Thanks!
hi , thanks for the video, but could you please direct me that which file in your github is the jupyter notebook for this video? I could not find it. thanks
Andy, thanks for sharing. I can’t find the notebook for this specific exercise. I am trying to follow along with a different dataset but I am getting an error “name ‘means’ is not defined” when trying to determine the number of clusters.
Hi Timothy, did you manage to resolve this? If not, I would go back and check you have ran all of the cells before trying to determine the number of clusters.
Hi, I have one question about scettering in 13:21. Why were 'NHPI' and 'RHOB' written in 'plt.scatter()' when all calculations were done according to scaled data (I mean 'NHPI_T' and 'RHOB_T')? I am just trying to learn it. Could you please help me?
Using the scaled data within certain algorithms can reduce the effect of different data ranges (e.g feature1 ranges from 0 to 1, and feature2 ranges from 0.1 to 10,000), and scaling can also help speed things up. Some algorithms such as decision trees/random forests don't really need scaling whereas Neural Networks and even clustering can benefit from this process. Plotting the data using the original curves allows us to see how the calculated clusters align with the original data. If we were using scaled data, then the numbers on the axes wouldn't make too much sense for petrophysical interpretation. Hope that helps :)
@@AndyMcDonald42 Yes. It helps. :) Thank you very much. Also I have other question. Is there any way to get information about point in the graph by click using mouse to see which point belongs to which data?
@@ahmetatasever8315 Yes, there certainly us, The plot shown in this video was done with matplotlib, which is used to create a basic and static figure. You could easily swap that out for Plotly, which will have the extra interactivity and give extra info on hover.
In the .fit() call at 12:00 you would pass in more variables. I have just used 2 for this example to illustrate what the output is like. Hope that helps :)
Can I say that at the end of the day, the way of interpreting the clusters is kind of subjective especially when the dataset gets more complex? Since the results could vary quite a lot as you apply different clustering algorithms or tuning some of their parameters. So it could be quite subjective, no?
Yes. That is very true. It is down to you or the person doing the interpretation to understand what the cluster may represent. If another person does there own interpretation they may have their own understanding of what the clusters represent
hello Andy, thanks for well-explained session,but on the final part can you assist to explain as to which features or measures differentiate one cluster from other,Thanks again
Thanks Dominic. One way would be to use a facet grid plot from seaborn and split by the clusters. You could then view the data by histograms, scatter plots and other plot types. That way you can see how the data features vary per cluster
Thank you so much for this video. I downloaded the data you used and found a negative relationship between RHOB and NPHI. Can tell me how your scatterplot shows a positive relationship between them? Thank you.
No problem. You are correct that NPHI and RHOB are usually anti-correlated. In petrophysics, we normally display RHOB on an inverted scale, often on the Y-axis. As RHOB values get lower, we likely have a higher porosity, and the values will plot higher up on the y-axis. For higher NPHI (neutron porosity) values, the points will plot further to the right. If we have a case where both NPHI and RHOB are high, they will then plot in the top right. It's a nice and easy way to visualise and identify potential reservoir intervals.
@@AndyMcDonald42 Thank you so much. I am using it to cluster customer data, but I wanted to make sure I could replicate yours before trying. Thank you again for the explanation and such an awesome tutorial.
Has anyone had an issue with In[9] when running from Jupyter Lab? Have fully checked for any spelling errors. The assignment and new columns seems to try to access. KeyError: "None of [Index(['RHOB_T', 'GR_T'. 'NPHI_T', 'PEF_T', 'DTC_T'], dtype='object')] are in the [columns]"
I'm working on clustering energy consumption profiles of a group of households, how should the starting dataset be structured? For each apartment I'm given the annual energy consumption profile (15 minutes frequency for 1 year), the number of appliances and the number of rooms
Sounds like an interesting task 🙂 If I understand correctly, you have a continuous variable for the energy consumption and then fixed variables for the rest? Have you considered clustering based on the profiles alone and grouping them into something like high energy users and low energy users or early birds and night owls? After that you could then try to use the other properties to gain more insights
I believe this may have been my original notebook. It contains much more detail than what I covered in the video. I hope this helps. github.com/andymcdgeo/Petrophysics-Python-Series/blob/master/18%20-%20Unsupervised%20Clustering%20for%20Lithofacies.ipynb
Thank you so much, Andy! I really find your video helpful. I am just wondering whether it would be possible for us to draw the scatter plot in multi-dimensions? Cuz I followed all of your steps but could not continue the step after the elbow plot when using my 500 columns dataframe.
Thanks Patty. You would only be able to draw the scatter plot up to 3 dimensions (X, Y and Z). However, you could look at using Seaborn's Pairplot to view 2d scatter plots of each of the variables versus the others: ruclips.net/video/D5DPZyge31g/видео.html I would be wary though of using 500 features with this plot as it will become unwieldy. I would be asking myself the following in your situation: - Do I require all 500 columns? - Are all of the columns relevant? - Can I reduce them manually or look at algorithms such as PCA to reduce the dimensionality of the dataset.
I have trouble using kemans.labels_ at the end it keeps showing this error: 'numpy.ndarray' object has no attribute 'labels_' can someone help me with this? Thank you!
I'm using k-means for the first time. my dataset has more than 400,000 rows and 11 columns, I run the k-means for k= 3, 5, 7, 9, and 10. it took more than 3 hours and still no output. is that normal?
Sorry for the late reply. I realised I hadn't uploaded the file to the repo. You can find it here: github.com/andymcdgeo/Petrophysics-Python-Series It is Notebook 18.
Andy, i get some NaN value on the datasets.. and then when i try to run the "df.dropna(inplace = True)", all of the datasets become empty (zero). How to handle this? Thankyou
![K-means Clustering From Scratch In Python [Machine Learning Tutorial]](http://i.ytimg.com/vi/lX-3nGHDhQg/mqdefault.jpg)
![Stromae, Pomme - “Ma Meilleure Ennemie” (from Arcane Season 2) [Official Visualizer]](http://i.ytimg.com/vi/1F3OGIFnW1k/mqdefault.jpg)
criminally underrated channel!
Your explanations are superb
Thanks ! I am geoscientist just starting my data sciences journey and I find your videos very helpful
Please can you help me I want to know more about data sciences applying in geosciences
Your fluency and skill, simply superb! Keep it up!
Thank you! 😃
Explained this better than my professor. Big W
That was the best explanation what i watch for KClustering thank you 😊
Great presentation. The clearest I've seen on RUclips, to date. 👍
Very nice, simple, clear and to the point. Thank you for sharing.
Thank you! The example script is a huge help
This was such a fantastic tutorial, thank you for putting quality content out there.
Glad you liked it!
very useful thank you! I'm midway through a data analysis apprenticeship and this helped me alot!
You're very welcome! I am glad to hear it has been helpful.
You’re a star. Thank you. Subscribed… very well explained
Amazing video you have put together here. I enjoyed how clear you were as well as the pace you took to go through the steps and explain everything. I am new to this kind of thing so does anyone have resources on where I can learn how to interpret cluster graphs
Andy - Your videos are very helpful and informative! Thank you!
Glad you like them! Thanks!
Excellent tutorial! Thank you very much for your time
Thanks buddy, your lesson helped me a lot
Precise and clear👍👍plz explain naive based, Support vector machine & decision tree as well
Thanks again for the content, Andy! You're a great teacher!
Thanks Allan. Glad to hear you are enjoying the content.
Hi Andy I think you start machine learning topic and it's my favorite topic thank you 🙏🙏
I will be jumping between some Python topics and machine learning topics over the future episodes. Is there any particular algorithms you would like to see covered?
Thanks! I have been doing this on resistivity and seismic values on different profiles in a catchment. However, everytime I get same trend but clusters change in their places. Would like to know about this issue...
pretty cool. I have used K-means and DBSCAN to identify electrofacies, but I am still working on a way to optimize this task.
It would be grade to see the Well Plots (depth Vs logs) with each point identified by its own cluster.
Thanks Jose. I did have a section of code for displaying the facies data on a log plot but I did not include it in the video. The full plotting code can be found here: towardsdatascience.com/how-to-use-unsupervised-learning-to-cluster-well-log-data-using-python-a552713748b5
hi. thank you for this wonderful tutorial. where do you recommend choosing data sets from?
Hi Andy, this was a great tutorial as it's something I would like to try on a csv file with various metrics in the design of a pharmaceutical. I have one question though: I will be wanting to use 5-7 columns on the csv file for clustering - how do you go about visually representing this? I can't think of a good way to do it. Thanks!
Excellent presentation and explanation
is there a place from where I see the code you have written for this as that would help me in learning. Thanks
hi , thanks for the video, but could you please direct me that which file in your github is the jupyter notebook for this video? I could not find it. thanks
Absolutely useful. Thank you Andy
Great to hear!
great tutorial, thank you.
I’ve searched for this file in the github repository and I didn’t find this tutorial’s code file
You are a hero!
Why are they useful? Do we know what qualities does these clusters have? Are they meaningful if we have lots of variables?
Andy, thanks for sharing. I can’t find the notebook for this specific exercise. I am trying to follow along with a different dataset but I am getting an error “name ‘means’ is not defined” when trying to determine the number of clusters.
Hi Timothy, did you manage to resolve this?
If not, I would go back and check you have ran all of the cells before trying to determine the number of clusters.
Hey! great video, only one question. What if I want to set my own centroids?
Hi, I have one question about scettering in 13:21. Why were 'NHPI' and 'RHOB' written in 'plt.scatter()' when all calculations were done according to scaled data (I mean 'NHPI_T' and 'RHOB_T')? I am just trying to learn it. Could you please help me?
Using the scaled data within certain algorithms can reduce the effect of different data ranges (e.g feature1 ranges from 0 to 1, and feature2 ranges from 0.1 to 10,000), and scaling can also help speed things up. Some algorithms such as decision trees/random forests don't really need scaling whereas Neural Networks and even clustering can benefit from this process.
Plotting the data using the original curves allows us to see how the calculated clusters align with the original data. If we were using scaled data, then the numbers on the axes wouldn't make too much sense for petrophysical interpretation.
Hope that helps :)
@@AndyMcDonald42 Yes. It helps. :) Thank you very much. Also I have other question. Is there any way to get information about point in the graph by click using mouse to see which point belongs to which data?
@@ahmetatasever8315 Yes, there certainly us, The plot shown in this video was done with matplotlib, which is used to create a basic and static figure. You could easily swap that out for Plotly, which will have the extra interactivity and give extra info on hover.
@@AndyMcDonald42 Thank you again :)
very helpful . If you could use example that can be easily understandable for non-science community would be extra helpful!!!
THANKS YOUUUUU AHHHHH SO HAPPY I DID IT
Okay. So how to draw conclusion from these clusters ? I mean, what are your insights from this model ?
Thank you Andy, great video! What if I want to cluster more than 2 variables?
In the .fit() call at 12:00 you would pass in more variables. I have just used 2 for this example to illustrate what the output is like.
Hope that helps :)
Amazing video, thank you Sir
Yasss! A fellow Scot!!!
Can I say that at the end of the day, the way of interpreting the clusters is kind of subjective especially when the dataset gets more complex? Since the results could vary quite a lot as you apply different clustering algorithms or tuning some of their parameters. So it could be quite subjective, no?
Yes. That is very true. It is down to you or the person doing the interpretation to understand what the cluster may represent. If another person does there own interpretation they may have their own understanding of what the clusters represent
hello Andy, thanks for well-explained session,but on the final part can you assist to explain as to which features or measures differentiate one cluster from other,Thanks again
Thanks Dominic.
One way would be to use a facet grid plot from seaborn and split by the clusters. You could then view the data by histograms, scatter plots and other plot types. That way you can see how the data features vary per cluster
@@AndyMcDonald42 thank Andy,this is useful,I real appriciate
thank you andy for your sharing 🙏🙏
My pleasure
Thank you so much for this video. I downloaded the data you used and found a negative relationship between RHOB and NPHI. Can tell me how your scatterplot shows a positive relationship between them? Thank you.
No problem. You are correct that NPHI and RHOB are usually anti-correlated. In petrophysics, we normally display RHOB on an inverted scale, often on the Y-axis. As RHOB values get lower, we likely have a higher porosity, and the values will plot higher up on the y-axis. For higher NPHI (neutron porosity) values, the points will plot further to the right. If we have a case where both NPHI and RHOB are high, they will then plot in the top right. It's a nice and easy way to visualise and identify potential reservoir intervals.
@@AndyMcDonald42 Thank you so much. I am using it to cluster customer data, but I wanted to make sure I could replicate yours before trying. Thank you again for the explanation and such an awesome tutorial.
What a great tutorial, thanks a lot🥰🥰
Glad you like it!
Has anyone had an issue with In[9] when running from Jupyter Lab? Have fully checked for any spelling errors. The assignment and new columns seems to try to access.
KeyError: "None of [Index(['RHOB_T', 'GR_T'. 'NPHI_T', 'PEF_T', 'DTC_T'], dtype='object')] are in the [columns]"
Solid video :)
Btw, where is your accent from?
do we only use 2 features of a data while using k means clustering or did you do it for visualization purposes?
I'm working on clustering energy consumption profiles of a group of households, how should the starting dataset be structured?
For each apartment I'm given the annual energy consumption profile (15 minutes frequency for 1 year), the number of appliances and the number of rooms
Sounds like an interesting task 🙂
If I understand correctly, you have a continuous variable for the energy consumption and then fixed variables for the rest?
Have you considered clustering based on the profiles alone and grouping them into something like high energy users and low energy users or early birds and night owls?
After that you could then try to use the other properties to gain more insights
Maybe have a look at time series clustering techniques for grouping the profiles
An error is raised after writing (kmeans_3) while plotting (NPHI vs. RHOB)
Thank you, Andy, I could not find the notebook in your github.
I believe this may have been my original notebook. It contains much more detail than what I covered in the video. I hope this helps.
github.com/andymcdgeo/Petrophysics-Python-Series/blob/master/18%20-%20Unsupervised%20Clustering%20for%20Lithofacies.ipynb
Thank you so much, Andy! I really find your video helpful. I am just wondering whether it would be possible for us to draw the scatter plot in multi-dimensions? Cuz I followed all of your steps but could not continue the step after the elbow plot when using my 500 columns dataframe.
Thanks Patty.
You would only be able to draw the scatter plot up to 3 dimensions (X, Y and Z). However, you could look at using Seaborn's Pairplot to view 2d scatter plots of each of the variables versus the others: ruclips.net/video/D5DPZyge31g/видео.html
I would be wary though of using 500 features with this plot as it will become unwieldy.
I would be asking myself the following in your situation:
- Do I require all 500 columns?
- Are all of the columns relevant?
- Can I reduce them manually or look at algorithms such as PCA to reduce the dimensionality of the dataset.
i liked it, had to hit that belllll
I have trouble using kemans.labels_ at the end it keeps showing this error: 'numpy.ndarray' object has no attribute 'labels_' can someone help me with this? Thank you!
Could you please share the link to get the dataset?
Thanks alot for your helpful videos..
Glad you like them!
I have problem when trying calculate using excel, the result is different with code, what can i do to fix it?
I'm using k-means for the first time. my dataset has more than 400,000 rows and 11 columns, I run the k-means for k= 3, 5, 7, 9, and 10. it took more than 3 hours and still no output. is that normal?
Thank you so much !!
Thank you Andy! I just want to ask you where can I find this notebook to download and work with it? Thanks again!
Sorry for the late reply. I realised I hadn't uploaded the file to the repo. You can find it here: github.com/andymcdgeo/Petrophysics-Python-Series
It is Notebook 18.
@@AndyMcDonald42 thank you!! Please, keep on doing videos like this, I've been learning a lot!
I cannot find notebook file of this video in your git
great content
Excellent thanks
You are welcome
Thanks a lot!
No problem 👍
Andy, i get some NaN value on the datasets.. and then when i try to run the "df.dropna(inplace = True)", all of the datasets become empty (zero). How to handle this? Thankyou
I would check if one or more columns are entirely nan.
how to create input and output lines? pls help
sir, how to clustering data 2d with size(512,512), please help me sir tq
thanks a lot
Too good
Where is the meaning the columns of Data?
It keeps saying name means not defined :(
finally, a non-indian accent speaker