Cluster analysis in R | Finding out Intra and Inter cluster distances and optimum number of clusters
HTML-код
- Опубликовано: 7 авг 2024
- ~Hello everyone in this tutorial, I am going to cover almost all concepts involved in cluster analysis, starting from how to find an optimum number of clusters using wss plot function, silhouette method using factoextra package and also by using NbClust package, for k-means and hierarchical clustering algorithms, in hierarchical clustering I will explain about dendrograms, how to plot, export and hang them. In this tutorial, I will also show how to find the inter and intracluster distances using an R package called clv. Finally how to find the cluster means or average
0:00 Introduction
2:10 Preliminary steps
6:40 k means
7:20 Opt no of clusters
12:50 Hierarchical clustering
19:00 Cluster distances
21:27 Cluster means
Other youtube videos
Coding
1) • Introduction to Cluste...
2) • Cluster analysis
Theory
1) • 4 Basic Types of Clust...
2) • Clustering: K-means an...
3) • StatQuest: K-means clu...
4) • StatQuest: Hierarchica...
Descriptive
~NbClust
1) www.rdocumentation.org/packag...
~Finding the optimum number of clusters
1) uc-r.github.io/kmeans_clustering
2) stackoverflow.com/questions/1...
~Distances
1) www.geeksforgeeks.org/ml-inte...
Script
docs.google.com/document/d/19...
![Hierarchical Cluster Analysis [Simply explained]](http://i.ytimg.com/vi/8QCBl-xdeZI/mqdefault.jpg)
Check the script in the video description............
Hi there, don't hesitate to ask your doubts here....... Feel free to ask me.
Excellent information, thanks for explaining every step very clearly
It is more informative Sir thank you more!.
Thank you. this is very helpful
very useful lecture sir, thanks
Amazing, BIG!
Nicely explained
Thanks more for sharing the video. If possible can you show an other video on Prep-design and lattice design (i.e. simple and triple lattice) data analysis ? Thank you...
You are welcome Zeru, you can check the recent video on Agricultural statistical tools which is uploaded recently in my data analysis playlist you may find what you're looking for in STAR app manual.
Very good explanation, the pea.xlsx file can you post it? Did you publish your findings ?
hello sir, my design was Augmented 2 with 7 blocks ..210 RILs of sorghum , with 6 checks replicated in each block . In case of this should igo for hierarchical clustering analysis using k means and include adjusted mean of 210 lines for analysis. . am I right s?
I have to do interpretation on the bases of fodder yield. ..so is their any specification for doing trait wise clustering..
I have taken quantative traits along with shoot fly associated dead heart percent etc traits ..
as my RILs are the population of shoot fly susceptible x resistance cross ....
so how should I proceed for the analysis on diversity for forage sorghum based on my characters ...total 18 characters were taken..... Should I limit my traits according to my objective of interpretation ..or should I subject all the traits ....
.is for non replicated data non hierarchical clustering Is different from hierarchical ?
sir,how to make the circular cluster dendrogram
Thank you for the video sir. If scaling is only for quantitative trait what could we do for qualitative traits...how to do cluster analysis for qualitative traits?
Most often we won't consider qualitative traits, if you are considering them the clustering will not be good, if you want to that you can do it in SPSS using two step clustering option.
@@Guruprasad_A Ok sir. Thank you for your reply.
The topic was very simple clearly & well explained.
Your video is now among my go to resources in data science.
Just a small constructive criticism : You have already added your R code however the dataset you used (pea.xlsx) hasn't been included.
Thus, your script won't work without it & it may have no use for people who wishes to try practice themselves.
So, that would be much better if you include the excel file as well, imho.
But overall fantastic tutorial & very helpful for rookies
👍
Which method will be suitable for cluster analysis for 300 treatments/genotypes and how can i represent it using dendrogram or any clustering technique for data visualization.
Ward method is more suitable.
How to get percent contribution of each independent trait on the dependent variable similar to that one used by Tocher's ?
stackoverflow.com/questions/62332254/is-there-a-way-to-determine-the-weight-of-different-attributes-used-for-r-cluste
Check out the above article on FeatureImpCluster, I think you need to get it from devtools.
I wanted to perform hierarchical clustering for my data which has 4k observations,
I have applied 'NbClust' with smaller number of clusters i.e 2 - 9, and it throws this error -
" Error in plot.new() : figure margins too large "
Try to adjust the number of clusters in between 1 to 15 or so..
@@Guruprasad_A Sure, I will try that
Thank you
Sir i have doubt, how to write multivariate D2 analysis and ward D2 method hierarchical clustering in thesis, they differing in cluster partition.
You go with one which suits you. Mostly for diversity studies D2 test is better.
After calculating d sq distance according to privious video.
Then according to this video number of cluster is different how we can solve this. Please respond
You won't get similar results stick to one method.
Can you please do analysis for path and corelation .i am not able to run the program
Check my video on correlation.
Hi. I am new to this field. I'am trying to conduct as you illustrated, but I got a pause at clustering with NbClust: Error in NbClust(data = newdata, diss = NULL, distance = "euclidean", :
The TSS matrix is indefinite. There must be too many missing values. The index cannot be calculated.
Could you help me?
stackoverflow.com/questions/46067602/the-tss-matrix-is-indefinite-there-must-be-too-many-missing-values-the-index
stats.stackexchange.com/questions/204760/r-how-to-fix-nbclust-error-with-error-message-the-tss-matrix-is-indefinite?newreg=6dbb3e394e5f40cf8b1e50a16f88ee84
Can you please make video on basics of R...or recommend any source for the same
Yup, I will do
I recommend you to join datacamp, to learn syntax
how to understand wss silhouette NbClust giving different clusters number suggestion? which we should believe more?
Just look the things given in my video description.
NBClust. because it is based on 30 different indices.
Hi I did clustering using ward.d , and got 5 optimal number of clusters but in cluster mean it is giving mean for only 4 clusters .,why
I hope there might be a problem while selecting number of clusters object, in case of that you can manual calculate the average by adding one more extra column alongside of treatments indicating which cluster they belong, later sort them and get the average using MSExcel.
Which inter class and intra class cluster should we use?
Most often it's average only
When I used NbClust I am getting this message 'Error in NbClust(data = Table.s, diss = NULL, distance = "euclidean", :
The TSS matrix is indefinite. There must be too many missing values. The index cannot be calculated."
The silhouette method is given number of clusters as 7
stats.stackexchange.com/questions/204760/r-how-to-fix-nbclust-error-with-error-message-the-tss-matrix-is-indefinite?newreg=6dbb3e394e5f40cf8b1e50a16f88ee84
Consider 7 only for k means clustering
Dear Sir, I am doing cluster analysis of 40,000 points. I used the efficient hierarchical clustering algorithm and successfully did the clustering. Now I want to choose the optimal no of clusters. Which one would you suggest for dealing with large dataset? Because I need to test from k=1 , … , k=40000, which is huge
www.datanovia.com/en/lessons/clara-in-r-clustering-large-applications/
?NbClust
Pass the above argument in the R script to get more help, where you can understand what you can make out with help of NbClust
@@Guruprasad_A hi i tried nbclust, but still its quite slow to compute all one by one.
@@minhaoling3056 Try to define a narrow range for the number of clusters in the NbClust function i.e min.nc and max.nc
I have loaded corrected means from augmented RCBD.. But one numerical trait is taken as character.. after changing to numeric it is showing values as 'NA' for all genotypes... This unable to proceed with k means .. it is showing error... How to go about..? Kindly help..
Check there might be problem in the data what you have entered. There might be double pulstop or a alphabet like o instead of zero.
Can we get label font size clear and small
Dendrogram labels?
How we generate output file for all this process ?
sink("output.txt")
# your code to get the output
.
.
sink()
in this result where is the genotypes grouping on the basis of clusters
When I run NbClust(data=rice.s, diss= NULL, distance= "euclidean", min.nc = 2, max.nc = 15,
method = "complete", index= "all", alphaBeale = 0.1), then I got the following error:
Error: division by zero!Error in solve.default(W) : Lapack routine dgesv: system is exactly singular: U[6,6] = 0
How can I solve this?
Thank you.
Change the index.
How to fill rectangle and lines with color in hierarchical dendrogram
ruclips.net/video/JRGMePP6A5k/видео.html
Check this out
can i have the code for vertical viewing of this dendrogram
www.sthda.com/english/wiki/beautiful-dendrogram-visualizations-in-r-5-must-known-methods-unsupervised-machine-learning
An error is comming while loading package factoextra : package or namespace load failed for 'ggplot2' in loadnamespace (i, c(lib.loc, .libPaths()), versioncheck = vI[[i]] : namespace 'ellipsis' 0.3 .1 is already loaded , but >= 0.3.2 is required.
Please provide solution for this
Have you already installed ggplot2. If no or already installed, also try installing once again with this command
Install.packages("ggplot2")
After loading ggplot2
Then load factoextra
@@Guruprasad_A thanks for guiding....an update was needed
where can i get the pea dataset
It's my research data.
hello sir , I'm getting error after running this command:
NbClust(data = clust.s, diss = NULL, distance = "euclidean", min.nc = 2,
max.nc =15,method = "kmeans",index = "all",alphaBeale = 0.1)
error is:
Error in NbClust(data = clust.s, diss = NULL, distance = "euclidean", :
The TSS matrix is indefinite. There must be too many missing values. The index cannot be calculated.
please gives your valuable suggestions regarding this
stackoverflow.com/questions/46067602/the-tss-matrix-is-indefinite-there-must-be-too-many-missing-values-the-index
stats.stackexchange.com/questions/204760/r-how-to-fix-nbclust-error-with-error-message-the-tss-matrix-is-indefinite?newreg=6dbb3e394e5f40cf8b1e50a16f88ee84
Consider silhouette method
Error in t(jeu) %*% jeu :
requires numeric/complex matrix/vector arguments
Getting this error while computing the optimum number of clusters using NbClust ...Please help
Does your dataset contain any non numeric variable.
@@Guruprasad_A: no it does not. I am using the adjusted mean data of the augmented RCBD design. I did that analysis using by seeing your video only "augmented RCBD in R".
While importing data check whether all variables in double.
@@Guruprasad_A : Got it fixed, thanks alot for the wonderful video,please share how to extract the analysis report in word format, as you did in previous video.
It's not possible in this package.
Error in hclust(d, "ward.D") : Invalid clustering method , getting this error
try using hclust(dist(data, "ward.D")
When I am running the NbClust it gives me error Error in solve.default(W) :
system is computationally singular: reciprocal condition number = 8.06347e-18
stackoverflow.com/questions/56160674/how-do-i-fix-nbclust-error-solve-defaultw-in-mtcars-dataset
It's because of your data, just ignore this method and if you are looking for the optimum number of cluster for k-means use silhouette method.
www.biostars.org/p/407232/
If possible try taking out too much correlated data, if you have replication use Tocher method/D2 statistics
@@Guruprasad_A Yes silhouette method is giving 2 clusters
> NbClust(data = pea.s, diss =NULL, distance = "euclidean", min.nc = 2, max.nc = 15,
+ method = "kmeans" , index = "all", alphaBeale = 0.1)
Error in NbClust(data = pea.s, diss = NULL, distance = "euclidean", min.nc = 2, :
The TSS matrix is indefinite. There must be too many missing values. The index cannot be calculated.
can you tell me how to overcome this problem???
You can use the results from other methods like silhouette.
When I went to make dendrogram by watching your video, it is saying that figure margins are too large and I m not getting anything in plots.
Try to export with optimum resolution
Or do it in any other computer and see.
stackoverflow.com/questions/23050928/error-in-plot-new-figure-margins-too-large-scatter-plot
Error in solve.default(W) :
system is computationally singular: reciprocal condition number = 6.36052e-19
In addition: Warning message:
In log(det(P)/det(W)) : NaNs produced
Sir please help with this error.
It's because of too much correlations in your data
Then how to solve this problem sir@@Guruprasad_A
Please share the data file "pea"
It's my research data sir, I have to wait until it get published in Krishikosh
I have problem with finding clusters!
library(cluster)
library(class)
library(clv)
data(faba.s)
# compute intercluster distances and intracluster diameters
#kmeans
kid
Could you share the data set pls
It's my research data, I will share the link once after it's published in Krishikosh by our university. I hope you understand.