Estimate Reliability in R with Alpha, Omega, and Kappa

☕If you would like to support, consider buying me a coffee ☕: buymeacoffee.c...
For one-on-one tutoring/consultation services: guide-tree-sta...
I offer one-on-one tutoring/consultation services for many topics related statistics/machine learning. You can also email me [e-mail]
For rcode and dataset: gist.github.co...
This is a tutorial that describes different types of reliability and shows how to estimate them using R. The types of reliability reviewed are test-retest, parallel form, inter-rater, and internal consistency. Reliability coefficients reviewed include Cohen's Kappa, Chronbach's Alpha, and McDonald's Omega (Hierarchical and Total). The 'psych' package is used to estimate the reliability of the Beck's Depression Inventory (BDI).

[@PeaceOfMake]

Can't believe I'm doing calculations for my Master's Thesis the same day this is uploaded. You are awesome!

[@statsguidetree]

Happy to hear it was helpful and best of luck completing your thesis.

[@statsguidetree]

Here is the rcode with notes:
# Reliability is distinct from Validity but you cannot have a
# valid instrument if it is not reliable.
# Different types of reliability:
# Test Re-test;
# Parallel Form;
# Inter-rater;
# and Internal Consistency (e.g.,
# Split-half Reliability, Chronbach's Alpa, etc.).
##### load dataset for example
#install.packages("KernSmoothIRT")
library(KernSmoothIRT)
data(BDI)
# remove NAs
bdi

[@quiqueclari8731]

Great video, thank you very much!

[@statsguidetree]

Thanks so much for the compliment.

[@lucyh1208]

Thank you for this! Still I am unsure, which of these omegas should be reported in a methods part of a paper?

[@statsguidetree]

Deciding whether to report the Omega or Omega Hierarchical should depend on your instrument. Is the underlying factor structure a bifactor model, then you could report Omega Hierarchical to get an overall estimate of reliability despite the multi factor structure. If not, you can report Omega total. I linked to another source: journals.sagepub.com/doi/full/10.1177/2515245920951747

[@mahdimazidi4656]

Thanks for the great video. had a question would appreciate your help. For computing the omega reliability of a from a measure, I was wondering whether I need to use the nfactors argument or not? I use function omega from psych package. I select the items for that subscale, and then use the omega function, however, output for total omega is different when I use the nfactor = 1 in the code. Should I write nfactor = 1 because it is one subscale of the measure or should I just left it empty? Thanks a lot!

[@statsguidetree]

The number of factors to set should depend on the number of sub-factors you suspect that exist in your measure. In addition, at least 2 factors would be needed for meaningful results (though an output may still be provided by the function if 1 is entered). If you suspect only 1 factor, in that case it may be more meaningful to use and report alpha.

[@amogh2101]

Is there a way to interpret Omega (like we do chronbach's alpha)?

[@statsguidetree]

Yes, you can interpret them very similarly (both Alpha and Omega) when you are talking about the degree of estimated reliability of the instrument. Higher values suggest higher reliability. Hope that helped.

[@adoi]

Hello! How do I calculate the test-retest reliability using ICC?