I think a lot of applied researchers see/hear the phrase "alpha underestimates reliability" and think that alpha is a good default because it is conservative in being careful to not overestimate reliability. However, getting a nearly acceptable .70 from a covariance matrix describing two orthogonal factors is scary (2:46). The total score of this scale could be a meaningless average between two unrelated constructs. As someone who does applied research, it was important for me to read the McNeish 2017 article and do my own simulation to see that alpha can actually overestimate reliability as well.
Alpha does not overestimate reliability in the example that you mention, but you are correct that alpha can be misleading. The problem with alpha is that it does not quantify whether the variables measure one thing or multiple things. If you have a score generated from scales measuring two orthogonal factors, the scale score is almost certainly meaningless. Nevertheless, we can estimate the reliability of that scale score (think e.g. test-retest reliability). Alpha underestimates this reliability. The problem is that an acceptable alpha does not guarantee that you have a useful scale score. At minimum, you should also do a factor analysis as well.
After hours of making sense of what my textbook is saying, I finally understood, thanks to you!
I think a lot of applied researchers see/hear the phrase "alpha underestimates reliability" and think that alpha is a good default because it is conservative in being careful to not overestimate reliability. However, getting a nearly acceptable .70 from a covariance matrix describing two orthogonal factors is scary (2:46). The total score of this scale could be a meaningless average between two unrelated constructs. As someone who does applied research, it was important for me to read the McNeish 2017 article and do my own simulation to see that alpha can actually overestimate reliability as well.
Alpha does not overestimate reliability in the example that you mention, but you are correct that alpha can be misleading. The problem with alpha is that it does not quantify whether the variables measure one thing or multiple things. If you have a score generated from scales measuring two orthogonal factors, the scale score is almost certainly meaningless. Nevertheless, we can estimate the reliability of that scale score (think e.g. test-retest reliability). Alpha underestimates this reliability. The problem is that an acceptable alpha does not guarantee that you have a useful scale score. At minimum, you should also do a factor analysis as well.
Great, professor. Your videos have helped me understand many things that I have simply done mechanically. Thanks.
Great to hear!
So helpful!! I spent hours trying to figure out what tau equivalence means and now I finally get it :)
Good that it helped. Understanding that tau is used for true scores in CTT helps a lot. Tau equivalence = the items have the same true score.
Gracias por la explicación
this was really helpful! thanks :)
Awesome! Thank you!
You are welcome!
thank u so much
You are welcome