This video describes how address issues of construct validity. Mostly intended for viewing by students in the PSC 41 class at UC Davis. ______________________ Transcript:
Time to go over one of the hardest concepts in the whole class: construct validity. I recommend that you watch this video at least a few times…it will really help you. I’m going to describe the three component parts of construct validity, and then talk about a strategy that you can use to address construct validity questions when you get them. Before we do that, though, I want to talk more generally about validity for a minute. In order to conduct good research, it needs to be valid. There are several component parts to that validity. The first big breakdown of component parts is between internal and external validity. Both of these things are equally important if research is going to convince me to change how I think about the world. Actually, the overall validity of a study is a result of nested validities. Within internal validity we have, for example, statistical validity and construct validity. Within construct validity we have face, procedural, and methodological validity. In order for a researcher’s conclusions to follow from the research that was done, it has to be valid in all of the ways. If any type of validity isn’t there, the research doesn’t quite tell us what the researchers want us to think it does. This doesn’t mean it tells us nothing, it just means that it tells us something different than what we might have learned otherwise. For instance, if a study lacks external validity, that might mean that the conclusions don’t generalize to the broader population. When we think about criticizing the validity of a study, we want to get the most bang for our buck. Since studies will have a stronger claim for validity in some areas of their study, the biggest bang for our buck would be in criticizing those strong areas. If we find that even the strongest aspects of a study are invalid, then we can safely disregard the conclusions of the study. For instance, let’s say a study gathered the smartest people in an area and used a spelling test to assess their intelligence. It is clear that we can’t conclude that the average intelligence of this group is the average intelligence of the whole population, because of poor sampling. But even if they would have used a good sampling strategy, would they have been able to say much about intelligence? Not really! A spelling test doesn’t say a whole lot about intelligence. By choosing the strongest part of the study to attack, this totally invalidates the findings of this research. We want to use the same strategy to address all other types of validity, including construct validity. Instead of picking the weakest part of the construct and addressing that, we should pick the strongest part of the construct. If we find problems there, we have done a lot more damage to the claim the researchers were trying to make than if we had chosen the weakest part of the construct. When addressing questions about construct validity, then, you should choose the strongest of procedural, face, or methodological validity to address, as that is the best way for you to make your claim. If the strongest one doesn’t hold up, you can stop there. Since all of these things have to be valid for the construct to be valid, and since we chose the strongest of these things, finding problems with it demonstrates that the construct is invalid. If there are no problems in the strongest component, focus on the next strongest component. If there are still no problems, go to the weakest component. Once you have described how one of these things is invalid, you don’t need to describe anymore since you have invalidated the whole construct by invalidating one aspect of it. If you think the construct is valid, though, you need to explain why each of the aspects are valid. To highlight each of these types of validity, I’m going to use a silly example. For my research project, I want to know if the sun is up. Maybe it’s a post-apocalyptic world, and I’m waiting for the nuclear winter to be over. Anyway, I’ll use that example with different constructs to demonstrate the different types of validity. First, face validity. Face validity can be summed up as when you read something and you’re like, “What the hell…?” You just read it and think, “That is the dumbest idea I’ve ever heard of.” For example, I want to know if the sun is up. Knowing something about the geothermal cycles and convection currents of the air, I propose measuring if the sun is up by measuring the dust in the air. I’m going to follow all directions for my fancy dust detector, and I got the best dust detector in the world, it perfectly measures dust. This should be one of those “what the hell is he thinking” moments. That’s ridiculous. If I want to know if the sun is up, why am I measuring dust in the air. That construct has poor methodological validity. Okay, that was just plain silly. Time to measure something real. So, I still want to know if the sun is up. Instead of buying a really expensive dust detector, I decide to go for something much cheaper-some light-sensitive photography film. We know this only responds to light, so it makes sense that I would use it. Also, I’m going to follow all the directions perfectly, so it should be consistent in that way. But this film is pretty cheap, so they don’t do a lot of quality control on it. I mean, you could always use new film if you need to, right?! So, the method that I have chosen will introduce some error into the system. Your first reaction to the film might have been, “But that’s still ridiculous.” That reaction wouldn’t totally be wrong. Sometimes it’s hard to see where face validity ends and methodological validity begins. Sometimes there is no hard boundary. The best way to think about it is that methodological validity is a more rigorous version of face validity. That’s not going to be true in all cases, but it’s a good guide. Okay, now that I’ve decided to measure light instead of dust and not to use crappy film, I’m on my way to having a valid construct. I decide, this time, to buy a fancy light detector from my local photography store. I’m so excited when it arrives that I don’t even read the directions. I grab the side with the white bulb on it, hold out the other end and turn on the detector. Every time I turn it off and back on, I get a different reading, and it really frustrates me. After reading the directions, it turns out that the white bulb is what measures the light! By not following directions, not doing the procedure correctly, I introduced error into my results. This construct lacked procedural validity. Before we move on, it is easy to get procedural and methodological validity mixed up. In normal, every day language, we often use the word method to mean exactly how something was done. But that’s not how we mean it in psychology. In psychology, the method is what was done, the procedure is exactly how it was done.
Hi, could I ask, if I have an experiment that seeks to measure which burger is more popular, through a forced choice survey that only allows a selection of one burger, Would that affect construct validity, considering that I have failed to account that someone may like both burgers equally much, or may dislike both burgers? Would this pertain to construct validity? Or internal validity?
Time to go over one of the hardest concepts in the whole class: construct validity. I recommend that you watch this video at least a few times…it will really help you. I’m going to describe the three component parts of construct validity, and then talk about a strategy that you can use to address construct validity questions when you get them.
Before we do that, though, I want to talk more generally about validity for a minute. In order to conduct good research, it needs to be valid. There are several component parts to that validity. The first big breakdown of component parts is between internal and external validity. Both of these things are equally important if research is going to convince me to change how I think about the world. Actually, the overall validity of a study is a result of nested validities. Within internal validity we have, for example, statistical validity and construct validity. Within construct validity we have face, procedural, and methodological validity. In order for a researcher’s conclusions to follow from the research that was done, it has to be valid in all of the ways. If any type of validity isn’t there, the research doesn’t quite tell us what the researchers want us to think it does. This doesn’t mean it tells us nothing, it just means that it tells us something different than what we might have learned otherwise. For instance, if a study lacks external validity, that might mean that the conclusions don’t generalize to the broader population.
When we think about criticizing the validity of a study, we want to get the most bang for our buck. Since studies will have a stronger claim for validity in some areas of their study, the biggest bang for our buck would be in criticizing those strong areas. If we find that even the strongest aspects of a study are invalid, then we can safely disregard the conclusions of the study. For instance, let’s say a study gathered the smartest people in an area and used a spelling test to assess their intelligence. It is clear that we can’t conclude that the average intelligence of this group is the average intelligence of the whole population, because of poor sampling. But even if they would have used a good sampling strategy, would they have been able to say much about intelligence? Not really! A spelling test doesn’t say a whole lot about intelligence. By choosing the strongest part of the study to attack, this totally invalidates the findings of this research.
We want to use the same strategy to address all other types of validity, including construct validity. Instead of picking the weakest part of the construct and addressing that, we should pick the strongest part of the construct. If we find problems there, we have done a lot more damage to the claim the researchers were trying to make than if we had chosen the weakest part of the construct. When addressing questions about construct validity, then, you should choose the strongest of procedural, face, or methodological validity to address, as that is the best way for you to make your claim. If the strongest one doesn’t hold up, you can stop there. Since all of these things have to be valid for the construct to be valid, and since we chose the strongest of these things, finding problems with it demonstrates that the construct is invalid. If there are no problems in the strongest component, focus on the next strongest component. If there are still no problems, go to the weakest component. Once you have described how one of these things is invalid, you don’t need to describe anymore since you have invalidated the whole construct by invalidating one aspect of it. If you think the construct is valid, though, you need to explain why each of the aspects are valid.
To highlight each of these types of validity, I’m going to use a silly example. For my research project, I want to know if the sun is up. Maybe it’s a post-apocalyptic world, and I’m waiting for the nuclear winter to be over. Anyway, I’ll use that example with different constructs to demonstrate the different types of validity.
First, face validity. Face validity can be summed up as when you read something and you’re like, “What the hell…?” You just read it and think, “That is the dumbest idea I’ve ever heard of.” For example, I want to know if the sun is up. Knowing something about the geothermal cycles and convection currents of the air, I propose measuring if the sun is up by measuring the dust in the air. I’m going to follow all directions for my fancy dust detector, and I got the best dust detector in the world, it perfectly measures dust. This should be one of those “what the hell is he thinking” moments. That’s ridiculous. If I want to know if the sun is up, why am I measuring dust in the air. That construct has poor methodological validity.
Okay, that was just plain silly. Time to measure something real. So, I still want to know if the sun is up. Instead of buying a really expensive dust detector, I decide to go for something much cheaper-some light-sensitive photography film. We know this only responds to light, so it makes sense that I would use it. Also, I’m going to follow all the directions perfectly, so it should be consistent in that way. But this film is pretty cheap, so they don’t do a lot of quality control on it. I mean, you could always use new film if you need to, right?! So, the method that I have chosen will introduce some error into the system.
Your first reaction to the film might have been, “But that’s still ridiculous.” That reaction wouldn’t totally be wrong. Sometimes it’s hard to see where face validity ends and methodological validity begins. Sometimes there is no hard boundary. The best way to think about it is that methodological validity is a more rigorous version of face validity. That’s not going to be true in all cases, but it’s a good guide.
Okay, now that I’ve decided to measure light instead of dust and not to use crappy film, I’m on my way to having a valid construct. I decide, this time, to buy a fancy light detector from my local photography store. I’m so excited when it arrives that I don’t even read the directions. I grab the side with the white bulb on it, hold out the other end and turn on the detector. Every time I turn it off and back on, I get a different reading, and it really frustrates me. After reading the directions, it turns out that the white bulb is what measures the light! By not following directions, not doing the procedure correctly, I introduced error into my results. This construct lacked procedural validity.
Before we move on, it is easy to get procedural and methodological validity mixed up. In normal, every day language, we often use the word method to mean exactly how something was done. But that’s not how we mean it in psychology. In psychology, the method is what was done, the procedure is exactly how it was done.
Hi, could I ask, if I have an experiment that seeks to measure which burger is more popular, through a forced choice survey that only allows a selection of one burger,
Would that affect construct validity, considering that I have failed to account that someone may like both burgers equally much, or may dislike both burgers?
Would this pertain to construct validity? Or internal validity?
Thank you for explaining this. I don't like statistics or anything it involves but you explained very well! :)
Fantastic, thank you
thank you
thanks alot :)