This is my first time leaving comments on RUclips, and this video has been that much valuable. The most valuable and helpful to me were the necessary references and the must-have critical viewpoints. I will need to read and study far more to come up with worthy questions for you, but until then, allow me to express my appreciation. Thank you, Mikko.
Hello Professor, thanks for this excellent video ! There were two questions that clicked was 1) Can we retain common latent factor and marker variable in our CFA and therefore SEM model if we get better model fit indices ? 2) How can we report/justify this ? 3) Also, can you please suggest any article that mentions about how to choose a marker variable?
1) The purpose of SEM is not to get better fit indices and model fit is not be an ideal metric for deciding whether a method factor is required. The reason for this is that even if there was a strong method effect in the data, it might not affect fit indices that much. You should also check how much the method factor explains the indicators. See ruclips.net/video/MVHSJLxtYRg/видео.html If you think that a source of method variance is present, you should include that in the main SEM model too. See the video linked above. 2) See the video linked above. 3) I think Spector et al 2019, cited in the video, is the best source at the moment. I appreciate the clear questions and the numbering, which made it easy to answer!
Thank you for making this video Mikko. Actually, I'm learning a lot from your academic uploads, have subscribed to it, and have already recommended it to my colleagues. I just have some questions sir regarding the CFA Marker Technique as follows which were not expounded in the article of Williams et al., 2010: 1. If method variance is detected through either Method C or Method U (whichever is better) and Method R is NOT significantly different from either Method C or Method U (whichever is better), how will the researcher reflect the CMB adjustments/corrections in the evaluation of the hypothesized model? 1a. Using either Method C or Method U (whichever is better), will the researcher transition from latent measurement model to composite scores to factor in the overall effect of the marker variable in the evaluation of the hypothesized model? 1b. Or will the researcher maintain the latent measurement model using either Method C or Method U (whichever is better) and build the structure of his/her hypothesized model from it? 1c. Or after detection of CMB, will the researcher reverts back to the baseline CFA model, report in the manuscript/paper that CMB was detected using CFA marker technique but did not skew the relationships between the substantive variables, and control the measured marker variable in the hypothesized model? 2. If in case Method R is different from either Method C or Method U (whichever is better), researcher is going to use Method R as final method for the CFA Marker Technique? In this regard, since in Method R, CFA marker skews the relationships between the substantive variables, how will the researcher going to proceed with the analysis given this information? Will it be treated like my questions in 1a, 1b, 1c, or a different approach? Thank you very much and hoping to get a clarification on this regard.
First, as I noted in the video, I think that the entire model that Williams proposes is a but illogical because if the marker indicators measure something together, they should load on a substantive factor (the marker construct) and the method factor, not just one factor that combines both these concepts. (For the record, I value Larry's work a great deal, but we all make mistakes.) As such, I would recommend more modern takes on markers such as the Spector et al paper that I cite in the video. Second, I would not myself ever constrain the factor loadings to be equal to the model with the baseline model. By doing so you would basically assume that there is no method variance present in the data. If that was indeed the case, the results from the Method-C and Method-U models would be very similar to the Method-R model, but the parameter estimate (and particularly their standard errors) would better reflect the uncertainty of the estimates given that you do not really know whether there is method variance or not in the data. I would instead use the most general model (with no fixed parameters except the first loadings for scale setting) and report that as my final model. That would be the most concervative approach. Finally, I would not recommend transiting to scale scores if you suspect method variance because the scales scores would be contaminated with method variance. See the video "Interpretation of method factor models" for a published example of how I implemented method factor in an empirical study. The paper was one of my early papers and there are some reporting issues, but I still think that the way we did it is the way to go. And read Spector's paper too.
Hello, For Lindell and Whitney's procedure, what would one do if the relationship between the marker variable and the substantive variable is negative. For example, if the lowest correlation with a marker variable is -.01, would one have to assess the partial correlation using? : r - (-.01) ------------ 1-(-.01)
I do not recommend the use of the Lindell and Whitney procedure at all. I think that I explain that in the video, but the lowest correlation is subject to sampling error and will underestimate the effect of method variance if such exists. If you have a valid marker, then the average correlation between the marker and other indicators can tell you something about method variance. But even if that is the case, the CFA based techniques are a lot better way to assess method variance.
Hi! Thank you for this video it was really helpful. I was wondering does the question relating to the marker variable appear at the beginning, end or middle of the survey? I have 7 question so would i place the marker variable questions in the middle of my survey or at the end? Thanks!
With such short survey, I do not think it makes a difference where you place the question. As a general rule of thumb, I always recommend randomisation of the question order unless there is a good reason not to do so. You should read about item priming effect to understand the effect of question ordering - it can be substantial.
This paper shows that this test doesn't work. Why do we need to do it then? Chin, W.W., Thatcher, J.B. and Wright, R.T. (2012), “Assessing common method bias: problems with the ULMC technique”, MIS Quarterly, Vol. 36 No. 3, pp. 1003-1019.
This is an interesting question and I could go on for hours answering it. But here is a quick four-part answer: 1) The video is about marker variable methods, not about unmeasured latent method factor (UMLC) model, so the paper that you cite is not directly relevant to the video. 2) Chin's conclusion does not follow from their simulation evidence. They show that UMLC is not useful when estimated with PLS. Contrary to what Chin claims, the problem is not UMLC but the fact that PLS is not a useful estimation technique for this kind of models. For a quick comment on this, see dx.doi.org/10.1016/j.jom.2016.05.002 p.22. 3) UMLC can work in specific scenarios, but it is often not identified and therefore often not useful. This is not really discussed in any published article, but you can see proofs of non identification on my method variance playlist. 4) Now to the main question, why do people use this kinds of models? In my work as a reviewer and editor, I have come to realize that most people who use methods do not really understand what they are based on (if anything) and simply follow what others have done in the past. I discuss this in my video on methodological myths and urban legends.
You can use SPSS AMOS to do the CFA-based models. The Lindell and Whitney "test" would be doable in SPSS, but I would not generally recommend that technique.
This is my first time leaving comments on RUclips, and this video has been that much valuable. The most valuable and helpful to me were the necessary references and the must-have critical viewpoints. I will need to read and study far more to come up with worthy questions for you, but until then, allow me to express my appreciation. Thank you, Mikko.
You are welcome. I am happy that you found the video useful.
Really helpful, Mikko! Thanks for sharing your analysis. Greetings from Portugal
Excellent! Very clear! Thanks very much!
You're welcome!
Thank you very much Mikko. Great help.
Hello Professor, thanks for this excellent video ! There were two questions that clicked was 1) Can we retain common latent factor and marker variable in our CFA and therefore SEM model if we get better model fit indices ?
2) How can we report/justify this ?
3) Also, can you please suggest any article that mentions about how to choose a marker variable?
1) The purpose of SEM is not to get better fit indices and model fit is not be an ideal metric for deciding whether a method factor is required. The reason for this is that even if there was a strong method effect in the data, it might not affect fit indices that much. You should also check how much the method factor explains the indicators. See ruclips.net/video/MVHSJLxtYRg/видео.html If you think that a source of method variance is present, you should include that in the main SEM model too. See the video linked above.
2) See the video linked above.
3) I think Spector et al 2019, cited in the video, is the best source at the moment.
I appreciate the clear questions and the numbering, which made it easy to answer!
Thank you for making this video Mikko. Actually, I'm learning a lot from your academic uploads, have subscribed to it, and have already recommended it to my colleagues. I just have some questions sir regarding the CFA Marker Technique as follows which were not expounded in the article of Williams et al., 2010:
1. If method variance is detected through either Method C or Method U (whichever is better) and Method R is NOT significantly different from either Method C or Method U (whichever is better), how will the researcher reflect the CMB adjustments/corrections in the evaluation of the hypothesized model?
1a. Using either Method C or Method U (whichever is better), will the researcher transition from latent measurement model to composite scores to factor in the overall effect of the marker variable in the evaluation of the hypothesized model?
1b. Or will the researcher maintain the latent measurement model using either Method C or Method U (whichever is better) and build the structure of his/her hypothesized model from it?
1c. Or after detection of CMB, will the researcher reverts back to the baseline CFA model, report in the manuscript/paper that CMB was detected using CFA marker technique but did not skew the relationships between the substantive variables, and control the measured marker variable in the hypothesized model?
2. If in case Method R is different from either Method C or Method U (whichever is better), researcher is going to use Method R as final method for the CFA Marker Technique? In this regard, since in Method R, CFA marker skews the relationships between the substantive variables, how will the researcher going to proceed with the analysis given this information? Will it be treated like my questions in 1a, 1b, 1c, or a different approach?
Thank you very much and hoping to get a clarification on this regard.
First, as I noted in the video, I think that the entire model that Williams proposes is a but illogical because if the marker indicators measure something together, they should load on a substantive factor (the marker construct) and the method factor, not just one factor that combines both these concepts. (For the record, I value Larry's work a great deal, but we all make mistakes.) As such, I would recommend more modern takes on markers such as the Spector et al paper that I cite in the video.
Second, I would not myself ever constrain the factor loadings to be equal to the model with the baseline model. By doing so you would basically assume that there is no method variance present in the data. If that was indeed the case, the results from the Method-C and Method-U models would be very similar to the Method-R model, but the parameter estimate (and particularly their standard errors) would better reflect the uncertainty of the estimates given that you do not really know whether there is method variance or not in the data. I would instead use the most general model (with no fixed parameters except the first loadings for scale setting) and report that as my final model. That would be the most concervative approach.
Finally, I would not recommend transiting to scale scores if you suspect method variance because the scales scores would be contaminated with method variance. See the video "Interpretation of method factor models" for a published example of how I implemented method factor in an empirical study. The paper was one of my early papers and there are some reporting issues, but I still think that the way we did it is the way to go. And read Spector's paper too.
Amazing video
Glad you think so!
Your video was really helpful. Thank you :)
Hello,
For Lindell and Whitney's procedure, what would one do if the relationship between the marker variable and the substantive variable is negative. For example, if the lowest correlation with a marker variable is -.01, would one have to assess the partial correlation using? :
r - (-.01)
------------
1-(-.01)
I do not recommend the use of the Lindell and Whitney procedure at all. I think that I explain that in the video, but the lowest correlation is subject to sampling error and will underestimate the effect of method variance if such exists. If you have a valid marker, then the average correlation between the marker and other indicators can tell you something about method variance. But even if that is the case, the CFA based techniques are a lot better way to assess method variance.
Great video!
Thanks
Hi! Thank you for this video it was really helpful. I was wondering does the question relating to the marker variable appear at the beginning, end or middle of the survey?
I have 7 question so would i place the marker variable questions in the middle of my survey or at the end?
Thanks!
With such short survey, I do not think it makes a difference where you place the question. As a general rule of thumb, I always recommend randomisation of the question order unless there is a good reason not to do so. You should read about item priming effect to understand the effect of question ordering - it can be substantial.
Really helpful, thanks a lot
Glad it was helpful!
Thank you, Mikko! When is a sample considered to be large?
What is large depends on the context and it is impossible to give a general answer.
This paper shows that this test doesn't work. Why do we need to do it then?
Chin, W.W., Thatcher, J.B. and Wright, R.T. (2012), “Assessing common method bias: problems with the ULMC technique”, MIS Quarterly, Vol. 36 No. 3, pp. 1003-1019.
This is an interesting question and I could go on for hours answering it. But here is a quick four-part answer:
1) The video is about marker variable methods, not about unmeasured latent method factor (UMLC) model, so the paper that you cite is not directly relevant to the video.
2) Chin's conclusion does not follow from their simulation evidence. They show that UMLC is not useful when estimated with PLS. Contrary to what Chin claims, the problem is not UMLC but the fact that PLS is not a useful estimation technique for this kind of models. For a quick comment on this, see dx.doi.org/10.1016/j.jom.2016.05.002 p.22.
3) UMLC can work in specific scenarios, but it is often not identified and therefore often not useful. This is not really discussed in any published article, but you can see proofs of non identification on my method variance playlist.
4) Now to the main question, why do people use this kinds of models? In my work as a reviewer and editor, I have come to realize that most people who use methods do not really understand what they are based on (if anything) and simply follow what others have done in the past. I discuss this in my video on methodological myths and urban legends.
@@mronkko Thank you very much for the explanation. It is very helpful.
sir can we do marker variable test in SPSS?
You can use SPSS AMOS to do the CFA-based models. The Lindell and Whitney "test" would be doable in SPSS, but I would not generally recommend that technique.
@@mronkko Thank you so much sir, it was really helpful