Succinct, clear, lucid, apt explanation demonstrated with empirical results from Standard economic journals. The Matching basics is truly fantastic. Slide on Matching the basics : The key to finding a good estimate for the counterfactual is to try to find someone who is nearly identical to the treated person but just did not get the treatment ... underscores the importance of getting the right counterfactual in impact evaluation. Propensity score introduced is also desirable.
Thanks a lot for your nice explanations. My question is, can we compare between 2 treatment groups, one group subjected to specific treatment and the second group subjected to another treatment instead if placebo or control as it is unethical to leave second group without treatment and we want to compare between the impacts of both treatments to know which one is better. Please need urgent answer. Thanks in advance
You sure could compare between treatment 1 and treatment 2 (no control) in theory. The design will still have internal validity. However, the problem with this is that it will likely lack external validity. This means that you will be picking up the differences between treatment1 and treatment 2....you won't be picking up the effect of treatment 1 all by itself. Usually, in these sorts of situations, where it is unethical to not offer treatment, what is done is a staggered implementation approach. Apply the treatment in 2-3 waves, perhaps 1 year apart (depending on what the treatment is), and then use the later waves as control groups for the earlier waves. Make sure that subjects are assigned randomly to the waves, however.
@Justin Jarvis thanks a lot for your valuable answer. Regarding the external validity, I designed a pre- & post test for each treatment , starting from an equal point as both groups comparable at baseline and then allocating subjects to 2 different treatments and then bivaruate statistics taking in my consideration evaluation for pre- test and post-test for both groups and for each and comparing the difference (pre value - post value) between both groups to figurout the improvement difference. Is a pre-post- test for each group not enough to evaluate the effect of each treatment ? As the durability of study is time (4 months apart), limited
@@fadhilfp6 Hello again. I might be misunderstanding your question. Since you were discussing the allocation of individuals to certain groups, I had assumed you were talking about an RCT-like design. Are you talking instead about a DinD design? The Differences in Differences technique is usually used when we CANNOT assign individuals to groups; that is, for some reason the data we have is generated by individuals who voluntarily signed up for the program. If we are able to assign, the RCT design will be cleaner, and as I said before, the later waves can be used as the control for the earlier waves. However, if you are thinking about a DinD design, with individuals assigned to two different treatment groups, this can be done (since you have pre and post data for each group). However, the alpha (coefficient of interest) you will calculate will not be the effectiveness of a treatment, rather it will be the (marginal) casual difference of treatment 1 as compared to treatment 2. This might yield useful data, depending on the context, but it will not yield information as to the effectiveness of treatment 1 (or treatment 2 for that matter) over the counterfactual. This is because there is no counterfactual generated.
Dr. Jarvis, I wish I had professors like you in my Uni. Thanks for the explanation! PS - Do you know of any papers I could read that use a combination of DD with Propensity Score Matching?
Sorry for the late reply. Off the top of my head, no. However, it's fairly common in the development literature to use DiD and then perform a "robustness check" with another form of impact evaluation, such as Propensity Score Matching.
Hello Justin! I am currently writing my master thesis on the influence of ESG on stock returns during the pandemic crisis. I am using a dif in dif for studying the causality between ESG score (treatment) and the current pandemic (time dummy). Is ESG as a dummy (one if the company qualifies in the top quartile, ESG score is last measured in 2018) qualified? I am worrying for self-selection bias. However, there are already academic studies published that use the same methodology. Thank you in advance!
Hello. I think it's OK. It's a good question and I'm not an expert in this field. But... let me tell you what I think. If you set up your model so that the impact estimate of interest is (return after pandemic-return before pandemic)[for the ESG=1]-(return after pandemic-return before pandemic)[for the ESG=0], then any selection bias will appear in both of these terms: return after pandemic[ESG=1] and also return before pandemic[ESG=1]. Since these terms are subtracted from each other, the selection bias should be cleaned off. This will enable you to talk about the heterogenous impact of the pandemic on the two different types of companies. I'm not sure that's exactly what you want to answer though...because in this model I've written the pandemic is the treatment.
@@justinjarvis5681 Thank you for your insight! I am trying to answer whether high ESG score companies have higher returns during the pandemic. The fact that I am determining the two groups (control and treatment) based on characteristics, I am worrying about self-selection bias. The companies decide either to spend more on sustainability or not.
Hi Dr Jarvis, I my proposal Topic is to measure the impact of internet access on student academic performance (using standardized math test scores) during the Covid pandemic in Rural China. Was wondering if I could use the DID model in measuring the impact
Hi Andam, sorry for the late reply. Do you still need help? I think the DiD model could be used, but somehow you'd want to find groups that were differentially exposed to internet access. Maybe the government started an internet program for a certain area and didn't for another area?
Wow, this marvelous and very rare content to find. It is complete, indepth, leaves no stone unturned
Thanks! I'm glad you like it.
This is amazing! Really can't believe I could relearn so well Impact Evaluation! Thank you very much!
Succinct, clear, lucid, apt explanation demonstrated with empirical results from Standard economic journals. The Matching basics is truly fantastic. Slide on Matching the basics : The key to finding a good estimate for the counterfactual is to try to find someone who is nearly identical to the treated person but just did not get the treatment ... underscores the importance of getting the right counterfactual in impact evaluation. Propensity score introduced is also desirable.
This video is so helpful that i liked it first, then took away my like, and then again liked it again - double liked it!
Thank you for such a detailed explanation from Gertler :)
Thank you for this helpful video!
Thanks a lot for your nice explanations. My question is, can we compare between 2 treatment groups, one group subjected to specific treatment and the second group subjected to another treatment instead if placebo or control as it is unethical to leave second group without treatment and we want to compare between the impacts of both treatments to know which one is better. Please need urgent answer. Thanks in advance
You sure could compare between treatment 1 and treatment 2 (no control) in theory. The design will still have internal validity. However, the problem with this is that it will likely lack external validity. This means that you will be picking up the differences between treatment1 and treatment 2....you won't be picking up the effect of treatment 1 all by itself. Usually, in these sorts of situations, where it is unethical to not offer treatment, what is done is a staggered implementation approach. Apply the treatment in 2-3 waves, perhaps 1 year apart (depending on what the treatment is), and then use the later waves as control groups for the earlier waves. Make sure that subjects are assigned randomly to the waves, however.
@Justin Jarvis thanks a lot for your valuable answer. Regarding the external validity, I designed a pre- & post test for each treatment , starting from an equal point as both groups comparable at baseline and then allocating subjects to 2 different treatments and then bivaruate statistics taking in my consideration evaluation for pre- test and post-test for both groups and for each and comparing the difference (pre value - post value) between both groups to figurout the improvement difference. Is a pre-post- test for each group not enough to evaluate the effect of each treatment ? As the durability of study is time (4 months apart), limited
@@fadhilfp6 Hello again. I might be misunderstanding your question. Since you were discussing the allocation of individuals to certain groups, I had assumed you were talking about an RCT-like design. Are you talking instead about a DinD design? The Differences in Differences technique is usually used when we CANNOT assign individuals to groups; that is, for some reason the data we have is generated by individuals who voluntarily signed up for the program. If we are able to assign, the RCT design will be cleaner, and as I said before, the later waves can be used as the control for the earlier waves.
However, if you are thinking about a DinD design, with individuals assigned to two different treatment groups, this can be done (since you have pre and post data for each group). However, the alpha (coefficient of interest) you will calculate will not be the effectiveness of a treatment, rather it will be the (marginal) casual difference of treatment 1 as compared to treatment 2. This might yield useful data, depending on the context, but it will not yield information as to the effectiveness of treatment 1 (or treatment 2 for that matter) over the counterfactual. This is because there is no counterfactual generated.
@Justin Jarvis
I am so thankful for your answer, Sir. I appreciated it.
Dr. Jarvis, I wish I had professors like you in my Uni. Thanks for the explanation!
PS - Do you know of any papers I could read that use a combination of DD with Propensity Score Matching?
Sorry for the late reply. Off the top of my head, no. However, it's fairly common in the development literature to use DiD and then perform a "robustness check" with another form of impact evaluation, such as Propensity Score Matching.
Maravilha!
Hello Justin! I am currently writing my master thesis on the influence of ESG on stock returns during the pandemic crisis. I am using a dif in dif for studying the causality between ESG score (treatment) and the current pandemic (time dummy). Is ESG as a dummy (one if the company qualifies in the top quartile, ESG score is last measured in 2018) qualified? I am worrying for self-selection bias. However, there are already academic studies published that use the same methodology. Thank you in advance!
Hello. I think it's OK. It's a good question and I'm not an expert in this field. But... let me tell you what I think. If you set up your model so that the impact estimate of interest is (return after pandemic-return before pandemic)[for the ESG=1]-(return after pandemic-return before pandemic)[for the ESG=0], then any selection bias will appear in both of these terms: return after pandemic[ESG=1] and also return before pandemic[ESG=1]. Since these terms are subtracted from each other, the selection bias should be cleaned off. This will enable you to talk about the heterogenous impact of the pandemic on the two different types of companies. I'm not sure that's exactly what you want to answer though...because in this model I've written the pandemic is the treatment.
@@justinjarvis5681 Thank you for your insight! I am trying to answer whether high ESG score companies have higher returns during the pandemic. The fact that I am determining the two groups (control and treatment) based on characteristics, I am worrying about self-selection bias. The companies decide either to spend more on sustainability or not.
Hi Dr Jarvis, I my proposal Topic is to measure the impact of internet access on student academic performance (using standardized math test scores) during the Covid pandemic in Rural China. Was wondering if I could use the DID model in measuring the impact
Hi Andam, sorry for the late reply. Do you still need help? I think the DiD model could be used, but somehow you'd want to find groups that were differentially exposed to internet access. Maybe the government started an internet program for a certain area and didn't for another area?
Hi codes and data please.
Hi! This doesn't make sense so me, so I won't be able to do it! Sorry about that.
@@justinjarvis5681 not a problem, thanks for the awesome content and your efforts.