I disagree that you should use the observed effect size, I think you should use the MCID or the effect size that was selected during the a priori power calculations
Okay. I am very new to this and still a rather confused on what is being discussed here--I get the general theme, but the specifics are eluding me. For example, at min. 17:00 (approx.), where you are unpacking the screen tests. I do not understand the prevalence number: how can our screening tell us that 100 people have a condition when whatever testing we're doing, the sensitivity factor discussed, can tell us that only 80 people have the condition??? To me, it seems our prevalence is 8/10 of 1% (0.008) rather than 1% How can we possibly know the prior prevalence when we can only empirically confirm a lesser number? Apologies for the glaring ignorance here for those more seasoned with this, but I'm not getting this. I'll keep trying.
Thanks for your question. The estimate of prevalence will, like everything else, be subject to some uncertainty. It will, I guess, usually be based on the number of confirmed cases. For example, you know (eventually) how many people have prostate cancer. That being said, an error of 20% in the value used for prevalence is not going to make a huge difference to the calculations in the sense that the actions that you take are not likely to change much if there are modest errors in estimating the prevalence.
Thank you very much for your very helpful contributions to EBM, I have very much enjoyed your papers. So the FPR is essentially 1-PPV ? How does this relate , conceptually, to positive likelihood value? ( which I understand as the risk a positive test is a true positive divided by the risk it is a false positive)
Good question. The term PPV is usually used for diagnostic screening and in that case you are right. For screening the p-less-than approach is sensible and that''s how PPV is usually calculated. But for tests of significance the p-equals approach is what makes sense -the distinction between these two approaches is discussed at 33:00 in the video (or in section 3 of my 2017 paper royalsocietypublishing.org/doi/full/10.1098/rsos.171085#d3e628 ). The web calculator gives the results of both approaches: fpr-calc.ucl.ac.uk/ and it's clear that the p-equals approach gives bigger FPR for any given p value.
@@DavidColquhoun1 Thank you so much for clarifying. I see now, it is important to differentiate between diagnostic testing and significance testing ! Thanks again, best wishes,
I made a ratio version of FPR calculater. www.tutorialspoint.com/execute_r_online.php?NGsTgJ It needs package 'pwr' to execute, but graph is shown without it. I am afraid there are many bug in it, due to my poor understanding. Any improvement or debugging is welcome.
the supplementary of the article www.tandfonline.com/doi/full/10.1080/00031305.2018.1529622 at www.tandfonline.com/doi/suppl/10.1080/00031305.2018.1529622/suppl_file/utas_a_1529622_sm1508.zip seems to have some code to get started :) There is also a shiny app here fpr-calc.ucl.ac.uk/ !
You should make it explicit that you’re defining the false positive risk as the posterior probability of H0! Also great lecture
An amazing introductory slide; thank you for your work, Professor!
I disagree that you should use the observed effect size, I think you should use the MCID or the effect size that was selected during the a priori power calculations
So lowering a given fields p-value threshold to 0.005 in terms of publication standards would achieve a FPR of
Okay. I am very new to this and still a rather confused on what is being discussed here--I get the general theme, but the specifics are eluding me. For example, at min. 17:00 (approx.), where you are unpacking the screen tests. I do not understand the prevalence number: how can our screening tell us that 100 people have a condition when whatever testing we're doing, the sensitivity factor discussed, can tell us that only 80 people have the condition??? To me, it seems our prevalence is 8/10 of 1% (0.008) rather than 1% How can we possibly know the prior prevalence when we can only empirically confirm a lesser number? Apologies for the glaring ignorance here for those more seasoned with this, but I'm not getting this. I'll keep trying.
Thanks for your question. The estimate of prevalence will, like everything else, be subject to some uncertainty. It will, I guess, usually be based on the number of confirmed cases. For example, you know (eventually) how many people have prostate cancer. That being said, an error of 20% in the value used for prevalence is not going to make a huge difference to the calculations in the sense that the actions that you take are not likely to change much if there are modest errors in estimating the prevalence.
You can find collected links to all the stuff I've written about his at www.onemol.org.uk/?page_id=456
Thank you very much for your very helpful contributions to EBM, I have very much enjoyed your papers. So the FPR is essentially 1-PPV ? How does this relate , conceptually, to positive likelihood value? ( which I understand as the risk a positive test is a true positive divided by the risk it is a false positive)
Good question. The term PPV is usually used for diagnostic screening and in that case you are right. For screening the p-less-than approach is sensible and that''s how PPV is usually calculated. But for tests of significance the p-equals approach is what makes sense -the distinction between these two approaches is discussed at 33:00 in the video (or in section 3 of my 2017 paper royalsocietypublishing.org/doi/full/10.1098/rsos.171085#d3e628 ).
The web calculator gives the results of both approaches: fpr-calc.ucl.ac.uk/ and it's clear that the p-equals approach gives bigger FPR for any given p value.
@@DavidColquhoun1 Thank you so much for clarifying. I see now, it is important to differentiate between diagnostic testing and significance testing ! Thanks again, best wishes,
I made a ratio version of FPR calculater.
www.tutorialspoint.com/execute_r_online.php?NGsTgJ
It needs package 'pwr' to execute,
but graph is shown without it.
I am afraid there are many bug in it, due to my poor understanding.
Any improvement or debugging is welcome.
the supplementary of the article www.tandfonline.com/doi/full/10.1080/00031305.2018.1529622 at www.tandfonline.com/doi/suppl/10.1080/00031305.2018.1529622/suppl_file/utas_a_1529622_sm1508.zip seems to have some code to get started :) There is also a shiny app here fpr-calc.ucl.ac.uk/ !
25:01