I have an MS in statistics and broadly agree with you. There are times when it does make sense to apply some kind of correction, but in most cases, it probably makes better sense to publish the results and simply talk about it as a limitation or area of further exploration in the discussion, like you said, especially if the p value is just under 0.05. I would argue we need to collectively move away from viewing the 0.05 as a black or white judgement on whether research in meaningful or not. Even an p value of 0.10 *might* indicate there is a relationship there that would show up more significantly with a larger sample size or if some other factor of the experiment was controlled for. Also, thank you for the references. I will be using them.
My Two Cents on this issue: if you have small sample size (n =< 100), I recommend NOT USING Bonferroni. Small sample size have less power or less Type I error, or less chance of proving something is statistically significant. Based on this video, Bonferroni is very strict and controlled on the Alpha or Type I error. We wouldn't worry much of the Type I error with small sample size. But if we have large sample sizes, then I highly suggest using the Bonferonni. Large sample size has also lots of power or higher chance of proving something is statistically significant. Having a Bonferroni adjustment will somehow regulate such power. Another important thing to consider is, Type II error is less evil than Type I error. Because Type II error supports the NORMAL while Type I proves the ABNORMAL. To disturb something that is normal should have an OVERWHELMING evidence than just enough evidence. In Research, when we prove something out of the normal (alternative hypothesis), we should actually very and meticulously sure that it is actually true. There are more damaging effects of falsely proving something out of the ordinary.
Nice, love it!!! To relax the apprehension pertaining family-wise error rate I believe that instead of looking at every single possible comparison, one needs to rather select only those that would be meaningful (that is, with regards to the outcome of interest). In this way the bonferroni wouldn't be so strict because every single comparison you make would be necessitated by sound reason and therefore justifiable. I am aware though, that reasons for selecting certain comparisons over others is a subjective matter. But, I believe this is where you're needed as a scientist, show us your justification for selecting those that you did, and show us what story you're selling. Let me know what you think about this.
While I agree with you, a small sample size has reduced power, but it does not have a reduced type 1 error rate. It instead has an increased type 2 error rate in order to keep the type 1 error rate at a constant value (usually of .05).
Video really helped me out, thanks a million. I think with exploratory studies there is real risk of underpowering and loss of potential rich data. A part of me suspects there is something very questionable about the fundamental premises Bonferroni adjustment logic is based on.
Great video! Thanks a lot! Wondering if it makes any sense to adjust for multiple comparisions when you are comparing means of a single variable between different groups e.g. pH values between 5 water treatments
My experience in the Medical Device industrial Process Validation is that there is very little place to apply the Bonferroni correction. However, for the bio-science research, where controlling variance is indeed a challenge for replications, it could be somewhat useful. So, in contrast with the shameful position of Harvard, MIT and Penn presidency the Bonferroni correction depends on the context.
The more I learn on FWE, the more errors I notice with a lot of the fundamental logic behind it and the less need I see for it in terms of the way it is normally corrected. Solutions that simply focus on expected significance based on p-value and number of tests performed are much more logically sound I find. Too many assumptions are made in FWER from the start as it assumes that every single test you perform will have a p-value of .05 and disregards the actual data collected. Besides, if you collect a mean that happens to be an extreme score, the number of tests you perform will not control for the errors you will find which is the core issue. Merely setting a more strict alpha is the best way to control for false positives if it is a large concern in a study. There’s also a huge theoretical issue that I often see not mentioned, which is that the original FWER calculation comes from probabilities of repeated sampling of the SAME exact population which is of course not true in any experiment. The FWER calculation is the exact same calculation used when asking the question, “what’s the percentage chance of me rolling a 1 on a 20 sided die over X rolls”. The idea that each comparison being made is the same as rolling the exact same die multiple times is problematic.
I have a problem of using the bonferroni adjustment.. I ran my data by Friedmans Test using spss software and the result is, it has significant difference.. And when I applied Post Hoc Test to determine the treatments with significant differences and using the boferroni adjustment it appeared that all the comparison of my data have no significant difference.. Is it okay not to use this adjustment in order to justify the significant difference result ran by the Friedman's Test?
Thank you for this very clear and didactic video. It has been very helpful. Myself I am used to post-hoc analysis in one way ANOVA, but a reviewer "forced me" to do the Bonferroni corrections because I used multiple chi-square tests to a small sample. I answered with a column of uncorrected p values and a column of corrected p values in my table. Do you think that's a good idea?
I'm not sure what you mean. There are no chi-square tests in the ANOVA context. I think what you mean is post-hoc testing the contingency table analysis case. I discuss that procedure in this video (and, yes, correcting one column is appropriate): ruclips.net/video/cOu9rv83G-I/видео.html
@@how2stats The problem is that the reviewer asked me to do Bonferroni corrections not in a ANOVA test but in a table where I was comparing nominal variables between two groups by means of chi-square tests. What do you think of that? Was he being reasonable by asking to do Bonferroni corrections in the context of chi-square tests? Post-hoc anaylsis in the context of ANOVA tests is a standard procedure, but what confused me was that he asked for Bonferroni corrections after I usted chi-square tests to compare nominal. dichotomic variables between two groups (in a small sample)
hi, my study is to look at pre and post treatment on control and experimental group. i did a 2x2 mixed anova so i had two main effects and an interaction effect. i ve been asked to do a further pre post t test for each group to interpret the interaction effect so i did a paired t test for each group. do i need to do a bonferroni correction? and if yes shall i divide the p value by 2?
If I decided to go on with Bonferroni correction. I am quantifying the expression levels of proteins in control (A) and mild (B), moderate (C), and severe (D) diseased groups (4 groups overall). I want to detect any significance between all pairwise comparisons AB, AC, AD, BC, BD, and CD. In that case, should I divide the α by 6 (All the pairwise comparisons) or 3 (Knowing that each group will have 3 independent comparisons only)?
One question please, I am using SmartPLS 3 to do Multi-group analysis. I have three groups and I have to do the permutation-based confidence intervals for the mean values and the variances. In this case, how do I report the measurement invariance of the composite models (MICOM) and the permutation results (with or without the Bonferroni adjustment)? Thank you in advance for all the help :)
hi. it is a very good video. I would like to ask you a question. I have carried out a paired sample t test with alpha .05, research design of 1 group pretest - posttest experiment. I am measuring 7 subscales of a construct. where i compare the mean before test and mean after test. i would like to ask. is bonferroni correction applicable to paired sample t test? as a lot of information stated that anova etc. but there isnt informaton related to my situation. hope to get your reply. thank you so much in advance
Hi, Thank you for the video. If you have time: I am wondering whether the Bonferroni Correction test can be applied when conducting multiple nonparametric tests on one sample, e.g. the Krushkal H Wallis test or Kolomogorov Smirnof Z test? I am unsure since SPSS automatically produces an adjusted significance result.
What about Bonferroni Correction in the framework of crosstabulations and difference in proportions? SPSS manual suggests using Bonferroni Correction. What is your opinion?
The Bonferroni correction is really your own option in the crosstabs case; or perhaps the sequential Bonferroni correction? You could possibly get away with the Fisher's protected LSD if it's a 2x3 crosstabs. You might want to check out this video: ruclips.net/video/cOu9rv83G-I/видео.html
Hello I am comparing gene expression between two groups diseased and controls. I have to compare gene expression of 7 genes between two groups. For each gene the CT values are different for each group, therefore it means I am not using same data for multiple comparisons.. do I still need to perform the correction test.. secondly the sample size is very small 8 controls and 16 diseased.
If I understand the nature of your data/design correctly, I'd say you need to consider some level of familywise error rate inflation correction, yes. Whether the Bonferroni correction is your best/only option, I don't know.
The ultimate reason is 'chance'. You simply need to accept that when you conduct statistical analyses on a sample of data, apparently statistically significant results can arise simply by chance (no systematic effect in the population).
Guys, what do you think about my data: I compared 14 morphological measures in lizards of two different environments with student t-test... Ex: head width of forest lizards x head width of dunes lizards... I have to use the Bonferroni correction?
I'm guessing your sample size isn't big enough to create body size factor scores. If so, you might have to rest on the argument that your 14 morphological measures are substantially positively inter-correlated, which would imply that a Bonferroni correction would be way too strict (as discussed in the video).
I have an MS in statistics and broadly agree with you. There are times when it does make sense to apply some kind of correction, but in most cases, it probably makes better sense to publish the results and simply talk about it as a limitation or area of further exploration in the discussion, like you said, especially if the p value is just under 0.05. I would argue we need to collectively move away from viewing the 0.05 as a black or white judgement on whether research in meaningful or not. Even an p value of 0.10 *might* indicate there is a relationship there that would show up more significantly with a larger sample size or if some other factor of the experiment was controlled for.
Also, thank you for the references. I will be using them.
My Two Cents on this issue: if you have small sample size (n =< 100), I recommend NOT USING Bonferroni. Small sample size have less power or less Type I error, or less chance of proving something is statistically significant. Based on this video, Bonferroni is very strict and controlled on the Alpha or Type I error. We wouldn't worry much of the Type I error with small sample size. But if we have large sample sizes, then I highly suggest using the Bonferonni. Large sample size has also lots of power or higher chance of proving something is statistically significant. Having a Bonferroni adjustment will somehow regulate such power.
Another important thing to consider is, Type II error is less evil than Type I error. Because Type II error supports the NORMAL while Type I proves the ABNORMAL. To disturb something that is normal should have an OVERWHELMING evidence than just enough evidence. In Research, when we prove something out of the normal (alternative hypothesis), we should actually very and meticulously sure that it is actually true. There are more damaging effects of falsely proving something out of the ordinary.
Nice, love it!!!
To relax the apprehension pertaining family-wise error rate I believe that instead of looking at every single possible comparison, one needs to rather select only those that would be meaningful (that is, with regards to the outcome of interest). In this way the bonferroni wouldn't be so strict because every single comparison you make would be necessitated by sound reason and therefore justifiable.
I am aware though, that reasons for selecting certain comparisons over others is a subjective matter. But, I believe this is where you're needed as a scientist, show us your justification for selecting those that you did, and show us what story you're selling.
Let me know what you think about this.
While I agree with you, a small sample size has reduced power, but it does not have a reduced type 1 error rate. It instead has an increased type 2 error rate in order to keep the type 1 error rate at a constant value (usually of .05).
I will be sending this link to my PhD committee member that keeps on pushing this analysis!
Video really helped me out, thanks a million. I think with exploratory studies there is real risk of underpowering and loss of potential rich data. A part of me suspects there is something very questionable about the fundamental premises Bonferroni adjustment logic is based on.
To support and appreciate how2stats, I suggest we at least don't skip the ads.
Great video! Thanks a lot! Wondering if it makes any sense to adjust for multiple comparisions when you are comparing means of a single variable between different groups e.g. pH values between 5 water treatments
My experience in the Medical Device industrial Process Validation is that there is very little place to apply the Bonferroni correction. However, for the bio-science research, where controlling variance is indeed a challenge for replications, it could be somewhat useful. So, in contrast with the shameful position of Harvard, MIT and Penn presidency the Bonferroni correction depends on the context.
Thank you so much for these videos!
The more I learn on FWE, the more errors I notice with a lot of the fundamental logic behind it and the less need I see for it in terms of the way it is normally corrected. Solutions that simply focus on expected significance based on p-value and number of tests performed are much more logically sound I find.
Too many assumptions are made in FWER from the start as it assumes that every single test you perform will have a p-value of .05 and disregards the actual data collected. Besides, if you collect a mean that happens to be an extreme score, the number of tests you perform will not control for the errors you will find which is the core issue. Merely setting a more strict alpha is the best way to control for false positives if it is a large concern in a study.
There’s also a huge theoretical issue that I often see not mentioned, which is that the original FWER calculation comes from probabilities of repeated sampling of the SAME exact population which is of course not true in any experiment. The FWER calculation is the exact same calculation used when asking the question, “what’s the percentage chance of me rolling a 1 on a 20 sided die over X rolls”. The idea that each comparison being made is the same as rolling the exact same die multiple times is problematic.
I have a problem of using the bonferroni adjustment.. I ran my data by Friedmans Test using spss software and the result is, it has significant difference.. And when I applied Post Hoc Test to determine the treatments with significant differences and using the boferroni adjustment it appeared that all the comparison of my data have no significant difference.. Is it okay not to use this adjustment in order to justify the significant difference result ran by the Friedman's Test?
Thank you for this very clear and didactic video. It has been very helpful. Myself I am used to post-hoc analysis in one way ANOVA, but a reviewer "forced me" to do the Bonferroni corrections because I used multiple chi-square tests to a small sample. I answered with a column of uncorrected p values and a column of corrected p values in my table. Do you think that's a good idea?
I'm not sure what you mean. There are no chi-square tests in the ANOVA context. I think what you mean is post-hoc testing the contingency table analysis case. I discuss that procedure in this video (and, yes, correcting one column is appropriate): ruclips.net/video/cOu9rv83G-I/видео.html
@@how2stats The problem is that the reviewer asked me to do Bonferroni corrections not in a ANOVA test but in a table where I was comparing nominal variables between two groups by means of chi-square tests. What do you think of that? Was he being reasonable by asking to do Bonferroni corrections in the context of chi-square tests? Post-hoc anaylsis in the context of ANOVA tests is a standard procedure, but what confused me was that he asked for Bonferroni corrections after I usted chi-square tests to compare nominal. dichotomic variables between two groups (in a small sample)
What about Benjamini Horchberg method?
hi, my study is to look at pre and post treatment on control and experimental group. i did a 2x2 mixed anova so i had two main effects and an interaction effect. i ve been asked to do a further pre post t test for each group to interpret the interaction effect so i did a paired t test for each group. do i need to do a bonferroni correction? and if yes shall i divide the p value by 2?
Excellent video👍🏽
Very informative :) Thank you
If I decided to go on with Bonferroni correction. I am quantifying the expression levels of proteins in control (A) and mild (B), moderate (C), and severe (D) diseased groups (4 groups overall).
I want to detect any significance between all pairwise comparisons AB, AC, AD, BC, BD, and CD. In that case, should I divide the α by 6 (All the pairwise comparisons) or 3 (Knowing that each group will have 3 independent comparisons only)?
Could you by any chance share what you found out? I know it's been two years haha. Thanks.
One question please, I am using SmartPLS 3 to do Multi-group analysis. I have three groups and I have to do the permutation-based confidence intervals for the mean values and the variances. In this case, how do I report the measurement invariance of the composite models (MICOM) and the permutation results (with or without the Bonferroni adjustment)? Thank you in advance for all the help :)
Hi, how to report in writing if we use Bonferroni correction in t-test?
How can I cite you in regards to the unnecessity of using Bonferroni correction? Thank you.
Check out: blog.apastyle.org/apastyle/2011/10/how-to-create-a-reference-for-a-youtube-video.html
(I'm Gignac, G. E.)
hi. it is a very good video. I would like to ask you a question. I have carried out a paired sample t test with alpha .05, research design of 1 group pretest - posttest experiment. I am measuring 7 subscales of a construct. where i compare the mean before test and mean after test. i would like to ask. is bonferroni correction applicable to paired sample t test? as a lot of information stated that anova etc. but there isnt informaton related to my situation. hope to get your reply. thank you so much in advance
I read that Bonferroni is too conservative. That we should consider False Discovery Rate instead. Thoughts?
Same question!
Hi,
Thank you for the video. If you have time: I am wondering whether the Bonferroni Correction test can be applied when conducting multiple nonparametric tests on one sample, e.g. the Krushkal H Wallis test or Kolomogorov Smirnof Z test? I am unsure since SPSS automatically produces an adjusted significance result.
Yes, the Bonferroni can be applied to non-parametric tests.
It is also automatically applied when you use 'Analyse > Nonparametric tests > Independent samples'.
What about Bonferroni Correction in the framework of crosstabulations and difference in proportions? SPSS manual suggests using Bonferroni Correction. What is your opinion?
The Bonferroni correction is really your own option in the crosstabs case; or perhaps the sequential Bonferroni correction? You could possibly get away with the Fisher's protected LSD if it's a 2x3 crosstabs. You might want to check out this video: ruclips.net/video/cOu9rv83G-I/видео.html
how2stats www.ibm.com/support/knowledgecenter/en/SSLVMB_23.0.0/spss/tutorials/xtab_demo_howto_02.html
Hello I am comparing gene expression between two groups diseased and controls. I have to compare gene expression of 7 genes between two groups. For each gene the CT values are different for each group, therefore it means I am not using same data for multiple comparisons.. do I still need to perform the correction test.. secondly the sample size is very small 8 controls and 16 diseased.
If I understand the nature of your data/design correctly, I'd say you need to consider some level of familywise error rate inflation correction, yes. Whether the Bonferroni correction is your best/only option, I don't know.
How exactly familywise error rate increases? What's the reason behind?
The ultimate reason is 'chance'. You simply need to accept that when you conduct statistical analyses on a sample of data, apparently statistically significant results can arise simply by chance (no systematic effect in the population).
@@how2stats Thank you very much!
Guys, what do you think about my data:
I compared 14 morphological measures in lizards of two different environments with student t-test... Ex: head width of forest lizards x head width of dunes lizards... I have to use the Bonferroni correction?
I'm guessing your sample size isn't big enough to create body size factor scores. If so, you might have to rest on the argument that your 14 morphological measures are substantially positively inter-correlated, which would imply that a Bonferroni correction would be way too strict (as discussed in the video).
@@how2stats Thank you very much
@@how2stats I have made 14 t-tests (one per measure)... It's ok?
💢💢💢
ok