Hope you all enjoy the video. Just note that the Bayesian approach isn't necessarily better than the frequentist in every way - it is another perspective of looking at data. In fact, @Cassie Kozyrkov (head of decision intelligence at Google) has an amazing video comparing the 2 approaches with a simple and fun example. Thought I'd give it a shout out since its super cool
Great video. I just wanted to add a few points : - Frequentist approach also allows you to monitor the evolution of p-value as week goes by - Frequentist approach (with 1 tailed t-test for example) also allowed for checking hypothesis like "Is the uplift at least 2%?" - Small correction on your explanation of Bayesian approach : You don't get the proba of having a lift of -0.14% or something. You get the probability of having a negative lift, and then if we have to estimate the lift, then it would be -0.14%. but that's independent
A/B testing for conversion rate should use Z test instead of chi-square test. It's never about independence of two random variables. It's about the difference of the two groups is by chance or by design.
Thank you for clearly demonstrating this technique. It will be useful in a lot of experiments because you can peek while still collecting outcomes and it won't mess up the decision making, making it more practical to use than the classical p-value based decision making.
Thank you this is very clear in terms of how you executed the Bayesian approach with priors and updating your beliefs. However I feel like your interpretation of the result is pretty subjective - it's true that you get more confident when you obtain more weeks of data (30% -> 19% ->15% probability treatment is better than control), so likely we will not roll out with new page based on conversion alone, but how are you "super confident" at 85% likelihood of control winning? With frequentist approach, often times we need p value
Thank you for awesome videos. I have a question: could you please explain line 255. I'm not sure why did you use means of prior but not directly prior? It is similar to bootstrapping sampling but you already have enough sample size, then what is the need of bootstrapping? I am new to this field, so sometimes concepts become really confusing :(
Thanks four your great explanation! Just want to check whether I understood one aspect correctly: in a chi-square test, dont we need to state null hypothesis like: "conversion and group are independent" instead of "control and treatment are independent"? The second option is very confusing...
More concretely to this problem, the chi squared test will determine if the purchase conversion should be independent of whether the user was shown the old page or new page. In other words H0 = "There is no difference in purchase conversion".
I'm not sure if 'probability that we are seeing a 2% lift' measure you computed toward the end is actually a probability. Because what's to keep it bounded between 0 and 1.
what happens if at the beginning, I don't want to eliminate the 1.3% of users that have seen both and want to make sure that doest happens again, like, trying to separate them??
Hope you all enjoy the video. Just note that the Bayesian approach isn't necessarily better than the frequentist in every way - it is another perspective of looking at data. In fact, @Cassie Kozyrkov (head of decision intelligence at Google) has an amazing video comparing the 2 approaches with a simple and fun example. Thought I'd give it a shout out since its super cool
Do you know which one of her videos on RUclips?
Great video. I just wanted to add a few points :
- Frequentist approach also allows you to monitor the evolution of p-value as week goes by
- Frequentist approach (with 1 tailed t-test for example) also allowed for checking hypothesis like "Is the uplift at least 2%?"
- Small correction on your explanation of Bayesian approach : You don't get the proba of having a lift of -0.14% or something. You get the probability of having a negative lift, and then if we have to estimate the lift, then it would be -0.14%. but that's independent
Very interesting. Thanks for the input. Learning new things everyday
A/B testing for conversion rate should use Z test instead of chi-square test. It's never about independence of two random variables. It's about the difference of the two groups is by chance or by design.
yea, that's the part where I was a bit confused. I thought it should be testing whether or not there is an improvement (the difference is > 0)
I think it should be Chi-squared Goodness of fit test or Z
Thank you for clearly demonstrating this technique. It will be useful in a lot of experiments because you can peek while still collecting outcomes and it won't mess up the decision making, making it more practical to use than the classical p-value based decision making.
your explanations and your voice are crystal clear. Thank you:)
Glad!:)
Thank you this is very clear in terms of how you executed the Bayesian approach with priors and updating your beliefs. However I feel like your interpretation of the result is pretty subjective - it's true that you get more confident when you obtain more weeks of data (30% -> 19% ->15% probability treatment is better than control), so likely we will not roll out with new page based on conversion alone, but how are you "super confident" at 85% likelihood of control winning? With frequentist approach, often times we need p value
To sum it up in business terms, you can use this approach to potentially save on testing costs. That is the benefit of sampling.
Wow! You are simply the best, thank you so much for this video!
This blew my mind. Thank you
Thank you so much for simplifying this! Really enjoyed your tutorial.
Amazing content please upload more videos.💛
Thank you for awesome videos. I have a question: could you please explain line 255. I'm not sure why did you use means of prior but not directly prior? It is similar to bootstrapping sampling but you already have enough sample size, then what is the need of bootstrapping? I am new to this field, so sometimes concepts become really confusing :(
You make the things I look for.
Good job BTW 👍 thanks!
Can you do more bayesian approaches for future videos, assuming there are others also interested.
Thank you so much for this. Super helpful!!!
This is great. Thank you!
How do you know which prior distribution to choose? And how do you choose the parameters of the chosen prior distribution?
Very nice explanation
Thank you!
Thanks four your great explanation! Just want to check whether I understood one aspect correctly: in a chi-square test, dont we need to state null hypothesis like: "conversion and group are independent" instead of "control and treatment are independent"? The second option is very confusing...
More concretely to this problem, the chi squared test will determine if the purchase conversion should be independent of whether the user was shown the old page or new page. In other words H0 = "There is no difference in purchase conversion".
great video! learned a lot, thank you.
Very useful, thanks a lot!
I'm not sure if 'probability that we are seeing a 2% lift' measure you computed toward the end is actually a probability. Because what's to keep it bounded between 0 and 1.
For those with treatment under group, would they not have new_page under landing_page? (Sorry, I saw your explanation later on in the video. Thanks.)
what happens if at the beginning, I don't want to eliminate the 1.3% of users that have seen both and want to make sure that doest happens again, like, trying to separate them??
Thank you man :3
Simply wow
Thank you :)
How does one learn all these panda tricks? I keep forgetting them if I don't end up using them.
is there academic protocol to do Bayesian Testing ?
a good one. Thanks
Most welcome 🙂
It's a lot to take in, il have to visit again to understand it better
When I try Bayesian Testing, Is it data distribution must be normal distribution?
What does "Lift" mean in the code?
ur a beast
Cool
For a second I thought you were heavenlycontroller 😂