Stanford Seminar: Peeking at A/B Tests - Why It Matters and What to Do About It

Ramesh Johari
Stanford University
I'll describe a novel statistical methodology that has been deployed by the commercial A/B testing platform Optimizely to communicate experimental results to their customers. Our methodology addresses the issue that traditional p-values and confidence intervals give unreliable inference. This is because users of A/B testing software are known to continuously monitor these measures as the experiment is running. We provide always valid p-values and confidence intervals that are provably robust to this effect. Not only does this make it safe for a user to continuously monitor, but it empowers her to detect true effects more efficiently. I'll cover why continuous monitoring is a problem, and how our solution addresses it.
The talk will be presented at a level that does not presume much statistical background.
Ramesh Johari is an Associate Professor at Stanford University, with a full-time appointment in the Department of Management Science and Engineering (MS&E), and courtesy appointments in the Departments of Computer Science (CS) and Electrical Engineering (EE). He is a member of the Operations Research group and the Social Algorithms Lab (SOAL) in MS&E, the Information Systems Laboratory (ISL) in EE, and the Institute for Computational and Mathematical Engineering (ICME).
Learn more about Stanford's Human-Computer Interaction Group: hci.stanford.edu
Learn about Stanford's Graduate Certificate in HCI: online.stanfor...
View the full playlist: • Stanford CS547 - Human...