Autoplay
Autocomplete
Previous Lesson
Complete and Continue
A/B Test like a Pro! (online course)
#0 - WELCOME
Welcome to the course (0:55)
About me (1:17)
What's in the course? How to get the most out of it. (4:15)
#1 - INTRODUCTION
What is A/B testing exactly? (3:01)
The correlation vs. causation issue (4:47)
Why do you need A/B testing? (a story and the real reason) (4:23)
What can you A/B test? (8:52)
EXTRA: Hand-picked A/B Testing case studies
ACTION: The A/B tests you've run so far -- and the A/B tests you want to run in the future
#2 - IMPORTANT INITIAL BUSINESS QUESTIONS
Important A/B Testing mindset: Conversion Rate Optimization or Research? (3:42)
Two key questions before starting an A/B Test. (And when is A/B testing a really bad idea?) (6:45)
When you don't need to run experiments (6:32)
When you need to run experiments (5:30)
A/B test only when you need it (2:09)
EXTRA: the "can I run this A/B test" checklist
ACTION: Filter out the bad A/B test ideas from your backlog!
#3 - STATISTICAL BACKGROUND
Statistics is important (1:30)
Randomization (we all are terrible "random-generators") (3:01)
Statistical significance and the certainty of your results (4:37)
An intuitive way of interpreting the importance of statistical significance (2:02)
80%? 95%? 99%? What's the right confidence level? (6:45)
Calculating the required sample size before the test (4:57)
EXTRA: online calculators (statistical significance and sample size)
ACTION: Review your A/B testing backlog!
VAULT: Q&A
Your questions answered.
#4 - EXECUTION - STEP 1: PREPARING YOUR A/B TEST(S)
The four steps of executing an A/B test (1:39)
The Research Framework (14:14)
Your success metric(s) to evaluate your test (7:36)
A few words about setting a hypothesis (5:05)
EXTRA: A/B Testing Hypothesis Form
ACTION: Run your first research round!
#4 - EXECUTION - STEP 2: IMPLEMENTATION
My favorite A/B testing tools (6:06)
Tool demo (Google Optimize) (16:01)
Avoiding website flickering in an A/B test (4:38)
The change-one-thing-at-a-time myth, multivariate tests, A/B/n tests and unusual audience splits (7:04)
A few typical mistakes at the implementation phase (3:13)
EXTRA: The "can I start the A/B test?" checklist
ACTION: Try out the simplest ever A/B test!
#4 - EXECUTION - STEP 3: RUNNING A TEST
How long should an A/B test run? (2:42)
Which part of the year/week shouldn't you run an A/B test? (1:53)
Typical mistakes while running an A/B test (5:57)
#4 - EXECUTION - STEP 4: EVALUATING YOUR EXPERIMENTS
Don't lie to yourself! (4:36)
What to do when a test didn't perform the way you expected (2:28)
What to do after evaluating your experiment (6:29)
ACTION: Start your own knowledge base!
#5 - A FEW SMALLER ADDITIONS
The limitations of A/B testing (2:26)
The art and science of A/B testing (3:15)
Segmentation (2:37)
EXTRA: Further reads and videos
#6 - SUMMARY
Summary & The right mindset to be successful with A/B testing (2:29)
Where to go next
VAULT: Q&A
Your questions answered
VAULT #2: Running a redirect A/B test without 3rd party tools -- by writing your own code.
Why?
1) Traffic allocation: sending visitors to the old/new landing pages (with JavaScript)
2) Data collection and visualization (Bash + SQL + Google Data Studio)
3) Evaluation: significance calculations (using Python)
Conclusion
Important A/B Testing mindset: Conversion Rate Optimization or Research?
references:
Fostering a Culture of Experimentation
at Skyscanner
Complete and Continue
Lesson Content Locked
Your session expired. Please sign in again to continue.
Login