Experiments With Few Users
Who this book is for
Người làm sản phẩm ở doanh nghiệp Việt Nam: product manager, product owner, trưởng nhóm sản phẩm ở công ty phần mềm, ứng dụng di động, sàn thương mại điện tử, fintech và bộ phận số của doanh nghiệp truyền thống; làm việc hằng ngày với kỹ sư, thiết kế, đội bán hàng và chăm sóc khách, thường không có đội nghiên cứu người dùng hay phân tích dữ liệu riêng.
Sound familiar?
- Runs an A/B test for two weeks and concludes from a difference of a handful of users.
- Doesn't know how many users are needed before a result can be trusted.
- Stops tests early as soon as the number looks good.
- Every change is called an experiment but none has a hypothesis written down first.
After reading, you can
Help product people design small experiments that conclude with their real traffic.
Each chapter answers one question
Chapter 1: If any result fits, why run the experiment?
What do I believe, and which result would make me change my mind?
An experiment without a hypothesis written in advance can explain away any result.
Chapter 2: Can we still experiment with so few users?
With my current users, what size of difference can I detect?
User volume decides what kind of experiment is possible, not the other way round.
Chapter 3: Is A/B testing the cheapest way to learn, or priciest?
How can I test this hypothesis without splitting users into two groups?
There are many ways to learn more cheaply than an A/B test.
Chapter 4: Is an early win skill, or just luck?
Would this result still hold if I ran it another week?
A great result from stopping early is usually luck.
Chapter 5: Is one win worth more than a year of notes?
What does my team remember from last year's experiments?
The value of experiments lies in a log kept over months, not in one win.