Chapter 2

Can we still experiment with so few users?

Ít người dùng thì có còn thí nghiệm được không?

The chapter's question

With my current users, what size of difference can I detect?

Big idea

User volume decides what kind of experiment is possible, not the other way round.

In this chapter

  1. Part 2.1

    Rough Sample Sizes

    How many people per arm are needed to see a five-point difference?

  2. Part 2.2

    Bigger Changes

    Instead of changing a button colour, what change is big enough to see?

  3. Part 2.3

    Closer Metrics

    Which metric happens more often so the experiment can finish sooner?

  4. Part 2.4

    When Traffic Is Too Low

    If there are never enough people, how else do I learn?

The reading for this chapter is inside Conan School

This chapter has 4 readings, with case stories and exercises. Chapter 1 of Experiments With Few Users is free to read right here.

Read in Conan School

Chapters of Experiments With Few Users

  1. 1.If any result fits, why run the experiment?
  2. 2.Can we still experiment with so few users?
  3. 3.Is A/B testing the cheapest way to learn, or priciest?
  4. 4.Is an early win skill, or just luck?
  5. 5.Is one win worth more than a year of notes?