Chapter 4

Is a more detailed rubric really a fairer one?

Thang chấm càng chi tiết thì càng công bằng, thật không?

The chapter's question

What criteria do I use to grade the machine's answers?

Big idea

A clear grading scale matters more than a sophisticated one.

In this chapter

  1. Part 4.1

    A Three-Level Scale

    Are pass, not yet and plain wrong enough for my task?

  2. Part 4.2

    When Graders Disagree

    Do two graders give the same score to the same answer?

  3. Part 4.3

    Letting a Model Grade

    When can another model do the grading, and when not?

  4. Part 4.4

    The Summary Number

    Does the average score hide seriously wrong answers?

The reading for this chapter is inside Conan School

This chapter has 4 readings, with case stories and exercises. Chapter 1 of An Exam Paper for the Machine is free to read right here.

Read in Conan School

Chapters of An Exam Paper for the Machine

  1. 1.Did yesterday's fix help, or just look better?
  2. 2.Do questions we invent test what users actually ask?
  3. 3.Do we really know what answer we want?
  4. 4.Is a more detailed rubric really a fairer one?
  5. 5.Which product is last year's test set still grading?