Lessons from learning theory for benchmark design
This discussion/tutorial consists of a few short discussions on the (theoretical) trade-offs of various choices in constructing benchmarks. Most of the results discussed here are "common sense" at high level. However, all of this "common sense" is (to some extent) quantifiable common sense, and occasionally that quantification is useful. These short discussions cover: 1) Prediction Domains & Loss functions, 2) Prediction settings, 3) Assumption Failures.