Google’s Latest Guidelines To Build Better NLU Benchmarks

Recently, researchers from Google Brain and New York University laid out four criteria to fix the issues plaguing natural language understanding  (NLU) benchmarks. For a few years now, research on natural language understanding has focused on improving benchmark datasets, which feature roughly independent and identically distributed (IID) training, testing sections, validation, and drawn from data collected or annotated by crowdsourcing. The researchers stated: “Progress suffers in the absence of a trustworthy metric for benchmark-driven work: Newcomers and non-specialists are discouraged from trying to contribute, and specialists are given significant freedom to cherry-pick ad-hoc evaluation settings that mask a lack of progress.” According to the researchers, unreliable