Revisiting ‘Is AI Progress Impossible To Predict?’

Alyssa Vance argued that AI progress on a task from one model to the next was unpredictable (Vance, 2022). Can we investigate this in more detail? For instance, the authors of Beyond the Imitation Game (Big Bench) find that for tasks where progress is “jumpy”, there are usually progress metrics that vary more smoothly (Srivastava, 2022). Can we use those metrics to predict progress?