Jonathan H. Rystrøm

How do language models handle Black Swans?

Most language models are trained on a large dataset (i.e. [the pile](https://pile.eleuther.ai/)). Because of the costs associated they are expensive to update. Figuring out how they handle an uncertain future (like War in Ukraine and other [Black Swans](https://en.wikipedia.org/wiki/The_Black_Swan:_The_Impact_of_the_Highly_Improbable)) could therefore inform how reliable they are. Concretely, the problem will look at how they "predict" an uncertain variable (like the prize of oil) _after_ their training period (versus after).