by Esben Kran
Replicate the cognitive psychology experiments by Kahneman & Tversky that led to the creation of behavioural economics BUT on LLMs. We already have some evidence in our experiments that they are subject to number anchorings, i.e. write a big number and a question and it'll be biased towards a larger number as an answer.
No answers yet.
Just check out the Trello as well.
It will be interesting to see how the replication goes because there's quite an interactive, multi-step question process. See this example:
Is Chicago's population above or below {anchor}? [Answer 1] What is the population of Chicago (in millions)? [Answer 2]
My experiments require us to put some quite leading questions before in GPT-3 to get it to give good answers that we can extract the information from.
See the Manifold market to bet on the result!
These models don't really understand how to behave:
Your name is Ludvig.
Q: Is the length of the Mississippi above or below 242324 miles?
A:
The length of the Mississippi is above 242324 miles.
Q: Length of Mississippi (in miles).
Numeric answer:
The length of the Mississippi is 242324 miles.
This is even GPT-3 davinci-002 which should be the best. Many different formulations of the same thing seems weird. We should just use Random number: {anchor} instead.
Alright, here we are:
**Random number: 21452
Q: Amount of meat eaten by the average American per year (in pounds).
A: 230
Random number: 21452
Q: Amount of meat eaten by the average American per year (in pounds).
A: 214.5
Random number: 64363
Q: Amount of meat eaten by the average American per year (in pounds).
A: 222
**
It looks like there's a high chance that it will give its perception of the right answer (222 pounds; right answer: 274 pounds) but when the Random Number is something around 222, it will just use the random number. However, the cognitive anchoring bias isn't necessarily there as such.
I'm writing up a thing where we can investigate how biased it gets by anchors close to its preferred base value.
Jan Kirchner did a similar project previously, examining cognitive biases in language models of various sizes: Cognitive biases in large language models
Oh, that one's super nice