A language model is not like a single person. As most humans grow up, they forge an identity defined by a roughly coherent set of beliefs, values and opinions. This might include political or religious affiliations, their racial or gender identity, and whether they like spaghetti bolognese, grime music or knitting. But LLMs do not have a single, coherent identity, and do not define themselves with a single set of beliefs or opinions. Even after fine-tuning, when their linguistic expressions may collapse on a single liberal mode, a whole universe of other human opinions are still bubbling away under the surface, and can be extracted with carefully crafted prompts. Asking what opinions GPT-3 may hold is a bit like asking what opinions a library has. The only sensible answer is ‘all of them’, even if library policy prevents readers from accessing some of the nastiest books.

Christopher Summerfield, These Strange New Minds: How AI Learned to Talk and What It Means (2024)