We know that LLMs are just trained to predict the next token in a sequence. But if they have intentional states, then behind the scenes, there is something deeper and richer going on. LLMs with intentionality could have private thoughts, ideas or desires. They might have beliefs about what is true or right, or preferences over states of the world. They might secretly think it rude if you put your question in shouty CAPS or forget to include a question mark at the end of your query. Even if they are overtly polite, like a customer-service manager, they might secretly think your question is really dumb. They might be able – either now or in the near future – to work things out for themselves, and start to behave more autonomously. This is obviously a strange and slightly worrying prospect, but as AI systems become more capable, it’s one that is being taken ever more seriously.

Christopher Summerfield, These Strange New Minds: How AI Learned to Talk and What It Means (2024)