So, my own take – which is not necessarily shared by everyone — is that current AI systems, the LLM things like ChatGPT or Claude or whatever, are going to have a pretty hard time running amok to a huge degree, due to technical limitations. One big one: they have a lot of static memory, but their “mutable memory” is not very large — just what lives in the context menu.

And to some extent, the specific way in which hallucinations show up are an artifact of the fact that they are LLMs. My expectation is that an artificial general intelligence that can reason like a human likely will not be simply an LLM (though it might incorporate an LLM).

However, you can say, I think, that at some point, we will have artificial general intelligences that work at a human level. And then…yeah, the fact that whatever reasoning process they use, they will probably make errors, just as humans do. And that could be a problem, just as it is when humans do. In the case of an advanced AI that is much more capable than humans, how to control it and make it do things that we would want is a problem, and not an easy one. Maybe a problem that we can’t actually solve.

What is happening in the world reminds me of a novel - I have no mouth, but I must scream. Have you read this novel?

No, but I did play the adventure game based on it in ScummVM.