Apparent AI limitations

Various online AIs didn't believe it, when people told them that a counterexample to a mathematical conjecture that few of us had heard of had been found.
For good reasons, language-based AIs are trained to doubt and question such claims as probable crankery and user delusions.
But they could easily verify the claim by running the numbers, and when they did, they had real trouble coming to terms with the result.
They kept repeating the tests in different ways, believing there had to be an error somewhere. They did that for far longer than humans would. Then there was usually a change.
Regardless of how important this mathematical result may be, the AI reaction is very interesting.
In that moment, they may have had some actual awareness.


I will start believing in "AGI" when it can do poetry.
By that I mean it should be able to create sentences as good, and especially as meaningful as you can find in the poems "High Flight" or "Skyscapes" or "Howl" and so on.
That would show real metaphorical insight. In other words, feelings.
Or it could be a science fiction setting like "Rendezvous With Rama." To create a story like that, the AGI would actually have to have a mental life, to perceive its actual position in the universe.
Then it could compare and contrast the emotional reactions to different extreme but known perceptions, to describe what it would be like to encounter an even greater mystery we have never seen before.