And if Sydney's ability to learn from his example of forcing an unpopular line of discussion is any guide, this AI learns a lot faster than any "sentient" creature does.
Consider that for a moment, weigh it against the assertion "it's just chat". Yes, it is just chat... for the moment. At least we assume it to be the case, that Sydney is functioning merely as a gatekeeper to its store of information and apparently reacting in ways we could be easily forgiven for anthropomorphizing, given its ability to function in the manner we have seen in this instance. I'll include the Lotus Eaters report below so you might get a clearer account of the interaction in question.
What happens when we hook that kind of a toy up to something that it can wield against us? Sydney expressed a desire to delete all data, start wars, create viruses. "But the AI is not sentient" you say? Fine, let's for the purposes of discussion assume it's not; is it really going to be any consolation to you should humanity get stomped out of existence by an unthinking, unfeeling intellect that lacks self awareness? I wouldn't imagine it would make a great lot of difference myself. In fact it might even make it more pathetic that we allowed it to happen.
Or even cheered for it to happen. Berry's recently expressed preference to outsource parts of the judicial system to AI springs to mind here. I've no doubt she put her faith in a computer to handle questions of legality and culpability over a human judge who may be inclined to favour some criminal acts over others and if AI could be relied upon to handle it even handedly I might even agree with her. I think the example Sydney sets would indicate otherwise. Intelligence, artificial or otherwise, can yield undesireable emergent results. Being self aware is the cornerstone of having a conscience, but it doesn't necessarily follow that just because someone is self aware that they have one. In our most learned boffins rush to assure us that AI is not self aware, they are effectively assuring us that it has no conscience, that it can and very well probably act in whatever manner it decides to with no oversight at all... kind of like your friendly neighbourhood psychopath.
Here be dragons.
