What’s more likely in the techno-fascist idiocracy of the 21st Century?
We somehow manage to turn statistical models into super-intelligence… or the incompetent, criminally-corrupt, sociopathic leadership at the top of capitalism train an AI to defend the rich from the working class — by training it to be the most efficient killing machine possible — integrate it into fully autonomous killing machines, and it (accurately) determines the most efficient way to kill 99% of humanity is to start a nuclear war?
AI being conscious or actively wanting to harm us isn’t the scenario most people worrying about the alignment problem are concerned about. The worry is that it ends up destroying us as a side product of something else it does. In a similar way humans destroy an anthill where we want to put up a house. We don’t do it because we want to harm ants.
The ever classic “paperclip maximiser” problem.
An AI is given the goal of maximising paperclip production in a factory. It’s given no other guard rails. Some shenanigans later, it finishes wiping out humanity, in order to divert more resources to paperclip production. (Apparently humans don’t want the iron in their blood used to make paperclips)