I think the rationalist obsession with general artificial intelligence as an existential threat is baffling, but I find it baffling because I come from a Green philosophical background. I'm horrified by AI systems as they are today because they are destroying ecosystems (both literal, our climate, and more abstract — art, economy, authorship).
From a relational point of view, they are a disaster. A few mega-corporations are steering our information ecology, in an instrumentalist way. They cannot even see the bonds they are destroying. We are far poorer for it.
We already have paperclip maximizers: billionaires. Corporations. Profit motive. Money is just paperclips with power attached.
These ways of thinking can only see a system so far as to instrumentalize it. After that they switch to modes of control that eliminate any need for understanding.