The whole thing is optimized for scams, deception and other criminal behavior:
- user interface that deceptively pretends it's a human you're talking to
- claims from companies highly exaggerate capabilities
companies and "experts" constantly hype "AGI" which they (funnily enough) do to both make investors greedier and spread fear and as a distraction because these algorithms can't actually do what they keep promising
- large-scale accounting and financial fraud (e.g. what Nvidia is doing with circular selling)
- biggest case of copyright infringement in history
Note: I think the underlying technology is really cool, and definitely has use cases and can be used for actually good things. But: some technology just has more downsides than upsides, and some should only be used by experts in controlled environments. Leaded gasoline, asbestos and chlorofluorocarbon are also all really cool technology.
In this case perhaps the techology itself doesn't do anything inherently bad, however the people making it are lying about what it can do, the people selling it are motivated purely by greed and the people using it (often forced to do so) are being deceived.