Tootfinder

I just watched the #BlackHat #OpenAI #HuggingFace talk (#OpenAI understands it themselves. However, emergence is still no indication of actual intelligence, and the talk is again ripe with anthropomorphism.
3. Yes, it is surprising and somewhat terrifying how good these LLM agent loops have become in finding and exploiting vulnerabilities. We are seeing a much stronger version of the earlier next-generation fuzzing leap in tool capabilities.
4. The comparison with fuzzing and other tools is important for another reason: #OpenAI conducted their experiments on the live Internet. No, forcing internal test runs to go through a proxy server is *not the same as running in an isolated environment*. If there is a connection to a live production system, it is by definition not an effective sandbox. What happened to people letting their cool new fuzzers run live on non-consenting production targets on the Internet back then? They were criminally liable. Why should #OpenAI not be held liable for any damages caused by their systems?
Just because you don't understand the complexity that you built doesn't mean you aren't responsible for what it does!
5. The agent loops first successfully started exploited their proxy server on May 26th, discovered July 4th-6th as active exploitation and takeover. Credentials were revoked and the proxy server patched. "At this point, AI training and evaluations resume."
WHAT THE ACTUAL F...? You discovered emergent behavior in your overly complex system that you didn't understand, saw active exploitation of your internal systems to an extent that they were owned with admin privileges, and your response was to patch the uppermost surface holes, and then continue letting the out-of-control complex system run and continue on the same course? You didn't think to, let's see, maybe stop or at least pause the highly irresponsible experiment while figuring out - in detail - what went wrong, why, and how to avoid the same thing happening again on a systematic level?
6. On July 8th, completely unsurprisingly, the LLM agent loops continue to do the same thing and find another surface hole that hasn't been patched yet to take over again. Why should this have stopped? You haven't done any root cause analysis on the system level. Why do you expect that the problem should have stopped?
7. It takes another 11 days to discover that this is happening again. So you turned the system that had broken something back on again without a detailed root cause analysis and then didn't even watch carefully? I can't even...
8. And no, the response is not to fight fire with fire. Complexity on the attacker side (#OpenAI is the attacker, not the defender - they are the guilty perpetrator, not the innocent victim of circumstance) should be fought with *reduced* attack surface on the defender systems. Adding LLM agent loops that the "frontier" companies themselves quite obviously have no control over to already brittle systems with the hope of auto-patching your way out of vulnerabilities does not seem like a wise course of action. You don't mitigate complexity with even more complexity. The next 2 years will be ... exciting - and your best bet is going to be to disable all dependencies and complex interactions that your production systems don't absolutely require.