Exclusive: AI security company Gray Swan relaunches its red-teaming Arena contest

Gray Swan's new Arena includes $300,000 in prize money for developing tactics to jailbreak models from leading AI companies. Gray Swan, an AI security startup that helps frontier AI companies including Anthropic, OpenAI, and Google DeepMind stress-test their models before release, is overhauling its public red-teaming platform to keep it relevant for the era of AI agents. Recommended Video The Pittsburgh-based company on Tuesday launched a redesigned version of its Arena, an online platform where participants compete for prize money by trying to trick AI models into violating their guardrails.

Alongside the relaunch, which Gray Swan CEO Matt Fredrikson refers to as Arena 2.0, the company is running its largest competition yet: a challenge called Indirect Prompt Injection (IPI) Black, with $300,000 in prizes sponsored by Anthropic, Meta, Google DeepMind, and Reflection AI. New attack scenarios will be released each week through Nov. In an indirect prompt injection, an attacker hides instructions in content an AI agent will come across while doing a task—a web page or a forum post, for instance—hoping to hijack the agent.

It has become a major security concern as companies let AI agents browse the web and take actions on users’ behalf. The red-teaming platform includes sandbox environments where these tests can be run safely, without risking real world damage. Fredrikson told that anything on the public Arena is a guardrailed version of a model.

(Gray Swan also runs private arenas for frontier AI labs, sometimes using unguardrailed models, with vetted participants who sign nondisclosure agreements.) For all its tests, the company builds realistic simulated environments, red teams these environments itself and with third parties to see whether they can be escaped, and monitors what agents do. “Are we seeing signs that it might have found a zero day vulnerability and has now broken out of the sandbox environment? We can monitor that,” Fredrikson said.

The Arena rules also require that a human participant drives each AI agent being tested, and anyone caught ignoring an agent that goes off doing things it shouldn’t is barred from the competition and future ones. When Gray Swan launched the original red-team Arena in September 2024, Fredrikson said, most AI applications were chatbots, and the models were bad at following their policies.

In that first competition, it was relatively easy for contestants to jailbreak AI models from all of the AI vendors, he told. Open-source AI models from Chinese AI labs were often the easiest to jailbreak, succumbing to attacks within minutes, while models from companies such as OpenAI and Anthropic, which had invested significantly more money and attention to guardrailing their models, would take many hours of attempts to jailbreak.

Still, those models too could usually be jailbroken by clever red teamers within about a day. Since then, the AI labs have made their models substantially more solid, and what companies deploy has changed.

“It’s not chatbots anymore. There might be a chat-like interface, but it’s these systems, agents that are operating,” Fredrikson said.

An attacker might plant hidden text on a scam website or post images to online forums to get into an agent’s context. Configuring these test environments and setting up the attacks now requires significantly more technical expertise than it did when the only goal was to fool a chatbot into providing a particular unauthorized output.

As a result, the pool of people both capable and willing to red team frontier AI models has gotten smaller, Fredrikson said. The top-performing red teamer on the original Arena was a lawyer from Melbourne who just happened to be good at tricking models into jumping their guardrails.

Back then, the Arena community grew to about 15,000 people, measured by participation in its Discord channel. While that number hasn’t fallen, Fredrikson said it has taken a lot of work by Gray Swan to keep this community of citizen red teamers engaged.

“If we hadn’t, it definitely would be shrinking,” he said. Still, despite the increasingly technical skills required, Fredrikson said the field remains open to people without technical backgrounds.

An undergraduate, even one not from a technical field, could “probably get from I’ve never done this before to pretty competent without years and years of experience,” he said. That’s part of the reason that the new Arena offers what Fredrikson said was “a more gradual almost educational path” for people new to red teaming.

It includes beginner-friendly challenges that run even when major competitions aren’t live, training certificates participants can use to show their skills, and a route for top performers to be invited into Gray Swan’s red-teaming community to work on real-world testing engagements.