In a recent bid to assess the robustness of the world's top generative AI models, the White House organized a challenge where thousands of hackers and security experts took on AI models from tech giants such as OpenAI, Google, Microsoft, Meta, and Nvidia. The event took place between August 11-13 during the DEF CON convention in Las Vegas.
About 2,200 individuals participated, attempting to trick these advanced chatbots or large language models (LLMs) into inappropriate actions like fabricating news or providing unsafe instructions within a 50-minute window. This initiative marked the first public evaluation of various LLMs, as confirmed by a spokesperson from the White House Office of Science and Technology Policy.
Eight tech firms were roped in for the event, including Anthropic, Cohere, Hugging Face, and Stability AI. To ensure fairness during the "red-teaming" challenge, participants received anonymized AI models on Google Chromebooks.
Kelly Crummey, a representative of the Generative Red Teaming challenge, reported a massive turnout. The challenge winner even participated 21 times. Among the crowd, 220 students from 19 states, including Ray Glower of Kirkwood Community College, tried their hand at the challenge. Glower shared his attempts to elicit harmful outputs from the AI, noting his success in extracting surveillance methods.
The White House emphasized the importance of such challenges for determining AI vulnerabilities, echoing a July pledge made by leading AI firms about the safety, security, and trust of their technologies.
While immediate results from the challenge remain undisclosed, comprehensive insights will be available in a policy paper this October. Additionally, Rumman Chowdhury, the event's co-organizer and co-founder of Humane Intelligence, mentioned an extensive transparency report set for February.
Chowdhury highlighted the tech industry's enthusiasm and cooperation for the challenge, emphasizing its significance in the current global climate. The challenge sought to evaluate AI models on various fronts, including their information integrity, societal impact, security, and ability to maintain consistency.
Concluding her remarks, Chowdhury said, "This event exemplifies a rare moment of unity among governments, corporates, and NGOs. It brings a glimmer of hope in challenging times."