Cal Students' AI Fight Club Makes Tech Giants Sweat in Free-for-All
By Kelli Luu | 28 Feb, 2025

A handful of UC Berkeley Students have built Chatbot Arena into the AI industry’s scorekeeper.

Behind closed doors in the C-suites of the world’s AI titans top execs huddle nervously around a website.  Their focus isn’t some game-changing development by some tech giant.  Their collective brows are furrowing over a student project at the University of California at Berkeley.

 It’s called Chatbot Arena, and it has recently emerged as the UFC battleground for AI models to go head-to-head for the title of Best Chatbot.  The design and engineering behind this AI leaderboard is the work of a team of scholars that is subtly shaping the future of AI.  And giants like Google, Microsoft and Meta can’t afford to look away. 

 Let’s go back to the very beginning of 2023 when Wei-Lin Chiang, a PhD student and an associate of Cal’s Large Model Systems Organization (LMSYS), set out to change how the world measures artificial intelligence.  He was joined by roommate Anastasios Angeleopoulos who, like Chiang, had focused his studies on large language models (LLMs), the technology used to train AI applications like chatbots with data scraped from the internet.  

 The duo’s key, perhaps revolutionary, insight was to evaluate AI performance with a ranking system based on how effectively everyday humans like you and I are able to use each AI model. 

 They launched Chatbot Arena in April 2023 after considering various testing algorithms and running many trial matches.  It didn’t take long before their modest AI leaderboard began attracting enough AI users that tech giants took notice.  It has been keeping them on their toes ever since.

 Chiang and Angelopoulos were the driving force behind Chatbot Arena, but were backed up by teammates who were mostly members of elite LMSYS, a group at UC Berkeley’s Sky Computing Lab dedicated to accelerating AI progress.  Lianmin Zheng, Zhanghao Wu, and Hao Zhang  became essential to the development of Chatbot Arena as a trusted industry tool.

 Chatbot Arena’s early development stages at the Sky Computing Lab often felt like 5 AM workouts for MVPs, guzzling caffeine and slurping instant ramen regardless of whether it was early morning or late night.  Endless days spent staring at code were punctuated by passionate debate over which AI model deserved to rise to number one.

 Chatbot Arena has helped push UC Berkeley to the forefront of the AI industry by creating an opportunity for models developed by small startups to go up against well-financed giants.  An illustration of CA’s impact on the budding AI industry occurred January 25, 2025 when DeepSeek, an unknown Chinese AI startup, shot up the Chatbot Arena leaderboard past Amazon-backed Anthropic and Elon Musk’s xAI. 

 The AI industry was turned upside down when it was discovered that DeepSeek had trained its AI model for a mere $5.6 million back in December 2024.  By comparison Anthropic had spent between $100 million and $1 billion to train the models that had been compared against DeepSeek — and found wanting.  And Elon Musk had revealed that he had raised $6 billion to finance xAI.  For their AI models to be taken down by a 1-year-old Chinese startup using a relatively small number of dumbed-down export-model Nvidia chips made the business world question the wisdom and necessity for the billions being pumped into buying Nvidia chips.

 Chiang and the Chatbot Arena team have succeeded in having top contenders like Google’s Gemini and ChatGPT constantly monitoring their Chatbot Arena rankings in order to make rapid updates in response to shifts in leaderboard.

 Chatbot Arena’s impact has made AI development responsive to the judgment of human users instead of technical performance metrics.  In turn this has given innovative upstarts an equal shot at moving up the leaderboard.

Perhaps as importantly, Wei-Lin Chiang and colleagues have also shown that the next stages of AI development are likely taking shape among brilliant and impatient young minds at university campuses striving to visualize the future.