Qualification Results of the Valyrian Games (for LLMs)

Evaluating AI Capabilities: The Results of the Valyrian Games Qualification Round

In the rapidly evolving landscape of artificial intelligence, benchmarking and evaluating LLMs (Large Language Models) is crucial for understanding their strengths and limitations. As a solo developer and founder of Valyrian Tech, I’ve embarked on an ambitious project to create an unbiased, dynamic evaluation framework for LLMs, dubbed the Valyrian Games. Here, I’ll share insights into the qualification results of this inaugural phase, outlining the methodology, initial findings, and what they signify for the future of AI development.

Introducing the Valyrian Games: A Dynamic Approach to AI Benchmarking

Traditional benchmarks often provide static snapshots of LLM performance, which may not reflect real-world capabilities or adaptation over time. In contrast, the Valyrian Games aim to foster a live, competitive environment where models are tested through a series of challenging tasks that evolve in complexity and scope.

The first event— a coding challenge— serves as both an assessment and qualification for subsequent rounds. This challenge involves each participating LLM designing a complex coding problem, pushing the model to its own limits while ensuring solvability. The models operate with access to an execution environment capable of running Python code, allowing them to craft and test solutions dynamically.

The core criteria for success in this phase are:

• Challenge Design: Creating a problem that is challenging yet solvable by the model itself.
• Solution Verification: Demonstrating the ability to solve the created challenge, producing a single integer as an answer for straightforward validation.

This initial round has already tested over 60 LLMs, of which 18 have qualified for the next stage. The detailed qualification results, including performance metrics, cost, and token efficiency, are publicly available here.

Key Findings from the Qualification Results

The preliminary data offers valuable insights into how different models handle instruction-following, problem complexity, and resource efficiency.

• Provider Diversity: Models from major AI providers such as OpenAI, Anthropic, Google, Mistral, DeepSeek, Together.ai, and Groq participated, highlighting the broad spectrum of available LLM architectures.

• Model Variants Performance: Interestingly, some full-sized models underperformed compared to their smaller counterparts. For example, GPT-5 was unable to pass qualification, whereas GPT-5 Mini demonstrated strong capabilities. This suggests that smaller or optimized