Ran a YouTube strategy test with Gemini, GPT-4, and Claude — Gemini nailed creativity, but it also made stuff up

Exploring AI-Generated Content Strategies: An Evaluation of Gemini, GPT-4, and Claude in YouTube Content Planning

In the rapidly evolving landscape of digital content creation, leveraging artificial intelligence (AI) tools has become an essential component of strategic planning. Recently, I conducted a comparative analysis of three leading AI models—Gemini 2.5 Pro, GPT-4, and Claude 3.5 Sonnet—to understand their capabilities in generating actionable insights for YouTube channel growth. This article summarizes the methodology, findings, and key takeaways from that experiment, offering insights for content creators and digital strategists alike.

Experimental Framework

The objective was to assess how each AI model interprets and processes structured input data related to a mid-sized YouTube channel (~60,000 subscribers). To ensure fairness and consistency, identical datasets were provided to each model, including:

• Ten comprehensive video scripts
• A spreadsheet detailing video titles, viewer metrics, and click-through rates (CTR)
• Screenshots from YouTube Studio showcasing audience retention and traffic sources

The prompt used was designed to solicit strategic recommendations and creative suggestions, framing the context clearly to each AI.

Key Findings and Comparative Performance

Gemini 2.5 Pro: A Creative Powerhouse with a Caveat

Gemini demonstrated remarkable speed and an impressive ability to generate well-structured, personalized suggestions. It proposed actionable ideas such as transitioning long-form content into a series format, identifying retention dips around the three-minute mark, and segmenting videos for Shorts. These insights could prove valuable for content optimization.

However, Gemini also exhibited a tendency to hallucinate—fabricating details not present in the input data. For instance, it referenced a “tech collaboration” video that did not exist in the dataset, and this error persisted despite multiple attempts. While its creative outputs are compelling, such inaccuracies necessitate careful fact-checking.

GPT-4: Reliable and Balanced

GPT-4 offered conservative, straightforward recommendations closely aligned with the provided data. Its outputs were clear, structured, and safe, making it a dependable choice for users seeking dependable insights without surprises. However, its responses tended to be less innovative and more cautious, which might limit creative exploration. Processing times were also comparatively longer.

Claude 3.5 Sonnet: Analytical and Precise

Claude exhibited superior accuracy, consistently aligning its observations with the input data. It identified meaningful patterns—such as a 20% higher CTR on videos under six minutes—