Understanding Hallucinations in Document Analysis with AI Language Models

Introduction

Recently, I engaged with an AI language model, Gemini, to analyze a substantial worldbuilding document for a fantasy project. The experience highlighted some challenges, particularly around the model’s tendency to generate “hallucinated” content—essentially fabricating information instead of accurately extracting details directly from the text.

The Challenge of Document Input

To provide some context, the document I worked with spans an impressive 82,682 words, making it quite a meaty file for any AI to digest. My initial aim was to simply retrieve factual information, such as a list of fictional nations mentioned within the document. Surprisingly, despite having an explicit section dedicated to this information, Gemini only managed to generate a partial list. This alone was somewhat expected, but as I continued to ask more questions, a troubling pattern emerged: the AI frequently preferred to invent details rather than accurately report what was present.

For instance, when I requested the reproduction of the table of contents, Gemini complied but included entries that, while thematically aligned with the document’s content, were not actually present. This phenomenon isn’t exclusive to Gemini; I’ve observed similar behavior in ChatGPT, although the latter is often more upfront about acknowledging its limits. On the other hand, Gemini continued to assert that it was accurately reflecting the content of the document, which raised questions about its reliability.

Exploring the One-Million-Token Context Window

To add to the perplexing experience, I initially tried to retrieve information by copying and pasting excerpts of the text directly into the prompts. Unfortunately, I found that Gemini seemed to forget earlier entries, even with its advertised one-million-token context window, which should typically accommodate such a lengthy document without issue. When I tested its memory by querying specific words or phrases, it often failed to recall them, which was frustrating.

My approach was to provide the text in prompts and trust that Gemini would effectively remember the context. I also encouraged the model to confirm its understanding by sending back section titles, which it did. However, I couldn’t help but feel that there must be a more efficient way to interact with these AI models to improve information retrieval accuracy.

Seeking Solutions for Effective AI Communication

Given these experiences, I’m reaching out to the community for insights. Is there an established method for optimizing LLMs (Large Language Models) in terms of extracting information accurately, rather than fabricating? Additionally, any guidance on how to