Training a Vision model on a Text-Only Dataset using Axolotl

Understanding Fine-Tuning of Vision-Instruct Models with Axolotl: Common Challenges and Solutions

Introduction

Fine-tuning large language models (LLMs) with vision capabilities, such as LLaMA variants designed for multimodal tasks, can unlock powerful functionalities. However, configuring these models correctly—particularly when using tools like Axolotl—can present unexpected challenges. This article explores common issues encountered during such fine-tuning exercises, explains their causes, and provides guidance on resolving them to ensure successful training without sacrificing the model’s vision capabilities.

Background

Axolotl is an innovative framework that simplifies fine-tuning powerful LLMs, including those with vision components. When adapting models like LLaMA 3.2 11B Vision-Instruct to specialized datasets, meticulous configuration of the YAML files is crucial. These configuration files specify the model architecture, tokenizer, dataset structure, training parameters, and more.

Common Challenges and Their Causes

1. Error: KeyError: 'Indexing with integers is not available when using Python based feature extractors'

2. Cause: This error typically arises when the processor_type is set to AutoProcessor along with a custom tokenizer_config. For vision-instruction models, the processor is an integrated component designed to handle both text and images, often expecting specific feature extractors.

3. Implication: Setting processor_type: AutoProcessor instructs Axolotl to initialize a feature extractor and tokenizer separately, which can conflict with the model’s built-in processor. If your dataset is text-only, using a vision processor can cause indexing issues because the processor expects image data, but only text data is provided.

4. Solution: If your dataset contains only text data, it’s sometimes best to omit the processor_type or set it to a text-only processor, depending on the model’s configuration. Alternatively, ensure that the dataset aligns with the expected inputs of your processor.

5. Error: AttributeError: 'MllamaTextSelfAttention' object has no attribute 'is_causal'

6. Cause: This occurs when attempting to initialize a model or processor incompatible with the configuration, especially when mixing vision-specific configurations with text-only setups. It indicates that the model’s architecture is expecting attributes related to multimodal attention, but those are missing or misconfigured.

7. Implication: The model’s internal design expects certain components (like `is_causal