Creating a Custom Memory System for Your AI with Minimal Hardware

In the evolving landscape of AI, personalized memory capabilities can significantly enhance the quality and relevance of responses. Interestingly, constructing a true memory system for AI models like GPT can be achieved using just a CPU—though it requires patience and careful setup. This guide provides a streamlined overview of how to generate, format, and upload custom memory data onto a platform such as CustomGPT, enabling more personalized and context-aware interactions.

Step 1: Export Your Data

Begin by exporting your conversation data. Depending on the platform you’re working with, this typically involves selecting and downloading a conversation archive. For example, after making your selection, you’ll obtain a file—commonly named conversations.json. This file contains your dialogue history in JSON format, serving as the foundation for creating your personalized memory.

[Sample image reference: Selection-999-1333.png]

Step 2: Prepare Your Data Files

Alongside your conversations.json, you’ll need two key Python scripts:

• batch_embedder.py: Responsible for processing your conversation data into embeddings suitable for fine-tuning.

• Access script here

• data_formatter.py: Formats the embedded data into a structured dataset compatible with training processes.

• Access script here

Run the batch_embedder.py script on your conversations.json to generate embeddings, then use data_formatter.py to structure the data appropriately. The process results in a directory named data_finetune, which contains a file titled train_detailed.jsonl.gz. For simplicity, rename this to memory.jsonl.gz.

Step 3: Upload and Integrate with CustomGPT

Download the memory search script:

• memory_search_enhanced.py

Upload memory.jsonl.gz to your CustomGPT instance and incorporate the script to enable efficient retrieval of stored memories.

Sample usage commands include:

“`bash

Search for mentions of “max richter” within your memory

!python3 /mnt/data/memory_search_enhanced.py /mnt/data/m