Maintaining Context in AI Companion Conversations

I am a product designer with extensive experience in creating solutions across various sectors, including SaaS, communication platforms, and other customer-facing applications. Recently, I started working on a project centered around AI and large language models (LLMs). This journey has been a deep dive into understanding the challenges and opportunities in building AI-powered tools. This article is part of my learnings and could be the first in a series of articles where I share insights and discoveries from my work in AI development.

When building my AI companion app, I encountered a challenge where the AI often lost track of previous interactions, leading to disconnected and frustrating experiences for users. After some research, I discovered that GPT-3.5-Turbo and GPT-4 do not inherently support context retention or real-time data. This limitation required me to devise a workaround to ensure the companion could maintain a consistent personality and remember key details about conversations and its relationship with the user.

In the age of AI-driven interactions, users expect their virtual companions or agents to remember and build upon previous conversations. This is especially important for applications like the AI companions I’m building, where the goal is to create seamless, proactive, and engaging dialogue.

Conversation showing distortion in dialogue context

Let’s address the user’s goals from having a seamless experience:

1. Memory of Previous Conversations: Users should feel like the AI remembers previous interactions and fosters a sense of continuity and connection. For example, if a user shares their thoughts about a key project in the recent past, such as “launching a new trial initiative for a product feature,” and discusses its goals, the AI should be able to recall this later and reference it in relevant contexts. This could lead to the AI saying, “You mentioned your trial initiative from last week — would you like insights on industry trends or examples of successful implementations?” This kind of continuity transforms interactions from basic to conversational and mirrors natural dialogue.
2. Relevant Context Responses: The AI’s responses must align with both the immediate query and the broader history of interactions. Let’s imagine a scenario where a user from a private equity firm asks, “Can you draft a summary about the Q1 sales of Apple?” and later follows up with, “Make sure it highlights their new product lines.” The AI should seamlessly integrate these two points, responding with, “Here’s a draft that summarizes the Q1 sales of Apple and introduces their new iPhones.” Such awareness in context can greatly enhance users’ productivity and satisfaction.
3. Consistent Personality and Relationship Details: The AI companion should maintain a consistent persona while remembering key details, such as user preferences and recurring topics. For instance, Alex is one of the AI companions in the app, tailored to a leadership coaching persona. It should consistently exhibit motivational and strategic advice. If the user often discusses challenges in team management, Alex could proactively suggest tips like, “Last time, we discussed improving team collaboration. Would you like some strategies for handling this alignment?”

There are several technical considerations that could influence or constrain the approaches to addressing the issues outlined above. These include:

1. Managing conversation history: Deciding how many previous messages to include in each API request is important here. The dilemma is, including too few previous messages can lead to disjointed responses, while add too many can exceed token limits. I’m currently in a scrapy development mode, with a small budget for OpenAI API credits, so what gives.
2. Storage efficiency vs. Quality of context: Striking a balance between storing the full conversation history between an AI companion and user, versus focusing on only high-priority details is key for the user’s experience, performance of the app and quality of the conversations.

Having found these technical considerations, I could implement a full history method if i can afford the credits for high token usage. This will store the complete conversation history and periodically summarize older parts to reduce token usage. An advantage to this option is, it’ll ensure complete context for generating responses.

Another implementation idea could be to store only important facts and moments separately while maintaining the flow of recent conversation. Although, how does one know which are important facts or just small talks. However, this implementation idea is an efficient use of gpt tokens and storage.

Lastly, this is another implementation idea that combines the strengths of the previous two ideas. We could maintain only recent messages in full while summarizing and storing key points from older interactions. Also, we update the summary as new important information emerges. I assume this will be complex to implement but it’s quite possible. On the bright side, this will balance the need for context and efficiency. It will also preserve most recent and important details while staying within token limits. I’m quite biased to go with this implementation idea for obvious reasons.

Conclusion

Maintaining context in AI companion conversations is quite important for creating meaningful and engaging interactions. By prioritizing the user’s experience goals, addressing technical considerations/limitations, and implementing the idea that stays within token limits, I should be able to enhance the experience such that companions feel truly attentive, personalized and transform into trusted conversational partners.