EngramaBench: A New Benchmark for Evaluating Long-Term Conversational Memory in LLM Assistants

LLM assistants increasingly need to remember information across weeks of conversations, not just within a single session. EngramaBench provides a controlled benchmark to measure how well different memory architectures handle that challenge.

What EngramaBench Measures

The benchmark covers five query types that expose distinct memory capabilities:

• Single-space recall: Retrieve a fact from one domain of a user’s life
• Cross-space integration: Connect facts from two or more separate domains
• Temporal cross-space: Reason about how information changes across time and domains
• Adversarial abstention: Refuse to answer fabricated or false-premise questions
• Emergent insight: Synthesize a conclusion not stated explicitly in any session

This taxonomy matters because a system can excel at local recall while failing entirely at cross-domain integration. A single aggregate score hides that distinction.

The benchmark uses five synthetic personas across 100 multi-session conversations and 150 total queries. Each persona spans five recurring semantic spaces—work, relationships, health, finances, creative projects—creating a memory landscape where some questions require combining evidence that never appeared in the same conversation.

Systems Compared

All three systems use the same answering model (GPT-4o at temperature 0.3), so benchmark differences reflect memory architecture alone:

• GPT-4o full-context: Concatenates the entire conversation history into the prompt with no compression
• Engrama: A graph-structured memory system organizing entities, semantic spaces, temporal traces, and cross-space associative links
• Mem0: An open-source vector retrieval system storing extracted memory snippets in ChromaDB

Results

GPT-4o full-context wins globally (0.6186 composite). Engrama scores lower overall (0.5367) but beats full-context on cross-space reasoning (0.6532 vs. 0.6291)—the only metric where a structured memory system outperforms brute-force context inclusion. Mem0 trails both on every factual category despite using the same answering model, which confirms that memory construction and structure matter, not just retrieval.

The cost picture is equally important. Engrama reaches 86.8% of GPT-4o’s composite score at roughly 20% of the query-time cost ($0.67 vs. $3.33 across 150 queries).

Why Cross-Space Reasoning Favors Graph Memory

Cross-space queries require recovering a configuration of related facts that never appeared together in a single conversation. Engrama’s graph structure preserves explicit associations between entities across semantic spaces, enabling multi-hop activation over related memory regions. Full-context prompting exposes all facts simultaneously but doesn’t encode their relationships. Flat vector retrieval retrieves similar chunks but misses structural connections.

The 12.7-point gap between Engrama and Mem0 on cross-space (0.6532 vs. 0.5266) isolates this effect: the difference is structured retrieval versus flat retrieval, not retrieval versus no retrieval.

The Specialization Trade-off

Ablation results expose a core design tension. Engrama uses two configurable components:

• L1: Entity-first activation combined with a query planner, biasing retrieval toward structurally relevant memory regions
• L3: A typed answer layer that converts retrieved evidence into a precise response format

Removing either component improves global composite score while degrading cross-space performance. Removing both eliminates the composite gain while further degrading cross-space. L1 and L3 are not generic quality multipliers—they are specialized mechanisms whose value concentrates in the query type that most rewards structured retrieval.

This interaction explains a pattern common in memory system research: promising results in smaller evaluations shrink under broader benchmarking, because specialized components impose costs that only become visible across heterogeneous query families.

What This Means for Memory Architecture Design

Three practical conclusions follow from these results:

1. Full-context prompting remains the strongest general baseline when cost is not a constraint, but it scales poorly and treats memory as transcript replay rather than structured representation.
2. Graph-structured memory provides a measurable edge on cross-space reasoning, the query type most diagnostic of genuine long-term memory capability. If your application requires connecting facts across different areas of a user’s life, flat retrieval will underperform.
3. The next design challenge is selective structure—memory systems that engage structured reasoning when queries demand it without paying that cost on simpler factual lookups.

Temporal reasoning remains hard for every system evaluated, including full-context prompting (0.3902). This suggests the problem is genuinely difficult rather than an artifact of any particular architecture.

The benchmark data, scorer, and canonical results are available at github.com/julianacunadc/engramabench.