TWIST: A Proposed Benchmark for Intervention Quality in Conversational Memory, with a Human-Validated Draft-Alignment Track
Abstract
Subrat Panda
Existing long-conversation benchmarks increasingly test recall and prompted knowledge updates, and recent work also studies evolving user beliefs and memory state. TWIST complements these evaluations with a proposed benchmark suite for intervention quality in a deployed memory system — exercised through its own ingest/recall/vet surface. Its four tracks address proactively detecting unresolved tensions, vetting proposed outgoing drafts against the record, answering with current beliefs while preserving supersession history, and governing sensitive recall. The suite extends LoCoMo’s corpora and harness pattern, with each track pairing every detect/block metric with a matched do-not-over-detect control: surface-matched hard negatives price false intervention, so no track can be gamed by flagging everything. The validation protocol treats the benchmark itself as the first system under test — independent, gold-blind double annotation with adjudication, an LLM-judge decoy calibration set, and a pre-release separability audit for the tracks where simple retrieval should be insufficient; the instantiated draft-alignment track instead uses matched hard negatives and evidence attribution to separate finding relevant evidence from safely acting on it.
On the human-validated Track B v1.0 key (161 items, post-adjudication κ =0.85), no tested configuration simultaneously achieves high contradiction recall, high hard-negative specificity, and high attribution accuracy: flat-RAG baselines detect 0.76–0.97 of true contradictions but falsely flag 16–43% of surface-matched safe drafts depending on backend, while a deployed coherence-oriented system almost never over-flags (0.98–1.00 specificity on both control classes) yet catches 42% of true contradictions — a trade-off no recall-only score can see. Gold-evidence and full-context baselines localize the causes: every gold contradiction is detectable from its evidence (recall 1.000 for all backends), and calibrated models nearly solve the track given the full transcript — a gap consistent with substantial retrieval- coverage limitations in the retrieval-based configurations — while how context volume shapes restraint proves strongly model-dependent. A system’s TWIST profile, reported alongside its recall score, measures whether a memory system knows both when to intervene and when not to.