Execute the LoCoMo benchmark and report F1 scores by category.
npx skills add https://github.com/genomewalker/cc-soul --skill locomo-benchmark
Please help me install this Agent Skill. Skill: locomo-benchmark Source: https://github.com/genomewalker/cc-soul/tree/main/skills/locomo-benchmark Command: npx skills add https://github.com/genomewalker/cc-soul --skill locomo-benchmark
This Skill addresses the need to quantitatively evaluate the effectiveness of long-term conversational memory systems, like cc-soul, against established academic benchmarks.
Run the locomo benchmark for conversation 26 using the command: python3 $PLUGIN_DIR/scripts/locomo-benchmark.py conv-26
High-intent search queries and answers about installing and using this skill.
To evaluate long-term conversational memory, the LoCoMo benchmark ingests conversations, retrieves context using specified tags, and calculates F1 scores against ground truth for various question types. Results are reported by category and conversation.
The LoCoMo benchmark evaluates conversational AI memory recall across five categories: Multi-hop, Single-hop, Temporal, Open-domain, and Adversarial. It calculates F1 scores for each category and compares results against human and AI baselines.
You can run a conversational memory benchmark evaluation on a specific conversation by executing the Python script with the conversation identifier. For example, use the command: python3 $PLUGIN_DIR/scripts/locomo-benchmark.py conv-26.
No external dependencies are required. The LoCoMo benchmark operates as a standalone script that downloads benchmark data, ingests conversations into memory, and evaluates question-answer pairs to calculate F1 scores.
Yes, you can compare your conversational AI system against established baselines. The benchmark reports F1 scores broken down by category and per conversation, comparing your results directly against human and AI baselines.
The LoCoMo benchmark specializes in long-term conversational memory evaluation rather than general NLP tasks. It focuses on assessing memory recall capabilities through question types like Multi-hop, Temporal, and Adversarial, providing F1 scores against ground truth.
PickleSodaInvoke relevant skills before responding to user queries in AI conversations.
apassuelloEvaluate LLM outputs with automated metrics, human judgments, and benchmarking.
AcosmiSearch long-term memory by keyword with semantic retrieval and relevance scores.
ricardorocheApply standardized metrics to LLM evaluation datasets for reproducible assessments.
