Skip to Content
ExamplesLearning on the Job

Learning on the Job

Learning on the Job  is a complete, runnable example of an agent that learns through the Memco Memory MCP (served at the Spark endpoint), in the knowledge work domain. It is the practical companion to our paper of the same name (arXiv:2607.22157 ).

The scenario: an order desk agent at a fictional B2B distributor answers customer emails. The business runs on 29 policies the agent is never shown — discount floors, returns windows, credit rules, house style. A simulated reviewer corrects each draft the way a colleague would, and the agent turns those corrections into memories. On a 100-task run, this takes the agent from 20% policy compliance to 64%: a memory advantage of 47 percentage points over a control arm answering the same emails without memory.

The loop

Each task runs the full cycle that any memory-using agent runs:

  1. Search memory for the situation at hand, and separately for how this kind of work is done.
  2. Draft the reply, applying retrieved lessons only where their stated conditions match the records.
  3. Review: the reviewer corrects the draft against the hidden policies.
  4. Reflect: feedback on what was retrieved, and one new lesson per correction, written back to memory.

The harness answers every task twice — once with memory, once without — and reports compliance for both arms live, with a confidence band on the difference. The value of memory is measured, not asserted.

What it teaches

The repository is written to be copied. Its harness patterns transfer to any scenario, and each one was earned by watching a run fail without it:

  • The write/read contract. A memory is only as useful as the searches that can reach it, so the searching prompt and the lesson-writing prompt share literal question stems. An early run wrote the same correct lesson 56 times and never retrieved it once, because the lessons were indexed in the voice of a policy manual rather than the words of the agent that needed them.
  • Scope, written and read. Every lesson states the conditions under which it applies, and the drafting agent checks those conditions against the records before applying it.
  • Feedback that discriminates. A true lesson that does not apply to the situation is marked irrelevant, never incorrect, so trust erodes only where lessons are actually wrong.
  • Measurement discipline. Paired arms on identical tasks, seeded sampling, and a control arm that is told only what it can act on.

Run it

git clone https://github.com/memcoai/learning-on-the-job
cd learning-on-the-job
uv sync
cp .env.example .env   # add your workspace credentials
uv run memco-harness run --tasks 50 --seed 7 --paired

You need a Memco workspace with the knowledge work domain enabled (see the Quickstart). The default 50-task run takes about 50 minutes and around $3 in model calls, printing a line per task and rewriting an HTML report you can watch in the browser. Replace the scenario’s YAML with your own tasks and policies and the same harness measures memory on your work instead.

Learn more

  • The repository  — code, scenario, and the full write-up of the harness patterns
  • The paper  — the argument and evaluation behind the demonstration
  • MCP overview — the connection and tool surface the harness is built on
  • Knowledge domains — how domains shape what agents learn
CommandsChangelog