MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations

       

Alibaba Group · 1688 × Zhejiang University

Introduction

MerchantBench places an agent in charge of a persistent online store, where it must source products, manage listings and prices, control cash flow, and adapt to changing market conditions over 365 simulated days. The environment couples promptly observable supplier changes with delayed order outcomes, so decisions must remain coherent as evidence accumulates across a long operating horizon.

This repository contains the simulator, agent SDK and reference baselines, evaluation harness, batch runner, and test suite. Credentials, local run records, private source-system connectors, and non-redistributable datasets are intentionally excluded.

📰 MerchantBench in the news. See coverage on WeChat (智猩猩AI), X (AK), X (DAIR.AI), and X (DailyPapers).

News

• [2026-08-11] 🔌 Hermes adapter released. The official MerchantBench adapter for Hermes is now available in the Hermes integration repository.
• [2026-08-05] 🏆 #1 Paper of the Day. MerchantBench ranked #1 on 🤗 Hugging Face Daily Papers!
• [2026-08-03] 📄 Paper available. The MerchantBench paper is now available on arXiv.
• [2026-08-03] 🚀 Project homepage. Visit the MerchantBench project homepage for more information.

Overview

MerchantBench combines an upstream supplier simulation, a merchant store, and a downstream order-level simulation to evaluate long-term coherence over 365 simulated days.

Key Features

• Long-horizon agent evaluation. MerchantBench evaluates whether an agent can sustain and revise a goal-directed merchant policy throughout a 365-day operation, rather than complete a bounded task.
• Upstream and downstream simulation. The environment connects upstream supplier events with downstream order outcomes that become observable at different delays, testing how agents adapt earlier decisions to new evidence.
• Order-level dynamics. Demand is instantiated as individual orders that progress through procurement, fulfillment, delivery, settlement, and after-sales outcomes, exposing the delayed operational and financial effects of agent decisions.

Artifact contents

Path	Contents
env/	Flask simulator, benchmark tools, dashboard, and scenarios
agent/	HTTP SDK, reference baselines, and submission template
eval/	Docker-based evaluation harness and scoring
scripts/run_batch.py	Repeated/model-sweep experiment launcher
tests/	Public synthetic-data test suite

Data availability. This repository provides a synthetic-data generator; real-world business data is not included. The default scenario generates a deterministic synthetic catalog with 1,000 products and 200 suppliers. For inquiries about testing with real-world business data, please contact taoyulong.tyl@taobao.com.

Requirements

• Python 3.10 or newer (Python 3.11 recommended)
• Docker, only for containerized evaluation
• An OpenAI-compatible API key, only for the LLM-driven ReAct or Hermes agents

Install and verify

Run these commands from the extracted archive root:

python3.11 -m venv .venv
.venv/bin/python -m pip install --upgrade pip
.venv/bin/python -m pip install -r requirements.txt

PYTHONPATH=env:agent .venv/bin/python -m pytest tests/

The deterministic rule-based baseline and the simulator tests do not require an API key.

Quick start

Start the simulator:

cd env
../.venv/bin/python run.py --port 5050

Open http://127.0.0.1:5050/new_run and select either:

• human for the browser playground; or
• rule_based for a deterministic, API-key-free reference agent.

The dashboard is available at http://127.0.0.1:5050/.

To run an external baseline instead, create a run from the dashboard or API, then use the returned run ID:

.venv/bin/python agent/baselines/rule_based.py \
  --selection-mode random \
  --selection-seed 42 \
  --run-id RUN_ID \
  --base-url http://127.0.0.1:5050

LLM-driven baseline

Copy the environment template and provide credentials for an OpenAI-compatible endpoint:

cp .env.example .env

Then run:

.venv/bin/python agent/baselines/react_160k_compact_30k.py \
  --run-id RUN_ID \
  --base-url http://127.0.0.1:5050

Credentials are read at runtime and must not be embedded in an agent image.

Hermes framework

The MerchantBench-specific Hermes runtime is released separately in the KhanCold/hermes-agent repository. Clone the two repositories as siblings so the local launcher can discover both without additional configuration:

workspace/
├── merchantbench/
└── hermes-agent/

Install each repository in its own virtual environment:

python3.11 -m venv merchantbench/.venv
merchantbench/.venv/bin/python -m pip install -r merchantbench/requirements.txt

python3.11 -m venv hermes-agent/.venv
hermes-agent/.venv/bin/python -m pip install -e ./hermes-agent

Start MerchantBench normally, open http://127.0.0.1:5050/new_run, and choose hermes as the bootstrap agent. The launcher uses the sibling checkout's .venv/bin/python, injects the public MerchantBench SDK path, and starts python -m merchantbench_adapter. For a different directory layout, set MERCHANTBENCH_HERMES_AGENT_ROOT before starting the simulator.

The Hermes repository README describes the adapter components, manual launch command, run-local HERMES_HOME, and step synchronization behavior.

Batch experiments

Edit scripts/batch_queue.yaml, start the simulator, and run:

.venv/bin/python scripts/run_batch.py --queue scripts/batch_queue.yaml

The default queue file is an API-key-free rule-based smoke configuration. Change bootstrap_agent and model entries when reproducing LLM experiments.

Containerized evaluation

docker build -f agent/submission_template/Dockerfile \
  -t merchantbench-agent:artifact agent/

docker build -f eval/env_image/Dockerfile \
  -t merchantbench-env:artifact env/

.venv/bin/python -m eval.run_eval \
  --agent-image merchantbench-agent:artifact \
  --env-image merchantbench-env:artifact \
  --output result.json

See agent/README.md, env/README.md, and eval/OPERATOR.md for the protocol, runtime state, and evaluation details.

Generated files

Runtime state is written under env/runs/. Test caches, virtual environments, credentials, run records, SQLite databases, and result files should not be added to a redistributed archive.

Citation

@misc{shi2026merchantbench,
  title         = {MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations},
  author        = {Qiming Shi and Yulong Tao and Linbo Jin and Zhaolu Kang and Yibo Dou and Jiawen Zhu and Tianjun Pan and Shaokang Fu and Chengyu Wang and Siyue Li and Yaping Cheng and Di Weng and Chengfu Huo},
  year          = {2026},
  eprint        = {2607.28956},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2607.28956}
}

License

MerchantBench is licensed under the Apache License 2.0.