Muses Bench: A Benchmark Framework for Evaluating Multi-User LLM Agents

The paper introduces Muses Bench, a formal benchmark framework for evaluating how LLM agents behave when multiple users interact. Unlike prior work that assumes a single user with one utility function, this setup models distinct roles, permissions, and preferences. The framework organizes evaluation around four capability dimensions: Secure credential management and access control Sequential coordination such as meeting scheduling Shared inference queue fairness Following per-user constraints in multi-stakeholder tasks From the repository, the implementation is structured around four scenario directories under data/scenarios/, each providing JSONL test datasets plus data-builder scripts for custom generation. Scenarios include access_control, meeting_scheduling, shared_queue, and multiuser_instruction_following. The README supports running scenarios through a single CLI entry point (run.py), accepting scenario selection, data paths, model identifiers, and provider flags. Provider support relies on LiteLLM, covering OpenAI, Anthropic, and other LiteLLM-compatible backends. Beyond evaluation, the repo ships