Eval Suites

Spawn agents and run many scripted and simulated scenarios concurrently from a single manifest with pipecat eval suite.

pipecat eval run tests scenarios against an agent you started yourself. A suite goes one step further: you list agents and scenarios in a manifest, and pipecat eval suite spawns each agent with its eval transport on its own port, runs its scenarios, tears it down, and aggregates the results, several runs at a time.

Suites are the right tool when you have more than one agent, more than a handful of scenarios, or want a single command for CI. They are also how a simulation runs more than once: each run gets a fresh agent. Pipecat’s own release evals are a manifest with 100+ example agents plus this command.

The manifest

manifest.yaml

concurrency: 4 # how many runs execute at once
runs_dir: eval-runs # logs + recordings go to <runs_dir>/<timestamp>/
record: false # record conversation audio (audio-mode scenarios)
scenarios_dir: scenarios # scenario names resolve to <dir>/<name>.yaml

# How to start each agent. {python}, {bot}, and {port} are substituted per run.
spawn: "{python} {bot} -t eval --port {port}"

suite:
  - bot: bots/support-agent.py
    scenarios: [greeting, capital_question, multi_turn]
  - bot: bots/sales-agent.py
    concurrency: 2 # this entry may hold two slots at once
    scenarios: [greeting, weather_function_call]
  - bot: bots/booking-agent.py
    scenarios: [greeting, simulated/book_table] # a simulation, listed the same way
  - bot: bots/vision-agent.py
    runner_body:
      path: scenarios/vision-body.yaml # optional --runner-body data, as a file
    scenarios: [vision_describe]
  - bot: bots/turn-agent.py
    name: openai/gpt-4o-mini # the entry's label, since two entries share this bot
    runner_body:
      data: { model: gpt-4o-mini } # or written inline
    scenarios: [turn_completion]
  - bot: bots/turn-agent.py
    name: groq/llama-3.3-70b
    runner_body:
      data: { model: llama-3.3-70b }
    scenarios: [turn_completion]

Paths in the manifest (bots_dir, scenarios_dir, runs_dir, the bot: entries) resolve relative to the manifest file, so a manifest is portable: check it into your repo and run it from anywhere. A scenario name can include a subfolder (simulated/book_table), and a name ending in .yaml is a path relative to the manifest.

A scenarios: entry names a scenario file, and the file contributes one run per scenario it holds, named <file>/<scenario> after the file’s name: and the scenario’s own. Scripted scenarios and simulations are listed the same way, and each scenario says which it is: turns: makes it scripted, persona: makes it a simulation. Pipecat’s release evals keep the two in scenarios/scripted/ and scenarios/simulated/ folders, which is a convention worth copying.

Scenarios are reusable across agents. One greeting scenario can cover every agent in the suite.

Entry options

An entry can carry three optional keys beside bot: and scenarios::

• runner_body: supplies session data the agent would normally receive in a /start request body, passed to it as --runner-body. It holds either path:, a YAML or JSON file resolved relative to the manifest, or data:, the body itself as a mapping, which the suite writes to a file among the run’s logs. An agent given a file runs with the file’s directory as its working directory, so a relative path inside the body (a vision agent’s image, say) resolves next to the file. A body that holds such paths belongs in a file for that reason.
• name: is the entry’s label: what the dashboard, the -p filter, results.jsonl, and the artifact file names use. It defaults to the bot: path, so it is only needed when several entries share a bot and differ in their runner_body:, as when sweeping a model. Two entries may not run the same scenario under one label.
• concurrency: is how many of the suite’s slots the entry may hold at once, one by default. See Concurrency.

A bare runner_body: <file> is deprecated since 1.11.0 and will be removed in 2.0.0. Write runner_body: { path: <file> } instead.

Running a suite

pipecat eval suite manifest.yaml

In a terminal, a live dashboard shows each run’s status, a running tally, and total time. When piped (in CI, or driven by a coding assistant), it streams one plain result line per run instead. The command exits 0 only if every run passes.

Useful flags:

pipecat eval suite manifest.yaml -p support       # only entries whose name or bot path contains "support"
pipecat eval suite manifest.yaml -s greeting      # every scenario in the greeting file
pipecat eval suite manifest.yaml -s book_table/available  # one scenario of a file
pipecat eval suite manifest.yaml -k simulation    # only simulations (or: -k script)
pipecat eval suite manifest.yaml -c 8             # 8 runs at a time
pipecat eval suite manifest.yaml -n nightly       # output to eval-runs/nightly/
pipecat eval suite manifest.yaml -a               # record conversation audio
pipecat eval suite manifest.yaml -d               # save full per-pipeline debug logs
pipecat eval suite manifest.yaml -r 5             # run each pair 5 times

-s takes a run’s full <file>/<scenario> name or either half of it, so a file’s name selects every scenario it holds, and a scenario’s name selects it in whichever file it appears.

Everything except the suite: list can live in the manifest or be passed on the command line (the command line wins), so a manifest can be as minimal as a suite: list.

Concurrency

concurrency: is how many runs execute at once. Each entry’s runs go one after another on a single slot, so its scenarios finish as a block and a slow or rate-limited provider holds no more than one slot, whatever the suite’s figure. An entry that can take more sets its own concurrency:, the number of slots it may hold at once, and runs that many of its scenarios side by side. Entries take free slots in manifest order, and the suite’s figure, or --concurrency, bounds the whole.

Run output

Each invocation writes to <runs_dir>/<name>/ (a timestamp when -n is omitted):

eval-runs/20260610_142200/
  results.jsonl                                          # one line per run
  logs/
    bots_support-agent.py__greeting__greeting.log        # the agent process output
    bots_support-agent.py__greeting__greeting.eval.log   # the harness's decision trace
    bots_support-agent.py__greeting__greeting.debug.log  # per-pipeline harness logs (-d only)
  recordings/
    bots_support-agent.py__greeting__greeting.wav        # conversation audio (record: true or -a)

An artifact’s name is the bot path, then the entry’s name: when it has one, then the scenario’s <file>/<scenario> name with its slash written as a double underscore, so bots/turn-agent.py under the label openai/gpt-4o-mini running turn_completion/short_answer writes bots_turn-agent.py__openai_gpt-4o-mini__turn_completion__short_answer.log.

results.jsonl carries one line per run, and every line names its bot, its entry’s name, its scenario, its attempt, and its kind, script or simulation. A scripted record carries the outcome, duration, failures (each with a kind), per-turn results, and paths to its artifacts. Each turn lists its expectations, one per expectation the turn resolved, with whether it passed and what it matched: the marker of an llm_marker, a function call’s signature, a reply’s text. What a passed run matched is only recorded here. A simulation record carries the outcome (passed, succeeded, error), how the run ended (ended_by), the persona’s turn count, every metric’s score, value, failure_kind, and per-turn verdicts (yes, no, or none when the judge gave none), the judge’s reason, the persona’s own end_call claim, and the whole conversation as messages. Lines are appended as each run finishes, so an interrupted sweep keeps everything already done. Runs that didn’t pass also carry events_seen.

Each run executes in its own process, so a harness that loads local audio models does so on its own, and a crash in one run doesn’t stop the others.

results.jsonl is written by pipecat eval suite. pipecat eval run doesn’t produce one.

When a run fails, start with the .eval.log decision trace: it’s a timestamped record of every event the harness saw, what it matched, what the judge said, and why an assertion failed. The agent’s own log sits next to it.

Testing one agent with many scenarios

If you just want to run a batch of scenarios against an agent you already have running, you don’t need a manifest. pipecat eval run accepts multiple scenario files and shares the suite’s dashboard and tally:

pipecat eval run scenarios/*.yaml --bot-url ws://localhost:7860
pipecat eval run scenarios/ --bot-url ws://localhost:7860       # the whole directory

A directory expands to its .yaml and .yml files in filename order, non-recursively, and files and directories can be mixed in one invocation. A directory holding no scenarios is an error rather than an empty run.

By default the agent is left running afterward so it can serve more evals; pass --stop-bot to shut it down when the batch finishes.

Running a simulation more than once

A persona doesn’t say the same thing twice, so one run of a simulation proves little. A simulation’s own runs: field tells the suite how many times to run it, and every run must pass. The suite reports a pass rate per simulation with a ✓ or ✗ beside it, and exits 1 when any run failed:

  Pass rate:
  bots/booking-agent.py  book_table/available      3/3 (100%)  ✓  ~12.4s each

A run that errored (the agent never came up, the persona’s LLM failed, the judge gave no verdict on the goal) is reported but kept out of the rate: it says nothing about whether the agent did its job.

Repeating a run

A behavior with a race in it passes sometimes. repeat: in the manifest, or --repeat / -r on the command line, runs every (bot, scenario) pair N times and reports a pass rate per pair instead of a single verdict:

# manifest.yaml
repeat: 5
suite:
  - bot: bots/support-agent.py
    scenarios: [interruption]

A repeated sweep is attempt-major: every entry’s first attempt runs before any entry’s second, each entry’s scenarios as a block, so every bot meets the same machine conditions in the same stretch and a transient slowdown shows as a band across all of them rather than a regression in one. Artifact filenames gain an attempt suffix (__001) only when repeat is above 1, so a single pass keeps the filenames it always had.

A repeated sweep always exits 0. A pass rate isn’t a pass or a fail, so the threshold is yours to choose: read results.jsonl and decide. Leave repeat unset for the CI gate below.

A repeat set on the command line or in the manifest applies to every run, simulations included, even when it is 1. It overrides a simulation’s runs: and turns the requirement into a measurement: rates are reported and the exit code stays 0. Leave repeat unset to let each simulation run its own runs: and gate on them.

Suites in CI

The exit code makes suites CI-ready with no extra glue:

# e.g. GitHub Actions
- name: Run behavioral evals
  run: pipecat eval suite manifest.yaml

For deterministic, key-free CI runs, prefer text-mode scripted scenarios and an OpenAI-compatible judge endpoint you control. Simulations run their persona on the local Ollama model by default, so they need no extra key, but they say something different each run by design, so give them runs: 3 and expect them to take longer. -k script runs the scripted half alone when you want a fast gate on every push and the simulations on a schedule. Audio-mode scenarios work in CI too, but need the harness’s TTS and STT services available (local models by default, which also need more CPU).