VoxLock Audio Filter

Keeps the caller’s voice and removes the other voices in the room before VAD and speech-to-text, on the CPU

Overview

VoxLockFilter is a BaseAudioFilter implementation backed by VoxLock, an open-source (Apache-2.0) voice isolation model. You attach it to a transport’s audio_in_filter, and it keeps the caller’s voice and removes the other people talking in the caller’s room, so VAD and your STT service only hear the caller. It runs locally with onnxruntime on the CPU: no API key and no network calls after the model download.

VoxLock locks on to one voice. Until it has heard 3 seconds of the caller’s speech, the filter passes audio through unchanged; it then takes that speech as the voice sample and isolates from then on. If you already have a sample of the caller, pass it in and isolation starts with the first frame.

Source Repository

Code, Pipecat example bot, benchmark harness and changelog

Model

Model card, benchmark results and ONNX files on Hugging Face

PyPI Package

The voxlock package on PyPI

Live Demo

Listen to calls with the filter off and on, in the browser

Installation

This is a community-maintained package distributed separately from pipecat-ai:

uv add "voxlock[pipecat]"

Requires Python 3.11 or later, like pipecat-ai.

Prerequisites

• No API key. The model (75 MB) downloads from Hugging Face on first use and is cached.
• CPU. With the default settings the filter uses 2 CPU threads per call. See Choosing the CPU settings.

Configuration

voice_sample · np.ndarray | None · default: "None"

About 3 seconds of the caller’s voice, 16 kHz mono float32. When given, isolation starts from the first audio frame.

sample_seconds · float · default: "3.0"

Seconds of speech to collect as the voice sample when voice_sample is not given. Audio passes through unchanged until then.

chunk_ms · int · default: "80"

Audio is run through the model in chunks of this length. Larger chunks use less CPU and add their length to the delay.

threads · int · default: "2"

onnxruntime threads for this filter.

precision · str · default: "\"int8\""

Model files: "int8", "int8-full" (fastest), "fp16" or "fp32".

model_dir · str | None · default: "None"

Folder with the ONNX files. None downloads them from Hugging Face.

Usage

Create one filter per call and pass it to your transport as audio_in_filter:

from pipecat.transports.base_transport import TransportParams
from voxlock.pipecat import VoxLockFilter

transport_params = TransportParams(
    audio_in_enabled=True,
    audio_in_filter=VoxLockFilter(),
)

The transport drives the filter lifecycle (start(sample_rate), filter(audio), process_frame(frame) and stop()), so no additional wiring is needed. A complete voice agent is in examples/pipecat_bot.py.

With a voice sample

import voxlock
from voxlock.pipecat import VoxLockFilter

sample = voxlock.load_audio("caller_sample.wav")  # about 3 s, any format
audio_filter = VoxLockFilter(voice_sample=sample)

Choosing the CPU settings

CPU	Settings	Real-time factor
2 or more threads (default)	VoxLockFilter()	0.67
One vCPU, e.g. Pipecat Cloud agent-2x	VoxLockFilter(chunk_ms=160, threads=1)	0.73

Measured on an 8-vCPU Xeon server; below 1 keeps up with the call. Half a vCPU (Pipecat Cloud agent-1x) is too little.

Input Frames

FilterEnableFrame · Frame

Control frame to toggle isolation on and off at runtime

from pipecat.frames.frames import FilterEnableFrame

# Bypass isolation: audio passes through unchanged
await task.queue_frame(FilterEnableFrame(enable=False))

# Resume isolation
await task.queue_frame(FilterEnableFrame(enable=True))

Audio Requirements

• Input is 16-bit mono PCM at the transport’s sample rate. The filter resamples to 16 kHz for the model and back, so 8 kHz telephony and 48 kHz WebRTC both work.
• Audio is collected into chunk_ms chunks, so some calls to filter() return no audio; the input transport skips those.

Compatibility

Tested with Pipecat 1.12.0. See the changelog for the latest tested version.