OpenVoice Multilingual Voice-Cloning API

A production-oriented Flask wrapper around MyShell OpenVoice V2 and MeloTTS. It turns text plus an authorised reference voice into multilingual speech through a validated API with lazy model loading, collision-safe temporary files, health checks, downloadable outputs, Docker and model-free tests.

This repository is a cleaned, reusable version of voice-generation service patterns built for multilingual media workflows. It does not include proprietary audio, company database tables, credentials, checkpoints or generated voices.

What it demonstrates

• Zero-shot tone-colour transfer from a short reference recording
• Japanese, English and Chinese synthesis configuration, extensible to other OpenVoice V2 languages
• Lazy loading of OpenVoice converters and MeloTTS language models
• CPU/CUDA device selection through environment variables
• JSON requests with a configured reference or multipart reference-audio uploads
• Safe generated filenames and an audio download endpoint
• Backward-compatible routes for legacy integrations
• CSV batch client without application-specific database coupling
• Dependency-injected tests, Ruff checks and GitHub Actions CI

Responsible-use boundary

Only clone a voice when the speaker has explicitly authorised that use. Do not use this service for impersonation, fraud, harassment or deceptive media. Production systems should retain consent records, disclose synthetic audio where appropriate, restrict access, rate-limit generation and consider audible or machine-detectable provenance controls.

Architecture

JSON or multipart request
  -> validation + request limits
  -> MeloTTS base speech for selected language
  -> OpenVoice reference embedding + tone-colour conversion
  -> UUID-named WAV output
  -> authenticated delivery layer in the consuming system

Prerequisites

• Python 3.10 (the most widely tested version for the upstream stack)
• FFmpeg
• OpenVoice V2 checkpoints downloaded according to the upstream instructions
• A reference WAV file for a voice you are authorised to use

Expected checkpoint layout by default:

checkpoints_v2/
  converter/config.json
  converter/checkpoint.pth
  base_speakers/ses/
    en-us.pth
    jp.pth
    zh.pth

Checkpoint filenames vary between upstream releases. Override each path with the environment variables below rather than renaming model artefacts.

Install and run

python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

export OPENVOICE_CHECKPOINT_ROOT=checkpoints_v2
export OPENVOICE_REFERENCE_AUDIO=references/authorised-speaker.wav
gunicorn -c gunicorn.conf.py "app:app"

The model is loaded on the first synthesis request, not during import. /health therefore works in orchestration checks without allocating model memory.

API

Synthesis with configured reference audio

curl http://localhost:5002/synthesize \
  -H "Content-Type: application/json" \
  -d '{"text":"こんにちは。音声生成のデモです。","language":"JP","speed":1.0}'

Synthesis with an uploaded reference

curl http://localhost:5002/synthesize \
  -F "text=Hello from a consented reference voice." \
  -F "language=EN" \
  -F "[email protected]"

The response includes an audio_url. Download it from GET /audio/<generated-id>.wav.

Legacy POST /get_openvoice, /get_openvoice_batch and /get_openvoice_batch_2 routes remain available and return the original audio_path field.

Batch client

Prepare a UTF-8 CSV:

text,language
Hello world,EN
こんにちは,JP

python batch_client.py input.csv output.csv --service-url http://localhost:5002

Configuration

Variable	Default	Purpose
OPENVOICE_CHECKPOINT_ROOT	checkpoints_v2	Root checkpoint directory
OPENVOICE_REFERENCE_AUDIO	unset	Authorised default reference WAV
OPENVOICE_OUTPUT_DIR	generated_audio	Generated WAV directory
OPENVOICE_DEVICE	auto	Explicit cpu, cuda:0, etc.
OPENVOICE_JP_SOURCE_SE	.../jp.pth	Japanese base-speaker embedding
OPENVOICE_EN_SOURCE_SE	.../en-us.pth	English base-speaker embedding
OPENVOICE_ZH_SOURCE_SE	.../zh.pth	Chinese base-speaker embedding
MAX_TEXT_CHARS	2000	Text-size limit
MAX_UPLOAD_MB	20	Request-size limit
PORT	5002	Development server port

Tests

python -m pip install -r requirements-dev.txt
ruff check .
pytest -q

CI injects a fake synthesiser, so it verifies validation, file handling and response contracts without downloading checkpoints.

Docker

docker build -t openvoice-api .
docker run --rm -p 5002:5002 \
  -e OPENVOICE_CHECKPOINT_ROOT=/models/checkpoints_v2 \
  -e OPENVOICE_REFERENCE_AUDIO=/references/authorised.wav \
  -v /absolute/path/checkpoints_v2:/models/checkpoints_v2:ro \
  -v /absolute/path/references:/references:ro \
  -v openvoice-output:/app/generated_audio \
  openvoice-api

Production checklist

• Add authentication and per-user authorisation in the product layer.
• Store consent and permitted-use metadata for every reference voice.
• Scan uploads, restrict media types and enforce gateway-level rate limits.
• Move long synthesis jobs to a queue if requests can exceed proxy timeouts.
• Monitor latency, failures, GPU memory and generated-file retention.
• Use object storage with expiring signed URLs instead of the built-in file route at scale.

License and attribution

Service code is MIT licensed. OpenVoice V1/V2 are MIT licensed; checkpoint and dependency terms still apply. OpenVoice is the work of its upstream authors and this independent API wrapper is not affiliated with MyShell.