CosyVoice

👉🏻 CosyVoice 演示 👈🏻

[CosyVoice 论文][CosyVoice 工作室][CosyVoice 代码]

关于 SenseVoice，请访问 SenseVoice 仓库和 SenseVoice 空间。

安装

克隆并安装

克隆仓库

git clone --recursive https://github.com/v3ucn/CosyVoice_For_Windows.git
# 如果由于网络问题克隆子模块失败，请重复运行以下命令直到成功
cd CosyVoice_For_Windows
git submodule update --init --recursive

安装 Conda：请参见 https://docs.conda.io/en/latest/miniconda.html
创建 Conda 环境：

conda create -n cosyvoice python=3.11
conda activate cosyvoice
pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com

从 https://github.com/S95Sedan/Deepspeed-Windows/releases/tag/v14.0%2Bpy311 安装 deepspeed

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

# 如果你使用 Windows
无需安装 sox

模型下载

我们强烈建议您下载我们预训练的 CosyVoice-300M、CosyVoice-300M-SFT、CosyVoice-300M-Instruct 模型和 speech_kantts_ttsfrd 资源。

如果您是该领域的专家，只对从头开始训练自己的 CosyVoice 模型感兴趣，可以跳过这一步。

# SDK模型下载
from modelscope import snapshot_download
snapshot_download('iic/CosyVoice-300M', local_dir='pretrained_models/CosyVoice-300M')
snapshot_download('iic/CosyVoice-300M-SFT', local_dir='pretrained_models/CosyVoice-300M-SFT')
snapshot_download('iic/CosyVoice-300M-Instruct', local_dir='pretrained_models/CosyVoice-300M-Instruct')
snapshot_download('speech_tts/speech_kantts_ttsfrd', local_dir='pretrained_models/speech_kantts_ttsfrd')

# git模型下载，请确保已安装git lfs
mkdir -p pretrained_models
git clone https://www.modelscope.cn/iic/CosyVoice-300M.git pretrained_models/CosyVoice-300M
git clone https://www.modelscope.cn/iic/CosyVoice-300M-SFT.git pretrained_models/CosyVoice-300M-SFT
git clone https://www.modelscope.cn/iic/CosyVoice-300M-Instruct.git pretrained_models/CosyVoice-300M-Instruct
git clone https://www.modelscope.cn/speech_tts/speech_kantts_ttsfrd.git pretrained_models/speech_kantts_ttsfrd

基本用法

对于零样本/跨语言推理，请使用 CosyVoice-300M 模型。对于 SFT 推理，请使用 CosyVoice-300M-SFT 模型。对于指令推理，请使用 CosyVoice-300M-Instruct 模型。首先，将 third_party/AcademiCodec 和 third_party/Matcha-TTS 添加到您的 PYTHONPATH。

set PYTHONPATH=third_party/AcademiCodec;third_party/Matcha-TTS

from cosyvoice.cli.cosyvoice import CosyVoice
from cosyvoice.utils.file_utils import load_wav
import torchaudio

cosyvoice = CosyVoice('speech_tts/CosyVoice-300M-SFT')
# sft 用法
print(cosyvoice.list_avaliable_spks())
output = cosyvoice.inference_sft('你好，我是通义生成式语音大模型，请问有什么可以帮您的吗？', '中文女')
torchaudio.save('sft.wav', output['tts_speech'], 22050)

cosyvoice = CosyVoice('speech_tts/CosyVoice-300M')

零样本使用

prompt_speech_16k = load_wav('zero_shot_prompt.wav', 16000) output = cosyvoice.inference_zero_shot('收到好友从远方寄来的生日礼物，那份意外的惊喜与深深的祝福让我心中充满了甜蜜的快乐，笑容如花儿般绽放。', '希望你以后能够做的比我还好呦。', prompt_speech_16k) torchaudio.save('zero_shot.wav', output['tts_speech'], 22050)

跨语言使用

prompt_speech_16k = load_wav('cross_lingual_prompt.wav', 16000) output = cosyvoice.inference_cross_lingual('<|en|>And then later on, fully acquiring that company. So keeping management in line, interest in line with the asset that's coming into the family is a reason why sometimes we don't buy the whole thing.', prompt_speech_16k) torchaudio.save('cross_lingual.wav', output['tts_speech'], 22050)

cosyvoice = CosyVoice('speech_tts/CosyVoice-300M-Instruct')

指令使用

output = cosyvoice.inference_instruct('在面对挑战时，他展现了非凡的勇气与智慧。', '中文男', 'Theo 'Crimson', is a fiery, passionate rebel leader. Fights with fervor for justice, but struggles with impulsiveness.') torchaudio.save('instruct.wav', output['tts_speech'], 22050)

启动网页演示

你可以使用我们的网页演示页面快速熟悉CosyVoice。我们在网页演示中支持sft/零样本/跨语言/指令推理。

请查看演示网站了解详情。

python3 webui.py --port 9886 --model_dir ./pretrained_models/CosyVoice-300M

高级用法

对于高级用户，我们在examples/libritts/cosyvoice/run.sh中提供了训练和推理脚本。你可以按照这个配方熟悉CosyVoice。

部署构建

可选地，如果你想使用grpc进行服务部署，你可以运行以下步骤。否则，你可以直接忽略这一步。

cd runtime/python docker build -t cosyvoice:v1.0 .

如果你想使用指令推理，请将speech_tts/CosyVoice-300M改为speech_tts/CosyVoice-300M-Instruct

docker run -d --runtime=nvidia -p 50000:50000 cosyvoice:v1.0 /bin/bash -c "cd /opt/CosyVoice/CosyVoice/runtime/python && python3 server.py --port 50000 --max_conc 4 --model_dir speech_tts/CosyVoice-300M && sleep infinity" python3 client.py --port 50000 --mode <sft|zero_shot|cross_lingual|instruct>