musiclm-pytorch - Pytorch实现的音乐生成模型MusicLM

MusicLM - Pytorch

使用注意力网络在Pytorch中实现MusicLM，Google最新的SOTA音乐生成模型。

他们基本上是在使用文本条件的AudioLM，但令人惊讶的是，使用了一个名为MuLan的文本-音频对比学习模型的嵌入。本库将构建MuLan，并从另一个库中修改AudioLM以支持此处的音乐生成需求。

如果你对与LAION社区一起进行复现感兴趣，请加入

感谢

感谢Stability.ai慷慨赞助以研究和开源尖端的人工智能技术
感谢🤗 Huggingface提供的accelerate训练库

使用方法

$ pip install musiclm-pytorch

使用方法

首先需要训练MuLaN

import torch
from musiclm_pytorch import MuLaN, AudioSpectrogramTransformer, TextTransformer

audio_transformer = AudioSpectrogramTransformer(
    dim = 512,
    depth = 6,
    heads = 8,
    dim_head = 64,
    spec_n_fft = 128,
    spec_win_length = 24,
    spec_aug_stretch_factor = 0.8
)

text_transformer = TextTransformer(
    dim = 512,
    depth = 6,
    heads = 8,
    dim_head = 64
)

mulan = MuLaN(
    audio_transformer = audio_transformer,
    text_transformer = text_transformer
)

# 获取大量<声音, 文本>对并进行训练

wavs = torch.randn(2, 1024)
texts = torch.randint(0, 20000, (2, 256))

loss = mulan(wavs, texts)
loss.backward()

# 经过大量训练后，你可以将声音和文本嵌入到一个联合嵌入空间
# 用于条件音频LM

embeds = mulan.get_audio_latents(wavs)  # 训练期间

embeds = mulan.get_text_latents(texts)  # 推理期间

要获取作为 AudioLM 部分的三个转换器的条件嵌入，你必须使用 MuLaNEmbedQuantizer 如下

from musiclm_pytorch import MuLaNEmbedQuantizer

# 使用命名空间条件嵌入设置量化器，每个量化器和命名空间都是唯一的（每个变换器）

quantizer = MuLaNEmbedQuantizer(
    mulan = mulan,                          # 传入上面训练好的Mulan
    conditioning_dims = (1024, 1024, 1024), # 假设所有三个转换器的模型维度均为1024
    namespaces = ('semantic', 'coarse', 'fine')
)

# 现在假设你需要语义转换器的条件嵌入

wavs = torch.randn(2, 1024)
conds = quantizer(wavs = wavs, namespace = 'semantic') # (2, 8, 1024) - 8是量化器的数量

要训练（或微调）作为AudioLM部分的三个转换器，只需按照audiolm-pytorch的训练说明进行操作，但需要将MulanEmbedQuantizer实例传递给训练类中的audio_conditioner关键字

例如 SemanticTransformerTrainer

import torch
from audiolm_pytorch import HubertWithKmeans, SemanticTransformer, SemanticTransformerTrainer

wav2vec = HubertWithKmeans(
    checkpoint_path = './hubert/hubert_base_ls960.pt',
    kmeans_path = './hubert/hubert_base_ls960_L9_km500.bin'
)

semantic_transformer = SemanticTransformer(
    num_semantic_tokens = wav2vec.codebook_size,
    dim = 1024,
    depth = 6,
    audio_text_condition = True      # 该设置必须为True（CoarseTransformer和FineTransformers也一样）
).cuda()

trainer = SemanticTransformerTrainer(
    transformer = semantic_transformer,
    wav2vec = wav2vec,
    audio_conditioner = quantizer,   # 传入上面的MulanEmbedQuantizer实例
    folder ='/path/to/audio/files',
    batch_size = 1,
    data_max_length = 320 * 32,
    num_train_steps = 1
)

trainer.train()

在对所有三个转换器（语义，粗略，精细）进行大量训练后，你将把微调或从头开始训练的AudioLM和用 MuLaNEmbedQuantizer 包装的MuLaN传给MusicLM

# 你需要从上面训练好的AudioLM（audio_lm）
# 与MulanEmbedQuantizer（mulan_embed_quantizer）

from musiclm_pytorch import MusicLM

musiclm = MusicLM(
    audio_lm = audio_lm,                 # `AudioLM`来自https://github.com/lucidrains/audiolm-pytorch
    mulan_embed_quantizer = quantizer    # 上面的`MuLaNEmbedQuantizer`
)

music = musiclm('the crystalline sounds of the piano in a ballroom', num_samples = 4) # 采样4个并使用Mulan选取最佳匹配

待办事项

mulan似乎在使用解耦的对比学习，提供该选项
用mulan包装器包装mulan并量化输出，将其投射到audiolm维度
修改audiolm以接受条件嵌入，可以选择通过单独的投影处理不同的维度
audiolm和mulan进入musiclm进行生成，并用mulan过滤
给AST中的自注意力赋予动态位置偏置
实现MusicLM生成多个样本并选择与MuLan最匹配的样本
支持音频变长度，使用音频转换器中的掩模
向open clip添加一个版本的mulan
设置所有适当的声谱图超参数

引用

@inproceedings{Agostinelli2023MusicLMGM,
    title     = {MusicLM: Generating Music From Text},
    author    = {Andrea Agostinelli and Timo I. Denk and Zal{\'a}n Borsos and Jesse Engel and Mauro Verzetti and Antoine Caillon and Qingqing Huang and Aren Jansen and Adam Roberts and Marco Tagliasacchi and Matthew Sharifi and Neil Zeghidour and C. Frank},
    year      = {2023}
}

@article{Huang2022MuLanAJ,
    title   = {MuLan: A Joint Embedding of Music Audio and Natural Language},
    author  = {Qingqing Huang and Aren Jansen and Joonseok Lee and Ravi Ganti and Judith Yue Li and Daniel P. W. Ellis},
    journal = {ArXiv},
    year    = {2022},
    volume  = {abs/2208.12415}
}

@misc{https://doi.org/10.48550/arxiv.2302.01327,
    doi     = {10.48550/ARXIV.2302.01327},
    url     = {https://arxiv.org/abs/2302.01327},
    author  = {Kumar, Manoj and Dehghani, Mostafa and Houlsby, Neil},
    title   = {Dual PatchNorm},
    publisher = {arXiv},
    year    = {2023},
    copyright = {Creative Commons Attribution 4.0 International}
}

@article{Liu2022PatchDropoutEV,
    title   = {PatchDropout: Economizing Vision Transformers Using Patch Dropout},
    author  = {Yue Liu and Christos Matsoukas and Fredrik Strand and Hossein Azizpour and Kevin Smith},
    journal = {ArXiv},
    year    = {2022},
    volume  = {abs/2208.07220}
}

@misc{liu2021swin,
    title   = {Swin Transformer V2: Scaling Up Capacity and Resolution},
    author  = {Ze Liu and Han Hu and Yutong Lin and Zhuliang Yao and Zhenda Xie and Yixuan Wei and Jia Ning and Yue Cao and Zheng Zhang and Li Dong and Furu Wei and Baining Guo},
    year    = {2021},
    eprint  = {2111.09883},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV}
}

@misc{gilmer2023intriguing
    title  = {Intriguing Properties of Transformer Training Instabilities},
    author = {Justin Gilmer, Andrea Schioppa, and Jeremy Cohen},
    year   = {2023},
    status = {to be published - one attention stabilization technique is circulating within Google Brain, being used by multiple teams}
}

@inproceedings{Shukor2022EfficientVP,
    title   = {Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment},
    author  = {Mustafa Shukor and Guillaume Couairon and Matthieu Cord},
    booktitle = {British Machine Vision Conference},
    year    = {2022}
}

@inproceedings{Zhai2023SigmoidLF,
    title   = {Sigmoid Loss for Language Image Pre-Training},
    author  = {Xiaohua Zhai and Basil Mustafa and Alexander Kolesnikov and Lucas Beyer},
    year    = {2023}
}

唯一的真理是音乐。 - 杰克·凯鲁亚克

音乐是人类的通用语言。 - 亨利·沃兹沃斯·朗费罗