Audio Development Tools (ADT) 🔥

This is a list of sound, audio and music development tools which contains machine learning, audio generation, audio signal processing, sound synthesis, game audio, digital audio workstation, spatial audio, music information retrieval, music generation, speech recognition, speech synthesis, singing voice synthesis and more.

Table of Contents

• Machine Learning (ML)
• Audio Generation (AG)
• Audio Signal Processing (ASP)
• Sound Synthesis (SS)
• Game Audio (GA)
• Digital Audio Workstation (DAW)
• Spatial Audio (SA)
• Web Audio Processing (WAP)
• Music Information Retrieval (MIR)
• Music Generation (MG)
• Speech Recognition (ASR)
• Speech Synthesis (TTS)
• Singing Voice Synthesis (SVS)

Project List

Machine Learning (ML)

• librosa - Librosa is a python package for music and audio analysis. It provides the building blocks necessary to create music information retrieval systems.
• Essentia - Essentia is an open-source C++ library for audio analysis and audio-based music information retrieval released under the Affero GPLv3 license. It contains an extensive collection of reusable algorithms which implement audio input/output functionality, standard digital signal processing blocks, statistical characterization of data, and a large set of spectral, temporal, tonal and high-level music descriptors. C++ library for audio and music analysis, description and synthesis, including Python bindings.
• DDSP - DDSP: Differentiable Digital Signal Processing. DDSP is a library of differentiable versions of common DSP functions (such as synthesizers, waveshapers, and filters). This allows these interpretable elements to be used as part of an deep learning model, especially as the output layers for audio generation.
• MIDI-DDSP - MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling. MIDI-DDSP is a hierarchical audio generation model for synthesizing MIDI expanded from DDSP.
• DDSP-VST - Realtime DDSP Neural Synthesizer and Effect. VST3/AU plugins and desktop applications built using the JUCE framework and DDSP.
• torchsynth - A GPU-optional modular synthesizer in pytorch, 16200x faster than realtime, for audio ML researchers.
• aubio - aubio is a tool designed for the extraction of annotations from audio signals. Its features include segmenting a sound file before each of its attacks, performing pitch detection, tapping the beat and producing midi streams from live audio.
• audioFlux - audioFlux is a deep learning tool library for audio and music analysis, feature extraction. It supports dozens of time-frequency analysis transformation methods and hundreds of corresponding time-domain and frequency-domain feature combinations. It can be provided to deep learning networks for training, and is used to study various tasks in the audio field such as Classification, Separation, Music Information Retrieval(MIR) and ASR etc.
• Polymath - Polymath uses machine learning to convert any music library (e.g from Hard-Drive or YouTube) into a music production sample-library. The tool automatically separates songs into stems (beats, bass, etc.), quantizes them to the same tempo and beat-grid (e.g. 120bpm), analyzes musical structure (e.g. verse, chorus, etc.), key (e.g C4, E3, etc.) and other infos (timbre, loudness, etc.), and converts audio to midi. The result is a searchable sample library that streamlines the workflow for music producers, DJs, and ML audio developers.
• IPython - IPython provides a rich toolkit to help you make the most of using Python interactively.
• torchaudio - an audio library for PyTorch. Data manipulation and transformation for audio signal processing, powered by PyTorch.
• TorchLibrosa - PyTorch implementation of Librosa.
• torch-audiomentations - Fast audio data augmentation in PyTorch. Inspired by audiomentations. Useful for deep learning.
• PyTorch Audio Augmentations - Audio data augmentations library for PyTorch for audio in the time-domain.
• Asteroid - Asteroid is a Pytorch-based audio source separation toolkit that enables fast experimentation on common datasets. It comes with a source code that supports a large range of datasets and architectures, and a set of recipes to reproduce some important papers.
• Kapre - Kapre: Keras Audio Preprocessors. Keras Audio Preprocessors - compute STFT, InverseSTFT, Melspectrogram, and others on GPU real-time.
• praudio - Audio preprocessing framework for Deep Learning audio applications.
• automix-toolkit - Models and datasets for training deep learning automatic mixing models.
• DeepAFx - DeepAFx: Deep Audio Effects. Audio signal processing effects (FX) are used to manipulate sound characteristics across a variety of media. Many FX, however, can be difficult or tedious to use, particularly for novice users. In our work, we aim to simplify how audio FX are used by training a machine to use FX directly and perform automatic audio production tasks. By using familiar and existing tools for processing and suggesting control parameters, we can create a unique paradigm that blends the power of AI with human creative control to empower creators.
• nnAudio - nnAudio is an audio processing toolbox using PyTorch convolutional neural network as its backend. By doing so, spectrograms can be generated from audio on-the-fly during neural network training and the Fourier kernels (e.g. or CQT kernels) can be trained.
• WavEncoder - WavEncoder is a Python library for encoding audio signals, transforms for audio augmentation, and training audio classification models with PyTorch backend.
• SciPy - SciPy (pronounced "Sigh Pie") is an open-source software for mathematics, science, and engineering. It includes modules for statistics, optimization, integration, linear algebra, Fourier transforms, signal and image processing, ODE solvers, and more.
• pyAudioAnalysis - Python Audio Analysis Library: Feature Extraction, Classification, Segmentation and Applications.
• Mutagen - Mutagen is a Python module to handle audio metadata. It supports ASF, FLAC, MP4, Monkey’s Audio, MP3, Musepack, Ogg Opus, Ogg FLAC, Ogg Speex, Ogg Theora, Ogg Vorbis, True Audio, WavPack, OptimFROG, and AIFF audio files. All versions of ID3v2 are supported, and all standard ID3v2.4 frames are parsed. It can read Xing headers to accurately calculate the bitrate and length of MP3s. ID3 and APEv2 tags can be edited regardless of audio format. It can also manipulate Ogg streams on an individual packet/page level.
• LibXtract - LibXtract is a simple, portable, lightweight library of audio feature extraction functions. The purpose of the library is to provide a relatively exhaustive set of feature extraction primatives that are designed to be 'cascaded' to create a extraction hierarchies.
• dejavu - Audio fingerprinting and recognition in Python. Dejavu can memorize audio by listening to it once and fingerprinting it. Then by playing a song and recording microphone input or reading from disk, Dejavu attempts to match the audio against the fingerprints held in the database, returning the song being played.
• Matchering - 🎚️ Open Source Audio Matching and Mastering. Matchering 2.0 is a novel Containerized Web Application and Python Library for audio matching and mastering.
• TimeSide - TimeSide is a python framework enabling low and high level audio analysis, imaging, transcoding, streaming and labelling. Its high-level API is designed to enable complex processing on very large datasets of any audio or video assets with a plug-in architecture, a secure scalable backend and an extensible dynamic web frontend.
• Meyda - Meyda is a Javascript audio feature extraction library. Meyda supports both offline feature extraction as well as real-time feature extraction using the Web Audio API. We wrote a paper about it, which is available here.
• Audiomentations - A Python library for audio data augmentation. Inspired by albumentations. Useful for deep learning. Runs on CPU. Supports mono audio and multichannel audio. Can be integrated in training pipelines in e.g. Tensorflow/Keras or Pytorch. Has helped people get world-class results in Kaggle competitions. Is used by companies making next-generation audio products.
• soundata - Python library for downloading, loading & working with sound datasets.
• auraloss - A collection of audio-focused loss functions in PyTorch.
• Neutone - AI audio plugin & community. Bridging the gap between AI research and creativity 🚀
• Waveformer - An efficient architecture for real-time target sound extraction.
• EfficientAT - Efficient Large-Scale Audio Tagging. We provide AudioSet pre-trained models ready for downstream training and extraction of audio embeddings.
• EfficientAT_HEAR - Evaluate EfficientAT models on the Holistic Evaluation of Audio Representations Benchmark.
• VAD-python - Voice Activity Detector in Python. Python code to apply voice activity detector to wave file. Voice activity detector based on ration between energy in speech band and total energy.
• Diffsynth - A Differentiable Musical Synthesizer in PyTorch.
• Realtime DDSP - Realtime (streaming) DDSP in PyTorch compatible with neutone.
• pc-ddsp - Pitch Controllable DDSP Vocoders.
• SSSSM-DDSP - Semi-supervised Synthesizer Sound Matching with Differentiable DSP.
• GOLF - A DDSP-based neural vocoder.
• audacitorch - PyTorch wrappers for using your model in audacity!
• Scyclone - Scyclone is an audio plugin that utilizes neural timbre transfer technology to offer a new approach to audio production.
• Scyclone AI - Create presets for Scyclone: a Real-time Neural Timbre Transfer Plug-in.
• Multi Task Automatic-Synthesizer-Programming - This is the code for the multi VST automatic synthesizer programming project.
• NeuralNote - Audio Plugin for Audio to MIDI transcription using deep learning.
• AudioDec - An Open-source Streaming High-fidelity Neural Audio Codec.
• PaSST - Efficient Training of Audio Transformers with Patchout.
• speech_data_augment - A summary of speech data augment algorithms.
• AugLy - A data augmentations library for audio, image, text, and video.
• NeuraFuzz - Neural audio plugin trained on custom analog fuzz circuit design.
• Ultimate Vocal Remover GUI - GUI for a Vocal Remover that uses Deep Neural Networks.
• Frechet Audio Distance - A lightweight library for Frechet Audio Distance calculation.
• LAPE - A unified framework for Low-resource Audio Processing and Evaluation (SSL Pre-training and Downstream Fine-tuning).
• Azure SDK for Python - This repository is for active development of the Azure SDK for Python.
• Panotti - A multi-channel neural network audio classifier using Keras.
• Allie - Allie is a framework for building machine learning models from audio, text, image, video, or .CSV files.
• Torchplugins - Max/MSP, PureData and Unity plugins to load Pytorch models.
• aeiou - (ML) audio engineering i/o utils.
• BirdNET-Analyzer - BirdNET analyzer for scientific audio data processing.
• spring-reverb-dl-models - Virtual Analog Modelling of the Spring Reverb with Deep Learning.
• EVAR ~ - EVAR ~ Evaluation package for Audio Representations.
• Julius - Fast PyTorch based DSP for audio and 1D signals.
• NeuralDX7 - Random machine learning experiments related to the classic Yamaha DX7.
• HANCE - HANCE offers top-quality signal-processing techniques developed by machine learning specialists, sound engineers, and audio processing experts. Our technology is designed to provide users with the highest possible audio experience by removing noise, reverb, and other signal impairments.
• IDEAW - Robust Neural Audio Watermarking with Invertible Dual-Embedding.
• SyNEThesia - SyNEThesia is a deep-learning-based music and sound visualizer, and a play of words on Synesthesia, a neurological condition where one perceives a stimulus in multiple ways (for example seeing sound).
• Voxaboxen - Voxaboxen is a deep learning framework designed to find the start and stop times of (possibly overlapping) sound events in a recording.
• vocal-separate - An extremely simple tool for separating vocals and background music, completely localized for web operation, using 2stems/4stems/5stems models.
• Speech-enhancement - Deep learning for audio denoising.
• SNAC - Multi-Scale Neural Audio Codec (SNAC) compressess 44.1 kHz audio into discrete codes at a low bitrate.
• Supervoice GPT - A GPT model that converts from text to phonemes with durations that is suitable to feed into voice synthesizer.
• AudioEditing - Zero-Shot Unsupervised and Text-Based Audio Editing Using DDPM Inversion.
• MAX-Audio-Classifier - IBM Developer Model Asset