Awesome Production Machine Learning
This repository contains a curated list of awesome open source libraries that will help you deploy, monitor, version, scale and secure your production machine learning 🚀
Quick links to sections in this page
10 Min Video Overview
This 10 minute video provides an overview of the motivations for machine learning operations as well as a high level overview on some of the tools in this repo. This newer video covers the an updated 2022 version of the state of MLOps
|
|
Want to receive recurrent updates on this repo and other advancements?
You can join the Machine Learning Engineer newsletter. Join over 10,000 ML professionals and enthusiasts who receive weekly curated articles & tutorials on production Machine Learning.
|
|
Also check out the Awesome Artificial Intelligence Guidelines List, where we aim to map the landscape of "Frameworks", "Codes of Ethics", "Guidelines", "Regulations", etc related to Artificial Intelligence.
|
|
Main Content
Explaining Black Box Models and Datasets
- Aequitas - An open-source bias audit toolkit for data scientists, machine learning researchers, and policymakers to audit machine learning models for discrimination and bias, and to make informed and equitable decisions around developing and deploying predictive risk-assessment tools.
- AI Explainability 360 - Interpretability and explainability of data and machine learning models including a comprehensive set of algorithms that cover different dimensions of explanations along with proxy explainability metrics.
- AI Fairness 360 - A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models.
- Alibi - Alibi is an open source Python library aimed at machine learning model inspection and interpretation. The initial focus on the library is on black-box, instance based model explanations.
- anchor - Code for the paper "High precision model agnostic explanations", a model-agnostic system that explains the behaviour of complex models with high-precision rules called anchors.
- captum - model interpretability and understanding library for PyTorch developed by Facebook. It contains general purpose implementations of integrated gradients, saliency maps, smoothgrad, vargrad and others for PyTorch models.
- CleverHans - An adversarial example library for constructing attacks, building defenses, and benchmarking both. A python library to benchmark system's vulnerability to adversarial examples.
- DeepLIFT - Codebase that contains the methods in the paper "Learning important features through propagating activation differences". Here is the slides and the video of the 15 minute talk given at ICML.
- DeepVis Toolbox - This is the code required to run the Deep Visualization Toolbox, as well as to generate the neuron-by-neuron visualizations using regularized optimisation. The toolbox and methods are described casually here and more formally in this paper.
- ELI5 - "Explain Like I'm 5" is a Python package which helps to debug machine learning classifiers and explain their predictions.
- FACETS - Facets contains two robust visualizations to aid in understanding and analyzing machine learning datasets. Get a sense of the shape of each feature of your dataset using Facets Overview, or explore individual observations using Facets Dive.
- Fairlearn - Fairlearn is a python toolkit to assess and mitigate unfairness in machine learning models.
- FairML - FairML is a python toolbox auditing the machine learning models for bias.
- Fairness Comparison - This repository is meant to facilitate the benchmarking of fairness aware machine learning algorithms based on this paper.
- Fairness Indicators - The tool supports teams in evaluating, improving, and comparing models for fairness concerns in partnership with the broader Tensorflow toolkit.
- iNNvestigate - An open-source library for analyzing Keras models visually by methods such as DeepTaylor-Decomposition, PatternNet, Saliency Maps, and Integrated Gradients.
- Integrated-Gradients - This repository provides code for implementing integrated gradients for networks with image inputs.
- InterpretML - InterpretML is an open-source package for training interpretable models and explaining blackbox systems.
- keras-vis - keras-vis is a high-level toolkit for visualizing and debugging your trained keras neural net models. Currently supported visualizations include: Activation maximization, Saliency maps, Class activation maps.
- L2X - Code for replicating the experiments in the paper "Learning to Explain: An Information-Theoretic Perspective on Model Interpretation" at ICML 2018.
- Lightly - A python framework for self-supervised learning on images. The learned representations can be used to analyze the distribution in unlabeled data and rebalance datasets.
- Lightwood - A Pytorch based framework that breaks down machine learning problems into smaller blocks that can be glued together seamlessly with an objective to build predictive models with one line of code.
- LIME - Local Interpretable Model-agnostic Explanations for machine learning models.
- LOFO Importance - LOFO (Leave One Feature Out) Importance calculates the importances of a set of features based on a metric of choice, for a model of choice, by iteratively removing each feature from the set, and evaluating the performance of the model, with a validation scheme of choice, based on the chosen metric.
- MindsDB - MindsDB is an Explainable AutoML framework for developers. With MindsDB you can build, train and use state of the art ML models in as simple as one line of code.
- mljar-supervised - A Python package for AutoML on tabular data with feature engineering, hyper-parameters tuning, explanations and automatic documentation.
- NETRON - Viewer for neural network, deep learning and machine learning models.
- SHAP - SHapley Additive exPlanations is a unified approach to explain the output of any machine learning model.
- SHAPash - Shapash is a Python library that provides several types of visualization that display explicit labels that everyone can understand.
- tensorflow's Model Analysis - TensorFlow Model Analysis (TFMA) is a library for evaluating TensorFlow models. It allows users to evaluate their models on large amounts of data in a distributed manner, using the same metrics defined in their trainer.
- themis-ml - themis-ml is a Python library built on top of pandas and sklearn that implements fairness-aware machine learning algorithms.
- Themis - Themis is a testing-based approach for measuring discrimination in a software system.
- Transformer Debugger - Transformer Debugger (TDB) is a tool developed by OpenAI's Superalignment team with the goal of supporting investigations into specific behaviors of small language models.
- TreeInterpreter - Package for interpreting scikit-learn's decision tree and random forest predictions. Allows decomposing each prediction into bias and feature contribution components as described here.
- WhatIf - An easy-to-use interface for expanding understanding of a black-box classification or regression ML model.
- woe - Tools for WoE Transformation mostly used in ScoreCard Model for credit rating
- XAI - eXplainableAI - An eXplainability toolbox for machine learning.
Privacy Preserving ML
- BastionLab - BastionLab is a framework for confidential data science collaboration. It uses Confidential Computing, Access control data science, and Differential Privacy to enable data scientists to remotely perform data exploration, statistics, and training on confidential data while ensuring maximal privacy for data owners.
- Concrete-ML - Concrete-ML is a Privacy-Preserving Machine Learning (PPML) open-source set of tools built on top of The Concrete Framework by Zama. It aims to simplify the use of fully homomorphic encryption (FHE) for data scientists to help them automatically turn machine learning models into their homomorphic equivalent.
- Fedlearner - Fedlearner is collaborative machine learning framework that enables joint modeling of data distributed between institutions.
- FATE - FATE (Federated AI Technology Enabler) is the world's first industrial grade federated learning open source framework to enable enterprises and institutions to collaborate on data while protecting data security and privacy.
- FedML - FedML provides a research and production integrated edge-cloud platform for Federated/Distributed Machine Learning at anywhere at any scale.
- Flower - Flower is a Federated Learning Framework with a unified approach. It enables the federation of any ML workload, with any ML framework, and any programming language.
- Google's Differential Privacy - This is a C++ library of ε-differentially private algorithms, which can be used to produce aggregate statistics over numeric data sets containing private or sensitive information.
- Intel Homomorphic Encryption Backend - The Intel HE transformer for nGraph is a Homomorphic Encryption (HE) backend to the Intel nGraph Compiler, Intel's graph compiler for Artificial Neural Networks.
- Microsoft SEAL - Microsoft SEAL is an easy-to-use open-source (MIT licensed) homomorphic encryption library developed by the Cryptography Research group at Microsoft.
- OpenFL - OpenFL is a Python framework for Federated Learning. OpenFL is designed to be a flexible, extensible and easily learnable tool for data scientists. OpenFL is developed by Intel Internet of Things Group (IOTG) and Intel Labs.
- PySyft - A Python library for secure, private Deep Learning. PySyft decouples private data from model training, using Multi-Party Computation (MPC) within PyTorch.
- Rosetta - A privacy-preserving framework based on TensorFlow with customized backend Operations using Multi-Party Computation (MPC). Rosetta reuses the APIs of TensorFlow and allows to transfer original TensorFlow codes into a privacy-preserving manner with minimal changes.
- Substra - Substra is an open-source framework for privacy-preserving, traceable and collaborative Machine Learning.
- Tensorflow Privacy - A Python library that includes implementations of TensorFlow optimizers for training machine learning models with differential privacy.
- TF Encrypted - A Framework for Confidential Machine Learning on Encrypted Data in TensorFlow.
Model and Data Versioning
- Aim - A super-easy way to record, search and compare AI experiments.
- Catalyst - High-level utils for PyTorch DL & RL research. It was developed with a focus on reproducibility, fast experimentation and code/ideas reusing.
- ClearML - Auto-Magical Experiment Manager & Version Control for AI (previously Trains).
- CodaLab - CodaLab Worksheets is a collaborative platform for reproducible research that allows researchers to run, manage, and share their experiments in the cloud. It helps researchers ensure that their runs are reproducible and consistent.
- Data Version Control (DVC) - A git fork that allows for version management of models.
- Deepkit - An open-source platform and cross-platform desktop application to execute, track, and debug modern machine learning experiments.
- Dolt - Dolt is a SQL database that you can fork, clone, branch, merge, push and pull just like a git repository.
- Flor - Easy to use logger and automatic version controller made for data scientists who write ML code.
- Guild AI - Open source toolkit that automates and optimizes machine learning experiments.
- Hangar - Version control for tensor data, git-like semantics on numerical data with high speed and efficiency.
- Keepsake - Version control for machine learning.
- lakeFS - Repeatable, atomic and versioned data lake on top of object storage.
- MLflow - Open source platform to manage the ML lifecycle, including experimentation, reproducibility and deployment.
- ModelDB - An open-source system to version machine learning models including their ingredients code, data, config, and environment and to track ML metadata across the model lifecycle.
- ModelStore - An open-source Python library that allows you to version, export, and save a machine learning model to your cloud storage provider.
- ormb - Docker for Your ML/DL Models Based on OCI Artifacts.
- Polyaxon - A platform for reproducible and scalable machine learning and deep learning on kubernetes - (Video).
- Quilt - Versioning, reproducibility and deployment of data and models.
- Sacred - Tool to help you configure, organize, log and reproduce machine learning experiments.
- Studio - Model management framework which minimizes the overhead involved with scheduling, running, monitoring and managing artifacts of your machine learning experiments.
- TerminusDB - A graph database management system that stores data like git.
Model Training Orchestration
- Accelerate - Accelerate abstracts exactly and only the boilerplate code related to multi-GPU/TPU/mixed-precision and leaves the rest of your code unchanged.
- Aqueduct - Aqueduct enables you to easily define, run, and manage AI & ML tasks on any cloud infrastructure.
- CML - Continuous Machine Learning (CML) is an open-source library for implementing continuous integration & delivery (CI/CD) in machine learning projects.
- CoreNet - CoreNet is a deep neural network toolkit that allows researchers and engineers to train standard and novel small and large-scale models for variety of tasks, including foundation models (e.g., CLIP and LLM), object classification, object detection, and semantic segmentation.
- Determined - Deep learning training platform with integrated support for distributed training, hyperparameter tuning, and model management (supports Tensorflow and Pytorch).
- envd - Machine learning development environment for data science and AI/ML engineering teams.
- Fabrik - Fabrik is an online collaborative platform to build, visualize and train deep learning models via a simple drag-and-drop interface.
- Hopsworks - Hopsworks is a data-intensive platform for the design and operation of machine learning pipelines that includes a Feature Store - (Video).
- Kubeflow - A cloud-native platform for machine learning based on Google’s internal machine learning pipelines.
- MFTCoder - MFTCoder is an open-source project of CodeFuse for accurate and efficient Multi-task Fine-tuning(MFT) on Large Language Models(LLMs), especially on Code-LLMs(large language model for code tasks).
- MLeap - Standardisation of pipeline and model serialization for Spark, Tensorflow and sklearn.
- Nanotron - Nanotron provides distributed primitives to train a variety of models efficiently using 3D parallelism.
- NeMo - NVIDIA NeMo is a scalable and cloud-native generative AI framework built for researchers and PyTorch developers working on Large Language Models (LLMs), Multimodal Models (MMs), Automatic Speech Recognition (ASR), Text to Speech (TTS), and Computer Vision (CV) domains. It is designed to help you efficiently create, customize, and deploy new generative AI models by leveraging existing code and pre-trained model checkpoints.
- Nos - Nos is an open-source platform to efficiently run AI workloads on Kubernetes, increasing GPU utilization and reducing infrastructure and operational costs.
- NVIDIA TensorRT - TensorRT is a C++ library for high-performance inference on NVIDIA GPUs and deep learning accelerators.
- Onepanel - Production scale vision AI platform, with fully integrated components for model building, automated labeling, data processing and model training pipelines.
- Open Platform for AI - Platform that provides complete AI model training and resource management capabilities.
- PyCaret ) - low-code library for training and deploying models (scikit-learn, XGBoost, LightGBM, spaCy)
- Sematic - Platform to build resource-intensive pipelines with simple Python.
- Skaffold - Skaffold is a command line tool that facilitates continuous development for Kubernetes applications. You can iterate on your application source code locally then deploy to local or remote Kubernetes clusters.
- SkyPilot - Run LLMs, AI, and batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed execution -- all with a simple interface.
- Streaming - A Data Streaming Library for Efficient Neural Network Training.
- Tensorflow Extended (TFX) - Production oriented configuration framework for ML based on TensorFlow, incl. monitoring and model version management.
- TonY