tianshou

Tianshou (天授) is a reinforcement learning platform based on pure PyTorch and Gymnasium. Unlike other reinforcement learning libraries, which may have complex codebases, unfriendly high-level APIs, or are not optimized for speed, Tianshou provides a high-performance, modularized framework and user-friendly interfaces for building deep reinforcement learning agents. One more aspect that sets Tianshou apart is its generality: it supports online and offline RL, multi-agent RL, and model-based algorithms.

Tianshou aims at enabling concise implementations, both for researchers and practitioners, without sacrificing flexibility.

Supported algorithms include:

• Deep Q-Network (DQN)
• Double DQN
• Dueling DQN
• Branching DQN
• Categorical DQN (C51)
• Rainbow DQN (Rainbow)
• Quantile Regression DQN (QRDQN)
• Implicit Quantile Network (IQN)
• Fully-parameterized Quantile Function (FQF)
• Policy Gradient (PG)
• Natural Policy Gradient (NPG)
• Advantage Actor-Critic (A2C)
• Trust Region Policy Optimization (TRPO)
• Proximal Policy Optimization (PPO)
• Deep Deterministic Policy Gradient (DDPG)
• Twin Delayed DDPG (TD3)
• Soft Actor-Critic (SAC)
• Randomized Ensembled Double Q-Learning (REDQ)
• Discrete Soft Actor-Critic (SAC-Discrete)
• Vanilla Imitation Learning
• Batch-Constrained deep Q-Learning (BCQ)
• Conservative Q-Learning (CQL)
• Twin Delayed DDPG with Behavior Cloning (TD3+BC)
• Discrete Batch-Constrained deep Q-Learning (BCQ-Discrete)
• Discrete Conservative Q-Learning (CQL-Discrete)
• Discrete Critic Regularized Regression (CRR-Discrete)
• Generative Adversarial Imitation Learning (GAIL)
• Prioritized Experience Replay (PER)
• Generalized Advantage Estimator (GAE)
• Posterior Sampling Reinforcement Learning (PSRL)
• Intrinsic Curiosity Module (ICM)
• Hindsight Experience Replay (HER)

Other noteworthy features:

• Elegant framework with dual APIs:
  • Tianshou's high-level API maximizes ease of use for application development while still retaining a high degree of flexibility.
  • The fundamental procedural API provides a maximum of flexibility for algorithm development without being overly verbose.
• State-of-the-art results in MuJoCo benchmarks for REINFORCE/A2C/TRPO/PPO/DDPG/TD3/SAC algorithms
• Support for vectorized environments (synchronous or asynchronous) for all algorithms (see usage)
• Support for super-fast vectorized environments based on EnvPool for all algorithms (see usage)
• Support for recurrent state representations in actor networks and critic networks (RNN-style training for POMDPs) (see usage)
• Support any type of environment state/action (e.g. a dict, a self-defined class, ...) Usage
• Support for customized training processes (see usage)
• Support n-step returns estimation and prioritized experience replay for all Q-learning based algorithms; GAE, nstep and PER are highly optimized thanks to numba's just-in-time compilation and vectorized numpy operations
• Support for multi-agent RL (see usage)
• Support for logging based on both TensorBoard and W&B
• Support for multi-GPU training (see usage)
• Comprehensive documentation, PEP8 code-style checking, type checking and thorough tests

In Chinese, Tianshou means divinely ordained, being derived to the gift of being born. Tianshou is a reinforcement learning platform, and the nature of RL is not learn from humans. So taking "Tianshou" means that there is no teacher to learn from, but rather to learn by oneself through constant interaction with the environment.

“天授”意指上天所授，引申为与生具有的天赋。天授是强化学习平台，而强化学习算法并不是向人类学习的，所以取“天授”意思是没有老师来教，而是自己通过跟环境不断交互来进行学习。

Installation

Tianshou is currently hosted on PyPI and conda-forge. It requires Python >= 3.11.

For installing the most recent version of Tianshou, the best way is clone the repository and install it with poetry (which you need to install on your system first)

git clone git@github.com:thu-ml/tianshou.git
cd tianshou
poetry install

You can also install the dev requirements by adding --with dev or the extras for say mujoco and acceleration by envpool by adding --extras "mujoco envpool"

If you wish to install multiple extras, ensure that you include them in a single command. Sequential calls to poetry install --extras xxx will overwrite prior installations, leaving only the last specified extras installed. Or you may install all the following extras by adding --all-extras.

Available extras are:

• atari (for Atari environments)
• box2d (for Box2D environments)
• classic_control (for classic control (discrete) environments)
• mujoco (for MuJoCo environments)
• mujoco-py (for legacy mujoco-py environments¹)
• pybullet (for pybullet environments)
• robotics (for gymnasium-robotics environments)
• vizdoom (for ViZDoom environments)
• envpool (for envpool integration)
• argparse (in order to be able to run the high level API examples)

Otherwise, you can install the latest release from PyPI (currently far behind the master) with the following command:

$ pip install tianshou

If you are using Anaconda or Miniconda, you can install Tianshou from conda-forge:

$ conda install tianshou -c conda-forge

Alternatively to the poetry install, you can also install the latest source version through GitHub:

$ pip install git+https://github.com/thu-ml/tianshou.git@master --upgrade

Finally, you may check the installation via your Python console as follows:

import tianshou
print(tianshou.__version__)

If no errors are reported, you have successfully installed Tianshou.

Documentation

Tutorials and API documentation are hosted on tianshou.readthedocs.io.

Find example scripts in the test/ and examples/ folders.

Why Tianshou?

Comprehensive Functionality

RL Platform	GitHub Stars	# of Alg. (1)	Custom Env	Batch Training	RNN Support	Nested Observation	Backend
Baselines		9	:heavy_check_mark: (gym)	:heavy_minus_sign: (2)	:heavy_check_mark:	:x:	TF1
Stable-Baselines		11	:heavy_check_mark: (gym)	:heavy_minus_sign: (2)	:heavy_check_mark:	:x:	TF1
Stable-Baselines3		7 (3)	:heavy_check_mark: (gym)	:heavy_minus_sign: (2)	:x:	:heavy_check_mark:	PyTorch
Ray/RLlib		16	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	TF/PyTorch
SpinningUp		6	:heavy_check_mark: (gym)	:heavy_minus_sign: (2)	:x:	:x:	PyTorch
Dopamine		7	:x:	:x:	:x:	:x:	TF/JAX
ACME		14	:heavy_check_mark: (dm_env)	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	TF/JAX
keras-rl		7	:heavy_check_mark: (gym)	:x:	:x:	:x:	Keras
rlpyt		11	:x:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	PyTorch
ChainerRL		18	:heavy_check_mark: (gym)	:heavy_check_mark:	:heavy_check_mark:	:x:	Chainer
Sample Factory		1 (4)