访问官网
Github
文档
论文
SmartSim
SmartSim is made up of two parts
- SmartSim Infrastructure Library (This repository)
- SmartRedis
The two library components are designed to work together, but can also be used independently.
SmartSim is a workflow library that makes it easier to use common Machine Learning (ML) libraries, like PyTorch and TensorFlow, in High Performance Computing (HPC) simulations and applications. SmartSim launches ML infrastructure on HPC systems alongside user workloads.
SmartRedis provides an API to connect HPC workloads, particularly (MPI + X) simulations, to the ML infrastructure, namely the The Orchestrator database, launched by SmartSim.
Applications integrated with the SmartRedis clients, written in Fortran, C, C++ and Python, can send data to and remotely request SmartSim infrastructure to execute ML models and scripts on GPU or CPU. The distributed Client-Server paradigm allows for data to be seamlessly exchanged between applications at runtime without the utilization of MPI.
Table of Contents
- SmartSim
- Quick Start
- SmartSim Infrastructure Library
- Infrastructure Library Applications
- SmartRedis
- SmartSim + SmartRedis Tutorials
- Publications
- Cite
Quick Start
The documentation has a number of tutorials that make it easy to get used to SmartSim locally before using it on your system. Each tutorial is a Jupyter notebook that can be run through the SmartSim Tutorials docker image which will run a jupyter lab with the tutorials, SmartSim, and SmartRedis installed.
docker pull ghcr.io/craylabs/smartsim-tutorials:latest
docker run -p 8888:8888 ghcr.io/craylabs/smartsim-tutorials:latest
# click on link to open jupyter lab
SmartSim Infrastructure Library
The Infrastructure Library (IL), the smartsim python package,
facilitates the launch of Machine Learning and simulation
workflows. The Python interface of the IL creates, configures, launches and monitors
applications.
Experiments
The Experiment object
is the main interface of SmartSim. Through the Experiment
users can create references to user applications called Models.
Hello World
Below is a simple example of a workflow that uses the IL to launch hello world program using the local launcher which is designed for laptops and single nodes.
from smartsim import Experiment
exp = Experiment("simple", launcher="local")
settings = exp.create_run_settings("echo", exe_args="Hello World")
model = exp.create_model("hello_world", settings)
exp.start(model, block=True)
print(exp.get_status(model))
Hello World MPI
The Experiment.create_run_settings method returns a RunSettings object which
defines how a model is launched. There are many types of RunSettings supported by
SmartSim.
RunSettingsMpirunSettingsSrunSettingsAprunSettingsJsrunSettings
The following example launches a hello world MPI program using the local launcher for single compute node, workstations and laptops.
from smartsim import Experiment
exp = Experiment("hello_world", launcher="local")
mpi_settings = exp.create_run_settings(exe="echo",
exe_args="Hello World!",
run_command="mpirun")
mpi_settings.set_tasks(4)
mpi_model = exp.create_model("hello_world", mpi_settings)
exp.start(mpi_model, block=True)
print(exp.get_status(model))
If an argument of run_command="auto" (the default) is passed to
Experiment.create_run_settings, SmartSim will attempt to find a run command on the
system with which it has a corresponding RunSettings class. If one can be found,
Experiment.create_run_settings will instance and return an object of that type.
Experiments on HPC Systems
SmartSim integrates with common HPC schedulers providing batch and interactive launch capabilities for all applications:
- Slurm
- LSF
- PBSPro
- Local (for laptops/single node, no batch)
In addition, on Slurm and PBS systems, Dragon can be used as a launcher. Please refer to the documentation for instructions on how to insall it on your system and use it in SmartSim.
Interactive Launch Example
The following launches the same hello_world model in an interactive allocation.
# get interactive allocation (Slurm)
salloc -N 3 --ntasks-per-node=20 --ntasks 60 --exclusive -t 00:10:00
# get interactive allocation (PBS)
qsub -l select=3:ncpus=20 -l walltime=00:10:00 -l place=scatter -I -q <queue>
# get interactive allocation (LSF)
bsub -Is -W 00:10 -nnodes 3 -P <project> $SHELL
This same script will run on a SLURM, PBS, or LSF system as the launcher
is set to auto in the Experiment
initialization. The run command like mpirun,
aprun or srun will be automatically detected from what is available on the
system.
# hello_world.py
from smartsim import Experiment
exp = Experiment("hello_world_exp", launcher="auto")
run = exp.create_run_settings(exe="echo", exe_args="Hello World!")
run.set_tasks(60)
run.set_tasks_per_node(20)
model = exp.create_model("hello_world", run)
exp.start(model, block=True, summary=True)
print(exp.get_status(model))
# in interactive terminal
python hello_world.py
This script could also be launched in a batch file instead of an interactive terminal. For example, for Slurm:
#!/bin/bash
#SBATCH --exclusive
#SBATCH --nodes=3
#SBATCH --ntasks-per-node=20
#SBATCH --time=00:10:00
python /path/to/hello_world.py
# on Slurm system
sbatch run_hello_world.sh
Batch Launch Examples
SmartSim can also launch workloads in a batch directly from Python, without the need
for a batch script. Users can launch groups of Model instances in a Ensemble.
The following launches 4 replicas of the the same hello_world model.
# hello_ensemble.py
from smartsim import Experiment
exp = Experiment("hello_world_batch", launcher="auto")
# define resources for all ensemble members
batch = exp.create_batch_settings(nodes=4, time="00:10:00", account="12345-Cray")
batch.set_queue("premium")
# define how each member should run
run = exp.create_run_settings(exe="echo", exe_args="Hello World!")
run.set_tasks(60)
run.set_tasks_per_node(20)
ensemble = exp.create_ensemble("hello_world",
batch_settings=batch,
run_settings=run,
replicas=4)
exp.start(ensemble, block=True, summary=True)
print(exp.get_status(ensemble))
python hello_ensemble.py
Similar to the interactive example, this same script will run on a SLURM, PBS,
or LSF system as the launcher is set to auto in the
Experiment
initialization. Local launching does not support batch workloads.
Infrastructure Library Applications
- Orchestrator - In-memory data store and Machine Learning Inference (Redis + RedisAI)
Redis + RedisAI
The Orchestrator is an in-memory database that utilizes Redis and RedisAI to provide
a distributed database and access to ML runtimes from Fortran, C, C++ and Python.
SmartSim provides classes that make it simple to launch the database in many configurations and optionally form a distributed database cluster. The examples below will show how to launch the database. Later in this document we will show how to use the database to perform ML inference and processing.
Local Launch
The following script launches a single database using the local launcher.
Experiment.create_database
will initialize an Orchestrator instance corresponding to the specified launcher.
# run_db_local.py
from smartsim import Experiment
exp = Experiment("local-db", launcher="local")
db = exp.create_database(port=6780, # database port
interface="lo") # network interface to use
# by default, SmartSim never blocks execution after the database is launched.
exp.start(db)
# launch models, analysis, training, inference sessions, etc
# that communicate with the database using the SmartRedis clients
# stop the database
exp.stop(db)
Interactive Launch
The Orchestrator, like Ensemble instances, can be launched locally, in interactive
allocations, or in a batch.
The following example launches a distributed (3 node) database cluster in an interactive allocation.
# get interactive allocation (Slurm)
salloc -N 3 --ntasks-per-node=1 --exclusive -t 00:10:00
# get interactive allocation (PBS)
qsub -l select=3:ncpus=1 -l walltime=00:10:00 -l place=scatter -I -q queue
# get interactive allocation (LSF)
bsub -Is -W 00:10 -nnodes 3 -P project $SHELL
# run_db.py
from smartsim import Experiment
# auto specified to work across launcher types
exp = Experiment("db-on-slurm", launcher="auto")
db_cluster = exp.create_database(db_nodes=3,
db_port=6780,
batch=False,
interface="ipogif0")
exp.start(db_cluster)
print(f"Orchestrator launched on nodes: {db_cluster.hosts}")
# launch models, analysis, training, inference sessions, etc
# that communicate with the database using the SmartRedis clients
exp.stop(db_cluster)
# in interactive terminal
python run_db.py
Batch Launch
The Orchestrator can also be launched in a batch without the need for an interactive allocation.
SmartSim will create the batch file, submit it to the batch system, and then wait for the database
to be launched. Users can hit CTRL-C to cancel the launch if needed.
# run_db_batch.py
from smartsim import Experiment
exp = Experiment("batch-db-on-pbs", launcher="auto")
db_cluster = exp.create_database(db_nodes=3,
db_port=6780,
batch=True,
time="00:10:00",
interface="ib0",
account="12345-Cray",
queue="cl40")
exp.start(db_cluster)
print(f"Orchestrator launched on nodes: {db_cluster.hosts}")
# launch models, analysis, training, inference sessions, etc
# that communicate with the database using the SmartRedis clients
exp.stop(db_cluster)
python run_db_batch.py
SmartRedis
The SmartSim IL Clients (SmartRedis) are implementations of Redis clients that implement the RedisAI API with additions specific to scientific workflows.
SmartRedis clients are available in Fortran, C, C++, and Python. Users can seamlessly pull and push data from the Orchestrator from different languages.
Tensors
Tensors are the fundamental data structure for the SmartRedis clients. The Clients use the native array format of the language. For example, in Python, a tensor is a NumPy array while the C/C++ clients accept nested and contiguous arrays.
When stored in the database, all tensors are stored in the same format. Hence, any language can receive a tensor from the database no matter what supported language the array was sent from. This enables applications in different languages to communicate numerical data with each other at runtime.
For more information on the tensor data structure, see the documentation
Datasets
Datasets are collections of Tensors and associated metadata. The Dataset class
is a user space object that can be created, added to, sent to, and retrieved from
the Orchestrator.
For an example of how to use the Dataset class, see the Online Analysis example
For more information on the API, see the API documentation
SmartSim + SmartRedis Tutorials
SmartSim and SmartRedis were designed to work together. When launched through SmartSim, applications using the SmartRedis clients are directly connected to any Orchestrator launched in the same Experiment.
In this way, a SmartSim Experiment becomes a driver for coupled ML and Simulation workflows. The following are simple examples of how to use SmartSim and SmartRedis together.
Run the Tutorials
Each tutorial is a Jupyter notebook that can be run through the SmartSim Tutorials docker image which will run a jupyter
