FasterTransformer4CodeFuse
| English |
简介
提供高性能模型推理,主要支持蚂蚁集团的CodeFuse模型。
与原始FT相比,本仓库具有以下特点:
- :white_check_mark: CodeFuse模型的Int8量化
- :white_check_mark: 提示词结尾不需要完整单词
- :white_check_mark: Python API
- :white_check_mark: Python API的流式输出
- :white_check_mark: 更高的模型加载速度
- :white_check_mark: 一些bug修复
性能
批处理大小:1
模型 | CodeFuse 13B | |||||
测量指标 | 延迟(毫秒) | |||||
GPU | 单个A100 | 2 * A100 张量并行 | ||||
数据类型 | fp16 | int8 | fp16 | int8 | ||
输入/输出长度 | 16 | 8 | 160 | 195 | 238 | 84 |
64 | 32 | 608 | 369 | 373 | 295 | |
256 | 128 | 2650 | 1530 | 1492 | 1130 | |
1024 | 512 | 10776 | 7054 | 6786 | 5415 | |
每秒生成的令牌数 | 48 | 75 | 77 | 98 |
快速开始
我们在容器环境中运行:nvcr.io/nvidia/pytorch:22.09-py3
。
1. 安装依赖
pip install --no-cache-dir pybind11==2.6.2 transformers accelerate sentencepiece
echo "export pybind11_DIR=/opt/conda/lib/python3.8/site-packages/pybind11/share/cmake/pybind11/" >> ~/.bashrc
export pybind11_DIR=/opt/conda/lib/python3.8/site-packages/pybind11/share/cmake/pybind11/
2. 构建
mkdir build ; cd build
export TORCH_PYTHON_LIBRARIES=/opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so
cmake -DCMAKE_BUILD_TYPE=Release -DSM="80;75" -DBUILD_PYT=ON -DSPARSITY_SUPPORT=OFF -DMEASURE_BUILD_TIME=ON \
-DBUILD_CUTLASS_MIXED_GEMM=ON -DBUILD_MULTI_GPU=ON -DBUILD_TRT=OFF \
-DENABLE_FP8=OFF -DBUILD_PYBIND=ON -DTORCH_PYTHON_LIBRARIES=${TORCH_PYTHON_LIBRARIES} ..
make -j"$(grep -c ^processor /proc/cpuinfo)"
3. 运行
您可以使用examples/pytorch/codefuse/huggingface_convert.py
脚本将检查点文件从HuggingFace转换为FasterTransformer。
export MODEL_NAME=codefuse
export TENSOR_PARA_SIZE=2
python ../examples/pytorch/codefuse/huggingface_convert.py \
-o ../models/${MODEL_NAME}/fastertransformer \
-i ../models/${MODEL_NAME}/transformers \
-infer_gpu_num ${TENSOR_PARA_SIZE} \
-processes 20 \
-weight_data_type fp16 \
-model_name gptneox
您可以使用examples/pytorch/codefuse/quant_and_save.py
脚本将fp16或fp32的FasterTransformer检查点文件转换为int8文件和尺度,获得更高的模型加载速度和更小的检查点文件。
export MODEL_NAME=codefuse
export TENSOR_PARA_SIZE=2
python ../examples/pytorch/codefuse/quant_and_save.py \
--in_dir ../models/${MODEL_NAME}/fastertransformer/${TENSOR_PARA_SIZE}-gpu \
--out_dir ../models/${MODEL_NAME}/fastertransformer/${TENSOR_PARA_SIZE}-gpu_int8 \
--lib_path ../build/lib/libth_common.so \
--tensor_para_size ${TENSOR_PARA_SIZE} \
--use_gptj_residual \
--data_type fp16
您可以使用examples/pytorch/codefuse/codefuse_example.py
运行模型推理。
export MODEL_NAME=codefuse
# fp16 1gpu
python ../examples/pytorch/codefuse/codefuse_example.py \
--ckpt_path ../models/${MODEL_NAME}/fastertransformer/1-gpu \
--tokenizer_path ../models/${MODEL_NAME}/transformers
# int8 1gpu
python ../examples/pytorch/codefuse/codefuse_example.py \
--ckpt_path ../models/${MODEL_NAME}/fastertransformer/1-gpu_int8 \
--tokenizer_path ../models/${MODEL_NAME}/transformers \
--int8_mode 1 \
--enable_int8_weights 1
# fp16 2gpus
torchrun --nproc_per_node 2 ../examples/pytorch/codefuse/codefuse_example.py \
--world_size 2 \
--ckpt_path ../models/${MODEL_NAME}/fastertransformer/2-gpu \
--tokenizer_path ../models/${MODEL_NAME}/transformers
# int8 2gpus
torchrun --nproc_per_node 2 ../examples/pytorch/codefuse/codefuse_example.py \
--world_size 2 \
--ckpt_path ../models/${MODEL_NAME}/fastertransformer/2-gpu_int8 \
--tokenizer_path ../models/${MODEL_NAME}/transformers \
--int8_mode 1 \
--enable_int8_weights 1