CANN LLaDA2.0 sglang推理

LLaDA2.0模型基于sglang框架在NPU实现推理【免费下载链接】cann-recipes-infer本项目针对LLM与多模态模型推理业务中的典型模型、加速算法提供基于CANN平台的优化样例项目地址: https://gitcode.com/cann/cann-recipes-infer概述LLaDA2.0模型是2025年开源的Diffusion LLM大语言模型本样例基于sglang开源框架llada2.py完成LLaDA2.0模型在sglang框架上的的适配。支持的产品型号Atlas A2 系列产品环境准备参考sglang-npu官方教程准备conda或者docker环境(具体的版本和docker链接也可直接参考官方链接)https://docs.sglang.io/platforms/ascend_npu.html 当前版本复现docker如下也可参考上述提到的官方环境准备教程拉取镜像docker pull quay.io/ascend/sglang:main-cann8.3.rc2-910b基于基础镜像创建sglang镜像# Clone the SGLang repository git clone https://github.com/sgl-project/sglang.git cd sglang/docker # Build the docker image docker build -t image_name -f npu.Dockerfile .创建容器并执行其中{your_offline_model_path}为host上下载好的模型权重路径#!/usr/bin/env bash set -euo pipefail # 用户配置 export IMAGE${IMAGE:-quay.io/ascend/sglang:main-cann8.3.rc2-910b} export NAME${NAME:-sglang-main-cann8.3.rc2-910b} export SHM_SIZE${SHM_SIZE:-16g} export CACHE_DIR${CACHE_DIR:-$HOME/.cache} export MODEL_DIR${MODEL_DIR:-/your_offline_model_path} export NPU_VISIBLE_DEVICES${NPU_VISIBLE_DEVICES:-auto} # DEVICES() for d in /dev/davinci_manager /dev/devmm_svm /dev/hisi_hdc; do [[ -e $d ]] DEVICES(--device $d) done if [[ $NPU_VISIBLE_DEVICES auto ]]; then mapfile -t DLIST (ls -1 /dev/davinci[0-9]* 2/dev/null | sort -V) else IFS, read -ra IDS $NPU_VISIBLE_DEVICES DLIST() for i in ${IDS[]}; do [[ -e /dev/davinci${i} ]] DLIST(/dev/davinci${i}) done fi for d in ${DLIST[]}; do DEVICES(--device $d); done VISIBLE_IDS$(printf %s\n ${DLIST[]} | sed -E s#.*/davinci([0-9])#\1# | paste -sd, -) MOUNTS( -v /usr/local/sbin:/usr/local/sbin -v /usr/local/dcmi:/usr/local/dcmi -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi -v /usr/local/Ascend/driver/:/usr/local/Ascend/driver/ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware -v /etc/ascend_install.info:/etc/ascend_install.info -v /var/queue_schedule:/var/queue_schedule -v ${MODEL_DIR}:/workspace/models/ -v ${CACHE_DIR}:/root/.cache ) # 组装容器内部初始化命令注意必须是一行 CMD set -e echo 可见 NPU: $NPU_VISIBLE_DEVICES if command -v npu-smi /dev/null 21; then npu-smi info || true fi echo 容器初始化完毕已进入交互模式。 exec bash docker run --rm \ --name ${NAME} \ ${DEVICES[]} \ ${MOUNTS[]} \ -e ASCEND_VISIBLE_DEVICES0,1,2,3,4,5,6,7 \ -e NPU_VISIBLE_DEVICES0,1,2,3,4,5,6,7\ --cap-addSYS_PTRACE \ --security-opt seccompunconfined \ --ipchost \ --ulimit memlock-1 \ --ulimit stack67108864 \ --shm-size ${SHM_SIZE} \ -it ${IMAGE} bash -c ${CMD}卸载容器内的官方sglang版本pip uninstall -y sglang权重准备下载[LLaDA2.x权重]以LLaDA2.1-mini为例 (https://huggingface.co/inclusionAI/LLaDA2.1-mini) 或者 LLaDA2.0-mini https://huggingface.co/inclusionAI/LLaDA2.0-mini代码准备下载本样例所在代码仓以master分支为例git clone https://gitcode.com/cann/cann-recipes-infer.git获取sglang主仓源码并应用patchgit clone https://github.com/sgl-project/sglang.git cd sglang git reset --hard 615a02dcd45adf875de698d2ba66b1cbef161aa5 # 将修改本仓中的修改patch应用到sglang代码中 git am ../cann-recipes-infer/models/llada2.x-sglang/patches/*.patch基于patched之后的代码重装sglang.cd ./python/ cp ../../cann-recipes-infer/models/llada2.0-sglang/install_sglang_dev.sh ./ cp pyproject_npu.toml.toml pyproject.toml chmod x install_sglang_dev.sh ./install_sglang_dev.sh推理执行llada 2.1拉起服务, 如果模型文件以及下载到本地替换--model-path的参数为本地参数路径export SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT1 python -m sglang.launch_server \ --model-path inclusionAI/LLaDA2.1-mini \ --host 0.0.0.0 \ --port 8000 \ --device npu \ --attention-backend ascend \ --dtype bfloat16 \ --kv-cache-dtype auto \ --trust-remote-code \ --disable-radix-cache \ --mem-fraction-static 0.90 \ --max-running-requests 1 \ --dllm-algorithm JointThreshold \ --enable-tokenizer-batch-encode \ --skip-server-warmup \ --enable-cache-report \ --tp 1 \llada 2.0拉起服务, 如果模型文件以及下载到本地替换--model-path的参数为本地参数路径export SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT1 python -m sglang.launch_server \ --model-path inclusionAI/LLaDA2.0-mini \ --host 0.0.0.0 \ --port 8000 \ --device npu \ --attention-backend ascend \ --dtype bfloat16 \ --kv-cache-dtype auto \ --trust-remote-code \ --disable-radix-cache \ --mem-fraction-static 0.90 \ --max-running-requests 1 \ --dllm-algorithm LowConfidence \ --enable-tokenizer-batch-encode \ --skip-server-warmup \ --enable-cache-report \ --tp 1 \执行推理curl http://localhost:8000/v1/chat/completions -H Content-Type: application/json -d { model: LLaDA, messages: [ {role: user, content: (Question: Elizas rate per hour for the first 40 hours she works each week is $10. She also receives an overtime pay of 1.2 times her regular hourly rate. If Eliza worked for 45 hours this week, how much are her earnings for this week? Answer:} ], temperature: 0.0, max_tokens: 25600, ignore_eos: false }gms8k精度测试修改对应的模型和算法vim test/registered/dllm/test_llada2_mini_ascend.pyllada2.1 对应参数:cls.model inclusionAI/LLaDA2.1-mini other_args [ --trust-remote-code, --disable-radix-cache, --mem-fraction-static, 0.9, --max-running-requests, 1, --tp, 1, --attention-backend, ascend, --dllm-algorithm, JointThreshold, # TODO: Add dLLM configurations ]llada2.0 对应参数:cls.model inclusionAI/LLaDA2.0-mini other_args [ --trust-remote-code, --disable-radix-cache, --mem-fraction-static, 0.9, --max-running-requests, 1, --tp, 1, --attention-backend, ascend, --dllm-algorithm, LowConfidence, # TODO: Add dLLM configurations --dllm-algorithm-config, dllm_config.yaml, ]执行脚本python test_llada2_mini_ascend.py附录speed 以及 gms8k 精度测试llada2.1 tp1-bs1 5shotllada2.1 tp1-bs1 zero shotllada2.0 tp1-bs1 5shotllada2.0 tp2-bs1 5shot文件说明文件路径说明0001-NPU-support-DLLM-ascend-backend-on-NPU-with-LLaDA2-t.patchLLaDA2.0 sglang-ascend支持主要涉及到ascend_backend的dllm适配page_size适配默认128需要适配到dllm block size大小.0002-NPU-support-dllm-model-LLaDA2.0-graph-capture-and-re.patch支持dllm在昇腾上的入图0003-NPU-speed-up-llada2.0-on-npu-with-graph-mode.patch入图后的算子融合及优化0004-NPU-update-test_llada2_mini_ascend.py-with-graph-mod.patch更新入图模式下的benchmark脚本0005-NPU-fix-bug-in-python-sglang-srt-server_args.py.patchfix bug in python/sglang/srt/server_args.py0006-NPU-remove-some-redundent-code.patch删除一些无关代码0007-NPU-Update-dllm-llada2.0-ci-for-npu.patchUpdate dllm-llada2.0 ci for npu0008-NPU-dllm-update-format-code-with-pre-commit.patchdllm update: format code with pre-commit0009-NPU-update-ci-by-introducing-the-SGLANG_NPU_DISABLE_.patchupdate ci by introducing the SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT env parameter, to avoid oom with tp10011-Update-joint_threshold.py.patch在npu上加速update-joint threhold 算法for llada2.10012-fuse-shared-output-into-moe-groupmatmul2-to-speedup-.patchfuse shared output into moe groupmatmul2 to speedup on npu0013-Update-unquant.py.patchcache the row idx for fusedMoE with NPU inference0014-update-the-ascend-dllm-accuracy-test-config.patchupdate the accuracy test config file0015-update-the-comments-in-test_llada2_mini_ascend.py.patchupdate the commaccuracy test config file0016-revert-the-moe-gate-accuracy-to-the-model-prededfine.patchrevert the moe gate accuracy to the model predefined accuracy【免费下载链接】cann-recipes-infer本项目针对LLM与多模态模型推理业务中的典型模型、加速算法提供基于CANN平台的优化样例项目地址: https://gitcode.com/cann/cann-recipes-infer创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考