lambda 实验室现在推出 gh200 半价优惠，以让更多人习惯 arm 工具。这意味着您实际上可能有能力运行最大的开源模型！唯一需要注意的是，您有时必须从源代码构建一些东西。以下是我如何让 llama 405b 在 gh200s 上高精度运行。
创建实例
llama 405b 约为 750gb，因此您需要大约 10 个 96gb gpu 来运行它。 （gh200 具有相当好的 cpu-gpu 内存交换速度——这就是 gh200 的全部意义——所以你可以使用少至 3 个。每个令牌的时间会很糟糕，但总吞吐量是可以接受的，如果您正在执行批处理。）登录 lambda 实验室并创建一堆 gh200 实例。 确保为它们提供相同的共享网络文件系统。
将 ip 地址保存到 ~/ips.txt。
批量 ssh 连接助手
我更喜欢直接 bash 和 ssh，而不是 kubernetes 或 slurm 等任何花哨的东西。借助一些助手即可轻松管理。
# skip fingerprint confirmationfor ip in $(cat ~/ips.txt); do echo "doing $ip" ssh-keyscan $ip >> ~/.ssh/known_hostsdonefunction run_ip() { ssh -i ~/.ssh/lambda_id_ed25519 ubuntu@$ip -- stdbuf -ol -el bash -l -c "$(printf "%q" "$*")" /dev/null}function runall() { ips="$(cat ~/ips.txt)" run_ips "$@"; }function runrest() { ips="$(tail -n+2 ~/ips.txt)" run_ips "$@"; }function ssh_k() { ip=$(sed -n "$k"p ~/ips.txt) ssh -i ~/.ssh/lambda_id_ed25519 ubuntu@$ip}alias ssh_head='k=1 ssh_k'function killall() { pkill -ife '.ssh/lambda_id_ed25519' sleep 1 pkill -ife -9 '.ssh/lambda_id_ed25519' while [[ -n "$(jobs -p)" ]]; do fg || true; done}
设置 nfs 缓存
我们将把 python 环境和模型权重放在 nfs 中。如果我们缓存它，加载速度会快得多。
# first, check the nfs works.# runall ln -s my_other_fs_name sharedrunhead 'echo world > shared/hello'runall cat shared/hello# install and enable cachefilesdrunall sudo apt-get updaterunall sudo apt-get install -y cachefilesdrunall "echo 'run=yescache_tag=mycachecache_backend=path=/var/cache/fscachecachefs_reclaim=0' | sudo tee -a /etc/default/cachefilesd"runall sudo systemctl restart cachefilesdrunall 'sudo journalctl -u cachefilesd | tail -n2'# set the "fsc" option on the nfs mountrunhead cat /etc/fstab # should have mount to ~/sharedrunall cp /etc/fstab etc-fstab-bak.txtrunall sudo sed -i 's/,proto=tcp,/,proto=tcp,fsc,/g' /etc/fstabrunall cat /etc/fstab# remountrunall sudo umount /home/ubuntu/wash2runall sudo mount /home/ubuntu/wash2runall cat /proc/fs/nfsfs/volumes # fsc column should say "yes"# test cache speeduprunhead dd if=/dev/urandom of=shared/bigfile bs=1m count=8192runall dd if=shared/bigfile of=/dev/null bs=1m # first one takes 8 secondsrunall dd if=shared/bigfile of=/dev/null bs=1m # seond takes 0.6 seconds
创建conda环境
我们可以在 nfs 中使用 conda 环境，并只用头节点来控制它，而不是在每台机器上小心地执行完全相同的命令。
# we'll also use a shared script instead of changing ~/.profile directly.# easier to fix mistakes that way.runhead 'echo ". /opt/miniconda/etc/profile.d/conda.sh" >> shared/common.sh'runall 'echo "source /home/ubuntu/shared/common.sh" >> ~/.profile'runall which conda# create the environmentrunhead 'conda create --prefix ~/shared/311 -y python=3.11'runhead '~/shared/311/bin/python --version' # double-check that it is executablerunhead 'echo "conda activate ~/shared/311" >> shared/common.sh'runall which python
安装阿芙罗狄蒂依赖项
aphrodite 是 vllm 的一个分支，启动速度更快，并且有一些额外的功能。
它将运行兼容 openai 的推理 api 和模型本身。
你需要手电筒、triton 和闪光注意。
您可以从 pytorch.org 获取 aarch64 torch 构建（您不想自己构建它）。
另外两个你可以自己搭建或者使用我做的轮子。
如果您从源代码构建，那么您可以通过在三台不同的机器上并行运行 triton、flash-attention 和 aphrodite 的 python setup.py bdist_wheel 来节省一些时间。或者您可以在同一台机器上逐一执行它们。
runhead pip install 'numpy<2' torch==2.4.0 --index-url 'https://download.pytorch.org/whl/cu124'# fix for "libstdc++.so.6: version `glibcxx_3.4.30' not found" error:runhead conda install -y -c conda-forge libstdcxx-ng=12runhead python -c 'import torch; print(torch.tensor(2).cuda() + 2, "torch ok")'
来自车轮的 triton 和闪光注意
runhead pip install 'https://github.com/qpwo/lambda-gh200-llama-405b-tutorial/releases/download/v0.1/triton-3.2.0+git755d4164-cp311-cp311-linux_aarch64.whl'runhead pip install 'https://github.com/qpwo/lambda-gh200-llama-405b-tutorial/releases/download/v0.1/aphrodite_flash_attn-2.6.1.post2-cp311-cp311-linux_aarch64.whl'
海卫一从源头
k=1 ssh_k # ssh into first machinepip install -u pip setuptools wheel ninja cmake setuptools_scmgit config --global feature.manyfiles true # faster clonesgit clone https://github.com/triton-lang/triton.git ~/shared/tritoncd ~/shared/triton/pythongit checkout 755d4164 # <-- optional, tested versions# note that ninja already parallelizes everything to the extent possible,# so no sense trying to change the cmake flags or anything.python setup.py bdist_wheelpip install --no-deps dist/*.whl # good idea to download this too for laterpython -c 'import triton; print("triton ok")'
来自源头的闪光注意力
k=2 ssh_k # go into second machinegit clone https://github.com/alpindale/flash-attention ~/shared/flash-attentioncd ~/shared/flash-attentionpython setup.py bdist_wheelpip install --no-deps dist/*.whlpython -c 'import aphrodite_flash_attn; import aphrodite_flash_attn_2_cuda; print("flash attn ok")'
安装阿芙罗狄蒂
你可以使用我的轮子或自己建造。
轮子上的阿佛洛狄忒
runhead pip install 'https://github.com/qpwo/lambda-gh200-llama-405b-tutorial/releases/download/v0.1/aphrodite_engine-0.6.4.post1-cp311-cp311-linux_aarch64.whl'
阿佛洛狄忒的来源
k=3 ssh_k # building this on the third machinegit clone https://github.com/pygmalionai/aphrodite-engine.git ~/shared/aphrodite-enginecd ~/shared/aphrodite-enginepip install protobuf==3.20.2 ninja msgspec coloredlogs portalocker pytimeparse -r requirements-common.txtpython setup.py bdist_wheelpip install --no-deps dist/*.whl
检查所有安装是否成功
function runallpyc() { runall "python -c $(printf %q "$*")"; }runallpyc 'import torch; print(torch.tensor(5).cuda() + 1, "torch ok")'runallpyc 'import triton; print("triton ok")'runallpyc 'import aphrodite_flash_attn; import aphrodite_flash_attn_2_cuda; print("flash attn ok")'runallpyc 'import aphrodite; print("aphrodite ok")'runall 'aphrodite run --help | head -n1'
下载权重
转到https://huggingface.co/meta-llama/llama-3.1-405b-instruct并确保您拥有正确的权限。批准通常需要大约一个小时。从https://huggingface.co/settings/tokens
获取令牌
pip install hf_transfer 'huggingface_hub[hf_transfer]'runall git config --global credential.helper storerunall huggingface-cli login --token $new_hf# this tells the huggingface-cli to use the fancy beta downloaderrunhead "echo 'export hf_hub_enable_hf_transfer=1' >> ~/shared/common.sh"runall 'echo $hf_hub_enable_hf_transfer'runall pkill -ife huggingface-cli # kill any stragglers# we can speed up the model download by having each server download partlocal_dir=/home/ubuntu/shared/hff/405b-instructk=1 run_k huggingface-cli download --max-workers=32 --revision="main" --include="model-000[0-4]?-of-00191.safetensors" --local-dir=$local_dir meta-llama/meta-llama-3.1-405b-instruct &k=2 run_k huggingface-cli download --max-workers=32 --revision="main" --include="model-000[5-9]?-of-00191.safetensors" --local-dir=$local_dir meta-llama/meta-llama-3.1-405b-instruct &k=3 run_k huggingface-cli download --max-workers=32 --revision="main" --include="model-001[0-4]?-of-00191.safetensors" --local-dir=$local_dir meta-llama/meta-llama-3.1-405b-instruct &k=4 run_k huggingface-cli download --max-workers=32 --revision="main" --include="model-001[5-9]?-of-00191.safetensors" --local-dir=$local_dir meta-llama/meta-llama-3.1-405b-instruct &wait# download misc remaining filesk=1 run_k huggingface-cli download --max-workers=32 --revision="main" --exclude='*.pth' --local-dir=$local_dir meta-llama/meta-llama-3.1-405b-instruct
奔跑骆驼 405b
我们将通过启动 ray 让服务器相互了解。
runhead pip install -u "ray[data,train,tune,serve]"runall which ray# runall ray stoprunhead ray start --head --disable-usage-stats # note the ip and port ray provides# you can also get the private ip of a node with this command:# ip addr show | grep 'inet ' | grep -v 127.0.0.1 | awk '{print $2}' | cut -d/ -f1 | head -n 1runrest ray start --address=?.?.?.?:6379runhead ray status # should see 0.0/10.0 gpu (or however many you set up)
我们可以在一个终端选项卡中启动阿芙罗狄蒂：
# ray provides a dashboard (similar to nvidia-smi) at http://localhost:8265# 2242 has the aphrodite api.ssh -l 8265:localhost:8265 -l 2242:localhost:2242 ubuntu@$(head -n1 ~/ips.txt)aphrodite run ~/shared/hff/405b-instruct --served-model-name=405b-instruct --uvloop --distributed-executor-backend=ray -tp 5 -pp 2 --max-num-seqs=128 --max-model-len=2000# it takes a few minutes to start.# it's ready when it prints "chat api: http://localhost:2242/v1/chat/completions"
并在第二个终端中从本地计算机运行查询：
pip install openaipython -c 'import timefrom openai import openaiclient = openai(api_key="empty", base_url="http://localhost:2242/v1")started = time.time()num_tok = 0for part in client.completions.create( model="405b-instruct", prompt="live free or die. that is", temperature=0.7, n=1, max_tokens=200, stream=true,): text = part.choices[0].text or "" print(text, end="", flush=true) num_tok += 1elapsed = time.time() - startedprint()print(f"{num_tok=} {elapsed=:.2} tokens_per_sec={num_tok / elapsed:.1f}")'
THE LIFE FOR ME."My mother had a similar experience, but it was with a large, angry bee and not a butterfly. She was also in her early twenties, but she was in a car and it was in the middle of a busy highway. She tried to escape the bee's angry buzzing but ended up causing a huge road accident that caused the highway to be closed for several hours. Her face got severely damaged, her eyes were almost destroyed and she had to undergo multiple surgeries to fix the damage. Her face never looked the same after the incident. She was lucky to have survived such a traumatic experience.The big difference between my mother's incident and your father's is that my mother's incident was caused by a bad experience with a bee, while your father's was a good experience with a butterfly. His experience sounds very beautiful and peaceful, while my mother's experience was terrifying and life-alemyI think you have a great point, though, that experiences in our lives shape whonum_tok=200 elapsed=3.8e+01 tokens_per_sec=5.2
对于文本来说速度不错，但是对于代码来说有点慢。如果您连接 2 台 8xh100 服务器，那么每秒会接近 16 个令牌，但成本是原来的三倍。
进一步阅读
理论上，您可以使用 lambda labs api 编写实例创建和销毁脚本https://cloud.lambdalabs.com/api/v1/docs阿佛洛狄忒文档https://aphrodite.pygmalion.chat/vllm 文档（api 大部分相同）https://docs.vllm.ai/en/latest/
以上就是如何使用 ghs 运行 llama b bf的详细内容，更多请关注创想鸟其它相关文章！
版权声明：本文内容由互联网用户自发贡献，该文观点仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。
如发现本站有涉嫌抄袭侵权/违法违规的内容， 请发送邮件至 chuangxiangniao@163.com 举报，一经查实，本站将立刻删除。
发布者：程序猿，转转请注明出处：https://www.chuangxiangniao.com/p/1354990.html
微信扫一扫 
支付宝扫一扫 
