Debian ROCm Toolchain

HOWTO rebuld all of ROCm for the 6.1.2 release on Debian stable (12/bookworm).

The main GPU toolchain is built from AMD upstream ROCm sources.

A build of all the sources requires building LLVM (clang) and the various ROCm libraries and applications.

There is a bit of a chicken and egg problem with getting the libraries and compilers built.

AMD’s ROCm source repositories contain most of the toolchain software that needs to be built. Note, there are many equivalent packages in Debian’s main free repository, but they are older versions than needed (including in unstable sid).

The amdgpu module is in the Linux kernel source, not part of this toolchain. See “Operating System” about building the amdgpu Linux kernel module. This toolchain isn’t used to build the Linux kernel module. The stock Debian gcc is used for the kernel module.

Uninstall

Starting from a “clean slate” Perhaps blow out all the old files and start from scratch…

sudo apt purge --autoremove amd-smi-lib comgr hipcc hipcc-nvidia hsa-rocr \
  hsa-rocr-dev hsakmt-roct-dev rocblas rocblas-dev rocm-bandwidth-test \
  rocm-cmake rocm-core rocm-device-libs rocm-ocl-icd rocm-opencl-dev \
  rocm-opencl rocm-smi-lib rocminfo rocsolver rocsolver-dev llvm

LLVM

First pass of LLVM build. Doesn’t include “everything” because it needs other dependencies built with itself first.

git clone --recursive https://github.com/ROCm/llvm-project
cd llvm-project/ 
git checkout rocm-6.1.2
rm -rf build/
cmake -S llvm -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_ARCHIVE_THREADS=0 \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_DEBIAN_COMPRESSION_TYPE=xz \
  -DCPACK_DEBIAN_FILE_NAME=DEB-DEFAULT \
  -DCPACK_DEBIAN_PACKAGE_MAINTAINER="root <root@127.0.0.1>" \
  -DCPACK_PACKAGE_CONTACT="root <root@127.0.0.1>" \
  -DCPACK_SOURCE_STGZ=OFF \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DCPACK_THREADS=0 \
  -DLIBCLANG_BUILD_STATIC=ON \
  -DLLVM_BUILD_BENCHMARKS=OFF \
  -DLLVM_BUILD_TESTS=OFF \
  -DLLVM_CCACHE_BUILD=ON \
  -DLLVM_ENABLE_LLD=OFF \
  -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;compiler-rt;libclc;lld;polly;pstl" \
  -DLLVM_ENABLE_RUNTIMES="all" \
  -DLLVM_ENABLE_Z3_SOLVER=ON \
  -DLLVM_INCLUDE_BENCHMARKS=OFF \
  -DLLVM_INCLUDE_DOCS=OFF \
  -DLLVM_INCLUDE_EXAMPLES=OFF \
  -DLLVM_INSTALL_UTILS=ON \
  -DLLVM_TARGETS_TO_BUILD="X86;host;AMDGPU" \
  -DLLVM_TOOL_COMPILER_RT_BUILD=ON \
  -DLLVM_TOOL_LLDB_BUILD=OFF \
  -DCLANG_TOOL_CLANG_NVLINK_WRAPPER_BUILD=OFF \
  -DCLANG_TOOL_NVPTX_ARCH_BUILD=OFF \
  -DCOMPILER_RT_BAREMETAL_BUILD=ON \
  -DLLVM_ENABLE_WARNINGS=OFF

ninja -C build package
sudo dpkg -i build/llvm_17.0.0git_amd64.deb 
sudo apt-mark hold llvm

hip

This just sets up the HIP headers for clr to use later, it doesn’t actually build anything.

git clone https://github.com/ROCm/HIP hip
cd hip/
git checkout rocm-6.1.2

rocm-core

Build rocm-core.

git clone https://github.com/ROCm/rocm-core
cd rocm-core/
git checkout rocm-6.1.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_VERSION=6.1.2

ninja -C build package
sudo dpkg -i build/rocm-core_6.1.2.60102-local_amd64.deb

rocm-cmake

Build rocm-cmake.

git clone https://github.com/ROCm/rocm-cmake
cd rocm-cmake/
git checkout rocm-6.1.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_DEP_ROCMCORE=OFF

ninja -C build package
sudo dpkg -i build/rocm-cmake_0.12.0-f6fcfe7_amd64.deb
sudo apt-mark hold rocm-cmake

amd-smi

Build amd-smi.

git clone https://github.com/ROCm/amdsmi
cd amdsmi/
git checkout rocm-6.1.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGE_CONTACT="root <root@127.0.0.1>" \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_DIR=/opt/rocm

ninja -C build package
sudo dpkg -i build/amd-smi-lib_24.5.2.99999-local_amd64.deb \
  build/amd-smi-lib-tests_24.5.2.99999-local_amd64.deb

roct-thunk-interface

Build roct-thunk-interface.

git clone --recursive https://github.com/ROCm/ROCT-Thunk-Interface
cd ROCT-Thunk-Interface/
git checkout rocm-6.1.2
rm -rf build
# XXX PATCH
cmake -B build -G Ninja \
  -DBUILD_SHARED_LIBS=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_CCACHE_BUILD=ON

ninja -C build package
sudo dpkg -i build/hsakmt-roct-dev_6.1.2.99999-local_amd64.deb

rocm-device-libs

Build rocm-device-libs.

Using the deprecated rocm-device-libs repository, as it is what is used for release 6.1.2.

git clone https://github.com/ROCm/ROCm-Device-Libs
cd ROCm-Device-Libs/
git checkout rocm-6.1.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_CCACHE_BUILD=ON \
  -DROCM_DIR=/opt/rocm

ninja -C build package
sudo dpkg -i build/rocm-device-libs_1.0.0.99999-local_amd64.deb
sudo apt-mark hold rocm-device-libs

In later releases, this package may be built under the llvm-project/amd directory.

rocr-runtime

Build rocr-runtime.

This has an option for TARGET_DEVICES. By default all targets are built. This adds a lot of time to the build for devices that won’t be used. But if they aren’t included, other packages further down the toolchain may complain, so include them all for now.

List of possible targets:

gfx700;gfx701;gfx702;gfx801;gfx802;gfx803;gfx805;gfx810;gfx900;gfx902;gfx904;gfx906;gfx908;gfx909;gfx90a;gfx90c;gfx940;gfx941;gfx942; gfx1010;gfx1011;gfx1012;gfx1013;gfx1030;gfx1031;gfx1032;gfx1033;gfx1034;gfx1035;gfx1036;gfx1100;gfx1101;gfx1102;gfx1103

The AMD Radeon 7900 XTX target is gfx1100.

git clone --recursive https://github.com/ROCm/ROCR-Runtime
cd ROCR-Runtime/src/
git checkout rocm-6.1.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_INSTALL_INCLUDEDIR=/opt/rocm/include \
  -DCMAKE_INSTALL_LIBDIR=/opt/rocm/lib \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_CCACHE_BUILD=ON

ninja -C build package
sudo dpkg -i build/hsa-rocr_1.13.0-local_amd64.deb \
  build/hsa-rocr-dev_1.13.0-local_amd64.deb

For some reason, this is installing headers to /usr/hsa instead of /opt/rocm. It is ignoring the PREFIX. Workaround…

hipcc

hipcc per upstream:

HIPCC has moved! This project is now located in the AMD Fork of the LLVM Project, under the “amd/hipcc” directory. This repository is now read-only.

So use the llvm-project repo built above to build hipcc…

cd llvm-project/amd/hipcc
sed -i -e 's/, hip-dev, rocm-core, rocm-llvm//g' CMakeLists.txt
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_C_COMPILER=clang \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_DIR=/opt/rocm

ninja -C build package
sudo dpkg -i build/hipcc_1.0.0.99999-local_amd64.deb
# It won't honor install paths, so cruft:
sudo ln -s /usr/hip/bin/hip* /opt/rocm/bin/

rocminfo

rocminfo

This ignores CMAKE_INSTALL_PREFIX=/opt/rocm and installs to /usr/bin which causes other applications to fail.

So cruftily add some symlinks at the end to appease these applications looking in the wrong spot.

git clone https://github.com/ROCm/rocminfo
cd rocminfo/
git checkout rocm-6.1.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF

ninja -C build package
sudo dpkg -i build/rocminfo_1.0.0.99999-local_amd64.deb
# XXX cruft
sudo ln -s /usr/bin/rocminfo /opt/rocm/bin/
sudo ln -s /usr/bin/rocm_agent_enumerator /opt/rocm/bin/

rocm-bandwidth-test

rocm-bandwidth-test.

git clone https://github.com/ROCm/rocm_bandwidth_test
cd rocm_bandwidth_test/
git checkout rocm-6.1.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_CXX_FLAGS="-I/opt/rocm/include/hsa" \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF

ninja -C build package
sudo dpkg -i build/rocm-bandwidth-test_1.4.0.99999-local_amd64.deb

comgr

AKA ROCm-CompilerSupport.

Build comgr.

This is another that in latest HEAD uses llvm-project/amd/ directory, similar to hipcc.

cd llvm-project/amd/comgr
rm -rf build
cmake -B build -G Ninja \
  -DAMDDeviceLibs_DIR=/usr/lib/cmake/AMDDeviceLibs \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_CCACHE_BUILD=ON \
  -DROCM_DIR=/opt/rocm \
  -Dhip_DIR=/home/jebba/devel/ROCm/hip
  #-Dhip_DIR=/opt/rocm/share/rocm/cmake

ninja -C build package
sudo dpkg -i build/comgr_2.7.0.99999-local_amd64.deb

clr

OpenCL and more.

git clone https://github.com/ROCm/clr
cd clr/
git checkout rocm-6.1.2
virtualenv venv
source venv/bin/activate
pip install -U setuptools pip wheel
pip install CppHeaderParser
rm -rf build
cmake -B build -G Ninja \
  -DCLR_BUILD_HIP=ON \
  -DCLR_BUILD_OCL=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DHIP_COMMON_DIR=/home/jebba/devel/ROCm/hip \
  -DCMAKE_PREFIX_PATH="/opt/rocm/" \
  -DHIP_PLATFORM=amd \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DROCM_PATH=/opt/rocm \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DHIPCC_BIN_DIR=/usr/bin \
  -DCLR_BUILD_HIP=ON \
  -DUSE_PROF_API=OFF \
  -DBUILD_TESTING=OFF \
  -DROCCLR_ENABLE_HSA=ON \
  -DROCCLR_ENABLE_HSAIL=OFF \
  -DOpenGL_GL_PREFERENCE=GLVND

ninja -C build package
sudo dpkg -i build/rocm-opencl_2.0.0-local_amd64.deb \
  build/rocm-opencl-dev_2.0.0-local_amd64.deb \
  build/rocm-ocl-icd_2.0.0-local_amd64.deb

rocBLAS

rocBLAS.

Fails to build for rocm-6.1.2. Tensile error.

This builds for many targets, which takes ~155 minutes to build, but only gfx1100 is really needed.

The cmake build scripts download Tensile from git. Not sure if it uses a specific tag, or is a moving build…

git clone https://github.com/ROCm/rocBLAS
cd rocBLAS/
# Broken Tensile:
git checkout rocm-6.1.2
# Broken Tensile:
#git checkout remotes/origin/release/rocm-rel-6.1
# test
#git checkout develop
deactivate
rm -rf venv
virtualenv venv
source venv/bin/activate
pip install -U setuptools pip wheel
pip install -r deps/requirements.txt
sed -i -e 's/hip-rocclr >= 4.0.0//g' CMakeLists.txt
rm -rf build
cmake -B build -G Ninja \
  -DBUILD_FORTRAN_CLIENTS=OFF \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=/usr/bin/clang++ \
  -DCMAKE_C_COMPILER=/usr/bin/clang \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCMAKE_PREFIX_PATH=/opt/rocm/ \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_DEP_ROCMCORE=OFF \
  -DBUILD_WITH_TENSILE=OFF
  -DTensile_COMPILER=hipcc 


ninja -C build package
sudo dpkg -i build/rocblas-dev_4.1.2-84435396~dirty_amd64.deb \
  build/rocblas_4.1.2-84435396~dirty_amd64.deb

# worked in 6.0.2, tensile fails in 6.1.2
  -DTENSILE_USE_LLVM=ON \
  -DTensile_COMPILER=hipcc 

# XXX fails, with, without
  -DAMDGPU_TARGETS=gfx1100 \
  -DTENSILE_GPU_ARCHS=gfx1100
  -DTENSILE_USE_HIP=OFF \
  -DTENSILE_USE_LLVM=ON

# last resort, if it won't build:
  -DBUILD_WITH_TENSILE=OFF
  -DTensile_COMPILER= ?

Other options, maybe:

-DBUILD_FORTRAN_CLIENTS=OFF
-DAMDGPU_TARGETS=gfx1100
-DTENSILE_GPU_ARCHS=gfx1100
-DTENSILE_USE_LLVM=ON
-DTensile_COMPILER=hipcc
-DBUILD_WITH_TENSILE=

XXX ? Fortran can’t be built because flang can’t be built in the LLVM build first pass. If LLVM is rebuilt with flang, then maybe Fortran can be enabled.

rocprim

rocprim.

git clone https://github.com/ROCm/rocprim
cd rocprim/
git checkout rocm-6.1.2
sed -i -e 's/hip-rocclr >= 3.5.0//g' CMakeLists.txt
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCMAKE_PREFIX_PATH=/opt/rocm/ \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF

ninja -C build package
sudo dpkg -i build/rocprim-dev_3.1.0-85253f87~dirty_amd64.deb

rocsparse

rocsparse.

Has GPU_TARGETS option, all are built.

git clone https://github.com/ROCm/rocsparse
cd rocsparse/
git checkout rocm-6.1.2
sed -i -e 's/hip-rocclr >= 3.5.0//g' CMakeLists.txt
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCMAKE_PREFIX_PATH=/opt/rocm/ \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF

ninja -C build package
sudo dpkg -i build/rocsparse-dev_3.1.2-edb27708~dirty_amd64.deb  \
  build/rocsparse_3.1.2-edb27708~dirty_amd64.deb

rocsolver

rocsolver for hipBLAS etc.

Fails because rocBLAS fails above.

This takes hour++ to build.

Has option: -DAMDGPU_TARGETS=gfx1100, but builds for all targets.

git clone https://github.com/ROCm/rocsolver
cd rocsolver/
git checkout rocm-6.1.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCMAKE_PREFIX_PATH=/opt/rocm/ \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_DEP_ROCMCORE=OFF \

ninja -C build package
sudo dpkg -i build/rocsolver_3.25.0-dd38229_amd64.deb \
  build/rocsolver-dev_3.25.0-dd38229_amd64.deb

hipBLAS

hipBLAS plz.

git clone https://github.com/ROCm/hipBLAS
cd hipBLAS/
git checkout rocm-6.1.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_Fortran_COMPILER_AR=/usr/bin/llvm-ari \
  -DCMAKE_Fortran_COMPILER_RANLIB=/usr/bin/llvm-ranlib \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCMAKE_PREFIX_PATH=/opt/rocm/ \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_DEP_ROCMCORE=OFF

ninja -C build package
sudo dpkg -i build/hipblas_2.1.0-8ca4bff_amd64.deb \
  build/hipblas-dev_2.1.0-8ca4bff_amd64.deb 

rocm_smi_lib

rocm_smi_lib plz.

rocRAND

rocRAND.

git clone https://github.com/ROCm/rocrand
cd rocrand/
git checkout rocm-6.1.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCMAKE_PREFIX_PATH=/opt/rocm/ \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF

ninja -C build package
sudo dpkg -i build/rocrand_3.0.1-77858b5_amd64.deb \
  build/rocrand-dev_3.0.1-77858b5_amd64.deb

OpenMPI

OpenMPI, not to be confused with OpenMP. They can be used independently of each other, or used together. Briefly, OpenMP parallelizes across one machine (GPUs/CPUs), and OpenMPI parallelizes across multiple machines (network). Both rebuilt for ROCm is ideal. Then applications need to be built against them.

No OpenMPI build yet…

Check Debian.

OpenMP

AMD’s OpenMP is called AOMP.

This build hasn’t been completed yet.

OpenMP can be built as a part of LLVM, but it fails in a first pass build. It can be built (perhaps) a rebuild of LLVM with LLVM.

OpenMP repos, check rebuild.

# sudo apt install cli-common-dev
git clone https://github.com/ROCm/aomp
cd aomp/
git checkout rocm-6.1.2
python -m venv venv
source venv/bin/activate
pip install -U setuptools pip wheel
# pip install CppHeaderParser argparse lit lxml barectf termcolor pandas
pip install CppHeaderParser argparse lit lxml termcolor pandas
pip install git+https://github.com/efficios/barectf
rm -rf build
./bin/build_aomp.sh

exit
# Using upstream build scripts. Fails.
export AOMP_VERSION=18.0
export AOMP_REPOS=$HOME/devel/ROCm/aomp${AOMP_VERSION}
export MAKEFLAGS="-j128"
mkdir -p $AOMP_REPOS
cd $AOMP_REPOS
git clone -b aomp-18.0-1 https://github.com/ROCm-Developer-Tools/aomp
$AOMP_REPOS/aomp/bin/clone_aomp.sh
$AOMP_REPOS/aomp/bin/build_prereq.sh
nohup $AOMP_REPOS/aomp/bin/build_aomp.sh &

rocALUTION

rocALUTION. Fails to build.

Needs rocrand.

Uses OpenMP, but can be built without it. # OpenMP not found. Compiling WITHOUT OpenMP

Builds using Debian’s libopenmpi-dev, but does that need recompiling with ROCm toolchain ? XXX

git clone https://github.com/ROCm/rocalution
cd rocalution/
git checkout rocm-6.1.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCMAKE_PREFIX_PATH=/opt/rocm/ \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DBUILD_CLIENTS_SAMPLES=OFF \
  -DSUPPORT_MPI=OFF \
  -DSUPPORT_OMP=OFF \
  -DCMAKE_MODULE_PATH="/opt/rocm/lib/cmake/hip" \
  -DROCM_PATH=/opt/rocm

ninja -C build package
sudo dpkg -i build/rocalution-dev_3.1.0-85253f87~dirty_amd64.deb