Debian ROCm Toolchain

HOWTO rebuld all of ROCm for the 6.0.2 release on Debian stable (12/bookworm).

The main GPU toolchain is built from AMD upstream ROCm sources.

A build of all the sources requires building LLVM (clang) and the various ROCm libraries and applications.

There is a bit of a chicken and egg problem with getting the libraries and compilers built.

AMD’s ROCm source repositories contain most of the toolchain software that needs to be built. Note, there are many equivalent packages in Debian’s main free repository, but they are older versions than needed (including in unstable sid).

The amdgpu module is in the Linux kernel source, not part of this toolchain. See “Operating System” about building the amdgpu Linux kernel module. This toolchain isn’t used to build the Linux kernel module. The stock Debian gcc is used for the kernel module.

Uninstall

Starting from a “clean slate” Perhaps blow out all the old files and start from scratch…

sudo apt purge --autoremove amd-smi-lib comgr hipcc hipcc-nvidia hsa-rocr \
  hsa-rocr-dev hsakmt-roct-dev rocblas rocblas-dev rocm-bandwidth-test \
  rocm-cmake rocm-core rocm-device-libs rocm-ocl-icd rocm-opencl-dev \
  rocm-opencl rocm-smi-lib rocminfo rocsolver rocsolver-dev llvm

LLVM

First pass of LLVM build. Doesn’t include “everything” because it needs other dependencies built with itself first.

git clone --recursive https://github.com/ROCm/llvm-project
cd llvm-project/ 
git checkout rocm-6.0.2
rm -rf build/
cmake -S llvm -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_ARCHIVE_THREADS=0 \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_DEBIAN_COMPRESSION_TYPE=xz \
  -DCPACK_DEBIAN_FILE_NAME=DEB-DEFAULT \
  -DCPACK_DEBIAN_PACKAGE_MAINTAINER="root <root@127.0.0.1>" \
  -DCPACK_PACKAGE_CONTACT="root <root@127.0.0.1>" \
  -DCPACK_SOURCE_STGZ=OFF \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DCPACK_THREADS=0 \
  -DLIBCLANG_BUILD_STATIC=ON \
  -DLLVM_BUILD_BENCHMARKS=OFF \
  -DLLVM_BUILD_TESTS=OFF \
  -DLLVM_CCACHE_BUILD=ON \
  -DLLVM_ENABLE_LLD=OFF \
  -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;compiler-rt;libclc;lld;polly;pstl" \
  -DLLVM_ENABLE_RUNTIMES="all" \
  -DLLVM_ENABLE_Z3_SOLVER=ON \
  -DLLVM_INCLUDE_BENCHMARKS=OFF \
  -DLLVM_INCLUDE_DOCS=OFF \
  -DLLVM_INCLUDE_EXAMPLES=OFF \
  -DLLVM_INSTALL_UTILS=ON \
  -DLLVM_TARGETS_TO_BUILD="X86;host;AMDGPU" \
  -DLLVM_TOOL_COMPILER_RT_BUILD=ON \
  -DLLVM_TOOL_LLDB_BUILD=OFF \
  -DCLANG_TOOL_CLANG_NVLINK_WRAPPER_BUILD=OFF \
  -DCLANG_TOOL_NVPTX_ARCH_BUILD=OFF \
  -DCOMPILER_RT_BAREMETAL_BUILD=ON \
  -DLLVM_ENABLE_WARNINGS=OFF

ninja -C build package
sudo dpkg -i build/llvm_17.0.0git_amd64.deb 
sudo apt-mark hold llvm
# LLVM_BUILD_LLVM_DYLIB
# LLVM_BUILD_EXTERNAL_COMPILER_RT
# LLVM_ENABLE_RTTI
# LLVM_OPTIMIZED_TABLEGEN
# LLVM_TOOL_LLVM_DRIVER_BUILD

hip

This just sets up the HIP headers for clr to use later, it doesn’t actually build anything.

git clone https://github.com/ROCm/HIP hip
cd hip/
git checkout rocm-6.0.2

rocm-core

Build rocm-core.

git clone https://github.com/ROCm/rocm-core
cd rocm-core/
git checkout rocm-6.0.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_VERSION=6.0.2

ninja -C build package
sudo dpkg -i build/rocm-core_6.0.2.60002-local_amd64.deb

rocm-cmake

Build rocm-cmake.

git clone https://github.com/ROCm/rocm-cmake
cd rocm-cmake/
git checkout rocm-6.0.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_DEP_ROCMCORE=OFF

ninja -C build package
sudo dpkg -i build/rocm-cmake_0.11.0-5a34e72_amd64.deb 
sudo apt-mark hold rocm-cmake

amd-smi

Build amd-smi.

git clone https://github.com/ROCm/amdsmi
cd amdsmi/
git checkout rocm-6.0.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGE_CONTACT="root <root@127.0.0.1>" \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_DIR=/opt/rocm

ninja -C build package
sudo dpkg -i build/amd-smi-lib_23.4.2.99999-local_amd64.deb \
  build/amd-smi-lib-tests_23.4.2.99999-local_amd64.deb

roct-thunk-interface

This needs a patchlet or other applications (e.g. rocminfo) won’t be able to build. Just needs a one-liner:

diff --git a/src/libhsakmt.ver b/src/libhsakmt.ver
index 15c2916..c04cefe 100644
--- a/src/libhsakmt.ver
+++ b/src/libhsakmt.ver
@@ -81,6 +81,7 @@ hsaKmtWaitOnEvent_Ext;
 hsaKmtWaitOnMultipleEvents_Ext;
 hsaKmtReplaceAsanHeaderPage;
 hsaKmtReturnAsanHeaderPage;
+hsaKmtGetAMDGPUDeviceHandle;
 
 local: *;
 };

Build roct-thunk-interface.

git clone --recursive https://github.com/ROCm/ROCT-Thunk-Interface
cd ROCT-Thunk-Interface/
git checkout rocm-6.0.2
rm -rf build
# XXX PATCH
cmake -B build -G Ninja \
  -DBUILD_SHARED_LIBS=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_CCACHE_BUILD=ON

ninja -C build package
sudo dpkg -i build/hsakmt-roct-dev_6.0.2.99999-local_amd64.deb

device-libs

Build device-libs.

Using the deprecated device-libs repository, as it is what is used for release 6.0.2. In later releases, this package is built under the llvm-project/amd directory.

git clone https://github.com/ROCm/ROCm-Device-Libs
cd ROCm-Device-Libs/
git checkout rocm-6.0.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_CCACHE_BUILD=ON \
  -DROCM_DIR=/opt/rocm

ninja -C build package
sudo dpkg -i build/rocm-device-libs_1.0.0.99999-local_amd64.deb
sudo apt-mark hold rocm-device-libs

rocr-runtime

Build rocr-runtime.

This has an option for TARGET_DEVICES. By default all targets are built. This adds a lot of time to the build for devices that won’t be used. But if they aren’t included, other packages further down the toolchain may complain, so include them all for now.

List of possible targets:

gfx700;gfx701;gfx702;gfx801;gfx802;gfx803;gfx805;gfx810;gfx900;gfx902;gfx904;gfx906;gfx908;gfx909;gfx90a;gfx90c;gfx940;gfx941;gfx942; gfx1010;gfx1011;gfx1012;gfx1013;gfx1030;gfx1031;gfx1032;gfx1033;gfx1034;gfx1035;gfx1036;gfx1100;gfx1101;gfx1102;gfx1103

The AMD Radeon 7900 XTX target is gfx1100.

git clone --recursive https://github.com/ROCm/ROCR-Runtime
cd ROCR-Runtime/src/
git checkout rocm-6.0.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_INSTALL_INCLUDEDIR=/opt/rocm/include \
  -DCMAKE_INSTALL_LIBDIR=/opt/rocm/lib \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_CCACHE_BUILD=ON

ninja -C build package
sudo dpkg -i build/hsa-rocr_1.12.0-local_amd64.deb \
  build/hsa-rocr-dev_1.12.0-local_amd64.deb

For some reason, this is installing headers to /usr/hsa instead of /opt/rocm. It is ignoring the PREFIX. Workaround…

hipcc

hipcc built under clr. This seems better.

git clone https://github.com/ROCm/HIPCC hipcc
cd hipcc/
git checkout rocm-6.0.2
sed -i -e 's/, hip-dev, rocm-core, rocm-llvm//g' CMakeLists.txt
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCMAKE_PREFIX_PATH=/opt/rocm/ \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DHIPCC_BACKWARD_COMPATIBILITY=ON

ninja -C build package
sudo dpkg -i build/hipcc_1.0.0.99999-local_amd64.deb

rocminfo

rocminfo

This ignores CMAKE_INSTALL_PREFIX=/opt/rocm and installs to /usr/bin which causes other applications to fail.

So cruftily add some symlinks at the end to appease these applications looking in the wrong spot.

git clone https://github.com/ROCm/rocminfo
cd rocminfo/
git checkout rocm-6.0.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF

ninja -C build package
sudo dpkg -i build/rocminfo_1.0.0.99999-local_amd64.deb
# XXX cruft
sudo ln -s /usr/bin/rocminfo /opt/rocm/bin/
sudo ln -s /usr/bin/rocm_agent_enumerator /opt/rocm/bin/

rocm-bandwidth-test

rocm-bandwidth-test.

git clone https://github.com/ROCm/rocm_bandwidth_test
cd rocm_bandwidth_test/
git checkout rocm-6.0.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_CXX_FLAGS="-I/opt/rocm/include/hsa" \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF

ninja -C build package
sudo dpkg -i build/rocm-bandwidth-test_1.4.0.99999-local_amd64.deb

comgr

AKA ROCm-CompilerSupport.

Build comgr.

This is another that in latest HEAD uses llvm-project/amd/ directory, but in 6.0.2 this isn’t available. So use the rocm-6.0.2 tag, like the others in this build.

The first time this is built, it has a non-fatal hip_DIR-NOTFOUND in cmake. This is because clr needs to be built. But clr can’t be build without comgr first. So build comgr, then clr, then rebuild comgr so it finds the HIP directory in the second build…

git clone https://github.com/ROCm/ROCm-CompilerSupport
cd ROCm-CompilerSupport/lib/comgr
git checkout rocm-6.0.2
rm -rf build
cmake -B build -G Ninja \
  -DAMDDeviceLibs_DIR=/usr/lib/cmake/AMDDeviceLibs \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_CCACHE_BUILD=ON \
  -DROCM_DIR=/opt/rocm \
  -Dhip_DIR=/opt/rocm/share/rocm/cmake

ninja -C build package
sudo dpkg -i build/comgr_2.6.0.99999-local_amd64.deb

clr

OpenCL and more.

Note HSAIL needs to be disabled or the build fails. -DROCCLR_ENABLE_HSAIL=OFF

git clone https://github.com/ROCm/clr
cd clr/
git checkout rocm-6.0.2
virtualenv venv
source venv/bin/activate
pip install -U setuptools pip wheel
pip install CppHeaderParser
rm -rf build
cmake -B build -G Ninja \
  -DCLR_BUILD_HIP=ON \
  -DCLR_BUILD_OCL=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DHIP_COMMON_DIR=/home/jebba/devel/ROCm/hip \
  -DCMAKE_PREFIX_PATH="/opt/rocm/" \
  -DHIP_PLATFORM=amd \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DROCM_PATH=/opt/rocm \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DHIPCC_BIN_DIR=/opt/rocm/bin \
  -DCLR_BUILD_HIP=ON \
  -DUSE_PROF_API=OFF \
  -DBUILD_TESTING=OFF \
  -DROCCLR_ENABLE_HSA=ON \
  -DROCCLR_ENABLE_HSAIL=OFF \
  -DOpenGL_GL_PREFERENCE=GLVND

ninja -C build package
sudo dpkg -i build/rocm-opencl_2.0.0-local_amd64.deb \
  build/rocm-opencl-dev_2.0.0-local_amd64.deb \
  build/rocm-ocl-icd_2.0.0-local_amd64.deb

comgr Second Pass

Re-build comgr now that clr is built, so the HIP directory is found.

The build commands are the same in both builds.

git clone https://github.com/ROCm/ROCm-CompilerSupport
cd ROCm-CompilerSupport/lib/comgr
git checkout rocm-6.0.2
rm -rf build
cmake -B build -G Ninja \
  -DAMDDeviceLibs_DIR=/usr/lib/cmake/AMDDeviceLibs \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_CCACHE_BUILD=ON \
  -DROCM_DIR=/opt/rocm \
  -Dhip_DIR=/opt/rocm/share/rocm/cmake

ninja -C build package
sudo dpkg -i build/comgr_2.6.0.99999-local_amd64.deb

comgr installs to /usr/lib. Some apps expect it elsewhere. Cruft XXX workaround:

sudo ln -s /usr/lib/libamd_comgr.so* /opt/rocm/lib/

rocBLAS

rocBLAS.

This builds for many targets, which takes ~155 minutes to build, but only gfx1100 is really needed.

The cmake build scripts download Tensile from git. Not sure if it uses a specific tag, or is a moving build…

git clone https://github.com/ROCm/rocBLAS
cd rocBLAS/
git checkout rocm-6.0.2
deactivate
rm -rf venv
virtualenv venv
source venv/bin/activate
pip install -U setuptools pip wheel
pip install -r deps/requirements.txt
sed -i -e 's/hip-rocclr >= 4.0.0//g' CMakeLists.txt
rm -rf build
cmake -B build -G Ninja \
  -DBUILD_FORTRAN_CLIENTS=OFF \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=/usr/bin/clang++ \
  -DCMAKE_C_COMPILER=/usr/bin/clang \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCMAKE_PREFIX_PATH=/opt/rocm/ \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_DEP_ROCMCORE=OFF

ninja -C build package
sudo dpkg -i build/rocblas-dev_4.0.0-88df9726~dirty_amd64.deb \
  build/rocblas_4.0.0-88df9726~dirty_amd64.deb

Other options, maybe:

-DBUILD_FORTRAN_CLIENTS=OFF
-DAMDGPU_TARGETS=gfx1100
-DTENSILE_GPU_ARCHS=gfx1100
-DTENSILE_USE_LLVM=ON
-DTensile_COMPILER=hipcc
-DBUILD_WITH_TENSILE=

XXX ? Fortran can’t be built because flang can’t be built in the LLVM build first pass. If LLVM is rebuilt with flang, then maybe Fortran can be enabled.

rocprim

rocprim.

git clone https://github.com/ROCm/rocprim
cd rocprim/
git checkout rocm-6.0.2
sed -i -e 's/hip-rocclr >= 3.5.0//g' CMakeLists.txt
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCMAKE_PREFIX_PATH=/opt/rocm/ \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF

ninja -C build package
sudo dpkg -i build/rocprim-dev_3.0.0-c8297d68~dirty_amd64.deb

rocsparse

rocsparse.

Has GPU_TARGETS option, all are built.

git clone https://github.com/ROCm/rocsparse
cd rocsparse/
git checkout rocm-6.0.2
sed -i -e 's/hip-rocclr >= 3.5.0//g' CMakeLists.txt
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCMAKE_PREFIX_PATH=/opt/rocm/ \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF

ninja -C build package
sudo dpkg -i build/rocsparse-dev_3.0.2-1c5d839c~dirty_amd64.deb \
  build/rocsparse_3.0.2-1c5d839c~dirty_amd64.deb

rocsolver

rocsolver for hipBLAS etc. This takes hour++ to build.

Has option: -DAMDGPU_TARGETS=gfx1100, but builds for all targets.

git clone https://github.com/ROCm/rocsolver
cd rocsolver/
git checkout rocm-6.0.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCMAKE_PREFIX_PATH=/opt/rocm/ \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_DEP_ROCMCORE=OFF \

ninja -C build package
sudo dpkg -i build/rocsolver-dev_3.24.0-447a52f_amd64.deb \
  build/rocsolver_3.24.0-447a52f_amd64.deb

hipBLAS

hipBLAS plz.

git clone https://github.com/ROCm/hipBLAS
cd hipBLAS/
git checkout rocm-6.0.2
rm -rf build
cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_Fortran_COMPILER_AR=/usr/bin/llvm-ari \
  -DCMAKE_Fortran_COMPILER_RANLIB=/usr/bin/llvm-ranlib \
  -DCMAKE_INSTALL_PREFIX=/opt/rocm \
  -DCMAKE_PREFIX_PATH=/opt/rocm/ \
  -DCPACK_BINARY_DEB=ON \
  -DCPACK_BINARY_STGZ=OFF \
  -DCPACK_BINARY_TGZ=OFF \
  -DCPACK_BINARY_TZ=OFF \
  -DCPACK_GENERATOR=DEB \
  -DCPACK_PACKAGING_INSTALL_PREFIX=/opt/rocm \
  -DCPACK_SOURCE_TBZ2=OFF \
  -DCPACK_SOURCE_TGZ=OFF \
  -DCPACK_SOURCE_TXZ=OFF \
  -DCPACK_SOURCE_TZ=OFF \
  -DROCM_DEP_ROCMCORE=OFF

ninja -C build package
sudo dpkg -i build/hipblas-dev_2.0.0-00c0676_amd64.deb \
  build/hipblas_2.0.0-00c0676_amd64.deb

OpenMP

OpenMP can be built as a part of LLVM, but it fails in a first pass build. It can be built (perhaps) a rebuild of LLVM with LLVm.

OpenMP repos, check rebuild.

OpenMPI

OpenMPI, not to be confused with OpenMP. They can be used independently of each other, or used together. Briefly, OpenMP parallelizes across one machine (GPUs/CPUs), and OpenMPI parallelizes across multiple machines (network). Both rebuilt for ROCm is ideal. Then applications need to be built against them.