Debian Linux Kernel

tinyrocs runs a Linux kernel.

Below is info on building a custom Linux kernel for tinyrocs under Debian. Note, this kernel doesn’t work with Ubuntu, due to EFI errors.

Linux Source

The kernel is built from AMD’s source repo, using the latest rocm-6.1.1 git tag.

Configuration

The latest kernel .config for the tinyrocs cluster is here:

Notable differences between the tinyrocs kernel and Debian:

  • Version 6.7.0.

  • Much smaller total binary size. The size of the modules plus kernel plus initrd size on latest Debian Bookworm backport kernel is 516 megs. The custom tinyrocs kernel total size is 26 megs.

  • Removed all sorts of video related options, except needed for amdgpu and the motherboard onboard AST chip.

  • Non-preemptable server kernel (e.g. not “responsive desktop”) at 100Hz.

  • Include network drivers for the Intel and Broadcom versions of the ROMED8-2T motherboards.

  • AMD Epyc kernel options (e.g. disable Intel specific CPU options).

  • Disable all sleep, hibernation, backlights, anything related to laptops.

  • No virtualization guest or host (no Xen, KVM, etc.).

  • Nearly all modules statically compiled in kernel, not as loadable modules, except for the amdgpu related bits.

  • Many security options disabled (e.g. SELinux, firewalling, crypto algorithms).

  • NUMA disabled (XXX revisit).

  • Various encryption options disabled (e.g. encryped RAM).

  • Removed a universe of hardware drivers. No wifi, SCSI, SLIP, sound, mice, etc. Only include what is needed.

  • No kexec and other options like that not needed.

  • No emulation, such as for 32-bit.

  • No hotplugging, except USB (XXX revisit).

  • Removed lots of debugging options, probing, etc.

  • Ceph filesystem in kernel.

  • Removed many filesystems, partition types.

  • Removed many networking options and protocols for other types of setups, including IPv6.

  • Includes modules for hardware sensors. Excludes sensor modules not needed.

  • Includes network drivers for various Mellanox Infiniband cards (hardware not selected yet). XXX revisit needed config after cards selected.

  • Includes USB keyboard, but not much else USB.

  • Disable audit.

  • Disable watchdogs.

Build

Build a tinyrocs kernel thusly.

Get kernel source.

At present, AMD’s ROCm kernel “fork” is used. There are endless other kernel source options.

tinyrocs needs a recent amdgpu module. The laziest way to build this is to just use AMD’s source. The kernel source can match the same tag as the rest of the toolchain as one big ROCm release from kernel to compiler to libraries.

git clone https://github.com/ROCm/ROCK-Kernel-Driver
cd ROCK-Kernel-Driver

Get git tag of latest ROCm release:

git tag -l | grep -v rc | sort -V | grep ^rocm
git checkout $(git tag -l | grep -v rc | sort -V | grep ^rocm | tail -1)

Get a .config kernel configuration file. To use a configuration from the running kernel:

cp -vp /boot/config-$(uname -r) .config

Or get the latest tinyrocs kernel config:

wget -O .config https://spacecruft.org/deepcrayon/tinyrocs/raw/branch/main/kernel/config

Further configure kernel. Increment kernel localversion.

scripts/config --disable DEBUG_INFO
make localmodconfig
make menuconfig

Build Debian package:

make -j$(nproc) deb-pkg

Install kernel .deb:

ls ../*.deb
sudo dpkg -i ../*.deb

amdgpu

Module parameters for 6.7 kernel, see here: