Abstract

Behold! A repeatable procedure for openSUSE Leap 16 with AMD Instinct MI100 “Arcturus” CDNA 1.0 (gfx908) accelerators. AMD does not officially list Leap in the ROCm compatibility matrix – only SLES. This guide uses the SLES 16 package repository, which matches Leap 16’s kernel build, and keeps AMD’s own driver package out of the transaction entirely.

This procedure works because Leap 16 is now built from the same package base as SLE 16, and the running kernel here (6.12.x) matches the 6.12 kernel AMD tests SLES 16.0 against. This as an unsupported-but-pragmatic path, because I’ll do what I want.

For hypervisor PCIe passthrough systems, we’ll also tackle the “AMD reset bug” – where some older AMD GPUs (consumer Vega/Navi, and Instinct cards like the MI100 built on the same silicon) don’t implement a real PCIe FLR or bus reset, so a VM using PCI passthrough leaves the card in whatever state the guest driver left it after shutdown. On Proxmox, this shows up on the next VM start: vfio-pci’s reset call fails (Inappropriate ioctl for device), and QEMU hands the guest a half-initialized GPU with garbage config space (invalid PCI interrupt pin 255). vendor-reset fixes it by hard-coding AMD’s actual internal reset sequence per GPU generation, so the card comes back clean on every VM start instead of needing a full host power cycle.

Be advised that I jump around logically. These guides are my progress notes, and I often have to go back and re-write whole sections as a result of failures or sub-optimal results. Pick and choose what matters to you, but I recommend you go the intellectually curious route and play the whole game <3

Table of Contents

Abstract
Proxmox VE Hypervisor Preparation
- PCIe Passthrough / VFIO
- Vendor Reset Bug
Install and Ready the Operating Environment
- Install Prerequisites and Add AMD’s ROCm Repository
Install the ROCm Userspace Stack
- Group Membership and PATH
- Verify GPU(s) are Visible
Helpful Links

Proxmox VE Hypervisor Preparation

Obviously you can skip this part if you’re on bare metal. If you’re using an inferior hypervisor, you can figure things out.

PCIe Passthrough / VFIO

PVE just doesn’t want to let go of devices. If left unaltered, it will claim all your GPUs – which makes them kinda hard to use in VMs. So let’s tell it what we want. What we really, really want.

Enable these in BIOS:

SVM (AMD CPU) or VT-x (Intel CPU) virtualization
IOMMU AMD-Vi or VT-d (Intel CPU)
Above 4G Decoding
SR-IOV, if present
UEFI boot / CSM disabled, ideally

Create a VFIO modules-load file:

root@deimos:~# cat > /etc/modules-load.d/vfio.conf <<'EOF'
vfio
vfio_iommu_type1
vfio_pci
EOF

Get Vendor IDs (substitute your model in place if not the MI100):

root@deimos:~# lspci -nnD | grep MI100
0000:45:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] Arcturus GL-XL [Instinct MI100] [1002:738c] (rev 01)
0000:48:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] Arcturus GL-XL [Instinct MI100] [1002:738c] (rev 01)
0000:c5:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] Arcturus GL-XL [Instinct MI100] [1002:738c] (rev 01)
0000:c8:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] Arcturus GL-XL [Instinct MI100] [1002:738c] (rev 01)

Add VFIO / Vendor IDs (replace the IDs with yours if different):

root@deimos:~# cat > /etc/modprobe.d/vfio-pci.conf <<'EOF'
options vfio-pci ids=1002:738c disable_vga=1
softdep amdgpu pre: vfio-pci
softdep radeon pre: vfio-pci
EOF

Blacklist host GPU drivers for passthrough cards:

root@deimos:~# cat > /etc/modprobe.d/blacklist-gpu-passthrough.conf <<'EOF'
blacklist amdgpu
blacklist radeon
EOF

We have to add a little something to /etc/default/grub to get IOMMU working.

For AMD CPU systems, this one liner does the trick:

root@deimos:~# sed -i.bak -E 's/^(GRUB_CMDLINE_LINUX_DEFAULT=")([^"]*)(")$/\1\2 iommu=pt\3/' /etc/default/grub

For Intel CPU system, do this:

root@deimos:~# sed -i.bak -E 's/^(GRUB_CMDLINE_LINUX_DEFAULT=")([^"]*)(")$/\1\2 intel_iommu=on iommu=pt\3/' /etc/default/grub

NOTE: Intel’s IOMMU grouping is often worse than AMD’s, so an extra NIC or USB controller sometimes rides along with the GPU in the same group. If your VM won’t start with vfio: device already attached that’s why.

Next, update grub and reboot!:

root@deimos:~# update-grub
root@deimos:~# update-initramfs -u -k all
root@deimos:~# reboot

Vendor Reset Bug

The MI100 (Arcturus), as well as others from the same silicon era, don’t implement the standard PCIe FLR, so vfio-pci‘s generic reset path fails when a virtual machine utilizing these cards via PCIe passthrough is rebooted. The device is left with garbage in its config space (with restart errors like “invalid PCI interrupt pin 255”), until something performs AMD’s actual vendor-specific reset sequence. That usually involves a full host power reset. But that is quite inconvenient. So somebody glue together a fix. Well, a patch. Whatever.

vendor-reset is the standard fix – a kernel module that implements the correct reset sequence for GPUs that lack real FLR support.

If you’ve already experienced the error after rebooting the guest and are staring at it now, you’ll need to power cycle the host first (not just reboot). If the GPUs are already stuck from the failed attempt, vendor-reset can’t recover a device that’s mid-broken; it needs to take over from a clean state.

Now that’s out of the way, let’s do some repairs.

Install vendor-reset:

root@deimos:~# apt update
root@deimos:~# apt install -y git dkms build-essential pve-headers-$(uname -r)
root@deimos:~# git clone https://github.com/gnif/vendor-reset.git
root@deimos:~# cd vendor-reset
root@deimos:~/vendor-reset# dkms install .
root@deimos:~/vendor-reset# echo vendor-reset >> /etc/modules
root@deimos:~/vendor-reset# update-initramfs -u -k all
root@deimos:~/vendor-reset# reboot

NOTE: vendor-reset has to load before vfio-pci claims the GPUs, or the kernel’s default (failing) reset path runs first and can wedge the device in a way vendor-reset can’t fix. Loading it via /etc/modules + regenerating initramfs ensures it’s available early in boot.

Verify after reboot…

lsmod | grep vendor_reset
dmesg | grep -i vendor-reset      # should show it claiming the Arcturus PCI IDs
lspci -vv -s 45:00.0 | grep FLReset

Install and Ready the Operating Environment

Do a “Server” installation of openSUSE Leap 16. Pretty simple. From there, the adventure begins.

First, we need to verify that the GPU(s) are properly detected. I have four of these specific cards – your quantity and exact model will probably vary):

beaker@HAL9000:~> lspci -Dnnk | grep -A3 MI100
0000:01:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] Arcturus GL-XL [Instinct MI100] [1002:738c] (rev 01)
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0c34]
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
0000:02:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] Arcturus GL-XL [Instinct MI100] [1002:738c] (rev 01)
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0c34]
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
0000:03:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] Arcturus GL-XL [Instinct MI100] [1002:738c] (rev 01)
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0c34]
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
0000:04:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] Arcturus GL-XL [Instinct MI100] [1002:738c] (rev 01)
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0c34]
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu

beaker@HAL9000:~> ls -la /dev/kfd /dev/dri
crw-rw----. 1 root render 239, 0 Sep  4 01:38 /dev/kfd

/dev/dri:
total 0
drwxr-xr-x.  3 root root        240 Sep  4 01:38 .
drwxr-xr-x. 18 root root       3780 Sep  4 01:38 ..
drwxr-xr-x.  2 root root        220 Sep  4 01:38 by-path
crw-rw----.  1 root video  226,   0 Sep  4 01:38 card0
crw-rw----.  1 root video  226,   1 Sep  4 01:38 card1
crw-rw----.  1 root video  226,   2 Sep  4 01:38 card2
crw-rw----.  1 root video  226,   3 Sep  4 01:38 card3
crw-rw----.  1 root video  226,   4 Sep  4 01:38 card4
crw-rw----.  1 root render 226, 128 Sep  4 01:38 renderD128
crw-rw----.  1 root render 226, 129 Sep  4 01:38 renderD129
crw-rw----.  1 root render 226, 130 Sep  4 01:38 renderD130
crw-rw----.  1 root render 226, 131 Sep  4 01:38 renderD131

Install Prerequisites and Add AMD’s ROCm Repository

beaker@SAL9000:~> sudo zypper install -y wget libatomic1 libquadmath0 gcc gcc-c++

We’re using the SLES 16.0 package set for this effort:

beaker@HAL9000:~> sudo tee /etc/zypp/repos.d/amdrocm-stable.repo >/dev/null <<'EOF'
> [amdrocm-stable]
> name=ROCm 10.0.0
> baseurl=https://stable.repo.amd.com/rocm/core/packages/sles16/x86_64
> enabled=1
> gpgcheck=1
> gpgkey=https://stable.repo.amd.com/rocm/gpg/packages.gpg
> EOF

beaker@HAL9000:~> sudo zypper --gpg-auto-import-keys refresh amdrocm-stable

Install the ROCm Userspace Stack

A lot of bits are going to be pulled in here. Don’t be alarmed:

beaker@HAL9000:~> sudo zypper install -y amdrocm10.0 amdrocm-core-devel10.0 amdrocm-developer-tools10.0

What you’re actually getting:

amdrocm10.0: HIP runtime, core libraries, system tools
amdrocm-core-devel10.0: compilers, CMake config, headers, static libs
amdrocm-developer-tools10.0: rocm-smi, profiling tools

Group Membership and PATH

beaker@SAL9000:~> sudo usermod -a -G render,video $LOGNAME

# ROCm installs versioned under /opt/rocm-10.0.0 with an /opt/rocm symlink

beaker@SAL9000:~> echo 'export PATH=/opt/rocm/bin:$PATH' | sudo tee /etc/profile.d/rocm.sh
beaker@SAL9000:~> sudo chmod +x /etc/profile.d/rocm.sh

Group membership only takes effect in new login sessions. Either start a fresh shell with newgrp render, or fully log out and back in. Otherwise rocminfo will fail to open /dev/kfd with a permissions error even though everything installed correctly.

Verify GPU(s) are Visible

Pay attention to the “Name: ” listed for your GPU model. That is the ROCm LLVM Target Name. Mine is gfx908 as shown here. This is what you’ll feed tools like llama.cpp during compilation so they get tuned for your specific hardware.

beaker@HAL9000:~> rocminfo | grep -E 'Name:|Marketing'
  Name:                    AMD EPYC 7K62 48-Core Processor    
  Marketing Name:          AMD EPYC 7K62 48-Core Processor    
  Vendor Name:             CPU                                
  Name:                    gfx908                             
  Marketing Name:          AMD Instinct MI100                 
  Vendor Name:             AMD                                
      Name:                    amdgcn-amd-amdhsa--gfx908:sramecc+:xnack-
  Name:                    gfx908                             
  Marketing Name:          AMD Instinct MI100                 
  Vendor Name:             AMD                                
      Name:                    amdgcn-amd-amdhsa--gfx908:sramecc+:xnack-
  Name:                    gfx908                             
  Marketing Name:          AMD Instinct MI100                 
  Vendor Name:             AMD                                
      Name:                    amdgcn-amd-amdhsa--gfx908:sramecc+:xnack-
  Name:                    gfx908                             
  Marketing Name:          AMD Instinct MI100                 
  Vendor Name:             AMD                                
      Name:                    amdgcn-amd-amdhsa--gfx908:sramecc+:xnack-

beaker@HAL9000:~> rocm-smi 


========================================= ROCm System Management Interface =========================================
=================================================== Concise Info ===================================================
Device  Node  IDs              Temp    Power  Partitions          SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
              (DID,     GUID)  (Edge)  (Avg)  (Mem, Compute, ID)                                                   
====================================================================================================================
0       1     0x738c,   29067  34.0°C  39.0W  N/A, N/A, 0         300Mhz  1200Mhz  0%   auto  290.0W  0%     0%    
1       2     0x738c,   32971  34.0°C  34.0W  N/A, N/A, 0         300Mhz  1200Mhz  0%   auto  290.0W  0%     0%    
2       3     0x738c,   4106   36.0°C  35.0W  N/A, N/A, 0         300Mhz  1200Mhz  0%   auto  290.0W  0%     0%    
3       4     0x738c,   8776   36.0°C  36.0W  N/A, N/A, 0         300Mhz  1200Mhz  0%   auto  290.0W  0%     0%    
====================================================================================================================
=============================================== End of ROCm SMI Log ================================================

Helpful Links

Proxmox PCI/GPU Passthrough

Proxmox VE PCI(e) Passthrough Wiki: The official reference: IOMMU, VFIO modules, device assignment.
PCI/GPU Passthrough on Proxmox VE 8 (forum tutorial): Widely-referenced community walkthrough, good for troubleshooting edge cases the wiki doesn’t cover.
Ultimate Beginner’s Guide to Proxmox GPU Passthrough (gist): Thorough, includes IOMMU group inspection steps like the ones we ran.

AMD reset bug / vendor-reset

gnif/vendor-reset (GitHub): The module itself; README lists supported GPU generations including Arcturus/MI100.
Instinct MI100 cluster fails to reset on restart — Issue #80: Worth reading the full thread for edge cases others hit.
Yet another AMD Reset Bug/Vendor Reset Thread (Proxmox forum): Proxmox-specific install/troubleshooting notes.

ROCm on openSUSE Leap

SDB:AMD GPGPU – openSUSE Wiki: Closest thing to an official community reference for ROCm/OpenCL on openSUSE.
Installing ROCm in TW and Leap (openSUSE Forums): AMD doesn’t officially support Leap (only SLES), so expect to lean on community threads like this one rather than AMD’s own docs.
Recommended way to install ROCm on Leap (openSUSE Forums): More recent thread, discusses using SLES packages/repos on Leap as a workaround.

llama.cpp + ROCm

llama.cpp on ROCm installation – AMD ROCm docs: AMD’s own build/install guide, uses -DGGML_HIP=ON and AMDGPU_TARGETS.
llama.cpp docs/build.md (GitHub): Canonical build doc covering all backends, including HIP.

EPYC / NUMA tuning background (if you want to get fancy)

AMD 2nd Gen EPYC CPU Tuning Guide for InfiniBand HPC: NPS/CCD background for the Rome generation – my CPU doesn’t support this 🙁
NUMA Configuration settings on AMD EPYC 2nd Generation (Dell): Despite being from Evil Dell, this is a vendor-neutral deep dive on NPS modes.

AMD Instinct MI100 (CDNA / Vega 20) GPU on openSUSE Leap