A project to build and configure the ScoreP performance measurement framework for AMD GPUs, specifically the Mi300, with support for the ROCm stack.

• Introduction
  • Purpose
  • Functionality
    • Patches to PAPI
• Before You Start
• Usage
  • 1. Directory Structure
  • 2. Building ScoreP
  • 3. Adding ScoreP to Your Environment
  • 4. Building ScoreP Plugins
  • 5. Choose Your Measurement Parameters
  • 6. Compiling CPU-Only Applications
  • 7. Compiling GPU Applications
  • 8. Running Applications
  • 9. Viewing Results
  • 10. Cleaning Up
• Conclusion

The purpose of this project is to provide a comprehensive setup for HPC researchers and developers to gather performance metrics on exascale systems using AMD GPUs, especially those based on the Mi300 and Mi200 architectures (e.g. El Capitan, Frontier, etc.).

Additionally, this project provides extensions to the PAPI interface so that researchers can perform analysis of the energy consumption of their applications on AMD GPUs. This is particularly useful for researchers who are interested in research questions surrounding how different applications or design choices affect the energy consumption of their applications on AMD GPUs, and how to optimize for energy efficiency.

This project performs the following tasks:

• Builds the ScoreP performance measurement framework with support for AMD GPUs, and all the necessary dependencies from source.
• Configures ScoreP to use the ROCm stack for performance measurement on AMD GPUs using the hipcc compiler infrastructure.
• Provides scripts to automate the build and configuration process.

Additionally, this project provides two patches which extend to the PAPI interface to provide access to some ROCm features not available in the standard PAPI interface

• The energy consumption of the GPU provided in the rocm-smi API. The documentation for this metric can be found here.
• The Mi300 GPU introduces new performance counters for collecting instantaneous GPU power consumption metrics in microwatts. These counters are not available in the standard PAPI interface, but can be accessed through the rocm-smi API. The documentation for this metric can be found here.

You need to have the following prerequisites installed:

• ROCm (Radeon Open Compute) stack - You'll need to check the ROCm documentation for installation instructions. The system requirements can be found here. This project assumes you're using Linux, and have a compatible AMD GPU for use with hipcc and LLVM.
• Your ROCm installation must be in the /opt/rocm-<version> directory. This script will adapt to your version automatically, assuming it's compatible with ScoreP.
• Your ROCm installation must support the rocm-smi API, which is used to query GPU information.
• You must have a C compiler installed. The dependencies of ScoreP in this project were built and tested with gcc version 13.3.0 on Ubuntu 24.04, but other versions of gcc and clang should work too.

If you encounter issues refer to the following documentation:

• Make sure you have permissions to access the API for your AMD GPU. To run GPU applications, your account may need to be in the render group.
• ScoreP documentation
• ROCm documentation

All sections of this project are designed to run in a bash environment without root privileges.

The project directory structure is as follows:

scorep-amd/
├── README.md             # This file
├── docs/                 # Directory containing documentation files
│   └── scorep.pdf        # Documentation for ScoreP
│
├── patches/              # Directory containing patches for PAPI
│   ├── coretemp/         # The PAPI patch for more SYSFS data (power usage)
│   ├── rocm_smi/         # The PAPI patch for rocm_smi (energy consumption)
│   └── rocm_smi_lib/     # The patched ROCm SMI library API
│
├── plugins/              # Directory for ScoreP plugins
│   ├── build-plugins.sh         # Script to build ScoreP plugins for AMD GPUs
│   ├── scorep-arocm-smi-plugin/ # ROCm SMI plugin for ScoreP
│   └── scorep-coretemp-plugin/  # Coretemp plugin for ScoreP
│
├── runs/                 # Directory for running ScoreP applications
│   ├── build-plugins.sh      # Script to build ScoreP plugins for AMD GPUs
│   └── setup-run-params.sh   # Script to set up ScoreP measurement parameters
│
├── build-scorep.sh       # Script to build ScoreP and its dependencies
├── clean.sh              # Script to clean builds, installs, environment
├── setup-env.sh          # Script to add ScoreP to the environment
├── build-papi.sh         # Script to build the patched PAPI and rocm_smi_lib
│
├── build/                # Auto-generated directory where everything is built
└── install/              # Auto-generated directory for all installed files

The details of each directory and file will be explained piecemeal in the following sections.

First, clone the repository and navigate to the project directory. The placement of this directory is not important to the project's functionality, as long as you have write permissions in the directory.

$ # Clone the repository
$ git clone https://github.com/adam-mcdaniel/scorep-amd.git
$ # Navigate to the project directory
$ cd scorep-amd

To build ScoreP, run the build-scorep.sh script. This script will:

• Download all the necessary dependencies.
• Add the patched PAPI components to the PAPI source tree, if they are not already present.
• Build each dependency from source in the ./build directory.
• Configure ScoreP to use the ROCm stack for performance measurement on AMD GPUs.
• Install ScoreP in the ./install directory.

$ ./build-scorep.sh

The script will automatically detect the version of ROCm installed on your system and adapt the build process accordingly. You will not need to run build-papi.sh separately, as it is called by the build-scorep.sh script.

After running the script, you should see output indicating that ScoreP and its dependencies have been built successfully. The final output will indicate the installation directory, which is ./install by default.

Check to make sure that all your desired binaries for ScoreP exist in the ./install/bin/ directory. For compiling applications that use GPU kernels, you should see the scorep-hipcc binary, which is a wrapper around the hipcc compiler that adds the necessary ScoreP instrumentation.

To use ScoreP in your applications conveniently, you need to add the ScoreP binaries to your environment. You can do this by running the setup-env.sh script:

$ source setup-env.sh

This script will:

• Add the ScoreP binaries to your PATH.
• Set the LD_LIBRARY_PATH to include the ScoreP libraries.
• Set the LIBRARY_PATH to include the ScoreP libraries for linking.
• Set the C_INCLUDE_PATH to include the ScoreP headers for compilation.
• Define the INSTALL_DIR environment variable to point to the ./install directory. This is used for the scripts in runs/ to find the ScoreP installation, and for setting up their internal paths.

After running the setup-env.sh script, you should be able to run scorep-hipcc and other ScoreP tools from any directory in your terminal. You can verify this by running:

$ scorep-info config-summary

If everything is set up correctly, you should see a summary of the ScoreP configuration, including the hipcc support and the ROCm provided LLVM tools.

To use the metrics provided by PAPI, whether you use the rocm_smi or coretemp patches, you need to build the ScoreP plugins. The plugins/build-plugins.sh script will do this for you.

$ # Make sure you've already run the previous steps!
$ # Navigate to the runs directory
$ cd plugins/
$ # Build the ScoreP plugins to add ROCm instrumentation
$ ./build-plugins.sh

If your environment is setup correctly and the ScoreP installation was successful, you should see output indicating that the plugins have been built successfully.

You can verify that the plugins are installed correctly with the following command:

$ ls $INSTALL_DIR/lib/lib*plugin*.so
/path/to/scorep-amd/install/lib/libarocm_smi_plugin.so
/path/to/scorep-amd/install/lib/libcoretemp_plugin.so

If you see the libarocm_smi_plugin.so and libcoretemp_plugin.so files, then the plugins have been built successfully.

To run applications with ScoreP instrumentation, you must first set up the ScoreP measurement parameters.

$ # Make sure you've already run the previous steps!
$ # Navigate to the runs/ directory
$ cd /path/to/scorep-amd/runs
$ # Set up the ScoreP measurement parameters
$ source ./setup-run-params.sh

This script will configure the ScoreP environment variables to specify the metrics to collect, the tracing/profiling options, the output directory and file formats, etc.

Now that you have built ScoreP and the necessary plugins, you can compile and run applications with ScoreP instrumentation!

You can compile CPU applications with ScoreP instrumentation by using scorep-clang, scorep-clang++, or scorep-flang as your compiler. For example:

$ scorep-clang -o my_cpu_app my_cpu_app.c

You may also write your own Makefiles that use the appropriate ScoreP wrapper as your CC or CXX variables, for example.

To compile GPU applications that use hipcc, you should use the scorep-hipcc wrapper. This wrapper will automatically instrument your code for ScoreP and link against the necessary libraries.

$ scorep-hipcc rocm-blas-gemm.cpp -o rocm-blas-gemm -lrocblas

To run your instrumented applications, simply execute them as you normally would. ScoreP will automatically collect performance data based on the configuration set in the setup-run-params.sh script, and produce output files in the specified output directories (e.g., experiments/scorep-experiment-<time>).

For example:

$ scorep-clang -o my_cpu_app my_cpu_app.c
$ ./my_cpu_app

Or for a GPU application:

$ scorep-hipcc rocm-blas-gemm.cpp -o rocm-blas-gemm -lrocblas
$ ./rocm-blas-gemm

By default, ScoreP will generate .otf2 files in the specified output directory (e.g., experiments/scorep-experiment-<time>).

To analyze the results, it is recommended (but not required) to use Vampir.

To clean up the build/ and install/ directories, you can run the clean.sh script:

$ ./clean.sh

This script provides you with the option to clean the specific directories you want, as well as the option to clear all the environment variables set by setup-env.sh. This is useful if you want to start with a fresh environment, or if you want to test a different configuration without the previous settings interfering.

This project provides a comprehensive setup for building and configuring ScoreP for AMD GPUs, specifically the Mi300, with support for the ROCm stack. By following the steps outlined in this README, you should be able to successfully build ScoreP, configure it for AMD GPUs, and run applications with performance measurement capabilities.

If you have any unexpected or unanticipated issues, please feel free to open an issue on the GitHub repository!