[SWDEV-535159] Add support for GPU partition metrics (#490)

[SWDEV-535159] Add support for GPU partition metrics

Changes include:
  - Internal logic to smart-switch between gpu_metrics/xcp_metrics files
  - [WIP] Initial plumbing for new partition metric API

Change-Id: I4340fb1b48bac0117d80d5d486b9e871430d5cd8
Signed-off-by: Charis Poag <[email protected]>

Add amdsmi_get_gpu_partition_metrics_info() + minor cleanup

Change-Id: I5d60604f18baddbd03852dc90e88aa0b8107d50e
Signed-off-by: Charis Poag <[email protected]>

Fix partition metric logic + update logging/tests

Change-Id: I9e89b19ead17694c54e224f8e13ff8ee3eb2e22a
Signed-off-by: Charis Poag <[email protected]>

Adjust amd-smi metric/monitor/default to show (some) partition information

Change-Id: I2e8d2745876a19bdaec3c039daa97345c9f701b5
Signed-off-by: Charis Poag <[email protected]>

Add C++ tests

Change-Id: Ib9eb0b57a6d7a280992e05a4c6eba632826952ef
Signed-off-by: Charis Poag <[email protected]>

Remove modification of energy counter, not needed

Change-Id: I5c48eaaae248ee6dc79abba609d837ec35d78022
Signed-off-by: Charis Poag <[email protected]>

[CLI] amd-smi metric: cleaned up N/A'd multi-valued to show just N/A

Changes:
1. amd-smi metric: cleaned up N/A'd multi-valued to show just N/A
ex.
JPEG_ACTIVITY: [N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A]

Now just shows: N/A

2. [Python Unit Test] Changed testname TestAmdSmiPythonBDF(unittest.TestCase) ->
 AmdSmiPythonUnitTest

Test name was confusing.

Change-Id: Ieb3b036f30002fd22362508eb9fc5d443df395ae
Signed-off-by: Charis Poag <[email protected]>

Log cleanup

Change-Id: I1b1a95f1844d35bec7a7bd8cb996f87e4914c069
Signed-off-by: Charis Poag <[email protected]>

Add amd-smi partition-metrics CLI + general cleanup

Change-Id: Ia91488e6cb3a4d62b4087afbddfe0b3bb9378fdc
Signed-off-by: Charis Poag <[email protected]>

[1.3 metrics] Remove forwards compatibility for partition metrics

Change-Id: Iab928983e6f6f1587bc9307f6f3fa2b2696ca6f7
Signed-off-by: Charis Poag <[email protected]>

Fixed violation output not showing % + general cleanup

Change-Id: Icac1b0a55b18c7628b07109ae0c377d17e0825f1
Signed-off-by: Charis Poag <[email protected]>

Clean up amdsmi_get_gpu_partition_metrics_info & amd-smi partition-metric outputs

Change-Id: I6427028b980874641e9ffb3b5d88ad493dbf9cf4
Signed-off-by: Charis Poag <[email protected]>

* Fix metrics not found + extra logging/formatting

Change-Id: I841a27bb2c305e97ec7579a13ac915e5be497c3a
Signed-off-by: Charis Poag <[email protected]>

* Update license to current default

Change-Id: I0de9b8a2d5dbbeab4491097f0354ba17b0d30866
Signed-off-by: Charis Poag <[email protected]>

* Cleanup for review

Change-Id: I96ed25c3f2b8968eea1af24c5e5860c2b4e74e6e
Signed-off-by: Charis Poag <[email protected]>

* Moderize updated/new interal APIs.

Change-Id: I3c48a250eeb703709b14cb5ffa68268d8321626c
Signed-off-by: Charis Poag <[email protected]>

* Remove extra logging in dynamic metrics

Change-Id: Idb97547bcbe143d6fa1cb5cb278ffe4da615ce14
Signed-off-by: Charis Poag <[email protected]>

* Remove amd-smi partition-metric command

Change-Id: Ib83c17e5cd7e0da3798198943bddd46c296b411c
Signed-off-by: Charis Poag <[email protected]>

* Move new CLI updates to another PR + minor fixes

Change-Id: I3b1163eec12f9b5f7d95ee33de08e168cec1b1fe
Signed-off-by: Charis Poag <[email protected]>

* Allow dynamic metrics to work for gpu/xcp metrics 1.9+/1.1+

Updated some logging as well.

Change-Id: I2ed9f5a5ef8afb1520508820ca6153525f0644b4
Signed-off-by: Charis Poag <[email protected]>

* Allow dyn gpu/xcp metric v1.9+/v1.1+

Added tests for quick check

Change-Id: I576d6f6582a55afb08e5ac57791ce95e2fa184a2
Signed-off-by: Charis Poag <[email protected]>

* Update tests for larger subset of version checks

Change-Id: I3cdf4f8bb4fc6161f4c76566939f90545d0f362a
Signed-off-by: Charis Poag <[email protected]>

* Fix XCP metrics in gpu/partition metric pre-v1.9/v1.1 (dynamic)

Change-Id: I4dabc1ed6bef6b86c8e7f92bf9cb5992f3966fe2
Signed-off-by: Charis Poag <[email protected]>

---------

Change-Id: I8ab1752743b04f1c7791d0405a7bccd7128b01ae
Signed-off-by: Charis Poag <[email protected]>

Author: charis-poag-amd

amdsmi_cli/amdsmi_commands.py

@@ -1672,6 +1672,7 @@ def metric_gpu(self, args, multiple_devices=False, watching_output=False, gpu=No
         # Add timestamp and store values for specified arguments
         values_dict = {}
 
+        is_partition_metrics = False  # True if we get the metrics from xcp_metrics file (amdsmi_get_gpu_partition_metrics_info)
         #get metric info only once per gpu, this will speed up data output
         try:
             # Get GPU Metrics table
@@ -1680,19 +1681,10 @@ def metric_gpu(self, args, multiple_devices=False, watching_output=False, gpu=No
             logging.debug("#3 - Unable to load GPU Metrics table for %s | %s", gpu_id, e.get_error_info())
             gpu_metric = amdsmi_interface._NA_amdsmi_get_gpu_metrics_info()
 
-        # Workaround for XCP (partition) metrics not providing num_partition in v1.0
-        # Confirmed with driver team that we can default to 1 if num_partition is not defined.
-        # Pending partitions exist, ie. partition_id > 0. See logic below.
-        try:
-            partition_id = amdsmi_interface.amdsmi_get_gpu_kfd_info(args.gpu)['current_partition_id']
-        except amdsmi_exception.AmdSmiLibraryException as e:
-            logging.debug("Failed to get current partition id for gpu %s | %s", gpu_id, e.get_error_info())
-            partition_id = "N/A"
-
-        num_partition = gpu_metric['num_partition']
-        if num_partition == "N/A":
-            num_partition = 1  # Workaround for XCP metrics not providing num_partition in v1.0
-            logging.debug(f"num_partition is N/A and partition_id: {partition_id} (greater > 0).\nModified num_partition: {num_partition} to adjust for XCP metrics.")
+        # Workaround for XCP (partition) metrics not providing num_partition in v1.9+/v1.1+
+        # Provides original formatting for earlier metric versions
+        partition_metric_info = self.helpers._get_metric_version_and_partition_info(gpu_metric, is_partition_metrics, gpu_id, args.gpu)
+        num_partition = partition_metric_info['num_partition']
 
         if self.logger.is_json_format():
             values_dict['gpu'] = int(gpu_id)
@@ -2719,7 +2711,7 @@ def metric_gpu(self, args, multiple_devices=False, watching_output=False, gpu=No
                                     value[k][index] = self.helpers.unit_format(self.logger, activity, activity_unit)
                                 value[k] = '[' + ", ".join(value[k]) + ']'
                         elif value != "N/A":
-                            value = self.helpers.unit_format(self.logger, value, activity_unit)
+                            throttle_status[key] = self.helpers.unit_format(self.logger, value, activity_unit)
                     if self.logger.is_json_format():
                         if isinstance(value, (list, dict)):
                             for k, v in value.items():
@@ -3130,7 +3122,6 @@ def metric_core(self, args, multiple_devices=False, core=None, core_boost_limit=
         if not self.logger.is_json_format():
             self.logger.print_output(multiple_device_enabled=multiple_devices_csv_override)
 
-
     def metric(self, args, multiple_devices=False, watching_output=False, gpu=None,
                 usage=None, watch=None, watch_time=None, iterations=None, power=None,
                 clock=None, temperature=None, ecc=None, ecc_blocks=None, pcie=None,
@@ -5744,6 +5735,7 @@ def monitor(self, args, multiple_devices=False, watching_output=False, gpu=None,
             except amdsmi_exception.AmdSmiLibraryException as e:
                 logging.debug("#5 - Unable to load GPU Metrics table for %s | %s", gpu_id, e.get_error_info())
 
+        is_partition_metrics = False  # True if we get the metrics from xcp_metrics file (amdsmi_get_gpu_partition_metrics_info)
         #get metric info only once per gpu, this will speed up data output
         try:
             # Get GPU Metrics table
@@ -5755,25 +5747,15 @@ def monitor(self, args, multiple_devices=False, watching_output=False, gpu=None,
             gpu_metrics_info = amdsmi_interface._NA_amdsmi_get_gpu_metrics_info()
             logging.debug("Unable to load GPU Metrics table for %s | %s", gpu_id, e.get_error_info())
 
-        # Workaround for XCP (partition) metrics not providing num_partition in v1.0
-        # Confirmed with driver team that we can default to 1 if num_partition is not defined.
-        # Pending partitions exist, ie. partition_id > 0. See logic below.
-        try:
-            partition_id = amdsmi_interface.amdsmi_get_gpu_kfd_info(args.gpu)['current_partition_id']
-        except amdsmi_exception.AmdSmiLibraryException as e:
-            logging.debug("Failed to get current partition id for gpu %s | %s", gpu_id, e.get_error_info())
-            partition_id = "N/A"
+        # Workaround for XCP (partition) metrics not providing num_partition in v1.9+/v1.1+
+        # Provides original formatting for earlier metric versions
+        partition_metric_info = self.helpers._get_metric_version_and_partition_info(gpu_metrics_info, is_partition_metrics, gpu_id, args.gpu)
+        partition_id = partition_metric_info['partition_id']
+        num_partition = partition_metric_info['num_partition']
 
-        num_partition = gpu_metrics_info['num_partition']
-        if num_partition == "N/A":
-            num_partition = partition_id
-
-        num_xcp = num_partition  # used later for XCP metrics
+        # Update logger for XCP display (only if applicable)
         self.logger.table_header += 'XCP'.rjust(5, ' ')
-        self.logger.store_output(args.gpu, 'xcp', partition_id)  # Starting with partition_id.
-                                                                 # Outputs which have xcp details
-                                                                 # will update this value via num_xcp.
-                                                                 # This value will help map to primary device.
+        self.logger.store_output(args.gpu, 'xcp', partition_id)  # Store partition_id initially; can be updated via num_xcp
 
         # Store the pcie_bw values due to possible increase in bandwidth due to repeated gpu_metrics calls
         if args.pcie:
@@ -6013,7 +5995,7 @@ def monitor(self, args, multiple_devices=False, watching_output=False, gpu=None,
                                                            "unit" : freq_unit}
             except (KeyError, amdsmi_exception.AmdSmiLibraryException) as e:
                 monitor_values['dclock'] = "N/A"
-                logging.debug("Failed to get vclock on gpu %s | %s", gpu_id, e)
+                logging.debug("Failed to get dclock on gpu %s | %s", gpu_id, e)
 
             self.logger.table_header += 'DCLOCK'.rjust(10)
 
@@ -6356,7 +6338,7 @@ def monitor(self, args, multiple_devices=False, watching_output=False, gpu=None,
                     self.logger.store_multiple_device_output()
                     current_xcp += 1
             else:
-                self.logger.store_output(args.gpu, 'xcp', num_xcp)
+                self.logger.store_output(args.gpu, 'xcp', partition_id)
                 self.logger.store_output(args.gpu, 'values', monitor_values)
 
         # Store typical output for all commands (XCP data will be handled separately, eg. violation status)

amdsmi_cli/amdsmi_helpers.py

@@ -1017,7 +1017,6 @@ def unit_format(self, logger, value, unit):
         """This function will format output with unit based on the logger output format
 
         params:
-            args - argparser args to pass to subcommand
             logger (AMDSMILogger) - Logger to print out output
             value - the value to be formatted
             unit - the unit to be formatted with the value
@@ -1040,6 +1039,9 @@ def unit_format(self, logger, value, unit):
                     return {"value": value, "unit": unit}
                 else:
                     return value
+            if logger.is_csv_format():
+                # For CSV, return the raw value (number or "N/A"), not a string
+                return value
             if logger.is_human_readable_format():
                 if unit:
                     return f"{value} {unit}".rstrip()
@@ -1637,3 +1639,70 @@ def average_flattened_ints(data, context="data"):
         # Flatten nested lists and filter integers
         flat = [v for value in data for v in (value if isinstance(value, list) else [value]) if isinstance(v, int)]
         return round(sum(flat) / len(flat)) if flat else "N/A"
+
+    def _get_metric_version_and_partition_info(self, gpu_metrics_info, is_partition_metrics, gpu_id, gpu_handle):
+        """
+        Helper method to compute metric version, partition ID, and num_partition for dynamic metrics.
+        Handles logging updates internally for reusability.
+        
+        Args:
+            gpu_metrics_info (dict): GPU metrics info from amdsmi_get_gpu_metrics_info.
+            is_partition_metrics (bool): Whether this is for partition metrics.
+            gpu_id (int): GPU ID for logging.
+            gpu_handle: GPU device handle for KFD info retrieval.
+        
+        Returns:
+            dict: {
+                'metric_version': float or "N/A",
+                'partition_id': int or "N/A",
+                'num_partition': int or "N/A",
+                'num_xcp': int or "N/A"  # Alias for num_partition
+            }
+        """
+        # Compute metric version from header revisions
+        metric_version = "N/A"
+        format_rev = gpu_metrics_info.get('common_header.format_revision', "N/A")
+        content_rev = gpu_metrics_info.get('common_header.content_revision', "N/A")
+        if format_rev != "N/A" and content_rev != "N/A":
+            try:
+                metric_version = float(f"{format_rev}.{content_rev}")
+            except ValueError:
+                metric_version = "N/A"  # Fallback if conversion fails
+        
+        # Retrieve partition ID from KFD info
+        partition_id = "N/A"
+        try:
+            kfd_info = amdsmi_interface.amdsmi_get_gpu_kfd_info(gpu_handle)
+            partition_id = kfd_info.get('current_partition_id', "N/A")
+        except amdsmi_exception.AmdSmiLibraryException as e:
+            logging.debug("Failed to get current partition ID for GPU %s | %s", gpu_id, e.get_error_info())
+        
+        # Determine num_partition with fallback logic for dynamic metrics
+        num_partition = gpu_metrics_info.get('num_partition', "N/A")
+        if metric_version != "N/A" and num_partition == "N/A":
+            # Workaround: Default to 1 for newer metric versions if num_partition is missing
+            # (Confirmed with driver team; applies to GPU and partition metrics)
+            if not is_partition_metrics and metric_version >= 1.9:
+                num_partition = 1
+            elif is_partition_metrics and metric_version >= 1.1:
+                num_partition = 1
+            elif partition_id != "N/A" and partition_id > 0:
+                # Fallback to partition_id if partitions exist but num_partition is unavailable
+                num_partition = partition_id
+            # Else: Remains "N/A" if no conditions match
+        
+        # Alias num_xcp for XCP metrics usage
+        num_xcp = num_partition
+        
+        # Debug logging
+        logging.debug(
+            "GPU %s | Metric version: %s, num_partition: %s, partition_id: %s, num_xcp: %s",
+            gpu_id, metric_version, num_partition, partition_id, num_xcp
+        )
+        
+        return {
+            'metric_version': metric_version,
+            'partition_id': partition_id,
+            'num_partition': num_partition,
+            'num_xcp': num_xcp
+        }    

amdsmi_cli/amdsmi_parser.py

@@ -918,7 +918,6 @@ def _add_bad_pages_parser(self, subparsers: argparse._SubParsersAction, func):
         self._add_device_arguments(bad_pages_parser, required=False)
         self._add_command_modifiers(bad_pages_parser)
 
-
     def _add_metric_parser(self, subparsers: argparse._SubParsersAction, func):
         # Subparser help text
         metric_help = "Gets metric/performance information about the specified GPU"

include/amd_smi/amdsmi.h

@@ -4055,6 +4055,30 @@ amdsmi_get_gpu_metrics_header_info(amdsmi_processor_handle processor_handle, amd
 amdsmi_status_t amdsmi_get_gpu_metrics_info(amdsmi_processor_handle processor_handle,
                                             amdsmi_gpu_metrics_t *pgpu_metrics);
 
+/**
+ *  @brief This function retrieves the partition metrics information.
+ *
+ *  @ingroup tagClkPowerPerfQuery
+ *
+ *  @platform{gpu_bm_linux} @platform{guest_1vf}
+ *
+ *  @details Given a processor handle @p processor_handle and a pointer to a
+ *  ::amdsmi_gpu_metrics_t structure @p pgpu_metrics, this function will populate
+ *  @p pgpu_metrics. See ::amdsmi_gpu_metrics_t for more details.
+ *
+ *  @param[in] processor_handle a processor handle
+ *
+ *  @param[in,out] pgpu_metrics a pointer to an ::amdsmi_gpu_metrics_t structure
+ *  If this parameter is nullptr, this function will return
+ *  ::AMDSMI_STATUS_INVAL if the function is supported with the provided,
+ *  arguments and ::AMDSMI_STATUS_NOT_SUPPORTED if it is not supported with the
+ *  provided arguments.
+ *
+ *  @return ::amdsmi_status_t | ::AMDSMI_STATUS_SUCCESS on success, non-zero on fail
+ */
+amdsmi_status_t amdsmi_get_gpu_partition_metrics_info(amdsmi_processor_handle processor_handle,
+                                                      amdsmi_gpu_metrics_t *pgpu_metrics);
+
 /**
  *  @brief Get the pm metrics table with provided device index.
  *

py-interface/__init__.py

@@ -170,6 +170,7 @@
 from .amdsmi_interface import amdsmi_get_clk_freq
 from .amdsmi_interface import amdsmi_get_gpu_od_volt_info
 from .amdsmi_interface import amdsmi_get_gpu_metrics_info
+from .amdsmi_interface import amdsmi_get_gpu_partition_metrics_info
 from .amdsmi_interface import amdsmi_get_gpu_od_volt_curve_regions
 from .amdsmi_interface import amdsmi_is_gpu_power_management_enabled