Add tool to compare built ipa and master ipa initramfs #1058

peppi-lotta · 2025-07-29T12:06:56Z

This PR introduces a script ipa_debug_tools.sh that provides a set of utilities to assist with debugging and comparing IPA (ironic-python-agent) initramfs builds. The script is particularly useful for identifying differences between locally built IPA images and the official master images.

This tool is meant especially for situations where built IPA size get over a certain limit limit and two file systems need to be compared.

🛠️ What's Included

set_up_debug_dirs
- Sets up a local debug environment under /tmp/debug-initramfs
- Builds the local IPA using jenkins/scripts/dynamic_worker_workflow/build_ipa.sh
- Downloads and extracts the master IPA image from upstream
compare_dir_sizes [path]
- Compares sizes of files/directories between the local build and master IPA
- Provides a colored, sorted table of size differences
- Highlights large discrepancies (10MB+, 1MB+, etc.)
- Lists files exclusive to either initramfs tree

compare_rpm_packages
- Extracts and compares installed RPMs between local and master IPA environments
- Outputs side-by-side view of mismatched or missing packages
- Helps identify unintended package changes

clean_up_debug_dirs
- Deletes all temporary directories and files created during debugging

metal3-io-bot · 2025-07-29T12:07:02Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tuminoid for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Rozzii

In general LGTM,
It is a very useful tool as I have mentioned offline.
I have one improvement proposal but it can be a later addition if you don't feel like implementing it now.

Rozzii · 2025-07-31T06:15:14Z

hack/ipa_debug_tools.sh

+    sudo mkdir -p /tmp/debug-initramfs/master-ipa-initramfs
+    cd /tmp/debug-initramfs || exit
+    sudo wget https://tarballs.opendev.org/openstack/ironic-python-agent/dib/ipa-centos9-master.tar.gz
+    sudo tar -xzf ipa-centos9-master.tar.gz


It would be nice if the upstream IPA URI and tar file name could be customized same as in IPA downloader, you could actually use IPA downloader here instead of your own download logic to keep it simple .

For debug tool, it might be good enough to be able to supply the URL. get-resource.sh vs one-liner... I'd go for the latter.

I am fine either way, but the url and the name of the tar file has to be customizable because otherwise only a specific master on a specific base image can be tested. I suggested to use the downloader because that has all the customization options implemented but I am fine with doing it with the one liners too.

Rozzii · 2025-07-31T06:22:40Z

/override metal3-ubuntu-e2e-integration-test-main
This is a hacktool for manual testing, not used in CI jet.

metal3-io-bot · 2025-07-31T06:22:45Z

@Rozzii: Overrode contexts on behalf of Rozzii: metal3-ubuntu-e2e-integration-test-main

In response to this:

/override metal3-ubuntu-e2e-integration-test-main
This is a hacktool for manual testing, not used in CI jet.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tuminoid

This is great addition to debug tools, thanks @peppi-lotta.

Some cleanup to do, some minor syntax things to fix and some suggestions to make it more readable and maintainable.

hack/ipa_debug_tools.sh

tuminoid · 2025-08-05T06:19:00Z

hack/ipa_debug_tools.sh

+    sudo mkdir -p /tmp/debug-initramfs/master-ipa-initramfs
+    cd /tmp/debug-initramfs || exit
+    sudo wget https://tarballs.opendev.org/openstack/ironic-python-agent/dib/ipa-centos9-master.tar.gz
+    sudo tar -xzf ipa-centos9-master.tar.gz


For debug tool, it might be good enough to be able to supply the URL. get-resource.sh vs one-liner... I'd go for the latter.

hack/ipa_debug_tools.sh

Rozzii · 2025-09-29T17:56:01Z

hack/ipa_debug_tools.sh

+# -----------------------------------------------------------------------------
+clean_up_debug_dirs() {
+    # Clean up debug directories
+    sudo rm -rf "${BASE_DEBUG_DIR}" /tmp/dib ipa-file-injector.service 2> /dev/null


I get "${BASE_DEBUG_DIR}" and /tmp/dib but how does ipa-file-injector.service end up here?
I don't see any other reference to that file, and AFAIK that is the systemd unit file of the similarly named DIB module. It could be that I am missing something here but looks like this is some leftover reference maybe.

ipa-file-injector.service appears in the /tmp directory after running the build-ip.sh. That is why I've added that to the clean up as well.

okay, that might be then some sort of residue from the template rendering process. I might remove it later, but then it is fine to remove it here.

tuminoid

IMO colors are fine, but emojis not.

hack/ipa_debug_tools.sh

Signed-off-by: peppi-lotta <[email protected]>

Rozzii

/lgtm

peppi-lotta · 2025-10-20T07:04:53Z

@tuminoid I've addressed all comment can you approve and override test?

tuminoid · 2025-10-20T07:39:00Z

It doesn't work?

2025-10-20 07:35:48.405 | + diskimage_builder/lib/common-functions:tmpfs_check:29                            :   return 0
2025-10-20 07:35:48.409 | + diskimage_builder/lib/common-functions:cleanup_image_dir:236                     :   timeout 120 sh -c 'while ! sudo umount -f /tmp/dib_image.Jc5gaKZ3; do sleep 1; done'
2025-10-20 07:35:48.426 | + diskimage_builder/lib/common-functions:cleanup_image_dir:241                     :   rm -rf --one-file-system /tmp/dib_image.Jc5gaKZ3
2025-10-20 07:35:48.431 | + diskimage_builder/lib/img-functions:trap_cleanup:37                              :   exit 1
Command '['disk-image-create', '-o', 'ironic-python-agent', 'ironic-python-agent-ramdisk', 'centos', 'dynamic-login', 'journal-to-console', 'devuser', 'openssh-server', 'extra-hardware', 'ipa-module-autoload', 'ipa-add-buildinfo', 'ipa-cleanup-dracut', 'simple-init', 'override-simple-init', 'ipa-file-injector', 'cleanup-package', '-x']' returned non-zero exit status 1.
build_ipa.sh failed, will check for artifact...
Required file '/tmp/dib/ironic-python-agent.initramfs' not created, exiting.

Running the test with using default options (centos10 master, opendev dib url).

tuminoid

Not exactly fond of that it just asks for sudo password and then goes and installs stuff on the system. Installing anything should be another "subcommand" that user must run explicitly.

The sudo usage in general is also very liberal (please don't run sudo wget and sudo tar .... Making a temporary directory with sudo is not typical either.

Sudo chroot also is not needed.

tuminoid · 2025-10-03T07:44:09Z

hack/ipa_debug_tools.sh

+# The path to the directory that holds this script
+CURRENT_SCRIPT_DIR="$(dirname -- "$(readlink -f "${BASH_SOURCE[0]}")")"
+BASE_DEBUG_DIR="/tmp/debug-initramfs"
+mkdir -p ${BASE_DEBUG_DIR}


Suggested change

mkdir -p ${BASE_DEBUG_DIR}

mkdir -p "${BASE_DEBUG_DIR}"

tuminoid · 2025-10-20T13:50:37Z

hack/ipa_debug_tools.sh

+    build_rpm_list_name_bases=$(mktemp -p "${BASE_HELPER_FILE_DIR}")
+    master_rpm_list_name_bases=$(mktemp -p "${BASE_HELPER_FILE_DIR}")
+
+    sudo chroot "${build_dir}" rpm -qa | sort > "${build_rpm_list}"


rpm --root "${build_dir}" -qa should do the trick without sudo chroot.

metal3-io-bot requested review from Rozzii and smoshiur1237 July 29, 2025 12:07

metal3-io-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 29, 2025

peppi-lotta force-pushed the peppi-lotta/add-ipa-debug-tool branch 2 times, most recently from 4c2b687 to 9e9d4b4 Compare July 30, 2025 06:15

Rozzii reviewed Jul 31, 2025

View reviewed changes

Rozzii added this to Metal3 - Roadmap Jul 31, 2025

Rozzii moved this to MISC WIP in Metal3 - Roadmap Jul 31, 2025

tuminoid reviewed Aug 5, 2025

View reviewed changes

peppi-lotta force-pushed the peppi-lotta/add-ipa-debug-tool branch 2 times, most recently from 00054eb to 7a7af56 Compare September 29, 2025 09:51

Rozzii reviewed Sep 29, 2025

View reviewed changes

tuminoid reviewed Sep 30, 2025

View reviewed changes

Add tool to compare built ipa and master ipa initramfs

27494cb

Signed-off-by: peppi-lotta <[email protected]>

peppi-lotta force-pushed the peppi-lotta/add-ipa-debug-tool branch from 7a7af56 to 27494cb Compare October 3, 2025 07:16

Rozzii reviewed Oct 7, 2025

View reviewed changes

metal3-io-bot added the lgtm Indicates that a PR is ready to be merged. label Oct 7, 2025

tuminoid reviewed Oct 20, 2025

View reviewed changes

Add tool to compare built ipa and master ipa initramfs #1058

Are you sure you want to change the base?

Add tool to compare built ipa and master ipa initramfs #1058

Uh oh!

Conversation

peppi-lotta commented Jul 29, 2025

🛠️ What's Included

Uh oh!

metal3-io-bot commented Jul 29, 2025

Uh oh!

Rozzii left a comment

Choose a reason for hiding this comment

Uh oh!

Rozzii Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tuminoid Aug 5, 2025

Choose a reason for hiding this comment

Uh oh!

Rozzii Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

Rozzii commented Jul 31, 2025

Uh oh!

metal3-io-bot commented Jul 31, 2025

Uh oh!

tuminoid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tuminoid Aug 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Rozzii Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peppi-lotta Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

Rozzii Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

tuminoid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Rozzii left a comment

Choose a reason for hiding this comment

Uh oh!

peppi-lotta commented Oct 20, 2025

Uh oh!

tuminoid commented Oct 20, 2025

Uh oh!

tuminoid left a comment

Choose a reason for hiding this comment

Uh oh!

tuminoid Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

tuminoid Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

Rozzii Jul 31, 2025 •

edited

Loading

Rozzii Sep 29, 2025 •

edited

Loading