CI: Generate RPM diff and auto-publish GitHub Release #63

unixtech-06 · 2025-10-03T02:56:40Z

Closes #60

Add RPM diff generation and GitHub Release automation to CI

Detect previous dated image tags with skopeo and compare RPM lists
Generate Markdown changelog (Added / Updated / Removed) per platform
Publish combined RPM diff as GitHub Release with attached artifacts
Mark release as pre-release when pushing to testing registries

…ous release in the container build workflow

alexiri · 2025-10-04T07:20:47Z

Great work!

For the atomic-desktop images, this changelog is generated by the rechunker, which is very useful. One of the things it does is attach the full package list (ie. your rpm_lists/.../*.txt) as a label to the container image itself. You can see it here, for example:

skopeo inspect docker://quay.io/almalinuxorg/atomic-desktop-gnome:10 | jq '.Labels["dev.hhd.rechunk.info"]'

How about doing this instead? This is kind of like a poor man's SBOM, and since it's tied to the image it makes it much easier for you to get the information of the previous released image and it also makes the information available for other future uses.

alexiri · 2025-10-04T06:40:35Z

.github/workflows/build-test-push.yml

+              delc=$(wc -l < removed.tmp | tr -d ' ')
+
+              echo "" >> "${out}"
+              echo "**Added (${addc})**" >> "${out}"


How about skipping the sections altogether if there are no packages in that category?

Thanks for the suggestion
I’ve updated the changelog generation so that it now skips any Added / Updated / Removed section when the count is zero

.github/workflows/build-test-push.yml

alexiri · 2025-10-04T07:06:07Z

.github/workflows/build-test-push.yml

+        run: |
+          IMAGE_REPO="${IMAGE_NAMES%%,*}"
+
+          # Fetch tag list and pick latest dated tag (<today): <VMJ><VMIN>-YYYYMMDD


Why not just use the current ${VMJ} tag, which should always be the latest released tag?

Thanks for pointing this out
I had overlooked that the ${VMJ} tag is always kept up to date with the latest release
I’ve updated the workflow to simply use ${VMJ} as the comparison base instead of scanning dated tags

Co-authored-by: Alex Iribarren <[email protected]>

alexiri · 2025-10-07T07:56:55Z

.github/workflows/build-test-push.yml

+          echo "Checking ${IMAGE_REPO}:${PREV_TAG}"
+          if ! skopeo inspect docker://"${IMAGE_REPO}:${PREV_TAG}" >/dev/null 2>&1; then
+            echo "::error ::Tag not found: ${IMAGE_REPO}:${PREV_TAG}"
+            exit 1


This makes it impossible for the workflow to run the first time, when there's no previous tag. Not too relevant now that it exists, but still...

updated the step so the workflow won’t fail on the very first run when there’s no previous tag

.github/workflows/build-test-push.yml

… loop ci: switch <details> blocks to printf and always close after platform loop

unixtech-06 · 2025-10-09T01:51:53Z

Great work!

For the atomic-desktop images, this changelog is generated by the rechunker, which is very useful. One of the things it does is attach the full package list (ie. your rpm_lists/.../*.txt) as a label to the container image itself. You can see it here, for example:
skopeo inspect docker://quay.io/almalinuxorg/atomic-desktop-gnome:10 | jq '.Labels["dev.hhd.rechunk.info"]'
How about doing this instead? This is kind of like a poor man's SBOM, and since it's tied to the image it makes it much easier for you to get the information of the previous released image and it also makes the information available for other future uses.

I agree that this approach makes perfect sense.

Here’s what I’m thinking:

For each platform, generate a rechunker-style JSON that includes the full RPM list, its SHA256, and metadata such as built_at and prev_tag.
Compress the JSON with gzip + base64 and store it in an environment variable RCHUNK_INFO_LABEL.
During the final image push step, add the label as follows:
dev.hhd.rechunk.info=${{ env.RCHUNK_INFO_LABEL }} (specified in docker/build-push-action@v5 under labels:).

The label can be verified in the same way as the atomic-desktop images:

skopeo inspect docker://<registry>/<repo>:<tag> \
  | jq -r '.Labels["dev.hhd.rechunk.info"]' \
  | base64 -d | gzip -d | jq .

…iner images feat: embed rechunker-style RPM metadata as compressed label in container images

alexiri · 2025-10-09T05:28:44Z

.github/workflows/build-test-push.yml

+          IMT: ${{ matrix.image_types }}
+          PREV_TAG: ${{ env.prev_tag }}
+          DTS: ${{ needs.init-data.outputs.date_time_stamp }}
+          LABEL_KEY: dev.hhd.rechunk.info


dev.hdd.rechunk.info isn't a standardized label or anything, and given the format is different maybe we shouldn't use the same label. How about org.almalinux.sbom? Reference: https://github.com/opencontainers/image-spec/blob/main/annotations.md

That sounds good
I'll use org.almalinux.sbom instead.

alexiri · 2025-10-09T05:29:43Z

.github/workflows/build-test-push.yml

+            rm -f pkgs_${key}.json
+          done
+
+          # Compress and encode JSON into a single-line lable payload


Suggested change

# Compress and encode JSON into a single-line lable payload

# Compress and encode JSON into a single-line label payload

alexiri · 2025-10-09T05:30:09Z

.github/workflows/build-test-push.yml

          push: true
          tags: ${{ steps.meta.outputs.tags }}
+          labels: |
+            ${{ env.LABLE_KEY }}=${{ env.RCHUNK_INFO_LABEL }}


Suggested change

${{ env.LABLE_KEY }}=${{ env.RCHUNK_INFO_LABEL }}

${{ env.LABEL_KEY }}=${{ env.RCHUNK_INFO_LABEL }}

alexiri · 2025-10-09T05:34:14Z

.github/workflows/build-test-push.yml

+
+            # Old image (if present): <image>:<tag>
+            if [ -n "${IMG_PREV:-}" ] && [ -n "${TAG_PREV:-}" ]; then
+              docker pull --platform "${plat}" "${IMG_PREV}:${TAG_PREV}" >/dev/null 2>&1 || true


You don't need to run the previous image and extract the list of packages again, now that it's attached as a label you could just get it from there. If you use crane, you can even get the label without having to pull the image!

I'll switch to using crane.
It seems that crane isn’t included in the default ubuntu-latest runner image, so I’ll add installation steps for it as well.
https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md

alexiri · 2025-10-09T05:44:38Z

.github/workflows/build-test-push.yml

+          done
+
+          # Compress and encode JSON into a single-line lable payload
+          RCHUNK_INFO_LABEL=$(gzip -c "rechunk_${VMJ}_${IMT}.json" | base64 -w0)


I understand wanting to reduce the size of this data, but making it binary makes it a bit less useful. It adds an extra couple of steps to be able to read and use it. There's no limit to the value size (as far as I can tell), so why not just store the json directly?

I’ll change it to store the JSON directly in the label instead of encoding it.

…nfo)

…ing old image ci: read previous RPM list from image label via crane instead of running old image ci: read previous RPM list from image label via crane instead of running old image ci: read previous RPM list from image label via crane instead of running old image ci: read previous RPM list from image label via crane instead of running old image

alexiri · 2025-10-09T12:43:59Z

This looks great! Are you running it anywhere so we can verify everything works as expected and looks good?

unixtech-06 · 2025-10-09T14:04:44Z

This looks great! Are you running it anywhere so we can verify everything works as expected and looks good?

I'll try it out in my own repository by creating a new branch for testing.

unixtech-06 · 2025-10-10T07:03:48Z

The workflow ran successfully and generated the changelog correctly.
The result is available here:
https://github.com/unixtech-06/container-images/releases/tag/v20251010

…d RPM diff output

alexiri · 2025-10-15T09:40:31Z

Hi @unixtech-06, please take a look at this conversation: https://chat.almalinux.org/almalinux/pl/yuun4ire5jbibxtid1mfubgzfy

Now would be a great opportunity to standardize on the same SBOM format for all AlmaLinux images!

unixtech-06 · 2025-10-21T07:41:29Z

Hi @unixtech-06, please take a look at this conversation: https://chat.almalinux.org/almalinux/pl/yuun4ire5jbibxtid1mfubgzfy

Now would be a great opportunity to standardize on the same SBOM format for all AlmaLinux images!

Sorry, I was mistaken earlier.

After thinking about it, the current rpm-based implementation doesn’t work for micro anyway since it doesn’t include the rpm command.
So for now, I think it’s fine to generate SBOMs for the other container images (excluding micro) by mounting their rootfs externally and using this script.

alexiri · 2025-12-16T19:34:09Z

@javihernandez, what do you think, good to go with the SBOM?

javihernandez · 2025-12-17T08:37:56Z

@javihernandez, what do you think, good to go with the SBOM?

Hey, currently, the sbom tools are not part of the cloud images pipelines, its inclusion in the gcp images pipelines is being finalized as part of AlmaLinux/cloud-images#266.

In any case, yes, I think that we can proceed with the SBOM tools approach. For that, @unixtech-06 needs to:

Create a PR with the rpm bindings fallback changes in https://github.com/AlmaLinux/cloud-images-sbom-tools
Update this PR accordingly. Please check the GCP images PR above for inspiration on how to include the tools into the container images pipeline(s)

Add a changelog feature that shows the RPM diff compared to the previ…

dd2b0e2

…ous release in the container build workflow

alexiri reviewed Oct 4, 2025

View reviewed changes

fix(diff): change arrow in awk output from '->' to '→'

b63a774

Co-authored-by: Alex Iribarren <[email protected]>

unixtech-06 force-pushed the changelog-clean branch from 23b0c9f to b63a774 Compare October 7, 2025 00:36

unixtech-06 added 3 commits October 7, 2025 09:45

ci: simplify previous image detection — use tag directly

b9bca8d

ci: skip empty RPM diff sections in changelog

c59a1bc

ci: simplify previous image detection — use tag directly

5ebb806

unixtech-06 force-pushed the changelog-clean branch 5 times, most recently from c49569a to 5ebb806 Compare October 7, 2025 03:58

ci: wrap per-platform RPM diff in collapsible <details> sections

8c0ac51

alexiri reviewed Oct 7, 2025

View reviewed changes

.github/workflows/build-test-push.yml Outdated Show resolved Hide resolved

ci: allow initial run when no previous tag exists

d299edb

unixtech-06 force-pushed the changelog-clean branch 2 times, most recently from 97f081e to d299edb Compare October 8, 2025 01:39

ci: switch <details> blocks to printf and always close after platform…

0074210

… loop ci: switch <details> blocks to printf and always close after platform loop

feat: embed rechunker-style RPM metadata as compressed label in conta…

4872506

…iner images feat: embed rechunker-style RPM metadata as compressed label in container images

alexiri reviewed Oct 9, 2025

View reviewed changes

unixtech-06 added 4 commits October 9, 2025 20:03

ci: change LABEL_KEY to org.almalinux.sbom (replace dev.hhd.rechunk.i…

6ae0feb

…nfo)

fix: correct label-related typos

e9967c8

refactor: store JSON label directly instead of gzip+base64

39a73ba

ci(registry,label,changelog): tweak testing registry, label fetch, an…

39bb749

…d RPM diff output

	# Compress and encode JSON into a single-line lable payload
	# Compress and encode JSON into a single-line label payload

	${{ env.LABLE_KEY }}=${{ env.RCHUNK_INFO_LABEL }}
	${{ env.LABEL_KEY }}=${{ env.RCHUNK_INFO_LABEL }}

Uh oh!

CI: Generate RPM diff and auto-publish GitHub Release #63

Are you sure you want to change the base?

CI: Generate RPM diff and auto-publish GitHub Release #63

Uh oh!

Conversation

unixtech-06 commented Oct 3, 2025

Uh oh!

alexiri commented Oct 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

unixtech-06 commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexiri commented Oct 9, 2025

Uh oh!

unixtech-06 commented Oct 9, 2025

Uh oh!

unixtech-06 commented Oct 10, 2025

Uh oh!

alexiri commented Oct 15, 2025

Uh oh!

unixtech-06 commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexiri commented Dec 16, 2025

Uh oh!

javihernandez commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

unixtech-06 commented Oct 9, 2025 •

edited

Loading

unixtech-06 commented Oct 21, 2025 •

edited

Loading

javihernandez commented Dec 17, 2025 •

edited

Loading