Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion meson_post_install.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,6 @@
euid = os.geteuid()

if not destdir and not os.path.exists('/run/.containerenv') and euid == 0:
subprocess.run(['systemd-tmpfiles', '--create'], check=True)
subprocess.run(['systemd-tmpfiles', '--create', '--prefix=/run'], check=True)

sys.exit(0)
48 changes: 37 additions & 11 deletions playbooks/dependencies-fedora-coreos.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,19 +14,45 @@
# limitations under the License.
#

- name: Synchronize glibc to repo version for gcc/glibc-devel compatibility
become: yes
command: >-
rpm-ostree override replace --experimental --from repo=updates
glibc
glibc-common
glibc-minimal-langpack
glibc-gconv-extra
ignore_errors: yes
# These package overrides get a little complicated due to the
# `next` stream pointing at pre-release repos during branching.
# With the below loop, we try and use the default `updates` repo,
# and fallback to `updates-testing` and `fedora` in turn so that we
# can update base packages on the system.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't the updates-testing aspect turn out to be a red herring? In this case the problem is that the Fedora CoreOS host image has an older libgcc from an older gcc build and the fedora repository has a newer one, and this trips up the rpm install ... gcc later.

It seems worth it to include fedora, because during freezes and right after a release, the ISOs and images might have older builds than fedora. The problem might go away once a release gets older and newer builds make their way to updates.

More importantly, I don't know if updates-testing is enabled during the rpm-ostree install ... later on Fedora CoreOS Next? Is it? I am guessing that it's not in Stable? If it's not enabled, then there will be a difference between the builds seen by rpm-ostree override replace ... and rpm-ostree install ..., and the problem will be back.

I suppose we could find a way to conditionally use updates-testing depending on the Fedora CoreOS stream, but my rudimentary Ansible skills are preventing me from conjuring up a quick solution for that. :)

So, if you want to fix the current problem with the minimum amount of fuss, and I am fine with that, then we should drop updates-testing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But some updates were in updates-testing (glibc) and some were in fedora (libgcc). See the below:

output
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="fedora" glibc glibc-common glibc-minimal-langpack glibc-gconv-extra
error: Parsing override source 'fedora': Not in KIND=NAME format
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=fedora" glibc glibc-common glibc-minimal-langpack glibc-gconv-extra
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Updating metadata for 'fedora-cisco-openh264'... done
Updating metadata for 'updates-testing'... done
Updating metadata for 'updates'... done
Updating metadata for 'fedora'... done
Updating metadata for 'updates-archive'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264'; generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing'; generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates'; generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora'; generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive'; generated: 2026-08-21T13:38:41Z solvables: 0
Resolving dependencies... done
error: Request to reinstall exact base package versions: glibc glibc-common glibc-gconv-extra glibc-minimal-langpack
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=updates-testing" glibc glibc-common glibc-minimal-langpack glibc-gconv-extra
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing' (cached); generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora' (cached); generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive' (cached); generated: 2026-08-21T13:38:41Z solvables: 0
Resolving dependencies... done
Will download: 4 packages (4.9 MB)
Downloading from 'updates-testing'... done
Importing packages... done
Applying 4 overrides
Processing packages... done
Running systemd-sysusers... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Upgraded:
  glibc 2.44-1.fc45 -> 2.44-2.fc45
  glibc-common 2.44-1.fc45 -> 2.44-2.fc45
  glibc-gconv-extra 2.44-1.fc45 -> 2.44-2.fc45
  glibc-minimal-langpack 2.44-1.fc45 -> 2.44-2.fc45
Run "systemctl reboot" to start a reboot

As for the install step, no updates-testing isn't in stable, but do the repos here not mean that updates-testing is being used in next?

[core@cosa-devsh ~]$ sudo rpm-ostree install gcc
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing' (cached); generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora' (cached); generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive' (cached); generated: 2026-08-21T13:38:41Z solvables: 0
Resolving dependencies... done
error: Could not depsolve transaction; 1 problem detected:
 Problem: libgcc-16.2.1-2.fc45.1.i686 from fedora does not belong to a distupgrade repository
  - package gcc-16.2.1-2.fc45.1.x86_64 from fedora requires libgcc >= 16.2.1-2.fc45.1, but none of the providers can be installed
  - cannot install both libgcc-16.2.1-2.fc45.1.x86_64 from fedora and libgcc-16.2.1-2.fc45.x86_64 from @System
  - conflicting requests

Could you explain where the difference would come from on stable even? The updates-testing repo just fails:

[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=updates-testing" glibc glibc-common glibc-minimal-langpack glibc-gconv-extra
Checking out tree 2c45d70... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora updates-archive
Updating metadata for 'fedora-cisco-openh264'... done
Updating metadata for 'updates'... done
Updating metadata for 'fedora'... done
Updating metadata for 'updates-archive'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264'; generated: 2026-03-24T10:37:04Z solvables: 3
rpm-md repo 'updates'; generated: 2026-10-13T01:05:00Z solvables: 30441
rpm-md repo 'fedora'; generated: 2026-04-22T13:34:32Z solvables: 76354
rpm-md repo 'updates-archive'; generated: 2026-10-12T03:51:56Z solvables: 47351
error: No matches for 'glibc' in repo 'updates-testing'

Apologies if I'm misunderstanding anything.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this also shows we still need updates-testing:

output
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=fedora" libgcc
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Updating metadata for 'fedora-cisco-openh264'... done
Updating metadata for 'updates-testing'... done
Updating metadata for 'updates'... done
Updating metadata for 'fedora'... done
Updating metadata for 'updates-archive'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264'; generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing'; generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates'; generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora'; generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive'; generated: 2026-08-21T13:38:41Z solvables: 0
Resolving dependencies... done
Will download: 1 package (139.8 kB)
Downloading from 'fedora'... done
Importing packages... done
Applying 1 override
Processing packages... done
Running systemd-sysusers... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Upgraded:
  libgcc 16.2.1-2.fc45 -> 16.2.1-2.fc45.1
Run "systemctl reboot" to start a reboot
[core@cosa-devsh ~]$ sudo     rpm-ostree install --allow-inactive --apply-live --assumeyes --idempotent ShellCheck bats codespell gcc golang golang-github-cpuguy83-md2man httpd-tools meson ninja-build shadow-utils-subid-devel udisks2
Inactive requests:
  udisks2 (already provided by udisks2-2.11.2-1.fc45.x86_64)
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing' (cached); generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora' (cached); generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive' (cached); generated: 2026-08-21T13:38:41Z solvables: 0
Resolving dependencies... done
Will download: 99 packages (143.9 MB)
Downloading from 'updates-testing'... done
Downloading from 'fedora'... done
Importing packages... done
Applying 1 override and 99 overlays
Processing packages... done
Running systemd-sysusers... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Freed: 31.1 MB (pkgcache branches: 0)
error: packages would be changed: 2, allow replacement to override
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=fedora" glibc glibc-common glibc-minimal-langpack glibc-gconv-extra
Inactive requests:
  udisks2 (already provided by udisks2-2.11.2-1.fc45.x86_64)
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing' (cached); generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora' (cached); generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive' (cached); generated: 2026-08-21T13:38:41Z solvables: 0
Resolving dependencies... done
error: Request to reinstall exact base package versions: glibc glibc-common glibc-gconv-extra glibc-minimal-langpack
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=updates" glibc glibc-common glibc-minimal-langpack glibc-gconv-extra
Inactive requests:
  udisks2 (already provided by udisks2-2.11.2-1.fc45.x86_64)
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing' (cached); generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora' (cached); generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive' (cached); generated: 2026-08-21T13:38:41Z solvables: 0
error: No matches for 'glibc' in repo 'updates'
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=updates-testing" glibc glibc-common glibc-minimal-langpack glibc-gconv-extra
Inactive requests:
  udisks2 (already provided by udisks2-2.11.2-1.fc45.x86_64)
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing' (cached); generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora' (cached); generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive' (cached); generated: 2026-08-21T13:38:41Z solvables: 0
Resolving dependencies... done
Will download: 5 packages (5.6 MB)
Downloading from 'updates-testing'... done
Importing packages... done
Applying 5 overrides and 99 overlays
Processing packages... done
Running systemd-sysusers... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Freed: 38.3 MB (pkgcache branches: 1)
Upgraded:
  glibc 2.44-1.fc45 -> 2.44-2.fc45
  glibc-common 2.44-1.fc45 -> 2.44-2.fc45
  glibc-gconv-extra 2.44-1.fc45 -> 2.44-2.fc45
  glibc-minimal-langpack 2.44-1.fc45 -> 2.44-2.fc45
  libgcc 16.2.1-2.fc45 -> 16.2.1-2.fc45.1
Added:
  ShellCheck-0.11.0-5.fc45.x86_64
  apr-1.7.6-6.fc45.x86_64
  apr-util-1.6.5-2.fc45.x86_64
  apr-util-lmdb-1.6.5-2.fc45.x86_64
  apr-util-openssl-1.6.5-2.fc45.x86_64
  bats-1.14.0-1.fc45.noarch
  binutils-2.47-1.fc45.x86_64
  codespell-2.4.2-2.fc45.noarch
  cpp-16.2.1-2.fc45.1.x86_64
  elfutils-debuginfod-client-0.196-1.fc45.x86_64
  gcc-16.2.1-2.fc45.1.x86_64
  glibc-devel-2.44-2.fc45.x86_64
  go-filesystem-3.8.0-3.fc45.x86_64
  go-md2man-2.0.7-2.fc45.x86_64
  golang-1.27.2-1.fc45.x86_64
  golang-bin-1.27.2-1.fc45.x86_64
  golang-src-1.27.2-1.fc45.noarch
  groff-base-1.24.1-2.fc45.x86_64
  httpd-tools-2.4.69-1.fc45.x86_64
  kernel-headers-7.2.4-300.fc45.x86_64
  libatomic-16.2.1-2.fc45.1.x86_64
  libgomp-16.2.1-2.fc45.1.x86_64
  libmpc-1.4.1-2.fc45.x86_64
  libxcrypt-devel-4.5.2-4.fc45.x86_64
  make-1:4.4.1-13.fc45.x86_64
  meson-1.12.1-2.fc45.noarch
  meson-srpm-macros-1.12.1-2.fc45.noarch
  mpdecimal-4.0.1-4.fc45.x86_64
  ninja-build-1.13.2-6.fc45.x86_64
  parallel-20260822-2.fc45.noarch
  perl-AutoLoader-5.74-527.fc45.noarch
  perl-B-1.92-527.fc45.x86_64
  perl-Carp-1.54-527.fc45.noarch
  perl-Class-Struct-0.68-527.fc45.noarch
  perl-Data-Dumper-2.192-528.fc45.x86_64
  perl-Digest-1.20-527.fc45.noarch
  perl-Digest-MD5-2.59-527.fc45.x86_64
  perl-DynaLoader-1.57-527.fc45.x86_64
  perl-Encode-4:3.24-527.fc45.x86_64
  perl-Errno-1.39-527.fc45.x86_64
  perl-Exporter-5.79-527.fc45.noarch
  perl-Fcntl-1.20-527.fc45.x86_64
  perl-File-Basename-2.86-527.fc45.noarch
  perl-File-Path-2.18-527.fc45.noarch
  perl-File-Temp-1:0.231.200-527.fc45.noarch
  perl-File-stat-1.15-527.fc45.noarch
  perl-FileHandle-2.05-527.fc45.noarch
  perl-Getopt-Long-1:2.58-527.fc45.noarch
  perl-Getopt-Std-1.15-527.fc45.noarch
  perl-HTTP-Tiny-0.096-528.fc45.noarch
  perl-IO-1.56-527.fc45.x86_64
  perl-IO-Socket-IP-0.44-527.fc45.noarch
  perl-IO-Socket-SSL-2.099-2.fc45.noarch
  perl-IPC-Open3-1.24-527.fc45.noarch
  perl-MIME-Base32-1.303-26.fc45.noarch
  perl-MIME-Base64-3.16-527.fc45.x86_64
  perl-NDBM_File-1.18-527.fc45.x86_64
  perl-Net-SSLeay-1.96-5.fc45.x86_64
  perl-POSIX-2.26-527.fc45.x86_64
  perl-PathTools-3.95-527.fc45.x86_64
  perl-Pod-Escapes-1:1.07-527.fc45.noarch
  perl-Pod-Perldoc-3.28.01-527.fc45.noarch
  perl-Pod-Simple-1:3.48-527.fc45.noarch
  perl-Pod-Usage-4:2.05-527.fc45.noarch
  perl-Scalar-List-Utils-5:1.70-527.fc45.x86_64
  perl-SelectSaver-1.03-527.fc45.noarch
  perl-Socket-4:2.043-1.fc45.x86_64
  perl-Storable-1:3.41-527.fc45.x86_64
  perl-Symbol-1.09-527.fc45.noarch
  perl-Term-ANSIColor-5.01-527.fc45.noarch
  perl-Term-Cap-1.18-527.fc45.noarch
  perl-Text-ParseWords-3.31-527.fc45.noarch
  perl-Text-Tabs+Wrap-2024.001-527.fc45.noarch
  perl-Thread-Queue-3.14-527.fc45.noarch
  perl-Time-HiRes-4:1.9780-527.fc45.x86_64
  perl-Time-Local-2:1.350-527.fc45.noarch
  perl-URI-5.37-1.fc45.noarch
  perl-Unicode-Normalize-1.32-527.fc45.x86_64
  perl-base-2.27-527.fc45.noarch
  perl-constant-1.33-528.fc45.noarch
  perl-if-0.61.000-527.fc45.noarch
  perl-interpreter-4:5.44.0-527.fc45.x86_64
  perl-libnet-3.15-528.fc45.noarch
  perl-libs-4:5.44.0-527.fc45.x86_64
  perl-locale-1.13-527.fc45.noarch
  perl-mro-1.30-527.fc45.x86_64
  perl-overload-1.40-527.fc45.noarch
  perl-overloading-0.03-527.fc45.noarch
  perl-parent-1:0.244-527.fc45.noarch
  perl-podlators-1:6.1.1-1.fc45.noarch
  perl-threads-1:2.45-527.fc45.x86_64
  perl-threads-shared-1.73-527.fc45.x86_64
  perl-vars-1.05-527.fc45.noarch
  python-pip-wheel-26.1.2-5.fc45.noarch
  python-unversioned-command-3.15.0-1.fc45.noarch
  python3-3.15.0-1.fc45.x86_64
  python3-libs-3.15.0-1.fc45.x86_64
  shadow-utils-subid-devel-2:4.20.0-1.fc45.x86_64
  vim-filesystem-2:9.2.1129-1.fc45.noarch
Run "systemctl reboot" to start a reboot
[core@cosa-devsh ~]$ sudo     rpm-ostree install --allow-inactive --apply-live --assumeyes --idempotent ShellCheck bats codespell gcc golang golang-github-cpuguy83-md2man httpd-tools meson ninja-build shadow-utils-subid-devel udisks2
Inactive requests:
  udisks2 (already provided by udisks2-2.11.2-1.fc45.x86_64)
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing' (cached); generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora' (cached); generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive' (cached); generated: 2026-08-21T13:38:41Z solvables: 0
Resolving dependencies... done
No change.

@debarshiray debarshiray Oct 14, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A quick reply to one of your questions, because I am running short on time.

But some updates were in updates-testing (glibc) and some were in fedora (libgcc). See the below:

As far as I can make out, the glibc builds in updates-testing aren't causing problems here. Those are missing updates and the CI will run with what's in fedora and updates-testing.

The problem here is that the libgcc in the Fedora CoreOS Next image is older than the gcc in the fedora repositories, and an attempt to rpm-ostree install gcc fails due to this mismatch. Probably, because the image was created from an older snapshot of builds, and just this gets it going:

$ rpm-ostree override replace --experimental --from repo=fedora libgcc

See #1851 for a full CI run with logs. Of course, we can't literally do what's in that throwaway PR. It's just a demo.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not entirely sure why the CI ends up behaving differently than #1847 (comment) (maybe different next image) but I think the output in that comment shows that it can be an issue if we don't take updates from updates-testing


- name: Synchronize shadow-utils to repo version for shadow-utils-subid-devel compatibility
- name: Synchronize base packages to repo versions
become: yes
command: rpm-ostree override replace --experimental --from repo=updates shadow-utils
shell: |
for repo in updates updates-testing fedora; do

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not fully sure about the rpm-ostree override replace semantics, and I am wondering if the order could/should be:

fedora updates updates-testing

... to match the ascending order of builds - fedora has the oldest builds, followed by updates and updates-testing has the newest.

Train of thought follows below.

output=$(rpm-ostree override replace --experimental --from repo="$repo" {{ item.packages }} 2>&1)
rc=$?

if [ $rc -eq 0 ]; then
echo "$output"
exit 0
fi
echo "$output" >&2

# Already at the latest version, nothing to do.
if echo "$output" | grep -qi 'reinstall exact'; then
exit 0
fi

# Package not found in this repo, try the next one.
if echo "$output" | grep -qi 'no matches'; then
continue
fi

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Train of thought follows from above.

Error handling, loops, and standard error/output stream handling in shell scripts stress me out under the best of conditions, and this is an Ansible playbook that's difficult to test locally. :)

So, I am wondering if something like this will actually work:

    for repo in fedora updates; do
      rpm-ostree override replace --experimental --from repo="$repo" {{ item.packages }} 2>&1
    done

It can be slower because it will always run rpm-ostree multiple times for a package, and not terminate the loop on success. The advantage is that it is so simple to understand that we won't have to worry about testing the error handling and grepping for sub-strings in the output. The test suite does so much disk and network I/O that the extra rpm-ostree invocations will be lost in the noise.

The problem is, I don't know if rpm-ostree override replace semantics match my expectations. I am hoping that first it will add an override for an older build from fedora, and then overwrite it with a newer one from updates. Is that how it works?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it does work like that yes. I'm not entirely sure how to confirm. It does seem to behave well for the series of overrides we're doing there though.

I'll make that change and we'll see what CI says.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, that's annoying. Doesn't look like it does the right thing:

[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=fedora" libgcc
Checking out tree 2c45d70... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:37:04Z solvables: 3
rpm-md repo 'updates' (cached); generated: 2026-10-13T01:05:00Z solvables: 30441
rpm-md repo 'fedora' (cached); generated: 2026-04-22T13:34:32Z solvables: 76354
rpm-md repo 'updates-archive' (cached); generated: 2026-10-12T03:51:56Z solvables: 47351
Resolving dependencies... done
Will download: 1 package (129.3 kB)
Downloading from 'fedora'... done
Importing packages... done
Applying 1 override
Processing packages... done
Running systemd-sysusers... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Downgraded:
  libgcc 16.2.1-2.fc44 -> 16.0.1-0.10.fc44
Run "systemctl reboot" to start a reboot
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=updates" libgcc
Checking out tree 2c45d70... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:37:04Z solvables: 3
rpm-md repo 'updates' (cached); generated: 2026-10-13T01:05:00Z solvables: 30441
rpm-md repo 'fedora' (cached); generated: 2026-04-22T13:34:32Z solvables: 76354
rpm-md repo 'updates-archive' (cached); generated: 2026-10-12T03:51:56Z solvables: 47351
Resolving dependencies... done
Applying 1 override
Processing packages... done
Running systemd-sysusers... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Freed: 29.2 MB (pkgcache branches: 0)
Downgraded:
  libgcc 16.2.1-2.fc44 -> 16.0.1-0.10.fc44
Run "systemctl reboot" to start a reboot
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=updates-testing" libgcc
Checking out tree 2c45d70... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:37:04Z solvables: 3
rpm-md repo 'updates' (cached); generated: 2026-10-13T01:05:00Z solvables: 30441
rpm-md repo 'fedora' (cached); generated: 2026-04-22T13:34:32Z solvables: 76354
rpm-md repo 'updates-archive' (cached); generated: 2026-10-12T03:51:56Z solvables: 47351
Resolving dependencies... done
Applying 1 override
Processing packages... done
Running systemd-sysusers... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Freed: 29.2 MB (pkgcache branches: 0)
Downgraded:
  libgcc 16.2.1-2.fc44 -> 16.0.1-0.10.fc44
Run "systemctl reboot" to start a reboot

That's on an F44 node


# Other error, bail out.
exit $rc
done
exit $rc
loop:
- { name: glibc, packages: "glibc glibc-common glibc-minimal-langpack glibc-gconv-extra" }
- { name: shadow-utils, packages: "shadow-utils" }
- { name: libgcc, packages: "libgcc" }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick. Since we are grouping things by the same source package, we should probably use name: gcc, because libgcc is a subpackage built from the gcc source package.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just thought it might be confusing since we're installing gcc below

loop_control:
label: "{{ item.name }}"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it doing anything? :)

The documentation says:

When looping over complex data structures, the console
output of your task can be enormous. To limit the displayed
output, use the label directive with loop_control.

However, it doesn't seem to make any difference to the console output.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just seemed correct to include, but we can drop it I guess

ignore_errors: yes

- name: Apply base package overrides
Expand Down
Loading