Repository navigation
playbooks/depencies-fedora-coreos: add a fallback to the updates-testing repo - #1847
Rolv-Apneseth wants to merge 2 commits into
Conversation
2773b57 to
51c1a42
Compare
51c1a42 to
9cc2e16
Compare
There was a problem hiding this comment.
Thanks for putting this together, @Rolv-Apneseth ! I don't think I understand the details of Fedora CoreOS Next, so let me ask what might be a stupid question. Does CoreOS have different updates and updates-testing repositories than the usual Fedora ones? If not, then I am a bit confused about what's going on.
First, we have these:
$ rpm-ostree override replace --experimental --from repo=updates glibc ...
...
error: No matches for 'glibc' in repo 'updates'
$ rpm-ostree override replace --experimental --from repo=updates shadow-utils
...
error: No matches for 'shadow-utils' in repo 'updates'From what you wrote, it seems that this is related to Fedora 45 being in Final Freeze from the 6th of October, which has caused some package updates to be stuck in the updates-testing repository. That makes sense, and explains the above based on what I can see in Bodhi and Koji.
These errors aren't fatal, because we are using ignore_errors: yes for those in the Ansible playbook.
Second, we have:
$ rpm-ostree install --allow-inactive --apply-live --assumeyes --idempotent gcc
...
error: Could not depsolve transaction; 1 problem detected:
Problem: libgcc-16.2.1-2.fc45.1.i686 from fedora does not belong to a distupgrade repository
- package gcc-16.2.1-2.fc45.1.x86_64 from fedora requires libgcc >= 16.2.1-2.fc45.1, but none of the providers can be installed
- cannot install both libgcc-16.2.1-2.fc45.1.x86_64 from fedora and libgcc-16.2.1-2.fc45.x86_64 from @System
- conflicting requestsThis is the part I don't understand, and this error is fatal because we don't use ignore_errors: yes for it.
The gcc-16.2.1-2.fc45.1 build was pushed to stable five days ago, and both the gcc and libgcc binary packages are built from the same gcc source. So, how can gcc and libgcc have a broken dependency chain between them? :)
Anyway, if the CI passes, then that's the final word. :)
|
So, I'm picking this up as I go, but my understanding is:
They are the same.
So for the install step, any package already installed on the system can't be updated automatically due to how rpm-ostree works, which is why we have the override steps. And we ignore the errors in case there are no updates. But you're right to ask, I missed the fact it's |
|
I think coreos/rpm-ostree#4708 would make things a lot easier here |
Thanks for explaining that detail about
Yes, if Take a step back from this specific problem, I am wondering if there's ever a valid reason for the |
9cc2e16 to
4d86f14
Compare
|
Trying something different, we'll see how CI goes.
I'm afraid I don't know. I don't have enough knowledge on the topics of Fedora package repos. |
|
Ok, seems like the fallbacks are working as expected in https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/local/build/91b9e465a1264a79bff668732ab506af/log/job-output.txt: outputHowever, looks like there's another issue in the |
debarshiray
left a comment
debarshiray
left a comment
There was a problem hiding this comment.
If libgcc is in the Fedora CoreOS host image, and it breaks rpm-ostree install ... gcc because it's older than the version in the repositories, then wouldn't it be enough to just do:
diff --git a/playbooks/dependencies-fedora-coreos.yaml b/playbooks/dependencies-fedora-coreos.yaml
index d676cb9f4175..27e7f5862134 100644
--- a/playbooks/dependencies-fedora-coreos.yaml
+++ b/playbooks/dependencies-fedora-coreos.yaml
@@ -22,6 +22,7 @@
glibc-common
glibc-minimal-langpack
glibc-gconv-extra
+ libgcc
ignore_errors: yes
- name: Synchronize shadow-utils to repo version for shadow-utils-subid-devel compatibility
Completely untested, because I didn't want to step on your toes.
Looks like this is coreos/fedora-coreos-tracker#2238 - can maybe be fixed in systemd, but likely will be a change needed on our side. If you want to move forward here, I think
We want them separate because a lack of an update in one package will fail the whole command, so we can only really group packages that are from the same SRPM. The current changes seem to pass that stage without issues, if you're happy with them? |
Sounds good to me, yes. Do you want to throw a patch in here?
Aha, I didn't know that. |
Sure yes, I'm working on understanding the upstream issue |
4d86f14 to
23a2a2b
Compare
|
Rebased and pushed a commit for that @debarshiray. Hopefully that lets the CI here avoid the issue. |
debarshiray
left a comment
debarshiray
left a comment
There was a problem hiding this comment.
I pulled out my Fedora 44 Silverblue laptop to play with rpm-ostree.
The lack of full-fledged manual pages for each sub-command always trips me up, and I wish --experimental and --from weren't hidden options. :)
It would have made things simpler if --from could be specified multiple times to enable multiple RPM repositories. It seems that currently, it can be specified multiple times and only the last one takes effect.
I used a throwaway pull request at #1851 to briefly test my theories and thoughts below. Feel free to use it for your own experiments.
| loop: | ||
| - { name: glibc, packages: "glibc glibc-common glibc-minimal-langpack glibc-gconv-extra" } | ||
| - { name: shadow-utils, packages: "shadow-utils" } | ||
| - { name: libgcc, packages: "libgcc" } |
There was a problem hiding this comment.
Nitpick. Since we are grouping things by the same source package, we should probably use name: gcc, because libgcc is a subpackage built from the gcc source package.
There was a problem hiding this comment.
I just thought it might be confusing since we're installing gcc below
| - { name: shadow-utils, packages: "shadow-utils" } | ||
| - { name: libgcc, packages: "libgcc" } | ||
| loop_control: | ||
| label: "{{ item.name }}" |
There was a problem hiding this comment.
Is it doing anything? :)
The documentation says:
When looping over complex data structures, the console
output of your task can be enormous. To limit the displayed
output, use the label directive with loop_control.
However, it doesn't seem to make any difference to the console output.
There was a problem hiding this comment.
Just seemed correct to include, but we can drop it I guess
| # `next` stream pointing at pre-release repos during branching. | ||
| # With the below loop, we try and use the default `updates` repo, | ||
| # and fallback to `updates-testing` and `fedora` in turn so that we | ||
| # can update base packages on the system. |
There was a problem hiding this comment.
Didn't the updates-testing aspect turn out to be a red herring? In this case the problem is that the Fedora CoreOS host image has an older libgcc from an older gcc build and the fedora repository has a newer one, and this trips up the rpm install ... gcc later.
It seems worth it to include fedora, because during freezes and right after a release, the ISOs and images might have older builds than fedora. The problem might go away once a release gets older and newer builds make their way to updates.
More importantly, I don't know if updates-testing is enabled during the rpm-ostree install ... later on Fedora CoreOS Next? Is it? I am guessing that it's not in Stable? If it's not enabled, then there will be a difference between the builds seen by rpm-ostree override replace ... and rpm-ostree install ..., and the problem will be back.
I suppose we could find a way to conditionally use updates-testing depending on the Fedora CoreOS stream, but my rudimentary Ansible skills are preventing me from conjuring up a quick solution for that. :)
So, if you want to fix the current problem with the minimum amount of fuss, and I am fine with that, then we should drop updates-testing.
There was a problem hiding this comment.
But some updates were in updates-testing (glibc) and some were in fedora (libgcc). See the below:
output
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="fedora" glibc glibc-common glibc-minimal-langpack glibc-gconv-extra
error: Parsing override source 'fedora': Not in KIND=NAME format
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=fedora" glibc glibc-common glibc-minimal-langpack glibc-gconv-extra
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Updating metadata for 'fedora-cisco-openh264'... done
Updating metadata for 'updates-testing'... done
Updating metadata for 'updates'... done
Updating metadata for 'fedora'... done
Updating metadata for 'updates-archive'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264'; generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing'; generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates'; generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora'; generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive'; generated: 2026-08-21T13:38:41Z solvables: 0
Resolving dependencies... done
error: Request to reinstall exact base package versions: glibc glibc-common glibc-gconv-extra glibc-minimal-langpack
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=updates-testing" glibc glibc-common glibc-minimal-langpack glibc-gconv-extra
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing' (cached); generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora' (cached); generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive' (cached); generated: 2026-08-21T13:38:41Z solvables: 0
Resolving dependencies... done
Will download: 4 packages (4.9 MB)
Downloading from 'updates-testing'... done
Importing packages... done
Applying 4 overrides
Processing packages... done
Running systemd-sysusers... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Upgraded:
glibc 2.44-1.fc45 -> 2.44-2.fc45
glibc-common 2.44-1.fc45 -> 2.44-2.fc45
glibc-gconv-extra 2.44-1.fc45 -> 2.44-2.fc45
glibc-minimal-langpack 2.44-1.fc45 -> 2.44-2.fc45
Run "systemctl reboot" to start a rebootAs for the install step, no updates-testing isn't in stable, but do the repos here not mean that updates-testing is being used in next?
[core@cosa-devsh ~]$ sudo rpm-ostree install gcc
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing' (cached); generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora' (cached); generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive' (cached); generated: 2026-08-21T13:38:41Z solvables: 0
Resolving dependencies... done
error: Could not depsolve transaction; 1 problem detected:
Problem: libgcc-16.2.1-2.fc45.1.i686 from fedora does not belong to a distupgrade repository
- package gcc-16.2.1-2.fc45.1.x86_64 from fedora requires libgcc >= 16.2.1-2.fc45.1, but none of the providers can be installed
- cannot install both libgcc-16.2.1-2.fc45.1.x86_64 from fedora and libgcc-16.2.1-2.fc45.x86_64 from @System
- conflicting requests
Could you explain where the difference would come from on stable even? The updates-testing repo just fails:
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=updates-testing" glibc glibc-common glibc-minimal-langpack glibc-gconv-extra
Checking out tree 2c45d70... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora updates-archive
Updating metadata for 'fedora-cisco-openh264'... done
Updating metadata for 'updates'... done
Updating metadata for 'fedora'... done
Updating metadata for 'updates-archive'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264'; generated: 2026-03-24T10:37:04Z solvables: 3
rpm-md repo 'updates'; generated: 2026-10-13T01:05:00Z solvables: 30441
rpm-md repo 'fedora'; generated: 2026-04-22T13:34:32Z solvables: 76354
rpm-md repo 'updates-archive'; generated: 2026-10-12T03:51:56Z solvables: 47351
error: No matches for 'glibc' in repo 'updates-testing'
Apologies if I'm misunderstanding anything.
There was a problem hiding this comment.
I think this also shows we still need updates-testing:
output
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=fedora" libgcc
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Updating metadata for 'fedora-cisco-openh264'... done
Updating metadata for 'updates-testing'... done
Updating metadata for 'updates'... done
Updating metadata for 'fedora'... done
Updating metadata for 'updates-archive'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264'; generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing'; generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates'; generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora'; generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive'; generated: 2026-08-21T13:38:41Z solvables: 0
Resolving dependencies... done
Will download: 1 package (139.8 kB)
Downloading from 'fedora'... done
Importing packages... done
Applying 1 override
Processing packages... done
Running systemd-sysusers... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Upgraded:
libgcc 16.2.1-2.fc45 -> 16.2.1-2.fc45.1
Run "systemctl reboot" to start a reboot
[core@cosa-devsh ~]$ sudo rpm-ostree install --allow-inactive --apply-live --assumeyes --idempotent ShellCheck bats codespell gcc golang golang-github-cpuguy83-md2man httpd-tools meson ninja-build shadow-utils-subid-devel udisks2
Inactive requests:
udisks2 (already provided by udisks2-2.11.2-1.fc45.x86_64)
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing' (cached); generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora' (cached); generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive' (cached); generated: 2026-08-21T13:38:41Z solvables: 0
Resolving dependencies... done
Will download: 99 packages (143.9 MB)
Downloading from 'updates-testing'... done
Downloading from 'fedora'... done
Importing packages... done
Applying 1 override and 99 overlays
Processing packages... done
Running systemd-sysusers... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Freed: 31.1 MB (pkgcache branches: 0)
error: packages would be changed: 2, allow replacement to override
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=fedora" glibc glibc-common glibc-minimal-langpack glibc-gconv-extra
Inactive requests:
udisks2 (already provided by udisks2-2.11.2-1.fc45.x86_64)
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing' (cached); generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora' (cached); generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive' (cached); generated: 2026-08-21T13:38:41Z solvables: 0
Resolving dependencies... done
error: Request to reinstall exact base package versions: glibc glibc-common glibc-gconv-extra glibc-minimal-langpack
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=updates" glibc glibc-common glibc-minimal-langpack glibc-gconv-extra
Inactive requests:
udisks2 (already provided by udisks2-2.11.2-1.fc45.x86_64)
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing' (cached); generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora' (cached); generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive' (cached); generated: 2026-08-21T13:38:41Z solvables: 0
error: No matches for 'glibc' in repo 'updates'
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=updates-testing" glibc glibc-common glibc-minimal-langpack glibc-gconv-extra
Inactive requests:
udisks2 (already provided by udisks2-2.11.2-1.fc45.x86_64)
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing' (cached); generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora' (cached); generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive' (cached); generated: 2026-08-21T13:38:41Z solvables: 0
Resolving dependencies... done
Will download: 5 packages (5.6 MB)
Downloading from 'updates-testing'... done
Importing packages... done
Applying 5 overrides and 99 overlays
Processing packages... done
Running systemd-sysusers... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Freed: 38.3 MB (pkgcache branches: 1)
Upgraded:
glibc 2.44-1.fc45 -> 2.44-2.fc45
glibc-common 2.44-1.fc45 -> 2.44-2.fc45
glibc-gconv-extra 2.44-1.fc45 -> 2.44-2.fc45
glibc-minimal-langpack 2.44-1.fc45 -> 2.44-2.fc45
libgcc 16.2.1-2.fc45 -> 16.2.1-2.fc45.1
Added:
ShellCheck-0.11.0-5.fc45.x86_64
apr-1.7.6-6.fc45.x86_64
apr-util-1.6.5-2.fc45.x86_64
apr-util-lmdb-1.6.5-2.fc45.x86_64
apr-util-openssl-1.6.5-2.fc45.x86_64
bats-1.14.0-1.fc45.noarch
binutils-2.47-1.fc45.x86_64
codespell-2.4.2-2.fc45.noarch
cpp-16.2.1-2.fc45.1.x86_64
elfutils-debuginfod-client-0.196-1.fc45.x86_64
gcc-16.2.1-2.fc45.1.x86_64
glibc-devel-2.44-2.fc45.x86_64
go-filesystem-3.8.0-3.fc45.x86_64
go-md2man-2.0.7-2.fc45.x86_64
golang-1.27.2-1.fc45.x86_64
golang-bin-1.27.2-1.fc45.x86_64
golang-src-1.27.2-1.fc45.noarch
groff-base-1.24.1-2.fc45.x86_64
httpd-tools-2.4.69-1.fc45.x86_64
kernel-headers-7.2.4-300.fc45.x86_64
libatomic-16.2.1-2.fc45.1.x86_64
libgomp-16.2.1-2.fc45.1.x86_64
libmpc-1.4.1-2.fc45.x86_64
libxcrypt-devel-4.5.2-4.fc45.x86_64
make-1:4.4.1-13.fc45.x86_64
meson-1.12.1-2.fc45.noarch
meson-srpm-macros-1.12.1-2.fc45.noarch
mpdecimal-4.0.1-4.fc45.x86_64
ninja-build-1.13.2-6.fc45.x86_64
parallel-20260822-2.fc45.noarch
perl-AutoLoader-5.74-527.fc45.noarch
perl-B-1.92-527.fc45.x86_64
perl-Carp-1.54-527.fc45.noarch
perl-Class-Struct-0.68-527.fc45.noarch
perl-Data-Dumper-2.192-528.fc45.x86_64
perl-Digest-1.20-527.fc45.noarch
perl-Digest-MD5-2.59-527.fc45.x86_64
perl-DynaLoader-1.57-527.fc45.x86_64
perl-Encode-4:3.24-527.fc45.x86_64
perl-Errno-1.39-527.fc45.x86_64
perl-Exporter-5.79-527.fc45.noarch
perl-Fcntl-1.20-527.fc45.x86_64
perl-File-Basename-2.86-527.fc45.noarch
perl-File-Path-2.18-527.fc45.noarch
perl-File-Temp-1:0.231.200-527.fc45.noarch
perl-File-stat-1.15-527.fc45.noarch
perl-FileHandle-2.05-527.fc45.noarch
perl-Getopt-Long-1:2.58-527.fc45.noarch
perl-Getopt-Std-1.15-527.fc45.noarch
perl-HTTP-Tiny-0.096-528.fc45.noarch
perl-IO-1.56-527.fc45.x86_64
perl-IO-Socket-IP-0.44-527.fc45.noarch
perl-IO-Socket-SSL-2.099-2.fc45.noarch
perl-IPC-Open3-1.24-527.fc45.noarch
perl-MIME-Base32-1.303-26.fc45.noarch
perl-MIME-Base64-3.16-527.fc45.x86_64
perl-NDBM_File-1.18-527.fc45.x86_64
perl-Net-SSLeay-1.96-5.fc45.x86_64
perl-POSIX-2.26-527.fc45.x86_64
perl-PathTools-3.95-527.fc45.x86_64
perl-Pod-Escapes-1:1.07-527.fc45.noarch
perl-Pod-Perldoc-3.28.01-527.fc45.noarch
perl-Pod-Simple-1:3.48-527.fc45.noarch
perl-Pod-Usage-4:2.05-527.fc45.noarch
perl-Scalar-List-Utils-5:1.70-527.fc45.x86_64
perl-SelectSaver-1.03-527.fc45.noarch
perl-Socket-4:2.043-1.fc45.x86_64
perl-Storable-1:3.41-527.fc45.x86_64
perl-Symbol-1.09-527.fc45.noarch
perl-Term-ANSIColor-5.01-527.fc45.noarch
perl-Term-Cap-1.18-527.fc45.noarch
perl-Text-ParseWords-3.31-527.fc45.noarch
perl-Text-Tabs+Wrap-2024.001-527.fc45.noarch
perl-Thread-Queue-3.14-527.fc45.noarch
perl-Time-HiRes-4:1.9780-527.fc45.x86_64
perl-Time-Local-2:1.350-527.fc45.noarch
perl-URI-5.37-1.fc45.noarch
perl-Unicode-Normalize-1.32-527.fc45.x86_64
perl-base-2.27-527.fc45.noarch
perl-constant-1.33-528.fc45.noarch
perl-if-0.61.000-527.fc45.noarch
perl-interpreter-4:5.44.0-527.fc45.x86_64
perl-libnet-3.15-528.fc45.noarch
perl-libs-4:5.44.0-527.fc45.x86_64
perl-locale-1.13-527.fc45.noarch
perl-mro-1.30-527.fc45.x86_64
perl-overload-1.40-527.fc45.noarch
perl-overloading-0.03-527.fc45.noarch
perl-parent-1:0.244-527.fc45.noarch
perl-podlators-1:6.1.1-1.fc45.noarch
perl-threads-1:2.45-527.fc45.x86_64
perl-threads-shared-1.73-527.fc45.x86_64
perl-vars-1.05-527.fc45.noarch
python-pip-wheel-26.1.2-5.fc45.noarch
python-unversioned-command-3.15.0-1.fc45.noarch
python3-3.15.0-1.fc45.x86_64
python3-libs-3.15.0-1.fc45.x86_64
shadow-utils-subid-devel-2:4.20.0-1.fc45.x86_64
vim-filesystem-2:9.2.1129-1.fc45.noarch
Run "systemctl reboot" to start a reboot
[core@cosa-devsh ~]$ sudo rpm-ostree install --allow-inactive --apply-live --assumeyes --idempotent ShellCheck bats codespell gcc golang golang-github-cpuguy83-md2man httpd-tools meson ninja-build shadow-utils-subid-devel udisks2
Inactive requests:
udisks2 (already provided by udisks2-2.11.2-1.fc45.x86_64)
Checking out tree 3b7be3f... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:56:00Z solvables: 3
rpm-md repo 'updates-testing' (cached); generated: 2026-10-13T01:52:04Z solvables: 14046
rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'fedora' (cached); generated: 2026-10-12T08:28:25Z solvables: 75855
rpm-md repo 'updates-archive' (cached); generated: 2026-08-21T13:38:41Z solvables: 0
Resolving dependencies... done
No change.There was a problem hiding this comment.
A quick reply to one of your questions, because I am running short on time.
But some updates were in updates-testing (glibc) and some were in fedora (libgcc). See the below:
As far as I can make out, the glibc builds in updates-testing aren't causing problems here. Those are missing updates and the CI will run with what's in fedora and updates-testing.
The problem here is that the libgcc in the Fedora CoreOS Next image is older than the gcc in the fedora repositories, and an attempt to rpm-ostree install gcc fails due to this mismatch. Probably, because the image was created from an older snapshot of builds, and just this gets it going:
$ rpm-ostree override replace --experimental --from repo=fedora libgccSee #1851 for a full CI run with logs. Of course, we can't literally do what's in that throwaway PR. It's just a demo.
There was a problem hiding this comment.
Not entirely sure why the CI ends up behaving differently than #1847 (comment) (maybe different next image) but I think the output in that comment shows that it can be an issue if we don't take updates from updates-testing
| become: yes | ||
| command: rpm-ostree override replace --experimental --from repo=updates shadow-utils | ||
| shell: | | ||
| for repo in updates updates-testing fedora; do |
There was a problem hiding this comment.
I am not fully sure about the rpm-ostree override replace semantics, and I am wondering if the order could/should be:
fedora updates updates-testing
... to match the ascending order of builds - fedora has the oldest builds, followed by updates and updates-testing has the newest.
Train of thought follows below.
| # Package not found in this repo, try the next one. | ||
| if echo "$output" | grep -qi 'no matches'; then | ||
| continue | ||
| fi |
There was a problem hiding this comment.
Train of thought follows from above.
Error handling, loops, and standard error/output stream handling in shell scripts stress me out under the best of conditions, and this is an Ansible playbook that's difficult to test locally. :)
So, I am wondering if something like this will actually work:
for repo in fedora updates; do
rpm-ostree override replace --experimental --from repo="$repo" {{ item.packages }} 2>&1
done
It can be slower because it will always run rpm-ostree multiple times for a package, and not terminate the loop on success. The advantage is that it is so simple to understand that we won't have to worry about testing the error handling and grepping for sub-strings in the output. The test suite does so much disk and network I/O that the extra rpm-ostree invocations will be lost in the noise.
The problem is, I don't know if rpm-ostree override replace semantics match my expectations. I am hoping that first it will add an override for an older build from fedora, and then overwrite it with a newer one from updates. Is that how it works?
There was a problem hiding this comment.
I think it does work like that yes. I'm not entirely sure how to confirm. It does seem to behave well for the series of overrides we're doing there though.
I'll make that change and we'll see what CI says.
There was a problem hiding this comment.
OK, that's annoying. Doesn't look like it does the right thing:
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=fedora" libgcc
Checking out tree 2c45d70... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:37:04Z solvables: 3
rpm-md repo 'updates' (cached); generated: 2026-10-13T01:05:00Z solvables: 30441
rpm-md repo 'fedora' (cached); generated: 2026-04-22T13:34:32Z solvables: 76354
rpm-md repo 'updates-archive' (cached); generated: 2026-10-12T03:51:56Z solvables: 47351
Resolving dependencies... done
Will download: 1 package (129.3 kB)
Downloading from 'fedora'... done
Importing packages... done
Applying 1 override
Processing packages... done
Running systemd-sysusers... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Downgraded:
libgcc 16.2.1-2.fc44 -> 16.0.1-0.10.fc44
Run "systemctl reboot" to start a reboot
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=updates" libgcc
Checking out tree 2c45d70... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:37:04Z solvables: 3
rpm-md repo 'updates' (cached); generated: 2026-10-13T01:05:00Z solvables: 30441
rpm-md repo 'fedora' (cached); generated: 2026-04-22T13:34:32Z solvables: 76354
rpm-md repo 'updates-archive' (cached); generated: 2026-10-12T03:51:56Z solvables: 47351
Resolving dependencies... done
Applying 1 override
Processing packages... done
Running systemd-sysusers... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Freed: 29.2 MB (pkgcache branches: 0)
Downgraded:
libgcc 16.2.1-2.fc44 -> 16.0.1-0.10.fc44
Run "systemctl reboot" to start a reboot
[core@cosa-devsh ~]$ sudo rpm-ostree override replace --experimental --from="repo=updates-testing" libgcc
Checking out tree 2c45d70... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2026-03-24T10:37:04Z solvables: 3
rpm-md repo 'updates' (cached); generated: 2026-10-13T01:05:00Z solvables: 30441
rpm-md repo 'fedora' (cached); generated: 2026-04-22T13:34:32Z solvables: 76354
rpm-md repo 'updates-archive' (cached); generated: 2026-10-12T03:51:56Z solvables: 47351
Resolving dependencies... done
Applying 1 override
Processing packages... done
Running systemd-sysusers... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Freed: 29.2 MB (pkgcache branches: 0)
Downgraded:
libgcc 16.2.1-2.fc44 -> 16.0.1-0.10.fc44
Run "systemctl reboot" to start a rebootThat's on an F44 node
|
I am taking an extended weekend break from Thursday to Monday. I will get back to this when I am back on Tuesday. |
|
No rush, enjoy the break |
…ng and fedora repos When trying to replace the existing versions of some base Fedora CoreOS packages, we had repo=updates hard-coded. However, while the `next` stream is pointing at a pre-release version, updates are either in repo=updates-testing or repo=fedora. Let's add some fallback logic to avoid failures when this is the case. Signed-off-by: Rolv Apneseth <[email protected]>
As of systemd v261, bare `systemd-tmpfiles --create` calls now attempt to set permissions on / on systems which are not set to 0o555. This currently fails on Fedora CoreOS systems due to them having the incorrect permissions set while also having a read-only root filesystem. This leads to the following error: fchmod() of / failed: Read-only file system Since both entries in data/tmpfiles.d/toolbox.conf are under /run, it's safe for us to restrict these calls to this directory with --prefix=/run in order to avoid that issue on FCOS. See: coreos/fedora-coreos-tracker#2238 Signed-off-by: Rolv Apneseth <[email protected]>
23a2a2b to
f74d1ec
Compare
|
@TristanCacqueray could I get logs for the above node failures on coreos-next? #1847 (comment) passed and this was just a rebase so must be a new issue |
When trying to replace the existing versions of some base Fedora CoreOS packages, we had repo=updates hard-coded. However, during branching, the
nextstream will only have updates from repo=updates-testing, so let's add a fallback to avoid failures when this is the case.Testing if this resolves the issues seen in #1846. Note that we need that merged first, then I'll rebase this.