Fedora CoreOS Testing Stream New Release Guide For August 5, 2025

by ADMIN 66 views
Iklan Headers

Hey everyone! Today, we're diving into the process of releasing a new Fedora CoreOS testing stream. This guide outlines all the steps necessary to ensure a smooth and successful release. Let's get started!

First off, make sure you've got all the prerequisites covered. It's crucial to have everything in place before we proceed. Once you're set, let's jump into the details.

Edit the issue title to include today's date. After the pipeline spits out the new version ID, you can append it to the title, like this: (31.20191117.2.0). This helps us keep track of the releases.

Pre-release

Promote testing-devel changes to testing

This is a critical step where we move changes from the development environment to the testing environment. It’s all about ensuring that the features and fixes we've been working on are ready for broader testing. This meticulous process helps us catch potential issues early and maintain the stability of our releases. Let's break down the steps involved in promoting these changes. The goal here is to ensure that the transition from development to testing is as seamless as possible, reducing the risk of unexpected problems down the line.

  • [ ] Add the ok-to-promote label to the issue. This label acts as a signal that the changes are reviewed and deemed ready for promotion. It's a crucial checkpoint in our workflow, preventing premature movement of code.
  • [ ] Review the promotion PR against the testing branch on https://github.com/coreos/fedora-coreos-config. This is where we carefully examine the changes in the pull request against the current state of the testing branch. We're looking for any potential conflicts, regressions, or unexpected behavior. It's a detailed comparison to ensure everything aligns with our expectations. This review process isn't just a formality; it's a critical safeguard that helps us maintain the integrity of our codebase and prevent introducing issues into the testing environment. Every line of code, every configuration change, is scrutinized to ensure it meets our standards and won't disrupt the existing system.
  • [ ] Once CI has passed, merge it. After our continuous integration (CI) system confirms that all automated tests have passed, we proceed to merge the changes into the testing branch. This step signifies that the code not only meets our functional requirements but also adheres to our quality standards. The passing CI is a green light, assuring us that the changes are safe to integrate. However, it's not the end of our checks; it's just one piece of the puzzle. We still rely on human review and manual testing to uncover any subtleties that automated systems might miss. This combination of automated and manual verification gives us a robust approach to quality assurance.
Manual alternative

Sometimes, you might need to run the process manually, especially if you need to add an extra commit to change something in manifest.yaml. Here’s how:

  • git fetch upstream
  • git checkout testing
  • git reset --hard upstream/testing
  • /path/to/fedora-coreos-releng-automation/scripts/promote-config.sh testing-devel
  • Open PR against the testing branch on https://github.com/coreos/fedora-coreos-config

Build

The build process is where we take the code and configurations and turn them into an actual, runnable system image. It's a complex process involving compilers, linkers, and various tools that work together to create the final product. This build step is essential because it's the foundation upon which everything else is built. A flawed build can lead to unstable systems, malfunctioning applications, and a host of other problems. That's why we pay close attention to this stage, ensuring that every component is correctly assembled and optimized for performance. Think of it like constructing a building; if the foundation isn't solid, the entire structure is at risk. Similarly, a well-executed build process ensures that our Fedora CoreOS images are robust, reliable, and ready for deployment.

  • [ ] Start a build job (select testing and enable EARLY_ARCH_JOBS, leave all other defaults). This will automatically run multi-arch builds. The architecture jobs we are dealing with are very critical for system compatibility.
  • Post links to the jobs as a comment to this issue
  • Wait for the jobs to finish and succeed
    • [ ] x86_64
    • [ ] aarch64
    • [ ] ppc64le
    • [ ] s390x

Sanity-check the build

Sanity-checking the build is like giving our newly constructed system a thorough once-over before we let it loose. It’s a crucial phase where we verify that the build behaves as expected, and that no obvious issues have crept in during the process. This step is all about catching the low-hanging fruit – the common errors that can be easily identified with a few quick tests. Think of it as a quick health check for our system. We're not diving into deep, complex testing at this stage; instead, we're focusing on the basics. Does the system boot up correctly? Are the essential services running? Are there any glaring error messages? These are the kinds of questions we're trying to answer. If we find any issues during the sanity check, it’s much easier and less costly to address them now than after the system has been deployed. So, this stage acts as a vital early warning system, helping us ensure the stability and reliability of our releases.

Using the the build browser for the testing stream:

  • Verify that the parent commit and version match the previous testing release (in the future, we'll want to integrate this check in the release job)
    • [ ] x86_64
    • [ ] aarch64
    • [ ] ppc64le
    • [ ] s390x
  • Check kola extended upgrade runs to make sure they didn't fail
    • [ ] x86_64
    • [ ] aarch64
    • [ ] ppc64le
    • [ ] s390x
  • Check kola AWS runs to make sure they didn't fail
    • [ ] x86_64
    • [ ] aarch64
  • Check kola OpenStack runs to make sure they didn't fail
    • [ ] x86_64
    • [ ] aarch64
  • Check kola Azure runs to make sure they didn't fail
    • [ ] x86_64
    • [ ] aarch64
  • Check kola GCP runs to make sure they didn't fail
    • [ ] x86_64
    • [ ] aarch64

IMPORTANT: this is the point of no return here. Once the OSTree commit is imported into the unified repo, any machine that manually runs rpm-ostree upgrade will have the new update.

Run the release job

Running the release job is the final countdown before we unleash the new version into the world. This stage is where we initiate the automated processes that propagate the build across our systems and make it available for users. It's a critical juncture because, once this job is executed, the new release is essentially live. We approach this step with the utmost care, ensuring that all prerequisites are met and that we're fully prepared for any eventuality. The release job is like the conductor of an orchestra, orchestrating a series of actions that must harmonize perfectly to achieve the desired outcome. From updating repositories to generating release notes, each task is carefully choreographed to minimize the risk of errors and ensure a seamless transition to the new version. It’s a moment of high stakes, but also one of great satisfaction, knowing that our hard work is about to benefit users worldwide.

  • [ ] Run the release job, filling in for parameters testing and the new version ID. This release job is very important to ensure the compatibility of the system.
  • [ ] Post a link to the job as a comment to this issue
  • [ ] Wait for job to finish

At this point, Cincinnati will see the new release on its next refresh and create a corresponding node in the graph without edges pointing to it yet.

Refresh metadata (stream and updates)

Refreshing metadata is like updating the map for our users, ensuring they know where the new release is and how to get there. This crucial step involves updating the stream and update information, so the system knows the latest version available and can guide users through the upgrade process. Think of it as updating the GPS coordinates for a destination; without accurate metadata, users might get lost or end up with outdated software. This process is not just about making the new release visible; it’s also about ensuring a smooth and seamless experience for our users. We want to make sure that the upgrade process is as straightforward as possible, with minimal disruption and maximum reliability. Accurate metadata is the key to achieving this goal, providing the necessary information for systems to correctly identify, download, and install the latest updates. It’s a vital part of our commitment to providing a high-quality, user-friendly experience.

  • [ ] Wait for all releases that will be released simultaneously to reach this step in the process
  • [ ] Go to the rollout workflow, click "Run workflow", and fill out the form
Rollout general guidelines
Risk Day of the week Rollout Start Time Time allocation
risky Tuesday 2PM UTC 72H
common Tuesday 2PM UTC 48H
rapid Tuesday 2PM UTC 24H

When setting a rollout start time ask "when would be the best time to react to any errors or regressions from updates?". Commonly we select 2PM UTC so that the rollout's start at 10am EST(±1 for daylight savings), but these can be fluid and adjust after talking with the fedora-coreos IRC. Note, this is impacted by the day of the week and holidays. The rollout time allocation is a key factor to consider.

The later in the week the release gets held up due to unforeseen issues the more likely the rollout time allocation will need to shrink or the release will need to be deferred.

Manual alternative
  • Make sure your fedora-coreos-stream-generator binary is up-to-date.

From a checkout of this repo:

  • Update stream metadata, by running:
fedora-coreos-stream-generator -releases=https://fcos-builds.s3.amazonaws.com/prod/streams/testing/releases.json  -output-file=streams/testing.json -pretty-print
  • Add a rollout. For example, for a 48-hour rollout starting at 10 AM ET the same day, run:
./rollout.py add testing <version> "10 am ET today" 48
  • Commit the changes and open a PR against the repo
Update graph manual check 
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=x86_64&stream=testing&rollout_wariness=0'
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=aarch64&stream=testing&rollout_wariness=0'
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=ppc64le&stream=testing&rollout_wariness=0'
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=s390x&stream=testing&rollout_wariness=0'

NOTE: In the future, most of these steps will be automated.

Housekeeping

Housekeeping is the unsung hero of the release process, ensuring that we're set up for success in the future. These final steps are about tidying up after a release and preparing for the next one. Think of it as putting away the tools and materials after a construction project, so they're ready for the next job. This includes tasks like creating issues for the next release, assigning responsibilities, and ensuring that our tracking systems are up-to-date. It’s about maintaining a smooth workflow and preventing future bottlenecks. Good housekeeping practices not only make our lives easier in the long run but also contribute to the overall efficiency and quality of our releases. By taking care of these details, we ensure that the process remains consistent, predictable, and manageable. So, while it might seem like a small thing, housekeeping is a crucial element in our commitment to continuous improvement.

  • [ ] If one doesn't already exist, open an issue in this repo for the next release in this stream. Use the approximate date of the release in the title.
  • [ ] Issues opened via the previous link will automatically create a linked Jira card. Assign the GitHub issue and Jira card to the next person in the rotation.

That's it, guys! We've walked through the entire process of releasing a new Fedora CoreOS testing stream. By following these steps, we can ensure a smooth and successful release every time. Keep up the great work!