Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
User Guide › Reference › Maintenance Support
ContentsContents
User Guide
  1. About this Guide
  2. I First Steps
    1. 1 Beginnerʼs Guide
  3. II Concepts
    1. 2 Supported Build Recipes and Package Formats
  4. III Setup
    1. 3 osc, the Command Line Tool
    2. 4 Build Configuration
  5. IV Usage
    1. 5 Basic OBS Workflow
    2. 6 Local Building
    3. 7 Using Source Services
    4. 8 SCM/CI Workflow Integration
    5. 9 Staging Workflow
    6. 10 Notifications
    7. 11 Moderation
  6. V Best Practices
    1. 12 Using the OBS Web UI
    2. 13 Basic Concepts and Work Styles
    3. 14 How to integrate external SCM sources
    4. 15 Publishing Upstream Binaries
    5. 16 Bootstrapping
    6. 17 osc Example Commands
    7. 18 Advanced Project Setups
    8. 19 Building Kernel Modules
    9. 20 Common Questions and Solutions
  7. VI Reference
    1. 21 OBS Architecture
    2. 22 OBS Concepts
    3. 23 Build Process
    4. 24 Build Containers
    5. 25 Source Management
    6. 26 SCM Bridge
    7. 27 Supported Formats
    8. 28 Request And Review System
    9. 29 Image Templates
    10. 30 Multiple Build Description File Handling
    11. 31 Maintenance Support
    12. 32 Binary Package Tracking
    13. 33 Administration
    14. 34 Scheduling and Dispatching
    15. 35 Build Constraints
    16. 36 Building Preinstall Images
    17. 37 Authorization
    18. 38 Quality Assurance(QA) Hooks
    19. 39 openSUSE Factory
  8. Glossary
  9. A GNU Licenses
Navigation←→
User Guide › Reference › Maintenance Support
Top
←→

31 Maintenance Support

31.1 Simple Project Setup
31.2 Project setup for the Maintenance Process
31.3 Using the Maintenance Process
31.4 OBS Internal Mechanisms
31.5 Setting Up Projects for a Maintenance Cycle
31.6 Optional Channel Setup

This chapter explains the setup and workflow of a maintenance update in the openSUSE way. However, this should not be limited to openSUSE distribution projects but be usable anywhere (the entire workflow or just parts of it).

The goal of the OBS maintenance process is to publish updates for a frozen project, in this example an entire distribution. These updates need to be approved by a maintenance team and the published result must contain documentation about the changes and be applicable in the easiest way by the users. The result is a package repository with additional information about the solved problems and defined groups of packages to achieve that. Binary delta data can also be generated to reduce the needed download size for the clients.

Technically this results in a frozen software repository containing the original package distribution and an additional update repository with a subset of updated packages that will be preferred by the package manager thus superseding the packages from the original distribution.

31.1 Simple Project Setup

Simple Project Setup
Figure 31.1: Simple Project Setup

This figure gives an overview about the project setup and general workflow for a single package and single maintained distribution. It shows the "openSUSE:11.4" project, which is considered to be frozen and not changing at all anymore. The "openSUSE:11.4:Update" projects hosts all officially released updates. It does not build any binary, just gets it sources and binaries from the maintenance incident project via the release process. The incident project is named "openSUSE:Maintenance:IDxxx" in this example, which is under control of the maintenance team. Official updates get built and reviewed here. QA teams are also testing the binaries from here. However, a user can prepare it in the same way in their project and start the maintenance process via doing a "maintenance" request.

  • openSUSE:11.4 is the GA Project in this example. It is locked and not changing anymore.

  • openSUSE:11.4:Update is the Update Project to release official updates for the locked openSUSE:11.4 project. Thus it links to the openSUSE:11.4 project, inheriting all package sources from there.

  • openSUSE:Maintenance is the Maintenance Project which in this case maintains the openSUSE:11.4:Update project (and optionally others as well).

  • openSUSE:Maintenance:IDxxx is a Incident project created automatically by accepting a maintenance request.

31.2 Project setup for the Maintenance Process

All workflow related projects must be set up with a proper project meta configuration.

  • It is recommended to lock the GA Project by the project maintainer by using the osc lock [PROJECT] command

  • The Update Project has to have the <link project="[PROJECT]"/> element in the project meta configuration.

    It is very useful to define groups of bugowners, maintainers and reviewers and to make use of bots for further quality assurance tasks.

  • The Maintenance Project has to have the 

    <project name=... kind="maintenance">

    attribute in the project meta configuration, as well as a 

    <maintenance>

    element containing one or more 

    <maintains project="[PROJECT]"/>

    elements.

    It is very useful to define groups of maintainers and reviewers and to make use of review bots to enforce desired quality properties here.

31.3 Using the Maintenance Process

This describes all required steps by all involved persons from preparing to releasing a maintenance update.

31.3.1 Workflow A: A Maintainer Builds an Entire Update Incident for Submission

A user is usually starting to prepare an update by creating a maintenance branch. This is typically done by creating an own maintenance project. Usually multiple released products are affected, so the server can find out which one are maintained by a given source package name, in this example for glibc including checkout via 

osc mbranch glibc
osc mbranch --checkout glibc

This is equivalent to the API call /source?cmd=branch&package=glibc.

It is also possible to branch only one defined version, if it is known that only one version is affected. In this example the openSUSE:12.1 version: 

osc branch --maintenance openSUSE:12.1 glibc
osc branch -M -c openSUSE:12.1 glibc

In a simple setup as described before, create the maintenance branch from the package of the Update Project as the GA Project can never be changed anymore.

NOTE: both branch commands do support the --noaccess parameter, which will create a hidden project. This may be used when a not yet publicly known security issue is get fixed.

Afterwards the user needs to do the needed modifications. Packages will be built and can be tested. Afterwards they may add information about the purpose of this maintenance update via 

osc patchinfo

If the source changes contain references to issue trackers (like Bugzilla, CVE or FATE) these will be added to the _patchinfo file.

The server will create a full maintenance channel now, in case the user wants to test this as well. After the user has tested, they have to create a maintenancerequest to ask the maintenance team to accept this as an official update incident: 

osc maintenancerequest

On accepting this request all sources of the entire project will get copied to the incident project and be rebuild. The origin project gets usually removed (based on the request cleanup options).

31.3.2 Workflow B: Submitting a Package Without Branching

You may submit a package source from a project which is not prepared as maintenance project. That works via the maintenancerequest mechanism by specifying one or more packages from one project. As a consequence it means also that the first testable build will happen in the maintenance incident project. Also, the maintenance team need to write the update information on their own. 

osc maintenancerequest [ SOURCEPROJECT [ SOURCEPACKAGES RELEASEPROJECT ] ]

The following example is submitting two packages (kdelibs4 and kdebase4) from the project KDE:Devel project as update for openSUSE:12.1

osc maintenancerequest KDE:Devel kdelibs4 kdebase4 openSUSE:12.1
Note
Note: Specifying an Existing Incident

It is also possible to specify an existing incident as target with the --incident parameter. The packages will then be merged into the existing incident project.

31.3.3 Workflow C: Process Gets Initiated By the Maintenance Team

The maintenance team may start the process (for example because a security issue was reported and the maintenance team decided that a fix is required). In this case the incident gets created via the Web UI or via the API call:

osc createincident [PROJECT]

osc api /source/PROJECT?cmd=createmaintenanceincident

osc api /source?cmd=createmaintenanceincident&attribute=OBS:Maintenance.

To document the expected work the creation of a patchinfo package is needed. This can be done via

osc patchinfo [PROJECT]

It is important to add Bugzilla entries inside of the _patchinfo file. As long these are open Bugzilla entries, the bug assignee will see this patchinfo on their "my work" Web UI and osc views, so they knows that work is expected from them.

31.3.4 Maintenance Incident Processing

The maintenance incidents are usually managed by a maintenance team. In case the incident got started by a maintainer a maintenance request is targeted towards the defined maintenance project, in our example this is openSUSE:Maintenance. The defined maintainer and reviewers in this project need to decide about this request. In case it gets accepted, the server is creating a subproject with a unique incident ID and copies the sources and build settings to it. The origin project will get removed usually via the cleanup option. This maintenance project is used to build the final packages.

If the maintenance team decides to merge a new maintenance request with an existing incident, they can run the osc rq setincident $REQUESTID $INCIDENT before accepting the request.

The maintenance team may still modify them or the patchinfo data at this point. An outside maintainer can still submit changes via standard submit request mechanism, but direct write permissions are not granted. When the maintenance people are satisfied with the update, they can create a request to release the sources and binaries to the final openSUSE:11.4:Update project. 

osc releaserequest

The release request needs to specify the source and target for each package. In case just the source package or project is specified the API is completing the request on creation time. It is using this based on the source link target of each package and the release information in the repository definitions.

31.3.5 Incident Gets Released

The release process gets usually started via creating a release request. This sets all affected packages to the locked state, which means that all commands for editing the source or triggering rebuilds are not allowed anymore.

The release request typically needs to be approved by QA and other teams as defined in the Update project. In case something gets declined, the necessary changes need to be submitted to the maintenance project and a new release request has to be created.

A unique release ID will be generated and become part of the updateinfo.xml file in the target project on release event. This ID is different from the incident ID and is usually in the style of "YEAR-COUNTER". The counter is strictly increasing on each release. In case of a re-release of the same incident a release counter will be added.

A different naming scheme can be defined via the OBS:MaintenanceIdTemplate attribute value. The release will move all packages to the update project and extend the target package name with the incident ID. Binaries will be moved as well without modification. The exception is the updateinfo.xml which will be modified by replacing its incident id with the release id.

31.3.6 Incident Gets Reopened and Re-Released

An update should not, but may have an undetected regression. In this case the update needs a re-release. (If another problem shall be fixed a new incident should be created instead.)

If the current update harms the systems, the maintenance team may decide to take it back immediately. It can be done by removing the patchinfo.ID package container in the Update projects. This will create a new update channel without this update.

To re-open a release incident project, it must get unlocked and marked as open again. Unlocking can be done either via revoking a release request or via explicit unlocking the incident. The explicit unlock via osc: osc unlock INCIDENT_PROJECT is also triggering a rebuild to ensure to have higher release numbers and adding the "trigger=maintenance" flags to the release target definitions. Afterwards the project can be edited again and also gets listed as running incident again.

31.3.7 Using Custom Update IDs

The used string of update IDs can be defined via the OBS:MaintenanceIdTemplate attribute value of the master maintenance project.

31.4 OBS Internal Mechanisms

OBS is tracking maintenance work and can be used as a database for future and past updates.

31.4.1 Maintenance Incident Workflow

A maintenance incident is started by creating the incident project, either via a developer request or by the maintenance team.

  1. Incident project container is created. This is always a sub project to the maintenance project. A unique ID (counter) is used as subproject name. Build is disabled by default project wide.

  2. Default content for an incident is added via branch by attribute call:

    • Package sources get added based on given package and attribute name from all existing project instances. The package name is extended by the source project name to allow multiple instances of same package in one project. Source revision links are using the xsrcmd5 to avoid that other releases will affect this package instance.

    • Build repositories are added if missing. All repositories from all projects where the package sources gets branched from are used. The build flags in the package instances gets switched on for these.

    • A release target definition is added to the repository configuration via additional releasetarget element. The special release condition "maintenance" gets defined for this.

  3. Fixes for the packages need to get submitted now.

  4. A patchinfo file need to get added describing the issue.

  5. OBS server is building packages according to the sources and update information according to the patchinfo data.

  6. one or more release requests get created. It does also set the project to "freeze" state by default, this means no source changes are possible anymore and all running builds get canceled.

  7. Usually the request is in review state with defined reviewers from the release project. All reviewers need to review the state in the incident project.

  8. Request changes into state "new" when all reviewers accepted the release request.

  9. The release happens on accepting the request by the maintainers of the release project.

    • All package sources and binaries get copied into a package container where the package name gets extended by the incident number.

    • A main package gets created or updated, it just contains a link to the current incident package. For example, glibc points to glibc.42. The purpose of this main package is to have a place to refer to the current sources of a package.

    • The release target condition=maintenance gets removed.

    • The updateinfo.xml gets updated with the existing or now created unique updateinfo ID.

    • The server will update the repository based on all existing binaries.

  10. OPTIONAL: A maintenance coordinator may remove the release by removing the package instances inside the release project. The source link has to be fixed manually. (We may offer a function for this).

  11. OPTIONAL: A maintenance incident can be restarted by

    • Removing the lock flag.

    • Adding again the condition=maintenance attribute to the release target which requires a re-release.

NOTE: The step 1 and 2 may be done via accepting an incident request instead.

31.4.2 Searching for Incidents

The Web UI shows the running and past incidents when going to the maintenance project (openSUSE:Maintenance in our example). It shows the open requests either for creating or release an incident. Also, the open incidents, which are not yet released are visible.

All users need usually just to visit their "my work" screen in Web UI or osc to see requests or patchinfos where actions of them are expected: osc my [work]

The following items list some common ways to search for maintenance incidents via the api:

  • A developer can see the work to be done by them via searching for patchinfos with open Bugzilla entries:

    /search/package?match=([kind='patchinfo' and issue/[@state='OPEN' and owner/@login='$USER_LOGIN']])

  • A maintenance coordinator can see requests for doing a maintenance release via searching for open requests with maintenance_incident action against the maintenance project. They are visible in the Web UI request page of that project or via

    /search/request?match=(state/@name='new') and action/@type='maintenance_incident' and action/target/@project='openSUSE:Maintenance')

  • A maintenance coordinator can see open incidents via searching for incidents project repositories which have a release target with maintenance trigger. Note: this search result is showing all repositories of a matching project.

    /search/project?match=(repository/releasetarget/@trigger='maintenance')

  • A maintenance coordinator can see updates which currently are reviewed (for example by a QA team) via

    /search/request?match=(state/@name='review') and action/@type='maintenance_release')

  • A maintenance coordinator can see updates ready to release via searching for open requests with maintenance_release action.

    /search/request?match=(state/@name='new') and action/@type='maintenance_release')

31.5 Setting Up Projects for a Maintenance Cycle

31.5.1 Defining a Maintenance Space

An OBS server is using by default a maintenance space defined via the OBS:Maintenance attribute. This must get created on a project where maintenance incident projects should get created below. This project is also defining the default maintenance maintainers and reviewers in its ACL list.

It is possible to have multiple and independent maintenance name spaces, however the maintenance request must be created against this other namespace manually or using a different attribute.

31.5.2 Maintained Project Setups

Maintained projects must be frozen, this means no changes in sources or binaries. All updates will be hosted in the defined update project. This project gets defined via the OBS:UpdateProject attribute which must contain a value with the update project name. In addition to this, an attribute to define the active maintenance should also be defined, by default the OBS:Maintained attribute. The osc mbranch command will take packages from this project as a result.

The Update project should be defined as build disabled as well. Also define a project link to the main project and at least one repository building against the main project.

31.6 Optional Channel Setup

Channels are optional definitions to publish a sub-set of binaries into own repositories. They can be used to maintain a larger amount of packages in a central place, but defining to published binaries with an independent workflow which requires an approval for each binary.

31.6.1 Defining a Channel

Channels get defined and maintained in an xml file inside of a package source. The file name of these lists must be _channel.

The file may contain a list of targets where binaries gets released to.

31.6.2 Using Channels in Maintenance Workflow

Channel definitions for existing packages do affect incident projects. Matching channel packages get automatically branched inside and additional repositories for the channels are created. The server will build the channel package by aggregating the binary packages into the channel repositories.

The _channel files can be modified inside of the incident project if needed. This can be necessary when binary packages get renamed or added with this update. The modification will be part of the maintenance release request as simple submit actions.

This example shows the setup where selected binary packages get released also to a defined channel. The openSUSE:11.4:SecurityChannel project contains a _channel definition inside of the channel package. This one gets branched as well into the incident in case a matching channel does exist. Also, the additional repository gets added. The resulting binaries will be transfer via a release request to the code stream project (openSUSE:11.4:Update) and the special channel project.

→Chapter 32 Binary Package Tracking←Chapter 30 Multiple Build Description File Handling
Print this page

© 2024 SUSE