Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]

2 File System Overview

2.1 Configuration Files

2.1.1 Front-end Configuration

The front-end is configured with 4 files:

  • /srv/www/obs/api/config/database.yml

  • /srv/www/obs/api/config/options.yml

  • /srv/www/obs/api/config/feature.yml

  • /etc/apache2/vhosts.d/obs.conf

2.1.1.1 database.yml

This file has the information needed to access the database. It contain credentials for the database access and should be only readable by root and the group running the Web server (www).

The file has settings for the production, development and test ruby environment, for production systems only the production section is important.

Example production section

production:
  adapter: mysql2
  database: api_production
  username: obsapiuser
  password: topsecret
  encoding: utf8
  timeout: 15
  pool: 30
Table 2.1: Database Configuration Keywords
keyword Description Remarks

adapter

Database driver

only MySQL databases are supported

database

Database name

do not change !

username

MySQL user name

database user, not a system user

password

password for this user

clear text

encoding

codetable

timeout

wait time in milliseconds

pool

number of open connections per thread

socket

path to the MySQL socket

same host only

host

IP address or hostname of the MySQL server

for remote servers

port

port number of the MySQL server

for remote servers

2.1.1.2 options.yml

The configuration file /srv/www/obs/api/config/options.yml is the default configuration file for the Open Build Service Web UI and API. It contains configuration parameters for example for back-end connections and connection to the API. Important are the configurations for source and front-end hosts. The configuration for LDAP authentication is also located in this file.

Configuration options can be set per Rails environment or as generic configuration option defined in default.

Note
Note

We've updated the format of the options.yml after the release of OBS 2.9. Old configuration files can be converted via

(cd /srv/www/obs/api/; rake migrate_options_yml)
Note
Note

More and more configurations will be moved to the database and do not longer exist in this file. The database configuration can be accessed via the API /configuration path.

Table 2.2: options.yml Configuration Items
Config item Description Values default Remarks

use_xforward

Use mod_xforward module

true false

Apache only, should be true

use_nginx_redirect

Use X-Accel-Redirect

/internal_redirect

Nginx only

min_votes_for_rating

Minimum votes for a rating

integer 3

response_schema_validation

Set to true to verify XML responses comply to the schema

true false

test/debug option

source_host

back-end source server host

localhost

source_port

back-end source server port

integer 5352

source_protocol

back-end source server protocol

http, https

front end_host

Front-end host

localhost

frontend_port

Front-end port

integer 443

frontend_protocol

Front-end protocol

http https

external_frontend_host

External Front-end host

if your users access the hosts through a proxy or different name

external_frontend_port

External Front-end port

integer 443

external_frontend_protocol

External Front-end protocol

http https

extended_backend_log

Extended back-end log

true false

test/debug option

proxy_auth_mode:

turn proxy mode on/off

:off :on

see LDAP section

proxy_auth_test_user

Test user

coolguy

test/debug option

proxy_auth_test_email

Email of Test user

coolguy@ example.com

test/debug option

global_write_through

if set to false, the API will only fake writes to back-end

true false

test/debug option

auto_cleanup_after_days

not longer used

30

moved to /configuration API

errbit_api_key

API key of the application

test/debug option

errbit_host

installation of errbit.com a Ruby error catcher

test/debug option

errbit_api_key

API key of the application

test/debug option

ldap_mode:

OBS LDAP mode on/off

:off :on

see LDAP section

Example options.yml

#
# This file contains the default configuration of the Open Build Service
# API.
#

default: &default
  # Make use of mod_xforward module in apache
  use_xforward: true

  # Make use of X-Accel-Redirect for Nginx.
  # http://kovyrin.net/2010/07/24/nginx-fu-x-accel-redirect-remote
  #use_nginx_redirect: /internal_redirect

  # Minimum count of rating votes a project/package needs to # be taken in
  # account
  # for global statistics:
  min_votes_for_rating: 3

  # Set to true to verify XML reponses comply to the schema
  response_schema_validation: false

  # backend source server
  source_host: localhost
  source_port: 5352
  #source_protocol: https

  # api access to this instance
  frontend_host: localhost
  frontend_port: 443
  frontend_protocol: https
  # if your users access the hosts through a proxy (or just a different name,
  # use this to
  # overwrite the settings for users)
  #external_frontend_host: api.opensuse.org
  #external_frontend_port: 443
  #external_frontend_protocol: https


  extended_backend_log: true

  # proxy_auth_mode can be :off, :on or :simulate
  proxy_auth_mode: :off

  # ATTENTION: If proxy_auth_mode'is :on, the frontend takes the user
  # name that is coming as headervalue X-username as a
  # valid user does no further authentication. So take care...
  proxy_auth_test_user: coolguy
  proxy_auth_test_email: coolguy@example.com

  # set this to enable auto cleanup requests after the given days
  auto_cleanup_after_days: 30

  #schema_location

  #version

  # if set to false, the API will only fake writes to backend (useful in
  # testing)
  # global_write_through: true

  # see
  # http://colszowka.heroku.com/2011/02/22/setting-up-your-custom-hoptoad-notifier-endpoint-for-free-using-errbit-on-heroku
  #errbit_api_key: api_key_of_your_app
  #errbit_host: installation.of.errbit.com

production:
  <<: *default

test:
  <<: *default
  source_host: backend
  memcached_host: cache

development:
  <<: *default
  source_host: backend
  memcached_host: cache

2.1.1.3 feature.yml

The configuration file /srv/www/obs/api/config/feature.yml contains the default configuration about features that can be enabled or disabled in Open Build Service.

Table 2.3: feature.yml Configuration Items
Config item Description Values default Remarks

image_templates

enable/disable image template feature

true false

see Reference Guide for more information

kiwi_image_editor

enable/disable Kiwi Image Editor

true false

cloud_upload

enable/disable Cloud Upload setup

true false

Example feature.yml

production:
  features: &default
    image_templates: true
    kiwi_image_editor: false
    cloud_upload: false

development:
  features:
    <<: *default
    kiwi_image_editor: true
    cloud_upload: true

test:
  features:
    <<: *default
    kiwi_image_editor: true
    cloud_upload: true

2.1.1.4 Apache Virtual Host obs.conf

The Apache configuration depends on the Apache version and which extra options are used, so use the documentation of the Apache version you are using.

Here, as an example, the standard configuration used by the appliance: Apache vhost example

Listen 82
# May needed on old distributions or after an update from them.
#Listen 443

# Passenger defaults
PassengerSpawnMethod "smart"
PassengerMaxPoolSize 20
#RailsEnv "development"

# allow long request urls and being part of headers
LimitRequestLine 20000
LimitRequestFieldsize 20000

# Just the overview page
<VirtualHost *:80>
        # just give an overview about this OBS instance via static web page
        DocumentRoot  "/srv/www/obs/overview"

        <Directory /srv/www/obs/overview>
           Options Indexes
           Require all granted
        </Directory>
</VirtualHost>


# Build Results
<VirtualHost *:82>
        # The resulting repositories
        DocumentRoot  "/srv/obs/repos"

        <Directory /srv/obs/repos>
           Options Indexes FollowSymLinks
           Require all granted
        </Directory>
</VirtualHost>


# OBS WEB UI & API
<VirtualHost *:443>
        ServerName api

        #  General setup for the virtual host
        DocumentRoot  "/srv/www/obs/api/public"
        ErrorLog /srv/www/obs/api/log/apache_error.log
        TransferLog /srv/www/obs/api/log/apache_access.log

        PassengerMinInstances 2
        PassengerPreStart https://api

        SSLEngine on

        #  SSL protocols
        #  Supporting TLS only is adequate nowadays
        SSLProtocol all -SSLv2 -SSLv3

        #   SSL Cipher Suite:
        #   List the ciphers that the client is permitted to negotiate.
        #   We disable weak ciphers by default.
        #   See the mod_ssl documentation or "openssl ciphers -v" for a
        #   complete list.
        SSLCipherSuite ALL:!aNULL:!eNULL:!SSLv2:!LOW:!EXP:!MD5:@STRENGTH

        SSLCertificateFile /srv/obs/certs/server.crt
        SSLCertificateKeyFile /srv/obs/certs/server.key

        <Directory /srv/www/obs/api/public>
           AllowOverride all
           Options -MultiViews

           # This requires mod_xforward loaded in apache
           # Enable the usage via options.yml
           # This will decrease the load due to long running requests a lot (unloading from rails stack)
           XForward on

           Require all granted
        </Directory>

        SetEnvIf User-Agent ".*MSIE [1-5].*" \
                 nokeepalive ssl-unclean-shutdown \
                 downgrade-1.0 force-response-1.0

        CustomLog /var/log/apache2/ssl_request_log   ssl_combined


        # from http://guides.rubyonrails.org/asset_pipeline.html
        <LocationMatch "^/assets/.*$">
          Header unset ETag
          FileETag None
          # RFC says only cache for 1 year
          ExpiresActive On
          ExpiresDefault "access plus 1 year"
       </LocationMatch>

        SetEnvIf User-Agent ".*MSIE [1-5].*" \
                 nokeepalive ssl-unclean-shutdown \
                 downgrade-1.0 force-response-1.0

        ## Older firefox versions needs this, otherwise it wont cache anything over SSL.
        Header append Cache-Control "public"

</VirtualHost>

2.1.2 Back-end Configuration

The Back-end is configured with 2 files:

  • /etc/sysconfig/obs-server - a shell script used for workers and the OBS start scripts

  • /usr/lib/obs/server/BSConfig.pm - a Perl script defining some global variables

2.1.2.1 /etc/sysconfig/obs-server

This script is used to set up the basic paths and the worker. the most important settings are the OBS_SRC_SERVER and OBS_REPO_SERVERS and the OBS_WORKER_INSTANCES.

Table 2.4: obs-server Variables
Variable Description Values default Remarks

OBS_BACKENDCODE_DIR

Path to the back-end scripts

/usr/lib/obs/server/

OBS_RUN_DIR

communication directory base

/srv/obs/run

OBS_LOG_DIR

logging directory

/srv/obs/log

OBS_BASE_DIR

base directory

/srv/obs

OBS_API_AUTOSETUP

Automatically setup API and Web UI

yes no

appliance only, will overwrite config files

OBS_SRC_SERVER

source server host

localhost:5352

only one

OBS_REPO_SERVERS

repository server host

localhost:5252

maybe a list

OBS_WORKER_INSTANCES

number of build instances

integer 0

OBS_WORKER_INSTANCE _NAMES

names of the workers

space-separated list

OBS_WORKER_DIRECTORY

worker base directory

OBS_WORKER_PORTBASE

The base for port numbers used by worker

integer 0

0 OS assign number

OBS_WORKER_JOBS

Number of parallel compile jobs

integer 1

OBS_WORKER_TEST_MODE

Run in test mode

yes no

OBS_WORKER_HOST LABELS

one or more labels for the build host

may used by constraints

OBS_USE_SLP

Register in SLP server

yes no

OBS_CACHE_DIR

cache directory for downloaded packages

OBS_CACHE_SIZE

package cache size

in MB

OBS_WORKER_NICE _LEVEL

nice level of running workers

18

OBS_VM_TYPE

VM type

auto Xen kvm lxc zvm emulator:$arch none

OBS_VM_KERNEL

Set kernel used by worker

none (/boot/vmlinuz)

KVM option

OBS_VM_INITRD

initrd used by worker

none (/boot/vmlinuz)

KVM option

OBS_VM_DISK_AUTOSETUP _ROOT_FILESIZE

Autosetup disk size

4096

in MB

OBS_VM_DISK_AUTOSETUP _SWAP_FILESIZE

Autosetup swap size

1024

on MB

OBS_VM_DISK_AUTOSETUP _FILESYSTEM

File System used with autosetup

ext3

OBS_VM_DISK_AUTOSETUP _MOUNT_OPTIONS

Special mount options

OBS_VM_USE_TMPFS

Enable build in memory

yes no

requires much memory

OBS_INSTANCE_MEMORY

Memory allocated for a VM

512

OBS_STORAGE_AUTOSETUP

storage auto configuration

yes no

may destroy disk content

OBS_SETUP_WORKER _PARTITIONS

LVM via obsstoragesetup

take_all use_obs_vg none

may destroy disk content

OBS_WORKER_CACHE_SIZE

LVM partition for cache size

OBS_WORKER_ROOT_SIZE

LVM partition for root size

OBS_WORKER_SWAP_SIZE

LVM partition for swap size

OBS_WORKER_BINARIES _PROXY

proxy service for caching binaries

OBS_ROOT_SSHD_KEY_URL

ssh pub key to allow root user login

for mass deployment

OBS_WORKER_SCRIPT_URL

URL to the initial script

For workers the settings could be declared in the /etc/buildhost.config file as well.

#
# NOTE: all these options can be also declared in /etc/buildhost.config on each worker differently.
#

## Path:        Applications/OBS
## Description: The OBS backend code directory
## Type:        string
## Default:     ""
## Config:      OBS
#
# An empty dir will lead to the fall back directory, typically /usr/lib/obs/server/
#
OBS_BACKENDCODE_DIR=""

## Path:        Applications/OBS
## Description: The base for OBS communication directory
## Type:        string
## Default:     ""
## Config:      OBS
#
# An empty dir will lead to the fall back directory, typically /srv/obs/run
#
OBS_RUN_DIR="/srv/obs/run"

## Path:        Applications/OBS
## Description: The base for OBS logging directory
## Type:        string
## Default:     ""
## Config:      OBS
#
# An empty dir will lead to the fall back directory, typically /srv/obs/log
#
OBS_LOG_DIR="/srv/obs/log"

## Path:        Applications/OBS
## Description: The base directory for OBS
## Type:        string
## Default:     ""
## Config:      OBS
#
# An empty dir will lead to the fall back directory, typically /srv/obs
#
OBS_BASE_DIR=""

## Path:        Applications/OBS
## Description: Automatically set up API and Web UI for OBS server, be warned, this will replace config files!
## Type:        ("yes" | "no")
## Default:     "no"
## Config:      OBS
#
# This is usually only enabled on the OBS Appliance
#
OBS_API_AUTOSETUP="yes"
#
# NOTE: all these options can be also declared in /etc/buildhost.config on each worker differently.
#

## Path:        Applications/OBS
## Description: define source server host to be used
## Type:        string
## Default:     ""
## Config:      OBS
#
# An empty setting will point to localhost:5352 by default
#
OBS_SRC_SERVER=""

## Path:        Applications/OBS
## Description: define repository server host to be used
## Type:        string
## Default:     ""
## Config:      OBS
#
# An empty setting will point to localhost:5252 by default
#
OBS_REPO_SERVERS=""

## Path:        Applications/OBS
## Description: define number of build instances
## Type:        integer
## Default:     0
## Config:      OBS
#
# 0 instances will automatically use the number of CPU's
#
OBS_WORKER_INSTANCES="0"

## Path:        Applications/OBS
## Description: define names of build instances for z/VM
## Type:        string
## Default:     ""
## Config:      OBS
#
# The names of the workers as defined in z/VM. These must have two minidisks
# assigned, and have a secondary console configured to the local machine:
# 0150 is the root device
# 0250 is the swap device
#
#OBS_WORKER_INSTANCE_NAMES="LINUX075 LINUX076 LINUX077"
OBS_WORKER_INSTANCE_NAMES=""

## Path:        Applications/OBS
## Description: The base directory, where sub directories for each worker will get created
## Type:        string
## Default:     ""
## Config:      OBS
#
#
OBS_WORKER_DIRECTORY=""

## Path:        Applications/OBS
## Description: The base for port numbers used by worker instances
## Type:        integer
## Default:     "0"
## Config:      OBS
#
# 0 means let the operating system assign a port number
#
OBS_WORKER_PORTBASE="0"

## Path:        Applications/OBS
## Description: Number of parallel compile jobs per worker
## Type:        integer
## Default:     "1"
## Config:      OBS
#
# this maps usually to "make -j1" during build
#
OBS_WORKER_JOBS="1"

## Path:        Applications/OBS
## Description: Run in test mode (build results will be ignore, no job blocking)
## Type:        ("yes" | "")
## Default:     ""
## Config:      OBS
#
OBS_WORKER_TEST_MODE=""

## Path:        Applications/OBS
## Description: define one or more labels for the build host.
## Type:        string
## Default:     ""
## Config:      OBS
#
# A label can be used to build specific packages only on dedicated hosts.
# For example for benchmarking.
#
OBS_WORKER_HOSTLABELS=""

## Path:        Applications/OBS
## Description: Register in SLP server
## Type:        ("yes" | "no")
## Default:     "yes"
## Config:      OBS
#
#
OBS_USE_SLP="yes"

## Path:        Applications/OBS
## Description: Use a common cache directory for downloaded packages
## Type:        string
## Default:     ""
## Config:      OBS
#
# Enable caching requires a given directory here. Be warned, content will be
# removed there !
#
OBS_CACHE_DIR=""

## Path:        Applications/OBS
## Description: Defines the package cache size
## Type:        size in MB
## Default:     ""
## Config:      OBS
#
# Set the size to 50% of the maximum usable size of this partition
#
OBS_CACHE_SIZE=""

## Path:        Applications/OBS
## Description: Defines the nice level of running workers
## Type:        integer
## Default:     18
## Config:      OBS
#
# Nicenesses range from -20 (most favorable  scheduling) to 19 (least
# favorable).
# Default to 18 as some testsuites depend on being able to switch to
# one priority below (19) _and_ having changed the numeric level
# (so going from 19->19 makes them fail).
#
OBS_WORKER_NICE_LEVEL=18

## Path:        Applications/OBS
## Description: Set used VM type by worker
## Type:        ("auto" | "xen" | "kvm" | "lxc" | "zvm" | "emulator:$arch" | "emulator:$arch:$script" | "none")
## Default:     "auto"
## Config:      OBS
#
#
OBS_VM_TYPE="auto"

## Path:        Applications/OBS
## Description: Set kernel used by worker (kvm)
## Type:        ("none" | "/boot/vmlinuz" | "/foo/bar/vmlinuz)
## Default:     "none"
## Config:      OBS
#
# For z/VM this is normally /boot/image
#
OBS_VM_KERNEL="none"

## Path:        Applications/OBS
## Description: Set initrd used by worker (kvm)
## Type:        ("none" | "/boot/initrd" | "/foo/bar/initrd-foo)
## Default:     "none"
## Config:      OBS
#
# for KVM, you have to create with (example for openSUSE 11.2):
#
# export rootfstype="ext4"
# mkinitrd -d /dev/null -m "ext4 binfmt_misc virtio_pci virtio_blk" -k vmlinuz-2.6.31.12-0.2-default -i initrd-2.6.31.12-0.2-default-obs_worker
#
# a working initrd file which includes virtio and binfmt_misc for OBS in order to work fine
#
# for z/VM, the build script will create a initrd at the given location if
# it does not yet exist.
#
OBS_VM_INITRD="none"

## Path:        Applications/OBS
## Description: Autosetup for XEN/KVM/TMPFS disk (root) - Filesize in MB
## Type:        integer
## Default:     "4096"
## Config:      OBS
#
#
OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE="4096"

## Path:        Applications/OBS
## Description: Autosetup for XEN/KVM disk (swap) - Filesize in MB
## Type:        integer
## Default:     "1024"
## Config:      OBS
#
#
OBS_VM_DISK_AUTOSETUP_SWAP_FILESIZE="1024"

## Path:        Applications/OBS
## Description: Filesystem to use for autosetup {none,ext4}=ext4, ext3=ext3
## Type:        string
## Default:     "ext3"
## Config:      OBS
#
#
OBS_VM_DISK_AUTOSETUP_FILESYSTEM="ext3"

## Path:        Applications/OBS
## Description: Filesystem mount options to use for autosetup
## Type:        string
## Default:     ""
## Config:      OBS
#
#
OBS_VM_DISK_AUTOSETUP_MOUNT_OPTIONS=""

## Path:        Applications/OBS
## Description: Enable build in memory
## Type:        ("yes" | "")
## Default:     ""
## Config:      OBS
#
# WARNING: this requires much memory!
#
OBS_VM_USE_TMPFS=""

## Path:        Applications/OBS
## Description: Memory allocated for each VM (512) if not set
## Type:        integer
## Default:     ""
## Config:      OBS
#
#
OBS_INSTANCE_MEMORY=""

## Path:        Applications/OBS
## Description: Enable storage auto configuration
## Type:        ("yes" | "")
## Default:     ""
## Config:      OBS
#
# WARNING: this may destroy data on your hard disk !
# This is usually only used on mass deployed worker instances
#
OBS_STORAGE_AUTOSETUP="yes"

## Path:        Applications/OBS
## Description: Setup LVM via obsstoragesetup
## Type:        ("take_all" | "use_obs_vg" | "none")
## Default:     "use_obs_vg"
## Config:      OBS
#
# take_all: WARNING: all LVM partitions will be used and all data erased !
# use_obs_vg:  A lvm volume group named "OBS" will be re-setup for the workers.
#
OBS_SETUP_WORKER_PARTITIONS="use_obs_vg"

## Path:        Applications/OBS
## Description: Size in MB when creating LVM partition for cache partition
## Type:        integer
## Default:     ""
## Config:      OBS
#
#
OBS_WORKER_CACHE_SIZE=""

## Path:        Applications/OBS
## Description: Size in MB when creating LVM partition for each worker root partition
## Type:        integer
## Default:     ""
## Config:      OBS
#
#
OBS_WORKER_ROOT_SIZE=""

## Path:        Applications/OBS
## Description: Size in MB when creating LVM partition for each worker swap partition
## Type:        integer
## Default:     ""
## Config:      OBS
#
#
OBS_WORKER_SWAP_SIZE=""

## Path:        Applications/OBS
## Description: URL to a proxy service for caching binaries used by worker
## Type:        string
## Default:     ""
## Config:      OBS
#
#
OBS_WORKER_BINARIES_PROXY=""

## Path:        Applications/OBS
## Description: URL to a ssh pub key to allow root user login
## Type:        string
## Default:     ""
## Config:      OBS
#
# This is usually used on mass (PXE) deployed workers)
#
OBS_ROOT_SSHD_KEY_URL=""

## Path:        Applications/OBS
## Description: URL to a script to be downloaded and executed
## Type:        string
## Default:     ""
## Config:      OBS
#
# This is a hook for doing special things in your setup at boot time
#
OBS_WORKER_SCRIPT_URL=""

2.1.2.2 BSConfig.pm

This file is a perl module used by most back-end scripts, it mainly defines global variables. Since it is a perl module, after changes the back-end servers need to be restarted to become aware of the changes.

Warning
Warning

If there is a Perl syntax error in this file, the services will not start. Most likely you forgot the semicolon on the end of a statement.

Table 2.5: BSConfig.pm Variables
Variable Description Values default Remarks

$hostname

FQDN of the back-end host

leave as it is

$ip

IP address of the back-end host

leave as it is

$frontend

FQDN of the front-end host

undef

set only if the front-end runs on another host

$ipaccess

Map of IP access rules

Add all hosts if partition are used

$srcserver

URL of the source server

'http://$hostname: 5352'

$reposerver

URL of the repo server

'http://$hostname: 5252'

partition specific

$serviceserver

URL of the service server

'http://$hostname: 5152'

$workersrcserver

URL of the source server

optional for worker access

$workerreposerver

URL of the repo server

optional for worker access

$clouduploadserver

URL of the cloud upload server

'http://$hostname: 5452'

$servicedir

Path to the service scripts

/usr/lib/obs/service/

$servicetempdir

Path to service temp dir

/var/tmp/

optional

$serviceroot

Prefix to servicedir

optional

$service_maxchild

Maximum number of concurrent jobs for source service

integer

unlimited if not set

$gpg_standard_key

Path to the standard sign key

$hermesserver

URL of the notification server

optional

$hermesnamespace

Namespace for the notifications

optional

$notification _plugin

notification plugins

optional

@reposervers

List of reposervers

("http://$hostname: 5252")

$bsdir

Path to the back-end directory

/srv/obs

$bsuser

OS user running the back-end

obsrun

$bsgroup

OS group running the back-end

obsrun

$bsquotafile

Package quota for projects

optional

$sched_asyncmode

Use asynchronous scheduler

Avoid issues with remote projects on slow networks

$sched_startupmode

Cold start mode

0 1 2

$disable_data_sync

fdatasync

may cause data corruption

$rundir

back-end communication

$bsdir/run

$logdir

log directory

$bsdir/log

$nosharedtrees

Shared trees 0=shared 1=not shared 2=not shared with fallback

0 1 2

optional for non-ACL systems, should be set for access control

$packtrack

enable binary release tracking

[]

$limit_projects

limit visibility of projects for some architectures

optional

$relsync_pool

allow separation of releasenumber syncing per architecture

$stageserver

stage server

rsync URI

$stageserver_sync

Extra stage sync server

rsync URI

$sign

Path to sign script

$sign_project

call sign with --project <project>

0 1

$keyfile

Global sign key

$localarch

Local architecture for product building

$buildlog_maxsize

worker max buildlog size

'500 * 1000000'

in bytes

$buildlog_maxidle

Time with no changes in the buildlog will kill the job

'8 * 3600'

in sec

$xenstore_maxsize

xenstore size

'20 * 1000000'

current XEN has no xenstore anymore

$gettimeout

Max timeout for get

'1 * 3600'

in sec

$workerhostcheck

check script for worker

$powerhosts

Worker with more resources

obsolete use constraints

$powerpkgs

packages which need workers with more resources

obsolete use constraints

$norootexceptions

List of packages need to build as root

$old_style_services

Use old style source service handling

0 1

$partition

Current partition

see Section 1.4, “Distributed Setup”

$partitioning

Partition project mapping

see Section 1.4, “Distributed Setup”

$partitionservers

Partition server mapping

see Section 1.4, “Distributed Setup”

$dispatch_adjust

Adjust dispatch priority

see Section 4.5.2, “dispatch_adjust Array”

$publishedhook_use _regex

Use regular expressions in publish hook map

0 1

see Section 4.6, “Publisher Hooks”

$publishedhook

Publish hook map

see Section 4.6, “Publisher Hooks”

$unpublishedhook_use _regex

Use regular expressions in unpublish hook map

0 1

see Section 4.7, “Unpublisher Hooks”

$unpublishedhook

Unpublish hook map

see Section 4.7, “Unpublisher Hooks”

Example BSConfig.pm

#
# Copyright (c) 2006, 2007 Michael Schroeder, Novell Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2 as
# published by the Free Software Foundation.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program (see the file COPYING); if not, write to the
# Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
#
################################################################
#
# Open Build Service Configuration
#

package BSConfig;

use Net::Domain;
use Socket;

my $hostname = Net::Domain::hostfqdn() || 'localhost';
# IP corresponding to hostname (only used for $ipaccess); fallback to localhost since inet_aton may fail to resolve at shutdown.
my $ip = quotemeta inet_ntoa(inet_aton($hostname) || inet_aton("localhost"));

my $frontend = undef; # FQDN of the Web UI/API server if it's not $hostname

# If defined, restrict access to the backend servers (bs_repserver, bs_srcserver, bs_service)
our $ipaccess = {
   '127\..*' => 'rw', # only the localhost can write to the backend
   "^$ip" => 'rw',    # Permit IP of FQDN
   '.*' => 'worker',  # build results can be delivered from any client in the network
};

# IP of the Web UI/API Server (only used for $ipaccess)
if ($frontend) {
  my $frontendip = quotemeta inet_ntoa(inet_aton($frontend) || inet_aton("localhost"));
  $ipaccess->{$frontendip} = 'rw' ; # in dotted.quad format
}

# Also change the SLP reg files in /etc/slp.reg.d/ when you touch hostname or port
our $srcserver = "http://$hostname:5352";
our $reposerver = "http://$hostname:5252";
our $serviceserver = "http://$hostname:5152";

# you can use different ports for worker connections
#our $workersrcserver = "http://$hostname:5353";
#our $workerreposerver = "http://$hostname:5253";

our $servicedir = "/usr/lib/obs/service/";
#our $servicetempdir = "/var/temp/";
#our $serviceroot = "/opt/obs/MyServiceSystem";

# Maximum number of concurrent jobs for source service
#our $service_maxchild = 20;

our $gpg_standard_key = "/srv/obs/obs-default-gpg.asc";
# optional notification service:
#our $hermesserver = "http://$hostname/hermes";
#our $hermesnamespace = "OBS";
#
# Notification Plugin, multiple plugins supported, separated by space
#our $notification_plugin = "notify_hermes notify_rabbitmq";
#

# For the workers only, it is possible to define multiple repository servers here.
# But only one source server is possible yet.
our @reposervers = ("http://$hostname:5252");

# Package defaults
our $bsdir = '/srv/obs';
our $bsuser = 'obsrun';
our $bsgroup = 'obsrun';
#our $bsquotafile = '/srv/obs/quota.xml';

# Use asynchronus scheduler. This avoids hanging schedulers on remote projects,
# when the network is slow or broken. This will become the default in future
# our $sched_asyncmode = 1;

# Define how the scheduler does a cold start. The default (0) is to request the
# data for all packages, (1) means that only the non-remote packages are fetched,
# (2) means that all of the package data fetches get delayed.
# our $sched_startupmode = 0;

# Disable fdatasync calls, increases the speed, but may lead to data
# corruption on system crash when the filesystem does not guarantees
# data write before rename.
# It is esp. required on XFS filesystem.
# It is safe to be disabled on ext4 and btrfs filesystems.
#our $disable_data_sync = 1;

# Package rc script / backend communication + log files
our $rundir = "$bsdir/run";
our $logdir = "$bsdir/log";

# optional for non-acl systems, should be set for access control
# 0: trees are shared between projects (built-in default)
# 1: trees are not shared (only usable for new installations)
# 2: new trees are not shared, in case of a missing tree the shared
#    location is also tried (package default)
our $nosharedtrees = 2;

# enable binary release tracking by default for release projects
our $packtrack = [];

# optional: limit visibility of projects for some architectures
#our $limit_projects = {
# "ppc" => [ "openSUSE:Factory", "FATE" ],
# "ppc64" => [ "openSUSE:Factory", "FATE" ],
#};

# optional: allow seperation of releasnumber syncing per architecture
# one counter pool for all ppc architectures, one for i586/x86_64,
# arm archs are separated and one for the rest in this example
our $relsync_pool = {
 "local" => "local",
 "i586" => "i586",
 "x86_64" => "i586",
 "ppc" => "ppc",
 "ppc64" => "ppc",
 "ppc64le" => "ppc",
 "mips" => "mips",
 "mips64" => "mips",
 "mipsel" => "mipsel",
 "mips64el" => "mipsel",
 "aarch64"  => "arm",
 "aarch64_ilp32"  => "arm",
 "armv4l"  => "arm",
 "armv5l"  => "arm",
 "armv6l"  => "arm",
 "armv6hl" => "arm",
 "armv7l"  => "arm",
 "armv7hl" => "arm",
 "armv5el" => "armv5el", # they do not exist
 "armv6el" => "armv6el",
 "armv7el" => "armv7el",
 "armv8el" => "armv8el",
 "sparcv9" => "sparcv9",
 "sparc64" => "sparcv9",
};

#No extra stage server sync
#our $stageserver = 'rsync://127.0.0.1/put-repos-main';
#our $stageserver_sync = 'rsync://127.0.0.1/trigger-repos-sync';

#No package signing server
our $sign = "/usr/bin/sign";
#Extend sign call with project name as argument "--project $NAME"
#our $sign_project = 1;
#Global sign key
our $keyfile = "/srv/obs/obs-default-gpg.asc";

# Use a special local arch for product building
# our $localarch = "x86_64";

# config options for the bs_worker
#
#our buildlog_maxsize = 500 * 1000000;
#our buildlog_maxidle =   8 * 3600;
#our xenstore_maxsize =  20 * 1000000;
#our gettimeout =         1 * 3600;
#
# run a script to check if the worker is good enough for the job
#our workerhostcheck = 'my_check_script';
#
# Allow to build as root, exceptions per package
# the keys are actually anchored regexes
# our $norootexceptions = { "my_project/my_package" => 1, "openSUSE:Factory.*/installation-images" => 1 };

# Use old style source service handling
# our $old_style_services = 1;

###
# Optional support to split the binary backend. This can be used on large servers
# to separate projects for better scalability.
# There is still just one source server, but there can be multiple servers which
# run each repserver, schedulers, dispatcher, warden and publisher
#
# This repo service is the 'home' server for all home:* projects. This and the
# $reposerver setting must be different on the binary backend servers.
# our $partition = 'home';
#
# this defines how the projects are split. All home: projects are hosted
# on an own server in this example. Order is important.
# our $partitioning = [ 'home:' => 'home',
#                       '.*'    => 'main',
#                     ];
#
# our $partitionservers = { 'home' => 'http://home-backend-server:5252',
#                           'main' => 'http://main-backend-server:5252',
#                         };

# Publish hooks
our $publishedhook_use_regex = 1;
our $publishedhook = {
    "Product\/SLES12"     => "/usr/local/bin/script2run_sles12",
    "Product\/SLES11.*"   => "/usr/local/bin/script2run_sles11",
};


# host specific configs
my $hostconfig = __FILE__;
$hostconfig =~ s/[^\/]*$/bsconfig.$hostname/;
if (-r $hostconfig) {
  print STDERR "reading $hostconfig...\n";
  require $hostconfig;
}

1;

2.2 Log Files

2.2.1 Front-end

The front-end log files are found under /srv/www/obs/api/log.

The following front-end log files exist:

  • apache_access.log - apache requests

  • apache_error.log - errors from apache

  • backend_access.log - API → backend requests

  • clockworkd.clock.output → timer event log

  • delayed_job.log → delayed job log

  • production.log→ main ruby log

  • production.searchd.log - search daemon log

  • production.searchd.query.log - search request logs

2.2.2 Back-end

The back-end log files are found by default under /srv/obs/log/.

The following back-end log files exist:

  • dispatcher.log - dispatcher log

  • dodup.log - download on demand log (since 2.7)

  • publisher.log - publisher log

  • rep_server.log - repo server log

  • scheduler_<arch>.log - scheduler log for each architecture

  • signer.log - sign service log

  • src_server.log - source server log

  • src_service.log - source service daemon log

  • warden.log - warden log

  • clouduploadserver.log - cloud upload server log

  • clouduploadworker.log - cloud upload worker log

The following log files for the upload jobs exist inside the /srv/obs/cloudupload directory (also linked in /bs/cloudupload):

  • <job_id>.log - log files for undone upload jobs

  • done/<job_id>.log - log files for already finished upload jobs

2.3 /srv/obs Tree

The default back-end data directory is located under /srv/obs/. Here are a bunch of subdirectories used for communication between the different server, to store data, status information and logs. Here is one file configuration.xml in the top directory, which stores the global OBS configuration for the back-end. You should not modify this file directly, but use the API /configuration interface instead, since this information needs to kept in sync with the front-end.

2.3.1 build Directory

In this subdirectory managed by the repo server daemon, all repository data, metadata and build results are stored in a hierarchical tree.

Example build directory tree of a binary imported distribution (OpenSUSE:13.2) and a small test project with 3 packages:

├── openSUSE:13.2
│   └── standard
│       ├── i586
│       │   └── :full
│       └── x86_64
│           └── :full
├── Test1
│   └── os13.2
│       ├── i586
│       │   ├── :full
│       │   ├── :logfiles.fail
│       │   ├── :logfiles.success
│       │   ├── :meta
│       │   ├── :repo
│       │   ├── rsync
│       │   ├── srtp
│       │   └── wget
│       └── x86_64
│           ├── :full
│           ├── :logfiles.fail
│           ├── :logfiles.success
│           ├── :meta
│           ├── :repo
│           ├── rsync
│           ├── srtp
│           └── wget

2.3.2 cloudupload Directory

Info for cloud upload jobs is stored here, it has a subdir named done for storing the already finished jobs.

2.3.3 db Directory

Back-end database root directory use by the source server, repo server scheduler and publisher. Nobody should touch this.

2.3.4 diffcache Directory

Cache for source server compare operations.

2.3.5 events Directory

Communication between services.

2.3.6 info Directory

Scheduler information managed by the scheduler and used by the repo server.

2.3.7 jobs Directory

The build jobs are stored in the /srv/obs/jobs directory. They are organized bybuild architecture:

jobs
├── armv7l
├── i586
├── load
└── x86_64
    └── Release:Stable::SLE-12_GA::CI-demo-36db80552b735e193dced13f058f866f

The jobs/load file contains statistical data about the build jobs.

2.3.8 log Directory

Contains the log files of the back-end daemons.

2.3.9 projects Directory

Contains the project hierarchy and metadata under revision control.

2.3.10 remotecache Directory

Cache for remote repository information.

2.3.11 repos Directory

Directory managed by the publisher to collect build results, also used by the repo server and scheduler to find build results.

2.3.12 repos_sync Directory

Directory with files pointing to the project root directories, helper for publisher rsync.

2.3.13 run Directory

State and lock information for the back-end daemons

2.3.14 sources Directory

All package sources under revision control in one directory per package, managed by the source server. Package sources are by default deduplicated across all projects, as long a source file has the same MD5 sum, it is only stored once. A pseudo '_project' package exist in the directory containing the project metadata revisions. ':service' and ':upload' are temporary directories used by the source server.

Example sources directory structure:

sources/
├── CI-demo
[...]
├── srtp
├── test1
├── _project
├── :service
└── :upload

2.3.15 trees Directory

Revision control data for project and packages, managed by the source server.

2.3.16 upload Directory

Temporary directory for uploading files for other back-end components.

2.3.17 workers Directory

Worker information

2.4 Metadata

2.4.1 OBS Revision Control

This section gives a short generic overview how the revision information are stored in the OBS back-end for packages and projects. The OBS back-end stores all files in a light weight content based hierarchical tree. Each file is hashed (with MD5) and stored with the hash as part of the filename under the /srv/obs/tree or /srv/obs/sources directories. The revision information is stored in separate files by the Source Server in the /srv/obs/projects directory.

2.4.1.1 OBS revision control files

The revision information is stored in simple CSV like file format with a bar (|) as delimiter between the 8 columns. The files do have the extension .rev for package/project revision data and .mref for meta revision data. The hash then points to a <hash>-MD5SUMS file in the /srv/obs/tree/ directories which have the file list with MD5 hashes of this revision. The hashes in this file list are pointing to the source files in the /srv/obs/sources tree.

An example revision file:

1|1|56cdd3adb778089d1fcc49b92bb93e5b|0.9|1464005086|user4|initial version|
2|2|fe7aa1ade5c9d005de738c234c90bc90|0.9|1464005304|user4|fix spec file|
3|1|72c7986e694f45ab1a62779e64e92a8f|1.0|1464005339|user4|new version|
4|2|699e9931e6f167d78e65bbe5853f592f|1.0|1464006221|user4|add patch file|
5|1|0cfc3a2297f38d2aa9d8d0e98fc22a38|1.1|1464007797|user4|new version|
Table 2.6: The 8 Columns
Column Content XML tag may empty

1

revision number

ref

no

2

version revision number

vref

yes

3

hash

srcmd5

no

4

version

version

yes

5

time stamp

time

no

6

user

user

no

7

commit message

comment

yes

8

request id

requestid

yes

Depending on the target (package, project or metadata) used, fields can be empty or have special values, for example, unknown for the version.

Example MD5SUMS file

/srv/obs # cat trees/Test1/package1/56cdd3adb778089d1fcc49b92bb93e5b-MD5SUMS
0a17daaa913df9e50ee65e83a1898363  package1.spec
1f810b3521242a98333b7bbf6b2b7ef7  test1.sh

2.4.1.2 OBS Revision API

The revision info can be retrieved via API calls for the specific package, for example, using /source/<project>/<package>/_history.

Specific revisions of files can be retrieved with the optional "rev=N" parameter, for example, /source/<project>/<package>/<file>?rev=N.

On PUT and POST methods for files the optional "comment=some+comment" can be used to set a commit message.

2.4.2 Project Metadata

Project metadata are XML files containing the meta project information, such as title, description, related user and groups with roles, build settings, repository settings, publish settings, debug settings and more.

Table 2.7: Project Meta XML
XML tag Attributes Content

project

name

project name

title

Short description

description

Developer information

person

userid

login name

person

role

role (maintainer, bugowner, …)

group

groupid

group name

group

role

role (maintainer, bugowner, …)

devel

An optional devel project

build

optional build flags

publish

optional publish flags

useforbuild

optional useforbuild flags

debuginfo

optional debuginfo flags

binarydownload

optional binarydownload flags

repository

name

name of the repository for build results

repository path

project

name of the source project for remaining build requires

repository path

repository

name of repository in the source project

repository arch

architecture name

remoteurl

path to a remote OBS API for interconnect

Example project metadata:

<project name="Test11">
  <title>Test project 11</title>
  <description>Project for demo</description>
  <person userid="Admin" role="maintainer"/>
  <person userid="user0" role="maintainer"/>
  <group groupid="obsprj3" role="maintainer"/>
  <repository name="openSUSE_13.2">
    <path project="openSUSE.org:openSUSE:13.2" repository="standard"/>
    <arch>x86_64</arch>
  </repository>
</project>

2.4.3 Package Metadata

XML file about package meta information, like Title, description, related user and groups with roles, build settings, publish settings, debug settings and more. Most XML tags are the same as for projects.

Example package metadata:

<package name="test1" project="Test11">
  <title>A test package for learning</title>
  <description>An example test package for learning.&#13;
</description>
  <person userid="user5" role="bugowner"/>
  <person userid="user5" role="maintainer"/>
  <build>
    <enable repository="openSUSE_13.2"/>
  </build>
  <publish>
    <disable repository="openSUSE_13.2"/>
  </publish>
  <debuginfo>
    <disable/>
  </debuginfo>
</package>

2.4.4 Attribute Metadata

Attributes can be used to add special information to packages. Attributes can be used to trigger special actions.

Example attribute data:

<attributes>
  <attribute name="Issues" namespace="OBS"/>
  <attribute name="AutoCleanup" namespace="OBS">
    <value>2016-06-30 00:00:00</value>
  </attribute>
  <attribute name="AutoCleanup" namespace="OBS">
    <value></value>
  </attribute>
</attributes>

2.4.5 Job Files

Jobs are stored by the scheduler in the /srv/obs/jobs directory and contain the build setup information for the package, for example, a reference to the exact source version, build dependencies, build repository information, timestamps.

Sample job file:

<buildinfo project="Release:Stable" repository="SLE-12_GA" package="CI-demo"
srcserver="http://obs.b1-systems.de:5352"
reposerver="http://obs.b1-systems.de:5252">
  <job>Release:Stable::SLE-12_GA::
 CI-demo-36db80552b735e193dced13f058f866f</job>
  <arch>x86_64</arch>
  <srcmd5>36db80552b735e193dced13f058f866f</srcmd5>
  <verifymd5>36db80552b735e193dced13f058f866f</verifymd5>
  <rev>2</rev>
  <disturl>obs://b1-systems.de/Release:Stable/SLE-12_GA/
  36db80552b735e193dced13f058f866f-CI-demo</disturl>
  <reason>new build</reason>
  <needed>0</needed>
  <revtime>1461077600</revtime>
  <readytime>1461077708</readytime>
  <file>CI-demo.spec</file>
  <versrel>0.1.9-2</versrel>
  <bcnt>1</bcnt>
  <release>2.1</release>
  <debuginfo>1</debuginfo>
  <prjconfconstraint>linux:version:min 3.0.0</prjconfconstraint>
  <bdep name="aaa_base" preinstall="1" runscripts="1" notmeta="1" />
  <bdep name="attr" preinstall="1" notmeta="1" />
  <bdep name="bash" preinstall="1" notmeta="1" />
  <bdep name="coreutils" preinstall="1" notmeta="1" />
  <bdep name="diffutils" preinstall="1" notmeta="1" />
  <bdep name="filesystem" preinstall="1" notmeta="1" />
  <bdep name="fillup" preinstall="1" notmeta="1" />
  <bdep name="glibc" preinstall="1" notmeta="1" />
  <bdep name="grep" preinstall="1" notmeta="1" />
  <bdep name="libbz2-1" preinstall="1" notmeta="1" />
  <bdep name="libgcc_s1" preinstall="1" notmeta="1" />
  <bdep name="m4" preinstall="1" notmeta="1" />
[...]
  <path project="Release:Stable" repository="SLE-12_GA"
 server="http://obs.b1-systems.de:5252" />
  <path project="SUSE:SLE-12:GA" repository="standard"
 server="http://obs.b1-systems.de:5252" />
</buildinfo>
Print this page