3 File System Overview #
3.1 Configuration Files #
3.1.1 Front-end Configuration #
The front-end is configured with four files:
/srv/www/obs/api/config/database.yml
/srv/www/obs/api/config/options.yml
/srv/www/obs/api/config/feature.yml
/etc/apache2/vhosts.d/obs.conf
3.1.1.1 database.yml
#
This file has the information needed to access the database. It contain credentials for the database access and should be only readable by root and the group running the Web server (www).
The file has settings for the production, development and test ruby environment, for production systems only the production section is important.
Example production section
production: adapter: mysql2 database: api_production username: obsapiuser password: topsecret encoding: utf8 timeout: 15 pool: 30
keyword | Description | Remarks |
---|---|---|
adapter |
Database driver |
only MySQL databases are supported |
database |
Database name |
do not change ! |
username |
MySQL user name |
database user, not a system user |
password |
password for this user |
clear text |
encoding |
codetable | |
timeout |
wait time in milliseconds | |
pool |
number of open connections per thread | |
socket |
path to the MySQL socket |
same host only |
host |
IP address or hostname of the MySQL server |
for remote servers |
port |
port number of the MySQL server |
for remote servers |
3.1.1.2 options.yml
#
The configuration file /srv/www/obs/api/config/options.yml is the default configuration file for the Open Build Service Web UI and API. It contains configuration parameters for example for back-end connections and connection to the API. Important are the configurations for source and front-end hosts. The configuration for LDAP authentication is also located in this file.
Configuration options can be set per Rails environment or as generic configuration option defined in default.
We've updated the format of the options.yml after the release of OBS 2.9. Old configuration files can be converted via
(cd /srv/www/obs/api/; rake migrate_options_yml)
More and more configurations will be moved to the database and do not longer exist in this file. The database configuration can be accessed via the API /configuration path.
options.yml
Configuration Items #Config item | Description | Values default
| Remarks |
---|---|---|---|
use_xforward |
Use mod_xforward module |
|
Apache only, should be true |
use_nginx_redirect |
Use X-Accel-Redirect |
|
Nginx only |
min_votes_for_rating |
Minimum votes for a rating |
integer | |
response_schema_validation |
Set to true to verify XML responses comply to the schema |
true |
test/debug option |
source_host |
back-end source server host |
| |
source_port |
back-end source server port |
integer | |
source_protocol |
back-end source server protocol |
| |
front end_host |
Front-end host |
| |
frontend_port |
Front-end port |
integer | |
frontend_protocol |
Front-end protocol |
http | |
external_frontend_host |
External Front-end host |
if your users access the hosts through a proxy or different name | |
external_frontend_port |
External Front-end port |
integer | |
external_frontend_protocol |
External Front-end protocol |
http | |
extended_backend_log |
Extended back-end log |
|
test/debug option |
proxy_auth_mode: |
turn proxy mode on/off |
|
see LDAP section |
proxy_auth_test_user |
Test user |
|
test/debug option |
proxy_auth_test_email |
Email of Test user |
|
test/debug option |
global_write_through |
if set to false, the API will only fake writes to back-end |
|
test/debug option |
auto_cleanup_after_days |
not longer used |
|
moved to /configuration API |
errbit_api_key |
API key of the application |
test/debug option | |
errbit_host |
installation of errbit.com a Ruby error catcher |
test/debug option | |
errbit_api_key |
API key of the application |
test/debug option | |
ldap_mode: |
OBS LDAP mode on/off |
|
see LDAP section |
Example options.yml
# # This file contains the default configuration of the Open Build Service # API. # default: &default # Make use of mod_xforward module in apache use_xforward: true # Make use of X-Accel-Redirect for Nginx. # http://kovyrin.net/2010/07/24/nginx-fu-x-accel-redirect-remote #use_nginx_redirect: /internal_redirect # Minimum count of rating votes a project/package needs to # be taken in # account # for global statistics: min_votes_for_rating: 3 # Set to true to verify XML reponses comply to the schema response_schema_validation: false # backend source server source_host: localhost source_port: 5352 #source_protocol: https # api access to this instance frontend_host: localhost frontend_port: 443 frontend_protocol: https # if your users access the hosts through a proxy (or just a different name, # use this to # overwrite the settings for users) #external_frontend_host: api.opensuse.org #external_frontend_port: 443 #external_frontend_protocol: https extended_backend_log: true # proxy_auth_mode can be :off, :on or :simulate proxy_auth_mode: :off # ATTENTION: If proxy_auth_mode'is :on, the frontend takes the user # name that is coming as headervalue X-username as a # valid user does no further authentication. So take care... proxy_auth_test_user: coolguy proxy_auth_test_email: coolguy@example.com # set this to enable auto cleanup requests after the given days auto_cleanup_after_days: 30 #schema_location #version # if set to false, the API will only fake writes to backend (useful in # testing) # global_write_through: true # see # http://colszowka.heroku.com/2011/02/22/setting-up-your-custom-hoptoad-notifier-endpoint-for-free-using-errbit-on-heroku #errbit_api_key: api_key_of_your_app #errbit_host: installation.of.errbit.com production: <<: *default test: <<: *default source_host: backend memcached_host: cache development: <<: *default source_host: backend memcached_host: cache
3.1.1.3 feature.yml
#
The configuration file /srv/www/obs/api/config/feature.yml contains the default configuration about features that can be enabled or disabled in Open Build Service.
feature.yml
Configuration Items #Config item | Description | Values default
| Remarks |
---|---|---|---|
image_templates |
enable/disable image template feature |
|
see Reference Guide for more information |
kiwi_image_editor |
enable/disable Kiwi Image Editor |
true | |
cloud_upload |
enable/disable Cloud Upload setup |
true |
Example feature.yml
production: features: &default image_templates: true kiwi_image_editor: false cloud_upload: false development: features: <<: *default kiwi_image_editor: true cloud_upload: true test: features: <<: *default kiwi_image_editor: true cloud_upload: true
3.1.1.4 Apache Virtual Host obs.conf
#
The Apache configuration depends on the Apache version and which extra options are used, so use the documentation of the Apache version you are using.
Here, as an example, the standard configuration used by the appliance: Apache vhost example
Listen 82 # May needed on old distributions or after an update from them. #Listen 443 # Passenger defaults PassengerSpawnMethod "smart" PassengerMaxPoolSize 20 #RailsEnv "development" # allow long request urls and being part of headers LimitRequestLine 20000 LimitRequestFieldsize 20000 # Just the overview page <VirtualHost *:80> # just give an overview about this OBS instance via static web page DocumentRoot "/srv/www/obs/overview" <Directory /srv/www/obs/overview> Options Indexes Require all granted </Directory> </VirtualHost> # Build Results <VirtualHost *:82> # The resulting repositories DocumentRoot "/srv/obs/repos" <Directory /srv/obs/repos> Options Indexes FollowSymLinks Require all granted </Directory> </VirtualHost> # OBS WEB UI & API <VirtualHost *:443> ServerName api # General setup for the virtual host DocumentRoot "/srv/www/obs/api/public" ErrorLog /srv/www/obs/api/log/apache_error.log TransferLog /srv/www/obs/api/log/apache_access.log PassengerMinInstances 2 PassengerPreStart https://api SSLEngine on # SSL protocols # Supporting TLS only is adequate nowadays SSLProtocol all -SSLv2 -SSLv3 # SSL Cipher Suite: # List the ciphers that the client is permitted to negotiate. # We disable weak ciphers by default. # See the mod_ssl documentation or "openssl ciphers -v" for a # complete list. SSLCipherSuite ALL:!aNULL:!eNULL:!SSLv2:!LOW:!EXP:!MD5:@STRENGTH SSLCertificateFile /srv/obs/certs/server.crt SSLCertificateKeyFile /srv/obs/certs/server.key <Directory /srv/www/obs/api/public> AllowOverride all Options -MultiViews # This requires mod_xforward loaded in apache # Enable the usage via options.yml # This will decrease the load due to long running requests a lot (unloading from rails stack) XForward on Require all granted </Directory> SetEnvIf User-Agent ".*MSIE [1-5].*" \ nokeepalive ssl-unclean-shutdown \ downgrade-1.0 force-response-1.0 CustomLog /var/log/apache2/ssl_request_log ssl_combined # from http://guides.rubyonrails.org/asset_pipeline.html <LocationMatch "^/assets/.*$"> Header unset ETag FileETag None # RFC says only cache for 1 year ExpiresActive On ExpiresDefault "access plus 1 year" </LocationMatch> SetEnvIf User-Agent ".*MSIE [1-5].*" \ nokeepalive ssl-unclean-shutdown \ downgrade-1.0 force-response-1.0 ## Older firefox versions needs this, otherwise it wont cache anything over SSL. Header append Cache-Control "public" </VirtualHost>
3.1.2 Back-end Configuration #
The Back-end is configured with 2 files:
/etc/sysconfig/obs-server - a shell script used for workers and the OBS start scripts
/usr/lib/obs/server/BSConfig.pm - a Perl script defining some global variables
3.1.2.1 /etc/sysconfig/obs-server
#
This script is used to set up the basic paths and the worker. the most important settings are the OBS_SRC_SERVER and OBS_REPO_SERVERS and the OBS_WORKER_INSTANCES.
obs-server
Variables #Variable | Description | Values default
| Remarks |
---|---|---|---|
OBS_BACKENDCODE_DIR |
Path to the back-end scripts |
/usr/lib/obs/server/ | |
OBS_RUN_DIR |
communication directory base |
/srv/obs/run | |
OBS_LOG_DIR |
logging directory |
/srv/obs/log | |
OBS_BASE_DIR |
base directory |
/srv/obs | |
OBS_API_AUTOSETUP |
Automatically setup API and Web UI |
yes |
appliance only, will overwrite config files |
OBS_SRC_SERVER |
source server host |
localhost:5352 |
only one |
OBS_REPO_SERVERS |
repository server host |
localhost:5252 |
maybe a list |
OBS_WORKER_INSTANCES |
number of build instances |
integer | |
OBS_WORKER_INSTANCE _NAMES |
names of the workers |
space-separated list | |
OBS_WORKER_DIRECTORY |
worker base directory | ||
OBS_WORKER_PORTBASE |
The base for port numbers used by worker |
integer |
0 OS assign number |
OBS_WORKER_JOBS |
Number of parallel compile jobs |
integer | |
OBS_WORKER_TEST_MODE |
Run in test mode |
yes | |
OBS_WORKER_HOST LABELS |
one or more labels for the build host |
may used by constraints | |
OBS_USE_SLP |
Register in SLP server |
| |
OBS_CACHE_DIR |
cache directory for downloaded packages | ||
OBS_CACHE_SIZE |
package cache size |
in MB | |
OBS_WORKER_NICE _LEVEL |
nice level of running workers |
| |
OBS_VM_TYPE |
VM type |
auto Xen kvm lxc zvm emulator:$arch none | |
OBS_VM_KERNEL |
Set kernel used by worker |
|
KVM option |
OBS_VM_INITRD |
initrd used by worker |
|
KVM option |
OBS_VM_DISK_AUTOSETUP _ROOT_FILESIZE |
Autosetup disk size |
|
in MB |
OBS_VM_DISK_AUTOSETUP _SWAP_FILESIZE |
Autosetup swap size |
|
on MB |
OBS_VM_DISK_AUTOSETUP _FILESYSTEM |
File System used with autosetup |
| |
OBS_VM_DISK_AUTOSETUP _MOUNT_OPTIONS |
Special mount options | ||
OBS_VM_USE_TMPFS |
Enable build in memory |
yes |
requires much memory |
OBS_INSTANCE_MEMORY |
Memory allocated for a VM |
| |
OBS_STORAGE_AUTOSETUP |
storage auto configuration |
yes |
may destroy disk content |
OBS_SETUP_WORKER _PARTITIONS |
LVM via obsstoragesetup |
take_all |
may destroy disk content |
OBS_WORKER_CACHE_SIZE |
LVM partition for cache size | ||
OBS_WORKER_ROOT_SIZE |
LVM partition for root size | ||
OBS_WORKER_SWAP_SIZE |
LVM partition for swap size | ||
OBS_WORKER_BINARIES _PROXY |
proxy service for caching binaries | ||
OBS_ROOT_SSHD_KEY_URL |
ssh pub key to allow root user login |
for mass deployment | |
OBS_WORKER_SCRIPT_URL |
URL to the initial script |
For workers the settings could be declared in the /etc/buildhost.config file as well.
# # NOTE: all these options can be also declared in /etc/buildhost.config on each worker differently. # ## Path: Applications/OBS ## Description: The OBS backend code directory ## Type: string ## Default: "" ## Config: OBS # # An empty dir will lead to the fall back directory, typically /usr/lib/obs/server/ # OBS_BACKENDCODE_DIR="" ## Path: Applications/OBS ## Description: The base for OBS communication directory ## Type: string ## Default: "" ## Config: OBS # # An empty dir will lead to the fall back directory, typically /srv/obs/run # OBS_RUN_DIR="/srv/obs/run" ## Path: Applications/OBS ## Description: The base for OBS logging directory ## Type: string ## Default: "" ## Config: OBS # # An empty dir will lead to the fall back directory, typically /srv/obs/log # OBS_LOG_DIR="/srv/obs/log" ## Path: Applications/OBS ## Description: The base directory for OBS ## Type: string ## Default: "" ## Config: OBS # # An empty dir will lead to the fall back directory, typically /srv/obs # OBS_BASE_DIR="" ## Path: Applications/OBS ## Description: Automatically set up API and Web UI for OBS server, be warned, this will replace config files! ## Type: ("yes" | "no") ## Default: "no" ## Config: OBS # # This is usually only enabled on the OBS Appliance # OBS_API_AUTOSETUP="yes" # # NOTE: all these options can be also declared in /etc/buildhost.config on each worker differently. # ## Path: Applications/OBS ## Description: define source server host to be used ## Type: string ## Default: "" ## Config: OBS # # An empty setting will point to localhost:5352 by default # OBS_SRC_SERVER="" ## Path: Applications/OBS ## Description: define repository server host to be used ## Type: string ## Default: "" ## Config: OBS # # An empty setting will point to localhost:5252 by default # OBS_REPO_SERVERS="" ## Path: Applications/OBS ## Description: define number of build instances ## Type: integer ## Default: 0 ## Config: OBS # # 0 instances will automatically use the number of CPU's # OBS_WORKER_INSTANCES="0" ## Path: Applications/OBS ## Description: define names of build instances for z/VM ## Type: string ## Default: "" ## Config: OBS # # The names of the workers as defined in z/VM. These must have two minidisks # assigned, and have a secondary console configured to the local machine: # 0150 is the root device # 0250 is the swap device # #OBS_WORKER_INSTANCE_NAMES="LINUX075 LINUX076 LINUX077" OBS_WORKER_INSTANCE_NAMES="" ## Path: Applications/OBS ## Description: The base directory, where sub directories for each worker will get created ## Type: string ## Default: "" ## Config: OBS # # OBS_WORKER_DIRECTORY="" ## Path: Applications/OBS ## Description: The base for port numbers used by worker instances ## Type: integer ## Default: "0" ## Config: OBS # # 0 means let the operating system assign a port number # OBS_WORKER_PORTBASE="0" ## Path: Applications/OBS ## Description: Number of parallel compile jobs per worker ## Type: integer ## Default: "1" ## Config: OBS # # this maps usually to "make -j1" during build # OBS_WORKER_JOBS="1" ## Path: Applications/OBS ## Description: Run in test mode (build results will be ignore, no job blocking) ## Type: ("yes" | "") ## Default: "" ## Config: OBS # OBS_WORKER_TEST_MODE="" ## Path: Applications/OBS ## Description: define one or more labels for the build host. ## Type: string ## Default: "" ## Config: OBS # # A label can be used to build specific packages only on dedicated hosts. # For example for benchmarking. # OBS_WORKER_HOSTLABELS="" ## Path: Applications/OBS ## Description: Register in SLP server ## Type: ("yes" | "no") ## Default: "yes" ## Config: OBS # # OBS_USE_SLP="yes" ## Path: Applications/OBS ## Description: Use a common cache directory for downloaded packages ## Type: string ## Default: "" ## Config: OBS # # Enable caching requires a given directory here. Be warned, content will be # removed there ! # OBS_CACHE_DIR="" ## Path: Applications/OBS ## Description: Defines the package cache size ## Type: size in MB ## Default: "" ## Config: OBS # # Set the size to 50% of the maximum usable size of this partition # OBS_CACHE_SIZE="" ## Path: Applications/OBS ## Description: Defines the nice level of running workers ## Type: integer ## Default: 18 ## Config: OBS # # Nicenesses range from -20 (most favorable scheduling) to 19 (least # favorable). # Default to 18 as some testsuites depend on being able to switch to # one priority below (19) _and_ having changed the numeric level # (so going from 19->19 makes them fail). # OBS_WORKER_NICE_LEVEL=18 ## Path: Applications/OBS ## Description: Set used VM type by worker ## Type: ("auto" | "xen" | "kvm" | "lxc" | "zvm" | "emulator:$arch" | "emulator:$arch:$script" | "none") ## Default: "auto" ## Config: OBS # # OBS_VM_TYPE="auto" ## Path: Applications/OBS ## Description: Set kernel used by worker (kvm) ## Type: ("none" | "/boot/vmlinuz" | "/foo/bar/vmlinuz) ## Default: "none" ## Config: OBS # # For z/VM this is normally /boot/image # OBS_VM_KERNEL="none" ## Path: Applications/OBS ## Description: Set initrd used by worker (kvm) ## Type: ("none" | "/boot/initrd" | "/foo/bar/initrd-foo) ## Default: "none" ## Config: OBS # # for KVM, you have to create with (example for openSUSE 11.2): # # export rootfstype="ext4" # mkinitrd -d /dev/null -m "ext4 binfmt_misc virtio_pci virtio_blk" -k vmlinuz-2.6.31.12-0.2-default -i initrd-2.6.31.12-0.2-default-obs_worker # # a working initrd file which includes virtio and binfmt_misc for OBS in order to work fine # # for z/VM, the build script will create a initrd at the given location if # it does not yet exist. # OBS_VM_INITRD="none" ## Path: Applications/OBS ## Description: Autosetup for XEN/KVM/TMPFS disk (root) - Filesize in MB ## Type: integer ## Default: "4096" ## Config: OBS # # OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE="4096" ## Path: Applications/OBS ## Description: Autosetup for XEN/KVM disk (swap) - Filesize in MB ## Type: integer ## Default: "1024" ## Config: OBS # # OBS_VM_DISK_AUTOSETUP_SWAP_FILESIZE="1024" ## Path: Applications/OBS ## Description: Filesystem to use for autosetup {none,ext4}=ext4, ext3=ext3 ## Type: string ## Default: "ext3" ## Config: OBS # # OBS_VM_DISK_AUTOSETUP_FILESYSTEM="ext3" ## Path: Applications/OBS ## Description: Filesystem mount options to use for autosetup ## Type: string ## Default: "" ## Config: OBS # # OBS_VM_DISK_AUTOSETUP_MOUNT_OPTIONS="" ## Path: Applications/OBS ## Description: Enable build in memory ## Type: ("yes" | "") ## Default: "" ## Config: OBS # # WARNING: this requires much memory! # OBS_VM_USE_TMPFS="" ## Path: Applications/OBS ## Description: Memory allocated for each VM (512) if not set ## Type: integer ## Default: "" ## Config: OBS # # OBS_INSTANCE_MEMORY="" ## Path: Applications/OBS ## Description: Enable storage auto configuration ## Type: ("yes" | "") ## Default: "" ## Config: OBS # # WARNING: this may destroy data on your hard disk ! # This is usually only used on mass deployed worker instances # OBS_STORAGE_AUTOSETUP="yes" ## Path: Applications/OBS ## Description: Setup LVM via obsstoragesetup ## Type: ("take_all" | "use_obs_vg" | "none") ## Default: "use_obs_vg" ## Config: OBS # # take_all: WARNING: all LVM partitions will be used and all data erased ! # use_obs_vg: A lvm volume group named "OBS" will be re-setup for the workers. # OBS_SETUP_WORKER_PARTITIONS="use_obs_vg" ## Path: Applications/OBS ## Description: Size in MB when creating LVM partition for cache partition ## Type: integer ## Default: "" ## Config: OBS # # OBS_WORKER_CACHE_SIZE="" ## Path: Applications/OBS ## Description: Size in MB when creating LVM partition for each worker root partition ## Type: integer ## Default: "" ## Config: OBS # # OBS_WORKER_ROOT_SIZE="" ## Path: Applications/OBS ## Description: Size in MB when creating LVM partition for each worker swap partition ## Type: integer ## Default: "" ## Config: OBS # # OBS_WORKER_SWAP_SIZE="" ## Path: Applications/OBS ## Description: URL to a proxy service for caching binaries used by worker ## Type: string ## Default: "" ## Config: OBS # # OBS_WORKER_BINARIES_PROXY="" ## Path: Applications/OBS ## Description: URL to a ssh pub key to allow root user login ## Type: string ## Default: "" ## Config: OBS # # This is usually used on mass (PXE) deployed workers) # OBS_ROOT_SSHD_KEY_URL="" ## Path: Applications/OBS ## Description: URL to a script to be downloaded and executed ## Type: string ## Default: "" ## Config: OBS # # This is a hook for doing special things in your setup at boot time # OBS_WORKER_SCRIPT_URL=""
3.1.2.2 BSConfig.pm #
This file is a perl module used by most back-end scripts, it mainly defines global variables. Since it is a perl module, after changes the back-end servers need to be restarted to become aware of the changes.
If there is a Perl syntax error in this file, the services will not start. Most likely you forgot the semicolon on the end of a statement.
BSConfig.pm
Variables #Variable | Description | Values default
| Remarks |
---|---|---|---|
$hostname |
FQDN of the back-end host |
leave as it is | |
$ip |
IP address of the back-end host |
leave as it is | |
$frontend |
FQDN of the front-end host |
|
set only if the front-end runs on another host |
$ipaccess |
Map of IP access rules |
Add all hosts if partition are used | |
$srcserver |
URL of the source server |
| |
$reposerver |
URL of the repo server |
|
partition specific |
$serviceserver |
URL of the service server |
| |
$workersrcserver |
URL of the source server |
optional for worker access | |
$workerreposerver |
URL of the repo server |
optional for worker access | |
$clouduploadserver |
URL of the cloud upload server |
| |
$servicedir |
Path to the service scripts |
| |
$servicetempdir |
Path to service temp dir |
|
optional |
$serviceroot |
Prefix to servicedir |
optional | |
$service_maxchild |
Maximum number of concurrent jobs for source service |
integer |
unlimited if not set |
$gpg_standard_key |
Path to the standard sign key | ||
$hermesserver |
URL of the notification server |
optional | |
$hermesnamespace |
Namespace for the notifications |
optional | |
$notification _plugin |
notification plugins |
optional | |
@reposervers |
List of reposervers |
| |
$bsdir |
Path to the back-end directory |
| |
$bsuser |
OS user running the back-end |
| |
$bsgroup |
OS group running the back-end |
| |
$bsquotafile |
Package quota for projects |
optional | |
$sched_asyncmode |
Use asynchronous scheduler |
Avoid issues with remote projects on slow networks | |
$sched_startupmode |
Cold start mode |
| |
$disable_data_sync |
fdatasync |
may cause data corruption | |
$rundir |
back-end communication |
| |
$logdir |
log directory |
| |
$nosharedtrees |
Shared trees 0=shared 1=not shared 2=not shared with fallback |
0 1 |
optional for non-ACL systems, should be set for access control |
$packtrack |
enable binary release tracking |
| |
$limit_projects |
limit visibility of projects for some architectures |
optional | |
$relsync_pool |
allow separation of releasenumber syncing per architecture | ||
$stageserver |
stage server |
rsync URI | |
$stageserver_sync |
Extra stage sync server |
rsync URI | |
$sign |
Path to sign script | ||
$sign_project |
call sign with --project <project> |
0 | |
$keyfile |
Global sign key | ||
$localarch |
Local architecture for product building | ||
$buildlog_maxsize |
worker max buildlog size |
|
in bytes |
$buildlog_maxidle |
Time with no changes in the buildlog will kill the job |
|
in sec |
$xenstore_maxsize |
xenstore size |
|
current XEN has no xenstore anymore |
$gettimeout |
Max timeout for get |
|
in sec |
$workerhostcheck |
check script for worker | ||
$powerhosts |
Worker with more resources |
obsolete use constraints | |
$powerpkgs |
packages which need workers with more resources |
obsolete use constraints | |
$norootexceptions |
List of packages need to build as root | ||
$old_style_services |
Use old style source service handling |
| |
$partition |
Current partition | ||
$partitioning |
Partition project mapping | ||
$partitionservers |
Partition server mapping | ||
$dispatch_adjust |
Adjust dispatch priority | ||
$publishedhook_use _regex |
Use regular expressions in publish hook map |
| |
$publishedhook |
Publish hook map | ||
$unpublishedhook_use _regex |
Use regular expressions in unpublish hook map |
| |
$unpublishedhook |
Unpublish hook map |
Example BSConfig.pm
# # Copyright (c) 2006, 2007 Michael Schroeder, Novell Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License version 2 as # published by the Free Software Foundation. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program (see the file COPYING); if not, write to the # Free Software Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA # ################################################################ # # Open Build Service Configuration # package BSConfig; use Net::Domain; use Socket; my $hostname = Net::Domain::hostfqdn() || 'localhost'; # IP corresponding to hostname (only used for $ipaccess); fallback to localhost since inet_aton may fail to resolve at shutdown. my $ip = quotemeta inet_ntoa(inet_aton($hostname) || inet_aton("localhost")); my $frontend = undef; # FQDN of the Web UI/API server if it's not $hostname # If defined, restrict access to the backend servers (bs_repserver, bs_srcserver, bs_service) our $ipaccess = { '127\..*' => 'rw', # only the localhost can write to the backend "^$ip" => 'rw', # Permit IP of FQDN '.*' => 'worker', # build results can be delivered from any client in the network }; # IP of the Web UI/API Server (only used for $ipaccess) if ($frontend) { my $frontendip = quotemeta inet_ntoa(inet_aton($frontend) || inet_aton("localhost")); $ipaccess->{$frontendip} = 'rw' ; # in dotted.quad format } # Also change the SLP reg files in /etc/slp.reg.d/ when you touch hostname or port our $srcserver = "http://$hostname:5352"; our $reposerver = "http://$hostname:5252"; our $serviceserver = "http://$hostname:5152"; # you can use different ports for worker connections #our $workersrcserver = "http://$hostname:5353"; #our $workerreposerver = "http://$hostname:5253"; our $servicedir = "/usr/lib/obs/service/"; #our $servicetempdir = "/var/temp/"; #our $serviceroot = "/opt/obs/MyServiceSystem"; # Maximum number of concurrent jobs for source service #our $service_maxchild = 20; our $gpg_standard_key = "/srv/obs/obs-default-gpg.asc"; # optional notification service: #our $hermesserver = "http://$hostname/hermes"; #our $hermesnamespace = "OBS"; # # Notification Plugin, multiple plugins supported, separated by space #our $notification_plugin = "notify_hermes notify_rabbitmq"; # # For the workers only, it is possible to define multiple repository servers here. # But only one source server is possible yet. our @reposervers = ("http://$hostname:5252"); # Package defaults our $bsdir = '/srv/obs'; our $bsuser = 'obsrun'; our $bsgroup = 'obsrun'; #our $bsquotafile = '/srv/obs/quota.xml'; # Use asynchronus scheduler. This avoids hanging schedulers on remote projects, # when the network is slow or broken. This will become the default in future # our $sched_asyncmode = 1; # Define how the scheduler does a cold start. The default (0) is to request the # data for all packages, (1) means that only the non-remote packages are fetched, # (2) means that all of the package data fetches get delayed. # our $sched_startupmode = 0; # Disable fdatasync calls, increases the speed, but may lead to data # corruption on system crash when the filesystem does not guarantees # data write before rename. # It is esp. required on XFS filesystem. # It is safe to be disabled on ext4 and btrfs filesystems. #our $disable_data_sync = 1; # Package rc script / backend communication + log files our $rundir = "$bsdir/run"; our $logdir = "$bsdir/log"; # optional for non-acl systems, should be set for access control # 0: trees are shared between projects (built-in default) # 1: trees are not shared (only usable for new installations) # 2: new trees are not shared, in case of a missing tree the shared # location is also tried (package default) our $nosharedtrees = 2; # enable binary release tracking by default for release projects our $packtrack = []; # optional: limit visibility of projects for some architectures #our $limit_projects = { # "ppc" => [ "openSUSE:Factory", "FATE" ], # "ppc64" => [ "openSUSE:Factory", "FATE" ], #}; # optional: allow seperation of releasnumber syncing per architecture # one counter pool for all ppc architectures, one for i586/x86_64, # arm archs are separated and one for the rest in this example our $relsync_pool = { "local" => "local", "i586" => "i586", "x86_64" => "i586", "ppc" => "ppc", "ppc64" => "ppc", "ppc64le" => "ppc", "mips" => "mips", "mips64" => "mips", "mipsel" => "mipsel", "mips64el" => "mipsel", "aarch64" => "arm", "aarch64_ilp32" => "arm", "armv4l" => "arm", "armv5l" => "arm", "armv6l" => "arm", "armv6hl" => "arm", "armv7l" => "arm", "armv7hl" => "arm", "armv5el" => "armv5el", # they do not exist "armv6el" => "armv6el", "armv7el" => "armv7el", "armv8el" => "armv8el", "sparcv9" => "sparcv9", "sparc64" => "sparcv9", }; #No extra stage server sync #our $stageserver = 'rsync://127.0.0.1/put-repos-main'; #our $stageserver_sync = 'rsync://127.0.0.1/trigger-repos-sync'; #No package signing server our $sign = "/usr/bin/sign"; #Extend sign call with project name as argument "--project $NAME" #our $sign_project = 1; #Global sign key our $keyfile = "/srv/obs/obs-default-gpg.asc"; # Use a special local arch for product building # our $localarch = "x86_64"; # config options for the bs_worker # #our buildlog_maxsize = 500 * 1000000; #our buildlog_maxidle = 8 * 3600; #our xenstore_maxsize = 20 * 1000000; #our gettimeout = 1 * 3600; # # run a script to check if the worker is good enough for the job #our workerhostcheck = 'my_check_script'; # # Allow to build as root, exceptions per package # the keys are actually anchored regexes # our $norootexceptions = { "my_project/my_package" => 1, "openSUSE:Factory.*/installation-images" => 1 }; # Use old style source service handling # our $old_style_services = 1; ### # Optional support to split the binary backend. This can be used on large servers # to separate projects for better scalability. # There is still just one source server, but there can be multiple servers which # run each repserver, schedulers, dispatcher, warden and publisher # # This repo service is the 'home' server for all home:* projects. This and the # $reposerver setting must be different on the binary backend servers. # our $partition = 'home'; # # this defines how the projects are split. All home: projects are hosted # on an own server in this example. Order is important. # our $partitioning = [ 'home:' => 'home', # '.*' => 'main', # ]; # # our $partitionservers = { 'home' => 'http://home-backend-server:5252', # 'main' => 'http://main-backend-server:5252', # }; # Publish hooks our $publishedhook_use_regex = 1; our $publishedhook = { "Product\/SLES12" => "/usr/local/bin/script2run_sles12", "Product\/SLES11.*" => "/usr/local/bin/script2run_sles11", }; # host specific configs my $hostconfig = __FILE__; $hostconfig =~ s/[^\/]*$/bsconfig.$hostname/; if (-r $hostconfig) { print STDERR "reading $hostconfig...\n"; require $hostconfig; } 1;
3.2 Log Files #
3.2.1 Front-end #
The front-end log files are found under /srv/www/obs/api/log.
The following front-end log files exist:
apache_access.log - apache requests
apache_error.log - errors from apache
backend_access.log - API → backend requests
clockworkd.clock.output → timer event log
delayed_job.log → delayed job log
production.log→ main ruby log
production.searchd.log - search daemon log
production.searchd.query.log - search request logs
3.2.2 Back-end #
The back-end log files are found by default under /srv/obs/log/.
The following back-end log files exist:
dispatcher.log - dispatcher log
dodup.log - download on demand log (since 2.7)
publisher.log - publisher log
rep_server.log - repo server log
scheduler_<arch>.log - scheduler log for each architecture
signer.log - sign service log
src_server.log - source server log
src_service.log - source service daemon log
warden.log - warden log
clouduploadserver.log - cloud upload server log
clouduploadworker.log - cloud upload worker log
The following log files for the upload jobs exist inside the /srv/obs/cloudupload directory (also linked in /bs/cloudupload):
<job_id>.log - log files for undone upload jobs
done/<job_id>.log - log files for already finished upload jobs
3.3 /srv/obs
Tree #
The default back-end data directory is located under /srv/obs/. Here are a bunch of subdirectories used for communication between the different server, to store data, status information and logs. Here is one file configuration.xml in the top directory, which stores the global OBS configuration for the back-end. You should not modify this file directly, but use the API /configuration interface instead, since this information needs to kept in sync with the front-end.
3.3.1 build
Directory #
In this subdirectory managed by the repo server daemon, all repository data, metadata and build results are stored in a hierarchical tree.
Example build directory tree of a binary imported distribution (OpenSUSE:13.2) and a small test project with 3 packages:
├── openSUSE:13.2 │ └── standard │ ├── i586 │ │ └── :full │ └── x86_64 │ └── :full ├── Test1 │ └── os13.2 │ ├── i586 │ │ ├── :full │ │ ├── :logfiles.fail │ │ ├── :logfiles.success │ │ ├── :meta │ │ ├── :repo │ │ ├── rsync │ │ ├── srtp │ │ └── wget │ └── x86_64 │ ├── :full │ ├── :logfiles.fail │ ├── :logfiles.success │ ├── :meta │ ├── :repo │ ├── rsync │ ├── srtp │ └── wget
3.3.2 cloudupload
Directory #
Info for cloud upload jobs is stored here, it has a subdir named done for storing the already finished jobs.
3.3.3 db
Directory #
Back-end database root directory use by the source server, repo server scheduler and publisher. Nobody should touch this.
3.3.4 diffcache
Directory #
Cache for source server compare operations.
3.3.5 events
Directory #
Communication between services.
3.3.6 info
Directory #
Scheduler information managed by the scheduler and used by the repo server.
3.3.7 jobs
Directory #
The build jobs are stored in the /srv/obs/jobs directory. They are organized bybuild architecture:
jobs ├── armv7l ├── i586 ├── load └── x86_64 └── Release:Stable::SLE-12_GA::CI-demo-36db80552b735e193dced13f058f866f
The jobs/load file contains statistical data about the build jobs.
3.3.8 log
Directory #
Contains the log files of the back-end daemons.
3.3.9 projects
Directory #
Contains the project hierarchy and metadata under revision control.
3.3.10 remotecache
Directory #
Cache for remote repository information.
3.3.11 repos
Directory #
Directory managed by the publisher to collect build results, also used by the repo server and scheduler to find build results.
3.3.12 repos_sync
Directory #
Directory with files pointing to the project root directories, helper for publisher rsync.
3.3.13 run
Directory #
State and lock information for the back-end daemons
3.3.14 sources
Directory #
All package sources under revision control in one directory per package, managed by the source server. Package sources are by default deduplicated across all projects, as long a source file has the same MD5 sum, it is only stored once. A pseudo '_project' package exist in the directory containing the project metadata revisions. ':service' and ':upload' are temporary directories used by the source server.
Example sources directory structure:
sources/ ├── CI-demo [...] ├── srtp ├── test1 ├── _project ├── :service └── :upload
3.3.15 trees
Directory #
Revision control data for project and packages, managed by the source server.
3.3.16 upload
Directory #
Temporary directory for uploading files for other back-end components.
3.3.17 workers
Directory #
Worker information
3.4 Metadata #
3.4.1 OBS Revision Control #
This section gives a short generic overview how the revision information are stored in the OBS back-end for packages and projects. The OBS back-end stores all files in a light weight content based hierarchical tree. Each file is hashed (with MD5) and stored with the hash as part of the filename under the /srv/obs/tree or /srv/obs/sources directories. The revision information is stored in separate files by the Source Server in the /srv/obs/projects directory.
3.4.1.1 OBS revision control files #
The revision information is stored in simple CSV like file format with a bar (|) as delimiter between the 8 columns. The files do have the extension .rev for package/project revision data and .mref for meta revision data. The hash then points to a <hash>-MD5SUMS file in the /srv/obs/tree/ directories which have the file list with MD5 hashes of this revision. The hashes in this file list are pointing to the source files in the /srv/obs/sources tree.
An example revision file:
1|1|56cdd3adb778089d1fcc49b92bb93e5b|0.9|1464005086|user4|initial version| 2|2|fe7aa1ade5c9d005de738c234c90bc90|0.9|1464005304|user4|fix spec file| 3|1|72c7986e694f45ab1a62779e64e92a8f|1.0|1464005339|user4|new version| 4|2|699e9931e6f167d78e65bbe5853f592f|1.0|1464006221|user4|add patch file| 5|1|0cfc3a2297f38d2aa9d8d0e98fc22a38|1.1|1464007797|user4|new version|
Column | Content | XML tag | may empty |
---|---|---|---|
1 |
revision number |
ref |
no |
2 |
version revision number |
vref |
yes |
3 |
hash |
srcmd5 |
no |
4 |
version |
version |
yes |
5 |
time stamp |
time |
no |
6 |
user |
user |
no |
7 |
commit message |
comment |
yes |
8 |
request id |
requestid |
yes |
Depending on the target (package, project or metadata)
used, fields can be empty or have special values, for example,
unknown
for the version.
Example MD5SUMS file
/srv/obs # cat trees/Test1/package1/56cdd3adb778089d1fcc49b92bb93e5b-MD5SUMS 0a17daaa913df9e50ee65e83a1898363 package1.spec 1f810b3521242a98333b7bbf6b2b7ef7 test1.sh
3.4.1.2 OBS Revision API #
The revision info can be retrieved via API calls for the specific
package, for example, using
/source/<project>/<package>/_history
.
Specific revisions of files can be retrieved with the optional "rev=N" parameter, for example, /source/<project>/<package>/<file>?rev=N.
On PUT and POST methods for files the optional "comment=some+comment" can be used to set a commit message.
3.4.2 Project Metadata #
Project metadata are XML files containing the meta project information, such as title, description, related user and groups with roles, build settings, repository settings, publish settings, debug settings and more.
XML tag | Attributes | Content |
---|---|---|
project |
name |
project name |
title |
Short description | |
description |
Developer information | |
person |
userid |
login name |
person |
role |
role (maintainer, bugowner, …) |
group |
groupid |
group name |
group |
role |
role (maintainer, bugowner, …) |
devel |
An optional devel project | |
build |
optional build flags | |
publish |
optional publish flags | |
useforbuild |
optional useforbuild flags | |
debuginfo |
optional debuginfo flags | |
binarydownload |
optional binarydownload flags | |
repository |
name |
name of the repository for build results |
repository path |
project |
name of the source project for remaining build requires |
repository path |
repository |
name of repository in the source project |
repository arch |
architecture name | |
remoteurl |
path to a remote OBS API for interconnect |
Example project metadata:
<project name="Test11"> <title>Test project 11</title> <description>Project for demo</description> <person userid="Admin" role="maintainer"/> <person userid="user0" role="maintainer"/> <group groupid="obsprj3" role="maintainer"/> <repository name="openSUSE_13.2"> <path project="openSUSE.org:openSUSE:13.2" repository="standard"/> <arch>x86_64</arch> </repository> </project>
3.4.3 Package Metadata #
XML file about package meta information, like Title, description, related user and groups with roles, build settings, publish settings, debug settings and more. Most XML tags are the same as for projects.
Example package metadata:
<package name="test1" project="Test11"> <title>A test package for learning</title> <description>An example test package for learning. </description> <person userid="user5" role="bugowner"/> <person userid="user5" role="maintainer"/> <build> <enable repository="openSUSE_13.2"/> </build> <publish> <disable repository="openSUSE_13.2"/> </publish> <debuginfo> <disable/> </debuginfo> </package>
3.4.4 Attribute Metadata #
Attributes can be used to add special information to packages. Attributes can be used to trigger special actions.
Example attribute data:
<attributes> <attribute name="Issues" namespace="OBS"/> <attribute name="AutoCleanup" namespace="OBS"> <value>2016-06-30 00:00:00</value> </attribute> <attribute name="AutoCleanup" namespace="OBS"> <value></value> </attribute> </attributes>
3.4.5 Job Files #
Jobs are stored by the scheduler in the
/srv/obs/jobs
directory and contain the build setup
information for the package, for example, a reference to the exact source version,
build dependencies, build repository information, timestamps.
Sample job file:
<buildinfo project="Release:Stable" repository="SLE-12_GA" package="CI-demo" srcserver="http://obs.b1-systems.de:5352" reposerver="http://obs.b1-systems.de:5252"> <job>Release:Stable::SLE-12_GA:: CI-demo-36db80552b735e193dced13f058f866f</job> <arch>x86_64</arch> <srcmd5>36db80552b735e193dced13f058f866f</srcmd5> <verifymd5>36db80552b735e193dced13f058f866f</verifymd5> <rev>2</rev> <disturl>obs://b1-systems.de/Release:Stable/SLE-12_GA/ 36db80552b735e193dced13f058f866f-CI-demo</disturl> <reason>new build</reason> <needed>0</needed> <revtime>1461077600</revtime> <readytime>1461077708</readytime> <file>CI-demo.spec</file> <versrel>0.1.9-2</versrel> <bcnt>1</bcnt> <release>2.1</release> <debuginfo>1</debuginfo> <prjconfconstraint>linux:version:min 3.0.0</prjconfconstraint> <bdep name="aaa_base" preinstall="1" runscripts="1" notmeta="1" /> <bdep name="attr" preinstall="1" notmeta="1" /> <bdep name="bash" preinstall="1" notmeta="1" /> <bdep name="coreutils" preinstall="1" notmeta="1" /> <bdep name="diffutils" preinstall="1" notmeta="1" /> <bdep name="filesystem" preinstall="1" notmeta="1" /> <bdep name="fillup" preinstall="1" notmeta="1" /> <bdep name="glibc" preinstall="1" notmeta="1" /> <bdep name="grep" preinstall="1" notmeta="1" /> <bdep name="libbz2-1" preinstall="1" notmeta="1" /> <bdep name="libgcc_s1" preinstall="1" notmeta="1" /> <bdep name="m4" preinstall="1" notmeta="1" /> [...] <path project="Release:Stable" repository="SLE-12_GA" server="http://obs.b1-systems.de:5252" /> <path project="SUSE:SLE-12:GA" repository="standard" server="http://obs.b1-systems.de:5252" /> </buildinfo>