+

=

Debian OpenStack packaging team home

Welcome!

You've reached the packaging team home page. This is the first step to contributing to the packaging of OpenStack in Debian. Feel free to discuss any of this page content with us on IRC: irc.debian.org, aka oftc network, on the #debian-openstack channel. You can also join the mailing list. Register here: http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/openstack-devel. If you don't know how to help, just ask, we'll be happy to explain. There's always a bunch of things to do, OpenStack packaging never stops. Also, there's a todo over here: https://wiki.debian.org/OpenStack/OpenStack/todo.

tl;dr: I just want to build an OpenStack package... upstream OpenStack contributors: this is for you!

This bit is intended for upstream OpenStack authors, who would like to (quickly) build a package to see why unit tests are breaking, and who will not necessarily know a lot about packaging.

Cloning and building

  1. If you don't want to break anything in your system, setup a new (Debian Jessie) VM to build package, using your virtualization of choice (KVM, VirtualBox, etc.).
  2. Get this package installed.
  3. Edit /etc/pkgos/pkgos.conf, and set the local mirror nearest to you in CLOSEST_DEBIAN_MIRROR.
  4. Launch pkgos-setup-sbuild: pkgos-setup-sbuild dev YOUR-SYSTEM-USERNAME. Note that it may take some time to get the gpg key generated.
  5. Edit /etc/schroot/setup.d/80sources and set the below packages repository instead of the localhost ones:
    deb http://kilo-jessie.pkgs.mirantis.com/debian/ jessie-kilo-backports main
    deb http://kilo-jessie.pkgs.mirantis.com/debian/ jessie-kilo-backports-nochange main
  6. Clone a package, for example: git clone git://git.debian.org/openstack/keystone.git && cd keystone (note: you can use ssh:// instead of git:// if you have an account on alioth.debian.org).
  7. Switch to the correct packaging branch. For example, with Kilo: git checkout -b debian/kilo origin/debian/kilo.
  8. Build using pkgos-bop: pkgos-bpo. Watch it and see why the unit test breaks...

Test a patch to fix something

If you want to test a patch, best is to generate one in debian/patches:
  1. Make sure you have the orig.tar.gz generated: ./debian/rules gen-orig-xz.
  2. Do any kind of modification to your code.
  3. Generate the patch using quilt: dpkg-source --commit.
  4. Remove the patch from the branch (as you only want it in the debian/patches folder): quilt pop -a && rm -rf .pc.
  5. Commit the resulting quilt patch with git: git add debian/patches/* && git commit debian/patches/*.
  6. Build the package again: pkgos-bop.
Note about dependencies/requirements: no, there is no tox, pip, or requirements.txt / test-requirements.txt involved here, only Build-Depends:, Build-Depends-Indep: and Depends in the debian/control file. Of course, these are kind of built out of the upstream informations in the requirements.txt / test-requirements.txt, but it's not just a copy/paste. However, you can check if you see obvious mistakes, by running pkgos-fetch-fake-repo (to initialize the /etc/pkgos/fake-jessie-mirror folder with Packages.gz and Sources.gz) then pkgos-reqsdiff to check for errors. Note that it is expected to see differences printed by pkgos-reqsdiff, as no upstream requirements is a perfect match for what should be done in a given distribution.

Yes, I can build, but how do I hack within the build environment?

That's easy. Get root, then do: schroot -c jessie-amd64-sbuild. This is a scratch pad, feel free to do anything from there, everything will be deleted when you exit this shell. You may want to install all of the build-dependencies of a package. First, make sure you have deb-src lines in your sources.list. For example:
deb-src http://kilo-jessie.pkgs.mirantis.com/debian/ jessie-kilo-backports main
deb-src http://kilo-jessie.pkgs.mirantis.com/debian/ jessie-kilo-backports-nochange main
Then simply fetch all build-dependencies with apt-get build-dep and download the package sources. For example, for Neutron:
apt-get install git-buildpackage fakeroot devscripts --no-install-recommends
apt-get build-dep neutron --no-install-recommends
git clone git://git.debian.org/openstack/neutron.git
cd neutron
git checkout -b debian/kilo
You can run the unit tests of each packages using ./debian/rules override_dh_auto_test, or simply run testr manually (remember the rules: you should never use tox, but you can apt-get...).

Still stuck?

Get in touch and join #debian-openstack on irc.debian.org (aka: the OFTC network). There, you can catch zigo (ie: myself) and ask whatever you need. Note that I'm also available on Freenode, but the Debian channel for packaging OpenStack is on OFTC.

Mandatory packaging workflow

The above is a snapshot of /usr/share/openstack-pkg-tools/pkgos.make

The first time

  1. Clone the repository:
    git clone git+ssh://git.debian.org/git/openstack/glance.git
  2. cd glance
  3. Add the upstream remote (only needed if you need to merge upstream master into the packaging branch):
    ./debian/rules fetch-upstream-remote
    Note that at no point, the upstream "master" bramch should ever be uploaded to Alioth. The upstream tags in the packaging branch are enough to generate the orig.tar.xz.

Get all openstack repositories

  1. Clone the opestack-myrepos repository:
    git clone git+ssh://git.debian.org/git/openstack/openstack-myrepos.git
  2. cd openstack-myrepos
  3. Add .mrconfig to your trust file:
    echo $PWD/.mrconfig >> ~/.mrtrust
  4. Update the repositories:
    mr update

Uploading a new version

  1. Fetch from it, so you'll get new tags:
    ./debian/rules fetch-upstream-remote
  2. If necessary, checkout the Debian branch you need:
    git checkout debian/unstable
  3. Merge upstream changes in the Debian branch by merging the tag you want, for example 2011.3:
    git merge -X theirs 2011.3
  4. If you need, build the orig.tar.xz:
    ./debian/rules gen-orig-xz
  5. Update your packaging if needed, commit as you need. Do small, atomic commits.
    git commit ...
  6. When finished, tag and sign what you uploaded to Debian, for example version 2011.3-1:
    git tag -s debian/2011.3-1 -m 'Debian release 2011.3-1'
  7. Then you can push everything on Alioth:
    git push && git push --tags

Managing upstream pre-release tags

In OpenStack, there are pre-releases that you might wish to package. These releases are using tags which are a bit annoying for us. For example, 2013.2.rc1. In this case, we need to make sure that it becomes 2013.2~rc1 once packaged. The issue is that the ~ char is not useable in Git. Therefore, you can use _, and the scripts in openstack-pkg-tools will do the rest (and convert the _ into ~). So you will need to retag: git tag 2013.2_rc1 2013.2.rc1 And don't forget to "git push --tags" so that the new tag is on Alioth. Remember that we do not need the master branch to be hosted in Alioth, and that the packaging branch should be the default one. If you do a mistake, you can always change the default branch by going on the Alioth machine, and inside your Git repository, do:
git symbolic-ref HEAD refs/head/debian/havana

Helper and tooling using openstack-pkg-tools (and its pkgos-* helpers)

What is openstack-pkg-tools for

The openstack-pkg-tools has multiple use. First, it is used as part of the build-depends of many OpenStack packages, to automatically generate init scripts (for sysv-rc, upstart, or systemd), using a minimalistic template in debian/*.init.in. It is also used as a collection of shell script functions which may be use in maintainer's scripts (ie: *.postinst, *.config). It also has some Makefile include, so that maintainers are able to do some action to manage upstream Git repositories. So, in other words, it helps factoring many scripts into a unified system, to avoid code duplication.

But that's not it. openstack-pkg-tools also provides some build tools for the OpenStack maintainers, like a setup script for sbuild and Jenkins, scripts to build packages automatically (using sbuild), and so on. What's below describes how to use all that on your latop/workstation.

pkgos-setup-sbuild

If you wish to reproduce the build of packages before uploading a change to Alioth, then best is if you use sbuild and all the same tooling which the OpenStack packaging team use. To do so, install openstack-pkg-tools on your machine (note: as of writing, you need version 23 from the debian/experimental branch on the Alioth git). Once that's installed, make sure to edit /etc/pkgos/pkgos.conf (especially, you will want to set the correct Debian repository close to you instead of http.debian.net). Then simply type pkgos-setup-sbuild dev USERNAME (replace USERNAME by your actual username on your system). This will install a local Debian repository in /home/ftp, setup apache and pure-ftpd to provide it to the sbuild chroot.

Once you have completed the installation of sbuild using the pkgos-setup-sbuild script, you may want to edit /etc/schroot/setup.d/80sources. This file contains the definitions of the OpenStack repositories which will be in use in the sbuild chroot. It is, by default, set to use the local repository of the system where openstack-pkg-tools is installed. If you haven't rebuilt all of the OpenStack packages, then maybe you want to use what's already built, and use something like this:

deb http://kilo-jessie.pkgs.mirantis.com/debian jessie-kilo-backports main
deb http://kilo-jessie.pkgs.mirantis.com/debian jessie-kilo-backports-nochange main
and then update your schroot using sbuild-update -udcar jessie-amd64-sbuild.

Bip ... bop!

Once your sbuild system is ready, simply close a package. Let's start with one of the smallest as a first try:
git clone ssh://git.debian.org/git/openstack/spice-html5.git
cd spice-html5
pkgos-bop
The pkgos-bop script will create the .orig.tar.xz file for you, then use git-buildpackage to build (which will use sbuild).

pkgos-debpypi

If you want a quick way to package a new python module, you can use pkgos-debpypi. Note that you will *have* to fix the debian/copyright and the long and short descriptions, plus reviewing all of the packaging is completely mandatory. This is just to make things faster, this doesn't remove completely the packaging work.

Once your done: uploading to Alioth with correct rights

It is easy to upload a git to Alioth. It isn't to do it correctly. If you don't pay attention, you will be the only one with write access to your Git repository. The below script, to be run on Alioth tells you what you have to do.
#!/bin/sh
set -e

PKG_NAME=python-tooz

DEST_PROJECT=openstack
cd /git/${DEST_PROJECT}

echo '===> Activating update-server-info hook'
mv ${PKG_NAME}.git/hooks/post-update.sample ${PKG_NAME}.git/hooks/post-update
cd /git/${DEST_PROJECT}/${PKG_NAME}.git
git --bare update-server-info

echo '===> Fxing g+w unix permissions'
find /git/${DEST_PROJECT}/${PKG_NAME}.git -exec chmod g+w {} \;
find . -type d -exec chmod g+s {} \;
git config core.sharedRepository group
Some of us use pkgos-alioth-new-git openstack for uploading a git repository to Alioth. To use it, simply go at the top of your local git repository, and type: pkgos-alioth-new-git openstack, and it will make an archive of a bare copy of your local repository, upload that archive to Alioth, uncompress it there in /git/openstack, then fix the access rights correctly. Note that this script is meant to be used by debian developers, so you will need to addapt it if your computer login doesn't match the login on Alioth (eg, add the -guest and so on).

Openstack Jenkins

All of OpenStack building is managed by a Jenkins machine which produces backports for various distributions (currently, Jessie and Trusty). The Jenkins setup is done automatically using pkgos-setup-sbuild jenkins (and we would welcome you to try it). Building also involves pkgos-bop, but also using a wrapper around it: pkgos-bop-jenkins. After a git push, it will automatically build the package that you are working on thanks to a recieve hook, and put it in a repository. For the moment, there's the following:
deb http://kilo-jessie.pkgs.mirantis.com/debian jessie-kilo-backports main
deb http://kilo-jessie.pkgs.mirantis.com/debian jessie-kilo-backports-nochange main


The 2nd repository is a bunch of backports made in a semi-automated fashion, while the first one contains only the packages maintained within the Debian packaging team for OpenStack.

We also maintain some Ubuntu packages, which are available from (this URL is to be updated):
deb http://havana.dev-ubuntu.pkgs.enovance.com/debian precise-havana-backports main
For the moment, you need all of the above 2 Debian repositories for Kilo, though this will change later on.

You can join #debian-openstack-commits to see the result of your git push as well. Once the build is completed, please test the resulting packages before uploading to Sid. Also, if you want to see the logs, you can browse the Jenkins:
https://kilo-jessie.pkgs.mirantis.com

Debconf handling

In many packages, there are automations to maintain the basic configuration. You can see examples on how to write them with a few Git commits (examples in the Trove and Neutron packages):