Introduction

This guides aims to kickstart people with working in existing Debian packages, either to backport software, patch existing packages or work on security issues as part of the security team or the LTS project.

This guide assumes that the Debian package already exists and you are making modifications to it. If you wish to make a new Debian package, there are 3 different guides that you can follow, and I am not going to create a fourth one (even this guide is duplicating existing efforts already). So go see one of those:

• Debian New Maintainer's Guide: the one I followed originally
• Guide for Debian Maintainers: a new version of the above, basically using the new debmake instead of dh-make, untested
• Introduction to Debian packaging: a set of slides, a good primer for people that like slides, although a bit technical

All those are part of the developer's manual suite which also includes the Debian policy and the Debian developer's reference, two more reference manuals which you may find useful when looking for more information.

Minimal packaging workflow

This guides tries to take an opinionated approach to maintaining Debian packages. It doesn't try to cover all cases, doesn't try to teach you about debhelper, cdbs, uscan or make. It assumes you will find that information elsewhere, for example in the above references, and that you are already somewhat familiar with Debian systems administration (you know how to use a shell) and Debian packages as a concept (you know what a .deb file is and know how to use dpkg -i).

This will guide you through a standardized approach to:

• download Debian package source code
• make modifications to packages
• build packages, especially with multiple environments (testing, unstable, backports)
• upload packages

It covers a workflow that could be summarily described by this diagram:

Find the source

In the following, I take the example of building a backport of the Calibre package, which I needed. It's a good example because it does not use a git repository to track the Debian package source code, but Bazaar, which I am not familiar enough with to feel comfortable working on.¹

To get the source code on an arbitrary package, visit the package tracker.² In this case, we look at the Calibre package tracker page and find the download links for the release we're interested in. Since we are doing a backport, we use the testing download link. If you are looking for an packages not in a distribution or antique package, you can also find download links on snapshot.debian.org and archive.debian.net.

debian:
 calibre | 0.7.7+dfsg-1squeeze1   | squeeze          | source, all
 calibre | 0.8.51+dfsg1-0.1       | wheezy           | source, all
 calibre | 1.22.0+dfsg1-1~bpo70+2 | wheezy-backports | source, all
 calibre | 2.5.0+dfsg-1           | jessie-kfreebsd  | source, all
 calibre | 2.5.0+dfsg-1           | jessie           | source, all
 calibre | 2.55.0+dfsg-1          | stretch          | source, all
 calibre | 2.55.0+dfsg-1          | sid              | source, all
ubuntu:
 calibre | 0.8.38+dfsg-1        | precise/universe        | source, all
 calibre | 1.25.0+dfsg-1build1  | trusty/universe         | source, all
 calibre | 1.25.0+dfsg-1ubuntu1 | trusty-updates/universe | source, all
 calibre | 2.20.0+dfsg-1        | vivid/universe          | source, all
 calibre | 2.33.0+dfsg-1build1  | wily/universe           | source, all
 calibre | 2.55.0+dfsg-1        | xenial/universe         | source, all
 calibre | 2.55.0+dfsg-1build1  | yakkety/universe        | source, all

RMADISON_DEFAULT_URL=debian,ubuntu

What we are looking for is the calibre_2.55.0+dfsg-1.dsc file, the "source description" file for the 2.55.0+dfsg-1 version that is currently in stretch. Using that .dsc file, we can get all we need to build the package. So the first step is to download the source code, using dget(1):

$ dget http://httpredir.debian.org/debian/pool/main/c/calibre/calibre_2.55.0+dfsg-1.dsc
dget: retrieving http://httpredir.debian.org/debian/pool/main/c/calibre/calibre_2.55.0+dfsg-1.dsc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  2620  100  2620    0     0   1349      0  0:00:01  0:00:01 --:--:-- 36388
dget: retrieving http://httpredir.debian.org/debian/pool/main/c/calibre/calibre_2.55.0+dfsg.orig.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 37.3M  100 37.3M    0     0  2931k      0  0:00:13  0:00:13 --:--:-- 3032k
dget: retrieving http://httpredir.debian.org/debian/pool/main/c/calibre/calibre_2.55.0+dfsg-1.debian.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 22800  100 22800    0     0  59423      0 --:--:-- --:--:-- --:--:-- 59423
calibre_2.55.0+dfsg-1.dsc:
      Good signature found
   validating calibre_2.55.0+dfsg.orig.tar.xz
   validating calibre_2.55.0+dfsg-1.debian.tar.xz
All files validated successfully.
dpkg-source: info: extraction de calibre dans calibre-2.55.0+dfsg
dpkg-source: info: extraction de calibre_2.55.0+dfsg.orig.tar.xz
dpkg-source: info: extraction de calibre_2.55.0+dfsg-1.debian.tar.xz
dpkg-source: info: mise en place de no_updates_dialog.patch
dpkg-source: info: mise en place de disable_plugins.py
dpkg-source: info: mise en place de use-system-feedparser.patch
dpkg-source: info: mise en place de python_multiarch_inc.patch
dpkg-source: info: mise en place de dont_build_unrar_plugin.patch
dpkg-source: info: mise en place de mips_no_build_threads.patch
dpkg-source: info: mise en place de links-privacy.patch

A lot of stuff has happened here!

First, dget downloaded the .dsc file, which includes references to the .orig.tar.xz file and the debian.tar.xz files. The .orig.tar.xz is the upstream source code,³ and the .debian.tar.xz file is basically the content of the debian/ subdirectory, the metadata used to build the debian package, including all the patches specific to Debian.

Then dget downloads the files .orig.tar.xz and .debian.tar.xz files.

Then the OpenPGP signature on the .dsc file is verified against the web of trust,⁴ using dscverify. The .dsc files includes checksums for the downloaded files, and those checksums are verified as well.

Then the files are extracted using dpkg-source -x.

Notice how dget is just a shortcut to commands you could all have ran by hand.

If the version control system the package uses is familiar to you, you can use debcheckout to checkout the source directly. However, keep in mind that it does not ensure end-to-end cryptographic integrity like the previous procedure does, and instead relies on HTTPS-level transport security.

It might be useful if you prefer to collaborate with GitLab merge requests over at salsa.debian.org, but be warned that not all maintainers watch their GitLab projects as closely as the bug tracking system.

Modifying the package

At this point, we have a shiny source tree available in the calibre-2.55.0+dfsg/ directory:

cd calibre-2.55.0+dfsg/

We can start looking around and make some changes.

Changing version

The first thing we want to make sure we do is to bump the version number so that we don't mistakenly build a new package with the same version number but with undocumented changes. The Debian package version is stored in debian/changelog.

In the case of calibre, I only needed to change that file to complete the backport. This is called a "trivial backport": we just need to recompile against the stable environment. In other cases, dependencies need to be modified, patches need to be included and so on. But let's stick with the version number for now.

The generate the new version, I use dch --bpo, which chooses the right version number for me and pops open my $EDITOR so I can add things to the changelog:

dch --bpo

In this case, this created an entry for the 2.55.0+dfsg-1~bpo8+1 version⁵. I also added a Closes entry to indicate that the upload will fix a bug report I opened about creating that very backport. It's important to describe exactly what you are doing in the changelog and what bugs are being fixed. Here's the result:

calibre (2.55.0+dfsg-1~bpo8+1) jessie-backports; urgency=medium

  * Rebuild for jessie-backports (Closes: #818309)

 -- Antoine Beaupré <anarcat@debian.org>  Tue, 26 Apr 2016 16:49:56 -0400

Note that there are other options you can pass to dch. I often use:

• --bpo for backports
• --security, -s for security uploads
• --nmu, -n for non-maintainer uploads
• --increment, -i for my own packages

There are more described in the dch manpage. The managing packages section of the developer's reference is useful in crafting packages specific to your situation.

Changing package metadata

If I needed to modify dependencies, I would have edited debian/control directly. Other modifications to the Debian package would also happen in the debian/ directory. The function of the various files in that directory vary a lot according to how the package is built, but a good starting point is Debian policy §4: Source packages.

Modifying the source code

If I needed to modify the source tree outside debian/, I can do the modifications directly, then use dpkg-source --commit to generate a patch that will end up added to the quilt patchset in debian/patches. New patches should follow the patch tagging guidelines and dpkg-source --commit will use that template when creating a new patch.

Applying patches

If I already have a patch I want to apply to the source tree, then quilt is even more important. The first step is to import the patch:

quilt import ~/patches/CVE-2015-8477.patch

The above simply adds the CVE-2015-8477.patch to the patch set, but does not apply. You can apply all patches with:

quilt push -a

Quilt may tell you that the patch fails to apply. You can try to force-apply it with a "fuzz" argument:

quilt push --fuzz 100

Note that if the patch fails to apply, quilt leaves the source tree intact, so you can try that again and again. If the above fails but you are confident you can manually fix the patch, you can force-apply the patch:

quilt push --force

... then apply the modifications by hand. Then you need to refresh the patch to make sure it is updated correctly:

quilt refresh

Review the patch in debian/patches to make sure it still looks sane.

Notice how you could also have bypassed quilt completely and applied the patch with the patch command directly, then use dpkg-source --commit to generate a completely new patch.

Also note that the above does not care where the patch comes from. I often extract the patch from a Git source tree fetched with debcheckout on the side, with, for example:

( cd ../source-git ; git show $hash ) > debian/patches/CVE-2015-8477.patch
quilt import debian/patches/CVE-2015-8477.patch
quilt push

Again, it's useful to add metadata to the patch and follow the patch tagging guidelines.

Building the package

Now that we are satisfied with our modified package, we need to build it. The generic command to build a Debian package is dpkg-buildpackage⁶. So at the root of the source tree (above the debian/ directory), simply run:

dpkg-buildpackage

dpkg-buildpackage will the .deb file. It also creates new .dsc, .debian.tar.gz and .changes files.⁷ Those files should all show up in the parent directory.

In any case, there's a catch here: you need all the build-dependencies for the above builds to succeed. You may not have all of those, so you can try to install them with:

sudo mk-build-deps -i -r calibre

mk-build-deps makes a dummy package to wrap them all up together, so they are easy to uninstall. But this still installs a lot of cruft on your system that you don't want in the first place.

Furthermore, the above procedure doesn't build the package in a "clean environment". For example, say I am building a package for a regular upload into unstable ("sid"). Yet, my workstation is running stable ("jessie", currently). I can't simply build the package in jessie and expect it to work in sid, I need to build it into a sid environment.

For this, we need more powerful tools.

Building in a clean environment

I am using sbuild to build packages in a dedicated clean build environment. This means I can build packages for arbitrary distributions and also make sure there aren't exotic local configurations that can contaminate the build.

sbuild takes your source package (the .dsc file), and builds it in a clean, temporary chroot. To create that .dsc file, you can use dpkg-buildpackage -S simply call sbuild in the source directory which will create it for you.

This manual covers two different ways of using sbuild:

1. with a schroot backend, which is basically just a chroot wrapper

2. with a autopkgtest and qemu backend, which enforces much stronger isolation at a (performance) cost

schroot instructions

To use sbuild with schroot, you first need to configure a schroot:

sudo sbuild-createchroot --include=eatmydata,gnupg unstable /srv/chroot/unstable-amd64-sbuild http://deb.debian.org/debian

The above will create chroots for the unstable suite with an amd64 architecture, using debootstrap. You may of course modify this to taste based on your requirements and available disk space. My build directories count for around 7GB (including ~3GB of cached .deb packages) and each chroot is between 500MB and 700MB.

aliases=bookworm-security-amd64-sbuild,bookworm-backports-amd64-build

Qemu configuration

To use qemu as a backend, use this instead:

sudo mkdir -p /srv/sbuild/qemu/
sudo apt install sbuild-qemu
sudo sbuild-qemu-create -a amd64 -o /srv/sbuild/qemu/unstable-amd64.img unstable https://deb.debian.org/debian

This will create a single virtual machine image from the Debian unstable suite with an amd64 architecture. Thanks to qemu, you can use this to build or test packages in completely different architectures.

Then to make this used by default, add this to ~/.sbuildrc:

# run autopkgtest inside the schroot
$run_autopkgtest = 1;
# tell sbuild to use autopkgtest as a chroot
$chroot_mode = 'autopkgtest';
# tell autopkgtest to use qemu
$autopkgtest_virt_server = 'qemu';
# tell autopkgtest-virt-qemu the path to the image
# use --debug there to show what autopkgtest is doing
$autopkgtest_virt_server_options = [ '--', '/srv/sbuild/qemu/%r-%a.img' ];
# tell plain autopkgtest to use qemu, and the right image
$autopkgtest_opts = [ '--', 'qemu', '/srv/sbuild/qemu/%r-%a.img' ];
# no need to cleanup the chroot after build, we run in a completely clean VM
$purge_build_deps = 'never';
# no need for sudo
$autopkgtest_root_args = '';

Note that the above will use the default autopkgtest (1GB, one core) and qemu (128MB, one core) configuration, which might be a little low on resources. You probably want to be explicit about this, with something like this:

# extra parameters to pass to qemu
# --enable-kvm is not necessary, detected on the fly by autopkgtest
my @_qemu_options = ['--ram-size=4096', '--cpus=2'];
# tell autopkgtest-virt-qemu the path to the image
# use --debug there to show what autopkgtest is doing
$autopkgtest_virt_server_options = [ @_qemu_options, '--', '/srv/sbuild/qemu/%r-%a.img' ];
$autopkgtest_opts = [ '--', 'qemu', @_qemu_options, '/srv/sbuild/qemu/%r-%a.img'];

Also see a more in-depth discussion about this configuration in this blog post.

unshare configuration

Everyone seems to be switching to unshare as a schroot backend, including the official debian.org builders. The official sbuild instructions also use unshare now, this is my take.

1. install necessary packages

   sudo apt install sbuild mmdebstrap uidmap
   

2. create directory to store images

   mkdir -p ~/.cache/sbuild
   

   Note that jumped around quite a lot for me. It used to be in /srv/sbuild for qemu, but this was bind-mounted to /home/sbuild, which begs the question of why it can't just be in ~/.cache after all, although I have Puppet managing /srv.

3. create an unstable image:

   mmdebstrap --include=ca-certificates --skip=output/dev --variant=buildd unstable ~/.cache/sbuild/unstable-amd64.tar.zst https://deb.debian.org/debian
   

   This install took 42 seconds in my experience.

4. tweak .sbuildrc:

   $chroot_mode = 'unshare';
   

   Remove any $piuparts_opts, $autopkgtest_virt_server_options, or $autopkgtest_opts

The guide argues against running this, but you can update the tarballs with:

sbuild-update --chroot-mode=unshare --update --upgrade --dist-upgrade --autoremove oldstable stable unstable

This takes 6 seconds in my tests, much faster than rebuilding from scratch.

Note that we don't care about doing overlayfs or fancy copy-on-write stuff, the tarballs apparently take only two seconds to decompresss. See also this discussion.

Building with sbuild

Now that a sbuild backend is setup, I can build packages in one of three ways.

1. If I have a .dsc already (again, that can be generated with dpkg-buildpackage -S in the source tree):

   sbuild calibre_2.55.0+dfsg-1~bpo8+1.dsc
   

2. If I'm in the source tree:

   sbuild
   

3. With git-buildpackage:

   git-buildpackage --git-builder=sbuild
   

   The above can be configured by default in ~/.gbp.conf:

   [DEFAULT]
   builder=sbuild
   # to force lintian to run since we don't use debuild anymore
   postbuild = lintian $GBP_CHANGES_FILE
   

sbuild will generate your binary package in the parent directory (..). It will build your package with the suite specified in the package's latest changelog entry.

To pass options to the underlying dpkg-buildpackage (for example, you often need -sa to provide the source tarball with the upload), you should use --debbuildopts -sa in sbuild. For git-buildpackage, simply add -sa to the commandline.

Build servers

Sometimes, your machine is too slow to build this stuff yourself. If you have a more powerful machine lying around, you can send a source package to the machine and build the package there.

You first need to create a source package (the .dsc file created with dpkg-buildpackage -S) and transfer the files over. An easy way to do the latter is with the dcmd command. For example, this will create a source package, transfer it to the remote host example.net and build it:

foo-1.0$ dpkg-buildpackage -S
foo-1.0$ dcmd scp ../foo_1.0.dsc example.net:dist/
foo-1.0$ ssh example.net sbuild dist/foo_1.0.dsc

The above might cause trouble if you are working within a git repository, as dpkg-source might bundle .git files that should be ignored.

To build from git, you first use gitpkg to generate a .dsc file from the git tree, where rev is the current commit of the debian package (can be master or a specific tag) and upstream is a revision pointing at the upstream release to generate the .orig tarball if not already present:

gitpkg rev upstream

Then you can use the resulting .dsc file as normal.

cowpoke --buildd buildd.example.com --dist sid --arch amd64 calibre_2.55.0+dfsg-1~bpo8+1.dsc

If you do not have your own host to build packages, you can upload source packages to another buildd using dput, for example through debomatic. You need to request access first, however. More documentation is available on the Deb-o-Matic site.

In my experiments, Debomatic was way faster than compiling on my laptop, so I sometimes use it for larger packages. For example, building Xen on Debomatic takes around 6 minutes while on my laptop it takes almost triple that time (17 minutes). Plus Debomatic runs lintian and piuparts.

The obvious downside is that I need to trust the remote server to generate the same package as I would do locally. Even if the package is reproducible (which is not always the case!), I would still have to build the package locally to be able to compare the results...

Testing packages

Some packages have a built-in test suite which you should make sure runs properly during the build. Sometimes, backporting that test suite explicitly can be useful to ensure that everything works properly after a backport or LTS security upload. A key component of this is DEP8, currently implemented as autopkgtest. piuparts can also be used to see if the package cleans up properly after itself.

With autopkgtest

When a package self-testing enabled, it will be ran by Debian CI at various times. While there can be build-time tests, CI runs more often (when dependencies are updated, for example), and on the installed package, which is a different environment. To reproduce test failures or make sure changes won't break the test suite, autopkgtest can be ran locally. First, install it:

sudo apt install autopkgtest

Then you can run the tests against the built package:

autopkgtest libotr_4.1.1-3_amd64.changes -- schroot unstable-amd64-sbuild

That's it! Tests can also be ran against the current directory, in which case autopkgtest will build the package for you first, also in the virtual environment.

Testing by hand

In some cases, however, those tests are unavailable or insufficient and you need to actually install and run the package somewhere.

Backports can obviously be tested directly on your local machine if you are running stable (which is likely if you are building backports, unless you are doing them for someone else of course).

Otherwise, it is likely that you will need to build a separate environment to test the package if it is built for another distribution. For this, you can use the debootstrap and chroot commands. But it is probably better to run tests within a completely isolated environment, often called a "Virtual Machine".

Vagrant virtual machines

Hashicorp's Vagrant is a useful shortcut you can use to build virtual machines consistently. You can get started by using the following commands to get SSH into a Bookworm machine, for example:

sudo apt install vagrant
mkdir bookworm64; cd bookworm64
vagrant init debian/bookworm64
vagrant up
vagrant ssh

And that's it. vagrant init creates a Vagrantfile that basically describes how vagrant up can recreate the VM. You can use shell commands, Ansible, Puppet or other provisionning tools to automatically configure the VM, of course. Use vagrant halt to stop the VM and vagrant destroy to actually remove all data associated with the VM.

If the official Debian images are not up to your taste, you can build your own or choose another one from the Hashicorp Atlas.

Also note that Vagrant can use a libvirt backend, with the following Vagrantfile snippet:

config.vm.provider :libvirt do |libvirt|
    libvirt.graphics_type = "spice"
    libvirt.video_type = "qxl"
    libvirt.channel :type => 'spicevmc', :target_name => 'com.redhat.spice.0', :target_type => 'virtio'
    # add two devices using spicevmc channel
    (1..2).each do
      libvirt.redirdev :type => "spicevmc"
    end
    libvirt.memory = "2096"
  end

Qemu

Another simple approach is to use plain Qemu. We will need to use a special tool to create the virtual machine as debootstrap only creates a chroot, which virtual machines do not necessarily understand.

sudo grml-deboostratp --vmfile --bootappend console=ttyS0 --arch $ARCH --release $DIST --target $DIST.qcow2

To boot those images with Qemu, use:

qemu-system-x86_64 -snapshot -enable-kvm -display none -serial mon:stdio $DIST-$ARCH.qcow2

-snapshot makes the image read-only, so it can be readily reused without worring about contaminating the environment. KVM is obviously optional here, but usually works in my tests and is much faster than non-HVM usage. The remaining options are to make sure I get a regular terminal from Qemu instead of a graphical window. This requires serial console to be configured in the image, otherwise you will get no output at all. Also, if you are testing GUIs, you will obviously want to remove those options and install a bunch of packages on top of the minimal install.

nc -q 0 -l -p 10080 <  /var/cache/pbuilder/wheezy-amd64/result/phpmyadmin_3.4.11.1-2+deb7u4_all.deb 

nc 10.0.2.2 10080 > phpmyadmin_3.4.11.1-2+deb7u4_all.deb

# qemu: specify architecture, enable serial port and common port
# forwards (HTTP and SSH), enable KVM support and don't write the
# image by default (can be worked around with C-a s at
# runtime). graphical display still enabled for POLA (some VMs don't
# have serial), can be turned off with -display none.
alias qemu="qemu-system-x86_64 -serial mon:stdio -net user,hostfwd=tcp::10080-:80 -net user,hostfwd=tcp::10022-:22 -net nic -enable-kvm -snapshot"

unshare

Another option is to use the "unshare" command, which launches another command in a different namespace:

sudo unshare --ipc --mount --net --pid --uts --user --cgroup --time --fork chroot /path/mountpoint qemu-arm-static /bin/bash

The above launches the bash binary using the a ARM emulation wrapper (qemu-arm-static) inside a chroot at /path/mountpoint in a different namespace.

Yes, there should really be an --all option here. A previous version of this document omitted the --user --cgroup --time flags, for example.

This can reuse previously created chroots, but the filesystem separation works only if /path/mountpoint is really a different mountpoint. Otherwise changes in the filesystem affect the parent host, in which case you can just copy over the chroot.

Uploading packages

Uploading packages can be done on your own personal archive if you have a webserver, using the following ~/.dput.cf configuration:

[people]
fqdn = people.debian.org
method = scp
incoming = /home/anarcat/public_html/debian/wheezy-lts/
run_dinstall = 0
progress_indicator = 2

The above archive is not usable in a sources.list file, because that would involve an incoming queue and all sorts of complicated things. Instead, people are expected to use dget to download the .changes file and check the signatures. See the HEADER.html on my personal archive for an example.

You can also upload to the official Debian archives and backports, naturally. If you do not have access to those, you can also upload to mentors and request sponsorship.

Failing all that, you can upload packages to a Launchpad PPA or host your own Debian repository using reprepro (Koumbit has some good documentation) or aptly (haven't tested it myself).

Further work and remaining issues

This guide should be integrated into the official documentation or the Debian wiki. It is eerily similar to this guide which itself is a duplicate of this other guide.

A blog post goes into depth about the alternatives to qemu and sbuild.