1. Quick Debian development guide
  2. Find the source
  3. Modifying the package
    1. Changing version
    2. Changing package metadata
    3. Modifying the source code
    4. Applying patches
  4. Building the package
    1. Building in a clean environment
    2. Offloading: cowpoke and debomatic
  5. Testing packages
    1. Vagrant virtual machines
  6. Uploading packages
  7. Further work and remaining issues
This guide is also available under the URL https://deb.li/quickdev.

This guides aims to kickstart people with working in existing Debian packages, either to backport software, patch existing packages or work on security issues as part of the security team or the LTS project.

This guide assumes that the Debian package already exists and you are making modifications to it. If you wish to make a new Debian package, there are 3 different guides that you can follow, and I am not going to create a fourth one (even this guide is duplicating existing efforts already). :/ So go see one of those:

All those are part of the developer's manual suite which also includes the Debian policy and the Debian developer's reference, two more reference manuals which you may find useful when looking for more information.

This guides tries to take an opinionated approach to maintaining Debian packages. It doesn't try to cover all cases, doesn't try to teach you about debhelper, cdbs, uscan or make. It assumes you will find that information elsewhere, for example in the above references, and that you are already somewhat familiar with Debian systems administration (you know how to use a shell) and Debian packages as a concept (you know what a .deb file is and know how to use dpkg -i).

This will guide you through a standardized approach to:

It covers a workflow that could be summarily described by this diagram:

Find the source

In the following, I take the example of building a backport of the Calibre package, which I needed. It's a good example because it does not use a git repository to track the Debian package source code, but Bazaar, which I am not familiar enough with to feel comfortable working on.1

To get the source code on an arbitrary package, visit the package tracker.2 In this case, we look at the Calibre package tracker page and find the download links for the release we're interested in. Since we are doing a backport, we use the testing download link. If you are looking for an antique package, you can also find download links on archive.debian.net.

It's also helpful to use rmadison, part of the devscripts package, to look at the various versions available for a specific package in Debian. For example, here are the versions of Calibre available at the time of writing:

debian:
 calibre | 0.7.7+dfsg-1squeeze1   | squeeze          | source, all
 calibre | 0.8.51+dfsg1-0.1       | wheezy           | source, all
 calibre | 1.22.0+dfsg1-1~bpo70+2 | wheezy-backports | source, all
 calibre | 2.5.0+dfsg-1           | jessie-kfreebsd  | source, all
 calibre | 2.5.0+dfsg-1           | jessie           | source, all
 calibre | 2.55.0+dfsg-1          | stretch          | source, all
 calibre | 2.55.0+dfsg-1          | sid              | source, all
ubuntu:
 calibre | 0.8.38+dfsg-1        | precise/universe        | source, all
 calibre | 1.25.0+dfsg-1build1  | trusty/universe         | source, all
 calibre | 1.25.0+dfsg-1ubuntu1 | trusty-updates/universe | source, all
 calibre | 2.20.0+dfsg-1        | vivid/universe          | source, all
 calibre | 2.33.0+dfsg-1build1  | wily/universe           | source, all
 calibre | 2.55.0+dfsg-1        | xenial/universe         | source, all
 calibre | 2.55.0+dfsg-1build1  | yakkety/universe        | source, all

To get the Ubuntu results, I added the following line to my ~/.devscripts file:

RMADISON_DEFAULT_URL=debian,ubuntu

What we are looking for is the calibre_2.55.0+dfsg-1.dsc file, the "source description" file for the 2.55.0+dfsg-1 version that is currently in stretch. Using that .dsc file, we can get all we need to build the package. So the first step is to download the source code, using dget(1):

$ dget http://httpredir.debian.org/debian/pool/main/c/calibre/calibre_2.55.0+dfsg-1.dsc
dget: retrieving http://httpredir.debian.org/debian/pool/main/c/calibre/calibre_2.55.0+dfsg-1.dsc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  2620  100  2620    0     0   1349      0  0:00:01  0:00:01 --:--:-- 36388
dget: retrieving http://httpredir.debian.org/debian/pool/main/c/calibre/calibre_2.55.0+dfsg.orig.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 37.3M  100 37.3M    0     0  2931k      0  0:00:13  0:00:13 --:--:-- 3032k
dget: retrieving http://httpredir.debian.org/debian/pool/main/c/calibre/calibre_2.55.0+dfsg-1.debian.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 22800  100 22800    0     0  59423      0 --:--:-- --:--:-- --:--:-- 59423
calibre_2.55.0+dfsg-1.dsc:
      Good signature found
   validating calibre_2.55.0+dfsg.orig.tar.xz
   validating calibre_2.55.0+dfsg-1.debian.tar.xz
All files validated successfully.
dpkg-source: info: extraction de calibre dans calibre-2.55.0+dfsg
dpkg-source: info: extraction de calibre_2.55.0+dfsg.orig.tar.xz
dpkg-source: info: extraction de calibre_2.55.0+dfsg-1.debian.tar.xz
dpkg-source: info: mise en place de no_updates_dialog.patch
dpkg-source: info: mise en place de disable_plugins.py
dpkg-source: info: mise en place de use-system-feedparser.patch
dpkg-source: info: mise en place de python_multiarch_inc.patch
dpkg-source: info: mise en place de dont_build_unrar_plugin.patch
dpkg-source: info: mise en place de mips_no_build_threads.patch
dpkg-source: info: mise en place de links-privacy.patch

A lot of stuff has happened here!

First, dget downloaded the .dsc file, which includes references to the .orig.tar.xz file and the debian.tar.xz files. The .orig.tar.xz is the upstream source code,3 and the .debian.tar.xz file is basically the content of the debian/ subdirectory, the metadata used to build the debian package, including all the patches specific to Debian.

Then dget downloads the files .orig.tar.xz and .debian.tar.xz files.

Then the OpenPGP signature on the .dsc file is verified against the web of trust,4 using dscverify. The .dsc files includes checksums for the downloaded files, and those checksums are verified as well.

Then the files are extracted using dpkg-source -x. Notice how dget is basically just a shortcut to commands you could all have ran by hand. This is something useful to keep in mind to understand how this process works.

If the version control system the package uses is familiar to you, you can use debcheckout to checkout the source directly. If you are comfortable with many revision control systems, this may be better for you in general. However, keep in mind that it does not ensure end-to-end cryptographic integrity like the previous procedure does. It will be useful, however, if you want to review the source code history of the package to figure out where things come from.

Modifying the package

At this point, we have a shiny source tree available in the calibre-2.55.0+dfsg/ directory:

cd calibre-2.55.0+dfsg/

We can start looking around and make some changes.

Changing version

The first thing we want to make sure we do is to bump the version number so that we don't mistakenly build a new package with the same version number but with undocumented changes. The Debian package version is stored in debian/changelog.

In the case of calibre, I only needed to change that file to complete the backport. This is called a "trivial backport": we just need to recompile against the stable environment. In other cases, dependencies need to be modified, patches need to be included and so on. But let's stick with the version number for now.

The generate the new version, I use dch --bpo, which chooses the right version number for me and pops open my $EDITOR so I can add things to the changelog:

dch --bpo

In this case, this created an entry for the 2.55.0+dfsg-1~bpo8+1 version5. I also added a Closes entry to indicate that the upload will fix a bug report I opened about creating that very backport. It's important to describe exactly what you are doing in the changelog and what bugs are being fixed. Here's the result:

calibre (2.55.0+dfsg-1~bpo8+1) jessie-backports; urgency=medium

  * Rebuild for jessie-backports (Closes: #818309)

 -- Antoine Beaupré <anarcat@debian.org>  Tue, 26 Apr 2016 16:49:56 -0400

Note that there are other options you can pass to dch. I often use:

There are more described in the dch manpage. The managing packages section of the developer's reference is useful in crafting those specific packages.

Changing package metadata

If I needed to modify dependencies, I would have edited debian/control directly. Other modifications to the Debian package would also happen in the debian/ directory. The function of the various files in that directory vary a lot according to how the package is built, but a good starting point is Debian policy §4: Source packages.

Modifying the source code

If I needed to modify the source tree outside debian/, I can do the modifications directly, then use dpkg-source --commit to generate a patch that will end up added to the quilt patchset in debian/patches. New patches should follow the patch tagging guidelines and dpkg-source --commit will use that template when creating a new patch.

Applying patches

If I already have a patch I want to apply to the source tree, then quilt is even more important. The first step is to import the patch:

quilt import ~/patches/CVE-2015-8477.patch

The above simply adds the CVE-2015-8477.patch to the patch set, but does not apply. You can apply all patches with:

quilt push -a

Quilt may tell you that the patch fails to apply. You can try to force-apply it with a "fuzz" argument:

quilt push --fuzz 100

Note that if the patch fails to apply, quilt leaves the source tree intact, so you can try that again and again. If the above fails but you are confident you can manually fix the patch, you can force-apply the patch:

quilt push --force

... then apply the modifications by hand. Then you need to refresh the patch to make sure it is updated correctly:

quilt refresh

Review the patch in debian/patches to make sure it still looks sane.

Notice how you could also have bypassed quilt completely and applied the patch with the patch command directly, then use dpkg-source --commit to generate a completely new patch.

Also note that the above does not care where the patch comes from. I often extract the patch from a Git source tree fetched with debcheckout on the side, with, for example:

( cd ../source-git ; git show $hash ) > debian/patches/CVE-2015-8477.patch
quilt import debian/patches/CVE-2015-8477.patch
quilt push

Again, it's useful to add metadata to the patch and follow the patch tagging guidelines.

Building the package

Now that we are satisfied with our modified package, we need to build it. The generic command to build a Debian package is dpkg-buildpackage. So at the root of the source tree (above the debian/ directory), simply run:

dpkg-buildpackage

dpkg-buildpackage will the .deb file. It also creates new .dsc, .debian.tar.gz and .changes files.6 Those files should all show up in the parent directory.

If you are building from a VCS (e.g. git) checkout, you will get a lot of garbage in your source package. To avoid this, you need to use a tool specifically crafted for your VCS. I use git-buildpackage (or gbp in short) for that purpose, but other also use the simpler git-pkg. I find that gbp has more error checking but it is more complicated and less intuitive if you actually know what you are doing, which wasn't my case when I started.7

In any case, there's a catch here. The catch is that you need all the build-dependencies for the above builds to succeed. You may not have all of those, so you can try to install them with:

sudo mk-build-deps -i -r calibre

But this installs a lot of cruft on your system! mk-build-deps makes a dummy package to wrap them all up together, so they are easy to uninstall.

Furthermore, the above procedure doesn't build the package in a "clean environment". For example, say I am building a package for a regular upload into unstable ("sid"). Yet, my workstation is running stable ("jessie", currently). I can't simply build the package in jessie and expect it to work in sid, I need to build it into a sid environment.

For this, we need more powerful tools.

Building in a clean environment

I am using sbuild to build packages in a dedicated clean build environment. This means I can build packages for arbitrary distributions and also make sure there aren't exotic local configurations that can contaminate the build.

sbuild takes your source package (the .dsc file), and builds it in a clean, temporary chroot. To create that .dsc file, you can use dpkg-source -b or simply call sbuild in the source directory. Some places, like Debomatic, require a full .changes file, which is generated with dpkg-buildpackage -S.

To use sbuild, you first need to configure an image:

sudo sbuild-createchroot --include=eatmydata,ccache,gnupg unstable /srv/chroot/unstable-amd64-sbuild http://deb.debian.org/debian

This assumes that:

  1. you are running Stretch or later (see the sbuild wiki docs for workarounds in Jessie)

  2. sbuild is already installed and you are in the right group, do this otherwise:

    sudo apt-get install sbuild
    sudo sbuild-adduser $LOGNAME
    
  3. you want to create an "unstable" image in amd64. to change the architecture, use the --arch argument, and to change the suite, change it in two the places that say unstable, obviously

  4. you won't need to use hardlinks. overlay filesystems do not support hardlinks and you may need to switch to a tarball image if you need that feature (e.g. the mercurial test suite relies on this). to create a tarball image, use this:

    sudo sbuild-createchroot --make-sbuild-tarball=/srv/chroot/unstable-amd64-sbuild.tar.gz unstable `mktemp -d` http://deb.debian.org/debian
    

The above will create chroots for all the main suites and two architectures, using debootstrap. You may of course modify this to taste based on your requirements and available disk space. My build directories count for around 7GB (including ~3GB of cached .deb packages) and each chroot is between 500MB and 700MB.

Then I build packages in one of three ways.

  1. If I have a .dsc already (again, that can be generated with dpkg-source -b in the source tree):

    sbuild calibre_2.55.0+dfsg-1~bpo8+1.dsc
    
  2. If I'm in the source tree:

    sbuild
    
  3. With git-buildpackage:

    git-buildpackage --git-builder=sbuild
    

    The above can be configured by default in ~/.gbp.conf:

    [DEFAULT]
    builder=sbuild
    # to force lintian to run since we don't use debuild anymore
    postbuild = lintian $GBP_CHANGES_FILE
    

sbuild will generate your binary package in the parent directory (..). It will build your package with the suite specified in the package's latest changelog entry. If you want to use another suite, you can specify an arbitarry chroot with the --chroot option. See schroot -l to list the available options.

To pass options to the underlying dpkg-buildpackage (for example, you often need -sa to provide the source tarball with the upload), you should use --debbuildopts -sa in sbuild. For git-buildpackage, simply add -sa to the commandline.

A few handy sbuild-related commands:

Also note that it is useful to add aliases to your schroot configuration files. This allows you, for example, to automatically build wheezy-security or wheezy-backports packages in the wheezy schroot. Just add this line to the relevant config in /etc/schroot/chroot.d/:

aliases=wheezy-security-amd64-sbuild,wheezy-backports-amd64-build

I was previously using pbuilder and switched in 2017 to sbuild. AskUbuntu.com has a good comparative between pbuilder and sbuild that shows they are pretty similar. The big advantage of sbuild is that it is the tool in use on the buildds and it's written in Perl instead of shell. My concerns about switching were POLA (I'm used to pbuilder), the fact that pbuilder runs as a separate user (works with sbuild as well now, if the _apt user is present), and setting up COW semantics in sbuild (can't just plug cowbuilder there, need to configure overlayfs or aufs, which is non-trivial in jessie with backports...). Ubuntu folks, again, have more documentation there. Debian also has extensive documentation, especially about how to configure overlays. I was convinced by stapelberg's post on the topic which shows how simpler sbuild really is...

Offloading: cowpoke and debomatic

Sometimes, your machine is too slow to build this stuff yourself. If you have a more powerful machine lying around, you can use cowpoke to send builds to that machine. cowpoke operates on a source package (the .dsc file created with dpkg-source -b) and works well with gitpkg. By default, cowpoke logs into a remote server and uses sudo to call cowbuilder to build a chroot. For example, this will build calibre on the remote host buildd.example.com, in a jessie/amd64 chroot:

cowpoke --buildd buildd.example.com --dist sid --arch amd64 calibre_2.55.0+dfsg-1~bpo8+1.dsc

This assume the chroot already exists of course. You can create it by using the --create argument. It also only works for .dsc files, so it doesn't cooperate well with git-buildpackage, which expects a debuild-like interface.

To build from git, you first use gitpkg to generate a .dsc file from the git tree, where rev is the current commit of the debian package (can be master or a specific tag) and upstream is a revision pointing at the upstream release to generate the .orig tarball if not already present:

gitpkg rev upstream

Then call cowpoke on the resulting .dsc file.

If you do not have your own host to build packages, you can upload source packages to another buildd using dput, for example through debomatic. You need to request access first, however. More documentation is available on the Deb-o-Matic site.

In my experiments, Debomatic was way faster than compiling on my laptop, so I sometimes use it for larger packages. For example, building Xen on Debomatic takes around 6 minutes while on my laptop it takes almost triple that time (17 minutes). Plus Debomatic runs lintian and piuparts.

The obvious downside is that I need to trust the remote server to generate the same package as I would do locally. Even if the package is reproducible (which is not always the case!), I would still have to build the package locally to be able to compare the results...

Testing packages

Some packages have a built-in test suite which you should make sure runs properly during the build. Sometimes, backporting that test suite explicitly can be useful to ensure that everything works properly after a backport or LTS security upload. A key component of this is DEP8, currently implemented as autopkgtest. piuparts can also be used to see if the package cleans up properly after itself.

In some cases, however, those tests are not available or not sufficient and you need to actually install and run the package somewhere.

Backports can obviously be tested directly on your local machine if you are running stable (which is likely if you are building backports, unless you are doing them for someone else of course).

Otherwise, it is very likely that you will need to build a separate environment to test the package if it is built for another distribution. For this, you can use the debootstrap and chroot commands. But it is probably better to run tests within a completely isolated environment, often called a "Virtual Machine".

There are a lot of different virtualization solutions you can use (e.g. Xen, KVM, Docker or Virtualbox), which are for now considered to be outside the scope of this tutorial. I have also found libguestfs to be useful to operate on virtual images in various ways. Libvirt and Vagrant are also useful wrappers on top of the above systems.

I was previously using Qemu to run virtual machines, and had to create VMs by hand with various tools. This didn't work so well so I switched to using Vagrant as a de-facto standard to build development environment machines. Previous instructions for using Qemu are in the history of this page.

Vagrant virtual machines

Hashicorp's Vagrant is a useful shortcut you can use to build virtual machines consistently. You can get started by using the following commands to get SSH into a Wheezy machine, for example:

sudo apt install vagrant
mkdir wheezy64; cd wheezy64
vagrant init debian/wheezy64
vagrant up
vagrant ssh

And that's it. vagrant init creates a Vagrantfile that basically describes how vagrant up can recreate the VM. You can use shell commands, Ansible, Puppet or other provisionning tools to automatically configure the VM, of course. Use vagrant halt to stop the VM and vagrant destroy to actually remove all data associated with the VM.

Note that the directory where the VagrantFile is store is, by default, shared with the virtual machine, in the /vagrant directory. This allows you to share files between the host and the VM, and allow for files to "stick" when destroying the VM. This is synchronized using vagrant rsync, see the official Debian baseboxes documentation for more information.

If the official Debian images are not up to your taste, you can build your own or choose another one from the Hashicorp Atlas.

Uploading packages

Uploading packages can be done on your own personal archive if you have a webserver, using the following ~/.dput.cf configuration:

[people]
fqdn = people.debian.org
method = scp
incoming = /home/anarcat/public_html/debian/wheezy-lts/
run_dinstall = 0
progress_indicator = 2

The above archive is not usable in a sources.list file, because that would involve an incoming queue and all sorts of complicated things. Instead, people are expected to use dget to download the .changes file and check the signatures. See the HEADER.html on my personal archive for an example.

You can also upload to the official Debian archives and backports, naturally. If you do not have access to those, you can also upload to mentors and request sponsorship.

Failing all that, you can upload packages to a Launchpad PPA or host your own Debian repository using reprepro (Koumbit has some good documentation) or aptly (haven't tested it myself).

Further work and remaining issues

I am curious about Whalebuilder, which uses Docker to build packages instead of pbuilder or sbuild. Docker provides more isolation than a simple chroot: in whalebuilder, packages are built without network access and inside a virtualized environment. Keep in mind there are limitations to Docker's security and that pbuilder and sbuild do build under a different user which will limit the security issues with building untrusted packages. Furthermore, whalebuilder is not currently packaged as an official Debian package (it is now, see whalebuilder) and lacks certain features (like passing custom arguments to dpkg-buildpackage) so I don't feel it is quite ready yet. For now, if you need better isolation, look towards qemubuilder or possibly kvmtool.

This guide should be integrated into the official documentation or the Debian wiki. It is eerily similar to this guide which itself is a duplicate of this other guide.


  1. Well, it's not exactly true: I know bzr well enough, but I'm lazy. Besides, the point is to have a procedure that works regardless of the version control used. Furthermore, some packages do not use version control at all! ↩

  2. we use the tracker instead of apt-get source or dget package=version because those rely on a sources.list with all possible distributions enabled, something that is really inconvenient and harder to configure and document. ↩

  3. Well, not exactly: in this case, it's a modification the upstream source code, prepared specifically to remove non-free software, hence the +dfsg suffix, which is an acronym for Debian Free Software Guidelines. The +dfsg is simply a naming convention used to designated such modified tarballs. ↩

  4. In my case, this works cleanly, but that is only because the key is known on my system. dget actually offloads that work to dscverify which looks into the official keyrings in the debian-keyring package. This package may be missing some keys because the maintainers are new and were not in the keyring when the Debian version you are using was released. In this case, you need to find the key yourself, add it to your keyring, and then adding the following to ~/.devscripts will leverage your personal keys into the web of trust:

    DSCVERIFY_KEYRINGS=~/.gnupg/pubring.gpg
    

    You can also use dscvrify --keyring key.gpg *.dsc to check the signature by hand against a given key file. ↩

  5. The "tilde" character to indicate that this is a lower version than the version I am backporting from, so that when the users eventually upgrade to the next stable (stretch, in this case), they will actually upgrade to the real version in stretch, and not keep the backport lying around. There is a detailed algorithm description of how version number are compared, which you can test using dpkg --compare-versions if you are unsure. ↩

  6. the .changes file is similar to the .dsc file, but also covers the .deb file. So it's the .dsc file for the binary package. [^debuild]: I also often use debuild instead of dpkg-buildpackage because it also runs lintian and signs the binary package with debsign. ↩

  7. git-pkg actually only extracts a source package from your git tree, and nothing else. There are hooks to trigger builds and so on, but it's basically expected that you do that yourself, and gitpkg is just there to clean things up for your. git-buildpackage does way more stuff, which can be confusing for people more familiar with the Debian toolchain. ↩

Created . Edited .