Bookworm upgrade

  1. services
  2. upgrades
  1. Procedure
    1. Conflicts resolution
  2. Notable changes
    1. New packages
    2. My packages
    3. Updated packages
    4. Removed packages
    5. Puppet server upgrade
    6. pip install changes (PEP 668)
  3. Issues
    1. Pending
      1. ZFS upgrade failure
      2. MPD fails to start
    2. Resolved
      1. libcrypt disappeared on upgrade
      2. Packages to remove from my configuration
      3. Packages restored
      4. pip installation
  4. Troubleshooting
    1. Upgrade failures
    2. Reboot failures
    3. Finding orphaned and weird packages
  5. References

It's the moost wonderful tiiime of the yeaar! Yes, my friends, it's that time again, when the northern hemisphere freezes over and tries to make us forget that it might stop doing that soon and kill us all...

And yes, it's also the time where Debian starts heading towards a freeze, and when, as a dare devil, try to upgrade as many Debian I can lay my hands on to bookworm.

This document contains my upgrade procedure, notable changes in the new version, issues I have stumbled upon (and possibly fixed), and troubleshooting instructions.

It does not hope to replace the official documentation: it is a personal, living document that I have started keeping back when I upgraded to jessie. The other documents can be found in the parent upgrades page.

Procedure

This procedure is designed to be applied, in batch, on multiple servers. Do NOT follow this procedure unless you are familiar with the command line and the Debian upgrade process. It has been crafted by and for experienced system administrators that have dozens if not hundreds of servers to upgrade.

In particular, it runs almost completely unattended: configuration changes are not prompted during the upgrade, and just not applied at all, which will break services in many cases. I use a clean-conflicts script to do this all in one shot to shorten the upgrade process (without it, configuration file changes stop the upgrade at more or less random times). Then those changes get applied after a reboot. And yes, that's even more dangerous.

IMPORTANT: if you are doing this procedure over SSH (I had the privilege of having a console), you may want to upgrade SSH first as it has a longer downtime period, especially if you are on a flaky connection.

See the "conflicts resolution" section below for how to handle clean_conflicts output.

This procedure will kill your graphical session, so make sure you can log back in over a serial console or virtual terminal.

  1. Preparation:

    : reset to the default locale
export LC_ALL=C.UTF-8 &&
: install some dependencies
sudo apt install ttyrec screen debconf-utils deborphan &&
: create ttyrec file with adequate permissions &&
sudo touch /var/log/upgrade-bookworm.ttyrec &&
sudo chmod 600 /var/log/upgrade-bookworm.ttyrec &&
sudo ttyrec -a -e screen /var/log/upgrade-bookworm.ttyrec

  2. Backups and checks:

    ( 
  umask 0077 &&
  tar cfz /var/backups/pre-bookworm-backup.tgz /etc /var/lib/dpkg /var/lib/apt/extended_states /var/cache/debconf $( [ -e /var/lib/aptitude/pkgstates ] && echo /var/lib/aptitude/pkgstates ) &&
  dpkg --get-selections "*" > /var/backups/dpkg-selections-pre-bookworm.txt &&
  debconf-get-selections > /var/backups/debconf-selections-pre-bookworm.txt
) &&
( puppet agent --test || true )&&
apt-mark showhold &&
dpkg --audit &&
: look for dkms packages and make sure they are relevant, if not, purge. &&
( dpkg -l '*dkms' || true ) &&
: look for leftover config files &&
/home/anarcat/src/koumbit-scripts/vps/clean_conflicts &&
: run backups &&
/home/anarcat/bin/backup-$(hostname) &&
printf "End of Step 2\a\n"

  3. Perform any pending upgrade and clear out old pins:

    puppet agent --disable "running major upgrade" &&
apt update && apt -y upgrade &&
: Check for pinned, on hold, packages, and possibly disable &&
rm -f /etc/apt/preferences /etc/apt/preferences.d/* &&
rm -f /etc/apt/sources.list.d/backports.debian.org.list &&
rm -f /etc/apt/sources.list.d/backports.list &&
rm -f /etc/apt/sources.list.d/bookworm.list &&
rm -f /etc/apt/sources.list.d/bullseye.list &&
rm -f /etc/apt/sources.list.d/buster-backports.list &&
rm -f /etc/apt/sources.list.d/experimental.list &&
rm -f /etc/apt/sources.list.d/incoming.list &&
rm -f /etc/apt/sources.list.d/proposed-updates.list &&
rm -f /etc/apt/sources.list.d/sid.list &&
rm -f /etc/apt/sources.list.d/testing.list &&
: purge removed packages &&
apt purge $(dpkg -l | awk '/^rc/ { print $2 }') &&
apt purge '?obsolete' &&
apt autoremove -y --purge &&
: possibly clean up old kernels &&
dpkg -l 'linux-image-*' &&
: look for packages from backports, other suites or archives &&
: if possible, switch to official packages by disabling third-party repositories &&
apt list '?narrow(?installed, ?not(?origin(Debian)))' &&
apt list "?narrow(?installed, ?not(?codename($(lsb_release -c -s | tail -1))))" &&
printf "End of Step 3\a\n"

  4. Check free space (see this guide to free up space), disable auto-upgrades, and download packages:

    systemctl stop apt-daily.timer &&
sed -i 's#bullseye-security#bookworm-security#' $(ls /etc/apt/sources.list /etc/apt/sources.list.d/*) &&
sed -i 's/bullseye/bookworm/g' $(ls /etc/apt/sources.list /etc/apt/sources.list.d/*) &&
apt update &&
apt -y -d full-upgrade &&
apt -y -d upgrade &&
apt -y -d dist-upgrade &&
df -h &&
printf "End of Step 4\a\n"

  5. Actual upgrade run:

    : put server in maintenance &&
sudo touch /etc/nologin &&
env DEBIAN_FRONTEND=noninteractive APT_LISTCHANGES_FRONTEND=none APT_LISTBUGS_FRONTEND=none UCF_FORCE_CONFFOLD=y \
    apt full-upgrade -y -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold' &&
printf "End of Step 5\a\n"

  6. Post-upgrade procedures:

    apt-get update --allow-releaseinfo-change &&
puppet agent --enable &&
puppet agent -t --noop &&
printf "Press enter to continue, Ctrl-C to abort." &&
read -r _ &&
(puppet agent -t || true) &&
: rm -f /etc/apt/apt.conf.d/50unattended-upgrades.dpkg-dist /etc/ca-certificates.conf.dpkg-old /etc/cron.daily/bsdmainutils.dpkg-remove /etc/default/prometheus-apache-exporter.dpkg-dist /etc/default/prometheus-node-exporter.dpkg-dist /etc/logrotate.d/apache2.dpkg-dist /etc/nagios/nrpe.cfg.dpkg-dist /etc/ssh/ssh_config.dpkg-dist /etc/ssh/sshd_config.ucf-dist /etc/unbound/unbound.conf.dpkg-dist &&
printf "\a" &&
/home/anarcat/src/koumbit-scripts/vps/clean_conflicts &&
systemctl start apt-daily.timer &&
printf "End of Step 6\a\n" &&
shutdown -r +1 "rebooting to get rid of old kernel image..."

  7. Post-upgrade checks:

    export LC_ALL=C.UTF-8 &&
sudo ttyrec -a -e screen /var/log/upgrade-bookworm.ttyrec

apt-mark manual bind9-dnsutils puppet-agent
apt purge gcc-9-base gcc-10-base
apt purge $(dpkg -l | awk '/^rc/ { print $2 }') # purge removed packages
apt autoremove -y --purge
apt purge $(deborphan --guess-dummy)
while deborphan -n | grep -q . ; do apt purge $(deborphan -n); done
apt autoremove -y --purge
apt clean
# review and purge older kernel if the new one boots properly
dpkg -l 'linux-image*'
# review packages that are not in the new distribution
apt purge '?obsolete'
apt list "?narrow(?installed, ?not(?codename($(lsb_release -c -s | tail -1))))" &&
printf "All procedures completed\a\n" &&

Conflicts resolution

When the clean_conflicts script gets run, it asks you to check each configuration file that was modified locally but that the Debian package upgrade wants to overwrite. You need to make a decision on each file. This section aims to provide guidance on how to handle those prompts.

Those config files should be manually checked on each host:

     /etc/default/grub.dpkg-dist
     /etc/initramfs-tools/initramfs.conf.dpkg-dist

If other files come up, they should be added in the above decision list, or in an operation in step 2 or 7 of the above procedure, before the clean_conflicts call.

Files that should be updated in Puppet are mentioned in the Issues section below as well.

Notable changes

Here are some packages with notable version changes that I noticed.

See also the wiki page about bookworm for another list.

New packages

This is a curated list of packages that were introduced in bookworm. There are actually thousands of new packages in the new Debian release, but this is a small selection of projects I found particularly interesting:

TODO

See also:

My packages

In packages I maintain, those are the important changes:

TODO

Updated packages

This table summarizes package version changes I find interesting.

Package Bullseye Bookworm Notes
Emacs 27.1 28.1 native compilation, seccomp, better emoji support, 24-bit true color support in terminals, C-x 4 4 to display next command in a new window, [xterm-mouse-mode][], [context-menu-mode][], [repeat-mode][]
Firefox 91.13 102.11 91.13 already in buster-security
GNOME 3.38 43
Inkscape 1.0 1.2 1.2 release notes
Libreoffice 7.0 7.4
OpenSSH 8.4 9.2 scp now uses SFTP, NTRU quantum-resistant key exchange, SHA-1 disabled EnableEscapeCommandline
Python 3.9.2 3.11 Python 2 removed completely
Puppet 5.5.22 7.23 major work from colleagues and myself

Note that this table may not be up to date with the current bullseye release. See the official release notes for a more up to date list.

Removed packages

TODO

Python 2 was completely removed from Debian, a long-term task that had already started with bullseye, but not completed.

At the time of this writing (during freeze), there's a significant number of packages gone from bookworm, which I actually wanted to have installed on my machines:

Unfortunately, this late in the freeze, it's unlikely they will re-enter testing, so in all likelihood, I will learn to live without those.

See also the noteworthy obsolete packages list.

Puppet server upgrade

I had to apt install postgresql puppetdb puppet-terminus-puppetdb and follow the connect instructions, as I was using the redis terminus before. I also had to adduser puppetdb puppet for it to be able to access the certs, and add the certs to the jetty config. Basically:

certname="$(puppet config print certname)"
hostcert="$(puppet config print hostcert)"
hostkey="$(puppet config print hostprivkey)"
cacert="$(puppet config print cacert)"

adduser puppetdb puppet

cat >>/etc/puppetdb/conf.d/jetty.ini <<-EOF
    ssl-host = 0.0.0.0
    ssl-port = 8081
    ssl-key = ${hostkey}
    ssl-cert = ${hostcert}
    ssl-ca-cert = ${cacert}
EOF

echo "Starting PuppetDB ..."
systemctl start puppetdb

cp /usr/share/doc/puppet-terminus-puppetdb/routes.yaml.example /etc/puppet/routes.yaml
cat >/etc/puppet/puppetdb.conf <<-EOF
    [main]
    server_urls = https://${certname}:8081

also:

apt install puppet-module-puppetlabs-cron-core puppet-module-puppetlabs-augeas-core puppet-module-puppetlabs-sshkeys-core
puppetserver gem install trocla:0.4.0 --no-document

pip install changes (PEP 668)

This is not yet documented in the Debian release notes (1033564), but we are now enforcing [PEP 668][] which means that a simple pip install foo will fail unless you pass --break-system-packages or use a virtual environment.

Issues

See also the official list of known issues.

Pending

ZFS upgrade failure

The zfs-dkms package has this weird bug where it tries to configure the old package:

Setting up linux-image-6.0.0-4-amd64 (6.0.8-1) ...
/etc/kernel/postinst.d/dkms:
dkms: running auto installation service for kernel 6.0.0-4-amd64:Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/2.0.3/source/dkms.conf does not exist.
 failed!
run-parts: /etc/kernel/postinst.d/dkms exited with return code 4
dpkg: error processing package linux-image-6.0.0-4-amd64 (--configure):
 installed linux-image-6.0.0-4-amd64 package post-installation script subprocess returned error exit status 1

The workaround:

rm -rf /var/lib/dkms/2.0.3

This bug was filed as bug 1024326 in the Debian package.

MPD fails to start

This one was hard to diagnose because the normal output does not show the real error:

$ mpd --no-daemon 
Nov 17 15:16 : server_socket: bind to '0.0.0.0:6600' failed (continuing anyway, because binding to '[::]:6600' succeeded): Failed to bind socket: Address already in use
Nov 17 15:16 : exception: Tag list mismatch, discarding database file
Abandon

The full error is visible only with --verbose:

anarcat@curie:~$ mpd --no-daemon --stdout --verbose
config_file: loading file /home/anarcat/.mpdconf
server_socket: bind to '0.0.0.0:6600' failed (continuing anyway, because binding to '[::]:6600' succeeded): Failed to bind socket: Address already in use
libsamplerate: libsamplerate converter 'Fastest Sinc Interpolator'
vorbis: Xiph.Org libVorbis 1.3.7
opus: libopus 1.3.1
sndfile: libsndfile-1.1.0
hybrid_dsd: The Hybrid DSD decoder is disabled because it was not explicitly enabled
adplug: adplug 2.3.3
simple_db: reading DB
exception: Tag list mismatch, discarding database file
curl: version 7.86.0
curl: with GnuTLS/3.7.8
update: spawned thread for update job id 1
state_file: Loading state file /home/anarcat/.mpd/state
update: starting
terminate called after throwing an instance of 'std::runtime_error'
  what():  io_uring_get_sqe() failed
Abandon

... and this is bug 1023872. The workaround is to install liburing2 from unstable, while we wait for upstream to fix this.

Resolved

libcrypt disappeared on upgrade

It's unclear how that happened, but I somehow lost libcrypt.so.1 during the upgrade, which pretty much broke everything, including Python, Perl, and Ruby, which, in turn, broke apt-listbugs, apt-listchanges, and, more critically, debconf. The later was especially hairy because libc requires debconf to at least function (if it's installed!) during its postinst.

The workaround was to add a bunch of false && in the libc postinst. I also had to manually install a bunch of packages with dpkg from /var/cache/apt/archives. And other tricks, I don't remember it all but it was scary and hairy.

Packages to remove from my configuration

Packages restored

Those packages were removed from bookworm and eventually restored.

pip installation

NOTE: this procedure used to advise to use pip to reinstall packages but my own packages are all in Debian now, so that is not an issue anymore. But in case that problem comes up again in the future, what I was doing was:

: reinstall Python packages to follow Python upgrade &&
for package in rsendmail ; do
    cd ~anarcat/src/$package && pip3 install . || echo WARNING: failed to install $package
done

This wasn't great. First, it now doesn't work in bookworm anymore because pip forbids installing packages that way, you must use virtual environments. Second, it doesn't cleanup old things and will forget stuff that hasn't been added to the procedure. I have considered this instead:

pip freeze --local > requirements.txt
apt upgrade
pip install -r requirements.txt
rm -rf .../python3.x # remove old crap

That way we reinstall what's already setup, and we have pinned versions, but not checksums.

Troubleshooting

Upgrade failures

Instructions on errors during upgrades can be found in the release notes troubleshooting section.

Reboot failures

If there's any trouble during reboots, you should use some recovery system. The release notes actually have good documentation on that, on top of "use a live filesystem".

Finding orphaned and weird packages

The apt-forktracer call above used to have many other different incantations, and it's not yet clear that it does everything we need. What we want to find are basically packages that are not "canonical Debian packages", which are shipped by the stable Debian distribution. Those are typically called "obsolete" packages in Debian, but that term is somewhat to narrow, as I also want to consider packages that were never part of Debian at all.

Weirdly, the release notes suggest three different methods to do this, in different part of the documentation. I filed this as a bug in 987017, but it's still not settled. The previous version of this guide (i.e. bullseye) discussed many alternatives but also did not settled on a single one.

The bug report seems to settle to running this before the upgrade, to see which packages are from backports or weird third-party repos, with:

apt list '?narrow(?installed, ?not(?origin(Debian)))'

... which is roughly equivalent to apt-forktracer, but with less details (e.g. it doesn't show the version in the repo).

Then, after the upgrade, we list obsolete packages, which are not managed by Debian anymore:

apt list '?obsolete'

To remove those:

apt purge '?obsolete'

Those didn't catch the non-standard versions that apt-show-versions caught however. This will:

apt list '?narrow(?installed, ?not(?codename(bookworm)))'

That effectively replaces the old apt-show-versions | grep -v /bookworm hack.

TODO: update actual procedure with the above.

References

Created . Edited .