Bullseye upgrade

  1. services
  2. upgrades
  1. Procedure
  2. Notable changes
    1. New packages
    2. My packages
    3. Updated packages
    4. Removed packages
    5. Security updates URL changes
  3. Issues
    1. Pending
      1. Critical packages missing
      2. Puppet breaks in bullseye/sid
    2. Resolved
      1. Cool things I want to try
      2. Browserpass fails to upgrade
      3. Double redshift
      4. Emacs took 2 minutes to start
      5. i3-focus and rsendmail delivery failed
      6. Not enough disk space
      7. Packages mistakenly removed
      8. LUKS password prompt in plain text instead of GUI
  4. Troubleshooting
    1. Upgrade failures
    2. Reboot failures
    3. Finding orphaned and weird packages
      1. aptitude search 1
      2. apt-show-versions
      3. aptitude 2: ~obsolete
      4. apt-forktracer
      5. apt list
  5. References

It's Debian major upgrade time again! My personal policy is generally to upgrade slightly before or during the freeze. This time I feel almost late because it seems we'll be releasing in almost a month now (May 2021, it's April 2021 now).

Update: this procedure was tested first on my workstation (curie). I have done another (the laptop, angela) in July 2021, and the server (marcos) in October 2021.

This document contains my upgrade procedure, notable changes in the new version, issues I have stumbled upon (and possibly fixed), and troubleshooting instructions.

It does not hope to replace the official documentation: it is a personal, living document that I have started keeping back when I upgraded to jessie. The other documents can be found in the parent upgrades page.

Procedure

This procedure is designed to be applied, in batch, on multiple servers. Do NOT follow this procedure unless you are familiar with the command line and the Debian upgrade process. It has been crafted by and for experienced system administrators that have dozens if not hundreds of servers to upgrade.

In particular, it runs almost completely unattended: configuration changes are not prompted during the upgrade, and just not applied at all, which will break services in many cases. I use a clean-conflicts script to do this all in one shot to shorten the upgrade process (without it, configuration file changes stop the upgrade at more or less random times). Then those changes get applied after a reboot. And yes, that's even more dangerous.

IMPORTANT: if you are doing this procedure over SSH (I had the privilege of having a console), you may want to upgrade SSH first as it has a longer downtime period, especially if you are on a flaky connection.

  1. Preparation:

    : reset to the default locale
export LC_ALL=C.UTF-8 &&
: put server in maintenance &&
sudo touch /etc/nologin &&
: install some dependencies
sudo apt install ttyrec screen debconf-utils deborphan apt-forktracer &&
: create ttyrec file with adequate permissions &&
sudo touch /var/log/upgrade-bullseye.ttyrec &&
sudo chmod 600 /var/log/upgrade-bullseye.ttyrec &&
sudo ttyrec -e screen /var/log/upgrade-bullseye.ttyrec

  2. Backups and checks:

    ( 
  umask 0077 &&
  tar cfz /var/backups/pre-bullseye-backup.tgz /etc /var/lib/dpkg /var/lib/apt/extended_states /var/cache/debconf $( [ -e /var/lib/aptitude/pkgstates ] && echo /var/lib/aptitude/pkgstates ) &&
  dpkg --get-selections "*" > /var/backups/dpkg-selections-pre-bullseye.txt &&
  debconf-get-selections > /var/backups/debconf-selections-pre-bullseye.txt
) &&
( puppet agent --test || true )&&
apt-mark showhold &&
dpkg --audit &&
: look for dkms packages and make sure they are relevant, if not, purge. &&
( dpkg -l '*dkms' || true ) &&
: look for leftover config files &&
/home/anarcat/src/koumbit-scripts/vps/clean_conflicts &&
: run backups &&
/home/anarcat/bin/backup-$(hostname) &&
printf "End of Step 2\a\n"

  3. Perform any pending upgrade and clear out old pins:

    puppet agent --disable "running major upgrade" &&
apt update && apt -y upgrade &&
: Check for pinned, on hold, packages, and possibly disable &&
rm -f /etc/apt/preferences /etc/apt/preferences.d/* &&
rm -f /etc/apt/sources.list.d/backports.debian.org.list &&
rm -f /etc/apt/sources.list.d/backports.list &&
rm -f /etc/apt/sources.list.d/bookworm.list &&
rm -f /etc/apt/sources.list.d/bullseye.list &&
rm -f /etc/apt/sources.list.d/buster-backports.list &&
rm -f /etc/apt/sources.list.d/experimental.list &&
rm -f /etc/apt/sources.list.d/incoming.list &&
rm -f /etc/apt/sources.list.d/proposed-updates.list &&
rm -f /etc/apt/sources.list.d/sid.list &&
rm -f /etc/apt/sources.list.d/testing.list &&
: purge removed packages &&
apt purge $(dpkg -l | awk '/^rc/ { print $2 }') &&
apt autoremove -y --purge &&
: possibly clean up old kernels &&
dpkg -l 'linux-image-*' &&
: look for packages from backports, other suites or archives &&
: if possible, switch to official packages by disabling third-party repositories &&
apt-forktracer | sort &&
printf "End of Step 3\a\n"

  4. Check free space, see this guide to free up space and download packages:

    systemctl stop apt-daily.timer &&
sed -i 's#buster/updates#bullseye-security#' $(ls /etc/apt/sources.list /etc/apt/sources.list.d/*) &&
sed -i 's/buster/bullseye/g' $(ls /etc/apt/sources.list /etc/apt/sources.list.d/*) &&
apt update &&
apt -y -d full-upgrade &&
apt -y -d upgrade &&
apt -y -d dist-upgrade &&
df -h &&
printf "End of Step 4\a\n"

  5. Actual upgrade run:

    env DEBIAN_FRONTEND=noninteractive APT_LISTCHANGES_FRONTEND=none APT_LISTBUGS_FRONTEND=none UCF_FORCE_CONFFOLD=y \
    apt full-upgrade -y -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold' &&
printf "End of Step 5\a\n"

  6. Post-upgrade procedures:

    apt-get update --allow-releaseinfo-change &&
puppet agent --enable &&
puppet agent -t --noop &&
printf "Press enter to continue, Ctrl-C to abort." &&
read -r _ &&
(puppet agent -t || true) &&
: reinstall Python packages to follow Python upgrade &&
for package in rsendmail pubpaste ; do
    cd ~/src/$package && pip3 install .
done &&
: rm -f /etc/apt/apt.conf.d/50unattended-upgrades.dpkg-dist /etc/ca-certificates.conf.dpkg-old /etc/cron.daily/bsdmainutils.dpkg-remove /etc/default/prometheus-apache-exporter.dpkg-dist /etc/default/prometheus-node-exporter.dpkg-dist /etc/logrotate.d/apache2.dpkg-dist /etc/nagios/nrpe.cfg.dpkg-dist /etc/ssh/ssh_config.dpkg-dist /etc/ssh/sshd_config.ucf-dist /etc/unbound/unbound.conf.dpkg-dist &&
printf "\a" &&
/home/anarcat/src/koumbit-scripts/vps/clean_conflicts &&
systemctl start apt-daily.timer &&
printf "End of Step 6\a\n" &&
shutdown -r +1 "rebooting to get rid of old kernel image..."

  7. Post-upgrade checks:

    export LC_ALL=C.UTF-8 &&
sudo ttyrec -a -e screen /var/log/upgrade-bullseye.ttyrec

apt-mark manual bind9-dnsutils
apt purge libgcc1:amd64 gcc-8-base:amd64
apt purge $(dpkg -l | awk '/^rc/ { print $2 }') # purge removed packages
apt autoremove -y --purge
apt purge $(deborphan --guess-dummy | grep -v python-is-python2)
while deborphan -n | grep -v python-is-python2 | grep -q . ; do apt purge $(deborphan -n | grep -v python-is-python2); done
apt autoremove -y --purge
apt clean
# review and purge older kernel if the new one boots properly
dpkg -l 'linux-image*'
# review packages that are not in the new distribution
apt-forktracer | sort
printf "All procedures completed\a\n" &&

Notable changes

Here are some packages with notable version changes that I noticed.

There is a more exhaustive review of server-level changes from mikas as well. Notable:

See also the wiki page about bullseye for another list.

New packages

This is a curated list of packages that were introduced in bullseye. There are actually thousands of new packages in the new Debian release, but this is a small selection of projects I found particularly interesting:

My packages

In packages I maintain, those are the important changes:

Updated packages

This table summarizes package version changes I find interesting.

Package Buster Bullseye Notes
APT 1.8 2.2 2.0, 2.2
Browserpass 2.0 3.7 Major usability improvements
Docker 18 20 Docker made it for a second release
Emacs 26 27 JSON parsing for LSP? ~/.config/emacs/? harfbuzz?? oh my! details
Firefox 68 78 78 was already in buster-updates
Ganeti 2.16.0 3.0.1 breaking upgrade?
GNOME 3.30 3.38 Missed the "GNOME 40" release
Inkscape 0.92 1.0 Finally, 1.0!
Libreoffice 6.2 7.0
OpenSSH 7.9 8.4 FIDO/U2F, Include, signatures, quantum-resistant key exchange, key fingerprint as confirmation
Postgresql 11 13
Python 3.7 3.9 walrus operator, importlib.metadata, dict unions, zoneinfo
Puppet 5.5 5.5 Missed the Puppet 6 (and 7!) releases

Note that this table may not be up to date with the current bullseye release. See the official release notes for a more up to date list.

Removed packages

See also the noteworthy obsolete packages list.

Security updates URL changes

So you might have noticed this little line in the upgrade procedure:

sed -i 's#buster/updates#bullseye-security#' /etc/apt/sources.list $(ls /etc/apt/sources.list.d/*) &&

This is a known issue in this release, it's because the security sources.list is now:

deb https://deb.debian.org/debian-security bullseye-security main contrib

In buster, it was:

deb http://security.debian.org/ buster/updates main contrib non-free

... although this presumably worked already as well:

deb https://deb.debian.org/debian-security buster/updates main contrib

Also note that the codename of the security changed, this time, from buster to bullseye-security. This affects you only if you have an APT configuration which involves pinning the APT::Default-Release manually.

At least that's what the release notes say. In reality, in my case, I had the codename set in the unattended-upgrades configuration file, which required a change (in Puppet).

I can't find any other recent changes in my notes, but I could have sworn it's not the first time a change like this happens. If someone remember what it was before that, I would be grateful if I could update that list!

Issues

See also the official list of known issues.

Pending

Critical packages missing

In the "removed packages" list above, i have decided to keep the following, even if they don't make it to bullseye:

I also particularly need to pay attention to usbguard, as it's quite possible I won't be able to do anything after reboot. :p

Some other removed packages I have just accepted the removal, with the following alternatives:

Package Alternative Rationale
gocode gopls LSP is the (ad-hoc) standard
gtk-recordmydesktop obs, peek peek is a nice, simple alternative, OBS Studio can also be used for live streaming
nomacs geeqie, many others i mostly use geeqie all the time, and only recently found nomacs, might be interesting if it gets packaged properly
usbguard-applet-qt usbguard allow-device GUI just gone, but commandline might work

For syncmaildir, the plan is to drop it. See this blog post. For elpy, I'm likely to use LSP as well, since it seems it won't make it after all. For qalculate, I'm betting on backports.

Puppet breaks in bullseye/sid

testing has this ... peculiar notion of itself. instead of announcing itself like a normal Debian stable release, for example:

anarcat@angela:~(main)$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 10 (buster)
Release:    10
Codename:   buster

It is kind of unsure about its identity:

vagrant@testing:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux bullseye/sid
Release:    testing/unstable
Codename:   n/a

When you know how Debian works (that testing is really just an old, partial copy of unstable), that makes sense. But when you create Puppet manifests, you expect stuff like:

if $facts['os']['release']['major'] < 11 {
    # stuff before bullseye
} else {
    # stuff after bullseye
}

To just work. But they don't. In bullseye/sid/testing/unstable, however you want to call it, os.release.major is actually "bullseye/sid". Not "bullseye", not "sid", and, of course, not "11". "bullseye/sid". So obviously that just totally breaks when comparing to "11".

I tried patching /etc/os-release:

cat >> /etc/os-release <<EOF
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
EOF

But that doesn't seem to work: it looks like facter -p, at least, takes the major/minor information from... /etc/debian_version! So you actually need to do this to fix your manifests:

echo 11.0 > /etc/debian_version

But that's really... quite a hack. To workaround this from the Puppet side, I ended up doing this ugly kludge:

# remove packages gone from bullseye
#
# we should really use < 11 here, but os.release.major is
# actually "bullseye/sid" now? ouch?
#
# remove this when we stop supporting buster
$bullseye_removed = $facts['os']['distro']['codename'] ? {
  'bullseye/sid' => absent,
  'bullseye' => absent,
  default => present,
}
package { 'gtk-recordmydesktop':
  ensure => $bullseye_removed,
}

It's unclear to me here where the fault lies. On the one hand, it seems that Puppet shouldn't change the type of one of its core facts, but on the other, /etc/debian_version is bullseye/sid, a string and not a version, in testing/unstable in Debian... Garbage-in, garbage-out? Why don't we set a real version number there in Debian in the first place?

Resolved

Cool things I want to try

Update: I postponed those. I feel sway (and Wayland in general) is not ready for prime time just yet. For example it requires pipewire for screensharing, Emacs is not yet ported yet, etc. Kind of a mess.

Debian Planet has the #newinbullseye stuff (mikas blog mostly), I added what I found here already. There's also this wiki page and, of course, the release notes.

Browserpass fails to upgrade

Upgrade crashed on this:

dpkg: error processing archive /var/cache/apt/archives/webext-browserpass_3.7.2-1+b1_amd64.deb (--unpack):
 unable to open '/usr/share/mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/browserpass@maximbaz.com/icon.png.dpkg-new': No such file or directory
Reinstalling /etc/chromium/native-messaging-hosts/com.dannyvankooten.browserpass.json that was moved away
Errors were encountered while processing:
 /var/cache/apt/archives/webext-browserpass_3.7.2-1+b1_amd64.deb

This is bug #982758. Workaround:

apt purge webext-browserpass

If the upgrade crashed, purge the package with the same Dpkg options:

apt -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold' purge webext-browserpass

Once the upgrade is completed, just reinstall:

apt install webext-browserpass

The extension is quite finicky: i had to disable and re-enable it to get the button to show up on the browser interface.

Double redshift

I had two Redshift items in the notification area (and presumably processes too) running after a reboot and re-login. Not sure what's going on, it made the monitor "flicker slowly" as it flipped between the two configurations somehow.

It seems like the Debian package now ships a systemd unit file in /usr/lib/systemd/user/redshift-gtk.service, which takes care of the startup, so I disabled the hack in my .xsession file, nice.

Emacs took 2 minutes to start

That was because I still had company-go in my .emacs configuration, which meant it was trying to fetch it from MELPA, which took forever. I removed it and, anyways, it wouldn't have done it a second time so that's fixed.

i3-focus and rsendmail delivery failed

I have this custom i3-focus script to improve on the "alt-tab" behavior, which depends on a python library not in Debian. I have this virtualenv to deploy it, but somehow it failed after the upgrade. Doing this fixed it:

mv .virtualenvs/i3_py .virtualenvs/i3_py.bak &&
python3 -m venv --system-site-packages ~/.virtualenvs/i3_py &&
.virtualenvs/i3_py/bin/pip3 install i3_py &&
rm -rf .virtualenvs/i3_py.bak

This is presumably because Python libraries get installed in a version-specific directory...

Note that this also crashed rsendmail which I really need to get around packaging (which would fix this issue). It also meant it totally lost the mails, because postfix panicked and drop the mails when it couldn't generate a bounce either.

Not enough disk space

I have too much stuff on my computers. I was already a bit short on my / partition before the upgrade:

Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/curie--vg-root   28G   25G  2.8G  90% /

The upgrade downloaded ~7GB of Debian packages, and required an extra 4.5GB of disk space! Clearly that wouldn't do here, so I had to expand the root partition, which ended up like this after the upgrade:

Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/curie--vg-root   38G   25G   13G  67% /

I'm surprised that Debian bullseye now would use an extra 4GB of disk space! The disk requirements don't seem to have changed in decades, yet I keep having to pile up more disk space only to store software... We'll see what the end result will be.

Packages I have removed:

After the complete upgrade procedure (but before removing the extra kernel):

Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/curie--vg-root   38G   28G  9.1G  76% /

So the upgrade did use about 3-4GB of disk space, which is quite significant!

Maybe there's a way to figure out which package ate all that much?

Packages mistakenly removed

Those packages were removed during the upgrade, yet I still want to use them:

Workaround: apt install $PACKAGE

The package libu2f-host0 was also removed and, typically, I needed it to make U2F authentication work (2FA) in Firefox and Chrome, but it seems it's not necessary in bullseye anymore at least, so I've just removed it altogether.

LUKS password prompt in plain text instead of GUI

It seems like Plymouth just disappeared? At least on curie. Could not reproduce on angela, suspecting this was never correctly setup on curie.

Troubleshooting

Upgrade failures

Instructions on errors during upgrades can be found in the release notes troubleshooting section.

Reboot failures

If there's any trouble during reboots, you should use some recovery system. The release notes actually have good documentation on that, on top of "use a live filesystem".

Finding orphaned and weird packages

The apt-forktracer call above used to have many other different incantations, and it's not yet clear that it does everything we need. What we want to find are basically packages that are not "canonical Debian packages", which are shipped by the stable Debian distribution. Those are typically called "obsolete" packages in Debian, but that term is somewhat to narrow, as I also want to consider packages that were never part of Debian at all.

Weirdly, the release notes suggest three different methods to do this, in different part of the documentation. (Filed this as a bug in 987017.)

This section tries to figure out the right way forward. See also step 4.2.2, 4.8 and this forum.

aptitude search 1

This is the first way I found:

aptitude search '?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))'

This incantation comes from the cross-upgrade documentation. It selects packages that are currently installed (?narrow(...,?version(CURRENT))) from an archive other than "now" (?not(?archive("^[^n][^o][^w].*$")). This was cargo-culted from Ewan's cross-upgrading documentation.

Nowadays, the release notes actually suggest a similar pattern:

aptitude search '?narrow(?installed, ?not(?origin(Debian)))'

apt-show-versions

I also found this somewhat works to find weird packages:

apt-show-versions | grep -v /bullseye

This uses the more flexible apt-show-version to list everything that is not in the bullseye repository. But the regex could hide third-party repositories that happen to reuse that codename. It can also yield strange results like:

linux-libc-dev:i386 not installed

Those are presumably harmless, so this might be a better call:

apt-show-versions | grep -v /bullseye | grep -v 'not installed$'

... to filter out those packages.

aptitude 2: ~obsolete

Then the release notes also suggest this:

aptitude search '?obsolete'

This command has been recommended to find obsolete packages since buster.

apt-forktracer

This one is fairly new to the game, at least as far as I am concerned:

apt-forktracer | sort

This will not find packages that are from a newer version (for example from "testing" in "stable").

It's also recommended by the release notes. I've settled on it because its output is so much simpler, but I still need to compare the various results.

apt list

Starting from bullseye, ironically, we have another way of doing this, since APT adopted the aptitude patterns:

apt list '?obsolete'

It works well, and the output is digestible, but it will not catch versions on the local machine newer than in the archive, which might be a problem in some cases.

References

Created . Edited .