Trixie upgrade

1. services
2. upgrades

This document contains my upgrade procedure, notable changes in the new version, issues I have stumbled upon (and possibly fixed), and troubleshooting instructions.

It does not hope to replace the official documentation: it is a personal, living document that I have started keeping back when I upgraded to jessie. The other documents can be found in the parent upgrades page.

Procedure

This procedure is designed to be applied, in batch, on multiple servers. Do NOT follow this procedure unless you are familiar with the command line and the Debian upgrade process. It has been crafted by and for experienced system administrators that have dozens if not hundreds of servers to upgrade.

In particular, it runs almost completely unattended: configuration changes are not prompted during the upgrade, and just not applied at all, which will break services in many cases. I use a clean-conflicts script to do this all in one shot to shorten the upgrade process (without it, configuration file changes stop the upgrade at more or less random times). Then those changes get applied after a reboot. And yes, that's even more dangerous.

IMPORTANT: if you are doing this procedure over SSH (I had the privilege of having a console), you may want to upgrade SSH first, especially if you are on a flaky connection.

Make sure you read the conflicts resolution section below for how to handle clean_conflicts output.

This procedure may kill your graphical session, so make sure you can log back in over a serial console or virtual terminal.

1. Preparation:

   echo reset to the default locale &&
   export LC_ALL=C.UTF-8 &&
   echo install some dependencies &&
   sudo apt install ttyrec screen debconf-utils deborphan &&
   echo create ttyrec file with adequate permissions &&
   sudo touch /var/log/upgrade-trixie.ttyrec &&
   sudo chmod 600 /var/log/upgrade-trixie.ttyrec &&
   sudo ttyrec -a -e screen /var/log/upgrade-trixie.ttyrec
   

2. Backups and checks:

   ( 
     umask 0077 &&
     tar cfz /var/backups/pre-trixie-backup.tgz /etc /var/lib/dpkg /var/lib/apt/extended_states /var/cache/debconf $( [ -e /var/lib/aptitude/pkgstates ] && echo /var/lib/aptitude/pkgstates ) &&
     dpkg --get-selections "*" > /var/backups/dpkg-selections-pre-trixie.txt &&
     debconf-get-selections > /var/backups/debconf-selections-pre-trixie.txt
   ) &&
   ( puppet agent --test || true )&&
   apt-mark showhold &&
   dpkg --audit &&
   echo look for dkms packages and make sure they are relevant, if not, purge. &&
   ( dpkg -l '*dkms' || true ) &&
   echo look for leftover config files &&
   /home/anarcat/src/koumbit-scripts/vps/clean_conflicts &&
   echo run backups &&
   /home/anarcat/bin/backup-$(hostname) &&
   printf "End of Step 2\a\n"
   

3. Perform any pending upgrade and clear out old pins:

   puppet agent --disable "running major upgrade" &&
   apt update && apt -y upgrade &&
   echo Check for pinned, on hold, packages, and possibly disable &&
   rm -f /etc/apt/preferences /etc/apt/preferences.d/* &&
   rm -f /etc/apt/sources.list.d/backports.debian.org.list &&
   rm -f /etc/apt/sources.list.d/backports.list &&
   rm -f /etc/apt/sources.list.d/*-backports.list &&
   rm -f /etc/apt/sources.list.d/trixie.list &&
   rm -f /etc/apt/sources.list.d/bookworm.list &&
   rm -f /etc/apt/sources.list.d/bullseye.list &&
   rm -f /etc/apt/sources.list.d/experimental.list &&
   rm -f /etc/apt/sources.list.d/incoming.list &&
   rm -f /etc/apt/sources.list.d/proposed-updates.list &&
   rm -f /etc/apt/sources.list.d/sid.list &&
   rm -f /etc/apt/sources.list.d/testing.list &&
   echo purge removed packages &&
   apt purge $(dpkg -l | awk '/^rc/ { print $2 }') &&
   echo purge obsolete packages &&
   apt purge '?obsolete' &&
   echo autoremove packages &&
   apt autoremove -y --purge &&
   echo possibly clean up old kernels &&
   dpkg -l 'linux-image-*' &&
   echo look for packages from backports, other suites or archives &&
   echo if possible, switch to official packages by disabling third-party repositories &&
   apt list "?narrow(?installed, ?not(?codename($(lsb_release -c -s | tail -1))))" &&
   printf "End of Step 3\a\n"
   

4. Check free space (see this guide to free up space), disable auto-upgrades, and download packages:

   systemctl stop apt-daily.timer &&
   sed -i 's#bookworm-security#trixie-security#' $(ls /etc/apt/sources.list /etc/apt/sources.list.d/*) &&
   sed -i 's/bookworm/trixie/g' $(ls /etc/apt/sources.list /etc/apt/sources.list.d/*) &&
   apt update &&
   apt -y -d full-upgrade &&
   apt -y -d upgrade &&
   apt -y -d dist-upgrade &&
   df -h &&
   echo make sure host is silenced in monitoring &&
   printf "End of Step 4\a\n"
   

5. Actual upgrade step. Put server in maintenance here.

   Optional, minimal upgrade run (avoids new installs or removals):

   sudo touch /etc/nologin &&
   env DEBIAN_FRONTEND=noninteractive APT_LISTCHANGES_FRONTEND=none APT_LISTBUGS_FRONTEND=none UCF_FORCE_CONFFOLD=y \
       apt upgrade --without-new-pkgs -y -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold' &&
   

   Full upgrade:

   sudo touch /etc/nologin &&
   env DEBIAN_FRONTEND=noninteractive APT_LISTCHANGES_FRONTEND=none APT_LISTBUGS_FRONTEND=none UCF_FORCE_CONFFOLD=y \
       apt full-upgrade -y -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold' &&
   printf "End of Step 5\a\n"
   

6. Post-upgrade procedures:

   apt-get update --allow-releaseinfo-change &&
   puppet agent --enable &&
   puppet agent -t --noop &&
   printf "Press enter to continue, Ctrl-C to abort." &&
   read -r _ &&
   (puppet agent -t || true) &&
   echo deploy upgrades after possible Puppet sources.list changes &&
   apt update && apt upgrade -y &&
   echo rm -f /etc/apt/apt.conf.d/50unattended-upgrades.dpkg-dist /etc/ca-certificates.conf.dpkg-old /etc/cron.daily/bsdmainutils.dpkg-remove /etc/default/prometheus-apache-exporter.dpkg-dist /etc/default/prometheus-node-exporter.dpkg-dist /etc/logrotate.d/apache2.dpkg-dist /etc/nagios/nrpe.cfg.dpkg-dist /etc/ssh/ssh_config.dpkg-dist /etc/ssh/sshd_config.ucf-dist /etc/unbound/unbound.conf.dpkg-dist &&
   printf "\a" &&
   /home/anarcat/src/koumbit-scripts/vps/clean_conflicts &&
   systemctl start apt-daily.timer &&
   rm /etc/nologin &&
   printf "End of Step 6\a\n" &&
   shutdown -r +1 "major upgrade step 6: removing old kernel image"
   

7. Post-upgrade cleanup:

   export LC_ALL=C.UTF-8 &&
   sudo ttyrec -a -e screen /var/log/upgrade-trixie.ttyrec
   
   echo consider apt-mark minimize-manual
   
   apt-mark manual bind9-dnsutils &&
   apt purge apt-forktracer &&
   echo purging removed packages &&
   apt purge $(dpkg -l | awk '/^rc/ { print $2 }') &&
   apt autopurge &&
   apt purge $(deborphan --guess-dummy) &&
   while deborphan -n | grep -q . ; do apt purge $(deborphan -n); done &&
   apt autopurge &&
   echo review obsolete and odd packages &&
   apt purge '?obsolete' && apt autopurge &&
   apt list "?narrow(?installed, ?not(?codename($(lsb_release -c -s | tail -1))))" &&
   apt clean &&
   echo review installed kernels: &&
   dpkg -l 'linux-image*' | less &&
   printf "End of Step 8\a\n" &&
   shutdown -r +1 "last major upgrade step: testing reboots one final time"
   

Conflicts resolution

When the clean_conflicts script gets run, it asks you to check each configuration file that was modified locally but that the Debian package upgrade wants to overwrite. You need to make a decision on each file. This section aims to provide guidance on how to handle those prompts.

Those config files should be manually checked on each host:

     /etc/default/grub.dpkg-dist
     /etc/initramfs-tools/initramfs.conf.dpkg-dist

If other files come up, they should be added in the above decision list, or in an operation in step 2 or 7 of the above procedure, before the clean_conflicts call.

Files that should be updated in Puppet are mentioned in the Issues section below as well.

Notable changes

Here are some packages with notable version changes that I noticed.

TODO: merge or point at blog/2024-08-15-why-trixie

See also the wiki page about trixie for another list.

New packages

This is a curated list of packages that were introduced in trixie. There are actually thousands of new packages in the new Debian release, but this is a small selection of projects I found particularly interesting:

TODO

My packages

In packages I maintain, those are the important changes:

TODO

Updated packages

This table summarizes package version changes I find interesting.

Package	Bookworm	Trixie	Notes
TODO

Note that this table may not be up to date with the current release. See the official release notes for a more up to date list.

Removed packages

TODO

See also the noteworthy obsolete packages list.

Other improvements

• pinentry-qt now has Wayland support
• Signal Desktop seems to work properly in Wayland

Issues

See also the official list of known issues.

Pending

Removed packages

• nomacs (needs an update to latest upstream)
• raysession (broken by a FTBFS in a dependency)

Webcam sharing stopped working in Firefox

Webcam isn't detected properly in Firefox. It works in guvcview and chrome, so it's not an issue with the webcam per se, seems specific to Firefox.

Screen sharing still works.

Resolved

Fluffychat fails to start

After the upgrade, Fluffychat completely failed to start with an error like:

flutter: [Matrix] !!!CRITICAL!!! Unable to construct database! - SqfliteFfiException(sqlite_error: 26, , SqliteException(26): while selecting from statement, file is not a database, file is not a database (code 26)

It was quite hard to parse in the UI, and after restarting, the error message went away, but it was as if the configuration was entirely reset. Looking at my logs, it seems the problem is with the keyring support:

fluffychat[228136]: libsecret_error: Failed to unlock the keyring
flutter: [Matrix] Unable to init database encryption - PlatformException(Libsecret error, Failed to unlock the keyring, null, null)

I switched to gomuks, but they are rewriting it in web which, ugh. So I also tested ement.el, an Emacs client, which works well: OIDC and image support, but it's lacking E2EE.

This might be unrelated to the upgrade, and more likely because I started using keyring_pass.

Thankfully, just logging in again and entering the recovery key mostly brought back Fluffychat in good working order.

gpg-agent / SSH failure

After a reboot, GCR (GNOME Keyring?) took over the SSH_AUTH_SOCK environment (set to /run/user/1000/gcr/ssh instead of /run/user/1000/gnupg/S.gpg-agent.ssh). That's somewhat easily worked around by hardcoding the environment in a shell, but even without that, SSH still fails, as scdaemon fails to talk with the YubiKey:

scdaemon[16500]: pcsc_connect failed: sharing violation (0x8010000b)

Restarting gpg-agent.service and disconnecting the YubiKey fixed the latter.

Purging gcr4 fixed the former.

In-upgrade issues

During the upgrade, magit started misbehaving, gpg-agent wouldn't authenticate to remove SSH servers or decrypt emails either. The problem is the pinentry program:

Cannot mix incompatible Qt library (5.15.15) with this library (5.15.8)

... and bash completion would fail with:

_comp_initialize: command not found

All of those issues fixed themselves after the upgrade was completed.

Out of space in /boot

The /boot has, again, become too small, which is getting a bit ridiculous. I had to jump through a couple hoops to make the upgrade complete:

1. MODULES=dep in /etc/initramfs-tools/initramfs.conf
2. cleaning up old kernels preemptively
3. regenerate the current initramfs with update-initramfs -u
4. rerun apt upgrade

After the upgrade, with two initramfs compressed with zstd and MODULES=dep, I get:

root@angela:/var/tmp# df -h  /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  456M  264M  167M  62% /boot

... which is a bit bizarre because du -schx /boot reports 88M used. After a reboot, however, free space returns properly:

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  456M   88M  344M  21% /boot

Packages not upgradable

The packages libboost-dev and source-extractor were marked as "not upgrading". Both packages were not directly used and just purged.

Pinentry reverted to GNOME

After a reboot, for some reason pinentry switched from pinentry-qt to pinentry-gnome3 which works but that I find kind of ugly. A simple fix is to:

update-alternatives --config pinentry

Removing pinentry-gnome3 also fixes this.

Puppet catalog fails

Puppet in trixie fails with:

Error while evaluating a Function Call, Unsupported OS family:  (file: /etc/puppet/code/production/modules/augeas/manifests/params.pp, line: 46, column: 17) on node angela.anarc.at

There were lots of issues like this, as Puppet 8 deprecated a ton of things. Adding include_legacy_facts=true to puppet.conf on the agent helped a lot, but ultimately I just upgraded all the modules that were failing, one at a time.

Removed packages

• gnuradio (reinstalled)
• python3-yubikey-manager (now called python3-ykman, normal transition)
• tlp (replaced with power-profiles-daemon during the upgrade, restored by Puppet)

Troubleshooting

Upgrade failures

Instructions on errors during upgrades can be found in the release notes "possible issues" section.

Reboot failures

If there's any trouble during reboots, you should use some recovery system. The release notes actually have good documentation on that, on top of "use a live filesystem".

References

• Official guide
• Release notes
• Koumbit guide (N/A, last checked 2024-11-27
• DSA guide (N/A, last checked 2024-11-27)
• TPA guide (N/A, last checked 2024-11-27)
• Solution proposal to automate this

TODO: replace releases/testing with releases/trixie after the release notes are published.