Buster upgrade

  1. services
  2. upgrades
  1. Procedure
  2. Notable changes
  3. Issues
  4. References

Yet again, this time has come. I usually upgrade at least one machine to testing during its development cycle, and now I need to do this on my workstation to get a working npm package which was removed from testing/stretch, because it was outdated, in 2017. Maintenance of the Debian package has now resumed and a fresh new version landed in unstable in August 2018 which looks promising, so here we go.

Procedure

  1. Preperation:

    : reset to the default locale
export LC_ALL=C.UTF-8 &&
sudo apt install ttyrec screen debconf-utils apt-show-versions deborphan apt-forktracer &&
sudo ttyrec -e screen /var/log/upgrade-buster.ttyrec

  2. Backups and checks:

    ( umask 0077 &&
  tar cfz /var/backups/pre-buster-backup.tgz /etc /var/lib/dpkg /var/lib/apt/extended_states /var/lib/aptitude/pkgstates /var/cache/debconf &&
  dpkg --get-selections "*" > /var/backups/dpkg-selections-pre-buster.txt &&
  debconf-get-selections > /var/backups/debconf-selections-pre-buster.txt
) &&
apt-mark showhold &&
dpkg --audit &&
: look for dkms packages and make sure they are relevant, if not, purge. &&
dpkg -l '*dkms' || true &&
/home/anarcat/bin/backup-$(hostname) &&
echo End of Step 2

  3. Perform any pending upgrade and clear out old pins:

    : Check for pinned, on hold, packages, and possibly disable &&
rm -f /etc/apt/preferences /etc/apt/preferences.d/* &&
rm -f /etc/apt/sources.list.d/testing.list &&
rm -f /etc/apt/sources.list.d/stretch-backports.list &&
rm -f /etc/apt/sources.list.d/backports.debian.org.list &&
apt update && apt -y upgrade &&
: list kernel images and purge unused packages &&
dpkg -l 'linux-image-*' &&
: look for packages from backports, other suites or archives &&
: if possible, switch to official packages by disabling third-party repositories &&
apt-forktracer &&
echo End of Step 3

  4. Check free space, see this guide to free up space and download packages:

    sed -i 's/stretch/buster/g' /etc/apt/sources.list $(ls /etc/apt/sources.list.d/*) &&
apt update && apt -o APT::Get::Trivial-Only=true dist-upgrade && df -h &&
apt -y -d upgrade && apt -y -d dist-upgrade &&
echo End of Step 4

  5. Actual upgrade run:

    export DEBIAN_FRONTEND=noninteractive APT_LISTCHANGES_FRONTEND=mail &&
apt dist-upgrade -y -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold' &&
/opt/bin/clean_conflicts &&
echo 'End step 6'

  6. Post-upgrade checks:

    dpkg -l '*-dbg' # look for dbg package and possible replace with -dbgsym
aptitude purge ~c # purge removed packages
apt autoremove -y --purge
while deborphan -n | grep -q . ; do apt purge $(deborphan -n); done
apt autoremove -y --purge
apt clean
reboot
# review and purge older kernel once the new one boots properly
dpkg -l 'linux-image*'
# review packages that are not in the new distribution
aptitude search '?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))'

The last incantation comes from the cross-upgrade documentation. It selects packages that are currently installed (?narrow(...,?version(CURRENT))) from an archive other than "now" (?not(?archive("^[^n][^o][^w].*$")). This was cargo-culted from Ewan's cross-upgrading documentation. Another way to do this is with apt-show-version:

apt-show-versions | grep -v /buster

... although that yields strange results like:

linux-libc-dev:i386 not installed

Those are presumably harmless, so this might be a better call:

apt-show-versions | grep -v /buster | grep -v 'not installed$'

Update: the first incantation was updated to use apt-forktracer instead.

Notable changes

Here are some packages with notable version changes that I noticed.

Note that this table is not up to date with the current buster release. See the official release notes for a more up to date list.

Package Stretch Buster Notes
Docker N/A 18 Finally, Docker is back in Debian?
Emacs 24.5 26
Firefox 52 60 Major upgrade, the "Quantum" release
GNOME 3.22 3.30
Libreoffice 5.2 6.2

Many packages were removed from Buster. I've built an exhaustive list on May 16th 2019, but it's probably changed since then. See also the noteworthy obsolete packages list.

Issues

See also the official list of known issues.

Pending

Spurious prompts

I actually tried the upgrade without DEBIAN_FRONTEND=noninteractive and got prompted for a bunch of irrelevant stuff like xastir, libc6 and others. I also got prompted by ucf even if I told dpkg to --force-confold, which I would have expected to trickle down.

Missing packages

Those are packages that are only available in sid or stretch:

Font rendering changes

Font rendering changed again:

screenshot of the rendering change before and after upgrade

The difference is subtle, but annoying enough. The NEWS file for the fontconfig package said it changed font hinting defaults in 2.12 to hintslight, but changing it to Full (or None) does not fix the problem. I've tried all sorts of configurations:

screenshot of many fontconfig configurations

We (thanks @jelly!) identified that subpixel rendering is not the issue: auto/full/never and auto/full/always are identical:

comparison of subpixel rendering

Turns out the problem is due to an upstream change documented in bug #866685. Basically, the way subpixel rendering works has changed in the new version, as it implements a "v40" TrueType rendered, as explained here. A workaround is to set the following environment:

FREETYPE_PROPERTIES=truetype:interpreter-version=35

To get the same pixel-level rendering as the original stretch version, I also had to pick the settings "Native, Full, Always, No". But after a few experiments, I found that the sub-pixel rendering did have an effect when the workaround was in place, so I picked "Native, Full, Auto, No" instead, which gives a more compact font. I have commented on the bug report to share this and hopefully this will be fixed before the buster release somehow...

Update: this was not fixed before the buster release, which shipped with the new interpreter. I have therefore followed suit and changed tack. Instead of battling the new rendering changes, I switched to a nicer font, Fira mono. See 2020-03-10-font-changes for details.

APT permission warnings

apt gives me a cryptic warning in the end that are probably harmless, but happens on every run:

W: Download is performed unsandboxed as root as file '/var/cache/apt/archives/partial/libpython3.6-minimal_3.6.6-1_amd64.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission non accordée)

SSH configuration changes

There was an issue with my SSH configuration:

Bad SSH2 Mac spec 'hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com,hmac-sha1-96,hmac-sha1'.

I commented out the line, but it would be nice if SSH actually said which item in there was the wrong one, instead of having me bisect the damn list.

mutt binary name change

mutt is now the official upstream version of mutt, so all settings specific to neomutt or the patched version traditionnally shipped with Debian yield warnings:

Erreur dans /home/anarcat/.mutt/options, ligne 63 : news_cache_dir : variable inconnue
Erreur dans /home/anarcat/.mutt/options, ligne 66 : news_server : variable inconnue
Erreur dans /home/anarcat/.mutt/options, ligne 75 : catchup_newsgroup : variable inconnue
Erreur dans /home/anarcat/.mutt/options, ligne 76 : nntp_context : variable inconnue
Erreur dans /home/anarcat/.mutt/options, ligne 77 : nntp_load_description : variable inconnue
Erreur dans /home/anarcat/.mutt/options, ligne 78 : nntp_poll : variable inconnue
Erreur dans /home/anarcat/.mutt/options, ligne 79 : post_moderated : variable inconnue
Erreur dans /home/anarcat/.mutt/options, ligne 80 : followup_to_poster : variable inconnue
Erreur dans /home/anarcat/.mutt/options, ligne 82 : change-newsgroup : cette fonction n'existe pas dans la table
Erreur dans /home/anarcat/.muttrc, ligne 3 : source : erreurs dans /home/anarcat/.mutt/options
source : erreurs dans /home/anarcat/.muttrc

Installing neomutt and removing mutt only solves part of this problem, as some programs just expect the mutt command to be available, for example bts. Setting the BTS_MAIL_READER environment to neomutt -f %s fixes that issue, but the mailscripts tools are also a good candidate for replacing bts show --mbox ### in my workflow, as they can inject bug reports straight into my normal notmuch workflow...

pulseaudio diversion issue

I had a weird problem with pulseaudio not starting. it would fail with:

$ pulseaudio 
E: [pulseaudio] module-alsa-card.c: Failed to find a working profile.
E: [pulseaudio] module.c: Failed to load module "module-alsa-card" (argument: "device_id="1" name="pci-0000_00_1f.3" card_name="alsa_card.pci-0000_00_1f.3" namereg_fail=false tsched=yes fixed_latency_range=no ignore_dB=no deferred_volume=yes use_ucm=yes card_properties="module-udev-detect.discovered=1""): initialization failed.
E: [pulseaudio] backend-ofono.c: Failed to register as a handsfree audio agent with ofono: org.freedesktop.DBus.Error.ServiceUnknown: The name org.ofono was not provided by any .service files

No audio would work, naturally. After running with -v, I found the following error:

I: [pulseaudio] (alsa-lib)conf.c: cannot access file /etc/alsa/conf.d/50-pulseaudio.conf

As it turns out, it seems I had hacked PA to not automatically start for all users or some weird thing, as documented in the Debian wiki. This would show up as a diversion when checkout out the target of the above symlink:

$ dpkg -S usr/share/alsa/alsa.conf.d/50-pulseaudio.conf
local diversion from: /usr/share/alsa/alsa.conf.d/50-pulseaudio.conf
local diversion to: /usr/share/alsa/alsa.conf.pulse/50-pulseaudio.conf
libasound2-plugins:amd64: /usr/share/alsa/alsa.conf.d/50-pulseaudio.conf

The fix was to remove those diversions and reinstall the package:

dpkg-divert --remove /usr/share/alsa/alsa.conf.d/pulse.conf
dpkg-divert --remove /usr/share/alsa/alsa.conf.d/99-pulseaudio-default.conf.example
dpkg-divert --remove /usr/share/alsa/alsa.conf.d/50-pulseaudio.conf
apt install --reinstall libasound2-plugins

Electrum

Electrum was removed from Debian because of a serious security issue. I don't have anything in my wallet anymore, but it's possible that people send money there, unfortunately. I should probably figure a way out of there.

Touchpad / trackpoint freeze after sleep

On a Thinkpad E431, the entire mouse interface (touch, trackpoint) freezes after sleep. Keyboard still works but not mouse until a reboot.

There's bug 922666 in Debian buster, without a fix. It also says it eventually recovers, which is not our experience. Possible dupe is bug 928189.

There's also bug 1791427 in Ubuntu 18.04 that seems related, and which proposes the following workarounds:

There's also bug 1442699 in Fedora, which suggests those workarounds:

Also related is this libinput bug that's closed as "not our bug" because they claim it's a bug in the kernel.

There are two patches on the Linux kernel which apparently fix the issue, still pending approval:

Possibly related: https://lkml.org/lkml/2016/8/18/134

5.1rc7 shipped two fixes against the synaptics-rmi4 module. A pull request has been merged in mainline with two other fixes on the module./ 5.0.11 also has fixes on the module. It's clearly a regression from Debian stretch (kernel 4.9) since it was working fine before.

Possibly related, two-finger scrolling bug in Ubuntu, which identifies this commit as the source of the regression. Upstream kernel bug, still open.

Antidote 9 failures

Antidote is a powerful spell-and-grammar-checking that's popular in the french-speaking community because it's incredibly powerful, probably unrivaled in free software.

The upgrade from stretch to buster broke it completely: it wouldn't start at all and the buttons wouldn't work in Libreoffice. Upgrading from the 9v5.1 to the 9v5.3 Antidote release fixed its startup: now the standalone version runs, but not the Libreoffice plugin, even after reinstalling that.

Of course, that's proprietary software which I don't support out of the box, but I gotta help our users... We've opened a ticket upstream, and they said that Debian is unsupported and they suggest upgrading to Antidote 10 (which costs money). They also explicitely say they don't provide refunds if the upgrade doesn't fail.

They also provide steps to debug the problem and a way to give them more information. So we'll see how that flies. The way that works is they make you download a file called DruideInfos.bash.tar.gz and, yes, that's a bash script in a tar archive. They tell you to "uncompress it on the desktop and run it", presumably a user-friendly version of curl | bash, although my friend had no idea how to actually do that. Not sure I like the idea of telling people to run a 50KB bash script, but at a cursory glance, the script looked more or less okay. Even though it does send a lot of private information to Druide, they warn about that profusely and it's in the context of the user asking for help, so it kind of makes sense. Still, there are a few things wrong with the way they're doing this:

  1. the link is HTTP, in plain text

  2. it's a huge bash script, for crying out loud, can't antidote ship a self-diagnostic tool in itself instead of telling people to download random stuff over the internet?

  3. minor: you don't need tar to compress a single file, gzip suffices

  4. probably minor: the username of the staff that created the script leaks in the gzip archive

  5. the insides of the scripts are of course absolutely horrendous. just to give an example, they write a JSON parser in an inline Python script, written with variables in french, for good measure (spotted by sdk)

Now we're waiting for a response from their tech support.

No sleep

The user noticed the laptop doesn't go to sleep when the lid is closed. Technically, the bug report is "big power draw even if lid is closed", which I concluded was related to suspend not working in some situation. I suspect the problem might simply be that suspend doesn't trigger if the lid is closed before the power cable is unplugged.

This might or might not be a regression, unclear.

Disk problems (unrelated to upgrade)

Unrelated to the upgrade, but I have to write this down somewhere: that laptop is also showing alarming disk errors when booting, since the HDD was replaced with a Samsung 860 SSD.

When chatting with Samsung support, they suggested checking if the disk is well-seated, which we still have to try.

Resolved

msmtp

Still had msmtp installed but I changed the mail setup to use rsendmail instead so I uninstalled it.

stray files

Had a few warnings like this, pretty harmless, but can easily be missed and leave stray files around:

dpkg: avertissement: impossible de supprimer l'ancien répertoire  « /etc/cryptsetup-initramfs » : Le dossier n'est pas vide

Those folders were:

dictd upgrade failure

dictd, dict-devil and dict-gcide packages failed to upgrade:

Paramétrage de dictd (1.12.1+dfsg-7) ...
update-inetd: cannot open /etc/inetd.conf: No such file or directory
dpkg: erreur de traitement du paquet dictd (--configure) :
 installed dictd package post-installation script subprocess returned error exit status 2

That bug #906420 was fixed in 1.12.1+dfsg-8.

missing space in /boot

initramfs failed because I had too many kernels installed, cleaning up obsolete kernels fixed this and I added it to the preparation procedure above.

unattended upgrade config patch

unattended-upgrades finally implemented a local change I had where I enabled "stable" updates, so I installed the version from upstream directly

aufs-dkms warnings

also seen some warnings with aufs-dkms:

Paramétrage de aufs-dkms (4.17+20180709-1) ...
Loading new aufs-4.17+20180709 DKMS files...
Building for 4.9.0-8-amd64 4.17.0-1-amd64
Building initial module for 4.9.0-8-amd64
Error!  The dkms.conf for this module includes a BUILD_EXCLUSIVE directive which
does not match this kernel/arch.  This indicates that it should not be built.
Skipped.
Building initial module for 4.17.0-1-amd64
Done.

aufs.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/4.17.0-1-amd64/updates/dkms/

depmod...

DKMS: install completed.

Probably harmless considering the latest kernel build worked.

snap startup failure

snap packages completely fail to start:

[1002]anarcat@curie:~$ signal-desktop 
snap-confine has elevated permissions and is not confined but should be. Refusing to continue to avoid permission escalation attacks

It's unclear what happened there. The workaround is to remove the snapd security keys (!?):

sudo rm /var/lib/snapd/system-key

I ended up switching to flatpak completely.

python3-sh errrors

python3-sh is throwing crazy weird errors which broke my LWN workflow. i had to revert to manually saving the page with control-s (which doesn't work, oddly) or copy-pasting the source in a file (which does, also oddly) and calling pandoc by hand with:

pandoc -f html -t markdown-simple_tables --filter lwn-clean.hs < ~/Downloads/dat.html > blog/dat.mdwn

I filed this as #907370 but I suspect this might be an upstream problem. it could also be related to the Python 3.7 migration (#902788) although that hasn't trickled down onto buster for obvious reasons. (fixed in NMU 1.12.14-1.1, pending DELAYED/10)

linux-headers errors

Some error happened during the linux-headers upgrade which I still need to parse:

Paramétrage de linux-headers-4.17.0-1-amd64 (4.17.8-1) ...
/etc/kernel/header_postinst.d/dkms:
Error! Bad return status for module build on kernel: 4.17.0-1-amd64 (x86_64)
Consult /var/lib/dkms/wireguard/0.0.20171017-1/build/make.log for more information.

Turns out I had installed wireguard from unstable a while ago, and it's still not in buster. So I made this special pin to get it from unstable until it's ready in buster:

Package: wireguard*
Pin: release n=sid
Pin-Priority: 501

Package: wireguard*
Pin: release n=buster
Pin-Priority: 502

Firefox upgrade

Firefox is a major upgrade, which I had already performed. In fact, it is the reason I switched back from Chromium, so the work was already done (see firefox for details). The new version is not yet in Buster at the time of writing, but will be by the time it is released, so a pin like this was necessary to install from sid while still allowing versions from buster to override when ready:

Package: firefox-esr
Pin: release n=sid
Pin-Priority: 501

Package: firefox-esr
Pin: release n=buster
Pin: release v=60
Pin-Priority: 502

One downside is that no extensions are currently packaged so we are trusting the Mozilla "addons" site for those. Update: this is now fixed as ESR landed in buster and the above pin can be removed.

U2F failure

I mistakenly removed the libu2f-udev library during the upgrade, which broke U2F in Firefox. Reinstalling the package and restarting udev fixed the issue:

sudo udevadm trigger

References

Created . Edited .