1. Debian considering automated upgrades
    1. Problems with automated upgrades
    2. Benefits of automated upgrades
    3. Possible non-issues
    4. Automating major upgrades?
    5. Future work

The Debian project is looking at possibly making automatic minor upgrades to installed packages the default for newly installed systems. While Debian has a reliable and stable package update system that has been an inspiration for multiple operating systems (the venerable APT), upgrades are, usually, a manual process on Debian for most users.

The proposal was brought up during the Debian Cloud sprint in November by longtime Debian Developer Steve McIntyre. The rationale was to make sure that users installing Debian in the cloud have a "secure" experience by default, by installing and configuring the unattended-upgrades package within the images. The unattended-upgrades package contains a Python program that automatically performs any pending upgrade and is designed to run unattended. It is roughly the equivalent of doing apt-get update; apt-get upgrade in a cron job, but has special code to handle error conditions, warn about reboots, and selectively upgrade packages. The package was originally written for Ubuntu by Michael Vogt, a longtime Debian developer and Canonical employee.

Since there was a concern that Debian cloud images would be different from normal Debian installs, McIntyre suggested installing unattended-upgrades by default on all Debian installs, so that people have a consistent experience inside and outside of the cloud. The discussion that followed was interesting as it brought up key issues one would have when deploying automated upgrade tools, outlining both the benefits and downsides to such systems.

Problems with automated upgrades

An issue raised in the following discussion is that automated upgrades may create unscheduled downtime for critical services. For example, certain sites may not be willing to tolerate a master MySQL server rebooting in conditions not controlled by the administrators. The consensus seems to be that experienced administrators will be able to solve this issue on their own, or are already doing so.

For example, Noah Meyerhans, a Debian developer, argued that "any reasonably well managed production host is going to be driven by some kind of configuration management system" where competent administrators can override the defaults. Debian, for example, provides the policy-rc.d mechanism to disable service restarts on certain packages out of the box. unattended-upgrades also features a way to disable upgrades on specific packages that administrators would consider too sensitive to restart automatically and will want to schedule during maintenance windows.

Reboots were another issue discussed: how and when to deploy kernel upgrades? Automating kernel upgrades may mean data loss if the reboot happens during a critical operation. On Debian systems, the kernel upgrade mechanisms already provide a /var/run/reboot-required flag file that tools can monitor to notify users of the required reboot. For example, some desktop environments will popup a warning prompting users to reboot when the file exists. Debian doesn't currently feature an equivalent warning for command-line operation: Vogt suggested that the warning could be shown along with the usual /etc/motd announcement.

The ideal solution here, of course, is reboot-less kernel upgrades, which is also known as "live patching" the kernel. Unfortunately, this area is still in development in the kernel (as was previously discussed here). Canonical deployed the feature for the Ubuntu 16.04 LTS release, but Debian doesn't yet have such capability, since it requires extra infrastructure among other issues.

Furthermore, system reboots are only one part of the problem. Currently, upgrading packages only replaces the code and restarts the primary service shipped with a given package. On library upgrades, however, dependent services may not necessarily notice and will keep running with older, possibly vulnerable, libraries. While libc6, in Debian, has special code to restart dependent services, other libraries like libssl do not notify dependent services that they need to restart to benefit from potentially critical security fixes.

One solution to this is the needrestart package which inspects all running processes and restarts services as necessary. It also covers interpreted code, specifically Ruby, Python, and Perl. In my experience, however, it can take up to a minute to inspect all processes, which degrades the interactivity of the usually satisfying apt-get install process. Nevertheless, it seems like needrestart is a key component of a properly deployed automated upgrade system.

Benefits of automated upgrades

One thing that was less discussed is the actual benefit of automating upgrades. It is merely described as "secure by default" by McIntyre in the proposal, but no one actually expanded on this much. For me, however, it is now obvious that any out-of-date system will be systematically attacked by automated probes and may be taken over to the detriment of the whole internet community, as we are seeing with Internet of Things devices. As Debian Developer Lars Wirzenius said:

The ecosystem-wide security benefits of having Debian systems keep up to date with security updates by default overweigh any inconvenience of having to tweak system configuration on hosts where the automatic updates are problematic.

One could compare automated upgrades with backups: if they are not automated, they do not exist and you will run into trouble without them. (Wirzenius, coincidentally, also works on the Obnam backup software.)

Another benefit that may be less obvious is the acceleration of the feedback loop between developers and users: developers like to know quickly when an update creates a regression. Automation does create the risk of a bad update affecting more users, but this issue is already present, to a lesser extent, with manual updates. And the same solution applies: have a staging area for security upgrades, the same way updates to Debian stable are first proposed before shipping a point release. This doesn't have to be limited to stable security updates either: more adventurous users could follow rolling distributions like Debian testing or unstable with unattended upgrades as well, with all the risks and benefits that implies.

Possible non-issues

That there was not a backlash against the proposal surprised me: I expected the privacy-sensitive Debian community to react negatively to another "phone home" system as it did with the Django proposal. This, however, is different than a phone home system: it merely leaks package lists and one has to leak that information to get the updated packages. Furthermore, privacy-sensitive administrators can use APT over Tor to fetch packages. In addition, the diversity of the mirror infrastructure makes it difficult for a single entity to profile users.

Automated upgrades do imply a culture change, however: administrators approve changes only a posteriori as opposed to deliberately deciding to upgrade parts they chose. I remember a time when I had to maintain proprietary operating systems and was reluctant to enable automated upgrades: such changes could mean degraded functionality or additional spyware. However, this is the free-software world and upgrades generally come with bug fixes and new features, not additional restrictions.

Automating major upgrades?

While automating minor upgrades is one part of the solution to the problem of security maintenance, the other is how to deal with major upgrades. Once a release becomes unsupported, security issues may come up and affect older software. While Debian LTS extends releases lifetimes significantly, it merely delays the inevitable major upgrades. In the grand scheme of things, the lifetimes of Linux systems (Debian: 3-5 years, Ubuntu: 1-5 years) versus other operating systems (Solaris: 10-15 years, Windows: 10+ years) is fairly short, which makes major upgrades especially critical.

While major upgrades are not currently automated in Debian, they are usually pretty simple: edit sources.list then:

    # apt-get update && apt-get dist-upgrade

But the actual upgrade process is really much more complex. If you run into problems with the above commands, you will quickly learn that you should have followed the release notes, a whopping 20,000-word, ten-section document that outlines all the gory details of the release. This is a real issue for large deployments and for users unfamiliar with the command line.

The solutions most administrators seem to use right now is to roll their own automated upgrade process. For example, the Debian.org system administrators have their own process for the "jessie" (8.0) upgrade. I have also written a specification of how major upgrades could be automated that attempts to take into account the wide variety of corner cases that occur during major upgrades, but it is currently at the design stage. Therefore, this problem space is generally unaddressed in Debian: Ubuntu does have a do-release-upgrade command but it is Ubuntu-specific and would need significant changes in order to work in Debian.

Future work

Ubuntu currently defaults to "no automation" but, on install, invites users to enable unattended-upgrades or Landscape, a proprietary system-management service from Canonical. According to Vogt, the company supports both projects equally as they differ in scope: unattended-upgrades just upgrades packages while Landscape aims at maintaining thousands of machines and handles user management, release upgrades, statistics, and aggregation.

It appears that Debian will enable unattended-upgrades on the images built for the cloud by default. For regular installs, the consensus that has emerged points at the Debian installer prompting users to ask if they want to disable the feature as well. One reason why this was not enabled before is that unattended-upgrades had serious bugs in the past that made it less attractive. For example, it would simply fail to follow security updates, a major bug that was fortunately promptly fixed by the maintainer.

In any case, it is important to distribute security and major upgrades on Debian machines in a timely manner. In my long experience in professionally administering Unix server farms, I have found the upgrade work to be a critical but time-consuming part of my work. During that time, I successfully deployed an automated upgrade system all the way back to Debian woody, using the simpler cron-apt. This approach is, unfortunately, a little brittle and non-standard; it doesn't address the need of automating major upgrades, for which I had to revert to tools like cluster-ssh or more specialized configuration management tools like Puppet. I therefore encourage any effort towards improving that process for the whole community.

More information about the configuration of unattended-upgrades can be found in the Ubuntu documentation or the Debian wiki.

Note: this article first appeared in the Linux Weekly News.

comment 1
Thanks! There is a small problem with the "Django proposal" link, which lacks a https://lwn.net :). You may delete this comment after it has been fixed ;-)
Comment by rohieb
great article!

Awesome article!

Thanks for sharing!

Just ask on install defaulting to yes. That should make everybody happy :)

Comment by aL
Good idea

I think it's a good idea to enable automatic upgrades by default. I do that allready myself. But sometimes I see there is a problem, it would be good to have a way to be noticed.

I would like a script what would do "apt-get dist-upgrade", otherwise not everything is updated. I do not mean upgrades from Debian 8 to Debian 9, but I still mean upgrades from Debian 8.1 to Debian 8.2, and from everything what's enabled in sources.list. Now I use "*" in the configuration.

Comment by Paul van der Vlis
Death by 1000 cuts?

I moved away from Ubuntu as I felt they were getting into bed with the devil (i.e. Microsoft). Now Automated "security" updates!!!!????

Any Admin worth their salt (and wishing to maintain an "employed" status HAS to comply with a server update regime, so some arguments fall at the first hurdle.

Personally I would not complain about an "Opt in" option during the installation process, but otherwise NO! I don't want to go Distribution dumping again :-(

Comment by bill
where to begin...

I moved away from Ubuntu as I felt they were getting into bed with the devil (i.e. Microsoft).

Ubuntu got in bed with Microsoft? I must have missed that meeting. There are plenty of wrong things with Ubuntu, but I don't see that one as a problem: privacy-invading online searches, proprietary drivers, CLAs and dubious copyright activities... But nope, no sign of microsoft there.

Besides, if you're worried about "Microsoft", you are going to be terrified when you learn (right now I guess) that Microsoft is now a platinum sponsor of the Linux Foundation, which in turns sponsors the one and only Linus Torvalds. Are you going to refuse their 500k$ a year because of religious reasons?

Turns out that satan isn't that bad after all... I would be more worried about Google, Facebook or Amazon these days.

Besides, satanists make the best music. :p

Now Automated "security" updates!!!!????

Why the quotes???!!!!1111oneoneone?

Those are security updates. Plus, you can choose whatever freaking upgrades you like, from proposed updates to backports to security in the unattended-upgrades configuration. Hell, you can just disable the whole thin if you want...

Any Admin worth their salt (and wishing to maintain an "employed" status HAS to comply with a server update regime, so some arguments fall at the first hurdle.

I fail to see the point here: any admin worth their salt will enable automated upgrades so ... we shouldn't do that? That makes no sense at all.

Personally I would not complain about an "Opt in" option during the installation process, but otherwise NO! I don't want to go Distribution dumping again :-(

I believe this is exactly what is being proposed, but if you dump distributions for such minor changes, you're going to be changing distributions like underwear and will never get anyting done.

This is about setting up tools for people to do stuff, while still allowing policies do be decided by the end users. We're not proposing to shove anything down the user's throat here, we're just building tools.

Please stay calm and carry on upgrading, however you wish.

Comment by anarcat
misc notes

@rohieb - thanks! i updated the links... turns out my linkchecker program was still broken (we need to fork that thing now). i fixed the link, but won't remove your comment so you get proper credit. :)

@paul - unattended-upgrade can do the equivalent of "dist-upgrade", for minor upgrades only of course. (it could possibly do the same for major upgrades as well, but i wouldn't recommend it.) you just need to configure it to allow more than just security upgrades... there's commented lines that may work. this is where the config is a bit more unreliable, as the commented lines do not always work.

Comment by anarcat
Interesting/important subject but...

Hi.

I have several machines on which I apply (almost) daily updates. Some are using debian, others are under some ubuntu flavor. Most of them are for my familly members and they often power-up their machine (autologin, SSD with fast boot), check if they've got mail and then power-down the machine.

There are (AFAIK) 3 ways of dealing with such "workload" :

  • wait for all updates to be applied to allow poweroff (that won't please my wife)

  • partially install updates with risk of FS/system corruption

  • install updates manually

Of course I choosed the 3rd one. I made my own script similar to apticron : at boot, apt-get update and, if updates are present, apt-get dist-upgrade -d to only download those updates (in this case a mail/sms is send to notify myself that they are present). Some later time I apply those updates while the system is running with desktop notification to the connected user so that the system is not powered off while the process is not finished.

I have seen so much rant from people about windows forbidding them to power-off their computer when they want that I think this should not be the course to follow...

Comment by Jean
debian-considering-automated-upgrades

no debian Upgrade ever worked for me. I had at least a Day of work to do to make it useable again after Upgrades. So I think this should be at least no default behaviour, at least not until Debian had mastered Upgrades such as Ubuntu.

Comment by Jan-Peter Rühmann
Created . Edited .