The Debian project is looking at possibly making automatic minor upgrades to installed packages the default for newly installed systems. While Debian has a reliable and stable package update system that has been an inspiration for multiple operating systems (the venerable APT), upgrades are, usually, a manual process on Debian for most users.
The proposal was brought
the Debian Cloud
November by longtime Debian Developer Steve McIntyre. The rationale was
to make sure that users installing Debian in the cloud have a "secure"
experience by default, by installing and configuring the
package within the images.
unattended-upgrades package contains a Python program that
automatically performs any pending upgrade and is designed to run
unattended. It is roughly the equivalent of doing
apt-get update; apt-get upgrade in a cron job, but has special code to
handle error conditions, warn about reboots, and selectively upgrade
packages. The package was originally written for Ubuntu by Michael Vogt,
a longtime Debian developer and Canonical employee.
Since there was a concern that Debian cloud images would be different
from normal Debian installs, McIntyre suggested installing
unattended-upgrades by default on all Debian installs, so that people
have a consistent experience inside and outside of the cloud. The
discussion that followed was interesting as it brought up key issues one
would have when deploying automated upgrade tools, outlining both the
benefits and downsides to such systems.
An issue raised in the following discussion is that automated upgrades may create unscheduled downtime for critical services. For example, certain sites may not be willing to tolerate a master MySQL server rebooting in conditions not controlled by the administrators. The consensus seems to be that experienced administrators will be able to solve this issue on their own, or are already doing so.
For example, Noah Meyerhans, a Debian developer,
that "any reasonably well managed production host is going to be driven
by some kind of configuration management system" where competent
administrators can override the defaults. Debian, for example, provides
to disable service restarts on certain packages out of the box.
unattended-upgrades also features a way to disable upgrades on
specific packages that administrators would consider too sensitive to
restart automatically and will want to schedule during maintenance
Reboots were another issue discussed: how and when to deploy kernel
upgrades? Automating kernel upgrades may mean data loss if the reboot
happens during a critical operation. On Debian systems, the kernel
upgrade mechanisms already provide a
file that tools can monitor to notify users of the required reboot. For
example, some desktop environments will popup a warning prompting users
to reboot when the file exists. Debian doesn't currently
feature an equivalent warning for command-line operation: Vogt
that the warning could be shown along with the usual
The ideal solution here, of course, is reboot-less kernel upgrades, which is also known as "live patching" the kernel. Unfortunately, this area is still in development in the kernel (as was previously discussed here). Canonical deployed the feature for the Ubuntu 16.04 LTS release, but Debian doesn't yet have such capability, since it requires extra infrastructure among other issues.
Furthermore, system reboots are only one part of the problem. Currently,
upgrading packages only replaces the code and restarts the primary
service shipped with a given package. On library upgrades, however,
dependent services may not necessarily notice and will keep running with
older, possibly vulnerable, libraries. While
libc6, in Debian, has
special code to restart dependent services, other libraries like
libssl do not notify dependent services that they need to restart to
benefit from potentially critical security fixes.
One solution to this is the
package which inspects all
running processes and restarts services as necessary. It also covers
interpreted code, specifically Ruby, Python, and Perl. In my experience,
however, it can take up to a minute to inspect all processes, which
degrades the interactivity of the usually satisfying
process. Nevertheless, it seems like
needrestart is a key component of
a properly deployed automated upgrade system.
One thing that was less discussed is the actual benefit of automating upgrades. It is merely described as "secure by default" by McIntyre in the proposal, but no one actually expanded on this much. For me, however, it is now obvious that any out-of-date system will be systematically attacked by automated probes and may be taken over to the detriment of the whole internet community, as we are seeing with Internet of Things devices. As Debian Developer Lars Wirzenius said:
The ecosystem-wide security benefits of having Debian systems keep up to date with security updates by default overweigh any inconvenience of having to tweak system configuration on hosts where the automatic updates are problematic.
One could compare automated upgrades with backups: if they are not automated, they do not exist and you will run into trouble without them. (Wirzenius, coincidentally, also works on the Obnam backup software.)
Another benefit that may be less obvious is the acceleration of the feedback loop between developers and users: developers like to know quickly when an update creates a regression. Automation does create the risk of a bad update affecting more users, but this issue is already present, to a lesser extent, with manual updates. And the same solution applies: have a staging area for security upgrades, the same way updates to Debian stable are first proposed before shipping a point release. This doesn't have to be limited to stable security updates either: more adventurous users could follow rolling distributions like Debian testing or unstable with unattended upgrades as well, with all the risks and benefits that implies.
That there was not a backlash against the proposal surprised me: I expected the privacy-sensitive Debian community to react negatively to another "phone home" system as it did with the Django proposal. This, however, is different than a phone home system: it merely leaks package lists and one has to leak that information to get the updated packages. Furthermore, privacy-sensitive administrators can use APT over Tor to fetch packages. In addition, the diversity of the mirror infrastructure makes it difficult for a single entity to profile users.
Automated upgrades do imply a culture change, however: administrators approve changes only a posteriori as opposed to deliberately deciding to upgrade parts they chose. I remember a time when I had to maintain proprietary operating systems and was reluctant to enable automated upgrades: such changes could mean degraded functionality or additional spyware. However, this is the free-software world and upgrades generally come with bug fixes and new features, not additional restrictions.
While automating minor upgrades is one part of the solution to the problem of security maintenance, the other is how to deal with major upgrades. Once a release becomes unsupported, security issues may come up and affect older software. While Debian LTS extends releases lifetimes significantly, it merely delays the inevitable major upgrades. In the grand scheme of things, the lifetimes of Linux systems (Debian: 3-5 years, Ubuntu: 1-5 years) versus other operating systems (Solaris: 10-15 years, Windows: 10+ years) is fairly short, which makes major upgrades especially critical.
While major upgrades are not currently automated in Debian, they are
usually pretty simple: edit
# apt-get update && apt-get dist-upgrade
But the actual upgrade process is really much more complex. If you run into problems with the above commands, you will quickly learn that you should have followed the release notes, a whopping 20,000-word, ten-section document that outlines all the gory details of the release. This is a real issue for large deployments and for users unfamiliar with the command line.
The solutions most administrators seem to use right now is to roll their own automated upgrade process. For example, the Debian.org system administrators have their own process for the "jessie" (8.0) upgrade. I have also written a specification of how major upgrades could be automated that attempts to take into account the wide variety of corner cases that occur during major upgrades, but it is currently at the design stage. Therefore, this problem space is generally unaddressed in Debian: Ubuntu does have a do-release-upgrade command but it is Ubuntu-specific and would need significant changes in order to work in Debian.
Ubuntu currently defaults to "no
but, on install, invites users to enable
Landscape, a proprietary
system-management service from Canonical. According to Vogt, the company
supports both projects equally as they differ in scope:
unattended-upgrades just upgrades packages while Landscape aims at
maintaining thousands of machines and handles user management, release
upgrades, statistics, and aggregation.
It appears that Debian will enable
unattended-upgrades on the images
built for the cloud by default. For regular installs, the consensus
points at the Debian installer prompting users to ask if they want to
disable the feature as well. One reason why this was not enabled
before is that
unattended-upgrades had serious bugs in the past that
made it less attractive. For example, it would simply fail to follow
major bug that was fortunately promptly fixed by the maintainer.
In any case, it is important to distribute security and major upgrades on Debian machines in a timely manner. In my long experience in professionally administering Unix server farms, I have found the upgrade work to be a critical but time-consuming part of my work. During that time, I successfully deployed an automated upgrade system all the way back to Debian woody, using the simpler cron-apt. This approach is, unfortunately, a little brittle and non-standard; it doesn't address the need of automating major upgrades, for which I had to revert to tools like cluster-ssh or more specialized configuration management tools like Puppet. I therefore encourage any effort towards improving that process for the whole community.