Debian considering automated upgrades
The Debian project is looking at possibly making automatic minor upgrades to installed packages the default for newly installed systems. While Debian has a reliable and stable package update system that has been an inspiration for multiple operating systems (the venerable APT), upgrades are, usually, a manual process on Debian for most users.
The proposal was brought
up during
the Debian Cloud
sprint in
November by longtime Debian Developer Steve McIntyre. The rationale was
to make sure that users installing Debian in the cloud have a "secure"
experience by default, by installing and configuring the
unattended-upgrades
package within the images.
The unattended-upgrades package contains a Python program that
automatically performs any pending upgrade and is designed to run
unattended. It is roughly the equivalent of doing
apt-get update; apt-get upgrade in a cron job, but has special code to
handle error conditions, warn about reboots, and selectively upgrade
packages. The package was originally written for Ubuntu by Michael Vogt,
a longtime Debian developer and Canonical employee.
Since there was a concern that Debian cloud images would be different
from normal Debian installs, McIntyre suggested installing
unattended-upgrades by default on all Debian installs, so that people
have a consistent experience inside and outside of the cloud. The
discussion that followed was interesting as it brought up key issues one
would have when deploying automated upgrade tools, outlining both the
benefits and downsides to such systems.
Problems with automated upgrades
An issue raised in the following discussion is that automated upgrades may create unscheduled downtime for critical services. For example, certain sites may not be willing to tolerate a master MySQL server rebooting in conditions not controlled by the administrators. The consensus seems to be that experienced administrators will be able to solve this issue on their own, or are already doing so.
For example, Noah Meyerhans, a Debian developer,
argued
that "any reasonably well managed production host is going to be driven
by some kind of configuration management system" where competent
administrators can override the defaults. Debian, for example, provides
the policy-rc.d
mechanism
to disable service restarts on certain packages out of the box.
unattended-upgrades also features a way to disable upgrades on
specific packages that administrators would consider too sensitive to
restart automatically and will want to schedule during maintenance
windows.
Reboots were another issue discussed: how and when to deploy kernel
upgrades? Automating kernel upgrades may mean data loss if the reboot
happens during a critical operation. On Debian systems, the kernel
upgrade mechanisms already provide a /var/run/reboot-required flag
file that tools can monitor to notify users of the required reboot. For
example, some desktop environments will popup a warning prompting users
to reboot when the file exists. Debian doesn't currently
feature an equivalent warning for command-line operation: Vogt
suggested
that the warning could be shown along with the usual /etc/motd
announcement.
The ideal solution here, of course, is reboot-less kernel upgrades, which is also known as "live patching" the kernel. Unfortunately, this area is still in development in the kernel (as was previously discussed here). Canonical deployed the feature for the Ubuntu 16.04 LTS release, but Debian doesn't yet have such capability, since it requires extra infrastructure among other issues.
Furthermore, system reboots are only one part of the problem. Currently,
upgrading packages only replaces the code and restarts the primary
service shipped with a given package. On library upgrades, however,
dependent services may not necessarily notice and will keep running with
older, possibly vulnerable, libraries. While libc6, in Debian, has
special code to restart dependent services, other libraries like
libssl do not notify dependent services that they need to restart to
benefit from potentially critical security fixes.
One solution to this is the needrestart
package which inspects all
running processes and restarts services as necessary. It also covers
interpreted code, specifically Ruby, Python, and Perl. In my experience,
however, it can take up to a minute to inspect all processes, which
degrades the interactivity of the usually satisfying apt-get install
process. Nevertheless, it seems like needrestart is a key component of
a properly deployed automated upgrade system.
Benefits of automated upgrades
One thing that was less discussed is the actual benefit of automating upgrades. It is merely described as "secure by default" by McIntyre in the proposal, but no one actually expanded on this much. For me, however, it is now obvious that any out-of-date system will be systematically attacked by automated probes and may be taken over to the detriment of the whole internet community, as we are seeing with Internet of Things devices. As Debian Developer Lars Wirzenius said:
The ecosystem-wide security benefits of having Debian systems keep up to date with security updates by default overweigh any inconvenience of having to tweak system configuration on hosts where the automatic updates are problematic.
One could compare automated upgrades with backups: if they are not automated, they do not exist and you will run into trouble without them. (Wirzenius, coincidentally, also works on the Obnam backup software.)
Another benefit that may be less obvious is the acceleration of the feedback loop between developers and users: developers like to know quickly when an update creates a regression. Automation does create the risk of a bad update affecting more users, but this issue is already present, to a lesser extent, with manual updates. And the same solution applies: have a staging area for security upgrades, the same way updates to Debian stable are first proposed before shipping a point release. This doesn't have to be limited to stable security updates either: more adventurous users could follow rolling distributions like Debian testing or unstable with unattended upgrades as well, with all the risks and benefits that implies.
Possible non-issues
That there was not a backlash against the proposal surprised me: I expected the privacy-sensitive Debian community to react negatively to another "phone home" system as it did with the Django proposal. This, however, is different than a phone home system: it merely leaks package lists and one has to leak that information to get the updated packages. Furthermore, privacy-sensitive administrators can use APT over Tor to fetch packages. In addition, the diversity of the mirror infrastructure makes it difficult for a single entity to profile users.
Automated upgrades do imply a culture change, however: administrators approve changes only a posteriori as opposed to deliberately deciding to upgrade parts they chose. I remember a time when I had to maintain proprietary operating systems and was reluctant to enable automated upgrades: such changes could mean degraded functionality or additional spyware. However, this is the free-software world and upgrades generally come with bug fixes and new features, not additional restrictions.
Automating major upgrades?
While automating minor upgrades is one part of the solution to the problem of security maintenance, the other is how to deal with major upgrades. Once a release becomes unsupported, security issues may come up and affect older software. While Debian LTS extends releases lifetimes significantly, it merely delays the inevitable major upgrades. In the grand scheme of things, the lifetimes of Linux systems (Debian: 3-5 years, Ubuntu: 1-5 years) versus other operating systems (Solaris: 10-15 years, Windows: 10+ years) is fairly short, which makes major upgrades especially critical.
While major upgrades are not currently automated in Debian, they are
usually pretty simple: edit sources.list then:
# apt-get update && apt-get dist-upgrade
But the actual upgrade process is really much more complex. If you run into problems with the above commands, you will quickly learn that you should have followed the release notes, a whopping 20,000-word, ten-section document that outlines all the gory details of the release. This is a real issue for large deployments and for users unfamiliar with the command line.
The solutions most administrators seem to use right now is to roll their own automated upgrade process. For example, the Debian.org system administrators have their own process for the "jessie" (8.0) upgrade. I have also written a specification of how major upgrades could be automated that attempts to take into account the wide variety of corner cases that occur during major upgrades, but it is currently at the design stage. Therefore, this problem space is generally unaddressed in Debian: Ubuntu does have a do-release-upgrade command but it is Ubuntu-specific and would need significant changes in order to work in Debian.
Future work
Ubuntu currently defaults to "no
automation"
but, on install, invites users to enable unattended-upgrades or
Landscape, a proprietary
system-management service from Canonical. According to Vogt, the company
supports both projects equally as they differ in scope:
unattended-upgrades just upgrades packages while Landscape aims at
maintaining thousands of machines and handles user management, release
upgrades, statistics, and aggregation.
It appears that Debian will enable unattended-upgrades on the images
built for the cloud by default. For regular installs, the consensus
that has
emerged
points at the Debian installer prompting users to ask if they want to
disable the feature as well. One reason why this was not enabled
before is that unattended-upgrades had serious bugs in the past that
made it less attractive. For example, it would simply fail to follow
security
updates, a
major bug that was fortunately promptly fixed by the maintainer.
In any case, it is important to distribute security and major upgrades on Debian machines in a timely manner. In my long experience in professionally administering Unix server farms, I have found the upgrade work to be a critical but time-consuming part of my work. During that time, I successfully deployed an automated upgrade system all the way back to Debian woody, using the simpler cron-apt. This approach is, unfortunately, a little brittle and non-standard; it doesn't address the need of automating major upgrades, for which I had to revert to tools like cluster-ssh or more specialized configuration management tools like Puppet. I therefore encourage any effort towards improving that process for the whole community.
More information about the configuration of unattended-upgrades can be found in the Ubuntu documentation or the Debian wiki.
Note: this article first appeared in the Linux Weekly News.
Awesome article!
Thanks for sharing!
Just ask on install defaulting to yes. That should make everybody happy
I think it's a good idea to enable automatic upgrades by default. I do that allready myself. But sometimes I see there is a problem, it would be good to have a way to be noticed.
I would like a script what would do "apt-get dist-upgrade", otherwise not everything is updated. I do not mean upgrades from Debian 8 to Debian 9, but I still mean upgrades from Debian 8.1 to Debian 8.2, and from everything what's enabled in sources.list. Now I use "*" in the configuration.
I moved away from Ubuntu as I felt they were getting into bed with the devil (i.e. Microsoft). Now Automated "security" updates!!!!????
Any Admin worth their salt (and wishing to maintain an "employed" status HAS to comply with a server update regime, so some arguments fall at the first hurdle.
Personally I would not complain about an "Opt in" option during the installation process, but otherwise NO! I don't want to go Distribution dumping again
Ubuntu got in bed with Microsoft? I must have missed that meeting. There are plenty of wrong things with Ubuntu, but I don't see that one as a problem: privacy-invading online searches, proprietary drivers, CLAs and dubious copyright activities... But nope, no sign of microsoft there.
Besides, if you're worried about "Microsoft", you are going to be terrified when you learn (right now I guess) that Microsoft is now a platinum sponsor of the Linux Foundation, which in turns sponsors the one and only Linus Torvalds. Are you going to refuse their 500k$ a year because of religious reasons?
Turns out that satan isn't that bad after all... I would be more worried about Google, Facebook or Amazon these days.
Besides, satanists make the best music. :p
Why the quotes???!!!!1111oneoneone?
Those are security updates. Plus, you can choose whatever freaking upgrades you like, from proposed updates to backports to security in the unattended-upgrades configuration. Hell, you can just disable the whole thin if you want...
I fail to see the point here: any admin worth their salt will enable automated upgrades so ... we shouldn't do that? That makes no sense at all.
I believe this is exactly what is being proposed, but if you dump distributions for such minor changes, you're going to be changing distributions like underwear and will never get anyting done.
This is about setting up tools for people to do stuff, while still allowing policies do be decided by the end users. We're not proposing to shove anything down the user's throat here, we're just building tools.
Please stay calm and carry on upgrading, however you wish.
@rohieb - thanks! i updated the links... turns out my linkchecker program was still broken (we need to fork that thing now). i fixed the link, but won't remove your comment so you get proper credit.
@paul - unattended-upgrade can do the equivalent of "dist-upgrade", for minor upgrades only of course. (it could possibly do the same for major upgrades as well, but i wouldn't recommend it.) you just need to configure it to allow more than just security upgrades... there's commented lines that may work. this is where the config is a bit more unreliable, as the commented lines do not always work.
Hi.
I have several machines on which I apply (almost) daily updates. Some are using debian, others are under some ubuntu flavor. Most of them are for my familly members and they often power-up their machine (autologin, SSD with fast boot), check if they've got mail and then power-down the machine.
There are (AFAIK) 3 ways of dealing with such "workload" :
wait for all updates to be applied to allow poweroff (that won't please my wife)
partially install updates with risk of FS/system corruption
install updates manually
Of course I choosed the 3rd one. I made my own script similar to apticron : at boot,
apt-get updateand, if updates are present,apt-get dist-upgrade -dto only download those updates (in this case a mail/sms is send to notify myself that they are present). Some later time I apply those updates while the system is running with desktop notification to the connected user so that the system is not powered off while the process is not finished.I have seen so much rant from people about windows forbidding them to power-off their computer when they want that I think this should not be the course to follow...
no debian Upgrade ever worked for me. I had at least a Day of work to do to make it useable again after Upgrades. So I think this should be at least no default behaviour, at least not until Debian had mastered Upgrades such as Ubuntu.