If you look around Google’s Mountain View, CA offices, you’ll see Windows machines, Chromebooks, Macs — and gLinux desktops. G what, you ask? Well, in addition to relying on Linux for its servers, Google has its very own Linux desktop distribution.
You can’t get it — darn it! — but for more than a decade, Google has been baking and eating its own homemade Linux desktop distribution. The first version was Goobuntu. (As you’d guess from the name, it was based on Ubuntu.)
In 2018, Google moved its in-house Linux desktop from the Goobuntu to a new Linux distro, the Debian-based gLinux. Why? Because, as Google explained, Ubuntu’s Long Term Support (LTS) two-year release “meant that we had to upgrade every machine in our fleet of over 100,000 devices before the end-of-life date of the OS.”
That was a pain. Add in the time-consuming need to fully customize engineers’ PCs, and Google decided that it cost too much. Besides, the “effort to upgrade our Goobuntu fleet usually took the better part of a year. With a two-year support window, there was only one year left until we had to go through the same process all over again for the next LTS. This entire process was a huge stress factor for our team, as we got hundreds of bugs with requests for help for corner cases.”
So, when Google had enough of that, it moved to Debian Linux (though not just vanilla Debian). The company created a rolling Debian distribution: GLinux Rolling Debian Testing (Rodete). The idea is that users and developers are best served by giving them the latest updates and patches as they’re created and deemed ready for production. Such distros include Arch Linux, Debian Testing, and openSUSE Tumbleweed.
For Google, the immediate goal was to get off the two-year upgrade cycle. As the move to Continuous Integration/Continuous Deployment (CI/CD) has shown, these incremental changes work well. They’re also easier to control and rollback if something goes wrong.
To make all this work without a lot of blood, sweat, and tears, Google created a new workflow system, Sieve. Whenever Sieve spots a new version of a Debian package, it starts a new build. These packages are built in package groups since separate packages often must be upgraded together. Once the whole group has been built, Google runs a virtualized test suite to ensure no core components and developer workflows are broken. Next, each group is tested separately with a full system installation, boot, and local test suite run. The package builds complete within minutes, but testing can take up to an hour.
Once that’s done, all the new packages are merged with the newest gLinux package pool. Then, when Google decides it’s time to release it into production, the team snapshots that pool. Finally, it rolls out the fresh release to the fleet. Of course, it’s not going to just dump it on users. Instead, it uses Site reliability engineering (SRE) principles such as incremental canarying to make sure nothing goes awry.
Over the years, Google has gotten better at this. Today, thanks to Sieve, the entire gLinux development team consists of a single on-duty release engineer position that rotates among team members. There are no big pushes to upgrade the fleet. No multi-stage alpha, betas, and general availability (GA) releases.
Better still, thanks to the rolling release schedule, Google can patch security holes on the entire fleet quickly without compromising stability. Previously, security engineers had to carefully review each Debian Security Advisory (DSA) to make sure the fix was in.
In addition, Google’s “improved testing suite and integration tests with key partner teams that run critical developer systems also yielded a more stable experience using a Linux distribution that provides the latest versions of the Linux Kernel. Our strong longing for automating everything in the pipeline has significantly reduced toil and stress within the team. It is now also possible for us to report bugs and incompatibilities with other library versions while making sure that Google tools work better within the Linux ecosystem.”
Looking ahead, Google’s team declared that it’ll work “more closely with upstream Debian and contribute more of our internal patches to maintain the Debian package ecosystem.”
That all sounds great. But I have two thoughts to share.
First, for some organizations, LTS releases still make sense. If you don’t need the newest, shiniest programs for your business, an Ubuntu or Red Hat LTS Linux still makes sense.
Second, and this is the important one: Sieve sounds like the cat’s meow. One program that can automate a rolling distro production pipeline to the point where it takes only one engineer to maintain a desktop used by 100,000+ users? Sign me up!
Better still, release Sieve’s code so we can all start producing rolling Linux desktop releases. How about it, Google? What do you say?
