Last December Red Hat announced that it would cease work on CentOS Linux, switch its focus to CentOS Stream, and sunset CentOS 8 in December 2021 instead of 2029. It was a change that shook the Linux community. CentOS is hugely popular: exact figures are difficult, but W3techs suggests that CentOS has 10.6 per cent of the websites running Linux, versus 0.9 per cent for Red Hat Enterprise Linux (RHEL). CentOS is a community build of RHEL whereas CentOS Stream is an upstream project, “a preview of upcoming Red Hat Enterprise Linux minor and major releases,” as described by Red Hat.
“The end of life of CentOS as we know it had a negative effect on my company and my company’s customers as well as pretty much everybody in the industry,” Kurtzer told The Register. “I felt it was important to take the lessons learned from CentOS and bring that into a new project… and I enjoy operating systems.”
Kurtzer was one of the original founders of CentOS, along with Rocky McGough (hence the new name) and Lance Davis. CentOS came about when Red Hat Linux gave way to Red Hat Enterprise Linux in 2003 – an equally traumatic change, according to Kurtzer. “They end-of-lifed multiple versions of the distribution all at once on one day basically, and the whole community needed something urgently. We already had a build system because we were building Caos Linux and we said, let’s see if this build system will build Red Hat source RPMS and it did,” he recalled.
What is different this time round? “The early versions of CentOS were built on laptops and personal workstations and that created a very small and cliquey culture,” he said. “That culture persisted for pretty much the entire life of CentOS, even after Red Hat acquired it, it was still seen to be a small number of core developers.”
“With Rocky we didn’t have the same urgency,” said Kurtzer, with CentOS 8 being supported to the end of 2021. “We had about a year to create the successor. I felt that enterprises needed six months to validate tests and do a transition. So I told the team, we’ve got six months. I wanted to create a much more diverse community of contributors and grow the project so it’s not a small clique.”
Much of the initial work for Rocky Linux was in creating the infrastructure. “It’s a fairly complicated infrastructure to build,” Kurtzer told us. “We had to write code to the orchestration of some of the workloads, so distrobuild is a piece of software that we’ve open-sourced. Building the operating system took about two months. It was an easier problem to solve than the infrastructure… we emulated the Fedora infrastructure because that has a lot of outside contributors.”
Kurtzer talked at length about the governance of Rocky Linux and the fact that he created a B (public benefit) Corporation rather than a 501(c) (non-profit) for the Rocky Enterprise Software Foundation, a move seemingly at odds with the community culture which he advocates. There is personal history here. “I was creating a 501(c)(3) for the Caos Foundation, which was the host of Caos Linux, which ended up being the host of CentOS, I had all my paperwork filled out and filed and that’s when the CentOS debacle started,” he said.
What was the CentOS debacle? The most public incident was this, but there is other history, some of which is mentioned here and in the linked video interview. Kurtzer now says: “What I’ve learned along the way is that a 501(c) anything is not a guarantee of integrity and honesty and good behaviour… and I said to myself, if I’m going to do this, I don’t want to put myself in that environment again.
“At some point maybe it will be the right thing to do, but initially the way that I can guarantee that’s not going to happen is if I do have some amount of control. It’s not a new thing for open-source projects to have a single leader in control,” he said, referencing Linus Torvalds and Linux, though now “he’s given a lot of the property over to the Linux Foundation which is a 501(c).”
He insisted that there are “checks and balances,” not least that “I actually cannot take over the infrastructure. I just physically can’t”. The reason is that credentials are held by teams, each of which controls their part of the project, such as engineering, testing, and release.
“If I were to do something that is a bad move for the project, or that the team does not agree with, they can change the name and fork or move the infrastructure almost immediately,” he said. “I want to encourage that… and a second check and balance is the transparency of the infrastructure, specifically for replication. If we fail in some way, I want to ensure that another project can pick up where we left off.”
According to Kurtzer, the key to longevity for an open-source project is “having multiple companies involved and having a voice… it’s going to create the stability that people need right now in open source, thanks to what’s been happening to everything from Mongo through to Elastic,” referencing two companies that have changed the licensing of their open-source products.
Rocky Linux, and other RHEL-compatible distros such as AlmaLinux and Oracle Linux, can only succeed while RHEL itself remains successful. How does the work of Rocky Linux affect Red Hat? “I actually think this is good for Red Hat,” Kurtzer claims, because every organisation that uses the RHEL-compatible distros helps to cement the idea that RHEL is the enterprise standard. “The other option? Everybody goes to Debian and Ubuntu, or SUSE,” he said.
Aside from that, does Rocky Linux give anything back to Red Hat? This is where Kurtzer says something surprising. “CentOS to me means Community Enterprise Operating System and it’s obvious that they killed the Community Enterprise Operating System by end-of-lifing it.
“Now, Neal Gompa [a member of the openSUSE board] challenged me two days ago on this, that the move to Stream is giving the community a more direct mechanism than Fedora to interoperate with this. CentOS has gone from being the operating system for the community enterprise to now being the developers’ interface to the enterprise operating system.
“It completely changes the perspective of what Stream is. I’m finally OK with calling it CentOS Stream. I was upset with it for a while because we came up with the name CentOS and then all of a sudden it was killed.”
I think Red Hat has done a tremendous job in terms of how they how they orchestrated this. I was slow on the uptake but I get what they’re doing now.
This is the second mutual benefit with Red Hat, he said. “We can interface with CentOS Stream. Enterprise Linux is pulling from the CentOS Git repository as we pull from the CentOS Git repository. We’re more of a peer to it. What we’re all downstream from is CentOS Stream. Now we can actually push bug fixes directly into that same git repository that Red Hat’s pulling from.
“So is there a mutualistic benefit? Absolutely, and I’m looking forward to being able to contribute back upstream to CentOS Stream. And then to have both Red Hat as well as Rocky, as well as all of the enterprise Linux distributions, benefit from that. I think Red Hat has done a tremendous job in terms of how they how they orchestrated this. I was slow on the uptake but I get what they’re doing now.”
Despite this change of perspective, Kurtzer still feels the way the change came about was disruptive. “I know very large enterprises that were in the process of transitioning to Ubuntu. I had a few of them contact me, I’m talking very large oil and gas, telecommunications, health and manufacturing, who said please give us a brief on what it is you’re doing because we’re nervous, we’re trying to convert all of our tooling to Ubuntu and Debian, but if we don’t have to do that, we don’t want to.”
Where is secure boot for Rocky Linux?
What is the progress with signing the Rocky Linux build for secure boot, which has been a frustration for some potential users? Kurtzer said that Microsoft, which issues the certificates, had delayed things. “We were actually just being held up by them… we’re now in the process of just getting everything validated.
“I’m hoping it’s going to be within weeks that we’re going to see secure boot released and properly done, the process took a little while because we did everything from scratch and we didn’t have any legacy.” According to Kurtzer, now that the infrastructure is in place, Rocky Linux should release at “about the same cadence” as other RHEL-compatible builds.
What will come next once Rocky Linux 8 is fully done?
“The goal is to extend the operating system via special interests groups (SIGs),” said Kurtzer. “I’m really interested in high-performance computing and in cloud hybrid. Other people have spun up a hyperscale SIG, a SIG for legacy hardware, a SIG for laptops and workstations, a SIG for media and entertainment, a SIG for storage, a SIG for EDA [Electronic Design Automation]. There’s going to be some overlap with CentOS SIGs as well as Fedora and EPEL packaging, but I think what we can do is be the new kid on the block.”
Finally, Kurtzer said that having multiple RHEL-compatible projects is OK. “Having multiple projects out there that are also solving this problem, I see as a benefit to the community.
“I don’t see it as a separation of resources… everything should be transparent and everybody should be able to see what the other people are doing. Consumers know that if something happens to Rocky, in a matter of minutes that can transition to Alma. If something happens to Alma, in a matter of minutes they can transition to Oracle or Red Hat because we’re all compatible. None of this even matters. We’re all solving the same problem.”
