Kubernetes: How a Rejected Internal Project Became a Global Standard

San Francisco, June 9-10, 2014. In the halls of the Marriott Marquis hotel, the first DockerCon is held. “Containers everywhere!” is printed on the back of event T-shirts. The first big news of DockerCon is the launch of Docker 1.0, but there is much more to come. It is the beginning of what is later named “The Container Orchestration War“. Apache Mesos, Red Hat’s GearD, Docker Libswarm, Facebook’s Tupperware and, most importantly, Google’s Kubernetes are all announced on the same two days, to the extent that Brad Rydzewski (Drone.io founder) comments:

“What I learned at #dockercon: everyone is building their own orchestration platform. Seriously. Everyone”.

Only one, however, emerged as a leader in container orchestration, so much so that it has become almost synonymous with it. We are talking about Kubernetes, which won that “technology war” in 2017, when AWS announced their Elastic Container Service for Kubernetes (EKS). And AWS was not the only one: other companies had already announced native support for Kubernetes, such as Mesosphere, Pivotal and Docker. That’s on top of Google and Azure offering managed Kubernetes services, and, you may have heard of it, Red Hat’s OpenShift Container Platform. Although it was not clear which secret sauce made Kubernetes craved by developers and operation teams, one thing was certain. Kubernetes is now a standard, not in the official sense but as a de facto common ground.

This article does not want to delve into the technical reasons why Kubernetes rose to prominence. It follows its story, from the early rejections of Google leadership, to the “right time and right place” on a company shuttle, and finally to international validation thanks to… Pokémon Go.

Today’s landscape

According to CNCF‘s Cloud Native Landscape, 58 companies sell Kubernetes distributions today, meaning core open source Kubernetes code packaged for redistribution. Big names here: Red Hat, with OpenShift Container Platform, AWS, Google, Oracle, Vcluster, Microsoft Azure, Ericsson, Rancher, VMware, and others. These are in addition to 48 hosted Kubernetes services, offered by infrastructure providers such as AWS, Digital Ocean, Alibaba Cloud, Microsoft Azure, Google, Huawei, Oracle, SAP (and, again, Red Hat).

Speed, efficiency, scalability on resources demands, high portability of applications from cloud providers to on-premise data centers, flexibility of deploying applications, whether they are monolithic or microservices, infrastructure resilience: a truly full recipe of chief ingredients for success, which explains why Kubernetes is the keyword today when it comes to breaking through the containerization wall.

The popularity of the container orchestrator is now unquestioned, and it was greeted with enthusiasm even on that day, June 10, 2014, when it was showcased at DockerCon. But its genesis has a complex story, and one that has seen its fortunes rise thanks to fortuitous coincidences.

The triple alignment of opportunities

Let’s take a step back to the previous year, 2013. Every company is an Internet company, and Silicon Valley is beginning to embrace the DevOps movement, where virtualization is gradually giving way to the cloud.

“The cloud was starting to become important. At Google, we realized that this was a space, which we needed to be in”.

says Craig McLukie (now VMware R&D vice president, and formerly at Google) in “Kubernetes: The Documentary” by Honeypot.

But not Google is the dominant cloud provider: it is the giant AWS, with S3, that has such a leading position that it seems difficult to strategically change the game for competing companies.

This is where Docker’s open source success of 2013 comes in: in the famous Lighting Talk of just 5 minutes, at PyCon US, Solomon Hykes illustrates for the first time Docker, founded on isolating processes and running them on any platform, bare-metal, virtual box or cloud instance. He gets a thunderous applause already in the third minute. Solomon Hykes is the founder and CEO of dotCloud: ever heard of it? It is nothing more than the former name of Docker Inc, changed that same year after the technology’s incredible success.

So we already have an alignment of two factors: the desire to undermine AWS from its predominant place in the cloud market, and the inspiring novelty of an open-source container engine that had shaken the IT world with its simplicity and efficiency. But it doesn’t end there: Red Hat also moved its pawns in this chess game, and in these years it has to figure out what would be the next flagship product after its Linux distribution RHEL. And Red Hat did contribute to the birth of Kubernetes, in the figure of the architect Clayton Coleman, but we will get to that later.

From Borg to Seven of Nine

The concept of containers is not new in 2013, and it is certainly not invented by Docker. In fact, Google has been using containers for years, in the cluster manager called Borg, which would later become the inspiration for Kubernetes. This curious name is inspired by the Star Trek movies franchise where The Borg is an antagonist alien group.

The system serves as an infrastructure to provide Google Cloud Platform users for computational needs. However, after the release of the IaaS Google Compute Engine platform, Google realizes that customers are paying a lot for CPUs but have extremely low utilization rates. The problem lies in Virtual Machines, while the future of computing is beginning to involve containers.

Joe Beda, co-founder of Kubernetes and now Principal Engineer at VMware, summarizes some of the questions that were swirling around in Google’s engineering:

“Can we create a Borg-like system in a way that applies to the cloud? And […] do we use Borg itself or we do something new, or we adapt an open project that’s already out there?”.

Joe, Craig McLukie and Brendan Burns (Senior Staff Software Engineer at Google) look to the Docker container system as the game-changer, even though v. 1.0 has not been announced yet. The three co-founder are soon followed by other Google engineers, including Brian Grant and Tim Hockin.

They begin working on the initial prototype, a sort of Borg spin-off, which, again to stay within the Star Trek framework, is called Project Seven of Nine or simply Project Seven, in honor of a member of Star Trek Voyager rescued by the Borg Collective. The project is designed to be a friendlier Borg, an easier tool for developers.

Open Sourcing the project: “You are crazy”.

Soon Brendan and Craig come to the idea to make the project open source, to achieve faster innovation, global collaboration and increased trust and adoption, to benefit both developers and users of the technology and establish it as a standard.

But this also complicates obtaining the green light from management. The team has to assure them that they were not enabling competition. When they try to pitch the prototype to Urs Hölzle, Head of Technical Infrastructure and Chief Architect of many of Google’s most important network innovations, the response is sharp:

“So let me get this straight. You want to build an external version of the Borg task scheduler. One of our most important competitive advantages. The one we don’t even talk about externally. And, on top of that, you want to open source it?”.

Not only he, but others wonder why, what they would get out of it, and what advantage it would bring to Google. “We went through a series of reviews with the executive team where we pitched and they were like, ‘you’re crazy, you’re crazy,'” Craig recalls.

Decision-making meetings are postponed from week to week, a stressful timeline for the team, which considers obvious the revolutionary scope of the project, and is ready to start working on it. Plus there are new ideas coming out every week, in other companies, that look similar to theirs: timing is critical, and if they don’t get the project out as soon as possible, then the world would move on and they’ll have to adapt to another company’s leading technology. However, the situation seems to be stuck.

The right place at the right moment

Returning from an off-site summit with members of Google’s technical infrastructure leadership team, Craig McLukie is sitting on a shuttle casually behind Eric Brewer (VP Infrastructure & Google Fellow). The two start talking, and Craig is just framing up what they were trying to do with the Project Seven, rejected so many times. Eric is also of the same mind, as he thinks it’s too core, too valuable for open source. However, he slowly realizes that one of the fundamental motivations for open source is the industry change in the cloud direction, that Google alone could not do, not without other contributors. Thousands of contributors.

From that day, something changes without the team even noticing. They get another meeting, again with Urs Hölzle, but Craig is at a point where he is not even going to travel for the pitch. So instead of going to Mountain View, he stays in Seattle on videocall. It is Brendan and Joe who meet with Urs and other senior leadership, still strongly opposed to open sourcing. But from home, Craig can see Urs playing on his phone for most of the talk. And that’s where he think they had made it.

They get the approval after six months of pitching and negotiation.

Now that the plans are concrete, they only have three months to develop the prototype and launch it at Dockercon. Craig recalls:

“Actually took less time to build the dot one release than it did to get approval to release it”.

There was still the question of the name.

Craig is driving: holding the steering wheel, he reflects that there must be another language for “pilot” or “someone steering”. That’s how the name is found in ancient Greek, κυβερνήτης, with the meaning of “helmsman” (someone who steers a ship, like a container ship). In homage to the original name, the logo, designed by Tim Hocklin, has seven sides. Interesting to know that they first asked Google’s marketing department for permission, which responded, “Open source? We don’t give a crap, just do whatever you want.”

Red Hat’s role

OpenShift began in 2010 but in 2013 Red Hat wants to use Docker inside OpenShift, even though it knows that’s not enough. OpenShift Container Platform 3 wouldn’t need a container engine but a container orchestrator.

Clayton Coleman (Architect, Containerized Application Infrastructure at Red Hat) contacts Google around this time asking if they were working on a similar project, and they respond by naming Project Seven, based on Borg and perhaps soon to be open-sourced. But they are elusive, giving few details and not knowing whether they will actually open source it.

Two weeks before DockerCon 2014, Clayton’s manager Matt Hicks (now Red Hat’s CEO) sends a message to Clayton, telling him that Google suddenly wants to know whether he is in on the project or not. That day, Clayton gets private access to the commit and begins to clean up and get much of the code into the correct Golang semantics.

“I was either the first or the second external contributor to Kubernetes,” he states. And that was the beginning of the kind of open community work they were looking for.

Announced to the world: DockerCon 2014

This brings us to the day Kubernetes was announced, June 10, 2014. It is the beginning of the Container Orchestration War.

“If I recall there were five or six other container management systems announced the same day. Some were proprietary and some were also open source. The same day”, Tim recalls. And he is not the only one, because Clayton also remarks: “Every big startup I felt like had a container orchestration project and half of them were announced at DockerCon 2014.”

Kubernetes, DockerSwarm and Mesos enter into competition on different levels. The goal of Kubernetes is a clean API fitting the issues of scheduling containers and working across many companies. Mesos has a very sophisticated scheduler and DockerSwarm wants to visualize a cluster of computers as one big computer, trying to make the experience as simple as possible, even at the cost of optimal capacity.

CNCF: a new home for Kubernetes

For a technology to be ubiquitous, used by everyone, driven by a league stronger than any individual’s contributions, it must be held under open and free governance. Instead, in 2014 Kubernetes is still actually in a GitHub organization owned by Google, and people in the community have to give Google the right to relicense the product. This discourages many people.

That is why the Cloud Native Computing Foundation is formed and announced at OSCON in June 2015 along with Kubernetes version 1.0. Its purpose is to take Kubernetes and adjacent technologies, harmonize them and advance the cloud native computing agenda for all.

Today it includes hundreds of products, including Kubernetes, Prometheus, Envoy and Fluentd, and provides a neutral home for all the projects and its community, ensuring that they remain open and accessible to all.

Pikachu was the one who validated Kubernetes

However, Kubernetes is still rivalring Mesos, DockerSwarn, and others, and the team is failing to capture large customers, such as Netflix and Airbnb, who choose the competition. In the meantime, they learn that Niantic is building Pokémon Go on GKE (Google Kubernetes Engine) and engage them to see if Kubernetes can fit their use case.

An incredible collaboration emerges: Niantic runs Pokémon Go on a single large cluster that, once launched, experiences 50 times the load of what they expected. As the game grows and grows, the Kubernetes team panicks because they don’t know how they are going to deliver more game space.

But, surprising, they we are actually ready for that.

The deal with Niantic is a big deal, not only financially, but also in validating the power of Kubernetes and establishing its success in the enterprise world.

The end of the war

Will Docker change to adopt Kubernetes? Is AWS suddenly going to adopt Kubernetes? At every DockerCon the team wonders, if someone would change everything with an announcement. And one day, in 2017, Solomon Hykes (from Docker) calls Tim. He has succeeded in building Docker on top of Kubernetes and he is announcing it at DockerCon in Copenhagen.

It is not the only company, though: during the same year many competitors embrace Kubernetes, such as Mesos and Pivotal. Towards the end of 2017, even AWS launches a Kubernetes product (EKS). Seeing Amazon closing the loop by supporting Kubernetes is a testament that it has become the standard across global cloud providers.

It is no coincidence that this article began with the introduction of AWS as the leading cloud provider that no one, not even Google, alone would be able to dethrone.

Kubernetes, as an open source project, has achieved its goal. It has taken on a life of its own among the worldwide companies and has evolved beyond what the small team at Google imagined at first.

Conclusions

The most innovative concepts from the “war” have persisted. And they have now been integrated and consolidated in the most recent checkpoint, Kubernetes. But, the reality is that Kubernetes will be replaced by something else, sooner or later. It may be a serverless component or a new way of considering APIs and how they should work together. Kubernetes is just the last checkpoint, and we look forward to what the next one will be.