CoreOS Fest and the world of containers, part 1

This article brought to you by LWN subscribers Subscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible.

It's been a Linux container bonanza in San Francisco recently, and that includes a series of events and announcements from multiple startups and cloud hosts. It seems like everyone is fighting for a piece of what they hope will be a new multi-billion-dollar market. This included Container Camp on April 17 and CoreOS Fest on May 5th and 6th, with DockerCon to come near the end of June. While there is a lot of hype, the current container gold rush has yielded more than a few benefits for users — and caused technological development so rapid it is hard to keep up with.

CoreOS Fest was a demonstration of how trendy containers are in the startup world right now. The event sold out for 300 attendees, despite being planned within the last six months and located in an ill-suited venue called The Village in San Francisco's Tenderloin. Despite the drawbacks, it was well-attended; I suspect that DockerCon will be even bigger. This audience, based on responses to speaker questions, was almost entirely made up of system administrators and dedicated DevOps staff.

Among the latest developments in the container world are new funding, a new appc committee, the release of CoreOS, Inc.'s Tectonic platform, Kubernetes, new tools and techniques for databases on containers, systemd integration, Project Calico, Sysdig, and more. Over this series of three articles, we're going to be exploring some of the developments in the world of Linux containers. But first, some Silicon Valley politics.

Note to forestall confusion: For the rest of this article, "Docker" and "CoreOS" refer to the respective open-source projects and related software, and "Docker, Inc." and "CoreOS, Inc." refer to the companies.

The orchestration gold rush and CoreOS vs. Docker

CoreOS, Inc. was Docker, Inc.'s strongest partner, but split with it only six months before CoreOS Fest, when it launched the competing container platform rkt (formerly also known as Rocket). The separation between the two companies seems to have become a divorce, as competition between them for users and capital has heated up. Docker, Inc. received $95 million in Series D funding on April 14th. That same week, CoreOS, Inc. raised $12 million, notably including an investment from Google Ventures.

The conference made it obvious that it's a strange separation, though. Probably 80% of the people in the room at the keynote were Docker users, and most of the technologies introduced are compatible with Docker. Yet few people on stage ever said the word "Docker"; one speaker even went so far as to use the phrase "the D word" instead of saying the name.

A lot of this competition centers around orchestration platforms, which refers to the suite of tools required to deploy, manage, and network large numbers of containers that make up a container-based software infrastructure. The idea is that while Linux containers on their own are useful as a development platform, for them to be useful as the basis for the whole software stack, there is a need for several orchestration tools. This includes: container schedulers that deploy groups of containers to physical servers, cluster information stores for container data sharing and coordination, software-defined networking for connecting containers, along with resource management and monitoring tools.

All of the companies in the container space seem to have decided that orchestration is where they can differentiate products, and is therefore the primary way to exert influence and create revenue. It's not just Docker, Inc. and CoreOS, Inc. in this field: Red Hat's Project Atomic, Ubuntu's Snappy Core, Joyent's Triton, and the Apache Mesos project are all strong contenders for the future of container orchestration. Notably, Microsoft has now announced that Windows Containers, which make container deployment available for users of Windows Server and .Net Stacks, will be available in 2016.

Perhaps because of this intense competition, there was a much stronger emphasis on talking about container security at CoreOS Fest than there has been at the prior CoreOS Meetups. Weak access controls and a lack of other security measures has been one of CoreOS, Inc.'s main criticisms of the Docker project from before the split in December.

The new appc committee

This security focus was evident in the App Container (appc) specification panel. The specification was created and had its 0.1 release in December. It describes the required properties of an Application Container Image (ACI); rkt is CoreOS's implementation of that specification as explained in an earlier article.

Before discussing any new features, CoreOS, Inc. CEO Alex Polvi cautioned the audience that the committee was still working on the security part of the specification; "sometimes it takes a while to get these things right", he said. He then introduced the members of the panel, who are also the committee in charge of the new "appc specification community": Vincent Batts of Red Hat, Tim Hockin of Google, Charles Aylward of Twitter, along with Brandon Philips and Jonathan Boulle of CoreOS, Inc. Ken Robertson of Apcera was also on the panel, although he is not a member of the committee.

That was one of the two big announcements of the morning: CoreOS has created a governance document and turned over the appc specification project to a committee of "maintainers", the majority of whom do not work for CoreOS, Inc. While this is not a foundation or other incorporated body, the move seems intended to make appc a real, independent specification. It was also a demonstration of partner support for the spec. "[appc] should feel like the HTML 5 standard. Shared standards plus competition creates better product", Polvi said.

To start out, each of the panelists explained their company's interest in the appc specification.

Apcera was working on its own closed-source container technology when the appc project was announced. It quickly worked to bring its own technology in line with the draft specification. "When we saw the rkt announcements, we thought 'damn, now we have to build an abstraction'", said Robertson. He also announced the release of Kurma, Apcera's bootable container infrastructure that is compatible with appc-compliant containers.

Twitter already had a lot of existing infrastructure, and Docker didn't fit in with what it had, Aylward said. Rkt and appc allowed the company to pick and choose what it implemented. Hockin noted that Google is looking to create an open-source platform that mirrors how its large-scale, proprietary container platform works, and has formed a tight partnership with CoreOS to support it. "Coming from Google, I'm interested in building the cathedral. But before you can build the cathedral, you need to pour the foundation", he said.

Batts was more equivocal, saying that Red Hat's interest in appc is in supporting standards and user choice. Since Red Hat's Project Atomic is also closely aligned with Docker, Red Hat's fairly neutral stance makes sense. He explained it as "finding commonalities and working with them which drives everything else forward."

Once corporate politics were out of the way, the panelists discussed the state of the spec and current development. They started with some of the major challenges and feature requests, such as making encryption work with service discovery, the need for a better ACI validator, and the need to lock down more system calls inside the container for better security. The main challenge, however, is that parts of the 0.5 specification are still vaguely described, which frequently forces the rkt team to halt work while the specification is hammered out.

"You can write a spec, but without an implementation, you don't know that you can build it. So implementation and spec need to go hand-in-hand", said Aylward.

The committee agreed on the main goal of the project: for ACI to be the reference format for container images, and for developers to build ACI images first, then use them to create whatever other packages are needed. They did not agree about everything else. For example, while CoreOS, Inc. is devoted to systemd for container bootstrapping and initialization, Google is not using systemd. Hockin also disagreed with the other committee members on how much container isolation could be part of the spec. He believes that, eventually, by separating the general "spec" and the "os-spec", appc can encompass a full application binary interface (ABI) in order to provide full isolation for container runtimes. "It's pretty well understood that containers are not a security barrier. This is something that needs to evolve from the inside out", Hockin said.

Tectonic

The other major announcement for the conference was CoreOS, Inc.'s launch of the Tectonic platform, which is the full CoreOS, Inc. suite of tools. That includes CoreOS Linux, the container deployment tool fleet, the clustered data store etcd, the flannel virtual networking system, and the image repository Quay.io, all combined with Google's Kubernetes project (see below). The idea is to present a single, user-friendly integrated platform for large-scale container orchestration. Polvi called it "Google's infrastructure for everyone else, or GIFEE".

Tectonic is proprietary, commercial software that CoreOS, Inc. plans to sell to customers who want a fully integrated stack with a nice GUI and are willing to pay for it. While all the tools used are available as open source — except for the GUI — doing your own orchestration is difficult due to the newness of the tools and the complex ways in which they interact.

CoreOS's fleet and flannel may seem to have overlapping and conflicting functionality with Kubernetes, but in Tectonic they are complementary. According to Kelsey Hightower of CoreOS, Inc., fleet is used in Tectonic to bootstrap and monitor Kubernetes, which can otherwise require a lot of hand configuration. Flannel supplies an overlay networking system that supports Kubernetes' service discovery features.

As a demonstration product, Intel, Supermicro, and data center vendor Redapt announced a joint venture in making preconfigured Tectonic stacks available. At the conference, they showed off a quarter-rack of servers that were running the beta version of Tectonic as a "plug and play" container infrastructure that was ready to go. It is also possible to run the Tectonic beta on top of Amazon EC2.

Kubernetes

The other project logo that was just as pervasive at CoreOS Fest as the CoreOS logo was the Kubernetes ship's wheel. Brendan Burns, head of the Kubernetes project at Google, explained what Kubernetes was, how it worked, and how it relates to CoreOS and containers.

He started by separating operations into four layers: application ops, cluster ops, kernel ops, and hardware ops. Kubernetes operates at the level of cluster ops, synchronizing servers into a "unified compute substrate", in order to decouple application requirements from specific knowledge of the hardware, in the same way that a public cloud does.

Developers interact with Kubernetes through its API server, which supports both a command-line interface and a JavaScript-based web API. All of its data is contained in etcd. Like the configuration management system Puppet, Kubernetes uses a declarative approach where users specify how the system should be, and then Kubernetes reconciles the actual state with the desired state. An example of this would be "exactly three Redis servers should be running", which would cause Kubernetes to either stop or start containers until that declaration was true.

Deploying containers to servers in order to provide requested services is known as "scheduling". The "atomic unit of scheduling" in Kubernetes is the "pod", a group of containers, networking, and data volumes. This allows Kubernetes to schedule services that require multiple components that don't work if not placed on the same physical server, such as a database and its file storage.

The other big feature of Kubernetes is service discovery, which lets application developers use service proxies to talk to services without knowing where those containers are on the network. This proxy network is driven by "labels" attached to each pod and container that show the services that they provide. In this model, multiple pods supplying the same service are treated as fungible units — Kubernetes will load-balance among them.

Compared with competing orchestration frameworks such as CoreOS's own fleet, Apache Mesos, or Docker, Inc.'s Swarm and Machine, Kubernetes feels more feature-complete and mature in simple trials at my company. Since it's a de facto port of Google's own, in-production orchestration software, this should not be surprising.

The only tool from the CoreOS stack that is actually required for Kubernetes is etcd, although flannel can be used to support the Kubernetes service discovery with virtual networking. Etcd can be used to share metadata for Docker and rkt containers equally well. However, given the close alliance between Google and CoreOS, Inc., further integration with CoreOS tools seems likely.

Next up

The pace of new tools, companies, techniques, and practices in the Linux container world has been extremely rapid, and it is only through events like CoreOS Fest that I have been able to keep pace. The shifting alliances between companies and open-source projects is constantly changing in a way that we haven't seen since the early days of mobile Linux.

In the next part of this series, we'll be covering systemd and CoreOS, the rise of the language Go as a container tool language, the new projects Calico and Sysdig. We will conclude with an article about the issues and solutions for storing persistent data on container infrastructures, including PostgreSQL Governor, CockroachDB, etcd, and the RAFT consensus algorithm.

Index entries for this article
GuestArticles	Berkus, Josh
Conference	CoreOS Fest/2015

---

(Log in to post comments)