Imagine if instead of individual physical servers, we could aggregate all the resources in a data center into a single large virtual pool, exposing not virtual machines but primitives such as CPU, RAM, and I/O?  In conjunction, imagine if we could break applications into small isolated units of tasks that could be dynamically assigned resources from our virtual data center pool, based on the needs of the applications in our data center? The analogy here would be a PC with an operating system that is pooling the PC’s processors and RAM and coordinating the allocation and deallocation of  those resources for use by different processes.  Now extend that analogy to make the data center the PC with Mesos as the operating system kernel.  That, in a nutshell, is how Mesos is transforming the data center and making true SDDC a reality.

A short history of Mesos may help set some context for the rest of this and upcoming blog posts.  You can trace the genesis of Mesos to a home-grown data center resource manager, created by Google, called Borg.  You can read more about the origins of Borg and it’s impact on Mesos in this article by WIRED Magazine.  Google’s use of Borg became the inspiration for Twitter to develop a similar resource manager that could help them get rid of the dreaded “Fail Whale.”  They latched on to a project being worked on at the University of California at Berkeley AMPLab called Mesos.  The UC Berkeley team working on the project was led by a PhD student named Ben Hindman who eventually went to work for Twitter and led the development and deployment of Mesos at Twitter.  Today, Mesos manages the placement of workloads for over 30,0000 servers at Twitter and the Fail Whale is a thing of the past.  Mesos is also deployed at companies such as Airbnb, eBay, and Netflix.

How does Mesos allow companies, like Twitter and Airbnb, scale their applications through more efficient management of their data center resources?  It begins with a rather simple but elegant two-level scheduler architecture.

The modified diagram above from the Apache Mesos website shows how Mesos implements it’s two-level scheduling architecture for managing multiple types of applications.  The first level is the master daemon which manages slave daemons running on each node in the Mesos cluster.  The cluster consists of all servers, physical or virtual, that will be running applications tasks, such as Hadoop and MPI jobs.  The second level consists of a component called a framework.  A framework includes a scheduler and an executor process, the latter of which also runs on each node.  Mesos is able to communicate with different types of frameworks with each one managing a different clustered application.  The diagram above shows Hadoop and MPI but other frameworks have been written as well for other types of applications.

The Mesos master works with its slaves to determine each node’s available resources, aggregates all reported available resources across all nodes, and offers them to frameworks that have registered with the master as a client.  Frameworks can choose to accept or reject resources offers from the master based on the requirements of the application using that framework.  Once offers are accepted, the master coordinates with the frameworks and the slaves to schedule tasks on the participating nodes and to execute them in containers so that multiple types of tasks, such as Hadoop and Cassandra, can run simultaneously on the same nodes.

I will be diving into the details of the Mesos architecture and workflow in future blog posts.  But this two-level scheduler architecture and the algorithms and isolation techniques that Mesos uses to enable resource sharing across multiple types of applications on the same nodes is, in my opinion, the future of the data center.  As I wrote earlier, it is the best readily available technology I’ve seen so far to deliver on the promise of the SDDC.

I hope this introduction was helpful and has whetted people’s appetite to learn more about Mesos.  In upcoming weeks, I’ll be diving deeper into the technology, giving folks some ways to get started, and talking about how to get involved in the community.

I dig deeper into the technology here, delve into persistent storage and fault tolerance in Mesos here and go under the hood of resource allocation in Mesos here.  You can also read my post on what I see as going right with the Mesos project.  If you are interested in spinning up and trying out Mesos, I link to some resources in another blog post.