Open and Invisible: A Glimpse at the Future of Infrastructure

LinkedIn’s mission is to connect the world's professionals to make them more productive and successful. I love the way that mission applies to the way we think about technology - from our philosophy governing open source, to our vision for frictionless data infrastructure. One of the reasons why this mission resonates with me is that, as a lifelong engineer, I enjoy taking a process or solution that was once time-consuming and onerous and making it seamless and easy.

Ask any software developer about their biggest pain points, and it’s likely that the acquisition, expansion and maintenance of well-built data infrastructure is high on their list. At many tech companies, creating the data infrastructure to support a new product or feature is one of the main bottlenecks to moving faster. It’s frustrating.

Until very recently, if you wanted to build a new product at LinkedIn, getting the data layer ready was a multi-step process. First, you needed to engage with the data infrastructure team to talk about the characteristics of your product in terms of size of the dataset, number of queries per second, type and complexity of such queries, and how these parameters would evolve over time. You would then have to work in collaboration with this team to provision the cluster and deploy it across our different fabrics, from test to production. This type of high touch engagement is time consuming and does not scale as the size of our application development team increases.

About a year ago, we kicked-off a project called Nuage (French for "Cloud"). Nuage is a service that exposes database provisioning functionality through a rich user interface and set of APIs. Through this new user interface, developers can specify the characteristics of the datastore they want to create, and Nuage will interact with the database system to provision your datastore on a pre-existing cluster. Of course, the user interface and gateway to interact with the underlying database system is only one piece of the puzzle. The underlying database system needs to support multi-tenancy and needs to be elastic such that it can expand its capacity automatically when the load on the system increases.

Developers can now automatically provision and deploy small to medium size datastores through Nuage across all fabrics, from dev to production. But that's just a small step. Based on this early success, we have developed a long term vision which is to build Data Infrastructure that works so well that it is Invisible. Characteristics of such 'invisible' infrastructure include:

• It is ultra-simple to use and it takes little or no time to setup the data layer for the application you’re building.
• It is elastic and scales automatically. New nodes are being provisioned automatically and as needed. 
• It has no single point of failure, is highly available, and fixes itself.
• It is highly operable such that hundreds of thousands of nodes can be managed by a handful of administrators.

In this vision, we’re treating data infrastructure with as much reverence and care as any product we’d release to our members. We want to operationalize this vision across our entire set of data systems: storage solutions such as Voldemort or Espresso, data streaming solutions such as Kafka, or analytics systems such as Pinot.

I feel privileged to pursue this vision with a world-class team of engineers. By building invisible infrastructure, we are making application developers more productive and successful, and we are helping LinkedIn fulfill its mission.