Kenneth Hui

Is Persistent Storage Good For Containers?

Listing-Cargo-Ship

The value of persistent storage for containers has been something I’ve been thinking about so I thought I would raise a couple of concerns I have and see what folks thinks. In particular, I am thinking of people who’ve thought through this topic and who are smarter than me, like Clint Kitson from EMC {code}, John Griffith from SolidFire, Ryan Wallner from ClusterHQ, and Shamail Tahir from IBM.

For folks who are new to the topic, let me give some quick background. Containers are fast becoming the standard unit of deployment for cloud-native applications that follow the design patterns of a Twelve-Factor App. Two of those patterns talks about stateless application processes and about backing services. The idea is that a container, which runs an app process, should be stateless and ephemeral with data that need to persist written to a backing service such as a database on a storage volume. But even that backing service should be treated as an attached resource that is loosely couple to a container and can be swapped out at any time without requiring major changes to an application. Conventional wisdom, therefore, says that containers should avoid the use of persistent storage for backing services whenever possible and even when persistent storage is used, they should be loosely attached and should not require storage that is difficult to swap out. Examples of the latter would include object stores which are accessed through a URL and not mounted in a tightly coupled manner. Further, given the ephemeral nature of how containers should be used in a cloud-native app, persistent storage should not need to be backed by solutions with rich data and infrastructure resilient services. Finally, cloud-native apps should be designed to scale-out which is difficult to achieve if containers have to be attached to storage that is tightly coupled to the container host and not built to scale out. Much more can be said here, but I’ve leave that to future posts.

Despite this conventional wisdom, there persists (pun intended) a desire to bring persistent storage to containers, particularly transactional storage such as block volumes. The reasons for this varies. In some cases, an application needs data to persist and its performance requirements can not be met through backends like objects stores or network file systems; typically, this is a SQL database like MySQL or Postgres that isn’t designed to scale out in the way a NoSQL database might. In other cases, a company that is moving to containers and cloud-native apps may have a desire to leverage existing technology when possible, such as a storage array. To address this need, several open source solutions have surfaced including:

To find out more about these solutions, use the provided hyperlinks to read the relevant documentation.

So what are my concerns with persistent storage for containers? There are two I want to bring up – one architectural and other behavioral. For the sake of brevity, I will keep my arguments short at the risk of leaving some points out.

  • Architecture – I’ve often said that spinning up a container is not a particularly interesting problem; the real challenge comes with managing containers at scale. In a small environment with only tens and hundreds of containers, persistent storage poorly architected may not become an issue. But as environments grow to thousands and tens of thousands of containers, this is not the case. When a large number of containers are ephemeral and constantly moved or re-instantiated on new container hosts, reliance on persistent storage that is tightly coupled to specific container hosts greatly inhibit scale and resiliency. Imagine building a large scale cloud-native application using container hosts that rely on SAN storage for persistence; your ability to scale is now limited to how many servers can be attached to the same SAN storage volumes. Or even if you use a shared file system like ZFS for persistence, the time required to move a volume from one container host to another is typically longer than the time it takes to spin up a new container. This creates an impedance mismatch because the need to spin up an app quickly in a container and the time required to mount the persistence layer.
  • Behavior – I could also argue that reliance on traditional storage paradigms and solutions enforce anti-patterns for cloud-native designs. I’ve already seen that in cases where some users have ported entire application servers, including their dependence on traditional storage, form virtual machines to containers and then wonder why they are not getting all the benefits of containers. By offering more persistent storage options, are we helping users to avoid following cloud-native design patters, such as Twelve-Factor App and enabling them to avoid the future? As I’ve often said, every architectural choice comes with constraints that have to be observed. We shouldn’t imply to users they can use old design patterns and still get all the benefits of containers with cloud-native applications.

Much more could be said but let me close with this… I am not binary when it comes to technology and don’t believe we should get rid of all persistent storage options. I agree that persistent storage has a place for containers, but only if done right to address the issues that are present and if we are willing to educate and to help users understand their constraints. Sometimes that means saying no to a storage solution if it “technically’ works because it doesn’t adhere to correct design patterns. Maybe it means requiring all persistent storage for containers to follow sound cloud-native design patterns. At the very least, it means educating user on the pros and the cons of persistent storage solutions.

Hopefully, this will start a healthy dialog on the issue. But now that I’ve had my say, Clint, John, Ryan, Shamail, and others, what say you all?

9 comments

  1. Nice post Ken! I have to say this is an interesting one for me, and you and I have yet to have a chance to really sit down over dinner and have a great debate over this (yet).

    So I do think that persistence in the form of Data-Volumes that are as you put it “loosely coupled” and easily moved throughout a Cluster is a key and critical component even for Cloud-Native apps. Data-Bases are always the most popular example, and honestly I think that’s because they’re the most appropriate. There are others but the point is that Persistence of some data is important and I’d even say necessary.

    The problem in my opinion is things can easily go too far. The Docker Volume API today pretty much gives everything I’d expect/want. The problem is that not all storage backends should be used in these environments and when you try and shoe-horn every vendors semantics and expose their particular strengths/features you end up with a watered down mess of things that in reality nobody needs and hurts the whole cloud-native paradigm. I do think that with the *right* storage you can in fact automate, scale and do all the cool things that cloud-native talks about. Where we run into trouble is when you try and fit the square peg into the round hole, and where you insist on throwing in every feature including the kitchen sink, that frankly nobody needs.

    I think that basic data-volumes are a great add, and important. You left my work off the list, and I’ll refrain from self-promotion here 🙂 I think the key here though is to offer something innovative and different, not water down Containers with the same old busted up garbage.

    Reply

  2. First off, great post Ken, glad to be a part of the conversation. I like to start by saying that Yes, data and storage portability with containers is a beneficial building block but requires deep integration with the containerizing layer and OF layers the we’re only just starting to begin to see. Not a surprising nor detailed view from myself, but hey it needs to be said.

    Few comments on a few items to get the conversations going. (Disclaimer: these comments are solely my opinion on the post and are aimed at churning a good dialog up with readers and the author.)

    > “The idea is that a container, which runs an app process, should be stateless and ephemeral with data that need to persist written to a backing service such as a database on a storage volume. “

    This idea is a strong one, and can really hold true in many cases. Micro services and like architectures (futuristic SOA, whatever you may call it) can be build in pieces and layers. This means you can still design your layers into stateless services but your not going to avoid needing persistence. Therefore, theres no reason in my mind why you should need a separate, off-to-the-side service to manage your persistence (stretching here on purpose), getting a seamless and cloud-native solution should be integrated, and persistence should be a first class citizen to the app and not to the side.

    > “Conventional wisdom, therefore, says that containers should avoid the use of persistent storage for backing services whenever possible and even when persistent storage is used, they should be loosely attached and should not require storage that is difficult to swap out.”

    I really agree here, but again you WILL in fact need it at some point. In short, times are changing and so are the applications and architectures, it will take a technology and mind shift to adopt this model correctly, though OFs are getting good at allowing this type of workflow, like Kubernetes, storage can be an easily discoverable service to the app.

    Also a quick not on stateless vs stateful, this sometimes can go down a rabbit whole but FWIW the container itself can remain ephemeral even when its acting as part of your persistence layer.

    > “Finally, cloud-native apps should be designed to scale-out which is difficult to achieve if containers have to be attached to storage that is tightly coupled to the container host and not built to scale out”

    Sure, and I agree with you here, but microservice architectures call for the separation of responsibilities at the organizational and technological levels. This means we can have layers of our applications architecture that scale very well in high demand, and also provide ways to scale out data services along side it, even though it may be harder to achieve.

    > “a company that is moving to containers and cloud-native apps may have a desire to leverage existing technology when possible, such as a storage array.”

    This is that “middle ground” or “chasm” (as Shamail and I have called it), the limbo in which a pool of infrastructure resources exists together and must co-exist well enough on the road to truly cloud-native if thats what your after. On the architecture, your never going to truly get past the issue of the impedance mismatch of the time it takes to create storage, unless said storage pre-exists, and even then your still bound by the time to attach/make it available which could be very minimal or longer than you want. This of course taking into consideration that your storage is part of the same workflow. To have a separate workflow, separate commands, separate pipeline for provisioning/setting up backing services, again is almost the same issue, less your application is now more separated and the data workflow isn’t a first class citizen to the app. (mind you, this can work just fine, not bashing but arguing the point of a single workflow for the developer)

    > “I could also argue that reliance on traditional storage paradigms and solutions enforce anti-patterns for cloud-native designs”

    I agree with you here, and this partly goes back to my comment about the middle ground that we’re in. Truly adopting cloud-native means a mind shift with developers and organizations, which doesn’t happen overnight and isn’t cheap for companies with expensive equipment sitting in their datacenter already. Storage itself needs to be re-thought, but in the meantime integrations are needed and some will last, but will not be the end-all but does seem to be an anti-pattern. Another layer of intelligence will need to be added for persistence to be truly as flexible as these architectures, how defining “as flexible” is another matter in itself.

    Cheers!
    R

    Reply

Leave a comment