ping @ibuildthecloud

Does this need to apply to other things too - like volumes, networks, etc? I would think a "system" volume might want to be hidden from normal users in the same way.

Its too bad that we don't have the notion of users as this is starting to head down the path of scoping the visibility of resources based on something and something like a user's ACL would be the most natural thing to use here.

If all we want to solve is this one problem then I like this solution, its nice and easy, but I think it might be worth exploring what people will naturally want to do next and make sure this one boolean flag is really enough, otherwise it might be deprecated pretty quickly.

Yes, a problem exists. But your flag trying to solve two problems at once: order and accidental removal. Fixing them separately or, at least, having the flexibility to turn one without other would be a great improvement too.

I could see modifying this to not deal with automatic filtering. As I was typing it out I was thinking it may be too much.

I tend to agree with duglin on this one, I think user ACL is the most natural way to deal with this. For example in Swarm, there would be an admin/maintainance account to deal with the Swarm containers and deploy more nodes/launch more agents and managers. This would ensure that other regular users cannot even list with docker ps -a and/or delete those containers by any mean.

This does not deal with the issue of starting containers in a specific order but with this basic construct then we can go on and have for example a system user for system containers that would have the priority at daemon startup.

The flag does solve the simple case of a set of containers to start before regular containers but what if I want to have a dependency graph and also start system containers in a specific order? ACL could solve that by giving priorities at daemon startup to users/system services (even though this solution is far from being ideal from a usability perspective, just the first thing that popped in my mind 😄)

What should be the result if a system container cannot start at all, or repeatedly starts and fails?

(I have discussed this at moby/libnetwork#813)

If the answer is "Docker should continue to attempt starting the system container and not let you do anything else before it succeeds", then there needs to be some way of defeating this, since you need to use Docker to change the configuration.

@bboreham In that case for a first step I'd just log the error as we do today and keep going.

@cpuguy83 This is not what happens for network plugins: the operation is retried several times, often enough to time-out the startup and leave you with nothing.

@bboreham That is how plugins work, the above proposal doesn't change that.
Container startup != plugin communication, and something that is --system would not necessarily be a plugin.

@cpuguy83 ok, thanks for clarifying.

Makes sure the container will startup before non-system containers

Can I just clarify some more: your proposal is to change the order in which the fork/exec operations are done, or to do something extra to establish that the system container has actually started, perhaps even to know that it is "ready"?

@bborehem neither... It just attempts to start them before normal containers.

I think your "attempts to start" is what I meant by "fork/exec"; the important part is that it doesn't give you any guarantees about the order in which those processes will then be scheduled by the kernel, and particularly no guarantee that the system containers will be ready before the non-system ones need them.

Right?

Docker provides the odering.
In terms of "readiness", that's up to the services which consume them, not Docker.

+1

• docker rm behavior: accidental removal of these containers can be damaging. Say, I want to remove all app-containers, for which I am used to doing docker rm -f $(docker ps -aq) - it might also remove all non system/infra containers. Perhaps the long-term answer would be to have ACLs as suggested by @duglin. It would be best to not show these containers in docker ps output all together unless explicitly asked for.
• starting a infra container later: it is best if order behavior is decoupled from whether it shows up in docker ps, etc. because I can think of starting a system/infra service afterwards (like how I can do it with systemd)

So, in summary -- what we really need is a way to specify that a container should start before any other containers.
What this also implies is that these "system" containers cannot be linked to non-"system" containers.

One thing to keep in mind is that the entire notion of ordering is one that we should probably discourage people from even worrying about. At any time any component can do down and users of that component need to be prepared to deal with it - whether its pausing, retrying, or something else. So, to me dealing with a dependent component going down/restarting isn't much different than me dealing with it being started after me. Providing a way to specify ordering could be just giving people a false sense of security.

I tried to get this across earlier but I think I failed:

The order in which you start some processes gives no guarantee over the order in which they will subsequently get CPU time in order to achieve anything. That is up to the OS scheduler, and it may have other things to think about.

It seems to me unwise to add a feature which usually does what you want but sometimes doesn't.

I agree, and in cases where these are docker plugins that plugin loading is handled correctly.. ie. containers that rely on a plugin resource are not delayed anyway until the plugin is available.

I'm going to close this as I think we can handle this in other ways.

As mentioned at moby/libnetwork#882, there is a mirror-image problem at shutdown - you shouldn't shut down a container that is implementing some "system" feature before shutting down a container using that feature.