New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remote drivers are (wrongly) assumed to be global #486
Comments
It may be difficult to do this without modifications to the plugin subsystem. My best guess at doing without that is to have two kinds of driver; I agree that this is a gap; but I wonder if it is exactly what e.g., kubernetes needs. What assumptions does libnetwork make about locally scoped drivers? |
@squaremo i dont think we would need to change the plugin subsystem for this. We could add an explicit call for capability negotiation before having to register the plugin with libnetwork. If it is a locally scoped driver, then libnetwork will not distribute the network or endpoint information or require a KV store to back these guarantees. |
For example, Kubernetes endpoints are not going to be used across hosts. On Wed, Sep 2, 2015 at 4:03 PM, Michael Bridgen notifications@github.com
|
@thockin as we discussed, this is a good reason to be a local-scoped driver. |
@mavenugo On host-1: My driver currently receives just a uuid. On a different host, if I run: docker service publish my-service.foo I will likely get a "foo" network not found error. What do you have in mind for a locally scoped driver? Can we get names instead of UUIDs? |
Also I think a locally scoped driver will work if commands like "docker network ls" ask the local driver to list networks instead of trying to list UUIDs on its own. The local drivers can provide back UUIDs and names back which is listed by docker. So in theory the local drivers do the job of libkv. |
The suggestion from @squaremo sounds sensible to me. The plugin already has to do a handshake to establish its capabilities so adding a different capability for "local" plugins sounds like a good idea |
Is "On a different host" compatible with a locally scoped driver? On Wed, Sep 2, 2015 at 4:29 PM, Gurucharan Shetty notifications@github.com
|
@thockin May be if you give an end to end workflow of a locally scoped driver that you have in mind, I will know better. |
@shettyg That's an interesting thought. You're treating locally scoped drivers as a way to sidestep multi-host parts of libnetwork. Which as you point out, only works if libnetwork defers more control to the driver. |
@shettyg locally scoped driver will make sure libnetwork doesnt synchronize the networks, endpoints and hence yes, it is upto the orchestration system to determine how this is handled across multi-hosts. |
Another thought. Isn't a locally scoped driver same as starting docker daemon with a libkv store that is local only? |
@shettyg @squaremo @tomdee i dont think having another plugin endpoint is a good idea. This is a property of the network driver and must be honored as such. Introducing another plugin type will call for more changes to the libnetwork core to look for more plugin types, when the functionality provided by the driver is the same. Hence my suggestion is to add a capability negotiation in the Plugin API & exchange this info, |
Not in general. Because each host will assume it's acting only locally, it will cons a new UUID for a network it hasn't seen on that host. If "LocalScope" is being used to mean "let me do my own co-ordination", this is going to fail to do the expected thing. |
@squaremo correct. bridge driver today is a localscoped driver and it doenst depend on the distributed states. Same I think will work for k8s and other drivers such as macvlan, ipvlan plugins. |
@mavenugo Fair point about plugin types; minor shame to have another handshake exchange, but I agree it is better overall. |
Right; this leaves systems that do their own co-ordination high and dry, unfortunately. |
@squaremo Are you looking for a notion of cluster to be provided to the drivers by libnetwork? |
This wouldn't help, since it would still be the case that drivers only see UUIDs, and these are constructed assuming that Docker's is the only own shared state. So I would lean towards giving the drivers the information they need to do their own co-ordination, which pretty much means the user-supplied names. |
Yeah, this would be better. I can then have kubernetes orchestrate each On Wed, Sep 2, 2015 at 5:11 PM, Michael Bridgen notifications@github.com
|
@thockin k8s can still do that. If you are planning on using local-scope driber and if the network is created by k8s, it has the mapping between the name <-> network-id across all the hosts. |
If I use local-scope driver, won't every node have a different network ID On Wed, Sep 2, 2015 at 5:26 PM, Madhu Venugopal notifications@github.com
|
@thockin yes. it will be. just like docker0 bridge today which is different in each host. |
I don't want Docker to try to manage it globally because we have our own On Wed, Sep 2, 2015 at 9:40 PM, Madhu Venugopal notifications@github.com
|
@thockin can you please help explain what you mean by |
If this works, then I would not have to think two ways to implement drivers when it runs as a plugin in Kubernetes vs natively as remote driver on libnetwork. Assume that I have KV store available. I imagine by |
Sorry, that came off snippier than I meant it. My primary key is the network name. That's the key my API knows (or will know). That key exists before I ever call "docker network create". I don't want docker to try to manage my driver globally because I already have a global control plane. So I have to use a local driver. Because I have a local driver, every node is going to "docker network create" and get a different UUID. When we join a container to a namespace you are only telling me the UUID. My driver can not use that UUID to look anything up in my own API. When I start offering multiple Networks, this just gets worse. I have to make my node-agent keep MORE side-band state that maps the UUID (returned from 'docker network create' right?) to network name and then publish that state to my driver. I can probably do that, but surely you see how this is a terrible hack just to work around docker. I know other people have asked for network name - why is docker stonewalling the community on this seemingly tiny thing? |
The current side-band hack to retrieve network name is made worse by the fact that the docker+libnetwork remote driver API is synchronous, so the driver cannot query docker during a driver operation or docker will deadlock. Instead the remote driver must cache the UUID and then somehow right after the CreateNetwork() hook request the network name from the docker API. That's obviously racy since there's no guarantee that docker will receive the ListNetworks() request from the driver before libnetwork calls the driver again with CreateEndpoint() or some other call. |
+1 |
Where is the place to make them work, @mrjana? |
@squaremo As I've mentioned in my other posts, the place to make them work is north of docker/libnetwork api, very similar to what k8s is trying to do. Driver api is definitely not the place to expect a gateway to interface into control plane. Control plane doesn't even belong in that layer. Once you have control plane integrated on top of docker api you can have the driver portion of your solution to be plugged into docker using driver api |
Most of the problems @thockin & co have been discussing are real problems I have been working around too. The network name, in particular, should be propagated to plugins. I have experimented with a plugin-per-network, plugin-per-tenant-per-network, plugin-that-calls-back-into-docker, etc. None of these solutions are good, even if they all do technically work. Labels and options get us some of the way but we're still playing "fit the square peg into the round hole", looking for a loophole to push a network name in. Docker should formally clarify the expectations of network plugins from a capability perspective, so we can design one that fits your model. Either that, or open up the capabilities of libnetwork to plugin authors so they can become more than what the creators expect. I personally favor the latter. :) |
@mrjana while it is natural for an orchestrator like Kubernetes to live north of docker, I struggle to see how a pure network implementation could have its control plane do anything north of the api. Can you expand a bit more how that would work? Disclaimer: I work on weave Net. |
AFAICT everybody who is trying to integrate their networking tech with Docker really wants the former, i.e. integrate both their data plane and control plane. AND they want to do so in a way that fits in with the concepts and UI Docker provides for networking, rather than opting out of that completely. |
@bboreham When I mentioned "control plane" I was referring mostly to the management side of the control plane i.e a control plane which has it's own UI/API and has different constructs and abstractions than Dockers Networks/Endpoints etc. For such a control plane, they most certainly need to sit on top of docker and treat docker as a low level ingredient. For control planes who want to use Docker's Networks/Endpoints abstraction there is no problem, until those control planes try to do things like advertise their drivers as This was mainly a response to somebody suggesting sending The only thing that can happen in the future is provide a remote api extension point, very similar to ClusterHQ's powerstrip solution if the ecosystem wants management plane hooks. But that's where it should happen if it ever happens and the shape of such an extension point is yet to be determined. |
@rade Please see my previous response |
So I'm back from vacation and I'd like to summarize my understanding of the discussion so far WRT kubernetes. It seems like @thockin and k8s would be happy if:
Does that sound right to everyone? |
@mrjana As I understand it, "GlobalScope" really means "use Docker/libnetwork control plane", right? A driver that has its own control plane should currently advertise itself as LocalScope, which is where the confusion stems from. That driver is actually "global" scope, since it has its own control plane and could be orchestrating multiple instances of docker across nodes, just that docker has no knowledge of that. So it would be clearer to just deprecate the Scope type, and instead add a new set of flags to the Capability type with one currently defined value: UseLibnetworkControlPlane. The 'overlay' driver would set this capability, and remote drivers could opt-in to it as well. All other drivers simply pass nothing for this flag. If that makes sense I'll do a PR for it? |
First I'm sorry that I didn't read all your comments so I may miss something of your talking. I'm doing some work on Libnetwork and find a possible solution for this, the related PR is already pushed. The code maybe need a little re-organised, but let's first see if this solution is OK for you. |
@WeiZhang555 thank you for actually pushing a PR to get it resolved :-) Have added comments with my opinion. PTAL. |
@mavenugo would you object to changing the Scope definitions into just a single capability for whether or not the plugin wants to use the libnetwork control plane? I think that would remove some of the confusion we have here. |
@dcbw I would say that strictly speaking, local-scoped drivers still use libnetwork's control plane. e.g. |
@lxpollitt I'm not sure I'd agree with "clear meanings". The meaning for GlobalScope is:
which is certainly true for drivers that do this outside of docker's control plane, yes these drivers have to advertise themselves as LocalScope. Perhaps just updating the API doc to specify that GlobalScope explicitly opts the driver into the docker control plane would be enough to clarify. |
It isn't a deal breaker - we can force users to pass that redundantly as a I don't like pushing boulders up hills, so I am going to stop assuming that On Thu, Sep 10, 2015 at 9:03 AM, Dan Williams notifications@github.com
|
The labels hack does not work in all cases. Specifically it wouldn't work for dynamically attaching containers to a network, since labels can only be set at container creation time. |
I agree with you that the functionality provided by the driver is the same, but why would it call for more changes to libnetwork core? I have tried to bring it to reality and found that it doesn't need too
Do you mean another negotiation after handshake or make negotiation during handshake? Does this need to change the whole plugin mechanism ? |
@WeiZhang555 yes, the current changes are small. but am more worried about the long-term impact and the correctness of the design. once we start to treat it as a different
can replace today's code that statically defines the capability as GlobalScope.
just before invoking the dc.RegisterDriver here : https://github.com/docker/libnetwork/blob/master/drivers/remote/driver.go#L33 Do you think we can get that done ? |
@rade I don't think anybody is talking about container labels here. These are network labels attached to network objects and created during network creation time. |
@mavenugo Got your point, you give a really clear and detailed illustration, I'd like to take another shot if you still need this, I think it won't take much time to get that done. :) |
Absolutely. Please go ahead. -Madhu
|
closing this issue via #516 . Thanks @WeiZhang555 for the patch & everyone for a good discussion. |
I read it all and very thankfully. Now, I'm a little less ignorant on how moby was born. 😉 |
https://github.com/docker/libnetwork/blob/master/drivers/remote/driver.go#L32
It should be possible to write local-only drivers.
The text was updated successfully, but these errors were encountered: