New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposal: docker embedded DNS server #17195
Comments
I agree DNS would be better than mucking (non-atomically!!) with /etc/hosts. However, if we go this route, please make it optional. Some of us already have DNS solutions at a higher-level and don't want this granularity. |
@thockin what do you mean optional? For docker to continue doing what it is today (modifying The only scenario I can think of where modifying |
I mean I don't want docker participating in name resolution neither by DNS nor by managing /etc/hosts - AT ALL. The name that docker has for a container is NOT the name I want reflected in DNS. The stuff that is happening to /etc/hosts right now is terrifying - I am pretty sure it can not possibly be done safely (atomically). I understand how people who are not using higher-level management systems might think built-in DNS is useful. But the higher levels already handle things like DNS with broader scope and visibility. It needs to be "batteries included, but removable" |
Can you point me at an example of this "higher-level management system"? It would be nice if we had something that worked with custom DNS solutions without having to provide another What if we had a flag to control the domain docker responds to queries on? E.G. |
@phemmer Kubernetes runs a real DNS server and populates it with the Kubernetes names of all the running containers. The name that is given to a container is NOT the name exposed in DNS. The container name includes things like a prefix to delineate user-defined containers from system-defined containers, a checksum, etc. On top of that our cluster space is chopped up into sub-domains (namespaces) wherein names are unique, but they are not unique across namespaces - I can have a 'foo' container in the 'a' subdomain and in the the 'b' subdomain. There are more issues with docker doing it automatically, but I won't bore you. The point is that it is a layering violation and we need it off. I think "should manage DNS" should be a docker daemon flag (and should default to false). |
As far as I can see, the |
@phemmer without a real spec to riff on, it's hard to know exactly what you mean :) I am saying that I want Docker doing exactly nothing in this space. It's a waste of cycles and RAM for Docker to muss with DNS (for our use case). |
@phemmer 👍 on the proposal. |
@thockin I don't see an inherent conflict here, depending on the options available to the user. I think it would be very useful in the default case to run a host DNS server exposed only to the containers, have it publish local container names, and configure However, it would also be useful to do the following to accommodate bespoke configurations:
|
@mfischer-zd run actually has |
@thaJeztah Thanks, updated comment |
I've actually started working on this on my own. The version I'm working on maintains the same functionality present today (including |
This sounds like it will be useful for a lot of people. But I still ask that it can be completely turned off. It''s a 100% waste on our machines, unless it is a totally generically programmable DNS server (which I don't think is right for Docker) |
Wouldn't it make sense for the service discovery to be plugin based, like networking and volumes are today? That way you could set up per container what kind of service discovery driver you want (/etc/hosts, host dns, kubernetes dns, custom solution, etc). |
@mariusGundersen We already have plugable service discovery via the |
Some datapoints on this topic: I've been using containers for testing and development starting from the 1.3 release
|
To be specific, what I meant with generically programmable was things like
On Mon, Oct 26, 2015 at 12:16 PM, Akshay Moghe notifications@github.com
|
@mavenugo @thaJeztah I'd be happy to try to contribute some resources to this, but would need to know where to begin. Feel free to drop me a line. |
@mfischer-zd thanks for offering, reading back here, @phemmer was already working on a PoC, so I suggest to help once that PR is created (we try to avoid having multiple "competing" PRs). @mavenugo I know this has been talked about, so I'm not sure if there was already worked on this internally? |
@phemmer let me know how I can help. |
PoC exists here which I submitted to the docker-dev mailing list soliciting feedback on the integration points. Been pretty quiet though. |
@phemmer apologies for that, it's been busy times, but I'll be my usual PITA and try to get some maintainers to have a look at your proposal. (thanks a lot for that PoC, please don't see the "quiet" as it not being appreciated) |
No worries, it's only been 1 working day. I've been around here enough to know it can usually take quite a few :-) |
@phemmer ❤️ |
I like the idea but making it optional is absolutely required (as mentioned by @thockin). It would make sense to me that per network you could set the default value of whether you want this Docker embedded DNS, then also at a container level is nice (basically at this point it's |
A DNS server pointed to by nothing costs little. I don't think it's necessary to have a master shut-off switch. |
Sure, the Docker daemon is already the single largest non-user process (in
|
@thockin Please check your sarcasm at the door. Your opinions and experiences are valuable, but your attitude is not. As for the substantive concerns: First, there aren't really ways to have your cake and eat it too with respect to adding a DNS service into the agent. AFAIK Go doesn't support shared libraries, so in order to gain text pages back by disabling the DNS service, either the DNS service has to be moved into a separate process (this would be a first for agent mode, and would add a significant amount of complexity to it), or it would have to be disabled at compile time. Second, as a practical matter, Docker memory consumption tends to be noise compared to the actual applications run on the system. (My current 1.8.3 RSS is approximately 45MB.) Modern servers are configured standard with at least 1GB RAM (t2.micro) in virtual instances; bare metal to 256GB and beyond. My experience tells me that most users will not miss a few megabytes. People running in extremely memory-constrained environments are more than welcome to run custom Docker builds if they desire to leave out features they don't need. If DNS service can be excluded via a compile-time switch, that would make it very easy to do. |
On Sun, Nov 1, 2015 at 9:05 AM, Michael S. Fischer
First, code pages that are not executed are not faulted in, so that's Running in-process vs out of process is a more relevant discussion.
45 MB of 1GB is 4.3%. That's "just a few megabytes" but it's actually |
You're right about text pages being demand-faulted in, so that's actually a point in my favor. That leaves the following, assuming the server isn't actually being addressed by any clients:
I suggest we actually instrument the impact before debating this further. |
Every page is reclaimable unless you have locked in all your pages into memory using
Why should it consume port 53 in the host name space? That's a question the design should provide but it's not a given. In fact for achieving SD within the
With the static linking of go programs let's not assume that splitting things into many processes is automatically going to save us memory. Remember the kernel has no idea how to share the same exact text pages which are duplicated in every go program because as far as kernel is concerned they are different text pages. Unless KSM is enabled which can get super heavy weight on CPU by it's own right.
In my 2GB linux machine, kernel consumes about 368M of physical memory and this is after dropping all caches. This is aproximately 17% of my total physical memory. So what should we do? |
On Sun, Nov 1, 2015 at 4:55 PM, Michael S. Fischer
Since so many people have DNS solutions, anything > 0 is a waste, but
Good point - it can be virtual.
See above - anything > 0 is pointless for many people with existing solutions. |
On Sun, Nov 1, 2015 at 4:56 PM, Jana Radhakrishnan
Every CLEAN page is reclaimable. Some dirty pages are reclaimable
It means the people who want the functionality spend money and the
No offense, but I have a LOT more faith in the kernel's memory |
A couple of weeks ago I submitted a PR to libnetwork to permit enabling and disabling service discovery on networks - moby/libnetwork#722. It needs design review to decide whether service discovery toggling should 1) be for the whole network or 2) just affect containers (aka services) started after the toggle point. If the decision is 1 (which I'm leaning towards), you can just keep a track of how many networks have service discovery enabled and, if there aren't any, shut down the dns server. There are going to be people who will want the resolution done via /etc/hosts rather than dns, so it'll need to be configurable regardless. |
But it doesn't work reliably. It seems to me that it's better to drop support for it than tell people not to use it when it gets corrupted. |
Agree. We should not offer people solutions that are broken-by-design if On Mon, Nov 16, 2015 at 5:37 AM, Michael S. Fischer <
|
I would prefer to make it an optional feature, it's ok to disable it by default. All alternatives rely on a special service running somewhere, while /etc/hosts is easily consumable by the most basic tools. With the Docker 1.9 overlay network it has become even more convenient. |
Once /etc/hosts is corrupted, it's likely to be corrupted for the lifetime of the container, since updates are edge-triggered. The temporary failure of a DNS server, on the other hand, could be due to any number of issues, and the probability of restoration during the lifetime of a container is much higher. I don't follow your consumption argument -- is there any evidence that users are regularly consuming /etc/hosts manually? My experience is that most applications are just using the standard resolver libraries, in which case they don't really care what the data source is as long as the nsswitch.conf file is correct, which it usually is by default. |
Yes. In order to address all the issues mentioned in the proposal, the clean solution is to cut-over seamlessly to DNS based discovery without impacting the applications. As @mfischer-zd pointed out in most cases applications should not be tied to /etc/hosts based resolution ( @gesellix please correct if this is an incorrect assumption that impacts your use-case ). We have been discussing and designing a proper /etc/hosts replacement with DNS targeting 1.10. Also, thanks to @phemmer's initial proposal (and initial implementation), the discussion is happening in docker-dev. @sanimej is working on a proposal with libnetwork design goals for this requirement. @mfischer-zd please participate in the discussion & also please reach out in #docker-network channel. |
Currently I do use the /etc/hosts, but I would love not to read the file manually. In my use case it's most important to have a stable configuration, i.e. I either need to rely on /etc/hosts being updated by Docker or by configuring a static dns ip address, where the dns server should be updated by Docker as easy as possible (which is what the proposals are addressing?). Now you'll tell me that there exist solutions which listen to Docker events and update a dns server, but when running the dns server in Docker its ip address might change, so in my use case I still prefer the /etc/hosts file. My problem is not that I prefer /etc/hosts, but the need for a stable/static container ip address. Maybe I'm missing something, so if you have any suggestions, I'd be glad. Otherwise I'd prefer if the proposals kept such a use case in mind - I'll try to participate in the discussions. |
I don't agree that /etc/hosts is broken by design - it's broken by implementation. On reflection, if the move to DNS comes with the ability to easily query services (either via I'm interested in how you're planning to allow different backend DNS servers (due to the |
@mavenugo taking off from the discussion on twitter and after having a read through the discuss here https://groups.google.com/forum/#!topic/docker-dev/WXkMiPJqh7I i have a suggestion to make , i may be wrong and may miss something so please forgive my error and correct me if i'm wrong. From what i understand the libnetwork uses a distributed kv store to store node-ip information or host records. This way we don't need to mess with either resolv.conf or /etc/hosts. Not just that when you have this data in the resolver the ns query overhead & latency will probably be lower. Most of the backing kv stores are fairly robust and reliable eliminating potential SPOF that would be incurred by an explicit instance running a dns server. Additionally there is even potential to modify this (perhaps out of the scope of discussion) to provide basic load balancing like yahoo's l3dsr Also as far as the user is concerned he has a prestine hosts and resolv file. We could potentially ensure the nsswith.conf adds this mechanism into its resolver array which very few people modify anyway. |
The problem with that is that go binaries (a popular choice for creating stripped down docker containers) don't look at nsswitch.conf because they're typically statically compiled and can't load the shared libraries. Concrete example of nsswitch.conf being ignored by a go binary we're all familiar with: #1715 I do like the idea. But the current 'docker best practices' and ecosystem of containers means that this probably isn't even an 80% solution. |
@dhananjaysathe Thats an interesting idea. In addition to the issue @aidanhs mentioned it also requires changing nsswitch.conf. Ideally we want a solution to replace /etc/hosts based discovery as transparently as possible. In the docker-dev mail thread I mentioned that libnetwork has all the required host/ip info without any driver dependency. Some of it could be from the distributed KV store (may not be etcd). But for local scope networks distributed KV store is not used. Not sure if that will work for nss-etcd. But I have to take a better look at its implementation |
I am not sure if Docker should have this DNS server ability on board, because I agree there are lots of use cases where you don't want Docker messing with DNS stuff at all. Furthermore, I think it is pretty easy to fix this problem by using: https://github.com/gliderlabs/resolvable. I use it in my local dev environment and it works straight out of the box. |
@phemmer moby/libnetwork#767 explains the proposal for embedded DNS server with the design we discussed last week in docker-dev mailer. PTAL |
@jethroborsje Not sure what you mean by Docker 'messing with DNS stuff'. Take a look at the proposal which has a different design. moby/libnetwork#767 As explained there, the docker embedded DNS server is not a general purpose DNS server and will not interfere with other external name servers. The problem with resolvable is that all the existing deployments have to do things differently for their existing containers to work which we want to avoid. In the current form doesn't look like it will work for multi-host networks. |
https://github.com/docker/dnsserver. I'm a visionary. :P |
The current implementation does not seem to have a CLI- or API-exposed way to disable it. Please add this, at least to provide a get-out option if any bugs surface. |
The same question as above. Do we have some flag or api option to disable the embedded dns? thanks very much. |
I think this can be closed now, because the embedded DNS server was implemented |
There are numerous open issues in regards to docker and DNS handling within containers: (#17190 #16619 #15978 #14627 #15819 and likely many others which I was fuzzy on)
One solution I think which would solve all these issues would be if docker acted as a DNS server. It would answer lookup requests for linked containers, and when the request isn't for a linked container, it would forward it upstream (to the host's name servers).
The
/etc/hosts
file inside the container would then be static, containing only the container itself.We could also not touch
/etc/hosts
at all, and leave the container's entry to DNS. This would allow image builds to manipulate the file and persist the changes.For performance, it would probably be good if docker cached the upstream DNS records. Records come back with a TTL, so docker should cache the record until this TTL expires.
The text was updated successfully, but these errors were encountered: