New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposal: Logging drivers #7195
Comments
What about per container driver choosing? For example I don't want logs for elasticsearch, but I want logs for prosody. |
A few questions that I have, what should we do about timestamps? Should the read return some type of structured data or should we still manage this as just steams? Any suggestion and modifications to this proposal is welcome. |
I would still manage them as just streams and allow the implementation to handle it. |
Hm, actually I think that we have perfect interface for logging - io.Writer :) And for none we have ioutil.Discard. |
I encourage being able to attach streams to syslog out of the box, while syslog may not be sexy in 2014, it works everywhere. |
@brianm would you be interested in working on a syslog driver for this initial push? |
Happy to! |
@brianm sounds good to me. I like to have a few different drives so that it keeps that interface honest and makes sure that we are accounting for different needs within the driver. I'm guessing for things like syslog we will need to pass options when we create the driver. Maybe something like: driver, err := syslog.NewDriver("/var/lib/docker/logging/syslog", map[string]string{
"priority": "1",
"socket": "/somepath",
}) |
+1 to syslog driver and config. |
I just added the syslog driver to the proposal |
I am currently evaluating a gazillion way of getting my logs to the right place with docker (app directly forward log, agent in the container, agent in another container, agent on the host, syslog, ...) and having docker logs directly to syslog would solve it easily. All apps could just use stdout/err. As per container configuration, we should be able to give the sender name and the facility. The other possibility is to turn off logging in docker and use systemd or supervisord to forward the logs of each containers to syslog. |
If someone miss my first comment:
|
I can see the need for this to be per container but configuration will be weird on the daemon running multiple logging drivers. -1 on the io.Writer, we need to distinguish stdout and stderr in some of the drivers os we cannot use one interface. the steams are already io.Writers coming in so we are still good. |
I updated the Driver interface to include CloseLog for signaling to the driver that no more logs will be written for a specific id. |
Yup, and I'm totally want possibility to have different drivers for them. |
Should the logging driver support a syslog server option ? |
@markcartertm Are you questioning the idea of including a syslog driver (which is much discussed above and part of the proposal) or did you not see the discussion? |
An implementation that chunked logs by time could be helpful for async log archiving strategies. For example, if logs were stored by minute. I'm not sure how this would interact with the |
A first step that would allow "usual" logging processing system to work is to make docker logrotate compliant. At present there is no way to make docker re-create the log files after a rotation without restarting the containers. |
I think the last question here that needs to be answered is, should this be per container or a daemon wide option? |
@crosbymichael People will want per container with a default set at the daemon level. |
Syslog configuration could be (in order or priority, from low to high):
The configuration at the image level would obviously not include all options (like where to log) but rather what to log (stderr/stdout/both, include timestamp or not or add formatting). |
@kuon we cannot do anything at the image level because it makes images lose portability. Things like this should be host specific/ runtime dependencies. |
On Wed, Jul 30, 2014 at 10:49:07AM -0700, Michael Crosby wrote:
Setting defaults at the image level shouldn't compromise portability. |
@wking all these things you listed are things that happen inside the container. |
@wking Yes, they do when it's something specific about what type of logging drivers you have installed on a specific docker host. The settings in the image are all portable. This is the reason why VOLUMES /home/michael:/root is not allowed in the image because not ever docker host will have a folder structure with |
I guess relying on environment variables (like -e LOGLEVEL=INFO and such) is OK for this use case. It was more an idea than a "thought through" proposal. |
IMVHO I think a syslog plugin for docker is completely pointless. Syslog daemons that run on practically every distribution already support everything that has been discussed here. This comment outlines precisely why I feel this way. This is not meant as a flame, but as an alternative discussion that relieves some pressure off the docker devs and puts more responsibility on the Dockerfile maintainer where in this case I feel (again, IMVHO) it belongs. I am not saying that the logging doesn't need some work; the piece about handling/rotating the stderr/stdout of the container itself is incredibly useful b/c for long-running containers pushing a lot of logs to those pipes results in the issues previously described regarding disk usage. This will at some point need to be solved, though, to cover the bevy of trusted builds that currently send everything to stderr/stdout. configuring syslog output within the containerI find that the following options work beautifully (these should obviously be expanded):
#!/bin/bash
if host syslog > /dev/null 2>&1; then HOST_SYSLOG_DAEMON=$(host syslog | head -n 1 | cut -f 4); fi
_enable_syslog=${SYSLOG:-true}
_host_syslog_daemon="${HOST_SYSLOG_DAEMON:-172.17.42.1}" # perhaps loaded by --env/-e; ${VAR:-DEFAULT} notation sets default if ENV variable does not exist
_unique_proc_name="randywallace/test_syslog_image"
_facility='local0' # or local1 thru local7, cron, user, etc...
_syslog_and_stdout_stderr=${TEE_OUTPUT:-false} # true/false; also could be a --env
# docker run -e TEE_OUTPUT=true -e HOST_SYSLOG_DAEMON=1.2.3.4 -d my_image
# or
# docker run --link my-rsyslog-container:syslog -d my_image
# or disabled completely
# docker run -e SYSLOG=false -d my_image
__logger() {
local LEVEL=${1:-info}
sed -u -r -e 's/\\n/ /g' -e 's/\s\-{3,}/;/g' -e 's/\-{3,}\s//g' |\
/usr/bin/logger -p ${_facility}.${LEVEL} -t "${_unique_proc_name}[$$]" -n "${_host_syslog_daemon}"
}
run_logger() {
if $_enable_syslog; then
if $_syslog_and_stdout_stderr; then
tee -a >(__logger $1)
else
__logger $1
fi
else
if [ "$1" = "err" ]; then
cat >&2
else
cat
fi
fi
}
# Catch all STDOUT and STDERR traffic
exec > >(run_logger info) 2> >(run_logger err)
log() { echo "INFO: $*" | run_logger ; }
error() { echo "ERROR: $*" | run_logger err ; }
critical() { echo "EMERGENCY: $*" | run_logger emerg; exit 1 ; }
alert() { echo "ALERT: $*" | run_logger alert ; }
notice() { echo "NOTICE: $*" | run_logger notice ; }
debug() { echo "DEBUG: $*" | run_logger debug ; }
warning() { echo "WARN: $*" | run_logger warn ; }
log "info"
error "error"
alert "alert"
notice "notice"
debug "debug"
warning "warning"
echo "STDOUT output"
echo "STDERR output" >&2
critical "critical... exiting" identifying the host to receive syslog traffic
Profit
ConclusionSolving the problem of logging is not a new one, and I seriously doubt that docker could create enough plugins, command line options, etc... to satisfy everybody. This is why rsyslog, syslog-ng, syslogd, papertrail, logstash, graylog2, splunk, fluentd, etc... exist. We've already seen this battle start here, and I don't want to be around when the smoke clears. I hope what I've said here, though, may help some of you to come up with your own solutions that could be working today! And, if you have problems with the container's logs getting too full (those that are generated from stderr/stdout), don't send them there at all and use my example wrapper above to get rid of that problem completely! |
For what's worth, I am now using systemd to launch containers and forward logs to syslog, in addition with logrotate in copytruncate mode. This works fine.. I am not arguing in one way or the other, just saying that this setup works today and give per container configuration option. |
@kuon This is good way, but docker internal mechanism of writing logs to stdout is not perfect. So if you write long lines or you have high flow of logs - you will get huge memory and CPU overhead just for writing logs to container stdout. So having native syslog support will be great anyway. |
@randywallace The point is to do something with the collected logs that we already have in Docker. Instead of forcing people to implement something on their own, Docker can provide the facility to do it without having to hack stuff around (like running a syslog daemon inside your container). |
I would like to echo the sentiment of @brianm and @zepouet and maybe suggest that there are really two discussions here: The first one is, frankly, up to @crosbymichael and co. and it concerns how Docker handles an issue with the current implementation of logging. I think the guys are trying hard to come up with forward-facing solutions that will provide paths to new features, and I applaud that effort. The second discussion, however, is being alluded to in previous comments; that is: is it appropriate for Docker to dictate a "logging framework"? No matter how many drivers "we" add, no matter how many options and config files, the tacit assumption becomes "everybody does logging like (or a subset of how) Docker does it". The UNIX way is to do one thing and do it well. In the case of logging, this means let syslog do the work; for rotating logs we have logrotate, etc. I think it would be better to have more intelligent ways to handle mounting and cross-mnt namespace access so that solutions described above like mounting @randywallace provides an example of how easy it is to setup logging already using existing tools. I just think we're not thinking outside the box on how to provide a generally useful solution that also handles this case. All of this is without mentioning the performance issues in containers at high load if the Docker daemon has to handle each packet. In high traffic situations, this is an absolute non-starter. We need to have access to kernel primitives at this point and a multi-layered userspace logging solution is going to force me to disable it and cobble my own each time. As I said above, the more immediate decision is important for handling issues with log management. Of course, that should be solved. But I would hate to feel that Docker as an organization was wasting money and time building stuff I already have. The featureset so far is so out-of-the-box (both in terms of innovation and usability) that I really hate to see such a mundane and already-solved concern become the responsibility of the docker daemon. To head off and forestall any other comments to the effect that "wouldn't it be nice if docker managed your logfiles" let me say that yes, it would. I would also like a built-in webserver so that I can launch my Node.js apps. I just don't feel that beyond the scope of improving the current stdout/stderr system this path leads anywhere but to having a whole group of people disabling docker log management and bending over backwards to use something else. Of course, this all should be taken in the context of an assumption that the goal is to have advanced container technology that does it's thing and otherwise let's you do what you want (ie. maintaining neutrality). If docker is becoming a bit more of an "app hosting platform" where you can fully customize the inside and all the plumbing is handled for you, then this is definitely the way to go. In case code speaks louder than words, I'm willing to work on a PR with a sketch of something if I get at least two people who will read it. |
I reviewed it all i think the only thing that needs to happen is that the log files get rotate able all else will break my existing setups probally and if i need logs from a process i can gather them via the dockerhost on os level via tools i don't need docker to handle that |
My 2 cents: It should be easy to get some default logging (aka something without that huge memory and CPU overhead) and possible (without unnecessary overhead) to integrate your own logging as @randywallace suggested, all that as lean as possible. Therefor I wouldn't try to interprete the logstream and just implement the bare minimal features for the default logging (truncated, rotate and maybe some 'tailing' to get only the last x lines). |
@frank-dspeed You can rotate docker logs with logrotate using copytruncate, see #7333 |
After reading all the comment I still feel that docker needs to do something to simplify logging. Some comments mentioned using Suggestions made by @randywallace don't help me at all unless I'm missing something, many apps don't have the capability to log to a remote syslog, they expect a local syslog device. |
Any news on this? What @tve and others have said is very important for anyone using
Which makes this a very brittle solution... |
Is there a reason I shouldn't just (as of v1.3) if I want to inspect a container by container, log file by log file ?
|
@mhart You shouldn't need to mount the log device. If your syslog daemon on the host is listening on the gateway of the docker bridge (172.17.42.1 by default), this should work just fine, even across host syslog daemon restarts:
|
Graylog2 developer here. Not having much practical experience with Docker yet so I can't go much into Docker configuration specifics but I thought I'd join and leave a few comments based on my experience from building a logging system in the last 4-5 years: Keep it as simple and hand off the logging to for example the local syslog subsystem as soon as possible. Tools like rsyslog or syslog-ng have spent enormous amounts of time to let the user flexibly configure stuff. Simple tasks like choosing the facility are easy to build but you can spent a lot of time implementing different TCP syslog framing methods for example. Do not try to build anything yourself that the popular syslog daemons are doing already. They should be available on basically every platform that Docker runs on. If you want to ship Docker with log management capabilities like basic search, archiving or live tailing then use a log management system that already exists. Graylog2 for example has REST APIs that can be built upon while all the data management is abstracted. Even implementing something like log rotation yourself can go wrong in many different ways and cause OS compatibility nightmares. You will also think about avoiding a Docker log silo that only contains Docker logs. You need to have all your logs (network hardware, OS, applications) in one place for proper correlation, Key=Value pairs are a good way to structure data. There is also GELF, which Graylog2, Logstash, fluentd and nxlog speak. Structured syslog as defined in RFC5424 is probably the most compliant approach but could cause issues with maximum message length. Just my suggestions. :) |
Hey guys, any volunteers to try #9513 patch and provide some feedback? |
I think @LK4D4 said my current proposal here is a little too complex and he is looking into something much simpler. |
@crosbymichael Not very much simpler, but yes :) I'll prepare "proposal with code" |
I think this is closed by #10568. |
Can we configure it to send applications logs to rsyslog of host machine? |
@psquickitjayant by using |
@oncletom Thanks for the information. According to my understanding, we send stdout / stderr logs to host syslog. Possible to send logs from applications running inside container like Apache, cron, Nginx etc.? |
Improved logging support
Topics:
Logging drivers
The driver interface should be able to support the smallest subset available for logging drivers to
implement their functionality. Stdout and stderr will still be the source of logging for containers
in this proposal. Docker will, however, take the raw streams from the containers and create discrete
messages delimited by writes. This parsed struct will then be sent to the logging drivers.
When creating or initializing the drivers they will be provided with a key/value map with the user defined configuration specific to the driver. Each driver will also be provided a root directory where it is able to store and manage any type of state on disk.
Initial logging drivers
none - This driver will ignore the streams and log nothing for the containers. This is a totally valid
driver as the docker daemon has to manage the logs for all container it's a memory and performance bottleneck
on the daemon.
default - This driver will be the current implementation of logs that docker currently has. It is a single
file on disk with json objects with the message, timestamp, and stream of the log message separated by a
new line char.
syslog - This driver will write to a syslog socket and use the tag field to insert the container id.
Default driver improvements
One of the biggest issues with the default driver is that there is no log truncation or rotation. Both of
these issues need to be addressed. We can either truncate based on filesize or date. I believe filesize
is better.
Truncation size can default to 10mb with an option when you select the driver to specify additional options.
Rotation can also be set a specific size limit defaulting to 500mb. To change the defaults I propose a
--logging-opt
flag on the daemon, similar to--storage-opt
for the storage drivers.Usage
The usage for this feature will be managed via the daemon:
The text was updated successfully, but these errors were encountered: