Allow setting ulimits for containers #9437

cpuguy83 · 2014-12-01T18:44:13Z

Add option to set ulimit settings for containers.
Currently the container inherits from the daemon's ulimit settings.

Support for this was added in libcontainer here docker-archive/libcontainer@7ce34f5

thaJeztah · 2014-12-01T18:49:21Z

docs/sources/reference/commandline/cli.md

+
+```
+    $ docker run --ulimit=nofile:1024:1024 --rm debian ulimit -n
+    1024


Unintended wrapping?

No, that would be the output of the run command.

Ah, sorry, mis-interpreted, I thought -n 1024 😄

crosbymichael · 2014-12-01T18:59:15Z

I like it LGTM

SvenDowideit · 2014-12-02T01:44:46Z

Doc LGTM - @fredlf @jamtur01

it might be good to add what happens if you try to set a limit in the container that is larger than that on the host.

cpuguy83 · 2014-12-02T01:46:21Z

@SvenDowideit I think it doesn't matter since Docker is setting the limit, which has full host access.

icecrime · 2014-12-02T05:58:19Z

docs/sources/reference/api/docker_remote_api_v1.16.md

@@ -150,7 +150,8 @@ Create a container
               "CapDrop": ["MKNOD"],
               "RestartPolicy": { "Name": "", "MaximumRetryCount": 0 },
               "NetworkMode": "bridge",
-               "Devices": []
+               "Devices": [],
+               "Ulimits": {},


Sorry for the super nitpicky remark, but I don't think the trailing comma is legal JSON.

fredlf · 2014-12-02T23:47:37Z

docs/sources/reference/commandline/cli.md

+
+>**Note:**
+> If you do not provide a `hard limit`, the `soft limit` will be used for both
+values. If no ulimits are set, they will inherit from the default ulimits set on


"...they will be inherited"

Check to make sure you are consistent with code style for all instances of ulimit.

fredlf · 2014-12-02T23:55:20Z

Docs LGTM, though I would like that one point clarified, ideally. Thanks.

cpuguy83 · 2014-12-03T18:34:53Z

@fredlf Updated.

jamtur01 · 2014-12-09T12:08:59Z

LGTM

fredlf · 2014-12-10T00:33:14Z

LGTM

fredlf · 2014-12-10T00:35:54Z

@cpuguy83 I'll merge if you can address the conflicts.

lexinator · 2014-12-15T22:41:28Z

sorry i'm doing a flyby here, @fredlf this may be of interest to you, http://blog.spreedly.com/2014/06/24/merge-pull-request-considered-harmful/#.U6w1vY1dVDs

SvenDowideit · 2014-12-16T02:31:35Z

@lexinator doesn't really apply, as this project only uses the merge-pull-request-button - and if you're involved enough that its a problem, you can request a change to the project's processes via PR :)

we also have a policy of taking over and updating PR's for contributors - but @cpuguy83 is a pretty core contributor, so :)

icecrime · 2015-01-14T01:05:17Z

docs/sources/reference/api/docker_remote_api_v1.16.md

@@ -232,6 +233,9 @@ Json Parameters:
  -   **Devices** - A list of devices to add to the container specified in the
        form
        `{ "PathOnHost": "/dev/deviceName", "PathInContainer": "/dev/deviceName", "CgroupPermissions": "mrw"}`
+  -   **Ulimits** - A list of ulimits to be set in the container, specified as
+        `<type>: { "Soft": <soft limit>, "Hard": <hard limit> }`, for example:
+        `Ulimits: { "nofile": { "Soft": 1024, "Hard", 2048 }}`


I think you need to revert changes to v1.16 and add apply them to v1.17 now.

icecrime · 2015-01-14T01:09:51Z

Talked with @crosbymichael: we believe this would get merged if you submitted an implementation. Do you want to close this one and make a new PR when ready, or add new commits here?

jeremyeder · 2015-01-14T13:02:58Z

I plan to test this when there's code.

cpuguy83 · 2015-01-14T14:14:53Z

Working on it. I'll put commits here.

cpuguy83 · 2015-01-14T22:43:53Z

Ok, it's done, tear it up!

Note, I changed the the HostConifg->Ulimits field to be an array instead of map.

philips · 2015-02-26T03:19:44Z

@cpuguy83 makes sense, I see it in the docs now, no idea how I missed that. Thanks.

jessfraz · 2015-03-04T11:59:50Z

what is this waiting on?

Allow setting ulimits for containers

liquid-sky · 2015-04-22T11:19:22Z

How can one provide RLIM_INFINITY value as per resource.h or equivalent of providing unlimited value when running ulimit for a resource?

cpuguy83 · 2015-04-22T11:22:43Z

@liquid-sky it is unsupported. There really isn't an unlimited, it's just a really high value.

frol · 2015-04-23T09:17:44Z

UPDATE: It seems that ulimit subsystem is not namespaced and applies limits to all processes based only on UID/GID.

Original post:

I cannot figure out why it happens, but it seems that resources that are limited by ulimit are counted across containers. Here is an example:

If I run one container with --ulimit 'nproc=2' it runs fine, but I cannot start the second container with the same --ulimit 'nproc=2':

Terminal #1:

$ docker run -it --rm --ulimit 'nproc=2' --user nobody debian bash
nobody@7005f259a827:/$

Terminal #2:

$ docker run -it --rm --ulimit 'nproc=2' --user nobody debian bash
resource temporarily unavailable
FATA[0000] Error response from daemon: Cannot start container 589299c070779487462393fcb04df05d619d2debe1b1197c41587ee53c2283b8: [8] System error: resource temporarily unavailable

I tested this on Ubuntu 14.04 (kernel 3.16) and Ubuntu 12.04 (kernel 3.13), Docker 1.6.0.

cpuguy83 · 2015-04-23T13:13:58Z

After some digging, it seems nproc is special and is per user.

frol · 2015-04-23T14:36:21Z

Note: another part of discussion is going in #6479 (Fork bomb prevention) and will probably continue there.

ashish235 · 2015-08-03T05:56:54Z

So how to set the Ulimit? Can it be done per container wise? I seem to be clueless with the discussions happening in this thread.. Can someone help please.
I need "ulimit -c unlimited" equivalent for docker.

ashish235 · 2015-08-03T06:43:32Z

[root@localhost ~]# docker run --ulimit core=unlimited -t ubuntu:12.04 bash -c "ulimit -c"
invalid value "core=unlimited" for flag --ulimit: strconv.ParseInt: parsing "unlimited": invalid syntax
See 'docker run --help'.

Unlimited is not a recognized keyword here.

My docker version is the latest 1.7.

[root@localhost ~]# docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): ba1f6c3/1.6.2
OS/Arch (client): linux/amd64
INFO[0203] GET /v1.18/version
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

Also setting big values doesn't help here.

[root@localhost ~]# docker run --ulimit core=99999 -t ubuntu:12.04 bash -c "ulimit -c"
97

It sets only 97.

thaJeztah · 2015-08-03T07:48:38Z

Hi @ashish235 please keep in mind that GitHub is an issue tracker, and not a general support forum.

A number of things I noticed; first of all, although your daemon is 1.7.1, your client is still at 1.6.2, so you might want to update that as well.

There were some fixes to ulimits in docker 1.7.x, but I'm not sure if those fixes were in the client or daemon, so this may be worth checking. (#12515)

W.r.t. "unlimited"; #12515 (comment)

unlimited is purposefully not support since it's not a real value, it's just a constant set to a really high number.

W.r.t. "setting big values doesn't help here"; please note this comment: #12957 (comment)

It sets the limit value as number of bytes, while ulimit reports the number of kb. So 32768 bytes == 32 kb

Converting 99999 to kb; https://www.google.com/search?q=99999%20bytes%20in%20kibibyte I get "97", so this looks right

First attempt: most probably Docker.io would not honor the values in /etc/security etc. so this would not work: # Core Generation: http://www.opensips.org/Documentation/TroubleShooting-Crash RUN \ echo -n -e 'opensips soft core unlimited\nopensips hard core unlimited\n' >> /etc/security/limits.conf \ && \ echo -n -e 'fs.suid_dumpable = 1\nkernel.core_uses_pid = 1\n' >> /etc/sysctl.conf See e.g. moby/moby#9437

AkihiroSuda · 2019-08-02T08:13:59Z

@cpuguy83 @cyphar @giuseppe

Do you think we can add support for RLIMIT_AS for rootless mode?

cpuguy83 force-pushed the set_rlimits_in_container branch from 46ffe5a to 8036187 Compare December 1, 2014 18:44

thaJeztah reviewed Dec 1, 2014
View reviewed changes

cpuguy83 force-pushed the set_rlimits_in_container branch from 8036187 to 61fa833 Compare December 1, 2014 19:04

jessfraz added the Proposal label Dec 1, 2014

SvenDowideit added the /project/doc label Dec 2, 2014

icecrime reviewed Dec 2, 2014
View reviewed changes

cpuguy83 mentioned this pull request Dec 2, 2014

sysctl tunables #4717

Closed

fredlf reviewed Dec 2, 2014
View reviewed changes

cpuguy83 force-pushed the set_rlimits_in_container branch from 61fa833 to 9681336 Compare December 3, 2014 18:33

icecrime reviewed Jan 14, 2015
View reviewed changes

cpuguy83 force-pushed the set_rlimits_in_container branch from 9681336 to 61445a5 Compare January 14, 2015 01:51

cpuguy83 force-pushed the set_rlimits_in_container branch from 61445a5 to d407d43 Compare January 14, 2015 22:42

cpuguy83 force-pushed the set_rlimits_in_container branch from d407d43 to 4a63d3f Compare January 14, 2015 22:44

cpuguy83 mentioned this pull request Feb 27, 2015

Fork bomb prevention #6479

Closed

bgrant0607 mentioned this pull request Feb 27, 2015

Take advantage of Docker ulimit support once it's merged kubernetes/kubernetes#4911

Closed

thaJeztah mentioned this pull request Mar 1, 2015

Filehandle limits in init scripts lead to extremely poor python performance #9876

Closed

jessfraz pushed a commit that referenced this pull request Mar 4, 2015

Merge pull request #9437 from cpuguy83/set_rlimits_in_container

df7ba57

Allow setting ulimits for containers

jessfraz merged commit df7ba57 into moby:master Mar 4, 2015

dmp42 removed the status/2-code-review label Mar 4, 2015

timothysc mentioned this pull request Mar 4, 2015

ulimit support from docker. kubernetes/kubernetes#5042

Closed

thaJeztah mentioned this pull request Mar 6, 2015

RUN apt-get update too slow during building an image, on EC2 #9766

Closed

cpuguy83 mentioned this pull request Mar 25, 2015

Add documentation on new --ulimit and --default-ulimit support #11754

Closed

cpuguy83 deleted the set_rlimits_in_container branch April 16, 2015 20:27

liquid-sky mentioned this pull request Apr 22, 2015

Adds support for --ulimit parameter sous-chefs/docker#306

Closed

liquid-sky mentioned this pull request Apr 22, 2015

Adds support for --ulimit parameter sous-chefs/docker#307

Merged

delcypher mentioned this pull request Jun 1, 2015

Setting stack size limit either prevents container from starting or the values set are ignored. #13521

Closed

yurtaev mentioned this pull request Dec 1, 2015

Reducing the size of docker image centrifugal/centrifugo#43

Closed

brunoborges mentioned this pull request Dec 20, 2015

ulimits/sysctls are not in effect oracle/docker-images#51

Closed

vcaputo mentioned this pull request Jan 22, 2016

How to read max open files of a process coreos/bugs#1069

Closed

kesor mentioned this pull request Jan 26, 2018

rlimit support kubernetes/kubernetes#3595

Open

ilyam8 mentioned this pull request Sep 13, 2020

Change file descriptors to meet the requirements of dbengine netdata/helmchart#136

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow setting ulimits for containers #9437

Allow setting ulimits for containers #9437

cpuguy83 commented Dec 1, 2014

thaJeztah Dec 1, 2014

cpuguy83 Dec 1, 2014

thaJeztah Dec 1, 2014

crosbymichael commented Dec 1, 2014

SvenDowideit commented Dec 2, 2014

cpuguy83 commented Dec 2, 2014

icecrime Dec 2, 2014

fredlf Dec 2, 2014

fredlf Dec 2, 2014

fredlf commented Dec 2, 2014

cpuguy83 commented Dec 3, 2014

jamtur01 commented Dec 9, 2014

fredlf commented Dec 10, 2014

fredlf commented Dec 10, 2014

lexinator commented Dec 15, 2014

SvenDowideit commented Dec 16, 2014

icecrime Jan 14, 2015

icecrime commented Jan 14, 2015

jeremyeder commented Jan 14, 2015

cpuguy83 commented Jan 14, 2015

cpuguy83 commented Jan 14, 2015

philips commented Feb 26, 2015

jessfraz commented Mar 4, 2015

liquid-sky commented Apr 22, 2015

cpuguy83 commented Apr 22, 2015

frol commented Apr 23, 2015

cpuguy83 commented Apr 23, 2015

frol commented Apr 23, 2015

ashish235 commented Aug 3, 2015

ashish235 commented Aug 3, 2015

thaJeztah commented Aug 3, 2015

AkihiroSuda commented Aug 2, 2019

Allow setting ulimits for containers #9437

Allow setting ulimits for containers #9437

Conversation

cpuguy83 commented Dec 1, 2014

thaJeztah Dec 1, 2014

Choose a reason for hiding this comment

cpuguy83 Dec 1, 2014

Choose a reason for hiding this comment

thaJeztah Dec 1, 2014

Choose a reason for hiding this comment

crosbymichael commented Dec 1, 2014

SvenDowideit commented Dec 2, 2014

cpuguy83 commented Dec 2, 2014

icecrime Dec 2, 2014

Choose a reason for hiding this comment

fredlf Dec 2, 2014

Choose a reason for hiding this comment

fredlf Dec 2, 2014

Choose a reason for hiding this comment

fredlf commented Dec 2, 2014

cpuguy83 commented Dec 3, 2014

jamtur01 commented Dec 9, 2014

fredlf commented Dec 10, 2014

fredlf commented Dec 10, 2014

lexinator commented Dec 15, 2014

SvenDowideit commented Dec 16, 2014

icecrime Jan 14, 2015

Choose a reason for hiding this comment

icecrime commented Jan 14, 2015

jeremyeder commented Jan 14, 2015

cpuguy83 commented Jan 14, 2015

cpuguy83 commented Jan 14, 2015

philips commented Feb 26, 2015

jessfraz commented Mar 4, 2015

liquid-sky commented Apr 22, 2015

cpuguy83 commented Apr 22, 2015

frol commented Apr 23, 2015

cpuguy83 commented Apr 23, 2015

frol commented Apr 23, 2015

ashish235 commented Aug 3, 2015

ashish235 commented Aug 3, 2015

thaJeztah commented Aug 3, 2015

AkihiroSuda commented Aug 2, 2019