Could you start the deamon in debug mode (with -D) and paste here the debug ?

Thanks.

As soon as I'm able to reproduce it I'll post (next broken image) Thank you !

[debug] api.go:883 Calling POST /containers/create
2013/07/09 16:11:25 POST /v1.3/containers/create
[debug] api.go:883 Calling POST /containers/{name:.*}/start
2013/07/09 16:11:25 POST /v1.3/containers/64e6343f4d63/start
2013/07/09 16:11:25 Kernel does not support AUFS, trying to load the AUFS module with modprobe...
2013/07/09 16:11:25 ...module loaded.

Nothing special in this output, aufs module is always loaded, the error is not there event if docker tries to modprobe again the driver. As I said before with another image it's working.

I'm using Ubuntu 12.04 LTS with PPA docker repository.

/cc @jpetazzo

I don't see any useful clue here, unfortunately!
Would there be a way to tar the broken image (and its dependencies) so we
can have a look?

Ok next case I'll export the container if that's possible ! (Maybe the same error will occure)

Broken Image :

docker run -i -t 298bd447a161 bash
2013/07/22 14:48:37 Error: Error starting container 2bdc2487652c: Unable to mount using aufs

Working Image :

docker run -i -t base bash
root@29dd2712d871:/# 

docker inspect 298bd447a161
[{
    "id": "298bd447a161615bd6cadb0a7e1a6646ae982d5ff46f676b873676498675bcb9",
    "parent": "a72f54bf6432c5961cbc777cc4cdbea54d11d6fae7483cd5e0d999d9f4aaf2fa",
    "created": "2013-07-22T14:38:37.039340267Z",
    "container": "8ff61bd27b30f5da3806c8f635ba6dc6aef489c247eceee276c5e566cff07985",
    "container_config": {
        "Hostname": "8ff61bd27b30",
        "User": "",
        "Memory": 0,
        "MemorySwap": 0,
        "CpuShares": 0,
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "PortSpecs": null,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": null,
        "Cmd": [
            "/buildpacks/builder"
        ],
        "Dns": null,
        "Image": "app/testapp",
        "Volumes": null,
        "VolumesFrom": "",
        "Entrypoint": null
    },
    "docker_version": "0.5.0",
    "config": {
        "Hostname": "",
        "User": "",
        "Memory": 0,
        "MemorySwap": 0,
        "CpuShares": 0,
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "PortSpecs": null,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": null,
        "Cmd": null,
        "Dns": null,
        "Image": "",
        "Volumes": null,
        "VolumesFrom": "",
        "Entrypoint": null
    },
    "architecture": "x86_64",
    "Size": 94881024

What is the best way to export the image ?

Thank you !

I'm tarring and uploading /var/lib/docker/graph/[IMAGE_ID] will you be able to do something with that ?

In fact this is the at least the 20th layers. (Each parent has a parent on more than 20 levels) Is their a limitation with AUFS on the number of layers we can mount ?

(Because the command which fails is directly moun -t aufs ... )

@Soulou no, because the parents arent' there. could you docker push it ?

@Soulou yes there is a limit in the number of aufs layers, I think it's 42 (at least in the default build on ubuntu kernels).
—
@solomonstre
@getdocker

On Mon, Jul 22, 2013 at 8:32 AM, Victor Vieux notifications@github.com
wrote:

@Soulou no, because the parents arent' there. could you docker push it ?

Reply to this email directly or view it on GitHub:
#1171 (comment)

I'm getting this also, 38 RUN commands plus one EXPOSE above the base image, on Ubuntu 13.0.4. Could it have something to do with the brs=1|0 option of http://manpages.ubuntu.com/manpages/karmic/man5/aufs.5.html?

              If the number of your branches is large or their  path  is  long
              and  you  meet the limitation of mount(8) ro /etc/mtab, you need
              to enable CONFIG_SYSFS and set aufs module parameter brs=1.   If
              your  linux  version  is  linux-2.6.24  and earlier, you need to
              enable CONFIG_AUFS_SYSAUFS too.

If it isn't feasible to increase aufs's maximum depth, implementing #332 would help a lot, especially if we could put some kind of FLATTEN command in dockerfiles.

One stopgap solution would be to let us group several shell commands into a single RUN command, so that we can decrease the number of layers we create without sacrificing too much readability:

RUN
    git clone git://github.com/apatil/repo
    cd repo
    ./configure
    make install

When you get the "Unable to mount using aufs" error, do you get something
more in the kernel logs or the output of the docker daemon itself?
(That would be very helpful to pinpoint the root cause of the problem.)

On Mon, Jul 22, 2013 at 10:50 PM, Anand Patil notifications@github.comwrote:

I'm getting this also, 38 RUN commands plus one EXPOSE above the base
image, on Ubuntu 13.0.4. Could it have something to do with the brs=1|0option of
http://manpages.ubuntu.com/manpages/karmic/man5/aufs.5.html?

          If the number of your branches is large or their  path  is  long
          and  you  meet the limitation of mount(8) ro /etc/mtab, you need
          to enable CONFIG_SYSFS and set aufs module parameter brs=1.   If
          your  linux  version  is  linux-2.6.24  and earlier, you need to
          enable CONFIG_AUFS_SYSAUFS too.

If it isn't feasible to increase aufs's maximum depth, implementing #332https://github.com/dotcloud/docker/issues/332would help a lot, especially if we could put some kind of FLATTEN command
in dockerfiles.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1171#issuecomment-21394503
.

From the Docker daemon:

[debug] api.go:64 [error 500] Unable to mount using aufs
2013/07/23 19:07:45 http: multiple response.WriteHeader calls

This is with revision a93a87f. Haven't found anything in the kernel logs.

I can reproduce the problem by putting

FROM base
RUN "ls"
RUN "ls" # repeat 40-50 times

in a dockerfile, and doing docker build . inside its folder.

@apatil there's a limit to how many AUFS layers can be created. I believe it's 42.

Should we have an error message that is returned before doing docker build or when the maximum is reached?

I'm having the same problem on Ubuntu 12.04 LTS with PPA docker repository.
I think some FLATTEN command like is described in #332 would be a good solution.

Reproduced. Gist with sample Dockerfile and build output:
https://gist.github.com/metalivedev/6085112

This is weird, once you reached the limit (use the Dockerfile fom @metalivedev)
You can't mount any more, doesn't look to be per image, but global.

ping @shykes @jpetazzo

Same problem here, also.

I understand why creating an image per step in the Dockerfile helps, noticeably while growing / debugging an image.
But maybe we should have the option to choose when this occurs, so that for the production of the final image, there are less intermediate ones produced.

Maybe either a flag passed to the command line to prevent the commit of intermediate images. Or directives inside the Dockerfile. (the former seems more interesting since you won't have to change the Dockerfile everytime you want to rewrite on it - and just before you want to create the final production image - )

@laurentpetit I believe #332 is the answer for that.

@fsouza It's answer proposal which would solve the same problem, but not the same proposal :-). I like it also, but am not seeing it make progress :-/

I had the same problem

2013/08/17 20:59:59 Error: Error starting container af267a84f9c4: Unable to mount using aufs

Merging a few RUN commands, and the container starts normally.

Hey Guys - We're experiencing the same problem with some images. We do not believe we have a layer issue (is there a way to check)?

In our kernel logs, we see this error
aufs opt_add:714:docker[15256]: lookup failed /var/lib/docker/graph/32737f8072d0 (-2)

so docker history shows that we are hitting the limit :(
any good ideas on compressing the number of steps to a single layer?

Yash, currently the only way to "squashing" the image is to create a
container from it, export that container into a raw tarball, and re-import
that as an image. Unfortunately that will cause all image metadata to be
lost, including its history but also ports, env, default command,
maintainer info etc. So it's really not great.

There are 2 things we can do to improve the situation:

1. A short-term solution is to implement a "lossless export" function,
   which would allow exporting an image to a tarball with all its metadata
   preserved, so that it can be re-imported on the other side without loss.
   This would preserve everything except history, because an image config does
   not currently carry all of its history. We could try to plan this for 0.7
   which is scheduled for mid-September. That is, if our 0.7 release manager
   @vieux decides we have time to fit it in the release :)

2. A 2nd step would be to add support for history as well. This is a little
   more work because we need to start storing an image's full history in each
   image, instead of spreading it out across all the aufs layers. This is
   planned for 0.8.

On Mon, Aug 19, 2013 at 9:00 PM, Yash Kumar notifications@github.comwrote:

so docker history shows that we are hitting the limit :(
any good ideas on compressing the number of steps to a single layer?

—

Reply to this email directly or view it on GitHubhttps://github.com//issues/1171#issuecomment-22921077
.

Thanks solomon. We're also playing around with aubrsync to copy down branches. Do you not recommend this approach?

We're expecting our commit history to get 100-150 layers long.

This is related to issue #332.

On Mon, Aug 19, 2013 at 9:24 PM, Yash Kumar notifications@github.comwrote:

Thanks solomon. We're also playing around with aubrsync to copy down
branches. Do you not recommend this approach?

We're expecting our commit history to get 100-150 layers long.

—

Reply to this email directly or view it on GitHubhttps://github.com//issues/1171#issuecomment-22921792
.

@lox can you post the output of docker info

A testcase to reproduce: https://gist.github.com/dhotson/8883c2e8ebf79ce95115

#7863 looks to be related to @lox's issue

After some testing, this appears to specifically affect the kernel used in EC2, can't reproduce the issue elsewhere, but on Linux ip-172-31-32-231 3.13.0-35-generic #62-Ubuntu SMP Fri Aug 15 01:58:42 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux it happens consistently.

Possibly also related to? #4338

We'll do some testing on whether it's related to having docker's layers stored in /mnt.

I seem to remember this being hard coded at 42 due to some AUFS limitations but after searching the code a tad now I’m not as sure.

On Sep 16, 2014, at 8:16 PM, Lachlan Donald notifications@github.com wrote:

Possibly also related to? #4338

We'll do some testing on whether it's related to having docker's layers stored in /mnt.

—
Reply to this email directly or view it on GitHub.

Its 127 now
On Sep 16, 2014 8:35 PM, "Erik Hollensbe" notifications@github.com wrote:

I seem to remember this being hard coded at 42 due to some AUFS
limitations but after searching the code a tad now I’m not as sure.

On Sep 16, 2014, at 8:16 PM, Lachlan Donald notifications@github.com
wrote:

Possibly also related to? #4338

We'll do some testing on whether it's related to having docker's layers
stored in /mnt.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#1171 (comment).

Strangely, the problem occurs with docker mounted on /mnt/docker but works fine with /mnt/anything_else_here/docker. Truly bizarre, any insights into what might be causing that?

I can reproduce this on 1.10.3 . Any reason why this is closed?

@agherzan If you are reproducing a 42-layer issue, then most likely it's due to how your aufs kernel module is compiled.
We do have a 127 layer hard limit for all storage drivers, however.

What's your setup?

Are you referring to the max branch config of aufs? @cpuguy83

EDIT: I run docker 1.10.3 on an armv7 board, aufs, linux 4.1.15, 42 layers

@agherzan Yes.

@cpuguy83 I have 127

CONFIG_AUFS_FS=y
CONFIG_AUFS_BRANCH_MAX_127=y

Just for reference I will tell here what happened in my case for people to benefit from it.

I was using golang 1.6 which hardcodes the page size of arm64 to 64. This made docker generate aufs mount commands that would make options not fit in 4K. In my case the kernel is configured without 64K page size support so mount command would return invalid argument due to string truncation as it uses a page size as maximum option size.

There are some patches in master (of golang) which removes this static behaviour and detects the pagesize at runtime. (https://go-review.googlesource.com/#/c/25021/) Backporting these fixed my problem.

@agherzan Thanks for the report!
I assume you mean go 1.6 and not docker 1.6?

Edited to clarify @cpuguy83 . Typo.

Thanks again!