@tianon should we implement the workaround to fix jdk 8 images and any children?

I suppose, although only if it's going to be temporary for sure. Has anyone figured out what's wrong upstream to cause it? I'd feel a lot better about carrying a temporary workaround if we made sure we did our due diligence and got a patch submitted upstream so we make sure it really is only temporary.

Looks like /etc/ca-certificates/update.d/jks-keystore can be used to generate /etc/ssl/certs/java/cacerts. I haven't dug in, but I'm assuming it generates it from the certs installed by ca-certificates. All of this is part of the ca-certificates-java package.

Update: And now I see that y'all are already talking about the the ca-certificates-java package... 🙈

Also, it appears that the right command to run would be update-ca-certificates. I'm not sure why it gets run in the java:7 case and not the java:8 case.

It looks like the issue is that there is no java in the $PATH. There are errors during the installation of both ca-certificates and ca-certificates-java that get ignored. Right after ca-certificates gets set up, then update-alternatives gets called to make java work.

Setting up ca-certificates-java (20140324) ...
/var/lib/dpkg/info/ca-certificates-java.postinst: line 53: java: command not found
/var/lib/dpkg/info/ca-certificates-java.postinst: line 66: java: command not found
done.
...
Processing triggers for ca-certificates (20141019) ...
Updating certificates in /etc/ssl/certs... 173 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....
/etc/ca-certificates/update.d/jks-keystore: 82: /etc/ca-certificates/update.d/jks-keystore: java: not found
E: /etc/ca-certificates/update.d/jks-keystore exited with code 1.
done.
Setting up openjdk-8-jre-headless:amd64 (8u40~b09-1) ...
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/rmid to provide /usr/bin/rmid (rmid) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java to provide /usr/bin/java (java) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/keytool to provide /usr/bin/keytool (keytool) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/jjs to provide /usr/bin/jjs (jjs) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/pack200 to provide /usr/bin/pack200 (pack200) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/rmiregistry to provide /usr/bin/rmiregistry (rmiregistry) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/unpack200 to provide /usr/bin/unpack200 (unpack200) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/orbd to provide /usr/bin/orbd (orbd) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/servertool to provide /usr/bin/servertool (servertool) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/tnameserv to provide /usr/bin/tnameserv (tnameserv) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jexec to provide /usr/bin/jexec (jexec) in auto mode

It looks like ca-certificates-java needs to declare its dependency on a java executable so that it doesn't get installed until after openjdk-8-jre-headless.

I think I found the root cause here: http://anonscm.debian.org/viewvc/pkg-java/trunk/ca-certificates-java/debian/postinst.in?view=markup#l29

It looks like the postinstall script attempts to find its java executable by shortcutting update-alternatives and looking for the JDK directly. To do this, it has a hard-coded list of possible JDK locations that doesn't include JDK 8, so the JDK 8 java command will not be used.

After the installation is complete (in particular the selection of JDK 8 as /usr/bin/java in its postinstall), the update-ca-certificates script will now work correctly because it can find java from $PATH.

It looks like one fix would be to update the postinstall script to look for the JDK in java-8-openjdk-$arch and java-8-openjdk. That will just break for the next JDK, but it fixes what's being done currently at least.

Proposal: Add the workaround:

RUN /var/lib/dpkg/info/ca-certificates-java.postinst configure

But also add ca-certificates-java to the apt-get install with the specific failing version 20140324, that way we can update the version and drop the RUN when it gets fixed (or just drop both). Either way we get a nice docker cache bust.

For the record, @yosifkit filed a patch upstream to update the lines @md5 pointed out: https://bugs.debian.org/775775 ❤️

I'm +1 on @yosifkit's proposal for now, especially since we've got a nice natural cache-bust when it's fixed upstream. 👍

Hello folks, I just wanted to point out that this workaround may not be necessary anymore, and thus the code for it may be able to be removed from the Dockerfile.

Here's my case:

FROM debian:jessie
MAINTAINER Bilal Akil <mail@bilalakil.me>

# Mostly copied from `docker-library/openjdk`'s
# `openjdk-8-jdk` Dockerfile, which uses Debian!
#
# Referenced during development:
#   https://github.com/docker-library/openjdk/blob/2dd0496901a9c01c78895d0e6618b36c08f
78bde/openjdk-8-jdk/Dockerfile
# Latest:
#   https://github.com/docker-library/openjdk/blob/master/openjdk-8-jdk/Dockerfile

ENV DBUILD_OPENJDK_VERSION 8u72-b15-1~bpo8+1
ENV DBUILD_CA_CERTIFICATES_JAVA_VERSION 20140324

ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64
ENV JAVA_VERSION 8u72

RUN \
  # We need to add OpenJDK's backport repository to find their Java 8.
  echo 'deb http://httpredir.debian.org/debian jessie-backports main' > /etc/apt/sourc
es.list.d/jessie-backports.list \
  \
  # Now we can `apt-get` it.
  && apt-get update \
  && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
    openjdk-8-jdk="$DBUILD_OPENJDK_VERSION" \
    ca-certificates-java="$DBUILD_CA_CERTIFICATES_JAVA_VERSION" \
  \
  # See https://github.com/docker-library/java/issues/19#issuecomment-70546872
  # (This has been commented out for demonstration.)
  # /var/lib/dpkg/info/ca-certificates-java.postinst configure
  \
  # And finally we can reverse as much of the damage as we can.
  && rm /etc/apt/sources.list.d/jessie-backports.list \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

This is my Dockerfile. It was originally using a different base image but for the purpose of this example debian:jessie is fine.

If I understand correctly, that's the same base operating system as in the OpenJDK 8 Dockerfile. Most of the contents of this Dockerfile and yours are identical, but with a bit of a writing style difference.

$ docker build -t test .
$ docker run --rm test ls -la /etc/ssl/certs/java/cacerts
-rw-r--r-- 1 root root 208916 Feb 11 07:32 /etc/ssl/certs/java/cacerts

So yeah, it doesn't seem necessary anymore..? I'm not sure what's changed since this issue was opened last year though, but hey, food for thought.

Hmm, that's strange -- the Debian bug is still open, and was set to severity "serious" just days ago by one of the Java maintainers. 😕

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=775775#13

Severity set to 'serious' from 'normal' Request was from Matthias Klose <doko@debian.org> to control@bugs.debian.org. (Mon, 08 Feb 2016 20:51:06 GMT)

I am trying to use the workaround in Ubuntu 14.04 since it obviously has the problem. I put it in the dockerfile.
RUN /var/lib/dpkg/info/ca-certificates-java.postinst configure
before
RUN apt-get install -y default-jdk
but I got
/bin/sh: 1: /var/lib/dpkg/info/ca-certificates-java.postinst: not found

The command works from terminal directly but that certainly does not help docker to build. Any sample dockerfile?

@hayday100 from my original post, you must put

RUN /var/lib/dpkg/info/ca-certificates-java.postinst configure

after the apt-get install command. If you do it before, then the file won't yet have been created, since it is created by installing java.

I tried that last night too. But it certainly got stuck at the

Setting up ca-certificates-java (20130815ubuntu1) ...

which obviously beat the purpose of the workaround.

@hayday100 That's not the purpose of this workaround. This workaround existed because cacerts was not being created. I actually think the fact that you're getting that hanging behavior shows that the package trigger that creates cacerts is trying to run.

The problem you appear to be running into is moby/moby#18180

Could you provide the output of docker version?

Client:
Version: 1.10.1
API version: 1.22
Go version: go1.5.3
Git commit: 9e83765
Built: Thu Feb 11 19:27:08 2016
OS/Arch: linux/amd64

Server:
Version: 1.10.1
API version: 1.22
Go version: go1.5.3
Git commit: 9e83765
Built: Thu Feb 11 19:27:08 2016
OS/Arch: linux/amd64

It is probably irrelevant but it is a VirtualBox 5.0.14 in Windows 8.1.

I am willing to upgrade Ubuntu if that helps but people say that this persists until Ubuntu 15.04 so that's why I am very interested in this workaround. But it looks like the workaround does not help in docker since there is not a right place for it in the dockerfile.

No. It is a local VirtualBox that I installed from the ISO from scratch.

I am creating the VM for this in case some one is interested in reproducing the error.
https://www.ibm.com/developerworks/cloud/library/cl-bluemix-minecraft-docker-trs-1/

Do you think that boot2docker can be an alternative for Ubuntu in light of this ca-certificates problem?

The lack of GUI in Boot2docker will make it difficult for part 2 of the same tutorial but possible. I just don't want to chase a dead end.

I am a docker rookie. According to my understanding, he docker server is where the images are and client is where the containers will run from the images generated. The error occurs when the docker server is trying to add a ssh certificate into the image yet the openjdk certificate is not recognized. I assume if docker is a full OS, the workaround will definitely work. But it is a mini-VM where things are built only from dockerfile following a sequential order (or at least before I know enough to tweak it).

Hanging on Setting up ca-certificates-java...

Hi @hayday100, I've experienced a very similar problem as you (where running a Docker container hangs on Setting up ca-certificates-java...) and believe @md5 is correct about the issue being docker/docker#18180.

It's a tricky problem and took me quite a while to deal with, however workarounds were covered extensively in that issue - look for @AkihiroSuda's big "LATEST QUICK WORKAROUNDS" comment (NB: is there any way for me to copy a link to the comment? For now just search the previously quoted words). Note that there's more than one of said comment from this generous contributor, so ensure you're reading the most up to date one.

In summary there's a kernel bug that has reached various *nix operating systems and affects particular versions of Docker. That is, it's the kernel of the OS running Docker that has the bug causing this particular problem, as opposed to the OS that Docker is running.

That being said, the workaround is to either up/downgrade your kernel to one without that problem, or downgrade Docker instead. The huge comment on that issue details exactly which kernel version to upgrade to depending on your underlying OS. Again, ensure you're reading the most up to date version of that comment.

I personally wasn't feeling up to playing with the kernel at the time so instead I downgraded my Docker to 1.8.3 and this problem disappeared. Obviously that's just a short-term solution, but hopefully that'll help you deal with your problem.

So what about the problem and workaround discussed in this issue?

$ docker run --rm -it java:openjdk-8-jdk ls -al /etc/ssl/certs/java/cacerts
...
ls: cannot access /etc/ssl/certs/java/cacerts: No such file or directory

In my experience this was a separate problem. Once you deal with the above issue that caused the hanging on Setting up ca-certificates-java... you should then consider this issue. Observe that this issue is only noticeable once the container is successfully running, as in the above snippet using docker run.

If you think you're experiencing the problem discussed here, your Docker image should still build fine (unless you're immediately using the subject files) and you should be able to check if you're having the problem and need to use the workaround discussed here.

I have done exactly that just a few comments up, so you can read that as a guide on how to test whether you're having the problem or not. To my surprise, I was not having the problem and didn't need that workaround code at all, and that's why I commented about it. I'd be interested to see if you end up needing the workaround too.

bilalakil: you got it precisely. The kernel must be downgraded to 3.13.0-71. There were a few hiccups while trying to do that. First, the generic kernel downgraded only to 3.13.0-79 so I had to manually download and downgrade. Then the virtualbox lost its mouse pointer. I later found that you just have to use "input -> mouse integration" to reset the pointing device. Thirdly, the docker daemon refused to start again after the downgrade and apt-get did not help. I went to the docker website and followed the longer process to reinstall docker. But finally everything works like a charm! You don't need the workaround, just a good old engine! Thanks to all!

This is still happening on Debian and the workaround doesn't fix it

Build your image on top of it.

FROM java:openjdk-8-jdk
RUN sed -i "2 iexport JAVA_HOME=${JAVA_HOME}" /etc/ca-certificates/update.d/docker-openjdk

at 2nd line, it will insert JAVA_HOME with a computed value ( that's why double quotes not single quotes)

Credits to : jenkinsci/docker#901