Skip to content
This repository has been archived by the owner on Feb 24, 2020. It is now read-only.

rkt fails to render Docker images with whiteout-ed hard links #1653

Closed
zopyx opened this issue Oct 21, 2015 · 3 comments
Closed

rkt fails to render Docker images with whiteout-ed hard links #1653

zopyx opened this issue Oct 21, 2015 · 3 comments
Assignees
@iaguis
Labels
depends-on/external kind/bug kind/support priority/P1 technology/docker
Milestone
v0.12.0

Comments

@zopyx
Copy link

zopyx commented Oct 21, 2015 

[ajung@vmd9666 rkt-v0.9.0]$ uname -a
Linux vmd9666.contabo.host 4.1.7-200.fc22.x86_64 #1 SMP Mon Sep 14 20:19:24 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[ajung@vmd9666 rkt-v0.9.0]$ cat /etc/issue
Fedora release 22 (Twenty Two)
Kernel \r on an \m (\l)

[ajung@vmd9666 rkt-v0.9.0]$ sudo ./rkt --insecure-skip-verify fetch docker://zopyx/xmldirector-plone
[ajung@vmd9666 rkt-v0.9.0]$ sudo ./rkt --insecure-skip-verify run sha512-414fcf004ed2282e45aa16fafcf08b07
[sudo] password for ajung:
rkt: using image from file /home/ajung/src/rkt-v0.9.0/stage1-coreos.aci
2015/10/21 14:51:40 Preparing stage1
2015/10/21 14:51:44 Writing image manifest
2015/10/21 14:51:44 Loading image sha512-414fcf004ed2282e45aa16fafcf08b071d5f4a4e6308622faac764cae5972ab0
run: error setting up stage0: error setting up image sha512-414fcf004ed2282e45aa16fafcf08b071d5f4a4e6308622faac764cae5972ab0: error rendering tree image: treestore: cannot render aci: error extracting ACI: extracttar error: exit status 1, output: error extracting tar: error extracting tarball: link /rootfs/var/lib/dnf/yumdb/b/de43dd8b006bf3b89c4e11c9176ba44a7a3f8ed2-bash-4.3.33-2.fc22-x86_64/reason /rootfs/var/lib/dnf/yumdb/e/cb9ee1ca63f1980ace7b8c577664eeb36488c47d-e2fsprogs-1.42.12-4.fc22-x86_64/reason: no such file or directory
@alban alban mentioned this issue Oct 21, 2015
Closed
@alban
Copy link
Member

alban commented Oct 21, 2015

Thanks for the bug report!

rkt uses docker2aci to convert the Docker image and it does not handle well hard links that get deleted in a subsequent Docker layer: after the rkt fetch, I tried the following:

# tar xf /var/lib/rkt/cas/blob/sha512/9f/sha512-9f66eec1ced37f239c2a94f5eb0df5e6620be823fca4cf23d87a8747d8c1b003 -C /tmp/extract/
tar: rootfs/var/lib/dnf/yumdb/e/cb9ee1ca63f1980ace7b8c577664eeb36488c47d-e2fsprogs-1.42.12-4.fc22-x86_64/reason: Cannot hard link to ‘rootfs/var/lib/dnf/yumdb/b/de43dd8b006bf3b89c4e11c9176ba44a7a3f8ed2-bash-4.3.33-2.fc22-x86_64/reason’: No such file or directory
tar: rootfs/var/lib/dnf/yumdb/f/fbb1718e2212451072f77f3aff0913efd9e39ccd-fedora-release-22-1-noarch/reason: Cannot hard link to ‘rootfs/var/lib/dnf/yumdb/b/de43dd8b006bf3b89c4e11c9176ba44a7a3f8ed2-bash-4.3.33-2.fc22-x86_64/reason’: No such file or directory
tar: rootfs/var/lib/dnf/yumdb/r/e1898da4a71f470ffefbddadfa3d12d55f4a151e-rootfiles-8.1-17.fc21-noarch/reason: Cannot hard link to ‘rootfs/var/lib/dnf/yumdb/b/de43dd8b006bf3b89c4e11c9176ba44a7a3f8ed2-bash-4.3.33-2.fc22-x86_64/reason’: No such file or directory
tar: Exiting with failure status due to previous errors

I reported the issue on appc/docker2aci#98

@jonboulle jonboulle changed the title rkt: 0.9.0: error rendering tree image: treestore: cannot render aci: error extracting ACI: extracttar error: exit status 1, output: error extracting tar: error extracting tarball: link rkt fails to render Docker images with whiteout-ed hard links Oct 21, 2015
@alban alban added this to the v0.11.0 milestone Oct 23, 2015
alban added a commit to alban/docker2aci that referenced this issue Nov 12, 2015
@alban
handle hard links and whiteouts correctly
f582fb1 
Hard links and whiteout were not handled correctly in this scenario:
- The first Docker layer contains:
  - file A
  - file B hard link to file A
- The second Docker layer contains:
  - whiteout file A

Then, the squashed ACI previously generated by docker2aci contained:
  - file B dangling hard link to file A

Hence the bug.

This patch changes:
- the converting algorithm lib/common/common.go:writeACI() to generate:
  - First docker layer
    - .hidden.docker2aci.sha512-xxx (hash of the layer and file name)
    - file A hard link to .hidden.docker2aci.sha512-xxx
    - file B hard link to .hidden.docker2aci.sha512-xxx
  - Second docker layer
    - whiteout file A
- the squashing algorithm lib/docker2aci.go:SquashLayers() to have two
  passes:
  - Pass one: build an in-memory map of hard links and whiteouts
  - Pass two: remove white-out and .hidden.docker2aci.sha512-xxx files

I tested the following images from rkt/rkt#1653:
- docker://albanc/busybox-hardlinks
- docker://zopyx/xmldirector-plone

TODO:
- not optimized if there are no hard links
- semantic changes on --nosquash

Fixes appc#98
@alban alban mentioned this issue Nov 12, 2015
Closed
@alban
Copy link
Member

alban commented Nov 12, 2015

I prepared a fix in docker2aci: appc/docker2aci#100

But I would like to have time to change it to do the conversion without an additional intermediary file. Then, it will need to be reviewed and more tested. => Moving to next milestone.

@alban alban modified the milestones: v0.12.0, v0.11.0 Nov 12, 2015
alban added a commit to alban/docker2aci that referenced this issue Nov 17, 2015
@alban
handle hard links and whiteouts correctly
6ddea53 
Hard links and whiteout were not handled correctly in this scenario:
- The first Docker layer contains:
  - file A
  - file B hard link to file A
- The second Docker layer contains:
  - whiteout file A

Then, the squashed ACI previously generated by docker2aci contained:
  - file B dangling hard link to file A

Hence the bug.

This patch changes:
- the converting algorithm lib/common/common.go:writeACI() to generate:
  - First docker layer
    - .hidden.docker2aci.sha512-xxx (hash of the layer and file name)
    - file A hard link to .hidden.docker2aci.sha512-xxx
    - file B hard link to .hidden.docker2aci.sha512-xxx
  - Second docker layer
    - whiteout file A
- the squashing algorithm lib/docker2aci.go:SquashLayers() to have two
  passes:
  - Pass one: build an in-memory map of hard links and whiteouts
  - Pass two: remove white-out and .hidden.docker2aci.sha512-xxx files

I tested the following images from rkt/rkt#1653:
- docker://albanc/busybox-hardlinks
- docker://zopyx/xmldirector-plone

TODO:
- not optimized if there are no hard links
- semantic changes on --nosquash

Fixes appc#98
alban added a commit to alban/docker2aci that referenced this issue Nov 19, 2015
@alban
handle hard links and whiteouts correctly
7b912ba 
Hard links and whiteout were not handled correctly in this scenario:
- The first Docker layer contains:
  - file A
  - file B hard link to file A
- The second Docker layer contains:
  - whiteout file A

Then, the squashed ACI previously generated by docker2aci contained:
  - file B dangling hard link to file A

Hence the bug.

This patch changes:
- the converting algorithm lib/common/common.go:writeACI() to generate:
  - First docker layer
    - .hidden.docker2aci.sha512-xxx (hash of the layer and file name)
    - file A hard link to .hidden.docker2aci.sha512-xxx
    - file B hard link to .hidden.docker2aci.sha512-xxx
  - Second docker layer
    - whiteout file A
- the squashing algorithm lib/docker2aci.go:SquashLayers() to have two
  passes:
  - Pass one: build an in-memory map of hard links and whiteouts
  - Pass two: remove white-out and .hidden.docker2aci.sha512-xxx files

I tested the following images from rkt/rkt#1653:
- docker://albanc/busybox-hardlinks
- docker://zopyx/xmldirector-plone

TODO:
- not optimized if there are no hard links
- semantic changes on --nosquash

Fixes appc#98
alban added a commit to alban/docker2aci that referenced this issue Nov 20, 2015
@alban
handle hard links and whiteouts correctly
091b568 
Hard links and whiteout were not handled correctly in this scenario:
- The first Docker layer contains:
  - file A
  - file B hard link to file A
- The second Docker layer contains:
  - whiteout file A

Then, the squashed ACI previously generated by docker2aci contained:
  - file B dangling hard link to file A

Hence the bug.

This patch changes:
- the converting algorithm lib/common/common.go:writeACI() to generate:
  - First docker layer
    - .hidden.docker2aci.sha512-xxx (hash of the layer and file name)
    - file A hard link to .hidden.docker2aci.sha512-xxx
    - file B hard link to .hidden.docker2aci.sha512-xxx
  - Second docker layer
    - whiteout file A
- the squashing algorithm lib/docker2aci.go:SquashLayers() to have two
  passes:
  - Pass one: build an in-memory map of hard links and whiteouts
  - Pass two: remove white-out and .hidden.docker2aci.sha512-xxx files

I tested the following images from rkt/rkt#1653:
- docker://albanc/busybox-hardlinks
- docker://zopyx/xmldirector-plone

TODO:
- not optimized if there are no hard links
- semantic changes on --nosquash

Fixes appc#98
alban added a commit to alban/docker2aci that referenced this issue Nov 20, 2015
@alban
handle hard links and whiteouts correctly
3459ce3 
Hard links and whiteout were not handled correctly in this scenario:
- The first Docker layer contains:
  - file A
  - file B hard link to file A
- The second Docker layer contains:
  - whiteout file A

Then, the squashed ACI previously generated by docker2aci contained:
  - file B dangling hard link to file A

Hence the bug.

This patch changes:
- the converting algorithm lib/common/common.go:writeACI() to generate:
  - First docker layer
    - .hidden.docker2aci.sha512-xxx (hash of the layer and file name)
    - file A hard link to .hidden.docker2aci.sha512-xxx
    - file B hard link to .hidden.docker2aci.sha512-xxx
  - Second docker layer
    - whiteout file A
- the squashing algorithm lib/docker2aci.go:SquashLayers() to have two
  passes:
  - Pass one: build an in-memory map of hard links and whiteouts
  - Pass two: remove white-out and .hidden.docker2aci.sha512-xxx files

I tested the following images from rkt/rkt#1653:
- docker://albanc/busybox-hardlinks
- docker://zopyx/xmldirector-plone

TODO:
- not optimized if there are no hard links
- semantic changes on --nosquash

Fixes appc#98
alban added a commit to alban/docker2aci that referenced this issue Nov 22, 2015
@alban
handle hard links and whiteouts correctly
0639869 
Hard links and whiteout were not handled correctly in this scenario:
- The first Docker layer contains:
  - file A
  - file B hard link to file A
- The second Docker layer contains:
  - whiteout file A

Then, the squashed ACI previously generated by docker2aci contained:
  - file B dangling hard link to file A

Hence the bug.

This patch changes the squashing algorithm lib/docker2aci.go:SquashLayers()
to have two passes:
  - Pass one: build an in-memory map of hard links and whiteouts
  - Pass two: write the archive using the map generated in pass one

I tested the following images from rkt/rkt#1653:
- docker://albanc/busybox-hardlinks
- docker://zopyx/xmldirector-plone

Fixes appc#98
@alban alban mentioned this issue Nov 23, 2015
Merged
alban added a commit to alban/rkt that referenced this issue Nov 23, 2015
@alban
Godeps: update to the last docker2aci
e07e2fd 
The last docker2aci should help with the following issues:
- rkt#1653 (whiteout-ed hard links)
- rkt#1617 (/dev/stdout)
@alban alban mentioned this issue Nov 23, 2015
Merged
@alban
Copy link
Member

alban commented Nov 24, 2015

appc/docker2aci#105 is merged and #1784 added the new docker2aci code into rkt. The image zopyx/xmldirector-plone works fine in rkt now.

$ sudo ./build-rkt-0.11.0+git/bin/rkt --insecure-skip-verify  \
        fetch docker://zopyx/xmldirector-plone
rkt: remote fetching from url docker://zopyx/xmldirector-plone
Downloading 48ecf305d2cf: [====================================] 32 B/32 B
Downloading ded7cd95e059: [====================================] 66.9 MB/66.9 MB
Downloading b6791c168813: [====================================] 32 B/32 B
Downloading fa5929338b78: [====================================] 99.4 MB/99.4 MB
Downloading b486fcef36eb: [====================================] 284 MB/284 MB 
Downloading c1dbd3a42b12: [====================================] 3.43 KB/3.43 KB
Downloading 67765b0e1857: [====================================] 32 B/32 B
Downloading 1a525f3e63df: [====================================] 32 B/32 B
Downloading e60b3290a68e: [====================================] 123 MB/123 MB 
Downloading 3cef685f61b7: [====================================] 365 B/365 B
Downloading 5a9d51cbcfc0: [====================================] 124 MB/124 MB 
Downloading 0eb44b73e3d3: [====================================] 196 B/196 B
Downloading 26bf62e64bb4: [====================================] 32 B/32 B
Downloading eb218f61198c: [====================================] 2.81 MB/2.81 MB
Downloading d8eaea66cf64: [====================================] 72.7 KB/72.7 KB
Downloading 3466cff753bd: [====================================] 38 MB/38 MB  
Downloading 3508bf6146e5: [====================================] 32 B/32 B
Downloading 88715e2416f6: [====================================] 873 B/873 B
Downloading e713f6d58073: [====================================] 257 KB/257 KB 
Downloading f3db17bfb8e7: [====================================] 156 MB/156 MB 
Downloading 2326b2e42cdc: [====================================] 32 B/32 B
Downloading f4a19130afb1: [====================================] 32 B/32 B
Downloading 832b5543eeec: [====================================] 359 B/359 B
Downloading 6fb78fc993fd: [====================================] 32 B/32 B
sha512-08225202dc341bbae70d2f005e68ed20

$ sudo ./build-rkt-0.11.0+git/bin/rkt --insecure-skip-verify run \
        --net=host \
        --volume dns,kind=host,source=/etc/resolv.conf
        sha512-08225202dc341bbae70d2f005e68ed20 \
        --mount volume=dns,target=/etc/resolv.conf
rkt: using image from local store for image name coreos.com/rkt/stage1-coreos:0.11.0+gitf0939ef
[260471.403382] sh[4]: From https://github.com/xml-director/xmldirector.plonecore
[260471.403783] sh[4]: baec170..5959b77  master     -> origin/master
[260471.404621] sh[4]: * [new branch]      ftp-fixes  -> origin/ftp-fixes
[260471.405257] sh[4]: 58de9b0..103f26f  plone5     -> origin/plone5
[260471.637149] sh[4]: Updating baec170..5959b77
[260471.637393] sh[4]: Fast-forward
...

It will be in the next release v0.12.0.

@alban alban closed this as completed Nov 24, 2015
alban added a commit to kinvolk/rkt that referenced this issue Nov 25, 2015
@alban @blixtra
Godeps: update to the last docker2aci
a80604c 
The last docker2aci should help with the following issues:
- rkt#1653 (whiteout-ed hard links)
- rkt#1617 (/dev/stdout)
blixtra pushed a commit to blixtra/rkt that referenced this issue Nov 25, 2015
@alban @blixtra
Godeps: update to the last docker2aci
a3b9c3f 
The last docker2aci should help with the following issues:
- rkt#1653 (whiteout-ed hard links)
- rkt#1617 (/dev/stdout)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@iaguis iaguis

Labels
depends-on/external kind/bug kind/support priority/P1 technology/docker
Projects
None yet
Milestone
v0.12.0
Development

No branches or pull requests
4 participants
@alban @iaguis @zopyx @jonboulle