Skip to content
This repository has been archived by the owner on Feb 24, 2020. It is now read-only.

rkt fails on Debian GNU/Linux 7 (wheezy) with unclear error #1042

Closed
alban opened this issue Jun 15, 2015 · 20 comments
Closed

rkt fails on Debian GNU/Linux 7 (wheezy) with unclear error #1042

alban opened this issue Jun 15, 2015 · 20 comments
Assignees
@alban
Labels
area/usability component/stage1 kind/bug kind/friction priority/P0
Milestone
v0.9.0

Comments

@alban
Copy link
Member

alban commented Jun 15, 2015

Debian GNU/Linux 7 (wheezy)

It fails without error message:

$ sudo bin/rkt --insecure-skip-verify   --debug run -interactive docker://busybox
rkt: fetching image from file:///tmp/rkt-ci-debian78/bin/stage1.aci
rkt: found image in local store, skipping fetching from docker://busybox
2015/06/15 12:06:41 Preparing stage1
2015/06/15 12:06:45 Writing image manifest
2015/06/15 12:06:45 Loading image sha512-e17fc826c76437429a085f2c276d3feb4312dd67f57390cefaf582739e8c8c19
2015/06/15 12:06:46 Writing pod manifest
2015/06/15 12:06:46 Setting up stage1
2015/06/15 12:06:46 Wrote filesystem to /var/lib/rkt/pods/run/3393cb47-43c5-4185-ac68-c85b98692f33
2015/06/15 12:06:46 Pivoting to filesystem /var/lib/rkt/pods/run/3393cb47-43c5-4185-ac68-c85b98692f33
2015/06/15 12:06:46 Execing /init
Spawning container rkt-3393cb47-43c5-4185-ac68-c85b98692f33 on /var/lib/rkt/pods/run/3393cb47-43c5-4185-ac68-c85b98692f33/stage1/rootfs.
Press ^] three times within 1s to kill container.
$ echo $?
1

When using SYSTEMD_LOG_LEVEL=debug, the error message is unclear:

$ sudo SYSTEMD_LOG_LEVEL=debug bin/rkt --insecure-skip-verify   --debug run -interactive docker://busybox
(...)
Press ^] three times within 1s to kill container.
namespace helper failed with error code 1.

When using strace, we see that the nspawn error is related to cgroups:

[pid 19528] open("/proc/cgroups", O_RDONLY|O_CLOEXEC) = 5
[pid 19528] fstat(5, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[pid 19528] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f798f4d7000
[pid 19528] read(5, "#subsys_name\thierarchy\tnum_cgroups\tenabled\ncpuset\t4\t34\t1\ncpu\t2\t34\t1\ncpuacct\t3\t34\t1\nmemory\t0\t1\t0\ndevices\t5\t34\t1\nfreezer\t6\t34\t1\nnet_cls\t7\t3
4\t1\nblkio\t1\t34\t1\nperf_event\t9\t34\t1\nnet_prio\t8\t34\t1\n", 1024) = 188
[pid 19528] open("/dev/urandom", O_RDONLY|O_NOCTTY|O_CLOEXEC) = 8
[pid 19528] read(8, "\30\36\226\30\177\317[\372J\241\367\31h#\323t", 16) = 16
[pid 19528] close(8)                    = 0
[pid 19528] read(5, "", 1024)           = 0
[pid 19528] close(5)                    = 0
[pid 19528] munmap(0x7f798f4d7000, 4096) = 0
[pid 19528] readlinkat(AT_FDCWD, "/sys/fs/cgroup/blkio", 0x7f7990330920, 99) = -1 ENOENT (No such file or directory)
[pid 19528] writev(2, [{"Failed to read link /sys/fs/cgroup/blkio: No such file or directory", 67}, {"\n", 1}], 2) = 68
[pid 19528] exit_group(1)               = ?

Indeed, the cgroup subsystems are not mounted:

$ cat /proc/cgroups 
#subsys_name    hierarchy       num_cgroups     enabled
cpuset  4       34      1
cpu     2       34      1
cpuacct 3       34      1
memory  0       1       0
devices 5       34      1
freezer 6       34      1
net_cls 7       34      1
blkio   1       34      1
perf_event      9       34      1
net_prio        8       34      1
$ ls -l /sys/fs/cgroup/
total 0
drwxr-xr-x 2 root root 60 Jun 15 10:21 cgmanager

Similar issue to #996.

The solution might be on https://wiki.debian.org/LXC#Prepare_the_host but rkt+nspawn should give better error messages.
@iaguis
Copy link
Member

iaguis commented Jun 15, 2015

Upstream issue: systemd/systemd#207

@jonboulle
Copy link
Contributor

Does this affect stock systemd-nspawn on wheezy?

@kler
Copy link

kler commented Jun 15, 2015

Don't know if interesting for you, but it fails in the same way when following the README.md instructions and using vagrant (in VirtualBox on OS X).

@jonboulle
Copy link
Contributor

thanks for the data point.
@alban @iaguis is this affecting nspawn more generally?

@iaguis
Copy link
Member

iaguis commented Jun 15, 2015

Yes. See systemd/systemd#207 and systemd/systemd#210

@jonboulle
Copy link
Contributor

@iaguis I should have been more specific: I was asking about the cgroups thing causing systemd-nspawn to always fail, rather than just the diagnostics issue of it not printing output when things go wrong.

@iaguis
Copy link
Member

iaguis commented Jun 16, 2015

Oh, sorry. Yes, nspawn alone fails too. It expects cgroups being mounted in /sys/fs/cgroup if /proc/cgroups says they're enabled.

@alban
Copy link
Member Author

alban commented Jun 18, 2015

The missing error message is fixed by systemd/systemd#207 systemd/systemd#272. It will be in systemd-v221.

I now have correct error messages:

Spawning container rkt-8fa79c17-35ea-48bb-b13a-9d1b2257fef1 on /tmp/datadir-712183490/pods/run/8fa79c17-35ea-48bb-b13a-9d1b2257fef1/stage1/rootfs.
Press ^] three times within 1s to kill container.
/etc/localtime is not a symlink, not updating container timezone.

Failed to read link /sys/fs/cgroup/freezer: No such file or directory

I don't know if we should do something else about this.

@iaguis
Copy link
Member

iaguis commented Jun 18, 2015

Maybe we should add something to the documentation saying that rkt expects the cgroup hierarchy mounted in /sys/fs/cgroup à la https://wiki.debian.org/LXC#Prepare_the_host

@jonboulle
Copy link
Contributor

+1
On Jun 18, 2015 6:30 AM, "Iago López Galeiras" notifications@github.com
wrote:

Maybe we should add something to the documentation saying that rkt expects
the cgroup hierarchy mounted in /sys/fs/cgroup and à la
https://wiki.debian.org/LXC#Prepare_the_host

—
Reply to this email directly or view it on GitHub
#1042 (comment).

@alban alban mentioned this issue Jun 22, 2015
Closed
@kontsevoy
Copy link

Even when I prepare the host according to https://wiki.debian.org/LXC#Prepare_the_host I still get a similar message. How did it work before?

I am on Ubuntu 14.02 LTS. Prior to rebuilding the latest rkt everything worked out of the box. Then I started getting an error similar to what's described above. My /sys/fs/cgroup has always been empty.

So I prepared host according to the Debian link.
Now here is my `/etc/fstab':

LABEL=cloudimg-rootfs   /    ext4   defaults    0 0
cgroup  /sys/fs/cgroup  cgroup  defaults  0   0

Here's my /sys/fs/cgroup:

sudo tree /sys/fs/cgroup/
/sys/fs/cgroup/
├── blkio.io_merged
├── blkio.io_merged_recursive
├── blkio.io_queued
├── blkio.io_queued_recursive
├── blkio.io_service_bytes
├── blkio.io_service_bytes_recursive
├── blkio.io_serviced
├── blkio.io_serviced_recursive
├── blkio.io_service_time
├── blkio.io_service_time_recursive
├── blkio.io_wait_time
├── blkio.io_wait_time_recursive
├── blkio.leaf_weight
├── blkio.leaf_weight_device
├── blkio.reset_stats
├── blkio.sectors
├── blkio.sectors_recursive
├── blkio.throttle.io_service_bytes
├── blkio.throttle.io_serviced
├── blkio.throttle.read_bps_device
├── blkio.throttle.read_iops_device
├── blkio.throttle.write_bps_device
├── blkio.throttle.write_iops_device
├── blkio.time
├── blkio.time_recursive
├── blkio.weight
├── blkio.weight_device
├── cgroup.clone_children
├── cgroup.event_control
├── cgroup.procs
├── cgroup.sane_behavior
├── cpuacct.stat
├── cpuacct.usage
├── cpuacct.usage_percpu
├── cpu.cfs_period_us
├── cpu.cfs_quota_us
├── cpuset.cpu_exclusive
├── cpuset.cpus
├── cpuset.mem_exclusive
├── cpuset.mem_hardwall
├── cpuset.memory_migrate
├── cpuset.memory_pressure
├── cpuset.memory_pressure_enabled
├── cpuset.memory_spread_page
├── cpuset.memory_spread_slab
├── cpuset.mems
├── cpuset.sched_load_balance
├── cpuset.sched_relax_domain_level
├── cpu.shares
├── cpu.stat
├── devices.allow
├── devices.deny
├── devices.list
├── hugetlb.2MB.failcnt
├── hugetlb.2MB.limit_in_bytes
├── hugetlb.2MB.max_usage_in_bytes
├── hugetlb.2MB.usage_in_bytes
├── memory.failcnt
├── memory.force_empty
├── memory.kmem.failcnt
├── memory.kmem.limit_in_bytes
├── memory.kmem.max_usage_in_bytes
├── memory.kmem.slabinfo
├── memory.kmem.tcp.failcnt
├── memory.kmem.tcp.limit_in_bytes
├── memory.kmem.tcp.max_usage_in_bytes
├── memory.kmem.tcp.usage_in_bytes
├── memory.kmem.usage_in_bytes
├── memory.limit_in_bytes
├── memory.max_usage_in_bytes
├── memory.move_charge_at_immigrate
├── memory.numa_stat
├── memory.oom_control
├── memory.pressure_level
├── memory.soft_limit_in_bytes
├── memory.stat
├── memory.swappiness
├── memory.usage_in_bytes
├── memory.use_hierarchy
├── notify_on_release
├── release_agent
├── systemd
│   ├── cgroup.clone_children
│   ├── cgroup.event_control
│   ├── cgroup.procs
│   ├── cgroup.sane_behavior
│   ├── notify_on_release
│   ├── release_agent
│   ├── tasks
│   └── user
│       ├── 1000.user
│       │   ├── 1.session
│       │   │   ├── cgroup.clone_children
│       │   │   ├── cgroup.event_control
│       │   │   ├── cgroup.procs
│       │   │   ├── notify_on_release
│       │   │   └── tasks
│       │   ├── 2.session
│       │   │   ├── cgroup.clone_children
│       │   │   ├── cgroup.event_control
│       │   │   ├── cgroup.procs
│       │   │   ├── notify_on_release
│       │   │   └── tasks
│       │   ├── cgroup.clone_children
│       │   ├── cgroup.event_control
│       │   ├── cgroup.procs
│       │   ├── notify_on_release
│       │   └── tasks
│       ├── cgroup.clone_children
│       ├── cgroup.event_control
│       ├── cgroup.procs
│       ├── notify_on_release
│       └── tasks
├── tasks
└── user
    ├── 1000.user
    │   ├── 2.session
    │   │   ├── blkio.io_merged
    │   │   ├── blkio.io_merged_recursive
    │   │   ├── blkio.io_queued
    │   │   ├── blkio.io_queued_recursive
    │   │   ├── blkio.io_service_bytes
    │   │   ├── blkio.io_service_bytes_recursive
    │   │   ├── blkio.io_serviced
    │   │   ├── blkio.io_serviced_recursive
    │   │   ├── blkio.io_service_time
    │   │   ├── blkio.io_service_time_recursive
    │   │   ├── blkio.io_wait_time
    │   │   ├── blkio.io_wait_time_recursive
    │   │   ├── blkio.leaf_weight
    │   │   ├── blkio.leaf_weight_device
    │   │   ├── blkio.reset_stats
    │   │   ├── blkio.sectors
    │   │   ├── blkio.sectors_recursive
    │   │   ├── blkio.throttle.io_service_bytes
    │   │   ├── blkio.throttle.io_serviced
    │   │   ├── blkio.throttle.read_bps_device
    │   │   ├── blkio.throttle.read_iops_device
    │   │   ├── blkio.throttle.write_bps_device
    │   │   ├── blkio.throttle.write_iops_device
    │   │   ├── blkio.time
    │   │   ├── blkio.time_recursive
    │   │   ├── blkio.weight
    │   │   ├── blkio.weight_device
    │   │   ├── cgroup.clone_children
    │   │   ├── cgroup.event_control
    │   │   ├── cgroup.procs
    │   │   ├── cpuacct.stat
    │   │   ├── cpuacct.usage
    │   │   ├── cpuacct.usage_percpu
    │   │   ├── cpu.cfs_period_us
    │   │   ├── cpu.cfs_quota_us
    │   │   ├── cpuset.cpu_exclusive
    │   │   ├── cpuset.cpus
    │   │   ├── cpuset.mem_exclusive
    │   │   ├── cpuset.mem_hardwall
    │   │   ├── cpuset.memory_migrate
    │   │   ├── cpuset.memory_pressure
    │   │   ├── cpuset.memory_spread_page
    │   │   ├── cpuset.memory_spread_slab
    │   │   ├── cpuset.mems
    │   │   ├── cpuset.sched_load_balance
    │   │   ├── cpuset.sched_relax_domain_level
    │   │   ├── cpu.shares
    │   │   ├── cpu.stat
    │   │   ├── devices.allow
    │   │   ├── devices.deny
    │   │   ├── devices.list
    │   │   ├── freezer.parent_freezing
    │   │   ├── freezer.self_freezing
    │   │   ├── freezer.state
    │   │   ├── hugetlb.2MB.failcnt
    │   │   ├── hugetlb.2MB.limit_in_bytes
    │   │   ├── hugetlb.2MB.max_usage_in_bytes
    │   │   ├── hugetlb.2MB.usage_in_bytes
    │   │   ├── memory.failcnt
    │   │   ├── memory.force_empty
    │   │   ├── memory.kmem.failcnt
    │   │   ├── memory.kmem.limit_in_bytes
    │   │   ├── memory.kmem.max_usage_in_bytes
    │   │   ├── memory.kmem.slabinfo
    │   │   ├── memory.kmem.tcp.failcnt
    │   │   ├── memory.kmem.tcp.limit_in_bytes
    │   │   ├── memory.kmem.tcp.max_usage_in_bytes
    │   │   ├── memory.kmem.tcp.usage_in_bytes
    │   │   ├── memory.kmem.usage_in_bytes
    │   │   ├── memory.limit_in_bytes
    │   │   ├── memory.max_usage_in_bytes
    │   │   ├── memory.move_charge_at_immigrate
    │   │   ├── memory.numa_stat
    │   │   ├── memory.oom_control
    │   │   ├── memory.pressure_level
    │   │   ├── memory.soft_limit_in_bytes
    │   │   ├── memory.stat
    │   │   ├── memory.swappiness
    │   │   ├── memory.usage_in_bytes
    │   │   ├── memory.use_hierarchy
    │   │   ├── notify_on_release
    │   │   ├── system.slice
    │   │   │   ├── blkio.io_merged
    │   │   │   ├── blkio.io_merged_recursive
    │   │   │   ├── blkio.io_queued
    │   │   │   ├── blkio.io_queued_recursive
    │   │   │   ├── blkio.io_service_bytes
    │   │   │   ├── blkio.io_service_bytes_recursive
    │   │   │   ├── blkio.io_serviced
    │   │   │   ├── blkio.io_serviced_recursive
    │   │   │   ├── blkio.io_service_time
    │   │   │   ├── blkio.io_service_time_recursive
    │   │   │   ├── blkio.io_wait_time
    │   │   │   ├── blkio.io_wait_time_recursive
    │   │   │   ├── blkio.leaf_weight
    │   │   │   ├── blkio.leaf_weight_device
    │   │   │   ├── blkio.reset_stats
    │   │   │   ├── blkio.sectors
    │   │   │   ├── blkio.sectors_recursive
    │   │   │   ├── blkio.throttle.io_service_bytes
    │   │   │   ├── blkio.throttle.io_serviced
    │   │   │   ├── blkio.throttle.read_bps_device
    │   │   │   ├── blkio.throttle.read_iops_device
    │   │   │   ├── blkio.throttle.write_bps_device
    │   │   │   ├── blkio.throttle.write_iops_device
    │   │   │   ├── blkio.time
    │   │   │   ├── blkio.time_recursive
    │   │   │   ├── blkio.weight
    │   │   │   ├── blkio.weight_device
    │   │   │   ├── cgroup.clone_children
    │   │   │   ├── cgroup.event_control
    │   │   │   ├── cgroup.procs
    │   │   │   ├── cpuacct.stat
    │   │   │   ├── cpuacct.usage
    │   │   │   ├── cpuacct.usage_percpu
    │   │   │   ├── cpu.cfs_period_us
    │   │   │   ├── cpu.cfs_quota_us
    │   │   │   ├── cpuset.cpu_exclusive
    │   │   │   ├── cpuset.cpus
    │   │   │   ├── cpuset.mem_exclusive
    │   │   │   ├── cpuset.mem_hardwall
    │   │   │   ├── cpuset.memory_migrate
    │   │   │   ├── cpuset.memory_pressure
    │   │   │   ├── cpuset.memory_spread_page
    │   │   │   ├── cpuset.memory_spread_slab
    │   │   │   ├── cpuset.mems
    │   │   │   ├── cpuset.sched_load_balance
    │   │   │   ├── cpuset.sched_relax_domain_level
    │   │   │   ├── cpu.shares
    │   │   │   ├── cpu.stat
    │   │   │   ├── devices.allow
    │   │   │   ├── devices.deny
    │   │   │   ├── devices.list
    │   │   │   ├── freezer.parent_freezing
    │   │   │   ├── freezer.self_freezing
    │   │   │   ├── freezer.state
    │   │   │   ├── hugetlb.2MB.failcnt
    │   │   │   ├── hugetlb.2MB.limit_in_bytes
    │   │   │   ├── hugetlb.2MB.max_usage_in_bytes
    │   │   │   ├── hugetlb.2MB.usage_in_bytes
    │   │   │   ├── memory.failcnt
    │   │   │   ├── memory.force_empty
    │   │   │   ├── memory.kmem.failcnt
    │   │   │   ├── memory.kmem.limit_in_bytes
    │   │   │   ├── memory.kmem.max_usage_in_bytes
    │   │   │   ├── memory.kmem.slabinfo
    │   │   │   ├── memory.kmem.tcp.failcnt
    │   │   │   ├── memory.kmem.tcp.limit_in_bytes
    │   │   │   ├── memory.kmem.tcp.max_usage_in_bytes
    │   │   │   ├── memory.kmem.tcp.usage_in_bytes
    │   │   │   ├── memory.kmem.usage_in_bytes
    │   │   │   ├── memory.limit_in_bytes
    │   │   │   ├── memory.max_usage_in_bytes
    │   │   │   ├── memory.move_charge_at_immigrate
    │   │   │   ├── memory.numa_stat
    │   │   │   ├── memory.oom_control
    │   │   │   ├── memory.pressure_level
    │   │   │   ├── memory.soft_limit_in_bytes
    │   │   │   ├── memory.stat
    │   │   │   ├── memory.swappiness
    │   │   │   ├── memory.usage_in_bytes
    │   │   │   ├── memory.use_hierarchy
    │   │   │   ├── notify_on_release
    │   │   │   ├── sha512-1eba37d9b344b33d272181e176da111e.service
    │   │   │   │   ├── blkio.io_merged
    │   │   │   │   ├── blkio.io_merged_recursive
    │   │   │   │   ├── blkio.io_queued
    │   │   │   │   ├── blkio.io_queued_recursive
    │   │   │   │   ├── blkio.io_service_bytes
    │   │   │   │   ├── blkio.io_service_bytes_recursive
    │   │   │   │   ├── blkio.io_serviced
    │   │   │   │   ├── blkio.io_serviced_recursive
    │   │   │   │   ├── blkio.io_service_time
    │   │   │   │   ├── blkio.io_service_time_recursive
    │   │   │   │   ├── blkio.io_wait_time
    │   │   │   │   ├── blkio.io_wait_time_recursive
    │   │   │   │   ├── blkio.leaf_weight
    │   │   │   │   ├── blkio.leaf_weight_device
    │   │   │   │   ├── blkio.reset_stats
    │   │   │   │   ├── blkio.sectors
    │   │   │   │   ├── blkio.sectors_recursive
    │   │   │   │   ├── blkio.throttle.io_service_bytes
    │   │   │   │   ├── blkio.throttle.io_serviced
    │   │   │   │   ├── blkio.throttle.read_bps_device
    │   │   │   │   ├── blkio.throttle.read_iops_device
    │   │   │   │   ├── blkio.throttle.write_bps_device
    │   │   │   │   ├── blkio.throttle.write_iops_device
    │   │   │   │   ├── blkio.time
    │   │   │   │   ├── blkio.time_recursive
    │   │   │   │   ├── blkio.weight
    │   │   │   │   ├── blkio.weight_device
    │   │   │   │   ├── cgroup.clone_children
    │   │   │   │   ├── cgroup.event_control
    │   │   │   │   ├── cgroup.procs
    │   │   │   │   ├── cpuacct.stat
    │   │   │   │   ├── cpuacct.usage
    │   │   │   │   ├── cpuacct.usage_percpu
    │   │   │   │   ├── cpu.cfs_period_us
    │   │   │   │   ├── cpu.cfs_quota_us
    │   │   │   │   ├── cpuset.cpu_exclusive
    │   │   │   │   ├── cpuset.cpus
    │   │   │   │   ├── cpuset.mem_exclusive
    │   │   │   │   ├── cpuset.mem_hardwall
    │   │   │   │   ├── cpuset.memory_migrate
    │   │   │   │   ├── cpuset.memory_pressure
    │   │   │   │   ├── cpuset.memory_spread_page
    │   │   │   │   ├── cpuset.memory_spread_slab
    │   │   │   │   ├── cpuset.mems
    │   │   │   │   ├── cpuset.sched_load_balance
    │   │   │   │   ├── cpuset.sched_relax_domain_level
    │   │   │   │   ├── cpu.shares
    │   │   │   │   ├── cpu.stat
    │   │   │   │   ├── devices.allow
    │   │   │   │   ├── devices.deny
    │   │   │   │   ├── devices.list
    │   │   │   │   ├── freezer.parent_freezing
    │   │   │   │   ├── freezer.self_freezing
    │   │   │   │   ├── freezer.state
    │   │   │   │   ├── hugetlb.2MB.failcnt
    │   │   │   │   ├── hugetlb.2MB.limit_in_bytes
    │   │   │   │   ├── hugetlb.2MB.max_usage_in_bytes
    │   │   │   │   ├── hugetlb.2MB.usage_in_bytes
    │   │   │   │   ├── memory.failcnt
    │   │   │   │   ├── memory.force_empty
    │   │   │   │   ├── memory.kmem.failcnt
    │   │   │   │   ├── memory.kmem.limit_in_bytes
    │   │   │   │   ├── memory.kmem.max_usage_in_bytes
    │   │   │   │   ├── memory.kmem.slabinfo
    │   │   │   │   ├── memory.kmem.tcp.failcnt
    │   │   │   │   ├── memory.kmem.tcp.limit_in_bytes
    │   │   │   │   ├── memory.kmem.tcp.max_usage_in_bytes
    │   │   │   │   ├── memory.kmem.tcp.usage_in_bytes
    │   │   │   │   ├── memory.kmem.usage_in_bytes
    │   │   │   │   ├── memory.limit_in_bytes
    │   │   │   │   ├── memory.max_usage_in_bytes
    │   │   │   │   ├── memory.move_charge_at_immigrate
    │   │   │   │   ├── memory.numa_stat
    │   │   │   │   ├── memory.oom_control
    │   │   │   │   ├── memory.pressure_level
    │   │   │   │   ├── memory.soft_limit_in_bytes
    │   │   │   │   ├── memory.stat
    │   │   │   │   ├── memory.swappiness
    │   │   │   │   ├── memory.usage_in_bytes
    │   │   │   │   ├── memory.use_hierarchy
    │   │   │   │   ├── notify_on_release
    │   │   │   │   └── tasks
    │   │   │   └── tasks
    │   │   └── tasks
    │   ├── blkio.io_merged
    │   ├── blkio.io_merged_recursive
    │   ├── blkio.io_queued
    │   ├── blkio.io_queued_recursive
    │   ├── blkio.io_service_bytes
    │   ├── blkio.io_service_bytes_recursive
    │   ├── blkio.io_serviced
    │   ├── blkio.io_serviced_recursive
    │   ├── blkio.io_service_time
    │   ├── blkio.io_service_time_recursive
    │   ├── blkio.io_wait_time
    │   ├── blkio.io_wait_time_recursive
    │   ├── blkio.leaf_weight
    │   ├── blkio.leaf_weight_device
    │   ├── blkio.reset_stats
    │   ├── blkio.sectors
    │   ├── blkio.sectors_recursive
    │   ├── blkio.throttle.io_service_bytes
    │   ├── blkio.throttle.io_serviced
    │   ├── blkio.throttle.read_bps_device
    │   ├── blkio.throttle.read_iops_device
    │   ├── blkio.throttle.write_bps_device
    │   ├── blkio.throttle.write_iops_device
    │   ├── blkio.time
    │   ├── blkio.time_recursive
    │   ├── blkio.weight
    │   ├── blkio.weight_device
    │   ├── cgroup.clone_children
    │   ├── cgroup.event_control
    │   ├── cgroup.procs
    │   ├── cpuacct.stat
    │   ├── cpuacct.usage
    │   ├── cpuacct.usage_percpu
    │   ├── cpu.cfs_period_us
    │   ├── cpu.cfs_quota_us
    │   ├── cpuset.cpu_exclusive
    │   ├── cpuset.cpus
    │   ├── cpuset.mem_exclusive
    │   ├── cpuset.mem_hardwall
    │   ├── cpuset.memory_migrate
    │   ├── cpuset.memory_pressure
    │   ├── cpuset.memory_spread_page
    │   ├── cpuset.memory_spread_slab
    │   ├── cpuset.mems
    │   ├── cpuset.sched_load_balance
    │   ├── cpuset.sched_relax_domain_level
    │   ├── cpu.shares
    │   ├── cpu.stat
    │   ├── devices.allow
    │   ├── devices.deny
    │   ├── devices.list
    │   ├── freezer.parent_freezing
    │   ├── freezer.self_freezing
    │   ├── freezer.state
    │   ├── hugetlb.2MB.failcnt
    │   ├── hugetlb.2MB.limit_in_bytes
    │   ├── hugetlb.2MB.max_usage_in_bytes
    │   ├── hugetlb.2MB.usage_in_bytes
    │   ├── memory.failcnt
    │   ├── memory.force_empty
    │   ├── memory.kmem.failcnt
    │   ├── memory.kmem.limit_in_bytes
    │   ├── memory.kmem.max_usage_in_bytes
    │   ├── memory.kmem.slabinfo
    │   ├── memory.kmem.tcp.failcnt
    │   ├── memory.kmem.tcp.limit_in_bytes
    │   ├── memory.kmem.tcp.max_usage_in_bytes
    │   ├── memory.kmem.tcp.usage_in_bytes
    │   ├── memory.kmem.usage_in_bytes
    │   ├── memory.limit_in_bytes
    │   ├── memory.max_usage_in_bytes
    │   ├── memory.move_charge_at_immigrate
    │   ├── memory.numa_stat
    │   ├── memory.oom_control
    │   ├── memory.pressure_level
    │   ├── memory.soft_limit_in_bytes
    │   ├── memory.stat
    │   ├── memory.swappiness
    │   ├── memory.usage_in_bytes
    │   ├── memory.use_hierarchy
    │   ├── notify_on_release
    │   └── tasks
    ├── blkio.io_merged
    ├── blkio.io_merged_recursive
    ├── blkio.io_queued
    ├── blkio.io_queued_recursive
    ├── blkio.io_service_bytes
    ├── blkio.io_service_bytes_recursive
    ├── blkio.io_serviced
    ├── blkio.io_serviced_recursive
    ├── blkio.io_service_time
    ├── blkio.io_service_time_recursive
    ├── blkio.io_wait_time
    ├── blkio.io_wait_time_recursive
    ├── blkio.leaf_weight
    ├── blkio.leaf_weight_device
    ├── blkio.reset_stats
    ├── blkio.sectors
    ├── blkio.sectors_recursive
    ├── blkio.throttle.io_service_bytes
    ├── blkio.throttle.io_serviced
    ├── blkio.throttle.read_bps_device
    ├── blkio.throttle.read_iops_device
    ├── blkio.throttle.write_bps_device
    ├── blkio.throttle.write_iops_device
    ├── blkio.time
    ├── blkio.time_recursive
    ├── blkio.weight
    ├── blkio.weight_device
    ├── cgroup.clone_children
    ├── cgroup.event_control
    ├── cgroup.procs
    ├── cpuacct.stat
    ├── cpuacct.usage
    ├── cpuacct.usage_percpu
    ├── cpu.cfs_period_us
    ├── cpu.cfs_quota_us
    ├── cpuset.cpu_exclusive
    ├── cpuset.cpus
    ├── cpuset.mem_exclusive
    ├── cpuset.mem_hardwall
    ├── cpuset.memory_migrate
    ├── cpuset.memory_pressure
    ├── cpuset.memory_spread_page
    ├── cpuset.memory_spread_slab
    ├── cpuset.mems
    ├── cpuset.sched_load_balance
    ├── cpuset.sched_relax_domain_level
    ├── cpu.shares
    ├── cpu.stat
    ├── devices.allow
    ├── devices.deny
    ├── devices.list
    ├── freezer.parent_freezing
    ├── freezer.self_freezing
    ├── freezer.state
    ├── hugetlb.2MB.failcnt
    ├── hugetlb.2MB.limit_in_bytes
    ├── hugetlb.2MB.max_usage_in_bytes
    ├── hugetlb.2MB.usage_in_bytes
    ├── memory.failcnt
    ├── memory.force_empty
    ├── memory.kmem.failcnt
    ├── memory.kmem.limit_in_bytes
    ├── memory.kmem.max_usage_in_bytes
    ├── memory.kmem.slabinfo
    ├── memory.kmem.tcp.failcnt
    ├── memory.kmem.tcp.limit_in_bytes
    ├── memory.kmem.tcp.max_usage_in_bytes
    ├── memory.kmem.tcp.usage_in_bytes
    ├── memory.kmem.usage_in_bytes
    ├── memory.limit_in_bytes
    ├── memory.max_usage_in_bytes
    ├── memory.move_charge_at_immigrate
    ├── memory.numa_stat
    ├── memory.oom_control
    ├── memory.pressure_level
    ├── memory.soft_limit_in_bytes
    ├── memory.stat
    ├── memory.swappiness
    ├── memory.usage_in_bytes
    ├── memory.use_hierarchy
    ├── notify_on_release
    └── tasks

10 directories, 519 files

Trying to run etcd.aci:

sudo ./rkt --insecure-skip-verify --mds-register=false run etcd.aci

Output:

2015/07/19 22:24:22 Preparing stage1
2015/07/19 22:24:22 Writing image manifest
2015/07/19 22:24:22 Loading image sha512-1eba37d9b344b33d272181e176da111ef2fdd4958b88ba4071e56db9ac07cf62
2015/07/19 22:24:22 Writing pod manifest
2015/07/19 22:24:22 Setting up stage1
2015/07/19 22:24:22 Wrote filesystem to /var/lib/rkt/pods/run/8fdee4d6-6ec3-40b8-ba4f-ea1904211cb2
2015/07/19 22:24:22 Pivoting to filesystem /var/lib/rkt/pods/run/8fdee4d6-6ec3-40b8-ba4f-ea1904211cb2
2015/07/19 22:24:22 Execing /init
Failed to read link /sys/fs/cgroup/freezer: No such file or directory

@kontsevoy
Copy link

Did something change? Previously everything worked without messing with mounting cgroup FS.

iaguis added a commit to endocode/rkt that referenced this issue Jul 20, 2015
@iaguis
scripts/vagrant: use ubuntu 15.04
b2229d8 
Ubuntu 14.04 has some problems with incorrectly set up cgroups (rkt#1042).
@iaguis
Copy link
Member

iaguis commented Jul 20, 2015

That cgroups hierarchy doesn't look right, there should be a directory per controller and there all the controller knobs are in /sys/fs/cgroup/.

Did something change? Previously everything worked without messing with mounting cgroup FS.

What's the last version that worked? In rkt v0.6.1 we updated stage1's systemd from v215 to v220 and v218 introduced the cgroup-tree creation code.

That might be the cause.

@rimmington
Copy link

To add another data point: 0.5.6 works with the provided 14.04 Vagrantfile, 0.6.1 does not. When I modified the Vagrantfile to use 15.04 (ubuntu/vivid64), 0.6.1 seems to work fine.

@nstott
Copy link

nstott commented Aug 24, 2015

I'm seeing this same thing, trying to get ubuntu 14.04 working, and seeing the cgroups set up as @kontsevoy describes

@nstott
Copy link

nstott commented Aug 25, 2015

installing the package cgroup-lite seems to help and sets up the cgroups as is expected

@iaguis
Copy link
Member

iaguis commented Aug 26, 2015

@nstott: so installing that package makes rkt work?

@nstott
Copy link

nstott commented Aug 26, 2015

@iaguis yeah, it set up the cgroups as a controler per directory,
I installed cgroup-lite (default config) instead of following the 'preparing a host' instructions that had you edit fstab and mount the cgroups

i was able to start and run rkt containers from the command line properly after that

i did have a problem (probably a seperate problem) when trying to start a container via upstart though, and got this

2015/08/25 15:06:31 Preparing stage1
2015/08/25 15:06:31 Writing image manifest
2015/08/25 15:06:32 Loading image sha512-f2bf68bbdadf9250cd4cfc5925607a37fd1f64c8aade3b496f549f5ade124432
2015/08/25 15:06:34 Writing pod manifest
2015/08/25 15:06:34 Setting up stage1
2015/08/25 15:06:34 Wrote filesystem to /var/lib/rkt/pods/run/12124274-1b09-49d3-9da4-d369cc04e1a5
2015/08/25 15:06:34 Pivoting to filesystem /var/lib/rkt/pods/run/12124274-1b09-49d3-9da4-d369cc04e1a5
2015/08/25 15:06:34 Execing /init
[!!!!!!] Failed to allocate manager object, freezing.

but that might be a seperate problem entirely

@rimmington
Copy link

I can confirm installing cgroup-lite and rebooting works with rkt 0.8.0 and Ubuntu 14.04. Thanks @nstott!

@jonboulle jonboulle added this to the v0.9.0 milestone Sep 21, 2015
@jonboulle jonboulle mentioned this issue Sep 23, 2015
Merged
@iaguis
Copy link
Member

iaguis commented Sep 24, 2015

After #1437, installing cgroup-lite should NOT be necessary.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@alban alban

Labels
area/usability component/stage1 kind/bug kind/friction priority/P0
Projects
None yet
Milestone
v0.9.0
Development

No branches or pull requests
7 participants
@alban @rimmington @nstott @kontsevoy @iaguis @kler @jonboulle