Container, rkt, Kubernetes 顾宜凡 (Yifan Gu) Software Engineer @ CoreOS github.com/yifan-gu 1. Containers, OCI, Appc 2. rkt 3. Kubernetes, and rkt + Kubernetes Overview Container Is HOT Container is not a new technology, Why? Container = Docker ? So, what is container? Control group - CPU - Memory - IO - Devices - ... - Namespaces - Network - IPC - ProcessID - ... So, what is container? Container = Docker ? No So, what is container? Container = Docker ? No Container = cgroup + namespace ? No So, what is container? Solaris Zones ~ 2005 FreeBSD ~ 2000 So, what is container? Container = package + runtime ! So, what is container? Container = package + runtime ! ● Easy packaging (build/push/pull) ● Isolated, controlled (run/stop) So, what is container? If it’s hot, then standardize it Container ~ 1950 Container ~ 2010 If it’s hot, then standardize it 2013.3 Docker 1.0 2016.32015.62014.12 Appc 0.1 OCI (Runtime Spec) OCI (Image Spec) Container Spec Timeline Open Container Specifications - Runtime Spec - config.json - runtime.json - rootfs - Image Spec - started from Docker v2 - absorb from Appc - discovery - signing - app configs Container Spec github.com/coreos/rkt rkt is a CLI for running app containers on Linux. rkt is designed to be secure, composable, and standards-based. rkt doesn’t require a long-running daemon and provides a powerful, pluggable, abstraction around isolation and runtime initialization. What is rkt ? ● GPG signatures to verify images ● SELinux contexts ● Can run containers in hypervisor ● Can do TPM measurements, provides a tamper-proof audit log How rkt does security ● Integrating well with init systems ● Aims to work well with other projects ● rkt has the concept of a “stage1”, which is a swappable component that actually runs the container ● Available stage1s ○ chroot ○ Linux namespaces (default) ○ LKVM How rkt does composability ● Implementation of AppC, a well defined spec ● Uses CNI for networking, common plumbing used by many other projects ● Can run docker images ● Will be fully OCI compliant How rkt does standards/compatibility Distributed Trusted Computing Distributed Trusted Computing Stack Distributed Trusted Computing Stack rkt internals modular architecture execution divided into stages stage0 → stage1 → stage2 ● Image discovery and fetching - Locate and download ACI and Docker images ● Unpacking/preparing the container manifest and filesystem ● Setting up network and namespace isolation ● Handle any needed runtime setup / features (e.g. systemd) ● Container entrypoint! Stage 0 Stage 1 Stage 2 rkt Stages $ rkt fetch example.com/redis $ rkt fetch docker://nginx $ rkt fetch https://my_web_container.aci $ rkt fetch ./my_container.aci Stage 0 - Fetch $ rkt fetch example.com/redis # GET https://example.com/redis # # GET https://example.com/redis.aci.asc # GET https://example.com/redis.aci Downloading signature: [=======================================] 287 B/287 B Downloading ACI: [=============================================] 10 MB/10 MB image: signature verified: Example sha512-... Stage 0 - Fetch $ rkt prepare example.com/redis uuid $ tree /var/lib/rkt/pods/prepared/uuid /var/lib/rkt/pods/prepared/uuid/ ├── appsinfo │ └── redis │ ├── manifest │ └── treeStoreID ├── overlay-prepared ├── pod ├── stage1 │ └── manifest └── stage1TreeStoreID Stage 0 - Prepare $ rkt run-prepared uuid Default: Systemd-nspawn ● Writes a unit file for each application based on its manifest ● Setup network namespaces (CNI) ● Handle mounts via systemd-nspawn (default) ● Hand off to systemd Stage 1 run-prepared $ rkt run-prepared uuid Default: Systemd-nspawn $ systemd-nspawn --boot --register=false --link-journal=try-guest --quiet -- uuid=a1caebb1-948b-4486-8133-bb21133a7090 --machine=rkt-a1caebb1- 948b-4486-8133-bb21133a7090 --directory=stage1/rootfs -- capability=CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE, CAP_FSETID,CAP_FOWNER,CAP_KILL,CAP_MKNOD,CAP_NET_RAW, CAP_NET_BIND_SERVICE,CAP_SETUID,CAP_SETGID,CAP_SETPCAP, CAP_SETFCAP,CAP_SYS_CHROOT -- --default-standard-output=tty --log- target=null --show-status=0 Stage 1 ● Distributed as a container image (rkt-fetchable， self- containerd) ● Run with no isolation on the host ● Support multiple stage1 for different purpose Examples ● stage1-coreos - Run all applications under systemd ● stage1-kvm - Run all applications under lkvm ● stage1-fly - Run an application under chroot “isolation” Stage 1 (Continued) ● App + some nice features: rkt───systemd-nspawn───systemd─┬─redis-server └─systemd-journal $ machinectl list rkt-uuid ... $ journalctl -M rkt-uuid ... _._ _.-``__ ＇＇-._ _.-`` `. `_. ＇＇-._ Redis 3.2.0 (00000000/0) .-`` .-```. ```＼/ _.,_ ＇＇-._ ( ＇ , .-` | `, ) Running in standalone mode |`-._`-...-` __...-.``-._|＇` _.-＇| Port: 6379 | `-._ `._ / _.-＇ | PID: 5 `-._ `-._ `-./ _.-＇ _.-＇ |`-._`-._ `-.__.-＇ _.-＇_.-＇| | `-._`-._ _.-＇_.-＇ | http://redis.io `-._ `-._`-.__.-＇_.-＇ _.-＇ |`-._`-._ `-.__.-＇ _.-＇_.-＇| | `-._`-._ _.-＇_.-＇ | `-._ `-._`-.__.-＇_.-＇ _.-＇ `-._ `-.__.-＇ _.-＇ `-._ _.-＇ `-.__.-＇ Stage 2 rkt (stage0) pod (stage1) bash/runit/systemd/... (invoking process) app1 (stage2) app2 (stage2) rkt Stages rkt is in production github.com/coreos/rkt Only Container Runtime Is Not Enough ● Fleet, ● Mesos (marathon) ● Docker Swarm ● Kubernetes ● ... Only Container Runtime Is Not Enough ● Pod (Co-located containers) ● Replication Controller (HA & Resize) ● Service (Service discovery, Load balancer) Kubernetes API Kubelet Kubelet Overview Kubelet Overview Kubelet Overview Started from nothing but Docker → Deep-coupled with Docker Kubelet -- Runtime Interface Started from nothing but Docker → Deep-coupled with Docker → started rkt integration → Pod level runtime interface Kubelet -- Runtime Interface Kubelet -- Runtime Interface ● GetPods() ● SyncPod() ● KillPod() ● GetPodStatus() ● ListImages() ● PullImage() ● RemoveImage() ● ImageStats() ● GetContainerLogs() ● ExecInContainer() ● ... Kubelet -- Runtime Interface ● GetPods() ● SyncPod(), declarative ● KillPod() ● GetPodStatus() ● ListImages() ● PullImage() ● RemoveImage() ● ImageStats() ● GetContainerLogs() ● ExecInContainer() ● ... Kubelet -- Runtime Interface func SyncPodIdeal (expectedPod, actualPod) { foreach container in actualPod { if container is not in expectedPod.Containers { KillContainer() } } foreach container in expectedPod { if container is not in actual.Containers { StartContainer() } } } Kubelet -- Runtime Interface func SyncPodLessIdeal (expectedPod, actualPod) { foreach container in actualPod { Has the container spec changed? Is the container healthy? if container is not in expectedPod.Containers { KillContainer() } } foreach container in expectedPod { if container is not in actualPod.Containers { Does the container needs to restart? Is the container a pod infra container? StartContainer() } } } Kubelet -- Runtime Interface func SyncPodRkt (expectedPod, actualPod) { foreach container in actualPod { Has the container spec changed? Is the container healthy? if container is not in expectedPod.Containers { goto restart } } foreach container in expectedPod { if container is not in actualPod.Containers { Does the container needs to restart? goto restart } } restart: RestartPod() } Kubelet -- Runtime Interface Pod level interface ● Simpler ● Coarse-grained ● Not every runtime implements “Pod” Container level interface ● More complexity in kubelet ● Doesn’t make sense to VM based runtime ● Fine-grained ● Runtime implementation can be easy Observation ● Improve extensibility: Easier container runtime integration. ● Improve feature velocity ● Improve code maintainability Container level, Pod level, or both? It’s under debate! https://github.com/kubernetes/kubernetes/pull/25899 Future of Runtime Interface ● Container is the future, standard is important ● rkt is a composable, secure container runtime ● Kubernetes, kubelet and container runtime interface Summary CoreOS is running the world’s containers We’re hiring: careers@coreos.com sales@coreos.com 90+ Projects on GitHub, 1,000+ Contributors coreos.com Support plans, training and more OPEN SOURCE ENTERPRISE