Mesos:数据库中心操作系统


Mesos: The Datacenter Operang System David Greenberg Two Sigma Who am I? •  Architected project to build a massive Mesos cluster •  Building custom framework and leveraging open source The Plan What is Mesos? How can I use Mesos? How can I build on Mesos? What is Mesos? A long me ago… Are you done with the machine? I need to load my cards. Lol no; maybe tomorrow. 1957 Oh man! Let’s all share the computer, AT THE SAME TIME! John McCarthy Popularized Timesharing A long me ago… Are you done with the Hadoop cluster? I need to run my analycs job. Lol no; maybe tomorrow. 2010 Oh man! Let’s all share the cluster, AT THE SAME TIME! Ben Hindman Popularized Mesos Good ideas today mirror good ideas of yesteryear Mesos: an Operang System Isolaon Resource Sharing Common Infrastructure •  read(), write(), open() •  bind(), connect() •  apt-get, yum •  launchTask(), killTask(), statusUpdate() •  Docker Distributed System* Anatomy Workers Coordinator * Excluding peer-to-peer systems Stac Paroning Coordinator (Hadoop) Coordinator (Storm) Mesos (slaves) Mesos: a Level of Indirecon Coordinator Mesos (master) Coordinator Mesos (slaves) Mesos: a Level of Indirecon Coordinator Mesos (master) Coordinator Mesos (slaves) Mesos: a Level of Indirecon Coordinator Mesos (master) Coordinator Mesos (slaves) Mesos: a Level of Indirecon Coordinator Mesos (master) Coordinator Mesos (slaves) Mesos: a Level of Indirecon Coordinator Mesos (master) Coordinator Coordinang Execuon ≈ Scheduling Mesos (slaves) Coordinator Mesos (master) s/Coordinator/Scheduler/ Mesos (slaves) Scheduler Mesos (master) s/Coordinator/Scheduler/ Mesos (slaves) JobTracker (Scheduler) Mesos (master) Apache Hadoop Distributed System ≈ (Mesos) framework a Mesos framework is a distributed system that has a coordinator a Mesos framework is a distributed system that has a coordinator a Mesos framework is a distributed system that has a scheduler a a Mesos framework is an app for your cluster How can I use Mesos? Tons of Flexibility! Jenkins •  Connuous build server •  Just install a plugin! Hadoop •  Mul-cluster isolaon •  Fast startup •  Just run the repacked Cloudera CDH 4.2.1 MR1 distribuon for Mesos Marathon •  PaaS on Mesos •  init.d for the cluster •  Docker support •  Scales at the click of a buon •  Manages edge routers - HAProxy Chronos •  Distributed cron •  Supports job dependencies •  REST API Aurora •  Advanced PaaS on Mesos •  Powers Twier •  Supports phased rollouts •  Supports complex deployments Spark •  In memory Map Reduce, built for “Medium Data” •  Supports SQL as well as Java, Python, and Scala •  Designed for interacve analysis via REPL How do I use these? •  Free online interacve tutorials! – hp://mesosphere.io/learn •  Covers all of the previously menoned and many more How can I build on Mesos? Cluster Manager Status Quo Cluster Manager Applicaon/Human Specificaon The specificaon includes as much informaon as possible to assist the cluster manager in scheduling and execuon Cluster Manager Status Quo Cluster Manager Applicaon/Human Wait for task to be executed Cluster Manager Status Quo Cluster Manager Applicaon/Human Result Problems with Specificaons ① Hard to specify certain desires or constraints ② Hard to update specificaons dynamically as tasks execute and finish/fail An Alternave Model Mesos Scheduler request 3 CPUs 2 GB RAM •  A request is purposely simplified subset of a specificaon •  It is just the required resources at that point in me What should you do if you can’t sasfy a request? What should you do if you can’t sasfy a request? ①  Wait unl you can … What should you do if you can’t sasfy a request? ①  Wait unl you can … ②  Offer best you can immediately What should you do if you can’t sasfy a request? ①  Wait unl you can … ②  Offer best you can immediately Mesos Model Mesos Scheduler offer hostname 4 CPUs 4 GB RAM •  Resources are allocated via resource offers •  A resource offer represents a snapshot of available resources that a scheduler can use to run tasks An Analogue: non-blocking sockets Kernel Applicaon write(s, buffer, size);! An Analogue: non-blocking sockets Kernel Applicaon 42 of 100 bytes written!! offer hostname 4 CPUs 4 GB RAM offer hostname 4 CPUs 4 GB RAM offer hostname 4 CPUs 4 GB RAM Mesos Model Mesos Scheduler offer hostname 4 CPUs 4 GB RAM Scheduler uses the offers to decide what tasks to run Mesos Model Mesos Scheduler Scheduler uses the offers to decide what tasks to run “Two-level scheduling” task 3 CPUs 2 GB RAM Two-level Scheduling •  Mesos: controls resource allocaons to schedulers •  Schedulers: make decisions about what tasks to run given allocated resources Two-level Scheduling Elsewhere •  Mesos influenced by operang system supported user-space scheduling – E.g. green threads, gorounes •  Mesos is designed less like a “cluster manager” and more like an operang system (or kernel) Language Bindings Should I build it on Mesos? •  Theme of MesosCon: it’s easy to build frameworks •  Open source and proprietary frameworks are being created all the me – Two Sigma – Nelix – Twier – Hubspot But should I really build it on Mesos? •  Most users just use Marathon, Hadoop, Spark, and Chronos •  Why did we build our own? – Exoc workload The Plan, redux What is Mesos? How can I use Mesos? How can I build on Mesos? Quesons? Thank you
还剩59页未读

继续阅读

下载pdf到电脑,查找使用更方便

pdf的实际排版效果,会与网站的显示效果略有不同!!

需要 8 金币 [ 分享pdf获得金币 ] 2 人已下载

下载pdf

pdf贡献者

lelezi257

贡献于2015-04-12

下载需要 8 金币 [金币充值 ]
亲,您也可以通过 分享原创pdf 来获得金币奖励!
下载pdf