《架构师》2014年10月刊


10 2014 QCon 11 1 QCon Twitter Raffi Mars Intel Andrew Xia Spark Steve Facebook Hedger Wang Kevin Roy …… QCon QCon QCon —— InfoQ —— QConArchSummit QClub InfoQ QCon —— shy —— “” “”InfoQ “” editors@cn.infoq.com Databricks Spark WEB WEB WEB Spark RDD TDW ——Shuffle SNG “” NoSQL | People IT IT 10 16 -18 QCon SDN InfoQ InfoQ TCP/IP // SDN VNF SDN InfoQ 67 19 /SDN | People InfoQ IT IT IT IBM IT IT IT GoogleAmazon IT IT IT IT IT IT IT IT GoogleFacebook IT IT IT | People InfoQ OpenFlow Controller Google Facebook OpenFlow Controller Controller Controller —— Controller InfoQ IT —— IT | People / VC InfoQ | Opinion Databricks Spark DataBricksErlang/OTP Erlang InfoQ Spark InfoQ Storm Spark Storm Spark SQL Storm Spark Spark Streaming Spark Streaming Storm Spark Streaming Storm Storm Spark Streaming RDD lineage DAG replication straggler Spark RDD Spark Storm Spark Streaming Matei Zaharia An Architecture for Fast and General Data Processing on Large Clusters | Opinion InfoQ2014 Databricks 2013 Spark Spark Spark Spark Spark Databricks InfoQ Spark Spark 1. Spark 2. immutability MapReduce / Hadoop Spark Akka actor modelSpark Akka Erlang InfoQSpark Hadoop 1. Spark | Opinion 2. RDD 3. Spark contributor Spark Spark “” :-) scale out 4. Hadoop Spark ETL Spark ClouderaMapRPivotalHortonworks Hadoop Spark InfoQSpark Spark 1. Scala 2. Spark eBay Spark InfoQ Spark Spark SQL 1.2 roadmap API API Spark SQL Cassandra HBase InfoQ Spark Spark • standaloneMesos YARN • Yahoo QoS eBay • KafkaFlumeHive JDBC Spark Spark | Opinion RDD 1.2 Spark SQL API • cache Spark • Spark • JVM full GC @ Databricks Spark U CD RAM O'Reilly Media Andy Oram AerospikeCassandra FoundationDBRethinkDB Tokutek • • / • FoundationDB CEO David Rosenthal 50 100 RethinkDB CEO Slava Akhmechetat 100 • Oram Aerospike RAM RAM RAM Cassandra LSM- Jonathan | Opinion Ellis LSM-Cassandra Rosenthal FoundationDB LSM Tokutek TokuDB TokuDB MySQL MariaDB 5 1 7 1 MongoDB 10 1 AerospikeFoundationDBRethinkDB Tokutek MVCC Bulkowski Aerospike 2 10 Rosenthal “ ”FoundationDB Bulkowski | Opinion | Opinion Code Review QA BIDS CTO Jim Bird 1 4% , 2 Bug Bug , 2 3 | Opinion 4 5 Bug Bug Bug Bug 25:75 Bug 6 7 8 Bug Bug | Topic WEB WEB • WEB WEB WEB • WEB • / WebIMLofter WEB • IPad Lofter • PC WEB • • WEB | Topic Trident Internet Explorer Webkit AppleGoogleAdobe Apple SafariGoogle Chrome Gecko Mozilla Mozilla Firefox Presto Opera Software Opera Opera 15 Blink Presto Blink Google Opera Software Webkit Chrome 28+Opera 15+ | Topic Trident Internet Explorer Internet Explorer Trident IE6 Trident IE7-9 Android Android Webkit iOS iOS Webkit WinPhone Windows Phone Trident PC CEF Webkit AOPAspect-Oriented Programming AOP | Topic • W3CES • • function doSomething(){ if(isTrident){ // TODO trident implement }else if(isWebkit){ // TODO webkit implement }else if(isGecko){ // TODO gecko implement }else if(isPresto){ // TODO presto implement }else{ // TODO w3c implement } } // doSomething(1,2,3); • • • webkit AOP function doSomething(){ // TODO w3c/es implement } // | Topic doSomething(1,2,3); Trident trident.js // trident implement doSomething = doSomething._$aop( function(_event){ // TODO trident implement }, function(_event){ // TODO trident implement } ); AOP • • • NEJ AOP NEJ NEJ widget.js API | Topic NEJ.define([ 'util/event', '{platform}api.js' ],function(t,h,p){ // TODO }); {platform}api.js ./ widget.js api.js API api.patch.js NEJ.patch NEJ.patch api.patch.js NEJ.patch // NEJ.patch // // NEJ.define([ './hack.js' ],function(h){ // trident NEJ.patch('TR',function(){ // TODO }); // gecko NEJ.patch('GR',[ './hack.firefox.js' ],function(fh){ // TODO }); // IE6 NEJ.patch('TR==2.0',['./hack.ie6.js']); // IE7-IE9 NEJ.patch('3.0<=TR<=5.0',function(){ // TODO }); // hack.js return h; }); | Topic • nxw • patch • | Topic 4.0 2005 2010 NEJ NEJ genify@163.com genify WEB | Topic WEB WEB Web WEB • • • • • WEB module.html | Topic html

aaaaaaaaaaaaaaaaaaa

bbbbbbbbbbbbbbbbbbb

parse scripttextarea module.html textarea • o /// o | Topic • o o • PACPresentationAbstractionControl • HMVCHierarchical model–view–controller WEB • • • • | Topic UMIUniform Module Identifier UMI • URI Path /m/m0/ • “/” • “/?” • /m/m0/ /m/m1/ /m UMI html UMI UMI | Topic “/” UMI UMI | Topic UMI | Topic '/m/blog/list/':{ module:'module/layout/blog.list/index.html', composite:{ box:'/?/blog/box/', tag:'/?/blog/tag/', list:'/?/blog/list/', clazz:'/?/blog/class/' } } • • • • /m/blog/list/ | Topic 1. 2. 3. /m/blog/list/ /m/setting/account/edu/ | Topic 1. 2. | Topic 4. 3. | Topic • UMI • UMI NEJ 4.0 WEB | Topic WEB WEB NEJ NEJ NEJ | Topic UMI • “/”“m”“/” • “/” | Topic • “/” UMI list UMI /m/blog/list • blog list list blog “/” | Topic blog setting m blog setting m | Topic • • —— | Topic /m /?/tab//m /?/tab/ | Topic webroot |- res |- src |- html |- module |- app.html • • • • css | Topic NEJ • cssjs html • @TEMPLATE | Topic NEJ util/dispatcher/module _$$ModuleAbstract /* * ------------------------------------------ * * @version 1.0 * @author genify(caijf@corp.netease.com) * ------------------------------------------ */ NEJ.define([ 'base/klass', 'util/dispatcher/module' ],function(_k,_t,_p){ // variable declaration var _pro; /** | Topic * * @class {_$$Module} * @extends {_$$ModuleAbstract} * @param {Object} */ _p._$$Module = _k._$klass(); _pro = _p._$$Module._$extend(_t._$$ModuleAbstract); /** * * @param {Object} * @return {Void} */ _pro.__doSomething = function(_args){ // TODO }; // TODO return _p; }); • __doBuild • __onShow __onRefresh • __onRefresh • __onHide__onShow __onRefresh __doBuild | Topic /* * ------------------------------------------ * * @version 1.0 * @author genify(caijf@corp.netease.com) * ------------------------------------------ */ NEJ.define([ 'base/klass', 'util/dispatcher/module', '/path/to/project/module.js' ],function(_k,_e,_t,_p){ // variable declaration var _pro; /** * * @class {_$$ModuleDemo} * @extends {_$$Module} * @param {Object} */ _p._$$ModuleDemo = _k._$klass(); _pro = _p._$$ModuleDemo._$extend(_t._$$Module); /** * * - * - * - * @return {Void} */ _pro.__doBuild = function(){ this.__super(); // TODO }; /** * * - * - * - * @param {Object} * @return {Void} */ _pro.__onShow = function(_options){ this.__super(_options); // TODO }; /** * * - * - * - * - * @return {Void} */ _pro.__onRefresh = function(_options){ this.__super(_options); | Topic // TODO }; /** * * - * - * - * @return {Void} */ _pro.__onHide = function(){ this.__super(); // TODO }; // notify dispatcher _e._$regist( 'umi_or_alias', _p._$$ModuleDemo ); return _p; }); __doSendMessage UMI __onMessage _pro.__doSomething = function(){ // TODO this.__doSendMessage( '/m/setting/account/',{ a:'aaaaaa', b:'bbbbbbbbb' } ); }; _pro.__onMessage = function(_event){ // _event.from // _event.data {a:'aaaaaa',b:'bbbbbbbbb'} // TODO }; | Topic _pro.__doSomething = function(){ // TODO this.__doPublishMessage( 'onok',{ a:'aaaaaa', b:'bbbbbbbb' } ); }; _pro.__doBuild = function(){ // TODO this.__doSubscribeMessage( '/m/message/account/','onok', this.__onMessageReceive._$bind(this) ); }; html document.mbody NEJ.define([ 'util/dispatcher/test' ],function(_e){ document.mbody = 'module-id-0'; // test module _e._$testByTemplate('template-box'); }); | Topic rules:{ rewrite:{ '404':'/m/blog/list/', '/m/blog/list/':'/m/blog/', '/m/setting/account/':'/m/setting/' }, title:{ '/m/blog/tag/':'', | Topic '/m/blog/list/':'', '/m/setting/permission/':'', '/m/setting/account/':'', '/m/setting/account/edu/':'' }, alias:{ 'system-tab':'/?/tab/', 'blog-tab':'/?/blog/tab/', 'blog-list-box':'/?/blog/box/', 'blog-list-tag':'/?/blog/tag/', 'blog-list-class':'/?/blog/class/', 'blog-list':'/?/blog/list/', 'setting-tab':'/?/setting/tab/', 'setting-account-tab':'/?/setting/account/tab/', 'layout-system':'/m', 'layout-blog':'/m/blog', 'layout-blog-list':'/m/blog/list/', 'layout-setting':'/m/setting', 'layout-setting-account':'/m/setting/account', 'blog-tag':'/m/blog/tag/', 'setting-edu':'/m/setting/account/edu/', 'setting-profile':'/m/setting/account/', 'setting-permission':'/m/setting/permission/' } } modules:{ '/?/tab/':'module/tab/index.html', '/?/blog/tab/':'module/blog/tab/index.html', '/?/blog/box/':'module/blog/list.box/index.html', '/?/blog/tag/':'module/blog/list.tag/index.html', '/?/blog/class/':'module/blog/list.class/index.html', '/?/blog/list/':'module/blog/list/index.html', '/?/setting/tab/':'module/setting/tab/index.html', '/?/setting/account/tab/':'module/setting/account.tab/index.html', '/m':{ module:'module/layout/system/index.html', composite:{ tab:'/?/tab/' } }, '/m/blog':{ module:'module/layout/blog/index.html', composite:{ tab:'/?/blog/tab/' } }, '/m/blog/list/':{ module:'module/layout/blog.list/index.html', composite:{ box:'/?/blog/box/', tag:'/?/blog/tag/', list:'/?/blog/list/', | Topic clazz:'/?/blog/class/' } }, '/m/blog/tag/':'module/blog/tag/index.html', '/m/setting':{ module:'module/layout/setting/index.html', composite:{ tab:'/?/setting/tab/' } }, '/m/setting/account':{ module:'module/layout/setting.account/index.html', composite:{ tab:'/?/setting/account/tab/' } }, '/m/setting/account/':'module/setting/profile/index.html', '/m/setting/account/edu/':'module/setting/edu/index.html', '/m/setting/permission/':'module/setting/permission/index.html' } __export __export parent “/m”__doParseParent _pro.__doBuild = function(){ this.__body = _e._$html2node( _e._$getTextTemplate('module-id-l2') ); // 0 - box select // 1 - class list box // 2 - tag list box // 3 - sub module box var _list = _e._$getByClassName(this.__body,'j-flag'); this.__export = { box:_list[0], clazz:_list[1], tag:_list[2], list:_list[3], parent:_list[3] }; }; composite '/m/blog/list/':{ module:'module/layout/blog.list/index.html', composite:{ box:'/?/blog/box/', tag:'/?/blog/tag/', list:'/?/blog/list/', clazz:'/?/blog/class/' } } | Topic • onshow refresh refresh • onrefresh refresh refresh • onshow onrefresh onrefresh composite:{ onshow:{ // onshow // onrefresh }, onrefresh{ // onshow // onrefresh } // onrefresh } NEJ.define([ 'util/dispatcher/dispatcher' ],function(_e){ _e._$startup({ // rules:{ rewrite:{ // }, title:{ // }, alias:{ // // } }, // modules:{ // UMI // } }); }); | Topic NEJ /m/setting/tag/ rules:{ // ... alias:{ // ... 'blog-tag':['/m/blog/tag/','/m/setting/tag/'] } } modules:{ | Topic // ... '/m/setting/tag/':'module/blog/tag/index.html' } /?/setting/tab UMI WEB SPA NEJ PC WebIM IPad Lofter Android 4.0 WEB | Article Amr Noaman Abdel-Hamid work in progress cycle time Throughput 1 Little’s Law1 John Little WIP = Th * CT Work in Process= Th= CT= 2Diagrams of Effects 1 Adam 2 1 2013 Adam CEO | Article Th = WIP / CT 1 2 | Article 3 4 2 = | Article 2 Ismail SLAService Level Agreement Ismail SLA 5 Ismail Ismail | Article 6 / 3 7 | Article 8 9 | Article 1. 2. 3. • John Little Arrival Rate Arrival Rate L = λ W L = λ = W = John Little “ ” 88 (WIP = Th*CT) | Article 0 0 Little “” 93 Input Rate= Output Rateλth 0 Little “ ” 93 • Diagrams of Effects: four-volume series Gerald Weinberg CLDCausal Loop Diagrams Amr Noaman Abdel-Hamid Amr Agile Academy Amr Lean&Agile Network GoAgile Amr 400 Amr emailLinked-in Twitter @amrnoaman Amr http://www.infoq.com/articles/how-kanban-works | Article Spark RDD Spark RDD MapReduceStreamingSQLMachine Learning Graph Matei Zaharia “Unified Programming Abstraction Spark Spark RDD RDD RDD Resilient Distributed Datasets RDD mapflatMapfilter monad Scala RDD join groupByreduceByKey reduceByKey action transformation Iterative AlgorithmsRelational QueriesMapReduceStream Processing Hadoop MapReduce MapReduces Storm Stream Processing RDD Spark | Article RDD RDD dataset RDD RDD Child RDD narrow dependency Child RDD wide dependency map narrow dependency join wide dependency Spark narrow wide narrow dependencies cluster node map filterwide dependencies MapReduce narrow dependencies parent partition wide dependencies RDD Parent Partitions narrow dependencies wide dependencies Matei Zaharia An Architecture for Fast and General Data Processing on Large Clusters box RDD partition | Article RDD RDD persistence patitioning persist patitionBy RDD RDD (RDD parallerize ) Spark input.map(parseArticle _).partitionBy(partitioner).cache() partitionBy Partitioner val partitioner = new HashPartitioner(sc.defaultParallelism) RDD RDD Int Float Int RDD Int RDD transformation action transformation RDD action RDD RDD transformation RDD map MappedRDD flatMap FlatMappedRDD map flatMap RDD RDD def map[U: ClassTag](f: T => U): RDD[U] = new MappedRDD(this, sc.clean(f)) RDD compute action private[spark] class MappedRDD[U: ClassTag, T: ClassTag](prev: RDD[T], f: T => U) extends RDD[U](prev) { override def getPartitions: Array[Partition] = firstParent[T].partitions override def compute(split: Partition, context: TaskContext) = firstParent[T].iterator(split, context).map(f) } | Article RDD RDD (immutable) Graph of Operation Worker replication Spark Spark Streaming update Streaming window Spark checkpoint checkpoint RDD wide dependency checkpoint Spark auto checkpointing RDD Spark Spark • • • key • • ThoughtWorks Spark RDD | Article RAD Game Tools Fabian “ryg” Giesen Cache coherency primer InfoQ Cache CPU CPU I/O Write-combined CPU /—— CPU CPU L1 Cache —— “”line 32 ARM90 /2000 x86 PowerPC64 ARM x86 128 Power ISA “” CPU L1D$“”cache“”cash | Article localized write- throughwrite-back “” “” “” CPU CPU 32 128 | Article cache associativitycache setswrite-allocate unaligned access Coherency protocols CPU CPU CPU CPU CPU CPU CPU CPU CPU “ snooping”“directory-based” “” arbitrate | Article “” —— MESI MESI ModifiedExclusiveSharedInvalid MESI “MESI ” MESI MESI MESI • Invalid • Shared • Exclusive S E “” “” • Modified E —— IS M / E “” E M “” — — | Article “” MESI “Read For Ownership” MESI M E MESI MESI CPU ARM MESI “O”Owned E “” MOSEI MERSI MESIF S R F R/F O MESI | Article “”—— MESI sequential consistency C++ 11 “” ARM POWER CPU reordering “ memory barrier”“” x86 “”—— MESI • CPU “”invalidation queue • Out-of-Order executionin-order execution CPU • “”store buffer “” “”drained | Article —— x86 “”retired MOBmemory ordering bufferx86 “”—— page faultbranch mispredict “” x86 “”x86 “memory ordering machine clear”x86 —— http://fgiesen.wordpress.com/2014/07/07/cache-coherency/ Cache Coherency Column TDW ——Shuffle Tencent distributed Data Warehouse, TDW Hadoop Hive 5600 100 TDW TDW MapReduce Spark ——Shuffle Shuffle Shuffle TDW Shuffle——MapReduce Shuffle Shuffle MapReduce shuffle MapReduce shuffle MapReduce map reduce Reduce mapmap reduce reduce shuffle map reduce shuffleShuffle map reduce map spill reduce copy sort Column Spill Spill Collect Map kvbuffer kvmeta kvbuffer IntBuffer kvbuffer spill 0 Column Kvbuffer bufindex bufindex 0 Int key bufindex 4 Int value bufindex 8 kvbuffer value key partition value Int kvmeta kvindex “” kvindex -4(kvindex+0) value (kvindex+1) key (kvindex+2) partition (kvindex+3) value kvindex -8 kvindex -32 Kvbuffer kvbuffer kvbuffer spill spill spill kvbuffer spill kvbuffer spill map spill kvbuffer 80% spill spill map spill map Spill spill spill map “” sortAndSpill spill spill sort Sort kvbuffer partition key kvmeta partition partition key Column Spill Spill spill “spill12.out”Spill kvmeta partition partition partition partition partition (segment) partition partition partition partition “spill12.out.index” crc32 (spill12.out.index 1M spill12.out ) spill out index spill spill sortAndSpill map Map kvbuffer bufindex kvmeta kvindex bufindex kvindex Map kvbuffer bufindex kvindex -16 Column spill Map Merge Map spillout index merge Merge spill spill Merge spill index spill spill kvbuffer I/O merge file.out file.out.index partition partition partition partition partition Column spill partition partition segment segment partition segment segment key segment segment segment index Map shuffle Copy Reduce http map http server reduce map mapOutput http http server map reduce reduce Reduce map Reduce map map map merge merge map map http 64K Column map merge map map reduce Merge Sort merge map merge Map merge reduce sort reduce copy sort copy sort Reduce shuffle Spark Shuffle Shuffle Writer Spark shuffle shuffle wide dependency group by key Spark shuffle map reduce bucketmap partitioner bucketId bucket map reduce map R bucketR reduce M map M*R bucket Map bucket map bucket blockFile DiskBlockManager hash map R map 100KB FastBufferedOutputStream shuffle Column shuffle Spark shuffle consolidation shuffle shuffle consolidation shuffle bucket segment Job map reduce bucket ShuffleFileGroup map ShuffleFileGroup map bucket ShuffleFileGroup segment map ShuffleFileGroup map ShuffleFileGroup bucket ShuffleFileGroup job 3 map 2 reduce(1) 3 core 3 map 3 map 2 shuffle 6 shuffle (2) 2 core 2 map 2 map 2 shuffle map map map shuffle map shuffle 4 shuffle (3) 2 2 core 1 core 2 map 1 map 2 map map shuffle map shuffle map ShuffleFileGroup 6 shuffle Shuffle Fetcher Reduce map Spark socket netty Executor BlockManager BlockManagerWorker reduce GET_BLOCK blockId reduce Netty BlockManager ShuffleSender shuffle map reduce Reduce Spark map spark shuffle spark shuffle reduce shuffle Reduce HashMap HashMap key Column map keymap key value HashMap valueSpark shuffle HashMap HashMap Shuffle map shuffle group by key reduce key value value Spark Shuffle 1000 70% MapReduce merge MapReduce Spark Shuffle MapReduce Spark collect map map sort map map merge spill map merge reduce copy jetty netty socket map Column copy merge sort Shuffle shuffle io shuffle shuffle “” shuffle io TDW MapReduce Shuffle • • reduce nb • shuffle • netty • map reduce Spark Shuffle Spark MapReduce shuffle Spark shuffle • • Spark map TDW ——Shuffle Column SNG “” SNG QQ QQ SNG 2013 QCon T4 QCon SNS SNG 2009 Leader SNG QCon 2014 SNG SNG InfoQ 2012 CTO Tony BG SNG SNG SNG QzoneQQ QQ QQ QQ InfoQ SNG “ ” SNG “” inotify Column InfoQ“” XEN LXC LXC CPU crontab LXC NAT LXC QQ hardcode IP “”“”“” SNG D/O DevOps —— SNG pkg svn CC apiL5 Column InfoQ“” IDC/ SET/ SET SET SET Qzone SET SNG + L5L5 L5ID L5api get_route L5ID IPport update_route L5ID IP:port L5 L5ID IP L5 IP:port IPport L5 L5 /CPU/ buff L5 InfoQ C/S C inotify leader S 1000 md5 md5md5 Column SNG Qzone 3 InfoQ —— CMDB / wiki SNG SNG svn diff / IP Column svn InfoQ“” SNG storm mongoDBrabbitMQ Infobright SNG InfoQ SNG Qzone SET SET SSD buff SDK MSNG Column SNG “” | Void InfoQ “” InfoQ • • • Google IntelMarvell kk | Void debug PCB ID SMT ID 10% 500 2000 500 VC 20 ———— 200 40 8000 5 iPhone4 quacomm iPhone5s iPhone iPhone5s | Void —— “ ” VC 10 Wiko2013 18% 250 Wiko 90% 100-300 200 Wiko 90%Wiko “” cost down 1 5k | Void PPT PPT Healthkit Healthkit VC “ ” 2003-2005 03 ZigBee——ZigBee IEEE ZigBee ZigBee ZigBee ZigBee—— IEEE W3C SOAP JSON JSON | Void “ ” ODM care TV 10 Google Android Wear care Android “” MTK Linkit IDH 20 4G Linkit | Void Jorjin QCon Glass Google Glass ODM 9 SDK Google Google Glass 1500 Google Jorjin SDK MCU Wifi Linux “” 20 5 15 Google Web Web 10 | Void David LI 1990 Free Software Foundation) Objectweb Apache 20 David 2010 2014 10 16-18 QCon | Void NoSQL Bigstep’s Full Metal Cloud Alex Bordei NoSQL Alex 1. Alex 2. Alex NoSQL 3. CPU CPU | Void 4. CPU 5. NoSQL NoSQL Alex 400% NoSQL /editors@cn.infoq.com InfoQ http://weibo.com/infoqchina sales@cn.infoq.com InfoQ Flash 51CTO Linux 丁
还剩102页未读

继续阅读

下载pdf到电脑,查找使用更方便

pdf的实际排版效果,会与网站的显示效果略有不同!!

需要 2 金币 [ 分享pdf获得金币 ] 1 人已下载

下载pdf

pdf贡献者

patrick002

贡献于2015-03-15

下载需要 2 金币 [金币充值 ]
亲,您也可以通过 分享原创pdf 来获得金币奖励!
下载pdf