• 1. 基于scala/akka构建响应式流计算@途牛-谢辉
  • 2. 议题Akka-actor 响应式流计算 与Akka-Stream对比 性能优化
  • 3. AKKA
  • 4. Actor(Questioner)MailboxDispatcherMessageMessage不变性 异步、Nonblocking Actor是线程安全,并发通过多个Actor实现 两两actor间message有序Actor(Answerer)AKKA
  • 5. Actor ModelEvent-DrivenActorBehaviorStateMailbox:消息队列 Behavior:响应消息的行为 State:FSM Run: 附着在线程上执行
  • 6. RootACFDEB/A/A/C/A/C/F树形结构: Parent-Children Supervisor Strategy One-For-One All-For-One Path Dispatcher ThreadPoolActor SystemActor System
  • 7. 消息驱动NonBlocking(异步) 无锁(消息不变性) Actor线程安全 状态拓扑 Message通信开销
  • 8. 响应式流计算
  • 9. 背景场景:列表页 资源价格、余位信息变更频繁 内存索引? 不同供应商能力差异 未命中、实时抓取
  • 10. 计算引擎Data-1Data-2Data-3cacheInput流计算数据源:Lucene、cache、RPC… Input: 流化管道 计算:一次Stream的聚合计算
  • 11. DataFlow WorkerExecutorWorkerPoolActiveWorkerStream ProcessWork InputExecutorProcessEngine………1:nFrontendMasterBackend
  • 12. ProcessADCBProcessEngineFJ
  • 13. WorkActivity行为 包含一个Worker担当行为人 多个Executor实现并发处理 In-Out IN: 1)前一活动 的Result 2)来自WorkInput 的Job流 ActivityWorkerExecutorExecutor…
  • 14. WorkActivity XORJob StreamSession:job流 JobSource(SessionId, Timeout, …) Job … Deadline(SessionId) JobSource XOR Deadline = 0jobSourcdeadlinejobjob…jobSourcdeadlinejobjob…jobSourcdeadlinejobjob…
  • 15. Worker FSMPrepareWorkerReadyDoingDoneRelease
  • 16. Exception PropagationRootProcessEngineWorkerExecutorProcessWorker-Executor: One-For-One Strategy Process-ProcessEngine-Worker: Exception作为Message向外传递
  • 17. Time OutProcess Time Out ProcessDefinition.timeout(5 seconds) Supervisor: StreamProcess Session Time Out JobSource(sessionId, 2 seconds, timeoutMsg) Supervisor:Worker Executor处理timeoutMsg
  • 18. 示例
  • 19. 一个计算流程两组随机数分别产生1000个1~50的整数,对这两组数进行碰撞,得到碰撞数(该数在两组中都存在,则产生碰撞,碰撞后需从各自集合中移除)的集合,后对集合进行排序,并且计算出碰撞数相同次数分别有是多少。CollisionSecondaryForkJoinSortCollectResultEndJob…
  • 20. 流程DSL
  • 21. Collision Executor
  • 22. Akka Stream
  • 23. Akka-StreamFrom http://akka.io
  • 24. 似dataflowAkka-stream计算节点Activity OnReady WorkActivity EndActivity ForkActivity/JoinActivityGraph:Module/Shape SourceModule/SourceShape FlowModule/FlowShape SinkModule/SinkShape JunctionModule/FanInShape、FanOutShape操作ExecutorStage:map、filter…流程ProcessDefinitionRunnableFlow运行Cluster 、 standaloneFlowMaterializer:ActorFlowMaterializer orElse
  • 25. 非dataflowAkka-stream运算模型有界流无界流运算单元有状态无状态数据源WorkInput 动态数据源Source消息传递Push messageBack-pressure Diymatic Push/Pull启动用时< 20ms> 100ms
  • 26. 性能调优
  • 27. DataFlow通信 Local > Remote 点到点(WorkInput) Actor Don’t Ask(NonBlocking) 循环利用 行为单调
  • 28. 应用优化Executor Behavior:简单 NonBlocking:异步库(akka-http、scalaredis…) 并发:足量保证消费能力,routing规则均衡 Streaming job-Stream:流化均匀 Thread parallelism(fork-join-executor) :min、max、factor 数据分类 计算型数据、渲染型数据
  • 29. Q & A