Jafka guide
•Download as PPTX, PDF•
8 likes•4,606 views
Jafka is a fast and lightweight message queue system that is implemented as a single 271KB JAR file. It uses Zookeeper for coordination and has dependencies on common Java libraries like Log4j and Jackson. Jafka aims to eventually become a full implementation of Apache Kafka with features like persistence, high throughput processing of millions of messages per second, load balancing and a simple message format. It currently focuses on providing basic queue functionality through a simple producer/consumer model.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Similar to Jafka guide
Similar to Jafka guide (20)
Recently uploaded
Recently uploaded (20)
Jafka guide
- 2. overview ● https://github.com/adyliu/jafka ● 271KB single jar ● 3.5MB with all dependencies and configurations ● So simple, so fast jafka mq 2
- 3. Maven Usage <dependency> <groupId>com.sohu.jafka</groupId> <artifactId>jafka</artifactId> <version>1.0-rc7</version> </dependency> jafka mq 3
- 4. Dependencies ● Zkclient – Zookeeper ● Log4j ● Jackson ● Optional dependencies: – Mx4j-tools – jopt-simple jafka mq 4
- 5. Future ● A full kafka clone (http://incubator.apache.org/kafka) ● Persisten messaging with O(1) ● Constant time performance even with TB messages ● High-throughput (100k/s) ● Distributed messaging system – producers/brokers/consumers ● Auto load balancing ● Simple message format jafka mq 5
- 6. Message format (1) jafka mq 6
- 7. Message format (2) jafka mq 7
- 8. Log Implementation jafka mq 8
- 9. Architech and workflow jafka mq 9
- 10. Zookeeper nodes jafka mq 10
- 11. Offset length limit ● Long.MAX_VALUE – 9223372036854775807 → 9*1018 ● If one message size is 1k – Messages: 9*1015 – If producing 1 billion messages per day ● Continuing days: 9*106 ● Continuing years: 24657 jafka mq 11
- 12. About Filesystem ● 6 7200rpm SATA RAID-5 * – 300MB/s linear writes (300k/s 1KB per message) – 50k/s random writes jafka mq 12 http://incubator.apache.org/kafka/design.html
- 13. sendfile() (1) ● Common data transfer 1. OS reads data from disk to pagecache 2. Application reads data from pagecache to buffer 3. Application writes data back into socket buffer 4. OS copies data from socket buffer to NIC buffer ● 4 copies + 2 system_call jafka mq 13
- 14. sendfile() (2) ● FileChannel.transferTo(long, long, WritableByteCh annel) – Zero copy * https://www.ibm.com/developerworks/linux/library/j-zerocopy/ jafka mq 14
- 15. Batch compression ● GZIP message compression default jafka mq 15
- 16. Sync or Async ● Both supported ● Setting at connecting ● Batch sending while using asynchronous sending jafka mq 16
- 17. Usage ● Simple & Funny jafka mq 17
- 18. Simple usage - dependencies <dependency> <groupId>com.sohu.suc*</groupId> <artifactId>suc-mq</artifactId> </dependency> Version >= 2.0-SNAPSHOT ● * our inner project jafka mq 18
- 19. Simple usage - producer Producers producers = Producers.buildProducer(); StringMessage data = new StringMessage("demo"); data.add("first message"); data.add("second message"); data.add("third message"); producers.send(data); jafka mq 19
- 20. Simple usage - consumer public static void main(String[] args) { Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { @Override public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); } }); } jafka mq 20
- 21. Consumer => Queue Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { public void onMessage(String message) { Client A System.out.println(message + " ==> " + new Date()); } }); Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { public void onMessage(String message) { Client A System.out.println(message + " ==> " + new Date()); } }); jafka mq 21
- 22. Consumer => Topic Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); Client A } }); Consumers.buildConsumer("demo", "zhendong", new IMessageListener<String>() { public void onMessage(String message) { Client B System.out.println("[震子] ==> " + message); } }); jafka mq 22
- 23. Full Producer code Properties props = new Properties(); props.put("zk.connect", "localhost:2181"); props.put("serializer.class", StringEncoder.class.getName()); // ProducerConfig config = new ProducerConfig(props); Producer<String, String> producer = new Producer<String, String>(config); // StringProducerData data = new StringProducerData("demo"); for(int i=0;i<100;i++) { data.add("Hello world #"+i); } try { long start = System.currentTimeMillis(); for (int i = 0; i < 100; i++) { producer.send(data); } long cost = System.currentTimeMillis() - start; System.out.println("send 10000 message cost: "+cost+" ms"); } finally { producer.close(); } jafka mq 23
- 24. Full Consumer Code Properties props = new Properties(); props.put("zk.connect", "localhost:2181"); props.put("groupid", "test_group"); // ConsumerConfig consumerConfig = new ConsumerConfig(props); ConsumerConnector connector = Consumer.create(consumerConfig); // Map<String, List<MessageStream<String>>> topicMessageStreams = connector.createMessageStreams(ImmutableMap.of("demo", 2), new StringDecoder()); List<MessageStream<String>> streams = topicMessageStreams.get("demo"); // ExecutorService executor = Executors.newFixedThreadPool(2); final AtomicInteger count = new AtomicInteger(); for (final MessageStream<String> stream : streams) { executor.submit(new Runnable() { public void run() { for (String message : stream) { System.out.println(count.incrementAndGet() + " => " + message); } } }); } jafka mq 24
- 25. More Documents ● https://github.com/adyliu/jafka/wiki jafka mq 25
- 26. Open source First Java open source in maven repository at groupId com.sohu. jafka mq 26
- 27. About me ● Ady Liu ● Email: imxylz@gmail.com ● Architech@SOHU.COM jafka mq 27