SlideShare a Scribd company logo
1 of 37
Download to read offline
深入了解Redis
               宋传胜
   Chuansheng.song@langtaojin.com
简单介绍
•   文本协议 memcached类似
•   KEY 可打印字符
•   VALUE支持的类型
    –   STRINGS
    –   LIST
    –   SET
    –   SORTED SET
    –   HASH
•   高性能 (100k+ SET/80k+ GET)/s
•   序列化支持
•   主从同步支持
•   客户端自己实现sharding
基本数据结构
• RedisObject (Redis.h)
  – key/value对象的基础
KEY
• 基本命令
 –   get/set/exists O(1)
 –   setnx/randomkey/rename/renamenx/dbsize/type
 –   keys pattern
 –   del 一次删除多个key
 –   expire key seconds 指定过期时间
 –   ttl key 剩余过期时间
 –   select db
 –   move key newdb
 –   flush db/flushall
EXPIRE AND LRU CACHE
• Volatile Key (Db.c)
   – When the key is set to a new value using the SET command, or when a
     key is destroyed via DEL, the timeout is removed from the key.
• Enhanced Lazy Expiration algorithm
   – Redis does not constantly monitor keys that are going to be expired.
     Keys are expired simply when some client tries to access a key, and the
     key is found to be timed out.
• Work as LRU cache (memcached)
   – Maxmemory/maxmemory-policy
   – When memory limit was already reached, server will remove some old
     data deleting a volatile key, even if the key is still far from expiring
     automatically.
   – Random get keys, delete by lru rules.
基本数据结构
• Hashtable(Dict.c)
基本Value数据结构
• STRING(sds.c)



  – set/setnx/get/getset/mget/mset/msetnx
    •nx – not exists
    •m -- multiple
  – incr/decr/incrby/decrby
  – getrange/append
基本Value数据结构
• LIST (T_list.c)
  – REDIS_LIST 类型, 如果其 entry 小于配置值: list-
    max-ziplist-entries 或 value字符串的长度小于 list-
    max-ziplist-value,使用ziplist数据结构,否则使用
    标准的Doubly linked list
  – (l/r)push/(l/r)pop/llen O(1)
  – b(l/r)pop支持阻塞等待,避免了轮循
  – lrange/ltrim/lrem/lset/rpoplpush
LIST
• Ziplist (Ziplist.c)
   – O(mem_size) add/delete
   – list-max-ziplist-entries (default: 1024)
   – list-max-ziplist-value (default: 32)
LIST
• Ziplist (continue)
  – zlentry (unencode structure)
LIST
• Doubly linked list (Adlist.c)
基本Value数据结构
• SET (T_set.c) – hashtable + Intset
  – String的无序集合
  – sadd/srem/sismember/scard O(1)
  – spop/srandmember
  – smove
  – smembers
  – sinter(store) O(C)
  – sunion(store)/sdiff(store) O(N)
INTSET
• INTSET (Intset.c)
  – 都是整型时数据结构退化成排序的intset
  – Good fit for size up to 20-50K
  – set-max-intset-entries (default: 4096)
  – O(n) search
  – O(log(n) + mem_size) add/del
基本Value数据结构
• SORTED SET(T_zset.c) hashtable + skip list
  – 按照key的score排序
  – zadd/zrem/zrank/zrevrank O(log(n))
  – zcard O(1)
  – zincrby 修改score
  – zrange/zrevrange/zrangebyscore/zscore
    O(log(N)+M)
  – zremrangebyrank/zremrangebyscore O(log(N)+M)
SKIP LIST
• Skip List (T_zset.c)
  – skip list  redblack tree  AVL tree (more
    balanced)
  – lockfree/concurrency
基本Value数据结构
• HASH (T_hash.c) zipmap + hashtable
  – 如果其 entry 小于配置值: hash-max-zipmap-
    entries 或 value字符串的长度小于 hash-max-
    zipmap-value ,使用zipmap数据结构
  – Value只能是string类型
  – hset/hget/hexists O(1)
  – hmset/hexists/hincrby/hdel/hlen
  – hmget O(N)
  – hkeys/hvals/hgetall
ZIPMAP
• zipmap
  – O(n) search O(mem_size) add/delete
内存管理
• zmalloc.c
  – 简单内存管理
  – 支持tcmalloc USE_TCMALLOC
  – zmalloc(size_t size)  malloc(size+PREFIX_SIZE)
  – fragmentation_ratio =
    zmalloc_get_rss/zmalloc_used_memory
PUBSUB
• SUBSCRIBE/UNSUBSCRIBE/PSUBSCRIBE/PU
  NSUBSCRIBE
• PUBLISH
• 连接断开后信息丢失
TRANSACTION
• MULTI (不支持CAS)
  –   MULTI
  –   SET foo 0
  –   INCR foo
  –   EXEC
• WATCH (Check and Set )
  –   WATCH theKey
  –   v = GET theKey
  –   MULTI
  –   SET theKey v+1
  –   EXEC
• UNWATCH/DISCARD
提升网络效率、减少内存占用
• MGET/MSET
• PIPELINING
  – $ (echo -en "PINGrnPINGrnPINGrn"; sleep 1) | nc localhost 6379
    +PONG
    +PONG
    +PONG
• 压缩数据,触发ziplist/zipmap/intset
• 使用 GETBIT/SETBIT/GETRANGE/SETRANGE 压缩多
  个属性
• 使用HASH
• 使用32位redis + sharding
Client Side Sharding
• Consistent Hashing
  – Redis Sharding at Craigslist
网络层
• Anet.c
• 事件驱动 ae.c 
  ae_epoll.c/ae_kqueue.c/ae_select.c
• 非阻塞单线程
• 支持timer
• 在VM场景下有可能多线程
网络层对比
• memcached/varnish/scribed的网络IO模型
 – 多线程,非阻塞IO复用
网络层对比
• lighttpd/nginx的网络IO模型
  – 多进程,单线程非阻塞IO
• apache的MPM模型
  – 多进程prefork,多线程
PERSISTENCE
• SNAPSHOT (Rdb.c)
• 可以关闭
• BGSAVE or save xseconds ychanges
• fork
   – parent - disable rehash, periodic check child on serverCron
     (100ms)
   – child - copy-on-write, save whole db, exit
• Need to turn on the overcommit_memory setting if you want to
  deal with a dataset more than 1/2 of RAM, which contradicts
  our habit to battle against OOM killer as a database
  administrator.
Append Only File
• Append to log file on every change (Aof.c)
• fsync() policy
   – always/os decide/every second
   – Always with battery backed raid controllers
   – Every second by default (innodb_flush_log_at_trx_commit=2 in
     mysql)
• Compact aof file
   – BGREWRITEAOF/REWRITEAOF
• Fork
   – child – write new aof in temporary file
   – parent – write change in both old aof file and memory buffer,
      append memory buffer to temporary file when child done, rename
      temporary to aof file.
• redis-check-aof --fix <filename>
Replication
• 支持多级同步 (Replication.c)
Replication
• 支持AUTH
• 起动过程
 – slave
   •SYNC
   •Wait
 – master
   •Issue a BGSAVE, or wait if BGSAVE in progress
Virtual Memory
• Redis VM is now deprecated. Redis 2.4 will be the
  latest Redis version featuring Virtual Memory!
• Why not OS VM
  – Varnish What’s wrong with 1975 programming ?
  – Redis What’s wrong with 2006 programming?
     • Blocking when page fault
     • 4k pages granularity
     • Optimal disk persistent format, 10:1
• Keys can’t be swapped out
• vm is read only when BGSAVE/BGREWRITEAOF in
  progress
• vm-max-memory/vm-pages/vm-page-size
Virtual Memory
• Blocking vm
• Threaded vm
  – Before command is executed, check the value
  – Block client and load, then signal by unix pipe
  – Continue
• Recommended filesystem – ext3 or any other file
  system with good support for sparse files
• redis-stat vmstat
• InnoDB innodb_buffer_pool_size – swap and
  durability
Future
• Diskstore (diskstore.c dscache.c)
  – All data on disk, B+tree in future
  – Swaps data into memory when needed
• Leveldb/InnoDB Change Buffer
  – LSM tree
其他相关的代码
• quicksort (pqsort.c)
• lzf compression (lzf_c.c lzf_d.c)
• Sha1 (Sha1.c)
• Syncio.c
• SORT (Sort.c)
线上Redis集群情况
• 16台服务器,兼做后端服务器
• 45个Redis Instance进程,三组服务
• 总内存 77G VIRT,71G RSS
• 使用了主从复制、snapshot持久化,VM支持
  ,客户端sharding、pipelining、hash优化
Redis应用
•   Resque – message queue of github (celery)
•   Nginx HttpRedis module
•   Redweb - Web administrative and query UI
•   A fast, fuzzy, full-text index using Redis
•   Redis-backed BloomFilter(s) in Ruby
•   Soulmate : Auto Complete with Redis
•   Rate limiting with Redis
•   Redis Sharding
•   Locking with setnx
•   A Collection Of Redis Use Cases
•   Radishapp Redis monitoring
•   openredis.com hosting service
Contribute to Redis
• Fork redis on github
  – https://github.com/antirez/redis
• Choose branch and commit your changes
• Create a pull request
• Wait for approval
  – https://github.com/antirez/redis/pulls
• My contribute for read only slave
  – https://github.com/antirez/redis/pull/47
Resources
• DOCUMENATION
 – http://redis.io/documentation
 – http://antirez.com/
 – http://groups.google.com/group/redis-db
• 本文参考的Redis资料集合
 – http://www.delicious.com/fakechris/redis
 – http://www.delicious.com/tag/redis

More Related Content

What's hot

A simple introduction to redis
A simple introduction to redisA simple introduction to redis
A simple introduction to redisZhichao Liang
 
Your 1st Ceph cluster
Your 1st Ceph clusterYour 1st Ceph cluster
Your 1st Ceph clusterMirantis
 
High Performance Weibo QCon Beijing 2011
High Performance Weibo QCon Beijing 2011High Performance Weibo QCon Beijing 2011
High Performance Weibo QCon Beijing 2011Tim Y
 
Redis overview for Software Architecture Forum
Redis overview for Software Architecture ForumRedis overview for Software Architecture Forum
Redis overview for Software Architecture ForumChristopher Spring
 
Ceph Performance and Sizing Guide
Ceph Performance and Sizing GuideCeph Performance and Sizing Guide
Ceph Performance and Sizing GuideJose De La Rosa
 
Development to Production with Sharded MongoDB Clusters
Development to Production with Sharded MongoDB ClustersDevelopment to Production with Sharded MongoDB Clusters
Development to Production with Sharded MongoDB ClustersSeveralnines
 
Caching solutions with Redis
Caching solutions   with RedisCaching solutions   with Redis
Caching solutions with RedisGeorge Platon
 
微博cache设计谈
微博cache设计谈微博cache设计谈
微博cache设计谈Tim Y
 
Red Hat Storage Server Administration Deep Dive
Red Hat Storage Server Administration Deep DiveRed Hat Storage Server Administration Deep Dive
Red Hat Storage Server Administration Deep DiveRed_Hat_Storage
 
Redis trouble shooting_eng
Redis trouble shooting_engRedis trouble shooting_eng
Redis trouble shooting_engDaeMyung Kang
 
Алексей Лесовский "Тюнинг Linux для баз данных. "
Алексей Лесовский "Тюнинг Linux для баз данных. "Алексей Лесовский "Тюнинг Linux для баз данных. "
Алексей Лесовский "Тюнинг Linux для баз данных. "Tanya Denisyuk
 

What's hot (20)

A simple introduction to redis
A simple introduction to redisA simple introduction to redis
A simple introduction to redis
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
redis basics
redis basicsredis basics
redis basics
 
Redis acc 2015_eng
Redis acc 2015_engRedis acc 2015_eng
Redis acc 2015_eng
 
Redis 101
Redis 101Redis 101
Redis 101
 
Redis ndc2013
Redis ndc2013Redis ndc2013
Redis ndc2013
 
Your 1st Ceph cluster
Your 1st Ceph clusterYour 1st Ceph cluster
Your 1st Ceph cluster
 
High Performance Weibo QCon Beijing 2011
High Performance Weibo QCon Beijing 2011High Performance Weibo QCon Beijing 2011
High Performance Weibo QCon Beijing 2011
 
Redis overview for Software Architecture Forum
Redis overview for Software Architecture ForumRedis overview for Software Architecture Forum
Redis overview for Software Architecture Forum
 
MyRocks Deep Dive
MyRocks Deep DiveMyRocks Deep Dive
MyRocks Deep Dive
 
Redis database
Redis databaseRedis database
Redis database
 
Ceph Performance and Sizing Guide
Ceph Performance and Sizing GuideCeph Performance and Sizing Guide
Ceph Performance and Sizing Guide
 
Development to Production with Sharded MongoDB Clusters
Development to Production with Sharded MongoDB ClustersDevelopment to Production with Sharded MongoDB Clusters
Development to Production with Sharded MongoDB Clusters
 
Caching solutions with Redis
Caching solutions   with RedisCaching solutions   with Redis
Caching solutions with Redis
 
微博cache设计谈
微博cache设计谈微博cache设计谈
微博cache设计谈
 
Redis introduction
Redis introductionRedis introduction
Redis introduction
 
Red Hat Storage Server Administration Deep Dive
Red Hat Storage Server Administration Deep DiveRed Hat Storage Server Administration Deep Dive
Red Hat Storage Server Administration Deep Dive
 
Bluestore
BluestoreBluestore
Bluestore
 
Redis trouble shooting_eng
Redis trouble shooting_engRedis trouble shooting_eng
Redis trouble shooting_eng
 
Алексей Лесовский "Тюнинг Linux для баз данных. "
Алексей Лесовский "Тюнинг Linux для баз данных. "Алексей Лесовский "Тюнинг Linux для баз данных. "
Алексей Лесовский "Тюнинг Linux для баз данных. "
 

Viewers also liked

新浪微博开放平台Redis实战
新浪微博开放平台Redis实战新浪微博开放平台Redis实战
新浪微博开放平台Redis实战mysqlops
 
高性能No sql数据库redis
高性能No sql数据库redis高性能No sql数据库redis
高性能No sql数据库redispaitoubing
 
redis 适用场景与实现
redis 适用场景与实现redis 适用场景与实现
redis 适用场景与实现iammutex
 
美团点评沙龙 飞行中换引擎--美团配送业务系统的架构演进之路
美团点评沙龙 飞行中换引擎--美团配送业务系统的架构演进之路美团点评沙龙 飞行中换引擎--美团配送业务系统的架构演进之路
美团点评沙龙 飞行中换引擎--美团配送业务系统的架构演进之路美团点评技术团队
 
Redis 常见使用模式分析
Redis 常见使用模式分析Redis 常见使用模式分析
Redis 常见使用模式分析vincent253
 
美团点评技术沙龙010-Redis Cluster运维实践
美团点评技术沙龙010-Redis Cluster运维实践美团点评技术沙龙010-Redis Cluster运维实践
美团点评技术沙龙010-Redis Cluster运维实践美团点评技术团队
 
Redis cluster
Redis clusterRedis cluster
Redis clusteriammutex
 
Everything you always wanted to know about Redis but were afraid to ask
Everything you always wanted to know about Redis but were afraid to askEverything you always wanted to know about Redis but were afraid to ask
Everything you always wanted to know about Redis but were afraid to askCarlos Abalde
 
Redis and Groovy and Grails - gr8conf 2011
Redis and Groovy and Grails - gr8conf 2011Redis and Groovy and Grails - gr8conf 2011
Redis and Groovy and Grails - gr8conf 2011Ted Naleid
 
美团点评沙龙012-初创电商的物流摸索
美团点评沙龙012-初创电商的物流摸索美团点评沙龙012-初创电商的物流摸索
美团点评沙龙012-初创电商的物流摸索美团点评技术团队
 
美团点评技术沙龙010-点评RDS系统介绍
美团点评技术沙龙010-点评RDS系统介绍美团点评技术沙龙010-点评RDS系统介绍
美团点评技术沙龙010-点评RDS系统介绍美团点评技术团队
 
美团点评技术沙龙09 - 美团配送智能调度实践
美团点评技术沙龙09 - 美团配送智能调度实践美团点评技术沙龙09 - 美团配送智能调度实践
美团点评技术沙龙09 - 美团配送智能调度实践美团点评技术团队
 
美团点评技术沙龙010-美团Atlas实践
美团点评技术沙龙010-美团Atlas实践美团点评技术沙龙010-美团Atlas实践
美团点评技术沙龙010-美团Atlas实践美团点评技术团队
 
美团点评技术沙龙011 - 移动app兼容性测试工具Spider
美团点评技术沙龙011 - 移动app兼容性测试工具Spider 美团点评技术沙龙011 - 移动app兼容性测试工具Spider
美团点评技术沙龙011 - 移动app兼容性测试工具Spider 美团点评技术团队
 
美团点评沙龙012-从零到千万量级的实时物流平台架构实践
美团点评沙龙012-从零到千万量级的实时物流平台架构实践美团点评沙龙012-从零到千万量级的实时物流平台架构实践
美团点评沙龙012-从零到千万量级的实时物流平台架构实践美团点评技术团队
 
美团点评技术沙龙011 - 团购系统流量和容量评估实践
美团点评技术沙龙011 - 团购系统流量和容量评估实践美团点评技术沙龙011 - 团购系统流量和容量评估实践
美团点评技术沙龙011 - 团购系统流量和容量评估实践美团点评技术团队
 
美团点评技术沙龙011 - 客户端用户体验数据量化
美团点评技术沙龙011 - 客户端用户体验数据量化美团点评技术沙龙011 - 客户端用户体验数据量化
美团点评技术沙龙011 - 客户端用户体验数据量化美团点评技术团队
 
Taste Rabbitmq
Taste RabbitmqTaste Rabbitmq
Taste Rabbitmqjeff kit
 

Viewers also liked (20)

Redis介绍
Redis介绍Redis介绍
Redis介绍
 
新浪微博开放平台Redis实战
新浪微博开放平台Redis实战新浪微博开放平台Redis实战
新浪微博开放平台Redis实战
 
高性能No sql数据库redis
高性能No sql数据库redis高性能No sql数据库redis
高性能No sql数据库redis
 
redis 适用场景与实现
redis 适用场景与实现redis 适用场景与实现
redis 适用场景与实现
 
美团点评沙龙 飞行中换引擎--美团配送业务系统的架构演进之路
美团点评沙龙 飞行中换引擎--美团配送业务系统的架构演进之路美团点评沙龙 飞行中换引擎--美团配送业务系统的架构演进之路
美团点评沙龙 飞行中换引擎--美团配送业务系统的架构演进之路
 
Redis 常见使用模式分析
Redis 常见使用模式分析Redis 常见使用模式分析
Redis 常见使用模式分析
 
美团点评技术沙龙010-Redis Cluster运维实践
美团点评技术沙龙010-Redis Cluster运维实践美团点评技术沙龙010-Redis Cluster运维实践
美团点评技术沙龙010-Redis Cluster运维实践
 
Redis cluster
Redis clusterRedis cluster
Redis cluster
 
Everything you always wanted to know about Redis but were afraid to ask
Everything you always wanted to know about Redis but were afraid to askEverything you always wanted to know about Redis but were afraid to ask
Everything you always wanted to know about Redis but were afraid to ask
 
Redis and Groovy and Grails - gr8conf 2011
Redis and Groovy and Grails - gr8conf 2011Redis and Groovy and Grails - gr8conf 2011
Redis and Groovy and Grails - gr8conf 2011
 
美团点评沙龙012-初创电商的物流摸索
美团点评沙龙012-初创电商的物流摸索美团点评沙龙012-初创电商的物流摸索
美团点评沙龙012-初创电商的物流摸索
 
美团点评技术沙龙010-点评RDS系统介绍
美团点评技术沙龙010-点评RDS系统介绍美团点评技术沙龙010-点评RDS系统介绍
美团点评技术沙龙010-点评RDS系统介绍
 
美团点评技术沙龙09 - 美团配送智能调度实践
美团点评技术沙龙09 - 美团配送智能调度实践美团点评技术沙龙09 - 美团配送智能调度实践
美团点评技术沙龙09 - 美团配送智能调度实践
 
美团点评技术沙龙010-美团Atlas实践
美团点评技术沙龙010-美团Atlas实践美团点评技术沙龙010-美团Atlas实践
美团点评技术沙龙010-美团Atlas实践
 
美团点评技术沙龙011 - 移动app兼容性测试工具Spider
美团点评技术沙龙011 - 移动app兼容性测试工具Spider 美团点评技术沙龙011 - 移动app兼容性测试工具Spider
美团点评技术沙龙011 - 移动app兼容性测试工具Spider
 
Redis Replication
Redis ReplicationRedis Replication
Redis Replication
 
美团点评沙龙012-从零到千万量级的实时物流平台架构实践
美团点评沙龙012-从零到千万量级的实时物流平台架构实践美团点评沙龙012-从零到千万量级的实时物流平台架构实践
美团点评沙龙012-从零到千万量级的实时物流平台架构实践
 
美团点评技术沙龙011 - 团购系统流量和容量评估实践
美团点评技术沙龙011 - 团购系统流量和容量评估实践美团点评技术沙龙011 - 团购系统流量和容量评估实践
美团点评技术沙龙011 - 团购系统流量和容量评估实践
 
美团点评技术沙龙011 - 客户端用户体验数据量化
美团点评技术沙龙011 - 客户端用户体验数据量化美团点评技术沙龙011 - 客户端用户体验数据量化
美团点评技术沙龙011 - 客户端用户体验数据量化
 
Taste Rabbitmq
Taste RabbitmqTaste Rabbitmq
Taste Rabbitmq
 

Similar to 深入了解Redis

Build an High-Performance and High-Durable Block Storage Service Based on Ceph
Build an High-Performance and High-Durable Block Storage Service Based on CephBuild an High-Performance and High-Durable Block Storage Service Based on Ceph
Build an High-Performance and High-Durable Block Storage Service Based on CephRongze Zhu
 
Bottom to Top Stack Optimization with LAMP
Bottom to Top Stack Optimization with LAMPBottom to Top Stack Optimization with LAMP
Bottom to Top Stack Optimization with LAMPkatzgrau
 
Bottom to Top Stack Optimization - CICON2011
Bottom to Top Stack Optimization - CICON2011Bottom to Top Stack Optimization - CICON2011
Bottom to Top Stack Optimization - CICON2011CodeIgniter Conference
 
Cassandra
CassandraCassandra
Cassandraexsuns
 
ZFS for Databases
ZFS for DatabasesZFS for Databases
ZFS for Databasesahl0003
 
Comparison of foss distributed storage
Comparison of foss distributed storageComparison of foss distributed storage
Comparison of foss distributed storageMarian Marinov
 
美团技术沙龙04 - 高性能服务器架构设计和调优
美团技术沙龙04 - 高性能服务器架构设计和调优美团技术沙龙04 - 高性能服务器架构设计和调优
美团技术沙龙04 - 高性能服务器架构设计和调优美团点评技术团队
 
Comparison of-foss-distributed-storage
Comparison of-foss-distributed-storageComparison of-foss-distributed-storage
Comparison of-foss-distributed-storageMarian Marinov
 
DEF CON 27- ITZIK KOTLER and AMIT KLEIN - gotta catch them all
DEF CON 27- ITZIK KOTLER and AMIT KLEIN - gotta catch them allDEF CON 27- ITZIK KOTLER and AMIT KLEIN - gotta catch them all
DEF CON 27- ITZIK KOTLER and AMIT KLEIN - gotta catch them allFelipe Prado
 
(DAT407) Amazon ElastiCache: Deep Dive
(DAT407) Amazon ElastiCache: Deep Dive(DAT407) Amazon ElastiCache: Deep Dive
(DAT407) Amazon ElastiCache: Deep DiveAmazon Web Services
 
What's new in Jewel and Beyond
What's new in Jewel and BeyondWhat's new in Jewel and Beyond
What's new in Jewel and BeyondSage Weil
 
Automatic tool for static analysis
Automatic tool for static analysisAutomatic tool for static analysis
Automatic tool for static analysisChong-Kuan Chen
 
Scaling web applications with cassandra presentation
Scaling web applications with cassandra presentationScaling web applications with cassandra presentation
Scaling web applications with cassandra presentationMurat Çakal
 
Community Update at OpenStack Summit Boston
Community Update at OpenStack Summit BostonCommunity Update at OpenStack Summit Boston
Community Update at OpenStack Summit BostonSage Weil
 
Talk about apache cassandra, TWJUG 2011
Talk about apache cassandra, TWJUG 2011Talk about apache cassandra, TWJUG 2011
Talk about apache cassandra, TWJUG 2011Boris Yen
 
Talk About Apache Cassandra
Talk About Apache CassandraTalk About Apache Cassandra
Talk About Apache CassandraJacky Chu
 
High Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseHigh Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseAltinity Ltd
 

Similar to 深入了解Redis (20)

Build an High-Performance and High-Durable Block Storage Service Based on Ceph
Build an High-Performance and High-Durable Block Storage Service Based on CephBuild an High-Performance and High-Durable Block Storage Service Based on Ceph
Build an High-Performance and High-Durable Block Storage Service Based on Ceph
 
Bottom to Top Stack Optimization with LAMP
Bottom to Top Stack Optimization with LAMPBottom to Top Stack Optimization with LAMP
Bottom to Top Stack Optimization with LAMP
 
Bottom to Top Stack Optimization - CICON2011
Bottom to Top Stack Optimization - CICON2011Bottom to Top Stack Optimization - CICON2011
Bottom to Top Stack Optimization - CICON2011
 
Strata - 03/31/2012
Strata - 03/31/2012Strata - 03/31/2012
Strata - 03/31/2012
 
Cassandra
CassandraCassandra
Cassandra
 
ZFS for Databases
ZFS for DatabasesZFS for Databases
ZFS for Databases
 
Comparison of foss distributed storage
Comparison of foss distributed storageComparison of foss distributed storage
Comparison of foss distributed storage
 
Scale 10x 01:22:12
Scale 10x 01:22:12Scale 10x 01:22:12
Scale 10x 01:22:12
 
美团技术沙龙04 - 高性能服务器架构设计和调优
美团技术沙龙04 - 高性能服务器架构设计和调优美团技术沙龙04 - 高性能服务器架构设计和调优
美团技术沙龙04 - 高性能服务器架构设计和调优
 
Comparison of-foss-distributed-storage
Comparison of-foss-distributed-storageComparison of-foss-distributed-storage
Comparison of-foss-distributed-storage
 
DEF CON 27- ITZIK KOTLER and AMIT KLEIN - gotta catch them all
DEF CON 27- ITZIK KOTLER and AMIT KLEIN - gotta catch them allDEF CON 27- ITZIK KOTLER and AMIT KLEIN - gotta catch them all
DEF CON 27- ITZIK KOTLER and AMIT KLEIN - gotta catch them all
 
(DAT407) Amazon ElastiCache: Deep Dive
(DAT407) Amazon ElastiCache: Deep Dive(DAT407) Amazon ElastiCache: Deep Dive
(DAT407) Amazon ElastiCache: Deep Dive
 
What's new in Jewel and Beyond
What's new in Jewel and BeyondWhat's new in Jewel and Beyond
What's new in Jewel and Beyond
 
Automatic tool for static analysis
Automatic tool for static analysisAutomatic tool for static analysis
Automatic tool for static analysis
 
Scaling web applications with cassandra presentation
Scaling web applications with cassandra presentationScaling web applications with cassandra presentation
Scaling web applications with cassandra presentation
 
Community Update at OpenStack Summit Boston
Community Update at OpenStack Summit BostonCommunity Update at OpenStack Summit Boston
Community Update at OpenStack Summit Boston
 
Talk about apache cassandra, TWJUG 2011
Talk about apache cassandra, TWJUG 2011Talk about apache cassandra, TWJUG 2011
Talk about apache cassandra, TWJUG 2011
 
Talk About Apache Cassandra
Talk About Apache CassandraTalk About Apache Cassandra
Talk About Apache Cassandra
 
Hbase: an introduction
Hbase: an introductionHbase: an introduction
Hbase: an introduction
 
High Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseHigh Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouse
 

More from iammutex

Scaling Instagram
Scaling InstagramScaling Instagram
Scaling Instagramiammutex
 
NoSQL误用和常见陷阱分析
NoSQL误用和常见陷阱分析NoSQL误用和常见陷阱分析
NoSQL误用和常见陷阱分析iammutex
 
MongoDB 在盛大大数据量下的应用
MongoDB 在盛大大数据量下的应用MongoDB 在盛大大数据量下的应用
MongoDB 在盛大大数据量下的应用iammutex
 
8 minute MongoDB tutorial slide
8 minute MongoDB tutorial slide8 minute MongoDB tutorial slide
8 minute MongoDB tutorial slideiammutex
 
Thoughts on Transaction and Consistency Models
Thoughts on Transaction and Consistency ModelsThoughts on Transaction and Consistency Models
Thoughts on Transaction and Consistency Modelsiammutex
 
Rethink db&tokudb调研测试报告
Rethink db&tokudb调研测试报告Rethink db&tokudb调研测试报告
Rethink db&tokudb调研测试报告iammutex
 
Introduction to couchdb
Introduction to couchdbIntroduction to couchdb
Introduction to couchdbiammutex
 
What every data programmer needs to know about disks
What every data programmer needs to know about disksWhat every data programmer needs to know about disks
What every data programmer needs to know about disksiammutex
 
redis运维之道
redis运维之道redis运维之道
redis运维之道iammutex
 
Realtime hadoopsigmod2011
Realtime hadoopsigmod2011Realtime hadoopsigmod2011
Realtime hadoopsigmod2011iammutex
 
[译]No sql生态系统
[译]No sql生态系统[译]No sql生态系统
[译]No sql生态系统iammutex
 
Couchdb + Membase = Couchbase
Couchdb + Membase = CouchbaseCouchdb + Membase = Couchbase
Couchdb + Membase = Couchbaseiammutex
 
Redis cluster
Redis clusterRedis cluster
Redis clusteriammutex
 
Hadoop introduction berlin buzzwords 2011
Hadoop introduction   berlin buzzwords 2011Hadoop introduction   berlin buzzwords 2011
Hadoop introduction berlin buzzwords 2011iammutex
 
Couchdb and me
Couchdb and meCouchdb and me
Couchdb and meiammutex
 
10 Key MongoDB Performance Indicators
10 Key MongoDB Performance Indicators  10 Key MongoDB Performance Indicators
10 Key MongoDB Performance Indicators iammutex
 
MongoDB开发应用实践
MongoDB开发应用实践MongoDB开发应用实践
MongoDB开发应用实践iammutex
 

More from iammutex (20)

Scaling Instagram
Scaling InstagramScaling Instagram
Scaling Instagram
 
NoSQL误用和常见陷阱分析
NoSQL误用和常见陷阱分析NoSQL误用和常见陷阱分析
NoSQL误用和常见陷阱分析
 
MongoDB 在盛大大数据量下的应用
MongoDB 在盛大大数据量下的应用MongoDB 在盛大大数据量下的应用
MongoDB 在盛大大数据量下的应用
 
8 minute MongoDB tutorial slide
8 minute MongoDB tutorial slide8 minute MongoDB tutorial slide
8 minute MongoDB tutorial slide
 
skip list
skip listskip list
skip list
 
Thoughts on Transaction and Consistency Models
Thoughts on Transaction and Consistency ModelsThoughts on Transaction and Consistency Models
Thoughts on Transaction and Consistency Models
 
Rethink db&tokudb调研测试报告
Rethink db&tokudb调研测试报告Rethink db&tokudb调研测试报告
Rethink db&tokudb调研测试报告
 
Introduction to couchdb
Introduction to couchdbIntroduction to couchdb
Introduction to couchdb
 
What every data programmer needs to know about disks
What every data programmer needs to know about disksWhat every data programmer needs to know about disks
What every data programmer needs to know about disks
 
Ooredis
OoredisOoredis
Ooredis
 
Ooredis
OoredisOoredis
Ooredis
 
redis运维之道
redis运维之道redis运维之道
redis运维之道
 
Realtime hadoopsigmod2011
Realtime hadoopsigmod2011Realtime hadoopsigmod2011
Realtime hadoopsigmod2011
 
[译]No sql生态系统
[译]No sql生态系统[译]No sql生态系统
[译]No sql生态系统
 
Couchdb + Membase = Couchbase
Couchdb + Membase = CouchbaseCouchdb + Membase = Couchbase
Couchdb + Membase = Couchbase
 
Redis cluster
Redis clusterRedis cluster
Redis cluster
 
Hadoop introduction berlin buzzwords 2011
Hadoop introduction   berlin buzzwords 2011Hadoop introduction   berlin buzzwords 2011
Hadoop introduction berlin buzzwords 2011
 
Couchdb and me
Couchdb and meCouchdb and me
Couchdb and me
 
10 Key MongoDB Performance Indicators
10 Key MongoDB Performance Indicators  10 Key MongoDB Performance Indicators
10 Key MongoDB Performance Indicators
 
MongoDB开发应用实践
MongoDB开发应用实践MongoDB开发应用实践
MongoDB开发应用实践
 

Recently uploaded

Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsNathaniel Shimoni
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPathCommunity
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...Wes McKinney
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesBernd Ruecker
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterMydbops
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityIES VE
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical InfrastructureVarsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructureitnewsafrica
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesManik S Magar
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Nikki Chapple
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 

Recently uploaded (20)

Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directions
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to Hero
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architectures
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL Router
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a reality
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical InfrastructureVarsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 

深入了解Redis

  • 1. 深入了解Redis 宋传胜 Chuansheng.song@langtaojin.com
  • 2. 简单介绍 • 文本协议 memcached类似 • KEY 可打印字符 • VALUE支持的类型 – STRINGS – LIST – SET – SORTED SET – HASH • 高性能 (100k+ SET/80k+ GET)/s • 序列化支持 • 主从同步支持 • 客户端自己实现sharding
  • 3. 基本数据结构 • RedisObject (Redis.h) – key/value对象的基础
  • 4. KEY • 基本命令 – get/set/exists O(1) – setnx/randomkey/rename/renamenx/dbsize/type – keys pattern – del 一次删除多个key – expire key seconds 指定过期时间 – ttl key 剩余过期时间 – select db – move key newdb – flush db/flushall
  • 5. EXPIRE AND LRU CACHE • Volatile Key (Db.c) – When the key is set to a new value using the SET command, or when a key is destroyed via DEL, the timeout is removed from the key. • Enhanced Lazy Expiration algorithm – Redis does not constantly monitor keys that are going to be expired. Keys are expired simply when some client tries to access a key, and the key is found to be timed out. • Work as LRU cache (memcached) – Maxmemory/maxmemory-policy – When memory limit was already reached, server will remove some old data deleting a volatile key, even if the key is still far from expiring automatically. – Random get keys, delete by lru rules.
  • 7. 基本Value数据结构 • STRING(sds.c) – set/setnx/get/getset/mget/mset/msetnx •nx – not exists •m -- multiple – incr/decr/incrby/decrby – getrange/append
  • 8. 基本Value数据结构 • LIST (T_list.c) – REDIS_LIST 类型, 如果其 entry 小于配置值: list- max-ziplist-entries 或 value字符串的长度小于 list- max-ziplist-value,使用ziplist数据结构,否则使用 标准的Doubly linked list – (l/r)push/(l/r)pop/llen O(1) – b(l/r)pop支持阻塞等待,避免了轮循 – lrange/ltrim/lrem/lset/rpoplpush
  • 9. LIST • Ziplist (Ziplist.c) – O(mem_size) add/delete – list-max-ziplist-entries (default: 1024) – list-max-ziplist-value (default: 32)
  • 10. LIST • Ziplist (continue) – zlentry (unencode structure)
  • 11. LIST • Doubly linked list (Adlist.c)
  • 12. 基本Value数据结构 • SET (T_set.c) – hashtable + Intset – String的无序集合 – sadd/srem/sismember/scard O(1) – spop/srandmember – smove – smembers – sinter(store) O(C) – sunion(store)/sdiff(store) O(N)
  • 13. INTSET • INTSET (Intset.c) – 都是整型时数据结构退化成排序的intset – Good fit for size up to 20-50K – set-max-intset-entries (default: 4096) – O(n) search – O(log(n) + mem_size) add/del
  • 14. 基本Value数据结构 • SORTED SET(T_zset.c) hashtable + skip list – 按照key的score排序 – zadd/zrem/zrank/zrevrank O(log(n)) – zcard O(1) – zincrby 修改score – zrange/zrevrange/zrangebyscore/zscore O(log(N)+M) – zremrangebyrank/zremrangebyscore O(log(N)+M)
  • 15. SKIP LIST • Skip List (T_zset.c) – skip list  redblack tree  AVL tree (more balanced) – lockfree/concurrency
  • 16. 基本Value数据结构 • HASH (T_hash.c) zipmap + hashtable – 如果其 entry 小于配置值: hash-max-zipmap- entries 或 value字符串的长度小于 hash-max- zipmap-value ,使用zipmap数据结构 – Value只能是string类型 – hset/hget/hexists O(1) – hmset/hexists/hincrby/hdel/hlen – hmget O(N) – hkeys/hvals/hgetall
  • 17. ZIPMAP • zipmap – O(n) search O(mem_size) add/delete
  • 18. 内存管理 • zmalloc.c – 简单内存管理 – 支持tcmalloc USE_TCMALLOC – zmalloc(size_t size)  malloc(size+PREFIX_SIZE) – fragmentation_ratio = zmalloc_get_rss/zmalloc_used_memory
  • 19. PUBSUB • SUBSCRIBE/UNSUBSCRIBE/PSUBSCRIBE/PU NSUBSCRIBE • PUBLISH • 连接断开后信息丢失
  • 20. TRANSACTION • MULTI (不支持CAS) – MULTI – SET foo 0 – INCR foo – EXEC • WATCH (Check and Set ) – WATCH theKey – v = GET theKey – MULTI – SET theKey v+1 – EXEC • UNWATCH/DISCARD
  • 21. 提升网络效率、减少内存占用 • MGET/MSET • PIPELINING – $ (echo -en "PINGrnPINGrnPINGrn"; sleep 1) | nc localhost 6379 +PONG +PONG +PONG • 压缩数据,触发ziplist/zipmap/intset • 使用 GETBIT/SETBIT/GETRANGE/SETRANGE 压缩多 个属性 • 使用HASH • 使用32位redis + sharding
  • 22. Client Side Sharding • Consistent Hashing – Redis Sharding at Craigslist
  • 23. 网络层 • Anet.c • 事件驱动 ae.c  ae_epoll.c/ae_kqueue.c/ae_select.c • 非阻塞单线程 • 支持timer • 在VM场景下有可能多线程
  • 25. 网络层对比 • lighttpd/nginx的网络IO模型 – 多进程,单线程非阻塞IO • apache的MPM模型 – 多进程prefork,多线程
  • 26. PERSISTENCE • SNAPSHOT (Rdb.c) • 可以关闭 • BGSAVE or save xseconds ychanges • fork – parent - disable rehash, periodic check child on serverCron (100ms) – child - copy-on-write, save whole db, exit • Need to turn on the overcommit_memory setting if you want to deal with a dataset more than 1/2 of RAM, which contradicts our habit to battle against OOM killer as a database administrator.
  • 27. Append Only File • Append to log file on every change (Aof.c) • fsync() policy – always/os decide/every second – Always with battery backed raid controllers – Every second by default (innodb_flush_log_at_trx_commit=2 in mysql) • Compact aof file – BGREWRITEAOF/REWRITEAOF • Fork – child – write new aof in temporary file – parent – write change in both old aof file and memory buffer, append memory buffer to temporary file when child done, rename temporary to aof file. • redis-check-aof --fix <filename>
  • 29. Replication • 支持AUTH • 起动过程 – slave •SYNC •Wait – master •Issue a BGSAVE, or wait if BGSAVE in progress
  • 30. Virtual Memory • Redis VM is now deprecated. Redis 2.4 will be the latest Redis version featuring Virtual Memory! • Why not OS VM – Varnish What’s wrong with 1975 programming ? – Redis What’s wrong with 2006 programming? • Blocking when page fault • 4k pages granularity • Optimal disk persistent format, 10:1 • Keys can’t be swapped out • vm is read only when BGSAVE/BGREWRITEAOF in progress • vm-max-memory/vm-pages/vm-page-size
  • 31. Virtual Memory • Blocking vm • Threaded vm – Before command is executed, check the value – Block client and load, then signal by unix pipe – Continue • Recommended filesystem – ext3 or any other file system with good support for sparse files • redis-stat vmstat • InnoDB innodb_buffer_pool_size – swap and durability
  • 32. Future • Diskstore (diskstore.c dscache.c) – All data on disk, B+tree in future – Swaps data into memory when needed • Leveldb/InnoDB Change Buffer – LSM tree
  • 33. 其他相关的代码 • quicksort (pqsort.c) • lzf compression (lzf_c.c lzf_d.c) • Sha1 (Sha1.c) • Syncio.c • SORT (Sort.c)
  • 34. 线上Redis集群情况 • 16台服务器,兼做后端服务器 • 45个Redis Instance进程,三组服务 • 总内存 77G VIRT,71G RSS • 使用了主从复制、snapshot持久化,VM支持 ,客户端sharding、pipelining、hash优化
  • 35. Redis应用 • Resque – message queue of github (celery) • Nginx HttpRedis module • Redweb - Web administrative and query UI • A fast, fuzzy, full-text index using Redis • Redis-backed BloomFilter(s) in Ruby • Soulmate : Auto Complete with Redis • Rate limiting with Redis • Redis Sharding • Locking with setnx • A Collection Of Redis Use Cases • Radishapp Redis monitoring • openredis.com hosting service
  • 36. Contribute to Redis • Fork redis on github – https://github.com/antirez/redis • Choose branch and commit your changes • Create a pull request • Wait for approval – https://github.com/antirez/redis/pulls • My contribute for read only slave – https://github.com/antirez/redis/pull/47
  • 37. Resources • DOCUMENATION – http://redis.io/documentation – http://antirez.com/ – http://groups.google.com/group/redis-db • 本文参考的Redis资料集合 – http://www.delicious.com/fakechris/redis – http://www.delicious.com/tag/redis