用ElasticSearch,LogStash,Kibana搭建实时日志收集系统
来自： http://blog.csdn.net//jiao_fuyou/article/details/46694125
用ElasticSearch,LogStash,Kibana搭建实时日志收集系统
介绍
- 这套系统，logstash负责收集处理日志文件内容存储到elasticsearch搜索引擎数据库中。kibana负责查询elasticsearch并在web中展示。
- logstash收集进程收获日志文件内容后，先输出到redis中缓存，另一logstash处理进程从redis中读出并转存到elasticsearch中，以解决读快写慢速度不一致问题。
- 官方在线文档：https://www.elastic.co/guide/index.html
一、安装jdk7
ElasticSearch,LogStash均是java程序，所以需要jdk环境。
需要注意的是，多节点通讯，必须保证JDK版本一致，不然可能会导致连接失败。下载：jdk-7u71-linux-x64.rpm
http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.htmlrpm -ivh jdk-7u71-linux-x64.rpm
配置JDK
编辑/etc/profile文件，在开头加入：export JAVA_HOME=/usr/java/jdk1.7.0_71 export JRE_HOME=$JAVA_HOME/jre export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
检查JDK环境
使用source /etc/profile命令，使环境变量立即生效。
查看当前安装的JDK版本，命令：java -version
检查环境变量，echo $PATH
二、安装elasticsearch
elasticsearch是一个搜索引擎，负责存储日志内容。
下载安装
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.4.4.tar.gz
tar zxvf elasticsearch-1.4.4.tar.gz修改config/elasticsearch.yml配置文件
bootstrap.mlockall: true index.number_of_shards: 1 index.number_of_replicas: 0 #index.translog.flush_threshold_ops: 100000 #index.refresh_interval: -1 index.translog.flush_threshold_ops: 5000 index.refresh_interval: 1 network.bind_host: 172.16.18.114 #节点间通讯发布到其它节点的IP地址 #如果不设置由ES自己决定它可能会发现一个地址，但是其它节点可能访问不了，这样节点间通讯将失败 network.publish_host: 172.16.18.114 # Security 允许所有http请求 http.cors.enabled: true http.cors.allow-origin: "/.*/"
- 修改bin/elasticsearch文件
# 使jvm使用os，max-open-files es_parms="-Delasticsearch -Des.max-open-files=ture" # Start up the service # 修改OS打开最大文件数 ulimit -n 1000000 ulimit -l unlimited launch_service "$pidfile" "$daemonized" "$properties"
- 修改bin/elasticsearch.in.sh文件
...... if [ "x$ES_MIN_MEM" = "x" ]; then ES_MIN_MEM=256m fi if [ "x$ES_MAX_MEM" = "x" ]; then ES_MAX_MEM=1g fi if [ "x$ES_HEAP_SIZE" != "x" ]; then ES_MIN_MEM=$ES_HEAP_SIZE ES_MAX_MEM=$ES_HEAP_SIZE fi #set min memory as 2g ES_MIN_MEM=2g #set max memory as 2g ES_MAX_MEM=2g ......
运行
./bin/elasticsearch -d
./logs下为日志文件检查节点状态
curl -XGET ‘http://localhost:9200/_nodes?os=true&process=true&pretty=true’{ "cluster_name" : "elasticsearch", "nodes" : { "7PEaZbvxToCL2O2KuMGRYQ" : { "name" : "Gertrude Yorkes", "transport_address" : "inet[/172.16.18.116:9300]", "host" : "casimbak", "ip" : "172.16.18.116", "version" : "1.4.4", "build" : "c88f77f", "http_address" : "inet[/172.16.18.116:9200]", "settings" : { "index": { "number_of_replicas": "0", "translog": { "flush_threshold_ops": "5000" }, "number_of_shards": "1", "refresh_interval": "1" }, "path" : { "logs" : "/home/jfy/soft/elasticsearch-1.4.4/logs", "home" : "/home/jfy/soft/elasticsearch-1.4.4" }, "cluster" : { "name" : "elasticsearch" }, "bootstrap" : { "mlockall" : "true" }, "client" : { "type" : "node" }, "http" : { "cors" : { "enabled" : "true", "allow-origin" : "/.*/" } }, "foreground" : "yes", "name" : "Gertrude Yorkes", "max-open-files" : "ture" }, "process" : { "refresh_interval_in_millis" : 1000, "id" : 13896, "max_file_descriptors" : 1000000, "mlockall" : true }, ... } } }表明ElasticSearch已运行，状态与配置相符
"index": { "number_of_replicas": "0", "translog": { "flush_threshold_ops": "5000" }, "number_of_shards": "1", "refresh_interval": "1" }, "process" : { "refresh_interval_in_millis" : 1000, "id" : 13896, "max_file_descriptors" : 1000000, "mlockall" : true },安装head插件操作elasticsearch
elasticsearch/bin/plugin -install mobz/elasticsearch-head
http://172.16.18.116:9200/_plugin/head/安装marvel插件监控elasticsearch状态
elasticsearch/bin/plugin -i elasticsearch/marvel/latest
http://172.16.18.116:9200/_plugin/marvel/
三、安装logstash
logstash一个日志收集处理过滤程序。
LogStash分为日志收集端进程和日志处理端进程，收集端负责收集多个日志文件实时的将日志内容输出到redis队列缓存，处理端负责将redis队列缓存中的内容输出到ElasticSarch中存储。收集端进程运行在产生日志文件的服务器上，处理端进程运行在redis,elasticsearch同一服务器上。
下载
wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gzredis安装配置略，但要注意监控redis队列长度，如果长时间堆集说明elasticsearch出问题了
每2S检查一下redis中数据列表长度，100次
redis-cli -r 100 -i 2 llen logstash:redis配置Logstash日志收集进程
vi ./lib/logstash/config/shipper.conf
input { #file { # type => "mysql_log" # path => "/usr/local/mysql/data/localhost.log" # codec => plain{ # charset => "GBK" # } #} file { type => "hostapd_log" path => "/root/hostapd/hostapd.log" sincedb_path => "/home/jfy/soft/logstash-1.4.2/sincedb_hostapd.access" #start_position => "beginning" #http://logstash.net/docs/1.4.2/codecs/plain codec => plain{ charset => "GBK" } } file { type => "hkt_log" path => "/usr1/app/log/bsapp.tr" sincedb_path => "/home/jfy/soft/logstash-1.4.2/sincedb_hkt.access" start_position => "beginning" codec => plain{ charset => "GBK" } } # stdin { # type => "hostapd_log" # } } #filter { # grep { # match => [ "@message", "mysql|GET|error" ] # } #} output { redis { host => '172.16.18.116' data_type => 'list' key => 'logstash:redis' # codec => plain{ # charset => "UTF-8" # } } # elasticsearch { # #embedded => true # host => "172.16.18.116" # } } 运行收集端进程
./bin/logstash agent -f ./lib/logstash/config/shipper.conf配置Logstash日志处理进程
vi ./lib/logstash/config/indexer.confinput { redis { host => '127.0.0.1' data_type => 'list' key => 'logstash:redis' #threads => 10 #batch_count => 1000 } } output { elasticsearch { #embedded => true host => localhost #workers => 10 } }运行处理端进程
./bin/logstash agent -f ./lib/logstash/config/indexer.conf
处理端从redis读出缓存的日志内容，输出到ElasticSarch中存储
四、安装kibana
kibana是elasticsearch搜索引擎的web展示界面，一套在webserver下的js脚本,可以定制复杂的查询过滤条件检索elasticsearch，并以多种方式（表格，图表）展示。
下载
wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.2.tar.gz
解压后将kibana目录放到webserver能访问到的地方配置
修改kibana/config.js:
如果kibana与elasticsearch不在同一机器则修改： elasticsearch: "http://192.168.91.128:9200", #这里实际上是浏览器直接访问该地址连接elasticsearch 否则默认，一定不要修改
如果出现connection failed，则修改elasticsearch/config/elasticsearch.yml，增加：
http.cors.enabled: true http.cors.allow-origin: "/.*/"
具体含义参见：
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-http.html
访问kibana
http://172.16.18.114:6090/kibana/index.html#/dashboard/file/logstash.json配置kibana界面
可以filtering中配置需要访问的日志type,如_type=voip_log,对应着上面logstash中shipper.conf的type
也可以点击右上角的save将当前界面的配置保存到elasticsearch中，默认是保存在kibana-int索引中