SlideShare a Scribd company logo
1 of 47
Download to read offline
Fluentd vs. Logstash
Masaki Matsushita
NTT Communications
About Me
● Masaki MATSUSHITA
● Software Engineer at
○ We are providing Internet access here!
● Github: mmasaki Twitter: @_mmasaki
● 16 Commits in Liberty
○ Trove, oslo_log, oslo_config
● CRuby Commiter
○ 100+ commits for performance improvement
2
What are Log Collectors?
● Provide pluggable and unified logging layer
Without Log Collectors With Log Collectors
Images from http://fluentd.org/ 3
Input, Filter and Output
4
Input Plugins
tail
syslog
Filter Plugins
grep
hostname
Output Plugins
InfluxDB
Elasticsearch
● They are implemented as plugins
● Can be replaced easily
Log FIles
Components
Two Popular Log Collectors
● Fluentd
○ Written in CRuby
○ Used in Kubernetes
○ Maintained by Treasure Data Inc.
● Logstash
○ Written in JRuby
○ Maintained by elastic.co
● They have similar features
● Which one is better for you? 5
Agenda
● Comparisons
○ Configuration
○ Supported Plugins
○ Performance
○ Transport Protocol
● Integrate OpenStack with Fluentd/Logstash
○ Considering High Availability 6
Configuration: Fluentd
● Every inputs are tagged
● Logs will be routed by tag
nova-api.log
(tag: openstack.nova)
cinder-api.log
(tag: openstack.cinder)
<match openstack.nova>
<match openstack.cinder>
Filter/Route
7
Fluentd Configuration: Input
<source>
@type tail
path /var/log/nova/nova-api.log
tag openstack.nova
</source>
Example of tailing nova-api log
● Every inputs will be tagged
8
Fluentd Configuration: Output
<match openstack.nova> # nova related logs
@type elasticsearch
host example.com
</match>
<match openstack.*> # all other OpenStack related logs
@type influxdb
# …
</match>
Routed by tag
(First match is priority)
Wildcards can be used
9
Fluentd Configuration: Copy
<match openstack.*>
@type copy
<store>
@type influxdb
</store>
<store>
@type elasticsearch
</store>
</match>
Copy plugin enables multiple
outputs for a tag
Copied Output
tag: openstack.*
10
Logstash Configuration
● No tags
● All inputs will be aggregated
● Logs will be scattered to outputs
nova-api.log
cinder-api.log
Filter/Aggregate
aggregated logs
11
Logstash Configuration
input {
file { path => “/var/log/nova/*.log” }
file { path => “/var/log/cinder/*.log” }
}
output {
elasticsearch { hosts => [“example.com”] }
influxdb { host => “example.com”... }
}
12
Case 1: Separated Streams
Input1
Input2
Input3
Output2
Output3
Output1
● Handle multiple streams separately
13
Case 1: Separated Streams
Fluentd: Simple matching by tag
<match input.input1>
@type output1
</match>
<match input.input2>
@type output2
</match>
<match input.input3>
@type output3
</match>
Logstash: Conditional Outputs
output {
if [type] == “input1” {
output1 {}
} else if [type] == “input2” {
output2 {}
} else if [type] == “input3” {
output3 {}
}
}
Need to split aggregated logs
14
Case 2: Aggregated Streams
Input1
Input2
Input3
Output2
Output3
Output1
● Streams will be aggregated and scattered
15
Case 2: Aggregated Streams
Fluentd: Copy plugins is needed
<match input.*>
@type copy
<store>
@type output1
</store>
<store>
@type output2
</store>
<store>
@type output3
</store>
</match>
Logstash: Quite simple
output {
output1 {}
output2 {}
output3 {}
}
16
Configuration
● Fluentd
○ Routed by simple tag matching
○ Suited to handle log streams separately
● Logstash
○ Logs are aggregated
○ Suited to handle logs in gather-scatter style
17
Plugins
● Both provide many plugins
○ Fluentd: 300+, Logstash: 200+
● Popular plugins are bundled with Logstash
○ They are maintained by the Logstash project
● Fluentd contains only minimal plugins
○ Most plugins are maintained by individuals
● Plugins can be installed easily by one command
18
Performance
● Depends on circumstances
● More than enough for OpenStack logs
○ Both can handle 10000+ logs/s
● Applying heavy filters is not a good idea
● CRuby is slow because of GVL?
○ GVL: Global VM (Interpreter) Lock
○ It’s not true for IO bound loads
19
GVL on IO bound loads
● IO operation can be performed in parallel
20
Thread 1 Thread 2
Idle :
User Space:
Kernel Space:
Actual Read/Write
Ruby Code Execution
GVL Released/
Acquired
IO operations
in parallel
Transport Protocol
● Both collectors have their own transport protocol.
○ Failure Detection and Fallback
● Logstash: Lumberjack protocol
○ Active-Standby only
● Fluentd: forward protocol
○ Active-Active (Load Balancing), Active-Standby
○ Some additional features
21
Logstash Transport: lumberjack
● Active-Standby lumberjack { #config@source
hosts => [
“primary”,
“secondary”
]
port => 1234
ssl_certificate => …
}
primary
secondary
source
secondary is used
when primary fails
Fail
Fallback
22
Fluentd Transport: forward
● Active-Active
(Load Balancing)
<match openstack.*>
type forward
<server>
host dest1
</server>
<server>
host dest2
</server>
</match>
source dest1
dest2
Equally balanced
outputs
23
Fluentd Transport: forward
● Active-Standby <match openstack.*>
type forward
<server>
host primary
</server>
<server>
host secondary
standby
</server>
</match>
primary
secondary
source
Fail
Fallback
24
Fluentd Transport: forward
● Weighted Load Balancing
<match openstack.*>
type forward
<server>
host dest1
weight 60
</server>
<server>
host dest2
weight 40
</server>
</match>
source dest1
dest2
60%
40%
25
Fluentd Transport: forward
● At-least-one Semantics
(may affect performance)
<match openstack.*>
type forward
require_ack_response
<server>
host dest
</server>
</match>
destsource
send logs
ACK
Logs are re-transmitted
until ACK is received
26
Transport Protocol
● Both can be configured as Active-Standby mode.
● Fluentd has great features:
○ Active-Active Mode (Load Balancing)
○ At-least-one Semantics
○ Weighted Load Balancing
27
Forwarders
● Fluentd/Logstash have their own “forwarders”
○ Lightweight implementation written in Golang
○ Low memory consumption
○ One binary: Less dependent and easy to install
28
Node
Tail log files
Forwarder
Log AggregatorForward/
Lumberjack
Protocol
Forwarders: Config Example
fluentd-forwarder:
[fluentd-forwarder]
to = fluent://fluentd1:24224
to = fluent://fluentd2:24224
logstash-forwarder:
"network": {
"servers": [
"logstash1:5043",
"logstash2:5043"
]
}Always send logs to both servers.
Pick one active server and send logs only to it.
Fallback to another server on failure. 29
Integration with OpenStack
● Tail log files by local Fluentd/Logstash
○ must parse many form of log files
● Rsyslog
○ installed by default in most distribution
○ can receive logs in JSON format
● Direct output from oslo_log
○ oslo_log: logging library used by components
○ Logging without any parsing 30
Log
Aggregators
OpenStack nodes
Tail Log Files
31
Tail log files
Forward Protocol
dest1
dest2
Tail Log Files
• Must handle many log files…
syslog
kern.log
apache2/access.log
apache2/error.log
keystone/keystone-all.log
keystone/keystone-manage.log
keystone/keystone.log
cinder/cinder-api.log
cinder/cinder-scheduler.log
neutron/neutron-server.log
neutron/neutron-server.log
nova/nova-api.log
nova/nova-conductor.log
nova/nova-consoleauth.log
nova/nova-manage.log
nova/nova-novncproxy.log
nova/nova-scheduler.log
mysql/error.log
mysql/mysql-slow.log
mysql.log
mysql.err
nova/nova-compute.log
nova/nova-manage.log...
32
Tail Log Files
• But you can use wildcard
Fluentd:
<source>
type tail
path /var/log/nova/*.log
tag openstack.nova
</source>
Logstash:
input {
file {
path => [“/var/log/nova/*.log”]
}
}
33
Parse Text Log
● Welcome to regular expression hell!
<source>
type tail # or syslog
path /var/log/nova/nova-api.log
format /^(?<asctime>.+) (?<process>d+) (?<loglevel>w+) (?
<objname>S+)( [(-|(?<request_id>.+?) (?<user_identity>.+))])?
((?<remote>S*) "(?<method>S+) (?<path>[^"]*) S*?" status: (?
<code>d*) len: (?<size>d*) time: (?<res_time>S)|(?<message>.
*))/
</source>
34
Log
Aggregators
OpenStack nodes
Rsyslog
35
via /dev/log
Syslog Protocol
(TCP or UDP)
rsyslog
Rsyslog: Logging.conf
● Logging Configuration in detail
● Handler: Syslog, Formatter: JSON
# /etc/{nova,cinder…}/logging.conf
[handler_syslog]
class = handlers.SysLogHandler
args = ('/dev/log', handlers.SysLogHandler.LOG_LOCAL1)
formatter = json
[formatter_json]
class = oslo_log.formatters.JSONFormatter 36
Example Output: JSONFormatter
{
"levelname": "INFO",
"funcname": "start",
"message": "Starting conductor node (version 13.0.0)",
"msg": "Starting %(topic)s node (version %(version)s)",
"asctime": "2015-09-29 18:29:57,690",
"relative_created": 2454.8499584198,
"process": 25204,
"created": 1443518997.690932,
"thread": 140119466896752,
"name": "nova.service",
"process_name": "MainProcess",
"thread_name": "GreenThread-1",
...
37
Syslog Facilities
● Assignment of local0..7 Facilities for components
● Logs are tagged as like “syslog.local0” in Fluentd
● Example:
○ local0: Keystone
○ local1: Nova
○ local2: Cinder
○ local3: Neutron
○ local4: Glance
38
Rsyslog: Config@OpenStack nodes
● Active-Standby Configuration
# /etc/rsyslog.d/rsyslog.conf
user.* @@primary:5140
$ActionExecOnlyWhenPreviousIsSuspended on
&@@secondary:5140
39
Rsyslog: Config@Aggregator
Fluentd:
<source>
type syslog
port 5140
protocol_type tcp
format json
tag syslog
</source>
Logstash:
input {
syslog {
codec => json
port => 5140
}
} Listen on both TCP and UDP
Specify TCP or UDP 40
Rsyslog: Config@Aggregator
Fluentd:
<source>
type syslog
port 5140
protocol_type tcp
format json
tag syslog
</source>
Logstash:
input {
syslog {
codec => json
port => 5140
}
}
41
Log
AggregatorsOpenStack nodes
42
via FluentHandler
Forward Protocol
Direct output from oslo_log
Local Fluentd for buffering/load balancing
(Logstash also can be used)
Direct output from oslo_log
# logging.conf:
[handler_fluent]
class = fluent.handler.FluentHandler # fluent-logger
formatter = fluent
args = (’openstack.nova', 'localhost', 24224)
[formatter_fluent]
class = fluent.handler.FluentFormatter # our Blueprint
43
Format logs as Dictionary
Our BP in oslo_log: FluentFormatter
{
"hostname":"allinone-vivid",
"extra":{"project":"unknown","version":"unknown"},
"process_name":"MainProcess",
"module":"wsgi",
"message":"(4132) wsgi starting up on http://0.0.0.0:8774/",
"filename":"wsgi.py",
"name":"nova.osapi_compute.wsgi.server",
"level":"INFO",
"traceback":null,
"funcname":"server",
"time":"2015-10-15 10:09:12,255"
}
Don’t need to parse!
44
Conclusion
● Log Handling
○ Fluentd: Logs are distinguished by tag
○ Logstash: No tags. Logs are aggregated
● Transport Protocol
○ Both supports active-standby mode
○ Fluentd supports some additional features
■ Client-side load balancing (Active-Active)
■ At-least-one semantics
■ Weighted load balancing 45
Conclusion
● Integration with OpenStack
○ Tail log files: regular expression hell
○ Rsyslog: No agents are needed
○ Direct output from oslo_log w/o any parsing
○ Review is welcome for our Blueprint
(oslo_log: fluent-formatter)
46
Thank you!
Please visit our booth!
Robot Racing over WebRTC! →

More Related Content

What's hot

Using eBPF for High-Performance Networking in Cilium
Using eBPF for High-Performance Networking in CiliumUsing eBPF for High-Performance Networking in Cilium
Using eBPF for High-Performance Networking in CiliumScyllaDB
 
Boosting I/O Performance with KVM io_uring
Boosting I/O Performance with KVM io_uringBoosting I/O Performance with KVM io_uring
Boosting I/O Performance with KVM io_uringShapeBlue
 
Fluentd1.2 & Fluent Bit
Fluentd1.2 & Fluent BitFluentd1.2 & Fluent Bit
Fluentd1.2 & Fluent BitSeiya Mizuno
 
ML2/OVN アーキテクチャ概観
ML2/OVN アーキテクチャ概観ML2/OVN アーキテクチャ概観
ML2/OVN アーキテクチャ概観Yamato Tanaka
 
eBPF - Rethinking the Linux Kernel
eBPF - Rethinking the Linux KerneleBPF - Rethinking the Linux Kernel
eBPF - Rethinking the Linux KernelThomas Graf
 
Pacemakerを使いこなそう
Pacemakerを使いこなそうPacemakerを使いこなそう
Pacemakerを使いこなそうTakatoshi Matsuo
 
Writing the Container Network Interface(CNI) plugin in golang
Writing the Container Network Interface(CNI) plugin in golangWriting the Container Network Interface(CNI) plugin in golang
Writing the Container Network Interface(CNI) plugin in golangHungWei Chiu
 
Common issues with Apache Kafka® Producer
Common issues with Apache Kafka® ProducerCommon issues with Apache Kafka® Producer
Common issues with Apache Kafka® Producerconfluent
 
LinuxCon 2015 Linux Kernel Networking Walkthrough
LinuxCon 2015 Linux Kernel Networking WalkthroughLinuxCon 2015 Linux Kernel Networking Walkthrough
LinuxCon 2015 Linux Kernel Networking WalkthroughThomas Graf
 
High-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uringHigh-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uringScyllaDB
 
DevConf 2014 Kernel Networking Walkthrough
DevConf 2014   Kernel Networking WalkthroughDevConf 2014   Kernel Networking Walkthrough
DevConf 2014 Kernel Networking WalkthroughThomas Graf
 
ストリーム処理を支えるキューイングシステムの選び方
ストリーム処理を支えるキューイングシステムの選び方ストリーム処理を支えるキューイングシステムの選び方
ストリーム処理を支えるキューイングシステムの選び方Yoshiyasu SAEKI
 
Cephのベンチマークをしました
CephのベンチマークをしましたCephのベンチマークをしました
CephのベンチマークをしましたOSSラボ株式会社
 
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptx
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptxGrafana Mimir and VictoriaMetrics_ Performance Tests.pptx
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptxRomanKhavronenko
 
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理NTT DATA Technology & Innovation
 
AvailabilityZoneとHostAggregate
AvailabilityZoneとHostAggregateAvailabilityZoneとHostAggregate
AvailabilityZoneとHostAggregateHiroki Ishikawa
 
DNSキャッシュサーバ チューニングの勘所
DNSキャッシュサーバ チューニングの勘所DNSキャッシュサーバ チューニングの勘所
DNSキャッシュサーバ チューニングの勘所hdais
 
BPF: Tracing and more
BPF: Tracing and moreBPF: Tracing and more
BPF: Tracing and moreBrendan Gregg
 

What's hot (20)

Using eBPF for High-Performance Networking in Cilium
Using eBPF for High-Performance Networking in CiliumUsing eBPF for High-Performance Networking in Cilium
Using eBPF for High-Performance Networking in Cilium
 
Boosting I/O Performance with KVM io_uring
Boosting I/O Performance with KVM io_uringBoosting I/O Performance with KVM io_uring
Boosting I/O Performance with KVM io_uring
 
Fluentd1.2 & Fluent Bit
Fluentd1.2 & Fluent BitFluentd1.2 & Fluent Bit
Fluentd1.2 & Fluent Bit
 
ML2/OVN アーキテクチャ概観
ML2/OVN アーキテクチャ概観ML2/OVN アーキテクチャ概観
ML2/OVN アーキテクチャ概観
 
eBPF - Rethinking the Linux Kernel
eBPF - Rethinking the Linux KerneleBPF - Rethinking the Linux Kernel
eBPF - Rethinking the Linux Kernel
 
Pacemakerを使いこなそう
Pacemakerを使いこなそうPacemakerを使いこなそう
Pacemakerを使いこなそう
 
Writing the Container Network Interface(CNI) plugin in golang
Writing the Container Network Interface(CNI) plugin in golangWriting the Container Network Interface(CNI) plugin in golang
Writing the Container Network Interface(CNI) plugin in golang
 
Common issues with Apache Kafka® Producer
Common issues with Apache Kafka® ProducerCommon issues with Apache Kafka® Producer
Common issues with Apache Kafka® Producer
 
LinuxCon 2015 Linux Kernel Networking Walkthrough
LinuxCon 2015 Linux Kernel Networking WalkthroughLinuxCon 2015 Linux Kernel Networking Walkthrough
LinuxCon 2015 Linux Kernel Networking Walkthrough
 
High-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uringHigh-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uring
 
DevConf 2014 Kernel Networking Walkthrough
DevConf 2014   Kernel Networking WalkthroughDevConf 2014   Kernel Networking Walkthrough
DevConf 2014 Kernel Networking Walkthrough
 
Airflow Intro-1.pdf
Airflow Intro-1.pdfAirflow Intro-1.pdf
Airflow Intro-1.pdf
 
ストリーム処理を支えるキューイングシステムの選び方
ストリーム処理を支えるキューイングシステムの選び方ストリーム処理を支えるキューイングシステムの選び方
ストリーム処理を支えるキューイングシステムの選び方
 
Cephのベンチマークをしました
CephのベンチマークをしましたCephのベンチマークをしました
Cephのベンチマークをしました
 
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptx
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptxGrafana Mimir and VictoriaMetrics_ Performance Tests.pptx
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptx
 
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
 
AvailabilityZoneとHostAggregate
AvailabilityZoneとHostAggregateAvailabilityZoneとHostAggregate
AvailabilityZoneとHostAggregate
 
DNSキャッシュサーバ チューニングの勘所
DNSキャッシュサーバ チューニングの勘所DNSキャッシュサーバ チューニングの勘所
DNSキャッシュサーバ チューニングの勘所
 
BPF: Tracing and more
BPF: Tracing and moreBPF: Tracing and more
BPF: Tracing and more
 
gRPC Overview
gRPC OverviewgRPC Overview
gRPC Overview
 

Viewers also liked

BI, Reporting and Analytics on Apache Cassandra
BI, Reporting and Analytics on Apache CassandraBI, Reporting and Analytics on Apache Cassandra
BI, Reporting and Analytics on Apache CassandraVictor Coustenoble
 
AdStage: Monacella: An Relational Object Database using Cassandra as the Data...
AdStage: Monacella: An Relational Object Database using Cassandra as the Data...AdStage: Monacella: An Relational Object Database using Cassandra as the Data...
AdStage: Monacella: An Relational Object Database using Cassandra as the Data...DataStax Academy
 
From Zero to Hero - Centralized Logging with Logstash & Elasticsearch
From Zero to Hero - Centralized Logging with Logstash & ElasticsearchFrom Zero to Hero - Centralized Logging with Logstash & Elasticsearch
From Zero to Hero - Centralized Logging with Logstash & ElasticsearchSematext Group, Inc.
 
Integrando Redis en aplicaciones Symfony2
Integrando Redis en aplicaciones Symfony2Integrando Redis en aplicaciones Symfony2
Integrando Redis en aplicaciones Symfony2Ronny López
 
Fluentd and PHP
Fluentd and PHPFluentd and PHP
Fluentd and PHPchobi e
 
Application Logging With The ELK Stack
Application Logging With The ELK StackApplication Logging With The ELK Stack
Application Logging With The ELK Stackbenwaine
 
Fluentd and docker monitoring
Fluentd and docker monitoringFluentd and docker monitoring
Fluentd and docker monitoringVinay Krishna
 
Application Logging With Logstash
Application Logging With LogstashApplication Logging With Logstash
Application Logging With Logstashbenwaine
 
Logging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & KibanaLogging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & KibanaAmazee Labs
 

Viewers also liked (9)

BI, Reporting and Analytics on Apache Cassandra
BI, Reporting and Analytics on Apache CassandraBI, Reporting and Analytics on Apache Cassandra
BI, Reporting and Analytics on Apache Cassandra
 
AdStage: Monacella: An Relational Object Database using Cassandra as the Data...
AdStage: Monacella: An Relational Object Database using Cassandra as the Data...AdStage: Monacella: An Relational Object Database using Cassandra as the Data...
AdStage: Monacella: An Relational Object Database using Cassandra as the Data...
 
From Zero to Hero - Centralized Logging with Logstash & Elasticsearch
From Zero to Hero - Centralized Logging with Logstash & ElasticsearchFrom Zero to Hero - Centralized Logging with Logstash & Elasticsearch
From Zero to Hero - Centralized Logging with Logstash & Elasticsearch
 
Integrando Redis en aplicaciones Symfony2
Integrando Redis en aplicaciones Symfony2Integrando Redis en aplicaciones Symfony2
Integrando Redis en aplicaciones Symfony2
 
Fluentd and PHP
Fluentd and PHPFluentd and PHP
Fluentd and PHP
 
Application Logging With The ELK Stack
Application Logging With The ELK StackApplication Logging With The ELK Stack
Application Logging With The ELK Stack
 
Fluentd and docker monitoring
Fluentd and docker monitoringFluentd and docker monitoring
Fluentd and docker monitoring
 
Application Logging With Logstash
Application Logging With LogstashApplication Logging With Logstash
Application Logging With Logstash
 
Logging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & KibanaLogging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & Kibana
 

Similar to Fluentd vs. Logstash for OpenStack Log Management

Building a Unified Logging Layer with Fluentd, Elasticsearch and Kibana
Building a Unified Logging Layer with Fluentd, Elasticsearch and KibanaBuilding a Unified Logging Layer with Fluentd, Elasticsearch and Kibana
Building a Unified Logging Layer with Fluentd, Elasticsearch and KibanaMushfekur Rahman
 
Experiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah WatkinsExperiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah WatkinsCeph Community
 
OpenTelemetry For Operators
OpenTelemetry For OperatorsOpenTelemetry For Operators
OpenTelemetry For OperatorsKevin Brockhoff
 
Logstash: Get to know your logs
Logstash: Get to know your logsLogstash: Get to know your logs
Logstash: Get to know your logsSmartLogic
 
Mirko Damiani - An Embedded soft real time distributed system in Go
Mirko Damiani - An Embedded soft real time distributed system in GoMirko Damiani - An Embedded soft real time distributed system in Go
Mirko Damiani - An Embedded soft real time distributed system in Golinuxlab_conf
 
[scala.by] Launching new application fast
[scala.by] Launching new application fast[scala.by] Launching new application fast
[scala.by] Launching new application fastDenis Karpenko
 
My Sql Proxy
My Sql ProxyMy Sql Proxy
My Sql ProxyLiu Lizhi
 
Turbo charge your logs
Turbo charge your logsTurbo charge your logs
Turbo charge your logsJeremy Cook
 
Monitoring.pptx
Monitoring.pptxMonitoring.pptx
Monitoring.pptxShadi Akil
 
Introduction to redis - version 2
Introduction to redis - version 2Introduction to redis - version 2
Introduction to redis - version 2Dvir Volk
 
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo..."Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...Yandex
 
MySQL for Oracle DBAs
MySQL for Oracle DBAsMySQL for Oracle DBAs
MySQL for Oracle DBAsFromDual GmbH
 
Modern Java Features
Modern Java Features Modern Java Features
Modern Java Features Florian Hopf
 
Building zero data loss pipelines with apache kafka
Building zero data loss pipelines with apache kafkaBuilding zero data loss pipelines with apache kafka
Building zero data loss pipelines with apache kafkaAvinash Ramineni
 
Logs @ OVHcloud
Logs @ OVHcloudLogs @ OVHcloud
Logs @ OVHcloudOVHcloud
 
BUD17-310: Introducing LLDB for linux on Arm and AArch64
BUD17-310: Introducing LLDB for linux on Arm and AArch64 BUD17-310: Introducing LLDB for linux on Arm and AArch64
BUD17-310: Introducing LLDB for linux on Arm and AArch64 Linaro
 
Apache Flink: Better, Faster & Uncut - Piotr Nowojski, data Artisans
Apache Flink: Better, Faster & Uncut - Piotr Nowojski, data ArtisansApache Flink: Better, Faster & Uncut - Piotr Nowojski, data Artisans
Apache Flink: Better, Faster & Uncut - Piotr Nowojski, data ArtisansEvention
 

Similar to Fluentd vs. Logstash for OpenStack Log Management (20)

Logging for Containers
Logging for ContainersLogging for Containers
Logging for Containers
 
Containers and Logging
Containers and LoggingContainers and Logging
Containers and Logging
 
Building a Unified Logging Layer with Fluentd, Elasticsearch and Kibana
Building a Unified Logging Layer with Fluentd, Elasticsearch and KibanaBuilding a Unified Logging Layer with Fluentd, Elasticsearch and Kibana
Building a Unified Logging Layer with Fluentd, Elasticsearch and Kibana
 
Experiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah WatkinsExperiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah Watkins
 
OpenTelemetry For Operators
OpenTelemetry For OperatorsOpenTelemetry For Operators
OpenTelemetry For Operators
 
Docker Logging Webinar
Docker Logging  WebinarDocker Logging  Webinar
Docker Logging Webinar
 
Logstash: Get to know your logs
Logstash: Get to know your logsLogstash: Get to know your logs
Logstash: Get to know your logs
 
Mirko Damiani - An Embedded soft real time distributed system in Go
Mirko Damiani - An Embedded soft real time distributed system in GoMirko Damiani - An Embedded soft real time distributed system in Go
Mirko Damiani - An Embedded soft real time distributed system in Go
 
[scala.by] Launching new application fast
[scala.by] Launching new application fast[scala.by] Launching new application fast
[scala.by] Launching new application fast
 
My Sql Proxy
My Sql ProxyMy Sql Proxy
My Sql Proxy
 
Turbo charge your logs
Turbo charge your logsTurbo charge your logs
Turbo charge your logs
 
Monitoring.pptx
Monitoring.pptxMonitoring.pptx
Monitoring.pptx
 
Introduction to redis - version 2
Introduction to redis - version 2Introduction to redis - version 2
Introduction to redis - version 2
 
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo..."Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...
 
MySQL for Oracle DBAs
MySQL for Oracle DBAsMySQL for Oracle DBAs
MySQL for Oracle DBAs
 
Modern Java Features
Modern Java Features Modern Java Features
Modern Java Features
 
Building zero data loss pipelines with apache kafka
Building zero data loss pipelines with apache kafkaBuilding zero data loss pipelines with apache kafka
Building zero data loss pipelines with apache kafka
 
Logs @ OVHcloud
Logs @ OVHcloudLogs @ OVHcloud
Logs @ OVHcloud
 
BUD17-310: Introducing LLDB for linux on Arm and AArch64
BUD17-310: Introducing LLDB for linux on Arm and AArch64 BUD17-310: Introducing LLDB for linux on Arm and AArch64
BUD17-310: Introducing LLDB for linux on Arm and AArch64
 
Apache Flink: Better, Faster & Uncut - Piotr Nowojski, data Artisans
Apache Flink: Better, Faster & Uncut - Piotr Nowojski, data ArtisansApache Flink: Better, Faster & Uncut - Piotr Nowojski, data Artisans
Apache Flink: Better, Faster & Uncut - Piotr Nowojski, data Artisans
 

More from NTT Communications Technology Development

クラウドを最大限活用するinfrastructure as codeを考えよう
クラウドを最大限活用するinfrastructure as codeを考えようクラウドを最大限活用するinfrastructure as codeを考えよう
クラウドを最大限活用するinfrastructure as codeを考えようNTT Communications Technology Development
 
【たぶん日本初導入!】Azure Stack Hub with GPUの性能と機能紹介
【たぶん日本初導入!】Azure Stack Hub with GPUの性能と機能紹介【たぶん日本初導入!】Azure Stack Hub with GPUの性能と機能紹介
【たぶん日本初導入!】Azure Stack Hub with GPUの性能と機能紹介NTT Communications Technology Development
 
macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~
macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~
macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~NTT Communications Technology Development
 
マルチクラウドでContinuous Deliveryを実現するSpinnakerについて
マルチクラウドでContinuous Deliveryを実現するSpinnakerについて マルチクラウドでContinuous Deliveryを実現するSpinnakerについて
マルチクラウドでContinuous Deliveryを実現するSpinnakerについて NTT Communications Technology Development
 
Can we boost more HPC performance? Integrate IBM POWER servers with GPUs to O...
Can we boost more HPC performance? Integrate IBM POWER servers with GPUs to O...Can we boost more HPC performance? Integrate IBM POWER servers with GPUs to O...
Can we boost more HPC performance? Integrate IBM POWER servers with GPUs to O...NTT Communications Technology Development
 
イケてない開発チームがイケてる開発を始めようとする軌跡
イケてない開発チームがイケてる開発を始めようとする軌跡イケてない開発チームがイケてる開発を始めようとする軌跡
イケてない開発チームがイケてる開発を始めようとする軌跡NTT Communications Technology Development
 

More from NTT Communications Technology Development (20)

クラウドを最大限活用するinfrastructure as codeを考えよう
クラウドを最大限活用するinfrastructure as codeを考えようクラウドを最大限活用するinfrastructure as codeを考えよう
クラウドを最大限活用するinfrastructure as codeを考えよう
 
【たぶん日本初導入!】Azure Stack Hub with GPUの性能と機能紹介
【たぶん日本初導入!】Azure Stack Hub with GPUの性能と機能紹介【たぶん日本初導入!】Azure Stack Hub with GPUの性能と機能紹介
【たぶん日本初導入!】Azure Stack Hub with GPUの性能と機能紹介
 
macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~
macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~
macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~
 
マルチクラウドでContinuous Deliveryを実現するSpinnakerについて
マルチクラウドでContinuous Deliveryを実現するSpinnakerについて マルチクラウドでContinuous Deliveryを実現するSpinnakerについて
マルチクラウドでContinuous Deliveryを実現するSpinnakerについて
 
Argo CDについて
Argo CDについてArgo CDについて
Argo CDについて
 
SpinnakerとKayentaで 高速・安全なデプロイ!
SpinnakerとKayentaで 高速・安全なデプロイ!SpinnakerとKayentaで 高速・安全なデプロイ!
SpinnakerとKayentaで 高速・安全なデプロイ!
 
100Gbps OpenStack For Providing High-Performance NFV
100Gbps OpenStack For Providing High-Performance NFV100Gbps OpenStack For Providing High-Performance NFV
100Gbps OpenStack For Providing High-Performance NFV
 
Can we boost more HPC performance? Integrate IBM POWER servers with GPUs to O...
Can we boost more HPC performance? Integrate IBM POWER servers with GPUs to O...Can we boost more HPC performance? Integrate IBM POWER servers with GPUs to O...
Can we boost more HPC performance? Integrate IBM POWER servers with GPUs to O...
 
AWS re:Invent2017で見た AWSの強さとは
AWS re:Invent2017で見た AWSの強さとは AWS re:Invent2017で見た AWSの強さとは
AWS re:Invent2017で見た AWSの強さとは
 
分散トレーシング技術について(Open tracingやjaeger)
分散トレーシング技術について(Open tracingやjaeger)分散トレーシング技術について(Open tracingやjaeger)
分散トレーシング技術について(Open tracingやjaeger)
 
Mexico ops meetup発表資料 20170905
Mexico ops meetup発表資料 20170905Mexico ops meetup発表資料 20170905
Mexico ops meetup発表資料 20170905
 
NTT Tech Conference #2 - closing -
NTT Tech Conference #2 - closing -NTT Tech Conference #2 - closing -
NTT Tech Conference #2 - closing -
 
イケてない開発チームがイケてる開発を始めようとする軌跡
イケてない開発チームがイケてる開発を始めようとする軌跡イケてない開発チームがイケてる開発を始めようとする軌跡
イケてない開発チームがイケてる開発を始めようとする軌跡
 
GPU Container as a Service を実現するための最新OSS徹底比較
GPU Container as a Service を実現するための最新OSS徹底比較GPU Container as a Service を実現するための最新OSS徹底比較
GPU Container as a Service を実現するための最新OSS徹底比較
 
SpinnakerとOpenStackの構築
SpinnakerとOpenStackの構築SpinnakerとOpenStackの構築
SpinnakerとOpenStackの構築
 
Troveコミュニティ動向
Troveコミュニティ動向Troveコミュニティ動向
Troveコミュニティ動向
 
Web rtc for iot, edge computing use cases
Web rtc for iot, edge computing use casesWeb rtc for iot, edge computing use cases
Web rtc for iot, edge computing use cases
 
OpenStack Ops Mid-Cycle Meetup & Project Team Gathering出張報告
OpenStack Ops Mid-Cycle Meetup & Project Team Gathering出張報告OpenStack Ops Mid-Cycle Meetup & Project Team Gathering出張報告
OpenStack Ops Mid-Cycle Meetup & Project Team Gathering出張報告
 
NTT Tech Conference #1 Opening Keynote
NTT Tech Conference #1 Opening KeynoteNTT Tech Conference #1 Opening Keynote
NTT Tech Conference #1 Opening Keynote
 
NTT Tech Conference #1 Closing Keynote
NTT Tech Conference #1 Closing KeynoteNTT Tech Conference #1 Closing Keynote
NTT Tech Conference #1 Closing Keynote
 

Recently uploaded

Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesSoftwareMill
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfBrain Inventory
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?AmeliaSmith90
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.Sharon Liu
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Incrobinwilliams8624
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyRaymond Okyere-Forson
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 

Recently uploaded (20)

Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retries
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdf
 
Salesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptxSalesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptx
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Inc
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human Beauty
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 

Fluentd vs. Logstash for OpenStack Log Management

  • 1. Fluentd vs. Logstash Masaki Matsushita NTT Communications
  • 2. About Me ● Masaki MATSUSHITA ● Software Engineer at ○ We are providing Internet access here! ● Github: mmasaki Twitter: @_mmasaki ● 16 Commits in Liberty ○ Trove, oslo_log, oslo_config ● CRuby Commiter ○ 100+ commits for performance improvement 2
  • 3. What are Log Collectors? ● Provide pluggable and unified logging layer Without Log Collectors With Log Collectors Images from http://fluentd.org/ 3
  • 4. Input, Filter and Output 4 Input Plugins tail syslog Filter Plugins grep hostname Output Plugins InfluxDB Elasticsearch ● They are implemented as plugins ● Can be replaced easily Log FIles Components
  • 5. Two Popular Log Collectors ● Fluentd ○ Written in CRuby ○ Used in Kubernetes ○ Maintained by Treasure Data Inc. ● Logstash ○ Written in JRuby ○ Maintained by elastic.co ● They have similar features ● Which one is better for you? 5
  • 6. Agenda ● Comparisons ○ Configuration ○ Supported Plugins ○ Performance ○ Transport Protocol ● Integrate OpenStack with Fluentd/Logstash ○ Considering High Availability 6
  • 7. Configuration: Fluentd ● Every inputs are tagged ● Logs will be routed by tag nova-api.log (tag: openstack.nova) cinder-api.log (tag: openstack.cinder) <match openstack.nova> <match openstack.cinder> Filter/Route 7
  • 8. Fluentd Configuration: Input <source> @type tail path /var/log/nova/nova-api.log tag openstack.nova </source> Example of tailing nova-api log ● Every inputs will be tagged 8
  • 9. Fluentd Configuration: Output <match openstack.nova> # nova related logs @type elasticsearch host example.com </match> <match openstack.*> # all other OpenStack related logs @type influxdb # … </match> Routed by tag (First match is priority) Wildcards can be used 9
  • 10. Fluentd Configuration: Copy <match openstack.*> @type copy <store> @type influxdb </store> <store> @type elasticsearch </store> </match> Copy plugin enables multiple outputs for a tag Copied Output tag: openstack.* 10
  • 11. Logstash Configuration ● No tags ● All inputs will be aggregated ● Logs will be scattered to outputs nova-api.log cinder-api.log Filter/Aggregate aggregated logs 11
  • 12. Logstash Configuration input { file { path => “/var/log/nova/*.log” } file { path => “/var/log/cinder/*.log” } } output { elasticsearch { hosts => [“example.com”] } influxdb { host => “example.com”... } } 12
  • 13. Case 1: Separated Streams Input1 Input2 Input3 Output2 Output3 Output1 ● Handle multiple streams separately 13
  • 14. Case 1: Separated Streams Fluentd: Simple matching by tag <match input.input1> @type output1 </match> <match input.input2> @type output2 </match> <match input.input3> @type output3 </match> Logstash: Conditional Outputs output { if [type] == “input1” { output1 {} } else if [type] == “input2” { output2 {} } else if [type] == “input3” { output3 {} } } Need to split aggregated logs 14
  • 15. Case 2: Aggregated Streams Input1 Input2 Input3 Output2 Output3 Output1 ● Streams will be aggregated and scattered 15
  • 16. Case 2: Aggregated Streams Fluentd: Copy plugins is needed <match input.*> @type copy <store> @type output1 </store> <store> @type output2 </store> <store> @type output3 </store> </match> Logstash: Quite simple output { output1 {} output2 {} output3 {} } 16
  • 17. Configuration ● Fluentd ○ Routed by simple tag matching ○ Suited to handle log streams separately ● Logstash ○ Logs are aggregated ○ Suited to handle logs in gather-scatter style 17
  • 18. Plugins ● Both provide many plugins ○ Fluentd: 300+, Logstash: 200+ ● Popular plugins are bundled with Logstash ○ They are maintained by the Logstash project ● Fluentd contains only minimal plugins ○ Most plugins are maintained by individuals ● Plugins can be installed easily by one command 18
  • 19. Performance ● Depends on circumstances ● More than enough for OpenStack logs ○ Both can handle 10000+ logs/s ● Applying heavy filters is not a good idea ● CRuby is slow because of GVL? ○ GVL: Global VM (Interpreter) Lock ○ It’s not true for IO bound loads 19
  • 20. GVL on IO bound loads ● IO operation can be performed in parallel 20 Thread 1 Thread 2 Idle : User Space: Kernel Space: Actual Read/Write Ruby Code Execution GVL Released/ Acquired IO operations in parallel
  • 21. Transport Protocol ● Both collectors have their own transport protocol. ○ Failure Detection and Fallback ● Logstash: Lumberjack protocol ○ Active-Standby only ● Fluentd: forward protocol ○ Active-Active (Load Balancing), Active-Standby ○ Some additional features 21
  • 22. Logstash Transport: lumberjack ● Active-Standby lumberjack { #config@source hosts => [ “primary”, “secondary” ] port => 1234 ssl_certificate => … } primary secondary source secondary is used when primary fails Fail Fallback 22
  • 23. Fluentd Transport: forward ● Active-Active (Load Balancing) <match openstack.*> type forward <server> host dest1 </server> <server> host dest2 </server> </match> source dest1 dest2 Equally balanced outputs 23
  • 24. Fluentd Transport: forward ● Active-Standby <match openstack.*> type forward <server> host primary </server> <server> host secondary standby </server> </match> primary secondary source Fail Fallback 24
  • 25. Fluentd Transport: forward ● Weighted Load Balancing <match openstack.*> type forward <server> host dest1 weight 60 </server> <server> host dest2 weight 40 </server> </match> source dest1 dest2 60% 40% 25
  • 26. Fluentd Transport: forward ● At-least-one Semantics (may affect performance) <match openstack.*> type forward require_ack_response <server> host dest </server> </match> destsource send logs ACK Logs are re-transmitted until ACK is received 26
  • 27. Transport Protocol ● Both can be configured as Active-Standby mode. ● Fluentd has great features: ○ Active-Active Mode (Load Balancing) ○ At-least-one Semantics ○ Weighted Load Balancing 27
  • 28. Forwarders ● Fluentd/Logstash have their own “forwarders” ○ Lightweight implementation written in Golang ○ Low memory consumption ○ One binary: Less dependent and easy to install 28 Node Tail log files Forwarder Log AggregatorForward/ Lumberjack Protocol
  • 29. Forwarders: Config Example fluentd-forwarder: [fluentd-forwarder] to = fluent://fluentd1:24224 to = fluent://fluentd2:24224 logstash-forwarder: "network": { "servers": [ "logstash1:5043", "logstash2:5043" ] }Always send logs to both servers. Pick one active server and send logs only to it. Fallback to another server on failure. 29
  • 30. Integration with OpenStack ● Tail log files by local Fluentd/Logstash ○ must parse many form of log files ● Rsyslog ○ installed by default in most distribution ○ can receive logs in JSON format ● Direct output from oslo_log ○ oslo_log: logging library used by components ○ Logging without any parsing 30
  • 31. Log Aggregators OpenStack nodes Tail Log Files 31 Tail log files Forward Protocol dest1 dest2
  • 32. Tail Log Files • Must handle many log files… syslog kern.log apache2/access.log apache2/error.log keystone/keystone-all.log keystone/keystone-manage.log keystone/keystone.log cinder/cinder-api.log cinder/cinder-scheduler.log neutron/neutron-server.log neutron/neutron-server.log nova/nova-api.log nova/nova-conductor.log nova/nova-consoleauth.log nova/nova-manage.log nova/nova-novncproxy.log nova/nova-scheduler.log mysql/error.log mysql/mysql-slow.log mysql.log mysql.err nova/nova-compute.log nova/nova-manage.log... 32
  • 33. Tail Log Files • But you can use wildcard Fluentd: <source> type tail path /var/log/nova/*.log tag openstack.nova </source> Logstash: input { file { path => [“/var/log/nova/*.log”] } } 33
  • 34. Parse Text Log ● Welcome to regular expression hell! <source> type tail # or syslog path /var/log/nova/nova-api.log format /^(?<asctime>.+) (?<process>d+) (?<loglevel>w+) (? <objname>S+)( [(-|(?<request_id>.+?) (?<user_identity>.+))])? ((?<remote>S*) "(?<method>S+) (?<path>[^"]*) S*?" status: (? <code>d*) len: (?<size>d*) time: (?<res_time>S)|(?<message>. *))/ </source> 34
  • 36. Rsyslog: Logging.conf ● Logging Configuration in detail ● Handler: Syslog, Formatter: JSON # /etc/{nova,cinder…}/logging.conf [handler_syslog] class = handlers.SysLogHandler args = ('/dev/log', handlers.SysLogHandler.LOG_LOCAL1) formatter = json [formatter_json] class = oslo_log.formatters.JSONFormatter 36
  • 37. Example Output: JSONFormatter { "levelname": "INFO", "funcname": "start", "message": "Starting conductor node (version 13.0.0)", "msg": "Starting %(topic)s node (version %(version)s)", "asctime": "2015-09-29 18:29:57,690", "relative_created": 2454.8499584198, "process": 25204, "created": 1443518997.690932, "thread": 140119466896752, "name": "nova.service", "process_name": "MainProcess", "thread_name": "GreenThread-1", ... 37
  • 38. Syslog Facilities ● Assignment of local0..7 Facilities for components ● Logs are tagged as like “syslog.local0” in Fluentd ● Example: ○ local0: Keystone ○ local1: Nova ○ local2: Cinder ○ local3: Neutron ○ local4: Glance 38
  • 39. Rsyslog: Config@OpenStack nodes ● Active-Standby Configuration # /etc/rsyslog.d/rsyslog.conf user.* @@primary:5140 $ActionExecOnlyWhenPreviousIsSuspended on &@@secondary:5140 39
  • 40. Rsyslog: Config@Aggregator Fluentd: <source> type syslog port 5140 protocol_type tcp format json tag syslog </source> Logstash: input { syslog { codec => json port => 5140 } } Listen on both TCP and UDP Specify TCP or UDP 40
  • 41. Rsyslog: Config@Aggregator Fluentd: <source> type syslog port 5140 protocol_type tcp format json tag syslog </source> Logstash: input { syslog { codec => json port => 5140 } } 41
  • 42. Log AggregatorsOpenStack nodes 42 via FluentHandler Forward Protocol Direct output from oslo_log Local Fluentd for buffering/load balancing (Logstash also can be used)
  • 43. Direct output from oslo_log # logging.conf: [handler_fluent] class = fluent.handler.FluentHandler # fluent-logger formatter = fluent args = (’openstack.nova', 'localhost', 24224) [formatter_fluent] class = fluent.handler.FluentFormatter # our Blueprint 43 Format logs as Dictionary
  • 44. Our BP in oslo_log: FluentFormatter { "hostname":"allinone-vivid", "extra":{"project":"unknown","version":"unknown"}, "process_name":"MainProcess", "module":"wsgi", "message":"(4132) wsgi starting up on http://0.0.0.0:8774/", "filename":"wsgi.py", "name":"nova.osapi_compute.wsgi.server", "level":"INFO", "traceback":null, "funcname":"server", "time":"2015-10-15 10:09:12,255" } Don’t need to parse! 44
  • 45. Conclusion ● Log Handling ○ Fluentd: Logs are distinguished by tag ○ Logstash: No tags. Logs are aggregated ● Transport Protocol ○ Both supports active-standby mode ○ Fluentd supports some additional features ■ Client-side load balancing (Active-Active) ■ At-least-one semantics ■ Weighted load balancing 45
  • 46. Conclusion ● Integration with OpenStack ○ Tail log files: regular expression hell ○ Rsyslog: No agents are needed ○ Direct output from oslo_log w/o any parsing ○ Review is welcome for our Blueprint (oslo_log: fluent-formatter) 46
  • 47. Thank you! Please visit our booth! Robot Racing over WebRTC! →