log parser for clusters
- Remote access to logs via a single CLI
- Multi-host summaries of log line frequencies
- Multi-host realtime tailing (like tail -f)
- Syslog
- Log4j
- Java GC log
For details please see LOGLINE_RE_LIST in hblog/lib/SingleFileLogAccessor.py
Usage: hblog [OPTIONS]... [TIER...] [TIER:HOST...]
Where TIER is one of:
dfs-nn hbase-master-gc mr-jt
dfs-nn-gc hbase-regionservers mr-jt-gc
dfs-slaves hbase-regionservers-gc mr-slaves
dfs-slaves-gc hbase-secondary mr-slaves-gc
dfs-sn hbase-secondary-gc syslog
dfs-sn-gc hbase-thrift
hbase-master hbase-zookeepers
hblog - a log paser for clusters
Options:
-h, --help show this help message and exit
-v, --verbose print extra information about the state of hblog
-n, --nowrap print characters only up to the width of your terminal
Modes:
Log lines are "fingerprinted", usually able to assign matching
fingerprints to log lines that differ only by timestamp, specific host
names, or other variables.
--summary host-vs-fingerprint frequency table (Default mode)
-d, --details print all matching log lines embellished with
hostnames and fingerprints
-f, --follow like --details but streaming, just like 'tail -f'
Select time:
If time selectors are not supplied, only the last one minute of logs
will be processed.
-s START, --start=START
process only lines after the time specified
in format YYYY-MM-DD hh:mm:ss
-e END, --end=END process only lines up to the time specified
in format YYYY-MM-DD hh:mm:ss
-t TAIL, --tail=TAIL
process only the last X minutes of each log specified
as one of these formats ":sec", "min", "hour:min"
-T TAIL_END, --tail-end=TAIL_END
process only up to the last X minutes of each log
specified as one of these formats ":sec", "min",
"hour:min"
Filters:
-l LEVEL, --level=LEVEL
the log level to filter for (default level: WARN)
-S SAMPLE, --sample=SAMPLE
sampling rate will be achieved by skipping log lines
(default: 1.0, read all lines)
-p FP, --fp=FP comma-separated list of fingerprints to include
-P FP_EXCLUDE, --fp-exclude=FP_EXCLUDE
comma-separated list of fingerprints to exclude
-r RE, --re=RE comma-separated list of regex to include (case
insensitive)
-R RE_EXCLUDE, --re-exclude=RE_EXCLUDE
comma-separated list of regex to exclude (case
insensitive)
--local To test hblog. Connect to localhost. Read logs from
./var/log/hadoop-example.log
$ hblog --level=INFO mycluster001-hbase-regionservers
---------------------------------------------------------------
---------------------------- To make these setting default run:
---------------------------------------------------------------
cat <<'EOF' > $HOME/.hblogrc
{
"fp": "",
"fp-exclude": "",
"level": "WARN",
"log-tiers": [
"hbase-regionservers"
],
"mode": "summary",
"nowrap": true,
"re": "",
"re-exclude": "^\t",
"sample": 1.0,
"tail": null,
"tail-end": null,
"verbose": false
}
EOF
---------------------------------------------------------------
log-tiers: hbase-regionservers
log-tiers-globs: /var/log/hadoop/*-HBASE/hbase-hadoop-regionserver-[!g][!c]*
hosts-list-size: 115
type: summary
start: 2013-10-17 12:34:22
end: 2013-10-17 12:35:22
duration: 0:01:00 hh:mm:ss
level: WARN
sample: 1.0
fp: []
fp-exclude: []
re: []
re-exclude: ['^\t']
---------------------------------------------------------------
WARNING:root:Connect error on fd 10: ECONNREFUSED
WARN: HTTP error from hbase999, blacklisting host. Error was HTTP 599: Connection closed.
WARNING:root:Connect error on fd 11: ECONNREFUSED
WARN: HTTP error from hbase998, blacklisting host. Error was HTTP 599: Connection closed.
---------------------------------------------------------------
Fingerprint summary:
count fingerprint level text
2898 822edf9 WARN org.apache.hadoop.hbase.regionserver.Store: Not in set org.apache.hadoop.hbase.regionserver.StoreScanner@#
392 aae50d9 WARN org.apache.hadoop.hdfs.DFSClient: Null blocks retrieved for : /## offset : #, len : #
290 d2c8224 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler # on # took # ms appending an edit to hlog;
editcount=#, len~=#.# KB
90 4fcd95b WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler # on # took # ms appending an edit to hlog;
editcount=#, len~=#.#
3 a0e71c9 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer:
1 e310f00 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler # on # took # ms appending an edit to hlog;
editcount=#, len~=#.# MB
1 b73e9eb WARN org.apache.hadoop.hbase.ipc.HBaseServer: IPC Server handler # on #, call get( ... ) from <<IP>>:#: error: java
.io.IOEx#ption: Could not seek StoreFileScanner[HFileScanner for reader reader=hdfs:/##<<HOST>>:#/## compressi
on=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndex#sOnWrite=fal
se] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=#:#@<<HOST>>/## las
tKey=#:#@<<HOST>>/##<<MID>>\x#\x#/## avgKeyLen=#, avgValueLen=#, entries=#, length=#, cur=null] to key #:#@<<H
OST>>/##=#
1 333fb44 WARN org.apache.hadoop.hbase.ipc.HBaseServer: IPC Server handler # on #, call get( ... ) from <<IP>>:#: error: java
.io.IOEx#ption: Could not seek StoreFileScanner[HFileScanner for reader reader=hdfs:/##<<HOST>>:#/## compressi
on=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndex#sOnWrite=fal
se] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=#:#@<<HOST>>/## las
tKey=#:#@<<HOST>>/## avgKeyLen=#, avgValueLen=#, entries=#, length=#, cur=null] to key #:#@<<HOST>>/##=#
1 03ebf69 WARN org.apache.hadoop.hbase.ipc.HBaseServer: IPC Server handler # on #, call get( ... ) from <<IP>>:#: error: java
.io.IOEx#ption: Could not seek StoreFileScanner[HFileScanner for reader reader=hdfs:/##<<HOST>>:#/## compressi
on=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndex#sOnWrite=fal
se] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=#:#@<<HOST>>/##<<MI
D>><<MID>>\x#\x#/## lastKey=#:#@<<HOST>>/##<<MID>>\x#\x#/## avgKeyLen=#, avgValueLen=#, entries=#, length=#, c
ur=null] to key #:#@<<HOST>>/##=#
---------------------------------------------------------------
Host sumary:
822edf9 aae50d9 d2c8224 4fcd95b a0e71c9 e310f00 b73e9eb 333fb44 03ebf69
host001 35
host002 40
host003 16 16
host004 58
host005 25 9
host006 15
host007 14
host008 49 20
host009 47
host010 42
host011 13
host012 11 21
host013 29
host014 65 1 1
host015 25
host016 14 7
host017 14
host018 36 13
host019 7
host020 71
host021 43
host022 24
host023 12 11
host024 40 4
host025 22
host026 39
host027 50
host028 20 8
host029 23
host030 25
host031 27
host032 24
host033 23
host034 4
host035 21
host036 13 14
host037 98 233 71 1
host038 68
host039 21 1
host040 42 21
host041 27
host042 22
host043 33
host044 23
host045 4
host046 16 15
host047 73
host048 14
host049 12
host050 41
host051 29 8
host052 26
host053 52
host054 11
host055 13
host056 27
host057 24
host058 12
host059 80 1
host060 9
host061 35
host062 33 18
host063 42 6
host064 29
host065 25
host066 14
host067 42
host068 23
host069 1 9
host070 26
host071 16
host072 50 1 1
host073 4
host074 25
host075 28
host076 19
host077 23
host078 47 5
host079 31
host080 3 20
host081 17 19
host082 24 5
host083 33 1 1
host084 29
host085 12
host086 26 13
host087 61
host088 18
host089 12
host090 37
host091 38
host092 11 2
host093 14 7 57 19
host094 14
host095 69
host096 1 19
host097 16
host098 51 12
host099 21
host100 12
host101 31 11
host102 49
host103 18 19
host104 13
---------------------------------------------------------------
2013-10-17 12:35:26.776816 BLACKL01 WARN - Blacklisted 2 hosts (of 115 in this session): set(['host998', 'host999']) - consider adding filters or lowering the sampling rate
This project is licensed under the Apache 2.0 license
- tornado 3.0.1 (https://github.com/facebook/tornado/tree/v3.0.1)
git clone https://github.com/facebook/hblog.git
cd hblog
git submodule init tornado/
git submodule update tornado/
cd hblog
./sbin/hblogd.py # in a separate tab or in screen/tmux
export PATH="$PATH:$(pwd)/bin" # for list_hosts_of_tier.sh
./bin/hblog.py --local --start '2011-03-27 12:48:18' nn
./bin/hblog.py --local --start '2011-03-27 12:48:18' nn-gc
./bin/hblog.py --local --start '2011-03-27 12:48:18' syslog