一、概述
Elasticsearch是个开源分布式搜索引擎,它的特点有:分布式,零配置,自动发现,索引自动分片,索引副本机制,restful风格接口,多数据源,自动搜索负载等。
Logstash是一个开源的用于收集,分析和存储日志的工具。
Kibana 也是一个开源和免费的工具,Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面,可以汇总、分析和搜索重要数据日志。
Beats是elasticsearch公司开源的一款采集系统监控数据的代理agent,是在被监控服务器上以客户端形式运行的数据收集器的统称,可以直接把数据发送给Elasticsearch或者通过Logstash发送给Elasticsearch,然后进行后续的数据分析活动。Beats由如下组成:
1.Packetbeat:是一个网络数据包分析器,用于监控、收集网络流量信息,
Packetbeat嗅探服务器之间的流量,解析应用层协议,并关联到消息的处理, 其支 持ICMP (v4 and v6)、DNS、HTTP、Mysql、PostgreSQL、Redis、
MongoDB、Memcache等协议;
2. Filebeat:用于监控、收集服务器日志文件,其已取代 logstash forwarder;
3. Metricbeat:可定期获取外部系统的监控指标信息,其可以监控、收集
Apache、HAProxy、MongoDB、MySQL、Nginx、PostgreSQL、
Redis、System、Zookeeper等服务;
4. Winlogbeat:用于监控、收集Windows系统的日志信息;
5. Create your own Beat:自定义beat ,如果上面的指标不能满足需求,elasticsarch鼓励开发者 使用go语言,扩展实现自定义的beats,只需要按照模板,实现监控的输入,日志,输出等即可。
Beats 将搜集到的数据发送到 Logstash,经 Logstash 解析、过滤后,将其发送到 Elasticsearch 存储,并由 Kibana 呈现给用户。
Beats 作为日志搜集器没有Logstash 作为日志搜集器消耗资源,解决了 Logstash 在各服务器节点上占用系统资源高的问题。
Elastic Stack官方下载地址:https://www.elastic.co/downloads。
二、开源实时日志分析系统Elastic Stack 5.0部署:
A.安装依赖包JDK
①关闭防火墙
1
2
3
4
|
# cat /etc/selinux/config |grep -v "#"
SELINUX=disabled
##关闭selinux
SELINUXTYPE=targeted
# systemctl stop firewalld ##关闭防火墙
|
②下载安装JDK
1
2
3
4
5
6
|
# wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# yum install jre ##这里我只安装的是jre环境,一样可用
# java -version
openjdk version
"1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-b15)
OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode
|
B.安装Elasticsearch
①下载安装elasticsearch
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
# cat > /etc/yum.repos.d/elasticsearch.repo <<EOF
[elasticsearch-5.x]
name=Elasticsearch repository
for
5.x packages
baseurl=https:
//artifacts
.elastic.co
/packages/5
.x
/yum
gpgcheck=1
gpgkey=https:
//artifacts
.elastic.co
/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type
=rpm-md
EOF
# yum install elasticsearch
# systemctl start elasticsearch
# systemctl status elasticsearch
# /usr/share/elasticsearch/bin/elasticsearch -V ##查看elasticsearch版本
Version: 5.0.1, Build: 080bb47
/2016-11-11T22
:08:49.812Z, JVM: 1.8.0_111
|
注:安装后出现如下问题,致使elasticsearch无法启动
1
2
3
4
5
6
7
|
# systemctl status elasticsearch
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one,
then
you should conf...CThreads=N
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000085330000, 2060255232, 0) failed; error=
'Cannot a ...'
(errno=12)
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 2060255232 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid11084.log
|
解决:
配置elasticsearch下的jvm.options:
1
2
3
4
5
|
# vi /etc/elasticsearch/jvm.options
-Xms4g
##启用如下两项
-Xmx4g
##-Xms2g ##关闭如下两项
##-Xmx2g
|
②ElasticSearch默认的对外服务的HTTP端口是9200,节点间交互的TCP端口是9300。
1
|
# ss -tlnp |grep -E '9200|9300'
|
③测试服务
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
# curl -X GET http://localhost:9200
{
"name"
:
"XVY0Ovb"
,
"cluster_name"
:
"elasticsearch"
,
"cluster_uuid"
:
"tR_H9avzT6Kf4hXWTIfWyA"
,
"version"
: {
"number"
:
"5.0.1"
,
"build_hash"
:
"080bb47"
,
"build_date"
:
"2016-11-11T22:08:49.812Z"
,
"build_snapshot"
:
false
,
"lucene_version"
:
"6.2.1"
},
"tagline"
:
"You Know, for Search"
}
|
也可以使用如下命令测试
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
# curl -i -XGET 'localhost:9200/'
HTTP
/1
.1 200 OK
content-
type
: application
/json
; charset=UTF-8
content-length: 327
{
"name"
:
"XVY0Ovb"
,
"cluster_name"
:
"elasticsearch"
,
"cluster_uuid"
:
"tR_H9avzT6Kf4hXWTIfWyA"
,
"version"
: {
"number"
:
"5.0.1"
,
"build_hash"
:
"080bb47"
,
"build_date"
:
"2016-11-11T22:08:49.812Z"
,
"build_snapshot"
:
false
,
"lucene_version"
:
"6.2.1"
},
"tagline"
:
"You Know, for Search"
}
|
C.安装Logstash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
# cat > /etc/yum.repos.d/logstash.repo <<EOF
[logstash-5.x]
name=Elastic repository
for
5.x packages
baseurl=https:
//artifacts
.elastic.co
/packages/5
.x
/yum
gpgcheck=1
gpgkey=https:
//artifacts
.elastic.co
/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type
=rpm-md
EOF
# yum install logstash
# systemctl start logstash
# systemctl status logstash
# /usr/share/logstash/bin/logstash -V ##查看logstash版本
logstash 5.0.1
|
D.安装Kibana
①安装Kibana
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
# cat > /etc/yum.repos.d/kibana.repo <<EOF
[kibana-5.x]
name=Kibana repository
for
5.x packages
baseurl=https:
//artifacts
.elastic.co
/packages/5
.x
/yum
gpgcheck=1
gpgkey=https:
//artifacts
.elastic.co
/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type
=rpm-md
EOF
# yum install kibana
# systemctl start kibana
# systemctl status kibana
# /usr/share/kibana/bin/kibana -V ##查看kibana版本
5.0.1
|
②只需更改如下配置
1
2
|
# cat /etc/kibana/kibana.yml |grep -v "#"
server.host:
"192.168.147.128"
|
③检测服务
1
|
# ss -tlnp|grep 5601 ##Kibana默认进程名:node ,端口5601
|
E、Beats安装部署
a.安装部署Filebeat
①安装Filebeat
1
2
3
4
5
6
|
# curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.0.1-x86_64.rpm
# rpm -ivh filebeat-5.0.1-x86_64.rpm
# systemctl start filebeat
# systemctl status filebeat
# filebeat.sh -version ##查看版本
filebeat version 5.0.1 (amd64), libbeat 5.0.1
|
②配置Filebeat
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
# cd /etc/filebeat
# cp filebeat.yml filebeat.yml.bak
# vi /etc/filebeat/filebeat.yml ##配置filebeat
#============= Filebeat prospectors ===============
filebeat.prospectors:
- input_type: log
paths:
-
/var/log/
*.log
#==================== Outputs =====================
#------------- Elasticsearch output ---------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: [
"localhost:9200"
]
#---------------- Logstash output -----------------
output.logstash:
# The Logstash hosts
hosts: [
"localhost:5043"
]
##只需配置该处,其他默认即可
# filebeat.sh -configtest -e ##验证配置文件
|
③配置Logstash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
# cat /etc/logstash/conf.d/logstash.conf
input {
beats {
port =>
"5043"
}
}
filter {
grok {
match => {
"message"
=>
"%{COMBINEDAPACHELOG}"
}
}
geoip {
source
=>
"clientip"
}
}
output {
elasticsearch {
hosts => [
"localhost:9200"
]
}
}
# systemctl restart logstash ##重启logstash
# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit ##验证配置文件
Sending Logstash logs to
/var/log/logstash
which
is now configured via log4j2.properties
Configuration OK
# ss -tlnp|grep -E '5043|9600'
|
④配置kibana
浏览器输入http://localhost:5601,配置filebeat的索引(只需输入filebeat-*即可)。
在第一个框里输入filebeat-*后稍等片刻,kibana会自动识别,OK后下面的按钮会由灰色变为可操控的按钮"Create",如上图所示。点击该按钮后,最后就会呈现如下图所示:
我们再回过头新建logstash的索引,浏览器输入http://localhost:5601,点击左边栏的”Management” ===> 然后点击“index Patterns” ===>
===> 然后点击“Add New” ===>
===> 点击“Crete”按钮创建logstash索引,创建完成后即会展现如下图所示:
b.安装部署Packetbeat
①安装配置Packetbeat
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
|
# yum install libpcap
# curl -L -O https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-5.0.1-x86_64.rpm
# rpm -ivh packetbeat-5.0.1-x86_64.rpm
# cat /etc/packetbeat/packetbeat.yml
==================== Network device ===================
packetbeat.interfaces.device: any
##捕获所有消息发送或接收的网络接口
======================== Flows ========================
packetbeat.flows:
timeout: 30s
period: 10s
================== Transaction protocols ==============
##如下是packetbeat默认支持的主要协议及端口
packetbeat.protocols.icmp:
enabled:
true
packetbeat.protocols.amqp:
ports: [5672]
packetbeat.protocols.cassandra:
ports: [9042]
packetbeat.protocols.dns:
ports: [53]
include_authorities:
true
include_additionals:
true
packetbeat.protocols.http:
ports: [80, 8080, 8000, 5000, 8002]
packetbeat.protocols.memcache:
ports: [11211]
packetbeat.protocols.mysql:
ports: [3306]
packetbeat.protocols.pgsql:
ports: [5432]
packetbeat.protocols.redis:
ports: [6379]
packetbeat.protocols.thrift:
ports: [9090]
packetbeat.protocols.mongodb:
ports: [27017]
packetbeat.protocols.nfs:
ports: [2049]
========================= General =========================
========================= Outputs =========================
------------------- Elasticsearch output ------------------
output.elasticsearch:
hosts: [
"localhost:9200"
]
--------------------- Logstash output ---------------------
output.logstash:
hosts: [
"localhost:5043"
]
##只需配置该处,其他默认即可
============================= Logging =====================
|
②验证配置并启动packetbeat
1
2
3
4
5
6
7
|
# packetbeat.sh -version ##查看packetbeat版本
packetbeat version 5.0.1 (amd64), libbeat 5.0.1
# packetbeat.sh -configtest -e ##测试配置文件
......
Config OK
# systemctl start packetbeat
# systemctl status packetbeat
|
③配置Kibana(新建packetbeat索引)
在http://localhost:5601下新建索引页面输入“packetbeat-*”,之后kibana会自动更新,在“Time-field name”下面的三个选项中选择“@timestamp”,最后点击“Create”创建即可。
创建完成后,kibana显示如下:
c.安装部署Metricbeat
①安装配置metricbeat
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
|
# curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-5.0.1-x86_64.rpm
# rpm -ivh metricbeat-5.0.1-x86_64.rpm
# cat /etc/metricbeat/metricbeat.yml
================= Modules configuration =================
metricbeat.modules:
---------------------- System Module ---------------------
- module: system
metricsets:
- cpu
- load
- filesystem
- fsstat
- memory
- network
- process
enabled:
true
period: 10s
processes: [
'.*'
]
========================= General =======================
========================= Outputs =======================
------------------- Elasticsearch output ----------------
output.elasticsearch:
hosts: [
"localhost:9200"
]
--------------------- Logstash output -------------------
output.logstash:
hosts: [
"localhost:5043"
]
##只需配置该项,其他默认即可
======================= Logging =========================
|
②验证配置并启动metricbeat
1
2
3
4
5
|
# metricbeat.sh -version ##查看版本信息
metricbeat version 5.0.1 (amd64), libbeat 5.0.1
# metricbeat.sh -configtest -e ##验证配置文件
# systemctl start metricbeat
# systemctl status metricbeat
|
③配置kibana(新建metricbeat索引)
在http://localhost:5601下新建索引页面输入“metricbeat-*”,之后kibana会自动更新,在“Time-field name”下面的选项中选择“@timestamp”,最后点击“Create”创建即可。
最后呈现如下图所示:
注1:
关于ELK Stack的一些查询语句:
①查询filebeat
1
|
# curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'
|
②查询packetbeat
1
|
# curl -XGET 'http://localhost:9200/packetbeat-*/_search?pretty'
|
③查询metricbeat
1
|
# curl -XGET 'http://localhost:9200/metricbeat-*/_search?pretty'
|
④查询集群健康度
1
|
# curl 'localhost:9200/_cat/health?v'
|
⑤查看节点列表
1
2
3
|
# curl 'localhost:9200/_cat/nodes?v'
ip heap.percent
ram
.percent cpu load_1m load_5m load_15m node.role master name
127.0.0.1 37 93 3 0.05 0.07 0.41 mdi * XVY0Ovb
|
⑥列出所有索引
1
2
3
4
5
6
7
8
9
10
11
12
|
# curl 'localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow
open
filebeat-2016.11.28 Mn4MzxdTRaCj9iseutcmqA 5 1 2 0 12kb 12kb
yellow
open
filebeat-2016.11.29 iMrr710mT42mApxdV62k-A 5 1 159 0 65.9kb 65.9kb
yellow
open
packetbeat-2016.11.29 wkTcIwD6RgiiCFwlWBIILA 5 1 5652 0 1.6mb 1.6mb
yellow
open
customer NvxXLgHoREefJLRhot13Ug 5 1 0 0 800b 800b
yellow
open
packetbeat-2016.11.28 Beoe07S7QB-dntNV4nxJNQ 5 1 2446 0 676.4kb 676.4kb
yellow
open
test
M7WbkYq2QNmeJ9NOyMfMZA 5 1 0 0 800b 800b
yellow
open
logstash-2016.11.28 pcb_84ChSBe9A7VRd-SQNw 5 1 161 0 123.2kb 123.2kb
yellow
open
metricbeat-2016.11.29 AmVeT1xCQGCnxlAFXUxhYw 5 1 94459 0 37.6mb 37.6mb
yellow
open
logstash-2016.11.29 6PCKMYKCSVmPfdg-Sx2ARA 5 1 85772 0 20.3mb 20.3mb
yellow
open
.kibana QYTg0I5KS-yc3d7GSey3Zw 1 1 5 0 102kb 102kb
|
注2:
如果搭建期间有什么不清楚或不了解的,建议去看官方文档,文档地址如下:
https://www.elastic.co/guide/index.html
注3:如需创建自己的beat,该处有教程可供学习:
http://www.itnose.net/detail/6675459.html