前言:我们使用nginx来模拟产生日志的服务,通过filebeat收集,交给kafka进行消息队列,再用logstash消费kafka集群中的数据,交给elasticsearch+kibana监控文章来源:https://www.toymoban.com/news/detail-776259.html
一,环境
服务器环境:文章来源地址https://www.toymoban.com/news/detail-776259.html
192.168.2.1:elasticsearch
192.168.2.2:filebeat+nginx
192.168.2.3:kafka
192.168.2.4:logstash
二,服务的安装
elasticseatch+filebeat+kafka+logsstash(6.60)清华源下载: https://mirrors.tuna.tsinghua.edu.cn/elasticstack/6.x/yum/6.6.0/
zookeeper官网下载: https://zookeeper.apache.org/releases.html
kafka官网下载: https://kafka.apache.org/downloads
一,配置elasticsearch
1.验证java环境(存在无需安装)
java -version#验证java环境
安装JDK1.8:yum -y install java-1.8.0-openjdk.x86_64
2.安装elasticsearch
rpm -ivh /mnt/elk-6.6/elasticsearch-6.6.0.rpm
3.修改配置文件
vi /etc/elasticsearch/elasticsearch.yml
修改一下内容:
node.name: node-1 #群集中本机节点名
network.host: 192.168.2.1,127.0.0.1 #监听的ip地址
http.port: 9200
4.开启elasticsearch
systemctl start elasticsearch
5.查看启动情况
[root@localhost ~]# netstat -anpt | grep java
tcp6 0 0192.168.2.1:9200 :::* LISTEN 12564/java
tcp6 0 0127.0.0.1:9200 :::* LISTEN 12564/java
tcp6 0 0192.168.2.1:9300 :::* LISTEN 12564/java
tcp6 0 0127.0.0.1:9300 :::* LISTEN 12564/java
tcp6 0 0192.168.2.1:9200 192.168.2.4:34428 ESTABLISHED 12564/java
tcp6 0 0192.168.2.1:9200 192.168.2.4:34436 ESTABLISHED 12564/java
二,配置filebeat+nginx
1.安装nginx
yum -y install nginx
2.安装filebeat
rpm -ivh /mnt/elk-6.6/filebeat-6.6.0-x86_64.rpm
3.修改filebeat配置文件
[root@localhost ~]# vi /etc/filebeat/filebeat.yml
添加一下内容:
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
output.kafka:
enabled: true
hosts: ["192.168.2.3:9092"] #kafka的IP地址和端口
topic: test1 #kafka的topic
4.开启服务
systemctl start nginx
systemctl start filebeat
三,配置kafka环境
1.安装java环境(存在无需安装)
yum -y install java-1.8.0-openjdk.x86_64
2.安装zookeeper
tar xf /mnt/zookeeper-3.4.9.tar.gz -C /usr/local/
mv /usr/local/zookeeper-3.4.9/ /usr/local/zookeeper
cd /usr/local/zookeeper/conf
cp zoo_sample.cfg zoo.cfg
vi zoo.cfg
mkdir data logs
echo 1 > data/myid
/usr/local/zookeeper/bin/zkServer.sh start
/usr/local/zookeeper/bin/zkServer.sh status
1.修改zookeeper配置文件
vi zoo.cfg
添加一下内容:
dataDir=/usr/local/zookeeper/data
dataLogDir=/usr/local/zookeeper/logs
server.1=192.168.2.3:3188:3288
保存退出
mkdir data logs
echo1 > data/myid
2.启动zookeeper文件
/usr/local/zookeeper/bin/zkServer.sh start
/usr/local/zookeeper/bin/zkServer.sh status
3.安装kafka
tar xf /mnt/kafka_2.11-2.2.1.tgz -C /usr/local/
mv /usr/local/kafka_2.11-2.2.1/ /usr/local/kafka
4.配置kafka
cd /usr/local/kafka/config/
cp server.properties server.properties.bak
vi server.properties
修改一下内容:
broker.id=1
listeners=PLAINTEXT://192.168.2.3:9092
zookeeper.connect=192.168.2.3:2181
5.启动kafka
cd /usr/local/kafka/
./bin/kafka-server-start.sh ./config/server.properties &
6.kafka创建topic
./bin/kafka-topics.sh --create--zookeeper localhost:2181 --replication-factor1--partitions1--topic test1 #创建名为test1的topic
./bin/kafka-topics.sh --list--zookeeper localhost:2181 #查看当前有哪些topic
./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test1 --from-beginning #查看test1中有那些信息
四,配置logstash
1.安装java环境(存在可省略)
yum -y install java-1.8.0-openjdk.x86_64
2.安装logstash
rpm -ivh /mnt/elk-6.6/logstash-6.6.0.rpm
3.编写配置文件
vi /etc/logstash/conf.d/kafka.conf
input {
kafka {
bootstrap_servers => ["192.168.2.3:9092"]
group_id => "es-test"
topics => ["test1"] #与filebeat使用的topic一致
codec => json
}
}
output {
kafka{
codec => json {
charset => "UTF-8"
}
topic_id => "test1"
bootstrap_servers => "192.168.2.3:9092"
}
elasticsearch {
hosts => "http://192.168.2.1:9200"
index => "kafka‐%{+YYYY.MM.dd}"
}
}
4.启动服务
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis.conf
五,配置kibana
1.安装kiana
rpm -ihv /mnt/elk-6.6/kibana-6.6.0-x86_64.rpm
2.配置kiana
vim /etc/kibana/kibana.yml
修改:
server.port: 5601
server.host: "192.168.2.5"
server.name: "db01"
elasticsearch.hosts: ["http://192.168.2.1:9200"] #es服务器的ip,便于接收日志数据
3.开启kiana服务
systemctl start kibana
三,收集日志
一,kibana收集日志
添加日志信息
选择日志格式
3.查看日志信息
到了这里,关于filebeat+kafka+logstash+elasticsearch+kibana实现日志收集解决方案的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!