Prometheus Consul Blackbox | Export 监控实现

作者：郭鹏超 2022-12-13 08:01:06安全 应用安全 使用上述方案，黑盒监控与自建cmdb 平台很容易进行集成，使其监控自动化，不需要过多的人工干预，可以省去大量的人工成本，grafana 的配置这里就不进行过多介绍，自行通过谷歌完成。

前言：

• blackbox_exporter

是Prometheus 官方提供的 exporter 之一,主要提供http、dns、tcp、icmp 的监控数据采集。

• Consul

主要提供，服务发现，健康检查，等功能，本次集成主要使用到服务发现功能。

本文主要实现，基于consul_sd_config  & consul 的 prometheus  服务发现，实现网路设备ping监控，站点可用行监控，以及证书相关信息监控。

安装环境：

• k8s
• consul
• Prometheus
• blackbox_exporter

1:  Consul 安装

1.1：使用helm 安装 consul

Bash
# 添加 consul helm 源
helm repo add hashicorp https://helm.releases.hashicorp.com
# 安装consul 
helm -n consul  install  \
--set storageClass=alicloud-disk-efficiency  \
consul hashicorp/consul \
--version=0.32.1

1.2：查看服务安装状态

Bash
[root@xxxxxxxx consul_install]# kubectl -n consul get pods
NAME                     READY   STATUS    RESTARTS   AGE
consul-consul-9lxfc      1/1     Running   0          6d1h
consul-consul-ntqcf      1/1     Running   0          6d1h
consul-consul-q7c6f      1/1     Running   0          6d1h
consul-consul-server-0   1/1     Running   0          6d1h
consul-consul-server-1   1/1     Running   0          6d1h
consul-consul-server-2   1/1     Running   0          6d1h

1.3：nginx-ingress consul

• consul_ingress.yml

Bash
# consul.xxxxxx.cn  ----->  替换为正确域名

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: consul-ingress
namespace: consul
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: consul.xxxxxx.cn
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: consul-consul-ui
port: 
number: 80

• 执行部署

Bash
kubectl apply -f consul_ingress.yml

1.4：访问测试

2: Blackbox_export

2.1：blackbox 安装

• blackbox-exporter-config.yaml

Bash
apiVersion: v1
kind: ConfigMap
metadata:
name: blackbox-exporter
labels:
app: blackbox-exporter
data:
blackbox.yml: |-
modules:
## ----------- DNS 检测配置 -----------
dns_tcp:  
prober: dns
dns:
transport_protocol: "tcp"
preferred_ip_protocol: "ip4"
query_name: "kubernetes.default.svc.cluster.local" # 用于检测域名可用的网址
query_type: "A" 
## ----------- TCP 检测模块配置 -----------
tcp_connect:
prober: tcp
timeout: 5s
## ----------- ICMP 检测配置 -----------
ping:
prober: icmp
timeout: 5s
icmp:
preferred_ip_protocol: "ip4"
## ----------- HTTP GET 2xx 检测模块配置 -----------
http_get_2xx:  
prober: http
timeout: 10s
http:
method: GET
preferred_ip_protocol: "ip4"
valid_http_versions: ["HTTP/1.1","HTTP/2"]
valid_status_codes: [200]           # 验证的HTTP状态码,默认为2xx
no_follow_redirects: false          # 是否不跟随重定向
## ----------- HTTP GET 3xx 检测模块配置 -----------
http_get_3xx:  
prober: http
timeout: 10s
http:
method: GET
preferred_ip_protocol: "ip4"
valid_http_versions: ["HTTP/1.1","HTTP/2"]
valid_status_codes: [301,302,304,305,306,307]  # 验证的HTTP状态码,默认为2xx
no_follow_redirects: false                     # 是否不跟随重定向
## ----------- HTTP POST 监测模块 -----------
http_post_2xx: 
prober: http
timeout: 10s
http:
method: POST
preferred_ip_protocol: "ip4"
valid_http_versions: ["HTTP/1.1", "HTTP/2"]
#headers:                             # HTTP头设置
#  Content-Type: application/json
#body: '{ }'                           # 请求体设置

• blackbox-exporter-deploy.yaml

Bash
apiVersion: v1
kind: Service
metadata:
name: blackbox-exporter
labels:
k8s-app: blackbox-exporter
spec:
type: ClusterIP
ports:
- name: http
port: 9115
targetPort: 9115
selector:
k8s-app: blackbox-exporter
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: blackbox-exporter
labels:
k8s-app: blackbox-exporter
spec:
replicas: 1
selector:
matchLabels:
k8s-app: blackbox-exporter
template:
metadata:
labels:
k8s-app: blackbox-exporter
spec:
containers:
- name: blackbox-exporter
image: prom/blackbox-exporter:v0.19.0
args:
- --config.file=/etc/blackbox_exporter/blackbox.yml
- --web.listen-address=:9115
- --log.level=info
ports:
- name: http
containerPort: 9115
resources:
limits:
cpu: 3
memory: 6000Mi
requests:
cpu: 100m
memory: 50Mi
livenessProbe:
tcpSocket:
port: 9115
initialDelaySeconds: 5
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
readinessProbe:
tcpSocket:
port: 9115
initialDelaySeconds: 5
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
volumeMounts:
- name: config
mountPath: /etc/blackbox_exporter
volumes:
- name: config
configMap:
name: blackbox-exporter
defaultMode: 420

• 执行安装

Bash
kubectl apply -f blackbox-exporter-deploy.yaml
kubectl apply -f blackbox-exporter-config.yaml

2.2：nginx ingress blackbox-exporter • blackbox_ingress.yml

Bash
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: blackbox-ingress
namespace: monitoring
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: blackbox-devops.lululemon.cn
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: blackbox-exporter
port:
number: 9115

• 执行安装

Bash
kubectl apply -f blackbox_ingress.yml

3: rometheus 添加 服务动态发现

Bash
#####  http_get_2xx 数据获取
- job_name: http_get_2xx
params:
module:
- http_get_2xx
scrape_interval: 2s
scrape_timeout: 2s
metrics_path: /probe
consul_sd_configs:
# consul 服务地址
- server: consul-consul-server.consul.svc.cluster.local:8500
tag_separator: ','
services:
- http_get_2xx
relabel_configs:
- source_labels: ['__meta_consul_service_address']
target_label: __param_target
- source_labels: ['__meta_consul_service_address']
target_label: instance
- target_label: __address__
## blackbox-export  地址
replacement: blackbox-exporter.monitoring.svc.cluster.local:9115
####### icmp 配置 
- job_name: blackbox_icmp
params:
module:
- ping
scrape_interval: 2s
scrape_timeout: 2s
metrics_path: /probe
consul_sd_configs:
# consul 服务地址
- server: consul-consul-server.consul.svc.cluster.local:8500
tag_separator: ','
services:
- ping
relabel_configs:
- source_labels: ['__meta_consul_service_address']
target_label: __param_target
- source_labels: ['__meta_consul_service_address']
target_label: instance
- target_label: __address__
## blackbox-export  地址
replacement: blackbox-exporter.monitoring.svc.cluster.local:9115

4：添加 icmp 监控

4.1：添加监控地址到consul

• icmp_list

Bash
192.168.1.1
192.168.1.2

• add_consul_service_icmp.sh

Bash
#!/usr/bin/env bash

ip_addr=$1


if test "$ip_addr";then
curl -X PUT -d '{ 
"id": "icmp_'${ ip_addr}'",
"name": "ping",
"address": "'${ ip_addr}'",
"port": 443,
"Meta": { 
"env": "prod",
"team": "network",
"project": "network",
"owner": "Mike"
},
"tags": ["node"],
"checks": [{ "http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}' \
http://consul-consul-server:8500/v1/agent/service/register
else
echo "请输入参数"
fi

• 添加service ping

Bash
for i in `cat icmp_list`;do bash add_consul_service_icmp.sh  $i;done

4.2：查看consul 服务

4.3：删除ping 监控地址脚本

Bash
#!/usr/bin/env bash
ip_addr=$1

curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/icmp_${ ip_addr}

5: 添加http_get_2xx

5.1：添加监控域名

• domain_name_list

Bash
wwww.baidu.com
wwww.1111.com
wwww.2222.com

• add_consul_service_http_get_2xx.sh

Bash
#!/usr/bin/env bash

service_name=$1


if test "$service_name";then
curl -X PUT -d '{ 
"id": "http_get_2xx_'${ service_name}'",
"name": "http_get_2xx",
"address": "https://'${ service_name}'",
"port": 443,
"Meta": { 
"env": "prod",
"team": "web",
"project": "web",
"owner": "Devops"
},
"tags": ["node"],
"checks": [{ "http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}' \
http://consul-consul-server:8500/v1/agent/service/register
else
echo "请输入参数"
fi

• 添加 service  http_get_2xx

Bash
for i  in `cat domain_name_list`;do bash  add_consul_service_http_get_2xx.sh  $i;done

5.2：查看consul 服务

5.3：删除域名监控脚本

• del_consul_service_http_get_2xx.sh

Bash
#!/usr/bin/env bash
ip_addr=$1

curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/http_get_2xx_${ ip_addr}

6：查看prometheus 监控

总结：

使用上述方案，黑盒监控与自建cmdb 平台很容易进行集成，使其监控自动化，不需要过多的人工干预，可以省去大量的人工成本，grafana 的配置这里就不进行过多介绍，自行通过谷歌完成。

责任编辑：武晓燕 来源： 新钛云服 监控黑盒集成

(责任编辑：休闲)