Prometheus Consul Blackbox | Export 监控实现-51CTO.COM

前言：

blackbox_exporter

是Prometheus 官方提供的 exporter 之一,主要提供http、dns、tcp、icmp 的监控数据采集。

Consul

主要提供，服务发现，健康检查，等功能，本次集成主要使用到服务发现功能。

本文主要实现，基于consul_sd_config & consul 的 prometheus 服务发现，实现网路设备ping监控，站点可用行监控，以及证书相关信息监控。

安装环境：

k8s
consul
Prometheus
blackbox_exporter

1: Consul 安装

1.1：使用helm 安装 consul

Bash
# 添加 consul helm 源
helm repo add hashicorp https://helm.releases.hashicorp.com
# 安装consul 
helm -n consul  install  \
--set storageClass=alicloud-disk-efficiency  \
consul hashicorp/consul \
--version=0.32.1

1.2：查看服务安装状态

Bash
[root@xxxxxxxx consul_install]# kubectl -n consul get pods
NAME                     READY   STATUS    RESTARTS   AGE
consul-consul-9lxfc      1/1     Running   0          6d1h
consul-consul-ntqcf      1/1     Running   0          6d1h
consul-consul-q7c6f      1/1     Running   0          6d1h
consul-consul-server-0   1/1     Running   0          6d1h
consul-consul-server-1   1/1     Running   0          6d1h
consul-consul-server-2   1/1     Running   0          6d1h

1.3：nginx-ingress consul

consul_ingress.yml

Bash
# consul.xxxxxx.cn  ----->  替换为正确域名

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: consul-ingress
  namespace: consul
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: consul.xxxxxx.cn
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: consul-consul-ui
            port: 
              number: 80

执行部署

Bash
kubectl apply -f consul_ingress.yml

1.4：访问测试

2: Blackbox_export

2.1：blackbox 安装

blackbox-exporter-config.yaml

Bash
apiVersion: v1
kind: ConfigMap
metadata:
  name: blackbox-exporter
  labels:
    app: blackbox-exporter
data:
  blackbox.yml: |-
    modules:
      ## ----------- DNS 检测配置 -----------
      dns_tcp:  
        prober: dns
        dns:
          transport_protocol: "tcp"
          preferred_ip_protocol: "ip4"
          query_name: "kubernetes.default.svc.cluster.local" # 用于检测域名可用的网址
          query_type: "A" 
      ## ----------- TCP 检测模块配置 -----------
      tcp_connect:
        prober: tcp
        timeout: 5s
      ## ----------- ICMP 检测配置 -----------
      ping:
        prober: icmp
        timeout: 5s
        icmp:
          preferred_ip_protocol: "ip4"
      ## ----------- HTTP GET 2xx 检测模块配置 -----------
      http_get_2xx:  
        prober: http
        timeout: 10s
        http:
          method: GET
          preferred_ip_protocol: "ip4"
          valid_http_versions: ["HTTP/1.1","HTTP/2"]
          valid_status_codes: [200]           # 验证的HTTP状态码,默认为2xx
          no_follow_redirects: false          # 是否不跟随重定向
      ## ----------- HTTP GET 3xx 检测模块配置 -----------
      http_get_3xx:  
        prober: http
        timeout: 10s
        http:
          method: GET
          preferred_ip_protocol: "ip4"
          valid_http_versions: ["HTTP/1.1","HTTP/2"]
          valid_status_codes: [301,302,304,305,306,307]  # 验证的HTTP状态码,默认为2xx
          no_follow_redirects: false                     # 是否不跟随重定向
      ## ----------- HTTP POST 监测模块 -----------
      http_post_2xx: 
        prober: http
        timeout: 10s
        http:
          method: POST
          preferred_ip_protocol: "ip4"
          valid_http_versions: ["HTTP/1.1", "HTTP/2"]
          #headers:                             # HTTP头设置
          #  Content-Type: application/json
          #body: '{}'                           # 请求体设置

blackbox-exporter-deploy.yaml

Bash
apiVersion: v1
kind: Service
metadata:
  name: blackbox-exporter
  labels:
    k8s-app: blackbox-exporter
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 9115
    targetPort: 9115
  selector:
    k8s-app: blackbox-exporter
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: blackbox-exporter
  labels:
    k8s-app: blackbox-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: blackbox-exporter
  template:
    metadata:
      labels:
        k8s-app: blackbox-exporter
    spec:
      containers:
      - name: blackbox-exporter
        image: prom/blackbox-exporter:v0.19.0
        args:
        - --config.file=/etc/blackbox_exporter/blackbox.yml
        - --web.listen-address=:9115
        - --log.level=info
        ports:
        - name: http
          containerPort: 9115
        resources:
          limits:
            cpu: 3
            memory: 6000Mi
          requests:
            cpu: 100m
            memory: 50Mi
        livenessProbe:
          tcpSocket:
            port: 9115
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        readinessProbe:
          tcpSocket:
            port: 9115
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        volumeMounts:
        - name: config
          mountPath: /etc/blackbox_exporter
      volumes:
      - name: config
        configMap:
          name: blackbox-exporter
          defaultMode: 420

执行安装

Bash
kubectl apply -f blackbox-exporter-deploy.yaml
kubectl apply -f blackbox-exporter-config.yaml

2.2：nginx ingress blackbox-exporter • blackbox_ingress.yml

Bash
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blackbox-ingress
  namespace: monitoring
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: blackbox-devops.lululemon.cn
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: blackbox-exporter
            port:
              number: 9115

执行安装

Bash
kubectl apply -f blackbox_ingress.yml

3: rometheus 添加服务动态发现

Bash
#####  http_get_2xx 数据获取
 - job_name: http_get_2xx
      params:
        module:
        - http_get_2xx
      scrape_interval: 2s
      scrape_timeout: 2s
      metrics_path: /probe
      consul_sd_configs:
      # consul 服务地址
      - server: consul-consul-server.consul.svc.cluster.local:8500
        tag_separator: ','
        services:
        - http_get_2xx
      relabel_configs:
        - source_labels: ['__meta_consul_service_address']
          target_label: __param_target
        - source_labels: ['__meta_consul_service_address']
          target_label: instance
        - target_label: __address__
          ## blackbox-export  地址
          replacement: blackbox-exporter.monitoring.svc.cluster.local:9115
 ####### icmp 配置 
     - job_name: blackbox_icmp
      params:
        module:
        - ping
      scrape_interval: 2s
      scrape_timeout: 2s
      metrics_path: /probe
      consul_sd_configs:
      # consul 服务地址
      - server: consul-consul-server.consul.svc.cluster.local:8500
        tag_separator: ','
        services:
        - ping
      relabel_configs:
        - source_labels: ['__meta_consul_service_address']
          target_label: __param_target
        - source_labels: ['__meta_consul_service_address']
          target_label: instance
        - target_label: __address__
          ## blackbox-export  地址
          replacement: blackbox-exporter.monitoring.svc.cluster.local:9115

4：添加 icmp 监控

4.1：添加监控地址到consul

icmp_list

Bash
192.168.1.1
192.168.1.2

add_consul_service_icmp.sh

Bash
#!/usr/bin/env bash

ip_addr=$1


if test "$ip_addr";then
        curl -X PUT -d '{
            "id": "icmp_'${ip_addr}'",
            "name": "ping",
            "address": "'${ip_addr}'",
            "port": 443,
            "Meta": {
              "env": "prod",
              "team": "network",
              "project": "network",
              "owner": "Mike"
            },
            "tags": ["node"],
            "checks": [{"http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}' \
            http://consul-consul-server:8500/v1/agent/service/register
else
        echo "请输入参数"
fi

添加service ping

Bash
for i in `cat icmp_list`;do bash add_consul_service_icmp.sh  $i;done

4.2：查看consul 服务

4.3：删除ping 监控地址脚本

Bash
#!/usr/bin/env bash
ip_addr=$1

curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/icmp_${ip_addr}

5: 添加http_get_2xx

5.1：添加监控域名

domain_name_list

Bash
wwww.baidu.com
wwww.1111.com
wwww.2222.com

add_consul_service_http_get_2xx.sh

Bash
#!/usr/bin/env bash

service_name=$1


if test "$service_name";then
        curl -X PUT -d '{
            "id": "http_get_2xx_'${service_name}'",
            "name": "http_get_2xx",
            "address": "https://'${service_name}'",
            "port": 443,
            "Meta": {
              "env": "prod",
              "team": "web",
              "project": "web",
              "owner": "Devops"
            },
            "tags": ["node"],
            "checks": [{"http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}' \
            http://consul-consul-server:8500/v1/agent/service/register
else
        echo "请输入参数"
fi

添加 service http_get_2xx

Bash
for i  in `cat domain_name_list`;do bash  add_consul_service_http_get_2xx.sh  $i;done

5.2：查看consul 服务

5.3：删除域名监控脚本

del_consul_service_http_get_2xx.sh

Bash
#!/usr/bin/env bash
ip_addr=$1

curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/http_get_2xx_${ip_addr}

6：查看prometheus 监控

总结：

使用上述方案，黑盒监控与自建cmdb 平台很容易进行集成，使其监控自动化，不需要过多的人工干预，可以省去大量的人工成本，grafana 的配置这里就不进行过多介绍，自行通过谷歌完成。

Prometheus Consul Blackbox | Export 监控实现

前言：