Prometheus-Operator 手动入门实战

prometheus operator 手动入门实战

虽然生产中我们大部分都是通过Helm的方式来安装Prometheus-Operator，但是我们也是需要明白Prometheus Operator的原理，以及其中5个CRD的YAML定义方式，这里我们以手动的方式来安装部署Prometheus Operator以及Prometheus Server、Alertmanager集群，配置我们需要监控的对象。

Prometheus Operator GitHub

本文采用的环境以及版本：

• Kubernetes 1.15.5 + Calico v3.6.5
• Prometheus-Operator v0.34.0

Prometheus Operator简介

Prometheus Operator 安装后提供如下功能：

• 很容易的在kubernetes集群中特定的Namespace中创建和销毁prometheus实例；
• 配置简单：使用kubernetes原生的方式（CRD）对prometheus的版本，持久化，数据保留，实例数进行配置；
• 通过labels来指定生成对目标监控的配置；

架构图如下：

Operator 监控Kubernetes集群中自定义资源Prometheus、Alertmanager的变化，部署和管理Prometheus Server以及Alertmanager集群，使其状态以及配置达到用户的期望状态，同时监控k8s集群中自定义资源ServiceMonitor（它是用来指定需要收集Metrics的目标的服务），来生成Prometheus的配置信息。

Prometheus Operator vs. kube-prometheus vs. community helm chart 都是什么？

• prometheus operator使用kubernetes原生的方式来管理和操作prometheus和alertmanager集群。
• kube-prometheus 联合prometheus operator和一些manifests来帮助监控kubernetes集群本身以及跑在kubernetes上面的应用。
• stable/prometheus-operator Helm chart 提供简单的功能集来创建kube-prometheus。

前提依赖以及兼容性：

推荐Kubernetes集群版本在1.8以上。

快速安装Kubernetes测试集群：

兼容的prometheus版本：v1.4.0–v2.10.0，参考

兼容的alertmanager版本：>= v0.15，参考

CRD（Custom Resource Definitions 自定义资源）

• Prometheus：定义期望的Prometheus实例，同时保证任何时候有期望的Prometheus实例在运行。
• ServiceMonitor：通过声明式的方式指定哪些服务需要被监控，它自动生成Prometheus 的scrape配置。
• PodMonitor：通过声明式的方式指定哪些pod需要被监控，它自动生成Prometheus 的scrape配置。
• PrometheusRule：配置Prometheus rule文件，包括recording rules和alerting，它能够自动被Prometheus加载。
• Alertmanager：定义期望的Alertmanager实例，同时保证任何时候有期望的Alertmanager实例在运行，对于指定多台Alertmanager，prometheus operator会自动将它们配置成集群。

安装prometheus-operator

下载prometheus-operator YAML文件：

$ wget -c "https://github.com/coreos/prometheus-operator/raw/master/bundle.yaml"

更改bundel.yaml 文件中的image地址，默认是使用quay.io的镜像，这里修改成quay.azk8s.cn的地址，可以快速的下载镜像，同时更改默认的namespace为prometheus

$ sed -i 's#quay.io#quay.azk8s.cn#g' bundle.yaml
$ sed -i 's#namespace: default#namespace: prometheus#g' bundle.yaml

更改后的显示如下：

containers:
- args:
  - --kubelet-service=kube-system/kubelet
  - --logtostderr=true
  - --config-reloader-image=quay.azk8s.cn/coreos/configmap-reload:v0.0.1
  - --prometheus-config-reloader=quay.azk8s.cn/coreos/prometheus-config-reloader:v0.34.0
  image: quay.azk8s.cn/coreos/prometheus-operator:v0.34.0

安装prometheus-operator 到prometheus namespace中：

$ kubectl create ns prometheus
namespace/prometheus created
$ kubectl get ns
NAME              STATUS   AGE
default           Active   4d6h
kube-node-lease   Active   4d6h
kube-public       Active   4d6h
kube-system       Active   4d6h
prometheus        Active   7s
$ kubectl apply -f bundle.yaml
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator configured
clusterrole.rbac.authorization.k8s.io/prometheus-operator unchanged
deployment.apps/prometheus-operator created
serviceaccount/prometheus-operator created
service/prometheus-operator created

检查安装状态：

$ kubectl -n prometheus get all
NAME                                       READY   STATUS    RESTARTS   AGE
pod/prometheus-operator-5748cc95dd-ssn7s   1/1     Running   0          2m45s


NAME                          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/prometheus-operator   ClusterIP   None         <none>        8080/TCP   2m46s


NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/prometheus-operator   1/1     1            1           2m46s

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/prometheus-operator-5748cc95dd   1         1         1       2m46s

查看prometheus operator的启动参数，以及命令帮助：

# 进入prometheus operator容器内：
$ kubectl -n prometheus exec -it prometheus-operator-5748cc95dd-ssn7s -- sh

# 查看operator启动参数：
$ ps -ef|more
PID   USER     TIME  COMMAND
    1 nobody    0:04 /bin/operator --kubelet-service=kube-system/kubelet --logtostderr=true --config-reloader-image=quay.azk8s.cn/coreos/configmap-reload:v0.0.1 --prometheus-config-reloader=quay.azk8s.cn/coreos/prometheus-config-reloader:v0.34.0
    
# 查看operator的命令帮助：
$ operator --help
Usage of operator:
  -alertmanager-default-base-image string
    	Alertmanager default base image (default "quay.io/prometheus/alertmanager")
  -alertmanager-instance-namespaces value
    	Namespaces where Alertmanager custom resources and corresponding StatefulSets are watched/created. If set this takes precedence over --namespaces or --deny-namespaces for Alertmanager custom resources.
  -alertmanager-instance-selector string
    	Label selector to filter AlertManager CRDs to manage
  -alsologtostderr
    	log to standard error as well as files
  -apiserver string
    	API Server addr, e.g. ' - NOT RECOMMENDED FOR PRODUCTION - http://127.0.0.1:8080'. Omit parameter to run in on-cluster mode and utilize the service account token.
  -ca-file string
    	- NOT RECOMMENDED FOR PRODUCTION - Path to TLS CA file.
  -cert-file string
    	 - NOT RECOMMENDED FOR PRODUCTION - Path to public TLS certificate file.
  -config-reloader-cpu string
    	Config Reloader CPU. Value "0" disables it and causes no limit to be configured. (default "100m")
  -config-reloader-image string
    	Reload Image (default "quay.io/coreos/configmap-reload:v0.0.1")
  -config-reloader-memory string
    	Config Reloader Memory. Value "0" disables it and causes no limit to be configured. (default "25Mi")
  -crd-kinds value
    	 - EXPERIMENTAL (could be removed in future releases) - customize CRD kind names
  -deny-namespaces value
    	Namespaces not to scope the interaction of the Prometheus Operator (deny list). This is mutually exclusive with --namespaces.
  -key-file string
    	- NOT RECOMMENDED FOR PRODUCTION - Path to private TLS certificate file.
  -kubelet-service string
    	Service/Endpoints object to write kubelets into in format "namespace/name"
  -labels value
    	Labels to be add to all resources created by the operator
  -localhost string
    	EXPERIMENTAL (could be removed in future releases) - Host used to communicate between local services on a pod. Fixes issues where localhost resolves incorrectly. (default "localhost")
  -log-format string
    	Log format to use. Possible values: logfmt, json (default "logfmt")
  -log-level string
    	Log level to use. Possible values: all, debug, info, warn, error, none (default "info")
  -log_backtrace_at value
    	when logging hits line file:N, emit a stack trace
  -log_dir string
    	If non-empty, write log files in this directory
  -log_file string
    	If non-empty, use this log file
  -log_file_max_size uint
    	Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
  -logtostderr
    	log to standard error instead of files (default true)
  -manage-crds
    	Manage all CRDs with the Prometheus Operator. (default true)
  -namespaces value
    	Namespaces to scope the interaction of the Prometheus Operator and the apiserver (allow list). This is mutually exclusive with --deny-namespaces.
  -prometheus-config-reloader string
    	Prometheus config reloader image (default "quay.io/coreos/prometheus-config-reloader:v0.34.0")
  -prometheus-default-base-image string
    	Prometheus default base image (default "quay.io/prometheus/prometheus")
  -prometheus-instance-namespaces value
    	Namespaces where Prometheus custom resources and corresponding Secrets, Configmaps and StatefulSets are watched/created. If set this takes precedence over --namespaces or --deny-namespaces for Prometheus custom resources.
  -prometheus-instance-selector string
    	Label selector to filter Prometheus CRDs to manage
  -skip_headers
    	If true, avoid header prefixes in the log messages
  -skip_log_headers
    	If true, avoid headers when opening log files
  -stderrthreshold value
    	logs at or above this threshold go to stderr (default 2)
  -thanos-default-base-image string
    	Thanos default base image (default "quay.io/thanos/thanos")
  -tls-insecure
    	- NOT RECOMMENDED FOR PRODUCTION - Don't verify API server's CA certificate.
  -v value
    	number for the log level verbosity
  -vmodule value
    	comma-separated list of pattern=N settings for file-filtered logging
  -with-validation
    	Include the validation spec in the CRD (default true)

安装Prometheus Operator CRDs ??

安装完prometheus operator 我们还需要手动的安装Prometheus-Operator所需的CRD嘛？

至少GitHub上没有写安装，但是如果不需要手动安装对应的CRD，那么是谁安装的呢？这是我的疑问，相信你们也会有；那么我们先检查一下是否已经有了这些CRD呢？

$ kubectl get crd | grep monitoring.coreos.com
alertmanagers.monitoring.coreos.com           2019-11-16T12:19:01Z
podmonitors.monitoring.coreos.com             2019-11-16T12:19:02Z
prometheuses.monitoring.coreos.com            2019-11-16T12:19:01Z
prometheusrules.monitoring.coreos.com         2019-11-16T12:19:02Z
servicemonitors.monitoring.coreos.com         2019-11-16T12:19:02Z

我们发现这些CRD已经安装上了，那么是谁安装的呢？答案只有一个，那就是Prometheus Operator了，我们来检查一下，首先检查一下安装Operator的yaml文件：bundle.yaml

# 其中的ClusterRole，我们发现它是有权限去安装CRD的，而且对应的权限都是*，也就是可以对需要的资源做任何操作。

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: prometheus-operator
    app.kubernetes.io/version: v0.34.0
  name: prometheus-operator
rules:
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - '*'
- apiGroups:
  - monitoring.coreos.com
  resources:
  - alertmanagers
  - prometheuses
  - prometheuses/finalizers
  - alertmanagers/finalizers
  - servicemonitors
  - podmonitors
  - prometheusrules
  verbs:
  - '*'
- apiGroups:
  - apps
  resources:
  - statefulsets
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - configmaps
  - secrets
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - list
  - delete
- apiGroups:
  - ""
  resources:
  - services
  - services/finalizers
  - endpoints
  verbs:
  - get
  - create
  - update
  - delete
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
  - list
  - watch

检查一下Operator的启动日志：

$ kubectl -n prometheus logs prometheus-operator-5748cc95dd-wgkhn 
ts=2019-11-17T01:21:50.319966698Z caller=main.go:199 msg="Starting Prometheus Operator version '0.34.0'."
ts=2019-11-17T01:21:50.324753193Z caller=main.go:96 msg="Staring insecure server on :8080"
level=info ts=2019-11-17T01:21:50.417372629Z caller=operator.go:441 component=prometheusoperator msg="connection established" cluster-version=v1.15.5
level=info ts=2019-11-17T01:21:50.417486093Z caller=operator.go:219 component=alertmanageroperator msg="connection established" cluster-version=v1.15.5
level=info ts=2019-11-17T01:21:51.315610123Z caller=operator.go:641 component=alertmanageroperator msg="CRD updated" crd=Alertmanager
level=info ts=2019-11-17T01:21:51.415575543Z caller=operator.go:1870 component=prometheusoperator msg="CRD updated" crd=Prometheus
level=info ts=2019-11-17T01:21:51.427497667Z caller=operator.go:1870 component=prometheusoperator msg="CRD updated" crd=ServiceMonitor
level=info ts=2019-11-17T01:21:51.439926809Z caller=operator.go:1870 component=prometheusoperator msg="CRD updated" crd=PodMonitor
level=info ts=2019-11-17T01:21:51.449866666Z caller=operator.go:1870 component=prometheusoperator msg="CRD updated" crd=PrometheusRule
level=info ts=2019-11-17T01:21:54.415708201Z caller=operator.go:235 component=alertmanageroperator msg="CRD API endpoints ready"
level=info ts=2019-11-17T01:21:54.616157837Z caller=operator.go:190 component=alertmanageroperator msg="successfully synced all caches"
level=info ts=2019-11-17T01:22:03.528619385Z caller=operator.go:457 component=prometheusoperator msg="CRD API endpoints ready"
level=info ts=2019-11-17T01:22:04.330326124Z caller=operator.go:387 component=prometheusoperator msg="successfully synced all caches"

我们发现有如下几个CRD被创建了：

msg="CRD updated" crd=Alertmanager
msg="CRD updated" crd=Prometheus
msg="CRD updated" crd=ServiceMonitor
msg="CRD updated" crd=PodMonitor
msg="CRD updated" crd=PrometheusRule

正式我们需要的5个。

我们查看一下Kubernetes中的API：

# 方式1
$ kubectl api-versions 
admissionregistration.k8s.io/v1beta1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1
apiregistration.k8s.io/v1beta1
apps/v1
apps/v1beta1
apps/v1beta2
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
autoscaling/v2beta1
autoscaling/v2beta2
batch/v1
batch/v1beta1
certificates.k8s.io/v1beta1
coordination.k8s.io/v1
coordination.k8s.io/v1beta1
crd.projectcalico.org/v1
events.k8s.io/v1beta1
extensions/v1beta1
monitoring.coreos.com/v1
networking.k8s.io/v1
networking.k8s.io/v1beta1
node.k8s.io/v1beta1
policy/v1beta1
rbac.authorization.k8s.io/v1
rbac.authorization.k8s.io/v1beta1
scheduling.k8s.io/v1
scheduling.k8s.io/v1beta1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1

# 方式2：
$ kubectl get APIService
NAME                                   SERVICE   AVAILABLE   AGE
v1.                                    Local     True        4d20h
v1.apps                                Local     True        4d20h
v1.authentication.k8s.io               Local     True        4d20h
v1.authorization.k8s.io                Local     True        4d20h
v1.autoscaling                         Local     True        4d20h
v1.batch                               Local     True        4d20h
v1.coordination.k8s.io                 Local     True        4d20h
v1.crd.projectcalico.org               Local     True        4d16h
v1.monitoring.coreos.com               Local     True        13h
v1.networking.k8s.io                   Local     True        4d20h
v1.rbac.authorization.k8s.io           Local     True        4d20h
v1.scheduling.k8s.io                   Local     True        4d20h
v1.storage.k8s.io                      Local     True        4d20h
v1beta1.admissionregistration.k8s.io   Local     True        4d20h
v1beta1.apiextensions.k8s.io           Local     True        4d20h
v1beta1.apps                           Local     True        4d20h
v1beta1.authentication.k8s.io          Local     True        4d20h
v1beta1.authorization.k8s.io           Local     True        4d20h
v1beta1.batch                          Local     True        4d20h
v1beta1.certificates.k8s.io            Local     True        4d20h
v1beta1.coordination.k8s.io            Local     True        4d20h
v1beta1.events.k8s.io                  Local     True        4d20h
v1beta1.extensions                     Local     True        4d20h
v1beta1.networking.k8s.io              Local     True        4d20h
v1beta1.node.k8s.io                    Local     True        4d20h
v1beta1.policy                         Local     True        4d20h
v1beta1.rbac.authorization.k8s.io      Local     True        4d20h
v1beta1.scheduling.k8s.io              Local     True        4d20h
v1beta1.storage.k8s.io                 Local     True        4d20h
v1beta2.apps                           Local     True        4d20h
v2beta1.autoscaling                    Local     True        4d20h
v2beta2.autoscaling                    Local     True        4d20h

我们发现有monitoring.coreos.com这个API：v1.monitoring.coreos.com，我们来具体的查看一下：

$ kubectl get --raw /apis/monitoring.coreos.com/v1 | python -m json.tool
{
    "apiVersion": "v1",
    "groupVersion": "monitoring.coreos.com/v1",
    "kind": "APIResourceList",
    "resources": [
        {
            "kind": "PodMonitor",
            "name": "podmonitors",
            "namespaced": true,
            "singularName": "podmonitor",
            "storageVersionHash": "t6BHpUAzPig=",
            "verbs": [
                "delete",
                "deletecollection",
                "get",
                "list",
                "patch",
                "create",
                "update",
                "watch"
            ]
        },
        {
            "kind": "PrometheusRule",
            "name": "prometheusrules",
            "namespaced": true,
            "singularName": "prometheusrule",
            "storageVersionHash": "RSJ8iG+KDOo=",
            "verbs": [
                "delete",
                "deletecollection",
                "get",
                "list",
                "patch",
                "create",
                "update",
                "watch"
            ]
        },
        {
            "kind": "Alertmanager",
            "name": "alertmanagers",
            "namespaced": true,
            "singularName": "alertmanager",
            "storageVersionHash": "NshW3zg1K7o=",
            "verbs": [
                "delete",
                "deletecollection",
                "get",
                "list",
                "patch",
                "create",
                "update",
                "watch"
            ]
        },
        {
            "kind": "ServiceMonitor",
            "name": "servicemonitors",
            "namespaced": true,
            "singularName": "servicemonitor",
            "storageVersionHash": "JLhPcfa+5xE=",
            "verbs": [
                "delete",
                "deletecollection",
                "get",
                "list",
                "patch",
                "create",
                "update",
                "watch"
            ]
        },
        {
            "kind": "Prometheus",
            "name": "prometheuses",
            "namespaced": true,
            "singularName": "prometheus",
            "storageVersionHash": "C8naPY4eojU=",
            "verbs": [
                "delete",
                "deletecollection",
                "get",
                "list",
                "patch",
                "create",
                "update",
                "watch"
            ]
        }
    ]
}

所以呢：在安装完Prometheus-Operator后不需要我们在手动的方式安装所需的CRD了。

来看一下GitHub上对应的CRD目录：

https://github.com/coreos/prometheus-operator/tree/master/example/prometheus-operator-crd

注意：再使用Helm的方式安装Operator的时候，可能需要提前的手动安装一下，可以通过下面的方式直接安装，总共5个CRD：参考

$ kubectl apply -f https://github.com/coreos/prometheus-operator/raw/master/example/prometheus-operator-crd/servicemonitor.crd.yaml
$ kubectl apply -f https://github.com/coreos/prometheus-operator/raw/master/example/prometheus-operator-crd/prometheusrule.crd.yaml
$ kubectl apply -f https://github.com/coreos/prometheus-operator/raw/master/example/prometheus-operator-crd/prometheus.crd.yaml
$ kubectl apply -f https://github.com/coreos/prometheus-operator/raw/master/example/prometheus-operator-crd/podmonitor.crd.yaml
$ kubectl apply -f https://github.com/coreos/prometheus-operator/raw/master/example/prometheus-operator-crd/alertmanager.crd.yaml

Prometheus Operator 以及对应的CRD都已经安装并且成功启动，下面我们来通过kubernetes原生的方式来创建prometheus，alertmanager实例，以及配置监控对象和规则等。

Prometheus Operator 实战

参考：

https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md

创建Prometheus实例：

不知道怎么创建，直接看上面的参考吧，我摘录下来：

Prometheus

Prometheus defines a Prometheus deployment.

Field	Description	Scheme	Required
metadata	Standard object’s metadata. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#metadata	metav1.ObjectMeta	false
spec	Specification of the desired behavior of the Prometheus cluster. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#spec-and-status	PrometheusSpec	true
status	Most recent observed status of the Prometheus cluster. Read-only. Not included when requesting from the apiserver, only from the Prometheus Operator API itself. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#spec-and-status	*PrometheusStatus	false

发现只有PrometheusSpec是必须的，看下面：

PrometheusSpec

PrometheusSpec is a specification of the desired behavior of the Prometheus cluster. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#spec-and-status

Field	Description	Scheme	Required
podMetadata	Standard object’s metadata. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#metadata Metadata Labels and Annotations gets propagated to the prometheus pods.	*metav1.ObjectMeta	false
serviceMonitorSelector	ServiceMonitors to be selected for target discovery.	*metav1.LabelSelector	false
serviceMonitorNamespaceSelector	Namespaces to be selected for ServiceMonitor discovery. If nil, only check own namespace.	*metav1.LabelSelector	false
podMonitorSelector	Experimental PodMonitors to be selected for target discovery.	*metav1.LabelSelector	false
podMonitorNamespaceSelector	Namespaces to be selected for PodMonitor discovery. If nil, only check own namespace.	*metav1.LabelSelector	false
version	Version of Prometheus to be deployed.	string	false
tag	Tag of Prometheus container image to be deployed. Defaults to the value of version. Version is ignored if Tag is set.	string	false
sha	SHA of Prometheus container image to be deployed. Defaults to the value of version. Similar to a tag, but the SHA explicitly deploys an immutable container image. Version and Tag are ignored if SHA is set.	string	false
paused	When a Prometheus deployment is paused, no actions except for deletion will be performed on the underlying objects.	bool	false
image	Image if specified has precedence over baseImage, tag and sha combinations. Specifying the version is still necessary to ensure the Prometheus Operator knows what version of Prometheus is being configured.	*string	false
baseImage	Base image to use for a Prometheus deployment.	string	false
imagePullSecrets	An optional list of references to secrets in the same namespace to use for pulling prometheus and alertmanager images from registries see http://kubernetes.io/docs/user-guide/images#specifying-imagepullsecrets-on-a-pod	[]v1.LocalObjectReference	false
replicas	Number of instances to deploy for a Prometheus deployment.	*int32	false
replicaExternalLabelName	Name of Prometheus external label used to denote replica name. Defaults to the value of prometheus_replica. External label will not be added when value is set to empty string (\"\").	*string	false
prometheusExternalLabelName	Name of Prometheus external label used to denote Prometheus instance name. Defaults to the value of prometheus. External label will not be added when value is set to empty string (\"\").	*string	false
retention	Time duration Prometheus shall retain data for. Default is ‘24h’, and must match the regular expression `[0-9]+(ms	s	m
retentionSize	Maximum amount of disk space used by blocks.	string	false
walCompression	Enable compression of the write-ahead log using Snappy. This flag is only available in versions of Prometheus >= 2.11.0.	*bool	false
logLevel	Log level for Prometheus to be configured with.	string	false
logFormat	Log format for Prometheus to be configured with.	string	false
scrapeInterval	Interval between consecutive scrapes.	string	false
evaluationInterval	Interval between consecutive evaluations.	string	false
rules	/–rules.*/ command-line arguments.	Rules	false
externalLabels	The labels to add to any time series or alerts when communicating with external systems (federation, remote storage, Alertmanager).	map[string]string	false
enableAdminAPI	Enable access to prometheus web admin API. Defaults to the value of false. WARNING: Enabling the admin APIs enables mutating endpoints, to delete data, shutdown Prometheus, and more. Enabling this should be done with care and the user is advised to add additional authentication authorization via a proxy to ensure only clients authorized to perform these actions can do so. For more information see https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-admin-apis	bool	false
externalUrl	The external URL the Prometheus instances will be available under. This is necessary to generate correct URLs. This is necessary if Prometheus is not served from root of a DNS name.	string	false
routePrefix	The route prefix Prometheus registers HTTP handlers for. This is useful, if using ExternalURL and a proxy is rewriting HTTP routes of a request, and the actual ExternalURL is still true, but the server serves requests under a different route prefix. For example for use with kubectl proxy.	string	false
query	QuerySpec defines the query command line flags when starting Prometheus.	*QuerySpec	false
storage	Storage spec to specify how storage shall be used.	*StorageSpec	false
volumes	Volumes allows configuration of additional volumes on the output StatefulSet definition. Volumes specified will be appended to other volumes that are generated as a result of StorageSpec objects.	[]v1.Volume	false
ruleSelector	A selector to select which PrometheusRules to mount for loading alerting rules from. Until (excluding) Prometheus Operator v0.24.0 Prometheus Operator will migrate any legacy rule ConfigMaps to PrometheusRule custom resources selected by RuleSelector. Make sure it does not match any config maps that you do not want to be migrated.	*metav1.LabelSelector	false
ruleNamespaceSelector	Namespaces to be selected for PrometheusRules discovery. If unspecified, only the same namespace as the Prometheus object is in is used.	*metav1.LabelSelector	false
alerting	Define details regarding alerting.	*AlertingSpec	false
resources	Define resources requests and limits for single Pods.	v1.ResourceRequirements	false
nodeSelector	Define which Nodes the Pods are scheduled on.	map[string]string	false
serviceAccountName	ServiceAccountName is the name of the ServiceAccount to use to run the Prometheus Pods.	string	false
secrets	Secrets is a list of Secrets in the same namespace as the Prometheus object, which shall be mounted into the Prometheus Pods. The Secrets are mounted into /etc/prometheus/secrets/.	[]string	false
configMaps	ConfigMaps is a list of ConfigMaps in the same namespace as the Prometheus object, which shall be mounted into the Prometheus Pods. The ConfigMaps are mounted into /etc/prometheus/configmaps/.	[]string	false
affinity	If specified, the pod’s scheduling constraints.	*v1.Affinity	false
tolerations	If specified, the pod’s tolerations.	[]v1.Toleration	false
remoteWrite	If specified, the remote_write spec. This is an experimental feature, it may change in any upcoming release in a breaking way.	[]RemoteWriteSpec	false
remoteRead	If specified, the remote_read spec. This is an experimental feature, it may change in any upcoming release in a breaking way.	[]RemoteReadSpec	false
securityContext	SecurityContext holds pod-level security attributes and common container settings. This defaults to the default PodSecurityContext.	*v1.PodSecurityContext	false
listenLocal	ListenLocal makes the Prometheus server listen on loopback, so that it does not bind against the Pod IP.	bool	false
containers	Containers allows injecting additional containers or modifying operator generated containers. This can be used to allow adding an authentication proxy to a Prometheus pod or to change the behavior of an operator generated container. Containers described here modify an operator generated container if they share the same name and modifications are done via a strategic merge patch. The current container names are: prometheus, prometheus-config-reloader, rules-configmap-reloader, and thanos-sidecar. Overriding containers is entirely outside the scope of what the maintainers will support and by doing so, you accept that this behaviour may break at any time without notice.	[]v1.Container	false
initContainers	InitContainers allows adding initContainers to the pod definition. Those can be used to e.g. fetch secrets for injection into the Prometheus configuration from external sources. Any errors during the execution of an initContainer will lead to a restart of the Pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ Using initContainers for any use case other then secret fetching is entirely outside the scope of what the maintainers will support and by doing so, you accept that this behaviour may break at any time without notice.	[]v1.Container	false
additionalScrapeConfigs	AdditionalScrapeConfigs allows specifying a key of a Secret containing additional Prometheus scrape configurations. Scrape configurations specified are appended to the configurations generated by the Prometheus Operator. Job configurations specified must have the form as specified in the official Prometheus documentation: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config. As scrape configs are appended, the user is responsible to make sure it is valid. Note that using this feature may expose the possibility to break upgrades of Prometheus. It is advised to review Prometheus release notes to ensure that no incompatible scrape configs are going to break Prometheus after the upgrade.	*v1.SecretKeySelector	false
additionalAlertRelabelConfigs	AdditionalAlertRelabelConfigs allows specifying a key of a Secret containing additional Prometheus alert relabel configurations. Alert relabel configurations specified are appended to the configurations generated by the Prometheus Operator. Alert relabel configurations specified must have the form as specified in the official Prometheus documentation: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#alert_relabel_configs. As alert relabel configs are appended, the user is responsible to make sure it is valid. Note that using this feature may expose the possibility to break upgrades of Prometheus. It is advised to review Prometheus release notes to ensure that no incompatible alert relabel configs are going to break Prometheus after the upgrade.	*v1.SecretKeySelector	false
additionalAlertManagerConfigs	AdditionalAlertManagerConfigs allows specifying a key of a Secret containing additional Prometheus AlertManager configurations. AlertManager configurations specified are appended to the configurations generated by the Prometheus Operator. Job configurations specified must have the form as specified in the official Prometheus documentation: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#alertmanager_config. As AlertManager configs are appended, the user is responsible to make sure it is valid. Note that using this feature may expose the possibility to break upgrades of Prometheus. It is advised to review Prometheus release notes to ensure that no incompatible AlertManager configs are going to break Prometheus after the upgrade.	*v1.SecretKeySelector	false
apiserverConfig	APIServerConfig allows specifying a host and auth methods to access apiserver. If left empty, Prometheus is assumed to run inside of the cluster and will discover API servers automatically and use the pod’s CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount/.	*APIServerConfig	false
thanos	Thanos configuration allows configuring various aspects of a Prometheus server in a Thanos environment.\n\nThis section is experimental, it may change significantly without deprecation notice in any release.\n\nThis is experimental and may change significantly without backward compatibility in any release.	*ThanosSpec	false
priorityClassName	Priority class assigned to the Pods	string	false
portName	Port name used for the pods and governing service. This defaults to web	string	false
arbitraryFSAccessThroughSMs	ArbitraryFSAccessThroughSMs configures whether configuration based on a service monitor can access arbitrary files on the file system of the Prometheus container e.g. bearer token files.	ArbitraryFSAccessThroughSMsConfig	false
overrideHonorLabels	OverrideHonorLabels if set to true overrides all user configured honor_labels. If HonorLabels is set in ServiceMonitor or PodMonitor to true, this overrides honor_labels to false.	bool	false
overrideHonorTimestamps	OverrideHonorTimestamps allows to globally enforce honoring timestamps in all scrape configs.	bool	false
ignoreNamespaceSelectors	IgnoreNamespaceSelectors if set to true will ignore NamespaceSelector settings from the podmonitor and servicemonitor configs, and they will only discover endpoints within their current namespace. Defaults to false.	bool	false
enforcedNamespaceLabel	EnforcedNamespaceLabel enforces adding a namespace label of origin for each alert and metric that is user created. The label value will always be the namespace of the object that is being created.	string	false

我们发现也没有必须要填的，那就先创建一个默认的吧：prometheus.yaml

kind: Prometheus
apiVersion: monitoring.coreos.com/v1
metadata:
  name: prometheus
  namespace: prometheus
spec: {}

# appley一下：
$ kubectl apply -f prometheus.yaml

查看一下：

$ kubectl -n prometheus get all
NAME                                       READY   STATUS              RESTARTS   AGE
pod/prometheus-operator-5748cc95dd-wgkhn   1/1     Running             0          31h
pod/prometheus-prometheus-0                0/3     ContainerCreating   0          4m11s


NAME                          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/prometheus-operated   ClusterIP   None         <none>        9090/TCP   4m11s
service/prometheus-operator   ClusterIP   None         <none>        8080/TCP   31h


NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/prometheus-operator   1/1     1            1           31h

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/prometheus-operator-5748cc95dd   1         1         1       31h

NAME                                     READY   AGE
statefulset.apps/prometheus-prometheus   0/1     4m11s

# 我们看到operator 创建了一个statefulSet 为prometheus，查看一下对应的pod
$ kubectl -n prometheus get po
NAME                                   READY   STATUS              RESTARTS   AGE
prometheus-operator-5748cc95dd-wgkhn   1/1     Running             0          31h
prometheus-prometheus-0                0/3     ContainerCreating   0          3m11s

# 查看一下描述信息
$ kubectl -n prometheus describe po prometheus-prometheus-0
...
Events:
  Type    Reason     Age   From                          Message
  ----    ------     ----  ----                          -------
  Normal  Scheduled  56s   default-scheduler             Successfully assigned prometheus/prometheus-prometheus-0 to k8s03.test.awsbj.cn
  Normal  Pulling    55s   kubelet, k8s03.test.awsbj.cn  Pulling image "quay.io/prometheus/prometheus:v2.7.1"

# 还在拉取镜像，而且版本还是v2.7.1的，截止到目前官方最新版本是：v2.14.0，operator兼容的最高prometheus版本是：v2.10.0

# 稍等片刻即可：
$ kubectl -n prometheus get po
NAME                                   READY   STATUS    RESTARTS   AGE
prometheus-operator-5748cc95dd-wgkhn   1/1     Running   0          32h
prometheus-prometheus-0                3/3     Running   1          53m

我们创建prometheus实例的service：

$ cat prometheus-svc.yaml 
kind: Service
apiVersion: v1
metadata:
  name: prometheus
  namespace: prometheus
spec:
  selector:
    app: prometheus
    prometheus: prometheus
  ports:
  - name: http
    port: 9090
    targetPort: 9090
    protocol: TCP
  type: NodePort

# appley一下：
$ kubectl apply -f prometheus-svc.yaml 

# 查看具体的svc的nodeport
$ kubectl -n prometheus get svc
NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
prometheus            NodePort    10.100.56.198   <none>        9090:32766/TCP   4s
prometheus-operated   ClusterIP   None            <none>        9090/TCP         65m
prometheus-operator   ClusterIP   None            <none>        8080/TCP         32h

我们访问测试一下，看看具体的配置文件吧：

指定prometheus server的版本

默认创建的prometheus server版本太低了，我们更换成operator兼容列表中最高的prometheus server版本：

我们注意到在上面的：PrometheusSpec跟版本有关的总共有三个：

Field	Description	Scheme	Required
version	Version of Prometheus to be deployed. 部署的prometheus server版本	string	false
tag	Tag of Prometheus container image to be deployed. Defaults to the value of version. Version is ignored if Tag is set. 部署的prometheus server版本，优先级高于version	string	false
image	Image if specified has precedence over baseImage, tag and sha combinations. Specifying the version is still necessary to ensure the Prometheus Operator knows what version of Prometheus is being configured. 当指定image时优先级高于baseImage，tag 和sha。但是仍然需要指定version来告诉operator是哪个版本的prometheus需要被配置	*string	false

所以呢：这里我们使用image+version的方式：

$ cat prometheus-image.yaml 
kind: Prometheus
apiVersion: monitoring.coreos.com/v1
metadata:
  name: prometheus
  namespace: prometheus
spec:
  version: v2.10.0
  image: quay.azk8s.cn/prometheus/prometheus:v2.10.0

# 应用
$ kubectl apply -f prometheus-image.yaml

注意：image中需要指定prometheus的版本v2.10.0*，否则就会拉取latest的版本;

这里的version只是告诉operator哪个prometheus版本需要被配置的，对于使用哪个版本是由image指定的。

version中的版本要和image中的版本一致，否则可能会出现未知的错误。

apply后，operator会根据我们定义的声明式yaml文件重新部署prometheus server，稍等片刻我们来验证一下：

$ kubectl -n prometheus get po prometheus-prometheus-0 -o yaml --export | grep image
Flag --export has been deprecated, This flag is deprecated and will be removed in future.
    image: quay.azk8s.cn/prometheus/prometheus:v2.10.0
    imagePullPolicy: IfNotPresent
    image: quay.azk8s.cn/coreos/prometheus-config-reloader:v0.34.0
    imagePullPolicy: IfNotPresent
    image: quay.azk8s.cn/coreos/configmap-reload:v0.0.1
    imagePullPolicy: IfNotPresent

我们发现连基础的镜像都是用了我们配置的image的镜像源：quay.azk8s.cn，所以呢，如果使用了自建的docker registry也需要提前拉取好这些镜像。

因为我们使用的Azure的镜像，它就是quay的镜像，所以默认都已经存在了，所以不需要手动再去拉取的。

所以推荐使用Azure中国镜像，使用方式请参考：docker.io gcr.io k8s.gcr.io quay.io 中国区加速

我们继续，我们看到prometheus-prometheus-0 这个statefulSet创建的pod中，另外两个关于重载prometheus配置的pod都已经配置了资源限制，只有prometheus server这个没有配置，显然不符合生产要求。

指定prometheus server的资源限制：

从上面的PrometheusSpec中我们看到有resource字段，内用和原声的kubernetes resource是一样的。

$ cat prometheus-image-resource.yaml 
kind: Prometheus
apiVersion: monitoring.coreos.com/v1
metadata:
  name: prometheus
  namespace: prometheus
spec:
  version: v2.10.0
  image: quay.azk8s.cn/prometheus/prometheus:v2.10.0
  resources:
    requests:
      memory: 400Mi
      cpu: 200m
    limits:
      memory: 512Mi
      cpu: 500m

# 应用一下：
$ kubectl apply -f prometheus-image-resource.yaml

检查一下：

$ kubectl -n prometheus get po prometheus-prometheus-0 -o yaml --export
...
spec:
  containers:
  - args:
    - --web.console.templates=/etc/prometheus/consoles
    - --web.console.libraries=/etc/prometheus/console_libraries
    - --config.file=/etc/prometheus/config_out/prometheus.env.yaml
    - --storage.tsdb.path=/prometheus
    - --storage.tsdb.retention.time=24h
    - --web.enable-lifecycle
    - --storage.tsdb.no-lockfile
    - --web.route-prefix=/
    image: quay.azk8s.cn/prometheus/prometheus:v2.10.0
    imagePullPolicy: IfNotPresent
  ...
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:
        cpu: 200m
        memory: 400Mi
...

配置prometheus server的存储以及数据保留期限

查看具体的api：

Field	Description	Scheme	Required
retention	Time duration Prometheus shall retain data for. Default is ‘24h’, and must match the regular expression `[0-9]+(ms	s	m
storage	Storage spec to specify how storage shall be used.	*StorageSpec	false

StorageSpec

StorageSpec defines the configured storage for a group Prometheus servers. If neither emptyDir nor volumeClaimTemplate is specified, then by default an EmptyDir will be used.

Field	Description	Scheme	Required
emptyDir	EmptyDirVolumeSource to be used by the Prometheus StatefulSets. If specified, used in place of any volumeClaimTemplate. More info: https://kubernetes.io/docs/concepts/storage/volumes/#emptydir	*v1.EmptyDirVolumeSource	false
volumeClaimTemplate	A PVC spec to be used by the Prometheus StatefulSets.	v1.PersistentVolumeClaim	false

volumeClaimTemplate 中和kubernetes statefulSet 中定义的是一样的。

参考

$ cat prometheus.yaml 
kind: Prometheus
apiVersion: monitoring.coreos.com/v1
metadata:
  name: prometheus
  namespace: prometheus
spec:
  version: v2.10.0
  image: quay.azk8s.cn/prometheus/prometheus:v2.10.0
  retention: 1d # 保留1天
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: ssd
        resources:
          requests:
            storage: 40Gi
  resources:
    requests:
      memory: 400Mi
      cpu: 200m
    limits:
      memory: 512Mi
      cpu: 500m

# 准备storageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2

由于我这里还没有storageClass的环境，所以不再测试，后期搭建一个Rook-Ceph测试一下。

配置prometheus server ingress，并配置basicAuth 用户名密码认证

Prometheus Operator参考

Nginx Ingress Basic Auth参考

由于默认的prometheus dashboard没有认证的，对于生产来说公网访问不安全，所以需要加上认证。

安装htpasswd命令：

$ sudo yum install httpd-tools -y

# 查看一下htpasswd命令帮助：
$ htpasswd --help
Usage:
4htpasswd [-cimBdpsDv] [-C cost] passwordfile username
4htpasswd -b[cmBdpsDv] [-C cost] passwordfile username password

4htpasswd -n[imBdps] [-C cost] username
4htpasswd -nb[mBdps] [-C cost] username password
 -c  Create a new file.
 -n  Don't update file; display results on stdout.
 -b  Use the password from the command line rather than prompting for it.
 -i  Read password from stdin without verification (for script usage).
 -m  Force MD5 encryption of the password (default).
 -B  Force bcrypt encryption of the password (very secure).
 -C  Set the computing time used for the bcrypt algorithm
     (higher is more secure but slower, default: 5, valid: 4 to 17).
 -d  Force CRYPT encryption of the password (8 chars max, insecure).
 -s  Force SHA encryption of the password (insecure).
 -p  Do not encrypt the password (plaintext, insecure).
 -D  Delete the specified user.
 -v  Verify password for the specified user.
On other systems than Windows and NetWare the '-p' flag will probably not work.
The SHA algorithm does not use a salt and is less secure than the MD5 algorithm.

创建nginx ingress所需的密码文件：

# auth 为创建的文件名，prometheus为用户名，然后输入两次一样的密码
$ htpasswd -c auth prometheus
New password: xxx
Re-type new password: xxx
Adding password for user prometheus

$ cat auth 
prometheus:$apr1$JhroPTnp$dzelC968rK0QkCnq1NnrY1

注意：文件名必须为auth，否则报503错误，用户名无所谓。

创建对应的kubernetes secret：

$ $ kubectl -n prometheus create secret generic basic-auth --from-file=auth

注意：secret必须要和ingress在同一个namespace，注意是ingress yaml文件的资源所在的namespace，不是nginx-ingress-controller所在的namespace。否则也会报503错误。

我这里ingress是准备在prometheus这个namespace下的，所以这个namespace也是在prometheus这个namespace。

创建prometheus的ingress，在prometheus这个namespace：我这里没有证书，所以把tls的部分注释掉了。

$ cat prometheus-ingress.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: prometheus
  namespace: prometheus
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    nginx.ingress.kubernetes.io/auth-realm: "Authentication Required Prometheus"
spec:
  #tls:
  #  - hosts:
  #    - prometheus.test.aws.test.com
  #    secretName: prometheus.test.aws.test.com
  rules:
  - host: prometheus.test.aws.test.com
    http:
      paths:
      - path: /
        backend:
          serviceName: prometheus
          servicePort: http

配置好域名解析访问就会提示输入我们设定的用户名密码。

配置ingress的prometheus path路径为 /prometheus

有的朋友可能会问到：

这里是直接把域名的/根路径给了prometheus，后期的alertmanager还得需要域名，不太好，建议用prometheus.test.aws.test.com/promethues 访问prometheus，prometheus.test.aws.test.com/alertmanager来访问alertmanager。

要实现这种：配置如下：

首先看一下prometheus的参数：

--web.external-url=<URL>   The URL under which Prometheus is externally reachable (for example, if Prometheus is served via a reverse proxy). Used for generating relative and absolute links back to Prometheus itself. If the URL has a path portion, it will be used to prefix all HTTP endpoints served by Prometheus. If omitted, relevant URL components will be derived automatically.

--web.route-prefix=<path>  Prefix for the internal routes of web endpoints. Defaults to path of --web.external-url.

直接在非docker环境启动测试OK，如下：

./prometheus --config.file=prometheus.yml --web.enable-lifecycle --web.enable-admin-api --web.route-prefix="/prometheus" --web.external-url="http://prometheus.test.aws.test.com/prometheus"

测试：发现都加上了我们的 /prometheus 这个path：

$ curl http://prometheus.test.aws.test.com
<a href="/prometheus">Found</a>.

$ curl http://prometheus.test.aws.test.com/prometheus
<a href="/prometheus/">Moved Permanently</a>.

$ curl http://prometheus.test.aws.test.com/prometheus -L
<a href="/prometheus/graph">Found</a>.

那么k8s环境中prometheus-operator如何配置呢？

查看一下：PrometheusSpec

Field	Description	Scheme	Required
externalUrl	The external URL the Prometheus instances will be available under. This is necessary to generate correct URLs. This is necessary if Prometheus is not served from root of a DNS name.	string	false
routePrefix	The route prefix Prometheus registers HTTP handlers for. This is useful, if using ExternalURL and a proxy is rewriting HTTP routes of a request, and the actual ExternalURL is still true, but the server serves requests under a different route prefix. For example for use with kubectl proxy.	string	false

$ cat prometheus-path.yaml 
kind: Prometheus
apiVersion: monitoring.coreos.com/v1
metadata:
  name: prometheus
  namespace: prometheus
spec:
  version: v2.10.0
  image: quay.azk8s.cn/prometheus/prometheus:v2.10.0
  retention: 1d
  externalUrl: http://prometheus.test.aws.microoak.cn/prometheus
  routePrefix: /prometheus
  resources:
    requests:
      memory: 400Mi
      cpu: 200m
    limits:
      memory: 512Mi
      cpu: 500m

# appley 一下
$ kubectl apply -f prometheus-path.yaml

# 记得把prometheus ingress中的path更换成
path: /prometheus

测试一下吧：

# curl -L 表示跟谁301/302.返回正常
$ curl -L http://prometheus.test.aws.test.com/prometheus -uprometheus -p 
Enter host password for user 'prometheus':
<!DOCTYPE html>
<html lang="en">
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
        <meta name="robots" content="noindex,nofollow">
        <title>Prometheus Time Series Collection and Processing Server</title>
        <link rel="shortcut icon" href="/prometheus/static/img/favicon.ico?v=d20e84d0fb64aff2f62a977adc8cfb656da4e286">
        <script src="/prometheus/static/vendor/js/jquery-3.3.1.min.js?v=d20e84d0fb64aff2f62a977adc8cfb656da4e286"></script>    
        <script src="/prometheus/static/vendor/js/popper.min.js?v=d20e84d0fb64aff2f62a977adc8cfb656da4e286"></script>
        <script src="/prometheus/static/vendor/bootstrap-4.3.1/js/bootstrap.min.js?v=d20e84d0fb64aff2f62a977adc8cfb656da4e286"></script>

注意：如果你使用的是Helm安装prometheus服务的，或者在k8s上手动安装prometheus，而不是通过prometheus-operator的，需要更改一些参数。

prometheus-operator只需要配置这两个参数即可，它自动的把下面的都配置好了。

需要更改如下：

• readinessProbe路径为：/prometheus/-/ready
• livenessProbe路径为：/prometheus/-/healthy
• prometheus sidecar用于发现配置更改重载prometheus的服务的参数：- –webhook-url=http://127.0.0.1:9090/prometheus/-/reload
• 如果你的prometheus也收集了本身，需要配置收集的路径为：/prometheus/metircs

好了，关于operator的Prometheus资源配置就到这里，如果有其他的配置，建议参考：PrometheusSpec

ServiceMonitor 配置

参考

我们首先部署一下我们要搜集的目标服务：注意是在default名称空间下：

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
    spec:
      containers:
      - name: example-app
        image: fabxc/instrumented_app
        ports:
        - name: web
          containerPort: 8080
---
kind: Service
apiVersion: v1
metadata:
  name: example-app
  labels:
    app: example-app
spec:
  selector:
    app: example-app
  ports:
  - name: web
    port: 8080

检查这个服务：

$ kubectl get svc,po
NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/example-app   ClusterIP   10.100.151.33   <none>        8080/TCP   45s
service/kubernetes    ClusterIP   10.100.0.1      <none>        443/TCP    7d2h
service/nginx         ClusterIP   10.100.54.82    <none>        80/TCP     5d23h

NAME                               READY   STATUS    RESTARTS   AGE
pod/example-app-66db748757-lmtsk   1/1     Running   0          45s
pod/example-app-66db748757-pzc6w   1/1     Running   0          45s
pod/example-app-66db748757-tchgl   1/1     Running   0          45s
pod/nginx-9cb7f8c7d-wk89s          1/1     Running   0          5d23h

# 查看其metrics：
$ curl 10.100.151.33:8080/metrics
# HELP codelab_api_http_requests_in_progress The current number of API HTTP requests in progress.
# TYPE codelab_api_http_requests_in_progress gauge
codelab_api_http_requests_in_progress 1
# HELP codelab_api_request_duration_seconds A histogram of the API HTTP request durations in seconds.
# TYPE codelab_api_request_duration_seconds histogram
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0001"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00015000000000000001"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00022500000000000002"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0003375"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00050625"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.000759375"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0011390624999999999"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0017085937499999998"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0025628906249999996"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0038443359374999994"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00576650390625"} 0

好了，让我们配置ServiceMonitor，来让prometheus搜集吧。

老规矩，先看ServiceMonitor的api吧，我也不知道怎么配置。

ServiceMonitor

ServiceMonitor defines monitoring for a set of services.

Field	Description	Scheme	Required
metadata	Standard object’s metadata. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#metadata	metav1.ObjectMeta	false
spec	Specification of desired Service selection for target discrovery by Prometheus.	ServiceMonitorSpec	true

ServiceMonitorSpec

ServiceMonitorSpec contains specification parameters for a ServiceMonitor.

Field	Description	Scheme	Required
jobLabel	The label to use to retrieve the job name from.	string	false
targetLabels	TargetLabels transfers labels on the Kubernetes Service onto the target.	[]string	false
podTargetLabels	PodTargetLabels transfers labels on the Kubernetes Pod onto the target.	[]string	false
endpoints	A list of endpoints allowed as part of this ServiceMonitor.	[]Endpoint	true
selector	Selector to select Endpoints objects.	metav1.LabelSelector	true
namespaceSelector	Selector to select which namespaces the Endpoints objects are discovered from.	NamespaceSelector	false
sampleLimit	SampleLimit defines per-scrape limit on number of scraped samples that will be accepted.	uint64	false

Endpoint

Endpoint defines a scrapeable endpoint serving Prometheus metrics.

Field	Description	Scheme	Required
port	Name of the service port this endpoint refers to. Mutually exclusive with targetPort.	string	false
targetPort	Name or number of the target port of the endpoint. Mutually exclusive with port.	*intstr.IntOrString	false
path	HTTP path to scrape for metrics.	string	false
scheme	HTTP scheme to use for scraping.	string	false
params	Optional HTTP URL parameters	map[string][]string	false
interval	Interval at which metrics should be scraped	string	false
scrapeTimeout	Timeout after which the scrape is ended	string	false
tlsConfig	TLS configuration to use when scraping the endpoint	*TLSConfig	false
bearerTokenFile	File to read bearer token for scraping targets.	string	false
bearerTokenSecret	Secret to mount to read bearer token for scraping targets. The secret needs to be in the same namespace as the service monitor and accessible by the Prometheus Operator.	v1.SecretKeySelector	false
honorLabels	HonorLabels chooses the metric’s labels on collisions with target labels.	bool	false
honorTimestamps	HonorTimestamps controls whether Prometheus respects the timestamps present in scraped data.	*bool	false
basicAuth	BasicAuth allow an endpoint to authenticate over basic authentication More info: https://prometheus.io/docs/operating/configuration/#endpoints	*BasicAuth	false
metricRelabelings	MetricRelabelConfigs to apply to samples before ingestion.	[]*RelabelConfig	false
relabelings	RelabelConfigs to apply to samples before scraping. More info: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config	[]*RelabelConfig	false
proxyUrl	ProxyURL eg http://proxyserver:2195 Directs scrapes to proxy through this endpoint.	*string	false

NamespaceSelector

NamespaceSelector is a selector for selecting either all namespaces or a list of namespaces.

Field	Description	Scheme	Required
any	Boolean describing whether all namespaces are selected in contrast to a list restricting them.	bool	false
matchNames	List of namespace names.	[]string	false

有两个必须要提供的：

ServiceMonitor.spec.endpoints

ServiceMonitor.spec.selector

我们来定义serviceMonitor：

$ cat example-app-ServiceMonitor.yaml 
kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
  name: example-app
  namespace: prometheus # 我把ServiceMonitor资源放到了prometheus 空间下了
  labels: # 需要配置自身的标签，以便prometheus能够通过标签找到自己
    team: frontend
spec:
  namespaceSelector: # 而目标的service在default 空间下，需要指定；如果不指定那么就默认找自己所在的空间
    matchNames:
    - default
  selector:
    matchLabels:
      app: example-app # 需要搜集对象service的labels，通过service的lables找到server对应的endpoint。
  endpoints:
  - port: web # 需要搜集对象service中对应的端口名称

# 应用一下：
$ kubectl apply -f example-app-ServiceMonitor.yaml

如果希望ServiceMonitor可以关联任何空间下的标签，可以通过下面的方式定义：

spec:
  namespaceSelector:
    any: true

如果监控的目标对应中有basicAuth认证的话，需要配置如下：

kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
  name: example-app
  namespace: prometheus
  labels:
    team: frontend
spec:
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      app: example-app
  endpoints:
  - port: web
    basicAuth:
      password:
        name: basic-auth
        key: password
      username:
        name: basic-auth
        key: username

# basicAuth 关联了目标对象的secret，需要手动创建
apiVersion: v1
kind: Secret
metadata:
  name: basic-auth
data:
  password: cHJvbWV0aGV1c0AxMjM=
  username: cHJvbWV0aGV1cw==
type: Opaque

检查一下：

$ kubectl -n prometheus get servicemonitors.monitoring.coreos.com 
NAME          AGE
example-app   7m58s

OK了，刷新一下prometheus dashboard看一下吧。呀，啥也没有呢，没生效？

其实是这样的：

ServiceMonitor定义了目标收集相关的，比如端口啊，哪个service啊等，但是还没有指定是由哪个prometheus去搜集呢？

Prometheus,Alertmanager,ServiceMonitor等自定义的资源在一个k8s集群中可能都有多个，比如定义了两台prometheus server，并且定义了多个ServiceMonitor，那么哪个prometheus通过哪个ServiceMonitor去搜集呢？是不是？

所以还需要更改之前定义的Prometheus自定义资源，因为prometheus server是通过kubernetes_sd_configs自动发现的，所以需要给prometheus server权限去发现：所以呢，首先定义prometheus server所需的RBAC：

参考

$ wget -c "https://github.com/coreos/prometheus-operator/raw/master/example/rbac/prometheus/prometheus-cluster-role-binding.yaml"
$ wget -c "https://github.com/coreos/prometheus-operator/raw/master/example/rbac/prometheus/prometheus-cluster-role.yaml"
$ wget -c "https://github.com/coreos/prometheus-operator/raw/master/example/rbac/prometheus/prometheus-service-account.yaml"

# 因为我把prometheus安装到了prometheus空间中，所以需要修改上面的namespace：这两个文件需要修改
prometheus-service-account.yaml 和 prometheus-cluster-role-binding.yaml

# 然后appley一下：
$ kubectl apply -f ./

serviceAccount 配置OK了，prometheus怎么用呢？以及怎么选择对应的serviceMonitor呢？当然是通过强大的labels选择器了啊：

$ cat prometheus-path.yaml 
kind: Prometheus
apiVersion: monitoring.coreos.com/v1
metadata:
  name: prometheus
  namespace: prometheus
spec:
  version: v2.10.0
  image: quay.azk8s.cn/prometheus/prometheus:v2.10.0
  retention: 1d
  externalUrl: http://prometheus.test.aws.microoak.cn/prometheus
  routePrefix: /prometheus
  serviceAccountName: prometheus # 指定需要serviceAccount
  serviceMonitorSelector: # 通过label找到具体的serviceMonitor
    matchLabels:
      team: frontend
  resources:
    requests:
      memory: 400Mi
      cpu: 200m
    limits:
      memory: 512Mi
      cpu: 500m

# 再次应用一下：
$ kubectl apply -f prometheus-path.yaml

好了，再次刷新一下prometheus dashboard：

target：

UI 上查看具体的prometheus 配置：

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 30s
  external_labels:
    prometheus: prometheus/prometheus
    prometheus_replica: prometheus-prometheus-0
alerting:
  alert_relabel_configs:
  - separator: ;
    regex: prometheus_replica
    replacement: $1
    action: labeldrop
rule_files:
- /etc/prometheus/rules/prometheus-prometheus-rulefiles-0/*.yaml
scrape_configs:
- job_name: prometheus/example-app/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - default
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: example-app
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: web
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: web
    action: replace

command-line flags 截图：

创建Alertmanager实例：

老规矩先看其api：

Alertmanager

Alertmanager describes an Alertmanager cluster.

Field	Description	Scheme	Required
metadata	Standard object’s metadata. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#metadata	metav1.ObjectMeta	false
spec	Specification of the desired behavior of the Alertmanager cluster. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#spec-and-status	AlertmanagerSpec	true
status	Most recent observed status of the Alertmanager cluster. Read-only. Not included when requesting from the apiserver, only from the Prometheus Operator API itself. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#spec-and-status	*AlertmanagerStatus	false

AlertmanagerSpec

AlertmanagerSpec is a specification of the desired behavior of the Alertmanager cluster. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#spec-and-status

Field	Description	Scheme	Required
podMetadata	Standard object’s metadata. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#metadata Metadata Labels and Annotations gets propagated to the prometheus pods.	*metav1.ObjectMeta	false
image	Image if specified has precedence over baseImage, tag and sha combinations. Specifying the version is still necessary to ensure the Prometheus Operator knows what version of Alertmanager is being configured.	*string	false
version	Version the cluster should be on.	string	false
tag	Tag of Alertmanager container image to be deployed. Defaults to the value of version. Version is ignored if Tag is set.	string	false
sha	SHA of Alertmanager container image to be deployed. Defaults to the value of version. Similar to a tag, but the SHA explicitly deploys an immutable container image. Version and Tag are ignored if SHA is set.	string	false
baseImage	Base image that is used to deploy pods, without tag.	string	false
imagePullSecrets	An optional list of references to secrets in the same namespace to use for pulling prometheus and alertmanager images from registries see http://kubernetes.io/docs/user-guide/images#specifying-imagepullsecrets-on-a-pod	[]v1.LocalObjectReference	false
secrets	Secrets is a list of Secrets in the same namespace as the Alertmanager object, which shall be mounted into the Alertmanager Pods. The Secrets are mounted into /etc/alertmanager/secrets/.	[]string	false
configMaps	ConfigMaps is a list of ConfigMaps in the same namespace as the Alertmanager object, which shall be mounted into the Alertmanager Pods. The ConfigMaps are mounted into /etc/alertmanager/configmaps/.	[]string	false
configSecret	ConfigSecret is the name of a Kubernetes Secret in the same namespace as the Alertmanager object, which contains configuration for this Alertmanager instance. Defaults to ‘alertmanager-‘ The secret is mounted into /etc/alertmanager/config.	string	false
logLevel	Log level for Alertmanager to be configured with.	string	false
logFormat	Log format for Alertmanager to be configured with.	string	false
replicas	Size is the expected size of the alertmanager cluster. The controller will eventually make the size of the running cluster equal to the expected size.	*int32	false
retention	Time duration Alertmanager shall retain data for. Default is ‘120h’, and must match the regular expression `[0-9]+(ms	s	m
storage	Storage is the definition of how storage will be used by the Alertmanager instances.	*StorageSpec	false
volumes	Volumes allows configuration of additional volumes on the output StatefulSet definition. Volumes specified will be appended to other volumes that are generated as a result of StorageSpec objects.	[]v1.Volume	false
volumeMounts	VolumeMounts allows configuration of additional VolumeMounts on the output StatefulSet definition. VolumeMounts specified will be appended to other VolumeMounts in the alertmanager container, that are generated as a result of StorageSpec objects.	[]v1.VolumeMount	false
externalUrl	The external URL the Alertmanager instances will be available under. This is necessary to generate correct URLs. This is necessary if Alertmanager is not served from root of a DNS name.	string	false
routePrefix	The route prefix Alertmanager registers HTTP handlers for. This is useful, if using ExternalURL and a proxy is rewriting HTTP routes of a request, and the actual ExternalURL is still true, but the server serves requests under a different route prefix. For example for use with kubectl proxy.	string	false
paused	If set to true all actions on the underlaying managed objects are not goint to be performed, except for delete actions.	bool	false
nodeSelector	Define which Nodes the Pods are scheduled on.	map[string]string	false
resources	Define resources requests and limits for single Pods.	v1.ResourceRequirements	false
affinity	If specified, the pod’s scheduling constraints.	*v1.Affinity	false
tolerations	If specified, the pod’s tolerations.	[]v1.Toleration	false
securityContext	SecurityContext holds pod-level security attributes and common container settings. This defaults to the default PodSecurityContext.	*v1.PodSecurityContext	false
serviceAccountName	ServiceAccountName is the name of the ServiceAccount to use to run the Prometheus Pods.	string	false
listenLocal	ListenLocal makes the Alertmanager server listen on loopback, so that it does not bind against the Pod IP. Note this is only for the Alertmanager UI, not the gossip communication.	bool	false
containers	Containers allows injecting additional containers. This is meant to allow adding an authentication proxy to an Alertmanager pod.	[]v1.Container	false
initContainers	InitContainers allows adding initContainers to the pod definition. Those can be used to e.g. fetch secrets for injection into the Alertmanager configuration from external sources. Any errors during the execution of an initContainer will lead to a restart of the Pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ Using initContainers for any use case other then secret fetching is entirely outside the scope of what the maintainers will support and by doing so, you accept that this behaviour may break at any time without notice.	[]v1.Container	false
priorityClassName	Priority class assigned to the Pods	string	false
additionalPeers	AdditionalPeers allows injecting a set of additional Alertmanagers to peer with to form a highly available cluster.	[]string	false
portName	Port name used for the pods and governing service. This defaults to web	string	false

好吧，既然没有必须的那就参考上面配置的Prometheus，指定如下配置：

• replicas为3，创建以3台alertmanager组成的集群
• 更改image资源为Azure中国的
• 配置一下resource限制
• 同时采用类似prometheus 的ingress path为 /alertmanager
• 配置数据保留期限

$ cat prometheus-alertmanager.yaml 
kind: Alertmanager
apiVersion: monitoring.coreos.com/v1
metadata:
  name: alertmanager
  namespace: prometheus
spec:
  replicas: 3
  image: quay.azk8s.cn/prometheus/alertmanager:v0.17.0
  version: v0.17.0
  retention: 24h # 注意这里最大的单位是h，没有d。
  externalUrl: http://prometheus.test.aws.microoak.cn/alertmanager
  routePrefix: /alertmanager
  resources:
    requests:
      memory: 256Mi
      cpu: 100m
    limits:
      memory: 256Mi
      cpu: 100m

# apply一下
$ kubectl apply -f prometheus-alertmanager.yaml

# 检查一下：
$ kubectl -n prometheus get po
NAME                                   READY   STATUS              RESTARTS   AGE
alertmanager-alertmanager-0            0/2     ContainerCreating   0          109s
alertmanager-alertmanager-1            0/2     ContainerCreating   0          109s
alertmanager-alertmanager-2            0/2     ContainerCreating   0          109s
prometheus-operator-5748cc95dd-wgkhn   1/1     Running             0          2d11h
prometheus-prometheus-0                3/3     Running             1          3h47m

# 发现需要创建配置文件：alertmanager-alertmanager 而且是secret
$ kubectl -n prometheus describe po alertmanager-alertmanager-0
Events:
  Type     Reason       Age                From                          Message
  ----     ------       ----               ----                          -------
  Normal   Scheduled    97s                default-scheduler             Successfully assigned prometheus/alertmanager-alertmanager-1 to k8s03.test.awsbj.cn
  Warning  FailedMount  33s (x8 over 96s)  kubelet, k8s03.test.awsbj.cn  MountVolume.SetUp failed for volume "config-volume" : secret "alertmanager-alertmanager" not found

准备alertmanager的配置文件：名为：alertmanager-<Alertmanager> 格式的secret，其中<Alertmanager>为创建Alertmanager资源对象的名称：

这里配置了email以及企业微信的告警：

$ cat alertmanager.yaml
global:
  resolve_timeout: 5m
  # smtp
  smtp_from: alert@xxx.xx
  smtp_smarthost: smtphm.qiye.163.com:465
  smtp_hello: alert@xxx.xx
  smtp_auth_username: alert@xxx.xx
  smtp_auth_password: xxx
  smtp_require_tls: false
  # wechat
  wechat_api_secret: PxxxxxxxxxxxxxxxxxxxxSc
  wechat_api_corp_id: wxxxxxxxxxxx7

route:
  group_by:
  - alertname
  - instance
  group_wait: 30s
  group_interval: 1m
  repeat_interval: 5m
  receiver: 'email-receiver'
  routes:
  - receiver: "wechat-receiver"
    match_re:
      job: pushgw|grafana

receivers:
- name: 'email-receiver'
  email_configs:
    - to: xxx@xxx.xx
      send_resolved: true

- name: 'wechat-receiver'
  wechat_configs:
    - send_resolved: true  
      agent_id: 1xxxx3
      to_user: '@all'

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'service', 'instance']

创建secret，注意是在和Alertmanager所在的namespace，我这里是prometheus：

$ kubectl -n prometheus create secret generic alertmanager-alertmanager --from-file=alertmanager.yaml

注意alertmanager的配置文件名称必须是alertmanager.yaml，为啥呢？

$ kubectl -n prometheus get statefulsets.apps alertmanager-alertmanager -o yaml
apiVersion: apps/v1
kind: StatefulSet
\......
    spec:
      containers:
      - args:
        - --config.file=/etc/alertmanager/config/alertmanager.yaml # 这里指定的就是alertmanager.yaml
        - --cluster.listen-address=[$(POD_IP)]:9094
        - --storage.path=/alertmanager
        - --data.retention=24h
        - --web.listen-address=:9093
        - --web.external-url=http://prometheus.test.aws.microoak.cn/alertmanager
        - --web.route-prefix=/alertmanager
        - --cluster.peer=alertmanager-alertmanager-0.alertmanager-operated.prometheus.svc:9094
        - --cluster.peer=alertmanager-alertmanager-1.alertmanager-operated.prometheus.svc:9094
        - --cluster.peer=alertmanager-alertmanager-2.alertmanager-operated.prometheus.svc:9094
        image: quay.azk8s.cn/prometheus/alertmanager:v0.17.0
        imagePullPolicy: IfNotPresent
        name: alertmanager
        \......
        volumeMounts:
        - mountPath: /etc/alertmanager/config
          name: config-volume
        - mountPath: /alertmanager
          name: alertmanager-alertmanager-db
      - args:
        - -webhook-url=http://localhost:9093/-/reload
        - -volume-dir=/etc/alertmanager/config
        image: quay.azk8s.cn/coreos/configmap-reload:v0.0.1
        imagePullPolicy: IfNotPresent
        name: config-reloader
       \......
        volumeMounts:
        - mountPath: /etc/alertmanager/config
          name: config-volume
          readOnly: true
     \......
      volumes:
      - name: config-volume
        secret: # secret类型的
          defaultMode: 420
          secretName: alertmanager-alertmanager # secret的名称
      - emptyDir: {}
        name: alertmanager-alertmanager-db

创建alertmanager的service：

$ cat alertmanager-svc.yaml 
kind: Service
apiVersion: v1
metadata:
  name: alertmanager
  namespace: prometheus
spec:
  selector:
    app: alertmanager
    alertmanager: alertmanager
  ports:
  - name: web
    port: 9093
    targetPort: 9093
    protocol: TCP
  type: ClusterIP

# apply
$ kubectl apply -f alertmanager-svc.yaml

创建alertmanager的ingress：用户名密码直接复用prometheus的

$ cat alertmanager-ingress.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: alertmanager
  namespace: prometheus
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    nginx.ingress.kubernetes.io/auth-realm: "Authentication Required Prometheus"
spec:
  #tls:
  #  - hosts:
  #    - prometheus.test.aws.test.com
  #    secretName: prometheus.test.aws.test.com
  rules:
  - host: prometheus.test.aws.test.com
    http:
      paths:
      - path: /alertmanager
        backend:
          serviceName: alertmanager
          servicePort: web

# apply
$ kubectl apply -f alertmanager-ingress.yaml

好了，访问测试一下：http://prometheus.test.aws.test.com

$ curl -L http://prometheus.test.aws.test.com/alertmanager -uprometheus -p
Enter host password for user 'prometheus':
<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
        <title>Alertmanager</title>
    </head>
    <body>
    ......

下图可以看到三台alertmanager已经组成了集群。

好了，alertmanager完事了，如果告知前面创建的prometheus呢？

还是查看一下PrometheusSpec：

Field	Description	Scheme	Required
alerting	Define details regarding alerting.	*AlertingSpec	false

AlertingSpec

AlertingSpec defines parameters for alerting configuration of Prometheus servers.

Field	Description	Scheme	Required
alertmanagers	AlertmanagerEndpoints Prometheus should fire alerts against.	[]AlertmanagerEndpoints	true

AlertmanagerEndpoints

AlertmanagerEndpoints defines a selection of a single Endpoints object containing alertmanager IPs to fire alerts against.

Field	Description	Scheme	Required
namespace	Namespace of Endpoints object.	string	true
name	Name of Endpoints object in Namespace.	string	true
port	Port the Alertmanager API is exposed on.	intstr.IntOrString	true
scheme	Scheme to use when firing alerts.	string	false
pathPrefix	Prefix for the HTTP path alerts are pushed to.	string	false
tlsConfig	TLS Config to use for alertmanager connection.	*TLSConfig	false
bearerTokenFile	BearerTokenFile to read from filesystem to use when authenticating to Alertmanager.	string	false

Prometheus定义如下：

$ cat prometheus-path.yaml 
kind: Prometheus
apiVersion: monitoring.coreos.com/v1
metadata:
  name: prometheus
  namespace: prometheus
spec:
  version: v2.10.0
  image: quay.azk8s.cn/prometheus/prometheus:v2.10.0
  retention: 1d
  externalUrl: http://prometheus.test.aws.microoak.cn/prometheus
  routePrefix: /prometheus
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  alerting: # 这里定义alerting
    alertmanagers:
    - namespace: prometheus # alertmanager endpoint对象所在namespace
      name: alertmanager # alertmanager endpoint名称，等同于service名称
      port: web # alertmanager端口
      pathPrefix: /alertmanager # alertmanager path的前缀，因为我们定义了/alertmanager
  resources:
    requests:
      memory: 400Mi
      cpu: 200m
    limits:
      memory: 512Mi
      cpu: 500m

注意：

如果你没有配置：routePrefix和externalUrl，那么这里也不需要配置pathPrefix

如果你配置了如上的：routePrefix和externalUrl，那么pathPrefix需要配置成和routePrefix一致：

pathPrefix: /alertmanager：

否则prometheus去通知alertmanager的时候会报404错误：

level=error ts=2019-11-20T03:01:24.934Z caller=notifier.go:487 component=notifier alertmanager=http://10.101.217.208:9093/api/v1/alerts count=0 msg=”Error sending alert” err=”bad response status 404 Not Found”
level=error ts=2019-11-20T03:01:24.934Z caller=notifier.go:487 component=notifier alertmanager=http://10.101.253.80:9093/api/v1/alerts count=0 msg=”Error sending alert” err=”bad response status 404 Not Found”
level=error ts=2019-11-20T03:01:24.934Z caller=notifier.go:487 component=notifier alertmanager=http://10.101.253.81:9093/api/v1/alerts count=0 msg=”Error sending alert” err=”bad response status 404 Not Found”

检查一下prometheus dashboard 中alerting的配置吧：

alerting:
  alert_relabel_configs:
  - separator: ;
    regex: prometheus_replica
    replacement: $1
    action: labeldrop
  alertmanagers:
  - kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names:
        - prometheus
    scheme: http
    path_prefix: /alertmanager
    timeout: 10s
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: alertmanager
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      separator: ;
      regex: web
      replacement: $1
      action: keep

查看prometheus是否发现了alertmanager实例呢：prometheus dashboard—Status—Runtime & BUild Information：

好了alertmanager 实例已经创建了，现在可以创建告警以及规则了。

通过PrometheusRule 创建rule，以及alert告警

老规矩，查看api：

PrometheusRule

PrometheusRule defines alerting rules for a Prometheus instance

Field	Description	Scheme	Required
metadata	Standard object’s metadata. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#metadata	metav1.ObjectMeta	false
spec	Specification of desired alerting rule definitions for Prometheus.	PrometheusRuleSpec	true

PrometheusRuleSpec 这里的定义是跟prometheus中定义rules，alert是一样的。具体可以alerting-rules，rules

PrometheusRuleSpec contains specification parameters for a Rule.

Field	Description	Scheme	Required
groups	Content of Prometheus rule file	[]RuleGroup	false

RuleGroup

RuleGroup is a list of sequentially evaluated recording and alerting rules.

Field	Description	Scheme	Required
name		string	true
interval		string	false
rules		[]Rule	true

Rule

Rule describes an alerting or recording rule.

Field	Description	Scheme	Required
record		string	false
alert		string	false
expr		intstr.IntOrString	true
for		string	false
labels		map[string]string	false
annotations		map[string]string	false

定义prometheusrule如下：

$ cat prometheusrule.yaml 
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: rules-alerts
  namespace: prometheus
  labels: # 需要定义labels，方便prometheus进行筛选。
    prometheus: prometheus
    role: rules-alerts
spec:
  groups:
  - name: alerting-group
    interval: 13s
    rules:
    - alert: InstanceDown
      expr: up == 0
      for: 1m
      labels:
        severity: error
      annotations:
        summary: "Instance {{ $labels.instance }} down"
        description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 3 minutes."
    - alert: WatchDog
      expr: vector(1)
      for: 10m
      labels:
        severity: info
      annotations:
        summary: "watchdog service every 10m send message."
  - name: rules-group
    interval: 30s
    rules:
    - record: http_requests_total:avg_rate1m
      expr: avg(rate(http_requests_total[1m])) by (instance,job,method,service,code)

# apply一下
$ kubectl apply -f prometheusrule.yaml

定义完了prometheusrule其实还不能用：类似与serviceMonitor、Alertmanager都需要在Prometheus资源中通过lables进行筛选prometheus需要的。

$ cat prometheus-path.yaml 
kind: Prometheus
apiVersion: monitoring.coreos.com/v1
metadata:
  name: prometheus
  namespace: prometheus
spec:
  version: v2.10.0
  image: quay.azk8s.cn/prometheus/prometheus:v2.10.0
  retention: 1d
  externalUrl: http://prometheus.test.aws.microoak.cn/prometheus
  routePrefix: /prometheus
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  alerting:
    alertmanagers:
    - namespace: prometheus
      name: alertmanager
      port: web
      pathPrefix: /alertmanager
  ruleSelector: # 通过labels筛选prometheusRule
    matchLabels:
      prometheus: prometheus
      role: rules-alerts
  resources:
    requests:
      memory: 400Mi
      cpu: 200m
    limits:
      memory: 512Mi
      cpu: 500m
      
# apply一下
$ kubectl apply -f prometheus-path.yaml

等待prometheus的自动reload，然后检查如下：

查询一下定义的rule：

查看alert：

等待10分钟再次查看：

在Alertmanager上查看：

查看邮件通知：

点击上面的Source能够跳转到Prometheus：

http://prometheus.test.aws.test.com/prometheus/graph?g0.expr=vector%281%29&g0.tab=1

总结：

看到上面的Prometheus、Alertmanager、ServiceMonitor、PrometheusRule、PodMonitor，以及其中的依赖关系，是不是觉得很懵逼呢？看个图就明白了很多：

Prometheus、Alertmanager、ServiceMonitor、PrometheusRule、PodMonitor在一个Kubernetes集群中都可能会有多个。

Alertmanager定义alertmanager server，负责等待prometheus server推送过来的告警，然后根据告警规则通知最终用户。

ServiceMonitor、PodMonitor都是定义要搜集的目标，以及怎么搜集，都是通过prometheus的kubernetes自动发现配置的，所以需要为prometheus配置RBAC权限，让它有权限去发现对应的目标。

然后就是Prometheus去部署一个prometheus server，然后通过标签选择要搜集的对象（ServiceMonitor、PodMonitor）、告警和规则（PrometheusRule）、以及当发生告警时要通知那些alertmanager（Alertmanager）。

所有的这些操作都是通过kubernetes的声明式API配置的，operator定义了如上几个CRD，用户定义期望的值（配置的这些的CRD），operator负责根据Prometheus、Alertmanager部署prometheus server，alertmanager server，然后根据ServiceMonitor、PrometheusRule、PodMonitor 来生成prometheus的配置文件，prometheus pod内的sidecar负责重载配置，最终尽可能的达到用户的期望值。

移除Prometheus Operator

我们以手动的方式测试完prometheus operator，了解其中的原理后，我们需要将其卸载掉，因为我们后续会使用Helm的方式来安装Prometheus-Operator（后面会有博文去介绍，欢迎期待），这样会帮助我们安装kube-prometheus、监控Kubernetes集群所需的Manifests，以及对应的Grafana和Dashboard，这样会大大的解放我们双手。

移除的方法参考GitHub

要移除prometheus operator首先要移除在每个namespace下面的定义的CRD（5个），operator会自动的的关闭prometheus和alertmanager pod以及其关联的ConfigMaps。

$ for n in $(kubectl get namespaces -o jsonpath={..metadata.name}); do
  kubectl delete --all --namespace=$n prometheus,servicemonitor,podmonitor,alertmanager
done

然后再移除operator本身：

$ kubectl delete -f bundle.yaml

清理operator自动在每个namespace下面创建的service，以及5个CRD：

for n in $(kubectl get namespaces -o jsonpath={..metadata.name}); do
  kubectl delete --ignore-not-found --namespace=$n service prometheus-operated alertmanager-operated
done

kubectl delete --ignore-not-found customresourcedefinitions \
  prometheuses.monitoring.coreos.com \
  servicemonitors.monitoring.coreos.com \
  podmonitors.monitoring.coreos.com \
  alertmanagers.monitoring.coreos.com \
  prometheusrules.monitoring.coreos.com

参考：

https://coreos.com/blog/the-prometheus-operator.html

https://github.com/coreos/prometheus-operator/blob/master/Documentation/troubleshooting.md

https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md # 重要

https://zhangguanzhang.github.io/2018/10/12/prometheus-operator/

https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/

https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/

---

本文到这里就结束了，欢迎期待后面的文章。您可以关注下方的公众号二维码，在第一时间查看新文章。

---