90 lines
3.6 KiB
Markdown
90 lines
3.6 KiB
Markdown
# kube-prometheus
|
|
|
|
This repository collects Kubernetes manifests, dashboards, and alerting rules
|
|
combined with documentation and scripts to deploy them to get a full cluster
|
|
monitoring setup working.
|
|
|
|
## Prerequisites
|
|
|
|
First, you need a running Kubernetes cluster. If you don't have one, follow
|
|
the instructions of [bootkube](https://github.com/kubernetes-incubator/bootkube)
|
|
or [minikube](https://github.com/kubernetes/minikube).
|
|
|
|
etcd is an important component of a working Kubernetes cluster, but it's deployed
|
|
outside of it. The monitoring setup below assumes that it is made visible from
|
|
within the cluster through a headless Kubernetes service.
|
|
An example for bootkube's multi-vagrant setup is [here](/manifests/etcd/etcd-bootkube-vagrant-multi.yaml).
|
|
|
|
## Monitoring Kubernetes
|
|
|
|
The manifests used here use the [Prometheus controller](https://github.com/coreos/kube-prometheus-controller),
|
|
which manages Prometheus servers and their configuration in your cluster. To install the
|
|
controller, the [node_exporter](https://github.com/prometheus/node_exporter),
|
|
[Grafana](https://grafana.org) including default dashboards, and the Prometheus server, run:
|
|
|
|
```bash
|
|
export KUBECONFIG=<path> # defaults to "~/.kube/config"
|
|
hack/cluster-monitoring/deploy
|
|
```
|
|
|
|
After all pods are ready, you can reach:
|
|
|
|
* Prometheus UI on node port `30900`
|
|
* Grafana on node port `30902`
|
|
|
|
To tear it all down again, run:
|
|
|
|
```bash
|
|
hack/cluster-monitoring/teardown
|
|
```
|
|
|
|
*All services in the manifest still contain the `prometheus.io/scrape = true` annotations. It is not
|
|
used by the Prometheus controller. They remain for convential deployments as in
|
|
[this example configuration](https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml).*
|
|
|
|
## Monitoring custom services
|
|
|
|
The example manifests in [/manifests/examples/example-app](/manifests/examples/example-app)
|
|
deploy a fake service into the `production` and `development` namespaces and define
|
|
a Prometheus server monitoring them.
|
|
|
|
```bash
|
|
kubectl --kubeconfig="$KUBECONFIG" create namespace production
|
|
kubectl --kubeconfig="$KUBECONFIG" create namespace development
|
|
hack/example-service-monitoring/deploy
|
|
```
|
|
|
|
After all pods are ready you can reach the Prometheus server monitoring your services
|
|
on node port `30100`.
|
|
|
|
Teardown:
|
|
|
|
```bash
|
|
hack/example-service-monitoring/teardown
|
|
```
|
|
|
|
## Dashboarding
|
|
|
|
The provided manifests deploy a Grafana instance serving dashboards provided via a ConfigMap.
|
|
To modify, delete, or add dashboards, the `grafana-dashboards` ConfigMap must be modified.
|
|
|
|
Currently, Grafana does not support serving dashboards from static files. Instead, the `grafana-watcher`
|
|
sidecar container aims to emulate the behavior, by keeping the Grafana database always in sync
|
|
with the provided ConfigMap. Hence, the Grafana pod is effectively stateless.
|
|
This allows managing dashboards via `git` etc. and easily deploying them via CD pipelines.
|
|
|
|
In the future, a separate Grafana controller should support gathering dashboards from multiple
|
|
ConfigMaps, which are selected by their labels.
|
|
Prometheus servers deployed by the Prometheus controller should be automatically added as
|
|
Grafana data sources.
|
|
|
|
## Roadmap
|
|
|
|
* Incorporate [Alertmanager controller](https://github.com/coreos/kube-alertmanager-controller)
|
|
* Grafana controller that dynamically discovers and deploys dashboards from ConfigMaps
|
|
* Collection of base alerting for cluster monitoring
|
|
* KPM/Helm packages to easily provide production-ready cluster-monitoring setup (essentially contents of `hack/cluster-monitoring`)
|
|
* Add meta-monitoring to default cluster monitoring setup
|
|
|
|
|