Add documentation for example service monitoring

README.md

@@ -22,23 +22,46 @@ which manages Prometheus servers and their configuration in your cluster. To ins
 controller, the [node_exporter](https://github.com/prometheus/node_exporter),
 [Grafana](https://grafana.org) including default dashboards, and the Prometheus server, run:
 
-```
+```bash
 export KUBECONFIG=<path>          # defaults to "~/.kube/config"
 export KUBE_NAMESPACE=<ns>        # defaults to "default"
 hack/cluster-monitoring/deploy
 ```
 
-To tear it all down again, run:
-
-```
-hack/cluster-monitoring/teardown
-```
-
 After all pods are ready, you can reach:
 
 * Prometheus UI on node port `30900`
 * Grafana on node port `30902`
 
+To tear it all down again, run:
+
+```bash
+hack/cluster-monitoring/teardown
+```
+
+__All services in the manifest still contain the `prometheus.io/scrape = true` annotations. It is not
+used by the Prometheus controller. They remain for convential deployments as in
+[this example configuration](https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml).__
+
 ## Monitoring custom services
 
-TODO
+The example manifests in [/manifests/examples/example-app](/manifests/examples/example-app)
+deploy a fake service into the `production` and `development` namespaces and define
+a Prometheus server monitoring them.
+
+```bash
+kubectl --kubeconfig="$KUBECONFIG" create namespace production
+kubectl --kubeconfig="$KUBECONFIG" create namespace development
+hack/cluster-monitoring/deploy
+```
+
+After all pods are ready you can reach the Prometheus server monitoring your services
+on node port `30100`.
+
+Teardown:
+
+```bash
+hack/cluster-monitoring/teardown
+```
+
+

hack/example-service-monitoring/deploy

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+
+if [ -z "${KUBECONFIG}" ]; then
+    KUBECONFIG=~/.kube/config
+fi
+
+kubectl --kubeconfig="$KUBECONFIG" create -f manifests/examples/example-app

hack/example-service-monitoring/teardown

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+
+if [ -z "${KUBECONFIG}" ]; then
+    KUBECONFIG=~/.kube/config
+fi
+
+kubectl --kubeconfig="$KUBECONFIG" -n "$KUBE_NAMESPACE" delete -f manifests/examples/example-app

manifests/examples/example-app/prometheus-frontend-svc.yaml

@@ -2,6 +2,7 @@ apiVersion: v1
 kind: Service
 metadata:
   name: prometheus-frontend
+  namespace: default
 spec:
   type: NodePort
   ports:

manifests/examples/example-app/prometheus-frontend.yaml

@@ -2,6 +2,7 @@ apiVersion: prometheus.coreos.com/v1alpha1
 kind: Prometheus
 metadata:
   name: prometheus-frontend
+  namespace: default
   labels:
     prometheus: frontend
 spec:

manifests/examples/example-app/servicemonitor-frontend.yaml

@@ -2,6 +2,7 @@ apiVersion: prometheus.coreos.com/v1alpha1
 kind: ServiceMonitor
 metadata:
   name: frontend
+  namespace: default
   labels:
     team: frontend
 spec: