Add documentation for example service monitoring

README.md

@@ -22,23 +22,46 @@ which manages Prometheus servers and their configuration in your cluster. To ins
 controller, the [node_exporter](https://github.com/prometheus/node_exporter),
 [Grafana](https://grafana.org) including default dashboards, and the Prometheus server, run:
 
-```
-export KUBECONFIG=<path>   # defaults to "~/.kube/config"
-export KUBE_NAMESPACE=<ns> # defaults to "default"
+```bash
+export KUBECONFIG=<path>          # defaults to "~/.kube/config"
+export KUBE_NAMESPACE=<ns>        # defaults to "default"
 hack/cluster-monitoring/deploy
 ```
 
-To tear it all down again, run:
-
-```
-hack/cluster-monitoring/teardown
-```
-
 After all pods are ready, you can reach:
 
 * Prometheus UI on node port `30900`
 * Grafana on node port `30902`
 
+To tear it all down again, run:
+
+```bash
+hack/cluster-monitoring/teardown
+```
+
+__All services in the manifest still contain the `prometheus.io/scrape = true` annotations. It is not
+used by the Prometheus controller. They remain for convential deployments as in
+[this example configuration](https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml).__
+
 ## Monitoring custom services
 
-TODO
+The example manifests in [/manifests/examples/example-app](/manifests/examples/example-app)
+deploy a fake service into the `production` and `development` namespaces and define
+a Prometheus server monitoring them.
+
+```bash
+kubectl --kubeconfig="$KUBECONFIG" create namespace production
+kubectl --kubeconfig="$KUBECONFIG" create namespace development
+hack/cluster-monitoring/deploy
+```
+
+After all pods are ready you can reach the Prometheus server monitoring your services
+on node port `30100`.
+
+Teardown:
+
+```bash
+hack/cluster-monitoring/teardown
+```
+
+