Addon: Use Hostprocess for Windows Exporter (#2048)

* Addon: Use Hostprocess for Windows Exporter #1627

This allows for Windows Exporter to be deployed and configured
dynamically without requiring to specify targets manually for each
Windows node

Documentation and example added

* Addon: Windows Exporter using Hostprocess

Fix some performance issues.
- Specify resource limits for windows exporter.
- Allow for scrape timeout and interval to be configured. Depending
on how many pods are running on a node it can take from 500ms to 15s
to scrape metrics from node. Default timeout is 10s.
- Allow for enabled collectors to be configured.
- Only enable collectors that are being used in rules and dashboards.

* Addon: Windows Exporter using Hostprocess

Fix formatting issues

* Fix formatting issues in windows-hostprocess addon

* Windows Addon: update doc to reflect both configs
- Hostprocess and static

* Windows Addon: update doc

---------

Co-authored-by: Dinesh Sharma <dinesh.sharma@health.telstra.com>

docs/windows.md

@@ -1,8 +1,27 @@
 # Windows
 
-The [Windows addon](../examples/windows.jsonnet) adds the dashboards and rules from [kubernetes-monitoring/kubernetes-mixin](https://github.com/kubernetes-monitoring/kubernetes-mixin#dashboards-for-windows-nodes).
+The [Windows hostprocess addon](../examples/windows-hostprocess.jsonnet) adds the dashboards and rules from [kubernetes-monitoring/kubernetes-mixin](https://github.com/kubernetes-monitoring/kubernetes-mixin#dashboards-for-windows-nodes).
 
-Currently, Docker based Windows does not support running with [windows_exporter](https://github.com/prometheus-community/windows_exporter) in a pod so this add on uses [additional scrape configuration](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/additional-scrape-config.md) to set up a static config to scrape the node ports where windows_exporter is configured.
+It also deploys [windows_exporter](https://github.com/prometheus-community/windows_exporter) as a [hostprocess pod](https://github.com/prometheus-community/windows_exporter/blob/master/kubernetes/kubernetes.md) as Kubernetes now supports HostProcess containers on Windows nodes (as of [v1.22](https://kubernetes.io/blog/2021/08/16/windows-hostprocess-containers/)). The cluster should be using containerd runtime.
+
+```
+local kp = (import 'kube-prometheus/main.libsonnet') +
+  (import 'kube-prometheus/addons/windows-hostprocess.libsonnet') +
+  {
+    values+:: {
+      windowsExporter+:: {
+        image: "ghcr.io/prometheus-community/windows-exporter",
+        version: "0.21.0",
+      },
+    },
+  };
+
+{ ['windows-exporter-' + name]: kp.windowsExporter[name] for name in std.objectFields(kp.windowsExporter) }
+```
+
+See the [full example](../examples/windows-hostprocess.jsonnet) for setup.
+
+If the cluster is running docker runtime then use the other [Windows addon](../examples/windows.jsonnet). The Windows addon does not deploy windows_exporter. Docker based Windows does not support running with [windows_exporter](https://github.com/prometheus-community/windows_exporter) in a pod so this add on uses [additional scrape configuration](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/additional-scrape-config.md) to set up a static config to scrape the node ports where windows_exporter is configured.
 
 The addon requires you to specify the node ips and ports where it can find the windows_exporter. See the [full example](../examples/windows.jsonnet) for setup.
 
@@ -19,5 +38,3 @@ local kp = (import 'kube-prometheus/main.libsonnet') +
     },
   };
 ```
-
-[Containerd](https://github.com/prometheus-community/windows_exporter/blob/master/kubernetes/kubernetes.md) version can run as pod.

examples/windows-hostprocess.jsonnet

@@ -0,0 +1,33 @@
+local kp =
+  (import 'kube-prometheus/main.libsonnet') +
+  (import 'kube-prometheus/addons/windows-hostprocess.libsonnet') +
+  {
+    values+:: {
+      common+: {
+        namespace: 'monitoring',
+      },
+      windowsExporter+:: {
+        image: 'ghcr.io/prometheus-community/windows-exporter',
+        version: '0.21.0',
+      },
+    },
+  };
+
+{ 'setup/0namespace-namespace': kp.kubePrometheus.namespace } +
+{
+  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
+  for name in std.filter((function(name) name != 'serviceMonitor' && name != 'prometheusRule'), std.objectFields(kp.prometheusOperator))
+} +
+// serviceMonitor and prometheusRule are separated so that they can be created after the CRDs are ready
+{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
+{ 'prometheus-operator-prometheusRule': kp.prometheusOperator.prometheusRule } +
+{ 'kube-prometheus-prometheusRule': kp.kubePrometheus.prometheusRule } +
+{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
+{ ['blackbox-exporter-' + name]: kp.blackboxExporter[name] for name in std.objectFields(kp.blackboxExporter) } +
+{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +
+{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
+{ ['kubernetes-' + name]: kp.kubernetesControlPlane[name] for name in std.objectFields(kp.kubernetesControlPlane) }
+{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
+{ ['windows-exporter-' + name]: kp.windowsExporter[name] for name in std.objectFields(kp.windowsExporter) } +
+{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
+{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) }

jsonnet/kube-prometheus/addons/windows-hostprocess.libsonnet

@@ -0,0 +1,197 @@
+local windowsdashboards = import 'github.com/kubernetes-monitoring/kubernetes-mixin/dashboards/windows.libsonnet';
+local windowsrules = import 'github.com/kubernetes-monitoring/kubernetes-mixin/rules/windows.libsonnet';
+
+local defaults = {
+  local defaults = self,
+  // Convention: Top-level fields related to CRDs are public, other fields are hidden
+  // If there is no CRD for the component, everything is hidden in defaults.
+  name:: 'windows-exporter',
+  namespace:: error 'must provide namespace',
+  version:: error 'must provide version',
+  image:: error 'must provide version',
+  resources:: {
+    requests: { cpu: '300m', memory: '200Mi' },
+    limits: { memory: '200Mi' },
+  },
+  collectorsEnabled:: 'cpu,logical_disk,net,os,system,container,memory',
+  scrapeTimeout:: '15s',
+  interval:: '30s',
+  listenAddress:: '127.0.0.1',
+  port:: 9182,
+  commonLabels:: {
+    'app.kubernetes.io/name': defaults.name,
+    'app.kubernetes.io/version': defaults.version,
+    'app.kubernetes.io/component': 'windows-exporter',
+    'app.kubernetes.io/part-of': 'kube-prometheus',
+  },
+  selectorLabels:: {
+    [labelName]: defaults.commonLabels[labelName]
+    for labelName in std.objectFields(defaults.commonLabels)
+    if !std.setMember(labelName, ['app.kubernetes.io/version'])
+  },
+};
+
+local windowsExporter = function(params) {
+  local we = self,
+  _config:: defaults + params,
+  // Safety check
+  assert std.isObject(we._config.resources),
+  _metadata:: {
+    name: we._config.name,
+    namespace: we._config.namespace,
+    labels: we._config.commonLabels,
+  },
+
+  daemonset: {
+    apiVersion: 'apps/v1',
+    kind: 'DaemonSet',
+    metadata: we._metadata,
+    spec: {
+      selector: {
+        matchLabels: we._config.selectorLabels,
+      },
+      updateStrategy: {
+        type: 'RollingUpdate',
+        rollingUpdate: { maxUnavailable: '10%' },
+      },
+      template: {
+        metadata: we._metadata,
+        spec: {
+          securityContext: {
+            windowsOptions: {
+              hostProcess: true,
+              runAsUserName: 'NT AUTHORITY\\system',
+            },
+          },
+          hostNetwork: true,
+          initContainers: [
+            {
+              name: 'configure-firewall',
+              image: 'mcr.microsoft.com/windows/nanoserver:1809',
+              resources: we._config.resources,
+              command: [
+                'powershell',
+              ],
+              args: [
+                'New-NetFirewallRule',
+                '-DisplayName',
+                "'windows-exporter'",
+                '-Direction',
+                'inbound',
+                '-Profile',
+                'Any',
+                '-Action',
+                'Allow',
+                '-LocalPort',
+                std.toString(we._config.port),
+                '-Protocol',
+                'TCP',
+              ],
+            },
+          ],
+          containers: [
+            {
+              args: [
+                '--config.file=%CONTAINER_SANDBOX_MOUNT_POINT%/config.yml',
+                '--collector.textfile.directory=%CONTAINER_SANDBOX_MOUNT_POINT%',
+              ],
+              name: we._config.name,
+              image: we._config.image + ':' + we._config.version,
+              imagePullPolicy: 'Always',
+              resources: we._config.resources,
+              ports: [
+                {
+                  containerPort: we._config.port,
+                  hostPort: we._config.port,
+                  name: 'http',
+                },
+              ],
+              volumeMounts: [
+                {
+                  name: 'windows-exporter-config',
+                  mountPath: '/config.yml',
+                  subPath: 'config.yml',
+                },
+              ],
+            },
+          ],
+          nodeSelector: {
+            'kubernetes.io/os': 'windows',
+          },
+          volumes: [
+            {
+              name: 'windows-exporter-config',
+              configMap: {
+                name: we._config.name,
+              },
+            },
+          ],
+        },
+      },
+    },
+  },
+  configmap: {
+    kind: 'ConfigMap',
+    apiVersion: 'v1',
+    metadata: we._metadata,
+    data: {
+      'config.yml': "collectors:\n  enabled: '" + we._config.collectorsEnabled + "'",
+    },
+  },
+  podmonitor: {
+    apiVersion: 'monitoring.coreos.com/v1',
+    kind: 'PodMonitor',
+    metadata: we._metadata,
+    spec: {
+      jobLabel: 'app.kubernetes.io/name',
+      selector: {
+        matchLabels: we._config.selectorLabels,
+      },
+      podMetricsEndpoints: [
+        {
+          port: 'http',
+          scheme: 'http',
+          scrapeTimeout: we._config.scrapeTimeout,
+          interval: we._config.interval,
+          relabelings: [
+            {
+              action: 'replace',
+              regex: '(.*)',
+              replacement: '$1',
+              sourceLabels: ['__meta_kubernetes_pod_node_name'],
+              targetLabel: 'instance',
+            },
+          ],
+        },
+      ],
+    },
+  },
+};
+
+{
+  values+:: {
+    windowsExporter+: {
+      name: defaults.name,
+      namespace: $.values.common.namespace,
+    },
+    grafana+:: {
+      dashboards+:: windowsdashboards {
+        _config: $.kubernetesControlPlane.mixin._config {
+          windowsExporterSelector: 'job="' + $.values.windowsExporter.name + '"',
+        },
+      }.grafanaDashboards,
+    },
+  },
+  kubernetesControlPlane+: {
+    mixin+:: {
+      prometheusRules+:: {
+        groups+: windowsrules {
+          _config: $.kubernetesControlPlane.mixin._config {
+            windowsExporterSelector: 'job="' + $.values.windowsExporter.name + '"',
+          },
+        }.prometheusRules.groups,
+      },
+    },
+  },
+  windowsExporter: windowsExporter($.values.windowsExporter),
+}