Commit Graph
11 Commits
Author SHA1 Message Date
Arunprasad Rajkumar 6ff8bfbb02 Adjust NodeFilesystemSpaceFillingUp thresholds according default kubelet GC behavior
Previously[1] we attempted to do the same, but there was a
misunderstanding about the GC behavior and it caused the alert to be
fired even before GC comes into play.

According to[2][3] kubelet GC kicks in only when `imageGCHighThresholdPercent` is hit which is set to 85% by default. However `NodeFilesystemSpaceFillingUp` is set to fire as soon as 80% usage is hit.

This commit changes the `fsSpaceFillingUpWarningThreshold` to 15% so
that we give ample time to GC to reclaim unwanted images. This commit
also changes `fsSpaceFillingUpCriticalThreshold` to 10% which gives more time to admins to react to warning before sending critical alert.

[1] https://github.com/prometheus-operator/kube-prometheus/pull/1357
[2] https://docs.openshift.com/container-platform/4.10/nodes/nodes/nodes-nodes-garbage-collection.html#nodes-nodes-garbage-collection-images_nodes-nodes-configuring
[3] https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/

Signed-off-by: Arunprasad Rajkumar <arajkuma@redhat.com>
2022-04-13 12:01:06 +05:30
Arunprasad Rajkumar 0eabbb5d0c Sanitize all regex denylist in ksm-lite addon
This is a follow up fix of https://github.com/prometheus-operator/kube-prometheus/pull/1613. @simonpasquier recommended to sanitize all denylist metrics.

Signed-off-by: Arunprasad Rajkumar <arajkuma@redhat.com>
2022-02-03 19:52:44 +05:30
Arunprasad RajkumarandFlorian Gleizes f16277e8e0 Sanitize regex denylist in ksm-lite addon
The following metrics are missing from kube-state-metrics:
- kube_pod_container_status_terminated_reason
- kube_pod_init_container_status_terminated_reason
- kube_pod_status_scheduled_time

Previously, some metrics were removed from kube-state-metrics by adding the following --metric-denylist argument to the kube-state-metrics container

```
--metric-denylist=
kube_.+_created,
kube_.+_metadata_resource_version,
kube_replicaset_metadata_generation,
kube_replicaset_status_observed_generation,
kube_pod_restart_policy,
kube_pod_init_container_status_terminated,
kube_pod_init_container_status_running,
kube_pod_container_status_terminated,
kube_pod_container_status_running,
kube_pod_completion_time,
kube_pod_status_scheduled
```

--metric-denylist: Comma-separated list of metrics not to be enabled. This list comprises of exact metric names and/or regex patterns. The allowlist and denylist are mutually exclusive.

However, all the list of metrics is managed as RegEx, thus "kube_pod_container_status_terminated" denies .*kube_pod_container_status_terminated.*, that's why kube_pod_init_container_status_terminated_reason is missing

Co-authored-by: Florian Gleizes <fgleizes@redhat.com>
Signed-off-by: Arunprasad Rajkumar <arajkuma@redhat.com>
2022-02-03 17:50:22 +05:30
Arunprasad RajkumarandPawe? Krupa b7a135a101 Fix thanos sidecar selector
Previous PR[1] which deprecated `thanosSelector` accidentally broke the
usage of it. This commit restores the old behaviour along with newly
introduced `thanos` config variable.

[1] https://github.com/prometheus-operator/kube-prometheus/pull/1543

Co-authored-by: Pawe? Krupa <pawel@krupa.net.pl>
Signed-off-by: Arunprasad Rajkumar <arajkuma@redhat.com>
2022-01-05 16:23:24 +05:30
Arunprasad Rajkumar 1ff5273f87 Deprecate thanosSelector and keep until 0.10.0
Signed-off-by: Arunprasad Rajkumar <arajkuma@redhat.com>
2021-12-13 16:23:37 +05:30
Arunprasad Rajkumar 1fb864ec21 Remove thanosSelector and expose mixin._config.thanos config variable for thanos sidecar
This commit removes existing `thanosSelector` and exposes a single config variable `mixin._config.thanos` to customize thanos sidecar mixins. It follows same structure as https://github.com/thanos-io/thanos/blob/d2d74dac9878f633f1fb0e253bcc0b1cdea376d3/mixin/config.libsonnet

Signed-off-by: Arunprasad Rajkumar <arajkuma@redhat.com>
2021-12-10 08:56:44 +05:30
Arunprasad Rajkumar 19347ed662 Fix typo in jsonnet dependency upgrade GH actions
Replace `commiter` to `committer`.

Refer https://github.com/peter-evans/create-pull-request#action-inputs for more
info about supported input properties for `peter-evans/create-pull-request@v3`.

Signed-off-by: Arunprasad Rajkumar <arajkuma@redhat.com>
2021-10-19 16:03:20 +05:30
Arunprasad Rajkumar c5d265a14e thanos: bump to latest and add thanosPrometheusCommonDimensions
This commit pulls latest changes from thanos mixins and sets `thanosPrometheusCommonDimensions`
to `namespace, pod` for k8s use case.

Refer https://github.com/thanos-io/thanos/pull/4508 for more details.

Signed-off-by: Arunprasad Rajkumar <arajkuma@redhat.com>
2021-09-27 12:07:08 +05:30
Arunprasad Rajkumar 4de44139ec add comments to reason fsSpaceFilling threshold adjustment
Signed-off-by: Arunprasad Rajkumar <arajkuma@redhat.com>
2021-09-02 17:38:02 +05:30
Arunprasad Rajkumar 16ef18ec54 assets: regenerate
Signed-off-by: Arunprasad Rajkumar <arajkuma@redhat.com>
2021-09-01 13:33:41 +05:30
Arunprasad Rajkumar 03471fd86f Adjust threshold for SpaceFillingUp warning alert
Reduce threshold of NodeFilesystemSpaceFillingUp warning alert to 20% space available, instead of 40% (default).

This will align the threshold according to default kubelet GC values
below[1],

"imageMinimumGCAge": "2m0s",
"imageGCHighThresholdPercent": 85,
"imageGCLowThresholdPercent": 80,

[1] https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/

Signed-off-by: Arunprasad Rajkumar <arajkuma@redhat.com>
2021-09-01 13:29:36 +05:30