<!-- .slide: class="prez-slide-title-page" -->
# PF-2942<br/>Upgrade Prometheus<br/>et Prometheus-Operator<!-- .element: style="font-size: 2em" -->
#### Démo Équipe Monitoring du 29/04/2024
---
# Contenu de la présentation
## (Yves)<!-- .element: class="prez-slide-qui" -->
- Les ChangeLogs
- Prometheus
- Prometheus-Operator
- la chart Helm
- Démo : ScrapeConfig
- Démo : ScapeClass
- Métrologie
- Prometheus et Thanos-Receiver
- Conclusion
---
<!-- .slide: class="prez-slide-changelog" -->
# ChangeLog : Prometheus
## (Yves)<!-- .element: class="prez-slide-qui" -->
#### [ChangeLog 2.42.0 -> 2.51.2](https://github.com/prometheus/prometheus/blob/main/CHANGELOG.md)
- modifications récurrentes sur plusieurs versions
- "performance improvements"
- plusieurs améliorations relatives à remoteWrite (source et destination), dont `sample_age_limit` ou des modifications de codes d'erreur HTTP (`503/422 -> 499` dans certains cas)
- refacto puis bugfixes sur les labels (à partir de 2.43.0)
- **2.48.0** [ENHANCEMENT] Scraping: Add configuration option for tracking staleness of scraped timestamps. [#13060](https://github.com/prometheus/prometheus/pull/13060)
- **2.49.0** [BUGFIX] Mixins: Fix url filter on remote write dashboards. [#10721](https://github.com/prometheus/prometheus/pull/10721)
- **2.50.0** [FEATURE] Add automatic memory limit handling. Activated using the feature flag. `auto-gomemlimit` [#13395](https://github.com/prometheus/prometheus/pull/13395) : **envisager un PF d'étude** ?
---
<!-- .slide: class="prez-slide-changelog" -->
# ChangeLog : Prometheus-Operator
## (Nolwenn)<!-- .element: class="prez-slide-qui" -->
#### [ChangeLog 0.63.0 -> 0.73.1](https://github.com/prometheus-operator/prometheus-operator/blob/main/CHANGELOG.md)
- **0.64.0** : nouveau CRD PrometheusAgent
- **0.65.0** : nouveau CRD ScrapeConfig
- **0.66.0** : option `keep_firing_for` dans les PrometheusRules
- **0.71.0** : ajout des métriques `prometheus_operator_status_update_errors_total` et `prometheus_operator_status_update_operations_total`
- Cela permet de savoir quand il y a des échecs sur la modification du status des ressources
- Nous avons une alerte déployée (`PrometheusOperatorStatusUpdateErrors`) utilisant ces métriques, elle va maintenant fonctionner.
- **0.71.0** : ajout d'un paramètre `maximumStartupDurationSeconds` dans le CRD Prometheus
- **0.72.0** : ajout d'un objet `ScrapeClass` dans le CRD Prometheus
- **0.73.0** : ajout du champ `sampleAgeLimit` dans le CRD Prometheus
---
<!-- .slide: class="prez-slide-changelog" -->
# ChangeLog : chart Helm
## (Nolwenn)<!-- .element: class="prez-slide-qui" -->
#### [ChangeLog 45.7.1 -> 58.1.3](https://github.com/prometheus-community/helm-charts/commits/main/charts/kube-prometheus-stack/Chart.yaml)
- le ChangeLog est long mais reporte globalement les modifications de Prometheus-Operator.
- la [section Upgrading...](https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/README.md#upgrading-an-existing-release-to-a-new-major-version) du README.md mentionne :
- 47.x to 48.x : les CRDs sont déplacés dans une subchart dédiée
- 51.x to 52.x : quelques changements pour Thanos
- nos changements notables sont :
- possibilité de configurer Thanos Ruler pour envoyer les recording rules en RemoteWrite sur les Thanos Receive (mode stateless)
- ajout du scraping des métriques du config reloader de Prometheus, nous avons une alerte `ConfigReloaderSidecarErrors` déployée utilisant ces métriques, elle va maintenant fonctionner.
---
<!-- .slide: class="prez-slide-scrapeconfig1" -->
# ScrapeConfig simple
## (Yves)<!-- .element: class="prez-slide-qui" -->
```
apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
name: pf-2942-simple
namespace: monitoring
labels:
cloudservicesfactory/managed-by: corp-cluster
spec:
staticConfigs:
- labels:
job: pf-2942-simple
targets:
- demo.do.prometheus.io:9100
```
[Grafana](https://grafana.obs-corp-stg.csfpriv.com/explore?orgId=1&left=%7B%22datasource%22:%22thanos%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22node_exporter_build_info%7Bgoversion%21%3D%5C%22go1.19.3%5C%22%7D%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22thanos%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D) : `node_exporter_build_info{goversion!="go1.19.3"}`
---
<!-- .slide: class="prez-slide-scrapeconfig1" -->
# ScrapeConfig basicAuth (1/2)
## (Yves)<!-- .element: class="prez-slide-qui" -->
Générer le secret (on va fédérer test01 qui expose son API) :
```
#! /bin/bash
export export VAULT_ADDR=https://vault.infra-stg.caascad.com; vault token lookup || vault login -method oidc
s=$(vault read /secret/zones/fe/svc-monitoring-stack-client-test01/prometheus-federate-auth --format=json)
cat <<EOF > secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: pf-2942-basic-auth
namespace: monitoring
data:
password: "$(echo -n "${s}" | jq -r .data.password | tr -d '\n' | base64)"
user: "$(echo -n "${s}" | jq -r .data.username | tr -d '\n' | base64 )"
type: Opaque
EOF
```
---
<!-- .slide: class="prez-slide-scrapeconfig1" -->
# ScrapeConfig basicAuth (2/2)
## (Yves)<!-- .element: class="prez-slide-qui" -->
```
apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
name: pf-2942-authconfig
namespace: monitoring
labels:
cloudservicesfactory/managed-by: corp-cluster
spec:
staticConfigs:
- labels:
job: pf-2942-authconfig
targets:
- prometheus.obs-test01.csfpriv.com
basicAuth:
password:
name: pf-2942-basic-auth
key: password
username:
name: pf-2942-basic-auth
key: user
metricsPath: "/federate"
params:
'match[]':
- '{__name__="stackdriver_exporter_build_info"}'
relabelings:
- replacement: "pf-2942"
targetLabel: "ticket"
```
[Grafana](https://grafana.obs-corp-stg.csfpriv.com/explore?orgId=1&left=%7B%22datasource%22:%22thanos%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22stackdriver_exporter_build_info%7Bticket%3D%5C%22pf-2942%5C%22%7D%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22thanos%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22from%22:%22now-30m%22,%22to%22:%22now%22%7D%7D) : `stackdriver_exporter_build_info{ticket="pf-2942"}`
---
<!-- .slide: class="prez-slide-scrapeconfig1" -->
# ScrapeClass (1/2)
## (Nolwenn)<!-- .element: class="prez-slide-qui" -->
:warning:Attention:warning: : fonctionnalité "expérimentale" (mais cool)
Sur `kub-10003` (par exemple) :
`kubectl -n monitoring edit prometheus cluster-prometheus`
```
spec:
scrapeClasses:
- name: prez20240425
relabelings:
- action: replace
replacement: pf-2942
targetLabel: testprez
```
---
<!-- .slide: class="prez-slide-scrapeconfig1" -->
# ScrapeClass (2/2)
## (Nolwenn)<!-- .element: class="prez-slide-qui" -->
Sur `kub-10003` (par exemple) :
`kubectl -n logging edit smon promtail`
```
spec:
scrapeClass: prez20240425
```
[Grafana](https://grafana.obs-corp-stg.csfpriv.com/explore?orgId=1&left=%7B%22datasource%22:%22thanos%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22promtail_build_info%7Bcluster%3D%5C%22kub-10003%5C%22%7D%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22thanos%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D) : `promtail_build_info{cluster="kub-10003"}`
---
<!-- .slide: class="prez-slide-metrologie" -->
# Métrologie
## (Yves)<!-- .element: class="prez-slide-qui" -->
| | CPU | Mémoire (en Mo) |
|-|-----|-----------------|
| prometheus-operator | 1,5m -> 3,5m (pics à 150m) | 69-133 -> 34-54 |
| config-reloader | 0,6m -> 1m | 18-26 -> 30-45 |
| prometheus-cluster | 230m -> 140m | 1300-2800 -> 1050-2350 |
| prometheus-central | **8 -> 6,5** | **64Go -> 56 Go** |
[Grafana CPU](https://grafana.obs-corp-prd.cloudservicesfactory.com/explore?left=%7B%22datasource%22:%22thanos%22,%22queries%22:%5B%7B%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22thanos%22%7D,%22expr%22:%22clamp_max%28max%28node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate%7Bnamespace%3D%5C%22monitoring%5C%22,%20container%3D%5C%22prometheus%5C%22%7D%29,0.4%29%22,%22format%22:%22time_series%22,%22intervalFactor%22:2,%22refId%22:%22A%22,%22interval%22:%22%22,%22editorMode%22:%22code%22,%22range%22:true,%22instant%22:true%7D,%7B%22refId%22:%22B%22,%22expr%22:%22vector%28.14%29%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22thanos%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22from%22:%22now-3d%22,%22to%22:%22now%22%7D%7D&orgId=1) - [Grafana Mémoire](https://grafana.obs-corp-prd.cloudservicesfactory.com/explore?left=%7B%22datasource%22:%22thanos%22,%22queries%22:%5B%7B%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22thanos%22%7D,%22expr%22:%22min%28container_memory_working_set_bytes%7Bjob%3D%5C%22kubelet%5C%22,%20metrics_path%3D%5C%22%2Fmetrics%2Fcadvisor%5C%22,%20%20namespace%3D%5C%22monitoring%5C%22,%20%20container%3D%5C%22prometheus%5C%22,%20cluster%21~%5C%22kub-36%7Ckub-41%7Ckub-42%7Ckub-43%5C%22%7D%29%2F1024%2F1024%22,%22format%22:%22time_series%22,%22intervalFactor%22:2,%22refId%22:%22A%22,%22interval%22:%22%22,%22editorMode%22:%22code%22,%22range%22:true,%22instant%22:true%7D,%7B%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22thanos%22%7D,%22expr%22:%22max%28container_memory_working_set_bytes%7Bjob%3D%5C%22kubelet%5C%22,%20metrics_path%3D%5C%22%2Fmetrics%2Fcadvisor%5C%22,%20%20namespace%3D%5C%22monitoring%5C%22,%20%20container%3D%5C%22prometheus%5C%22,%20cluster%21~%5C%22kub-36%7Ckub-41%7Ckub-42%7Ckub-43%5C%22%7D%29%20%2F1024%2F1024%22,%22format%22:%22time_series%22,%22intervalFactor%22:2,%22refId%22:%22B%22,%22interval%22:%22%22,%22editorMode%22:%22code%22,%22range%22:true,%22instant%22:true%7D,%7B%22refId%22:%22C%22,%22expr%22:%22vector%282800%29%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22thanos%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22from%22:%22now-3d%22,%22to%22:%22now%22%7D%7D&orgId=1)
<!-- .element: style="text-align: center" -->
---
# Prometheus et Thanos-Receiver
## (Nolwenn)<!-- .element: class="prez-slide-qui" -->
- Paramètre `sample_age_limit`
- l'upgrade a provoqué des "incidents". Ce paramètre n'a pas eu l'effet escompté et provoque même un bug. Il a été temporairement supprimé.
- :warning: Mimir, Thanos-Receiver et Prometheus fonctionnent de la même façon : si des métriques sont "bloquées" dans l'agent (Prometheus-cluster sur NGOT), le problème existera aussi sur les nouvelles stacks.
- Il est maintenant possible de choisir le mode de déploiement de Thanos-Ruler (stateful vs stateless)
- Les versions de la chart kube-prometheus-stack, Prometheus-Operator et Thanos-Ruler sont en phase.
---
# Conclusion
## (Nolwenn)<!-- .element: class="prez-slide-qui" -->
- L'upgrade était nécessaire pour avancer sur le sujet Thanos-Receiver (PF-2640).
- Les avancées espérées (paramètre `sample_age_limit`, bugfix divers) sont décevantes.
- Notre satisfaction principale est d'avoir Prometheus-Operator et Prometheus à jour (ou presque).
<style type="text/css">
p {
font-size: 0.8em;
}
.reveal ul li {
font-size: 0.8em;
}
.reveal ul ul li {
font-size: 0.6em;
}
.reveal ol li {
font-size: 0.8em;
}
.reveal section {
text-align: left;
}
}
.reveal h3 {
color: orange;
text-align: center;
border-bottom: 1px solid orange;
font-size: 1em;
}
.reveal h1 {
color: orange;
text-align: center;
border-bottom: 1px solid orange;
margin-bottom: 0.4em;
font-size: 1.2em;
}
.reveal code {
color: aquamarine;
font-size: smaller;
}
.prez-slide-title-page div {
text-align: center;
font-size: smaller;
}
.prez-slide-changelog div {
font-size: .8em;
}
.prez-slide-scrapeconfig1 div pre {
border: 2px solid white;
}
.prez-slide-metrologie table td {
font-size: 0.6em;
border: 1px solid white;
}
.prez-slide-metrologie table th {
border: 1px solid white;
}
.prez-slide-metrologie table tr {
font-size: 0.8em;
border: 1px solid white;
}
.prez-slide-metrologie toto table tr {
font-size: 0.8em;
border: 1px solid white;
}
h2.prez-slide-qui {
margin: 0;
font-size: .2em;
text-align: right;
color: white;
}
</style>