mirror of
https://github.com/Alkatrazz24/Funk-lab.git
synced 2026-07-08 21:54:42 +02:00
docs: ArgoCD + monitoring — documentation complète
- admin/argocd.md (NEW) : installation bootstrap, principe GitOps, structure k8s/, pattern App of Apps, déployer une app/chart Helm, commandes utiles, dépannage (OutOfSync, SSH key, field schema, StatefulSet PVC) - admin/monitoring.md (NEW) : accès+comptes Grafana/Prometheus/AlertManager, architecture scrape, node_exporter storage-01+gpu-01, ROCm sysfs collector, llama-server /metrics, webhook AlertManager→Hermes, admin Grafana+Prometheus, dashboards recommandés, points d'attention (persistence, RAM, Talos) - admin/cluster.md : section GitOps ArgoCD ajoutée - admin/README.md : index mis à jour avec argocd.md et monitoring.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
alkatrazz
parent
92a882aa94
commit
727d7351eb
4 changed files with 499 additions and 0 deletions
229
admin/monitoring.md
Normal file
229
admin/monitoring.md
Normal file
|
|
@ -0,0 +1,229 @@
|
|||
# Monitoring — Grafana + Prometheus + AlertManager
|
||||
|
||||
Stack de monitoring déployée via ArgoCD dans le namespace `monitoring`.
|
||||
Chart : `kube-prometheus-stack` v85.0.2 (Prometheus Operator).
|
||||
|
||||
---
|
||||
|
||||
## Accès et comptes
|
||||
|
||||
| Service | URL | Login | Mot de passe |
|
||||
|---|---|---|---|
|
||||
| **Grafana** | http://grafana.lab.local | `admin` | `funk-grafana` |
|
||||
| **AlertManager** | http://alertmanager.lab.local ¹ | — | — |
|
||||
| **Prometheus** | http://prometheus.lab.local ¹ | — | — |
|
||||
|
||||
¹ Ingress non encore créés — accessible via port-forward :
|
||||
```bash
|
||||
kubectl port-forward svc/kube-prometheus-stack-prometheus -n monitoring 9090:9090
|
||||
kubectl port-forward svc/kube-prometheus-stack-alertmanager -n monitoring 9093:9093
|
||||
```
|
||||
|
||||
> DNS `*.lab.local` ne résout que depuis le cluster LAN (192.168.10.0/24).
|
||||
> Depuis le poste perso (192.168.1.x), les entrées sont dans `/etc/hosts` :
|
||||
> ```
|
||||
> 192.168.10.200 grafana.lab.local argocd.lab.local
|
||||
> ```
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Prometheus (k8s, namespace monitoring)
|
||||
├── kube-state-metrics → ressources k8s (pods, deployments, nodes)
|
||||
├── node-exporter (daemonset) → métriques système compute-01/02/03
|
||||
├── scrape storage-01:9100 → métriques système storage-01
|
||||
├── scrape gpu-01:9100 → métriques système gpu-01
|
||||
├── scrape gpu-01:9101 (ROCm) → température GPU, VRAM, utilisation
|
||||
└── scrape gpu-01:1234/1236/1237 → llama-server (tokens/s, slots, queue)
|
||||
↓
|
||||
AlertManager
|
||||
└── webhook → storage-01:9093/webhook → ask-agent monitor → Hermes profil monitor
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Installation (déjà faite via ArgoCD)
|
||||
|
||||
Le monitoring est géré par ArgoCD. Pour le redéployer depuis zéro :
|
||||
|
||||
```bash
|
||||
# Ajouter le repo Helm
|
||||
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
|
||||
helm repo update
|
||||
|
||||
# L'application est créée automatiquement par ArgoCD depuis k8s/infra/monitoring/
|
||||
# Forcer un refresh si nécessaire :
|
||||
kubectl -n argocd annotate application kube-prometheus-stack \
|
||||
argocd.argoproj.io/refresh=hard --overwrite
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Sources de métriques
|
||||
|
||||
### Nœuds hors cluster (storage-01, gpu-01)
|
||||
|
||||
`node_exporter` v1.9.1 déployé via Ansible (rôle `node_exporter`).
|
||||
|
||||
```bash
|
||||
# Vérifier node_exporter
|
||||
systemctl status node_exporter
|
||||
curl -s http://192.168.10.1:9100/metrics | grep node_memory_MemAvailable
|
||||
curl -s http://192.168.10.20:9100/metrics | grep node_memory_MemAvailable
|
||||
```
|
||||
|
||||
### Métriques GPU AMD (gpu-01)
|
||||
|
||||
Collecteur sysfs via `rocm_scraper.timer` (toutes les 30s).
|
||||
Lit directement `/sys/class/drm/card0/device/` — pas de dépendance `rocm-smi`.
|
||||
|
||||
```bash
|
||||
# Vérifier les métriques ROCm
|
||||
ssh ansible@192.168.10.20 "cat /var/lib/node_exporter/textfile_collector/rocm.prom"
|
||||
|
||||
# Forcer une collecte manuelle
|
||||
ssh ansible@192.168.10.20 "sudo systemctl start rocm_scraper"
|
||||
```
|
||||
|
||||
Métriques disponibles :
|
||||
| Métrique | Description |
|
||||
|---|---|
|
||||
| `rocm_gpu_temperature_celsius` | Température (°C) |
|
||||
| `rocm_gpu_utilization_percent` | Utilisation GPU (%) |
|
||||
| `rocm_vram_used_bytes` | VRAM utilisée (bytes) |
|
||||
| `rocm_vram_total_bytes` | VRAM totale (bytes) |
|
||||
|
||||
### llama-server /metrics
|
||||
|
||||
llama.cpp expose des métriques Prometheus nativement sur `/metrics` :
|
||||
- `llamacpp:kv_cache_usage_ratio` — utilisation du contexte
|
||||
- `llamacpp:requests_processing` — requêtes en cours
|
||||
- `llamacpp:tokens_per_second` — débit
|
||||
|
||||
```bash
|
||||
curl -s http://192.168.10.20:1234/metrics | grep llamacpp
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## AlertManager → Hermes monitor
|
||||
|
||||
AlertManager envoie les alertes en POST vers `http://192.168.10.1:9093/webhook`.
|
||||
Le service `alertmanager-webhook` (Python, storage-01) reçoit les alertes et appelle :
|
||||
|
||||
```bash
|
||||
ask-agent monitor "Alertes Prometheus : <données> — analyse et recommandations"
|
||||
```
|
||||
|
||||
Hermes (profil `monitor`) analyse et répond dans ses logs.
|
||||
|
||||
```bash
|
||||
# Vérifier le webhook
|
||||
systemctl status alertmanager-webhook
|
||||
journalctl -u alertmanager-webhook -n 20 --no-pager --output=cat
|
||||
|
||||
# Tester manuellement
|
||||
curl -s -X POST http://192.168.10.1:9093/webhook \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"alerts":[{"status":"firing","labels":{"alertname":"TestAlert","severity":"warning","instance":"storage-01"},"annotations":{"summary":"Test alerte","description":"Vérification du webhook"}}]}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Administration Grafana
|
||||
|
||||
### Changer le mot de passe admin
|
||||
|
||||
```bash
|
||||
# Via l'API
|
||||
curl -X PUT http://admin:funk-grafana@grafana.lab.local/api/user/password \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"oldPassword":"funk-grafana","newPassword":"<nouveau>","confirmNew":"<nouveau>"}'
|
||||
```
|
||||
|
||||
Ou mettre à jour `k8s/infra/monitoring/values.yaml` → `grafana.adminPassword` → `git push`.
|
||||
|
||||
### Importer un dashboard
|
||||
|
||||
```bash
|
||||
# Via l'API (ID depuis grafana.com/dashboards)
|
||||
curl -X POST http://admin:funk-grafana@grafana.lab.local/api/dashboards/import \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"dashboard":{"id":null},"overwrite":true,"inputs":[{"name":"DS_PROMETHEUS","type":"datasource","pluginId":"prometheus","value":"prometheus"}],"folderId":0}'
|
||||
```
|
||||
|
||||
Dashboards recommandés :
|
||||
| Dashboard | ID Grafana.com | Usage |
|
||||
|---|---|---|
|
||||
| Node Exporter Full | 1860 | CPU, RAM, disque, réseau par nœud |
|
||||
| Kubernetes cluster | 7249 | Vue d'ensemble k8s |
|
||||
| kube-prometheus-stack | 15761 | Santé du monitoring lui-même |
|
||||
|
||||
### Datasources configurées automatiquement
|
||||
|
||||
| Nom | Type | URL |
|
||||
|---|---|---|
|
||||
| Prometheus | prometheus | `http://kube-prometheus-stack-prometheus:9090` |
|
||||
| Alertmanager | alertmanager | `http://kube-prometheus-stack-alertmanager:9093` |
|
||||
|
||||
---
|
||||
|
||||
## Administration Prometheus
|
||||
|
||||
### Vérifier les targets de scrape
|
||||
|
||||
```bash
|
||||
# Via port-forward
|
||||
kubectl port-forward svc/kube-prometheus-stack-prometheus -n monitoring 9090:9090 &
|
||||
curl -s http://localhost:9090/api/v1/targets | python3 -m json.tool | grep -E '"health"|"job"' | head -30
|
||||
```
|
||||
|
||||
### Ajouter un scrape config
|
||||
|
||||
Modifier `k8s/infra/monitoring/values.yaml` → section `additionalScrapeConfigs` → `git push`.
|
||||
|
||||
### Règles d'alerte
|
||||
|
||||
Les règles par défaut couvrent : OOMKill, NodeNotReady, PodCrashLooping, DiskPressure, etc.
|
||||
Pour ajouter des règles custom, créer un `PrometheusRule` dans `k8s/infra/monitoring/`.
|
||||
|
||||
---
|
||||
|
||||
## Commandes d'administration
|
||||
|
||||
```bash
|
||||
# État de tous les pods monitoring
|
||||
kubectl get pods -n monitoring
|
||||
|
||||
# Logs Prometheus
|
||||
kubectl logs -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -c prometheus --tail=50
|
||||
|
||||
# Logs Grafana
|
||||
kubectl logs -n monitoring -l app.kubernetes.io/name=grafana --tail=50
|
||||
|
||||
# Logs AlertManager
|
||||
kubectl logs -n monitoring alertmanager-kube-prometheus-stack-alertmanager-0 --tail=50
|
||||
|
||||
# Redémarrer Grafana
|
||||
kubectl rollout restart deployment kube-prometheus-stack-grafana -n monitoring
|
||||
|
||||
# Forcer re-sync ArgoCD
|
||||
kubectl -n argocd annotate application kube-prometheus-stack \
|
||||
argocd.argoproj.io/refresh=hard --overwrite
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Points d'attention
|
||||
|
||||
| Sujet | Détail |
|
||||
|---|---|
|
||||
| Persistance Prometheus | Désactivée (emptyDir) — données perdues au redémarrage du pod. Installer Longhorn pour activer |
|
||||
| Persistance Grafana | Désactivée — dashboards custom perdus au redémarrage. Exporter en JSON ou activer Longhorn |
|
||||
| RAM compute nodes | Prometheus + Grafana : ~1 GB total. Limites définies dans values.yaml |
|
||||
| Talos — composants désactivés | `kubeEtcd`, `kubeScheduler`, `kubeControllerManager`, `kubeProxy` désactivés (non accessibles depuis Talos) |
|
||||
| DNS lab.local | Résout uniquement depuis LAN (192.168.10.0/24) — ajouter `/etc/hosts` sur le poste perso pour grafana/argocd |
|
||||
| ROCm scraper | Lit sysfs directement (`/sys/class/drm/`), pas de `rocm-smi` (retiré dans ROCm 7.x) |
|
||||
| Webhook port 9093 | Ouvert uniquement depuis 192.168.10.0/24 (nftables storage-01) |
|
||||
Loading…
Add table
Add a link
Reference in a new issue