mirror of
https://github.com/Alkatrazz24/Funk-lab.git
synced 2026-07-08 12:34:42 +02:00
docs: réorganisation complète de admin/ en 4 domaines thématiques
Structure avant : 20 fichiers à plat dans admin/ — difficile à naviguer.
Structure après : 4 sous-répertoires thématiques + index clair.
Réorganisation :
admin/ops/ → cluster.md, ansible.md, systeme.md
admin/infra/ → reseau.md, nfs.md, dnsmasq.md, ssh.md
admin/k8s/ → talos.md, argocd.md, monitoring.md
admin/ia/ → llama_server.md, rocm.md, litellm.md, hermes.md
Suppressions :
- ask-agent.md : contenu fusionné dans ia/hermes.md (section ask-agent)
- lm_studio.md : obsolète (LM Studio remplacé par llama-server)
Mises à jour contenu :
- ia/hermes.md : fusion complète avec ask-agent.md (profils, skills,
SOUL.md, ask-agent CLI, dépannage) — doc unifiée sans redondance
- ops/cluster.md : section GitOps réduite à 2 lignes + lien argocd.md
- incidents.md : tableau de résumé en tête + 4 nouveaux incidents
(Grafana OOMKilled, AlertManager null receiver, llama-server 501,
nftables règle après drop)
- README.md : réécrit — navigation rapide + index par domaine
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
alkatrazz
parent
d552abd2ef
commit
5bcf95b82e
19 changed files with 633 additions and 677 deletions
416
admin/k8s/monitoring.md
Normal file
416
admin/k8s/monitoring.md
Normal file
|
|
@ -0,0 +1,416 @@
|
|||
# Monitoring — Grafana + Prometheus + AlertManager
|
||||
|
||||
Stack déployée via ArgoCD dans le namespace `monitoring`.
|
||||
Chart : `kube-prometheus-stack` v85.0.2 (Prometheus Operator).
|
||||
Dashboards et alertes custom versionnés dans `k8s/infra/monitoring/`.
|
||||
|
||||
---
|
||||
|
||||
## Accès et comptes
|
||||
|
||||
| Service | URL | Login | Mot de passe |
|
||||
|---|---|---|---|
|
||||
| **Grafana** | http://grafana.lab.local | `admin` | `funk-grafana` |
|
||||
| **Prometheus** | http://prometheus.lab.local | — | — |
|
||||
| **AlertManager** | http://alertmanager.lab.local | — | — |
|
||||
|
||||
> DNS `*.lab.local` ne résout que depuis le LAN cluster (192.168.10.0/24).
|
||||
> Depuis le poste perso (192.168.1.x), ajouter dans `/etc/hosts` :
|
||||
> ```
|
||||
> 192.168.10.200 grafana.lab.local argocd.lab.local prometheus.lab.local alertmanager.lab.local
|
||||
> ```
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Prometheus (k8s, namespace monitoring)
|
||||
├── kube-state-metrics → ressources k8s (pods, deployments, nodes)
|
||||
├── node-exporter (DaemonSet) → métriques système compute-01/02/03
|
||||
├── scrape storage-01:9100 → métriques système storage-01 (job=storage-01)
|
||||
├── scrape gpu-01:9100 → métriques système + ROCm gpu-01 (job=gpu-01-node)
|
||||
│ └── textfile_collector → rocm_gpu_temperature/utilization/vram (rocm_scraper.timer)
|
||||
├── scrape gpu-01:1234 → llama-server GPU Qwen3-8B (job=llama-server-gpu)
|
||||
├── scrape gpu-01:1236 → llama-server CPU system (job=llama-server-system)
|
||||
└── scrape gpu-01:1237 → llama-server CPU monitor (job=llama-server-monitor)
|
||||
↓
|
||||
AlertManager
|
||||
├── receiver "null" → alerte Watchdog (heartbeat interne — silencé)
|
||||
└── receiver "hermes-monitor" → webhook storage-01:9093 → ask-agent monitor → Hermes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dashboards Grafana
|
||||
|
||||
Trois dashboards custom auto-importés via ConfigMaps (label `grafana_dashboard: "1"`).
|
||||
Versionnés dans `k8s/infra/monitoring/dashboards/`.
|
||||
|
||||
| Dashboard | UID | Contenu |
|
||||
|---|---|---|
|
||||
| **Funk — Kubernetes** | `funk-k8s1` | Cluster overview, CPU/RAM par nœud, pod restarts, PVCs |
|
||||
| **Funk — Infrastructure** | `funk-inf1` | Hardware compute/storage-01/gpu-01 — CPU, RAM, disque, réseau, GPU ROCm |
|
||||
| **Funk — IA / Hermes** | `funk-ai01` | Tokens/s llama-server (GPU+CPU), requêtes, VRAM, température GPU |
|
||||
|
||||
URLs directes :
|
||||
- http://grafana.lab.local/d/funk-k8s1
|
||||
- http://grafana.lab.local/d/funk-inf1
|
||||
- http://grafana.lab.local/d/funk-ai01
|
||||
|
||||
### Ajouter un dashboard custom (GitOps)
|
||||
|
||||
1. Créer `k8s/infra/monitoring/dashboards/mon-dashboard.yaml` :
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: grafana-dashboard-mon-dashboard
|
||||
namespace: monitoring
|
||||
labels:
|
||||
grafana_dashboard: "1"
|
||||
data:
|
||||
mon-dashboard.json: |
|
||||
{ ... JSON exporté depuis Grafana ... }
|
||||
```
|
||||
|
||||
2. `git add` + `git push` → ArgoCD applique → sidecar Grafana importe automatiquement.
|
||||
|
||||
Pour exporter un dashboard depuis Grafana : **Dashboard → Share → Export → Save to file**.
|
||||
|
||||
---
|
||||
|
||||
## Règles d'alerte
|
||||
|
||||
Versionnées dans `k8s/infra/monitoring/alerts/` comme `PrometheusRule`.
|
||||
Déclenchent AlertManager → webhook storage-01:9093 → profil `monitor` Hermes.
|
||||
|
||||
### Nœuds (`alerts-node.yaml`)
|
||||
|
||||
| Alerte | Seuil | Sévérité |
|
||||
|---|---|---|
|
||||
| `NodeHighCPU` | CPU > 90% pendant 5 min | warning |
|
||||
| `NodeCriticalCPU` | CPU > 98% pendant 2 min | critical |
|
||||
| `NodeHighMemory` | RAM > 90% pendant 5 min | warning |
|
||||
| `NodeCriticalMemory` | RAM > 97% pendant 2 min | critical |
|
||||
| `NodeDiskSpaceLow` | Disque > 80% pendant 10 min | warning |
|
||||
| `NodeDiskSpaceCritical` | Disque > 93% pendant 5 min | critical |
|
||||
| `Storage01Down` | `up{job="storage-01"} == 0` pendant 2 min | critical |
|
||||
| `Gpu01Down` | `up{job="gpu-01-node"} == 0` pendant 2 min | critical |
|
||||
| `NodeHighLoad` | Load5 > 8 pendant 10 min | warning |
|
||||
|
||||
### Kubernetes (`alerts-k8s.yaml`)
|
||||
|
||||
| Alerte | Seuil | Sévérité |
|
||||
|---|---|---|
|
||||
| `KubeNodeNotReady` | Nœud NotReady pendant 5 min | critical |
|
||||
| `PodCrashLooping` | > 3 restarts en 15 min | warning |
|
||||
| `PodFailedLong` | Pod Failed/Unknown pendant 10 min | warning |
|
||||
| `PodPendingLong` | Pod Pending pendant 15 min | warning |
|
||||
| `DeploymentUnavailable` | Réplicas < spec pendant 5 min | warning |
|
||||
| `PVCNotBound` | PVC non Bound pendant 10 min | warning |
|
||||
| `ArgoCDAppOutOfSync` | App ArgoCD OutOfSync pendant 30 min | warning |
|
||||
|
||||
### IA / LLM (`alerts-ai.yaml`)
|
||||
|
||||
| Alerte | Seuil | Sévérité |
|
||||
|---|---|---|
|
||||
| `LlamaServerGPUDown` | `up{job="llama-server-gpu"} == 0` pendant 2 min | critical |
|
||||
| `LlamaServerSystemDown` | `up{job="llama-server-system"} == 0` pendant 2 min | warning |
|
||||
| `LlamaServerMonitorDown` | `up{job="llama-server-monitor"} == 0` pendant 2 min | warning |
|
||||
| `GPUTemperatureHigh` | Temp GPU > 80°C pendant 5 min | warning |
|
||||
| `GPUTemperatureCritical` | Temp GPU > 90°C pendant 2 min | critical |
|
||||
| `GPUVRAMAlmostFull` | VRAM > 95% pendant 5 min | warning |
|
||||
| `LlamaServerHighQueueGPU` | Requêtes deferred > 5 pendant 2 min | warning |
|
||||
|
||||
### Ajouter une règle d'alerte
|
||||
|
||||
```yaml
|
||||
# k8s/infra/monitoring/alerts/ma-regle.yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: funk-ma-regle
|
||||
namespace: monitoring
|
||||
labels:
|
||||
release: kube-prometheus-stack # obligatoire — sélecteur du Prometheus operator
|
||||
spec:
|
||||
groups:
|
||||
- name: funk.custom
|
||||
rules:
|
||||
- alert: MonAlerte
|
||||
expr: ma_metrique > seuil
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Description courte"
|
||||
description: "Détail avec {{ $value }}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Sources de métriques
|
||||
|
||||
### Nœuds hors cluster (storage-01, gpu-01)
|
||||
|
||||
`node_exporter` v1.9.1 déployé via Ansible (rôle `node_exporter`).
|
||||
Prérequis firewall : port 9100 ouvert depuis 192.168.10.0/24 (nftables storage-01, firewalld gpu-01).
|
||||
|
||||
```bash
|
||||
# État node_exporter
|
||||
ssh storage-01 "systemctl status node_exporter"
|
||||
ssh gpu-01 "systemctl status node_exporter"
|
||||
|
||||
# Vérifier les métriques directement
|
||||
curl -s http://192.168.10.1:9100/metrics | grep node_memory_MemAvailable
|
||||
curl -s http://192.168.10.20:9100/metrics | grep node_memory_MemAvailable
|
||||
|
||||
# Vérifier le statut des targets dans Prometheus
|
||||
curl -s http://prometheus.lab.local/api/v1/targets | python3 -c "
|
||||
import sys, json
|
||||
d = json.load(sys.stdin)
|
||||
for t in d['data']['activeTargets']:
|
||||
job = t['labels'].get('job','?')
|
||||
print(t['health'], job, t.get('lastError','')[:60])
|
||||
" | grep -E "storage-01|gpu-01|llama"
|
||||
```
|
||||
|
||||
### Métriques GPU AMD (gpu-01) — ROCm via textfile_collector
|
||||
|
||||
Collecteur sysfs `rocm_scraper.timer` (toutes les 30s) écrit dans `/var/lib/node_exporter/textfile_collector/rocm.prom`.
|
||||
Servi par node_exporter sur port 9100 — **pas de port séparé**.
|
||||
Lit `/sys/class/drm/card0/device/` directement (rocm-smi retiré dans ROCm 7.x).
|
||||
|
||||
```bash
|
||||
# Vérifier les métriques ROCm
|
||||
ssh gpu-01 "cat /var/lib/node_exporter/textfile_collector/rocm.prom"
|
||||
|
||||
# Forcer une collecte manuelle
|
||||
ssh gpu-01 "sudo systemctl start rocm_scraper"
|
||||
|
||||
# Vérifier le timer
|
||||
ssh gpu-01 "systemctl status rocm_scraper.timer"
|
||||
```
|
||||
|
||||
Métriques disponibles (labels : `gpu="0"`, `model="gfx1031"`) :
|
||||
|
||||
| Métrique | Description | Valeur typique |
|
||||
|---|---|---|
|
||||
| `rocm_gpu_temperature_celsius` | Température (°C) | 40–85 |
|
||||
| `rocm_gpu_utilization_percent` | Utilisation GPU (%) | 0–100 |
|
||||
| `rocm_vram_used_bytes` | VRAM utilisée | ~10 GB (modèle chargé) |
|
||||
| `rocm_vram_total_bytes` | VRAM totale | 12.9 GB (RX 6700XT) |
|
||||
|
||||
### llama-server /metrics
|
||||
|
||||
llama.cpp expose ses métriques sur `/metrics` — activé via `--metrics` dans le service systemd.
|
||||
|
||||
| Métrique | Description |
|
||||
|---|---|
|
||||
| `llamacpp:prompt_tokens_seconds` | Débit prompt moyen (tokens/s) |
|
||||
| `llamacpp:predicted_tokens_seconds` | Débit génération moyen (tokens/s) |
|
||||
| `llamacpp:prompt_tokens_total` | Tokens prompt totaux (compteur) |
|
||||
| `llamacpp:tokens_predicted_total` | Tokens générés totaux (compteur) |
|
||||
| `llamacpp:requests_processing` | Requêtes en cours de traitement |
|
||||
| `llamacpp:requests_deferred` | Requêtes en file d'attente |
|
||||
| `llamacpp:n_decode_total` | Nombre total d'appels llama_decode() |
|
||||
|
||||
```bash
|
||||
# Métriques GPU (Qwen3-8B)
|
||||
curl -s http://192.168.10.20:1234/metrics | grep llamacpp
|
||||
|
||||
# Métriques CPU system
|
||||
curl -s http://192.168.10.20:1236/metrics | grep llamacpp
|
||||
|
||||
# Métriques CPU monitor
|
||||
curl -s http://192.168.10.20:1237/metrics | grep llamacpp
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## AlertManager → Hermes monitor
|
||||
|
||||
AlertManager envoie les alertes en POST vers `http://192.168.10.1:9093/webhook`.
|
||||
Le service `alertmanager-webhook` (Python, storage-01) reçoit les alertes et appelle `ask-agent monitor`.
|
||||
Hermes (profil `monitor`) analyse et répond dans ses logs.
|
||||
|
||||
```bash
|
||||
# Vérifier le webhook
|
||||
ssh storage-01 "systemctl status alertmanager-webhook"
|
||||
ssh storage-01 "journalctl -u alertmanager-webhook -n 20 --no-pager --output=cat"
|
||||
|
||||
# Tester manuellement
|
||||
curl -s -X POST http://192.168.10.1:9093/webhook \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"alerts":[{"status":"firing","labels":{"alertname":"TestAlert","severity":"warning","instance":"storage-01"},"annotations":{"summary":"Test alerte","description":"Vérification du webhook"}}]}'
|
||||
|
||||
# Voir les alertes actives dans AlertManager
|
||||
curl -s http://alertmanager.lab.local/api/v2/alerts | python3 -m json.tool
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Persistance des données
|
||||
|
||||
Prometheus et Grafana stockent leurs données sur le RAID5 de storage-01 via NFS.
|
||||
Provisioner : `nfs-subdir-external-provisioner`, StorageClass `nfs`.
|
||||
|
||||
| Service | PVC | Taille | Rétention |
|
||||
|---|---|---|---|
|
||||
| Prometheus | `prometheus-...-db-prometheus-...-0` | 20 Gi | 15 jours / 8 GB |
|
||||
| Grafana | `kube-prometheus-stack-grafana` | 2 Gi | — |
|
||||
|
||||
```bash
|
||||
# Espace consommé sur le RAID5
|
||||
ssh storage-01 "du -sh /srv/data/nfs/k8s/*"
|
||||
|
||||
# État des PVCs
|
||||
kubectl get pvc -n monitoring
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Administration Grafana
|
||||
|
||||
### Changer le mot de passe admin
|
||||
|
||||
Via API :
|
||||
```bash
|
||||
curl -X PUT http://admin:funk-grafana@grafana.lab.local/api/user/password \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"oldPassword":"funk-grafana","newPassword":"<nouveau>","confirmNew":"<nouveau>"}'
|
||||
```
|
||||
|
||||
Ou modifier `k8s/infra/monitoring/values.yaml` → `grafana.adminPassword` → `git push`.
|
||||
|
||||
### Datasources
|
||||
|
||||
| Nom | Type | URL (interne cluster) |
|
||||
|---|---|---|
|
||||
| Prometheus | prometheus | `http://kube-prometheus-stack-prometheus:9090` |
|
||||
| Alertmanager | alertmanager | `http://kube-prometheus-stack-alertmanager:9093` |
|
||||
|
||||
Configurées automatiquement au démarrage — ne pas modifier depuis l'UI (écrasé à chaque redémarrage).
|
||||
|
||||
### Lister les dashboards disponibles
|
||||
|
||||
```bash
|
||||
curl -s http://admin:funk-grafana@grafana.lab.local/api/search?type=dash-db \
|
||||
| python3 -c "import sys,json; [print(d['title'],'—',d['uid']) for d in json.load(sys.stdin)]"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Administration Prometheus
|
||||
|
||||
### Vérifier les targets de scrape
|
||||
|
||||
```bash
|
||||
# Depuis l'UI
|
||||
open http://prometheus.lab.local/targets
|
||||
|
||||
# Via API
|
||||
curl -s http://prometheus.lab.local/api/v1/targets | python3 -c "
|
||||
import sys, json
|
||||
d = json.load(sys.stdin)
|
||||
for t in d['data']['activeTargets']:
|
||||
print(t['health'], t['labels'].get('job','?'), t.get('lastError','')[:60])
|
||||
" | sort
|
||||
```
|
||||
|
||||
### Ajouter un scrape config
|
||||
|
||||
Modifier `k8s/infra/monitoring/values.yaml` → section `additionalScrapeConfigs` → `git push`.
|
||||
|
||||
```yaml
|
||||
prometheus:
|
||||
prometheusSpec:
|
||||
additionalScrapeConfigs:
|
||||
- job_name: mon-service
|
||||
static_configs:
|
||||
- targets: ['192.168.10.x:9xxx']
|
||||
labels:
|
||||
instance: mon-instance
|
||||
```
|
||||
|
||||
### Vérifier les règles d'alerte
|
||||
|
||||
```bash
|
||||
# Depuis l'UI
|
||||
open http://prometheus.lab.local/alerts
|
||||
|
||||
# Via API — alertes actuellement firing
|
||||
curl -s http://prometheus.lab.local/api/v1/alerts | python3 -c "
|
||||
import sys, json
|
||||
d = json.load(sys.stdin)
|
||||
for a in d['data']['alerts']:
|
||||
if a['state'] == 'firing':
|
||||
print(a['labels']['alertname'], a['labels'].get('instance',''), a['state'])
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Commandes d'administration
|
||||
|
||||
```bash
|
||||
# État de tous les pods monitoring
|
||||
kubectl get pods -n monitoring
|
||||
|
||||
# Logs Prometheus
|
||||
kubectl logs -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -c prometheus --tail=50
|
||||
|
||||
# Logs Grafana
|
||||
kubectl logs -n monitoring -l app.kubernetes.io/name=grafana --tail=50
|
||||
|
||||
# Logs AlertManager
|
||||
kubectl logs -n monitoring alertmanager-kube-prometheus-stack-alertmanager-0 --tail=50
|
||||
|
||||
# Redémarrer Grafana (recharge les ConfigMaps dashboards)
|
||||
kubectl rollout restart deployment kube-prometheus-stack-grafana -n monitoring
|
||||
|
||||
# Forcer re-sync ArgoCD (dashboards, alertes, config)
|
||||
kubectl -n argocd annotate application monitoring argocd.argoproj.io/refresh=hard --overwrite
|
||||
kubectl -n argocd annotate application kube-prometheus-stack argocd.argoproj.io/refresh=hard --overwrite
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Structure des fichiers
|
||||
|
||||
```
|
||||
k8s/infra/monitoring/
|
||||
├── helmrelease.yaml # Application ArgoCD (multi-source Helm kube-prometheus-stack)
|
||||
├── namespace.yaml # Namespace monitoring (labels PodSecurity privileged)
|
||||
├── values.yaml # Valeurs Helm : ressources, ingress, scrape configs, alertmanager
|
||||
├── dashboards/
|
||||
│ ├── dashboard-kubernetes.yaml # ConfigMap — Funk Kubernetes (uid: funk-k8s1)
|
||||
│ ├── dashboard-infrastructure.yaml # ConfigMap — Funk Infrastructure (uid: funk-inf1)
|
||||
│ └── dashboard-ai.yaml # ConfigMap — Funk IA/Hermes (uid: funk-ai01)
|
||||
└── alerts/
|
||||
├── alerts-node.yaml # PrometheusRule — CPU, RAM, disque, nodes hors cluster
|
||||
├── alerts-k8s.yaml # PrometheusRule — k8s : pods, nodes, PVCs, ArgoCD
|
||||
└── alerts-ai.yaml # PrometheusRule — llama-server, GPU temp, VRAM
|
||||
```
|
||||
|
||||
> ArgoCD Application `monitoring` utilise `directory.recurse: true` pour scanner
|
||||
> récursivement le répertoire et ses sous-dossiers.
|
||||
|
||||
---
|
||||
|
||||
## Points d'attention
|
||||
|
||||
| Sujet | Détail |
|
||||
|---|---|
|
||||
| Persistance | **Active** — Prometheus 20 Gi + Grafana 2 Gi sur RAID5 NFS storage-01 |
|
||||
| Dashboards GitOps | ConfigMaps dans `dashboards/` → importés automatiquement par le sidecar Grafana |
|
||||
| Alertes GitOps | PrometheusRules dans `alerts/` → label `release: kube-prometheus-stack` obligatoire |
|
||||
| Receiver `null` | Doit exister dans la config AlertManager — requis par les sous-routes internes du chart (Watchdog) |
|
||||
| nftables storage-01 | Port 9100 doit être avant la règle `drop` dans la chaîne `input` (sinon context deadline) |
|
||||
| ROCm port | Métriques GPU via textfile_collector sur port **9100** — pas de port séparé 9101 |
|
||||
| llama-server `--metrics` | Flag requis dans le service systemd pour activer l'endpoint `/metrics` |
|
||||
| Talos — composants désactivés | `kubeEtcd`, `kubeScheduler`, `kubeControllerManager`, `kubeProxy` désactivés (non accessibles) |
|
||||
| PodSecurity privileged | Namespace `monitoring` doit être `privileged` pour node-exporter (hostNetwork/hostPID/hostPath) |
|
||||
| nfs-provisioner | StorageClass peut disparaître si ArgoCD prune l'app — forcer sync si PVC Pending |
|
||||
Loading…
Add table
Add a link
Reference in a new issue