OllamaをPrometheus+Grafanaで監視する方法｜GPU使用率・レスポンス時間・エラー率をリアルタイムダッシュボードで可視化する

この記事のポイント

・Ollamaは標準で `/metrics` エンドポイントを持つためPrometheusと追加プラグインなしに連携できる
・Prometheus＋node_exporter＋nvidia_gpu_exporterの3点セットでシステム・GPU・LLMを一元監視できる
・Grafanaのデータソースにprometheusを登録するだけでリアルタイムダッシュボードが構成できる
・アラートルールを設定すればVRAM逼迫やレスポンス遅延を自動検知してSlack通知できる

# prometheus専用ユーザーを作成
$ sudo useradd --no-create-home --shell /bin/false prometheus

# バイナリのダウンロードと展開
$ cd /tmp
$ wget https://github.com/prometheus/prometheus/releases/download/v2.52.0/prometheus-2.52.0.linux-amd64.tar.gz
$ tar xvf prometheus-2.52.0.linux-amd64.tar.gz
$ sudo cp prometheus-2.52.0.linux-amd64/prometheus /usr/local/bin/
$ sudo cp prometheus-2.52.0.linux-amd64/promtool /usr/local/bin/
$ sudo chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool

# 設定ファイルとデータディレクトリの作成
$ sudo mkdir /etc/prometheus /var/lib/prometheus
$ sudo cp -r prometheus-2.52.0.linux-amd64/consoles /etc/prometheus/
$ sudo cp -r prometheus-2.52.0.linux-amd64/console_libraries /etc/prometheus/
$ sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

$ sudo nano /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

$ sudo nano /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --storage.tsdb.retention.time=30d

[Install]
WantedBy=multi-user.target

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now prometheus
$ sudo systemctl status prometheus
* prometheus.service - Prometheus
     Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2026-07-04 10:00:00 JST
   Main PID: 12345 (prometheus)
      Tasks: 8 (limit: 9400)
     Memory: 52.3M

$ curl -s http://localhost:11434/metrics | head -30
# HELP ollama_request_duration_seconds Time taken to generate a response
# TYPE ollama_request_duration_seconds histogram
ollama_request_duration_seconds_bucket{model="llama3.3:70b-instruct-q4_0",le="0.1"} 0
ollama_request_duration_seconds_bucket{model="llama3.3:70b-instruct-q4_0",le="0.5"} 3
ollama_request_duration_seconds_bucket{model="llama3.3:70b-instruct-q4_0",le="1"} 7
ollama_request_duration_seconds_bucket{model="llama3.3:70b-instruct-q4_0",le="+Inf"} 47
# HELP ollama_requests_total Total number of requests processed
# TYPE ollama_requests_total counter
ollama_requests_total{model="llama3.3:70b-instruct-q4_0",status="success"} 45
ollama_requests_total{model="llama3.3:70b-instruct-q4_0",status="error"} 2
# HELP ollama_loaded_models Number of models currently loaded in memory
# TYPE ollama_loaded_models gauge
ollama_loaded_models 1

  - job_name: 'ollama'
    static_configs:
      - targets: ['localhost:11434']
    metrics_path: /metrics
    scrape_interval: 10s

# SIGHUPでリロード（サービス停止なし）
$ sudo systemctl reload prometheus

# Web APIでリロードする場合（ExecStartに --web.enable-lifecycle が必要）
$ curl -X POST http://localhost:9090/-/reload

$ cd /tmp
$ wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
$ tar xvf node_exporter-1.8.2.linux-amd64.tar.gz
$ sudo cp node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/
$ sudo useradd --no-create-home --shell /bin/false node_exporter
$ sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

$ sudo nano /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now node_exporter

# 動作確認
$ curl -s http://localhost:9100/metrics | grep node_cpu_seconds | head -3
# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 12345.67

$ cd /tmp
$ wget https://github.com/utkuozdemir/nvidia_gpu_exporter/releases/download/v1.2.1/nvidia_gpu_exporter_1.2.1_linux_amd64.tar.gz
$ tar xvf nvidia_gpu_exporter_1.2.1_linux_amd64.tar.gz
$ sudo cp nvidia_gpu_exporter /usr/local/bin/
$ sudo chmod +x /usr/local/bin/nvidia_gpu_exporter

$ sudo nano /etc/systemd/system/nvidia_gpu_exporter.service

[Unit]
Description=NVIDIA GPU Exporter
After=network.target

[Service]
User=root
Type=simple
ExecStart=/usr/local/bin/nvidia_gpu_exporter

[Install]
WantedBy=multi-user.target

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now nvidia_gpu_exporter

# GPU使用率の取得確認
$ curl -s http://localhost:9835/metrics | grep nvidia_smi_utilization_gpu
nvidia_smi_utilization_gpu_ratio{uuid="GPU-xxxx-xxxx",name="NVIDIA GeForce RTX 4090"} 0.87

  - job_name: 'nvidia_gpu'
    static_configs:
      - targets: ['localhost:9835']

$ sudo apt-get install -y apt-transport-https software-properties-common wget
$ sudo mkdir -p /etc/apt/keyrings
$ wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
$ echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
$ sudo apt-get update
$ sudo apt-get install -y grafana
$ sudo systemctl enable --now grafana-server

$ sudo systemctl status grafana-server
* grafana-server.service - Grafana instance
     Loaded: loaded (/lib/systemd/system/grafana-server.service; enabled)
     Active: active (running) since Fri 2026-07-04 10:10:00 JST

# UFW有効確認
$ sudo ufw status

# 各ポートを社内ネットワークに限定して開放する例（CIDRは環境に合わせて変更）
$ sudo ufw allow from 192.168.0.0/24 to any port 9090
$ sudo ufw allow from 192.168.0.0/24 to any port 9100
$ sudo ufw allow from 192.168.0.0/24 to any port 9835
$ sudo ufw allow from 192.168.0.0/24 to any port 3000

$ ollama --version
ollama version is 0.3.14

# 0.5.0未満なのでアップデートする
$ curl -fsSL https://ollama.com/install.sh | sh

$ ollama --version
ollama version is 0.5.7

# まずnvidia-smiが動くか確認
$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07    Driver Version: 550.90.07    CUDA Version: 12.4    |
|-------------------------------+----------------------+----------------------+

# nvidia_gpu_exporterのエラーをログで確認
$ journalctl -u nvidia_gpu_exporter -n 20 --no-pager

OllamaサーバーをGPU環境で構築・監視まで2日間のハンズオンで体験する

Prometheus監視スタックの構築からGrafanaダッシュボード設計まで一気通貫で習得したい方向けに、「ローカルAIマスターセミナー」を開催しています。
少人数（最大8名）ZOOMハンズオン形式で実施しています。