mooncake-track
/
Mooncake
0
0
Fork 12
Code Issues Pull Requests 13 Actions Packages Projects Releases Wiki Activity
Mooncake/docs/source/getting_started/observability.md

7.4 KiB
Raw Blame History

Observability

This document describes how to monitor and observe a running Mooncake Store deployment.

Master Metrics Log

When started, the Mooncake master periodically prints a metrics summary log every 10 seconds (configurable via kMetricReportIntervalSeconds). This log provides a comprehensive snapshot of the master's runtime state.

Log Format

I0512 15:03:30.321475 239489 rpc_service.cpp:269] Master Admin Metrics: role=leader, state=serving, service_ready=true, master={...}, ha={...}, leader=127.0.0.1:50051, view_version=1

Each log line contains:

  • GLog header: timestamp, thread ID, source file and line number
  • role: HA role — leader or standby
  • state: HA runtime state — serving, starting, stopping, etc.
  • service_ready: whether the gRPC service is accepting requests
  • master: master metrics block (see below)
  • ha: HA metrics block
  • leader: (only when available) the current leader address and view version

Master Metrics Block

A typical master={...} block looks like this:

Mem Storage: 94.09 MB / 100.00 MB (94.1%) | SSD Storage: 0 B / 0 B | Keys: 16058 (soft-pinned: 0) | Clients: 1 | Requests (Success/Total per sec): PutStart=0.00/0.00, PutEnd=0.00/0.00, PutRevoke=0.00/0.00, Get=0.00/0.00, Exist=0.00/0.00, Del=0.00/0.00, DelAll=0.00/0.00, Ping=1.00/1.00, CopyStart=0.00/0.00, CopyEnd=0.00/0.00, CopyRevoke=0.00/0.00, MoveStart=0.00/0.00, MoveEnd=0.00/0.00, MoveRevoke=0.00/0.00, EvictDiskReplica=0.00/0.00 | Batch Requests (per sec, Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=0.00/0.00/0.00, Item=0.00/0.00), PutEnd:(Req=0.00/0.00/0.00, Item=0.00/0.00), PutRevoke:(Req=0.00/0.00/0.00, Item=0.00/0.00), Get:(Req=0.00/0.00/0.00, Item=0.00/0.00), ExistKey:(Req=0.00/0.00/0.00, Item=0.00/0.00), QueryIp:(Req=0.00/0.00/0.00, Item=0.00/0.00), Clear:(Req=0.00/0.00/0.00, Item=0.00/0.00), CreateMoveTask:(Req=0.00/0.00), CreateCopyTask:(Req=0.00/0.00), QueryTask:(Req=0.00/0.00), FetchTasks:(Req=0.00/0.00), MarkTaskToComplete:(Req=0.00/0.00) | Eviction: Success/Attempts=0/0, AllocFail=0, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B | Snapshots: Success=0, Fail=0

Request counters are reported as rates per second over the time window between two consecutive log outputs (10 seconds by default). Real-time state values (storage, key count, client count, discard staging size) are not rate-limited and reflect the current value at log time.

The metrics block consists of the following sections:

Storage

Field Description
Mem Storage Current memory usage / total memory capacity, with percentage
SSD Storage Current SSD-backed storage usage / total SSD capacity

Keys and Clients

Field Description
Keys Total number of keys managed by the master
soft-pinned Number of keys with active soft-pin leases (protected from eviction)
Clients Number of currently connected clients

Requests (Success/Total per sec)

Rate counters for individual (non-batch) RPC requests over the last time window. Each shows <success_rate>/<total_rate> in requests per second:

Counter Description
PutStart Put object allocation requests
PutEnd Put object commit requests
PutRevoke Put object cancellation requests
Get Get replica list requests
Exist Key existence check requests
Del Single key deletion requests
DelAll Delete-all objects requests
Ping Client heartbeat/ping requests
CopyStart Copy object allocation requests
CopyEnd Copy object commit requests
CopyRevoke Copy object cancellation requests
MoveStart Move object allocation requests
MoveEnd Move object commit requests
MoveRevoke Move object cancellation requests
EvictDiskReplica Evict disk replica requests

Batch Requests (per sec)

Batch operations aggregate multiple items into a single RPC. Rates are per second over the last time window. Format: Req=<success>/<partial_success>/<total>, Item=<success_items>/<total_items>:

Counter Description
PutStart Batch put object allocation requests
PutEnd Batch put object commit requests
PutRevoke Batch put object cancellation requests
Get Batch get replica list requests
ExistKey Batch key existence check requests
QueryIp Batch query IP requests
Clear Batch replica clear requests

A request is considered "partial success" when it succeeds for some items but not all.

Task Operations

Counter Description
CreateMoveTask Move task creation requests
CreateCopyTask Copy task creation requests
QueryTask Task status query requests
FetchTasks Pending task fetch requests (polled by store clients)
MarkTaskToComplete Task completion acknowledgement requests

Eviction & Discard

Eviction counters are deltas between two consecutive log outputs — they show what happened in the time window, not cumulative totals.

Field Description
Eviction: Success/Attempts Eviction rounds that succeeded at least partially vs. total attempts in this window
AllocFail Number of PutStart/UpsertStart failures caused by replica allocation failure (triggers eviction) in this window
keys Number of keys evicted in this window
size Total size of evicted data in this window
Discard: Released/Total Released (cleaned up) vs. total discarded PutStart staging replicas (live values)
StagingSize Current size of discarded but not-yet-released staging buffers (live value)

Prometheus Metrics Endpoint

Mooncake master exposes Prometheus-format metrics at the HTTP admin endpoint. This allows integration with Prometheus, Grafana, or any Prometheus-compatible monitoring stack.

Endpoints

The admin HTTP server runs on metrics_port (default: 9003) and exposes the following endpoints:

Endpoint Content-Type Description
GET /metrics text/plain; version=0.0.4 All metrics in Prometheus exposition format
GET /metrics/summary text/plain; version=0.0.4 Human-readable summary (same content as the periodic log)
GET /health application/json Health check with role, HA state, and service readiness
GET /role text/plain Current HA role (leader / standby)
GET /ha_status text/plain Current HA runtime state (serving / starting / etc.)

Usage

Scrape the /metrics endpoint with Prometheus:

Add a scrape config to your prometheus.yml:

scrape_configs:
  - job_name: 'mooncake-master'
    static_configs:
      - targets: ['<master-host>:9003']
    metrics_path: '/metrics'

Quick check with curl:

# Get Prometheus metrics
curl http://<master-host>:9003/metrics

# Get human-readable summary
curl http://<master-host>:9003/metrics/summary

# Check health
curl http://<master-host>:9003/health

Configuration

The admin HTTP server is configured in the master config file (master.json or master.yaml):

{
  "enable_metric_reporting": true,
  "metrics_port": 9003,
  ...
}

Set enable_metric_reporting to false to disable the periodic metrics log. HTTP endpoints (/metrics, /health, etc.) remain available regardless of this setting.