natas_hw
/
dolphinscheduler0701
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
dolphinscheduler0701/docs/docs/en/guide/metrics/metrics.md

11 KiB
Raw Blame History

Introduction

Apache DolphinScheduler exports metrics for system observability. We use Micrometer as application metrics facade. Currently, we only support Prometheus Exporter but more are coming soon.

Quick Start

  • We enable Apache DolphinScheduler to export metrics in standalone mode to help users get hands dirty easily.
  • After triggering tasks in standalone mode, you could access metrics list by visiting url http://localhost:12345/dolphinscheduler/actuator/metrics.
  • After triggering tasks in standalone mode, you could access prometheus-format metrics by visiting url http://localhost:12345/dolphinscheduler/actuator/prometheus.
  • For a better experience with Prometheus and Grafana, we have prepared the out-of-the-box Grafana configurations for you, you could find the Grafana dashboards at dolphinscheduler-meter/resources/grafana and directly import these dashboards to your Grafana instance.
  • If you want to try with docker, you can use the following command to start the out-of-the-box Prometheus and Grafana:
cd dolphinscheduler-meter/src/main/resources/grafana-demo
docker compose up

then access the Grafana by the url: http://localhost:3001 for dashboards.

image.png image.png image.png

  • If you prefer to have some experiments in cluster mode, please refer to the Configuration section below:

Configuration

  • Please add the following config in master/worker/alert/api's yaml file to enable the metrics exporter.
metrics:
  enabled: true
  • Once the metrics exporter enabled, you could access the metrics by the url http://ip:port/actuator/prometheus.

The exporter port is the server.port defined in application.yaml, e.g: master: server.port: 5679, worker: server.port: 1235, alert: server.port: 50053, api: server.port: 12345.

For example, you can get the master metrics by curl http://localhost:5679/actuator/prometheus.

Naming Convention & Mapping

  • Naming of Apache DolphinScheduler metrics follows the officially-recommended approach by Micrometer
  • Micrometer automatically maps the metrics names to suit the external metrics system you configured. Currently, we only support Prometheus Exporter but more are coming soon.

Prometheus

  • all dots mapped to underscores
  • metric name starting with number added with prefix m_
  • COUNTER: add _total suffix if not ending with it
  • LONG_TASK_TIMER: _timer_seconds suffix added if not ending with them
  • GAUGE: _baseUnit suffix added if not ending with it

Dolphin Scheduler Metrics Cheatsheet

  • We categorize metrics by dolphin scheduler components such as master server, worker server, api server and alert server.
  • Although task / workflow related metrics exported by master server and worker server, we categorize them separately for users to query them more conveniently.

Task Related Metrics

  • ds.task.instance.count: (counter) the number of task instances, sliced by the tag state:
    • timeout: the number of timeout tasks
    • finish: the number of finished tasks, both succeeded and failed included
    • success: the number of successful tasks
    • fail: the number of failed tasks
    • stop: the number of stopped tasks
    • retry: the number of retried tasks
    • submit: the number of submitted tasks
    • failover: the number of task fail-overs
  • ds.task.dispatch.count: (counter) the number of tasks dispatched to worker
  • ds.task.dispatch.failure.count: (counter) the number of tasks failed to dispatch, retry failure included
  • ds.task.dispatch.error.count: (counter) the number of task dispatch errors
  • ds.task.execution.count.by.type: (counter) the number of task executions grouped by tag task_type
  • ds.task.prepared: (gauge) the number of tasks prepared for task queue
  • ds.task.execution.count: (counter) the number of executed tasks
  • ds.task.execution.duration: (histogram) duration of task executions

Workflow Related Metrics

  • ds.workflow.create.command.count: (counter) the number of commands created and inserted by workflows
  • ds.workflow.instance.submit.count: (counter) the number of submitted workflow instances
  • ds.workflow.instance.running: (gauge) the number of running workflow instances
  • ds.workflow.instance.count: (counter) the number of workflow instances, sliced by tags process.definition.code and state. To monitor a specific workflow, you could filter the metrics by tag process.definition.code, which refers to the definition code of your workflow. There are seven different states for workflow instances as follows:
    • submit: the number of submitted workflow instances
    • timeout: the number of timeout workflow instances
    • finish: the number of finished workflow instances, both successes and failures included
    • success: the number of successful workflow instances
    • fail: the number of failed workflow instances
    • stop: the number of stopped workflow instances
    • failover: the number of workflow instance fail-overs

RPC Related Metrics

  • ds.rpc.client.sync.request.exception.count: (counter) the number of exceptions occurred in sync rpc requests
  • ds.rpc.client.sync.request.duration.time: (histogram) the time cost of sync rpc requests

Master Server Metrics

  • ds.master.overload.count: (counter) the number of times the master overloaded
  • ds.master.consume.command.count: (counter) the number of commands consumed by master
  • ds.master.scheduler.failover.check.count: (counter) the number of scheduler (master) fail-over checks
  • ds.master.scheduler.failover.check.time: (histogram) the total time cost of scheduler (master) fail-over checks
  • ds.master.quartz.job.executed: the total number of quartz jobs executed
  • ds.master.quartz.job.execution.time: the total execution time of quartz jobs

Worker Server Metrics

  • ds.worker.overload.count: (counter) the number of times the worker overloaded
  • ds.worker.task: (gauge) the number of tasks on the worker, including pending and running ones
  • ds.worker.execute.queue.size: (gauge) the number of pending tasks on the worker
  • ds.worker.active.execute.thread: (gauge) the number of running tasks on the worker
  • ds.worker.memory.available: (gauge) the available physical memory of the worker (GB)
  • ds.worker.cpu.usage: (gauge) the cpu usage percentage of the worker
  • ds.worker.memory.usage: (gauge) the memory usage percentage of the worker
  • ds.worker.full.submit.queue.count: (counter) the number of times the worker's submit queue being full
  • ds.worker.resource.download.count: (counter) the number of downloaded resource files on workers, sliced by tag status
  • ds.worker.resource.download.duration: (histogram) the time cost of resource download on workers
  • ds.worker.resource.download.size: (histogram) the sizes of downloaded resource files on workers (bytes)

Api Server Metrics

  • ds.api.request.count: (counter) the number of requests received by the api server
  • ds.api.response.count: (counter) the number of responses received by the api server, sliced by tag code
  • ds.api.response.time: (timer) the response time distribution of the api server, sliced by tag user_id
  • ds.api.resource.upload.size: (histogram) size distribution of resource files uploaded by the api server (bytes)
  • ds.api.resource.download.size: (histogram) size distribution of resource files download by the api server (bytes)

Alert Server Related

  • ds.alert.send.count: (counter) the number of sent alerts, sliced by tag status
  • ds.alert.pending: (gauge) the number of alerts waiting to get fired

In each server, there are some default system-level metrics related to database connection, JVM, etc. We list them below for your reference:

Database Related Metrics (Default)

  • hikaricp.connections: the total number of connections
  • hikaricp.connections.creation: connection creation time (max, count, sum included)
  • hikaricp.connections.acquire: connection acquirement time (max, count, sum included)
  • hikaricp.connections.usage: connection usage time (max, count, sum included)
  • hikaricp.connections.max: the max number of connections
  • hikaricp.connections.min: the min number of connections
  • hikaricp.connections.active: the number of active connections
  • hikaricp.connections.idle: the number of idle connections
  • hikaricp.connections.pending: the number of pending connections
  • hikaricp.connections.timeout: the total number of timeout connections
  • jdbc.connections.max: the max number of active connections that can be allocated at the same time
  • jdbc.connections.min: the min number of idle connections in the pool
  • jdbc.connections.idle: the number of established but idle connections
  • jdbc.connections.active: the current number of active connections allocated from the data source

JVM Related Metrics (Default)

  • jvm.buffer.total.capacity: an estimate of the total capacity of the buffers in the pool
  • jvm.buffer.count: an estimate of the number of buffers in the pool
  • jvm.buffer.memory.used: an estimate of the memory that the JVM is using for this buffer pool
  • jvm.memory.committed: the amount of memory in bytes committed for the JVM to use
  • jvm.memory.max: the maximum amount of memory in bytes that can be used for memory management
  • jvm.memory.used: the amount of used memory
  • jvm.threads.peak: the peak live thread count since the JVM started or peak reset
  • jvm.threads.states: the current number of threads having NEW state
  • jvm.gc.memory.allocated: incremented for an increase in the size of the (young) heap memory pool after one GC to before the next
  • jvm.gc.max.data.size: max size of long-lived heap memory pool
  • jvm.gc.pause: time spent in GC pause (count, sum, max included)
  • jvm.gc.live.data.size: the size of long-lived heap memory pool after reclamation
  • jvm.gc.memory.promoted: the count of positive increases in the size of the old generation memory pool before GC to after GC.
  • jvm.classes.loaded: the number of classes currently loaded in the JVM
  • jvm.threads.live: the current number of live threads including both daemon and non-daemon threads
  • jvm.threads.daemon: the current number of live daemon threads
  • jvm.classes.unloaded: the total number of classes unloaded since the JVM started
  • process.cpu.usage: the recent cpu usage for the JVM process
  • process.start.time: start time of the process since unix epoch
  • process.uptime: the uptime of the JVM

Others (Default)

  • jetty.threads.config.max: the max number of threads in the pool
  • jetty.threads.config.min: the min number of threads in the pool
  • jetty.threads.current: the total number of threads in the pool
  • jetty.threads.idle: the number of idle threads in the pool
  • jetty.threads.busy: the number of busy threads in the pool
  • jetty.threads.jobs: number of queued jobs waiting for a thread
  • process.files.max: the max number of file descriptors
  • process.files.open: the number of open file descriptors
  • system.cpu.usage: the recent cpu usage for the whole system
  • system.cpu.count: the number of processors available to the JVM
  • system.load.average.1m: the total number of runnable entities queued to available processors and runnable entities running on the available processors averaged over a period
  • logback.events: the number of events that made it to the logs grouped by the tag level
  • http.server.requests: total number of http requests