Compare commits

...

69 Commits

Author SHA1 Message Date
Amy0104 836f58de05
[Feature][Doc] Added change log. (#13128) 2022-12-08 11:36:06 +08:00
Jay Chung 418cef3421 [fix][doc] python api release link 2022-11-26 15:22:41 +08:00
Tq 2202b83996
[fix][doc] Fix sql-hive and hive-cli doc (#12765) (#12807) 2022-11-08 13:54:19 +08:00
caishunfeng 7e39396a76 [maven-release-plugin] prepare for next development iteration 2022-09-26 21:50:03 +08:00
caishunfeng ae33ba5947 [maven-release-plugin] prepare release 3.1.0 2022-09-26 21:49:27 +08:00
Devosend beebc5e0ad fix env config space bug (#12147) 2022-09-26 19:06:46 +08:00
caishunfeng f982e1d2a2
[3.1.0-prepare] Update version to 310 (#12146)
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-09-26 18:26:30 +08:00
Tq 3efe4bc308 [BUG][BE-12113]fix htrace dependency conflicts (#12126)
* fix htrace dependency conflicts

* replace htrace-core with hbase-noop-htrace
2022-09-26 13:49:11 +08:00
陈家名 399f62cba2 [bug][k8s] Change the config file keys in K8s to be the same as those in common.properties (#11812) 2022-09-26 12:05:46 +08:00
陈家名 7a85b930d7 [k8s] Delete unnecessary configurations. (#11808) 2022-09-26 12:05:17 +08:00
caishunfeng a5a941e079
fix upgrade for 310 (#12137) 2022-09-26 10:16:12 +08:00
Yann Ann 3e899bee06 [fix#12000]Cannot remove the WorkerGroup from the master service (#12050)
* [Bug] [Master] Cannot remove the WorkerGroup from the master service. #12000

* remove unnecessary locks

* Update dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/registry/ServerNodeManager.java

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-09-25 23:18:09 +08:00
Eric Gao 42d8308940 [Doc][Improvement] Add instructions for process execution type and data complement (#12121)
* Add instructions for process execution type and data complement

* Fix and polish the docs
2022-09-25 22:58:32 +08:00
Jiajie Zhong b0b29ed8e1 [fix][python] Task switch branch not show in webui (#12120)
when create task switch in python api, switch branch not show correctly,
due to we add some unnecessary attribute to switch task, this patch we
add `_task_ignore_attr` in class `Task` to ignore those unnecessary
attribute from `_task_default_attr`

(cherry picked from commit a86f4e2693)
2022-09-23 19:46:38 +08:00
Jiajie Zhong 71c51c3c3d [Fix] Also auto formatter workflow instance if location is null (#12080)
* [fix] Also auto formatter workflow instance if location is null

In #11535(096fae77) and #11681(4dca488c), we already formatter
workflow definition, but I find out we forget formatter workflow
instance with definition's location is null, this patch also
formatter workflow instance.

* remove console log

(cherry picked from commit 2531c222de)
2022-09-23 19:44:20 +08:00
JieguangZhou f871a4a41f fix python home conflict (#12112)
(cherry picked from commit 6eb1eb722a)
2022-09-23 19:43:56 +08:00
Amy0104 c47e088b73
[Fix][UI] Fix workflow name overlaps bug (#11862) (#12125)
Co-authored-by: Devosend <devosend@gmail.com>
2022-09-23 17:40:13 +08:00
jackfanwan 1e59250055 [BUG]Delete Setting ID (#12096)
* Delete Setting ID

* modify id

Co-authored-by: jackfanwan <fanwanlong@kezaihui.com>
2022-09-22 15:21:24 +08:00
Wenjun Ruan 1a63f8672a
Fix insert command error due to the id is not null (#12092) (#12098)
(cherry picked from commit fba5a8eaa0)
2022-09-22 15:21:07 +08:00
Eric Gao e7b12bf205
[Cherry-Pick][3.1.0-prepare] Cherry-Pick OssOperator for Resource Center to 3.1.0 (#12094)
* Cherry OssOperator for Resource Center to 3.1.0

* Fix doc dead link
2022-09-22 15:00:25 +08:00
Eric Gao 0d16d7b323
[Cherry-Pick][3.1.0-Prepare] Cherry pick docs format fix for 3.1.0 (#12095)
* Cherry pick docs format fix for 3.1.0

* Fix doc dead link

* Fix sphinx dead link
2022-09-22 14:11:43 +08:00
JieguangZhou a2b3659fe9 [improve] Optimize MLFlow task plugin for easy of use (#12071)
optimize code

(cherry picked from commit cf522e2f)
2022-09-21 13:50:48 +08:00
limaiwang f0fda2a9aa add(dependent node's add unprivileged project list): dependent node's add unprivileged project list (#12073) 2022-09-21 10:40:12 +08:00
jackfanwan 280b7c8545 [Improvement] Add Set cluster name (#12074) 2022-09-21 10:17:04 +08:00
jackfanwan cda3110409 [improvement]Add Set cluster name (#12058)
* Add Set cluster name

* add unit test

Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
2022-09-21 10:16:39 +08:00
Wenjun Ruan c286c5567a
Script cannot contains ''' in params (#12068)
(cherry picked from commit f40a831453)
2022-09-21 08:51:44 +08:00
Wenjun Ruan f4babb773e
Fix update TaskDefinition error (#12064)
(cherry picked from commit 96c12f9592)
2022-09-20 22:24:11 +08:00
rickchengx 806ad1d98c [Improvement-11291][sql] Improvement the log output while executing multiple sql statements (#11292) 2022-09-20 10:47:18 +08:00
songjianet 23d77f8a2a [Feat][UI] Update package. (#12046)
* [Feat][UI] Update package.

* [Feat][UI] Update package.
2022-09-20 10:46:10 +08:00
juzimao 34950d3c09 and task instance date range search (#11963) 2022-09-20 10:45:00 +08:00
caishunfeng 780a509f67 fix workflow keep running when task fail (#11930) 2022-09-19 17:39:18 +08:00
Wenjun Ruan 1513363eed Fix cannot save processDefinition (#11931) 2022-09-19 17:38:46 +08:00
caishunfeng 704043f229 fix import error 2022-09-19 17:07:02 +08:00
Stalary d7f40b19b5 [Bug](dependent) Dependent downstream trigger error when schedule cycle not day. (#11734)
* FIX: dependent

* FIX: version

* MOD: for review
2022-09-19 17:01:14 +08:00
kezhenxu94 775ef98b64 Add validations of possible malicious keys (#11966) 2022-09-19 16:32:15 +08:00
Wenjun Ruan 6f7ba2c634 Add debug logs for properties read from `common.properties` (#11910) 2022-09-19 16:17:10 +08:00
Stalary 714e258be6 [Bug][Dependent]: Id also clone due to duplicate when use dependent mode. (#11929) 2022-09-19 16:14:23 +08:00
juzimao 8c6658e3f9 Optimize date range condition (#11911) 2022-09-19 16:12:34 +08:00
Daniel Y 2a437607ae [Improvement-11834] Upgrade docker base image to support python3.9 (#11835)
* use openjdk:8-jre-slim-bullseye instead of buster to support python3.9

* use openjdk:8-jre-slim-bullseye instead of buster across all components
2022-09-19 15:50:25 +08:00
Wenjun Ruan 7ab4412b5e Fix listing processDefinition the schedulerReleaseState will never be null (#11888) 2022-09-19 15:39:10 +08:00
W.XING 9ad6a049c5 Add chunjun in dolphinscheduler-task-all/pom.xml (#11375) 2022-09-19 15:34:54 +08:00
ouyangyewei 7b58737e22 Add default constructor (#11858) 2022-09-19 15:32:07 +08:00
LiuBodong 0f3b42925f [Fix][Monitor]Monitor UI not show DisakAvailable and MemoryUsage correctly (#11870) 2022-09-19 15:30:49 +08:00
kezhenxu94 89b4192d3d Remove htrace dependencies (#11881)
* Remove htrace dependencies

* Update pom.xml
2022-09-19 15:27:41 +08:00
Kengo Seki 216ceea641 [Improve] Upgrade Hadoop to 3.2.4 (#11832) 2022-09-19 15:26:58 +08:00
Yann Ann c11ec05d61 [Improvement-#11768][Monitor] Support monitor h2 database in monitor page (#11813) 2022-09-19 15:23:11 +08:00
kezhenxu94 fcc75ef1c6 Bump up dependencies to fix cves (#11765) 2022-09-19 15:10:56 +08:00
Wenjun Ruan 1aba077fb8 Remove `not use web service` comment in alert server (#11819) 2022-09-19 14:50:16 +08:00
Kengo Seki 52b79b017e [Improvement] Replace commons-lang 2 function invocations with commons-lang3 (#11810)
* [Improvement] Replace commons-lang 2 function invocations with commons-lang3
2022-09-19 14:48:17 +08:00
Jiajie Zhong ed33066178 [python] Fix permission error when create related objects (#11783)
after we add permission to our codebase in dev branch, will got error
when create objects from pydolphinscheduler, because users with user_id
equal to `Integer.MAX_VALUE` is ordinary users which can not have access
to create unless we grant permission to the role manually. this patch
change dummyUser to admin user's user_id == 1 to fix it
2022-09-19 14:46:26 +08:00
fuchanghai f1de1707d5 Update dolphinscheduler_ddl.sql fix alert t_ds_user add time_zone is not correct (#11795) 2022-09-19 14:41:50 +08:00
caishunfeng 113dc4b5c5
[Feature-10273][Datasource] Add dataSource for AWS Athena (#11157) (#12039)
Co-authored-by: ZhaoGuodong <guodongym@163.com>
2022-09-19 13:45:19 +08:00
JieguangZhou dde6f63c31 [Feature][PyDolphinScheduler] Support DVC task in pyds #11922 (#11941)
* add dvc task in pyds

* add BaseDVC class

(cherry picked from commit bbc228e93f)
2022-09-18 16:49:34 +08:00
JieguangZhou cb063732d7 [feat][python] Support MLflow task in python api (#11962)
(cherry picked from commit ad683c3c42)
2022-09-18 16:46:45 +08:00
JieguangZhou f66dedc9da [feat][python] Support OpenMLDB task in python api (#11944)
Using the whole word zookeeper instead of short cut zk
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>

(cherry picked from commit e50a32d9cc)
2022-09-18 16:43:36 +08:00
JieguangZhou 1ac2e4a8f3 [feat][python] Add sagemaker yaml example (#11925)
(cherry picked from commit e05f14ba23)
2022-09-18 16:43:24 +08:00
JieguangZhou 6f3b4c1624 [feat][python] Support Pytorch task in python api (#11975)
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
(cherry picked from commit 5202e5cfc6)
2022-09-18 16:43:08 +08:00
JieguangZhou e462918ac9
[Feat][python] Add parameter environment to task (#11763) (#11989)
(cherry picked from commit 8988492c43)
2022-09-17 10:24:07 +08:00
Wenjun Ruan 1ea2e848cd
[Feature][Registry] Support etcd as registry (#10981) (#11997)
(cherry picked from commit 2eb8b9f7f0)

Co-authored-by: wjf <jffwang@qq.com>
2022-09-17 09:01:24 +08:00
Wenjun Ruan 8eadf5e5aa
Fix insertOrUpdate plugin may failed due to concurrent operation (#11471) (#11996)
(cherry picked from commit 864a90820d)
2022-09-17 08:59:45 +08:00
JieguangZhou 5e42f52bdf
[3.1.0-prepare][cherry-pick][DSIP-11][python] create workflows from YAML configuration (#11988)
* [DSIP-11][python] create workflows from YAML configuration (#11611)

(cherry picked from commit 38ee91fb1e)

* fix pydolphin yaml doc link (#11817)

(cherry picked from commit c41fa5a8b1)
2022-09-17 08:56:37 +08:00
caishunfeng c4953e8660
Example Change the regular expression of the data source password (#11459) (#11990)
Co-authored-by: jackfanwan <61672564+jackfanwan@users.noreply.github.com>
2022-09-16 15:16:05 +08:00
caishunfeng 951f707b61
[fix][doc] Update the registry related configuration in values.yaml (#11444) (#11981)
Co-authored-by: rickchengx <38122586+rickchengx@users.noreply.github.com>
2022-09-16 11:17:33 +08:00
caishunfeng cd8f32d876
[Bug][Workflow Definition] fix ordinary users can not create depend task (#11961) (#11977) 2022-09-16 09:42:59 +08:00
caishunfeng f034a09d25
[Bug-11650][worker] #11650 fix SQL type task, stop task cause NPE (#11668) (#11958)
Co-authored-by: 冯剑 <35831367+fengjian1129@users.noreply.github.com>
2022-09-15 14:24:34 +08:00
caishunfeng 0647b3e10c
[Cherry-pick][Improvement] [Doc] Put default task parameters in a new file #11776 (#11957)
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
2022-09-15 14:21:05 +08:00
caishunfeng 27b69e608a
[Bug] [Master] WorkflowExecuteRunnable will face a infinite loop #11838 (#11864) (#11950)
* [Bug] [Master] WorkflowExecuteRunnable will face a infinite loop #11838

Co-authored-by: Yann Ann <2993643785@qq.com>
2022-09-15 10:39:00 +08:00
caishunfeng 0f6cc4fe33
[Fix][db] fix init&upgrade mysql-meta-schema bugs (#11887) (#11935)
Co-authored-by: wendongdi <49931055+wendongdi@users.noreply.github.com>
2022-09-14 18:49:49 +08:00
caishunfeng 8d1c2d3eeb
cherry-pick [Bug] Fix procedure task param pass #11919 (#11926) 2022-09-14 15:44:01 +08:00
764 changed files with 25202 additions and 17789 deletions

View File

@ -1,6 +1,5 @@
<!--Thanks very much for contributing to Apache DolphinScheduler. Please review https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html before opening a pull request.-->
## Purpose of the pull request
<!--(For example: This pull request adds checkstyle plugin).-->
@ -8,8 +7,9 @@
## Brief change log
<!--*(for example:)*
- *Add maven-checkstyle-plugin to root pom.xml*
- *Add maven-checkstyle-plugin to root pom.xml*
-->
## Verify this pull request
<!--*(Please pick either of the following options)*-->
@ -25,9 +25,9 @@ This pull request is already covered by existing tests, such as *(please describ
This change added tests and can be verified as follows:
<!--*(example:)*
- *Added dolphinscheduler-dao tests for end-to-end.*
- *Added CronUtilsTest to verify the change.*
- *Manually verified the change by testing locally.* -->
- *Added dolphinscheduler-dao tests for end-to-end.*
- *Added CronUtilsTest to verify the change.*
- *Manually verified the change by testing locally.* -->
(or)

View File

@ -15,7 +15,7 @@
# limitations under the License.
#
FROM openjdk:8-jre-slim-buster
FROM openjdk:8-jre-slim-bullseye
RUN apt update ; \
apt install -y curl wget default-mysql-client sudo openssh-server netcat-traditional ;

View File

@ -15,7 +15,7 @@
# limitations under the License.
#
FROM openjdk:8-jre-slim-buster
FROM openjdk:8-jre-slim-bullseye
RUN apt update ; \
apt install -y curl wget sudo openssh-server netcat-traditional ;

View File

@ -6,11 +6,11 @@ Start by forking the dolphinscheduler GitHub repository, make changes in a branc
There are three branches in the remote repository currently:
- `master` : normal delivery branch. After the stable version is released, the code for the stable version branch is merged into the master branch.
- `master` : normal delivery branch. After the stable version is released, the code for the stable version branch is merged into the master branch.
- `dev` : daily development branch. The daily development branch, the newly submitted code can pull requests to this branch.
- `dev` : daily development branch. The daily development branch, the newly submitted code can pull requests to this branch.
- `x.x.x-release` : the stable release version.
- `x.x.x-release` : the stable release version.
So, you should fork the `dev` branch.
@ -40,7 +40,6 @@ There will be two repositories at this time: origin (your own warehouse) and ups
Get/update remote repository code (already the latest code, skip it).
```sh
git fetch upstream
```
@ -91,7 +90,6 @@ After submitting changes to your remote repository, you should click on the new
<img src = "http://geek.analysys.cn/static/upload/221/2019-04-02/90f3abbf-70ef-4334-b8d6-9014c9cf4c7f.png" width ="60%"/>
</p>
Select the modified local branch and the branch to merge past to create a pull request.
<p align = "center">

View File

@ -1,6 +1,6 @@
Dolphin Scheduler Official Website
[dolphinscheduler.apache.org](https://dolphinscheduler.apache.org)
============
==================================================================
[![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html)
[![codecov](https://codecov.io/gh/apache/dolphinscheduler/branch/dev/graph/badge.svg)](https://codecov.io/gh/apache/dolphinscheduler/branch/dev)
@ -8,9 +8,6 @@ Dolphin Scheduler Official Website
[![Twitter Follow](https://img.shields.io/twitter/follow/dolphinschedule.svg?style=social&label=Follow)](https://twitter.com/dolphinschedule)
[![Slack Status](https://img.shields.io/badge/slack-join_chat-white.svg?logo=slack&style=social)](https://s.apache.org/dolphinscheduler-slack)
[![Stargazers over time](https://starchart.cc/apache/dolphinscheduler.svg)](https://starchart.cc/apache/dolphinscheduler)
[![EN doc](https://img.shields.io/badge/document-English-blue.svg)](README.md)
@ -21,35 +18,35 @@ Dolphin Scheduler Official Website
DolphinScheduler is a distributed and extensible workflow scheduler platform with powerful DAG visual interfaces, dedicated to solving complex job dependencies in the data pipeline and providing various types of jobs available `out of the box`.
Its main objectives are as follows:
- Highly Reliable,
- Highly Reliable,
DolphinScheduler adopts a decentralized multi-master and multi-worker architecture design, which naturally supports easy expansion and high availability (not restricted by a single point of bottleneck), and its performance increases linearly with the increase of machines
- High performance, supporting tens of millions of tasks every day
- Support multi-tenant.
- Cloud Native, DolphinScheduler supports multi-cloud/data center workflow management, also
- High performance, supporting tens of millions of tasks every day
- Support multi-tenant.
- Cloud Native, DolphinScheduler supports multi-cloud/data center workflow management, also
supports Kubernetes, Docker deployment and custom task types, distributed
scheduling, with overall scheduling capability increased linearly with the
scale of the cluster
- Support various task types: Shell, MR, Spark, SQL (MySQL, PostgreSQL, hive, spark SQL), Python, Sub_Process, Procedure, etc.
- Support scheduling of workflows and dependencies, manual scheduling to pause/stop/recover task, support failure task retry/alarm, recover specified nodes from failure, kill task, etc.
- Associate the tasks according to the dependencies of the tasks in a DAG graph, which can visualize the running state of the task in real-time.
- WYSIWYG online editing tasks
- Support the priority of workflows & tasks, task failover, and task timeout alarm or failure.
- Support workflow global parameters and node customized parameter settings.
- Support online upload/download/management of resource files, etc. Support online file creation and editing.
- Support task log online viewing and scrolling and downloading, etc.
- Support the viewing of Master/Worker CPU load, memory, and CPU usage metrics.
- Support displaying workflow history in tree/Gantt chart, as well as statistical analysis on the task status & process status in each workflow.
- Support back-filling data.
- Support internationalization.
- More features waiting for partners to explore...
- Support various task types: Shell, MR, Spark, SQL (MySQL, PostgreSQL, hive, spark SQL), Python, Sub_Process, Procedure, etc.
- Support scheduling of workflows and dependencies, manual scheduling to pause/stop/recover task, support failure task retry/alarm, recover specified nodes from failure, kill task, etc.
- Associate the tasks according to the dependencies of the tasks in a DAG graph, which can visualize the running state of the task in real-time.
- WYSIWYG online editing tasks
- Support the priority of workflows & tasks, task failover, and task timeout alarm or failure.
- Support workflow global parameters and node customized parameter settings.
- Support online upload/download/management of resource files, etc. Support online file creation and editing.
- Support task log online viewing and scrolling and downloading, etc.
- Support the viewing of Master/Worker CPU load, memory, and CPU usage metrics.
- Support displaying workflow history in tree/Gantt chart, as well as statistical analysis on the task status & process status in each workflow.
- Support back-filling data.
- Support internationalization.
- More features waiting for partners to explore...
## What's in DolphinScheduler
Stability | Accessibility | Features | Scalability |
--------- | ------------- | -------- | ------------|
Decentralized multi-master and multi-worker | Visualization of workflow key information, such as task status, task type, retry times, task operation machine information, visual variables, and so on at a glance.  |  Support pause, recover operation | Support customized task types
support HA | Visualization of all workflow operations, dragging tasks to draw DAGs, configuring data sources and resources. At the same time, for third-party systems, provide API mode operations. | Users on DolphinScheduler can achieve many-to-one or one-to-one mapping relationship through tenants and Hadoop users, which is very important for scheduling large data jobs. | The scheduler supports distributed scheduling, and the overall scheduling capability will increase linearly with the scale of the cluster. Master and Worker support dynamic adjustment.
Overload processing: By using the task queue mechanism, the number of schedulable tasks on a single machine can be flexibly configured. Machine jam can be avoided with high tolerance to numbers of tasks cached in task queue. | One-click deployment | Support traditional shell tasks, and big data platform task scheduling: MR, Spark, SQL (MySQL, PostgreSQL, hive, spark SQL), Python, Procedure, Sub_Process | |
| Stability | Accessibility | Features | Scalability |
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Decentralized multi-master and multi-worker | Visualization of workflow key information, such as task status, task type, retry times, task operation machine information, visual variables, and so on at a glance.  |  Support pause, recover operation | Support customized task types |
| support HA | Visualization of all workflow operations, dragging tasks to draw DAGs, configuring data sources and resources. At the same time, for third-party systems, provide API mode operations. | Users on DolphinScheduler can achieve many-to-one or one-to-one mapping relationship through tenants and Hadoop users, which is very important for scheduling large data jobs. | The scheduler supports distributed scheduling, and the overall scheduling capability will increase linearly with the scale of the cluster. Master and Worker support dynamic adjustment. |
| Overload processing: By using the task queue mechanism, the number of schedulable tasks on a single machine can be flexibly configured. Machine jam can be avoided with high tolerance to numbers of tasks cached in task queue. | One-click deployment | Support traditional shell tasks, and big data platform task scheduling: MR, Spark, SQL (MySQL, PostgreSQL, hive, spark SQL), Python, Procedure, Sub_Process | |
## User Interface Screenshots

View File

@ -1,12 +1,11 @@
Dolphin Scheduler Official Website
[dolphinscheduler.apache.org](https://dolphinscheduler.apache.org)
============
==================================================================
[![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html)
[![codecov](https://codecov.io/gh/apache/dolphinscheduler/branch/dev/graph/badge.svg)](https://codecov.io/gh/apache/dolphinscheduler/branch/dev)
[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=apache-dolphinscheduler&metric=alert_status)](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler)
[![Stargazers over time](https://starchart.cc/apache/dolphinscheduler.svg)](https://starchart.cc/apache/dolphinscheduler)
[![CN doc](https://img.shields.io/badge/文档-中文版-blue.svg)](README_zh_CN.md)
@ -18,20 +17,20 @@ Dolphin Scheduler Official Website
其主要目标如下:
- 以DAG图的方式将Task按照任务的依赖关系关联起来可实时可视化监控任务的运行状态
- 支持丰富的任务类型Shell、MR、Spark、SQL(mysql、postgresql、hive、sparksql)、Python、Sub_Process、Procedure等
- 支持工作流定时调度、依赖调度、手动调度、手动暂停/停止/恢复,同时支持失败重试/告警、从指定节点恢复失败、Kill任务等操作
- 支持工作流优先级、任务优先级及任务的故障转移及任务超时告警/失败
- 支持工作流全局参数及节点自定义参数设置
- 支持资源文件的在线上传/下载,管理等,支持在线文件创建、编辑
- 支持任务日志在线查看及滚动、在线下载日志等
- 实现集群HA通过Zookeeper实现Master集群和Worker集群去中心化
- 支持对`Master/Worker` cpu loadmemorycpu在线查看
- 支持工作流运行历史树形/甘特图展示、支持任务状态统计、流程状态统计
- 支持补数
- 支持多租户
- 支持国际化
- 还有更多等待伙伴们探索
- 以DAG图的方式将Task按照任务的依赖关系关联起来可实时可视化监控任务的运行状态
- 支持丰富的任务类型Shell、MR、Spark、SQL(mysql、postgresql、hive、sparksql)、Python、Sub_Process、Procedure等
- 支持工作流定时调度、依赖调度、手动调度、手动暂停/停止/恢复,同时支持失败重试/告警、从指定节点恢复失败、Kill任务等操作
- 支持工作流优先级、任务优先级及任务的故障转移及任务超时告警/失败
- 支持工作流全局参数及节点自定义参数设置
- 支持资源文件的在线上传/下载,管理等,支持在线文件创建、编辑
- 支持任务日志在线查看及滚动、在线下载日志等
- 实现集群HA通过Zookeeper实现Master集群和Worker集群去中心化
- 支持对`Master/Worker` cpu loadmemorycpu在线查看
- 支持工作流运行历史树形/甘特图展示、支持任务状态统计、流程状态统计
- 支持补数
- 支持多租户
- 支持国际化
- 还有更多等待伙伴们探索
## 系统部分截图

View File

@ -2,3 +2,4 @@
* [Start Up DolphinScheduler with Docker](https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/guide/start/docker.html)
* [Start Up DolphinScheduler with Kubernetes](https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/guide/installation/kubernetes.html)

View File

@ -15,8 +15,8 @@
# specific language governing permissions and limitations
# under the License.
#
HUB=ghcr.io/apache/dolphinscheduler
TAG=latest
HUB=apache
TAG=3.1.0
TZ=Asia/Shanghai
DATABASE=postgresql

View File

@ -39,7 +39,7 @@ version: 2.0.0
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application.
appVersion: dev-SNAPSHOT
appVersion: 3.1.0
dependencies:
- name: postgresql

View File

@ -182,19 +182,6 @@ Create a registry environment variables.
{{- end }}
{{- end -}}
{{/*
Create a common fs_s3a environment variables.
*/}}
{{- define "dolphinscheduler.fs_s3a.env_vars" -}}
{{- if eq (default "HDFS" .Values.common.configmap.RESOURCE_STORAGE_TYPE) "S3" -}}
- name: FS_S3A_SECRET_KEY
valueFrom:
secretKeyRef:
key: fs-s3a-secret-key
name: {{ include "dolphinscheduler.fullname" . }}-fs-s3a
{{- end -}}
{{- end -}}
{{/*
Create a sharedStoragePersistence volume.
*/}}

View File

@ -23,7 +23,7 @@ metadata:
app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-common
{{- include "dolphinscheduler.common.labels" . | nindent 4 }}
data:
{{- range $key, $value := (omit .Values.common.configmap "FS_S3A_SECRET_KEY") }}
{{- range $key, $value := .Values.common.configmap }}
{{ $key }}: {{ $value | quote }}
{{- end }}
{{- end }}

View File

@ -71,7 +71,6 @@ spec:
value: {{ .Values.timezone }}
{{- include "dolphinscheduler.database.env_vars" . | nindent 12 }}
{{- include "dolphinscheduler.registry.env_vars" . | nindent 12 }}
{{- include "dolphinscheduler.fs_s3a.env_vars" . | nindent 12 }}
{{ range $key, $value := .Values.api.env }}
- name: {{ $key }}
value: {{ $value | quote }}

View File

@ -46,7 +46,6 @@ spec:
value: {{ .Values.timezone }}
{{- include "dolphinscheduler.database.env_vars" . | nindent 12 }}
{{- include "dolphinscheduler.registry.env_vars" . | nindent 12 }}
{{- include "dolphinscheduler.fs_s3a.env_vars" . | nindent 12 }}
envFrom:
- configMapRef:
name: {{ include "dolphinscheduler.fullname" . }}-common

View File

@ -1,28 +0,0 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
{{- if eq (default "HDFS" .Values.common.configmap.RESOURCE_STORAGE_TYPE) "S3" }}
apiVersion: v1
kind: Secret
metadata:
name: {{ include "dolphinscheduler.fullname" . }}-fs-s3a
labels:
app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-fs-s3a
{{- include "dolphinscheduler.common.labels" . | nindent 4 }}
type: Opaque
data:
fs-s3a-secret-key: {{ .Values.common.configmap.FS_S3A_SECRET_KEY | b64enc | quote }}
{{- end }}

View File

@ -66,7 +66,6 @@ spec:
value: {{ .Values.timezone }}
{{- include "dolphinscheduler.database.env_vars" . | nindent 12 }}
{{- include "dolphinscheduler.registry.env_vars" . | nindent 12 }}
{{- include "dolphinscheduler.fs_s3a.env_vars" . | nindent 12 }}
{{ range $key, $value := .Values.master.env }}
- name: {{ $key }}
value: {{ $value | quote }}

View File

@ -68,7 +68,6 @@ spec:
value: {{ include "dolphinscheduler.fullname" . }}-alert
{{- include "dolphinscheduler.database.env_vars" . | nindent 12 }}
{{- include "dolphinscheduler.registry.env_vars" . | nindent 12 }}
{{- include "dolphinscheduler.fs_s3a.env_vars" . | nindent 12 }}
{{ range $key, $value := .Values.worker.env }}
- name: {{ $key }}
value: {{ $value | quote }}

View File

@ -23,7 +23,7 @@ timezone: "Asia/Shanghai"
image:
registry: "dolphinscheduler.docker.scarf.sh/apache"
tag: "dev-SNAPSHOT"
tag: "3.1.0"
pullPolicy: "IfNotPresent"
pullSecret: ""
@ -74,8 +74,8 @@ conf:
# resource storage type: HDFS, S3, NONE
resource.storage.type: HDFS
# resource store on HDFS/S3 path, resource file will store to this hadoop hdfs path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended
resource.upload.path: /dolphinscheduler
# resource store on HDFS/S3 path, resource file will store to this base path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended
resource.storage.upload.base.path: /dolphinscheduler
# whether to startup kerberos
hadoop.security.authentication.startup.state: false
@ -93,14 +93,20 @@ conf:
kerberos.expire.time: 2
# resource view suffixs
#resource.view.suffixs: txt,log,sh,bat,conf,cfg,py,java,sql,xml,hql,properties,json,yml,yaml,ini,js
# if resource.storage.type: HDFS, the user must have the permission to create directories under the HDFS root path
hdfs.root.user: hdfs
# if resource.storage.type: S3, the value like: s3a://dolphinscheduler; if resource.storage.type: HDFS and namenode HA is enabled, you need to copy core-site.xml and hdfs-site.xml to conf dir
fs.defaultFS: file:///
aws.access.key.id: minioadmin
aws.secret.access.key: minioadmin
aws.region: us-east-1
aws.endpoint: http://localhost:9000
# if resource.storage.type=HDFS, the user must have the permission to create directories under the HDFS root path
resource.hdfs.root.user: hdfs
# if resource.storage.type=S3, the value like: s3a://dolphinscheduler; if resource.storage.type=HDFS and namenode HA is enabled, you need to copy core-site.xml and hdfs-site.xml to conf dir
resource.hdfs.fs.defaultFS: hdfs://mycluster:8020
# The AWS access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required
resource.aws.access.key.id: minioadmin
# The AWS secret access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required
resource.aws.secret.access.key: minioadmin
# The AWS Region to use. if resource.storage.type=S3 or use EMR-Task, This configuration is required
resource.aws.region: cn-north-1
# The name of the bucket. You need to create them by yourself. Otherwise, the system cannot start. All buckets in Amazon S3 share a single namespace; ensure the bucket is given a unique name.
resource.aws.s3.bucket.name: dolphinscheduler
# You need to set this parameter when private cloud s3. If S3 uses public cloud, you only need to set resource.aws.region or set to the endpoint of a public cloud such as S3.cn-north-1.amazonaws.com.cn
resource.aws.s3.endpoint: http://localhost:9000
# resourcemanager port, the default value is 8088 if not specified
resource.manager.httpaddress.port: 8088
# if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single, keep this value empty
@ -149,25 +155,8 @@ common:
configmap:
DOLPHINSCHEDULER_OPTS: ""
DATA_BASEDIR_PATH: "/tmp/dolphinscheduler"
RESOURCE_STORAGE_TYPE: "HDFS"
RESOURCE_UPLOAD_PATH: "/dolphinscheduler"
FS_DEFAULT_FS: "file:///"
FS_S3A_ENDPOINT: "s3.xxx.amazonaws.com"
FS_S3A_ACCESS_KEY: "xxxxxxx"
FS_S3A_SECRET_KEY: "xxxxxxx"
HADOOP_SECURITY_AUTHENTICATION_STARTUP_STATE: "false"
JAVA_SECURITY_KRB5_CONF_PATH: "/opt/krb5.conf"
LOGIN_USER_KEYTAB_USERNAME: "hdfs@HADOOP.COM"
LOGIN_USER_KEYTAB_PATH: "/opt/hdfs.keytab"
KERBEROS_EXPIRE_TIME: "2"
HDFS_ROOT_USER: "hdfs"
RESOURCE_MANAGER_HTTPADDRESS_PORT: "8088"
YARN_RESOURCEMANAGER_HA_RM_IDS: ""
YARN_APPLICATION_STATUS_ADDRESS: "http://ds1:%s/ws/v1/cluster/apps/%s"
YARN_JOB_HISTORY_STATUS_ADDRESS: "http://ds1:19888/ws/v1/history/mapreduce/jobs/%s"
DATASOURCE_ENCRYPTION_ENABLE: "false"
DATASOURCE_ENCRYPTION_SALT: "!@#$%^&*"
SUDO_ENABLE: "true"
# dolphinscheduler env
HADOOP_HOME: "/opt/soft/hadoop"
HADOOP_CONF_DIR: "/opt/soft/hadoop/etc/hadoop"

File diff suppressed because it is too large Load Diff

View File

@ -89,3 +89,4 @@ closed and transfer from [current DSIPs][current-DSIPs] to [past DSIPs][past-DSI
[github-issue-choose]: https://github.com/apache/dolphinscheduler/issues/new/choose
[mail-to-dev]: mailto:dev@dolphinscheduler.apache.org
[DSIP-1]: https://github.com/apache/dolphinscheduler/issues/6407

View File

@ -17,3 +17,4 @@
## High Scalability
- **Scalability**: Supports multitenancy and online resource management. Stable operation of 100,000 data tasks per day is supported.

View File

@ -71,4 +71,3 @@ process fails and ends
From the perspective of scheduling, this article preliminarily introduces the architecture principles and implementation
ideas of the big data distributed workflow scheduling system-DolphinScheduler. To be continued

View File

@ -6,15 +6,15 @@ This section briefs about the hardware requirements for DolphinScheduler. Dolphi
The Linux operating systems specified below can run on physical servers and mainstream virtualization environments such as VMware, KVM, and XEN.
| Operating System | Version |
| :----------------------- | :----------: |
| Red Hat Enterprise Linux | 7.0 and above |
| CentOS | 7.0 and above |
| Oracle Enterprise Linux | 7.0 and above |
| Operating System | Version |
|:-------------------------|:---------------:|
| Red Hat Enterprise Linux | 7.0 and above |
| CentOS | 7.0 and above |
| Oracle Enterprise Linux | 7.0 and above |
| Ubuntu LTS | 16.04 and above |
> **Note:**
>The above Linux operating systems can run on physical servers and mainstream virtualization environments such as VMware, KVM, and XEN.
> The above Linux operating systems can run on physical servers and mainstream virtualization environments such as VMware, KVM, and XEN.
## Server Configuration
@ -23,8 +23,8 @@ DolphinScheduler supports 64-bit hardware platforms with Intel x86-64 architectu
### Production Environment
| **CPU** | **MEM** | **HD** | **NIC** | **Num** |
| --- | --- | --- | --- | --- |
| 4 core+ | 8 GB+ | SAS | GbE | 1+ |
|---------|---------|--------|---------|---------|
| 4 core+ | 8 GB+ | SAS | GbE | 1+ |
> **Note:**
> - The above recommended configuration is the minimum configuration for deploying DolphinScheduler. Higher configuration is strongly recommended for production environments.
@ -34,11 +34,11 @@ DolphinScheduler supports 64-bit hardware platforms with Intel x86-64 architectu
DolphinScheduler provides the following network port configurations for normal operation:
| Server | Port | Desc |
| --- | --- | --- |
| MasterServer | 5678 | not the communication port, require the native ports do not conflict |
| WorkerServer | 1234 | not the communication port, require the native ports do not conflict |
| ApiApplicationServer | 12345 | backend communication port |
| Server | Port | Desc |
|----------------------|-------|----------------------------------------------------------------------|
| MasterServer | 5678 | not the communication port, require the native ports do not conflict |
| WorkerServer | 1234 | not the communication port, require the native ports do not conflict |
| ApiApplicationServer | 12345 | backend communication port |
> **Note:**
> - MasterServer and WorkerServer do not need to enable communication between the networks. As long as the local ports do not conflict.

View File

@ -101,8 +101,6 @@ The directory structure of DolphinScheduler is as follows:
## Configurations in Details
### dolphinscheduler-daemon.sh [startup or shutdown DolphinScheduler application]
dolphinscheduler-daemon.sh is responsible for DolphinScheduler startup and shutdown.
@ -110,6 +108,7 @@ Essentially, start-all.sh or stop-all.sh startup and shutdown the cluster via do
Currently, DolphinScheduler just makes a basic config, remember to config further JVM options based on your practical situation of resources.
Default simplified parameters are:
```bash
export DOLPHINSCHEDULER_OPTS="
-server
@ -157,8 +156,8 @@ The default configuration is as follows:
Note that DolphinScheduler also supports database configuration through `bin/env/dolphinscheduler_env.sh`.
### Zookeeper related configuration
DolphinScheduler uses Zookeeper for cluster management, fault tolerance, event monitoring and other functions. Configuration file location:
|Service| Configuration file |
|--|--|
@ -226,8 +225,8 @@ The default configuration is as follows:
|alert.rpc.port | 50052 | the RPC port of Alert Server|
|zeppelin.rest.url | http://localhost:8080 | the RESTful API url of zeppelin|
### Api-server related configuration
Location: `api-server/conf/application.yaml`
|Parameters | Default value| Description|
@ -257,6 +256,7 @@ Location: `api-server/conf/application.yaml`
|traffic.control.customize-tenant-qps-rate||customize tenant max request number per second|
### Master Server related configuration
Location: `master-server/conf/application.yaml`
|Parameters | Default value| Description|
@ -278,8 +278,8 @@ Location: `master-server/conf/application.yaml`
|master.registry-disconnect-strategy.strategy|stop|Used when the master disconnect from registry, default value: stop. Optional values include stop, waiting|
|master.registry-disconnect-strategy.max-waiting-time|100s|Used when the master disconnect from registry, and the disconnect strategy is waiting, this config means the master will waiting to reconnect to registry in given times, and after the waiting times, if the master still cannot connect to registry, will stop itself, if the value is 0s, the Master will waitting infinitely|
### Worker Server related configuration
Location: `worker-server/conf/application.yaml`
|Parameters | Default value| Description|
@ -298,6 +298,7 @@ Location: `worker-server/conf/application.yaml`
|worker.registry-disconnect-strategy.max-waiting-time|100s|Used when the worker disconnect from registry, and the disconnect strategy is waiting, this config means the worker will waiting to reconnect to registry in given times, and after the waiting times, if the worker still cannot connect to registry, will stop itself, if the value is 0s, will waitting infinitely |
### Alert Server related configuration
Location: `alert-server/conf/application.yaml`
|Parameters | Default value| Description|
@ -305,7 +306,6 @@ Location: `alert-server/conf/application.yaml`
|server.port|50053|the port of Alert Server|
|alert.port|50052|the port of alert|
### Quartz related configuration
This part describes quartz configs and configure them based on your practical situation and resources.
@ -335,7 +335,6 @@ The default configuration is as follows:
|spring.quartz.properties.org.quartz.jobStore.driverDelegateClass | org.quartz.impl.jdbcjobstore.PostgreSQLDelegate|
|spring.quartz.properties.org.quartz.jobStore.clusterCheckinInterval | 5000|
### dolphinscheduler_env.sh [load environment variables configs]
When using shell to commit tasks, DolphinScheduler will export environment variables from `bin/env/dolphinscheduler_env.sh`. The

View File

@ -24,46 +24,46 @@
* **MasterServer**
MasterServer adopts a distributed and decentralized design concept. MasterServer is mainly responsible for DAG task segmentation, task submission monitoring, and monitoring the health status of other MasterServer and WorkerServer at the same time.
When the MasterServer service starts, register a temporary node with ZooKeeper, and perform fault tolerance by monitoring changes in the temporary node of ZooKeeper.
MasterServer provides monitoring services based on netty.
MasterServer adopts a distributed and decentralized design concept. MasterServer is mainly responsible for DAG task segmentation, task submission monitoring, and monitoring the health status of other MasterServer and WorkerServer at the same time.
When the MasterServer service starts, register a temporary node with ZooKeeper, and perform fault tolerance by monitoring changes in the temporary node of ZooKeeper.
MasterServer provides monitoring services based on netty.
#### The Service Mainly Includes:
#### The Service Mainly Includes:
- **DistributedQuartz** distributed scheduling component, which is mainly responsible for the start and stop operations of scheduled tasks. When quartz start the task, there will be a thread pool inside the Master responsible for the follow-up operation of the processing task;
- **DistributedQuartz** distributed scheduling component, which is mainly responsible for the start and stop operations of scheduled tasks. When quartz start the task, there will be a thread pool inside the Master responsible for the follow-up operation of the processing task;
- **MasterSchedulerService** is a scanning thread that regularly scans the `t_ds_command` table in the database, runs different business operations according to different **command types**;
- **MasterSchedulerService** is a scanning thread that regularly scans the `t_ds_command` table in the database, runs different business operations according to different **command types**;
- **WorkflowExecuteRunnable** is mainly responsible for DAG task segmentation, task submission monitoring, and logical processing of different event types;
- **WorkflowExecuteRunnable** is mainly responsible for DAG task segmentation, task submission monitoring, and logical processing of different event types;
- **TaskExecuteRunnable** is mainly responsible for the processing and persistence of tasks, and generates task events and submits them to the event queue of the process instance;
- **TaskExecuteRunnable** is mainly responsible for the processing and persistence of tasks, and generates task events and submits them to the event queue of the process instance;
- **EventExecuteService** is mainly responsible for the polling of the event queue of the process instances;
- **EventExecuteService** is mainly responsible for the polling of the event queue of the process instances;
- **StateWheelExecuteThread** is mainly responsible for process instance and task timeout, task retry, task-dependent polling, and generates the corresponding process instance or task event and submits it to the event queue of the process instance;
- **StateWheelExecuteThread** is mainly responsible for process instance and task timeout, task retry, task-dependent polling, and generates the corresponding process instance or task event and submits it to the event queue of the process instance;
- **FailoverExecuteThread** is mainly responsible for the logic of Master fault tolerance and Worker fault tolerance;
- **FailoverExecuteThread** is mainly responsible for the logic of Master fault tolerance and Worker fault tolerance;
* **WorkerServer**
WorkerServer also adopts a distributed and decentralized design concept. WorkerServer is mainly responsible for task execution and providing log services.
WorkerServer also adopts a distributed and decentralized design concept. WorkerServer is mainly responsible for task execution and providing log services.
When the WorkerServer service starts, register a temporary node with ZooKeeper and maintain a heartbeat.
WorkerServer provides monitoring services based on netty.
When the WorkerServer service starts, register a temporary node with ZooKeeper and maintain a heartbeat.
WorkerServer provides monitoring services based on netty.
#### The Service Mainly Includes:
#### The Service Mainly Includes:
- **WorkerManagerThread** is mainly responsible for the submission of the task queue, continuously receives tasks from the task queue, and submits them to the thread pool for processing;
- **WorkerManagerThread** is mainly responsible for the submission of the task queue, continuously receives tasks from the task queue, and submits them to the thread pool for processing;
- **TaskExecuteThread** is mainly responsible for the process of task execution, and the actual processing of tasks according to different task types;
- **TaskExecuteThread** is mainly responsible for the process of task execution, and the actual processing of tasks according to different task types;
- **RetryReportTaskStatusThread** is mainly responsible for regularly polling to report the task status to the Master until the Master replies to the status ack to avoid the loss of the task status;
- **RetryReportTaskStatusThread** is mainly responsible for regularly polling to report the task status to the Master until the Master replies to the status ack to avoid the loss of the task status;
* **ZooKeeper**
ZooKeeper service, MasterServer and WorkerServer nodes in the system all use ZooKeeper for cluster management and fault tolerance. In addition, the system implements event monitoring and distributed locks based on ZooKeeper.
ZooKeeper service, MasterServer and WorkerServer nodes in the system all use ZooKeeper for cluster management and fault tolerance. In addition, the system implements event monitoring and distributed locks based on ZooKeeper.
We have also implemented queues based on Redis, but we hope DolphinScheduler depends on as few components as possible, so we finally removed the Redis implementation.
We have also implemented queues based on Redis, but we hope DolphinScheduler depends on as few components as possible, so we finally removed the Redis implementation.
* **AlertServer**
@ -71,7 +71,7 @@
* **API**
The API interface layer is mainly responsible for processing requests from the front-end UI layer. The service uniformly provides RESTful APIs to provide request services to external.
The API interface layer is mainly responsible for processing requests from the front-end UI layer. The service uniformly provides RESTful APIs to provide request services to external.
* **UI**
@ -84,6 +84,7 @@
##### Centralized Thinking
The centralized design concept is relatively simple. The nodes in the distributed cluster are roughly divided into two roles according to responsibilities:
<p align="center">
<img src="https://analysys.github.io/easyscheduler_docs_cn/images/master_slave.png" alt="master-slave character" width="50%" />
</p>
@ -120,8 +121,6 @@ The service fault-tolerance design relies on ZooKeeper's Watcher mechanism, and
</p>
Among them, the Master monitors the directories of other Masters and Workers. If the remove event is triggered, perform fault tolerance of the process instance or task instance according to the specific business logic.
- Master fault tolerance
<p align="center">
@ -146,7 +145,7 @@ Fault-tolerant content: When sending the remove event of the Worker node, the Ma
Fault-tolerant post-processing: Once the Master Scheduler thread finds that the task instance is in the "fault-tolerant" state, it takes over the task and resubmits it.
Note: Due to "network jitter", the node may lose heartbeat with ZooKeeper in a short period of time, and the node's remove event may occur. For this situation, we use the simplest way, that is, once the node and ZooKeeper timeout connection occurs, then directly stop the Master or Worker service.
Note: Due to "network jitter", the node may lose heartbeat with ZooKeeper in a short period of time, and the node's remove event may occur. For this situation, we use the simplest way, that is, once the node and ZooKeeper timeout connection occurs, then directly stop the Master or Worker service.
##### Task Failed and Try Again
@ -170,26 +169,26 @@ If there is a task failure in the workflow that reaches the maximum retry times,
In the early schedule design, if there is no priority design and use the fair scheduling, the task submitted first may complete at the same time with the task submitted later, thus invalid the priority of process or task. So we have re-designed this, and the following is our current design:
- According to **the priority of different process instances** prior over **priority of the same process instance** prior over **priority of tasks within the same process** prior over **tasks within the same process**, process task submission order from highest to Lowest.
- The specific implementation is to parse the priority according to the JSON of the task instance, and then save the **process instance priority_process instance id_task priority_task id** information to the ZooKeeper task queue. When obtain from the task queue, we can get the highest priority task by comparing string.
- According to **the priority of different process instances** prior over **priority of the same process instance** prior over **priority of tasks within the same process** prior over **tasks within the same process**, process task submission order from highest to Lowest.
- The specific implementation is to parse the priority according to the JSON of the task instance, and then save the **process instance priority_process instance id_task priority_task id** information to the ZooKeeper task queue. When obtain from the task queue, we can get the highest priority task by comparing string.
- The priority of the process definition is to consider that some processes need to process before other processes. Configure the priority when the process starts or schedules. There are 5 levels in total, which are HIGHEST, HIGH, MEDIUM, LOW, and LOWEST. As shown below
- The priority of the process definition is to consider that some processes need to process before other processes. Configure the priority when the process starts or schedules. There are 5 levels in total, which are HIGHEST, HIGH, MEDIUM, LOW, and LOWEST. As shown below
<p align="center">
<img src="https://user-images.githubusercontent.com/10797147/146744784-eb351b14-c94a-4ed6-8ba4-5132c2a3d116.png" alt="Process priority configuration" width="40%" />
</p>
<p align="center">
<img src="https://user-images.githubusercontent.com/10797147/146744784-eb351b14-c94a-4ed6-8ba4-5132c2a3d116.png" alt="Process priority configuration" width="40%" />
</p>
- The priority of the task is also divides into 5 levels, ordered by HIGHEST, HIGH, MEDIUM, LOW, LOWEST. As shown below:
<p align="center">
<img src="https://user-images.githubusercontent.com/10797147/146744830-5eac611f-5933-4f53-a0c6-31613c283708.png" alt="Task priority configuration" width="35%" />
</p>
- The priority of the task is also divides into 5 levels, ordered by HIGHEST, HIGH, MEDIUM, LOW, LOWEST. As shown below:
<p align="center">
<img src="https://user-images.githubusercontent.com/10797147/146744830-5eac611f-5933-4f53-a0c6-31613c283708.png" alt="Task priority configuration" width="35%" />
</p>
#### Logback and Netty Implement Log Access
- Since Web (UI) and Worker are not always on the same machine, to view the log cannot be like querying a local file. There are two options:
- Put logs on the ES search engine.
- Obtain remote log information through netty communication.
- In consideration of the lightness of DolphinScheduler as much as possible, so choose gRPC to achieve remote access to log information.
- Since Web (UI) and Worker are not always on the same machine, to view the log cannot be like querying a local file. There are two options:
- Put logs on the ES search engine.
- Obtain remote log information through netty communication.
- In consideration of the lightness of DolphinScheduler as much as possible, so choose gRPC to achieve remote access to log information.
<p align="center">
<img src="https://analysys.github.io/easyscheduler_docs_cn/images/grpc.png" alt="grpc remote access" width="50%" />

View File

@ -57,3 +57,4 @@ You can customise the configuration by changing the following properties in work
- worker.max.cpuload.avg=-1 (worker max cpuload avg, only higher than the system cpu load average, worker server can be dispatched tasks. default value -1: the number of cpu cores * 2)
- worker.reserved.memory=0.3 (worker reserved memory, only lower than system available memory, worker server can be dispatched tasks. default value 0.3, the unit is G)

View File

@ -1,8 +1,8 @@
# MetaData
## Table Schema
see sql files in `dolphinscheduler/dolphinscheduler-dao/src/main/resources/sql`
see sql files in `dolphinscheduler/dolphinscheduler-dao/src/main/resources/sql`
---
@ -26,6 +26,7 @@ see sql files in `dolphinscheduler/dolphinscheduler-dao/src/main/resources/sql`
- The `user_id` in the `t_ds_udfs` table represents the user who create the UDF, and the `user_id` in the `t_ds_relation_udfs_user` table represents a user who has permission to the UDF.
### Project - Tenant - ProcessDefinition - Schedule
![image.png](../../../img/metadata-erd/project_tenant_process_definition_schedule.png)
- A project can have multiple process definitions, and each process definition belongs to only one project.
@ -33,8 +34,10 @@ see sql files in `dolphinscheduler/dolphinscheduler-dao/src/main/resources/sql`
- A workflow definition can have one or more schedules.
### Process Definition Execution
![image.png](../../../img/metadata-erd/process_definition.png)
- A process definition corresponds to multiple task definitions, which are associated through `t_ds_process_task_relation` and the associated key is `code + version`. When the pre-task of the task is empty, the corresponding `pre_task_node` and `pre_task_version` are 0.
- A process definition can have multiple process instances `t_ds_process_instance`, one process instance corresponds to one or more task instances `t_ds_task_instance`.
- The data stored in the `t_ds_relation_process_instance` table is used to handle the case that the process definition contains sub-processes. `parent_process_instance_id` represents the id of the main process instance containing the sub-process, `process_instance_id` represents the id of the sub-process instance, `parent_task_instance_id` represents the task instance id of the sub-process node. The process instance table and the task instance table correspond to the `t_ds_process_instance` table and the `t_ds_task_instance` table, respectively.

View File

@ -6,28 +6,28 @@ All tasks in DolphinScheduler are saved in the `t_ds_process_definition` table.
The following shows the `t_ds_process_definition` table structure:
No. | field | type | description
-------- | ---------| -------- | ---------
1|id|int(11)|primary key
2|name|varchar(255)|process definition name
3|version|int(11)|process definition version
4|release_state|tinyint(4)|release status of process definition: 0 not released, 1 released
5|project_id|int(11)|project id
6|user_id|int(11)|user id of the process definition
7|process_definition_json|longtext|process definition JSON
8|description|text|process definition description
9|global_params|text|global parameters
10|flag|tinyint(4)|specify whether the process is available: 0 is not available, 1 is available
11|locations|text|node location information
12|connects|text|node connectivity info
13|receivers|text|receivers
14|receivers_cc|text|CC receivers
15|create_time|datetime|create time
16|timeout|int(11) |timeout
17|tenant_id|int(11) |tenant id
18|update_time|datetime|update time
19|modify_by|varchar(36)|specify the user that made the modification
20|resource_ids|varchar(255)|resource ids
| No. | field | type | description |
|-----|-------------------------|--------------|------------------------------------------------------------------------------|
| 1 | id | int(11) | primary key |
| 2 | name | varchar(255) | process definition name |
| 3 | version | int(11) | process definition version |
| 4 | release_state | tinyint(4) | release status of process definition: 0 not released, 1 released |
| 5 | project_id | int(11) | project id |
| 6 | user_id | int(11) | user id of the process definition |
| 7 | process_definition_json | longtext | process definition JSON |
| 8 | description | text | process definition description |
| 9 | global_params | text | global parameters |
| 10 | flag | tinyint(4) | specify whether the process is available: 0 is not available, 1 is available |
| 11 | locations | text | node location information |
| 12 | connects | text | node connectivity info |
| 13 | receivers | text | receivers |
| 14 | receivers_cc | text | CC receivers |
| 15 | create_time | datetime | create time |
| 16 | timeout | int(11) | timeout |
| 17 | tenant_id | int(11) | tenant id |
| 18 | update_time | datetime | update time |
| 19 | modify_by | varchar(36) | specify the user that made the modification |
| 20 | resource_ids | varchar(255) | resource ids |
The `process_definition_json` field is the core field, which defines the task information in the DAG diagram, and it is stored in JSON format.
@ -40,6 +40,7 @@ No. | field | type | description
4|timeout|int|timeout
Data example:
```bash
{
"globalParams":[
@ -238,38 +239,38 @@ No.|parameter name||type|description |note
**The following shows the node data structure:**
No.|parameter name||type|description |notes
-------- | ---------| ---------| -------- | --------- | ---------
1|id | |String| task Id|
2|type ||String |task type |SPARK
3| name| |String|task name |
4| params| |Object|customized parameters |JSON format
5| |mainClass |String | main class
6| |mainArgs | String| execution arguments
7| |others | String| other arguments
8| |mainJar |Object | application jar package
9| |deployMode |String |deployment mode |local,client,cluster
10| |driverCores | String| driver cores
11| |driverMemory | String| driver memory
12| |numExecutors |String | executor count
13| |executorMemory |String | executor memory
14| |executorCores |String | executor cores
15| |programType | String| program type|JAVA,SCALA,PYTHON
16| | sparkVersion| String| Spark version| SPARK1 , SPARK2
17| | localParams| Array|customized local parameters
18| | resourceList| Array|resource files
19|description | |String|description | |
20|runFlag | |String |execution flag| |
21|conditionResult | |Object|condition branch| |
22| | successNode| Array|jump to node if success| |
23| | failedNode|Array|jump to node if failure|
24| dependence| |Object |task dependency |mutual exclusion with params
25|maxRetryTimes | |String|max retry times | |
26|retryInterval | |String |retry interval| |
27|timeout | |Object|timeout | |
28| taskInstancePriority| |String|task priority | |
29|workerGroup | |String |Worker group| |
30|preTasks | |Array|preposition tasks| |
| No. | parameter name || type | description | notes |
|-----|----------------------|----------------|--------|-----------------------------|------------------------------|
| 1 | id | | String | task Id |
| 2 | type || String | task type | SPARK |
| 3 | name | | String | task name |
| 4 | params | | Object | customized parameters | JSON format |
| 5 | | mainClass | String | main class |
| 6 | | mainArgs | String | execution arguments |
| 7 | | others | String | other arguments |
| 8 | | mainJar | Object | application jar package |
| 9 | | deployMode | String | deployment mode | local,client,cluster |
| 10 | | driverCores | String | driver cores |
| 11 | | driverMemory | String | driver memory |
| 12 | | numExecutors | String | executor count |
| 13 | | executorMemory | String | executor memory |
| 14 | | executorCores | String | executor cores |
| 15 | | programType | String | program type | JAVA,SCALA,PYTHON |
| 16 | | sparkVersion | String | Spark version | SPARK1 , SPARK2 |
| 17 | | localParams | Array | customized local parameters |
| 18 | | resourceList | Array | resource files |
| 19 | description | | String | description | |
| 20 | runFlag | | String | execution flag | |
| 21 | conditionResult | | Object | condition branch | |
| 22 | | successNode | Array | jump to node if success | |
| 23 | | failedNode | Array | jump to node if failure |
| 24 | dependence | | Object | task dependency | mutual exclusion with params |
| 25 | maxRetryTimes | | String | max retry times | |
| 26 | retryInterval | | String | retry interval | |
| 27 | timeout | | Object | timeout | |
| 28 | taskInstancePriority | | String | task priority | |
| 29 | workerGroup | | String | Worker group | |
| 30 | preTasks | | Array | preposition tasks | |
**Node data example:**
@ -336,31 +337,31 @@ No.|parameter name||type|description |notes
**The following shows the node data structure:**
No.|parameter name||type|description |notes
-------- | ---------| ---------| -------- | --------- | ---------
1|id | |String| task Id|
2|type ||String |task type |MR
3| name| |String|task name |
4| params| |Object|customized parameters |JSON format
5| |mainClass |String | main class
6| |mainArgs | String|execution arguments
7| |others | String|other arguments
8| |mainJar |Object | application jar package
9| |programType | String|program type|JAVA,PYTHON
10| | localParams| Array|customized local parameters
11| | resourceList| Array|resource files
12|description | |String|description | |
13|runFlag | |String |execution flag| |
14|conditionResult | |Object|condition branch| |
15| | successNode| Array|jump to node if success| |
16| | failedNode|Array|jump to node if failure|
17| dependence| |Object |task dependency |mutual exclusion with params
18|maxRetryTimes | |String|max retry times | |
19|retryInterval | |String |retry interval| |
20|timeout | |Object|timeout | |
21| taskInstancePriority| |String|task priority| |
22|workerGroup | |String |Worker group| |
23|preTasks | |Array|preposition tasks| |
| No. | parameter name || type | description | notes |
|-----|----------------------|--------------|--------|-----------------------------|------------------------------|
| 1 | id | | String | task Id |
| 2 | type || String | task type | MR |
| 3 | name | | String | task name |
| 4 | params | | Object | customized parameters | JSON format |
| 5 | | mainClass | String | main class |
| 6 | | mainArgs | String | execution arguments |
| 7 | | others | String | other arguments |
| 8 | | mainJar | Object | application jar package |
| 9 | | programType | String | program type | JAVA,PYTHON |
| 10 | | localParams | Array | customized local parameters |
| 11 | | resourceList | Array | resource files |
| 12 | description | | String | description | |
| 13 | runFlag | | String | execution flag | |
| 14 | conditionResult | | Object | condition branch | |
| 15 | | successNode | Array | jump to node if success | |
| 16 | | failedNode | Array | jump to node if failure |
| 17 | dependence | | Object | task dependency | mutual exclusion with params |
| 18 | maxRetryTimes | | String | max retry times | |
| 19 | retryInterval | | String | retry interval | |
| 20 | timeout | | Object | timeout | |
| 21 | taskInstancePriority | | String | task priority | |
| 22 | workerGroup | | String | Worker group | |
| 23 | preTasks | | Array | preposition tasks | |
**Node data example:**
@ -493,36 +494,36 @@ No.|parameter name||type|description |notes
**The following shows the node data structure:**
No.|parameter name||type|description |notes
-------- | ---------| ---------| -------- | --------- | ---------
1|id | |String|task Id|
2|type ||String |task type|FLINK
3| name| |String|task name|
4| params| |Object|customized parameters |JSON format
5| |mainClass |String |main class
6| |mainArgs | String|execution arguments
7| |others | String|other arguments
8| |mainJar |Object |application jar package
9| |deployMode |String |deployment mode |local,client,cluster
10| |slot | String| slot count
11| |taskManager |String | taskManager count
12| |taskManagerMemory |String |taskManager memory size
13| |jobManagerMemory |String | jobManager memory size
14| |programType | String| program type|JAVA,SCALA,PYTHON
15| | localParams| Array|local parameters
16| | resourceList| Array|resource files
17|description | |String|description | |
18|runFlag | |String |execution flag| |
19|conditionResult | |Object|condition branch| |
20| | successNode| Array|jump node if success| |
21| | failedNode|Array|jump node if failure|
22| dependence| |Object |task dependency |mutual exclusion with params
23|maxRetryTimes | |String|max retry times| |
24|retryInterval | |String |retry interval| |
25|timeout | |Object|timeout | |
26| taskInstancePriority| |String|task priority| |
27|workerGroup | |String |Worker group| |
38|preTasks | |Array|preposition tasks| |
| No. | parameter name || type | description | notes |
|-----|----------------------|-------------------|--------|-------------------------|------------------------------|
| 1 | id | | String | task Id |
| 2 | type || String | task type | FLINK |
| 3 | name | | String | task name |
| 4 | params | | Object | customized parameters | JSON format |
| 5 | | mainClass | String | main class |
| 6 | | mainArgs | String | execution arguments |
| 7 | | others | String | other arguments |
| 8 | | mainJar | Object | application jar package |
| 9 | | deployMode | String | deployment mode | local,client,cluster |
| 10 | | slot | String | slot count |
| 11 | | taskManager | String | taskManager count |
| 12 | | taskManagerMemory | String | taskManager memory size |
| 13 | | jobManagerMemory | String | jobManager memory size |
| 14 | | programType | String | program type | JAVA,SCALA,PYTHON |
| 15 | | localParams | Array | local parameters |
| 16 | | resourceList | Array | resource files |
| 17 | description | | String | description | |
| 18 | runFlag | | String | execution flag | |
| 19 | conditionResult | | Object | condition branch | |
| 20 | | successNode | Array | jump node if success | |
| 21 | | failedNode | Array | jump node if failure |
| 22 | dependence | | Object | task dependency | mutual exclusion with params |
| 23 | maxRetryTimes | | String | max retry times | |
| 24 | retryInterval | | String | retry interval | |
| 25 | timeout | | Object | timeout | |
| 26 | taskInstancePriority | | String | task priority | |
| 27 | workerGroup | | String | Worker group | |
| 38 | preTasks | | Array | preposition tasks | |
**Node data example:**
@ -588,30 +589,30 @@ No.|parameter name||type|description |notes
**The following shows the node data structure:**
No.|parameter name||type|description |notes
-------- | ---------| ---------| -------- | --------- | ---------
1|id | |String|task Id|
2|type ||String |task type|HTTP
3| name| |String|task name|
4| params| |Object|customized parameters |JSON format
5| |url |String |request url
6| |httpMethod | String|http method|GET,POST,HEAD,PUT,DELETE
7| | httpParams| Array|http parameters
8| |httpCheckCondition | String|validation of HTTP code status|default code 200
9| |condition |String |validation conditions
10| | localParams| Array|customized local parameters
11|description | |String|description| |
12|runFlag | |String |execution flag| |
13|conditionResult | |Object|condition branch| |
14| | successNode| Array|jump node if success| |
15| | failedNode|Array|jump node if failure|
16| dependence| |Object |task dependency |mutual exclusion with params
17|maxRetryTimes | |String|max retry times | |
18|retryInterval | |String |retry interval| |
19|timeout | |Object|timeout | |
20| taskInstancePriority| |String|task priority| |
21|workerGroup | |String |Worker group| |
22|preTasks | |Array|preposition tasks| |
| No. | parameter name || type | description | notes |
|-----|----------------------|--------------------|--------|--------------------------------|------------------------------|
| 1 | id | | String | task Id |
| 2 | type || String | task type | HTTP |
| 3 | name | | String | task name |
| 4 | params | | Object | customized parameters | JSON format |
| 5 | | url | String | request url |
| 6 | | httpMethod | String | http method | GET,POST,HEAD,PUT,DELETE |
| 7 | | httpParams | Array | http parameters |
| 8 | | httpCheckCondition | String | validation of HTTP code status | default code 200 |
| 9 | | condition | String | validation conditions |
| 10 | | localParams | Array | customized local parameters |
| 11 | description | | String | description | |
| 12 | runFlag | | String | execution flag | |
| 13 | conditionResult | | Object | condition branch | |
| 14 | | successNode | Array | jump node if success | |
| 15 | | failedNode | Array | jump node if failure |
| 16 | dependence | | Object | task dependency | mutual exclusion with params |
| 17 | maxRetryTimes | | String | max retry times | |
| 18 | retryInterval | | String | retry interval | |
| 19 | timeout | | Object | timeout | |
| 20 | taskInstancePriority | | String | task priority | |
| 21 | workerGroup | | String | Worker group | |
| 22 | preTasks | | Array | preposition tasks | |
**Node data example:**
@ -1112,3 +1113,4 @@ No.|parameter name||type|description |notes
]
}
```

675
docs/docs/en/changelog.md Normal file
View File

@ -0,0 +1,675 @@
# Change Log
## Feature
- [Feature-8252][doc] K8s and namespace manager docs and web page update (#9881)
- [feat][task plugin] Add Jupyter task plugin (#9872)
- [Feature-9474] [CI] Add cluster test script verify on shell script (#9997)
- [Feature-10034][ci] Add postgresql cluster test in CI (#10035)
- [feat] Add batch rerun and stop for process instance (#10013)
- [Feat][ui] Add resource username for function and resource manage (#10086)
- [Feature][plugin] Add k8s task in task plugin (#9425)
- [Feature][task plugin] Add mlflow task plugin for MLops scenario. (#10036)
- [Feature] [CI] Add alert server cluster test #10125
- [python] Change distribute name avoid confusion with main pkg (#10134)
- [Feature][mlops] Support MLflow Models to deploy model service (MLflow models serve and Docker) (#10150)
- [Feature][ci] Schedule Dead Link Checker (#10000)
- [Feature][mlops] Add MLflow Projects (custom project) component (#10175)
- [Feature][permission] Project management module authority control refactoring, authority abstract interface setting. (#10261)
- [Feature][datasource]replace DataSourceController API with string JSON and cast to DTO (#10276)
- [Feature][api] Modify the permissions of project management, security center, data source center and data quality module. (#10289)
- [Feature][permission] Reconstruction of permissions of resource center and monitoring center. (#10307)
- [feat][task] Add OpenMLDB task plugin (#10198)
- [Feature][task plugin] Add support for dynamic form for zeppelin task plugin (#9977) (#10269)
- [Feature-10290] [API] Refactor org.apache.dolphinscheduler.api.controller.AccessTokenController#createToken api (#10301)
- [Feature] Should not add refresh logic in the e2e case. (#10331)
- [Feature][monitoring] Added disk available capacity monitoring (#10287)
- [Feature-6758][task] Add limit resource usage for tasks base on cgroup (#10373)
- [Feature][metrics] Add master, worker metrics (#10326)
- [Feature] [MLOps] support mlflow deploy with docker compose (#10217)
- [Feature][metrics] Enable prometheus to collect metrics in standalone mode demo (#10398)
- [Feature][ui] Add manual input for scheduling dates. (#10396)
- Add mysql registry plugin (#10406)
- [CI] Add OWASP Dependency Check (#10058)
- [Feature][improvement] Support multi cluster environments - k8s type (#10096)
- [Feature-10411] Add tenant api test (#10442)
- [Feature-10363][server] S3 Resource center supports bucket customization (#10433)
- [feature][task] Enable zeppelin schedule a whole zeppelin note (#10434)
- [Feature] Enable users to switch and install conda env in jupyter task (#10337)
- [Feature][task plugin] Add DVC task plugin for MLops scenario (#10372) (#10407)
- [Feature] [Permission]Data permission module added and Query class api remove verification. (#10504)
- [Feature] TaskGroup remove query func permission check. #10557
- [python] Support using full-name definitions for resources (#10551)
- [feat] ui build no inherit from maven proxy (#10590)
- [Feature] Time function analysis extension. (#10624)
- [feature][task-flink] Support Flink application mode (#9577)
- [Feature][master] Failover workflow instance base on priority (#10754)
- [Feature] Enable users to create python env from requirements.txt (#10658)
- [Feature-10629][improvement] Support multi cluster environments - namespace use cluster (#10650)
- [Feature][dependencies] Bump spring-core from 5.3.12 to 5.3.19 (#10857)
- [Feature-10219][emr] EMR supports use <add-Steps> to add steps to an existing cluster (#10657)
- [Feature-10871] add workflow executing data query (#10875)
- [Feat][ui] Uniform search button size. (#10880)
- [Feature][metrics] Add resource download related metrics for workers (#10749)
- [Feat][ui] Add title display to the table in the security center. (#10969)
- feat: Added global trim method (#10975)
- [Feature][task plugin] Enable users to switch endpoints in zeppelin tasks (#10925)
- [Feature][datax task]Datax task plugin support custom parameters (#10950)
- feat: Add trim method to all input (#10984)
- [Feat][ui] Added LDAP function. (#11029)
- [Feature][task plugin] Increase zeppelin task stability in production (#11010)
- [Feature][task plugin] Add SageMaker Pipeline task plugin for MLOps scenario (#10826) (#10935)
- [python] add import SageMaker task (#11121)
- Add base loop task execotor and http template parser (#11137)
- [Feature-10604] Task name in the copied and imported workflow without copy suffix (#10607)
- [Feature][api] Refactor org.apache.dolphinscheduler.api.controller.ProjectController (#10832)
- [Feature-10837][task plugin] Sqoop Task Add --split-by Parameter (#10960)
- [Feature] Add CURD to the project/tenant/user section of the API and add its tests. (#10913)
- [Feature] Add new task type chunjun (#10937)
- Write alert result into db (#11221)
- [Python] Migrate exists method call in PythonGateway to corresponding service (#11188)
- [Feature][style] Add spotless maven plugin for automatic style fix. (#11272)
- [Python] Combine gateway.entry_point call in python api side (#11330)
- [Feature-9978][ui] add node dependency type feature (#11235)
- [Feature][api] Refactor org.apache.dolphinscheduler.api.controller.QueueController (#11154)
- [python] Add local dev mod runing python integate tests (#11362)
- [Feature-10117][ui] Hide node execution when start from the workflow definition list page (#11372)
- [Feature-11223] support stream task (#11350)
- [python] Add multiple version of document (#11391)
- [python] Add support interpreter version 310 and 311-dev (#11170)
- [Feature-7024] Add waiting strategy to support master/worker can recover from registry lost (#11368)
- [Feature][style] Enable spotless to manage imports (#11458)
- Support reading application properties from kubernetes config map (#11402)
- [Feature-10182][task-datax] datax access kerberos authenticated bigdata cluster (#11387)
- [Feature][style] Make sure there is one and only one empty line at the end of each java file (#11520)
- [Feature] New task plugin Pytorch (#11498)
- [feature-11408] add hive option (#11493)
- feat: Locations Formats the layout if it does not exist (#11535)
- [Feature][ui] Enable highlight and auto-complete for SQL scripts (#11659)
- [python] Auto location when create process definition (#11681)
- [Feature][task plugin] Support hive cli task plugin (#11651)
- [feature][api]feature: add task type list and enable user to add task type to fav (#11727)
- Add Kubernetes configmap reload to all components (#11730)
- [Feature-11530] add state history for process instance (#11757)
- [DSIP-11][python] create workflows from YAML configuration (#11988)
- [Feature][registry] Support etcd as registry (#10981) (#11997)
- [Feat][python] Add parameter environment to task (#11763) (#11989)
- [feat][python] Support Pytorch task in python api (#11975)
- [feat][python] Add sagemaker yaml example (#11925)
- [feat][python] Support OpenMLDB task in python api (#11944)
- [feat][python] Support MLflow task in python api (#11962)
- [Feature][pydolphinscheduler] Support DVC task in pyds #11922 (#11941)
- [Feature-10273][datasource] Add dataSource for AWS Athena (#11157) (#12039)
- [Feat][ui] Update package. (#12046)
- [python] Support SageMaker task type (#11002)
- [Feature][resource center] Add support for Alibaba Cloud OSS as storage of resource center (#11708)
## Improvement
- [refactor][perf] cache compiled regex pattern (#10099)
- [improve] Change Mysql Driver (#10220)
- [Style][ui] Split multilingual modules. (#10256)
- Clean up unused dependencies and packaging issues (#9944)
- [Style][ui] Format code. (#10314)
- Add dolphinscheduler-task-all in api module (#10350)
- Move datasource params to datasouce plugin module (#10353)
- issues-10354: upgrade commons-io to fix CVE (#10355)
- issue-10356: upgrade logback to fix cve (#10357)
- Remove quartz at WorkerServer (#10358)
- Add dolphinscheduler-scheduler module (#10360)
- [improve] Using create or replace function in sql task (#10170)
- Optimize MasterServer, add MasterRPCService (#10371)
- change to common lang3 and fix checkstyle (#10380)
- Upgrade log4j in E2E (#10359)
- Add some warning log in master (#10383)
- [Improve] Enhance complement function transformation (#10376)
- Update chart.yaml dependencies version (#10420)
- [imporve] Add index to table t_ds_relation_process_instance (#10461)
- [Improvement][api] Interface and entity optimization to improve code scalability (#10466). (#10468)
- [Improve] Add LDAP user not exsitst action config (#10451)
- Modify obsolete swagger parameters in AuditLogController (#10494)
- Remove the schedule thread in LowerWeightHostManager (#10310)
- [Improvement][metrics] Apply micrometer naming convention to metrics (#10477)
- change import method access to protected (#10536)
- [Improve]Remove the code logger.isDebugEnabled() #5918 (#10542)
- [Improvement][metrics] Update some metrics names in grafana-demo dashboards (#10432) (#10552)
- Optimize master log, use MDC to inject workflow instance id and task instance id in log (#10516)
- add datasource health check to the healthcheck endpoint (#10588)
- dependent node log output optimization and fix checkstyle (#10593)
- [Improvement][master] batch remove TaskInstaceId and workflowInstanceId (#10620)
- [Improvement][worker] fixed naming of rpc package (#10614)
- Add JDK11 on unit-test (#10364)
- Validate master/worker config (#10649)
- [Improvement]Delete the getSession in the Registry interface (#10633)
- [Refactor][ui] Refactor yarn queue manage using NSpace component. (#10680)
- [Refactor][ui] Refactor token manage using NSpace component. (#10681)
- [Refactor][ui] Refactor k8s namespace manage using NSpace component. (#10682)
- [Improvement-10617] Add comment in slot check (#10618)
- [Improvement] heartbeat info improvement (#10699)
- Optimize sql script use insert ignore (#10700)
- [Optimization] Calculate global parameter and local parameter at master. (#10704)
- resource modules query fix. (#10687)
- [Refactor][ui] Refactor environment manage using NSpace component. (#10702)
- [Refactor][ui] Refactor cluster manage using NSpace component. (#10701)
- [Enhancement][api] Support batch copy and export online workflow #10678
- [Refactor][ui] Refactor worker group manage using NSpace component. (#10735)
- [Task] Add Dinky task to better support the development and execution of FlinkSQL (#10640)
- [Optimization]Optimize some details of MLFlow task plugin #10740 (#10739)
- [Optimization] Config module resource relation fix. (#10718)
- [Improvement] [Alert Server] Add default datasource url in alert server application.yaml for consistency (#10790)
- [python] Correct some docs for python api (#10772)
- [Task] Added Apache SeaTunnel 2.x task node (#10480)
- [Refactor][ui] Refactor alarm instance manage using NSpace component. (#10799)
- [Refactor][ui] Refactor alarm group manage using NSpace component. (#10803)
- Remove quartz in service (#10748)
- [Refactor][ui] Refactor user manage using NSpace component. (#10816)
- [Improvement] Resource authority module database table unique index increased. (#10807)
- [ui] Data Quality Result Display Optimize (#10835)
- [Improvement] [CI] Add label robot to pull request (#10848)
- [Refactor][ui] Refactor tenant manage using NSpace component. (#10879)
- [Python] Supports creating or editing resources. (#10823)
- [Refactor][ui] Refactor audio log using NSpace component. (#10991)
- impv: not mark issue stale with security label (#10982)
- [Refactor][ui] Refactor datasource using NSpace component. (#11022)
- [Refactor][ui] Refactor task result using NSpace component. (#11054)
- Add dolphinscheduler-bom to manage the dependency version (#11025)
- [Improvement][task plugin]Refactor JupyterParameters.java with lombok (#11045)
- [Improvement][task]Improve seatunnel task command log (#11027)
- [Improve] Hide Change Pwd Page When Login With LDAP (#11041)
- [Refactor][ui] Refactor rule manage using NSpace component. (#11069)
- [Refactor][ui] Refactor time display use datatable render function (#11078)
- [Refactor][ui] Refactor file manage using NSpace component. (#11073)
- [Improvement][task log] Task status log print description instead of code (#11009)
- [Improvement-11013][dolphinscheduler-common] YarnHAAdminUtils#getActiveRMName function Add HTTPS Hadoop environment support (#11017)
- [Refactor][ui] Refactor function manage using NSpace component. (#11124)
- [Improvement 11072][common] Added HTTPS request support in kerberos environment (#11127)
- [Refactor][ui] Refactor resource manage using NSpace component. (#11098)
- [Improvement][metrics] Switch to use tags to indicate task / workflow execution status for metrics (#11128)
- [Enhancement][api] Copy workflow with scheduler info (#10677)
- Refactor alert-api with lombok (#11133)
- [Improvement][executorserviceimpl] remove some unnecessary (#11122)
- Add is success in loop task status (#11169)
- [python] Refactor structure to avoid cycle import (#11167)
- [Improvement][metrics] Upgrade plugin versions in grafana demo dashboards (#11192) (#11193)
- [Improvement][ui] Unified local parameters UI (#11190)
- [Refactor][ui] Refactor task group option using NSpace component. (#11209)
- [Improvement-10760][dolphinscheduler-api] Some variables are redundant and fix checkstyle (#11213)
- [Improvement][ui] Improve monaco-editor component in ts. (#11274)
- [Improvement][metrics] Add metrics for alert server (#11240)
- [Improvement-11293][dolphinscheduler-api] remove some variables that redundant in subclasses of AbstractAuthenticator (#11305)
- [Improvement][ui] Scrolling will be applied to parent model on editor scrolled to top/bottom (#11335)
- [Refactor][ui] Refactor task group queue using NSpace component. (#11341)
- [Refactor][ui] Refactor project list using NSpace component. (#11369)
- Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus (#11340)
- [Improvement-11394][task plugin] Remove Passed Parameter --queue in Seatunnel Task Plugin (#11395)
- Add pause to task instance status (#11390)
- [Refactor][ui] Refactor project workflow definition using NSpace component. (#11398)
- [Improvement-11380][scp-host.sh] Set StrictHostKeyChecking=no option to ssh (#11382)
- [Refactor][ui] Refactor workflow definition timing using NSpace component. (#11415)
- [Refactor][ui] Refactor workflow definition version using NSpace component. (#11420)
- [Improvement] Avoid using search in for and start using testSaveTaskDefine (#11383)
- Clear thread name in BaseTaskProcessor (#11422)
- [Refactor][ui] Refactor task instance batch using NSpace component. (#11447)
- [Refactor][ui] Refactor workflow instance using NSpace component. (#11446)
- [Improvement-11386][ui] Concise the logic available for task action buttons (#11419)
- [Refactor][ui] Refactor task instance stream using NSpace component. (#11454)
- optimize sorting priority for workflow define (#11462)
- [Refactor][ui] Refactor task definition batch stream using NSpace component. (#11485)
- [Refactor][ui] Refactor task definition version using NSpace component. (#11495)
- [test-11432][httptask] Add unit tests to http task (#11453)
- [Improvement] File deletion logic optimization (#11519)
- [Improvement-11468] Support re running historical version workflow (#11489)
- [Improvement]Add a stop function when you are ready to do pause operation (#11543)
- [Improvement] [LDAP] Bug fix: Config value return "null" instead of real null currently. (#11325)
- Remove logger header in task log file (#11555)
- [Improvement-11550] [Doc]Document content update (#11577)
- Upgrade spring boot and swagger (#11574)
- Add dependent task instance log (#11541)
- [improvement] Add two parameters in workergroup, and support the application of description display and other parameters (#11542)
- [Improvement][taskinstance] reduce database queries (#11522)
- [Improvement-10942][ui]The workflow or task name is too long and results in incomplete display (#10957)
- [Improvement][user] Query all users filter un enabled. (#11504)
- [Improvement-#11613] Add spi priority factory (#11614)
- [improvement-#11630]fix document about common.properties (#11653)
- Refactor worker execute task process (#11540)
- add unit test ci timeout (#11655)
- [Improvement-11662][dao] fix foreach null items in mapper.xml
- Refactor heart beat task, use json to serialize/deserialize (#11702)
- optimize the process instance query, change the date time range (#11719)
- reduce task dependencies (#11731)
- [Improvement] Replace commons-lang 2 function invocations with commons-lang3 (#11810)
- Bump up dependencies to fix cves (#11765)
- Remove `not use web service` comment in alert server (#11819)
- [Improvement-#11768][monitor] Support monitor h2 database in monitor page (#11813)
- [Improve] Upgrade Hadoop to 3.2.4 (#11832)
- Remove htrace dependencies (#11881)
- [Improvement-11834] Upgrade docker base image to support python3.9 (#11835)
- Optimize date range condition (#11911)
- Add debug logs for properties read from `common.properties` (#11910)
- Add validations of possible malicious keys (#11966)
- [Improvement] and task instance date range search (#11963)
- [Improvement-11291][sql] Improvement the log output while executing multiple sql statements (#11292)
- Script cannot contains ''' in params (#12068)
- [improvement]Add Set cluster name (#12058)
- [Improvement] Add Set cluster name (#12074)
- add(dependent node's add unprivileged project list): dependent node's add unprivileged project list (#12073)
- [improve] Optimize MLFlow task plugin for easy of use (#12071)
- Remove dao in worker (#10994)
- Move utils, builder under master (#11000)
- [Improvement][worker] Resource may store in hadoop/S3 (#11206)
## Bugfix
- [Fix][ui][V1.0.0-Beta] Fix the problem of packing failure. (#9962)
- [Bug] [MASTER-9811]fix cmd param to overwrite global param when executing complement (#9952)
- [Fix-10002] Fix some bugs in datasource list (#10005)
- [Fix][ui] Fix the task name validator error. (#10008)
- [Fix-9975] The selected task instance was recreated when the Master service failed over (#9976)
- [Fix][ui] Support only one file upload on the file manage page. (#10007)
- [Fix][ui] Fix errorOutputPath column in dataquality page (#10015)
- [fix-9991][worker]fix statement is closed before resultSet.getMetaData() (#10014)
- [dataquality] Fix task commnd null bug (#9974)
- [Fix-10039] Flink run command when perfecting Python jobs (#10042)
- [Bug] fix run on docker and k8s (#10026)
- [python] Fix pre_submit_check error when local params in tasks (#10032)
- [Fix][ui]Modify error message (#10054)
- Fix flaky filemanage_e2e_test (#10068)
- [Fix][ui] Fix in the Chinese state, there is a problem with the multi-language support of the form. (#10066)
- [Fix][ui] Fix bug where username is empty in file manage (#10070)
- [Bug][master] fix master task failover (#10065)
- [Fix-10080]When the created tenant name is too long, an error message will be prompted (#10081)
- Create process definition with task group and task group priority cannot be save into db (#10078)
- [Fix][docker] Support Multi-Architecture Docker Images (#10052)
- [Fix-10082]Resource userName is null and udf function manager add a username (#10064)
- [Fix-10049] Conditions Task branch flow failed (#10077)
- [Fix][ui] fulfill missing fields in DataQuality (#10098)
- [fix] Enhance name pre checker in resource center (#10094)
- [BUG][taskgroup] Task group does not take effect (#10093)
- [Build][ui] Solve vite config throws terser warning. (#10105)
- Fix alert server curator health check not work (#10111)
- Fix unit test and remove unused import (#10113)
- [Bug][doc] Fix the open-api docs request error (#10100)
- [Fix][ui] Add the task status of pending. (#10129)
- [Fix][ui] Limit task group resource capacity to only positive integers. (#10158)
- [Fix]Change the HTTP or HTTPS regular expression (#10166)
- [Fix][ui] Fix the problem that the failure strategy field in the time management form is multilingual. (#10169)
- [Fix-10057][api] api parameter name misspelling (#10063)
- [Fix][ui] Add seatunnel icon (#10178)
- fix spcial week init bug (#10183)
- [Fix-10076] queryUserByToken use server timezone (#10177)
- [Fix][ui] Fix data-quality component multi-language support. (#10202)
- [Fix-10199] [Workflow/Workflow Definition] After deleting messages in batches, the error message is repeated (#10201)
- [Bug][deploy]Fix worker-server path in script scp-hosts.sh (#10208) (#10209)
- fix#10214 Delete online workflows in batches, prompt information is accurate (#10215)
- [python] Fix process definition attr release state not work (#10151)
- [Fix][ui] add assignment of totalPage (#10165)
- [Fix-10103][k8s]Fix k8s Change DataSource Error (#10128)
- [Fix][ui] Fix the problem of multi-language support of workflow instance host in Chinese state. (#10223)
- [fix-10207] When the registered alarm plugin is empty, the error message is not clear (#10224)
- [Fix][ui] Fix issue with treemap depth in workflow relationships. (#10229)
- [fix] Task group queue query all list error (#10237)
- [Fix#9923] The alarm instance management interface is always in the loading state (#9924)
- [Fix][ui] Set task group name empty of default value (#10243)
- [fix][deploy] File overwrite when specific value set to installPath (#10241)
- [fix] Deploy scp-hosts use fullpath avoid dangling (#10249)
- [Bug] init docker compose DATABASE env (#10244)
- [Fix-10181] Fix the logic of judging that the tenant does not exist (#10185)
- [Fix][ui] Fix status statistics card height is inconsistent. (#10268)
- [Fix] Fix the timeout alarm button is invalid (#10260)
- [Fix] Fix the line.separator problem for alert-plugin' test case #10277 (#10278)
- [fix] Fix the SERIAL_DISCARD execution_type not working (#10258)
- [Fix][ui] Add mapping_columns filed for the data_quality ruleInputParamter. (#10286)
- [Fix][ui] Display the result page when the DB page has no data. (#10298)
- [Fix-8828] [Master] Assign tasks to worker optimization (#9919)
- Fix database initialize late in standalone. (#10296)
- [Fix-10317] [ui] Fix the tips of startTimeout in the jupyter form (#10319)
- [Bug][api-server]Fix bug when user without tenant uploads file in resource center (#10303)
- [Fix] Privilege Reconstruction Data Query Fix & Resource Creation Post Operations (#10313)
- [Fix][ui] Fixed the issue that the status statistics on the project overview page were inconsistent. (#10328)
- [Fix-10305][common] AWS config Constant definitions are repeated (#10311)
- fix bug_10343 (#10344)
- [Fix-10318][ui] Fix EMR Task UI custom parameters Is not correct (#10334)
- [Fix-10338] Tenant does not belong to resource permission. (#10339)
- [Fix] [Worker] Fix worker will hang if fails to start (#10342)
- [fix][deploy]fix run on kubernetes bug (#10379)
- [Fix-10367] Permission check related key adjustment. (#10368)
- [Fix-10412] [Standalone] Failed to start standalone server according to the official document (#10414)
- Fixed the problem of not having permission to modify the allocated resource data. (#10410)
- [fix][python] Support same task name in project defferent process definition (#10428)
- [DolphinScheduler] Fix wrong class name in doc (#10438)
- [Bug][tools] Add datasource url for mysql profile in tools application.yaml (#10399)
- [Fix-10425]Recovery LDAP Config (#10429)
- [fix-10386]LDAP user create with status enable (#10426)
- [Fix][ui] Add loading before file modification data is returned. (#10450)
- [Bug-10394] [Update Possword] Change password shows error (#10437)
- [fix][python] Correct variable type of SQL type (#10464)
- fix rename form not init bug in file manange (#10471)
- Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe (#10479)
- [Fix] Fix the JSONUtils tool class time zone problem #10282 (#10284)
- [fix][master]: fix condition npe (#10478)
- Fix PeerTaskInstancePriorityQueue cannot contains method use taskInstanceId to check (#10491)
- [Fix]Recovery Traffic Control Config (#10505)
- [Fix-10413] Fix Master startup failure the server still hang (#10500)
- [FIX] Could not find or load main class org.apache.zookeeper.ZooKeeperMain (#10535)
- [Fix-10522] [API] Fix get executable time null pointer exception (#10523)
- [Fix-10452] Serial wait for policy recovery (#10453)
- [Fix-10274]Close InitialLDAPContext In LDAP (#10532)
- fix: Current node settings number of failed retries may not be negative (#10530)
- [fix][doc] Add DVC task plugin doc link (#10529)
- [Fix] fix the HttpUtilsTest' test case , it is always time out. (#10294)
- [fix] Change K8s postgresql password key name (#10545)
- [Fix-10513] Fix the doc of pseudo-cluster mysql installation error #10564
- [Fix-10547][data-quality] Fix Table or view not found #10549
- [Fix-10514] The master frequently print the same info or error logs, leads to insufficient disk space #10568
- fix standalone start error (#10577)
- fix dependent task re-run params error (#10580)
- [Fix-10400]Fix K8s deployment bitnami version error (#10567)
- [Fix][dolphinscheduler-spi] json parse support LocalDateTime (#10440)
- [Fix-10539] [Bug] [UI] Copy task a to B, change the user-defined parameter (#10611)
- [Fix][ui] Fix bytesToSize function calculation error. (#10627)
- [fix-10596] Workflow relationship cannot be displayed normally (#10625)
- [fix-10632][ui] fix the workflow name with underline cannot be displayed correctly error. (#10636)
- [Bug] [Master] Worker failover will cause task cannot be failover (#10631)
- [Bug] [Api] The copied workflow create time and update time is still copied from the original workflow
- [Fix-10641] [Common] Fix the hdfs NPE (#10642)
- [Bug] [Task] SQL task no exception stack trace in task log (#10656)
- [Fix-10652] fix docker-compose.yml (#10653)
- [Fix][ui] Fix bug where the home page displays no data. (#10697)
- fix bug where project overview page display no data (#10720)
- [Bug] [Api-Service] Workflow and subworkflow instance does not change after clicking rerun。 (#10727)
- [Fix-10666] Workflow submit failed will still in memory and never retry (#10667)
- Fix language not matched (#10775)
- [Bug][script] Remove deprecated arg in remove-zk-node.sh (#10789)
- [Fix][ui] Fix the environment name display back error. (#10797)
- [API] Do not specify rollback for ensure in case of any exception or error. (#10793)
- [Fix-10785] Fix state event handle error will not retry (#10786)
- [CI] Fix checkstyle & review dog don't work (#10812)
- [python] fix doc build error (#10810)
- [python] Fix submit and run error (#10792)
- [Fix-10829][ui] display real workflow relation in `WorkFlow Relation` page (#10836)
- [Fix-10842] Fix master/worker failover will cause status incorrect (#10839)
- Complement numbers will run in a loop under the serial strategy fixed. (#10862)
- [Bug][api] Complete Data workflow remove duplicates. (#10850)
- [Fixed-10833] [Bug] [Quartz] timezone display doesn't match the next_fire_time in ds 3.0.0-beta1 version (#10865)
- [Bug][api] Complete Data workflow remove duplicates. (#10868)
- [Fix-10863] Command execution result data statistics remove userId condition. (#10864)
- [Fix-10854] Fix database restart may lost task instance status (#10866)
- [fix] update queue and tenant with same code error (#10877)
- [Fix][ui] Fix hive datasource doesn't show principal bug (#10876)
- [Fix-10827] Fix network error cause worker cannot send message to master (#10886)
- [bug][task-plugins]fix switch class name and add param check to the switch task plugin (#10894)
- [fix] mlflow project success status (#10897)
- [fix] delete workflow or task used by other tasks (#10873)
- display timing record start time (#10915)
- fix bug where task name display in gantt (#10900)
- [Fix][ui] Add validator to the next node field of the switch task. (#10898)
- Catch exception when check state in StateWheelExecuteThread (#10908)
- Solve transaction problems under multi-threading (#10917)
- [fix][plugin]: fix spark sql use custom parma. (#10821)
- [FIX-10784][bug] [ALERT-SERVER] FEISHU Plugin might block the whole alert process loop (#10888)
- [fix] Can not query exists environments (#10884)
- [Fix-10909] Returns executionStatus for each process instance status and access token module support. (#10922)
- Fix taskInstance's host is not worker nettyServer address (#10926)
- Fix kill yarn job error when failover caused by doesn't set ProcessDefinition (#10948)
- [Fix][ui] Set the next node to not required. (#10932)
- [fix-10954]Querying the task group list returns empty data (#10955)
- fix: typo task definition. (#10958)
- [Fix-10890] fix some bugs in the Resource page (#10907)
- [Fix-10595][api] Fix Dependent Node Only View Projects Created By Self (#10834)
- [python] Fix tasks with multiple upstream and workflow query error (#10941)
- [fix] [MLflow Task plugin] Sometimes errors are reported using preset algorithms (#10986) (#10987)
- Kill task when we do master failover to avoid task lost (#10997)
- fix: typo executor service (#10980)
- Fix worker cannot shutdown due to resource close failed or heart beat check failed (#10979)
- [hotfix][worker] Remove service dependency from worker module (#11008)
- [Fix-11003]Task group queue is not updated to final state (#11004)
- [Bug] fix task params pass (#11053)
- [fix-10729]The request's url given by the test case looks different from the controller layer. Some usages and parameter format errors (#10734)
- [Fix-11031] AccessToken can only be used by the creator. (#11032)
- [E2E] [CI] Fix flaky e2e test (#11077)
- Try to fix filemanage e2e flaky test (#11097)
- add oracle dependency (#11095)
- [Fix][ui] Widen the segment separator input box. (#11096)
- fix view variable display bug (#11100)
- [Fix][ui] Fix the problem of too large spacing in operation. (#11090)
- fix space validate for create folder bug (#11106)
- [fix] py sagemaker doc (#11109)
- [Fix-10918] Close datasource when expire from guava cache (#11120)
- [fix][data quality] Data Quality Report Display For Admin Role #10919 (#10977)
- [Fix][ui] fix bug where The Gantt chart does not show the execution status (#11135)
- [Bug] [API] If the single log length is long, it will causes the View Log page to hang (#10612)
- [Fix][ui] Add duplicate of file suffix (#11136)
- fix k8s master and worker start error (#11156)
- [Bug-11101] fix task failover NPE (#11168)
- [Fix-10665] [S3] Fix s3 download method (#10675)
- [Fix-11140] Added resource parameters for function verification (#11141)
- [Fix] Parameter values disappear when editing Zeppelin task (#11115)
- [Bug-11071] fix local params (#11184)
- fix nodeSize is not a function bug (#11200)
- [BUG-11181] [Document] text check (#11183)
- [Fix-11007] [Master] fix forced_success bug (#11088)
- [Fix][ui] Fix datax task data instance replay error. (#11222)
- fix work oom when task logs's size is very large (#11224)
- Add alertPluginInstanceId in AlertInfo (#11231)
- Fix recovery from failed task will dead loop (#11239)
- fix bug where confirm disable error (#11251)
- [Fix][ui] fix bug where log modal display object (#11271)
- [fix-11233][ui] Remove custom template switch button in chunjun component module (#11234)
- Fix TaskGroup cannot work and will cause master dead loop (#11254)
- [fix] Process definition length too long when copy or import (#10621)
- no-admin user query alert group fix (#11301)
- [Fix][ui] Fix the taskPriority options not showing when hover. (#11294)
- [Bug][doc] Fix typos in docs which causes links to upgrade section to fail (#11333)
- [Fix][ui] fix bug where icons are displayed in one line. (#11320)
- [fix-11295] remove duplicate classes (#11355)
- [Fix-11366][ui] Workflow instance should not support right-click running (#11367)
- [Fix-11376][task plugin]Seatunnel Task cannot save successfully when select spark engine (#11377)
- [fix-11378] fix API-Server release response message typo (#11400)
- [Fix-10392][sqltask]: Fix list param error when use sql task (#11285)
- [Fix-11051][task]Fix the process exitValue exception when the process times out and the process does not end. (#11099)
- [Fix-1360]: Fix Description length check (#11371)
- [Fix-11413] Cannot set task status to kill if the task is not in running(#11414)
- [Fix][ui] fix bug where warn group name not display (#11429)
- [fix-10961][alert server]Change the content of alert to an array (#11033)
- Fix find last process instance in dependent task is not correct (#11424)
- Fix workflow instance may failover many times due to doesn't check the restart time (#11445)
- Set master's task running status in `runTask` to avoid the task group acquire failed, but the task status is in running (#11451)
- If the task processor is not found need to throw error rather than exception (#11461)
- [Fix-11465] Serial first, the last executing instance will be stopped (#11466)
- [Fix][ui] Fix the table header in the workflow instance. (#11487)
- [Fix][ui] Fix the preTask options are unavailable when creating a task in the task definition page. (#11501)
- [fix] Database change not sync (#11258)
- [Fix][ui] Remove unsupported datasource types in the datax task. (#11527)
- Set process instance and task priority default value as 2 (#11539)
- [fix-11404]: make the common.properties to configurable on values.yaml (#11441)
- Fix some resource leak (#11576)
- [Fix][issue-11579] Cannot check the application status. (#11580)
- Fix vulnerability in LDAP login (#11586)
- Refresh server startup time when recovery from waiting (#11595)
- Fix quartz threadPriority config name error (#11596)
- Fix kill task failed will cause the taskGroup cannot release and add taskGroup log (#11469)
- [Fix-9980] [Server] fix heartBeatTaskCount bug (#11232)
- Fix homepage email subscription link (#11622)
- [Fix][ui] Fix the port in the datasource edit. (#11624)
- [Fix-11616] Workflow Definition Name Change Validation Optimization. (#11617)
- [Fix][ui] Fix the problem that the pagination component is not centered in the environment management. (#11646)
- [Fix][ui] Fix bug where crontab special of month and year (#11661)
- [fix][python] Sql pattern add truncate. (#11666)
- [bug][python] Fix task switch error when default branch not defined last (#11606)
- [Fix-11633] The data type of the PostgreSQL database column is wrong. (#11633) (#11634)
- [Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task (#11701)
- [Fix][ui] Add the task name entry when the workflow instance detail page goes to the task instance page. (#11761)
- [fix][doc] Update the registry related configuration in values.yaml (#11444) (#11981)
- [Bug][workflow definition] fix ordinary users can not create depend task (#11961) (#11977)
- [Bug-11650][worker] #11650 fix SQL type task, stop task cause NPE (#11668) (#11958)
- [Bug] [Master] WorkflowExecuteRunnable will face a infinite loop #11838 (#11864) (#11950)
- [Fix][db] fix init&upgrade mysql-meta-schema bugs (#11887) (#11935)
- [Bug] Fix procedure task param pass #11919 (#11926)
- [Fix-11217] [Doc] add postgresql config in doc: datasource-setting (#11326)
- [fix-#11689] change replaceAll to replace in method of tryExecuteSqlResolveColumnNames (#11696)
- [fix-#11691] Add lz4-java in clickhouse datasource. (#11697)
- Example Change the regular expression of the data source password (#11459) (#11990)
- Fix insertOrUpdate plugin may failed due to concurrent operation (#11471) (#11996)
- Update dolphinscheduler_ddl.sql fix alert t_ds_user add time_zone is not correct (#11795)
- [python] Fix permission error when create related objects (#11783)
- [Bug][dependent]: Id also clone due to duplicate when use dependent mode. (#11929)
- Fix listing processDefinition the schedulerReleaseState will never be null (#11888)
- Add chunjun in dolphinscheduler-task-all/pom.xml (#11375)
- [Bug] [dolphinscheduler-api] Data Analysis Service API responds with jackson deserialization exception (#11858)
- [Fix][monitor]Monitor UI not show DisakAvailable and MemoryUsage correctly (#11870)
- fix workflow keep running when task fail (#11930)
- Fix cannot save processDefinition (#11931)
- [Bug](dependent) Dependent downstream trigger error when schedule cycle not day. (#11734)
- Fix update TaskDefinition error (#12064)
- [fix][python] Task switch branch not show in webui (#12120)
- [Fix] Also auto formatter workflow instance if location is null (#12080)
- fix python home conflict (#12112)
- [Fix][ui] Fix workflow name overlaps bug (#11862) (#12125)
- [BUG]Delete Setting ID (#12096)
- Fix insert command error due to the id is not null (#12092) (#12098)
- [BUG][be-12113]fix htrace dependency conflicts (#12126)
- [bug][k8s] Change the config file keys in K8s to be the same as those in common.properties (#11812)
- [k8s] Delete unnecessary configurations. (#11808)
- fix upgrade for 310 (#12137)
- [fix#12000]Cannot remove the WorkerGroup from the master service (#12050)
- fix env config space bug (#12147)
## Doc
- [Docs] Upgrade license. (#9967)
- [Docs] Update README. (#9970)
- [doc] Fix docs deadlink in readme (#9972)
- [doc] Update the document picture to new UI (#9953)
- [Doc] Add example and notice about task type Conditions (#9945)
- [Docs] Fix docker link. (#9998)
- [doc] Add the description about execute type in SQL task (#9987)
- [doc] Add example and notice about task type Dependent (#10001)
- [doc] Correct kubernetes (#9985)
- [doc] Correct docs of development-environment-setup (#9995)
- [Bug][document] Fix task group management document link error (#10062)
- [Fix-10083][doc]Change Docker cmd && Change WebexTeams Chinese Translation (#10084)
- [doc] Show all the functions of the home page through screenshots (#10118)
- [doc] change document structure (#10124)
- [doc] Remove flink call docs (#10122)
- [doc] Update editorial workflow content and images (#10130)
- [doc] Update website docs link (#10161)
- [doc] Refactor global parameter docment (#10045)
- [doc] Update project directory structure in docs (#10107)
- [Doc] Add complie supporting system (#10188)
- [doc] Update delete workflow description (#10203)
- [doc] Update task DataX document (#10218)
- [doc] Update kubernetes' FAQ (#10221)
- [doc] Refactor local parameter docment (#10119)
- [doc] Delete SPRING_DATASOURCE_DRIVER_CLASS_NAME in doc (#10228)
- [doc] Add setValue syntax in local param (#10245)
- [doc] Correct all setup documents (#10074)
- [doc] Upgrade screenshot to latest version in dev document (#10242)
- [doc] Refactor context parameter docment (#10227)
- [docs] Added local file configuration guide for resource center (#10264)
- [doc] Update the development environment to build the software version (#10246)
- [doc] Fix api standard error (#10292)
- [doc] Add how to use resource in shell task (#10308)
- [doc] Use related path for img (#10325)
- [Docs]Update Slack Invite Link (#10362)
- [Docs] Upgrade license. (#10365)
- [doc] Add switch task demo (#10247)
- [doc] Correct resource in HDFS HA cluster (#10445)
- [Doc] Add how to set the alarm mailbox in faq (#10401)
- [Doc] Add api test doc (#10499)
- [doc] Correct the complement add docs (#10521)
- [doc] Add docs about complement dependent mode (#10587)
- [doc] Add doc about how to use healthcheck endpoints (#10592)
- [doc] Add run the task alone doc. (#10581)
- [docs] Add links to metrics docs (#10583)
- [doc] Correct installPath when using related path (#10597)
- [doc] Modify Shell task file (#10634)
- [doc] Change subprocess dependent (#10637)
- [doc] Modified parameter doc files. (#10659)
- [doc] Replace the dag screenshot of the README. (#10662)
- [doc] Update metadata and design doc (#10600)
- [doc] Add doc for pre-sql and post-sql (#10685)
- [doc] Change datasource doc (#10660)
- [doc] Change tasks doc (#10639)
- [doc] Modified alert doc files (#10714)
- [doc] Modify quick start. (#10712)
- Merging general settings to faq (#10716)
- [Docs] Replace the dag screenshot of the README. (#10741)
- [doc] Modified project doc. (#10711)
- [doc] Change tasks docs (#10751)
- [doc] Unify data source switching of the docs during installation (#10688)
- [doc] Replace the dag screenshot of the README. (#10798)
- [doc] Modified dq, monitor, security, resources (#10715)
- [doc] Modify features, hw, intro and homepage docs. (#10713)
- [doc] Correct custom jdbc driver for K8s Doc (#10768)
- [Docs] Updated license. (#10838)
- [Doc][upgrade] Fix tiny error in upgrade docs (#10847)
- [doc] Add missing doc change by snehal (#10892)
- [Doc]Chinese translation updated (#11146)
- [doc] Fix bugs in doc installation, design and glossary (#11084)
- [doc] Add new page incompatible change (#11153)
- [doc] Add notice when environment can not select (#11276)
- [Doc] update the configuration doc (#11113)
- [doc] migrate community and development docs into main repo (#11313)
- [Doc] Add tools/libs to the init database steps (#11345)
- [doc] Fixed hadoop link to avoid dead link (#11363)
- [doc] Improve the document introduction of complement (#11275)
- [doc] Refine the deployment documentation (#11361)
- [doc] Recover traffic configuration (#11288)
- [doc] Add how obtain IP address from network card in FAQ (#11311)
- [Doc][resources] Instruct users to use local storage if they have remote storage mounted to local (#11435)
- [Doc][dsip] Move DSIP docs to the right place (#11442) (#11443)
- [doc] Use Rainbond deployment DolphinScheduler Cluster (#10951)
- [Doc][security] Update instructions on worker groups (#11483)
- [Doc] Update UI readme. (#11496)
- [Doc][bug] Fix resource center docs for standalone mode (#11600)
- [DOC] should notice that need setting the full path when call the resource file #11620 (#11621)
- [Docs] Update README. (#11641)
- [doc] Correct E2E Doc, fix WorkerGroupPage typo (#11629)
- [DOC] improve zk digest doc (#11703)
- [Doc][improvement] Add instructions for process execution type and data complement (#12121)
- add Feishu alert doc and configure email alert doc to the docsdev.js (#11173)
- [doc] Fix installation typo (#11220)
- [Improvement-11481][docs] Add log specification document to contribution guidelines. (#11484)
- Enable spotless to fix markdown checkstyle and refactor faq.md as an example (#11384)
- Update doc according to recent update of Spring / Swagger (#11755)
- [Doc] Put default task parameters in a new file #11776 (#11957)
- [Fix-10830][doc] Change the start or install script command sh in the document to bash (#10831)
- [Doc][style] Fix doc format once for all (#12006)
## Chore
- [ci] Correct CODEOWNERS and add new pom.xml owner (#9954)
- [Build][ui] Upgrade dependencies. (#9956)
- [chore] Combine version 206 and 210 sql upgrade to 300 (#9959)
- [ci] Dead link check all markdown file (#10004)
- [chore] update by release version (#10299)
- [ci] Fix mergeable for ddl change and remove label unexpect (#7099)
- [CI] bumping actions/cache to v3 to fix ci error (#10345)
- [Build][ui] Upgrade dependencies. (#10346)
- [CI] Add code owner of .github (#10347)
- [CI] Add some codeowner (#10385)
- [chore] Remove unnecessary badge total lines (#10405)
- add codeowner (#10457)
- add_some_code_owners (#10462)
- Open checkstyle, change checkstyle severity from info to warning (#10492)
- add codeowner (#10537)
- [chore] update by release version (#10742)
- [ci] mergable do not handle label (#10638)
- Add improvement template (#10764)
- Add doc issue template (#10804)
- [API-Test] [Checkstyle] Reformat api-test code and enable checkstyle (#10805)
- add reviewdog token to enable checkstyle (#10819)
- [Build][ui] Updated dependencies. (#10813)
- [CI] Try to fix label bot (#10852)
- [ci] Fix mergeable will get mad add/remove label sometimes (#10856)
- fix label bot again (#10859)
- [python] Integrate test run strategy change (#10825)
- Add EricGao888 to help review docs, tools, etc. (#11047)
- [python] Add directly install without clone source code (#11179)
- [ci] Make cluster test work on release branch (#11199)
- [community] Remove stackoverflow ref link in website (#11342)
- Modifying Slack Links (#11416)
- update PyDolphinScheduler documentation link. (#11474)
- update slack (#11578)
- [CI] Fix cluster test abnormal (#11688)
- add lombok config (#11212)

View File

@ -1,9 +1,11 @@
# API design standard
A standardized and unified API is the cornerstone of project design.The API of DolphinScheduler follows the REST ful standard. REST ful is currently the most popular Internet software architecture. It has a clear structure, conforms to standards, is easy to understand and extend.
This article uses the DolphinScheduler API as an example to explain how to construct a Restful API.
## 1. URI design
REST is "Representational State Transfer".The design of Restful URI is based on resources.The resource corresponds to an entity on the network, for example: a piece of text, a picture, and a service. And each resource corresponds to a URI.
+ One Kind of Resource: expressed in the plural, such as `task-instances`、`groups` ;
@ -12,36 +14,43 @@ REST is "Representational State Transfer".The design of Restful URI is based on
+ A Sub Resource`/instances/{instanceId}/tasks/{taskId}`;
## 2. Method design
We need to locate a certain resource by URI, and then use Method or declare actions in the path suffix to reflect the operation of the resource.
### ① Query - GET
Use URI to locate the resource, and use GET to indicate query.
+ When the URI is a type of resource, it means to query a type of resource. For example, the following example indicates paging query `alter-groups`.
```
Method: GET
/dolphinscheduler/alert-groups
```
+ When the URI is a single resource, it means to query this resource. For example, the following example means to query the specified `alter-group`.
```
Method: GET
/dolphinscheduler/alter-groups/{id}
```
+ In addition, we can also express query sub-resources based on URI, as follows:
```
Method: GET
/dolphinscheduler/projects/{projectId}/tasks
```
**The above examples all represent paging query. If we need to query all data, we need to add `/list` after the URI to distinguish. Do not mix the same API for both paged query and query.**
```
Method: GET
/dolphinscheduler/alert-groups/list
```
### ② Create - POST
Use URI to locate the resource, use POST to indicate create, and then return the created id to requester.
+ create an `alter-group`
@ -52,35 +61,42 @@ Method: POST
```
+ create sub-resources is also the same as above.
```
Method: POST
/dolphinscheduler/alter-groups/{alterGroupId}/tasks
```
### ③ Modify - PUT
Use URI to locate the resource, use PUT to indicate modify.
+ modify an `alert-group`
```
Method: PUT
/dolphinscheduler/alter-groups/{alterGroupId}
```
### ④ Delete -DELETE
Use URI to locate the resource, use DELETE to indicate delete.
+ delete an `alert-group`
```
Method: DELETE
/dolphinscheduler/alter-groups/{alterGroupId}
```
+ batch deletion: batch delete the id arraywe should use POST. **Do not use the DELETE method, because the body of the DELETE request has no semantic meaning, and it is possible that some gateways, proxies, and firewalls will directly strip off the request body after receiving the DELETE request.**
```
Method: POST
/dolphinscheduler/alter-groups/batch-delete
```
### ⑤ Partial Modifications -PATCH
Use URI to locate the resource, use PATCH to partial modifications.
```
@ -89,20 +105,27 @@ Method: PATCH
```
### ⑥ Others
In addition to creating, deleting, modifying and quering, we also locate the corresponding resource through url, and then append operations to it after the path, such as:
```
/dolphinscheduler/alert-groups/verify-name
/dolphinscheduler/projects/{projectCode}/process-instances/{code}/view-gantt
```
## 3. Parameter design
There are two types of parameters, one is request parameter and the other is path parameter. And the parameter must use small hump.
In the case of paging, if the parameter entered by the user is less than 1, the front end needs to automatically turn to 1, indicating that the first page is requested; When the backend finds that the parameter entered by the user is greater than the total number of pages, it should directly return to the last page.
## 4. Others design
### base URL
The URI of the project needs to use `/<project_name>` as the base path, so as to identify that these APIs are under this project.
```
/dolphinscheduler
```

View File

@ -10,7 +10,6 @@ In contrast, API testing focuses on whether a complete operation chain can be co
For example, the API test of the tenant management interface focuses on whether users can log in normally; If the login fails, whether the error message can be displayed correctly. After logging in, you can perform tenant management operations through the sessionid you carry.
## API Test
### API-Pages
@ -49,7 +48,6 @@ In addition, during the testing process, the interface are not requested directl
On the login page, only the input parameter specification of the interface request is defined. For the output parameter of the interface request, only the unified basic response structure is defined. The data actually returned by the interface is tested in the actual test case. Whether the input and output of main test interfaces can meet the requirements of test cases.
### API-Cases
The following is an example of a tenant management test. As explained earlier, we use docker-compose for deployment, so for each test case, we need to import the corresponding file in the form of an annotation.

View File

@ -1,4 +1,5 @@
## Architecture Design
Before explaining the architecture of the schedule system, let us first understand the common nouns of the schedule system.
### 1.Noun Interpretation
@ -34,11 +35,10 @@ Before explaining the architecture of the schedule system, let us first understa
**Complement**: Complement historical data, support **interval parallel and serial** two complement methods
### 2.System architecture
#### 2.1 System Architecture Diagram
<p align="center">
<img src="../../../img/architecture.jpg" alt="System Architecture Diagram" />
<p align="center">
@ -46,60 +46,51 @@ Before explaining the architecture of the schedule system, let us first understa
</p>
</p>
#### 2.2 Architectural description
* **MasterServer**
MasterServer adopts the distributed non-central design concept. MasterServer is mainly responsible for DAG task split, task submission monitoring, and monitoring the health status of other MasterServer and WorkerServer.
When the MasterServer service starts, it registers a temporary node with Zookeeper, and listens to the Zookeeper temporary node state change for fault tolerance processing.
MasterServer adopts the distributed non-central design concept. MasterServer is mainly responsible for DAG task split, task submission monitoring, and monitoring the health status of other MasterServer and WorkerServer.
When the MasterServer service starts, it registers a temporary node with Zookeeper, and listens to the Zookeeper temporary node state change for fault tolerance processing.
##### The service mainly contains:
- **Distributed Quartz** distributed scheduling component, mainly responsible for the start and stop operation of the scheduled task. When the quartz picks up the task, the master internally has a thread pool to be responsible for the subsequent operations of the task.
##### The service mainly contains:
- **Distributed Quartz** distributed scheduling component, mainly responsible for the start and stop operation of the scheduled task. When the quartz picks up the task, the master internally has a thread pool to be responsible for the subsequent operations of the task.
- **MasterSchedulerThread** is a scan thread that periodically scans the **command** table in the database for different business operations based on different **command types**
- **MasterExecThread** is mainly responsible for DAG task segmentation, task submission monitoring, logic processing of various command types
- **MasterTaskExecThread** is mainly responsible for task persistence
- **MasterSchedulerThread** is a scan thread that periodically scans the **command** table in the database for different business operations based on different **command types**
- **MasterExecThread** is mainly responsible for DAG task segmentation, task submission monitoring, logic processing of various command types
- **MasterTaskExecThread** is mainly responsible for task persistence
* **WorkerServer**
- WorkerServer also adopts a distributed, non-central design concept. WorkerServer is mainly responsible for task execution and providing log services. When the WorkerServer service starts, it registers the temporary node with Zookeeper and maintains the heartbeat.
- WorkerServer also adopts a distributed, non-central design concept. WorkerServer is mainly responsible for task execution and providing log services. When the WorkerServer service starts, it registers the temporary node with Zookeeper and maintains the heartbeat.
##### This service contains:
##### This service contains:
- **FetchTaskThread** is mainly responsible for continuously receiving tasks from **Task Queue** and calling **TaskScheduleThread** corresponding executors according to different task types.
- **FetchTaskThread** is mainly responsible for continuously receiving tasks from **Task Queue** and calling **TaskScheduleThread** corresponding executors according to different task types.
- **ZooKeeper**
- **ZooKeeper**
The ZooKeeper service, the MasterServer and the WorkerServer nodes in the system all use the ZooKeeper for cluster management and fault tolerance. In addition, the system also performs event monitoring and distributed locking based on ZooKeeper.
We have also implemented queues based on Redis, but we hope that DolphinScheduler relies on as few components as possible, so we finally removed the Redis implementation.
The ZooKeeper service, the MasterServer and the WorkerServer nodes in the system all use the ZooKeeper for cluster management and fault tolerance. In addition, the system also performs event monitoring and distributed locking based on ZooKeeper.
We have also implemented queues based on Redis, but we hope that DolphinScheduler relies on as few components as possible, so we finally removed the Redis implementation.
- **Task Queue**
- **Task Queue**
The task queue operation is provided. Currently, the queue is also implemented based on Zookeeper. Since there is less information stored in the queue, there is no need to worry about too much data in the queue. In fact, we have over-measured a million-level data storage queue, which has no effect on system stability and performance.
The task queue operation is provided. Currently, the queue is also implemented based on Zookeeper. Since there is less information stored in the queue, there is no need to worry about too much data in the queue. In fact, we have over-measured a million-level data storage queue, which has no effect on system stability and performance.
- **Alert**
- **Alert**
Provides alarm-related interfaces. The interfaces mainly include **Alarms**. The storage, query, and notification functions of the two types of alarm data. The notification function has two types: **mail notification** and **SNMP (not yet implemented)**.
Provides alarm-related interfaces. The interfaces mainly include **Alarms**. The storage, query, and notification functions of the two types of alarm data. The notification function has two types: **mail notification** and **SNMP (not yet implemented)**.
- **API**
- **API**
The API interface layer is mainly responsible for processing requests from the front-end UI layer. The service provides a RESTful api to provide request services externally.
Interfaces include workflow creation, definition, query, modification, release, offline, manual start, stop, pause, resume, start execution from this node, and more.
- **UI**
The front-end page of the system provides various visual operation interfaces of the system. For details, see the [quick start](https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/about/introduction.html) section.
The API interface layer is mainly responsible for processing requests from the front-end UI layer. The service provides a RESTful api to provide request services externally.
Interfaces include workflow creation, definition, query, modification, release, offline, manual start, stop, pause, resume, start execution from this node, and more.
- **UI**
The front-end page of the system provides various visual operation interfaces of the system. For details, see the [quick start](https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/about/introduction.html) section.
#### 2.3 Architectural Design Ideas
@ -130,10 +121,9 @@ Problems in the design of centralized :
- In the decentralized design, there is usually no Master/Slave concept, all roles are the same, the status is equal, the global Internet is a typical decentralized distributed system, networked arbitrary node equipment down machine , all will only affect a small range of features.
- The core design of decentralized design is that there is no "manager" that is different from other nodes in the entire distributed system, so there is no single point of failure problem. However, since there is no "manager" node, each node needs to communicate with other nodes to get the necessary machine information, and the unreliable line of distributed system communication greatly increases the difficulty of implementing the above functions.
- In fact, truly decentralized distributed systems are rare. Instead, dynamic centralized distributed systems are constantly emerging. Under this architecture, the managers in the cluster are dynamically selected, rather than preset, and when the cluster fails, the nodes of the cluster will spontaneously hold "meetings" to elect new "managers". Go to preside over the work. The most typical case is the Etcd implemented in ZooKeeper and Go.
- Decentralization of DolphinScheduler is the registration of Master/Worker to ZooKeeper. The Master Cluster and the Worker Cluster are not centered, and the Zookeeper distributed lock is used to elect one Master or Worker as the “manager” to perform the task.
##### 二、Distributed lock practice
##### 二、Distributed lock practice
DolphinScheduler uses ZooKeeper distributed locks to implement only one Master to execute the Scheduler at the same time, or only one Worker to perform task submission.
@ -184,8 +174,6 @@ Service fault tolerance design relies on ZooKeeper's Watcher mechanism. The impl
The Master monitors the directories of other Masters and Workers. If the remove event is detected, the process instance is fault-tolerant or the task instance is fault-tolerant according to the specific business logic.
- Master fault tolerance flow chart:
<p align="center">
@ -194,8 +182,6 @@ The Master monitors the directories of other Masters and Workers. If the remove
After the ZooKeeper Master is fault-tolerant, it is rescheduled by the Scheduler thread in DolphinScheduler. It traverses the DAG to find the "Running" and "Submit Successful" tasks, and monitors the status of its task instance for the "Running" task. You need to determine whether the Task Queue already exists. If it exists, monitor the status of the task instance. If it does not exist, resubmit the task instance.
- Worker fault tolerance flow chart:
<p align="center">
@ -204,7 +190,7 @@ After the ZooKeeper Master is fault-tolerant, it is rescheduled by the Scheduler
Once the Master Scheduler thread finds the task instance as "need to be fault tolerant", it takes over the task and resubmits.
Note: Because the "network jitter" may cause the node to lose the heartbeat of ZooKeeper in a short time, the node's remove event occurs. In this case, we use the easiest way, that is, once the node has timeout connection with ZooKeeper, it will directly stop the Master or Worker service.
Note: Because the "network jitter" may cause the node to lose the heartbeat of ZooKeeper in a short time, the node's remove event occurs. In this case, we use the easiest way, that is, once the node has timeout connection with ZooKeeper, it will directly stop the Master or Worker service.
###### 2. Task failure retry
@ -214,8 +200,6 @@ Here we must first distinguish between the concept of task failure retry, proces
- Process failure recovery is process level, is done manually, recovery can only be performed **from the failed node** or **from the current node**
- Process failure rerun is also process level, is done manually, rerun is from the start node
Next, let's talk about the topic, we divided the task nodes in the workflow into two types.
- One is a business node, which corresponds to an actual script or processing statement, such as a Shell node, an MR node, a Spark node, a dependent node, and so on.
@ -225,16 +209,12 @@ Each **service node** can configure the number of failed retries. When the task
If there is a task failure in the workflow that reaches the maximum number of retries, the workflow will fail to stop, and the failed workflow can be manually rerun or process resumed.
##### V. Task priority design
In the early scheduling design, if there is no priority design and fair scheduling design, it will encounter the situation that the task submitted first may be completed simultaneously with the task submitted subsequently, but the priority of the process or task cannot be set. We have redesigned this, and we are currently designing it as follows:
- According to **different process instance priority** prioritizes **same process instance priority** prioritizes **task priority within the same process** takes precedence over **same process** commit order from high Go to low for task processing.
- The specific implementation is to resolve the priority according to the json of the task instance, and then save the **process instance priority _ process instance id_task priority _ task id** information in the ZooKeeper task queue, when obtained from the task queue, Through string comparison, you can get the task that needs to be executed first.
- The priority of the process definition is that some processes need to be processed before other processes. This can be configured at the start of the process or at the time of scheduled start. There are 5 levels, followed by HIGHEST, HIGH, MEDIUM, LOW, and LOWEST. As shown below
<p align="center">
@ -308,8 +288,6 @@ Public class TaskLogFilter extends Filter<ILoggingEvent> {
}
```
### summary
Starting from the scheduling, this paper introduces the architecture principle and implementation ideas of the big data distributed workflow scheduling system-DolphinScheduler. To be continued

View File

@ -59,3 +59,4 @@ Assign the parameters with matching values to varPool (List, which contains the
* Format the varPool as json and pass it to master.
* The parameters that are OUT would be written into the localParam after the master has received the varPool.

View File

@ -1,6 +1,6 @@
# Overview
<!-- TODO Since the side menu does not support multiple levels, add new page to keep all sub page here -->
* [Global Parameter](global-parameter.md)
* [Switch Task type](task/switch.md)

View File

@ -6,3 +6,4 @@ Switch task workflow step as follows
* `SwitchTaskExecThread` processes the expressions defined in `switch` from top to bottom, obtains the value of the variable from `varPool`, and parses the expression through `javascript`. If the expression returns true, stop checking and record The order of the expression, here we record as resultConditionLocation. The task of SwitchTaskExecThread is over
* After the `switch` task runs, if there is no error (more commonly, the user-defined expression is out of specification or there is a problem with the parameter name), then `MasterExecThread.submitPostNode` will obtain the downstream node of the `DAG` to continue execution.
* If it is found in `DagHelper.parsePostNodes` that the current node (the node that has just completed the work) is a `switch` node, the `resultConditionLocation` will be obtained, and all branches except `resultConditionLocation` in the SwitchParameters will be skipped. In this way, only the branches that need to be executed are left

View File

@ -26,8 +26,8 @@ If you don't care about its internal design, but simply want to know how to deve
This module is currently a plug-in provided by us, and now we have supported dozens of plug-ins, such as Email, DingTalk, Script, etc.
#### Alert SPI Main class information.
AlertChannelFactory
Alarm plug-in factory interface. All alarm plug-ins need to implement this interface. This interface is used to define the name of the alarm plug-in and the required parameters. The create method is used to create a specific alarm plug-in instance.
@ -56,36 +56,40 @@ The specific design of alert_spi can be seen in the issue: [Alert Plugin Design]
* Email
Email alert notification
Email alert notification
* DingTalk
Alert for DingTalk group chat bots
Alert for DingTalk group chat bots
Related parameter configuration can refer to the DingTalk robot document.
Related parameter configuration can refer to the DingTalk robot document.
* EnterpriseWeChat
EnterpriseWeChat alert notifications
EnterpriseWeChat alert notifications
Related parameter configuration can refer to the EnterpriseWeChat robot document.
Related parameter configuration can refer to the EnterpriseWeChat robot document.
* Script
We have implemented a shell script for alerting. We will pass the relevant alert parameters to the script and you can implement your alert logic in the shell. This is a good way to interface with internal alerting applications.
We have implemented a shell script for alerting. We will pass the relevant alert parameters to the script and you can implement your alert logic in the shell. This is a good way to interface with internal alerting applications.
* SMS
SMS alerts
SMS alerts
* FeiShu
FeiShu alert notification
* Slack
Slack alert notification
* PagerDuty
PagerDuty alert notification
* WebexTeams
WebexTeams alert notification
@ -101,3 +105,4 @@ The specific design of alert_spi can be seen in the issue: [Alert Plugin Design]
* Http
We have implemented a Http script for alerting. And calling most of the alerting plug-ins end up being Http requests, if we not support your alert plug-in yet, you can use Http to realize your alert login. Also welcome to contribute your common plug-ins to the community :)

View File

@ -6,9 +6,10 @@ Make the following configuration (take zookeeper as an example)
* Registry plug-in configuration, take Zookeeper as an example (registry.properties)
dolphinscheduler-service/src/main/resources/registry.properties
```registry.properties
registry.plugin.name=zookeeper
registry.servers=127.0.0.1:2181
registry.plugin.name=zookeeper
registry.servers=127.0.0.1:2181
```
For specific configuration information, please refer to the parameter information provided by the specific plug-in, for example zk: `org/apache/dolphinscheduler/plugin/registry/zookeeper/ZookeeperConfiguration.java`

View File

@ -1,6 +1,7 @@
# DolphinScheduler development
## Software Requirements
Before setting up the DolphinScheduler development environment, please make sure you have installed the software as below:
* [Git](https://git-scm.com/downloads)
@ -46,6 +47,7 @@ fix things for you.
DolphinScheduler will release new Docker images after it released, you could find them in [Docker Hub](https://hub.docker.com/search?q=DolphinScheduler).
* If you want to modify DolphinScheduler source code, and build Docker images locally, you can run when finished the modification
```shell
cd dolphinscheduler
./mvnw -B clean package \
@ -59,6 +61,7 @@ cd dolphinscheduler
When the command is finished you could find them by command `docker imaegs`.
* If you want to modify DolphinScheduler source code, build and push Docker images to your registry <HUB_URL>you can run when finished the modification
```shell
cd dolphinscheduler
./mvnw -B clean deploy \
@ -92,7 +95,6 @@ RUN apt update ; \
>
> Have to use version after Docker 19.03, because after 19.03 docker contains buildx
## Notice
There are two ways to configure the DolphinScheduler development environment, standalone mode and normal mode
@ -122,6 +124,7 @@ Find the class `org.apache.dolphinscheduler.StandaloneServer` in Intellij IDEA a
### Start frontend server
Install frontend dependencies and run it.
> Note: You can see more detail about the frontend setting in [frontend development](./frontend-development.md).
```shell
@ -130,7 +133,7 @@ pnpm install
pnpm run dev
```
The browser access address [http://localhost:3000](http://localhost:3000) can login DolphinScheduler UI. The default username and password are **admin/dolphinscheduler123**
The browser access address [http://localhost:5173](http://localhost:5173) can login DolphinScheduler UI. The default username and password are **admin/dolphinscheduler123**
## DolphinScheduler Normal Mode
@ -143,12 +146,11 @@ Download [ZooKeeper](https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3
* Create directory `zkData` and `zkLog`
* Go to the zookeeper installation directory, copy configure file `zoo_sample.cfg` to `conf/zoo.cfg`, and change value of dataDir in conf/zoo.cfg to dataDir=./tmp/zookeeper
```shell
# We use path /data/zookeeper/data and /data/zookeeper/datalog here as example
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/datalog
```
```shell
# We use path /data/zookeeper/data and /data/zookeeper/datalog here as example
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/datalog
```
* Run `./bin/zkServer.sh` in terminal by command `./bin/zkServer.sh start`.
#### Database
@ -166,21 +168,22 @@ Following steps will guide how to start the DolphinScheduler backend service
* Open project: Use IDE open the project, here we use Intellij IDEA as an example, after opening it will take a while for Intellij IDEA to complete the dependent download
* File change
* If you use MySQL as your metadata database, you need to modify `dolphinscheduler/pom.xml` and change the `scope` of the `mysql-connector-java` dependency to `compile`. This step is not necessary to use PostgreSQL
* Modify database configuration, modify the database configuration in the `dolphinscheduler-master/src/main/resources/application.yaml`
* Modify database configuration, modify the database configuration in the `dolphinscheduler-worker/src/main/resources/application.yaml`
* Modify database configuration, modify the database configuration in the `dolphinscheduler-api/src/main/resources/application.yaml`
We here use MySQL with database, username, password named dolphinscheduler as an example
```application.yaml
spring:
datasource:
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8
username: dolphinscheduler
password: dolphinscheduler
```
```application.yaml
spring:
datasource:
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8
username: dolphinscheduler
password: dolphinscheduler
```
* Log level: add a line `<appender-ref ref="STDOUT"/>` to the following configuration to enable the log to be displayed on the command line
@ -220,4 +223,4 @@ pnpm install
pnpm run dev
```
The browser access address [http://localhost:3000](http://localhost:3000) can login DolphinScheduler UI. The default username and password are **admin/dolphinscheduler123**
The browser access address [http://localhost:5173](http://localhost:5173) can login DolphinScheduler UI. The default username and password are **admin/dolphinscheduler123**

View File

@ -77,31 +77,31 @@ In addition, during the testing process, the elements are not manipulated direct
The SecurityPage provides goToTab methods to test the corresponding sidebar jumps, mainly including TenantPage, UserPage, WorkerGroupPage and QueuePage. These pages are implemented in the same way, mainly to test whether the input, add and delete buttons of the form can return to the corresponding page.
```java
public <T extends SecurityPage.Tab> T goToTab(Class<T> tab) {
if (tab == TenantPage.class) {
WebElement menuTenantManageElement = new WebDriverWait(driver, 60)
.until(ExpectedConditions.elementToBeClickable(menuTenantManage));
((JavascriptExecutor)driver).executeScript("arguments[0].click();", menuTenantManageElement);
return tab.cast(new TenantPage(driver));
}
if (tab == UserPage.class) {
WebElement menUserManageElement = new WebDriverWait(driver, 60)
.until(ExpectedConditions.elementToBeClickable(menUserManage));
((JavascriptExecutor)driver).executeScript("arguments[0].click();", menUserManageElement);
return tab.cast(new UserPage(driver));
}
if (tab == WorkerGroupPage.class) {
WebElement menWorkerGroupManageElement = new WebDriverWait(driver, 60)
.until(ExpectedConditions.elementToBeClickable(menWorkerGroupManage));
((JavascriptExecutor)driver).executeScript("arguments[0].click();", menWorkerGroupManageElement);
return tab.cast(new WorkerGroupPage(driver));
}
if (tab == QueuePage.class) {
menuQueueManage().click();
return tab.cast(new QueuePage(driver));
}
throw new UnsupportedOperationException("Unknown tab: " + tab.getName());
}
public <T extends SecurityPage.Tab> T goToTab(Class<T> tab) {
if (tab == TenantPage.class) {
WebElement menuTenantManageElement = new WebDriverWait(driver, 60)
.until(ExpectedConditions.elementToBeClickable(menuTenantManage));
((JavascriptExecutor)driver).executeScript("arguments[0].click();", menuTenantManageElement);
return tab.cast(new TenantPage(driver));
}
if (tab == UserPage.class) {
WebElement menUserManageElement = new WebDriverWait(driver, 60)
.until(ExpectedConditions.elementToBeClickable(menUserManage));
((JavascriptExecutor)driver).executeScript("arguments[0].click();", menUserManageElement);
return tab.cast(new UserPage(driver));
}
if (tab == WorkerGroupPage.class) {
WebElement menWorkerGroupManageElement = new WebDriverWait(driver, 60)
.until(ExpectedConditions.elementToBeClickable(menWorkerGroupManage));
((JavascriptExecutor)driver).executeScript("arguments[0].click();", menWorkerGroupManageElement);
return tab.cast(new WorkerGroupPage(driver));
}
if (tab == QueuePage.class) {
menuQueueManage().click();
return tab.cast(new QueuePage(driver));
}
throw new UnsupportedOperationException("Unknown tab: " + tab.getName());
}
```
![SecurityPage](../../../img/e2e-test/SecurityPage.png)
@ -146,14 +146,14 @@ The following is an example of a tenant management test. As explained earlier, w
The browser is loaded using the RemoteWebDriver provided with Selenium. Before each test case is started there is some preparation work that needs to be done. For example: logging in the user, jumping to the corresponding page (depending on the specific test case).
```java
@BeforeAll
public static void setup() {
new LoginPage(browser)
.login("admin", "dolphinscheduler123")
.goToNav(SecurityPage.class)
.goToTab(TenantPage.class)
;
}
@BeforeAll
public static void setup() {
new LoginPage(browser)
.login("admin", "dolphinscheduler123")
.goToNav(SecurityPage.class)
.goToTab(TenantPage.class)
;
}
```
When the preparation is complete, it is time for the formal test case writing. We use a form of @Order() annotation for modularity, to confirm the order of the tests. After the tests have been run, assertions are used to determine if the tests were successful, and if the assertion returns true, the tenant creation was successful. The following code can be used as a reference:
@ -176,7 +176,7 @@ The rest are similar cases and can be understood by referring to the specific so
https://github.com/apache/dolphinscheduler/tree/dev/dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/java/org/apache/dolphinscheduler/e2e/cases
## III. Supplements
## III. Supplements
When running E2E tests locally, First, you need to start the local service, you can refer to this page:
[development-environment-setup](./development-environment-setup.md)

View File

@ -1,6 +1,7 @@
# Front-end development documentation
### Technical selection
```
Vue mvvm framework
@ -17,10 +18,16 @@ Lodash high performance JavaScript utility library
### Development environment
- #### Node installation
-
#### Node installation
Node package download (note version v12.20.2) `https://nodejs.org/download/release/v12.20.2/`
- #### Front-end project construction
-
#### Front-end project construction
Use the command line mode `cd` enter the `dolphinscheduler-ui` project directory and execute `npm install` to pull the project dependency package.
> If `npm install` is very slow, you can set the taobao mirror
@ -36,13 +43,16 @@ npm config set registry http://registry.npm.taobao.org/
API_BASE = http://127.0.0.1:12345
```
> ##### ! ! ! Special attention here. If the project reports a "node-sass error" error while pulling the dependency package, execute the following command again after execution.
##### ! ! ! Special attention here. If the project reports a "node-sass error" error while pulling the dependency package, execute the following command again after execution.
```bash
npm install node-sass --unsafe-perm #Install node-sass dependency separately
```
- #### Development environment operation
-
#### Development environment operation
- `npm start` project development environment (after startup address http://localhost:8888)
#### Front-end project release
@ -140,6 +150,7 @@ Public module and utill `src/js/module`
Home => `http://localhost:8888/#/home`
Project Management => `http://localhost:8888/#/projects/list`
```
| Project Home
| Workflow
@ -149,6 +160,7 @@ Project Management => `http://localhost:8888/#/projects/list`
```
Resource Management => `http://localhost:8888/#/resource/file`
```
| File Management
| udf Management
@ -159,6 +171,7 @@ Resource Management => `http://localhost:8888/#/resource/file`
Data Source Management => `http://localhost:8888/#/datasource/list`
Security Center => `http://localhost:8888/#/security/tenant`
```
| Tenant Management
| User Management
@ -174,16 +187,19 @@ User Center => `http://localhost:8888/#/user/account`
The project `src/js/conf/home` is divided into
`pages` => route to page directory
```
The page file corresponding to the routing address
The page file corresponding to the routing address
```
`router` => route management
```
vue router, the entry file index.js in each page will be registered. Specific operations: https://router.vuejs.org/zh/
```
`store` => status management
```
The page corresponding to each route has a state management file divided into:
@ -201,9 +217,13 @@ Specific actionhttps://vuex.vuejs.org/zh/
```
## specification
## Vue specification
##### 1.Component name
The component is named multiple words and is connected with a wire (-) to avoid conflicts with HTML tags and a clearer structure.
```
// positive example
export default {
@ -212,7 +232,9 @@ export default {
```
##### 2.Component files
The internal common component of the `src/js/module/components` project writes the folder name with the same name as the file name. The subcomponents and util tools that are split inside the common component are placed in the internal `_source` folder of the component.
```
└── components
├── header
@ -228,6 +250,7 @@ The internal common component of the `src/js/module/components` project writes t
```
##### 3.Prop
When you define Prop, you should always name it in camel format (camelCase) and use the connection line (-) when assigning values to the parent component.
This follows the characteristics of each language, because it is case-insensitive in HTML tags, and the use of links is more friendly; in JavaScript, the more natural is the hump name.
@ -270,7 +293,9 @@ props: {
```
##### 4.v-for
When performing v-for traversal, you should always bring a key value to make rendering more efficient when updating the DOM.
```
<ul>
<li v-for="item in list" :key="item.id">
@ -280,6 +305,7 @@ When performing v-for traversal, you should always bring a key value to make ren
```
v-for should be avoided on the same element as v-if (`for example: <li>`) because v-for has a higher priority than v-if. To avoid invalid calculations and rendering, you should try to use v-if Put it on top of the container's parent element.
```
<ul v-if="showList">
<li v-for="item in list" :key="item.id">
@ -289,7 +315,9 @@ v-for should be avoided on the same element as v-if (`for example: <li>`) becaus
```
##### 5.v-if / v-else-if / v-else
If the elements in the same set of v-if logic control are logically identical, Vue reuses the same part for more efficient element switching, `such as: value`. In order to avoid the unreasonable effect of multiplexing, you should add key to the same element for identification.
```
<div v-if="hasData" key="mazey-data">
<span>{{ mazeyData }}</span>
@ -300,12 +328,15 @@ If the elements in the same set of v-if logic control are logically identical, V
```
##### 6.Instruction abbreviation
In order to unify the specification, the instruction abbreviation is always used. Using `v-bind`, `v-on` is not bad. Here is only a unified specification.
```
<input :value="mazeyUser" @click="verifyUser">
```
##### 7.Top-level element order of single file components
Styles are packaged in a file, all the styles defined in a single vue file, the same name in other files will also take effect. All will have a top class name before creating a component.
Note: The sass plugin has been added to the project, and the sas syntax can be written directly in a single vue file.
For uniformity and ease of reading, they should be placed in the order of `<template>`、`<script>`、`<style>`.
@ -357,25 +388,31 @@ For uniformity and ease of reading, they should be placed in the order of `<tem
## JavaScript specification
##### 1.var / let / const
It is recommended to no longer use var, but use let / const, prefer const. The use of any variable must be declared in advance, except that the function defined by function can be placed anywhere.
##### 2.quotes
```
const foo = 'after division'
const bar = `${foo}ront-end engineer`
```
##### 3.function
Anonymous functions use the arrow function uniformly. When multiple parameters/return values are used, the object's structure assignment is used first.
```
function getPersonInfo ({name, sex}) {
// ...
return {name, gender}
}
```
The function name is uniformly named with a camel name. The beginning of the capital letter is a constructor. The lowercase letters start with ordinary functions, and the new operator should not be used to operate ordinary functions.
##### 4.object
```
const foo = {a: 0, b: 1}
const bar = JSON.parse(JSON.stringify(foo))
@ -393,7 +430,9 @@ for (let [key, value] of myMap.entries()) {
```
##### 5.module
Unified management of project modules using import / export.
```
// lib.js
export default {}
@ -411,13 +450,16 @@ If the module has only one output value, use `export default`otherwise no.
##### 1.Label
Do not write the type attribute when referencing external CSS or JavaScript. The HTML5 default type is the text/css and text/javascript properties, so there is no need to specify them.
```
<link rel="stylesheet" href="//www.test.com/css/test.css">
<script src="//www.test.com/js/test.js"></script>
```
##### 2.Naming
The naming of Class and ID should be semantic, and you can see what you are doing by looking at the name; multiple words are connected by a link.
```
// positive example
.test-header{
@ -426,6 +468,7 @@ The naming of Class and ID should be semantic, and you can see what you are doin
```
##### 3.Attribute abbreviation
CSS attributes use abbreviations as much as possible to improve the efficiency and ease of understanding of the code.
```
@ -439,6 +482,7 @@ border: 1px solid #ccc;
```
##### 4.Document type
The HTML5 standard should always be used.
```
@ -446,7 +490,9 @@ The HTML5 standard should always be used.
```
##### 5.Notes
A block comment should be written to a module file.
```
/**
* @module mazey/api
@ -458,6 +504,7 @@ A block comment should be written to a module file.
## interface
##### All interfaces are returned as Promise
Note that non-zero is wrong for catching catch
```
@ -477,6 +524,7 @@ test.then(res => {
```
Normal return
```
{
code:0,
@ -486,6 +534,7 @@ Normal return
```
Error return
```
{
code:10000,
@ -493,8 +542,10 @@ Error return
msg:'failed'
}
```
If the interface is a post request, the Content-Type defaults to application/x-www-form-urlencoded; if the Content-Type is changed to application/json,
Interface parameter transfer needs to be changed to the following way
```
io.post('url', payload, null, null, { emulateJSON: false } res => {
resolve(res)
@ -524,6 +575,7 @@ User Center Related Interfaces `src/js/conf/home/store/user/actions.js`
(1) First place the icon icon of the node in the `src/js/conf/home/pages/dag/img `folder, and note the English name of the node defined by the `toolbar_${in the background. For example: SHELL}.png`
(2) Find the `tasksType` object in `src/js/conf/home/pages/dag/_source/config.js` and add it to it.
```
'DEPENDENT': { // The background definition node type English name is used as the key value
desc: 'DEPENDENT', // tooltip desc
@ -532,6 +584,7 @@ User Center Related Interfaces `src/js/conf/home/store/user/actions.js`
```
(3) Add a `${node type (lowercase)}`.vue file in `src/js/conf/home/pages/dag/_source/formModel/tasks`. The contents of the components related to the current node are written here. Must belong to a node component must have a function _verification () After the verification is successful, the relevant data of the current component is thrown to the parent component.
```
/**
* Verification
@ -566,6 +619,7 @@ User Center Related Interfaces `src/js/conf/home/store/user/actions.js`
(4) Common components used inside the node component are under` _source`, and `commcon.js` is used to configure public data.
##### 2.Increase the status type
(1) Find the `tasksState` object in `src/js/conf/home/pages/dag/_source/config.js` and add it to it.
```
@ -579,7 +633,9 @@ User Center Related Interfaces `src/js/conf/home/store/user/actions.js`
```
##### 3.Add the action bar tool
(1) Find the `toolOper` object in `src/js/conf/home/pages/dag/_source/config.js` and add it to it.
```
{
code: 'pointer', // tool identifier
@ -599,13 +655,12 @@ User Center Related Interfaces `src/js/conf/home/store/user/actions.js`
`util.js` => belongs to the `plugIn` tool class
The operation is handled in the `src/js/conf/home/pages/dag/_source/dag.js` => `toolbarEvent` event.
##### 3.Add a routing page
(1) First add a routing address`src/js/conf/home/router/index.js` in route management
```
routing address{
path: '/test', // routing address
@ -619,12 +674,12 @@ routing address{
(2)Create a `test` folder in `src/js/conf/home/pages` and create an `index.vue `entry file in the folder.
This will give you direct access to`http://localhost:8888/#/test`
This will give you direct access to`http://localhost:8888/#/test`
##### 4.Increase the preset mailbox
Find the `src/lib/localData/email.js` startup and timed email address input to automatically pull down the match.
```
export default ["test@analysys.com.cn","test1@analysys.com.cn","test3@analysys.com.cn"]
```

View File

@ -21,8 +21,9 @@ Some quick tips when using email:
- Tagging the subject line of your email will help you get a faster response, e.g. [api-server]: How to get open api interface?
- Tags may help identify a topic by:
- Component: MasterServer,ApiServer,WorkerServer,AlertServer, etc
- Level: Beginner, Intermediate, Advanced
- Scenario: Debug, How-to
- For error logs or long code examples, please use [GitHub gist](https://gist.github.com/) and include only a few lines of the pertinent code / log within the email.

View File

@ -20,7 +20,6 @@ Moreover, when we intend to refer a new software ( not limited to 3rd party jar,
* [COMMUNITY-LED DEVELOPMENT "THE APACHE WAY"](https://apache.org/dev/licensing-howto.html)
For example, we should contain the NOTICE file (every open-source project has NOTICE file, generally under root directory) of ZooKeeper in our project when we are using ZooKeeper. As the Apache explains, "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work.
We are not going to dive into every 3rd party open-source license policy, you may look up them if interested.
@ -40,3 +39,4 @@ We need to follow the following steps when we need to add new jars or external r
* [COMMUNITY-LED DEVELOPMENT "THE APACHE WAY"](https://apache.org/dev/licensing-howto.html)
* [ASF 3RD PARTY LICENSE POLICY](https://apache.org/legal/resolved.html)

View File

@ -3,66 +3,67 @@
The following Code of Conduct is based on full compliance with the [Apache Software Foundation Code of Conduct](https://www.apache.org/foundation/policies/conduct.html).
## Development philosophy
- **Consistent** code style, naming, and usage are consistent.
- **Easy to read** code is obvious, easy to read and understand, when debugging one knows the intent of the code.
- **Neat** agree with the concepts of《Refactoring》and《Code Cleanliness》and pursue clean and elegant code.
- **Abstract** hierarchy is clear and the concepts are refined and reasonable. Keep methods, classes, packages, and modules at the same level of abstraction.
- **Heart** Maintain a sense of responsibility and continue to be carved in the spirit of artisans.
- **Consistent** code style, naming, and usage are consistent.
- **Easy to read** code is obvious, easy to read and understand, when debugging one knows the intent of the code.
- **Neat** agree with the concepts of《Refactoring》and《Code Cleanliness》and pursue clean and elegant code.
- **Abstract** hierarchy is clear and the concepts are refined and reasonable. Keep methods, classes, packages, and modules at the same level of abstraction.
- **Heart** Maintain a sense of responsibility and continue to be carved in the spirit of artisans.
## Development specifications
- Executing `mvn -U clean package -Prelease` can compile and test through all test cases.
- The test coverage tool checks for no less than dev branch coverage.
- In the root directory, use Checkstyle to check your code for special reasons for violating validation rules. The template location is located at ds_check_style.xml.
- Follow the coding specifications.
- Executing `mvn -U clean package -Prelease` can compile and test through all test cases.
- The test coverage tool checks for no less than dev branch coverage.
- In the root directory, use Checkstyle to check your code for special reasons for violating validation rules. The template location is located at ds_check_style.xml.
- Follow the coding specifications.
## Coding specifications
- Use linux line breaks.
- Indentation (including empty lines) is consistent with the last line.
- An empty line is required between the class declaration and the following variable or method.
- There should be no meaningless empty lines.
- Classes, methods, and variables should be named as the name implies and abbreviations should be avoided.
- Return value variables are named after `result`; `each` is used in loops to name loop variables; and `entry` is used in map instead of `each`.
- The cached exception is called `e`; Catch the exception and do nothing, and the exception is named `ignored`.
- Configuration Files are named in camelCase, and file names are lowercase with uppercase initial/starting letter.
- Code that requires comment interpretation should be as small as possible and interpreted by method name.
- `equals` and `==` In a conditional expression, the constant is left, the variable is on the right, and in the expression greater than less than condition, the variable is left and the constant is right.
- In addition to the abstract classes used for inheritance, try to design the class as `final`.
- Nested loops are as much a method as possible.
- The order in which member variables are defined and the order in which parameters are passed is consistent across classes and methods.
- Priority is given to the use of guard statements.
- Classes and methods have minimal access control.
- The private method used by the method should follow the method, and if there are multiple private methods, the writing private method should appear in the same order as the private method in the original method.
- Method entry and return values are not allowed to be `null`.
- The return and assignment statements of if else are preferred with the tri-objective operator.
- Priority is given to `LinkedList` and only use `ArrayList` if you need to get element values in the collection through the index.
- Collection types such as `ArrayList``HashMap` that may produce expansion must specify the initial size of the collection to avoid expansion.
- Logs and notes are always in English.
- Comments can only contain `javadoc`, `todo` and `fixme`.
- Exposed classes and methods must have javadoc, other classes and methods and methods that override the parent class do not require javadoc.
- Use linux line breaks.
- Indentation (including empty lines) is consistent with the last line.
- An empty line is required between the class declaration and the following variable or method.
- There should be no meaningless empty lines.
- Classes, methods, and variables should be named as the name implies and abbreviations should be avoided.
- Return value variables are named after `result`; `each` is used in loops to name loop variables; and `entry` is used in map instead of `each`.
- The cached exception is called `e`; Catch the exception and do nothing, and the exception is named `ignored`.
- Configuration Files are named in camelCase, and file names are lowercase with uppercase initial/starting letter.
- Code that requires comment interpretation should be as small as possible and interpreted by method name.
- `equals` and `==` In a conditional expression, the constant is left, the variable is on the right, and in the expression greater than less than condition, the variable is left and the constant is right.
- In addition to the abstract classes used for inheritance, try to design the class as `final`.
- Nested loops are as much a method as possible.
- The order in which member variables are defined and the order in which parameters are passed is consistent across classes and methods.
- Priority is given to the use of guard statements.
- Classes and methods have minimal access control.
- The private method used by the method should follow the method, and if there are multiple private methods, the writing private method should appear in the same order as the private method in the original method.
- Method entry and return values are not allowed to be `null`.
- The return and assignment statements of if else are preferred with the tri-objective operator.
- Priority is given to `LinkedList` and only use `ArrayList` if you need to get element values in the collection through the index.
- Collection types such as `ArrayList``HashMap` that may produce expansion must specify the initial size of the collection to avoid expansion.
- Logs and notes are always in English.
- Comments can only contain `javadoc`, `todo` and `fixme`.
- Exposed classes and methods must have javadoc, other classes and methods and methods that override the parent class do not require javadoc.
## Unit test specifications
- Test code and production code are subject to the same code specifications.
- Unit tests are subject to AIR (Automatic, Independent, Repeatable) Design concept.
- Automatic: Unit tests should be fully automated, not interactive. Manual checking of output results is prohibited, `System.out`, `log`, etc. are not allowed, and must be verified with assertions.
- Independent: It is prohibited to call each other between unit test cases and to rely on the order of execution. Each unit test can be run independently.
- Repeatable: Unit tests cannot be affected by the external environment and can be repeated.
- Unit tests are subject to BCDEBorder, Correct, Design, Error) Design principles.
- Border (Boundary value test): The expected results are obtained by entering the boundaries of loop boundaries, special values, data order, etc.
- Correct (Correctness test): The expected results are obtained with the correct input.
- Design (Rationality Design): Design high-quality unit tests in combination with production code design.
- Error (Fault tolerance test): The expected results are obtained through incorrect input such as illegal data, abnormal flow, etc.
- If there is no special reason, the test needs to be fully covered.
- Each test case needs to be accurately asserted.
- Prepare the environment for code separation from the test code.
- Only jUnit `Assert`hamcrest `CoreMatchers`Mockito Correlation can use static import.
- Single-data assertions should use `assertTrue``assertFalse``assertNull` and `assertNotNull`.
- Multi-data assertions should use `assertThat`.
- Accurate assertion, try not to use `not``containsString` assertion.
- The true value of the test case should be named actualXXX, and the expected value should be named expectedXXX.
- Classes and Methods with `@Test` labels do not require javadoc.
- Test code and production code are subject to the same code specifications.
- Unit tests are subject to AIR (Automatic, Independent, Repeatable) Design concept.
- Automatic: Unit tests should be fully automated, not interactive. Manual checking of output results is prohibited, `System.out`, `log`, etc. are not allowed, and must be verified with assertions.
- Independent: It is prohibited to call each other between unit test cases and to rely on the order of execution. Each unit test can be run independently.
- Repeatable: Unit tests cannot be affected by the external environment and can be repeated.
- Unit tests are subject to BCDEBorder, Correct, Design, Error) Design principles.
- Border (Boundary value test): The expected results are obtained by entering the boundaries of loop boundaries, special values, data order, etc.
- Correct (Correctness test): The expected results are obtained with the correct input.
- Design (Rationality Design): Design high-quality unit tests in combination with production code design.
- Error (Fault tolerance test): The expected results are obtained through incorrect input such as illegal data, abnormal flow, etc.
- If there is no special reason, the test needs to be fully covered.
- Each test case needs to be accurately asserted.
- Prepare the environment for code separation from the test code.
- Only jUnit `Assert`hamcrest `CoreMatchers`Mockito Correlation can use static import.
- Single-data assertions should use `assertTrue``assertFalse``assertNull` and `assertNotNull`.
- Multi-data assertions should use `assertThat`.
- Accurate assertion, try not to use `not``containsString` assertion.
- The true value of the test case should be named actualXXX, and the expected value should be named expectedXXX.
- Classes and Methods with `@Test` labels do not require javadoc.
- Public specifications.
- Each line is no longer than `200` in length, ensuring that each line is semantically complete for easy understanding.
- Public specifications.
- Each line is no longer than `200` in length, ensuring that each line is semantically complete for easy understanding.

View File

@ -13,7 +13,7 @@ We encourage any form of participation in the community that will eventually bec
* Help promote DolphinScheduler, participate in technical conferences or meetup, sharing and more.
Welcome to the contributing team and join open source starting with submitting your first PR.
- For example, add code comments or find "easy to fix" tags or some very simple issue (misspellings, etc.) and so on, first familiarize yourself with the submission process through the first simple PR.
- For example, add code comments or find "easy to fix" tags or some very simple issue (misspellings, etc.) and so on, first familiarize yourself with the submission process through the first simple PR.
Note: Contributions are not limited to PR Only, but contribute to the development of the project.

View File

@ -2,7 +2,7 @@
Good documentation is critical for any type of software. Any contribution that can improve the DolphinScheduler documentation is welcome.
### Get the document project
### Get the document project
Documentation for the DolphinScheduler project is maintained in a separate [git repository](https://github.com/apache/dolphinscheduler-website).
@ -52,8 +52,8 @@ Now you can run and build the website in your local environment.
2. Simply push the changed files, for example:
* `*.md`
* `blog.js or docs.js or site.js`
* `*.md`
* `blog.js or docs.js or site.js`
3. Submit the Pull Request to the **master** branch.

View File

@ -1,6 +1,7 @@
# Issue Notice
## Preface
Issues function is used to track various Features, Bugs, Functions, etc. The project maintainer can organize the tasks to be completed through issues.
Issue is an important step in drawing out a feature or bug,
@ -129,8 +130,8 @@ The main purpose of this is to avoid wasting time caused by different opinions o
- How to deal with the user who raises an issue does not know the module corresponding to the issue.
It is true that most users when raising issue do not know which module the issue belongs to.
In fact, this is very common in many open source communities. In this case, the committer / contributor actually knows the module affected by the issue.
If the issue is really valuable after being approved by committer and contributor, then the committer can modify the issue title according to the specific module involved in the issue,
or leave a message to the user who raises the issue to modify it into the corresponding title.
It is true that most users when raising issue do not know which module the issue belongs to.
In fact, this is very common in many open source communities. In this case, the committer / contributor actually knows the module affected by the issue.
If the issue is really valuable after being approved by committer and contributor, then the committer can modify the issue title according to the specific module involved in the issue,
or leave a message to the user who raises the issue to modify it into the corresponding title.

View File

@ -1,6 +1,7 @@
# Pull Request Notice
## Preface
Pull Request is a way of software cooperation, which is a process of bringing code involving different functions into the trunk. During this process, the code can be discussed, reviewed, and modified.
In Pull Request, we try not to discuss the implementation of the code. The general implementation of the code and its logic should be determined in Issue. In the Pull Request, we only focus on the code format and code specification, so as to avoid wasting time caused by different opinions on implementation.
@ -75,3 +76,4 @@ see [Code Style](../development-environment-setup.md#code-style) for details.
the second is multiple issues have subtle differences.
In this scenario, the responsibilities of each issue can be clearly divided. The type of each issue is marked as Sub-Task, and then these sub task type issues are associated with one issue.
And each Pull Request is submitted should be associated with only one issue of a sub task.

View File

@ -27,38 +27,38 @@ go to section [review Pull Requests](#pull-requests).
Review Issues means discuss [Issues][all-issues] in GitHub and give suggestions on it. Include but are not limited to the following situations
| Situation | Reason | Label | Action |
| ------ | ------ | ------ | ------ |
| wont fix | Has been fixed in dev branch | [wontfix][label-wontfix] | Close Issue, inform creator the fixed version if it already release |
| duplicate issue | Had the same problem before | [duplicate][label-duplicate] | Close issue, inform creator the link of same issue |
| Description not clearly | Without detail reproduce step | [need more information][label-need-more-information] | Inform creator add more description |
| Situation | Reason | Label | Action |
|-------------------------|-------------------------------|------------------------------------------------------|---------------------------------------------------------------------|
| wont fix | Has been fixed in dev branch | [wontfix][label-wontfix] | Close Issue, inform creator the fixed version if it already release |
| duplicate issue | Had the same problem before | [duplicate][label-duplicate] | Close issue, inform creator the link of same issue |
| Description not clearly | Without detail reproduce step | [need more information][label-need-more-information] | Inform creator add more description |
In addition give suggestion, add label for issue is also important during review. The labeled issues can be retrieved
better, which convenient for further processing. An issue can with more than one label. Common issue categories are:
| Label | Meaning |
| ------ | ------ |
| [UI][label-UI] | UI and front-end related |
| [security][label-security] | Security Issue |
| [user experience][label-user-experience] | User experience Issue |
| [development][label-development] | Development Issue |
| [Python][label-Python] | Python Issue |
| [plug-in][label-plug-in] | Plug-in Issue |
| [document][label-document] | Document Issue |
| [docker][label-docker] | Docker Issue |
| [need verify][label-need-verify] | Need verify Issue |
| [e2e][label-e2e] | E2E Issue |
| [win-os][label-win-os] | windows operating system Issue |
| [suggestion][label-suggestion] | Give suggestion to us |
| Label | Meaning |
|------------------------------------------|--------------------------------|
| [UI][label-UI] | UI and front-end related |
| [security][label-security] | Security Issue |
| [user experience][label-user-experience] | User experience Issue |
| [development][label-development] | Development Issue |
| [Python][label-Python] | Python Issue |
| [plug-in][label-plug-in] | Plug-in Issue |
| [document][label-document] | Document Issue |
| [docker][label-docker] | Docker Issue |
| [need verify][label-need-verify] | Need verify Issue |
| [e2e][label-e2e] | E2E Issue |
| [win-os][label-win-os] | windows operating system Issue |
| [suggestion][label-suggestion] | Give suggestion to us |
Beside classification, label could also set the priority of Issues. The higher the priority, the more attention pay
in the community, the easier it is to be fixed or implemented. The priority label are as follows
| Label | priority |
| ------ | ------ |
| [priority:high][label-priority-high] | High priority |
| Label | priority |
|------------------------------------------|-----------------|
| [priority:high][label-priority-high] | High priority |
| [priority:middle][label-priority-middle] | Middle priority |
| [priority:low][label-priority-low] | Low priority |
| [priority:low][label-priority-low] | Low priority |
All the labels above in common label. For all labels in this project you could see in [full label list][label-all-list]
@ -74,12 +74,12 @@ Before reading following content, please make sure you have labeled the Issue.
When an Issue need to create Pull Requests, you could also labeled it from below.
| Label | Mean |
| ------ | ------ |
| [Chore][label-Chore] | Chore for project |
| [Good first issue][label-good-first-issue] | Good first issue for new contributor |
| [easy to fix][label-easy-to-fix] | Easy to fix, harder than `Good first issue` |
| [help wanted][label-help-wanted] | Help wanted |
| Label | Mean |
|--------------------------------------------|---------------------------------------------|
| [Chore][label-Chore] | Chore for project |
| [Good first issue][label-good-first-issue] | Good first issue for new contributor |
| [easy to fix][label-easy-to-fix] | Easy to fix, harder than `Good first issue` |
| [help wanted][label-help-wanted] | Help wanted |
> Note: Only members have permission to add or delete label. When you need to add or remove lebals but are not member,
> you can `@` members to do that. But as long as you have a GitHub account, you can comment on issues and give suggestions.
@ -90,14 +90,14 @@ When an Issue need to create Pull Requests, you could also labeled it from below
<!-- markdown-link-check-disable -->
Review Pull mean discussing in [Pull Requests][all-PRs] in GitHub and giving suggestions to it. DolphinScheduler's
Pull Requests reviewing are the same as [GitHub's reviewing changes in pull requests][gh-review-pr]. You can give your
suggestions in Pull Requests
suggestions in Pull Reque-->
* When you think the Pull Request is OK to be merged, you can agree to the Pull Request according to the "Approve" process
in [GitHub's reviewing changes in pull requests][gh-review-pr].
* When you think Pull Request needs to be changed, you can comment it according to the "Comment" process in
[GitHub's reviewing changes in pull requests][gh-review-pr]. And when you think issues that must be fixed before they
merged, please follow "Request changes" in [GitHub's reviewing changes in pull requests][gh-review-pr] to ask contributors
modify it.
<!-- markdown-link-check-enable -->
Labeled Pull Requests is an important part. Reasonable classification can save a lot of time for reviewers. The good news
@ -107,11 +107,11 @@ and [priority:high][label-priority-high].
Pull Requests have some unique labels of it own
| Label | Mean |
| ------ | ------ |
| [miss document][label-miss-document] | Pull Requests miss document, and should be add |
| [first time contributor][label-first-time-contributor] | Pull Requests submit by first time contributor |
| [don't merge][label-do-not-merge] | Pull Requests have some problem and should not be merged |
| Label | Mean |
|--------------------------------------------------------|----------------------------------------------------------|
| [miss document][label-miss-document] | Pull Requests miss document, and should be add |
| [first time contributor][label-first-time-contributor] | Pull Requests submit by first time contributor |
| [don't merge][label-do-not-merge] | Pull Requests have some problem and should not be merged |
> Note: Only members have permission to add or delete label. When you need to add or remove lebals but are not member,
> you can `@` members to do that. But as long as you have a GitHub account, you can comment on Pull Requests and give suggestions.
@ -151,3 +151,4 @@ Pull Requests have some unique labels of it own
[all-issues]: https://github.com/apache/dolphinscheduler/issues
[all-PRs]: https://github.com/apache/dolphinscheduler/pulls
[gh-review-pr]: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/about-pull-request-reviews

View File

@ -3,26 +3,23 @@
* First from the remote repository *https://github.com/apache/dolphinscheduler.git* fork a copy of the code into your own repository
* There are currently three branches in the remote repository:
* master normal delivery branch
After the stable release, merge the code from the stable branch into the master.
    
* dev daily development branch
Every day dev development branch, newly submitted code can pull request to this branch.
* master normal delivery branch
After the stable release, merge the code from the stable branch into the master.
    
* dev daily development branch
Every day dev development branch, newly submitted code can pull request to this branch.
* Clone your repository to your local
`git clone https://github.com/apache/dolphinscheduler.git`
`git clone https://github.com/apache/dolphinscheduler.git`
* Add remote repository address, named upstream
`git remote add upstream https://github.com/apache/dolphinscheduler.git`
`git remote add upstream https://github.com/apache/dolphinscheduler.git`
* View repository
    `git remote -v`
    `git remote -v`
>At this time, there will be two repositories: origin (your own repository) and upstream (remote repository)
> At this time, there will be two repositories: origin (your own repository) and upstream (remote repository)
* Get/Update remote repository code
    `git fetch upstream`
    `git fetch upstream`
* Synchronize remote repository code to local repository
@ -39,6 +36,7 @@ git push --set-upstream origin dev-1.0
```
* Create new branch
```
git checkout -b xxx origin/dev
```
@ -60,4 +58,3 @@ Make sure that the branch `xxx` is building successfully on the latest code of t
* Finally, congratulations, you have become an official contributor to dolphinscheduler!

View File

@ -21,3 +21,4 @@ Unsubscribe from the mailing list steps are as follows:
2. Receive confirmation email and reply. After completing step 1, you will receive a confirmation email from dev-help@dolphinscheduler.apache.org (if not received, please confirm whether the email is automatically classified as spam, promotion email, subscription email, etc.) . Then reply directly to the email, or click on the link in the email to reply quickly, the subject and content are arbitrary.
3. Receive a goodbye email. After completing the above steps, you will receive a goodbye email with the subject GOODBYE from dev@dolphinscheduler.apache.org, and you have successfully unsubscribed to the Apache DolphinScheduler mailing list, and you will not receive emails from dev@dolphinscheduler.apache.org.

View File

@ -2,25 +2,27 @@
### 1. The Benefits of Writing Unit Tests
- Unit tests help everyone to get into the details of the code and understand how it works.
- Through test cases we can find bugs and submit robust code.
- The test case is also a demo usage of the code.
- Unit tests help everyone to get into the details of the code and understand how it works.
- Through test cases we can find bugs and submit robust code.
- The test case is also a demo usage of the code.
### 2. Some design principles for unit test cases
- The steps, granularity and combination of conditions should be carefully designed.
- Pay attention to boundary conditions.
- Unit tests should be well designed as well as avoiding useless code.
- When you find a `method` is difficult to write unit test, and if you confirm that the `method` is `bad code`, then refactor it with the developer.
- The steps, granularity and combination of conditions should be carefully designed.
- Pay attention to boundary conditions.
- Unit tests should be well designed as well as avoiding useless code.
- When you find a `method` is difficult to write unit test, and if you confirm that the `method` is `bad code`, then refactor it with the developer.
<!-- markdown-link-check-disable -->
- DolphinScheduler: [mockito](http://site.mockito.org/). Here are some development guides: [mockito tutorial](http://www.baeldung.com/bdd-mockito), [mockito refcard](https://dzone.com/refcardz/mockito)
- DolphinScheduler: [mockito](http://site.mockito.org/). Here are some development guides: [mockito tutorial](http://www.baeldung.com/bdd-mockito), [mockito refcard](https://dzone.com/refcardz/mockito)
<!-- markdown-link-check-enable -->
- TDD(option): When you start writing a new feature, you can try writing test cases first.
- TDD(option): When you start writing a new feature, you can try writing test cases first.
### 3. Test coverage setpoint
- At this stage, the default value for test coverage of Delta change codes is >= 60%, the higher the better.
- We can see the test reports on this page: https://codecov.io/gh/apache/dolphinscheduler
- At this stage, the default value for test coverage of Delta change codes is >= 60%, the higher the better.
- We can see the test reports on this page: https://codecov.io/gh/apache/dolphinscheduler
## Fundamental guidelines for unit test
@ -64,13 +66,13 @@ Invalid assertions make the test itself meaningless, it has little to do with wh
There are several types of invalid assertions:
1. Different types of comparisons.
1. Different types of comparisons.
2. Determines that an object or variable with a default value is not null.
2. Determines that an object or variable with a default value is not null.
This seems meaningless. Therefore, when making the relevant judgements you should pay attention to whether it contains a default value itself.
This seems meaningless. Therefore, when making the relevant judgements you should pay attention to whether it contains a default value itself.
3. Assertions should be affirmative rather than negative if possible. Assertions should be within a range of predicted results, or exact values, whenever possible (otherwise you may end up with something that doesn't match your actual expectations but passes the assertion) unless your code only cares about whether it is empty or not.
3. Assertions should be affirmative rather than negative if possible. Assertions should be within a range of predicted results, or exact values, whenever possible (otherwise you may end up with something that doesn't match your actual expectations but passes the assertion) unless your code only cares about whether it is empty or not.
### 8. Some points to note for unit tests
@ -90,16 +92,17 @@ For example @Ignore("see #1").
The test will fail when the code in the unit test throws an exception. Therefore, there is no need to use try-catch to catch exceptions.
```java
@Test
public void testMethod() {
try {
// Some code
} catch (MyException e) {
Assert.fail(e.getMessage()); // Noncompliant
}
}
```
```java
@Test
public void testMethod() {
try {
// Some code
} catch (MyException e) {
Assert.fail(e.getMessage()); // Noncompliant
}
}
```
You should this:
```java

View File

@ -50,3 +50,4 @@ That is, the workflow instance ID and task instance ID are injected in the print
- Branch printing of logs is prohibited. The contents of the logs need to be associated with the relevant information in the log format, and printing them in separate lines will cause the contents of the logs to not match the time and other information, and cause the logs to be mixed in a large number of log environments, which will make log retrieval more difficult.
- The use of the "+" operator for splicing log content is prohibited. Use placeholders for formatting logs for printing to improve memory usage efficiency.
- When the log content includes object instances, you need to make sure to override the toString() method to prevent printing meaningless hashcode.

View File

@ -22,7 +22,7 @@ We could reuse the main command the CI run and publish our Docker images to Dock
## Publish pydolphinscheduler to PyPI
Python API need to release to PyPI for easier download and use, you can see more detail in [Python API release](https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-python/pydolphinscheduler/RELEASE.md#to-pypi)
Python API need to release to PyPI for easier download and use, you can see more detail in [Python API release](https://github.com/apache/dolphinscheduler/blob/3.1.0/dolphinscheduler-python/pydolphinscheduler/RELEASE.md#to-pypi)
to finish PyPI release.
## Get All Contributors

View File

@ -4,9 +4,9 @@
Compared with the last release, the `release-docs` of the current release needs to be updated to the latest, if there are dependencies and versions changes
- `dolphinscheduler-dist/release-docs/LICENSE`
- `dolphinscheduler-dist/release-docs/NOTICE`
- `dolphinscheduler-dist/release-docs/licenses`
- `dolphinscheduler-dist/release-docs/LICENSE`
- `dolphinscheduler-dist/release-docs/NOTICE`
- `dolphinscheduler-dist/release-docs/licenses`
## Update Version
@ -29,3 +29,4 @@ For example, to release `x.y.z`, the following updates are required:
- Add new history version
- `docs/docs/en/history-versions.md` and `docs/docs/zh/history-versions.md`: Add the new version and link for `x.y.z`
- `docs/configs/docsdev.js`: change `/dev/` to `/x.y.z/`

View File

@ -210,7 +210,7 @@ git push origin --tags
> Note1: In this step, you should use github token for password because native password no longer supported, you can see
> https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token for more
> detail about how to create token about it.
>
> Note2: After the command done, it will auto-created `release.properties` file and `*.Backup` files, their will be need
> in the following command and DO NOT DELETE THEM
@ -293,6 +293,7 @@ cd ~/ds_svn/dev/dolphinscheduler
svn add *
svn --username="${A_USERNAME}" commit -m "release ${VERSION}"
```
## Check Release
### Check sha512 hash
@ -353,14 +354,14 @@ cd ../
Decompress `apache-dolphinscheduler-<VERSION>-src.tar.gz` and `python/apache-dolphinscheduler-python-<VERSION>.tar.gz` then check the following items:
* Check whether source tarball is oversized for including nonessential files
* `LICENSE` and `NOTICE` files exist
* Correct year in `NOTICE` file
* There is only text files but no binary files
* All source files have ASF headers
* Codes can be compiled and pass the unit tests (mvn install)
* The contents of the release match with what's tagged in version control (diff -r a verify_dir tag_dir)
* Check if there is any extra files or folders, empty folders for example
* Check whether source tarball is oversized for including nonessential files
* `LICENSE` and `NOTICE` files exist
* Correct year in `NOTICE` file
* There is only text files but no binary files
* All source files have ASF headers
* Codes can be compiled and pass the unit tests (mvn install)
* The contents of the release match with what's tagged in version control (diff -r a verify_dir tag_dir)
* Check if there is any extra files or folders, empty folders for example
#### Check binary packages
@ -387,8 +388,8 @@ maybe not correct, you should filter them by yourself) and classify them and pas
### Vote procedure
1. DolphinScheduler community vote: send the vote e-mail to `dev@dolphinscheduler.apache.org`.
PMC needs to check the rightness of the version according to the document before they vote.
After at least 72 hours and with at least 3 `+1 and no -1 PMC member` votes, it can come to the next stage of the vote.
PMC needs to check the rightness of the version according to the document before they vote.
After at least 72 hours and with at least 3 `+1 and no -1 PMC member` votes, it can come to the next stage of the vote.
2. Announce the vote result: send the result vote e-mail to `dev@dolphinscheduler.apache.org`
@ -538,3 +539,4 @@ DolphinScheduler Resources:
- Mailing list: dev@dolphinscheduler.apache.org
- Documents: https://dolphinscheduler.apache.org/zh-cn/docs/<VERSION>/user_doc/about/introduction.html
```

View File

@ -280,7 +280,7 @@ A Will hive pom
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>2.1.0</version>
<version>2.3.9</version>
</dependency>
```

View File

@ -8,20 +8,21 @@ The following shows the `DingTalk` configuration example:
## Parameter Configuration
| **Parameter** | **Description** |
| --- | --- |
| Warning Type | Alert on success or failure or both. |
| WebHook | The format is: [https://oapi.dingtalk.com/robot/send?access\_token=XXXXXX](https://oapi.dingtalk.com/robot/send?access_token=XXXXXX) |
| Keyword | Custom keywords for security settings. |
| Secret | Signature of security settings |
| Msg Type | Message parse type (support txt, markdown, markdownV2, html). |
| **Parameter** | **Description** |
|----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Warning Type | Alert on success or failure or both. |
| WebHook | The format is: [https://oapi.dingtalk.com/robot/send?access\_token=XXXXXX](https://oapi.dingtalk.com/robot/send?access_token=XXXXXX) |
| Keyword | Custom keywords for security settings. |
| Secret | Signature of security settings |
| Msg Type | Message parse type (support txt, markdown, markdownV2, html). |
| At User Mobile | When a custom bot sends a message, you can specify the "@person list" by their mobile phone number. When the selected people in the "@people list" receive the message, there will be a `@` message reminder. `No disturb` mode always receives reminders, and "someone @ you" appears in the message. The "At User Mobile" represents mobile phone number of the "@person" |
| At User Ids | The user ID by "@person" |
| Proxy | The proxy address of the proxy server. |
| Port | The proxy port of Proxy-Server. |
| User | Authentication(Username) for the proxy server. |
| Password | Authentication(Password) for the proxy server. |
| At User Ids | The user ID by "@person" |
| Proxy | The proxy address of the proxy server. |
| Port | The proxy port of Proxy-Server. |
| User | Authentication(Username) for the proxy server. |
| Password | Authentication(Password) for the proxy server. |
## Reference
- [DingTalk Custom Robot Access Development Documentation](https://open.dingtalk.com/document/robots/custom-robot-access)

View File

@ -1,4 +1,5 @@
# Email
If you need to use `Email` for alerting, create an alert instance in the alert instance management and select the Email plugin.
The following shows the `Email` configuration example:

View File

@ -7,14 +7,14 @@ The following is the `WebexTeams` configuration example:
## Parameter Configuration
| **Parameter** | **Description** |
| --- | --- |
| botAccessToken | The access token of robot. |
| roomID | The ID of the room that receives message (only support one room ID). |
| toPersonId | The person ID of the recipient when sending a private 1:1 message. |
| toPersonEmail | The email address of the recipient when sending a private 1:1 message. |
| **Parameter** | **Description** |
|-----------------|-------------------------------------------------------------------------------------------------------------------------|
| botAccessToken | The access token of robot. |
| roomID | The ID of the room that receives message (only support one room ID). |
| toPersonId | The person ID of the recipient when sending a private 1:1 message. |
| toPersonEmail | The email address of the recipient when sending a private 1:1 message. |
| atSomeoneInRoom | If the message destination is room, the emails of the person being @, use `,` (eng commas) to separate multiple emails. |
| destination |The destination of the message (one message only support one destination). |
| destination | The destination of the message (one message only support one destination). |
## Create Bot
@ -59,3 +59,4 @@ The `Room ID` we can acquire it from the `id` of creating a new group chat room
- [WebexTeams Application Bot Guide](https://developer.webex.com/docs/bots)
- [WebexTeams Message Guide](https://developer.webex.com/docs/api/v1/messages/create-a-message)

View File

@ -40,7 +40,6 @@ The following is the `query userId` API example:
APP: https://work.weixin.qq.com/api/doc/90000/90135/90236
### Group Chat
The Group Chat send type means to notify the alert results via group chat created by Enterprise WeChat API, sending messages to all members of the group and specified users are not supported.
@ -69,3 +68,4 @@ The following is the `create new group chat` API and `query userId` API example:
## Reference
- Group Chathttps://work.weixin.qq.com/api/doc/90000/90135/90248

View File

@ -10,6 +10,7 @@ The following shows the `Feishu` configuration example:
## Parameter Configuration
* Webhook
> Copy the robot webhook URL shown below:
![alert-feishu-webhook](../../../../img/new_ui/dev/alert/alert_feishu_webhook.png)

View File

@ -4,13 +4,13 @@ If you need to use `Http script` for alerting, create an alert instance in the a
## Parameter Configuration
| **Parameter** | **Description** |
| --- | --- |
| URL | The `Http` request URL needs to contain protocol, host, path and parameters if the method is `GET`. |
| Request Type | Select the request type from `POST` or `GET`. |
| Headers | The headers of the `Http` request in JSON format. |
| Body | The request body of the `Http` request in JSON format, when using `POST` method to alert. |
| Content Field | The field name to place the alert information. |
| **Parameter** | **Description** |
|---------------|-----------------------------------------------------------------------------------------------------|
| URL | The `Http` request URL needs to contain protocol, host, path and parameters if the method is `GET`. |
| Request Type | Select the request type from `POST` or `GET`. |
| Headers | The headers of the `Http` request in JSON format. |
| Body | The request body of the `Http` request in JSON format, when using `POST` method to alert. |
| Content Field | The field name to place the alert information. |
## Send Type

View File

@ -7,11 +7,11 @@ The following shows the `Script` configuration example:
## Parameter Configuration
| **Parameter** | **Description** |
| --- | --- |
| User Params | User defined parameters will pass to the script. |
| Script Path |The file location path in the server. |
| Type | Support `Shell` script. |
| **Parameter** | **Description** |
|---------------|--------------------------------------------------|
| User Params | User defined parameters will pass to the script. |
| Script Path | The file location path in the server. |
| Type | Support `Shell` script. |
### Note

View File

@ -7,17 +7,17 @@ The following shows the `Telegram` configuration example:
## Parameter Configuration
| **Parameter** | **Description** |
| --- | --- |
| WebHook | The WebHook of Telegram when use robot to send message. |
| botToken | The access token of robot. |
| chatId | Sub Telegram Channel. |
| parseMode | Message parse type (support txt, markdown, markdownV2, html). |
| EnableProxy | Enable proxy sever. |
| Proxy | The proxy address of the proxy server. |
| Port | The proxy port of proxy server. |
| User | Authentication(Username) for the proxy server. |
| Password | Authentication(Password) for the proxy server. |
| **Parameter** | **Description** |
|---------------|---------------------------------------------------------------|
| WebHook | The WebHook of Telegram when use robot to send message. |
| botToken | The access token of robot. |
| chatId | Sub Telegram Channel. |
| parseMode | Message parse type (support txt, markdown, markdownV2, html). |
| EnableProxy | Enable proxy sever. |
| Proxy | The proxy address of the proxy server. |
| Port | The proxy port of proxy server. |
| User | Authentication(Username) for the proxy server. |
| Password | Authentication(Password) for the proxy server. |
### NOTE
@ -35,3 +35,4 @@ The webhook needs to be able to receive and use the same JSON body of HTTP POST
- [Telegram Application Bot Guide](https://core.telegram.org/bots)
- [Telegram Bots Api](https://core.telegram.org/bots/api)
- [Telegram SendMessage Api](https://core.telegram.org/bots/api#sendmessage)

View File

@ -1,4 +1,5 @@
# Data Quality
## Introduction
The data quality task is used to check the data accuracy during the integration and processing of data. Data quality tasks in this release include single-table checking, single-table custom SQL checking, multi-table accuracy, and two-table value comparisons. The running environment of the data quality task is Spark 2.4.0, and other versions have not been verified, and users can verify by themselves.
@ -9,7 +10,7 @@ The execution logic of the data quality task is as follows:
- When running a task, `Master` will parse `TaskParam`, encapsulate the parameters required by `DataQualityTask` and send it to `Worker`.
- Worker runs the data quality task. After the data quality task finishes running, it writes the statistical results to the specified storage engine.
- The current data quality task result is stored in the `t_ds_dq_execute_result` table of `dolphinscheduler`
`Worker` sends the task result to `Master`, after `Master` receives `TaskResponse`, it will judge whether the task type is `DataQualityTask`, if so, it will read the corresponding result from `t_ds_dq_execute_result` according to `taskInstanceId`, and then The result is judged according to the check mode, operator and threshold configured by the user.
`Worker` sends the task result to `Master`, after `Master` receives `TaskResponse`, it will judge whether the task type is `DataQualityTask`, if so, it will read the corresponding result from `t_ds_dq_execute_result` according to `taskInstanceId`, and then The result is judged according to the check mode, operator and threshold configured by the user.
- If the result is a failure, the corresponding operation, alarm or interruption will be performed according to the failure policy configured by the user.
- Add config : `<server-name>/conf/common.properties`
@ -27,13 +28,13 @@ data-quality.jar.name=dolphinscheduler-data-quality-dev-SNAPSHOT.jar
## Detailed Inspection Logic
| **Parameter** | **Description** |
| ----- | ---- |
| CheckMethod | [CheckFormula][Operator][Threshold], if the result is true, it indicates that the data does not meet expectations, and the failure strategy is executed. |
| CheckFormula | <ul><li>Expected-Actual</li><li>Actual-Expected</li><li>(Actual/Expected)x100%</li><li>(Expected-Actual)/Expected x100%</li></ul> |
| Operator | =, >, >=, <, <=, != |
| **Parameter** | **Description** |
|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| CheckMethod | [CheckFormula][Operator][Threshold], if the result is true, it indicates that the data does not meet expectations, and the failure strategy is executed. |
| CheckFormula | <ul><li>Expected-Actual</li><li>Actual-Expected</li><li>(Actual/Expected)x100%</li><li>(Expected-Actual)/Expected x100%</li></ul> |
| Operator | =, >, >=, <, <=, != |
| ExpectedValue | <ul><li>FixValue</li><li>DailyAvg</li><li>WeeklyAvg</li><li>MonthlyAvg</li><li>Last7DayAvg</li><li>Last30DayAvg</li><li>SrcTableTotalRows</li><li>TargetTableTotalRows</li></ul> |
| Example |<ul><li>CheckFormulaExpected-Actual</li><li>Operator></li><li>Threshold0</li><li>ExpectedValueFixValue=9</li></ul>
| Example | <ul><li>CheckFormulaExpected-Actual</li><li>Operator></li><li>Threshold0</li><li>ExpectedValueFixValue=9</li></ul> |
In the example, assuming that the actual value is 10, the operator is >, and the expected value is 9, then the result 10 -9 > 0 is true, which means that the row data in the empty column has exceeded the threshold, and the task is judged to fail.
@ -50,7 +51,6 @@ The goal of the null value check is to check the number of empty rows in the spe
```sql
SELECT COUNT(*) AS miss FROM ${src_table} WHERE (${src_field} is null or ${src_field} = '') AND (${src_filter})
```
- The SQL to calculate the total number of rows in the table is as follows:
```sql
@ -61,155 +61,163 @@ The goal of the null value check is to check the number of empty rows in the spe
![dataquality_null_check](../../../img/tasks/demo/null_check.png)
| **Parameter** | **Description** |
| ----- | ---- |
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | Drop-down to select the table where the validation data is located. |
| Src filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Src table check column | Drop-down to select the check column name. |
| Check method | <ul><li>[Expected-Actual]</li><li>[Actual-Expected]</li><li>[Actual/Expected]x100%</li><li>[(Expected-Actual)/Expected]x100%</li></ul> |
| Check operators | =, >, >=, <, <=, ! = |
| Threshold | The value used in the formula for comparison. |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
| Expected value type | Select the desired type from the drop-down menu. |
| **Parameter** | **Description** |
|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | Drop-down to select the table where the validation data is located. |
| Src filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Src table check column | Drop-down to select the check column name. |
| Check method | <ul><li>[Expected-Actual]</li><li>[Actual-Expected]</li><li>[Actual/Expected]x100%</li><li>[(Expected-Actual)/Expected]x100%</li></ul> |
| Check operators | =, >, >=, <, <=, ! = |
| Threshold | The value used in the formula for comparison. |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
| Expected value type | Select the desired type from the drop-down menu. |
## Timeliness Check of Single Table Check
### Inspection Introduction
The timeliness check is used to check whether the data is processed within the expected time. The start time and end time can be specified to define the time range. If the amount of data within the time range does not reach the set threshold, the check task will be judged as fail.
### Interface Operation Guide
![dataquality_timeliness_check](../../../img/tasks/demo/timeliness_check.png)
| **Parameter** | **Description** |
| ----- | ---- |
| Source data type | Select MySQL, PostgreSQL, etc.
| Source data source | The corresponding data source under the source data type.
| Source data table | Drop-down to select the table where the validation data is located. |
| Src filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Src table check column | Drop-down to select check column name. |
| Start time | The start time of a time range. |
| end time | The end time of a time range. |
| Time Format | Set the corresponding time format. |
| Check method | <ul><li>[Expected-Actual]</li><li>[Actual-Expected]</li><li>[Actual/Expected]x100%</li><li>[(Expected-Actual)/Expected]x100%</li></ul> |
| Check operators | =, >, >=, <, <=, ! = |
| Threshold | The value used in the formula for comparison. |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
| Expected value type | Select the desired type from the drop-down menu. |
| **Parameter** | **Description** |
|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | Drop-down to select the table where the validation data is located. |
| Src filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Src table check column | Drop-down to select check column name. |
| Start time | The start time of a time range. |
| end time | The end time of a time range. |
| Time Format | Set the corresponding time format. |
| Check method | <ul><li>[Expected-Actual]</li><li>[Actual-Expected]</li><li>[Actual/Expected]x100%</li><li>[(Expected-Actual)/Expected]x100%</li></ul> |
| Check operators | =, >, >=, <, <=, ! = |
| Threshold | The value used in the formula for comparison. |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
| Expected value type | Select the desired type from the drop-down menu. |
## Field Length Check for Single Table Check
### Inspection Introduction
The goal of field length verification is to check whether the length of the selected field meets the expectations. If there is data that does not meet the requirements, and the number of rows exceeds the threshold, the task will be judged to fail.
### Interface Operation Guide
![dataquality_length_check](../../../img/tasks/demo/field_length_check.png)
| **Parameter** | **Description** |
| ----- | ---- |
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | Drop-down to select the table where the validation data is located. |
| Src filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Src table check column | Drop-down to select the check column name. |
| Logical operators | =, >, >=, <, <=, ! = |
| Field length limit | Like the title. |
| Check method | <ul><li>[Expected-Actual]</li><li>[Actual-Expected]</li><li>[Actual/Expected]x100%</li><li>[(Expected-Actual)/Expected]x100%</li></ul> |
| Check operators | =, >, >=, <, <=, ! = |
| Threshold | The value used in the formula for comparison. |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
| Expected value type | Select the desired type from the drop-down menu. |
| **Parameter** | **Description** |
|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | Drop-down to select the table where the validation data is located. |
| Src filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Src table check column | Drop-down to select the check column name. |
| Logical operators | =, >, >=, <, <=, ! = |
| Field length limit | Like the title. |
| Check method | <ul><li>[Expected-Actual]</li><li>[Actual-Expected]</li><li>[Actual/Expected]x100%</li><li>[(Expected-Actual)/Expected]x100%</li></ul> |
| Check operators | =, >, >=, <, <=, ! = |
| Threshold | The value used in the formula for comparison. |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
| Expected value type | Select the desired type from the drop-down menu. |
## Uniqueness Check for Single Table Check
### Inspection Introduction
The goal of the uniqueness check is to check whether the fields are duplicated. It is generally used to check whether the primary key is duplicated. If there are duplicates and the threshold is reached, the check task will be judged to be failed.
### Interface Operation Guide
![dataquality_uniqueness_check](../../../img/tasks/demo/uniqueness_check.png)
| **Parameter** | **Description** |
| ----- | ---- |
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | Drop-down to select the table where the validation data is located. |
| Src filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Src table check column | Drop-down to select the check column name. |
| Check method | <ul><li>[Expected-Actual]</li><li>[Actual-Expected]</li><li>[Actual/Expected]x100%</li><li>[(Expected-Actual)/Expected]x100%</li></ul> |
| Check operators | =, >, >=, <, <=, ! = |
| Threshold | The value used in the formula for comparison. |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
| Expected value type | Select the desired type from the drop-down menu. |
| **Parameter** | **Description** |
|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | Drop-down to select the table where the validation data is located. |
| Src filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Src table check column | Drop-down to select the check column name. |
| Check method | <ul><li>[Expected-Actual]</li><li>[Actual-Expected]</li><li>[Actual/Expected]x100%</li><li>[(Expected-Actual)/Expected]x100%</li></ul> |
| Check operators | =, >, >=, <, <=, ! = |
| Threshold | The value used in the formula for comparison. |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
| Expected value type | Select the desired type from the drop-down menu. |
## Regular Expression Check for Single Table Check
### Inspection Introduction
The goal of regular expression verification is to check whether the format of the value of a field meets the requirements, such as time format, email format, ID card format, etc. If there is data that does not meet the format and exceeds the threshold, the task will be judged as failed.
### Interface Operation Guide
![dataquality_regex_check](../../../img/tasks/demo/regexp_check.png)
| **Parameter** | **Description** |
| ----- | ---- |
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | Drop-down to select the table where the validation data is located. |
| Src filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Src table check column | Drop-down to select check column name. |
| Regular expression | As title. |
| Check method | <ul><li>[Expected-Actual]</li><li>[Actual-Expected]</li><li>[Actual/Expected]x100%</li><li>[(Expected-Actual)/Expected]x100%</li></ul> |
| Check operators | =, >, >=, <, <=, ! = |
| Threshold | The value used in the formula for comparison. |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
| Expected value type | Select the desired type from the drop-down menu. |
| **Parameter** | **Description** |
|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | Drop-down to select the table where the validation data is located. |
| Src filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Src table check column | Drop-down to select check column name. |
| Regular expression | As title. |
| Check method | <ul><li>[Expected-Actual]</li><li>[Actual-Expected]</li><li>[Actual/Expected]x100%</li><li>[(Expected-Actual)/Expected]x100%</li></ul> |
| Check operators | =, >, >=, <, <=, ! = |
| Threshold | The value used in the formula for comparison. |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
| Expected value type | Select the desired type from the drop-down menu. |
## Enumeration Value Validation for Single Table Check
### Inspection Introduction
The goal of enumeration value verification is to check whether the value of a field is within the range of the enumeration value. If there is data that is not in the range of the enumeration value and exceeds the threshold, the task will be judged to fail.
### Interface Operation Guide
![dataquality_enum_check](../../../img/tasks/demo/enumeration_check.png)
| **Parameter** | **Description** |
| ----- | ---- |
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | Drop-down to select the table where the validation data is located. |
| Src table filter conditions | Such as title, also used when counting the total number of rows in the table, optional. |
| Src table check column | Drop-down to select the check column name. |
| List of enumeration values | Separated by commas. |
| Check method | <ul><li>[Expected-Actual]</li><li>[Actual-Expected]</li><li>[Actual/Expected]x100%</li><li>[(Expected-Actual)/Expected]x100%</li></ul> |
| Check operators | =, >, >=, <, <=, ! = |
| Threshold | The value used in the formula for comparison. |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
| Expected value type | Select the desired type from the drop-down menu. |
| **Parameter** | **Description** |
|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | Drop-down to select the table where the validation data is located. |
| Src table filter conditions | Such as title, also used when counting the total number of rows in the table, optional. |
| Src table check column | Drop-down to select the check column name. |
| List of enumeration values | Separated by commas. |
| Check method | <ul><li>[Expected-Actual]</li><li>[Actual-Expected]</li><li>[Actual/Expected]x100%</li><li>[(Expected-Actual)/Expected]x100%</li></ul> |
| Check operators | =, >, >=, <, <=, ! = |
| Threshold | The value used in the formula for comparison. |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
| Expected value type | Select the desired type from the drop-down menu. |
## Table Row Number Verification for Single Table Check
### Inspection Introduction
The goal of table row number verification is to check whether the number of rows in the table reaches the expected value. If the number of rows does not meet the standard, the task will be judged as failed.
### Interface Operation Guide
![dataquality_count_check](../../../img/tasks/demo/table_count_check.png)
| **Parameter** | **Description** |
| ----- | ---- |
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | Drop-down to select the table where the validation data is located. |
| Src filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Src table check column | Drop-down to select the check column name. |
| Check method | <ul><li>[Expected-Actual]</li><li>[Actual-Expected]</li><li>[Actual/Expected]x100%</li><li>[(Expected-Actual)/Expected]x100%</li></ul> |
| Check operators | =, >, >=, <, <=, ! = |
| Threshold | The value used in the formula for comparison. |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
| Expected value type | Select the desired type from the drop-down menu. |
| **Parameter** | **Description** |
|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | Drop-down to select the table where the validation data is located. |
| Src filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Src table check column | Drop-down to select the check column name. |
| Check method | <ul><li>[Expected-Actual]</li><li>[Actual-Expected]</li><li>[Actual/Expected]x100%</li><li>[(Expected-Actual)/Expected]x100%</li></ul> |
| Check operators | =, >, >=, <, <=, ! = |
| Threshold | The value used in the formula for comparison. |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
| Expected value type | Select the desired type from the drop-down menu. |
## Custom SQL Check for Single Table Check
@ -217,36 +225,38 @@ The goal of table row number verification is to check whether the number of rows
![dataquality_custom_sql_check](../../../img/tasks/demo/custom_sql_check.png)
| **Parameter** | **Description** |
| ----- | ---- |
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | Drop-down to select the table where the data to be verified is located. |
| Actual value name | Alias in SQL for statistical value calculation, such as max_num. |
|Actual value calculation SQL | SQL for outputting actual values. Note:<ul><li>The SQL must be statistical SQL, such as counting the number of rows, calculating the maximum value, minimum value, etc.</li><li>Select max(a) as max_num from ${src_table}, the table name must be filled like this.</li></ul> |
| Src filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Check method | <ul><li>[Expected-Actual]</li><li>[Actual-Expected]</li><li>[Actual/Expected]x100%</li><li>[(Expected-Actual)/Expected]x100%</li></ul> |
| Check operators | =, >, >=, <, <=, ! = |
| Threshold | The value used in the formula for comparison. |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
| Expected value type | Select the desired type from the drop-down menu. |
| **Parameter** | **Description** |
|------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | Drop-down to select the table where the data to be verified is located. |
| Actual value name | Alias in SQL for statistical value calculation, such as max_num. |
| Actual value calculation SQL | SQL for outputting actual values. Note:<ul><li>The SQL must be statistical SQL, such as counting the number of rows, calculating the maximum value, minimum value, etc.</li><li>Select max(a) as max_num from ${src_table}, the table name must be filled like this.</li></ul> |
| Src filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Check method | <ul><li>[Expected-Actual]</li><li>[Actual-Expected]</li><li>[Actual/Expected]x100%</li><li>[(Expected-Actual)/Expected]x100%</li></ul> |
| Check operators | =, >, >=, <, <=, ! = |
| Threshold | The value used in the formula for comparison. |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
| Expected value type | Select the desired type from the drop-down menu. |
## Accuracy Check of Multi-table
### Inspection Introduction
Accuracy checks are performed by comparing the accuracy differences of data records for selected fields between two tables, examples are as follows
- table test1
| c1 | c2 |
| :---: | :---: |
| a | 1 |
| b | 2 |
|:--:|:--:|
| a | 1 |
| b | 2 |
- table test2
| c21 | c22 |
| :---: | :---: |
| a | 1 |
| b | 3 |
|:---:|:---:|
| a | 1 |
| b | 3 |
If you compare the data in c1 and c21, the tables test1 and test2 are exactly the same. If you compare c2 and c22, the data in table test1 and table test2 are inconsistent.
@ -254,45 +264,47 @@ If you compare the data in c1 and c21, the tables test1 and test2 are exactly th
![dataquality_multi_table_accuracy_check](../../../img/tasks/demo/multi_table_accuracy_check.png)
| **Parameter** | **Description** |
| ----- | ---- |
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | Drop-down to select the table where the data to be verified is located. |
| Src filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Target data type | Choose MySQL, PostgreSQL, etc. |
| Target data source | The corresponding data source under the source data type. |
| Target data table | Drop-down to select the table where the data to be verified is located. |
| Target filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Check column | Fill in the source data column, operator and target data column respectively. |
| Verification method | Select the desired verification method. |
| Operators | =, >, >=, <, <=, ! = |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li><ul> |
| Expected value type | Select the desired type in the drop-down menu, only `SrcTableTotalRow`, `TargetTableTotalRow` and fixed value are suitable for selection here. |
| **Parameter** | **Description** |
|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | Drop-down to select the table where the data to be verified is located. |
| Src filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Target data type | Choose MySQL, PostgreSQL, etc. |
| Target data source | The corresponding data source under the source data type. |
| Target data table | Drop-down to select the table where the data to be verified is located. |
| Target filter conditions | Such as the title, it will also be used when counting the total number of rows in the table, optional. |
| Check column | Fill in the source data column, operator and target data column respectively. |
| Verification method | Select the desired verification method. |
| Operators | =, >, >=, <, <=, ! = |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li><ul> |
| Expected value type | Select the desired type in the drop-down menu, only `SrcTableTotalRow`, `TargetTableTotalRow` and fixed value are suitable for selection here. |
## Comparison of the values checked by the two tables
### Inspection Introduction
Two-table value comparison allows users to customize different SQL statistics for two tables and compare the corresponding values. For example, for the source table A, the total amount of a certain column is calculated, and for the target table, the total amount of a certain column is calculated. value sum2, compare sum1 and sum2 to determine the check result.
### Interface Operation Guide
![dataquality_multi_table_comparison_check](../../../img/tasks/demo/multi_table_comparison_check.png)
| **Parameter** | **Description** |
| ----- | ---- |
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | The table where the data is to be verified. |
| Actual value name | Calculate the alias in SQL for the actual value, such as max_age1. |
| Actual value calculation SQL | SQL for outputting actual values. Note: <ul><li>The SQL must be statistical SQL, such as counting the number of rows, calculating the maximum value, minimum value, etc.</li><li>Select max(age) as max_age1 from ${src_table} The table name must be filled like this.</li></ul> |
| Target data type | Choose MySQL, PostgreSQL, etc. |
| Target data source | The corresponding data source under the source data type. |
| Target data table | The table where the data is to be verified. |
| Expected value name | Calculate the alias in SQL for the expected value, such as max_age2. |
| **Parameter** | **Description** |
|--------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Source data type | Select MySQL, PostgreSQL, etc. |
| Source data source | The corresponding data source under the source data type. |
| Source data table | The table where the data is to be verified. |
| Actual value name | Calculate the alias in SQL for the actual value, such as max_age1. |
| Actual value calculation SQL | SQL for outputting actual values. Note: <ul><li>The SQL must be statistical SQL, such as counting the number of rows, calculating the maximum value, minimum value, etc.</li><li>Select max(age) as max_age1 from ${src_table} The table name must be filled like this.</li></ul> |
| Target data type | Choose MySQL, PostgreSQL, etc. |
| Target data source | The corresponding data source under the source data type. |
| Target data table | The table where the data is to be verified. |
| Expected value name | Calculate the alias in SQL for the expected value, such as max_age2. |
| Expected value calculation SQL | SQL for outputting expected value. Note: <ul><li>The SQL must be statistical SQL, such as counting the number of rows, calculating the maximum value, minimum value, etc.</li><li>Select max(age) as max_age2 from ${target_table} The table name must be filled like this.</li></ul> |
| Verification method | Select the desired verification method. |
| Operators | =, >, >=, <, <=, ! = |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
| Verification method | Select the desired verification method. |
| Operators | =, >, >=, <, <=, ! = |
| Failure strategy | <ul><li>Alert: The data quality task failed, the DolphinScheduler task result is successful, and an alert is sent.</li><li>Blocking: The data quality task fails, the DolphinScheduler task result is failed, and an alarm is sent.</li></ul> |
## Task result view

View File

@ -0,0 +1,23 @@
# AWS Athena
![AWS Athena](../../../../img/new_ui/dev/datasource/athena.png)
## Datasource Parameters
| **Datasource** | **Description** |
|----------------------------|-----------------------------------------------------------|
| Datasource | Select ATHENA. |
| Datasource name | Enter the name of the DataSource. |
| Description | Enter a description of the DataSource. |
| Username | Set the AWS access key. |
| Password | Set the AWS secret access key. |
| AwsRegion | Set the AWS region. |
| Database name | Enter the database name of the ATHENA connection. |
| Jdbc connection parameters | Parameter settings for ATHENA connection, in JSON format. |
## Native Supported
- No, read section example in [datasource-setting](../howto/datasource-setting.md) `DataSource Center` section to activate this datasource.
- JDBC driver configuration reference document [athena-connect-with-jdbc](https://docs.amazonaws.cn/athena/latest/ug/connect-with-jdbc.html)
- Driver download link [SimbaAthenaJDBC-2.0.31.1000/AthenaJDBC42.jar](https://s3.cn-north-1.amazonaws.com.cn/athena-downloads-cn/drivers/JDBC/SimbaAthenaJDBC-2.0.31.1000/AthenaJDBC42.jar)

View File

@ -4,16 +4,16 @@
## Datasource Parameters
| **Datasource** | **Description** |
| --- | --- |
| Datasource | Select CLICKHOUSE. |
| Datasource Name | Enter the name of the datasource. |
| Description | Enter a description of the datasource. |
| IP/Host Name | Enter the CLICKHOUSE service IP. |
| Port | Enter the CLICKHOUSE service port. |
| Username | Set the username for CLICKHOUSE connection. |
| Password | Set the password for CLICKHOUSE connection. |
| Database Name | Enter the database name of the CLICKHOUSE connection. |
| **Datasource** | **Description** |
|-------------------------|---------------------------------------------------------------|
| Datasource | Select CLICKHOUSE. |
| Datasource Name | Enter the name of the datasource. |
| Description | Enter a description of the datasource. |
| IP/Host Name | Enter the CLICKHOUSE service IP. |
| Port | Enter the CLICKHOUSE service port. |
| Username | Set the username for CLICKHOUSE connection. |
| Password | Set the password for CLICKHOUSE connection. |
| Database Name | Enter the database name of the CLICKHOUSE connection. |
| jdbc connect parameters | Parameter settings for CLICKHOUSE connection, in JSON format. |
## Native Supported

View File

@ -4,16 +4,16 @@
## Datasource Parameters
| **Datasource** | **Description** |
| --- | --- |
| Datasource | Select DB2. |
| Datasource Name | Enter the name of the datasource. |
| Description | Enter a description of the datasource. |
| IP/Host Name | Enter the DB2 service IP. |
| Port | Enter the DB2 service port. |
| Username | Set the username for DB2 connection. |
| Password | Set the password for DB2 connection. |
| Database Name | Enter the database name of the DB2 connection. |
| **Datasource** | **Description** |
|-------------------------|--------------------------------------------------------|
| Datasource | Select DB2. |
| Datasource Name | Enter the name of the datasource. |
| Description | Enter a description of the datasource. |
| IP/Host Name | Enter the DB2 service IP. |
| Port | Enter the DB2 service port. |
| Username | Set the username for DB2 connection. |
| Password | Set the password for DB2 connection. |
| Database Name | Enter the database name of the DB2 connection. |
| jdbc connect parameters | Parameter settings for DB2 connection, in JSON format. |
## Native Supported

View File

@ -6,16 +6,16 @@
## Datasource Parameters
| **Datasource** | **Description** |
| --- | --- |
| Datasource | Select HIVE. |
| Datasource name | Enter the name of the DataSource. |
| Description | Enter a description of the DataSource. |
| IP/Host Name | Enter the HIVE service IP. |
| Port | Enter the HIVE service port. |
| Username | Set the username for HIVE connection. |
| Password | Set the password for HIVE connection. |
| Database name | Enter the database name of the HIVE connection. |
| **Datasource** | **Description** |
|----------------------------|---------------------------------------------------------|
| Datasource | Select HIVE. |
| Datasource name | Enter the name of the DataSource. |
| Description | Enter a description of the DataSource. |
| IP/Host Name | Enter the HIVE service IP. |
| Port | Enter the HIVE service port. |
| Username | Set the username for HIVE connection. |
| Password | Set the password for HIVE connection. |
| Database name | Enter the database name of the HIVE connection. |
| Jdbc connection parameters | Parameter settings for HIVE connection, in JSON format. |
> NOTICE: If you wish to execute multiple HIVE SQL in the same session, you could set `support.hive.oneSession = true` in `common.properties`.

View File

@ -4,16 +4,16 @@
## Datasource Parameters
| **Datasource** | **Description** |
| --- | --- |
| Datasource | Select MYSQL. |
| Datasource name | Enter the name of the DataSource. |
| Description | Enter a description of the DataSource. |
| IP/Host Name | Enter the MYSQL service IP. |
| Port | Enter the MYSQL service port. |
| Username | Set the username for MYSQL connection. |
| Password | Set the password for MYSQL connection. |
| Database name | Enter the database name of the MYSQL connection. |
| **Datasource** | **Description** |
|----------------------------|----------------------------------------------------------|
| Datasource | Select MYSQL. |
| Datasource name | Enter the name of the DataSource. |
| Description | Enter a description of the DataSource. |
| IP/Host Name | Enter the MYSQL service IP. |
| Port | Enter the MYSQL service port. |
| Username | Set the username for MYSQL connection. |
| Password | Set the password for MYSQL connection. |
| Database name | Enter the database name of the MYSQL connection. |
| Jdbc connection parameters | Parameter settings for MYSQL connection, in JSON format. |
## Native Supported

View File

@ -4,16 +4,16 @@
## Datasource Parameters
| **Datasource** | **Description** |
| --- | --- |
| Datasource | Select Oracle. |
| Datasource Name | Enter the name of the datasource. |
| Description | Enter a description of the datasource. |
| IP/Host Name | Enter the Oracle service IP. |
| Port | Enter the Oracle service port. |
| Username | Set the username for Oracle connection. |
| Password | Set the password for Oracle connection. |
| Database Name | Enter the database name of the Oracle connection. |
| **Datasource** | **Description** |
|-------------------------|-----------------------------------------------------------|
| Datasource | Select Oracle. |
| Datasource Name | Enter the name of the datasource. |
| Description | Enter a description of the datasource. |
| IP/Host Name | Enter the Oracle service IP. |
| Port | Enter the Oracle service port. |
| Username | Set the username for Oracle connection. |
| Password | Set the password for Oracle connection. |
| Database Name | Enter the database name of the Oracle connection. |
| jdbc connect parameters | Parameter settings for Oracle connection, in JSON format. |
## Native Supported

View File

@ -4,16 +4,16 @@
## Datasource Parameters
| **Datasource** | **Description** |
| --- | --- |
| Datasource | Select POSTGRESQL. |
| Datasource name | Enter the name of the DataSource. |
| Description | Enter a description of the DataSource. |
| IP/Host Name | Enter the PostgreSQL service IP. |
| Port | Enter the PostgreSQL service port. |
| Username | Set the username for PostgreSQL connection. |
| Password | Set the password for PostgreSQL connection. |
| Database name | Enter the database name of the PostgreSQL connection. |
| **Datasource** | **Description** |
|----------------------------|---------------------------------------------------------------|
| Datasource | Select POSTGRESQL. |
| Datasource name | Enter the name of the DataSource. |
| Description | Enter a description of the DataSource. |
| IP/Host Name | Enter the PostgreSQL service IP. |
| Port | Enter the PostgreSQL service port. |
| Username | Set the username for PostgreSQL connection. |
| Password | Set the password for PostgreSQL connection. |
| Database name | Enter the database name of the PostgreSQL connection. |
| Jdbc connection parameters | Parameter settings for PostgreSQL connection, in JSON format. |
## Native Supported

View File

@ -4,19 +4,18 @@
## Datasource Parameters
| **Datasource** | **Description** |
| --- | --- |
| Datasource | Select Presto. |
| Datasource Name | Enter the name of the datasource. |
| Description | Enter a description of the datasource. |
| IP/Host Name | Enter the Presto service IP. |
| Port | Enter the Presto service port. |
| Username | Set the username for Presto connection. |
| Password | Set the password for Presto connection. |
| Database Name | Enter the database name of the Presto connection. |
| **Datasource** | **Description** |
|-------------------------|-----------------------------------------------------------|
| Datasource | Select Presto. |
| Datasource Name | Enter the name of the datasource. |
| Description | Enter a description of the datasource. |
| IP/Host Name | Enter the Presto service IP. |
| Port | Enter the Presto service port. |
| Username | Set the username for Presto connection. |
| Password | Set the password for Presto connection. |
| Database Name | Enter the database name of the Presto connection. |
| jdbc connect parameters | Parameter settings for Presto connection, in JSON format. |
## Native Supported
Yes, could use this datasource by default.

View File

@ -4,16 +4,16 @@
## Datasource Parameters
| **Datasource** | **Description** |
| --- | --- |
| Datasource | Select Redshift. |
| Datasource Name | Enter the name of the datasource. |
| Description | Enter a description of the datasource. |
| IP/Host Name | Enter the Redshift service IP. |
| Port | Enter the Redshift service port. |
| Username | Set the username for Redshift connection. |
| Password | Set the password for Redshift connection. |
| Database Name | Enter the database name of the Redshift connection. |
| **Datasource** | **Description** |
|-------------------------|-------------------------------------------------------------|
| Datasource | Select Redshift. |
| Datasource Name | Enter the name of the datasource. |
| Description | Enter a description of the datasource. |
| IP/Host Name | Enter the Redshift service IP. |
| Port | Enter the Redshift service port. |
| Username | Set the username for Redshift connection. |
| Password | Set the password for Redshift connection. |
| Database Name | Enter the database name of the Redshift connection. |
| jdbc connect parameters | Parameter settings for Redshift connection, in JSON format. |
## Native Supported

View File

@ -4,16 +4,16 @@
## Datasource Parameters
| **Datasource** | **Description** |
| --- | --- |
| Datasource | Select Spark. |
| Datasource name | Enter the name of the DataSource. |
| Description | Enter a description of the DataSource. |
| IP/Host Name | Enter the Spark service IP. |
| Port | Enter the Spark service port. |
| Username | Set the username for Spark connection. |
| Password | Set the password for Spark connection. |
| Database name | Enter the database name of the Spark connection. |
| **Datasource** | **Description** |
|----------------------------|----------------------------------------------------------|
| Datasource | Select Spark. |
| Datasource name | Enter the name of the DataSource. |
| Description | Enter a description of the DataSource. |
| IP/Host Name | Enter the Spark service IP. |
| Port | Enter the Spark service port. |
| Username | Set the username for Spark connection. |
| Password | Set the password for Spark connection. |
| Database name | Enter the database name of the Spark connection. |
| Jdbc connection parameters | Parameter settings for Spark connection, in JSON format. |
## Native Supported

View File

@ -4,16 +4,16 @@
## Datasource Parameters
| **Datasource** | **Description** |
| --- | --- |
| Datasource | Select SQLSERVER. |
| Datasource Name | Enter the name of the datasource. |
| Description | Enter a description of the datasource. |
| IP/Host Name | Enter the SQLSERVER service IP. |
| Port | Enter the SQLSERVER service port. |
| Username | Set the username for SQLSERVER connection. |
| Password | Set the password for SQLSERVER connection. |
| Database Name | Enter the database name of the SQLSERVER connection. |
| **Datasource** | **Description** |
|-------------------------|--------------------------------------------------------------|
| Datasource | Select SQLSERVER. |
| Datasource Name | Enter the name of the datasource. |
| Description | Enter a description of the datasource. |
| IP/Host Name | Enter the SQLSERVER service IP. |
| Port | Enter the SQLSERVER service port. |
| Username | Set the username for SQLSERVER connection. |
| Password | Set the password for SQLSERVER connection. |
| Database Name | Enter the database name of the SQLSERVER connection. |
| jdbc connect parameters | Parameter settings for SQLSERVER connection, in JSON format. |
## Native Supported

View File

@ -5,8 +5,8 @@
This article describes how to add a new master service or worker service to an existing DolphinScheduler cluster.
```
Attention: There cannot be more than one master service process or worker service process on a physical machine.
If the physical machine which locate the expansion master or worker node has already installed the scheduled service, check the [1.4 Modify configuration] and edit the configuration file `conf/config/install_config.conf` on ** all ** nodes, add masters or workers parameter, and restart the scheduling cluster.
Attention: There cannot be more than one master service process or worker service process on a physical machine.
If the physical machine which locate the expansion master or worker node has already installed the scheduled service, check the [1.4 Modify configuration] and edit the configuration file `conf/config/install_config.conf` on ** all ** nodes, add masters or workers parameter, and restart the scheduling cluster.
```
### Basic software installation
@ -14,9 +14,8 @@ This article describes how to add a new master service or worker service to an e
* [required] [JDK](https://www.oracle.com/technetwork/java/javase/downloads/index.html) (version 1.8+): must install, install and configure `JAVA_HOME` and `PATH` variables under `/etc/profile`
* [optional] If the expansion is a worker node, you need to consider whether to install an external client, such as Hadoop, Hive, Spark Client.
```markdown
Attention: DolphinScheduler itself does not depend on Hadoop, Hive, Spark, but will only call their Client for the corresponding task submission.
Attention: DolphinScheduler itself does not depend on Hadoop, Hive, Spark, but will only call their Client for the corresponding task submission.
```
### Get Installation Package
@ -31,13 +30,13 @@ This article describes how to add a new master service or worker service to an e
mkdir -p /opt
cd /opt
# decompress
tar -zxvf apache-dolphinscheduler-<version>-bin.tar.gz -C /opt
tar -zxvf apache-dolphinscheduler-3.1.0-bin.tar.gz -C /opt
cd /opt
mv apache-dolphinscheduler-<version>-bin dolphinscheduler
mv apache-dolphinscheduler-3.1.0-bin dolphinscheduler
```
```markdown
Attention: You can copy the installation package directly from an existing environment to an expanded physical machine.
Attention: You can copy the installation package directly from an existing environment to an expanded physical machine.
```
### Create Deployment Users
@ -58,53 +57,49 @@ sed -i 's/Defaults requirett/#Defaults requirett/g' /etc/sudoers
```
```markdown
Attention:
- Since it is `sudo -u {linux-user}` to switch between different Linux users to run multi-tenant jobs, the deploying user needs to have sudo privileges and be password free.
- If you find the line `Default requiretty` in the `/etc/sudoers` file, please also comment it out.
- If have needs to use resource uploads, you also need to assign read and write permissions to the deployment user on `HDFS or MinIO`.
Attention:
- Since it is `sudo -u {linux-user}` to switch between different Linux users to run multi-tenant jobs, the deploying user needs to have sudo privileges and be password free.
- If you find the line `Default requiretty` in the `/etc/sudoers` file, please also comment it out.
- If have needs to use resource uploads, you also need to assign read and write permissions to the deployment user on `HDFS or MinIO`.
```
### Modify Configuration
- From an existing node such as `Master/Worker`, copy the configuration directory directly to replace the configuration directory in the new node. After finishing the file copy, check whether the configuration items are correct.
```markdown
Highlights:
datasource.properties: database connection information
zookeeper.properties: information for connecting zk
common.properties: Configuration information about the resource store (if hadoop is set up, please check if the core-site.xml and hdfs-site.xml configuration files exist).
dolphinscheduler_env.sh: environment Variables
````
```markdown
Highlights:
datasource.properties: database connection information
zookeeper.properties: information for connecting zk
common.properties: Configuration information about the resource store (if hadoop is set up, please check if the core-site.xml and hdfs-site.xml configuration files exist).
dolphinscheduler_env.sh: environment Variables
```
- Modify the `dolphinscheduler_env.sh` environment variable in the `bin/env/dolphinscheduler_env.sh` directory according to the machine configuration (the following is the example that all the used software install under `/opt/soft`)
```shell
export HADOOP_HOME=/opt/soft/hadoop
export HADOOP_CONF_DIR=/opt/soft/hadoop/etc/hadoop
# export SPARK_HOME1=/opt/soft/spark1
export SPARK_HOME2=/opt/soft/spark2
export PYTHON_HOME=/opt/soft/python
export JAVA_HOME=/opt/soft/jav
export HIVE_HOME=/opt/soft/hive
export FLINK_HOME=/opt/soft/flink
export DATAX_HOME=/opt/soft/datax/bin/datax.py
export PATH=$HADOOP_HOME/bin:$SPARK_HOME2/bin:$PYTHON_HOME:$JAVA_HOME/bin:$HIVE_HOME/bin:$PATH:$FLINK_HOME/bin:$DATAX_HOME:$PATH
```shell
export HADOOP_HOME=/opt/soft/hadoop
export HADOOP_CONF_DIR=/opt/soft/hadoop/etc/hadoop
# export SPARK_HOME1=/opt/soft/spark1
export SPARK_HOME2=/opt/soft/spark2
export PYTHON_HOME=/opt/soft/python
export JAVA_HOME=/opt/soft/jav
export HIVE_HOME=/opt/soft/hive
export FLINK_HOME=/opt/soft/flink
export DATAX_HOME=/opt/soft/datax/bin/datax.py
export PATH=$HADOOP_HOME/bin:$SPARK_HOME2/bin:$PYTHON_HOME:$JAVA_HOME/bin:$HIVE_HOME/bin:$PATH:$FLINK_HOME/bin:$DATAX_HOME:$PATH
```
`Attention: This step is very important, such as `JAVA_HOME` and `PATH` is necessary to configure if haven not used just ignore or comment out`
```
`Attention: This step is very important, such as `JAVA_HOME` and `PATH` is necessary to configure if haven not used just ignore or comment out`
- Soft link the `JDK` to `/usr/bin/java` (still using `JAVA_HOME=/opt/soft/java` as an example)
```shell
sudo ln -s /opt/soft/java/bin/java /usr/bin/java
```
- Modify the configuration file `conf/config/install_config.conf` on the **all** nodes, synchronizing the following configuration.
* To add a new master node, you need to modify the IPs and masters parameters.
* To add a new worker node, modify the IPs and workers parameters.
```shell
sudo ln -s /opt/soft/java/bin/java /usr/bin/java
```
- Modify the configuration file `conf/config/install_config.conf` on the **all** nodes, synchronizing the following configuration.
* To add a new master node, you need to modify the IPs and masters parameters.
* To add a new worker node, modify the IPs and workers parameters.
```shell
# which machines to deploy DS services on, separated by commas between multiple physical machines
@ -120,6 +115,7 @@ masters="existing master01,existing master02,ds1,ds2"
workers="existing worker01:default,existing worker02:default,ds3:default,ds4:default"
```
- If the expansion is for worker nodes, you need to set the worker group, refer to the security of the [Worker grouping](./security.md)
- On all new nodes, change the directory permissions so that the deployment user has access to the DolphinScheduler directory
@ -154,26 +150,26 @@ bash bin/dolphinscheduler-daemon.sh start alert-server # start alert service
```
```
Attention: When using `stop-all.sh` or `stop-all.sh`, if the physical machine execute the command is not configured to be ssh-free on all machines, it will prompt to enter the password
Attention: When using `stop-all.sh` or `stop-all.sh`, if the physical machine execute the command is not configured to be ssh-free on all machines, it will prompt to enter the password
```
- After completing the script, use the `jps` command to see if every node service is started (`jps` comes with the `Java JDK`)
```
MasterServer ----- master service
WorkerServer ----- worker service
ApiApplicationServer ----- api service
AlertServer ----- alert service
MasterServer ----- master service
WorkerServer ----- worker service
ApiApplicationServer ----- api service
AlertServer ----- alert service
```
After successful startup, you can view the logs, which are stored in the `logs` folder.
```Log Path
logs/
├── dolphinscheduler-alert-server.log
├── dolphinscheduler-master-server.log
├── dolphinscheduler-worker-server.log
├── dolphinscheduler-api-server.log
logs/
├── dolphinscheduler-alert-server.log
├── dolphinscheduler-master-server.log
├── dolphinscheduler-worker-server.log
├── dolphinscheduler-api-server.log
```
If the above services start normally and the scheduling system page is normal, check whether there is an expanded Master or Worker service in the [Monitor] of the web system. If it exists, the expansion is complete.
@ -187,8 +183,8 @@ There are two steps for shrinking. After performing the following two steps, the
### Stop the Service on the Scaled-Down Node
* If you are scaling down the master node, identify the physical machine where the master service is located, and stop the master service on the physical machine.
* If scale down the worker node, determine the physical machine where the worker service scale down and stop the worker services on the physical machine.
* If you are scaling down the master node, identify the physical machine where the master service is located, and stop the master service on the physical machine.
* If scale down the worker node, determine the physical machine where the worker service scale down and stop the worker services on the physical machine.
```shell
# stop command:
@ -211,26 +207,25 @@ bash bin/dolphinscheduler-daemon.sh start alert-server # start alert service
```
```
Attention: When using `stop-all.sh` or `stop-all.sh`, if the machine without the command is not configured to be ssh-free for all machines, it will prompt to enter the password
Attention: When using `stop-all.sh` or `stop-all.sh`, if the machine without the command is not configured to be ssh-free for all machines, it will prompt to enter the password
```
- After the script is completed, use the `jps` command to see if every node service was successfully shut down (`jps` comes with the `Java JDK`)
```
MasterServer ----- master service
WorkerServer ----- worker service
ApiApplicationServer ----- api service
AlertServer ----- alert service
MasterServer ----- master service
WorkerServer ----- worker service
ApiApplicationServer ----- api service
AlertServer ----- alert service
```
If the corresponding master service or worker service does not exist, then the master or worker service is successfully shut down.
If the corresponding master service or worker service does not exist, then the master or worker service is successfully shut down.
### Modify the Configuration File
- modify the configuration file `conf/config/install_config.conf` on the **all** nodes, synchronizing the following configuration.
* to scale down the master node, modify the IPs and masters parameters.
* to scale down worker nodes, modify the IPs and workers parameters.
- modify the configuration file `conf/config/install_config.conf` on the **all** nodes, synchronizing the following configuration.
* to scale down the master node, modify the IPs and masters parameters.
* to scale down worker nodes, modify the IPs and workers parameters.
```shell
# which machines to deploy DS services on, "localhost" for this machine
@ -246,3 +241,4 @@ masters="existing master01,existing master02,ds1,ds2"
workers="existing worker01:default,existing worker02:default,ds3:default,ds4:default"
```

View File

@ -39,3 +39,4 @@ curl --request GET 'http://localhost:50053/actuator/health'
```
> Notice: If you modify the default service port and address, you need to modify the IP+Port to the modified value.

View File

@ -5,7 +5,7 @@
We here use MySQL as an example to illustrate how to configure an external database:
> NOTE: If you use MySQL, you need to manually download [mysql-connector-java driver][mysql] (8.0.16) and move it to the libs directory of DolphinScheduler
which is `api-server/libs` and `alert-server/libs` and `master-server/libs` and `worker-server/libs`.
> which is `api-server/libs` and `alert-server/libs` and `master-server/libs` and `worker-server/libs`.
* First of all, follow the instructions in [datasource-setting](datasource-setting.md) `Pseudo-Cluster/Cluster Initialize the Database` section to create and initialize database
* Set the following environment variables in your terminal or modify the `bin/env/dolphinscheduler_env.sh` with your database username and password for `{user}` and `{password}`:
@ -26,7 +26,6 @@ DolphinScheduler stores metadata in `relational database`. Currently, we support
> If you use MySQL, you need to manually download [mysql-connector-java driver][mysql] (8.0.16) and move it to the libs directory of DolphinScheduler which is `api-server/libs` and `alert-server/libs` and `master-server/libs` and `worker-server/libs`.
For mysql 5.6 / 5.7
```shell
@ -57,6 +56,7 @@ mysql> FLUSH PRIVILEGES;
```
For PostgreSQL:
```shell
# Use psql-tools to login PostgreSQL
psql
@ -75,6 +75,7 @@ pg_ctl reload
Then, modify `./bin/env/dolphinscheduler_env.sh`, change {user} and {password} to what you set in the previous step.
For MySQL:
```shell
# for mysql
export DATABASE=${DATABASE:-mysql}
@ -85,6 +86,7 @@ export SPRING_DATASOURCE_PASSWORD={password}
```
For PostgreSQL:
```shell
# for postgresql
export DATABASE=${DATABASE:-postgresql}
@ -125,3 +127,4 @@ like Docker.
> But if you want to use MySQL as the metabase of DolphinScheduler, it only supports [8.0.16 and above](https:/ /repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar) version.
[mysql]: https://downloads.MySQL.com/archives/c-j/

View File

@ -6,9 +6,9 @@ If you are a new hand and want to experience DolphinScheduler functions, we reco
## Prerequisites
- [Helm](https://helm.sh/) version 3.1.0+
- [Kubernetes](https://kubernetes.io/) version 1.12+
- PV provisioner support in the underlying infrastructure
- [Helm](https://helm.sh/) version 3.1.0+
- [Kubernetes](https://kubernetes.io/) version 1.12+
- PV provisioner support in the underlying infrastructure
## Install DolphinScheduler
@ -93,8 +93,8 @@ The configuration file is `values.yaml`, and the [Appendix-Configuration](#appen
## Support Matrix
| Type | Support | Notes |
| ------------------------------------------------------------ | ------------ | ------------------------------------- |
| Type | Support | Notes |
|--------------------------------------------------------------|--------------|---------------------------------------|
| Shell | Yes | |
| Python2 | Yes | |
| Python3 | Indirect Yes | Refer to FAQ |
@ -241,7 +241,7 @@ externalDatabase:
> You need to change the two service images including dolphinscheduler-worker, dolphinscheduler-api.
1. Download the MySQL driver [mysql-connector-java-8.0.16.jar](https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar).
or download the Oracle driver [ojdbc8.jar](https://repo1.maven.org/maven2/com/oracle/database/jdbc/ojdbc8/) (such as `ojdbc8-19.9.0.0.jar`)
or download the Oracle driver [ojdbc8.jar](https://repo1.maven.org/maven2/com/oracle/database/jdbc/ojdbc8/) (such as `ojdbc8-19.9.0.0.jar`)
2. Create a new `Dockerfile` to add MySQL or Oracle driver:
@ -521,226 +521,228 @@ common:
## Appendix-Configuration
| Parameter | Description | Default |
| --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------- |
| `timezone` | World time and date for cities in all time zones | `Asia/Shanghai` |
| | | |
| `image.repository` | Docker image repository for the DolphinScheduler | `apache/dolphinscheduler` |
| `image.tag` | Docker image version for the DolphinScheduler | `latest` |
| `image.pullPolicy` | Image pull policy. Options: Always, Never, IfNotPresent | `IfNotPresent` |
| `image.pullSecret` | Image pull secret. An optional reference to secret in the same namespace to use for pulling any of the images | `nil` |
| | | |
| `postgresql.enabled` | If not exists external PostgreSQL, by default, the DolphinScheduler will use a internal PostgreSQL | `true` |
| `postgresql.postgresqlUsername` | The username for internal PostgreSQL | `root` |
| `postgresql.postgresqlPassword` | The password for internal PostgreSQL | `root` |
| `postgresql.postgresqlDatabase` | The database for internal PostgreSQL | `dolphinscheduler` |
| `postgresql.persistence.enabled` | Set `postgresql.persistence.enabled` to `true` to mount a new volume for internal PostgreSQL | `false` |
| `postgresql.persistence.size` | `PersistentVolumeClaim` size | `20Gi` |
| `postgresql.persistence.storageClass` | PostgreSQL data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
| `externalDatabase.type` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database type will use it | `postgresql` |
| `externalDatabase.driver` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database driver will use it | `org.postgresql.Driver` |
| `externalDatabase.host` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database host will use it | `localhost` |
| `externalDatabase.port` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database port will use it | `5432` |
| `externalDatabase.username` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database username will use it | `root` |
| `externalDatabase.password` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database password will use it | `root` |
| `externalDatabase.database` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database database will use it | `dolphinscheduler` |
| `externalDatabase.params` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database params will use it | `characterEncoding=utf8` |
| | | |
| `zookeeper.enabled` | If not exists external ZooKeeper, by default, the DolphinScheduler will use a internal ZooKeeper | `true` |
| `zookeeper.fourlwCommandsWhitelist` | A list of comma separated Four Letter Words commands to use | `srvr,ruok,wchs,cons` |
| `zookeeper.persistence.enabled` | Set `zookeeper.persistence.enabled` to `true` to mount a new volume for internal ZooKeeper | `false` |
| `zookeeper.persistence.size` | `PersistentVolumeClaim` size | `20Gi` |
| `zookeeper.persistence.storageClass` | ZooKeeper data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
| `zookeeper.zookeeperRoot` | Specify dolphinscheduler root directory in ZooKeeper | `/dolphinscheduler` |
| `externalZookeeper.zookeeperQuorum` | If exists external ZooKeeper, and set `zookeeper.enabled` value to false. Specify Zookeeper quorum | `127.0.0.1:2181` |
| `externalZookeeper.zookeeperRoot` | If exists external ZooKeeper, and set `zookeeper.enabled` value to false. Specify dolphinscheduler root directory in Zookeeper | `/dolphinscheduler` |
| | | |
| `common.configmap.DOLPHINSCHEDULER_OPTS` | The jvm options for dolphinscheduler, suitable for all servers | `""` |
| `common.configmap.DATA_BASEDIR_PATH` | User data directory path, self configuration, please make sure the directory exists and have read write permissions | `/tmp/dolphinscheduler` |
| `common.configmap.RESOURCE_STORAGE_TYPE` | Resource storage type: HDFS, S3, NONE | `HDFS` |
| `common.configmap.RESOURCE_UPLOAD_PATH` | Resource store on HDFS/S3 path, please make sure the directory exists on hdfs and have read write permissions | `/dolphinscheduler` |
| `common.configmap.FS_DEFAULT_FS` | Resource storage file system like `file:///`, `hdfs://mycluster:8020` or `s3a://dolphinscheduler` | `file:///` |
| `common.configmap.FS_S3A_ENDPOINT` | S3 endpoint when `common.configmap.RESOURCE_STORAGE_TYPE` is set to `S3` | `s3.xxx.amazonaws.com` |
| `common.configmap.FS_S3A_ACCESS_KEY` | S3 access key when `common.configmap.RESOURCE_STORAGE_TYPE` is set to `S3` | `xxxxxxx` |
| `common.configmap.FS_S3A_SECRET_KEY` | S3 secret key when `common.configmap.RESOURCE_STORAGE_TYPE` is set to `S3` | `xxxxxxx` |
| `common.configmap.HADOOP_SECURITY_AUTHENTICATION_STARTUP_STATE` | Whether to startup kerberos | `false` |
| `common.configmap.JAVA_SECURITY_KRB5_CONF_PATH` | The java.security.krb5.conf path | `/opt/krb5.conf` |
| `common.configmap.LOGIN_USER_KEYTAB_USERNAME` | The login user from keytab username | `hdfs@HADOOP.COM` |
| `common.configmap.LOGIN_USER_KEYTAB_PATH` | The login user from keytab path | `/opt/hdfs.keytab` |
| `common.configmap.KERBEROS_EXPIRE_TIME` | The kerberos expire time, the unit is hour | `2` |
| `common.configmap.HDFS_ROOT_USER` | The HDFS root user who must have the permission to create directories under the HDFS root path | `hdfs` |
| `common.configmap.RESOURCE_MANAGER_HTTPADDRESS_PORT` | Set resource manager httpaddress port for yarn | `8088` |
| `common.configmap.YARN_RESOURCEMANAGER_HA_RM_IDS` | If resourcemanager HA is enabled, please set the HA IPs | `nil` |
| `common.configmap.YARN_APPLICATION_STATUS_ADDRESS` | If resourcemanager is single, you only need to replace ds1 to actual resourcemanager hostname, otherwise keep default | `http://ds1:%s/ws/v1/cluster/apps/%s` |
| `common.configmap.SKYWALKING_ENABLE` | Set whether to enable skywalking | `false` |
| `common.configmap.SW_AGENT_COLLECTOR_BACKEND_SERVICES` | Set agent collector backend services for skywalking | `127.0.0.1:11800` |
| `common.configmap.SW_GRPC_LOG_SERVER_HOST` | Set grpc log server host for skywalking | `127.0.0.1` |
| `common.configmap.SW_GRPC_LOG_SERVER_PORT` | Set grpc log server port for skywalking | `11800` |
| `common.configmap.HADOOP_HOME` | Set `HADOOP_HOME` for DolphinScheduler's task environment | `/opt/soft/hadoop` |
| `common.configmap.HADOOP_CONF_DIR` | Set `HADOOP_CONF_DIR` for DolphinScheduler's task environment | `/opt/soft/hadoop/etc/hadoop` |
| `common.configmap.SPARK_HOME1` | Set `SPARK_HOME1` for DolphinScheduler's task environment | `/opt/soft/spark1` |
| `common.configmap.SPARK_HOME2` | Set `SPARK_HOME2` for DolphinScheduler's task environment | `/opt/soft/spark2` |
| `common.configmap.PYTHON_HOME` | Set `PYTHON_HOME` for DolphinScheduler's task environment | `/usr/bin/python` |
| `common.configmap.JAVA_HOME` | Set `JAVA_HOME` for DolphinScheduler's task environment | `/usr/local/openjdk-8` |
| `common.configmap.HIVE_HOME` | Set `HIVE_HOME` for DolphinScheduler's task environment | `/opt/soft/hive` |
| `common.configmap.FLINK_HOME` | Set `FLINK_HOME` for DolphinScheduler's task environment | `/opt/soft/flink` |
| `common.configmap.DATAX_HOME` | Set `DATAX_HOME` for DolphinScheduler's task environment | `/opt/soft/datax` |
| `common.sharedStoragePersistence.enabled` | Set `common.sharedStoragePersistence.enabled` to `true` to mount a shared storage volume for Hadoop, Spark binary and etc | `false` |
| `common.sharedStoragePersistence.mountPath` | The mount path for the shared storage volume | `/opt/soft` |
| `common.sharedStoragePersistence.accessModes` | `PersistentVolumeClaim` access modes, must be `ReadWriteMany` | `[ReadWriteMany]` |
| `common.sharedStoragePersistence.storageClassName` | Shared Storage persistent volume storage class, must support the access mode: ReadWriteMany | `-` |
| `common.sharedStoragePersistence.storage` | `PersistentVolumeClaim` size | `20Gi` |
| `common.fsFileResourcePersistence.enabled` | Set `common.fsFileResourcePersistence.enabled` to `true` to mount a new file resource volume for `api` and `worker` | `false` |
| `common.fsFileResourcePersistence.accessModes` | `PersistentVolumeClaim` access modes, must be `ReadWriteMany` | `[ReadWriteMany]` |
| `common.fsFileResourcePersistence.storageClassName` | Resource persistent volume storage class, must support the access mode: ReadWriteMany | `-` |
| `common.fsFileResourcePersistence.storage` | `PersistentVolumeClaim` size | `20Gi` |
| | | |
| `master.podManagementPolicy` | PodManagementPolicy controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down | `Parallel` |
| `master.replicas` | Replicas is the desired number of replicas of the given Template | `3` |
| `master.annotations` | The `annotations` for master server | `{}` |
| `master.affinity` | If specified, the pod's scheduling constraints | `{}` |
| `master.nodeSelector` | NodeSelector is a selector which must be true for the pod to fit on a node | `{}` |
| `master.tolerations` | If specified, the pod's tolerations | `{}` |
| `master.resources` | The `resource` limit and request config for master server | `{}` |
| `master.configmap.MASTER_SERVER_OPTS` | The jvm options for master server | `-Xms1g -Xmx1g -Xmn512m` |
| `master.configmap.MASTER_EXEC_THREADS` | Master execute thread number to limit process instances | `100` |
| `master.configmap.MASTER_EXEC_TASK_NUM` | Master execute task number in parallel per process instance | `20` |
| `master.configmap.MASTER_DISPATCH_TASK_NUM` | Master dispatch task number per batch | `3` |
| `master.configmap.MASTER_HOST_SELECTOR` | Master host selector to select a suitable worker, optional values include Random, RoundRobin, LowerWeight | `LowerWeight` |
| `master.configmap.MASTER_HEARTBEAT_INTERVAL` | Master heartbeat interval, the unit is second | `10` |
| `master.configmap.MASTER_TASK_COMMIT_RETRYTIMES` | Master commit task retry times | `5` |
| `master.configmap.MASTER_TASK_COMMIT_INTERVAL` | master commit task interval, the unit is second | `1` |
| `master.configmap.MASTER_MAX_CPULOAD_AVG` | Master max cpuload avg, only higher than the system cpu load average, master server can schedule | `-1` (`the number of cpu cores * 2`) |
| `master.configmap.MASTER_RESERVED_MEMORY` | Master reserved memory, only lower than system available memory, master server can schedule, the unit is G | `0.3` |
| `master.livenessProbe.enabled` | Turn on and off liveness probe | `true` |
| `master.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | `30` |
| `master.livenessProbe.periodSeconds` | How often to perform the probe | `30` |
| `master.livenessProbe.timeoutSeconds` | When the probe times out | `5` |
| `master.livenessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
| `master.livenessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
| `master.readinessProbe.enabled` | Turn on and off readiness probe | `true` |
| `master.readinessProbe.initialDelaySeconds` | Delay before readiness probe is initiated | `30` |
| `master.readinessProbe.periodSeconds` | How often to perform the probe | `30` |
| `master.readinessProbe.timeoutSeconds` | When the probe times out | `5` |
| `master.readinessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
| `master.readinessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
| `master.persistentVolumeClaim.enabled` | Set `master.persistentVolumeClaim.enabled` to `true` to mount a new volume for `master` | `false` |
| `master.persistentVolumeClaim.accessModes` | `PersistentVolumeClaim` access modes | `[ReadWriteOnce]` |
| `master.persistentVolumeClaim.storageClassName` | `Master` logs data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
| `master.persistentVolumeClaim.storage` | `PersistentVolumeClaim` size | `20Gi` |
| | | |
| `worker.podManagementPolicy` | PodManagementPolicy controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down | `Parallel` |
| `worker.replicas` | Replicas is the desired number of replicas of the given Template | `3` |
| `worker.annotations` | The `annotations` for worker server | `{}` |
| `worker.affinity` | If specified, the pod's scheduling constraints | `{}` |
| `worker.nodeSelector` | NodeSelector is a selector which must be true for the pod to fit on a node | `{}` |
| `worker.tolerations` | If specified, the pod's tolerations | `{}` |
| `worker.resources` | The `resource` limit and request config for worker server | `{}` |
| `worker.configmap.WORKER_SERVER_OPTS` | The jvm options for worker server | `-Xms1g -Xmx1g -Xmn512m` |
| `worker.configmap.WORKER_EXEC_THREADS` | Worker execute thread number to limit task instances | `100` |
| `worker.configmap.WORKER_HEARTBEAT_INTERVAL` | Worker heartbeat interval, the unit is second | `10` |
| `worker.configmap.WORKER_MAX_CPULOAD_AVG` | Worker max cpuload avg, only higher than the system cpu load average, worker server can be dispatched tasks | `-1` (`the number of cpu cores * 2`) |
| `worker.configmap.WORKER_RESERVED_MEMORY` | Worker reserved memory, only lower than system available memory, worker server can be dispatched tasks, the unit is G | `0.3` |
| `worker.configmap.WORKER_GROUPS` | Worker groups | `default` |
| `worker.livenessProbe.enabled` | Turn on and off liveness probe | `true` |
| `worker.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | `30` |
| `worker.livenessProbe.periodSeconds` | How often to perform the probe | `30` |
| `worker.livenessProbe.timeoutSeconds` | When the probe times out | `5` |
| `worker.livenessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
| `worker.livenessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
| `worker.readinessProbe.enabled` | Turn on and off readiness probe | `true` |
| `worker.readinessProbe.initialDelaySeconds` | Delay before readiness probe is initiated | `30` |
| `worker.readinessProbe.periodSeconds` | How often to perform the probe | `30` |
| `worker.readinessProbe.timeoutSeconds` | When the probe times out | `5` |
| `worker.readinessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
| `worker.readinessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
| `worker.persistentVolumeClaim.enabled` | Set `worker.persistentVolumeClaim.enabled` to `true` to enable `persistentVolumeClaim` for `worker` | `false` |
| `worker.persistentVolumeClaim.dataPersistentVolume.enabled` | Set `worker.persistentVolumeClaim.dataPersistentVolume.enabled` to `true` to mount a data volume for `worker` | `false` |
| `worker.persistentVolumeClaim.dataPersistentVolume.accessModes` | `PersistentVolumeClaim` access modes | `[ReadWriteOnce]` |
| `worker.persistentVolumeClaim.dataPersistentVolume.storageClassName` | `Worker` data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
| `worker.persistentVolumeClaim.dataPersistentVolume.storage` | `PersistentVolumeClaim` size | `20Gi` |
| `worker.persistentVolumeClaim.logsPersistentVolume.enabled` | Set `worker.persistentVolumeClaim.logsPersistentVolume.enabled` to `true` to mount a logs volume for `worker` | `false` |
| `worker.persistentVolumeClaim.logsPersistentVolume.accessModes` | `PersistentVolumeClaim` access modes | `[ReadWriteOnce]` |
| `worker.persistentVolumeClaim.logsPersistentVolume.storageClassName` | `Worker` logs data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
| `worker.persistentVolumeClaim.logsPersistentVolume.storage` | `PersistentVolumeClaim` size | `20Gi` |
| | | |
| `alert.replicas` | Replicas is the desired number of replicas of the given Template | `1` |
| `alert.strategy.type` | Type of deployment. Can be "Recreate" or "RollingUpdate" | `RollingUpdate` |
| `alert.strategy.rollingUpdate.maxSurge` | The maximum number of pods that can be scheduled above the desired number of pods | `25%` |
| `alert.strategy.rollingUpdate.maxUnavailable` | The maximum number of pods that can be unavailable during the update | `25%` |
| `alert.annotations` | The `annotations` for alert server | `{}` |
| `alert.affinity` | If specified, the pod's scheduling constraints | `{}` |
| `alert.nodeSelector` | NodeSelector is a selector which must be true for the pod to fit on a node | `{}` |
| `alert.tolerations` | If specified, the pod's tolerations | `{}` |
| `alert.resources` | The `resource` limit and request config for alert server | `{}` |
| `alert.configmap.ALERT_SERVER_OPTS` | The jvm options for alert server | `-Xms512m -Xmx512m -Xmn256m` |
| `alert.configmap.XLS_FILE_PATH` | XLS file path | `/tmp/xls` |
| `alert.configmap.MAIL_SERVER_HOST` | Mail `SERVER HOST ` | `nil` |
| `alert.configmap.MAIL_SERVER_PORT` | Mail `SERVER PORT` | `nil` |
| `alert.configmap.MAIL_SENDER` | Mail `SENDER` | `nil` |
| `alert.configmap.MAIL_USER` | Mail `USER` | `nil` |
| `alert.configmap.MAIL_PASSWD` | Mail `PASSWORD` | `nil` |
| `alert.configmap.MAIL_SMTP_STARTTLS_ENABLE` | Mail `SMTP STARTTLS` enable | `false` |
| `alert.configmap.MAIL_SMTP_SSL_ENABLE` | Mail `SMTP SSL` enable | `false` |
| `alert.configmap.MAIL_SMTP_SSL_TRUST` | Mail `SMTP SSL TRUST` | `nil` |
| `alert.configmap.ENTERPRISE_WECHAT_ENABLE` | `Enterprise Wechat` enable | `false` |
| `alert.configmap.ENTERPRISE_WECHAT_CORP_ID` | `Enterprise Wechat` corp id | `nil` |
| `alert.configmap.ENTERPRISE_WECHAT_SECRET` | `Enterprise Wechat` secret | `nil` |
| `alert.configmap.ENTERPRISE_WECHAT_AGENT_ID` | `Enterprise Wechat` agent id | `nil` |
| `alert.configmap.ENTERPRISE_WECHAT_USERS` | `Enterprise Wechat` users | `nil` |
| `alert.livenessProbe.enabled` | Turn on and off liveness probe | `true` |
| `alert.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | `30` |
| `alert.livenessProbe.periodSeconds` | How often to perform the probe | `30` |
| `alert.livenessProbe.timeoutSeconds` | When the probe times out | `5` |
| `alert.livenessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
| `alert.livenessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
| `alert.readinessProbe.enabled` | Turn on and off readiness probe | `true` |
| `alert.readinessProbe.initialDelaySeconds` | Delay before readiness probe is initiated | `30` |
| `alert.readinessProbe.periodSeconds` | How often to perform the probe | `30` |
| `alert.readinessProbe.timeoutSeconds` | When the probe times out | `5` |
| `alert.readinessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
| `alert.readinessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
| `alert.persistentVolumeClaim.enabled` | Set `alert.persistentVolumeClaim.enabled` to `true` to mount a new volume for `alert` | `false` |
| `alert.persistentVolumeClaim.accessModes` | `PersistentVolumeClaim` access modes | `[ReadWriteOnce]` |
| `alert.persistentVolumeClaim.storageClassName` | `Alert` logs data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
| `alert.persistentVolumeClaim.storage` | `PersistentVolumeClaim` size | `20Gi` |
| | | |
| `api.replicas` | Replicas is the desired number of replicas of the given Template | `1` |
| `api.strategy.type` | Type of deployment. Can be "Recreate" or "RollingUpdate" | `RollingUpdate` |
| `api.strategy.rollingUpdate.maxSurge` | The maximum number of pods that can be scheduled above the desired number of pods | `25%` |
| `api.strategy.rollingUpdate.maxUnavailable` | The maximum number of pods that can be unavailable during the update | `25%` |
| `api.annotations` | The `annotations` for api server | `{}` |
| `api.affinity` | If specified, the pod's scheduling constraints | `{}` |
| `api.nodeSelector` | NodeSelector is a selector which must be true for the pod to fit on a node | `{}` |
| `api.tolerations` | If specified, the pod's tolerations | `{}` |
| `api.resources` | The `resource` limit and request config for api server | `{}` |
| `api.configmap.API_SERVER_OPTS` | The jvm options for api server | `-Xms512m -Xmx512m -Xmn256m` |
| `api.livenessProbe.enabled` | Turn on and off liveness probe | `true` |
| `api.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | `30` |
| `api.livenessProbe.periodSeconds` | How often to perform the probe | `30` |
| `api.livenessProbe.timeoutSeconds` | When the probe times out | `5` |
| `api.livenessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
| `api.livenessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
| `api.readinessProbe.enabled` | Turn on and off readiness probe | `true` |
| `api.readinessProbe.initialDelaySeconds` | Delay before readiness probe is initiated | `30` |
| `api.readinessProbe.periodSeconds` | How often to perform the probe | `30` |
| `api.readinessProbe.timeoutSeconds` | When the probe times out | `5` |
| `api.readinessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
| `api.readinessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
| `api.persistentVolumeClaim.enabled` | Set `api.persistentVolumeClaim.enabled` to `true` to mount a new volume for `api` | `false` |
| `api.persistentVolumeClaim.accessModes` | `PersistentVolumeClaim` access modes | `[ReadWriteOnce]` |
| `api.persistentVolumeClaim.storageClassName` | `api` logs data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
| `api.persistentVolumeClaim.storage` | `PersistentVolumeClaim` size | `20Gi` |
| `api.service.type` | `type` determines how the Service is exposed. Valid options are ExternalName, ClusterIP, NodePort, and LoadBalancer | `ClusterIP` |
| `api.service.clusterIP` | `clusterIP` is the IP address of the service and is usually assigned randomly by the master | `nil` |
| `api.service.nodePort` | `nodePort` is the port on each node on which this service is exposed when type=NodePort | `nil` |
| `api.service.externalIPs` | `externalIPs` is a list of IP addresses for which nodes in the cluster will also accept traffic for this service | `[]` |
| `api.service.externalName` | `externalName` is the external reference that kubedns or equivalent will return as a CNAME record for this service | `nil` |
| `api.service.loadBalancerIP` | `loadBalancerIP` when service.type is LoadBalancer. LoadBalancer will get created with the IP specified in this field | `nil` |
| `api.service.annotations` | `annotations` may need to be set when service.type is LoadBalancer | `{}` |
| | | |
| `ingress.enabled` | Enable ingress | `false` |
| `ingress.host` | Ingress host | `dolphinscheduler.org` |
| `ingress.path` | Ingress path | `/dolphinscheduler` |
| `ingress.tls.enabled` | Enable ingress tls | `false` |
| `ingress.tls.secretName` | Ingress tls secret name | `dolphinscheduler-tls` |
| Parameter | Description | Default |
|----------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|---------------------------------------|
| `timezone` | World time and date for cities in all time zones | `Asia/Shanghai` |
| | | |
| `image.repository` | Docker image repository for the DolphinScheduler | `apache/dolphinscheduler` |
| `image.tag` | Docker image version for the DolphinScheduler | `latest` |
| `image.pullPolicy` | Image pull policy. Options: Always, Never, IfNotPresent | `IfNotPresent` |
| `image.pullSecret` | Image pull secret. An optional reference to secret in the same namespace to use for pulling any of the images | `nil` |
| | | |
| `postgresql.enabled` | If not exists external PostgreSQL, by default, the DolphinScheduler will use a internal PostgreSQL | `true` |
| `postgresql.postgresqlUsername` | The username for internal PostgreSQL | `root` |
| `postgresql.postgresqlPassword` | The password for internal PostgreSQL | `root` |
| `postgresql.postgresqlDatabase` | The database for internal PostgreSQL | `dolphinscheduler` |
| `postgresql.persistence.enabled` | Set `postgresql.persistence.enabled` to `true` to mount a new volume for internal PostgreSQL | `false` |
| `postgresql.persistence.size` | `PersistentVolumeClaim` size | `20Gi` |
| `postgresql.persistence.storageClass` | PostgreSQL data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
| `externalDatabase.type` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database type will use it | `postgresql` |
| `externalDatabase.driver` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database driver will use it | `org.postgresql.Driver` |
| `externalDatabase.host` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database host will use it | `localhost` |
| `externalDatabase.port` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database port will use it | `5432` |
| `externalDatabase.username` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database username will use it | `root` |
| `externalDatabase.password` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database password will use it | `root` |
| `externalDatabase.database` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database database will use it | `dolphinscheduler` |
| `externalDatabase.params` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database params will use it | `characterEncoding=utf8` |
| | | |
| `zookeeper.enabled` | If not exists external ZooKeeper, by default, the DolphinScheduler will use a internal ZooKeeper | `true` |
| `zookeeper.service.port` | The port of zookeeper | `2181` |
| `zookeeper.fourlwCommandsWhitelist` | A list of comma separated Four Letter Words commands to use | `srvr,ruok,wchs,cons` |
| `zookeeper.persistence.enabled` | Set `zookeeper.persistence.enabled` to `true` to mount a new volume for internal ZooKeeper | `false` |
| `zookeeper.persistence.size` | `PersistentVolumeClaim` size | `20Gi` |
| `zookeeper.persistence.storageClass` | ZooKeeper data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
| `externalRegistry.registryPluginDir` | If exists external registry and set `zookeeper.enable` to `false`, specify the external registry plugin directory | `lib/plugin/registry` |
| `externalRegistry.registryPluginName` | If exists external registry and set `zookeeper.enable` to `false`, specify the external registry plugin name | `zookeeper` |
| `externalRegistry.registryServers` | If exists external registry and set `zookeeper.enable` to `false`, specify the external registry servers | `127.0.0.1:2181` |
| | | |
| `common.configmap.DOLPHINSCHEDULER_OPTS` | The jvm options for dolphinscheduler, suitable for all servers | `""` |
| `common.configmap.DATA_BASEDIR_PATH` | User data directory path, self configuration, please make sure the directory exists and have read write permissions | `/tmp/dolphinscheduler` |
| `common.configmap.RESOURCE_STORAGE_TYPE` | Resource storage type: HDFS, S3, NONE | `HDFS` |
| `common.configmap.RESOURCE_UPLOAD_PATH` | Resource store on HDFS/S3 path, please make sure the directory exists on hdfs and have read write permissions | `/dolphinscheduler` |
| `common.configmap.FS_DEFAULT_FS` | Resource storage file system like `file:///`, `hdfs://mycluster:8020` or `s3a://dolphinscheduler` | `file:///` |
| `common.configmap.FS_S3A_ENDPOINT` | S3 endpoint when `common.configmap.RESOURCE_STORAGE_TYPE` is set to `S3` | `s3.xxx.amazonaws.com` |
| `common.configmap.FS_S3A_ACCESS_KEY` | S3 access key when `common.configmap.RESOURCE_STORAGE_TYPE` is set to `S3` | `xxxxxxx` |
| `common.configmap.FS_S3A_SECRET_KEY` | S3 secret key when `common.configmap.RESOURCE_STORAGE_TYPE` is set to `S3` | `xxxxxxx` |
| `common.configmap.HADOOP_SECURITY_AUTHENTICATION_STARTUP_STATE` | Whether to startup kerberos | `false` |
| `common.configmap.JAVA_SECURITY_KRB5_CONF_PATH` | The java.security.krb5.conf path | `/opt/krb5.conf` |
| `common.configmap.LOGIN_USER_KEYTAB_USERNAME` | The login user from keytab username | `hdfs@HADOOP.COM` |
| `common.configmap.LOGIN_USER_KEYTAB_PATH` | The login user from keytab path | `/opt/hdfs.keytab` |
| `common.configmap.KERBEROS_EXPIRE_TIME` | The kerberos expire time, the unit is hour | `2` |
| `common.configmap.HDFS_ROOT_USER` | The HDFS root user who must have the permission to create directories under the HDFS root path | `hdfs` |
| `common.configmap.RESOURCE_MANAGER_HTTPADDRESS_PORT` | Set resource manager httpaddress port for yarn | `8088` |
| `common.configmap.YARN_RESOURCEMANAGER_HA_RM_IDS` | If resourcemanager HA is enabled, please set the HA IPs | `nil` |
| `common.configmap.YARN_APPLICATION_STATUS_ADDRESS` | If resourcemanager is single, you only need to replace ds1 to actual resourcemanager hostname, otherwise keep default | `http://ds1:%s/ws/v1/cluster/apps/%s` |
| `common.configmap.SKYWALKING_ENABLE` | Set whether to enable skywalking | `false` |
| `common.configmap.SW_AGENT_COLLECTOR_BACKEND_SERVICES` | Set agent collector backend services for skywalking | `127.0.0.1:11800` |
| `common.configmap.SW_GRPC_LOG_SERVER_HOST` | Set grpc log server host for skywalking | `127.0.0.1` |
| `common.configmap.SW_GRPC_LOG_SERVER_PORT` | Set grpc log server port for skywalking | `11800` |
| `common.configmap.HADOOP_HOME` | Set `HADOOP_HOME` for DolphinScheduler's task environment | `/opt/soft/hadoop` |
| `common.configmap.HADOOP_CONF_DIR` | Set `HADOOP_CONF_DIR` for DolphinScheduler's task environment | `/opt/soft/hadoop/etc/hadoop` |
| `common.configmap.SPARK_HOME1` | Set `SPARK_HOME1` for DolphinScheduler's task environment | `/opt/soft/spark1` |
| `common.configmap.SPARK_HOME2` | Set `SPARK_HOME2` for DolphinScheduler's task environment | `/opt/soft/spark2` |
| `common.configmap.PYTHON_HOME` | Set `PYTHON_HOME` for DolphinScheduler's task environment | `/usr/bin/python` |
| `common.configmap.JAVA_HOME` | Set `JAVA_HOME` for DolphinScheduler's task environment | `/usr/local/openjdk-8` |
| `common.configmap.HIVE_HOME` | Set `HIVE_HOME` for DolphinScheduler's task environment | `/opt/soft/hive` |
| `common.configmap.FLINK_HOME` | Set `FLINK_HOME` for DolphinScheduler's task environment | `/opt/soft/flink` |
| `common.configmap.DATAX_HOME` | Set `DATAX_HOME` for DolphinScheduler's task environment | `/opt/soft/datax` |
| `common.sharedStoragePersistence.enabled` | Set `common.sharedStoragePersistence.enabled` to `true` to mount a shared storage volume for Hadoop, Spark binary and etc | `false` |
| `common.sharedStoragePersistence.mountPath` | The mount path for the shared storage volume | `/opt/soft` |
| `common.sharedStoragePersistence.accessModes` | `PersistentVolumeClaim` access modes, must be `ReadWriteMany` | `[ReadWriteMany]` |
| `common.sharedStoragePersistence.storageClassName` | Shared Storage persistent volume storage class, must support the access mode: ReadWriteMany | `-` |
| `common.sharedStoragePersistence.storage` | `PersistentVolumeClaim` size | `20Gi` |
| `common.fsFileResourcePersistence.enabled` | Set `common.fsFileResourcePersistence.enabled` to `true` to mount a new file resource volume for `api` and `worker` | `false` |
| `common.fsFileResourcePersistence.accessModes` | `PersistentVolumeClaim` access modes, must be `ReadWriteMany` | `[ReadWriteMany]` |
| `common.fsFileResourcePersistence.storageClassName` | Resource persistent volume storage class, must support the access mode: ReadWriteMany | `-` |
| `common.fsFileResourcePersistence.storage` | `PersistentVolumeClaim` size | `20Gi` |
| | | |
| `master.podManagementPolicy` | PodManagementPolicy controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down | `Parallel` |
| `master.replicas` | Replicas is the desired number of replicas of the given Template | `3` |
| `master.annotations` | The `annotations` for master server | `{}` |
| `master.affinity` | If specified, the pod's scheduling constraints | `{}` |
| `master.nodeSelector` | NodeSelector is a selector which must be true for the pod to fit on a node | `{}` |
| `master.tolerations` | If specified, the pod's tolerations | `{}` |
| `master.resources` | The `resource` limit and request config for master server | `{}` |
| `master.configmap.MASTER_SERVER_OPTS` | The jvm options for master server | `-Xms1g -Xmx1g -Xmn512m` |
| `master.configmap.MASTER_EXEC_THREADS` | Master execute thread number to limit process instances | `100` |
| `master.configmap.MASTER_EXEC_TASK_NUM` | Master execute task number in parallel per process instance | `20` |
| `master.configmap.MASTER_DISPATCH_TASK_NUM` | Master dispatch task number per batch | `3` |
| `master.configmap.MASTER_HOST_SELECTOR` | Master host selector to select a suitable worker, optional values include Random, RoundRobin, LowerWeight | `LowerWeight` |
| `master.configmap.MASTER_HEARTBEAT_INTERVAL` | Master heartbeat interval, the unit is second | `10` |
| `master.configmap.MASTER_TASK_COMMIT_RETRYTIMES` | Master commit task retry times | `5` |
| `master.configmap.MASTER_TASK_COMMIT_INTERVAL` | master commit task interval, the unit is second | `1` |
| `master.configmap.MASTER_MAX_CPULOAD_AVG` | Master max cpuload avg, only higher than the system cpu load average, master server can schedule | `-1` (`the number of cpu cores * 2`) |
| `master.configmap.MASTER_RESERVED_MEMORY` | Master reserved memory, only lower than system available memory, master server can schedule, the unit is G | `0.3` |
| `master.livenessProbe.enabled` | Turn on and off liveness probe | `true` |
| `master.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | `30` |
| `master.livenessProbe.periodSeconds` | How often to perform the probe | `30` |
| `master.livenessProbe.timeoutSeconds` | When the probe times out | `5` |
| `master.livenessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
| `master.livenessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
| `master.readinessProbe.enabled` | Turn on and off readiness probe | `true` |
| `master.readinessProbe.initialDelaySeconds` | Delay before readiness probe is initiated | `30` |
| `master.readinessProbe.periodSeconds` | How often to perform the probe | `30` |
| `master.readinessProbe.timeoutSeconds` | When the probe times out | `5` |
| `master.readinessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
| `master.readinessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
| `master.persistentVolumeClaim.enabled` | Set `master.persistentVolumeClaim.enabled` to `true` to mount a new volume for `master` | `false` |
| `master.persistentVolumeClaim.accessModes` | `PersistentVolumeClaim` access modes | `[ReadWriteOnce]` |
| `master.persistentVolumeClaim.storageClassName` | `Master` logs data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
| `master.persistentVolumeClaim.storage` | `PersistentVolumeClaim` size | `20Gi` |
| | | |
| `worker.podManagementPolicy` | PodManagementPolicy controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down | `Parallel` |
| `worker.replicas` | Replicas is the desired number of replicas of the given Template | `3` |
| `worker.annotations` | The `annotations` for worker server | `{}` |
| `worker.affinity` | If specified, the pod's scheduling constraints | `{}` |
| `worker.nodeSelector` | NodeSelector is a selector which must be true for the pod to fit on a node | `{}` |
| `worker.tolerations` | If specified, the pod's tolerations | `{}` |
| `worker.resources` | The `resource` limit and request config for worker server | `{}` |
| `worker.configmap.WORKER_SERVER_OPTS` | The jvm options for worker server | `-Xms1g -Xmx1g -Xmn512m` |
| `worker.configmap.WORKER_EXEC_THREADS` | Worker execute thread number to limit task instances | `100` |
| `worker.configmap.WORKER_HEARTBEAT_INTERVAL` | Worker heartbeat interval, the unit is second | `10` |
| `worker.configmap.WORKER_MAX_CPULOAD_AVG` | Worker max cpuload avg, only higher than the system cpu load average, worker server can be dispatched tasks | `-1` (`the number of cpu cores * 2`) |
| `worker.configmap.WORKER_RESERVED_MEMORY` | Worker reserved memory, only lower than system available memory, worker server can be dispatched tasks, the unit is G | `0.3` |
| `worker.configmap.WORKER_GROUPS` | Worker groups | `default` |
| `worker.livenessProbe.enabled` | Turn on and off liveness probe | `true` |
| `worker.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | `30` |
| `worker.livenessProbe.periodSeconds` | How often to perform the probe | `30` |
| `worker.livenessProbe.timeoutSeconds` | When the probe times out | `5` |
| `worker.livenessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
| `worker.livenessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
| `worker.readinessProbe.enabled` | Turn on and off readiness probe | `true` |
| `worker.readinessProbe.initialDelaySeconds` | Delay before readiness probe is initiated | `30` |
| `worker.readinessProbe.periodSeconds` | How often to perform the probe | `30` |
| `worker.readinessProbe.timeoutSeconds` | When the probe times out | `5` |
| `worker.readinessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
| `worker.readinessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
| `worker.persistentVolumeClaim.enabled` | Set `worker.persistentVolumeClaim.enabled` to `true` to enable `persistentVolumeClaim` for `worker` | `false` |
| `worker.persistentVolumeClaim.dataPersistentVolume.enabled` | Set `worker.persistentVolumeClaim.dataPersistentVolume.enabled` to `true` to mount a data volume for `worker` | `false` |
| `worker.persistentVolumeClaim.dataPersistentVolume.accessModes` | `PersistentVolumeClaim` access modes | `[ReadWriteOnce]` |
| `worker.persistentVolumeClaim.dataPersistentVolume.storageClassName` | `Worker` data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
| `worker.persistentVolumeClaim.dataPersistentVolume.storage` | `PersistentVolumeClaim` size | `20Gi` |
| `worker.persistentVolumeClaim.logsPersistentVolume.enabled` | Set `worker.persistentVolumeClaim.logsPersistentVolume.enabled` to `true` to mount a logs volume for `worker` | `false` |
| `worker.persistentVolumeClaim.logsPersistentVolume.accessModes` | `PersistentVolumeClaim` access modes | `[ReadWriteOnce]` |
| `worker.persistentVolumeClaim.logsPersistentVolume.storageClassName` | `Worker` logs data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
| `worker.persistentVolumeClaim.logsPersistentVolume.storage` | `PersistentVolumeClaim` size | `20Gi` |
| | | |
| `alert.replicas` | Replicas is the desired number of replicas of the given Template | `1` |
| `alert.strategy.type` | Type of deployment. Can be "Recreate" or "RollingUpdate" | `RollingUpdate` |
| `alert.strategy.rollingUpdate.maxSurge` | The maximum number of pods that can be scheduled above the desired number of pods | `25%` |
| `alert.strategy.rollingUpdate.maxUnavailable` | The maximum number of pods that can be unavailable during the update | `25%` |
| `alert.annotations` | The `annotations` for alert server | `{}` |
| `alert.affinity` | If specified, the pod's scheduling constraints | `{}` |
| `alert.nodeSelector` | NodeSelector is a selector which must be true for the pod to fit on a node | `{}` |
| `alert.tolerations` | If specified, the pod's tolerations | `{}` |
| `alert.resources` | The `resource` limit and request config for alert server | `{}` |
| `alert.configmap.ALERT_SERVER_OPTS` | The jvm options for alert server | `-Xms512m -Xmx512m -Xmn256m` |
| `alert.configmap.XLS_FILE_PATH` | XLS file path | `/tmp/xls` |
| `alert.configmap.MAIL_SERVER_HOST` | Mail `SERVER HOST ` | `nil` |
| `alert.configmap.MAIL_SERVER_PORT` | Mail `SERVER PORT` | `nil` |
| `alert.configmap.MAIL_SENDER` | Mail `SENDER` | `nil` |
| `alert.configmap.MAIL_USER` | Mail `USER` | `nil` |
| `alert.configmap.MAIL_PASSWD` | Mail `PASSWORD` | `nil` |
| `alert.configmap.MAIL_SMTP_STARTTLS_ENABLE` | Mail `SMTP STARTTLS` enable | `false` |
| `alert.configmap.MAIL_SMTP_SSL_ENABLE` | Mail `SMTP SSL` enable | `false` |
| `alert.configmap.MAIL_SMTP_SSL_TRUST` | Mail `SMTP SSL TRUST` | `nil` |
| `alert.configmap.ENTERPRISE_WECHAT_ENABLE` | `Enterprise Wechat` enable | `false` |
| `alert.configmap.ENTERPRISE_WECHAT_CORP_ID` | `Enterprise Wechat` corp id | `nil` |
| `alert.configmap.ENTERPRISE_WECHAT_SECRET` | `Enterprise Wechat` secret | `nil` |
| `alert.configmap.ENTERPRISE_WECHAT_AGENT_ID` | `Enterprise Wechat` agent id | `nil` |
| `alert.configmap.ENTERPRISE_WECHAT_USERS` | `Enterprise Wechat` users | `nil` |
| `alert.livenessProbe.enabled` | Turn on and off liveness probe | `true` |
| `alert.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | `30` |
| `alert.livenessProbe.periodSeconds` | How often to perform the probe | `30` |
| `alert.livenessProbe.timeoutSeconds` | When the probe times out | `5` |
| `alert.livenessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
| `alert.livenessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
| `alert.readinessProbe.enabled` | Turn on and off readiness probe | `true` |
| `alert.readinessProbe.initialDelaySeconds` | Delay before readiness probe is initiated | `30` |
| `alert.readinessProbe.periodSeconds` | How often to perform the probe | `30` |
| `alert.readinessProbe.timeoutSeconds` | When the probe times out | `5` |
| `alert.readinessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
| `alert.readinessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
| `alert.persistentVolumeClaim.enabled` | Set `alert.persistentVolumeClaim.enabled` to `true` to mount a new volume for `alert` | `false` |
| `alert.persistentVolumeClaim.accessModes` | `PersistentVolumeClaim` access modes | `[ReadWriteOnce]` |
| `alert.persistentVolumeClaim.storageClassName` | `Alert` logs data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
| `alert.persistentVolumeClaim.storage` | `PersistentVolumeClaim` size | `20Gi` |
| | | |
| `api.replicas` | Replicas is the desired number of replicas of the given Template | `1` |
| `api.strategy.type` | Type of deployment. Can be "Recreate" or "RollingUpdate" | `RollingUpdate` |
| `api.strategy.rollingUpdate.maxSurge` | The maximum number of pods that can be scheduled above the desired number of pods | `25%` |
| `api.strategy.rollingUpdate.maxUnavailable` | The maximum number of pods that can be unavailable during the update | `25%` |
| `api.annotations` | The `annotations` for api server | `{}` |
| `api.affinity` | If specified, the pod's scheduling constraints | `{}` |
| `api.nodeSelector` | NodeSelector is a selector which must be true for the pod to fit on a node | `{}` |
| `api.tolerations` | If specified, the pod's tolerations | `{}` |
| `api.resources` | The `resource` limit and request config for api server | `{}` |
| `api.configmap.API_SERVER_OPTS` | The jvm options for api server | `-Xms512m -Xmx512m -Xmn256m` |
| `api.livenessProbe.enabled` | Turn on and off liveness probe | `true` |
| `api.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | `30` |
| `api.livenessProbe.periodSeconds` | How often to perform the probe | `30` |
| `api.livenessProbe.timeoutSeconds` | When the probe times out | `5` |
| `api.livenessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
| `api.livenessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
| `api.readinessProbe.enabled` | Turn on and off readiness probe | `true` |
| `api.readinessProbe.initialDelaySeconds` | Delay before readiness probe is initiated | `30` |
| `api.readinessProbe.periodSeconds` | How often to perform the probe | `30` |
| `api.readinessProbe.timeoutSeconds` | When the probe times out | `5` |
| `api.readinessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
| `api.readinessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
| `api.persistentVolumeClaim.enabled` | Set `api.persistentVolumeClaim.enabled` to `true` to mount a new volume for `api` | `false` |
| `api.persistentVolumeClaim.accessModes` | `PersistentVolumeClaim` access modes | `[ReadWriteOnce]` |
| `api.persistentVolumeClaim.storageClassName` | `api` logs data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
| `api.persistentVolumeClaim.storage` | `PersistentVolumeClaim` size | `20Gi` |
| `api.service.type` | `type` determines how the Service is exposed. Valid options are ExternalName, ClusterIP, NodePort, and LoadBalancer | `ClusterIP` |
| `api.service.clusterIP` | `clusterIP` is the IP address of the service and is usually assigned randomly by the master | `nil` |
| `api.service.nodePort` | `nodePort` is the port on each node on which this service is exposed when type=NodePort | `nil` |
| `api.service.externalIPs` | `externalIPs` is a list of IP addresses for which nodes in the cluster will also accept traffic for this service | `[]` |
| `api.service.externalName` | `externalName` is the external reference that kubedns or equivalent will return as a CNAME record for this service | `nil` |
| `api.service.loadBalancerIP` | `loadBalancerIP` when service.type is LoadBalancer. LoadBalancer will get created with the IP specified in this field | `nil` |
| `api.service.annotations` | `annotations` may need to be set when service.type is LoadBalancer | `{}` |
| | | |
| `ingress.enabled` | Enable ingress | `false` |
| `ingress.host` | Ingress host | `dolphinscheduler.org` |
| `ingress.path` | Ingress path | `/dolphinscheduler` |
| `ingress.tls.enabled` | Enable ingress tls | `false` |
| `ingress.tls.secretName` | Ingress tls secret name | `dolphinscheduler-tls` |

View File

@ -154,7 +154,7 @@ bash ./bin/install.sh
```
> **_Note:_** For the first time deployment, there maybe occur five times of `sh: bin/dolphinscheduler-daemon.sh: No such file or directory` in the terminal,
this is non-important information that you can ignore.
> this is non-important information that you can ignore.
## Login DolphinScheduler
@ -190,7 +190,7 @@ bash ./bin/dolphinscheduler-daemon.sh stop alert-server
> for micro-services need. It means that you could start all servers by command `<service>/bin/start.sh` with different
> environment variable from `<service>/conf/dolphinscheduler_env.sh`. But it will use file `bin/env/dolphinscheduler_env.sh` overwrite
> `<service>/conf/dolphinscheduler_env.sh` if you start server with command `/bin/dolphinscheduler-daemon.sh start <service>`.
>
> **_Note2:_**: Please refer to the section of "System Architecture Design" for service usage. Python gateway service is
> started along with the api-server, and if you do not want to start Python gateway service please disabled it by changing
> the yaml config `python-gateway.enabled : false` in api-server's configuration path `api-server/conf/application.yaml`
@ -198,3 +198,4 @@ bash ./bin/dolphinscheduler-daemon.sh stop alert-server
[jdk]: https://www.oracle.com/technetwork/java/javase/downloads/index.html
[zookeeper]: https://zookeeper.apache.org/releases.html
[issue]: https://github.com/apache/dolphinscheduler/issues/6597

View File

@ -14,12 +14,12 @@ This section describes the one-click deployment of high availability DolphinSche
* Click `install` on the right side of DolphinScheduler to go to the installation page. Fill in the corresponding information and click `OK` to start the installation. You will get automatically redirected to the application view.
| Select item | Description |
| ------------ | ------------------------------------ |
| Select item | Description |
|--------------|-------------------------------------|
| Team name | user workspaceIsolate by namespace |
| Cluster name | select kubernetes cluster |
| Select app | select application |
| app version | select DolphinScheduler version |
| Cluster name | select kubernetes cluster |
| Select app | select application |
| app version | select DolphinScheduler version |
![](../../../../img/rainbond/install-dolphinscheduler.png)
@ -42,6 +42,7 @@ Take `worker` as an example: enter the `component -> Telescopic` page, and set t
To verify `worker` node, enter `DolphinScheduler UI -> Monitoring -> Worker` page to view detailed node information.
![](../../../../img/rainbond/monitor-dolphinscheduler.png)
## Configuration file
API and Worker Services share the configuration file `/opt/dolphinscheduler/conf/common.properties`. To modify the configurations, you only need to modify that of the API service.
@ -60,5 +61,7 @@ Take `DataX` as an example:
* FILE_PATH/opt/soft
* LOCK_PATH/opt/soft
3. Update component, the plug-in `Datax` will be downloaded automatically and decompress to `/opt/soft`
![](../../../../img/rainbond/plugin.png)
![](../../../../img/rainbond/plugin.png)
---

View File

@ -9,7 +9,7 @@ Currently, we only support `Prometheus Exporter` but more are coming soon.
- After triggering tasks in `standalone` mode, you could access metrics list by visiting url `http://localhost:12345/dolphinscheduler/actuator/metrics`.
- After triggering tasks in `standalone` mode, you could access `prometheus-format` metrics by visiting url `http://localhost:12345/dolphinscheduler/actuator/prometheus`.
- For a better experience with `Prometheus` and `Grafana`, we have prepared the out-of-the-box `Grafana` configurations for you, you could find the `Grafana` dashboards
at `dolphinscheduler-meter/resources/grafana` and directly import these dashboards to your `Grafana` instance.
at `dolphinscheduler-meter/resources/grafana` and directly import these dashboards to your `Grafana` instance.
- If you want to try with `docker`, you can use the following command to start the out-of-the-box `Prometheus` and `Grafana`:
```shell
@ -78,7 +78,6 @@ For example, you can get the master metrics by `curl http://localhost:5679/actua
- ds.task.execution.count: (counter) the number of executed tasks
- ds.task.execution.duration: (histogram) duration of task executions
### Workflow Related Metrics
- ds.workflow.create.command.count: (counter) the number of commands created and inserted by workflows
@ -175,3 +174,4 @@ For example, you can get the master metrics by `curl http://localhost:5679/actua
- system.load.average.1m: the total number of runnable entities queued to available processors and runnable entities running on the available processors averaged over a period
- logback.events: the number of events that made it to the logs grouped by the tag `level`
- http.server.requests: total number of http requests

View File

@ -28,12 +28,12 @@
![statistics](../../../img/new_ui/dev/monitor/statistics.png)
| **Parameter** | **Description** |
| ----- | ----- |
| Number of commands wait to be executed | Statistics of the `t_ds_command` table data. |
| The number of failed commands | Statistics of the `t_ds_error_command` table data. |
| Number of tasks wait to run | Count the data of `task_queue` in the ZooKeeper. |
| Number of tasks wait to be killed | Count the data of `task_kill` in the ZooKeeper. |
| **Parameter** | **Description** |
|----------------------------------------|----------------------------------------------------|
| Number of commands wait to be executed | Statistics of the `t_ds_command` table data. |
| The number of failed commands | Statistics of the `t_ds_error_command` table data. |
| Number of tasks wait to run | Count the data of `task_queue` in the ZooKeeper. |
| Number of tasks wait to be killed | Count the data of `task_kill` in the ZooKeeper. |
### Audit Log

View File

@ -28,13 +28,13 @@ Generally, projects and processes are created through pages, but considering the
2. select a test API, the API selected for this test is `queryAllProjectList`
> projects/list
>
> projects/list
3. Open `Postman`, fill in the API address, enter the `Token` in `Headers`, and then send the request to view the result:
```
token: The Token just generated
```
token: The Token just generated
```
![api-test](../../../img/new_ui/dev/open-api/api_test.png)

View File

@ -2,11 +2,11 @@
## Basic Built-in Parameter
| Variable | Declaration Method | Meaning |
| ---- | ---- | -----------------------------|
| system.biz.date | `${system.biz.date}` | The day before the schedule time of the daily scheduling instance, the format is `yyyyMMdd` |
| system.biz.curdate | `${system.biz.curdate}` | The schedule time of the daily scheduling instance, the format is `yyyyMMdd` |
| system.datetime | `${system.datetime}` | The schedule time of the daily scheduling instance, the format is `yyyyMMddHHmmss` |
| Variable | Declaration Method | Meaning |
|--------------------|-------------------------|---------------------------------------------------------------------------------------------|
| system.biz.date | `${system.biz.date}` | The day before the schedule time of the daily scheduling instance, the format is `yyyyMMdd` |
| system.biz.curdate | `${system.biz.curdate}` | The schedule time of the daily scheduling instance, the format is `yyyyMMdd` |
| system.datetime | `${system.datetime}` | The schedule time of the daily scheduling instance, the format is `yyyyMMddHHmmss` |
## Extended Built-in Parameter
@ -16,19 +16,19 @@
- Or define by the following two ways:
1. Use add_month(yyyyMMdd, offset) function to add or minus number of months.
The first parameter of this function is [yyyyMMdd], represents the time format and the second parameter is offset, represents the number of months the user wants to add or minus.
- Next N years`$[add_months(yyyyMMdd,12*N)]`
- N years before`$[add_months(yyyyMMdd,-12*N)]`
- Next N months`$[add_months(yyyyMMdd,N)]`
- N months before`$[add_months(yyyyMMdd,-N)]`
1. Use add_month(yyyyMMdd, offset) function to add or minus number of months.
The first parameter of this function is [yyyyMMdd], represents the time format and the second parameter is offset, represents the number of months the user wants to add or minus.
- Next N years`$[add_months(yyyyMMdd,12*N)]`
- N years before`$[add_months(yyyyMMdd,-12*N)]`
- Next N months`$[add_months(yyyyMMdd,N)]`
- N months before`$[add_months(yyyyMMdd,-N)]`
2. Add or minus numbers directly after the time format.
- Next N weeks`$[yyyyMMdd+7*N]`
- First N weeks`$[yyyyMMdd-7*N]`
- Next N days`$[yyyyMMdd+N]`
- N days before`$[yyyyMMdd-N]`
- Next N hours`$[HHmmss+N/24]`
- First N hours`$[HHmmss-N/24]`
- Next N minutes`$[HHmmss+N/24/60]`
- First N minutes`$[HHmmss-N/24/60]`
2. Add or minus numbers directly after the time format.
- Next N weeks`$[yyyyMMdd+7*N]`
- First N weeks`$[yyyyMMdd-7*N]`
- Next N days`$[yyyyMMdd+N]`
- N days before`$[yyyyMMdd-N]`
- Next N hours`$[HHmmss+N/24]`
- First N hours`$[HHmmss-N/24]`
- Next N minutes`$[HHmmss+N/24/60]`
- First N minutes`$[HHmmss-N/24/60]`

View File

@ -49,7 +49,7 @@ When the SHELL task is completed, we can use the output passed upstream as the q
> Note: If the result of the SQL node has only one row, one or multiple fields, the name of the `prop` needs to be the same as the field name. The data type can choose structure except `LIST`. The parameter assigns the value according to the same column name in the SQL query result.
>
>If the result of the SQL node has multiple rows, one or more fields, the name of the `prop` needs to be the same as the field name. Choose the data type structure as `LIST`, and the SQL query result will be converted to `LIST<VARCHAR>`, and forward to convert to JSON as the parameter value.
> If the result of the SQL node has multiple rows, one or more fields, the name of the `prop` needs to be the same as the field name. Choose the data type structure as `LIST`, and the SQL query result will be converted to `LIST<VARCHAR>`, and forward to convert to JSON as the parameter value.
#### Save the workflow and set the global parameters

View File

@ -2,14 +2,14 @@
This page describes details regarding Project screen in Apache DolphinScheduler. Here, you will see all the functions which can be handled in this screen. The following table explains commonly used terms in Apache DolphinScheduler:
| Glossary | description |
| ------ |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| DAG | Tasks in a workflow are assembled in form of Directed Acyclic Graph (DAG). A topological traversal is performed from nodes with zero degrees of entry until there are no subsequent nodes. |
| Glossary | description |
|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| DAG | Tasks in a workflow are assembled in form of Directed Acyclic Graph (DAG). A topological traversal is performed from nodes with zero degrees of entry until there are no subsequent nodes. |
| Workflow Definition | Visualization formed by dragging task nodes and establishing task node associations (DAG). |
| Workflow Instance | Instantiation of the workflow definition, which can be generated by manual start or scheduled scheduling. Each time the process definition runs, a workflow instance is generated. |
| Workflow Relation | Shows dynamic status of all the workflows in a project. |
| Task | Task is a discrete action in a Workflow. Apache DolphinScheduler supports SHELL, SQL, SUB_PROCESS (sub-process), PROCEDURE, MR, SPARK, PYTHON, DEPENDENT ( depends), and plans to support dynamic plug-in expansion, (SUB_PROCESS). It is also a separate process definition that can be started and executed separately. |
| Task Instance | Instantiation of the task node in the process definition, which identifies the specific task execution status. |
| Workflow Instance | Instantiation of the workflow definition, which can be generated by manual start or scheduled scheduling. Each time the process definition runs, a workflow instance is generated. |
| Workflow Relation | Shows dynamic status of all the workflows in a project. |
| Task | Task is a discrete action in a Workflow. Apache DolphinScheduler supports SHELL, SQL, SUB_PROCESS (sub-process), PROCEDURE, MR, SPARK, PYTHON, DEPENDENT ( depends), and plans to support dynamic plug-in expansion, (SUB_PROCESS). It is also a separate process definition that can be started and executed separately. |
| Task Instance | Instantiation of the task node in the process definition, which identifies the specific task execution status. |
## Project List

View File

@ -1,6 +1,7 @@
# Task Definition
## Batch Task Definition
Task definition allows to modify or operate tasks at the task level rather than modifying them in the workflow definition.
We already have workflow level task editor in [workflow definition](workflow-definition.md) which you can click the specific
workflow and then edit its task definition. It is depressing when you want to edit the task definition but do not remember
@ -14,6 +15,7 @@ name but forget which workflow it belongs to. It is also supported query by the
`Workflow Name`
## Stream Task Definition
Stream task definitions are created in the workflow definition, and can be modified and executed.
![task-definition](../../../../img/new_ui/dev/project/stream-task-definition.png)

View File

@ -1,6 +1,7 @@
# Task Instance
## Batch Task Instance
### Create Task Instance
Click `Project Management -> Workflow -> Task Instance` to enter the task instance page, as shown in the figure below, click the name of the workflow instance to jump to the DAG diagram of the workflow instance to view the task status.
@ -21,3 +22,4 @@ Click the `View Log` button in the operation column to view the log of the task
- SavePoint: Click the `SavePoint` button in the operation column to do stream task savepoint.
- Stop: Click the `Stop` button in the operation column to stop the stream task.

View File

@ -35,9 +35,10 @@ Click the plus sign on the right of the task node to connect the task; as shown
![workflow-dependent](../../../../img/new_ui/dev/project/workflow-dependent.png)
### Dependencies with stream task
If the DAG contains stream tasks, the relationship between stream tasks is displayed as a dotted line, and the execution of stream tasks will be skipped when the workflow instance is executed.
![workflow-dependent](../../../../img/new_ui/dev/project/workflow-definition-with-stream-task.png)
![workflow-dependent](../../../../img/new_ui/dev/project/workflow-definition-with-stream-task.png)
**Delete dependencies:** Using your mouse to select the connection line, and click the "Delete" icon in the upper right corner <img src= "../../../../img/delete.png" width="35"/>, delete dependencies between tasks.
@ -49,6 +50,17 @@ Click the `Save` button, and the "Set DAG chart name" window pops up, as shown i
![workflow-save](../../../../img/new_ui/dev/project/workflow-save.png)
### Configure workflow (process) execution type
Click the `Save` button and configure `process execution type` in the pop-up window. There are four process execution types:
- `Parallel`: If there are multiple instances of the same workflow definition, execute the instances in parallel.
- `Serial Wait`: If there are multiple instances of the same workflow definition, execute the instances in serial.
- `Serial Discard`: If there are multiple instances of the same workflow definition, discard the later ones and kill the current running ones.
- `Serial Priority`: If there are multiple instances of the same workflow definition, execute the instances according to the priority in serial.
![workflow-execution-type](../../../../img/new_ui/dev/project/workflow-execution-type.png)
## Workflow Definition Operation Function
Click `Project Management -> Workflow -> Workflow Definition` to enter the workflow definition page, as shown below:
@ -59,14 +71,49 @@ Workflow running parameter description:
* **Failure strategy**: When a task node fails to execute, other parallel task nodes need to execute the strategy. "Continue" means: After a task fails, other task nodes execute normally; "End" means: Terminate all tasks being executed, and terminate the entire process.
* **Notification strategy**: When the process ends, send process execution information notification emails according to the process status, including no status, success, failure, success or failure.
* **Process priority**: the priority of process operation, divided into five levels: the highest (HIGHEST), high (HIGH), medium (MEDIUM), low (LOW), the lowest (LOWEST). When the number of master threads is insufficient, processes with higher levels will be executed first in the execution queue, and processes with the same priority will be executed in the order of first-in, first-out.
* **Process priority**: The priority of process execution, there are five different priorities: the highest (HIGHEST), high (HIGH), medium (MEDIUM), low (LOW), the lowest (LOWEST). When the number of master threads is insufficient, processes with higher priorities in the execution queue will run first. Processes with the same priority will run in first-come-first-served fashion.
* **Worker grouping**: This process can only be executed in the specified worker machine group. The default is Default, which can be executed on any worker.
* **Notification Group**: Select Notification Policy||Timeout Alarm||When fault tolerance occurs, process information or emails will be sent to all members in the notification group.
* **Recipient**: Select Notification Policy||Timeout Alarm||When fault tolerance occurs, process information or alarm email will be sent to the recipient list.
* **Cc**: Select Notification Policy||Timeout Alarm||When fault tolerance occurs, the process information or alarm email will be copied to the Cc list.
* **Startup parameters**: Set or override the value of global parameters when starting a new process instance.
* **Complement**: There are 2 modes of serial complement and parallel complement. Serial complement: within the specified time range, perform complements in sequence from the start date to the end date, and generate N process instances in turn; parallel complement: within the specified time range, perform multiple complements at the same time, and generate N process instances at the same time .
* **Complement**: Execute the workflow definition of the specified date, you can select the time range of the supplement (currently only supports the supplement for consecutive days), for example, the data from May 1st to May 10th needs to be supplemented, as shown in the following figure:
* **Complement(Backfill)**: Run workflow for a specified historical period. There are two strategies: serial complement and parallel complement. You could select the time period or fill in it manually in UI.
* Serial complement: Run the workflow from start date to end date according to the time period you set in serial.
![workflow-serial](../../../../img/new_ui/dev/project/workflow-serial.png)
* Parallel complement: Run the workflow from start date to end date according to the time period you set in parallel.
![workflow-parallel](../../../../img/new_ui/dev/project/workflow-parallel.png)
* Parallelism: The max number of workflow instances of the workflow definition you choose for complement.
![workflow-concurrency-from](../../../../img/new_ui/dev/project/workflow-concurrency-from.png)
![workflow-concurrency](../../../../img/new_ui/dev/project/workflow-concurrency.png)
* Mode of dependent: Whether to trigger downstream workflow definition for complement.
![workflow-dependency](../../../../img/new_ui/dev/project/workflow-dependency.png)
* Schedule date
1. Select from pop-up window:
![workflow-pageSelection](../../../../img/new_ui/dev/project/workflow-pageSelection.png)
2. Fill in the time period manually:
![workflow-input](../../../../img/new_ui/dev/project/workflow-input.png)
* Complement with or without scheduling
1. Without scheduling: Run workflow every day from start date to end date according to the time period you set. e.g. Do complement from July 7th to 10th without scheduling:
![workflow-unconfiguredTimingResult](../../../../img/new_ui/dev/project/workflow-unconfiguredTimingResult.png)
2. With scheduling: Run workflow from start date to end date on schedule according to the time period and schedule you set. e.g. Do complement from July 7th to 10th with the schedule of 5 AM every day:
![workflow-configuredTiming](../../../../img/new_ui/dev/project/workflow-configuredTiming.png)
![workflow-configuredTimingResult](../../../../img/new_ui/dev/project/workflow-configuredTimingResult.png)
The following are the operation functions of the workflow definition list:
@ -92,57 +139,57 @@ The following are the operation functions of the workflow definition list:
![workflow-run](../../../../img/new_ui/dev/project/workflow-run.png)
Description of workflow operating parameters:
Description of workflow operating parameters:
* Failure strategy: When a task node fails to execute, other parallel task nodes need to execute this strategy. "Continue" means: after a certain task fails, other task nodes execute normally; "End" means: terminate all tasks execution, and terminate the entire process.
* Notification strategy: When the process is over, send the process execution result notification email according to the process status, options including no send, send if sucess, send of failure, send whatever result.
* Process priority: The priority of process operation, divide into five levels: highest (HIGHEST), high (HIGH), medium (MEDIUM), low (LOW), and lowest (LOWEST). When the number of master threads is insufficient, high priority processes will execute first in the execution queue, and processes with the same priority will execute in the order of first in, first out.
* Worker group: The process can only be executed in the specified worker machine group. The default is `Default`, which can execute on any worker.
* Notification group: select notification strategy||timeout alarm||when fault tolerance occurs, process result information or email will send to all members in the notification group.
* Recipient: select notification policy||timeout alarm||when fault tolerance occurs, process result information or alarm email will be sent to the recipient list.
* Cc: select notification policy||timeout alarm||when fault tolerance occurs, the process result information or warning email will be copied to the CC list.
* Startup parameter: Set or overwrite global parameter values when starting a new process instance.
* Complement: refers to running the workflow definition within the specified date range and generating the corresponding workflow instance according to the complement policy. The complement policy includes two modes: **serial complement** and **parallel complement**. The date can be selected on the page or entered manually.
* Failure strategy: When a task node fails to execute, other parallel task nodes need to execute this strategy. "Continue" means: after a certain task fails, other task nodes execute normally; "End" means: terminate all tasks execution, and terminate the entire process.
* Notification strategy: When the process is over, send the process execution result notification email according to the process status, options including no send, send if sucess, send of failure, send whatever result.
* Process priority: The priority of process operation, divide into five levels: highest (HIGHEST), high (HIGH), medium (MEDIUM), low (LOW), and lowest (LOWEST). When the number of master threads is insufficient, high priority processes will execute first in the execution queue, and processes with the same priority will execute in the order of first in, first out.
* Worker group: The process can only be executed in the specified worker machine group. The default is `Default`, which can execute on any worker.
* Notification group: select notification strategy||timeout alarm||when fault tolerance occurs, process result information or email will send to all members in the notification group.
* Recipient: select notification policy||timeout alarm||when fault tolerance occurs, process result information or alarm email will be sent to the recipient list.
* Cc: select notification policy||timeout alarm||when fault tolerance occurs, the process result information or warning email will be copied to the CC list.
* Startup parameter: Set or overwrite global parameter values when starting a new process instance.
* Complement: refers to running the workflow definition within the specified date range and generating the corresponding workflow instance according to the complement policy. The complement policy includes two modes: **serial complement** and **parallel complement**. The date can be selected on the page or entered manually.
* Serial complement: within the specified time range, complement is executed from the start date to the end date, and multiple process instances are generated in turn; Click Run workflow and select the serial complement mode: for example, from July 9 to July 10, execute in sequence, and generate two process instances in sequence on the process instance page.
* Serial complement: within the specified time range, complement is executed from the start date to the end date, and multiple process instances are generated in turn; Click Run workflow and select the serial complement mode: for example, from July 9 to July 10, execute in sequence, and generate two process instances in sequence on the process instance page.
![workflow-serial](../../../../img/new_ui/dev/project/workflow-serial.png)
![workflow-serial](../../../../img/new_ui/dev/project/workflow-serial.png)
* Parallel Replenishment: within the specified time range, replenishment is performed simultaneously for multiple days, and multiple process instances are generated at the same time. Enter date manually: manually enter a date in the comma separated date format of 'yyyy MM DD hh:mm:ss'.Click Run workflow and select the parallel complement mode: for example, execute the workflow definition from July 9 to July 10 at the same time, and generate two process instances on the process instance page at the same time.
* Parallel Replenishment: within the specified time range, replenishment is performed simultaneously for multiple days, and multiple process instances are generated at the same time. Enter date manually: manually enter a date in the comma separated date format of 'yyyy MM DD hh:mm:ss'.Click Run workflow and select the parallel complement mode: for example, execute the workflow definition from July 9 to July 10 at the same time, and generate two process instances on the process instance page at the same time.
![workflow-parallel](../../../../img/new_ui/dev/project/workflow-parallel.png)
![workflow-parallel](../../../../img/new_ui/dev/project/workflow-parallel.png)
* Concurrency: refers to the maximum number of instances executed in parallel in the parallel complement mode.For example, if tasks from July 6 to July 10 are executed at the same time, and the concurrency is 2, then the process instance is:
* Concurrency: refers to the maximum number of instances executed in parallel in the parallel complement mode.For example, if tasks from July 6 to July 10 are executed at the same time, and the concurrency is 2, then the process instance is:
![workflow-concurrency-from](../../../../img/new_ui/dev/project/workflow-concurrency-from.png)
![workflow-concurrency-from](../../../../img/new_ui/dev/project/workflow-concurrency-from.png)
![workflow-concurrency](../../../../img/new_ui/dev/project/workflow-concurrency.png)
![workflow-concurrency](../../../../img/new_ui/dev/project/workflow-concurrency.png)
* Dependency mode: whether to trigger the replenishment of workflow instances that downstream dependent nodes depend on the current workflow (the timing status of workflow instances that require the current replenishment is online, which will only trigger the replenishment of downstream directly dependent on the current workflow).
* Dependency mode: whether to trigger the replenishment of workflow instances that downstream dependent nodes depend on the current workflow (the timing status of workflow instances that require the current replenishment is online, which will only trigger the replenishment of downstream directly dependent on the current workflow).
![workflow-dependency](../../../../img/new_ui/dev/project/workflow-dependency.png)
![workflow-dependency](../../../../img/new_ui/dev/project/workflow-dependency.png)
* Date selection:
* Date selection:
1. Select the date through the page:
1. Select the date through the page:
![workflow-pageSelection](../../../../img/new_ui/dev/project/workflow-pageSelection.png)
![workflow-pageSelection](../../../../img/new_ui/dev/project/workflow-pageSelection.png)
2. Manual input:
2. Manual input:
![workflow-input](../../../../img/new_ui/dev/project/workflow-input.png)
![workflow-input](../../../../img/new_ui/dev/project/workflow-input.png)
* Relationship between complement and timing configuration:
* Relationship between complement and timing configuration:
1. Unconfigured timing: When there is no timing configuration, the daily replenishment will be performed by default according to the selected time range. For example, the workflow scheduling date is July 7 to July 10. If timing is not configured, the process instance is:
1. Unconfigured timing: When there is no timing configuration, the daily replenishment will be performed by default according to the selected time range. For example, the workflow scheduling date is July 7 to July 10. If timing is not configured, the process instance is:
![workflow-unconfiguredTimingResult](../../../../img/new_ui/dev/project/workflow-unconfiguredTimingResult.png)
![workflow-unconfiguredTimingResult](../../../../img/new_ui/dev/project/workflow-unconfiguredTimingResult.png)
2. Configured timing: If there is a timing configuration, it will be supplemented according to the selected time range in combination with the timing configuration. For example, the workflow scheduling date is July 7 to July 10, and the timing is configured (running every 5 a.m.). The process example is:
2. Configured timing: If there is a timing configuration, it will be supplemented according to the selected time range in combination with the timing configuration. For example, the workflow scheduling date is July 7 to July 10, and the timing is configured (running every 5 a.m.). The process example is:
![workflow-configuredTiming](../../../../img/new_ui/dev/project/workflow-configuredTiming.png)
![workflow-configuredTiming](../../../../img/new_ui/dev/project/workflow-configuredTiming.png)
![workflow-configuredTimingResult](../../../../img/new_ui/dev/project/workflow-configuredTimingResult.png)
![workflow-configuredTimingResult](../../../../img/new_ui/dev/project/workflow-configuredTimingResult.png)
## Run the task alone
- Right-click the task and click the `Start` button (only online tasks can be clicked to run).
@ -160,12 +207,15 @@ The following are the operation functions of the workflow definition list:
![workflow-time01](../../../../img/new_ui/dev/project/workflow-time01.png)
- Select a start and end time. Within the start and end time range, the workflow is run regularly; outside the start and end time range, no timed workflow instance will be generated.
- Add a timing that execute 5 minutes once, as shown in the following figure:
![workflow-time02](../../../../img/new_ui/dev/project/workflow-time02.png)
- Failure strategy, notification strategy, process priority, worker group, notification group, recipient, and CC are the same as workflow running parameters.
- Click the "Create" button to create the timing. Now the timing status is "**Offline**" and the timing needs to be **Online** to make effect.
- Schedule online: Click the `Timing Management` button <img src="../../../../img/timeManagement.png" width="35"/>, enter the timing management page, click the `online` button, the timing status will change to `online`, as shown in the below figure, the workflow makes effect regularly.
![workflow-time03](../../../../img/new_ui/dev/project/workflow-time03.png)

View File

@ -43,15 +43,23 @@ Click `Project Management -> Workflow -> Workflow Instance`, enter the workflow
![workflow-instance](../../../../img/new_ui/dev/project/workflow-instance.png)
- **Edit:** Only processes with success/failed/stop status can be edited. Click the "Edit" button or the workflow instance name to enter the DAG edit page. After the edit, click the "Save" button to confirm, as shown in the figure below. In the pop-up box, check "Whether to update the workflow definition", after saving, the information modified by the instance will be updated to the workflow definition; if not checked, the workflow definition would not be updated.
<p align="center">
<img src="../../../../img/editDag-en.png" width="80%" />
</p>
- **Rerun** Re-execute the terminated process
- **Recovery Failed:** For failed processes, you can perform failure recovery operations, starting from the failed node
- **Stop:** **Stop** the running process, the background code will first `kill` the worker process, and then execute `kill -9` operation
- **Pause:** **Pause** the running process, the system status will change to **waiting for execution**, it will wait for the task to finish, and pause the next sequence task.
- **Resume pause:** Resume the paused process, start running directly from the **paused node**
- **Delete:** Delete the workflow instance and the task instance under the workflow instance
- **Gantt Chart:** The vertical axis of the Gantt chart is the topological sorting of task instances of the workflow instance, and the horizontal axis is the running time of the task instances, as shown in the figure:
![instance-gantt](../../../../img/new_ui/dev/project/instance-gantt.png)

View File

@ -45,9 +45,9 @@ data.basedir.path=/tmp/dolphinscheduler
# resource view suffixs
#resource.view.suffixs=txt,log,sh,bat,conf,cfg,py,java,sql,xml,hql,properties,json,yml,yaml,ini,js
# resource storage type: HDFS, S3, NONE
# resource storage type: HDFS, S3, OSS, NONE
resource.storage.type=NONE
# resource store on HDFS/S3 path, resource file will store to this base path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended
# resource store on HDFS/S3/OSS path, resource file will store to this base path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended
resource.storage.upload.base.path=/tmp/dolphinscheduler
# The AWS access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required
@ -61,6 +61,17 @@ resource.aws.s3.bucket.name=dolphinscheduler
# You need to set this parameter when private cloud s3. If S3 uses public cloud, you only need to set resource.aws.region or set to the endpoint of a public cloud such as S3.cn-north-1.amazonaws.com.cn
resource.aws.s3.endpoint=http://localhost:9000
# alibaba cloud access key id, required if you set resource.storage.type=OSS
resource.alibaba.cloud.access.key.id=<your-access-key-id>
# alibaba cloud access key secret, required if you set resource.storage.type=OSS
resource.alibaba.cloud.access.key.secret=<your-access-key-secret>
# alibaba cloud region, required if you set resource.storage.type=OSS
resource.alibaba.cloud.region=cn-hangzhou
# oss bucket name, required if you set resource.storage.type=OSS
resource.alibaba.cloud.oss.bucket.name=dolphinscheduler
# oss bucket endpoint, required if you set resource.storage.type=OSS
resource.alibaba.cloud.oss.endpoint=https://oss-cn-hangzhou.aliyuncs.com
# if resource.storage.type=HDFS, the user must have the permission to create directories under the HDFS root path
resource.hdfs.root.user=hdfs
# if resource.storage.type=S3, the value like: s3a://dolphinscheduler; if resource.storage.type=HDFS and namenode HA is enabled, you need to copy core-site.xml and hdfs-site.xml to conf dir

View File

@ -65,6 +65,7 @@ In the workflow definition module of project Manage, create a new workflow using
- Script: 'sh hello.sh'
- Resource: Select 'hello.sh'
> Notice: When using a resource file in the script, the file name needs to be the same as the full path of the selected resource:
> For example: if the resource path is `/resource/hello.sh`, you need to use the full path of `/resource/hello.sh` to use it in the script.

View File

@ -139,7 +139,7 @@ worker:
```
- You can add new worker groups for the workers during runtime regardless of the configurations in `application.yaml` as below:
`Security Center` -> `Worker Group Manage` -> `Create Worker Group` -> fill in `Group Name` and `Worker Addresses` -> click `confirm`.
`Security Center` -> `Worker Group Manage` -> `Create Worker Group` -> fill in `Group Name` and `Worker Addresses` -> click `confirm`.
## Environmental Management
@ -164,10 +164,10 @@ Create a task node in the workflow definition, select the worker group and the e
## Cluster Management
> Add or update cluster
- Each process can be related to zero or several clusters to support multiple environment, now just support k8s.
> - Each process can be related to zero or several clusters to support multiple environment, now just support k8s.
>
> Usage cluster
- After creation and authorization, k8s namespaces and processes will associate clusters. Each cluster will have separate workflows and task instances running independently.
> - After creation and authorization, k8s namespaces and processes will associate clusters. Each cluster will have separate workflows and task instances running independently.
![create-cluster](../../../img/new_ui/dev/security/create-cluster.png)
@ -183,4 +183,3 @@ Create a task node in the workflow definition, select the worker group and the e
![create-environment](../../../img/new_ui/dev/security/create-namespace.png)

View File

@ -19,7 +19,7 @@ Start DolphinScheduler with standalone-server Docker images is the easiest way t
you can learn DolphinScheduler's concepts and usage, with minimal cost.
```shell
$ DOLPHINSCHEDULER_VERSION=<version>
$ DOLPHINSCHEDULER_VERSION=3.1.0
$ docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:25333 -d apache/dolphinscheduler-standalone-server:"${DOLPHINSCHEDULER_VERSION}"
```
@ -42,14 +42,14 @@ modify docker-compose's free memory up to 4 GB.
- MacClick `Docker Desktop -> Preferences -> Resources -> Memory` modified it
- Windows Docker Desktop
- Hyper-V mode: Click `Docker Desktop -> Settings -> Resources -> Memory` modified it
- WSL 2 mode: see [WSL 2 utility VM](https://docs.microsoft.com/zh-cn/windows/wsl/wsl-config#configure-global-options-with-wslconfig) for more detail.
- Hyper-V mode: Click `Docker Desktop -> Settings -> Resources -> Memory` modified it
- WSL 2 mode: see [WSL 2 utility VM](https://docs.microsoft.com/zh-cn/windows/wsl/wsl-config#configure-global-options-with-wslconfig) for more detail.
After complete the configuration, we can get the `docker-compose.yaml` file from [download page](/en-us/download/download.html)
form its source package, and make sure you get the right version. After download the package, you can run the commands as below.
```shell
$ DOLPHINSCHEDULER_VERSION=<version>
$ DOLPHINSCHEDULER_VERSION=3.1.0
$ tar -zxf apache-dolphinscheduler-"${DOLPHINSCHEDULER_VERSION}"-src.tar.gz
# Going to docker-compose's location
# For Mac or Linux users
@ -71,9 +71,8 @@ $ docker-compose --profile all up -d
[Using docker-compose to start server](#using-docker-compose-to-start-server) will create new a database and the ZooKeeper
container when it up. You could start DolphinScheduler server separately if you want to reuse your exists services.
```shell
$ DOLPHINSCHEDULER_VERSION=<version>
$ DOLPHINSCHEDULER_VERSION=3.1.0
# Initialize the database, make sure database <DATABASE> already exists
$ docker run -d --name dolphinscheduler-tools \
-e DATABASE="postgresql" \
@ -134,5 +133,5 @@ and use `admin` and `dolphinscheduler123` as default username and password in th
You can modify some environment variables to change configurations when you are starting servers through Docker. We have
an example in [using exists PostgreSQL ZooKeeper](#using-exists-postgresql-zookeeper) to change database and ZooKeeper configurations,
and you could find all environment variables in [all environment variables](https://github.com/apache/dolphinscheduler/blob/<version>/script/env/dolphinscheduler_env.sh) <!-- markdown-link-check-disable-line -->
and you could find all environment variables in [all environment variables](https://github.com/apache/dolphinscheduler/blob/3.1.0/script/env/dolphinscheduler_env.sh) <!-- markdown-link-check-disable-line -->
and change them if you want.

View File

@ -0,0 +1,24 @@
# DolphinScheduler Task Parameters Appendix
DolphinScheduler task plugins share some common default parameters. Each type of task contains all or **some** default parameters as follows:
## Default Task Parameters
| **Parameter** | **Description** |
|--------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Node Name | The name of the task. Node names within the same workflow must be unique. |
| Run Flag | Indicating whether to schedule the task. If you do not need to execute the task, you can turn on the `Prohibition execution` switch. |
| Description | Describing the function of this node. |
| Task Priority | When the number of the worker threads is insufficient, the worker executes task according to the priority. When two tasks have the same priority, the worker will execute them in `first come first served` fashion. |
| Worker Group | Machines which execute the tasks. If you choose `default`, scheduler will send the task to a random worker. |
| Task Group Name | Resource group of tasks. It will not take effect if not configured. |
| Environment Name | Environment to execute the task. |
| Number of Failed Retries | The number of task retries for failures. You could select it by drop-down menu or fill it manually. |
| Failure Retry Interval | Interval of task retries for failures. You could select it by drop-down menu or fill it manually. |
| CPU Quota | Assign the specified CPU time quota to the task executed. Takes a percentage value. Default -1 means unlimited. For example, the full CPU load of one core is 100%, and that of 16 cores is 1600%. You could configure it by [task.resource.limit.state](../../architecture/configuration.md). |
| Max Memory | Assign the specified max memory to the task executed. Exceeding this limit will trigger oom to be killed and will not automatically retry. Takes an MB value. Default -1 means unlimited. You could configure it by [task.resource.limit.state](../../architecture/configuration.md). |
| Timeout Alarm | Alarm for task timeout. When the task exceeds the "timeout threshold", an alarm email will send. |
| Delayed Execution Time | The time that a task delays for execution in minutes. |
| Resources | Resources which your task node uses. |
| Predecessor Task | The upstream task of the current task node. |

View File

@ -11,26 +11,16 @@ ChunJun task type for executing ChunJun programs. For ChunJun nodes, the worker
## Task Parameters
| **Parameter** | **Description** |
| ------- | ---------- |
| Node name | The node name in a workflow definition is unique. |
| Run flag | Identifies whether this node schedules normally, if it does not need to execute, select the prohibition execution. |
| Task priority | When the number of worker threads is insufficient, execute in the order of priority from high to low, and tasks with the same priority will execute in a first-in first-out order. |
| Description | Describe the function of the node. |
| Worker group | Assign tasks to the machines of the worker group to execute. If `Default` is selected, randomly select a worker machine for execution. |
| Environment Name | Configure the environment name in which run the script. |
| Number of failed retries | The number of times the task failed to resubmit. |
| Failed retry interval | The time interval (unit minute) for resubmitting the task after a failed task. |
| Task group name | The task group name. |
| Priority | The task priority. |
| Delayed execution time | The time, in minutes, that a task is delayed in execution. |
| Timeout alarm | Check the timeout alarm and timeout failure. When the task exceeds the "timeout period", an alarm email will be sent and the task execution will fail. |
| Custom template | Custom the content of the ChunJun node's json profile. |
| json | json configuration file for ChunJun synchronization. |
| Custom parameters | It is a user-defined parameter, and will replace the content with `${variable}` in the script.
| Deploy mode | Execute chunjun task mode, eg local standalone. |
| Option Parameters | Support such as `-confProp "{\"flink.checkpoint.interval\":60000}"` |
| Predecessor task | Selecting a predecessor task for the current task will set the selected predecessor task as upstream of the current task. |
- Please refer to [DolphinScheduler Task Parameters Appendix](appendix.md#default-task-parameters) for default parameters.
| **Parameter** | **Description** |
|-------------------|---------------------------------------------------------------------------------------------------------------------------|
| Custom template | Custom the content of the ChunJun node's json profile. |
| json | json configuration file for ChunJun synchronization. |
| Custom parameters | It is a user-defined parameter, and will replace the content with `${variable}` in the script. |
| Deploy mode | Execute chunjun task mode, eg local standalone. |
| Option Parameters | Support such as `-confProp "{\"flink.checkpoint.interval\":60000}"` |
| Predecessor task | Selecting a predecessor task for the current task will set the selected predecessor task as upstream of the current task. |
## Task Example
@ -60,7 +50,7 @@ After writing the required json file, you can configure the node content by foll
Before execute ${CHUNJUN_HOME}/bin/start-chunjun, need to change the shell ${CHUNJUN_HOME}/bin/start-chunjun, remove '&' in order to run in front.
such as:
such as:
```shell
nohup $JAVA_RUN -cp $JAR_DIR $CLASS_NAME $@ &
@ -71,3 +61,4 @@ update to following:
```shell
nohup $JAVA_RUN -cp $JAR_DIR $CLASS_NAME $@
```

View File

@ -9,19 +9,11 @@ Condition is a conditional node, that determines which downstream task should ru
## Task Parameters
| **Parameter** | **Description** |
| -------- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Node Name | Set the name of the task. Node names within a workflow definition are unique. |
| Run flag | Indicates whether the node can be scheduled normally. If it is not necessary to execute, you can turn on the prohibiting execution switch. |
| Description | Describes the function of this node. |
| Task priority | When the number of worker threads is insufficient, they are executed in order from high to low according to the priority, and they are executed according to the first-in, first-out principle when the priority is the same. |
| Worker group | The task is assigned to the machines in the worker group for execution. If Default is selected, a worker machine will be randomly selected for execution. |
| Task group name | The group in Resources, if not configured, it will not be used. |
| Environment Name | Configure the environment in which to run the script. |
| Number of failed retries | The number of times the task is resubmitted after failure. It supports drop-down and manual filling. |
| Failure Retry Interval | The time interval for resubmitting the task if the task fails. It supports drop-down and manual filling. |
| Timeout alarm | Check Timeout Alarm and Timeout Failure. When the task exceeds the "timeout duration", an alarm email will be sent and the task execution will fail. |
| Downstream tasks selection | Depending on the status of the predecessor task, you can jump to the corresponding branch, currently two branches are supported: success, failure <ul><li>Success: When the upstream task runs successfully, run the success branch.</li><li>Failure: When the upstream task runs failed, run the failure branch.</li></ul></li></ul> |
- Please refer to [DolphinScheduler Task Parameters Appendix](appendix.md#default-task-parameters) for default parameters.
| **Parameter** | **Description** |
|------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Downstream tasks selection | Depending on the status of the predecessor task, you can jump to the corresponding branch, currently two branches are supported: success, failure <ul><li>Success: When the upstream task runs successfully, run the success branch.</li><li>Failure: When the upstream task runs failed, run the failure branch.</li></ul></li></ul> |
| Upstream condition selection | Can select one or more upstream tasks for conditions.<ul><li>Add an upstream dependency: the first parameter is to choose a specified task name, and the second parameter is to choose the upstream task status to trigger conditions.</li><li>Select upstream task relationship: use `and` and `or` operators to handle the complex relationship of upstream when there are multiple upstream tasks for conditions.</li></ul></li></ul> |
## Related Task

View File

@ -11,33 +11,22 @@ DataX task type for executing DataX programs. For DataX nodes, the worker will e
## Task Parameters
| **Parameter** | **Description** |
| ------- | ---------- |
| Node name | The node name in a workflow definition is unique. |
| Run flag | Identifies whether this node schedules normally, if it does not need to execute, select the prohibition execution. |
| Task priority | When the number of worker threads is insufficient, execute in the order of priority from high to low, and tasks with the same priority will execute in a first-in first-out order. |
| Description | Describe the function of the node. |
| Worker group | Assign tasks to the machines of the worker group to execute. If `Default` is selected, randomly select a worker machine for execution. |
| Environment Name | Configure the environment name in which run the script. |
| Number of failed retries | The number of times the task failed to resubmit. |
| Failed retry interval | The time interval (unit minute) for resubmitting the task after a failed task. |
| Cpu quota | Assign the specified CPU time quota to the task executed. Takes a percentage value. Default -1 means unlimited. For example, the full CPU load of one core is 100%,and that of 16 cores is 1600%. This function is controlled by [task.resource.limit.state](../../architecture/configuration.md) |
| Max memory | Assign the specified max memory to the task executed. Exceeding this limit will trigger oom to be killed and will not automatically retry. Takes an MB value. Default -1 means unlimited. This function is controlled by [task.resource.limit.state](../../architecture/configuration.md) |
| Delayed execution time | The time, in cents, that a task is delayed in execution. |
| Timeout alarm | Check the timeout alarm and timeout failure. When the task exceeds the "timeout period", an alarm email will be sent and the task execution will fail. |
| Custom template | Custom the content of the DataX node's json profile when the default data source provided does not meet the required requirements. |
| json | json configuration file for DataX synchronization. |
| Resource | When using custom json, if the cluster has kerberos authentication enabled, and datax needs to use the relevant keytab, xml file, etc. when reading or writing plug-ins such as hdfs and hbase, you can use this option. and the files uploaded or created in Resource Center - File Management.|
| Custom parameters | SQL task type, and stored procedure is a custom parameter order to set values for the method. The custom parameter type and data type are the same as the stored procedure task type. The difference is that the SQL task type custom parameter will replace the \${variable} in the SQL statement. |
| Data source | Select the data source from which the data will be extracted. |
| sql statement | the sql statement used to extract data from the target database, the sql query column name is automatically parsed when the node is executed, and mapped to the target table synchronization column name. When the source table and target table column names are inconsistent, they can be converted by column alias. |
| Target library | Select the target library for data synchronization. |
| Pre-sql | Pre-sql is executed before the sql statement (executed by the target library). |
| Post-sql | Post-sql is executed after the sql statement (executed by the target library). |
| Stream limit (number of bytes) | Limits the number of bytes in the query. |
| Limit flow (number of records) | Limit the number of records for a query. |
| Running memory | the minimum and maximum memory required can be configured to suit the actual production environment. |
| Predecessor task | Selecting a predecessor task for the current task will set the selected predecessor task as upstream of the current task. |
- Please refer to [DolphinScheduler Task Parameters Appendix](appendix.md#default-task-parameters) for default parameters.
| **Parameter** | **Description** |
|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Custom template | Custom the content of the DataX node's json profile when the default data source provided does not meet the required requirements. |
| json | json configuration file for DataX synchronization. |
| Resource | When using custom json, if the cluster has kerberos authentication enabled, and datax needs to use the relevant keytab, xml file, etc. when reading or writing plug-ins such as hdfs and hbase, you can use this option. and the files uploaded or created in Resource Center - File Management. |
| Custom parameters | SQL task type, and stored procedure is a custom parameter order to set values for the method. The custom parameter type and data type are the same as the stored procedure task type. The difference is that the SQL task type custom parameter will replace the \${variable} in the SQL statement. |
| Data source | Select the data source from which the data will be extracted. |
| sql statement | the sql statement used to extract data from the target database, the sql query column name is automatically parsed when the node is executed, and mapped to the target table synchronization column name. When the source table and target table column names are inconsistent, they can be converted by column alias. |
| Target library | Select the target library for data synchronization. |
| Pre-sql | Pre-sql is executed before the sql statement (executed by the target library). |
| Post-sql | Post-sql is executed after the sql statement (executed by the target library). |
| Stream limit (number of bytes) | Limits the number of bytes in the query. |
| Limit flow (number of records) | Limit the number of records for a query. |
| Running memory | the minimum and maximum memory required can be configured to suit the actual production environment. |
## Task Example

View File

@ -4,7 +4,6 @@
Dependent nodes are **dependency check nodes**. For example, process A depends on the successful execution of process B from yesterday, and the dependent node will check whether process B run successful yesterday.
## Create Task
- Click `Project Management -> Project Name -> Workflow Definition`, and click the `Create Workflow` button to enter the DAG editing page.
@ -12,20 +11,11 @@ Dependent nodes are **dependency check nodes**. For example, process A depends o
## Task Parameter
| **Parameter** | **Description** |
| ----- | -----------|
| Node name | Unique name of node in workflow definition. |
| Run flag | Identifies whether this node schedules normally. |
| Description | Describe the function of the node. |
| Task priority | When the number of worker threads is insufficient, execute in the order of priority from high to low, and tasks with the same priority will execute in a first-in first-out order. |
| Worker group | Assign tasks to the machines of the worker group to execute. If `Default` is selected, randomly select a worker machine for execution. |
| Task group name | The group in Resources, if not configured, it will not be used. |
| Environment Name | Configure the environment name in which run the script. |
| Number of failed retries | The number of times the task failed to resubmit. |
| Failed retry interval | The time interval (unit minute) for resubmitting the task after a failed task. |
| Delayed execution time | The time (unit minute) that a task delays in execution. |
| Pre task | Selecting a predecessor task for the current task, will set the selected predecessor task as upstream of the current task. |
- Please refer to [DolphinScheduler Task Parameters Appendix](appendix.md#default-task-parameters) for default parameters.
| **Parameter** | **Description** |
|------------------|---------------------------------------------|
| Predecessor Task | The upstream task of the current task node. |
## Task Examples

Some files were not shown because too many files have changed in this diff Show More