[Imporvement] Improve the MasterCofnig module comments of dolphinscheduler (#16134)

Co-authored-by: xiangzihao <460888207@qq.com>

.asf.yaml

@@ -45,7 +45,8 @@ github:
           - Unit Test
           - E2E
           - Docs
-          - "Mergeable: milestone-label-check"
+          - Frontend Build
+#          - "Mergeable: milestone-label-check"
       required_pull_request_reviews:
         dismiss_stale_reviews: true
-        required_approving_review_count: 1
+        required_approving_review_count: 2

.github/CODEOWNERS

@@ -23,7 +23,7 @@
 /dolphinscheduler-registry/ @caishunfeng @ruanwenjun
 /dolphinscheduler-api/ @caishunfeng @SbloodyS
 /dolphinscheduler-dao/ @caishunfeng @SbloodyS
-/dolphinscheduler-dao/src/main/resources/sql/ @zhongjiajie
+/dolphinscheduler-dao/src/main/resources/sql/ @EricGao888
 /dolphinscheduler-common/ @caishunfeng
 /dolphinscheduler-standalone-server/ @caishunfeng
 /dolphinscheduler-datasource-plugin/ @caishunfeng
@@ -33,13 +33,13 @@
 /dolphinscheduler-master/ @caishunfeng @SbloodyS @ruanwenjun
 /dolphinscheduler-worker/ @caishunfeng @SbloodyS @ruanwenjun
 /dolphinscheduler-service/ @caishunfeng
-/dolphinscheduler-remote/ @caishunfeng
+/dolphinscheduler-extract/ @caishunfeng @ruanwenjun
 /dolphinscheduler-spi/ @caishunfeng
 /dolphinscheduler-task-plugin/ @caishunfeng @SbloodyS @zhuangchong
-/dolphinscheduler-tools/ @caishunfeng @SbloodyS @zhongjiajie @EricGao888
-/script/ @caishunfeng @SbloodyS @zhongjiajie @EricGao888
+/dolphinscheduler-tools/ @caishunfeng @SbloodyS @EricGao888
+/script/ @caishunfeng @SbloodyS @EricGao888
 /dolphinscheduler-ui/ @songjianet @Amy0104
-/docs/ @zhongjiajie @EricGao888
-/licenses/ @zhongjiajie
-/images/ @zhongjiajie @EricGao888
+/docs/ @EricGao888
+/licenses/ @EricGao888
+/images/ @EricGao888
 /style/ @caishunfeng

.github/ISSUE_TEMPLATE/dsip-request.yml

@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+name: DSIP
+description: Suggest an idea for this project
+title: "[DSIP-][Module Name] DSIP title"
+labels: [ "DSIP", "Waiting for reply" ]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        For better global communication, Please write in English.
+
+        If you feel the description in English is not clear, then you can append description in Chinese, thanks!
+
+  - type: checkboxes
+    attributes:
+      label: Search before asking
+      description: >
+        Please make sure to search in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) first
+        to see whether the same DSIP was created already.
+      options:
+        - label: >
+            I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no
+            similar DSIP.
+          required: true
+
+  - type: textarea
+    attributes:
+      label: Motivation
+      description: Why you want to do this change?
+
+  - type: textarea
+    attributes:
+      label: Design Detail
+      description: Your design.
+      placeholder: >
+        It's better to provide a detailed design, such as the design of the interface, the design of the database, etc.
+
+  - type: textarea
+    attributes:
+      label: Compatibility, Deprecation, and Migration Plan
+      description: >
+        If this feature is related to compatibility, deprecation, or migration, please describe it here.
+
+  - type: textarea
+    attributes:
+      label: Test Plan
+      description: >
+        How to test this improvement.
+
+  - type: checkboxes
+    attributes:
+      label: Code of Conduct
+      description: |
+        The Code of Conduct helps create a safe space for everyone. We require that everyone agrees to it.
+      options:
+        - label: |
+            I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
+          required: true
+
+  - type: markdown
+    attributes:
+      value: "Thanks for completing our form!"

.github/actions/labeler/labeler.yml

@@ -25,7 +25,7 @@ backend:
   - 'dolphinscheduler-dist/**/*'
   - 'dolphinscheduler-master/**/*'
   - 'dolphinscheduler-registry/**/*'
-  - 'dolphinscheduler-remote/**/*'
+  - 'dolphinscheduler-extract/**/*'
   - 'dolphinscheduler-scheduler-plugin/**/*'
   - 'dolphinscheduler-service/**/*'
   - 'dolphinscheduler-spi/**/*'

.github/mergeable.yml

@@ -1,62 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
----
-version: 2
-mergeable:
-  # we can not use `pull_request.*` which including event `pull_request.labeled`, according to https://github.com/mergeability/mergeable/issues/643,
-  # otherwise mergeable will keep add or remove label endless, we just need this CI act like the default behavior as
-  # GitHub action workflow `pull_requests` https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request like,
-  # which only trigger runs when a pull_request event's activity type is opened, synchronize, or reopened
-  - when: pull_request.opened, pull_request.reopened, pull_request.synchronize
-    name: sync-sql-ddl
-    validate:
-      # Sql files must change synchronize
-      - do: dependent
-        files:
-          - 'dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_h2.sql'
-          - 'dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql'
-          - 'dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_postgresql.sql'
-        message: 'Sql files not change synchronize'
-    # Add labels 'sql not sync' and comment to reviewers if Sql files not change synchronize
-    fail:
-      - do: comment
-        payload:
-          body: >
-            :warning: This PR do not change database DDL synchronize.
-        leave_old_comment: false
-      - do: labels
-        add: 'sql not sync'
-    # Remove labels 'sql not sync' if pass
-    pass:
-      - do: labels
-        delete: 'sql not sync'
-
-  - when: pull_request.*
-    name: milestone-label-check
-    validate:
-      - do: milestone
-        no_empty:
-          enabled: false # Cannot be empty when true.
-          message: 'Milestone is required and cannot be empty.'
-      - do: label
-        and:
-          - must_include:
-              regex: 'feature|bug|improvement|document|chore|revert'
-              message: 'Label must include one of the following: `feature`, `bug`, `improvement`, `document`, `chore`, `revert`'
-          - must_include:
-              regex: 'ready-to-merge'
-              message: 'Please check if there are PRs that already have a `ready-to-merge` label and can be merged, if exists please merge them first.'

.github/workflows/backend.yml

@@ -106,10 +106,14 @@ jobs:
     strategy:
       matrix:
         case:
-          - name: cluster-test-mysql
-            script: .github/workflows/cluster-test/mysql/start-job.sh
-          - name: cluster-test-postgresql
-            script: .github/workflows/cluster-test/postgresql/start-job.sh
+          - name: cluster-test-mysql-with-zookeeper-registry
+            script: .github/workflows/cluster-test/mysql_with_zookeeper_registry/start-job.sh
+          - name: cluster-test-mysql-with-mysql-registry
+            script: .github/workflows/cluster-test/mysql_with_mysql_registry/start-job.sh
+          - name: cluster-test-postgresql-zookeeper-registry
+            script: .github/workflows/cluster-test/postgresql_with_zookeeper_registry/start-job.sh
+          - name: cluster-test-postgresql-with-postgresql-registry
+            script: .github/workflows/cluster-test/postgresql_with_postgresql_registry/start-job.sh
     steps:
       - uses: actions/checkout@v2
         with:
@@ -149,7 +153,7 @@ jobs:
       fail-fast: false
       matrix:
         db: ["mysql", "postgresql"]
-        version: ["2.0.9", "3.0.6", "3.1.8"]
+        version: ["3.1.9", "3.2.0"]
     steps:
       - name: Set up JDK 8
         uses: actions/setup-java@v2
@@ -171,7 +175,7 @@ jobs:
           echo "DATABASE_VERSION=${VERSION//\./}" >> $GITHUB_ENV
       - name: Prepare
         run: |
-          wget https://dlcdn.apache.org/dolphinscheduler/${{ matrix.version }}/apache-dolphinscheduler-${{ matrix.version }}-bin.tar.gz -P dolphinscheduler/${{ matrix.version }}
+          wget https://archive.apache.org/dist/dolphinscheduler/${{ matrix.version }}/apache-dolphinscheduler-${{ matrix.version }}-bin.tar.gz -P dolphinscheduler/${{ matrix.version }}
           tar -xzf dolphinscheduler/${{ matrix.version }}/apache-dolphinscheduler-${{ matrix.version }}-bin.tar.gz -C dolphinscheduler/${{ matrix.version }} --strip-components 1
           tar -xzf dolphinscheduler/dev/apache-dolphinscheduler-*-bin.tar.gz -C dolphinscheduler/dev --strip-components 1
 
@@ -267,7 +271,7 @@ jobs:
             echo "Skip Build!"
             exit 0
           fi
-          if [[ ${{ needs.build.result }} != 'success' || ${{ needs.cluster-test.result }} != 'success' || ${{ needs.schema-check.result }} != 'success' ]]; then
+          if [[ ${{ needs.build.result }} != 'success' || ${{ needs.cluster-test.result }} != 'success' ]]; then
             echo "Build Failed!"
             exit -1
           fi

.github/workflows/cluster-test/mysql_with_mysql_registry/Dockerfile

@@ -28,10 +28,10 @@ RUN mv /root/apache-dolphinscheduler-*-SNAPSHOT-bin /root/apache-dolphinschedule
 ENV DOLPHINSCHEDULER_HOME /root/apache-dolphinscheduler-test-SNAPSHOT-bin
 
 #Setting install.sh
-COPY .github/workflows/cluster-test/mysql/install_env.sh $DOLPHINSCHEDULER_HOME/bin/env/install_env.sh
+COPY .github/workflows/cluster-test/mysql_with_mysql_registry/install_env.sh $DOLPHINSCHEDULER_HOME/bin/env/install_env.sh
 
 #Setting dolphinscheduler_env.sh
-COPY .github/workflows/cluster-test/mysql/dolphinscheduler_env.sh $DOLPHINSCHEDULER_HOME/bin/env/dolphinscheduler_env.sh
+COPY .github/workflows/cluster-test/mysql_with_mysql_registry/dolphinscheduler_env.sh $DOLPHINSCHEDULER_HOME/bin/env/dolphinscheduler_env.sh
 
 #Download mysql jar
 ENV MYSQL_URL "https://repo.maven.apache.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar"
@@ -43,6 +43,6 @@ cp $DOLPHINSCHEDULER_HOME/alert-server/libs/$MYSQL_DRIVER $DOLPHINSCHEDULER_HOME
 cp $DOLPHINSCHEDULER_HOME/alert-server/libs/$MYSQL_DRIVER $DOLPHINSCHEDULER_HOME/tools/libs/$MYSQL_DRIVER
 
 #Deploy
-COPY .github/workflows/cluster-test/mysql/deploy.sh /root/deploy.sh
+COPY .github/workflows/cluster-test/mysql_with_mysql_registry/deploy.sh /root/deploy.sh
 
 CMD [ "/bin/bash", "/root/deploy.sh" ]

.github/workflows/cluster-test/mysql_with_mysql_registry/deploy.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+set -euox pipefail
+
+
+USER=root
+
+#Create database
+mysql -hmysql -P3306 -uroot -p123456 -e "CREATE DATABASE IF NOT EXISTS dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;"
+
+#Sudo
+sed -i '$a'$USER'  ALL=(ALL)  NOPASSWD: NOPASSWD: ALL' /etc/sudoers
+sed -i 's/Defaults    requirett/#Defaults    requirett/g' /etc/sudoers
+
+#SSH
+ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
+cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
+chmod 600 ~/.ssh/authorized_keys
+service ssh start
+
+#Init schema
+/bin/bash $DOLPHINSCHEDULER_HOME/tools/bin/upgrade-schema.sh
+/bin/bash $DOLPHINSCHEDULER_HOME/tools/bin/initialize-jdbc-registry.sh
+
+#Start Cluster
+/bin/bash $DOLPHINSCHEDULER_HOME/bin/start-all.sh
+
+#Keep running
+tail -f /dev/null

.github/workflows/cluster-test/mysql_with_mysql_registry/docker-compose-base.yaml

@@ -0,0 +1,34 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+version: "3"
+
+services:
+  mysql:
+    container_name: mysql
+    image: mysql:5.7.36
+    command: --default-authentication-plugin=mysql_native_password
+    restart: always
+    environment:
+      MYSQL_ROOT_PASSWORD: 123456
+    ports:
+      - "3306:3306"
+    healthcheck:
+      test: mysqladmin ping -h 127.0.0.1 -u root --password=$$MYSQL_ROOT_PASSWORD
+      interval: 5s
+      timeout: 60s
+      retries: 120

.github/workflows/cluster-test/mysql_with_mysql_registry/dolphinscheduler_env.sh

@@ -0,0 +1,57 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# JAVA_HOME, will use it to start DolphinScheduler server
+export JAVA_HOME=${JAVA_HOME:-/opt/java/openjdk}
+
+# Database related configuration, set database type, username and password
+export DATABASE=${DATABASE:-mysql}
+export SPRING_PROFILES_ACTIVE=${DATABASE}
+export SPRING_DATASOURCE_URL="jdbc:mysql://mysql:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false"
+export SPRING_DATASOURCE_USERNAME=root
+export SPRING_DATASOURCE_PASSWORD=123456
+
+# DolphinScheduler server related configuration
+export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
+export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
+
+# Registry center configuration, determines the type and link of the registry center
+export REGISTRY_TYPE=${REGISTRY_TYPE:-jdbc}
+export REGISTRY_HIKARI_CONFIG_JDBC_URL="jdbc:mysql://mysql:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false"
+export REGISTRY_HIKARI_CONFIG_USERNAME=root
+export REGISTRY_HIKARI_CONFIG_PASSWORD=123456
+
+# Tasks related configurations, need to change the configuration if you use the related tasks.
+export HADOOP_HOME=${HADOOP_HOME:-/opt/soft/hadoop}
+export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/soft/hadoop/etc/hadoop}
+export SPARK_HOME=${SPARK_HOME:-/opt/soft/spark}
+export PYTHON_LAUNCHER=${PYTHON_LAUNCHER:-/opt/soft/python/bin/python3}
+export HIVE_HOME=${HIVE_HOME:-/opt/soft/hive}
+export FLINK_HOME=${FLINK_HOME:-/opt/soft/flink}
+export DATAX_LAUNCHER=${DATAX_LAUNCHER:-/opt/soft/datax/bin/datax.py}
+
+export PATH=$HADOOP_HOME/bin:$SPARK_HOME/bin:$PYTHON_LAUNCHER:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_LAUNCHER:$PATH
+
+export MASTER_RESERVED_MEMORY=0.01
+export WORKER_RESERVED_MEMORY=0.01
+
+# applicationId auto collection related configuration, the following configurations are unnecessary if setting appId.collect=log
+#export HADOOP_CLASSPATH=`hadoop classpath`:${DOLPHINSCHEDULER_HOME}/tools/libs/*
+#export SPARK_DIST_CLASSPATH=$HADOOP_CLASSPATH:$SPARK_DIST_CLASS_PATH
+#export HADOOP_CLIENT_OPTS="-javaagent:${DOLPHINSCHEDULER_HOME}/tools/libs/aspectjweaver-1.9.7.jar":$HADOOP_CLIENT_OPTS
+#export SPARK_SUBMIT_OPTS="-javaagent:${DOLPHINSCHEDULER_HOME}/tools/libs/aspectjweaver-1.9.7.jar":$SPARK_SUBMIT_OPTS
+#export FLINK_ENV_JAVA_OPTS="-javaagent:${DOLPHINSCHEDULER_HOME}/tools/libs/aspectjweaver-1.9.7.jar":$FLINK_ENV_JAVA_OPTS

.github/workflows/cluster-test/mysql_with_mysql_registry/install_env.sh

@@ -0,0 +1,58 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# ---------------------------------------------------------
+# INSTALL MACHINE
+# ---------------------------------------------------------
+# A comma separated list of machine hostname or IP would be installed DolphinScheduler,
+# including master, worker, api, alert. If you want to deploy in pseudo-distributed
+# mode, just write a pseudo-distributed hostname
+# Example for hostnames: ips="ds1,ds2,ds3,ds4,ds5", Example for IPs: ips="192.168.8.1,192.168.8.2,192.168.8.3,192.168.8.4,192.168.8.5"
+ips=${ips:-"localhost"}
+
+# Port of SSH protocol, default value is 22. For now we only support same port in all `ips` machine
+# modify it if you use different ssh port
+sshPort=${sshPort:-"22"}
+
+# A comma separated list of machine hostname or IP would be installed Master server, it
+# must be a subset of configuration `ips`.
+# Example for hostnames: masters="ds1,ds2", Example for IPs: masters="192.168.8.1,192.168.8.2"
+masters=${masters:-"localhost"}
+
+# A comma separated list of machine <hostname>:<workerGroup> or <IP>:<workerGroup>.All hostname or IP must be a
+# subset of configuration `ips`, And workerGroup have default value as `default`, but we recommend you declare behind the hosts
+# Example for hostnames: workers="ds1:default,ds2:default,ds3:default", Example for IPs: workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
+workers=${workers:-"localhost:default"}
+
+# A comma separated list of machine hostname or IP would be installed Alert server, it
+# must be a subset of configuration `ips`.
+# Example for hostname: alertServer="ds3", Example for IP: alertServer="192.168.8.3"
+alertServer=${alertServer:-"localhost"}
+
+# A comma separated list of machine hostname or IP would be installed API server, it
+# must be a subset of configuration `ips`.
+# Example for hostname: apiServers="ds1", Example for IP: apiServers="192.168.8.1"
+apiServers=${apiServers:-"localhost"}
+
+# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
+# Do not set this configuration same as the current path (pwd)
+installPath=${installPath:-"/root/apache-dolphinscheduler-*-SNAPSHOT-bin"}
+
+# The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running `install.sh`
+# script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
+# to be created by this user
+deployUser=${deployUser:-"dolphinscheduler"}

.github/workflows/cluster-test/mysql_with_mysql_registry/start-job.sh

@@ -18,16 +18,16 @@
 set -euox pipefail
 
 #Start base service containers
-docker-compose -f .github/workflows/cluster-test/mysql/docker-compose-base.yaml up -d
+docker-compose -f .github/workflows/cluster-test/mysql_with_mysql_registry/docker-compose-base.yaml up -d
 
 #Build ds mysql cluster image
-docker build -t jdk8:ds_mysql_cluster -f .github/workflows/cluster-test/mysql/Dockerfile .
+docker build -t jdk8:ds_mysql_cluster -f .github/workflows/cluster-test/mysql_with_mysql_registry/Dockerfile .
 
 #Start ds mysql cluster container
-docker-compose -f .github/workflows/cluster-test/mysql/docker-compose-cluster.yaml up -d
+docker-compose -f .github/workflows/cluster-test/mysql_with_mysql_registry/docker-compose-cluster.yaml up -d
 
 #Running tests
-/bin/bash .github/workflows/cluster-test/mysql/running_test.sh
+/bin/bash .github/workflows/cluster-test/mysql_with_mysql_registry/running_test.sh
 
 #Cleanup
 docker rm -f $(docker ps -aq)

.github/workflows/cluster-test/mysql_with_zookeeper_registry/Dockerfile

@@ -0,0 +1,48 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM eclipse-temurin:8-jre
+
+RUN apt update ; \
+    apt install -y wget default-mysql-client sudo openssh-server netcat-traditional ;
+
+COPY ./apache-dolphinscheduler-*-SNAPSHOT-bin.tar.gz /root
+RUN tar -zxvf /root/apache-dolphinscheduler-*-SNAPSHOT-bin.tar.gz -C ~
+
+RUN mv /root/apache-dolphinscheduler-*-SNAPSHOT-bin /root/apache-dolphinscheduler-test-SNAPSHOT-bin
+
+ENV DOLPHINSCHEDULER_HOME /root/apache-dolphinscheduler-test-SNAPSHOT-bin
+
+#Setting install.sh
+COPY .github/workflows/cluster-test/mysql_with_zookeeper_registry/install_env.sh $DOLPHINSCHEDULER_HOME/bin/env/install_env.sh
+
+#Setting dolphinscheduler_env.sh
+COPY .github/workflows/cluster-test/mysql_with_zookeeper_registry/dolphinscheduler_env.sh $DOLPHINSCHEDULER_HOME/bin/env/dolphinscheduler_env.sh
+
+#Download mysql jar
+ENV MYSQL_URL "https://repo.maven.apache.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar"
+ENV MYSQL_DRIVER "mysql-connector-java-8.0.16.jar"
+RUN wget -O $DOLPHINSCHEDULER_HOME/alert-server/libs/$MYSQL_DRIVER $MYSQL_URL ; \
+cp $DOLPHINSCHEDULER_HOME/alert-server/libs/$MYSQL_DRIVER $DOLPHINSCHEDULER_HOME/api-server/libs/$MYSQL_DRIVER ; \
+cp $DOLPHINSCHEDULER_HOME/alert-server/libs/$MYSQL_DRIVER $DOLPHINSCHEDULER_HOME/master-server/libs/$MYSQL_DRIVER ; \
+cp $DOLPHINSCHEDULER_HOME/alert-server/libs/$MYSQL_DRIVER $DOLPHINSCHEDULER_HOME/worker-server/libs/$MYSQL_DRIVER ; \
+cp $DOLPHINSCHEDULER_HOME/alert-server/libs/$MYSQL_DRIVER $DOLPHINSCHEDULER_HOME/tools/libs/$MYSQL_DRIVER
+
+#Deploy
+COPY .github/workflows/cluster-test/mysql_with_zookeeper_registry/deploy.sh /root/deploy.sh
+
+CMD [ "/bin/bash", "/root/deploy.sh" ]

.github/workflows/cluster-test/mysql_with_zookeeper_registry/docker-compose-cluster.yaml

@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+version: "3"
+
+services:
+  ds:
+    container_name: ds
+    image: jdk8:ds_mysql_cluster
+    restart: always
+    ports:
+      - "12345:12345"
+      - "5679:5679"
+      - "1235:1235"
+      - "50053:50053"

.github/workflows/cluster-test/mysql_with_zookeeper_registry/dolphinscheduler_env.sh

@@ -28,7 +28,6 @@ export SPRING_DATASOURCE_PASSWORD=123456
 # DolphinScheduler server related configuration
 export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
 export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
-export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}
 
 # Registry center configuration, determines the type and link of the registry center
 export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}

.github/workflows/cluster-test/mysql_with_zookeeper_registry/running_test.sh

@@ -0,0 +1,108 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+set -x
+
+
+API_HEALTHCHECK_COMMAND="curl -I -m 10 -o /dev/null -s -w %{http_code} http://0.0.0.0:12345/dolphinscheduler/actuator/health"
+MASTER_HEALTHCHECK_COMMAND="curl -I -m 10 -o /dev/null -s -w %{http_code} http://0.0.0.0:5679/actuator/health"
+WORKER_HEALTHCHECK_COMMAND="curl -I -m 10 -o /dev/null -s -w %{http_code} http://0.0.0.0:1235/actuator/health"
+ALERT_HEALTHCHECK_COMMAND="curl -I -m 10 -o /dev/null -s -w %{http_code} http://0.0.0.0:50053/actuator/health"
+
+#Cluster start health check
+TIMEOUT=180
+START_HEALTHCHECK_EXITCODE=0
+
+for ((i=1; i<=TIMEOUT; i++))
+do
+  MASTER_HTTP_STATUS=$(eval "$MASTER_HEALTHCHECK_COMMAND")
+  WORKER_HTTP_STATUS=$(eval "$WORKER_HEALTHCHECK_COMMAND")
+  API_HTTP_STATUS=$(eval "$API_HEALTHCHECK_COMMAND")
+  ALERT_HTTP_STATUS=$(eval "$ALERT_HEALTHCHECK_COMMAND")
+  if [[ $MASTER_HTTP_STATUS -eq 200 && $WORKER_HTTP_STATUS -eq 200 && $API_HTTP_STATUS -eq 200 && $ALERT_HTTP_STATUS -eq 200 ]];then
+    START_HEALTHCHECK_EXITCODE=0
+  else
+    START_HEALTHCHECK_EXITCODE=2
+  fi
+
+  if [[ $START_HEALTHCHECK_EXITCODE -eq 0 ]];then
+    echo "cluster start health check success"
+    break
+  fi
+
+  if [[ $i -eq $TIMEOUT ]];then
+    if [[ $MASTER_HTTP_STATUS -ne 200 ]];then
+      docker exec -u root ds bash -c "cat /root/apache-dolphinscheduler-*-SNAPSHOT-bin/master-server/logs/dolphinscheduler-master.log"
+      docker exec -u root ds bash -c "cat /root/apache-dolphinscheduler-*-SNAPSHOT-bin/master-server/logs/*.out"
+      echo "master start health check failed"
+    fi
+    if [[ $WORKER_HTTP_STATUS -ne 200 ]]; then
+      docker exec -u root ds bash -c "cat /root/apache-dolphinscheduler-*-SNAPSHOT-bin/worker-server/logs/dolphinscheduler-worker.log"
+      docker exec -u root ds bash -c "cat /root/apache-dolphinscheduler-*-SNAPSHOT-bin/worker-server/logs/*.out"
+      echo "worker start health check failed"
+    fi
+    if [[ $API_HTTP_STATUS -ne 200 ]]; then
+      docker exec -u root ds bash -c "cat /root/apache-dolphinscheduler-*-SNAPSHOT-bin/api-server/logs/dolphinscheduler-api.log"
+      docker exec -u root ds bash -c "cat /root/apache-dolphinscheduler-*-SNAPSHOT-bin/api-server/logs/*.out"
+      echo "api start health check failed"
+    fi
+    if [[ $ALERT_HTTP_STATUS -ne 200 ]]; then
+      docker exec -u root ds bash -c "cat /root/apache-dolphinscheduler-*-SNAPSHOT-bin/alert-server/logs/dolphinscheduler-alert.log"
+      docker exec -u root ds bash -c "cat /root/apache-dolphinscheduler-*-SNAPSHOT-bin/alert-server/logs/*.out"
+      echo "alert start health check failed"
+    fi
+    exit $START_HEALTHCHECK_EXITCODE
+  fi
+  sleep 1
+done
+
+#Stop Cluster
+docker exec -u root ds bash -c "/root/apache-dolphinscheduler-*-SNAPSHOT-bin/bin/stop-all.sh"
+
+#Cluster stop health check
+sleep 5
+MASTER_HTTP_STATUS=$(eval "$MASTER_HEALTHCHECK_COMMAND")
+if [[ $MASTER_HTTP_STATUS -ne 200 ]];then
+  echo "master stop health check success"
+else
+  echo "master stop health check failed"
+  exit 3
+fi
+
+WORKER_HTTP_STATUS=$(eval "$WORKER_HEALTHCHECK_COMMAND")
+if [[ $WORKER_HTTP_STATUS -ne 200 ]];then
+  echo "worker stop health check success"
+else
+  echo "worker stop health check failed"
+  exit 3
+fi
+
+API_HTTP_STATUS=$(eval "$API_HEALTHCHECK_COMMAND")
+if [[ $API_HTTP_STATUS -ne 200 ]];then
+  echo "api stop health check success"
+else
+  echo "api stop health check failed"
+  exit 3
+fi
+
+ALERT_HTTP_STATUS=$(eval "$ALERT_HEALTHCHECK_COMMAND")
+if [[ $ALERT_HTTP_STATUS -ne 200 ]];then
+  echo "alert stop health check success"
+else
+  echo "alert stop health check failed"
+  exit 3
+fi

.github/workflows/cluster-test/mysql_with_zookeeper_registry/start-job.sh

@@ -0,0 +1,33 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+set -euox pipefail
+
+#Start base service containers
+docker-compose -f .github/workflows/cluster-test/mysql_with_zookeeper_registry/docker-compose-base.yaml up -d
+
+#Build ds mysql cluster image
+docker build -t jdk8:ds_mysql_cluster -f .github/workflows/cluster-test/mysql_with_zookeeper_registry/Dockerfile .
+
+#Start ds mysql cluster container
+docker-compose -f .github/workflows/cluster-test/mysql_with_zookeeper_registry/docker-compose-cluster.yaml up -d
+
+#Running tests
+/bin/bash .github/workflows/cluster-test/mysql_with_zookeeper_registry/running_test.sh
+
+#Cleanup
+docker rm -f $(docker ps -aq)

.github/workflows/cluster-test/postgresql_with_postgresql_registry/Dockerfile

@@ -0,0 +1,39 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM eclipse-temurin:8-jre
+
+RUN apt update ; \
+    apt install -y wget sudo openssh-server netcat-traditional ;
+
+COPY ./apache-dolphinscheduler-*-SNAPSHOT-bin.tar.gz /root
+RUN tar -zxvf /root/apache-dolphinscheduler-*-SNAPSHOT-bin.tar.gz -C ~
+
+RUN mv /root/apache-dolphinscheduler-*-SNAPSHOT-bin /root/apache-dolphinscheduler-test-SNAPSHOT-bin
+
+ENV DOLPHINSCHEDULER_HOME /root/apache-dolphinscheduler-test-SNAPSHOT-bin
+
+#Setting install.sh
+COPY .github/workflows/cluster-test/postgresql_with_postgresql_registry/install_env.sh $DOLPHINSCHEDULER_HOME/bin/env/install_env.sh
+
+#Setting dolphinscheduler_env.sh
+COPY .github/workflows/cluster-test/postgresql_with_postgresql_registry/dolphinscheduler_env.sh $DOLPHINSCHEDULER_HOME/bin/env/dolphinscheduler_env.sh
+
+#Deploy
+COPY .github/workflows/cluster-test/postgresql_with_postgresql_registry/deploy.sh /root/deploy.sh
+
+CMD [ "/bin/bash", "/root/deploy.sh" ]

.github/workflows/cluster-test/postgresql_with_postgresql_registry/deploy.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+set -euox pipefail
+
+
+USER=root
+
+#Sudo
+sed -i '$a'$USER'  ALL=(ALL)  NOPASSWD: NOPASSWD: ALL' /etc/sudoers
+sed -i 's/Defaults    requirett/#Defaults    requirett/g' /etc/sudoers
+
+#SSH
+ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
+cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
+chmod 600 ~/.ssh/authorized_keys
+service ssh start
+
+#Init schema
+/bin/bash $DOLPHINSCHEDULER_HOME/tools/bin/upgrade-schema.sh
+/bin/bash $DOLPHINSCHEDULER_HOME/tools/bin/initialize-jdbc-registry.sh
+
+#Start Cluster
+/bin/bash $DOLPHINSCHEDULER_HOME/bin/start-all.sh
+
+#Keep running
+tail -f /dev/null

.github/workflows/cluster-test/postgresql_with_postgresql_registry/docker-compose-base.yaml

@@ -0,0 +1,35 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+version: "3"
+
+services:
+  postgres:
+    container_name: postgres
+    image: postgres:14.1
+    restart: always
+    environment:
+      POSTGRES_PASSWORD: postgres
+      POSTGRES_DB: dolphinscheduler
+    ports:
+      - "5432:5432"
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U postgres"]
+      interval: 5s
+      timeout: 60s
+      retries: 120
+

.github/workflows/cluster-test/postgresql_with_postgresql_registry/dolphinscheduler_env.sh

@@ -0,0 +1,57 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# JAVA_HOME, will use it to start DolphinScheduler server
+export JAVA_HOME=${JAVA_HOME:-/opt/java/openjdk}
+
+# Database related configuration, set database type, username and password
+export DATABASE=${DATABASE:-postgresql}
+export SPRING_PROFILES_ACTIVE=${DATABASE}
+export SPRING_DATASOURCE_URL="jdbc:postgresql://postgres:5432/dolphinscheduler"
+export SPRING_DATASOURCE_USERNAME=postgres
+export SPRING_DATASOURCE_PASSWORD=postgres
+
+# DolphinScheduler server related configuration
+export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
+export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
+
+# Registry center configuration, determines the type and link of the registry center
+export REGISTRY_TYPE=jdbc
+export REGISTRY_HIKARI_CONFIG_JDBC_URL="jdbc:postgresql://postgres:5432/dolphinscheduler"
+export REGISTRY_HIKARI_CONFIG_USERNAME=postgres
+export REGISTRY_HIKARI_CONFIG_PASSWORD=postgres
+
+# Tasks related configurations, need to change the configuration if you use the related tasks.
+export HADOOP_HOME=${HADOOP_HOME:-/opt/soft/hadoop}
+export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/soft/hadoop/etc/hadoop}
+export SPARK_HOME=${SPARK_HOME:-/opt/soft/spark}
+export PYTHON_LAUNCHER=${PYTHON_LAUNCHER:-/opt/soft/python/bin/python3}
+export HIVE_HOME=${HIVE_HOME:-/opt/soft/hive}
+export FLINK_HOME=${FLINK_HOME:-/opt/soft/flink}
+export DATAX_LAUNCHER=${DATAX_LAUNCHER:-/opt/soft/datax/bin/datax.py}
+
+export PATH=$HADOOP_HOME/bin:$SPARK_HOME/bin:$PYTHON_LAUNCHER:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_LAUNCHER:$PATH
+
+export MASTER_RESERVED_MEMORY=0.01
+export WORKER_RESERVED_MEMORY=0.01
+
+# applicationId auto collection related configuration, the following configurations are unnecessary if setting appId.collect=log
+#export HADOOP_CLASSPATH=`hadoop classpath`:${DOLPHINSCHEDULER_HOME}/tools/libs/*
+#export SPARK_DIST_CLASSPATH=$HADOOP_CLASSPATH:$SPARK_DIST_CLASS_PATH
+#export HADOOP_CLIENT_OPTS="-javaagent:${DOLPHINSCHEDULER_HOME}/tools/libs/aspectjweaver-1.9.7.jar":$HADOOP_CLIENT_OPTS
+#export SPARK_SUBMIT_OPTS="-javaagent:${DOLPHINSCHEDULER_HOME}/tools/libs/aspectjweaver-1.9.7.jar":$SPARK_SUBMIT_OPTS
+#export FLINK_ENV_JAVA_OPTS="-javaagent:${DOLPHINSCHEDULER_HOME}/tools/libs/aspectjweaver-1.9.7.jar":$FLINK_ENV_JAVA_OPTS

.github/workflows/cluster-test/postgresql_with_postgresql_registry/install_env.sh

@@ -0,0 +1,58 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# ---------------------------------------------------------
+# INSTALL MACHINE
+# ---------------------------------------------------------
+# A comma separated list of machine hostname or IP would be installed DolphinScheduler,
+# including master, worker, api, alert. If you want to deploy in pseudo-distributed
+# mode, just write a pseudo-distributed hostname
+# Example for hostnames: ips="ds1,ds2,ds3,ds4,ds5", Example for IPs: ips="192.168.8.1,192.168.8.2,192.168.8.3,192.168.8.4,192.168.8.5"
+ips=${ips:-"localhost"}
+
+# Port of SSH protocol, default value is 22. For now we only support same port in all `ips` machine
+# modify it if you use different ssh port
+sshPort=${sshPort:-"22"}
+
+# A comma separated list of machine hostname or IP would be installed Master server, it
+# must be a subset of configuration `ips`.
+# Example for hostnames: masters="ds1,ds2", Example for IPs: masters="192.168.8.1,192.168.8.2"
+masters=${masters:-"localhost"}
+
+# A comma separated list of machine <hostname>:<workerGroup> or <IP>:<workerGroup>.All hostname or IP must be a
+# subset of configuration `ips`, And workerGroup have default value as `default`, but we recommend you declare behind the hosts
+# Example for hostnames: workers="ds1:default,ds2:default,ds3:default", Example for IPs: workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
+workers=${workers:-"localhost:default"}
+
+# A comma separated list of machine hostname or IP would be installed Alert server, it
+# must be a subset of configuration `ips`.
+# Example for hostname: alertServer="ds3", Example for IP: alertServer="192.168.8.3"
+alertServer=${alertServer:-"localhost"}
+
+# A comma separated list of machine hostname or IP would be installed API server, it
+# must be a subset of configuration `ips`.
+# Example for hostname: apiServers="ds1", Example for IP: apiServers="192.168.8.1"
+apiServers=${apiServers:-"localhost"}
+
+# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
+# Do not set this configuration same as the current path (pwd)
+installPath=${installPath:-"/root/apache-dolphinscheduler-*-SNAPSHOT-bin"}
+
+# The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running `install.sh`
+# script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
+# to be created by this user
+deployUser=${deployUser:-"dolphinscheduler"}

.github/workflows/cluster-test/postgresql_with_postgresql_registry/start-job.sh

@@ -0,0 +1,33 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+set -euox pipefail
+
+#Start base service containers
+docker-compose -f .github/workflows/cluster-test/postgresql_with_postgresql_registry/docker-compose-base.yaml up -d
+
+#Build ds postgresql cluster image
+docker build -t jdk8:ds_postgresql_cluster -f .github/workflows/cluster-test/postgresql_with_postgresql_registry/Dockerfile .
+
+#Start ds postgresql cluster container
+docker-compose -f .github/workflows/cluster-test/postgresql_with_postgresql_registry/docker-compose-cluster.yaml up -d
+
+#Running tests
+/bin/bash .github/workflows/cluster-test/postgresql_with_postgresql_registry/running_test.sh
+
+#Cleanup
+docker rm -f $(docker ps -aq)

.github/workflows/cluster-test/postgresql_with_zookeeper_registry/Dockerfile

@@ -28,12 +28,12 @@ RUN mv /root/apache-dolphinscheduler-*-SNAPSHOT-bin /root/apache-dolphinschedule
 ENV DOLPHINSCHEDULER_HOME /root/apache-dolphinscheduler-test-SNAPSHOT-bin
 
 #Setting install.sh
-COPY .github/workflows/cluster-test/postgresql/install_env.sh $DOLPHINSCHEDULER_HOME/bin/env/install_env.sh
+COPY .github/workflows/cluster-test/postgresql_with_zookeeper_registry/install_env.sh $DOLPHINSCHEDULER_HOME/bin/env/install_env.sh
 
 #Setting dolphinscheduler_env.sh
-COPY .github/workflows/cluster-test/postgresql/dolphinscheduler_env.sh $DOLPHINSCHEDULER_HOME/bin/env/dolphinscheduler_env.sh
+COPY .github/workflows/cluster-test/postgresql_with_zookeeper_registry/dolphinscheduler_env.sh $DOLPHINSCHEDULER_HOME/bin/env/dolphinscheduler_env.sh
 
 #Deploy
-COPY .github/workflows/cluster-test/postgresql/deploy.sh /root/deploy.sh
+COPY .github/workflows/cluster-test/postgresql_with_zookeeper_registry/deploy.sh /root/deploy.sh
 
 CMD [ "/bin/bash", "/root/deploy.sh" ]

.github/workflows/cluster-test/postgresql_with_zookeeper_registry/docker-compose-cluster.yaml

@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+version: "3"
+
+services:
+  ds:
+    container_name: ds
+    image: jdk8:ds_postgresql_cluster
+    restart: always
+    ports:
+      - "12345:12345"
+      - "5679:5679"
+      - "1235:1235"
+      - "50053:50053"

.github/workflows/cluster-test/postgresql_with_zookeeper_registry/dolphinscheduler_env.sh

@@ -28,7 +28,6 @@ export SPRING_DATASOURCE_PASSWORD=postgres
 # DolphinScheduler server related configuration
 export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
 export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
-export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}
 
 # Registry center configuration, determines the type and link of the registry center
 export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}

.github/workflows/cluster-test/postgresql_with_zookeeper_registry/running_test.sh

@@ -0,0 +1,109 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+set -x
+
+
+API_HEALTHCHECK_COMMAND="curl -I -m 10 -o /dev/null -s -w %{http_code} http://0.0.0.0:12345/dolphinscheduler/actuator/health"
+MASTER_HEALTHCHECK_COMMAND="curl -I -m 10 -o /dev/null -s -w %{http_code} http://0.0.0.0:5679/actuator/health"
+WORKER_HEALTHCHECK_COMMAND="curl -I -m 10 -o /dev/null -s -w %{http_code} http://0.0.0.0:1235/actuator/health"
+ALERT_HEALTHCHECK_COMMAND="curl -I -m 10 -o /dev/null -s -w %{http_code} http://0.0.0.0:50053/actuator/health"
+
+#Cluster start health check
+TIMEOUT=180
+START_HEALTHCHECK_EXITCODE=0
+
+for ((i=1; i<=TIMEOUT; i++))
+do
+  MASTER_HTTP_STATUS=$(eval "$MASTER_HEALTHCHECK_COMMAND")
+  WORKER_HTTP_STATUS=$(eval "$WORKER_HEALTHCHECK_COMMAND")
+  API_HTTP_STATUS=$(eval "$API_HEALTHCHECK_COMMAND")
+  ALERT_HTTP_STATUS=$(eval "$ALERT_HEALTHCHECK_COMMAND")
+  if [[ $MASTER_HTTP_STATUS -eq 200 && $WORKER_HTTP_STATUS -eq 200 && $API_HTTP_STATUS -eq 200 && $ALERT_HTTP_STATUS -eq 200 ]];then
+    START_HEALTHCHECK_EXITCODE=0
+  else
+    START_HEALTHCHECK_EXITCODE=2
+  fi
+
+  if [[ $START_HEALTHCHECK_EXITCODE -eq 0 ]];then
+    echo "cluster start health check success"
+    break
+  fi
+
+  if [[ $i -eq $TIMEOUT ]];then
+      if [[ $MASTER_HTTP_STATUS -ne 200 ]];then
+        docker exec -u root ds bash -c "cat /root/apache-dolphinscheduler-*-SNAPSHOT-bin/master-server/logs/dolphinscheduler-master.log"
+        docker exec -u root ds bash -c "cat /root/apache-dolphinscheduler-*-SNAPSHOT-bin/master-server/logs/*.out"
+        echo "master start health check failed"
+      fi
+      if [[ $WORKER_HTTP_STATUS -ne 200 ]]; then
+        docker exec -u root ds bash -c "cat /root/apache-dolphinscheduler-*-SNAPSHOT-bin/worker-server/logs/dolphinscheduler-worker.log"
+        docker exec -u root ds bash -c "cat /root/apache-dolphinscheduler-*-SNAPSHOT-bin/worker-server/logs/*.out"
+        echo "worker start health check failed"
+      fi
+      if [[ $API_HTTP_STATUS -ne 200 ]]; then
+        docker exec -u root ds bash -c "cat /root/apache-dolphinscheduler-*-SNAPSHOT-bin/api-server/logs/dolphinscheduler-api.log"
+        docker exec -u root ds bash -c "cat /root/apache-dolphinscheduler-*-SNAPSHOT-bin/api-server/logs/*.out"
+        echo "api start health check failed"
+      fi
+      if [[ $ALERT_HTTP_STATUS -ne 200 ]]; then
+        docker exec -u root ds bash -c "cat /root/apache-dolphinscheduler-*-SNAPSHOT-bin/alert-server/logs/dolphinscheduler-alert.log"
+        docker exec -u root ds bash -c "cat /root/apache-dolphinscheduler-*-SNAPSHOT-bin/alert-server/logs/*.out"
+        echo "alert start health check failed"
+      fi
+      exit $START_HEALTHCHECK_EXITCODE
+    fi
+
+  sleep 1
+done
+
+#Stop Cluster
+docker exec -u root ds bash -c "/root/apache-dolphinscheduler-*-SNAPSHOT-bin/bin/stop-all.sh"
+
+#Cluster stop health check
+sleep 5
+MASTER_HTTP_STATUS=$(eval "$MASTER_HEALTHCHECK_COMMAND")
+if [[ $MASTER_HTTP_STATUS -ne 200 ]];then
+  echo "master stop health check success"
+else
+  echo "master stop health check failed"
+  exit 3
+fi
+
+WORKER_HTTP_STATUS=$(eval "$WORKER_HEALTHCHECK_COMMAND")
+if [[ $WORKER_HTTP_STATUS -ne 200 ]];then
+  echo "worker stop health check success"
+else
+  echo "worker stop health check failed"
+  exit 3
+fi
+
+API_HTTP_STATUS=$(eval "$API_HEALTHCHECK_COMMAND")
+if [[ $API_HTTP_STATUS -ne 200 ]];then
+  echo "api stop health check success"
+else
+  echo "api stop health check failed"
+  exit 3
+fi
+
+ALERT_HTTP_STATUS=$(eval "$ALERT_HEALTHCHECK_COMMAND")
+if [[ $ALERT_HTTP_STATUS -ne 200 ]];then
+  echo "alert stop health check success"
+else
+  echo "alert stop health check failed"
+  exit 3
+fi

.github/workflows/cluster-test/postgresql_with_zookeeper_registry/start-job.sh

@@ -18,16 +18,16 @@
 set -euox pipefail
 
 #Start base service containers
-docker-compose -f .github/workflows/cluster-test/postgresql/docker-compose-base.yaml up -d
+docker-compose -f .github/workflows/cluster-test/postgresql_with_zookeeper_registry/docker-compose-base.yaml up -d
 
 #Build ds postgresql cluster image
-docker build -t jdk8:ds_postgresql_cluster -f .github/workflows/cluster-test/postgresql/Dockerfile .
+docker build -t jdk8:ds_postgresql_cluster -f .github/workflows/cluster-test/postgresql_with_zookeeper_registry/Dockerfile .
 
 #Start ds postgresql cluster container
-docker-compose -f .github/workflows/cluster-test/postgresql/docker-compose-cluster.yaml up -d
+docker-compose -f .github/workflows/cluster-test/postgresql_with_zookeeper_registry/docker-compose-cluster.yaml up -d
 
 #Running tests
-/bin/bash .github/workflows/cluster-test/postgresql/running_test.sh
+/bin/bash .github/workflows/cluster-test/postgresql_with_zookeeper_registry/running_test.sh
 
 #Cleanup
 docker rm -f $(docker ps -aq)

.github/workflows/docs.yml

@@ -55,13 +55,50 @@ jobs:
     timeout-minutes: 30
     steps:
       - uses: actions/checkout@v2
-      - run: sudo npm install -g markdown-link-check@3.10.0
+      - run: sudo npm install -g markdown-link-check@3.11.2
+      - run: sudo apt install plocate -y
       # NOTE: Change command from `find . -name "*.md"` to `find . -not -path "*/node_modules/*" -not -path "*/.tox/*" -name "*.md"`
       # if you want to run check locally
       - run: |
-          for file in $(find . -name "*.md" -not \( -path ./deploy/terraform/aws/README.md -prune \)); do
-            markdown-link-check -c .dlc.json -q "$file"
+          for file in $(locate "$PWD*/*.md" | grep -v ./deploy/terraform/aws/README.md); do
+            markdown-link-check -c .dlc.json -q "$file" &
           done
+          wait
+  paths-filter:
+    name: Helm-Doc-Path-Filter
+    runs-on: ubuntu-latest
+    outputs:
+      helm-doc: ${{ steps.filter.outputs.helm-doc }}
+    steps:
+      - uses: actions/checkout@v2
+      - uses: dorny/paths-filter@b2feaf19c27470162a626bd6fa8438ae5b263721
+        id: filter
+        with:
+          filters: |
+            helm-doc:
+              - 'deploy/**'
+  helm-doc:
+    name: Helm-Doc-Execute
+    needs: paths-filter
+    if: ${{ (needs.paths-filter.outputs.helm-doc == 'true') || (github.event_name == 'push') }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+      - name: Generating helm-doc
+        run: |
+          ./mvnw validate -P helm-doc -pl :dolphinscheduler
+      - name: Check helm-doc
+        run: |
+          DIFF=$(git diff ${GITHUB_WORKSPACE}/deploy/kubernetes/*md)
+          if [ ! -z "$DIFF" ]; then
+              echo "###### ERROR: helm-doc is not up to date ######"
+              echo "Please execute './mvnw validate -P helm-doc -pl :dolphinscheduler' in your clone, of your fork, of the project, and commit an updated deploy/kubernetes/README.md for the chart."
+              echo "###### ERROR: helm-doc is not up to date ######"
+          fi
+          git diff --exit-code
   result:
     name: Docs
     runs-on: ubuntu-latest
@@ -70,9 +107,13 @@ jobs:
       - style
       - img-check
       - dead-link
-    if: success()
+    if: always()
     steps:
-      - name: success
+      - name: Status
         run: |
-          echo "Docs check success"
-          exit 0
+          if [[ ${{ contains(needs.*.result, 'failure') }} == 'true' || ${{ contains(needs.*.result, 'cancelled') }} == 'true' ]]; then
+            echo "Build Failed!"
+            exit 1
+          else
+            echo "Build Success!"
+          fi

.github/workflows/e2e-k8s.yml

@@ -70,8 +70,8 @@ jobs:
       - name: Create k8s Kind Cluster
         run: |
           # install kubectl
-          curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
-          sudo chmod +x kubectl /usr/local/bin/kubectl
+          curl -LO "https://dl.k8s.io/release/v1.28.3/bin/linux/amd64/kubectl"
+          sudo chmod +x kubectl
           sudo mv kubectl /usr/local/bin/kubectl
 
           # install kind

.github/workflows/e2e.yml

@@ -106,6 +106,10 @@ jobs:
             class: org.apache.dolphinscheduler.e2e.cases.TokenE2ETest
           - name: Workflow
             class: org.apache.dolphinscheduler.e2e.cases.WorkflowE2ETest
+          - name: WorkflowHttp
+            class: org.apache.dolphinscheduler.e2e.cases.WorkflowHttpTaskE2ETest
+          - name: WorkflowJava
+            class: org.apache.dolphinscheduler.e2e.cases.WorkflowJavaTaskE2ETest
 #          - name: WorkflowForSwitch
 #            class: org.apache.dolphinscheduler.e2e.cases.WorkflowSwitchE2ETest
           - name: FileManage

.github/workflows/frontend.yml

@@ -25,9 +25,6 @@ on:
       - '.github/workflows/frontend.yml'
       - 'dolphinscheduler-ui/**'
   pull_request:
-    paths:
-      - '.github/workflows/frontend.yml'
-      - 'dolphinscheduler-ui/**'
 
 defaults:
   run:
@@ -38,10 +35,25 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
+  paths-filter:
+    name: Frontend-Path-Filter
+    runs-on: ubuntu-latest
+    outputs:
+      not-ignore: ${{ steps.filter.outputs.not-ignore }}
+    steps:
+      - uses: actions/checkout@v2
+      - uses: dorny/paths-filter@b2feaf19c27470162a626bd6fa8438ae5b263721
+        id: filter
+        with:
+          filters: |
+            not-ignore:
+              - '!(docs/**)'
   build:
     name: Build
+    needs: paths-filter
+    if: ${{ (needs.paths-filter.outputs.not-ignore == 'true') || (github.event_name == 'push') }}
     runs-on: ${{ matrix.os }}
-    timeout-minutes: 20
+    timeout-minutes: 30
     strategy:
       matrix:
         os: [ ubuntu-latest, macos-latest ]
@@ -56,9 +68,28 @@ jobs:
         uses: actions/setup-node@v2
         with:
           node-version: 16
-      - name: Compile and Build
+      - name: Compile and Build on ${{ matrix.os }}
         run: |
+          set -x
           npm install pnpm@7 -g
           pnpm install
           pnpm run lint
           pnpm run build:prod
+  result:
+    name: Frontend Build
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    needs: [ build, paths-filter ]
+    if: always()
+    steps:
+      - uses: actions/checkout@v2
+      - name: Status
+        run: |
+          if [[ ${{ needs.paths-filter.outputs.not-ignore }} == 'false' && ${{ github.event_name }} == 'pull_request' ]]; then
+            echo "Skip Frontend Build!"
+            exit 0
+          fi
+          if [[ ${{ needs.build.result }} != 'success' ]]; then
+            echo "Frontend Build Failed!"
+            exit -1
+          fi

.github/workflows/mergeable.yml

@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+---
+#version: 2
+
+on:
+  pull_request:
+
+name: "Mergeable"
+
+jobs:
+  result:
+    name: "Mergeable: milestone-label-check"
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    steps:
+      - name: Status
+        run: |
+          echo "Temporary skipping this check"
+
+#mergeable:
+#  # we can not use `pull_request.*` which including event `pull_request.labeled`, according to https://github.com/mergeability/mergeable/issues/643,
+#  # otherwise mergeable will keep add or remove label endless, we just need this CI act like the default behavior as
+#  # GitHub action workflow `pull_requests` https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request like,
+#  # which only trigger runs when a pull_request event's activity type is opened, synchronize, or reopened
+#  - when: pull_request.opened, pull_request.reopened, pull_request.synchronize
+#    name: sync-sql-ddl
+#    validate:
+#      # Sql files must change synchronize
+#      - do: dependent
+#        files:
+#          - 'dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_h2.sql'
+#          - 'dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql'
+#          - 'dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_postgresql.sql'
+#        message: 'Sql files not change synchronize'
+#    # Add labels 'sql not sync' and comment to reviewers if Sql files not change synchronize
+#    fail:
+#      - do: comment
+#        payload:
+#          body: >
+#            :warning: This PR do not change database DDL synchronize.
+#        leave_old_comment: false
+#      - do: labels
+#        add: 'sql not sync'
+#    # Remove labels 'sql not sync' if pass
+#    pass:
+#      - do: labels
+#        delete: 'sql not sync'
+#
+#  - when: pull_request.*
+#    name: milestone-label-check
+#    validate:
+#      - do: milestone
+#        no_empty:
+#          enabled: false # Cannot be empty when true.
+#          message: 'Milestone is required and cannot be empty.'
+#      - do: label
+#        and:
+#          - must_include:
+#              regex: 'feature|bug|improvement|document|chore|revert'
+#              message: 'Label must include one of the following: `feature`, `bug`, `improvement`, `document`, `chore`, `revert`'
+#          - must_include:
+#              regex: 'ready-to-merge'
+#              message: 'Please check if there are PRs that already have a `ready-to-merge` label and can be merged, if exists please merge them first.'

.github/workflows/unit-test.yml

@@ -76,7 +76,7 @@ jobs:
           restore-keys: ${{ runner.os }}-maven-
 
       - name: Run Unit tests
-        run: ./mvnw clean verify -B -Dmaven.test.skip=false -Dspotless.skip=true
+        run: ./mvnw clean verify -B -Dmaven.test.skip=false -Dspotless.skip=true -DskipUT=false
       - name: Upload coverage report to codecov
         run: CODECOV_TOKEN="09c2663f-b091-4258-8a47-c981827eb29a" bash <(curl -s https://codecov.io/bash)
 
@@ -99,22 +99,11 @@ jobs:
           -Dsonar.login=e4058004bc6be89decf558ac819aa1ecbee57682
           -Dsonar.exclusions=,dolphinscheduler-ui/src/**/i18n/locale/*.js,dolphinscheduler-microbench/src/**/*
           -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.httpconnectionManager.ttlSeconds=120
+          -DskipUT=true
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
 
-      - name: Collect logs
-        continue-on-error: true
-        run: |
-          mkdir -p ${LOG_DIR}
-          docker-compose -f $(pwd)/docker/docker-swarm/docker-compose.yml logs dolphinscheduler-postgresql > ${LOG_DIR}/db.txt
-
-      - name: Upload logs
-        uses: actions/upload-artifact@v2
-        continue-on-error: true
-        with:
-          name: unit-test-logs
-          path: ${LOG_DIR}
   result:
     name: Unit Test
     runs-on: ubuntu-latest

.licenserc.yaml

@@ -50,5 +50,6 @@ header:
     - tools/dependencies/known-dependencies.txt
     - '**/banner.txt'
     - '.terraform.lock.hcl'
+    - deploy/kubernetes/dolphinscheduler/README.md.gotmpl
 
   comment: on-failure

README.md

@@ -3,7 +3,7 @@
 [![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html)
 ![codecov](https://codecov.io/gh/apache/dolphinscheduler/branch/dev/graph/badge.svg)
 [![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=apache-dolphinscheduler&metric=alert_status)](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler)
-[![Twitter Follow](https://img.shields.io/twitter/follow/dolphinschedule.svg?style=social&label=Follow)](https://twitter.com/dolphinschedule)
+[![Twitter Follow](https://img.shields.io/twitter/follow/dolphinschedule.svg?style=social&label=Follow)](https://twitter.com/dolphinschedule) <!-- markdown-link-check-disable-line -->
 [![Slack Status](https://img.shields.io/badge/slack-join_chat-white.svg?logo=slack&style=social)](https://s.apache.org/dolphinscheduler-slack)
 [![CN doc](https://img.shields.io/badge/文档-中文版-blue.svg)](README_zh_CN.md)
 
@@ -31,6 +31,8 @@ The key features for DolphinScheduler are as follows:
   - Want to [start with Docker](https://dolphinscheduler.apache.org/en-us/docs/3.1.5/guide/start/docker)
 - For Kubernetes
   - [Start with Kubernetes](https://dolphinscheduler.apache.org/en-us/docs/3.1.5/guide/installation/kubernetes)
+- For Terraform
+  - [Start with Terraform](deploy/terraform/README.md) 
 
 ## User Interface Screenshots
 
@@ -63,15 +65,14 @@ find the good first issue in [here](https://github.com/apache/dolphinscheduler/c
 Welcome to join the Apache DolphinScheduler community by:
 
 - Join the [DolphinScheduler Slack](https://s.apache.org/dolphinscheduler-slack) to keep in touch with the community
-- Follow the [DolphinScheduler Twitter](https://twitter.com/dolphinschedule) and get the latest news
-- Subscribe DolphinScheduler mail list, users@dolphinscheduler.apache.org for user and dev@dolphinscheduler.apache.org for developer
+- Follow the [DolphinScheduler Twitter](https://twitter.com/dolphinschedule) and get the latest news <!-- markdown-link-check-disable-line -->
+- Subscribe DolphinScheduler mail list, [users@dolphinscheduler.apache.org](mailto:users-subscribe@dolphinscheduler.apache.org) for user and [dev@dolphinscheduler.apache.org](mailto:dev-subscribe@dolphinscheduler.apache.org) for developer
 
 # Landscapes
 
 <p align="center">
 <br/><br/>
-<img src="https://landscape.cncf.io/images/left-logo.svg" width="150"/>&nbsp;&nbsp;<img src="https://landscape.cncf.io/images/right-logo.svg" width="200"/>
+<img src="./images/cncf-landscape-white-bg.jpg" width="175" alt="cncf-landscape"/>&nbsp;&nbsp;<img src="./images/cncf-white-bg.jpg" width="200" alt="cncf-logo"/>
 <br/><br/>
-DolphinScheduler enriches the <a href="https://landscape.cncf.io/?landscape=observability-and-analysis&license=apache-license-2-0">CNCF CLOUD NATIVE Landscape.</a >
-
+DolphinScheduler enriches the <a href="https://landscape.cncf.io/?item=orchestration-management--scheduling-orchestration--dolphinscheduler">CNCF CLOUD NATIVE Landscape.</a >
 </p >

README_zh_CN.md

@@ -3,7 +3,7 @@
 [![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html)
 [![codecov](https://codecov.io/gh/apache/dolphinscheduler/branch/dev/graph/badge.svg)]()
 [![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=apache-dolphinscheduler&metric=alert_status)](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler)
-[![Twitter Follow](https://img.shields.io/twitter/follow/dolphinschedule.svg?style=social&label=Follow)](https://twitter.com/dolphinschedule)
+[![Twitter Follow](https://img.shields.io/twitter/follow/dolphinschedule.svg?style=social&label=Follow)](https://twitter.com/dolphinschedule) <!-- markdown-link-check-disable-line -->
 [![Slack Status](https://img.shields.io/badge/slack-join_chat-white.svg?logo=slack&style=social)](https://s.apache.org/dolphinscheduler-slack)
 [![EN doc](https://img.shields.io/badge/document-English-blue.svg)](README.md)
 
@@ -61,15 +61,15 @@ DolphinScheduler 的主要特性如下：
 欢迎通过以方式加入社区：
 
 - 加入 [DolphinScheduler Slack](https://s.apache.org/dolphinscheduler-slack)
-- 关注 [DolphinScheduler Twitter](https://twitter.com/dolphinschedule) 来获取最新消息
-- 订阅 DolphinScheduler 邮件列表, 用户订阅 users@dolphinscheduler.apache.org 开发者请订阅 dev@dolphinscheduler.apache.org
+- 关注 [DolphinScheduler Twitter](https://twitter.com/dolphinschedule) 来获取最新消息 <!-- markdown-link-check-disable-line -->
+- 订阅 DolphinScheduler 邮件列表, 用户订阅 [users@dolphinscheduler.apache.org](mailto:users-subscribe@dolphinscheduler.apache.org) 开发者请订阅 [dev@dolphinscheduler.apache.org](mailto:dev-subscribe@dolphinscheduler.apache.org)
 
 # Landscapes
 
 <p align="center">
 <br/><br/>
-<img src="https://landscape.cncf.io/images/left-logo.svg" width="150"/>&nbsp;&nbsp;<img src="https://landscape.cncf.io/images/right-logo.svg" width="200"/>
+<img src="./images/cncf-landscape-white-bg.jpg" width="175" alt="cncf-landscape"/>&nbsp;&nbsp;<img src="./images/cncf-white-bg.jpg" width="200" alt="cncf-logo"/>
 <br/><br/>
-DolphinScheduler enriches the <a href="https://landscape.cncf.io/?landscape=observability-and-analysis&license=apache-license-2-0">CNCF CLOUD NATIVE Landscape.</a >
+DolphinScheduler enriches the <a href="https://landscape.cncf.io/?item=orchestration-management--scheduling-orchestration--dolphinscheduler">CNCF CLOUD NATIVE Landscape.</a >
 
 </p >

deploy/README.md

@@ -2,3 +2,4 @@
 
 - [Start Up DolphinScheduler with Docker](https://dolphinscheduler.apache.org/en-us/docs/3.1.2/guide/start/docker)
 - [Start Up DolphinScheduler with Kubernetes](https://dolphinscheduler.apache.org/en-us/docs/3.1.2/guide/installation/kubernetes)
+- [Start Up DolphinScheduler with Terraform](./terraform/README.md)

deploy/docker/.env

@@ -15,8 +15,8 @@
 # specific language governing permissions and limitations
 # under the License.
 #
-HUB=apache
-TAG=3.2.0
+HUB=ghcr.io/apache/dolphinscheduler
+TAG=latest
 
 TZ=Asia/Shanghai
 DATABASE=postgresql

deploy/kubernetes/dolphinscheduler/.helmignore

@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/

deploy/kubernetes/dolphinscheduler/Chart.yaml

@@ -35,11 +35,11 @@ type: application
 
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
-version: 3.2.0
+version: 3.1.0
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application.
-appVersion: 3.2.0
+appVersion: 3.1.0
 
 dependencies:
 - name: postgresql

deploy/kubernetes/dolphinscheduler/README.md

@@ -0,0 +1,359 @@
+## About DolphinScheduler for Kubernetes
+
+Dolphin Scheduler is a distributed and easy-to-expand visual DAG workflow scheduling system, dedicated to solving the complex dependencies in data processing, making the scheduling system out of the box for data processing.
+
+This chart bootstraps all the components needed to run Apache DolphinScheduler on a Kubernetes Cluster using [Helm](https://helm.sh).
+
+## QuickStart in Kubernetes
+
+Please refer to the [Quick Start in Kubernetes](../../../docs/docs/en/guide/installation/kubernetes.md)
+
+## Values
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| alert.affinity | object | `{}` | Affinity is a group of affinity scheduling rules. If specified, the pod's scheduling constraints. More info: [node-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity) |
+| alert.annotations | object | `{}` | You can use annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata. |
+| alert.enabled | bool | `true` | Enable or disable the Alert-Server component |
+| alert.env.JAVA_OPTS | string | `"-Xms512m -Xmx512m -Xmn256m"` | The jvm options for alert server |
+| alert.livenessProbe | object | `{"enabled":true,"failureThreshold":"3","initialDelaySeconds":"30","periodSeconds":"30","successThreshold":"1","timeoutSeconds":"5"}` | Periodic probe of container liveness. Container will be restarted if the probe fails. More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) |
+| alert.livenessProbe.enabled | bool | `true` | Turn on and off liveness probe |
+| alert.livenessProbe.failureThreshold | string | `"3"` | Minimum consecutive failures for the probe |
+| alert.livenessProbe.initialDelaySeconds | string | `"30"` | Delay before liveness probe is initiated |
+| alert.livenessProbe.periodSeconds | string | `"30"` | How often to perform the probe |
+| alert.livenessProbe.successThreshold | string | `"1"` | Minimum consecutive successes for the probe |
+| alert.livenessProbe.timeoutSeconds | string | `"5"` | When the probe times out |
+| alert.nodeSelector | object | `{}` | NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info: [assign-pod-node](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) |
+| alert.persistentVolumeClaim | object | `{"accessModes":["ReadWriteOnce"],"enabled":false,"storage":"20Gi","storageClassName":"-"}` | PersistentVolumeClaim represents a reference to a PersistentVolumeClaim in the same namespace. More info: [persistentvolumeclaims](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) |
+| alert.persistentVolumeClaim.accessModes | list | `["ReadWriteOnce"]` | `PersistentVolumeClaim` access modes |
+| alert.persistentVolumeClaim.enabled | bool | `false` | Set `alert.persistentVolumeClaim.enabled` to `true` to mount a new volume for `alert` |
+| alert.persistentVolumeClaim.storage | string | `"20Gi"` | `PersistentVolumeClaim` size |
+| alert.persistentVolumeClaim.storageClassName | string | `"-"` | `Alert` logs data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning |
+| alert.readinessProbe | object | `{"enabled":true,"failureThreshold":"3","initialDelaySeconds":"30","periodSeconds":"30","successThreshold":"1","timeoutSeconds":"5"}` | Periodic probe of container service readiness. Container will be removed from service endpoints if the probe fails. More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) |
+| alert.readinessProbe.enabled | bool | `true` | Turn on and off readiness probe |
+| alert.readinessProbe.failureThreshold | string | `"3"` | Minimum consecutive failures for the probe |
+| alert.readinessProbe.initialDelaySeconds | string | `"30"` | Delay before readiness probe is initiated |
+| alert.readinessProbe.periodSeconds | string | `"30"` | How often to perform the probe |
+| alert.readinessProbe.successThreshold | string | `"1"` | Minimum consecutive successes for the probe |
+| alert.readinessProbe.timeoutSeconds | string | `"5"` | When the probe times out |
+| alert.replicas | int | `1` | Number of desired pods. This is a pointer to distinguish between explicit zero and not specified. Defaults to 1. |
+| alert.resources | object | `{}` | Compute Resources required by this container. More info: [manage-resources-containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) |
+| alert.service.annotations | object | `{}` | annotations may need to be set when want to scrapy metrics by prometheus but not install prometheus operator |
+| alert.service.serviceMonitor | object | `{"annotations":{},"enabled":false,"interval":"15s","labels":{},"path":"/actuator/prometheus"}` | serviceMonitor for prometheus operator |
+| alert.service.serviceMonitor.annotations | object | `{}` | serviceMonitor.annotations ServiceMonitor annotations |
+| alert.service.serviceMonitor.enabled | bool | `false` | Enable or disable alert-server serviceMonitor |
+| alert.service.serviceMonitor.interval | string | `"15s"` | serviceMonitor.interval interval at which metrics should be scraped |
+| alert.service.serviceMonitor.labels | object | `{}` | serviceMonitor.labels ServiceMonitor extra labels |
+| alert.service.serviceMonitor.path | string | `"/actuator/prometheus"` | serviceMonitor.path path of the metrics endpoint |
+| alert.strategy | object | `{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"}` | The deployment strategy to use to replace existing pods with new ones. |
+| alert.strategy.rollingUpdate.maxSurge | string | `"25%"` | The maximum number of pods that can be scheduled above the desired number of pods |
+| alert.strategy.rollingUpdate.maxUnavailable | string | `"25%"` | The maximum number of pods that can be unavailable during the update |
+| alert.strategy.type | string | `"RollingUpdate"` | Type of deployment. Can be "Recreate" or "RollingUpdate" |
+| alert.tolerations | list | `[]` | Tolerations are appended (excluding duplicates) to pods running with this RuntimeClass during admission, effectively unioning the set of nodes tolerated by the pod and the RuntimeClass. |
+| api.affinity | object | `{}` | Affinity is a group of affinity scheduling rules. If specified, the pod's scheduling constraints. More info: [node-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity) |
+| api.annotations | object | `{}` | You can use annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata. |
+| api.enabled | bool | `true` | Enable or disable the API-Server component |
+| api.env.JAVA_OPTS | string | `"-Xms512m -Xmx512m -Xmn256m"` | The jvm options for api server |
+| api.livenessProbe | object | `{"enabled":true,"failureThreshold":"3","initialDelaySeconds":"30","periodSeconds":"30","successThreshold":"1","timeoutSeconds":"5"}` | Periodic probe of container liveness. Container will be restarted if the probe fails. More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) |
+| api.livenessProbe.enabled | bool | `true` | Turn on and off liveness probe |
+| api.livenessProbe.failureThreshold | string | `"3"` | Minimum consecutive failures for the probe |
+| api.livenessProbe.initialDelaySeconds | string | `"30"` | Delay before liveness probe is initiated |
+| api.livenessProbe.periodSeconds | string | `"30"` | How often to perform the probe |
+| api.livenessProbe.successThreshold | string | `"1"` | Minimum consecutive successes for the probe |
+| api.livenessProbe.timeoutSeconds | string | `"5"` | When the probe times out |
+| api.nodeSelector | object | `{}` | NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info: [assign-pod-node](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) |
+| api.persistentVolumeClaim | object | `{"accessModes":["ReadWriteOnce"],"enabled":false,"storage":"20Gi","storageClassName":"-"}` | PersistentVolumeClaim represents a reference to a PersistentVolumeClaim in the same namespace. More info: [persistentvolumeclaims](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) |
+| api.persistentVolumeClaim.accessModes | list | `["ReadWriteOnce"]` | `PersistentVolumeClaim` access modes |
+| api.persistentVolumeClaim.enabled | bool | `false` | Set `api.persistentVolumeClaim.enabled` to `true` to mount a new volume for `api` |
+| api.persistentVolumeClaim.storage | string | `"20Gi"` | `PersistentVolumeClaim` size |
+| api.persistentVolumeClaim.storageClassName | string | `"-"` | `api` logs data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning |
+| api.readinessProbe | object | `{"enabled":true,"failureThreshold":"3","initialDelaySeconds":"30","periodSeconds":"30","successThreshold":"1","timeoutSeconds":"5"}` | Periodic probe of container service readiness. Container will be removed from service endpoints if the probe fails. More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) |
+| api.readinessProbe.enabled | bool | `true` | Turn on and off readiness probe |
+| api.readinessProbe.failureThreshold | string | `"3"` | Minimum consecutive failures for the probe |
+| api.readinessProbe.initialDelaySeconds | string | `"30"` | Delay before readiness probe is initiated |
+| api.readinessProbe.periodSeconds | string | `"30"` | How often to perform the probe |
+| api.readinessProbe.successThreshold | string | `"1"` | Minimum consecutive successes for the probe |
+| api.readinessProbe.timeoutSeconds | string | `"5"` | When the probe times out |
+| api.replicas | string | `"1"` | Number of desired pods. This is a pointer to distinguish between explicit zero and not specified. Defaults to 1. |
+| api.resources | object | `{}` | Compute Resources required by this container. More info: [manage-resources-containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) |
+| api.service.annotations | object | `{}` | annotations may need to be set when service.type is LoadBalancer service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:EXAMPLE_CERT |
+| api.service.clusterIP | string | `""` | clusterIP is the IP address of the service and is usually assigned randomly by the master |
+| api.service.externalIPs | list | `[]` | externalIPs is a list of IP addresses for which nodes in the cluster will also accept traffic for this service |
+| api.service.externalName | string | `""` | externalName is the external reference that kubedns or equivalent will return as a CNAME record for this service, requires Type to be ExternalName |
+| api.service.loadBalancerIP | string | `""` | loadBalancerIP when service.type is LoadBalancer. LoadBalancer will get created with the IP specified in this field |
+| api.service.nodePort | string | `""` | nodePort is the port on each node on which this api service is exposed when type=NodePort |
+| api.service.pythonNodePort | string | `""` | pythonNodePort is the port on each node on which this python api service is exposed when type=NodePort |
+| api.service.serviceMonitor | object | `{"annotations":{},"enabled":false,"interval":"15s","labels":{},"path":"/dolphinscheduler/actuator/prometheus"}` | serviceMonitor for prometheus operator |
+| api.service.serviceMonitor.annotations | object | `{}` | serviceMonitor.annotations ServiceMonitor annotations |
+| api.service.serviceMonitor.enabled | bool | `false` | Enable or disable api-server serviceMonitor |
+| api.service.serviceMonitor.interval | string | `"15s"` | serviceMonitor.interval interval at which metrics should be scraped |
+| api.service.serviceMonitor.labels | object | `{}` | serviceMonitor.labels ServiceMonitor extra labels |
+| api.service.serviceMonitor.path | string | `"/dolphinscheduler/actuator/prometheus"` | serviceMonitor.path path of the metrics endpoint |
+| api.service.type | string | `"ClusterIP"` | type determines how the Service is exposed. Defaults to ClusterIP. Valid options are ExternalName, ClusterIP, NodePort, and LoadBalancer |
+| api.strategy | object | `{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"}` | The deployment strategy to use to replace existing pods with new ones. |
+| api.strategy.rollingUpdate.maxSurge | string | `"25%"` | The maximum number of pods that can be scheduled above the desired number of pods |
+| api.strategy.rollingUpdate.maxUnavailable | string | `"25%"` | The maximum number of pods that can be unavailable during the update |
+| api.strategy.type | string | `"RollingUpdate"` | Type of deployment. Can be "Recreate" or "RollingUpdate" |
+| api.taskTypeFilter.enabled | bool | `false` | Enable or disable the task type filter. If set to true, the API-Server will return tasks of a specific type set in api.taskTypeFilter.task Note: This feature only filters tasks to return a specific type on the WebUI. However, you can still create any task that DolphinScheduler supports via the API. |
+| api.taskTypeFilter.task | object | `{}` | ref: [task-type-config.yaml](https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-api/src/main/resources/task-type-config.yaml) |
+| api.tolerations | list | `[]` | Tolerations are appended (excluding duplicates) to pods running with this RuntimeClass during admission, effectively unioning the set of nodes tolerated by the pod and the RuntimeClass. |
+| common.configmap.DATAX_LAUNCHER | string | `"/opt/soft/datax/bin/datax.py"` | Set `DATAX_LAUNCHER` for DolphinScheduler's task environment |
+| common.configmap.DATA_BASEDIR_PATH | string | `"/tmp/dolphinscheduler"` | User data directory path, self configuration, please make sure the directory exists and have read write permissions |
+| common.configmap.DOLPHINSCHEDULER_OPTS | string | `""` | The jvm options for dolphinscheduler, suitable for all servers |
+| common.configmap.FLINK_HOME | string | `"/opt/soft/flink"` | Set `FLINK_HOME` for DolphinScheduler's task environment |
+| common.configmap.HADOOP_CONF_DIR | string | `"/opt/soft/hadoop/etc/hadoop"` | Set `HADOOP_CONF_DIR` for DolphinScheduler's task environment |
+| common.configmap.HADOOP_HOME | string | `"/opt/soft/hadoop"` | Set `HADOOP_HOME` for DolphinScheduler's task environment |
+| common.configmap.HIVE_HOME | string | `"/opt/soft/hive"` | Set `HIVE_HOME` for DolphinScheduler's task environment |
+| common.configmap.JAVA_HOME | string | `"/opt/java/openjdk"` | Set `JAVA_HOME` for DolphinScheduler's task environment |
+| common.configmap.PYTHON_LAUNCHER | string | `"/usr/bin/python/bin/python3"` | Set `PYTHON_LAUNCHER` for DolphinScheduler's task environment |
+| common.configmap.RESOURCE_UPLOAD_PATH | string | `"/dolphinscheduler"` | Resource store on HDFS/S3 path, please make sure the directory exists on hdfs and have read write permissions |
+| common.configmap.SPARK_HOME | string | `"/opt/soft/spark"` | Set `SPARK_HOME` for DolphinScheduler's task environment |
+| common.fsFileResourcePersistence.accessModes | list | `["ReadWriteMany"]` | `PersistentVolumeClaim` access modes, must be `ReadWriteMany` |
+| common.fsFileResourcePersistence.enabled | bool | `false` | Set `common.fsFileResourcePersistence.enabled` to `true` to mount a new file resource volume for `api` and `worker` |
+| common.fsFileResourcePersistence.storage | string | `"20Gi"` | `PersistentVolumeClaim` size |
+| common.fsFileResourcePersistence.storageClassName | string | `"-"` | Resource persistent volume storage class, must support the access mode: `ReadWriteMany` |
+| common.sharedStoragePersistence.accessModes | list | `["ReadWriteMany"]` | `PersistentVolumeClaim` access modes, must be `ReadWriteMany` |
+| common.sharedStoragePersistence.enabled | bool | `false` | Set `common.sharedStoragePersistence.enabled` to `true` to mount a shared storage volume for Hadoop, Spark binary and etc |
+| common.sharedStoragePersistence.mountPath | string | `"/opt/soft"` | The mount path for the shared storage volume |
+| common.sharedStoragePersistence.storage | string | `"20Gi"` | `PersistentVolumeClaim` size |
+| common.sharedStoragePersistence.storageClassName | string | `"-"` | Shared Storage persistent volume storage class, must support the access mode: ReadWriteMany |
+| conf.auto | bool | `false` | auto restart, if true, all components will be restarted automatically after the common configuration is updated. if false, you need to restart the components manually. default is false |
+| conf.common."alert.rpc.port" | int | `50052` | rpc port |
+| conf.common."appId.collect" | string | `"log"` | way to collect applicationId: log, aop |
+| conf.common."aws.credentials.provider.type" | string | `"AWSStaticCredentialsProvider"` |  |
+| conf.common."aws.s3.access.key.id" | string | `"minioadmin"` | The AWS access key. if resource.storage.type=S3, and credentials.provider.type is AWSStaticCredentialsProvider. This configuration is required |
+| conf.common."aws.s3.access.key.secret" | string | `"minioadmin"` | The AWS secret access key. if resource.storage.type=S3, and credentials.provider.type is AWSStaticCredentialsProvider. This configuration is required |
+| conf.common."aws.s3.bucket.name" | string | `"dolphinscheduler"` | The name of the bucket. You need to create them by yourself. Otherwise, the system cannot start. All buckets in Amazon S3 share a single namespace; ensure the bucket is given a unique name. |
+| conf.common."aws.s3.endpoint" | string | `"http://minio:9000"` | You need to set this parameter when private cloud s3. If S3 uses public cloud, you only need to set resource.aws.region or set to the endpoint of a public cloud such as S3.cn-north-1.amazonaws.com.cn |
+| conf.common."aws.s3.region" | string | `"ca-central-1"` | The AWS Region to use. if resource.storage.type=S3, This configuration is required |
+| conf.common."conda.path" | string | `"/opt/anaconda3/etc/profile.d/conda.sh"` | set path of conda.sh |
+| conf.common."data-quality.jar.dir" | string | `nil` | data quality option |
+| conf.common."data.basedir.path" | string | `"/tmp/dolphinscheduler"` | user data local directory path, please make sure the directory exists and have read write permissions |
+| conf.common."datasource.encryption.enable" | bool | `false` | datasource encryption enable |
+| conf.common."datasource.encryption.salt" | string | `"!@#$%^&*"` | datasource encryption salt |
+| conf.common."development.state" | bool | `false` | development state |
+| conf.common."hadoop.security.authentication.startup.state" | bool | `false` | whether to startup kerberos |
+| conf.common."java.security.krb5.conf.path" | string | `"/opt/krb5.conf"` | java.security.krb5.conf path |
+| conf.common."kerberos.expire.time" | int | `2` | kerberos expire time, the unit is hour |
+| conf.common."login.user.keytab.path" | string | `"/opt/hdfs.headless.keytab"` | login user from keytab path |
+| conf.common."login.user.keytab.username" | string | `"hdfs-mycluster@ESZ.COM"` | login user from keytab username |
+| conf.common."ml.mlflow.preset_repository" | string | `"https://github.com/apache/dolphinscheduler-mlflow"` | mlflow task plugin preset repository |
+| conf.common."ml.mlflow.preset_repository_version" | string | `"main"` | mlflow task plugin preset repository version |
+| conf.common."resource.alibaba.cloud.access.key.id" | string | `"<your-access-key-id>"` | alibaba cloud access key id, required if you set resource.storage.type=OSS |
+| conf.common."resource.alibaba.cloud.access.key.secret" | string | `"<your-access-key-secret>"` | alibaba cloud access key secret, required if you set resource.storage.type=OSS |
+| conf.common."resource.alibaba.cloud.oss.bucket.name" | string | `"dolphinscheduler"` | oss bucket name, required if you set resource.storage.type=OSS |
+| conf.common."resource.alibaba.cloud.oss.endpoint" | string | `"https://oss-cn-hangzhou.aliyuncs.com"` | oss bucket endpoint, required if you set resource.storage.type=OSS |
+| conf.common."resource.alibaba.cloud.region" | string | `"cn-hangzhou"` | alibaba cloud region, required if you set resource.storage.type=OSS |
+| conf.common."resource.azure.client.id" | string | `"minioadmin"` | azure storage account name, required if you set resource.storage.type=ABS |
+| conf.common."resource.azure.client.secret" | string | `"minioadmin"` | azure storage account key, required if you set resource.storage.type=ABS |
+| conf.common."resource.azure.subId" | string | `"minioadmin"` | azure storage subId, required if you set resource.storage.type=ABS |
+| conf.common."resource.azure.tenant.id" | string | `"minioadmin"` | azure storage tenantId, required if you set resource.storage.type=ABS |
+| conf.common."resource.hdfs.fs.defaultFS" | string | `"hdfs://mycluster:8020"` | if resource.storage.type=S3, the value like: s3a://dolphinscheduler; if resource.storage.type=HDFS and namenode HA is enabled, you need to copy core-site.xml and hdfs-site.xml to conf dir |
+| conf.common."resource.hdfs.root.user" | string | `"hdfs"` | if resource.storage.type=HDFS, the user must have the permission to create directories under the HDFS root path |
+| conf.common."resource.manager.httpaddress.port" | int | `8088` | resourcemanager port, the default value is 8088 if not specified |
+| conf.common."resource.storage.type" | string | `"S3"` | resource storage type: HDFS, S3, OSS, GCS, ABS, NONE |
+| conf.common."resource.storage.upload.base.path" | string | `"/dolphinscheduler"` | resource store on HDFS/S3 path, resource file will store to this base path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended |
+| conf.common."sudo.enable" | bool | `true` | use sudo or not, if set true, executing user is tenant user and deploy user needs sudo permissions; if set false, executing user is the deploy user and doesn't need sudo permissions |
+| conf.common."support.hive.oneSession" | bool | `false` | Whether hive SQL is executed in the same session |
+| conf.common."task.resource.limit.state" | bool | `false` | Task resource limit state |
+| conf.common."yarn.application.status.address" | string | `"http://ds1:%s/ws/v1/cluster/apps/%s"` | if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single, you only need to replace ds1 to actual resourcemanager hostname |
+| conf.common."yarn.job.history.status.address" | string | `"http://ds1:19888/ws/v1/history/mapreduce/jobs/%s"` | job history status url when application number threshold is reached(default 10000, maybe it was set to 1000) |
+| conf.common."yarn.resourcemanager.ha.rm.ids" | string | `"192.168.xx.xx,192.168.xx.xx"` | if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single, keep this value empty |
+| datasource.profile | string | `"postgresql"` | The profile of datasource |
+| externalDatabase.database | string | `"dolphinscheduler"` | The database of external database |
+| externalDatabase.driverClassName | string | `"org.postgresql.Driver"` | The driverClassName of external database |
+| externalDatabase.enabled | bool | `false` | If exists external database, and set postgresql.enable value to false. external database will be used, otherwise Dolphinscheduler's internal database will be used. |
+| externalDatabase.host | string | `"localhost"` | The host of external database |
+| externalDatabase.params | string | `"characterEncoding=utf8"` | The params of external database |
+| externalDatabase.password | string | `"root"` | The password of external database |
+| externalDatabase.port | string | `"5432"` | The port of external database |
+| externalDatabase.type | string | `"postgresql"` | The type of external database, supported types: postgresql, mysql |
+| externalDatabase.username | string | `"root"` | The username of external database |
+| externalRegistry.registryPluginName | string | `"zookeeper"` | If exists external registry and set `zookeeper.enable` && `registryEtcd.enabled` && `registryJdbc.enabled` to false, specify the external registry plugin name |
+| externalRegistry.registryServers | string | `"127.0.0.1:2181"` | If exists external registry and set `zookeeper.enable` && `registryEtcd.enabled` && `registryJdbc.enabled` to false, specify the external registry servers |
+| image.alert | string | `"dolphinscheduler-alert-server"` | alert-server image |
+| image.api | string | `"dolphinscheduler-api"` | api-server image |
+| image.master | string | `"dolphinscheduler-master"` | master image |
+| image.pullPolicy | string | `"IfNotPresent"` | Image pull policy. Options: Always, Never, IfNotPresent |
+| image.pullSecret | string | `""` | Specify a imagePullSecrets |
+| image.registry | string | `"apache"` | Docker image repository for the DolphinScheduler |
+| image.tag | string | `"latest"` | Docker image version for the DolphinScheduler |
+| image.tools | string | `"dolphinscheduler-tools"` | tools image |
+| image.worker | string | `"dolphinscheduler-worker"` | worker image |
+| ingress.annotations | object | `{}` | Ingress annotations |
+| ingress.enabled | bool | `false` | Enable ingress |
+| ingress.host | string | `"dolphinscheduler.org"` | Ingress host |
+| ingress.path | string | `"/dolphinscheduler"` | Ingress path |
+| ingress.tls.enabled | bool | `false` | Enable ingress tls |
+| ingress.tls.secretName | string | `"dolphinscheduler-tls"` | Ingress tls secret name |
+| initImage | object | `{"busybox":"busybox:1.30.1","pullPolicy":"IfNotPresent"}` | Used to detect whether dolphinscheduler dependent services such as database are ready |
+| initImage.busybox | string | `"busybox:1.30.1"` | Specify initImage repository |
+| initImage.pullPolicy | string | `"IfNotPresent"` | Image pull policy. Options: Always, Never, IfNotPresent |
+| master.affinity | object | `{}` | Affinity is a group of affinity scheduling rules. If specified, the pod's scheduling constraints. More info: [node-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity) |
+| master.annotations | object | `{}` | You can use annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata. |
+| master.enabled | bool | `true` | Enable or disable the Master component |
+| master.env.JAVA_OPTS | string | `"-Xms1g -Xmx1g -Xmn512m"` | The jvm options for master server |
+| master.env.MASTER_DISPATCH_TASK_NUM | string | `"3"` | Master dispatch task number per batch |
+| master.env.MASTER_EXEC_TASK_NUM | string | `"20"` | Master execute task number in parallel per process instance |
+| master.env.MASTER_EXEC_THREADS | string | `"100"` | Master execute thread number to limit process instances |
+| master.env.MASTER_FAILOVER_INTERVAL | string | `"10m"` | Master failover interval, the unit is minute |
+| master.env.MASTER_HEARTBEAT_ERROR_THRESHOLD | string | `"5"` | Master heartbeat error threshold |
+| master.env.MASTER_HOST_SELECTOR | string | `"LowerWeight"` | Master host selector to select a suitable worker, optional values include Random, RoundRobin, LowerWeight |
+| master.env.MASTER_KILL_APPLICATION_WHEN_HANDLE_FAILOVER | string | `"true"` | Master kill application when handle failover |
+| master.env.MASTER_MAX_HEARTBEAT_INTERVAL | string | `"10s"` | Master max heartbeat interval |
+| master.env.MASTER_SERVER_LOAD_PROTECTION_ENABLED | bool | `false` | If set true, will open master overload protection |
+| master.env.MASTER_SERVER_LOAD_PROTECTION_MAX_DISK_USAGE_PERCENTAGE_THRESHOLDS | float | `0.7` | Master max disk usage , when the master's disk usage is smaller then this value, master server can execute workflow. |
+| master.env.MASTER_SERVER_LOAD_PROTECTION_MAX_JVM_CPU_USAGE_PERCENTAGE_THRESHOLDS | float | `0.7` | Master max jvm cpu usage, when the master's jvm cpu usage is smaller then this value, master server can execute workflow. |
+| master.env.MASTER_SERVER_LOAD_PROTECTION_MAX_SYSTEM_CPU_USAGE_PERCENTAGE_THRESHOLDS | float | `0.7` | Master max system cpu usage, when the master's system cpu usage is smaller then this value, master server can execute workflow. |
+| master.env.MASTER_SERVER_LOAD_PROTECTION_MAX_SYSTEM_MEMORY_USAGE_PERCENTAGE_THRESHOLDS | float | `0.7` | Master max System memory usage , when the master's system memory usage is smaller then this value, master server can execute workflow. |
+| master.env.MASTER_STATE_WHEEL_INTERVAL | string | `"5s"` | master state wheel interval, the unit is second |
+| master.env.MASTER_TASK_COMMIT_INTERVAL | string | `"1s"` | master commit task interval, the unit is second |
+| master.env.MASTER_TASK_COMMIT_RETRYTIMES | string | `"5"` | Master commit task retry times |
+| master.livenessProbe | object | `{"enabled":true,"failureThreshold":"3","initialDelaySeconds":"30","periodSeconds":"30","successThreshold":"1","timeoutSeconds":"5"}` | Periodic probe of container liveness. Container will be restarted if the probe fails. More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) |
+| master.livenessProbe.enabled | bool | `true` | Turn on and off liveness probe |
+| master.livenessProbe.failureThreshold | string | `"3"` | Minimum consecutive failures for the probe |
+| master.livenessProbe.initialDelaySeconds | string | `"30"` | Delay before liveness probe is initiated |
+| master.livenessProbe.periodSeconds | string | `"30"` | How often to perform the probe |
+| master.livenessProbe.successThreshold | string | `"1"` | Minimum consecutive successes for the probe |
+| master.livenessProbe.timeoutSeconds | string | `"5"` | When the probe times out |
+| master.nodeSelector | object | `{}` | NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info: [assign-pod-node](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) |
+| master.persistentVolumeClaim | object | `{"accessModes":["ReadWriteOnce"],"enabled":false,"storage":"20Gi","storageClassName":"-"}` | PersistentVolumeClaim represents a reference to a PersistentVolumeClaim in the same namespace. The StatefulSet controller is responsible for mapping network identities to claims in a way that maintains the identity of a pod. Every claim in this list must have at least one matching (by name) volumeMount in one container in the template. A claim in this list takes precedence over any volumes in the template, with the same name. |
+| master.persistentVolumeClaim.accessModes | list | `["ReadWriteOnce"]` | `PersistentVolumeClaim` access modes |
+| master.persistentVolumeClaim.enabled | bool | `false` | Set `master.persistentVolumeClaim.enabled` to `true` to mount a new volume for `master` |
+| master.persistentVolumeClaim.storage | string | `"20Gi"` | `PersistentVolumeClaim` size |
+| master.persistentVolumeClaim.storageClassName | string | `"-"` | `Master` logs data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning |
+| master.podManagementPolicy | string | `"Parallel"` | PodManagementPolicy controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down. |
+| master.readinessProbe | object | `{"enabled":true,"failureThreshold":"3","initialDelaySeconds":"30","periodSeconds":"30","successThreshold":"1","timeoutSeconds":"5"}` | Periodic probe of container service readiness. Container will be removed from service endpoints if the probe fails. More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) |
+| master.readinessProbe.enabled | bool | `true` | Turn on and off readiness probe |
+| master.readinessProbe.failureThreshold | string | `"3"` | Minimum consecutive failures for the probe |
+| master.readinessProbe.initialDelaySeconds | string | `"30"` | Delay before readiness probe is initiated |
+| master.readinessProbe.periodSeconds | string | `"30"` | How often to perform the probe |
+| master.readinessProbe.successThreshold | string | `"1"` | Minimum consecutive successes for the probe |
+| master.readinessProbe.timeoutSeconds | string | `"5"` | When the probe times out |
+| master.replicas | string | `"3"` | Replicas is the desired number of replicas of the given Template. |
+| master.resources | object | `{}` | Compute Resources required by this container. More info: [manage-resources-containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) |
+| master.service.annotations | object | `{}` | annotations may need to be set when want to scrapy metrics by prometheus but not install prometheus operator |
+| master.service.serviceMonitor | object | `{"annotations":{},"enabled":false,"interval":"15s","labels":{},"path":"/actuator/prometheus"}` | serviceMonitor for prometheus operator |
+| master.service.serviceMonitor.annotations | object | `{}` | serviceMonitor.annotations ServiceMonitor annotations |
+| master.service.serviceMonitor.enabled | bool | `false` | Enable or disable master serviceMonitor |
+| master.service.serviceMonitor.interval | string | `"15s"` | serviceMonitor.interval interval at which metrics should be scraped |
+| master.service.serviceMonitor.labels | object | `{}` | serviceMonitor.labels ServiceMonitor extra labels |
+| master.service.serviceMonitor.path | string | `"/actuator/prometheus"` | serviceMonitor.path path of the metrics endpoint |
+| master.tolerations | list | `[]` | Tolerations are appended (excluding duplicates) to pods running with this RuntimeClass during admission, effectively unioning the set of nodes tolerated by the pod and the RuntimeClass. |
+| minio.auth.rootPassword | string | `"minioadmin"` | minio password |
+| minio.auth.rootUser | string | `"minioadmin"` | minio username |
+| minio.defaultBuckets | string | `"dolphinscheduler"` | minio default buckets |
+| minio.enabled | bool | `true` | Deploy minio and configure it as the default storage for DolphinScheduler, note this is for demo only, not for production. |
+| minio.persistence.enabled | bool | `false` | Set minio.persistence.enabled to true to mount a new volume for internal minio |
+| mysql.auth.database | string | `"dolphinscheduler"` | mysql database |
+| mysql.auth.params | string | `"characterEncoding=utf8"` | mysql params |
+| mysql.auth.password | string | `"ds"` | mysql password |
+| mysql.auth.username | string | `"ds"` | mysql username |
+| mysql.driverClassName | string | `"com.mysql.cj.jdbc.Driver"` | mysql driverClassName |
+| mysql.enabled | bool | `false` | If not exists external MySQL, by default, the DolphinScheduler will use a internal MySQL |
+| mysql.primary.persistence.enabled | bool | `false` | Set mysql.primary.persistence.enabled to true to mount a new volume for internal MySQL |
+| mysql.primary.persistence.size | string | `"20Gi"` | `PersistentVolumeClaim` size |
+| mysql.primary.persistence.storageClass | string | `"-"` | MySQL data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning |
+| postgresql.driverClassName | string | `"org.postgresql.Driver"` | The driverClassName for internal PostgreSQL |
+| postgresql.enabled | bool | `true` | If not exists external PostgreSQL, by default, the DolphinScheduler will use a internal PostgreSQL |
+| postgresql.params | string | `"characterEncoding=utf8"` | The params for internal PostgreSQL |
+| postgresql.persistence.enabled | bool | `false` | Set postgresql.persistence.enabled to true to mount a new volume for internal PostgreSQL |
+| postgresql.persistence.size | string | `"20Gi"` | `PersistentVolumeClaim` size |
+| postgresql.persistence.storageClass | string | `"-"` | PostgreSQL data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning |
+| postgresql.postgresqlDatabase | string | `"dolphinscheduler"` | The database for internal PostgreSQL |
+| postgresql.postgresqlPassword | string | `"root"` | The password for internal PostgreSQL |
+| postgresql.postgresqlUsername | string | `"root"` | The username for internal PostgreSQL |
+| registryEtcd.authority | string | `""` | Etcd authority |
+| registryEtcd.enabled | bool | `false` | If you want to use Etcd for your registry center, change this value to true. And set zookeeper.enabled to false |
+| registryEtcd.endpoints | string | `""` | Etcd endpoints |
+| registryEtcd.namespace | string | `"dolphinscheduler"` | Etcd namespace |
+| registryEtcd.passWord | string | `""` | Etcd passWord |
+| registryEtcd.ssl.certFile | string | `"etcd-certs/ca.crt"` | CertFile file path |
+| registryEtcd.ssl.enabled | bool | `false` | If your Etcd server has configured with ssl, change this value to true. About certification files you can see [here](https://github.com/etcd-io/jetcd/blob/main/docs/SslConfig.md) for how to convert. |
+| registryEtcd.ssl.keyCertChainFile | string | `"etcd-certs/client.crt"` | keyCertChainFile file path |
+| registryEtcd.ssl.keyFile | string | `"etcd-certs/client.pem"` | keyFile file path |
+| registryEtcd.user | string | `""` | Etcd user |
+| registryJdbc.enabled | bool | `false` | If you want to use JDbc for your registry center, change this value to true. And set zookeeper.enabled and registryEtcd.enabled to false |
+| registryJdbc.hikariConfig.driverClassName | string | `"com.mysql.cj.jdbc.Driver"` | Default use same Dolphinscheduler's database if you don't change this value. If you set this value, Registry jdbc's database type will use it |
+| registryJdbc.hikariConfig.enabled | bool | `false` | Default use same Dolphinscheduler's database, if you want to use other database please change `enabled` to `true` and change other configs |
+| registryJdbc.hikariConfig.jdbcurl | string | `"jdbc:mysql://"` | Default use same Dolphinscheduler's database if you don't change this value. If you set this value, Registry jdbc's database type will use it |
+| registryJdbc.hikariConfig.password | string | `""` | Default use same Dolphinscheduler's database if you don't change this value. If you set this value, Registry jdbc's database type will use it |
+| registryJdbc.hikariConfig.username | string | `""` | Default use same Dolphinscheduler's database if you don't change this value. If you set this value, Registry jdbc's database type will use it |
+| registryJdbc.termExpireTimes | int | `3` | Used to calculate the expire time |
+| registryJdbc.termRefreshInterval | string | `"2s"` | Used to schedule refresh the ephemeral data/ lock |
+| security.authentication.ldap.basedn | string | `"dc=example,dc=com"` | LDAP base dn |
+| security.authentication.ldap.password | string | `"password"` | LDAP password |
+| security.authentication.ldap.ssl.enable | bool | `false` | LDAP ssl switch |
+| security.authentication.ldap.ssl.jksbase64content | string | `""` | LDAP jks file base64 content. If you use macOS, please run `base64 -b 0 -i /path/to/your.jks`. If you use Linux, please run `base64 -w 0 /path/to/your.jks`. If you use Windows, please run `certutil -f -encode /path/to/your.jks`. Then copy the base64 content to below field in one line |
+| security.authentication.ldap.ssl.truststore | string | `"/opt/ldapkeystore.jks"` | LDAP jks file absolute path, do not change this value |
+| security.authentication.ldap.ssl.truststorepassword | string | `""` | LDAP jks password |
+| security.authentication.ldap.urls | string | `"ldap://ldap.forumsys.com:389/"` | LDAP urls |
+| security.authentication.ldap.user.admin | string | `"read-only-admin"` | Admin user account when you log-in with LDAP |
+| security.authentication.ldap.user.emailattribute | string | `"mail"` | LDAP user email attribute |
+| security.authentication.ldap.user.identityattribute | string | `"uid"` | LDAP user identity attribute |
+| security.authentication.ldap.user.notexistaction | string | `"CREATE"` | action when ldap user is not exist,default value: CREATE. Optional values include(CREATE,DENY) |
+| security.authentication.ldap.username | string | `"cn=read-only-admin,dc=example,dc=com"` | LDAP username |
+| security.authentication.type | string | `"PASSWORD"` | Authentication types (supported types: PASSWORD,LDAP,CASDOOR_SSO) |
+| timezone | string | `"Asia/Shanghai"` | World time and date for cities in all time zones |
+| worker.affinity | object | `{}` | Affinity is a group of affinity scheduling rules. If specified, the pod's scheduling constraints. More info: [node-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity) |
+| worker.annotations | object | `{}` | You can use annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata. |
+| worker.enabled | bool | `true` | Enable or disable the Worker component |
+| worker.env.DEFAULT_TENANT_ENABLED | bool | `false` | If set true, will use worker bootstrap user as the tenant to execute task when the tenant is `default`; |
+| worker.env.WORKER_EXEC_THREADS | string | `"100"` | Worker execute thread number to limit task instances |
+| worker.env.WORKER_HOST_WEIGHT | string | `"100"` | Worker host weight to dispatch tasks |
+| worker.env.WORKER_MAX_HEARTBEAT_INTERVAL | string | `"10s"` | Worker heartbeat interval |
+| worker.env.WORKER_SERVER_LOAD_PROTECTION_ENABLED | bool | `false` | If set true, will open worker overload protection |
+| worker.env.WORKER_SERVER_LOAD_PROTECTION_MAX_DISK_USAGE_PERCENTAGE_THRESHOLDS | float | `0.7` | Worker max disk usage , when the worker's disk usage is smaller then this value, worker server can be dispatched tasks. |
+| worker.env.WORKER_SERVER_LOAD_PROTECTION_MAX_JVM_CPU_USAGE_PERCENTAGE_THRESHOLDS | float | `0.7` | Worker max jvm cpu usage, when the worker's jvm cpu usage is smaller then this value, worker server can be dispatched tasks. |
+| worker.env.WORKER_SERVER_LOAD_PROTECTION_MAX_SYSTEM_CPU_USAGE_PERCENTAGE_THRESHOLDS | float | `0.7` | Worker max system cpu usage, when the worker's system cpu usage is smaller then this value, worker server can be dispatched tasks. |
+| worker.env.WORKER_SERVER_LOAD_PROTECTION_MAX_SYSTEM_MEMORY_USAGE_PERCENTAGE_THRESHOLDS | float | `0.7` | Worker max memory usage , when the worker's memory usage is smaller then this value, worker server can be dispatched tasks. |
+| worker.env.WORKER_TENANT_CONFIG_AUTO_CREATE_TENANT_ENABLED | bool | `true` | tenant corresponds to the user of the system, which is used by the worker to submit the job. If system does not have this user, it will be automatically created after the parameter worker.tenant.auto.create is true. |
+| worker.keda.advanced | object | `{}` | Specify HPA related options |
+| worker.keda.cooldownPeriod | int | `30` | How many seconds KEDA will wait before scaling to zero. Note that HPA has a separate cooldown period for scale-downs |
+| worker.keda.enabled | bool | `false` | Enable or disable the Keda component |
+| worker.keda.maxReplicaCount | int | `3` | Maximum number of workers created by keda |
+| worker.keda.minReplicaCount | int | `0` | Minimum number of workers created by keda |
+| worker.keda.namespaceLabels | object | `{}` | Keda namespace labels |
+| worker.keda.pollingInterval | int | `5` | How often KEDA polls the DolphinScheduler DB to report new scale requests to the HPA |
+| worker.livenessProbe | object | `{"enabled":true,"failureThreshold":"3","initialDelaySeconds":"30","periodSeconds":"30","successThreshold":"1","timeoutSeconds":"5"}` | Periodic probe of container liveness. Container will be restarted if the probe fails. More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) |
+| worker.livenessProbe.enabled | bool | `true` | Turn on and off liveness probe |
+| worker.livenessProbe.failureThreshold | string | `"3"` | Minimum consecutive failures for the probe |
+| worker.livenessProbe.initialDelaySeconds | string | `"30"` | Delay before liveness probe is initiated |
+| worker.livenessProbe.periodSeconds | string | `"30"` | How often to perform the probe |
+| worker.livenessProbe.successThreshold | string | `"1"` | Minimum consecutive successes for the probe |
+| worker.livenessProbe.timeoutSeconds | string | `"5"` | When the probe times out |
+| worker.nodeSelector | object | `{}` | NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info: [assign-pod-node](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) |
+| worker.persistentVolumeClaim | object | `{"dataPersistentVolume":{"accessModes":["ReadWriteOnce"],"enabled":false,"storage":"20Gi","storageClassName":"-"},"enabled":false,"logsPersistentVolume":{"accessModes":["ReadWriteOnce"],"enabled":false,"storage":"20Gi","storageClassName":"-"}}` | PersistentVolumeClaim represents a reference to a PersistentVolumeClaim in the same namespace. The StatefulSet controller is responsible for mapping network identities to claims in a way that maintains the identity of a pod. Every claim in this list must have at least one matching (by name) volumeMount in one container in the template. A claim in this list takes precedence over any volumes in the template, with the same name. |
+| worker.persistentVolumeClaim.dataPersistentVolume.accessModes | list | `["ReadWriteOnce"]` | `PersistentVolumeClaim` access modes |
+| worker.persistentVolumeClaim.dataPersistentVolume.enabled | bool | `false` | Set `worker.persistentVolumeClaim.dataPersistentVolume.enabled` to `true` to mount a data volume for `worker` |
+| worker.persistentVolumeClaim.dataPersistentVolume.storage | string | `"20Gi"` | `PersistentVolumeClaim` size |
+| worker.persistentVolumeClaim.dataPersistentVolume.storageClassName | string | `"-"` | `Worker` data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning |
+| worker.persistentVolumeClaim.enabled | bool | `false` | Set `worker.persistentVolumeClaim.enabled` to `true` to enable `persistentVolumeClaim` for `worker` |
+| worker.persistentVolumeClaim.logsPersistentVolume.accessModes | list | `["ReadWriteOnce"]` | `PersistentVolumeClaim` access modes |
+| worker.persistentVolumeClaim.logsPersistentVolume.enabled | bool | `false` | Set `worker.persistentVolumeClaim.logsPersistentVolume.enabled` to `true` to mount a logs volume for `worker` |
+| worker.persistentVolumeClaim.logsPersistentVolume.storage | string | `"20Gi"` | `PersistentVolumeClaim` size |
+| worker.persistentVolumeClaim.logsPersistentVolume.storageClassName | string | `"-"` | `Worker` logs data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning |
+| worker.podManagementPolicy | string | `"Parallel"` | PodManagementPolicy controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down. |
+| worker.readinessProbe | object | `{"enabled":true,"failureThreshold":"3","initialDelaySeconds":"30","periodSeconds":"30","successThreshold":"1","timeoutSeconds":"5"}` | Periodic probe of container service readiness. Container will be removed from service endpoints if the probe fails. More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) |
+| worker.readinessProbe.enabled | bool | `true` | Turn on and off readiness probe |
+| worker.readinessProbe.failureThreshold | string | `"3"` | Minimum consecutive failures for the probe |
+| worker.readinessProbe.initialDelaySeconds | string | `"30"` | Delay before readiness probe is initiated |
+| worker.readinessProbe.periodSeconds | string | `"30"` | How often to perform the probe |
+| worker.readinessProbe.successThreshold | string | `"1"` | Minimum consecutive successes for the probe |
+| worker.readinessProbe.timeoutSeconds | string | `"5"` | When the probe times out |
+| worker.replicas | string | `"3"` | Replicas is the desired number of replicas of the given Template. |
+| worker.resources | object | `{}` | Compute Resources required by this container. More info: [manage-resources-containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) |
+| worker.service.annotations | object | `{}` | annotations may need to be set when want to scrapy metrics by prometheus but not install prometheus operator |
+| worker.service.serviceMonitor | object | `{"annotations":{},"enabled":false,"interval":"15s","labels":{},"path":"/actuator/prometheus"}` | serviceMonitor for prometheus operator |
+| worker.service.serviceMonitor.annotations | object | `{}` | serviceMonitor.annotations ServiceMonitor annotations |
+| worker.service.serviceMonitor.enabled | bool | `false` | Enable or disable worker serviceMonitor |
+| worker.service.serviceMonitor.interval | string | `"15s"` | serviceMonitor.interval interval at which metrics should be scraped |
+| worker.service.serviceMonitor.labels | object | `{}` | serviceMonitor.labels ServiceMonitor extra labels |
+| worker.service.serviceMonitor.path | string | `"/actuator/prometheus"` | serviceMonitor.path path of the metrics endpoint |
+| worker.tolerations | list | `[]` | Tolerations are appended (excluding duplicates) to pods running with this RuntimeClass during admission, effectively unioning the set of nodes tolerated by the pod and the RuntimeClass. |
+| zookeeper.enabled | bool | `true` | If not exists external registry, the zookeeper registry will be used by default. |
+| zookeeper.fourlwCommandsWhitelist | string | `"srvr,ruok,wchs,cons"` | A list of comma separated Four Letter Words commands to use |
+| zookeeper.persistence.enabled | bool | `false` | Set `zookeeper.persistence.enabled` to true to mount a new volume for internal ZooKeeper |
+| zookeeper.persistence.size | string | `"20Gi"` | PersistentVolumeClaim size |
+| zookeeper.persistence.storageClass | string | `"-"` | ZooKeeper data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning |
+| zookeeper.service.port | int | `2181` | The port of zookeeper |

deploy/kubernetes/dolphinscheduler/README.md.gotmpl

@@ -0,0 +1,11 @@
+## About DolphinScheduler for Kubernetes
+
+{{ template "chart.description" . }}
+
+This chart bootstraps all the components needed to run Apache DolphinScheduler on a Kubernetes Cluster using [Helm](https://helm.sh).
+
+## QuickStart in Kubernetes
+
+Please refer to the [Quick Start in Kubernetes](../../../docs/docs/en/guide/installation/kubernetes.md)
+
+{{ template "chart.valuesSection" . }}

deploy/kubernetes/dolphinscheduler/templates/_helpers.tpl

@@ -51,7 +51,6 @@ Create a default common labels.
 {{- define "dolphinscheduler.common.labels" -}}
 app.kubernetes.io/instance: {{ .Release.Name }}
 app.kubernetes.io/managed-by: {{ .Release.Service }}
-app.kubernetes.io/version: {{ .Chart.AppVersion }}
 {{- end -}}
 
 {{/*
@@ -146,6 +145,10 @@ Create a database environment variables.
   {{- else }}
   value: {{ .Values.externalDatabase.type | quote }}
   {{- end }}
+{{- if or .Values.mysql.enabled (eq .Values.externalDatabase.type "mysql") }}
+- name: SPRING_PROFILES_ACTIVE
+  value: mysql
+{{- end }}
 - name: SPRING_DATASOURCE_URL
   {{- if .Values.postgresql.enabled }}
   value: jdbc:postgresql://{{ template "dolphinscheduler.postgresql.fullname" . }}:5432/{{ .Values.postgresql.postgresqlDatabase }}?{{ .Values.postgresql.params }}
@@ -252,28 +255,48 @@ Create a registry environment variables.
 - name: REGISTRY_TYPE
   {{- if .Values.zookeeper.enabled }}
   value: "zookeeper"
-  {{- else if .Values.etcd.enabled }}
+  {{- else if .Values.registryEtcd.enabled }}
   value: "etcd"
+  {{- else if .Values.registryJdbc.enabled }}
+  value: "jdbc"
   {{- else }}
   value: {{ .Values.externalRegistry.registryPluginName }}
   {{- end }}
-{{- if .Values.etcd.enabled }}
+{{- if .Values.registryEtcd.enabled }}
 - name: REGISTRY_ENDPOINTS
-  value: {{ .Values.etcd.endpoints }}
+  value: {{ .Values.registryEtcd.endpoints }}
 - name: REGISTRY_NAMESPACE
-  value: {{ .Values.etcd.namespace }}
+  value: {{ .Values.registryEtcd.namespace }}
 - name: REGISTRY_USER
-  value: {{ .Values.etcd.user }}
+  value: {{ .Values.registryEtcd.user }}
 - name: REGISTRY_PASSWORD
-  value: {{ .Values.etcd.passWord }}
+  value: {{ .Values.registryEtcd.passWord }}
 - name: REGISTRY_AUTHORITY
-  value: {{ .Values.etcd.authority }}
+  value: {{ .Values.registryEtcd.authority }}
 - name: REGISTRY_CERT_FILE
-  value: {{ .Values.etcd.ssl.certFile }}
+  value: {{ .Values.registryEtcd.ssl.certFile }}
 - name: REGISTRY_KEY_CERT_CHAIN_FILE
-  value: {{ .Values.etcd.ssl.keyCertChainFile }}
+  value: {{ .Values.registryEtcd.ssl.keyCertChainFile }}
 - name: REGISTRY_KEY_FILE
-  value: {{ .Values.etcd.ssl.keyFile }}
+  value: {{ .Values.registryEtcd.ssl.keyFile }}
+{{- else if .Values.registryJdbc.enabled }}
+- name: REGISTRY_TERM_REFRESH_INTERVAL
+  value: {{ .Values.registryJdbc.termRefreshInterval }}
+- name: REGISTRY_TERM_EXPIRE_TIMES
+  value: {{ .Values.registryJdbc.termExpireTimes | quote}}
+{{- if .Values.registryJdbc.hikariConfig.enabled }}
+- name: REGISTRY_HIKARI_CONFIG_DRIVER_CLASS_NAME
+  value: {{ .Values.registryJdbc.hikariConfig.driverClassName }}
+- name: REGISTRY_HIKARI_CONFIG_JDBC_URL
+  value: {{ .Values.registryJdbc.hikariConfig.jdbcurl }}
+- name: REGISTRY_HIKARI_CONFIG_USERNAME
+  value: {{ .Values.registryJdbc.hikariConfig.username }}
+- name: REGISTRY_HIKARI_CONFIG_PASSWORD
+  valueFrom:
+    secretKeyRef:
+      name: {{ include "dolphinscheduler.fullname" . }}-registry-db
+      key: registry-password
+{{- end }}
 {{- else }}
 - name: REGISTRY_ZOOKEEPER_CONNECT_STRING
   {{- if .Values.zookeeper.enabled }}
@@ -330,7 +353,7 @@ Create a fsFileResourcePersistence volumeMount.
 Create a etcd ssl volume.
 */}}
 {{- define "dolphinscheduler.etcd.ssl.volume" -}}
-{{- if .Values.etcd.ssl.enabled -}}
+{{- if .Values.registryEtcd.ssl.enabled -}}
 - name: etcd-ssl
   secret:
     secretName: {{ include "dolphinscheduler.fullname" . }}-etcd-ssl
@@ -341,14 +364,14 @@ Create a etcd ssl volume.
 Create a etcd ssl volumeMount.
 */}}
 {{- define "dolphinscheduler.etcd.ssl.volumeMount" -}}
-{{- if .Values.etcd.ssl.enabled -}}
-- mountPath: /opt/dolphinscheduler/{{ .Values.etcd.ssl.certFile }}
+{{- if .Values.registryEtcd.ssl.enabled -}}
+- mountPath: /opt/dolphinscheduler/{{ .Values.registryEtcd.ssl.certFile }}
   name: etcd-ssl
   subPath: cert-file
-- mountPath: /opt/dolphinscheduler/{{ .Values.etcd.ssl.keyCertChainFile  }}
+- mountPath: /opt/dolphinscheduler/{{ .Values.registryEtcd.ssl.keyCertChainFile  }}
   name: etcd-ssl
   subPath: key-cert-chain-file
-- mountPath: /opt/dolphinscheduler/{{ .Values.etcd.ssl.keyFile }}
+- mountPath: /opt/dolphinscheduler/{{ .Values.registryEtcd.ssl.keyFile }}
   name: etcd-ssl
   subPath: key-file
 {{- end -}}

deploy/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-alert.yaml

@@ -0,0 +1,115 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+{{- if and .Values.alert.enabled }}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ include "dolphinscheduler.fullname" . }}-alert
+  labels:
+    app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-alert
+    {{- include "dolphinscheduler.alert.labels" . | nindent 4 }}
+data:
+  application.yaml: |
+    spring:
+      profiles:
+        active: {{ .Values.datasource.profile }}
+      jackson:
+        time-zone: UTC
+        date-format: "yyyy-MM-dd HH:mm:ss"
+      banner:
+        charset: UTF-8
+      datasource:
+        driver-class-name: org.postgresql.Driver
+        url: jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
+        username: root
+        password: root
+        hikari:
+          connection-test-query: select 1
+          pool-name: DolphinScheduler
+    
+    # Mybatis-plus configuration, you don't need to change it
+    mybatis-plus:
+      mapper-locations: classpath:org/apache/dolphinscheduler/dao/mapper/*Mapper.xml
+      type-aliases-package: org.apache.dolphinscheduler.dao.entity
+      configuration:
+        cache-enabled: false
+        call-setters-on-nulls: true
+        map-underscore-to-camel-case: true
+        jdbc-type-for-null: NULL
+      global-config:
+        db-config:
+          id-type: auto
+        banner: false
+    
+    server:
+      port: 50053
+    
+    management:
+      endpoints:
+        web:
+          exposure:
+            include: health,metrics,prometheus
+      endpoint:
+        health:
+          enabled: true
+          show-details: always
+      health:
+        db:
+          enabled: true
+        defaults:
+          enabled: false
+      metrics:
+        tags:
+          application: ${spring.application.name}
+    
+    alert:
+      port: 50052
+      # Mark each alert of alert server if late after x milliseconds as failed.
+      # Define value is (0 = infinite), and alert server would be waiting alert result.
+      wait-timeout: 0
+      max-heartbeat-interval: 60s
+      query_alert_threshold: 100
+    
+    registry:
+      type: zookeeper
+      zookeeper:
+        namespace: dolphinscheduler
+        connect-string: localhost:2181
+        retry-policy:
+          base-sleep-time: 60ms
+          max-sleep: 300ms
+          max-retries: 5
+        session-timeout: 30s
+        connection-timeout: 9s
+        block-until-connected: 600ms
+        digest: ~
+    
+    metrics:
+      enabled: true
+    
+    # Override by profile
+    ---
+    spring:
+      config:
+        activate:
+          on-profile: mysql
+      datasource:
+        driver-class-name: com.mysql.cj.jdbc.Driver
+        url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler
+        username: root
+        password: root
+{{- end }}

deploy/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-api.yaml

@@ -0,0 +1,258 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+{{- if and .Values.api.enabled }}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ include "dolphinscheduler.fullname" . }}-api
+  labels:
+    app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-api
+    {{- include "dolphinscheduler.api.labels" . | nindent 4 }}
+data:
+  application.yaml: |
+    server:
+      port: 12345
+      servlet:
+        session:
+          timeout: 120m
+        context-path: /dolphinscheduler/
+      compression:
+        enabled: true
+        mime-types: text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml
+      jetty:
+        max-http-form-post-size: 5000000
+        accesslog:
+          enabled: true
+          custom-format: '%{client}a - %u %t "%r" %s %O %{ms}Tms'
+    
+    spring:
+      profiles:
+        active: {{ .Values.datasource.profile }}
+      banner:
+        charset: UTF-8
+      jackson:
+        time-zone: UTC
+        date-format: "yyyy-MM-dd HH:mm:ss"
+      servlet:
+        multipart:
+          max-file-size: 1024MB
+          max-request-size: 1024MB
+      messages:
+        basename: i18n/messages
+      datasource:
+        driver-class-name: org.postgresql.Driver
+        url: jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
+        username: root
+        password: root
+        hikari:
+          connection-test-query: select 1
+          pool-name: DolphinScheduler
+      quartz:
+        auto-startup: false
+        job-store-type: jdbc
+        jdbc:
+          initialize-schema: never
+        properties:
+          org.quartz.jobStore.isClustered: true
+          org.quartz.jobStore.class: org.springframework.scheduling.quartz.LocalDataSourceJobStore
+          org.quartz.scheduler.instanceId: AUTO
+          org.quartz.jobStore.tablePrefix: QRTZ_
+          org.quartz.jobStore.acquireTriggersWithinLock: true
+          org.quartz.scheduler.instanceName: DolphinScheduler
+          org.quartz.threadPool.class: org.apache.dolphinscheduler.scheduler.quartz.QuartzZeroSizeThreadPool
+          org.quartz.jobStore.useProperties: false
+          org.quartz.jobStore.misfireThreshold: 60000
+          org.quartz.scheduler.makeSchedulerThreadDaemon: true
+          org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
+          org.quartz.jobStore.clusterCheckinInterval: 5000
+          org.quartz.scheduler.batchTriggerAcquisitionMaxCount: 1
+      mvc:
+        pathmatch:
+          matching-strategy: ANT_PATH_MATCHER
+        static-path-pattern: /static/**
+    springdoc:
+      swagger-ui:
+        path: /swagger-ui.html
+      packages-to-scan: org.apache.dolphinscheduler.api
+    
+    # Mybatis-plus configuration, you don't need to change it
+    mybatis-plus:
+      mapper-locations: classpath:org/apache/dolphinscheduler/dao/mapper/*Mapper.xml
+      type-aliases-package: org.apache.dolphinscheduler.dao.entity
+      configuration:
+        cache-enabled: false
+        call-setters-on-nulls: true
+        map-underscore-to-camel-case: true
+        jdbc-type-for-null: NULL
+      global-config:
+        db-config:
+          id-type: auto
+        banner: false
+    
+    management:
+      endpoints:
+        web:
+          exposure:
+            include: health,metrics,prometheus
+      endpoint:
+        health:
+          enabled: true
+          show-details: always
+      health:
+        db:
+          enabled: true
+        defaults:
+          enabled: false
+      metrics:
+        tags:
+          application: ${spring.application.name}
+    
+    registry:
+      type: zookeeper
+      zookeeper:
+        namespace: dolphinscheduler
+        connect-string: localhost:2181
+        retry-policy:
+          base-sleep-time: 60ms
+          max-sleep: 300ms
+          max-retries: 5
+        session-timeout: 60s
+        connection-timeout: 15s
+        block-until-connected: 15s
+        digest: ~
+    
+    api:
+      audit-enable: false
+      # Traffic control, if you turn on this config, the maximum number of request/s will be limited.
+      # global max request number per second
+      # default tenant-level max request number
+      traffic-control:
+        global-switch: false
+        max-global-qps-rate: 300
+        tenant-switch: false
+        default-tenant-qps-rate: 10
+          #customize-tenant-qps-rate:
+        # eg.
+        #tenant1: 11
+        #tenant2: 20
+      python-gateway:
+        # Weather enable python gateway server or not. The default value is false.
+        enabled: false
+        # Authentication token for connection from python api to python gateway server. Should be changed the default value
+        # when you deploy in public network.
+        auth-token: jwUDzpLsNKEFER4*a8gruBH_GsAurNxU7A@Xc
+        # The address of Python gateway server start. Set its value to `0.0.0.0` if your Python API run in different
+        # between Python gateway server. It could be be specific to other address like `127.0.0.1` or `localhost`
+        gateway-server-address: 0.0.0.0
+        # The port of Python gateway server start. Define which port you could connect to Python gateway server from
+        # Python API side.
+        gateway-server-port: 25333
+        # The address of Python callback client.
+        python-address: 127.0.0.1
+        # The port of Python callback client.
+        python-port: 25334
+        # Close connection of socket server if no other request accept after x milliseconds. Define value is (0 = infinite),
+        # and socket server would never close even though no requests accept
+        connect-timeout: 0
+        # Close each active connection of socket server if python program not active after x milliseconds. Define value is
+        # (0 = infinite), and socket server would never close even though no requests accept
+        read-timeout: 0
+    
+    metrics:
+      enabled: true
+    
+    security:
+      authentication:
+        # Authentication types (supported types: PASSWORD,LDAP,CASDOOR_SSO)
+        type: PASSWORD
+        # IF you set type `LDAP`, below config will be effective
+        ldap:
+          # ldap server config
+          urls: ldap://ldap.forumsys.com:389/
+          base-dn: dc=example,dc=com
+          username: cn=read-only-admin,dc=example,dc=com
+          password: password
+          user:
+            # admin userId when you use LDAP login
+            admin: read-only-admin
+            identity-attribute: uid
+            email-attribute: mail
+            # action when ldap user is not exist (supported types: CREATE,DENY)
+            not-exist-action: CREATE
+          ssl:
+            enable: false
+            # jks file absolute path && password
+            trust-store: "/ldapkeystore.jks"
+            trust-store-password: "password"
+        casdoor:
+          user:
+            admin: ""
+        oauth2:
+          enable: false
+          provider:
+            github:
+              authorizationUri: ""
+              redirectUri: ""
+              clientId: ""
+              clientSecret: ""
+              tokenUri: ""
+              userInfoUri: ""
+              callbackUrl: ""
+              iconUri: ""
+              provider: github
+            google:
+              authorizationUri: ""
+              redirectUri: ""
+              clientId: ""
+              clientSecret: ""
+              tokenUri: ""
+              userInfoUri: ""
+              callbackUrl: ""
+              iconUri: ""
+              provider: google
+    casdoor:
+      # Your Casdoor server url
+      endpoint: ""
+      client-id: ""
+      client-secret: ""
+      # The certificate may be multi-line, you can use `|-` for ease
+      certificate: ""
+      # Your organization name added in Casdoor
+      organization-name: ""
+      # Your application name added in Casdoor
+      application-name: ""
+      # Doplhinscheduler login url
+      redirect-url: ""
+    
+    # Override by profile
+    ---
+    spring:
+      config:
+        activate:
+          on-profile: mysql
+      datasource:
+        driver-class-name: com.mysql.cj.jdbc.Driver
+        url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler
+        username: root
+        password: root
+      quartz:
+        properties:
+          org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.StdJDBCDelegate
+{{- end }}
+
+
+

deploy/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-master.yaml

@@ -0,0 +1,173 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+{{- if and .Values.master.enabled }}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ include "dolphinscheduler.fullname" . }}-master
+  labels:
+    app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-master
+    {{- include "dolphinscheduler.master.labels" . | nindent 4 }}
+data:
+  application.yaml: |
+    spring:
+      profiles:
+        active: {{ .Values.datasource.profile }}
+      banner:
+        charset: UTF-8
+      jackson:
+        time-zone: UTC
+        date-format: "yyyy-MM-dd HH:mm:ss"
+      datasource:
+        driver-class-name: org.postgresql.Driver
+        url: jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
+        username: root
+        password: root
+        hikari:
+          connection-test-query: select 1
+          pool-name: DolphinScheduler
+      quartz:
+        job-store-type: jdbc
+        jdbc:
+          initialize-schema: never
+        properties:
+          org.quartz.threadPool.threadPriority: 5
+          org.quartz.jobStore.isClustered: true
+          org.quartz.jobStore.class: org.springframework.scheduling.quartz.LocalDataSourceJobStore
+          org.quartz.scheduler.instanceId: AUTO
+          org.quartz.jobStore.tablePrefix: QRTZ_
+          org.quartz.jobStore.acquireTriggersWithinLock: true
+          org.quartz.scheduler.instanceName: DolphinScheduler
+          org.quartz.threadPool.class: org.quartz.simpl.SimpleThreadPool
+          org.quartz.jobStore.useProperties: false
+          org.quartz.threadPool.makeThreadsDaemons: true
+          org.quartz.threadPool.threadCount: 25
+          org.quartz.jobStore.misfireThreshold: 60000
+          org.quartz.scheduler.batchTriggerAcquisitionMaxCount: 1
+          org.quartz.scheduler.makeSchedulerThreadDaemon: true
+          org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
+          org.quartz.jobStore.clusterCheckinInterval: 5000
+    
+    # Mybatis-plus configuration, you don't need to change it
+    mybatis-plus:
+      mapper-locations: classpath:org/apache/dolphinscheduler/dao/mapper/*Mapper.xml
+      type-aliases-package: org.apache.dolphinscheduler.dao.entity
+      configuration:
+        cache-enabled: false
+        call-setters-on-nulls: true
+        map-underscore-to-camel-case: true
+        jdbc-type-for-null: NULL
+      global-config:
+        db-config:
+          id-type: auto
+        banner: false
+    
+    
+    registry:
+      type: zookeeper
+      zookeeper:
+        namespace: dolphinscheduler
+        connect-string: localhost:2181
+        retry-policy:
+          base-sleep-time: 60ms
+          max-sleep: 300ms
+          max-retries: 5
+        session-timeout: 30s
+        connection-timeout: 9s
+        block-until-connected: 600ms
+        digest: ~
+    
+    master:
+      listen-port: 5678
+      # master fetch command num
+      fetch-command-num: 10
+      # master prepare execute thread number to limit handle commands in parallel
+      pre-exec-threads: 10
+      # master execute thread number to limit process instances in parallel
+      exec-threads: 100
+      # master dispatch task number per batch, if all the tasks dispatch failed in a batch, will sleep 1s.
+      dispatch-task-number: 3
+      # master host selector to select a suitable worker, default value: LowerWeight. Optional values include random, round_robin, lower_weight
+      host-selector: lower_weight
+      # master heartbeat interval
+      max-heartbeat-interval: 10s
+      # master commit task retry times
+      task-commit-retry-times: 5
+      # master commit task interval
+      task-commit-interval: 1s
+      state-wheel-interval: 5s
+      server-load-protection:
+        # If set true, will open master overload protection
+        enabled: true
+        # Master max system cpu usage, when the master's system cpu usage is smaller then this value, master server can execute workflow.
+        max-system-cpu-usage-percentage-thresholds: 0.7
+        # Master max jvm cpu usage, when the master's jvm cpu usage is smaller then this value, master server can execute workflow.
+        max-jvm-cpu-usage-percentage-thresholds: 0.7
+        # Master max System memory usage , when the master's system memory usage is smaller then this value, master server can execute workflow.
+        max-system-memory-usage-percentage-thresholds: 0.7
+        # Master max disk usage , when the master's disk usage is smaller then this value, master server can execute workflow.
+        max-disk-usage-percentage-thresholds: 0.7
+      # failover interval, the unit is minute
+      failover-interval: 10m
+      # kill yarn / k8s application when failover taskInstance, default true
+      kill-application-when-task-failover: true
+      registry-disconnect-strategy:
+        # The disconnect strategy: stop, waiting
+        strategy: waiting
+        # The max waiting time to reconnect to registry if you set the strategy to waiting
+        max-waiting-time: 100s
+      worker-group-refresh-interval: 10s
+    
+    server:
+      port: 5679
+    
+    management:
+      endpoints:
+        web:
+          exposure:
+            include: health,metrics,prometheus
+      endpoint:
+        health:
+          enabled: true
+          show-details: always
+      health:
+        db:
+          enabled: true
+        defaults:
+          enabled: false
+      metrics:
+        tags:
+          application: ${spring.application.name}
+    
+    metrics:
+      enabled: true
+    
+    # Override by profile
+    ---
+    spring:
+      config:
+        activate:
+          on-profile: mysql
+      datasource:
+        driver-class-name: com.mysql.cj.jdbc.Driver
+        url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler
+        username: root
+        password: root
+      quartz:
+        properties:
+          org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.StdJDBCDelegate
+{{- end }}

deploy/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-worker.yaml

@@ -0,0 +1,108 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+{{- if and .Values.worker.enabled }}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ include "dolphinscheduler.fullname" . }}-worker
+  labels:
+    app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-worker
+    {{- include "dolphinscheduler.worker.labels" . | nindent 4 }}
+data:
+  application.yaml: |
+    spring:
+      banner:
+        charset: UTF-8
+      jackson:
+        time-zone: UTC
+        date-format: "yyyy-MM-dd HH:mm:ss"
+      autoconfigure:
+        exclude:
+          - org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration
+    
+    registry:
+      type: zookeeper
+      zookeeper:
+        namespace: dolphinscheduler
+        connect-string: localhost:2181
+        retry-policy:
+          base-sleep-time: 60ms
+          max-sleep: 300ms
+          max-retries: 5
+        session-timeout: 30s
+        connection-timeout: 9s
+        block-until-connected: 600ms
+        digest: ~
+    
+    worker:
+      # worker listener port
+      listen-port: 1234
+      # worker execute thread number to limit task instances in parallel
+      exec-threads: 100
+      # worker heartbeat interval
+      max-heartbeat-interval: 10s
+      # worker host weight to dispatch tasks, default value 100
+      host-weight: 100
+      server-load-protection:
+        # If set true, will open worker overload protection
+        enabled: true
+        # Worker max system cpu usage, when the worker's system cpu usage is smaller then this value, worker server can be dispatched tasks.
+        max-system-cpu-usage-percentage-thresholds: 0.7
+        # Worker max jvm cpu usage, when the worker's jvm cpu usage is smaller then this value, worker server can be dispatched tasks.
+        max-jvm-cpu-usage-percentage-thresholds: 0.7
+        # Worker max System memory usage , when the master's system memory usage is smaller then this value, master server can execute workflow.
+        max-system-memory-usage-percentage-thresholds: 0.7
+        # Worker max disk usage , when the worker's disk usage is smaller then this value, worker server can be dispatched tasks.
+        max-disk-usage-percentage-thresholds: 0.7
+      registry-disconnect-strategy:
+        # The disconnect strategy: stop, waiting
+        strategy: waiting
+        # The max waiting time to reconnect to registry if you set the strategy to waiting
+        max-waiting-time: 100s
+      task-execute-threads-full-policy: REJECT
+      tenant-config:
+        # tenant corresponds to the user of the system, which is used by the worker to submit the job. If system does not have this user, it will be automatically created after the parameter worker.tenant.auto.create is true.
+        auto-create-tenant-enabled: true
+        # Scenes to be used for distributed users. For example, users created by FreeIpa are stored in LDAP. This parameter only applies to Linux, When this parameter is true, auto-create-tenant-enabled has no effect and will not automatically create tenants.
+        distributed-tenant-enabled: false
+        # If set true, will use worker bootstrap user as the tenant to execute task when the tenant is `default`.
+        default-tenant-enabled: false
+    
+    server:
+      port: 1235
+    
+    management:
+      endpoints:
+        web:
+          exposure:
+            include: health,metrics,prometheus
+      endpoint:
+        health:
+          enabled: true
+          show-details: always
+      health:
+        db:
+          enabled: true
+        defaults:
+          enabled: false
+      metrics:
+        tags:
+          application: ${spring.application.name}
+    
+    metrics:
+      enabled: true
+{{- end }}

deploy/kubernetes/dolphinscheduler/templates/configmap.yaml

@@ -22,12 +22,22 @@ metadata:
     app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-common
     {{- include "dolphinscheduler.common.labels" . | nindent 4 }}
 data:
-  common_properties: |-
+  {{- if and .Values.api.taskTypeFilter.enabled }}
+  task-type-config.yaml: |-
+    {{- if .Values.api.taskTypeFilter.task }}
+    task:
+      {{- range $key, $value := .Values.api.taskTypeFilter.task }}
+      {{ $key }}:
+        {{- toYaml $value | nindent 8 }}
+      {{- end }}
+    {{- end }}
+  {{- end }}
+  common.properties: |-
   {{- if index .Values.conf "common" }}
   {{- range $key, $value := index .Values.conf "common" }}
     {{- if and $.Values.minio.enabled }}
     {{- if eq $key "resource.storage.type" }}{{ $value = "S3" }}{{- end }}
-    {{- if eq $key "resource.aws.s3.endpoint" }}{{ $value = print "http://" (include "dolphinscheduler.minio.fullname" $) ":9000" }}{{- end }}
+    {{- if eq $key "aws.s3.endpoint" }}{{ $value = print "http://" (include "dolphinscheduler.minio.fullname" $) ":9000" }}{{- end }}
     {{- end }}
     {{ $key }}={{ $value }}
   {{- end }}

deploy/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-alert.yaml

@@ -114,7 +114,10 @@ spec:
               name: {{ include "dolphinscheduler.fullname" . }}-alert
             - name: config-volume
               mountPath: /opt/dolphinscheduler/conf/common.properties
-              subPath: common_properties
+              subPath: common.properties
+            - name: alert-config-volume
+              mountPath: /opt/dolphinscheduler/conf/application.yaml
+              subPath: application.yaml
       volumes:
         - name: {{ include "dolphinscheduler.fullname" . }}-alert
           {{- if .Values.alert.persistentVolumeClaim.enabled }}
@@ -126,4 +129,7 @@ spec:
         - name: config-volume
           configMap:
             name: {{ include "dolphinscheduler.fullname" . }}-configs
+        - name: alert-config-volume
+          configMap:
+            name: {{ include "dolphinscheduler.fullname" . }}-alert
 {{- end }}

deploy/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-api.yaml

@@ -115,7 +115,15 @@ spec:
               name: {{ include "dolphinscheduler.fullname" . }}-api
             - name: config-volume
               mountPath: /opt/dolphinscheduler/conf/common.properties
-              subPath: common_properties
+              subPath: common.properties
+            {{- if .Values.api.taskTypeFilter.enabled }}
+            - name: config-volume
+              mountPath: /opt/dolphinscheduler/conf/task-type-config.yaml
+              subPath: task-type-config.yaml
+            {{- end }}
+            - name: api-config-volume
+              mountPath: /opt/dolphinscheduler/conf/application.yaml
+              subPath: application.yaml
             {{- include "dolphinscheduler.sharedStorage.volumeMount" . | nindent 12 }}
             {{- include "dolphinscheduler.fsFileResource.volumeMount" . | nindent 12 }}
             {{- include "dolphinscheduler.ldap.ssl.volumeMount" . | nindent 12 }}
@@ -131,6 +139,9 @@ spec:
         - name: config-volume
           configMap:
             name: {{ include "dolphinscheduler.fullname" . }}-configs
+        - name: api-config-volume
+          configMap:
+            name: {{ include "dolphinscheduler.fullname" . }}-api
         {{- include "dolphinscheduler.sharedStorage.volume" . | nindent 8 }}
         {{- include "dolphinscheduler.fsFileResource.volume" . | nindent 8 }}
         {{- include "dolphinscheduler.ldap.ssl.volume" . | nindent 8 }}

deploy/kubernetes/dolphinscheduler/templates/secret-external-database.yaml

@@ -25,4 +25,4 @@ metadata:
 type: Opaque
 data:
   database-password: {{ .Values.externalDatabase.password | b64enc | quote }}
-{{- end }}
+{{- end }}

deploy/kubernetes/dolphinscheduler/templates/secret-external-etcd-ssl.yaml

@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-{{- if .Values.etcd.ssl.enabled }}
+{{- if .Values.registryEtcd.ssl.enabled }}
 apiVersion: v1
 kind: Secret
 metadata:
@@ -24,7 +24,7 @@ metadata:
     {{- include "dolphinscheduler.common.labels" . | nindent 4 }}
 type: Opaque
 data:
-  cert-file: {{ .Files.Get .Values.etcd.ssl.certFile | b64enc | quote }}
-  key-cert-chain-file: {{ .Files.Get .Values.etcd.ssl.keyCertChainFile | b64enc | quote }}
-  key-file: {{ .Files.Get .Values.etcd.ssl.keyFile | b64enc | quote }}
+  cert-file: {{ .Files.Get .Values.registryEtcd.ssl.certFile | b64enc | quote }}
+  key-cert-chain-file: {{ .Files.Get .Values.registryEtcd.ssl.keyCertChainFile | b64enc | quote }}
+  key-file: {{ .Files.Get .Values.registryEtcd.ssl.keyFile | b64enc | quote }}
 {{- end }}

deploy/kubernetes/dolphinscheduler/templates/secret-registry-database.yaml

@@ -0,0 +1,28 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+{{- if .Values.registryJdbc.hikariConfig.enabled }}
+apiVersion: v1
+kind: Secret
+metadata:
+  name: {{ include "dolphinscheduler.fullname" . }}-registry-db
+  labels:
+    app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-registry-db
+    {{- include "dolphinscheduler.common.labels" . | nindent 4 }}
+type: Opaque
+data:
+  registry-password: {{ .Values.registryJdbc.hikariConfig.password | b64enc | quote }}
+{{- end }}

deploy/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-master.yaml

@@ -109,10 +109,13 @@ spec:
           volumeMounts:
             - mountPath: "/opt/dolphinscheduler/logs"
               name: {{ include "dolphinscheduler.fullname" . }}-master
+            - name: master-config-volume
+              mountPath: /opt/dolphinscheduler/conf/application.yaml
+              subPath: application.yaml
             {{- include "dolphinscheduler.sharedStorage.volumeMount" . | nindent 12 }}
             - name: config-volume
               mountPath: /opt/dolphinscheduler/conf/common.properties
-              subPath: common_properties
+              subPath: common.properties
             {{- include "dolphinscheduler.etcd.ssl.volumeMount" . | nindent 12 }}
       volumes:
         - name: {{ include "dolphinscheduler.fullname" . }}-master
@@ -122,6 +125,9 @@ spec:
           {{- else }}
           emptyDir: {}
           {{- end }}
+        - name: master-config-volume
+          configMap:
+            name: {{ include "dolphinscheduler.fullname" . }}-master
         {{- include "dolphinscheduler.sharedStorage.volume" . | nindent 8 }}
         - name: config-volume
           configMap:

deploy/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-worker.yaml

@@ -111,9 +111,12 @@ spec:
               name: {{ include "dolphinscheduler.fullname" . }}-worker-data
             - mountPath: "/opt/dolphinscheduler/logs"
               name: {{ include "dolphinscheduler.fullname" . }}-worker-logs
+            - name: worker-config-volume
+              mountPath: /opt/dolphinscheduler/conf/application.yaml
+              subPath: application.yaml
             - name: config-volume
               mountPath: /opt/dolphinscheduler/conf/common.properties
-              subPath: common_properties
+              subPath: common.properties
             {{- include "dolphinscheduler.sharedStorage.volumeMount" . | nindent 12 }}
             {{- include "dolphinscheduler.fsFileResource.volumeMount" . | nindent 12 }}
             {{- include "dolphinscheduler.etcd.ssl.volumeMount" . | nindent 12 }}
@@ -139,6 +142,9 @@ spec:
         - name: {{ include "dolphinscheduler.fullname" . }}-worker-logs
           emptyDir: {}
         {{- end }}
+        - name: worker-config-volume
+          configMap:
+            name: {{ include "dolphinscheduler.fullname" . }}-worker
         - name: config-volume
           configMap:
             name: {{ include "dolphinscheduler.fullname" . }}-configs

doap.rdf

@@ -0,0 +1,45 @@
+<?xml version="1.0"?>
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one or more
+    contributor license agreements.  See the NOTICE file distributed with
+    this work for additional information regarding copyright ownership.
+    The ASF licenses this file to You under the Apache License, Version 2.0
+    (the "License"); you may not use this file except in compliance with
+    the License.  You may obtain a copy of the License at
+   
+         https://www.apache.org/licenses/LICENSE-2.0
+   
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+-->
+<?xml-stylesheet type="text/xsl"?>
+<rdf:RDF xml:lang="en"
+         xmlns="http://usefulinc.com/ns/doap#" 
+         xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
+         xmlns:asfext="http://projects.apache.org/ns/asfext#"
+         xmlns:foaf="http://xmlns.com/foaf/0.1/">
+  <Project rdf:about="https://dolphinscheduler.apache.org">
+    <created>2023-09-09</created>
+    <license rdf:resource="https://spdx.org/licenses/Apache-2.0" />
+    <name>Apache DolphinScheduler</name>
+    <homepage rdf:resource="https://dolphinscheduler.apache.org" />
+    <asfext:pmc rdf:resource="https://dolphinscheduler.apache.org" />
+    <shortdesc>Apache DolphinScheduler is a modern data orchestration platform.</shortdesc>
+    <description>Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code. It is also provided powerful user interface, dedicated to solving complex task dependencies in the data pipeline and providing various types of jobs available out of the box</description>
+    <bug-database rdf:resource="https://github.com/apache/dolphinscheduler/issues" />
+    <mailing-list rdf:resource="https://lists.apache.org/list.html?dev@dolphinscheduler.apache.org" />
+    <download-page rdf:resource="https://dolphinscheduler.apache.org/en-us/download" />
+    <programming-language>Java</programming-language>
+    <category rdf:resource="https://projects.apache.org/category/big-data" />
+    <repository>
+      <GitRepository>
+        <location rdf:resource="https://github.com/apache/dolphinscheduler.git"/>
+        <browse rdf:resource="https://github.com/apache/dolphinscheduler"/>
+      </GitRepository>
+    </repository>
+  </Project>
+</rdf:RDF>
+

docs/configs/index.md.jsx

@@ -60,6 +60,9 @@ import docs315Config from '../../../site_config/docs3-1-5';
 import docs316Config from '../../../site_config/docs3-1-6';
 import docs317Config from '../../../site_config/docs3-1-7';
 import docs318Config from '../../../site_config/docs3-1-8';
+import docs319Config from '../../../site_config/docs3-1-9';
+import docs320Config from '../../../site_config/docs3-2-0';
+import docs310Config from '../../../site_config/docs3-2-1';
 import docsDevConfig from '../../../site_config/docsdev';
 
 const docsSource = {
@@ -96,6 +99,9 @@ const docsSource = {
   '3.1.6': docs316Config,
   '3.1.7': docs317Config,
   '3.1.8': docs318Config,
+  '3.1.9': docs319Config,
+  '3.2.0': docs320Config,
+  '3.2.1': docs321Config,
   dev: docsDevConfig,
 };
 

docs/configs/site.js

@@ -24,7 +24,7 @@ export default {
   port: 8080,
   domain: 'dolphinscheduler.apache.org',
   copyToDist: ['asset', 'img', 'file', '.asf.yaml', 'sitemap.xml', '.nojekyll', '.htaccess', 'googled0df7b96f277a143.html'],
-  docsLatest: '3.1.8',
+  docsLatest: '3.2.1',
   defaultSearch: 'google', // default search engine
   defaultLanguage: 'en-us',
   'en-us': {
@@ -45,13 +45,13 @@ export default {
         children: [
           {
             key: 'docs0',
-            text: 'latest(3.1.8)',
+            text: 'latest(3.2.1)',
             link: '/en-us/docs/latest/user_doc/about/introduction.html',
           },
           {
             key: 'docs1',
-            text: '3.0.6',
-            link: '/en-us/docs/3.0.6/user_doc/about/introduction.html',
+            text: '3.1.9',
+            link: '/en-us/docs/3.1.9/user_doc/about/introduction.html',
           },
           {
             key: 'docs2',
@@ -173,13 +173,13 @@ export default {
         children: [
           {
             key: 'docs0',
-            text: '最新版本latest(3.1.8)',
+            text: '最新版本latest(3.2.1)',
             link: '/zh-cn/docs/latest/user_doc/about/introduction.html',
           },
           {
             key: 'docs1',
-            text: '3.0.6',
-            link: '/zh-cn/docs/3.0.6/user_doc/about/introduction.html',
+            text: '3.1.9',
+            link: '/zh-cn/docs/3.1.9/user_doc/about/introduction.html',
           },
           {
             key: 'docs2',

docs/docs/en/about/glossary.md

@@ -61,7 +61,7 @@ process fails and ends
 
 - dolphinscheduler-dao provides operations such as database access.
 
-- dolphinscheduler-remote client and server based on netty
+- dolphinscheduler-extract dolphinscheduler extract module, providing master/worker/alert sdk.
 
 - dolphinscheduler-service service module, including Quartz, Zookeeper, log client access service, easy to call server
   module and api module

docs/docs/en/architecture/cache.md

@@ -1,42 +0,0 @@
-# Cache
-
-## Purpose
-
-Due to the large database read operations during the master-server scheduling process. Such as read tables like `tenant`, `user`, `processDefinition`, etc. Operations stress read pressure to the DB, and slow down the entire core scheduling process.
-
-By considering this part of the business data is a high-read and low-write scenario, a cache module is introduced to reduce the DB read pressure and speed up the core scheduling process.
-
-## Cache Settings
-
-```yaml
-spring:
-  cache:
-    # default disable cache, you can enable by `type: caffeine`
-    type: none
-    cache-names:
-      - tenant
-      - user
-      - processDefinition
-      - processTaskRelation
-      - taskDefinition
-    caffeine:
-      spec: maximumSize=100,expireAfterWrite=300s,recordStats
-```
-
-The cache module uses [spring-cache](https://spring.io/guides/gs/caching/), so you can set cache config like whether to enable cache (`none` to disable by default), cache types in the spring `application.yaml` directly.
-
-Currently, implements the config of [caffeine](https://github.com/ben-manes/caffeine), you can assign cache configs like cache size, expire time, etc.
-
-## Cache Read
-
-The cache module adopts the `@Cacheable` annotation from spring-cache and you can annotate the annotation in the related mapper layer. Refer to the `TenantMapper`.
-
-## Cache Evict
-
-The business data updates come from the api-server, and the cache side is in the master-server. Then it is necessary to monitor the data updates from the api-server (use aspect point cut interceptor `@CacheEvict`), and notify the master-server of `cacheEvictCommand` when processing a cache eviction.
-
-Note: the final strategy for cache update comes from the expiration strategy configuration in caffeine, therefore configure it under the business scenarios;
-
-The sequence diagram shows below:
-
-<img src="../../../img/cache-evict.png" alt="cache-evict" style="zoom: 67%;" />

docs/docs/en/architecture/configuration.md

@@ -110,7 +110,8 @@ The directory structure of DolphinScheduler is as follows:
 
 dolphinscheduler-daemon.sh is responsible for DolphinScheduler startup and shutdown.
 Essentially, start-all.sh or stop-all.sh startup and shutdown the cluster via dolphinscheduler-daemon.sh.
-Currently, DolphinScheduler just makes a basic config, remember to config further JVM options based on your practical situation of resources.
+Currently, DolphinScheduler just makes a basic config, remember to config further JVM options based on your practical
+situation of resources.
 
 Default simplified parameters are:
 
@@ -128,44 +129,47 @@ export DOLPHINSCHEDULER_OPTS="
 "
 ```
 
-> "-XX:DisableExplicitGC" is not recommended due to may lead to memory link (DolphinScheduler dependent on Netty to communicate).
-> If add "-Djava.net.preferIPv6Addresses=true" will use ipv6 address, if add "-Djava.net.preferIPv4Addresses=true" will use ipv4 address, if doesn't set the two parameter will use ipv4 or ipv6.
+> "-XX:DisableExplicitGC" is not recommended due to may lead to memory link (DolphinScheduler dependent on Netty to
+> communicate).
+> If add "-Djava.net.preferIPv6Addresses=true" will use ipv6 address, if add "-Djava.net.preferIPv4Addresses=true" will
+> use ipv4 address, if doesn't set the two parameter will use ipv4 or ipv6.
 
 ### Database connection related configuration
 
 DolphinScheduler uses Spring Hikari to manage database connections, configuration file location:
 
-|Service| Configuration file  |
-|--|--|
-|Master Server | `master-server/conf/application.yaml`|
-|Api Server| `api-server/conf/application.yaml`|
-|Worker Server| `worker-server/conf/application.yaml`|
-|Alert Server| `alert-server/conf/application.yaml`|
+|    Service    |          Configuration file           |
+|---------------|---------------------------------------|
+| Master Server | `master-server/conf/application.yaml` |
+| Api Server    | `api-server/conf/application.yaml`    |
+| Worker Server | `worker-server/conf/application.yaml` |
+| Alert Server  | `alert-server/conf/application.yaml`  |
 
 The default configuration is as follows:
 
-|Parameters | Default value| Description|
-|--|--|--|
-|spring.datasource.driver-class-name| org.postgresql.Driver |datasource driver|
-|spring.datasource.url| jdbc:postgresql://127.0.0.1:5432/dolphinscheduler |datasource connection url|
-|spring.datasource.username|root|datasource username|
-|spring.datasource.password|root|datasource password|
-|spring.datasource.hikari.connection-test-query|select 1|validate connection by running the SQL|
-|spring.datasource.hikari.minimum-idle| 5| minimum connection pool size number|
-|spring.datasource.hikari.auto-commit|true|whether auto commit|
-|spring.datasource.hikari.pool-name|DolphinScheduler|name of the connection pool|
-|spring.datasource.hikari.maximum-pool-size|50| maximum connection pool size number|
-|spring.datasource.hikari.connection-timeout|30000|connection timeout|
-|spring.datasource.hikari.idle-timeout|600000|Maximum idle connection survival time|
-|spring.datasource.hikari.leak-detection-threshold|0|Connection leak detection threshold|
-|spring.datasource.hikari.initialization-fail-timeout|1|Connection pool initialization failed timeout|
+|                      Parameters                      |                   Default value                   |                  Description                  |
+|------------------------------------------------------|---------------------------------------------------|-----------------------------------------------|
+| spring.datasource.driver-class-name                  | org.postgresql.Driver                             | datasource driver                             |
+| spring.datasource.url                                | jdbc:postgresql://127.0.0.1:5432/dolphinscheduler | datasource connection url                     |
+| spring.datasource.username                           | root                                              | datasource username                           |
+| spring.datasource.password                           | root                                              | datasource password                           |
+| spring.datasource.hikari.connection-test-query       | select 1                                          | validate connection by running the SQL        |
+| spring.datasource.hikari.minimum-idle                | 5                                                 | minimum connection pool size number           |
+| spring.datasource.hikari.auto-commit                 | true                                              | whether auto commit                           |
+| spring.datasource.hikari.pool-name                   | DolphinScheduler                                  | name of the connection pool                   |
+| spring.datasource.hikari.maximum-pool-size           | 50                                                | maximum connection pool size number           |
+| spring.datasource.hikari.connection-timeout          | 30000                                             | connection timeout                            |
+| spring.datasource.hikari.idle-timeout                | 600000                                            | Maximum idle connection survival time         |
+| spring.datasource.hikari.leak-detection-threshold    | 0                                                 | Connection leak detection threshold           |
+| spring.datasource.hikari.initialization-fail-timeout | 1                                                 | Connection pool initialization failed timeout |
 
 Note that DolphinScheduler also supports database configuration through `bin/env/dolphinscheduler_env.sh`.
 
-### Zookeeper related configuration
+### Registry Related configuration
 
-DolphinScheduler uses Zookeeper for cluster management, fault tolerance, event monitoring and other functions. Configuration file location:
-|Service| Configuration file  |
+DolphinScheduler uses Zookeeper for cluster management, fault tolerance, event monitoring and other functions.
+Configuration file location:
+|Service| Configuration file |
 |--|--|
 |Master Server | `master-server/conf/application.yaml`|
 |Api Server| `api-server/conf/application.yaml`|
@@ -173,30 +177,35 @@ DolphinScheduler uses Zookeeper for cluster management, fault tolerance, event m
 
 The default configuration is as follows:
 
-|Parameters | Default value| Description|
-|--|--|--|
-|registry.zookeeper.namespace|dolphinscheduler|namespace of zookeeper|
-|registry.zookeeper.connect-string|localhost:2181| the connection string of zookeeper|
-|registry.zookeeper.retry-policy.base-sleep-time|60ms|time to wait between subsequent retries|
-|registry.zookeeper.retry-policy.max-sleep|300ms|maximum time to wait between subsequent retries|
-|registry.zookeeper.retry-policy.max-retries|5|maximum retry times|
-|registry.zookeeper.session-timeout|30s|session timeout|
-|registry.zookeeper.connection-timeout|30s|connection timeout|
-|registry.zookeeper.block-until-connected|600ms|waiting time to block until the connection succeeds|
-|registry.zookeeper.digest|{username}:{password}|digest of zookeeper to access znode, works only when acl is enabled, for more details please check [https://zookeeper.apache.org/doc/r3.4.14/zookeeperAdmin.html](Apache Zookeeper doc) |
+|                   Parameters                    |     Default value     |                                                                                       Description                                                                                       |
+|-------------------------------------------------|-----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| registry.zookeeper.namespace                    | dolphinscheduler      | namespace of zookeeper                                                                                                                                                                  |
+| registry.zookeeper.connect-string               | localhost:2181        | the connection string of zookeeper                                                                                                                                                      |
+| registry.zookeeper.retry-policy.base-sleep-time | 60ms                  | time to wait between subsequent retries                                                                                                                                                 |
+| registry.zookeeper.retry-policy.max-sleep       | 300ms                 | maximum time to wait between subsequent retries                                                                                                                                         |
+| registry.zookeeper.retry-policy.max-retries     | 5                     | maximum retry times                                                                                                                                                                     |
+| registry.zookeeper.session-timeout              | 30s                   | session timeout                                                                                                                                                                         |
+| registry.zookeeper.connection-timeout           | 30s                   | connection timeout                                                                                                                                                                      |
+| registry.zookeeper.block-until-connected        | 600ms                 | waiting time to block until the connection succeeds                                                                                                                                     |
+| registry.zookeeper.digest                       | {username}:{password} | digest of zookeeper to access znode, works only when acl is enabled, for more details please check [https://zookeeper.apache.org/doc/r3.4.14/zookeeperAdmin.html](Apache Zookeeper doc) |
 
 Note that DolphinScheduler also supports zookeeper related configuration through `bin/env/dolphinscheduler_env.sh`.
 
+For ETCD Registry, please see more details
+on [link](https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-etcd/README.md).
+For JDBC Registry, please see more details
+on [link](https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-jdbc/README.md).
+
 ### common.properties [hadoop、s3、yarn config properties]
 
 Currently, common.properties mainly configures Hadoop,s3a related configurations. Configuration file location:
 
-|Service| Configuration file  |
-|--|--|
-|Master Server | `master-server/conf/common.properties`|
-|Api Server| `api-server/conf/common.properties`|
-|Worker Server| `worker-server/conf/common.properties`|
-|Alert Server| `alert-server/conf/common.properties`|
+|    Service    |                          Configuration file                           |
+|---------------|-----------------------------------------------------------------------|
+| Master Server | `master-server/conf/common.properties`                                |
+| Api Server    | `api-server/conf/common.properties`, `api-server/conf/aws.yaml`       |
+| Worker Server | `worker-server/conf/common.properties`, `worker-server/conf/aws.yaml` |
+| Alert Server  | `alert-server/conf/common.properties`                                 |
 
 The default configuration is as follows:
 
@@ -205,10 +214,6 @@ The default configuration is as follows:
 | data.basedir.path                             | /tmp/dolphinscheduler                            | local directory used to store temp files                                                                                                                                                                                                                                                                                                                                                                                             |
 | resource.storage.type                         | NONE                                             | type of resource files: HDFS, S3, OSS, GCS, ABS, NONE                                                                                                                                                                                                                                                                                                                                                                                |
 | resource.upload.path                          | /dolphinscheduler                                | storage path of resource files                                                                                                                                                                                                                                                                                                                                                                                                       |
-| aws.access.key.id                             | minioadmin                                       | access key id of S3                                                                                                                                                                                                                                                                                                                                                                                                                  |
-| aws.secret.access.key                         | minioadmin                                       | secret access key of S3                                                                                                                                                                                                                                                                                                                                                                                                              |
-| aws.region                                    | us-east-1                                        | region of S3                                                                                                                                                                                                                                                                                                                                                                                                                         |
-| aws.s3.endpoint                               | http://minio:9000                                | endpoint of S3                                                                                                                                                                                                                                                                                                                                                                                                                       |
 | hdfs.root.user                                | hdfs                                             | configure users with corresponding permissions if storage type is HDFS                                                                                                                                                                                                                                                                                                                                                               |
 | fs.defaultFS                                  | hdfs://mycluster:8020                            | If resource.storage.type=S3, then the request url would be similar to 's3a://dolphinscheduler'. Otherwise if resource.storage.type=HDFS and hadoop supports HA, copy core-site.xml and hdfs-site.xml into 'conf' directory                                                                                                                                                                                                           |
 | hadoop.security.authentication.startup.state  | false                                            | whether hadoop grant kerberos permission                                                                                                                                                                                                                                                                                                                                                                                             |
@@ -226,7 +231,7 @@ The default configuration is as follows:
 | yarn.job.history.status.address               | http://ds1:19888/ws/v1/history/mapreduce/jobs/%s | job history status url of yarn                                                                                                                                                                                                                                                                                                                                                                                                       |
 | datasource.encryption.enable                  | false                                            | whether to enable datasource encryption                                                                                                                                                                                                                                                                                                                                                                                              |
 | datasource.encryption.salt                    | !@#$%^&*                                         | the salt of the datasource encryption                                                                                                                                                                                                                                                                                                                                                                                                |
-| data-quality.jar.name                         | dolphinscheduler-data-quality-dev-SNAPSHOT.jar   | the jar of data quality                                                                                                                                                                                                                                                                                                                                                                                                              |
+| data-quality.jar.dir                          |                                                  | the jar of data quality                                                                                                                                                                                                                                                                                                                                                                                                              |
 | support.hive.oneSession                       | false                                            | specify whether hive SQL is executed in the same session                                                                                                                                                                                                                                                                                                                                                                             |
 | sudo.enable                                   | true                                             | whether to enable sudo                                                                                                                                                                                                                                                                                                                                                                                                               |
 | alert.rpc.port                                | 50052                                            | the RPC port of Alert Server                                                                                                                                                                                                                                                                                                                                                                                                         |
@@ -237,104 +242,111 @@ The default configuration is as follows:
 
 Location: `api-server/conf/application.yaml`
 
-|Parameters | Default value| Description|
-|--|--|--|
-|server.port|12345|api service communication port|
-|server.servlet.session.timeout|120m|session timeout|
-|server.servlet.context-path|/dolphinscheduler/ |request path|
-|spring.servlet.multipart.max-file-size|1024MB|maximum file size|
-|spring.servlet.multipart.max-request-size|1024MB|maximum request size|
-|server.jetty.max-http-post-size|5000000|jetty maximum post size|
-|spring.banner.charset|UTF-8|message encoding|
-|spring.jackson.time-zone|UTC|time zone|
-|spring.jackson.date-format|"yyyy-MM-dd HH:mm:ss"|time format|
-|spring.messages.basename|i18n/messages|i18n config|
-|security.authentication.type|PASSWORD|authentication type|
-|security.authentication.ldap.user.admin|read-only-admin|admin user account when you log-in with LDAP|
-|security.authentication.ldap.urls|ldap://ldap.forumsys.com:389/|LDAP urls|
-|security.authentication.ldap.base.dn|dc=example,dc=com|LDAP base dn|
-|security.authentication.ldap.username|cn=read-only-admin,dc=example,dc=com|LDAP username|
-|security.authentication.ldap.password|password|LDAP password|
-|security.authentication.ldap.user.identity-attribute|uid|LDAP user identity attribute|
-|security.authentication.ldap.user.email-attribute|mail|LDAP user email attribute|
-|security.authentication.ldap.user.not-exist-action|CREATE|action when ldap user is not exist,default value: CREATE. Optional values include(CREATE,DENY)|
-|security.authentication.ldap.ssl.enable|false|LDAP switch|
-|security.authentication.ldap.ssl.trust-store|ldapkeystore.jks|LDAP jks file absolute path|
-|security.authentication.ldap.ssl.trust-store-password|password|LDAP jks password|
-|security.authentication.casdoor.user.admin||admin user account when you log-in with Casdoor|
-|casdoor.endpoint||Casdoor server url|
-|casdoor.client-id||id in Casdoor|
-|casdoor.client-secret||secret in Casdoor|
-|casdoor.certificate||certificate in Casdoor|
-|casdoor.organization-name||organization name in Casdoor|
-|casdoor.application-name||application name in Casdoor|
-|casdoor.redirect-url||doplhinscheduler login url|
-|api.traffic.control.global.switch|false|traffic control global switch|
-|api.traffic.control.max-global-qps-rate|300|global max request number per second|
-|api.traffic.control.tenant-switch|false|traffic control tenant switch|
-|api.traffic.control.default-tenant-qps-rate|10|default tenant max request number per second|
-|api.traffic.control.customize-tenant-qps-rate||customize tenant max request number per second|
+|                      Parameters                       |            Default value             |                                          Description                                           |
+|-------------------------------------------------------|--------------------------------------|------------------------------------------------------------------------------------------------|
+| server.port                                           | 12345                                | api service communication port                                                                 |
+| server.servlet.session.timeout                        | 120m                                 | session timeout                                                                                |
+| server.servlet.context-path                           | /dolphinscheduler/                   | request path                                                                                   |
+| spring.servlet.multipart.max-file-size                | 1024MB                               | maximum file size                                                                              |
+| spring.servlet.multipart.max-request-size             | 1024MB                               | maximum request size                                                                           |
+| server.jetty.max-http-post-size                       | 5000000                              | jetty maximum post size                                                                        |
+| spring.banner.charset                                 | UTF-8                                | message encoding                                                                               |
+| spring.jackson.time-zone                              | UTC                                  | time zone                                                                                      |
+| spring.jackson.date-format                            | "yyyy-MM-dd HH:mm:ss"                | time format                                                                                    |
+| spring.messages.basename                              | i18n/messages                        | i18n config                                                                                    |
+| security.authentication.type                          | PASSWORD                             | authentication type                                                                            |
+| security.authentication.ldap.user.admin               | read-only-admin                      | admin user account when you log-in with LDAP                                                   |
+| security.authentication.ldap.urls                     | ldap://ldap.forumsys.com:389/        | LDAP urls                                                                                      |
+| security.authentication.ldap.base.dn                  | dc=example,dc=com                    | LDAP base dn                                                                                   |
+| security.authentication.ldap.username                 | cn=read-only-admin,dc=example,dc=com | LDAP username                                                                                  |
+| security.authentication.ldap.password                 | password                             | LDAP password                                                                                  |
+| security.authentication.ldap.user.identity-attribute  | uid                                  | LDAP user identity attribute                                                                   |
+| security.authentication.ldap.user.email-attribute     | mail                                 | LDAP user email attribute                                                                      |
+| security.authentication.ldap.user.not-exist-action    | CREATE                               | action when ldap user is not exist,default value: CREATE. Optional values include(CREATE,DENY) |
+| security.authentication.ldap.ssl.enable               | false                                | LDAP ssl switch                                                                                |
+| security.authentication.ldap.ssl.trust-store          | ldapkeystore.jks                     | LDAP jks file absolute path                                                                    |
+| security.authentication.ldap.ssl.trust-store-password | password                             | LDAP jks password                                                                              |
+| security.authentication.casdoor.user.admin            |                                      | admin user account when you log-in with Casdoor                                                |
+| casdoor.endpoint                                      |                                      | Casdoor server url                                                                             |
+| casdoor.client-id                                     |                                      | id in Casdoor                                                                                  |
+| casdoor.client-secret                                 |                                      | secret in Casdoor                                                                              |
+| casdoor.certificate                                   |                                      | certificate in Casdoor                                                                         |
+| casdoor.organization-name                             |                                      | organization name in Casdoor                                                                   |
+| casdoor.application-name                              |                                      | application name in Casdoor                                                                    |
+| casdoor.redirect-url                                  |                                      | doplhinscheduler login url                                                                     |
+| api.traffic.control.global.switch                     | false                                | traffic control global switch                                                                  |
+| api.traffic.control.max-global-qps-rate               | 300                                  | global max request number per second                                                           |
+| api.traffic.control.tenant-switch                     | false                                | traffic control tenant switch                                                                  |
+| api.traffic.control.default-tenant-qps-rate           | 10                                   | default tenant max request number per second                                                   |
+| api.traffic.control.customize-tenant-qps-rate         |                                      | customize tenant max request number per second                                                 |
 
 ### Master Server related configuration
 
 Location: `master-server/conf/application.yaml`
 
-|Parameters | Default value| Description|
-|--|--|--|
-|master.listen-port|5678|master listen port|
-|master.fetch-command-num|10|the number of commands fetched by master|
-|master.pre-exec-threads|10|master prepare execute thread number to limit handle commands in parallel|
-|master.exec-threads|100|master execute thread number to limit process instances in parallel|
-|master.dispatch-task-number|3|master dispatch task number per batch|
-|master.host-selector|lower_weight|master host selector to select a suitable worker, default value: LowerWeight. Optional values include random, round_robin, lower_weight|
-|master.heartbeat-interval|10|master heartbeat interval, the unit is second|
-|master.task-commit-retry-times|5|master commit task retry times|
-|master.task-commit-interval|1000|master commit task interval, the unit is millisecond|
-|master.state-wheel-interval|5|time to check status|
-|master.max-cpu-load-avg|1|master max cpuload avg percentage, only higher than the system cpu load average, master server can schedule. default value 1: will use 100% cpu|
-|master.reserved-memory|0.3|master reserved memory, only lower than system available memory, master server can schedule. default value 0.3, only the available memory is higher than 30%, master server can schedule.|
-|master.failover-interval|10|failover interval, the unit is minute|
-|master.kill-application-when-task-failover|true|whether to kill yarn/k8s application when failover taskInstance|
-|master.registry-disconnect-strategy.strategy|stop|Used when the master disconnect from registry, default value: stop. Optional values include stop, waiting|
-|master.registry-disconnect-strategy.max-waiting-time|100s|Used when the master disconnect from registry, and the disconnect strategy is waiting, this config means the master will waiting to reconnect to registry in given times, and after the waiting times, if the master still cannot connect to registry, will stop itself, if the value is 0s, the Master will wait infinitely|
-|master.worker-group-refresh-interval|10s|The interval to refresh worker group from db to memory|
+|                                 Parameters                                  | Default value |                                                                                                                                                         Description                                                                                                                                                          |
+|-----------------------------------------------------------------------------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| master.listen-port                                                          | 5678          | master listen port                                                                                                                                                                                                                                                                                                           |
+| master.pre-exec-threads                                                     | 10            | master prepare execute thread number to limit handle commands in parallel                                                                                                                                                                                                                                                    |
+| master.exec-threads                                                         | 100           | master execute thread number to limit process instances in parallel                                                                                                                                                                                                                                                          |
+| master.dispatch-task-number                                                 | 3             | master dispatch task number per batch                                                                                                                                                                                                                                                                                        |
+| master.host-selector                                                        | lower_weight  | master host selector to select a suitable worker, default value: LowerWeight. Optional values include random, round_robin, lower_weight                                                                                                                                                                                      |
+| master.max-heartbeat-interval                                               | 10s           | master max heartbeat interval                                                                                                                                                                                                                                                                                                |
+| master.task-commit-retry-times                                              | 5             | master commit task retry times                                                                                                                                                                                                                                                                                               |
+| master.task-commit-interval                                                 | 1000          | master commit task interval, the unit is millisecond                                                                                                                                                                                                                                                                         |
+| master.state-wheel-interval                                                 | 5             | time to check status                                                                                                                                                                                                                                                                                                         |
+| master.server-load-protection.enabled                                       | true          | If set true, will open master overload protection                                                                                                                                                                                                                                                                            |
+| master.server-load-protection.max-system-cpu-usage-percentage-thresholds    | 0.7           | Master max system cpu usage, when the master's system cpu usage is smaller then this value, master server can execute workflow.                                                                                                                                                                                              |
+| master.server-load-protection.max-jvm-cpu-usage-percentage-thresholds       | 0.7           | Master max JVM cpu usage, when the master's jvm cpu usage is smaller then this value, master server can execute workflow.                                                                                                                                                                                                    |
+| master.server-load-protection.max-system-memory-usage-percentage-thresholds | 0.7           | Master max system memory usage , when the master's system memory usage is smaller then this value, master server can execute workflow.                                                                                                                                                                                       |
+| master.server-load-protection.max-disk-usage-percentage-thresholds          | 0.7           | Master max disk usage , when the master's disk usage is smaller then this value, master server can execute workflow.                                                                                                                                                                                                         |
+| master.failover-interval                                                    | 10            | failover interval, the unit is minute                                                                                                                                                                                                                                                                                        |
+| master.kill-application-when-task-failover                                  | true          | whether to kill yarn/k8s application when failover taskInstance                                                                                                                                                                                                                                                              |
+| master.registry-disconnect-strategy.strategy                                | stop          | Used when the master disconnect from registry, default value: stop. Optional values include stop, waiting                                                                                                                                                                                                                    |
+| master.registry-disconnect-strategy.max-waiting-time                        | 100s          | Used when the master disconnect from registry, and the disconnect strategy is waiting, this config means the master will waiting to reconnect to registry in given times, and after the waiting times, if the master still cannot connect to registry, will stop itself, if the value is 0s, the Master will wait infinitely |
+| master.worker-group-refresh-interval                                        | 10s           | The interval to refresh worker group from db to memory                                                                                                                                                                                                                                                                       |
+| master.command-fetch-strategy.type                                          | ID_SLOT_BASED | The command fetch strategy, only support `ID_SLOT_BASED`                                                                                                                                                                                                                                                                     |
+| master.command-fetch-strategy.config.id-step                                | 1             | The id auto incremental step of t_ds_command in db                                                                                                                                                                                                                                                                           |
+| master.command-fetch-strategy.config.fetch-size                             | 10            | The number of commands fetched by master                                                                                                                                                                                                                                                                                     |
 
 ### Worker Server related configuration
 
 Location: `worker-server/conf/application.yaml`
 
-|Parameters | Default value| Description|
-|--|--|--|
-|worker.listen-port|1234|worker-service listen port|
-|worker.exec-threads|100|worker-service execute thread number, used to limit the number of task instances in parallel|
-|worker.heartbeat-interval|10|worker-service heartbeat interval, the unit is second|
-|worker.host-weight|100|worker host weight to dispatch tasks|
-|worker.tenant-auto-create|true|tenant corresponds to the user of the system, which is used by the worker to submit the job. If system does not have this user, it will be automatically created after the parameter worker.tenant.auto.create is true.|
-|worker.max-cpu-load-avg|1|worker max cpuload avg, only higher than the system cpu load average, worker server can be dispatched tasks. default value 1: will use 100% cpu.|
-|worker.reserved-memory|0.3|worker reserved memory, only lower than system available memory, worker server can be dispatched tasks. default value 0.3, only the available memory is higher than 30%, worker server can receive task.|
-|worker.alert-listen-host|localhost|the alert listen host of worker|
-|worker.alert-listen-port|50052|the alert listen port of worker|
-|worker.registry-disconnect-strategy.strategy|stop|Used when the worker disconnect from registry, default value: stop. Optional values include stop, waiting|
-|worker.registry-disconnect-strategy.max-waiting-time|100s|Used when the worker disconnect from registry, and the disconnect strategy is waiting, this config means the worker will waiting to reconnect to registry in given times, and after the waiting times, if the worker still cannot connect to registry, will stop itself, if the value is 0s, will wait infinitely |
-|worker.task-execute-threads-full-policy|REJECT|If REJECT, when the task waiting in the worker reaches exec-threads, it will reject the received task and the Master will redispatch it; If CONTINUE, it will put the task into the worker's execution queue and wait for a free thread to start execution|
+|                                 Parameters                                  | Default value |                                                                                                                                                    Description                                                                                                                                                    |
+|-----------------------------------------------------------------------------|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| worker.listen-port                                                          | 1234          | worker-service listen port                                                                                                                                                                                                                                                                                        |
+| worker.exec-threads                                                         | 100           | worker-service execute thread number, used to limit the number of task instances in parallel                                                                                                                                                                                                                      |
+| worker.max-heartbeat-interval                                               | 10s           | worker-service max heartbeat interval                                                                                                                                                                                                                                                                             |
+| worker.host-weight                                                          | 100           | worker host weight to dispatch tasks                                                                                                                                                                                                                                                                              |
+| worker.server-load-protection.enabled                                       | true          | If set true will open worker overload protection                                                                                                                                                                                                                                                                  |
+| worker.server-load-protection.max-system-cpu-usage-percentage-thresholds    | 0.7           | Worker max system cpu usage, when the worker's system cpu usage is smaller then this value, master server can execute workflow.                                                                                                                                                                                   |
+| worker.server-load-protection.max-jvm-cpu-usage-percentage-thresholds       | 0.7           | Worker max JVM cpu usage, when the worker's jvm cpu usage is smaller then this value, master server can execute workflow.                                                                                                                                                                                         |
+| worker.server-load-protection.max-system-memory-usage-percentage-thresholds | 0.7           | Worker max system memory usage , when the worker's system memory usage is smaller then this value, master server can execute workflow.                                                                                                                                                                            |
+| worker.server-load-protection.max-disk-usage-percentage-thresholds          | 0.7           | Worker max disk usage , when the worker's disk usage is smaller then this value, master server can execute workflow.                                                                                                                                                                                              |
+| worker.registry-disconnect-strategy.strategy                                | stop          | Used when the worker disconnect from registry, default value: stop. Optional values include stop, waiting                                                                                                                                                                                                         |
+| worker.registry-disconnect-strategy.max-waiting-time                        | 100s          | Used when the worker disconnect from registry, and the disconnect strategy is waiting, this config means the worker will waiting to reconnect to registry in given times, and after the waiting times, if the worker still cannot connect to registry, will stop itself, if the value is 0s, will wait infinitely |
+| worker.task-execute-threads-full-policy                                     | REJECT        | If REJECT, when the task waiting in the worker reaches exec-threads, it will reject the received task and the Master will redispatch it; If CONTINUE, it will put the task into the worker's execution queue and wait for a free thread to start execution                                                        |
+| worker.tenant-config.auto-create-tenant-enabled                             | true          | tenant corresponds to the user of the system, which is used by the worker to submit the job. If system does not have this user, it will be automatically created after the parameter worker.tenant.auto.create is true.                                                                                           |
+| worker.tenant-config.default-tenant-enabled                                 | false         | If set true, will use worker bootstrap user as the tenant to execute task when the tenant is `default`.                                                                                                                                                                                                           |
 
 ### Alert Server related configuration
 
 Location: `alert-server/conf/application.yaml`
 
-|Parameters | Default value| Description|
-|--|--|--|
-|server.port|50053|the port of Alert Server|
-|alert.port|50052|the port of alert|
+| Parameters  | Default value |       Description        |
+|-------------|---------------|--------------------------|
+| server.port | 50053         | the port of Alert Server |
+| alert.port  | 50052         | the port of alert        |
 
 ### Quartz related configuration
 
 This part describes quartz configs and configure them based on your practical situation and resources.
 
-|Service| Configuration file  |
-|--|--|
-|Master Server | `master-server/conf/application.yaml`|
-|Api Server| `api-server/conf/application.yaml`|
+|    Service    |          Configuration file           |
+|---------------|---------------------------------------|
+| Master Server | `master-server/conf/application.yaml` |
+| Api Server    | `api-server/conf/application.yaml`    |
 
 The default configuration is as follows:
 
@@ -352,7 +364,8 @@ The default configuration is as follows:
 | spring.quartz.properties.org.quartz.jobStore.driverDelegateClass        | org.quartz.impl.jdbcjobstore.PostgreSQLDelegate |
 | spring.quartz.properties.org.quartz.jobStore.clusterCheckinInterval     | 5000                                            |
 
-The above configuration items is the same in *Master Server* and *Api Server*, but their *Quartz Scheduler* threadpool configuration is different.
+The above configuration items is the same in *Master Server* and *Api Server*, but their *Quartz Scheduler* threadpool
+configuration is different.
 
 The default quartz threadpool configuration in *Master Server* is as follows:
 
@@ -363,7 +376,8 @@ The default quartz threadpool configuration in *Master Server* is as follows:
 | spring.quartz.properties.org.quartz.threadPool.threadPriority     | 5                                 |
 | spring.quartz.properties.org.quartz.threadPool.class              | org.quartz.simpl.SimpleThreadPool |
 
-Since *Api Server* will not start *Quartz Scheduler* instance, as a client only, therefore it's threadpool is configured as `QuartzZeroSizeThreadPool` which has zero thread;
+Since *Api Server* will not start *Quartz Scheduler* instance, as a client only, therefore it's threadpool is configured
+as `QuartzZeroSizeThreadPool` which has zero thread;
 The default configuration is as follows:
 
 |                      Parameters                      |                             Default value                             |
@@ -372,7 +386,8 @@ The default configuration is as follows:
 
 ### dolphinscheduler_env.sh [load environment variables configs]
 
-When using shell to commit tasks, DolphinScheduler will export environment variables from `bin/env/dolphinscheduler_env.sh`. The
+When using shell to commit tasks, DolphinScheduler will export environment variables
+from `bin/env/dolphinscheduler_env.sh`. The
 mainly configuration including `JAVA_HOME` and other environment paths.
 
 ```bash
@@ -400,9 +415,10 @@ export FLINK_ENV_JAVA_OPTS="-javaagent:${DOLPHINSCHEDULER_HOME}/tools/libs/aspec
 
 ### Log related configuration
 
-|Service| Configuration file  |
-|--|--|
-|Master Server | `master-server/conf/logback-spring.xml`|
-|Api Server| `api-server/conf/logback-spring.xml`|
-|Worker Server| `worker-server/conf/logback-spring.xml`|
-|Alert Server| `alert-server/conf/logback-spring.xml`|
+|    Service    |           Configuration file            |
+|---------------|-----------------------------------------|
+| Master Server | `master-server/conf/logback-spring.xml` |
+| Api Server    | `api-server/conf/logback-spring.xml`    |
+| Worker Server | `worker-server/conf/logback-spring.xml` |
+| Alert Server  | `alert-server/conf/logback-spring.xml`  |
+

docs/docs/en/contribute/development-environment-setup.md

@@ -153,7 +153,7 @@ The browser access address [http://localhost:5173](http://localhost:5173) can lo
 
 #### zookeeper
 
-Download [ZooKeeper](https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.6.3), and extract it.
+Download [ZooKeeper](https://zookeeper.apache.org/releases.html), and extract it.
 
 - Create directory `zkData` and `zkLog`
 - Go to the zookeeper installation directory, copy configure file `zoo_sample.cfg` to `conf/zoo.cfg`, and change value of dataDir in conf/zoo.cfg to dataDir=./tmp/zookeeper
@@ -209,7 +209,6 @@ spring:
   <root level="INFO">
   +  <appender-ref ref="STDOUT"/>
     <appender-ref ref="APILOGFILE"/>
-    <appender-ref ref="SKYWALKING-LOG"/>
   </root>
   ```
 

docs/docs/en/contribute/frontend-development.md

@@ -33,7 +33,7 @@ Use the command line mode `cd`  enter the `dolphinscheduler-ui` project director
 > If `npm install` is very slow, you can set the taobao mirror
 
 ```
-npm config set registry http://registry.npm.taobao.org/
+npm config set registry http://registry.npmmirror.com/
 ```
 
 - Modify `API_BASE` in the file `dolphinscheduler-ui/.env` to interact with the backend:

docs/docs/en/faq.md

@@ -459,11 +459,11 @@ A: 1, cd dolphinscheduler-ui and delete node_modules directory
 sudo rm -rf node_modules
 ```
 
-​ 2, install node-sass through npm.taobao.org
+​ 2, install node-sass through npmmirror.com
 
 ```
 sudo npm uninstall node-sass
-sudo npm i node-sass --sass_binary_site=https://npm.taobao.org/mirrors/node-sass/
+sudo npm i node-sass --sass_binary_site=https://npmmirror.com/mirrors/node-sass/
 ```
 
 3, if the 2nd step failure, please, [referer url](https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/frontend-development.md)

docs/docs/en/guide/alert/alert_plugin_user_guide.md

@@ -13,6 +13,8 @@ Steps to be used are as follows:
 - Select the corresponding alarm plug-in and fill in the relevant alarm parameters.
 - Select `Alarm Group Management`, create an alarm group, and choose the corresponding alarm instance.
 
+> You can click `Test Send` button to test whether the alarm instance is configured correctly.
+
 ![alert-instance01](../../../../img/new_ui/dev/alert/alert_instance01.png)
 
 ![alert-instance02](../../../../img/new_ui/dev/alert/alert_instance02.png)

docs/docs/en/guide/alert/aliyun-voice.md

@@ -0,0 +1,32 @@
+# Alibaba Cloud - Voice alarm
+
+If you need to use `Voice` for alerting, create an alert instance in the alert instance management and select the `Voice` plugin.
+
+## 参数配置
+
+* Called number
+
+  > Called display number
+
+* Call display number
+
+  > The number for receiving voice notifications
+
+* VoiceCode
+
+  > Voice ID of the VoiceCode
+
+* accessKeyId
+
+  > Your AccessKey ID
+
+* accessKeySecret
+
+  > Your AccessKey Secret
+
+### example
+
+Voice notification Sends a voice notification of the file type to a specified number。
+The following shows the `voice` configuration example:
+
+![enterprise-wechat-app-msg-config](../../../../img/alert/aliyun-vocie-form-example.png)

docs/docs/en/guide/alert/script.md

@@ -7,11 +7,11 @@ The following shows the `Script` configuration example:
 
 ## Parameter Configuration
 
-| **Parameter** |                 **Description**                  |
-|---------------|--------------------------------------------------|
-| User Params   | User defined parameters will pass to the script. |
-| Script Path   | The file location path in the server.            |
-| Type          | Support `Shell` script.                          |
+| **Parameter** |                       **Description**                       |
+|---------------|-------------------------------------------------------------|
+| User Params   | User defined parameters will pass to the script.            |
+| Script Path   | The file location path in the server, only support .sh file |
+| Type          | Support `Shell` script.                                     |
 
 ### Note
 

docs/docs/en/guide/data-quality.md

@@ -12,15 +12,7 @@ The execution logic of the data quality task is as follows:
 - The current data quality task result is stored in the `t_ds_dq_execute_result` table of `dolphinscheduler`
   `Worker` sends the task result to `Master`, after `Master` receives `TaskResponse`, it will judge whether the task type is `DataQualityTask`, if so, it will read the corresponding result from `t_ds_dq_execute_result` according to `taskInstanceId`, and then The result is judged according to the check mode, operator and threshold configured by the user.
 - If the result is a failure, the corresponding operation, alarm or interruption will be performed according to the failure policy configured by the user.
-- Add config : `<server-name>/conf/common.properties`
-
-```properties
-# Change to specific version if you not use dev branch
-data-quality.jar.name=dolphinscheduler-data-quality-dev-SNAPSHOT.jar
-```
-
-- Please fill in `data-quality.jar.name` according to the actual package name.
-- If you package `data-quality` separately, remember to modify the package name to be consistent with `data-quality.jar.name`.
+- If you package `data-quality` separately, remember to modify the package name to be consistent with `data-quality.jar.dir` in `common.properties` with attribute name `data-quality.jar.dir`
 - If the old version is upgraded and used, you need to execute the `sql` update script to initialize the database before running.
 - `dolphinscheduler-data-quality-dev-SNAPSHOT.jar` was built with no dependencies. If a `JDBC` driver is required, you can set the `-jars` parameter in the `node settings` `Option Parameters`, e.g. `--jars /lib/jars/mysql-connector-java-8.0.16.jar`.
 - Currently only `MySQL`, `PostgreSQL` and `HIVE` data sources have been tested, other data sources have not been tested yet.

docs/docs/en/guide/expansion-reduction.md

@@ -30,9 +30,9 @@ Attention: DolphinScheduler itself does not depend on Hadoop, Hive, Spark, but w
 mkdir -p /opt
 cd /opt
 # decompress
-tar -zxvf apache-dolphinscheduler-3.2.0-bin.tar.gz -C /opt 
+tar -zxvf apache-dolphinscheduler-<version>-bin.tar.gz -C /opt 
 cd /opt
-mv apache-dolphinscheduler-3.2.0-bin  dolphinscheduler
+mv apache-dolphinscheduler-<version>-bin  dolphinscheduler
 ```
 
 ```markdown

docs/docs/en/guide/installation/cluster.md

@@ -14,7 +14,7 @@ Configure all the configurations refer to [pseudo-cluster deployment](pseudo-clu
 
 ### Modify Configuration
 
-This step differs quite a lot from [pseudo-cluster deployment](pseudo-cluster.md), because the deployment script transfers the required resources for installation to each deployment machine by using `scp`. So we only need to modify the configuration of the machine that runs `install.sh` script and configurations will dispatch to cluster by `scp`. The configuration file is under the path `bin/env/install_env.sh`, here we only need to modify section **INSTALL MACHINE**, **DolphinScheduler ENV, Database, Registry Server** and keep other sections the same as [pseudo-cluster deployment](pseudo-cluster .md), the following describes the parameters that must be modified:
+This step differs quite a lot from [pseudo-cluster deployment](pseudo-cluster.md), because the deployment script transfers the required resources for installation to each deployment machine by using `scp`. So we only need to modify the configuration of the machine that runs `install.sh` script and configurations will dispatch to cluster by `scp`. The configuration file is under the path `bin/env/install_env.sh`, here we only need to modify section **INSTALL MACHINE**, **DolphinScheduler ENV, Database, Registry Server** and keep other sections the same as [pseudo-cluster deployment](pseudo-cluster.md), the following describes the parameters that must be modified:
 
 ```shell
 # ---------------------------------------------------------

docs/docs/en/guide/installation/kubernetes.md

@@ -4,6 +4,8 @@ Kubernetes deployment is DolphinScheduler deployment in a Kubernetes cluster, wh
 
 If you are a new hand and want to experience DolphinScheduler functions, we recommend you install follow [Standalone deployment](standalone.md). If you want to experience more complete functions and schedule massive tasks, we recommend you install follow [pseudo-cluster deployment](pseudo-cluster.md). If you want to deploy DolphinScheduler in production, we recommend you follow [cluster deployment](cluster.md) or [Kubernetes deployment](kubernetes.md).
 
+> **Tip**: You can also try [DolphinScheduler K8S Operator](https://github.com/apache/dolphinscheduler-operator)，which is current on alpha1 stage
+
 ## Prerequisites
 
 - [Helm](https://helm.sh/) version 3.1.0+
@@ -12,16 +14,15 @@ If you are a new hand and want to experience DolphinScheduler functions, we reco
 
 ## Install DolphinScheduler
 
-Please download the source code package `apache-dolphinscheduler-3.2.0-src.tar.gz`, download address: [download address](https://dolphinscheduler.apache.org/en-us/download)
-
-To publish the release name `dolphinscheduler` version, please execute the following commands:
-
-```
-$ tar -zxvf apache-dolphinscheduler-3.2.0-src.tar.gz
-$ cd apache-dolphinscheduler-3.2.0-src/deploy/kubernetes/dolphinscheduler
-$ helm repo add bitnami https://charts.bitnami.com/bitnami
-$ helm dependency update .
-$ helm install dolphinscheduler . --set image.tag=3.2.0
+```bash
+# Choose the corresponding version yourself
+export VERSION=3.2.1
+helm pull oci://registry-1.docker.io/apache/dolphinscheduler-helm --version ${VERSION}
+tar -xvf dolphinscheduler-helm-${VERSION}.tgz
+cd dolphinscheduler-helm
+helm repo add bitnami https://charts.bitnami.com/bitnami
+helm dependency update .
+helm install dolphinscheduler .
 ```
 
 To publish the release name `dolphinscheduler` version to `test` namespace:
@@ -232,9 +233,9 @@ kubectl scale --replicas=6 sts dolphinscheduler-worker -n test # with test names
 2. Create a new `Dockerfile` to add MySQL driver:
 
 ```
-FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:3.2.0
+FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:<version>
 # For example
-# FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-tools:3.2.0
+# FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-tools:<version>
 
 # Attention Please, If the build is dolphinscheduler-tools image
 # You need to change the following line to: COPY mysql-connector-java-8.0.16.jar /opt/dolphinscheduler/tools/libs
@@ -283,9 +284,9 @@ externalDatabase:
 2. Create a new `Dockerfile` to add MySQL or Oracle driver:
 
 ```
-FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:3.2.0
+FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:<version>
 # For example
-# FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:3.2.0
+# FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:<version>
 
 # If you want to support MySQL Datasource
 COPY mysql-connector-java-8.0.16.jar /opt/dolphinscheduler/libs
@@ -315,7 +316,7 @@ docker build -t apache/dolphinscheduler-<service>:new-driver .
 1. Create a new `Dockerfile` to install pip:
 
 ```
-FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:3.2.0
+FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:<version>
 COPY requirements.txt /tmp
 RUN apt-get update && \
     apt-get install -y --no-install-recommends python-pip && \
@@ -350,7 +351,7 @@ docker build -t apache/dolphinscheduler-worker:pip .
 1. Create a new `Dockerfile` to install Python 3:
 
 ```
-FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:3.2.0
+FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:<version>
 RUN apt-get update && \
     apt-get install -y --no-install-recommends python3 && \
     rm -rf /var/lib/apt/lists/*
@@ -501,19 +502,6 @@ Modify `BUCKET_NAME`, `MINIO_IP`, `MINIO_ACCESS_KEY` and `MINIO_SECRET_KEY` to a
 
 > **Note**: `MINIO_IP` can only use IP instead of the domain name, because DolphinScheduler currently doesn't support S3 path style access.
 
-### How to Configure SkyWalking?
-
-Modify SkyWalking configurations in `values.yaml`:
-
-```yaml
-common:
-  configmap:
-    SKYWALKING_ENABLE: "true"
-    SW_AGENT_COLLECTOR_BACKEND_SERVICES: "127.0.0.1:11800"
-    SW_GRPC_LOG_SERVER_HOST: "127.0.0.1"
-    SW_GRPC_LOG_SERVER_PORT: "11800"
-```
-
 ### How to deploy specific components separately?
 
 Modify the `api.enabled`, `alert.enabled`, `master.enabled`, or `worker.enabled` configuration items in the `values.yaml` file.
@@ -537,234 +525,8 @@ helm install dolphinscheduler-gpu-worker . \
      --set externalRegistry.registryPluginName=zookeeper --set externalRegistry.registryServers=dolphinscheduler-zookeeper:2181
 ```
 
-Please note that the above steps are for reference only, and specific operations need to be adjusted according to the actual situation.
+> **Note**: the above steps are for reference only, and specific operations need to be adjusted according to the actual situation.
 
 ## Appendix-Configuration
 
-|                              Parameter                               |                                                          Description                                                          |                Default                |
-|----------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|---------------------------------------|
-| `timezone`                                                           | World time and date for cities in all time zones                                                                              | `Asia/Shanghai`                       |
-|                                                                      |                                                                                                                               |                                       |
-| `image.repository`                                                   | Docker image repository for the DolphinScheduler                                                                              | `apache/dolphinscheduler`             |
-| `image.tag`                                                          | Docker image version for the DolphinScheduler                                                                                 | `latest`                              |
-| `image.pullPolicy`                                                   | Image pull policy. Options: Always, Never, IfNotPresent                                                                       | `IfNotPresent`                        |
-| `image.pullSecret`                                                   | Image pull secret. An optional reference to secret in the same namespace to use for pulling any of the images                 | `nil`                                 |
-|                                                                      |                                                                                                                               |                                       |
-| `postgresql.enabled`                                                 | If not exists external PostgreSQL, by default, the DolphinScheduler will use a internal PostgreSQL                            | `true`                                |
-| `postgresql.postgresqlUsername`                                      | The username for internal PostgreSQL                                                                                          | `root`                                |
-| `postgresql.postgresqlPassword`                                      | The password for internal PostgreSQL                                                                                          | `root`                                |
-| `postgresql.postgresqlDatabase`                                      | The database for internal PostgreSQL                                                                                          | `dolphinscheduler`                    |
-| `postgresql.persistence.enabled`                                     | Set `postgresql.persistence.enabled` to `true` to mount a new volume for internal PostgreSQL                                  | `false`                               |
-| `postgresql.persistence.size`                                        | `PersistentVolumeClaim` size                                                                                                  | `20Gi`                                |
-| `postgresql.persistence.storageClass`                                | PostgreSQL data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning     | `-`                                   |
-| `minio.enabled`                                                      | Deploy minio and configure it as the default storage for DolphinScheduler, note this is for demo only, not for production.    | `false`                               |
-| `externalDatabase.type`                                              | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database type will use it      | `postgresql`                          |
-| `externalDatabase.driver`                                            | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database driver will use it    | `org.postgresql.Driver`               |
-| `externalDatabase.host`                                              | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database host will use it      | `localhost`                           |
-| `externalDatabase.port`                                              | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database port will use it      | `5432`                                |
-| `externalDatabase.username`                                          | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database username will use it  | `root`                                |
-| `externalDatabase.password`                                          | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database password will use it  | `root`                                |
-| `externalDatabase.database`                                          | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database database will use it  | `dolphinscheduler`                    |
-| `externalDatabase.params`                                            | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database params will use it    | `characterEncoding=utf8`              |
-|                                                                      |                                                                                                                               |                                       |
-| `zookeeper.enabled`                                                  | If not exists external ZooKeeper, by default, the DolphinScheduler will use a internal ZooKeeper                              | `true`                                |
-| `zookeeper.service.port`                                             | The port of zookeeper                                                                                                         | `2181`                                |
-| `zookeeper.fourlwCommandsWhitelist`                                  | A list of comma separated Four Letter Words commands to use                                                                   | `srvr,ruok,wchs,cons`                 |
-| `zookeeper.persistence.enabled`                                      | Set `zookeeper.persistence.enabled` to `true` to mount a new volume for internal ZooKeeper                                    | `false`                               |
-| `zookeeper.persistence.size`                                         | `PersistentVolumeClaim` size                                                                                                  | `20Gi`                                |
-| `zookeeper.persistence.storageClass`                                 | ZooKeeper data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning      | `-`                                   |
-| `externalRegistry.registryPluginName`                                | If exists external registry and set `zookeeper.enable` to `false`, specify the external registry plugin name                  | `zookeeper`                           |
-| `externalRegistry.registryServers`                                   | If exists external registry and set `zookeeper.enable` to `false`, specify the external registry servers                      | `127.0.0.1:2181`                      |
-|                                                                      |                                                                                                                               |                                       |
-| `common.configmap.DOLPHINSCHEDULER_OPTS`                             | The jvm options for dolphinscheduler, suitable for all servers                                                                | `""`                                  |
-| `common.configmap.DATA_BASEDIR_PATH`                                 | User data directory path, self configuration, please make sure the directory exists and have read write permissions           | `/tmp/dolphinscheduler`               |
-| `common.configmap.RESOURCE_STORAGE_TYPE`                             | Resource storage type: HDFS, S3, OSS, GCS, ABS, NONE                                                                          | `HDFS`                                |
-| `common.configmap.RESOURCE_UPLOAD_PATH`                              | Resource store on HDFS/S3 path, please make sure the directory exists on hdfs and have read write permissions                 | `/dolphinscheduler`                   |
-| `common.configmap.FS_DEFAULT_FS`                                     | Resource storage file system like `file:///`, `hdfs://mycluster:8020` or `s3a://dolphinscheduler`                             | `file:///`                            |
-| `common.configmap.FS_S3A_ENDPOINT`                                   | S3 endpoint when `common.configmap.RESOURCE_STORAGE_TYPE` is set to `S3`                                                      | `s3.xxx.amazonaws.com`                |
-| `common.configmap.FS_S3A_ACCESS_KEY`                                 | S3 access key when `common.configmap.RESOURCE_STORAGE_TYPE` is set to `S3`                                                    | `xxxxxxx`                             |
-| `common.configmap.FS_S3A_SECRET_KEY`                                 | S3 secret key when `common.configmap.RESOURCE_STORAGE_TYPE` is set to `S3`                                                    | `xxxxxxx`                             |
-| `common.configmap.HADOOP_SECURITY_AUTHENTICATION_STARTUP_STATE`      | Whether to startup kerberos                                                                                                   | `false`                               |
-| `common.configmap.JAVA_SECURITY_KRB5_CONF_PATH`                      | The java.security.krb5.conf path                                                                                              | `/opt/krb5.conf`                      |
-| `common.configmap.LOGIN_USER_KEYTAB_USERNAME`                        | The login user from keytab username                                                                                           | `hdfs@HADOOP.COM`                     |
-| `common.configmap.LOGIN_USER_KEYTAB_PATH`                            | The login user from keytab path                                                                                               | `/opt/hdfs.keytab`                    |
-| `common.configmap.KERBEROS_EXPIRE_TIME`                              | The kerberos expire time, the unit is hour                                                                                    | `2`                                   |
-| `common.configmap.HDFS_ROOT_USER`                                    | The HDFS root user who must have the permission to create directories under the HDFS root path                                | `hdfs`                                |
-| `common.configmap.RESOURCE_MANAGER_HTTPADDRESS_PORT`                 | Set resource manager httpaddress port for yarn                                                                                | `8088`                                |
-| `common.configmap.YARN_RESOURCEMANAGER_HA_RM_IDS`                    | If resourcemanager HA is enabled, please set the HA IPs                                                                       | `nil`                                 |
-| `common.configmap.YARN_APPLICATION_STATUS_ADDRESS`                   | If resourcemanager is single, you only need to replace ds1 to actual resourcemanager hostname, otherwise keep default         | `http://ds1:%s/ws/v1/cluster/apps/%s` |
-| `common.configmap.SKYWALKING_ENABLE`                                 | Set whether to enable skywalking                                                                                              | `false`                               |
-| `common.configmap.SW_AGENT_COLLECTOR_BACKEND_SERVICES`               | Set agent collector backend services for skywalking                                                                           | `127.0.0.1:11800`                     |
-| `common.configmap.SW_GRPC_LOG_SERVER_HOST`                           | Set grpc log server host for skywalking                                                                                       | `127.0.0.1`                           |
-| `common.configmap.SW_GRPC_LOG_SERVER_PORT`                           | Set grpc log server port for skywalking                                                                                       | `11800`                               |
-| `common.configmap.HADOOP_HOME`                                       | Set `HADOOP_HOME` for DolphinScheduler's task environment                                                                     | `/opt/soft/hadoop`                    |
-| `common.configmap.HADOOP_CONF_DIR`                                   | Set `HADOOP_CONF_DIR` for DolphinScheduler's task environment                                                                 | `/opt/soft/hadoop/etc/hadoop`         |
-| `common.configmap.SPARK_HOME`                                        | Set `SPARK_HOME` for DolphinScheduler's task environment                                                                      | `/opt/soft/spark`                     |
-| `common.configmap.PYTHON_LAUNCHER`                                   | Set `PYTHON_LAUNCHER` for DolphinScheduler's task environment                                                                 | `/usr/bin/python`                     |
-| `common.configmap.JAVA_HOME`                                         | Set `JAVA_HOME` for DolphinScheduler's task environment                                                                       | `/opt/java/openjdk`                   |
-| `common.configmap.HIVE_HOME`                                         | Set `HIVE_HOME` for DolphinScheduler's task environment                                                                       | `/opt/soft/hive`                      |
-| `common.configmap.FLINK_HOME`                                        | Set `FLINK_HOME` for DolphinScheduler's task environment                                                                      | `/opt/soft/flink`                     |
-| `common.configmap.DATAX_LAUNCHER`                                    | Set `DATAX_LAUNCHER` for DolphinScheduler's task environment                                                                  | `/opt/soft/datax`                     |
-| `common.sharedStoragePersistence.enabled`                            | Set `common.sharedStoragePersistence.enabled` to `true` to mount a shared storage volume for Hadoop, Spark binary and etc     | `false`                               |
-| `common.sharedStoragePersistence.mountPath`                          | The mount path for the shared storage volume                                                                                  | `/opt/soft`                           |
-| `common.sharedStoragePersistence.accessModes`                        | `PersistentVolumeClaim` access modes, must be `ReadWriteMany`                                                                 | `[ReadWriteMany]`                     |
-| `common.sharedStoragePersistence.storageClassName`                   | Shared Storage persistent volume storage class, must support the access mode: ReadWriteMany                                   | `-`                                   |
-| `common.sharedStoragePersistence.storage`                            | `PersistentVolumeClaim` size                                                                                                  | `20Gi`                                |
-| `common.fsFileResourcePersistence.enabled`                           | Set `common.fsFileResourcePersistence.enabled` to `true` to mount a new file resource volume for `api` and `worker`           | `false`                               |
-| `common.fsFileResourcePersistence.accessModes`                       | `PersistentVolumeClaim` access modes, must be `ReadWriteMany`                                                                 | `[ReadWriteMany]`                     |
-| `common.fsFileResourcePersistence.storageClassName`                  | Resource persistent volume storage class, must support the access mode: ReadWriteMany                                         | `-`                                   |
-| `common.fsFileResourcePersistence.storage`                           | `PersistentVolumeClaim` size                                                                                                  | `20Gi`                                |
-|                                                                      |                                                                                                                               |                                       |
-| `master.enabled`                                                     | Enable or disable the Master component                                                                                        | true                                  |
-| `master.podManagementPolicy`                                         | PodManagementPolicy controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down | `Parallel`                            |
-| `master.replicas`                                                    | Replicas is the desired number of replicas of the given Template                                                              | `3`                                   |
-| `master.annotations`                                                 | The `annotations` for master server                                                                                           | `{}`                                  |
-| `master.affinity`                                                    | If specified, the pod's scheduling constraints                                                                                | `{}`                                  |
-| `master.nodeSelector`                                                | NodeSelector is a selector which must be true for the pod to fit on a node                                                    | `{}`                                  |
-| `master.tolerations`                                                 | If specified, the pod's tolerations                                                                                           | `{}`                                  |
-| `master.resources`                                                   | The `resource` limit and request config for master server                                                                     | `{}`                                  |
-| `master.env.JAVA_OPTS`                                               | The jvm options for master server                                                                                             | `-Xms1g -Xmx1g -Xmn512m`              |
-| `master.env.MASTER_EXEC_THREADS`                                     | Master execute thread number to limit process instances                                                                       | `100`                                 |
-| `master.env.MASTER_EXEC_TASK_NUM`                                    | Master execute task number in parallel per process instance                                                                   | `20`                                  |
-| `master.env.MASTER_DISPATCH_TASK_NUM`                                | Master dispatch task number per batch                                                                                         | `3`                                   |
-| `master.env.MASTER_HOST_SELECTOR`                                    | Master host selector to select a suitable worker, optional values include Random, RoundRobin, LowerWeight                     | `LowerWeight`                         |
-| `master.env.MASTER_HEARTBEAT_INTERVAL`                               | Master heartbeat interval, the unit is second                                                                                 | `10s`                                 |
-| `master.env.MASTER_TASK_COMMIT_RETRYTIMES`                           | Master commit task retry times                                                                                                | `5`                                   |
-| `master.env.MASTER_TASK_COMMIT_INTERVAL`                             | master commit task interval, the unit is second                                                                               | `1s`                                  |
-| `master.env.MASTER_MAX_CPULOAD_AVG`                                  | Master max cpuload avg, only higher than the system cpu load average, master server can schedule                              | `-1` (`the number of cpu cores * 2`)  |
-| `master.env.MASTER_RESERVED_MEMORY`                                  | Master reserved memory, only lower than system available memory, master server can schedule, the unit is G                    | `0.3`                                 |
-| `master.livenessProbe.enabled`                                       | Turn on and off liveness probe                                                                                                | `true`                                |
-| `master.livenessProbe.initialDelaySeconds`                           | Delay before liveness probe is initiated                                                                                      | `30`                                  |
-| `master.livenessProbe.periodSeconds`                                 | How often to perform the probe                                                                                                | `30`                                  |
-| `master.livenessProbe.timeoutSeconds`                                | When the probe times out                                                                                                      | `5`                                   |
-| `master.livenessProbe.failureThreshold`                              | Minimum consecutive successes for the probe                                                                                   | `3`                                   |
-| `master.livenessProbe.successThreshold`                              | Minimum consecutive failures for the probe                                                                                    | `1`                                   |
-| `master.readinessProbe.enabled`                                      | Turn on and off readiness probe                                                                                               | `true`                                |
-| `master.readinessProbe.initialDelaySeconds`                          | Delay before readiness probe is initiated                                                                                     | `30`                                  |
-| `master.readinessProbe.periodSeconds`                                | How often to perform the probe                                                                                                | `30`                                  |
-| `master.readinessProbe.timeoutSeconds`                               | When the probe times out                                                                                                      | `5`                                   |
-| `master.readinessProbe.failureThreshold`                             | Minimum consecutive successes for the probe                                                                                   | `3`                                   |
-| `master.readinessProbe.successThreshold`                             | Minimum consecutive failures for the probe                                                                                    | `1`                                   |
-| `master.persistentVolumeClaim.enabled`                               | Set `master.persistentVolumeClaim.enabled` to `true` to mount a new volume for `master`                                       | `false`                               |
-| `master.persistentVolumeClaim.accessModes`                           | `PersistentVolumeClaim` access modes                                                                                          | `[ReadWriteOnce]`                     |
-| `master.persistentVolumeClaim.storageClassName`                      | `Master` logs data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning  | `-`                                   |
-| `master.persistentVolumeClaim.storage`                               | `PersistentVolumeClaim` size                                                                                                  | `20Gi`                                |
-|                                                                      |                                                                                                                               |                                       |
-| `worker.enabled`                                                     | Enable or disable the Worker component                                                                                        | true                                  |
-| `worker.podManagementPolicy`                                         | PodManagementPolicy controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down | `Parallel`                            |
-| `worker.replicas`                                                    | Replicas is the desired number of replicas of the given Template                                                              | `3`                                   |
-| `worker.annotations`                                                 | The `annotations` for worker server                                                                                           | `{}`                                  |
-| `worker.affinity`                                                    | If specified, the pod's scheduling constraints                                                                                | `{}`                                  |
-| `worker.nodeSelector`                                                | NodeSelector is a selector which must be true for the pod to fit on a node                                                    | `{}`                                  |
-| `worker.tolerations`                                                 | If specified, the pod's tolerations                                                                                           | `{}`                                  |
-| `worker.resources`                                                   | The `resource` limit and request config for worker server                                                                     | `{}`                                  |
-| `worker.env.WORKER_EXEC_THREADS`                                     | Worker execute thread number to limit task instances                                                                          | `100`                                 |
-| `worker.env.WORKER_HEARTBEAT_INTERVAL`                               | Worker heartbeat interval, the unit is second                                                                                 | `10s`                                 |
-| `worker.env.WORKER_MAX_CPU_LOAD_AVG`                                 | Worker max cpu load avg, only higher than the system cpu load average, worker server can be dispatched tasks                  | `-1` (`the number of cpu cores * 2`)  |
-| `worker.env.WORKER_RESERVED_MEMORY`                                  | Worker reserved memory, only lower than system available memory, worker server can be dispatched tasks, the unit is G         | `0.3`                                 |
-| `worker.env.HOST_WEIGHT`                                             | Worker host weight to dispatch tasks                                                                                          | `100`                                 |
-| `worker.livenessProbe.enabled`                                       | Turn on and off liveness probe                                                                                                | `true`                                |
-| `worker.livenessProbe.initialDelaySeconds`                           | Delay before liveness probe is initiated                                                                                      | `30`                                  |
-| `worker.livenessProbe.periodSeconds`                                 | How often to perform the probe                                                                                                | `30`                                  |
-| `worker.livenessProbe.timeoutSeconds`                                | When the probe times out                                                                                                      | `5`                                   |
-| `worker.livenessProbe.failureThreshold`                              | Minimum consecutive successes for the probe                                                                                   | `3`                                   |
-| `worker.livenessProbe.successThreshold`                              | Minimum consecutive failures for the probe                                                                                    | `1`                                   |
-| `worker.readinessProbe.enabled`                                      | Turn on and off readiness probe                                                                                               | `true`                                |
-| `worker.readinessProbe.initialDelaySeconds`                          | Delay before readiness probe is initiated                                                                                     | `30`                                  |
-| `worker.readinessProbe.periodSeconds`                                | How often to perform the probe                                                                                                | `30`                                  |
-| `worker.readinessProbe.timeoutSeconds`                               | When the probe times out                                                                                                      | `5`                                   |
-| `worker.readinessProbe.failureThreshold`                             | Minimum consecutive successes for the probe                                                                                   | `3`                                   |
-| `worker.readinessProbe.successThreshold`                             | Minimum consecutive failures for the probe                                                                                    | `1`                                   |
-| `worker.persistentVolumeClaim.enabled`                               | Set `worker.persistentVolumeClaim.enabled` to `true` to enable `persistentVolumeClaim` for `worker`                           | `false`                               |
-| `worker.persistentVolumeClaim.dataPersistentVolume.enabled`          | Set `worker.persistentVolumeClaim.dataPersistentVolume.enabled` to `true` to mount a data volume for `worker`                 | `false`                               |
-| `worker.persistentVolumeClaim.dataPersistentVolume.accessModes`      | `PersistentVolumeClaim` access modes                                                                                          | `[ReadWriteOnce]`                     |
-| `worker.persistentVolumeClaim.dataPersistentVolume.storageClassName` | `Worker` data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning       | `-`                                   |
-| `worker.persistentVolumeClaim.dataPersistentVolume.storage`          | `PersistentVolumeClaim` size                                                                                                  | `20Gi`                                |
-| `worker.persistentVolumeClaim.logsPersistentVolume.enabled`          | Set `worker.persistentVolumeClaim.logsPersistentVolume.enabled` to `true` to mount a logs volume for `worker`                 | `false`                               |
-| `worker.persistentVolumeClaim.logsPersistentVolume.accessModes`      | `PersistentVolumeClaim` access modes                                                                                          | `[ReadWriteOnce]`                     |
-| `worker.persistentVolumeClaim.logsPersistentVolume.storageClassName` | `Worker` logs data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning  | `-`                                   |
-| `worker.persistentVolumeClaim.logsPersistentVolume.storage`          | `PersistentVolumeClaim` size                                                                                                  | `20Gi`                                |
-|                                                                      |                                                                                                                               |                                       |
-| `alert.enabled`                                                      | Enable or disable the Alert-Server component                                                                                  | true                                  |
-| `alert.replicas`                                                     | Replicas is the desired number of replicas of the given Template                                                              | `1`                                   |
-| `alert.strategy.type`                                                | Type of deployment. Can be "Recreate" or "RollingUpdate"                                                                      | `RollingUpdate`                       |
-| `alert.strategy.rollingUpdate.maxSurge`                              | The maximum number of pods that can be scheduled above the desired number of pods                                             | `25%`                                 |
-| `alert.strategy.rollingUpdate.maxUnavailable`                        | The maximum number of pods that can be unavailable during the update                                                          | `25%`                                 |
-| `alert.annotations`                                                  | The `annotations` for alert server                                                                                            | `{}`                                  |
-| `alert.affinity`                                                     | If specified, the pod's scheduling constraints                                                                                | `{}`                                  |
-| `alert.nodeSelector`                                                 | NodeSelector is a selector which must be true for the pod to fit on a node                                                    | `{}`                                  |
-| `alert.tolerations`                                                  | If specified, the pod's tolerations                                                                                           | `{}`                                  |
-| `alert.resources`                                                    | The `resource` limit and request config for alert server                                                                      | `{}`                                  |
-| `alert.configmap.ALERT_SERVER_OPTS`                                  | The jvm options for alert server                                                                                              | `-Xms512m -Xmx512m -Xmn256m`          |
-| `alert.configmap.XLS_FILE_PATH`                                      | XLS file path                                                                                                                 | `/tmp/xls`                            |
-| `alert.configmap.MAIL_SERVER_HOST`                                   | Mail `SERVER HOST `                                                                                                           | `nil`                                 |
-| `alert.configmap.MAIL_SERVER_PORT`                                   | Mail `SERVER PORT`                                                                                                            | `nil`                                 |
-| `alert.configmap.MAIL_SENDER`                                        | Mail `SENDER`                                                                                                                 | `nil`                                 |
-| `alert.configmap.MAIL_USER`                                          | Mail `USER`                                                                                                                   | `nil`                                 |
-| `alert.configmap.MAIL_PASSWD`                                        | Mail `PASSWORD`                                                                                                               | `nil`                                 |
-| `alert.configmap.MAIL_SMTP_STARTTLS_ENABLE`                          | Mail `SMTP STARTTLS` enable                                                                                                   | `false`                               |
-| `alert.configmap.MAIL_SMTP_SSL_ENABLE`                               | Mail `SMTP SSL` enable                                                                                                        | `false`                               |
-| `alert.configmap.MAIL_SMTP_SSL_TRUST`                                | Mail `SMTP SSL TRUST`                                                                                                         | `nil`                                 |
-| `alert.configmap.ENTERPRISE_WECHAT_ENABLE`                           | `Enterprise Wechat` enable                                                                                                    | `false`                               |
-| `alert.configmap.ENTERPRISE_WECHAT_CORP_ID`                          | `Enterprise Wechat` corp id                                                                                                   | `nil`                                 |
-| `alert.configmap.ENTERPRISE_WECHAT_SECRET`                           | `Enterprise Wechat` secret                                                                                                    | `nil`                                 |
-| `alert.configmap.ENTERPRISE_WECHAT_AGENT_ID`                         | `Enterprise Wechat` agent id                                                                                                  | `nil`                                 |
-| `alert.configmap.ENTERPRISE_WECHAT_USERS`                            | `Enterprise Wechat` users                                                                                                     | `nil`                                 |
-| `alert.livenessProbe.enabled`                                        | Turn on and off liveness probe                                                                                                | `true`                                |
-| `alert.livenessProbe.initialDelaySeconds`                            | Delay before liveness probe is initiated                                                                                      | `30`                                  |
-| `alert.livenessProbe.periodSeconds`                                  | How often to perform the probe                                                                                                | `30`                                  |
-| `alert.livenessProbe.timeoutSeconds`                                 | When the probe times out                                                                                                      | `5`                                   |
-| `alert.livenessProbe.failureThreshold`                               | Minimum consecutive successes for the probe                                                                                   | `3`                                   |
-| `alert.livenessProbe.successThreshold`                               | Minimum consecutive failures for the probe                                                                                    | `1`                                   |
-| `alert.readinessProbe.enabled`                                       | Turn on and off readiness probe                                                                                               | `true`                                |
-| `alert.readinessProbe.initialDelaySeconds`                           | Delay before readiness probe is initiated                                                                                     | `30`                                  |
-| `alert.readinessProbe.periodSeconds`                                 | How often to perform the probe                                                                                                | `30`                                  |
-| `alert.readinessProbe.timeoutSeconds`                                | When the probe times out                                                                                                      | `5`                                   |
-| `alert.readinessProbe.failureThreshold`                              | Minimum consecutive successes for the probe                                                                                   | `3`                                   |
-| `alert.readinessProbe.successThreshold`                              | Minimum consecutive failures for the probe                                                                                    | `1`                                   |
-| `alert.persistentVolumeClaim.enabled`                                | Set `alert.persistentVolumeClaim.enabled` to `true` to mount a new volume for `alert`                                         | `false`                               |
-| `alert.persistentVolumeClaim.accessModes`                            | `PersistentVolumeClaim` access modes                                                                                          | `[ReadWriteOnce]`                     |
-| `alert.persistentVolumeClaim.storageClassName`                       | `Alert` logs data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning   | `-`                                   |
-| `alert.persistentVolumeClaim.storage`                                | `PersistentVolumeClaim` size                                                                                                  | `20Gi`                                |
-|                                                                      |                                                                                                                               |                                       |
-| `api.enabled`                                                        | Enable or disable the API-Server component                                                                                    | true                                  |
-| `api.replicas`                                                       | Replicas is the desired number of replicas of the given Template                                                              | `1`                                   |
-| `api.strategy.type`                                                  | Type of deployment. Can be "Recreate" or "RollingUpdate"                                                                      | `RollingUpdate`                       |
-| `api.strategy.rollingUpdate.maxSurge`                                | The maximum number of pods that can be scheduled above the desired number of pods                                             | `25%`                                 |
-| `api.strategy.rollingUpdate.maxUnavailable`                          | The maximum number of pods that can be unavailable during the update                                                          | `25%`                                 |
-| `api.annotations`                                                    | The `annotations` for api server                                                                                              | `{}`                                  |
-| `api.affinity`                                                       | If specified, the pod's scheduling constraints                                                                                | `{}`                                  |
-| `api.nodeSelector`                                                   | NodeSelector is a selector which must be true for the pod to fit on a node                                                    | `{}`                                  |
-| `api.tolerations`                                                    | If specified, the pod's tolerations                                                                                           | `{}`                                  |
-| `api.resources`                                                      | The `resource` limit and request config for api server                                                                        | `{}`                                  |
-| `api.configmap.API_SERVER_OPTS`                                      | The jvm options for api server                                                                                                | `-Xms512m -Xmx512m -Xmn256m`          |
-| `api.livenessProbe.enabled`                                          | Turn on and off liveness probe                                                                                                | `true`                                |
-| `api.livenessProbe.initialDelaySeconds`                              | Delay before liveness probe is initiated                                                                                      | `30`                                  |
-| `api.livenessProbe.periodSeconds`                                    | How often to perform the probe                                                                                                | `30`                                  |
-| `api.livenessProbe.timeoutSeconds`                                   | When the probe times out                                                                                                      | `5`                                   |
-| `api.livenessProbe.failureThreshold`                                 | Minimum consecutive successes for the probe                                                                                   | `3`                                   |
-| `api.livenessProbe.successThreshold`                                 | Minimum consecutive failures for the probe                                                                                    | `1`                                   |
-| `api.readinessProbe.enabled`                                         | Turn on and off readiness probe                                                                                               | `true`                                |
-| `api.readinessProbe.initialDelaySeconds`                             | Delay before readiness probe is initiated                                                                                     | `30`                                  |
-| `api.readinessProbe.periodSeconds`                                   | How often to perform the probe                                                                                                | `30`                                  |
-| `api.readinessProbe.timeoutSeconds`                                  | When the probe times out                                                                                                      | `5`                                   |
-| `api.readinessProbe.failureThreshold`                                | Minimum consecutive successes for the probe                                                                                   | `3`                                   |
-| `api.readinessProbe.successThreshold`                                | Minimum consecutive failures for the probe                                                                                    | `1`                                   |
-| `api.persistentVolumeClaim.enabled`                                  | Set `api.persistentVolumeClaim.enabled` to `true` to mount a new volume for `api`                                             | `false`                               |
-| `api.persistentVolumeClaim.accessModes`                              | `PersistentVolumeClaim` access modes                                                                                          | `[ReadWriteOnce]`                     |
-| `api.persistentVolumeClaim.storageClassName`                         | `api` logs data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning     | `-`                                   |
-| `api.persistentVolumeClaim.storage`                                  | `PersistentVolumeClaim` size                                                                                                  | `20Gi`                                |
-| `api.service.type`                                                   | `type` determines how the Service is exposed. Valid options are ExternalName, ClusterIP, NodePort, and LoadBalancer           | `ClusterIP`                           |
-| `api.service.clusterIP`                                              | `clusterIP` is the IP address of the service and is usually assigned randomly by the master                                   | `nil`                                 |
-| `api.service.nodePort`                                               | `nodePort` is the port on each node on which this service is exposed when type=NodePort                                       | `nil`                                 |
-| `api.service.externalIPs`                                            | `externalIPs` is a list of IP addresses for which nodes in the cluster will also accept traffic for this service              | `[]`                                  |
-| `api.service.externalName`                                           | `externalName` is the external reference that kubedns or equivalent will return as a CNAME record for this service            | `nil`                                 |
-| `api.service.loadBalancerIP`                                         | `loadBalancerIP` when service.type is LoadBalancer. LoadBalancer will get created with the IP specified in this field         | `nil`                                 |
-| `api.service.annotations`                                            | `annotations` may need to be set when service.type is LoadBalancer                                                            | `{}`                                  |
-|                                                                      |                                                                                                                               |                                       |
-| `ingress.enabled`                                                    | Enable ingress                                                                                                                | `false`                               |
-| `ingress.host`                                                       | Ingress host                                                                                                                  | `dolphinscheduler.org`                |
-| `ingress.path`                                                       | Ingress path                                                                                                                  | `/dolphinscheduler`                   |
-| `ingress.tls.enabled`                                                | Enable ingress tls                                                                                                            | `false`                               |
-| `ingress.tls.secretName`                                             | Ingress tls secret name                                                                                                       | `dolphinscheduler-tls`                |
-
+Ref: [DolphinScheduler Helm Charts](https://github.com/apache/dolphinscheduler/blob/dev/deploy/kubernetes/dolphinscheduler/README.md)

docs/docs/en/guide/installation/pseudo-cluster.md

@@ -2,7 +2,7 @@
 
 The purpose of the pseudo-cluster deployment is to deploy the DolphinScheduler service on a single machine. In this mode, DolphinScheduler's master, worker, API server, are all on the same machine.
 
-If you are a new hand and want to experience DolphinScheduler functions, we recommend you install follow [Standalone deployment](standalone.md). If you want to experience more complete functions and schedule massive tasks, we recommend you install follow[pseudo-cluster deployment. If you want to deploy DolphinScheduler in production, we recommend you follow [cluster deployment](cluster.md) or [Kubernetes deployment](kubernetes.md).
+If you are a new hand and want to experience DolphinScheduler functions, we recommend you install follow [Standalone deployment](standalone.md). If you want to experience more complete functions and schedule massive tasks, we recommend you install follow [pseudo-cluster deployment](pseudo-cluster.md). If you want to deploy DolphinScheduler in production, we recommend you follow [cluster deployment](cluster.md) or [Kubernetes deployment](kubernetes.md).
 
 ## Preparation
 
@@ -123,7 +123,6 @@ export SPRING_DATASOURCE_PASSWORD={password}
 # DolphinScheduler server related configuration
 export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
 export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
-export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}
 
 # Registry center configuration, determines the type and link of the registry center
 export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}

docs/docs/en/guide/installation/standalone.md

@@ -12,7 +12,7 @@ If you want to deploy DolphinScheduler in production, we recommend you follow [c
 ## Preparation
 
 - JDK：download [JDK][jdk] (1.8+), install and configure environment variable `JAVA_HOME` and append `bin` dir (included in `JAVA_HOME`) to `PATH` variable. You can skip this step if it already exists in your environment.
-- Binary package: download the DolphinScheduler binary package at [download page](https://dolphinscheduler.apache.org/en-us/download/3.2.0).  <!-- markdown-link-check-disable-line -->
+- Binary package: download the DolphinScheduler binary package at [download page](https://dolphinscheduler.apache.org/en-us/download/<version>).  <!-- markdown-link-check-disable-line -->
 
 ## Start DolphinScheduler Standalone Server
 
@@ -36,7 +36,7 @@ Access address `http://localhost:12345/dolphinscheduler/ui` and login DolphinSch
 
 ### Start or Stop Server
 
-The script `./bin/dolphinscheduler-daemon.sh`can be used not only quickly start standalone, but also to stop the service operation. The following are all the commands:
+The script `./bin/dolphinscheduler-daemon.sh` can be used not only quickly start standalone, but also to stop the service operation. The following are all the commands:
 
 ```shell
 # Start Standalone Server

docs/docs/en/guide/metrics/metrics.md

@@ -91,6 +91,11 @@ For example, you can get the master metrics by `curl http://localhost:5679/actua
   - stop: the number of stopped workflow instances
   - failover: the number of workflow instance fail-overs
 
+### RPC Related Metrics
+
+- ds.rpc.client.sync.request.exception.count: (counter) the number of exceptions occurred in sync rpc requests
+- ds.rpc.client.sync.request.duration.time: (histogram) the time cost of sync rpc requests
+
 ### Master Server Metrics
 
 - ds.master.overload.count: (counter) the number of times the master overloaded

docs/docs/en/guide/monitor.md

@@ -16,6 +16,12 @@
 
 ![worker](../../../img/new_ui/dev/monitor/worker.png)
 
+### Alert Server
+
+- Mainly related to alert server information.
+
+![alert-server](../../../img/new_ui/dev/monitor/alert-server.png)
+
 ### Database
 
 - Mainly the health status of the DB.
@@ -26,18 +32,17 @@
 
 ### Statistics
 
-![statistics](../../../img/new_ui/dev/monitor/statistics.png)
+![Command Statistics List](../../../img/new_ui/dev/monitor/command-list.png)
 
-|             **Parameter**              |                  **Description**                   |
-|----------------------------------------|----------------------------------------------------|
-| Number of commands wait to be executed | Statistics of the `t_ds_command` table data.       |
-| The number of failed commands          | Statistics of the `t_ds_error_command` table data. |
-| Number of tasks wait to run            | Count the data of `task_queue` in the ZooKeeper.   |
-| Number of tasks wait to be killed      | Count the data of `task_kill` in the ZooKeeper.    |
+Shows the command list in the system. Data is from the `t_ds_command` table.
+
+![Failure Command Statistics List](../../../img/new_ui/dev/monitor/failure-command-list.png)
+
+Shows the failure command list in the system. Data is from the `t_ds_error_command` table.
 
 ### Audit Log
 
 The audit log provides information about who accesses the system and the operations made to the system and record related
 time, which strengthen the security of the system and maintenance.
 
-![audit-log](../../../img/new_ui/dev/monitor/audit-log.jpg)
+![audit-log](../../../img/new_ui/dev/monitor/audit-log.png)

docs/docs/en/guide/parameter/context.md

@@ -92,6 +92,8 @@ For example
 
 ![python_context_param](../../../../img/new_ui/dev/parameter/python_context_param.png)
 
+Attention: When the variable value contains the `\n` identifier, such as ` value = "hello \n world" `, value needs to be carried out in a special way. You need to use `print('${setValue(key=%s)}' % repr(value))`, otherwise the argument cannot be passed to the subsequent flow.
+
 #### Pass parameter from SubProcess task to downstream
 
 In the workflow definition of the subprocess, define `OUT` direction parameters as output parameters, and these parameters can be passed to the downstream tasks of the subprocess node.
@@ -120,7 +122,7 @@ Although the two parameters var1 and var2 are output in the A task, only the `OU
 
 #### Pass parameter from Kubernetes task to downstream
 
-Different programming languages may use different logging frameworks in Kubernetes tasks. To be compatible with these frameworks, DolphinScheduler provides a universal logging data format `${(key=value)dsVal}`. Users can output log data in the format `${(key=value)dsVal}` in the terminal logs of their applications, where `key` is the corresponding parameter prop and `value` is the value of that parameter. DolphinScheduler will capture the `${(key=value)dsVal}` in the output logs to capture the parameters and pass them downstream.
+Different programming languages may use different logging frameworks in Kubernetes tasks. To be compatible with these frameworks, DolphinScheduler provides a universal logging data format `${(key=value)}` or `#{(key=value)}`. Users can output log data in the format in the terminal logs of their applications, where `key` is the corresponding parameter prop and `value` is the value of that parameter. DolphinScheduler will capture the `${(key=value)}` or `#{(key=value)}` in the output logs to capture the parameters and pass them downstream.
 
 For example
 

docs/docs/en/guide/parameter/global.md

@@ -22,7 +22,7 @@ Create a shell task and enter `echo ${dt}` in the script content. In this case,
 
 ### Save the workflow and set global parameters
 
-Set global parameter: On the workflow definition page, click the plus sign to the right of "Set Global", after filling in the variable name and value, save it.
+Set global parameter: On the workflow definition page, click the plus sign to the right of "Set Global", after filling in the variable name and value, select the appropriate parameter value type, save it.
 
 ![global-parameter02](../../../../img/new_ui/dev/parameter/global_parameter02.png)
 

docs/docs/en/guide/parameter/project-parameter.md

@@ -8,7 +8,7 @@ Project-level parameters are valid for all task nodes under the entire project.
 
 ### Define project-level parameters
 
-On the project page, click Project Parameters and Create Parameters, and fill in the parameter name and parameter value. As shown below:
+On the project page, click Project Parameters and Create Parameters, and fill in the parameter name and parameter value, select the appropriate parameter value type. As shown below:
 
 ![project-parameter01](../../../../img/new_ui/dev/parameter/project_parameter01.png)
 

docs/docs/en/guide/parameter/startup-parameter.md

@@ -6,7 +6,7 @@ Parameters are valid for all task nodes of the entire workflow. It can be config
 
 ## Usage
 
-Usage of startup parameters is: at the task launch page, click the '+' below the 'Startup Parameter' and fill in the key and value to save. The workflow will add them into global parameters.
+Usage of startup parameters is: at the task launch page, click the '+' below the 'Startup Parameter' and fill in the key and value, select the appropriate parameter value type then to save. The workflow will add them into global parameters.
 
 ## Example
 

docs/docs/en/guide/project/workflow-definition.md

@@ -70,10 +70,12 @@ Click `Project Management -> Workflow -> Workflow Definition` to enter the workf
 ### Actions Supported by a Single Workflow
 
 - **Edit:** Only "Offline" workflow definitions can be edited. Workflow DAG editing is the same as [Create Workflow Definition](#create-workflow-definition)
+- **Run:** Only workflow in the online state can run. See [Run Workflow](#run-the-workflow) for the operation steps.
 - **Online:** When the workflow status is "Offline", used to make workflow online. Only the workflow in the "Online" state can run, but cannot edit.
 - **Offline:** When the workflow status is "Online", used to make workflow offline. Only the workflow in the "Offline" state can be edited, but cannot run.
-- **Run:** Only workflow in the online state can run. See [Run Workflow](#run-the-workflow) for the operation steps.
-- **Timing:** Timing can only set to online workflows, and the system automatically schedules to run the workflow on time. The status after creating a timing setting is "offline", and the timing must set online on the timing management page to make effect. See [Workflow Timing](#workflow-schedule) for timing operation steps.
+- **Timing:** Timing can only set to online workflows, and the system automatically schedules to run the workflow on time. The status after creating a timing setting is "offline", You need to click the <img src="../../../../img/scheduler_online.png" width="20"/>button on the right to go online to take effect. See [Workflow Timing](#workflow-schedule) for timing operation steps.
+- **Timing Online：** Scheduled online, schedule can only be clicked after the schedule is created and the workflow status is "online".
+- **Timing Offline：** Scheduled offline.
 - **Timing Management:** The timing management page can edit, online or offline and delete timing.
 - **Delete:** Delete the workflow definition. In the same project, only the workflow definition created by yourself can be deleted, and the workflow definition of other users cannot be deleted. If you need to delete it, please contact the user who created it or the administrator.
 - **Download:** Download workflow definition to local.
@@ -179,10 +181,13 @@ Description of workflow operating parameters:
 
 - Click the "Create" button to create the timing. Now the timing status is "**Offline**" and the timing needs to be **Online** to make effect.
 
-- Schedule online: Click the `Timing Management` button <img src="../../../../img/timeManagement.png" width="35"/>, enter the timing management page, click the `online` button, the timing status will change to `online`, as shown in the below figure, the workflow makes effect regularly.
+- Schedule online: Click the `Timing online` button <img src="../../../../img/scheduler_online.png" width="30"/>, click the `online` button, the timing status will change to `online`, as shown in the below figure, the workflow makes effect regularly.
 
   ![workflow-time03](../../../../img/new_ui/dev/project/workflow-time03.png)
 
+- View the Schedule Timing：After the scheduled time goes online, you can check the current timing situation through the "Timing" button<img src="../../../../img/timing.png" width="35"/>. But it cannot be modified, as shown in the below figure.
+  ![workflow-time04](../../../../img/new_ui/dev/project/workflow-time04.png)
+
 ## Import Workflow
 
 Click `Project Management -> Workflow -> Workflow Definition` to enter the workflow definition page, click the `Import Workflow` button to import the local workflow file, the workflow definition list displays the imported workflow and the status is offline.

docs/docs/en/guide/remote-logging.md

@@ -10,7 +10,7 @@ If you deploy DolphinScheduler in `Standalone` mode, you only need to configure
 ```properties
 # Whether to enable remote logging
 remote.logging.enable=false
-# if remote.logging.enable = true, set the target of remote logging
+# if remote.logging.enable = true, set the target of remote logging, currently support OSS, S3, GCS, ABS
 remote.logging.target=OSS
 # if remote.logging.enable = true, set the log base directory
 remote.logging.base.dir=logs
@@ -66,12 +66,12 @@ remote.logging.google.cloud.storage.bucket.name=<your-bucket>
 Configure `common.properties` as follows:
 
 ```properties
-# abs container name, required if you set resource.storage.type=ABS
-resource.azure.blob.storage.container.name=<your-container>
 # abs account name, required if you set resource.storage.type=ABS
-resource.azure.blob.storage.account.name=<your-account-name>
-# abs connection string, required if you set resource.storage.type=ABS
-resource.azure.blob.storage.connection.string=<your-connection-string>
+remote.logging.abs.account.name=<your-account-name>
+# abs account key, required if you set resource.storage.type=ABS
+remote.logging.abs.account.key=<your-account-key>
+# abs container name, required if you set resource.storage.type=ABS
+remote.logging.abs.container.name=<your-container-name>
 ```
 
 ### Notice

docs/docs/en/guide/resource/configuration.md

@@ -2,8 +2,8 @@
 
 - You could use `Resource Center` to upload text files, UDFs and other task-related files.
 - You could configure `Resource Center` to use distributed file system like [Hadoop](https://hadoop.apache.org/docs/r2.7.0/) (2.6+), [MinIO](https://github.com/minio/minio) cluster or remote storage products like [AWS S3](https://aws.amazon.com/s3/), [Alibaba Cloud OSS](https://www.aliyun.com/product/oss), [Huawei Cloud OBS](https://support.huaweicloud.com/obs/index.html) etc.
-- You could configure `Resource Center` to use local file system. If you deploy `DolphinScheduler` in `Standalone` mode, you could configure it to use local file system for `Resouce Center` without the need of an external `HDFS` system or `S3`.
-- Furthermore, if you deploy `DolphinScheduler` in `Cluster` mode, you could use [S3FS-FUSE](https://github.com/s3fs-fuse/s3fs-fuse) to mount `S3` or [JINDO-FUSE](https://help.aliyun.com/document_detail/187410.html) to mount `OSS` to your machines and use the local file system for `Resouce Center`. In this way, you could operate remote files as if on your local machines.
+- You could configure `Resource Center` to use local file system. If you deploy `DolphinScheduler` in `Standalone` mode, you could configure it to use local file system for `Resource Center` without the need of an external `HDFS` system or `S3`.
+- Furthermore, if you deploy `DolphinScheduler` in `Cluster` mode, you could use [S3FS-FUSE](https://github.com/s3fs-fuse/s3fs-fuse) to mount `S3` or [JINDO-FUSE](https://help.aliyun.com/document_detail/187410.html) to mount `OSS` to your machines and use the local file system for `Resource Center`. In this way, you could operate remote files as if on your local machines.
 
 ## Use Local File System
 
@@ -28,74 +28,37 @@ The configuration you may need to change:
 
 ## connect AWS S3
 
-if you want to upload resources to `Resource Center` connected to `S3`, you need to configure `api-server/conf/common.properties` and `worker-server/conf/common.properties`. You can refer to the following:
+if you want to upload resources to `Resource Center` connected to `S3`, you need to configure `api-server/conf/common.properties`, `api-server/conf/aws.yaml` and `worker-server/conf/common.properties`, `worker-server/conf/aws.yaml`. You can refer to the following:
 
 config the following fields
 
 ```properties
-......
 
 resource.storage.type=S3
-
-......
-
-resource.aws.access.key.id=aws_access_key_id
-# The AWS secret access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required
-resource.aws.secret.access.key=aws_secret_access_key
-# The AWS Region to use. if resource.storage.type=S3 or use EMR-Task, This configuration is required
-resource.aws.region=us-west-2
-# The name of the bucket. You need to create them by yourself. Otherwise, the system cannot start. All buckets in Amazon S3 share a single namespace; ensure the bucket is given a unique name.
-resource.aws.s3.bucket.name=dolphinscheduler
-# You need to set this parameter when private cloud s4. If S3 uses public cloud, you only need to set resource.aws.region or set to the endpoint of a public cloud such as S3.cn-north-1.amazonaws.com.cn
-resource.aws.s3.endpoint=
-
-......
 ```
 
-## Use HDFS or Remote Object Storage
+```yaml
+aws:
+    s3:
+        # The AWS credentials provider type. support: AWSStaticCredentialsProvider, InstanceProfileCredentialsProvider
+        # AWSStaticCredentialsProvider: use the access key and secret key to authenticate
+        # InstanceProfileCredentialsProvider: use the IAM role to authenticate
+        credentials.provider.type: AWSStaticCredentialsProvider
+        access.key.id: <access.key.id>
+        access.key.secret: <access.key.secret>
+        region: <region>
+        bucket.name: <bucket.name>
+        endpoint: <endpoint>
 
-After version 3.0.0-alpha, if you want to upload resources to `Resource Center` connected to `HDFS`, you need to configure `api-server/conf/common.properties` and `worker-server/conf/common.properties`.
+```
+
+## connect OSS S3
+
+if you want to upload resources to `Resource Center` connected to `OSS`, you need to configure `api-server/conf/common.properties` and `worker-server/conf/common.properties`. You can refer to the following:
+
+config the following fields
 
 ```properties
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# user data local directory path, please make sure the directory exists and have read write permissions
-data.basedir.path=/tmp/dolphinscheduler
-
-# resource view suffixs
-#resource.view.suffixs=txt,log,sh,bat,conf,cfg,py,java,sql,xml,hql,properties,json,yml,yaml,ini,js
-
-# resource storage type: LOCAL, HDFS, S3, OSS, GCS, ABS, OBS
-resource.storage.type=LOCAL
-# resource store on HDFS/S3/OSS path, resource file will store to this base path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended
-resource.storage.upload.base.path=/tmp/dolphinscheduler
-
-# The AWS access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required
-resource.aws.access.key.id=minioadmin
-# The AWS secret access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required
-resource.aws.secret.access.key=minioadmin
-# The AWS Region to use. if resource.storage.type=S3 or use EMR-Task, This configuration is required
-resource.aws.region=cn-north-1
-# The name of the bucket. You need to create them by yourself. Otherwise, the system cannot start. All buckets in Amazon S3 share a single namespace; ensure the bucket is given a unique name.
-resource.aws.s3.bucket.name=dolphinscheduler
-# You need to set this parameter when private cloud s3. If S3 uses public cloud, you only need to set resource.aws.region or set to the endpoint of a public cloud such as S3.cn-north-1.amazonaws.com.cn
-resource.aws.s3.endpoint=http://localhost:9000
-
 # alibaba cloud access key id, required if you set resource.storage.type=OSS 
 resource.alibaba.cloud.access.key.id=<your-access-key-id>
 # alibaba cloud access key secret, required if you set resource.storage.type=OSS
@@ -107,87 +70,24 @@ resource.alibaba.cloud.oss.bucket.name=dolphinscheduler
 # oss bucket endpoint, required if you set resource.storage.type=OSS
 resource.alibaba.cloud.oss.endpoint=https://oss-cn-hangzhou.aliyuncs.com
 
-# alibaba cloud access key id, required if you set resource.storage.type=OBS
+```
+
+## connect OBS S3
+
+if you want to upload resources to `Resource Center` connected to `OBS`, you need to configure `api-server/conf/common.properties` and `worker-server/conf/common.properties`. You can refer to the following:
+
+config the following fields
+
+```properties
+# access key id, required if you set resource.storage.type=OBS
 resource.huawei.cloud.access.key.id=<your-access-key-id>
-# alibaba cloud access key secret, required if you set resource.storage.type=OBS
+# access key secret, required if you set resource.storage.type=OBS
 resource.huawei.cloud.access.key.secret=<your-access-key-secret>
 # oss bucket name, required if you set resource.storage.type=OBS
 resource.huawei.cloud.obs.bucket.name=dolphinscheduler
 # oss bucket endpoint, required if you set resource.storage.type=OBS
 resource.huawei.cloud.obs.endpoint=obs.cn-southwest-2.huaweicloud.com
 
-# if resource.storage.type=HDFS, the user must have the permission to create directories under the HDFS root path
-resource.hdfs.root.user=hdfs
-# if resource.storage.type=S3, the value like: s3a://dolphinscheduler; if resource.storage.type=HDFS and namenode HA is enabled, you need to copy core-site.xml and hdfs-site.xml to conf dir
-resource.hdfs.fs.defaultFS=hdfs://mycluster:8020
-
-# whether to startup kerberos
-hadoop.security.authentication.startup.state=false
-
-# java.security.krb5.conf path
-java.security.krb5.conf.path=/opt/krb5.conf
-
-# login user from keytab username
-login.user.keytab.username=hdfs-mycluster@ESZ.COM
-
-# login user from keytab path
-login.user.keytab.path=/opt/hdfs.headless.keytab
-
-# kerberos expire time, the unit is hour
-kerberos.expire.time=2
-
-
-# resourcemanager port, the default value is 8088 if not specified
-resource.manager.httpaddress.port=8088
-# if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single, keep this value empty
-yarn.resourcemanager.ha.rm.ids=192.168.xx.xx,192.168.xx.xx
-# if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single, you only need to replace ds1 to actual resourcemanager hostname
-yarn.application.status.address=http://ds1:%s/ws/v1/cluster/apps/%s
-# job history status url when application number threshold is reached(default 10000, maybe it was set to 1000)
-yarn.job.history.status.address=http://ds1:19888/ws/v1/history/mapreduce/jobs/%s
-
-# datasource encryption enable
-datasource.encryption.enable=false
-
-# datasource encryption salt
-datasource.encryption.salt=!@#$%^&*
-
-# data quality option
-data-quality.jar.name=dolphinscheduler-data-quality-dev-SNAPSHOT.jar
-
-#data-quality.error.output.path=/tmp/data-quality-error-data
-
-# Network IP gets priority, default inner outer
-
-# Whether hive SQL is executed in the same session
-support.hive.oneSession=false
-
-# use sudo or not, if set true, executing user is tenant user and deploy user needs sudo permissions; if set false, executing user is the deploy user and doesn't need sudo permissions
-sudo.enable=true
-
-# network interface preferred like eth0, default: empty
-#dolphin.scheduler.network.interface.preferred=
-
-# network IP gets priority, default: inner outer
-#dolphin.scheduler.network.priority.strategy=default
-
-# system env path
-#dolphinscheduler.env.path=dolphinscheduler_env.sh
-
-# development state
-development.state=false
-
-# rpc port
-alert.rpc.port=50052
-
-# set path of conda.sh
-conda.path=/opt/anaconda3/etc/profile.d/conda.sh
-
-# Task resource limit state
-task.resource.limit.state=false
-
-# way to collect applicationId: log(original regex match), aop
-appId.collect: log
 ```
 
 > **Note:**

docs/docs/en/guide/resource/intro.md

@@ -2,4 +2,4 @@
 
 The Resource Center is typically used for uploading files, UDF functions, and task group management. For a stand-alone
 environment, you can select the local file directory as the upload folder (**this operation does not require Hadoop or HDFS deployment**).
-Of course, you can also choose to upload to Hadoop or MinIO cluster. In this case, you need to have Hadoop (2.6+) or MinIOn and other related environments.
+Of course, you can also choose to upload to Hadoop or MinIO cluster. In this case, you need to have Hadoop (2.6+) or MinIO and other related environments.

docs/docs/en/guide/resource/task-group.md

@@ -2,6 +2,8 @@
 
 The task group is mainly used to control the concurrency of task instances, and is designed to control the pressure of other resources (it can also control the pressure of the Hadoop cluster, the cluster will have queue control it). When creating a new task definition, you can configure the corresponding task group and configure the priority of the task running in the task group. The user can only view the task groups belongs to authorized projects, and can create or update task groups belongs to one project only if they have write permission.
 
+> Note: The task group's resource restrictions are on the project level and unrelated to tenants.
+
 ## Task Group Configuration
 
 ### Create Task Group

docs/docs/en/guide/security/security.md

@@ -9,6 +9,8 @@ Administrator login, default username/password: admin/dolphinscheduler123
 - The queue is used when executing programs such as spark and mapreduce, and the "queue" parameter needs to be used.
 - The administrator enters the `Security Center->Queue Management` page and clicks the "Create Queue" button to create a new queue.
 
+> Note: Currently, only admin users can modify queues.
+
 ![create-queue](../../../../img/new_ui/dev/security/create-queue.png)
 
 ## Add Tenant
@@ -17,6 +19,8 @@ Administrator login, default username/password: admin/dolphinscheduler123
 - Tenant Code: **The tenant code is the user on Linux, unique and cannot be repeated**
 - The administrator enters the `Security Center->Tenant Management` page, and clicks the `Create Tenant` button to create a tenant.
 
+> Note: Currently, only admin users can modify tenant.
+
 ![create-tenant](../../../../img/new_ui/dev/security/create-tenant.png)
 
 ## Create Normal User

docs/docs/en/guide/start/docker.md

@@ -19,7 +19,7 @@ Start DolphinScheduler with standalone-server Docker images is the easiest way t
 you can learn DolphinScheduler's concepts and usage, with minimal cost.
 
 ```shell
-$ DOLPHINSCHEDULER_VERSION=3.2.0
+$ DOLPHINSCHEDULER_VERSION=<version>
 $ docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:25333 -d apache/dolphinscheduler-standalone-server:"${DOLPHINSCHEDULER_VERSION}"
 ```
 
@@ -37,11 +37,11 @@ be stored on disks after you change docker-compose configuration, and it is robu
 DolphinScheduler in a long term. You have to install [docker-compose](https://docs.docker.com/compose/install/) before you
 start servers.
 
-After complete the installation, get the `docker-compose.yaml` file from [download page](https://dolphinscheduler.apache.org/en-us/download/3.2.0)
+After complete the installation, get the `docker-compose.yaml` file from [download page](https://dolphinscheduler.apache.org/en-us/download/<version>)
 form its source package, and make sure you get the right version. After download the package, you can run the commands as below.
 
 ```shell
-$ DOLPHINSCHEDULER_VERSION=3.2.0
+$ DOLPHINSCHEDULER_VERSION=<version>
 $ tar -zxf apache-dolphinscheduler-"${DOLPHINSCHEDULER_VERSION}"-src.tar.gz
 # Going to docker-compose's location
 # For Mac or Linux users
@@ -68,7 +68,7 @@ $ docker-compose --profile all up -d
 container when it up. You could start DolphinScheduler server separately if you want to reuse your exists services.
 
 ```shell
-$ DOLPHINSCHEDULER_VERSION=3.2.0
+$ DOLPHINSCHEDULER_VERSION=<version>
 # Initialize the database, make sure database <DATABASE> already exists
 $ docker run -d --name dolphinscheduler-tools \
     -e DATABASE="postgresql" \
@@ -81,7 +81,7 @@ $ docker run -d --name dolphinscheduler-tools \
 # Starting DolphinScheduler service
 $ docker run -d --name dolphinscheduler-master \
     -e DATABASE="postgresql" \
-    -e SPRING_DATASOURCE_URL="jdbc:postgresql://localhost:5432/dolphinscheduler" \
+    -e SPRING_DATASOURCE_URL="jdbc:postgresql://localhost:5432/<DATABASE>" \
     -e SPRING_DATASOURCE_USERNAME="<USER>" \
     -e SPRING_DATASOURCE_PASSWORD="<PASSWORD>" \
     -e SPRING_JACKSON_TIME_ZONE="UTC" \
@@ -90,7 +90,7 @@ $ docker run -d --name dolphinscheduler-master \
     -d apache/dolphinscheduler-master:"${DOLPHINSCHEDULER_VERSION}"
 $ docker run -d --name dolphinscheduler-worker \
     -e DATABASE="postgresql" \
-    -e SPRING_DATASOURCE_URL="jdbc:postgresql://localhost:5432/dolphinscheduler" \
+    -e SPRING_DATASOURCE_URL="jdbc:postgresql://localhost:5432/<DATABASE>" \
     -e SPRING_DATASOURCE_USERNAME="<USER>" \
     -e SPRING_DATASOURCE_PASSWORD="<PASSWORD>" \
     -e SPRING_JACKSON_TIME_ZONE="UTC" \
@@ -99,7 +99,7 @@ $ docker run -d --name dolphinscheduler-worker \
     -d apache/dolphinscheduler-worker:"${DOLPHINSCHEDULER_VERSION}"
 $ docker run -d --name dolphinscheduler-api \
     -e DATABASE="postgresql" \
-    -e SPRING_DATASOURCE_URL="jdbc:postgresql://localhost:5432/dolphinscheduler" \
+    -e SPRING_DATASOURCE_URL="jdbc:postgresql://localhost:5432/<DATABASE>" \
     -e SPRING_DATASOURCE_USERNAME="<USER>" \
     -e SPRING_DATASOURCE_PASSWORD="<PASSWORD>" \
     -e SPRING_JACKSON_TIME_ZONE="UTC" \
@@ -108,7 +108,7 @@ $ docker run -d --name dolphinscheduler-api \
     -d apache/dolphinscheduler-api:"${DOLPHINSCHEDULER_VERSION}"
 $ docker run -d --name dolphinscheduler-alert-server \
     -e DATABASE="postgresql" \
-    -e SPRING_DATASOURCE_URL="jdbc:postgresql://localhost:5432/dolphinscheduler" \
+    -e SPRING_DATASOURCE_URL="jdbc:postgresql://localhost:5432/<DATABASE>" \
     -e SPRING_DATASOURCE_USERNAME="<USER>" \
     -e SPRING_DATASOURCE_PASSWORD="<PASSWORD>" \
     -e SPRING_JACKSON_TIME_ZONE="UTC" \
@@ -128,11 +128,11 @@ and use `admin` and `dolphinscheduler123` as default username and password in th
 ![login](../../../../img/new_ui/dev/quick-start/login.png)
 
 > Note: If you start the services by the way [using exists PostgreSQL ZooKeeper](#using-exists-postgresql-zookeeper), and
-> strating with multiple machine, you should change URL domain from `localhost` to IP or hostname the api server running.
+> starting with multiple machine, you should change URL domain from `localhost` to IP or hostname the api server running.
 
 ## Change Environment Variable
 
 You can modify some environment variables to change configurations when you are starting servers through Docker. We have
 an example in [using exists PostgreSQL ZooKeeper](#using-exists-postgresql-zookeeper) to change database and ZooKeeper configurations,
-and you could find all environment variables in [all environment variables](https://github.com/apache/dolphinscheduler/blob/3.2.0/script/env/dolphinscheduler_env.sh) <!-- markdown-link-check-disable-line -->
+and you could find all environment variables in [all environment variables](https://github.com/apache/dolphinscheduler/blob/<version>/script/env/dolphinscheduler_env.sh) <!-- markdown-link-check-disable-line -->
 and change them if you want.

docs/docs/en/guide/task/datafactory.md

@@ -19,11 +19,11 @@ DolphinScheduler DataFactory functions:
 
 ### Application Permission Setting
 
-First, visit the `Subcription` page and choose `Access control (IAM)`, then click `Add role assignment` to the authorization page.
-![Subcription-IAM](../../../../img/tasks/demo/datafactory_auth1.png)
+First, visit the `Subscription` page and choose `Access control (IAM)`, then click `Add role assignment` to the authorization page.
+![Subscription-IAM](../../../../img/tasks/demo/datafactory_auth1.png)
 After that, select `Contributor` role which satisfy functions calls in data factory. Then click `Members` page, and click `Select members`.
 Search application name or application `Object ID` to assign `Contributor` role to application.
-![Subcription-Role](../../../../img/tasks/demo/datafactory_auth2.png)
+![Subscription-Role](../../../../img/tasks/demo/datafactory_auth2.png)
 
 ## Configurations