Compare commits

...

18 Commits

Author SHA1 Message Date
wangxj3 0a21422558 [maven-release-plugin] prepare release 3.2.2 2024-06-30 17:27:43 +08:00
Wenjun Ruan 1e3bbfce77
[Fix-16211] Remove ExecutePath in buildJarCommand to fix Java Task in Jar Mode (#16238) (#16241)
(cherry picked from commit d42ac96c55)

Co-authored-by: blink <lanxing2@users.noreply.github.com>
2024-06-29 18:00:33 +08:00
Wenjun Ruan f13f070242
Fix worker overload will casue task failed (#16223)
(cherry picked from commit 4a41e4290e)
2024-06-27 08:25:42 +08:00
xiangzihao 66b7d1274a
[Chore] [Cherry-Pick] cherry pick some pr to 3.2.2 (#16215)
* [Fix-16174] Incorrect cluster installation guide. (#16208)

* [Fix][CI] fix the ci error of Values.datasource.profile (#16031)

* [Improvement][Helm] Make configmap of api/master/worker/alert configuration (#16058)

Update deploy/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-worker.yaml

* Update deploy/kubernetes/dolphinscheduler/values.yaml

* [helm] remove appversion from labels (#16066)
2024-06-26 14:45:37 +08:00
xiangzihao fe96907fbb
[Chore] Revert some incompatible change for 3.2.2 (#16214)
* revert pr 14977

* fix spotless

* revert pr 14977

* revert pr 14977
2024-06-26 13:17:51 +08:00
Wenjun Ruan e65208b34e
Fix task cannot use workflow's environment (#16199) (#16206)
(cherry picked from commit b34fe46044)
2024-06-24 23:00:18 +08:00
Wenjun Ruan 2f928b15c9
[Chore] Disable composite indicator (#16172) (#16205)
(cherry picked from commit 3f3f0cd4cb)
2024-06-24 22:02:13 +08:00
zhuxt2015 e524cfe744
[Improvement-16125][dolphinscheduler-dist] Reduce binary distribution tarball file size (#16192) 2024-06-24 21:50:18 +08:00
xiangzihao 16bc1e7bfb
Revert "3.2.2 release add task-pure delete some task module to reduce the si…" (#16184)
This reverts commit 3ad53c5200.
2024-06-20 13:44:55 +08:00
xiangzihao 3cdf754b1b
Fix admin login tenant unspecified issue after upgrade to v3.2.1 (#16064) (#16185) 2024-06-20 13:44:28 +08:00
Wenjun Ruan 82ec84f85c
Fix condition task will cause workflow instance failed (#16152) (#16177)
(cherry picked from commit 5df4b1cbc3)
2024-06-19 12:39:48 +08:00
wangxj3 3ad53c5200
3.2.2 release add task-pure delete some task module to reduce the size of the deployment package (#16094)
* add task-pure delete some task module to reduce the size of the deployment package

* add version of 3.2.2

* Update Chart.yaml Rollback APPversion

* remove task pure plugin

* fix ci

* add pure datasource plugin

* remove ck,sr from datasource pure

* remove task plugin chunjun ,dms ,datafactory from pure

* add api pom dependency

* spotless:apply

* add dependency license

* [maven-release-plugin] prepare release 3.2.2

* [maven-release-plugin] prepare for next development iteration

* remove datasource-pure:
dolphinscheduler-datasource-sqlserver
dolphinscheduler-datasource-oracle

remove task-pure:
dolphinscheduler-task-dinky
dolphinscheduler-task-pytorch
dolphinscheduler-task-datasync
dolphinscheduler-task-kubeflow
dolphinscheduler-task-emr
dolphinscheduler-task-pigeon
dolphinscheduler-task-flink-stream

* add register pure remove jdbc and etcd

* add dolphinscheduler-storage-pure remove task of
dolphinscheduler-storage-obs
dolphinscheduler-storage-abs
dolphinscheduler-storage-gcs
dolphinscheduler-storage-oss

* mvn spotless:apply

---------

Co-authored-by: wangxj959 <wangxj959@chinaunicom.cn>
Co-authored-by: wangxj <wangxj31>
Co-authored-by: wangxj3 <wangxj@apache.org>
2024-06-08 17:05:20 +08:00
Wenjun Ruan e91a272965
Fix SqlServer split merge sql will miss ; (#16118) (#16120)
(cherry picked from commit d6714bb77a)
2024-06-06 17:11:51 +08:00
Wenjun Ruan 31e79d10ea
Fix task nodes under switch task will not be skipped (#16109)
(cherry picked from commit f687a7d851c206ae30377ac3f9dc025cec35ffb2)
2024-06-05 23:41:52 +08:00
Wenjun Ruan 50c434e6c4
Fix task instance which status is dispatch will not be failover (#16110)
(cherry picked from commit a59ecf2aa19f4acd8d9981d14abe4d0c0d8e7fec)
2024-06-05 13:31:29 +08:00
Wenjun Ruan 72b402422a
Revert "[Improvement] Abnormal characters check (#15824)" (#16102) (#16103)
This reverts commit e5e7749251.

(cherry picked from commit 7ae9ca5a33)
2024-05-31 21:21:14 +08:00
xiangzihao fd864bf351
Revert "3.2.2 release cherry-pick cp: Reduce the size of tarball to continue …" (#16068)
This reverts commit 646bbf4881.
2024-05-25 10:50:37 +08:00
wangxj3 646bbf4881
3.2.2 release cherry-pick cp: Reduce the size of tarball to continue ASF release (#16042)
* add version of 3.2.2

* Fix rpc channel leak due to concurrent operation (#16021)

* Fix rpc channel leak due to concurrent operation

* Throw channel create failed exception

---------

Co-authored-by: Rick Cheng <rickchengx@gmail.com>

* Fix WorkerTaskExecutorThreadPool#isOverload is incorrect (#16027)

* cp: Reduce the size of tarball to continue ASF release (#15004) (#15540)

* Reduce the size of tarball to continue ASF release

for more detail you can see https://lists.apache.org/thread/rmp7fghlj0n7h9y2v3p8gkw9f9qbo6qt
# Conflicts:
#	tools/dependencies/known-dependencies.txt

* Increase block-until-connected in ZookeeperRegistryTestCase (#16041)

* [Fix][CI] fix the ci error of Values.datasource.profile (#16031)

Co-authored-by: Eric Gao <ericgao.apache@gmail.com>

* Correction of Typos in the Chinese Document Appendix for Task Parameters (#16033)

Co-authored-by: Rick Cheng <rickchengx@gmail.com>

* [DSIP-39][parameter] Improvement startup parameters/global parameters/project parameters data type (#15967)

* [Improvement][parameter] New data types and type filtering

* [Improvement][parameter] Improvement startup parameters/global parameters data type

* fix api interfaces compatible

* add project parameter data type default value

* [Improvement][parameter] New data types and type filtering

* [Improvement][parameter] Improvement startup parameters/global parameters data type

* fix api interfaces compatible

* add project parameter data type default value

* improvement project code

* remove useless imports

* remove method onClearSearchTaskType

* add parameter doc

* optimisation logic

* code conflict resolution

* code conflict resolution

* [Improvement][Monitor] Show master && worker Busy Or Normal Status and Show Commands table list (#15978)

* update

* test

* add monitor enhance ui

* update

* update

* update doc

* fix spotless

* update

* update

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/CommandMapper.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* update

* fix spotless

* update

---------

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* [Improvement][Monitor] Add UT for montor (#15998)

* formatting Code

* add pom

* Fix NettyRemotingClient might throw IllegalMonitorStateException (#16038)

* Removed unused StateEventHandleFailure (#16052)

* Remove OmitStackTraceInFastThrow in start.sh (#16054)

* Update Chart.yaml Rollback APPversion

* modify log msg (#16062)

Co-authored-by: xiangzihao <460888207@qq.com>

* [DS-16046][fix] Set PreparedStatement parameter type (#16050)

Fix the PreparedStatement parameter is TIME, set it to java.sql.Time

Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>

* Update pom.xml

* [DSIP-42] Add dolphinscheduler-aws-authentication module  (#16043)

---------

Co-authored-by: wangxj <wangxj31>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
Co-authored-by: TianXinCoord <tianxincoord@163.com>
Co-authored-by: 小可耐 <46134044+sdhzwc@users.noreply.github.com>
Co-authored-by: sleo <97011595+alei1206@users.noreply.github.com>
Co-authored-by: xiangzihao <460888207@qq.com>
Co-authored-by: yinxiaolog <43245392+yinxiaolog@users.noreply.github.com>
Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
2024-05-25 10:16:18 +08:00
222 changed files with 1078 additions and 2224 deletions

View File

@ -14,6 +14,8 @@ Please refer to the [Quick Start in Kubernetes](../../../docs/docs/en/guide/inst
|-----|------|---------|-------------| |-----|------|---------|-------------|
| alert.affinity | object | `{}` | Affinity is a group of affinity scheduling rules. If specified, the pod's scheduling constraints. More info: [node-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity) | | alert.affinity | object | `{}` | Affinity is a group of affinity scheduling rules. If specified, the pod's scheduling constraints. More info: [node-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity) |
| alert.annotations | object | `{}` | You can use annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata. | | alert.annotations | object | `{}` | You can use annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata. |
| alert.customizedConfig | object | `{}` | configure aligned with https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/resources/application.yaml |
| alert.enableCustomizedConfig | bool | `false` | enable configure custom config |
| alert.enabled | bool | `true` | Enable or disable the Alert-Server component | | alert.enabled | bool | `true` | Enable or disable the Alert-Server component |
| alert.env.JAVA_OPTS | string | `"-Xms512m -Xmx512m -Xmn256m"` | The jvm options for alert server | | alert.env.JAVA_OPTS | string | `"-Xms512m -Xmx512m -Xmn256m"` | The jvm options for alert server |
| alert.livenessProbe | object | `{"enabled":true,"failureThreshold":"3","initialDelaySeconds":"30","periodSeconds":"30","successThreshold":"1","timeoutSeconds":"5"}` | Periodic probe of container liveness. Container will be restarted if the probe fails. More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) | | alert.livenessProbe | object | `{"enabled":true,"failureThreshold":"3","initialDelaySeconds":"30","periodSeconds":"30","successThreshold":"1","timeoutSeconds":"5"}` | Periodic probe of container liveness. Container will be restarted if the probe fails. More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) |
@ -52,6 +54,8 @@ Please refer to the [Quick Start in Kubernetes](../../../docs/docs/en/guide/inst
| alert.tolerations | list | `[]` | Tolerations are appended (excluding duplicates) to pods running with this RuntimeClass during admission, effectively unioning the set of nodes tolerated by the pod and the RuntimeClass. | | alert.tolerations | list | `[]` | Tolerations are appended (excluding duplicates) to pods running with this RuntimeClass during admission, effectively unioning the set of nodes tolerated by the pod and the RuntimeClass. |
| api.affinity | object | `{}` | Affinity is a group of affinity scheduling rules. If specified, the pod's scheduling constraints. More info: [node-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity) | | api.affinity | object | `{}` | Affinity is a group of affinity scheduling rules. If specified, the pod's scheduling constraints. More info: [node-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity) |
| api.annotations | object | `{}` | You can use annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata. | | api.annotations | object | `{}` | You can use annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata. |
| api.customizedConfig | object | `{}` | configure aligned with https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-api/src/main/resources/application.yaml |
| api.enableCustomizedConfig | bool | `false` | enable configure custom config |
| api.enabled | bool | `true` | Enable or disable the API-Server component | | api.enabled | bool | `true` | Enable or disable the API-Server component |
| api.env.JAVA_OPTS | string | `"-Xms512m -Xmx512m -Xmn256m"` | The jvm options for api server | | api.env.JAVA_OPTS | string | `"-Xms512m -Xmx512m -Xmn256m"` | The jvm options for api server |
| api.livenessProbe | object | `{"enabled":true,"failureThreshold":"3","initialDelaySeconds":"30","periodSeconds":"30","successThreshold":"1","timeoutSeconds":"5"}` | Periodic probe of container liveness. Container will be restarted if the probe fails. More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) | | api.livenessProbe | object | `{"enabled":true,"failureThreshold":"3","initialDelaySeconds":"30","periodSeconds":"30","successThreshold":"1","timeoutSeconds":"5"}` | Periodic probe of container liveness. Container will be restarted if the probe fails. More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) |
@ -158,6 +162,7 @@ Please refer to the [Quick Start in Kubernetes](../../../docs/docs/en/guide/inst
| conf.common."yarn.application.status.address" | string | `"http://ds1:%s/ws/v1/cluster/apps/%s"` | if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single, you only need to replace ds1 to actual resourcemanager hostname | | conf.common."yarn.application.status.address" | string | `"http://ds1:%s/ws/v1/cluster/apps/%s"` | if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single, you only need to replace ds1 to actual resourcemanager hostname |
| conf.common."yarn.job.history.status.address" | string | `"http://ds1:19888/ws/v1/history/mapreduce/jobs/%s"` | job history status url when application number threshold is reached(default 10000, maybe it was set to 1000) | | conf.common."yarn.job.history.status.address" | string | `"http://ds1:19888/ws/v1/history/mapreduce/jobs/%s"` | job history status url when application number threshold is reached(default 10000, maybe it was set to 1000) |
| conf.common."yarn.resourcemanager.ha.rm.ids" | string | `"192.168.xx.xx,192.168.xx.xx"` | if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single, keep this value empty | | conf.common."yarn.resourcemanager.ha.rm.ids" | string | `"192.168.xx.xx,192.168.xx.xx"` | if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single, keep this value empty |
| datasource.profile | string | `"postgresql"` | The profile of datasource |
| externalDatabase.database | string | `"dolphinscheduler"` | The database of external database | | externalDatabase.database | string | `"dolphinscheduler"` | The database of external database |
| externalDatabase.driverClassName | string | `"org.postgresql.Driver"` | The driverClassName of external database | | externalDatabase.driverClassName | string | `"org.postgresql.Driver"` | The driverClassName of external database |
| externalDatabase.enabled | bool | `false` | If exists external database, and set postgresql.enable value to false. external database will be used, otherwise Dolphinscheduler's internal database will be used. | | externalDatabase.enabled | bool | `false` | If exists external database, and set postgresql.enable value to false. external database will be used, otherwise Dolphinscheduler's internal database will be used. |
@ -189,6 +194,8 @@ Please refer to the [Quick Start in Kubernetes](../../../docs/docs/en/guide/inst
| initImage.pullPolicy | string | `"IfNotPresent"` | Image pull policy. Options: Always, Never, IfNotPresent | | initImage.pullPolicy | string | `"IfNotPresent"` | Image pull policy. Options: Always, Never, IfNotPresent |
| master.affinity | object | `{}` | Affinity is a group of affinity scheduling rules. If specified, the pod's scheduling constraints. More info: [node-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity) | | master.affinity | object | `{}` | Affinity is a group of affinity scheduling rules. If specified, the pod's scheduling constraints. More info: [node-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity) |
| master.annotations | object | `{}` | You can use annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata. | | master.annotations | object | `{}` | You can use annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata. |
| master.customizedConfig | object | `{}` | configure aligned with https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-master/src/main/resources/application.yaml |
| master.enableCustomizedConfig | bool | `false` | enable configure custom config |
| master.enabled | bool | `true` | Enable or disable the Master component | | master.enabled | bool | `true` | Enable or disable the Master component |
| master.env.JAVA_OPTS | string | `"-Xms1g -Xmx1g -Xmn512m"` | The jvm options for master server | | master.env.JAVA_OPTS | string | `"-Xms1g -Xmx1g -Xmn512m"` | The jvm options for master server |
| master.env.MASTER_DISPATCH_TASK_NUM | string | `"3"` | Master dispatch task number per batch | | master.env.MASTER_DISPATCH_TASK_NUM | string | `"3"` | Master dispatch task number per batch |
@ -295,6 +302,8 @@ Please refer to the [Quick Start in Kubernetes](../../../docs/docs/en/guide/inst
| timezone | string | `"Asia/Shanghai"` | World time and date for cities in all time zones | | timezone | string | `"Asia/Shanghai"` | World time and date for cities in all time zones |
| worker.affinity | object | `{}` | Affinity is a group of affinity scheduling rules. If specified, the pod's scheduling constraints. More info: [node-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity) | | worker.affinity | object | `{}` | Affinity is a group of affinity scheduling rules. If specified, the pod's scheduling constraints. More info: [node-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity) |
| worker.annotations | object | `{}` | You can use annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata. | | worker.annotations | object | `{}` | You can use annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata. |
| worker.customizedConfig | object | `{}` | configure aligned with https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-worker/src/main/resources/application.yaml |
| worker.enableCustomizedConfig | bool | `false` | enable configure custom config |
| worker.enabled | bool | `true` | Enable or disable the Worker component | | worker.enabled | bool | `true` | Enable or disable the Worker component |
| worker.env.DEFAULT_TENANT_ENABLED | bool | `false` | If set true, will use worker bootstrap user as the tenant to execute task when the tenant is `default`; | | worker.env.DEFAULT_TENANT_ENABLED | bool | `false` | If set true, will use worker bootstrap user as the tenant to execute task when the tenant is `default`; |
| worker.env.WORKER_EXEC_THREADS | string | `"100"` | Worker execute thread number to limit task instances | | worker.env.WORKER_EXEC_THREADS | string | `"100"` | Worker execute thread number to limit task instances |
@ -314,7 +323,6 @@ Please refer to the [Quick Start in Kubernetes](../../../docs/docs/en/guide/inst
| worker.keda.minReplicaCount | int | `0` | Minimum number of workers created by keda | | worker.keda.minReplicaCount | int | `0` | Minimum number of workers created by keda |
| worker.keda.namespaceLabels | object | `{}` | Keda namespace labels | | worker.keda.namespaceLabels | object | `{}` | Keda namespace labels |
| worker.keda.pollingInterval | int | `5` | How often KEDA polls the DolphinScheduler DB to report new scale requests to the HPA | | worker.keda.pollingInterval | int | `5` | How often KEDA polls the DolphinScheduler DB to report new scale requests to the HPA |
| worker.livenessProbe | object | `{"enabled":true,"failureThreshold":"3","initialDelaySeconds":"30","periodSeconds":"30","successThreshold":"1","timeoutSeconds":"5"}` | Periodic probe of container liveness. Container will be restarted if the probe fails. More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) |
| worker.livenessProbe.enabled | bool | `true` | Turn on and off liveness probe | | worker.livenessProbe.enabled | bool | `true` | Turn on and off liveness probe |
| worker.livenessProbe.failureThreshold | string | `"3"` | Minimum consecutive failures for the probe | | worker.livenessProbe.failureThreshold | string | `"3"` | Minimum consecutive failures for the probe |
| worker.livenessProbe.initialDelaySeconds | string | `"30"` | Delay before liveness probe is initiated | | worker.livenessProbe.initialDelaySeconds | string | `"30"` | Delay before liveness probe is initiated |

View File

@ -51,7 +51,6 @@ Create a default common labels.
{{- define "dolphinscheduler.common.labels" -}} {{- define "dolphinscheduler.common.labels" -}}
app.kubernetes.io/instance: {{ .Release.Name }} app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }} app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/version: {{ .Chart.AppVersion }}
{{- end -}} {{- end -}}
{{/* {{/*

View File

@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
# #
{{- if and .Values.alert.enabled }} {{- if .Values.alert.enableCustomizedConfig }}
apiVersion: v1 apiVersion: v1
kind: ConfigMap kind: ConfigMap
metadata: metadata:
@ -23,83 +23,8 @@ metadata:
app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-alert app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-alert
{{- include "dolphinscheduler.alert.labels" . | nindent 4 }} {{- include "dolphinscheduler.alert.labels" . | nindent 4 }}
data: data:
application.yaml: | {{- range $path, $config := .Values.alert.customizedConfig }}
spring: {{ $path }}: |
profiles: {{ $config | indent 4 -}}
active: {{ .Values.datasource.profile }} {{- end -}}
jackson: {{- end -}}
time-zone: UTC
date-format: "yyyy-MM-dd HH:mm:ss"
banner:
charset: UTF-8
datasource:
profile: postgresql
config:
driver-class-name: org.postgresql.Driver
url: jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
username: root
password: root
hikari:
connection-test-query: select 1
pool-name: DolphinScheduler
# Mybatis-plus configuration, you don't need to change it
mybatis-plus:
mapper-locations: classpath:org/apache/dolphinscheduler/dao/mapper/*Mapper.xml
type-aliases-package: org.apache.dolphinscheduler.dao.entity
configuration:
cache-enabled: false
call-setters-on-nulls: true
map-underscore-to-camel-case: true
jdbc-type-for-null: NULL
global-config:
db-config:
id-type: auto
banner: false
server:
port: 50053
management:
endpoints:
web:
exposure:
include: health,metrics,prometheus
endpoint:
health:
enabled: true
show-details: always
health:
db:
enabled: true
defaults:
enabled: false
metrics:
tags:
application: ${spring.application.name}
alert:
port: 50052
# Mark each alert of alert server if late after x milliseconds as failed.
# Define value is (0 = infinite), and alert server would be waiting alert result.
wait-timeout: 0
max-heartbeat-interval: 60s
query_alert_threshold: 100
registry:
type: zookeeper
zookeeper:
namespace: dolphinscheduler
connect-string: localhost:2181
retry-policy:
base-sleep-time: 60ms
max-sleep: 300ms
max-retries: 5
session-timeout: 30s
connection-timeout: 9s
block-until-connected: 600ms
digest: ~
metrics:
enabled: true
{{- end }}

View File

@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
# #
{{- if and .Values.api.enabled }} {{- if .Values.api.enableCustomizedConfig }}
apiVersion: v1 apiVersion: v1
kind: ConfigMap kind: ConfigMap
metadata: metadata:
@ -23,223 +23,8 @@ metadata:
app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-api app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-api
{{- include "dolphinscheduler.api.labels" . | nindent 4 }} {{- include "dolphinscheduler.api.labels" . | nindent 4 }}
data: data:
application.yaml: | {{- range $path, $config := .Values.api.customizedConfig }}
server: {{ $path }}: |
port: 12345 {{ $config | indent 4 -}}
servlet: {{- end -}}
session: {{- end -}}
timeout: 120m
context-path: /dolphinscheduler/
compression:
enabled: true
mime-types: text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml
jetty:
max-http-form-post-size: 5000000
accesslog:
enabled: true
custom-format: '%{client}a - %u %t "%r" %s %O %{ms}Tms'
spring:
profiles:
active: {{ .Values.datasource.profile }}
banner:
charset: UTF-8
jackson:
time-zone: UTC
date-format: "yyyy-MM-dd HH:mm:ss"
servlet:
multipart:
max-file-size: 1024MB
max-request-size: 1024MB
messages:
basename: i18n/messages
datasource:
profile: postgresql
config:
driver-class-name: org.postgresql.Driver
url: jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
username: root
password: root
hikari:
connection-test-query: select 1
pool-name: DolphinScheduler
quartz:
auto-startup: false
job-store-type: jdbc
jdbc:
initialize-schema: never
properties:
org.quartz.jobStore.isClustered: true
org.quartz.jobStore.class: org.springframework.scheduling.quartz.LocalDataSourceJobStore
org.quartz.scheduler.instanceId: AUTO
org.quartz.jobStore.tablePrefix: QRTZ_
org.quartz.jobStore.acquireTriggersWithinLock: true
org.quartz.scheduler.instanceName: DolphinScheduler
org.quartz.threadPool.class: org.apache.dolphinscheduler.scheduler.quartz.QuartzZeroSizeThreadPool
org.quartz.jobStore.useProperties: false
org.quartz.jobStore.misfireThreshold: 60000
org.quartz.scheduler.makeSchedulerThreadDaemon: true
org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
org.quartz.jobStore.clusterCheckinInterval: 5000
org.quartz.scheduler.batchTriggerAcquisitionMaxCount: 1
mvc:
pathmatch:
matching-strategy: ANT_PATH_MATCHER
static-path-pattern: /static/**
springdoc:
swagger-ui:
path: /swagger-ui.html
packages-to-scan: org.apache.dolphinscheduler.api
# Mybatis-plus configuration, you don't need to change it
mybatis-plus:
mapper-locations: classpath:org/apache/dolphinscheduler/dao/mapper/*Mapper.xml
type-aliases-package: org.apache.dolphinscheduler.dao.entity
configuration:
cache-enabled: false
call-setters-on-nulls: true
map-underscore-to-camel-case: true
jdbc-type-for-null: NULL
global-config:
db-config:
id-type: auto
banner: false
management:
endpoints:
web:
exposure:
include: health,metrics,prometheus
endpoint:
health:
enabled: true
show-details: always
health:
db:
enabled: true
defaults:
enabled: false
metrics:
tags:
application: ${spring.application.name}
registry:
type: zookeeper
zookeeper:
namespace: dolphinscheduler
connect-string: localhost:2181
retry-policy:
base-sleep-time: 60ms
max-sleep: 300ms
max-retries: 5
session-timeout: 60s
connection-timeout: 15s
block-until-connected: 15s
digest: ~
api:
audit-enable: false
# Traffic control, if you turn on this config, the maximum number of request/s will be limited.
# global max request number per second
# default tenant-level max request number
traffic-control:
global-switch: false
max-global-qps-rate: 300
tenant-switch: false
default-tenant-qps-rate: 10
#customize-tenant-qps-rate:
# eg.
#tenant1: 11
#tenant2: 20
python-gateway:
# Weather enable python gateway server or not. The default value is false.
enabled: false
# Authentication token for connection from python api to python gateway server. Should be changed the default value
# when you deploy in public network.
auth-token: jwUDzpLsNKEFER4*a8gruBH_GsAurNxU7A@Xc
# The address of Python gateway server start. Set its value to `0.0.0.0` if your Python API run in different
# between Python gateway server. It could be be specific to other address like `127.0.0.1` or `localhost`
gateway-server-address: 0.0.0.0
# The port of Python gateway server start. Define which port you could connect to Python gateway server from
# Python API side.
gateway-server-port: 25333
# The address of Python callback client.
python-address: 127.0.0.1
# The port of Python callback client.
python-port: 25334
# Close connection of socket server if no other request accept after x milliseconds. Define value is (0 = infinite),
# and socket server would never close even though no requests accept
connect-timeout: 0
# Close each active connection of socket server if python program not active after x milliseconds. Define value is
# (0 = infinite), and socket server would never close even though no requests accept
read-timeout: 0
metrics:
enabled: true
security:
authentication:
# Authentication types (supported types: PASSWORD,LDAP,CASDOOR_SSO)
type: PASSWORD
# IF you set type `LDAP`, below config will be effective
ldap:
# ldap server config
urls: ldap://ldap.forumsys.com:389/
base-dn: dc=example,dc=com
username: cn=read-only-admin,dc=example,dc=com
password: password
user:
# admin userId when you use LDAP login
admin: read-only-admin
identity-attribute: uid
email-attribute: mail
# action when ldap user is not exist (supported types: CREATE,DENY)
not-exist-action: CREATE
ssl:
enable: false
# jks file absolute path && password
trust-store: "/ldapkeystore.jks"
trust-store-password: "password"
casdoor:
user:
admin: ""
oauth2:
enable: false
provider:
github:
authorizationUri: ""
redirectUri: ""
clientId: ""
clientSecret: ""
tokenUri: ""
userInfoUri: ""
callbackUrl: ""
iconUri: ""
provider: github
google:
authorizationUri: ""
redirectUri: ""
clientId: ""
clientSecret: ""
tokenUri: ""
userInfoUri: ""
callbackUrl: ""
iconUri: ""
provider: google
casdoor:
# Your Casdoor server url
endpoint: ""
client-id: ""
client-secret: ""
# The certificate may be multi-line, you can use `|-` for ease
certificate: ""
# Your organization name added in Casdoor
organization-name: ""
# Your application name added in Casdoor
application-name: ""
# Doplhinscheduler login url
redirect-url: ""
{{- end }}

View File

@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
# #
{{- if and .Values.master.enabled }} {{- if .Values.master.enableCustomizedConfig }}
apiVersion: v1 apiVersion: v1
kind: ConfigMap kind: ConfigMap
metadata: metadata:
@ -23,138 +23,8 @@ metadata:
app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-master app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-master
{{- include "dolphinscheduler.master.labels" . | nindent 4 }} {{- include "dolphinscheduler.master.labels" . | nindent 4 }}
data: data:
application.yaml: | {{- range $path, $config := .Values.master.customizedConfig }}
spring: {{ $path }}: |
profiles: {{ $config | indent 4 -}}
active: {{ .Values.datasource.profile }} {{- end -}}
banner: {{- end -}}
charset: UTF-8
jackson:
time-zone: UTC
date-format: "yyyy-MM-dd HH:mm:ss"
datasource:
profile: postgresql
config:
driver-class-name: org.postgresql.Driver
url: jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
username: root
password: root
hikari:
connection-test-query: select 1
pool-name: DolphinScheduler
quartz:
job-store-type: jdbc
jdbc:
initialize-schema: never
properties:
org.quartz.threadPool.threadPriority: 5
org.quartz.jobStore.isClustered: true
org.quartz.jobStore.class: org.springframework.scheduling.quartz.LocalDataSourceJobStore
org.quartz.scheduler.instanceId: AUTO
org.quartz.jobStore.tablePrefix: QRTZ_
org.quartz.jobStore.acquireTriggersWithinLock: true
org.quartz.scheduler.instanceName: DolphinScheduler
org.quartz.threadPool.class: org.quartz.simpl.SimpleThreadPool
org.quartz.jobStore.useProperties: false
org.quartz.threadPool.makeThreadsDaemons: true
org.quartz.threadPool.threadCount: 25
org.quartz.jobStore.misfireThreshold: 60000
org.quartz.scheduler.batchTriggerAcquisitionMaxCount: 1
org.quartz.scheduler.makeSchedulerThreadDaemon: true
org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
org.quartz.jobStore.clusterCheckinInterval: 5000
# Mybatis-plus configuration, you don't need to change it
mybatis-plus:
mapper-locations: classpath:org/apache/dolphinscheduler/dao/mapper/*Mapper.xml
type-aliases-package: org.apache.dolphinscheduler.dao.entity
configuration:
cache-enabled: false
call-setters-on-nulls: true
map-underscore-to-camel-case: true
jdbc-type-for-null: NULL
global-config:
db-config:
id-type: auto
banner: false
registry:
type: zookeeper
zookeeper:
namespace: dolphinscheduler
connect-string: localhost:2181
retry-policy:
base-sleep-time: 60ms
max-sleep: 300ms
max-retries: 5
session-timeout: 30s
connection-timeout: 9s
block-until-connected: 600ms
digest: ~
master:
listen-port: 5678
# master fetch command num
fetch-command-num: 10
# master prepare execute thread number to limit handle commands in parallel
pre-exec-threads: 10
# master execute thread number to limit process instances in parallel
exec-threads: 100
# master dispatch task number per batch, if all the tasks dispatch failed in a batch, will sleep 1s.
dispatch-task-number: 3
# master host selector to select a suitable worker, default value: LowerWeight. Optional values include random, round_robin, lower_weight
host-selector: lower_weight
# master heartbeat interval
max-heartbeat-interval: 10s
# master commit task retry times
task-commit-retry-times: 5
# master commit task interval
task-commit-interval: 1s
state-wheel-interval: 5s
server-load-protection:
# If set true, will open master overload protection
enabled: true
# Master max system cpu usage, when the master's system cpu usage is smaller then this value, master server can execute workflow.
max-system-cpu-usage-percentage-thresholds: 0.7
# Master max jvm cpu usage, when the master's jvm cpu usage is smaller then this value, master server can execute workflow.
max-jvm-cpu-usage-percentage-thresholds: 0.7
# Master max System memory usage , when the master's system memory usage is smaller then this value, master server can execute workflow.
max-system-memory-usage-percentage-thresholds: 0.7
# Master max disk usage , when the master's disk usage is smaller then this value, master server can execute workflow.
max-disk-usage-percentage-thresholds: 0.7
# failover interval, the unit is minute
failover-interval: 10m
# kill yarn / k8s application when failover taskInstance, default true
kill-application-when-task-failover: true
registry-disconnect-strategy:
# The disconnect strategy: stop, waiting
strategy: waiting
# The max waiting time to reconnect to registry if you set the strategy to waiting
max-waiting-time: 100s
worker-group-refresh-interval: 10s
server:
port: 5679
management:
endpoints:
web:
exposure:
include: health,metrics,prometheus
endpoint:
health:
enabled: true
show-details: always
health:
db:
enabled: true
defaults:
enabled: false
metrics:
tags:
application: ${spring.application.name}
metrics:
enabled: true
{{- end }}

View File

@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
# #
{{- if and .Values.worker.enabled }} {{- if .Values.worker.enableCustomizedConfig }}
apiVersion: v1 apiVersion: v1
kind: ConfigMap kind: ConfigMap
metadata: metadata:
@ -23,86 +23,8 @@ metadata:
app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-worker app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-worker
{{- include "dolphinscheduler.worker.labels" . | nindent 4 }} {{- include "dolphinscheduler.worker.labels" . | nindent 4 }}
data: data:
application.yaml: | {{- range $path, $config := .Values.worker.customizedConfig }}
spring: {{ $path }}: |
banner: {{ $config | indent 4 -}}
charset: UTF-8 {{- end -}}
jackson: {{- end -}}
time-zone: UTC
date-format: "yyyy-MM-dd HH:mm:ss"
autoconfigure:
exclude:
- org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration
registry:
type: zookeeper
zookeeper:
namespace: dolphinscheduler
connect-string: localhost:2181
retry-policy:
base-sleep-time: 60ms
max-sleep: 300ms
max-retries: 5
session-timeout: 30s
connection-timeout: 9s
block-until-connected: 600ms
digest: ~
worker:
# worker listener port
listen-port: 1234
# worker execute thread number to limit task instances in parallel
exec-threads: 100
# worker heartbeat interval
max-heartbeat-interval: 10s
# worker host weight to dispatch tasks, default value 100
host-weight: 100
server-load-protection:
# If set true, will open worker overload protection
enabled: true
# Worker max system cpu usage, when the worker's system cpu usage is smaller then this value, worker server can be dispatched tasks.
max-system-cpu-usage-percentage-thresholds: 0.7
# Worker max jvm cpu usage, when the worker's jvm cpu usage is smaller then this value, worker server can be dispatched tasks.
max-jvm-cpu-usage-percentage-thresholds: 0.7
# Worker max System memory usage , when the master's system memory usage is smaller then this value, master server can execute workflow.
max-system-memory-usage-percentage-thresholds: 0.7
# Worker max disk usage , when the worker's disk usage is smaller then this value, worker server can be dispatched tasks.
max-disk-usage-percentage-thresholds: 0.7
registry-disconnect-strategy:
# The disconnect strategy: stop, waiting
strategy: waiting
# The max waiting time to reconnect to registry if you set the strategy to waiting
max-waiting-time: 100s
task-execute-threads-full-policy: REJECT
tenant-config:
# tenant corresponds to the user of the system, which is used by the worker to submit the job. If system does not have this user, it will be automatically created after the parameter worker.tenant.auto.create is true.
auto-create-tenant-enabled: true
# Scenes to be used for distributed users. For example, users created by FreeIpa are stored in LDAP. This parameter only applies to Linux, When this parameter is true, auto-create-tenant-enabled has no effect and will not automatically create tenants.
distributed-tenant-enabled: false
# If set true, will use worker bootstrap user as the tenant to execute task when the tenant is `default`.
default-tenant-enabled: false
server:
port: 1235
management:
endpoints:
web:
exposure:
include: health,metrics,prometheus
endpoint:
health:
enabled: true
show-details: always
health:
db:
enabled: true
defaults:
enabled: false
metrics:
tags:
application: ${spring.application.name}
metrics:
enabled: true
{{- end }}

View File

@ -115,9 +115,11 @@ spec:
- name: config-volume - name: config-volume
mountPath: /opt/dolphinscheduler/conf/common.properties mountPath: /opt/dolphinscheduler/conf/common.properties
subPath: common.properties subPath: common.properties
{{- if .Values.alert.enableCustomizedConfig }}
- name: alert-config-volume - name: alert-config-volume
mountPath: /opt/dolphinscheduler/conf/application.yaml mountPath: /opt/dolphinscheduler/conf/application.yaml
subPath: application.yaml subPath: application.yaml
{{- end }}
volumes: volumes:
- name: {{ include "dolphinscheduler.fullname" . }}-alert - name: {{ include "dolphinscheduler.fullname" . }}-alert
{{- if .Values.alert.persistentVolumeClaim.enabled }} {{- if .Values.alert.persistentVolumeClaim.enabled }}
@ -129,7 +131,9 @@ spec:
- name: config-volume - name: config-volume
configMap: configMap:
name: {{ include "dolphinscheduler.fullname" . }}-configs name: {{ include "dolphinscheduler.fullname" . }}-configs
{{- if .Values.alert.enableCustomizedConfig }}
- name: alert-config-volume - name: alert-config-volume
configMap: configMap:
name: { { include "dolphinscheduler.fullname" . } }-alert name: {{ include "dolphinscheduler.fullname" . }}-alert
{{- end }}
{{- end }} {{- end }}

View File

@ -121,9 +121,11 @@ spec:
mountPath: /opt/dolphinscheduler/conf/task-type-config.yaml mountPath: /opt/dolphinscheduler/conf/task-type-config.yaml
subPath: task-type-config.yaml subPath: task-type-config.yaml
{{- end }} {{- end }}
{{- if .Values.api.enableCustomizedConfig }}
- name: api-config-volume - name: api-config-volume
mountPath: /opt/dolphinscheduler/conf/application.yaml mountPath: /opt/dolphinscheduler/conf/application.yaml
subPath: application.yaml subPath: application.yaml
{{- end }}
{{- include "dolphinscheduler.sharedStorage.volumeMount" . | nindent 12 }} {{- include "dolphinscheduler.sharedStorage.volumeMount" . | nindent 12 }}
{{- include "dolphinscheduler.fsFileResource.volumeMount" . | nindent 12 }} {{- include "dolphinscheduler.fsFileResource.volumeMount" . | nindent 12 }}
{{- include "dolphinscheduler.ldap.ssl.volumeMount" . | nindent 12 }} {{- include "dolphinscheduler.ldap.ssl.volumeMount" . | nindent 12 }}
@ -139,9 +141,11 @@ spec:
- name: config-volume - name: config-volume
configMap: configMap:
name: {{ include "dolphinscheduler.fullname" . }}-configs name: {{ include "dolphinscheduler.fullname" . }}-configs
{{- if .Values.api.enableCustomizedConfig }}
- name: api-config-volume - name: api-config-volume
configMap: configMap:
name: { { include "dolphinscheduler.fullname" . } }-api name: {{ include "dolphinscheduler.fullname" . }}-api
{{- end }}
{{- include "dolphinscheduler.sharedStorage.volume" . | nindent 8 }} {{- include "dolphinscheduler.sharedStorage.volume" . | nindent 8 }}
{{- include "dolphinscheduler.fsFileResource.volume" . | nindent 8 }} {{- include "dolphinscheduler.fsFileResource.volume" . | nindent 8 }}
{{- include "dolphinscheduler.ldap.ssl.volume" . | nindent 8 }} {{- include "dolphinscheduler.ldap.ssl.volume" . | nindent 8 }}

View File

@ -109,9 +109,11 @@ spec:
volumeMounts: volumeMounts:
- mountPath: "/opt/dolphinscheduler/logs" - mountPath: "/opt/dolphinscheduler/logs"
name: {{ include "dolphinscheduler.fullname" . }}-master name: {{ include "dolphinscheduler.fullname" . }}-master
{{- if .Values.master.enableCustomizedConfig }}
- name: master-config-volume - name: master-config-volume
mountPath: /opt/dolphinscheduler/conf/application.yaml mountPath: /opt/dolphinscheduler/conf/application.yaml
subPath: application.yaml subPath: application.yaml
{{- end }}
{{- include "dolphinscheduler.sharedStorage.volumeMount" . | nindent 12 }} {{- include "dolphinscheduler.sharedStorage.volumeMount" . | nindent 12 }}
- name: config-volume - name: config-volume
mountPath: /opt/dolphinscheduler/conf/common.properties mountPath: /opt/dolphinscheduler/conf/common.properties
@ -125,9 +127,11 @@ spec:
{{- else }} {{- else }}
emptyDir: {} emptyDir: {}
{{- end }} {{- end }}
{{- if .Values.master.enableCustomizedConfig }}
- name: master-config-volume - name: master-config-volume
configMap: configMap:
name: { { include "dolphinscheduler.fullname" . } }-master name: {{ include "dolphinscheduler.fullname" . }}-master
{{- end }}
{{- include "dolphinscheduler.sharedStorage.volume" . | nindent 8 }} {{- include "dolphinscheduler.sharedStorage.volume" . | nindent 8 }}
- name: config-volume - name: config-volume
configMap: configMap:

View File

@ -111,9 +111,11 @@ spec:
name: {{ include "dolphinscheduler.fullname" . }}-worker-data name: {{ include "dolphinscheduler.fullname" . }}-worker-data
- mountPath: "/opt/dolphinscheduler/logs" - mountPath: "/opt/dolphinscheduler/logs"
name: {{ include "dolphinscheduler.fullname" . }}-worker-logs name: {{ include "dolphinscheduler.fullname" . }}-worker-logs
{{- if .Values.worker.enableCustomizedConfig }}
- name: worker-config-volume - name: worker-config-volume
mountPath: /opt/dolphinscheduler/conf/application.yaml mountPath: /opt/dolphinscheduler/conf/application.yaml
subPath: application.yaml subPath: application.yaml
{{- end }}
- name: config-volume - name: config-volume
mountPath: /opt/dolphinscheduler/conf/common.properties mountPath: /opt/dolphinscheduler/conf/common.properties
subPath: common.properties subPath: common.properties
@ -142,9 +144,11 @@ spec:
- name: {{ include "dolphinscheduler.fullname" . }}-worker-logs - name: {{ include "dolphinscheduler.fullname" . }}-worker-logs
emptyDir: {} emptyDir: {}
{{- end }} {{- end }}
{{- if .Values.worker.enableCustomizedConfig }}
- name: worker-config-volume - name: worker-config-volume
configMap: configMap:
name: { { include "dolphinscheduler.fullname" . } }-worker name: {{ include "dolphinscheduler.fullname" . }}-worker
{{- end }}
- name: config-volume - name: config-volume
configMap: configMap:
name: {{ include "dolphinscheduler.fullname" . }}-configs name: {{ include "dolphinscheduler.fullname" . }}-configs

View File

@ -49,6 +49,10 @@ image:
# -- tools image # -- tools image
tools: dolphinscheduler-tools tools: dolphinscheduler-tools
datasource:
# -- The profile of datasource
profile: postgresql
postgresql: postgresql:
# -- If not exists external PostgreSQL, by default, the DolphinScheduler will use a internal PostgreSQL # -- If not exists external PostgreSQL, by default, the DolphinScheduler will use a internal PostgreSQL
enabled: true enabled: true
@ -440,7 +444,19 @@ master:
# requests: # requests:
# memory: "2Gi" # memory: "2Gi"
# cpu: "500m" # cpu: "500m"
# -- enable configure custom config
enableCustomizedConfig: false
# -- configure aligned with https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-master/src/main/resources/application.yaml
customizedConfig: { }
# customizedConfig:
# application.yaml: |
# profiles:
# active: postgresql
# banner:
# charset: UTF-8
# jackson:
# time-zone: UTC
# date-format: "yyyy-MM-dd HH:mm:ss"
# -- Periodic probe of container liveness. Container will be restarted if the probe fails. # -- Periodic probe of container liveness. Container will be restarted if the probe fails.
# More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) # More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes)
livenessProbe: livenessProbe:
@ -569,6 +585,17 @@ worker:
# -- Periodic probe of container liveness. Container will be restarted if the probe fails. # -- Periodic probe of container liveness. Container will be restarted if the probe fails.
# More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) # More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes)
# -- enable configure custom config
enableCustomizedConfig: false
# -- configure aligned with https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-worker/src/main/resources/application.yaml
customizedConfig: { }
# customizedConfig:
# application.yaml: |
# banner:
# charset: UTF-8
# jackson:
# time-zone: UTC
# date-format: "yyyy-MM-dd HH:mm:ss"
livenessProbe: livenessProbe:
# -- Turn on and off liveness probe # -- Turn on and off liveness probe
enabled: true enabled: true
@ -733,7 +760,19 @@ alert:
# requests: # requests:
# memory: "1Gi" # memory: "1Gi"
# cpu: "500m" # cpu: "500m"
# -- enable configure custom config
enableCustomizedConfig: false
# -- configure aligned with https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/resources/application.yaml
customizedConfig: { }
# customizedConfig:
# application.yaml: |
# profiles:
# active: postgresql
# banner:
# charset: UTF-8
# jackson:
# time-zone: UTC
# date-format: "yyyy-MM-dd HH:mm:ss"
# -- Periodic probe of container liveness. Container will be restarted if the probe fails. # -- Periodic probe of container liveness. Container will be restarted if the probe fails.
# More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) # More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes)
livenessProbe: livenessProbe:
@ -833,7 +872,19 @@ api:
# requests: # requests:
# memory: "1Gi" # memory: "1Gi"
# cpu: "500m" # cpu: "500m"
# -- enable configure custom config
enableCustomizedConfig: false
# -- configure aligned with https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-api/src/main/resources/application.yaml
customizedConfig: { }
# customizedConfig:
# application.yaml: |
# profiles:
# active: postgresql
# banner:
# charset: UTF-8
# jackson:
# time-zone: UTC
# date-format: "yyyy-MM-dd HH:mm:ss"
# -- Periodic probe of container liveness. Container will be restarted if the probe fails. # -- Periodic probe of container liveness. Container will be restarted if the probe fails.
# More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) # More info: [container-probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes)
livenessProbe: livenessProbe:

View File

@ -14,21 +14,7 @@ Configure all the configurations refer to [pseudo-cluster deployment](pseudo-clu
### Modify Configuration ### Modify Configuration
This step differs quite a lot from [pseudo-cluster deployment](pseudo-cluster.md), because the deployment script transfers the required resources for installation to each deployment machine by using `scp`. So we only need to modify the configuration of the machine that runs `install.sh` script and configurations will dispatch to cluster by `scp`. The configuration file is under the path `bin/env/install_env.sh`, here we only need to modify section **INSTALL MACHINE**, **DolphinScheduler ENV, Database, Registry Server** and keep other sections the same as [pseudo-cluster deployment](pseudo-cluster .md), the following describes the parameters that must be modified: This step differs quite a lot from [pseudo-cluster deployment](pseudo-cluster.md), please use `scp` or other methods to distribute the configuration files to each machine, then modify the configuration files.
```shell
# ---------------------------------------------------------
# INSTALL MACHINE
# ---------------------------------------------------------
# Using IP or machine hostname for the server going to deploy master, worker, API server, the IP of the server
# If you using a hostname, make sure machines could connect each other by hostname
# As below, the hostname of the machine deploying DolphinScheduler is ds1, ds2, ds3, ds4, ds5, where ds1, ds2 install the master server, ds3, ds4, and ds5 installs worker server, the alert server is installed in ds4, and the API server is installed in ds5
ips="ds1,ds2,ds3,ds4,ds5"
masters="ds1,ds2"
workers="ds3:default,ds4:default,ds5:default"
alertServer="ds4"
apiServers="ds5"
```
## Start and Login DolphinScheduler ## Start and Login DolphinScheduler

View File

@ -71,31 +71,7 @@ Go to the ZooKeeper installation directory, copy configure file `zoo_sample.cfg`
## Modify Configuration ## Modify Configuration
After completing the preparation of the basic environment, you need to modify the configuration file according to the After completing the preparation of the basic environment, you need to modify the configuration file according to the
environment you used. Change the environment configurations via `export <ENV_NAME>=<VALUE>`. The configuration files are located in directory `bin/env` as `install_env.sh` and `dolphinscheduler_env.sh`. environment you used. Change the environment configurations via `export <ENV_NAME>=<VALUE>`. The configuration files are located in directory `bin/env` as `dolphinscheduler_env.sh`.
### Modify `install_env.sh`
File `install_env.sh` describes which machines will be installed DolphinScheduler and what server will be installed on
each machine. You could find this file in the path `bin/env/install_env.sh` and the detail of the configuration as below.
```shell
# ---------------------------------------------------------
# INSTALL MACHINE
# ---------------------------------------------------------
# Due to the master, worker, and API server being deployed on a single node, the IP of the server is the machine IP or localhost
ips="localhost"
sshPort="22"
masters="localhost"
workers="localhost:default"
alertServer="localhost"
apiServers="localhost"
# DolphinScheduler installation path, it will auto-create if not exists
installPath=~/dolphinscheduler
# Deploy user, use the user you create in section **Configure machine SSH password-free login**
deployUser="dolphinscheduler"
```
### Modify `dolphinscheduler_env.sh` ### Modify `dolphinscheduler_env.sh`
@ -146,11 +122,7 @@ Follow the instructions in [datasource-setting](../howto/datasource-setting.md)
## Start DolphinScheduler ## Start DolphinScheduler
Use **deployment user** you created above, running the following command to complete the deployment, and the server log will be stored in the logs folder. Use **deployment user** you created above, running the command to complete the deployment, and the server log will be stored in the logs folder.
```shell
bash ./bin/install.sh
```
> **_Note:_** For the first time deployment, there maybe occur five times of `sh: bin/dolphinscheduler-daemon.sh: No such file or directory` in the terminal, > **_Note:_** For the first time deployment, there maybe occur five times of `sh: bin/dolphinscheduler-daemon.sh: No such file or directory` in the terminal,
> this is non-important information that you can ignore. > this is non-important information that you can ignore.

View File

@ -14,21 +14,7 @@
### 修改相关配置 ### 修改相关配置
这个是与[伪集群部署](pseudo-cluster.md)差异较大的一步,因为部署脚本会通过 `scp` 的方式将安装需要的资源传输到各个机器上,所以这一步我们仅需要修改运行`install.sh`脚本的所在机器的配置即可。配置文件在路径在`bin/env/install_env.sh`下,此处我们仅需修改**INSTALL MACHINE****DolphinScheduler ENV、Database、Registry Server**与伪集群部署保持一致,下面对必须修改参数进行说明 这个是与[伪集群部署](pseudo-cluster.md)差异较大的一步,请使用 scp 等方式将配置文件分发到各台机器上,然后修改配置文件
```shell
# ---------------------------------------------------------
# INSTALL MACHINE
# ---------------------------------------------------------
# 需要配置master、worker、API server所在服务器的IP均为机器IP或者localhost
# 如果是配置hostname的话需要保证机器间可以通过hostname相互链接
# 如下图所示,部署 DolphinScheduler 机器的 hostname 为 ds1,ds2,ds3,ds4,ds5其中 ds1,ds2 安装 master 服务ds3,ds4,ds5安装 worker 服务alert server安装在ds4中api server 安装在ds5中
ips="ds1,ds2,ds3,ds4,ds5"
masters="ds1,ds2"
workers="ds3:default,ds4:default,ds5:default"
alertServer="ds4"
apiServers="ds5"
```
## 启动 DolphinScheduler && 登录 DolphinScheduler && 启停服务 ## 启动 DolphinScheduler && 登录 DolphinScheduler && 启停服务

View File

@ -70,30 +70,7 @@ chmod 600 ~/.ssh/authorized_keys
## 修改相关配置 ## 修改相关配置
完成基础环境的准备后,需要根据你的机器环境修改配置文件。配置文件可以在目录 `bin/env` 中找到,他们分别是 并命名为 `install_env.sh``dolphinscheduler_env.sh` 完成基础环境的准备后,需要根据你的机器环境修改配置文件。配置文件可以在目录 `bin/env/dolphinscheduler_env.sh` 中找到。
### 修改 `install_env.sh` 文件
文件 `install_env.sh` 描述了哪些机器将被安装 DolphinScheduler 以及每台机器对应安装哪些服务。您可以在路径 `bin/env/install_env.sh` 中找到此文件,可通过以下方式更改 env 变量,export <ENV_NAME>=<VALUE>,配置详情如下。
```shell
# ---------------------------------------------------------
# INSTALL MACHINE
# ---------------------------------------------------------
# Due to the master, worker, and API server being deployed on a single node, the IP of the server is the machine IP or localhost
ips="localhost"
sshPort="22"
masters="localhost"
workers="localhost:default"
alertServer="localhost"
apiServers="localhost"
# DolphinScheduler installation path, it will auto-create if not exists
installPath=~/dolphinscheduler
# Deploy user, use the user you create in section **Configure machine SSH password-free login**
deployUser="dolphinscheduler"
```
### 修改 `dolphinscheduler_env.sh` 文件 ### 修改 `dolphinscheduler_env.sh` 文件
@ -141,11 +118,7 @@ export PATH=$HADOOP_HOME/bin:$SPARK_HOME/bin:$PYTHON_LAUNCHER:$JAVA_HOME/bin:$HI
## 启动 DolphinScheduler ## 启动 DolphinScheduler
使用上面创建的**部署用户**运行以下命令完成部署,部署后的运行日志将存放在 logs 文件夹内 使用上面创建的**部署用户**运行命令完成部署,部署后的运行日志将存放在 logs 文件夹内
```shell
bash ./bin/install.sh
```
> **_注意:_** 第一次部署的话,可能出现 5 次`sh: bin/dolphinscheduler-daemon.sh: No such file or directory`相关信息,此为非重要信息直接忽略即可 > **_注意:_** 第一次部署的话,可能出现 5 次`sh: bin/dolphinscheduler-daemon.sh: No such file or directory`相关信息,此为非重要信息直接忽略即可

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-aliyunVoice</artifactId> <artifactId>dolphinscheduler-alert-aliyunVoice</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-all</artifactId> <artifactId>dolphinscheduler-alert-all</artifactId>

View File

@ -17,13 +17,12 @@
~ specific language governing permissions and limitations ~ specific language governing permissions and limitations
~ under the License. ~ under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-api</artifactId> <artifactId>dolphinscheduler-alert-api</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-dingtalk</artifactId> <artifactId>dolphinscheduler-alert-dingtalk</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-email</artifactId> <artifactId>dolphinscheduler-alert-email</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-feishu</artifactId> <artifactId>dolphinscheduler-alert-feishu</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-http</artifactId> <artifactId>dolphinscheduler-alert-http</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-pagerduty</artifactId> <artifactId>dolphinscheduler-alert-pagerduty</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-prometheus</artifactId> <artifactId>dolphinscheduler-alert-prometheus</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-script</artifactId> <artifactId>dolphinscheduler-alert-script</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-slack</artifactId> <artifactId>dolphinscheduler-alert-slack</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-telegram</artifactId> <artifactId>dolphinscheduler-alert-telegram</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-webexteams</artifactId> <artifactId>dolphinscheduler-alert-webexteams</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-wechat</artifactId> <artifactId>dolphinscheduler-alert-wechat</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert</artifactId> <artifactId>dolphinscheduler-alert</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<packaging>pom</packaging> <packaging>pom</packaging>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert</artifactId> <artifactId>dolphinscheduler-alert</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-server</artifactId> <artifactId>dolphinscheduler-alert-server</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -17,13 +17,12 @@
~ specific language governing permissions and limitations ~ specific language governing permissions and limitations
~ under the License. ~ under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert</artifactId> <artifactId>dolphinscheduler-alert</artifactId>
<packaging>pom</packaging> <packaging>pom</packaging>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-aop</artifactId> <artifactId>dolphinscheduler-aop</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-api</artifactId> <artifactId>dolphinscheduler-api</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -19,11 +19,11 @@ package org.apache.dolphinscheduler.api.dto.task;
import static org.apache.dolphinscheduler.common.constants.Constants.VERSION_FIRST; import static org.apache.dolphinscheduler.common.constants.Constants.VERSION_FIRST;
import org.apache.dolphinscheduler.common.constants.Constants;
import org.apache.dolphinscheduler.common.enums.Flag; import org.apache.dolphinscheduler.common.enums.Flag;
import org.apache.dolphinscheduler.common.enums.Priority; import org.apache.dolphinscheduler.common.enums.Priority;
import org.apache.dolphinscheduler.common.enums.TimeoutFlag; import org.apache.dolphinscheduler.common.enums.TimeoutFlag;
import org.apache.dolphinscheduler.dao.entity.TaskDefinition; import org.apache.dolphinscheduler.dao.entity.TaskDefinition;
import org.apache.dolphinscheduler.dao.utils.WorkerGroupUtils;
import org.apache.dolphinscheduler.plugin.task.api.enums.TaskTimeoutStrategy; import org.apache.dolphinscheduler.plugin.task.api.enums.TaskTimeoutStrategy;
import java.util.Date; import java.util.Date;
@ -107,7 +107,7 @@ public class TaskCreateRequest {
taskDefinition.setProjectCode(this.projectCode); taskDefinition.setProjectCode(this.projectCode);
taskDefinition.setTaskType(this.taskType); taskDefinition.setTaskType(this.taskType);
taskDefinition.setTaskParams(this.taskParams); taskDefinition.setTaskParams(this.taskParams);
taskDefinition.setWorkerGroup(this.workerGroup == null ? Constants.DEFAULT_WORKER_GROUP : this.workerGroup); taskDefinition.setWorkerGroup(WorkerGroupUtils.getWorkerGroupOrDefault(workerGroup));
taskDefinition.setEnvironmentCode(this.environmentCode); taskDefinition.setEnvironmentCode(this.environmentCode);
taskDefinition.setFailRetryTimes(this.failRetryTimes); taskDefinition.setFailRetryTimes(this.failRetryTimes);
taskDefinition.setFailRetryInterval(this.failRetryInterval); taskDefinition.setFailRetryInterval(this.failRetryInterval);

View File

@ -18,7 +18,6 @@
package org.apache.dolphinscheduler.api.service; package org.apache.dolphinscheduler.api.service;
import org.apache.dolphinscheduler.api.utils.Result; import org.apache.dolphinscheduler.api.utils.Result;
import org.apache.dolphinscheduler.dao.entity.TaskInstance;
import org.apache.dolphinscheduler.dao.entity.User; import org.apache.dolphinscheduler.dao.entity.User;
import java.util.List; import java.util.List;
@ -77,13 +76,6 @@ public interface WorkerGroupService {
*/ */
Map<String, Object> getWorkerAddressList(); Map<String, Object> getWorkerAddressList();
/**
* Get task instance's worker group
* @param taskInstance task instance
* @return worker group
*/
String getTaskWorkerGroup(TaskInstance taskInstance);
/** /**
* Query worker group by process definition codes * Query worker group by process definition codes
* @param processDefinitionCodeList processDefinitionCodeList * @param processDefinitionCodeList processDefinitionCodeList

View File

@ -34,7 +34,6 @@ import static org.apache.dolphinscheduler.api.enums.Status.PROCESS_DEFINE_NOT_EX
import static org.apache.dolphinscheduler.common.constants.CommandKeyConstants.CMD_PARAM_SUB_PROCESS_DEFINE_CODE; import static org.apache.dolphinscheduler.common.constants.CommandKeyConstants.CMD_PARAM_SUB_PROCESS_DEFINE_CODE;
import static org.apache.dolphinscheduler.common.constants.Constants.COPY_SUFFIX; import static org.apache.dolphinscheduler.common.constants.Constants.COPY_SUFFIX;
import static org.apache.dolphinscheduler.common.constants.Constants.DATA_LIST; import static org.apache.dolphinscheduler.common.constants.Constants.DATA_LIST;
import static org.apache.dolphinscheduler.common.constants.Constants.DEFAULT_WORKER_GROUP;
import static org.apache.dolphinscheduler.common.constants.Constants.GLOBAL_PARAMS; import static org.apache.dolphinscheduler.common.constants.Constants.GLOBAL_PARAMS;
import static org.apache.dolphinscheduler.common.constants.Constants.IMPORT_SUFFIX; import static org.apache.dolphinscheduler.common.constants.Constants.IMPORT_SUFFIX;
import static org.apache.dolphinscheduler.common.constants.Constants.LOCAL_PARAMS; import static org.apache.dolphinscheduler.common.constants.Constants.LOCAL_PARAMS;
@ -109,6 +108,7 @@ import org.apache.dolphinscheduler.dao.model.PageListingResult;
import org.apache.dolphinscheduler.dao.repository.ProcessDefinitionDao; import org.apache.dolphinscheduler.dao.repository.ProcessDefinitionDao;
import org.apache.dolphinscheduler.dao.repository.ProcessDefinitionLogDao; import org.apache.dolphinscheduler.dao.repository.ProcessDefinitionLogDao;
import org.apache.dolphinscheduler.dao.repository.TaskDefinitionLogDao; import org.apache.dolphinscheduler.dao.repository.TaskDefinitionLogDao;
import org.apache.dolphinscheduler.dao.utils.WorkerGroupUtils;
import org.apache.dolphinscheduler.plugin.task.api.TaskPluginManager; import org.apache.dolphinscheduler.plugin.task.api.TaskPluginManager;
import org.apache.dolphinscheduler.plugin.task.api.enums.SqlType; import org.apache.dolphinscheduler.plugin.task.api.enums.SqlType;
import org.apache.dolphinscheduler.plugin.task.api.enums.TaskTimeoutStrategy; import org.apache.dolphinscheduler.plugin.task.api.enums.TaskTimeoutStrategy;
@ -1390,7 +1390,7 @@ public class ProcessDefinitionServiceImpl extends BaseServiceImpl implements Pro
taskDefinition.setFailRetryTimes(0); taskDefinition.setFailRetryTimes(0);
taskDefinition.setFailRetryInterval(0); taskDefinition.setFailRetryInterval(0);
taskDefinition.setTimeoutFlag(TimeoutFlag.CLOSE); taskDefinition.setTimeoutFlag(TimeoutFlag.CLOSE);
taskDefinition.setWorkerGroup(DEFAULT_WORKER_GROUP); taskDefinition.setWorkerGroup(WorkerGroupUtils.getDefaultWorkerGroup());
taskDefinition.setTaskPriority(Priority.MEDIUM); taskDefinition.setTaskPriority(Priority.MEDIUM);
taskDefinition.setEnvironmentCode(-1); taskDefinition.setEnvironmentCode(-1);
taskDefinition.setTimeout(0); taskDefinition.setTimeout(0);

View File

@ -32,6 +32,7 @@ import org.apache.dolphinscheduler.dao.mapper.ProjectWorkerGroupMapper;
import org.apache.dolphinscheduler.dao.mapper.ScheduleMapper; import org.apache.dolphinscheduler.dao.mapper.ScheduleMapper;
import org.apache.dolphinscheduler.dao.mapper.TaskDefinitionMapper; import org.apache.dolphinscheduler.dao.mapper.TaskDefinitionMapper;
import org.apache.dolphinscheduler.dao.mapper.WorkerGroupMapper; import org.apache.dolphinscheduler.dao.mapper.WorkerGroupMapper;
import org.apache.dolphinscheduler.dao.utils.WorkerGroupUtils;
import org.apache.commons.collections.CollectionUtils; import org.apache.commons.collections.CollectionUtils;
import org.apache.commons.collections4.SetUtils; import org.apache.commons.collections4.SetUtils;
@ -118,7 +119,7 @@ public class ProjectWorkerGroupRelationServiceImpl extends BaseServiceImpl
workerGroupMapper.queryAllWorkerGroup().stream().map(WorkerGroup::getName).collect( workerGroupMapper.queryAllWorkerGroup().stream().map(WorkerGroup::getName).collect(
Collectors.toSet()); Collectors.toSet());
workerGroupNames.add(Constants.DEFAULT_WORKER_GROUP); workerGroupNames.add(WorkerGroupUtils.getDefaultWorkerGroup());
Set<String> assignedWorkerGroupNames = new HashSet<>(workerGroups); Set<String> assignedWorkerGroupNames = new HashSet<>(workerGroups);

View File

@ -17,7 +17,6 @@
package org.apache.dolphinscheduler.api.service.impl; package org.apache.dolphinscheduler.api.service.impl;
import static org.apache.dolphinscheduler.api.utils.CheckUtils.checkFilePath;
import static org.apache.dolphinscheduler.common.constants.Constants.ALIAS; import static org.apache.dolphinscheduler.common.constants.Constants.ALIAS;
import static org.apache.dolphinscheduler.common.constants.Constants.CONTENT; import static org.apache.dolphinscheduler.common.constants.Constants.CONTENT;
import static org.apache.dolphinscheduler.common.constants.Constants.EMPTY_STRING; import static org.apache.dolphinscheduler.common.constants.Constants.EMPTY_STRING;
@ -1291,10 +1290,6 @@ public class ResourcesServiceImpl extends BaseServiceImpl implements ResourcesSe
if (FOLDER_SEPARATOR.equalsIgnoreCase(fullName)) { if (FOLDER_SEPARATOR.equalsIgnoreCase(fullName)) {
return; return;
} }
// abnormal characters check
if (!checkFilePath(fullName)) {
throw new ServiceException(Status.ILLEGAL_RESOURCE_PATH);
}
// Avoid returning to the parent directory // Avoid returning to the parent directory
if (fullName.contains("../")) { if (fullName.contains("../")) {
throw new ServiceException(Status.ILLEGAL_RESOURCE_PATH, fullName); throw new ServiceException(Status.ILLEGAL_RESOURCE_PATH, fullName);

View File

@ -32,7 +32,6 @@ import org.apache.dolphinscheduler.dao.entity.EnvironmentWorkerGroupRelation;
import org.apache.dolphinscheduler.dao.entity.ProcessInstance; import org.apache.dolphinscheduler.dao.entity.ProcessInstance;
import org.apache.dolphinscheduler.dao.entity.Schedule; import org.apache.dolphinscheduler.dao.entity.Schedule;
import org.apache.dolphinscheduler.dao.entity.TaskDefinition; import org.apache.dolphinscheduler.dao.entity.TaskDefinition;
import org.apache.dolphinscheduler.dao.entity.TaskInstance;
import org.apache.dolphinscheduler.dao.entity.User; import org.apache.dolphinscheduler.dao.entity.User;
import org.apache.dolphinscheduler.dao.entity.WorkerGroup; import org.apache.dolphinscheduler.dao.entity.WorkerGroup;
import org.apache.dolphinscheduler.dao.mapper.EnvironmentWorkerGroupRelationMapper; import org.apache.dolphinscheduler.dao.mapper.EnvironmentWorkerGroupRelationMapper;
@ -41,6 +40,7 @@ import org.apache.dolphinscheduler.dao.mapper.ProcessInstanceMapper;
import org.apache.dolphinscheduler.dao.mapper.ScheduleMapper; import org.apache.dolphinscheduler.dao.mapper.ScheduleMapper;
import org.apache.dolphinscheduler.dao.mapper.TaskDefinitionMapper; import org.apache.dolphinscheduler.dao.mapper.TaskDefinitionMapper;
import org.apache.dolphinscheduler.dao.mapper.WorkerGroupMapper; import org.apache.dolphinscheduler.dao.mapper.WorkerGroupMapper;
import org.apache.dolphinscheduler.dao.utils.WorkerGroupUtils;
import org.apache.dolphinscheduler.registry.api.RegistryClient; import org.apache.dolphinscheduler.registry.api.RegistryClient;
import org.apache.dolphinscheduler.registry.api.enums.RegistryNodeType; import org.apache.dolphinscheduler.registry.api.enums.RegistryNodeType;
import org.apache.dolphinscheduler.service.process.ProcessService; import org.apache.dolphinscheduler.service.process.ProcessService;
@ -357,12 +357,12 @@ public class WorkerGroupServiceImpl extends BaseServiceImpl implements WorkerGro
workerGroups = workerGroupMapper.queryAllWorkerGroup(); workerGroups = workerGroupMapper.queryAllWorkerGroup();
} }
boolean containDefaultWorkerGroups = workerGroups.stream() boolean containDefaultWorkerGroups = workerGroups.stream()
.anyMatch(workerGroup -> Constants.DEFAULT_WORKER_GROUP.equals(workerGroup.getName())); .anyMatch(workerGroup -> WorkerGroupUtils.isWorkerGroupEmpty(workerGroup.getName()));
if (!containDefaultWorkerGroups) { if (!containDefaultWorkerGroups) {
// there doesn't exist a default WorkerGroup, we will add all worker to the default worker group. // there doesn't exist a default WorkerGroup, we will add all worker to the default worker group.
Set<String> activeWorkerNodes = registryClient.getServerNodeSet(RegistryNodeType.WORKER); Set<String> activeWorkerNodes = registryClient.getServerNodeSet(RegistryNodeType.WORKER);
WorkerGroup defaultWorkerGroup = new WorkerGroup(); WorkerGroup defaultWorkerGroup = new WorkerGroup();
defaultWorkerGroup.setName(Constants.DEFAULT_WORKER_GROUP); defaultWorkerGroup.setName(WorkerGroupUtils.getDefaultWorkerGroup());
defaultWorkerGroup.setAddrList(String.join(Constants.COMMA, activeWorkerNodes)); defaultWorkerGroup.setAddrList(String.join(Constants.COMMA, activeWorkerNodes));
defaultWorkerGroup.setCreateTime(new Date()); defaultWorkerGroup.setCreateTime(new Date());
defaultWorkerGroup.setUpdateTime(new Date()); defaultWorkerGroup.setUpdateTime(new Date());
@ -431,27 +431,6 @@ public class WorkerGroupServiceImpl extends BaseServiceImpl implements WorkerGro
return result; return result;
} }
@Override
public String getTaskWorkerGroup(TaskInstance taskInstance) {
if (taskInstance == null) {
return null;
}
String workerGroup = taskInstance.getWorkerGroup();
if (StringUtils.isNotEmpty(workerGroup)) {
return workerGroup;
}
int processInstanceId = taskInstance.getProcessInstanceId();
ProcessInstance processInstance = processService.findProcessInstanceById(processInstanceId);
if (processInstance != null) {
return processInstance.getWorkerGroup();
}
log.info("task : {} will use default worker group", taskInstance.getId());
return Constants.DEFAULT_WORKER_GROUP;
}
@Override @Override
public Map<Long, String> queryWorkerGroupByProcessDefinitionCodes(List<Long> processDefinitionCodeList) { public Map<Long, String> queryWorkerGroupByProcessDefinitionCodes(List<Long> processDefinitionCodeList) {
List<Schedule> processDefinitionScheduleList = List<Schedule> processDefinitionScheduleList =

View File

@ -158,14 +158,4 @@ public class CheckUtils {
return pattern.matcher(str).matches(); return pattern.matcher(str).matches();
} }
/**
* regex FilePath check,only use a to z, A to Z, 0 to 9, and _./-
*
* @param str input string
* @return true if regex pattern is right, otherwise return false
*/
public static boolean checkFilePath(String str) {
return regexChecks(str, Constants.REGEX_FILE_PATH);
}
} }

View File

@ -75,6 +75,8 @@ spring:
pathmatch: pathmatch:
matching-strategy: ANT_PATH_MATCHER matching-strategy: ANT_PATH_MATCHER
static-path-pattern: /static/** static-path-pattern: /static/**
cloud.discovery.client.composite-indicator.enabled: false
springdoc: springdoc:
swagger-ui: swagger-ui:
path: /swagger-ui.html path: /swagger-ui.html

View File

@ -67,6 +67,7 @@ import org.apache.dolphinscheduler.dao.mapper.TaskDefinitionMapper;
import org.apache.dolphinscheduler.dao.mapper.TaskGroupQueueMapper; import org.apache.dolphinscheduler.dao.mapper.TaskGroupQueueMapper;
import org.apache.dolphinscheduler.dao.mapper.TenantMapper; import org.apache.dolphinscheduler.dao.mapper.TenantMapper;
import org.apache.dolphinscheduler.dao.repository.ProcessInstanceDao; import org.apache.dolphinscheduler.dao.repository.ProcessInstanceDao;
import org.apache.dolphinscheduler.dao.utils.WorkerGroupUtils;
import org.apache.dolphinscheduler.registry.api.enums.RegistryNodeType; import org.apache.dolphinscheduler.registry.api.enums.RegistryNodeType;
import org.apache.dolphinscheduler.service.command.CommandService; import org.apache.dolphinscheduler.service.command.CommandService;
import org.apache.dolphinscheduler.service.process.ProcessService; import org.apache.dolphinscheduler.service.process.ProcessService;
@ -271,7 +272,7 @@ public class ExecuteFunctionServiceTest {
null, null, null, null,
null, null, null, null, null, null,
RunMode.RUN_MODE_SERIAL, RunMode.RUN_MODE_SERIAL,
Priority.LOW, Constants.DEFAULT_WORKER_GROUP, tenantCode, 100L, 10, null, null, Priority.LOW, WorkerGroupUtils.getDefaultWorkerGroup(), tenantCode, 100L, 10, null, null,
Constants.DRY_RUN_FLAG_NO, Constants.DRY_RUN_FLAG_NO,
Constants.TEST_FLAG_NO, Constants.TEST_FLAG_NO,
ComplementDependentMode.OFF_MODE, null, ComplementDependentMode.OFF_MODE, null,
@ -298,7 +299,7 @@ public class ExecuteFunctionServiceTest {
null, "123456789,987654321", null, "123456789,987654321",
null, null, null, null, null, null,
RunMode.RUN_MODE_SERIAL, RunMode.RUN_MODE_SERIAL,
Priority.LOW, Constants.DEFAULT_WORKER_GROUP, tenantCode, 100L, 110, null, null, Priority.LOW, WorkerGroupUtils.getDefaultWorkerGroup(), tenantCode, 100L, 110, null, null,
Constants.DRY_RUN_FLAG_NO, Constants.DRY_RUN_FLAG_NO,
Constants.TEST_FLAG_NO, Constants.TEST_FLAG_NO,
ComplementDependentMode.OFF_MODE, null, ComplementDependentMode.OFF_MODE, null,
@ -323,7 +324,7 @@ public class ExecuteFunctionServiceTest {
null, "1123456789,987654321", null, "1123456789,987654321",
null, null, null, null, null, null,
RunMode.RUN_MODE_SERIAL, RunMode.RUN_MODE_SERIAL,
Priority.LOW, Constants.DEFAULT_WORKER_GROUP, tenantCode, 100L, 110, null, 0, Priority.LOW, WorkerGroupUtils.getDefaultWorkerGroup(), tenantCode, 100L, 110, null, 0,
Constants.DRY_RUN_FLAG_NO, Constants.DRY_RUN_FLAG_NO,
Constants.TEST_FLAG_NO, Constants.TEST_FLAG_NO,
ComplementDependentMode.OFF_MODE, null, ComplementDependentMode.OFF_MODE, null,
@ -354,14 +355,14 @@ public class ExecuteFunctionServiceTest {
dependentProcessDefinition.setProcessDefinitionCode(2); dependentProcessDefinition.setProcessDefinitionCode(2);
dependentProcessDefinition.setProcessDefinitionVersion(1); dependentProcessDefinition.setProcessDefinitionVersion(1);
dependentProcessDefinition.setTaskDefinitionCode(1); dependentProcessDefinition.setTaskDefinitionCode(1);
dependentProcessDefinition.setWorkerGroup(Constants.DEFAULT_WORKER_GROUP); dependentProcessDefinition.setWorkerGroup(WorkerGroupUtils.getDefaultWorkerGroup());
dependentProcessDefinition.setTaskParams( dependentProcessDefinition.setTaskParams(
"{\"localParams\":[],\"resourceList\":[],\"dependence\":{\"relation\":\"AND\",\"dependTaskList\":[{\"relation\":\"AND\",\"dependItemList\":[{\"depTaskCode\":2,\"status\":\"SUCCESS\"}]}]},\"conditionResult\":{\"successNode\":[1],\"failedNode\":[1]}}"); "{\"localParams\":[],\"resourceList\":[],\"dependence\":{\"relation\":\"AND\",\"dependTaskList\":[{\"relation\":\"AND\",\"dependItemList\":[{\"depTaskCode\":2,\"status\":\"SUCCESS\"}]}]},\"conditionResult\":{\"successNode\":[1],\"failedNode\":[1]}}");
Mockito.when(processService.queryDependentProcessDefinitionByProcessDefinitionCode(processDefinitionCode)) Mockito.when(processService.queryDependentProcessDefinitionByProcessDefinitionCode(processDefinitionCode))
.thenReturn(Lists.newArrayList(dependentProcessDefinition)); .thenReturn(Lists.newArrayList(dependentProcessDefinition));
Map<Long, String> processDefinitionWorkerGroupMap = new HashMap<>(); Map<Long, String> processDefinitionWorkerGroupMap = new HashMap<>();
processDefinitionWorkerGroupMap.put(1L, Constants.DEFAULT_WORKER_GROUP); processDefinitionWorkerGroupMap.put(1L, WorkerGroupUtils.getDefaultWorkerGroup());
Mockito.when(workerGroupService.queryWorkerGroupByProcessDefinitionCodes(Lists.newArrayList(1L))) Mockito.when(workerGroupService.queryWorkerGroupByProcessDefinitionCodes(Lists.newArrayList(1L)))
.thenReturn(processDefinitionWorkerGroupMap); .thenReturn(processDefinitionWorkerGroupMap);
@ -370,7 +371,7 @@ public class ExecuteFunctionServiceTest {
command.setCommandType(CommandType.COMPLEMENT_DATA); command.setCommandType(CommandType.COMPLEMENT_DATA);
command.setCommandParam( command.setCommandParam(
"{\"StartNodeList\":\"1\",\"complementStartDate\":\"2020-01-01 00:00:00\",\"complementEndDate\":\"2020-01-31 23:00:00\"}"); "{\"StartNodeList\":\"1\",\"complementStartDate\":\"2020-01-01 00:00:00\",\"complementEndDate\":\"2020-01-31 23:00:00\"}");
command.setWorkerGroup(Constants.DEFAULT_WORKER_GROUP); command.setWorkerGroup(WorkerGroupUtils.getDefaultWorkerGroup());
command.setProcessDefinitionCode(processDefinitionCode); command.setProcessDefinitionCode(processDefinitionCode);
command.setExecutorId(1); command.setExecutorId(1);
@ -383,7 +384,7 @@ public class ExecuteFunctionServiceTest {
childDependent.setProcessDefinitionCode(3); childDependent.setProcessDefinitionCode(3);
childDependent.setProcessDefinitionVersion(1); childDependent.setProcessDefinitionVersion(1);
childDependent.setTaskDefinitionCode(4); childDependent.setTaskDefinitionCode(4);
childDependent.setWorkerGroup(Constants.DEFAULT_WORKER_GROUP); childDependent.setWorkerGroup(WorkerGroupUtils.getDefaultWorkerGroup());
childDependent.setTaskParams( childDependent.setTaskParams(
"{\"localParams\":[],\"resourceList\":[],\"dependence\":{\"relation\":\"AND\",\"dependTaskList\":[{\"relation\":\"AND\",\"dependItemList\":[{\"depTaskCode\":3,\"status\":\"SUCCESS\"}]}]},\"conditionResult\":{\"successNode\":[1],\"failedNode\":[1]}}"); "{\"localParams\":[],\"resourceList\":[],\"dependence\":{\"relation\":\"AND\",\"dependTaskList\":[{\"relation\":\"AND\",\"dependItemList\":[{\"depTaskCode\":3,\"status\":\"SUCCESS\"}]}]},\"conditionResult\":{\"successNode\":[1],\"failedNode\":[1]}}");
Mockito.when(processService.queryDependentProcessDefinitionByProcessDefinitionCode( Mockito.when(processService.queryDependentProcessDefinitionByProcessDefinitionCode(
@ -409,7 +410,8 @@ public class ExecuteFunctionServiceTest {
null, null, null, null,
null, null, null, null, null, null,
RunMode.RUN_MODE_SERIAL, RunMode.RUN_MODE_SERIAL,
Priority.LOW, Constants.DEFAULT_WORKER_GROUP, tenantCode, 100L, 110, null, 2, Constants.DRY_RUN_FLAG_NO, Priority.LOW, WorkerGroupUtils.getDefaultWorkerGroup(), tenantCode, 100L, 110, null, 2,
Constants.DRY_RUN_FLAG_NO,
Constants.TEST_FLAG_NO, Constants.TEST_FLAG_NO,
ComplementDependentMode.OFF_MODE, null, ComplementDependentMode.OFF_MODE, null,
false, false,
@ -434,7 +436,7 @@ public class ExecuteFunctionServiceTest {
null, null, null, null,
null, null, null, null, null, null,
RunMode.RUN_MODE_SERIAL, RunMode.RUN_MODE_SERIAL,
Priority.LOW, Constants.DEFAULT_WORKER_GROUP, tenantCode, 100L, 110, null, null, Priority.LOW, WorkerGroupUtils.getDefaultWorkerGroup(), tenantCode, 100L, 110, null, null,
Constants.DRY_RUN_FLAG_NO, Constants.DRY_RUN_FLAG_NO,
Constants.TEST_FLAG_NO, Constants.TEST_FLAG_NO,
ComplementDependentMode.OFF_MODE, null, ComplementDependentMode.OFF_MODE, null,
@ -460,7 +462,8 @@ public class ExecuteFunctionServiceTest {
null, null, null, null,
null, null, null, null, null, null,
RunMode.RUN_MODE_PARALLEL, RunMode.RUN_MODE_PARALLEL,
Priority.LOW, Constants.DEFAULT_WORKER_GROUP, tenantCode, 100L, 110, null, 2, Constants.DRY_RUN_FLAG_NO, Priority.LOW, WorkerGroupUtils.getDefaultWorkerGroup(), tenantCode, 100L, 110, null, 2,
Constants.DRY_RUN_FLAG_NO,
Constants.TEST_FLAG_NO, Constants.TEST_FLAG_NO,
ComplementDependentMode.OFF_MODE, null, ComplementDependentMode.OFF_MODE, null,
false, false,
@ -486,7 +489,7 @@ public class ExecuteFunctionServiceTest {
null, null, null, null,
null, null, null, null, null, null,
RunMode.RUN_MODE_PARALLEL, RunMode.RUN_MODE_PARALLEL,
Priority.LOW, Constants.DEFAULT_WORKER_GROUP, tenantCode, 100L, 110, null, 15, Priority.LOW, WorkerGroupUtils.getDefaultWorkerGroup(), tenantCode, 100L, 110, null, 15,
Constants.DRY_RUN_FLAG_NO, Constants.DRY_RUN_FLAG_NO,
Constants.TEST_FLAG_NO, Constants.TEST_FLAG_NO,
ComplementDependentMode.OFF_MODE, null, ComplementDependentMode.OFF_MODE, null,
@ -514,7 +517,7 @@ public class ExecuteFunctionServiceTest {
null, null,
RunMode.RUN_MODE_PARALLEL, RunMode.RUN_MODE_PARALLEL,
Priority.LOW, Priority.LOW,
Constants.DEFAULT_WORKER_GROUP, WorkerGroupUtils.getDefaultWorkerGroup(),
tenantCode, tenantCode,
100L, 100L,
110, 110,
@ -553,7 +556,7 @@ public class ExecuteFunctionServiceTest {
null, null, null, null,
null, null, 0, null, null, 0,
RunMode.RUN_MODE_PARALLEL, RunMode.RUN_MODE_PARALLEL,
Priority.LOW, Constants.DEFAULT_WORKER_GROUP, tenantCode, 100L, 110, null, 15, Priority.LOW, WorkerGroupUtils.getDefaultWorkerGroup(), tenantCode, 100L, 110, null, 15,
Constants.DRY_RUN_FLAG_NO, Constants.DRY_RUN_FLAG_NO,
Constants.TEST_FLAG_YES, Constants.TEST_FLAG_YES,
ComplementDependentMode.OFF_MODE, null, ComplementDependentMode.OFF_MODE, null,

View File

@ -72,6 +72,7 @@ import org.apache.dolphinscheduler.dao.model.PageListingResult;
import org.apache.dolphinscheduler.dao.repository.ProcessDefinitionDao; import org.apache.dolphinscheduler.dao.repository.ProcessDefinitionDao;
import org.apache.dolphinscheduler.dao.repository.ProcessDefinitionLogDao; import org.apache.dolphinscheduler.dao.repository.ProcessDefinitionLogDao;
import org.apache.dolphinscheduler.dao.repository.TaskDefinitionLogDao; import org.apache.dolphinscheduler.dao.repository.TaskDefinitionLogDao;
import org.apache.dolphinscheduler.dao.utils.WorkerGroupUtils;
import org.apache.dolphinscheduler.service.alert.ListenerEventAlertManager; import org.apache.dolphinscheduler.service.alert.ListenerEventAlertManager;
import org.apache.dolphinscheduler.service.process.ProcessService; import org.apache.dolphinscheduler.service.process.ProcessService;
import org.apache.dolphinscheduler.spi.enums.DbType; import org.apache.dolphinscheduler.spi.enums.DbType;
@ -1143,7 +1144,7 @@ public class ProcessDefinitionServiceTest extends BaseServiceTestTool {
schedule.setProcessInstancePriority(Priority.MEDIUM); schedule.setProcessInstancePriority(Priority.MEDIUM);
schedule.setWarningType(WarningType.NONE); schedule.setWarningType(WarningType.NONE);
schedule.setWarningGroupId(1); schedule.setWarningGroupId(1);
schedule.setWorkerGroup(Constants.DEFAULT_WORKER_GROUP); schedule.setWorkerGroup(WorkerGroupUtils.getDefaultWorkerGroup());
return schedule; return schedule;
} }

View File

@ -30,7 +30,6 @@ import org.apache.dolphinscheduler.common.constants.Constants;
import org.apache.dolphinscheduler.common.enums.AuthorizationType; import org.apache.dolphinscheduler.common.enums.AuthorizationType;
import org.apache.dolphinscheduler.common.enums.UserType; import org.apache.dolphinscheduler.common.enums.UserType;
import org.apache.dolphinscheduler.dao.entity.ProcessInstance; import org.apache.dolphinscheduler.dao.entity.ProcessInstance;
import org.apache.dolphinscheduler.dao.entity.TaskInstance;
import org.apache.dolphinscheduler.dao.entity.User; import org.apache.dolphinscheduler.dao.entity.User;
import org.apache.dolphinscheduler.dao.entity.WorkerGroup; import org.apache.dolphinscheduler.dao.entity.WorkerGroup;
import org.apache.dolphinscheduler.dao.mapper.EnvironmentWorkerGroupRelationMapper; import org.apache.dolphinscheduler.dao.mapper.EnvironmentWorkerGroupRelationMapper;
@ -65,8 +64,6 @@ import org.slf4j.LoggerFactory;
@MockitoSettings(strictness = Strictness.LENIENT) @MockitoSettings(strictness = Strictness.LENIENT)
public class WorkerGroupServiceTest { public class WorkerGroupServiceTest {
private static final Logger logger = LoggerFactory.getLogger(WorkerGroupServiceTest.class);
private static final Logger baseServiceLogger = LoggerFactory.getLogger(BaseServiceImpl.class); private static final Logger baseServiceLogger = LoggerFactory.getLogger(BaseServiceImpl.class);
private static final Logger serviceLogger = LoggerFactory.getLogger(WorkerGroupService.class); private static final Logger serviceLogger = LoggerFactory.getLogger(WorkerGroupService.class);
@ -288,47 +285,6 @@ public class WorkerGroupServiceTest {
Assertions.assertEquals("default", workerGroups.toArray()[0]); Assertions.assertEquals("default", workerGroups.toArray()[0]);
} }
@Test
public void giveNull_whenGetTaskWorkerGroup_expectNull() {
String nullWorkerGroup = workerGroupService.getTaskWorkerGroup(null);
Assertions.assertNull(nullWorkerGroup);
}
@Test
public void giveCorrectTaskInstance_whenGetTaskWorkerGroup_expectTaskWorkerGroup() {
TaskInstance taskInstance = new TaskInstance();
taskInstance.setId(1);
taskInstance.setWorkerGroup("cluster1");
String workerGroup = workerGroupService.getTaskWorkerGroup(taskInstance);
Assertions.assertEquals("cluster1", workerGroup);
}
@Test
public void giveNullWorkerGroup_whenGetTaskWorkerGroup_expectProcessWorkerGroup() {
TaskInstance taskInstance = new TaskInstance();
taskInstance.setId(1);
taskInstance.setProcessInstanceId(1);
ProcessInstance processInstance = new ProcessInstance();
processInstance.setId(1);
processInstance.setWorkerGroup("cluster1");
Mockito.when(processService.findProcessInstanceById(1)).thenReturn(processInstance);
String workerGroup = workerGroupService.getTaskWorkerGroup(taskInstance);
Assertions.assertEquals("cluster1", workerGroup);
}
@Test
public void giveNullTaskAndProcessWorkerGroup_whenGetTaskWorkerGroup_expectDefault() {
TaskInstance taskInstance = new TaskInstance();
taskInstance.setId(1);
taskInstance.setProcessInstanceId(1);
Mockito.when(processService.findProcessInstanceById(1)).thenReturn(null);
String defaultWorkerGroup = workerGroupService.getTaskWorkerGroup(taskInstance);
Assertions.assertEquals(Constants.DEFAULT_WORKER_GROUP, defaultWorkerGroup);
}
/** /**
* get Group * get Group
*/ */

View File

@ -92,24 +92,4 @@ public class CheckUtilsTest {
Assertions.assertTrue(CheckUtils.checkPhone("17362537263")); Assertions.assertTrue(CheckUtils.checkPhone("17362537263"));
} }
/**
* check file path
*/
@Test
public void testCheckFilePath() {
// true
Assertions.assertTrue(CheckUtils.checkFilePath("/"));
Assertions.assertTrue(CheckUtils.checkFilePath("xx/"));
Assertions.assertTrue(CheckUtils.checkFilePath("/xx"));
Assertions.assertTrue(CheckUtils.checkFilePath("14567134578654"));
Assertions.assertTrue(CheckUtils.checkFilePath("/admin/root/"));
Assertions.assertTrue(CheckUtils.checkFilePath("/admin/root/1531531..13513/153135.."));
// false
Assertions.assertFalse(CheckUtils.checkFilePath(null));
Assertions.assertFalse(CheckUtils.checkFilePath("file://xxx/ss"));
Assertions.assertFalse(CheckUtils.checkFilePath("/xxx/ss;/dasd/123"));
Assertions.assertFalse(CheckUtils.checkFilePath("/xxx/ss && /dasd/123"));
Assertions.assertFalse(CheckUtils.checkFilePath("/xxx/ss || /dasd/123"));
}
} }

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-bom</artifactId> <artifactId>dolphinscheduler-bom</artifactId>
<packaging>pom</packaging> <packaging>pom</packaging>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-common</artifactId> <artifactId>dolphinscheduler-common</artifactId>

View File

@ -47,6 +47,7 @@ public final class Constants {
public static final String RESOURCE_TYPE_UDF = "udfs"; public static final String RESOURCE_TYPE_UDF = "udfs";
public static final String EMPTY_STRING = ""; public static final String EMPTY_STRING = "";
public static final int MAX_FILE_SIZE = 1024 * 1024 * 1024;
/** /**
* resource.hdfs.fs.defaultFS * resource.hdfs.fs.defaultFS
@ -249,11 +250,6 @@ public final class Constants {
*/ */
public static final Pattern REGEX_USER_NAME = Pattern.compile("^[a-zA-Z0-9._-]{3,39}$"); public static final Pattern REGEX_USER_NAME = Pattern.compile("^[a-zA-Z0-9._-]{3,39}$");
/**
* file path regex
*/
public static final Pattern REGEX_FILE_PATH = Pattern.compile("^[a-zA-Z0-9_./-]+$");
/** /**
* read permission * read permission
*/ */
@ -537,14 +533,9 @@ public final class Constants {
* session timeout * session timeout
*/ */
public static final int SESSION_TIME_OUT = 7200; public static final int SESSION_TIME_OUT = 7200;
public static final int MAX_FILE_SIZE = 1024 * 1024 * 1024;
public static final String UDF = "UDF"; public static final String UDF = "UDF";
public static final String CLASS = "class"; public static final String CLASS = "class";
/**
* default worker group
*/
public static final String DEFAULT_WORKER_GROUP = "default";
/** /**
* authorize writable perm * authorize writable perm
*/ */

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao-plugin</artifactId> <artifactId>dolphinscheduler-dao-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-dao-api</artifactId> <artifactId>dolphinscheduler-dao-api</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao-plugin</artifactId> <artifactId>dolphinscheduler-dao-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-dao-h2</artifactId> <artifactId>dolphinscheduler-dao-h2</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao-plugin</artifactId> <artifactId>dolphinscheduler-dao-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-dao-mysql</artifactId> <artifactId>dolphinscheduler-dao-mysql</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao-plugin</artifactId> <artifactId>dolphinscheduler-dao-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-dao-plugin-all</artifactId> <artifactId>dolphinscheduler-dao-plugin-all</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao-plugin</artifactId> <artifactId>dolphinscheduler-dao-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-dao-postgresql</artifactId> <artifactId>dolphinscheduler-dao-postgresql</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-dao-plugin</artifactId> <artifactId>dolphinscheduler-dao-plugin</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-dao</artifactId> <artifactId>dolphinscheduler-dao</artifactId>
<name>${project.artifactId}</name> <name>${project.artifactId}</name>

View File

@ -31,11 +31,15 @@ import org.apache.dolphinscheduler.common.enums.TaskExecuteType;
import org.apache.dolphinscheduler.common.utils.DateUtils; import org.apache.dolphinscheduler.common.utils.DateUtils;
import org.apache.dolphinscheduler.common.utils.JSONUtils; import org.apache.dolphinscheduler.common.utils.JSONUtils;
import org.apache.dolphinscheduler.plugin.task.api.enums.TaskExecutionStatus; import org.apache.dolphinscheduler.plugin.task.api.enums.TaskExecutionStatus;
import org.apache.dolphinscheduler.plugin.task.api.parameters.ConditionsParameters;
import org.apache.dolphinscheduler.plugin.task.api.parameters.DependentParameters; import org.apache.dolphinscheduler.plugin.task.api.parameters.DependentParameters;
import org.apache.dolphinscheduler.plugin.task.api.parameters.SwitchParameters; import org.apache.dolphinscheduler.plugin.task.api.parameters.SwitchParameters;
import org.apache.commons.lang3.StringUtils;
import java.io.Serializable; import java.io.Serializable;
import java.util.Date; import java.util.Date;
import java.util.HashMap;
import java.util.Map; import java.util.Map;
import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeUnit;
@ -192,6 +196,9 @@ public class TaskInstance implements Serializable {
@TableField(exist = false) @TableField(exist = false)
private DependentParameters dependency; private DependentParameters dependency;
@TableField(exist = false)
private ConditionsParameters conditionsParameters;
/** /**
* switch dependency * switch dependency
*/ */
@ -318,6 +325,43 @@ public class TaskInstance implements Serializable {
this.dependency = dependency; this.dependency = dependency;
} }
public ConditionsParameters getConditionsParameters() {
if (this.conditionsParameters == null) {
Map<String, Object> taskParamsMap =
JSONUtils.parseObject(this.getTaskParams(), new TypeReference<Map<String, Object>>() {
});
this.conditionsParameters =
JSONUtils.parseObject((String) taskParamsMap.get(Constants.DEPENDENCE), ConditionsParameters.class);
}
return conditionsParameters;
}
public ConditionsParameters.ConditionResult getConditionResult() {
Map<String, Object> taskParamsMap =
JSONUtils.parseObject(this.getTaskParams(), new TypeReference<Map<String, Object>>() {
});
String conditionResult = (String) taskParamsMap.getOrDefault(Constants.CONDITION_RESULT, "");
if (StringUtils.isNotEmpty(conditionResult)) {
return JSONUtils.parseObject(conditionResult, new TypeReference<ConditionsParameters.ConditionResult>() {
});
}
return null;
}
public void setConditionResult(ConditionsParameters conditionsParameters) {
if (conditionsParameters == null) {
return;
}
Map<String, Object> taskParamsMap =
JSONUtils.parseObject(this.getTaskParams(), new TypeReference<Map<String, Object>>() {
});
if (taskParamsMap == null) {
taskParamsMap = new HashMap<>();
}
taskParamsMap.put(Constants.CONDITION_RESULT, JSONUtils.toJsonString(conditionsParameters));
this.setTaskParams(JSONUtils.toJsonString(taskParamsMap));
}
public SwitchParameters getSwitchDependency() { public SwitchParameters getSwitchDependency() {
// todo: We need to directly use Jackson to deserialize the taskParam, rather than parse the map and get from // todo: We need to directly use Jackson to deserialize the taskParam, rather than parse the map and get from
// field. // field.

View File

@ -0,0 +1,54 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.dao.utils;
public class EnvironmentUtils {
private static final long EMPTY_ENVIRONMENT_CODE = -1L;
/**
* Check if the environment code is empty (we should use null instead of -1, this is used to comply with the original code)
*
* @return true if the environment code is empty, false otherwise
*/
public static boolean isEnvironmentCodeEmpty(Long environmentCode) {
return environmentCode == null || environmentCode <= 0;
}
/**
* Get the empty environment code
*/
public static Long getDefaultEnvironmentCode() {
return EMPTY_ENVIRONMENT_CODE;
}
/**
* Get the environment code or the default environment code if the environment code is empty
*/
public static Long getEnvironmentCodeOrDefault(Long environmentCode) {
return getEnvironmentCodeOrDefault(environmentCode, getDefaultEnvironmentCode());
}
/**
* Get the environment code or the default environment code if the environment code is empty
*/
public static Long getEnvironmentCodeOrDefault(Long environmentCode, Long defaultEnvironmentCode) {
return isEnvironmentCodeEmpty(environmentCode) ? defaultEnvironmentCode : environmentCode;
}
}

View File

@ -0,0 +1,45 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.dao.utils;
import org.apache.commons.lang3.StringUtils;
public class WorkerGroupUtils {
private static final String DEFAULT_WORKER_GROUP = "default";
/**
* Check if the worker group is empty, if the worker group is default, it is considered empty
*/
public static boolean isWorkerGroupEmpty(String workerGroup) {
return StringUtils.isEmpty(workerGroup) || getDefaultWorkerGroup().equals(workerGroup);
}
public static String getWorkerGroupOrDefault(String workerGroup) {
return getWorkerGroupOrDefault(workerGroup, getDefaultWorkerGroup());
}
public static String getWorkerGroupOrDefault(String workerGroup, String defaultWorkerGroup) {
return isWorkerGroupEmpty(workerGroup) ? defaultWorkerGroup : workerGroup;
}
public static String getDefaultWorkerGroup() {
return DEFAULT_WORKER_GROUP;
}
}

View File

@ -30,6 +30,9 @@ END IF;
END; END;
d// d//
-- If the admin account is not associated with a tenant, the admin's tenant will be set to the default tenant.
UPDATE `t_ds_user` SET `tenant_id` = '-1' WHERE (`user_name` = 'admin') AND (`tenant_id` = '0');
delimiter ; delimiter ;
CALL dolphin_t_ds_tenant_insert_default(); CALL dolphin_t_ds_tenant_insert_default();
DROP PROCEDURE dolphin_t_ds_tenant_insert_default; DROP PROCEDURE dolphin_t_ds_tenant_insert_default;

View File

@ -21,6 +21,9 @@ INSERT INTO t_ds_tenant(id, tenant_code, description, queue_id, create_time, upd
UPDATE t_ds_schedules as t1 SET tenant_code = COALESCE(t3.tenant_code, 'default') FROM t_ds_process_definition as t2 LEFT JOIN t_ds_tenant t3 ON t2.tenant_id = t3.id WHERE t1.process_definition_code = t2.code; UPDATE t_ds_schedules as t1 SET tenant_code = COALESCE(t3.tenant_code, 'default') FROM t_ds_process_definition as t2 LEFT JOIN t_ds_tenant t3 ON t2.tenant_id = t3.id WHERE t1.process_definition_code = t2.code;
UPDATE t_ds_process_instance SET tenant_code = 'default' WHERE tenant_code IS NULL; UPDATE t_ds_process_instance SET tenant_code = 'default' WHERE tenant_code IS NULL;
-- If the admin account is not associated with a tenant, the admin's tenant will be set to the default tenant.
UPDATE t_ds_user SET tenant_id = '-1' WHERE (user_name = 'admin') AND (tenant_id = '0');
-- data quality support choose database -- data quality support choose database
INSERT INTO t_ds_dq_rule_input_entry INSERT INTO t_ds_dq_rule_input_entry
(id, field, "type", title, value, "options", placeholder, option_source_type, value_type, input_type, is_show, can_edit, is_emit, is_validate, create_time, update_time) (id, field, "type", title, value, "options", placeholder, option_source_type, value_type, input_type, is_show, can_edit, is_emit, is_validate, create_time, update_time)

View File

@ -19,7 +19,6 @@ package org.apache.dolphinscheduler.dao.mapper;
import static com.google.common.truth.Truth.assertThat; import static com.google.common.truth.Truth.assertThat;
import org.apache.dolphinscheduler.common.constants.Constants;
import org.apache.dolphinscheduler.common.enums.CommandType; import org.apache.dolphinscheduler.common.enums.CommandType;
import org.apache.dolphinscheduler.common.enums.FailureStrategy; import org.apache.dolphinscheduler.common.enums.FailureStrategy;
import org.apache.dolphinscheduler.common.enums.Flag; import org.apache.dolphinscheduler.common.enums.Flag;
@ -32,6 +31,7 @@ import org.apache.dolphinscheduler.dao.BaseDaoTest;
import org.apache.dolphinscheduler.dao.entity.Command; import org.apache.dolphinscheduler.dao.entity.Command;
import org.apache.dolphinscheduler.dao.entity.CommandCount; import org.apache.dolphinscheduler.dao.entity.CommandCount;
import org.apache.dolphinscheduler.dao.entity.ProcessDefinition; import org.apache.dolphinscheduler.dao.entity.ProcessDefinition;
import org.apache.dolphinscheduler.dao.utils.WorkerGroupUtils;
import java.util.Date; import java.util.Date;
import java.util.HashMap; import java.util.HashMap;
@ -303,7 +303,7 @@ public class CommandMapperTest extends BaseDaoTest {
command.setProcessInstancePriority(Priority.MEDIUM); command.setProcessInstancePriority(Priority.MEDIUM);
command.setStartTime(DateUtils.stringToDate("2019-12-29 10:10:00")); command.setStartTime(DateUtils.stringToDate("2019-12-29 10:10:00"));
command.setUpdateTime(DateUtils.stringToDate("2019-12-29 10:10:00")); command.setUpdateTime(DateUtils.stringToDate("2019-12-29 10:10:00"));
command.setWorkerGroup(Constants.DEFAULT_WORKER_GROUP); command.setWorkerGroup(WorkerGroupUtils.getDefaultWorkerGroup());
command.setProcessInstanceId(0); command.setProcessInstanceId(0);
command.setProcessDefinitionVersion(0); command.setProcessDefinitionVersion(0);
commandMapper.insert(command); commandMapper.insert(command);

View File

@ -19,7 +19,6 @@ package org.apache.dolphinscheduler.dao.repository.impl;
import static com.google.common.truth.Truth.assertThat; import static com.google.common.truth.Truth.assertThat;
import org.apache.dolphinscheduler.common.constants.Constants;
import org.apache.dolphinscheduler.common.enums.CommandType; import org.apache.dolphinscheduler.common.enums.CommandType;
import org.apache.dolphinscheduler.common.enums.FailureStrategy; import org.apache.dolphinscheduler.common.enums.FailureStrategy;
import org.apache.dolphinscheduler.common.enums.Priority; import org.apache.dolphinscheduler.common.enums.Priority;
@ -29,6 +28,7 @@ import org.apache.dolphinscheduler.common.utils.DateUtils;
import org.apache.dolphinscheduler.dao.BaseDaoTest; import org.apache.dolphinscheduler.dao.BaseDaoTest;
import org.apache.dolphinscheduler.dao.entity.Command; import org.apache.dolphinscheduler.dao.entity.Command;
import org.apache.dolphinscheduler.dao.repository.CommandDao; import org.apache.dolphinscheduler.dao.repository.CommandDao;
import org.apache.dolphinscheduler.dao.utils.WorkerGroupUtils;
import org.apache.commons.lang3.RandomUtils; import org.apache.commons.lang3.RandomUtils;
@ -80,7 +80,7 @@ class CommandDaoImplTest extends BaseDaoTest {
command.setProcessInstancePriority(Priority.MEDIUM); command.setProcessInstancePriority(Priority.MEDIUM);
command.setStartTime(DateUtils.stringToDate("2019-12-29 10:10:00")); command.setStartTime(DateUtils.stringToDate("2019-12-29 10:10:00"));
command.setUpdateTime(DateUtils.stringToDate("2019-12-29 10:10:00")); command.setUpdateTime(DateUtils.stringToDate("2019-12-29 10:10:00"));
command.setWorkerGroup(Constants.DEFAULT_WORKER_GROUP); command.setWorkerGroup(WorkerGroupUtils.getDefaultWorkerGroup());
command.setProcessInstanceId(0); command.setProcessInstanceId(0);
command.setProcessDefinitionVersion(0); command.setProcessDefinitionVersion(0);
commandDao.insert(command); commandDao.insert(command);

View File

@ -0,0 +1,72 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.dao.utils;
import static com.google.common.truth.Truth.assertThat;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import org.junit.jupiter.params.provider.ValueSource;
class EnvironmentUtilsTest {
@ParameterizedTest
@ValueSource(longs = {0, -1})
void testIsEnvironmentCodeEmpty_emptyEnvironmentCode(Long environmentCode) {
assertThat(EnvironmentUtils.isEnvironmentCodeEmpty(environmentCode)).isTrue();
}
@ParameterizedTest
@ValueSource(longs = {123})
void testIsEnvironmentCodeEmpty_nonEmptyEnvironmentCode(Long environmentCode) {
assertThat(EnvironmentUtils.isEnvironmentCodeEmpty(environmentCode)).isFalse();
}
@Test
void testGetDefaultEnvironmentCode() {
assertThat(EnvironmentUtils.getDefaultEnvironmentCode()).isEqualTo(-1L);
}
@ParameterizedTest
@ValueSource(longs = {0, -1})
void testGetEnvironmentCodeOrDefault_emptyEnvironmentCode(Long environmentCode) {
assertThat(EnvironmentUtils.getEnvironmentCodeOrDefault(environmentCode)).isEqualTo(-1L);
}
@ParameterizedTest
@ValueSource(longs = {123})
void testGetEnvironmentCodeOrDefault_nonEmptyEnvironmentCode(Long environmentCode) {
assertThat(EnvironmentUtils.getEnvironmentCodeOrDefault(environmentCode)).isEqualTo(environmentCode);
}
@ParameterizedTest
@CsvSource(value = {",123", "-1,123"})
void testGetEnvironmentCodeOrDefault_withDefaultValue_emptyEnvironmentCode(Long environmentCode,
Long defaultValue) {
assertThat(EnvironmentUtils.getEnvironmentCodeOrDefault(environmentCode, defaultValue)).isEqualTo(defaultValue);
}
@ParameterizedTest
@CsvSource(value = {"1,123"})
void testGetEnvironmentCodeOrDefault_withDefaultValue_nonEmptyEnvironmentCode(Long environmentCode,
Long defaultValue) {
assertThat(EnvironmentUtils.getEnvironmentCodeOrDefault(environmentCode, defaultValue))
.isEqualTo(environmentCode);
}
}

View File

@ -0,0 +1,70 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.dao.utils;
import static com.google.common.truth.Truth.assertThat;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import org.junit.jupiter.params.provider.ValueSource;
class WorkerGroupUtilsTest {
@ParameterizedTest
@ValueSource(strings = {"", "default"})
void testIsWorkerGroupEmpty_emptyWorkerGroup(String workerGroup) {
assertThat(WorkerGroupUtils.isWorkerGroupEmpty(workerGroup)).isTrue();
}
@ParameterizedTest
@ValueSource(strings = {"123", "default1"})
void testIsWorkerGroupEmpty_nonEmptyWorkerGroup(String workerGroup) {
assertThat(WorkerGroupUtils.isWorkerGroupEmpty(workerGroup)).isFalse();
}
@ParameterizedTest
@ValueSource(strings = {"", "default"})
void testGetWorkerGroupOrDefault_emptyWorkerGroup(String workerGroup) {
assertThat(WorkerGroupUtils.getWorkerGroupOrDefault(workerGroup))
.isEqualTo(WorkerGroupUtils.getDefaultWorkerGroup());
}
@ParameterizedTest
@ValueSource(strings = {"test"})
void testGetWorkerGroupOrDefault_nonEmptyWorkerGroup(String workerGroup) {
assertThat(WorkerGroupUtils.getWorkerGroupOrDefault(workerGroup)).isEqualTo(workerGroup);
}
@ParameterizedTest
@CsvSource(value = {",test", "default,test"})
void testGetWorkerGroupOrDefault_withDefaultValue_emptyWorkerGroup(String workerGroup, String defaultValue) {
assertThat(WorkerGroupUtils.getWorkerGroupOrDefault(workerGroup, defaultValue)).isEqualTo(defaultValue);
}
@ParameterizedTest
@CsvSource(value = {"test1,test"})
void testGetWorkerGroupOrDefault_withDefaultValue_nonEmptyWorkerGroup(String workerGroup, String defaultValue) {
assertThat(WorkerGroupUtils.getWorkerGroupOrDefault(workerGroup)).isEqualTo(workerGroup);
}
@Test
void getDefaultWorkerGroup() {
assertThat(WorkerGroupUtils.getDefaultWorkerGroup()).isEqualTo("default");
}
}

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-data-quality</artifactId> <artifactId>dolphinscheduler-data-quality</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-all</artifactId> <artifactId>dolphinscheduler-datasource-all</artifactId>
@ -148,11 +147,6 @@
<artifactId>dolphinscheduler-datasource-sagemaker</artifactId> <artifactId>dolphinscheduler-datasource-sagemaker</artifactId>
<version>${project.version}</version> <version>${project.version}</version>
</dependency> </dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-k8s</artifactId>
<version>${project.version}</version>
</dependency>
<dependency> <dependency>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-hana</artifactId> <artifactId>dolphinscheduler-datasource-hana</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-api</artifactId> <artifactId>dolphinscheduler-datasource-api</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-athena</artifactId> <artifactId>dolphinscheduler-datasource-athena</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-azure-sql</artifactId> <artifactId>dolphinscheduler-datasource-azure-sql</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-clickhouse</artifactId> <artifactId>dolphinscheduler-datasource-clickhouse</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-dameng</artifactId> <artifactId>dolphinscheduler-datasource-dameng</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-databend</artifactId> <artifactId>dolphinscheduler-datasource-databend</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-db2</artifactId> <artifactId>dolphinscheduler-datasource-db2</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-doris</artifactId> <artifactId>dolphinscheduler-datasource-doris</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-hana</artifactId> <artifactId>dolphinscheduler-datasource-hana</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-hive</artifactId> <artifactId>dolphinscheduler-datasource-hive</artifactId>

View File

@ -1,50 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-k8s</artifactId>
<packaging>jar</packaging>
<name>${project.artifactId}</name>
<dependencies>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-spi</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-api</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-client</artifactId>
</dependency>
</dependencies>
</project>

View File

@ -1,55 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.datasource.k8s;
import lombok.extern.slf4j.Slf4j;
import io.fabric8.kubernetes.api.model.NamespaceList;
import io.fabric8.kubernetes.client.Config;
import io.fabric8.kubernetes.client.KubernetesClient;
import io.fabric8.kubernetes.client.KubernetesClientBuilder;
@Slf4j
public class K8sClientWrapper implements AutoCloseable {
private KubernetesClient client;
public K8sClientWrapper() {
}
public boolean checkConnect(String kubeConfigYaml, String namespace) {
try {
Config config = Config.fromKubeconfig(kubeConfigYaml);
client = new KubernetesClientBuilder().withConfig(config).build();
NamespaceList namespaceList = client.namespaces().list();
if (!namespaceList.getItems().stream().anyMatch(ns -> ns.getMetadata().getName().equals(namespace))) {
log.info("failed to connect to the K8S cluster, namespace not found\n");
return false;
}
log.info("successfully connected to the K8S cluster");
return true;
} catch (Exception e) {
log.info("failed to connect to the K8S cluster\n");
return false;
}
}
@Override
public void close() throws Exception {
}
}

View File

@ -1,37 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.datasource.k8s;
import org.apache.dolphinscheduler.spi.datasource.AdHocDataSourceClient;
import org.apache.dolphinscheduler.spi.datasource.BaseConnectionParam;
import org.apache.dolphinscheduler.spi.datasource.DataSourceChannel;
import org.apache.dolphinscheduler.spi.datasource.PooledDataSourceClient;
import org.apache.dolphinscheduler.spi.enums.DbType;
public class K8sDataSourceChannel implements DataSourceChannel {
@Override
public AdHocDataSourceClient createAdHocDataSourceClient(BaseConnectionParam baseConnectionParam, DbType dbType) {
throw new UnsupportedOperationException("K8S AdHocDataSourceClient is not supported");
}
@Override
public PooledDataSourceClient createPooledDataSourceClient(BaseConnectionParam baseConnectionParam, DbType dbType) {
throw new UnsupportedOperationException("K8S AdHocDataSourceClient is not supported");
}
}

View File

@ -1,39 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.datasource.k8s;
import org.apache.dolphinscheduler.spi.datasource.DataSourceChannel;
import org.apache.dolphinscheduler.spi.datasource.DataSourceChannelFactory;
import org.apache.dolphinscheduler.spi.enums.DbType;
import com.google.auto.service.AutoService;
@AutoService(DataSourceChannelFactory.class)
public class K8sDataSourceChannelFactory implements DataSourceChannelFactory {
@Override
public DataSourceChannel create() {
return new K8sDataSourceChannel();
}
@Override
public String getName() {
return DbType.K8S.getName();
}
}

View File

@ -1,32 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.datasource.k8s.param;
import org.apache.dolphinscheduler.spi.datasource.ConnectionParam;
import lombok.Data;
import com.fasterxml.jackson.annotation.JsonInclude;
@Data
@JsonInclude(JsonInclude.Include.NON_NULL)
public class K8sConnectionParam implements ConnectionParam {
protected String kubeConfig;
protected String namespace;
}

View File

@ -1,35 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.datasource.k8s.param;
import org.apache.dolphinscheduler.plugin.datasource.api.datasource.BaseDataSourceParamDTO;
import org.apache.dolphinscheduler.spi.enums.DbType;
import lombok.Data;
@Data
public class K8sDataSourceParamDTO extends BaseDataSourceParamDTO {
protected String kubeConfig;
protected String namespace;
@Override
public DbType getType() {
return DbType.K8S;
}
}

View File

@ -1,129 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.datasource.k8s.param;
import org.apache.dolphinscheduler.common.utils.JSONUtils;
import org.apache.dolphinscheduler.plugin.datasource.api.datasource.AbstractDataSourceProcessor;
import org.apache.dolphinscheduler.plugin.datasource.api.datasource.BaseDataSourceParamDTO;
import org.apache.dolphinscheduler.plugin.datasource.api.datasource.DataSourceProcessor;
import org.apache.dolphinscheduler.plugin.datasource.api.utils.PasswordUtils;
import org.apache.dolphinscheduler.plugin.datasource.k8s.K8sClientWrapper;
import org.apache.dolphinscheduler.spi.datasource.ConnectionParam;
import org.apache.dolphinscheduler.spi.enums.DbType;
import org.apache.commons.lang3.StringUtils;
import java.io.IOException;
import java.sql.Connection;
import java.sql.SQLException;
import java.text.MessageFormat;
import lombok.extern.slf4j.Slf4j;
import com.google.auto.service.AutoService;
@AutoService(DataSourceProcessor.class)
@Slf4j
public class K8sDataSourceProcessor extends AbstractDataSourceProcessor {
@Override
public BaseDataSourceParamDTO castDatasourceParamDTO(String paramJson) {
return JSONUtils.parseObject(paramJson, K8sDataSourceParamDTO.class);
}
@Override
public void checkDatasourceParam(BaseDataSourceParamDTO datasourceParam) {
K8sDataSourceParamDTO k8sDataSourceParamDTO = (K8sDataSourceParamDTO) datasourceParam;
if (StringUtils.isEmpty(k8sDataSourceParamDTO.getKubeConfig())) {
throw new IllegalArgumentException("sagemaker datasource param is not valid");
}
}
@Override
public String getDatasourceUniqueId(ConnectionParam connectionParam, DbType dbType) {
K8sConnectionParam baseConnectionParam = (K8sConnectionParam) connectionParam;
return MessageFormat.format("{0}@{1}@{2}", dbType.getName(),
PasswordUtils.encodePassword(baseConnectionParam.getKubeConfig()), baseConnectionParam.getNamespace());
}
@Override
public BaseDataSourceParamDTO createDatasourceParamDTO(String connectionJson) {
K8sConnectionParam connectionParams = (K8sConnectionParam) createConnectionParams(connectionJson);
K8sDataSourceParamDTO k8sDataSourceParamDTO = new K8sDataSourceParamDTO();
k8sDataSourceParamDTO.setKubeConfig(connectionParams.getKubeConfig());
k8sDataSourceParamDTO.setNamespace(connectionParams.getNamespace());
return k8sDataSourceParamDTO;
}
@Override
public K8sConnectionParam createConnectionParams(BaseDataSourceParamDTO datasourceParam) {
K8sDataSourceParamDTO k8sDataSourceParam = (K8sDataSourceParamDTO) datasourceParam;
K8sConnectionParam k8sConnectionParam = new K8sConnectionParam();
k8sConnectionParam.setKubeConfig(k8sDataSourceParam.getKubeConfig());
k8sConnectionParam.setNamespace(k8sDataSourceParam.getNamespace());
return k8sConnectionParam;
}
@Override
public ConnectionParam createConnectionParams(String connectionJson) {
return JSONUtils.parseObject(connectionJson, K8sConnectionParam.class);
}
@Override
public String getDatasourceDriver() {
return "";
}
@Override
public String getValidationQuery() {
return "";
}
@Override
public String getJdbcUrl(ConnectionParam connectionParam) {
return "";
}
@Override
public Connection getConnection(ConnectionParam connectionParam) throws ClassNotFoundException, SQLException, IOException {
return null;
}
@Override
public boolean checkDataSourceConnectivity(ConnectionParam connectionParam) {
K8sConnectionParam baseConnectionParam = (K8sConnectionParam) connectionParam;
try (
K8sClientWrapper k8sClientWrapper = new K8sClientWrapper()) {
return k8sClientWrapper.checkConnect(baseConnectionParam.kubeConfig, baseConnectionParam.namespace);
} catch (Exception e) {
log.error("failed to connect to the K8S cluster", e);
return false;
}
}
@Override
public DbType getDbType() {
return DbType.K8S;
}
@Override
public DataSourceProcessor create() {
return new K8sDataSourceProcessor();
}
}

View File

@ -1,107 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.datasource.k8s;
import org.apache.dolphinscheduler.plugin.datasource.k8s.param.K8sConnectionParam;
import org.apache.dolphinscheduler.plugin.datasource.k8s.param.K8sDataSourceParamDTO;
import org.apache.dolphinscheduler.plugin.datasource.k8s.param.K8sDataSourceProcessor;
import org.apache.dolphinscheduler.spi.enums.DbType;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.MockedConstruction;
import org.mockito.Mockito;
import org.mockito.junit.jupiter.MockitoExtension;
@ExtendWith(MockitoExtension.class)
public class K8sDataSourceProcessorTest {
private K8sDataSourceProcessor k8sDataSourceProcessor;
private String connectJson =
"{\"namespace\":\"namespace\",\"kubeConfig\":\"kubeConfig\"}";
@BeforeEach
public void init() {
k8sDataSourceProcessor = new K8sDataSourceProcessor();
}
@Test
void testCheckDatasourceParam() {
K8sDataSourceParamDTO k8sDataSourceParamDTO = new K8sDataSourceParamDTO();
Assertions.assertThrows(IllegalArgumentException.class,
() -> k8sDataSourceProcessor.checkDatasourceParam(k8sDataSourceParamDTO));
k8sDataSourceParamDTO.setNamespace("namespace");
Assertions.assertThrows(IllegalArgumentException.class,
() -> k8sDataSourceProcessor.checkDatasourceParam(k8sDataSourceParamDTO));
k8sDataSourceParamDTO.setKubeConfig("kubeConfig");
Assertions
.assertDoesNotThrow(
() -> k8sDataSourceProcessor.checkDatasourceParam(k8sDataSourceParamDTO));
}
@Test
void testGetDatasourceUniqueId() {
K8sConnectionParam k8sConnectionParam = new K8sConnectionParam();
k8sConnectionParam.setNamespace("namespace");
k8sConnectionParam.setKubeConfig("kubeConfig");
Assertions.assertEquals("k8s@kubeConfig@namespace",
k8sDataSourceProcessor.getDatasourceUniqueId(k8sConnectionParam, DbType.K8S));
}
@Test
void testCreateDatasourceParamDTO() {
K8sDataSourceParamDTO k8sDataSourceParamDTO =
(K8sDataSourceParamDTO) k8sDataSourceProcessor.createDatasourceParamDTO(connectJson);
Assertions.assertEquals("namespace", k8sDataSourceParamDTO.getNamespace());
Assertions.assertEquals("kubeConfig", k8sDataSourceParamDTO.getKubeConfig());
}
@Test
void testCreateConnectionParams() {
K8sDataSourceParamDTO k8sDataSourceParamDTO =
(K8sDataSourceParamDTO) k8sDataSourceProcessor.createDatasourceParamDTO(connectJson);
K8sConnectionParam k8sConnectionParam =
k8sDataSourceProcessor.createConnectionParams(k8sDataSourceParamDTO);
Assertions.assertEquals("namespace", k8sConnectionParam.getNamespace());
Assertions.assertEquals("kubeConfig", k8sConnectionParam.getKubeConfig());
}
@Test
void testTestConnection() {
K8sDataSourceParamDTO k8sDataSourceParamDTO =
(K8sDataSourceParamDTO) k8sDataSourceProcessor.createDatasourceParamDTO(connectJson);
K8sConnectionParam connectionParam =
k8sDataSourceProcessor.createConnectionParams(k8sDataSourceParamDTO);
Assertions.assertFalse(k8sDataSourceProcessor.checkDataSourceConnectivity(connectionParam));
try (
MockedConstruction<K8sClientWrapper> k8sClientWrapperMockedConstruction =
Mockito.mockConstruction(K8sClientWrapper.class, (mock, context) -> {
Mockito.when(
mock.checkConnect(connectionParam.getKubeConfig(), connectionParam.getNamespace()))
.thenReturn(true);
})) {
Assertions.assertTrue(k8sDataSourceProcessor.checkDataSourceConnectivity(connectionParam));
}
}
}

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-kyuubi</artifactId> <artifactId>dolphinscheduler-datasource-kyuubi</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-mysql</artifactId> <artifactId>dolphinscheduler-datasource-mysql</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-oceanbase</artifactId> <artifactId>dolphinscheduler-datasource-oceanbase</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-oracle</artifactId> <artifactId>dolphinscheduler-datasource-oracle</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-postgresql</artifactId> <artifactId>dolphinscheduler-datasource-postgresql</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-presto</artifactId> <artifactId>dolphinscheduler-datasource-presto</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-redshift</artifactId> <artifactId>dolphinscheduler-datasource-redshift</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-sagemaker</artifactId> <artifactId>dolphinscheduler-datasource-sagemaker</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-snowflake</artifactId> <artifactId>dolphinscheduler-datasource-snowflake</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-spark</artifactId> <artifactId>dolphinscheduler-datasource-spark</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-sqlserver</artifactId> <artifactId>dolphinscheduler-datasource-sqlserver</artifactId>

View File

@ -35,6 +35,7 @@ import java.sql.DriverManager;
import java.sql.SQLException; import java.sql.SQLException;
import java.util.List; import java.util.List;
import java.util.Map; import java.util.Map;
import java.util.stream.Collectors;
import com.alibaba.druid.sql.parser.SQLParserUtils; import com.alibaba.druid.sql.parser.SQLParserUtils;
import com.google.auto.service.AutoService; import com.google.auto.service.AutoService;
@ -128,8 +129,10 @@ public class SQLServerDataSourceProcessor extends AbstractDataSourceProcessor {
@Override @Override
public List<String> splitAndRemoveComment(String sql) { public List<String> splitAndRemoveComment(String sql) {
String cleanSQL = SQLParserUtils.removeComment(sql, com.alibaba.druid.DbType.sqlserver); return SQLParserUtils.splitAndRemoveComment(sql, com.alibaba.druid.DbType.sqlserver)
return SQLParserUtils.split(cleanSQL, com.alibaba.druid.DbType.sqlserver); .stream()
.map(subSql -> subSql.concat(";"))
.collect(Collectors.toList());
} }
private String transformOther(Map<String, String> otherMap) { private String transformOther(Map<String, String> otherMap) {

View File

@ -17,12 +17,15 @@
package org.apache.dolphinscheduler.plugin.datasource.sqlserver.param; package org.apache.dolphinscheduler.plugin.datasource.sqlserver.param;
import static com.google.common.truth.Truth.assertThat;
import org.apache.dolphinscheduler.common.constants.DataSourceConstants; import org.apache.dolphinscheduler.common.constants.DataSourceConstants;
import org.apache.dolphinscheduler.common.utils.JSONUtils; import org.apache.dolphinscheduler.common.utils.JSONUtils;
import org.apache.dolphinscheduler.plugin.datasource.api.utils.PasswordUtils; import org.apache.dolphinscheduler.plugin.datasource.api.utils.PasswordUtils;
import org.apache.dolphinscheduler.spi.enums.DbType; import org.apache.dolphinscheduler.spi.enums.DbType;
import java.util.HashMap; import java.util.HashMap;
import java.util.List;
import java.util.Map; import java.util.Map;
import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Assertions;
@ -95,4 +98,32 @@ public class SQLServerDataSourceProcessorTest {
Assertions.assertEquals(DataSourceConstants.SQLSERVER_VALIDATION_QUERY, Assertions.assertEquals(DataSourceConstants.SQLSERVER_VALIDATION_QUERY,
sqlServerDatasourceProcessor.getValidationQuery()); sqlServerDatasourceProcessor.getValidationQuery());
} }
@Test
void splitAndRemoveComment_singleSelect() {
String sql = "select * from table;";
List<String> subSqls = sqlServerDatasourceProcessor.splitAndRemoveComment(sql);
assertThat(subSqls).hasSize(1);
assertThat(subSqls.get(0)).isEqualTo("select * from table;");
}
@Test
void splitAndRemoveComment_singleMerge() {
String sql = "MERGE\n" +
" [ TOP ( expression ) [ PERCENT ] ]\n" +
" [ INTO ] <target_table> [ WITH ( <merge_hint> ) ] [ [ AS ] table_alias ]\n" +
" USING <table_source> [ [ AS ] table_alias ]\n" +
" ON <merge_search_condition>\n" +
" [ WHEN MATCHED [ AND <clause_search_condition> ]\n" +
" THEN <merge_matched> ] [ ...n ]\n" +
" [ WHEN NOT MATCHED [ BY TARGET ] [ AND <clause_search_condition> ]\n" +
" THEN <merge_not_matched> ]\n" +
" [ WHEN NOT MATCHED BY SOURCE [ AND <clause_search_condition> ]\n" +
" THEN <merge_matched> ] [ ...n ]\n" +
" [ <output_clause> ]\n" +
" [ OPTION ( <query_hint> [ ,...n ] ) ];";
List<String> subSqls = sqlServerDatasourceProcessor.splitAndRemoveComment(sql);
assertThat(subSqls).hasSize(1);
assertThat(subSqls.get(0)).isEqualTo(sql);
}
} }

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-ssh</artifactId> <artifactId>dolphinscheduler-datasource-ssh</artifactId>

View File

@ -15,13 +15,12 @@
~ See the License for the specific language governing permissions and ~ See the License for the specific language governing permissions and
~ limitations under the License. ~ limitations under the License.
--> -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.2</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-starrocks</artifactId> <artifactId>dolphinscheduler-datasource-starrocks</artifactId>

Some files were not shown because too many files have changed in this diff Show More