Compare commits

...

7 Commits

Author SHA1 Message Date
wangxj3 3ad53c5200
3.2.2 release add task-pure delete some task module to reduce the size of the deployment package (#16094)
* add task-pure delete some task module to reduce the size of the deployment package

* add version of 3.2.2

* Update Chart.yaml Rollback APPversion

* remove task pure plugin

* fix ci

* add pure datasource plugin

* remove ck,sr from datasource pure

* remove task plugin chunjun ,dms ,datafactory from pure

* add api pom dependency

* spotless:apply

* add dependency license

* [maven-release-plugin] prepare release 3.2.2

* [maven-release-plugin] prepare for next development iteration

* remove datasource-pure:
dolphinscheduler-datasource-sqlserver
dolphinscheduler-datasource-oracle

remove task-pure:
dolphinscheduler-task-dinky
dolphinscheduler-task-pytorch
dolphinscheduler-task-datasync
dolphinscheduler-task-kubeflow
dolphinscheduler-task-emr
dolphinscheduler-task-pigeon
dolphinscheduler-task-flink-stream

* add register pure remove jdbc and etcd

* add dolphinscheduler-storage-pure remove task of
dolphinscheduler-storage-obs
dolphinscheduler-storage-abs
dolphinscheduler-storage-gcs
dolphinscheduler-storage-oss

* mvn spotless:apply

---------

Co-authored-by: wangxj959 <wangxj959@chinaunicom.cn>
Co-authored-by: wangxj <wangxj31>
Co-authored-by: wangxj3 <wangxj@apache.org>
2024-06-08 17:05:20 +08:00
Wenjun Ruan e91a272965
Fix SqlServer split merge sql will miss ; (#16118) (#16120)
(cherry picked from commit d6714bb77a)
2024-06-06 17:11:51 +08:00
Wenjun Ruan 31e79d10ea
Fix task nodes under switch task will not be skipped (#16109)
(cherry picked from commit f687a7d851c206ae30377ac3f9dc025cec35ffb2)
2024-06-05 23:41:52 +08:00
Wenjun Ruan 50c434e6c4
Fix task instance which status is dispatch will not be failover (#16110)
(cherry picked from commit a59ecf2aa19f4acd8d9981d14abe4d0c0d8e7fec)
2024-06-05 13:31:29 +08:00
Wenjun Ruan 72b402422a
Revert "[Improvement] Abnormal characters check (#15824)" (#16102) (#16103)
This reverts commit e5e7749251.

(cherry picked from commit 7ae9ca5a33)
2024-05-31 21:21:14 +08:00
xiangzihao fd864bf351
Revert "3.2.2 release cherry-pick cp: Reduce the size of tarball to continue …" (#16068)
This reverts commit 646bbf4881.
2024-05-25 10:50:37 +08:00
wangxj3 646bbf4881
3.2.2 release cherry-pick cp: Reduce the size of tarball to continue ASF release (#16042)
* add version of 3.2.2

* Fix rpc channel leak due to concurrent operation (#16021)

* Fix rpc channel leak due to concurrent operation

* Throw channel create failed exception

---------

Co-authored-by: Rick Cheng <rickchengx@gmail.com>

* Fix WorkerTaskExecutorThreadPool#isOverload is incorrect (#16027)

* cp: Reduce the size of tarball to continue ASF release (#15004) (#15540)

* Reduce the size of tarball to continue ASF release

for more detail you can see https://lists.apache.org/thread/rmp7fghlj0n7h9y2v3p8gkw9f9qbo6qt
# Conflicts:
#	tools/dependencies/known-dependencies.txt

* Increase block-until-connected in ZookeeperRegistryTestCase (#16041)

* [Fix][CI] fix the ci error of Values.datasource.profile (#16031)

Co-authored-by: Eric Gao <ericgao.apache@gmail.com>

* Correction of Typos in the Chinese Document Appendix for Task Parameters (#16033)

Co-authored-by: Rick Cheng <rickchengx@gmail.com>

* [DSIP-39][parameter] Improvement startup parameters/global parameters/project parameters data type (#15967)

* [Improvement][parameter] New data types and type filtering

* [Improvement][parameter] Improvement startup parameters/global parameters data type

* fix api interfaces compatible

* add project parameter data type default value

* [Improvement][parameter] New data types and type filtering

* [Improvement][parameter] Improvement startup parameters/global parameters data type

* fix api interfaces compatible

* add project parameter data type default value

* improvement project code

* remove useless imports

* remove method onClearSearchTaskType

* add parameter doc

* optimisation logic

* code conflict resolution

* code conflict resolution

* [Improvement][Monitor] Show master && worker Busy Or Normal Status and Show Commands table list (#15978)

* update

* test

* add monitor enhance ui

* update

* update

* update doc

* fix spotless

* update

* update

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/CommandMapper.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* update

* fix spotless

* update

---------

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* [Improvement][Monitor] Add UT for montor (#15998)

* formatting Code

* add pom

* Fix NettyRemotingClient might throw IllegalMonitorStateException (#16038)

* Removed unused StateEventHandleFailure (#16052)

* Remove OmitStackTraceInFastThrow in start.sh (#16054)

* Update Chart.yaml Rollback APPversion

* modify log msg (#16062)

Co-authored-by: xiangzihao <460888207@qq.com>

* [DS-16046][fix] Set PreparedStatement parameter type (#16050)

Fix the PreparedStatement parameter is TIME, set it to java.sql.Time

Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>

* Update pom.xml

* [DSIP-42] Add dolphinscheduler-aws-authentication module  (#16043)

---------

Co-authored-by: wangxj <wangxj31>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
Co-authored-by: TianXinCoord <tianxincoord@163.com>
Co-authored-by: 小可耐 <46134044+sdhzwc@users.noreply.github.com>
Co-authored-by: sleo <97011595+alei1206@users.noreply.github.com>
Co-authored-by: xiangzihao <460888207@qq.com>
Co-authored-by: yinxiaolog <43245392+yinxiaolog@users.noreply.github.com>
Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
2024-05-25 10:16:18 +08:00
170 changed files with 1080 additions and 639 deletions

View File

@ -15,8 +15,8 @@
# specific language governing permissions and limitations # specific language governing permissions and limitations
# under the License. # under the License.
# #
HUB=ghcr.io/apache/dolphinscheduler HUB=apache
TAG=latest TAG=3.2.2
TZ=Asia/Shanghai TZ=Asia/Shanghai
DATABASE=postgresql DATABASE=postgresql

View File

@ -35,7 +35,7 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes # This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version. # to the chart and its templates, including the app version.
version: 3.1.0 version: 3.2.2
# This is the version number of the application being deployed. This version number should be # This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. # incremented each time you make changes to the application.

View File

@ -33,7 +33,7 @@ image:
# -- Docker image repository for the DolphinScheduler # -- Docker image repository for the DolphinScheduler
registry: apache registry: apache
# -- Docker image version for the DolphinScheduler # -- Docker image version for the DolphinScheduler
tag: latest tag: 3.2.2
# -- Image pull policy. Options: Always, Never, IfNotPresent # -- Image pull policy. Options: Always, Never, IfNotPresent
pullPolicy: "IfNotPresent" pullPolicy: "IfNotPresent"
# -- Specify a imagePullSecrets # -- Specify a imagePullSecrets

File diff suppressed because it is too large Load Diff

View File

@ -340,7 +340,7 @@ svn --username="${A_USERNAME}" commit -m "release ${VERSION}"
#### Check source package #### Check source package
Decompress `apache-dolphinscheduler-<VERSION>-src.tar.gz` then check the following items: Decompress `apache-dolphinscheduler-3.2.2-src.tar.gz` then check the following items:
- Check whether source tarball is oversized for including nonessential files - Check whether source tarball is oversized for including nonessential files
- `LICENSE` and `NOTICE` files exist - `LICENSE` and `NOTICE` files exist
@ -353,7 +353,7 @@ Decompress `apache-dolphinscheduler-<VERSION>-src.tar.gz` then check the followi
#### Check binary packages #### Check binary packages
Decompress `apache-dolphinscheduler-<VERSION>-bin.tar.gz` to check the following items: Decompress `apache-dolphinscheduler-3.2.2-bin.tar.gz` to check the following items:
- `LICENSE` and `NOTICE` files exist - `LICENSE` and `NOTICE` files exist
- Correct year in `NOTICE` file - Correct year in `NOTICE` file
@ -389,7 +389,7 @@ Announce the vote result: send the result vote e-mail to `dev@dolphinscheduler.a
Title Title
```txt ```txt
[VOTE] Release Apache DolphinScheduler <VERSION> [VOTE] Release Apache DolphinScheduler 3.2.2
``` ```
Body Body
@ -397,15 +397,15 @@ Body
```txt ```txt
Hello DolphinScheduler Community, Hello DolphinScheduler Community,
This is a call for vote to release Apache DolphinScheduler version <VERSION> This is a call for vote to release Apache DolphinScheduler version 3.2.2
Release notes: https://github.com/apache/dolphinscheduler/releases/tag/<VERSION> Release notes: https://github.com/apache/dolphinscheduler/releases/tag/3.2.2
The release candidates: https://dist.apache.org/repos/dist/dev/dolphinscheduler/<VERSION>/ The release candidates: https://dist.apache.org/repos/dist/dev/dolphinscheduler/3.2.2/
Maven 2 staging repository: https://repository.apache.org/content/repositories/<STAGING.REPOSITORY>/org/apache/dolphinscheduler/ Maven 2 staging repository: https://repository.apache.org/content/repositories/<STAGING.REPOSITORY>/org/apache/dolphinscheduler/
Git tag for the release: https://github.com/apache/dolphinscheduler/tree/<VERSION> Git tag for the release: https://github.com/apache/dolphinscheduler/tree/3.2.2
Release Commit ID: https://github.com/apache/dolphinscheduler/commit/<SHA-VALUE> Release Commit ID: https://github.com/apache/dolphinscheduler/commit/<SHA-VALUE>
@ -436,13 +436,13 @@ Checklist for reference:
Title Title
```txt ```txt
[RESULT][VOTE] Release Apache DolphinScheduler <VERSION> [RESULT][VOTE] Release Apache DolphinScheduler 3.2.2
``` ```
Body Body
```txt ```txt
The vote to release Apache DolphinScheduler <VERSION> has passed.Here is the vote result, The vote to release Apache DolphinScheduler 3.2.2 has passed.Here is the vote result,
4 PMC member +1 votes: 4 PMC member +1 votes:
@ -478,19 +478,19 @@ and then find DolphinScheduler in [apache staging repositories](https://reposito
### Update Document ### Update Document
Website should be present before you send the announce mail this section will tell you how to change the website. For example, Website should be present before you send the announce mail this section will tell you how to change the website. For example,
the release version is `<VERSION>`, the following updates are required(note it will take effect immediately when the PR is merged): the release version is `3.2.2`, the following updates are required(note it will take effect immediately when the PR is merged):
- Repository **apache/dolphinscheduler-website**: - Repository **apache/dolphinscheduler-website**:
- `config/download.json`: add the download of the `<VERSION>` release package - `config/download.json`: add the download of the `3.2.2` release package
- `scripts/conf.sh`: Add new release version `<VERSION>` key-value pair to variable `DEV_RELEASE_DOCS_VERSIONS` - `scripts/conf.sh`: Add new release version `3.2.2` key-value pair to variable `DEV_RELEASE_DOCS_VERSIONS`
- Repository **apache/dolphinscheduler** (dev branch): - Repository **apache/dolphinscheduler** (dev branch):
- `docs/configs/site.js`: - `docs/configs/site.js`:
- `docsLatest`: update to `<VERSION>` - `docsLatest`: update to `3.2.2`
- `docs0`: The `text` of two places of `en-us/zh-cn` needs to be updated to `latest(<VERSION>)` - `docs0`: The `text` of two places of `en-us/zh-cn` needs to be updated to `latest(3.2.2)`
- `docs/configs/index.md.jsx`: Add `<VERSION>: docsxyzConfig` and add new `import` for the new `docsxyzConfig` - `docs/configs/index.md.jsx`: Add `3.2.2: docsxyzConfig` and add new `import` for the new `docsxyzConfig`
- `docs/docs/en/history-versions.md` and `docs/docs/zh/history-versions.md`: Add new `<VERSION>` release docs. - `docs/docs/en/history-versions.md` and `docs/docs/zh/history-versions.md`: Add new `3.2.2` release docs.
- `.github/ISSUE_TEMPLATE/bug-report.yml`: DolphinScheduler's GitHub [bug-report](https://github.com/apache/dolphinscheduler/blob/dev/.github/ISSUE_TEMPLATE/bug-report.yml) - `.github/ISSUE_TEMPLATE/bug-report.yml`: DolphinScheduler's GitHub [bug-report](https://github.com/apache/dolphinscheduler/blob/dev/.github/ISSUE_TEMPLATE/bug-report.yml)
issue template have **Version** selection bottom. So after released we should add the new `<VERSION>` to issue template have **Version** selection bottom. So after released we should add the new `3.2.2` to
bug-report.yml bug-report.yml
### Publish Docker Image and Helm Chart ### Publish Docker Image and Helm Chart
@ -511,7 +511,7 @@ Announcement e-mail template as below
Title Title
```txt ```txt
[ANNOUNCE] Release Apache DolphinScheduler <VERSION> [ANNOUNCE] Release Apache DolphinScheduler 3.2.2
``` ```
Body Body
@ -519,7 +519,7 @@ Body
```txt ```txt
Hi all, Hi all,
We are glad to announce the release of Apache DolphinScheduler <VERSION>. Once again I would like to express my thanks to your help. We are glad to announce the release of Apache DolphinScheduler 3.2.2. Once again I would like to express my thanks to your help.
Dolphin Scheduler is a distributed and easy-to-extend visual workflow scheduler system, Dolphin Scheduler is a distributed and easy-to-extend visual workflow scheduler system,
dedicated to solving the complex task dependencies in data processing, making the scheduler system out of the box for data processing. dedicated to solving the complex task dependencies in data processing, making the scheduler system out of the box for data processing.
@ -527,14 +527,14 @@ dedicated to solving the complex task dependencies in data processing, making th
Download Links: https://dolphinscheduler.apache.org/en-us/download Download Links: https://dolphinscheduler.apache.org/en-us/download
Release Notes: https://github.com/apache/dolphinscheduler/releases/tag/<VERSION> Release Notes: https://github.com/apache/dolphinscheduler/releases/tag/3.2.2
Website: https://dolphinscheduler.apache.org/ Website: https://dolphinscheduler.apache.org/
DolphinScheduler Resources: DolphinScheduler Resources:
- Issue: https://github.com/apache/dolphinscheduler/issues/ - Issue: https://github.com/apache/dolphinscheduler/issues/
- Mailing list: dev@dolphinscheduler.apache.org - Mailing list: dev@dolphinscheduler.apache.org
- Documents: https://dolphinscheduler.apache.org/en-us/docs/<VERSION>/about/introduction - Documents: https://dolphinscheduler.apache.org/en-us/docs/3.2.2/about/introduction
``` ```
## News ## News

View File

@ -30,9 +30,9 @@ Attention: DolphinScheduler itself does not depend on Hadoop, Hive, Spark, but w
mkdir -p /opt mkdir -p /opt
cd /opt cd /opt
# decompress # decompress
tar -zxvf apache-dolphinscheduler-<version>-bin.tar.gz -C /opt tar -zxvf apache-dolphinscheduler-3.2.2-bin.tar.gz -C /opt
cd /opt cd /opt
mv apache-dolphinscheduler-<version>-bin dolphinscheduler mv apache-dolphinscheduler-3.2.2-bin dolphinscheduler
``` ```
```markdown ```markdown

View File

@ -233,9 +233,9 @@ kubectl scale --replicas=6 sts dolphinscheduler-worker -n test # with test names
2. Create a new `Dockerfile` to add MySQL driver: 2. Create a new `Dockerfile` to add MySQL driver:
``` ```
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:<version> FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:3.2.2
# For example # For example
# FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-tools:<version> # FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-tools:3.2.2
# Attention Please, If the build is dolphinscheduler-tools image # Attention Please, If the build is dolphinscheduler-tools image
# You need to change the following line to: COPY mysql-connector-java-8.0.16.jar /opt/dolphinscheduler/tools/libs # You need to change the following line to: COPY mysql-connector-java-8.0.16.jar /opt/dolphinscheduler/tools/libs
@ -284,9 +284,9 @@ externalDatabase:
2. Create a new `Dockerfile` to add MySQL or Oracle driver: 2. Create a new `Dockerfile` to add MySQL or Oracle driver:
``` ```
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:<version> FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:3.2.2
# For example # For example
# FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:<version> # FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:3.2.2
# If you want to support MySQL Datasource # If you want to support MySQL Datasource
COPY mysql-connector-java-8.0.16.jar /opt/dolphinscheduler/libs COPY mysql-connector-java-8.0.16.jar /opt/dolphinscheduler/libs
@ -316,7 +316,7 @@ docker build -t apache/dolphinscheduler-<service>:new-driver .
1. Create a new `Dockerfile` to install pip: 1. Create a new `Dockerfile` to install pip:
``` ```
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:<version> FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:3.2.2
COPY requirements.txt /tmp COPY requirements.txt /tmp
RUN apt-get update && \ RUN apt-get update && \
apt-get install -y --no-install-recommends python-pip && \ apt-get install -y --no-install-recommends python-pip && \
@ -351,7 +351,7 @@ docker build -t apache/dolphinscheduler-worker:pip .
1. Create a new `Dockerfile` to install Python 3: 1. Create a new `Dockerfile` to install Python 3:
``` ```
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:<version> FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:3.2.2
RUN apt-get update && \ RUN apt-get update && \
apt-get install -y --no-install-recommends python3 && \ apt-get install -y --no-install-recommends python3 && \
rm -rf /var/lib/apt/lists/* rm -rf /var/lib/apt/lists/*

View File

@ -12,7 +12,7 @@ If you want to deploy DolphinScheduler in production, we recommend you follow [c
## Preparation ## Preparation
- JDKdownload [JDK][jdk] (1.8+), install and configure environment variable `JAVA_HOME` and append `bin` dir (included in `JAVA_HOME`) to `PATH` variable. You can skip this step if it already exists in your environment. - JDKdownload [JDK][jdk] (1.8+), install and configure environment variable `JAVA_HOME` and append `bin` dir (included in `JAVA_HOME`) to `PATH` variable. You can skip this step if it already exists in your environment.
- Binary package: download the DolphinScheduler binary package at [download page](https://dolphinscheduler.apache.org/en-us/download/<version>). <!-- markdown-link-check-disable-line --> - Binary package: download the DolphinScheduler binary package at [download page](https://dolphinscheduler.apache.org/en-us/download/3.2.2). <!-- markdown-link-check-disable-line -->
## Start DolphinScheduler Standalone Server ## Start DolphinScheduler Standalone Server

View File

@ -19,7 +19,7 @@ Start DolphinScheduler with standalone-server Docker images is the easiest way t
you can learn DolphinScheduler's concepts and usage, with minimal cost. you can learn DolphinScheduler's concepts and usage, with minimal cost.
```shell ```shell
$ DOLPHINSCHEDULER_VERSION=<version> $ DOLPHINSCHEDULER_VERSION=3.2.2
$ docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:25333 -d apache/dolphinscheduler-standalone-server:"${DOLPHINSCHEDULER_VERSION}" $ docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:25333 -d apache/dolphinscheduler-standalone-server:"${DOLPHINSCHEDULER_VERSION}"
``` ```
@ -37,11 +37,11 @@ be stored on disks after you change docker-compose configuration, and it is robu
DolphinScheduler in a long term. You have to install [docker-compose](https://docs.docker.com/compose/install/) before you DolphinScheduler in a long term. You have to install [docker-compose](https://docs.docker.com/compose/install/) before you
start servers. start servers.
After complete the installation, get the `docker-compose.yaml` file from [download page](https://dolphinscheduler.apache.org/en-us/download/<version>) After complete the installation, get the `docker-compose.yaml` file from [download page](https://dolphinscheduler.apache.org/en-us/download/3.2.2)
form its source package, and make sure you get the right version. After download the package, you can run the commands as below. form its source package, and make sure you get the right version. After download the package, you can run the commands as below.
```shell ```shell
$ DOLPHINSCHEDULER_VERSION=<version> $ DOLPHINSCHEDULER_VERSION=3.2.2
$ tar -zxf apache-dolphinscheduler-"${DOLPHINSCHEDULER_VERSION}"-src.tar.gz $ tar -zxf apache-dolphinscheduler-"${DOLPHINSCHEDULER_VERSION}"-src.tar.gz
# Going to docker-compose's location # Going to docker-compose's location
# For Mac or Linux users # For Mac or Linux users
@ -68,7 +68,7 @@ $ docker-compose --profile all up -d
container when it up. You could start DolphinScheduler server separately if you want to reuse your exists services. container when it up. You could start DolphinScheduler server separately if you want to reuse your exists services.
```shell ```shell
$ DOLPHINSCHEDULER_VERSION=<version> $ DOLPHINSCHEDULER_VERSION=3.2.2
# Initialize the database, make sure database <DATABASE> already exists # Initialize the database, make sure database <DATABASE> already exists
$ docker run -d --name dolphinscheduler-tools \ $ docker run -d --name dolphinscheduler-tools \
-e DATABASE="postgresql" \ -e DATABASE="postgresql" \
@ -134,5 +134,5 @@ and use `admin` and `dolphinscheduler123` as default username and password in th
You can modify some environment variables to change configurations when you are starting servers through Docker. We have You can modify some environment variables to change configurations when you are starting servers through Docker. We have
an example in [using exists PostgreSQL ZooKeeper](#using-exists-postgresql-zookeeper) to change database and ZooKeeper configurations, an example in [using exists PostgreSQL ZooKeeper](#using-exists-postgresql-zookeeper) to change database and ZooKeeper configurations,
and you could find all environment variables in [all environment variables](https://github.com/apache/dolphinscheduler/blob/<version>/script/env/dolphinscheduler_env.sh) <!-- markdown-link-check-disable-line --> and you could find all environment variables in [all environment variables](https://github.com/apache/dolphinscheduler/blob/3.2.2/script/env/dolphinscheduler_env.sh) <!-- markdown-link-check-disable-line -->
and change them if you want. and change them if you want.

View File

@ -4,6 +4,10 @@
#### Setup instructions, are available for each stable version of Apache DolphinScheduler below: #### Setup instructions, are available for each stable version of Apache DolphinScheduler below:
### Versions: 3.2.2
#### Links [3.2.2 Document](../3.2.2/user_doc/about/introduction.md)
### Versions: 3.2.1 ### Versions: 3.2.1
#### Links [3.2.1 Document](../3.2.1/user_doc/about/introduction.md) #### Links [3.2.1 Document](../3.2.1/user_doc/about/introduction.md)

View File

@ -347,7 +347,7 @@ svn --username="${A_USERNAME}" commit -m "release ${VERSION}"
#### 检查源码包的文件内容 #### 检查源码包的文件内容
解压缩`apache-dolphinscheduler-<VERSION>-src.tar.gz`,进行如下检查: 解压缩`apache-dolphinscheduler-3.2.2-src.tar.gz`,进行如下检查:
- 检查源码包是否包含由于包含不必要文件,致使 tarball 过于庞大 - 检查源码包是否包含由于包含不必要文件,致使 tarball 过于庞大
- 存在`LICENSE`和`NOTICE`文件 - 存在`LICENSE`和`NOTICE`文件
@ -359,7 +359,7 @@ svn --username="${A_USERNAME}" commit -m "release ${VERSION}"
#### 检查二进制包的文件内容 #### 检查二进制包的文件内容
解压缩`apache-dolphinscheduler-<VERSION>-bin.tar.gz`进行如下检查: 解压缩`apache-dolphinscheduler-3.2.2-bin.tar.gz`进行如下检查:
- 存在`LICENSE`和`NOTICE`文件 - 存在`LICENSE`和`NOTICE`文件
- 所有文本文件开头都有 ASF 许可证 - 所有文本文件开头都有 ASF 许可证
@ -394,7 +394,7 @@ DolphinScheduler 社区投票,发起投票邮件到`dev@dolphinscheduler.apach
标题: 标题:
```txt ```txt
[VOTE] Release Apache DolphinScheduler <VERSION> [VOTE] Release Apache DolphinScheduler 3.2.2
``` ```
正文: 正文:
@ -402,15 +402,15 @@ DolphinScheduler 社区投票,发起投票邮件到`dev@dolphinscheduler.apach
```txt ```txt
Hello DolphinScheduler Community, Hello DolphinScheduler Community,
This is a call for vote to release Apache DolphinScheduler version <VERSION> This is a call for vote to release Apache DolphinScheduler version 3.2.2
Release notes: https://github.com/apache/dolphinscheduler/releases/tag/<VERSION> Release notes: https://github.com/apache/dolphinscheduler/releases/tag/3.2.2
The release candidates: https://dist.apache.org/repos/dist/dev/dolphinscheduler/<VERSION>/ The release candidates: https://dist.apache.org/repos/dist/dev/dolphinscheduler/3.2.2/
Maven 2 staging repository: https://repository.apache.org/content/repositories/<STAGING.REPOSITORY>/org/apache/dolphinscheduler/ Maven 2 staging repository: https://repository.apache.org/content/repositories/<STAGING.REPOSITORY>/org/apache/dolphinscheduler/
Git tag for the release: https://github.com/apache/dolphinscheduler/tree/<VERSION> Git tag for the release: https://github.com/apache/dolphinscheduler/tree/3.2.2
Release Commit ID: https://github.com/apache/dolphinscheduler/commit/<SHA-VALUE> Release Commit ID: https://github.com/apache/dolphinscheduler/commit/<SHA-VALUE>
@ -441,11 +441,11 @@ Checklist for reference:
Title Title
```txt ```txt
[RESULT][VOTE] Release Apache DolphinScheduler <VERSION> [RESULT][VOTE] Release Apache DolphinScheduler 3.2.2
``` ```
```txt ```txt
The vote to release Apache DolphinScheduler <VERSION> has passed.Here is the vote result, The vote to release Apache DolphinScheduler 3.2.2 has passed.Here is the vote result,
4 PMC member +1 votes: 4 PMC member +1 votes:
@ -480,17 +480,17 @@ git push --delete "${GH_REMOTE}" "${VERSION}-prepare"
### 更新文档 ### 更新文档
官网应该在您发送通知邮件之前完成更新,本节将告诉您如何更改网站。假设发版的版本是 `<VERSION>`,需要进行以下更新(注意,当修改 pull requests 被 merge 后就会生效): 官网应该在您发送通知邮件之前完成更新,本节将告诉您如何更改网站。假设发版的版本是 `3.2.2`,需要进行以下更新(注意,当修改 pull requests 被 merge 后就会生效):
- **apache/dolphinscheduler-website** 仓库: - **apache/dolphinscheduler-website** 仓库:
- `config/download.json`: 增加 `<VERSION>` 版本发布包的下载 - `config/download.json`: 增加 `3.2.2` 版本发布包的下载
- `scripts/conf.sh`: 在变量 `DEV_RELEASE_DOCS_VERSIONS` 中增加版本为 `<VERSION>` 的新键值对 - `scripts/conf.sh`: 在变量 `DEV_RELEASE_DOCS_VERSIONS` 中增加版本为 `3.2.2` 的新键值对
- **apache/dolphinscheduler** 仓库 (dev 分支) - **apache/dolphinscheduler** 仓库 (dev 分支)
- `docs/configs/site.js`: - `docs/configs/site.js`:
- `docsLatest`: 更新为 `<VERSION>` - `docsLatest`: 更新为 `3.2.2`
- `docs0`: 两处 `en-us/zh-cn``text` 更新为 `latest(<VERSION>)` - `docs0`: 两处 `en-us/zh-cn``text` 更新为 `latest(3.2.2)`
- `docs/configs/index.md.jsx`: 增加 `'<VERSION>': docsxyzConfig,` 以及新的 `import` - `docs/configs/index.md.jsx`: 增加 `'3.2.2': docsxyzConfig,` 以及新的 `import`
- `docs/docs/en/history-versions.md``docs/docs/zh/history-versions.md`: 增加新的发版版本 `<VERSION>` 的链接 - `docs/docs/en/history-versions.md``docs/docs/zh/history-versions.md`: 增加新的发版版本 `3.2.2` 的链接
- `.github/ISSUE_TEMPLATE/bug-report.yml`: DolphinScheduler 在 GitHub bug report 的 issue 中有版本选择,当有新的版本发版后,需要更新 - `.github/ISSUE_TEMPLATE/bug-report.yml`: DolphinScheduler 在 GitHub bug report 的 issue 中有版本选择,当有新的版本发版后,需要更新
[bug-report](https://github.com/apache/dolphinscheduler/blob/dev/.github/ISSUE_TEMPLATE/bug-report.yml) 中的 **Version** 部分。 [bug-report](https://github.com/apache/dolphinscheduler/blob/dev/.github/ISSUE_TEMPLATE/bug-report.yml) 中的 **Version** 部分。
@ -510,7 +510,7 @@ git push --delete "${GH_REMOTE}" "${VERSION}-prepare"
标题: 标题:
```txt ```txt
[ANNOUNCE] Release Apache DolphinScheduler <VERSION> [ANNOUNCE] Release Apache DolphinScheduler 3.2.2
``` ```
正文: 正文:
@ -518,7 +518,7 @@ git push --delete "${GH_REMOTE}" "${VERSION}-prepare"
```txt ```txt
Hi all, Hi all,
We are glad to announce the release of Apache DolphinScheduler <VERSION>. Once again I would like to express my thanks to your help. We are glad to announce the release of Apache DolphinScheduler 3.2.2. Once again I would like to express my thanks to your help.
Dolphin Scheduler is a distributed and easy-to-extend visual workflow scheduler system, Dolphin Scheduler is a distributed and easy-to-extend visual workflow scheduler system,
dedicated to solving the complex task dependencies in data processing, making the scheduler system out of the box for data processing. dedicated to solving the complex task dependencies in data processing, making the scheduler system out of the box for data processing.
@ -526,14 +526,14 @@ dedicated to solving the complex task dependencies in data processing, making th
Download Links: https://dolphinscheduler.apache.org/zh-cn/download Download Links: https://dolphinscheduler.apache.org/zh-cn/download
Release Notes: https://github.com/apache/dolphinscheduler/releases/tag/<VERSION> Release Notes: https://github.com/apache/dolphinscheduler/releases/tag/3.2.2
Website: https://dolphinscheduler.apache.org/ Website: https://dolphinscheduler.apache.org/
DolphinScheduler Resources: DolphinScheduler Resources:
- Issue: https://github.com/apache/dolphinscheduler/issues/ - Issue: https://github.com/apache/dolphinscheduler/issues/
- Mailing list: dev@dolphinscheduler.apache.org - Mailing list: dev@dolphinscheduler.apache.org
- Documents: https://dolphinscheduler.apache.org/zh-cn/docs/<VERSION>/about/introduction - Documents: https://dolphinscheduler.apache.org/zh-cn/docs/3.2.2/about/introduction
``` ```
## News ## News

View File

@ -30,9 +30,9 @@
mkdir -p /opt mkdir -p /opt
cd /opt cd /opt
# 解压缩 # 解压缩
tar -zxvf apache-dolphinscheduler-<version>-bin.tar.gz -C /opt tar -zxvf apache-dolphinscheduler-3.2.2-bin.tar.gz -C /opt
cd /opt cd /opt
mv apache-dolphinscheduler-<version>-bin dolphinscheduler mv apache-dolphinscheduler-3.2.2-bin dolphinscheduler
``` ```
```markdown ```markdown

View File

@ -232,9 +232,9 @@ kubectl scale --replicas=6 sts dolphinscheduler-worker -n test # with test names
2. 创建一个新的 `Dockerfile`,用于添加 MySQL 的驱动包: 2. 创建一个新的 `Dockerfile`,用于添加 MySQL 的驱动包:
``` ```
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:<version> FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:3.2.2
# 例如 # 例如
# FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-tools:<version> # FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-tools:3.2.2
# 注意如果构建的是dolphinscheduler-tools镜像 # 注意如果构建的是dolphinscheduler-tools镜像
# 需要将下面一行修改为COPY mysql-connector-java-8.0.16.jar /opt/dolphinscheduler/tools/libs # 需要将下面一行修改为COPY mysql-connector-java-8.0.16.jar /opt/dolphinscheduler/tools/libs
@ -283,9 +283,9 @@ externalDatabase:
2. 创建一个新的 `Dockerfile`,用于添加 MySQL 或者 Oracle 驱动包: 2. 创建一个新的 `Dockerfile`,用于添加 MySQL 或者 Oracle 驱动包:
``` ```
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:<version> FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:3.2.2
# 例如 # 例如
# FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:<version> # FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:3.2.2
# 如果你想支持 MySQL 数据源 # 如果你想支持 MySQL 数据源
COPY mysql-connector-java-8.0.16.jar /opt/dolphinscheduler/libs COPY mysql-connector-java-8.0.16.jar /opt/dolphinscheduler/libs
@ -315,7 +315,7 @@ docker build -t apache/dolphinscheduler-<service>:new-driver .
1. 创建一个新的 `Dockerfile`,用于安装 pip: 1. 创建一个新的 `Dockerfile`,用于安装 pip:
``` ```
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:<version> FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:3.2.2
COPY requirements.txt /tmp COPY requirements.txt /tmp
RUN apt-get update && \ RUN apt-get update && \
apt-get install -y --no-install-recommends python-pip && \ apt-get install -y --no-install-recommends python-pip && \
@ -350,7 +350,7 @@ docker build -t apache/dolphinscheduler-worker:pip .
1. 创建一个新的 `Dockerfile`,用于安装 Python 3: 1. 创建一个新的 `Dockerfile`,用于安装 Python 3:
``` ```
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:<version> FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:3.2.2
RUN apt-get update && \ RUN apt-get update && \
apt-get install -y --no-install-recommends python3 && \ apt-get install -y --no-install-recommends python3 && \
rm -rf /var/lib/apt/lists/* rm -rf /var/lib/apt/lists/*

View File

@ -10,7 +10,7 @@ Standalone 仅适用于 DolphinScheduler 的快速体验.
## 前置准备工作 ## 前置准备工作
- JDK下载[JDK][jdk] (1.8+),安装并配置 `JAVA_HOME` 环境变量,并将其下的 `bin` 目录追加到 `PATH` 环境变量中。如果你的环境中已存在,可以跳过这步。 - JDK下载[JDK][jdk] (1.8+),安装并配置 `JAVA_HOME` 环境变量,并将其下的 `bin` 目录追加到 `PATH` 环境变量中。如果你的环境中已存在,可以跳过这步。
- 二进制包:在[下载页面](https://dolphinscheduler.apache.org/en-us/download/<version>)下载 DolphinScheduler 二进制包 <!-- markdown-link-check-disable-line --> - 二进制包:在[下载页面](https://dolphinscheduler.apache.org/en-us/download/3.2.2)下载 DolphinScheduler 二进制包 <!-- markdown-link-check-disable-line -->
## 启动 DolphinScheduler Standalone Server ## 启动 DolphinScheduler Standalone Server

View File

@ -18,7 +18,7 @@
你可以最快速的体验到 DolphinScheduler 的大部分功能,了解主要和概念和内容。 你可以最快速的体验到 DolphinScheduler 的大部分功能,了解主要和概念和内容。
```shell ```shell
$ DOLPHINSCHEDULER_VERSION=<version> $ DOLPHINSCHEDULER_VERSION=3.2.2
$ docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:25333 -d apache/dolphinscheduler-standalone-server:"${DOLPHINSCHEDULER_VERSION}" $ docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:25333 -d apache/dolphinscheduler-standalone-server:"${DOLPHINSCHEDULER_VERSION}"
``` ```
@ -33,11 +33,11 @@ $ docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:2
服务重启的时候保留元数据(如需要挂载到本地路径需要做指定)。他更健壮,能保证用户体验更加完整的 DolphinScheduler 服务。这种方式需要先安装 服务重启的时候保留元数据(如需要挂载到本地路径需要做指定)。他更健壮,能保证用户体验更加完整的 DolphinScheduler 服务。这种方式需要先安装
[docker-compose](https://docs.docker.com/compose/install/),链接适用于 MacLinuxWindows。 [docker-compose](https://docs.docker.com/compose/install/),链接适用于 MacLinuxWindows。
确保 docker-compose 顺利安装后,需要获取 `docker-compose.yaml` 文件,通过[下载页面](https://dolphinscheduler.apache.org/en-us/download/<version>) 确保 docker-compose 顺利安装后,需要获取 `docker-compose.yaml` 文件,通过[下载页面](https://dolphinscheduler.apache.org/en-us/download/3.2.2)
下载对应版本源码包可能是最快的方法,当下载完源码后就可以运行命令进行部署了。 下载对应版本源码包可能是最快的方法,当下载完源码后就可以运行命令进行部署了。
```shell ```shell
$ DOLPHINSCHEDULER_VERSION=<version> $ DOLPHINSCHEDULER_VERSION=3.2.2
$ tar -zxf apache-dolphinscheduler-"${DOLPHINSCHEDULER_VERSION}"-src.tar.gz $ tar -zxf apache-dolphinscheduler-"${DOLPHINSCHEDULER_VERSION}"-src.tar.gz
# Mac Linux 用户 # Mac Linux 用户
$ cd apache-dolphinscheduler-"${DOLPHINSCHEDULER_VERSION}"-src/deploy/docker $ cd apache-dolphinscheduler-"${DOLPHINSCHEDULER_VERSION}"-src/deploy/docker
@ -61,7 +61,7 @@ $ docker-compose --profile all up -d
ZooKeeper 且不想启动新的服务,可以使用这个方式分别启动 DolphinScheduler 容器。 ZooKeeper 且不想启动新的服务,可以使用这个方式分别启动 DolphinScheduler 容器。
```shell ```shell
$ DOLPHINSCHEDULER_VERSION=<version> $ DOLPHINSCHEDULER_VERSION=3.2.2
# 初始化数据库,其确保数据库 <DATABASE> 已经存在 # 初始化数据库,其确保数据库 <DATABASE> 已经存在
$ docker run -d --name dolphinscheduler-tools \ $ docker run -d --name dolphinscheduler-tools \
-e DATABASE="postgresql" \ -e DATABASE="postgresql" \

View File

@ -4,6 +4,10 @@
#### 以下是Apache DolphinScheduler每个稳定版本的设置说明。 #### 以下是Apache DolphinScheduler每个稳定版本的设置说明。
### Versions: 3.2.2
#### Links [3.2.2 Document](../3.2.2/user_doc/about/introduction.md)
### Versions: 3.2.1 ### Versions: 3.2.1
#### Links [3.2.1 Document](../3.2.1/user_doc/about/introduction.md) #### Links [3.2.1 Document](../3.2.1/user_doc/about/introduction.md)

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-aliyunVoice</artifactId> <artifactId>dolphinscheduler-alert-aliyunVoice</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-all</artifactId> <artifactId>dolphinscheduler-alert-all</artifactId>

View File

@ -23,7 +23,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-api</artifactId> <artifactId>dolphinscheduler-alert-api</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-dingtalk</artifactId> <artifactId>dolphinscheduler-alert-dingtalk</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-email</artifactId> <artifactId>dolphinscheduler-alert-email</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-feishu</artifactId> <artifactId>dolphinscheduler-alert-feishu</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-http</artifactId> <artifactId>dolphinscheduler-alert-http</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-pagerduty</artifactId> <artifactId>dolphinscheduler-alert-pagerduty</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-prometheus</artifactId> <artifactId>dolphinscheduler-alert-prometheus</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-script</artifactId> <artifactId>dolphinscheduler-alert-script</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-slack</artifactId> <artifactId>dolphinscheduler-alert-slack</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-telegram</artifactId> <artifactId>dolphinscheduler-alert-telegram</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-webexteams</artifactId> <artifactId>dolphinscheduler-alert-webexteams</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-wechat</artifactId> <artifactId>dolphinscheduler-alert-wechat</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert</artifactId> <artifactId>dolphinscheduler-alert</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-plugins</artifactId> <artifactId>dolphinscheduler-alert-plugins</artifactId>
<packaging>pom</packaging> <packaging>pom</packaging>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert</artifactId> <artifactId>dolphinscheduler-alert</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert-server</artifactId> <artifactId>dolphinscheduler-alert-server</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>
@ -43,11 +43,6 @@
<artifactId>dolphinscheduler-dao</artifactId> <artifactId>dolphinscheduler-dao</artifactId>
</dependency> </dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-all</artifactId>
</dependency>
<dependency> <dependency>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-extract-alert</artifactId> <artifactId>dolphinscheduler-extract-alert</artifactId>
@ -120,6 +115,30 @@
</plugins> </plugins>
</build> </build>
</profile> </profile>
<profile>
<id>task-pure</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<dependencies>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-pure</artifactId>
</dependency>
</dependencies>
</profile>
<profile>
<id>task-all</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<dependencies>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-all</artifactId>
</dependency>
</dependencies>
</profile>
</profiles> </profiles>
</project> </project>

View File

@ -23,7 +23,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-alert</artifactId> <artifactId>dolphinscheduler-alert</artifactId>
<packaging>pom</packaging> <packaging>pom</packaging>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-aop</artifactId> <artifactId>dolphinscheduler-aop</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-api</artifactId> <artifactId>dolphinscheduler-api</artifactId>
<packaging>jar</packaging> <packaging>jar</packaging>
@ -66,33 +66,14 @@
<artifactId>dolphinscheduler-data-quality</artifactId> <artifactId>dolphinscheduler-data-quality</artifactId>
</dependency> </dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-all</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-task-all</artifactId>
</dependency>
<dependency> <dependency>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-ui</artifactId> <artifactId>dolphinscheduler-ui</artifactId>
</dependency> </dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-all</artifactId>
</dependency>
<dependency> <dependency>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-scheduler-all</artifactId> <artifactId>dolphinscheduler-scheduler-all</artifactId>
</dependency> </dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-storage-all</artifactId>
</dependency>
<dependency> <dependency>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
@ -233,6 +214,10 @@
<groupId>com.azure.resourcemanager</groupId> <groupId>com.azure.resourcemanager</groupId>
<artifactId>azure-resourcemanager-datafactory</artifactId> <artifactId>azure-resourcemanager-datafactory</artifactId>
</dependency> </dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-identity</artifactId>
</dependency>
<dependency> <dependency>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-extract-alert</artifactId> <artifactId>dolphinscheduler-extract-alert</artifactId>
@ -295,5 +280,53 @@
</plugins> </plugins>
</build> </build>
</profile> </profile>
<profile>
<id>task-pure</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<dependencies>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-task-pure</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-pure</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-pure</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-storage-pure</artifactId>
</dependency>
</dependencies>
</profile>
<profile>
<id>task-all</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<dependencies>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-task-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-storage-all</artifactId>
</dependency>
</dependencies>
</profile>
</profiles> </profiles>
</project> </project>

View File

@ -17,7 +17,6 @@
package org.apache.dolphinscheduler.api.service.impl; package org.apache.dolphinscheduler.api.service.impl;
import static org.apache.dolphinscheduler.api.utils.CheckUtils.checkFilePath;
import static org.apache.dolphinscheduler.common.constants.Constants.ALIAS; import static org.apache.dolphinscheduler.common.constants.Constants.ALIAS;
import static org.apache.dolphinscheduler.common.constants.Constants.CONTENT; import static org.apache.dolphinscheduler.common.constants.Constants.CONTENT;
import static org.apache.dolphinscheduler.common.constants.Constants.EMPTY_STRING; import static org.apache.dolphinscheduler.common.constants.Constants.EMPTY_STRING;
@ -1291,10 +1290,6 @@ public class ResourcesServiceImpl extends BaseServiceImpl implements ResourcesSe
if (FOLDER_SEPARATOR.equalsIgnoreCase(fullName)) { if (FOLDER_SEPARATOR.equalsIgnoreCase(fullName)) {
return; return;
} }
// abnormal characters check
if (!checkFilePath(fullName)) {
throw new ServiceException(Status.ILLEGAL_RESOURCE_PATH);
}
// Avoid returning to the parent directory // Avoid returning to the parent directory
if (fullName.contains("../")) { if (fullName.contains("../")) {
throw new ServiceException(Status.ILLEGAL_RESOURCE_PATH, fullName); throw new ServiceException(Status.ILLEGAL_RESOURCE_PATH, fullName);

View File

@ -158,14 +158,4 @@ public class CheckUtils {
return pattern.matcher(str).matches(); return pattern.matcher(str).matches();
} }
/**
* regex FilePath check,only use a to z, A to Z, 0 to 9, and _./-
*
* @param str input string
* @return true if regex pattern is right, otherwise return false
*/
public static boolean checkFilePath(String str) {
return regexChecks(str, Constants.REGEX_FILE_PATH);
}
} }

View File

@ -92,24 +92,4 @@ public class CheckUtilsTest {
Assertions.assertTrue(CheckUtils.checkPhone("17362537263")); Assertions.assertTrue(CheckUtils.checkPhone("17362537263"));
} }
/**
* check file path
*/
@Test
public void testCheckFilePath() {
// true
Assertions.assertTrue(CheckUtils.checkFilePath("/"));
Assertions.assertTrue(CheckUtils.checkFilePath("xx/"));
Assertions.assertTrue(CheckUtils.checkFilePath("/xx"));
Assertions.assertTrue(CheckUtils.checkFilePath("14567134578654"));
Assertions.assertTrue(CheckUtils.checkFilePath("/admin/root/"));
Assertions.assertTrue(CheckUtils.checkFilePath("/admin/root/1531531..13513/153135.."));
// false
Assertions.assertFalse(CheckUtils.checkFilePath(null));
Assertions.assertFalse(CheckUtils.checkFilePath("file://xxx/ss"));
Assertions.assertFalse(CheckUtils.checkFilePath("/xxx/ss;/dasd/123"));
Assertions.assertFalse(CheckUtils.checkFilePath("/xxx/ss && /dasd/123"));
Assertions.assertFalse(CheckUtils.checkFilePath("/xxx/ss || /dasd/123"));
}
} }

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-bom</artifactId> <artifactId>dolphinscheduler-bom</artifactId>
<packaging>pom</packaging> <packaging>pom</packaging>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-common</artifactId> <artifactId>dolphinscheduler-common</artifactId>

View File

@ -249,11 +249,6 @@ public final class Constants {
*/ */
public static final Pattern REGEX_USER_NAME = Pattern.compile("^[a-zA-Z0-9._-]{3,39}$"); public static final Pattern REGEX_USER_NAME = Pattern.compile("^[a-zA-Z0-9._-]{3,39}$");
/**
* file path regex
*/
public static final Pattern REGEX_FILE_PATH = Pattern.compile("^[a-zA-Z0-9_./-]+$");
/** /**
* read permission * read permission
*/ */

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao-plugin</artifactId> <artifactId>dolphinscheduler-dao-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-dao-api</artifactId> <artifactId>dolphinscheduler-dao-api</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao-plugin</artifactId> <artifactId>dolphinscheduler-dao-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-dao-h2</artifactId> <artifactId>dolphinscheduler-dao-h2</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao-plugin</artifactId> <artifactId>dolphinscheduler-dao-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-dao-mysql</artifactId> <artifactId>dolphinscheduler-dao-mysql</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao-plugin</artifactId> <artifactId>dolphinscheduler-dao-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-dao-plugin-all</artifactId> <artifactId>dolphinscheduler-dao-plugin-all</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao-plugin</artifactId> <artifactId>dolphinscheduler-dao-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-dao-postgresql</artifactId> <artifactId>dolphinscheduler-dao-postgresql</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-dao-plugin</artifactId> <artifactId>dolphinscheduler-dao-plugin</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-dao</artifactId> <artifactId>dolphinscheduler-dao</artifactId>
<name>${project.artifactId}</name> <name>${project.artifactId}</name>

View File

@ -1070,7 +1070,7 @@ CREATE TABLE t_ds_version
-- Records of t_ds_version -- Records of t_ds_version
-- ---------------------------- -- ----------------------------
INSERT INTO t_ds_version INSERT INTO t_ds_version
VALUES ('1', '3.3.0'); VALUES ('1', '3.2.2');
-- ---------------------------- -- ----------------------------

View File

@ -1044,7 +1044,7 @@ CREATE TABLE `t_ds_version` (
-- ---------------------------- -- ----------------------------
-- Records of t_ds_version -- Records of t_ds_version
-- ---------------------------- -- ----------------------------
INSERT IGNORE INTO `t_ds_version` VALUES ('1', '3.3.0'); INSERT IGNORE INTO `t_ds_version` VALUES ('1', '3.2.2');
-- ---------------------------- -- ----------------------------

View File

@ -1063,7 +1063,7 @@ INSERT INTO t_ds_queue(queue_name, queue, create_time, update_time)
VALUES ('default', 'default', '2018-11-29 10:22:33', '2018-11-29 10:22:33'); VALUES ('default', 'default', '2018-11-29 10:22:33', '2018-11-29 10:22:33');
-- Records of t_ds_queue,default queue name : default -- Records of t_ds_queue,default queue name : default
INSERT INTO t_ds_version(version) VALUES ('3.3.0'); INSERT INTO t_ds_version(version) VALUES ('3.2.2');
-- --
-- Table structure for table t_ds_plugin_define -- Table structure for table t_ds_plugin_define

View File

@ -1 +1 @@
3.3.0 3.2.2

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-data-quality</artifactId> <artifactId>dolphinscheduler-data-quality</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-all</artifactId> <artifactId>dolphinscheduler-datasource-all</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-api</artifactId> <artifactId>dolphinscheduler-datasource-api</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-athena</artifactId> <artifactId>dolphinscheduler-datasource-athena</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-azure-sql</artifactId> <artifactId>dolphinscheduler-datasource-azure-sql</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-clickhouse</artifactId> <artifactId>dolphinscheduler-datasource-clickhouse</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-dameng</artifactId> <artifactId>dolphinscheduler-datasource-dameng</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-databend</artifactId> <artifactId>dolphinscheduler-datasource-databend</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-db2</artifactId> <artifactId>dolphinscheduler-datasource-db2</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-doris</artifactId> <artifactId>dolphinscheduler-datasource-doris</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-hana</artifactId> <artifactId>dolphinscheduler-datasource-hana</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-hive</artifactId> <artifactId>dolphinscheduler-datasource-hive</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-k8s</artifactId> <artifactId>dolphinscheduler-datasource-k8s</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-kyuubi</artifactId> <artifactId>dolphinscheduler-datasource-kyuubi</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-mysql</artifactId> <artifactId>dolphinscheduler-datasource-mysql</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-oceanbase</artifactId> <artifactId>dolphinscheduler-datasource-oceanbase</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-oracle</artifactId> <artifactId>dolphinscheduler-datasource-oracle</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-postgresql</artifactId> <artifactId>dolphinscheduler-datasource-postgresql</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-presto</artifactId> <artifactId>dolphinscheduler-datasource-presto</artifactId>

View File

@ -0,0 +1,66 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-pure</artifactId>
<dependencies>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-hive</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-mysql</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-postgresql</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-spark</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-ssh</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-doris</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-k8s</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</project>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-redshift</artifactId> <artifactId>dolphinscheduler-datasource-redshift</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-sagemaker</artifactId> <artifactId>dolphinscheduler-datasource-sagemaker</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-snowflake</artifactId> <artifactId>dolphinscheduler-datasource-snowflake</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-spark</artifactId> <artifactId>dolphinscheduler-datasource-spark</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-sqlserver</artifactId> <artifactId>dolphinscheduler-datasource-sqlserver</artifactId>

View File

@ -35,6 +35,7 @@ import java.sql.DriverManager;
import java.sql.SQLException; import java.sql.SQLException;
import java.util.List; import java.util.List;
import java.util.Map; import java.util.Map;
import java.util.stream.Collectors;
import com.alibaba.druid.sql.parser.SQLParserUtils; import com.alibaba.druid.sql.parser.SQLParserUtils;
import com.google.auto.service.AutoService; import com.google.auto.service.AutoService;
@ -128,8 +129,10 @@ public class SQLServerDataSourceProcessor extends AbstractDataSourceProcessor {
@Override @Override
public List<String> splitAndRemoveComment(String sql) { public List<String> splitAndRemoveComment(String sql) {
String cleanSQL = SQLParserUtils.removeComment(sql, com.alibaba.druid.DbType.sqlserver); return SQLParserUtils.splitAndRemoveComment(sql, com.alibaba.druid.DbType.sqlserver)
return SQLParserUtils.split(cleanSQL, com.alibaba.druid.DbType.sqlserver); .stream()
.map(subSql -> subSql.concat(";"))
.collect(Collectors.toList());
} }
private String transformOther(Map<String, String> otherMap) { private String transformOther(Map<String, String> otherMap) {

View File

@ -17,12 +17,15 @@
package org.apache.dolphinscheduler.plugin.datasource.sqlserver.param; package org.apache.dolphinscheduler.plugin.datasource.sqlserver.param;
import static com.google.common.truth.Truth.assertThat;
import org.apache.dolphinscheduler.common.constants.DataSourceConstants; import org.apache.dolphinscheduler.common.constants.DataSourceConstants;
import org.apache.dolphinscheduler.common.utils.JSONUtils; import org.apache.dolphinscheduler.common.utils.JSONUtils;
import org.apache.dolphinscheduler.plugin.datasource.api.utils.PasswordUtils; import org.apache.dolphinscheduler.plugin.datasource.api.utils.PasswordUtils;
import org.apache.dolphinscheduler.spi.enums.DbType; import org.apache.dolphinscheduler.spi.enums.DbType;
import java.util.HashMap; import java.util.HashMap;
import java.util.List;
import java.util.Map; import java.util.Map;
import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Assertions;
@ -95,4 +98,32 @@ public class SQLServerDataSourceProcessorTest {
Assertions.assertEquals(DataSourceConstants.SQLSERVER_VALIDATION_QUERY, Assertions.assertEquals(DataSourceConstants.SQLSERVER_VALIDATION_QUERY,
sqlServerDatasourceProcessor.getValidationQuery()); sqlServerDatasourceProcessor.getValidationQuery());
} }
@Test
void splitAndRemoveComment_singleSelect() {
String sql = "select * from table;";
List<String> subSqls = sqlServerDatasourceProcessor.splitAndRemoveComment(sql);
assertThat(subSqls).hasSize(1);
assertThat(subSqls.get(0)).isEqualTo("select * from table;");
}
@Test
void splitAndRemoveComment_singleMerge() {
String sql = "MERGE\n" +
" [ TOP ( expression ) [ PERCENT ] ]\n" +
" [ INTO ] <target_table> [ WITH ( <merge_hint> ) ] [ [ AS ] table_alias ]\n" +
" USING <table_source> [ [ AS ] table_alias ]\n" +
" ON <merge_search_condition>\n" +
" [ WHEN MATCHED [ AND <clause_search_condition> ]\n" +
" THEN <merge_matched> ] [ ...n ]\n" +
" [ WHEN NOT MATCHED [ BY TARGET ] [ AND <clause_search_condition> ]\n" +
" THEN <merge_not_matched> ]\n" +
" [ WHEN NOT MATCHED BY SOURCE [ AND <clause_search_condition> ]\n" +
" THEN <merge_matched> ] [ ...n ]\n" +
" [ <output_clause> ]\n" +
" [ OPTION ( <query_hint> [ ,...n ] ) ];";
List<String> subSqls = sqlServerDatasourceProcessor.splitAndRemoveComment(sql);
assertThat(subSqls).hasSize(1);
assertThat(subSqls.get(0)).isEqualTo(sql);
}
} }

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-ssh</artifactId> <artifactId>dolphinscheduler-datasource-ssh</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-starrocks</artifactId> <artifactId>dolphinscheduler-datasource-starrocks</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-trino</artifactId> <artifactId>dolphinscheduler-datasource-trino</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-vertica</artifactId> <artifactId>dolphinscheduler-datasource-vertica</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-zeppelin</artifactId> <artifactId>dolphinscheduler-datasource-zeppelin</artifactId>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-datasource-plugin</artifactId> <artifactId>dolphinscheduler-datasource-plugin</artifactId>
<packaging>pom</packaging> <packaging>pom</packaging>
@ -56,6 +56,7 @@
<module>dolphinscheduler-datasource-sagemaker</module> <module>dolphinscheduler-datasource-sagemaker</module>
<module>dolphinscheduler-datasource-k8s</module> <module>dolphinscheduler-datasource-k8s</module>
<module>dolphinscheduler-datasource-hana</module> <module>dolphinscheduler-datasource-hana</module>
<module>dolphinscheduler-datasource-pure</module>
</modules> </modules>
<dependencyManagement> <dependencyManagement>

View File

@ -21,7 +21,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-dist</artifactId> <artifactId>dolphinscheduler-dist</artifactId>

View File

@ -313,6 +313,8 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
jetty-xml 9.4.48.v20220622: https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-xml/9.4.48.v20220622, Apache 2.0 and EPL 1.0 jetty-xml 9.4.48.v20220622: https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-xml/9.4.48.v20220622, Apache 2.0 and EPL 1.0
jna 5.10.0: https://mvnrepository.com/artifact/net.java.dev.jna/jna/5.10.0, Apache 2.0 and LGPL 2.1 jna 5.10.0: https://mvnrepository.com/artifact/net.java.dev.jna/jna/5.10.0, Apache 2.0 and LGPL 2.1
jna-platform 5.10.0: https://mvnrepository.com/artifact/net.java.dev.jna/jna-platform/5.10.0, Apache 2.0 and LGPL 2.1 jna-platform 5.10.0: https://mvnrepository.com/artifact/net.java.dev.jna/jna-platform/5.10.0, Apache 2.0 and LGPL 2.1
jna 5.5.0: https://mvnrepository.com/artifact/net.java.dev.jna/jna/5.5.0, Apache 2.0 and LGPL 2.1
jna-platform 5.6.0: https://mvnrepository.com/artifact/net.java.dev.jna/jna-platform/5.6.0, Apache 2.0 and LGPL 2.1
joda-time 2.10.13: https://github.com/JodaOrg/joda-time, Apache 2.0 joda-time 2.10.13: https://github.com/JodaOrg/joda-time, Apache 2.0
jose4j 0.7.8: https://mvnrepository.com/artifact/org.bitbucket.b_c/jose4j, Apache 2.0 jose4j 0.7.8: https://mvnrepository.com/artifact/org.bitbucket.b_c/jose4j, Apache 2.0
jpam 1.1: https://mvnrepository.com/artifact/net.sf.jpam/jpam/1.1, Apache 2.0 jpam 1.1: https://mvnrepository.com/artifact/net.sf.jpam/jpam/1.1, Apache 2.0
@ -509,6 +511,7 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
third-party-jackson-core 2.17.282: https://mvnrepository.com/artifact/software.amazon.awssdk/third-party-jackson-core/2.17.282, Apache 2.0 third-party-jackson-core 2.17.282: https://mvnrepository.com/artifact/software.amazon.awssdk/third-party-jackson-core/2.17.282, Apache 2.0
utils 2.17.282: https://mvnrepository.com/artifact/software.amazon.awssdk/utils/2.17.282, Apache 2.0 utils 2.17.282: https://mvnrepository.com/artifact/software.amazon.awssdk/utils/2.17.282, Apache 2.0
zt-zip 1.15: https://github.com/zeroturnaround/zt-zip/blob/master/LICENSE, Apache 2.0 zt-zip 1.15: https://github.com/zeroturnaround/zt-zip/blob/master/LICENSE, Apache 2.0
content-type 2.2: https://mvnrepository.com/artifact/com.nimbusds/content-type/2.2, Apache 2.0 content-type 2.2: https://mvnrepository.com/artifact/com.nimbusds/content-type/2.2, Apache 2.0
jackson-dataformat-xml 2.13.3: https://mvnrepository.com/artifact/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.13.3, Apache 2.0 jackson-dataformat-xml 2.13.3: https://mvnrepository.com/artifact/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.13.3, Apache 2.0
lang-tag 1.6: https://mvnrepository.com/artifact/com.nimbusds/lang-tag/1.6, Apache 2.0 lang-tag 1.6: https://mvnrepository.com/artifact/com.nimbusds/lang-tag/1.6, Apache 2.0

View File

@ -24,7 +24,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-extract</artifactId> <artifactId>dolphinscheduler-extract</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-extract-alert</artifactId> <artifactId>dolphinscheduler-extract-alert</artifactId>

View File

@ -24,7 +24,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-extract</artifactId> <artifactId>dolphinscheduler-extract</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-extract-base</artifactId> <artifactId>dolphinscheduler-extract-base</artifactId>

View File

@ -24,7 +24,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-extract</artifactId> <artifactId>dolphinscheduler-extract</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>

View File

@ -24,7 +24,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-extract</artifactId> <artifactId>dolphinscheduler-extract</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-extract-master</artifactId> <artifactId>dolphinscheduler-extract-master</artifactId>

View File

@ -24,7 +24,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-extract</artifactId> <artifactId>dolphinscheduler-extract</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-extract-worker</artifactId> <artifactId>dolphinscheduler-extract-worker</artifactId>

View File

@ -24,7 +24,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-extract</artifactId> <artifactId>dolphinscheduler-extract</artifactId>

View File

@ -24,7 +24,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-master</artifactId> <artifactId>dolphinscheduler-master</artifactId>
@ -55,25 +55,11 @@
<artifactId>dolphinscheduler-service</artifactId> <artifactId>dolphinscheduler-service</artifactId>
</dependency> </dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-all</artifactId>
</dependency>
<dependency> <dependency>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-scheduler-all</artifactId> <artifactId>dolphinscheduler-scheduler-all</artifactId>
</dependency> </dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-task-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-storage-all</artifactId>
</dependency>
<dependency> <dependency>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-extract-master</artifactId> <artifactId>dolphinscheduler-extract-master</artifactId>
@ -339,5 +325,51 @@
</plugins> </plugins>
</build> </build>
</profile> </profile>
<profile>
<id>task-pure</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<dependencies>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-task-pure</artifactId>
<exclusions>
<exclusion>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-all</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-pure</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-storage-pure</artifactId>
</dependency>
</dependencies>
</profile>
<profile>
<id>task-all</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<dependencies>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-task-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-storage-all</artifactId>
</dependency>
</dependencies>
</profile>
</profiles> </profiles>
</project> </project>

View File

@ -24,7 +24,7 @@ import java.util.function.Supplier;
import lombok.experimental.UtilityClass; import lombok.experimental.UtilityClass;
import com.facebook.presto.jdbc.internal.guava.collect.ImmutableSet; import com.google.common.collect.ImmutableSet;
import io.micrometer.core.instrument.Counter; import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.Gauge; import io.micrometer.core.instrument.Gauge;

View File

@ -1226,12 +1226,12 @@ public class WorkflowExecuteRunnable implements IWorkflowExecuteRunnable {
|| state == TaskExecutionStatus.DISPATCH || state == TaskExecutionStatus.DISPATCH
|| state == TaskExecutionStatus.SUBMITTED_SUCCESS || state == TaskExecutionStatus.SUBMITTED_SUCCESS
|| state == TaskExecutionStatus.DELAY_EXECUTION) { || state == TaskExecutionStatus.DELAY_EXECUTION) {
// try to take over task instance if (state == TaskExecutionStatus.SUBMITTED_SUCCESS
if (state == TaskExecutionStatus.SUBMITTED_SUCCESS || state == TaskExecutionStatus.DELAY_EXECUTION || state == TaskExecutionStatus.DELAY_EXECUTION) {
|| state == TaskExecutionStatus.DISPATCH) {
// The taskInstance is not in running, directly takeover it // The taskInstance is not in running, directly takeover it
} else if (tryToTakeOverTaskInstance(existTaskInstance)) { } else if (tryToTakeOverTaskInstance(existTaskInstance)) {
log.info("Success take over task {}", existTaskInstance.getName()); // If the taskInstance has already dispatched to worker then will try to take-over it
log.info("Success take over task {} -> status: {}", existTaskInstance.getName(), state);
continue; continue;
} else { } else {
// set the task instance state to fault tolerance // set the task instance state to fault tolerance

View File

@ -61,26 +61,30 @@ public class SwitchLogicTask extends BaseSyncLogicTask<SwitchParameters> {
this.taskInstance = workflowExecuteRunnable.getTaskInstance(taskExecutionContext.getTaskInstanceId()) this.taskInstance = workflowExecuteRunnable.getTaskInstance(taskExecutionContext.getTaskInstanceId())
.orElseThrow(() -> new LogicTaskInitializeException( .orElseThrow(() -> new LogicTaskInitializeException(
"Cannot find the task instance in workflow execute runnable")); "Cannot find the task instance in workflow execute runnable"));
// Since the default branch is not in the dependTaskList, we need to add it to the end
// otherwise the default branch will never be skipped in DAGHelper
addDefaultBranchToEnd();
} }
@Override @Override
public void handle() throws MasterTaskExecuteException { public void handle() throws MasterTaskExecuteException {
// Calculate the condition result and get the next node
if (CollectionUtils.isEmpty(taskParameters.getDependTaskList())) { if (CollectionUtils.isEmpty(taskParameters.getDependTaskList())) {
// If the branch is empty then will go into the default branch
// This case shouldn't happen, we can directly throw exception and forbid the user to set branch
moveToDefaultBranch(); moveToDefaultBranch();
} else { } else {
calculateSwitchBranch(); calculateSwitchBranch();
} }
taskInstance.setSwitchDependency(taskParameters); taskInstance.setSwitchDependency(taskParameters);
log.info("Switch task execute finished");
taskExecutionContext.setCurrentExecutionStatus(TaskExecutionStatus.SUCCESS); taskExecutionContext.setCurrentExecutionStatus(TaskExecutionStatus.SUCCESS);
log.info("Switch task execute finished: {}", taskExecutionContext.getCurrentExecutionStatus().name());
} }
private void moveToDefaultBranch() { private void moveToDefaultBranch() {
checkIfBranchExist(taskParameters.getNextNode());
List<SwitchResultVo> switchResultVos = taskParameters.getDependTaskList(); List<SwitchResultVo> switchResultVos = taskParameters.getDependTaskList();
switchResultVos.add(new SwitchResultVo(null, taskParameters.getNextNode())); SwitchResultVo defaultSwitchResultVo = getDefaultSwitchResultVo();
checkIfBranchExist(defaultSwitchResultVo.getNextNode());
taskParameters.setResultConditionLocation(switchResultVos.size() - 1); taskParameters.setResultConditionLocation(switchResultVos.size() - 1);
log.info("The condition is not satisfied, move to the default branch: {}", log.info("The condition is not satisfied, move to the default branch: {}",
@ -90,9 +94,6 @@ public class SwitchLogicTask extends BaseSyncLogicTask<SwitchParameters> {
private void calculateSwitchBranch() { private void calculateSwitchBranch() {
List<SwitchResultVo> switchResultVos = taskParameters.getDependTaskList(); List<SwitchResultVo> switchResultVos = taskParameters.getDependTaskList();
if (CollectionUtils.isEmpty(switchResultVos)) {
moveToDefaultBranch();
}
Map<String, Property> globalParams = taskExecutionContext.getPrepareParamsMap(); Map<String, Property> globalParams = taskExecutionContext.getPrepareParamsMap();
Map<String, Property> varParams = JSONUtils Map<String, Property> varParams = JSONUtils
.toList(taskInstance.getVarPool(), Property.class) .toList(taskInstance.getVarPool(), Property.class)
@ -100,7 +101,8 @@ public class SwitchLogicTask extends BaseSyncLogicTask<SwitchParameters> {
.collect(Collectors.toMap(Property::getProp, Property -> Property)); .collect(Collectors.toMap(Property::getProp, Property -> Property));
int finalConditionLocation = -1; int finalConditionLocation = -1;
for (int i = 0; i < switchResultVos.size(); i++) { // The last one is the default branch, no need to calculate
for (int i = 0; i < switchResultVos.size() - 1; i++) {
SwitchResultVo switchResultVo = switchResultVos.get(i); SwitchResultVo switchResultVo = switchResultVos.get(i);
log.info("Begin to execute {} condition: {} ", i, switchResultVo.getCondition()); log.info("Begin to execute {} condition: {} ", i, switchResultVo.getCondition());
String content = SwitchTaskUtils.generateContentWithTaskParams(switchResultVo.getCondition(), globalParams, String content = SwitchTaskUtils.generateContentWithTaskParams(switchResultVo.getCondition(), globalParams,
@ -111,14 +113,18 @@ public class SwitchLogicTask extends BaseSyncLogicTask<SwitchParameters> {
result = SwitchTaskUtils.evaluate(content); result = SwitchTaskUtils.evaluate(content);
log.info("Execute condition sentence: {} successfully: {}", content, result); log.info("Execute condition sentence: {} successfully: {}", content, result);
if (result) { if (result) {
// If matched, break the loop
finalConditionLocation = i; finalConditionLocation = i;
break;
} }
} catch (Exception e) { } catch (Exception e) {
log.info("Execute condition sentence: {} failed", content, e); log.info("Execute condition sentence: {} failed", content, e);
} }
} }
// If the finalConditionLocation is -1, then the default branch will be executed
if (finalConditionLocation >= 0) { if (finalConditionLocation >= 0) {
checkIfBranchExist(switchResultVos.get(finalConditionLocation).getNextNode()); List<Long> nextNodes = switchResultVos.get(finalConditionLocation).getNextNode();
checkIfBranchExist(nextNodes);
log.info("The condition is satisfied, move to the branch: {}", log.info("The condition is satisfied, move to the branch: {}",
switchResultVos.get(finalConditionLocation).getNextNode().stream() switchResultVos.get(finalConditionLocation).getNextNode().stream()
.map(node -> workflowExecuteRunnable.getWorkflowExecuteContext().getWorkflowGraph().getDag() .map(node -> workflowExecuteRunnable.getWorkflowExecuteContext().getWorkflowGraph().getDag()
@ -126,7 +132,6 @@ public class SwitchLogicTask extends BaseSyncLogicTask<SwitchParameters> {
.collect(Collectors.toList())); .collect(Collectors.toList()));
taskParameters.setResultConditionLocation(finalConditionLocation); taskParameters.setResultConditionLocation(finalConditionLocation);
} else { } else {
log.info("All conditions are not satisfied, move to the default branch");
moveToDefaultBranch(); moveToDefaultBranch();
} }
} }
@ -147,4 +152,16 @@ public class SwitchLogicTask extends BaseSyncLogicTask<SwitchParameters> {
} }
} }
private void addDefaultBranchToEnd() {
SwitchResultVo switchResultVo = new SwitchResultVo(null, taskParameters.getNextNode());
List<SwitchResultVo> dependTaskList = taskParameters.getDependTaskList();
if (!dependTaskList.contains(switchResultVo)) {
dependTaskList.add(switchResultVo);
}
}
private SwitchResultVo getDefaultSwitchResultVo() {
return taskParameters.getDependTaskList().get(taskParameters.getDependTaskList().size() - 1);
}
} }

View File

@ -63,7 +63,6 @@ public class SwitchTaskUtils {
if (MapUtils.isNotEmpty(varParams)) { if (MapUtils.isNotEmpty(varParams)) {
params.putAll(varParams); params.putAll(varParams);
} }
String originContent = content;
Pattern pattern = Pattern.compile(rgex); Pattern pattern = Pattern.compile(rgex);
Matcher m = pattern.matcher(content); Matcher m = pattern.matcher(content);
while (m.find()) { while (m.find()) {
@ -82,10 +81,6 @@ public class SwitchTaskUtils {
content = content.replace("${" + paramName + "}", value); content = content.replace("${" + paramName + "}", value);
} }
// if not replace any params, throw exception to avoid illegal condition
if (originContent.equals(content)) {
throw new IllegalArgumentException("condition is not valid, please check it. condition: " + condition);
}
return content; return content;
} }

View File

@ -50,7 +50,7 @@ public class SwitchTaskUtilsTest {
Map<String, Property> globalParams = new HashMap<>(); Map<String, Property> globalParams = new HashMap<>();
Map<String, Property> varParams = new HashMap<>(); Map<String, Property> varParams = new HashMap<>();
globalParams.put("test", new Property("test", Direct.IN, DataType.INTEGER, "1")); globalParams.put("test", new Property("test", Direct.IN, DataType.INTEGER, "1"));
Assertions.assertThrowsExactly(IllegalArgumentException.class, () -> { Assertions.assertDoesNotThrow(() -> {
SwitchTaskUtils.generateContentWithTaskParams(content, globalParams, varParams); SwitchTaskUtils.generateContentWithTaskParams(content, globalParams, varParams);
}); });
@ -70,15 +70,5 @@ public class SwitchTaskUtilsTest {
SwitchTaskUtils.evaluate(script); SwitchTaskUtils.evaluate(script);
}); });
String contentWithSpecify1 = "cmd.abc";
Assertions.assertThrowsExactly(IllegalArgumentException.class, () -> {
SwitchTaskUtils.generateContentWithTaskParams(contentWithSpecify1, globalParams, varParams);
});
String contentWithSpecify2 = "cmd()";
Assertions.assertThrowsExactly(IllegalArgumentException.class, () -> {
SwitchTaskUtils.generateContentWithTaskParams(contentWithSpecify2, globalParams, varParams);
});
} }
} }

View File

@ -24,7 +24,7 @@
<parent> <parent>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId> <artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</parent> </parent>
<artifactId>dolphinscheduler-meter</artifactId> <artifactId>dolphinscheduler-meter</artifactId>
@ -45,7 +45,7 @@
<dependency> <dependency>
<groupId>org.apache.dolphinscheduler</groupId> <groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-common</artifactId> <artifactId>dolphinscheduler-common</artifactId>
<version>dev-SNAPSHOT</version> <version>3.2.3-SNAPSHOT</version>
</dependency> </dependency>
<dependency> <dependency>

Some files were not shown because too many files have changed in this diff Show More