Compare commits

...

7 Commits

Author SHA1 Message Date
wangxj3 3ad53c5200
3.2.2 release add task-pure delete some task module to reduce the size of the deployment package (#16094)
* add task-pure delete some task module to reduce the size of the deployment package

* add version of 3.2.2

* Update Chart.yaml Rollback APPversion

* remove task pure plugin

* fix ci

* add pure datasource plugin

* remove ck,sr from datasource pure

* remove task plugin chunjun ,dms ,datafactory from pure

* add api pom dependency

* spotless:apply

* add dependency license

* [maven-release-plugin] prepare release 3.2.2

* [maven-release-plugin] prepare for next development iteration

* remove datasource-pure:
dolphinscheduler-datasource-sqlserver
dolphinscheduler-datasource-oracle

remove task-pure:
dolphinscheduler-task-dinky
dolphinscheduler-task-pytorch
dolphinscheduler-task-datasync
dolphinscheduler-task-kubeflow
dolphinscheduler-task-emr
dolphinscheduler-task-pigeon
dolphinscheduler-task-flink-stream

* add register pure remove jdbc and etcd

* add dolphinscheduler-storage-pure remove task of
dolphinscheduler-storage-obs
dolphinscheduler-storage-abs
dolphinscheduler-storage-gcs
dolphinscheduler-storage-oss

* mvn spotless:apply

---------

Co-authored-by: wangxj959 <wangxj959@chinaunicom.cn>
Co-authored-by: wangxj <wangxj31>
Co-authored-by: wangxj3 <wangxj@apache.org>
2024-06-08 17:05:20 +08:00
Wenjun Ruan e91a272965
Fix SqlServer split merge sql will miss ; (#16118) (#16120)
(cherry picked from commit d6714bb77a)
2024-06-06 17:11:51 +08:00
Wenjun Ruan 31e79d10ea
Fix task nodes under switch task will not be skipped (#16109)
(cherry picked from commit f687a7d851c206ae30377ac3f9dc025cec35ffb2)
2024-06-05 23:41:52 +08:00
Wenjun Ruan 50c434e6c4
Fix task instance which status is dispatch will not be failover (#16110)
(cherry picked from commit a59ecf2aa19f4acd8d9981d14abe4d0c0d8e7fec)
2024-06-05 13:31:29 +08:00
Wenjun Ruan 72b402422a
Revert "[Improvement] Abnormal characters check (#15824)" (#16102) (#16103)
This reverts commit e5e7749251.

(cherry picked from commit 7ae9ca5a33)
2024-05-31 21:21:14 +08:00
xiangzihao fd864bf351
Revert "3.2.2 release cherry-pick cp: Reduce the size of tarball to continue …" (#16068)
This reverts commit 646bbf4881.
2024-05-25 10:50:37 +08:00
wangxj3 646bbf4881
3.2.2 release cherry-pick cp: Reduce the size of tarball to continue ASF release (#16042)
* add version of 3.2.2

* Fix rpc channel leak due to concurrent operation (#16021)

* Fix rpc channel leak due to concurrent operation

* Throw channel create failed exception

---------

Co-authored-by: Rick Cheng <rickchengx@gmail.com>

* Fix WorkerTaskExecutorThreadPool#isOverload is incorrect (#16027)

* cp: Reduce the size of tarball to continue ASF release (#15004) (#15540)

* Reduce the size of tarball to continue ASF release

for more detail you can see https://lists.apache.org/thread/rmp7fghlj0n7h9y2v3p8gkw9f9qbo6qt
# Conflicts:
#	tools/dependencies/known-dependencies.txt

* Increase block-until-connected in ZookeeperRegistryTestCase (#16041)

* [Fix][CI] fix the ci error of Values.datasource.profile (#16031)

Co-authored-by: Eric Gao <ericgao.apache@gmail.com>

* Correction of Typos in the Chinese Document Appendix for Task Parameters (#16033)

Co-authored-by: Rick Cheng <rickchengx@gmail.com>

* [DSIP-39][parameter] Improvement startup parameters/global parameters/project parameters data type (#15967)

* [Improvement][parameter] New data types and type filtering

* [Improvement][parameter] Improvement startup parameters/global parameters data type

* fix api interfaces compatible

* add project parameter data type default value

* [Improvement][parameter] New data types and type filtering

* [Improvement][parameter] Improvement startup parameters/global parameters data type

* fix api interfaces compatible

* add project parameter data type default value

* improvement project code

* remove useless imports

* remove method onClearSearchTaskType

* add parameter doc

* optimisation logic

* code conflict resolution

* code conflict resolution

* [Improvement][Monitor] Show master && worker Busy Or Normal Status and Show Commands table list (#15978)

* update

* test

* add monitor enhance ui

* update

* update

* update doc

* fix spotless

* update

* update

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/CommandMapper.java

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* update

* fix spotless

* update

---------

Co-authored-by: Wenjun Ruan <wenjun@apache.org>

* [Improvement][Monitor] Add UT for montor (#15998)

* formatting Code

* add pom

* Fix NettyRemotingClient might throw IllegalMonitorStateException (#16038)

* Removed unused StateEventHandleFailure (#16052)

* Remove OmitStackTraceInFastThrow in start.sh (#16054)

* Update Chart.yaml Rollback APPversion

* modify log msg (#16062)

Co-authored-by: xiangzihao <460888207@qq.com>

* [DS-16046][fix] Set PreparedStatement parameter type (#16050)

Fix the PreparedStatement parameter is TIME, set it to java.sql.Time

Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>

* Update pom.xml

* [DSIP-42] Add dolphinscheduler-aws-authentication module  (#16043)

---------

Co-authored-by: wangxj <wangxj31>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
Co-authored-by: TianXinCoord <tianxincoord@163.com>
Co-authored-by: 小可耐 <46134044+sdhzwc@users.noreply.github.com>
Co-authored-by: sleo <97011595+alei1206@users.noreply.github.com>
Co-authored-by: xiangzihao <460888207@qq.com>
Co-authored-by: yinxiaolog <43245392+yinxiaolog@users.noreply.github.com>
Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
2024-05-25 10:16:18 +08:00
170 changed files with 1080 additions and 639 deletions

View File

@ -15,8 +15,8 @@
# specific language governing permissions and limitations
# under the License.
#
HUB=ghcr.io/apache/dolphinscheduler
TAG=latest
HUB=apache
TAG=3.2.2
TZ=Asia/Shanghai
DATABASE=postgresql

View File

@ -35,7 +35,7 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
version: 3.1.0
version: 3.2.2
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application.

View File

@ -33,7 +33,7 @@ image:
# -- Docker image repository for the DolphinScheduler
registry: apache
# -- Docker image version for the DolphinScheduler
tag: latest
tag: 3.2.2
# -- Image pull policy. Options: Always, Never, IfNotPresent
pullPolicy: "IfNotPresent"
# -- Specify a imagePullSecrets

File diff suppressed because it is too large Load Diff

View File

@ -340,7 +340,7 @@ svn --username="${A_USERNAME}" commit -m "release ${VERSION}"
#### Check source package
Decompress `apache-dolphinscheduler-<VERSION>-src.tar.gz` then check the following items:
Decompress `apache-dolphinscheduler-3.2.2-src.tar.gz` then check the following items:
- Check whether source tarball is oversized for including nonessential files
- `LICENSE` and `NOTICE` files exist
@ -353,7 +353,7 @@ Decompress `apache-dolphinscheduler-<VERSION>-src.tar.gz` then check the followi
#### Check binary packages
Decompress `apache-dolphinscheduler-<VERSION>-bin.tar.gz` to check the following items:
Decompress `apache-dolphinscheduler-3.2.2-bin.tar.gz` to check the following items:
- `LICENSE` and `NOTICE` files exist
- Correct year in `NOTICE` file
@ -389,7 +389,7 @@ Announce the vote result: send the result vote e-mail to `dev@dolphinscheduler.a
Title
```txt
[VOTE] Release Apache DolphinScheduler <VERSION>
[VOTE] Release Apache DolphinScheduler 3.2.2
```
Body
@ -397,15 +397,15 @@ Body
```txt
Hello DolphinScheduler Community,
This is a call for vote to release Apache DolphinScheduler version <VERSION>
This is a call for vote to release Apache DolphinScheduler version 3.2.2
Release notes: https://github.com/apache/dolphinscheduler/releases/tag/<VERSION>
Release notes: https://github.com/apache/dolphinscheduler/releases/tag/3.2.2
The release candidates: https://dist.apache.org/repos/dist/dev/dolphinscheduler/<VERSION>/
The release candidates: https://dist.apache.org/repos/dist/dev/dolphinscheduler/3.2.2/
Maven 2 staging repository: https://repository.apache.org/content/repositories/<STAGING.REPOSITORY>/org/apache/dolphinscheduler/
Git tag for the release: https://github.com/apache/dolphinscheduler/tree/<VERSION>
Git tag for the release: https://github.com/apache/dolphinscheduler/tree/3.2.2
Release Commit ID: https://github.com/apache/dolphinscheduler/commit/<SHA-VALUE>
@ -436,13 +436,13 @@ Checklist for reference:
Title
```txt
[RESULT][VOTE] Release Apache DolphinScheduler <VERSION>
[RESULT][VOTE] Release Apache DolphinScheduler 3.2.2
```
Body
```txt
The vote to release Apache DolphinScheduler <VERSION> has passed.Here is the vote result,
The vote to release Apache DolphinScheduler 3.2.2 has passed.Here is the vote result,
4 PMC member +1 votes:
@ -478,19 +478,19 @@ and then find DolphinScheduler in [apache staging repositories](https://reposito
### Update Document
Website should be present before you send the announce mail this section will tell you how to change the website. For example,
the release version is `<VERSION>`, the following updates are required(note it will take effect immediately when the PR is merged):
the release version is `3.2.2`, the following updates are required(note it will take effect immediately when the PR is merged):
- Repository **apache/dolphinscheduler-website**:
- `config/download.json`: add the download of the `<VERSION>` release package
- `scripts/conf.sh`: Add new release version `<VERSION>` key-value pair to variable `DEV_RELEASE_DOCS_VERSIONS`
- `config/download.json`: add the download of the `3.2.2` release package
- `scripts/conf.sh`: Add new release version `3.2.2` key-value pair to variable `DEV_RELEASE_DOCS_VERSIONS`
- Repository **apache/dolphinscheduler** (dev branch):
- `docs/configs/site.js`:
- `docsLatest`: update to `<VERSION>`
- `docs0`: The `text` of two places of `en-us/zh-cn` needs to be updated to `latest(<VERSION>)`
- `docs/configs/index.md.jsx`: Add `<VERSION>: docsxyzConfig` and add new `import` for the new `docsxyzConfig`
- `docs/docs/en/history-versions.md` and `docs/docs/zh/history-versions.md`: Add new `<VERSION>` release docs.
- `docsLatest`: update to `3.2.2`
- `docs0`: The `text` of two places of `en-us/zh-cn` needs to be updated to `latest(3.2.2)`
- `docs/configs/index.md.jsx`: Add `3.2.2: docsxyzConfig` and add new `import` for the new `docsxyzConfig`
- `docs/docs/en/history-versions.md` and `docs/docs/zh/history-versions.md`: Add new `3.2.2` release docs.
- `.github/ISSUE_TEMPLATE/bug-report.yml`: DolphinScheduler's GitHub [bug-report](https://github.com/apache/dolphinscheduler/blob/dev/.github/ISSUE_TEMPLATE/bug-report.yml)
issue template have **Version** selection bottom. So after released we should add the new `<VERSION>` to
issue template have **Version** selection bottom. So after released we should add the new `3.2.2` to
bug-report.yml
### Publish Docker Image and Helm Chart
@ -511,7 +511,7 @@ Announcement e-mail template as below
Title
```txt
[ANNOUNCE] Release Apache DolphinScheduler <VERSION>
[ANNOUNCE] Release Apache DolphinScheduler 3.2.2
```
Body
@ -519,7 +519,7 @@ Body
```txt
Hi all,
We are glad to announce the release of Apache DolphinScheduler <VERSION>. Once again I would like to express my thanks to your help.
We are glad to announce the release of Apache DolphinScheduler 3.2.2. Once again I would like to express my thanks to your help.
Dolphin Scheduler is a distributed and easy-to-extend visual workflow scheduler system,
dedicated to solving the complex task dependencies in data processing, making the scheduler system out of the box for data processing.
@ -527,14 +527,14 @@ dedicated to solving the complex task dependencies in data processing, making th
Download Links: https://dolphinscheduler.apache.org/en-us/download
Release Notes: https://github.com/apache/dolphinscheduler/releases/tag/<VERSION>
Release Notes: https://github.com/apache/dolphinscheduler/releases/tag/3.2.2
Website: https://dolphinscheduler.apache.org/
DolphinScheduler Resources:
- Issue: https://github.com/apache/dolphinscheduler/issues/
- Mailing list: dev@dolphinscheduler.apache.org
- Documents: https://dolphinscheduler.apache.org/en-us/docs/<VERSION>/about/introduction
- Documents: https://dolphinscheduler.apache.org/en-us/docs/3.2.2/about/introduction
```
## News

View File

@ -30,9 +30,9 @@ Attention: DolphinScheduler itself does not depend on Hadoop, Hive, Spark, but w
mkdir -p /opt
cd /opt
# decompress
tar -zxvf apache-dolphinscheduler-<version>-bin.tar.gz -C /opt
tar -zxvf apache-dolphinscheduler-3.2.2-bin.tar.gz -C /opt
cd /opt
mv apache-dolphinscheduler-<version>-bin dolphinscheduler
mv apache-dolphinscheduler-3.2.2-bin dolphinscheduler
```
```markdown

View File

@ -233,9 +233,9 @@ kubectl scale --replicas=6 sts dolphinscheduler-worker -n test # with test names
2. Create a new `Dockerfile` to add MySQL driver:
```
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:<version>
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:3.2.2
# For example
# FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-tools:<version>
# FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-tools:3.2.2
# Attention Please, If the build is dolphinscheduler-tools image
# You need to change the following line to: COPY mysql-connector-java-8.0.16.jar /opt/dolphinscheduler/tools/libs
@ -284,9 +284,9 @@ externalDatabase:
2. Create a new `Dockerfile` to add MySQL or Oracle driver:
```
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:<version>
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:3.2.2
# For example
# FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:<version>
# FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:3.2.2
# If you want to support MySQL Datasource
COPY mysql-connector-java-8.0.16.jar /opt/dolphinscheduler/libs
@ -316,7 +316,7 @@ docker build -t apache/dolphinscheduler-<service>:new-driver .
1. Create a new `Dockerfile` to install pip:
```
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:<version>
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:3.2.2
COPY requirements.txt /tmp
RUN apt-get update && \
apt-get install -y --no-install-recommends python-pip && \
@ -351,7 +351,7 @@ docker build -t apache/dolphinscheduler-worker:pip .
1. Create a new `Dockerfile` to install Python 3:
```
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:<version>
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:3.2.2
RUN apt-get update && \
apt-get install -y --no-install-recommends python3 && \
rm -rf /var/lib/apt/lists/*

View File

@ -12,7 +12,7 @@ If you want to deploy DolphinScheduler in production, we recommend you follow [c
## Preparation
- JDKdownload [JDK][jdk] (1.8+), install and configure environment variable `JAVA_HOME` and append `bin` dir (included in `JAVA_HOME`) to `PATH` variable. You can skip this step if it already exists in your environment.
- Binary package: download the DolphinScheduler binary package at [download page](https://dolphinscheduler.apache.org/en-us/download/<version>). <!-- markdown-link-check-disable-line -->
- Binary package: download the DolphinScheduler binary package at [download page](https://dolphinscheduler.apache.org/en-us/download/3.2.2). <!-- markdown-link-check-disable-line -->
## Start DolphinScheduler Standalone Server

View File

@ -19,7 +19,7 @@ Start DolphinScheduler with standalone-server Docker images is the easiest way t
you can learn DolphinScheduler's concepts and usage, with minimal cost.
```shell
$ DOLPHINSCHEDULER_VERSION=<version>
$ DOLPHINSCHEDULER_VERSION=3.2.2
$ docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:25333 -d apache/dolphinscheduler-standalone-server:"${DOLPHINSCHEDULER_VERSION}"
```
@ -37,11 +37,11 @@ be stored on disks after you change docker-compose configuration, and it is robu
DolphinScheduler in a long term. You have to install [docker-compose](https://docs.docker.com/compose/install/) before you
start servers.
After complete the installation, get the `docker-compose.yaml` file from [download page](https://dolphinscheduler.apache.org/en-us/download/<version>)
After complete the installation, get the `docker-compose.yaml` file from [download page](https://dolphinscheduler.apache.org/en-us/download/3.2.2)
form its source package, and make sure you get the right version. After download the package, you can run the commands as below.
```shell
$ DOLPHINSCHEDULER_VERSION=<version>
$ DOLPHINSCHEDULER_VERSION=3.2.2
$ tar -zxf apache-dolphinscheduler-"${DOLPHINSCHEDULER_VERSION}"-src.tar.gz
# Going to docker-compose's location
# For Mac or Linux users
@ -68,7 +68,7 @@ $ docker-compose --profile all up -d
container when it up. You could start DolphinScheduler server separately if you want to reuse your exists services.
```shell
$ DOLPHINSCHEDULER_VERSION=<version>
$ DOLPHINSCHEDULER_VERSION=3.2.2
# Initialize the database, make sure database <DATABASE> already exists
$ docker run -d --name dolphinscheduler-tools \
-e DATABASE="postgresql" \
@ -134,5 +134,5 @@ and use `admin` and `dolphinscheduler123` as default username and password in th
You can modify some environment variables to change configurations when you are starting servers through Docker. We have
an example in [using exists PostgreSQL ZooKeeper](#using-exists-postgresql-zookeeper) to change database and ZooKeeper configurations,
and you could find all environment variables in [all environment variables](https://github.com/apache/dolphinscheduler/blob/<version>/script/env/dolphinscheduler_env.sh) <!-- markdown-link-check-disable-line -->
and you could find all environment variables in [all environment variables](https://github.com/apache/dolphinscheduler/blob/3.2.2/script/env/dolphinscheduler_env.sh) <!-- markdown-link-check-disable-line -->
and change them if you want.

View File

@ -4,6 +4,10 @@
#### Setup instructions, are available for each stable version of Apache DolphinScheduler below:
### Versions: 3.2.2
#### Links [3.2.2 Document](../3.2.2/user_doc/about/introduction.md)
### Versions: 3.2.1
#### Links [3.2.1 Document](../3.2.1/user_doc/about/introduction.md)

View File

@ -347,7 +347,7 @@ svn --username="${A_USERNAME}" commit -m "release ${VERSION}"
#### 检查源码包的文件内容
解压缩`apache-dolphinscheduler-<VERSION>-src.tar.gz`,进行如下检查:
解压缩`apache-dolphinscheduler-3.2.2-src.tar.gz`,进行如下检查:
- 检查源码包是否包含由于包含不必要文件,致使 tarball 过于庞大
- 存在`LICENSE`和`NOTICE`文件
@ -359,7 +359,7 @@ svn --username="${A_USERNAME}" commit -m "release ${VERSION}"
#### 检查二进制包的文件内容
解压缩`apache-dolphinscheduler-<VERSION>-bin.tar.gz`进行如下检查:
解压缩`apache-dolphinscheduler-3.2.2-bin.tar.gz`进行如下检查:
- 存在`LICENSE`和`NOTICE`文件
- 所有文本文件开头都有 ASF 许可证
@ -394,7 +394,7 @@ DolphinScheduler 社区投票,发起投票邮件到`dev@dolphinscheduler.apach
标题:
```txt
[VOTE] Release Apache DolphinScheduler <VERSION>
[VOTE] Release Apache DolphinScheduler 3.2.2
```
正文:
@ -402,15 +402,15 @@ DolphinScheduler 社区投票,发起投票邮件到`dev@dolphinscheduler.apach
```txt
Hello DolphinScheduler Community,
This is a call for vote to release Apache DolphinScheduler version <VERSION>
This is a call for vote to release Apache DolphinScheduler version 3.2.2
Release notes: https://github.com/apache/dolphinscheduler/releases/tag/<VERSION>
Release notes: https://github.com/apache/dolphinscheduler/releases/tag/3.2.2
The release candidates: https://dist.apache.org/repos/dist/dev/dolphinscheduler/<VERSION>/
The release candidates: https://dist.apache.org/repos/dist/dev/dolphinscheduler/3.2.2/
Maven 2 staging repository: https://repository.apache.org/content/repositories/<STAGING.REPOSITORY>/org/apache/dolphinscheduler/
Git tag for the release: https://github.com/apache/dolphinscheduler/tree/<VERSION>
Git tag for the release: https://github.com/apache/dolphinscheduler/tree/3.2.2
Release Commit ID: https://github.com/apache/dolphinscheduler/commit/<SHA-VALUE>
@ -441,11 +441,11 @@ Checklist for reference:
Title
```txt
[RESULT][VOTE] Release Apache DolphinScheduler <VERSION>
[RESULT][VOTE] Release Apache DolphinScheduler 3.2.2
```
```txt
The vote to release Apache DolphinScheduler <VERSION> has passed.Here is the vote result,
The vote to release Apache DolphinScheduler 3.2.2 has passed.Here is the vote result,
4 PMC member +1 votes:
@ -480,17 +480,17 @@ git push --delete "${GH_REMOTE}" "${VERSION}-prepare"
### 更新文档
官网应该在您发送通知邮件之前完成更新,本节将告诉您如何更改网站。假设发版的版本是 `<VERSION>`,需要进行以下更新(注意,当修改 pull requests 被 merge 后就会生效):
官网应该在您发送通知邮件之前完成更新,本节将告诉您如何更改网站。假设发版的版本是 `3.2.2`,需要进行以下更新(注意,当修改 pull requests 被 merge 后就会生效):
- **apache/dolphinscheduler-website** 仓库:
- `config/download.json`: 增加 `<VERSION>` 版本发布包的下载
- `scripts/conf.sh`: 在变量 `DEV_RELEASE_DOCS_VERSIONS` 中增加版本为 `<VERSION>` 的新键值对
- `config/download.json`: 增加 `3.2.2` 版本发布包的下载
- `scripts/conf.sh`: 在变量 `DEV_RELEASE_DOCS_VERSIONS` 中增加版本为 `3.2.2` 的新键值对
- **apache/dolphinscheduler** 仓库 (dev 分支)
- `docs/configs/site.js`:
- `docsLatest`: 更新为 `<VERSION>`
- `docs0`: 两处 `en-us/zh-cn``text` 更新为 `latest(<VERSION>)`
- `docs/configs/index.md.jsx`: 增加 `'<VERSION>': docsxyzConfig,` 以及新的 `import`
- `docs/docs/en/history-versions.md``docs/docs/zh/history-versions.md`: 增加新的发版版本 `<VERSION>` 的链接
- `docsLatest`: 更新为 `3.2.2`
- `docs0`: 两处 `en-us/zh-cn``text` 更新为 `latest(3.2.2)`
- `docs/configs/index.md.jsx`: 增加 `'3.2.2': docsxyzConfig,` 以及新的 `import`
- `docs/docs/en/history-versions.md``docs/docs/zh/history-versions.md`: 增加新的发版版本 `3.2.2` 的链接
- `.github/ISSUE_TEMPLATE/bug-report.yml`: DolphinScheduler 在 GitHub bug report 的 issue 中有版本选择,当有新的版本发版后,需要更新
[bug-report](https://github.com/apache/dolphinscheduler/blob/dev/.github/ISSUE_TEMPLATE/bug-report.yml) 中的 **Version** 部分。
@ -510,7 +510,7 @@ git push --delete "${GH_REMOTE}" "${VERSION}-prepare"
标题:
```txt
[ANNOUNCE] Release Apache DolphinScheduler <VERSION>
[ANNOUNCE] Release Apache DolphinScheduler 3.2.2
```
正文:
@ -518,7 +518,7 @@ git push --delete "${GH_REMOTE}" "${VERSION}-prepare"
```txt
Hi all,
We are glad to announce the release of Apache DolphinScheduler <VERSION>. Once again I would like to express my thanks to your help.
We are glad to announce the release of Apache DolphinScheduler 3.2.2. Once again I would like to express my thanks to your help.
Dolphin Scheduler is a distributed and easy-to-extend visual workflow scheduler system,
dedicated to solving the complex task dependencies in data processing, making the scheduler system out of the box for data processing.
@ -526,14 +526,14 @@ dedicated to solving the complex task dependencies in data processing, making th
Download Links: https://dolphinscheduler.apache.org/zh-cn/download
Release Notes: https://github.com/apache/dolphinscheduler/releases/tag/<VERSION>
Release Notes: https://github.com/apache/dolphinscheduler/releases/tag/3.2.2
Website: https://dolphinscheduler.apache.org/
DolphinScheduler Resources:
- Issue: https://github.com/apache/dolphinscheduler/issues/
- Mailing list: dev@dolphinscheduler.apache.org
- Documents: https://dolphinscheduler.apache.org/zh-cn/docs/<VERSION>/about/introduction
- Documents: https://dolphinscheduler.apache.org/zh-cn/docs/3.2.2/about/introduction
```
## News

View File

@ -30,9 +30,9 @@
mkdir -p /opt
cd /opt
# 解压缩
tar -zxvf apache-dolphinscheduler-<version>-bin.tar.gz -C /opt
tar -zxvf apache-dolphinscheduler-3.2.2-bin.tar.gz -C /opt
cd /opt
mv apache-dolphinscheduler-<version>-bin dolphinscheduler
mv apache-dolphinscheduler-3.2.2-bin dolphinscheduler
```
```markdown

View File

@ -232,9 +232,9 @@ kubectl scale --replicas=6 sts dolphinscheduler-worker -n test # with test names
2. 创建一个新的 `Dockerfile`,用于添加 MySQL 的驱动包:
```
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:<version>
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:3.2.2
# 例如
# FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-tools:<version>
# FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-tools:3.2.2
# 注意如果构建的是dolphinscheduler-tools镜像
# 需要将下面一行修改为COPY mysql-connector-java-8.0.16.jar /opt/dolphinscheduler/tools/libs
@ -283,9 +283,9 @@ externalDatabase:
2. 创建一个新的 `Dockerfile`,用于添加 MySQL 或者 Oracle 驱动包:
```
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:<version>
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-<service>:3.2.2
# 例如
# FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:<version>
# FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:3.2.2
# 如果你想支持 MySQL 数据源
COPY mysql-connector-java-8.0.16.jar /opt/dolphinscheduler/libs
@ -315,7 +315,7 @@ docker build -t apache/dolphinscheduler-<service>:new-driver .
1. 创建一个新的 `Dockerfile`,用于安装 pip:
```
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:<version>
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:3.2.2
COPY requirements.txt /tmp
RUN apt-get update && \
apt-get install -y --no-install-recommends python-pip && \
@ -350,7 +350,7 @@ docker build -t apache/dolphinscheduler-worker:pip .
1. 创建一个新的 `Dockerfile`,用于安装 Python 3:
```
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:<version>
FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:3.2.2
RUN apt-get update && \
apt-get install -y --no-install-recommends python3 && \
rm -rf /var/lib/apt/lists/*

View File

@ -10,7 +10,7 @@ Standalone 仅适用于 DolphinScheduler 的快速体验.
## 前置准备工作
- JDK下载[JDK][jdk] (1.8+),安装并配置 `JAVA_HOME` 环境变量,并将其下的 `bin` 目录追加到 `PATH` 环境变量中。如果你的环境中已存在,可以跳过这步。
- 二进制包:在[下载页面](https://dolphinscheduler.apache.org/en-us/download/<version>)下载 DolphinScheduler 二进制包 <!-- markdown-link-check-disable-line -->
- 二进制包:在[下载页面](https://dolphinscheduler.apache.org/en-us/download/3.2.2)下载 DolphinScheduler 二进制包 <!-- markdown-link-check-disable-line -->
## 启动 DolphinScheduler Standalone Server

View File

@ -18,7 +18,7 @@
你可以最快速的体验到 DolphinScheduler 的大部分功能,了解主要和概念和内容。
```shell
$ DOLPHINSCHEDULER_VERSION=<version>
$ DOLPHINSCHEDULER_VERSION=3.2.2
$ docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:25333 -d apache/dolphinscheduler-standalone-server:"${DOLPHINSCHEDULER_VERSION}"
```
@ -33,11 +33,11 @@ $ docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:2
服务重启的时候保留元数据(如需要挂载到本地路径需要做指定)。他更健壮,能保证用户体验更加完整的 DolphinScheduler 服务。这种方式需要先安装
[docker-compose](https://docs.docker.com/compose/install/),链接适用于 MacLinuxWindows。
确保 docker-compose 顺利安装后,需要获取 `docker-compose.yaml` 文件,通过[下载页面](https://dolphinscheduler.apache.org/en-us/download/<version>)
确保 docker-compose 顺利安装后,需要获取 `docker-compose.yaml` 文件,通过[下载页面](https://dolphinscheduler.apache.org/en-us/download/3.2.2)
下载对应版本源码包可能是最快的方法,当下载完源码后就可以运行命令进行部署了。
```shell
$ DOLPHINSCHEDULER_VERSION=<version>
$ DOLPHINSCHEDULER_VERSION=3.2.2
$ tar -zxf apache-dolphinscheduler-"${DOLPHINSCHEDULER_VERSION}"-src.tar.gz
# Mac Linux 用户
$ cd apache-dolphinscheduler-"${DOLPHINSCHEDULER_VERSION}"-src/deploy/docker
@ -61,7 +61,7 @@ $ docker-compose --profile all up -d
ZooKeeper 且不想启动新的服务,可以使用这个方式分别启动 DolphinScheduler 容器。
```shell
$ DOLPHINSCHEDULER_VERSION=<version>
$ DOLPHINSCHEDULER_VERSION=3.2.2
# 初始化数据库,其确保数据库 <DATABASE> 已经存在
$ docker run -d --name dolphinscheduler-tools \
-e DATABASE="postgresql" \

View File

@ -4,6 +4,10 @@
#### 以下是Apache DolphinScheduler每个稳定版本的设置说明。
### Versions: 3.2.2
#### Links [3.2.2 Document](../3.2.2/user_doc/about/introduction.md)
### Versions: 3.2.1
#### Links [3.2.1 Document](../3.2.1/user_doc/about/introduction.md)

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-alert-aliyunVoice</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-alert-all</artifactId>

View File

@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-alert-api</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-alert-dingtalk</artifactId>
<packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-alert-email</artifactId>
<packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-alert-feishu</artifactId>
<packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-alert-http</artifactId>
<packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-alert-pagerduty</artifactId>
<packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-alert-prometheus</artifactId>
<packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-alert-script</artifactId>
<packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-alert-slack</artifactId>
<packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-alert-telegram</artifactId>
<packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-alert-webexteams</artifactId>
<packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert-plugins</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-alert-wechat</artifactId>
<packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-alert-plugins</artifactId>
<packaging>pom</packaging>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-alert-server</artifactId>
<packaging>jar</packaging>
@ -43,11 +43,6 @@
<artifactId>dolphinscheduler-dao</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-extract-alert</artifactId>
@ -120,6 +115,30 @@
</plugins>
</build>
</profile>
<profile>
<id>task-pure</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<dependencies>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-pure</artifactId>
</dependency>
</dependencies>
</profile>
<profile>
<id>task-all</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<dependencies>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-all</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>
</project>

View File

@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-alert</artifactId>
<packaging>pom</packaging>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-aop</artifactId>
<packaging>jar</packaging>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-api</artifactId>
<packaging>jar</packaging>
@ -66,33 +66,14 @@
<artifactId>dolphinscheduler-data-quality</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-all</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-task-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-ui</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-scheduler-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-storage-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
@ -233,6 +214,10 @@
<groupId>com.azure.resourcemanager</groupId>
<artifactId>azure-resourcemanager-datafactory</artifactId>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-identity</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-extract-alert</artifactId>
@ -295,5 +280,53 @@
</plugins>
</build>
</profile>
<profile>
<id>task-pure</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<dependencies>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-task-pure</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-pure</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-pure</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-storage-pure</artifactId>
</dependency>
</dependencies>
</profile>
<profile>
<id>task-all</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<dependencies>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-task-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-storage-all</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>
</project>

View File

@ -17,7 +17,6 @@
package org.apache.dolphinscheduler.api.service.impl;
import static org.apache.dolphinscheduler.api.utils.CheckUtils.checkFilePath;
import static org.apache.dolphinscheduler.common.constants.Constants.ALIAS;
import static org.apache.dolphinscheduler.common.constants.Constants.CONTENT;
import static org.apache.dolphinscheduler.common.constants.Constants.EMPTY_STRING;
@ -1291,10 +1290,6 @@ public class ResourcesServiceImpl extends BaseServiceImpl implements ResourcesSe
if (FOLDER_SEPARATOR.equalsIgnoreCase(fullName)) {
return;
}
// abnormal characters check
if (!checkFilePath(fullName)) {
throw new ServiceException(Status.ILLEGAL_RESOURCE_PATH);
}
// Avoid returning to the parent directory
if (fullName.contains("../")) {
throw new ServiceException(Status.ILLEGAL_RESOURCE_PATH, fullName);

View File

@ -158,14 +158,4 @@ public class CheckUtils {
return pattern.matcher(str).matches();
}
/**
* regex FilePath check,only use a to z, A to Z, 0 to 9, and _./-
*
* @param str input string
* @return true if regex pattern is right, otherwise return false
*/
public static boolean checkFilePath(String str) {
return regexChecks(str, Constants.REGEX_FILE_PATH);
}
}

View File

@ -92,24 +92,4 @@ public class CheckUtilsTest {
Assertions.assertTrue(CheckUtils.checkPhone("17362537263"));
}
/**
* check file path
*/
@Test
public void testCheckFilePath() {
// true
Assertions.assertTrue(CheckUtils.checkFilePath("/"));
Assertions.assertTrue(CheckUtils.checkFilePath("xx/"));
Assertions.assertTrue(CheckUtils.checkFilePath("/xx"));
Assertions.assertTrue(CheckUtils.checkFilePath("14567134578654"));
Assertions.assertTrue(CheckUtils.checkFilePath("/admin/root/"));
Assertions.assertTrue(CheckUtils.checkFilePath("/admin/root/1531531..13513/153135.."));
// false
Assertions.assertFalse(CheckUtils.checkFilePath(null));
Assertions.assertFalse(CheckUtils.checkFilePath("file://xxx/ss"));
Assertions.assertFalse(CheckUtils.checkFilePath("/xxx/ss;/dasd/123"));
Assertions.assertFalse(CheckUtils.checkFilePath("/xxx/ss && /dasd/123"));
Assertions.assertFalse(CheckUtils.checkFilePath("/xxx/ss || /dasd/123"));
}
}

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-bom</artifactId>
<packaging>pom</packaging>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-common</artifactId>

View File

@ -249,11 +249,6 @@ public final class Constants {
*/
public static final Pattern REGEX_USER_NAME = Pattern.compile("^[a-zA-Z0-9._-]{3,39}$");
/**
* file path regex
*/
public static final Pattern REGEX_FILE_PATH = Pattern.compile("^[a-zA-Z0-9_./-]+$");
/**
* read permission
*/

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-dao-api</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-dao-h2</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-dao-mysql</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-dao-plugin-all</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-dao-postgresql</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-dao-plugin</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-dao</artifactId>
<name>${project.artifactId}</name>

View File

@ -1070,7 +1070,7 @@ CREATE TABLE t_ds_version
-- Records of t_ds_version
-- ----------------------------
INSERT INTO t_ds_version
VALUES ('1', '3.3.0');
VALUES ('1', '3.2.2');
-- ----------------------------

View File

@ -1044,7 +1044,7 @@ CREATE TABLE `t_ds_version` (
-- ----------------------------
-- Records of t_ds_version
-- ----------------------------
INSERT IGNORE INTO `t_ds_version` VALUES ('1', '3.3.0');
INSERT IGNORE INTO `t_ds_version` VALUES ('1', '3.2.2');
-- ----------------------------

View File

@ -1063,7 +1063,7 @@ INSERT INTO t_ds_queue(queue_name, queue, create_time, update_time)
VALUES ('default', 'default', '2018-11-29 10:22:33', '2018-11-29 10:22:33');
-- Records of t_ds_queue,default queue name : default
INSERT INTO t_ds_version(version) VALUES ('3.3.0');
INSERT INTO t_ds_version(version) VALUES ('3.2.2');
--
-- Table structure for table t_ds_plugin_define

View File

@ -1 +1 @@
3.3.0
3.2.2

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-data-quality</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-all</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-api</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-athena</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-azure-sql</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-clickhouse</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-dameng</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-databend</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-db2</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-doris</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-hana</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-hive</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-k8s</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-kyuubi</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-mysql</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-oceanbase</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-oracle</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-postgresql</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-presto</artifactId>

View File

@ -0,0 +1,66 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-pure</artifactId>
<dependencies>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-hive</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-mysql</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-postgresql</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-spark</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-ssh</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-doris</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-k8s</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</project>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-redshift</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-sagemaker</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-snowflake</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-spark</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-sqlserver</artifactId>

View File

@ -35,6 +35,7 @@ import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
import com.alibaba.druid.sql.parser.SQLParserUtils;
import com.google.auto.service.AutoService;
@ -128,8 +129,10 @@ public class SQLServerDataSourceProcessor extends AbstractDataSourceProcessor {
@Override
public List<String> splitAndRemoveComment(String sql) {
String cleanSQL = SQLParserUtils.removeComment(sql, com.alibaba.druid.DbType.sqlserver);
return SQLParserUtils.split(cleanSQL, com.alibaba.druid.DbType.sqlserver);
return SQLParserUtils.splitAndRemoveComment(sql, com.alibaba.druid.DbType.sqlserver)
.stream()
.map(subSql -> subSql.concat(";"))
.collect(Collectors.toList());
}
private String transformOther(Map<String, String> otherMap) {

View File

@ -17,12 +17,15 @@
package org.apache.dolphinscheduler.plugin.datasource.sqlserver.param;
import static com.google.common.truth.Truth.assertThat;
import org.apache.dolphinscheduler.common.constants.DataSourceConstants;
import org.apache.dolphinscheduler.common.utils.JSONUtils;
import org.apache.dolphinscheduler.plugin.datasource.api.utils.PasswordUtils;
import org.apache.dolphinscheduler.spi.enums.DbType;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.junit.jupiter.api.Assertions;
@ -95,4 +98,32 @@ public class SQLServerDataSourceProcessorTest {
Assertions.assertEquals(DataSourceConstants.SQLSERVER_VALIDATION_QUERY,
sqlServerDatasourceProcessor.getValidationQuery());
}
@Test
void splitAndRemoveComment_singleSelect() {
String sql = "select * from table;";
List<String> subSqls = sqlServerDatasourceProcessor.splitAndRemoveComment(sql);
assertThat(subSqls).hasSize(1);
assertThat(subSqls.get(0)).isEqualTo("select * from table;");
}
@Test
void splitAndRemoveComment_singleMerge() {
String sql = "MERGE\n" +
" [ TOP ( expression ) [ PERCENT ] ]\n" +
" [ INTO ] <target_table> [ WITH ( <merge_hint> ) ] [ [ AS ] table_alias ]\n" +
" USING <table_source> [ [ AS ] table_alias ]\n" +
" ON <merge_search_condition>\n" +
" [ WHEN MATCHED [ AND <clause_search_condition> ]\n" +
" THEN <merge_matched> ] [ ...n ]\n" +
" [ WHEN NOT MATCHED [ BY TARGET ] [ AND <clause_search_condition> ]\n" +
" THEN <merge_not_matched> ]\n" +
" [ WHEN NOT MATCHED BY SOURCE [ AND <clause_search_condition> ]\n" +
" THEN <merge_matched> ] [ ...n ]\n" +
" [ <output_clause> ]\n" +
" [ OPTION ( <query_hint> [ ,...n ] ) ];";
List<String> subSqls = sqlServerDatasourceProcessor.splitAndRemoveComment(sql);
assertThat(subSqls).hasSize(1);
assertThat(subSqls.get(0)).isEqualTo(sql);
}
}

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-ssh</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-starrocks</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-trino</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-vertica</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-zeppelin</artifactId>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<packaging>pom</packaging>
@ -56,6 +56,7 @@
<module>dolphinscheduler-datasource-sagemaker</module>
<module>dolphinscheduler-datasource-k8s</module>
<module>dolphinscheduler-datasource-hana</module>
<module>dolphinscheduler-datasource-pure</module>
</modules>
<dependencyManagement>

View File

@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-dist</artifactId>

View File

@ -313,6 +313,8 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
jetty-xml 9.4.48.v20220622: https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-xml/9.4.48.v20220622, Apache 2.0 and EPL 1.0
jna 5.10.0: https://mvnrepository.com/artifact/net.java.dev.jna/jna/5.10.0, Apache 2.0 and LGPL 2.1
jna-platform 5.10.0: https://mvnrepository.com/artifact/net.java.dev.jna/jna-platform/5.10.0, Apache 2.0 and LGPL 2.1
jna 5.5.0: https://mvnrepository.com/artifact/net.java.dev.jna/jna/5.5.0, Apache 2.0 and LGPL 2.1
jna-platform 5.6.0: https://mvnrepository.com/artifact/net.java.dev.jna/jna-platform/5.6.0, Apache 2.0 and LGPL 2.1
joda-time 2.10.13: https://github.com/JodaOrg/joda-time, Apache 2.0
jose4j 0.7.8: https://mvnrepository.com/artifact/org.bitbucket.b_c/jose4j, Apache 2.0
jpam 1.1: https://mvnrepository.com/artifact/net.sf.jpam/jpam/1.1, Apache 2.0
@ -509,6 +511,7 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
third-party-jackson-core 2.17.282: https://mvnrepository.com/artifact/software.amazon.awssdk/third-party-jackson-core/2.17.282, Apache 2.0
utils 2.17.282: https://mvnrepository.com/artifact/software.amazon.awssdk/utils/2.17.282, Apache 2.0
zt-zip 1.15: https://github.com/zeroturnaround/zt-zip/blob/master/LICENSE, Apache 2.0
content-type 2.2: https://mvnrepository.com/artifact/com.nimbusds/content-type/2.2, Apache 2.0
jackson-dataformat-xml 2.13.3: https://mvnrepository.com/artifact/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.13.3, Apache 2.0
lang-tag 1.6: https://mvnrepository.com/artifact/com.nimbusds/lang-tag/1.6, Apache 2.0

View File

@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-extract</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-extract-alert</artifactId>

View File

@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-extract</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-extract-base</artifactId>

View File

@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-extract</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<groupId>org.apache.dolphinscheduler</groupId>

View File

@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-extract</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-extract-master</artifactId>

View File

@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-extract</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-extract-worker</artifactId>

View File

@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-extract</artifactId>

View File

@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-master</artifactId>
@ -55,25 +55,11 @@
<artifactId>dolphinscheduler-service</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-scheduler-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-task-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-storage-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-extract-master</artifactId>
@ -339,5 +325,51 @@
</plugins>
</build>
</profile>
<profile>
<id>task-pure</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<dependencies>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-task-pure</artifactId>
<exclusions>
<exclusion>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-all</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-pure</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-storage-pure</artifactId>
</dependency>
</dependencies>
</profile>
<profile>
<id>task-all</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<dependencies>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-task-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-all</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-storage-all</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>
</project>

View File

@ -24,7 +24,7 @@ import java.util.function.Supplier;
import lombok.experimental.UtilityClass;
import com.facebook.presto.jdbc.internal.guava.collect.ImmutableSet;
import com.google.common.collect.ImmutableSet;
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.Gauge;

View File

@ -1226,12 +1226,12 @@ public class WorkflowExecuteRunnable implements IWorkflowExecuteRunnable {
|| state == TaskExecutionStatus.DISPATCH
|| state == TaskExecutionStatus.SUBMITTED_SUCCESS
|| state == TaskExecutionStatus.DELAY_EXECUTION) {
// try to take over task instance
if (state == TaskExecutionStatus.SUBMITTED_SUCCESS || state == TaskExecutionStatus.DELAY_EXECUTION
|| state == TaskExecutionStatus.DISPATCH) {
if (state == TaskExecutionStatus.SUBMITTED_SUCCESS
|| state == TaskExecutionStatus.DELAY_EXECUTION) {
// The taskInstance is not in running, directly takeover it
} else if (tryToTakeOverTaskInstance(existTaskInstance)) {
log.info("Success take over task {}", existTaskInstance.getName());
// If the taskInstance has already dispatched to worker then will try to take-over it
log.info("Success take over task {} -> status: {}", existTaskInstance.getName(), state);
continue;
} else {
// set the task instance state to fault tolerance

View File

@ -61,26 +61,30 @@ public class SwitchLogicTask extends BaseSyncLogicTask<SwitchParameters> {
this.taskInstance = workflowExecuteRunnable.getTaskInstance(taskExecutionContext.getTaskInstanceId())
.orElseThrow(() -> new LogicTaskInitializeException(
"Cannot find the task instance in workflow execute runnable"));
// Since the default branch is not in the dependTaskList, we need to add it to the end
// otherwise the default branch will never be skipped in DAGHelper
addDefaultBranchToEnd();
}
@Override
public void handle() throws MasterTaskExecuteException {
// Calculate the condition result and get the next node
if (CollectionUtils.isEmpty(taskParameters.getDependTaskList())) {
// If the branch is empty then will go into the default branch
// This case shouldn't happen, we can directly throw exception and forbid the user to set branch
moveToDefaultBranch();
} else {
calculateSwitchBranch();
}
taskInstance.setSwitchDependency(taskParameters);
log.info("Switch task execute finished");
taskExecutionContext.setCurrentExecutionStatus(TaskExecutionStatus.SUCCESS);
log.info("Switch task execute finished: {}", taskExecutionContext.getCurrentExecutionStatus().name());
}
private void moveToDefaultBranch() {
checkIfBranchExist(taskParameters.getNextNode());
List<SwitchResultVo> switchResultVos = taskParameters.getDependTaskList();
switchResultVos.add(new SwitchResultVo(null, taskParameters.getNextNode()));
SwitchResultVo defaultSwitchResultVo = getDefaultSwitchResultVo();
checkIfBranchExist(defaultSwitchResultVo.getNextNode());
taskParameters.setResultConditionLocation(switchResultVos.size() - 1);
log.info("The condition is not satisfied, move to the default branch: {}",
@ -90,9 +94,6 @@ public class SwitchLogicTask extends BaseSyncLogicTask<SwitchParameters> {
private void calculateSwitchBranch() {
List<SwitchResultVo> switchResultVos = taskParameters.getDependTaskList();
if (CollectionUtils.isEmpty(switchResultVos)) {
moveToDefaultBranch();
}
Map<String, Property> globalParams = taskExecutionContext.getPrepareParamsMap();
Map<String, Property> varParams = JSONUtils
.toList(taskInstance.getVarPool(), Property.class)
@ -100,7 +101,8 @@ public class SwitchLogicTask extends BaseSyncLogicTask<SwitchParameters> {
.collect(Collectors.toMap(Property::getProp, Property -> Property));
int finalConditionLocation = -1;
for (int i = 0; i < switchResultVos.size(); i++) {
// The last one is the default branch, no need to calculate
for (int i = 0; i < switchResultVos.size() - 1; i++) {
SwitchResultVo switchResultVo = switchResultVos.get(i);
log.info("Begin to execute {} condition: {} ", i, switchResultVo.getCondition());
String content = SwitchTaskUtils.generateContentWithTaskParams(switchResultVo.getCondition(), globalParams,
@ -111,14 +113,18 @@ public class SwitchLogicTask extends BaseSyncLogicTask<SwitchParameters> {
result = SwitchTaskUtils.evaluate(content);
log.info("Execute condition sentence: {} successfully: {}", content, result);
if (result) {
// If matched, break the loop
finalConditionLocation = i;
break;
}
} catch (Exception e) {
log.info("Execute condition sentence: {} failed", content, e);
}
}
// If the finalConditionLocation is -1, then the default branch will be executed
if (finalConditionLocation >= 0) {
checkIfBranchExist(switchResultVos.get(finalConditionLocation).getNextNode());
List<Long> nextNodes = switchResultVos.get(finalConditionLocation).getNextNode();
checkIfBranchExist(nextNodes);
log.info("The condition is satisfied, move to the branch: {}",
switchResultVos.get(finalConditionLocation).getNextNode().stream()
.map(node -> workflowExecuteRunnable.getWorkflowExecuteContext().getWorkflowGraph().getDag()
@ -126,7 +132,6 @@ public class SwitchLogicTask extends BaseSyncLogicTask<SwitchParameters> {
.collect(Collectors.toList()));
taskParameters.setResultConditionLocation(finalConditionLocation);
} else {
log.info("All conditions are not satisfied, move to the default branch");
moveToDefaultBranch();
}
}
@ -147,4 +152,16 @@ public class SwitchLogicTask extends BaseSyncLogicTask<SwitchParameters> {
}
}
private void addDefaultBranchToEnd() {
SwitchResultVo switchResultVo = new SwitchResultVo(null, taskParameters.getNextNode());
List<SwitchResultVo> dependTaskList = taskParameters.getDependTaskList();
if (!dependTaskList.contains(switchResultVo)) {
dependTaskList.add(switchResultVo);
}
}
private SwitchResultVo getDefaultSwitchResultVo() {
return taskParameters.getDependTaskList().get(taskParameters.getDependTaskList().size() - 1);
}
}

View File

@ -63,7 +63,6 @@ public class SwitchTaskUtils {
if (MapUtils.isNotEmpty(varParams)) {
params.putAll(varParams);
}
String originContent = content;
Pattern pattern = Pattern.compile(rgex);
Matcher m = pattern.matcher(content);
while (m.find()) {
@ -82,10 +81,6 @@ public class SwitchTaskUtils {
content = content.replace("${" + paramName + "}", value);
}
// if not replace any params, throw exception to avoid illegal condition
if (originContent.equals(content)) {
throw new IllegalArgumentException("condition is not valid, please check it. condition: " + condition);
}
return content;
}

View File

@ -50,7 +50,7 @@ public class SwitchTaskUtilsTest {
Map<String, Property> globalParams = new HashMap<>();
Map<String, Property> varParams = new HashMap<>();
globalParams.put("test", new Property("test", Direct.IN, DataType.INTEGER, "1"));
Assertions.assertThrowsExactly(IllegalArgumentException.class, () -> {
Assertions.assertDoesNotThrow(() -> {
SwitchTaskUtils.generateContentWithTaskParams(content, globalParams, varParams);
});
@ -70,15 +70,5 @@ public class SwitchTaskUtilsTest {
SwitchTaskUtils.evaluate(script);
});
String contentWithSpecify1 = "cmd.abc";
Assertions.assertThrowsExactly(IllegalArgumentException.class, () -> {
SwitchTaskUtils.generateContentWithTaskParams(contentWithSpecify1, globalParams, varParams);
});
String contentWithSpecify2 = "cmd()";
Assertions.assertThrowsExactly(IllegalArgumentException.class, () -> {
SwitchTaskUtils.generateContentWithTaskParams(contentWithSpecify2, globalParams, varParams);
});
}
}

View File

@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-meter</artifactId>
@ -45,7 +45,7 @@
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-common</artifactId>
<version>dev-SNAPSHOT</version>
<version>3.2.3-SNAPSHOT</version>
</dependency>
<dependency>

Some files were not shown because too many files have changed in this diff Show More