Commit Graph

202 Commits

Author SHA1 Message Date
amao b71d4983dd
[Feature-6586][Server]add some ds process definition demo when init (#11759)
* [Feature-6586][Server]add some ds process definition demo when init
1.add some ds process definition demo when init, to display what task type can run and make user easy to
use ds.
2.need configure the JVM parameters (-Ddemo=true) to turn on the StandaloneServer service
3.need modify the tenant information in it
2022-10-31 22:35:11 +08:00
rickchengx 3c31ddfd9f
[Improvement-12536][k8s] Support the command for the container in k8s task plugin (#12538) 2022-10-31 19:06:41 +08:00
Aaron Wang 08335b1032
[Improvement][Task] Improved way to collect yarn job's appIds (#12197)
* Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code.
* Add user property `appId.collect` for user to decide how to collect applicationId.
* Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh`
* Update docs to declare how to use aop way.
* Update `LogUtils` to support fetch applicationId in different ways based on the user property.

Co-authored-by: gabrywu <gabrywu@apache.com>
2022-10-31 16:52:53 +08:00
HanayoZz 489e7fe4e2
[Feature-10495][Resource Center] Resource Center Refactor (#12076)
* resource center refactor - S3 services connection

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-10-26 13:53:44 +08:00
caishunfeng dedff70f90
[Improvement] Merge spi.utils into common.utils (#12502)
* improve JSONUtil

* merge spi.utils into common.utils

* split constants && remove some copy utils method

* remove StringUtils
2022-10-23 21:23:54 +08:00
fuchanghai f0a34cddec
[improvement-#12416] Use CollectionUtils.isEmpty judge the collection isEmpty to avoid NPE (#12449) 2022-10-20 14:26:56 +08:00
Eric Gao d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports (#12412)
* Fix remaining style errors and wildcard imports

* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link

* Fix spotless matching scope of markdown files
2022-10-19 11:08:04 +08:00
insist777 1e11034049
[Fix][Task] Fix dependent task can not predicate the status of the corresponding task correctly (#12253)
* [Fix][Task] dependent task can not predicate the status of the corresponding task correctly
2022-10-19 10:47:08 +08:00
simsicon ba538067f2
[fix] Change the default unix shell executor from sh to bash (#12180)
Co-authored-by: Kerwin <37063904+zhuangchong@users.noreply.github.com>
2022-10-18 12:57:37 +08:00
Yann Ann 3bef85f546
fix datax NPE issue (#12388) 2022-10-18 09:03:16 +08:00
Kerwin 55004bebe0
[Improvement-12391][API] Workflow definitions that contain logical task nodes support the copy function (#12392)
* Workflow definitions that contain logical task nodes support the copy function.
2022-10-17 10:28:58 +08:00
fuchanghai b7bd8d7803
[Improvement] add presto in datax (#12371) 2022-10-16 17:19:29 +08:00
Yann Ann 7aa8a77fb2
[Migrate][Test] Migrate other UT cases from jUnit 4 to 5, change @Ignore to @Disable 2022-10-16 12:49:11 +08:00
rickchengx 7b44612f28
[Improvement-12372][k8s] Update the deprecated k8s api (#12373) 2022-10-14 16:18:35 +08:00
Eric Gao 2f37da0dbc
[Improvement][Test] Block the usage of powermock and move mockito dependencies from sub-modules to root pom (#12311)
* move mockito dependencies from sub-modules to root pom

* Add check in CI to block the usage of powermock
2022-10-14 12:06:51 +08:00
kezhenxu94 db1d319159
Reorganize some classes in common module, remove duplicate classes (#12321) 2022-10-13 10:23:27 +08:00
rickchengx 05099bd8f7
[Improvement-12296][Test] Close static mock (#12300) 2022-10-11 16:13:05 +08:00
Eric Gao 875682d267
[Improvement][Test] Migrate all UT cases from jUnit4 to jUnit5 in task-plugin module as an example (#12299)
* Migrate all UT cases from jUnit4 to jUnit5 in task-plugin module as an example
2022-10-11 11:54:38 +08:00
Eric Gao a865be09b6
Remove powermock from DatasyncTaskTest (#12270) 2022-10-09 12:38:03 +08:00
Tq c7435a51ac
[Feature][task-plugins] aws datasync task plugins (#11865)
* submit datasync task

* add aws software package license
2022-10-09 10:47:01 +08:00
FlechazoW 0687697869
[Improvement][Task Plugin] Modify the comment of 'deployMode'. (#12163) 2022-10-07 21:21:13 +08:00
Yann Ann c87f6d89a4
[fix#12195] Change date to timestamp to support cross diff time zones (#12239)
* fix DS support cross time zone, use timestamp replace date
2022-10-01 12:50:42 +08:00
JieguangZhou 308f632fcc
[Feature][TASK-PLUGIN] AWS Database Migration Service (DMS) (#11868)
* add aws dms
2022-09-30 10:16:16 +08:00
Stalary aa5526ef5d
[Improvement][Test] Remove Powermock in dolphinscheduler-task-plugin (#12153)
* Remove the usage of powermock in task-plugin module
2022-09-28 14:13:23 +08:00
冯剑 779a0b9c75
[Improvement-#11608][task-plugin] New time variables are added to facilitate business development (#11667)
Add new time expressions for users' convenience

Co-authored-by: 冯剑 Jian <jian.feng@jiduauto.com>
Co-authored-by: David <dailidong66@gmail.com>
2022-09-26 16:01:32 +08:00
zhuxt2015 e6832220c3
[Improvement][SQL Task]use default sql segment separator (#10869) 2022-09-26 11:08:57 +08:00
JieguangZhou 6eb1eb722a
fix python home conflict (#12112) 2022-09-23 09:40:18 +08:00
冯剑 9b3b4e22b4
[Bug-#12057][task-plugin] fix when the sql query result is empty, the email fails to send the attachment, and an exception will be reported (#12059)
* [Bug-#12057][task-plugin] fix when the sql query result is empty, the email fails to send the attachment, and an exception will be reported

Co-authored-by: 冯剑 Jian <jian.feng@jiduauto.com>
2022-09-22 09:39:02 +08:00
rickchengx 08a4c7981f
[improve] Remove the spark version of spark task (#11860) 2022-09-21 20:27:28 +08:00
JieguangZhou cf522e2fc0
[improve] Optimize MLFlow task plugin for easy of use (#12071)
optimize code
2022-09-21 10:45:05 +08:00
Eric Gao 66958b91d7
[Feature][Resource Center] Add support for Alibaba Cloud OSS as storage of resource center (#11708)
* Add support for Alibaba Cloud OSS as storage of resource center

* Add dependency licenses

* Refactor code, add unit tests and related docs
2022-09-20 22:44:06 +08:00
Eric Gao e1b55db72a
[Improvement][Test] Give an example on replacing Powermock with Mockito (#11588)
* Give an example on replacing powermock with mockito

* Remove redundant comments

* Refactoring related UTs for better readability.
2022-09-20 09:48:02 +08:00
rickchengx b3bbf69b2e
[Improvement-11291][sql] Improvement the log output while executing multiple sql statements (#11292) 2022-09-20 09:30:27 +08:00
JieguangZhou bbc228e93f
[Feature][PyDolphinScheduler] Support DVC task in pyds #11922 (#11941)
* add dvc task in pyds

* add BaseDVC class
2022-09-17 16:55:51 +08:00
insist777 5b384f3fab
[Feature-11473][Task]Support test task (#11670)
* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* Update TaskExecuteThread.java

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update DataSource.java

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update messages_zh_CN.properties

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update messages.properties

* Update messages_en_US.properties

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks
2022-09-17 13:40:56 +08:00
itzhang89 511149f644
[fix-10938]: use dot to replace source and make the default env work in shell (#11625)
* [fix-10938]: use the dot to replace source and make the default env work in the sh
Co-authored-by: Kerwin <37063904+zhuangchong@users.noreply.github.com>
2022-09-15 15:05:09 +08:00
Wenjun Ruan d3a77c68e6
Make LogServiceClient Singleton (#11777) 2022-09-14 16:18:08 +08:00
caishunfeng 0deafba12c
fix procedure task param pass (#11919) 2022-09-14 13:44:01 +08:00
limaiwang 25b78a8003
[Bug] [spark-sql] In spark-sql, select both SPARK1 and SPARK2 versions and execute ${SPARK_HOME2}/bin/spark-sql (#11721)
select different versions of spark-sql to execute different versions of spark-sql
2022-09-12 10:35:27 +08:00
kezhenxu94 147e0ae8d7
Remove duplicate Spring version management (#11899) 2022-09-11 20:50:07 +08:00
W.XING 1164cb776d
Add chunjun in dolphinscheduler-task-all/pom.xml (#11375) 2022-09-11 19:14:05 +08:00
juzimao 5d9331f4ee
[Feature-10683][Task Plugin] Add Java Task Plugin. (#10689) 2022-09-09 10:37:24 +08:00
fuchanghai e101a5cb2c
[fix-#11815] fix ck column names contain special characters (#11818)
Co-authored-by: fuchanghai <changhai.fu@marketingforce.com>
2022-09-08 15:09:17 +08:00
caishunfeng 2e61c76c22
[Improvement] Add remote task model (#11767) 2022-09-08 13:41:36 +08:00
kezhenxu94 efedc9a53c
Bump up dependencies to fix cves (#11765) 2022-09-08 09:05:05 +08:00
Kengo Seki 444c7e6eb7
[Improvement] Replace commons-lang 2 function invocations with commons-lang3 (#11810)
* [Improvement] Replace commons-lang 2 function invocations with commons-lang3
2022-09-07 13:48:09 +08:00
fuchanghai f8b9aad239
[feature-#11689] change replaceAll to replace in method of tryExecuteSqlResolveColumnNames (#11696) 2022-09-05 09:54:18 +08:00
caishunfeng f95cdd350a
reduce task dependencies (#11731) 2022-09-01 17:04:07 +08:00
Wenjun Ruan 03e1e6fe45
[Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task (#11701)
* Fix kill job may cause worker oom
2022-08-31 15:25:01 +08:00
Eric Gao 4a37345436
[Feature][Task Plugin] Support hive cli task plugin (#11651)
* Add hive cli task plugin

* Add docs for hive-cli task plugin

* Fix hive-cli task related front-end nits and checkstyle

* Fix exception handling logic
2022-08-30 10:45:51 +08:00