f64b254809
[maven-release-plugin] prepare release 3.2.0
2023-10-11 16:59:08 +08:00
6d7dbe6733
Reduce the size of tarball to continue ASF release ( #15004 )
...
* Reduce the size of tarball to continue ASF release
for more detail you can see https://lists.apache.org/thread/rmp7fghlj0n7h9y2v3p8gkw9f9qbo6qt
* rename the new module
* spotless
* license
* fix api test
* fix api test
2023-10-11 16:19:25 +08:00
a1a041082d
[Improvement][K8S] K8S task support passing context parameters to downstream task ( #14934 )
...
* [Improvement][K8S] K8S task support passing context parameters to downstream task
Signed-off-by: Gallardot <gallardot@apache.org>
* [Improvement][K8S] K8S task support passing context parameters to downstream task
Signed-off-by: Gallardot <gallardot@apache.org>
* chore: update doc img
Signed-off-by: Gallardot <gallardot@apache.org>
* chore: update doc
Signed-off-by: Gallardot <gallardot@apache.org>
* chore: add UT
Signed-off-by: Gallardot <gallardot@apache.org>
* merge suggestion
Co-authored-by: 旺阳 <wang@lqwang.net>
* merge suggestion
Co-authored-by: 旺阳 <wang@lqwang.net>
* merge suggestion
Co-authored-by: 旺阳 <wang@lqwang.net>
---------
Signed-off-by: Gallardot <gallardot@apache.org>
Co-authored-by: David Zollo <dailidong66@gmail.com>
Co-authored-by: 旺阳 <qingwli@cisco.com>
Co-authored-by: 旺阳 <wang@lqwang.net>
Co-authored-by: xiangzihao <460888207@qq.com>
(cherry picked from commit 049c1511fa
2023-10-11 15:58:18 +08:00
c2dd874c82
[feature-#14449] add secret in k8s Task ( #14930 )
...
* [feature-#14449] add secret
* [feature-#14449] upgrade UT and rename parameter ```secret```
* [feature-#14449] fix UT
* [feature-#14449] fix UT
* [feature-#14449] improvement
* [feature-#14449] format
---------
Co-authored-by: fuchanghai <‘2875334588@qq.com’>
(cherry picked from commit f90eadd40f
2023-10-11 15:56:27 +08:00
cb80df4570
Fix the error of using shell task to obtain Home variable in Ubuntu system. ( #14964 )
...
(cherry picked from commit 6de1e2c1b3
2023-10-11 15:55:43 +08:00
06a555e494
fix the #14729 problem ( #14902 )
...
Co-authored-by: 刘阳 <liuy2590@chinaunicom.cn>
Co-authored-by: xiangzihao <460888207@qq.com>
(cherry picked from commit e748c2eb9a
2023-10-11 15:52:42 +08:00
87d6092ef9
[Fix-14546] seatunnel unable submit spark job to spark ( #14617 )
...
(cherry picked from commit 9083f9d0e0
2023-10-11 15:50:45 +08:00
12baee69d8
[Bug][Worker] Global and local parameters are not replaced in SQL components ( #14926 )
...
Signed-off-by: Gallardot <gallardot@apache.org>
(cherry picked from commit f6a6794289
2023-10-11 15:49:23 +08:00
e71ae0da55
[Bug][Worker] Fix sql/shell task doesn't replace parameter ( #14876 )
...
Fix parameter assign error, fix #14869 which including
* shell task param not work, botch local and global
* others task chain assign not work
(cherry picked from commit 3fa69d2db8
2023-10-11 15:44:01 +08:00
f3affa0d75
fix: null app-info-path cause NPE ( #14752 )
...
(cherry picked from commit a491e3a849
2023-08-30 17:09:57 +08:00
5d9d6061fd
[Improvement] add dispatch fail times when compare ( #14761 )
...
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
(cherry picked from commit 59998f4c75
2023-08-30 17:09:52 +08:00
22eac768da
[Fix-14646][Worker] Fix match wrong pid on linux server ( #14652 )
...
* [DS-14646][Worker] Fix match wrong pid on linux server
This closes #14646
---------
Co-authored-by: yangxin3 <yangxin3@newhope.cn>
(cherry picked from commit b3af374fe6
2023-08-30 17:09:25 +08:00
3e0569b4fa
[Feature][Resource Center] Add support for Huawei Cloud OBS ( #14643 )
...
* [Feature][Resource Center] Add support for Huawei Cloud OBS as storage of resource center
* add license and add doc
* add 3-party dependency license
* Update LICENSE
* fix
* Update pom.xml
* fix
* fix
---------
Co-authored-by: sunkang <sunkang@hjfruit.com>
Co-authored-by: xiangzihao <460888207@qq.com>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
(cherry picked from commit 25480ae9e7
2023-08-30 17:08:39 +08:00
6617a00b9b
[Fix-14721] [K8S Task] Handle job delete event when user manually delete the job in k8s cluster ( #14722 )
...
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
(cherry picked from commit cb55476235
2023-08-30 17:08:24 +08:00
3499c25f84
fix: do not override k8sConfigYaml in ShShellInterceptorBuilder ( #14682 )
...
Co-authored-by: xiangzihao <460888207@qq.com>
(cherry picked from commit 5ec9085113
2023-08-30 17:04:01 +08:00
92146ddc2d
Support get property from env/jvm/common.properties ( #14645 )
...
(cherry picked from commit 9ce8871158
2023-08-30 17:01:06 +08:00
f65dab2e21
[Fix-10795][plugin] sqoop task query columns bug ( #10776 )
...
* [Improvement][Api] add @Override on the Override method
* [FIX]sqoop task query columns bug
* merge two if
* update checkstyle problem
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-sqoop/src/main/java/org/apache/dolphinscheduler/plugin/task/sqoop/SqoopColumnType.java
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-sqoop/src/main/java/org/apache/dolphinscheduler/plugin/task/sqoop/generator/sources/MySQLSourceGenerator.java
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
* code format
* code format
---------
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: xiangzihao <460888207@qq.com>
(cherry picked from commit 512f4c7021
2023-08-30 16:59:43 +08:00
5f48b01591
Use AdHoc datasource client in sqlTask ( #14631 )
...
* Use AdHoc datasource client in sqlTask
* Add method in DataSourceChannel to create PooledDataSourceClient and AdHocDataSourceClient
(cherry picked from commit 4aab0b234f
2023-08-30 16:59:38 +08:00
e6e012091d
fix process utils comment typo ( #14613 )
...
(cherry picked from commit d6a4e99e19
2023-08-30 13:57:29 +08:00
f24e700419
[Feature-13763][K8S Task] collect real-time log ( #14379 )
...
* [Improvement-13763][K8S Task] collect real-time log
* fix codesmell
* get pod watcher until pod is ready
* fix codesmell
* specify container name & loop waiting pod creation
* sleep when pod is not initialized
---------
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
(cherry picked from commit 04800a48d1
2023-08-30 13:54:50 +08:00
10157e1c69
Fix wrong judgment condition when building parameters of seatunnel task node. ( #14594 )
2023-08-02 17:25:50 +08:00
e677a3816a
impv: Correct datax execute and python execute script name ( #14523 )
...
* impv: Correct datax execute and python execute script name
we use PYTHON_LAUNCHER for python script execute and
DATAX_LAUNCHER for datax script name
* Add pr number
* fix ut
* style
---------
Co-authored-by: xiangzihao <460888207@qq.com>
(cherry picked from commit de2d215aee
2023-07-20 11:30:46 +08:00
436c8e251e
Support execute shell in different interceptor ( #14582 )
...
(cherry picked from commit c30cca9d9a
2023-07-20 11:30:43 +08:00
d2ac5999c6
[Improvement-14136][task] Support submitting spark-sql task with the sql file in resource center ( #14527 )
...
Co-authored-by: xiangzihao <460888207@qq.com>
(cherry picked from commit 9778652f17
2023-07-20 11:30:43 +08:00
55b94db3c5
[Typo] standardized license header. ( #14548 )
...
Co-authored-by: wentz <wentz@getui.com>
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
(cherry picked from commit 027d66d200
2023-07-20 11:30:38 +08:00
adb106f47c
Add WorkflowExecuteContext ( #14544 )
...
(cherry picked from commit e6d94632b9
2023-07-20 11:30:38 +08:00
91e9cc25f4
[Improvement][Worker] No need to kill remote app when appid is empty ( #14534 )
...
(cherry picked from commit 09afa75a3d
2023-07-20 11:30:21 +08:00
2818294ac7
[Improvement] Remove the unused newDaemonSingleThreadExecutor ( #14520 )
...
(cherry picked from commit c1cffba349
2023-07-20 11:30:21 +08:00
c0b2b1c0e3
[Bug] [Seatunnel-Task] Run Mode defaults to run, and you cannot select a null value ( #14267 )
...
Co-authored-by: zhang <zhangyfcoder@163.com>
(cherry picked from commit 4111acb58e
2023-07-20 11:30:20 +08:00
d414f2631b
[Feature][Data Quality] Data Quality Support Choose Database ( #14406 )
...
* add dataquality database api
* change ui
* api change
* update
* fix spotless
* fix h2
* fix pg
* fix-dead-line
* update
* fix-spotless
* update pg sql
* add ut
* fix ut
(cherry picked from commit 175d976cd2
2023-07-20 11:30:20 +08:00
3add3f1dbb
[improvement] DataQuality module improve ( #14463 )
...
(cherry picked from commit 75d29f6101
2023-07-20 11:29:53 +08:00
a99596a2df
[Feature-14251][Task] Support yarn queue definition in yarn task ( #14310 )
...
(cherry picked from commit 0b69236b04
2023-07-20 11:29:21 +08:00
5ce72b3455
[Fix-14447][K8S Task] Fix IndexOutofBoundsException for pod name without dash ( #14448 )
...
(cherry picked from commit 742a2dc780
2023-07-20 11:29:17 +08:00
6a38b0e6ed
[Improvement-14464][Task] Remove the useless taskResultString ( #14465 )
...
(cherry picked from commit cff53feaeb
2023-07-20 11:28:40 +08:00
01a2f734c2
[Improvement-14457][UT] Improve the expected and actual value of Assertions ( #14459 )
...
(cherry picked from commit 68a3a75211
2023-07-20 11:26:43 +08:00
09e0a11f21
[Feature-14421][K8S Task] Configurable image pull policy ( #14426 )
...
(cherry picked from commit 65b6a4b097
2023-07-20 11:26:28 +08:00
3d6e145202
[Improvement-14371][Hivecli] Add support for hivevar setting in hivecli task ( #14423 )
...
(cherry picked from commit 13dfa859b5
2023-07-20 11:26:09 +08:00
75dedf46dd
fix: default datax plugin ( #14204 )
...
(cherry picked from commit 9c92b4b240
2023-07-20 11:23:19 +08:00
299c2e2b75
[Improvement-14331][common] Remove the duplicate ParameterUtils ( #14332 )
...
(cherry picked from commit b49b91c45d
2023-07-20 10:00:25 +08:00
f46d07de45
Fix switch task use boolean type error ( #14326 )
...
(cherry picked from commit a55612e4f7
2023-07-20 10:00:08 +08:00
2cb8186915
[Improvement-14329][common] Remove the duplicate BusinessTimeUtils ( #14330 )
...
(cherry picked from commit 1cd5070b5d
2023-07-20 09:59:55 +08:00
4be93325b0
[feature][Resource] Add support for Azure Blob Storage in the resource center ( #14082 )
...
(cherry picked from commit 4fa0ab4d7e
2023-07-20 09:55:26 +08:00
b2ee8d5e6f
[bug][plugin]Fix: Correct the way to determine the yarn queue in Flink CommandLine and SQL mode ( #14237 )
...
* Fix: Correct the way to determine the yarn queue in Flink CommandLine
* fix the yarn queue in sql mode && refine the code
* refine code
* remove unnecessary comment
* fix yarn queue properties
* remove redundant variable
(cherry picked from commit de2cc0e235
2023-07-20 09:55:22 +08:00
1f8a77d79d
fix: trim node selector requirement value ( #14307 )
...
(cherry picked from commit b7fb393eed
2023-07-20 09:54:58 +08:00
b659f1af7a
[Improvement-14295][common] Remove the duplicate FileUtils ( #14301 )
...
(cherry picked from commit 66e09dc66e
2023-07-20 09:54:46 +08:00
1168fabf13
[Feature][Task Plugin] Add Dynamic task to generate dynamic worklfows with list parameters ( #14127 )
...
* add dynamic task plugin
* fix ui
* update code
* add DOC
* ADD UT
* fix doc
* fix sanity check
* add upgrade ddl
* fix sql
(cherry picked from commit e7cdc7c136
2023-07-20 09:54:42 +08:00
95e2279bfb
[Improvement-14292][Common] Remove the duplicate OSUtils ( #14293 )
...
(cherry picked from commit 2c057d6a8f
2023-07-20 09:54:25 +08:00
bbd7c2a146
[Improvement-14280][Task] update the task execution status check ( #14281 )
...
(cherry picked from commit 4d798e774e
2023-07-20 09:53:58 +08:00
2634cf6350
[Bug][Master][Fixes 14216]workflow state keep READY_STOP when sub_process state … ( #14221 )
...
* [Bug] [Master] workflow state keep READY_STOP when sub_process state STOP
Fixes #14216
* [Bug] [Master] workflow state keep READY_STOP when sub_process state STOP
Fixes mvn spotless:apply#14216
* Update testFile.txt.crc
* Update testFile.txt.crc
* Delete testFile.txt.crc
(cherry picked from commit d587b5f03d
2023-07-20 09:53:43 +08:00
9e0991145e
[Improvement-14146][K8S Task] print complete error logs in task log file ( #14147 )
2023-05-27 17:39:03 +08:00
Jay Chung