2119e41800
[Improvement] Move delay calculation to Master ( #15278 )
2023-12-06 09:37:14 +08:00
db3d84b73d
[HotFix] Fix TaskOutputParameterParser might OOM if meed a bad output param expression ( #15264 )
2023-12-04 18:38:38 +08:00
ef2e9c9b67
Set the tenant as the owner in final stage ( #15256 )
2023-12-01 21:49:28 +08:00
71ee1f0c3d
Support parse task output params under multiple log ( #15244 )
2023-12-01 10:16:36 +08:00
df656a7671
Fix home page workflow instance miss status ( #15193 )
2023-11-22 10:33:24 +08:00
5afd94136c
[Fix][Data Quality] Change t_ds_dq_rule_input_entry field name fix PostgreSQL not support value issue ( #14992 )
...
* change data quality field name
* fix-dao
* fix-dao
* fix-dao
* fix-dao
* update
* test
* update
* update
2023-11-13 11:04:30 +08:00
c532fea00e
[Improvement] Refactoring K8S task plugin with connections managed in connection center ( #14977 )
2023-10-31 17:07:24 +08:00
5a3827eef4
[DSIP-19] Support sagemaker connections in the connection center, as well as external connections to the connection center in sagemaker tasks ( #14976 )
...
* Refactoring `Sagemaker` task plugin with connections managed in connection center.
---------
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
2023-10-31 10:07:05 +08:00
e99c5950b9
[DSIP-19] Support zeppelin connections in the connection center, as well as external connections to the connection center in zeppelin tasks ( #14434 )
...
* Refactoring zeppelin task plugin with connections managed in connection center
---------
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
2023-10-30 16:44:54 +08:00
99214d0fc3
Fix SqlTask cannot split the given sql when browser in windows ( #15062 )
2023-10-26 19:50:32 +08:00
688f84452d
[Feature-14678][Master][UI]Dependent task parameter passing ( #14702 )
...
* feat: dependent task parameter passing
* style: code format
* refactor: remove inappropriate log
* refactor: Modify unreasonable code structure
* refactor: Modify unreasonable code structure
* refactor: Solve conflicts caused by merge
---------
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
Co-authored-by: BaoLiang <29528966+lenboo@users.noreply.github.com>
2023-10-25 16:24:30 +08:00
049c1511fa
[Improvement][K8S] K8S task support passing context parameters to downstream task ( #14934 )
...
* [Improvement][K8S] K8S task support passing context parameters to downstream task
Signed-off-by: Gallardot <gallardot@apache.org>
* [Improvement][K8S] K8S task support passing context parameters to downstream task
Signed-off-by: Gallardot <gallardot@apache.org>
* chore: update doc img
Signed-off-by: Gallardot <gallardot@apache.org>
* chore: update doc
Signed-off-by: Gallardot <gallardot@apache.org>
* chore: add UT
Signed-off-by: Gallardot <gallardot@apache.org>
* merge suggestion
Co-authored-by: 旺阳 <wang@lqwang.net>
* merge suggestion
Co-authored-by: 旺阳 <wang@lqwang.net>
* merge suggestion
Co-authored-by: 旺阳 <wang@lqwang.net>
---------
Signed-off-by: Gallardot <gallardot@apache.org>
Co-authored-by: David Zollo <dailidong66@gmail.com>
Co-authored-by: 旺阳 <qingwli@cisco.com>
Co-authored-by: 旺阳 <wang@lqwang.net>
Co-authored-by: xiangzihao <460888207@qq.com>
2023-10-08 17:27:10 +08:00
f90eadd40f
[feature-#14449] add secret in k8s Task ( #14930 )
...
* [feature-#14449] add secret
* [feature-#14449] upgrade UT and rename parameter ```secret```
* [feature-#14449] fix UT
* [feature-#14449] fix UT
* [feature-#14449] improvement
* [feature-#14449] format
---------
Co-authored-by: fuchanghai <‘2875334588@qq.com’>
2023-09-28 11:06:35 +08:00
6de1e2c1b3
Fix the error of using shell task to obtain Home variable in Ubuntu system. ( #14964 )
2023-09-26 17:18:38 +08:00
e748c2eb9a
fix the #14729 problem ( #14902 )
...
Co-authored-by: 刘阳 <liuy2590@chinaunicom.cn>
Co-authored-by: xiangzihao <460888207@qq.com>
2023-09-21 18:46:14 +08:00
9083f9d0e0
[Fix-14546] seatunnel unable submit spark job to spark ( #14617 )
2023-09-19 13:23:27 +08:00
f6a6794289
[Bug][Worker] Global and local parameters are not replaced in SQL components ( #14926 )
...
Signed-off-by: Gallardot <gallardot@apache.org>
2023-09-15 10:31:14 +08:00
3fa69d2db8
[Bug][Worker] Fix sql/shell task doesn't replace parameter ( #14876 )
...
Fix parameter assign error, fix #14869 which including
* shell task param not work, botch local and global
* others task chain assign not work
2023-09-08 10:09:16 +08:00
7640c29495
[Improvement] [Task] Change Seatunnel task run-mode to deploy-mode ( #14800 )
2023-09-01 12:35:39 +08:00
55cc77226f
[RPC] Refactor the remote module. ( #14776 )
...
* Add dolphinscheduler-extract-base
* Add dolphinscheduler-extract-master
* Add dolphinscheduler-extract-worker
* Add dolphinscheduler-extract-alert
2023-08-23 09:43:17 +08:00
a491e3a849
fix: null app-info-path cause NPE ( #14752 )
2023-08-19 13:28:45 +08:00
59998f4c75
[Improvement] add dispatch fail times when compare ( #14761 )
...
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
2023-08-18 15:26:01 +08:00
b3af374fe6
[Fix-14646][Worker] Fix match wrong pid on linux server ( #14652 )
...
* [DS-14646][Worker] Fix match wrong pid on linux server
This closes #14646
---------
Co-authored-by: yangxin3 <yangxin3@newhope.cn>
2023-08-17 19:17:16 +08:00
25480ae9e7
[Feature][Resource Center] Add support for Huawei Cloud OBS ( #14643 )
...
* [Feature][Resource Center] Add support for Huawei Cloud OBS as storage of resource center
* add license and add doc
* add 3-party dependency license
* Update LICENSE
* fix
* Update pom.xml
* fix
* fix
---------
Co-authored-by: sunkang <sunkang@hjfruit.com>
Co-authored-by: xiangzihao <460888207@qq.com>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
2023-08-14 13:25:48 +08:00
cb55476235
[Fix-14721] [K8S Task] Handle job delete event when user manually delete the job in k8s cluster ( #14722 )
...
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
2023-08-11 17:15:07 +08:00
5ec9085113
fix: do not override k8sConfigYaml in ShShellInterceptorBuilder ( #14682 )
...
Co-authored-by: xiangzihao <460888207@qq.com>
2023-08-02 18:01:34 +08:00
9ce8871158
Support get property from env/jvm/common.properties ( #14645 )
2023-07-28 14:41:58 +08:00
512f4c7021
[Fix-10795][plugin] sqoop task query columns bug ( #10776 )
...
* [Improvement][Api] add @Override on the Override method
* [FIX]sqoop task query columns bug
* merge two if
* update checkstyle problem
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-sqoop/src/main/java/org/apache/dolphinscheduler/plugin/task/sqoop/SqoopColumnType.java
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-sqoop/src/main/java/org/apache/dolphinscheduler/plugin/task/sqoop/generator/sources/MySQLSourceGenerator.java
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
* code format
* code format
---------
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: xiangzihao <460888207@qq.com>
2023-07-26 10:42:29 +08:00
4aab0b234f
Use AdHoc datasource client in sqlTask ( #14631 )
...
* Use AdHoc datasource client in sqlTask
* Add method in DataSourceChannel to create PooledDataSourceClient and AdHocDataSourceClient
2023-07-26 08:58:50 +08:00
d6a4e99e19
fix process utils comment typo ( #14613 )
2023-07-21 13:18:10 +08:00
04800a48d1
[Feature-13763][K8S Task] collect real-time log ( #14379 )
...
* [Improvement-13763][K8S Task] collect real-time log
* fix codesmell
* get pod watcher until pod is ready
* fix codesmell
* specify container name & loop waiting pod creation
* sleep when pod is not initialized
---------
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
2023-07-20 18:38:25 +08:00
9664bef933
Fix wrong judgment condition when building parameters of seatunnel task node. ( #14594 )
2023-07-20 12:26:10 +08:00
de2d215aee
impv: Correct datax execute and python execute script name ( #14523 )
...
* impv: Correct datax execute and python execute script name
we use PYTHON_LAUNCHER for python script execute and
DATAX_LAUNCHER for datax script name
* Add pr number
* fix ut
* style
---------
Co-authored-by: xiangzihao <460888207@qq.com>
2023-07-20 09:13:55 +08:00
c30cca9d9a
Support execute shell in different interceptor ( #14582 )
2023-07-19 14:50:04 +08:00
9778652f17
[Improvement-14136][task] Support submitting spark-sql task with the sql file in resource center ( #14527 )
...
Co-authored-by: xiangzihao <460888207@qq.com>
2023-07-19 10:09:30 +08:00
027d66d200
[Typo] standardized license header. ( #14548 )
...
Co-authored-by: wentz <wentz@getui.com>
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
2023-07-17 13:56:23 +08:00
e6d94632b9
Add WorkflowExecuteContext ( #14544 )
2023-07-14 18:40:01 +08:00
09afa75a3d
[Improvement][Worker] No need to kill remote app when appid is empty ( #14534 )
2023-07-13 15:40:58 +08:00
c1cffba349
[Improvement] Remove the unused newDaemonSingleThreadExecutor ( #14520 )
2023-07-12 21:03:41 +08:00
4111acb58e
[Bug] [Seatunnel-Task] Run Mode defaults to run, and you cannot select a null value ( #14267 )
...
Co-authored-by: zhang <zhangyfcoder@163.com>
2023-07-12 09:06:19 +08:00
175d976cd2
[Feature][Data Quality] Data Quality Support Choose Database ( #14406 )
...
* add dataquality database api
* change ui
* api change
* update
* fix spotless
* fix h2
* fix pg
* fix-dead-line
* update
* fix-spotless
* update pg sql
* add ut
* fix ut
2023-07-12 09:00:28 +08:00
75d29f6101
[improvement] DataQuality module improve ( #14463 )
2023-07-10 13:38:09 +08:00
0b69236b04
[Feature-14251][Task] Support yarn queue definition in yarn task ( #14310 )
2023-07-07 14:24:47 +08:00
742a2dc780
[Fix-14447][K8S Task] Fix IndexOutofBoundsException for pod name without dash ( #14448 )
2023-07-07 12:53:16 +08:00
cff53feaeb
[Improvement-14464][Task] Remove the useless taskResultString ( #14465 )
2023-07-06 17:31:33 +08:00
68a3a75211
[Improvement-14457][UT] Improve the expected and actual value of Assertions ( #14459 )
2023-07-06 13:12:48 +08:00
65b6a4b097
[Feature-14421][K8S Task] Configurable image pull policy ( #14426 )
2023-07-05 09:17:47 +08:00
13dfa859b5
[Improvement-14371][Hivecli] Add support for hivevar setting in hivecli task ( #14423 )
2023-07-04 14:02:23 +08:00
9c92b4b240
fix: default datax plugin ( #14204 )
2023-06-25 16:18:38 +08:00
b49b91c45d
[Improvement-14331][common] Remove the duplicate ParameterUtils ( #14332 )
2023-06-14 15:00:54 +08:00
a55612e4f7
Fix switch task use boolean type error ( #14326 )
2023-06-13 13:26:05 +08:00
1cd5070b5d
[Improvement-14329][common] Remove the duplicate BusinessTimeUtils ( #14330 )
2023-06-13 08:30:34 +08:00
4fa0ab4d7e
[feature][Resource] Add support for Azure Blob Storage in the resource center ( #14082 )
2023-06-12 12:05:49 +08:00
de2cc0e235
[bug][plugin]Fix: Correct the way to determine the yarn queue in Flink CommandLine and SQL mode ( #14237 )
...
* Fix: Correct the way to determine the yarn queue in Flink CommandLine
* fix the yarn queue in sql mode && refine the code
* refine code
* remove unnecessary comment
* fix yarn queue properties
* remove redundant variable
2023-06-12 11:04:39 +08:00
b7fb393eed
fix: trim node selector requirement value ( #14307 )
2023-06-09 18:06:52 +08:00
66e09dc66e
[Improvement-14295][common] Remove the duplicate FileUtils ( #14301 )
2023-06-08 08:59:53 +08:00
e7cdc7c136
[Feature][Task Plugin] Add Dynamic task to generate dynamic worklfows with list parameters ( #14127 )
...
* add dynamic task plugin
* fix ui
* update code
* add DOC
* ADD UT
* fix doc
* fix sanity check
* add upgrade ddl
* fix sql
2023-06-07 16:01:06 +08:00
2c057d6a8f
[Improvement-14292][Common] Remove the duplicate OSUtils ( #14293 )
2023-06-07 14:59:25 +08:00
4d798e774e
[Improvement-14280][Task] update the task execution status check ( #14281 )
2023-06-06 15:25:57 +08:00
d587b5f03d
[Bug][Master][Fixes 14216]workflow state keep READY_STOP when sub_process state … ( #14221 )
...
* [Bug] [Master] workflow state keep READY_STOP when sub_process state STOP
Fixes #14216
* [Bug] [Master] workflow state keep READY_STOP when sub_process state STOP
Fixes mvn spotless:apply#14216
* Update testFile.txt.crc
* Update testFile.txt.crc
* Delete testFile.txt.crc
2023-06-06 11:04:32 +08:00
9e0991145e
[Improvement-14146][K8S Task] print complete error logs in task log file ( #14147 )
2023-05-27 17:39:03 +08:00
1011852ab2
[Improvement-14195][Parameter] remove misleading comment of combining local and global parameter ( #14196 )
2023-05-26 09:29:09 +08:00
4e78248238
[Feature][K8S Task] support node selector ( #14126 )
...
* feat: support K8S node selector
* fix CI
2023-05-26 09:21:23 +08:00
af7da0630e
[Feature] Add project name/workflow definition name/instance id/task name to the built-in variables ( #14099 )
...
* [Feature] Add project name/workflow definition name/instance id/task name to the built-in variables
2023-05-25 11:02:39 +08:00
606b56403e
[Fix-13381][plugin] fix error while parsing hive load sql ( #13378 )
...
* [fix][task plugin][task sql] fix parsing error while using variables sometimes parsing exception happened if task sql contains a variable followed by \" or \' .
for example:
input sql: load inpath '/tmp/test_table/dt=${dt}'
into table test_table partition(dt=${dt})
after replace: preparing : load inpath '/tmp/test_table/dt=?
into table test_table partition(dt=?)
2023-05-22 11:19:30 +08:00
8315279620
[Feature-#14148][Task] Added status check interval and dependent failure policy parameters for dependent task nodes ( #14150 )
...
* Added a dependent task failure policy in the dependent task node
2023-05-19 14:49:50 +08:00
56b0f91f6e
fix: Save task seatunnel error ( #14129 )
2023-05-19 10:47:16 +08:00
67f50e3904
[fix-#13940] add status of ``` stop``` in class of TaskExecutionStatus ( #14016 )
2023-05-18 13:31:02 +08:00
c3809fbecf
[Improvement-13868][K8S Task] Support the custom labels for k8s task ( #14067 )
...
* [Improvement-13868] Support the custom labels for k8s task
* fix front-end type mismatched
2023-05-16 12:04:33 +08:00
a3b2cc0275
[CI] Try fix api test error ( #14094 )
...
* try fix api test error
* add workergroupapitest to ci and fix log output
* fix log output
* skip remote shell UT
---------
Co-authored-by: SbloodyS <460888207@qq.com>
2023-05-12 21:39:00 +08:00
3eb4c0ed1d
[Improvement-14054][Worker] Kill multiple yarn apps at the same time ( #14055 )
...
* [Improvement-14054][Worker] Kill multiple yarn apps at the same time
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/am/YarnApplicationManager.java
Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
2023-05-06 19:13:12 +08:00
84edea7593
[Improvement-14052][Log] Remove the useless filter in getAppIdsFromLogFile ( #14053 )
2023-05-06 17:00:53 +08:00
3bad68a942
[Bug-13929][K8S Task] Set command contains single quote is invalid ( #13930 )
...
* [Fix-13929][K8S Task] Set command contains single quote is invalid
* fix UT
2023-05-01 11:45:03 +08:00
cf91544b99
Refactor logic task ( #13948 )
2023-04-26 10:01:51 +08:00
db6c061c8b
[Feature][plugin] Add databend datasource in datasource plugin ( #13866 )
...
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
2023-04-24 21:28:14 +08:00
ec29220977
[improve-#13966] mask datax log ( #13969 )
...
Co-authored-by: fuchanghai <‘2875334588@qq.com’>
2023-04-21 14:51:37 +08:00
34f5b009e1
[Fix 13941]Fix replace time variable ( #13975 )
...
* fix replace time variable
Co-authored-by: <ziheng.zhan@longbridge.sg>
2023-04-21 14:37:52 +08:00
45586f71d5
[BUG][Resource Center] Task can not select main JAR ( #13536 )
2023-04-18 18:38:16 +08:00
35bb533a50
[Improvement][Task Plugin] Integrate zeppelin password authentication with DS zeppelin task plugin ( #13861 )
...
---------
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
2023-04-18 12:18:51 +08:00
119f08d2ba
[Bug-13882][Worker] Upgrade kubernetes-client version to 6.0.0 ( #13891 )
2023-04-17 12:04:36 +08:00
4439c1ec90
[Feature-13052][Task Plugin] Support Apache Seatunnel Connector-V2 ( #13086 )
2023-04-08 22:02:19 +08:00
d772f2492b
fix: when sudo enable is false, execute command will failed ( #13860 )
2023-04-07 17:13:22 +08:00
d6fcf60534
[Improvement-13751][Worker] Support real-time pod log collection ( #13752 )
...
* Support real-time pod log collection
* datax task test have to mock ShellCommandExecutor with whole dependencies
* replace \n with EMPTY_STRING
* modify error log msg
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/AbstractCommandExecutor.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
* throw exception in pod log collection thread
---------
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2023-04-05 16:13:45 +08:00
a1861bc2df
Use DATAX_PYTHON to specify a datax python version ( #13849 )
...
(cherry picked from commit 752452ecbd2f8094b819eae219f3eec0f6a6427b)
2023-04-03 08:47:32 +08:00
273a5e78eb
[FEATURE][Task Plugin]Add remote-shell task plugin ( #13801 )
2023-03-30 20:17:28 +08:00
9df89cf0cf
Use bootstrap user as the default tenant ( #13825 )
2023-03-30 14:11:23 +08:00
85605bbef9
[Improvement-13653][Remote Logging] Support api-server to get task log from remote target ( #13654 )
2023-03-30 12:07:58 +08:00
d1b6e6f02c
Take over task instance in master failover ( #13798 )
2023-03-28 16:02:57 +08:00
b98566e070
[Bug-13782] match ')' in setValue regular expression ( #13787 )
2023-03-23 22:56:22 +08:00
2dceb6b8ca
[Feature-13428][Resource Center] Add support for Google Cloud Storage in the resource center ( #13562 )
2023-03-15 19:29:27 +08:00
aef5524ee7
Clearer task runnable. ( #13689 )
...
* Clean unused method in AbstractTask
* Kill task by process.destroy
* wait 5/s after destroy process
2023-03-09 20:46:29 +08:00
6218f818c0
[improve-#13665] improve datax parser if sql contain * ( #13680 )
...
* [improve-#13665] improve datax parser if sql contain *
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-datax/src/main/java/org/apache/dolphinscheduler/plugin/task/datax/DataxTask.java
Co-authored-by: fuchanghai <‘2875334588@qq.com’>
Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
2023-03-07 11:16:53 +08:00
c9066e8de9
Use MDC to filter task instance log ( #13673 )
...
* Use MDC to collect task instance log
* Use MDCAutoClosableContext to remove the MDC key
2023-03-06 17:44:23 +08:00
fd720880c7
[Improvement] Add missing placeholder ( #13663 )
...
* Add missing placeholder
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/am/KubernetesApplicationManager.java
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
---------
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2023-03-03 18:02:46 +08:00
ca9df868f1
Fix task instance log order might be wrong due to async print ( #13660 )
2023-03-02 22:37:05 +08:00
43d79e45b4
Fix data quality pwd error ( #13643 )
2023-02-28 18:50:44 +08:00
7586710d49
[fix-13625] Modify systemd-run from MemoryMax to MemoryLimit for more generality ( #13627 )
...
Co-authored-by: xiecao <haolong.zhl@alibaba-inc.com>
2023-02-28 14:17:54 +08:00
4351a25f2a
Add execute function to handle the workflow instance operation ( #13610 )
2023-02-24 13:47:41 +08:00
79d3ed0744
Update DataxUtils.java ( #13604 )
2023-02-22 19:01:43 +08:00
7bf3e3cdd6
[improve-#13201] update pid during running ( #13206 )
2023-02-22 14:22:49 +08:00
Wenjun Ruan