9e0991145e
[Improvement-14146][K8S Task] print complete error logs in task log file ( #14147 )
2023-05-27 17:39:03 +08:00
4e78248238
[Feature][K8S Task] support node selector ( #14126 )
...
* feat: support K8S node selector
* fix CI
2023-05-26 09:21:23 +08:00
af7da0630e
[Feature] Add project name/workflow definition name/instance id/task name to the built-in variables ( #14099 )
...
* [Feature] Add project name/workflow definition name/instance id/task name to the built-in variables
2023-05-25 11:02:39 +08:00
606b56403e
[Fix-13381][plugin] fix error while parsing hive load sql ( #13378 )
...
* [fix][task plugin][task sql] fix parsing error while using variables sometimes parsing exception happened if task sql contains a variable followed by \" or \' .
for example:
input sql: load inpath '/tmp/test_table/dt=${dt}'
into table test_table partition(dt=${dt})
after replace: preparing : load inpath '/tmp/test_table/dt=?
into table test_table partition(dt=?)
2023-05-22 11:19:30 +08:00
8315279620
[Feature-#14148][Task] Added status check interval and dependent failure policy parameters for dependent task nodes ( #14150 )
...
* Added a dependent task failure policy in the dependent task node
2023-05-19 14:49:50 +08:00
67f50e3904
[fix-#13940] add status of ``` stop``` in class of TaskExecutionStatus ( #14016 )
2023-05-18 13:31:02 +08:00
c3809fbecf
[Improvement-13868][K8S Task] Support the custom labels for k8s task ( #14067 )
...
* [Improvement-13868] Support the custom labels for k8s task
* fix front-end type mismatched
2023-05-16 12:04:33 +08:00
3eb4c0ed1d
[Improvement-14054][Worker] Kill multiple yarn apps at the same time ( #14055 )
...
* [Improvement-14054][Worker] Kill multiple yarn apps at the same time
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/am/YarnApplicationManager.java
Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
2023-05-06 19:13:12 +08:00
84edea7593
[Improvement-14052][Log] Remove the useless filter in getAppIdsFromLogFile ( #14053 )
2023-05-06 17:00:53 +08:00
3bad68a942
[Bug-13929][K8S Task] Set command contains single quote is invalid ( #13930 )
...
* [Fix-13929][K8S Task] Set command contains single quote is invalid
* fix UT
2023-05-01 11:45:03 +08:00
cf91544b99
Refactor logic task ( #13948 )
2023-04-26 10:01:51 +08:00
db6c061c8b
[Feature][plugin] Add databend datasource in datasource plugin ( #13866 )
...
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
2023-04-24 21:28:14 +08:00
45586f71d5
[BUG][Resource Center] Task can not select main JAR ( #13536 )
2023-04-18 18:38:16 +08:00
119f08d2ba
[Bug-13882][Worker] Upgrade kubernetes-client version to 6.0.0 ( #13891 )
2023-04-17 12:04:36 +08:00
d772f2492b
fix: when sudo enable is false, execute command will failed ( #13860 )
2023-04-07 17:13:22 +08:00
d6fcf60534
[Improvement-13751][Worker] Support real-time pod log collection ( #13752 )
...
* Support real-time pod log collection
* datax task test have to mock ShellCommandExecutor with whole dependencies
* replace \n with EMPTY_STRING
* modify error log msg
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/AbstractCommandExecutor.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
* throw exception in pod log collection thread
---------
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2023-04-05 16:13:45 +08:00
9df89cf0cf
Use bootstrap user as the default tenant ( #13825 )
2023-03-30 14:11:23 +08:00
85605bbef9
[Improvement-13653][Remote Logging] Support api-server to get task log from remote target ( #13654 )
2023-03-30 12:07:58 +08:00
d1b6e6f02c
Take over task instance in master failover ( #13798 )
2023-03-28 16:02:57 +08:00
b98566e070
[Bug-13782] match ')' in setValue regular expression ( #13787 )
2023-03-23 22:56:22 +08:00
2dceb6b8ca
[Feature-13428][Resource Center] Add support for Google Cloud Storage in the resource center ( #13562 )
2023-03-15 19:29:27 +08:00
aef5524ee7
Clearer task runnable. ( #13689 )
...
* Clean unused method in AbstractTask
* Kill task by process.destroy
* wait 5/s after destroy process
2023-03-09 20:46:29 +08:00
c9066e8de9
Use MDC to filter task instance log ( #13673 )
...
* Use MDC to collect task instance log
* Use MDCAutoClosableContext to remove the MDC key
2023-03-06 17:44:23 +08:00
fd720880c7
[Improvement] Add missing placeholder ( #13663 )
...
* Add missing placeholder
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/am/KubernetesApplicationManager.java
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
---------
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2023-03-03 18:02:46 +08:00
ca9df868f1
Fix task instance log order might be wrong due to async print ( #13660 )
2023-03-02 22:37:05 +08:00
7586710d49
[fix-13625] Modify systemd-run from MemoryMax to MemoryLimit for more generality ( #13627 )
...
Co-authored-by: xiecao <haolong.zhl@alibaba-inc.com>
2023-02-28 14:17:54 +08:00
4351a25f2a
Add execute function to handle the workflow instance operation ( #13610 )
2023-02-24 13:47:41 +08:00
7bf3e3cdd6
[improve-#13201] update pid during running ( #13206 )
2023-02-22 14:22:49 +08:00
047fa2f65e
[Feature-13511] Submit Spark task directly on Kubernetes ( #13550 )
2023-02-21 23:26:21 +08:00
2bd65fb2df
[Feature][Remote Logging] Add support for writing task logs to OSS ( #13332 )
2023-02-13 16:38:26 +08:00
16b193454b
[Improvement][Task Plugin] Support Task Plugin output response as a parameter ( #13529 )
...
* support http output response as param
* add UT
* fix sqoop UT
2023-02-08 23:03:21 +08:00
8d12dc0702
[Improvement-13491] Use lombok @Slf4j annotation to generate logger ( #13509 )
2023-02-07 20:32:53 +08:00
5a9cb7ee64
Fix task log might delay until task finished due to batch flush ( #13502 )
2023-02-02 21:48:03 +08:00
78ba71bdbd
[Feature][task]add azure datafactory task type ( #13417 )
...
* init data factory
* add task-type-config.yaml
* add data factory ui
* submit doc
* submit license
---------
Co-authored-by: devosend <devosend@gmail.com>
2023-01-30 11:15:20 +08:00
ba0a253f09
fix k8s task run fail ( #13348 )
2023-01-06 09:44:12 +08:00
6634ff98af
Add source env list in common.properties ( #13317 )
2023-01-04 20:53:52 +08:00
14ec4a2398
Remove dao module in worker ( #13242 )
2022-12-22 12:25:29 +08:00
042ec74a24
[refactor] cache compiled regex pattern ( #13208 )
...
* [refactor] cache compiled regex pattern
* [refactor] cache compiled regex pattern
* [refactor] cache compiled regex pattern
2022-12-18 11:24:11 +08:00
70ccffeee2
Format task parameter as pretty json ( #13173 )
2022-12-13 16:30:21 +08:00
560d232cbd
[Bug][Worker] Fix duplicated key bug when have out parameters of same name ( #12930 )
2022-12-12 11:07:23 +08:00
32a8241ef2
Don't source env file in task, this will override tenant's environment ( #13028 )
2022-11-28 21:09:50 +08:00
d480c271ec
[Bug-12994] [Worker] Fix kill process does not take effect ( #12995 )
2022-11-27 14:27:48 +08:00
92a9b7e243
Use System#lineSeparator to replace /n ( #12889 )
2022-11-15 13:57:45 +08:00
229c554912
[feature][task] Add Kubeflow task plugin for MLOps scenario ( #12843 )
2022-11-11 16:08:38 +08:00
43d69ae157
Upgrade clickhouse jdbc driver ( #12639 )
2022-11-11 13:38:58 +08:00
c4fb555883
Modify shell env to bash ( #12742 )
2022-11-07 11:21:11 +08:00
eaef68856b
Change command file permission to 755 ( #12678 )
...
(cherry picked from commit 95484d930762712f4ab68ece27ca3b65a2073ab2)
2022-11-03 22:51:37 +08:00
a47b4d0672
[Feature][Task] Transfer files between tasks ( #12552 )
...
* add data transfer between tasks
* add delete DATA_TRANSFER API
* convert Result to DeleteDataTransferResponse
* add api UT
* fix final
* fix doc
2022-11-03 16:33:39 +08:00
b71d4983dd
[Feature-6586][Server]add some ds process definition demo when init ( #11759 )
...
* [Feature-6586][Server]add some ds process definition demo when init
1.add some ds process definition demo when init, to display what task type can run and make user easy to
use ds.
2.need configure the JVM parameters (-Ddemo=true) to turn on the StandaloneServer service
3.need modify the tenant information in it
2022-10-31 22:35:11 +08:00
3c31ddfd9f
[Improvement-12536][k8s] Support the command for the container in k8s task plugin ( #12538 )
2022-10-31 19:06:41 +08:00
Aaron Wang