Commit Graph

255 Commits

Author SHA1 Message Date
Wenjun Ruan 9df89cf0cf
Use bootstrap user as the default tenant (#13825) 2023-03-30 14:11:23 +08:00
Rick Cheng 85605bbef9
[Improvement-13653][Remote Logging] Support api-server to get task log from remote target (#13654) 2023-03-30 12:07:58 +08:00
Wenjun Ruan d1b6e6f02c
Take over task instance in master failover (#13798) 2023-03-28 16:02:57 +08:00
Aaron Wang b98566e070
[Bug-13782] match ')' in setValue regular expression (#13787) 2023-03-23 22:56:22 +08:00
Rick Cheng 2dceb6b8ca
[Feature-13428][Resource Center] Add support for Google Cloud Storage in the resource center (#13562) 2023-03-15 19:29:27 +08:00
Wenjun Ruan aef5524ee7
Clearer task runnable. (#13689)
* Clean unused method in AbstractTask

* Kill task by process.destroy

* wait 5/s after destroy process
2023-03-09 20:46:29 +08:00
fuchanghai 6218f818c0
[improve-#13665] improve datax parser if sql contain * (#13680)
* [improve-#13665] improve datax parser if sql contain *

* Update dolphinscheduler-task-plugin/dolphinscheduler-task-datax/src/main/java/org/apache/dolphinscheduler/plugin/task/datax/DataxTask.java

Co-authored-by: fuchanghai <‘2875334588@qq.com’>
Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
2023-03-07 11:16:53 +08:00
Wenjun Ruan c9066e8de9
Use MDC to filter task instance log (#13673)
* Use MDC to collect task instance log
* Use MDCAutoClosableContext to remove the MDC key
2023-03-06 17:44:23 +08:00
Aaron Wang fd720880c7
[Improvement] Add missing placeholder (#13663)
* Add missing placeholder

* Update dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/am/KubernetesApplicationManager.java

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>

---------

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2023-03-03 18:02:46 +08:00
Wenjun Ruan ca9df868f1
Fix task instance log order might be wrong due to async print (#13660) 2023-03-02 22:37:05 +08:00
旺阳 43d79e45b4
Fix data quality pwd error (#13643) 2023-02-28 18:50:44 +08:00
QuantumXiecao 7586710d49
[fix-13625] Modify systemd-run from MemoryMax to MemoryLimit for more generality (#13627)
Co-authored-by: xiecao <haolong.zhl@alibaba-inc.com>
2023-02-28 14:17:54 +08:00
Wenjun Ruan 4351a25f2a
Add execute function to handle the workflow instance operation (#13610) 2023-02-24 13:47:41 +08:00
fuchanghai 79d3ed0744
Update DataxUtils.java (#13604) 2023-02-22 19:01:43 +08:00
fuchanghai 7bf3e3cdd6
[improve-#13201] update pid during running (#13206) 2023-02-22 14:22:49 +08:00
Aaron Wang 047fa2f65e
[Feature-13511] Submit Spark task directly on Kubernetes (#13550) 2023-02-21 23:26:21 +08:00
Rick Cheng 2bd65fb2df
[Feature][Remote Logging] Add support for writing task logs to OSS (#13332) 2023-02-13 16:38:26 +08:00
fuchanghai 37939add0c
[fix-13492]add single quotes (#13534)
* [fix-13492]add single quotes

* [improve-#13201] fix test error

---------

Co-authored-by: fuchanghai <‘2875334588@qq.com’>
2023-02-09 19:02:27 +08:00
JieguangZhou 16b193454b
[Improvement][Task Plugin] Support Task Plugin output response as a parameter (#13529)
* support http output response as param

* add UT

* fix sqoop UT
2023-02-08 23:03:21 +08:00
seedscoder 8d12dc0702
[Improvement-13491] Use lombok @Slf4j annotation to generate logger (#13509) 2023-02-07 20:32:53 +08:00
shenyun 8fa90c2d8f
feature flinksql support standalone deploy mode (#13495) 2023-02-05 20:37:56 +08:00
Wenjun Ruan 5a9cb7ee64
Fix task log might delay until task finished due to batch flush (#13502) 2023-02-02 21:48:03 +08:00
Tq 78ba71bdbd
[Feature][task]add azure datafactory task type (#13417)
* init data factory

* add task-type-config.yaml

* add data factory ui

* submit doc

* submit license

---------

Co-authored-by: devosend <devosend@gmail.com>
2023-01-30 11:15:20 +08:00
LucasClt a1f5675012
[Bug][TaskPlugin] Task execution error occurs when the Zeppelin task zeppelinNoteId parameter value is an empty string (#13162)
* [Bug] [TaskPlugin-Zeppelin] Task execution error occurs when the Zeppelin task zeppelinNoteId parameter value is an empty string. #13161
2023-01-06 19:29:05 +08:00
caishunfeng ba0a253f09
fix k8s task run fail (#13348) 2023-01-06 09:44:12 +08:00
Wenjun Ruan 6634ff98af
Add source env list in common.properties (#13317) 2023-01-04 20:53:52 +08:00
JieguangZhou df491f78cc
fix sagemaker sync status (#13264) 2022-12-22 20:17:17 +08:00
Wenjun Ruan 14ec4a2398
Remove dao module in worker (#13242) 2022-12-22 12:25:29 +08:00
Wenjun Ruan b2336b0ce9
Remove datasource in procedure task (#13198) 2022-12-19 17:14:59 +08:00
youzipi 042ec74a24
[refactor] cache compiled regex pattern (#13208)
* [refactor] cache compiled regex pattern

* [refactor] cache compiled regex pattern

* [refactor] cache compiled regex pattern
2022-12-18 11:24:11 +08:00
HuangWei 43e6ac32dd
[Fix-13187] refactor OpenmldbParameters (#13190) 2022-12-14 20:13:45 +08:00
Wenjun Ruan 838a0df783
Remove logic task plugin module (#13171) 2022-12-13 16:30:44 +08:00
Wenjun Ruan 70ccffeee2
Format task parameter as pretty json (#13173) 2022-12-13 16:30:21 +08:00
Wenjun Ruan ab96a3d0c0
Fix datasource jdbc url param doesn't work (#13098) 2022-12-12 11:46:40 +08:00
Aaron Wang 560d232cbd
[Bug][Worker] Fix duplicated key bug when have out parameters of same name (#12930) 2022-12-12 11:07:23 +08:00
Yann Ann 6ef74073cc
[Improve-13001]migrate commons-collections -> commons-collections4 (#13002) 2022-12-10 23:50:19 +08:00
Wenjun Ruan 169168ef34
Add plugin-all module (#13079) 2022-12-02 23:19:08 +08:00
Eric Gao c2ae76c0bd
Expand all static imports and add CI check to block wildcard static imports (#13067) 2022-12-01 14:15:28 +08:00
Wenjun Ruan 32a8241ef2
Don't source env file in task, this will override tenant's environment (#13028) 2022-11-28 21:09:50 +08:00
Kerwin ae670e6e1c
[Bug-13010] [Task] The Flink SQL task page selects the pre-job deployment mode, but the task executed by the worker is the Flink local mode (#13011) 2022-11-28 10:09:49 +08:00
Kerwin d480c271ec
[Bug-12994] [Worker] Fix kill process does not take effect (#12995) 2022-11-27 14:27:48 +08:00
Wenjun Ruan a2ff140f43
Fix sql task will send alert if we don't choose the send email (#12984) 2022-11-24 21:26:52 +08:00
rickchengx 38b876733c
[Feature-10498] Mask the password in the log of sqoop task (#11589) 2022-11-24 14:54:54 +08:00
JieguangZhou 6e224d08a2
[fix] Python task can not pass the parameters to downstream task. (#12961) 2022-11-22 16:11:45 +08:00
JieguangZhou 7c90bf01bc
[Improvement][Task Plugin] Improvement Kubeflow task plugin (#12928)
add example
check phase in status:conditions
2022-11-17 15:59:44 +08:00
Assert 2dbc79693e
[feature][task plugin] Add new task plugin for apache linkis (#12693) 2022-11-17 10:14:36 +08:00
Wenjun Ruan 92a9b7e243
Use System#lineSeparator to replace /n (#12889) 2022-11-15 13:57:45 +08:00
JieguangZhou 229c554912
[feature][task] Add Kubeflow task plugin for MLOps scenario (#12843) 2022-11-11 16:08:38 +08:00
simsicon 43d69ae157
Upgrade clickhouse jdbc driver (#12639) 2022-11-11 13:38:58 +08:00
Kerwin c4fb555883
Modify shell env to bash (#12742) 2022-11-07 11:21:11 +08:00