0c470ffe66
[ Fix #15129 ] [Dependent] The date rules of the dependent node are ambiguous. ( #15289 )
...
* [Fix-15129][Dependent] Fix the ambiguity in date rules for dependent node.
* [fix #15129 ] Revert ddl
* restore findLastProcessInterval
* update: mvn spotless:apply
---------
Co-authored-by: 李乐 <lile@boimc.com>
Co-authored-by: xiangzihao <460888207@qq.com>
2023-12-08 16:18:33 +08:00
14272dafab
Fix failover Master might not release taskGroup ( #15287 )
2023-12-06 13:31:46 +08:00
2119e41800
[Improvement] Move delay calculation to Master ( #15278 )
2023-12-06 09:37:14 +08:00
43f5f24529
Add dolphinscheduler-extract-common module ( #15266 )
2023-12-05 21:06:39 +08:00
b5eddc0ce8
fix switch condition ( #15228 )
...
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
2023-11-27 14:03:37 +08:00
1d23e4fe5d
Remove spring cache for dao ( #15184 )
2023-11-24 14:54:05 +08:00
cca1c4a172
fix can't stop bug ( #15191 )
...
Co-authored-by: Leoric Yue <leoric.yue@zoom.us>
2023-11-20 21:02:44 +08:00
d6175f3bbf
Exclude DataSourceAutoConfiguration in worker server ( #15169 )
2023-11-16 19:17:39 +08:00
5afd94136c
[Fix][Data Quality] Change t_ds_dq_rule_input_entry field name fix PostgreSQL not support value issue ( #14992 )
...
* change data quality field name
* fix-dao
* fix-dao
* fix-dao
* fix-dao
* update
* test
* update
* update
2023-11-13 11:04:30 +08:00
2622685784
[Bug] Fix endless loop ( #15092 )
...
* [fix-#13045] fix endless loop
---------
Co-authored-by: fuchanghai <‘2875334588@qq.com’>
2023-11-08 11:03:55 +08:00
d675d32771
Fix confusing constant string for unit convertor ( #15126 )
2023-11-06 21:32:36 +08:00
c0ed68121d
[Feature-14832][Listener]Implementation of Listener Mechanism ( #14981 )
...
* first commit
* 1. sql: sync ddl
2. front-end: change to ternary expression
3. back-end: correct license header in ListenerEvent.java
* test case
* frontend: remove unnecessary console
* fix unit test
* remove log depends on user-provided value
* fix dolphinscheduler_postgresql.sql
* sync database schema
* fix unit test
* fix unit test
* fix some NIT.
* extract GLOBAL_ALERT_GROUP_ID into variable
* fix ddl bug
* add column task_type in t_ds_fav_task in upgrade/3.2.0_schema
* add unit test
2023-10-29 20:28:05 +08:00
a5284e4024
Remove mapper usage in tools ( #15073 )
2023-10-25 22:58:47 +08:00
688f84452d
[Feature-14678][Master][UI]Dependent task parameter passing ( #14702 )
...
* feat: dependent task parameter passing
* style: code format
* refactor: remove inappropriate log
* refactor: Modify unreasonable code structure
* refactor: Modify unreasonable code structure
* refactor: Solve conflicts caused by merge
---------
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
Co-authored-by: BaoLiang <29528966+lenboo@users.noreply.github.com>
2023-10-25 16:24:30 +08:00
c59b2d5b8c
remove sub workflow finish notify ( #15057 )
...
Co-authored-by: xiangzihao <460888207@qq.com>
2023-10-23 10:19:57 +08:00
33084d281f
Add dolphinscheduler-dao-plugin module ( #15019 )
2023-10-14 17:58:40 +08:00
d683553324
[Improvement][Master] Calculate the remainTime then we set the delay execution. ( #15012 )
...
* [#15007 ][fix][worker-server] fix bug task delay execution
* [#15007 ][fix][worker-server] fix bug task delay execution
mvn spotless:apply
* [#15007 ][fix][worker-server] fix bug task delay execution
worker fix task delay logic
---------
Co-authored-by: 旺阳 <wang@lqwang.net>
2023-10-11 02:32:07 -05:00
629fced9ef
Fix task log appender will not be closed immediate after throwing exception ( #14989 )
2023-10-07 20:19:02 -05:00
73e846d03e
[Improvement-14884][Master] Add overload state in master heartbeat to trigger slot change ( #14887 )
2023-09-14 10:36:42 +08:00
76bbcbeb30
[Improvement][Master] Split the task dependency mode of the dependent node into workflow dependent and task dependent ( #14824 )
...
* [Improvement][Master] Split the task dependency mode of the dependent node into workflow dependent and task dependent. (#11970 )
* [Improvement][Master] Split the task dependency mode of the dependent node into workflow dependent and task dependent. (#11970 )
- delete useless output
* [Improvement][Master] Split the task dependency mode of the dependent node into workflow dependent and task dependent. (#11970 )
- add log
- run spotless check
---------
Co-authored-by: 旺阳 <qingwli@cisco.com>
Co-authored-by: JinYong Li <42576980+JinyLeeChina@users.noreply.github.com>
2023-09-09 17:23:34 +08:00
7ae4fb3787
[Fix-14149][master] task finished so not dispatch ( #14161 )
2023-08-31 20:08:09 +08:00
33525a17c3
[Fix-14828][master] check task id not null to avoid NPE ( #14829 )
...
Co-authored-by: eye <eye.gu@aloudata.com>
Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2023-08-31 18:34:31 +08:00
742f0c3e60
Add workflowInstanceHost and taskInstanceHost in ITaskInstanceExecutionEvent ( #14827 )
2023-08-29 13:32:17 +08:00
d223d654cc
Add rpc benchmark test ( #14797 )
2023-08-27 20:59:55 +08:00
9c2838ccc9
fix the status of logical task is not be correctly changed when task is finished. ( #14820 ) ( #14822 )
2023-08-27 13:20:37 +08:00
ac04288e2b
fix the full log path of logical task is not be set when task is dispatched ( #14819 ) ( #14821 )
2023-08-27 12:24:04 +08:00
55cc77226f
[RPC] Refactor the remote module. ( #14776 )
...
* Add dolphinscheduler-extract-base
* Add dolphinscheduler-extract-master
* Add dolphinscheduler-extract-worker
* Add dolphinscheduler-extract-alert
2023-08-23 09:43:17 +08:00
59998f4c75
[Improvement] add dispatch fail times when compare ( #14761 )
...
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
2023-08-18 15:26:01 +08:00
8e64026e93
fix fault tolerant enviroment missing ( #14735 )
2023-08-11 14:35:31 +08:00
7c37a7f9fe
[Fix-14714][master] package task instance before put into cache ( #14718 )
...
Co-authored-by: eye <eye.gu@aloudata.com>
Co-authored-by: xiangzihao <460888207@qq.com>
2023-08-09 13:41:28 +08:00
db62ce0e47
Fix serial mode will cause NPE in Workflow bootstrap ( #14703 )
2023-08-04 23:43:57 +08:00
9963bfc612
[Workflow] Print WorkflowInstance detail/ TaskInstance detail when finished ( #14677 )
...
* [Workflow] Print WorkflowInstance detail/ TaskInstance detail when finished
* [Workflow] minor fix
* [Workflow] add ut
* [Workflow] add header
---------
Co-authored-by: tengting.xu <xtt@dipeak.com>
2023-08-01 20:50:00 +08:00
4a80e3a021
[DS-14566][master] fix serial_wait command_param does not replace global_params ( #14606 )
...
This closes #14566
Co-authored-by: yangxin3 <yangxin3@newhope.cn>
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
Co-authored-by: xiangzihao <460888207@qq.com>
2023-07-25 13:36:35 +08:00
ca760aabd4
[Fix-14512][BOM] kubernetes-client version conflict ( #14561 )
2023-07-21 18:17:53 +08:00
cbd354d134
Add host/port in heartbeat ( #14591 )
2023-07-20 21:26:35 +08:00
af2986eca1
[Feature-14545][Master][UI]Cross workflow parameter passing ( #14552 )
...
* feat: cross-workflow parameter passing
* refactor: remove useless parameters
* refactor: remove useless code
* refactor: code format
* docs: update docs
* docs: docs format
* more effective information
Co-authored-by: xiangzihao <zihaoxiang@apache.org>
* more effective information
Co-authored-by: xiangzihao <zihaoxiang@apache.org>
* fix: wrong writing of log code
* fix: perfect error log
---------
Co-authored-by: xiangzihao <zihaoxiang@apache.org>
Co-authored-by: xiangzihao <460888207@qq.com>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
2023-07-19 18:00:44 +08:00
399d0d0ce9
Add log for NetUtils ( #14578 )
2023-07-18 21:17:17 +08:00
e6d94632b9
Add WorkflowExecuteContext ( #14544 )
2023-07-14 18:40:01 +08:00
c812bf9d49
[Bug] [Mater] Not got latest task intance by task code ( #14529 )
...
* add task code->instance map
---------
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
2023-07-14 14:51:39 +08:00
0246327083
Polish config, expose rpc config in application.yml ( #14501 )
2023-07-11 09:29:31 +08:00
406aeaf659
Use jvm_args_env.sh to set JVM args ( #14487 )
2023-07-10 17:39:57 +08:00
0d2ea3e27b
Fix worker failover service comment ( #14494 )
2023-07-10 15:54:06 +08:00
0b69236b04
[Feature-14251][Task] Support yarn queue definition in yarn task ( #14310 )
2023-07-07 14:24:47 +08:00
d38d504332
[Fix] Fix running task instance throught api gots failed ( #14433 )
...
* update logic
* split method
2023-07-06 11:38:26 +08:00
64e5741063
[Bug-14378][Master] Trigger task execution in workflow instance page cause workflow instance keep running status ( #14389 )
2023-06-24 22:10:14 +08:00
5cbe170e43
[Bug] [Master] Fix complementListDate will throw IndexOutOfBoundsException ( #14361 )
2023-06-19 23:23:05 +08:00
b49b91c45d
[Improvement-14331][common] Remove the duplicate ParameterUtils ( #14332 )
2023-06-14 15:00:54 +08:00
1c0dfbb044
[Improvement][Etcd] Support SSL In Etcd And Enhance Etcd In Helm ( #13924 )
2023-06-14 13:29:50 +08:00
465e7ae6ee
[Bug] [Master] Workflow keep running when task has no id ( #14315 )
...
* change completeTaskMap to completeTaskSet
* Update dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
* change method name
* fix ut
* fix spotless
---------
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
2023-06-14 12:53:16 +08:00
a55612e4f7
Fix switch task use boolean type error ( #14326 )
2023-06-13 13:26:05 +08:00
75f4fa2881
Fix the complement function does not generate the next running instances in parallel mode ( #14335 )
2023-06-12 19:40:26 +08:00
aa2d9a7f05
Add baseDao ( #14316 )
2023-06-09 19:02:50 +08:00
e7cdc7c136
[Feature][Task Plugin] Add Dynamic task to generate dynamic worklfows with list parameters ( #14127 )
...
* add dynamic task plugin
* fix ui
* update code
* add DOC
* ADD UT
* fix doc
* fix sanity check
* add upgrade ddl
* fix sql
2023-06-07 16:01:06 +08:00
d587b5f03d
[Bug][Master][Fixes 14216]workflow state keep READY_STOP when sub_process state … ( #14221 )
...
* [Bug] [Master] workflow state keep READY_STOP when sub_process state STOP
Fixes #14216
* [Bug] [Master] workflow state keep READY_STOP when sub_process state STOP
Fixes mvn spotless:apply#14216
* Update testFile.txt.crc
* Update testFile.txt.crc
* Delete testFile.txt.crc
2023-06-06 11:04:32 +08:00
5e9f1de075
Remove state check for dependent/subProcess in StateWheelExecuteThread ( #14242 )
2023-05-31 16:39:12 +08:00
d810c487fd
Fix taskGroupQueue doesn't removed from inQueue where wakeup failed ( #14200 )
...
* Fix taskGroupQueue doesn't removed from inQueue where wakeup failed
* Fix avaliable taskGroup is 0 the TaskGroupQueue doesn't out queue
2023-05-25 14:42:32 +08:00
deca0a7ea1
Fix master memory leak due to MasterTaskExecuteRunnableDoesn't be removed ( #14162 )
...
* Fix master memory leak due to MasterTaskExecuteRunnableDoesn't be removed
* Update dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/execute/MasterTaskExecuteRunnable.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2023-05-24 13:49:32 +08:00
05df0269f6
Fix task exception might not exist in task instance log ( #14085 )
2023-05-21 09:55:35 +08:00
8315279620
[Feature-#14148][Task] Added status check interval and dependent failure policy parameters for dependent task nodes ( #14150 )
...
* Added a dependent task failure policy in the dependent task node
2023-05-19 14:49:50 +08:00
c6512edc3a
revert [fix-#13956] [Master]taskId is null cause NPE #13980 ( #14142 )
2023-05-18 19:28:27 +08:00
503d54aff9
Fix dependent task nodes in workflow instances cannot display data ( #14141 )
2023-05-18 19:24:53 +08:00
cf91544b99
Refactor logic task ( #13948 )
2023-04-26 10:01:51 +08:00
d617620b00
[improve-#13956] taskId is null ( #13980 )
...
Co-authored-by: fuchanghai <‘2875334588@qq.com’>
2023-04-21 15:49:43 +08:00
1c836bf011
[Fix-13950][master] fix the key of master node failover ( #13952 )
...
Co-authored-by: eye <eye.gu@aloudata.com>
2023-04-19 17:52:38 +08:00
f78ef9ec08
[Fix-13815][master] Fix dependent task incorrect timeout alerts ( #13822 )
2023-04-19 16:14:13 +08:00
7835ddfdb7
[Fix-13780][api] Fix the problem of process instance DAG can't show dependent task ( #13823 )
2023-04-19 15:40:51 +08:00
45586f71d5
[BUG][Resource Center] Task can not select main JAR ( #13536 )
2023-04-18 18:38:16 +08:00
79cb707b57
[Improvement-13932][API] Tenant parameters and Worker group parameters are put together in the task startup parameters ( #13935 )
...
* [Improvement][API] Tenant parameters and Worker group parameters are put together in the task startup parameters
2023-04-18 14:56:38 +08:00
61a689aa5a
Use percentage to represent memory/cpu usage ( #13896 )
2023-04-13 15:48:20 +08:00
7ee66f2d02
[Feature-13429][Remote Logging] Add support for writing task logs to Google Cloud Storage ( #13777 )
2023-04-06 17:49:33 +08:00
41a8ba9aab
Support alert server HA ( #13865 )
...
* Support alert server cluster
* Remove WORKER_ALERT_LISTEN_HOST in statefulset-dolphinscheduler-worker.yaml
2023-04-05 17:31:50 +08:00
aec1366732
[Improvement]Update spelling mistake for the name of WorkflowSubmitStatue ( #13850 )
...
* Correct the spelling of an enum
2023-04-03 11:10:57 +08:00
9b269b3c53
Use common response for sync request ( #13847 )
2023-04-01 10:45:21 +08:00
d64004abea
Update WorkflowExecuteRunnable.java ( #13155 )
2023-03-30 15:12:40 +08:00
9df89cf0cf
Use bootstrap user as the default tenant ( #13825 )
2023-03-30 14:11:23 +08:00
fba59981d4
[Fix-13657][Master]NPE caused by the execution of workflow with startNode and forbidden task ( #13668 )
...
Co-authored-by: HomminLee <lihongmin@ysstech.com>
2023-03-30 09:56:57 +08:00
9d9ae9ad54
[improvement] support self-dependent ( #13818 )
2023-03-29 18:18:03 +08:00
68660ec96b
Refactor remote command ( #13809 )
...
* Refactor remote command
* Rename Command to Message
2023-03-29 17:54:57 +08:00
d1b6e6f02c
Take over task instance in master failover ( #13798 )
2023-03-28 16:02:57 +08:00
d91bdeff37
Fix retry task instance will loss varpool ( #13791 )
2023-03-25 12:26:59 +08:00
1f365819a6
Ignore unknown VM options in start.sh ( #13719 )
2023-03-10 17:34:09 +08:00
394805b2c7
[Feature][Metrics] Tag workflow related metrics with process definition code (workflow id) ( #13640 )
...
* Tag workflow related metrics with process definition code (workflow id)
* Clean up related metrics when deleting workflow definition
* Add license headers
* Update related UT cases
* Add an example in grafana-demo
* Add related docs
2023-03-09 11:30:21 +08:00
c9066e8de9
Use MDC to filter task instance log ( #13673 )
...
* Use MDC to collect task instance log
* Use MDCAutoClosableContext to remove the MDC key
2023-03-06 17:44:23 +08:00
7bf3e3cdd6
[improve-#13201] update pid during running ( #13206 )
2023-02-22 14:22:49 +08:00
047fa2f65e
[Feature-13511] Submit Spark task directly on Kubernetes ( #13550 )
2023-02-21 23:26:21 +08:00
701d67c831
[improve-#13045] after a submit failure, stop the processInstance to avoid an endless loop ( #13051 )
...
* [improve-#13045] add max submit number of workflow
* [fix-13405] remove max times
* Update dolphinscheduler-master/src/main/resources/application.yaml
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
---------
Co-authored-by: fuchanghai <‘2875334588@qq.com’>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
2023-02-20 14:10:59 +08:00
e8b20def54
Fix task instance will generate multiple times when retry interval is 0/s ( #13571 )
2023-02-18 23:41:16 +08:00
2bd65fb2df
[Feature][Remote Logging] Add support for writing task logs to OSS ( #13332 )
2023-02-13 16:38:26 +08:00
8d12dc0702
[Improvement-13491] Use lombok @Slf4j annotation to generate logger ( #13509 )
2023-02-07 20:32:53 +08:00
12dd60fa46
Fix task wake up failed will block the event ( #13466 )
2023-02-01 15:31:04 +08:00
385d781ebc
Fix minor spelling and punctuation errors ( #13452 )
2023-01-29 16:51:03 +08:00
ce34e21960
Task instance failure when worker group doesn't exist ( #13448 )
2023-01-29 16:27:56 +08:00
8be32d4145
[Feature][Api] When use api to run a process we want get processInstanceId ( #13184 )
...
* add sql
* add mapper
* add dao
* add excutor
Co-authored-by: qianl4 <qianl4@cicso.com>
2023-01-18 17:58:32 +08:00
3b980cb06a
[fix][worker][bug] master/worker crash when registry recover from SUSPENDED to RECONNECTED ( #13328 )
2023-01-03 19:24:11 +08:00
ccad56e88e
[Improvement][Master] Validate same content of input file when using task cache ( #13298 )
...
* support file content checksum
* fix inject null storageOperate bug
2023-01-03 11:38:13 +08:00
8a479927f3
Add projectCode in t_ds_process_instance and t_ds_task_instance to remove join ( #13284 )
2023-01-03 09:52:28 +08:00
52134277a3
Fix task group cannot release when kill task ( #13314 )
2023-01-03 09:52:03 +08:00
2e95a020ab
fix dag.getPreviousNodes miss upstream node ( #13255 )
2022-12-22 15:51:40 +08:00
14ec4a2398
Remove dao module in worker ( #13242 )
2022-12-22 12:25:29 +08:00
5fe25c995f
Fix spelling ( #13237 )
2022-12-21 16:22:49 +08:00
ll