ce34e21960
Task instance failure when worker group doesn't exist ( #13448 )
2023-01-29 16:27:56 +08:00
8be32d4145
[Feature][Api] When use api to run a process we want get processInstanceId ( #13184 )
...
* add sql
* add mapper
* add dao
* add excutor
Co-authored-by: qianl4 <qianl4@cicso.com>
2023-01-18 17:58:32 +08:00
3b980cb06a
[fix][worker][bug] master/worker crash when registry recover from SUSPENDED to RECONNECTED ( #13328 )
2023-01-03 19:24:11 +08:00
ccad56e88e
[Improvement][Master] Validate same content of input file when using task cache ( #13298 )
...
* support file content checksum
* fix inject null storageOperate bug
2023-01-03 11:38:13 +08:00
8a479927f3
Add projectCode in t_ds_process_instance and t_ds_task_instance to remove join ( #13284 )
2023-01-03 09:52:28 +08:00
52134277a3
Fix task group cannot release when kill task ( #13314 )
2023-01-03 09:52:03 +08:00
2e95a020ab
fix dag.getPreviousNodes miss upstream node ( #13255 )
2022-12-22 15:51:40 +08:00
14ec4a2398
Remove dao module in worker ( #13242 )
2022-12-22 12:25:29 +08:00
5fe25c995f
Fix spelling ( #13237 )
2022-12-21 16:22:49 +08:00
d13cd55281
fix spell error and move comment to correct describe location ( #13233 )
2022-12-21 16:22:03 +08:00
66e20271ad
[Feature][Master] Add task caching mechanism to improve the running speed of repetitive tasks ( #13194 )
...
* Supports task instance cache operation
* add task plugin cache
* use SHA-256 to generate key
* Update dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
* Update dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_postgresql.sql
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
* Optimizing database Scripts
* Optimize clear cache operation
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
2022-12-18 18:17:09 +08:00
7a0a2c2a46
Solve the deadlock problem caused by queuing ( #13191 )
...
* Solve the deadlock problem caused by queuing
* Solve the deadlock problem caused by queuing
* Solve the deadlock problem caused by queuing
* Solve the deadlock problem caused by queuing,move the event to the tail by throwing a exception
Co-authored-by: wfs <wangfushun@cdqcp.cpm>
2022-12-16 19:55:02 +08:00
be81b222d4
Optimize event loop ( #13193 )
2022-12-15 14:34:52 +08:00
e4b9b67255
Allow execute task in workflow instance ( #13103 )
2022-12-13 16:43:44 +08:00
70ccffeee2
Format task parameter as pretty json ( #13173 )
2022-12-13 16:30:21 +08:00
6ef74073cc
[Improve-13001]migrate commons-collections -> commons-collections4 ( #13002 )
2022-12-10 23:50:19 +08:00
a7ecc5a8b3
fix retry task failure ( #13077 )
2022-12-05 21:22:51 +08:00
169168ef34
Add plugin-all module ( #13079 )
2022-12-02 23:19:08 +08:00
12a6138d33
fix issue 13035 ( #13065 )
...
Co-authored-by: shenk-b <shenk-b@glodon.com>
2022-12-01 13:58:50 +08:00
ffc9fb280a
Add gc timestampt ( #13059 )
2022-12-01 10:00:27 +08:00
1a8811cb41
Set max loop times when consume StateEvent to avoid dead loop influence the thread. ( #13007 )
2022-11-27 15:34:26 +08:00
3106054ea7
[Improvement-12907] Change heartbeat log level to debug ( #12980 )
2022-11-25 17:37:30 +08:00
50779ea1e6
[Bug-12963] [Master] Fix dependent task node null pointer exception ( #12965 )
...
* Fix that there are both manual and scheduled workflow instances in dependent nodes, and one of them will report a null pointer exception during execution.
2022-11-24 19:00:46 +08:00
38b876733c
[Feature-10498] Mask the password in the log of sqoop task ( #11589 )
2022-11-24 14:54:54 +08:00
3747029cc0
[fix-#12932] when subprocess's processInstance is fail,not notify parent processInstance ( #12933 )
2022-11-22 15:07:50 +08:00
27c37b8828
Fix grammar and spelling ( #12937 )
2022-11-18 23:03:34 +08:00
c916c60853
fix NPE while retry task ( #12903 )
2022-11-16 10:31:52 +08:00
d99ba29b66
Fix master cluster may loop command unbalanced ( #12891 )
...
(cherry picked from commit 3b2b86661be76b7c1404a910c865d78b7936313d)
2022-11-16 10:20:22 +08:00
229c554912
[feature][task] Add Kubeflow task plugin for MLOps scenario ( #12843 )
2022-11-11 16:08:38 +08:00
7cdb926a5f
[Improvement][Batch Query] Batch query ProcessDefinitions belongs to need failover ProcessInstance. ( #12506 )
2022-11-03 09:15:19 +08:00
9e0c9af1a5
Fix the waiting strategy cannot recovery if the serverstate is already in running ( #12651 )
2022-11-02 14:06:01 +08:00
08335b1032
[Improvement][Task] Improved way to collect yarn job's appIds ( #12197 )
...
* Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code.
* Add user property `appId.collect` for user to decide how to collect applicationId.
* Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh`
* Update docs to declare how to use aop way.
* Update `LogUtils` to support fetch applicationId in different ways based on the user property.
Co-authored-by: gabrywu <gabrywu@apache.com>
2022-10-31 16:52:53 +08:00
e6da1ccf81
Add worker-group-refresh-interval in master config ( #12601 )
...
* Add worker-group-refresh-interval in master config
* Set interval cannot smaller than 10s
* Update dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/config/MasterConfig.java
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2022-10-31 09:37:26 +08:00
065d5caccc
Only expose necessary actuator endpoints ( #12571 )
2022-10-28 07:40:32 +08:00
489e7fe4e2
[Feature-10495][Resource Center] Resource Center Refactor ( #12076 )
...
* resource center refactor - S3 services connection
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-10-26 13:53:44 +08:00
04aa125ba2
[Refactor]migrate some task dao functions from ProcessServiceImpl to dao modules. ( #12505 )
...
* migrate some task DAO to dao modules.
* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/repository/impl/ProcessInstanceMapDaoImpl.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-10-26 11:51:58 +08:00
4aa9d60683
[Bug][Master] Add the aws-java-sdk-s3 jar package to the master module ( #12259 ) ( #12512 )
...
Co-authored-by: qiquan.li <qiquan.li@linkflowtech.com>
2022-10-25 14:53:18 +08:00
dedff70f90
[Improvement] Merge spi.utils into common.utils ( #12502 )
...
* improve JSONUtil
* merge spi.utils into common.utils
* split constants && remove some copy utils method
* remove StringUtils
2022-10-23 21:23:54 +08:00
b936b882bb
[Refactor] Migrate all command-related interface functions from ProcessServiceImpl ( #12474 )
...
* migrate all command-related interface functions to CommonService
2022-10-22 12:39:25 +08:00
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports ( #12412 )
...
* Fix remaining style errors and wildcard imports
* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link
* Fix spotless matching scope of markdown files
2022-10-19 11:08:04 +08:00
38b643f69b
make sure all failed task will save in errorTaskMap ( #12424 )
2022-10-19 09:36:47 +08:00
4ea7eec750
Use temurin Java docker images instead of deprecated ones ( #12334 )
2022-10-17 15:46:21 +08:00
bc80f34056
[Migrate][Test] Migrate all UT cases from jUnit 4 to 5 in master, worker, and remote modules ( #12362 )
...
* migrate junit4 -> junit5 in master, worker, and remote modules
2022-10-15 21:58:31 +08:00
2f37da0dbc
[Improvement][Test] Block the usage of powermock and move mockito dependencies from sub-modules to root pom ( #12311 )
...
* move mockito dependencies from sub-modules to root pom
* Add check in CI to block the usage of powermock
2022-10-14 12:06:51 +08:00
db1d319159
Reorganize some classes in common module, remove duplicate classes ( #12321 )
2022-10-13 10:23:27 +08:00
05099bd8f7
[Improvement-12296][Test] Close static mock ( #12300 )
2022-10-11 16:13:05 +08:00
165b9a58de
[DS-12131][master] Optimize the log printing of the master module acc… ( #12152 )
...
* [DS-12131][master] Optimize the log printing of the master module according to the log specification.
2022-10-09 13:56:50 +08:00
c87f6d89a4
[fix#12195] Change date to timestamp to support cross diff time zones ( #12239 )
...
* fix DS support cross time zone, use timestamp replace date
2022-10-01 12:50:42 +08:00
55388be21f
[Improvement][Master] LowerWeightRoundRobin doSelect may be null ( #12159 )
2022-09-30 20:00:23 +08:00
40090d459e
[Improvement] remove log-server and server module ( #12206 )
...
* remove log-server module
* remove dolphinscheduler-server module
2022-09-30 16:41:41 +08:00
aacabc6ec7
Remove workerGroup in registry ( #12217 )
2022-09-30 13:58:53 +08:00
e27c79974d
[fix-#11753] send alert error alert data id ( #11774 )
...
* [fix-#11753] send alert error alert data id
Co-authored-by: fuchanghai <changhai.fu@marketingforce.com>
2022-09-29 15:34:40 +08:00
d526dd89f6
[hotfix][UI] Remove SURVEIL and NEXT_LOOP in ui ( #12160 )
...
* Remove SURVEIL and NEXT_LOOP in ui
* Fix UT
2022-09-28 09:17:12 +08:00
cc82206c33
Remove powermock in dolphinscheduler-master module ( #12143 )
...
* Remove powermock related code in dolphinscheduler-master module
* Remove redundant comments, useless code and unnecessary stubbings
2022-09-26 17:02:14 +08:00
cf5a8894e2
[BUG][BE-12113]fix htrace dependency conflicts ( #12126 )
...
* fix htrace dependency conflicts
* replace htrace-core with hbase-noop-htrace
2022-09-26 13:49:08 +08:00
ada7cf71d5
[fix#12000]Cannot remove the WorkerGroup from the master service ( #12050 )
...
* [Bug] [Master] Cannot remove the WorkerGroup from the master service. #12000
* remove unnecessary locks
* Update dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/registry/ServerNodeManager.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-09-24 18:57:20 +08:00
5b384f3fab
[Feature-11473][Task]Support test task ( #11670 )
...
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* Update TaskExecuteThread.java
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update DataSource.java
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update messages_zh_CN.properties
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update messages.properties
* Update messages_en_US.properties
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
2022-09-17 13:40:56 +08:00
e938fdbe96
[Bug] [Master] WorkflowExecuteRunnable will face a infinite loop #11838 ( #11864 )
...
* [Bug] [Master] WorkflowExecuteRunnable will face a infinite loop #11838
2022-09-15 09:46:30 +08:00
6868876a29
fix workflow keep running when task fail ( #11930 )
2022-09-15 09:06:10 +08:00
d3a77c68e6
Make LogServiceClient Singleton ( #11777 )
2022-09-14 16:18:08 +08:00
009cb68011
[Improvement-11834] Upgrade docker base image to support python3.9 ( #11835 )
...
* use openjdk:8-jre-slim-bullseye instead of buster to support python3.9
* use openjdk:8-jre-slim-bullseye instead of buster across all components
2022-09-14 09:29:45 +08:00
3ca9680b20
[Fix][Monitor]Monitor UI not show DisakAvailable and MemoryUsage correctly ( #11870 )
2022-09-11 19:11:03 +08:00
105773f54d
Remove htrace dependencies ( #11881 )
...
* Remove htrace dependencies
* Update pom.xml
2022-09-10 22:54:38 +08:00
3664d85143
[Improve] Upgrade Hadoop to 3.2.4 ( #11832 )
2022-09-09 10:47:03 +08:00
2e61c76c22
[Improvement] Add remote task model ( #11767 )
2022-09-08 13:41:36 +08:00
efedc9a53c
Bump up dependencies to fix cves ( #11765 )
2022-09-08 09:05:05 +08:00
444c7e6eb7
[Improvement] Replace commons-lang 2 function invocations with commons-lang3 ( #11810 )
...
* [Improvement] Replace commons-lang 2 function invocations with commons-lang3
2022-09-07 13:48:09 +08:00
f8d46a26c1
[Feature-11530] add state history for process instance ( #11757 )
...
* [Feature] add state history for process instance (#97 )
* add state history for process instance
* upsertProcessInstance
* remove unuse method
* fix UT
Co-authored-by: caishunfeng <534328519@qq.com>
2022-09-03 10:14:05 +08:00
277f137358
Add Kubernetes configmap reload to all components ( #11730 )
2022-09-02 12:03:19 +08:00
f95cdd350a
reduce task dependencies ( #11731 )
2022-09-01 17:04:07 +08:00
67e7f88d8b
Refactor heart beat task, use json to serialize/deserialize ( #11702 )
...
* Refactor heart beat task, use json to serialize/deserialize
2022-08-31 16:20:23 +08:00
b96d69701a
[Improvement-#11613] Add spi priority factory ( #11614 )
...
* Add spi priority factory
* Add doc
* Add override log
* Use lombok
* Add comment
2022-08-24 19:51:47 +08:00
3f2ca7bca3
[Fix-9980] [Server] fix heartBeatTaskCount bug ( #11232 )
...
* fix heartBeat bug
* modify class name
* fix conflict
Co-authored-by: JinyLeeChina <jiny.li@foxmail.com>
2022-08-23 11:30:13 +08:00
abfef1a929
[improvement] Add two parameters in workergroup, and support the application of description display and other parameters ( #11542 )
...
* [Improvement] Improvement default worker group
* [Improvement] Improvement default worker group
* [Improvement] Improvement default worker group
* [Improvement] Improvement default worker group
* [Improvement] Improvement default worker group
* [improvement] The default worker grouping supports editing
* [improvement] The default worker grouping supports editing
* Update WorkerGroupServiceImpl.java
* Update WorkerGroupServiceImpl.java
* The default worker grouping supports editing
* update sql file
* update
* update
* update
* update
2022-08-23 10:54:17 +08:00
4d13a5104b
Fix kill task failed will cause the taskGroup cannot release and add taskGroup log ( #11469 )
2022-08-23 10:05:03 +08:00
a41c6824fc
Add dependent task instance log ( #11541 )
...
* Add dependent task instance log
* Optimize log
* Fix dependent task initialize failed will throw exception
2022-08-23 09:55:38 +08:00
8886d5a126
Fix quartz threadPriority config name error ( #11596 )
...
* Fix quartz threadPriority config name error
* Add batchTriggerAcquisitionMaxCount config
2022-08-23 09:32:05 +08:00
3516533017
Remove logger header in task log file ( #11555 )
2022-08-19 14:01:52 +08:00
f6ef628030
Support reading application properties from kubernetes config map ( #11402 )
2022-08-16 20:25:17 +08:00
4362fb8448
If the task processor is not found need to throw error rather than exception ( #11461 )
2022-08-13 22:01:39 +08:00
05589606a2
Set master's task running status in `runTask` to avoid the task group acquire failed, but the task status is in running ( #11451 )
2022-08-13 14:19:13 +08:00
7ff34c3947
[Feature-7024] Add waiting strategy to support master/worker can recover from registry lost ( #11368 )
...
* Add waiting strategy to support master/worker can recover from registry lost
* throw exception when zookeeper registry start failed due to interrupted
2022-08-13 09:52:03 +08:00
0ca3086296
Fix workflow instance may failover many times due to doesn't check the restart time ( #11445 )
2022-08-13 00:05:37 +08:00
a2b5219315
Fix find last process instance in dependent task is not correct ( #11424 )
2022-08-12 14:11:28 +08:00
47b50067ef
Clear thread name in BaseTaskProcessor ( #11422 )
2022-08-12 09:38:48 +08:00
496c2d4bfa
[Fix-11413] Cannot set task status to kill if the task is not in running( #11414 )
2022-08-11 13:38:49 +08:00
5fa3e7b1ed
Add pause to task instance status ( #11390 )
2022-08-11 09:25:53 +08:00
0464123c2b
[Feature-11223] support stream task ( #11350 )
...
* add task execute type
* update task definition list paging
* update task instance list paging
* stream task start
* [Feature][UI] Some changes to execute task.
* Set the connection edge to dashed line.
* Add FLINK_STREAM task.
* add stream task
* flink savepoint and cancel
* fix query bug
* add stream task definition
* add task instance for stream task
* delete stream task definition state
* update api for stream task definition edit
* modify search for stream task instance
* add language
* delete task type search for stream task definition
* change task type search for stream task instance
* add jump button
* add savepoint
* add down log for stream task instance
* ui test
* stream task start
* run DAG
* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.
* [Feature][UI] Remove some fields for FLINK_STREAM.
* add start modal
* add dryRun column for stream task instance
* fix duration
* fix pon
* fix build error
* Add success tip
* add auto sync for stream task instance
* remove forgien key for task instance
* license header
* UT fix
* modify locales
* recover common config
* fix UT
* add doc
Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
2022-08-10 21:44:43 +08:00
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus ( #11340 )
2022-08-10 11:00:23 +08:00
acd3d3fab1
Fix TaskGroup cannot work and will cause master dead loop ( #11254 )
...
* Fix TaskGroup cannot work and will cause master dead loop
* Remove acquireTaskGroupAgain in ProcessServiceImpl
2022-08-04 14:43:45 +08:00
9ca1eb96c4
[Improvement][Metrics] Add metrics for alert server ( #11240 )
...
* [Improvement][Metrics] Add metrics for alert server (#11131 )
* Update related docs of metrics
* Add grafana demo dashboards for alert server metrics
* Refactor metric classes with UtilityClass annotation
* Refactor meter names in camelCase for checkstyle
2022-08-03 15:42:06 +08:00
04f3aa9713
Fix recovery from failed task will dead loop ( #11239 )
2022-08-02 09:30:18 +08:00
8e21c38c00
Write alert result into db ( #11221 )
2022-08-01 13:04:27 +08:00
6c7e00c4aa
[Bug-11101] fix task failover NPE ( #11168 )
2022-07-27 19:21:21 +08:00
0e15ce3389
[Improvement][Metrics] Switch to use tags to indicate task / workflow execution status for metrics ( #11128 )
...
* [Improvement][Metrics] Switch to use tags to indicate task / workflow execution status for metrics (#10867 )
* Update docs and grafana demo dashboards
2022-07-26 13:44:02 +08:00
3701a24d15
[Improvement][Task Log] Task status log print description instead of code ( #11009 )
...
* use execution status instead of status code
2022-07-22 13:34:31 +08:00
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version ( #11025 )
2022-07-20 10:37:31 +08:00
e1c78d1a49
Move utils, builder under master ( #11000 )
2022-07-16 23:18:00 +08:00
083ab2b5c9
Remove dao in worker ( #10994 )
2022-07-15 20:07:18 +08:00
2be1d4bf0a
Fix worker cannot shutdown due to resource close failed or heart beat check failed ( #10979 )
...
* Use try-with-resource to close resource, and add heart error threshold to avoid worker cannot close due to heart beat check failed
* Move heartbeat error threshold to applicaiton.yml
2022-07-15 20:06:53 +08:00
998e4d74dd
Kill task when we do master failover to avoid task lost ( #10997 )
2022-07-15 18:44:42 +08:00
b245e7c973
Fix kill yarn job error when failover caused by doesn't set ProcessDefinition ( #10948 )
2022-07-13 22:32:30 +08:00
df0416c193
Fix taskInstance's host is not worker nettyServer address ( #10926 )
...
* Fix taskInstance's host is not worker nettyServer address
* Remove unnecessary mock
2022-07-13 20:46:33 +08:00
9f34a837b8
[Fix-10909] Returns executionStatus for each process instance status and access token module support. ( #10922 )
...
* Returns executionStatus for each process instance status and access token module support
2022-07-13 20:45:52 +08:00
2a67866718
Catch exception when check state in StateWheelExecuteThread ( #10908 )
...
* Catch exception when check state
2022-07-13 10:51:20 +08:00
cade66a9b6
[Fix-10827] Fix network error cause worker cannot send message to master ( #10886 )
...
* Fix network error cause worker cannot send message to master
2022-07-12 14:08:42 +08:00
aa8b88a875
[Feature-10871] add workflow executing data query ( #10875 )
...
* add workflow executing data query
* fix sonar check for interrupted
2022-07-11 16:55:55 +08:00
f639a2eed4
[Fix-10854] Fix database restart may lost task instance status ( #10866 )
...
* Fix database update error doesn't rollback the task instance status
* Fix database error may cause workflow dead with running status
2022-07-11 09:57:00 +08:00
b5653ea7f2
Complement numbers will run in a loop under the serial strategy fixed. ( #10862 )
...
* Complement numbers will run in a loop under the serial strategy fixed.
* e2e rerun
2022-07-09 16:41:51 +08:00
3f69ec8f28
[Fix-10842] Fix master/worker failover will cause status incorrect ( #10839 )
...
* Fix master failover will not update task instance status
* Add some failover log
* Fix worker failover will rerun task more than once
* Fix workflowInstance failover may rerun already success taskInstance
2022-07-09 11:54:59 +08:00
426567348e
Remove quartz in service ( #10748 )
...
* Remove quartz in service
2022-07-06 15:43:55 +08:00
67d14fb7b3
[Fix-10785] Fix state event handle error will not retry ( #10786 )
...
* Fix state event handle error will not retry
* Use state event handler to deal with the event
2022-07-06 14:53:28 +08:00
35a10d092f
[Fix-10666] Workflow submit failed will still in memory and never retry ( #10667 )
...
* Workflow submit failed will still in memory and never retry
2022-07-04 22:08:15 +08:00
8f621ff98b
[Optimization] Calculate global parameter and local parameter at master. ( #10704 )
...
* Global parameter and local parameter calculation external expansion.
* k8s task ut fix.
* TimePlaceholderUtils import DateUtils fix
* follow the review comments to fix.
* follow the review comments to fix.
* e2e rerun
2022-06-30 22:45:25 +08:00
247ca4ae8a
[Improvement-10617] Add comment in slot check ( #10618 )
2022-06-30 15:58:49 +08:00
35b25da863
Validate master/worker config ( #10649 )
2022-06-28 20:17:43 +08:00
66624c5c86
[Bug] [Master] Worker failover will cause task cannot be failover ( #10631 )
...
* fix worker failover may lose event
2022-06-28 16:08:35 +08:00
b5184138fa
[Feature] Time function analysis extension. ( #10624 )
...
* Time function analysis extension.
* param add.
* clear useless logs and update method notes
* permission omission fix.
* extending time functions to optimize static methods
* e2e rerun.
2022-06-28 11:24:51 +08:00
628c8ab2a1
[Improvement][Master] batch remove TaskInstaceId and workflowInstanceId ( #10620 )
2022-06-26 10:07:35 +08:00
397e51fa63
dependent node log output optimization and fix checkstyle ( #10593 )
2022-06-24 16:48:58 +08:00
1111371c9a
add datasource health check to the healthcheck endpoint ( #10588 )
2022-06-24 13:29:49 +08:00
0f38217b12
fix_10514 ( #10568 )
2022-06-23 16:15:08 +08:00
db595b3eff
Optimize master log, use MDC to inject workflow instance id and task instance id in log ( #10516 )
...
* Optimize master log, add workflow instance id and task instance id in log
* Use MDC to set the workflow info in log4j
* Add workflowInstanceId and taskInstanceId in MDC
2022-06-23 11:45:06 +08:00
f46faa02c3
[Fix-10452] Serial wait for policy recovery ( #10453 )
...
* Serial wait for policy recovery
* processInstance state check null
* add sendresult(Host host, Command command) method and replace the original.
2022-06-22 14:28:50 +08:00
cc06eaaf54
[Improvement][Metrics] Apply micrometer naming convention to metrics ( #10477 )
...
* Apply micrometer naming convention to worker metrics
* Apply micrometer naming convention all current metrics
* Fix remaining metrics names, update English docs and add Chinese docs
* Fix metrics names in grafana-demo dashboards
2022-06-21 14:27:06 +08:00
b100f6c489
Remove the schedule thread in LowerWeightHostManager ( #10310 )
2022-06-21 13:30:07 +08:00
117f78ec4b
[Fix-10413] Fix Master startup failure the server still hang ( #10500 )
...
* Fix Master startup failure the server still hang
2022-06-20 22:35:06 +08:00
0bdfa0cff9
Fix PeerTaskInstancePriorityQueue cannot contains method use taskInstanceId to check ( #10491 )
2022-06-17 16:14:56 +08:00
ad2646ff1f
Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe ( #10479 )
...
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
2022-06-16 21:46:18 +08:00
78c5fcc6ac
Add mysql registry plugin ( #10406 )
...
* Add mysql registry plugin
2022-06-13 11:24:42 +08:00
f6fea06f10
[Improve] Enhance complement function transformation ( #10376 )
2022-06-12 12:35:08 +08:00
b0d9d3f9ab
Add some warning log in master ( #10383 )
...
* Add some warn log in master
* fix may skip sleep
2022-06-10 19:38:20 +08:00
e21d7b1551
[Feature][metrics] Add master, worker metrics ( #10326 )
...
* Add mater metrics
* fix UT
* Add url to mysql profile
* Add worker metrics
* Update grafana config
* Add system metrics doc
* Add process failover counter
* Add metrics image
* Change jpg to png
* Add command insert metrics
* Fix UT
* Revert UT
2022-06-09 10:55:39 +08:00
3ecbee3885
Optimize MasterServer, add MasterRPCService ( #10371 )
...
* Optimize MasterServer, avoid NPE
2022-06-09 10:00:05 +08:00
b1fb17119b
[Feature-6758][Task] Add limit resource usage for tasks base on cgroup ( #10373 )
2022-06-08 19:42:05 +08:00
2d3be6b36c
Add dolphinscheduler-scheduler module ( #10360 )
...
* Add dolphinscheduler-scheduler module
2022-06-04 16:39:33 +08:00
d80cf21456
Clean up unused dependencies and packaging issues ( #9944 )
2022-05-31 15:22:41 +08:00
49979c658e
[Fix-8828] [Master] Assign tasks to worker optimization ( #9919 )
...
* fix 9584
* master recall
* fix ut
* update logger
* update delay queue
* fix ut
* remove sleep
Co-authored-by: 进勇 <lijinyong@cai-inc.com>
Co-authored-by: JinyLeeChina <jiny.li@foxmail.com>
2022-05-31 11:49:54 +08:00
aba5f8a40e
[improve] Change Mysql Driver ( #10220 )
2022-05-25 14:09:15 +08:00
ee2b855ced
[BUG][TaskGroup] Task group does not take effect ( #10093 )
...
* fix 10092: Task group does not take effect
* fix 10092: Task group does not take effect
* fix 10092: Task group does not take effect
2022-05-18 18:40:36 +08:00
5bb1eb04fc
[Feature][plugin] Add k8s task in task plugin ( #9425 )
...
* [Feature][plugin] Add k8s task in task plugin
* [Feature][plugin] fix dos and code problems
* [Feature][plugin] refactor some code based on sonar
* [Feature][UI] front-end for k8s task plugin
* [Feature][plugin] delete some front files
* [Feature][plugin] update document
* Update docs/docs/zh/guide/task/k8s.md
* Update docs/docs/en/guide/task/k8s.md
* Update docs/docs/en/guide/task/k8s.md
* Update docs/docs/en/guide/task/k8s.md
* [Feature][UI] front-end change from review comments
* [Feature][UI] replace get namespace list api
* [Feature][plugin] change file name
* Add kubernetes to zh task list
Co-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: William Tong <weitong@cisco.com>
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-05-18 17:23:42 +08:00
225cb332d1
[Fix-10049] Conditions Task branch flow failed ( #10077 )
2022-05-18 11:40:32 +08:00
0cc0ee77fa
[Bug][Master] fix master task failover ( #10065 )
...
* fix master task failover
* ui
2022-05-17 16:36:24 +08:00
dbdbfeaeee
[Fix-9975] The selected task instance was recreated when the Master service fail… ( #9976 )
...
* The selected task instance was recreated when the Master service failed over.
* Returns the expression result directly.
* Use Recovery to determine whether to use the old task instance.
2022-05-12 16:31:53 +08:00
d4aeee16e5
[Bug] [MASTER-9811]fix cmd param to overwrite global param when executing complement ( #9952 )
...
* fix cmd param to overwrite global param when executing complement
* fix cmd param to overwrite global param when executing complement
2022-05-11 18:37:03 +08:00
8562f6a878
[Feature][Log]Add timezone information in log output ( #9913 )
2022-05-06 17:31:44 +08:00
a1b6b033ad
[Fix-9906] After the serial wait execution strategy stops the running workflow instance, the instance will be woken up and executed if there is a wait instance. ( #9907 )
...
* After the serial wait execution strategy stops the running workflow instance, the instance will be woken up and executed if there is a wait instance.
* clear logic
* Resource overloading
2022-05-06 15:01:21 +08:00
fb0f96ed94
[Fix-9868] A task flow definition isolates the runs of different execution strategies by version numbers. ( #9869 )
...
* The thread cache task flow definition should get the latest version.
* Coverage on New Code
* Coverage on New Code
* Coverage on New Code
* use an existing method.
* Increase unit test coverage.
* Task flow definitions enforce policy isolation.
2022-05-03 22:52:42 +08:00
31cd1b5e61
Serial wait for subsequent execution ( #9847 )
2022-04-29 15:44:05 +08:00
3cea039239
Task queue status update. ( #9832 )
2022-04-28 16:41:00 +08:00
5c0be8a3d7
A task instance that normally queries the serial wait state. ( #9777 )
...
Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-26 15:47:01 +08:00
897d7cb555
Add the host address of the execution server to the sub task task instance. ( #9758 )
...
Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-25 18:17:48 +08:00
de50f43de6
[common] Make dolphinscheduler_env.sh work when start server ( #9726 )
...
* [common] Make dolphinscheduler_env.sh work
* Change dist tarball `dolphinscheduler_env.sh` location
from `bin/` to `conf/`, which users could finish their
change configuration operation in one single directory.
and we only need to add `$DOLPHINSCHEDULER_HOME/conf`
when we start our sever instead of adding both
`$DOLPHINSCHEDULER_HOME/conf` and `$DOLPHINSCHEDULER_HOME/bin`
* Change the `start.sh`'s path of `dolphinscheduler_env.sh`
* Change the setting order of `dolphinscheduler_env.sh`
* `bin/env/dolphinscheduler_env.sh` will overwrite the `<server>/conf/dolphinscheduler_env.sh`
when start the server using `bin/dolphinsceduler_daemon.sh` or `bin/install.sh`
* Change the related docs
2022-04-25 15:35:43 +08:00
7bcec7115a
[Fix-9717] The failure policy of the task flow takes effect ( #9718 )
...
* Failure policy takes effect.
* Coverage on New Code
* correct description logic
* Compatible with all scenarios
* clearer logic
Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
2022-04-25 15:29:18 +08:00
5657cb9aec
[Bug-9719][Master] fix failover fail because task plugins has not been loaded ( #9720 )
2022-04-24 20:34:21 +08:00
ebc4253d50
[fix][Service] BusinessTime should format with schedule timezone ( #9714 )
...
* BusinessTime should format with schedule timezone
* fix test error
* fix test error
* fix test error
2022-04-24 19:21:21 +08:00
88d2803fe1
fix task dispatch error overload resource pool of task group ( #9667 )
2022-04-22 18:39:40 +08:00
63638601b0
fix process pause and rerun ( #9568 )
2022-04-19 10:23:56 +08:00
508ed9769a
[Fix][Master Server] handle warn+failed timeout strategy in workflow execute thread of master server ( #8077 ) ( #9485 )
...
Co-authored-by: xudong.zhang <xudong.zhang@nio.com>
2022-04-18 20:34:22 +08:00
3815a86a3b
[Improvement][Master] Fix typo for MasterTaskExecThreadTest ( #9513 )
2022-04-15 11:32:22 +08:00
66d148872d
[Bug-9501][Worker] fix kill task error before running ( #9509 )
2022-04-14 21:34:02 +08:00
706cdb6a8e
[Fix-9174] [Alert] Fix deduplication of alarm information ( #9371 )
...
* feat(issue #9174 ):
Fix-9174
2022-04-14 10:30:25 +08:00
69923546a1
[DS-9263][Improvement][master]optimize failover ( #9281 )
...
- add FailoverService.java
- move failover method from MasterRegistryClient to FailoverService
- move failover code from FailoverExecuteThread to FailoverService
This closes #9263
2022-04-12 11:53:18 +08:00
b285ccf930
[Future-9396]Support output parameters transfer from parent workflow to child work flow ( #9410 )
...
* [Future-9396]Support output parameters transfer from parent workflow to child work flow
* fix note
2022-04-11 20:03:16 +08:00
dce3c132ca
[DS-9387][refactor]Remove the lock in the start method of the MasterRegistryClient class ( #9389 )
2022-04-08 21:06:28 +08:00
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT ( #9223 ) ( #9299 )
2022-04-07 18:15:19 +08:00
40e44f6bc4
fix repeated submit task ( #9323 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2022-04-02 14:06:35 +08:00
3e5781b6e1
[Fix-9065] [master] when task submit failed, remove from queue and add state event to thread ( #9186 )
...
* [Fix-9065] [master] when task submit failed, remove from queue and add state event to thread
This closes #9065
* Update dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteThread.java
yes, your idea is better
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
* [Bug-9065] [master] update logger info
Co-authored-by: guoshupei <guoshupei@lixiang.com>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-04-01 22:23:35 +08:00
1073fcae44
[Bug-9295][Master] fix repeated submit task ( #9304 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-31 21:10:09 +08:00
52ba2c6475
[Improvement-9288][Master] add task event thread pool ( #9293 )
...
* add task event thread
* license heander
* ci test
* delete unuse file
* fix CI test
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-31 14:13:33 +08:00
ddf1ff98fa
[Improvement-7697][Master/Worker] Change the task ack to runnning callback ( #8719 )
...
* rebase dev
* change task state to dispatch if dispatch success
* update task host when dispatch
* add dispatch task event
* test
* check tenant after enable auto create
* handle dispatch state
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-30 17:35:05 +08:00
b4b5241797
[DS-9266][feat][server]Implement show dolphinscheduler banner at launch ( #9273 )
...
- add banner.txt
- modify application.yaml
This closes #9266
2022-03-30 16:45:24 +08:00
fd76436d87
[Feature-#8318] [master-server] Sub process transfer parameters to father process ( #8522 )
...
* load param from subprocess
* unit test
* modify code style
Co-authored-by: wangxj <wangxj31>
2022-03-30 14:55:59 +08:00
df519f8044
remove common task log ( #9274 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-30 11:42:47 +08:00
e2c1cc0579
[Fix-9222] [master] Support for deploying multiple masters on one node ( #9240 )
...
This closes #9222
Co-authored-by: guoshupei <guoshupei@lixiang.com>
2022-03-29 22:46:02 +08:00
85e56b5e2a
fix 9243 ( #9244 )
2022-03-28 20:45:01 +08:00
d3251c9bcc
[Improvement-9227][master]implement use the slot to scan the database ( #9228 )
...
when the master assigns tasks by slot,implement use the slot to scan the database.
This closes #9227
2022-03-28 09:52:12 +08:00
263791a63e
[Bug-9137][Server] fix timezone init ( #9219 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-28 09:38:51 +08:00
258285e6bb
[Fix-8980] [master] fixed repeat processing command ( #9220 )
...
This closes #8980
Co-authored-by: guoshupei <guoshupei@lixiang.com>
2022-03-27 23:33:30 +08:00
d7d756e7b0
Fix DataX and Sqoop Task execute NullPointerException ( #9192 )
2022-03-26 20:15:41 +08:00
0e3cafec1d
[Feature-8612][RESOURCE] extend s3 to the storage of ds ( #8637 )
...
* feat(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config
* fix(resource manager): update the dependency
* fix(resource manager): extend s3 to the storage of ds
fix the constant of hadooputils
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager):
merge the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl
* fix(resource manager): extend s3 to the storage of ds
merge the resourceServiceTest
* fix(resource manager): test cancel the test method
createTenant verifyTenant
* fix(resource manager): merge the code follow the check-result of sonar
* fix(resource manager): extend s3 to the storage of ds
fit the spell question
* fix(resource manager): extend s3 to the storage of ds
revert the common.properties
* fix(resource manager): extend s3 to the storage of ds
update the storageConfig with None
* fix(resource manager): extend s3 to the storage of ds
fix the judge of resourceType
* fix(resource manager): extend s3 to the storage of ds
undo the compile-mysql
* fix(resource manager): extend s3 to the storage of ds
delete hadoop aws
* fix(resource manager): extend s3 to the storage of ds
update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties
* fix(resource manager): extend s3 to the storage of ds
update the aws-region
* fix(resource manager): extend s3 to the storage of ds
fix the storageconfig init
* fix(resource manager): update e2e docker-compose
update e2e docker-compose
* fix(resource manager): extend s3 to the storage of ds
revent the e2e common.proprites
print the resource type in propertyUtil
* fix(resource manager): extend s3 to the storage of ds
1.println the properties
* fix(resource manager): println the s3 info
* fix(resource manager): extend s3 to the storage of ds
delete the info and upgrade the s3 info to e2e
* fix(resource manager): extend s3 to the storage of ds
add the bucket init
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager): extend s3 to the storage of ds
upgrade the s3 endpoint
* fix(resource manager): withPathStyleAccessEnabled(true)
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager): upgrade the s3client builder
* fix(resource manager): correct the s3 point to s3client
* fix(resource manager): update the constant BUCKET_NAME
* fix(resource manager): e2e s3 endpoint -> s3:9000
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* style(resource manager): add info to createBucket
* style(resource manager): debug the log
* ci(resource manager): test
test s3
* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql
* fix(resource manager): update the h2 sql
* fix(resource manager): solve to delete the tenant
* style(resource manager): merge the style end delete the unuse s3 config
* fix(resource manager): extend s3 to the storage of ds
UPDATE the rename resources when s3
* fix(resource manager): extend s3 to the storage of ds
1.fix the code style of QuartzImpl
* fix(resource manager): extend s3 to the storage of ds
1.impoort restore_type to CommonUtils
* fix(resource manager): update the work thread
* fix(resource manager): update the baseTaskProcessor
* fix(resource manager): upgrade dolphinscheduler-standalone-server.xml
* fix(resource manager): add user Info to dolphinscheduler_h2.sql
* fix(resource manager): merge the resourceType to NONE
* style(upgrade the log level to info):
* fix(resource manager): sysnc the h2.sql
* fix(resource manager): update the merge the user tenant
* fix(resource manager): merge the resourcesServiceImpl
* fix(resource manager):
when the storage is s3 ,that the directory can't be renamed
* fix(resource manager): in s3 ,the directory cannot be renamed
* fix(resource manager): delete the deleteRenameDirectory in E2E
* fix(resource manager): check the style and recoverd the test
* fix(resource manager): delete the log.print(LoginUser)
2022-03-25 13:28:13 +08:00
852597c9bf
[Bug-9070][Server]fix taskExecutionContext json convert error ( #9072 )
...
* [Bug-9070][Server]fix taskExecutionContext json convert error
* test
* add license header
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-22 14:02:07 +08:00
3114c91748
[Bug-8810] [master] add try catch to onSuccess method in the WorkflowExecuteThreadPool ( #8896 )
...
This closes #8810
Co-authored-by: guoshupei <guoshupei@lixiang.com>
2022-03-15 16:34:19 +08:00
a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module ( #8584 )
...
* Task plug-in optimization, task-related classes are migrated to the task plugin module
* remove tasktype enum class.
* fix code style.
* fix code style
* fix code style
* fix ProcessServiceTest.testUpdateResourceInfo test class
* fix code style
* fix test task plugin manager check parameters null
* fix test ut
* update aws license
* change TaskTimeoutStrategy class
2022-03-15 15:16:36 +08:00
c4fc59c9bb
[Fix][Master] Fix master-server execution logic thread unsafe problem ( #8707 )
...
* Fix thread safety issue in master service
* optimize code.
2022-03-14 16:22:48 +08:00
b0fc6e7a69
[Feature-5752][server]Add blocking task ( #6272 )
...
* fix python task execution error
* modify code style
* delete AbstractTask#setCommand
* migrate blocking task alert feature and common attribute
* blocking task migrate(complete base features)
* migrate blocking task test
* resolve TaskNode.java conflict
* fix package problem in BlockingParameters
* resolve duplicate code
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* optimise code based on pr#6283 in ds master repo
* add log
* modify log
* resolve conflicts
* change hardcode to variable
* Add blocking task UI
* refactor old code to compatable with the newest branch
* add override annotation
* remove old code
* remove never used packages
* add missing packages declaration
* Refine the i18n description
* add UT
* resolve some code smell problems
* add fetch-depth
* enable unit test in sonar analysis
* disable unit test
* remove unused packages
* resolve duplicate codes
* add logs when task status changed and rename some variables
* add BLOCK & READY_BLOCK status and corresponding processing code
* revoke UI
* revert unit-test
Co-authored-by: Jave-Chen <baicai.chen@gmail.com>
2022-03-07 10:50:40 +08:00
67a6813e9c
remove no useful code ( #8552 )
...
Co-authored-by: ywang46 <ywang46@paypal.com>
2022-03-03 19:09:35 +08:00
74afdcf8f3
[Fix-8506][Master] process instance stop result wrong ( #8529 )
...
* fix bug_8506
* recover success state
2022-03-01 15:20:02 +08:00
a72742b9de
remove unuse import jsonFormat ( #8548 )
2022-02-25 22:05:05 +08:00
84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified ( #8477 )
...
* date convert of timezone
* remove @JsonFormat
* add unit test
* fix time preview in scheduler
* optimization & add env config
Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-24 18:20:34 +08:00
fa19006a18
Add a property to enable/disable metrics module ( #8484 )
2022-02-24 10:21:47 +08:00
234f399488
remove worker group lowercase ( #8448 )
2022-02-19 21:38:58 +08:00
8200a3f15a
add try catch ( #8433 )
2022-02-18 21:46:52 +08:00
864ceb164f
[Fix-Typo] Fix typo in dev branch. ( #8395 )
...
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* test
* fix dev_bug_8337
* fix dev_bug_8337
* fix dev_bug_8337
* fix bug_8337
* fix typo
2022-02-16 17:38:43 +08:00
08f642c756
[Fix-8337][Master] Process instance can not be kill when task is failure and can be retry ( #8387 )
...
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* test
* fix dev_bug_8337
* fix dev_bug_8337
* fix dev_bug_8337
* fix bug_8337
2022-02-15 19:04:19 +08:00
66dcf2376a
Optimize some code ( #8324 )
2022-02-10 19:07:53 +08:00
4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 ( #6718 )
...
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
* fix Constants bug
* remove the unused class
* add unit test
* fix code style error
* add unit test
* refactor data quality common entity
* fix code style error
* add unit test
* close e2e test
* fix code smell bug
* modify dataquality enum value to 14 in TaskType
* add data qualtiy task
* update
* add getDatasourceOptions interface
* fix checkstyle
* close e2e test
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* add unit test and alter dataquality task result
* fix checkstyle
* fix process service test error
* add unit test and fix code smells
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* change execute sql type name
* revert ui pom.xml
* fix data quality task error
* fix checkstyle error
* fix dq task src_connector_type ui select bug
* fix spark rw postgresql bug
* change mysql driver scope
* fix form-create json bug
* fix code smell
* fix DolphinException Bug
* fix ui validate rule and Alert title
* fix target connection param bug
* fix threshold validate change
* add rule input entry index
* change statistic_comparison_check logic
* remove check type change
* add DateExpressionReplaceUtil
* fix null point expetion
* fix null point expetion
* fix test error
* add more sql driver
* fix test error and remove DateExprReplaceUtil
* add get datasource tables and columns
* add get datasource tables and columns
* remove hive-jdbc in pom.xml
* fix code smells
* update sql
* change the pom.xml
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
* update sql
* [Feature][DataQuality] Add data quality task ui (#5054 )
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* fix dq task src_connector_type ui select bug
* fix threshold validate change
* remove check type change
* add get datasource tables and columns
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality module (#4830 )
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* update
* close e2e test
* fix spark rw postgresql bug
* change mysql driver scope
* add more sql driver
* remove hive-jdbc in pom.xml
* change the pom.xml
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality task backend (#4883 )
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* refactor data_quality_module
* add header license
* data quality module refactor
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix code smell
* fix check style
* fix unit test error
* task statistics value add unique code
* fix unit test error
* fix checkstyle error
* fix checkstyle
* fix security hotspot
* fix unit test error
* fix security hotspot
* fix check
* add data quality task error handling
* fix unit test error
* add unit test
* add unit test
* optimize data quality result alert
* fix unit test
* fix sql script error
* fix bug
* update sql script
* fix checkstyle
* add license
* fix checkstyle
* fix checkstyle
* fix unit test
* add jacoco dependencies
* fix unit test
* fix unit test
* add jacoco dependencies
* add unit test
* add unit test
* add license
* fix checkstyle
* fix pom
* fix checkstyle
* fix checkstyle
* merge dev
* fix ui error
* fix pom error
* fix pom error
* fix test error
* fix test error
* mssql-jdbc exclude azure-keyvault
* fix test error
* merge dev and add unit test
* add notes
* rollback the CollectionUtils
* fix
* update sql
* fix
* fix
* fix query rule page error
* change dq.jar path
* fix sql error
* fix ui error
* fix(dq): jar path&task enum description
* add notes on DataQualityApplication
* fix dq result jump error
* fix(ui): page condition
* feat(ui): add show error output path
* change version
* remove all chinese word in sql
* merge
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2022-01-27 15:55:45 +08:00
f88def8ef7
[IMPROVEMENT-8178] Add Netty processor in Spring container ( #8179 )
...
* Add Netty processor in Spring container
2022-01-25 21:33:09 +08:00
103ccb01da
change version to 2.0.4-SNAPSHOT ( #8188 )
2022-01-25 15:17:17 +08:00
b5fa54b6be
[Bug-7865][MasterServer] retry logic optimization ( #8156 )
...
* submit task optimization
* cloneAndReset
* update
* tolerant task restart when init
* fix snoar check
* fix test
* delete unuse file
* taskInstance key
* code style
* skip complete when retry
Co-authored-by: caishunfeng <534328519@qq.com>
2022-01-25 09:37:57 +08:00
Wenjun Ruan