45586f71d5
[BUG][Resource Center] Task can not select main JAR ( #13536 )
2023-04-18 18:38:16 +08:00
79cb707b57
[Improvement-13932][API] Tenant parameters and Worker group parameters are put together in the task startup parameters ( #13935 )
...
* [Improvement][API] Tenant parameters and Worker group parameters are put together in the task startup parameters
2023-04-18 14:56:38 +08:00
119f08d2ba
[Bug-13882][Worker] Upgrade kubernetes-client version to 6.0.0 ( #13891 )
2023-04-17 12:04:36 +08:00
4bf097ee40
[Improvement-13824][Resource Center] User can only visit resources under tenant defaultPath ( #13848 )
...
* [Improvement-13824][Resource Center] User can only visit resources under tenant defaultPath
* remove semicolon
* set null in some specific scenario
* remove useless console.log
2023-04-17 11:56:42 +08:00
51afbfe159
Fix schedule time data primary key conflict when importing workflow definition ( #13926 )
2023-04-13 17:01:10 +08:00
2177d70150
Set user default tenant ( #13909 )
2023-04-13 15:43:24 +08:00
bae8df6aa1
[Improvement-13911][JSON] Improve the readability of workflow exported json files ( #13912 )
2023-04-13 11:15:07 +08:00
61637cc0a3
[Bug] [Cron] Parse corn expression error ( #13841 )
...
* fix-cron
* update catch exception
2023-04-09 12:50:42 +08:00
7ee66f2d02
[Feature-13429][Remote Logging] Add support for writing task logs to Google Cloud Storage ( #13777 )
2023-04-06 17:49:33 +08:00
41a8ba9aab
Support alert server HA ( #13865 )
...
* Support alert server cluster
* Remove WORKER_ALERT_LISTEN_HOST in statefulset-dolphinscheduler-worker.yaml
2023-04-05 17:31:50 +08:00
8f3a23dae5
[Fix-13828][api] Fix the problem of work groups have two defult records ( #13829 )
2023-03-31 11:18:17 +08:00
273a5e78eb
[FEATURE][Task Plugin]Add remote-shell task plugin ( #13801 )
2023-03-30 20:17:28 +08:00
9df89cf0cf
Use bootstrap user as the default tenant ( #13825 )
2023-03-30 14:11:23 +08:00
85605bbef9
[Improvement-13653][Remote Logging] Support api-server to get task log from remote target ( #13654 )
2023-03-30 12:07:58 +08:00
68660ec96b
Refactor remote command ( #13809 )
...
* Refactor remote command
* Rename Command to Message
2023-03-29 17:54:57 +08:00
4e2c22061a
[Fix-11583][UI] Fix the API usage of gantt graph ( #13778 )
2023-03-28 17:17:10 +08:00
8895921d87
[Improvement-13651] Remove useless resources_task ( #13721 )
2023-03-23 22:52:23 +08:00
1075be6e62
[Improvement] Test DataSource connection return msg may not internationalization ( #13767 )
...
Co-authored-by: xuhaihui <xuhaihui@cmss.chinamobile.com>
2023-03-21 09:58:24 +08:00
1f365819a6
Ignore unknown VM options in start.sh ( #13719 )
2023-03-10 17:34:09 +08:00
394805b2c7
[Feature][Metrics] Tag workflow related metrics with process definition code (workflow id) ( #13640 )
...
* Tag workflow related metrics with process definition code (workflow id)
* Clean up related metrics when deleting workflow definition
* Add license headers
* Update related UT cases
* Add an example in grafana-demo
* Add related docs
2023-03-09 11:30:21 +08:00
76f3076481
[Bug] Fix ResourcesServiceImpl.java pageInfo ( #13709 )
2023-03-09 11:05:52 +08:00
30fd43e32b
feat: Python add online_schedule for workflow control schedule state ( #13551 )
...
related: apache/dolphinscheduler-sdk-python#73
2023-03-07 11:35:01 +08:00
512899ed14
feat: Python return datasource object and allow filter by type ( #13616 )
...
related: https://github.com/apache/dolphinscheduler-sdk-python/pull/75
2023-03-07 11:34:28 +08:00
a8e3deca8b
[Improvement][api] Remove unless code in ResourcesServiceImpl ( #13676 )
...
* remove unless code in ResourcesServiceImpl
---------
Co-authored-by: chenjiaming <chenjiaming@kezaihui.com>
2023-03-06 10:40:17 +08:00
7c97febefb
[Improvement-12697][UT] Improve the ProjectServiceTest ( #12698 )
2023-03-01 11:37:52 +08:00
eb4cba8884
[Improvement-12782][UT] Add UT for the 13 implementations of ResourceAcquisitionAndPermissionCheck ( #12784 )
2023-03-01 11:37:19 +08:00
4351a25f2a
Add execute function to handle the workflow instance operation ( #13610 )
2023-02-24 13:47:41 +08:00
df963f8393
[Improvement] Add call cost time for api ( #13518 )
2023-02-09 14:29:23 +08:00
8d12dc0702
[Improvement-13491] Use lombok @Slf4j annotation to generate logger ( #13509 )
2023-02-07 20:32:53 +08:00
8a59ab44eb
[Fix-13434][Resource Center] fix the NPE when use OSS as the resource center ( #13435 )
2023-02-07 11:55:07 +08:00
3627bf25ef
[Improvement-13472][Api] Removes the overhead of idle Quartz Threadpool of Api ( #13475 )
...
* [fix-13472][Api] change Threadpool of Quartz Scheduler to ZeroSizeThreadPool
* update doc about quartz threadpool configuration
2023-02-06 10:55:19 +08:00
0ee82c5492
[doc] Update LDAP config doc ( #13507 )
2023-02-04 11:05:51 +08:00
19ffd64e23
Fix complement data date order will change due to use set ( #13487 )
2023-02-03 19:44:45 +08:00
80da35e39c
[Fix-12828][api] Add permission check when query specific datasource ( #12830 )
2023-02-01 10:41:05 +08:00
78ba71bdbd
[Feature][task]add azure datafactory task type ( #13417 )
...
* init data factory
* add task-type-config.yaml
* add data factory ui
* submit doc
* submit license
---------
Co-authored-by: devosend <devosend@gmail.com>
2023-01-30 11:15:20 +08:00
713fdbc444
Split the upgrade code ( #13436 )
...
1. Split the upgrade code into different version
2. Log the dml/ddl sql in origin friendly format
3. Fix ddl of 3.0.0 and 3.1.0
4. Fix search resource regular expression injection
2023-01-20 20:48:07 +08:00
3b54de0c90
fix when the task details are not modified, the predecessor node cannot be modified ( #13422 )
...
* fix when the task details are not modified, the predecessor node cannot be modified
* deleted useless code
Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
2023-01-18 20:08:28 +08:00
8be32d4145
[Feature][Api] When use api to run a process we want get processInstanceId ( #13184 )
...
* add sql
* add mapper
* add dao
* add excutor
Co-authored-by: qianl4 <qianl4@cicso.com>
2023-01-18 17:58:32 +08:00
01f4fb26f0
Change remove task instance log to async request ( #13399 )
2023-01-16 15:49:23 +08:00
8439b5dc69
[fix-12721] Fix cannot modify the upstream task in task definition page. ( #12722 )
2023-01-15 19:58:47 +08:00
98dfd64444
Clear the task group data when delete a project/workflowInstance ( #13383 )
2023-01-12 21:51:51 +08:00
7027d3d768
[improve-#13053] in method of ``` execProcessInstance``` check ```startNodeList``` ( #13057 )
...
* [improve-#13053] in method of ``` execProcessInstance``` check ```startNodeList```
* [improve-#13053] format
* [improve-#13053] fix UT
* Update Status.java
update description
* [improve-#13053] add parameter ```version ```
* [improve-#13053] format
* [improve-#13053] version is not necessary
* [improve-#13053] format
* Update use-form.ts
change ```version``` default vaule to ``` null```
* [improve-#13053] format
* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
* Update ExecutorServiceImpl.java
Co-authored-by: fuchanghai <‘2875334588@qq.com’>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2023-01-07 20:49:59 +08:00
a074f7e2e6
Delete workflow will delete workflow instance ( #13336 )
2023-01-04 22:22:03 +08:00
8a479927f3
Add projectCode in t_ds_process_instance and t_ds_task_instance to remove join ( #13284 )
2023-01-03 09:52:28 +08:00
d42f576268
[fix][python] schedule unexpect offline each time update from api ( #13301 )
...
* Alway set workflow online before set schedule online
* Avoid return map in interface setScheduleState
2022-12-30 09:59:16 +08:00
04ea8f7e28
Revert "[Bug] [Server] Once click online schedule time, task will be automatically scheduled ( #13092 )" ( #13295 )
...
This reverts commit 7497b26979
2022-12-28 22:02:48 +08:00
7497b26979
[Bug] [Server] Once click online schedule time, task will be automatically scheduled ( #13092 )
...
* fix bug that trigger mis-fire strategy when setting start time earlier than current time
* update ut
* add warning msg
* add check start time when set schedule online
2022-12-28 14:51:59 +08:00
df32ef0efb
Fix when update scheduler will execute workflow ( #13285 )
2022-12-28 14:33:38 +08:00
fd3afd84ad
Remove alert when delete workflow instance ( #13281 )
2022-12-28 11:40:12 +08:00
a75a69f3ec
[Improvement][api]Start the downstream workflow when it is online ( #13257 )
...
Start the downstream workflow when it is online
2022-12-26 11:58:37 +08:00
8870464c02
[Improvement][api] Optimize batch query in process-list interface ( #13222 )
...
Co-authored-by: chenjiaming <chenjiaming@kezaihui.com>
2022-12-23 23:33:42 +08:00
a98cae7065
[Fix-13260][API] Ironed out the issue with lacking the value of user's name when querying the list of workflow definitions. ( #13261 )
...
* fix the issue with lacking user name
2022-12-23 19:22:08 +08:00
a235d3cd6a
[Improvement][UI] Support to display extra columns such as 'Worker Group' and 'Environment Name' on the page of cron manage. ( #13267 )
2022-12-23 15:24:59 +08:00
14ec4a2398
Remove dao module in worker ( #13242 )
2022-12-22 12:25:29 +08:00
5cd8a02f87
[fix] Remove tenant arg in workflow definition ( #13212 )
...
current tenant in workflow only work when the first time
user do not exist, when user change the tenant in workflow
but tenant exist, it will be ignore, so we try to remove it
from workflow, and in #40 we try to create both user and tenant
vis cli instead of auto create
2022-12-19 16:06:41 +08:00
66e20271ad
[Feature][Master] Add task caching mechanism to improve the running speed of repetitive tasks ( #13194 )
...
* Supports task instance cache operation
* add task plugin cache
* use SHA-256 to generate key
* Update dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
* Update dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_postgresql.sql
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
* Optimizing database Scripts
* Optimize clear cache operation
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
2022-12-18 18:17:09 +08:00
042ec74a24
[refactor] cache compiled regex pattern ( #13208 )
...
* [refactor] cache compiled regex pattern
* [refactor] cache compiled regex pattern
* [refactor] cache compiled regex pattern
2022-12-18 11:24:11 +08:00
0e3a924a37
move v2folder ( #13192 )
2022-12-16 10:30:01 +08:00
ca5af013ae
[Fix] openapi version inconsistency when updating workflows, tasks and relationship ( #13094 )
...
Co-authored-by: xiangzihao <460888207@qq.com>
2022-12-14 14:57:57 +08:00
631e3a81d1
[Feature][API] New restful API for taskInstance ( #13070 )
2022-12-14 14:47:36 +08:00
e4b9b67255
Allow execute task in workflow instance ( #13103 )
2022-12-13 16:43:44 +08:00
0db1bb6ba8
[Improvement][ApiServer]Task defination list paging sorted by updateTime not effect ( #12622 )
2022-12-13 16:35:54 +08:00
ab96a3d0c0
Fix datasource jdbc url param doesn't work ( #13098 )
2022-12-12 11:46:40 +08:00
f5801a0e1c
Fix task group might be rob by multiple task ( #13115 )
2022-12-11 22:58:21 +08:00
6ef74073cc
[Improve-13001]migrate commons-collections -> commons-collections4 ( #13002 )
2022-12-10 23:50:19 +08:00
f9cf9331a6
Throws the correct exception for the updateResource interface ( #13120 )
...
Co-authored-by: rqyin <rqyin@easipass.com>
2022-12-09 16:17:20 +08:00
cb8d125e0f
[Improvement] optimization task definition & fix in the task definition list, if one task have more pre task, the task list can't show all task ( #13106 )
...
* fix in the task definition list, if one task have more pre task, the task list can't show all task
* modify code style
* in the task definition, delete search workflow name filter, and fix in the task definition list, if one task have more pre task, the task list can't show all task
* modify code style
* modify code style
* delete useless select sql
* add annotation
Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
2022-12-07 09:52:53 +08:00
8a152aebc7
Fix task log file might not be clear ( #13102 )
2022-12-06 11:23:20 +08:00
fed1e85fb8
Fix the invalidate task instance never be deleted ( #13091 )
2022-12-04 18:38:54 +08:00
169168ef34
Add plugin-all module ( #13079 )
2022-12-02 23:19:08 +08:00
1192720ddc
[ci] Update skywalking-eye and fix block CI ( #13076 )
2022-12-02 17:54:00 +08:00
f5a59982b3
[fix] Python gateway can not upload to resource center ( #13042 )
...
in #12076 we refactor our resource center, remove the resource table,
it is a good refactor but it failed python api upload, this patch try
to fix python api upload function
ref: #12076
2022-12-02 10:11:57 +08:00
1064680ded
[feat][API] New restful API for workflow state ( #13031 )
2022-12-01 14:21:01 +08:00
c2ae76c0bd
Expand all static imports and add CI check to block wildcard static imports ( #13067 )
2022-12-01 14:15:28 +08:00
ffc9fb280a
Add gc timestampt ( #13059 )
2022-12-01 10:00:27 +08:00
a0eb45b9e4
[Feature][API] New restful API for workflowInstance ( #12990 )
2022-11-28 12:58:46 +08:00
7336afaa65
[Fix-12916] Add permission check when query or download log ( #12917 )
2022-11-25 17:59:28 +08:00
d09e02e5a9
[Improvement-12804][UT] Remove the unused method in DataAnalysisControllerTest ( #12805 )
2022-11-21 20:59:30 +08:00
20518682bb
[Improvement-12853][UT] Improve the ut of ResourcePermissionCheckServiceTest ( #12854 )
2022-11-19 15:12:12 +08:00
528f45acc5
[improve]Source skip check ( #12900 )
...
* Source skip check
2022-11-17 14:23:12 +08:00
2dbc79693e
[feature][task plugin] Add new task plugin for apache linkis ( #12693 )
2022-11-17 10:14:36 +08:00
f20c9b3102
[chore][python] Change name from process definition to workflow ( #12918 )
...
only change its name in python gateway server code, incluing
* Function name: all related to process definition
* Parameter name and comment related
ref: apache/dolphinscheduler-sdk-python#22
2022-11-16 21:23:39 +08:00
0186413a9f
[Feature][Api] Refactor org.apache.dolphinscheduler.api.controller.TaskInstanceController ( #11747 )
...
* Feature ：Refactor org.apache.dolphinscheduler.api.controller.TaskInstanceController
2022-11-16 18:32:42 +08:00
6d8befa075
[fix] Add token as authentication for python gateway ( #12893 )
...
separate from #6407 . Authentication, add secret to ensure only trusted people could connect to gateway.
fix : #8255
2022-11-14 18:43:08 +08:00
d324cce6f1
[Feature-12526][Api] add dynamic task function: shell task ( #12840 )
...
* submit dynamic task code
* submit shell.json
* add dynamic-task-type-config.yaml to standalone server
2022-11-14 13:52:44 +08:00
813c44b22b
[Improvement-12838][UT] Improve the ut of datasource ( #12839 )
2022-11-14 11:30:10 +08:00
e3cf72cbcd
[Fix-12832][API] Fix update worker group exception group name already exists. ( #12874 )
2022-11-14 11:20:51 +08:00
87a88e3662
[feat] Support set execute type to pydolphinscheduler ( #12871 )
...
Up to now, we can only submit workflow with parallel mode. this patch give users ability specific execute type to workflow
2022-11-12 10:31:08 +08:00
229c554912
[feature][task] Add Kubeflow task plugin for MLOps scenario ( #12843 )
2022-11-11 16:08:38 +08:00
1347a8f94c
[chore] Separate Python API into another repository ( #12779 )
...
Currently, our Python API code is a module in apache/dolphinscheduler codebase,
each time users change Python API code, they need to run all requests CI check
for dolphinscheduler and Python API, But if the user does only change Python
code, it could be merged if Python API CI pass and do not dependent on others CI.
Besides, we release Python API as the same version of dolphinscheduler. It is
easy for user to match Python API version. But when Python API does not change
any code, but dolphinscheduler release a bugfix version, Python API has to
release the new version to match dolphinscheduler. This happened when we
released Python API 2.0.6 and 2.0.7. 2.0.6 and 2.0.7 is bugfix version, and
Python API does not change any code, so the PyPI package is the same.
Separate Python API also makes our code more sense, we will have more
distinguished code in dolphinscheduler and Python API new repository.
Have separate issue tracker and changelog for information to users.
ref PR in other repository: apache/dolphinscheduler-sdk-python#1
see more detail in mail thread: https://lists.apache.org/thread/4z7l5l54c4d81smjlk1n8nq380p9f0oo
2022-11-09 21:20:15 +08:00
641a2559ea
[fix-12687]when timeout_notify_strategy is null, modify timeout_notify_strategy ( #12689 )
...
* According to timeoutFlag, assign timeoutNotifyStrategy value
Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
2022-11-09 11:48:12 +08:00
04c6b5e413
[Bug][Api] Added param description. ( #12813 )
...
Co-authored-by: houshitao <shitaohou@163.com>
2022-11-08 21:17:40 +08:00
23ead4f0e8
[Bug] [dolphinscheduler-api] in the task define list, when edit task, should show pre node list ( #12695 )
...
* backend : add pre node list
* resolve workflow error
Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
2022-11-08 13:31:20 +08:00
6d79414491
Add instructions on how to configure logging level for debugging purpose ( #12699 )
2022-11-07 10:30:56 +08:00
606a5aeee5
[Bug] Resource default auth function disabled false. ( #12467 )
...
* resource function disable check fix
Co-authored-by: Kerwin <37063904+zhuangchong@users.noreply.github.com>
2022-11-06 21:15:27 +08:00
f249f2b257
[fix-12675]edit workflow related task, workflow's task version change ( #12692 )
...
* edit workflow related task, workflow's task version change
* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java
Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-11-06 20:49:07 +08:00
04e23ed88f
Update Status.java ( #12724 )
2022-11-05 21:09:17 +08:00
7d0e2cbbb9
when delete workflow, delete related task ( #12681 )
...
* when delete workflow, delete related task
Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
2022-11-04 14:40:15 +08:00
a47b4d0672
[Feature][Task] Transfer files between tasks ( #12552 )
...
* add data transfer between tasks
* add delete DATA_TRANSFER API
* convert Result to DeleteDataTransferResponse
* add api UT
* fix final
* fix doc
2022-11-03 16:33:39 +08:00
c69376e74c
[Improvement-12658][Permission] Improve the readability of setPerm() in grantProjectbyCode() ( #12660 )
2022-11-03 09:29:04 +08:00
44e0935f56
[Improvement-12650][Permission] Improve the check of resourcePermissionCheck() ( #12652 )
2022-11-03 09:12:56 +08:00
3ff328c961
[Improvement-12623][Permission]Remove the unused projectIds in operationPermissionCheck() ( #12624 )
2022-11-02 11:20:00 +08:00
d84f1ef269
[Improvement][UI] Support to view the process variables on the page of DAG. ( #12609 )
2022-10-30 20:31:40 +08:00
fb9761f27a
Fix alert status spelling error ( #12592 )
...
* fix alert status spelling error
* fix-ut
2022-10-29 21:33:03 +08:00
065d5caccc
Only expose necessary actuator endpoints ( #12571 )
2022-10-28 07:40:32 +08:00
dc8d18cf87
[Feature-12040][api][ui] Add authorization management of read and write permissions for project center ( #12048 )
...
[Feature-12040][api][ui] Add authorization management of read and write permissions for project center
2022-10-27 18:01:51 +08:00
4a13148bfc
[Bug] [API] Before deleting a worker group, check whether there is environment that reference the worker group. ( #12534 )
...
* [Bug] [API] Before deleting a worker group, check whether there is environment that reference the worker group.
Co-authored-by: houshitao <shitaohou@163.com>
2022-10-27 16:34:08 +08:00
4b4d0b92a3
[FIX][DAO-12277] change t_ds_fav_task column task_name to task_type and related code ( #12276 )
...
* change t_ds_fav_task column task_name to task_type and related code
* change task_name to task_type
* change FavTask taskName to taskType, taskType to taskCategory
Co-authored-by: labbomb <739955946@qq.com>
2022-10-27 09:36:26 +08:00
ed209bdf82
[Refactor] Migrate all workergroup-related interface functions from ProcessServiceImpl ( #12493 )
...
* Migrate all workergroup-related interface functions from ProcessServiceImpl
2022-10-26 19:18:32 +08:00
547aa437ab
[Bug] [API] The workflow definition and the tenant in the workflow instance are inconsistent. ( #12533 )
...
* [Bug] [API] The workflow definition and the tenant in the workflow instance are inconsistent
* fix NPE.
Co-authored-by: houshitao <shitaohou@163.com>
2022-10-26 16:11:24 +08:00
489e7fe4e2
[Feature-10495][Resource Center] Resource Center Refactor ( #12076 )
...
* resource center refactor - S3 services connection
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-10-26 13:53:44 +08:00
04aa125ba2
[Refactor]migrate some task dao functions from ProcessServiceImpl to dao modules. ( #12505 )
...
* migrate some task DAO to dao modules.
* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/repository/impl/ProcessInstanceMapDaoImpl.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-10-26 11:51:58 +08:00
af9374c2c2
fill up missing i18 properties ( #12532 )
2022-10-26 11:39:26 +08:00
233bbdb09e
[Fix-12451][k8s] Read the kubeconfig from cluster conf ( #12452 )
2022-10-25 11:25:30 +08:00
77e29b59b2
[Improvement] add springdoc-openapi-ui ( #12379 )
...
Co-authored-by: fuchanghai <‘2875334588@qq.com’>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
2022-10-23 23:17:53 +08:00
dedff70f90
[Improvement] Merge spi.utils into common.utils ( #12502 )
...
* improve JSONUtil
* merge spi.utils into common.utils
* split constants && remove some copy utils method
* remove StringUtils
2022-10-23 21:23:54 +08:00
b936b882bb
[Refactor] Migrate all command-related interface functions from ProcessServiceImpl ( #12474 )
...
* migrate all command-related interface functions to CommonService
2022-10-22 12:39:25 +08:00
1384d8f4fa
set @Transactional rollbackFor ( #12445 )
2022-10-22 11:33:17 +08:00
b5193ee4e5
Remove equals in User to fix UT ( #12487 )
2022-10-22 08:59:10 +08:00
1436ad65fc
[Bug-12410] [API]Fix the worker list result in workflow definition only has default ( #12411 )
...
* Fix the worker list result in workflow definition only has default
* Fix tenant, k8s namespace similar permission issues.
* code clean in the k8s namespace mapper
* delete redundant sql method
2022-10-21 13:54:28 +08:00
f0a34cddec
[improvement-#12416] Use CollectionUtils.isEmpty judge the collection isEmpty to avoid NPE ( #12449 )
2022-10-20 14:26:56 +08:00
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports ( #12412 )
...
* Fix remaining style errors and wildcard imports
* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link
* Fix spotless matching scope of markdown files
2022-10-19 11:08:04 +08:00
fccbe5593a
When the workflow definition is copied, the operation user of the timed instance is changed to the current user. ( #12400 )
2022-10-18 08:56:19 +08:00
4ea7eec750
Use temurin Java docker images instead of deprecated ones ( #12334 )
2022-10-17 15:46:21 +08:00
55004bebe0
[Improvement-12391][API] Workflow definitions that contain logical task nodes support the copy function ( #12392 )
...
* Workflow definitions that contain logical task nodes support the copy function.
2022-10-17 10:28:58 +08:00
2f37da0dbc
[Improvement][Test] Block the usage of powermock and move mockito dependencies from sub-modules to root pom ( #12311 )
...
* move mockito dependencies from sub-modules to root pom
* Add check in CI to block the usage of powermock
2022-10-14 12:06:51 +08:00
224508850e
[Migrate][Test] Migrate all UT cases from jUnit 4 to jUnit 5 in alert and api module ( #12337 )
...
* Migrate all UT cases from jUnit 4 to jUnit 5 in alert and api module
* Fix potential problems
2022-10-13 14:40:57 +08:00
db1d319159
Reorganize some classes in common module, remove duplicate classes ( #12321 )
2022-10-13 10:23:27 +08:00
17cd644506
[Fix-#11669][Workflow Instance Page] Fix the duration in Workflow Instance page. ( #12264 )
...
* fix process duration issue
* use processInstance.getState().isFinished() replace new isFinish function
* add WorkflowUtils
2022-10-13 09:50:31 +08:00
0e1c8d8153
[Feature][API] New restful API task and task relation ( #12116 )
2022-10-13 09:41:03 +08:00
05099bd8f7
[Improvement-12296][Test] Close static mock ( #12300 )
2022-10-11 16:13:05 +08:00
18ef0a66cb
Set the id to null when copying t_ds_schedules ( #12280 )
...
Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
2022-10-11 10:53:25 +08:00
b4947b471c
fix this issue ( #12281 )
2022-10-10 09:34:32 +08:00
c7435a51ac
[Feature][task-plugins] aws datasync task plugins ( #11865 )
...
* submit datasync task
* add aws software package license
2022-10-09 10:47:01 +08:00
585efe3ad5
[Improvement][Test] Remove the usage of powermock in datasource-plugin module ( #12194 )
...
* Remove the usage of powermock in datasource-plugin module
* Switch datasource-plugin module to use PropertyUtils in common module
2022-09-30 17:38:24 +08:00
aacabc6ec7
Remove workerGroup in registry ( #12217 )
2022-09-30 13:58:53 +08:00
308f632fcc
[Feature][TASK-PLUGIN] AWS Database Migration Service (DMS) ( #11868 )
...
* add aws dms
2022-09-30 10:16:16 +08:00
1a4d7a8426
[Improvement-11678][API] Improvement the error message when batch delete workflow ( #11682 )
...
* [Improvement] Improvement the error message when batch delete workflow
2022-09-29 19:08:26 +08:00
e27c79974d
[fix-#11753] send alert error alert data id ( #11774 )
...
* [fix-#11753] send alert error alert data id
Co-authored-by: fuchanghai <changhai.fu@marketingforce.com>
2022-09-29 15:34:40 +08:00
57fafe4256
[Improvement-12144] Remove powermock in dolphinschduler-api ( #12169 )
2022-09-29 12:00:41 +08:00
d526dd89f6
[hotfix][UI] Remove SURVEIL and NEXT_LOOP in ui ( #12160 )
...
* Remove SURVEIL and NEXT_LOOP in ui
* Fix UT
2022-09-28 09:17:12 +08:00
7ed52c3ecb
[improve][python] Validate version of Python API at launch ( #11626 )
2022-09-27 21:28:20 +08:00
cf5a8894e2
[BUG][BE-12113]fix htrace dependency conflicts ( #12126 )
...
* fix htrace dependency conflicts
* replace htrace-core with hbase-noop-htrace
2022-09-26 13:49:08 +08:00
64f4cb4f3e
refactor: dag-menu refactor ( #11965 )
...
* refactor: dag-menu refactor
* feat: add i18n support for the dag-menu
* feat: add e2e class
Co-authored-by: devosend <devosend@gmail.com>
Co-authored-by: Tq <tianqitobethefirst@gmail.com>
2022-09-26 11:06:39 +08:00
82ddd72e4a
[Feature][API] New restful API for workflow and schedule ( #11912 )
...
* [feat] New restful API for workflow and schedule
CURD for workflow and schedule, different with exists
API, this new restful api only operate single resource
in each request, and return the latest. For example,
previous workflow should also need to post tasks definition
and tasks relation definition, but this patch will allow
you to create workflow without task relate information
* use checkProjectAndAuthThrowException, and fix CI error
* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ScheduleV2Controller.java
* change method name
from createProcessDefinitionV2 to createSingleProcessDefinition
from updateProcessDefinitionV2 to updateSingleProcessDefinition
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-09-23 22:25:19 +08:00
6e03a7e557
[BUG]Delete Setting ID ( #12096 )
...
* Delete Setting ID
* modify id
Co-authored-by: jackfanwan <fanwanlong@kezaihui.com>
2022-09-22 12:12:43 +08:00
cc492c3e13
[Feature] Add CURD to the project/tenant/user section of the python-DS ( #11162 )
...
- Add CURD in project
- Add CURD in tenant
- Add CURD in user
- Add test in user
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-09-22 11:03:27 +08:00
fba5a8eaa0
Fix insert command error due to the id is not null ( #12092 )
2022-09-22 09:52:32 +08:00
08a4c7981f
[improve] Remove the spark version of spark task ( #11860 )
2022-09-21 20:27:28 +08:00
b52da64001
[Improvement][UT] Upgrade jUnit to 5.+ ( #10976 ) ( #11332 )
...
* [Improvement][UT] Upgrade jUnit to 5.+ (#10976 )
* Refactor AlertServerTest with jUnit-5 as an example
2022-09-21 10:15:21 +08:00
ed1d1e8856
add(dependent node's add unprivileged project list): dependent node's add unprivileged project list ( #12073 )
2022-09-21 09:28:22 +08:00
66958b91d7
[Feature][Resource Center] Add support for Alibaba Cloud OSS as storage of resource center ( #11708 )
...
* Add support for Alibaba Cloud OSS as storage of resource center
* Add dependency licenses
* Refactor code, add unit tests and related docs
2022-09-20 22:44:06 +08:00
26dad5e2a8
Fix update TaskDefinition error ( #12060 )
2022-09-20 22:23:16 +08:00
0bf639b851
[improvement]Add Set cluster name ( #12058 )
...
* Add Set cluster name
* add unit test
Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
2022-09-20 21:26:05 +08:00
955e8ebe0f
[Improve] Fix support test tasks UT ( #12062 )
2022-09-20 15:46:29 +08:00
649015bfff
[Bug][ApiServer] When modifying the task definition, inserting the task definition log table will report the bug of primary key conflict #12037
...
Co-authored-by: 金鑫 <jinyanhui@huansi.net>
2022-09-20 14:42:21 +08:00
5b384f3fab
[Feature-11473][Task]Support test task ( #11670 )
...
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* Update TaskExecuteThread.java
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update DataSource.java
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update messages_zh_CN.properties
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update messages.properties
* Update messages_en_US.properties
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
2022-09-17 13:40:56 +08:00
1a7c6eabf3
[Improvement-11773][api] Optimize the log printing of the api module according… ( #11782 )
...
* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Resolve conflict.
* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- fix ProcessDefinitionServiceImpl.java dependency
* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Accept proposed change in ResourcePermissionCheckServiceImpl.java
* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Update the UT case in TenantServiceTest.java, UsersServiceTest.java, TaskGroupServiceTest.java.
- Update log printing in TaskGroupServiceImpl.java.
- Fix NPE of log parameter in ExecutorServiceImpl.java when call python api.
* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix NPE of log in WorkerGroupServiceImpl.java.
- Fix log parameter missing bug in ResourcesServiceImpl.java.
* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix bugs and handle some vulnerability codes.
* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix bugs and handle some vulnerability codes.
* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix e2e User test.
* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix codeQL check & SonarCloud Code Analysis.
* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Resolve import conflict.
* [DS-11773][api] Optimize the log printing of the api module according to the log specification doc.
- Fix codeQL check & SonarCloud Code Analysis.
2022-09-17 11:28:04 +08:00
864a90820d
Fix insertOrUpdate plugin may failed due to concurrent operation ( #11471 )
2022-09-16 15:39:04 +08:00
d833a28b2e
Fix cannot save processDefinition ( #11931 )
2022-09-16 13:54:18 +08:00
bd0fa79e37
[Bug][Dependent]: Id also clone due to duplicate when use dependent mode. ( #11929 )
2022-09-15 10:00:38 +08:00
dec6197a67
[Feature][Api] Refactor&Add api.controller.QueueV2Controller ( #11389 )
...
* [Feature][Api] Refactor&Add api.controller.QueueV2Controller
* update review
2022-09-15 09:37:12 +08:00
d3a77c68e6
Make LogServiceClient Singleton ( #11777 )
2022-09-14 16:18:08 +08:00
009cb68011
[Improvement-11834] Upgrade docker base image to support python3.9 ( #11835 )
...
* use openjdk:8-jre-slim-bullseye instead of buster to support python3.9
* use openjdk:8-jre-slim-bullseye instead of buster across all components
2022-09-14 09:29:45 +08:00
ca22c083b3
[Fix] [Quartz] cron did not work as expected ( #11851 )
...
* [Fix] [Quartz] cron did not work as expected
* [Fix] [Quartz] make the value of batchTriggerAcquisitionMaxCount to 1(default value)
Co-authored-by: hezhao2 <hezhao2@cisco.com>
2022-09-13 16:05:40 +08:00
8a47785f42
Fix listing processDefinition the schedulerReleaseState will never be null ( #11888 )
2022-09-13 10:07:12 +08:00
147e0ae8d7
Remove duplicate Spring version management ( #11899 )
2022-09-11 20:50:07 +08:00
db46e4786a
Add default constructor ( #11858 )
2022-09-11 19:12:03 +08:00
37325b4c34
[Bug](dependent) Dependent downstream trigger error when schedule cycle not day. ( #11734 )
...
* FIX: dependent
* FIX: version
* MOD: for review
2022-09-08 15:08:10 +08:00
2e61c76c22
[Improvement] Add remote task model ( #11767 )
2022-09-08 13:41:36 +08:00
efedc9a53c
Bump up dependencies to fix cves ( #11765 )
2022-09-08 09:05:05 +08:00
444c7e6eb7
[Improvement] Replace commons-lang 2 function invocations with commons-lang3 ( #11810 )
...
* [Improvement] Replace commons-lang 2 function invocations with commons-lang3
2022-09-07 13:48:09 +08:00
0e534dd61f
[python] Fix permission error when create related objects ( #11783 )
...
after we add permission to our codebase in dev branch, will got error
when create objects from pydolphinscheduler, because users with user_id
equal to `Integer.MAX_VALUE` is ordinary users which can not have access
to create unless we grant permission to the role manually. this patch
change dummyUser to admin user's user_id == 1 to fix it
2022-09-06 12:00:08 +08:00
8988492c43
[Feat][python] Add parameter environment to task ( #11763 )
2022-09-06 09:26:30 +08:00
4283cfd7a1
[fix][API]fix task types sequence ( #11775 )
...
* fix task types sequence
* fix mapper table name t_ds_fav_task
* fix sequence
2022-09-05 20:29:03 +08:00
3c86e52075
[Fix-11583][UI] Fix the API usage of gantt graph ( #11642 )
2022-09-05 14:20:48 +08:00
f8d46a26c1
[Feature-11530] add state history for process instance ( #11757 )
...
* [Feature] add state history for process instance (#97 )
* add state history for process instance
* upsertProcessInstance
* remove unuse method
* fix UT
Co-authored-by: caishunfeng <534328519@qq.com>
2022-09-03 10:14:05 +08:00
277f137358
Add Kubernetes configmap reload to all components ( #11730 )
2022-09-02 12:03:19 +08:00
b3cc8a55b9
[feature][API]feature: add task type list and enable user to add task type to fav ( #11727 )
...
* feature: add task type list and enable user to add task type to fav
* make DependentExecute.java variable protected
* add standalone server fileSet
2022-09-02 11:40:23 +08:00
67e7f88d8b
Refactor heart beat task, use json to serialize/deserialize ( #11702 )
...
* Refactor heart beat task, use json to serialize/deserialize
2022-08-31 16:20:23 +08:00
4dca488cd5
[python] Auto location when create process definition ( #11681 )
...
* Add auto format when submit process definition from python api
* add UPDATING.md
close : #8254
2022-08-27 23:38:46 +08:00
cf3b4424d7
[Fix-11616] Workflow Definition Name Change Validation Optimization. ( #11617 )
...
* Workflow Definition Name Change Validation Optimization
* [Fix][UI] Add the workflow code entry to the verify name interface.
* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ProcessDefinitionService.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
* processDefinitionCode param name change
Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-08-24 19:59:38 +08:00
961e67f524
[Improvement][User] Query all users filter un enabled. ( #11504 )
...
* user query all filter un enabled
* restore permissions
2022-08-24 11:55:18 +08:00
4893bef5a7
[Improvement][TaskInstance] reduce database queries ( #11522 )
...
* [Improvement][TaskInstance] reduce database queries
* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/TaskInstanceMapper.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
* [Improvement][TaskInstance] queryByInstanceIdsAndCodes -> queryByProcessInstanceIdsAndTaskCodes
Co-authored-by: zhangshunmin <zhangshunmin@kezaihui.com>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-08-23 14:27:26 +08:00
abfef1a929
[improvement] Add two parameters in workergroup, and support the application of description display and other parameters ( #11542 )
...
* [Improvement] Improvement default worker group
* [Improvement] Improvement default worker group
* [Improvement] Improvement default worker group
* [Improvement] Improvement default worker group
* [Improvement] Improvement default worker group
* [improvement] The default worker grouping supports editing
* [improvement] The default worker grouping supports editing
* Update WorkerGroupServiceImpl.java
* Update WorkerGroupServiceImpl.java
* The default worker grouping supports editing
* update sql file
* update
* update
* update
* update
2022-08-23 10:54:17 +08:00
8886d5a126
Fix quartz threadPriority config name error ( #11596 )
...
* Fix quartz threadPriority config name error
* Add batchTriggerAcquisitionMaxCount config
2022-08-23 09:32:05 +08:00
17a9dd25fa
Fix vulnerability in LDAP login ( #11586 )
2022-08-22 13:49:14 +08:00
13e4720aa2
Upgrade spring boot and swagger ( #11574 )
2022-08-22 10:27:40 +08:00
0b82755796
Fix some resource leak ( #11576 )
2022-08-21 01:19:16 +08:00
8b3f60ed88
[Improvement] [LDAP] Bug fix: Config value return "null" instead of real null currently. ( #11325 )
2022-08-19 10:34:00 +08:00
3b72c6efe7
[Improvement]Add a stop function when you are ready to do pause operation ( #11543 )
...
* add can stop for execute state machine
* add execute type check for pause operation
2022-08-18 21:32:53 +08:00
2862f5b67c
Set process instance and task priority default value as 2 ( #11539 )
2022-08-18 16:21:12 +08:00
ebdf903dcc
[Improvement-11468] Support re running historical version workflow ( #11489 )
...
* Support re running historical version workflow
Support re running historical version workflow
* Update ExecutorServiceImpl.java
* Update ExecutorServiceImpl.java
2022-08-18 09:46:54 +08:00
006d8e21a6
[Improvement] File deletion logic optimization ( #11519 )
...
* [Improvement] File deletion logic optimization
2022-08-18 09:38:06 +08:00
f6ef628030
Support reading application properties from kubernetes config map ( #11402 )
2022-08-16 20:25:17 +08:00
7ff34c3947
[Feature-7024] Add waiting strategy to support master/worker can recover from registry lost ( #11368 )
...
* Add waiting strategy to support master/worker can recover from registry lost
* throw exception when zookeeper registry start failed due to interrupted
2022-08-13 09:52:03 +08:00
8217784044
[Fix-1360]: Fix Description length check ( #11371 )
...
* [Fix-1360]: Fix Description length check
* Update dolphinscheduler_h2.sql
h2 Increase the length of some fields
* Update dolphinscheduler_ddl.sql
2022-08-11 13:26:50 +08:00
6e2bed4d6b
[fix-11378] fix API-Server release response message typo ( #11400 )
...
* [fix-11295] remove duplicate classes
* [fix-11378] fix response message error
Co-authored-by: fuchanghai <changhai.fu@marketingforce.com>
2022-08-11 09:21:32 +08:00
0464123c2b
[Feature-11223] support stream task ( #11350 )
...
* add task execute type
* update task definition list paging
* update task instance list paging
* stream task start
* [Feature][UI] Some changes to execute task.
* Set the connection edge to dashed line.
* Add FLINK_STREAM task.
* add stream task
* flink savepoint and cancel
* fix query bug
* add stream task definition
* add task instance for stream task
* delete stream task definition state
* update api for stream task definition edit
* modify search for stream task instance
* add language
* delete task type search for stream task definition
* change task type search for stream task instance
* add jump button
* add savepoint
* add down log for stream task instance
* ui test
* stream task start
* run DAG
* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.
* [Feature][UI] Remove some fields for FLINK_STREAM.
* add start modal
* add dryRun column for stream task instance
* fix duration
* fix pon
* fix build error
* Add success tip
* add auto sync for stream task instance
* remove forgien key for task instance
* license header
* UT fix
* modify locales
* recover common config
* fix UT
* add doc
Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
2022-08-10 21:44:43 +08:00
a50f110d3e
[Feature][Api] Refactor org.apache.dolphinscheduler.api.controller.QueueController ( #11154 )
...
* [Feature][Api] Refactor org.apache.dolphinscheduler.api.controller.QueueController
2022-08-10 11:34:37 +08:00
Aaron Wang