Commit Graph

736 Commits

Author SHA1 Message Date
caishunfeng 151c2d2c70
fix upgrade sql (#12141) 2022-09-26 15:21:52 +08:00
labbomb 64f4cb4f3e
refactor: dag-menu refactor (#11965)
* refactor: dag-menu refactor

* feat: add i18n support for the dag-menu

* feat: add e2e class

Co-authored-by: devosend <devosend@gmail.com>
Co-authored-by: Tq <tianqitobethefirst@gmail.com>
2022-09-26 11:06:39 +08:00
Jiajie Zhong 82ddd72e4a
[Feature][API] New restful API for workflow and schedule (#11912)
* [feat] New restful API for workflow and schedule

CURD for workflow and schedule, different with exists
API, this new restful api only operate single resource
in each request, and return the latest. For example,
previous workflow should also need to post tasks definition
and tasks relation definition, but this patch will allow
you to create workflow without task relate information

* use checkProjectAndAuthThrowException, and fix CI error

* Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ScheduleV2Controller.java

* change method name
from createProcessDefinitionV2 to createSingleProcessDefinition
from updateProcessDefinitionV2 to updateSingleProcessDefinition

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-09-23 22:25:19 +08:00
rickchengx 08a4c7981f
[improve] Remove the spark version of spark task (#11860) 2022-09-21 20:27:28 +08:00
limaiwang ed1d1e8856
add(dependent node's add unprivileged project list): dependent node's add unprivileged project list (#12073) 2022-09-21 09:28:22 +08:00
juzimao c8884e4f2f
and task instance date range search (#11963) 2022-09-19 17:50:24 +08:00
insist777 5b384f3fab
[Feature-11473][Task]Support test task (#11670)
* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* [improve] support test tasks

* Update TaskExecuteThread.java

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* try solve e2e q

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update DataSource.java

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update messages_zh_CN.properties

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* Update messages.properties

* Update messages_en_US.properties

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks

* [Feature] support test tasks
2022-09-17 13:40:56 +08:00
Wenjun Ruan 864a90820d
Fix insertOrUpdate plugin may failed due to concurrent operation (#11471) 2022-09-16 15:39:04 +08:00
Wenjun Ruan d833a28b2e
Fix cannot save processDefinition (#11931) 2022-09-16 13:54:18 +08:00
Wenjun Ruan 4ad34483a5
Fix alert_send_status may throw duplicate key exception and add limit for query alert (#11953) 2022-09-16 13:26:33 +08:00
caishunfeng 925e2fa551
[Bug][Workflow Definition] fix ordinary users can not create depend task (#11961) 2022-09-16 09:12:33 +08:00
wendongdi e70c501f9a
[Fix][db] fix init&upgrade mysql-meta-schema bugs (#11887) 2022-09-14 18:03:07 +08:00
juzimao b1068e27ba
Optimize date range condition (#11911) 2022-09-14 16:55:10 +08:00
Wenjun Ruan 8a47785f42
Fix listing processDefinition the schedulerReleaseState will never be null (#11888) 2022-09-13 10:07:12 +08:00
Yann Ann e20f17a7b7
[Improvement-#11768][Monitor] Support monitor h2 database in monitor page (#11813) 2022-09-09 09:43:56 +08:00
Stalary 37325b4c34
[Bug](dependent) Dependent downstream trigger error when schedule cycle not day. (#11734)
* FIX: dependent

* FIX: version

* MOD: for review
2022-09-08 15:08:10 +08:00
caishunfeng 2e61c76c22
[Improvement] Add remote task model (#11767) 2022-09-08 13:41:36 +08:00
kezhenxu94 efedc9a53c
Bump up dependencies to fix cves (#11765) 2022-09-08 09:05:05 +08:00
fuchanghai 7c0c1f4b78
Update dolphinscheduler_ddl.sql fix alert t_ds_user add time_zone is not correct (#11795) 2022-09-06 23:14:41 +08:00
Tq 4283cfd7a1
[fix][API]fix task types sequence (#11775)
* fix task types sequence

* fix mapper table name t_ds_fav_task

* fix sequence
2022-09-05 20:29:03 +08:00
caishunfeng f8d46a26c1
[Feature-11530] add state history for process instance (#11757)
* [Feature] add state history for process instance (#97)

* add state history for process instance

* upsertProcessInstance

* remove unuse method

* fix UT

Co-authored-by: caishunfeng <534328519@qq.com>
2022-09-03 10:14:05 +08:00
Tq b3cc8a55b9
[feature][API]feature: add task type list and enable user to add task type to fav (#11727)
* feature: add task type list and enable user to add task type to fav

* make DependentExecute.java variable protected

* add standalone server fileSet
2022-09-02 11:40:23 +08:00
juzimao d0f5e7edf3
optimize the process instance query, change the date time range (#11719) 2022-08-31 19:53:41 +08:00
seedscoder 145e441afb
[Fix-11633] The data type of the PostgreSQL database column is wrong. (#11633) (#11634) 2022-08-29 16:33:16 +08:00
SongTao Zhuang 6a5367fd57
[Improvement-11662][dao] fix foreach null items in mapper.xml 2022-08-28 20:58:12 +08:00
JiPeng Wang 961e67f524
[Improvement][User] Query all users filter un enabled. (#11504)
* user query all filter un enabled
* restore permissions
2022-08-24 11:55:18 +08:00
longtb 4893bef5a7
[Improvement][TaskInstance] reduce database queries (#11522)
* [Improvement][TaskInstance] reduce database queries

* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/TaskInstanceMapper.java

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>

* [Improvement][TaskInstance] queryByInstanceIdsAndCodes -> queryByProcessInstanceIdsAndTaskCodes

Co-authored-by: zhangshunmin <zhangshunmin@kezaihui.com>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-08-23 14:27:26 +08:00
insist777 abfef1a929
[improvement] Add two parameters in workergroup, and support the application of description display and other parameters (#11542)
* [Improvement] Improvement default worker group

* [Improvement] Improvement default worker group

* [Improvement] Improvement default worker group

* [Improvement] Improvement default worker group

* [Improvement] Improvement default worker group

* [improvement] The default worker grouping supports editing

* [improvement] The default worker grouping supports editing

* Update WorkerGroupServiceImpl.java

* Update WorkerGroupServiceImpl.java

* The default worker grouping supports editing

* update sql file

* update

* update

* update

* update
2022-08-23 10:54:17 +08:00
kezhenxu94 0b82755796
Fix some resource leak (#11576) 2022-08-21 01:19:16 +08:00
Wenjun Ruan 2862f5b67c
Set process instance and task priority default value as 2 (#11539) 2022-08-18 16:21:12 +08:00
Jiajie Zhong 5983e41f4b
[fix] Database change not sync (#11258)
* [fix] Database change not sync

* Make del sql upgrade same as release version
* Correct data quality dml sql
* Upgrade database DDL to avoid log path too long error

* Add missing if exists
2022-08-17 10:50:34 +08:00
juzimao f3d28112ce
optimize sorting priority for workflow define (#11462) 2022-08-13 13:28:35 +08:00
insist777 8217784044
[Fix-1360]: Fix Description length check (#11371)
* [Fix-1360]: Fix Description length check

* Update dolphinscheduler_h2.sql

h2 Increase the length of some fields

* Update dolphinscheduler_ddl.sql
2022-08-11 13:26:50 +08:00
caishunfeng 0464123c2b
[Feature-11223] support stream task (#11350)
* add task execute type

* update task definition list paging

* update task instance list paging

* stream task start

* [Feature][UI] Some changes to execute task.
    * Set the connection edge to dashed line.
    * Add FLINK_STREAM task.

* add stream task

* flink savepoint and cancel

* fix query bug

* add stream task definition

* add task instance for stream task

* delete stream task definition state

* update api for stream task definition edit

* modify search for stream task instance

* add language

* delete task type search for stream task definition

* change task type search for stream task instance

* add jump button

* add savepoint

* add down log for stream task instance

* ui test

* stream task start

* run DAG

* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.

* [Feature][UI] Remove some fields for FLINK_STREAM.

* add start modal

* add dryRun column for stream task instance

* fix duration

* fix pon

* fix build error

* Add success tip

* add auto sync for stream task instance

* remove forgien key for task instance

* license header

* UT fix

* modify locales

* recover common config

* fix UT

* add doc

Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
2022-08-10 21:44:43 +08:00
Wenjun Ruan 8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus (#11340) 2022-08-10 11:00:23 +08:00
Wenjun Ruan acd3d3fab1
Fix TaskGroup cannot work and will cause master dead loop (#11254)
* Fix TaskGroup cannot work and will cause master dead loop

* Remove acquireTaskGroupAgain in ProcessServiceImpl
2022-08-04 14:43:45 +08:00
Wenjun Ruan 8e21c38c00
Write alert result into db (#11221) 2022-08-01 13:04:27 +08:00
zhuxt2015 4d9db3451e
[Bug] [API] If the single log length is long, it will causes the View Log page to hang (#10612) 2022-07-26 10:14:45 +08:00
jackfanwan ccb7641468
修改数据质量查询的权限判断 (#10977)
Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>
2022-07-25 09:35:53 +08:00
WangJPLeo fd59f0bb32
[Fix-11031] AccessToken can only be used by the creator. (#11032)
* AccessToken can only be used by the creator.

* supplement ut

* checkstyle fix
2022-07-20 16:32:03 +08:00
Wenjun Ruan 5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version (#11025) 2022-07-20 10:37:31 +08:00
Jiajie Zhong 81930e5420
[python] Fix tasks with multiple upstream and workflow query error (#10941)
* when task with more than one upstreams, mapper
   TaskDefinitionMapper method queryByName will return
   more than one record, and failed the mybatis result
   type, so we have to add `limit 1` to it to
* add multiple runs of example in integrate test
* Change from subprocess.Popen to subprocess.call_check
  in integrating test which will raise an error when failed
2022-07-14 22:06:07 +08:00
Assert 4ff8eaa996
[Fix-10595][api] Fix Dependent Node Only View Projects Created By Self (#10834)
* [Fix-10595][api] Fix Dependent Node Only View Projects Created By Self
2022-07-14 19:03:34 +08:00
SongTao Zhuang 99a134febb
fix: typo task definition. (#10958) 2022-07-14 11:40:46 +08:00
WangJPLeo 9f34a837b8
[Fix-10909] Returns executionStatus for each process instance status and access token module support. (#10922)
* Returns executionStatus for each process instance status and access token module support
2022-07-13 20:45:52 +08:00
Jiajie Zhong 0db9bbd538
[fix] delete workflow or task used by other tasks (#10873)
currently, users can delete process definitions used
in other sub-process tasks or in other dependent tasks.
but this change will break the dependence of those task
and failed DAG, this patch add validation of those
delete behavior to avoid this error
2022-07-12 21:22:06 +08:00
Jiajie Zhong a6154220dc
[fix] update queue and tenant with same code error (#10877)
in ae6aa53f I use the same logic test to create or update
tenant and queue which failed create with the same value,
this patch will fix it. and also I use the constructor to reduce
the new object logic here
2022-07-11 11:06:30 +08:00
Wenjun Ruan f639a2eed4
[Fix-10854] Fix database restart may lost task instance status (#10866)
* Fix database update error doesn't rollback the task instance status

* Fix database error may cause workflow dead with running status
2022-07-11 09:57:00 +08:00
WangJPLeo 3756a233d8
[Fix-10863] Command execution result data statistics remove userId condition. (#10864)
* Command execution result data statistics remove userId condition.

* e2e rerun
2022-07-10 17:17:35 +08:00
Jiajie Zhong ae6aa53f96
[python] Fix submit and run error (#10792)
because we add permission, so it failed out
createQueue method. this patch fix it and
do some refactor of our tenant and queue
validator code
2022-07-08 10:54:41 +08:00