natas_hw
/
dolphinscheduler
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6,854 Commits 27 Branches 54 Tags 416 MiB
Commit Graph

107 Commits

Author SHA1 Message Date
Kerwin ea8b00fa89
[Fix-11051][Task]Fix the process exitValue exception when the process times out and th… (#11099) (#11983) 2022-09-16 19:03:01 +08:00
Kerwin 2e64e1ca2f
cherry-pick [Bug] [spark-sql] In spark-sql, select both SPARK1 and SPARK2 versions and execute /bin/spark-sql (#11971) 2022-09-16 13:01:03 +08:00
fuchanghai 15506a8785
[fix-#11815] fix ck column names contain special characters (#11973) 
Co-authored-by: fuchanghai <changhai.fu@marketingforce.com>
 2022-09-16 11:41:20 +08:00
Kerwin 350a3632d3
cherry-pick [fix-10938]: use dot to replace source and make the default env work in shell (#11937) 2022-09-15 15:05:39 +08:00
Jiajie Zhong 889fd580e3 [maven-release-plugin] prepare for next development iteration 2022-08-05 18:16:57 +08:00
Jiajie Zhong 9efd1ace78 [maven-release-plugin] prepare release 3.0.0 2022-08-05 18:16:57 +08:00
xuhhui d550f9917e fix error (#11206) 
(cherry picked from commit bfff3a7c5d)
 2022-08-02 18:57:18 +08:00
caishunfeng a8b1b61bd2 [Bug-11071] fix local params (#11184) 
* fix local params

* add CI case

(cherry picked from commit 818648df7d)
 2022-08-02 18:49:53 +08:00
caishunfeng cec5e88319
[Bug-11071] Fix task local params NPE (#11166) 
* fix local param NPE

* delete empty files
 2022-07-28 09:43:58 +08:00
Wenjun Ruan db31deb54f [Bug] [Master] Worker failover will cause task cannot be failover (#10631) 
* fix worker failover may lose event

(cherry picked from commit 66624c5c86)
 2022-07-19 11:41:25 +08:00
Wenjun Ruan 3ab9ee13fc Optimize master log, use MDC to inject workflow instance id and task instance id in log (#10516) 
* Optimize master log, add workflow instance id and task instance id in log

* Use MDC to set the workflow info in log4j

* Add workflowInstanceId and taskInstanceId in MDC

(cherry picked from commit db595b3eff)
 2022-07-19 11:29:41 +08:00
Wenjun Ruan 4ceb420873 Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe (#10479) 
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe

(cherry picked from commit ad2646ff1f)
 2022-07-19 11:02:57 +08:00
Tq 1543cdd1b5 [bug][task-plugins]fix switch class name and add param check to the switch task plugin (#10894) 
* fix switch class name and add param check to the switch task plugin
 2022-07-18 17:13:56 +08:00
zhuxt2015 cb14726bdd [Bug] [Task] SQL task no exception stack trace in task log (#10656) 2022-07-18 16:32:59 +08:00
devosend 7ddaa2f47d [maven-release-plugin] prepare for next development iteration 2022-06-28 16:41:31 +08:00
devosend 0a1b9bdd52 [maven-release-plugin] prepare release 3.0.0-beta-2 2022-06-28 16:41:30 +08:00
devosend d68dcda2bb [chore] pre-release change pom.xml 2022-06-28 08:24:21 +08:00
Jiajie Zhong 57ade38939 [maven-release-plugin] prepare release 3.0.0-beta-1 2022-05-26 20:26:32 +08:00
Dannila d74aa47196 [Fix-10039] Flink run command when perfecting Python jobs (#10042) 
* [fix] flink task

* [fix] flink task

(cherry picked from commit d643e1c1cf)
 2022-05-16 20:17:26 +08:00
zixi0825 25538aaf67 [dataquality] Fix task commnd null bug (#9974) 
(cherry picked from commit 359cbe2a1e)
 2022-05-16 20:16:36 +08:00
chuxing 44c6a053e7 [fix-9991][worker]fix statement is closed before resultSet.getMetaData() (#10014) 
(cherry picked from commit df04c4aef0)
 2022-05-16 20:15:46 +08:00
Jiajie Zhong a9fa6b33a4
[chore] Change release version to 3.0.0-beta-1 (#9957) 2022-05-12 21:49:21 +08:00
Dannila 2d36449444
[Feature- 9837][plugin/ui] support FlinkSQL Task (#9840) 
* flink_sql

* '[refactor]flinksql'

* '[refactor]flinksql'

* '[refactor]flinksql'

* '[refactor]flinksql'

* '[refactor]flinksql'

* '[refactor]flinksql'

* '[refactor]flinksql'

* '[refactor]flinksql'

* '[refactor]flinksql'

* '[refactor]flinksql'

* '[refactor]flinksql'

* '[refactor]flinksql'

* [refactor] flinksql

* '[refactor]flinksql'

* [refactor] dolphinscheduler flinksql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* [refactor] flink sql

* Update docs/docs/en/guide/task/flink.md

* Update docs/docs/zh/guide/task/flink.md

* prettier front-end code

Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
Co-authored-by: devosend <devosend@gmail.com>
 2022-05-09 18:51:55 +08:00
Kerwin 0e7c98d465
[Feature-9177][Task] The sql task supports configuring segmentation notation to provide execution of multiple statements (#9917) 
* Support sql segmentation to execute multiple functions.
 2022-05-08 15:04:35 +08:00
sq-q ed425d2d2d
[Feature-9772][plugin/ui] support SparkSQL Task (#9790) 
* [refactor] Dolphinscheduler sparkSQL

* [refactor] Dolphinscheduler SparkSQL

* [refactor] Dolphinscheduler SparkSQL

* [refactor] Dolphinscheduler SparkSQL

* [refactor] Dolphinscheduler SparkSQL

* [refactor] Dolphinscheduler SparkSQL

* [refactor] Dolphinscheduler SparkSQL

* [refactor] Refactor ui code and add sparksql test cases

* [refactor] refactor dolphinscheduler SparkSQL

* [refactor] refactor dolphinscheduler plugin-sparkSQL

* [refactor] refactor dolphinscheduler plugin-SparkSQL

* [refactor] dolphinscheduler plugin-SparkTaskTest

* [refactor] dolphinscheduler plugin-SparkTask

* [refactor] dolphinscheduler plugin-Spark

* [refactor] dolphinscheduler plugin-SparkTask-SparkSQL

* [refactor] dolphinscheduler plugin-spark-SparkTask

* [refactor] dolphinscheduler plugin-spark-SparkTask redefine code
 2022-04-27 17:44:05 +08:00
zhuxt2015 206b7c1c51
[Fix-9352][datasource-api]data source client uses a cache with an expiration time (#9353) 
* datasource client cache with expiration time

* remove unused import

* fixed unit test

* fix

* fix

* fix

* fix

* fix

* fix

* fix
 2022-04-27 16:56:24 +08:00
exmy 36f01155b5
[Improvement][server] varPool support syntax #{setValue(key=value)} (#9586) 2022-04-22 14:10:21 +08:00
Tq c5b7e5adff
[Bug] [API-9558]fix homepage task instance count method to use submit time to recount (#9559) 
* fix homepage task instance count method to use submit time to recount

* fix homepage task instance count method to use submit time to recount

* fix homepage task instance count method to use submit time to recount

* fix homepage task instance count method JUNIT

* fix homepage task instance count method JUNIT

* fix homepage task instance count method JUNIT
 2022-04-19 15:23:57 +08:00
Eric Gao a5bbf7852d
[Feature][Task-Plugin]Add zeppelin task-plugin to support Apache Zeppelin (#9327) 2022-04-17 22:10:10 +08:00
xiangzihao 7f41a96fc1
[Fix-9525] [Worker] Environment did not work as expected (#9527) 
* fix #9525

* change to ${PYTHON_HOME}

* remove import

* fix ut error
 2022-04-16 18:57:33 +08:00
Tq 04d4e4e0c4
fix the OUT direct param could pass to the current script (#9463) 2022-04-13 10:43:12 +08:00
caishunfeng b285ccf930
[Future-9396]Support output parameters transfer from parent workflow to child work flow (#9410) 
* [Future-9396]Support output parameters transfer from parent workflow to child work flow

* fix note
 2022-04-11 20:03:16 +08:00
Kerwin 923f3f38e3
[Fix-9316] [Task] Configure DB2 data source SQL script execution report ResultSet has been closed exception in SQL task (#9317) 
* fix db2 error in the sql task

* update limit in sql task
 2022-04-11 13:19:48 +08:00
nobolity 037692517a
[Fix-9251] [WORKER] reslove the sql task about of add the udf resource failed (#9319) 
* feat(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config

* fix(resource  manager): update the dependency

* fix(resource  manager): extend s3 to the storage of ds

fix the constant of hadooputils

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager):

merge  the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl

* fix(resource  manager): extend s3 to the storage of ds

merge the resourceServiceTest

* fix(resource  manager): test  cancel the test method

createTenant verifyTenant

* fix(resource  manager): merge the code  follow the check-result of sonar

* fix(resource  manager): extend s3 to the storage of ds

fit the spell question

* fix(resource  manager): extend s3 to the storage of ds

revert the common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the storageConfig with None

* fix(resource  manager): extend s3 to the storage of ds

fix the judge of resourceType

* fix(resource  manager): extend s3 to the storage of ds

undo the compile-mysql

* fix(resource  manager): extend s3 to the storage of ds

delete hadoop aws

* fix(resource  manager): extend s3 to the storage of ds

update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the aws-region

* fix(resource  manager): extend s3 to the storage of ds

fix the storageconfig init

* fix(resource  manager): update e2e docker-compose

update e2e docker-compose

* fix(resource  manager): extend s3 to the storage of ds

revent the e2e common.proprites

print the resource type in propertyUtil

* fix(resource  manager): extend s3 to the storage of ds
1.println the properties

* fix(resource  manager): println the s3 info

* fix(resource  manager): extend s3 to the storage of ds

delete the info  and upgrade the s3 info to e2e

* fix(resource  manager): extend s3 to the storage of ds

add the bucket init

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): extend s3 to the storage of ds

upgrade the s3 endpoint

* fix(resource  manager): withPathStyleAccessEnabled(true)

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): upgrade the  s3client builder

* fix(resource  manager): correct  the s3 point to s3client

* fix(resource  manager): update the constant BUCKET_NAME

* fix(resource  manager): e2e  s3 endpoint -> s3:9000

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* style(resource  manager): add info to createBucket

* style(resource  manager): debug the log

* ci(resource  manager): test

test s3

* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql

* fix(resource  manager): update the h2 sql

* fix(resource  manager): solve to delete the tenant

* style(resource  manager): merge the style end delete the unuse s3 config

* fix(resource  manager): extend s3 to the storage of ds

UPDATE the rename resources when s3

* fix(resource  manager): extend s3 to the storage of ds

1.fix the code style of QuartzImpl

* fix(resource  manager): extend s3 to the storage of ds

1.impoort restore_type to CommonUtils

* fix(resource  manager): update the work thread

* fix(resource  manager): update  the baseTaskProcessor

* fix(resource  manager): upgrade dolphinscheduler-standalone-server.xml

* fix(resource  manager): add  user Info to dolphinscheduler_h2.sql

* fix(resource  manager): merge  the resourceType to NONE

* style(upgrade the log level to info):

* fix(resource  manager): sysnc the h2.sql

* fix(resource  manager): update the merge the user tenant

* fix(resource  manager): merge the resourcesServiceImpl

* fix(resource  manager):

when the storage is s3 ,that the directory can't be renamed

* fix(resource  manager): in s3 ,the directory cannot be renamed

* fix(resource  manager): delete the deleteRenameDirectory in E2E

* fix(resource  manager): check the style and  recoverd the test

* fix(resource  manager): delete the log.print(LoginUser)

* fix(server): fix the  udf serialize

* fix(master  task): update the udfTest to update the json string

* fix(test): update the udfFuncTest

* fix(common): syn the common.properties

* fix(udfTest): upgrade the udfTest

* fix(common): revent the common.properties
 2022-04-11 10:49:46 +08:00
Tq 2a4fa9cdb1
[BUG][WORKER-9349]fix param priority (#9379) 
* fix param priority

* fix params priority code logic
 2022-04-07 20:09:53 +08:00
Eric Gao fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT (#9223) (#9299) 2022-04-07 18:15:19 +08:00
caishunfeng ddf1ff98fa
[Improvement-7697][Master/Worker] Change the task ack to runnning callback (#8719) 
* rebase dev

* change task state to dispatch if dispatch success

* update task host when dispatch

* add dispatch task event

* test

* check tenant after enable auto create

* handle dispatch state

Co-authored-by: caishunfeng <534328519@qq.com>
 2022-03-30 17:35:05 +08:00
Kerwin d7d756e7b0
Fix DataX and Sqoop Task execute NullPointerException (#9192) 2022-03-26 20:15:41 +08:00
nobolity 0e3cafec1d
[Feature-8612][RESOURCE] extend s3 to the storage of ds (#8637) 
* feat(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config

* fix(resource  manager): update the dependency

* fix(resource  manager): extend s3 to the storage of ds

fix the constant of hadooputils

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager):

merge  the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl

* fix(resource  manager): extend s3 to the storage of ds

merge the resourceServiceTest

* fix(resource  manager): test  cancel the test method

createTenant verifyTenant

* fix(resource  manager): merge the code  follow the check-result of sonar

* fix(resource  manager): extend s3 to the storage of ds

fit the spell question

* fix(resource  manager): extend s3 to the storage of ds

revert the common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the storageConfig with None

* fix(resource  manager): extend s3 to the storage of ds

fix the judge of resourceType

* fix(resource  manager): extend s3 to the storage of ds

undo the compile-mysql

* fix(resource  manager): extend s3 to the storage of ds

delete hadoop aws

* fix(resource  manager): extend s3 to the storage of ds

update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the aws-region

* fix(resource  manager): extend s3 to the storage of ds

fix the storageconfig init

* fix(resource  manager): update e2e docker-compose

update e2e docker-compose

* fix(resource  manager): extend s3 to the storage of ds

revent the e2e common.proprites

print the resource type in propertyUtil

* fix(resource  manager): extend s3 to the storage of ds
1.println the properties

* fix(resource  manager): println the s3 info

* fix(resource  manager): extend s3 to the storage of ds

delete the info  and upgrade the s3 info to e2e

* fix(resource  manager): extend s3 to the storage of ds

add the bucket init

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): extend s3 to the storage of ds

upgrade the s3 endpoint

* fix(resource  manager): withPathStyleAccessEnabled(true)

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): upgrade the  s3client builder

* fix(resource  manager): correct  the s3 point to s3client

* fix(resource  manager): update the constant BUCKET_NAME

* fix(resource  manager): e2e  s3 endpoint -> s3:9000

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* style(resource  manager): add info to createBucket

* style(resource  manager): debug the log

* ci(resource  manager): test

test s3

* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql

* fix(resource  manager): update the h2 sql

* fix(resource  manager): solve to delete the tenant

* style(resource  manager): merge the style end delete the unuse s3 config

* fix(resource  manager): extend s3 to the storage of ds

UPDATE the rename resources when s3

* fix(resource  manager): extend s3 to the storage of ds

1.fix the code style of QuartzImpl

* fix(resource  manager): extend s3 to the storage of ds

1.impoort restore_type to CommonUtils

* fix(resource  manager): update the work thread

* fix(resource  manager): update  the baseTaskProcessor

* fix(resource  manager): upgrade dolphinscheduler-standalone-server.xml

* fix(resource  manager): add  user Info to dolphinscheduler_h2.sql

* fix(resource  manager): merge  the resourceType to NONE

* style(upgrade the log level to info):

* fix(resource  manager): sysnc the h2.sql

* fix(resource  manager): update the merge the user tenant

* fix(resource  manager): merge the resourcesServiceImpl

* fix(resource  manager):

when the storage is s3 ,that the directory can't be renamed

* fix(resource  manager): in s3 ,the directory cannot be renamed

* fix(resource  manager): delete the deleteRenameDirectory in E2E

* fix(resource  manager): check the style and  recoverd the test

* fix(resource  manager): delete the log.print(LoginUser)
 2022-03-25 13:28:13 +08:00
Kerwin 25fc1dcb5f
Fix the exception of task getting external resource data during execution (#9134) 2022-03-23 19:00:09 +08:00
caishunfeng 852597c9bf
[Bug-9070][Server]fix taskExecutionContext json convert error (#9072) 
* [Bug-9070][Server]fix taskExecutionContext json convert error

* test

* add license header

Co-authored-by: caishunfeng <534328519@qq.com>
 2022-03-22 14:02:07 +08:00
zixi0825 63a10037a6
fix package bug (#9051) 2022-03-21 19:57:03 +08:00
Kerwin a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module (#8584) 
* Task plug-in optimization, task-related classes are migrated to the task plugin module

* remove tasktype enum class.

* fix code style.

* fix code style

* fix code style

* fix ProcessServiceTest.testUpdateResourceInfo test class

* fix code style

* fix test task plugin manager check parameters null

* fix test ut

* update aws license

* change TaskTimeoutStrategy class
 2022-03-15 15:16:36 +08:00
zixi0825 6a02b15e40
[Fix][data-quality] change data-quality task main jar path (#8752) 
* feat(data-quality) change data-quality task main jar path

* feat(data-quality) change data-quality task main jar path

Co-authored-by: zixi0825 <sunzhaohe@linklogis.com>
 2022-03-09 21:49:19 +08:00
ronyang1985 20ee9a37fe
[Feature-8485][task] Add EMR task plugin to support submit task to AWS EMR (#8503) 
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
 2022-03-01 11:43:37 +08:00
yimaixinchen a72742b9de
remove unuse import jsonFormat (#8548) 2022-02-25 22:05:05 +08:00
caishunfeng 84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified (#8477) 
* date convert of timezone

* remove @JsonFormat

* add unit test

* fix time preview in scheduler

* optimization & add env config

Co-authored-by: caishunfeng <534328519@qq.com>
 2022-02-24 18:20:34 +08:00
Yao WANG dd1397aaed
[Bug] [Task-plugin] ShellTask add the parentfolder exist check and create if not exit (#8526) 
* check the parent dir exist and create it

* fix lint

Co-authored-by: ywang46 <ywang46@paypal.com>
 2022-02-24 17:36:54 +08:00
Ivan0626 60ddede6a6
fixbug: varPool value is error (#7724) 2022-02-23 14:53:20 +08:00
ShuiMuNianHuaLP 806333a11c
fix sudo.enable=false Is invalid (#8388) 2022-02-17 15:23:21 +08:00