b12df72a0c
[Fix][Resource Center] Remove NONE storage type and set LOCAL type as default ( #14661 )
2023-09-07 14:09:15 +08:00
55cc77226f
[RPC] Refactor the remote module. ( #14776 )
...
* Add dolphinscheduler-extract-base
* Add dolphinscheduler-extract-master
* Add dolphinscheduler-extract-worker
* Add dolphinscheduler-extract-alert
2023-08-23 09:43:17 +08:00
70731a1617
[Feature][Authentication] support oauth2 login ( #14743 )
...
* support oauth2 login
---------
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
2023-08-21 15:57:30 +08:00
25480ae9e7
[Feature][Resource Center] Add support for Huawei Cloud OBS ( #14643 )
...
* [Feature][Resource Center] Add support for Huawei Cloud OBS as storage of resource center
* add license and add doc
* add 3-party dependency license
* Update LICENSE
* fix
* Update pom.xml
* fix
* fix
---------
Co-authored-by: sunkang <sunkang@hjfruit.com>
Co-authored-by: xiangzihao <460888207@qq.com>
Co-authored-by: Rick Cheng <rickchengx@gmail.com>
2023-08-14 13:25:48 +08:00
93b0283dfe
Set tenant as the the resource file owner ( #13832 )
2023-08-11 00:57:47 +08:00
9ce8871158
Support get property from env/jvm/common.properties ( #14645 )
2023-07-28 14:41:58 +08:00
2b99451ccd
Support filter restrict network interface ( #14638 )
2023-07-26 23:03:58 +08:00
cbd354d134
Add host/port in heartbeat ( #14591 )
2023-07-20 21:26:35 +08:00
1d0d85b988
[Feature] doris add datasource ( #14391 )
...
* Added doris data source plug-in
* Update DbType.java
* XML configuration
* doc and ui
* Missing catch of NumberFormatException
* code format
---------
Co-authored-by: xiangzihao <460888207@qq.com>
2023-07-20 17:15:22 +08:00
c30cca9d9a
Support execute shell in different interceptor ( #14582 )
2023-07-19 14:50:04 +08:00
399d0d0ce9
Add log for NetUtils ( #14578 )
2023-07-18 21:17:17 +08:00
9d10de6efa
[Improvement][Task] Mask configYaml in task log ( #14571 )
...
* [Improvement][Task] Mask configYaml in task log
Signed-off-by: Gallardot <gallardot@apache.org>
---------
Signed-off-by: Gallardot <gallardot@apache.org>
Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
2023-07-18 15:02:20 +08:00
e6d94632b9
Add WorkflowExecuteContext ( #14544 )
2023-07-14 18:40:01 +08:00
175d976cd2
[Feature][Data Quality] Data Quality Support Choose Database ( #14406 )
...
* add dataquality database api
* change ui
* api change
* update
* fix spotless
* fix h2
* fix pg
* fix-dead-line
* update
* fix-spotless
* update pg sql
* add ut
* fix ut
2023-07-12 09:00:28 +08:00
76b1eefb68
[Feature-14321][API] Support to complement data in descending or ascending order of date ( #14450 )
2023-07-05 14:35:18 +08:00
6f90156577
oss avoid listBuckets permission in bucket check ( #14414 )
...
---------
Co-authored-by: kuantian.zhang <kuantian.zhang@inceptio.ai>
2023-06-28 14:21:17 +08:00
7a24be0672
[Feature-14404][datasource-plugin] add HANA datasource ( #14404 )
...
---------
Co-authored-by: WeiJian Kong <>
Co-authored-by: xujiaqiang <Xjq634828243>
2023-06-28 10:56:41 +08:00
4fa0ab4d7e
[feature][Resource] Add support for Azure Blob Storage in the resource center ( #14082 )
2023-06-12 12:05:49 +08:00
66e09dc66e
[Improvement-14295][common] Remove the duplicate FileUtils ( #14301 )
2023-06-08 08:59:53 +08:00
e7cdc7c136
[Feature][Task Plugin] Add Dynamic task to generate dynamic worklfows with list parameters ( #14127 )
...
* add dynamic task plugin
* fix ui
* update code
* add DOC
* ADD UT
* fix doc
* fix sanity check
* add upgrade ddl
* fix sql
2023-06-07 16:01:06 +08:00
2c057d6a8f
[Improvement-14292][Common] Remove the duplicate OSUtils ( #14293 )
2023-06-07 14:59:25 +08:00
18d176c5f4
feat: Introduce new datasource vertica ( #14232 )
...
Co-authored-by: Zinway Liu <zinway.liu@ucloud.cn>
2023-06-02 13:57:37 +08:00
43d7b35977
[fix-14119] The checkBucketNameExists method should not use s3Client. listBuckets() ( #14188 )
...
* optimize checkBucketNameExists
* fix code style
---------
Co-authored-by: 詹子恒 <ziheng.zhan@longbridge.sg>
2023-05-24 13:43:44 +08:00
5e87b2d58a
[Bug][Worker] generate kill command with unknown user: default ( #14145 )
2023-05-22 20:48:40 +08:00
f1e78627fc
[feature][plugin] Add snowflake datasource in datasource plugin ( #13729 )
...
---------
Co-authored-by: weitong <weitong@cisco.com>
2023-05-06 19:30:50 +08:00
Bingchang Chen
77e58ce245
[Feature] Add SSO login by Casdoor ( #8706 )
2023-05-05 14:43:06 +08:00
cf91544b99
Refactor logic task ( #13948 )
2023-04-26 10:01:51 +08:00
db6c061c8b
[Feature][plugin] Add databend datasource in datasource plugin ( #13866 )
...
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
2023-04-24 21:28:14 +08:00
ec29220977
[improve-#13966] mask datax log ( #13969 )
...
Co-authored-by: fuchanghai <‘2875334588@qq.com’>
2023-04-21 14:51:37 +08:00
c0126b7f30
[Feature][plugin] Support Kyuubi datasource ( #13642 )
...
---------
Co-authored-by: weitong <weitong@cisco.com>
Co-authored-by: Rick Cheng <38122586+rickchengx@users.noreply.github.com>
2023-04-18 11:12:39 +08:00
61a689aa5a
Use percentage to represent memory/cpu usage ( #13896 )
2023-04-13 15:48:20 +08:00
7ee66f2d02
[Feature-13429][Remote Logging] Add support for writing task logs to Google Cloud Storage ( #13777 )
2023-04-06 17:49:33 +08:00
41a8ba9aab
Support alert server HA ( #13865 )
...
* Support alert server cluster
* Remove WORKER_ALERT_LISTEN_HOST in statefulset-dolphinscheduler-worker.yaml
2023-04-05 17:31:50 +08:00
69fc217ab2
Fix trim might cause NPE in PropertyUtils ( #13853 )
2023-04-04 12:22:34 +08:00
9df89cf0cf
Use bootstrap user as the default tenant ( #13825 )
2023-03-30 14:11:23 +08:00
85605bbef9
[Improvement-13653][Remote Logging] Support api-server to get task log from remote target ( #13654 )
2023-03-30 12:07:58 +08:00
1ac76e1cbb
[Improvement]update spelling mistake for Constants ( #13738 )
...
* [Improvement][Document]Update architecture and process-start-flow pic
* [Improvement]update spelling mistake for Constants
---------
Co-authored-by: xuhaihui <xuhaihui@cmss.chinamobile.com>
2023-03-16 16:52:17 +08:00
2dceb6b8ca
[Feature-13428][Resource Center] Add support for Google Cloud Storage in the resource center ( #13562 )
2023-03-15 19:29:27 +08:00
aa79c01ce9
[Feature-13419][Remote Logging] Add support for writing task logs to S3 ( #13649 )
...
* [Feature-13419][Remote Logging] Add support for writing task logs to S3
* Update dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/log/remote/S3RemoteLogHandler.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
* use try with resource
* remove unnecessary log
---------
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2023-03-13 19:36:12 +08:00
35d8e8ac17
[Feature][DataSource] add oceanbase datasource ( #13675 )
...
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
2023-03-07 10:12:58 +08:00
7bf3e3cdd6
[improve-#13201] update pid during running ( #13206 )
2023-02-22 14:22:49 +08:00
047fa2f65e
[Feature-13511] Submit Spark task directly on Kubernetes ( #13550 )
2023-02-21 23:26:21 +08:00
701d67c831
[improve-#13045] after a submit failure, stop the processInstance to avoid an endless loop ( #13051 )
...
* [improve-#13045] add max submit number of workflow
* [fix-13405] remove max times
* Update dolphinscheduler-master/src/main/resources/application.yaml
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
---------
Co-authored-by: fuchanghai <‘2875334588@qq.com’>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
2023-02-20 14:10:59 +08:00
2bd65fb2df
[Feature][Remote Logging] Add support for writing task logs to OSS ( #13332 )
2023-02-13 16:38:26 +08:00
e368bde7ee
Publish Helm Chart snapshot to ghcr.io ( #13546 )
2023-02-11 12:40:35 +08:00
8d12dc0702
[Improvement-13491] Use lombok @Slf4j annotation to generate logger ( #13509 )
2023-02-07 20:32:53 +08:00
b11431db95
Use single thread to refresh kerberos ( #13456 )
2023-02-01 20:41:05 +08:00
12dd60fa46
Fix task wake up failed will block the event ( #13466 )
2023-02-01 15:31:04 +08:00
78e5569afd
[feat] Add type local turn on resource center by default ( #13303 )
2023-01-31 17:39:20 +08:00
78ba71bdbd
[Feature][task]add azure datafactory task type ( #13417 )
...
* init data factory
* add task-type-config.yaml
* add data factory ui
* submit doc
* submit license
---------
Co-authored-by: devosend <devosend@gmail.com>
2023-01-30 11:15:20 +08:00
713fdbc444
Split the upgrade code ( #13436 )
...
1. Split the upgrade code into different version
2. Log the dml/ddl sql in origin friendly format
3. Fix ddl of 3.0.0 and 3.1.0
4. Fix search resource regular expression injection
2023-01-20 20:48:07 +08:00
8be32d4145
[Feature][Api] When use api to run a process we want get processInstanceId ( #13184 )
...
* add sql
* add mapper
* add dao
* add excutor
Co-authored-by: qianl4 <qianl4@cicso.com>
2023-01-18 17:58:32 +08:00
3a6e10397f
[feature][api-datasource]add redshift auth mode to get connection by access key ( #13379 )
...
* submit redshift auth mode
* add front end
* add doc and fix frontend
Co-authored-by: devosend <devosend@gmail.com>
2023-01-16 16:21:26 +08:00
6634ff98af
Add source env list in common.properties ( #13317 )
2023-01-04 20:53:52 +08:00
bcf03ad4d1
[Feature-9467] add DAMENG DataSource ( #12860 )
...
* [Feature-9467][DataSource] add DAMENG （达梦） data source
* [Feature-9467] add DM datasource doc (Issue #9467 )
* [Feature-9467] add DM IDataBaseOptionKeys
* [Feature-9467] add DM DataSource: DmDataSourceChannelFactoryTest remove testGetDmConnection method
* [Feature-9467] modification package dm to dameng;
* [Feature-9467] add Dameng DmJdbcDriver18 License
* [Feature-9467] add DM DataSource: remove DruidDataSourceClient
Co-authored-by: lenian <yuenian@mananacare.com>
2023-01-04 15:00:24 +08:00
ccad56e88e
[Improvement][Master] Validate same content of input file when using task cache ( #13298 )
...
* support file content checksum
* fix inject null storageOperate bug
2023-01-03 11:38:13 +08:00
fd3afd84ad
Remove alert when delete workflow instance ( #13281 )
2022-12-28 11:40:12 +08:00
14ec4a2398
Remove dao module in worker ( #13242 )
2022-12-22 12:25:29 +08:00
f0dd4f081f
[Feature][task-plugins-12820] azure azure-sql datasource plugins ( #13209 )
...
* submit azure sql code
* change mssql-jdbc version
* merge changes to delete other
* merge changes to delete other
* add dependencies
2022-12-19 10:27:03 +08:00
66e20271ad
[Feature][Master] Add task caching mechanism to improve the running speed of repetitive tasks ( #13194 )
...
* Supports task instance cache operation
* add task plugin cache
* use SHA-256 to generate key
* Update dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
* Update dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_postgresql.sql
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
* Optimizing database Scripts
* Optimize clear cache operation
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
2022-12-18 18:17:09 +08:00
e4b9b67255
Allow execute task in workflow instance ( #13103 )
2022-12-13 16:43:44 +08:00
70ccffeee2
Format task parameter as pretty json ( #13173 )
2022-12-13 16:30:21 +08:00
6ef74073cc
[Improve-13001]migrate commons-collections -> commons-collections4 ( #13002 )
2022-12-10 23:50:19 +08:00
1064680ded
[feat][API] New restful API for workflow state ( #13031 )
2022-12-01 14:21:01 +08:00
38b876733c
[Feature-10498] Mask the password in the log of sqoop task ( #11589 )
2022-11-24 14:54:54 +08:00
b40c63f02d
[Feature][DataSource]Add trino datasource plugin ( #12911 )
2022-11-23 10:09:02 +08:00
43d69ae157
Upgrade clickhouse jdbc driver ( #12639 )
2022-11-11 13:38:58 +08:00
ff59acd02f
Splite cpu/memory info in OSUtils#isOverload ( #12663 )
2022-11-02 21:15:03 +08:00
9e0c9af1a5
Fix the waiting strategy cannot recovery if the serverstate is already in running ( #12651 )
2022-11-02 14:06:01 +08:00
b71d4983dd
[Feature-6586][Server]add some ds process definition demo when init ( #11759 )
...
* [Feature-6586][Server]add some ds process definition demo when init
1.add some ds process definition demo when init, to display what task type can run and make user easy to
use ds.
2.need configure the JVM parameters (-Ddemo=true) to turn on the StandaloneServer service
3.need modify the tenant information in it
2022-10-31 22:35:11 +08:00
08335b1032
[Improvement][Task] Improved way to collect yarn job's appIds ( #12197 )
...
* Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code.
* Add user property `appId.collect` for user to decide how to collect applicationId.
* Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh`
* Update docs to declare how to use aop way.
* Update `LogUtils` to support fetch applicationId in different ways based on the user property.
Co-authored-by: gabrywu <gabrywu@apache.com>
2022-10-31 16:52:53 +08:00
a0d37fc7ae
Set tenantDir permission ( #12486 )
2022-10-23 21:54:01 +08:00
dedff70f90
[Improvement] Merge spi.utils into common.utils ( #12502 )
...
* improve JSONUtil
* merge spi.utils into common.utils
* split constants && remove some copy utils method
* remove StringUtils
2022-10-23 21:23:54 +08:00
f0a34cddec
[improvement-#12416] Use CollectionUtils.isEmpty judge the collection isEmpty to avoid NPE ( #12449 )
2022-10-20 14:26:56 +08:00
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports ( #12412 )
...
* Fix remaining style errors and wildcard imports
* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link
* Fix spotless matching scope of markdown files
2022-10-19 11:08:04 +08:00
2f37da0dbc
[Improvement][Test] Block the usage of powermock and move mockito dependencies from sub-modules to root pom ( #12311 )
...
* move mockito dependencies from sub-modules to root pom
* Add check in CI to block the usage of powermock
2022-10-14 12:06:51 +08:00
b357781df2
[Migrate][Test] Migrate junit4 -> junit5 test cases in common, service, and spi modules ( #12363 )
2022-10-14 11:06:29 +08:00
db1d319159
Reorganize some classes in common module, remove duplicate classes ( #12321 )
2022-10-13 10:23:27 +08:00
17cd644506
[Fix-#11669][Workflow Instance Page] Fix the duration in Workflow Instance page. ( #12264 )
...
* fix process duration issue
* use processInstance.getState().isFinished() replace new isFinish function
* add WorkflowUtils
2022-10-13 09:50:31 +08:00
9ab79e064c
[Improvement][Test] Fully remove the usage of powermock from the whole project ( #12244 )
...
* Fully remove the usage of powermock from the whole project
* Upgrade org.reflections to 0.10.12
2022-10-10 10:43:38 +08:00
b4947b471c
fix this issue ( #12281 )
2022-10-10 09:34:32 +08:00
c87f6d89a4
[fix#12195] Change date to timestamp to support cross diff time zones ( #12239 )
...
* fix DS support cross time zone, use timestamp replace date
2022-10-01 12:50:42 +08:00
cf5a8894e2
[BUG][BE-12113]fix htrace dependency conflicts ( #12126 )
...
* fix htrace dependency conflicts
* replace htrace-core with hbase-noop-htrace
2022-09-26 13:49:08 +08:00
289e1ecdc6
[Bug] [Common] update IPv4 Pattern ( #11762 )
...
* [Bug] [Common] IPv4 Pattern is wrong. #11758
2022-09-26 09:10:22 +08:00
08a4c7981f
[improve] Remove the spark version of spark task ( #11860 )
2022-09-21 20:27:28 +08:00
cf522e2fc0
[improve] Optimize MLFlow task plugin for easy of use ( #12071 )
...
optimize code
2022-09-21 10:45:05 +08:00
66958b91d7
[Feature][Resource Center] Add support for Alibaba Cloud OSS as storage of resource center ( #11708 )
...
* Add support for Alibaba Cloud OSS as storage of resource center
* Add dependency licenses
* Refactor code, add unit tests and related docs
2022-09-20 22:44:06 +08:00
5b384f3fab
[Feature-11473][Task]Support test task ( #11670 )
...
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* Update TaskExecuteThread.java
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update DataSource.java
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update messages_zh_CN.properties
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update messages.properties
* Update messages_en_US.properties
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
2022-09-17 13:40:56 +08:00
d833a28b2e
Fix cannot save processDefinition ( #11931 )
2022-09-16 13:54:18 +08:00
71b4087421
Example Change the regular expression of the data source password ( #11459 )
2022-09-16 13:34:22 +08:00
9813ded981
Add debug logs for properties read from `common.properties` ( #11910 )
2022-09-15 10:53:19 +08:00
3ca9680b20
[Fix][Monitor]Monitor UI not show DisakAvailable and MemoryUsage correctly ( #11870 )
2022-09-11 19:11:03 +08:00
3664d85143
[Improve] Upgrade Hadoop to 3.2.4 ( #11832 )
2022-09-09 10:47:03 +08:00
2e61c76c22
[Improvement] Add remote task model ( #11767 )
2022-09-08 13:41:36 +08:00
efedc9a53c
Bump up dependencies to fix cves ( #11765 )
2022-09-08 09:05:05 +08:00
ebcffb04aa
fix hdfs defaultFs not working ( #11823 )
2022-09-07 16:40:43 +08:00
444c7e6eb7
[Improvement] Replace commons-lang 2 function invocations with commons-lang3 ( #11810 )
...
* [Improvement] Replace commons-lang 2 function invocations with commons-lang3
2022-09-07 13:48:09 +08:00
b3cc8a55b9
[feature][API]feature: add task type list and enable user to add task type to fav ( #11727 )
...
* feature: add task type list and enable user to add task type to fav
* make DependentExecute.java variable protected
* add standalone server fileSet
2022-09-02 11:40:23 +08:00
67e7f88d8b
Refactor heart beat task, use json to serialize/deserialize ( #11702 )
...
* Refactor heart beat task, use json to serialize/deserialize
2022-08-31 16:20:23 +08:00
03e1e6fe45
[Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task ( #11701 )
...
* Fix kill job may cause worker oom
2022-08-31 15:25:01 +08:00
Aaron Wang
Bingchang Chen