dedff70f90
[Improvement] Merge spi.utils into common.utils ( #12502 )
...
* improve JSONUtil
* merge spi.utils into common.utils
* split constants && remove some copy utils method
* remove StringUtils
2022-10-23 21:23:54 +08:00
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports ( #12412 )
...
* Fix remaining style errors and wildcard imports
* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link
* Fix spotless matching scope of markdown files
2022-10-19 11:08:04 +08:00
b357781df2
[Migrate][Test] Migrate junit4 -> junit5 test cases in common, service, and spi modules ( #12363 )
2022-10-14 11:06:29 +08:00
db1d319159
Reorganize some classes in common module, remove duplicate classes ( #12321 )
2022-10-13 10:23:27 +08:00
9ab79e064c
[Improvement][Test] Fully remove the usage of powermock from the whole project ( #12244 )
...
* Fully remove the usage of powermock from the whole project
* Upgrade org.reflections to 0.10.12
2022-10-10 10:43:38 +08:00
c87f6d89a4
[fix#12195] Change date to timestamp to support cross diff time zones ( #12239 )
...
* fix DS support cross time zone, use timestamp replace date
2022-10-01 12:50:42 +08:00
e6832220c3
[Improvement][SQL Task]use default sql segment separator ( #10869 )
2022-09-26 11:08:57 +08:00
66958b91d7
[Feature][Resource Center] Add support for Alibaba Cloud OSS as storage of resource center ( #11708 )
...
* Add support for Alibaba Cloud OSS as storage of resource center
* Add dependency licenses
* Refactor code, add unit tests and related docs
2022-09-20 22:44:06 +08:00
86291bb5e6
[Feature-10273][Datasource] Add dataSource for AWS Athena ( #11157 )
2022-09-16 09:59:20 +08:00
b96d69701a
[Improvement-#11613] Add spi priority factory ( #11614 )
...
* Add spi priority factory
* Add doc
* Add override log
* Use lombok
* Add comment
2022-08-24 19:51:47 +08:00
71cf7e1baa
[Fix-10918] Close datasource when expire from guava cache ( #11120 )
...
* Close datasource when expire from guava cache
* Remove duplicate datasource in HiveDataSourceClient
2022-07-23 20:43:37 +08:00
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version ( #11025 )
2022-07-20 10:37:31 +08:00
71f0168510
[Feature] Enable users to create python env from requirements.txt ( #10658 )
2022-07-06 15:56:39 +08:00
dafed572d1
[Fix][dolphinscheduler-spi] json parse support LocalDateTime ( #10440 )
...
* JSONUtils support LocalDateTime
* remove extraneous unit tests
* remove unused classes import.
Co-authored-by: HomminLee <lihongmin@ysstech.com>
2022-06-27 10:19:28 +08:00
d80cf21456
Clean up unused dependencies and packaging issues ( #9944 )
2022-05-31 15:22:41 +08:00
592e433b1e
[Feature][Datasource]replace DataSourceController API with string JSON and cast to DTO ( #10276 )
...
* replace DataSourceController API with string JSON and cast to DTO
* add DataSourceProcessorManager and DataSourceProcessorProvider to datasource processor functions
2022-05-30 14:12:19 +08:00
31cd1b5e61
Serial wait for subsequent execution ( #9847 )
2022-04-29 15:44:05 +08:00
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT ( #9223 ) ( #9299 )
2022-04-07 18:15:19 +08:00
263791a63e
[Bug-9137][Server] fix timezone init ( #9219 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-28 09:38:51 +08:00
a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module ( #8584 )
...
* Task plug-in optimization, task-related classes are migrated to the task plugin module
* remove tasktype enum class.
* fix code style.
* fix code style
* fix code style
* fix ProcessServiceTest.testUpdateResourceInfo test class
* fix code style
* fix test task plugin manager check parameters null
* fix test ut
* update aws license
* change TaskTimeoutStrategy class
2022-03-15 15:16:36 +08:00
7b66b66e8f
[Feature-8527][Datasource] Add dataSource for Amazon Redshift ( #8564 )
...
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
2022-03-01 19:29:52 +08:00
20ee9a37fe
[Feature-8485][task] Add EMR task plugin to support submit task to AWS EMR ( #8503 )
...
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2022-03-01 11:43:37 +08:00
a72742b9de
remove unuse import jsonFormat ( #8548 )
2022-02-25 22:05:05 +08:00
e174d7a40e
[Fix-8394][Server]fix when the data is supplemented business time still date+1 ( #8532 )
2022-02-25 10:27:08 +08:00
84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified ( #8477 )
...
* date convert of timezone
* remove @JsonFormat
* add unit test
* fix time preview in scheduler
* optimization & add env config
Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-24 18:20:34 +08:00
f2b9796bc2
[Bug-8110][WorkerServer] kill all running task before worker stop ( #8490 )
...
* kill all running task before worker stop
* kill local process
* remove kill yarn job
* adjust the orders of close
Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-23 18:22:42 +08:00
60ddede6a6
fixbug: varPool value is error ( #7724 )
2022-02-23 14:53:20 +08:00
703d790415
[Bug] [Parameter Context] The preparameter OUT is null ( #8315 )
...
* fix:param context prop value is null
2022-02-14 14:25:23 +08:00
4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 ( #6718 )
...
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
* fix Constants bug
* remove the unused class
* add unit test
* fix code style error
* add unit test
* refactor data quality common entity
* fix code style error
* add unit test
* close e2e test
* fix code smell bug
* modify dataquality enum value to 14 in TaskType
* add data qualtiy task
* update
* add getDatasourceOptions interface
* fix checkstyle
* close e2e test
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* add unit test and alter dataquality task result
* fix checkstyle
* fix process service test error
* add unit test and fix code smells
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* change execute sql type name
* revert ui pom.xml
* fix data quality task error
* fix checkstyle error
* fix dq task src_connector_type ui select bug
* fix spark rw postgresql bug
* change mysql driver scope
* fix form-create json bug
* fix code smell
* fix DolphinException Bug
* fix ui validate rule and Alert title
* fix target connection param bug
* fix threshold validate change
* add rule input entry index
* change statistic_comparison_check logic
* remove check type change
* add DateExpressionReplaceUtil
* fix null point expetion
* fix null point expetion
* fix test error
* add more sql driver
* fix test error and remove DateExprReplaceUtil
* add get datasource tables and columns
* add get datasource tables and columns
* remove hive-jdbc in pom.xml
* fix code smells
* update sql
* change the pom.xml
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
* update sql
* [Feature][DataQuality] Add data quality task ui (#5054 )
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* fix dq task src_connector_type ui select bug
* fix threshold validate change
* remove check type change
* add get datasource tables and columns
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality module (#4830 )
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* update
* close e2e test
* fix spark rw postgresql bug
* change mysql driver scope
* add more sql driver
* remove hive-jdbc in pom.xml
* change the pom.xml
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality task backend (#4883 )
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* refactor data_quality_module
* add header license
* data quality module refactor
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix code smell
* fix check style
* fix unit test error
* task statistics value add unique code
* fix unit test error
* fix checkstyle error
* fix checkstyle
* fix security hotspot
* fix unit test error
* fix security hotspot
* fix check
* add data quality task error handling
* fix unit test error
* add unit test
* add unit test
* optimize data quality result alert
* fix unit test
* fix sql script error
* fix bug
* update sql script
* fix checkstyle
* add license
* fix checkstyle
* fix checkstyle
* fix unit test
* add jacoco dependencies
* fix unit test
* fix unit test
* add jacoco dependencies
* add unit test
* add unit test
* add license
* fix checkstyle
* fix pom
* fix checkstyle
* fix checkstyle
* merge dev
* fix ui error
* fix pom error
* fix pom error
* fix test error
* fix test error
* mssql-jdbc exclude azure-keyvault
* fix test error
* merge dev and add unit test
* add notes
* rollback the CollectionUtils
* fix
* update sql
* fix
* fix
* fix query rule page error
* change dq.jar path
* fix sql error
* fix ui error
* fix(dq): jar path&task enum description
* add notes on DataQualityApplication
* fix dq result jump error
* fix(ui): page condition
* feat(ui): add show error output path
* change version
* remove all chinese word in sql
* merge
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2022-01-27 15:55:45 +08:00
103ccb01da
change version to 2.0.4-SNAPSHOT ( #8188 )
2022-01-25 15:17:17 +08:00
898727709f
[fix-7671][plugin] Supports whether SQL is placed in the same session for configuration(fix-7671) ( #7675 )
...
-Supports whether SQL is placed in the same session for configuration
2022-01-13 18:06:40 +08:00
e4cdd82b7c
[Improvement][TaskLog] Unified task log ( #7831 )
...
* task logger
* update
* log format
* remove master config task-logger
* udpate log level
Co-authored-by: caishunfeng <534328519@qq.com>
2022-01-07 21:04:54 +08:00
8e62dd8246
[Fix-7291][datasource] Compatible with different versions of MySQL data source loading ( #7291 ) ( #7305 )
2021-12-13 22:49:46 +08:00
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties ( #7160 )
...
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
2021-12-10 11:28:53 +08:00
802fc498b5
[Fix-#6605][Common] Fix the situation where the obtained string property is empty. ( #6661 )
...
* Fix the situation where the obtained string property is empty.
* recovery test
* update code style.
* megre dev.
2021-11-19 10:58:17 +08:00
861aaaf971
[FIX-#6505][Dao] upgrade the MySQL driver package for building MySQL jdbcUrl ( #6708 )
...
* replace com.mysql.jdbc.Driver everywhere with com.mysql.cj.jdbc.Driver
2021-11-18 08:39:11 +08:00
4114cb07f6
Add actuator and enable metrics, add Grafana configs ( #6840 )
2021-11-14 20:22:13 +08:00
11844415b0
Refactor datasource and task plugins and simplify its usage ( #6767 )
2021-11-10 18:29:17 +08:00
0aea5ae11a
[feature-6071][plugin] add datasource plugin. ( #6071 ) ( #6361 )
...
* [feature-6071][plugin] add datasource plugin. (#6071 )
* remove the dependency of datasource in common and task-plugin module.
* the new datasource-api module provides data source connection acquisition.
* add sqlserver data source plug-in.
* add ds license
* fix check style
* update spi pom
* frx conflict resolution
* fix check style
* fix datasource ut
* fix code check style
* fix code check style
* fix datasource ut
* add pg hive mysql oracle clickhouse db2 datasource
* fix datasource package
* update datasource pom
* update datasource pom
* update datasource pom
* update datasource pom
* update pom
* update pom dep
* update pom dep
* fix license
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* add oracle package
* modify comments
* conflict resolution
* rename log comment
* rename log comment
2021-11-09 20:20:07 +08:00
05bae0b226
Refactor alert plugins and simplify its usage ( #6723 )
2021-11-09 18:59:34 +08:00
5400abc74f
Refactor registry plugin and simplify its usage ( #6712 )
2021-11-07 15:18:10 +08:00
5e923971bf
Migrate http to https in maven address ( #6698 )
2021-11-05 20:17:27 +08:00
72daba58cd
[DS-6694][Master] Reduce the refresh resource interval of LowerWeight ( #6695 )
...
* [DS-6694][Master] Reduce the refresh resource interval of LowerWeight
* add server node update event handle
Co-authored-by: caishunfeng <534328519@qq.com>
2021-11-05 19:10:36 +08:00
71047db39e
[FIX][Standalone Server] check if maven settings.xml not exists ( #6691 )
2021-11-04 20:17:53 +08:00
61a8726ccc
[Fix][Standalone Server] fix plugin load failed when local maven repository path is not default value ( #6625 ) ( #6626 )
...
* [Fix][Standalone Server] fix plugin load failed when local maven repository path is not default value
2021-11-03 18:46:25 +08:00
86c85114b2
make alert instance page display more friendly ( #6635 )
...
* make alert instance page display more friendly
* Update EmailAlertChannelTest.java
2021-10-31 21:16:10 +08:00
9b1984c244
[fix-6570][task] add sql task udf resource load ( #6570 ) ( #6571 )
...
* [fix-6570][task] add sql task udf resource load (#6570 )
-add sql task udf resource load
-Fix UdfFuncRequest serialization
* fix checkstyle
* add defaultFS in sqlTaskContext
* fix checkstyle
2021-10-21 15:23:49 +08:00
a80fca70f7
[Fix-6497][Server] Fix the missing code about the feature of supporting multiple environments. ( #6509 )
2021-10-12 17:47:04 +08:00
7dbea9218d
bug fix ( #6381 )
...
The newly introduced spi can't work in windows os.
2021-09-28 12:26:55 +08:00
67508d8d1b
update project version to 2.0.0-SNAPSHOT ( #6383 )
2021-09-28 12:00:43 +08:00
caishunfeng