* Add waiting strategy to support master/worker can recover from registry lost
* throw exception when zookeeper registry start failed due to interrupted
(cherry picked from commit 7ff34c3947)
* [python] Add missing document
which including `configuration`, `run example`
`how to connect remote server`
close: #9286, #9284, #8917
* [python] Recover python release properties
This patch recovers the properties `python.sign.skip=false`
when the combined profile `release,python` is used.
also close: #9433
* [doc] Add some dev missing doc
Including general-setting, task-definition, audit-log
and they related img
Co-authored-by: Tq <tianqitobethefirst@gmail.com>
Co-authored-by: Tq <tianqitobethefirst@gmail.com>
* [Fix][UI Next][V1.0.0-Alpha]Add zh for dag execution policy (#9363)
* [Bug-9235][Alert]Fix wechat markdown message and change wechat form structure (#9367)
* fix wechat issues:
1. change table msg type to markdown.
2. change userId to not required and enrich hints
3. change 'app id' to 'app id and chat id'
* fix wechat issues:
1. revert table showtype and add markdown showtype.
2. enrich hints.
3. delete 'chatid', rename agentid to weChatAgentIdChatId.
4. modify code to send markdown message.
* fix wechat issues: Change the language pack of agentId to agentId/chatId.
* fix format
* fix param name
Co-authored-by: Amy <amywang0104@163.com>
* [FIX-9355] Fix scheduleTime of start-process-instance api in api-doc (#9359)
* fix#9355
* fix#9355
* fix ut error
* fix ut error
* [CI] try to fix ci (#9366)
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* [optimization] [Service] Optimization ProcessService and add ProcessService interface (#9370)
* [task-spark][docs] Corrected notice section (#9375)
* [python] Migrate pythonGatewayServer into api server (#9372)
Currently the size of our distribute package is up to
800MB, this patch is migrate python gateway server into
api server
The distribute package size before and after this patch is:
```sh
# before
796M apache-dolphinscheduler-2.0.4-SNAPSHOT-bin.tar.gz
# after
647M apache-dolphinscheduler-2.0.4-SNAPSHOT-bin.tar.gz
```
* [Fix][UI Next][V1.0.0-Alpha] Add light color theme to echarts. (#9381)
* [Bug][API-9364]fix ProcessInstance wrong alert group id (#9383)
* fix ProcessInstance wrong alert group id
* change createComplementCommandList method to protected
* [BUG][WORKER-9349]fix param priority (#9379)
* fix param priority
* fix params priority code logic
* [Improvement] change method access (#9390)
* change method to protected
* change method access
* [Fix-9221] [alert-server] optimization and gracefully close (#9246)
* [Fix-9221] [alert-server] optimization and gracefully close
This closes#9221
* [Fix-9221] [alert-server] remove unused mock data
This closes#9221
* [Fix-9221] [alert-server] remove unused mock data
This closes#9221
* [Fix-9221] [alert-server] remove unnecessary Mockito stubbings
* [Fix-9221] [alert-server] init AlertPluginManager in AlertServer
* [Fix-9221] [alert-server] AlertServerTest add AlertPluginManager installPlugin
* [Fix-9221] [alert-server] replace @Eventlistener with @PostConstruct
* [Fix-9221] [alert-server] sonar check solution
* [Improvement-9221] [alert] update constructor injection and replace IStoppable with Closeable
Co-authored-by: guoshupei <guoshupei@lixiang.com>
* [Fix][UI Next][V1.0.0-Alpha] Fix the task instance forced success button multi-language support error. (#9392)
* [doc] Change get help from dev mail list to slack (#9377)
* Change all get help from dev mailing list to slack, because
we find out mailing list have many users ask for subscribe
and they maybe subscribe by accident.
* remove join dev mailing list in faq.md because we already
have it in https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html
* Add new code owner of docs module (#9388)
* [CI] Enable CI to remove unexpected files in /docs/img dir (#9393)
* [Bug][UI Next]Modify the display state logic of save buttons under workflow definition (#9403)
* Modifying site Configurations
* Modify the display state logic of save buttons under workflow definition
* [doc] Remove observability (#9402)
SkyWalking v9 is coming soon and there are without
DolphinScheduler menus anymore, So we should remove
the SW agent to avoid confusion.
close: #9242
* [DS-9387][refactor]Remove the lock in the start method of the MasterRegistryClient class (#9389)
* [Fix-9251] [WORKER] reslove the sql task about of add the udf resource failed (#9319)
* feat(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config
* fix(resource manager): update the dependency
* fix(resource manager): extend s3 to the storage of ds
fix the constant of hadooputils
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager):
merge the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl
* fix(resource manager): extend s3 to the storage of ds
merge the resourceServiceTest
* fix(resource manager): test cancel the test method
createTenant verifyTenant
* fix(resource manager): merge the code follow the check-result of sonar
* fix(resource manager): extend s3 to the storage of ds
fit the spell question
* fix(resource manager): extend s3 to the storage of ds
revert the common.properties
* fix(resource manager): extend s3 to the storage of ds
update the storageConfig with None
* fix(resource manager): extend s3 to the storage of ds
fix the judge of resourceType
* fix(resource manager): extend s3 to the storage of ds
undo the compile-mysql
* fix(resource manager): extend s3 to the storage of ds
delete hadoop aws
* fix(resource manager): extend s3 to the storage of ds
update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties
* fix(resource manager): extend s3 to the storage of ds
update the aws-region
* fix(resource manager): extend s3 to the storage of ds
fix the storageconfig init
* fix(resource manager): update e2e docker-compose
update e2e docker-compose
* fix(resource manager): extend s3 to the storage of ds
revent the e2e common.proprites
print the resource type in propertyUtil
* fix(resource manager): extend s3 to the storage of ds
1.println the properties
* fix(resource manager): println the s3 info
* fix(resource manager): extend s3 to the storage of ds
delete the info and upgrade the s3 info to e2e
* fix(resource manager): extend s3 to the storage of ds
add the bucket init
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager): extend s3 to the storage of ds
upgrade the s3 endpoint
* fix(resource manager): withPathStyleAccessEnabled(true)
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager): upgrade the s3client builder
* fix(resource manager): correct the s3 point to s3client
* fix(resource manager): update the constant BUCKET_NAME
* fix(resource manager): e2e s3 endpoint -> s3:9000
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* style(resource manager): add info to createBucket
* style(resource manager): debug the log
* ci(resource manager): test
test s3
* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql
* fix(resource manager): update the h2 sql
* fix(resource manager): solve to delete the tenant
* style(resource manager): merge the style end delete the unuse s3 config
* fix(resource manager): extend s3 to the storage of ds
UPDATE the rename resources when s3
* fix(resource manager): extend s3 to the storage of ds
1.fix the code style of QuartzImpl
* fix(resource manager): extend s3 to the storage of ds
1.impoort restore_type to CommonUtils
* fix(resource manager): update the work thread
* fix(resource manager): update the baseTaskProcessor
* fix(resource manager): upgrade dolphinscheduler-standalone-server.xml
* fix(resource manager): add user Info to dolphinscheduler_h2.sql
* fix(resource manager): merge the resourceType to NONE
* style(upgrade the log level to info):
* fix(resource manager): sysnc the h2.sql
* fix(resource manager): update the merge the user tenant
* fix(resource manager): merge the resourcesServiceImpl
* fix(resource manager):
when the storage is s3 ,that the directory can't be renamed
* fix(resource manager): in s3 ,the directory cannot be renamed
* fix(resource manager): delete the deleteRenameDirectory in E2E
* fix(resource manager): check the style and recoverd the test
* fix(resource manager): delete the log.print(LoginUser)
* fix(server): fix the udf serialize
* fix(master task): update the udfTest to update the json string
* fix(test): update the udfFuncTest
* fix(common): syn the common.properties
* fix(udfTest): upgrade the udfTest
* fix(common): revent the common.properties
* [Fix-9316] [Task] Configure DB2 data source SQL script execution report ResultSet has been closed exception in SQL task (#9317)
* fix db2 error in the sql task
* update limit in sql task
* [UI] Migrate NPM to PNPM in CI builds (#9431)
Co-authored-by: Devosend <devosend@gmail.com>
Co-authored-by: Tq <tianqitobethefirst@gmail.com>
Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: xiangzihao <460888207@qq.com>
Co-authored-by: gaojun2048 <gaojun2048@gmail.com>
Co-authored-by: mans2singh <mans2singh@users.noreply.github.com>
Co-authored-by: Jiajie Zhong <zhongjiajie955@hotmail.com>
Co-authored-by: Amy0104 <97265214+Amy0104@users.noreply.github.com>
Co-authored-by: guoshupei <15764973965@163.com>
Co-authored-by: guoshupei <guoshupei@lixiang.com>
Co-authored-by: songjianet <1778651752@qq.com>
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
Co-authored-by: labbomb <739955946@qq.com>
Co-authored-by: worry <7039986@qq.com>
Co-authored-by: nobolity <nobolity@users.noreply.github.com>
Co-authored-by: Kerwin <37063904+zhuangchong@users.noreply.github.com>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
* [python] Fix change exists pd attribute user error
* Remove attribute from class ProcessDefinition, cause it should to
object User instead of object ProcessDefinition.
* Grant project to user if attribute user change for exists ProcessDefinition
close: #8751
* Add py.test conftest.py for package integration
* Add new profile `python` to separate build python
package from `release`. Now someone who does not interested
in python api could build package without python environment
* Add auto gpg asc sign when run `mvn instal -Ppython`
* Add property to skip gpg sign `python.sign.skip`
* Set `python.sign.skip=false` sign python dist by default
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
* [python] Add integrated test to python gateway server
* Build java code and create standalone server image in GA
* Add component start docker in python
* Run example to make sure it work to it
close: #8035
* Fix build docker image working directory
* Fix working directory
* Add a new commit hook to pre-commit, to auto
remove unused imports
* Remove all unused pass statements
* Add autoflake to tox and out CI
close: #8592
* add `.pre-commit-config.yaml` in the pydolphinscheduler module and configured basic check rules like `isort`, `black`, `flake8`
* add some instructions in `DEVELOP.MD`
* Fix some mistake and add some docs on it
Co-authored-by: Jiajie Zhong <zhongjiajie955@hotmail.com>
* [python] Add tox for out of box test
* Add `tox` for easy and complete test
* Correct package data attribute
* Add more dirs in flake8 ignore
* Change GA from raw to tox
* Fix env setter
* Temp skip sanity
* Rm not exists job
* Add code test
* fix
* rm all need
* fix
* fix
* fix
* fix
* fix
* fix
* Migrate pip upgrede to tox setting
* Change install_commands to commands
* fix config of install command
* Add GA env to do that
* Fix env error
* Finial commit
* remove duplicate
* Change doc
* Change dependence of ci
* Change dependence of ci
* Change `yaml_parser.py` method `to_string` to magic method
`__str__` make it more pythonic.
* Add some tests to magic method `__str__` and `__repr__`
* Add LICENSE and NOTICE to Python API
* Add new license dir `python-api-license` to keep all
Python API's LICENSE files
* Change LICENSE in distribute package
* Add content about should add LICENSE when new dependence
adding
* [python] refactor yaml file parser
* Change yaml parser package to ruamel.yaml
* Refactor configuration.py module
* And file.py to write file locally
* Add more tests on it
close: #8593
* Fix UT error
* Remove pypyaml from tests
* Fix file error when param create is False
* Fix error logic
* And tests to avoid regression
* [python] Add config mechanism and cli subcommand config
* Add configuration.py mechanism for more easy change config
and move some configs to it. It mechanism including
configuration.py module and default_config.yaml file
* Add `config` for cli subcommand allow users initialize, get,
set configs
close: #8344
* Change setup.py format
This patch add python api package as independent
directory in final dist directory when user run
command `mvn -U install package -Prelease`.
There have `tar.gz` and `whl` in the python dist directory
```
<the-old-dist-package>
python
|--- python-independent-pkg.tar.gz
|--- python-independent-pkg.whl
```
close: #8343
* Give dedicated control to docker build/push goals
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
* date convert of timezone
* remove @JsonFormat
* add unit test
* fix time preview in scheduler
* optimization & add env config
Co-authored-by: caishunfeng <534328519@qq.com>
* Add basic cli mechanism with Click, for now just including
one single subcommand `version`
* Add general and easy test class in tests/testing/cli, and
test to version
* Add sphinx-click to general cli docs basic on click
For now, python API could only communicate python gateway server
in the same hosts, this patch makes it could work with different hosts,
and export java gateway setting to configure file
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
Co-authored-by: ruanwenjun <861923274@qq.com>
* [python] Fix database error handler about no datasource name
The original code show too many unrelated log when database name
do not exists. BTW, it would continue the code when error miss.
This patch is try to fix it.
close: #7616
* Use java exception instead of self define
* Enhance
* [python] Add checker pd attr param and fix switch example
* Correct submit self.param to java gateway
* Fix missing parameter for switch example
* Add mechanism checker before submit to java gateway
close: #7803
* Fix mock get task code and version
* Change judge statement and add comment
* [python] Task condition missing two downstream param
We add two downstream tasks to set task condition
success and failed node
close: #7763
* Add getter and setter property condition_resulth in base task
* [python] Enhance task datax example
* Add full example for `CustomDataX.json`
* Add comment about datasource need to exists.
close: #7800
* Add missing parameter setting
* Make sure all file with `.py` extension and all file
end with "_example"
* Make sure all examples have `__doc__`
* Make sure not have duplicate process definition name
* Make sure process definition same as filename
close: #7729
After #7505 merged. we could use condition task type
but our dependent set in the wrong direction, all
the condition operators should be upstream of the
current task instead of downstream
fix: #7649
* [python] Fix task relation wrong hashable function
Currently our :class:`TaskRelation` have wrong __hash__
function, would mistake a object as same object. It failed
our dependence and process definition. this patch fix it
by adding `_KEY_ATTR` to overwrite TaskRelation's __eq__
and correct __hash__ func
close: #7627
* Fix code format
Wenjun Ruan