Commit Graph

29 Commits

Author SHA1 Message Date
Jay Chung a645cb1c2a cherry-pick [chore] Separate Python API into another repository
#12779
2023-01-03 18:18:25 +08:00
JieguangZhou a2b3659fe9 [improve] Optimize MLFlow task plugin for easy of use (#12071)
optimize code

(cherry picked from commit cf522e2f)
2022-09-21 13:50:48 +08:00
JieguangZhou dde6f63c31 [Feature][PyDolphinScheduler] Support DVC task in pyds #11922 (#11941)
* add dvc task in pyds

* add BaseDVC class

(cherry picked from commit bbc228e93f)
2022-09-18 16:49:34 +08:00
JieguangZhou cb063732d7 [feat][python] Support MLflow task in python api (#11962)
(cherry picked from commit ad683c3c42)
2022-09-18 16:46:45 +08:00
JieguangZhou f66dedc9da [feat][python] Support OpenMLDB task in python api (#11944)
Using the whole word zookeeper instead of short cut zk
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>

(cherry picked from commit e50a32d9cc)
2022-09-18 16:43:36 +08:00
JieguangZhou 1ac2e4a8f3 [feat][python] Add sagemaker yaml example (#11925)
(cherry picked from commit e05f14ba23)
2022-09-18 16:43:24 +08:00
JieguangZhou 6f3b4c1624 [feat][python] Support Pytorch task in python api (#11975)
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
(cherry picked from commit 5202e5cfc6)
2022-09-18 16:43:08 +08:00
JieguangZhou 5e42f52bdf
[3.1.0-prepare][cherry-pick][DSIP-11][python] create workflows from YAML configuration (#11988)
* [DSIP-11][python] create workflows from YAML configuration (#11611)

(cherry picked from commit 38ee91fb1e)

* fix pydolphin yaml doc link (#11817)

(cherry picked from commit c41fa5a8b1)
2022-09-17 08:56:37 +08:00
Eric Gao 49a1270ef5
[Feature-8222][python] move examples into the scope of source package (#8340) 2022-02-14 15:07:00 +08:00
Jiajie Zhong 35578efd08
[python] Initiate document for pydolphinscheduler (#8005) 2022-01-26 12:37:57 +08:00
Devosend 7aeac75860
[Feature-7348] [Python] Add workflow as code task type mr (#8140) 2022-01-21 09:54:36 +08:00
Devosend 027af091c7
[Feature-7346] [Python]Add workflow as code task type spark (#7968)
* add spark task

* fix code format

* add parent class for flink and spark

* modify Engine docstring

* modify docstring of Engine
2022-01-15 23:24:55 +08:00
Jiajie Zhong b1edce2b11
[python] Add checker pd attr param and fix switch example (#7818)
* [python] Add checker pd attr param and fix switch example

* Correct submit self.param to java gateway
* Fix missing parameter for switch example
* Add mechanism checker before submit to java gateway

close: #7803

* Fix mock get task code and version

* Change judge statement and add comment
2022-01-05 19:59:20 +08:00
Jiajie Zhong 1417967d9e
[python] Task condition missing two downstream param (#7783)
* [python] Task condition missing two downstream param

We add two downstream tasks to set task condition
success and failed node

close: #7763

* Add getter and setter property condition_resulth in base task
2022-01-05 19:58:44 +08:00
Jiajie Zhong fcbb5f4d8f
[python] Enhance task datax example (#7801)
* [python] Enhance task datax example

* Add full example for `CustomDataX.json`
* Add comment about datasource need to exists.

close: #7800

* Add missing parameter setting
2022-01-05 19:57:28 +08:00
Jiajie Zhong f324b2f884
[python] Add test for examples (#7759)
* Make sure all file with `.py` extension and all file
  end with "_example"
* Make sure all examples have `__doc__`
* Make sure not have duplicate process definition name
* Make sure process definition same as filename

close: #7729
2022-01-04 10:09:38 +08:00
Devosend 081adf4aaa
[Feature-7347] [Python]Add workflow as code task type flink (#7632)
* add Flink task for code as flow

Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-01-04 09:45:54 +08:00
Jiajie Zhong cc8fbe3e14
[python] Fix task condition set wrong deps (#7650)
After #7505 merged. we could use condition task type
but our dependent set in the wrong direction, all
the condition operators should be upstream of the
current task instead of downstream

fix: #7649
2021-12-27 20:37:40 +08:00
Jiajie Zhong fd6eb1f830
[python] Correct python import syntax in example (#7617)
fix: #7615
2021-12-24 20:07:49 +08:00
Jiajie Zhong 946a0c7c57
[python] Add task switch (#7531)
* [python] Add task switch

close: #6928

* Fix code style
2021-12-22 11:46:34 +08:00
Jiajie Zhong e23a4848c0
[python] Add task condition (#7505)
* [python] Add task condition

close: #6927

* Add example and set downstream dep
2021-12-22 11:06:45 +08:00
Devosend 0c7aa4e2c5
[Feature-7349] [Python]Add workflow as code task type datax (#7437)
* support datax task by python
* add task datax example

Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2021-12-17 11:12:48 +08:00
Jiajie Zhong 6f93ebf3ba
[python] Add task dependent (#7405)
fix: #6926
2021-12-16 09:58:50 +08:00
Jiajie Zhong 7c5c7ec907
[ci][python] Add isort to sort out import (#6871)
* Add isort config file to fix conflict with black
* Add some doc about isort
2021-11-17 11:48:52 +08:00
Jiajie Zhong e76cf77040
[python] Add parameter schedule for process definition (#6664)
* [python] Add parameter schedule for process definition

* Rebase and fix some code style

* May schedule work on both string and datetime

* Fix flaky test

* Add comment about freeze time

* Add edge test for schedule_json with None schedule

* Fix test function name

* Fix rebase error
2021-11-13 17:21:40 +08:00
Jiajie Zhong 32d4411469
[python] Add flake8 and black for code style and integrated to GA (#6679)
* [python] Add code style lint for GA

* Change github action name

* Auto change by black

* Fix flake8

* Fix broken link for pyds README.md

* Auto fix by black

* Separate GitHub workflows

* Add Black badge and CI locally in README.md
2021-11-05 15:14:28 +08:00
Jiajie Zhong 9fb6d6ba9f
Fix f-strings variable error (#6677) 2021-11-04 16:15:16 +08:00
Jiajie Zhong ef9de9de3b
Add example about how to bulk create workflow (#6659) 2021-11-03 11:23:09 +08:00
Jiajie Zhong dd6ed36f65
Add Python API implementation of workflows-as-code (#6269)
* Init DS python SDK pydolphinscheduler: python code definition

* Doc first

* Add quick start and developer doc

* Java documentation change

* Add LICENSE-py4j.txt

* Add py4j to release-docs/LICENSE

* Move dependency version to parent pom

* Remove outdated code

* Add tenant parameter to tutorial
2021-10-31 20:35:46 +08:00