natas_hw
/
dolphinscheduler0701
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[Feature][task]add azure datafactory task type (#13417) 

* init data factory

* add task-type-config.yaml

* add data factory ui

* submit doc

* submit license

---------

Co-authored-by: devosend <devosend@gmail.com>
This commit is contained in:
Tq 2023-01-30 11:15:20 +08:00 committed by GitHub
parent 23c7363f64
commit 78ba71bdbd
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
74 changed files with 1874 additions and 16 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

70
docs/docs/en/guide/task/datafactory.md Normal file
View File

@ -0,0 +1,70 @@
# Azure DataFactory Node
## Overview
In the world of big data, raw, unorganized data is often stored in relational, non-relational, and other storage systems.[Azure DataFactory](https://learn.microsoft.com/en-us/azure/data-factory/introduction) is a managed cloud service that's built for these complex hybrid extract-transform-load (ETL), extract-load-transform (ELT), and data integration projects.
DolphinScheduler DataFactory functions:
- Create Azure DataFactory task enable DolphinScheduler to schedule DataFactory pipelines, keep query pipeline execution status until finish the execution.
## Prerequisite
- **ResourceGroup**: Azure owns a resource group
- **DataFactory**: Azure owns a data factory under the resource group
- **Pipeline**: Azure owns pipelines under the assigned resource group and data factory
- **Application**: Azure owns an application has permission to visit data factory, enables using SDK to call data factory functions
- **ApplicationClientSecret**: To apply for client secret under Application `Certificates & secrets`
- **AZURE-CLI**: Install Azure auth application AZURE-CLI on machine, please refer to [Install the Azure CLI on Linux](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux)
### Application Permission Setting
First, visit the `Subcription` page and choose `Access control (IAM)`, then click `Add role assignment` to the authorization page.
![Subcription-IAM](../../../../img/tasks/demo/datafactory_auth1.png)
After that, select `Contributor` role which satisfy functions calls in data factory. Then click `Members` page, and click `Select members`.
Search application name or application `Object ID` to assign `Contributor` role to application.
![Subcription-Role](../../../../img/tasks/demo/datafactory_auth2.png)
## Configurations
Config Azure configurations, modify `azure` related configurations in `common.properties`
- **resource.azure.client.id**: Azure Application Application (client) ID
- **resource.azure.client.secret**: Azure Application client secret under `Certificates & secrets`
- **resource.azure.subId**: Data factory subscription ID
- **resource.azure.tenant.id**: Azure Active Directory tenant ID
```yaml
# The Azure client ID (Azure Application (client) ID)
resource.azure.client.id=minioadmin
# The Azure client secret in the Azure application
resource.azure.client.secret=minioadmin
# The Azure data factory subscription ID
resource.azure.subId=minioadmin
# The Azure tenant ID in the Azure Active Directory
resource.azure.tenant.id=minioadmin
```
## Create Task
- Click `Project -> Management-Project -> Name-Workflow Definition`, and click the "Create Workflow" button to enter the
DAG editing page.
- Drag from the toolbar <img src="../../../../img/tasks/icons/datafactory.png" width="15"/> task node to canvas.
## Task Example
[//]: # (TODO: use the commented anchor below once our website template supports this syntax)
[//]: # (- Please refer to [DolphinScheduler Task Parameters Appendix]&#40;appendix.md#default-task-parameters&#41; `Default Task Parameters` section for default parameters.)
- Please refer to [DolphinScheduler Task Parameters Appendix](appendix.md) `Default Task Parameters` section for default parameters.
Here are some specific parameters for the DataFactory plugin:
- **factoryName**: data factory name
- **resourceGroupName**: resource group name of the data factory
- **pipelineName**: pipeline name under the resource group and data factory
The following shows the task plugin example:
![data-factory](../../../../img/tasks/demo/datafactory.png)

68
docs/docs/zh/guide/task/datafactory.md Normal file
View File

@ -0,0 +1,68 @@
# Azure DataFactory 节点
## 综述
在大数据环境中，原始、散乱的数据通常存储在关系、非关系和其他存储系统中。[Azure DataFactory](https://learn.microsoft.com/en-us/azure/data-factory/introduction) 是为这些复杂的混合提取-转换-加载 (ETL)、提取-加载-转换 (ELT) 和数据集成项目而构建的托管云服务。
DolphinScheduler DataFactory 组件的功能:
- 创建 Azure DataFactory 任务可以调度DataFactory中的pipeline，持续获取执行状态，直至任务执行完成。
## 前置条件
- **资源组**: 拥有一个资源组
- **数据工厂**: 拥有一个资源组下的数据工厂
- **Pipeline**: 拥有前两者对应的pipeline
- **应用**: 拥有一个有数据工厂访问权限的应用，才能赋予SDK调用数据工厂
- **应用客户端密钥**: 在应用的`Certificates & secrets`申请客户端密钥
- **AZURE-CLI**: 在机器上安装了AZURE的鉴权应用AZURE-CLI，请参考[在LINUX上安装AZURE-CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux)
### 应用权限设置
首先打开当前`Subcription`页面，点击`Access control (IAM)`，再点击`Add role assignment`进入授权页面。
![Subcription-IAM](../../../../img/tasks/demo/datafactory_auth1.png)
首先选择`Contributor`角色足够满足调用数据工厂。然后选择`Members`页面，再选择`Select members`，检索APP名称或APP的`Object ID`并添加，从给指定APP添加权限.
![Subcription-Role](../../../../img/tasks/demo/datafactory_auth2.png)
## 环境配置
需要进行Azure的一些配置，修改`common.properties`中的`azure`相关配置信息
- **resource.azure.client.id**: Azure Application应用的Application (client) ID
- **resource.azure.client.secret**: Azure Application应用的`Certificates & secrets`下的客户端密钥
- **resource.azure.subId**: 数据工厂的subscription ID
- **resource.azure.tenant.id**: Azure Active Directory下的tenant ID
```yaml
# The Azure client ID (Azure Application (client) ID)
resource.azure.client.id=minioadmin
# The Azure client secret in the Azure application
resource.azure.client.secret=minioadmin
# The Azure data factory subscription ID
resource.azure.subId=minioadmin
# The Azure tenant ID in the Azure Active Directory
resource.azure.tenant.id=minioadmin
```
## 创建任务
- 点击项目管理-项目名称-工作流定义，点击“创建工作流”按钮，进入 DAG 编辑页面；
- 拖动工具栏的 <img src="../../../../img/tasks/icons/datafactory.png" width="15"/> 任务节点到画板中。
## 任务样例
[//]: # (TODO: use the commented anchor below once our website template supports this syntax)
[//]: # (- 默认参数说明请参考[DolphinScheduler任务参数附录]&#40;appendix.md#默认任务参数&#41;`默认任务参数`一栏。)
- 默认参数说明请参考[DolphinScheduler任务参数附录](appendix.md)`默认任务参数`一栏。
### 独有参数
- **factoryName**: 数据工厂名称
- **resourceGroupName**: 与数据工厂相关的资源组名称
- **pipelineName**: 选取数据工厂以及资源组下的对应pipeline名称
组件图示如下：
![data-factory](../../../../img/tasks/demo/datafactory.png)

BIN
docs/img/tasks/demo/datafactory.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 7.0 KiB

BIN
docs/img/tasks/demo/datafactory_auth1.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 31 KiB

BIN
docs/img/tasks/demo/datafactory_auth2.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 38 KiB

BIN
docs/img/tasks/icons/datafactory.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.6 KiB

8
dolphinscheduler-api/pom.xml
View File

@ -198,6 +198,14 @@
<groupId>org.springdoc</groupId>
<artifactId>springdoc-openapi-ui</artifactId>
</dependency>
<dependency>
<groupId>com.azure.resourcemanager</groupId>
<artifactId>azure-resourcemanager</artifactId>
</dependency>
<dependency>
<groupId>com.azure.resourcemanager</groupId>
<artifactId>azure-resourcemanager-datafactory</artifactId>
</dependency>
</dependencies>
<build>
<testResources>

108
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/CloudController.java Normal file
View File

@ -0,0 +1,108 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.api.controller;
import static org.apache.dolphinscheduler.api.enums.Status.LIST_AZURE_DATA_FACTORY_ERROR;
import static org.apache.dolphinscheduler.api.enums.Status.LIST_AZURE_DATA_FACTORY_PIPELINE_ERROR;
import static org.apache.dolphinscheduler.api.enums.Status.LIST_AZURE_RESOURCE_GROUP_ERROR;
import org.apache.dolphinscheduler.api.aspect.AccessLogAnnotation;
import org.apache.dolphinscheduler.api.enums.Status;
import org.apache.dolphinscheduler.api.exceptions.ApiException;
import org.apache.dolphinscheduler.api.service.CloudService;
import org.apache.dolphinscheduler.api.utils.Result;
import org.apache.dolphinscheduler.common.constants.Constants;
import org.apache.dolphinscheduler.dao.entity.User;
import java.util.List;
import javax.annotation.Resource;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestAttribute;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseStatus;
import org.springframework.web.bind.annotation.RestController;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.tags.Tag;
/**
* cloud controller
*/
@Tag(name = "CLOUD_TAG")
@RestController
@RequestMapping("/cloud")
public class CloudController extends BaseController {
@Resource
private CloudService cloudService;
/**
* get datafactory list
*
* @param loginUser login user
* @return datafactory name list
*/
@Operation(summary = "listDataFactory", description = "LIST_DATA_FACTORY")
@GetMapping(value = "/azure/datafactory/factories")
@ResponseStatus(HttpStatus.OK)
@ApiException(LIST_AZURE_DATA_FACTORY_ERROR)
@AccessLogAnnotation(ignoreRequestArgs = "loginUser")
public Result listDataFactory(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser) {
List<String> factoryNames = cloudService.listDataFactory(loginUser);
return success(Status.SUCCESS.getMsg(), factoryNames);
}
/**
* get resourceGroup list
*
* @param loginUser login user
* @return resourceGroup list
*/
@Operation(summary = "listResourceGroup", description = "LIST_RESOURCE_GROUP")
@GetMapping(value = "/azure/datafactory/resourceGroups")
@ResponseStatus(HttpStatus.OK)
@ApiException(LIST_AZURE_RESOURCE_GROUP_ERROR)
@AccessLogAnnotation(ignoreRequestArgs = "loginUser")
public Result listResourceGroup(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser) {
List<String> resourceGroupNames = cloudService.listResourceGroup(loginUser);
return success(Status.SUCCESS.getMsg(), resourceGroupNames);
}
/**
* get resourceGroup list
*
* @param loginUser login user
* @return resourceGroup list
*/
@Operation(summary = "listPipeline", description = "LIST_PIPELINE")
@GetMapping(value = "/azure/datafactory/pipelines")
@ResponseStatus(HttpStatus.OK)
@ApiException(LIST_AZURE_DATA_FACTORY_PIPELINE_ERROR)
@AccessLogAnnotation(ignoreRequestArgs = "loginUser")
public Result listPipeline(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
@RequestParam("factoryName") String factoryName,
@RequestParam("resourceGroupName") String resourceGroupName) {
List<String> pipelineNames = cloudService.listPipeline(loginUser, factoryName, resourceGroupName);
return success(Status.SUCCESS.getMsg(), pipelineNames);
}
}

3
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java
View File

@ -265,6 +265,9 @@ public enum Status {
"请先保存后再执行"),
START_NODE_NOT_EXIST_IN_LAST_PROCESS(10207, "this node {0} does not exist in the latest process definition",
"该节点 {0} 不存在于最新的流程定义中"),
LIST_AZURE_DATA_FACTORY_ERROR(10208, "list azure data factory error", "查询AZURE数据工厂列表错误"),
LIST_AZURE_RESOURCE_GROUP_ERROR(10209, "list azure resource group error", "查询AZURE资源组列表错误"),
LIST_AZURE_DATA_FACTORY_PIPELINE_ERROR(10210, "list azure data factory pipeline error", "查询AZURE数据工厂pipeline列表错误"),
UDF_FUNCTION_NOT_EXIST(20001, "UDF function not found", "UDF函数不存在"),
UDF_FUNCTION_EXISTS(20002, "UDF function already exists", "UDF函数已存在"),

31
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/CloudService.java Normal file
View File

@ -0,0 +1,31 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.api.service;
import org.apache.dolphinscheduler.dao.entity.User;
import java.util.List;
public interface CloudService {
List<String> listDataFactory(User loginUser);
List<String> listResourceGroup(User loginUser);
List<String> listPipeline(User loginUser, String factoryName, String resourceGroupName);
}

88
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/CloudServiceImpl.java Normal file
View File

@ -0,0 +1,88 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.api.service.impl;
import org.apache.dolphinscheduler.api.service.CloudService;
import org.apache.dolphinscheduler.common.utils.PropertyUtils;
import org.apache.dolphinscheduler.dao.entity.User;
import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
import java.util.ArrayList;
import java.util.List;
import org.springframework.stereotype.Service;
import com.azure.core.http.rest.PagedIterable;
import com.azure.core.management.AzureEnvironment;
import com.azure.core.management.profile.AzureProfile;
import com.azure.identity.ClientSecretCredential;
import com.azure.identity.ClientSecretCredentialBuilder;
import com.azure.resourcemanager.AzureResourceManager;
import com.azure.resourcemanager.datafactory.DataFactoryManager;
import com.azure.resourcemanager.datafactory.models.Factories;
import com.azure.resourcemanager.datafactory.models.PipelineResource;
import com.azure.resourcemanager.resources.models.ResourceGroups;
@Service
public class CloudServiceImpl extends BaseServiceImpl implements CloudService {
private static final String AZURE_CLIENT_ID = PropertyUtils.getString(TaskConstants.AZURE_CLIENT_ID);
private static final String AZURE_CLIENT_SECRET = PropertyUtils.getString(TaskConstants.AZURE_CLIENT_SECRET);
private static final String AZURE_ACCESS_SUB_ID = PropertyUtils.getString(TaskConstants.AZURE_ACCESS_SUB_ID);
private static final String AZURE_SECRET_TENANT_ID = PropertyUtils.getString(TaskConstants.AZURE_SECRET_TENANT_ID);
private static final AzureProfile profile =
new AzureProfile(AZURE_SECRET_TENANT_ID, AZURE_ACCESS_SUB_ID, AzureEnvironment.AZURE);
private static final ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder()
.clientId(AZURE_CLIENT_ID)
.clientSecret(AZURE_CLIENT_SECRET)
.tenantId(AZURE_SECRET_TENANT_ID)
.authorityHost(profile.getEnvironment().getActiveDirectoryEndpoint())
.build();
private static final DataFactoryManager manager;
private static final AzureResourceManager azure;
static {
manager = DataFactoryManager.authenticate(clientSecretCredential, profile);
azure = AzureResourceManager.authenticate(clientSecretCredential, profile).withDefaultSubscription();
}
@Override
public List<String> listDataFactory(User loginUser) {
Factories factories = manager.factories();
List<String> names = new ArrayList<>();
factories.list().stream().forEach(e -> names.add(e.name()));
return names;
}
@Override
public List<String> listResourceGroup(User loginUser) {
ResourceGroups resourceGroups = azure.resourceGroups();
List<String> names = new ArrayList<>();
resourceGroups.list().stream().forEach(e -> names.add(e.name()));
return names;
}
@Override
public List<String> listPipeline(User loginUser, String factoryName, String resourceGroupName) {
PagedIterable<PipelineResource> pipelineResources =
manager.pipelines().listByFactory(resourceGroupName, factoryName);
List<String> names = new ArrayList<>();
pipelineResources.stream().forEach(e -> names.add(e.name()));
return names;
}
}

1
dolphinscheduler-api/src/main/resources/task-type-config.yaml
View File

@ -33,6 +33,7 @@ task:
- 'EMR'
- 'K8S'
- 'DMS'
- 'DATA_FACTORY'
logic:
- 'SUB_PROCESS'
- 'DEPENDENT'

12
dolphinscheduler-bom/pom.xml
View File

@ -103,6 +103,8 @@
<azure-identity.version>1.7.1</azure-identity.version>
<redshift-jdbc42.version>2.1.0.9</redshift-jdbc42.version>
<aws-java-sdk-redshift.version>1.12.300</aws-java-sdk-redshift.version>
<azure-resourcemanager.version>2.21.0</azure-resourcemanager.version>
<azure-resourcemanager-datafactory.version>1.0.0-beta.19</azure-resourcemanager-datafactory.version>
</properties>
<dependencyManagement>
@ -798,6 +800,16 @@
<artifactId>azure-identity</artifactId>
<version>${azure-identity.version}</version>
</dependency>
<dependency>
<groupId>com.azure.resourcemanager</groupId>
<artifactId>azure-resourcemanager</artifactId>
<version>${azure-resourcemanager.version}</version>
</dependency>
<dependency>
<groupId>com.azure.resourcemanager</groupId>
<artifactId>azure-resourcemanager-datafactory</artifactId>
<version>${azure-resourcemanager-datafactory.version}</version>
</dependency>
<dependency>
<groupId>com.amazon.redshift</groupId>
<artifactId>redshift-jdbc42</artifactId>

11
dolphinscheduler-common/src/main/resources/common.properties
View File

@ -26,6 +26,17 @@ resource.storage.type=NONE
# resource store on HDFS/S3 path, resource file will store to this base path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended
resource.storage.upload.base.path=/dolphinscheduler
# The Azure client ID (Azure Application (client) ID)
resource.azure.client.id=minioadmin
# The Azure client secret in the Azure application
resource.azure.client.secret=minioadmin
# The Azure data factory subscription ID
resource.azure.subId=minioadmin
# The Azure tenant id in the Azure Active Directory
resource.azure.tenant.id=minioadmin
# The query interval
resource.query.interval=10000
# The AWS access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required
resource.aws.access.key.id=minioadmin
# The AWS secret access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required

44
dolphinscheduler-dist/release-docs/LICENSE
View File

@ -517,6 +517,8 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
reactor-netty-core 1.0.22: https://mvnrepository.com/artifact/io.projectreactor.netty/reactor-netty-core/1.0.22, Apache 2.0
reactor-netty-http 1.0.22: https://mvnrepository.com/artifact/io.projectreactor.netty/reactor-netty-http/1.0.22, Apache 2.0
DmJdbcDriver18 8.1.2.79: https://mvnrepository.com/artifact/com.dameng/DmJdbcDriver18/8.1.2.79, Apache 2.0
nimbus-jose-jwt 9.22: https://mvnrepository.com/artifact/com.nimbusds/nimbus-jose-jwt/9.22, Apache 2.0
woodstox-core 6.4.0: https://mvnrepository.com/artifact/com.fasterxml.woodstox/woodstox-core/6.4.0, Apache 2.0
jna-platform
@ -605,12 +607,42 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
bcpkix-jdk15on 1.68: https://mvnrepository.com/artifact/org.bouncycastle/bcpkix-jdk15on, MIT
bcprov-jdk15on 1.68: https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15on, MIT
reactive-streams 1.0.4: https://mvnrepository.com/artifact/org.reactivestreams/reactive-streams/1.0.4, MIT
webjars-locator-core 0.50 https://mvnrepository.com/artifact/org.webjars/webjars-locator-core/0.50, MIT
azure-core 1.34.0 https://mvnrepository.com/artifact/com.azure/azure-core/1.34.0, MIT
azure-core-http-netty 1.12.7 https://mvnrepository.com/artifact/com.azure/azure-core-http-netty/1.12.7, MIT
azure-identity 1.7.1 https://mvnrepository.com/artifact/com.azure/azure-identity/1.7.1, MIT
msal4j 1.13.3 https://mvnrepository.com/artifact/com.microsoft.azure/msal4j/1.13.3, MIT
msal4j-persistence-extension 1.1.0 https://mvnrepository.com/artifact/com.microsoft.azure/msal4j-persistence-extension/1.1.0, MIT
webjars-locator-core 0.50: https://mvnrepository.com/artifact/org.webjars/webjars-locator-core/0.50, MIT
azure-core 1.34.0: https://mvnrepository.com/artifact/com.azure/azure-core/1.34.0, MIT
azure-core-http-netty 1.12.7: https://mvnrepository.com/artifact/com.azure/azure-core-http-netty/1.12.7, MIT
azure-identity 1.7.1: https://mvnrepository.com/artifact/com.azure/azure-identity/1.7.1, MIT
msal4j 1.13.3: https://mvnrepository.com/artifact/com.microsoft.azure/msal4j/1.13.3, MIT
msal4j-persistence-extension 1.1.0: https://mvnrepository.com/artifact/com.microsoft.azure/msal4j-persistence-extension/1.1.0, MIT
azure-resourcemanager 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager/2.21.0, MIT
azure-resourcemanager-appplatform 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-appplatform/2.21.0, MIT
azure-resourcemanager-appservice 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-appservice/2.21.0, MIT
azure-resourcemanager-authorization 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-authorization/2.21.0, MIT
azure-resourcemanager-cdn 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-cdn/2.21.0, MIT
azure-resourcemanager-compute 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-compute/2.21.0, MIT
azure-resourcemanager-containerinstance 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-containerinstance/2.21.0, MIT
azure-resourcemanager-containerregistry 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-containerregistry/2.21.0, MIT
azure-resourcemanager-containerservice 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-containerservice/2.21.0, MIT
azure-resourcemanager-cosmos 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-cosmos/2.21.0, MIT
azure-resourcemanager-datafactory 1.0.0-beta.19 https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-datafactory/1.0.0-beta.19, MIT
azure-resourcemanager-dns 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-dns/2.21.0, MIT
azure-resourcemanager-eventhubs 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-eventhubs/2.21.0, MIT
azure-resourcemanager-keyvault 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-keyvault/2.21.0, MIT
azure-resourcemanager-monitor 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-monitor/2.21.0, MIT
azure-resourcemanager-msi 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-msi/2.21.0, MIT
azure-resourcemanager-network 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-network/2.21.0, MIT
azure-resourcemanager-privatedns 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-privatedns/2.21.0, MIT
azure-resourcemanager-redis 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-redis/2.21.0, MIT
azure-resourcemanager-resources 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-resources/2.21.0, MIT
azure-resourcemanager-search 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-search/2.21.0, MIT
azure-resourcemanager-servicebus 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-servicebus/2.21.0, MIT
azure-resourcemanager-sql 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-sql/2.21.0, MIT
azure-resourcemanager-storage 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-storage/2.21.0, MIT
azure-resourcemanager-trafficmanager 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-trafficmanager/2.21.0, MIT
azure-security-keyvault-keys 4.5.2: https://mvnrepository.com/artifact/com.azure/azure-security-keyvault-keys/4.5.2, MIT
azure-security-keyvault-secrets 4.5.2: https://mvnrepository.com/artifact/com.azure/azure-security-keyvault-secrets/4.5.2, MIT
azure-storage-common 12.19.1: https://mvnrepository.com/artifact/com.azure/azure-storage-common/12.19.1, MIT
azure-storage-file-share 12.16.1: https://mvnrepository.com/artifact/com.azure/azure-storage-file-share/12.16.1, MIT
azure-core-management 1.9.0: https://mvnrepository.com/artifact/com.azure/azure-core-management/1.9.0, MIT
========================================================================
MPL 1.1 licenses

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-core-management.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-appplatform.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-appservice.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-authorization.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-cdn.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-compute.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-containerinstance.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-containerregistry.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-containerservice.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-cosmos.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-datafactory.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-dns.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-eventhubs.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-keyvault.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-monitor.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-msi.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-network.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-privatedns.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-redis.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-resources.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-search.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-servicebus.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-sql.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-storage.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-trafficmanager.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-security-keyvault-keys.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-security-keyvault-secrets.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-storage-common.txt Normal file
View File

@ -0,0 +1,14 @@
LICENSE-msal4j.txtCopyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

14
dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-storage-file-share.txt Normal file
View File

@ -0,0 +1,14 @@
Copyright(c) 2016 Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"),
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

11
dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/resources/docker/file-manage/common.properties
View File

@ -26,6 +26,17 @@ resource.storage.type=S3
# resource store on HDFS/S3 path, resource file will store to this base path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended
resource.storage.upload.base.path=/dolphinscheduler
# The Azure client ID (Azure Application (client) ID)
resource.azure.client.id=minioadmin
# The Azure client secret in the Azure application
resource.azure.client.secret=minioadmin
# The Azure data factory subscription ID
resource.azure.subId=minioadmin
# The Azure tenant id in the Azure Active Directory
resource.azure.tenant.id=minioadmin
# The query interval
resource.query.interval=10000
# The AWS access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required
resource.aws.access.key.id=accessKey123
# The AWS secret access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required

5
dolphinscheduler-task-plugin/dolphinscheduler-task-all/pom.xml
View File

@ -212,6 +212,11 @@
<artifactId>dolphinscheduler-task-linkis</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-task-datafactory</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</project>

9
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/TaskConstants.java
View File

@ -443,6 +443,15 @@ public class TaskConstants {
public static final String TASK_TYPE_STREAM = "STREAM";
/**
* azure config
*/
public static final String AZURE_CLIENT_ID = "resource.azure.client.id";
public static final String AZURE_CLIENT_SECRET = "resource.azure.client.secret";
public static final String AZURE_ACCESS_SUB_ID = "resource.azure.subId";
public static final String AZURE_SECRET_TENANT_ID = "resource.azure.tenant.id";
public static final String QUERY_INTERVAL = "resource.query.interval";
/**
* aws config
*/

57
dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/pom.xml Normal file
View File

@ -0,0 +1,57 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-task-plugin</artifactId>
<version>dev-SNAPSHOT</version>
</parent>
<artifactId>dolphinscheduler-task-datafactory</artifactId>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-spi</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-task-api</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-common</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.azure.resourcemanager</groupId>
<artifactId>azure-resourcemanager-datafactory</artifactId>
<version>1.0.0-beta.19</version>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-identity</artifactId>
</dependency>
</dependencies>
</project>

148
dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryHook.java Normal file
View File

@ -0,0 +1,148 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.task.datafactory;
import org.apache.dolphinscheduler.common.utils.PropertyUtils;
import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
import org.apache.commons.lang3.StringUtils;
import java.util.Arrays;
import java.util.List;
import lombok.Data;
import lombok.SneakyThrows;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.azure.core.management.AzureEnvironment;
import com.azure.core.management.profile.AzureProfile;
import com.azure.identity.ClientSecretCredential;
import com.azure.identity.ClientSecretCredentialBuilder;
import com.azure.resourcemanager.datafactory.DataFactoryManager;
import com.azure.resourcemanager.datafactory.models.CreateRunResponse;
import com.azure.resourcemanager.datafactory.models.PipelineResource;
import com.azure.resourcemanager.datafactory.models.PipelineRun;
import com.azure.resourcemanager.datafactory.models.PipelineRuns;
@Data
public class DatafactoryHook {
public static DatafactoryStatus[] taskFinishFlags =
{DatafactoryStatus.Failed, DatafactoryStatus.Succeeded, DatafactoryStatus.Cancelled};
protected final Logger logger =
LoggerFactory.getLogger(String.format(TaskConstants.TASK_LOG_LOGGER_NAME_FORMAT, getClass()));
private final int QUERY_INTERVAL = PropertyUtils.getInt(TaskConstants.QUERY_INTERVAL, 10000);
private DataFactoryManager client;
private static AzureProfile profile;
private static ClientSecretCredential credential;
private String runId;
public DatafactoryHook() {
logger.info("initDatafactoryClient ......");
client = createClient();
}
protected static DataFactoryManager createClient() {
final String AZURE_ACCESS_SUB_ID = PropertyUtils.getString(TaskConstants.AZURE_ACCESS_SUB_ID);
final String AZURE_SECRET_TENANT_ID = PropertyUtils.getString(TaskConstants.AZURE_SECRET_TENANT_ID);
final String AZURE_CLIENT_ID = PropertyUtils.getString(TaskConstants.AZURE_CLIENT_ID);
final String AZURE_CLIENT_SECRET = PropertyUtils.getString(TaskConstants.AZURE_CLIENT_SECRET);
profile =
new AzureProfile(AZURE_SECRET_TENANT_ID, AZURE_ACCESS_SUB_ID, AzureEnvironment.AZURE);
credential = new ClientSecretCredentialBuilder()
.clientId(AZURE_CLIENT_ID)
.clientSecret(AZURE_CLIENT_SECRET)
.tenantId(AZURE_SECRET_TENANT_ID)
.authorityHost(profile.getEnvironment().getActiveDirectoryEndpoint())
.build();
return DataFactoryManager.authenticate(credential, profile);
}
public Boolean startDatafactoryTask(DatafactoryParameters parameters) {
logger.info("initDatafactoryTask ......");
PipelineResource pipelineResource = getPipelineResource(parameters);
if (pipelineResource == null) {
return false;
}
logger.info("startDatafactoryTask ......");
CreateRunResponse run = pipelineResource.createRun();
if (StringUtils.isEmpty(run.runId())) {
return false;
}
runId = run.runId();
parameters.setRunId(runId);
return true;
}
public Boolean cancelDatafactoryTask(DatafactoryParameters parameters) {
logger.info("cancelTask ......");
PipelineRuns pipelineRuns = client.pipelineRuns();
try {
pipelineRuns.cancel(parameters.getResourceGroupName(), parameters.getFactoryName(), runId);
} catch (RuntimeException e) {
logger.error("failed to cancel datafactory task: " + e.getMessage());
return false;
}
return true;
}
public DatafactoryStatus queryDatafactoryTaskStatus(DatafactoryParameters parameters) {
logger.info("queryDatafactoryTaskStatus ......");
PipelineRuns pipelineRuns = client.pipelineRuns();
PipelineRun pipelineRun =
pipelineRuns.get(parameters.getResourceGroupName(), parameters.getFactoryName(), runId);
if (pipelineRun != null) {
logger.info("queryDatafactoryTaskStatus ......{}", pipelineRun.status());
return DatafactoryStatus.valueOf(pipelineRun.status());
}
return null;
}
private PipelineResource getPipelineResource(DatafactoryParameters parameters) {
return client.pipelines().get(parameters.getResourceGroupName(), parameters.getFactoryName(),
parameters.getPipelineName());
}
@SneakyThrows
public Boolean queryStatus(DatafactoryParameters parameters) {
List<DatafactoryStatus> stopStatusSet = Arrays.asList(taskFinishFlags);
int maxRetry = 5;
while (maxRetry > 0) {
DatafactoryStatus status = queryDatafactoryTaskStatus(parameters);
if (status == null) {
maxRetry--;
continue;
}
if (stopStatusSet.contains(status)) {
if (status.equals(DatafactoryStatus.Succeeded)) {
return true;
}
return false;
}
logger.debug("wait {}ms to recheck finish status....", QUERY_INTERVAL);
Thread.sleep(QUERY_INTERVAL);
}
return false;
}
}

45
dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryParameters.java Normal file
View File

@ -0,0 +1,45 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.task.datafactory;
import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
import org.apache.commons.lang3.StringUtils;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import lombok.ToString;
@Setter
@Getter
@NoArgsConstructor
@ToString
public class DatafactoryParameters extends AbstractParameters {
private String factoryName;
private String resourceGroupName;
private String pipelineName;
private String runId;
@Override
public boolean checkParameters() {
return StringUtils.isNotEmpty(factoryName) && StringUtils.isNotEmpty(resourceGroupName)
&& StringUtils.isNotEmpty(pipelineName);
}
}

38
dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryStatus.java Normal file
View File

@ -0,0 +1,38 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.task.datafactory;
public enum DatafactoryStatus {
Queued,
InProgress,
Succeeded,
Failed,
Canceling,
Cancelled,
;
/**
* Gets the status property: The status of a pipeline run. Possible values: Queued, InProgress, Succeeded, Failed,
* Canceling, Cancelled.
*
* @return the status value.
*/
DatafactoryStatus() {
}
}

118
dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTask.java Normal file
View File

@ -0,0 +1,118 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.task.datafactory;
import org.apache.dolphinscheduler.common.utils.JSONUtils;
import org.apache.dolphinscheduler.plugin.task.api.AbstractRemoteTask;
import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
import org.apache.dolphinscheduler.plugin.task.api.TaskException;
import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
import org.apache.commons.lang3.StringUtils;
import java.util.Collections;
import java.util.List;
import lombok.Getter;
import lombok.Setter;
@Setter
@Getter
public class DatafactoryTask extends AbstractRemoteTask {
private final TaskExecutionContext taskExecutionContext;
private DatafactoryParameters parameters;
private DatafactoryHook hook;
public DatafactoryTask(TaskExecutionContext taskExecutionContext) {
super(taskExecutionContext);
this.taskExecutionContext = taskExecutionContext;
}
@Override
public List<String> getApplicationIds() throws TaskException {
return Collections.emptyList();
}
@Override
public void init() {
parameters = JSONUtils.parseObject(taskExecutionContext.getTaskParams(), DatafactoryParameters.class);
logger.info("Initialize Datafactory task params {}", JSONUtils.toPrettyJsonString(parameters));
hook = new DatafactoryHook();
}
@Override
public void submitApplication() throws TaskException {
try {
// start task
exitStatusCode = startDatafactoryTask();
setExitStatusCode(exitStatusCode);
} catch (Exception e) {
setExitStatusCode(TaskConstants.EXIT_CODE_FAILURE);
throw new TaskException("data factory start task error", e);
}
// set runId to the appIds if start success
setAppIds(parameters.getRunId());
}
@Override
public void cancelApplication() throws TaskException {
checkApplicationId();
hook.cancelDatafactoryTask(parameters);
exitStatusCode = TaskConstants.EXIT_CODE_KILL;
}
@Override
public void trackApplicationStatus() throws TaskException {
checkApplicationId();
Boolean isFinishedSuccessfully;
isFinishedSuccessfully = hook.queryStatus(parameters);
if (!isFinishedSuccessfully) {
exitStatusCode = TaskConstants.EXIT_CODE_FAILURE;
} else {
exitStatusCode = TaskConstants.EXIT_CODE_SUCCESS;
}
}
/**
* check datafactory applicationId or get it from appId
*/
private void checkApplicationId() {
String taskExecArn = hook.getRunId();
if (StringUtils.isEmpty(taskExecArn)) {
if (StringUtils.isEmpty(getAppIds())) {
throw new TaskException("datafactory runId is null, not created yet");
}
hook.setRunId(getAppIds());
}
}
public int startDatafactoryTask() {
Boolean isStartSuccessfully = hook.startDatafactoryTask(parameters);
if (!isStartSuccessfully) {
return TaskConstants.EXIT_CODE_FAILURE;
}
return TaskConstants.EXIT_CODE_SUCCESS;
}
@Override
public DatafactoryParameters getParameters() {
return parameters;
}
}

47
dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTaskChannel.java Normal file
View File

@ -0,0 +1,47 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.task.datafactory;
import org.apache.dolphinscheduler.common.utils.JSONUtils;
import org.apache.dolphinscheduler.plugin.task.api.TaskChannel;
import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
import org.apache.dolphinscheduler.plugin.task.api.parameters.ParametersNode;
import org.apache.dolphinscheduler.plugin.task.api.parameters.resource.ResourceParametersHelper;
public class DatafactoryTaskChannel implements TaskChannel {
@Override
public void cancelApplication(boolean status) {
}
@Override
public DatafactoryTask createTask(TaskExecutionContext taskRequest) {
return new DatafactoryTask(taskRequest);
}
@Override
public AbstractParameters parseParameters(ParametersNode parametersNode) {
return JSONUtils.parseObject(parametersNode.getTaskParams(), DatafactoryParameters.class);
}
@Override
public ResourceParametersHelper getResources(String parameters) {
return null;
}
}

46
dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTaskChannelFactory.java Normal file
View File

@ -0,0 +1,46 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.task.datafactory;
import org.apache.dolphinscheduler.plugin.task.api.TaskChannel;
import org.apache.dolphinscheduler.plugin.task.api.TaskChannelFactory;
import org.apache.dolphinscheduler.spi.params.base.PluginParams;
import java.util.Collections;
import java.util.List;
import com.google.auto.service.AutoService;
@AutoService(TaskChannelFactory.class)
public class DatafactoryTaskChannelFactory implements TaskChannelFactory {
@Override
public TaskChannel create() {
return new DatafactoryTaskChannel();
}
@Override
public String getName() {
return "DATA_FACTORY";
}
@Override
public List<PluginParams> getParams() {
return Collections.emptyList();
}
}

120
dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/test/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTaskTest.java Normal file
View File

@ -0,0 +1,120 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.task.datafactory;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.mockStatic;
import static org.mockito.Mockito.when;
import org.apache.dolphinscheduler.common.utils.JSONUtils;
import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.MockedStatic;
import org.mockito.Mockito;
import org.mockito.junit.jupiter.MockitoExtension;
import com.azure.resourcemanager.datafactory.DataFactoryManager;
import com.azure.resourcemanager.datafactory.models.CreateRunResponse;
import com.azure.resourcemanager.datafactory.models.PipelineResource;
import com.azure.resourcemanager.datafactory.models.Pipelines;
@ExtendWith(MockitoExtension.class)
public class DatafactoryTaskTest {
private static final String mockRunId =
"3c7182f4-d107-43c7-af2a-8c7b3ed1d614";
DatafactoryTask datafactoryTask;
@Mock
DataFactoryManager client;
MockedStatic<DatafactoryHook> datafactoryHookMockedStatic;
@BeforeEach
public void before() throws IllegalAccessException {
client = mock(DataFactoryManager.class);
datafactoryHookMockedStatic = mockStatic(DatafactoryHook.class);
when(DatafactoryHook.createClient()).thenReturn(client);
DatafactoryParameters datafactoryParameters = new DatafactoryParameters();
datafactoryTask = initTask(datafactoryParameters);
}
@Test
public void testConvertParam() {
DatafactoryParameters parameters = castParam();
Assertions.assertEquals("whale1", parameters.getFactoryName());
Assertions.assertEquals("factory1", parameters.getResourceGroupName());
Assertions.assertEquals("CopyPipeline_ps8", parameters.getPipelineName());
datafactoryHookMockedStatic.close();
}
@Test
public void testCheckCreateTask() {
DatafactoryHook hook = new DatafactoryHook();
DataFactoryManager client1 = DatafactoryHook.createClient();
hook.setClient(client1);
PipelineResource pipelineResource = mock(PipelineResource.class);
Pipelines pipelines = mock(Pipelines.class);
when(client1.pipelines()).thenReturn(pipelines);
when(pipelines.get(any(), any(), any())).thenReturn(pipelineResource);
CreateRunResponse response = mock(CreateRunResponse.class);
when(pipelineResource.createRun()).thenReturn(response);
when(response.runId()).thenReturn(mockRunId);
hook.startDatafactoryTask(castParam());
Assertions.assertEquals(mockRunId, hook.getRunId());
datafactoryHookMockedStatic.close();
}
@Test
public void testCancelTask() {
datafactoryTask.cancelApplication();
Assertions.assertEquals(datafactoryTask.getExitStatusCode(), TaskConstants.EXIT_CODE_KILL);
datafactoryHookMockedStatic.close();
}
private DatafactoryParameters castParam() {
String input =
"{\"localParams\":[],\"resourceList\":[],\"factoryName\":\"whale1\",\"resourceGroupName\":\"factory1\",\"pipelineName\":\"CopyPipeline_ps8\"}";
return JSONUtils.parseObject(input, DatafactoryParameters.class);
}
private DatafactoryTask initTask(DatafactoryParameters DatasyncParameters) {
TaskExecutionContext taskExecutionContext = createContext(DatasyncParameters);
DatafactoryTask task = new DatafactoryTask(taskExecutionContext);
task.init();
task.getHook().setRunId(mockRunId);
return task;
}
public TaskExecutionContext createContext(DatafactoryParameters datafactoryParameters) {
String parameters = JSONUtils.toJsonString(datafactoryParameters);
TaskExecutionContext taskExecutionContext = Mockito.mock(TaskExecutionContext.class);
Mockito.when(taskExecutionContext.getTaskParams()).thenReturn(parameters);
return taskExecutionContext;
}
}

1
dolphinscheduler-task-plugin/pom.xml
View File

@ -61,6 +61,7 @@
<module>dolphinscheduler-task-datasync</module>
<module>dolphinscheduler-task-kubeflow</module>
<module>dolphinscheduler-task-linkis</module>
<module>dolphinscheduler-task-datafactory</module>
</modules>
<dependencyManagement>

BIN
dolphinscheduler-ui/public/images/task-icons/data_factory.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.8 KiB

BIN
dolphinscheduler-ui/public/images/task-icons/data_factory_hover.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.6 KiB

21
dolphinscheduler-ui/src/locales/en_US/project.ts
View File

@ -37,10 +37,10 @@ export default {
confirm: 'Confirm',
cancel: 'Cancel',
delete_confirm: 'Delete?',
authorize_level:'Authorize Level',
authorize_level: 'Authorize Level',
no_permission: 'No Permission',
read_permission: 'Read Permission',
all_permission: 'All Permission',
all_permission: 'All Permission'
},
workflow: {
on_line: 'Online',
@ -215,7 +215,7 @@ export default {
workflow_relation_no_data_result_desc:
'There is not any workflows. Please create a workflow, and then visit this page again.',
ready_to_block: 'Ready to block',
block: 'Block',
block: 'Block'
},
task: {
on_line: 'Online',
@ -329,7 +329,7 @@ export default {
online: 'Online'
},
node: {
is_cache: "Cache Execution",
is_cache: 'Cache Execution',
jvm_args: 'Java VM Parameters',
jvm_args_tips: 'Please enter virtual machine parameters',
run_type: 'Run Type',
@ -377,7 +377,8 @@ export default {
image: 'Image',
image_tips: 'Please enter image',
command: 'Command',
command_tips: 'Please enter the container execution command, for example: /bin/echo hello world',
command_tips:
'Please enter the container execution command, for example: /bin/echo hello world',
min_memory_tips: 'Please enter min memory',
state: 'State',
branch_flow: 'Branch flow',
@ -798,7 +799,13 @@ export default {
pytorch_requirements: 'Requirement File',
pytorch_conda_python_version: 'Python Version',
pytorch_conda_python_version_tips:
'Please enter the version number, such as 3.6, 3.7, 3.x'
'Please enter the version number, such as 3.6, 3.7, 3.x',
factory_name: 'Factory Name',
resource_group_name: 'Resource Group Name',
pipeline_name: 'Pipeline Name',
factory_tips: 'Please select factory',
resource_group_tips: 'Please select resource group',
pipeline_tips: 'Please select pipeline'
},
menu: {
fav: 'Favorites',
@ -808,6 +815,6 @@ export default {
di: 'Data Integration',
dq: 'Data Quality',
ml: 'Machine Learning',
other: 'Other',
other: 'Other'
}
}

8
dolphinscheduler-ui/src/locales/zh_CN/project.ts
View File

@ -777,7 +777,13 @@ export default {
pytorch_python_env_tool: 'python环境管理工具',
pytorch_requirements: '依赖文件',
pytorch_conda_python_version: 'python版本',
pytorch_conda_python_version_tips: '请输入版本号，如 3.6, 3.7, 3.x等'
pytorch_conda_python_version_tips: '请输入版本号，如 3.6, 3.7, 3.x等',
factory_name: '工厂名称',
resource_group_name: '资源组名称',
pipeline_name: 'pipeline名称',
factory_tips: '请选择工厂',
resource_group_tips: '请选择资源组',
pipeline_tips: '请选择pipeline'
},
menu: {
fav: '收藏组件',

41
dolphinscheduler-ui/src/service/modules/azure/index.ts Normal file
View File

@ -0,0 +1,41 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { axios } from '@/service/service'
import { DataFactoryReq } from './types'
export function queryDataFactoryFactories(): any {
return axios({
url: '/cloud/azure/datafactory/factories',
method: 'get'
})
}
export function queryDataFactoryResourceGroups(): any {
return axios({
url: '/cloud/azure/datafactory/resourceGroups',
method: 'get'
})
}
export function queryDataFactoryPipelines(params: DataFactoryReq): any {
return axios({
url: '/cloud/azure/datafactory/pipelines',
method: 'get',
params
})
}

23
dolphinscheduler-ui/src/service/modules/azure/types.ts Normal file
View File

@ -0,0 +1,23 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
interface DataFactoryReq {
factoryName: string
resourceGroupName: string
}
export { DataFactoryReq }

4
dolphinscheduler-ui/src/store/project/task-type.ts
View File

@ -149,6 +149,10 @@ export const TASK_TYPES_MAP = {
LINKIS: {
alias: 'LINKIS',
helperLinkDisable: true
},
DATA_FACTORY: {
alias: 'DATA_FACTORY',
helperLinkDisable: true
}
} as {
[key in TaskType]: {

1
dolphinscheduler-ui/src/store/project/types.ts
View File

@ -56,6 +56,7 @@ type TaskType =
| 'DATASYNC'
| 'KUBEFLOW'
| 'LINKIS'
| 'DATA_FACTORY'
type ProgramType = 'JAVA' | 'SCALA' | 'PYTHON'

1
dolphinscheduler-ui/src/views/projects/task/components/node/fields/index.ts
View File

@ -85,3 +85,4 @@ export { useDms } from './use-dms'
export { useDatasync } from './use-datasync'
export { useKubeflow } from './use-kubeflow'
export { useLinkis } from './use-linkis'
export { useDataFactory } from './use-data-factory'

152
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-data-factory.ts Normal file
View File

@ -0,0 +1,152 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import {
queryDataFactoryFactories,
queryDataFactoryPipelines,
queryDataFactoryResourceGroups
} from '@/service/modules/azure'
import { onMounted, ref, watch } from 'vue'
import { useI18n } from 'vue-i18n'
import { useCustomParams } from '.'
import type { IJsonItem } from '../types'
export function useDataFactory(model: { [field: string]: any }): IJsonItem[] {
const { t } = useI18n()
const factoryLoading = ref(false)
const resourceGroupLoading = ref(false)
const pipelineLoading = ref(false)
const factoryOptions = ref([] as { label: string; value: number }[])
const resourceGroupOptions = ref([] as { label: string; value: number }[])
const pipelineOptions = ref([] as { label: string; value: number }[])
const getFactoryOptions = async () => {
if (factoryLoading.value) return
factoryLoading.value = true
const factories = await queryDataFactoryFactories()
factoryOptions.value = factories.map((factory: string) => ({
label: factory,
value: factory
}))
factoryLoading.value = false
}
const getResourceGroupName = async () => {
if (resourceGroupLoading.value) return
resourceGroupLoading.value = true
const groupNames = await queryDataFactoryResourceGroups()
resourceGroupOptions.value = groupNames.map((groupName: string) => ({
label: groupName,
value: groupName
}))
resourceGroupLoading.value = false
}
const getPipelineName = async (
factoryName: string,
resourceGroupName: string
) => {
if (pipelineLoading.value) return
pipelineLoading.value = true
const pipelineNames = await queryDataFactoryPipelines({
factoryName,
resourceGroupName
})
pipelineOptions.value = pipelineNames.map((pipelineName: string) => ({
label: pipelineName,
value: pipelineName
}))
pipelineLoading.value = false
}
const onChange = () => {
model['pipelineName'] = ''
if (model['factoryName'] && model['resourceGroupName']) {
getPipelineName(model['factoryName'], model['resourceGroupName'])
}
}
watch(
() => model['pipelineName'],
() => {
if (model['pipelineName'] && pipelineOptions.value.length === 0) {
getPipelineName(model['factoryName'], model['resourceGroupName'])
}
}
)
onMounted(() => {
getFactoryOptions()
getResourceGroupName()
})
return [
{
type: 'select',
field: 'factoryName',
span: 24,
name: t('project.node.factory_name'),
options: factoryOptions,
props: {
'on-update:value': onChange,
loading: factoryLoading
},
validate: {
trigger: ['input', 'trigger'],
required: true,
message: t('project.node.factory_tips')
}
},
{
type: 'select',
field: 'resourceGroupName',
span: 24,
name: t('project.node.resource_group_name'),
options: resourceGroupOptions,
props: {
'on-update:value': onChange,
loading: resourceGroupLoading
},
validate: {
trigger: ['input', 'trigger'],
required: true,
message: t('project.node.resource_group_tips')
}
},
{
type: 'select',
field: 'pipelineName',
span: 24,
name: t('project.node.pipeline_name'),
options: pipelineOptions,
props: {
loading: pipelineLoading
},
validate: {
trigger: ['input', 'trigger'],
required: true,
message: t('project.node.pipeline_tips')
}
},
...useCustomParams({
model,
field: 'localParams',
isSimple: true
})
]
}

8
dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts
View File

@ -449,13 +449,19 @@ export function formatParams(data: INodeData): {
taskParams.yamlContent = data.yamlContent
taskParams.namespace = data.namespace
}
if (data.taskType === 'LINKIS') {
taskParams.useCustom = data.useCustom
taskParams.paramScript = data.paramScript
taskParams.rawScript = data.rawScript
}
if (data.taskType === 'DATA_FACTORY') {
taskParams.factoryName = data.factoryName
taskParams.resourceGroupName = data.resourceGroupName
taskParams.pipelineName = data.pipelineName
}
let timeoutNotifyStrategy = ''
if (data.timeoutNotifyStrategy) {
if (data.timeoutNotifyStrategy.length === 1) {

4
dolphinscheduler-ui/src/views/projects/task/components/node/tasks/index.ts
View File

@ -50,6 +50,7 @@ import { useDms } from './use-dms'
import { useDatasync } from './use-datasync'
import { useKubeflow } from './use-kubeflow'
import { useLinkis } from './use-linkis'
import { useDataFactory } from './use-data-factory'
export default {
SHELL: useShell,
@ -86,5 +87,6 @@ export default {
DMS: useDms,
DATASYNC: useDatasync,
KUBEFLOW: useKubeflow,
LINKIS: useLinkis
LINKIS: useLinkis,
DATA_FACTORY: useDataFactory
}

72
dolphinscheduler-ui/src/views/projects/task/components/node/tasks/use-data-factory.ts Normal file
View File

@ -0,0 +1,72 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { reactive } from 'vue'
import * as Fields from '../fields/index'
import type { IJsonItem, INodeData } from '../types'
import { ITaskData } from '../types'
export function useDataFactory({
projectCode,
from = 0,
readonly,
data
}: {
projectCode: number
from?: number
readonly?: boolean
data?: ITaskData
}) {
const model = reactive({
name: '',
taskType: 'DATA_FACTORY',
flag: 'YES',
description: '',
timeoutFlag: false,
localParams: [],
environmentCode: null,
failRetryInterval: 1,
failRetryTimes: 0,
workerGroup: 'default',
delayTime: 0,
timeout: 30,
timeoutNotifyStrategy: ['WARN'],
factoryName: '',
resourceGroupName: '',
pipelineName: ''
} as INodeData)
return {
json: [
Fields.useName(from),
...Fields.useTaskDefinition({ projectCode, from, readonly, data, model }),
Fields.useRunFlag(),
Fields.useCache(),
Fields.useDescription(),
Fields.useTaskPriority(),
Fields.useWorkerGroup(),
Fields.useEnvironmentName(model, !data?.id),
...Fields.useTaskGroup(model, projectCode),
...Fields.useFailed(),
Fields.useDelayTime(model),
...Fields.useTimeoutAlarm(model),
...Fields.useDataFactory(model),
Fields.usePreTasks()
] as IJsonItem[],
model
}
}

3
dolphinscheduler-ui/src/views/projects/task/components/node/types.ts
View File

@ -405,6 +405,9 @@ interface ITaskParams {
cloudWatchLogGroupArn?: string
yamlContent?: string
paramScript?: ILocalParam[]
factoryName?: string
resourceGroupName?: string
pipelineName?: string
}
interface INodeData

5
dolphinscheduler-ui/src/views/projects/task/constants/task-type.ts
View File

@ -50,6 +50,7 @@ export type TaskType =
| 'DATASYNC'
| 'KUBEFLOW'
| 'LINKIS'
| 'DATA_FACTORY'
export type TaskExecuteType = 'STREAM' | 'BATCH'
@ -180,6 +181,10 @@ export const TASK_TYPES_MAP = {
LINKIS: {
alias: 'LINKIS',
helperLinkDisable: true
},
DATA_FACTORY: {
alias: 'DATA_FACTORY',
helperLinkDisable: true
}
} as {
[key in TaskType]: {

6
dolphinscheduler-ui/src/views/projects/workflow/components/dag/dag.module.scss
View File

@ -204,6 +204,9 @@ $bgLight: #ffffff;
&.icon-kubeflow {
background-image: url('/images/task-icons/kubeflow.png');
}
&.icon-data_factory {
background-image: url('/images/task-icons/data_factory.png');
}
}
&:hover {
@ -311,6 +314,9 @@ $bgLight: #ffffff;
&.icon-kubeflow {
background-image: url('/images/task-icons/kubeflow_hover.png');
}
&.icon-data_factory {
background-image: url('/images/task-icons/data_factory_hover.png');
}
}
}

32
tools/dependencies/known-dependencies.txt
View File

@ -399,3 +399,35 @@ reactor-core-3.4.22.jar
reactor-netty-core-1.0.22.jar
reactor-netty-http-1.0.22.jar
DmJdbcDriver18-8.1.2.79.jar
azure-resourcemanager-2.21.0.jar
azure-resourcemanager-appplatform-2.21.0.jar
azure-resourcemanager-appservice-2.21.0.jar
azure-resourcemanager-authorization-2.21.0.jar
azure-resourcemanager-cdn-2.21.0.jar
azure-resourcemanager-compute-2.21.0.jar
azure-resourcemanager-containerinstance-2.21.0.jar
azure-resourcemanager-containerregistry-2.21.0.jar
azure-resourcemanager-containerservice-2.21.0.jar
azure-resourcemanager-cosmos-2.21.0.jar
azure-resourcemanager-datafactory-1.0.0-beta.19.jar
azure-resourcemanager-dns-2.21.0.jar
azure-resourcemanager-eventhubs-2.21.0.jar
azure-resourcemanager-keyvault-2.21.0.jar
azure-resourcemanager-monitor-2.21.0.jar
azure-resourcemanager-msi-2.21.0.jar
azure-resourcemanager-network-2.21.0.jar
azure-resourcemanager-privatedns-2.21.0.jar
azure-resourcemanager-redis-2.21.0.jar
azure-resourcemanager-resources-2.21.0.jar
azure-resourcemanager-search-2.21.0.jar
azure-resourcemanager-servicebus-2.21.0.jar
azure-resourcemanager-sql-2.21.0.jar
azure-resourcemanager-storage-2.21.0.jar
azure-resourcemanager-trafficmanager-2.21.0.jar
azure-security-keyvault-keys-4.5.2.jar
azure-security-keyvault-secrets-4.5.2.jar
azure-storage-common-12.19.1.jar
azure-storage-file-share-12.16.1.jar
nimbus-jose-jwt-9.22.jar
woodstox-core-6.4.0.jar
azure-core-management-1.9.0.jar