diff --git a/docs/docs/en/guide/task/datafactory.md b/docs/docs/en/guide/task/datafactory.md new file mode 100644 index 0000000000..1936db81f1 --- /dev/null +++ b/docs/docs/en/guide/task/datafactory.md @@ -0,0 +1,70 @@ +# Azure DataFactory Node + +## Overview + +In the world of big data, raw, unorganized data is often stored in relational, non-relational, and other storage systems.[Azure DataFactory](https://learn.microsoft.com/en-us/azure/data-factory/introduction) is a managed cloud service that's built for these complex hybrid extract-transform-load (ETL), extract-load-transform (ELT), and data integration projects. + +DolphinScheduler DataFactory functions: + +- Create Azure DataFactory task enable DolphinScheduler to schedule DataFactory pipelines, keep query pipeline execution status until finish the execution. + +## Prerequisite + +- **ResourceGroup**: Azure owns a resource group +- **DataFactory**: Azure owns a data factory under the resource group +- **Pipeline**: Azure owns pipelines under the assigned resource group and data factory +- **Application**: Azure owns an application has permission to visit data factory, enables using SDK to call data factory functions +- **ApplicationClientSecret**: To apply for client secret under Application `Certificates & secrets` +- **AZURE-CLI**: Install Azure auth application AZURE-CLI on machine, please refer to [Install the Azure CLI on Linux](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux) + +### Application Permission Setting + +First, visit the `Subcription` page and choose `Access control (IAM)`, then click `Add role assignment` to the authorization page. +![Subcription-IAM](../../../../img/tasks/demo/datafactory_auth1.png) +After that, select `Contributor` role which satisfy functions calls in data factory. Then click `Members` page, and click `Select members`. +Search application name or application `Object ID` to assign `Contributor` role to application. +![Subcription-Role](../../../../img/tasks/demo/datafactory_auth2.png) + +## Configurations + +Config Azure configurations, modify `azure` related configurations in `common.properties` +- **resource.azure.client.id**: Azure Application Application (client) ID +- **resource.azure.client.secret**: Azure Application client secret under `Certificates & secrets` +- **resource.azure.subId**: Data factory subscription ID +- **resource.azure.tenant.id**: Azure Active Directory tenant ID + +```yaml +# The Azure client ID (Azure Application (client) ID) +resource.azure.client.id=minioadmin +# The Azure client secret in the Azure application +resource.azure.client.secret=minioadmin +# The Azure data factory subscription ID +resource.azure.subId=minioadmin +# The Azure tenant ID in the Azure Active Directory +resource.azure.tenant.id=minioadmin + +``` + +## Create Task + +- Click `Project -> Management-Project -> Name-Workflow Definition`, and click the "Create Workflow" button to enter the + DAG editing page. +- Drag from the toolbar task node to canvas. + +## Task Example + +[//]: # (TODO: use the commented anchor below once our website template supports this syntax) +[//]: # (- Please refer to [DolphinScheduler Task Parameters Appendix](appendix.md#default-task-parameters) `Default Task Parameters` section for default parameters.) + +- Please refer to [DolphinScheduler Task Parameters Appendix](appendix.md) `Default Task Parameters` section for default parameters. + +Here are some specific parameters for the DataFactory plugin: + +- **factoryName**: data factory name +- **resourceGroupName**: resource group name of the data factory +- **pipelineName**: pipeline name under the resource group and data factory + +The following shows the task plugin example: + +![data-factory](../../../../img/tasks/demo/datafactory.png) + diff --git a/docs/docs/zh/guide/task/datafactory.md b/docs/docs/zh/guide/task/datafactory.md new file mode 100644 index 0000000000..0fa3375bc1 --- /dev/null +++ b/docs/docs/zh/guide/task/datafactory.md @@ -0,0 +1,68 @@ +# Azure DataFactory 节点 + +## 综述 + +在大数据环境中，原始、散乱的数据通常存储在关系、非关系和其他存储系统中。[Azure DataFactory](https://learn.microsoft.com/en-us/azure/data-factory/introduction) 是为这些复杂的混合提取-转换-加载 (ETL)、提取-加载-转换 (ELT) 和数据集成项目而构建的托管云服务。 + +DolphinScheduler DataFactory 组件的功能: + +- 创建 Azure DataFactory 任务可以调度DataFactory中的pipeline，持续获取执行状态，直至任务执行完成。 + +## 前置条件 + +- **资源组**: 拥有一个资源组 +- **数据工厂**: 拥有一个资源组下的数据工厂 +- **Pipeline**: 拥有前两者对应的pipeline +- **应用**: 拥有一个有数据工厂访问权限的应用，才能赋予SDK调用数据工厂 +- **应用客户端密钥**: 在应用的`Certificates & secrets`申请客户端密钥 +- **AZURE-CLI**: 在机器上安装了AZURE的鉴权应用AZURE-CLI，请参考[在LINUX上安装AZURE-CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux) + +### 应用权限设置 + +首先打开当前`Subcription`页面，点击`Access control (IAM)`，再点击`Add role assignment`进入授权页面。 +![Subcription-IAM](../../../../img/tasks/demo/datafactory_auth1.png) +首先选择`Contributor`角色足够满足调用数据工厂。然后选择`Members`页面，再选择`Select members`，检索APP名称或APP的`Object ID`并添加，从给指定APP添加权限. +![Subcription-Role](../../../../img/tasks/demo/datafactory_auth2.png) + +## 环境配置 + +需要进行Azure的一些配置，修改`common.properties`中的`azure`相关配置信息 +- **resource.azure.client.id**: Azure Application应用的Application (client) ID +- **resource.azure.client.secret**: Azure Application应用的`Certificates & secrets`下的客户端密钥 +- **resource.azure.subId**: 数据工厂的subscription ID +- **resource.azure.tenant.id**: Azure Active Directory下的tenant ID + +```yaml +# The Azure client ID (Azure Application (client) ID) +resource.azure.client.id=minioadmin +# The Azure client secret in the Azure application +resource.azure.client.secret=minioadmin +# The Azure data factory subscription ID +resource.azure.subId=minioadmin +# The Azure tenant ID in the Azure Active Directory +resource.azure.tenant.id=minioadmin + +``` + +## 创建任务 + +- 点击项目管理-项目名称-工作流定义，点击“创建工作流”按钮，进入 DAG 编辑页面； +- 拖动工具栏的 任务节点到画板中。 + +## 任务样例 + +[//]: # (TODO: use the commented anchor below once our website template supports this syntax) +[//]: # (- 默认参数说明请参考[DolphinScheduler任务参数附录](appendix.md#默认任务参数)`默认任务参数`一栏。) + +- 默认参数说明请参考[DolphinScheduler任务参数附录](appendix.md)`默认任务参数`一栏。 + +### 独有参数 + +- **factoryName**: 数据工厂名称 +- **resourceGroupName**: 与数据工厂相关的资源组名称 +- **pipelineName**: 选取数据工厂以及资源组下的对应pipeline名称 + +组件图示如下： + +![data-factory](../../../../img/tasks/demo/datafactory.png) + diff --git a/docs/img/tasks/demo/datafactory.png b/docs/img/tasks/demo/datafactory.png new file mode 100644 index 0000000000..711f8982ae Binary files /dev/null and b/docs/img/tasks/demo/datafactory.png differ diff --git a/docs/img/tasks/demo/datafactory_auth1.png b/docs/img/tasks/demo/datafactory_auth1.png new file mode 100644 index 0000000000..5e4bd71500 Binary files /dev/null and b/docs/img/tasks/demo/datafactory_auth1.png differ diff --git a/docs/img/tasks/demo/datafactory_auth2.png b/docs/img/tasks/demo/datafactory_auth2.png new file mode 100644 index 0000000000..1a135191ce Binary files /dev/null and b/docs/img/tasks/demo/datafactory_auth2.png differ diff --git a/docs/img/tasks/icons/datafactory.png b/docs/img/tasks/icons/datafactory.png new file mode 100644 index 0000000000..9cebaceee4 Binary files /dev/null and b/docs/img/tasks/icons/datafactory.png differ diff --git a/dolphinscheduler-api/pom.xml b/dolphinscheduler-api/pom.xml index 8badbf4e74..9696ee7e2f 100644 --- a/dolphinscheduler-api/pom.xml +++ b/dolphinscheduler-api/pom.xml @@ -198,6 +198,14 @@ org.springdoc springdoc-openapi-ui + + com.azure.resourcemanager + azure-resourcemanager + + + com.azure.resourcemanager + azure-resourcemanager-datafactory + diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/CloudController.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/CloudController.java new file mode 100644 index 0000000000..a5d73748cb --- /dev/null +++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/CloudController.java @@ -0,0 +1,108 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.dolphinscheduler.api.controller; + +import static org.apache.dolphinscheduler.api.enums.Status.LIST_AZURE_DATA_FACTORY_ERROR; +import static org.apache.dolphinscheduler.api.enums.Status.LIST_AZURE_DATA_FACTORY_PIPELINE_ERROR; +import static org.apache.dolphinscheduler.api.enums.Status.LIST_AZURE_RESOURCE_GROUP_ERROR; + +import org.apache.dolphinscheduler.api.aspect.AccessLogAnnotation; +import org.apache.dolphinscheduler.api.enums.Status; +import org.apache.dolphinscheduler.api.exceptions.ApiException; +import org.apache.dolphinscheduler.api.service.CloudService; +import org.apache.dolphinscheduler.api.utils.Result; +import org.apache.dolphinscheduler.common.constants.Constants; +import org.apache.dolphinscheduler.dao.entity.User; + +import java.util.List; + +import javax.annotation.Resource; + +import org.springframework.http.HttpStatus; +import org.springframework.web.bind.annotation.GetMapping; +import org.springframework.web.bind.annotation.RequestAttribute; +import org.springframework.web.bind.annotation.RequestMapping; +import org.springframework.web.bind.annotation.RequestParam; +import org.springframework.web.bind.annotation.ResponseStatus; +import org.springframework.web.bind.annotation.RestController; + +import io.swagger.v3.oas.annotations.Operation; +import io.swagger.v3.oas.annotations.Parameter; +import io.swagger.v3.oas.annotations.tags.Tag; + +/** + * cloud controller + */ +@Tag(name = "CLOUD_TAG") +@RestController +@RequestMapping("/cloud") +public class CloudController extends BaseController { + + @Resource + private CloudService cloudService; + + /** + * get datafactory list + * + * @param loginUser login user + * @return datafactory name list + */ + @Operation(summary = "listDataFactory", description = "LIST_DATA_FACTORY") + @GetMapping(value = "/azure/datafactory/factories") + @ResponseStatus(HttpStatus.OK) + @ApiException(LIST_AZURE_DATA_FACTORY_ERROR) + @AccessLogAnnotation(ignoreRequestArgs = "loginUser") + public Result listDataFactory(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser) { + List factoryNames = cloudService.listDataFactory(loginUser); + return success(Status.SUCCESS.getMsg(), factoryNames); + } + + /** + * get resourceGroup list + * + * @param loginUser login user + * @return resourceGroup list + */ + @Operation(summary = "listResourceGroup", description = "LIST_RESOURCE_GROUP") + @GetMapping(value = "/azure/datafactory/resourceGroups") + @ResponseStatus(HttpStatus.OK) + @ApiException(LIST_AZURE_RESOURCE_GROUP_ERROR) + @AccessLogAnnotation(ignoreRequestArgs = "loginUser") + public Result listResourceGroup(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser) { + List resourceGroupNames = cloudService.listResourceGroup(loginUser); + return success(Status.SUCCESS.getMsg(), resourceGroupNames); + } + + /** + * get resourceGroup list + * + * @param loginUser login user + * @return resourceGroup list + */ + @Operation(summary = "listPipeline", description = "LIST_PIPELINE") + @GetMapping(value = "/azure/datafactory/pipelines") + @ResponseStatus(HttpStatus.OK) + @ApiException(LIST_AZURE_DATA_FACTORY_PIPELINE_ERROR) + @AccessLogAnnotation(ignoreRequestArgs = "loginUser") + public Result listPipeline(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser, + @RequestParam("factoryName") String factoryName, + @RequestParam("resourceGroupName") String resourceGroupName) { + List pipelineNames = cloudService.listPipeline(loginUser, factoryName, resourceGroupName); + return success(Status.SUCCESS.getMsg(), pipelineNames); + } +} diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java index d60a867d77..97db8eb78f 100644 --- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java +++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java @@ -265,6 +265,9 @@ public enum Status { "请先保存后再执行"), START_NODE_NOT_EXIST_IN_LAST_PROCESS(10207, "this node {0} does not exist in the latest process definition", "该节点 {0} 不存在于最新的流程定义中"), + LIST_AZURE_DATA_FACTORY_ERROR(10208, "list azure data factory error", "查询AZURE数据工厂列表错误"), + LIST_AZURE_RESOURCE_GROUP_ERROR(10209, "list azure resource group error", "查询AZURE资源组列表错误"), + LIST_AZURE_DATA_FACTORY_PIPELINE_ERROR(10210, "list azure data factory pipeline error", "查询AZURE数据工厂pipeline列表错误"), UDF_FUNCTION_NOT_EXIST(20001, "UDF function not found", "UDF函数不存在"), UDF_FUNCTION_EXISTS(20002, "UDF function already exists", "UDF函数已存在"), diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/CloudService.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/CloudService.java new file mode 100644 index 0000000000..057974bf11 --- /dev/null +++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/CloudService.java @@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.dolphinscheduler.api.service; + +import org.apache.dolphinscheduler.dao.entity.User; + +import java.util.List; + +public interface CloudService { + + List listDataFactory(User loginUser); + + List listResourceGroup(User loginUser); + + List listPipeline(User loginUser, String factoryName, String resourceGroupName); +} diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/CloudServiceImpl.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/CloudServiceImpl.java new file mode 100644 index 0000000000..38f7351ce8 --- /dev/null +++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/CloudServiceImpl.java @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.dolphinscheduler.api.service.impl; + +import org.apache.dolphinscheduler.api.service.CloudService; +import org.apache.dolphinscheduler.common.utils.PropertyUtils; +import org.apache.dolphinscheduler.dao.entity.User; +import org.apache.dolphinscheduler.plugin.task.api.TaskConstants; + +import java.util.ArrayList; +import java.util.List; + +import org.springframework.stereotype.Service; + +import com.azure.core.http.rest.PagedIterable; +import com.azure.core.management.AzureEnvironment; +import com.azure.core.management.profile.AzureProfile; +import com.azure.identity.ClientSecretCredential; +import com.azure.identity.ClientSecretCredentialBuilder; +import com.azure.resourcemanager.AzureResourceManager; +import com.azure.resourcemanager.datafactory.DataFactoryManager; +import com.azure.resourcemanager.datafactory.models.Factories; +import com.azure.resourcemanager.datafactory.models.PipelineResource; +import com.azure.resourcemanager.resources.models.ResourceGroups; + +@Service +public class CloudServiceImpl extends BaseServiceImpl implements CloudService { + + private static final String AZURE_CLIENT_ID = PropertyUtils.getString(TaskConstants.AZURE_CLIENT_ID); + private static final String AZURE_CLIENT_SECRET = PropertyUtils.getString(TaskConstants.AZURE_CLIENT_SECRET); + private static final String AZURE_ACCESS_SUB_ID = PropertyUtils.getString(TaskConstants.AZURE_ACCESS_SUB_ID); + private static final String AZURE_SECRET_TENANT_ID = PropertyUtils.getString(TaskConstants.AZURE_SECRET_TENANT_ID); + private static final AzureProfile profile = + new AzureProfile(AZURE_SECRET_TENANT_ID, AZURE_ACCESS_SUB_ID, AzureEnvironment.AZURE); + private static final ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder() + .clientId(AZURE_CLIENT_ID) + .clientSecret(AZURE_CLIENT_SECRET) + .tenantId(AZURE_SECRET_TENANT_ID) + .authorityHost(profile.getEnvironment().getActiveDirectoryEndpoint()) + .build(); + private static final DataFactoryManager manager; + private static final AzureResourceManager azure; + + static { + manager = DataFactoryManager.authenticate(clientSecretCredential, profile); + azure = AzureResourceManager.authenticate(clientSecretCredential, profile).withDefaultSubscription(); + } + + @Override + public List listDataFactory(User loginUser) { + Factories factories = manager.factories(); + List names = new ArrayList<>(); + factories.list().stream().forEach(e -> names.add(e.name())); + return names; + } + + @Override + public List listResourceGroup(User loginUser) { + ResourceGroups resourceGroups = azure.resourceGroups(); + List names = new ArrayList<>(); + resourceGroups.list().stream().forEach(e -> names.add(e.name())); + return names; + } + + @Override + public List listPipeline(User loginUser, String factoryName, String resourceGroupName) { + PagedIterable pipelineResources = + manager.pipelines().listByFactory(resourceGroupName, factoryName); + List names = new ArrayList<>(); + pipelineResources.stream().forEach(e -> names.add(e.name())); + return names; + } +} diff --git a/dolphinscheduler-api/src/main/resources/task-type-config.yaml b/dolphinscheduler-api/src/main/resources/task-type-config.yaml index c4f3c148dd..688f02930d 100644 --- a/dolphinscheduler-api/src/main/resources/task-type-config.yaml +++ b/dolphinscheduler-api/src/main/resources/task-type-config.yaml @@ -33,6 +33,7 @@ task: - 'EMR' - 'K8S' - 'DMS' + - 'DATA_FACTORY' logic: - 'SUB_PROCESS' - 'DEPENDENT' diff --git a/dolphinscheduler-bom/pom.xml b/dolphinscheduler-bom/pom.xml index a691d3f2cc..5ff6d57743 100644 --- a/dolphinscheduler-bom/pom.xml +++ b/dolphinscheduler-bom/pom.xml @@ -103,6 +103,8 @@ 1.7.1 2.1.0.9 1.12.300 + 2.21.0 + 1.0.0-beta.19 @@ -798,6 +800,16 @@ azure-identity ${azure-identity.version} + + com.azure.resourcemanager + azure-resourcemanager + ${azure-resourcemanager.version} + + + com.azure.resourcemanager + azure-resourcemanager-datafactory + ${azure-resourcemanager-datafactory.version} + com.amazon.redshift redshift-jdbc42 diff --git a/dolphinscheduler-common/src/main/resources/common.properties b/dolphinscheduler-common/src/main/resources/common.properties index 0ea630f412..b0a45b9cdd 100644 --- a/dolphinscheduler-common/src/main/resources/common.properties +++ b/dolphinscheduler-common/src/main/resources/common.properties @@ -26,6 +26,17 @@ resource.storage.type=NONE # resource store on HDFS/S3 path, resource file will store to this base path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended resource.storage.upload.base.path=/dolphinscheduler +# The Azure client ID (Azure Application (client) ID) +resource.azure.client.id=minioadmin +# The Azure client secret in the Azure application +resource.azure.client.secret=minioadmin +# The Azure data factory subscription ID +resource.azure.subId=minioadmin +# The Azure tenant id in the Azure Active Directory +resource.azure.tenant.id=minioadmin +# The query interval +resource.query.interval=10000 + # The AWS access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required resource.aws.access.key.id=minioadmin # The AWS secret access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required diff --git a/dolphinscheduler-dist/release-docs/LICENSE b/dolphinscheduler-dist/release-docs/LICENSE index 2eb79349d3..8dc6103409 100644 --- a/dolphinscheduler-dist/release-docs/LICENSE +++ b/dolphinscheduler-dist/release-docs/LICENSE @@ -517,6 +517,8 @@ The text of each license is also included at licenses/LICENSE-[project].txt. reactor-netty-core 1.0.22: https://mvnrepository.com/artifact/io.projectreactor.netty/reactor-netty-core/1.0.22, Apache 2.0 reactor-netty-http 1.0.22: https://mvnrepository.com/artifact/io.projectreactor.netty/reactor-netty-http/1.0.22, Apache 2.0 DmJdbcDriver18 8.1.2.79: https://mvnrepository.com/artifact/com.dameng/DmJdbcDriver18/8.1.2.79, Apache 2.0 + nimbus-jose-jwt 9.22: https://mvnrepository.com/artifact/com.nimbusds/nimbus-jose-jwt/9.22, Apache 2.0 + woodstox-core 6.4.0: https://mvnrepository.com/artifact/com.fasterxml.woodstox/woodstox-core/6.4.0, Apache 2.0 jna-platform @@ -605,12 +607,42 @@ The text of each license is also included at licenses/LICENSE-[project].txt. bcpkix-jdk15on 1.68: https://mvnrepository.com/artifact/org.bouncycastle/bcpkix-jdk15on, MIT bcprov-jdk15on 1.68: https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15on, MIT reactive-streams 1.0.4: https://mvnrepository.com/artifact/org.reactivestreams/reactive-streams/1.0.4, MIT - webjars-locator-core 0.50 https://mvnrepository.com/artifact/org.webjars/webjars-locator-core/0.50, MIT - azure-core 1.34.0 https://mvnrepository.com/artifact/com.azure/azure-core/1.34.0, MIT - azure-core-http-netty 1.12.7 https://mvnrepository.com/artifact/com.azure/azure-core-http-netty/1.12.7, MIT - azure-identity 1.7.1 https://mvnrepository.com/artifact/com.azure/azure-identity/1.7.1, MIT - msal4j 1.13.3 https://mvnrepository.com/artifact/com.microsoft.azure/msal4j/1.13.3, MIT - msal4j-persistence-extension 1.1.0 https://mvnrepository.com/artifact/com.microsoft.azure/msal4j-persistence-extension/1.1.0, MIT + webjars-locator-core 0.50: https://mvnrepository.com/artifact/org.webjars/webjars-locator-core/0.50, MIT + azure-core 1.34.0: https://mvnrepository.com/artifact/com.azure/azure-core/1.34.0, MIT + azure-core-http-netty 1.12.7: https://mvnrepository.com/artifact/com.azure/azure-core-http-netty/1.12.7, MIT + azure-identity 1.7.1: https://mvnrepository.com/artifact/com.azure/azure-identity/1.7.1, MIT + msal4j 1.13.3: https://mvnrepository.com/artifact/com.microsoft.azure/msal4j/1.13.3, MIT + msal4j-persistence-extension 1.1.0: https://mvnrepository.com/artifact/com.microsoft.azure/msal4j-persistence-extension/1.1.0, MIT + azure-resourcemanager 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager/2.21.0, MIT + azure-resourcemanager-appplatform 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-appplatform/2.21.0, MIT + azure-resourcemanager-appservice 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-appservice/2.21.0, MIT + azure-resourcemanager-authorization 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-authorization/2.21.0, MIT + azure-resourcemanager-cdn 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-cdn/2.21.0, MIT + azure-resourcemanager-compute 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-compute/2.21.0, MIT + azure-resourcemanager-containerinstance 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-containerinstance/2.21.0, MIT + azure-resourcemanager-containerregistry 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-containerregistry/2.21.0, MIT + azure-resourcemanager-containerservice 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-containerservice/2.21.0, MIT + azure-resourcemanager-cosmos 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-cosmos/2.21.0, MIT + azure-resourcemanager-datafactory 1.0.0-beta.19 https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-datafactory/1.0.0-beta.19, MIT + azure-resourcemanager-dns 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-dns/2.21.0, MIT + azure-resourcemanager-eventhubs 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-eventhubs/2.21.0, MIT + azure-resourcemanager-keyvault 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-keyvault/2.21.0, MIT + azure-resourcemanager-monitor 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-monitor/2.21.0, MIT + azure-resourcemanager-msi 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-msi/2.21.0, MIT + azure-resourcemanager-network 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-network/2.21.0, MIT + azure-resourcemanager-privatedns 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-privatedns/2.21.0, MIT + azure-resourcemanager-redis 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-redis/2.21.0, MIT + azure-resourcemanager-resources 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-resources/2.21.0, MIT + azure-resourcemanager-search 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-search/2.21.0, MIT + azure-resourcemanager-servicebus 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-servicebus/2.21.0, MIT + azure-resourcemanager-sql 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-sql/2.21.0, MIT + azure-resourcemanager-storage 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-storage/2.21.0, MIT + azure-resourcemanager-trafficmanager 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-trafficmanager/2.21.0, MIT + azure-security-keyvault-keys 4.5.2: https://mvnrepository.com/artifact/com.azure/azure-security-keyvault-keys/4.5.2, MIT + azure-security-keyvault-secrets 4.5.2: https://mvnrepository.com/artifact/com.azure/azure-security-keyvault-secrets/4.5.2, MIT + azure-storage-common 12.19.1: https://mvnrepository.com/artifact/com.azure/azure-storage-common/12.19.1, MIT + azure-storage-file-share 12.16.1: https://mvnrepository.com/artifact/com.azure/azure-storage-file-share/12.16.1, MIT + azure-core-management 1.9.0: https://mvnrepository.com/artifact/com.azure/azure-core-management/1.9.0, MIT ======================================================================== MPL 1.1 licenses diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-core-management.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-core-management.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-core-management.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-appplatform.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-appplatform.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-appplatform.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-appservice.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-appservice.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-appservice.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-authorization.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-authorization.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-authorization.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-cdn.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-cdn.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-cdn.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-compute.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-compute.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-compute.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-containerinstance.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-containerinstance.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-containerinstance.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-containerregistry.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-containerregistry.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-containerregistry.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-containerservice.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-containerservice.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-containerservice.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-cosmos.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-cosmos.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-cosmos.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-datafactory.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-datafactory.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-datafactory.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-dns.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-dns.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-dns.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-eventhubs.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-eventhubs.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-eventhubs.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-keyvault.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-keyvault.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-keyvault.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-monitor.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-monitor.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-monitor.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-msi.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-msi.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-msi.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-network.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-network.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-network.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-privatedns.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-privatedns.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-privatedns.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-redis.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-redis.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-redis.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-resources.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-resources.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-resources.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-search.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-search.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-search.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-servicebus.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-servicebus.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-servicebus.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-sql.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-sql.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-sql.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-storage.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-storage.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-storage.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-trafficmanager.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-trafficmanager.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-trafficmanager.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-security-keyvault-keys.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-security-keyvault-keys.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-security-keyvault-keys.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-security-keyvault-secrets.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-security-keyvault-secrets.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-security-keyvault-secrets.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-storage-common.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-storage-common.txt new file mode 100644 index 0000000000..b5747801a6 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-storage-common.txt @@ -0,0 +1,14 @@ +LICENSE-msal4j.txtCopyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-storage-file-share.txt b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-storage-file-share.txt new file mode 100644 index 0000000000..d332c60cd0 --- /dev/null +++ b/dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-storage-file-share.txt @@ -0,0 +1,14 @@ +Copyright(c) 2016 Microsoft Corporation +All rights reserved. + +MIT License +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, +and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions : + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. \ No newline at end of file diff --git a/dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/resources/docker/file-manage/common.properties b/dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/resources/docker/file-manage/common.properties index f4bba5438d..644b8825a9 100644 --- a/dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/resources/docker/file-manage/common.properties +++ b/dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/resources/docker/file-manage/common.properties @@ -26,6 +26,17 @@ resource.storage.type=S3 # resource store on HDFS/S3 path, resource file will store to this base path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended resource.storage.upload.base.path=/dolphinscheduler +# The Azure client ID (Azure Application (client) ID) +resource.azure.client.id=minioadmin +# The Azure client secret in the Azure application +resource.azure.client.secret=minioadmin +# The Azure data factory subscription ID +resource.azure.subId=minioadmin +# The Azure tenant id in the Azure Active Directory +resource.azure.tenant.id=minioadmin +# The query interval +resource.query.interval=10000 + # The AWS access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required resource.aws.access.key.id=accessKey123 # The AWS secret access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-all/pom.xml b/dolphinscheduler-task-plugin/dolphinscheduler-task-all/pom.xml index cd8a33b8a0..9d11d6fe58 100644 --- a/dolphinscheduler-task-plugin/dolphinscheduler-task-all/pom.xml +++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-all/pom.xml @@ -212,6 +212,11 @@ dolphinscheduler-task-linkis ${project.version} + + org.apache.dolphinscheduler + dolphinscheduler-task-datafactory + ${project.version} + diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/TaskConstants.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/TaskConstants.java index d2514e349e..8337dc6f31 100644 --- a/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/TaskConstants.java +++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/TaskConstants.java @@ -443,6 +443,15 @@ public class TaskConstants { public static final String TASK_TYPE_STREAM = "STREAM"; + /** + * azure config + */ + public static final String AZURE_CLIENT_ID = "resource.azure.client.id"; + public static final String AZURE_CLIENT_SECRET = "resource.azure.client.secret"; + public static final String AZURE_ACCESS_SUB_ID = "resource.azure.subId"; + public static final String AZURE_SECRET_TENANT_ID = "resource.azure.tenant.id"; + public static final String QUERY_INTERVAL = "resource.query.interval"; + /** * aws config */ diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/pom.xml b/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/pom.xml new file mode 100644 index 0000000000..bbc1e59a1f --- /dev/null +++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/pom.xml @@ -0,0 +1,57 @@ + + + + 4.0.0 + + org.apache.dolphinscheduler + dolphinscheduler-task-plugin + dev-SNAPSHOT + + + dolphinscheduler-task-datafactory + jar + + + + org.apache.dolphinscheduler + dolphinscheduler-spi + provided + + + org.apache.dolphinscheduler + dolphinscheduler-task-api + provided + + + org.apache.dolphinscheduler + dolphinscheduler-common + provided + + + com.azure.resourcemanager + azure-resourcemanager-datafactory + 1.0.0-beta.19 + + + com.azure + azure-identity + + + + diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryHook.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryHook.java new file mode 100644 index 0000000000..763024169d --- /dev/null +++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryHook.java @@ -0,0 +1,148 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.dolphinscheduler.plugin.task.datafactory; + +import org.apache.dolphinscheduler.common.utils.PropertyUtils; +import org.apache.dolphinscheduler.plugin.task.api.TaskConstants; + +import org.apache.commons.lang3.StringUtils; + +import java.util.Arrays; +import java.util.List; + +import lombok.Data; +import lombok.SneakyThrows; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.azure.core.management.AzureEnvironment; +import com.azure.core.management.profile.AzureProfile; +import com.azure.identity.ClientSecretCredential; +import com.azure.identity.ClientSecretCredentialBuilder; +import com.azure.resourcemanager.datafactory.DataFactoryManager; +import com.azure.resourcemanager.datafactory.models.CreateRunResponse; +import com.azure.resourcemanager.datafactory.models.PipelineResource; +import com.azure.resourcemanager.datafactory.models.PipelineRun; +import com.azure.resourcemanager.datafactory.models.PipelineRuns; + +@Data +public class DatafactoryHook { + + public static DatafactoryStatus[] taskFinishFlags = + {DatafactoryStatus.Failed, DatafactoryStatus.Succeeded, DatafactoryStatus.Cancelled}; + protected final Logger logger = + LoggerFactory.getLogger(String.format(TaskConstants.TASK_LOG_LOGGER_NAME_FORMAT, getClass())); + private final int QUERY_INTERVAL = PropertyUtils.getInt(TaskConstants.QUERY_INTERVAL, 10000); + private DataFactoryManager client; + private static AzureProfile profile; + private static ClientSecretCredential credential; + private String runId; + + public DatafactoryHook() { + logger.info("initDatafactoryClient ......"); + client = createClient(); + } + + protected static DataFactoryManager createClient() { + final String AZURE_ACCESS_SUB_ID = PropertyUtils.getString(TaskConstants.AZURE_ACCESS_SUB_ID); + final String AZURE_SECRET_TENANT_ID = PropertyUtils.getString(TaskConstants.AZURE_SECRET_TENANT_ID); + final String AZURE_CLIENT_ID = PropertyUtils.getString(TaskConstants.AZURE_CLIENT_ID); + final String AZURE_CLIENT_SECRET = PropertyUtils.getString(TaskConstants.AZURE_CLIENT_SECRET); + profile = + new AzureProfile(AZURE_SECRET_TENANT_ID, AZURE_ACCESS_SUB_ID, AzureEnvironment.AZURE); + credential = new ClientSecretCredentialBuilder() + .clientId(AZURE_CLIENT_ID) + .clientSecret(AZURE_CLIENT_SECRET) + .tenantId(AZURE_SECRET_TENANT_ID) + .authorityHost(profile.getEnvironment().getActiveDirectoryEndpoint()) + .build(); + return DataFactoryManager.authenticate(credential, profile); + } + + public Boolean startDatafactoryTask(DatafactoryParameters parameters) { + logger.info("initDatafactoryTask ......"); + PipelineResource pipelineResource = getPipelineResource(parameters); + if (pipelineResource == null) { + return false; + } + logger.info("startDatafactoryTask ......"); + CreateRunResponse run = pipelineResource.createRun(); + if (StringUtils.isEmpty(run.runId())) { + return false; + } + runId = run.runId(); + parameters.setRunId(runId); + return true; + } + + public Boolean cancelDatafactoryTask(DatafactoryParameters parameters) { + logger.info("cancelTask ......"); + PipelineRuns pipelineRuns = client.pipelineRuns(); + try { + pipelineRuns.cancel(parameters.getResourceGroupName(), parameters.getFactoryName(), runId); + } catch (RuntimeException e) { + logger.error("failed to cancel datafactory task: " + e.getMessage()); + return false; + } + return true; + } + + public DatafactoryStatus queryDatafactoryTaskStatus(DatafactoryParameters parameters) { + logger.info("queryDatafactoryTaskStatus ......"); + + PipelineRuns pipelineRuns = client.pipelineRuns(); + PipelineRun pipelineRun = + pipelineRuns.get(parameters.getResourceGroupName(), parameters.getFactoryName(), runId); + + if (pipelineRun != null) { + logger.info("queryDatafactoryTaskStatus ......{}", pipelineRun.status()); + return DatafactoryStatus.valueOf(pipelineRun.status()); + } + return null; + } + + private PipelineResource getPipelineResource(DatafactoryParameters parameters) { + return client.pipelines().get(parameters.getResourceGroupName(), parameters.getFactoryName(), + parameters.getPipelineName()); + } + + @SneakyThrows + public Boolean queryStatus(DatafactoryParameters parameters) { + List stopStatusSet = Arrays.asList(taskFinishFlags); + int maxRetry = 5; + while (maxRetry > 0) { + DatafactoryStatus status = queryDatafactoryTaskStatus(parameters); + + if (status == null) { + maxRetry--; + continue; + } + + if (stopStatusSet.contains(status)) { + if (status.equals(DatafactoryStatus.Succeeded)) { + return true; + } + return false; + } + logger.debug("wait {}ms to recheck finish status....", QUERY_INTERVAL); + Thread.sleep(QUERY_INTERVAL); + } + return false; + } +} diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryParameters.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryParameters.java new file mode 100644 index 0000000000..b0bbb75e8d --- /dev/null +++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryParameters.java @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.dolphinscheduler.plugin.task.datafactory; + +import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters; + +import org.apache.commons.lang3.StringUtils; + +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.Setter; +import lombok.ToString; + +@Setter +@Getter +@NoArgsConstructor +@ToString +public class DatafactoryParameters extends AbstractParameters { + + private String factoryName; + private String resourceGroupName; + private String pipelineName; + private String runId; + + @Override + public boolean checkParameters() { + return StringUtils.isNotEmpty(factoryName) && StringUtils.isNotEmpty(resourceGroupName) + && StringUtils.isNotEmpty(pipelineName); + } +} diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryStatus.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryStatus.java new file mode 100644 index 0000000000..7f2af88877 --- /dev/null +++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryStatus.java @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.dolphinscheduler.plugin.task.datafactory; + +public enum DatafactoryStatus { + + Queued, + InProgress, + Succeeded, + Failed, + Canceling, + Cancelled, + ; + + /** + * Gets the status property: The status of a pipeline run. Possible values: Queued, InProgress, Succeeded, Failed, + * Canceling, Cancelled. + * + * @return the status value. + */ + DatafactoryStatus() { + } +} diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTask.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTask.java new file mode 100644 index 0000000000..b8a5db4371 --- /dev/null +++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTask.java @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.dolphinscheduler.plugin.task.datafactory; + +import org.apache.dolphinscheduler.common.utils.JSONUtils; +import org.apache.dolphinscheduler.plugin.task.api.AbstractRemoteTask; +import org.apache.dolphinscheduler.plugin.task.api.TaskConstants; +import org.apache.dolphinscheduler.plugin.task.api.TaskException; +import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext; + +import org.apache.commons.lang3.StringUtils; + +import java.util.Collections; +import java.util.List; + +import lombok.Getter; +import lombok.Setter; + +@Setter +@Getter +public class DatafactoryTask extends AbstractRemoteTask { + + private final TaskExecutionContext taskExecutionContext; + private DatafactoryParameters parameters; + private DatafactoryHook hook; + + public DatafactoryTask(TaskExecutionContext taskExecutionContext) { + super(taskExecutionContext); + this.taskExecutionContext = taskExecutionContext; + } + + @Override + public List getApplicationIds() throws TaskException { + return Collections.emptyList(); + } + + @Override + public void init() { + parameters = JSONUtils.parseObject(taskExecutionContext.getTaskParams(), DatafactoryParameters.class); + logger.info("Initialize Datafactory task params {}", JSONUtils.toPrettyJsonString(parameters)); + hook = new DatafactoryHook(); + } + + @Override + public void submitApplication() throws TaskException { + try { + // start task + exitStatusCode = startDatafactoryTask(); + setExitStatusCode(exitStatusCode); + } catch (Exception e) { + setExitStatusCode(TaskConstants.EXIT_CODE_FAILURE); + throw new TaskException("data factory start task error", e); + } + // set runId to the appIds if start success + setAppIds(parameters.getRunId()); + } + + @Override + public void cancelApplication() throws TaskException { + checkApplicationId(); + hook.cancelDatafactoryTask(parameters); + exitStatusCode = TaskConstants.EXIT_CODE_KILL; + } + + @Override + public void trackApplicationStatus() throws TaskException { + checkApplicationId(); + Boolean isFinishedSuccessfully; + isFinishedSuccessfully = hook.queryStatus(parameters); + if (!isFinishedSuccessfully) { + exitStatusCode = TaskConstants.EXIT_CODE_FAILURE; + } else { + exitStatusCode = TaskConstants.EXIT_CODE_SUCCESS; + } + } + + /** + * check datafactory applicationId or get it from appId + */ + private void checkApplicationId() { + String taskExecArn = hook.getRunId(); + if (StringUtils.isEmpty(taskExecArn)) { + if (StringUtils.isEmpty(getAppIds())) { + throw new TaskException("datafactory runId is null, not created yet"); + } + hook.setRunId(getAppIds()); + } + } + + public int startDatafactoryTask() { + Boolean isStartSuccessfully = hook.startDatafactoryTask(parameters); + if (!isStartSuccessfully) { + return TaskConstants.EXIT_CODE_FAILURE; + } + return TaskConstants.EXIT_CODE_SUCCESS; + } + + @Override + public DatafactoryParameters getParameters() { + return parameters; + } + +} diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTaskChannel.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTaskChannel.java new file mode 100644 index 0000000000..d8c099bb09 --- /dev/null +++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTaskChannel.java @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.dolphinscheduler.plugin.task.datafactory; + +import org.apache.dolphinscheduler.common.utils.JSONUtils; +import org.apache.dolphinscheduler.plugin.task.api.TaskChannel; +import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext; +import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters; +import org.apache.dolphinscheduler.plugin.task.api.parameters.ParametersNode; +import org.apache.dolphinscheduler.plugin.task.api.parameters.resource.ResourceParametersHelper; + +public class DatafactoryTaskChannel implements TaskChannel { + + @Override + public void cancelApplication(boolean status) { + } + + @Override + public DatafactoryTask createTask(TaskExecutionContext taskRequest) { + return new DatafactoryTask(taskRequest); + } + + @Override + public AbstractParameters parseParameters(ParametersNode parametersNode) { + return JSONUtils.parseObject(parametersNode.getTaskParams(), DatafactoryParameters.class); + } + + @Override + public ResourceParametersHelper getResources(String parameters) { + return null; + } +} diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTaskChannelFactory.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTaskChannelFactory.java new file mode 100644 index 0000000000..e9dce1ac96 --- /dev/null +++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTaskChannelFactory.java @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.dolphinscheduler.plugin.task.datafactory; + +import org.apache.dolphinscheduler.plugin.task.api.TaskChannel; +import org.apache.dolphinscheduler.plugin.task.api.TaskChannelFactory; +import org.apache.dolphinscheduler.spi.params.base.PluginParams; + +import java.util.Collections; +import java.util.List; + +import com.google.auto.service.AutoService; + +@AutoService(TaskChannelFactory.class) +public class DatafactoryTaskChannelFactory implements TaskChannelFactory { + + @Override + public TaskChannel create() { + return new DatafactoryTaskChannel(); + } + + @Override + public String getName() { + return "DATA_FACTORY"; + } + + @Override + public List getParams() { + return Collections.emptyList(); + } +} diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/test/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTaskTest.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/test/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTaskTest.java new file mode 100644 index 0000000000..cf2c5a55ea --- /dev/null +++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/test/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTaskTest.java @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.dolphinscheduler.plugin.task.datafactory; + +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.mockStatic; +import static org.mockito.Mockito.when; + +import org.apache.dolphinscheduler.common.utils.JSONUtils; +import org.apache.dolphinscheduler.plugin.task.api.TaskConstants; +import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.MockedStatic; +import org.mockito.Mockito; +import org.mockito.junit.jupiter.MockitoExtension; + +import com.azure.resourcemanager.datafactory.DataFactoryManager; +import com.azure.resourcemanager.datafactory.models.CreateRunResponse; +import com.azure.resourcemanager.datafactory.models.PipelineResource; +import com.azure.resourcemanager.datafactory.models.Pipelines; + +@ExtendWith(MockitoExtension.class) +public class DatafactoryTaskTest { + + private static final String mockRunId = + "3c7182f4-d107-43c7-af2a-8c7b3ed1d614"; + + DatafactoryTask datafactoryTask; + + @Mock + DataFactoryManager client; + MockedStatic datafactoryHookMockedStatic; + + @BeforeEach + public void before() throws IllegalAccessException { + client = mock(DataFactoryManager.class); + datafactoryHookMockedStatic = mockStatic(DatafactoryHook.class); + when(DatafactoryHook.createClient()).thenReturn(client); + + DatafactoryParameters datafactoryParameters = new DatafactoryParameters(); + datafactoryTask = initTask(datafactoryParameters); + } + + @Test + public void testConvertParam() { + DatafactoryParameters parameters = castParam(); + Assertions.assertEquals("whale1", parameters.getFactoryName()); + Assertions.assertEquals("factory1", parameters.getResourceGroupName()); + Assertions.assertEquals("CopyPipeline_ps8", parameters.getPipelineName()); + datafactoryHookMockedStatic.close(); + } + + @Test + public void testCheckCreateTask() { + DatafactoryHook hook = new DatafactoryHook(); + DataFactoryManager client1 = DatafactoryHook.createClient(); + hook.setClient(client1); + + PipelineResource pipelineResource = mock(PipelineResource.class); + Pipelines pipelines = mock(Pipelines.class); + when(client1.pipelines()).thenReturn(pipelines); + when(pipelines.get(any(), any(), any())).thenReturn(pipelineResource); + + CreateRunResponse response = mock(CreateRunResponse.class); + when(pipelineResource.createRun()).thenReturn(response); + when(response.runId()).thenReturn(mockRunId); + hook.startDatafactoryTask(castParam()); + Assertions.assertEquals(mockRunId, hook.getRunId()); + datafactoryHookMockedStatic.close(); + } + + @Test + public void testCancelTask() { + datafactoryTask.cancelApplication(); + Assertions.assertEquals(datafactoryTask.getExitStatusCode(), TaskConstants.EXIT_CODE_KILL); + datafactoryHookMockedStatic.close(); + } + + private DatafactoryParameters castParam() { + String input = + "{\"localParams\":[],\"resourceList\":[],\"factoryName\":\"whale1\",\"resourceGroupName\":\"factory1\",\"pipelineName\":\"CopyPipeline_ps8\"}"; + return JSONUtils.parseObject(input, DatafactoryParameters.class); + } + + private DatafactoryTask initTask(DatafactoryParameters DatasyncParameters) { + TaskExecutionContext taskExecutionContext = createContext(DatasyncParameters); + DatafactoryTask task = new DatafactoryTask(taskExecutionContext); + task.init(); + task.getHook().setRunId(mockRunId); + return task; + } + + public TaskExecutionContext createContext(DatafactoryParameters datafactoryParameters) { + String parameters = JSONUtils.toJsonString(datafactoryParameters); + TaskExecutionContext taskExecutionContext = Mockito.mock(TaskExecutionContext.class); + Mockito.when(taskExecutionContext.getTaskParams()).thenReturn(parameters); + return taskExecutionContext; + } +} diff --git a/dolphinscheduler-task-plugin/pom.xml b/dolphinscheduler-task-plugin/pom.xml index 6cd6c8a76a..27b741f06b 100644 --- a/dolphinscheduler-task-plugin/pom.xml +++ b/dolphinscheduler-task-plugin/pom.xml @@ -61,6 +61,7 @@ dolphinscheduler-task-datasync dolphinscheduler-task-kubeflow dolphinscheduler-task-linkis + dolphinscheduler-task-datafactory diff --git a/dolphinscheduler-ui/public/images/task-icons/data_factory.png b/dolphinscheduler-ui/public/images/task-icons/data_factory.png new file mode 100644 index 0000000000..fb3bbd33d5 Binary files /dev/null and b/dolphinscheduler-ui/public/images/task-icons/data_factory.png differ diff --git a/dolphinscheduler-ui/public/images/task-icons/data_factory_hover.png b/dolphinscheduler-ui/public/images/task-icons/data_factory_hover.png new file mode 100644 index 0000000000..9cebaceee4 Binary files /dev/null and b/dolphinscheduler-ui/public/images/task-icons/data_factory_hover.png differ diff --git a/dolphinscheduler-ui/src/locales/en_US/project.ts b/dolphinscheduler-ui/src/locales/en_US/project.ts index 5a42e12f53..3ee2cef106 100644 --- a/dolphinscheduler-ui/src/locales/en_US/project.ts +++ b/dolphinscheduler-ui/src/locales/en_US/project.ts @@ -37,10 +37,10 @@ export default { confirm: 'Confirm', cancel: 'Cancel', delete_confirm: 'Delete?', - authorize_level:'Authorize Level', + authorize_level: 'Authorize Level', no_permission: 'No Permission', read_permission: 'Read Permission', - all_permission: 'All Permission', + all_permission: 'All Permission' }, workflow: { on_line: 'Online', @@ -215,7 +215,7 @@ export default { workflow_relation_no_data_result_desc: 'There is not any workflows. Please create a workflow, and then visit this page again.', ready_to_block: 'Ready to block', - block: 'Block', + block: 'Block' }, task: { on_line: 'Online', @@ -329,7 +329,7 @@ export default { online: 'Online' }, node: { - is_cache: "Cache Execution", + is_cache: 'Cache Execution', jvm_args: 'Java VM Parameters', jvm_args_tips: 'Please enter virtual machine parameters', run_type: 'Run Type', @@ -377,7 +377,8 @@ export default { image: 'Image', image_tips: 'Please enter image', command: 'Command', - command_tips: 'Please enter the container execution command, for example: /bin/echo hello world', + command_tips: + 'Please enter the container execution command, for example: /bin/echo hello world', min_memory_tips: 'Please enter min memory', state: 'State', branch_flow: 'Branch flow', @@ -798,7 +799,13 @@ export default { pytorch_requirements: 'Requirement File', pytorch_conda_python_version: 'Python Version', pytorch_conda_python_version_tips: - 'Please enter the version number, such as 3.6, 3.7, 3.x' + 'Please enter the version number, such as 3.6, 3.7, 3.x', + factory_name: 'Factory Name', + resource_group_name: 'Resource Group Name', + pipeline_name: 'Pipeline Name', + factory_tips: 'Please select factory', + resource_group_tips: 'Please select resource group', + pipeline_tips: 'Please select pipeline' }, menu: { fav: 'Favorites', @@ -808,6 +815,6 @@ export default { di: 'Data Integration', dq: 'Data Quality', ml: 'Machine Learning', - other: 'Other', + other: 'Other' } } diff --git a/dolphinscheduler-ui/src/locales/zh_CN/project.ts b/dolphinscheduler-ui/src/locales/zh_CN/project.ts index 54c77d4633..1d5224022b 100644 --- a/dolphinscheduler-ui/src/locales/zh_CN/project.ts +++ b/dolphinscheduler-ui/src/locales/zh_CN/project.ts @@ -777,7 +777,13 @@ export default { pytorch_python_env_tool: 'python环境管理工具', pytorch_requirements: '依赖文件', pytorch_conda_python_version: 'python版本', - pytorch_conda_python_version_tips: '请输入版本号，如 3.6, 3.7, 3.x等' + pytorch_conda_python_version_tips: '请输入版本号，如 3.6, 3.7, 3.x等', + factory_name: '工厂名称', + resource_group_name: '资源组名称', + pipeline_name: 'pipeline名称', + factory_tips: '请选择工厂', + resource_group_tips: '请选择资源组', + pipeline_tips: '请选择pipeline' }, menu: { fav: '收藏组件', diff --git a/dolphinscheduler-ui/src/service/modules/azure/index.ts b/dolphinscheduler-ui/src/service/modules/azure/index.ts new file mode 100644 index 0000000000..28f83db45c --- /dev/null +++ b/dolphinscheduler-ui/src/service/modules/azure/index.ts @@ -0,0 +1,41 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { axios } from '@/service/service' +import { DataFactoryReq } from './types' + +export function queryDataFactoryFactories(): any { + return axios({ + url: '/cloud/azure/datafactory/factories', + method: 'get' + }) +} + +export function queryDataFactoryResourceGroups(): any { + return axios({ + url: '/cloud/azure/datafactory/resourceGroups', + method: 'get' + }) +} + +export function queryDataFactoryPipelines(params: DataFactoryReq): any { + return axios({ + url: '/cloud/azure/datafactory/pipelines', + method: 'get', + params + }) +} diff --git a/dolphinscheduler-ui/src/service/modules/azure/types.ts b/dolphinscheduler-ui/src/service/modules/azure/types.ts new file mode 100644 index 0000000000..5f59d70bde --- /dev/null +++ b/dolphinscheduler-ui/src/service/modules/azure/types.ts @@ -0,0 +1,23 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +interface DataFactoryReq { + factoryName: string + resourceGroupName: string +} + +export { DataFactoryReq } diff --git a/dolphinscheduler-ui/src/store/project/task-type.ts b/dolphinscheduler-ui/src/store/project/task-type.ts index b1f426bef8..24bc05614d 100644 --- a/dolphinscheduler-ui/src/store/project/task-type.ts +++ b/dolphinscheduler-ui/src/store/project/task-type.ts @@ -149,6 +149,10 @@ export const TASK_TYPES_MAP = { LINKIS: { alias: 'LINKIS', helperLinkDisable: true + }, + DATA_FACTORY: { + alias: 'DATA_FACTORY', + helperLinkDisable: true } } as { [key in TaskType]: { diff --git a/dolphinscheduler-ui/src/store/project/types.ts b/dolphinscheduler-ui/src/store/project/types.ts index e801f4a1eb..0df130e77b 100644 --- a/dolphinscheduler-ui/src/store/project/types.ts +++ b/dolphinscheduler-ui/src/store/project/types.ts @@ -56,6 +56,7 @@ type TaskType = | 'DATASYNC' | 'KUBEFLOW' | 'LINKIS' + | 'DATA_FACTORY' type ProgramType = 'JAVA' | 'SCALA' | 'PYTHON' diff --git a/dolphinscheduler-ui/src/views/projects/task/components/node/fields/index.ts b/dolphinscheduler-ui/src/views/projects/task/components/node/fields/index.ts index e7b4ab06f8..4510f47183 100644 --- a/dolphinscheduler-ui/src/views/projects/task/components/node/fields/index.ts +++ b/dolphinscheduler-ui/src/views/projects/task/components/node/fields/index.ts @@ -85,3 +85,4 @@ export { useDms } from './use-dms' export { useDatasync } from './use-datasync' export { useKubeflow } from './use-kubeflow' export { useLinkis } from './use-linkis' +export { useDataFactory } from './use-data-factory' diff --git a/dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-data-factory.ts b/dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-data-factory.ts new file mode 100644 index 0000000000..8dc6f5213c --- /dev/null +++ b/dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-data-factory.ts @@ -0,0 +1,152 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import { + queryDataFactoryFactories, + queryDataFactoryPipelines, + queryDataFactoryResourceGroups +} from '@/service/modules/azure' +import { onMounted, ref, watch } from 'vue' +import { useI18n } from 'vue-i18n' +import { useCustomParams } from '.' +import type { IJsonItem } from '../types' + +export function useDataFactory(model: { [field: string]: any }): IJsonItem[] { + const { t } = useI18n() + + const factoryLoading = ref(false) + const resourceGroupLoading = ref(false) + const pipelineLoading = ref(false) + + const factoryOptions = ref([] as { label: string; value: number }[]) + const resourceGroupOptions = ref([] as { label: string; value: number }[]) + const pipelineOptions = ref([] as { label: string; value: number }[]) + + const getFactoryOptions = async () => { + if (factoryLoading.value) return + factoryLoading.value = true + const factories = await queryDataFactoryFactories() + factoryOptions.value = factories.map((factory: string) => ({ + label: factory, + value: factory + })) + factoryLoading.value = false + } + + const getResourceGroupName = async () => { + if (resourceGroupLoading.value) return + resourceGroupLoading.value = true + const groupNames = await queryDataFactoryResourceGroups() + resourceGroupOptions.value = groupNames.map((groupName: string) => ({ + label: groupName, + value: groupName + })) + resourceGroupLoading.value = false + } + + const getPipelineName = async ( + factoryName: string, + resourceGroupName: string + ) => { + if (pipelineLoading.value) return + pipelineLoading.value = true + const pipelineNames = await queryDataFactoryPipelines({ + factoryName, + resourceGroupName + }) + pipelineOptions.value = pipelineNames.map((pipelineName: string) => ({ + label: pipelineName, + value: pipelineName + })) + pipelineLoading.value = false + } + + const onChange = () => { + model['pipelineName'] = '' + if (model['factoryName'] && model['resourceGroupName']) { + getPipelineName(model['factoryName'], model['resourceGroupName']) + } + } + + watch( + () => model['pipelineName'], + () => { + if (model['pipelineName'] && pipelineOptions.value.length === 0) { + getPipelineName(model['factoryName'], model['resourceGroupName']) + } + } + ) + + onMounted(() => { + getFactoryOptions() + getResourceGroupName() + }) + + return [ + { + type: 'select', + field: 'factoryName', + span: 24, + name: t('project.node.factory_name'), + options: factoryOptions, + props: { + 'on-update:value': onChange, + loading: factoryLoading + }, + validate: { + trigger: ['input', 'trigger'], + required: true, + message: t('project.node.factory_tips') + } + }, + { + type: 'select', + field: 'resourceGroupName', + span: 24, + name: t('project.node.resource_group_name'), + options: resourceGroupOptions, + props: { + 'on-update:value': onChange, + loading: resourceGroupLoading + }, + validate: { + trigger: ['input', 'trigger'], + required: true, + message: t('project.node.resource_group_tips') + } + }, + { + type: 'select', + field: 'pipelineName', + span: 24, + name: t('project.node.pipeline_name'), + options: pipelineOptions, + props: { + loading: pipelineLoading + }, + validate: { + trigger: ['input', 'trigger'], + required: true, + message: t('project.node.pipeline_tips') + } + }, + ...useCustomParams({ + model, + field: 'localParams', + isSimple: true + }) + ] +} diff --git a/dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts b/dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts index ea804dc1d7..7d227db0e7 100644 --- a/dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts +++ b/dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts @@ -449,13 +449,19 @@ export function formatParams(data: INodeData): { taskParams.yamlContent = data.yamlContent taskParams.namespace = data.namespace } - + if (data.taskType === 'LINKIS') { taskParams.useCustom = data.useCustom taskParams.paramScript = data.paramScript taskParams.rawScript = data.rawScript } + if (data.taskType === 'DATA_FACTORY') { + taskParams.factoryName = data.factoryName + taskParams.resourceGroupName = data.resourceGroupName + taskParams.pipelineName = data.pipelineName + } + let timeoutNotifyStrategy = '' if (data.timeoutNotifyStrategy) { if (data.timeoutNotifyStrategy.length === 1) { diff --git a/dolphinscheduler-ui/src/views/projects/task/components/node/tasks/index.ts b/dolphinscheduler-ui/src/views/projects/task/components/node/tasks/index.ts index 1b6f998b4e..04a3518242 100644 --- a/dolphinscheduler-ui/src/views/projects/task/components/node/tasks/index.ts +++ b/dolphinscheduler-ui/src/views/projects/task/components/node/tasks/index.ts @@ -50,6 +50,7 @@ import { useDms } from './use-dms' import { useDatasync } from './use-datasync' import { useKubeflow } from './use-kubeflow' import { useLinkis } from './use-linkis' +import { useDataFactory } from './use-data-factory' export default { SHELL: useShell, @@ -86,5 +87,6 @@ export default { DMS: useDms, DATASYNC: useDatasync, KUBEFLOW: useKubeflow, - LINKIS: useLinkis + LINKIS: useLinkis, + DATA_FACTORY: useDataFactory } diff --git a/dolphinscheduler-ui/src/views/projects/task/components/node/tasks/use-data-factory.ts b/dolphinscheduler-ui/src/views/projects/task/components/node/tasks/use-data-factory.ts new file mode 100644 index 0000000000..c845fd0246 --- /dev/null +++ b/dolphinscheduler-ui/src/views/projects/task/components/node/tasks/use-data-factory.ts @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { reactive } from 'vue' +import * as Fields from '../fields/index' +import type { IJsonItem, INodeData } from '../types' +import { ITaskData } from '../types' + +export function useDataFactory({ + projectCode, + from = 0, + readonly, + data +}: { + projectCode: number + from?: number + readonly?: boolean + data?: ITaskData +}) { + const model = reactive({ + name: '', + taskType: 'DATA_FACTORY', + flag: 'YES', + description: '', + timeoutFlag: false, + localParams: [], + environmentCode: null, + failRetryInterval: 1, + failRetryTimes: 0, + workerGroup: 'default', + delayTime: 0, + timeout: 30, + timeoutNotifyStrategy: ['WARN'], + factoryName: '', + resourceGroupName: '', + pipelineName: '' + } as INodeData) + + return { + json: [ + Fields.useName(from), + ...Fields.useTaskDefinition({ projectCode, from, readonly, data, model }), + Fields.useRunFlag(), + Fields.useCache(), + Fields.useDescription(), + Fields.useTaskPriority(), + Fields.useWorkerGroup(), + Fields.useEnvironmentName(model, !data?.id), + ...Fields.useTaskGroup(model, projectCode), + ...Fields.useFailed(), + Fields.useDelayTime(model), + ...Fields.useTimeoutAlarm(model), + ...Fields.useDataFactory(model), + Fields.usePreTasks() + ] as IJsonItem[], + model + } +} diff --git a/dolphinscheduler-ui/src/views/projects/task/components/node/types.ts b/dolphinscheduler-ui/src/views/projects/task/components/node/types.ts index 282948345b..490503aafd 100644 --- a/dolphinscheduler-ui/src/views/projects/task/components/node/types.ts +++ b/dolphinscheduler-ui/src/views/projects/task/components/node/types.ts @@ -405,6 +405,9 @@ interface ITaskParams { cloudWatchLogGroupArn?: string yamlContent?: string paramScript?: ILocalParam[] + factoryName?: string + resourceGroupName?: string + pipelineName?: string } interface INodeData diff --git a/dolphinscheduler-ui/src/views/projects/task/constants/task-type.ts b/dolphinscheduler-ui/src/views/projects/task/constants/task-type.ts index 2606bea240..738892f68b 100644 --- a/dolphinscheduler-ui/src/views/projects/task/constants/task-type.ts +++ b/dolphinscheduler-ui/src/views/projects/task/constants/task-type.ts @@ -50,6 +50,7 @@ export type TaskType = | 'DATASYNC' | 'KUBEFLOW' | 'LINKIS' + | 'DATA_FACTORY' export type TaskExecuteType = 'STREAM' | 'BATCH' @@ -180,6 +181,10 @@ export const TASK_TYPES_MAP = { LINKIS: { alias: 'LINKIS', helperLinkDisable: true + }, + DATA_FACTORY: { + alias: 'DATA_FACTORY', + helperLinkDisable: true } } as { [key in TaskType]: { diff --git a/dolphinscheduler-ui/src/views/projects/workflow/components/dag/dag.module.scss b/dolphinscheduler-ui/src/views/projects/workflow/components/dag/dag.module.scss index 7aa0c9c709..65b7aa0067 100644 --- a/dolphinscheduler-ui/src/views/projects/workflow/components/dag/dag.module.scss +++ b/dolphinscheduler-ui/src/views/projects/workflow/components/dag/dag.module.scss @@ -204,6 +204,9 @@ $bgLight: #ffffff; &.icon-kubeflow { background-image: url('/images/task-icons/kubeflow.png'); } + &.icon-data_factory { + background-image: url('/images/task-icons/data_factory.png'); + } } &:hover { @@ -311,6 +314,9 @@ $bgLight: #ffffff; &.icon-kubeflow { background-image: url('/images/task-icons/kubeflow_hover.png'); } + &.icon-data_factory { + background-image: url('/images/task-icons/data_factory_hover.png'); + } } } diff --git a/tools/dependencies/known-dependencies.txt b/tools/dependencies/known-dependencies.txt index 5e7b64c525..eeb308e7bf 100644 --- a/tools/dependencies/known-dependencies.txt +++ b/tools/dependencies/known-dependencies.txt @@ -399,3 +399,35 @@ reactor-core-3.4.22.jar reactor-netty-core-1.0.22.jar reactor-netty-http-1.0.22.jar DmJdbcDriver18-8.1.2.79.jar +azure-resourcemanager-2.21.0.jar +azure-resourcemanager-appplatform-2.21.0.jar +azure-resourcemanager-appservice-2.21.0.jar +azure-resourcemanager-authorization-2.21.0.jar +azure-resourcemanager-cdn-2.21.0.jar +azure-resourcemanager-compute-2.21.0.jar +azure-resourcemanager-containerinstance-2.21.0.jar +azure-resourcemanager-containerregistry-2.21.0.jar +azure-resourcemanager-containerservice-2.21.0.jar +azure-resourcemanager-cosmos-2.21.0.jar +azure-resourcemanager-datafactory-1.0.0-beta.19.jar +azure-resourcemanager-dns-2.21.0.jar +azure-resourcemanager-eventhubs-2.21.0.jar +azure-resourcemanager-keyvault-2.21.0.jar +azure-resourcemanager-monitor-2.21.0.jar +azure-resourcemanager-msi-2.21.0.jar +azure-resourcemanager-network-2.21.0.jar +azure-resourcemanager-privatedns-2.21.0.jar +azure-resourcemanager-redis-2.21.0.jar +azure-resourcemanager-resources-2.21.0.jar +azure-resourcemanager-search-2.21.0.jar +azure-resourcemanager-servicebus-2.21.0.jar +azure-resourcemanager-sql-2.21.0.jar +azure-resourcemanager-storage-2.21.0.jar +azure-resourcemanager-trafficmanager-2.21.0.jar +azure-security-keyvault-keys-4.5.2.jar +azure-security-keyvault-secrets-4.5.2.jar +azure-storage-common-12.19.1.jar +azure-storage-file-share-12.16.1.jar +nimbus-jose-jwt-9.22.jar +woodstox-core-6.4.0.jar +azure-core-management-1.9.0.jar \ No newline at end of file