[Feature] Enable users to switch and install conda env in jupyter task (#10337)
This commit is contained in:
Eric Gao
GitHub
parent
79ce5d78cd
commit
64cee03fed
|
|
@ -11,6 +11,41 @@ it will use `papermill` to evaluate jupyter notes. Click [here](https://papermil
|
|||
Click [here](https://docs.conda.io/en/latest/) for more information about `conda`.
|
||||
- `conda.path` is set to `/opt/anaconda3/etc/profile.d/conda.sh` by default. If you have no idea where your `conda` is, simply run `conda info | grep -i 'base environment'`.
|
||||
|
||||
> NOTICE: `Jupyter Task Plugin` uses `source` command to activate conda environment.
|
||||
> If your tenant does not have permission to use `source`, `Jupyter Task Plugin` will not function.
|
||||
|
||||
|
||||
## Python Dependency Management
|
||||
|
||||
### Use Pre-Installed Conda Environment
|
||||
|
||||
1. Create a conda environment manually or using `shell task` on your target worker.
|
||||
2. In your `jupyter task`, set `condaEnvName` as the name of the conda environment you just created.
|
||||
|
||||
### Use Packed Conda Environment
|
||||
|
||||
1. Use [Conda-Pack](https://conda.github.io/conda-pack/) to pack your conda environment into `tarball`.
|
||||
2. Upload packed conda environment to `resource center`.
|
||||
3. Select your packed conda environment as `resource` in your `jupyter task`, e.g. `jupyter_env.tar.gz`.
|
||||
|
||||
> **_Note:_** Make sure you follow the [Conda-Pack](https://conda.github.io/conda-pack/) official instructions.
|
||||
> If you unpack your packed conda environment, the directory structure should be the same as below:
|
||||
|
||||
```
|
||||
.
|
||||
├── bin
|
||||
├── conda-meta
|
||||
├── etc
|
||||
├── include
|
||||
├── lib
|
||||
├── share
|
||||
└── ssl
|
||||
```
|
||||
|
||||
> NOTICE: Please follow the `conda pack` instructions above strictly, and DO NOT modify `bin/activate`.
|
||||
> `Jupyter Task Plugin` uses `source` command to activate your packed conda environment.
|
||||
> If you are concerned about using `source`, choose other options to manage your python dependency.
|
||||
|
||||
## Create Task
|
||||
|
||||
- Click Project Management-Project Name-Workflow Definition, and click the "Create Workflow" button to enter the DAG editing page.
|
||||
|
|
@ -28,7 +63,7 @@ Click [here](https://docs.conda.io/en/latest/) for more information about `conda
|
|||
- Cpu quota: Assign the specified CPU time quota to the task executed. Takes a percentage value. Default -1 means unlimited. For example, the full CPU load of one core is 100%,and that of 16 cores is 1600%. This function is controlled by [task.resource.limit.state](../../architecture/configuration.md)
|
||||
- Max memory：Assign the specified max memory to the task executed. Exceeding this limit will trigger oom to be killed and will not automatically retry. Takes an MB value. Default -1 means unlimited. This function is controlled by [task.resource.limit.state](../../architecture/configuration.md)
|
||||
- Timeout alarm: Check the timeout alarm and timeout failure. When the task exceeds the "timeout period", an alarm email will send and the task execution will fail.
|
||||
- Conda Env Name: Name of conda environment.
|
||||
- Conda Env Name: Name of conda environment or packed conda environment tarball.
|
||||
- Input Note Path: Path of input jupyter note template.
|
||||
- Out Note Path: Path of output note.
|
||||
- Jupyter Parameters: Parameters in json format used for jupyter note parameterization.
|
||||
|
|
|
|||
|
|
@ -1,23 +1,56 @@
|
|||
# Jupyter
|
||||
|
||||
## Overview
|
||||
## 综述
|
||||
|
||||
`Jupyter`任务类型，用于创建并执行`Jupyter`类型任务。worker 执行该任务的时候，会通过`papermill`执行`jupyter note`。
|
||||
点击[这里](https://papermill.readthedocs.io/en/latest/) 获取更多关于`papermill`的信息。
|
||||
|
||||
## Conda Configuration
|
||||
## Conda虚拟环境配置
|
||||
|
||||
- 在`common.properties`配置`conda.path`，将其指向您的`conda.sh`。这里的`conda`应该是您用来管理您的 `papermill`和`jupyter`所在python环境的相同`conda`。
|
||||
点击 [这里](https://docs.conda.io/en/latest/) 获取更多关于`conda`的信息.
|
||||
- `conda.path`默认设置为`/opt/anaconda3/etc/profile.d/conda.sh`。 如果您不清楚您的`conda`环境在哪里，只需要在命令行执行`conda info | grep -i 'base environment'`即可获得。
|
||||
|
||||
> 注意：`Jupyter任务插件`使用`source`命令激活conda环境，
|
||||
> 如果您的租户没有`source`命令使用权限，`Jupyter任务插件`将无法使用。
|
||||
|
||||
## Create Task
|
||||
## Python依赖管理
|
||||
|
||||
### 使用预装好的Conda环境
|
||||
|
||||
1. 手动或使用`shell任务`在您的目标机器上创建conda环境。
|
||||
2. 在您的`jupyter任务`中，将`condaEnvName`设置为您在上一步创建的conda环境名。
|
||||
|
||||
### 使用打包的Conda环境
|
||||
|
||||
1. 使用 [Conda-Pack](https://conda.github.io/conda-pack/) 将您的conda环境打包成`tarball`.
|
||||
2. 将您打包好的conda环境上传到`资源中心`.
|
||||
3. 在您的`jupyter任务`资源设置中，添加您在上一步中上传的conda环境包，如`jupyter_env.tar.gz`.
|
||||
|
||||
> **_提示:_** 请您按照 [Conda-Pack](https://conda.github.io/conda-pack/) 官方指导打包conda环境，
|
||||
> 正确打包出的conda环境包解压后文件目录结构应和下图完全一致：
|
||||
|
||||
```
|
||||
.
|
||||
├── bin
|
||||
├── conda-meta
|
||||
├── etc
|
||||
├── include
|
||||
├── lib
|
||||
├── share
|
||||
└── ssl
|
||||
```
|
||||
|
||||
> 注意: 请严格按照上述`conda pack`指示操作，并且不要随意修改`bin/activate`。
|
||||
> `Jupyter任务插件`使用`source`命令激活您打包的conda环境。
|
||||
> 若您对使用`source`命令有安全性上的担忧，请使用其他方法管理您的python依赖。
|
||||
|
||||
## 创建任务
|
||||
|
||||
- 点击项目管理-项目名称-工作流定义，点击"创建工作流"按钮，进入DAG编辑页面。
|
||||
- 工具栏中拖动 <img src="../../../../img/tasks/icons/jupyter.png" width="15"/> 到画板中，即可完成创建。
|
||||
|
||||
## Task Parameter
|
||||
## 任务参数
|
||||
|
||||
- 任务名称：设置任务的名称。一个工作流定义中的节点名称是唯一的。
|
||||
- 运行标志：标识这个节点是否能正常调度,如果不需要执行，可以打开禁止执行开关。
|
||||
|
|
@ -30,7 +63,7 @@
|
|||
- 最大内存：为执行的任务分配指定的内存大小，超过会触发OOM被Kill同时不会进行自动重试，单位MB，默认-1代表不限制。这个功能由 [task.resource.limit.state](../../architecture/configuration.md) 控制
|
||||
- 超时告警：勾选超时告警、超时失败，当任务超过"超时时长"后，会发送告警邮件并且任务执行失败.这个功能由 [task.resource.limit.state](../../architecture/configuration.md) 控制
|
||||
- 前置任务：选择当前任务的前置任务，会将被选择的前置任务设置为当前任务的上游。
|
||||
- Conda Env Name: Conda环境名称。
|
||||
- Conda Env Name: Conda环境或打包的Conda环境包名称
|
||||
- Input Note Path: 输入的jupyter note模板路径。
|
||||
- Out Note Path: 输出的jupyter note路径。
|
||||
- Jupyter Parameters: 用于对接jupyter note参数化的JSON格式参数。
|
||||
|
|
@ -40,9 +73,9 @@
|
|||
- Jupyter Start Timeout: 对于jupyter notebook kernel设定的启动超时时间。
|
||||
- Others: 传入papermill命令的其他参数。
|
||||
|
||||
## Task Example
|
||||
## 任务样例
|
||||
|
||||
### Jupyter Task Example
|
||||
### 简单的Jupyter任务样例
|
||||
|
||||
这个示例展示了如何创建Jupyter任务节点：
|
||||
|
||||
|
|
|
|||
|
|
@ -33,6 +33,18 @@ public class JupyterConstants {
|
|||
*/
|
||||
public static final String CONDA_ACTIVATE = "conda activate";
|
||||
|
||||
/**
|
||||
* create and activate conda env from tar
|
||||
*/
|
||||
public static final String CREATE_ENV_FROM_TAR = "mkdir jupyter_env && " +
|
||||
"tar -xzf %s -C jupyter_env && " +
|
||||
"source jupyter_env/bin/activate";
|
||||
|
||||
/**
|
||||
* file suffix tar.gz
|
||||
*/
|
||||
public static final String TAR_SUFFIX = ".tar.gz";
|
||||
|
||||
/**
|
||||
* jointer to combine two command
|
||||
*/
|
||||
|
|
|
|||
|
|
@ -17,8 +17,11 @@
|
|||
|
||||
package org.apache.dolphinscheduler.plugin.task.jupyter;
|
||||
|
||||
import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
|
||||
import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* jupyter parameters
|
||||
*/
|
||||
|
|
@ -69,6 +72,10 @@ public class JupyterParameters extends AbstractParameters {
|
|||
*/
|
||||
private String others;
|
||||
|
||||
/**
|
||||
* resource list
|
||||
*/
|
||||
private List<ResourceInfo> resourceList;
|
||||
|
||||
public String getCondaEnvName() {
|
||||
return condaEnvName;
|
||||
|
|
@ -142,6 +149,19 @@ public class JupyterParameters extends AbstractParameters {
|
|||
this.others = others;
|
||||
}
|
||||
|
||||
public List<ResourceInfo> getResourceList() {
|
||||
return resourceList;
|
||||
}
|
||||
|
||||
public void setResourceList(List<ResourceInfo> resourceList) {
|
||||
this.resourceList = resourceList;
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<ResourceInfo> getResourceFilesList() {
|
||||
return resourceList;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean checkParameters() {
|
||||
return condaEnvName != null && inputNotePath != null && outputNotePath != null;
|
||||
|
|
|
|||
|
|
@ -107,8 +107,14 @@ public class JupyterTask extends AbstractTaskExecutor {
|
|||
args.add(JupyterConstants.CONDA_INIT);
|
||||
args.add(condaPath);
|
||||
args.add(JupyterConstants.JOINTER);
|
||||
args.add(JupyterConstants.CONDA_ACTIVATE);
|
||||
args.add(jupyterParameters.getCondaEnvName());
|
||||
String condaEnvName = jupyterParameters.getCondaEnvName();
|
||||
if (condaEnvName.endsWith(JupyterConstants.TAR_SUFFIX)) {
|
||||
args.add(String.format(JupyterConstants.CREATE_ENV_FROM_TAR, condaEnvName));
|
||||
} else {
|
||||
args.add(JupyterConstants.CONDA_ACTIVATE);
|
||||
args.add(jupyterParameters.getCondaEnvName());
|
||||
}
|
||||
|
||||
args.add(JupyterConstants.JOINTER);
|
||||
args.add(JupyterConstants.PAPERMILL);
|
||||
args.add(jupyterParameters.getInputNotePath());
|
||||
|
|
|
|||
|
|
@ -46,8 +46,8 @@ import static org.powermock.api.mockito.PowerMockito.when;
|
|||
public class JupyterTaskTest {
|
||||
|
||||
@Test
|
||||
public void testBuildJupyterCommand() throws Exception {
|
||||
String parameters = buildJupyterCommand();
|
||||
public void testBuildJupyterCommandWithLocalEnv() throws Exception {
|
||||
String parameters = buildJupyterCommandWithLocalEnv();
|
||||
TaskExecutionContext taskExecutionContext = PowerMockito.mock(TaskExecutionContext.class);
|
||||
when(taskExecutionContext.getTaskParams()).thenReturn(parameters);
|
||||
PowerMockito.mockStatic(PropertyUtils.class);
|
||||
|
|
@ -71,7 +71,35 @@ public class JupyterTaskTest {
|
|||
"--progress-bar");
|
||||
}
|
||||
|
||||
private String buildJupyterCommand() {
|
||||
@Test
|
||||
public void testBuildJupyterCommandWithPackedEnv() throws Exception {
|
||||
String parameters = buildJupyterCommandWithPackedEnv();
|
||||
TaskExecutionContext taskExecutionContext = PowerMockito.mock(TaskExecutionContext.class);
|
||||
when(taskExecutionContext.getTaskParams()).thenReturn(parameters);
|
||||
PowerMockito.mockStatic(PropertyUtils.class);
|
||||
when(PropertyUtils.getString(any())).thenReturn("/opt/anaconda3/etc/profile.d/conda.sh");
|
||||
JupyterTask jupyterTask = spy(new JupyterTask(taskExecutionContext));
|
||||
jupyterTask.init();
|
||||
Assert.assertEquals(jupyterTask.buildCommand(),
|
||||
"source /opt/anaconda3/etc/profile.d/conda.sh && " +
|
||||
"mkdir jupyter_env && " +
|
||||
"tar -xzf jupyter.tar.gz -C jupyter_env && " +
|
||||
"source jupyter_env/bin/activate && " +
|
||||
"papermill " +
|
||||
"/test/input_note.ipynb " +
|
||||
"/test/output_note.ipynb " +
|
||||
"--parameters city Shanghai " +
|
||||
"--parameters factor 0.01 " +
|
||||
"--kernel python3 " +
|
||||
"--engine default_engine " +
|
||||
"--execution-timeout 10 " +
|
||||
"--start-timeout 3 " +
|
||||
"--version " +
|
||||
"--inject-paths " +
|
||||
"--progress-bar");
|
||||
}
|
||||
|
||||
private String buildJupyterCommandWithLocalEnv() {
|
||||
JupyterParameters jupyterParameters = new JupyterParameters();
|
||||
jupyterParameters.setCondaEnvName("jupyter-lab");
|
||||
jupyterParameters.setInputNotePath("/test/input_note.ipynb");
|
||||
|
|
@ -85,4 +113,18 @@ public class JupyterTaskTest {
|
|||
return JSONUtils.toJsonString(jupyterParameters);
|
||||
}
|
||||
|
||||
private String buildJupyterCommandWithPackedEnv() {
|
||||
JupyterParameters jupyterParameters = new JupyterParameters();
|
||||
jupyterParameters.setCondaEnvName("jupyter.tar.gz");
|
||||
jupyterParameters.setInputNotePath("/test/input_note.ipynb");
|
||||
jupyterParameters.setOutputNotePath("/test/output_note.ipynb");
|
||||
jupyterParameters.setParameters("{\"city\": \"Shanghai\", \"factor\": \"0.01\"}");
|
||||
jupyterParameters.setKernel("python3");
|
||||
jupyterParameters.setEngine("default_engine");
|
||||
jupyterParameters.setExecutionTimeout("10");
|
||||
jupyterParameters.setStartTimeout("3");
|
||||
jupyterParameters.setOthers("--version");
|
||||
return JSONUtils.toJsonString(jupyterParameters);
|
||||
}
|
||||
|
||||
}
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@
|
|||
* limitations under the License.
|
||||
*/
|
||||
import { useI18n } from 'vue-i18n'
|
||||
import { useCustomParams } from '.'
|
||||
import { useCustomParams, useResources } from '.'
|
||||
import type { IJsonItem } from '../types'
|
||||
|
||||
export function useJupyter(model: { [field: string]: any }): IJsonItem[] {
|
||||
|
|
@ -121,6 +121,7 @@ export function useJupyter(model: { [field: string]: any }): IJsonItem[] {
|
|||
placeholder: t('project.node.jupyter_others_tips')
|
||||
}
|
||||
},
|
||||
useResources(),
|
||||
...useCustomParams({ model, field: 'localParams', isSimple: false })
|
||||
]
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in New Issue