[fix] Enhance name pre checker in resource center (#10094)
* [fix] Enhance name pre checker in resource center
Add file name and directory checker to avoid directory traversal
* add some missing change and change docs
* change var name in directoryTraversal
* Fix ci
(cherry picked from commit 63f835715f)
This commit is contained in:
parent
2797fcfc68
commit
cbfeb7f777
|
|
@ -4,40 +4,47 @@ When third party jars are used in the scheduling process or user defined scripts
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
- Create a file
|
## Basic Operator
|
||||||
> The file format supports the following types: txt, log, sh, conf, cfg, py, java, sql, xml, hql, properties.
|
|
||||||
|
### Create a File
|
||||||
|
|
||||||
|
The file format supports the following types: txt, log, sh, conf, cfg, py, java, sql, xml, hql, properties.
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
- upload files
|
### Upload Files
|
||||||
|
|
||||||
> Upload file: Click the "Upload File" button to upload, drag the file to the upload area, the file name will be automatically completed with the uploaded file name.
|
Click the "Upload File" button to upload, drag the file to the upload area, the file name will be automatically completed with the uploaded file name.
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
- File View
|
### View File Content
|
||||||
|
|
||||||
> For the files that can be viewed, click the file name to view the file details.
|
For the files that can be viewed, click the file name to view the file details.
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
- Download file
|
### Download file
|
||||||
|
|
||||||
> Click the "Download" button in the file list to download the file or click the "Download" button in the upper right corner of the file details to download the file.
|
> Click the "Download" button in the file list to download the file or click the "Download" button in the upper right corner of the file details to download the file.
|
||||||
|
|
||||||
- File rename
|
### Rename File
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
- delete
|
### Delete File
|
||||||
> File list -> Click the "Delete" button to delete the specified file.
|
|
||||||
|
|
||||||
- Re-upload file
|
File list -> Click the "Delete" button to delete the specified file.
|
||||||
|
|
||||||
> Re-upload file: Click the "Re-upload File" button to upload a new file to replace the old file, drag the file to the re-upload area, the file name will be automatically completed with the new uploaded file name.
|
### Re-upload file
|
||||||
|
|
||||||
|
Click the "Re-upload File" button to upload a new file to replace the old file, drag the file to the re-upload area, the file name will be automatically completed with the new uploaded file name.
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
|
> Note: File name or source name of your local file can not contain specific characters like `.` or `/` when you trying to
|
||||||
|
> upload, create or rename file in resource center.
|
||||||
|
|
||||||
## Example
|
## Example
|
||||||
|
|
||||||
The example uses a simple shell script to demonstrate the use of resource center files in workflow definitions. The same is true for tasks such as MR and Spark, which require jar packages.
|
The example uses a simple shell script to demonstrate the use of resource center files in workflow definitions. The same is true for tasks such as MR and Spark, which require jar packages.
|
||||||
|
|
|
||||||
|
|
@ -2,43 +2,48 @@
|
||||||
|
|
||||||
当在调度过程中需要使用到第三方的 jar 或者用户需要自定义脚本的情况,可以通过在该页面完成相关操作。可创建的文件类型包括:`txt/log/sh/conf/py/java` 等。并且可以对文件进行编辑、重命名、下载和删除等操作。
|
当在调度过程中需要使用到第三方的 jar 或者用户需要自定义脚本的情况,可以通过在该页面完成相关操作。可创建的文件类型包括:`txt/log/sh/conf/py/java` 等。并且可以对文件进行编辑、重命名、下载和删除等操作。
|
||||||
|
|
||||||
|
## 基础操作
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
* 创建文件
|
### 创建文件
|
||||||
|
|
||||||
> 文件格式支持以下几种类型:txt、log、sh、conf、cfg、py、java、sql、xml、hql、properties
|
文件格式支持以下几种类型:txt、log、sh、conf、cfg、py、java、sql、xml、hql、properties
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
* 上传文件
|
### 上传文件
|
||||||
> 上传文件:点击"上传文件"按钮进行上传,将文件拖拽到上传区域,文件名会自动以上传的文件名称补全
|
|
||||||
|
上传文件:点击"上传文件"按钮进行上传,将文件拖拽到上传区域,文件名会自动以上传的文件名称补全
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
* 文件查看
|
### 文件查看
|
||||||
|
|
||||||
> 对可查看的文件类型,点击文件名称,可查看文件详情
|
对可查看的文件类型,点击文件名称,可查看文件详情
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
* 下载文件
|
### 下载文件
|
||||||
|
|
||||||
> 点击文件列表的"下载"按钮下载文件或者在文件详情中点击右上角"下载"按钮下载文件
|
点击文件列表的"下载"按钮下载文件或者在文件详情中点击右上角"下载"按钮下载文件
|
||||||
|
|
||||||
* 文件重命名
|
### 文件重命名
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
* 删除
|
### 删除文件
|
||||||
|
|
||||||
> 文件列表->点击"删除"按钮,删除指定文件
|
文件列表->点击"删除"按钮,删除指定文件
|
||||||
|
|
||||||
* 重新上传文件
|
### 重新上传文件
|
||||||
|
|
||||||
> 点击文件列表中的”重新上传文件“按钮进行重新上传文件,将文件拖拽到上传区域,文件名会自动以上传的文件名称补全
|
点击文件列表中的”重新上传文件“按钮进行重新上传文件,将文件拖拽到上传区域,文件名会自动以上传的文件名称补全
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
|
> 注意:上传、创建、重命名文件时,文件名和源文件名(上传时)均不能带有 `.` 以及 `/` 特殊符号。
|
||||||
|
|
||||||
## 任务样例
|
## 任务样例
|
||||||
|
|
||||||
该样例主要通过一个简单的 shell 脚本,来演示如何在工作流定义中使用资源中心的文件。像 MR、Spark 等任务需要用到 jar 包,也是同理。
|
该样例主要通过一个简单的 shell 脚本,来演示如何在工作流定义中使用资源中心的文件。像 MR、Spark 等任务需要用到 jar 包,也是同理。
|
||||||
|
|
|
||||||
|
|
@ -395,7 +395,7 @@ public enum Status {
|
||||||
K8S_CLIENT_OPS_ERROR(1300006, "k8s error with exception {0}", "k8s操作报错[{0}]"),
|
K8S_CLIENT_OPS_ERROR(1300006, "k8s error with exception {0}", "k8s操作报错[{0}]"),
|
||||||
VERIFY_K8S_NAMESPACE_ERROR(1300007, "verify k8s and namespace error", "验证k8s命名空间信息错误"),
|
VERIFY_K8S_NAMESPACE_ERROR(1300007, "verify k8s and namespace error", "验证k8s命名空间信息错误"),
|
||||||
DELETE_K8S_NAMESPACE_BY_ID_ERROR(1300008, "delete k8s namespace by id error", "删除命名空间错误"),
|
DELETE_K8S_NAMESPACE_BY_ID_ERROR(1300008, "delete k8s namespace by id error", "删除命名空间错误"),
|
||||||
VERIFY_PARAMETER_NAME_FAILED(1300009, "The file name verify failed", "文件命名校验失败"),
|
VERIFY_PARAMETER_NAME_FAILED(1300009, "The file name verify failed", "文件命名校验失败"),
|
||||||
STORE_OPERATE_CREATE_ERROR(1300010, "create the resource failed", "存储操作失败"),
|
STORE_OPERATE_CREATE_ERROR(1300010, "create the resource failed", "存储操作失败"),
|
||||||
GRANT_K8S_NAMESPACE_ERROR(1300011, "grant namespace error", "授权资源错误"),
|
GRANT_K8S_NAMESPACE_ERROR(1300011, "grant namespace error", "授权资源错误"),
|
||||||
QUERY_UNAUTHORIZED_NAMESPACE_ERROR(1300012, "query unauthorized namespace error", "查询未授权命名空间错误"),
|
QUERY_UNAUTHORIZED_NAMESPACE_ERROR(1300012, "query unauthorized namespace error", "查询未授权命名空间错误"),
|
||||||
|
|
|
||||||
|
|
@ -129,8 +129,8 @@ public class ResourcesServiceImpl extends BaseServiceImpl implements ResourcesSe
|
||||||
if (!result.getCode().equals(Status.SUCCESS.getCode())) {
|
if (!result.getCode().equals(Status.SUCCESS.getCode())) {
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
if (name.endsWith(FOLDER_SEPARATOR)) {
|
if (FileUtils.directoryTraversal(name)) {
|
||||||
result.setCode(Status.VERIFY_PARAMETER_NAME_FAILED.getCode());
|
putMsg(result, Status.VERIFY_PARAMETER_NAME_FAILED);
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
String fullName = getFullName(currentDir, name);
|
String fullName = getFullName(currentDir, name);
|
||||||
|
|
@ -517,6 +517,19 @@ public class ResourcesServiceImpl extends BaseServiceImpl implements ResourcesSe
|
||||||
private Result<Object> verifyFile(String name, ResourceType type, MultipartFile file) {
|
private Result<Object> verifyFile(String name, ResourceType type, MultipartFile file) {
|
||||||
Result<Object> result = new Result<>();
|
Result<Object> result = new Result<>();
|
||||||
putMsg(result, Status.SUCCESS);
|
putMsg(result, Status.SUCCESS);
|
||||||
|
|
||||||
|
if (FileUtils.directoryTraversal(name)) {
|
||||||
|
logger.error("file alias name {} verify failed", name);
|
||||||
|
putMsg(result, Status.VERIFY_PARAMETER_NAME_FAILED);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (file != null && FileUtils.directoryTraversal(Objects.requireNonNull(file.getOriginalFilename()))) {
|
||||||
|
logger.error("file original name {} verify failed", file.getOriginalFilename());
|
||||||
|
putMsg(result, Status.VERIFY_PARAMETER_NAME_FAILED);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
if (file != null) {
|
if (file != null) {
|
||||||
// file is empty
|
// file is empty
|
||||||
if (file.isEmpty()) {
|
if (file.isEmpty()) {
|
||||||
|
|
@ -973,6 +986,10 @@ public class ResourcesServiceImpl extends BaseServiceImpl implements ResourcesSe
|
||||||
if (!result.getCode().equals(Status.SUCCESS.getCode())) {
|
if (!result.getCode().equals(Status.SUCCESS.getCode())) {
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
|
if (FileUtils.directoryTraversal(fileName)) {
|
||||||
|
putMsg(result, Status.VERIFY_PARAMETER_NAME_FAILED);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
//check file suffix
|
//check file suffix
|
||||||
String nameSuffix = fileSuffix.trim();
|
String nameSuffix = fileSuffix.trim();
|
||||||
|
|
|
||||||
|
|
@ -204,4 +204,25 @@ public class FileUtils {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check whether the given string type of path can be traversal or not, return true if path could
|
||||||
|
* traversal, and return false if it is not.
|
||||||
|
*
|
||||||
|
* @param filename String type of filename
|
||||||
|
* @return whether file path could be traversal or not
|
||||||
|
*/
|
||||||
|
public static boolean directoryTraversal(String filename){
|
||||||
|
if (filename.contains(FOLDER_SEPARATOR)) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
File file = new File(filename);
|
||||||
|
try {
|
||||||
|
File canonical = file.getCanonicalFile();
|
||||||
|
File absolute = file.getAbsoluteFile();
|
||||||
|
return !canonical.equals(absolute);
|
||||||
|
} catch (IOException e) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -88,4 +88,34 @@ public class FileUtilsTest {
|
||||||
Assert.assertEquals(content, fileContent);
|
Assert.assertEquals(content, fileContent);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
public void testDirectoryTraversal() {
|
||||||
|
// test case which do not directory traversal
|
||||||
|
String path;
|
||||||
|
path = "abc.txt";
|
||||||
|
Assert.assertFalse(FileUtils.directoryTraversal(path));
|
||||||
|
|
||||||
|
path = "abc...txt";
|
||||||
|
Assert.assertFalse(FileUtils.directoryTraversal(path));
|
||||||
|
|
||||||
|
path = "..abc.txt";
|
||||||
|
Assert.assertFalse(FileUtils.directoryTraversal(path));
|
||||||
|
|
||||||
|
// test case which will directory traversal
|
||||||
|
path = "../abc.txt";
|
||||||
|
Assert.assertTrue(FileUtils.directoryTraversal(path));
|
||||||
|
|
||||||
|
path = "../../abc.txt";
|
||||||
|
Assert.assertTrue(FileUtils.directoryTraversal(path));
|
||||||
|
|
||||||
|
path = "abc../def.txt";
|
||||||
|
Assert.assertTrue(FileUtils.directoryTraversal(path));
|
||||||
|
|
||||||
|
path = "abc./def.txt";
|
||||||
|
Assert.assertTrue(FileUtils.directoryTraversal(path));
|
||||||
|
|
||||||
|
path = "abc/def...txt";
|
||||||
|
Assert.assertTrue(FileUtils.directoryTraversal(path));
|
||||||
|
}
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue