Compare commits

...

37 Commits

Author SHA1 Message Date
i-robot be08033552
!1464 modify web-interface to add step of change max length of favorite sql
Merge pull request !1464 from 孙锐/Branch-KunPengZhongzhi
2022-05-16 06:52:16 +00:00
sunrui c5622813f7 change hetu-docs web-interface.md 2022-05-11 16:13:15 +08:00
i-robot 669d8bb4dc
!1452 fixed bug of I52KEW and I51RTJ
Merge pull request !1452 from 孙锐/Branch-KunPengZhongzhi
2022-05-04 10:23:51 +00:00
sunrui 15f851fc94 add show more. 2022-05-04 15:24:36 +08:00
sunrui f9303807d9 Modify the style and text of pop-up window, fixed bug of https://gitee.com/openlookeng/hetu-core/issues/I52KEW 2022-04-25 16:32:04 +08:00
sunrui df7220a517 change daterangepicker.js, make minute disable 2022-04-22 17:04:33 +08:00
i-robot bec7170aad
!1442 fixed bug of issues I52KEW
Merge pull request !1442 from 孙锐/Branch-KunPengZhongzhi
2022-04-20 08:13:54 +00:00
sunrui 7cc29a611d fixed bug of https://gitee.com/openlookeng/hetu-core/issues/I52KEW 2022-04-15 17:26:34 +08:00
i-robot 01a38285c4
!1440 fixed bug of issues#I52AVF
Merge pull request !1440 from 孙锐/Branch-KunPengZhongzhi
2022-04-14 01:32:27 +00:00
sunrui 961cbcaee5 change ShowCatalogStore.java, fixed bug of https://gitee.com/openlookeng/hetu-core/issues/I52AVF 2022-04-13 15:03:14 +08:00
sunrui 485d00e52b change ShowCatalogStore.java, fixed bug of https://gitee.com/openlookeng/hetu-core/issues/I52AVF 2022-04-13 14:33:33 +08:00
sunrui a56a2050b7 change ShowCatalogStore.java, fixed bug of https://gitee.com/openlookeng/hetu-core/issues/I52AVF 2022-04-13 09:17:00 +08:00
sunrui 29d9999fa9 change ShowCatalogStore.java, fixed bug of https://gitee.com/openlookeng/hetu-core/issues/I52AVF 2022-04-12 17:32:24 +08:00
i-robot bf974e238d
!1437 change ShowCatalogStore.java, fixed bug of NPE
Merge pull request !1437 from 孙锐/Branch-KunPengZhongzhi
2022-04-09 08:15:55 +00:00
sunrui 83d4bfd241 change ShowCatalogStore.java, fixed codestyle bug 2022-04-09 14:43:49 +08:00
sunrui 848301ceda change ShowCatalogStore.java, fixed codestyle bug 2022-04-09 11:11:30 +08:00
sunrui 213663ae79 change ShowCatalogStore.java, fixed bug of NPE 2022-04-08 09:22:13 +08:00
i-robot 82b05ffc88
!1426 fixed bug from Branch-KunPengZhongzhi
Merge pull request !1426 from 孙锐/Branch-KunPengZhongzhi
2022-04-01 01:49:05 +00:00
sunrui 70e2bd1080 fixed bug of codecheck 2022-03-31 20:38:46 +08:00
sunrui 1a14a5e35b fixed bug as the following list;
1. https://gitee.com/openlookeng/hetu-core/issues/I4YX6M
2. https://gitee.com/openlookeng/hetu-core/issues/I4YX37
3. https://gitee.com/openlookeng/hetu-core/issues/I4YVHQ
4. https://gitee.com/openlookeng/hetu-core/issues/I4Z2PC
5. https://gitee.com/openlookeng/hetu-core/issues/I4YSI7
2022-03-25 17:02:25 +08:00
i-robot e0c9459314
!1418 Branch-KunPengZhongZhi WebUI Enhancement js code
Merge pull request !1418 from 孙锐/Branch-KunPengZhongzhi
2022-03-17 04:15:51 +00:00
sunrui 7dff349b00 upload js from webapp/dist 2022-03-16 15:06:14 +08:00
i-robot d05a786cc3
!1352 Branch-KunPengZhongZhi WebUI Enhancement
Merge pull request !1352 from 孙锐/Branch-KunPengZhongzhi
2022-03-15 11:06:33 +00:00
sunrui 549fc0a37a update web-interface.md 2022-03-15 14:51:53 +08:00
sunrui 43d4c6f0ab update hetu-metastore to fixed bug 2022-03-14 19:24:55 +08:00
孙锐 82d0216071
update hetu-listener/src/main/java/io/hetu/core/eventlistener/util/HetuLogUtil.java. 2022-03-11 01:18:59 +00:00
sunrui 6f36cd2194 update hetu-doc about webUI 2022-03-08 19:45:09 +08:00
sunrui 6330095466 fixed some bug from codecheck 2022-03-08 14:57:39 +08:00
sunrui 5980ea1e66 fixed bug from Testhetulog 2022-03-04 17:24:09 +08:00
sunrui 2a3980a3fe fixed code check 2022-03-04 16:47:14 +08:00
sunrui 318c76f952 use try catch finally to open file 2022-03-04 15:36:27 +08:00
sunrui 7039d9a4d2 use airlift.log to enhance L rather than log4j;
change collect button;
add config for queryHistory-max-count and collectSql-max-count;
2022-03-04 15:03:30 +08:00
sunrui 41522939cf add pagenation for AuditLog WebUi 2022-01-15 02:56:58 -08:00
sunrui c9b7e28ec1 Code completion of sql-editor 2022-01-07 03:36:03 -08:00
sunrui 2656701286 make queryInfo persistence by hetu-metadata, collect user sql by hetu-metadata 2022-01-07 02:28:10 -08:00
sunrui 17455b0876 show catalog by reading properties file 2022-01-07 02:20:17 -08:00
sunrui 3a54b7ba01 Modify Hetu-listener to monitor openLooKeng cluster startup and shutdown,WebUi user login and exit 2022-01-07 02:04:39 -08:00
124 changed files with 142801 additions and 3763 deletions

View File

@ -1,6 +1,6 @@
# Audit Log
openLooKeng audit logging functionality is a custom event listener that is invoked for query creation and query completion (success or failure)
openLooKeng audit logging functionality is a custom event listener, which monitors the start and stop of openLooKeng cluster and the dynamic addition and deletion of nodes in the cluster; Listen to WebUi user login and exit events; Listen for query events and call when the query is created and completed (success or failure).
An audit log contains the following information:
1. time when an event occurs
@ -24,15 +24,17 @@ To enable audit logging feature, the following configs must be present in `etc/e
hetu.event.listener.type=AUDIT
hetu.event.listener.listen.query.creation=true
hetu.event.listener.listen.query.completion=true
hetu.auditlog.logoutput=/var/log/
hetu.auditlog.logconversionpattern=yyyy-MM-dd.HH
```
Other audit logging properties include:
The following is a detailed description of audit logging properties:
`hetu.event.listener.audit.file`: Optional property to define absolute file path for the audit file. Ensure the process running the openLooKeng server has write access to this directory.
`hetu.event.listener.type`: property to define logging type for audit files. Allowed values are AUDIT and LOGGER.
`hetu.event.listener.audit.filecount`: Optional property to define the number of files to use
`hetu.auditlog.logoutput`: property to define absolute file directory for audit files. Ensure the process running the openLooKeng server has write access to this directory.
`hetu.event.listener.audit.limit`: Optional property to define the maximum number of bytes to write to any one file
`hetu.auditlog.logconversionpattern`: property to define the conversion pattern of audit files. Allowed values are yyyy-MM-dd.HH and yyyy-MM-dd.
Example configuration file:
@ -41,7 +43,6 @@ event-listener.name=hetu-listener
hetu.event.listener.type=AUDIT
hetu.event.listener.listen.query.creation=true
hetu.event.listener.listen.query.completion=true
hetu.event.listener.audit.file=/var/log/hetu/hetu-audit.log
hetu.event.listener.audit.filecount=1
hetu.event.listener.audit.limit=100000
hetu.auditlog.logoutput=/var/log/
hetu.auditlog.logconversionpattern=yyyy-MM-dd.HH
```

View File

@ -34,3 +34,25 @@ and statistics about the query is available by clicking the *JSON* link. These v
> - **Default value:** `false`
>
> Insecure authentication over HTTP is disabled by default. This could be overridden via "hetu.queryeditor-ui.allow-insecure-over-http" property of "etc/config.properties" (e.g. hetu.queryeditor-ui.allow-insecure-over-http=true).
### `hetu.queryhistory.max-count`
> - **Type:** `int`
> - **Default value:** `1000`
>
> The maximum number of query history stored by openLooKeng. This could be overridden via "hetu.queryhistory.max-count" of "etc/config.properties".
### `hetu.collectionsql.max-count`
> - **Type:** `int`
> - **Default value:** `100`
>
> The Maximum number of SQL collected by each user. This could be overridden via "hetu.collectionsql.max-count" of "etc/config.properties".
## Remarks
The max length of the favorite SQL is 600 by default. You can modify it through the following steps:
1. Login MySQL database according to the JDBC configuration of `hetu-metastore.properties`
2. Select table hetu_favorite, execute script `alter table hetu_favorite modify query varchar(2000) not null;` to modify the max length of the favorite SQL.

View File

@ -1,6 +1,6 @@
# 审计日志
openLooKeng审计日志记录功能是一个自定义事件监听器在查询创建和完成成功或失败时调用。审计日志包含以下信息
openLooKeng审计日志记录功能是一个自定义事件监听器监听openLooKeng集群启停与集群中节点的动态添加与删除事件监听WebUi用户登录与退出事件监听查询事件,在查询创建和完成(成功或失败)时调用。审计日志包含以下信息:
1. 事件发生时间
2. 用户ID
@ -23,15 +23,17 @@ openLooKeng审计日志记录功能是一个自定义事件监听器在查询
hetu.event.listener.type=AUDIT
hetu.event.listener.listen.query.creation=true
hetu.event.listener.listen.query.completion=true
hetu.auditlog.logoutput=/var/log/
hetu.auditlog.logconversionpattern=yyyy-MM-dd.HH
```
其他审计日志记录属性包括:
`hetu.event.listener.audit.file`可选属性用于定义审计文件的绝对文件路径。确保运行openLooKeng服务器的进程对该目录有写权限
`hetu.event.listener.type`用于定义审计日志的记录类型允许的值为AUDIT和LOGGER
`hetu.event.listener.audit.filecount`:可选属性,用于定义要使用的文件数
`hetu.auditlog.logoutput`用于定义审计文件的绝对目录路径。确保运行openLooKeng服务器的进程对该目录有写权限
`hetu.event.listener.audit.limit`:可选属性,用于定义写入任一文件的最大字节数
`hetu.auditlog.logconversionpattern`用于定义审计日志的轮转模式。允许的值为yyyy-MM-dd.HH和yyyy-MM-dd
配置文件示例:
@ -43,4 +45,6 @@ hetu.event.listener.listen.query.completion=true
hetu.event.listener.audit.file=/var/log/hetu/hetu-audit.log
hetu.event.listener.audit.filecount=1
hetu.event.listener.audit.limit=100000
hetu.auditlog.logoutput=/var/log/
hetu.auditlog.logconversionpattern=yyyy-MM-dd.HH
```

View File

@ -31,3 +31,26 @@ openLooKeng提供了一个用于监视和管理查询的Web界面。Web界面可
> - **默认值:** `false`
>
> 默认情况下基于HTTP的非安全环境禁用WEB UI。可以通过配置`etc/config.properties`文件的`hetu.queryeditor-ui.allow-insecure-over-http`属性启用(例子: hetu.queryeditor-ui.allow-insecure-over-http=true)。
### `hetu.queryhistory.max-count`
> - **Type:** `int`
> - **Default value:** `1000`
>
> openLooKeng储存的历史查询记录最大数量。可以通过配置`etc/config.properties`文件的`hetu.queryhistory.max-count`属性修改。
### `hetu.collectionsql.max-count`
> - **Type:** `int`
> - **Default value:** `100`
>
> 每位用户收藏sql语句条数上限.可以通过配置`etc/config.properties`文件的`hetu.collectionsql.max-count`属性修改。
## 备注
收藏sql语句的最大长度默认为600可通过如下步骤对其进行修改
1. 根据hetu-metastore.properties文件中jdbc配置信息登录mysql数据库。
2. 选中hetu_favorite表使用命令`alter table hetu_favorite modify query varchar(2000) not null;`修改收藏语句最大长度为2000。

View File

@ -58,6 +58,12 @@
<groupId>io.hetu.core</groupId>
<artifactId>presto-main</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>log4j-over-slf4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>

View File

@ -42,6 +42,10 @@ public class HetuEventListenerConfig
private int auditFileCount = 1;
private String auditLogFile;
private String auditLogConversionPattern;
public BaseEventListener.Type getType()
{
return type;
@ -239,4 +243,42 @@ public class HetuEventListenerConfig
this.auditFileLimit = auditFileLimit;
return this;
}
/**
* set auditLogfile
*
* @param auditLogFile auditlogfile from properties file
* @return config object
*/
@Config("hetu.auditlog.logoutput")
@ConfigDescription("AuditLog event output file path")
public HetuEventListenerConfig setAuditLogFile(String auditLogFile)
{
this.auditLogFile = auditLogFile;
return this;
}
/**
* set auditLogConversionPattern
*
* @param auditLogConversionPattern auditLogConversionPattern from properties file
* @return config object
*/
@Config("hetu.auditlog.logconversionpattern")
@ConfigDescription("define the ConversionPattern of AuditLog event")
public HetuEventListenerConfig setAuditLogConversionPattern(String auditLogConversionPattern)
{
this.auditLogConversionPattern = auditLogConversionPattern;
return this;
}
public String getAuditLogFile()
{
return auditLogFile;
}
public String getAuditLogConversionPattern()
{
return auditLogConversionPattern;
}
}

View File

@ -16,6 +16,7 @@ package io.hetu.core.eventlistener;
import io.airlift.configuration.ConfigurationFactory;
import io.hetu.core.eventlistener.listeners.BaseEventListener;
import io.hetu.core.eventlistener.util.HetuLogUtil;
import io.prestosql.spi.eventlistener.EventListener;
import io.prestosql.spi.eventlistener.EventListenerFactory;
@ -39,6 +40,7 @@ public class HetuEventListenerFactory
{
ConfigurationFactory configurationFactory = new ConfigurationFactory(properties);
HetuEventListenerConfig config = configurationFactory.build(HetuEventListenerConfig.class);
HetuLogUtil.create(config);
return BaseEventListener.create(config);
}
}

View File

@ -16,6 +16,7 @@ package io.hetu.core.eventlistener.listeners;
import io.airlift.log.Logger;
import io.hetu.core.eventlistener.HetuEventListenerConfig;
import io.hetu.core.eventlistener.util.HetuLogUtil;
import io.hetu.core.eventlistener.util.ListenerErrorCode;
import io.prestosql.spi.PrestoException;
import io.prestosql.spi.eventlistener.QueryCompletedEvent;
@ -66,6 +67,16 @@ class AuditEventLogger
protected void onQueryCreatedEvent(QueryCreatedEvent queryCreatedEvent)
{
QueryContext queryContext = queryCreatedEvent.getContext();
java.util.logging.Logger log = HetuLogUtil.getLoggerByName(queryContext.getUser(), "INFO", HetuLogUtil.AuditType.Sql);
log.info(queryCreatedEvent.getCreateTime().atZone(ZoneId.systemDefault()) +
" UserName=" + queryContext.getUser() +
" UserIp=" + queryContext.getRemoteClientAddress().orElse(null) +
" queryId=" + queryCreatedEvent.getMetadata().getQueryId() +
" operation= " + queryCreatedEvent.getClass().getSimpleName() +
" stmt={" + queryCreatedEvent.getMetadata().getQuery() +
"} status=" + queryCreatedEvent.getMetadata().getQueryState() + " " +
queryCreatedEvent.getClass().getCanonicalName());
logger.info(queryCreatedEvent.getCreateTime().atZone(ZoneId.systemDefault()) +
" UserName=" + queryContext.getUser() +
" UserIp=" + queryContext.getRemoteClientAddress().orElse(null) +
@ -80,6 +91,18 @@ class AuditEventLogger
protected void onQueryCompletedEvent(QueryCompletedEvent queryCompletedEvent)
{
QueryContext queryContext = queryCompletedEvent.getContext();
java.util.logging.Logger log = HetuLogUtil.getLoggerByName(queryContext.getUser(), "INFO", HetuLogUtil.AuditType.Sql);
log.info(queryCompletedEvent.getCreateTime().atZone(ZoneId.systemDefault()) +
" UserName=" + queryContext.getUser() +
" UserIp=" + queryContext.getRemoteClientAddress().orElse(null) +
" queryId=" + queryCompletedEvent.getMetadata().getQueryId() +
" operation= " + queryCompletedEvent.getClass().getSimpleName() +
" WrittenRows= " + queryCompletedEvent.getStatistics().getWrittenRows() +
" TotalRows= " + queryCompletedEvent.getStatistics().getTotalRows() +
" OutputRows= " + queryCompletedEvent.getStatistics().getOutputRows() +
" stmt={" + queryCompletedEvent.getMetadata().getQuery() +
"} status=" + queryCompletedEvent.getMetadata().getQueryState() +
" " + queryCompletedEvent.getClass().getCanonicalName());
logger.info(queryCompletedEvent.getCreateTime().atZone(ZoneId.systemDefault()) +
" UserName=" + queryContext.getUser() +
" UserIp=" + queryContext.getRemoteClientAddress().orElse(null) +

View File

@ -15,11 +15,18 @@
package io.hetu.core.eventlistener.listeners;
import io.hetu.core.eventlistener.HetuEventListenerConfig;
import io.hetu.core.eventlistener.util.HetuLogUtil;
import io.prestosql.spi.eventlistener.AuditLogEvent;
import io.prestosql.spi.eventlistener.EventListener;
import io.prestosql.spi.eventlistener.QueryCompletedEvent;
import io.prestosql.spi.eventlistener.QueryCreatedEvent;
import io.prestosql.spi.eventlistener.SplitCompletedEvent;
import java.time.LocalDateTime;
import java.time.ZoneId;
import java.time.ZonedDateTime;
import java.util.logging.Logger;
import static java.util.Objects.requireNonNull;
public abstract class BaseEventListener
@ -60,7 +67,6 @@ public abstract class BaseEventListener
public static EventListener create(HetuEventListenerConfig config)
{
Type type = config.getType();
if (type == Type.AUDIT) {
return new AuditEventLogger(config);
}
@ -104,6 +110,30 @@ public abstract class BaseEventListener
}
}
@Override
public final void auditLogged(AuditLogEvent auditLogEvent)
{
Logger log = HetuLogUtil.getLoggerByName(auditLogEvent.getUser(), auditLogEvent.getLevel(), HetuLogUtil.AuditType.valueOf(auditLogEvent.getType()));
ZonedDateTime zonedDateTime = LocalDateTime.now().atZone(ZoneId.systemDefault());
String logRecord = zonedDateTime + " " + auditLogEvent;
switch (auditLogEvent.getLevel()) {
case "INFO":
log.info(logRecord);
break;
case "ERROR":
log.severe(logRecord);
break;
case "WARN":
log.warning(logRecord);
break;
case "DEBUG":
log.fine(logRecord);
break;
default:
break;
}
}
/**
* call back for query created
*

View File

@ -17,9 +17,11 @@ package io.hetu.core.eventlistener.listeners;
import io.airlift.log.Logger;
import io.hetu.core.eventlistener.HetuEventListenerConfig;
import io.hetu.core.eventlistener.util.EventUtility;
import io.hetu.core.eventlistener.util.HetuLogUtil;
import io.hetu.core.eventlistener.util.ListenerErrorCode;
import io.prestosql.spi.PrestoException;
import io.prestosql.spi.eventlistener.QueryCompletedEvent;
import io.prestosql.spi.eventlistener.QueryContext;
import io.prestosql.spi.eventlistener.QueryCreatedEvent;
import io.prestosql.spi.eventlistener.SplitCompletedEvent;
@ -65,12 +67,18 @@ class QueryEventLogger
@Override
protected void onQueryCreatedEvent(QueryCreatedEvent queryCreatedEvent)
{
QueryContext queryContext = queryCreatedEvent.getContext();
java.util.logging.Logger log = HetuLogUtil.getLoggerByName(queryContext.getUser(), "INFO", HetuLogUtil.AuditType.Sql);
log.info(EventUtility.toString(queryCreatedEvent));
logger.info(EventUtility.toString(queryCreatedEvent));
}
@Override
protected void onQueryCompletedEvent(QueryCompletedEvent queryCompletedEvent)
{
QueryContext queryContext = queryCompletedEvent.getContext();
java.util.logging.Logger log = HetuLogUtil.getLoggerByName(queryContext.getUser(), "INFO", HetuLogUtil.AuditType.Sql);
log.info(EventUtility.toString(queryCompletedEvent));
logger.info(EventUtility.toString(queryCompletedEvent));
}

View File

@ -0,0 +1,129 @@
/*
* Copyright (C) 2018-2020. Huawei Technologies Co., Ltd. All rights reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.hetu.core.eventlistener.util;
import io.hetu.core.eventlistener.HetuEventListenerConfig;
import io.prestosql.spi.PrestoException;
import java.io.File;
import java.io.IOException;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.concurrent.ConcurrentHashMap;
import java.util.logging.FileHandler;
import java.util.logging.Formatter;
import java.util.logging.Handler;
import java.util.logging.LogRecord;
import java.util.logging.Logger;
public class HetuLogUtil
{
private static String logConversionPattern;
private static String logOutput;
private static ConcurrentHashMap<String, Logger> loggerConcurrentHashMap;
private HetuLogUtil(){}
public static void create(HetuEventListenerConfig config)
{
logConversionPattern = config.getAuditLogConversionPattern();
logOutput = config.getAuditLogFile();
loggerConcurrentHashMap = new ConcurrentHashMap<>();
createFile(config.getAuditLogFile());
}
private static String getCurrentDate(String logConversionPattern)
{
String hetuLogConversionPattern = logConversionPattern;
if (hetuLogConversionPattern == null) {
hetuLogConversionPattern = "yyyy-MM-dd.HH";
}
LocalDateTime now = LocalDateTime.now();
DateTimeFormatter format = DateTimeFormatter.ofPattern(hetuLogConversionPattern);
return now.format(format);
}
public static Logger getLoggerByName(String username, String level, AuditType type)
{
String fileName = logOutput + "/" + type + "/" + level + "#" + username + "#" + getCurrentDate(logConversionPattern) + ".log";
java.util.logging.Logger logger = java.util.logging.Logger.getLogger(fileName);
if (logOutput == null) {
return logger;
}
try {
if (loggerConcurrentHashMap.containsKey(fileName)) {
return loggerConcurrentHashMap.get(fileName);
}
synchronized (HetuLogUtil.class) {
if (loggerConcurrentHashMap.containsKey(fileName)) {
return loggerConcurrentHashMap.get(fileName);
}
FileHandler fileHandler = new FileHandler(fileName, 0, 1, true);
fileHandler.setFormatter(new Formatter(){
@Override
public String format(LogRecord logRecord)
{
return logRecord.getMessage() + "\n";
}
});
logger.addHandler(fileHandler);
logger.setUseParentHandlers(false);
loggerConcurrentHashMap.put(fileName, logger);
clearUnusedLogger();
}
return loggerConcurrentHashMap.get(fileName);
}
catch (IOException ex) {
throw new PrestoException(ListenerErrorCode.LOCAL_FILE_FILESYSTEM_ERROR,
"failed to create logger writing to " + fileName, ex);
}
}
private static void clearUnusedLogger()
{
String currentTime = getCurrentDate(logConversionPattern);
for (String key : loggerConcurrentHashMap.keySet()) {
String loggerTime = key.substring(key.lastIndexOf("#") + 1, key.length() - 4);
if (loggerTime.compareTo(currentTime) < 0) {
Logger deadLogger = loggerConcurrentHashMap.get(key);
for (Handler h : deadLogger.getHandlers()) {
h.close();
}
loggerConcurrentHashMap.remove(key);
}
}
}
private static void createFile(String auditLogFile)
{
if (auditLogFile == null) {
return;
}
File logFileSql = new File(auditLogFile + "/Sql");
File logFileWebUi = new File(auditLogFile + "/WebUi");
File logFileCluster = new File(auditLogFile + "/Cluster");
logFileSql.mkdir();
logFileWebUi.mkdir();
logFileCluster.mkdir();
}
public enum AuditType{
WebUi,
Sql,
Cluster
}
}

View File

@ -38,7 +38,9 @@ public class TestHetuEventListenerConfig
.setLogFileCount(1)
.setAuditFile(null)
.setAuditFileLimit(0)
.setAuditFileCount(1));
.setAuditFileCount(1)
.setAuditLogFile(null)
.setAuditLogConversionPattern(null));
}
@Test
@ -55,6 +57,8 @@ public class TestHetuEventListenerConfig
.put("hetu.event.listener.audit.filecount", "10")
.put("hetu.event.listener.audit.limit", "1024")
.put("hetu.event.listener.audit.file", "/var/hetu-audit.log")
.put("hetu.auditlog.logoutput", "/var/log")
.put("hetu.auditlog.logconversionpattern", "yyyy-MM-dd.HH")
.build();
HetuEventListenerConfig expected = new HetuEventListenerConfig().setType(BaseEventListener.Type.LOGGER)
@ -66,7 +70,9 @@ public class TestHetuEventListenerConfig
.setLogFileLimit(1024)
.setAuditFile("/var/hetu-audit.log")
.setAuditFileLimit(1024)
.setAuditFileCount(10);
.setAuditFileCount(10)
.setAuditLogFile("/var/log")
.setAuditLogConversionPattern("yyyy-MM-dd.HH");
assertFullMapping(properties, expected);
}
@ -85,6 +91,8 @@ public class TestHetuEventListenerConfig
.put("hetu.event.listener.audit.filecount", "10")
.put("hetu.event.listener.audit.limit", "1024")
.put("hetu.event.listener.audit.file", "/var/hetu-audit.log")
.put("hetu.auditlog.logoutput", "/var/log")
.put("hetu.auditlog.logconversionpattern", "yyyy-MM-dd.HH")
.build();
HetuEventListenerConfig expected = new HetuEventListenerConfig().setType(BaseEventListener.Type.AUDIT)
@ -96,7 +104,9 @@ public class TestHetuEventListenerConfig
.setLogFileLimit(1024)
.setAuditFile("/var/hetu-audit.log")
.setAuditFileLimit(1024)
.setAuditFileCount(10);
.setAuditFileCount(10)
.setAuditLogFile("/var/log")
.setAuditLogConversionPattern("yyyy-MM-dd.HH");
assertFullMapping(properties, expected);
}

View File

@ -15,11 +15,15 @@
package io.hetu.core.metastore;
import io.airlift.log.Logger;
import io.prestosql.spi.favorite.FavoriteEntity;
import io.prestosql.spi.favorite.FavoriteResult;
import io.prestosql.spi.metastore.HetuCache;
import io.prestosql.spi.metastore.HetuMetastore;
import io.prestosql.spi.metastore.model.CatalogEntity;
import io.prestosql.spi.metastore.model.DatabaseEntity;
import io.prestosql.spi.metastore.model.TableEntity;
import io.prestosql.spi.queryhistory.QueryHistoryEntity;
import io.prestosql.spi.queryhistory.QueryHistoryResult;
import java.util.List;
import java.util.Optional;
@ -306,6 +310,60 @@ public class HetuMetastoreCache
}
}
@Override
public void insertQueryHistory(QueryHistoryEntity queryHistoryEntity, String jsonString)
{
delegate.insertQueryHistory(queryHistoryEntity, jsonString);
}
@Override
public void deleteQueryHistoryBatch()
{
delegate.deleteQueryHistoryBatch();
}
@Override
public long getAllQueryHistoryNum()
{
return delegate.getAllQueryHistoryNum();
}
@Override
public String getQueryDetail(String queryId)
{
return delegate.getQueryDetail(queryId);
}
@Override
public QueryHistoryResult getQueryHistory(int startNum, int pageSize,
String user, String startTime, String endTime,
String queryId, String query, String resourceGroup,
String resource, List<String> state, List<String> failed,
String sort, String sortOrder)
{
return delegate.getQueryHistory(startNum, pageSize,
user, startTime, endTime, queryId, query,
resourceGroup, resource, state, failed, sort, sortOrder);
}
@Override
public void insertFavorite(FavoriteEntity favoriteEntity)
{
delegate.insertFavorite(favoriteEntity);
}
@Override
public void deleteFavorite(FavoriteEntity favoriteEntity)
{
delegate.deleteFavorite(favoriteEntity);
}
@Override
public FavoriteResult getFavorite(int startNum, int pageSize, String user)
{
return delegate.getFavorite(startNum, pageSize, user);
}
@Override
public Optional<TableEntity> getTable(String catalogName, String databaseName, String tableName)
{

View File

@ -14,10 +14,14 @@
*/
package io.hetu.core.metastore;
import io.prestosql.spi.favorite.FavoriteEntity;
import io.prestosql.spi.favorite.FavoriteResult;
import io.prestosql.spi.metastore.HetuMetastore;
import io.prestosql.spi.metastore.model.CatalogEntity;
import io.prestosql.spi.metastore.model.DatabaseEntity;
import io.prestosql.spi.metastore.model.TableEntity;
import io.prestosql.spi.queryhistory.QueryHistoryEntity;
import io.prestosql.spi.queryhistory.QueryHistoryResult;
import javax.inject.Inject;
@ -160,4 +164,58 @@ public class HetuMetastoreNone
{
delegate.alterTableParameter(catalogName, databaseName, tableName, key, value);
}
@Override
public void insertQueryHistory(QueryHistoryEntity queryHistoryEntity, String jsonString)
{
delegate.insertQueryHistory(queryHistoryEntity, jsonString);
}
@Override
public void deleteQueryHistoryBatch()
{
delegate.deleteQueryHistoryBatch();
}
@Override
public long getAllQueryHistoryNum()
{
return delegate.getAllQueryHistoryNum();
}
@Override
public String getQueryDetail(String queryId)
{
return delegate.getQueryDetail(queryId);
}
@Override
public QueryHistoryResult getQueryHistory(int startNum, int pageSize,
String user, String startTime, String endTime,
String queryId, String query, String resourceGroup,
String resource, List<String> state, List<String> failed,
String sort, String sortOrder)
{
return delegate.getQueryHistory(startNum, pageSize,
user, startTime, endTime, queryId, query,
resourceGroup, resource, state, failed, sort, sortOrder);
}
@Override
public void insertFavorite(FavoriteEntity favoriteEntity)
{
delegate.insertFavorite(favoriteEntity);
}
@Override
public void deleteFavorite(FavoriteEntity favoriteEntity)
{
delegate.deleteFavorite(favoriteEntity);
}
@Override
public FavoriteResult getFavorite(int startNum, int pageSize, String user)
{
return delegate.getFavorite(startNum, pageSize, user);
}
}

View File

@ -30,12 +30,16 @@ import io.prestosql.spi.connector.SchemaNotFoundException;
import io.prestosql.spi.connector.SchemaTableName;
import io.prestosql.spi.connector.TableAlreadyExistsException;
import io.prestosql.spi.connector.TableNotFoundException;
import io.prestosql.spi.favorite.FavoriteEntity;
import io.prestosql.spi.favorite.FavoriteResult;
import io.prestosql.spi.filesystem.FileBasedLock;
import io.prestosql.spi.filesystem.HetuFileSystemClient;
import io.prestosql.spi.metastore.HetuMetastore;
import io.prestosql.spi.metastore.model.CatalogEntity;
import io.prestosql.spi.metastore.model.DatabaseEntity;
import io.prestosql.spi.metastore.model.TableEntity;
import io.prestosql.spi.queryhistory.QueryHistoryEntity;
import io.prestosql.spi.queryhistory.QueryHistoryResult;
import javax.inject.Inject;
@ -796,4 +800,52 @@ public class HetuFsMetastore
}
});
}
@Override
public void insertQueryHistory(QueryHistoryEntity queryHistoryEntity, String jsonString)
{
}
@Override
public void deleteQueryHistoryBatch()
{
}
@Override
public long getAllQueryHistoryNum()
{
return 0;
}
@Override
public String getQueryDetail(String queryId)
{
return null;
}
@Override
public QueryHistoryResult getQueryHistory(int startNum, int pageSize,
String user, String startTime, String endTime,
String queryId, String query, String resourceGroup,
String resource, List<String> state, List<String> failed,
String sort, String sortOrder)
{
return null;
}
@Override
public void insertFavorite(FavoriteEntity favoriteEntity)
{
}
@Override
public void deleteFavorite(FavoriteEntity favoriteEntity)
{
}
@Override
public FavoriteResult getFavorite(int startNum, int pageSize, String user)
{
return null;
}
}

View File

@ -0,0 +1,32 @@
/*
* Copyright (C) 2018-2021. Huawei Technologies Co., Ltd. All rights reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.hetu.core.metastore.jdbc;
import io.prestosql.spi.favorite.FavoriteEntity;
import org.jdbi.v3.core.result.LinkedHashMapRowReducer;
import org.jdbi.v3.core.result.RowView;
import java.util.Map;
public class FavoriteEntityReducer
implements LinkedHashMapRowReducer<Long, FavoriteEntity>
{
@Override
public void accumulate(Map<Long, FavoriteEntity> map, RowView rowView)
{
map.computeIfAbsent(rowView.getColumn("id", Long.class),
id -> rowView.getRow(FavoriteEntity.class));
}
}

View File

@ -20,11 +20,15 @@ import io.prestosql.spi.connector.CatalogNotFoundException;
import io.prestosql.spi.connector.SchemaNotFoundException;
import io.prestosql.spi.connector.SchemaTableName;
import io.prestosql.spi.connector.TableNotFoundException;
import io.prestosql.spi.favorite.FavoriteEntity;
import io.prestosql.spi.favorite.FavoriteResult;
import io.prestosql.spi.metastore.HetuMetastore;
import io.prestosql.spi.metastore.model.CatalogEntity;
import io.prestosql.spi.metastore.model.ColumnEntity;
import io.prestosql.spi.metastore.model.DatabaseEntity;
import io.prestosql.spi.metastore.model.TableEntity;
import io.prestosql.spi.queryhistory.QueryHistoryEntity;
import io.prestosql.spi.queryhistory.QueryHistoryResult;
import org.jdbi.v3.core.Jdbi;
import org.jdbi.v3.sqlobject.SqlObjectPlugin;
@ -432,6 +436,92 @@ public class JdbcHetuMetastore
});
}
@Override
public void insertQueryHistory(QueryHistoryEntity queryHistoryEntity, String jsonString)
{
runTransactionWithLock(jdbi, handle -> {
JdbcMetadataDao transactionDao = handle.attach(JdbcMetadataDao.class);
transactionDao.insertQueryHistory(queryHistoryEntity, jsonString);
});
}
@Override
public void deleteQueryHistoryBatch()
{
runTransactionWithLock(jdbi, handle -> {
JdbcMetadataDao transactionDao = handle.attach(JdbcMetadataDao.class);
transactionDao.deleteQueryHistoryBatch();
});
}
@Override
public long getAllQueryHistoryNum()
{
long[] ans = new long[1];
runTransactionWithLock(jdbi, handle -> {
JdbcMetadataDao transactionDao = handle.attach(JdbcMetadataDao.class);
ans[0] = transactionDao.getAllQueryHistoryNum();
});
return ans[0];
}
@Override
public String getQueryDetail(String queryId)
{
String[] jsonString = new String[1];
runTransactionWithLock(jdbi, handle -> {
JdbcMetadataDao transactionDao = handle.attach(JdbcMetadataDao.class);
jsonString[0] = transactionDao.getQueryDetail(queryId);
});
return jsonString[0];
}
@Override
public QueryHistoryResult getQueryHistory(int startNum, int pageSize,
String user, String startTime, String endTime,
String queryId, String query, String resourceGroup,
String resource, List<String> state, List<String> failed,
String sort, String sortOrder)
{
QueryHistoryResult queryHistoryResult = new QueryHistoryResult();
runTransactionWithLock(jdbi, handle -> {
JdbcMetadataDao transactionDao = handle.attach(JdbcMetadataDao.class);
queryHistoryResult.setTotal(transactionDao.getQueryHistoryCount(user, startTime, endTime, queryId, query, resourceGroup, resource, state, failed));
queryHistoryResult.setQueries(transactionDao.getQueryHistory(startNum, pageSize, user, startTime, endTime, queryId, query, resourceGroup, resource, state, failed, sort, sortOrder));
});
return queryHistoryResult;
}
@Override
public void insertFavorite(FavoriteEntity favoriteEntity)
{
runTransactionWithLock(jdbi, handle -> {
JdbcMetadataDao transactionDao = handle.attach(JdbcMetadataDao.class);
transactionDao.insertFavorite(favoriteEntity);
});
}
@Override
public void deleteFavorite(FavoriteEntity favoriteEntity)
{
runTransactionWithLock(jdbi, handle -> {
JdbcMetadataDao transactionDao = handle.attach(JdbcMetadataDao.class);
transactionDao.deleteFavorite(favoriteEntity);
});
}
@Override
public FavoriteResult getFavorite(int startNum, int pageSize, String user)
{
FavoriteResult favoriteResult = new FavoriteResult();
runTransactionWithLock(jdbi, handle -> {
JdbcMetadataDao transactionDao = handle.attach(JdbcMetadataDao.class);
favoriteResult.setQueries(transactionDao.getFavorite(startNum, pageSize, user));
favoriteResult.setTotal(transactionDao.getFavoriteCount(user));
});
return favoriteResult;
}
private void getTableColumns(List<Map.Entry<Long, TableEntity>> tableEntries,
ImmutableList.Builder<TableEntity> tables, JdbcMetadataDao transactionDao)
{

View File

@ -14,13 +14,17 @@
*/
package io.hetu.core.metastore.jdbc;
import io.prestosql.spi.favorite.FavoriteEntity;
import io.prestosql.spi.metastore.model.CatalogEntity;
import io.prestosql.spi.metastore.model.ColumnEntity;
import io.prestosql.spi.metastore.model.DatabaseEntity;
import io.prestosql.spi.metastore.model.TableEntity;
import io.prestosql.spi.queryhistory.QueryHistoryEntity;
import org.jdbi.v3.sqlobject.config.RegisterBeanMapper;
import org.jdbi.v3.sqlobject.customizer.Bind;
import org.jdbi.v3.sqlobject.customizer.BindBean;
import org.jdbi.v3.sqlobject.customizer.BindList;
import org.jdbi.v3.sqlobject.customizer.Define;
import org.jdbi.v3.sqlobject.statement.GetGeneratedKeys;
import org.jdbi.v3.sqlobject.statement.SqlBatch;
import org.jdbi.v3.sqlobject.statement.SqlQuery;
@ -41,6 +45,8 @@ import java.util.Optional;
@RegisterBeanMapper(value = TableEntity.class, prefix = "t")
@RegisterBeanMapper(value = ColumnEntity.class, prefix = "c")
@RegisterBeanMapper(value = PropertyEntity.class, prefix = "p")
@RegisterBeanMapper(value = QueryHistoryEntity.class)
@RegisterBeanMapper(value = FavoriteEntity.class)
public interface JdbcMetadataDao
{
@ -467,4 +473,142 @@ public interface JdbcMetadataDao
*/
@SqlQuery("SELECT id FROM hetu_tab_lock WHERE resource=1")
Long getLockId();
/**
* insert queryHistoryEntity
*
* @param queryHistoryEntity queryHistory Entity
*/
@SqlUpdate("INSERT IGNORE INTO hetu_query_history \n" +
"(user,source,queryId,resource,query,state,failed,\n" +
"createTime,elapsedTime,cpuTime,executionTime,\n" +
"catalog,schemata,currentMemory,cumulativeUserMemory,\n" +
"jsonString,completedDrivers,runningDrivers,\n" +
"queuedDrivers,totalCpuTime,totalMemoryReservation,\n" +
"peakTotalMemoryReservation) \n" +
"VALUES (:user,:source,:queryId,:resource,:query,:state,:failed,\n" +
":createTime,:elapsedTime,:cpuTime,:executionTime,\n" +
":catalog,:schemata,:currentMemory,:cumulativeUserMemory,\n" +
":jsonString,:completedDrivers,:runningDrivers,\n" +
":queuedDrivers,:totalCpuTime,:totalMemoryReservation,\n" +
":peakTotalMemoryReservation)")
void insertQueryHistory(@BindBean QueryHistoryEntity queryHistoryEntity, @Bind("jsonString") String jsonString);
/**
* delete the oldest 100 queryHistoryEntities
*
*/
@SqlUpdate("DELETE FROM hetu_query_history \n" +
"WHERE 1=1 \n" +
"ORDER BY id \n" +
"LIMIT 100")
void deleteQueryHistoryBatch();
/**
* get QueryDetail
*
* @param queryId
* @return QueryDetail
*/
@SqlQuery("SELECT jsonString FROM hetu_query_history WHERE queryId = :queryId")
String getQueryDetail(@Bind("queryId") String queryId);
/**
* get all QueryHistory
*
* @param
* @return QueryHistoryEntities
*/
@SqlQuery("SELECT id,user,source,queryId,resource,query,state,failed, " +
"createTime,elapsedTime,cpuTime,executionTime," +
"catalog,schemata,currentMemory,cumulativeUserMemory," +
"completedDrivers,runningDrivers,queuedDrivers,totalCpuTime," +
"totalMemoryReservation,peakTotalMemoryReservation " +
"FROM hetu_query_history WHERE\n" +
"(user like concat(\'%\',:user,\'%\')) AND" +
"(:startTime='' or createTime >= :startTime) AND" +
"(:endTime='' or createTime <= :endTime) AND" +
"(queryId like concat(\'%\',:queryId)) AND" +
"(query like concat(\'%\',:query,\'%\')) AND" +
"(:resourceGroup='' or resource = :resourceGroup) AND" +
"(:resource='' or source = :resource) AND" +
"(state in (<state>) ) AND" +
"(failed in (<failed>) )" +
"ORDER BY <sort> <sortOrder> LIMIT :startNum , :pageSize")
@UseRowReducer(QueryHistoryEntityReducer.class)
List<QueryHistoryEntity> getQueryHistory(@Bind("startNum") int startNum, @Bind("pageSize") int pageSize,
@Bind("user") String user, @Bind("startTime") String startTime, @Bind("endTime") String endTime,
@Bind("queryId") String queryId, @Bind("query") String query, @Bind("resourceGroup") String resourceGroup,
@Bind("resource") String resource, @BindList("state") List<String> state, @BindList("failed") List<String> failed,
@Define("sort") String sort, @Define("sortOrder") String sortOrder);
/**
* get QueryHistory counts
*
* @param
* @return counts
*/
@SqlQuery("SELECT count(*) FROM hetu_query_history WHERE \n" +
"(user like concat(\'%\',:user,\'%\')) AND" +
"(:startTime='' or createTime >= :startTime) AND" +
"(:endTime='' or createTime <= :endTime) AND" +
"(queryId like concat(\'%\',:queryId)) AND" +
"(query like concat(\'%\',:query,\'%\')) AND" +
"(:resourceGroup='' or resource = :resourceGroup) AND" +
"(:resource='' or source = :resource) AND" +
"(state in (<state>)) AND" +
"(failed in (<failed>))")
long getQueryHistoryCount(@Bind("user") String user, @Bind("startTime") String startTime, @Bind("endTime") String endTime,
@Bind("queryId") String queryId, @Bind("query") String query, @Bind("resourceGroup") String resourceGroup,
@Bind("resource") String resource, @BindList("state") List<String> state, @BindList("failed") List<String> failed);
/**
* get all QueryHistory counts
*
* @param
* @return counts
*/
@SqlQuery("SELECT count(1) FROM hetu_query_history")
long getAllQueryHistoryNum();
/**
* insert favoriteEntity
*
* @param favoriteEntity favorite Entity
*/
@SqlUpdate("INSERT IGNORE INTO hetu_favorite \n" +
"(creationTime, user, query, catalog, schemata)\n" +
"VALUES(DEFAULT, :user, :query, :catalog, :schemata)")
void insertFavorite(@BindBean FavoriteEntity favoriteEntity);
/**
* delete favoriteEntity
*
* @param favoriteEntity
*/
@SqlUpdate("DELETE FROM hetu_favorite WHERE \n" +
" user = :user and query = :query and catalog = :catalog and schemata = :schemata")
void deleteFavorite(@BindBean FavoriteEntity favoriteEntity);
/**
* get all favoriteEntity
*
* @param user
* @return favoriteEntities
*/
@SqlQuery("SELECT id, user, query, catalog, schemata \n" +
"FROM hetu_favorite WHERE user = :user \n" +
"ORDER BY creationTime desc \n" +
"LIMIT :startNum, :pageSize")
@UseRowReducer(FavoriteEntityReducer.class)
List<FavoriteEntity> getFavorite(@Bind("startNum")int startNum, @Bind("pageSize")int pageSize, @Bind("user")String user);
/**
* get all favoriteEntity
*
* @param user
* @return favoriteEntities
*/
@SqlQuery("SELECT count(*) FROM hetu_favorite WHERE user = :user")
long getFavoriteCount(@Bind("user")String user);
}

View File

@ -178,5 +178,9 @@ public class JdbcMetadataUtil
tableDao.createTableColumnParameters();
// hetu_tab_lock table
tableDao.createTableLock();
//hetu_query_history table
tableDao.createTableQueryHistory();
//hetu_favorite table
tableDao.createTableFavorite();
}
}

View File

@ -144,4 +144,51 @@ public interface MetadataTableDao
+ "UNIQUE (resource)\n"
+ ")")
void createTableLock();
/**
* create table of query_history
*/
@SqlUpdate("CREATE TABLE IF NOT EXISTS hetu_query_history (\n" +
" id BIGINT NOT NULL AUTO_INCREMENT,\n" +
" user varchar(20) NOT NULL,\n" +
" source varchar(30) NOT NULL,\n" +
" queryId varchar(50) NOT NULL,\n" +
" resource varchar(30) NOT NULL,\n" +
" query varchar(1000) NOT NULL,\n" +
" state varchar(30) NOT NULL,\n" +
" failed varchar(30) DEFAULT NULL,\n" +
" createTime varchar(30) DEFAULT NULL,\n" +
" elapsedTime varchar(30) NOT NULL,\n" +
" cpuTime varchar(30) NOT NULL,\n" +
" executionTime varchar(30) NOT NULL,\n" +
" catalog varchar(30) NOT NULL,\n" +
" schemata varchar(30) NOT NULL,\n" +
" currentMemory varchar(30) NOT NULL,\n" +
" cumulativeUserMemory double NOT NULL,\n" +
" jsonString mediumtext NOT NULL,\n" +
" completedDrivers int(11) NOT NULL,\n" +
" runningDrivers int(11) NOT NULL,\n" +
" queuedDrivers int(11) NOT NULL,\n" +
" totalCpuTime varchar(50) NOT NULL,\n" +
" totalMemoryReservation varchar(50) NOT NULL,\n" +
" peakTotalMemoryReservation varchar(50) NOT NULL,\n" +
" PRIMARY KEY (id),\n" +
" UNIQUE KEY query_history_queryId_uindex (queryId)\n" +
")")
void createTableQueryHistory();
/**
* create table of hatu_favorite
*/
@SqlUpdate("CREATE TABLE IF NOT EXISTS hetu_favorite (\n" +
" creationTime datetime DEFAULT CURRENT_TIMESTAMP,\n" +
" user varchar(20) NOT NULL,\n" +
" query varchar(600) NOT NULL,\n" +
" catalog varchar(30) NOT NULL,\n" +
" schemata varchar(30) NOT NULL,\n" +
" id BIGINT NOT NULL AUTO_INCREMENT,\n" +
" PRIMARY KEY (id),\n" +
" UNIQUE KEY favorite_uni (user,query,catalog,schemata)\n" +
")")
void createTableFavorite();
}

View File

@ -0,0 +1,32 @@
/*
* Copyright (C) 2018-2021. Huawei Technologies Co., Ltd. All rights reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.hetu.core.metastore.jdbc;
import io.prestosql.spi.queryhistory.QueryHistoryEntity;
import org.jdbi.v3.core.result.LinkedHashMapRowReducer;
import org.jdbi.v3.core.result.RowView;
import java.util.Map;
public class QueryHistoryEntityReducer
implements LinkedHashMapRowReducer<Long, QueryHistoryEntity>
{
@Override
public void accumulate(Map<Long, QueryHistoryEntity> map, RowView rowView)
{
map.computeIfAbsent(rowView.getColumn("id", Long.class),
id -> rowView.getRow(QueryHistoryEntity.class));
}
}

76
hetu-server/package-lock.json generated Normal file
View File

@ -0,0 +1,76 @@
{
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"@babel/runtime": {
"version": "7.16.3",
"resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.16.3.tgz",
"integrity": "sha512-WBwekcqacdY2e9AF/Q7WLFUWmdJGJTkbjqTjoMDgXkVZ3ZRUvOPsLb5KdwISoQVsbP+DQzVZW4Zhci0DvpbNTQ==",
"requires": {
"regenerator-runtime": "^0.13.4"
}
},
"@juggle/resize-observer": {
"version": "3.3.1",
"resolved": "https://registry.npmjs.org/@juggle/resize-observer/-/resize-observer-3.3.1.tgz",
"integrity": "sha512-zMM9Ds+SawiUkakS7y94Ymqx+S0ORzpG3frZirN3l+UlXUmSUR7hF4wxCVqW+ei94JzV5kt0uXBcoOEAuiydrw=="
},
"@rsuite/icon-font": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@rsuite/icon-font/-/icon-font-4.0.0.tgz",
"integrity": "sha512-rZTgpTH3H3HLczCA2rnkWfoMKm0ZXoRzsrkVujfP/FfslnKUMvO6w56pa8pCvhWGpNEPUsLS2ULnFGpTEcup/Q=="
},
"@rsuite/icons": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/@rsuite/icons/-/icons-1.0.2.tgz",
"integrity": "sha512-Y7vJNDQpJnFlyYSUXQ2iQ9Meg7+ZKcrIenhpYDdM3c7vYDE/L7pml+hrK28jk6QfV/QkVv5B504D+l7aM6AAJQ==",
"requires": {
"@rsuite/icon-font": "^4.0.0",
"classnames": "^2.2.5",
"insert-css": "^2.0.0",
"lodash": "^4.17.20"
}
},
"classnames": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/classnames/-/classnames-2.3.1.tgz",
"integrity": "sha512-OlQdbZ7gLfGarSqxesMesDa5uz7KFbID8Kpq/SxIoNGDqY8lSYs0D+hhtBXhcdB3rcbXArFr7vlHheLk1voeNA=="
},
"dom-lib": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/dom-lib/-/dom-lib-3.0.0.tgz",
"integrity": "sha512-3US7OyceYve90NSU4MhuE0lK4a7aB1hbb14h45v3CPV/ZolLoEOeuPrdS/A0L8kLnwgVCDiKwlPy/Ajq0u9vfA==",
"requires": {
"@babel/runtime": "^7.16.0"
}
},
"insert-css": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/insert-css/-/insert-css-2.0.0.tgz",
"integrity": "sha1-610Ql7dUL0x56jBg067gfQU4gPQ="
},
"lodash": {
"version": "4.17.21",
"resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz",
"integrity": "sha512-v2kDEe57lecTulaDIuNTPy3Ry4gLGJ6Z1O3vE1krgXZNrsQ+LFTGHVxVjcXPs17LhbZVGedAJv8XZ1tvj5FvSg=="
},
"regenerator-runtime": {
"version": "0.13.9",
"resolved": "https://registry.npmjs.org/regenerator-runtime/-/regenerator-runtime-0.13.9.tgz",
"integrity": "sha512-p3VT+cOEgxFsRRA9X4lkI1E+k2/CtnKtU4gcxyaCUreilL/vqI6CdZ3wxVUx3UOUg+gnUOQQcRI7BmSI656MYA=="
},
"rsuite-table": {
"version": "5.2.0",
"resolved": "https://registry.npmjs.org/rsuite-table/-/rsuite-table-5.2.0.tgz",
"integrity": "sha512-MiJtfZYSMXqaUdEFQ0Q5A5mNjiB0upZumoqMyd0WGsJDC6Ff8tbnMEDBbcMcQ6GcL2MSJoxi1vz3kIo/mZ1Wew==",
"requires": {
"@babel/runtime": "^7.12.5",
"@juggle/resize-observer": "^3.3.1",
"@rsuite/icons": "^1.0.0",
"classnames": "^2.2.5",
"dom-lib": "^3.0.0",
"lodash": "^4.17.21"
}
}
}
}

View File

@ -26,10 +26,14 @@ import io.hetu.core.cube.startree.tree.StarTreeMetadataBuilder;
import io.hetu.core.spi.cube.CubeMetadata;
import io.hetu.core.spi.cube.CubeStatus;
import io.hetu.core.spi.cube.io.CubeMetaStore;
import io.prestosql.spi.favorite.FavoriteEntity;
import io.prestosql.spi.favorite.FavoriteResult;
import io.prestosql.spi.metastore.HetuMetastore;
import io.prestosql.spi.metastore.model.CatalogEntity;
import io.prestosql.spi.metastore.model.DatabaseEntity;
import io.prestosql.spi.metastore.model.TableEntity;
import io.prestosql.spi.queryhistory.QueryHistoryEntity;
import io.prestosql.spi.queryhistory.QueryHistoryResult;
import io.prestosql.sql.analyzer.FeaturesConfig;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
@ -305,5 +309,53 @@ public class TestStarTreeMetaStore
@Override
public void alterTableParameter(String catalogName, String databaseName, String tableName, String key, String value) {}
@Override
public void insertQueryHistory(QueryHistoryEntity queryHistoryEntity, String jsonString)
{
}
@Override
public void deleteQueryHistoryBatch()
{
}
@Override
public long getAllQueryHistoryNum()
{
return 0;
}
@Override
public String getQueryDetail(String queryId)
{
return null;
}
@Override
public QueryHistoryResult getQueryHistory(int startNum, int pageSize,
String user, String startTime, String endTime,
String queryId, String query, String resourceGroup,
String resource, List<String> state, List<String> failed,
String sort, String sortOrder)
{
return null;
}
@Override
public void insertFavorite(FavoriteEntity favoriteEntity)
{
}
@Override
public void deleteFavorite(FavoriteEntity favoriteEntity)
{
}
@Override
public FavoriteResult getFavorite(int startNum, int pageSize, String user)
{
return null;
}
}
}

13
pom.xml
View File

@ -1023,7 +1023,7 @@
<artifactId>hadoop-distcp</artifactId>
<version>2.10.1</version>
</dependency>
<dependency>
<groupId>org.apache.hudi</groupId>
<artifactId>hudi-hadoop-mr</artifactId>
@ -1572,6 +1572,11 @@
<artifactId>calcite-core</artifactId>
<version>1.18.0</version>
</dependency>
<dependency>
<groupId>org.glassfish</groupId>
<artifactId>javax.el</artifactId>
<version>3.0.1-b08</version>
</dependency>
</dependencies>
</dependencyManagement>
@ -1850,7 +1855,7 @@
</profile>
</profiles>
<scm>
<tag>1.4.0</tag>
</scm>
<scm>
<tag>1.4.0</tag>
</scm>
</project>

View File

@ -295,10 +295,6 @@ public class BaseJdbcConfig
return connectionUrl;
}
@Mandatory(name = "connection-url",
description = "the address of datasource",
defaultValue = "host1",
required = true)
@Config("connection-url")
public BaseJdbcConfig setConnectionUrl(String connectionUrl)
{

View File

@ -7,4 +7,6 @@ hetu.event.listener.type=AUDIT
#hetu.event.listener.logger.limit=0
#hetu.event.listener.audit.file=/var/log/hetu/hetu-audit.log
#hetu.event.listener.audit.filecount=1
#hetu.event.listener.audit.limit=100000
#hetu.event.listener.audit.limit=100000
hetu.auditlog.logoutput=/var/log/
hetu.auditlog.logconversionpattern=yyyy-MM-dd.HH

View File

@ -19,6 +19,7 @@ import com.google.common.io.ByteStreams;
import io.airlift.json.JsonCodec;
import io.airlift.log.Logger;
import io.hetu.core.common.util.SecurePathWhiteList;
import io.prestosql.catalog.showcatalog.ShowCatalogStore;
import io.prestosql.spi.PrestoException;
import io.prestosql.spi.filesystem.FileBasedLock;
import io.prestosql.spi.filesystem.HetuFileSystemClient;
@ -51,7 +52,6 @@ public abstract class AbstractCatalogStore
private static final Logger log = Logger.get(AbstractCatalogStore.class);
private static final JsonCodec<List<String>> LIST_CODEC = JsonCodec.listJsonCodec(String.class);
private static final String CATALOG_NAME_PROPERTY = "connector.name";
protected final String baseDirectory;
protected final HetuFileSystemClient fileSystemClient;
private final int maxFileSizeInBytes;
@ -235,6 +235,25 @@ public abstract class AbstractCatalogStore
return new CatalogInfo(catalogName, connectorName, null, createdTime, version, catalogProperties);
}
public Map<String, String> getCatalogProperties(String catalogName, ShowCatalogStore.CatalogConfigType state, String baseDirectory)
throws IOException
{
Properties properties = new Properties();
CatalogFilePath catalogPath = new CatalogFilePath(baseDirectory, catalogName);
Path path;
if (state == ShowCatalogStore.CatalogConfigType.DYNAMIC) {
path = catalogPath.getPropertiesPath();
}
else {
path = catalogPath.getStaticPath();
}
try (InputStream inputStream = fileSystemClient.newInputStream(path)) {
properties.load(inputStream);
}
Map<String, String> catalogProperties = new HashMap<>(fromProperties(properties));
return catalogProperties;
}
public CatalogFileInputStream getCatalogFiles(String catalogName)
throws IOException
{

View File

@ -31,6 +31,7 @@ public final class CatalogFilePath
private final Path propertiesPath;
private final Path metadataPath;
private final Path lockPath;
private final Path staticPath;
/*
* baseDirectory
@ -73,6 +74,7 @@ public final class CatalogFilePath
this.metadataPath = Paths.get(catalogDirPath.toString(), catalogName + ".metadata");
// lock files directory
this.lockPath = Paths.get(catalogBasePath);
this.staticPath = Paths.get(baseDirectory, catalogName + ".properties");
}
public static Path getCatalogBasePath(String baseDirectory)
@ -104,4 +106,9 @@ public final class CatalogFilePath
{
return lockPath;
}
public Path getStaticPath()
{
return staticPath;
}
}

View File

@ -193,7 +193,6 @@ public class CatalogResource
@Context HttpServletRequest servletRequest)
{
CatalogInfo catalogInfo = toCatalogInfo(catalogInfoJson);
try (CatalogFileInputStream configFiles = toCatalogFiles(catalogConfigFileBodyParts, globalConfigFilesBodyParts)) {
return service.createCatalog(catalogInfo,
configFiles,

View File

@ -15,7 +15,10 @@
package io.prestosql.catalog;
import io.prestosql.catalog.showcatalog.ShowCatalogStore;
import java.io.IOException;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.locks.Lock;
@ -86,5 +89,7 @@ public interface CatalogStore
*/
void releaseCatalogLock(String catalogName);
Map<String, String> getCatalogProperties(String catalogName, ShowCatalogStore.CatalogConfigType state, String baseDirectory) throws IOException;
void close();
}

View File

@ -27,6 +27,7 @@ import io.prestosql.filesystem.FileSystemClientManager;
import io.prestosql.metadata.CatalogManager;
import io.prestosql.metadata.InternalNode;
import io.prestosql.metadata.InternalNodeManager;
import io.prestosql.metadata.StaticCatalogStoreConfig;
import io.prestosql.spi.PrestoException;
import io.prestosql.spi.connector.CatalogName;
@ -76,6 +77,7 @@ public class DynamicCatalogStore
public DynamicCatalogStore(ConnectorManager connectorManager,
DataCenterConnectorManager dataCenterConnectorManager,
DynamicCatalogConfig dynamicCatalogConfig,
StaticCatalogStoreConfig staticCatalogConfig,
CatalogManager catalogManager,
InternalNodeManager nodeManager,
ServiceSelectorManager selectorManager,

View File

@ -0,0 +1,35 @@
/*
* Copyright (C) 2018-2020. Huawei Technologies Co., Ltd. All rights reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.prestosql.catalog.showcatalog;
import com.google.inject.Binder;
import com.google.inject.Scopes;
import io.airlift.configuration.AbstractConfigurationAwareModule;
import org.glassfish.jersey.media.multipart.MultiPartFeature;
import static io.airlift.jaxrs.JaxrsBinder.jaxrsBinder;
public class ShowCatalogModule
extends AbstractConfigurationAwareModule
{
@Override
protected void setup(Binder binder)
{
jaxrsBinder(binder).bind(ShowCatalogResource.class);
jaxrsBinder(binder).bind(MultiPartFeature.class);
binder.bind(ShowCatalogService.class).in(Scopes.SINGLETON);
}
}

View File

@ -0,0 +1,99 @@
/*
* Copyright (C) 2018-2021. Huawei Technologies Co., Ltd. All rights reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.prestosql.catalog.showcatalog;
import com.google.inject.Inject;
import io.prestosql.spi.security.GroupProvider;
import org.assertj.core.util.VisibleForTesting;
import javax.servlet.http.HttpServletRequest;
import javax.validation.constraints.NotNull;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import java.util.Map;
import static io.prestosql.catalog.DynamicCatalogService.badRequest;
import static java.util.Objects.requireNonNull;
import static javax.ws.rs.core.Response.Status.BAD_REQUEST;
@Path("/v1/showCatalog")
public class ShowCatalogResource
{
private static final int MAX_NAME_LENGTH = 255;
private static final String VALID_CATALOG_NAME_REGEX = "[\\p{Alnum}_]+";
private final ShowCatalogService service;
private final GroupProvider groupProvider;
@Inject
public ShowCatalogResource(ShowCatalogService service, GroupProvider groupProvider)
{
this.service = requireNonNull(service, "service is null");
this.groupProvider = groupProvider;
}
@VisibleForTesting
void checkCatalogName(String catalogName)
{
if (catalogName.length() > MAX_NAME_LENGTH) {
throw badRequest(BAD_REQUEST, "The length of catalog name is too long");
}
// check dc
int dotIndex = catalogName.indexOf(".");
if (dotIndex >= 0) {
if (dotIndex == 0 || dotIndex == catalogName.length() - 1) {
throw badRequest(BAD_REQUEST, "Invalid catalog name");
}
String dc = catalogName.substring(0, dotIndex);
String catalog = catalogName.substring(dotIndex + 1);
if (!dc.matches(VALID_CATALOG_NAME_REGEX) || !catalog.matches(VALID_CATALOG_NAME_REGEX)) {
throw badRequest(BAD_REQUEST, "Invalid catalog name");
}
return;
}
if (!catalogName.matches(VALID_CATALOG_NAME_REGEX)) {
throw badRequest(BAD_REQUEST, "Invalid catalog name");
}
}
@GET
@Path("/{catalogName}")
@Produces(MediaType.APPLICATION_JSON)
public Map<String, String> getCatalogpropertites(@NotNull @PathParam("catalogName") String catalogName,
@Context HttpServletRequest servletRequest)
{
checkCatalogName(catalogName);
Map<String, String> response;
try {
response = service.getCatalogpropertites(catalogName);
}
catch (WebApplicationException ex) {
throw ex;
}
catch (Throwable ex) {
throw badRequest(BAD_REQUEST, "show catalog failed. please check your request info.");
}
return response;
}
}

View File

@ -0,0 +1,53 @@
/*
* Copyright (C) 2018-2021. Huawei Technologies Co., Ltd. All rights reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.prestosql.catalog.showcatalog;
import com.google.inject.Inject;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.Response;
import java.io.IOException;
import java.util.Map;
import static java.util.Objects.requireNonNull;
import static javax.ws.rs.core.MediaType.TEXT_PLAIN_TYPE;
public class ShowCatalogService
{
private final ShowCatalogStore showCatalogStore;
@Inject
public ShowCatalogService(ShowCatalogStore showCatalogStore)
{
this.showCatalogStore = requireNonNull(showCatalogStore, "dynamicCatalogStore is null");
}
public static WebApplicationException badRequest(Response.Status status, String message)
{
throw new WebApplicationException(
Response.status(status)
.type(TEXT_PLAIN_TYPE)
.entity(message)
.build());
}
public Map<String, String> getCatalogpropertites(String catalogname)
throws IOException
{
return showCatalogStore.getCatalogProperties(catalogname);
}
}

View File

@ -0,0 +1,114 @@
/*
* Copyright (C) 2018-2021. Huawei Technologies Co., Ltd. All rights reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.prestosql.catalog.showcatalog;
import com.google.common.collect.ImmutableSet;
import com.google.inject.Inject;
import io.prestosql.catalog.CatalogStore;
import io.prestosql.catalog.DynamicCatalogConfig;
import io.prestosql.catalog.LocalCatalogStore;
import io.prestosql.catalog.ShareCatalogStore;
import io.prestosql.filesystem.FileSystemClientManager;
import io.prestosql.metadata.StaticCatalogStoreConfig;
import java.io.IOException;
import java.nio.file.Paths;
import java.util.Map;
import java.util.Properties;
import java.util.Set;
public class ShowCatalogStore
{
private final StaticCatalogStoreConfig staticCatalogConfig;
private final DynamicCatalogConfig dynamicCatalogConfig;
private CatalogStore localCatalogStore;
private CatalogStore shareCatalogStore;
@Inject
public ShowCatalogStore(DynamicCatalogConfig dynamicCatalogConfig,
StaticCatalogStoreConfig staticCatalogConfig)
{
this.dynamicCatalogConfig = dynamicCatalogConfig;
this.staticCatalogConfig = staticCatalogConfig;
}
public void loadCatalogStores(FileSystemClientManager fileSystemClientManager)
throws IOException
{
int maxCatalogFileSize = (int) dynamicCatalogConfig.getCatalogMaxFileSize().toBytes();
String localConfigurationDir = dynamicCatalogConfig.getCatalogConfigurationDir();
Properties properties = new Properties();
properties.put("fs.client.type", "local");
String shareConfigurationDir = dynamicCatalogConfig.getCatalogShareConfigurationDir();
String shareFileSystemProfile = dynamicCatalogConfig.getShareFileSystemProfile();
if (!dynamicCatalogConfig.isDynamicCatalogEnabled()) {
localConfigurationDir = localConfigurationDir == null ? staticCatalogConfig.getCatalogConfigurationDir().getCanonicalPath() : localConfigurationDir;
this.shareCatalogStore = new ShareCatalogStore(localConfigurationDir,
fileSystemClientManager.getFileSystemClient(properties, Paths.get(localConfigurationDir)),
maxCatalogFileSize);
}
else {
this.shareCatalogStore = new ShareCatalogStore(shareConfigurationDir,
fileSystemClientManager.getFileSystemClient(shareFileSystemProfile, Paths.get(shareConfigurationDir)),
maxCatalogFileSize);
}
this.localCatalogStore = new LocalCatalogStore(localConfigurationDir,
fileSystemClientManager.getFileSystemClient(properties, Paths.get(localConfigurationDir)),
maxCatalogFileSize);
}
private CatalogStore getCatalogStore(CatalogStoreType type)
{
if (type == CatalogStoreType.LOCAL) {
return localCatalogStore;
}
else {
return shareCatalogStore;
}
}
public synchronized Set<String> listCatalogNames(CatalogStoreType type)
throws IOException
{
return ImmutableSet.copyOf(getCatalogStore(type).listCatalogNames());
}
public Map<String, String> getCatalogProperties(String catalogName)
throws IOException
{
Set<String> localCatalogs = listCatalogNames(CatalogStoreType.LOCAL);
Set<String> shareCatalogs = listCatalogNames(CatalogStoreType.SHARE);
if (localCatalogs.contains(catalogName)) {
return localCatalogStore.getCatalogProperties(catalogName, CatalogConfigType.DYNAMIC, dynamicCatalogConfig.getCatalogConfigurationDir());
}
else if (shareCatalogs.contains(catalogName)) {
return shareCatalogStore.getCatalogProperties(catalogName, CatalogConfigType.DYNAMIC, dynamicCatalogConfig.getCatalogShareConfigurationDir());
}
else {
return localCatalogStore.getCatalogProperties(catalogName, CatalogConfigType.STATIC, staticCatalogConfig.getCatalogConfigurationDir().toString());
}
}
public enum CatalogStoreType {
LOCAL,
SHARE
}
public enum CatalogConfigType {
DYNAMIC,
STATIC
}
}

View File

@ -1,4 +1,5 @@
/*
* Copyright (C) 2018-2020. Huawei Technologies Co., Ltd. All rights reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
@ -28,6 +29,7 @@ import io.prestosql.execution.QueryTracker;
import io.prestosql.execution.resourcegroups.ResourceGroupManager;
import io.prestosql.metadata.SessionPropertyManager;
import io.prestosql.queryeditorui.QueryEditorUIModule;
import io.prestosql.queryhistory.QueryHistoryService;
import io.prestosql.security.AccessControl;
import io.prestosql.server.BasicQueryInfo;
import io.prestosql.server.SessionContext;
@ -92,6 +94,7 @@ public class DispatchManager
private StateFetcher stateFetcher;
private final HetuConfig hetuConfig;
private final int maxQueryLength;
private final QueryHistoryService queryHistoryService;
private final Executor queryExecutor;
@ -113,7 +116,8 @@ public class DispatchManager
QueryManagerConfig queryManagerConfig,
DispatchExecutor dispatchExecutor,
StateStoreProvider stateStoreProvider,
HetuConfig hetuConfig)
HetuConfig hetuConfig,
QueryHistoryService queryHistoryService)
{
this.queryIdGenerator = requireNonNull(queryIdGenerator, "queryIdGenerator is null");
this.queryPreparer = requireNonNull(queryPreparer, "queryPreparer is null");
@ -127,6 +131,7 @@ public class DispatchManager
// StateStoreProvider and load HetuConfig
this.stateStoreProvider = requireNonNull(stateStoreProvider, "stateStoreProvider is null");
this.hetuConfig = requireNonNull(hetuConfig, "hetuConfig is null");
this.queryHistoryService = requireNonNull(queryHistoryService, "queryHistoryService is null");
PropertyService.setProperty(HetuConstant.MULTI_COORDINATOR_ENABLED, hetuConfig.isMultipleCoordinatorEnabled());
requireNonNull(queryManagerConfig, "queryManagerConfig is null");
@ -289,6 +294,10 @@ public class DispatchManager
queryTracker.removeQuery(dispatchQuery.getQueryId());
}
else {
if (!isUiQuery && dispatchQuery.getBasicQueryInfo().getState() == QueryState.FAILED) {
QueryInfo queryInfo = queryHistoryService.toFullQueryInfo(dispatchQuery);
queryHistoryService.insert(queryInfo);
}
// execution MUST be added to the expiration queue or there will be a leak
queryTracker.expireQuery(dispatchQuery.getQueryId());
}

View File

@ -17,6 +17,7 @@ import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableMap;
import io.airlift.log.Logger;
import io.prestosql.spi.classloader.ThreadContextClassLoader;
import io.prestosql.spi.eventlistener.AuditLogEvent;
import io.prestosql.spi.eventlistener.EventListener;
import io.prestosql.spi.eventlistener.EventListenerFactory;
import io.prestosql.spi.eventlistener.QueryCompletedEvent;
@ -41,6 +42,8 @@ public class EventListenerManager
private static final Logger log = Logger.get(EventListenerManager.class);
private static final File EVENT_LISTENER_CONFIGURATION = new File("etc/event-listener.properties");
private static final String EVENT_LISTENER_PROPERTY_NAME = "event-listener.name";
private static String logOutput = "etc/log/";
private static String logconversionpattern = "yyyy-MM-dd.HH";
private final Map<String, EventListenerFactory> eventListenerFactories = new ConcurrentHashMap<>();
private final AtomicReference<Optional<EventListener>> configuredEventListener = new AtomicReference<>(Optional.empty());
@ -63,7 +66,8 @@ public class EventListenerManager
String eventListenerName = properties.remove(EVENT_LISTENER_PROPERTY_NAME);
checkArgument(!isNullOrEmpty(eventListenerName),
"Access control configuration %s does not contain %s", EVENT_LISTENER_CONFIGURATION.getAbsoluteFile(), EVENT_LISTENER_PROPERTY_NAME);
logOutput = properties.get("hetu.auditlog.logoutput");
logconversionpattern = properties.get("hetu.auditlog.logconversionpattern");
setConfiguredEventListener(eventListenerName, properties);
}
}
@ -87,6 +91,16 @@ public class EventListenerManager
log.info("-- Loaded event listener %s --", name);
}
public String getLogOutput()
{
return logOutput;
}
public String getLogconversionpattern()
{
return logconversionpattern;
}
public void queryCompleted(QueryCompletedEvent queryCompletedEvent)
{
if (configuredEventListener.get().isPresent()) {
@ -107,4 +121,15 @@ public class EventListenerManager
configuredEventListener.get().get().splitCompleted(splitCompletedEvent);
}
}
//Expand event, it includes
//1.user login and logout
//2.cluster node added and deleted
//3.openLooKeng start and close
public void eventEnhanced(AuditLogEvent auditLogEvent)
{
if (configuredEventListener.get().isPresent()) {
configuredEventListener.get().get().auditLogged(auditLogEvent);
}
}
}

View File

@ -29,6 +29,7 @@ import io.prestosql.execution.StateMachine.StateChangeListener;
import io.prestosql.memory.ClusterMemoryManager;
import io.prestosql.metadata.SessionPropertyManager;
import io.prestosql.queryeditorui.QueryEditorUIModule;
import io.prestosql.queryhistory.QueryHistoryService;
import io.prestosql.server.BasicQueryInfo;
import io.prestosql.spi.ErrorType;
import io.prestosql.spi.PrestoException;
@ -99,14 +100,16 @@ public class SqlQueryManager
// LocalStateProvider
private final StateStoreProvider stateStoreProvider;
private final SessionPropertyManager sessionPropertyManager;
private final QueryHistoryService queryHistoryService;
// Inject LocalStateProvider
@Inject
public SqlQueryManager(ClusterMemoryManager memoryManager, QueryMonitor queryMonitor, EmbedVersion embedVersion, QueryManagerConfig queryManagerConfig, StateStoreProvider stateStoreProvider, SessionPropertyManager sessionPropertyManager)
public SqlQueryManager(ClusterMemoryManager memoryManager, QueryMonitor queryMonitor, EmbedVersion embedVersion, QueryManagerConfig queryManagerConfig, StateStoreProvider stateStoreProvider, SessionPropertyManager sessionPropertyManager, QueryHistoryService queryHistoryService)
{
this.memoryManager = requireNonNull(memoryManager, "memoryManager is null");
this.queryMonitor = requireNonNull(queryMonitor, "queryMonitor is null");
this.embedVersion = requireNonNull(embedVersion, "embedVersion is null");
this.queryHistoryService = requireNonNull(queryHistoryService, "embedVersion is null");
this.maxQueryCpuTime = queryManagerConfig.getQueryMaxCpuTime();
@ -274,6 +277,9 @@ public class SqlQueryManager
queryExecution.addFinalQueryInfoListener(finalQueryInfo -> {
try {
queryMonitor.queryCompletedEvent(finalQueryInfo);
if (!(finalQueryInfo.getSession().getSource().map(source -> QueryEditorUIModule.UI_QUERY_SOURCE.equals(source)).orElse(false))) {
queryHistoryService.insert(finalQueryInfo);
}
}
finally {
// execution MUST be added to the expiration queue or there will be a leak

View File

@ -28,6 +28,7 @@ import io.airlift.json.JsonCodec;
import io.airlift.log.Logger;
import io.airlift.units.DataSize;
import io.airlift.units.Duration;
import io.prestosql.eventlistener.EventListenerManager;
import io.prestosql.execution.LocationFactory;
import io.prestosql.execution.QueryExecution;
import io.prestosql.execution.QueryIdGenerator;
@ -42,6 +43,7 @@ import io.prestosql.server.InternalCommunicationConfig;
import io.prestosql.server.ServerConfig;
import io.prestosql.spi.PrestoException;
import io.prestosql.spi.QueryId;
import io.prestosql.spi.eventlistener.AuditLogEvent;
import io.prestosql.spi.memory.ClusterMemoryPoolManager;
import io.prestosql.spi.memory.MemoryPoolId;
import io.prestosql.spi.memory.MemoryPoolInfo;
@ -110,6 +112,7 @@ public class ClusterMemoryManager
private final ExecutorService listenerExecutor = Executors.newSingleThreadExecutor();
private final ClusterMemoryLeakDetector memoryLeakDetector = new ClusterMemoryLeakDetector();
private final InternalNodeManager nodeManager;
private final EventListenerManager eventListenerManager;
private final LocationFactory locationFactory;
private final HttpClient httpClient;
private final MBeanExporter exporter;
@ -154,6 +157,7 @@ public class ClusterMemoryManager
public ClusterMemoryManager(
@ForMemoryManager HttpClient httpClient,
InternalNodeManager nodeManager,
EventListenerManager eventListenerManager,
LocationFactory locationFactory,
MBeanExporter exporter,
JsonCodec<MemoryInfo> memoryInfoJsonCodec,
@ -173,6 +177,7 @@ public class ClusterMemoryManager
requireNonNull(nodeMemoryConfig, "nodeMemoryConfig is null");
requireNonNull(serverConfig, "serverConfig is null");
this.nodeManager = requireNonNull(nodeManager, "nodeManager is null");
this.eventListenerManager = requireNonNull(eventListenerManager, "eventListenerManager is null");
this.locationFactory = requireNonNull(locationFactory, "locationFactory is null");
this.httpClient = requireNonNull(httpClient, "httpClient is null");
this.exporter = requireNonNull(exporter, "exporter is null");
@ -531,14 +536,21 @@ public class ClusterMemoryManager
// Remove nodes that don't exist anymore
// Make a copy to materialize the set difference
Set<String> deadNodes = ImmutableSet.copyOf(difference(nodes.keySet(), aliveNodeIds));
for (String deadNode : deadNodes) {
if (nodes.get(deadNode) == null) {
continue;
}
InternalNode removedNode = nodes.get(deadNode).getNode();
eventListenerManager.eventEnhanced(new AuditLogEvent("Unknown", removedNode.getInternalUri().toString(), "Remove Node: " + removedNode.getNodeIdentifier(), "Cluster", "WARN"));
}
nodes.keySet().removeAll(deadNodes);
// Add new nodes
for (InternalNode node : aliveNodes) {
if (!nodes.containsKey(node.getNodeIdentifier()) && shouldIncludeNode(node)) {
// nodes.put(node.getNodeIdentifier(), new RemoteNodeMemory(node, httpClient, memoryInfoCodec, assignmentsRequestCodec, locationFactory.createMemoryInfoLocation(node), isBinaryEncoding));
nodes.put(node.getInternalUri().toString(), new RemoteNodeMemory(node, httpClient, memoryInfoCodec, assignmentsRequestCodec, locationFactory.createMemoryInfoLocation(node), isBinaryEncoding));
allNodes.put(node.getInternalUri().toString(), new RemoteNodeMemory(node, httpClient, memoryInfoCodec, assignmentsRequestCodec, locationFactory.createMemoryInfoLocation(node), isBinaryEncoding));
nodes.put(node.getNodeIdentifier(), new RemoteNodeMemory(node, httpClient, memoryInfoCodec, assignmentsRequestCodec, locationFactory.createMemoryInfoLocation(node), isBinaryEncoding));
allNodes.put(node.getNodeIdentifier(), new RemoteNodeMemory(node, httpClient, memoryInfoCodec, assignmentsRequestCodec, locationFactory.createMemoryInfoLocation(node), isBinaryEncoding));
eventListenerManager.eventEnhanced(new AuditLogEvent("Unknown", node.getInternalUri().toString(), "Add Node: " + node.getNodeIdentifier(), "Cluster", "INFO"));
}
}

View File

@ -74,7 +74,6 @@ public class HetuMetaStoreManager
public void addHetuMetaStoreFactory(HetuMetaStoreFactory hetuMetaStoreFactory)
{
requireNonNull(hetuMetaStoreFactory, "hetuMetaStoreFactory is null");
if (hetuMetastoreFactories.putIfAbsent(hetuMetaStoreFactory.getName(), hetuMetaStoreFactory) != null) {
throw new IllegalArgumentException(format("HetuMetastore '%s' is already registered", hetuMetaStoreFactory.getName()));
}
@ -124,4 +123,9 @@ public class HetuMetaStoreManager
{
return hetuMetastore;
}
public String getHetuMetastoreType()
{
return hetuMetastoreType;
}
}

View File

@ -39,6 +39,7 @@ import io.prestosql.queryeditorui.output.persistors.CSVPersistorFactory;
import io.prestosql.queryeditorui.output.persistors.PersistorFactory;
import io.prestosql.queryeditorui.protocol.ExecutionStatus.ExecutionError;
import io.prestosql.queryeditorui.protocol.ExecutionStatus.ExecutionSuccess;
import io.prestosql.queryeditorui.resources.AuditLogResource;
import io.prestosql.queryeditorui.resources.ConnectorResource;
import io.prestosql.queryeditorui.resources.FilesResource;
import io.prestosql.queryeditorui.resources.LoginResource;
@ -90,7 +91,6 @@ public class QueryEditorUIModule
protected void setup(Binder binder)
{
configBinder(binder).bindConfig(QueryEditorConfig.class);
//resources
jsonCodecBinder(binder).bindJsonCodec(ExecutionSuccess.class);
jsonCodecBinder(binder).bindJsonCodec(ExecutionError.class);
@ -102,7 +102,7 @@ public class QueryEditorUIModule
jaxrsBinder(binder).bind(ConnectorResource.class);
jaxrsBinder(binder).bind(LoginResource.class);
jaxrsBinder(binder).bind(UserResource.class);
jaxrsBinder(binder).bind(AuditLogResource.class);
binder.bind(SchemaService.class).in(Scopes.SINGLETON);
binder.bind(ColumnService.class).in(Scopes.SINGLETON);
binder.bind(PreviewTableService.class).in(Scopes.SINGLETON);

View File

@ -0,0 +1,264 @@
/*
* Copyright (C) 2018-2020. Huawei Technologies Co., Ltd. All rights reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.prestosql.queryeditorui.resources;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import io.prestosql.eventlistener.EventListenerManager;
import io.prestosql.protocol.ObjectMapperProvider;
import io.prestosql.security.AccessControl;
import io.prestosql.security.AccessControlUtil;
import io.prestosql.server.HttpRequestSessionContext;
import io.prestosql.server.ServerConfig;
import io.prestosql.spi.security.GroupProvider;
import org.eclipse.jetty.server.Response;
import javax.inject.Inject;
import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.ws.rs.FormParam;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.Context;
import java.io.BufferedOutputStream;
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Optional;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
import static com.google.common.base.Strings.emptyToNull;
import static java.util.Objects.requireNonNull;
@Path("/v1/audit")
public class AuditLogResource
{
private final AccessControl accessControl;
private final ServerConfig serverConfig;
private final GroupProvider groupProvider;
private final EventListenerManager eventListenerManager;
@Inject
public AuditLogResource(AccessControl accessControl,
ServerConfig serverConfig,
GroupProvider groupProvider,
EventListenerManager eventListenerManager)
{
this.accessControl = requireNonNull(accessControl, "httpServerInfo is null");
this.serverConfig = requireNonNull(serverConfig, "httpServerInfo is null");
this.groupProvider = requireNonNull(groupProvider, "groupProvider is null");
this.eventListenerManager = requireNonNull(eventListenerManager, "eventListenerManager is null");
}
@POST
@Path("/{type}")
public void getAuditLog(
@FormParam("user") String username,
@FormParam("beginTime") String beginTime,
@FormParam("endTime") String endTime,
@FormParam("level") String level,
@PathParam("type") String type,
@Context HttpServletResponse response,
@Context HttpServletRequest servletRequest) throws IOException
{
//if the user is admin, don't filter results by user.
Optional<String> filterUser = AccessControlUtil.getUserForFilter(accessControl, serverConfig, servletRequest, groupProvider);
if (filterUser.isPresent()) {
response.setStatus(Response.SC_FORBIDDEN);
return;
}
String inputUsername = emptyToNull(username);
String inputBeginTime = emptyToNull(beginTime);
String inputEndTime = emptyToNull(endTime);
String inputLevel = emptyToNull(level);
List<String> logFiles = getLogFiles(type, inputBeginTime, inputEndTime, inputUsername, inputLevel);
if (logFiles.isEmpty()) {
response.setStatus(Response.SC_NOT_FOUND);
return;
}
response.reset();
response.setCharacterEncoding("utf-8");
response.setContentType("application/json");
ServletOutputStream out = response.getOutputStream();
List<String> resLog = new ArrayList<>(100);
int auditLogLimit = 99; // show 100 logs in webUI
for (int i = logFiles.size() - 1; auditLogLimit >= 0 && i >= 0; i--) {
BufferedReader br = new BufferedReader(new FileReader(logFiles.get(i)));
String str = null;
while ((str = br.readLine()) != null) {
resLog.add(str + System.lineSeparator());
auditLogLimit--;
if (auditLogLimit < 0) {
break;
}
}
br.close();
}
ObjectMapper objectMapper = new ObjectMapperProvider().get();
objectMapper.disable(SerializationFeature.FAIL_ON_EMPTY_BEANS);
objectMapper.enable(SerializationFeature.INDENT_OUTPUT);
out.println(objectMapper.writeValueAsString(resLog));
out.flush();
resLog.clear();
}
//downloadLogFiles, return compressed package in. Zip format
@GET
@Path("/download")
public void downloadLogFiles(
@QueryParam("user") String username,
@QueryParam("beginTime") String beginTime,
@QueryParam("endTime") String endTime,
@QueryParam("level") String level,
@QueryParam("type") String type,
@Context HttpServletResponse response,
@Context HttpServletRequest servletRequest) throws IOException
{
//if the user is admin, don't filter results by user.
Optional<String> filterUser = AccessControlUtil.getUserForFilter(accessControl, serverConfig, servletRequest, groupProvider);
String sessionUser = AccessControlUtil.getUser(accessControl, new HttpRequestSessionContext(servletRequest, groupProvider));
if (filterUser.isPresent()) {
response.setStatus(Response.SC_FORBIDDEN);
return;
}
String inputUsername = emptyToNull(username);
String inputBeginTime = emptyToNull(beginTime);
String inputEndTime = emptyToNull(endTime);
String inputLevel = emptyToNull(level);
List<String> logFiles = getLogFiles(type, inputBeginTime, inputEndTime, inputUsername, inputLevel);
if (logFiles.isEmpty()) {
response.setStatus(Response.SC_NOT_FOUND);
return;
}
response.reset();
response.setContentType("multipart/form-data");
String downloadName = getCurrentDate() + "_" + sessionUser + "_auditLog.zip";
response.setHeader("Content-Disposition", "attachment;fileName=" + downloadName);
//set compression stream: write response directly to achieve compression while downloading
ZipOutputStream zipos = new ZipOutputStream(new BufferedOutputStream(response.getOutputStream()));
zipos.setMethod(ZipOutputStream.DEFLATED);
DataOutputStream os = null;
try {
for (String filePath : logFiles) {
File file = new File(filePath);
zipos.putNextEntry(new ZipEntry(file.getName()));
os = new DataOutputStream(zipos);
InputStream is = new FileInputStream(file);
byte[] b = new byte[1024];
int length = 0;
while ((length = is.read(b)) != -1) {
os.write(b, 0, length);
}
is.close();
zipos.closeEntry();
}
}
finally {
os.flush();
os.close();
zipos.close();
}
}
@GET
@Path("/pattern")
public String getPattern()
{
String pattern = eventListenerManager.getLogconversionpattern();
if (pattern == null || !pattern.contains(".")) {
pattern = "YYYY-MM-DD";
}
else {
pattern = "YYYY-MM-DD.HH";
}
return pattern;
}
public List<String> getLogFiles(String type, String beginTime, String endTime, String user, String level) throws IOException
{
ArrayList<String> res = new ArrayList<>();
String logPath = eventListenerManager.getLogOutput() + "/" + type;
File f = new File(logPath);
File[] file = f.listFiles();
if (file == null) {
return res;
}
Arrays.sort(file, (o1, o2) -> Long.valueOf(o1.lastModified()).compareTo(o2.lastModified()));
for (File tmpFile : file) {
if (filterTimeAndUser(beginTime, endTime, user, level, tmpFile.getName())) {
res.add(tmpFile.getCanonicalPath());
}
}
return res;
}
private static Boolean filterTimeAndUser(String beginTime, String endTime, String user, String level, String filename)
{
if (!filename.contains(".log") || filename.contains(".lck") || filename.contains(".log.")) {
return false;
}
String[]values = filename.split("#");
String tmpLevel = values[0];
String tmpUser = values[1];
String tmpTime = values[2].substring(0, values[2].length() - 4); // cut ".log" from filename
if (beginTime != null && beginTime.contains(".") && tmpTime.length() != beginTime.length()) {
tmpTime = tmpTime + ".00";
}
if (beginTime != null && tmpTime.compareTo(beginTime) < 0) {
return false;
}
if (endTime != null && tmpTime.compareTo(endTime) > 0) {
return false;
}
if (user != null && !tmpUser.equals(user)) {
return false;
}
if (level != null && !tmpLevel.equals(level)) {
return false;
}
return true;
}
private static String getCurrentDate()
{
LocalDateTime now = LocalDateTime.now();
DateTimeFormatter format = DateTimeFormatter.ofPattern("yyyy-MM-dd.HH");
return now.format(format);
}
}

View File

@ -15,9 +15,12 @@
package io.prestosql.queryeditorui.resources;
import com.google.inject.Inject;
import io.prestosql.eventlistener.EventListenerManager;
import io.prestosql.queryeditorui.security.UiAuthenticator;
import io.prestosql.server.security.WebUIAuthenticator;
import io.prestosql.spi.eventlistener.AuditLogEvent;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.FormParam;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
@ -29,16 +32,20 @@ import javax.ws.rs.core.SecurityContext;
import java.util.Optional;
import static com.google.common.base.Strings.emptyToNull;
import static io.prestosql.client.PrestoHeaders.PRESTO_USER;
@Path("/")
public class LoginResource
{
private WebUIAuthenticator uiAuthenticator;
private EventListenerManager eventListenerManager;
@Inject
public LoginResource(WebUIAuthenticator uiAuthenticator)
public LoginResource(WebUIAuthenticator uiAuthenticator, EventListenerManager eventListenerManager)
{
this.uiAuthenticator = uiAuthenticator;
this.eventListenerManager = eventListenerManager;
}
@POST
@ -47,24 +54,40 @@ public class LoginResource
@FormParam("username") String username,
@FormParam("password") String password,
@FormParam("redirectPath") String redirectPath,
@Context SecurityContext securityContext)
@Context SecurityContext securityContext,
@Context HttpServletRequest request)
{
username = emptyToNull(username);
password = emptyToNull(password);
redirectPath = emptyToNull(redirectPath);
Optional<NewCookie> newCookie = uiAuthenticator.checkLoginCredentials(username, password, securityContext.isSecure());
String localPassword = password;
String inputUsername = emptyToNull(username);
localPassword = emptyToNull(localPassword);
String inputRedirectPath = emptyToNull(redirectPath);
String userAgent = request.getHeader("User-Agent");
Optional<NewCookie> newCookie = uiAuthenticator.checkLoginCredentials(inputUsername, localPassword, securityContext.isSecure());
AuditLogEvent enhancedEvent = new AuditLogEvent(username, request.getRemoteAddr(), "", "WebUi", "");
if (newCookie.isPresent()) {
return UiAuthenticator.redirectFromSuccessfulLoginResponse(redirectPath)
enhancedEvent.setOperation("WebUi login!, UserAgent=\'" + userAgent);
enhancedEvent.setLevel("INFO");
eventListenerManager.eventEnhanced(enhancedEvent);
return UiAuthenticator.redirectFromSuccessfulLoginResponse(inputRedirectPath)
.cookie(newCookie.get()).build();
}
// authentication failed, redirect back to the login page
enhancedEvent.setOperation("Attempt to login WebUi failed! UserAgent=\'" + userAgent);
enhancedEvent.setLevel("WARN");
eventListenerManager.eventEnhanced(enhancedEvent);
return Response.seeOther(UiAuthenticator.LOGIN_FORM_URI).build();
}
@POST
@Path("/ui/api/logout")
public Response logout(@Context SecurityContext securityContext)
public Response logout(@Context SecurityContext securityContext, @Context HttpServletRequest request)
{
String user = emptyToNull(request.getHeader(PRESTO_USER));
if (user == null) {
user = (String) request.getAttribute(PRESTO_USER);
}
String operation = "WebUi logout!, UserAgent=" + request.getHeader("User-Agent");
eventListenerManager.eventEnhanced(new AuditLogEvent(user, request.getRemoteAddr(), operation, "WebUi", "INFO"));
return Response.seeOther(UiAuthenticator.LOGIN_FORM_URI)
.cookie(UiAuthenticator.getDeleteCookie(securityContext.isSecure()))
.build();

View File

@ -36,6 +36,7 @@ import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import java.util.List;
import java.util.Set;
import java.util.concurrent.ExecutionException;
@Path("/api/metadata")
@ -105,6 +106,18 @@ public class MetadataResource
return Response.ok(preview).build();
}
@GET
@Produces(MediaType.APPLICATION_JSON)
@Path("catalogs")
public Response getCatalogs(
@Context HttpServletRequest servletRequest)
throws ExecutionException
{
String user = AccessControlUtil.getUser(accessControl, new HttpRequestSessionContext(servletRequest, groupProvider));
Set<String> result = schemaService.queryCatalogs(user);
return Response.ok(result).build();
}
@GET
@Produces(MediaType.APPLICATION_JSON)
@Path("schemas")

View File

@ -0,0 +1,47 @@
/*
* Copyright (C) 2018-2020. Huawei Technologies Co., Ltd. All rights reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.prestosql.queryhistory;
import io.airlift.configuration.Config;
public class QueryHistoryConfig
{
private int maxQueryHistoryCount = 1000;
private int maxCollectionSqlCount = 100;
public int getMaxQueryHistoryCount()
{
return maxQueryHistoryCount;
}
@Config("hetu.queryhistory.max-count")
public QueryHistoryConfig setMaxQueryHistoryCount(int maxQueryHistoryCount)
{
this.maxQueryHistoryCount = maxQueryHistoryCount;
return this;
}
public int getMaxCollectionSqlCount()
{
return maxCollectionSqlCount;
}
@Config("hetu.collectionsql.max-count")
public QueryHistoryConfig setMaxCollectionSqlCount(int maxCollectionSqlCount)
{
this.maxCollectionSqlCount = maxCollectionSqlCount;
return this;
}
}

View File

@ -0,0 +1,37 @@
/*
* Copyright (C) 2018-2020. Huawei Technologies Co., Ltd. All rights reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.prestosql.queryhistory;
import com.google.inject.Binder;
import com.google.inject.Scopes;
import io.airlift.configuration.AbstractConfigurationAwareModule;
import io.prestosql.queryhistory.collectionsql.CollectionSqlResource;
import io.prestosql.queryhistory.collectionsql.CollectionSqlService;
import static io.airlift.configuration.ConfigBinder.configBinder;
import static io.airlift.jaxrs.JaxrsBinder.jaxrsBinder;
public class QueryHistoryModule
extends AbstractConfigurationAwareModule
{
@Override
protected void setup(Binder binder)
{
configBinder(binder).bindConfig(QueryHistoryConfig.class);
jaxrsBinder(binder).bind(CollectionSqlResource.class);
binder.bind(CollectionSqlService.class).in(Scopes.SINGLETON);
binder.bind(QueryHistoryService.class).in(Scopes.SINGLETON);
}
}

View File

@ -0,0 +1,323 @@
/*
* Copyright (C) 2018-2020. Huawei Technologies Co., Ltd. All rights reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.prestosql.queryhistory;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import com.google.common.base.Strings;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.inject.Inject;
import io.airlift.log.Logger;
import io.airlift.units.DataSize;
import io.airlift.units.Duration;
import io.prestosql.dispatcher.DispatchQuery;
import io.prestosql.execution.QueryInfo;
import io.prestosql.execution.QueryStats;
import io.prestosql.metastore.HetuMetaStoreManager;
import io.prestosql.protocol.ObjectMapperProvider;
import io.prestosql.queryhistory.model.Info;
import io.prestosql.server.BasicQueryInfo;
import io.prestosql.server.BasicQueryStats;
import io.prestosql.spi.queryhistory.QueryHistoryEntity;
import io.prestosql.spi.queryhistory.QueryHistoryResult;
import java.util.ArrayList;
import java.util.Locale;
import java.util.Optional;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import static com.google.common.base.Preconditions.checkArgument;
import static java.util.Objects.requireNonNull;
public class QueryHistoryService
{
private static final Logger log = Logger.get(QueryHistoryService.class);
private final HetuMetaStoreManager hetuMetaStoreManager;
private final QueryHistoryConfig queryHistoryConfig;
private static final DataSize ZERO_BYTES = new DataSize(0, DataSize.Unit.BYTE);
private static final Duration ZERO_MILLIS = new Duration(0, TimeUnit.MILLISECONDS);
private static AtomicLong currentQueries = new AtomicLong(0);
@Inject
public QueryHistoryService(HetuMetaStoreManager hetuMetaStoreManager, QueryHistoryConfig queryHistoryConfig)
{
this.hetuMetaStoreManager = requireNonNull(hetuMetaStoreManager, "metaStoreManager is null");
this.queryHistoryConfig = requireNonNull(queryHistoryConfig, "queryHistoryConfig is null");
}
public void insert(QueryInfo queryInfo)
{
if (queryInfo == null) {
return;
}
String user = queryInfo.getSession().getUser();
String source = queryInfo.getSession().getSource().get();
if (source == null) {
return;
}
String queryId = queryInfo.getQueryId().getId();
String resource = null;
if (queryInfo.getResourceGroupId().isPresent()) {
resource = queryInfo.getResourceGroupId().get().getLastSegment();
}
String query = queryInfo.getQuery();
String catalog = queryInfo.getSession().getCatalog().get();
String schemata = queryInfo.getSession().getSchema().get();
String state = queryInfo.getState().toString();
String failed = "null";
if (queryInfo.getErrorType() != null) {
failed = queryInfo.getErrorType().toString();
}
String createTime = queryInfo.getQueryStats().getCreateTime().toString();
String elapsedTime = queryInfo.getQueryStats().getElapsedTime().toString();
String cpuTime = queryInfo.getQueryStats().getTotalCpuTime().toString();
String executionTime = queryInfo.getQueryStats().getExecutionTime().toString();
String currentMemory = queryInfo.getQueryStats().getUserMemoryReservation().toString();
double cumulativeUserMemory = queryInfo.getQueryStats().getCumulativeUserMemory();
ObjectMapper objectMapper = new ObjectMapperProvider().get();
objectMapper.disable(SerializationFeature.FAIL_ON_EMPTY_BEANS);
objectMapper.enable(SerializationFeature.INDENT_OUTPUT);
String jsonString = null;
try {
jsonString = objectMapper.writerWithDefaultPrettyPrinter().writeValueAsString(queryInfo);
}
catch (JsonProcessingException e) {
log.error(e);
}
int completedDrivers = queryInfo.getQueryStats().getCompletedDrivers();
int runningDrivers = queryInfo.getQueryStats().getRunningDrivers();
int queuedDrivers = queryInfo.getQueryStats().getQueuedDrivers();
String totalCpuTime = queryInfo.getQueryStats().getTotalCpuTime().toString();
String totalMemoryReservation = queryInfo.getQueryStats().getTotalMemoryReservation().toString();
String peakTotalMemoryReservation = queryInfo.getQueryStats().getPeakTotalMemoryReservation().toString();
QueryHistoryEntity queryHistoryEntity = new QueryHistoryEntity.Builder()
.setQueryId(queryId)
.setState(state)
.setFailed(failed)
.setQuery(query)
.setUser(user)
.setSource(source)
.setResource(resource)
.setCatalog(catalog)
.setSchemata(schemata)
.setCurrentMemory(currentMemory)
.setCompletedDrivers(completedDrivers)
.setRunningDrivers(runningDrivers)
.setQueuedDrivers(queuedDrivers)
.setCreateTime(createTime)
.setExecutionTime(executionTime)
.setElapsedTime(elapsedTime)
.setCpuTime(cpuTime)
.setTotalCpuTime(totalCpuTime)
.setTotalMemoryReservation(totalMemoryReservation)
.setPeakTotalMemoryReservation(peakTotalMemoryReservation)
.setCumulativeUserMemory(cumulativeUserMemory)
.build();
hetuMetaStoreManager.getHetuMetastore().insertQueryHistory(queryHistoryEntity, jsonString);
currentQueries.incrementAndGet();
}
public String getQueryDetail(String queryId)
{
String res = hetuMetaStoreManager.getHetuMetastore().getQueryDetail(queryId);
return res;
}
public QueryHistoryResult Query(Info info, int pageNum, int pageSize)
{
String user = info.getUser();
String startTime = info.getStartTime();
String endTime = info.getEndTime();
String query = info.getQuery();
String queryId = info.getQueryId();
String source = info.getSource();
String resource = info.getResource();
String state = info.getState();
String failed = info.getFailed();
String sort = info.getSort();
String sortOrder = info.getSortorder();
int startNum = (pageNum - 1) * pageSize;
String[] stateStr = state.split(",");
String[] failedStr = failed.split(",");
if (sortOrder.equals("descending")) {
sortOrder = "desc";
}
else {
sortOrder = "asc";
}
switch (sort) {
case "cumulative":
sort = "cumulativeUserMemory";
break;
case "memory":
sort = "currentMemory";
break;
case "elapsed":
sort = "elapsedTime";
break;
case "cpu":
sort = "cpuTime";
break;
case "execution":
sort = "executionTime";
break;
default:
sort = "createTime";
}
if (stateStr == null || stateStr.equals("")) {
return new QueryHistoryResult(0, null);
}
if (!Strings.isNullOrEmpty(startTime)) {
startTime = startTime.replace(".", "T");
}
if (!Strings.isNullOrEmpty(endTime)) {
endTime = endTime.replace(".", "T");
}
ArrayList<String> stateFilter = new ArrayList<>();
ArrayList<String> failFilter = new ArrayList<>();
for (String s : stateStr) {
stateFilter.add(s.toUpperCase(Locale.ENGLISH));
}
stateFilter.add("null");
for (String s : failedStr) {
failFilter.add(s.toUpperCase(Locale.ENGLISH));
}
failFilter.add("null");
QueryHistoryResult queryHistoryResult = hetuMetaStoreManager.getHetuMetastore().getQueryHistory(startNum, pageSize, user, startTime, endTime,
queryId, query, resource, source, stateFilter, failFilter, sort, sortOrder);
return queryHistoryResult;
}
// assign queryHistoryCount to currentQueries
// if queryHistoryCount > 1000, then delete the oldest 100 queryHistory
public Long getCurrentQueries()
{
if (currentQueries.get() == 0) {
currentQueries.set(hetuMetaStoreManager.getHetuMetastore().getAllQueryHistoryNum());
}
if (currentQueries.get() > queryHistoryConfig.getMaxQueryHistoryCount()) {
hetuMetaStoreManager.getHetuMetastore().deleteQueryHistoryBatch();
currentQueries.set(0);
}
return currentQueries.get();
}
public String getHetuMetastoreType()
{
return hetuMetaStoreManager.getHetuMetastoreType();
}
public static QueryInfo toFullQueryInfo(DispatchQuery query)
{
checkArgument(query.isDone(), "query is not done");
BasicQueryInfo info = query.getBasicQueryInfo();
BasicQueryStats stats = info.getQueryStats();
QueryStats queryStats = new QueryStats(
query.getCreateTime(),
query.getExecutionStartTime().orElse(null),
query.getLastHeartbeat(),
query.getEndTime().orElse(null),
stats.getElapsedTime(),
stats.getQueuedTime(),
ZERO_MILLIS,
ZERO_MILLIS,
ZERO_MILLIS,
ZERO_MILLIS,
ZERO_MILLIS,
ZERO_MILLIS,
ZERO_MILLIS,
ZERO_MILLIS,
ZERO_MILLIS,
0,
0,
0,
0,
0,
0,
0,
0,
0,
ZERO_BYTES,
ZERO_BYTES,
ZERO_BYTES,
ZERO_BYTES,
ZERO_BYTES,
ZERO_BYTES,
ZERO_BYTES,
ZERO_BYTES,
ZERO_BYTES,
info.isScheduled(),
ZERO_MILLIS,
ZERO_MILLIS,
ZERO_MILLIS,
false,
ImmutableSet.of(),
ZERO_BYTES,
0,
ZERO_BYTES,
0,
ZERO_BYTES,
0,
ZERO_BYTES,
0,
ZERO_BYTES,
0,
ZERO_BYTES,
ImmutableList.of(),
ImmutableList.of());
return new QueryInfo(
info.getQueryId(),
info.getSession(),
info.getState(),
info.getMemoryPool(),
info.isScheduled(),
info.getSelf(),
ImmutableList.of(),
info.getQuery(),
info.getPreparedQuery(),
queryStats,
Optional.empty(),
Optional.empty(),
Optional.empty(),
ImmutableMap.of(),
ImmutableSet.of(),
ImmutableMap.of(),
ImmutableMap.of(),
ImmutableSet.of(),
Optional.empty(),
false,
null,
Optional.empty(),
query.getDispatchInfo().getFailureInfo().orElse(null),
info.getErrorCode(),
ImmutableList.of(),
ImmutableSet.of(),
Optional.empty(),
true,
info.getResourceGroupId(), false);
}
}

View File

@ -0,0 +1,94 @@
/*
* Copyright (C) 2018-2020. Huawei Technologies Co., Ltd. All rights reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.prestosql.queryhistory.collectionsql;
import com.google.inject.Inject;
import io.prestosql.queryhistory.model.FavoriteInfo;
import io.prestosql.security.AccessControl;
import io.prestosql.security.AccessControlUtil;
import io.prestosql.server.HttpRequestSessionContext;
import io.prestosql.spi.favorite.FavoriteResult;
import io.prestosql.spi.security.GroupProvider;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.DELETE;
import javax.ws.rs.FormParam;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
@Path("/v1/myCollection")
public class CollectionSqlResource
{
private final AccessControl accessControl;
private final CollectionSqlService collectionSqlService;
private final GroupProvider groupProvider;
@Inject
private CollectionSqlResource(
AccessControl accessControl,
CollectionSqlService collectionSqlService,
GroupProvider groupProvider)
{
this.accessControl = accessControl;
this.collectionSqlService = collectionSqlService;
this.groupProvider = groupProvider;
}
@POST
@Produces(MediaType.APPLICATION_JSON)
@Path("collection")
public Response insertQueryText(
@FormParam("queryText") String queryText,
@FormParam("catalog") String catalog,
@FormParam("schema") String schema,
@Context HttpServletRequest servletRequest)
{
String user = AccessControlUtil.getUser(accessControl, new HttpRequestSessionContext(servletRequest, groupProvider));
Boolean res = collectionSqlService.insert(new FavoriteInfo(user, queryText, catalog, schema));
return Response.ok(res).build();
}
@DELETE
@Path("collection/delete")
public Response deleteQueryText(
@FormParam("queryText") String queryText,
@FormParam("catalog") String catalog,
@FormParam("schema") String schema,
@Context HttpServletRequest servletRequest)
{
String user = AccessControlUtil.getUser(accessControl, new HttpRequestSessionContext(servletRequest, groupProvider));
Boolean res = collectionSqlService.delete(new FavoriteInfo(user, queryText, catalog, schema));
return Response.ok(res).build();
}
@GET
@Produces(MediaType.APPLICATION_JSON)
@Path("collection/query")
public Response getQueryText(
@QueryParam("pageSize") int pageSize,
@QueryParam("pageNum") int pageNum,
@Context HttpServletRequest servletRequest)
{
String user = AccessControlUtil.getUser(accessControl, new HttpRequestSessionContext(servletRequest, groupProvider));
FavoriteResult res = collectionSqlService.query(pageNum, pageSize, user);
return Response.ok(res).build();
}
}

View File

@ -0,0 +1,67 @@
/*
* Copyright (C) 2018-2020. Huawei Technologies Co., Ltd. All rights reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.prestosql.queryhistory.collectionsql;
import com.google.inject.Inject;
import io.prestosql.metastore.HetuMetaStoreManager;
import io.prestosql.queryhistory.QueryHistoryConfig;
import io.prestosql.queryhistory.model.FavoriteInfo;
import io.prestosql.spi.favorite.FavoriteEntity;
import io.prestosql.spi.favorite.FavoriteResult;
import static java.util.Objects.requireNonNull;
public class CollectionSqlService
{
private final HetuMetaStoreManager hetuMetaStoreManager;
private final QueryHistoryConfig queryHistoryConfig;
@Inject
public CollectionSqlService(HetuMetaStoreManager hetuMetaStoreManager, QueryHistoryConfig queryHistoryConfig)
{
this.hetuMetaStoreManager = requireNonNull(hetuMetaStoreManager, "MetaStoreManager is null");
this.queryHistoryConfig = requireNonNull(queryHistoryConfig, "queryHistoryConfig is null");
}
public Boolean insert(FavoriteInfo favoriteInfo)
{
String user = favoriteInfo.getUser();
String query = favoriteInfo.getQuery();
String catalog = favoriteInfo.getCatalog();
String schema = favoriteInfo.getSchemata();
if (hetuMetaStoreManager.getHetuMetastore().getFavorite(1, 1, user).getTotal() == queryHistoryConfig.getMaxCollectionSqlCount()) {
return false;
}
hetuMetaStoreManager.getHetuMetastore().insertFavorite(new FavoriteEntity(user, query, catalog, schema));
return true;
}
public Boolean delete(FavoriteInfo favoriteInfo)
{
String user = favoriteInfo.getUser();
String query = favoriteInfo.getQuery();
String catalog = favoriteInfo.getCatalog();
String schemata = favoriteInfo.getSchemata();
hetuMetaStoreManager.getHetuMetastore().deleteFavorite(new FavoriteEntity(user, query, catalog, schemata));
return true;
}
public FavoriteResult query(int pageNum, int pageSize, String user)
{
int startNum = (pageNum - 1) * pageSize;
FavoriteResult favoriteResult = hetuMetaStoreManager.getHetuMetastore().getFavorite(startNum, pageSize, user);
return favoriteResult;
}
}

View File

@ -0,0 +1,93 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.prestosql.queryhistory.model;
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
public class FavoriteInfo
{
@JsonProperty
private String user;
@JsonProperty
private String query;
@JsonProperty
private String catalog;
@JsonProperty
private String schemata;
@JsonCreator
public FavoriteInfo()
{
}
@JsonCreator
public FavoriteInfo(@JsonProperty("user") String user,
@JsonProperty("query") String query,
@JsonProperty("catalog") String catalog,
@JsonProperty("schemata") String schemata)
{
this.user = user;
this.query = query;
this.catalog = catalog;
this.schemata = schemata;
}
@JsonProperty
public String getUser()
{
return user;
}
@JsonProperty
public String getQuery()
{
return query;
}
@JsonProperty
public String getCatalog()
{
return catalog;
}
@JsonProperty
public String getSchemata()
{
return schemata;
}
public void setUser(String user)
{
this.user = user;
}
public void setQuery(String query)
{
this.query = query;
}
public void setCatalog(String catalog)
{
this.catalog = catalog;
}
public void setSchemata(String schemata)
{
this.schemata = schemata;
}
}

View File

@ -0,0 +1,142 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.prestosql.queryhistory.model;
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
public class Info
{
@JsonProperty
private String user;
@JsonProperty
private String startTime;
@JsonProperty
private String endTime;
@JsonProperty
private String queryid;
@JsonProperty
private String query;
@JsonProperty
private String source;
@JsonProperty
private String resource;
@JsonProperty
private String state;
@JsonProperty
private String failed;
@JsonProperty
private String sort;
@JsonProperty
private String sortorder;
@JsonCreator
public Info(
@JsonProperty("user") final String user,
@JsonProperty("starttime") final String startTime,
@JsonProperty("endtime") final String endTime,
@JsonProperty("queryid") final String queryid,
@JsonProperty("query") final String query,
@JsonProperty("source") final String source,
@JsonProperty("resource") final String resource,
@JsonProperty("state") final String state,
@JsonProperty("failed") final String failed,
@JsonProperty("sort") final String sort,
@JsonProperty("sortorder") final String sortorder)
{
this.user = user;
this.startTime = startTime;
this.endTime = endTime;
this.queryid = queryid;
this.query = query;
this.source = source;
this.resource = resource;
this.state = state;
this.failed = failed;
this.sort = sort;
this.sortorder = sortorder;
}
@JsonProperty
public String getQuery()
{
return query;
}
@JsonProperty
public String getUser()
{ return user; }
@JsonProperty
public String getQueryId()
{ return queryid; }
@JsonProperty
public String getStartTime()
{
return startTime;
}
@JsonProperty
public String getEndTime()
{
return endTime;
}
@JsonProperty
public String getSource()
{
return source;
}
@JsonProperty
public String getResource()
{
return resource;
}
@JsonProperty
public String getState()
{
return state;
}
@JsonProperty
public String getFailed()
{
return failed;
}
@JsonProperty
public String getSort()
{
return sort;
}
@JsonProperty
public String getSortorder()
{
return sortorder;
}
}

View File

@ -26,6 +26,7 @@ import io.airlift.discovery.server.EmbeddedDiscoveryModule;
import io.airlift.units.Duration;
import io.prestosql.catalog.CatalogModule;
import io.prestosql.catalog.DynamicCatalogConfig;
import io.prestosql.catalog.showcatalog.ShowCatalogModule;
import io.prestosql.client.QueryResults;
import io.prestosql.cost.CostCalculator;
import io.prestosql.cost.CostCalculator.EstimatedExchanges;
@ -113,6 +114,7 @@ import io.prestosql.memory.TotalReservationOnBlockedNodesLowMemoryKiller;
import io.prestosql.metadata.CatalogManager;
import io.prestosql.operator.ForScheduler;
import io.prestosql.queryeditorui.QueryEditorUIModule;
import io.prestosql.queryhistory.QueryHistoryModule;
import io.prestosql.server.remotetask.RemoteTaskStats;
import io.prestosql.spi.memory.ClusterMemoryPoolManager;
import io.prestosql.spi.resourcegroups.QueryType;
@ -227,6 +229,12 @@ public class CoordinatorModule
// catalog resource
install(installModuleIf(DynamicCatalogConfig.class, DynamicCatalogConfig::isDynamicCatalogEnabled, new CatalogModule()));
//showcatalog resourece
install(new ShowCatalogModule());
//QueryHistory Module
binder.install(new QueryHistoryModule());
// resource for serving static content
jaxrsBinder(binder).bind(WebUiResource.class);

View File

@ -18,11 +18,14 @@ import com.google.common.annotations.VisibleForTesting;
import com.google.inject.Inject;
import io.airlift.bootstrap.LifeCycleManager;
import io.airlift.log.Logger;
import io.airlift.node.NodeInfo;
import io.airlift.units.Duration;
import io.prestosql.eventlistener.EventListenerManager;
import io.prestosql.execution.QueryManager;
import io.prestosql.execution.TaskManager;
import io.prestosql.execution.scheduler.NodeSchedulerConfig;
import io.prestosql.metadata.NodeState;
import io.prestosql.spi.eventlistener.AuditLogEvent;
import javax.annotation.concurrent.GuardedBy;
@ -59,6 +62,8 @@ public class NodeStateChangeHandler
private final TaskManager sqlTaskManager;
private final QueryManager sqlQueryManager;
private final ShutdownAction shutdownAction;
private final EventListenerManager eventListenerManager;
private final NodeInfo nodeInfo;
private final boolean isCoordinator;
private final boolean allowTaskOnCoordinator;
@ -76,7 +81,9 @@ public class NodeStateChangeHandler
ServerConfig serverConfig,
NodeSchedulerConfig nodeSchedulerConfig,
ShutdownAction shutdownAction,
LifeCycleManager lifeCycleManager)
LifeCycleManager lifeCycleManager,
EventListenerManager eventListenerManager,
NodeInfo nodeInfo)
{
this.sqlTaskManager = requireNonNull(sqlTaskManager, "sqlTaskManager is null");
this.sqlQueryManager = requireNonNull(sqlQueryManager, "sqlQueryManager is null");
@ -85,6 +92,8 @@ public class NodeStateChangeHandler
this.isCoordinator = requireNonNull(serverConfig, "serverConfig is null").isCoordinator();
this.allowTaskOnCoordinator = requireNonNull(nodeSchedulerConfig, "nodeScheulerConfig is null").isIncludeCoordinator();
this.gracePeriod = serverConfig.getGracePeriod();
this.nodeInfo = requireNonNull(nodeInfo, "nodeInfo is null");
this.eventListenerManager = requireNonNull(eventListenerManager, "eventListenerManager is null");
}
private synchronized void requestIsolation(boolean forced)
@ -307,6 +316,8 @@ public class NodeStateChangeHandler
default:
break;
}
eventListenerManager.eventEnhanced(new AuditLogEvent("Unknown", nodeInfo.getInternalAddress(),
"Updating node " + nodeInfo.getNodeId() + " state from " + state + " to " + newState, "Cluster", "INFO"));
LOG.info(String.format("Updating node state from %s to %s", state, newState));
state = newState;
return true;

View File

@ -33,6 +33,7 @@ import io.airlift.node.NodeModule;
import io.airlift.tracetoken.TraceTokenModule;
import io.prestosql.catalog.DynamicCatalogScanner;
import io.prestosql.catalog.DynamicCatalogStore;
import io.prestosql.catalog.showcatalog.ShowCatalogStore;
import io.prestosql.discovery.HetuDiscoveryModule;
import io.prestosql.dynamicfilter.CrossRegionDynamicFilterListener;
import io.prestosql.dynamicfilter.DynamicFilterCacheManager;
@ -154,6 +155,7 @@ public class PrestoServer
injector.getInstance(StaticFunctionNamespaceStore.class).loadFunctionNamespaceManagers();
injector.getInstance(StaticCatalogStore.class).loadCatalogs();
injector.getInstance(DynamicCatalogStore.class).loadCatalogStores(fileSystemClientManager);
injector.getInstance(ShowCatalogStore.class).loadCatalogStores(fileSystemClientManager);
injector.getInstance(DynamicCatalogScanner.class).start();
injector.getInstance(SessionPropertyDefaults.class).loadConfigurationManager();
injector.getInstance(ResourceGroupManager.class).loadConfigurationManager();

View File

@ -26,12 +26,15 @@ import io.prestosql.execution.QueryManager;
import io.prestosql.execution.QueryState;
import io.prestosql.execution.QueryStats;
import io.prestosql.execution.StageId;
import io.prestosql.queryhistory.QueryHistoryService;
import io.prestosql.queryhistory.model.Info;
import io.prestosql.security.AccessControl;
import io.prestosql.security.AccessControlUtil;
import io.prestosql.server.security.SecurityRequireNonNull;
import io.prestosql.spi.ErrorType;
import io.prestosql.spi.PrestoException;
import io.prestosql.spi.QueryId;
import io.prestosql.spi.queryhistory.QueryHistoryResult;
import io.prestosql.spi.security.GroupProvider;
import javax.inject.Inject;
@ -41,6 +44,7 @@ import javax.ws.rs.GET;
import javax.ws.rs.PUT;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.core.Context;
@ -57,6 +61,7 @@ import java.util.Optional;
import java.util.concurrent.TimeUnit;
import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.base.Strings.nullToEmpty;
import static io.prestosql.connector.system.KillQueryProcedure.createKillQueryException;
import static io.prestosql.connector.system.KillQueryProcedure.createPreemptQueryException;
import static java.util.Objects.requireNonNull;
@ -79,6 +84,7 @@ public class QueryResource
private final ServerConfig serverConfig;
private final GroupProvider groupProvider;
private final QueryHistoryService queryHistoryService;
public enum SortOrder
{
@ -92,7 +98,8 @@ public class QueryResource
HttpServerInfo httpServerInfo,
AccessControl accessControl,
ServerConfig serverConfig,
GroupProvider groupProvider)
GroupProvider groupProvider,
QueryHistoryService queryHistoryService)
{
this.dispatchManager = requireNonNull(dispatchManager, "dispatchManager is null");
this.queryManager = requireNonNull(queryManager, "queryManager is null");
@ -100,6 +107,63 @@ public class QueryResource
this.accessControl = requireNonNull(accessControl, "httpServerInfo is null");
this.serverConfig = requireNonNull(serverConfig, "httpServerInfo is null");
this.groupProvider = requireNonNull(groupProvider, "groupProvider is null");
this.queryHistoryService = requireNonNull(queryHistoryService, "queryHistoryService is null");
}
@GET
@Path("queryInfo")
public Response getAllQueryInfo(
@QueryParam("state") String stateFilter,
@QueryParam("failed") String failedFilter,
@QueryParam("sort") String sortFilter,
@QueryParam("sortOrder") String sortOrder,
@QueryParam("searchUser") String searchUser,
@QueryParam("searchDate") String searchDate,
@QueryParam("searchSql") String searchSql,
@QueryParam("searchQueryId") String searchQueryId,
@QueryParam("searchResourceGroup") String resourceGroup,
@QueryParam("pageNum") Integer pageNum,
@QueryParam("pageSize") Integer pageSize,
@Context HttpServletRequest servletRequest)
{
// if the user is admin, don't filter results by user.
if (pageNum != null && pageNum <= 0) {
return Response.status(Status.BAD_REQUEST).build();
}
if (pageSize != null && pageSize <= 0) {
return Response.status(Status.BAD_REQUEST).build();
}
String inputSearchUser = nullToEmpty(searchUser);
String inputSearchDate = nullToEmpty(searchDate);
String inputSearchSql = nullToEmpty(searchSql);
String inputSearchQueryId = nullToEmpty(searchQueryId);
String inputResourceGroup = nullToEmpty(resourceGroup);
String startTime = "";
String endTime = "";
if (!"".equals(inputSearchDate)) {
inputSearchDate = inputSearchDate.replaceAll(" ", "");
startTime = inputSearchDate.split("~")[0];
endTime = inputSearchDate.split("~")[1];
}
Info info = new Info(inputSearchUser, startTime, endTime, inputSearchQueryId, inputSearchSql, "", inputResourceGroup, stateFilter, failedFilter, sortFilter, sortOrder);
QueryHistoryResult res = queryHistoryService.Query(info, pageNum, pageSize);
return Response.ok(res).build();
}
@GET
@Path("/queryNums")
public Response getCurrentQueries()
{
return Response.ok(queryHistoryService.getCurrentQueries()).build();
}
@GET
@Path("/hetuMetastoreType")
public Response getHetuMetastoreType()
{
return Response.ok(queryHistoryService.getHetuMetastoreType()).build();
}
@GET
@ -252,6 +316,25 @@ public class QueryResource
}
}
@GET
@Produces("application/json")
@Path("history_{queryId}")
public Response getHistoryQueryInfo(@PathParam("queryId") QueryId queryId)
{
try {
requireNonNull(queryId, "queryId is null");
}
catch (Exception ex) {
return Response.status(Status.BAD_REQUEST).build();
}
String queryJson = queryHistoryService.getQueryDetail(queryId.toString());
if (queryJson == null || "".equals(queryJson)) {
return Response.status(Status.GONE).build();
}
return Response.ok(queryJson).build();
}
@DELETE
@Path("{queryId}")
public void cancelQuery(@PathParam("queryId") QueryId queryId)

View File

@ -16,8 +16,10 @@ package io.prestosql.server;
import io.airlift.node.NodeInfo;
import io.prestosql.client.NodeVersion;
import io.prestosql.client.ServerInfo;
import io.prestosql.eventlistener.EventListenerManager;
import io.prestosql.metadata.NodeState;
import io.prestosql.security.AccessControl;
import io.prestosql.spi.eventlistener.AuditLogEvent;
import io.prestosql.spi.security.AccessDeniedException;
import io.prestosql.spi.security.GroupProvider;
@ -56,21 +58,26 @@ public class ServerInfoResource
private final AtomicBoolean startupComplete = new AtomicBoolean();
private final AccessControl accessControl;
private final GroupProvider groupProvider;
private final EventListenerManager eventListenerManager;
private final NodeInfo nodeInfo;
@Inject
public ServerInfoResource(NodeVersion nodeVersion,
NodeInfo nodeInfo,
ServerConfig serverConfig,
NodeStateChangeHandler nodeStateChangeHandler,
AccessControl accessControl,
GroupProvider groupProvider)
NodeInfo nodeInfo,
ServerConfig serverConfig,
NodeStateChangeHandler nodeStateChangeHandler,
AccessControl accessControl,
GroupProvider groupProvider,
EventListenerManager eventListenerManager)
{
this.version = requireNonNull(nodeVersion, "nodeVersion is null");
this.environment = requireNonNull(nodeInfo, "nodeInfo is null").getEnvironment();
this.nodeInfo = requireNonNull(nodeInfo, "nodeInfo is null");
this.coordinator = requireNonNull(serverConfig, "serverConfig is null").isCoordinator();
this.nodeStateChangeHandler = requireNonNull(nodeStateChangeHandler, "nodeStateChangeHandler");
this.accessControl = requireNonNull(accessControl, "accessControl is null");
this.groupProvider = requireNonNull(groupProvider, "groupProvider is null");
this.eventListenerManager = requireNonNull(eventListenerManager, "eventListenerManager is null");
}
@GET
@ -125,6 +132,7 @@ public class ServerInfoResource
}
try {
eventListenerManager.eventEnhanced(new AuditLogEvent(servletRequest.getHeader("X-Presto-User"), nodeInfo.getInternalAddress(), state.toString(), "Cluster", "INFO"));
nodeStateChangeHandler.doStateTransition(state);
return Response.ok().build();
}
@ -159,6 +167,7 @@ public class ServerInfoResource
public void startupComplete()
{
eventListenerManager.eventEnhanced(new AuditLogEvent("Unknown", nodeInfo.getInternalAddress(), "Server: " + nodeInfo.getNodeId() + " Start!", "Cluster", "INFO"));
checkState(startupComplete.compareAndSet(false, true), "Server startup already marked as complete");
}

View File

@ -37,6 +37,7 @@ import io.prestosql.catalog.CatalogStoreUtil;
import io.prestosql.catalog.DynamicCatalogConfig;
import io.prestosql.catalog.DynamicCatalogScanner;
import io.prestosql.catalog.DynamicCatalogStore;
import io.prestosql.catalog.showcatalog.ShowCatalogStore;
import io.prestosql.client.NodeVersion;
import io.prestosql.client.ServerInfo;
import io.prestosql.connector.CatalogConnectorStore;
@ -515,6 +516,9 @@ public class ServerMainModule
binder.bind(CatalogStoreUtil.class).in(Scopes.SINGLETON);
configBinder(binder).bindConfig(DynamicCatalogConfig.class);
// show catalog
binder.bind(ShowCatalogStore.class).in(Scopes.SINGLETON);
// plugin manager
binder.bind(PluginManager.class).in(Scopes.SINGLETON);
configBinder(binder).bindConfig(PluginManagerConfig.class);

View File

@ -390,7 +390,7 @@ pre {
}
.query-snippet {
height: 138px;
height: 155px;
overflow: hidden;
}
@ -449,7 +449,40 @@ pre {
color: #092ff1;
}
/** Query List Toolbar **/
.progress-catalog-schema-container {
float: left;
font-size: 120%;
color: black;
font-weight: bold;
}
.progress-catalog-schema {
margin: 0 5%;
}
.text-catalog-schema {
color: white;
}
.progress-state {
float: right;
color: black;
margin-right: 2%;
}
.query-showmore {
margin: 10px;
}
.modal-querytext {
margin: 15px;
word-break: break-all;
white-space: pre-line;
max-height: 500px;
max-width: 780px;
}
/** Query Monitor Toolbar **/
/** ================== **/
.toolbar-row {
@ -518,7 +551,7 @@ pre {
border: none;
}
.toolbar-row .input-group .form-control {
.query-monitor .input-group .form-control {
margin-top: 2px;
margin-bottom: -2px;
width: 98%;
@ -550,6 +583,83 @@ pre {
margin-left: 143px;
}
/** Audit log **/
/** ========= **/
.toolbar-log {
padding-right: 15px;
}
.toolbar-audit {
background-color: #e6e6e6;
margin-top: 5px;
padding: 10px;
border-bottom: 1px solid black;
margin-bottom: 20px;
}
.toolbar-log .form-control {
color: #000;
}
.toolbar-log .input-group-audit input{
margin-top: 2px;
margin-bottom: -2px;
width: 30%;
border: 1px solid #000;
display: inline;
border-radius: 0;
}
.container .toolbar-log .input-group-audit .select{
color: black;
background-color: #e6e6e6;
border-color: black;
}
.container .toolbar-log .input-group-audit .bootstrap-select {
width: 7.5%;
margin: 0 10px;
}
.auditlog-text {
overflow: auto;
height: auto;
max-height: calc(100vh - 220px);
padding: 20px 20px 0 20px;
background: white;
margin: 1% 0 2% 0;
}
.auditlog-row {
margin-bottom: 2%;
}
.container .tab-container {
padding: 0px;
}
.small-container {
padding: 0px;
}
.tab-group {
width: 33%;
text-align: center;
border-radius: 8px;
border: 1px solid gray;
}
/** Query History **/
/** ========= **/
.query-history .input-group .form-control {
margin-top: 2px;
margin-bottom: -2px;
width: 17%;
border: 1px solid #000;
}
/** Loading indicators **/
/** ================== **/

View File

@ -93,6 +93,32 @@ body {
justify-content: flex-end;
}
.collect {
padding: 5px 280px;
}
.collection-btn {
border: 1px solid #d5d8db;
width: 50px;
height: 30px;
background-color: #fcfcfd;
border-radius: .25em;
margin: 0 10px 10px 10px;
}
.collection-btn:hover {
background-color: #efefef;
}
.collection-btn:active {
background-color: #e8e6e6;
}
.collection-btn .fa-plus {
font-size: 75%;
}
.hetu-app {
position: relative;
top: 0;
@ -1091,6 +1117,10 @@ button.hetu-form-label-plus:active,button.hetu-form-label-plus:active:hover,butt
color: #3a8edc;
}
.collection-btn .icon-center {
margin-right: 0px;
}
.fa-form-label {
position: relative;
font-size: 20px;
@ -1276,4 +1306,8 @@ button.hetu-form-label-plus:active,button.hetu-form-label-plus:active:hover,butt
.contextmenu-icons.remove-favorite {
color: #fd082d;
}
.contextmenu-icons.show-catalog {
color: #626e86;
}

View File

@ -0,0 +1,67 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="description" content="Audit Log - openLooKeng">
<title>Audit Log - openLooKeng</title>
<!-- <link rel="icon" href="assets/favicon.ico">-->
<!--EMPTY ICON-->
<link rel="icon" href="assets/favicon.ico"/>
<!-- Bootstrap core -->
<link href="vendor/bootstrap/css/bootstrap.css" rel="stylesheet">
<!-- Latest compiled and minified CSS -->
<link rel="stylesheet" href="vendor/bootstrap-select/dist/css/bootstrap-select.min.css">
<!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->
<!-- jQuery -->
<script type="text/javascript" src="vendor/jquery/jquery-2.2.3.min.js"></script>
<!-- Bootstrap JS -->
<script type="text/javascript" src="vendor/bootstrap/js/bootstrap.js"></script>
<!-- Sparkline -->
<script type="text/javascript" src="vendor/jquery.sparkline/jquery.sparkline.min.js"></script>
<!-- Latest compiled and minified JavaScript -->
<script src="vendor/bootstrap-select/dist/js/bootstrap-select.min.js"></script>
<!-- (Optional) Latest compiled and minified JavaScript translation files -->
<script src="vendor/bootstrap-select/dist/js/i18n/defaults-*.min.js"></script>
<!-- CSS loader -->
<link href="vendor/css-loaders/loader.css" rel="stylesheet">
<link href="assets/presto.css" rel="stylesheet">
<link rel="stylesheet" href="assets/stylesheets/themes/default.css">
<link rel="stylesheet" href="assets/fonts/font-awesome/css/font-awesome.min.css">
<link rel="stylesheet" href="assets/stylesheets/hetuui.css">
<link rel="stylesheet" href="assets/stylesheets/headerfooter.css"/>
<link rel="stylesheet" href="assets/stylesheets/plugins/rc-pagination.css"/>
<link rel="stylesheet" href="assets/stylesheets/plugins/rc-select.css"/>
<link rel="stylesheet" href="assets/stylesheets/plugins/react-tabs.css">
<!-- Custom CSS -->
<link href="assets/presto.css" rel="stylesheet">
<!-- Date -->
<link href="src/dateRangePicker/daterangepicker.css" rel="stylesheet">
<script src="src/dateRangePicker/moment.js"></script>
<script src="src/dateRangePicker/daterangepicker.js"></script>
</head>
<body>
<div id="audit-log" class="font-white"><div class="loader">Loading...</div></div>
<!-- Fonts -->
<link href="assets/stylesheets/fonts.googleapis.com.css" rel="stylesheet">
<script type="text/javascript" src="dist/auditlog.js"></script>
</body>
</html>

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@ -28,6 +28,9 @@
<!-- Sparkline -->
<script type="text/javascript" src="vendor/jquery.sparkline/jquery.sparkline.min.js"></script>
<!-- alert -->
<script type="text/javascript" src="src/alert.js"></script>
<!-- CSS loader -->
<link href="vendor/css-loaders/loader.css" rel="stylesheet">

View File

@ -7,7 +7,7 @@
<meta name="description" content="Query List - openLooKeng">
<title>Query List - openLooKeng</title>
<!-- <link rel="icon" href="assets/favicon.ico">-->
<!-- <link rel="icon" href="assets/favicon.ico">-->
<!--EMPTY ICON-->
<link rel="icon" href="assets/favicon.ico"/>
<!-- Bootstrap core -->
@ -29,6 +29,9 @@
<!-- Sparkline -->
<script type="text/javascript" src="vendor/jquery.sparkline/jquery.sparkline.min.js"></script>
<!-- alert -->
<script type="text/javascript" src="src/alert.js"></script>
<!-- CSS loader -->
<link href="vendor/css-loaders/loader.css" rel="stylesheet">
<link href="assets/presto.css" rel="stylesheet">
@ -40,6 +43,12 @@
<link rel="stylesheet" href="assets/stylesheets/plugins/rc-select.css"/>
<!-- Custom CSS -->
<link href="assets/presto.css" rel="stylesheet">
<!-- Date -->
<link href="src/dateRangePicker/daterangepicker.css" rel="stylesheet">
<script src="src/dateRangePicker/moment.js"></script>
<script src="src/dateRangePicker/daterangepicker.js"></script>
</head>
<body>
@ -48,10 +57,10 @@
<div id="query-list" class="font-white"><div class="loader">Loading...</div></div>
</div>
<script type="text/javascript" src="dist/index.js"></script>
<!-- Fonts -->
<link href="assets/stylesheets/fonts.googleapis.com.css" rel="stylesheet">
<script type="text/javascript" src="dist/index.js"></script>
</body>
</html>

View File

@ -0,0 +1,57 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="description" content="Query List - openLooKeng">
<title>Query Monitor - openLooKeng</title>
<!-- <link rel="icon" href="assets/favicon.ico">-->
<!--EMPTY ICON-->
<link rel="icon" href="assets/favicon.ico"/>
<!-- Bootstrap core -->
<link href="vendor/bootstrap/css/bootstrap.css" rel="stylesheet">
<!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->
<!-- jQuery -->
<script type="text/javascript" src="vendor/jquery/jquery-2.2.3.min.js"></script>
<!-- Bootstrap JS -->
<script type="text/javascript" src="vendor/bootstrap/js/bootstrap.js"></script>
<!-- Sparkline -->
<script type="text/javascript" src="vendor/jquery.sparkline/jquery.sparkline.min.js"></script>
<!-- CSS loader -->
<link href="vendor/css-loaders/loader.css" rel="stylesheet">
<link href="assets/presto.css" rel="stylesheet">
<link rel="stylesheet" href="assets/stylesheets/themes/default.css">
<link rel="stylesheet" href="assets/fonts/font-awesome/css/font-awesome.min.css">
<link rel="stylesheet" href="assets/stylesheets/hetuui.css">
<link rel="stylesheet" href="assets/stylesheets/headerfooter.css"/>
<link rel="stylesheet" href="assets/stylesheets/plugins/rc-pagination.css"/>
<link rel="stylesheet" href="assets/stylesheets/plugins/rc-select.css"/>
<!-- Custom CSS -->
<link href="assets/presto.css" rel="stylesheet">
</head>
<body>
<div id="query-list-container">
<div id="query-monitor" class="font-white"><div class="loader">Loading...</div></div>
</div>
<script type="text/javascript" src="dist/querymonitor.js"></script>
<!-- Fonts -->
<link href="assets/stylesheets/fonts.googleapis.com.css" rel="stylesheet">
</body>
</html>

View File

@ -42,12 +42,24 @@ export default class NavigationMenu extends React.Component {
<div>Nodes</div>
</a>
</li>
<li className={this.props.active === 'querymonitor' ? "active" : ""}>
<a href={this.props.active === 'querymonitor' ? "#" : "./querymonitor.html"}>
<div><i className="fa fa-desktop"></i></div>
<div>Query Monitor</div>
</a>
</li>
<li className={this.props.active === 'queryhistory' ? "active" : ""}>
<a href={this.props.active === 'queryhistory' ? "#" : "./queryhistory.html"}>
<div><i className="fa fa-history"></i></div>
<div>Query History</div>
</a>
</li>
<li className={this.props.active === 'auditlog' ? "active" : ""}>
<a href={this.props.active === 'auditlog' ? "#" : "./auditlog.html"}>
<div><i className="fa fa-file-text-o"></i></div>
<div>Audit Log</div>
</a>
</li>
</ul>
</div>
);

View File

@ -0,0 +1,82 @@
window.alert = function(msg){
const body = document.getElementsByTagName('body')[0];
var alertMask = document.createElement('div');
var alertContainer = document.createElement('div');
var alertDes = document.createElement('div');
var alertConfirmParent = document.createElement('div');
var alertConfirmBtn = document.createElement('button');
css(alertMask, {
"position" : "fixed", /*生成绝对定位的元素,相对于浏览器窗口进行定位*/
"display": "flex",
"flex-direction": "row",
"align-items": "center",
"justify-content": "center",
"width": "100%",
"height": "100%",
"top": "0",
"left": "0",
"z-index": "99999",
"background": "#0000002b",
})
css(alertContainer, {
"min-width": "500px", /*容器最小240px*/
"max-width": "580px", /*容器最大320px*/
"background":"#fff",
"border": "1px solid #0000002b",
"border-radius": "3px",
"color": "#1f0000",
"overflow": "hidden",
})
css(alertDes, {
"padding": "35px 30px",
"text-align": "center",
"letter-spacing": "1px",
"font-size": "14px",
"color": "#1f0000",
})
css(alertConfirmParent, {
"width": "100%",
"padding": "20px 30px",
"text-align": "right",
"box-sizing": "border-box",
"background": "#f2f2f2",
})
css(alertConfirmBtn, {
"cursor": "pointer",
"padding": "8px 10px",
"border": "none",
"border-radius": "2px",
"color": "#fff",
"background-color": "gray",
"box-shadow": "0 0 2px gray",
})
body.append(alertMask);
alertMask.append(alertContainer);
alertContainer.append(alertDes);
alertContainer.append(alertConfirmParent);
alertConfirmParent.append(alertConfirmBtn);
alertConfirmBtn.innerText = '确定';
//加载提示信息
alertDes.innerHTML = msg;
//关闭当前alert弹窗
function alertBtnClick(){
body.removeChild(alertMask);
}
alertConfirmBtn.addEventListener("click", alertBtnClick);
}
function css(targetObj, cssObj) {
var str = targetObj.getAttribute("style") ? targetObj.getAttribute("style") : "";
for(var i in cssObj) {
str += i + ":" + cssObj[i] + ";";
}
targetObj.style.cssText = str;
}

View File

@ -0,0 +1,9 @@
import React from "react";
import ReactDOM from "react-dom";
import {AuditLog} from "./components/AuditLog";
ReactDOM.render(
<AuditLog />,
document.getElementById('audit-log')
);

View File

@ -0,0 +1,434 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react";
import Header from "../queryeditor/components/Header";
import Footer from "../queryeditor/components/Footer";
import StatusFooter from "../queryeditor/components/StatusFooter";
import NavigationMenu from "../NavigationMenu";
import _ from "lodash";
import TabStore from "../queryeditor/stores/TabStore";
import RunActions from "../queryeditor/actions/RunActions";
import TableStore from "../queryeditor/stores/TableStore";
import AuditLogActions from "../queryeditor/actions/AuditLogActions";
import MultiSelect from 'react-simple-multi-select';
import Select from "rc-select";
import Pagination from "rc-pagination";
import localeInfo from "../node_modules/rc-pagination/es/locale/en_US";
export class AuditLog extends React.Component {
constructor(props) {
super(props);
this.state = {
auditlog:{},
searchStringUser: '',
searchStringDate:'',
searchStringBeginTime:'',
searchStringEndTime:'',
searchStringLogLevel:'INFO',
searchStringType:'Sql',
selectedTab: TabStore.getSelectedTab(),
currentPage: 1,
pageSize: 10,
total: 0,
tableWidth: 400,
tableHeight: 400,
checkStatusLogLevel:{
INFO:true,
WARN:true
},
checkStatusType: {
Sql: true,
WebUI: true,
Cluster: true
},
itemListLogLevel: [
{key: "INFO", value: "INFO"},
{key: "WARN", value: "WARN"}
],
selectedItemListLogLevel: [
{key: "INFO", value: "INFO"},
{key: "WARN", value: "WARN"}
],
itemListType: [
{key: "Sql", value: "Sql"},
{key: "WebUI", value: "WebUi"},
{key: "Cluster", value: "Cluster"}
],
selectedItemListType: [
{key: "Sql", value: "Sql"},
{key: "WebUI", value: "WebUi"},
{key: "Cluster", value: "Cluster"}
],
dateRangePickerPattern: '',
};
this.flag = {
searchStringUser: '',
searchStringBeginTime:'',
searchStringEndTime:'',
searchStringLogLevel:'',
searchStringType:''
}
this.refreshLoop = this.refreshLoop.bind(this);
this.handleSearchStringChange = this.handleSearchStringChange.bind(this);
this.renderDateRangePicker = this.renderDateRangePicker.bind(this);
this.handleSortClick = this.handleSortClick.bind(this);
this.refreshData = this.refreshData.bind(this);
this.onPageChange = this.onPageChange.bind(this);
this.onPageSizeChange = this.onPageSizeChange.bind(this);
this.debounceSearch = _.debounce(() => { _.defer(this.refreshData) }, 1000)
this.update = this.update.bind(this);
this.onChange = this.onChange.bind(this);
this.onResize = this.onResize.bind(this);
this.onTabChange = this.onTabChange.bind(this);
this.renderAuditLog = this.renderAuditLog.bind(this);
this.handleDownload = this.handleDownload.bind(this);
}
resetTimer() {
clearTimeout(this.timeoutId);
// stop refreshing when query finishes or fails
if (this.state.query === null || !this.state.ended) {
this.timeoutId = setTimeout(this.refreshLoop, 1000);
}
}
refreshLoop() {
clearTimeout(this.timeoutId); // to stop multiple series of refreshLoop from going on simultaneously
this.refreshData();
}
refreshData() {
const {searchStringUser,searchStringBeginTime,searchStringEndTime,searchStringLogLevel,searchStringType} = this.state;
if (this.flag.searchStringUser != searchStringUser || this.flag.searchStringBeginTime != searchStringBeginTime || this.flag.searchStringEndTime != searchStringEndTime||
this.flag.searchStringLogLevel != searchStringLogLevel || this.flag.searchStringType != searchStringType) {
AuditLogActions.getAuditLog(searchStringUser, searchStringBeginTime,
searchStringEndTime, searchStringLogLevel, searchStringType).then((data) => {
this.setState({
auditlog: data,
total: data.length
})
this.resetTimer();
})
}
this.flag = {searchStringUser,searchStringBeginTime,searchStringEndTime,searchStringLogLevel,searchStringType};
}
componentDidMount() {
$.get(`../v1/audit/pattern`, function (pattern) {
this.setState({
dateRangePickerPattern: pattern
});
this.resetTimer();
}.bind(this))
.error(function () {
this.resetTimer();
}.bind(this));
this.refreshLoop();
RunActions.connect();
TableStore.listen(this.onChange);
TabStore.listen(this.onTabChange);
this.update();
let win = window;
if (win.addEventListener) {
win.addEventListener('resize', this.onResize, false);
} else if (win.attachEvent) {
win.attachEvent('onresize', this.onResize);
} else {
win.onresize = this.onResize;
}
$(window).on('resize', this.update);
}
componentWillUnmount() {
RunActions.disconnect();
TableStore.unlisten(this.onChange);
TabStore.unlisten(this.onTabChange);
}
update() {
let windowHeight = document.body.clientHeight;
let windowWidth = document.documentElement.clientWidth;
let newWidth = windowWidth - 75 - 270 - 10; //left side size
newWidth = 400 > newWidth ? 400 : newWidth;
let newHeight = windowHeight - ((0.3 * (windowHeight + 50)) + 40 + 57 + 46 + 6); // editor + header + tabs + footer+ extra 6(unknown) heights
newHeight = 400 > newHeight ? 400 : newHeight;
this.setState({
tableWidth: newWidth,
tableHeight: newHeight,
});
}
handleSearchStringChange(event) {
const newSearchString = event.target.value;
if (event.target.name === "User") {
this.setState({
currentPage: 1,
searchStringUser: newSearchString
});
}
else if (event.target.name === "Date"){
this.setState({
currentPage: 1,
searchStringDate: newSearchString
});
}
else if (event.target.name === "LogLevel") {
let obj = document.getElementById('loglevel');
let index = obj.selectedIndex;
this.setState({
currentPage: 1,
searchStringLogLevel: obj.options[index].value
})
}
else if (event.target.name === "AuditType") {
let obj = document.getElementById('audittype');
let index = obj.selectedIndex;
this.setState({
currentPage: 1,
searchStringType: obj.options[index].value
})
}
this.debounceSearch();
}
renderDateRangePicker() {
let that = this;
let pattern = this.state.dateRangePickerPattern;
let flag = true;
let today = new Date();
let tomorrow = new Date();
let todayText = "";
let tomorrowText = "";
if(pattern === "YYYY-MM-DD") {
flag = false;
todayText = today.getFullYear() + '/' + (today.getMonth() + 1) + '/' + today.getDate();
tomorrow.setTime(tomorrow.getTime()+24*60*60*1000);
tomorrowText = tomorrow.getFullYear()+ '/' + (tomorrow.getMonth()+1) + '/' + tomorrow.getDate();
}
else {
todayText = today.getFullYear() + '/' + (today.getMonth() + 1) + '/' + today.getDate() + '.00';
tomorrow.setTime(tomorrow.getTime()+24*60*60*1000);
tomorrowText = tomorrow.getFullYear()+ '/' + (tomorrow.getMonth()+1) + '/' + tomorrow.getDate() + '.00';
}
$('#date-picker').daterangepicker({
startDate:that.state.searchStringBeginTime == "" ? todayText : that.state.searchStringBeginTime,
endDate:that.state.searchStringEndTime == "" ? tomorrowText : that.state.searchStringEndTime,
"timePicker": flag,
"timePicker24Hour": true,
"linkedCalendars": false,
"autoUpdateInput": false,
"timePickerMinutes": false,
"timePickerIncrement": 60,
"locale": {
format: pattern,
separator: ' ~ ',
applyLabel: "apply",
cancelLabel: "clear",
resetLabel: "reset",
}
}, function(start, end, label) {
});
$('#date-picker').on('apply.daterangepicker', function(ev, picker) {
picker.element.val(picker.startDate.format(picker.locale.format) + picker.locale.separator + picker.endDate.format(picker.locale.format));
that.state.searchStringDate = picker.startDate.format(picker.locale.format) + picker.locale.separator + picker.endDate.format(picker.locale.format);
that.state.searchStringBeginTime = picker.startDate.format(picker.locale.format);
that.state.searchStringEndTime = picker.endDate.format(picker.locale.format);
setTimeout(that.debounceSearch,0);
});
$('#date-picker').on('cancel.daterangepicker', function(ev, picker) {
$('#date-picker').val('');
that.state.searchStringBeginTime = "";
that.state.searchStringEndTime = "";
that.state.searchStringDate = "";
setTimeout(that.debounceSearch,0);
});
$('#date-picker').on('hide.daterangepicker', function(ev, picker) {
if(that.state.searchStringDate === "") {
that.state.searchStringBeginTime = "";
that.state.searchStringEndTime = "";
}
else {
that.state.searchStringBeginTime = picker.startDate.format(pattern);
that.state.searchStringEndTime = picker.endDate.format(pattern);
}
setTimeout(that.debounceSearch,0);
});
}
handleDownload() {
const {searchStringUser,searchStringBeginTime,searchStringEndTime,searchStringLogLevel,searchStringType} = this.state;
if (searchStringUser != '' && searchStringBeginTime != '' && searchStringEndTime != '') {
AuditLogActions.downloadAuditLog(searchStringUser, searchStringBeginTime,
searchStringEndTime, searchStringLogLevel, searchStringType);
}
}
handleSortClick(sortType) {
const newSortType = sortType;
let newSortOrder = SORT_ORDER.DESCENDING;
if (this.state.currentSortType === sortType && this.state.currentSortOrder === SORT_ORDER.DESCENDING) {
newSortOrder = SORT_ORDER.ASCENDING;
}
this.setState({
currentPage: 1,
currentSortType: newSortType,
currentSortOrder: newSortOrder
});
_.defer(this.refreshData);
}
renderAuditLog(obj, currentPage, pageSize) {
let arr = [];
let showIndex = (currentPage-1)*pageSize;
for (let i = showIndex; i < showIndex + pageSize; i++) {
arr.push(
<div className="auditlog-row">
{obj[i]}
</div>
)
}
return arr ;
}
onChange() {
const table = TableStore.getActiveTable();
if (!table) return;
if (this.state.dataPreview && table.name === this.state.dataPreview.name) return;
this.setState({
dataPreview: table,
});
// TabActions.selectTab.defer(TabConstants.DATA_PREVIEW);
}
onTabChange() {
const selectedTab = TabStore.getSelectedTab();
this.setState({selectedTab});
}
onResize() {
this.update();
}
onPageChange(current, pageSize) {
this.setState({ currentPage: current });
_.defer(this.refreshData);
}
onPageSizeChange(current, pageSize) {
this.setState({ pageSize: pageSize });
_.defer(this.refreshData);
}
render() {
const {currentPage, total, pageSize } = this.state;
const { auditlog } = this.state;
const {searchStringUser,searchStringBeginTime,searchStringEndTime,searchStringLogLevel,searchStringType} = this.state;
let downloadParam = {
user: searchStringUser,
beginTime: searchStringBeginTime,
endTime: searchStringEndTime,
level: searchStringLogLevel,
type: searchStringType,
}
let downloadArray = [];
_.each(downloadParam, (value, key) => {
downloadArray.push(`${key}=${value}`);
})
let downloadString = downloadArray.join("&&");
return (
<div>
<div className='flex flex-row flex-initial header'>
<Header />
</div>
<div className='flex flex-row content'>
<NavigationMenu active={"auditlog"} />
<div className="container">
<div className="log toolbar-log">
<div className="col-xs-12 toolbar-audit">
<div className="input-group-audit">
<input type="text" className="form-control form-control-auditlog search-bar" placeholder="User" name="User"
onChange={this.handleSearchStringChange} value={this.state.searchStringUser} />
<input type="text" id="date-picker" className="form-control form-control-auditlog search-bar" placeholder="Date" name="Date"
onFocus={this.renderDateRangePicker} onChange={this.handleSearchStringChange} value={this.state.searchStringDate}/>
<select id="loglevel" className="selectpicker" data-style="select" onChange={this.handleSearchStringChange} value={this.state.searchStringLogLevel} name="LogLevel" >
<option>INFO </option>
<option>WARN </option>
</select>
<select id="audittype" className="selectpicker" data-style="select" onChange={this.handleSearchStringChange} value={this.state.searchStringType} name="AuditType" >
<option value="Sql">Sql</option>
<option value="WebUi">WebUI</option>
<option value="Cluster">Cluster</option>
</select>
<a href={"../v1/audit/download?"+downloadString} target="_blank" className="btn"> Download <i className="icon fa fa-download"/> </a>
</div>
</div>
<div className="col-xs-12 auditlog-text">
{JSON.stringify(this.state.auditlog) == "{}" ?
<div className="row error-message">
<div className="col-xs-12"><h4>{"No Audit Log"}</h4></div>
</div>
:
this.renderAuditLog(auditlog,currentPage,pageSize)
}
</div>
</div>
<Pagination
defaultCurrent={1}
current={currentPage}
total={total}
defaultPageSize={20}
pageSize={pageSize}
onChange={this.onPageChange}
showTotal={total => `Total ${total} `}
style={{ marginTop: 10 }}
locale={localeInfo}
showSizeChanger
onShowSizeChange={this.onPageSizeChange}
selectComponentClass={Select}
/>
</div>
</div>
<div className='flex flex-row flex-initial statusFooter'>
<StatusFooter />
</div>
<div className='flex flex-row flex-initial footer'>
<Footer />
</div>
</div>
);
}
}

View File

@ -17,13 +17,12 @@ import React, { Fragment } from "react";
import {
formatDataSizeBytes,
formatShortTime,
getProgressBarPercentage,
getProgressBarTitle,
getHumanReadableState,
getQueryStateColor,
GLYPHICON_DEFAULT,
GLYPHICON_HIGHLIGHT,
truncateString
} from "../utils";
} from "../newUtils";
import Header from "../queryeditor/components/Header";
import Footer from "../queryeditor/components/Footer";
import StatusFooter from "../queryeditor/components/StatusFooter";
@ -32,9 +31,19 @@ import Pagination from 'rc-pagination';
import Select from "rc-select";
import localeInfo from "../node_modules/rc-pagination/es/locale/en_US";
import _ from "lodash";
import ModalDialog from "../queryeditor/components/ModalDialog";
export class QueryListItem extends React.Component {
static stripQueryTextWhitespace(queryText) {
constructor(props) {
super(props);
this.state = {
show: false,
}
this.showModal = this.showModal.bind(this);
this.isShowMore = this.isShowMore.bind(this);
}
static getAllQueryText(queryText) {
const lines = queryText.split("\n");
let minLeadingWhitespace = -1;
for (let i = 0; i < lines.length; i++) {
@ -67,26 +76,51 @@ export class QueryListItem extends React.Component {
}
}
return truncateString(formattedQueryText, 300);
return formattedQueryText;
}
static stripQueryTextWhitespace(queryText) {
let formattedQueryText = QueryListItem.getAllQueryText(queryText);
return truncateString(formattedQueryText, 550);
}
showModal(e) {
let newShow = this.state.show;
this.setState({
show: !newShow
})
}
isShowMore(queryText) {
let formattedQueryText = QueryListItem.getAllQueryText(queryText);
let row = formattedQueryText.split("\n").length;
if (formattedQueryText && (formattedQueryText.length > 550 || row > 8)) {
return true;
}
else {
return false;
}
}
render() {
const query = this.props.query;
const progressBarStyle = { width: getProgressBarPercentage(query) + "%", backgroundColor: getQueryStateColor(query) };
const progressBarStyle = {width: 100 + "%", backgroundColor: getQueryStateColor(query) };
const splitDetails = (
<div className="col-xs-12 tinystat-row">
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="Completed splits" data-container="body">
<span className="glyphicon glyphicon-ok" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{query.queryStats.completedDrivers}
{query.completedDrivers}
</span>
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="Running splits" data-container="body">
<span className="glyphicon glyphicon-play" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{(query.state === "FINISHED" || query.state === "FAILED") ? 0 : query.queryStats.runningDrivers}
{(query.state === "FINISHED" || query.state === "FAILED") ? 0 : query.runningDrivers}
</span>
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="Queued splits" data-container="body">
<span className="glyphicon glyphicon-pause" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{(query.state === "FINISHED" || query.state === "FAILED") ? 0 : query.queryStats.queuedDrivers}
{(query.state === "FINISHED" || query.state === "FAILED") ? 0 : query.queuedDrivers}
</span>
</div>);
@ -94,15 +128,15 @@ export class QueryListItem extends React.Component {
<div className="col-xs-12 tinystat-row">
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="Wall time spent executing the query (not including queued time)" data-container="body">
<span className="glyphicon glyphicon-hourglass" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{query.queryStats.executionTime}
{query.executionTime}
</span>
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="Total query wall time" data-container="body">
<span className="glyphicon glyphicon-time" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{query.queryStats.elapsedTime}
{query.elapsedTime}
</span>
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="CPU time spent by this query" data-container="body">
<span className="glyphicon glyphicon-dashboard" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{query.queryStats.totalCpuTime}
{query.totalCpuTime}
</span>
</div>);
@ -110,24 +144,24 @@ export class QueryListItem extends React.Component {
<div className="col-xs-12 tinystat-row">
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="Current total reserved memory" data-container="body">
<span className="glyphicon glyphicon-scale" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{query.queryStats.totalMemoryReservation}
{query.totalMemoryReservation}
</span>
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="Peak total memory" data-container="body">
<span className="glyphicon glyphicon-fire" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{query.queryStats.peakTotalMemoryReservation}
{query.peakTotalMemoryReservation}
</span>
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="Cumulative user memory" data-container="body">
<span className="glyphicon glyphicon-equalizer" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{formatDataSizeBytes(query.queryStats.cumulativeUserMemory / 1000.0)}
{formatDataSizeBytes(query.cumulativeUserMemory / 1000.0)}
</span>
</div>);
let user = (<span>{query.session.user}</span>);
if (query.session.principal) {
user = (
<span>{query.session.user}<span className="glyphicon glyphicon-lock-inverse" style={GLYPHICON_DEFAULT} /></span>
);
}
let user = (<span>{query.user}</span>);
// if (query.session.principal) {
// user = (
// <span>{query.user}<span className="glyphicon glyphicon-lock-inverse" style={GLYPHICON_DEFAULT} /></span>
// );
// }
return (
<div className="query">
@ -135,10 +169,10 @@ export class QueryListItem extends React.Component {
<div className="col-xs-4">
<div className="row stat-row query-header query-header-queryid">
<div className="col-xs-9" data-toggle="tooltip" data-placement="bottom" title="Query ID">
<a href={"" + query.self.replace(/.*\/v1\/query\//, "query.html?")} target="_blank">{query.queryId}</a>
<a href={"../ui/query.html?history_"+query.queryId} target="_blank">{query.queryId}</a>
</div>
<div className="col-xs-3 query-header-timestamp" data-toggle="tooltip" data-placement="bottom" title="Submit time">
<span>{formatShortTime(new Date(Date.parse(query.queryStats.createTime)))}</span>
<span>{formatShortTime(new Date(Date.parse(query.createTime)))}</span>
</div>
</div>
<div className="row stat-row">
@ -153,7 +187,7 @@ export class QueryListItem extends React.Component {
<div className="col-xs-12">
<span data-toggle="tooltip" data-placement="right" title="Source">
<span className="glyphicon glyphicon-log-in" style={GLYPHICON_DEFAULT} />&nbsp;&nbsp;
<span>{truncateString(query.session.source, 35)}</span>
<span>{truncateString(query.source, 35)}</span>
</span>
</div>
</div>
@ -161,7 +195,7 @@ export class QueryListItem extends React.Component {
<div className="col-xs-12">
<span data-toggle="tooltip" data-placement="right" title="Resource Group">
<span className="glyphicon glyphicon-road" style={GLYPHICON_DEFAULT} />&nbsp;&nbsp;
<span>{truncateString(query.resourceGroupId ? query.resourceGroupId.join(".") : "n/a", 35)}</span>
<span>{truncateString(query.resource ? query.resource : "n/a", 35)}</span>
</span>
</div>
</div>
@ -179,9 +213,26 @@ export class QueryListItem extends React.Component {
<div className="row query-header">
<div className="col-xs-12 query-progress-container">
<div className="progress">
<div className="progress-bar progress-bar-info" role="progressbar" aria-valuenow={getProgressBarPercentage(query)} aria-valuemin="0"
aria-valuemax="100" style={progressBarStyle}>
{getProgressBarTitle(query)}
<div className="progress-bar progress-bar-info" role="progressbar" aria-valuemin="0"
aria-valuemax="100" style={progressBarStyle}>
<div className="progress-catalog-schema-container">
<span className="progress-catalog-schema">
catalog:
</span>
<span className="text-catalog-schema">
{query.catalog}
</span>
&nbsp;&nbsp;
<span className="progress-catalog-schema">
schema:
</span>
<span className="text-catalog-schema">
{query.schemata}
</span>
</div>
<div className="progress-state">
{getHumanReadableState(query)}
</div>
</div>
</div>
</div>
@ -189,6 +240,22 @@ export class QueryListItem extends React.Component {
<div className="row query-row-bottom">
<div className="col-xs-12">
<pre className="query-snippet"><code className="sql">{QueryListItem.stripQueryTextWhitespace(query.query)}</code></pre>
{this.isShowMore(query.query) ?
<div className="query-showmore">
<a onClick={this.showModal} style={{color: '#1E90FF',cursor: 'pointer'}}>
<span className="glyphicon glyphicon-arrow-right" style={{color: '#1E90FF'}}/>
&nbsp;Show More
</a>
<ModalDialog onClose={this.showModal} header={getHumanReadableState(query)} footer={""}
show={this.state.show}>
<div className="modal-querytext">
{QueryListItem.getAllQueryText(query.query)}
</div>
</ModalDialog>
</div>
:
null
}
</div>
</div>
</div>
@ -273,14 +340,22 @@ export class QueryList extends React.Component {
],
stateSelectAll: true,
errorSelectAll: true,
searchString: '',
searchStringUser: '',
searchStringDate:'',
searchStringSql:'',
searchStringQueryId:'',
searchStringResourceGroup:'',
currentPage: 1,
pageSize: 10,
total: 0,
queryString: '',
queryNums: -1,
metaStoreType:''
};
this.refreshLoop = this.refreshLoop.bind(this);
this.handleSearchStringChange = this.handleSearchStringChange.bind(this);
this.renderDateRangePicker = this.renderDateRangePicker.bind(this);
this.handleSortClick = this.handleSortClick.bind(this);
this.refreshData = this.refreshData.bind(this);
this.onPageChange = this.onPageChange.bind(this);
@ -302,14 +377,18 @@ export class QueryList extends React.Component {
}
refreshData() {
const { stateFilters, errorTypeFilters, currentSortType, currentSortOrder, searchString,
currentPage, pageSize } = this.state;
const { stateFilters, errorTypeFilters, currentSortType, currentSortOrder, searchStringUser,searchStringDate,searchStringSql
,searchStringQueryId,searchStringResourceGroup,currentPage, pageSize } = this.state;
let queryParam = {
state: stateFilters.join(","),
failed: errorTypeFilters.join(","),
sort: currentSortType,
sortOrder: currentSortOrder,
search: searchString,
searchUser: searchStringUser,
searchDate:searchStringDate,
searchSql:searchStringSql,
searchQueryId:searchStringQueryId,
searchResourceGroup:searchStringResourceGroup,
pageNum: currentPage,
pageSize: pageSize,
}
@ -318,35 +397,134 @@ export class QueryList extends React.Component {
queryArray.push(`${key}=${value}`);
})
let queryString = queryArray.join("&&");
$.get(`../v1/query/queryNums`, function (num) {
let queryNum = this.state.queryNums;
if(queryNum==num){
$.get(`../v1/query?${queryString}`, function (queryList) {
this.setState({
allQueries: queryList.queries,
total: queryList.total,
});
}
else {
this.state.queryNums = num;
$.get(`../v1/query/queryInfo?${queryString}`, function (queryList) {
this.setState({
allQueries: queryList.queries,
total: queryList.total,
});
this.resetTimer();
}.bind(this))
.error(function () {
this.resetTimer();
}.bind(this));
}
this.resetTimer();
}.bind(this))
if (queryString !== this.state.queryString) {
$.get(`../v1/query/queryInfo?${queryString}`, function (queryList) {
this.setState({
allQueries: queryList.queries,
total: queryList.total,
});
this.resetTimer();
}.bind(this))
.error(function () {
this.resetTimer();
}.bind(this));
}
this.setState({
queryString : queryString
})
}
componentDidMount() {
$.get(`../v1/query/hetuMetastoreType`, function (metaStoreType) {
this.state.metaStoreType = metaStoreType;
if(this.state.metaStoreType != "jdbc") {
alert("Page error. Please change the storage type of hetu-metastore to JDBC.");
}
this.resetTimer();
}.bind(this))
.error(function () {
this.resetTimer();
}.bind(this));
}
componentDidMount() {
this.refreshLoop();
}
handleSearchStringChange(event) {
const newSearchString = event.target.value;
this.setState({
currentPage: 1,
searchString: newSearchString
});
if (event.target.name === "User") {
this.setState({
currentPage: 1,
searchStringUser: newSearchString
});
}
else if (event.target.name === "Date"){
this.setState({
currentPage: 1,
searchStringDate: newSearchString
});
}
else if (event.target.name === "Sql"){
this.setState({
currentPage: 1,
searchStringSql: newSearchString
});
}
else if (event.target.name === "QueryId"){
this.setState({
currentPage: 1,
searchStringQueryId: newSearchString
});
}
else if (event.target.name === "ResourceGroup"){
this.setState({
currentPage: 1,
searchStringResourceGroup: newSearchString
})
}
this.debounceSearch();
}
renderDateRangePicker() {
let that = this;
$('#date-picker').daterangepicker({
"timePicker": true,
"timePicker24Hour": true,
"linkedCalendars": false,
"autoUpdateInput": false,
"timePickerMinutes": true,
"locale": {
format: 'YYYY-MM-DD.HH:mm',
separator: ' ~ ',
applyLabel: "apply",
cancelLabel: "clear",
resetLabel: "reset",
}
}, function(start, end, label) {
});
$('#date-picker').on('apply.daterangepicker', function(ev, picker) {
picker.element.val(picker.startDate.format(picker.locale.format) + picker.locale.separator + picker.endDate.format(picker.locale.format));
that.state.searchStringDate = picker.startDate.format(picker.locale.format) + picker.locale.separator + picker.endDate.format(picker.locale.format);
setTimeout(that.debounceSearch,0);
});
$('#date-picker').on('hide.daterangepicker', function(ev, picker) {
if(that.state.searchStringDate !== "") {
that.state.searchStringDate = picker.startDate.format(picker.locale.format) + picker.locale.separator + picker.endDate.format(picker.locale.format);
}
setTimeout(that.debounceSearch,0);
});
$('#date-picker').on('cancel.daterangepicker', function(ev, picker) {
$('#date-picker').val('');
that.state.searchStringDate = "";
setTimeout(that.debounceSearch,0);
});
}
renderSortListItem(sortType, sortText) {
if (this.state.currentSortType === sortType) {
@ -382,6 +560,7 @@ export class QueryList extends React.Component {
currentSortOrder: newSortOrder
});
_.defer(this.refreshData);
document.getElementById("query-history-sort").setAttribute("class", "input-group-btn open")
}
renderFilterButton(filterType, filterText) {
@ -418,13 +597,13 @@ export class QueryList extends React.Component {
}
stateSelectAll = newFilters.length == Object.keys(FILTER_TYPE).length ? true : false;
}
this.setState({
currentPage: 1,
stateSelectAll,
stateFilters: newFilters,
});
_.defer(this.refreshData);
document.getElementById("query-history-filter").setAttribute("class", "input-group-btn open")
}
renderErrorTypeListItem(errorType, errorTypeText) {
@ -468,6 +647,7 @@ export class QueryList extends React.Component {
errorTypeFilters: newFilters,
});
_.defer(this.refreshData);
document.getElementById("query-history-error").setAttribute("class", "input-group-btn open")
}
onPageChange(current, pageSize) {
@ -482,10 +662,11 @@ export class QueryList extends React.Component {
render() {
const { allQueries, currentPage, total, stateFilters, pageSize } = this.state;
let queryList = <DisplayedQueriesList queries={allQueries} />;
if (allQueries.queries === null || total === 0) {
// let queryList = <DisplayedQueriesList queries={allQueries} />;
let queryList;
if (allQueries == null || total === 0) {
let label = (<div className="loader">Loading...</div>);
if (allQueries.queries === null || total === 0) {
if (allQueries == null || total === 0) {
label = "No queries";
}
queryList = (
@ -494,6 +675,9 @@ export class QueryList extends React.Component {
</div>
);
}
else {
queryList = <DisplayedQueriesList queries={allQueries} />;
}
return (
<div>
@ -504,11 +688,20 @@ export class QueryList extends React.Component {
<NavigationMenu active={"queryhistory"} />
<div className="container">
<div className="row toolbar-row">
<div className="col-xs-12 toolbar-col">
<div className="col-xs-12 toolbar-col query-history">
<div className="input-group input-group-sm">
<input type="text" className="form-control form-control-small search-bar" placeholder="User, source, query ID, resource group, or query text"
onChange={this.handleSearchStringChange} value={this.state.searchString} />
<div className="input-group-btn">
<input type="text" className="form-control form-control-small search-bar" placeholder="User" name="User"
onChange={this.handleSearchStringChange} value={this.state.searchStringUser} />
<input type="text" id="date-picker" className="form-control form-control-small search-bar" placeholder="Date" name="Date"
onFocus={this.renderDateRangePicker} onChange={this.handleSearchStringChange}
value={this.state.searchStringDate}/>
<input type="text" className="form-control form-control-small search-bar" placeholder="Sql" name="Sql"
onChange={this.handleSearchStringChange} value={this.state.searchStringSql} />
<input type="text" className="form-control form-control-small search-bar" placeholder="QueryId" name="QueryId"
onChange={this.handleSearchStringChange} value={this.state.searchStringQueryId} />
<input type="text" className="form-control form-control-small search-bar" placeholder="ResourceGroup" name="ResourceGroup"
onChange={this.handleSearchStringChange} value={this.state.searchStringResourceGroup}/>
<div className="input-group-btn" id="query-history-filter">
<button type="button" className="btn btn-default dropdown-toggle" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
State <span className="caret" />
</button>
@ -527,23 +720,23 @@ export class QueryList extends React.Component {
</div>
&nbsp;
{stateFilters.indexOf(FILTER_TYPE.FAILED) > -1 &&
<Fragment>
<div className="input-group-btn">
<button type="button" className="btn btn-default dropdown-toggle" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
Failed <span className="caret" />
</button>
<ul className="dropdown-menu">
{this.renderErrorTypeListItem(SELECT_ALL, "Select All")}
{this.renderErrorTypeListItem(ERROR_TYPE.INTERNAL_ERROR, "Internal Error")}
{this.renderErrorTypeListItem(ERROR_TYPE.EXTERNAL, "External Error")}
{this.renderErrorTypeListItem(ERROR_TYPE.INSUFFICIENT_RESOURCES, "Resources Error")}
{this.renderErrorTypeListItem(ERROR_TYPE.USER_ERROR, "User Error")}
</ul>
</div>
&nbsp;
</Fragment>
<Fragment>
<div className="input-group-btn" id="query-history-error">
<button type="button" className="btn btn-default dropdown-toggle" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
Failed <span className="caret" />
</button>
<ul className="dropdown-menu">
{this.renderErrorTypeListItem(SELECT_ALL, "Select All")}
{this.renderErrorTypeListItem(ERROR_TYPE.INTERNAL_ERROR, "Internal Error")}
{this.renderErrorTypeListItem(ERROR_TYPE.EXTERNAL, "External Error")}
{this.renderErrorTypeListItem(ERROR_TYPE.INSUFFICIENT_RESOURCES, "Resources Error")}
{this.renderErrorTypeListItem(ERROR_TYPE.USER_ERROR, "User Error")}
</ul>
</div>
&nbsp;
</Fragment>
}
<div className="input-group-btn">
<div className="input-group-btn" id="query-history-sort">
<button type="button" className="btn btn-default dropdown-toggle" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
Sort <span className="caret" />
</button>
@ -560,22 +753,20 @@ export class QueryList extends React.Component {
</div>
</div>
{queryList}
{allQueries.length > 0 &&
<Pagination
defaultCurrent={1}
current={currentPage}
total={total}
defaultPageSize={10}
pageSize={pageSize}
onChange={this.onPageChange}
showTotal={total => `Total ${total} queries`}
style={{ marginTop: 10 }}
locale={localeInfo}
showSizeChanger
onShowSizeChange={this.onPageSizeChange}
selectComponentClass={Select}
/>
}
<Pagination
defaultCurrent={1}
current={currentPage}
total={total}
defaultPageSize={10}
pageSize={pageSize}
onChange={this.onPageChange}
showTotal={total => `Total ${total} queries`}
style={{ marginTop: 10 }}
locale={localeInfo}
showSizeChanger
onShowSizeChange={this.onPageSizeChange}
selectComponentClass={Select}
/>
</div>
</div>
<div className='flex flex-row flex-initial statusFooter'>
@ -586,6 +777,7 @@ export class QueryList extends React.Component {
</div>
</div>
);
}
}

View File

@ -0,0 +1,644 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React, { Fragment } from "react";
import {
formatDataSizeBytes,
formatShortTime,
getProgressBarPercentage,
getProgressBarTitle,
getQueryStateColor,
GLYPHICON_DEFAULT,
GLYPHICON_HIGHLIGHT,
truncateString
} from "../utils";
import Header from "../queryeditor/components/Header";
import Footer from "../queryeditor/components/Footer";
import StatusFooter from "../queryeditor/components/StatusFooter";
import NavigationMenu from "../NavigationMenu";
import Pagination from 'rc-pagination';
import Select from "rc-select";
import localeInfo from "../node_modules/rc-pagination/es/locale/en_US";
import _ from "lodash";
import ModalDialog from "../queryeditor/components/ModalDialog";
import {getHumanReadableState} from "../newUtils";
export class QueryListItem extends React.Component {
constructor(props) {
super(props);
this.state = {
show: false,
}
this.showModal = this.showModal.bind(this);
this.isShowMore = this.isShowMore.bind(this);
}
static getAllQueryText(queryText) {
const lines = queryText.split("\n");
let minLeadingWhitespace = -1;
for (let i = 0; i < lines.length; i++) {
if (minLeadingWhitespace === 0) {
break;
}
if (lines[i].trim().length === 0) {
continue;
}
const leadingWhitespace = lines[i].search(/\S/);
if (leadingWhitespace > -1 && ((leadingWhitespace < minLeadingWhitespace) || minLeadingWhitespace === -1)) {
minLeadingWhitespace = leadingWhitespace;
}
}
let formattedQueryText = "";
for (let i = 0; i < lines.length; i++) {
const trimmedLine = lines[i].substring(minLeadingWhitespace).replace(/\s+$/g, '');
if (trimmedLine.length > 0) {
formattedQueryText += trimmedLine;
if (i < (lines.length - 1)) {
formattedQueryText += "\n";
}
}
}
return formattedQueryText;
}
static stripQueryTextWhitespace(queryText) {
let formattedQueryText = QueryListItem.getAllQueryText(queryText);
return truncateString(formattedQueryText, 550);
}
showModal(e) {
let newShow = this.state.show;
this.setState({
show: !newShow
})
}
isShowMore(queryText) {
let formattedQueryText = QueryListItem.getAllQueryText(queryText);
let row = formattedQueryText.split("\n").length;
if (formattedQueryText && (formattedQueryText.length > 550 || row > 8)) {
return true;
}
else {
return false;
}
}
render() {
const query = this.props.query;
const progressBarStyle = { width: getProgressBarPercentage(query) + "%", backgroundColor: getQueryStateColor(query) };
const splitDetails = (
<div className="col-xs-12 tinystat-row">
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="Completed splits" data-container="body">
<span className="glyphicon glyphicon-ok" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{query.queryStats.completedDrivers}
</span>
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="Running splits" data-container="body">
<span className="glyphicon glyphicon-play" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{(query.state === "FINISHED" || query.state === "FAILED") ? 0 : query.queryStats.runningDrivers}
</span>
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="Queued splits" data-container="body">
<span className="glyphicon glyphicon-pause" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{(query.state === "FINISHED" || query.state === "FAILED") ? 0 : query.queryStats.queuedDrivers}
</span>
</div>);
const timingDetails = (
<div className="col-xs-12 tinystat-row">
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="Wall time spent executing the query (not including queued time)" data-container="body">
<span className="glyphicon glyphicon-hourglass" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{query.queryStats.executionTime}
</span>
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="Total query wall time" data-container="body">
<span className="glyphicon glyphicon-time" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{query.queryStats.elapsedTime}
</span>
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="CPU time spent by this query" data-container="body">
<span className="glyphicon glyphicon-dashboard" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{query.queryStats.totalCpuTime}
</span>
</div>);
const memoryDetails = (
<div className="col-xs-12 tinystat-row">
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="Current total reserved memory" data-container="body">
<span className="glyphicon glyphicon-scale" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{query.queryStats.totalMemoryReservation}
</span>
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="Peak total memory" data-container="body">
<span className="glyphicon glyphicon-fire" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{query.queryStats.peakTotalMemoryReservation}
</span>
<span className="tinystat" data-toggle="tooltip" data-placement="top" title="Cumulative user memory" data-container="body">
<span className="glyphicon glyphicon-equalizer" style={GLYPHICON_HIGHLIGHT} />&nbsp;&nbsp;
{formatDataSizeBytes(query.queryStats.cumulativeUserMemory / 1000.0)}
</span>
</div>);
let user = (<span>{query.session.user}</span>);
if (query.session.principal) {
user = (
<span>{query.session.user}<span className="glyphicon glyphicon-lock-inverse" style={GLYPHICON_DEFAULT} /></span>
);
}
return (
<div className="query">
<div className="row">
<div className="col-xs-4">
<div className="row stat-row query-header query-header-queryid">
<div className="col-xs-9" data-toggle="tooltip" data-placement="bottom" title="Query ID">
<a href={"" + query.self.replace(/.*\/v1\/query\//, "query.html?")} target="_blank">{query.queryId}</a>
</div>
<div className="col-xs-3 query-header-timestamp" data-toggle="tooltip" data-placement="bottom" title="Submit time">
<span>{formatShortTime(new Date(Date.parse(query.queryStats.createTime)))}</span>
</div>
</div>
<div className="row stat-row">
<div className="col-xs-12">
<span data-toggle="tooltip" data-placement="right" title="User">
<span className="glyphicon glyphicon-user" style={GLYPHICON_DEFAULT} />&nbsp;&nbsp;
<span>{truncateString(user, 35)}</span>
</span>
</div>
</div>
<div className="row stat-row">
<div className="col-xs-12">
<span data-toggle="tooltip" data-placement="right" title="Source">
<span className="glyphicon glyphicon-log-in" style={GLYPHICON_DEFAULT} />&nbsp;&nbsp;
<span>{truncateString(query.session.source, 35)}</span>
</span>
</div>
</div>
<div className="row stat-row">
<div className="col-xs-12">
<span data-toggle="tooltip" data-placement="right" title="Resource Group">
<span className="glyphicon glyphicon-road" style={GLYPHICON_DEFAULT} />&nbsp;&nbsp;
<span>{truncateString(query.resourceGroupId ? query.resourceGroupId.join(".") : "n/a", 35)}</span>
</span>
</div>
</div>
<div className="row stat-row">
{splitDetails}
</div>
<div className="row stat-row">
{timingDetails}
</div>
<div className="row stat-row">
{memoryDetails}
</div>
</div>
<div className="col-xs-8">
<div className="row query-header">
<div className="col-xs-12 query-progress-container">
<div className="progress">
<div className="progress-bar progress-bar-info" role="progressbar" aria-valuenow={getProgressBarPercentage(query)} aria-valuemin="0"
aria-valuemax="100" style={progressBarStyle}>
{getProgressBarTitle(query)}
</div>
</div>
</div>
</div>
<div className="row query-row-bottom">
<div className="col-xs-12">
<pre className="query-snippet"><code className="sql">{QueryListItem.stripQueryTextWhitespace(query.query)}</code></pre>
{this.isShowMore(query.query) ?
<div className="query-showmore">
<a onClick={this.showModal} style={{color: '#1E90FF',cursor: 'pointer'}}>
<span className="glyphicon glyphicon-arrow-right" style={{color: '#1E90FF'}}/>
&nbsp;Show More
</a>
<ModalDialog onClose={this.showModal} header={getHumanReadableState(query)} footer={""}
show={this.state.show}>
<div className="modal-querytext">
{QueryListItem.getAllQueryText(query.query)}
</div>
</ModalDialog>
</div>
:
null
}
</div>
</div>
</div>
</div>
</div>
);
}
}
class DisplayedQueriesList extends React.Component {
render() {
const queryNodes = this.props.queries.map(function (query) {
return (
<QueryListItem key={query.queryId} query={query} />
);
}.bind(this));
return (
<div className="queryListContainer">
{queryNodes}
</div>
);
}
}
const SELECT_ALL = "all";
const FILTER_TYPE = {
QUEUED: "queued",
WAITING_FOR_RESOURCES: "waiting_for_resources",
DISPATCHING: "dispatching",
PLANNING: "planning",
STARTING: "starting",
RUNNING: "running",
FINISHING: "finishing",
FINISHED: "finished",
FAILED: "failed",
};
const SORT_TYPE = {
CREATED: "creation",
ELAPSED: "elapsed",
EXECUTION: "execution",
CPU: "cpu",
CUMULATIVE_MEMORY: "cumulative",
CURRENT_MEMORY: "memory",
};
const ERROR_TYPE = {
USER_ERROR: "user_error",
INTERNAL_ERROR: "internal_error",
INSUFFICIENT_RESOURCES: "insufficient_resources",
EXTERNAL: "external",
};
const SORT_ORDER = {
ASCENDING: "ascending",
DESCENDING: "descending",
};
export class QueryMonitor extends React.Component {
constructor(props) {
super(props);
this.state = {
allQueries: [],
currentSortType: SORT_TYPE.CREATED,
currentSortOrder: SORT_ORDER.DESCENDING,
stateFilters: [
FILTER_TYPE.QUEUED,
FILTER_TYPE.WAITING_FOR_RESOURCES,
FILTER_TYPE.DISPATCHING,
FILTER_TYPE.PLANNING,
FILTER_TYPE.STARTING,
FILTER_TYPE.RUNNING,
FILTER_TYPE.FINISHING,
FILTER_TYPE.FINISHED,
FILTER_TYPE.FAILED],
errorTypeFilters: [
ERROR_TYPE.INTERNAL_ERROR,
ERROR_TYPE.INSUFFICIENT_RESOURCES,
ERROR_TYPE.EXTERNAL,
ERROR_TYPE.USER_ERROR,
],
stateSelectAll: true,
errorSelectAll: true,
searchString: '',
currentPage: 1,
pageSize: 10,
total: 0,
};
this.refreshLoop = this.refreshLoop.bind(this);
this.handleSearchStringChange = this.handleSearchStringChange.bind(this);
this.handleSortClick = this.handleSortClick.bind(this);
this.refreshData = this.refreshData.bind(this);
this.onPageChange = this.onPageChange.bind(this);
this.onPageSizeChange = this.onPageSizeChange.bind(this);
this.debounceSearch = _.debounce(() => { _.defer(this.refreshData) }, 200)
}
resetTimer() {
clearTimeout(this.timeoutId);
// stop refreshing when query finishes or fails
if (this.state.query === null || !this.state.ended) {
this.timeoutId = setTimeout(this.refreshLoop, 1000);
}
}
refreshLoop() {
clearTimeout(this.timeoutId); // to stop multiple series of refreshLoop from going on simultaneously
this.refreshData();
}
refreshData() {
const { stateFilters, errorTypeFilters, currentSortType, currentSortOrder, searchString,
currentPage, pageSize } = this.state;
let queryParam = {
state: stateFilters.join(","),
failed: errorTypeFilters.join(","),
sort: currentSortType,
sortOrder: currentSortOrder,
search: searchString,
pageNum: currentPage,
pageSize: pageSize,
}
let queryArray = [];
_.each(queryParam, (value, key) => {
queryArray.push(`${key}=${value}`);
})
let queryString = queryArray.join("&&");
$.get(`../v1/query?${queryString}`, function (queryList) {
this.setState({
allQueries: queryList.queries,
total: queryList.total,
});
this.resetTimer();
}.bind(this))
.error(function () {
this.resetTimer();
}.bind(this));
}
componentDidMount() {
this.refreshLoop();
}
handleSearchStringChange(event) {
const newSearchString = event.target.value;
this.setState({
currentPage: 1,
searchString: newSearchString
});
this.debounceSearch();
}
renderSortListItem(sortType, sortText) {
if (this.state.currentSortType === sortType) {
const directionArrow = this.state.currentSortOrder === SORT_ORDER.ASCENDING ? <span className="glyphicon glyphicon-triangle-top" /> :
<span className="glyphicon glyphicon-triangle-bottom" />;
return (
<li>
<a href="#" className="selected" onClick={this.handleSortClick.bind(this, sortType)}>
{sortText} {directionArrow}
</a>
</li>);
}
else {
return (
<li>
<a href="#" onClick={this.handleSortClick.bind(this, sortType)}>
{sortText}
</a>
</li>);
}
}
handleSortClick(sortType) {
const newSortType = sortType;
let newSortOrder = SORT_ORDER.DESCENDING;
if (this.state.currentSortType === sortType && this.state.currentSortOrder === SORT_ORDER.DESCENDING) {
newSortOrder = SORT_ORDER.ASCENDING;
}
this.setState({
currentPage: 1,
currentSortType: newSortType,
currentSortOrder: newSortOrder
});
_.defer(this.refreshData);
}
renderFilterButton(filterType, filterText) {
const { stateSelectAll } = this.state;
let checkmarkStyle = { color: '#ffffff' };
if (this.state.stateFilters.indexOf(filterType) > -1 ||
(filterType == SELECT_ALL && stateSelectAll)) {
checkmarkStyle = GLYPHICON_HIGHLIGHT;
}
return (
<li>
<a href="#" onClick={this.handleStateFilterClick.bind(this, filterType)}>
<span className="glyphicon glyphicon-ok" style={checkmarkStyle} />
&nbsp;{filterText}
</a>
</li>);
}
handleStateFilterClick(filter) {
let newFilters = [];
let { stateSelectAll } = this.state;
if (filter == SELECT_ALL) {
!stateSelectAll && _.each(FILTER_TYPE, (value, key) => {
newFilters.push(value);
})
stateSelectAll = !stateSelectAll;
} else {
newFilters = this.state.stateFilters.slice();
if (this.state.stateFilters.indexOf(filter) > -1) {
newFilters.splice(newFilters.indexOf(filter), 1);
}
else {
newFilters.push(filter);
}
stateSelectAll = newFilters.length == Object.keys(FILTER_TYPE).length ? true : false;
}
this.setState({
currentPage: 1,
stateSelectAll,
stateFilters: newFilters,
});
_.defer(this.refreshData);
document.getElementById("query-monitor-filter").setAttribute("class", "input-group-btn open")
}
renderErrorTypeListItem(errorType, errorTypeText) {
const { errorSelectAll } = this.state;
let checkmarkStyle = { color: '#ffffff' };
if (this.state.errorTypeFilters.indexOf(errorType) > -1 ||
(errorType == SELECT_ALL && errorSelectAll)) {
checkmarkStyle = GLYPHICON_HIGHLIGHT;
}
return (
<li>
<a href="#" onClick={this.handleErrorTypeFilterClick.bind(this, errorType)}>
<span className="glyphicon glyphicon-ok" style={checkmarkStyle} />
&nbsp;{errorTypeText}
</a>
</li>);
}
handleErrorTypeFilterClick(errorType) {
let newFilters = [];
let { errorSelectAll } = this.state;
if (errorType == SELECT_ALL) {
!errorSelectAll && _.each(ERROR_TYPE, (value, key) => {
newFilters.push(value);
})
errorSelectAll = !errorSelectAll;
} else {
newFilters = this.state.errorTypeFilters.slice();
if (this.state.errorTypeFilters.indexOf(errorType) > -1) {
newFilters.splice(newFilters.indexOf(errorType), 1);
}
else {
newFilters.push(errorType);
}
errorSelectAll = newFilters.length == Object.keys(ERROR_TYPE).length ? true : false;
}
this.setState({
currentPage: 1,
errorSelectAll,
errorTypeFilters: newFilters,
});
_.defer(this.refreshData);
document.getElementById("query-monitor-error").setAttribute("class", "input-group-btn open")
}
onPageChange(current, pageSize) {
this.setState({ currentPage: current });
_.defer(this.refreshData);
}
onPageSizeChange(current, pageSize) {
this.setState({ pageSize: pageSize });
_.defer(this.refreshData);
}
render() {
const { allQueries, currentPage, total, stateFilters, pageSize } = this.state;
let queryList = <DisplayedQueriesList queries={allQueries} />;
if (allQueries.queries === null || total === 0) {
let label = (<div className="loader">Loading...</div>);
if (allQueries.queries === null || total === 0) {
label = "No queries";
}
queryList = (
<div className="row error-message">
<div className="col-xs-12"><h4>{label}</h4></div>
</div>
);
}
return (
<div>
<div className='flex flex-row flex-initial header'>
<Header />
</div>
<div className='flex flex-row content'>
<NavigationMenu active={"querymonitor"} />
<div className="container">
<div className="row toolbar-row">
<div className="col-xs-12 toolbar-col query-monitor">
<div className="input-group input-group-sm">
<input type="text" className="form-control form-control-small search-bar" placeholder="User, source, query ID, resource group, or query text"
onChange={this.handleSearchStringChange} value={this.state.searchString} />
<div className="input-group-btn" id="query-monitor-filter">
<button type="button" className="btn btn-default dropdown-toggle" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
State <span className="caret" />
</button>
<ul className="dropdown-menu dropdown-menu-left">
{this.renderFilterButton(SELECT_ALL, "Select All")}
{this.renderFilterButton(FILTER_TYPE.QUEUED, "Queued")}
{this.renderFilterButton(FILTER_TYPE.WAITING_FOR_RESOURCES, "Waiting For Resources")}
{this.renderFilterButton(FILTER_TYPE.DISPATCHING, "Dispatching")}
{this.renderFilterButton(FILTER_TYPE.PLANNING, "Planning")}
{this.renderFilterButton(FILTER_TYPE.STARTING, "Starting")}
{this.renderFilterButton(FILTER_TYPE.RUNNING, "Running")}
{this.renderFilterButton(FILTER_TYPE.FINISHING, "Finishing")}
{this.renderFilterButton(FILTER_TYPE.FINISHED, "Finished")}
{this.renderFilterButton(FILTER_TYPE.FAILED, "Failed")}
</ul>
</div>
&nbsp;
{stateFilters.indexOf(FILTER_TYPE.FAILED) > -1 &&
<Fragment>
<div className="input-group-btn" id="query-monitor-error">
<button type="button" className="btn btn-default dropdown-toggle" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
Failed <span className="caret" />
</button>
<ul className="dropdown-menu">
{this.renderErrorTypeListItem(SELECT_ALL, "Select All")}
{this.renderErrorTypeListItem(ERROR_TYPE.INTERNAL_ERROR, "Internal Error")}
{this.renderErrorTypeListItem(ERROR_TYPE.EXTERNAL, "External Error")}
{this.renderErrorTypeListItem(ERROR_TYPE.INSUFFICIENT_RESOURCES, "Resources Error")}
{this.renderErrorTypeListItem(ERROR_TYPE.USER_ERROR, "User Error")}
</ul>
</div>
&nbsp;
</Fragment>
}
<div className="input-group-btn">
<button type="button" className="btn btn-default dropdown-toggle" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
Sort <span className="caret" />
</button>
<ul className="dropdown-menu dropdown-menu-right">
{this.renderSortListItem(SORT_TYPE.CREATED, "Creation Time")}
{this.renderSortListItem(SORT_TYPE.ELAPSED, "Elapsed Time")}
{this.renderSortListItem(SORT_TYPE.CPU, "CPU Time")}
{this.renderSortListItem(SORT_TYPE.EXECUTION, "Execution Time")}
{this.renderSortListItem(SORT_TYPE.CURRENT_MEMORY, "Current Memory")}
{this.renderSortListItem(SORT_TYPE.CUMULATIVE_MEMORY, "Cumulative User Memory")}
</ul>
</div>
</div>
</div>
</div>
{queryList}
{allQueries.length > 0 &&
<Pagination
defaultCurrent={1}
current={currentPage}
total={total}
defaultPageSize={10}
pageSize={pageSize}
onChange={this.onPageChange}
showTotal={total => `Total ${total} queries`}
style={{ marginTop: 10 }}
locale={localeInfo}
showSizeChanger
onShowSizeChange={this.onPageSizeChange}
selectComponentClass={Select}
/>
}
</div>
</div>
<div className='flex flex-row flex-initial statusFooter'>
<StatusFooter />
</div>
<div className='flex flex-row flex-initial footer'>
<Footer />
</div>
</div>
);
}
}

View File

@ -0,0 +1,269 @@
.daterangepicker {
position: absolute;
color: inherit;
background-color: #fff;
border-radius: 4px;
width: 278px;
padding: 4px;
margin-top: 1px;
top: 100px;
left: 20px;
/* Calendars */ }
.daterangepicker:before, .daterangepicker:after {
position: absolute;
display: inline-block;
border-bottom-color: rgba(0, 0, 0, 0.2);
content: ''; }
.daterangepicker:before {
top: -7px;
border-right: 7px solid transparent;
border-left: 7px solid transparent;
border-bottom: 7px solid #ccc; }
.daterangepicker:after {
top: -6px;
border-right: 6px solid transparent;
border-bottom: 6px solid #fff;
border-left: 6px solid transparent; }
.daterangepicker.opensleft:before {
right: 9px; }
.daterangepicker.opensleft:after {
right: 10px; }
.daterangepicker.openscenter:before {
left: 0;
right: 0;
width: 0;
margin-left: auto;
margin-right: auto; }
.daterangepicker.openscenter:after {
left: 0;
right: 0;
width: 0;
margin-left: auto;
margin-right: auto; }
.daterangepicker.opensright:before {
left: 9px; }
.daterangepicker.opensright:after {
left: 10px; }
.daterangepicker.dropup {
margin-top: -5px; }
.daterangepicker.dropup:before {
top: initial;
bottom: -7px;
border-bottom: initial;
border-top: 7px solid #ccc; }
.daterangepicker.dropup:after {
top: initial;
bottom: -6px;
border-bottom: initial;
border-top: 6px solid #fff; }
.daterangepicker.dropdown-menu {
max-width: none;
z-index: 3001; }
.daterangepicker.single .ranges, .daterangepicker.single .calendar {
float: none; }
.daterangepicker.show-calendar .calendar {
display: block; }
.daterangepicker .calendar {
display: none;
max-width: 270px;
margin: 4px; }
.daterangepicker .calendar.single .calendar-table {
border: none; }
.daterangepicker .calendar th, .daterangepicker .calendar td {
white-space: nowrap;
text-align: center;
min-width: 32px; }
.daterangepicker .calendar-table {
border: 1px solid #fff;
padding: 4px;
border-radius: 4px;
background-color: #fff; }
.daterangepicker table {
width: 100%;
margin: 0; }
.daterangepicker td, .daterangepicker th {
text-align: center;
width: 20px;
height: 20px;
border-radius: 4px;
border: 1px solid transparent;
white-space: nowrap;
cursor: pointer; }
.daterangepicker td.available:hover, .daterangepicker th.available:hover {
background-color: #eee;
border-color: transparent;
color: inherit; }
.daterangepicker td.week, .daterangepicker th.week {
font-size: 80%;
color: #ccc; }
.daterangepicker td.off, .daterangepicker td.off.in-range, .daterangepicker td.off.start-date, .daterangepicker td.off.end-date {
background-color: #fff;
border-color: transparent;
color: #999; }
.daterangepicker td.in-range {
background-color: #ebf4f8;
border-color: transparent;
color: #000;
border-radius: 0; }
.daterangepicker td.start-date {
border-radius: 4px 0 0 4px; }
.daterangepicker td.end-date {
border-radius: 0 4px 4px 0; }
.daterangepicker td.start-date.end-date {
border-radius: 4px; }
.daterangepicker td.active, .daterangepicker td.active:hover {
background-color: #357ebd;
border-color: transparent;
color: #fff; }
.daterangepicker th.month {
width: auto; }
.daterangepicker td.disabled, .daterangepicker option.disabled {
color: #999;
cursor: not-allowed;
text-decoration: line-through; }
.daterangepicker select.monthselect, .daterangepicker select.yearselect {
font-size: 12px;
padding: 1px;
height: auto;
margin: 0;
cursor: default; }
.daterangepicker select.monthselect {
margin-right: 2%;
width: 56%; }
.daterangepicker select.yearselect {
width: 40%; }
.daterangepicker select.hourselect, .daterangepicker select.minuteselect, .daterangepicker select.secondselect, .daterangepicker select.ampmselect {
width: 50px;
margin-bottom: 0; }
.daterangepicker .input-mini {
border: 1px solid #ccc;
border-radius: 4px;
color: #555;
height: 30px;
line-height: 30px;
display: block;
vertical-align: middle;
margin: 0 0 5px 0;
padding: 0 6px 0 28px;
width: 100%; }
.daterangepicker .input-mini.active {
border: 1px solid #08c;
border-radius: 4px; }
.daterangepicker .daterangepicker_input {
position: relative; }
.daterangepicker .daterangepicker_input i {
position: absolute;
left: 8px;
top: 8px; }
.daterangepicker.rtl .input-mini {
padding-right: 28px;
padding-left: 6px; }
.daterangepicker.rtl .daterangepicker_input i {
left: auto;
right: 8px; }
.daterangepicker .calendar-time {
text-align: center;
margin: 5px auto;
line-height: 30px;
position: relative;
padding-left: 28px; }
.daterangepicker .calendar-time select.disabled {
color: #ccc;
cursor: not-allowed; }
.ranges {
font-size: 11px;
float: none;
margin: 4px;
text-align: left; }
.ranges ul {
list-style: none;
margin: 0 auto;
padding: 0;
width: 100%; }
.ranges li {
font-size: 13px;
background-color: #f5f5f5;
border: 1px solid #f5f5f5;
border-radius: 4px;
color: #08c;
padding: 3px 12px;
margin-bottom: 8px;
cursor: pointer; }
.ranges li:hover {
background-color: #08c;
border: 1px solid #08c;
color: #fff; }
.ranges li.active {
background-color: #08c;
border: 1px solid #08c;
color: #fff; }
/* Larger Screen Styling */
@media (min-width: 564px) {
.daterangepicker {
width: auto; }
.daterangepicker .ranges ul {
width: 160px; }
.daterangepicker.single .ranges ul {
width: 100%; }
.daterangepicker.single .calendar.left {
clear: none; }
.daterangepicker.single.ltr .ranges, .daterangepicker.single.ltr .calendar {
float: left; }
.daterangepicker.single.rtl .ranges, .daterangepicker.single.rtl .calendar {
float: right; }
.daterangepicker.ltr {
direction: ltr;
text-align: left; }
.daterangepicker.ltr .calendar.left {
clear: left;
margin-right: 0; }
.daterangepicker.ltr .calendar.left .calendar-table {
border-right: none;
border-top-right-radius: 0;
border-bottom-right-radius: 0; }
.daterangepicker.ltr .calendar.right {
margin-left: 0; }
.daterangepicker.ltr .calendar.right .calendar-table {
border-left: none;
border-top-left-radius: 0;
border-bottom-left-radius: 0; }
.daterangepicker.ltr .left .daterangepicker_input {
padding-right: 12px; }
.daterangepicker.ltr .calendar.left .calendar-table {
padding-right: 12px; }
.daterangepicker.ltr .ranges, .daterangepicker.ltr .calendar {
float: left; }
.daterangepicker.rtl {
direction: rtl;
text-align: right; }
.daterangepicker.rtl .calendar.left {
clear: right;
margin-left: 0; }
.daterangepicker.rtl .calendar.left .calendar-table {
border-left: none;
border-top-left-radius: 0;
border-bottom-left-radius: 0; }
.daterangepicker.rtl .calendar.right {
margin-right: 0; }
.daterangepicker.rtl .calendar.right .calendar-table {
border-right: none;
border-top-right-radius: 0;
border-bottom-right-radius: 0; }
.daterangepicker.rtl .left .daterangepicker_input {
padding-left: 12px; }
.daterangepicker.rtl .calendar.left .calendar-table {
padding-left: 12px; }
.daterangepicker.rtl .ranges, .daterangepicker.rtl .calendar {
text-align: right;
float: right; } }
@media (min-width: 730px) {
.daterangepicker .ranges {
width: auto; }
.daterangepicker.ltr .ranges {
float: left; }
.daterangepicker.rtl .ranges {
float: right; }
.daterangepicker .calendar.left {
clear: none !important; } }

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,485 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
//@flow
import * as dagreD3 from "dagre-d3";
import * as d3 from "d3";
// Query display
// =============
export const GLYPHICON_DEFAULT = {color: '#0e5201'};
export const GLYPHICON_HIGHLIGHT = {color: '#999999'};
const STATE_COLOR_MAP = {
QUEUED: '#1b8f72',
RUNNING: '#19874e',
PLANNING: '#674f98',
FINISHED: '#678975',
BLOCKED: '#61003b',
USER_ERROR: '#9a7d66',
CANCELED: '#858959',
INSUFFICIENT_RESOURCES: '#7f5b72',
EXTERNAL_ERROR: '#ca7640',
UNKNOWN_ERROR: '#943524'
};
export function getQueryStateColor(query: any): string
{
switch (query.state) {
case "QUEUED":
return STATE_COLOR_MAP.QUEUED;
case "PLANNING":
return STATE_COLOR_MAP.PLANNING;
case "STARTING":
case "FINISHING":
case "RUNNING":
if (query.queryStats && query.queryStats.fullyBlocked) {
return STATE_COLOR_MAP.BLOCKED;
}
return STATE_COLOR_MAP.RUNNING;
case "FAILED":
switch (query.failed) {
case "USER_ERROR":
return STATE_COLOR_MAP.USER_ERROR;
case "EXTERNAL":
return STATE_COLOR_MAP.EXTERNAL_ERROR;
case "INSUFFICIENT_RESOURCES":
return STATE_COLOR_MAP.INSUFFICIENT_RESOURCES;
default:
return STATE_COLOR_MAP.UNKNOWN_ERROR;
}
case "FINISHED":
return STATE_COLOR_MAP.FINISHED;
default:
return STATE_COLOR_MAP.QUEUED;
}
}
export function getStageStateColor(stage: any): string
{
switch (stage.state) {
case "PLANNED":
return STATE_COLOR_MAP.QUEUED;
case "SCHEDULING":
case "SCHEDULING_SPLITS":
case "SCHEDULED":
return STATE_COLOR_MAP.PLANNING;
case "RUNNING":
if (stage.stageStats && stage.stageStats.fullyBlocked) {
return STATE_COLOR_MAP.BLOCKED;
}
return STATE_COLOR_MAP.RUNNING;
case "FINISHED":
return STATE_COLOR_MAP.FINISHED;
case "CANCELED":
case "ABORTED":
return STATE_COLOR_MAP.CANCELED;
case "FAILED":
return STATE_COLOR_MAP.UNKNOWN_ERROR;
default:
return "#b5b5b5"
}
}
// This relies on the fact that BasicQueryInfo and QueryInfo have all the fields
// necessary to compute this string, and that these fields are consistently named.
export function getHumanReadableState(query: any): string
{
if (query.state === "RUNNING") {
let title = "RUNNING";
if (query.scheduled && query.queryStats.totalDrivers > 0 && query.queryStats.runningDrivers >= 0) {
if (query.queryStats.fullyBlocked) {
title = "BLOCKED";
if (query.queryStats.blockedReasons && query.queryStats.blockedReasons.length > 0) {
title += " (" + query.queryStats.blockedReasons.join(", ") + ")";
}
}
if (query.memoryPool === "reserved") {
title += " (RESERVED)"
}
return title;
}
}
if (query.state === "FAILED") {
switch (query.failed) {
case "USER_ERROR":
return "USER ERROR";
case "INTERNAL_ERROR":
return "INTERNAL ERROR";
case "INSUFFICIENT_RESOURCES":
return "INSUFFICIENT RESOURCES";
case "EXTERNAL":
return "EXTERNAL ERROR";
}
}
return query.state;
}
export function isQueryEnded(query: any): boolean
{
return ["FINISHED", "FAILED", "CANCELED"].indexOf(query.state) > -1;
}
// Sparkline-related functions
// ===========================
// display at most 5 minutes worth of data on the sparklines
const MAX_HISTORY = 60 * 5;
// alpha param of exponentially weighted moving average. picked arbitrarily - lower values means more smoothness
const MOVING_AVERAGE_ALPHA = 0.2;
export function addToHistory (value: number, valuesArray: number[]): number[] {
if (valuesArray.length === 0) {
return valuesArray.concat([value]);
}
return valuesArray.concat([value]).slice(Math.max(valuesArray.length - MAX_HISTORY, 0));
}
export function addExponentiallyWeightedToHistory (value: number, valuesArray: number[]): number[] {
if (valuesArray.length === 0) {
return valuesArray.concat([value]);
}
let movingAverage = (value * MOVING_AVERAGE_ALPHA) + (valuesArray[valuesArray.length - 1] * (1 - MOVING_AVERAGE_ALPHA));
if (value < 1) {
movingAverage = 0;
}
return valuesArray.concat([movingAverage]).slice(Math.max(valuesArray.length - MAX_HISTORY, 0));
}
// DagreD3 Graph-related functions
// ===============================
export function initializeGraph()
{
return new dagreD3.graphlib.Graph({compound: true})
.setGraph({rankdir: 'BT'})
.setDefaultEdgeLabel(function () { return {}; });
}
export function initializeSvg(selector: any)
{
const svg = d3.select(selector);
svg.append("g");
return svg;
}
export function getChildren(nodeInfo: any)
{
const nodeType = removeNodeTypePackage(nodeInfo["@type"]);
switch (nodeType) {
case "OutputNode":
case "ExplainAnalyzeNode":
case "ProjectNode":
case "FilterNode":
case "AggregationNode":
case "SortNode":
case "MarkDistinctNode":
case "WindowNode":
case "RowNumberNode":
case "TopNRowNumberNode":
case "LimitNode":
case "DistinctLimitNode":
case "TopNNode":
case "SampleNode":
case "TableWriterNode":
case "DeleteNode":
case 'TableDeleteNode':
case 'TableFinishNode':
case 'GroupIdNode':
case 'CTEScanNode':
case 'UnnestNode':
case 'EnforceSingleRowNode':
return [nodeInfo.source];
case 'JoinNode':
return [nodeInfo.left, nodeInfo.right];
case 'SemiJoinNode':
return [nodeInfo.source, nodeInfo.filteringSource];
case 'SpatialJoinNode':
return [nodeInfo.left, nodeInfo.right];
case 'IndexJoinNode':
return [nodeInfo.probeSource, nodeInfo.indexSource];
case 'UnionNode':
case 'ExchangeNode':
return nodeInfo.sources;
case 'RemoteSourceNode':
case 'TableScanNode':
case 'ValuesNode':
case 'IndexSourceNode':
break;
default:
console.log("NOTE: Unhandled PlanNode: " + nodeType);
}
return [];
}
// Utility functions
// =================
export function truncateString(inputString: string, length: number): string {
if (inputString && inputString.length > length) {
return inputString.substring(0, length) + "...";
}
return inputString;
}
export function getStageNumber(stageId: string): number {
return Number.parseInt(stageId.slice(stageId.indexOf('.') + 1, stageId.length))
}
export function getTaskIdSuffix(taskId: string): string {
return taskId.slice(taskId.indexOf('.') + 1, taskId.length)
}
export function getTaskNumber(taskId: string): number {
return Number.parseInt(getTaskIdSuffix(getTaskIdSuffix(taskId)));
}
export function getFirstParameter(searchString: string): string {
const searchText = searchString.substring(1);
if (searchText.indexOf('&') !== -1) {
return searchText.substring(0, searchText.indexOf('&'));
}
return searchText;
}
export function getHostname(url: string): string {
let hostname = new URL(url).hostname;
if ((hostname.charAt(0) === '[') && (hostname.charAt(hostname.length - 1) === ']')) {
hostname = hostname.substr(1, hostname.length - 2);
}
return hostname;
}
export function getPort(url: string): string {
return new URL(url).port;
}
export function getHostAndPort(urlStr: string): string {
const url = new URL(urlStr);
return url.hostname + ":" + url.port;
}
export function computeRate(count: number, ms: number): number {
if (ms === 0) {
return 0;
}
return (count / ms) * 1000.0;
}
export function precisionRound(n: number): string {
if (n < 10) {
return n.toFixed(2);
}
if (n < 100) {
return n.toFixed(1);
}
return Math.round(n).toString();
}
export function formatDuration(duration: number): string {
let unit = "ms";
if (duration > 1000) {
duration /= 1000;
unit = "s";
}
if (unit === "s" && duration > 60) {
duration /= 60;
unit = "m";
}
if (unit === "m" && duration > 60) {
duration /= 60;
unit = "h";
}
if (unit === "h" && duration > 24) {
duration /= 24;
unit = "d";
}
if (unit === "d" && duration > 7) {
duration /= 7;
unit = "w";
}
return precisionRound(duration) + unit;
}
export function formatRows(count: number): string {
if (count === 1) {
return "1 row";
}
return formatCount(count) + " rows";
}
export function formatCount(count: number): string {
let unit = "";
if (count > 1000) {
count /= 1000;
unit = "K";
}
if (count > 1000) {
count /= 1000;
unit = "M";
}
if (count > 1000) {
count /= 1000;
unit = "B";
}
if (count > 1000) {
count /= 1000;
unit = "T";
}
if (count > 1000) {
count /= 1000;
unit = "Q";
}
return precisionRound(count) + unit;
}
export function formatDataSizeBytes(size: number): string {
return formatDataSizeMinUnit(size, "");
}
export function formatDataSize(size: number): string {
return formatDataSizeMinUnit(size, "B");
}
function formatDataSizeMinUnit(size: number, minUnit: string): string {
let unit = minUnit;
if (size === 0) {
return "0" + unit;
}
if (size >= 1024) {
size /= 1024;
unit = "K" + minUnit;
}
if (size >= 1024) {
size /= 1024;
unit = "M" + minUnit;
}
if (size >= 1024) {
size /= 1024;
unit = "G" + minUnit;
}
if (size >= 1024) {
size /= 1024;
unit = "T" + minUnit;
}
if (size >= 1024) {
size /= 1024;
unit = "P" + minUnit;
}
return precisionRound(size) + unit;
}
export function parseDataSize(value: string): ?number {
const DATA_SIZE_PATTERN = /^\s*(\d+(?:\.\d+)?)\s*([a-zA-Z]+)\s*$/;
const match = DATA_SIZE_PATTERN.exec(value);
if (match === null) {
return null;
}
const number = parseFloat(match[1]);
switch (match[2]) {
case "B":
return number;
case "kB":
return number * Math.pow(2, 10);
case "MB":
return number * Math.pow(2, 20);
case "GB":
return number * Math.pow(2, 30);
case "TB":
return number * Math.pow(2, 40);
case "PB":
return number * Math.pow(2, 50);
default:
return null;
}
}
export function parseDuration(value: string): ?number {
const DURATION_PATTERN = /^\s*(\d+(?:\.\d+)?)\s*([a-zA-Z]+)\s*$/;
const match = DURATION_PATTERN.exec(value);
if (match === null) {
return null;
}
const number = parseFloat(match[1]);
switch (match[2]) {
case "ns":
return number / 1000000.0;
case "us":
return number / 1000.0;
case "ms":
return number;
case "s":
return number * 1000;
case "m":
return number * 1000 * 60;
case "h":
return number * 1000 * 60 * 60;
case "d":
return number * 1000 * 60 * 60 * 24;
default:
return null;
}
}
export function formatShortTime(date: Date): string {
const hours = (date.getHours() % 12) || 12;
const minutes = (date.getMinutes() < 10 ? "0" : "") + date.getMinutes();
return hours + ":" + minutes + (date.getHours() >= 12 ? "pm" : "am");
}
export function formatShortDateTime(date: Date): string {
const year = date.getFullYear();
const month = "" + (date.getMonth() + 1);
const dayOfMonth = "" + date.getDate();
return year + "-" + (month[1] ? month : "0" + month[0]) + "-" + (dayOfMonth[1] ? dayOfMonth: "0" + dayOfMonth[0]) + " " + formatShortTime(date);
}
export function bubbleSort(arr): void {
let len = arr.length;
for(let i = 0; i < len - 1; i++) {
for(let j = 0; j < len - 1 - i; j++) {
if(arr[j].value < arr[j+1].value) {
let temp = arr[j+1];
arr[j+1] = arr[j];
arr[j] = temp;
}
}
}
return arr;
}
// Remove the Java package from each node type to convert the node type to the short name.
// For example, in the response sent from the server, an output node is represented by
// "io.prestosql.sql.planner.plan.OutputNode". After the invocation of this function,
// the short name "OutputNode" will be returned.
export function removeNodeTypePackage(nodeType: string): string {
const classEndIndex = nodeType.lastIndexOf(".");
return nodeType.substr(classEndIndex + 1);
}

View File

@ -11,10 +11,11 @@
"babel-preset-react": "^6.24.1",
"flow-bin": "^0.85.0",
"webpack": "^4.43.0",
"webpack-cli": "^3.3.11",
"webpack-cli": "^4.9.1",
"webpack-command": "^0.4.1"
},
"dependencies": {
"bootstrap-select": "^1.13.18",
"@bosket/core": "latest",
"@bosket/react": "latest",
"@bosket/tools": "latest",

View File

@ -0,0 +1,46 @@
/*
* Copyright (C) 2018-2020. Huawei Technologies Co., Ltd. All rights reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import xhr from "../utils/xhr";
import alt from "../alt";
class AuditLogActions {
constructor() {
}
getAuditLog(user , beginTime , endTime , level , type) {
return xhr(`../v1/audit/${type}`, {
method: 'post',
headers :{
'Content-Type': 'application/x-www-form-urlencoded'
},
body: 'user=' + user + '&' + 'beginTime=' + beginTime + '&' + 'endTime=' + endTime + '&' + 'level=' + level + '&' + 'type=' + type
}).then((data)=> {
return data ;
});
}
downloadAuditLog(user , beginTime , endTime , level , type) {
return xhr(`../v1/audit/download`, {
method: 'post',
headers :{
'Content-Type': 'application/x-www-form-urlencoded'
},
body: 'user=' + user + '&' + 'beginTime=' + beginTime + '&' + 'endTime=' + endTime + '&' + 'level=' + level + '&' + 'type=' + type
});
}
}
export default alt.createActions(AuditLogActions);

View File

@ -14,6 +14,8 @@
*/
import alt from '../alt';
import CatalogApiUtils from "../utils/CatalogApiUtils";
import xhrform from "../utils/xhrform";
import UserStore from "../stores/UserStore";
class CatalogActions {
constructor() {

View File

@ -0,0 +1,61 @@
/*
* Copyright (C) 2018-2020. Huawei Technologies Co., Ltd. All rights reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import xhr from "../utils/xhr";
import alt from "../alt";
class CollectionActions {
constructor() {
}
addToCollection(queryText,catalog,schema) {
return xhr("../v1/myCollection/collection", {
method: 'post',
headers :{
'Content-Type': 'application/x-www-form-urlencoded'
},
body: 'queryText=' + queryText + '&' + 'catalog=' + catalog + '&' + 'schema=' + schema
}).then((data)=> {
if(data==false||data=="false") {
alert("Error! Your favorite SQLs are full! Please delete some favorite SQL statement!");
}
});
}
getCollection(queryText) {
return xhr(`../v1/myCollection/collection/query?${queryText}`).then((data) => {
return data;
})
}
deleteCollection(queryText,catalog,schema) {
return xhr(`../v1/myCollection/collection/delete`, {
headers: {
'Content-Type': 'application/x-www-form-urlencoded'
},
method: 'DELETE',
body: 'queryText=' + queryText + '&' + 'catalog=' + catalog + '&' + 'schema=' + schema
}).then((data)=> {
if(data==true||data=="true") {
}
else{
alert("delete failed!");
}
});
}
}
export default alt.createActions(CollectionActions);

View File

@ -46,6 +46,7 @@ class SchemaActions {
message: "Success"
}
}).catch((error) => {
//console.log('here',error);
return {
result: false,
message: error.message
@ -53,6 +54,24 @@ class SchemaActions {
})
}
fetchCatalogs(){
return xhr("../api/metadata/catalogs").then((data) => {
return data;
})
}
fetchOnlySchema(catalog){
return xhr("../api/metadata/schemas/"+catalog).then((data) => {
return data;
})
}
fetchTable(catalog,schema){
return xhr("../api/metadata/tables/"+catalog+"/"+schema).then((data) => {
return data;
})
}
fetchSchemas(catalogs) {
return xhr("../api/metadata/schemas").then((data) => {
catalogs = [];

View File

@ -21,7 +21,7 @@ class AllRunningQueries
super(props);
}
componentWillMount() {
componentDidMount() {
}
render() {

View File

@ -0,0 +1,195 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from 'react';
import 'moment-timezone';
import UserStore from "../stores/UserStore";
import RunStore from "../stores/RunStore";
import Pagination from "rc-pagination";
import Select from "rc-select";
import _ from "lodash";
import CollectionAction from "../actions/CollectionActions";
import {Table, Column, Cell} from 'fixed-data-table-2';
import {setSelection} from "../../../vendor/vis/dist/vis";
import QueryActions from "../actions/QueryActions";
let columnWidths = {
user: 150,
context:200,
query: 800,
delete: 100,
}
class CollectionTable extends React.Component {
constructor(props) {
super(props);
this.state = {
user: UserStore.getCurrentUser().name,
pageSize: 10,
total: 0,
currentPage: 1,
allCollection: [],
metaStoreType: ''
}
this.onPageChange = this.onPageChange.bind(this);
this.onPageSizeChange = this.onPageSizeChange.bind(this);
this.refreshData = this.refreshData.bind(this);
this.handleDeleteButton = this.handleDeleteButton.bind(this);
this.renderQuery = this.renderQuery.bind(this);
}
refreshData(){
const {currentPage,pageSize} = this.state;
let queryText = "pageSize=" + pageSize + "&&" + "pageNum=" + currentPage;
$.get(`../v1/myCollection/collection/query?${queryText}`, function (queryList) {
this.setState({
allCollection: queryList.queries,
total:queryList.total
});
}.bind(this))
}
componentDidMount() {
$.get(`../v1/query/hetuMetastoreType`, function (metaStoreType) {
this.state.metaStoreType = metaStoreType;
if(this.state.metaStoreType != "jdbc") {
alert("Collection failed! Only hetu-metastore storage type configured as JDBC is supported!");
}
}.bind(this))
this.refreshData();
}
componentWillUnmount() {
}
handleEvent(){
_.defer(this.refreshData);
}
onPageChange(current, pageSize) {
this.state.currentPage = current
_.defer(this.refreshData);
}
onPageSizeChange(current, pageSize) {
this.state.pageSize = pageSize;
_.defer(this.refreshData);
}
handleDeleteButton(e){
let rowIndex = e.target.id;
let queryString = this.state.allCollection;
let deleteText = queryString[rowIndex].query;
let catalog = queryString[rowIndex].catalog;
let schema = queryString[rowIndex].schemata;
CollectionAction.deleteCollection(deleteText,catalog,schema).then();
_.defer(this.refreshData);
}
renderQuery(rowIndex) {
let sessionContext = {catalog : this.state.allCollection[rowIndex].catalog ,
schema : this.state.allCollection[rowIndex].schemata}
let query = this.state.allCollection[rowIndex].query;
return (
<a href="#" onClick={selectQuery.bind(null, query, sessionContext)}>
<code title={query}>{query}</code>
</a>
);
}
render() {
let rowsCount = 0;
let compare = this.state.total - (this.state.currentPage-1)*this.state.pageSize;
if (compare < this.state.pageSize){
rowsCount = compare;
}
else {
rowsCount = this.state.pageSize;
}
const {allCollection, total, pageSize, currentPage, user} = this.state;
if (allCollection == null){
return (
<p className="info panel-body text-light text-center">No queries to show</p>
);
}
else {
return (
<div className="flex collection-table hetu-table">
<div className="col-xs-12 row toolbar-row">
<Table
className="collection_table"
rowHeight={50}
rowsCount={rowsCount}
width={this.props.tableWidth}
height={this.props.tableHeight}
headerHeight={30}>
<Column
header={<Cell>User</Cell>}
cell={({rowIndex}) => <Cell> {user} </Cell>}
width={columnWidths.user}
/>
<Column
header={<Cell>Context</Cell>}
cell={({rowIndex}) =>
<Cell> {allCollection[rowIndex].catalog}.{allCollection[rowIndex].schemata}</Cell>}
width={columnWidths.context}
/>
<Column
header={<Cell>Query</Cell>}
cell={({rowIndex}) => <Cell> {this.renderQuery(rowIndex)} </Cell>}
width={columnWidths.query}
/>
<Column
header={<Cell>Delete</Cell>}
cell={({rowIndex}) => <Cell>
<button className="collection-btn" id={rowIndex} onClick={this.handleDeleteButton}>
<i className="icon icon-center fa fa-trash-o" />
</button>
</Cell>}
width={columnWidths.delete}
/>
</Table>
<Pagination
defaultCurrent={1}
current={currentPage}
total={total}
defaultPageSize={10}
pageSize={pageSize}
onChange={this.onPageChange}
showTotal={total => `Total ${total} queries`}
style={{marginTop: 10}}
showSizeChanger
onShowSizeChange={this.onPageSizeChange}
selectComponentClass={Select}
/>
</div>
</div>
)
}
}
renderEmptyMessage() {
return (
<p className="info panel-body text-light text-center">No queries to show</p>
);
}
}
function selectQuery(query, sessionContext, e) {
e.preventDefault();
QueryActions.selectQuery({query, sessionContext});
}
export default CollectionTable;

View File

@ -0,0 +1,32 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react";
import CollectionTable from './CollectionTable';
class MyCollection
extends React.Component {
constructor(props) {
super(props);
}
render() {
return(
<CollectionTable
tableWidth={this.props.tableWidth}
tableHeight={this.props.tableHeight} />
);
}
}
export default MyCollection;

View File

@ -28,6 +28,8 @@ import CnxnMonitorStore from "../stores/CnxnMonitorStore";
import CnxnMonitorActions from "../actions/CnxnMonitorActions";
import SchemaStore from "../stores/SchemaStore";
import {Form} from "react-bootstrap";
import SchemaActions, { dataType } from "../actions/SchemaActions";
import CollectionAction from "../actions/CollectionActions";
class QueryEditor
extends React.Component {
@ -44,8 +46,22 @@ class QueryEditor
schema: "runtime"
},
buttonState:false
}
;
};
this.stateType= {
emptyCatalog: 1,
emptySchema: 2,
emptyNeither: 3
};
this.catalogs = [];
this.schemas = [];
this.sql = (
"select|insert|update|delete|from|where|and|or|group|by|order|limit|offset|having|as|case|" +
"when|else|end|type|left|right|join|on|outer|desc|asc|union|create|table|primary|key|if|" +
"foreign|not|references|default|null|inner|cross|natural|database|drop|grant|"+
"avg|count|first|last|max|min|sum|ucase|lcase|mid|len|round|rank|now|format|coalesce|ifnull|isnull|nvl|"+
"money|real|number|integer|true|false"
).split('|');
this.query = "select * from system.information_schema.tables;";
this.queryEditorRef = React.createRef();
this.queryContainerRef = React.createRef();
@ -62,6 +78,7 @@ class QueryEditor
this.closeErrorDialog = this.closeErrorDialog.bind(this);
this.errorCollapseToggle = this.errorCollapseToggle.bind(this);
this.getBriefErrorMessage = this.getBriefErrorMessage.bind(this);
this.onload = this.onload.bind(this);
}
componentDidMount() {
@ -78,7 +95,7 @@ class QueryEditor
closeErrorDialog() {
CnxnMonitorActions.clear();
var state = this.state;
let state = this.state;
Object.assign(state, {
lastSubmissionResult: "",
lastPollingError: "",
@ -126,7 +143,7 @@ class QueryEditor
header={header}
footer={this.renderErrorDialogFooter()}
onClose={this.closeErrorDialog}>
<div style={{color: '#ff0000',padding:'10px 20px'}}>
<div style={{color: '#ffffff',padding:'10px 20px'}}>
<h4 style={{alignSelf: "center"}}>
{this.getBriefErrorMessage()}
</h4>
@ -233,6 +250,11 @@ class QueryEditor
}
render() {
setInterval(function(){
SchemaActions.fetchCatalogs().then((data) => {
this.catalogs=data;
});
}, 10000);
return (
<div className="flex flex-initial flex-column">
<div className="flex flex-row editor-menu">
@ -249,6 +271,7 @@ class QueryEditor
ref={this.queryEditorRef}
mode="sql"
theme={"eclipse"}
onLoad={this.onload}
onChange={this.onChange}
name="editor"
highlightActiveLine={true}
@ -269,7 +292,8 @@ class QueryEditor
/>
</div>
</div>
</div>);
</div>
);
}
// - Internal events ----------------------------------------------------- //
@ -347,10 +371,93 @@ class QueryEditor
this.setState(state);
}
onload(){
SchemaActions.fetchCatalogs().then((data) => {
this.catalogs=data;
});
}
onChange(newValue) {
var value = newValue.substring( newValue.lastIndexOf(";")+1,newValue.length);
var catalog = '';
var schema = '';
var value = value.substring(value.lastIndexOf(' ')+1,value.length).replace(/[\r\n]/g, "");
var index = value.indexOf('.');
if(index!=-1) {
catalog =value.substring(0,index);
var value = value.substring(index+1,value.length);
var next_index = value.indexOf('.');
if(next_index!=-1) {
schema = value.substring(0,next_index);
}
}
const editor= this.queryEditorRef.current.editor;
let currentCatalog = this.state.currentContext.catalog;
let currentSchema = this.state.currentContext.schema;
if(catalog==='') {
SchemaActions.fetchTable(currentCatalog,currentSchema).then((tables)=>{
let table_data = [];
for (const value of tables){
table_data.push(value.table)
}
this.addCompleters(editor,this.catalogs,this.stateType.emptyCatalog,table_data);
});
}
else if(catalog!=''&&schema==='') {
if(this.catalogs.includes(catalog)){
SchemaActions.fetchOnlySchema(catalog).then((data) => {
this.addCompleters(editor,data.schemas,this.stateType.emptySchema,[]);
})
}
}
else if(catalog!=''&&schema!='') {
if(this.catalogs.includes(catalog)&&this.schemas.includes(schema)){
SchemaActions.fetchTable(catalog,schema).then((data)=>{
let tablesdata = [];
for (const value of data){
tablesdata.push(value.table)
}
this.addCompleters(editor,tablesdata,this.stateType.emptyNeither,[]);
})
}
}
this.query = newValue;
}
addCompleters(editor,data,state,table_data) {
const arr = [];
editor.completers=arr;
var data_complete = [];
for(let val of data){
data_complete.push(val);
}
if(state===this.stateType.emptyCatalog) {
for(let val of this.sql){
data_complete.push(val);
}
}
if(state===this.stateType.emptySchema) {
this.schemas=data;
}
for(let val of table_data) {
if(!data_complete.includes(val))
data_complete.push(val);
}
let completers = data_complete.map(item=>({
name:item,
value:item,
score:100,
meta:'',
}));
editor.completers.push({
getCompletions: function (editors, session, pos, prefix, callback) {
callback(null, completers);
}
});
editor.resize();
}
schemaContextHandler() {
let model = SchemaStore.getModel();
let state = this.state;

View File

@ -26,6 +26,7 @@ import TabConstants from '../constants/TabConstants';
import {Tab, TabList, TabPanel, Tabs} from 'react-tabs';
import MySavedQueries from "./MySavedQueries";
import MyCollection from "./MyCollection";
class QueryInformation
extends React.Component {
@ -114,6 +115,7 @@ class QueryInformation
<Tab>All Queries</Tab>
<Tab>Results</Tab>
<Tab>Data Preview</Tab>
<Tab>My Collection</Tab>
</TabList>
<TabPanel style={{height:"calc(70vh - 200px)", overflowY:'auto'}}>
<MySavedQueries/>
@ -133,6 +135,11 @@ class QueryInformation
tableWidth={this.state.tableWidth}
tableHeight={this.state.tableHeight}/>
</TabPanel>
<TabPanel style={{overflowY:'auto',height:"calc(70vh - 200px)"}}>
<MyCollection
tableWidth={this.state.tableWidth}
tableHeight={this.state.tableHeight}/>
</TabPanel>
</Tabs>
</div>);
}

View File

@ -26,6 +26,7 @@ import {Cell, Column, Table} from 'fixed-data-table-2';
import ProgressBar from "./ProgressBar";
import ModalDialog from "./ModalDialog";
import UserStore from "../stores/UserStore";
import CollectionAction from "../actions/CollectionActions";
var classNames = require('classnames');
let isColumnResizing = false;
@ -37,7 +38,9 @@ let columnWidths = {
started: 180,
duration: 80,
output: 150,
addtocollection: 120,
};
let metaStoreType = '';
// State actions
function getRuns(user) {
@ -61,7 +64,8 @@ class RunsTable extends React.Component {
currentErrorMessage: null,
showModal: false,
runs: getRuns(this.props.user),
user: UserStore.getCurrentUser()
user: UserStore.getCurrentUser(),
isCollected:[]
};
this.getStateFromStore = this.getStateFromStore.bind(this);
this._onChange = this._onChange.bind(this);
@ -71,6 +75,7 @@ class RunsTable extends React.Component {
this.renderErrorDialogFooter = this.renderErrorDialogFooter.bind(this);
this.rowGetter = this.rowGetter.bind(this);
this.columnGetter = this.columnGetter.bind(this);
this.handleCollection = this.handleCollection.bind(this);
}
getStateFromStore() {
@ -83,6 +88,11 @@ class RunsTable extends React.Component {
}
componentDidMount() {
$.get(`../v1/query/hetuMetastoreType`, function (type) {
metaStoreType = type;
this.setState();
}.bind(this))
UserStore.listen(this._onChange);
RunStore.listen(this._onChange);
}
@ -118,7 +128,32 @@ class RunsTable extends React.Component {
);
}
handleCollection(e) {
this.setState({
isCollected: isCollected
})
if (this.state.isCollected){
// delete
}
else {
CollectionAction.addToCollection()
}
}
render() {
const CollectionButton = ({isCollected}) =>
<Cell>
{isCollected ?
<button className="collection-btn" onClick={this.handleCollection}>
<i className="icon fa fa-star"/>
</button>
:
<button className="collection-btn" onClick={this.handleCollection}>
<i className="icon fa fa-star-o"/>
</button>
}
</Cell>
if (this.state.runs.length === 0) {
return this.renderEmptyMessage();
}
@ -169,6 +204,11 @@ class RunsTable extends React.Component {
cell={({rowIndex}) => <Cell> {this.columnGetter('output', rowIndex)}</Cell>}
width={columnWidths.output}
/>
<Column
header={<Cell>Collect</Cell>}
cell={({rowIndex}) => <Cell> {this.columnGetter('collect',rowIndex)}</Cell>}
width={columnWidths.addtocollection}
/>
</Table>
</div>);
}
@ -195,12 +235,15 @@ class RunsTable extends React.Component {
return row.duration;
} else if ("output" == key) {
return CellRenderers.output(row.output, "output", row, this);
} else if ("collect" == key) {
return CellRenderers.collect(row.context,row.query,rowIndex,this);
}
}
rowGetter(rowIndex) {
//Show latest on top
let descIndex = this.state.runs.length - rowIndex - 1;
this.state.isCollected.push(0);
return formatRun(this.state.runs[descIndex], this.state.user);
}
@ -272,6 +315,36 @@ function toggleModal(component, msg=null) {
component.setState(state);
}
function handleCollection(e,rowIndex,parentObj) {
let queryText = e.query;
if(queryText.length > 1000) {
alert("Error! Your SQL statement is too long! Please shorten it!");
}
else if(metaStoreType != "jdbc") {
alert("Collection failed! Only hetu-metastore storage type configured as JDBC is supported!");
}
else {
let catalog = e.sessionContext.catalog;
let schema = e.sessionContext.schema;
let state = parentObj.state;
if (state.isCollected[rowIndex] == 0) {
state.isCollected[rowIndex] = 1
CollectionAction.addToCollection(queryText, catalog, schema);
} else {
state.isCollected[rowIndex] = 0
CollectionAction.deleteCollection(queryText, catalog, schema);
}
parentObj.setState(state);
}
}
function handleDelete(e) {
let queryText = e.query;
let catalog = e.sessionContext.catalog;
let schema = e.sessionContext.schema;
CollectionAction.deleteCollection(queryText,catalog,schema);
}
let CellRenderers = {
user(cellData) {
return <span title={cellData}>{cellData}</span>;
@ -379,12 +452,21 @@ let CellRenderers = {
},
started(cellData) {
// let m = moment.utc(cellData, 'yyyy-mm-ddTHH:MM:ss');
// let utc = m.format();
// let human = m.format('lll');
// return <span title={utc}>{human} UTC</span>;
return <Moment utc local>{cellData}</Moment>
}
},
collect(sessionContext,query,rowIndex,parentObj) {
const collection = {sessionContext,query}
let descIndex = parentObj.state.runs.length - rowIndex - 1;
return (
<span>
<button className="collection-btn" onClick={() => {handleCollection(collection,descIndex,parentObj)}}>
<i className="icon-center fa fa-plus icon"/>
<i className="icon-center fa fa-star icon"/>
</button>
</span>
)
},
};
function getProgressFromStats(stats) {

View File

@ -23,6 +23,11 @@ import TabActions from "../actions/TabActions";
import TabConstants from "../constants/TabConstants";
import _ from "lodash";
import QueryActions from "../actions/QueryActions";
import ModalDialog from "./ModalDialog";
import ShowCatalog from "./ShowCatalog";
import {color} from "echarts/lib/export";
let flag = true;
function getIcon(type) {
switch (type) {
@ -35,8 +40,12 @@ function getIcon(type) {
// return (<i className="material-icons">storage</i>);
}
case dataType.CATALOG: {
return (<i className="icon fa fa-server valign-middle"></i>);
// return (<i className="material-icons">source</i>);
if (!flag) {
return (<i className="icon fa fa-server valign-middle"></i>);
}
else {
return <i className="icon fa fa-server valign-middle" style={{marginLeft: "14.5px", color : 'gray'}}></i>
}
}
default: {
return (<i className="material-icons">dashboard</i>);
@ -45,9 +54,9 @@ function getIcon(type) {
}
function renderItem(tree, item) {
let style = (item.children == undefined || item.children instanceof Array && item.children.length == 0) ? { marginLeft: "14.5px" } : {};
flag = item.children == undefined || item.children instanceof Array && item.children.length == 0 ;
let style = (item.children == undefined || item.children instanceof Array && item.children.length == 0) ? { marginLeft: "14.5px",color: "gray" } : {};
let tableStyle = {};
Object.assign(tableStyle, style, { cursor: "pointer" })
let favorite = tree.isFavorite(item);
if (item.type == dataType.TABLE) {
if (item.fqn == tree.selectedTableName) {
@ -59,7 +68,7 @@ function renderItem(tree, item) {
{getIcon(item.type)}<span>{item.name}</span>{favorite.found ? <i className="icon fa fa-star valign-middle schema-tree-icons favorite" /> : null}
</ContextMenuTrigger>
<ContextMenu id={item.fqn}>
{favorite.found && favorite.self ?
{favorite.found && favorite.self && !flag ?
<MenuItem data={{ item: item, tree: tree }} onClick={(e, data) => {
tree.removeFromFavorites(item);
}}>
@ -105,7 +114,7 @@ function renderItem(tree, item) {
"icon fa fa-star valign-middle schema-tree-icons favoriteParent"} /> : null}
</ContextMenuTrigger>
<ContextMenu id={item.fqn}>
{favorite.found && favorite.self ?
{favorite.found && favorite.self && !flag ?
<MenuItem data={{ item: item, tree: tree }} onClick={(e, data) => {
tree.removeFromFavorites(item);
}}>
@ -123,6 +132,14 @@ function renderItem(tree, item) {
}}>
<i className="icon fa fa-refresh valign-middle" /><span>Refresh</span>
</MenuItem>
{item.type == dataType.CATALOG ?
<MenuItem data={{ item: item, tree: tree }} onClick={(e, data) => {
tree.showCatalog(item);
}}>
<i className="icon fa fa-file-text-o valign-middle contextmenu-icons show-catalog" /><span>Show Catalog</span>
</MenuItem>
: null
}
{item.type == dataType.CATALOG ?
<MenuItem data={{ item: item, tree: tree }} onClick={(e, data) => {
tree.deleteCatalog(item);
@ -146,7 +163,6 @@ function renderItem(tree, item) {
</a>
)
}
// return (<a style={style}>{getIcon(item.type)}<span>{item.name}</span></a>);
}
function sortItems(tree, item1, item2) {
@ -185,7 +201,13 @@ class SchemaTree extends React.Component {
},
height: 0,
model: this.getInitialModel(),
name: "name"
name: "name",
show: false,
catalog_name:"",
connection_password: "",
connector_name: "",
url: "",
user: ""
};
this.selectedTableName = "";
this.treeRef = React.createRef();
@ -194,6 +216,7 @@ class SchemaTree extends React.Component {
schemas: [],
tables: []
}
this.showObj = {}
this.updateTree = this.updateTree.bind(this);
this.selectTable = this.selectTable.bind(this);
this.unselectTable = this.unselectTable.bind(this);
@ -203,6 +226,9 @@ class SchemaTree extends React.Component {
this.reloadItem = this.reloadItem.bind(this);
this.refreshItem = this.refreshItem.bind(this);
this.deleteCatalog = this.deleteCatalog.bind(this);
this.showCatalog = this.showCatalog.bind(this);
this.showModal = this.showModal.bind(this);
this._objToStrMap = this._objToStrMap.bind(this);
}
updateTree() {
@ -256,6 +282,40 @@ class SchemaTree extends React.Component {
element.style.color = "#222222";
}
showModal() {
let newState = !this.state.show;
this.setState({
show: newState,
});
}
_objToStrMap(obj){
let strMap = new Map();
for (let k of Object.keys(obj)) {
strMap.set(k,obj[k]);
}
return strMap;
}
showCatalog(item) {
this.state.catalog_name = item.name;
let showText = item.name;
$.get(`../v1/showCatalog/${showText}`, function (showList) {
let showMap = this._objToStrMap(showList);
this.showObj = showList;
this.setState({
connection_password : showMap.get('connection-password'),
connector_name : showMap.get('connector.name'),
url : showMap.get('connection-url'),
user : showMap.get('connection-user')
})
}.bind(this))
let newState = !this.state.show;
this.setState({
show: newState,
});
}
deleteCatalog(item) {
let msg = 'Are you sure you want to delete the catalog?';
if (item.name.indexOf(".") !== -1) {
@ -368,7 +428,7 @@ class SchemaTree extends React.Component {
return { found: false, self: false };
}
reloadItem(item) {
reloadItem() {
let model = this.state.model;
this.state.model = [];
this.setState(this.state);
@ -406,7 +466,11 @@ class SchemaTree extends React.Component {
<div style={{ height: "calc(100vh - 200px)" }}>
<TreeView {...this.state} ref={this.treeRef}></TreeView>
</div>
</div>
<ModalDialog onClose={this.showModal} header={"Show Catalog"} footer={""}
show={this.state.show}>
<ShowCatalog onClose={this.showModal.bind(this)} catalog_name={this.state.catalog_name} showText={this.showObj}/>
</ModalDialog>
</div>
);
}
}

View File

@ -0,0 +1,144 @@
/*
* Copyright (C) 2018-2020. Huawei Technologies Co., Ltd. All rights reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from 'react';
import {Button, Col, Form, FormControl, FormGroup, FormLabel, Row} from "react-bootstrap";
const props = {};
class ShowCatalog extends React.Component {
constructor(props) {
super(props);
this.state = {
catalog_name:"",
connection_password: "",
connector_name: "",
url: "",
user: ""
}
this.showObj = {}
this.handleClose = this.handleClose.bind(this);
this.componentWillReceiveProps = this.componentWillReceiveProps.bind(this);
this._objToStrMap = this._objToStrMap.bind(this);
this.deleteMapKey = this.deleteMapKey.bind(this);
this.renderCatalogProperties = this.renderCatalogProperties.bind(this);
}
componentWillReceiveProps(nextProps) {
if (this.props.showText != nextProps.showText){
this.state.catalog_name = nextProps.catalog_name;
this.showObj = nextProps.showText
this.setState();
}
}
handleClose() {
this.props.onClose && this.props.onClose();
}
_objToStrMap(obj){
let strMap = new Map();
for (let k of Object.keys(obj)) {
strMap.set(k,obj[k]);
}
return strMap;
}
deleteMapKey(obj,key) {
let newMap = new Map();
for (let item of obj.keys()){
if (item != key){
newMap.set(item,obj.get(item));
}
}
return newMap;
}
renderCatalogProperties(obj) {
let arr = []
for (let item of obj.keys()) {
arr.push(
<Row style={{display: "flex",alignItems:'center'}}>
<Col style={{flexDirection: "column", marginRight: '10px', width:"70%"}}>
<FormControl name="name" type="text" placeholder={"property name"}
value={item}
readOnly="true"/>
</Col>
<Col style={{flexDirection: "column", width:"100%"}}>
<FormControl name="value" type="text" placeholder={"property value"}
value={obj.get(item)}
readOnly="true"/>
</Col>
</Row>
)
}
return (
arr
)
}
render() {
let showMap = this._objToStrMap(this.showObj);
let catalogProperties = this.deleteMapKey(showMap,'connector.name');
return(
<div>
<div className={"hetu-foem-body"} style={{position: "relative", paddingLeft: "20px", paddingRight: "20px", overflowY: "auto"}}>
<Form>
<FormGroup>
<Row>
<Col style={{display: "flex", alignItems:'center'}}>
<FormLabel className={"hetu-form-label"}>Data Source Type</FormLabel>
</Col>
<Col>
<div style={{display: "block"}}>
<input className="form-control" value={showMap.get('connector.name')} readOnly="true"/>
</div>
</Col>
</Row>
<Row>
<Col style={{display: "flex", alignItems:'center'}}>
<FormLabel className={"hetu-form-label"}>Catalog Name</FormLabel>
</Col>
<Col>
<div style={{display: "block"}}>
<input className="form-control" value={this.state.catalog_name} disabled="true"/>
</div>
</Col>
</Row>
{catalogProperties.size == 0 ?
null
:
<Row>
<Col style={{display: "flex", alignItems: 'center'}}>
<FormLabel className={"hetu-form-label"}>Catalog Properties</FormLabel>
</Col>
<Col>
<div style={{display: "block"}}>
{this.renderCatalogProperties(catalogProperties)}
</div>
</Col>
</Row>
}
</FormGroup>
</Form>
</div>
<div className={'catalog-btn-part'}>
<Button onClick={this.handleClose} className={"btn btn-lg"}>Close</Button>
</div>
</div>
)
}
}
export default ShowCatalog

View File

@ -13,10 +13,10 @@
*/
import xhr from './xhr';
const fetchColumData = (table) => xhr(`/api/metadata/columns/${table.url}`);
const fetchColumData = (table) => xhr(`../api/metadata/columns/${table.url}`);
const fetchPreviewData = (table, partition = {}) => {
let url = `/api/metadata/preview/${table.url}`;
let url = `../api/metadata/preview/${table.url}`;
if (partition.name && partition.value) {
url += '?' +
`partitionName=${partition.name}&` +
@ -26,7 +26,7 @@ const fetchPreviewData = (table, partition = {}) => {
return xhr(url);
};
const fetchPartitionData = (table) => xhr(`/api/metadata/partitions/${table.url}`);
const fetchPartitionData = (table) => xhr(`../api/metadata/partitions/${table.url}`);
export default {
fetchTableData(table) {

View File

@ -0,0 +1,21 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import React from "react";
import ReactDOM from "react-dom";
import {QueryMonitor} from "./components/QueryMonitor";
ReactDOM.render(
<QueryMonitor />,
document.getElementById('query-monitor')
);

Some files were not shown because too many files have changed in this diff Show More