diff --git a/hetu-docs/en/releasenotes/releasenotes-0.1.0.md b/hetu-docs/en/releasenotes/releasenotes-0.1.0.md index a335ed16b..debcc714b 100644 --- a/hetu-docs/en/releasenotes/releasenotes-0.1.0.md +++ b/hetu-docs/en/releasenotes/releasenotes-0.1.0.md @@ -1,6 +1,8 @@ # Release 0.1.0 +## Key Features + | Feature | Description | | -------------------------------------------- | ------------------------------------------------------------ | | Adaptive Dynamic Filter | The dynamic feature is enhanced so that in addition to bloom filter, hashsets can be used to store the build side values to filter out the probe side. The filters are stored in a distributed memory store so that they can be reused by subsequent queries without having to be rebuilt. | @@ -21,3 +23,7 @@ | High Availability (Active-Active) | Supports HA AA mode by storing runtime state information into a distributed cache like Hazelcast. Hazelcast cluster formation is done using a seed file. Discovery Service, OOM, & CPU Usage uses a distributed lock to ensure only one coordinator starts these services. | | Sql Migration Tool | Supplementary tool to assist in migrating the Hive SQL to openLooKeng Compatible SQLs | | Code Enhancement | RawSlice Optimization: This optimization reduces the memory footprint of openLooKeng by reusing the RawSlice object instead of constructing new Slice objects; Implicit Conversion: This feature supports data type implicit conversion. For example: If the query type does not match the table type in the Insert statement, it can convert the query type to the table type implicitly.| + +## Obtaining the Document + +For details, see [https://gitee.com/openlookeng/hetu-core/tree/010/hetu-docs/en](https://gitee.com/openlookeng/hetu-core/tree/010/hetu-docs/en) \ No newline at end of file diff --git a/hetu-docs/en/releasenotes/releasenotes-1.0.0.md b/hetu-docs/en/releasenotes/releasenotes-1.0.0.md index af342c8f6..3d5458736 100644 --- a/hetu-docs/en/releasenotes/releasenotes-1.0.0.md +++ b/hetu-docs/en/releasenotes/releasenotes-1.0.0.md @@ -1,5 +1,22 @@ # Release 1.0.0 +## Key Features + +This release focused on making improvements in 3 main areas: Performance, Security, and Usability. + +* Performance + + New enhancements were made to the engine to further improve the performance of ad-hoc interactive queries. These enhancements include changes to the dynamic filter feature to use a more efficient implementation of bloom filters, as well as optimizing the dynamic filter source operator so that dynamic filters can be collected and used as soon as possible. Other performance optimizations include the introduction of predicate pushdown to allow OR predicates to be pushed to the ORC reader, as well as supporting OR predicates during split filtering with the heuristic index feature. Finally, auto compaction was introduced for ORC files to reduce the number of ORC files after insert and update operations. + +* Security + + In this release the community focused on further securing the query engine. Several vulnerabilities were addressed in the code, and an audit log was introduced. Furthermore, a new feature was introduced to allow administrators to encrypt sensitive information like data source credentials found in catalog property files. + +* Usability + + Usability was another area of main focus for this release. In order to provide a database-like user experience, the community introduced a migration tool that helps migrate SQL queries from other engines to work in openLooKeng. Other usability enhancements include capturing ORC cache metrics via JMX to allow administrators to view cache hits and misses, and also improving the CACHE sql command to provide more flexibility as to what can be cached. Furthermore, changes were made to installation and deployment scripts to support deployments that contain a mix of ARM and x86 nodes. New scripts are introduced to allow administrators to deploy openLooKeng service based on containers. + + | Area | Feature | PR #s | | ----------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | | Security | Audit Logging | 160 | @@ -26,17 +43,8 @@ | | CACHE sql command usability enhancements to provide more flexibility as to what partitions can be cached. | 90, 99 | +## Obtaining the Document -This release focused on making improvements in 3 main areas: Performance, Security, and Usability. +For details, see [https://gitee.com/openlookeng/hetu-core/tree/1.0.0/hetu-docs/en](https://gitee.com/openlookeng/hetu-core/tree/1.0.0/hetu-docs/en) -**Performance** -New enhancements were made to the engine to further improve the performance of ad-hoc interactive queries. These enhancements include changes to the dynamic filter feature to use a more efficient implementation of bloom filters, as well as optimizing the dynamic filter source operator so that dynamic filters can be collected and used as soon as possible. Other performance optimizations include the introduction of predicate pushdown to allow OR predicates to be pushed to the ORC reader, as well as supporting OR predicates during split filtering with the heuristic index feature. Finally, auto compaction was introduced for ORC files to reduce the number of ORC files after insert and update operations. - -**Security** - -In this release the community focused on further securing the query engine. Several vulnerabilities were addressed in the code, and an audit log was introduced. Furthermore, a new feature was introduced to allow administrators to encrypt sensitive information like data source credentials found in catalog property files. - -**Usability** - -Usability was another area of main focus for this release. In order to provide a database-like user experience, the community introduced a migration tool that helps migrate SQL queries from other engines to work in openLooKeng. Other usability enhancements include capturing ORC cache metrics via JMX to allow administrators to view cache hits and misses, and also improving the CACHE sql command to provide more flexibility as to what can be cached. Furthermore, changes were made to installation and deployment scripts to support deployments that contain a mix of ARM and x86 nodes. New scripts are introduced to allow administrators to deploy openLooKeng service based on containers. \ No newline at end of file diff --git a/hetu-docs/en/releasenotes/releasenotes-1.0.1.md b/hetu-docs/en/releasenotes/releasenotes-1.0.1.md index 58aafaa49..03feec6da 100644 --- a/hetu-docs/en/releasenotes/releasenotes-1.0.1.md +++ b/hetu-docs/en/releasenotes/releasenotes-1.0.1.md @@ -1,9 +1,14 @@ # Release 1.0.1 +## Key Features + This release is mainly for fixing version number issue and index-cli issues. Besides, it has enhanced the code to improve the security. | Area | Feature | PR #s | | ----------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | | Index | Merge the index-cli into hetu-cli, and introduce the basic access control to index command | 283, 298 | | Security | Handle CVEs | 289,287,295,288,285,292，297,249,302 | -| Documentation | Update the documentation to reflect the latest code | 273,293,303 | + +## Obtaining the Document + +For details, see [https://gitee.com/openlookeng/hetu-core/tree/1.0.1/hetu-docs/en](https://gitee.com/openlookeng/hetu-core/tree/1.0.1/hetu-docs/en) \ No newline at end of file diff --git a/hetu-docs/en/releasenotes/releasenotes-1.1.0.md b/hetu-docs/en/releasenotes/releasenotes-1.1.0.md index d9487533b..a4773c2d8 100644 --- a/hetu-docs/en/releasenotes/releasenotes-1.1.0.md +++ b/hetu-docs/en/releasenotes/releasenotes-1.1.0.md @@ -1,5 +1,7 @@ # Release 1.1.0 +## Key Features + | Area | Feature | PR #s | | ---------------- | ------------------------------------------------------------ | ------------------------------- | | Heuristic Index | Btree index – BTree index is used for split filtering, and is used only by the coordinator nodes. If an index is created on a column which is part of a predicate in the query, then openLooKeng may be able to improve the performance of the query by filtering out splits during scheduling time. | 392,437,452,457 | @@ -18,9 +20,10 @@ | | Simplify the configurations | 397,369,449 | | Security | Support integration with Apache Ranger for unified permission control | 491 | -**Known Issues** -| **Category** | Description | Gitee issue | +## Known Issues + +| Category | Description | Gitee issue | | --------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | | Data Center Connector | If the Data center has many more than 3 tiers, and when the Join condition is on a small value, enabling dynamic filtering takes longer time than disabling dynamic filtering. This is because the column data is not high-cardinality, we need a better filter. | [I2BAZZ](https://gitee.com/openlookeng/hetu-core/issues/I2BAZZ) | | Heuristic Index | After drop index ,the index files folders would not be removed from HDFS. However the files inside them are removed | [I2BB1N](https://gitee.com/openlookeng/hetu-core/issues/I2BB1N) | @@ -31,3 +34,7 @@ | Installation | By using the auto deploy script, the system reused the old configuration file without any prompt messages, this will mislead user. | [I2BB52](https://gitee.com/openlookeng/hetu-core/issues/I2BB52) | | Vacuum | When a vacuum is run more than once on a table which had delete/update operations performed on it, the hive statistics might get corrupted and the queries would fail. To avoid this, set the session flag hive.collect_column_statistics_on_write to false (`set session hive.collect_column_statistics_on_write=false`) before running VACUUM command. In case the command was run without setting the above flag, run ANALYZE `table name` command to rectify the statistics. This issue is rectified in PR 517, but not merged into 1.1.0 release. | [I2BFH9](https://gitee.com/openlookeng/hetu-core/issues/I2BFH9) | | Reuse Exchange | When reuse exchange feature is enabled in config.properties (reuse_table_scan=true), querying non hive catalogs would fail. It is recommended use (set session reuse_table_scan=true;) when querying hive catalogs and disable it for other catalogs. This issue is rectified in PR 516, but not merged into 1.1.0 release. | [I2BEWV](https://gitee.com/openlookeng/hetu-core/issues/I2BEWV) | + +## Obtaining the Document + +For details, see [https://gitee.com/openlookeng/hetu-core/tree/1.1.0/hetu-docs/en](https://gitee.com/openlookeng/hetu-core/tree/1.1.0/hetu-docs/en) \ No newline at end of file diff --git a/hetu-docs/zh/releasenotes/releasenotes-0.1.0.md b/hetu-docs/zh/releasenotes/releasenotes-0.1.0.md index ca7e52cc5..98dcfe66f 100644 --- a/hetu-docs/zh/releasenotes/releasenotes-0.1.0.md +++ b/hetu-docs/zh/releasenotes/releasenotes-0.1.0.md @@ -1,6 +1,8 @@ # Release 0.1.0 +## 关键特性 + |特性|描述| | -------------------------------------------- | ------------------------------------------------------------ | | Adaptive Dynamic Filter |动态特性增强，除了bloom filter外，还可以使用hashsets存储build侧值，用于过滤探针侧。筛选器存储在分布式内存存储中，以便后续查询可以重用它们，而不必重新构建。| @@ -20,4 +22,8 @@ | Cross DC Connector |新的连接器被引入，以支持跨广域网的响应查询，允许客户端查询位于物理上遥远的另一个数据中心的数据源。| | High Availability (Active-Active) |支持HA AA模式，将运行时状态信息存储在Hazelcast等分布式缓存中。黑兹尔卡斯特集群的形成是使用种子文件完成的。发现服务、OOM、CPU使用率使用分布式锁，以确保只有一个协调器启动这些服务。| | Sql Migration Tool |协助Hive SQL迁移到openLooKeng兼容SQL的补充工具| -| 代码增强 |RawSlice Optimization:通过重用RawSlice对象，而不是构造新的Slice对象，减少openLooKeng的内存占用。Implicit Conversion ：支持数据类型隐式转换。例如：如果Insert语句中的查询类型与表类型不匹配，则可以将查询类型隐式转换为表类型。|| +| 代码增强 |RawSlice Optimization:通过重用RawSlice对象，而不是构造新的Slice对象，减少openLooKeng的内存占用。Implicit Conversion ：支持数据类型隐式转换。例如：如果Insert语句中的查询类型与表类型不匹配，则可以将查询类型隐式转换为表类型。| + +## 获取文档 + +请参考：[https://gitee.com/openlookeng/hetu-core/tree/010/hetu-docs/zh](https://gitee.com/openlookeng/hetu-core/tree/010/hetu-docs/zh) \ No newline at end of file diff --git a/hetu-docs/zh/releasenotes/releasenotes-1.0.0.md b/hetu-docs/zh/releasenotes/releasenotes-1.0.0.md index a3eb2b8b0..c841d3c0e 100644 --- a/hetu-docs/zh/releasenotes/releasenotes-1.0.0.md +++ b/hetu-docs/zh/releasenotes/releasenotes-1.0.0.md @@ -1,5 +1,23 @@ # Release 1.0.0 +## 关键特性 + +该版本主要在三个主要领域进行了改进：性能、安全性和可用性。 + +* 性能 + + 对引擎进行了新的性能特性增强，以进一步提高即席交互式查询的性能。这些增强包括：增强动态过滤器以使用更高效的bloom过滤器实现；以及优化动态过滤器的收集侧算子，以便能够尽快收集和应用动态过滤器。其他性能优化还包括引入谓词下推以允许OR谓词被推送到ORC读取器，以及启发式索引支持OR谓词对分片进行过滤。最后，为了减少插入和更新操作后ORC文件的数量，引入了ORC文件的自动压缩功能。 + +* 安全性 + + 在本版本中，社区聚焦在进一步强化引擎的安全性。代码中解决了多个漏洞，并引入了审计日志。此外，还引入了一个新的功能，允许管理员对Catalog配置文件的敏感信息进行加密处理，比如数据源凭据。 + +* 可用性 + + 可用性是本发行版的另一个重点关注的领域。为了提供数据库般的用户体验，社区引入了SQL迁移工具，该工具帮助将SQL语句从其他引擎迁移到openLooKeng中。其他可用性增强包括通过JMX监控ORC缓存的各项指标，以便管理员查看索引及数据缓存命中率；以及改进CACHE sql命令，以便为数据缓存提供更大的灵活性。此外，还对安装和部署脚本进行了更改，以支持包含ARM和x86节点的混合部署，并引入了新的脚本允许管理员基于容器部署openLooKeng服务。 + +详细特性描述见下表。 + | 类别 | 特性 | PR #s | | ------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | | 安全 | 新增审计日志功能。 | 160 | @@ -25,18 +43,6 @@ | ORC数据缓存 | 通过JMX监控ORC缓存的各项指标。 | 33、142 | | | CACHE sql命令的可用性增强，为选择缓存哪些分区提供了更大的灵活性。 | 90、99 | +## 获取文档 -此版本主要在三个主要领域进行了改进：性能、安全性和可用性。 - -**性能** - -对引擎进行了新的性能特性增强，以进一步提高即席交互式查询的性能。这些增强包括：改进动态过滤器以使用更高效的bloom过滤器实现；以及优化动态过滤器的源算子，以便能够尽快收集和应用动态过滤器。其他性能优化还包括引入谓词下推以允许OR谓词被推送到ORC读取器，以及启发式索引支持OR谓词对分片进行过滤。最后，为了减少插入和更新操作后ORC文件的数量，引入了ORC文件的自动压缩功能。 - -**安全** - -在本版本中，社区聚焦在进一步强化引擎的安全性。代码中解决了多个漏洞，并引入了审计日志。此外，还引入了一个新的功能，允许管理员对Catalog配置文件的敏感信息进行加密处理，比如数据源凭据。 - -**易用性** - -可用性是本发行版的另一个重点关注的领域。为了提供数据库般的用户体验，社区引入了SQL迁移工具，该工具帮助将SQL语句从其他引擎迁移到openLooKeng中。其他可用性增强包括通过JMX监控ORC缓存的各项指标，以便管理员查看索引及数据缓存命中率；以及改进CACHE sql命令，以便为数据缓存提供更大的灵活性。此外，还对安装和部署脚本进行了更改，以支持包含ARM和x86节点的混合部署，并引入了新的脚本允许管理员基于容器部署openLooKeng服务。 - +请参考：[https://gitee.com/openlookeng/hetu-core/tree/1.0.0/hetu-docs/zh](https://gitee.com/openlookeng/hetu-core/tree/1.0.0/hetu-docs/zh ) \ No newline at end of file diff --git a/hetu-docs/zh/releasenotes/releasenotes-1.0.1.md b/hetu-docs/zh/releasenotes/releasenotes-1.0.1.md index e596a9cf0..e9177fe6b 100644 --- a/hetu-docs/zh/releasenotes/releasenotes-1.0.1.md +++ b/hetu-docs/zh/releasenotes/releasenotes-1.0.1.md @@ -1,9 +1,14 @@ # Release 1.0.1 +## 关键特性 + 本次发布主要修复了版本号问题和Index-cli的问题。 此外，还在安全方面做了更多增强。 -| Area | Feature | PR #s | +| 类别 | 特性 | PR #s | | ----------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | | 索引 | 将Index cli合并到hetu-cli中，并为index的命令引入基本的权限控制 | 283, 298 | | 安全 | 处理CVEs问题 | 289,287,295,288,285,292，297,249,302 | -| 文档 | 更新文档以匹配最新的代码 | 273,293,303 | + +## 获取文档 + +请参考：[https://gitee.com/openlookeng/hetu-core/tree/1.0.1/hetu-docs/zh](https://gitee.com/openlookeng/hetu-core/tree/1.0.1/hetu-docs/zh ) \ No newline at end of file diff --git a/hetu-docs/zh/releasenotes/releasenotes-1.1.0.md b/hetu-docs/zh/releasenotes/releasenotes-1.1.0.md index 6b7f01264..a5c9e1999 100644 --- a/hetu-docs/zh/releasenotes/releasenotes-1.1.0.md +++ b/hetu-docs/zh/releasenotes/releasenotes-1.1.0.md @@ -1,5 +1,7 @@ # Release 1.1.0 +## 关键特性 + | 分类 | 特性 | PR #s | | ---------- | ------------------------------------------------------------ | ------------------------------- | | 启发式索引 | Btree索引-BTree索用于Split过滤，并且只能使用在Coordinator节点上，如果对作为查询谓词一部分的某一列创建索引，那么openLooKeng可以在执行的过程中通过索引来过滤掉Split，从而提高查询性能。 | 392,437,452,457 | @@ -18,9 +20,9 @@ | | 简化配置项 | 397,369,449 | | 安全 | 支持与Apache Ranger 集成以实现统一的权限控制 | 491 | -**已知问题** +## 已知问题 -| **分类** | 描述 | Gitee问题 | +| 分类 | 描述 | Gitee问题 | | ---------- | ------------------------------------------------------------ | ------------------------------------------------------------ | | DC连接器 | 分层的数据中心模型中，分层大于3层，开启动态过滤的耗时可能会比关闭动态过滤的耗时更长，由于过滤的列不是高基数的，会导致过滤效果变差，所以需要引入更高效的过滤器。 | [I2BAZZ](https://gitee.com/openlookeng/hetu-core/issues/I2BAZZ) | | 启发式索引 | 当删除index的时候，原HDFS上的文件夹并没有被删除，但里面的文件被删除。 | [I2BB1N](https://gitee.com/openlookeng/hetu-core/issues/I2BB1N) | @@ -30,4 +32,8 @@ | | 如果有很多计划信息，文字会溢出文字区域。 | [I2BB4E](https://gitee.com/openlookeng/hetu-core/issues/I2BB4E) | | 安装 | 使用自动部署脚本，系统会使用旧的配置，没有任何提示，这样会误导用户。 | [I2BB52](https://gitee.com/openlookeng/hetu-core/issues/I2BB52) | | Vacuum | 如果对已执行删除/更新操作的表多次运行vacuum操作，则统计信息可能会被损坏，并导致查询将失败。为了避免这种情况，请在运行vaccum命令前将会话标志hive.collect_column_statistics_on_write设置为false (`set session hive.collect_column_statistics_on_write=false`)。如果在运行该命令时没有设置上述参数flag，执行ANALYZE <表名>命令进行统计信息修正。PR 517已修改，未合入1.1.0版本 | [I2BFH9](https://gitee.com/openlookeng/hetu-core/issues/I2BFH9) | -| Reuse Exchange | 当在config.properties中启用Reuse Exchange特性（reuse_table_scan=true）时，查询非hive catalog会失败。建议在查询hive catalog时使用（设置session reuse_table_scan=true;），其他catalog时禁用。PR 516已修改，未合入1.1.0版本。 | [I2BEWV](https://gitee.com/openlookeng/hetu-core/issues/I2BEWV) | \ No newline at end of file +| Reuse Exchange | 当在config.properties中启用Reuse Exchange特性（reuse_table_scan=true）时，查询非hive catalog会失败。建议在查询hive catalog时使用（设置session reuse_table_scan=true;），其他catalog时禁用。PR 516已修改，未合入1.1.0版本。 | [I2BEWV](https://gitee.com/openlookeng/hetu-core/issues/I2BEWV) | + +## 获取文档 + +请参考：[https://gitee.com/openlookeng/hetu-core/tree/1.1.0/hetu-docs/zh](https://gitee.com/openlookeng/hetu-core/tree/1.1.0/hetu-docs/zh ) \ No newline at end of file