From 170091e26d33a587c8c411a589619c34ce32bc82 Mon Sep 17 00:00:00 2001 From: arjun-krishna1 Date: Thu, 12 Aug 2021 15:10:26 -0400 Subject: [PATCH 1/2] add sample Hetu Metastore and State Store config to mem conn docs --- hetu-docs/en/connector/memory.md | 70 ++++++++++++++++++++++++++++---- 1 file changed, 61 insertions(+), 9 deletions(-) diff --git a/hetu-docs/en/connector/memory.md b/hetu-docs/en/connector/memory.md index e830ae52a..518943e73 100644 --- a/hetu-docs/en/connector/memory.md +++ b/hetu-docs/en/connector/memory.md @@ -16,7 +16,66 @@ For example, you can write: connector.name=memory memory.max-data-per-node=200GB memory.spill-path=/opt/hetu/data/spill -``` +``` +#### Other Required Configurations +- This section will cover other configurations required to use Memory Connector. +- Refer to the documentation of [Hetu Metastore](../admin/meta-store.md) and [State Store](../admin/state-store.md) for more information. + + ##### Single Node Setup + - This section will give an example configuration for Memory Connector on a single node cluster. + - Create a file `etc/catalog/memory.properties` with the following information: + ``` properties + connector.name=memory + memory.max-data-per-node=200GB + memory.spill-path=/opt/hetu/data/spill + ``` + - Create the file `etc/hetu-metastore.properties` with these configurations: + ```properties + hetu.metastore.type=hetufilesystem + hetu.metastore.hetufilesystem.profile-name=default + hetu.metastore.hetufilesystem.path=/tmp/hetu/metastore + hetu.metastore.cache.type=local + ``` + ##### Multi-Node Setup + - This section will give an example configuration for Memory Connector an a cluster with more than one node. + - Create a file `etc/catalog/memory.properties` with the following information: + ``` properties + connector.name=memory + memory.max-data-per-node=200GB + memory.spill-path=/opt/hetu/data/spill + ``` + - Add this line in `etc/config.properties` to enable State Store: + - State Store allows Memory Connector to automatically clean up dropped tables, otherwise tables will only be cleaned up when another table is created. + ```properties + hetu.embedded-state-store.enabled=true + ``` + - Create a file called `etc/state-store.properties` with the following contents: + ```properties + state-store.type=hazelcast + state-store.name=test + state-store.cluster=test-cluster + hazelcast.discovery.mode=tcp-ip + hazelcast.discovery.port=7980 + # The ip address and hazelcast discovery ports of each server should be included here + # formatted like `hazelcast.discovery.tcp-ip.seeds=host1:port,host2:port` + hazelcast.discovery.tcp-ip.seeds=host1:7980, host2:7980 + ``` + - Create the file `etc/hetu-metastore.properties` with these configurations: + ```properties + hetu.metastore.type=hetufilesystem + hetu.metastore.hetufilesystem.profile-name=hdfs + hetu.metastore.hetufilesystem.path=/tmp/hetu/metastore + # make sure to use global cache! + hetu.metastore.cache.type=global + ``` + - Create the file `etc/filesystem/hdfs.properties` to direct openLooKeng to your hdfs file system: + ```properties + fs.client.type=hdfs + # Path to hdfs resource files (e.g. core-site.xml, hdfs-site.xml) + hdfs.config.resources=/tmp/hetu/hdfs-site.xml + # hdfs authentication, accepted values: KERBEROS/NONE + hdfs.authentication.type=NONE + ``` **Note:** - `spill-path` should be set to a directory with enough free space to hold @@ -26,12 +85,6 @@ memory.spill-path=/opt/hetu/data/spill - In `etc/config.properties` ensure that `task.writer-count` is set to `>=` number of nodes in the cluster running openLooKeng. This will help distribute the data uniformly between all the workers. -- Hetu Metastore must be configured. The default settings are included in - `etc/hetu-metastore.properties`. - - Check [Hetu Metastore](../admin/meta-store.md) for more information. -- State Store must be configured to enable automatic memory refresh on workers. - - Check [State Store](../admin/state-store.md) for more information - - Automatic memory refresh will allow Memory Connector to clean unused tables more often resulting in more efficient use of memory. Examples -------- @@ -158,7 +211,6 @@ For queries containing > >= < <= BETWEEN IN similar logic is applied. Limitations and known Issues --------------------------------------------- -- After `DROP TABLE`, memory is not released immediately. It is released on next `CREATE TABLE` operation. - - A simple workaround is to create a small temporary table to trigger a cleanup `CREATE TABLE memory.default.tmp AS SELECT * FROM tpch.tiny.nation;` +- Without State Store and Hetu Metastore with global cache, after `DROP TABLE`, memory is not released immediately on the workers. It is released on the next `CREATE TABLE` operation. - Currently only a single column in ascending order is supported by `sorted_by` - If a CTAS (CREATE TABLE AS) query fails or is cancelled, an invalid table will remain. This table must be dropped manually. \ No newline at end of file From 45df6a8b33fd731b1a3eb6acaabd0d0c7fd51a8a Mon Sep 17 00:00:00 2001 From: peiwangdb Date: Wed, 18 Aug 2021 13:03:40 -0700 Subject: [PATCH 2/2] Memory Connector: keep zh-version up-to-date --- hetu-docs/zh/connector/memory.md | 66 +++++++++++++++++++++++++++++--- 1 file changed, 60 insertions(+), 6 deletions(-) diff --git a/hetu-docs/zh/connector/memory.md b/hetu-docs/zh/connector/memory.md index 0d7a01d22..46c90f17e 100644 --- a/hetu-docs/zh/connector/memory.md +++ b/hetu-docs/zh/connector/memory.md @@ -16,14 +16,69 @@ memory.max-data-per-node=200GB memory.spill-path=/opt/hetu/data/spill ``` +#### 其他必要的配置 +- 本节将介绍使用内存连接器所需的其他配置。 +- 更多信息请参考[Hetu Metastore](../admin/meta-store.md)和[State Store](../admin/state-store.md)的文档。 + ##### 单个节点设置 + - 本节将给出单节点集群上内存连接器的示例配置。 + - 创建文件 `etc/catalog/memory.properties`并填入以下配置: + ``` properties + connector.name=memory + memory.max-data-per-node=200GB + memory.spill-path=/opt/hetu/data/spill + ``` + - 创建文件 `etc/hetu-metastore.properties` 并填入以下配置: + ```properties + hetu.metastore.type=hetufilesystem + hetu.metastore.hetufilesystem.profile-name=default + hetu.metastore.hetufilesystem.path=/tmp/hetu/metastore + hetu.metastore.cache.type=local + ``` + ##### 多节点设置 + - 本节将为具有多个节点的集群提供内存连接器的示例配置。 + - 创建文件 `etc/catalog/memory.properties` 并填入以下配置: + ``` properties + connector.name=memory + memory.max-data-per-node=200GB + memory.spill-path=/opt/hetu/data/spill + ``` + - 在 `etc/config.properties` 文件中加入以下代码来启用 State Store: + - State Store 允许 Memory Connector 自动清理删除的表，否则只有在创建另一个表时才会清理表。 + ```properties + hetu.embedded-state-store.enabled=true + ``` + - 创建文件 `etc/state-store.properties` 并填入以下配置: + ```properties + state-store.type=hazelcast + state-store.name=test + state-store.cluster=test-cluster + hazelcast.discovery.mode=tcp-ip + hazelcast.discovery.port=7980 + # 每个服务器的ip地址和hazelcast端口应该被声明在这里。 + # 格式：`hazelcast.discovery.tcp-ip.seeds=host1:port,host2:port` + hazelcast.discovery.tcp-ip.seeds=host1:7980, host2:7980 + ``` + - 创建文件 `etc/hetu-metastore.properties` 并填入以下配置: + ```properties + hetu.metastore.type=hetufilesystem + hetu.metastore.hetufilesystem.profile-name=hdfs + hetu.metastore.hetufilesystem.path=/tmp/hetu/metastore + # 确认使用全局缓存! + hetu.metastore.cache.type=global + ``` + - 创建文件 `etc/filesystem/hdfs.properties` 使openLooKeng使用相应的文件系统: + ```properties + fs.client.type=hdfs + # Path to hdfs resource files (e.g. core-site.xml, hdfs-site.xml) + hdfs.config.resources=/tmp/hetu/hdfs-site.xml + # hdfs authentication, accepted values: KERBEROS/NONE + hdfs.authentication.type=NONE + ``` + **提示：** - `spill-path`必须设置为一个有充足存储空间的路径。推荐使用SSD以获得更好性能。 可以依照需求自定义。 - 关于更多详细信息与其他可选配置项，请参见**配置属性** 章节。 - 在`etc/config.properties`中，请确保`task.writer-count`的数字不小于配置的openLooKeng集群的节点个数。这会帮助把所有数据更均匀地分配到各个节点上。 -- Hetu Metastore必须被妥善配置来保证内存连接器的正常功能。请参阅[Hetu Metastore](../admin/meta-store.md)。 -- 必须配置StateStore使得worker上的表自动刷新生效 - - 请参阅[State Store](../admin/state-store.md) - - 自动刷新特性使得worker节点定期从metastore获取最新的表的列表，并在本地清理已经删除的表 ## 示例 使用内存连接器创建表： @@ -142,8 +197,7 @@ LogicalPart 中创建了布隆过滤器、稀疏索引和 MinMax 索引。 ## 内存连接器限制和已知问题 -- `DROP TABLE`之后，worker上的内存没有立即释放。内存在下一次对内存连接器进行创建操作后释放。 - - 可以通过创建一个临时的表来在worker上强制清理，如`CREATE TABLE memory.default.tmp AS SELECT * FROM tpch.tiny.nation;` +- 如果没有 State Store 和带有全局缓存的 Hetu Metastore，在 `DROP TABLE` 之后，内存不会立即释放到 worker 上。它将在下一个“CREATE TABLE”操作时被释放。 - 当前`sorted_by`只支持按一个列排序。 - 如果一个CTAS (CREATE TABLE AS)查询失败或被取消，一个无效的表的记录会留在系统中。该表将需要被手动删除。