!1068 Add documentation explaining Hetu Metastore and State Store setup with Memory Connector

Merge pull request !1068 from arjunkrishna/global-cache-docs

hetu-docs/en/connector/memory.md

@@ -16,7 +16,66 @@ For example, you can write:
 connector.name=memory
 memory.max-data-per-node=200GB
 memory.spill-path=/opt/hetu/data/spill          
-```   
+```
+#### Other Required Configurations
+- This section will cover other configurations required to use Memory Connector.
+- Refer to the documentation of [Hetu Metastore](../admin/meta-store.md) and [State Store](../admin/state-store.md) for more information.
+
+  ##### Single Node Setup
+  - This section will give an example configuration for Memory Connector on a single node cluster.
+  - Create a file `etc/catalog/memory.properties` with the following information:
+  ``` properties
+  connector.name=memory
+  memory.max-data-per-node=200GB
+  memory.spill-path=/opt/hetu/data/spill
+  ```
+  - Create the file `etc/hetu-metastore.properties` with these configurations:
+  ```properties
+  hetu.metastore.type=hetufilesystem
+  hetu.metastore.hetufilesystem.profile-name=default
+  hetu.metastore.hetufilesystem.path=/tmp/hetu/metastore
+  hetu.metastore.cache.type=local
+  ```
+  ##### Multi-Node Setup
+  - This section will give an example configuration for Memory Connector an a cluster with more than one node.
+  - Create a file `etc/catalog/memory.properties` with the following information:
+  ``` properties
+  connector.name=memory
+  memory.max-data-per-node=200GB
+  memory.spill-path=/opt/hetu/data/spill
+  ```
+  - Add this line in `etc/config.properties` to enable State Store:
+      - State Store allows Memory Connector to automatically clean up dropped tables, otherwise tables will only be cleaned up when another table is created.
+  ```properties
+  hetu.embedded-state-store.enabled=true
+  ```
+  - Create a file called `etc/state-store.properties` with the following contents:
+  ```properties
+  state-store.type=hazelcast
+  state-store.name=test
+  state-store.cluster=test-cluster
+  hazelcast.discovery.mode=tcp-ip
+  hazelcast.discovery.port=7980
+  # The ip address and hazelcast discovery ports of each server should be included here
+  # formatted like `hazelcast.discovery.tcp-ip.seeds=host1:port,host2:port` 
+  hazelcast.discovery.tcp-ip.seeds=host1:7980, host2:7980
+  ```
+  - Create the file `etc/hetu-metastore.properties` with these configurations:
+  ```properties
+  hetu.metastore.type=hetufilesystem
+  hetu.metastore.hetufilesystem.profile-name=hdfs
+  hetu.metastore.hetufilesystem.path=/tmp/hetu/metastore
+  # make sure to use global cache!
+  hetu.metastore.cache.type=global
+  ```
+  - Create the file `etc/filesystem/hdfs.properties`  to direct openLooKeng to your hdfs file system:
+  ```properties
+  fs.client.type=hdfs
+  # Path to hdfs resource files (e.g. core-site.xml, hdfs-site.xml)
+  hdfs.config.resources=/tmp/hetu/hdfs-site.xml
+  # hdfs authentication, accepted values: KERBEROS/NONE
+  hdfs.authentication.type=NONE
+  ```
 
 **Note:**
 - `spill-path` should be set to a directory with enough free space to hold
@@ -26,12 +85,6 @@ memory.spill-path=/opt/hetu/data/spill
 - In `etc/config.properties` ensure that `task.writer-count` is set to
  `>=` number of nodes in the cluster running openLooKeng. This will help
   distribute the data uniformly between all the workers.
-- Hetu Metastore must be configured. The default settings are included in
-  `etc/hetu-metastore.properties`. 
-    - Check [Hetu Metastore](../admin/meta-store.md) for more information. 
-- State Store must be configured to enable automatic memory refresh on workers.
-    - Check [State Store](../admin/state-store.md) for more information
-    - Automatic memory refresh will allow Memory Connector to clean unused tables more often resulting in more efficient use of memory.
 
 Examples
 --------
@@ -158,7 +211,6 @@ For queries containing > >= < <= BETWEEN IN similar logic is applied.
 Limitations and known Issues
 ---------------------------------------------
 
-- After `DROP TABLE`, memory is not released immediately. It is released on next `CREATE TABLE` operation.
-    - A simple workaround is to create a small temporary table to trigger a cleanup `CREATE TABLE memory.default.tmp AS SELECT * FROM tpch.tiny.nation;`
+- Without State Store and Hetu Metastore with global cache, after `DROP TABLE`, memory is not released immediately on the workers. It is released on the next `CREATE TABLE` operation.
 - Currently only a single column in ascending order is supported by `sorted_by`
 - If a CTAS (CREATE TABLE AS) query fails or is cancelled, an invalid table will remain. This table must be dropped manually.

hetu-docs/zh/connector/memory.md

@@ -16,14 +16,69 @@ memory.max-data-per-node=200GB
 memory.spill-path=/opt/hetu/data/spill    
 ```
 
+#### 其他必要的配置
+- 本节将介绍使用内存连接器所需的其他配置。
+- 更多信息请参考[Hetu Metastore](../admin/meta-store.md)和[State Store](../admin/state-store.md)的文档。
+  ##### 单个节点设置
+  - 本节将给出单节点集群上内存连接器的示例配置。
+  - 创建文件 `etc/catalog/memory.properties`并填入以下配置:
+  ``` properties
+  connector.name=memory
+  memory.max-data-per-node=200GB
+  memory.spill-path=/opt/hetu/data/spill
+  ```
+  - 创建文件 `etc/hetu-metastore.properties` 并填入以下配置:
+  ```properties
+  hetu.metastore.type=hetufilesystem
+  hetu.metastore.hetufilesystem.profile-name=default
+  hetu.metastore.hetufilesystem.path=/tmp/hetu/metastore
+  hetu.metastore.cache.type=local
+  ```
+  ##### 多节点设置
+  - 本节将为具有多个节点的集群提供内存连接器的示例配置。
+  - 创建文件 `etc/catalog/memory.properties` 并填入以下配置:
+  ``` properties
+  connector.name=memory
+  memory.max-data-per-node=200GB
+  memory.spill-path=/opt/hetu/data/spill
+  ```
+  - 在 `etc/config.properties` 文件中加入以下代码来启用 State Store:
+    - State Store 允许 Memory Connector 自动清理删除的表，否则只有在创建另一个表时才会清理表。
+  ```properties
+  hetu.embedded-state-store.enabled=true
+  ```
+  - 创建文件 `etc/state-store.properties` 并填入以下配置:
+  ```properties
+  state-store.type=hazelcast
+  state-store.name=test
+  state-store.cluster=test-cluster
+  hazelcast.discovery.mode=tcp-ip
+  hazelcast.discovery.port=7980
+  # 每个服务器的ip地址和hazelcast端口应该被声明在这里。
+  # 格式：`hazelcast.discovery.tcp-ip.seeds=host1:port,host2:port` 
+  hazelcast.discovery.tcp-ip.seeds=host1:7980, host2:7980
+  ```
+  - 创建文件 `etc/hetu-metastore.properties` 并填入以下配置:
+  ```properties
+  hetu.metastore.type=hetufilesystem
+  hetu.metastore.hetufilesystem.profile-name=hdfs
+  hetu.metastore.hetufilesystem.path=/tmp/hetu/metastore
+  # 确认使用全局缓存!
+  hetu.metastore.cache.type=global
+  ```
+  - 创建文件 `etc/filesystem/hdfs.properties`  使openLooKeng使用相应的文件系统:
+  ```properties
+  fs.client.type=hdfs
+  # Path to hdfs resource files (e.g. core-site.xml, hdfs-site.xml)
+  hdfs.config.resources=/tmp/hetu/hdfs-site.xml
+  # hdfs authentication, accepted values: KERBEROS/NONE
+  hdfs.authentication.type=NONE
+  ```
+
 **提示：**
 - `spill-path`必须设置为一个有充足存储空间的路径。推荐使用SSD以获得更好性能。 可以依照需求自定义。 
 - 关于更多详细信息与其他可选配置项，请参见**配置属性** 章节。
 - 在`etc/config.properties`中，请确保`task.writer-count`的数字不小于配置的openLooKeng集群的节点个数。这会帮助把所有数据更均匀地分配到各个节点上。
-- Hetu Metastore必须被妥善配置来保证内存连接器的正常功能。请参阅[Hetu Metastore](../admin/meta-store.md)。
-- 必须配置StateStore使得worker上的表自动刷新生效
-  - 请参阅[State Store](../admin/state-store.md)
-  - 自动刷新特性使得worker节点定期从metastore获取最新的表的列表，并在本地清理已经删除的表
 ## 示例
 
 使用内存连接器创建表：
@@ -142,8 +197,7 @@ LogicalPart 中创建了布隆过滤器、稀疏索引和 MinMax 索引。
 
 ## 内存连接器限制和已知问题
 
-- `DROP TABLE`之后，worker上的内存没有立即释放。内存在下一次对内存连接器进行创建操作后释放。
-    - 可以通过创建一个临时的表来在worker上强制清理，如`CREATE TABLE memory.default.tmp AS SELECT * FROM tpch.tiny.nation;`
+- 如果没有 State Store 和带有全局缓存的 Hetu Metastore，在 `DROP TABLE` 之后，内存不会立即释放到 worker 上。它将在下一个“CREATE TABLE”操作时被释放。
 - 当前`sorted_by`只支持按一个列排序。
 - 如果一个CTAS (CREATE TABLE AS)查询失败或被取消，一个无效的表的记录会留在系统中。该表将需要被手动删除。