From 8a033df19d6a7604f4e78d44fe17555be0ddd04c Mon Sep 17 00:00:00 2001 From: Jack Guo Date: Thu, 17 Sep 2020 18:10:59 -0400 Subject: [PATCH] add tcp-ip for HA deployment documentation --- hetu-docs/en/installation/deployment-ha.md | 101 ++++++++++++++++++--- hetu-docs/zh/installation/deployment-ha.md | 93 ++++++++++++++++--- 2 files changed, 168 insertions(+), 26 deletions(-) diff --git a/hetu-docs/en/installation/deployment-ha.md b/hetu-docs/en/installation/deployment-ha.md index 7d5943d03..9b1cbe64b 100644 --- a/hetu-docs/en/installation/deployment-ha.md +++ b/hetu-docs/en/installation/deployment-ha.md @@ -4,32 +4,108 @@ The openLooKeng HA solves single point failure of coordinators. Users can submit queries to any coordinator to balance the workloads. ## Installing HA -openLooKeng with HA is required to be installed with minimum of 2 coordinators in the cluster. +openLooKeng with HA is required to be installed with minimum of 2 coordinators in the cluster. Make sure the time on all coordinators have been sync up. Please follow [Deploying openLooKeng Manually](./deployment.md) or [Deploying openLooKeng Automatically](./deployment-auto.md) for basic installation. ## Configuring HA +- There are 2 mechanisms of configuring HA + 1. TCP-IP : Coordinators discover each other based on their seeds (ie. coordinator's IP address). + 2. Multicast : Coordinators discover each other under the same network. -- Create an `etc\state-store.properties` file inside both coordinators and workers installation directories. +- Multicast mechanism is not recommended for production since UDP is often blocked in production environments. +### Configuring HA with TCP-IP -### State Store Properties +#### State Store Properties -The state store properties file, `etc/state-store.properties`, contains configuration to state store. -State store is used to store states that are shared between coordinators and workers. The following is a minimal configuration. +Create an `etc\state-store.properties` property file inside both coordinators and workers installation directories. + +State store is used to store states that are shared between coordinators and workers. ``` properties state-store.type=hazelcast state-store.name=query state-store.cluster=cluster1 + +hazelcast.discovery.mode=tcp-ip +hazelcast.discovery.port=5701 #optional, the default port is 5701 +``` + +The above properties are described below: +- `state-store.type`: The type of the state store. For now, only support hazelcast. +- `state-store.name`: User defined name of state store. +- `state-store.cluster`: User defined cluster name of state store. +- `hazelcast.discovery.mode` : The discovery mode of hazelcast state store, now support tcp-ip and multicast(default). +- `hazelcast.discovery.port` : The user defined port of hazelcast state store to launch. + +#### Seed Store Properties +Create an `etc\seed-store.properties` property file inside both coordinators and workers installation directories. + +Seed store is used to store seeds of coordinators to discover each other. + +``` +seed-store.type=filebased +seed-store.filesystem.seed-dir=/tmp/openlookeng/ha +seed-store.filesystem.profile=hdfs-config-default ``` The above properties are described below: -- `state-store.type`: The type of the state store. For now, only support hazelcast -- `state-store.name`: User defined name of state store -- `state-store.cluster`: User defined cluster name of state store +- `seed-store.type`: The type of the seed store. For now, only support filebased. +- `seed-store.filesystem.seed-dir`: The directory to store seeds. +- `seed-store.filesystem.profile.`: The name of filesystem configuration file under `etc\filesystem` directory. -### Coordinator and Worker Properties +#### Filesystem Properties + +Create an `etc\filesystem\hdfs-config-default.properties` property file inside both coordinators and workers installation directories. + +Filesystem is required to be distributed filesystem so that all coordinators and workers can access (ie. HDFS). +``` +fs.client.type=hdfs +hdfs.config.resources=/path/to/core-site.xml,/path/to/hdfs-site.xml +hdfs.authentication.type=NONE +fs.hdfs.impl.disable.cache=true +``` +The above properties are described in [Filesystem Access Utilities](../develop/filesystem.md). + +#### Coordinator and Worker Properties + +Add following configuration to `etc/config.properties` on all coordinators. + +``` properties +hetu.multiple-coordinator.enabled=true +hetu.embedded-state-store.enabled=true +hetu.seed-store.enabled=true +``` + +Add following configuration to ``etc/config.properties`` on all workers. + +``` properties +hetu.multiple-coordinator.enabled=true +hetu.seed-store.enabled=true +``` + +The above properties are described below: + +- `hetu.multiple-coordinator.enabled`: Multiple coordinator mode is enabled. +- `hetu.embedded-state-store.enabled`: Allow coordinators to use embedded state store. +- `hetu.seed-store.enabled`: Allow coordinator and worker to use seed store. + +### Configuring HA with Multicast + +#### State Store Properties + +Create an `etc\state-store.properties` file inside both coordinators and workers installation directories. +``` properties +state-store.type=hazelcast +state-store.name=query +state-store.cluster=cluster1 + +hazelcast.discovery.mode=multicast #optional, the default is multicast +hazelcast.discovery.port=5701 #optional, the default port is 5701 +``` + +#### Coordinator and Worker Properties Add following configuration to `etc/config.properties` on all coordinators. @@ -44,11 +120,6 @@ Add following configuration to ``etc/config.properties`` on all workers. hetu.multiple-coordinator.enabled=true ``` -The above properties are described below: - -- `hetu.multiple-coordinator.enabled`: Multiple coordinator mode is enabled -- `hetu.embedded-state-store.enabled`: Allow this coordinator to use embedded state store - ## HA Cluster behind Reverse Proxy To achieve the full benefit of HA, clients(ie. openLooKeng CLI, JDBC, etc) are encouraged to _not_ connect to specific coordinators directly. Instead, they should connect to multiple coordinators through some sort of _reverse proxy_, for example through a load balancer, or a Kubernetes service. This allows the client to continue to work even if a specific coordinator is not working as expected. @@ -59,7 +130,7 @@ When the connection is made through a reverse proxy, it is required for a given ### Configure Reverse Proxy Example (Nginx) -Please include configuration below in the Nginx configuration file (ie. `nginx.conf`) +Please include configuration below in the Nginx configuration file (ie. `nginx.conf`). ``` http { diff --git a/hetu-docs/zh/installation/deployment-ha.md b/hetu-docs/zh/installation/deployment-ha.md index a967eaa4b..b7c6a4805 100644 --- a/hetu-docs/zh/installation/deployment-ha.md +++ b/hetu-docs/zh/installation/deployment-ha.md @@ -5,20 +5,28 @@ openLooKeng HA解决协调节点单点故障问题。用户可以在任何协调 ## 安装HA -安装具有HA的openLooKeng要求集群中至少有2个协调节点。请按照[手动部署openLooKeng](deployment.md)或[自动部署openLooKeng](deployment-auto.md)进行基本设置。 +安装具有HA的openLooKeng要求集群中至少有2个协调节点。请确保协调节点上的时间一致性。请按照[手动部署openLooKeng](deployment.md)或[自动部署openLooKeng](deployment-auto.md)进行基本设置。 ## 配置HA +- 目前有2种方法配置HA + 1. TCP-IP : 协调节点根据种子（即协调节点ip地址）相互发现。 + 2. Multicast : 协调节点在同一网络下相互发现。 +- 不建议在生产环境中使用Multicast机制，因为UDP通常在生产环境中被阻止。 -- 在协调节点和工作节点安装目录中创建文件`etc\state-store.properties`。 +### TCP-IP方法配置HA -### 状态存储属性 +#### 状态存储属性 +在协调节点和工作节点安装目录中创建文件`etc\state-store.properties`。 -状态存储属性文件`etc/state-store.properties`包含状态存储的配置。状态存储用于与协调节点和工作节点共享的存储状态。以下是最基本配置。 +状态存储用于与协调节点和工作节点共享的存储状态。 ``` properties state-store.type=hazelcast state-store.name=query state-store.cluster=cluster1 + +hazelcast.discovery.mode=tcp-ip +hazelcast.discovery.port=5701 #可选，默认端口为5701 ``` 上述属性说明如下： @@ -26,8 +34,76 @@ state-store.cluster=cluster1 - `state-store.type`：状态存储的类型。目前仅支持Hazelcast。 - `state-store.name`：用户定义的状态存储名称。 - `state-store.cluster`：用户定义的状态存储集群名称。 +- `hazelcast.discovery.mode` : Hazelcast状态存储发现模式, 目前支持tcp-ip和multicast（默认）。 +- `hazelcast.discovery.port` : 用户定义hazelcast启动端口。 -### 协调节点和工作节点属性 +#### 种子存储属性 +在协调节点和工作节点安装目录中创建文件`etc\seed-store.properties`。 + +种子存储用于存储协调节点的种子以相互发现。 + +``` +seed-store.type=filebased +seed-store.filesystem.seed-dir=/tmp/openlookeng/ha +seed-store.filesystem.profile=hdfs-config-default +``` + +上述属性说明如下： + +- `seed-store.type`: 种子存储的类型。目前仅支持Filebased。 +- `seed-store.filesystem.seed-dir`: 存储种子的目录。 +- `seed-store.filesystem.profile.`: `etc\filesystem`目录下的文件系统中配置文件的名称。 + + +#### 文件系统属性 + +在协调节点和工作节点安装目录中创建文件`etc\filesystem\hdfs-config-default.properties`。 + +文件系统必须是分布式文件系统，以便所有协调节点和工作节点都可以访问（例如HDFS）。 +``` +fs.client.type=hdfs +hdfs.config.resources=/path/to/core-site.xml,/path/to/hdfs-site.xml +hdfs.authentication.type=NONE +fs.hdfs.impl.disable.cache=true +``` +上述属性说明请参考[文件系统访问实用程序](../develop/filesystem.md)。 + +#### 协调节点和工作节点属性 + +在所有协调节点上的`etc/config.properties`中添加如下配置。 + +``` properties +hetu.multiple-coordinator.enabled=true +hetu.embedded-state-store.enabled=true +hetu.seed-store.enabled=true +``` + +在所有工作节点上的`etc/config.properties`中添加如下配置。 + +``` properties +hetu.multiple-coordinator.enabled=true +hetu.seed-store.enabled=true +``` + +上述属性说明如下： + +- `hetu.multiple-coordinator.enabled`：开启了多协调节点模式。 +- `hetu.embedded-state-store.enabled`：允许协调节点使用嵌入式状态存储。 +- `hetu.seed-store.enabled`: 允许协调节点和工作节点使用嵌入式状态存储。 + +### Multicast方法配置HA + +#### 状态存储属性 + +状态存储属性文件`etc/state-store.properties`包含状态存储的配置。 + +``` properties +state-store.type=hazelcast +state-store.name=query +state-store.cluster=cluster1 +``` + +#### 协调节点和工作节点属性 在所有协调节点上的`etc/config.properties`中添加如下配置。 @@ -42,11 +118,6 @@ hetu.embedded-state-store.enabled=true hetu.multiple-coordinator.enabled=true ``` -上述属性说明如下： - -- `hetu.multiple-coordinator.enabled`：开启了多协调节点模式。 -- `hetu.embedded-state-store.enabled`：允许此协调节点使用嵌入式状态存储。 - ## HA及反向代理 要尽可能的显示HA的优势，建议客户端(例如openLooKeng CLI, JDBC等)不直接连接特定的协调节点，而是通过反向代理连接多个协调节点，例如使用负载平衡器或者Kubernetes服务，这样即使某个协调节点无法正常工作，客户端也可以继续使用其它的协调节点。 @@ -57,7 +128,7 @@ hetu.multiple-coordinator.enabled=true ### 配置反向代理 (Nginx) -请在Nginx配置文件中包含以下配置（nginx.conf） +请在Nginx配置文件中包含以下配置（nginx.conf）。 ``` http {