add tcp-ip for HA deployment documentation
This commit is contained in:
Jack Guo
parent
20b66f53cf
commit
8a033df19d
|
|
@ -4,32 +4,108 @@
|
|||
The openLooKeng HA solves single point failure of coordinators. Users can submit queries to any coordinator to balance the workloads.
|
||||
|
||||
## Installing HA
|
||||
openLooKeng with HA is required to be installed with minimum of 2 coordinators in the cluster.
|
||||
openLooKeng with HA is required to be installed with minimum of 2 coordinators in the cluster. Make sure the time on all coordinators have been sync up.
|
||||
Please follow [Deploying openLooKeng Manually](./deployment.md) or [Deploying openLooKeng Automatically](./deployment-auto.md) for basic installation.
|
||||
|
||||
## Configuring HA
|
||||
- There are 2 mechanisms of configuring HA
|
||||
1. TCP-IP : Coordinators discover each other based on their seeds (ie. coordinator's IP address).
|
||||
2. Multicast : Coordinators discover each other under the same network.
|
||||
|
||||
- Create an `etc\state-store.properties` file inside both coordinators and workers installation directories.
|
||||
- Multicast mechanism is not recommended for production since UDP is often blocked in production environments.
|
||||
|
||||
### Configuring HA with TCP-IP
|
||||
|
||||
### State Store Properties
|
||||
#### State Store Properties
|
||||
|
||||
The state store properties file, `etc/state-store.properties`, contains configuration to state store.
|
||||
State store is used to store states that are shared between coordinators and workers. The following is a minimal configuration.
|
||||
Create an `etc\state-store.properties` property file inside both coordinators and workers installation directories.
|
||||
|
||||
State store is used to store states that are shared between coordinators and workers.
|
||||
|
||||
``` properties
|
||||
state-store.type=hazelcast
|
||||
state-store.name=query
|
||||
state-store.cluster=cluster1
|
||||
|
||||
hazelcast.discovery.mode=tcp-ip
|
||||
hazelcast.discovery.port=5701 #optional, the default port is 5701
|
||||
```
|
||||
|
||||
The above properties are described below:
|
||||
- `state-store.type`: The type of the state store. For now, only support hazelcast.
|
||||
- `state-store.name`: User defined name of state store.
|
||||
- `state-store.cluster`: User defined cluster name of state store.
|
||||
- `hazelcast.discovery.mode` : The discovery mode of hazelcast state store, now support tcp-ip and multicast(default).
|
||||
- `hazelcast.discovery.port` : The user defined port of hazelcast state store to launch.
|
||||
|
||||
#### Seed Store Properties
|
||||
Create an `etc\seed-store.properties` property file inside both coordinators and workers installation directories.
|
||||
|
||||
Seed store is used to store seeds of coordinators to discover each other.
|
||||
|
||||
```
|
||||
seed-store.type=filebased
|
||||
seed-store.filesystem.seed-dir=/tmp/openlookeng/ha
|
||||
seed-store.filesystem.profile=hdfs-config-default
|
||||
```
|
||||
|
||||
The above properties are described below:
|
||||
|
||||
- `state-store.type`: The type of the state store. For now, only support hazelcast
|
||||
- `state-store.name`: User defined name of state store
|
||||
- `state-store.cluster`: User defined cluster name of state store
|
||||
- `seed-store.type`: The type of the seed store. For now, only support filebased.
|
||||
- `seed-store.filesystem.seed-dir`: The directory to store seeds.
|
||||
- `seed-store.filesystem.profile.`: The name of filesystem configuration file under `etc\filesystem` directory.
|
||||
|
||||
### Coordinator and Worker Properties
|
||||
#### Filesystem Properties
|
||||
|
||||
Create an `etc\filesystem\hdfs-config-default.properties` property file inside both coordinators and workers installation directories.
|
||||
|
||||
Filesystem is required to be distributed filesystem so that all coordinators and workers can access (ie. HDFS).
|
||||
```
|
||||
fs.client.type=hdfs
|
||||
hdfs.config.resources=/path/to/core-site.xml,/path/to/hdfs-site.xml
|
||||
hdfs.authentication.type=NONE
|
||||
fs.hdfs.impl.disable.cache=true
|
||||
```
|
||||
The above properties are described in [Filesystem Access Utilities](../develop/filesystem.md).
|
||||
|
||||
#### Coordinator and Worker Properties
|
||||
|
||||
Add following configuration to `etc/config.properties` on all coordinators.
|
||||
|
||||
``` properties
|
||||
hetu.multiple-coordinator.enabled=true
|
||||
hetu.embedded-state-store.enabled=true
|
||||
hetu.seed-store.enabled=true
|
||||
```
|
||||
|
||||
Add following configuration to ``etc/config.properties`` on all workers.
|
||||
|
||||
``` properties
|
||||
hetu.multiple-coordinator.enabled=true
|
||||
hetu.seed-store.enabled=true
|
||||
```
|
||||
|
||||
The above properties are described below:
|
||||
|
||||
- `hetu.multiple-coordinator.enabled`: Multiple coordinator mode is enabled.
|
||||
- `hetu.embedded-state-store.enabled`: Allow coordinators to use embedded state store.
|
||||
- `hetu.seed-store.enabled`: Allow coordinator and worker to use seed store.
|
||||
|
||||
### Configuring HA with Multicast
|
||||
|
||||
#### State Store Properties
|
||||
|
||||
Create an `etc\state-store.properties` file inside both coordinators and workers installation directories.
|
||||
``` properties
|
||||
state-store.type=hazelcast
|
||||
state-store.name=query
|
||||
state-store.cluster=cluster1
|
||||
|
||||
hazelcast.discovery.mode=multicast #optional, the default is multicast
|
||||
hazelcast.discovery.port=5701 #optional, the default port is 5701
|
||||
```
|
||||
|
||||
#### Coordinator and Worker Properties
|
||||
|
||||
Add following configuration to `etc/config.properties` on all coordinators.
|
||||
|
||||
|
|
@ -44,11 +120,6 @@ Add following configuration to ``etc/config.properties`` on all workers.
|
|||
hetu.multiple-coordinator.enabled=true
|
||||
```
|
||||
|
||||
The above properties are described below:
|
||||
|
||||
- `hetu.multiple-coordinator.enabled`: Multiple coordinator mode is enabled
|
||||
- `hetu.embedded-state-store.enabled`: Allow this coordinator to use embedded state store
|
||||
|
||||
## HA Cluster behind Reverse Proxy
|
||||
|
||||
To achieve the full benefit of HA, clients(ie. openLooKeng CLI, JDBC, etc) are encouraged to _not_ connect to specific coordinators directly. Instead, they should connect to multiple coordinators through some sort of _reverse proxy_, for example through a load balancer, or a Kubernetes service. This allows the client to continue to work even if a specific coordinator is not working as expected.
|
||||
|
|
@ -59,7 +130,7 @@ When the connection is made through a reverse proxy, it is required for a given
|
|||
|
||||
### Configure Reverse Proxy Example (Nginx)
|
||||
|
||||
Please include configuration below in the Nginx configuration file (ie. `nginx.conf`)
|
||||
Please include configuration below in the Nginx configuration file (ie. `nginx.conf`).
|
||||
|
||||
```
|
||||
http {
|
||||
|
|
|
|||
|
|
@ -5,20 +5,28 @@ openLooKeng HA解决协调节点单点故障问题。用户可以在任何协调
|
|||
|
||||
## 安装HA
|
||||
|
||||
安装具有HA的openLooKeng要求集群中至少有2个协调节点。请按照[手动部署openLooKeng](deployment.md)或[自动部署openLooKeng](deployment-auto.md)进行基本设置。
|
||||
安装具有HA的openLooKeng要求集群中至少有2个协调节点。请确保协调节点上的时间一致性。请按照[手动部署openLooKeng](deployment.md)或[自动部署openLooKeng](deployment-auto.md)进行基本设置。
|
||||
|
||||
## 配置HA
|
||||
- 目前有2种方法配置HA
|
||||
1. TCP-IP : 协调节点根据种子（即协调节点ip地址）相互发现。
|
||||
2. Multicast : 协调节点在同一网络下相互发现。
|
||||
- 不建议在生产环境中使用Multicast机制，因为UDP通常在生产环境中被阻止。
|
||||
|
||||
- 在协调节点和工作节点安装目录中创建文件`etc\state-store.properties`。
|
||||
### TCP-IP方法配置HA
|
||||
|
||||
### 状态存储属性
|
||||
#### 状态存储属性
|
||||
在协调节点和工作节点安装目录中创建文件`etc\state-store.properties`。
|
||||
|
||||
状态存储属性文件`etc/state-store.properties`包含状态存储的配置。状态存储用于与协调节点和工作节点共享的存储状态。以下是最基本配置。
|
||||
状态存储用于与协调节点和工作节点共享的存储状态。
|
||||
|
||||
``` properties
|
||||
state-store.type=hazelcast
|
||||
state-store.name=query
|
||||
state-store.cluster=cluster1
|
||||
|
||||
hazelcast.discovery.mode=tcp-ip
|
||||
hazelcast.discovery.port=5701 #可选，默认端口为5701
|
||||
```
|
||||
|
||||
上述属性说明如下：
|
||||
|
|
@ -26,8 +34,76 @@ state-store.cluster=cluster1
|
|||
- `state-store.type`：状态存储的类型。目前仅支持Hazelcast。
|
||||
- `state-store.name`：用户定义的状态存储名称。
|
||||
- `state-store.cluster`：用户定义的状态存储集群名称。
|
||||
- `hazelcast.discovery.mode` : Hazelcast状态存储发现模式, 目前支持tcp-ip和multicast（默认）。
|
||||
- `hazelcast.discovery.port` : 用户定义hazelcast启动端口。
|
||||
|
||||
### 协调节点和工作节点属性
|
||||
#### 种子存储属性
|
||||
在协调节点和工作节点安装目录中创建文件`etc\seed-store.properties`。
|
||||
|
||||
种子存储用于存储协调节点的种子以相互发现。
|
||||
|
||||
```
|
||||
seed-store.type=filebased
|
||||
seed-store.filesystem.seed-dir=/tmp/openlookeng/ha
|
||||
seed-store.filesystem.profile=hdfs-config-default
|
||||
```
|
||||
|
||||
上述属性说明如下：
|
||||
|
||||
- `seed-store.type`: 种子存储的类型。目前仅支持Filebased。
|
||||
- `seed-store.filesystem.seed-dir`: 存储种子的目录。
|
||||
- `seed-store.filesystem.profile.`: `etc\filesystem`目录下的文件系统中配置文件的名称。
|
||||
|
||||
|
||||
#### 文件系统属性
|
||||
|
||||
在协调节点和工作节点安装目录中创建文件`etc\filesystem\hdfs-config-default.properties`。
|
||||
|
||||
文件系统必须是分布式文件系统，以便所有协调节点和工作节点都可以访问（例如HDFS）。
|
||||
```
|
||||
fs.client.type=hdfs
|
||||
hdfs.config.resources=/path/to/core-site.xml,/path/to/hdfs-site.xml
|
||||
hdfs.authentication.type=NONE
|
||||
fs.hdfs.impl.disable.cache=true
|
||||
```
|
||||
上述属性说明请参考[文件系统访问实用程序](../develop/filesystem.md)。
|
||||
|
||||
#### 协调节点和工作节点属性
|
||||
|
||||
在所有协调节点上的`etc/config.properties`中添加如下配置。
|
||||
|
||||
``` properties
|
||||
hetu.multiple-coordinator.enabled=true
|
||||
hetu.embedded-state-store.enabled=true
|
||||
hetu.seed-store.enabled=true
|
||||
```
|
||||
|
||||
在所有工作节点上的`etc/config.properties`中添加如下配置。
|
||||
|
||||
``` properties
|
||||
hetu.multiple-coordinator.enabled=true
|
||||
hetu.seed-store.enabled=true
|
||||
```
|
||||
|
||||
上述属性说明如下：
|
||||
|
||||
- `hetu.multiple-coordinator.enabled`：开启了多协调节点模式。
|
||||
- `hetu.embedded-state-store.enabled`：允许协调节点使用嵌入式状态存储。
|
||||
- `hetu.seed-store.enabled`: 允许协调节点和工作节点使用嵌入式状态存储。
|
||||
|
||||
### Multicast方法配置HA
|
||||
|
||||
#### 状态存储属性
|
||||
|
||||
状态存储属性文件`etc/state-store.properties`包含状态存储的配置。
|
||||
|
||||
``` properties
|
||||
state-store.type=hazelcast
|
||||
state-store.name=query
|
||||
state-store.cluster=cluster1
|
||||
```
|
||||
|
||||
#### 协调节点和工作节点属性
|
||||
|
||||
在所有协调节点上的`etc/config.properties`中添加如下配置。
|
||||
|
||||
|
|
@ -42,11 +118,6 @@ hetu.embedded-state-store.enabled=true
|
|||
hetu.multiple-coordinator.enabled=true
|
||||
```
|
||||
|
||||
上述属性说明如下：
|
||||
|
||||
- `hetu.multiple-coordinator.enabled`：开启了多协调节点模式。
|
||||
- `hetu.embedded-state-store.enabled`：允许此协调节点使用嵌入式状态存储。
|
||||
|
||||
## HA及反向代理
|
||||
|
||||
要尽可能的显示HA的优势，建议客户端(例如openLooKeng CLI, JDBC等)不直接连接特定的协调节点，而是通过反向代理连接多个协调节点，例如使用负载平衡器或者Kubernetes服务，这样即使某个协调节点无法正常工作，客户端也可以继续使用其它的协调节点。
|
||||
|
|
@ -57,7 +128,7 @@ hetu.multiple-coordinator.enabled=true
|
|||
|
||||
### 配置反向代理 (Nginx)
|
||||
|
||||
请在Nginx配置文件中包含以下配置（nginx.conf）
|
||||
请在Nginx配置文件中包含以下配置（nginx.conf）。
|
||||
|
||||
```
|
||||
http {
|
||||
|
|
|
|||
Loading…
Reference in New Issue