Update ORC Cache, Heuristic Index, Execution Plan Cache documentation
This commit is contained in:
Sundar Annamalai
parent
05406a0af6
commit
502204f885
|
|
@ -321,13 +321,6 @@ Exchanges transfer data between openLooKeng nodes for different stages of a quer
|
|||
>
|
||||
> Sets the network topology to use when scheduling splits. `legacy` will ignore the topology when scheduling splits. `flat` will try to schedule splits on the host where the data is located by reserving 50% of the work queue for local splits. It is recommended to use `flat` for clusters where distributed storage runs on the same nodes as openLooKeng workers.
|
||||
|
||||
### `node-scheduler.enable-split-cache-map`
|
||||
|
||||
> - **Type:** `boolean`
|
||||
> - **Default value:** `true`
|
||||
>
|
||||
> This property enables SplitCacheAwareNodeSelector. Split cache aware node selector keeps track of all splits and the nodes on which splits were previously scheduled. Split cache map is used in subsequent query execution to schedule the tasks on the same nodes. However, split cache map would be effective only after cache table query is executed.
|
||||
|
||||
## Optimizer Properties
|
||||
|
||||
### `optimizer.dictionary-aggregation`
|
||||
|
|
@ -437,46 +430,96 @@ The following properties allow tuning the [regexp](../functions/regexp.html).
|
|||
|
||||
## Heuristic Index Properties
|
||||
|
||||
Heuristic index is external index module that which can be applied to filter to out rows at the connector level. Bitmap, Bloom, MinMaxIndex are list of indexes provided by openLooKeng. As of now, heuristic index is used only in hive connector and specifically ORC format.
|
||||
|
||||
- `hetu.filter.enabled`
|
||||
|
||||
**Type:** `boolean` **Default value:** `false` This property enables heuristic index.
|
||||
|
||||
- `hetu.filter.cache.max-indices-number`
|
||||
|
||||
**Type:** `integer` **Default value:** `10,000,000` Caching the index files provides better performance, index files are read only and modified very rarely. Caching saves time spent on reading the files from index store. Cache in part of This property controls maximum number of index files that can be cached. When limit exceeded, existing entries will be removed from cache based on LRU and new entry will be added to cache.
|
||||
|
||||
- `hetu.filter.plugins`
|
||||
|
||||
**Type:** `string` This property is used to defined the location of the plugins required to support heuristic index. Property accepts multiple plugins separated by comma.
|
||||
|
||||
- `hetu.filter.indexstore.uri`
|
||||
|
||||
**Type:** `string` **Default value:** `/opt/hetu/indices/` Directory under which all index files are stored. Each index will be stored in its own subdirectory.
|
||||
|
||||
- `hetu.filter.indexstore.type`
|
||||
|
||||
**Type** `string` **Allowed values:** `hdfs, local` **Default value:** `local` This property defines the persistence store for the index files. Additional store specific properties must be defined.
|
||||
|
||||
Heuristic index is external index module that which can be used to filter to out rows at the connector level. Bitmap, Bloom, MinMaxIndex are list of indexes provided by openLooKeng. As of now, bitmap index supports supports hive connector for tables with ORC storage format.
|
||||
|
||||
|
||||
```
|
||||
hetu.filter.hdfs.**
|
||||
```
|
||||
|
||||
### `hetu.filter.enabled`
|
||||
|
||||
|
||||
> Properties for HDFS indexstore. Few examples are
|
||||
> - **Type:** `boolean`
|
||||
> - **Default value:** `false`
|
||||
>
|
||||
> hetu.filter.indexstore.hdfs.config.resource hetu.filter.indexstore.hdfs.authentication.type hetu.filter.indexstore.hdfs.krb5.keytab.path hetu.filter.indexstore.hdfs.krb5.conf.path hetu.filter.indexstore.hdfs.krb5.principal
|
||||
|
||||
> This property enables heuristic index.
|
||||
|
||||
|
||||
```
|
||||
hetu.filter.local.**
|
||||
```
|
||||
|
||||
### `hetu.filter.cache.max-indices-number`
|
||||
|
||||
> - **Type:** `integer`
|
||||
> - **Default value:** `10,000,000`
|
||||
>
|
||||
> Caching the index files provides better performance, index files are read only and modified very rarely. Caching saves time spent on reading the files from index store. Cache in part of This property controls maximum number of index files that can be cached. When limit exceeded, existing entries will be removed from cache based on LRU and new entry will be added to cache.
|
||||
|
||||
### `hetu.filter.plugins`
|
||||
|
||||
> - **Type:** `string`
|
||||
>
|
||||
> This property is used to defined the location of the plugins required to support heuristic index. Property accepts multiple plugins separated by comma.
|
||||
|
||||
### `hetu.filter.indexstore.uri`
|
||||
|
||||
> - **Type:** `string`
|
||||
> - **Default value:** `/opt/hetu/indices/`
|
||||
>
|
||||
> Directory under which all index files are stored. Each index will be stored in its own subdirectory.
|
||||
|
||||
### `hetu.filter.indexstore.type`
|
||||
|
||||
> - **Type** `string`
|
||||
> - **Allowed values:** `hdfs, local`
|
||||
> - **Default value:** `local`
|
||||
>
|
||||
> This property defines the persistence store for the index files. Additional properties must be provider for HDFS index store.
|
||||
>
|
||||
#### Properties for HDFS indexstore
|
||||
|
||||
| Property Name | Mandatory | Description |
|
||||
| ---------------------------------------------------------- | -------------------------------- | ----------------------------------------------------------------- |
|
||||
| `hetu.filter.indexstore.hdfs.config.resources` | YES | Path to hdfs resource files (e.g. core-site.xml, hdfs-site.xml) |
|
||||
| `hetu.filter.indexstore.hdfs.authentication.type` | YES | hdfs authentication Accepted values: `KERBEROS`, `NONE` |
|
||||
| `hetu.filter.indexstore.hdfs.krb5.conf.path` | YES if auth type set to KERBEROS | Path to the krb5 config file |
|
||||
| `hetu.filter.indexstore.hdfs.krb5.keytab.path` | YES if auth type set to KERBEROS | Path to the kerberos keytab file |
|
||||
| `hetu.filter.indexstore.hdfs.krb5.principal` | YES if auth type set to KERBEROS | Principal of kerberos authentication |
|
||||
|
||||
##Execution Plan Cache Properties
|
||||
|
||||
> Properties for Local indexstore.
|
||||
Execution plan cache feature allows the coordinator to reuse execution plans between identical queries, instead
|
||||
of constructing another execution plan, thus reducing the amount of query pre-processing required.
|
||||
|
||||
### `hetu.executionplan.cache.enabled`
|
||||
>
|
||||
> - **Type:** `boolean`
|
||||
> - **Default value:** `false`
|
||||
>
|
||||
> Enable or disable execution plan cache. Disabled by default.
|
||||
|
||||
### `hetu.executionplan.cache.limit`
|
||||
>
|
||||
> - **Type:** `integer`
|
||||
> - **Default value:** `1000`
|
||||
>
|
||||
> Maximum number of execution plans to keep in the cache
|
||||
|
||||
### `hetu.executionplan.cache.timeout`
|
||||
>
|
||||
> - **Type:** `integer`
|
||||
> - **Default value:** `60000 ms`
|
||||
>
|
||||
> Time in milliseconds to expire cached execution plans after the last access
|
||||
|
||||
## SplitCacheMap Properties
|
||||
|
||||
SplitCacheMap must be enabled to support caching row data. When enabled, the coordinator stores table, partition and split scheduling metadata that
|
||||
helps with cache affinity scheduling.
|
||||
|
||||
### `hetu.split-cache-map.enabled`
|
||||
|
||||
> - **Type:** `boolean`
|
||||
> - **Default value:** `false`
|
||||
>
|
||||
> This property enables split caching functionality.
|
||||
> If state store is enabled, the split cache map configuration is automatically replicated in state store as well.
|
||||
> In case of HA setup with multiple coordinators, the state store is used to share split cache map between the coordinators.
|
||||
|
||||
### `hetu.split-cache-map.state-update-interval`
|
||||
|
||||
> - **Type:** `integer`
|
||||
> - **Default value:** `2 seconds`
|
||||
>
|
||||
> This property controls how frequently the split cache map is updated in state store. It is primarily applicable for HA deployment.
|
||||
|
|
|
|||
|
|
@ -269,15 +269,15 @@ The Hive connector can access data stored in GCS, using the `gs://` URI prefix.
|
|||
| `hive.gcs.json-key-file-path` | JSON key file used to authenticate with Google Cloud Storage. |
|
||||
| `hive.gcs.use-access-token` | Use client-provided OAuth token to access Google Cloud Storage. This is mutually exclusive with a global JSON key file. |
|
||||
|
||||
ORC Cache Configuration
|
||||
-----------------------
|
||||
|
||||
Hive connector caches the ORC file data to provide better performance. The data is cached in worker local memory. Support for caching partitioned table is only available. `cache sql` can be used to customize the table and partition that should be cached by the Connector.
|
||||
ORC Cache Configuration
|
||||
-----------------------
|
||||
|
||||
Hive connector caches the ORC file data to provide better performance and reduce query latency. Workers cache the data on their local memory.
|
||||
When enabled, Workers cache all ORC files tail, stripe-footer, row-index, bloom-filter information. However, the workers cache row data of only specific ORC
|
||||
files that are matching the predicates provided via `cache table` sql statement.
|
||||
|
||||
### ORC Cache Properties
|
||||
|
||||
|
||||
|
||||
| Property Name | Description | Default |
|
||||
| :----------------------------------------- | :--------------------------------------------------- | :-------- |
|
||||
| `hive.orc.file-tail.cache.enabled` | Enable ORC file tail cache | `false` |
|
||||
|
|
@ -296,6 +296,9 @@ Hive connector caches the ORC file data to provide better performance. The data
|
|||
| `hive.orc.row-data.block.cache.ttl` | TTL for ORC row group cache | `30 mins` |
|
||||
| `hive.orc.row-data.block.cache.max.weight` | Maximum weight of ORC row group cache | `500 MB` |
|
||||
|
||||
TTL is time taken since cache entry was last accessed by read or write. Timed expiration is performed with periodic maintenance during writes
|
||||
and occasionally during reads, as discussed below.
|
||||
|
||||
Table Statistics
|
||||
----------------
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,51 @@
|
|||
ORC Cache
|
||||
=========
|
||||
ORC Cache feature improves the query performance by caching frequently accessed data. ORC Cache reduces time spent on TableScan operation because
|
||||
the network IO This in turn reduces the query latency.
|
||||
|
||||
This feature is most beneficial for caching raw data from tables that are most frequently accessed and not co-located with
|
||||
the openLooKeng deployment. If enabled, workers automatically cache file tail, stripe footer, row index, bloom index
|
||||
of all ORC files because they are small. However, row group data tends to be huge and caching row group data for all
|
||||
files in not practically feasible because of the limitation with cache size.
|
||||
|
||||
Check this [link](https://orc.apache.org/specification/ORCv1/) to know about ORC specification.
|
||||
|
||||
``CACHE TABLE`` SQL command can be used to configure the table and partition for which row data should be cached by the Worker.
|
||||
|
||||
The following sections briefly explains how the entire row data cache implementation works.
|
||||
|
||||
SplitCacheMap
|
||||
-------------
|
||||
Users can use `CACHE TABLE` sql statement to configure which table and data must be cached by Hive connector. The partitions to cache are defined
|
||||
as predicates and are stored in `SplitCacheMap`. SplitCacheMap is stored in local memory of the coordinator.
|
||||
|
||||
Sample query to cache sales table data for days between 2020-01-04 and 2020-01-11.
|
||||
|
||||
`cache table hive.default.sales where sales_date BETWEEN date '2020-01-04' AND date'2020-01-11'`
|
||||
|
||||
Check `CACHE TABLE`, `SHOW CACHE`, and `DROP CACHE` commands for more information.
|
||||
|
||||
SplitCacheMap stores two kinds of information
|
||||
1. Table name along with predicates provided via `CACHE TABLE` command.
|
||||
2. Split to Worker mapping
|
||||
|
||||
Connector
|
||||
---------
|
||||
When caching is enabled and a predicate is provided through ``CACHE TABLE`` SQL command, HiveSplits will be flagged by the connector
|
||||
as cacheable if the corresponding partitioned ORC file matches the predicate.
|
||||
|
||||
SplitCacheAwareNodeSelector
|
||||
---------------------------
|
||||
SplitCacheAwareNodeSelector is implemented to support cache affinity scheduling. SplitCacheAwareNodeSelector is like any other node selector
|
||||
responsible for assigning splits to workers. When a split is scheduled for first time, the node selector stores the split and worker on which
|
||||
the split was scheduled. For subsequent scheduling, this information is used to determine whether split has already been processed by a worker.
|
||||
If so, the node selector schedules the split on the worker that previously processed it. If not, SplitCacheAwareNodeSelector falls back to default
|
||||
node selector to schedule the split. Workers which process the splits will cache the data mapped by the split in local memory.
|
||||
|
||||
Workers
|
||||
-------
|
||||
Workers rely on `ConnectorSplit.isCacheable` method to determine whether split data must be cached. If property is set
|
||||
to true, the HiveConnector tries to retrieve the data from Cache. In case of cache miss, the data is read from HDFS and stored in Cache for future
|
||||
use. Workers will purge their caches by expiry time or by reaching size limit, independently of the coordinator.
|
||||
|
||||
Check `ORC Cache Configuration` under Hive connector to know more about cache config.
|
||||
|
|
@ -18,19 +18,26 @@ Description
|
|||
|
||||
`CACHE TABLE` updates coordinator metadata with table name and partition that should be cached by the connector.
|
||||
|
||||
`condition` must be provided and is defined on only partition column(s).
|
||||
`condition` must be provided and is defined on only partition column(s) at this time.
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
Cache table :
|
||||
Cache all sales data for stores located in 'CA':
|
||||
|
||||
``` sql
|
||||
CACHE TABLE store_sales where location = 'CA';
|
||||
|
||||
```
|
||||
Cache all sales data from stores sold after 20 Feb 2020.
|
||||
|
||||
``` sql
|
||||
CACHE TABLE store_sales where ss_sold_date_sk > 20200220;
|
||||
```
|
||||
|
||||
Limitations
|
||||
-----------
|
||||
|
||||
Only Hive connector support this functionality. See connector documentation for more details.
|
||||
Only Hive connector support this functionality at this time. See connector documentation for more details.
|
||||
|
||||
See Also
|
||||
--------
|
||||
|
|
|
|||
|
|
@ -16,19 +16,22 @@ DROP CACHE table
|
|||
Description
|
||||
-----------
|
||||
|
||||
`DROP CACHE` deletes cache metadata of the `table` from coordinator.
|
||||
`DROP CACHE` deletes cache metadata of the `table` from coordinator only. Workers' caches purged automatically by expiry time
|
||||
or by reaching size limit but recurring splits will not reuse any cached node assignments.
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
Drop cache :
|
||||
|
||||
DROP CACHE table
|
||||
Drop cache metadata for `sales' table
|
||||
|
||||
```sql
|
||||
DROP CACHE sales
|
||||
```
|
||||
|
||||
Limitations
|
||||
-----------
|
||||
|
||||
Only Hive connector support this functionality. See connector documentation for more details.
|
||||
Only Hive connector support this functionality at this time. See connector documentation for more details.
|
||||
|
||||
See Also
|
||||
--------
|
||||
|
|
|
|||
|
|
@ -21,16 +21,24 @@ Description
|
|||
Examples
|
||||
--------
|
||||
|
||||
Show cache :
|
||||
Show all cache metadata
|
||||
|
||||
```sql
|
||||
SHOW CACHE;
|
||||
```
|
||||
|
||||
SHOW CACHE
|
||||
Show cache metadata for sales table
|
||||
|
||||
```sql
|
||||
SHOW CACHE sales;
|
||||
```
|
||||
|
||||
Limitations
|
||||
-----------
|
||||
|
||||
Only Hive connector support this functionality. See connector documentation for more details.
|
||||
Only Hive connector support this functionality at this time. See connector documentation for more details.
|
||||
|
||||
See Also
|
||||
--------
|
||||
|
||||
[cache-table](./cache-table.html), [drop-cache](./drop-cache.html)
|
||||
[cache-table](./cache-table.html), [drop-cache](./drop-cache.html)
|
||||
|
|
|
|||
|
|
@ -300,13 +300,6 @@ title = "属性参考"
|
|||
>
|
||||
> 设置调度拆分时使用的网络拓扑。`legacy`调度拆分时忽略拓扑。`flat`会尝试在数据所在的主机上调度拆分，为本地拆分预留50%的工作队列。对于分布式存储与openLooKeng worker运行在相同节点上的集群，推荐使用`flat`。
|
||||
|
||||
### `node-scheduler.enable-split-cache-map`
|
||||
|
||||
> - 类型：`boolean`
|
||||
> - 默认值：`true`
|
||||
>
|
||||
> 此属性启**用SplitCacheAwareNodeSelector**。拆分缓存感知节点选择器跟踪所有拆分以及以前调度过拆分的节点。在后续的查询执行中，使用拆分缓存映射来调度同一节点上的任务。但是，拆分缓存映射只有在执行缓存表查询之后才有效。
|
||||
|
||||
## 优化器属性
|
||||
|
||||
### `optimizer.dictionary-aggregation`
|
||||
|
|
@ -406,38 +399,96 @@ title = "属性参考"
|
|||
|
||||
## 启发式索引属性
|
||||
|
||||
启发式索引是外部索引模块，可用于筛选连接器级别的行。Bitmap、Bloom、MinMaxIndex是openLooKeng提供的索引列表。到目前为止，启发式索引仅用于Hive连接器，特别是ORC格式。
|
||||
启发式索引是外部索引模块，可用于过滤连接器级别的行。 位图，Bloom和MinMaxIndex是openLooKeng提供的索引列表。 到目前为止，位图索引支持使用ORC存储格式的表支持蜂巢连接器。
|
||||
|
||||
- `hetu.filter.enabled`
|
||||
|
||||
类型：`boolean` 默认值：`false` 此属性启用启发式索引。
|
||||
|
||||
- `hetu.filter.cache.max-indices-number`
|
||||
|
||||
类型：`integer` 默认值：`10,000,000` 缓存索引文件可以提供更好的性能，索引文件是只读的，很少被修改。缓存节省了从索引存储读取文件的时间。部分缓存此属性控制可以缓存的索引文件的最大数量。当超过限制时，基于LRU的现有条目将从缓存中移除，新条目将添加到缓存中。
|
||||
|
||||
- `hetu.filter.plugins`
|
||||
|
||||
类型：`string` 此属性用于定义支持启发式索引所需的插件的位置。属性接受多个插件，由逗号分隔。
|
||||
|
||||
- `hetu.filter.indexstore.uri`
|
||||
|
||||
类型：`string` 默认值：`/opt/hetu/indices/` 存放所有索引文件的目录。每个索引将存储在自己的子目录中。
|
||||
|
||||
- `hetu.filter.indexstore.type`
|
||||
|
||||
类型`string` 允许值：`hdfs, local` 默认值：`local` 此属性定义索引文件的持久性存储区。必须定义其他特定于存储的属性。
|
||||
|
||||
```
|
||||
hetu.filter.hdfs.**
|
||||
```
|
||||
|
||||
> HDFS索引存储的属性。一些例子如下：
|
||||
### `hetu.filter.enabled`
|
||||
|
||||
> - **类型:** `boolean`
|
||||
> - **默认值:** `false`
|
||||
>
|
||||
> 此属性启用启发式索引.
|
||||
|
||||
### `hetu.filter.cache.max-indices-number`
|
||||
|
||||
> - **类型:** `integer`
|
||||
> - **默认值：** `10,000,000`
|
||||
>
|
||||
> 缓存索引文件可提供更好的性能，索引文件是只读的，很少修改。 缓存节省了从索引存储读取文件所花费的时间。 部分缓存该属性控制可以缓存的索引文件的最大数量。 当超过限制时，将基于LRU从缓存中删除现有条目，并将新条目添加到缓存中。
|
||||
|
||||
### `hetu.filter.plugins`
|
||||
|
||||
> - **类型:** `string`
|
||||
>
|
||||
> 此属性用于定义支持启发式索引所需的插件的位置。 属性接受多个用逗号分隔的插件。
|
||||
|
||||
### `hetu.filter.indexstore.uri`
|
||||
|
||||
> - **类型:** `string`
|
||||
> - **默认值：** `/opt/hetu/indices/`
|
||||
>
|
||||
> hetu.filter.indexstore.hdfs.config.resource hetu.filter.indexstore.hdfs.authentication.type hetu.filter.indexstore.hdfs.krb5.keytab.path hetu.filter.indexstore.hdfs.krb5.conf.path hetu.filter.indexstore.hdfs.krb5.principal
|
||||
> 所有索引文件存储在的目录。 每个索引将存储在其自己的子目录中。
|
||||
|
||||
### `hetu.filter.indexstore.type`
|
||||
|
||||
> - **类型** `string`
|
||||
> - **允许的值：** `hdfs, local`
|
||||
> - **默认值：** `local`
|
||||
>
|
||||
> 此属性定义索引文件的持久性存储。 其他属性必须是HDFS索引存储的提供者。
|
||||
>
|
||||
#### HDFS索引存储的属性
|
||||
|
||||
```
|
||||
hetu.filter.local.**
|
||||
```
|
||||
| 物业名称 | 强制性的 | 描述 |
|
||||
| ---------------------------------------------------------- | -------------------------------- | ----------------------------------------------------------------- |
|
||||
| `hetu.filter.indexstore.hdfs.config.resources` | 是 | hdfs资源文件的路径 (e.g. core-site.xml, hdfs-site.xml) |
|
||||
| `hetu.filter.indexstore.hdfs.authentication.type` | 是 | hdfs身份验证接受的值：`KERBEROS`, `NONE` |
|
||||
| `hetu.filter.indexstore.hdfs.krb5.conf.path` | 如果身份验证类型设置为YES KERBEROS | krb5配置文件的路径 |
|
||||
| `hetu.filter.indexstore.hdfs.krb5.keytab.path` | 如果身份验证类型设置为YES KERBEROS | kerberos keytab文件的路径 |
|
||||
| `hetu.filter.indexstore.hdfs.krb5.principal` | 如果身份验证类型设置为YES KERBEROS | Kerberos认证主体 |
|
||||
|
||||
##执行计划缓存属性
|
||||
|
||||
> 本地索引存储的属性。
|
||||
执行计划缓存功能允许协调器在相同的查询之间重用执行计划，
|
||||
构建另一个执行计划的过程，从而减少了所需的查询预处理量。
|
||||
|
||||
### `hetu.executionplan.cache.enabled`
|
||||
>
|
||||
> - **类型:** `boolean`
|
||||
> - **默认值：** `false`
|
||||
>
|
||||
> 启用或禁用执行计划缓存。 默认禁用。
|
||||
|
||||
### `hetu.executionplan.cache.limit`
|
||||
>
|
||||
> - **类型:** `integer`
|
||||
> - **默认值：** `1000`
|
||||
>
|
||||
> 保留在缓存中的最大执行计划数
|
||||
|
||||
### `hetu.executionplan.cache.timeout`
|
||||
>
|
||||
> - **类型:** `integer`
|
||||
> - **默认值：** `60000 ms`
|
||||
>
|
||||
> 上次访问后使缓存的执行计划失效的时间（以毫秒为单位）
|
||||
|
||||
## SplitCacheMap属性
|
||||
|
||||
必须启用SplitCacheMap以支持缓存行数据。 启用后，协调器将存储表，分区和拆分调度元数据
|
||||
帮助进行缓存亲和力调度。
|
||||
|
||||
### `hetu.split-cache-map.enabled`
|
||||
|
||||
> - **类型:** `boolean`
|
||||
> - **默认值：** `false`
|
||||
>
|
||||
> 此属性启用拆分缓存功能。
|
||||
> 如果启用了状态存储，则拆分缓存映射配置也会自动复制到状态存储中。
|
||||
> 在具有多个协调器的HA设置的情况下，状态存储用于在协调器之间共享拆分的缓存映射。
|
||||
|
||||
### `hetu.split-cache-map.state-update-interval`
|
||||
|
||||
> - **类型:** `integer`
|
||||
> - **默认值：** `2 seconds`
|
||||
>
|
||||
> 此属性控制在状态存储中更新分割缓存映射的频率。 它主要适用于HA部署。
|
||||
|
|
|
|||
|
|
@ -272,12 +272,12 @@ The Hive connector can access data stored in GCS, using the `gs://` URI prefix.
|
|||
ORC Cache Configuration
|
||||
-----------------------
|
||||
|
||||
Hive connector caches the ORC file data to provide better performance. The data is cached in worker local memory. Support for caching partitioned table is only available. `cache sql` can be used to customize the table and partition that should be cached by the Connector.
|
||||
Hive connector caches the ORC file data to provide better performance and reduce query latency. Workers cache the data on their local memory.
|
||||
When enabled, Workers cache all ORC files tail, stripe-footer, row-index, bloom-filter information. However, the workers cache row data of only specific ORC
|
||||
files that are matching the predicates provided via `cache table` sql statement.
|
||||
|
||||
### ORC Cache Properties
|
||||
|
||||
|
||||
|
||||
| Property Name | Description | Default |
|
||||
| :----------------------------------------- | :--------------------------------------------------- | :-------- |
|
||||
| `hive.orc.file-tail.cache.enabled` | Enable ORC file tail cache | `false` |
|
||||
|
|
@ -296,6 +296,9 @@ Hive connector caches the ORC file data to provide better performance. The data
|
|||
| `hive.orc.row-data.block.cache.ttl` | TTL for ORC row group cache | `30 mins` |
|
||||
| `hive.orc.row-data.block.cache.max.weight` | Maximum weight of ORC row group cache | `500 MB` |
|
||||
|
||||
TTL is time taken since cache entry was last accessed by read or write. Timed expiration is performed with periodic maintenance during writes
|
||||
and occasionally during reads, as discussed below.
|
||||
|
||||
Table Statistics
|
||||
----------------
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,51 @@
|
|||
ORC Cache
|
||||
=========
|
||||
ORC Cache feature improves the query performance by caching frequently accessed data. ORC Cache reduces time spent on TableScan operation because
|
||||
the network IO This in turn reduces the query latency.
|
||||
|
||||
This feature is most beneficial for caching raw data from tables that are most frequently accessed and not co-located with
|
||||
the openLooKeng deployment. If enabled, workers automatically cache file tail, stripe footer, row index, bloom index
|
||||
of all ORC files because they are small. However, row group data tends to be huge and caching row group data for all
|
||||
files in not practically feasible because of the limitation with cache size.
|
||||
|
||||
Check this [link](https://orc.apache.org/specification/ORCv1/) to know about ORC specification.
|
||||
|
||||
``CACHE TABLE`` SQL command can be used to configure the table and partition for which row data should be cached by the Worker.
|
||||
|
||||
The following sections briefly explains how the entire row data cache implementation works.
|
||||
|
||||
SplitCacheMap
|
||||
-------------
|
||||
Users can use `CACHE TABLE` sql statement to configure which table and data must be cached by Hive connector. The partitions to cache are defined
|
||||
as predicates and are stored in `SplitCacheMap`. SplitCacheMap is stored in local memory of the coordinator.
|
||||
|
||||
Sample query to cache sales table data for days between 2020-01-04 and 2020-01-11.
|
||||
|
||||
`cache table hive.default.sales where sales_date BETWEEN date '2020-01-04' AND date'2020-01-11'`
|
||||
|
||||
Check `CACHE TABLE`, `SHOW CACHE`, and `DROP CACHE` commands for more information.
|
||||
|
||||
SplitCacheMap stores two kinds of information
|
||||
1. Table name along with predicates provided via `CACHE TABLE` command.
|
||||
2. Split to Worker mapping
|
||||
|
||||
Connector
|
||||
---------
|
||||
When caching is enabled and a predicate is provided through ``CACHE TABLE`` SQL command, HiveSplits will be flagged by the connector
|
||||
as cacheable if the corresponding partitioned ORC file matches the predicate.
|
||||
|
||||
SplitCacheAwareNodeSelector
|
||||
---------------------------
|
||||
SplitCacheAwareNodeSelector is implemented to support cache affinity scheduling. SplitCacheAwareNodeSelector is like any other node selector
|
||||
responsible for assigning splits to workers. When a split is scheduled for first time, the node selector stores the split and worker on which
|
||||
the split was scheduled. For subsequent scheduling, this information is used to determine whether split has already been processed by a worker.
|
||||
If so, the node selector schedules the split on the worker that previously processed it. If not, SplitCacheAwareNodeSelector falls back to default
|
||||
node selector to schedule the split. Workers which process the splits will cache the data mapped by the split in local memory.
|
||||
|
||||
Workers
|
||||
-------
|
||||
Workers rely on `ConnectorSplit.isCacheable` method to determine whether split data must be cached. If property is set
|
||||
to true, the HiveConnector tries to retrieve the data from Cache. In case of cache miss, the data is read from HDFS and stored in Cache for future
|
||||
use. Workers will purge their caches by expiry time or by reaching size limit, independently of the coordinator.
|
||||
|
||||
Check `ORC Cache Configuration` under Hive connector to know more about cache config.
|
||||
|
|
@ -15,17 +15,25 @@ CACHE TABLE table_name WHERE condition
|
|||
|
||||
`CACHE TABLE` 使用连接器应缓存的表名和分区更新协调器元数据。
|
||||
|
||||
必须提供 `condition`，并且仅在分区列上对其进行定义。
|
||||
目前必须提供“ condition”，并且仅在分区列上定义。
|
||||
|
||||
## 示例
|
||||
|
||||
缓存表：
|
||||
缓存“ CA”中商店的所有销售数据:
|
||||
|
||||
``` sql
|
||||
CACHE TABLE store_sales where location = 'CA';
|
||||
```
|
||||
缓存2020年2月20日之后出售的商店中的所有销售数据。
|
||||
|
||||
``` sql
|
||||
CACHE TABLE store_sales where ss_sold_date_sk > 20200220;
|
||||
```
|
||||
|
||||
|
||||
## 限制
|
||||
|
||||
仅 Hive 连接器支持该功能。有关更多详细信息，请参见连接器文档。
|
||||
目前只有Hive连接器支持此功能。 有关更多详细信息，请参见连接器文档。
|
||||
|
||||
## 另请参见
|
||||
|
||||
|
|
|
|||
|
|
@ -13,17 +13,19 @@ DROP CACHE table
|
|||
|
||||
## 说明
|
||||
|
||||
`DROP CACHE` 从协调器中删除 `table` 的缓存元数据。
|
||||
`DROP CACHE`仅从协调器删除`table`的缓存元数据。 到期时间自动清除工作人员的缓存 或达到大小限制，但重复进行的拆分将不会重用任何缓存的节点分配。
|
||||
|
||||
## 示例
|
||||
|
||||
删除缓存：
|
||||
删除“ sales”表的缓存元数据
|
||||
|
||||
DROP CACHE table
|
||||
```sql
|
||||
DROP CACHE sales
|
||||
```
|
||||
|
||||
## 限制
|
||||
|
||||
仅 Hive 连接器支持该功能。有关更多详细信息，请参见连接器文档。
|
||||
目前只有Hive连接器支持此功能。 有关更多详细信息，请参见连接器文档。
|
||||
|
||||
## 另请参见
|
||||
|
||||
|
|
|
|||
|
|
@ -17,13 +17,21 @@ SHOW CACHE
|
|||
|
||||
## 示例
|
||||
|
||||
显示缓存：
|
||||
显示所有缓存元数据
|
||||
|
||||
SHOW CACHE
|
||||
```sql
|
||||
SHOW CACHE;
|
||||
```
|
||||
|
||||
显示销售表的缓存元数据
|
||||
|
||||
```sql
|
||||
SHOW CACHE sales;
|
||||
```
|
||||
|
||||
## 限制
|
||||
|
||||
仅 Hive 连接器支持该功能。有关更多详细信息，请参见连接器文档。
|
||||
目前只有Hive连接器支持此功能。 有关更多详细信息，请参见连接器文档。
|
||||
|
||||
## 另请参见
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue