update hindex docs
This commit is contained in:
farhan3
parent
1dea9c09c3
commit
f853ec6ab7
|
|
@ -536,7 +536,11 @@ Heuristic index is external index module that which can be used to filter to out
|
|||
|
||||
> - **Type** `string`
|
||||
>
|
||||
> This property defines the filesystem profile used to read and write index. The corresponding profile must exist in `etc/filesystem`. For example, if this property is set as `hetu.heuristicindex.filter.indexstore.filesystem.profile=index-hdfs1`, a profile describing this filesystem access `index-hdfs1.properties` must be created in `etc/filesystem` with necessary information including authentication type, config, and keytabs (if applicable).
|
||||
> This property defines the filesystem profile used to read and write index. The corresponding profile must exist in `etc/filesystem`. For example, if this property is set as `hetu.heuristicindex.filter.indexstore.filesystem.profile=index-hdfs1`, a profile describing this filesystem access `index-hdfs1.properties` must be created in `etc/filesystem` with necessary information including authentication type, config, and keytabs (if applicable).
|
||||
>
|
||||
> `LOCAL` filesystem type should only be used during testing or in single node clusters.
|
||||
>
|
||||
> `HDFS` filesystem type should be used in production in order for the index to be accessible by all nodes in the cluster. All nodes should be configured to use the same filesystem profile.
|
||||
|
||||
## Execution Plan Cache Properties
|
||||
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
|
|
@ -1,19 +1,42 @@
|
|||
|
||||
# Bitmap Index
|
||||
|
||||
## Use cases
|
||||
Bitmap Index utilizes Bitmaps. The size of the index increases as the number
|
||||
of unique values in the column increases. For example, a column like gender
|
||||
will have a small size. Whereas a column like ID will have an extremely
|
||||
large size (not recommended).
|
||||
|
||||
Bitmap Index is used for filtering data read from ORC files and is used only by the **worker** nodes
|
||||
Note: Bitmap Index can additionally benefit when ORC predicate pushdown is enabled.
|
||||
This can be enabled by setting `hive.orc-predicate-pushdown-enabled=true`
|
||||
in `hive.properties` or setting the session using `set session hive.orc_predicate_pushdown_enabled=true;`.
|
||||
Setting this to true will improve improve the performance of queries that utilize Bitmap Index.
|
||||
See [Properties](../admin/properties.md) for details.
|
||||
|
||||
- If this index exists on a column which is part of a predicate in the query, the performance may be improved while reading the ORC files.
|
||||
## Filtering
|
||||
|
||||
For example, if an index exists on column `country` and the query is
|
||||
1. Bitmap Index is used on workers for filtering rows when reading ORC files.
|
||||
|
||||
``` sql
|
||||
select * from table where country="China"
|
||||
## Selecting column for Bitmap Index
|
||||
|
||||
Bitmap Index works on columns that have a low cardinality (i.e. few unique values),
|
||||
such as a Gender column.
|
||||
|
||||
## Supported operators
|
||||
|
||||
= Equality
|
||||
|
||||
## Examples
|
||||
|
||||
Creating index:
|
||||
```sql
|
||||
create index idx using bitmap on hive.hindex.users (gender);
|
||||
create index idx using bitmap on hive.hindex.users (gender) where regionkey=1;
|
||||
create index idx using bitmap on hive.hindex.users (gender) where regionkey in (3, 1);
|
||||
```
|
||||
|
||||
- This index works best if the column's values are not too distinct (e.g. country) and are distributed.
|
||||
|
||||
For example, assume that the table stores information about where users are from and the table data is in 10 files. There maybe be several users from a particular country, so each file will have some users from the country. If we create a bitmap index on the country column, we can perform filtering early on while reading the data files. i.e. the predicate is pushed down to the reading of the file. Without this index, all the data files will need to be read into memory as Pages and then the filtering would happen. With the index, we can ensure that the Pages already only contain the rows matching the predicate. This can help reduce the memory and CPU usage and can result in improved performance when many concurrent queries are running.
|
||||
* assuming users table is partitioned on `regionkey`
|
||||
|
||||
Using index:
|
||||
```sql
|
||||
select name from hive.hindex.users where gender="female"
|
||||
```
|
||||
|
|
@ -1,22 +1,48 @@
|
|||
|
||||
# Bloom Index
|
||||
|
||||
## Use cases
|
||||
Bloom Index utilizes Bloom Filters and index size will be fairly small.
|
||||
|
||||
Bloom Index is used for split filtering, and is used only by the **coordinator** nodes.
|
||||
## Filtering
|
||||
|
||||
- If this index exists on a column which is part of a predicate in the query, openLooKeng may be able to improve performance by filtering scheduled splits.
|
||||
1. Bloom Index is used on coordinator for filtering splits during scheduling
|
||||
2. Bloom Index is used on workers for filtering Stripes when reading ORC files
|
||||
|
||||
For example, if an index exists on column `id` and the query is:
|
||||
## Selecting column for Bloom Index
|
||||
|
||||
Bloom Index works on columns that have high cardinality (i.e. unique values),
|
||||
such as an ID column.
|
||||
|
||||
## Supported operators
|
||||
|
||||
= Equality
|
||||
|
||||
## Configurations
|
||||
|
||||
### `bloom.fpp`
|
||||
|
||||
> - **Type:** `Double`
|
||||
> - **Default value:** `0.001`
|
||||
>
|
||||
> Changes the FPP (false positive probability) value of the Bloom filter.
|
||||
> Making this value smaller will increase the effectiveness of the index but
|
||||
> will also increase the index size. The default value should be sufficient
|
||||
> in most usecases. If the index is too large, this value can be increased
|
||||
> e.g. 0.05.
|
||||
|
||||
## Examples
|
||||
|
||||
Creating index:
|
||||
```sql
|
||||
select * from table where id=12345
|
||||
create index idx using bloom on hive.hindex.users (id);
|
||||
create index idx using bloom on hive.hindex.users (id) where regionkey=1;
|
||||
create index idx using bloom on hive.hindex.users (id) where regionkey in (3, 1);
|
||||
create index idx using bloom on hive.hindex.users (id) WITH ("bloom.fpp" = '0.001');
|
||||
```
|
||||
|
||||
* assuming users table is partitioned on `regionkey`
|
||||
|
||||
|
||||
- Bloom index works best if the column's values are unique (e.g. userid) and are not too distributed.
|
||||
|
||||
For example, assume the tables stores information about users and the table data is in 10 files. For a given userid, only one file will contain the data. Therefore creating an index on userid will help us to filter out 9 out of the 10 files at scheduling time and will save significant IO time that would've therwise been used to read each of the files.
|
||||
|
||||
*Tip: if possible, it is recommended to sort the data on the column being indexed.*
|
||||
Using index:
|
||||
```sql
|
||||
select name from hive.hindex.users where id=123
|
||||
```
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
# BTree Index
|
||||
|
||||
BTree Index utilizes the B-Tree data structure.
|
||||
The size of the index increases as the number
|
||||
of unique values in the column increases.
|
||||
|
||||
## Filtering
|
||||
|
||||
1. BTree Index is used on coordinator for filtering splits during scheduling
|
||||
|
||||
## Selecting column for BTree Index
|
||||
|
||||
BTree Index works on columns that have high cardinality (i.e. unique values),
|
||||
such as an ID column, additionally it requires that the table be partitioned,
|
||||
e.g. by date.
|
||||
|
||||
When selecting between BTree Index, the following should be considered:
|
||||
- Bloom index only supports `=`
|
||||
- Btree index requires the table to be partitioned
|
||||
- Bloom index is probabilistic, whereas Btree index is deterministic. This means Btree will perform better filtering.
|
||||
- Btree index size will be larger than Bloom index
|
||||
|
||||
## Supported operators
|
||||
|
||||
= Equality
|
||||
> Greater than
|
||||
>= Greater than or equal
|
||||
< Less than
|
||||
<= Less than or equal
|
||||
BETWEEN Between range
|
||||
IN IN set
|
||||
|
||||
## Examples
|
||||
|
||||
Creating index:
|
||||
```sql
|
||||
create index idx using btree on hive.hindex.orders (orderid) with (level=partition) where orderDate='01-10-2020' ;
|
||||
create index idx using btree on hive.hindex.orders (orderid) with (level=partition) where orderDate in ('01-10-2020', '01-10-2020');
|
||||
```
|
||||
|
||||
* assuming orders table is partitioned on `orderDate`; table must be partitioned
|
||||
|
||||
Using index:
|
||||
```sql
|
||||
select * from hive.hindex.orders where orderid=12345
|
||||
select * from hive.hindex.orders where orderid>12345
|
||||
select * from hive.hindex.orders where orderid<12345
|
||||
select * from hive.hindex.orders where orderid>=12345
|
||||
select * from hive.hindex.orders where orderid<=12345
|
||||
select * from hive.hindex.orders where orderid between (10000, 20000)
|
||||
select * from hive.hindex.orders where orderid in (12345, 7890)
|
||||
```
|
||||
|
|
@ -0,0 +1,68 @@
|
|||
|
||||
|
||||
# Usage
|
||||
|
||||
Index can be managed using any of the supported clients, such as hetu-cli located under the `bin` directory in the installation.
|
||||
|
||||
|
||||
## CREATE
|
||||
To create an index you can run sql statements of the form:
|
||||
```roomsql
|
||||
CREATE INDEX [ IF NOT EXISTS ] index_name
|
||||
USING [ BITMAP | BLOOM | BTREE | MINMAX ]
|
||||
ON tbl_name (col_name)
|
||||
WITH ( "level" = ['STRIPE', 'PARTITION'], "bloom.fpp" = '0.001', [, …] )
|
||||
WHERE predicate;
|
||||
```
|
||||
|
||||
- `WHERE` predicate can be used to create index on select partition(s)
|
||||
- `WITH` can be used to specify index properties or index level. See individual index documentation to support properties.
|
||||
- `"level"='STRIPE'` if not specified
|
||||
|
||||
If the table is partitioned, you can specify a single partition to create an index on, or an in-predicate to specify multiple partitions:
|
||||
|
||||
```roomsql
|
||||
CREATE INDEX index_name USING bloom ON hive.schema.table (column1);
|
||||
CREATE INDEX index_name USING bloom ON hive.schema.table (column1) WITH ("bloom.fpp"="0.01") WHERE p=part1;
|
||||
CREATE INDEX index_name USING bloom ON hive.schema.table (column1) WHERE p in (part1, part2, part3);
|
||||
```
|
||||
|
||||
## SHOW
|
||||
|
||||
To show all indexes or a specific index_name:
|
||||
```roomsql
|
||||
SHOW INDEX;
|
||||
SHOW INDEX index_name;
|
||||
```
|
||||
|
||||
## DROP
|
||||
|
||||
To delete an index by name:
|
||||
```roomsql
|
||||
DROP INDEX index_name
|
||||
WHERE predicate;
|
||||
```
|
||||
|
||||
- `WHERE` predicate be used to delete index for specific partition(s). However, if index was initially created on the entire table, it is not possible to delete index for a single partition.
|
||||
|
||||
```roomsql
|
||||
DROP INDEX index_name where p=part1;
|
||||
```
|
||||
|
||||
Note: dropped index will be removed from cache after a few seconds, so you may still see the next few queries still using the index.
|
||||
|
||||
|
||||
## Notes on resource usage
|
||||
|
||||
### Disk usage
|
||||
Heuristic index uses the local temporary directory (default `/tmp` on linux) while creating and processing indexes while running.
|
||||
Therefore, the temporary directory should have sufficient space. To change the temporary directory set the following property in `etc/jvm.config`:
|
||||
|
||||
```
|
||||
-Djava.io.tmpdir=/path/to/another/dir
|
||||
```
|
||||
|
||||
The size of the index depends closely on the column properties such as number of unique values.
|
||||
As a rough estimate, the available temporary disk space should be table size divided by the number of columns.
|
||||
For example, e.g. for a table of 100GB with five mostly unique columns, 25GB of temporary disk space should be available.
|
||||
|
||||
|
|
@ -1,97 +0,0 @@
|
|||
|
||||
|
||||
# Index Command Line Interface
|
||||
|
||||
## Usage
|
||||
|
||||
The indexer can be utilized using the hetu-cli executable located under the `bin` directory in the installation.
|
||||
|
||||
To create an index you can run sql queries of the form:
|
||||
```roomsql
|
||||
CREATE INDEX [ IF NOT EXISTS ] index_name
|
||||
USING [ BITMAP | BLOOM | MINMAX ]
|
||||
ON tbl_name (col_name)
|
||||
WITH ( "bloom.fpp" = '0.001', [, …] )
|
||||
WHERE predicate;
|
||||
```
|
||||
|
||||
To show all indexes or a specific index_name:
|
||||
```roomsql
|
||||
SHOW INDEX;
|
||||
SHOW INDEX index_name;
|
||||
```
|
||||
|
||||
To delete an index by name:
|
||||
```roomsql
|
||||
DROP INDEX index_name;
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
Path white list：["/tmp", "/opt/hetu", "/opt/openlookeng", "/etc/hetu", "/etc/openlookeng", current workspace]
|
||||
|
||||
`etc` directory includes config.properties, and --config should specify an absolute path,
|
||||
the path should be children directory of Path white list
|
||||
|
||||
### Create index
|
||||
|
||||
``` shell
|
||||
$ java -jar ./hetu-cli-*.jar --config /xxx/etc --execute 'CREATE INDEX index_name USING bloom ON hive.schema.table (column1) WITH ("bloom.fpp"="0.01", verbose=true) WHERE p=part1'
|
||||
```
|
||||
|
||||
### Show index
|
||||
|
||||
``` shell
|
||||
$ java -jar ./hetu-cli-*.jar --config /xxx/etc --execute "SHOW INDEX index_name"
|
||||
```
|
||||
|
||||
### Drop index
|
||||
|
||||
``` shell
|
||||
$ java -jar ./hetu-cli-*.jar --config /xxx/etc --execute "DROP INDEX index_name"
|
||||
```
|
||||
|
||||
*Note*: Dropping an index will not remove the cached index from hetu server. This means the index may still be used until it expires from cache based on `hetu.heuristicindex.filter.cache.ttl` value or hetu server is restarted.
|
||||
|
||||
## Notes on resource usage
|
||||
|
||||
### Memory
|
||||
|
||||
By default the default JVM MaxHeapSize will be used (`java -XX:+PrintFlagsFinal -version | grep MaxHeapSize`). For improved performance, it is recommended to increase the MaxHeapSize. This can be
|
||||
done by setting -Xmx value:
|
||||
|
||||
*Note*: This should be done before executing the hetu-cli.
|
||||
``` shell
|
||||
export JAVA_TOOL_OPTIONS="-Xmx100G"
|
||||
```
|
||||
|
||||
In this example the MaxHeapSize will be set to 100G.
|
||||
|
||||
### Disk usage
|
||||
Heuristic index uses local temporary storage to create the indices then tar them onto hdfs. Therefore, it requires enough local disk space where the temporary directory (e.g. `/tmp` for linux) is mounted. Users can specify the temporary directory for the index writer to use if the default one is mounted on the disk with insufficient space. To do this, run the cli with `-Djava.io.tmpdir` flag:
|
||||
|
||||
```bash
|
||||
java -Djava.io.tmpdir=/path/to/another/dir -jar ./hetu-cli-*.jar
|
||||
```
|
||||
|
||||
We have a rough estimation of how much disk space the indices may take for bloom index. It is linear to the original size of the table on which the index is created, and is linear to the negative logarithmic of `fpp`. The smaller fpp and larger dataset, the bigger index will be created:
|
||||
|
||||
index size = -log(fpp) * table size on disk * C
|
||||
|
||||
The coefficient C varies depending on some other factors, which are not as significant and usually change less, such as how much the selected column weighs in the table. As a typical value, its around 0.04. Therefore, for a typical table with a few columns which is 100GB on hdfs, creating a bloom index with `fpp=0.001` on a column will take about 12GB space, while it takes around 16GB if `fpp` is set at `0.0001`.
|
||||
|
||||
### Indexing in parallel
|
||||
|
||||
If creating the index for a large table is too slow on one machine, you can create an index for different partitions in parallel on different machines. Just make sure they are assigned with different partitions. For example:
|
||||
|
||||
On machine 1:
|
||||
|
||||
``` bash
|
||||
$ java -jar ./hetu-cli-*.jar --config /xxx/etc --execute 'CREATE INDEX index_name USING bloom ON hive.schema.table (column1) WITH ("bloom.fpp"="0.01") WHERE p=part1'
|
||||
```
|
||||
|
||||
On machine 2:
|
||||
|
||||
``` shell
|
||||
$ java -jar ./hetu-cli-*.jar --config /xxx/etc --execute 'CREATE INDEX index_name USING bloom ON hive.schema.table (column1) WITH ("bloom.fpp"="0.01") WHERE p=part2'
|
||||
```
|
||||
|
|
@ -1,22 +1,44 @@
|
|||
|
||||
# Minmax Index
|
||||
# MinMax Index
|
||||
|
||||
## Use cases
|
||||
MinMax simply keeps tracks of the largest and smallest value.
|
||||
The size of the index is extremely small.
|
||||
However, this index will only be useful if the table is sorted
|
||||
on the indexed column.
|
||||
|
||||
MinMax Index is used for split filtering, and is used only by the **coordinator** nodes.
|
||||
## Filtering
|
||||
|
||||
If this index exists on a column which is part of a predicate in the query, the engine may be able to improve performance by filtering scheduled splits similar to Bloom Index.
|
||||
1. MinMax Index is used on coordinator for filtering splits during scheduling
|
||||
|
||||
For example if an index exists on column
|
||||
## Selecting column for MinMax Index
|
||||
|
||||
`age`
|
||||
MinMax Index will only work well on columns on which the table is sorted.
|
||||
For example, ID or age.
|
||||
|
||||
and the query is
|
||||
## Supported operators
|
||||
|
||||
= Equality
|
||||
> Greater than
|
||||
>= Greater than or equal
|
||||
< Less than
|
||||
<= Less than or equal
|
||||
|
||||
## Examples
|
||||
|
||||
Creating index:
|
||||
```sql
|
||||
select * from table where age > 50
|
||||
create index idx using minmax on hive.hindex.users (age);
|
||||
create index idx using minmax on hive.hindex.users (age) where regionkey=1;
|
||||
create index idx using minmax on hive.hindex.users (age) where regionkey in (3, 1);
|
||||
```
|
||||
|
||||
* assuming users table is partitioned on `regionkey`
|
||||
|
||||
|
||||
*Tip: sorting the data on the index column will provide the best results*
|
||||
Using index:
|
||||
```sql
|
||||
select name from hive.hindex.users where age=20
|
||||
select name from hive.hindex.users where age>20
|
||||
select name from hive.hindex.users where age<20
|
||||
select name from hive.hindex.users where age>=20
|
||||
select name from hive.hindex.users where age<=20
|
||||
```
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
# Adding your own Index Type
|
||||
|
||||
## Basic ideas
|
||||
|
||||
### ID
|
||||
|
||||
Each index type must have an `ID`, which is a unique identifier and the canonical name of an index type used in CLI, server config, file path, etc.
|
||||
|
||||
The `getID()` method in the `Index` interface returns the ID of this index type to any infrastructure that uses it.
|
||||
|
||||
### Level
|
||||
|
||||
A heuristic index stores additional and usually partial information of a dataset in a more compact way to speed up lookups in various ways. Therefore, each index must
|
||||
have a domain on which it is applied. For instance, if an index marks the max value of a data set, we must know how big the data set is when we define the "max" value (i.e.
|
||||
it can be the max of a group of rows, a data partition, or even a whole table). When a new index type is created, it must implement a method `Set<Level> getSupportedIndexLevels();`
|
||||
which returns the data set level it can support. The levels are defined as an enum in `Index` interface.
|
||||
|
||||
## Interface overlook
|
||||
|
||||
### Indexing methods
|
||||
|
||||
Apart from the methods metioned above, this section gives a quick guide on the most important methods needed to create a new index type. For the complete
|
||||
document on `Index` interface, please refer to the Java Doc of the source code.
|
||||
|
||||
There are two main functionalities in the `Index` interface:
|
||||
|
||||
```java
|
||||
boolean matches(Object expression) throws UnsupportedOperationException;
|
||||
|
||||
<I> Iterator<I> lookUp(Object expression) throws UnsupportedOperationException;
|
||||
```
|
||||
|
||||
The first `matches()` method takes an expression, and returns if a predicate (the expression) can hold on a specific **Level** of data. For example, if an index records
|
||||
the max value of an integer column, it could easily tell if `col_val > 5` can hold by comparing `5` to the max value.
|
||||
|
||||
The second `lookUp()` method is optional. Instead of only returning a boolean about whether a predicate can hold, it returns an iterator of all the possible positions
|
||||
where the data can be found. An index should always have the same result on `matches()` and `lookUp().hasNext()`.
|
||||
|
||||
### Adding and persisting values
|
||||
|
||||
The following methods are used to add values to the index and persist index objects onto disk:
|
||||
|
||||
```java
|
||||
boolean addValues(Map<String, List<Object>> values) throws IOException;
|
||||
|
||||
Index deserialize(InputStream in) throws IOException;
|
||||
|
||||
void serialize(OutputStream out) throws IOException;
|
||||
```
|
||||
|
||||
The usage of them are pretty straightforward. A good example to help understand their usage is the source code of `MinMaxIndex`, where adding values is just to
|
||||
update the `max` and `min` variables according to the input number, and `serialize()/deserialize()`
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
|
||||
# openLooKeng Heuristic Indexer
|
||||
# openLooKeng Heuristic Index
|
||||
|
||||
## Introduction
|
||||
|
||||
|
|
@ -12,73 +12,168 @@ The Heuristic Indexer allows creating indexes on existing data but stores the in
|
|||
- New index types not supported by the underlying data source can be created
|
||||
- Index data does not use the storage space of the data source
|
||||
|
||||
|
||||
## Use cases
|
||||
|
||||
Currently, heuristic indexer is supported on hive ORC data source to reduce the number of splits or rows read.
|
||||
**Currently, Heuristic Index is only supports the Hive connector with
|
||||
tables using ORC storage format.**
|
||||
|
||||
### 1. Filtering scheduled Splits during query execution
|
||||
|
||||
When the engine needs to schedule a TableScan operation, it schedules Splits on the workers. These Splits are responsible for reading a portion of the source data. However, not all Splits will return data if a predicate is applied.
|
||||
*Index types supported: Bloom Index, Btree Index, MinMax Index*
|
||||
|
||||
When the engine needs to read data from a data source it schedules Splits.
|
||||
However, not all Splits will return data if a predicate is applied.
|
||||
|
||||
For example: `select * from test_base where j1='070299439'`
|
||||
|
||||
By keeping an external index for the predicate column, the Heuristic Indexer can determine whether each split contains the values being searched for and only schedule the read operation for the splits which possibly contain the value.
|
||||
|
||||
|
||||
|
||||
### 2. Filtering Block early when reading ORC files
|
||||
### 2. Filtering Stripes when reading ORC files
|
||||
|
||||
When data needs to be read from an ORC file, the ORCRecordReader is used. This reader reads data from Stripes as batches (e.g. 1024 rows), which then form Pages. However, if a predicate is present, not all entries in the batch are required, some may be filtered out later by the Filter operator.
|
||||
*Index types supported: Bloom Index, MinMax Index*
|
||||
|
||||
By keeping an external bitmap index for the predicate column, the Heuristic Indexer can filter out rows which do not match the predicates before the Filter operator is even applied.
|
||||
Similar to Split filtering above, when using the Hive connector to read ORC tables,
|
||||
Stripes can be filtered out based on the specified predicate. This reduces the amount
|
||||
of data read and improves query performance.
|
||||
|
||||
## Example tutorial
|
||||
### 3. Filtering rows when reading ORC files
|
||||
|
||||
This section gives a short tutorial which introduces the basic usage of heuristic index through a sample query.
|
||||
*Index types supported: Bitmap Index*
|
||||
|
||||
### Identify a type of index on a column that can help
|
||||
Going one level lower, once the rows are read, they must be filtered if a predicate is present.
|
||||
This involves reading rows and then using the Filter operator to discard
|
||||
rows that do not match the predicate.
|
||||
|
||||
For a query:
|
||||
By creating a Bitmap Index for the predicate column, the Heuristic Indexer will only read
|
||||
rows which match the predicate, before the Filter operator is even applied. This can reduce
|
||||
memory and cpu usage and result in improved query performance, especially at higher concurrency.
|
||||
|
||||
SELECT * FROM table1 WHERE id="abcd1234";
|
||||
|
||||
A bloom index on id column can significantly decrease the splits to read when scanning table1.
|
||||
|
||||
We will use this example throughout this tutorial.
|
||||
## Getting started
|
||||
|
||||
### Configure indexer settings
|
||||
This section gives a short tutorial which introduces the basic usage of Heuristic Index through a sample query.
|
||||
For a complete list of configuration properties see [Properties](../admin/properties.md).
|
||||
|
||||
### 1. Configure indexer settings
|
||||
|
||||
In `etc/config.properties`, add these lines:
|
||||
|
||||
Path white list：["/tmp", "/opt/hetu", "/opt/openlookeng", "/etc/hetu", "/etc/openlookeng", current workspace]
|
||||
|
||||
Notice：avoid to choose root directory; ../ can't include in path; if you config node.date_dir, then the current workspace is the parent of node.data_dir;
|
||||
otherwise, the current workspace is the openlookeng server's directory.
|
||||
|
||||
hetu.heuristicindex.filter.enabled=true
|
||||
hetu.heuristicindex.filter.cache.max-memory=2GB
|
||||
hetu.heuristicindex.filter.cache.max-memory=10GB
|
||||
hetu.heuristicindex.indexstore.uri=/opt/hetu/indices
|
||||
hetu.heuristicindex.indexstore.filesystem.profile=index-store-profile
|
||||
|
||||
Then create an hdfs client profile in `etc/filesystem/index-store-profile.properties`, where `index-store-profile` is what specified above as the file name:
|
||||
|
||||
Path whitelist：`["/tmp", "/opt/hetu", "/opt/openlookeng", "/etc/hetu", "/etc/openlookeng", current workspace]`
|
||||
|
||||
**Note**：
|
||||
- `LOCAL` filesystem type should only be used in testing or single node clusters.
|
||||
- `HDFS` filesystem type should be used in production in order for the index to be accessible by all nodes in the cluster.
|
||||
- All nodes should be configured to use the same filesystem profile.
|
||||
- Heuristic Index can be disabled while the engine is running by setting: `set session heuristicindex_filter_enabled=false;`
|
||||
|
||||
Filesystem profile file should be placed at `etc/filesystem/index-store-profile.properties`,
|
||||
where `index-store-profile` is the name referenced above in `config.properties`:
|
||||
|
||||
fs.client.type=hdfs
|
||||
hdfs.config.resources=/path/to/core-site.xml,/path/to/hdfs-site.xml
|
||||
hdfs.authentication.type=NONE
|
||||
fs.hdfs.impl.disable.cache=true
|
||||
|
||||
If the hdfs cluster enables kerberos authentication, additional configs `hdfs.krb5.conf.path, hdfs.krb5.keytab.path, hdfs.krb5.principal` needs to be configured.
|
||||
If the kerberos authentication is enabled, additional configs `hdfs.krb5.conf.path`,
|
||||
`hdfs.krb5.keytab.path`, and `hdfs.krb5.principal` must be provided.
|
||||
|
||||
In this example we use a hdfs cluster to store the index files, which can be shard across different hetu servers. If you would like to use local disk to store index, simply change `index-store-profile.properties` to:
|
||||
With Heuristic Indexer configured, start the engine.
|
||||
|
||||
fs.client.type=local
|
||||
### 2. Identify a column to create index on
|
||||
|
||||
For a query like:
|
||||
|
||||
SELECT * FROM hive.schema.table1 WHERE id="abcd1234";
|
||||
|
||||
Where `id` is unique, a Bloom Index on can significantly decrease the splits to read
|
||||
when scanning table1.
|
||||
|
||||
### 3. Create index
|
||||
|
||||
To create index run the following statement:
|
||||
|
||||
CREATE INDEX index_name USING bloom ON table1 (column);
|
||||
|
||||
Note that you may create multiple filesystem profiles in `etc/filesystem`, several for different hdfs clusters and one for local, so you can easily switch between them by just changing the value of `hetu.heuristicindex.indexstore.filesystem.profile`.
|
||||
### 4. Run query
|
||||
|
||||
### Create index
|
||||
After index is created, run the query; index will start loading in the background.
|
||||
Subsequent queries will utilize the index to reduce the amount of data read
|
||||
and query performance will be improved.
|
||||
|
||||
To write index to the indexstore specified above, just change directory to your hetu installation's `bin` folder, then run:
|
||||
|
||||
java -jar ./hetu-cli-*.jar --config <your-etc-folder-directory> --execute 'CREATE INDEX index_name USING bloom ON table1 (column)'
|
||||
## Index Statements
|
||||
|
||||
See [Heuristic Index Statements](./hindex-statements.md).
|
||||
|
||||
-----
|
||||
|
||||
## Supported Index Types
|
||||
|
||||
| Index ID | Filtering type | Best Column type | Supported query operators | Notes | Example |
|
||||
|----------|-----------------|--------------------------------------------|---------------------------------------|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| [Bloom](./bloom.md) | Split<br>Stripe | High cardinality<br>(such as an ID column) | `=` `IN` | | `create index idx using bloom on hive.hindex.users (id);`<br>`select name from hive.hindex.users where id=123` |
|
||||
| [Btree](./btree.md) | Split | High cardinality<br>(such as an ID column) | `=` `>` `>=` `<` `<=` `IN` `BETWEEN` | Table must be partitioned | `create index idx using btree on hive.hindex.users (id) where regionkey IN (1,4) with ("level"='partition')`<br>(assuming table is partitioned on regionkey)<br>`select name from hive.hindex.users where id>123` |
|
||||
| [MinMax](./minmax.md) | Split<br>Stripe | Column which table is sorted on | `=` `>` `>=` `<` `<=` | | `create index idx using bloom on hive.hindex.users (age);`<br>(assuming users is sorted by age)<br>`select name from hive.hindex.users where age>25` |
|
||||
| [Bitmap](./bitmap.md) | Row | Low cardinality<br>(such as Gender column) | `=` `IN` | | `create index idx using bitmap on hive.hindex.users (gender);`<br>`select name from hive.hindex.users where gender='female'` |
|
||||
|
||||
**Note:** unsupported operators will still function correctly but will not benefit from the index.
|
||||
|
||||
|
||||
## Choosing Index Type
|
||||
|
||||
The Heuristic Indexer helps with queries where data is being filtered by a predicate.
|
||||
Identify the column on which data is being filtered and use the decision flowchart to
|
||||
help decide what type of index will work best.
|
||||
|
||||
Cardinality means the number of distinct values in the column relative to
|
||||
number of total rows. For example, an `ID` column has a high cardinality
|
||||
because IDs are unique. Whereas `employeeType` will have low cardinality
|
||||
because there are likely only a few different types (e.g. Manager, Developer,
|
||||
Tester).
|
||||
|
||||
|
||||
|
||||
Example queries:
|
||||
|
||||
1. `SELECT id FROM employees WHERE site = 'lab';`
|
||||
|
||||
In this query `site` has a low cardinality (i.e. not many sites) so **Bitmap Index** will help.
|
||||
|
||||
2. `SELECT * FROM visited WHERE id = '34857' AND date < '2020-01-01';`
|
||||
|
||||
In this query `id` has a high cardinality (i.e. IDs are likely unique)
|
||||
and table is partitioned on `date` so **Btree Index** will help.
|
||||
|
||||
### Run query
|
||||
3. `SELECT * FROM salaries WHERE salary > 50251.40;`
|
||||
|
||||
After this is finished, run the query on hetu server again, and it will start loading index in the background. Keep running the same query while it's loading, then you should see decrease in the number of splits processed, which finally goes down to a rather small value.
|
||||
In this query `salary` has a high cardinality (i.e. salary of employees
|
||||
will slightly vary) and assuming `salaries` table is sorted on `salary`,
|
||||
**MinMax Index** will help.
|
||||
|
||||
4. `SELECT * FROM assets WHERE id = 50;`
|
||||
|
||||
In this query `id` has a high cardinality (i.e. IDs are likely unique)
|
||||
but the table is not partitioned, so **Bloom Index** will help.
|
||||
|
||||
5. `SELECT * FROM phoneRecords WHERE phone='1234567890' and type = 'outgoing' and date > '2020-01-01';`
|
||||
|
||||
In this query `phone` has a high cardinality (i.e. there are many phone numbers, even if they
|
||||
made multiple calls), `type` has low cardinality (only outgoing or incoming),
|
||||
and the data is partitioned on date. Creating a **Btree Index** on `phone`
|
||||
and a **Bitmap Index** on `type` will help.
|
||||
|
||||
## Adding your own Index Type
|
||||
|
||||
See [Adding your own Index Type](./new-index.md).
|
||||
|
||||
## Access Control
|
||||
|
||||
See [Built-in System Access Control](../security/built-in-system-access-control.md).
|
||||
|
|
|
|||
|
|
@ -181,4 +181,31 @@ For example, if you want to allow only the user `admin` and `alice` to update th
|
|||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Heuristic Index Rules
|
||||
|
||||
The rules govern the Heuristic Index operations particular users can perform.
|
||||
|
||||
Each rule is composed of the following fields:
|
||||
|
||||
- `user` (required): regex to match against user name. Defaults to `.*`.
|
||||
- `privileges` (optional): list of privileges granted to user (`ALL`, `SHOW`, `CREATE`, `DROP`, `RENAME`, and `UPDATE`). Defaults to `ALL`.
|
||||
|
||||
For example, here user `tom` can only execute `SHOW INDEX` or `CREATE INDEX` statements.
|
||||
But user `admin` can execute all statements `CREATE INDEX`, `SHOW INDEX`, `DROP INDEX`, etc.
|
||||
|
||||
```json
|
||||
{
|
||||
"indexAccess": [
|
||||
{
|
||||
"user": "tom",
|
||||
"privileges": ["SHOW", "CREATE"]
|
||||
},
|
||||
{
|
||||
"user": "admin",
|
||||
"privileges": ["ALL"]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
|
|
@ -1,19 +1,37 @@
|
|||
|
||||
# 位图索引
|
||||
# BitMap（位图）索引
|
||||
|
||||
BitMap索引使用位图。索引的大小随着索引列中不同值的个数而增加。例如，一个标记性别的列很小，而一个ID列的索引则会极大（不推荐）。
|
||||
|
||||
注意：在ORC算子下推启用时，BitMap索引效果更好。可以通过设置`hive.properties`中的`hive.orc-predicate-pushdown-enabled=true`来启用，
|
||||
或者在命令行中启用`set session hive.orc_predicate_pushdown_enabled=true;`。
|
||||
|
||||
参见[Properties](../admin/properties.md)获得更多信息。
|
||||
|
||||
## 过滤
|
||||
|
||||
1. BitMap索引用于过滤从ORC文件中读取的数据，且仅供worker节点使用。
|
||||
|
||||
## 选择适用的列
|
||||
|
||||
BitMap索引在拥有较少不同值数量的列上比较适用，例如：性别。
|
||||
|
||||
## 支持的运算符
|
||||
|
||||
= Equality
|
||||
|
||||
## 用例
|
||||
|
||||
位图索引用于过滤从ORC文件中读取的数据，且仅供**worker**节点使用。
|
||||
|
||||
- 如果包含这个索引的列是查询中谓词的一部分，那么读取ORC文件的性能可能会得提升。
|
||||
|
||||
例如，如果索引在`country`列，并且查询语句是
|
||||
|
||||
``` sql
|
||||
select * from table where country="China"
|
||||
创建：
|
||||
```sql
|
||||
create index idx using bitmap on hive.hindex.users (gender);
|
||||
create index idx using bitmap on hive.hindex.users (gender) where regionkey=1;
|
||||
create index idx using bitmap on hive.hindex.users (gender) where regionkey in (3, 1);
|
||||
```
|
||||
|
||||
- 如果列的值不是太明显（例如国家）和分散，则此索引最有效。
|
||||
|
||||
例如，假设表存储的是用户来自何处的信息，并且表数据存在于10个文件中。可能有多个用户来自某一国家，因此每个文件将有一些来自该国的用户。如果我们在国家列创建一个位图索引，那么在读取数据文件时，我们可以在早期执行过滤。即，谓词被下推到文件读取。如果没有这个索引，所有的数据文件将会作为页读入内存，然后再过滤。如果有此索引，我们可以确保内存页中已经只包含与谓词匹配的行。这有助于减少内存和CPU使用率，并且提高多并发查询的性能。
|
||||
* 假设表已按照`regionkey`列分区
|
||||
|
||||
使用:
|
||||
```sql
|
||||
select name from hive.hindex.users where gender="female"
|
||||
```
|
||||
|
|
@ -1,20 +1,45 @@
|
|||
|
||||
# Bloom索引
|
||||
|
||||
Bloom索引实用布隆过滤器来过滤数据。索引体积非常小。
|
||||
|
||||
## 过滤
|
||||
|
||||
1. Bloom索引用于调度时的分片过滤，被coordinator节点使用。
|
||||
2. Bloom索引也用于worker节点上，用于在读取ORC文件是过滤stripes。
|
||||
|
||||
## 选择适用的列
|
||||
|
||||
位图索引在拥有较多不同值数量的列上比较适用，例如：ID。
|
||||
|
||||
## 支持的运算符
|
||||
|
||||
= Equality
|
||||
|
||||
## 配置参数
|
||||
|
||||
### `bloom.fpp`
|
||||
|
||||
> - **类型:** `Double`
|
||||
> - **默认值:** `0.001`
|
||||
>
|
||||
> 改变布隆过滤器的FPP (false positive probability)。
|
||||
> 更小的FPP会提高索引的过滤能力，但是会增加索引的体积。在大多数情况下默认值就足以够用。
|
||||
> 如果创建的索引太大，可以考虑增加这个值（例如，至0.05)。
|
||||
|
||||
## 用例
|
||||
|
||||
Bloom索引用于分片过滤，且仅被**coordinator**节点使用。
|
||||
|
||||
- 如果查询中作为谓词一部分的列存在此索引，openLooKeng可以通过筛选预定Splits来提高查询性能。
|
||||
|
||||
例如，如果列`id`包含此索引，并且查询语句如下：
|
||||
|
||||
创建索引:
|
||||
```sql
|
||||
select * from table where id=12345
|
||||
create index idx using bloom on hive.hindex.users (id);
|
||||
create index idx using bloom on hive.hindex.users (id) where regionkey=1;
|
||||
create index idx using bloom on hive.hindex.users (id) where regionkey in (3, 1);
|
||||
create index idx using bloom on hive.hindex.users (id) WITH ("bloom.fpp" = '0.001');
|
||||
```
|
||||
|
||||
- 如果列的值是唯一的（例如userid）并且不太分散，则Bloom索引最有效。
|
||||
* 假设表已按照`regionkey`列分区
|
||||
|
||||
例如，假设表中存储了用户信息，且表数据存在于10个文件中。 对于给定的用户ID，只有一个文件包含该数据。因此，对用户ID创建索引将帮助我们在调度时间内过滤掉10个文件中的9个文件，并节省大量用于读取每个文件的IO时间。
|
||||
|
||||
*提示：如果可能，建议对索引列中的数据进行排序*。
|
||||
使用:
|
||||
```sql
|
||||
select name from hive.hindex.users where id=123
|
||||
```
|
||||
|
|
@ -0,0 +1,48 @@
|
|||
# BTree索引
|
||||
|
||||
BTree索引使用二叉树数据结构存储。索引的大小随着索引列中不同值的个数而增加。
|
||||
|
||||
## 过滤
|
||||
|
||||
1. Bloom索引用于调度时的分片过滤，被coordinator节点使用。
|
||||
|
||||
## 选择适用的列
|
||||
|
||||
位图索引在拥有较多不同值数量的列上比较适用，例如：ID。除此之外，BTree索引还要求表是分区的。
|
||||
|
||||
在BTree和Bloom索引之间选择时，需要考虑：
|
||||
- Bloom索引只支持`=`
|
||||
- Btree索引要求表是分区的
|
||||
- Bloom索引是不确定的，而BTree索引是确定的。因此BTree通常有更好的过滤性能
|
||||
- BTree索引比Bloom索引更大
|
||||
|
||||
## 支持的运算符
|
||||
|
||||
= Equality
|
||||
> Greater than
|
||||
>= Greater than or equal
|
||||
< Less than
|
||||
<= Less than or equal
|
||||
BETWEEN Between range
|
||||
IN IN set
|
||||
|
||||
## 用例
|
||||
|
||||
创建索引:
|
||||
```sql
|
||||
create index idx using btree on hive.hindex.orders (orderid) with (level=partition) where orderDate='01-10-2020' ;
|
||||
create index idx using btree on hive.hindex.orders (orderid) with (level=partition) where orderDate in ('01-10-2020', '01-10-2020');
|
||||
```
|
||||
|
||||
* 假设表已按照`orderDate`列分区
|
||||
|
||||
使用索引:
|
||||
```sql
|
||||
select * from hive.hindex.orders where orderid=12345
|
||||
select * from hive.hindex.orders where orderid>12345
|
||||
select * from hive.hindex.orders where orderid<12345
|
||||
select * from hive.hindex.orders where orderid>=12345
|
||||
select * from hive.hindex.orders where orderid<=12345
|
||||
select * from hive.hindex.orders where orderid between (10000, 20000)
|
||||
select * from hive.hindex.orders where orderid in (12345, 7890)
|
||||
```
|
||||
|
|
@ -0,0 +1,65 @@
|
|||
|
||||
# 使用
|
||||
|
||||
索引命令行接口集成于`hetu-cli`中, 在安装目录的`bin`目录下运行。
|
||||
|
||||
|
||||
## 创建
|
||||
索引创建方法如下:
|
||||
```roomsql
|
||||
CREATE INDEX [ IF NOT EXISTS ] index_name
|
||||
USING [ BITMAP | BLOOM | BTREE | MINMAX ]
|
||||
ON tbl_name (col_name)
|
||||
WITH ( "level" = ['STRIPE', 'PARTITION'], "bloom.fpp" = '0.001', [, …] )
|
||||
WHERE predicate;
|
||||
```
|
||||
|
||||
- `WHERE` 用于选择部分分区创建索引
|
||||
- `WITH` 用于设置索引属性。参见各个索引的文档来查看支持的配置
|
||||
- `"level"='STRIPE'` 如缺省，默认创建级别是STRIPE
|
||||
|
||||
如果表是分区的，可以用一个等于表达式来指定一个创建的分区，或使用IN来指定多个。
|
||||
```roomsql
|
||||
CREATE INDEX index_name USING bloom ON hive.schema.table (column1);
|
||||
CREATE INDEX index_name USING bloom ON hive.schema.table (column1) WITH ("bloom.fpp"="0.01") WHERE p=part1;
|
||||
CREATE INDEX index_name USING bloom ON hive.schema.table (column1) WHERE p in (part1, part2, part3);
|
||||
```
|
||||
|
||||
## SHOW
|
||||
|
||||
显示所有索引或只根据名字显示一个索引：
|
||||
```roomsql
|
||||
SHOW INDEX;
|
||||
SHOW INDEX index_name;
|
||||
```
|
||||
|
||||
## DROP
|
||||
|
||||
根据名字删除一条索引：
|
||||
```roomsql
|
||||
DROP INDEX index_name
|
||||
WHERE predicate;
|
||||
```
|
||||
|
||||
- `WHERE` 表达式用于只删除索引的几个分区。但是，如果在创建时是全表创建的索引，则只删除部分的索引将不可用（只能整条删除）。
|
||||
|
||||
```roomsql
|
||||
DROP INDEX index_name where p=part1;
|
||||
```
|
||||
|
||||
删除的索引不会立即被从服务器的缓存中清除，直到下一次刷新缓存。刷新时间与设置的缓存加载延迟有关，通常在几秒钟左右。
|
||||
|
||||
## 资源使用说明
|
||||
|
||||
### 磁盘使用
|
||||
启发式索引使用本地临时文件夹存储创建的索引，然后打包上传至hdfs。因此，它需要临时文件夹（例如，linux上的`/tmp`)挂载的磁盘分区在本地有足够的可用空间。如果挂载的磁盘分区可用空间不足，用户可以在worker节点的`jvm.config`中通过`-Djava.io.tmpdir`来指定使用的临时路径：
|
||||
|
||||
```
|
||||
-Djava.io.tmpdir=/path/to/another/dir
|
||||
```
|
||||
|
||||
下面的公式给出了一个对于Bloom索引占用磁盘空间的大致估计。Bloom索引使用的空间大致与用于创建索引的表的大小成正比，同时与指定的`fpp`值的对数相反数成正比。因此，更小的fpp值和更大的数据集会使得创建的索引更大：
|
||||
|
||||
索引大小 = -log(fpp) * 表占用空间 * C
|
||||
|
||||
系数C还与其他许多因素相关，例如创建索引的列占表总数据的比重，但这些因素的影响应当不如fpp和表的大小重要，且变化较小。作为一个典型的拥有几个列的数据表，这个系数C在0.04左右。这就是说，为一个100GB的数据表的一列创建一个`fpp=0.001`的索引大致需要12GB磁盘空间，而创建`fpp=0.0001`的索引则需要16GB左右。
|
||||
|
|
@ -1,95 +0,0 @@
|
|||
|
||||
# 索引命令行接口
|
||||
|
||||
## 用法
|
||||
|
||||
索引命令行接口集成于`hetu-cli`中, 在安装目录的`bin`目录下运行。
|
||||
|
||||
命令的使用方式如下:
|
||||
```roomsql
|
||||
CREATE INDEX [ IF NOT EXISTS ] index_name
|
||||
USING [ BITMAP | BLOOM | MINMAX ]
|
||||
ON tbl_name (col_name)
|
||||
WITH ( "bloom.fpp" = '0.001', [, …] )
|
||||
WHERE predicate;
|
||||
```
|
||||
|
||||
显示所有索引或特定的索引名称的方法如下：
|
||||
```roomsql
|
||||
SHOW INDEX;
|
||||
SHOW INDEX index_name;
|
||||
```
|
||||
|
||||
通过索引名称删除索引的方法如下：
|
||||
```roomsql
|
||||
DROP INDEX index_name;
|
||||
```
|
||||
|
||||
## 示例
|
||||
|
||||
路径配置白名单：["/tmp", "/opt/hetu", "/opt/openlookeng", "/etc/hetu", "/etc/openlookeng", 工作目录]
|
||||
|
||||
`etc` 目录包含config.properties，在指定config时，我们需要写绝对路径，该路径必须是白名单中路径的子目录
|
||||
|
||||
### 创建索引
|
||||
|
||||
``` shell
|
||||
$ java -jar ./hetu-cli-*.jar --config /xxx/etc --execute 'CREATE INDEX index_name USING bloom ON hive.schema.table (column1) WITH ("bloom.fpp"="0.01", debugEnabled=true) WHERE p=part1'
|
||||
```
|
||||
|
||||
### 显示索引
|
||||
|
||||
``` shell
|
||||
$ java -jar ./hetu-cli-*.jar --config /xxx/etc --execute "SHOW INDEX index_name"
|
||||
$ java -jar ./hetu-cli-*.jar --config /xxx/etc --execute "SHOW INDEX"
|
||||
```
|
||||
|
||||
### 删除索引
|
||||
|
||||
``` shell
|
||||
$ java -jar ./hetu-cli-*.jar --config /xxx/etc --execute "DROP INDEX index_name"
|
||||
```
|
||||
|
||||
*注意*: 删除索引命令不会将索引文件从hetu server的内存中清除。意味着索引会被继续使用，直到达到`hetu.heuristicindex.filter.cache.ttl`所设置的值或者hetu server被重启.
|
||||
|
||||
|
||||
## 资源使用说明
|
||||
|
||||
### 内存
|
||||
|
||||
默认情况下，将使用默认的JVMMaxHeapSize(`java -XX:+PrintFlagsFinal -version | grep MaxHeapSize`)。为了提高性能，建议增大MaxHeapSize的取值。可以通过设置-Xmx值来实现：
|
||||
|
||||
|
||||
``` shell
|
||||
export JAVA_TOOL_OPTIONS="-Xmx100G"
|
||||
```
|
||||
|
||||
在此示例中，MaxHeapSize被设置为100 GB。
|
||||
|
||||
### 磁盘使用
|
||||
启发式索引使用本地临时文件夹存储创建的索引，然后打包上传至hdfs。因此，它需要临时文件夹（例如，linux上的`/tmp`)挂载的磁盘分区在本地有足够的可用空间。如果挂载的磁盘分区可用空间不足，用户可以在运行命令行时通过`-Djava.io.tmpdir`来指定使用的临时路径：
|
||||
```bash
|
||||
java -Djava.io.tmpdir=/path/to/another/dir -jar ./hetu-cli-*.jar
|
||||
```
|
||||
|
||||
下面的公式给出了一个对于Bloom索引占用磁盘空间的大致估计。Bloom索引使用的空间大致与用于创建索引的表的大小成正比，同时与指定的`fpp`值的对数相反数成正比。因此，更小的fpp值和更大的数据集会使得创建的索引更大：
|
||||
|
||||
索引大小 = -log(fpp) * 表占用空间 * C
|
||||
|
||||
系数C还与其他许多因素相关，例如创建索引的列占表总数据的比重，但这些因素的影响应当不如fpp和表的大小重要，且变化较小。作为一个典型的拥有几个列的数据表，这个系数C在0.04左右。这就是说，为一个100GB的数据表的一列创建一个`fpp=0.001`的索引大致需要12GB磁盘空间，而创建`fpp=0.0001`的索引则需要16GB左右。
|
||||
|
||||
### 并行索引
|
||||
|
||||
如果在一台机器上为一个大表创建索引的速度太慢，则可以在不同的机器上并行为不同的分区创建索引。只要保证这些并行创建的分区不冲突即可。例如：
|
||||
|
||||
在机器1上：
|
||||
|
||||
``` bash
|
||||
$ java -jar ./hetu-cli-*.jar --config /xxx/etc --execute 'CREATE INDEX index_name USING bloom ON hive.schema.table (column1) WITH ("bloom.fpp"="0.01") WHERE p=part1'
|
||||
```
|
||||
|
||||
在机器2上：
|
||||
|
||||
``` shell
|
||||
$ java -jar ./hetu-cli-*.jar --config /xxx/etc --execute 'CREATE INDEX index_name USING bloom ON hive.schema.table (column1) WITH ("bloom.fpp"="0.01") WHERE p=part2'
|
||||
```
|
||||
|
|
@ -1,20 +1,41 @@
|
|||
|
||||
# Minmax索引
|
||||
# MinMax索引
|
||||
|
||||
MinMax索引简单地记录数据的最大和最小值，占用空间极小。
|
||||
因此，这一索引仅仅能被用于已经排序的数据列。
|
||||
|
||||
## 过滤
|
||||
|
||||
1. MinMax索引用于调度时的分片过滤，被coordinator节点使用。
|
||||
|
||||
## 选择适用的列
|
||||
|
||||
MinMax索引仅仅能被用于已经排序的数据列。例如，ID或年龄.
|
||||
|
||||
## 支持的运算符
|
||||
|
||||
= Equality
|
||||
> Greater than
|
||||
>= Greater than or equal
|
||||
< Less than
|
||||
<= Less than or equal
|
||||
|
||||
## 用例
|
||||
|
||||
MinMax 索引用于分片过滤，仅被**coordinator**节点使用。
|
||||
|
||||
如果查询中作为谓词一部分的列存在此索引，则引擎可以通过筛选预定Splits来提高性能，类似于Bloom索引。
|
||||
|
||||
例如，如果索引在列
|
||||
|
||||
`age`
|
||||
|
||||
查询语句为
|
||||
|
||||
创建索引:
|
||||
```sql
|
||||
select * from table where age > 50
|
||||
create index idx using minmax on hive.hindex.users (age);
|
||||
create index idx using minmax on hive.hindex.users (age) where regionkey=1;
|
||||
create index idx using minmax on hive.hindex.users (age) where regionkey in (3, 1);
|
||||
```
|
||||
|
||||
*提示：对索引列中的数据进行排序将提供最佳结果。*
|
||||
* 假设表已按照`regionkey`列分区
|
||||
|
||||
使用索引:
|
||||
```sql
|
||||
select name from hive.hindex.users where age=20
|
||||
select name from hive.hindex.users where age>20
|
||||
select name from hive.hindex.users where age<20
|
||||
select name from hive.hindex.users where age>=20
|
||||
select name from hive.hindex.users where age<=20
|
||||
```
|
||||
|
|
@ -0,0 +1,48 @@
|
|||
# 创建自定义索引
|
||||
|
||||
## 基本概念
|
||||
|
||||
### ID
|
||||
|
||||
每一个索引类型都必须有一个`ID`变量作为其唯一标识符。这一名称将用于CLI，服务器配置，存储索引的文件路径中等各种场合，用于唯一地确定这一索引类型。
|
||||
|
||||
`Index`接口的`getID()`方法会将索引的ID返回给它的使用者。
|
||||
|
||||
### Level （数据）层级
|
||||
|
||||
启发式索引用各种不同的方法，存储了在原始数据之外额外并且通常更小的信息，用于提高随机查询的效率。因此，每一条索引都必须有它作用的数据域范围。例如，如果一个索引
|
||||
记录了数据集的"最大"数值，那这一"最大"数值必须拥有它作用的范围（可以是一组数据行，也可以是一个数据分区，甚至整张数据表的最大值）。当创建一个新的索引类型是，必须
|
||||
实现`Set<Level> getSupportedIndexLevels();` 方法来告知它可以支持的数据层级。这些层级由`Index`接口中的一个枚举类型定义。
|
||||
|
||||
## 接口概览
|
||||
|
||||
### 索引方法
|
||||
|
||||
除了上面提到的方法以外，这一段落将介绍其他几个实现新索引类型最重要的方法。要获取`Index`接口的完整文档，请参阅源代码的Java Doc。
|
||||
|
||||
`Index`接口有两个最重要的方法：
|
||||
|
||||
```java
|
||||
boolean matches(Object expression) throws UnsupportedOperationException;
|
||||
|
||||
<I> Iterator<I> lookUp(Object expression) throws UnsupportedOperationException;
|
||||
```
|
||||
|
||||
第一个`matches()`方法接收一个表达式对象，并返回基于索引存储的信息，表达式对应的数据是否可能存在。例如，如果一个索引标记了数据中的最大值，那他可以容易地判断`col_val > 5`是否可能成立。
|
||||
|
||||
第二个方法`lookUp()`不是必须的。在返回数据是否可能存在以外，他还能返回一个迭代器来返回数据可能存在的位置。对于一个索引，它的`matches()`方法和`lookUp().hasNext()`应当总是返回相同的结果。
|
||||
|
||||
### 插入和存取索引
|
||||
|
||||
下面的几个方法用于向一个索引中添加值，以及在磁盘上存储/读取索引实例：
|
||||
|
||||
```java
|
||||
boolean addValues(Map<String, List<Object>> values) throws IOException;
|
||||
|
||||
Index deserialize(InputStream in) throws IOException;
|
||||
|
||||
void serialize(OutputStream out) throws IOException;
|
||||
```
|
||||
|
||||
这些方法的使用非常直观。为了更好地理解他们的用法，`MinMaxIndex`的源代码可以作为一个很好的例子。在这个索引中，添加数据仅仅需要通过新的数据更新`max`和`min`变量即可。
|
||||
`serialize()/deserialize()`方法则只需要向/从磁盘写入/读取最大和最小值这两个值。
|
||||
|
|
@ -14,50 +14,57 @@
|
|||
|
||||
## 使用场景
|
||||
|
||||
当前，启发式索引支持ORC存储格式的hive数据源，帮助减少读取的分段或行数。
|
||||
**当前，启发式索引支持ORC存储格式的hive数据源。**
|
||||
|
||||
### 1.查询过程中过滤预定分段
|
||||
### 1.查询过程中过滤预定分片
|
||||
|
||||
支持的索引：Bloom, BTree, MinMax
|
||||
|
||||
当引擎需要调度一个TableScan操作时，它可以调度worker节点上的Split。这些Split负责读取部分源数据。但是如果应用了谓词，则并非所有Split都会返回数据。
|
||||
|
||||
例如，`select * from test_base where j1='070299439'`
|
||||
|
||||
通过为谓词列保留外部索引，启发式索引可以确定每个Split是否包含正在搜索的值，并且只对可能包含该值的Split安排读操作。
|
||||
|
||||
|
||||
|
||||
### 2.读取ORC文件时提前筛选块
|
||||
### 2.读取ORC文件时提前筛选Stripes
|
||||
|
||||
当需要从ORC文件中读取数据时，使用ORCRecordReader读取器。此读取器从数据条带批量（例如1024行）读取数据，然后形成页。但是，如果有一个谓词存在，那么就不需要批量读取中的所有条目，有些条目可能会在稍后被Filter运算符过滤掉。
|
||||
支持的索引：Bloom, MinMax
|
||||
|
||||
通过为谓词列保留外部位图索引，启发式索引甚至可以在应用Filter运算符之前筛选出与谓词不匹配的行。
|
||||
与分片过滤类似，当使用Hive Connector读取ORC文件时，Stripe可以被提前过滤来减少读取的数据量，从而提升查询性能。
|
||||
|
||||
### 3.读取ORC文件时筛选行
|
||||
|
||||
支持的索引：Bitmap
|
||||
|
||||
当需要从ORC文件中读取数据时，如果有一个谓词存在，那么就不需要批量读取中的所有行。
|
||||
|
||||
通过为谓词列保留外部位图索引，将实现只读取匹配当行，来提升内存和处理器表现。在服务器高并发时提升尤其明显。
|
||||
|
||||
## 示例教程
|
||||
|
||||
这一教程将通过一个示例查询语句来展示索引的用法。
|
||||
这一教程将通过一个示例查询语句来展示索引的用法。完整的配置请参见[Properties](../admin/properties.md)。
|
||||
|
||||
### 确定索引建立的列
|
||||
|
||||
对于这样的语句：
|
||||
|
||||
SELECT * FROM table1 WHERE id="abcd1234";
|
||||
|
||||
如果id比较唯一，bloom索引可以大大较少读取的分段数量。
|
||||
|
||||
在本教程中我们将以这个语句为例。
|
||||
|
||||
### 配置索引
|
||||
### 1. 配置索引
|
||||
|
||||
在 `etc/config.properties` 中加入这些行：
|
||||
|
||||
路径配置白名单：["/tmp", "/opt/hetu", "/opt/openlookeng", "/etc/hetu", "/etc/openlookeng", 工作目录]
|
||||
|
||||
注意：避免选择根目录；路径不能包含../；如果配置了node.data_dir,那么当前工作目录为node.data_dir的父目录；
|
||||
如果没有配置，那么当前工作目录为openlookeng server的目录
|
||||
|
||||
hetu.heuristicindex.filter.enabled=true
|
||||
hetu.heuristicindex.filter.cache.max-memory=2GB
|
||||
hetu.heuristicindex.filter.cache.max-memory=10GB
|
||||
hetu.heuristicindex.indexstore.uri=/opt/hetu/indices
|
||||
hetu.heuristicindex.indexstore.filesystem.profile=index-store-profile
|
||||
|
||||
|
||||
路径配置白名单：["/tmp", "/opt/hetu", "/opt/openlookeng", "/etc/hetu", "/etc/openlookeng", 工作目录]
|
||||
|
||||
避免选择根目录；路径不能包含../；如果配置了node.data_dir,那么当前工作目录为node.data_dir的父目录；如果没有配置，那么当前工作目录为openlookeng server的目录
|
||||
|
||||
**注意**：
|
||||
- `LOCAL` 本地文件系统只应该被用于本地测试，或单节点部署情形。（否则索引文件将无法在机器之间共享）
|
||||
- `HDFS` 应用于生产环境来在集群中共享数据。
|
||||
- 所有节点必须有相同的文件系统配置。
|
||||
- 在服务器运行中可以通过`set session heuristicindex_filter_enabled=false;`关闭启发式索引。
|
||||
|
||||
然后在`etc/filesystem/index-store-profile.properties`中创建一个HDFS描述文件, 其中`index-store-profile`是上面使用的名字：
|
||||
|
||||
fs.client.type=hdfs
|
||||
|
|
@ -67,18 +74,80 @@
|
|||
|
||||
如果HDFS集群开启了KERBEROS验证，还需要配置`hdfs.krb5.conf.path, hdfs.krb5.keytab.path, hdfs.krb5.principal`.
|
||||
|
||||
在上面这个例子中使用了HDFS作为存储索引文件的位置，这使得索引可以在多个Hetu服务器间共享。如果要使用本地磁盘来存储索引，只要将 `index-store-profile.properties` 的内容改为:
|
||||
### 2. 确定索引建立的列
|
||||
|
||||
fs.client.type=local
|
||||
对于这样的语句：
|
||||
|
||||
SELECT * FROM table1 WHERE id="abcd1234";
|
||||
|
||||
如果id比较唯一，bloom索引可以大大较少读取的分段数量。
|
||||
|
||||
在本教程中我们将以这个语句为例。
|
||||
|
||||
### 3. 创建索引
|
||||
|
||||
要创建索引, 在命令行中输入：
|
||||
|
||||
CREATE INDEX index_name USING bloom ON table1 (column);
|
||||
|
||||
注意：可以在`etc/filesystem`中配置多个不同名字的描述文件, 例如多个HDFS和一个Local，这样就可以通过改变`hetu.heuristicindex.indexstore.filesystem.profile`来在他们之中快速切换。
|
||||
### 4. 运行语句
|
||||
|
||||
### 创建索引
|
||||
完成上面的操作后，再次在Hetu服务器运行这个语句，服务器将在后台自动加载索引。接下来的语句将会从中获得性能提升。
|
||||
|
||||
要创建索引，首先将工作目录cd至安装目录的`bin`文件夹，然后运行：
|
||||
## 索引语句
|
||||
|
||||
java -jar ./hetu-cli-*.jar --config <your-etc-folder-directory> --execute 'CREATE INDEX index_name USING bloom ON table1 (column)'
|
||||
参见 [Heuristic Index Statements](./hindex-statements.md).
|
||||
|
||||
-----
|
||||
|
||||
## 支持的索引类型
|
||||
|
||||
| 索引 ID | 过滤类型 | 最适用的列 | 支持的运算符 | 注释 | 用例 |
|
||||
|----------|-----------------|--------------------------------------------|---------------------------------------|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| [Bloom](./bloom.md) | Split<br>Stripe | 大量不同数据值<br>(如ID) | `=` `IN` | | `create index idx using bloom on hive.hindex.users (id);`<br>`select name from hive.hindex.users where id=123` |
|
||||
| [Btree](./btree.md) | Split | 大量不同数据值<br>(如ID) | `=` `>` `>=` `<` `<=` `IN` `BETWEEN` | 表必须被分区 | `create index idx using btree on hive.hindex.users (id) where regionkey IN (1,4) with ("level"='partition')`<br>(假设表根据regionkey分区)<br>`select name from hive.hindex.users where id>123` |
|
||||
| [MinMax](./minmax.md) | Split<br>Stripe | 列数据被排序 | `=` `>` `>=` `<` `<=` | | `create index idx using bloom on hive.hindex.users (age);`<br>(假设数据根据年龄已排序)<br>`select name from hive.hindex.users where age>25` |
|
||||
| [Bitmap](./bitmap.md) | Row | 少量不同数据值<br>(如性别) | `=` `IN` | | `create index idx using bitmap on hive.hindex.users (gender);`<br>`select name from hive.hindex.users where gender='female'` |
|
||||
|
||||
**注意:** 包含不支持的运算符的语句依然会正常运行，但是不会从启发式索引中获得性能提升。
|
||||
|
||||
|
||||
## 选择索引类型
|
||||
|
||||
启发式索引用于根据谓词表达式过滤数据。请根据下面的决策流程图选择适用于数据列的最佳索引。
|
||||
|
||||
Cardinality 是指数据集中值域的大小。例如，`ID`列通常有很大的cardinality，
|
||||
而`employeeType`列通常cardinality很小(如 Manager, Developer, Tester)。
|
||||
|
||||
|
||||
|
||||
用例:
|
||||
|
||||
1. `SELECT id FROM employees WHERE site = 'lab';`
|
||||
|
||||
在这个语句中`site`的cardinality很小（没有很多不同的地点取值)。 因此，**Bitmap索引**比较适合。
|
||||
|
||||
2. `SELECT * FROM visited WHERE id = '34857' AND date < '2020-01-01';`
|
||||
|
||||
在这个语句中`id`有很高的cardinality （每一个ID是唯一的)。同时，表根据`date`已经分区。因此**Btree索引**比较适合。
|
||||
|
||||
### 运行语句
|
||||
3. `SELECT * FROM salaries WHERE salary > 50251.40;`
|
||||
|
||||
完成上面的操作后，再次在Hetu服务器运行这个语句，服务器将在后台自动加载索引。在此期间如果反复执行同一语句，应该观察到分段的数量不断降低，最后降至一个很小的值。
|
||||
在这个语句中`salary`有很高的cardinality（每个员工的收入总有些许不同)。假设表已经根据`salary`排序, 则**MinMax索引**最为适合。
|
||||
|
||||
4. `SELECT * FROM assets WHERE id = 50;`
|
||||
|
||||
在这个语句中`id`有很高的cardinality （每一个ID是唯一的)。但是，表没有分区。因此**Bloom索引**比较适合。
|
||||
|
||||
5. `SELECT * FROM phoneRecords WHERE phone='1234567890' and type = 'outgoing' and date > '2020-01-01';`
|
||||
|
||||
在这个语句中`phone`有很高的cardinality (即使有重复的电话，绝大部分号码总是不同的), `type`的cardinality较低 (只有两种：呼出/呼入),
|
||||
同时数据根据`date`已分区。因此，在`phone`上创建**Btree索引**并在`type`上创建**Bitmap索引**最为适合。
|
||||
|
||||
## 添加自定义的索引类型
|
||||
|
||||
参见 [Adding your own Index Type](./new-index.md).
|
||||
|
||||
## 权限控制
|
||||
|
||||
参见 [Built-in System Access Control](../security/built-in-system-access-control.md).
|
||||
|
|
|
|||
|
|
@ -180,4 +180,31 @@ security.refresh-period=1s
|
|||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 启发式索引控制规则
|
||||
|
||||
这些规则控制了允许的启发式索引操作。
|
||||
|
||||
每条规则由以下部分组成:
|
||||
|
||||
- `user` (必要): 匹配用户名称的正则表达式。默认值：`.*`.
|
||||
- `privileges` (可选): 授予用户的权限 (`ALL`, `SHOW`, `CREATE`, `DROP`, `RENAME`, and `UPDATE`). 默认值 `ALL`.
|
||||
|
||||
在下面这个例子中，用户`tom`只能执行`SHOW INDEX`或`CREATE INDEX`指令。
|
||||
用户`admin`可以执行`CREATE INDEX`, `SHOW INDEX`, `DROP INDEX`等所有指令.
|
||||
|
||||
```json
|
||||
{
|
||||
"indexAccess": [
|
||||
{
|
||||
"user": "tom",
|
||||
"privileges": ["SHOW", "CREATE"]
|
||||
},
|
||||
{
|
||||
"user": "admin",
|
||||
"privileges": ["ALL"]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
Loading…
Reference in New Issue