!639 Heuristic Index Features Documentation Edits
Merge pull request !639 from Daniel Zhang/hindex-splits-test
This commit is contained in:
Gitee
commit
9f24bbca7f
|
|
@ -559,7 +559,14 @@ Heuristic index is external index module that which can be used to filter to out
|
|||
> - **Default value:** `10s`
|
||||
>
|
||||
> The delay to wait before async loading task starts to load index cache from indexstore.
|
||||
|
||||
|
||||
### `hetu.heuristicindex.filter.cache.preload-indices`
|
||||
|
||||
> - **Type**：`string`
|
||||
> - **Default value：** ``
|
||||
>
|
||||
> Preload the specified indices (comma-separated) when the server starts. Put `ALL` to load all indices.
|
||||
|
||||
### `hetu.heuristicindex.indexstore.uri`
|
||||
|
||||
> - **Type:** `string`
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 35 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 50 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 37 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 37 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 47 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 38 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 33 KiB After Width: | Height: | Size: 32 KiB |
|
|
@ -1,25 +1,41 @@
|
|||
|
||||
# Bitmap Index
|
||||
# BitmapIndex
|
||||
|
||||
Bitmap Index utilizes Bitmaps. The size of the index increases as the number
|
||||
of unique values in the column increases. For example, a column like gender
|
||||
will have a small size. Whereas a column like ID will have an extremely
|
||||
BitmapIndex utilizes Bitmaps to allow for early row filtering which can help reduce CPU and memory usage.
|
||||
This can be beneficial in high concurrency queries.
|
||||
|
||||
BitmapIndex works well for columns with low cardinality (i.e. not many unique values) because
|
||||
the size of the index increases as the number
|
||||
of unique values in the column increases. For example, a column like `gender`
|
||||
will have a small size. Whereas a column like `id` will have an extremely
|
||||
large size (not recommended).
|
||||
|
||||
Note: Bitmap Index can additionally benefit when ORC predicate pushdown is enabled.
|
||||
A Bitmap is constructed for each unique column value to record the row numbers where the value can be found.
|
||||
A B+Tree is then used to store the mapping between the value and its Bitmap.
|
||||
By utilizing a B+Tree, BitmapIndex can support range queries with operators such as
|
||||
greater-than (`>`), less-than (`<`), `BETWEEN` and more.
|
||||
|
||||
**Note:** BitmapIndex can additionally benefit when ORC predicate pushdown is enabled.
|
||||
This can be enabled by setting `hive.orc-predicate-pushdown-enabled=true`
|
||||
in `hive.properties` or setting the session using `set session hive.orc_predicate_pushdown_enabled=true;`.
|
||||
Setting this to true will improve improve the performance of queries that utilize Bitmap Index.
|
||||
Setting this to true will improve the performance of queries that utilize BitmapIndex.
|
||||
See [Properties](../admin/properties.md) for details.
|
||||
|
||||
## Filtering
|
||||
## Use case(s)
|
||||
|
||||
1. Bitmap Index is used on workers for filtering rows when reading ORC files.
|
||||
**Note: Currently, Heuristic Index only supports the Hive connector with
|
||||
tables using ORC storage format.**
|
||||
|
||||
## Selecting column for Bitmap Index
|
||||
BitmapIndex is used on workers for filtering rows when reading ORC files.
|
||||
|
||||
Bitmap Index works on columns that have a low cardinality (i.e. few unique values),
|
||||
such as a Gender column.
|
||||
## Selecting column for BitmapIndex
|
||||
|
||||
Queries that are run in high concurrency and have a filter predicate on a column with
|
||||
low cardinality (i.e. not many unique values) can benefit from BitmapIndex.
|
||||
|
||||
For example, a query like `SELECT * FROM employees WHERE gender='M' AND type='FULLTIME' AND salary>10000`
|
||||
can benefit from having a BitmapIndex on both `gender` and `type` columns because
|
||||
data is being filtered on both columns and they both have low cardinality.
|
||||
|
||||
## Supported operators
|
||||
|
||||
|
|
@ -36,7 +52,7 @@ such as a Gender column.
|
|||
|
||||
## Examples
|
||||
|
||||
Creating index:
|
||||
**Creating index:**
|
||||
```sql
|
||||
create index idx using bitmap on hive.hindex.users (gender);
|
||||
create index idx using bitmap on hive.hindex.users (gender) where regionkey=1;
|
||||
|
|
@ -45,13 +61,64 @@ create index idx using bitmap on hive.hindex.users (gender) where regionkey in (
|
|||
|
||||
* assuming users table is partitioned on `regionkey`
|
||||
|
||||
Using index:
|
||||
**Using index:**
|
||||
```sql
|
||||
select name from hive.hindex.users where gender="female"
|
||||
select * from hive.hindex.users where id>123
|
||||
select * from hive.hindex.users where id<123
|
||||
select * from hive.hindex.users where id>=123
|
||||
select * from hive.hindex.users where id<=123
|
||||
select * from hive.hindex.users where id between (100, 200)
|
||||
select * from hive.hindex.users where id in (123, 199)
|
||||
```
|
||||
select * from hive.hindex.users where gender="female"
|
||||
select * from hive.hindex.users where age>20
|
||||
select * from hive.hindex.users where age<25
|
||||
select * from hive.hindex.users where age>=21
|
||||
select * from hive.hindex.users where age<=24
|
||||
select * from hive.hindex.users where age between (20, 25)
|
||||
select * from hive.hindex.users where age in (22, 23)
|
||||
```
|
||||
|
||||
## How BitmapIndex is created
|
||||
|
||||
1. BitmapIndex is created for each Stripe in an ORC file and allows us to know which rows contain a value.
|
||||
2. Data is inserted as an ordered list, the order in which the data appears in the Stripe.
|
||||
For the example below, data for `/hive/database.db/animals/000.orc stripe 1` would be inserted as follows:
|
||||
`["Ant", "Crab", "Bat", "Whale", "Ant", "Monkey"]`
|
||||
Additional information such as last modified time is stored as metadata to ensure a stale index is not used.
|
||||
3. When data insertion is finished, a Bitmap is created for each unique value. This is a compact way of tracking which rows the value is present in. (see Table)
|
||||
4. Once Bitmaps are created for the unique values. The value and the corresponding Bitmap is compressed and stored in a B+Tree to allow for quick lookup in `O(log(n))`.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## How BitmapIndex is used for Row Filtering
|
||||
|
||||
For filter queries like `SELECT * FROM animals WHERE type=LAND` normally all data needs to be read into memory and filtering will be applied to only return rows matching the predicate.
|
||||
|
||||
For example, for `/hive/database.db/animals/000.orc stripe 1` the following data will be read into memory:
|
||||
```
|
||||
Ant, LAND
|
||||
Crab, WATER
|
||||
Bat, AERIAL
|
||||
Whale, WATER
|
||||
Ant, LAND
|
||||
Monkey, LAND
|
||||
```
|
||||
Then, filtering would be applied to remove rows not matching the predicate:
|
||||
```
|
||||
Ant, LAND
|
||||
Ant, LAND
|
||||
Monkey, LAND
|
||||
```
|
||||
By using the BitmapIndex, we can improve this process.
|
||||
Instead of reading all the rows in the Stripe.
|
||||
BitmapIndex can return a list of matching rows which should be read.
|
||||
This can reduce both memory consumption and improve query execution time.
|
||||
|
||||
If we create a BitmapIndex on the `type` column, before data is read from the Stripe,
|
||||
the BitmapIndex for the Stripe will be queried for `"LAND"` and will return an iterator with values:
|
||||
`[1, 5, 6]`
|
||||
|
||||
These correspond to the row numbers which match the value
|
||||
(i.e. only these rows should be read into memory), the rest can be skipped.
|
||||
|
||||
For queries with multiple values like `SELECT * FROM animals WHERE type=LAND or type=AERIAL;`,
|
||||
BitmapIndex will perform two lookups. A union will be performed on the two Bitmaps to get the final result
|
||||
(i.e. `[001000] UNION [100011] = [101011]`), therefore the returned iterator will be `[1, 3, 5, 6]`.
|
||||
|
|
|
|||
|
|
@ -1,17 +1,30 @@
|
|||
|
||||
# Bloom Index
|
||||
# BloomIndex
|
||||
|
||||
Bloom Index utilizes Bloom Filters and index size will be fairly small.
|
||||
BloomIndex utilizes Bloom Filters to allow for filtering during scheduling and while reading data.
|
||||
|
||||
## Filtering
|
||||
BloomIndex works well for columns with high cardinality and index size is fairly small.
|
||||
|
||||
1. Bloom Index is used on coordinator for filtering splits during scheduling
|
||||
2. Bloom Index is used on workers for filtering Stripes when reading ORC files
|
||||
A Bloom Filter is constructed using column values. Then during lookup, the Bloom Filter can tell us if a given value is not in the Bloom Filter.
|
||||
|
||||
## Selecting column for Bloom Index
|
||||
BloomIndex can only support equality expression, e.g. `name='Monkey'`.
|
||||
|
||||
Bloom Index works on columns that have high cardinality (i.e. unique values),
|
||||
such as an ID column.
|
||||
## Use case(s)
|
||||
|
||||
**Note: Currently, Heuristic Index is only supports the Hive connector with
|
||||
tables using ORC storage format.**
|
||||
|
||||
1. BloomIndex is used on coordinator for filtering splits during scheduling
|
||||
2. BloomIndex is used on workers for filtering Stripes when reading ORC files
|
||||
|
||||
## Selecting column for BloomIndex
|
||||
|
||||
Queries that have a filter predicate on a column with
|
||||
high cardinality (i.e. many unique values) can benefit from BloomIndex.
|
||||
|
||||
For example, a query like `SELECT name FROM users WHERE phone=123456789`
|
||||
can benefit from having a BloomIndex on the `phone` column because
|
||||
data is being filtered on the column and `phone` column has a high cardinality.
|
||||
|
||||
## Supported operators
|
||||
|
||||
|
|
@ -35,7 +48,7 @@ such as an ID column.
|
|||
|
||||
## Examples
|
||||
|
||||
Creating index:
|
||||
**Creating index:**
|
||||
```sql
|
||||
create index idx using bloom on hive.hindex.users (id);
|
||||
create index idx using bloom on hive.hindex.users (id) where regionkey=1;
|
||||
|
|
@ -45,7 +58,42 @@ create index idx using bloom on hive.hindex.users (id) WITH ("bloom.fpp" = '0.00
|
|||
|
||||
* assuming users table is partitioned on `regionkey`
|
||||
|
||||
Using index:
|
||||
**Using index:**
|
||||
```sql
|
||||
select name from hive.hindex.users where id=123
|
||||
```
|
||||
```
|
||||
|
||||
## How BloomIndex is created
|
||||
|
||||
1. BloomIndex is created for each Stripe and allows us to know if the Stripe does not contain a given value.
|
||||
2. Data is inserted as a list, the order is not important and duplicates are acceptable.
|
||||
For the example below, data for `/hive/database.db/animals/000.orc stripe 1` would be inserted as follows:
|
||||
`["Ant", "Crab", "Bat", "Whale", "Ant", "Monkey"]`
|
||||
Additional information such as last modified time is stored as metadata to ensure a stale index is not used.
|
||||
3. When data insertion is finished, the BloomIndex can be serialized into the index store.
|
||||
|
||||
|
||||
|
||||
|
||||
## How BloomIndex is used for Split Filtering
|
||||
|
||||
When OLK engine needs to read data it schedules Splits.
|
||||
Each Split is responsible for reading a portion of the data.
|
||||
For example, when reading a Hive table with ORC data format,
|
||||
each Split will be responsible for reading a portion of the ORC file between the specified offsets.
|
||||
|
||||
E.g. `/hive/database.db/animals/000.orc`, starting offset `0`, ending offset `2000`.
|
||||
|
||||
For simplicity, we can assume each Split corresponds to a Stripe.
|
||||
|
||||
For a point query like `SELECT * FROM animals WHERE name='Monkey';`
|
||||
all data would normally need to be read and filtering will be applied to only return rows matching the predicate.
|
||||
In the example, all four Stripes will be read although only one of them contains the value.
|
||||
|
||||
By using the BloomIndex, only Stripes matching the predicate can be scheduled, therefore reducing the data that is read.
|
||||
This can significantly reduce the query execution time.
|
||||
|
||||
In this example, a lookup operation is performed on the BloomIndex for `Monkey`, which returns true for only the first Stripe.
|
||||
|
||||
Additionally, the last modified time is stored as part of the metadata and can be used to ensure that the index is still valid.
|
||||
If the original ORC file had been modified since the index was created, then the index is invalid and should not be used for filtering.
|
||||
|
|
|
|||
|
|
@ -1,24 +1,37 @@
|
|||
# BTree Index
|
||||
# BTreeIndex
|
||||
|
||||
BTree Index utilizes the B-Tree data structure.
|
||||
The size of the index increases as the number
|
||||
BTreeIndex utilizes the B+Tree data structure to allow for filtering during scheduling.
|
||||
|
||||
BTreeIndex is similar to BloomIndex and works well for columns with high cardinality.
|
||||
However, index size is can be large because the size of the index increases as the number
|
||||
of unique values in the column increases.
|
||||
|
||||
## Filtering
|
||||
A B+Tree is constructed using the unique column values as keys and the values are where the column value can be found.
|
||||
In order to reduce the index size, a dictionary is created for the values so large duplicate entries do not need to be stored multiple times.
|
||||
|
||||
1. BTree Index is used on coordinator for filtering splits during scheduling
|
||||
Unlike BloomIndex, BTreeIndex can also support range queries with operators such as
|
||||
greater-than (`>`), less-than (`<`), `BETWEEN` and more.
|
||||
|
||||
## Selecting column for BTree Index
|
||||
## Use case(s)
|
||||
|
||||
BTree Index works on columns that have high cardinality (i.e. unique values),
|
||||
such as an ID column, additionally it requires that the table be partitioned,
|
||||
e.g. by date.
|
||||
**Note: Currently, Heuristic Index is only supports the Hive connector with
|
||||
tables using ORC storage format.**
|
||||
|
||||
When selecting between BTree Index, the following should be considered:
|
||||
- Bloom index only supports `=`
|
||||
- Btree index requires the table to be partitioned
|
||||
- Bloom index is probabilistic, whereas Btree index is deterministic. This means Btree will perform better filtering.
|
||||
- Btree index size will be larger than Bloom index
|
||||
BTreeIndex is used on a coordinator for filtering splits during scheduling.
|
||||
|
||||
## Selecting column for BTreeIndex
|
||||
|
||||
Queries that have a filter predicate on a column with
|
||||
high cardinality (i.e. many unique values) can benefit from BTreeIndex.
|
||||
|
||||
For example, a query like `SELECT name FROM users WHERE phone>123456789`
|
||||
can benefit from having a BTreeIndex on the `phone` column because
|
||||
data is being filtered on the column and `phone` column has a high cardinality.
|
||||
|
||||
When selecting between BTreeIndex and BloomIndex, the following should be considered:
|
||||
- BloomIndex only supports `=`, whereas BTreeIndex supports range queries
|
||||
- BloomIndex is probabilistic, whereas BTreeIndex is deterministic. This means BTreeIndex will perform better filtering.
|
||||
- BTreeIndex size will be much larger than BloomIndex
|
||||
|
||||
## Supported operators
|
||||
|
||||
|
|
@ -35,15 +48,17 @@ When selecting between BTree Index, the following should be considered:
|
|||
|
||||
## Examples
|
||||
|
||||
Creating index:
|
||||
**Creating index:**
|
||||
|
||||
```sql
|
||||
create index idx using btree on hive.hindex.orders (orderid) with (level=partition) where orderDate='01-10-2020' ;
|
||||
create index idx using btree on hive.hindex.orders (orderid) with (level=table)';
|
||||
create index idx using btree on hive.hindex.orders (orderid) with (level=partition) where orderDate='01-10-2020';
|
||||
create index idx using btree on hive.hindex.orders (orderid) with (level=partition) where orderDate in ('01-10-2020', '01-10-2020');
|
||||
```
|
||||
|
||||
* assuming orders table is partitioned on `orderDate`; table must be partitioned
|
||||
* assuming orders table is partitioned on `orderDate`
|
||||
|
||||
Using index:
|
||||
**Using index:**
|
||||
```sql
|
||||
select * from hive.hindex.orders where orderid=12345
|
||||
select * from hive.hindex.orders where orderid>12345
|
||||
|
|
@ -52,4 +67,49 @@ select * from hive.hindex.orders where orderid>=12345
|
|||
select * from hive.hindex.orders where orderid<=12345
|
||||
select * from hive.hindex.orders where orderid between (10000, 20000)
|
||||
select * from hive.hindex.orders where orderid in (12345, 7890)
|
||||
```
|
||||
```
|
||||
|
||||
## How BTreeIndex is created
|
||||
|
||||
1. BTreeIndex is created at the Table level (or Partition level if table is partitioned).
|
||||
2. Data is inserted as `<Key,Value>` pairs. The `Keys` are the column values and the `Values` are the Stripes containing the column value.
|
||||
For the example below, data would be inserted as follows:
|
||||
```
|
||||
<"Ant", "/hive/database.db/animals/000.orc+3+1023+12345">
|
||||
<"Ant", "/hive/database.db/animals/000.orc+1024+2044+12345">
|
||||
<"Ant", "/hive/database.db/animals/001.orc+3+1023+12348">
|
||||
<"Crab", "/hive/database.db/animals/000.orc+3+1023+12345">
|
||||
...
|
||||
```
|
||||
Additional information about the stripe is included in the Value to help with filtering and to ensure a stale index is not used.
|
||||
3. Since storing the long value `"/hive/database.db/animals/000.orc+3+1023+12345"` multiple times would take up too much space, a dictionary is used. This dictionary maps the values to an integer. Instead of storing the long string value, the integer is stored in the B+Tree.
|
||||
4. As more data is inserted, the B+Tree is reblanced to ensure the height of the tree doesn't grow too much and lookup remains `O(log(n))`.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## How BTreeIndex is used for Split Filtering
|
||||
|
||||
When OLK engine needs to read data it schedules Splits. Each Split is responsible for reading a portion of the data.
|
||||
For example, when reading a Hive table with ORC data format, each Split will be responsible for reading a portion of the ORC file between the specified offsets.
|
||||
|
||||
E.g. `/hive/database.db/animals/000.orc`, starting offset `0`, ending offset `2000`.
|
||||
|
||||
For a point query like `SELECT * FROM animals WHERE name=Ant;`
|
||||
all data would normally need to be read and filtering will be applied to only return rows matching the predicate.
|
||||
|
||||
By using the BTreeIndex, only Splits matching the predicate can be scheduled, therefore reducing the data that is read.
|
||||
This can significantly reduce the query execution time.
|
||||
|
||||
In this example, a lookup operation is performed on the BTreeIndex for `Ant`, which returns an iterator with values:
|
||||
```
|
||||
"/hive/database.db/animals/000.orc+3+1023+12345"
|
||||
"/hive/database.db/animals/000.orc+1024+2044+12345"
|
||||
"/hive/database.db/animals/001.orc+3+1023+12348"
|
||||
```
|
||||
The file name and offsets can be used to filter out Splits which do not match the predicate.
|
||||
|
||||
Additionally, the last modified time can be used to ensure that the index is still valid.
|
||||
If the original ORC file had been modified since the index was created,
|
||||
then the index is invalid and should not be used for filtering.
|
||||
|
|
|
|||
|
|
@ -1,19 +1,26 @@
|
|||
|
||||
# MinMax Index
|
||||
# MinMaxIndex
|
||||
|
||||
MinMax simply keeps tracks of the largest and smallest value.
|
||||
MinMaxIndex simply keeps tracks of the largest and smallest value.
|
||||
The size of the index is extremely small.
|
||||
However, this index will only be useful if the table is sorted
|
||||
on the indexed column.
|
||||
|
||||
## Filtering
|
||||
## Use case(s)
|
||||
|
||||
1. MinMax Index is used on coordinator for filtering splits during scheduling
|
||||
**Note: Currently, Heuristic Index only supports the Hive connector with
|
||||
tables using ORC storage format.**
|
||||
|
||||
## Selecting column for MinMax Index
|
||||
MinMaxIndex is used on coordinator for filtering splits during scheduling.
|
||||
|
||||
MinMax Index will only work well on columns on which the table is sorted.
|
||||
For example, ID or age.
|
||||
## Selecting column for MinMaxIndex
|
||||
|
||||
Queries that have a filter predicate on a column on which data is sorted
|
||||
can benefit from MinMaxIndex.
|
||||
|
||||
For example, a query like `SELECT name FROM users WHERE age>25`
|
||||
can benefit from having a MinMaxIndex on the `age` column if
|
||||
the data is sorted on `age` column.
|
||||
|
||||
## Supported operators
|
||||
|
||||
|
|
|
|||
|
|
@ -12,15 +12,18 @@ The Heuristic Indexer allows creating indexes on existing data but stores the in
|
|||
- New index types not supported by the underlying data source can be created
|
||||
- Index data does not use the storage space of the data source
|
||||
|
||||
## Use case(s)
|
||||
|
||||
## Use cases
|
||||
|
||||
**Currently, Heuristic Index is only supports the Hive connector with
|
||||
**Note: Currently, Heuristic Index only supports the Hive connector with
|
||||
tables using ORC storage format.**
|
||||
|
||||
1. BloomIndex, MinMaxIndex and BtreeIndex can be used on a coordinator for filtering splits during scheduling
|
||||
2. When reading ORC files, to filter Stripes, MinMaxIndex can be used or BloomIndex can be used on workers
|
||||
2. BitmapIndex can used on workers for filtering rows when reading ORC files
|
||||
|
||||
### 1. Filtering scheduled Splits during query execution
|
||||
|
||||
*Index types supported: Bloom Index, Btree Index, MinMax Index*
|
||||
*Index types supported: BloomIndex, BtreeIndex, MinMaxIndex*
|
||||
|
||||
When the engine needs to read data from a data source it schedules Splits.
|
||||
However, not all Splits will return data if a predicate is applied.
|
||||
|
|
@ -33,7 +36,7 @@ By keeping an external index for the predicate column, the Heuristic Indexer can
|
|||
|
||||
### 2. Filtering Stripes when reading ORC files
|
||||
|
||||
*Index types supported: Bloom Index, MinMax Index*
|
||||
*Index types supported: BloomIndex, MinMaxIndex*
|
||||
|
||||
Similar to Split filtering above, when using the Hive connector to read ORC tables,
|
||||
Stripes can be filtered out based on the specified predicate. This reduces the amount
|
||||
|
|
@ -41,13 +44,13 @@ of data read and improves query performance.
|
|||
|
||||
### 3. Filtering rows when reading ORC files
|
||||
|
||||
*Index types supported: Bitmap Index*
|
||||
*Index types supported: BitmapIndex*
|
||||
|
||||
Going one level lower, once the rows are read, they must be filtered if a predicate is present.
|
||||
This involves reading rows and then using the Filter operator to discard
|
||||
rows that do not match the predicate.
|
||||
|
||||
By creating a Bitmap Index for the predicate column, the Heuristic Indexer will only read
|
||||
By creating a BitmapIndex for the predicate column, the Heuristic Indexer will only read
|
||||
rows which match the predicate, before the Filter operator is even applied. This can reduce
|
||||
memory and cpu usage and result in improved query performance, especially at higher concurrency.
|
||||
|
||||
|
|
@ -69,7 +72,7 @@ In `etc/config.properties`, add these lines:
|
|||
Path whitelist：`["/tmp", "/opt/hetu", "/opt/openlookeng", "/etc/hetu", "/etc/openlookeng", current workspace]`
|
||||
|
||||
**Note**：
|
||||
- `LOCAL` filesystem type is NOT supported anymore.
|
||||
- `LOCAL` filesystem type is NOT supported.
|
||||
- `HDFS` filesystem type should be used in production in order for the index to be accessible by all nodes in the cluster.
|
||||
- All nodes should be configured to use the same filesystem profile.
|
||||
- Heuristic Index can be disabled while the engine is running by setting: `set session heuristicindex_filter_enabled=false;`
|
||||
|
|
@ -122,7 +125,7 @@ Subsequent queries will utilize the index to reduce the amount of data read
|
|||
| hetu.heuristicindex.filter.cache.loading-delay | 10s | No | The delay to wait before async loading task starts to load index cache from indexstore|
|
||||
| hetu.heuristicindex.indexstore.uri | /opt/hetu/indices/ | No | Directory under which all index files are stored|
|
||||
| hetu.heuristicindex.indexstore.filesystem.profile | local-config-default| No | This property defines the filesystem profile used to read and write index|
|
||||
| hetu.heuristicindex.filter.cache.preload.indices | | No | Preload the specified indices (comma-separated) when the server starts. Put `ALL` to load all indices|
|
||||
| hetu.heuristicindex.filter.cache.preload-indices | | No | Preload the specified indices (comma-separated) when the server starts. Put `ALL` to load all indices|
|
||||
|
||||
Heuristic indexer now uses Hetu Metastore to manage its metadata. Please check [Hetu Metastore](../admin/meta-store.md) for more information.
|
||||
|
||||
|
|
@ -134,12 +137,12 @@ See [Heuristic Index Statements](./hindex-statements.md).
|
|||
|
||||
## Supported Index Types
|
||||
|
||||
| Index ID | Filtering type | Best Column type | Supported query operators | Notes | Example |
|
||||
|----------|-----------------|--------------------------------------------|---------------------------------------|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| [Bloom](./bloom.md) | Split<br>Stripe | High cardinality<br>(such as an ID column) | `=` `IN` | | `create index idx using bloom on hive.hindex.users (id);`<br>`select name from hive.hindex.users where id=123` |
|
||||
| [Btree](./btree.md) | Split | High cardinality<br>(such as an ID column) | `=` `>` `>=` `<` `<=` `IN` `BETWEEN` | Table must be partitioned | `create index idx using btree on hive.hindex.users (id) where regionkey IN (1,4) with ("level"='partition')`<br>(assuming table is partitioned on regionkey)<br>`select name from hive.hindex.users where id>123` |
|
||||
| [MinMax](./minmax.md) | Split<br>Stripe | Column which table is sorted on | `=` `>` `>=` `<` `<=` | | `create index idx using bloom on hive.hindex.users (age);`<br>(assuming users is sorted by age)<br>`select name from hive.hindex.users where age>25` |
|
||||
| [Bitmap](./bitmap.md) | Row | Low cardinality<br>(such as Gender column) | `=` `IN` | | `create index idx using bitmap on hive.hindex.users (gender);`<br>`select name from hive.hindex.users where gender='female'` |
|
||||
| Index ID | Filtering type | Best Column type | Supported query operators | Example |
|
||||
|----------|-----------------|--------------------------------------------|---------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| [Bloom](./bloom.md) | Split<br>Stripe | High cardinality<br>(such as an ID column) | `=` `IN` | `create index idx using bloom on hive.hindex.users (id);`<br>`select name from hive.hindex.users where id=123` |
|
||||
| [Btree](./btree.md) | Split | High cardinality<br>(such as an ID column) | `=` `>` `>=` `<` `<=` `IN` `BETWEEN` | `create index idx using btree on hive.hindex.users (id) where regionkey IN (1,4)`<br>`select name from hive.hindex.users where id>123` |
|
||||
| [MinMax](./minmax.md) | Split<br>Stripe | Column which table is sorted on | `=` `>` `>=` `<` `<=` | `create index idx using bloom on hive.hindex.users (age);`<br>(assuming users is sorted by age)<br>`select name from hive.hindex.users where age>25` |
|
||||
| [Bitmap](./bitmap.md) | Row | Low cardinality<br>(such as Gender column) | `=` `>` `>=` `<` `<=` `IN` `BETWEEN` | `create index idx using bitmap on hive.hindex.users (gender);`<br>`select name from hive.hindex.users where gender='female'` |
|
||||
|
||||
**Note:** unsupported operators will still function correctly but will not benefit from the index.
|
||||
|
||||
|
|
@ -162,30 +165,30 @@ Example queries:
|
|||
|
||||
1. `SELECT id FROM employees WHERE site = 'lab';`
|
||||
|
||||
In this query `site` has a low cardinality (i.e. not many sites) so **Bitmap Index** will help.
|
||||
In this query `site` has a low cardinality (i.e. not many sites) so **BitmapIndex** will help.
|
||||
|
||||
2. `SELECT * FROM visited WHERE id = '34857' AND date < '2020-01-01';`
|
||||
|
||||
In this query `id` has a high cardinality (i.e. IDs are likely unique)
|
||||
and table is partitioned on `date` so **Btree Index** will help.
|
||||
In this query `id` has a high cardinality (i.e. IDs are likely unique).
|
||||
**BloomIndex** or **BtreeIndex** will help.
|
||||
|
||||
3. `SELECT * FROM salaries WHERE salary > 50251.40;`
|
||||
|
||||
In this query `salary` has a high cardinality (i.e. salary of employees
|
||||
will slightly vary) and assuming `salaries` table is sorted on `salary`,
|
||||
**MinMax Index** will help.
|
||||
**MinMaxIndex** will help.
|
||||
|
||||
4. `SELECT * FROM assets WHERE id = 50;`
|
||||
|
||||
In this query `id` has a high cardinality (i.e. IDs are likely unique)
|
||||
but the table is not partitioned, so **Bloom Index** will help.
|
||||
In this query `id` has a high cardinality (i.e. IDs are likely unique).
|
||||
**BloomIndex** or **BtreeIndex** will help.
|
||||
|
||||
5. `SELECT * FROM phoneRecords WHERE phone='1234567890' and type = 'outgoing' and date > '2020-01-01';`
|
||||
|
||||
In this query `phone` has a high cardinality (i.e. there are many phone numbers, even if they
|
||||
made multiple calls), `type` has low cardinality (only outgoing or incoming),
|
||||
and the data is partitioned on date. Creating a **Btree Index** on `phone`
|
||||
and a **Bitmap Index** on `type` will help.
|
||||
and the data is partitioned on date. Creating a **BloomIndex** or **BtreeIndex** on `phone`
|
||||
and a **BitmapIndex** on `type` will help.
|
||||
|
||||
## Adding your own Index Type
|
||||
|
||||
|
|
|
|||
|
|
@ -490,6 +490,13 @@
|
|||
>
|
||||
> 在异步加载索引到缓存前等待的时长。
|
||||
|
||||
### `hetu.heuristicindex.filter.cache.preload-indices`
|
||||
|
||||
> - 类型：`string`
|
||||
> - **默认值：** ``
|
||||
>
|
||||
> 在服务器启动时预加载指定名称的索引(用逗号分隔), 当值为`ALL`时将预载入全部索引。
|
||||
|
||||
### `hetu.heuristicindex.indexstore.uri`
|
||||
|
||||
> - 类型：`string`
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 35 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 50 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 37 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 37 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 47 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 38 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 33 KiB After Width: | Height: | Size: 32 KiB |
|
|
@ -1,20 +1,37 @@
|
|||
|
||||
# BitMap（位图）索引
|
||||
# BitmapIndex（位图索引）
|
||||
|
||||
BitMap索引使用位图。索引的大小随着索引列中不同值的个数而增加。例如，一个标记性别的列很小，而一个ID列的索引则会极大（不推荐）。
|
||||
BitmapIndex使用位图来进行早期行过滤，这可以帮助减少CPU和内存使用量。
|
||||
这在高并发queries中是有益的。
|
||||
|
||||
注意：在ORC算子下推启用时，BitMap索引效果更好。可以通过设置`hive.properties`中的`hive.orc-predicate-pushdown-enabled=true`来启用，
|
||||
BitmapIndex对于低基数（即独特数据不多的）的列效果很好，
|
||||
因为index的大小随着独特数量的增加而增加。
|
||||
例如，`gender`之类的列将具有较小的尺寸。
|
||||
而像`id`这样的列将具有一个极高的大小（不推荐）。
|
||||
|
||||
Bitmap是为每个独特列值而构造一个位图，可以用来记录并且在其中找到该值的行号。
|
||||
然后，B+Tree会被用来存储值与其位图之间的映射。
|
||||
通过使用B+Tree，BitmapIndex可以支持使用运算符之类的范围query，例如
|
||||
大于（`>`），小于（`<`），`BETWEEN`等。
|
||||
|
||||
**注意：** 在ORC算子下推启用时，BitmapIndex效果更好。可以通过设置`hive.properties`中的`hive.orc-predicate-pushdown-enabled=true`来启用，
|
||||
或者在命令行中启用`set session hive.orc_predicate_pushdown_enabled=true;`。
|
||||
|
||||
参见[Properties](../admin/properties.md)获得更多信息。
|
||||
|
||||
## 过滤
|
||||
## 使用场景
|
||||
|
||||
1. BitMap索引用于过滤从ORC文件中读取的数据，且仅供worker节点使用。
|
||||
**注意：当前，启发式索引仅支持ORC存储格式的Hive数据源。**
|
||||
|
||||
BitmapIndex用于过滤从ORC文件中读取的数据，且仅供worker节点使用。
|
||||
|
||||
## 选择适用的列
|
||||
|
||||
BitMap索引在拥有较少不同值数量的列上比较适用，例如：性别。
|
||||
以高并发率运行的queries，并且在具有低基数（独特值不多的）条件的列上具有过滤predicates
|
||||
可以从BitmapIndex中得到好的效果。
|
||||
|
||||
例如，类似`SELECT * FROM Employees WHERE gender='M' AND type='FULLTIME' AND salary>10000`的query
|
||||
可以在`gender`和`type`列上用BitmapIndex并且得到好的效果，因为数据在两列上都被过滤，并且两者的基数都很低。
|
||||
|
||||
## 支持的运算符
|
||||
|
||||
|
|
@ -31,7 +48,7 @@ BitMap索引在拥有较少不同值数量的列上比较适用，例如：性
|
|||
|
||||
## 用例
|
||||
|
||||
创建：
|
||||
**创建：**
|
||||
```sql
|
||||
create index idx using bitmap on hive.hindex.users (gender);
|
||||
create index idx using bitmap on hive.hindex.users (gender) where regionkey=1;
|
||||
|
|
@ -40,13 +57,61 @@ create index idx using bitmap on hive.hindex.users (gender) where regionkey in (
|
|||
|
||||
* 假设表已按照`regionkey`列分区
|
||||
|
||||
使用:
|
||||
**使用:**
|
||||
```sql
|
||||
select name from hive.hindex.users where gender="female"
|
||||
select * from hive.hindex.users where id>123
|
||||
select * from hive.hindex.users where id<123
|
||||
select * from hive.hindex.users where id>=123
|
||||
select * from hive.hindex.users where id<=123
|
||||
select * from hive.hindex.users where id between (100, 200)
|
||||
select * from hive.hindex.users where id in (123, 199)
|
||||
```
|
||||
select * from hive.hindex.users where gender="female"
|
||||
select * from hive.hindex.users where age>20
|
||||
select * from hive.hindex.users where age<25
|
||||
select * from hive.hindex.users where age>=21
|
||||
select * from hive.hindex.users where age<=24
|
||||
select * from hive.hindex.users where age between (20, 25)
|
||||
select * from hive.hindex.users where age in (22, 23)
|
||||
```
|
||||
|
||||
## 如何创建BitmapIndex
|
||||
|
||||
1. BitmapIndex是为每一个在ORC文件中的Stripe创建的，并使我们知道哪些行包含值。
|
||||
2. 数据作为有序列表插入，数据顺序是根据在Stripe中的出现顺序。
|
||||
对于以下示例，`/hive/database.db/animals/000.orc stripe 1`的数据将如下插入：
|
||||
`["Ant", "Crab", "Bat", "Whale", "Ant", "Monkey"]`
|
||||
诸如上次修改时间之类的其他信息将作为元数据存储，以确保不使用陈旧索引。
|
||||
3. 数据插入完成后，将为每个独特值创建一个Bitmap。这是一种跟踪值存在的行的紧凑方式。（请参见表）
|
||||
4. 一旦为独特值创建了Bitmap。该值和相应的Bitmap被压缩并存储在B+Tree中，以允许在`O(log(n))`之内的运行速度来快速查找。
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## 如何将BitmapIndex用于行过滤
|
||||
|
||||
对于诸如`SELECT * FROM Animal WHERE type = LAND`之类的过滤器queries，通常，所有数据都需要读入内存，并且过滤将仅应用于与predicates匹配的返回行。
|
||||
|
||||
例如，对于`/hive/database.db/animals/000.orc stripe 1`，以下数据将被读入内存：
|
||||
```
|
||||
Ant, LAND
|
||||
Crab, WATER
|
||||
Bat, AERIAL
|
||||
Whale, WATER
|
||||
Ant, LAND
|
||||
Monkey, LAND
|
||||
```
|
||||
然后，将应用过滤以删除与predicate不匹配的行：
|
||||
```
|
||||
Ant, LAND
|
||||
Ant, LAND
|
||||
Monkey, LAND
|
||||
```
|
||||
通过使用BitmapIndex，我们可以改进此过程。而不是读取Stripe中的所有行。
|
||||
BitmapIndex可以返回应读取的匹配行的列表。这样既可以减少内存消耗，又可以缩短查询执行时间。
|
||||
|
||||
如果我们在`type`列上创建BitmapIndex，则在从Stripe读取数据之前，
|
||||
将为Stripe的BitmapIndex查询`LAND`，并将返回具有以下值的迭代器：
|
||||
`[1, 5, 6]`
|
||||
|
||||
这些对应于与值匹配的行号（即仅应将这些行读入内存），其余的可以跳过。
|
||||
|
||||
对于具有多个值的queries，例如`SELECT * FROM animes WHERE type=LAND OR type=AERIAL;`，
|
||||
BitmapIndex将执行两次查找。将对两个Bitmaps执行联合以得到最终结果
|
||||
（例如，`[001000] UNION [100011] = [101011]`），因此返回的迭代器将为`[1、3、5、6]`。
|
||||
|
|
|
|||
|
|
@ -1,16 +1,27 @@
|
|||
|
||||
# Bloom索引
|
||||
# BloomIndex
|
||||
|
||||
Bloom索引实用布隆过滤器来过滤数据。索引体积非常小。
|
||||
BloomIndex使用Bloom Filters(布隆过滤器)来在计划期间和读取数据时进行过滤。
|
||||
|
||||
## 过滤
|
||||
BloomIndex对于具有高基数的列以及索引大小很小的列都适用。
|
||||
|
||||
1. Bloom索引用于调度时的分片过滤，被coordinator节点使用。
|
||||
2. Bloom索引也用于worker节点上，用于在读取ORC文件是过滤stripes。
|
||||
布隆过滤器是使用列值构造的。然后在查找过程中，布隆过滤器会告诉我们布隆过滤器中是否有给定值。
|
||||
|
||||
BloomIndex仅支持相等表达式，例如`name='monkey'`。
|
||||
|
||||
## 使用场景
|
||||
|
||||
**注意：当前，启发式索引仅支持ORC存储格式的Hive数据源。**
|
||||
|
||||
1. BloomIndex用于调度时的分片过滤，被coordinator节点使用。
|
||||
2. BloomIndex也用于worker节点上，用于在读取ORC文件是过滤stripes。
|
||||
|
||||
## 选择适用的列
|
||||
|
||||
位图索引在拥有较多不同值数量的列上比较适用，例如：ID。
|
||||
在具有高基数（即许多独特值）条件的列上具有过滤predicate的queries可以从BloomIndex中得到好的效果。
|
||||
|
||||
例如，类似`SELECT name FROM users WHERE phone=123456789`之类的query
|
||||
可以通过在`phone`列上使用BloomIndex而得到好的效果，因为列中的数据已被过滤，`phone`列的基数较高。
|
||||
|
||||
## 支持的运算符
|
||||
|
||||
|
|
@ -32,7 +43,7 @@ Bloom索引实用布隆过滤器来过滤数据。索引体积非常小。
|
|||
|
||||
## 用例
|
||||
|
||||
创建索引:
|
||||
**创建索引:**
|
||||
```sql
|
||||
create index idx using bloom on hive.hindex.users (id);
|
||||
create index idx using bloom on hive.hindex.users (id) where regionkey=1;
|
||||
|
|
@ -42,7 +53,41 @@ create index idx using bloom on hive.hindex.users (id) WITH ("bloom.fpp" = '0.00
|
|||
|
||||
* 假设表已按照`regionkey`列分区
|
||||
|
||||
使用:
|
||||
**使用:**
|
||||
```sql
|
||||
select name from hive.hindex.users where id=123
|
||||
```
|
||||
```
|
||||
|
||||
## 如何创建BloomIndex
|
||||
|
||||
1. BloomIndex是为每一个Stripe创建的，并让我们知道Stripe是否不包含给定值。
|
||||
2. 数据作为列表插入，顺序不重要，并且可以接受重复项。
|
||||
对于以下示例，将插入`/hive/database.db/animals/000.orc stripe 1`的数据，如下所示：
|
||||
`["Ant", "Crab", "Bat", "Whale", "Ant", "Monkey"]`
|
||||
诸如上次修改时间之类的其他信息将作为元数据存储，以确保不使用陈旧索引。
|
||||
3. 数据插入完成后，可以将BloomIndex序列化到索引存储中。
|
||||
|
||||
|
||||
|
||||
## 如何将BloomIndex用于分片过滤
|
||||
|
||||
当OLK引擎需要读取数据时，它会计划Splits（分片）。
|
||||
每个分片负责读取一部分数据。
|
||||
例如，当读取具有ORC数据格式的Hive表时，
|
||||
每个分割将负责读取指定偏移量之间的ORC文件的一部分。
|
||||
|
||||
例如，`/hive/database.db/animals/000.orc`，从偏移量`0`开始，从偏移量`2000`开始。
|
||||
|
||||
为简单起见，我们可以假定每个分片对应于一个Stripe。
|
||||
|
||||
对于类似`SELECT * FROM animals WHERE name='Monkey';`的点查询（point query）
|
||||
通常将需要读取所有数据，并且过滤将仅应用于与predicates匹配的返回行。
|
||||
在该示例中，将读取所有四个条带，尽管其中只有一个包含该值。
|
||||
|
||||
通过使用BloomIndex，只能调度与predicates匹配的Stripes，因此减少了读取的数据。
|
||||
这样可以大大减少查询的执行时间。
|
||||
|
||||
在此示例中，对`Monkey`的BloomIndex执行查找操作，该操作仅对第一个Stripe返回true。
|
||||
|
||||
此外，上次修改时间存储为元数据的一部分，可用于确保索引仍然有效。
|
||||
如果自创建索引以来已对原始ORC文件进行了修改，则该索引无效，因此不应将其用于过滤。
|
||||
|
|
|
|||
|
|
@ -1,20 +1,37 @@
|
|||
# BTree索引
|
||||
# BTreeIndex
|
||||
|
||||
BTree索引使用二叉树数据结构存储。索引的大小随着索引列中不同值的个数而增加。
|
||||
BTreeIndex使用二叉树数据结构存储。索引的大小随着索引列中不同值的个数而增加。
|
||||
|
||||
## 过滤
|
||||
BTreeIndex利用B+Tree数据结构来允许在调度期间进行过滤。
|
||||
|
||||
1. Bloom索引用于调度时的分片过滤，被coordinator节点使用。
|
||||
BTreeIndex与BloomIndex相似，并且对于具有高基数的列非常适用。
|
||||
但是，索引的大小可能会很大，因为索引的大小会随着数量的增加而增加
|
||||
列中独特值的百分比增加。
|
||||
|
||||
B+Tree的建立构造使用独特列值作为键，并且可以在其中找到该列值的值。
|
||||
为了减小索引的大小，将为这些值创建一部Dictionary，因此不需要将大型重复项存储多次。
|
||||
|
||||
与BloomIndex不同，BTreeIndex还可以使用以下运算符来支持范围查询
|
||||
大于（`>`），小于（`<`），`BETWEEN`等。
|
||||
|
||||
## 使用场景
|
||||
|
||||
**注意：当前，启发式索引仅支持ORC存储格式的Hive数据源。**
|
||||
|
||||
BTreeIndex用于调度时的分片(Split)过滤，被coordinator节点使用。
|
||||
|
||||
## 选择适用的列
|
||||
|
||||
位图索引在拥有较多不同值数量的列上比较适用，例如：ID。除此之外，BTree索引还要求表是分区的。
|
||||
在具有高基数（即许多独特值）条件的列上具有过滤predicate的queries可以从BTreeIndex中达到好的效果。
|
||||
|
||||
在BTree和Bloom索引之间选择时，需要考虑：
|
||||
- Bloom索引只支持`=`
|
||||
- Btree索引要求表是分区的
|
||||
- Bloom索引是不确定的，而BTree索引是确定的。因此BTree通常有更好的过滤性能
|
||||
- BTree索引比Bloom索引更大
|
||||
例如，类似`SELECT FROM FROM users WHERE phone>123456789`的query
|
||||
可以通过在`phone`列上使用BTreeIndex而达到好的效果，因为
|
||||
列中的数据已被过滤，`phone`列的基数较高。
|
||||
|
||||
在BTreeIndex和BloomIndex索引之间选择时，需要考虑：
|
||||
- BloomIndex只支持`=`，而BTreeIndex提供范围咨询
|
||||
- BloomIndex是不确定的，而BTreeIndex是确定的。因此BTreeIndex通常有更好的过滤性能
|
||||
- BTreeIndex比BloomIndex索引更大
|
||||
|
||||
## 支持的运算符
|
||||
|
||||
|
|
@ -31,15 +48,17 @@ BTree索引使用二叉树数据结构存储。索引的大小随着索引列中
|
|||
|
||||
## 用例
|
||||
|
||||
创建索引:
|
||||
**创建索引:**
|
||||
|
||||
```sql
|
||||
create index idx using btree on hive.hindex.orders (orderid) with (level=partition) where orderDate='01-10-2020' ;
|
||||
create index idx using btree on hive.hindex.orders (orderid) with (level=table)';
|
||||
create index idx using btree on hive.hindex.orders (orderid) with (level=partition) where orderDate='01-10-2020';
|
||||
create index idx using btree on hive.hindex.orders (orderid) with (level=partition) where orderDate in ('01-10-2020', '01-10-2020');
|
||||
```
|
||||
|
||||
* 假设表已按照`orderDate`列分区
|
||||
|
||||
使用索引:
|
||||
**使用索引:**
|
||||
```sql
|
||||
select * from hive.hindex.orders where orderid=12345
|
||||
select * from hive.hindex.orders where orderid>12345
|
||||
|
|
@ -48,4 +67,49 @@ select * from hive.hindex.orders where orderid>=12345
|
|||
select * from hive.hindex.orders where orderid<=12345
|
||||
select * from hive.hindex.orders where orderid between (10000, 20000)
|
||||
select * from hive.hindex.orders where orderid in (12345, 7890)
|
||||
```
|
||||
```
|
||||
|
||||
## 如何创建BTreeIndex
|
||||
|
||||
1. BTreeIndex在表级别（table level）（如果表已分区，则在分区级别）创建。
|
||||
2. 数据作为`<Key,Value>`对插入。`Keys`是列值，`Values`是包含列值的Stripes。
|
||||
对于下面的示例，数据将按以下方式插入：
|
||||
```
|
||||
<"Ant", "/hive/database.db/animals/000.orc+3+1023+12345">
|
||||
<"Ant", "/hive/database.db/animals/000.orc+1024+2044+12345">
|
||||
<"Ant", "/hive/database.db/animals/001.orc+3+1023+12348">
|
||||
<"Crab", "/hive/database.db/animals/000.orc+3+1023+12345">
|
||||
...
|
||||
```
|
||||
值中包含有关条带的其他信息，以帮助进行过滤并确保不使用陈旧的索引。
|
||||
3. 由于多次存储长值`"/hive/database.db/animals/000.orc+3+1023+12345"`会占用太多空间，因此使用了Dictionary。该Dictionary将值映射为整数。而不是存储长字符串值，而是将整数存储在B+Tree中。
|
||||
4. 随着更多数据的插入，B+Tree重新平衡以确保树的高度不会增加太多，并且查找运行时间保持为`O(log(n))`。
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## 如何将BTreeIndex用于分片过滤
|
||||
|
||||
当OLK引擎需要读取数据时，它会计划分片。每个分片负责读取一部分数据。
|
||||
例如，当读取具有ORC数据格式的Hive表时，每个Split将负责读取指定偏移量之间的ORC文件的一部分。
|
||||
|
||||
例如，`/hive/database.db/animals/000.orc`，起始偏移量`0`，结束偏移量`2000`。
|
||||
|
||||
对于诸如`SELECT * FROM animals WHERE name=Ant;`的点查询（point query），
|
||||
通常将需要读取所有数据，并且过滤将仅应用于与predicates匹配的返回行。
|
||||
|
||||
通过使用BTreeIndex，只会调度与predicates匹配的分片，因此减少了读取的数据。
|
||||
这样可以大大减少查询的执行时间。
|
||||
|
||||
在此示例中，对Ant的BTreeIndex执行查找操作，该操作返回具有以下值的迭代器：
|
||||
```
|
||||
“ /hive/database.db/animals/000.orc+3+1023+12345”
|
||||
“ /hive/database.db/animals/000.orc+1024+2044+12345”
|
||||
“ /hive/database.db/animals/001.orc+3+1023+12348”
|
||||
```
|
||||
文件名和偏移量可用于筛选出与predicates不匹配的分片。
|
||||
|
||||
此外，上次修改的时间可以用来确保索引仍然有效。
|
||||
如果自创建索引以来已修改原始ORC文件，
|
||||
则索引无效，不应将其用于过滤。
|
||||
|
|
|
|||
|
|
@ -1,16 +1,21 @@
|
|||
|
||||
# MinMax索引
|
||||
# MinMaxIndex
|
||||
|
||||
MinMax索引简单地记录数据的最大和最小值，占用空间极小。
|
||||
MinMaxIndex简单地记录数据的最大和最小值，占用空间极小。
|
||||
因此，这一索引仅仅能被用于已经排序的数据列。
|
||||
|
||||
## 过滤
|
||||
## 使用场景
|
||||
|
||||
1. MinMax索引用于调度时的分片过滤，被coordinator节点使用。
|
||||
**注意：当前，启发式索引仅支持ORC存储格式的Hive数据源。**
|
||||
|
||||
MinMaxIndex用于调度时的分片过滤，被coordinator节点使用。
|
||||
|
||||
## 选择适用的列
|
||||
|
||||
MinMax索引仅仅能被用于已经排序的数据列。例如，ID或年龄.
|
||||
在对数据进行排序的列上具有过滤predicate的query可以从MinMaxIndex中得到好的效果。
|
||||
|
||||
例如，如果一下数据是根据`age`来排序的，那么一个像`SELECT name from users WHERE age> 25`
|
||||
之类的query则可以因有效地在`age`上利用MinMaxIndex，而从中得到好的效果。
|
||||
|
||||
## 支持的运算符
|
||||
|
||||
|
|
|
|||
|
|
@ -14,33 +14,37 @@
|
|||
|
||||
## 使用场景
|
||||
|
||||
**当前，启发式索引支持ORC存储格式的hive数据源。**
|
||||
**注意：当前，启发式索引仅支持ORC存储格式的Hive数据源。**
|
||||
|
||||
1. BloomIndex，MinMaxIndex和BtreeIndex可以在Coordinator上使用，以在调度期间过滤Splits
|
||||
2. 在读取ORC文件时，为了过滤Splits，可以使用MinMaxIndex或者在workers上使用BloomIndex
|
||||
3. 在读取ORC文件时，BitmapIndex可以在workers上用于过滤数据行
|
||||
|
||||
### 1.查询过程中过滤预定分片
|
||||
|
||||
支持的索引：Bloom, BTree, MinMax
|
||||
*支持的索引：BloomIndex, BTreeIndex, MinMaxIndex*
|
||||
|
||||
当引擎需要调度一个TableScan操作时，它可以调度worker节点上的Split。这些Split负责读取部分源数据。但是如果应用了谓词，则并非所有Split都会返回数据。
|
||||
当引擎需要调度一个TableScan操作时，它可以调度worker节点上的Split。这些Split负责读取部分源数据。但是如果应用了predicates，则并非所有Split都会返回数据。
|
||||
|
||||
例如，`select * from test_base where j1='070299439'`
|
||||
|
||||
通过为谓词列保留外部索引，启发式索引可以确定每个Split是否包含正在搜索的值，并且只对可能包含该值的Split安排读操作。
|
||||
通过为predicate列保留外部索引，启发式索引可以确定每个Split是否包含正在搜索的值，并且只对可能包含该值的Split安排读操作。
|
||||
|
||||
|
||||
|
||||
### 2.读取ORC文件时提前筛选Stripes
|
||||
|
||||
支持的索引：Bloom, MinMax
|
||||
*支持的索引：BloomIndex, MinMaxIndex*
|
||||
|
||||
与分片过滤类似，当使用Hive Connector读取ORC文件时，Stripe可以被提前过滤来减少读取的数据量，从而提升查询性能。
|
||||
|
||||
### 3.读取ORC文件时筛选行
|
||||
|
||||
支持的索引：Bitmap
|
||||
*支持的索引：BitmapIndex*
|
||||
|
||||
当需要从ORC文件中读取数据时，如果有一个谓词存在，那么就不需要批量读取中的所有行。
|
||||
当需要从ORC文件中读取数据时，如果有一个predicate存在，那么就不需要批量读取中的所有行。
|
||||
|
||||
通过为谓词列保留外部位图索引，将实现只读取匹配当行，来提升内存和处理器表现。在服务器高并发时提升尤其明显。
|
||||
通过为predicate列保留外部位图索引，将实现只读取匹配当行，来提升内存和处理器表现。在服务器高并发时提升尤其明显。
|
||||
|
||||
## 示例教程
|
||||
|
||||
|
|
@ -60,7 +64,7 @@
|
|||
避免选择根目录；路径不能包含../；如果配置了node.data_dir,那么当前工作目录为node.data_dir的父目录；如果没有配置，那么当前工作目录为openlookeng server的目录
|
||||
|
||||
**注意**：
|
||||
- `LOCAL` 本地文件系统将*不再*被支持
|
||||
- `LOCAL` 本地文件系统是*不*被支持的。
|
||||
- `HDFS` 应用于生产环境来在集群中共享数据。
|
||||
- 所有节点必须有相同的文件系统配置。
|
||||
- 在服务器运行中可以通过`set session heuristicindex_filter_enabled=false;`关闭启发式索引。
|
||||
|
|
@ -80,7 +84,7 @@
|
|||
|
||||
SELECT * FROM table1 WHERE id="abcd1234";
|
||||
|
||||
如果id比较唯一，bloom索引可以大大较少读取的分段数量。
|
||||
如果id比较独特，bloom索引可以大大较少读取的分段数量。
|
||||
|
||||
在本教程中我们将以这个语句为例。
|
||||
|
||||
|
|
@ -106,7 +110,7 @@
|
|||
| hetu.heuristicindex.filter.cache.loading-delay | 10s | 否 | 在异步加载索引到缓存前等待的时长|
|
||||
| hetu.heuristicindex.indexstore.uri | /opt/hetu/indices/ | 否 | 所有索引文件存储的目录|
|
||||
| hetu.heuristicindex.indexstore.filesystem.profile | local-config-default| 否 | 用于存储索引文件的文件系统属性描述文件名称|
|
||||
| hetu.heuristicindex.filter.cache.preload.indices | | 否 | 在服务器启动时预加载指定名称的索引(用逗号分隔), 当值为`ALL`时将预载入全部索引|
|
||||
| hetu.heuristicindex.filter.cache.preload-indices | | 否 | 在服务器启动时预加载指定名称的索引(用逗号分隔), 当值为`ALL`时将预载入全部索引|
|
||||
|
||||
索引功能现使用Hetu Metastore管理元数据。请参阅 [Hetu Metastore](../admin/meta-store.md) 获取关于如何配置的更多信息。
|
||||
|
||||
|
|
@ -118,19 +122,19 @@
|
|||
|
||||
## 支持的索引类型
|
||||
|
||||
| 索引 ID | 过滤类型 | 最适用的列 | 支持的运算符 | 注释 | 用例 |
|
||||
|----------|-----------------|--------------------------------------------|---------------------------------------|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| [Bloom](./bloom.md) | Split<br>Stripe | 大量不同数据值<br>(如ID) | `=` `IN` | | `create index idx using bloom on hive.hindex.users (id);`<br>`select name from hive.hindex.users where id=123` |
|
||||
| [Btree](./btree.md) | Split | 大量不同数据值<br>(如ID) | `=` `>` `>=` `<` `<=` `IN` `BETWEEN` | 表必须被分区 | `create index idx using btree on hive.hindex.users (id) where regionkey IN (1,4) with ("level"='partition')`<br>(假设表根据regionkey分区)<br>`select name from hive.hindex.users where id>123` |
|
||||
| [MinMax](./minmax.md) | Split<br>Stripe | 列数据被排序 | `=` `>` `>=` `<` `<=` | | `create index idx using bloom on hive.hindex.users (age);`<br>(假设数据根据年龄已排序)<br>`select name from hive.hindex.users where age>25` |
|
||||
| [Bitmap](./bitmap.md) | Row | 少量不同数据值<br>(如性别) | `=` `IN` | | `create index idx using bitmap on hive.hindex.users (gender);`<br>`select name from hive.hindex.users where gender='female'` |
|
||||
| 索引 ID | 过滤类型 | 最适用的列 | 支持的运算符 | 用例 |
|
||||
|----------|-----------------|--------------------------------------------|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| [Bloom](./bloom.md) | Split<br>Stripe | 大量不同数据值<br>(如ID) | `=` `IN` | `create index idx using bloom on hive.hindex.users (id);`<br>`select name from hive.hindex.users where id=123` |
|
||||
| [Btree](./btree.md) | Split | 大量不同数据值<br>(如ID) | `=` `>` `>=` `<` `<=` `IN` `BETWEEN` | `create index idx using btree on hive.hindex.users (id) where regionkey IN (1,4)`<br>`select name from hive.hindex.users where id>123` |
|
||||
| [MinMax](./minmax.md) | Split<br>Stripe | 列数据被排序 | `=` `>` `>=` `<` `<=` | `create index idx using bloom on hive.hindex.users (age);`<br>(假设数据根据年龄已排序)<br>`select name from hive.hindex.users where age>25` |
|
||||
| [Bitmap](./bitmap.md) | Row | 少量不同数据值<br>(如性别) | `=` `>` `>=` `<` `<=` `IN` `BETWEEN` | `create index idx using bitmap on hive.hindex.users (gender);`<br>`select name from hive.hindex.users where gender='female'` |
|
||||
|
||||
**注意:** 包含不支持的运算符的语句依然会正常运行，但是不会从启发式索引中获得性能提升。
|
||||
|
||||
|
||||
## 选择索引类型
|
||||
|
||||
启发式索引用于根据谓词表达式过滤数据。请根据下面的决策流程图选择适用于数据列的最佳索引。
|
||||
启发式索引用于根据predicate表达式过滤数据。请根据下面的决策流程图选择适用于数据列的最佳索引。
|
||||
|
||||
Cardinality 是指数据集中值域的大小。例如，`ID`列通常有很大的cardinality，
|
||||
而`employeeType`列通常cardinality很小(如 Manager, Developer, Tester)。
|
||||
|
|
@ -141,24 +145,24 @@ Cardinality 是指数据集中值域的大小。例如，`ID`列通常有很大
|
|||
|
||||
1. `SELECT id FROM employees WHERE site = 'lab';`
|
||||
|
||||
在这个语句中`site`的cardinality很小（没有很多不同的地点取值)。 因此，**Bitmap索引**比较适合。
|
||||
在这个语句中`site`的cardinality很小（没有很多不同的地点取值)。 因此，**BitmapIndex**比较适合。
|
||||
|
||||
2. `SELECT * FROM visited WHERE id = '34857' AND date < '2020-01-01';`
|
||||
|
||||
在这个语句中`id`有很高的cardinality （每一个ID是唯一的)。同时，表根据`date`已经分区。因此**Btree索引**比较适合。
|
||||
在这个语句中`id`有很高的cardinality （每一个ID是唯一的)。**BloomIndex**或者**BtreeIndex**都比较适合。
|
||||
|
||||
3. `SELECT * FROM salaries WHERE salary > 50251.40;`
|
||||
|
||||
在这个语句中`salary`有很高的cardinality（每个员工的收入总有些许不同)。假设表已经根据`salary`排序, 则**MinMax索引**最为适合。
|
||||
在这个语句中`salary`有很高的cardinality（每个员工的收入总有些许不同)。假设表已经根据`salary`排序, 则**MinMaxIndex**最为适合。
|
||||
|
||||
4. `SELECT * FROM assets WHERE id = 50;`
|
||||
|
||||
在这个语句中`id`有很高的cardinality （每一个ID是唯一的)。但是，表没有分区。因此**Bloom索引**比较适合。
|
||||
在这个语句中`id`有很高的cardinality （每一个ID是唯一的)。**BloomIndex**或者**BtreeIndex**都比较适合。
|
||||
|
||||
5. `SELECT * FROM phoneRecords WHERE phone='1234567890' and type = 'outgoing' and date > '2020-01-01';`
|
||||
|
||||
在这个语句中`phone`有很高的cardinality (即使有重复的电话，绝大部分号码总是不同的), `type`的cardinality较低 (只有两种：呼出/呼入),
|
||||
同时数据根据`date`已分区。因此，在`phone`上创建**Btree索引**并在`type`上创建**Bitmap索引**最为适合。
|
||||
在这个语句中`phone`有很高的cardinality (即使有重复的电话，绝大部分号码总是不同的), `type`的cardinality较低 (只有两种：呼出/呼入)。
|
||||
在`phone`上创建**BloomIndex**或者**BtreeIndex**并在`type`上创建**BitmapIndex**比较适合。
|
||||
|
||||
## 添加自定义的索引类型
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue