From ec363bfb428dbd8feee9a6e414db188ce7dbe44a Mon Sep 17 00:00:00 2001 From: fhxing168 Date: Tue, 7 Jul 2020 09:55:19 +0800 Subject: [PATCH] Add the chinese documents of language, rest, optimizer and indexer. --- hetu-docs/en/indexer/bitmap.md | 5 + hetu-docs/en/indexer/bloom.md | 5 + hetu-docs/en/indexer/indexer-cli.md | 5 + hetu-docs/en/indexer/overview.md | 5 + hetu-docs/en/language/reserved.md | 5 + hetu-docs/en/language/timestamp.md | 7 +- hetu-docs/en/language/types.md | 10 +- .../en/optimizer/cost-based-optimizations.md | 13 +- hetu-docs/en/optimizer/cost-in-explain.md | 5 + hetu-docs/en/optimizer/statistics.md | 9 +- hetu-docs/en/rest/node.md | 91 ++-- hetu-docs/en/rest/query.md | 7 +- hetu-docs/en/rest/stage.md | 5 + hetu-docs/en/rest/statement.md | 82 +-- hetu-docs/en/rest/task.md | 11 +- hetu-docs/zh/indexer/bitmap.md | 23 +- hetu-docs/zh/indexer/bloom.md | 31 +- hetu-docs/zh/indexer/indexer-cli.md | 89 ++-- hetu-docs/zh/indexer/minmax.md | 27 +- hetu-docs/zh/indexer/overview.md | 35 +- hetu-docs/zh/language/reserved.md | 157 +++--- hetu-docs/zh/language/timestamp.md | 46 +- hetu-docs/zh/language/types.md | 496 +++++++++--------- .../zh/optimizer/cost-based-optimizations.md | 59 ++- hetu-docs/zh/optimizer/cost-in-explain.md | 62 ++- hetu-docs/zh/optimizer/statistics.md | 86 +-- hetu-docs/zh/rest/node.md | 128 ++--- hetu-docs/zh/rest/query.md | 28 +- hetu-docs/zh/rest/stage.md | 12 +- hetu-docs/zh/rest/statement.md | 153 +++--- hetu-docs/zh/rest/task.md | 53 +- 31 files changed, 918 insertions(+), 832 deletions(-) diff --git a/hetu-docs/en/indexer/bitmap.md b/hetu-docs/en/indexer/bitmap.md index 06f3b9671..bd522b7a7 100644 --- a/hetu-docs/en/indexer/bitmap.md +++ b/hetu-docs/en/indexer/bitmap.md @@ -1,3 +1,8 @@ ++++ +weight = 3 +title = "Bitmap Index" ++++ + # Bitmap Index ## Use cases diff --git a/hetu-docs/en/indexer/bloom.md b/hetu-docs/en/indexer/bloom.md index 6f726cd6e..dc9bc87a1 100644 --- a/hetu-docs/en/indexer/bloom.md +++ b/hetu-docs/en/indexer/bloom.md @@ -1,3 +1,8 @@ ++++ +weight = 5 +title = "Bloom Index" ++++ + # Bloom Index ## Use cases diff --git a/hetu-docs/en/indexer/indexer-cli.md b/hetu-docs/en/indexer/indexer-cli.md index e8b789e1b..92b7b2072 100644 --- a/hetu-docs/en/indexer/indexer-cli.md +++ b/hetu-docs/en/indexer/indexer-cli.md @@ -1,3 +1,8 @@ ++++ +weight = 2 +title = "Index Command Line Interface" ++++ + # Index Command Line Interface ## Usage diff --git a/hetu-docs/en/indexer/overview.md b/hetu-docs/en/indexer/overview.md index 615d67a15..50f28c80b 100644 --- a/hetu-docs/en/indexer/overview.md +++ b/hetu-docs/en/indexer/overview.md @@ -1,3 +1,8 @@ ++++ +weight = 1 +title = "openLooKeng Heuristic Indexer" ++++ + # openLooKeng Heuristic Indexer ## Introduction diff --git a/hetu-docs/en/language/reserved.md b/hetu-docs/en/language/reserved.md index 88828a5a2..16a92835b 100644 --- a/hetu-docs/en/language/reserved.md +++ b/hetu-docs/en/language/reserved.md @@ -1,3 +1,8 @@ ++++ +weight = 2 +title = "Reserved Keywords" ++++ + Reserved Keywords ================= diff --git a/hetu-docs/en/language/timestamp.md b/hetu-docs/en/language/timestamp.md index 6f60b22a0..4c4f894f2 100644 --- a/hetu-docs/en/language/timestamp.md +++ b/hetu-docs/en/language/timestamp.md @@ -1,3 +1,8 @@ ++++ +weight = 3 +title = "Legacy and New Timestamp" ++++ + Legacy and New Timestamp ======================== @@ -20,7 +25,7 @@ Additionally, it can be enabled or disabled on a per-session basis with the `leg Previously, the `TIMESTAMP` type described an instance in time in the openLooKeng session\'s time zone. Now, openLooKeng treats `TIMESTAMP` values as a set of the following fields representing wall time: -- `YEAR OF ERA` +- `YEAR OF ERA` - `MONTH OF YEAR` - `DAY OF MONTH` - `HOUR OF DAY` diff --git a/hetu-docs/en/language/types.md b/hetu-docs/en/language/types.md index 7f548478c..b56b2bfc9 100644 --- a/hetu-docs/en/language/types.md +++ b/hetu-docs/en/language/types.md @@ -1,3 +1,8 @@ ++++ +weight = 1 +title = "Data Types" ++++ + Data Types ========== @@ -6,7 +11,7 @@ openLooKeng has a set of built-in data types, described below. Additional types **Note** -*Connectors are not required to support all types. See connector* *documentation for details on supported types.* +*Connectors are not required to support all types. See [connector](../connector.html) for details on supported types.* Boolean ------- @@ -126,7 +131,7 @@ String Date and Time ------------- -See also `/language/timestamp` +See also [timestamp](timestamp.html). ### `DATE` @@ -179,6 +184,7 @@ Structural > > Example: `ARRAY[1, 2, 3]` + ### `MAP` > A map between the given component types. diff --git a/hetu-docs/en/optimizer/cost-based-optimizations.md b/hetu-docs/en/optimizer/cost-based-optimizations.md index 5cc736f21..3e896956e 100644 --- a/hetu-docs/en/optimizer/cost-based-optimizations.md +++ b/hetu-docs/en/optimizer/cost-based-optimizations.md @@ -1,3 +1,8 @@ ++++ +weight = 3 +title = "Cost based optimizations" ++++ + Cost based optimizations ======================== @@ -16,9 +21,9 @@ The join enumeration strategy is governed by the `join_reordering_strategy` sess The valid values are: -- `AUTOMATIC` (default) - full automatic join enumeration enabled +- `AUTOMATIC` (default) - full automatic join enumeration enabled - `ELIMINATE_CROSS_JOINS` - eliminate unnecessary cross joins -- `NONE` - purely syntactic join order +- `NONE` - purely syntactic join order If using `AUTOMATIC` and statistics are not available, or if for any other reason a cost could not be computed, the `ELIMINATE_CROSS_JOINS` strategy is used instead. @@ -43,8 +48,8 @@ The join distribution strategy is governed by the `join_distribution_type` sessi The valid values are: - `AUTOMATIC` (default) - join distribution type is determined automatically for each join - - `BROADCAST` - broadcast join distribution is used for all joins - - PARTITIONED` - partitioned join distribution is used for all join +- `BROADCAST` - broadcast join distribution is used for all joins +- `PARTITIONED` - partitioned join distribution is used for all join Connector Implementations ------------------------- diff --git a/hetu-docs/en/optimizer/cost-in-explain.md b/hetu-docs/en/optimizer/cost-in-explain.md index 4379c3ef4..ee5e0e49b 100644 --- a/hetu-docs/en/optimizer/cost-in-explain.md +++ b/hetu-docs/en/optimizer/cost-in-explain.md @@ -1,3 +1,8 @@ ++++ +weight = 2 +title = "Cost in EXPLAIN" ++++ + Cost in EXPLAIN =============== diff --git a/hetu-docs/en/optimizer/statistics.md b/hetu-docs/en/optimizer/statistics.md index d8cd56a6a..72055ef03 100644 --- a/hetu-docs/en/optimizer/statistics.md +++ b/hetu-docs/en/optimizer/statistics.md @@ -1,9 +1,14 @@ ++++ +weight = 1 +title = "Table Statistics" ++++ + Table Statistics ================ openLooKeng supports statistics based optimizations for queries. For a query to take advantage of these optimizations, openLooKeng must have statistical information for the tables in that query. -Table statistics are provided to the query planner by connectors. Currently, the only connector that supports statistics is the [hive](../en/connector/hive.html). +Table statistics are provided to the query planner by connectors. Currently, the only connector that supports statistics is the [hive connector](../connector/hive.html). Table Layouts ------------- @@ -39,4 +44,4 @@ The following statistics are available in openLooKeng: The set of statistics available for a particular query depends on the connector being used and can also vary by table or even by table layout. For example, the Hive connector does not currently provide statistics on data size. -Table statistics can be displayed via the openLooKeng SQL interface using the [show-stats](../sql/show-stats.html) command. For the Hive connector, refer to the [Hive connector](../en/connector/hive#hive_analyze.html) documentation to learn how to update table statistics. \ No newline at end of file +Table statistics can be displayed via the openLooKeng SQL interface using the [show-stats](../sql/show-stats.html) command. For the Hive connector, refer to the [Hive connector](../connector/hive.html) documentation to learn how to update table statistics. \ No newline at end of file diff --git a/hetu-docs/en/rest/node.md b/hetu-docs/en/rest/node.md index 6cc6cc9cf..7b61e691f 100644 --- a/hetu-docs/en/rest/node.md +++ b/hetu-docs/en/rest/node.md @@ -1,3 +1,8 @@ ++++ +weight = 1 +title = "Node Resource" ++++ + Node Resource ============= @@ -42,9 +47,9 @@ about traffic uptime, and failures. > "recentFailureRatio":0.0, > "recentFailuresByType":{} > } -> ``` > -> \] +> ] +> ``` If a node is experiencing errors, you\'ll see a response that looks like the following. Here we have a node which has experienced a spate of @@ -55,31 +60,31 @@ have occurred recently on a particular node. > ``` http > HTTP/1.1 200 OK -> ``` > -> Vry: Accept Content-Type: text/javascript +> Vary: Accept Content-Type: text/javascript > -> \[ +> [ > -> { +> { > -> ​ \"age\": \"4.45m\", \"lastFailureInfo\": { \"message\": ->​ \"Connect Timeout\", \"stack\": \[ -> ​ \"org.eclipse.jetty.io.ManagedSelector\$ConnectTimeout.run(ManagedSelector.java:683)\", -> ​ \.... \"java.lang.Thread.run(Thread.java:745)\" \], -> ​ \"suppressed\": \[\], \"type\": -> ​ \"java.net.SocketTimeoutException\" }, \"lastRequestTime\": -> ​ \"2017-08-05T11:53:00.647Z\", \"lastResponseTime\": -> ​ \"2017-08-05T11:53:00.647Z\", \"recentFailureRatio\": -> ​ 0.47263053472046446, \"recentFailures\": 2.8445543205610617, -> ​ \"recentFailuresByType\": { -> ​ \"java.net.SocketTimeoutException\": 2.8445543205610617 }, -> ​ \"recentRequests\": 6.018558073577414, \"recentSuccesses\": -> ​ 3.1746446343010297, \"uri\": \"\" +> ​ "age": "4.45m", "lastFailureInfo": { "message": +>​ "Connect Timeout", "stack": [ +> ​ "org.eclipse.jetty.io.ManagedSelector$ConnectTimeout.run(ManagedSelector.java:683)", +> ​ .... "java.lang.Thread.run(Thread.java:745)" ], +> ​ "suppressed": [], "type": +> ​ "java.net.SocketTimeoutException" }, "lastRequestTime": +> ​ "2017-08-05T11:53:00.647Z", "lastResponseTime": +> ​ "2017-08-05T11:53:00.647Z", "recentFailureRatio": +> ​ 0.47263053472046446, "recentFailures": 2.8445543205610617, +> ​ "recentFailuresByType": { +> ​ "java.net.SocketTimeoutException": 2.8445543205610617 }, +> ​ "recentRequests": 6.018558073577414, "recentSuccesses": +> ​ 3.1746446343010297, "uri": "" > -> } +> } > -> \] +> ] +> ``` @@ -92,31 +97,27 @@ call is the same as the information returned by the previous service. **Example response**: -> ``` http +> ``` json > -> ``` -> -> \[ -> +> [ > { > -> \"age\": \"1.37m\", \"lastFailureInfo\": { \"message\": -> \"Connect Timeout\", \"stack\": \[ -> \"org.eclipse.jetty.io.ManagedSelector\$ConnectTimeout.run(ManagedSelector.java:683)\", -> \..... -> \"java.util.concurrent.ThreadPoolExecutor\$Worker.run(ThreadPoolExecutor.java:617)\", -> \"java.lang.Thread.run(Thread.java:745)\" \], \"suppressed\": -> \[\], \"type\": \"java.net.SocketTimeoutException\" }, -> \"lastRequestTime\": \"2017-08-05T11:52:42.647Z\", -> \"lastResponseTime\": \"2017-08-05T11:52:42.647Z\", -> \"recentFailureRatio\": 0.22498784153043677, -> \"recentFailures\": 20.11558290058638, -> \"recentFailuresByType\": { -> \"java.net.SocketTimeoutException\": 20.11558290058638 }, -> \"recentRequests\": 89.40742203558189, \"recentSuccesses\": -> 69.30583024727453, \"uri\": \"\" +> "age": "1.37m", "lastFailureInfo": { "message": +> "Connect Timeout", "stack": [ +> "org.eclipse.jetty.io.ManagedSelector$ConnectTimeout.run(ManagedSelector.java:683)", +> ... +> "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)", +> "java.lang.Thread.run(Thread.java:745)" ], "suppressed": +> [], "type": "java.net.SocketTimeoutException" }, +> "lastRequestTime": "2017-08-05T11:52:42.647Z", +> "lastResponseTime": "2017-08-05T11:52:42.647Z", +> "recentFailureRatio": 0.22498784153043677, +> "recentFailures": 20.11558290058638, +> "recentFailuresByType": { +> "java.net.SocketTimeoutException": 20.11558290058638 }, +> "recentRequests": 89.40742203558189, "recentSuccesses": +> 69.30583024727453, "uri": "" > -> } -> -> \] -> +> } +> ] +> ``` diff --git a/hetu-docs/en/rest/query.md b/hetu-docs/en/rest/query.md index 6f8041240..eb447b0c4 100644 --- a/hetu-docs/en/rest/query.md +++ b/hetu-docs/en/rest/query.md @@ -1,3 +1,8 @@ ++++ +weight = 2 +title = "Query Resource" ++++ + Query Resource ============== @@ -25,7 +30,7 @@ query will reveal a link to this service. **Example response**: -> ``` http +> ``` json > { > "queryId" : "20131229_211533_00017_dk5x2", > "session" : { diff --git a/hetu-docs/en/rest/stage.md b/hetu-docs/en/rest/stage.md index 3aabc0819..738451653 100644 --- a/hetu-docs/en/rest/stage.md +++ b/hetu-docs/en/rest/stage.md @@ -1,3 +1,8 @@ ++++ +weight = 3 +title = "Stage Resource" ++++ + Stage Resource ============== diff --git a/hetu-docs/en/rest/statement.md b/hetu-docs/en/rest/statement.md index 7d9b2b086..afcc15fb8 100644 --- a/hetu-docs/en/rest/statement.md +++ b/hetu-docs/en/rest/statement.md @@ -1,3 +1,8 @@ ++++ +weight = 4 +title = "Statement Resource" ++++ + Statement Resource ================== @@ -49,46 +54,47 @@ additional results from the query. > ``` http > POST /v1/statement HTTP/1.1 -> ``` > > Host: localhost:8001 X-Presto-Catalog: jmx X-Presto-Source: presto-cli > X-Presto-Schema: jmx User-Agent: StatementClient/0.55-SNAPSHOT > X-Presto-User: tobrie1 Content-Length: 41 > -> select name from \"java.lang:type=runtime\" +> select name from "java.lang:type=runtime" +> ``` + **Example response**: > ``` http > HTTP/1.1 200 OK -> ``` > > Content-Type: application/json X-Content-Type-Options: nosniff > Transfer-Encoding: chunked > > { > -> : \"id\":\"20140108\_110629\_00011\_dk5x2\", -> \"infoUri\":\"\", -> \"partialCancelUri\":\"\", -> \"nextUri\":\"\", -> \"columns\": \[ { \"name\":\"name\", \"type\":\"varchar\" } \], -> \"stats\": { \"state\":\"RUNNING\", \"scheduled\":false, -> \"nodes\":1, \"totalSplits\":0, \"queuedSplits\":0, -> \"runningSplits\":0, \"completedSplits\":0, \"cpuTimeMillis\":0, -> \"wallTimeMillis\":0, \"processedRows\":0, \"processedBytes\":0, -> \"rootStage\": { \"stageId\":\"0\", \"state\":\"SCHEDULED\", -> \"done\":false, \"nodes\":1, \"totalSplits\":0, -> \"queuedSplits\":0, \"runningSplits\":0, \"completedSplits\":0, -> \"cpuTimeMillis\":0, \"wallTimeMillis\":0, \"processedRows\":0, -> \"processedBytes\":0, \"subStages\": \[ { \"stageId\":\"1\", -> \"state\":\"SCHEDULED\", \"done\":false, \"nodes\":1, -> \"totalSplits\":0, \"queuedSplits\":0, \"runningSplits\":0, -> \"completedSplits\":0, \"cpuTimeMillis\":0, \"wallTimeMillis\":0, -> \"processedRows\":0, \"processedBytes\":0, \"subStages\":\[\] } \] +> "id":"20140108_110629_00011_dk5x2", +> "infoUri":"", +> "partialCancelUri":"", +> "nextUri":"", +> "columns": [ { "name":"name", "type":"varchar" } ], +> "stats": { "state":"RUNNING", "scheduled":false, +> "nodes":1, "totalSplits":0, "queuedSplits":0, +> "runningSplits":0, "completedSplits":0, "cpuTimeMillis":0, +> "wallTimeMillis":0, "processedRows":0, "processedBytes":0, +> "rootStage": { "stageId":"0", "state":"SCHEDULED", +> "done":false, "nodes":1, "totalSplits":0, +> "queuedSplits":0, "runningSplits":0, "completedSplits":0, +> "cpuTimeMillis":0, "wallTimeMillis":0, "processedRows":0, +> "processedBytes":0, "subStages": [ { "stageId":"1", +> "state":"SCHEDULED", "done":false, "nodes":1, +> "totalSplits":0, "queuedSplits":0, "runningSplits":0, +> "completedSplits":0, "cpuTimeMillis":0, "wallTimeMillis":0, +> "processedRows":0, "processedBytes":0, "subStages":[] } ] > } } > > } +> ``` - GET /v1/statement/{queryId}/{token} @@ -119,28 +125,28 @@ in progress or it can deliver the final results to the client. > ``` http > HTTP/1.1 200 OK -> ``` > > Content-Type: application/json X-Content-Type-Options: nosniff Vary: > Accept-Encoding, User-Agent Transfer-Encoding: chunked > -> 383 { \"id\":\"20140108\_110629\_00011\_dk5x2\", -> \"infoUri\":\"\", -> \"columns\": \[ { \"name\":\"name\", \"type\":\"varchar\" } \], -> \"data\": \[ \[\"<4165@domU-12-31-39-0F-CC-72>\"\] \], \"stats\": { -> \"state\":\"FINISHED\", \"scheduled\":true, \"nodes\":1, -> \"totalSplits\":2, \"queuedSplits\":0, \"runningSplits\":0, -> \"completedSplits\":2, \"cpuTimeMillis\":1, \"wallTimeMillis\":4, -> \"processedRows\":1, \"processedBytes\":27, \"rootStage\": { -> \"stageId\":\"0\", \"state\":\"FINISHED\", \"done\":true, \"nodes\":1, -> \"totalSplits\":1, \"queuedSplits\":0, \"runningSplits\":0, -> \"completedSplits\":1, \"cpuTimeMillis\":0, \"wallTimeMillis\":0, -> \"processedRows\":1, \"processedBytes\":32, \"subStages\": \[ { -> \"stageId\":\"1\", \"state\":\"FINISHED\", \"done\":true, \"nodes\":1, -> \"totalSplits\":1, \"queuedSplits\":0, \"runningSplits\":0, -> \"completedSplits\":1, \"cpuTimeMillis\":0, \"wallTimeMillis\":4, -> \"processedRows\":1, \"processedBytes\":27, \"subStages\":\[\] } \] } +> 383 { "id":"20140108_110629_00011_dk5x2", +> "infoUri":"", +> "columns": [ { "name":"name", "type":"varchar" } ], +> "data": [ ["<4165@domU-12-31-39-0F-CC-72>"] ], "stats": { +> "state":"FINISHED", "scheduled":true, "nodes":1, +> "totalSplits":2, "queuedSplits":0, "runningSplits":0, +> "completedSplits":2, "cpuTimeMillis":1, "wallTimeMillis":4, +> "processedRows":1, "processedBytes":27, "rootStage": { +> "stageId":"0", "state":"FINISHED", "done":true, "nodes":1, +> "totalSplits":1, "queuedSplits":0, "runningSplits":0, +> "completedSplits":1, "cpuTimeMillis":0, "wallTimeMillis":0, +> "processedRows":1, "processedBytes":32, "subStages": [ { +> "stageId":"1", "state":"FINISHED", "done":true, "nodes":1, +> "totalSplits":1, "queuedSplits":0, "runningSplits":0, +> "completedSplits":1, "cpuTimeMillis":0, "wallTimeMillis":4, +> "processedRows":1, "processedBytes":27, "subStages":[] } ] } > } } +> ``` - DELETE /v1/statement/{queryId}/{token} diff --git a/hetu-docs/en/rest/task.md b/hetu-docs/en/rest/task.md index 49bad82d2..efb7f8829 100644 --- a/hetu-docs/en/rest/task.md +++ b/hetu-docs/en/rest/task.md @@ -1,3 +1,8 @@ ++++ +weight = 5 +title = "Task Resource" ++++ + Task Resource ============= @@ -8,7 +13,7 @@ execution of queries on a openLooKeng installation. - GET /v1/task -Returns information about all tasks known to a openLooKengServer. +Returns information about all tasks known to a openLooKeng server. Note that the output of a call to `/v1/task` can be quite large. If you execute this against a busy openLooKeng server the response received will @@ -22,7 +27,7 @@ server would generate pages and pages of output. Here there is a **Example response**: -``` http +``` json [ { "taskId" : "20131222_183944_00011_dk5x2.1.0", "version" : 9223372036854775807, @@ -89,7 +94,7 @@ particular query. This is call is used by openLooKeng to coordinate a queries. **Example response**: -``` http +``` json { "taskId" : "20140115_170528_00004_dk5x2.0.0", "version" : 42, diff --git a/hetu-docs/zh/indexer/bitmap.md b/hetu-docs/zh/indexer/bitmap.md index 06f3b9671..e3e46bc06 100644 --- a/hetu-docs/zh/indexer/bitmap.md +++ b/hetu-docs/zh/indexer/bitmap.md @@ -1,18 +1,23 @@ -# Bitmap Index ++++ +weight = 3 +title = "位图索引" ++++ + +# 位图索引 -## Use cases +## 用例 -Bitmap Index is used for filtering data read from ORC files and is used only by the **worker** nodes +位图索引用于过滤从ORC文件中读取的数据，且仅供**worker**节点使用。 -- If this index exists on a column which is part of a predicate in the query, the performance may be improved while reading the ORC files. +- 如果包含这个索引的列是查询中谓词的一部分，那么读取ORC文件的性能可能会得提升。 -For example, if an index exists on column `country` and the query is +例如，如果索引在`country`列，并且查询语句是 -``` sql -select * from table where country="China" +``` sql +select * from table where country="China" ``` -- This index works best if the column's values are not too distinct (e.g. country) and are distributed. +- 如果列的值不是太明显（例如国家）和分散，则此索引最有效。 -For example, assume that the table stores information about where users are from and the table data is in 10 files. There maybe be several users from a particular country, so each file will have some users from the country. If we create a bitmap index on the country column, we can perform filtering early on while reading the data files. i.e. the predicate is pushed down to the reading of the file. Without this index, all the data files will need to be read into memory as Pages and then the filtering would happen. With the index, we can ensure that the Pages already only contain the rows matching the predicate. This can help reduce the memory and CPU usage and can result in improved performance when many concurrent queries are running. +例如，假设表存储的是用户来自何处的信息，并且表数据存在于10个文件中。可能有多个用户来自某一国家，因此每个文件将有一些来自该国的用户。如果我们在国家列创建一个位图索引，那么在读取数据文件时，我们可以在早期执行过滤。即，谓词被下推到文件读取。如果没有这个索引，所有的数据文件将会作为页读入内存，然后再过滤。如果有此索引，我们可以确保内存页中已经只包含与谓词匹配的行。这有助于减少内存和CPU使用率，并且提高多并发查询的性能。 diff --git a/hetu-docs/zh/indexer/bloom.md b/hetu-docs/zh/indexer/bloom.md index 6f726cd6e..2a3e2b31f 100644 --- a/hetu-docs/zh/indexer/bloom.md +++ b/hetu-docs/zh/indexer/bloom.md @@ -1,21 +1,24 @@ -# Bloom Index ++++ +weight = 5 +title = "Bloom索引" ++++ + +# Bloom索引 -## Use cases +## 用例 -Bloom Index is used for split filtering, and is used only by the **coordinator** nodes. +Bloom索引用于拆分过滤，且仅被**coordinator**节点使用。 -- If this index exists on a column which is part of a predicate in the query, openLooKeng may be able to improve performance by filtering scheduled splits. +- 如果查询中作为谓词一部分的列存在此索引，openLooKeng可以通过筛选预定Splits来提高查询性能。 -For example, if an index exists on column `id` and the query is: - -```sql -select * from table where id=12345 +例如，如果列`id`包含此索引，并且查询语句如下： + +```sql +select * from table where id=12345 ``` +- 如果列的值是唯一的（例如userid）并且不太分散，则Bloom索引最有效。 - -- Bloom index works best if the column's values are unique (e.g. userid) and are not too distributed. - -For example, assume the tables stores information about users and the table data is in 10 files. For a given userid, only one file will contain the data. Therefore creating an index on userid will help us to filter out 9 out of the 10 files at scheduling time and will save significant IO time that would've therwise been used to read each of the files. - -*Tip: if possible, it is recommended to sort the data on the column being indexed.* \ No newline at end of file +例如，假设表中存储了用户信息，且表数据存在于10个文件中。 对于给定的用户ID，只有一个文件包含该数据。因此，对用户ID创建索引将帮助我们在调度时间内过滤掉10个文件中的9个文件，并节省大量用于读取每个文件的IO时间。 + +*提示：如果可能，建议对索引列中的数据进行排序*。 diff --git a/hetu-docs/zh/indexer/indexer-cli.md b/hetu-docs/zh/indexer/indexer-cli.md index e8b789e1b..4b468f694 100644 --- a/hetu-docs/zh/indexer/indexer-cli.md +++ b/hetu-docs/zh/indexer/indexer-cli.md @@ -1,99 +1,98 @@ -# Index Command Line Interface ++++ +weight = 2 +title = "索引命令行接口" ++++ + +# 索引命令行接口 -## Usage +## 用法 -The index executable will be located under the `bin` directory in the installation. - -For example, `/bin/index` and must be executed from the `bin` directory because it uses relative paths by default. +可执行索引文件在安装中位于`bin`目录下，例如`<安装路径>/bin/index`。因为默认使用相对路径，它必须在`bin`目录下运行。 ``` -Usage: index [-v] [--debug] [--disableLocking] --table= +使用方法：index [-v] [--debug] [--disableLocking] --table=
[-c=] [--column=[,...]]... [--partition=[,...]]... - [--type=[,...]]... [-p=[, - ...]]... + [--type=[,...]]...[-p=[, + ...]]... -Using this index tool, you can CREATE, SHOW and DELETE indexes. +使用此索引工具，您可以创建、显示和删除索引。 -Supported index types: BITMAP, BLOOM, MINMAX +支持的索引类型如下：BITMAP, BLOOM, MINMAX -Supported index stores: LOCAL, HDFS (must be configured in {--config}/config.properties +支持的索引存储：LOCAL, HDFS (必须 在{--config}/config.properties配置 -Supported data sources: HIVE using ORC files (must be configured in {--config}/catalog/catalog_name.properties +支持的数据源：HIVE using ORC files (必须在{--config}/catalog/catalog_name.properties配置 - command types, e.g. create, delete, show; Note: delete command - works a column level only. + 命令类型，如create、delete、show等；说明：delete命令只作用于列级。 + --column=[,...] - column, comma separated format for multiple columns - --debug if enabled the original data for each split will - also be written to a file alongside the index - --disableLocking by default locking is enabled at the table level; if this - is set to false, the user must ensure that the same data - is not indexed by multiple callers at the same time - (indexing different columns or partitions in parallel is - allowed) + 列，使用逗号分隔多个列 + --debug 如果启用，则每个Split的原始数据也将随索引一起写入文件 + + --disableLocking 默认锁定在表级别启用；如果设置为false，用户必须确保相同的数据没有被多个调用方同时索引（允许不同列或分区并行索引） + --partition=[,...] - only create index for these partitions, comma separated - format for multiple partitions - --table=
fully qualified table name + 只为这些分区创建索引，用逗号分隔多个分区 + + --table=
全量表名 --type=[,...] - index type, comma separated format for multiple types - (supported types: BLOOM, BITMAP, MINMAX + 索引类型，用逗号分隔多种类型 (支持的类型：BLOOM, BITMAP, MINMAX -c, --config= - root folder of openLooKeng etc directory (default: ../etc) + openLooKeng etc目录的根目录（默认为../etc） -p, --plugins=[,...] - plugins dir or file, defaults to (default: . - /hetu-heuristic-index/plugins) + plugins目录或文件(默认为. /hetu-heuristic-index/plugins） + -v verbose ``` -## Examples +## 示例 -### Create index +### 创建索引 ``` shell $ ./index -v -c ../etc --table hive.schema.table --column column1,column2 --type bloom,minmax,bitmap --partition p=part1 create ``` -### Show index +### 显示索引 ``` shell $ ./index -v -c ../etc --table hive.schema.table show ``` -### Delete index +### 删除索引 -*Note:* index can only be deleted at table or column level, i.e. all index types will be deleted +*注意：*索引只能在表或列级别删除，即所有索引类型都将被删除。 ``` shell $ ./index -v -c ../etc --table hive.schema.table --column column1 delete ``` -## Notes on resource usage +## 资源使用说明 -### Memory +### 内存 + +默认情况下，将使用默认的JVMMaxHeapSize(`java -XX:+PrintFlagsFinal -version | grep MaxHeapSize`)。为了提高性能，建议增大MaxHeapSize的取值。可以通过设置-Xmx值来实现： -By default the default JVM MaxHeapSize will be used (`java -XX:+PrintFlagsFinal -version | grep MaxHeapSize`). For improved performance, it is recommended to increase the MaxHeapSize. This can be -done by setting -Xmx value: ``` shell export JAVA_TOOL_OPTIONS="-Xmx100G" ``` -In this example the MaxHeapSize will be set to 100G. +在此示例中，MaxHeapSize被设置为100 GB。 -### Indexing in parallel +###并行索引 -If creating the index for a large table is too slow on one machine, you can create index for different partitions in parallel on different machines. This requires setting the --disableLocking flag and specifying the partition(s). For example: +如果在一台机器上为一个大表创建索引的速度太慢，则可以在不同的机器上并行为不同的分区创建索引。这需要设置--disableLocking标志并指定分区。例如： -On machine 1: +在机器1上： -``` bash +``` bash $ ./index -v ---disableLocking c ../etc --table hive.schema.table --columncolumn1,column2 --type bloom,minmax,bitmap --partition p=part1 create ``` -On machine 2: +在机器2上： ``` shell $ ./index -v ---disableLocking c ../etc --table hive.schema.table --columncolumn1,column2 --type bloom,minmax,bitmap --partition p=part2 create diff --git a/hetu-docs/zh/indexer/minmax.md b/hetu-docs/zh/indexer/minmax.md index dbfbc56cb..d0f8fbcd4 100644 --- a/hetu-docs/zh/indexer/minmax.md +++ b/hetu-docs/zh/indexer/minmax.md @@ -1,21 +1,24 @@ -# Minmax Index ++++ +weight = 4 +title = "Minmax索引" ++++ + +# Minmax索引 -## Use cases +## 用例 -MinMax Index is used for split filtering, and is used only by the **coordinator** nodes. +MinMax 索引用于拆分过滤，仅被**coordinator**节点使用。 -If this index exists on a column which is part of a predicate in the query, the engine may be able to improve performance by filtering scheduled splits similar to Bloom Index. +如果查询中作为谓词一部分的列存在此索引，则引擎可以通过筛选预定Splits来提高性能，类似于Bloom索引。 -For example if an index exists on column +例如，如果索引在列 `age` -and the query is +查询语句为 -```sql -select * from table where age > 50 +```sql +select * from table where age > 50 ``` - - - -*Tip: sorting the data on the index column will provide the best results* \ No newline at end of file + +*提示：对索引列中的数据进行排序将提供最佳结果。* diff --git a/hetu-docs/zh/indexer/overview.md b/hetu-docs/zh/indexer/overview.md index 615d67a15..e622123bc 100644 --- a/hetu-docs/zh/indexer/overview.md +++ b/hetu-docs/zh/indexer/overview.md @@ -1,29 +1,34 @@ -# openLooKeng Heuristic Indexer ++++ +weight = 1 +title = "openLooKeng启发式索引器" ++++ + +# openLooKeng启发式索引器 -## Introduction +## 简介 -Indexes can be created using one or more columns of a database table, providing faster random lookups. Most Big Data formats such as ORC, Parquet and CarbonData already have indices embedded in them. +可以在数据库表的一个或多个列创建索引，从而提供更快的随机查找。大多数大数据格式，如ORC、Parquet和CarbonData，都已经内置了索引。 -The Heuristic Indexer allows creating indexes on existing data but stores the index external to the original data source. This provides several benefits: +启发式索引器允许在现有数据上创建索引，但将索引存储在外部的的原始数据源中。这样做好处如下： - - The index is agnostic to the underlying data source and can be used by any query engine - - Existing data can be indexed without having to rewrite the existing data files - - New index types not supported by the underlying data source can be created - - Index data does not use the storage space of the data source + - 索引对底层数据源不可知，并且可由任何查询引擎使用 + - 无需重写现有数据文件即可对现有数据进行索引 + - 可以创建底层数据源不支持的新索引类型 + - 索引数据不占用数据源存储空间 -## Example Usecases +## 示例 -### 1. Filtering scheduled Splits during query execution +### 1.查询过程中过滤预定分段 -When the engine needs to schedule a TableScan operation, it schedules Splits on the workers. These Splits are responsible for reading a portion of the source data. However, not all Splits will return data if a predicate is applied. +当引擎需要调度一个TableScan操作时，它可以调度worker节点上的Split。这些Split负责读取部分源数据。但是如果应用了谓词，则并非所有Split都会返回数据。 -By keeping an external index for the predicate column, the Heuristic Indexer can determine whether each split contains the values being searched for and only schedule the read operation for the splits which possibly contain the value. +通过为谓词列保留外部索引，启发式索引器可以确定每个Split是否包含正在搜索的值，并且只对可能包含该值的Split安排读操作。 ![](indexer_filter_splits.png) -### 2. Filtering Block early when reading ORC files +### 2.读取ORC文件时提前筛选块 -When data needs to be read from an ORC file, the ORCRecordReader is used. This reader reads data from Stripes as batches (e.g. 1024 rows), which then form Pages. However, if a predicate is present, not all entries in the batch are required, some may be filtered out later by the Filter operator. +当需要从ORC文件中读取数据时，使用ORCRecordReader读取器。此读取器从数据条带批量（例如1024行）读取数据，然后形成页。但是，如果有一个谓词存在，那么就不需要批量读取中的所有条目，有些条目可能会在稍后被Filter运算符过滤掉。 -By keeping an external bitmap index for the predicate column, the Heuristic Indexer can filter out rows which do not match the predicates before the Filter operator is even applied. +通过为谓词列保留外部位图索引，启发式索引器甚至可以在应用Filter运算符之前筛选出与谓词不匹配的行。 diff --git a/hetu-docs/zh/language/reserved.md b/hetu-docs/zh/language/reserved.md index 88828a5a2..7263b63d3 100644 --- a/hetu-docs/zh/language/reserved.md +++ b/hetu-docs/zh/language/reserved.md @@ -1,82 +1,87 @@ -Reserved Keywords ++++ +weight = 2 +title = "预留关键字" ++++ + +预留关键字 ================= -The following table lists all of the keywords that are reserved in openLooKeng, along with their status in the SQL standard. These reserved keywords must be quoted (using double quotes) in order to be used as an identifier. +下表列出了openLooKeng中预留的所有关键字，以及它们在SQL标准中的状态。这些预留的关键字必须加引号（使用双引号），以便用作标识符。 -| Keyword | SQL:2016 | SQL-92 | +|关键字 | SQL:2016 | SQL-92 | | ------------------- | -------- | -------- | -| `ALTER` | reserved | reserved | -| `AND` | reserved | reserved | -| `AS` | reserved | reserved | -| `BETWEEN` | reserved | reserved | -| `BY` | reserved | reserved | -| `CASE` | reserved | reserved | +| `ALTER` | 预留 | 预留 | +| `AND` | 预留 | 预留 | +| `AS` | 预留 | 预留 | +| `BETWEEN` | 预留 | 预留 | +| `BY` | 预留 | 预留 | +| `CASE` | 预留 | 预留 | | `CACHE` | | | -| `CAST` | reserved | reserved | -| `CONSTRAINT` | reserved | reserved | -| `CREATE` | reserved | reserved | -| `CROSS` | reserved | reserved | -| `CUBE` | reserved | | -| `CURRENT_DATE` | reserved | reserved | -| `CURRENT_PATH` | reserved | | -| `CURRENT_ROLE` | reserved | reserved | -| `CURRENT_TIME` | reserved | reserved | -| `CURRENT_TIMESTAMP` | reserved | reserved | -| `CURRENT_USER` | reserved | | -| `DEALLOCATE` | reserved | reserved | -| `DELETE` | reserved | reserved | -| `DESCRIBE` | reserved | reserved | -| `DISTINCT` | reserved | reserved | -| `DROP` | reserved | reserved | -| `ELSE` | reserved | reserved | -| `END` | reserved | reserved | -| `ESCAPE` | reserved | reserved | -| `EXCEPT` | reserved | reserved | -| `EXECUTE` | reserved | reserved | -| `EXISTS` | reserved | reserved | -| `EXTRACT` | reserved | reserved | -| `FALSE` | reserved | reserved | -| `FOR` | reserved | reserved | -| `FROM` | reserved | reserved | -| `FULL` | reserved | reserved | -| `GROUP` | reserved | reserved | -| `GROUPING` | reserved | | -| `HAVING` | reserved | reserved | -| `IN` | reserved | reserved | -| `INNER` | reserved | reserved | -| `INSERT` | reserved | reserved | -| `INTERSECT` | reserved | reserved | -| `INTO` | reserved | reserved | -| `IS` | reserved | reserved | -| `JOIN` | reserved | reserved | -| `LEFT` | reserved | reserved | -| `LIKE` | reserved | reserved | -| `LOCALTIME` | reserved | | -| `LOCALTIMESTAMP` | reserved | | -| `NATURAL` | reserved | reserved | -| `NORMALIZE` | reserved | | -| `NOT` | reserved | reserved | -| `NULL` | reserved | reserved | -| `ON` | reserved | reserved | -| `OR` | reserved | reserved | -| `ORDER` | reserved | reserved | -| `OUTER` | reserved | reserved | -| `OVERWRITE` | reserved | | -| `PREPARE` | reserved | reserved | -| `RECURSIVE` | reserved | | -| `RIGHT` | reserved | reserved | -| `ROLLUP` | reserved | | -| `SELECT` | reserved | reserved | -| `TABLE` | reserved | reserved | -| `THEN` | reserved | reserved | -| `TRUE` | reserved | reserved | -| `UESCAPE` | reserved | | -| `UNION` | reserved | reserved | -| `UNNEST` | reserved | | -| `UPDATE` | reserved | | -| `USING` | reserved | reserved | -| `VALUES` | reserved | reserved | +| `CAST` | 预留 | 预留 | +| `CONSTRAINT` | 预留 | 预留 | +| `CREATE` | 预留 | 预留 | +| `CROSS` | 预留 | 预留 | +| `CUBE` | 预留 | | +| `CURRENT_DATE` | 预留 | 预留 | +| `CURRENT_PATH` | 预留 | | +| `CURRENT_ROLE` | 预留 | 预留 | +| `CURRENT_TIME` | 预留 | 预留 | +| `CURRENT_TIMESTAMP` | 预留 | 预留 | +| `CURRENT_USER` | 预留 | | +| `DEALLOCATE` | 预留 | 预留 | +| `DELETE` | 预留 | 预留 | +| `DESCRIBE` | 预留 | 预留 | +| `DISTINCT` | 预留 | 预留 | +| `DROP` | 预留 | 预留 | +| `ELSE` | 预留 | 预留 | +| `END` | 预留 | 预留 | +| `ESCAPE` | 预留 | 预留 | +| `EXCEPT` | 预留 | 预留 | +| `EXECUTE` | 预留 | 预留 | +| `EXISTS` | 预留 | 预留 | +| `EXTRACT` | 预留 | 预留 | +| `FALSE` | 预留 | 预留 | +| `FOR` | 预留 | 预留 | +| `FROM` | 预留 | 预留 | +| `FULL` | 预留 | 预留 | +| `GROUP` | 预留 | 预留 | +| `GROUPING` | 预留 | | +| `HAVING` | 预留 | 预留 | +| `IN` | 预留 | 预留 | +| `INNER` | 预留 | 预留 | +| `INSERT` | 预留 | 预留 | +| `INTERSECT` | 预留 | 预留 | +| `INTO` | 预留 | 预留 | +| `IS` | 预留 | 预留 | +| `JOIN` | 预留 | 预留 | +| `LEFT` | 预留 | 预留 | +| `LIKE` | 预留 | 预留 | +| `LOCALTIME` | 预留 | | +| `LOCALTIMESTAMP` | 预留 | | +| `NATURAL` | 预留 | 预留 | +| `NORMALIZE` | 预留 | | +| `NOT` | 预留 | 预留 | +| `NULL` | 预留 | 预留 | +| `ON` | 预留 | 预留 | +| `OR` | 预留 | 预留 | +| `ORDER` | 预留 | 预留 | +| `OUTER` | 预留 | 预留 | +| `OVERWRITE` | 预留 | | +| `PREPARE` | 预留 | 预留 | +| `RECURSIVE` | 预留 | | +| `RIGHT` | 预留 | 预留 | +| `ROLLUP` | 预留 | | +| `SELECT` | 预留 | 预留 | +| `TABLE` | 预留 | 预留 | +| `THEN` | 预留 | 预留 | +| `TRUE` | 预留 | 预留 | +| `UESCAPE` | 预留 | | +| `UNION` | 预留 | 预留 | +| `UNNEST` | 预留 | | +| `UPDATE` | 预留 | | +| `USING` | 预留 | 预留 | +| `VALUES` | 预留 | 预留 | | `VACUUM` | | | -| `WHEN` | reserved | reserved | -| `WHERE` | reserved | reserved | -| `WITH` | reserved | reserved | \ No newline at end of file +| `WHEN` | 预留 | 预留 | +| `WHERE` | 预留 | 预留 | +| `WITH` | 预留 | 预留 | \ No newline at end of file diff --git a/hetu-docs/zh/language/timestamp.md b/hetu-docs/zh/language/timestamp.md index 6f60b22a0..d972534be 100644 --- a/hetu-docs/zh/language/timestamp.md +++ b/hetu-docs/zh/language/timestamp.md @@ -1,45 +1,51 @@ -Legacy and New Timestamp ++++ +weight = 3 +title = "旧时间戳和新时间戳" ++++ + +旧时间戳和新时间戳 ======================== -New `TIMESTAMP` and `TIME` semantics align the types with the SQL standard. See the following sections for details. +新的`TIMESTAMP`和`TIME`语义使类型与SQL标准保持一致。详见以下章节。 -**Note** +**注意** -*The new `TIMESTAMP` semantics is still experimental. It\'s recommended* *to keep the legacy `TIMESTAMP` semantics enabled. You can experiment* *with the new semantics by configuring it globally or on a per-session* *basis. The legacy semantics may be deprecated in a future release.* +*新的`TIMESTAMP`语义仍在试验中。建议* 保持原有`TIMESTAMP`语义为启用状态。您可以通过全局配置或基于每个会话配置新的语义来验证新语义**。在新版本中，可能会废弃旧版的语义。 -Configuration +配置 ------------- -The legacy semantics can be enabled using the `deprecated.legacy-timestamp` config property. Setting it to `true` (the default) enables the legacy semantics, whereas setting it to `false` enables the new semantics. +可以使用`deprecated.legacy-timestamp`配置属性启用旧语义。将其设置为`true`（默认）将启用旧语义，而将其设置为`false`将启用新语义。 -Additionally, it can be enabled or disabled on a per-session basis with the `legacy_timestamp` session property. +此外，可以通过`legacy_timestamp`会话属性实现基于会话的语义启用或禁用。 -### TIMESTAMP semantic changes +### TIMESTAMP语义变化 -Previously, the `TIMESTAMP` type described an instance in time in the openLooKeng session\'s time zone. Now, openLooKeng treats `TIMESTAMP` values as a set of the following fields representing wall time: +以前，`TIMESTAMP`类型描述的是openLooKeng会话时区中的时间实例。现在，openLooKeng将`TIMESTAMP`值视为一组表示实际时间的字段： -- `YEAR OF ERA` +- `YEAR OF ERA` - `MONTH OF YEAR` - `DAY OF MONTH` - `HOUR OF DAY` - `MINUTE OF HOUR` - `SECOND OF MINUTE` - as `DECIMAL(5, 3)` -For that reason, a `TIMESTAMP` value is not linked with the session time zone in any way until a time zone is needed explicitly, such as when casting to a `TIMESTAMP WITH TIME ZONE` or `TIME WITH TIME ZONE`. In -those cases, the time zone offset of the session time zone is applied, as specified in the SQL standard. +因此，除非明确需要某个时区，例如在转换为`TIMESTAMP WITH TIME ZONE` 或 `TIME WITH TIME ZONE`时，`TIMESTAMP`值不会以任何方式与会话时区链接。在这些情况下，如SQL标准中所约定的，将会使用会话时区的时区偏移量。 -### TIME semantic changes -The `TIME` type was changed similarly to the `TIMESTAMP` type. +### TIME语义变化 -### TIME WITH TIME ZONE semantic changes +`TIME`类型变得与`TIMESTAMP`类型相似。 -Due to compatibility requirements, having `TIME WITH TIME ZONE` completely aligned with the SQL standard was not possible yet. For that reason, when calculating the time zone offset for `TIME WITH TIME ZONE`, openLooKeng uses the session\'s start date and time. +### TIME with TIME ZONE语义变化 -This can be seen in queries using `TIME WITH TIME ZONE` in a time zone that has had time zone policy changes or uses DST. For example, with a session start time of 2017-03-01: +由于兼容性要求，`TIME WITH TIME ZONE`还不可能与SQL标准完全对齐。因此，在计算`TIME WITH TIME ZONE`的时区偏移量时，openLooKeng使用会话的开始日期和时间。 -- Query: `SELECT TIME '10:00:00 Asia/Kathmandu' AT TIME ZONE 'UTC'` -- Legacy result: `04:30:00.000 UTC` -- New result: `04:15:00.000 UTC` + +在使用`TIME WITH TIME ZONE`的查询中可以看到，查询的时区已经发生了时区策略更改或使用了夏令时。例如，会话开始时间为2017-03-01： + +- 查询: `SELECT TIME '10:00:00 Asia/Kathmandu' AT TIME ZONE 'UTC'` +- 旧的查询结果: `04:30:00.000 UTC` +- 新的查询结果: `04:15:00.000 UTC` diff --git a/hetu-docs/zh/language/types.md b/hetu-docs/zh/language/types.md index 7f548478c..192e36524 100644 --- a/hetu-docs/zh/language/types.md +++ b/hetu-docs/zh/language/types.md @@ -1,247 +1,249 @@ -Data Types -========== - -openLooKeng has a set of built-in data types, described below. Additional types can be provided by plugins. - - -**Note** - -*Connectors are not required to support all types. See connector* *documentation for details on supported types.* - -Boolean -------- - -### `BOOLEAN` - -> This type captures boolean values `true` and `false`. - -Integer -------- - -### `TINYINT` - -> A 8-bit signed two\'s complement integer with a minimum value of `-2^7` and a maximum value of `2^7 - 1`. - -### `SMALLINT` - -> A 16-bit signed two\'s complement integer with a minimum value of `-2^15` and a maximum value of `2^15 - 1`. - -### `INTEGER` - -> A 32-bit signed two\'s complement integer with a minimum value of `-2^31` and a maximum value of `2^31 - 1`. The name `INT` is also available for this type. - -### `BIGINT` - -> A 64-bit signed two\'s complement integer with a minimum value of `-2^63` and a maximum value of `2^63 - 1`. - -Floating-Point --------------- - -### `REAL` - -> A real is a 32-bit inexact, variable-precision implementing the IEEE Standard 754 for Binary Floating-Point Arithmetic. -> -> The name `FLOAT` is also available for this type. -> -> Example: REAL '10.3', REAL '10.3e0', REAL '1.03e1' - -### `DOUBLE` - -> A double is a 64-bit inexact, variable-precision implementing the IEEE Standard 754 for Binary Floating-Point Arithmetic. -> -> The name `DOUBLE PRECISION` is also available for this type. - -Fixed-Precision ---------------- - -### `DECIMAL` - -> A fixed precision decimal number. Precision up to 38 digits is supported but performance is best up to 18 digits. -> -> The name `NUMERIC` and `DEC` is also available for this type. -> -> The decimal type takes two literal parameters: -> -> - **precision** - total number of digits -> - **scale** - number of digits in fractional part. Scale is optional and defaults to 0. -> -> Example type definitions: `DECIMAL(10,3)`, `DECIMAL(20)` -> -> Example literals: `DECIMAL '10.3'`, `DECIMAL '1234567890'`, `1.1` -> -> -> **Note** -> -> For compatibility reasons decimal literals without explicit type specifier (e.g. `1.2`) are treated as the values of the `DOUBLE` type by default, but this is subject to change in future releases. This -> behavior is controlled by: -> -> - System wide property: `parse-decimal-literals-as-double` -> - Session wide property: `parse_decimal_literals_as_double` - -String ------- - -### `VARCHAR` - -> Variable length character data with an optional maximum length. -> -> The name `STRING` is also available for this type. Please be advised, `STRING` defined an unlimited length character data, you should not specify the length, otherwise it will be same as `VARCHAR(length)`. -> -> Example type definitions: `varchar`, `varchar(20)`, `string` -> -> SQL statements support simple literal, as well as Unicode usage: -> -> - literal string : `'Hello winter !'` -> - Unicode string with default escape character: `U&'Hello winter \2603 !'` -> - Unicode string with custom escape character: `U&'Hello winter #2603 !' UESCAPE '#'` -> -> A Unicode string is prefixed with `U&` and requires an escape -> character before any Unicode character usage with 4 digits. In the -> examples above `\2603` and `#2603` represent a snowman character. Long -> Unicode codes with 6 digits require usage of the plus symbol before -> the code. For example, you need to use `\+01F600` for a grinning face -> emoji. - -### `CHAR` - -> Fixed length character data. A `CHAR` type without length specified has a default length of 1. A `CHAR(x)` value always has `x` characters. For instance, casting `dog` to `CHAR(7)` adds 4 implicit trailing spaces. Leading and trailing spaces are included in comparisons of `CHAR` values. As a result, two character values with different lengths (`CHAR(x)` and `CHAR(y)` where `x != y`) will never -> be equal. -> -> Example type definitions: `char`, `char(20)` - -### `VARBINARY` - -> Variable length binary data. -> -> -> **Note** -> -> Binary strings with length are not yet supported: `varbinary(n)` -> - -### `JSON` - -> JSON value type, which can be a JSON object, a JSON array, a JSON number, a JSON string, `true`, `false` or `null`. - -Date and Time -------------- - -See also `/language/timestamp` - -### `DATE` - -> Calendar date (year, month, day). -> -> Example: `DATE '2001-08-22'` - -### `TIME` - -> Time of day (hour, minute, second, millisecond) without a time zone. Values of this type are parsed and rendered in the session time zone. -> -> Example: `TIME '01:02:03.456'` - -### `TIME WITH TIME ZONE` - -> Time of day (hour, minute, second, millisecond) with a time zone. Values of this type are rendered using the time zone from the value. -> -> Example: `TIME '01:02:03.456 America/Los_Angeles'` - -### `TIMESTAMP` - -> Instant in time that includes the date and time of day without a time zone. Values of this type are parsed and rendered in the session time zone. -> -> Example: `TIMESTAMP '2001-08-22 03:04:05.321'` - -### `TIMESTAMP WITH TIME ZONE` - -> Instant in time that includes the date and time of day with a time zone. Values of this type are rendered using the time zone from the value. -> -> Example: `TIMESTAMP '2001-08-22 03:04:05.321 America/Los_Angeles'` - -### `INTERVAL YEAR TO MONTH` - -> Span of years and months. -> -> Example: `INTERVAL '3' MONTH` - -### `INTERVAL DAY TO SECOND` - -> Span of days, hours, minutes, seconds and milliseconds. -> -> Example: `INTERVAL '2' DAY` - -Structural ----------- - -### `ARRAY` - -> An array of the given component type. -> -> Example: `ARRAY[1, 2, 3]` - -### `MAP` - -> A map between the given component types. -> -> Example: `MAP(ARRAY['foo', 'bar'], ARRAY[1, 2])` - -### `ROW` - -> A structure made up of fields that allows mixed types. The fields may be of any SQL type. -> -> By default, row fields are not named, but names can be assigned. -> -> Example: `CAST(ROW(1, 2.0) AS ROW(x BIGINT, y DOUBLE))` -> -> Named row fields are accessed with field reference operator `.`. -> -> Example: `CAST(ROW(1, 2.0) AS ROW(x BIGINT, y DOUBLE)).x` -> -> Named or unnamed row fields are accessed by position with the subscript operator `[]`. The position starts at `1` and must be a constant. -> -> Example: `ROW(1, 2.0)[1]` - -Network Address ---------------- - -### `IPADDRESS` - -> An IP address that can represent either an IPv4 or IPv6 address. Internally, the type is a pure IPv6 address. Support for IPv4 is handled using the *IPv4-mapped IPv6 address* range ([RFC 4291#section-2.5.5.2](https://tools.ietf.org/html/rfc4291.html#section-2.5.5.2 )). When creating an `IPADDRESS`, IPv4 addresses will be mapped into that range. When formatting an `IPADDRESS`, any address within the mapped range will be formatted as an IPv4 address. Other addresses will be formatted as IPv6 using the canonical format defined in [RFC 5929](https://tools.ietf.org/html/rfc5952.html ). -> -> Examples: `IPADDRESS '10.0.0.1'`, `IPADDRESS '2001:db8::1'` - -UUID ----- - -### `UUID` - -> This type represents a UUID (Universally Unique IDentifier), also known as a GUID (Globally Unique IDentifier), using the format defined in [RFC 4122](https://tools.ietf.org/html/rfc4122.html ). -> -> Example: `UUID '12151fd2-7586-11e9-8f9e-2a86e4085a59'` - -HyperLogLog ------------ - -Calculating the approximate distinct count can be done much more cheaply than an exact count using the -[HyperLogLog](https://en.wikipedia.org/wiki/HyperLogLog) data sketch. See [HyperLogLog Functions](../functions/hyperloglog.html). - -### `HyperLogLog` - -> A HyperLogLog sketch allows efficient computation of [approx_distinct()](../functions/aggregate.html). It starts as a sparse representation, switching to a dense representation when it becomes more efficient. - -### `P4HyperLogLog` - -> A P4HyperLogLog sketch is similar to `hyperloglog_type`, but it starts (and remains) in the dense representation. - -Quantile Digest ---------------- - -### `QDigest` - -> A quantile digest (qdigest) is a summary structure which captures the approximate distribution of data for a given input set, and can be queried to retrieve approximate quantile values from the distribution. -> The level of accuracy for a qdigest is tunable, allowing for more precise results at the expense of space. -> -> A qdigest can be used to give approximate answer to queries asking for what value belongs at a certain quantile. A useful property of qdigests is that they are additive, meaning they can be merged together without losing precision. -> -> A qdigest may be helpful whenever the partial results of `approx_percentile` can be reused. For example, one may be interested in a daily reading of the 99th percentile values that are read over -> the course of a week. Instead of calculating the past week of data with `approx_percentile`, `qdigest`s could be stored daily, and quickly merged to retrieve the 99th percentile value. ++++ +weight = 1 +title = "数据类型" ++++ + +数据类型 +========== + +openLooKeng有一组内置的数据类型，如下所述。可能通过插件提供更多的类型。 + + +**注意** + +*连接器不需要支持所有的数据类型。连接器支持的数据类型，请参见[连接器](../connector.html)文档。* + +布尔类型 +------- + +### `BOOLEAN` + +> 捕获布尔值`true`和`false`。 + +整数类型 +------- + +### `TINYINT` + +> 一个8比特带符号的2补码整数，最小值为`-2^7`，最大值为`2^7 - 1`。 + +### `SMALLINT` + +> 一个16比特带符号的2补码整数，最小值为`-2^15`，最大值为`2^15 - 1`。 + +### `INTEGER` + +> 一个32比特带符号的2补码整数，最小值为`-2^31`，最大值为`2^31 - 1`。该类型也被称为`INT`。 + +### `BIGINT` + +> 一个64比特带符号的2补码整数，最小值为`-2^63`，最大值为`2^63 - 1`。 + +浮点类型 +-------------- + +### `REAL` + +> 一种不精确的可变精度的32比特的数据类型，实现IEEE 754标准定义的二进制浮点运算。 +> +> 此类型也被称为`FLOAT`。 +> +> 例如: REAL '10.3', REAL '10.3e0', REAL '1.03e1' + +### `DOUBLE` + +> 一种不精确的可变精度的64比特的数据类型，它实现了IEEE 754标准定义的二进制浮点运算。 +> +> 该类型也被称为`DOUBLE PRECISION`。 + +固定精度类型 +--------------- + +### `DECIMAL` + +> 固定精度的十进制数。精度最高可达38位，但性能最高可达18位。 +> +>该类型也被称为`NUMERIC`和`DEC`。 +> +> 此数据类型有两个参数： +> +> - **precision** - 表示总位数。 +> - **scale** - 小数位数。Scale可选，默认为0。 +> +> 类型定义示例：`DECIMAL(10,3)`, `DECIMAL(20)` +> +> 文本示例：`DECIMAL '10.3'`, `DECIMAL '1234567890'`, `1.1` +> +> +> **注意** +> +>由于兼容性原因，没有显式类型说明符的十进制文字（例如 `1.2`）默认作为`DOUBLE`类型的值处理。但在以后的版本中也可能会有变化。此行为受以下控制： + +> +> - 系统属性: `parse-decimal-literals-as-double` +> - 会话属性: `parse_decimal_literals_as_double` + +字符串类型 +------ + +### `VARCHAR` + +> 可变长度字符数据，最大长度可选。 +> +> 该类型也被称为`STRING`。请注意，`STRING`定义的是一个无限长的字符数据，您不能指定长度，否则它就成为`VARCHAR(length)`类型了。 +> +> 类型定义示例：`varchar`, `varchar(20)`, `string` +> +> SQL语句支持简单文字和Unicode使用： +> +> - 文本字符串: `'Hello winter !'` +> - 含默认转义字符的Unicode字符串：`U&'Hello winter \2603 !'` +> - 含自定义转义字符的Unicode字符串：`U&'Hello winter #2603 !'UESCAPE '#'` +> +> Unicode字符串以`U&`为前缀，任何4位Unicode字符前都需要转义符。 +> 在上例中，`\2603`和`#2603`代表雪人符号。 +> 对于6位的Unicode长编码，需要在代码前使用一个加号。例如，您需要在笑脸表情前使用`\+01F600`。 + + +### `CHAR` + +> 固定长度字符。对于不指定长度的`CHAR`类型，默认长度为1。一个`CHAR(x)`包含`x`个字符。例如，将`dog`转换为`CHAR(7)`会增加4个隐式尾部空格。在比较`CHAR`值时包括前导和尾部空格。因此，两个不同长度的字符值（`CHAR(x）`和`CHAR(y)`，其中`x != y`)永远不会相等。 + +> +> 类型定义示例：`char`, `char(20)` + +### `VARBINARY` + +> 可变长度二进制数据。 +> +> +> **注意** +> +> 目前仍不支持带长度的二进制字符：`varbinary(n)` +> + +### `JSON` + +> JSON数值类型，可以是JSON对象、JSON数组、JSON数字、JSON字符串、`true`、`false`或`null`。 + +日期和时间类型 +------------- + +另见[timestamp](timestamp.html)。 + +### `DATE` + +> 日历日期（年、月、日） +> +> 例如: `DATE '2001-08-22'` + +### `TIME` + +> 不带时区的时间（时、分、秒、毫秒）此类型的值将在会话时区中解析和呈现。 +> +> 例如: `TIME '01:02:03.456'` + +### `TIME WITH TIME ZONE` + +> 带时区的时间（时、分、秒、毫秒）。此类型的值将使用该值中的时区进行呈现。 +> +> 例如: `TIME '01:02:03.456 America/Los_Angeles'` + +### `TIMESTAMP` + +> 包含日期和时间的即时时间，不包含时区。此类型的值将在会话时区中解析和呈现。 +> +> 例如: `TIMESTAMP '2001-08-22 03:04:05.321'` + +### `TIMESTAMP WITH TIME ZONE` + +> 即时时间，包括日期、时间和时区。此类型的值将使用该值中的时区进行呈现。 +> +> 例如: `TIMESTAMP '2001-08-22 03:04:05.321 America/Los_Angeles'` + +### `INTERVAL YEAR TO MONTH` + +> 年和月的跨度。 +> +> 例如: `INTERVAL '3' MONTH` + +### `INTERVAL DAY TO SECOND` + +> 天数、小时、分钟、秒和毫秒的跨度。 +> +> 例如: `INTERVAL '2' DAY` + +结构类型 +---------- + +### `ARRAY` + +> 指定组件类型的数组。 +> +> 例如: `ARRAY[1, 2, 3]` + +### `MAP` +> 指定组件类型之间的映射。 +> +> 例如: `MAP(ARRAY['foo', 'bar'], ARRAY[1, 2])` + +### `ROW` + +> 由混合类型的字段组成的结构。字段可以是任何SQL类型。 +> +> 行字段默认不命名，但可以指定名称。 +> +> 例如: `CAST(ROW(1, 2.0) AS ROW(x BIGINT, y DOUBLE))` +> +> 已命名的行字段通过字段引用运算符`.`访问。 +> +> 例如: `CAST(ROW(1, 2.0) AS ROW(x BIGINT, y DOUBLE)).x` +> +> 已命名或未命名的行字段通过下标运算符`[]`按位置访问。位置从`1`开始且必须是常量。 +> +> 例如: `ROW(1, 2.0)[1]` + +网络地址 +--------------- + +### `IPADDRESS` + +> 可表示IPv4地址或IPv6地址。对内为纯IPv6地址。可通过*IPv4-mapped IPv6 address* range ([RFC 4291#section-2.5.5.2](https://tools.ietf.org/html/rfc4291.html#section-2.5.5.2 ))来支持对IPv4地址的处理。在创建`IPADDRESS`时，IPv4地址将映射到指定的范围。格式化`IPADDRESS`时，在映射范围内的任何地址都会被格式化为IPv4地址。其他地址将使用[RFC 5929] (https://tools.ietf.org/html/rfc5952.html)中定义的规范格式格式化为IPv6地址。 +> +> 例如: `IPADDRESS '10.0.0.1'`, `IPADDRESS '2001:db8::1'` + +UUID +---- + +### `UUID` + +> 此类型表示UUID（通用唯一标识符），也称为GUID（全局唯一标识符），使用[RFC 4122](https://tools.ietf.org/html/rfc4122.html)中定义的格式。 +> +> 例如: `UUID '12151fd2-7586-11e9-8f9e-2a86e4085a59'` + +HyperLogLog +----------- + +计算近似的非重复计数比使用[HyperLogLog](https://en.wikipedia.org/wiki/HyperLogLogLog)数据草图进行精确计数成本低得多。 +请参见 [HyperLogLog Functions](../functions/hyperloglog.html)。 + +### `HyperLogLog` + +> HyperLogLog 草图可高效地计算 [approx_distinct()](../functions/aggregate.html)。它开始时是稀疏表示，当效率提高时，就切换到密集表示。 + +### `P4HyperLogLog` + +> P4HyperLogLog草图类似于`hyperloglog_type`，但它从始至终都采用密集表示形式。 + +分位点摘要 +--------------- + +### `QDigest` + +> 分位点摘要（qdigest）是一种摘要结构，它捕捉指定输入集的数据的近似分布，并且可以通过查询从分布中检索近似分位点值。 +> qdigest的准确程度是可调的，使更精确的结果会占用更多空间。 +> +> 对于在某一分位数处属于什么值的查询，可用qdigest提供近似回答。qdigest的一个有用的特性是它们是可加的，这意味着它们可以合并在一起而不损失精度。 +> +> 当`approx_percentile`的部分结果可以重用时，qdigest就会发挥更大作用。例如，人们可能对在一周内每天读取的第99百分位数值感兴趣。 +这种情况下，与其使用`approx_percentile`计算过去一周的数据，不如使用`qdigest`。`qdigest`可以每天存储，并快速合并以检索第99个百分位值。 diff --git a/hetu-docs/zh/optimizer/cost-based-optimizations.md b/hetu-docs/zh/optimizer/cost-based-optimizations.md index 5cc736f21..f99c2871d 100644 --- a/hetu-docs/zh/optimizer/cost-based-optimizations.md +++ b/hetu-docs/zh/optimizer/cost-based-optimizations.md @@ -1,52 +1,53 @@ -Cost based optimizations ++++ +weight = 3 +title = "基于成本的优化" ++++ + +基于成本的优化 ======================== -openLooKeng supports several cost based optimizations, described below. +openLooKeng支持多种基于成本的优化，如下所述。 -Join Enumeration +Join枚举 ---------------- -The order in which joins are executed in a query can have a significant impact on the query\'s performance. The aspect of join ordering that has the largest impact on performance is the size of the data being processed and transferred over the network. If a join that produces a lot of data is performed early in the execution, then subsequent stages will need to process large amounts of data for longer than necessary, -increasing the time and resources needed for the query. +在查询中执行join的顺序会对查询的性能产生重大影响。join排序方面对性能影响最大的是网络处理和传输的数据的大小。如果一个产生大量数据的join在早期执行，那么后续的阶段需要处理大量数据的时间将长于需要的时间, 这样会增加查询所需的时间和资源。 -With cost based join enumeration, openLooKeng uses `/optimizer/statistics` provided by connectors to estimate the costs for different join orders and automatically pick the join order with the lowest computed costs. +使用基于成本的join枚举，openLooKeng使用连接器提供的`/optimizer/statistics`来评估不同join排序的成本，并自动选择计算成本最低的join排序。 -The join enumeration strategy is governed by the `join_reordering_strategy` session property, with the -`optimizer.join-reordering-strategy` configuration property providing the default value. +join枚举策略由`join_reordering_strategy`会话属性控制，其中`optimizer.join-reordering-strategy`配置属性提供默认值。 -The valid values are: +有效值如下： -- `AUTOMATIC` (default) - full automatic join enumeration enabled -- `ELIMINATE_CROSS_JOINS` - eliminate unnecessary cross joins -- `NONE` - purely syntactic join order +- `AUTOMATIC`（默认值） -启用全自动join枚举 +- `ELIMINATE_CROSS_JOINS` -消除不必要的交叉join +- `NONE` -纯句法join顺序 -If using `AUTOMATIC` and statistics are not available, or if for any other reason a cost could not be computed, the `ELIMINATE_CROSS_JOINS` strategy is used instead. +如果使用`AUTOMATIC`但没有统计数据，或由于任何其他原因成本无法计算，则改用`ELIMINATE_CROSS_JOINS`策略。 -Join Distribution Selection +Join分布选择 --------------------------- -openLooKeng uses a hash based join algorithm. That implies that for each join operator a hash table must be created from one join input (called build side). The other input (probe side) is then iterated and for each row the hash table is queried to find matching rows. +openLooKeng使用基于哈希的join算法。这意味着对于每个join操作符，必须从一个join输入（称为构建侧）创建哈希表。然后，迭代另一个输入（探针侧），并查询哈希表以找到匹配的行。 -There are two types of join distributions: +有两种类型的join分布： -- Partitioned: each node participating in the query builds a hash table from only fraction of the data -- Broadcast: each node participating in the query builds a hash table from all of the data (data is replicated to each node) +- 分区模式：参与查询的每个节点仅从部分数据构建哈希表 +- 广播模式：参与查询的每个节点从所有数据构建一个哈希表（数据复制到每个节点） -Each type have their trade offs. Partitioned joins require redistributing both tables using a hash of the join key. This can be slower (sometimes substantially) than broadcast joins, but allows much larger joins. In particular, broadcast joins will be faster if the build side is much smaller than the probe side. However, broadcast joins require that the tables on the build side of the join after filtering fit in memory on each node, whereas distributed joins only need to fit in distributed memory across all nodes. +这两种类型各有利弊。分区join要求使用join键的散列重分布这两个表。这使得分区join比广播join慢（有时慢很多），但允许更大的join。特别是，当构建端比探测端小得多时，广播join将更快。但是，广播join要求join的构建端上的表在过滤后适合每个节点上的内存，而分布式join只需要适合所有节点上的分布式内存。 -With cost based join distribution selection, openLooKeng automatically chooses to use a partitioned or broadcast join. With cost based join enumeration, openLooKeng automatically chooses which side is the probe and -which is the build. +使用基于成本的join分布选择，openLooKeng自动选择使用分区join或广播join。使用基于成本的join枚举，openLooKeng自动选择哪一侧是探针侧，哪一侧是构建侧。 -The join distribution strategy is governed by the `join_distribution_type` session property, with the -`join-distribution-type` configuration property providing the default value. +join分发策略由`join_distribution_type`会话属性控制，其中`join-distribution-type`配置属性提供默认值。 -The valid values are: +有效值如下： -- `AUTOMATIC` (default) - join distribution type is determined automatically for each join - - `BROADCAST` - broadcast join distribution is used for all joins - - PARTITIONED` - partitioned join distribution is used for all join +- `AUTOMATIC`（默认值） -自动为每个join确定join分布类型 +- `BROADCAST` -对所有join使用广播join分布 +- `PARTITIONED` -对所有join使用分区join分布 -Connector Implementations +连接器实现 ------------------------- -In order for the openLooKeng optimizer to use the cost based strategies, the connector implementation must provide `statistics`. +为了使openLooKeng优化器使用基于成本的策略，连接器实现必须提供`statistics`。 diff --git a/hetu-docs/zh/optimizer/cost-in-explain.md b/hetu-docs/zh/optimizer/cost-in-explain.md index 4379c3ef4..2154280c4 100644 --- a/hetu-docs/zh/optimizer/cost-in-explain.md +++ b/hetu-docs/zh/optimizer/cost-in-explain.md @@ -1,29 +1,35 @@ -Cost in EXPLAIN -=============== ++++ +weight = 2 +title = "EXPLAIN成本" ++++ -During planning, the cost associated with each node of the plan is computed based on the table statistics for the tables in the query. This calculated cost is printed as part of the output of an [EXPLAIN](../sql/explain.html) statement. - -Cost information is displayed in the plan tree using the format `{rows: XX (XX), cpu: XX, memory: XX, network: XX}`. `rows` refers to the expected number of rows output by each plan node during execution. The value in the parentheses following the number of rows refers to the expected size of the data output by each plan node in bytes. Other parameters indicate the estimated amount of CPU, memory, and network utilized by the execution of a plan node. These values do not represent any actual unit, but are numbers that are used to compare the relative costs between plan nodes, allowing the optimizer to choose the best plan for executing a query. If any of the values is not known, a `?` is printed. - - - -For example: - -``` sql -lk:default> EXPLAIN SELECT comment FROM tpch.sf1.nation WHERE nationkey > 3; - -- Output[comment] => [[comment]] - Estimates: {rows: 22 (1.69kB), cpu: 6148.25, memory: 0.00, network: 1734.25} - - RemoteExchange[GATHER] => [[comment]] - Estimates: {rows: 22 (1.69kB), cpu: 6148.25, memory: 0.00, network: 1734.25} - - ScanFilterProject[table = tpch:nation:sf1.0, filterPredicate = ("nationkey" > BIGINT '3')] => [[comment]] - Estimates: {rows: 25 (1.94kB), cpu: 2207.00, memory: 0.00, network: 0.00}/{rows: 22 (1.69kB), cpu: 4414.00, memory: 0.00, network: 0.00}/{rows: 22 (1.69kB), cpu: 6148.25, memory: 0.00, network: 0.00} - nationkey := tpch:nationkey - comment := tpch:comment -``` - -Generally, there is only one cost printed for each plan node. However, when a `Scan` operator is combined with a `Filter` and/or `Project` operator, then multiple cost structures will be printed, each corresponding to an individual logical part of the combined operator. For example, three cost structures will be printed for a `ScanFilterProject` operator, corresponding to the `Scan`, `Filter`, and `Project` parts of the operator, in that order. - - - -Estimated cost is also printed in [explain-analyze](../sql/explain-analyze.html) in addition to actual runtime statistics. \ No newline at end of file +EXPLAIN成本 +=============== + +在计划期间，与计划的每个节点相关联的开销将根据查询中的表的表统计信息计算。此计算的成本将作为[EXPLAIN](../sql/explain.html)语句的输出的一部分打印出来。 + +在计划树中，成本信息以`{rows: XX (XX), cpu: XX, memory: XX, network: XX}`格式呈现。`rows`是指每个计划节点在执行过程中期望输出的行数。行数后面括号中的值表示每个计划节点期望输出的数据大小（以字节为单位）。其他参数表示计划节点执行时预计占用的CPU、内存、网络等资源。这些值并不代表任何实际的单位，而是用于比较计划节点之间的相对成本，使优化器能够选择执行查询的最佳计划。如果其中任何一个值未知，则打印`?`。 + + + +例如： + +``` sql +lk:default> EXPLAIN SELECT comment FROM tpch.sf1.nation WHERE nationkey > 3; + +- Output[comment] => [[comment]] + Estimates: {rows: 22 (1.69kB), cpu: 6148.25, memory: 0.00, network: 1734.25} + - RemoteExchange[GATHER] => [[comment]] + Estimates: {rows: 22 (1.69kB), cpu: 6148.25, memory: 0.00, network: 1734.25} + - ScanFilterProject[table = tpch:nation:sf1.0, filterPredicate = ("nationkey" > BIGINT '3')] => [[comment]] + Estimates: {rows: 25 (1.94kB), cpu: 2207.00, memory: 0.00, network: 0.00}/{rows: 22 (1.69kB), cpu: 4414.00, memory: 0.00, network: 0.00}/{rows: 22 (1.69kB), cpu: 6148.25, memory: 0.00, network: 0.00} + nationkey := tpch:nationkey + comment := tpch:comment +``` + + +通常情况下，每个计划节点只打印一份成本。但是，当`Scan`运算符与`Filter`和/或`Project`运算符组合使用时，将打印多个成本结构，每个成本结构对应组合运算符的一个逻辑部分。例如，`ScanFilterProject`运算符将打印三个成本结构，分别对应`Scan`、`Filter`和`Project`部分。 + + + +除了实际运行时统计数据外，预估成本也在[explain-analyze](../sql/explain-analyze.html)中打印。 diff --git a/hetu-docs/zh/optimizer/statistics.md b/hetu-docs/zh/optimizer/statistics.md index d8cd56a6a..135f48660 100644 --- a/hetu-docs/zh/optimizer/statistics.md +++ b/hetu-docs/zh/optimizer/statistics.md @@ -1,42 +1,46 @@ -Table Statistics -================ ++++ +weight = 1 +title = "表统计" ++++ -openLooKeng supports statistics based optimizations for queries. For a query to take advantage of these optimizations, openLooKeng must have statistical information for the tables in that query. - -Table statistics are provided to the query planner by connectors. Currently, the only connector that supports statistics is the [hive](../en/connector/hive.html). - -Table Layouts -------------- - -- Statistics are exposed to the query planner by a table layout. A table layout represents a subset of a table’s data and contains information about the organizational properties of that data (like sort order and bucketing). - - - - The number of table layouts available for a table and the details of those table layouts are specific to each connector. Using the Hive connector as an example: - - - - - Non-partitioned tables have just one table layout representing all data in the table - - Partitioned tables have a family of table layouts. Each set of partitions to be scanned represents one table layout. openLooKeng will try to pick a table layout consisting of the smallest number of partitions based on filtering predicates from the query. - -Available Statistics --------------------- - -The following statistics are available in openLooKeng: - - - -> - For a table: -> - **row count**: the total number of rows in the table layout -> - For each column in a table: -> - **data size**: the size of the data that needs to be read -> - **nulls fraction**: the fraction of null values -> - **distinct value count**: the number of distinct values -> - **low value**: the smallest value in the column -> - **high value**: the largest value in the column - - - -The set of statistics available for a particular query depends on the connector being used and can also vary by table or even by table layout. For example, the Hive connector does not currently provide statistics on data size. - -Table statistics can be displayed via the openLooKeng SQL interface using the [show-stats](../sql/show-stats.html) command. For the Hive connector, refer to the [Hive connector](../en/connector/hive#hive_analyze.html) documentation to learn how to update table statistics. \ No newline at end of file +表统计 +================ + +openLooKeng支持基于统计信息的查询优化。为了让查询利用这些优化，openLooKeng必须提供该查询中表的统计信息。 + +表统计信息由连接器提供给查询计划器。目前仅[hive connector](../connector/hive.html)连接器支持统计。 + +表格布局 +------------- + +- 统计信息通过表布局向查询计划器呈现。一个表布局表示表数据的子集，并包含有关该数据的组织属性（例如排序顺序和分桶）的信息。 + + + + 可用于表的表布局的数量和表布局的详细信息是每个连接器特有的。以Hive连接器为例： + + + - 非分区表只有一个表布局，表示表中的所有数据 + - 分区表拥有一系列表布局。每一组要扫描的分区代表一个表布局。openLooKeng将根据查询中的筛选谓词，尝试选择由最少数量的分区组成的表布局。 + +可用统计 +-------------------- + +openLooKeng提供以下统计数据： + + + +> - 对于表： +> - **row count**：表示表布局的总行数。 +> - 对于表格中的每一列： +> - **data size**：表示需要读取的数据大小 +> - **nulls fraction**：表示空值的部分 +> - **distinct value count**：表示独特值的个数。 +> - **low value**：表示列中最小值。 +> - **high value**：表示列中最大值 + + + +可用于特定查询的统计集取决于所使用的连接器，也因表甚至表布局而异。例如，Hive连接器目前不提供数据大小的统计信息。 + +可以通过openLooKeng SQL接口使用[show-stats](../sql/show-stats.html)命令查询表统计信息。Hive连接器，请参考[Hive Connector](../connector/hive.html)文档了解如何更新表的统计信息。 diff --git a/hetu-docs/zh/rest/node.md b/hetu-docs/zh/rest/node.md index 6cc6cc9cf..9eb4ac05d 100644 --- a/hetu-docs/zh/rest/node.md +++ b/hetu-docs/zh/rest/node.md @@ -1,29 +1,24 @@ -Node Resource ++++ +weight = 1 +title = "节点资源" ++++ + +节点资源 ============= - GET /v1/node -Returns a list of nodes known to a openLooKeng Server. This call doesn\'t -require a query parameter of headers, it simply returns an array with -each known node in a openLooKeng installation. +返回openLooKeng服务器已知的节点列表。此调用不需要标头的查询参数，它只返回一个数组，数组中包含openLooKeng安装中的每个已知节点。 -In the response, the values `recentRequests`, `recentFailures`, and -`recentSuccesses` are decaying counters set to decay exponentially over -time with a decay parameter of one minute. This decay rate means that if -a openLooKeng node has 1000 successes in a few seconds, this statistic value -will drop to 367 in one minute. +响应消息中，`recentRequests`、`recentFailures`和`recentSuccesses`的取值是随时间指数衰减的计数器，衰减参数为1分钟。这个衰减率意味着，如果一个openLooKeng节点在几秒内有1000次成功，那么这个统计值在一分钟内就会下降到367次。 -`age` shows you how long a particular node has been running, and uri -points you to the HTTP server for a particular node. The last request -and last response times show you how recently a node has been used. +`age`显示一个节点的运行时长，uri指向该节点的HTTP服务器。最后一次请求和响应时间显示了一个节点的最近使用情况。 -The following example response displays a single node which has not -experienced any failure conditions. Each node also reports statistics -about traffic uptime, and failures. +下面的响应示例显示一个未经历了任何故障条件的节点。各节点同时上报流量正常、故障等统计数据。 -**Example response**: +**响应样例**： > ``` http > HTTP/1.1 200 OK @@ -42,81 +37,72 @@ about traffic uptime, and failures. > "recentFailureRatio":0.0, > "recentFailuresByType":{} > } -> ``` > -> \] +> ] +> ``` -If a node is experiencing errors, you\'ll see a response that looks like -the following. Here we have a node which has experienced a spate of -errors. The recentFailuresByType field lists the Java exception which -have occurred recently on a particular node. -**Example response with Errors**: +如果一个节点发生故障，您会看到如下所示的响应。此例中，一个节点发生了一连串的错误。recentFailuresByType字段列出了节点上发生的Java异常。 + +**错误响应样例**： > ``` http > HTTP/1.1 200 OK -> ``` > -> Vry: Accept Content-Type: text/javascript +> Vary: Accept Content-Type: text/javascript > -> \[ +> [ > -> { +> { > -> ​ \"age\": \"4.45m\", \"lastFailureInfo\": { \"message\": ->​ \"Connect Timeout\", \"stack\": \[ -> ​ \"org.eclipse.jetty.io.ManagedSelector\$ConnectTimeout.run(ManagedSelector.java:683)\", -> ​ \.... \"java.lang.Thread.run(Thread.java:745)\" \], -> ​ \"suppressed\": \[\], \"type\": -> ​ \"java.net.SocketTimeoutException\" }, \"lastRequestTime\": -> ​ \"2017-08-05T11:53:00.647Z\", \"lastResponseTime\": -> ​ \"2017-08-05T11:53:00.647Z\", \"recentFailureRatio\": -> ​ 0.47263053472046446, \"recentFailures\": 2.8445543205610617, -> ​ \"recentFailuresByType\": { -> ​ \"java.net.SocketTimeoutException\": 2.8445543205610617 }, -> ​ \"recentRequests\": 6.018558073577414, \"recentSuccesses\": -> ​ 3.1746446343010297, \"uri\": \"\" +> ​ "age": "4.45m", "lastFailureInfo": { "message": +>​ "Connect Timeout", "stack": [ +> ​ "org.eclipse.jetty.io.ManagedSelector$ConnectTimeout.run(ManagedSelector.java:683)", +> ​ .... "java.lang.Thread.run(Thread.java:745)" ], +> ​ "suppressed": [], "type": +> ​ "java.net.SocketTimeoutException" }, "lastRequestTime": +> ​ "2017-08-05T11:53:00.647Z", "lastResponseTime": +> ​ "2017-08-05T11:53:00.647Z", "recentFailureRatio": +> ​ 0.47263053472046446, "recentFailures": 2.8445543205610617, +> ​ "recentFailuresByType": { +> ​ "java.net.SocketTimeoutException": 2.8445543205610617 }, +> ​ "recentRequests": 6.018558073577414, "recentSuccesses": +> ​ 3.1746446343010297, "uri": "" > -> } +> } > -> \] - +> ] +> ``` - GET /v1/node/failed -Calling this service returns a JSON document listing all the nodes that -have failed the last heartbeat check. The information returned by this -call is the same as the information returned by the previous service. +调用此服务将返回一个JSON文档，其中列出了所有上次心跳检测失败的节点。由此调用返回的信息与前一个业务返回的信息相同。 -**Example response**: +**响应样例**： -> ``` http +> ``` json > -> ``` -> -> \[ -> +> [ > { > -> \"age\": \"1.37m\", \"lastFailureInfo\": { \"message\": -> \"Connect Timeout\", \"stack\": \[ -> \"org.eclipse.jetty.io.ManagedSelector\$ConnectTimeout.run(ManagedSelector.java:683)\", -> \..... -> \"java.util.concurrent.ThreadPoolExecutor\$Worker.run(ThreadPoolExecutor.java:617)\", -> \"java.lang.Thread.run(Thread.java:745)\" \], \"suppressed\": -> \[\], \"type\": \"java.net.SocketTimeoutException\" }, -> \"lastRequestTime\": \"2017-08-05T11:52:42.647Z\", -> \"lastResponseTime\": \"2017-08-05T11:52:42.647Z\", -> \"recentFailureRatio\": 0.22498784153043677, -> \"recentFailures\": 20.11558290058638, -> \"recentFailuresByType\": { -> \"java.net.SocketTimeoutException\": 20.11558290058638 }, -> \"recentRequests\": 89.40742203558189, \"recentSuccesses\": -> 69.30583024727453, \"uri\": \"\" +> "age": "1.37m", "lastFailureInfo": { "message": +> "Connect Timeout", "stack": [ +> "org.eclipse.jetty.io.ManagedSelector$ConnectTimeout.run(ManagedSelector.java:683)", +> ... +> "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)", +> "java.lang.Thread.run(Thread.java:745)" ], "suppressed": +> [], "type": "java.net.SocketTimeoutException" }, +> "lastRequestTime": "2017-08-05T11:52:42.647Z", +> "lastResponseTime": "2017-08-05T11:52:42.647Z", +> "recentFailureRatio": 0.22498784153043677, +> "recentFailures": 20.11558290058638, +> "recentFailuresByType": { +> "java.net.SocketTimeoutException": 20.11558290058638 }, +> "recentRequests": 89.40742203558189, "recentSuccesses": +> 69.30583024727453, "uri": "" > -> } -> -> \] -> +> } +> ] +> ``` diff --git a/hetu-docs/zh/rest/query.md b/hetu-docs/zh/rest/query.md index 6f8041240..e945fded8 100644 --- a/hetu-docs/zh/rest/query.md +++ b/hetu-docs/zh/rest/query.md @@ -1,31 +1,27 @@ -Query Resource ++++ +weight = 2 +title = "查询资源" ++++ + +查询资源 ============== -The Query REST service is the most complex of the rest services. It -contains detailed information about nodes, and other details that -capture the state and history of a query being executed on a openLooKeng -installation. +Query REST服务是rest服务中最复杂的。它包含节点的详细信息，以及其它在openLooKeng上执行的查询的状态和历史的详细信息。 - GET /v1/query -This service returns information and statistics about queries that are -currently being executed on a openLooKeng coordinator. +此服务返回有关当前在openLooKeng协调器上执行的查询的信息和统计信息。 -When you point a web browser at a openLooKeng coordinate you\'ll see a rendered -version of the output from this service which will display recent -queries that have executed on a openLooKeng installation. +当你将浏览器指向一个openLooKeng坐标时，你会看到此服务输出的一个渲染的版本，将显示最近的在openLooKeng上执行的查询。 - GET /v1/query/{queryId} -If you are looking to gather very detailed statistics about a query, -this is the service you would call. If you load the web interface of a -openLooKeng coordinator you will see a list of current queries. Clicking on a -query will reveal a link to this service. +可调用此服务收集详细的有关查询的统计数据。如果您加载openLooKeng协调器的Web界面，您会看到一个关于当前查询的清单。单击查询将显示指向此服务的链接。 -**Example response**: +**响应样例**： -> ``` http +> ``` json > { > "queryId" : "20131229_211533_00017_dk5x2", > "session" : { diff --git a/hetu-docs/zh/rest/stage.md b/hetu-docs/zh/rest/stage.md index 3aabc0819..0e76a953f 100644 --- a/hetu-docs/zh/rest/stage.md +++ b/hetu-docs/zh/rest/stage.md @@ -1,10 +1,16 @@ -Stage Resource ++++ +weight = 3 +title = "阶段资源" ++++ + +阶段资源 ============== - GET /v1/stage -Returns detail about a stage in a openLooKeng query. + +返回openLooKeng查询中有关阶段的详细信息。 - DELETE /v1/stage/{stageId} -Deletes a stage in a openLooKeng query. +删除openLooKeng查询中的指定阶段。 diff --git a/hetu-docs/zh/rest/statement.md b/hetu-docs/zh/rest/statement.md index 7d9b2b086..af270290c 100644 --- a/hetu-docs/zh/rest/statement.md +++ b/hetu-docs/zh/rest/statement.md @@ -1,113 +1,103 @@ -Statement Resource ++++ +weight = 4 +title = "语句资源" ++++ + +语句资源 ================== - POST /v1/statement query query -: SQL Query to execute +: 需要执行的SQL查询 reqheader X-Presto-User -: User to execute statement on behalf of (optional) + +: 代表(optional)执行语句的用户 reqheader X-Presto-Source -: Source of query +: 查询的来源 reqheader X-Presto-Catalog -: Catalog to execute query against +: 执行查询基于的目录 reqheader X-Presto-Schema -: Schema to execute query against +: 执行查询的模式 -Submits a statement to openLooKeng for execution. The openLooKeng client executes -queries on behalf of a user against a catalog and a schema. When you run -a query with the openLooKeng CLI, it is calling out to the statement resource -on the openLooKeng coordinator. +提交语句给openLooKeng执行。openLooKeng客户端代表用户根据指定的目录和架构进行查询。当使用openLooKeng命令行进行查询时，则调用openLooKeng协调器的语句资源。 -The request to the statement resource is the SQL query to execute as a -post along with the standard X-Presto-Catalog, X-Presto-Source, -X-Presto-Schema, and X-Presto-User headers. +对语句资源的请求就是和标准X-Presto-Catalog、X-Presto-Source、X-Presto-Schema和X-Presto-User头域一起作为post执行的SQL查询。 -The response from the statement resource contains a query identifier -which can be used to gather detailed information about a query. This -initial response also includes information about the stages that have -been created to execute this query on openLooKeng workers. Every query has a -root stage and the root stage is given a stage identifier of \"0\" as -shown in the following example response. +语句资源的响应包含一个查询标识符，该标识符可用于收集有关查询的详细信息。初始响应还包括有关为执行此查询而在openLooKeng worker上创建的阶段的信息。每个查询都有一个根阶段，并且根阶段被赋予一个\"0\"的阶段标识符，如下面的响应示例所示。 -This root stage aggregates the responses from other stages running on -openLooKeng workers and delivers them to the client via the openLooKeng coordinator. -When a client receives a response to this POST it will contain a -\"nextUri\" property which directs the client to query this address for -additional results from the query. +这个根阶段聚合了在openLooKeng worker上运行的其他阶段的响应，并通过openLooKeng协调器将它们传递给客户端。 +当一个客户端收到这个POST的响应时，它将包含一个\"nextUri\"属性，这个属性指示客户端查询这个地址以获得查询的更多结果。 -**Example request**: +**请求样例**： > ``` http > POST /v1/statement HTTP/1.1 -> ``` > > Host: localhost:8001 X-Presto-Catalog: jmx X-Presto-Source: presto-cli > X-Presto-Schema: jmx User-Agent: StatementClient/0.55-SNAPSHOT > X-Presto-User: tobrie1 Content-Length: 41 > -> select name from \"java.lang:type=runtime\" +> select name from "java.lang:type=runtime" +> ``` -**Example response**: +**响应样例**： > ``` http > HTTP/1.1 200 OK -> ``` > > Content-Type: application/json X-Content-Type-Options: nosniff > Transfer-Encoding: chunked > > { > -> : \"id\":\"20140108\_110629\_00011\_dk5x2\", -> \"infoUri\":\"\", -> \"partialCancelUri\":\"\", -> \"nextUri\":\"\", -> \"columns\": \[ { \"name\":\"name\", \"type\":\"varchar\" } \], -> \"stats\": { \"state\":\"RUNNING\", \"scheduled\":false, -> \"nodes\":1, \"totalSplits\":0, \"queuedSplits\":0, -> \"runningSplits\":0, \"completedSplits\":0, \"cpuTimeMillis\":0, -> \"wallTimeMillis\":0, \"processedRows\":0, \"processedBytes\":0, -> \"rootStage\": { \"stageId\":\"0\", \"state\":\"SCHEDULED\", -> \"done\":false, \"nodes\":1, \"totalSplits\":0, -> \"queuedSplits\":0, \"runningSplits\":0, \"completedSplits\":0, -> \"cpuTimeMillis\":0, \"wallTimeMillis\":0, \"processedRows\":0, -> \"processedBytes\":0, \"subStages\": \[ { \"stageId\":\"1\", -> \"state\":\"SCHEDULED\", \"done\":false, \"nodes\":1, -> \"totalSplits\":0, \"queuedSplits\":0, \"runningSplits\":0, -> \"completedSplits\":0, \"cpuTimeMillis\":0, \"wallTimeMillis\":0, -> \"processedRows\":0, \"processedBytes\":0, \"subStages\":\[\] } \] +> "id":"20140108_110629_00011_dk5x2", +> "infoUri":"", +> "partialCancelUri":"", +> "nextUri":"", +> "columns": [ { "name":"name", "type":"varchar" } ], +> "stats": { "state":"RUNNING", "scheduled":false, +> "nodes":1, "totalSplits":0, "queuedSplits":0, +> "runningSplits":0, "completedSplits":0, "cpuTimeMillis":0, +> "wallTimeMillis":0, "processedRows":0, "processedBytes":0, +> "rootStage": { "stageId":"0", "state":"SCHEDULED", +> "done":false, "nodes":1, "totalSplits":0, +> "queuedSplits":0, "runningSplits":0, "completedSplits":0, +> "cpuTimeMillis":0, "wallTimeMillis":0, "processedRows":0, +> "processedBytes":0, "subStages": [ { "stageId":"1", +> "state":"SCHEDULED", "done":false, "nodes":1, +> "totalSplits":0, "queuedSplits":0, "runningSplits":0, +> "completedSplits":0, "cpuTimeMillis":0, "wallTimeMillis":0, +> "processedRows":0, "processedBytes":0, "subStages":[] } ] > } } > > } +> ``` + - GET /v1/statement/{queryId}/{token} + query queryId -: The query identifier returned from the initial POST to /v1/statement +: 初始POST返回给/v1/statement的查询标识 query token -: The token returned from the initial POST to /v1/statement or from a - previous call to this same call +: 初始POST返回给/v1/statement的令牌或先前调用返回给此调用的令牌。 +当openLooKeng客户端提交语句执行时，openLooKeng创建一个查询，然后它向客户端返回一个nextUri。此调用与nextUri调用相对应，可以包含正在进行中的查询的状态更新，也可以将最终结果传递给客户端。 -When a openLooKeng client submits a statement for execution, openLooKeng creates a -query and then it returns a nextUri to the client. This call corresponds -to that nextUri call and can contain either a status update for a query -in progress or it can deliver the final results to the client. - -**Example request**: +**请求样例**： > ``` http > GET /v1/statement/20140108_110629_00011_dk5x2/1 HTTP/1.1 @@ -115,52 +105,55 @@ in progress or it can deliver the final results to the client. > User-Agent: StatementClient/0.55-SNAPSHOT > ``` -**Example response**: +**响应样例**： > ``` http > HTTP/1.1 200 OK -> ``` > > Content-Type: application/json X-Content-Type-Options: nosniff Vary: > Accept-Encoding, User-Agent Transfer-Encoding: chunked > -> 383 { \"id\":\"20140108\_110629\_00011\_dk5x2\", -> \"infoUri\":\"\", -> \"columns\": \[ { \"name\":\"name\", \"type\":\"varchar\" } \], -> \"data\": \[ \[\"<4165@domU-12-31-39-0F-CC-72>\"\] \], \"stats\": { -> \"state\":\"FINISHED\", \"scheduled\":true, \"nodes\":1, -> \"totalSplits\":2, \"queuedSplits\":0, \"runningSplits\":0, -> \"completedSplits\":2, \"cpuTimeMillis\":1, \"wallTimeMillis\":4, -> \"processedRows\":1, \"processedBytes\":27, \"rootStage\": { -> \"stageId\":\"0\", \"state\":\"FINISHED\", \"done\":true, \"nodes\":1, -> \"totalSplits\":1, \"queuedSplits\":0, \"runningSplits\":0, -> \"completedSplits\":1, \"cpuTimeMillis\":0, \"wallTimeMillis\":0, -> \"processedRows\":1, \"processedBytes\":32, \"subStages\": \[ { -> \"stageId\":\"1\", \"state\":\"FINISHED\", \"done\":true, \"nodes\":1, -> \"totalSplits\":1, \"queuedSplits\":0, \"runningSplits\":0, -> \"completedSplits\":1, \"cpuTimeMillis\":0, \"wallTimeMillis\":4, -> \"processedRows\":1, \"processedBytes\":27, \"subStages\":\[\] } \] } +> 383 { "id":"20140108_110629_00011_dk5x2", +> "infoUri":"", +> "columns": [ { "name":"name", "type":"varchar" } ], +> "data": [ ["<4165@domU-12-31-39-0F-CC-72>"] ], "stats": { +> "state":"FINISHED", "scheduled":true, "nodes":1, +> "totalSplits":2, "queuedSplits":0, "runningSplits":0, +> "completedSplits":2, "cpuTimeMillis":1, "wallTimeMillis":4, +> "processedRows":1, "processedBytes":27, "rootStage": { +> "stageId":"0", "state":"FINISHED", "done":true, "nodes":1, +> "totalSplits":1, "queuedSplits":0, "runningSplits":0, +> "completedSplits":1, "cpuTimeMillis":0, "wallTimeMillis":0, +> "processedRows":1, "processedBytes":32, "subStages": [ { +> "stageId":"1", "state":"FINISHED", "done":true, "nodes":1, +> "totalSplits":1, "queuedSplits":0, "runningSplits":0, +> "completedSplits":1, "cpuTimeMillis":0, "wallTimeMillis":4, +> "processedRows":1, "processedBytes":27, "subStages":[] } ] } > } } - +> ``` + - DELETE /v1/statement/{queryId}/{token} query queryId -: The query identifier returned from the initial POST to /v1/statement + + +: 初始POST返回给/v1/statement的查询标识 reqheader X-Presto-User -: User to execute statement on behalf of (optional) +: 代表(optional)执行语句的用户 reqheader X-Presto-Source -: Source of query +: 查询来源 reqheader X-Presto-Catalog -: Catalog to execute query against + +: 执行查询基于的目录 reqheader X-Presto-Schema -: Schema to execute query against +: 执行查询的模式 diff --git a/hetu-docs/zh/rest/task.md b/hetu-docs/zh/rest/task.md index 49bad82d2..d4c2db76e 100644 --- a/hetu-docs/zh/rest/task.md +++ b/hetu-docs/zh/rest/task.md @@ -1,28 +1,23 @@ -Task Resource ++++ +weight = 5 +title = "任务资源" ++++ + +任务资源 ============= -The Task resource provides a set of REST endpoints that give openLooKeng -servers the ability to communicate about tasks and task output. This -isn\'t a service that will be used by end users, but it supports the -execution of queries on a openLooKeng installation. +任务资源提供了一组REST端点，这些端点使openLooKeng服务器能够就任务和任务输出进行交流。这不是一个供最终用户使用的服务，但它支持在openLooKeng上执行查询任务。 - GET /v1/task -Returns information about all tasks known to a openLooKengServer. +返回openLooKeng服务器已知的所有任务的信息。 -Note that the output of a call to `/v1/task` can be quite large. If you -execute this against a busy openLooKeng server the response received will -include a listing of every task known to that server along with detailed -statistics about operators and drivers. +注意，对`/v1/task`的调用的输出可能相当大。如果你对忙碌的openLooKeng服务器执行这个命令，收到的响应将包括该服务器已知的每个任务的列表以及详细的操作符和驱动的统计信息。 +为了便于本手册描述，下列的示例响应经过适当的删减。一个高负荷的openLooKeng服务器实际的响应输出可能有很多页。下面是一个状态为`CANCELED`的任务的`taskId`。 -The following example response shows a trivial task response that has -been truncated to fit this manual. A real response from a busy openLooKeng -server would generate pages and pages of output. Here there is a -`taskId` for a task which is in the `CANCELED` state. +**响应样例**： -**Example response**: - -``` http +``` json [ { "taskId" : "20131222_183944_00011_dk5x2.1.0", "version" : 9223372036854775807, @@ -67,15 +62,14 @@ server would generate pages and pages of output. Here there is a - DELETE /v1/task/{taskId} -Deletes a given task from a openLooKeng server. +从openLooKeng服务器删除指定的任务。 -- - GET /v1/task/{taskId} -Retrieves information about a specific task by `taskId`. +- GET /v1/task/{taskId} -The following example lists the output of a task. It contains the -following high-level sections: +通过`taskId`检索指定任务的信息。 + +下面是一个任务的输出结果示例。它包含以下为高级别部分： - `outputBuffers` - `noMoreSplits` @@ -83,13 +77,11 @@ following high-level sections: - `failures` - `outputs` -This is the same output that is also present in the response from the -Query resource which lists all of the stages and tasks involved in a -particular query. This is call is used by openLooKeng to coordinate a queries. +查询资源的响应中也有相同的输出。该响应列出了特定查询中涉及的所有阶段和任务。openLooKeng使用此调用协调查询。 -**Example response**: +**响应样例**： -``` http +``` json { "taskId" : "20140115_170528_00004_dk5x2.0.0", "version" : 42, @@ -306,12 +298,13 @@ particular query. This is call is used by openLooKeng to coordinate a queries. ``` + - GET /v1/task/{taskId}/results/{outputId}/{token} -This service is used by openLooKeng to retrieve task output. +openLooKeng通过此服务检索任务输出。 - DELETE /v1/task/{taskId}/results/{outputId} +openLooKeng通过此服务删除任务输出。 -This service is used by openLooKeng to delete task output.