!1531 Fixed 0 row count issue while recovering

Merge pull request !1531 from Kishore Kumar M V/0rows_after_recovery

hetu-docs/en/admin/properties.md

@@ -378,8 +378,6 @@ Exchanges transfer data between openLooKeng nodes for different stages of a quer
 >
 > Maximum size of a response returned from an exchange request. The response will be placed in the exchange client buffer which is shared across all concurrent requests for the exchange.
 >
->
->
 > Increasing the value may improve network throughput if there is high latency. Decreasing the value may improve query performance for large clusters as it reduces skew due to the exchange client buffer holding responses for more tasks (rather than hold more data from fewer tasks).
 
 ### `sink.max-buffer-size`
@@ -1005,6 +1003,14 @@ helps with cache affinity scheduling.
 >
 > Enables Kryo based serialization for snapshot, instead of default java serializer.
 
+### `experimental.eliminate-duplicate-spill-files`
+
+> -   **Type:** `boolean`
+> -   **Default value:** `false`
+>
+> Enables elimination of duplicate spill files storage as part of snapshot capture.
+
+
 
 ## HTTP Client Configurations
 
@@ -1027,3 +1033,12 @@ helps with cache affinity scheduling.
 > After the configured time elapsed and no response received, then client connection consider that to be failure in submission of request.
 >
 > (Note: this parameter should be configured with higher time when in high load environment)
+
+## Connector Properties configuration
+
+### `case-insensitive-name-matching`
+>
+> -    **Type:** `boolean`
+> -    **Default value:** `false`
+>
+> Case-insensitive matching between database and collection names. The default is case sensitive.

hetu-docs/en/admin/reliable-execution.md

@@ -71,4 +71,4 @@ Additionally, while query is in progress number of capturing snapshots and id of
 
 ## Configurations
 
-Configurations related to recovery framework feature can be found in [Properties Reference](properties.md#Recovery framework).
+Configurations related to recovery framework feature can be found in [Properties Reference](properties.md#Query Recovery).

hetu-docs/en/connector/Hudi.md

@@ -1,5 +1,8 @@
 # Hudi  Connector
 
+### Release Notes
+Currently Hudi only supports version 0.7.0.
+
 ### Hudi Introduction
 
 Apache Hudi is a fast growing data lake storage system that helps organizations build and manage petabyte-scale data lakes. Hudi enables storing vast amounts of data on top of existing DFS compatible storage while also enabling stream processing in addition to typical batch-processing. This is made possible by providing two new primitives. Specifically,

hetu-docs/en/develop/_index.md

@@ -1,5 +1,5 @@
 
 # Developer Guide
 
-openLooKeng is based on Trino(formerly known as PrestoSQL), and has been forked from the Trino open source project. openLooKeng has additional optimizations, and enhanced features to allow in-situ analytics on any data, anywhere, including geographically remote data sources. This guide is intended for openLooKeng contributors and plugin developers.
+openLooKeng is based on Trino 316(formerly known as PrestoSQL), and has been forked from the Trino open source project. openLooKeng has additional optimizations, and enhanced features to allow in-situ analytics on any data, anywhere, including geographically remote data sources. This guide is intended for openLooKeng contributors and plugin developers.
 

hetu-docs/en/faq/faq.md

@@ -40,6 +40,10 @@
    >
    > openLooKeng Bilibili channel:  https://space.bilibili.com/627629884   
 
+8. Which version of Trino is openLooKeng developed on?
+
+   > Based on Trino 316 version development.
+
 ## Functions
 
 1. What connectors does the openLooKeng support?

hetu-docs/en/security/ranger-access-control.md

@@ -7,7 +7,7 @@ Overview
 
 Apache Ranger delivers a comprehensive approach to security for a Hadoop cluster. It provides a centralized platform to define, administer and manage security policies consistently across Hadoop components. Check [Apache Ranger Wiki](https://cwiki.apache.org/confluence/display/RANGER/Index) for detail introduction and user guide.
 
-[openlookeng-ranger-plugin](https://gitee.com/openlookeng/openlookeng-ranger-plugin) is a ranger plugin for openLooKeng to enable, monitor and manage comprehensive data security.
+[openlookeng-ranger-plugin](https://gitee.com/openlookeng/openlookeng-ranger-plugin) is developed based on Ranger 2.1.0, which is a ranger plugin for openLooKeng to enable, monitor and manage comprehensive data security.
 
 Build Process
 -------------------------

hetu-docs/zh/admin/properties.md

@@ -115,6 +115,20 @@
 > 
 > 此属性是在JVM堆中为openLooKeng不跟踪的分配留作裕量/缓冲区的内存量。
 
+### `query.suspend-query-enabled`
+
+> -   **类型：** `boolean`
+> -   **默认值：** `false`
+>
+> 系统资源不足时，临时挂起运行中的查询。
+
+### `query.max-suspended-queries`
+
+> -   **类型：** `integer`
+> -   **默认值：** `10`
+>
+> 终止查询之前，查询挂起尝试的最大次数。仅当`query.suspend-query-enabled`设置为`true`时，此属性才生效。
+
 ## 溢出属性
 
 ### `experimental.spill-enabled`
@@ -340,35 +354,124 @@
 > 
 > 如果网络延迟较高，增大该值可以提高网络吞吐量。减小该值可以提高大型集群的查询性能，因为它减少了由于交换客户端缓冲区保存了较多任务（而不是保存较少任务中的较多数据）的响应而导致的倾斜。
 
-### `exchange.max-error-duration`
-
-> - **类型：** `duration`
-> - **最小值：** `1m`
-> - **默认值：** `7m`
-> 
-> 交换错误最大缓冲时间，超过该时限则查询失败。
-
-### `exchange.is-timeout-failure-detection-enabled`
-
-> -   **类型：** `boolean`
-> -   **默认值：** `true`
->
-> 正在使用的故障检测机制。默认值是基于超时的故障检测。但是，当该属性设置为`false`时，启用基于最大重试次数的故障检测机制。
-
-### `exchange.max-retry-count`
-
-> -   **类型：** `integer`
-> -   **默认值：** `10`
->
-> Coordinator在将失败任务视为永久失败之前对其执行的最大重试次数。仅当`exchange.is-timeout-failure-detection-enabled`设置为`false`时，才使用此属性。
-
 ### `sink.max-buffer-size`
 
 > - **类型：** `data size`
 > - **默认值：** `32MB`
 > 
-> 上游任务等待拉取任务数据的输出缓冲区大小。如果任务输出是经过哈希分区的，那么缓冲区将在所有分区的使用者之间共享。如果网络延迟较高或集群中有多个节点，增加此值可以提高在阶段之间传输的数据的网络吞吐量。
+>等待上游任务拉取的任务数据的输出缓冲区大小。如果任务输出是哈希分区的，则缓冲区将在所有分区的消费者之间共享。如果网络延迟高或集群中有许多节点，则增加此值可以提高阶段之间传输数据的网络吞吐量。
  
+ ## 故障恢复处理属性
+ 
+ ### 失败重试策略
+ 
+ ### `failure.recovery.retry.profile`
+ 
+ > -   **类型：** `string`
+ > -   **默认值：** `default`
+ >
+ > 此属性定义用于确定HTTP客户端上是否发生故障的故障检测配置文件。此属性的值`<profile-name>`必须对应`etc/failure-retry-policy/`路径中的`<profile-name>.properties`文件。如果没有此类配置文件可用，并且未设置此属性，则使用“default”配置文件。
+ > 例如，`failure.recovery.retry.profile="test"`要求`test.properties`文件存在于`etc/failure-retry-policy`路径中。
+ > `test.properties`文件必须包含指定的`failure.recovery.retry.type`。
+ 
+ 
+ ### `failure.recovery.retry.type`
+ 
+ > -   **类型：** `string`
+ > -   **默认值：** `timeout`
+ >
+ > 此属性用来设置正在使用的故障检测机制。默认值是基于`timeout`的故障检测。
+ 
+ #### 基于`timeout`的故障检测
+ 
+ > 如果使用此机制，HTTP客户端故障将在指定时间段内重试，重试失败则被视为永久故障。
+ >
+ > 可以为此类故障检测定义`max.error.duration`属性。
+ 
+ #### 基于`max-retry`的故障检测
+ 
+ > 如果使用此机制，HTTP客户端故障将在被视为永久故障之前重试指定次数。
+ > 可以为此类故障检测定义`max.retry.count`和`max.error.duration`属性。
+ > 在这种类型的故障检测中，在查询故障检测模块之前，会执行`max.retry.count`次重试。当故障检测器模块检测到远程节点发生故障时，HTTP客户端将此故障视为永久故障。否则，例如，当远程工作节点处于活动状态但没有响应时，在`max.error.duration`指定的时间段内重试，重试失败则被视为永久故障。
+ 
+ ### `max.error.duration`
+ 
+ > -   **类型：** `duration`
+ > -   **默认值：** `300s`
+ >
+ > 被视为永久故障前，协调器等待解决任务间相关错误的最长时间。
+ 
+ ### `max.retry.count`
+ 
+ > -   **类型：** `integer`
+ > -   **默认值：** `100`
+ >
+ > 协调器在向故障检测器模块查询远程节点状态之前，对失败任务执行的最大重试次数。
+ > 此属性指定查询失败检测模块之前的最小重试次数。因此，实际故障数量可能会因为集群大小和集群负载而略有不同。
+ > 此属性仅用于基于`max-retry`的故障检测配置文件。
+ > 最小值为100。
+ 
+ ### 故障检测Gossip协议配置
+ 
+ ### `failure-detection-protocol`
+ 
+ >- **类型：** `string`
+ >- **默认值：** `heartbeat`
+ >
+ >此属性定义正在使用的故障检测器的类型。默认配置为`heartbeat`故障检测器。
+ >在`config.properties`文件中，将此属性配置为`gossip`，可以启用Gossip协议。
+ >集群中的所有节点（即协调器和工作节点）都应在其各自的`etc/config.properties`文件中指定此属性。
+ 
+ ### `failure-detector.heartbeat-interval`
+ 
+ >- **类型：** `duration`
+ >- **默认值：** `500ms` （500毫秒）
+ >
+ >集群中两个节点之间的消息散播间隔。
+ >在Gossip协议中，两个工作节点间的消息散播频率高于协调器和一个工作节点间。
+ >在协调器的`config.properties`文件中，可以为此属性配置一个较大的值，例如`5s`（5秒）。
+ >在工作节点中，可以使用默认值。
+ 
+ ### `failure-detector.worker-gossip-probe-interval`
+ 
+ >- **类型：** `duration`
+ >- **默认值：** `5s`（5秒）
+ >
+ >Gossip协议使用监控任务（与`heartbeat`故障检测器相同）来监控其他节点。
+ >此属性指定监控任务刷新间隔，以触发工作节点消息散播。
+ >仅可以为工作节点指定默认值以外的任何其他值。
+ >该属性的值必须大于`failure-detector.heartbeat-interval`的值。
+ 
+ ### `failure-detector.coordinator-gossip-probe-interval`
+ 
+ >- **类型：** `duration`
+ >- **默认值：** `5s`（5秒）
+ >
+ >Gossip协议使用监控任务（与heartbeat故障检测器相同）来监控其他节点。
+ >此属性指定监控任务刷新间隔，以触发协调器参与工作节点消息散播。
+ >仅可以为协调器指定默认值以外的任何其他值。
+ >该属性的值必须大于`failure-detector.heartbeat-interval`和`failure-detector.worker-gossip-probe-interval`的值。
+ 
+ ### `failure-detector.coordinator-gossip-collate-interval`
+ 
+ >- **类型：** `duration`
+ >- **默认值：** `2s`（2秒）
+ >
+ >此属性指定协调器整理从所有工作节点获得的所有散播消息的间隔。
+ >此属性只支持为协调器配置。
+ >该属性的值必须大于`failure-detector.heartbeat-interval`的值。
+ 
+ ### `failure-detector.gossip-group-size`
+ 
+ >- **类型：** `integer`
+ >- **默认值：** `Integer.MAX_VALUE`
+ >
+ >此属性定义单个工作节点在集群中散播消息的工作节点数量。
+ >任何大于集群大小（即工作节点数量）的值都意味着all-to-all消息散播。
+ >要保持较低的网络开销，针对大型集群，请将此属性设置为一个较小的值（例如，100工作节点的集群设置为10）。
+ >每次刷新协调器上的工作节点监视任务时，协调器都会定义工作节点URI列表，其大小由`failure-detector.gossip-group-size`指定，以触发worker-to-worker消息散播。
+ >
+>
 ## 任务属性
 
 ### `task.concurrency`
@@ -612,7 +715,7 @@
 
 ## 启发式索引属性
 
-启发式索引是外部索引模块，可用于过滤连接器级别的行。 位图，Bloom和MinMaxIndex是openLooKeng提供的索引列表。 到目前为止，位图索引支持使用ORC存储格式的表支持蜂巢连接器。
+启发式索引是外部索引模块，可用于过滤连接器级别的行。 位图，Bloom和MinMaxIndex是openLooKeng提供的索引列表。 到目前为止，位图索引支持Hive连接器的ORC存储格式的表。
 
 ### `hetu.heuristicindex.filter.enabled`
 
@@ -795,18 +898,25 @@
 > 
 > 远程任务错误最大缓冲时间，超过该时限则查询失败。
 
-## 分布式快照
+## 查询恢复
+
+### `recovery_enabled`
+
+> -   **类型：** `boolean`
+> -   **默认值：** `false`
+>
+> 此会话属性用于启用或禁用恢复框架，该框架在发生故障时启用或禁用查询重启/恢复。
 
 ### `snapshot_enabled`
 
-> - 类型：`boolean`
+> -   **类型：** `boolean`
-> - **默认值**：`false`
+> -   **默认值：** `false`
 >
-> 此会话属性用于启用或禁用分布式快照功能。
+> 启用恢复框架时，启用此会话属性可以在查询执行期间捕获快照。如果未启用恢复框架，则此属性不生效。
 
 ### `hetu.experimental.snapshot.profile`
 
-> - 类型：`string`
+> - **类型：**`string`
 >
 > 此属性定义用于存储快照的[文件系统](../develop/filesystem.md)配置文件。对应的配置文件必须存在于`etc/filesystem`中。例如，如果将该属性设置为`hetu.experimental.snapshot.profile=snapshot-hdfs1`，则必须在`etc/filesystem`中创建描述此文件系统的配置文件`snapshot-hdfs1.properties`，其中包含的必要信息包括身份验证类型、配置和密钥表（如适用）。具体细节请参考[文件系统](../develop/filesystem.md)相关章节。
 >
@@ -816,21 +926,21 @@
 
 ### `hetu.recovery.maxRetries`
 
-> - 类型：`int`
+> - **类型：** `integer`
-> - **默认值**：`10`
+> - **默认值：** `10`
 >
-> 此属性定义查询的错误恢复尝试的最大次数。达到限制时，查询失败。
+> 此属性定义查询错误恢复尝试的最大次数。当达到限制时，查询失败。
 >
-> 也可以使用`recovery_max_retries`会话属性在每个查询基础上指定。
+> 也可以使用`recovery_max_retries`会话属性为每个查询指定此属性。
 
 ### `hetu.recovery.retryTimeout`
 
 > - **类型：** `duration`
 > - **默认值：** `10m`（10分钟）
 >
-> 此属性定义系统等待所有任务成功恢复的最大时长。如果在此超时时限内任何任务未就绪，则认为恢复失败，查询将尝试从较早快照恢复（如果可用）。
+> 此属性定义系统等待所有任务成功恢复的最长时间。如果在此时间内有任何任务未就绪，则恢复尝试将被视为失败，查询将尝试从较早的快照恢复（如果可用）。
 >
-> 也可以使用`recovery_retry_timeout`会话属性在每个查询基础上指定。
+> 也可以使用`recovery_retry_timeout`会话属性为每个查询指定此属性。
 
 ### `hetu.snapshot.useKryoSerialization`
 
@@ -860,3 +970,12 @@
 > 当超过所配置时间，客户端没有接收到任何响应，则视为客户端的请求提交失败。
 >
 > （注意： 建议在高负载环境下，该参数配置大一点。）
+
+## 连接器属性配置
+
+### `case-insensitive-name-matching`
+
+> -   **类型:** `boolean`
+> -   **默认值:** `false`
+>
+> 不区分大小写匹配数据库和集合名称，默认区分大小写。

hetu-docs/zh/admin/reliable-execution.md

@@ -10,9 +10,9 @@
 
 自版本1.2.0起，openLooKeng支持恢复任务和工作节点故障。
 
-## 启用分布式快照
+## 启用恢复框架
      
-分布式快照适用于长时间运行的查询任务。该功能默认为禁用状态，可以通过会话属性[`snapshot_enabled`](properties.md#snapshot_enabled)启用或禁用。建议仅在对可靠性要求高的复杂查询场景下启用该功能。
+恢复框架对于长时间运行的查询最有用。默认禁用，可以使用会话属性[`recovery_enabled`](properties.md#recovery_enabled)启用和禁用恢复框架。建议仅对可靠性要求高的复杂查询启用该功能。
 
 ## 要求
 
@@ -35,7 +35,7 @@
 
 ## 检测
 
-协调节点与远程任务之间的通信长时间失败时，将触发错误恢复，由[`query.remote-task.max-error-duration`](properties.md#queryremote-taskmax-error-duration)配置控制。
+当协调器与远程任务之间的通信长时间失败时，将触发错误恢复，由[`故障恢复处理属性`](properties.md#故障恢复处理属性)配置控制。
 
 ## 存储注意事项
 
@@ -53,18 +53,20 @@
 
 从错误和快照中恢复需要成本。捕获快照需要时间，时间长短取决于复杂性。因此，需要在性能和可靠性之间进行权衡。
 
-建议仅在必要时启用分布式快照，如运行时间较长的查询任务。对于这些类型的工作负载，捕获快照的开销可以忽略不计。
+建议在必要时打开快照捕获，例如对于长时间运行的查询。对于这些类型的工作负载，拍摄快照的开销可以忽略不计。
 
 ## 快照统计信息
 
 在调试模式下启动CLI时，快照捕获信息和恢复信息将与查询结果一起显示在CLI中。
 
-快照捕获信息包括查询期间捕获的快照大小、捕获快照的CPU时间和捕获快照的挂钟时间。分别显示所有快照和最后一个快照的捕获信息。
+快照捕获统计信息包括捕获的快照数量、捕获的快照大小、捕获快照所需的CPU时间和在查询期间捕获快照所需的挂钟时间。所有快照和最后一个快照的统计信息会分别显示。
 
 快照恢复信息包括查询期间从快照恢复的次数、加载用于恢复的快照大小、从快照恢复所需的CPU时间和从快照恢复所需的挂钟时间。仅当查询期间发生恢复时，才会显示恢复信息。
 
-![](../images/snapshot_statistics.png)
+此外，在查询正在进行时，将显示捕获的快照数量和恢复的快照的ID。更多详细信息，见下图。
+
+![](../images/snapshot_statistics_cn.png)
 
 ## 配置
 
-与分布式快照功能相关的配置可参见[属性参考](properties.md#分布式快照)。
+与恢复框架功能相关的配置，请参见[属性参考](properties.md#查询恢复)。

hetu-docs/zh/admin/spill.md

@@ -57,6 +57,8 @@ openLooKeng将溢出路径视为独立的磁盘（参见[JBOD](https://en.wikipe
 
 通过这种机制，联接操作符使用的峰值内存可以降低到最大构建表分区的大小。假设没有数据倾斜，这个值将是整个构建表大小的`1 / task.concurrency`倍。
 
+注意：spill-to-disk不支持交叉连接。
+
 ### 聚合
 
 聚合函数对一组值执行操作并返回一个值。如果要聚合的组数量很大，可能需要大量内存。当启用溢出到磁盘时，如果没有足够的内存，则中间累积的聚合结果将写入磁盘。结果被重新加载回来，并以较低的内存占用量合并。

hetu-docs/zh/connector/Hudi.md

@@ -1,5 +1,8 @@
 # Hudi连接器
 
+### 版本说明
+目前Hudi只支持0.7.0版本。
+
 ### Hudi介绍
 
 Apache Hudi是一个快速迭代的数据湖存储系统，可以帮助企业构建和管理PB级数据湖。它提供在DFS上存储超大规模数据集，同时使得流式处理如果批处理一样，该实现主要是通过如下两个原语实现。

hetu-docs/zh/develop/_index.md

@@ -1,4 +1,4 @@
 
 # 开发者指南
 
-openLooKeng基于Trino(之前称为PrestoSQL)，是Trino开源项目的一个分支。与Trino开源项目相比，openLooKeng提供了额外的优化和增强功能，可以在任何位置对任何数据（包括远程数据源）进行现场分析。本指南适用于openLooKeng参与者和插件开发人员。
+openLooKeng基于Trino 316版本(之前称为PrestoSQL)，是Trino开源项目的一个分支。与Trino开源项目相比，openLooKeng提供了额外的优化和增强功能，可以在任何位置对任何数据（包括远程数据源）进行现场分析。本指南适用于openLooKeng参与者和插件开发人员。

hetu-docs/zh/faq/faq.md

@@ -35,6 +35,10 @@
    > openLooKeng微信群：请添加openLooKeng小助手微信号：openLooKengoss，小助手会拉您入群  
    > openLooKeng B站： https://space.bilibili.com/627629884 
 
+8. openLooKeng基于Trino哪个版本进行开发？
+
+   > 基于Trino 316版本进行开发。
+
 ## 功能
 
 1. openLooKeng目前支持哪些连接器?

hetu-docs/zh/security/ranger-access-control.md

@@ -7,7 +7,7 @@ Ranger 访问控制
 
 Apache Ranger 为 Hadoop 集群提供了一种全面的安全框架，以跨组件的、一致性的方式进行定义、授权、管理安全策略。Ranger 详细介绍和用户指导可以参考[Apache Ranger Wiki](https://cwiki.apache.org/confluence/display/RANGER/Index )。
 
-[openlookeng-ranger-plugin](https://gitee.com/openlookeng/openlookeng-ranger-plugin) 是为 openLooKeng 开发的 Ranger 插件，用于全面的数据安全监控和权限管理。
+[openlookeng-ranger-plugin](https://gitee.com/openlookeng/openlookeng-ranger-plugin) 基于Ranger 2.1.0版本进行开发，是为 openLooKeng 开发的 Ranger 插件，用于全面的数据安全监控和权限管理。
 
 编译过程
 -------------------------

hetu-hazelcast/pom.xml

@@ -17,7 +17,7 @@
         <jline.version>3.21.0</jline.version>
         <hazelcast.version>5.1</hazelcast.version>
         <aspectjweaver.version>1.9.4</aspectjweaver.version>
-        <spring-context.version>5.2.19.RELEASE</spring-context.version>
+        <spring-context.version>5.3.20</spring-context.version>
         <cache-api.version>1.1.1</cache-api.version>
         <jmockit.version>1.38</jmockit.version>
     </properties>

hetu-hbase/pom.xml

@@ -14,7 +14,7 @@
 
     <properties>
         <air.main.basedir>${project.parent.basedir}</air.main.basedir>
-        <dep.log4j.version>2.17.1</dep.log4j.version>
+        <dep.log4j.version>2.17.2</dep.log4j.version>
         <version.annotations>13.0</version.annotations>
         <version.validation-api>2.0.1.Final</version.validation-api>
         <version.commons-lang3>3.10</version.commons-lang3>
@@ -249,7 +249,7 @@
         <dependency>
             <groupId>org.eclipse.jetty</groupId>
             <artifactId>jetty-webapp</artifactId>
-            <version>9.4.43.v20210629</version>
+            <version>9.4.46.v20220331</version>
             <scope>runtime</scope>
         </dependency>
         <dependency>

hetu-hive-functions/src/main/java/io/hetu/core/hive/dynamicfunctions/utils/HiveTypeTranslator.java

@@ -197,7 +197,7 @@ public class HiveTypeTranslator
             if (charLength <= HiveChar.MAX_CHAR_LENGTH) {
                 return getCharTypeInfo(charLength);
             }
-            throw new PrestoException(NOT_SUPPORTED, String.format("Unsupported Hive type: %s." +
+            throw new PrestoException(NOT_SUPPORTED, String.format(Locale.ROOT, "Unsupported Hive type: %s." +
                             " Supported CHAR types: CHAR(<=%d).",
                     type, HiveChar.MAX_CHAR_LENGTH));
         }

hetu-metastore/src/main/java/io/hetu/core/metastore/jdbc/MetadataTableDao.java

@@ -154,7 +154,7 @@ public interface MetadataTableDao
             "  source varchar(30) NOT NULL,\n" +
             "  queryId varchar(50) NOT NULL,\n" +
             "  resource varchar(30) NOT NULL,\n" +
-            "  query varchar(1000) NOT NULL,\n" +
+            "  query varchar(10000) NOT NULL,\n" +
             "  state varchar(30) NOT NULL,\n" +
             "  failed varchar(30) DEFAULT NULL,\n" +
             "  createTime varchar(30) DEFAULT NULL,\n" +

hetu-mongodb/src/main/java/io/hetu/core/plugin/mongodb/MongoClientConfig.java

@@ -19,6 +19,7 @@ import com.mongodb.MongoCredential;
 import com.mongodb.ServerAddress;
 import io.airlift.configuration.Config;
 import io.airlift.configuration.DefunctConfig;
+import io.prestosql.spi.function.Mandatory;
 
 import javax.validation.constraints.Min;
 import javax.validation.constraints.NotNull;
@@ -90,6 +91,10 @@ public class MongoClientConfig
         return seeds;
     }
 
+    @Mandatory(name = "mongodb.seeds",
+            description = "List of mongodb servers",
+            defaultValue = "ip:port",
+            required = true)
     @Config("mongodb.seeds")
     public MongoClientConfig setSeeds(String commaSeparatedList)
     {

hetu-mongodb/src/main/java/io/hetu/core/plugin/mongodb/MongoPlugin.java

@@ -17,12 +17,24 @@ import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableSet;
 import io.prestosql.spi.Plugin;
 import io.prestosql.spi.connector.ConnectorFactory;
+import io.prestosql.spi.function.ConnectorConfig;
+import io.prestosql.spi.queryeditorui.ConnectorUtil;
+import io.prestosql.spi.queryeditorui.ConnectorWithProperties;
 import io.prestosql.spi.type.Type;
 
+import java.util.Arrays;
+import java.util.Optional;
 import java.util.Set;
 
 import static io.hetu.core.plugin.mongodb.ObjectIdType.OBJECT_ID;
 
+/**
+ * hetu plugin to use mongodb as a data source.
+ */
+@ConnectorConfig(connectorLabel = "MongoDB: Allow access to MongoDB data from openLooKeng",
+        propertiesEnabled = true,
+        docLink = "https://openlookeng.io/docs/docs/connector/mongodb.html",
+        configLink = "https://openlookeng.io/docs/docs/connector/mongodb.html#configuration")
 public class MongoPlugin
         implements Plugin
 {
@@ -43,4 +55,12 @@ public class MongoPlugin
     {
         return ImmutableList.of(new MongoConnectorFactory("mongodb"));
     }
+
+    @Override
+    public Optional<ConnectorWithProperties> getConnectorWithProperties()
+    {
+        ConnectorConfig connectorConfig = MongoPlugin.class.getAnnotation(ConnectorConfig.class);
+        return ConnectorUtil.assembleConnectorProperties(connectorConfig,
+                Arrays.asList(MongoClientConfig.class.getDeclaredMethods()));
+    }
 }

hetu-transport/src/main/java/io/hetu/core/transport/execution/buffer/PagesSerdeUtil.java

@@ -184,6 +184,11 @@ public class PagesSerdeUtil
         return new PageReader(serde, sliceInput);
     }
 
+    public static Iterator<Page> readPages(GenericPagesSerde serde, SliceInput sliceInput, int spillPrefetchReadPages, Predicate<InputStream> eof)
+    {
+        return new PageReader(serde, sliceInput, spillPrefetchReadPages, eof);
+    }
+
     public static Iterator<Page> readPagesDirect(GenericPagesSerde serde, InputStream input, Predicate<InputStream> eof, int spillPrefetchReadPages)
     {
         return new PageReaderDirect(serde, input, eof, spillPrefetchReadPages);
@@ -195,30 +200,44 @@ public class PagesSerdeUtil
         private final GenericPagesSerde serde;
         private final SliceInput input;
         private final int spillPrefetchReadPages;
+        Predicate<InputStream> eof;
         LinkedList<Page> prefetchedPages;
 
         PageReader(GenericPagesSerde serde, SliceInput input)
         {
-            this(serde, input, DEFAULT_NUM_PAGES_PREFETCH);
+            this(serde, input, DEFAULT_NUM_PAGES_PREFETCH, null);
         }
 
-        PageReader(GenericPagesSerde serde, SliceInput input, int spillPrefetchReadPages)
+        PageReader(GenericPagesSerde serde, SliceInput sliceInput, int spillPrefetchReadPages)
+        {
+            this(serde, sliceInput, spillPrefetchReadPages, null);
+        }
+
+        PageReader(GenericPagesSerde serde, SliceInput input, int spillPrefetchReadPages, Predicate<InputStream> eof)
         {
             this.serde = requireNonNull(serde, "serde is null");
             this.input = requireNonNull(input, "input is null");
             checkArgument(spillPrefetchReadPages > 0, "spillPrefetchReadPages cannot be less then 1");
             this.prefetchedPages = new LinkedList<>();
             this.spillPrefetchReadPages = spillPrefetchReadPages;
+            this.eof = eof;
         }
 
         @Override
         protected Page computeNext()
         {
-            if (!input.isReadable() && prefetchedPages.size() == 0) {
+            if (eof != null) {
-                return endOfData();
+                if (eof.test(input) && prefetchedPages.size() == 0) {
+                    return endOfData();
+                }
+            }
+            else {
+                if (!input.isReadable() && prefetchedPages.size() == 0) {
+                    return endOfData();
+                }
             }
             if (prefetchedPages.size() == 0) {
-                for (int i = 0; i < spillPrefetchReadPages && input.isReadable(); i++) {
+                for (int i = 0; i < spillPrefetchReadPages && (eof != null ? !eof.test(input) : input.isReadable()); i++) {
                     prefetchedPages.add(serde.deserialize(readSerializedPage(input)));
                 }
             }

pom.xml

@@ -59,7 +59,7 @@
         <dep.prestosql.version>316</dep.prestosql.version>
         <dep.jackson.version>2.11.4</dep.jackson.version>
         <dep.hetu.version>${project.version}</dep.hetu.version>
-        <dep.jetty.version>9.4.43.v20210629</dep.jetty.version>
+        <dep.jetty.version>9.4.46.v20220331</dep.jetty.version>
         <dep.bcprov.version>1.66</dep.bcprov.version>
         <dep.hudi.version>0.7.0</dep.hudi.version>
         <dep.kryo.version>5.0.3</dep.kryo.version>
@@ -912,7 +912,7 @@
             <dependency>
                 <groupId>mysql</groupId>
                 <artifactId>mysql-connector-java</artifactId>
-                <version>5.1.47</version>
+                <version>5.1.49</version>
             </dependency>
 
             <dependency>

presto-benchmark/src/main/java/io/prestosql/benchmark/OrderByBenchmark.java

@@ -59,7 +59,7 @@ public class OrderByBenchmark
                 false,
                 Optional.empty(),
                 new OrderingCompiler(),
-                false);
+                false, false);
 
         return ImmutableList.of(tableScanOperator, limitOperator, orderByOperator);
     }

presto-elasticsearch/pom.xml

@@ -14,7 +14,7 @@
 
     <properties>
         <air.main.basedir>${project.parent.basedir}</air.main.basedir>
-        <dep.log4j.version>2.17.1</dep.log4j.version>
+        <dep.log4j.version>2.17.2</dep.log4j.version>
         <dep.elasticsearch.version>6.0.0</dep.elasticsearch.version>
         <httpcore-nio.version>4.4.5</httpcore-nio.version>
     </properties>

presto-main/src/main/java/io/prestosql/SystemSessionProperties.java

@@ -14,6 +14,7 @@
 package io.prestosql;
 
 import com.google.common.collect.ImmutableList;
+import io.airlift.log.Logger;
 import io.airlift.units.DataSize;
 import io.airlift.units.Duration;
 import io.prestosql.execution.QueryManagerConfig;
@@ -161,6 +162,7 @@ public final class SystemSessionProperties
     public static final String SPILL_THRESHOLD_REUSE_TABLESCAN = "spill_threshold_reuse_tablescan";
     public static final String SORT_BASED_AGGREGATION_ENABLED = "sort_based_aggregation_enabled";
     public static final String PRCNT_DRIVERS_FOR_PARTIAL_AGGR = "prcnt_drivers_for_partial_aggr";
+    public static final String SPILL_TO_HDFS_ENABLED = "spill_to_hdfs_enabled";
     // CTE Optimization configurations
     public static final String CTE_REUSE_ENABLED = "cte_reuse_enabled";
     public static final String CTE_MAX_QUEUE_SIZE = "cte_max_queue_size";
@@ -178,9 +180,12 @@ public final class SystemSessionProperties
     public static final String SNAPSHOT_SPLIT_COUNT_INTERVAL = "snapshot_split_count_interval";
     public static final String SKIP_ATTACHING_STATS_WITH_PLAN = "skip_attaching_stats_with_plan";
     public static final String SKIP_NON_APPLICABLE_RULES_ENABLED = "skip_non_applicable_rules_enabled";
+    public static final String ELIMINATE_DUPLICATE_SPILL_FILES = "eliminate_duplicate_spill_files";
 
     private final List<PropertyMetadata<?>> sessionProperties;
 
+    private static Logger log = Logger.get(SystemSessionProperties.class);
+
     public SystemSessionProperties()
     {
         this(new QueryManagerConfig(), new TaskManagerConfig(), new MemoryManagerConfig(), new FeaturesConfig(), new HetuConfig(), new RecoveryConfig());
@@ -836,6 +841,31 @@ public final class SystemSessionProperties
                         EXTENSION_EXECUTION_PLANNER_ENABLED,
                         "extension execution planner enabled",
                         hetuConfig.getExtensionExecutionPlannerEnabled(),
+                        false),
+                new PropertyMetadata<>(
+                        SPILL_TO_HDFS_ENABLED,
+                        "Enable Spill To HDFS",
+                        BOOLEAN,
+                        Boolean.class,
+                        featuresConfig.isSpillToHdfs(),
+                        false,
+                        value -> {
+                            boolean spillToHdfs = (Boolean) value;
+                            if (spillToHdfs && (featuresConfig.getSpillProfile() == null || featuresConfig.getSpillProfile().isEmpty())) {
+                                throw new PrestoException(
+                                        INVALID_SESSION_PROPERTY,
+                                        format("%s cannot be set to true; no spill profile is configured", SPILL_TO_HDFS_ENABLED));
+                            }
+                            if (spillToHdfs) {
+                                log.warn("Spiller path configured by user is ignored and /tmp/hetu/snapshot is used by default");
+                            }
+                            return spillToHdfs;
+                        },
+                        value -> value),
+                booleanProperty(
+                        ELIMINATE_DUPLICATE_SPILL_FILES,
+                        "Eliminates back up of spill files",
+                        recoveryConfig.isEliminateDuplicateSpillFilesEnabled(),
                         false));
     }
 
@@ -1465,4 +1495,14 @@ public final class SystemSessionProperties
     {
         return session.getSystemProperty(EXTENSION_EXECUTION_PLANNER_CLASS_PATH, String.class);
     }
+
+    public static boolean isSpillToHdfsEnabled(Session session)
+    {
+        return session.getSystemProperty(SPILL_TO_HDFS_ENABLED, Boolean.class);
+    }
+
+    public static boolean isEliminateDuplicateSpillFilesEnabled(Session session)
+    {
+        return session.getSystemProperty(ELIMINATE_DUPLICATE_SPILL_FILES, Boolean.class);
+    }
 }

presto-main/src/main/java/io/prestosql/dispatcher/DispatchManager.java

@@ -296,7 +296,7 @@ public class DispatchManager
                     }
                     else {
                         if (!isUiQuery && dispatchQuery.getBasicQueryInfo().getState() == QueryState.FAILED) {
-                            QueryInfo queryInfo = queryHistoryService.toFullQueryInfo(dispatchQuery);
+                            QueryInfo queryInfo = QueryHistoryService.toFullQueryInfo(dispatchQuery);
                             queryHistoryService.insert(queryInfo);
                         }
                         // execution MUST be added to the expiration queue or there will be a leak

presto-main/src/main/java/io/prestosql/execution/QueryInfo.java

@@ -75,6 +75,7 @@ public class QueryInfo
     private final boolean completeInfo;
     private final Optional<ResourceGroupId> resourceGroupId;
     private final boolean runningAsync;
+    private final boolean recoveryEnabled;
 
     @JsonCreator
     public QueryInfo(
@@ -107,7 +108,8 @@ public class QueryInfo
             @JsonProperty("output") Optional<Output> output,
             @JsonProperty("completeInfo") boolean completeInfo,
             @JsonProperty("resourceGroupId") Optional<ResourceGroupId> resourceGroupId,
-            @JsonProperty("runningAsync") boolean runningAsync)
+            @JsonProperty("runningAsync") boolean runningAsync,
+            @JsonProperty("recoveryEnabled") boolean recoveryEnabled)
     {
         requireNonNull(queryId, "queryId is null");
         requireNonNull(session, "session is null");
@@ -162,6 +164,7 @@ public class QueryInfo
         this.completeInfo = completeInfo;
         this.resourceGroupId = resourceGroupId;
         this.runningAsync = runningAsync;
+        this.recoveryEnabled = recoveryEnabled;
     }
 
     @JsonProperty
@@ -194,6 +197,12 @@ public class QueryInfo
         return scheduled;
     }
 
+    @JsonProperty
+    public boolean isRecoveryEnabled()
+    {
+        return recoveryEnabled;
+    }
+
     @JsonProperty
     public URI getSelf()
     {

presto-main/src/main/java/io/prestosql/execution/QueryStateMachine.java

@@ -162,6 +162,7 @@ public class QueryStateMachine
     private final WarningCollector warningCollector;
 
     private final AtomicBoolean isRunningAsync = new AtomicBoolean();
+    private final boolean recoveryEnabled;
 
     private QueryStateMachine(
             String query,
@@ -193,6 +194,7 @@ public class QueryStateMachine
         this.finalQueryInfo = new StateMachine<>("finalQueryInfo-" + queryId, executor, Optional.empty());
         this.outputManager = new QueryOutputManager(executor);
         this.warningCollector = requireNonNull(warningCollector, "warningCollector is null");
+        this.recoveryEnabled = SystemSessionProperties.isRecoveryEnabled(getSession());
     }
 
     /**
@@ -404,7 +406,8 @@ public class QueryStateMachine
                 preparedQuery,
                 queryStats,
                 errorCode == null ? null : errorCode.getType(),
-                errorCode);
+                errorCode,
+                recoveryEnabled);
     }
 
     @VisibleForTesting
@@ -458,7 +461,8 @@ public class QueryStateMachine
                 output.get(),
                 completeInfo,
                 Optional.of(resourceGroup),
-                isRunningAsync.get());
+                isRunningAsync.get(),
+                recoveryEnabled);
     }
 
     private QueryStats getQueryStats(Optional<StageInfo> rootStage)
@@ -1161,7 +1165,8 @@ public class QueryStateMachine
                 queryInfo.getInputs(),
                 queryInfo.getOutput(),
                 queryInfo.isCompleteInfo(),
-                queryInfo.getResourceGroupId(), false);
+                queryInfo.getResourceGroupId(), false,
+                queryInfo.isRecoveryEnabled());
         finalQueryInfo.compareAndSet(finalInfo, Optional.of(prunedQueryInfo));
     }
 

presto-main/src/main/java/io/prestosql/execution/SqlQueryExecution.java

@@ -889,15 +889,16 @@ public class SqlQueryExecution
     {
         try (SetThreadName ignored = new SetThreadName("Query-%s", stateMachine.getQueryId())) {
             SqlQueryScheduler scheduler = queryScheduler.get();
-            stateMachine.transitionToSuspend();
             if (scheduler != null) {
-                boolean useSnapshot = SystemSessionProperties.isSnapshotEnabled(stateMachine.getSession()) && snapshotManager.isSuccessfulSnapshotExist();
+                boolean useSnapshot = SystemSessionProperties.isRecoveryEnabled(stateMachine.getSession());
                 if (useSnapshot) {
+                    stateMachine.transitionToSuspend();
                     suspendedWithRecoveryManager.set(true);
                     queryRecoveryManager.suspendQuery();
                 }
                 else {
-                    scheduler.suspend();
+                    // if both snapshot and recovery are disabled.
+                    throw new PrestoException(NOT_SUPPORTED, "Either Snapshot or Recovery should be enabled to suspend Query");
                 }
             }
         }
@@ -908,7 +909,7 @@ public class SqlQueryExecution
     {
         try (SetThreadName ignored = new SetThreadName("Query-%s", stateMachine.getQueryId())) {
             SqlQueryScheduler scheduler = queryScheduler.get();
-            if (SystemSessionProperties.isSnapshotEnabled(stateMachine.getSession()) && suspendedWithRecoveryManager.get()) {
+            if (SystemSessionProperties.isRecoveryEnabled(stateMachine.getSession()) && suspendedWithRecoveryManager.get()) {
                 queryRecoveryManager.resumeQuery();
                 suspendedWithRecoveryManager.set(false);
             }

presto-main/src/main/java/io/prestosql/execution/SqlStageExecution.java

@@ -83,6 +83,7 @@ import static io.prestosql.failuredetector.FailureDetector.State.GONE;
 import static io.prestosql.operator.ExchangeOperator.REMOTE_CONNECTOR_ID;
 import static io.prestosql.spi.StandardErrorCode.GENERIC_INTERNAL_ERROR;
 import static io.prestosql.spi.StandardErrorCode.REMOTE_HOST_GONE;
+import static io.prestosql.util.Failures.WORKER_NODE_ERROR;
 import static java.util.Objects.requireNonNull;
 
 @ThreadSafe
@@ -652,6 +653,12 @@ public final class SqlStageExecution
                         queryRecoveryManager.startRecovery();
                         return;
                     }
+                    else if (message.contains(WORKER_NODE_ERROR)) {
+                        // Handling timeout interaction between worker - to - worker.
+                        log.debug(failure, "WorkerToWorker interaction timeout. Task %s on node %s failed but is resumable. Triggering rescheduling.", taskStatus.getTaskId(), taskStatus.getNodeId());
+                        queryRecoveryManager.startRecovery();
+                        return;
+                    }
                 }
                 stateMachine.transitionToFailed(failure);
             }

presto-main/src/main/java/io/prestosql/execution/SqlTaskExecution.java

@@ -838,14 +838,24 @@ public class SqlTaskExecution
 
     public void suspendTask()
     {
-        taskExecutor.suspendTask(taskId);
+        if (taskStateMachine.getState() == TaskState.RUNNING) {
-        taskStateMachine.suspend();
+            taskExecutor.suspendTask(taskId);
+            taskStateMachine.suspend();
+        }
+        else {
+            log.warn("Task Suspend requested when its not running: %s", taskStateMachine.getState().toString());
+        }
     }
 
     public void resumeTask()
     {
-        taskExecutor.resumeTask(taskId);
+        if (taskStateMachine.getState() == TaskState.SUSPENDED) {
-        taskStateMachine.resume();
+            taskExecutor.resumeTask(taskId);
+            taskStateMachine.resume();
+        }
+        else {
+            log.warn("Task Resume requested when its not suspended: %s", taskStateMachine.getState().toString());
+        }
     }
 
     // Splits for a particular plan node (all driver groups)

presto-main/src/main/java/io/prestosql/execution/executor/TaskExecutor.java

@@ -287,6 +287,8 @@ public class TaskExecutor
             List<PrioritizedSplitRunner> splits;
 
             suspendedTasks.remove(taskHandle);
+
+            taskHandle.resume();
             tasks.add(taskHandle);
 
             splits = taskHandle.getRunningIntermediateSplits();
@@ -568,7 +570,9 @@ public class TaskExecutor
                                     blockedSplits.remove(split);
                                     // reset the level priority to prevent previously-blocked splits from starving existing splits
                                     split.resetLevelPriority();
-                                    waitingSplits.offer(split);
+                                    if (!split.getTaskHandle().isSuspended()) {
+                                        waitingSplits.offer(split);
+                                    }
                                 }, executor);
                             }
                         }

presto-main/src/main/java/io/prestosql/execution/executor/TaskHandle.java

@@ -26,6 +26,7 @@ import java.util.ArrayList;
 import java.util.List;
 import java.util.OptionalInt;
 import java.util.Queue;
+import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicInteger;
 import java.util.concurrent.atomic.AtomicReference;
 import java.util.function.DoubleSupplier;
@@ -53,6 +54,8 @@ public class TaskHandle
     @GuardedBy("this")
     protected final SplitConcurrencyController concurrencyController;
 
+    private AtomicBoolean isSuspended = new AtomicBoolean(false);
+
     private final AtomicInteger nextSplitId = new AtomicInteger();
 
     protected final AtomicReference<Priority> priority = new AtomicReference<>(new Priority(0, 0));
@@ -122,6 +125,9 @@ public class TaskHandle
     public synchronized List<PrioritizedSplitRunner> suspend()
     {
         checkState(!destroyed, "Cannot suspend task as already destroyed");
+
+        isSuspended.compareAndSet(false, true);
+
         ImmutableList.Builder<PrioritizedSplitRunner> builder = ImmutableList.builder();
         builder.addAll(runningIntermediateSplits);
         builder.addAll(runningLeafSplits);
@@ -133,6 +139,16 @@ public class TaskHandle
         return builder.build();
     }
 
+    public synchronized void resume()
+    {
+        isSuspended.compareAndSet(true, false);
+    }
+
+    public synchronized boolean isSuspended()
+    {
+        return isSuspended.get();
+    }
+
     // Returns any remaining splits. The caller must destroy these.
     public synchronized List<PrioritizedSplitRunner> destroy()
     {
@@ -192,6 +208,7 @@ public class TaskHandle
         concurrencyController.splitFinished(split.getScheduledNanos(), utilizationSupplier.getAsDouble(), runningLeafSplits.size());
         runningIntermediateSplits.remove(split);
         runningLeafSplits.remove(split);
+        queuedLeafSplits.remove(split); // for case, task is suspended and an already scheduled split finishes
     }
 
     public int getNextSplitId()

presto-main/src/main/java/io/prestosql/execution/scheduler/SqlQueryScheduler.java

@@ -97,6 +97,7 @@ import static io.prestosql.SystemSessionProperties.getConcurrentLifespansPerNode
 import static io.prestosql.SystemSessionProperties.getWriterMinSize;
 import static io.prestosql.SystemSessionProperties.isReuseTableScanEnabled;
 import static io.prestosql.execution.BasicStageStats.aggregateBasicStageStats;
+import static io.prestosql.execution.QueryState.RECOVERING;
 import static io.prestosql.execution.SqlStageExecution.createSqlStageExecution;
 import static io.prestosql.execution.StageState.ABORTED;
 import static io.prestosql.execution.StageState.CANCELED;
@@ -106,7 +107,6 @@ import static io.prestosql.execution.StageState.RUNNING;
 import static io.prestosql.execution.StageState.SCHEDULED;
 import static io.prestosql.execution.scheduler.SourcePartitionedScheduler.newSourcePartitionedSchedulerAsStageScheduler;
 import static io.prestosql.snapshot.RecoveryConfig.calculateTaskCount;
-import static io.prestosql.snapshot.RecoveryState.STOPPING_FOR_RESCHEDULE;
 import static io.prestosql.spi.StandardErrorCode.GENERIC_INTERNAL_ERROR;
 import static io.prestosql.spi.StandardErrorCode.NO_NODES_AVAILABLE;
 import static io.prestosql.spi.connector.CatalogName.isInternalSystemConnector;
@@ -260,6 +260,9 @@ public class SqlQueryScheduler
         OutputBufferId rootBufferId = Iterables.getOnlyElement(rootOutputBuffers.getBuffers().keySet());
         visitedPlanFrags.add(plan.getFragment().getId());
         final boolean isRecoveryEnabled = SystemSessionProperties.isRecoveryEnabled(session);
+        if (isResumeScheduler) {
+            nodeScheduler.refreshNodeStates();
+        }
         List<SqlStageExecution> stageExecutions = createStages(
                 (fragmentId, tasks, noMoreExchangeLocations) -> updateQueryOutputLocations(queryStateMachine, rootBufferId, tasks, noMoreExchangeLocations),
                 new AtomicInteger(),
@@ -471,9 +474,6 @@ public class SqlQueryScheduler
             if (isInternalSystemConnector(catalogName)) {
                 catalogName = null;
             }
-            if (isResumeScheduler) {
-                nodeScheduler.refreshNodeStates();
-            }
             NodeSelector nodeSelector = nodeScheduler.createNodeSelector(catalogName, keepConsumerOnFeederNodes, feederScheduledNodes);
             if (isRecoveryEnabled) {
                 // When snapshot is enabled, then no task can be added after the query started running,
@@ -859,7 +859,7 @@ public class SqlQueryScheduler
             RuntimeException closeError = new RuntimeException();
             for (StageScheduler scheduler : stageSchedulers.values()) {
                 try {
-                    if (queryRecoveryManager.getState() != STOPPING_FOR_RESCHEDULE) {
+                    if (queryStateMachine.getQueryState() != RECOVERING && !queryRecoveryManager.isRecovering()) {
                         scheduler.close();
                     }
                 }

presto-main/src/main/java/io/prestosql/failuredetector/AbstractFailureRetryPolicy.java

@@ -28,10 +28,12 @@ public abstract class AbstractFailureRetryPolicy
         this.backoff = backoff;
     }
 
+    @Override
     public IBackoff getBackoff()
     {
         return this.backoff;
     }
 
+    @Override
     public abstract boolean hasFailed(HostAddress address);
 }

presto-main/src/main/java/io/prestosql/failuredetector/HeartbeatFailureDetector.java

@@ -510,7 +510,7 @@ public class HeartbeatFailureDetector
                 lastCompleteTimestamp = System.nanoTime();
                 lastFailureCount = stats.getRecentFailures();
             }
-            else if (Duration.nanosSince(lastCompleteTimestamp).compareTo(new Duration(1, TimeUnit.SECONDS)) > 0 && stats.getRecentFailures() == lastFailureCount) {
+            else if (Duration.nanosSince(lastCompleteTimestamp).compareTo(new Duration(1, TimeUnit.SECONDS)) > 0 && stats.getRecentFailures() <= lastFailureCount) {
                 lastCompleteTimestamp = System.nanoTime();
             }
         }

presto-main/src/main/java/io/prestosql/memory/QueryContext.java

@@ -280,7 +280,7 @@ public class QueryContext
                 parent.orElse(null),
                 serdeFactory,
                 new TaskSnapshotManager(taskStateMachine.getTaskId(), resumeCount, recoveryUtils),
-                recoveryUtils.getOrCreateRecoveryManager(queryId, session));
+                recoveryUtils.getRecoveryManager(queryId));
         taskContexts.put(taskInstanceId, taskContext);
         return taskContext;
     }

presto-main/src/main/java/io/prestosql/operator/HashAggregationOperator.java

@@ -70,7 +70,7 @@ public class HashAggregationOperator
                     false,
                     new DataSize(0, MEGABYTE),
                     new DataSize(0, MEGABYTE),
-                    (types, spillContext, memoryContext) -> {
+                    (types, spillContext, memoryContext, isSnapshotEnabled, queryId, isSpillToHdfs) -> {
                         throw new UnsupportedOperationException();
                     },
                     joinCompiler,

presto-main/src/main/java/io/prestosql/operator/HashBuilderOperator.java

@@ -18,6 +18,7 @@ import com.google.common.collect.AbstractIterator;
 import com.google.common.collect.ImmutableList;
 import com.google.common.io.Closer;
 import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.SettableFuture;
 import io.prestosql.execution.Lifespan;
 import io.prestosql.memory.context.LocalMemoryContext;
 import io.prestosql.snapshot.SingleInputSnapshotState;
@@ -32,7 +33,10 @@ import io.prestosql.spi.type.BigintType;
 import io.prestosql.spi.util.BloomFilter;
 import io.prestosql.spiller.SingleStreamSpiller;
 import io.prestosql.spiller.SingleStreamSpillerFactory;
+import io.prestosql.spiller.Spiller;
+import io.prestosql.spiller.SpillerFactory;
 import io.prestosql.sql.gen.JoinFilterFunctionCompiler.JoinFilterFunctionFactory;
+import org.apache.commons.lang3.tuple.Pair;
 
 import javax.annotation.Nullable;
 import javax.annotation.concurrent.ThreadSafe;
@@ -43,7 +47,6 @@ import java.io.IOException;
 import java.io.Serializable;
 import java.io.UncheckedIOException;
 import java.nio.file.Path;
-import java.util.ArrayDeque;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.HashMap;
@@ -53,13 +56,14 @@ import java.util.Map;
 import java.util.Optional;
 import java.util.OptionalInt;
 import java.util.OptionalLong;
-import java.util.Queue;
 import java.util.concurrent.atomic.AtomicBoolean;
 
 import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Preconditions.checkState;
 import static com.google.common.base.Verify.verify;
 import static com.google.common.util.concurrent.Futures.immediateFuture;
+import static com.google.common.util.concurrent.Futures.whenAllComplete;
+import static com.google.common.util.concurrent.MoreExecutors.directExecutor;
 import static io.airlift.concurrent.MoreFutures.checkSuccess;
 import static io.airlift.concurrent.MoreFutures.getDone;
 import static io.prestosql.SystemSessionProperties.isInnerJoinSpillFilteringEnabled;
@@ -77,7 +81,7 @@ import static java.util.Objects.requireNonNull;
 @RestorableConfig(uncapturedFields = {"lookupSourceFactory", "lookupSourceFactoryDestroyed", "outputChannels",
         "hashChannels", "filterFunctionFactory", "sortChannel", "searchFunctionFactories", "singleStreamSpillerFactory",
         "lookupSourceNotNeeded", "spilledLookupSourceHandle", "spillInProgress", "unspillInProgress", "lookupSourceSupplier", "lookupSourceChecksum",
-        "finishMemoryRevoke", "snapshotState", "lastMarker"})
+        "finishMemoryRevoke", "snapshotState", "lastMarker", "finishInProgress", "spillerFactory"})
 public class HashBuilderOperator
         implements SinkOperator, Spillable
 {
@@ -102,7 +106,10 @@ public class HashBuilderOperator
         private final Map<Lifespan, Integer> partitionIndexManager = new HashMap<>();
 
         private boolean closed;
+        private boolean spillToHdfsEnabled;
+        private SpillerFactory spillerFactory;
 
+        @VisibleForTesting
         public HashBuilderOperatorFactory(
                 int operatorId,
                 PlanNodeId planNodeId,
@@ -117,6 +124,40 @@ public class HashBuilderOperator
                 PagesIndex.Factory pagesIndexFactory,
                 boolean spillEnabled,
                 SingleStreamSpillerFactory singleStreamSpillerFactory)
+        {
+            this(operatorId,
+                    planNodeId,
+                    lookupSourceFactoryManager,
+                    outputChannels,
+                    hashChannels,
+                    preComputedHashChannel,
+                    filterFunctionFactory,
+                    sortChannel,
+                    searchFunctionFactories,
+                    expectedPositions,
+                    pagesIndexFactory,
+                    spillEnabled,
+                    singleStreamSpillerFactory,
+                    null,
+                    false);
+        }
+
+        public HashBuilderOperatorFactory(
+                int operatorId,
+                PlanNodeId planNodeId,
+                JoinBridgeManager<PartitionedLookupSourceFactory> lookupSourceFactoryManager,
+                List<Integer> outputChannels,
+                List<Integer> hashChannels,
+                OptionalInt preComputedHashChannel,
+                Optional<JoinFilterFunctionFactory> filterFunctionFactory,
+                Optional<Integer> sortChannel,
+                List<JoinFilterFunctionFactory> searchFunctionFactories,
+                int expectedPositions,
+                PagesIndex.Factory pagesIndexFactory,
+                boolean spillEnabled,
+                SingleStreamSpillerFactory singleStreamSpillerFactory,
+                SpillerFactory spillerFactory,
+                boolean spillToHdfsEnabled)
         {
             this.operatorId = operatorId;
             this.planNodeId = requireNonNull(planNodeId, "planNodeId is null");
@@ -134,7 +175,8 @@ public class HashBuilderOperator
             this.pagesIndexFactory = requireNonNull(pagesIndexFactory, "pagesIndexFactory is null");
             this.spillEnabled = spillEnabled;
             this.singleStreamSpillerFactory = requireNonNull(singleStreamSpillerFactory, "singleStreamSpillerFactory is null");
-
+            this.spillToHdfsEnabled = spillToHdfsEnabled;
+            this.spillerFactory = spillerFactory;
             this.expectedPositions = expectedPositions;
         }
 
@@ -165,7 +207,9 @@ public class HashBuilderOperator
                     expectedPositions,
                     pagesIndexFactory,
                     spillEnabled,
-                    singleStreamSpillerFactory);
+                    singleStreamSpillerFactory,
+                    spillerFactory,
+                    spillToHdfsEnabled);
         }
 
         @Override
@@ -253,8 +297,10 @@ public class HashBuilderOperator
     private Optional<ListenableFuture<?>> lookupSourceNotNeeded = Optional.empty();
     private final SpilledLookupSourceHandle spilledLookupSourceHandle;
     private Optional<SingleStreamSpiller> spiller = Optional.empty();
+    private Optional<Spiller> genericSpiller = Optional.empty();
     private ListenableFuture<?> spillInProgress = NOT_BLOCKED;
-    private Optional<ListenableFuture<List<Page>>> unspillInProgress = Optional.empty();
+    private SettableFuture<?> finishInProgress;
+    private Optional<ListenableFuture<Integer>> unspillInProgress = Optional.empty();
     @Nullable
     private LookupSourceSupplier lookupSourceSupplier;
     private OptionalLong lookupSourceChecksum = OptionalLong.empty();
@@ -280,6 +326,8 @@ public class HashBuilderOperator
     private MarkerPage lastMarker;
     // If this flag is true, then it was restored from the extra snapshot, and should discard incoming pages.
     private boolean alreadyFinished;
+    private boolean spillToHdfsEnabled;
+    private final SpillerFactory spillerFactory;
 
     public HashBuilderOperator(
             OperatorContext operatorContext,
@@ -294,7 +342,9 @@ public class HashBuilderOperator
             int expectedPositions,
             PagesIndex.Factory pagesIndexFactory,
             boolean spillEnabled,
-            SingleStreamSpillerFactory singleStreamSpillerFactory)
+            SingleStreamSpillerFactory singleStreamSpillerFactory,
+            SpillerFactory spillerFactory,
+            boolean spillToHdfsEnabled)
     {
         requireNonNull(pagesIndexFactory, "pagesIndexFactory is null");
 
@@ -327,8 +377,12 @@ public class HashBuilderOperator
         else {
             this.spillBloom = null;
         }
+        this.finishInProgress = SettableFuture.create();
+        this.finishInProgress.set(null);
 
         spilledLookupSourceHandle = new SpilledLookupSourceHandle(spillBloom);
+        this.spillToHdfsEnabled = spillToHdfsEnabled;
+        this.spillerFactory = spillerFactory;
     }
 
     @Override
@@ -348,7 +402,7 @@ public class HashBuilderOperator
     {
         switch (state) {
             case CONSUMING_INPUT:
-                return NOT_BLOCKED;
+                return finishInProgress;
 
             case SPILLING_INPUT:
                 return spillInProgress;
@@ -374,8 +428,7 @@ public class HashBuilderOperator
     @Override
     public boolean needsInput()
     {
-        boolean stateNeedsInput = (state == State.CONSUMING_INPUT)
+        boolean stateNeedsInput = (state == State.CONSUMING_INPUT || state == State.SPILLING_INPUT) && spillInProgress.isDone() && finishInProgress.isDone();
-                || (state == State.SPILLING_INPUT && spillInProgress.isDone());
 
         return stateNeedsInput && !lookupSourceFactoryDestroyed.isDone();
     }
@@ -455,11 +508,14 @@ public class HashBuilderOperator
         checkState(spillEnabled, "Spill not enabled, no revokable memory should be reserved");
 
         if (state == State.CONSUMING_INPUT) {
+            finishInProgress = SettableFuture.create();
             long indexSizeBeforeCompaction = index.getEstimatedSize().toBytes();
             index.compact();
             long indexSizeAfterCompaction = index.getEstimatedSize().toBytes();
             if (indexSizeAfterCompaction < indexSizeBeforeCompaction * INDEX_COMPACTION_ON_REVOCATION_TARGET) {
-                finishMemoryRevoke = Optional.of(() -> {});
+                finishMemoryRevoke = Optional.of(() -> {
+                    finishInProgress.set(null);
+                });
                 return immediateFuture(null);
             }
 
@@ -469,10 +525,12 @@ public class HashBuilderOperator
                 localRevocableMemoryContext.setBytes(0);
                 lookupSourceFactory.setPartitionSpilledLookupSourceHandle(partitionIndex, spilledLookupSourceHandle);
                 state = State.SPILLING_INPUT;
+                finishInProgress.set(null);
             });
             return spillIndex();
         }
         if (state == State.LOOKUP_SOURCE_BUILT) {
+            finishInProgress = SettableFuture.create();
             finishMemoryRevoke = Optional.of(() -> {
                 if (spillBloom != null) {
                     spillBloom.markReady();
@@ -484,7 +542,11 @@ public class HashBuilderOperator
                 localRevocableMemoryContext.setBytes(0);
                 lookupSourceChecksum = OptionalLong.of(lookupSourceSupplier.checksum());
                 lookupSourceSupplier = null;
+                if (spillToHdfsEnabled) {
+                    getSpiller().closeSessionSpiller();
+                }
                 state = State.INPUT_SPILLED;
+                finishInProgress.set(null);
             });
             return spillIndex();
         }
@@ -500,10 +562,25 @@ public class HashBuilderOperator
     private ListenableFuture<?> spillIndex()
     {
         checkState(!spiller.isPresent(), "Spiller already created");
-        spiller = Optional.of(singleStreamSpillerFactory.create(
+        checkState(!genericSpiller.isPresent(), "Generic Spiller already created");
-                index.getTypes(),
+        if (spillToHdfsEnabled) {
-                operatorContext.getSpillContext().newLocalSpillContext(),
+            genericSpiller = Optional.of(spillerFactory.create(
-                operatorContext.newLocalSystemMemoryContext(HashBuilderOperator.class.getSimpleName())));
+                    index.getTypes(),
+                    operatorContext.getSpillContext().newLocalSpillContext(),
+                    operatorContext.newAggregateSystemMemoryContext(),
+                    operatorContext.isSnapshotEnabled(),
+                    operatorContext.getDriverContext().getTaskId().getQueryId().toString(), spillToHdfsEnabled));
+            createSessionSpiller();
+        }
+        else {
+            spiller = Optional.of(singleStreamSpillerFactory.create(
+                    index.getTypes(),
+                    operatorContext.getSpillContext().newLocalSpillContext(),
+                    operatorContext.newLocalSystemMemoryContext(HashBuilderOperator.class.getSimpleName()),
+                    spillToHdfsEnabled,
+                    operatorContext.isSnapshotEnabled(),
+                    operatorContext.getDriverContext().getTaskId().getQueryId().toString(), spillToHdfsEnabled));
+        }
         return getSpiller().spill(new AbstractIterator<Page>()
         {
             private Iterator<Page> spillPages = index.getPages();
@@ -532,6 +609,15 @@ public class HashBuilderOperator
     @Override
     public void finish()
     {
+        if (lookupSourceFactoryDestroyed.isDone()) {
+            close();
+            return;
+        }
+
+        if (finishMemoryRevoke.isPresent()) {
+            return;
+        }
+
         if (snapshotState != null && !alreadyFinished && lastMarker != null) {
             // Update alreadyFinished field *before* calling captureState
             alreadyFinished = true;
@@ -646,6 +732,9 @@ public class HashBuilderOperator
         if (spillBloom != null) {
             spillBloom.markReady();
         }
+        if (spillToHdfsEnabled) {
+            getSpiller().closeSessionSpiller();
+        }
         state = State.INPUT_SPILLED;
     }
 
@@ -660,8 +749,27 @@ public class HashBuilderOperator
         verify(spiller.isPresent());
         verify(!unspillInProgress.isPresent());
 
-        localUserMemoryContext.setBytes(getSpiller().getSpilledPagesInMemorySize() + index.getEstimatedSize().toBytes());
+        if (spillToHdfsEnabled) {
-        unspillInProgress = Optional.of(getSpiller().getAllSpilledPages());
+            localUserMemoryContext.setBytes(getGenericSpiller().getSpilledPagesInMemorySize() + index.getEstimatedSize().toBytes());
+            unspillInProgress = Optional.of(getGenericSpiller().getAllSpilledPages(pages -> {
+                for (Page page : pages) {
+                    index.addPage(page);
+                    localUserMemoryContext.setBytes(index.getEstimatedSize().toBytes());
+                }
+                return 0;
+            }));
+        }
+        else {
+            localUserMemoryContext.setBytes(getSpiller().getSpilledPagesInMemorySize() + index.getEstimatedSize().toBytes());
+            ListenableFuture<List<Page>> listenableFuture = getSpiller().getAllSpilledPages();
+            unspillInProgress = Optional.of(whenAllComplete(listenableFuture).call(() -> {
+                for (Page page : getDone(listenableFuture)) {
+                    index.addPage(page);
+                    localUserMemoryContext.setBytes(index.getEstimatedSize().toBytes());
+                }
+                return 0;
+            }, directExecutor()));
+        }
 
         state = State.INPUT_UNSPILLING;
     }
@@ -674,21 +782,7 @@ public class HashBuilderOperator
             return;
         }
 
-        // Use Queue so that Pages already consumed by Index are not retained by us.
+        getDone(unspillInProgress.get());
-        Queue<Page> pages = new ArrayDeque<>(getDone(unspillInProgress.get()));
-        long memoryRetainedByRemainingPages = pages.stream()
-                .mapToLong(Page::getRetainedSizeInBytes)
-                .sum();
-        localUserMemoryContext.setBytes(memoryRetainedByRemainingPages + index.getEstimatedSize().toBytes());
-
-        while (!pages.isEmpty()) {
-            Page next = pages.remove();
-            index.addPage(next);
-            // There is no attempt to compact index, since unspilled pages are unlikely to have blocks with retained size > logical size.
-            memoryRetainedByRemainingPages -= next.getRetainedSizeInBytes();
-            localUserMemoryContext.setBytes(memoryRetainedByRemainingPages + index.getEstimatedSize().toBytes());
-        }
-
         LookupSourceSupplier partition = buildLookupSource();
         lookupSourceChecksum.ifPresent(checksum ->
                 checkState(partition.checksum() == checksum, "Unspilled lookupSource checksum does not match original one"));
@@ -739,6 +833,16 @@ public class HashBuilderOperator
         return spiller.orElseThrow(() -> new IllegalStateException("Spiller not created"));
     }
 
+    private Spiller getGenericSpiller()
+    {
+        return genericSpiller.orElseThrow(() -> new IllegalStateException("Generic Spiller not created"));
+    }
+
+    private void createSessionSpiller()
+    {
+        spiller = Optional.of(genericSpiller.get().createSessionSpiller());
+    }
+
     @Override
     public void close()
     {
@@ -753,6 +857,7 @@ public class HashBuilderOperator
 
         try (Closer closer = Closer.create()) {
             closer.register(index::clear);
+            genericSpiller.ifPresent(closer::register);
             spiller.ifPresent(closer::register);
             closer.register(() -> localUserMemoryContext.setBytes(0));
             closer.register(() -> localRevocableMemoryContext.setBytes(0));
@@ -773,11 +878,22 @@ public class HashBuilderOperator
         return state == State.SPILLING_INPUT || state == State.INPUT_SPILLED || state == State.INPUT_UNSPILLING;
     }
 
+    @Override
+    public boolean isSpillToHdfsEnabled()
+    {
+        return spillToHdfsEnabled;
+    }
+
     @Override
     public List<Path> getSpilledFilePaths()
     {
         if (isSpilled()) {
-            return ImmutableList.of(spiller.get().getFile());
+            if (spillToHdfsEnabled) {
+                return genericSpiller.get().getSpilledFilePaths(true);
+            }
+            else {
+                return ImmutableList.of(spiller.get().getFile());
+            }
         }
         return ImmutableList.of();
     }
@@ -851,14 +967,15 @@ public class HashBuilderOperator
                 }
             }
         }
+    }
 
-        private class SpillBloomState
+    @Override
-                implements Serializable
+    public List<Pair<Path, Long>> getSpilledFileInfo()
-        {
+    {
-            boolean isReady;
+        if (isSpilled()) {
-            int counter;
+            return ImmutableList.of(spiller.get().getSpilledFileInfo());
-            List<Object> blooms = new ArrayList<>();
         }
+        return ImmutableList.of();
     }
 
     @Override
@@ -870,17 +987,25 @@ public class HashBuilderOperator
         myState.localRevocableMemoryContext = localRevocableMemoryContext.getBytes();
         myState.index = index.capture(serdeProvider);
         myState.hashCollisionsCounter = hashCollisionsCounter.capture(serdeProvider);
-        myState.state = state.toString();
         myState.alreadyFinished = alreadyFinished;
 
         // Capture spill related to spill
-        if (spiller.isPresent()) {
+        if (spillToHdfsEnabled) {
-            myState.spiller = spiller.get().capture(serdeProvider);
+            if (genericSpiller.isPresent()) {
+                myState.genericSpiller = genericSpiller.get().capture(serdeProvider);
+            }
+        }
+        else {
+            if (spiller.isPresent()) {
+                myState.spiller = spiller.get().capture(serdeProvider);
+            }
         }
 
         if (spillBloom != null) {
             myState.spillBloom = spillBloom.capture();
         }
+        myState.spillToHdfsEnabled = spillToHdfsEnabled;
+        myState.state = state.toString();
         return myState;
     }
 
@@ -898,21 +1023,40 @@ public class HashBuilderOperator
         State oldState = this.state;
         this.state = State.valueOf(myState.state);
         this.alreadyFinished = myState.alreadyFinished;
-
+        this.spillToHdfsEnabled = myState.spillToHdfsEnabled;
         // Restore spill related fields
-        if (myState.spiller != null) {
+        if (myState.spiller != null || myState.genericSpiller != null) {
-            if (!spiller.isPresent()) {
+            if (!spiller.isPresent() || !genericSpiller.isPresent()) {
-                spiller = Optional.of(singleStreamSpillerFactory.create(
+                if (spillToHdfsEnabled) {
-                        index.getTypes(),
+                    genericSpiller = Optional.of(spillerFactory.create(
-                        operatorContext.getSpillContext().newLocalSpillContext(),
+                            index.getTypes(),
-                        operatorContext.newLocalSystemMemoryContext(HashBuilderOperator.class.getSimpleName())));
+                            operatorContext.getSpillContext().newLocalSpillContext(),
+                            operatorContext.newAggregateSystemMemoryContext(),
+                            operatorContext.isSnapshotEnabled(),
+                            operatorContext.getDriverContext().getTaskId().getQueryId().toString(), spillToHdfsEnabled));
+
+                    genericSpiller.get().restore(myState.genericSpiller, serdeProvider);
+                    createSessionSpiller();
+                }
+                else {
+                    spiller = Optional.of(singleStreamSpillerFactory.create(
+                            index.getTypes(),
+                            operatorContext.getSpillContext().newLocalSpillContext(),
+                            operatorContext.newLocalSystemMemoryContext(HashBuilderOperator.class.getSimpleName()),
+                            spillToHdfsEnabled,
+                            operatorContext.isSnapshotEnabled(),
+                            operatorContext.getDriverContext().getTaskId().getQueryId().toString(), spillToHdfsEnabled));
+
+                    this.spiller.get().restore(myState.spiller, serdeProvider);
+                }
             }
-            this.spiller.get().restore(myState.spiller, serdeProvider);
         }
         if (myState.spillBloom != null) {
             this.spillBloom.restore(myState.spillBloom);
         }
-        if (oldState == State.CONSUMING_INPUT && this.state == State.SPILLING_INPUT) {
+
+        // snapshot markers are allowed only if the state is either CONSUMING_INPUT or SPILLING_INPUT
+        if (oldState == State.CONSUMING_INPUT && (this.state == State.SPILLING_INPUT)) {
             lookupSourceFactory.setPartitionSpilledLookupSourceHandle(partitionIndex, spilledLookupSourceHandle);
         }
     }
@@ -926,6 +1070,7 @@ public class HashBuilderOperator
     private static class HashBuilderOperatorState
             implements Serializable
     {
+        private Object genericSpiller;
         private Object operatorContext;
         private long localUserMemoryContext;
         private long localRevocableMemoryContext;
@@ -936,7 +1081,15 @@ public class HashBuilderOperator
         private String state;
         private boolean alreadyFinished;
         private Object spiller;
-
+        private boolean spillToHdfsEnabled;
         private Object spillBloom;
     }
+
+    private static class SpillBloomState
+            implements Serializable
+    {
+        boolean isReady;
+        int counter;
+        List<Object> blooms = new ArrayList<>();
+    }
 }

presto-main/src/main/java/io/prestosql/operator/LookupJoinOperator.java

@@ -13,6 +13,7 @@
  */
 package io.prestosql.operator;
 
+import com.google.common.collect.AbstractIterator;
 import com.google.common.collect.ImmutableList;
 import com.google.common.io.Closer;
 import com.google.common.util.concurrent.ListenableFuture;
@@ -34,18 +35,21 @@ import io.prestosql.spi.type.Type;
 import io.prestosql.spiller.PartitioningSpiller;
 import io.prestosql.spiller.PartitioningSpiller.PartitioningSpillResult;
 import io.prestosql.spiller.PartitioningSpillerFactory;
+import org.apache.commons.lang3.tuple.Pair;
 
 import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
 import java.nio.file.Path;
+import java.util.ArrayList;
 import java.util.HashMap;
 import java.util.Iterator;
 import java.util.List;
 import java.util.Map;
 import java.util.Optional;
 import java.util.OptionalInt;
+import java.util.Set;
 import java.util.function.BiPredicate;
 import java.util.function.IntPredicate;
 import java.util.function.Supplier;
@@ -56,6 +60,7 @@ import static com.google.common.util.concurrent.Futures.immediateFuture;
 import static io.airlift.concurrent.MoreFutures.addSuccessCallback;
 import static io.airlift.concurrent.MoreFutures.checkSuccess;
 import static io.airlift.concurrent.MoreFutures.getDone;
+import static io.airlift.concurrent.MoreFutures.getFutureValue;
 import static io.prestosql.SystemSessionProperties.isInnerJoinSpillFilteringEnabled;
 import static io.prestosql.operator.LookupJoinOperators.JoinType.FULL_OUTER;
 import static io.prestosql.operator.LookupJoinOperators.JoinType.PROBE_OUTER;
@@ -77,7 +82,8 @@ import static java.util.Objects.requireNonNull;
 @RestorableConfig(uncapturedFields = {"probeTypes", "joinProbeFactory", "afterClose", "hashGenerator", "lookupSourceFactory",
         "partitioningSpillerFactory", "lookupSourceProviderFuture", "lookupSourceProvider", "probe", "outputPage",
         "partitionGenerator", "spillInProgress", "unspilling", "currentPartition",
-        "unspilledLookupSource", "unspilledInputPages", "snapshotState", "afterMemOpFinish"})
+        "unspilledLookupSource", "unspilledInputPages", "snapshotState", "afterMemOpFinish", "backUpRestoredPages", "isSpillerRestored",
+        "restoredPartition", "backUpUnspilledMemoryPartitions", "spilledPartitionsList", "unspilledMemoryPartitions"})
 public class LookupJoinOperator
         implements Operator, Spillable
 {
@@ -114,6 +120,7 @@ public class LookupJoinOperator
     private boolean closed;
     private boolean finishing;
     private boolean unspilling;
+    private boolean isSpillerRestored;
     private boolean finished;
     private long joinPosition = -1;
     private int joinSourcePositions;
@@ -128,8 +135,14 @@ public class LookupJoinOperator
     private Optional<Partition<Supplier<LookupSource>>> currentPartition = Optional.empty();
     private Optional<ListenableFuture<Supplier<LookupSource>>> unspilledLookupSource = Optional.empty();
     private Iterator<Page> unspilledInputPages = emptyIterator();
+    private Iterator<Page> unspilledMemoryPartitions = emptyIterator();
+    private Map<Integer, Iterator<Page>> backUpUnspilledMemoryPartitions = new HashMap<>();
+    private Map<Integer, Page> backUpRestoredPages = new HashMap<>();
+    private Integer restoredPartition;
 
     private final SingleInputSnapshotState snapshotState;
+    private boolean isSingleSessionSpiller;
+    private List<Integer> spilledPartitionsList = new ArrayList<>();
 
     public LookupJoinOperator(
             OperatorContext operatorContext,
@@ -143,7 +156,8 @@ public class LookupJoinOperator
             OptionalInt lookupJoinsCount,
             HashGenerator hashGenerator,
             PartitioningSpillerFactory partitioningSpillerFactory,
-            Runnable afterMemOpFinish)
+            Runnable afterMemOpFinish,
+            boolean isSingleSessionSpiller)
     {
         this.operatorContext = requireNonNull(operatorContext, "operatorContext is null");
         this.forked = forked;
@@ -169,6 +183,7 @@ public class LookupJoinOperator
         this.snapshotState = operatorContext.isSnapshotEnabled() ? SingleInputSnapshotState.forOperator(this, operatorContext) : null;
 
         this.afterMemOpFinish = afterMemOpFinish;
+        this.isSingleSessionSpiller = isSingleSessionSpiller;
     }
 
     @Override
@@ -255,17 +270,40 @@ public class LookupJoinOperator
         return spiller.isPresent();
     }
 
+    @Override
+    public boolean isSpillToHdfsEnabled()
+    {
+        return isSingleSessionSpiller;
+    }
+
     @Override
     public List<Path> getSpilledFilePaths()
     {
         if (isSpilled()) {
-            return spiller.get().getSpilledFilePaths();
+            if (isSingleSessionSpiller) {
+                return spiller.get().getSpilledFilePaths(true);
+            }
+            return spiller.get().getSpilledFilePaths(false);
+        }
+        return ImmutableList.of();
+    }
+
+    @Override
+    public List<Pair<Path, Long>> getSpilledFileInfo()
+    {
+        if (isSpilled()) {
+            return spiller.get().getSpilledFileInfo();
         }
         return ImmutableList.of();
     }
 
     @Override
     public void addInput(Page page)
+    {
+        addInput(page, false);
+    }
+
+    private void addInput(Page page, boolean isRestoredPage)
     {
         requireNonNull(page, "page is null");
         checkState(probe == null, "Current page has not been completely processed yet");
@@ -285,6 +323,12 @@ public class LookupJoinOperator
         checkState(tryFetchLookupSourceProvider(), "Not ready to handle input yet");
 
         SpillInfoSnapshot spillInfoSnapshot = lookupSourceProvider.withLease(SpillInfoSnapshot::from);
+        if (isRestoredPage && spillInfoSnapshot.hasSpilled() && spillInfoSnapshot.getSpillMask().test(restoredPartition)) {
+            backUpRestoredPages.put(restoredPartition, page);
+            backUpUnspilledMemoryPartitions.put(restoredPartition, unspilledMemoryPartitions);
+            unspilledMemoryPartitions = emptyIterator();
+            return;
+        }
         addInput(page, spillInfoSnapshot);
     }
 
@@ -293,7 +337,7 @@ public class LookupJoinOperator
         requireNonNull(spillInfoSnapshot, "spillInfoSnapshot is null");
 
         Page newPage = page;
-        if (spillInfoSnapshot.hasSpilled()) {
+        if (restoredPartition == null && spillInfoSnapshot.hasSpilled()) {
             newPage = spillAndMaskSpilledPositions(page,
                     spillInfoSnapshot.getSpillMask(),
                     (spillBypassEnabled) ? (i, j) -> true : spillInfoSnapshot.getSpillMatcher());
@@ -338,7 +382,10 @@ public class LookupJoinOperator
                     getPartitionGenerator(),
                     operatorContext.getSpillContext().newLocalSpillContext(),
                     operatorContext.newAggregateSystemMemoryContext(),
-                    hashGenerator::hashPosition));
+                    hashGenerator::hashPosition,
+                    isSingleSessionSpiller,
+                    operatorContext.isSnapshotEnabled(),
+                    operatorContext.getDriverContext().getTaskId().getQueryId().toString()));
         }
 
         PartitioningSpillResult result = spiller.get().partitionAndSpill(page, spillMask, spillMatcher);
@@ -390,6 +437,45 @@ public class LookupJoinOperator
             lookupSourceProvider = new StaticLookupSourceProvider(new EmptyLookupSource());
         }
 
+        if (probe == null) {
+            if (unspilledMemoryPartitions.hasNext()) {
+                Page page = unspilledMemoryPartitions.next();
+                addInput(page, true);
+                if (probe == null) {
+                    return null;
+                }
+            }
+            else {
+                restoredPartition = null;
+            }
+        }
+
+        if (isSpillerRestored && finishing) {
+            if (spiller.isPresent()) {
+                Set<Integer> spilledPartitions = spiller.get().getSpilledPartitions();
+                spilledPartitions.removeAll(lookupSourceFactory.getSpilledPartitions());
+                spilledPartitionsList = new ArrayList<>(spilledPartitions);
+                isSpillerRestored = false;
+            }
+        }
+
+        if (probe == null) {
+            if (!spilledPartitionsList.isEmpty()) {
+                restoredPartition = spilledPartitionsList.remove(0);
+                if (isSingleSessionSpiller) {
+                    getFutureValue(spiller.get().flushPartition(restoredPartition));
+                    spiller.get().closeSessionSpiller(restoredPartition);
+                }
+                unspilledMemoryPartitions = spiller.get().getSpilledPages(restoredPartition);
+                if (savedRows.containsKey(restoredPartition)) {
+                    addInput(savedRows.remove(restoredPartition).row, true);
+                }
+                if (probe == null) {
+                    return null;
+                }
+            }
+        }
+
         if (probe == null && finishing && !unspilling) {
             /*
              * We do not have input probe and we won't have any, as we're finishing.
@@ -442,12 +528,20 @@ public class LookupJoinOperator
             return;
         }
 
+        if (spiller.isPresent() && isSingleSessionSpiller) {
+            if (currentPartition.isPresent()) {
+                int partition = currentPartition.get().number();
+                getFutureValue(spiller.get().flushPartition(partition));
+                spiller.get().closeSessionSpiller(partition);
+            }
+        }
+
         if (lookupPartitions == null) {
             lookupPartitions = getDone(partitionedConsumption).beginConsumption();
         }
 
         if (unspilledInputPages.hasNext()) {
-            addInput(unspilledInputPages.next());
+            addInput(unspilledInputPages.next(), false);
             return;
         }
 
@@ -466,8 +560,27 @@ public class LookupJoinOperator
             statisticsCounter.updateLookupSourcePositions(lookupSource.getJoinPositionCount());
 
             int partition = currentPartition.get().number();
-            unspilledInputPages = spiller.map(spiller -> spiller.getSpilledPages(partition))
+            if (backUpUnspilledMemoryPartitions.containsKey(partition)) {
-                    .orElse(emptyIterator());
+                unspilledInputPages = new AbstractIterator<Page>() {
+                    Iterator<Page> pageIterator = backUpUnspilledMemoryPartitions.remove(partition);
+
+                    @Override
+                    protected Page computeNext()
+                    {
+                        if (backUpRestoredPages.containsKey(partition)) {
+                            return backUpRestoredPages.remove(partition);
+                        }
+                        else if (pageIterator.hasNext()) {
+                            return pageIterator.next();
+                        }
+                        return endOfData();
+                    }
+                };
+            }
+            else {
+                unspilledInputPages = spiller.map(spiller -> spiller.getSpilledPages(partition))
+                        .orElse(emptyIterator());
+            }
 
             Optional.ofNullable(savedRows.remove(partition)).ifPresent(savedRow -> {
                 restoreProbe(
@@ -550,15 +663,28 @@ public class LookupJoinOperator
             boolean currentRowSpilled = spillInfoSnapshot.getSpillMask().test(currentRowPartition);
 
             if (currentRowSpilled) {
-                savedRows.merge(
+                if (restoredPartition != null) {
-                        currentRowPartition,
+                    verify(restoredPartition.equals(currentRowPartition), "restored partition doesn't match");
-                        new SavedRow(currentPage, currentPosition, joinPositionWithinPartition, probePositionProducedRow, joinSourcePositions),
+                    if (currentPosition > 0) {
-                        (oldValue, newValue) -> {
+                        backUpRestoredPages.put(restoredPartition, pageTail(currentPage, currentPosition));
-                            throw new IllegalStateException(format("Partition %s is already spilled", currentRowPartition));
+                    }
-                        });
+                    else {
-                joinSourcePositions = 0;
+                        backUpRestoredPages.put(restoredPartition, currentPage);
-                Page unprocessed = pageTail(currentPage, currentPosition + 1);
+                    }
-                addInput(unprocessed, spillInfoSnapshot);
+                    backUpUnspilledMemoryPartitions.put(restoredPartition, unspilledMemoryPartitions);
+                    unspilledMemoryPartitions = emptyIterator();
+                }
+                else {
+                    savedRows.merge(
+                            currentRowPartition,
+                            new SavedRow(currentPage, currentPosition, joinPositionWithinPartition, probePositionProducedRow, joinSourcePositions),
+                            (oldValue, newValue) -> {
+                                throw new IllegalStateException(format("Partition %s is already spilled", currentRowPartition));
+                            });
+                    joinSourcePositions = 0;
+                    Page unprocessed = pageTail(currentPage, currentPosition + 1);
+                    addInput(unprocessed, spillInfoSnapshot);
+                }
             }
             else {
                 Page remaining = pageTail(currentPage, currentPosition);
@@ -598,7 +724,9 @@ public class LookupJoinOperator
         verify(probe == null);
 
         addInput(probePage, spillInfoSnapshot);
-        verify(probe.advanceNextPosition());
+        if (probe != null) {
+            verify(probe.advanceNextPosition());
+        }
         this.joinPosition = joinPosition;
         this.currentProbePositionProducedRow = currentProbePositionProducedRow;
         this.joinSourcePositions = joinSourcePositions;
@@ -877,6 +1005,7 @@ public class LookupJoinOperator
         for (Map.Entry<Integer, SavedRow> entry : savedRows.entrySet()) {
             myState.savedRows.put(entry.getKey(), entry.getValue().capture(serdeProvider));
         }
+        myState.isSingleSessionSpiller = isSingleSessionSpiller;
         return myState;
     }
 
@@ -918,15 +1047,21 @@ public class LookupJoinOperator
             this.lookupPartitions = null;
         }
 
+        this.isSingleSessionSpiller = myState.isSingleSessionSpiller;
+
         if (myState.spiller != null) {
             if (!spiller.isPresent()) {
                 spiller = Optional.of(partitioningSpillerFactory.create(
                         probeTypes,
                         getPartitionGenerator(),
                         operatorContext.getSpillContext().newLocalSpillContext(),
-                        operatorContext.newAggregateSystemMemoryContext()));
+                        operatorContext.newAggregateSystemMemoryContext(),
+                        isSingleSessionSpiller,
+                        operatorContext.isSnapshotEnabled(),
+                        operatorContext.getDriverContext().getTaskId().getQueryId().toString()));
             }
             this.spiller.get().restore(myState.spiller, serdeProvider);
+            this.isSpillerRestored = true;
         }
 
         this.savedRows.clear();
@@ -963,5 +1098,6 @@ public class LookupJoinOperator
 
         private Object spiller;
         private Map<Integer, Object> savedRows;
+        private boolean isSingleSessionSpiller;
     }
 }

presto-main/src/main/java/io/prestosql/operator/LookupJoinOperatorFactory.java

@@ -29,6 +29,7 @@ import java.util.OptionalInt;
 import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Preconditions.checkState;
 import static com.google.common.collect.ImmutableList.toImmutableList;
+import static io.prestosql.SystemSessionProperties.isSpillToHdfsEnabled;
 import static io.prestosql.operator.LookupJoinOperators.JoinType.INNER;
 import static io.prestosql.operator.LookupJoinOperators.JoinType.PROBE_OUTER;
 import static java.util.Objects.requireNonNull;
@@ -158,7 +159,8 @@ public class LookupJoinOperatorFactory
                 totalOperatorsCount,
                 probeHashGenerator,
                 partitioningSpillerFactory,
-                () -> joinBridgeManager.probeOperatorFinished(driverContext.getLifespan()));
+                () -> joinBridgeManager.probeOperatorFinished(driverContext.getLifespan()),
+                isSpillToHdfsEnabled(driverContext.getPipelineContext().getTaskContext().getSession()));
     }
 
     @Override

presto-main/src/main/java/io/prestosql/operator/LookupSourceFactory.java

@@ -22,6 +22,7 @@ import io.prestosql.spi.type.Type;
 import java.util.List;
 import java.util.Map;
 import java.util.OptionalInt;
+import java.util.Set;
 import java.util.function.Supplier;
 
 import static com.google.common.util.concurrent.Futures.immediateFuture;
@@ -105,4 +106,9 @@ public interface LookupSourceFactory
     {
         throw new UnsupportedOperationException();
     }
+
+    default Set<Integer> getSpilledPartitions()
+    {
+        throw new UnsupportedOperationException();
+    }
 }

presto-main/src/main/java/io/prestosql/operator/OrderByOperator.java

@@ -75,6 +75,7 @@ public class OrderByOperator
         private final Optional<SpillerFactory> spillerFactory;
         private final OrderingCompiler orderingCompiler;
         private final boolean spillNonBlocking;
+        private final boolean isSpillToHdfsEnabled;
 
         private boolean closed;
 
@@ -90,7 +91,8 @@ public class OrderByOperator
                 boolean spillEnabled,
                 Optional<SpillerFactory> spillerFactory,
                 OrderingCompiler orderingCompiler,
-                boolean spillNonBlocking)
+                boolean spillNonBlocking,
+                boolean isSpillToHdfsEnabled)
         {
             this.operatorId = operatorId;
             this.planNodeId = requireNonNull(planNodeId, "planNodeId is null");
@@ -106,6 +108,7 @@ public class OrderByOperator
             this.orderingCompiler = requireNonNull(orderingCompiler, "orderingCompiler is null");
             checkArgument(!spillEnabled || spillerFactory.isPresent(), "Spiller Factory is not present when spill is enabled");
             this.spillNonBlocking = spillNonBlocking;
+            this.isSpillToHdfsEnabled = isSpillToHdfsEnabled;
         }
 
         @Override
@@ -125,7 +128,8 @@ public class OrderByOperator
                     spillEnabled,
                     spillerFactory,
                     orderingCompiler,
-                    spillNonBlocking);
+                    spillNonBlocking,
+                    isSpillToHdfsEnabled);
         }
 
         @Override
@@ -149,7 +153,8 @@ public class OrderByOperator
                     spillEnabled,
                     spillerFactory,
                     orderingCompiler,
-                    spillNonBlocking);
+                    spillNonBlocking,
+                    isSpillToHdfsEnabled);
         }
     }
 
@@ -198,6 +203,7 @@ public class OrderByOperator
     private boolean secondarySpillRunning;
 
     private boolean primarySpillRunning;
+    private boolean isSpillToHdfsEnabled;
 
     public OrderByOperator(
             OperatorContext operatorContext,
@@ -210,7 +216,8 @@ public class OrderByOperator
             boolean spillEnabled,
             Optional<SpillerFactory> spillerFactory,
             OrderingCompiler orderingCompiler,
-            boolean spillNonBlocking)
+            boolean spillNonBlocking,
+            boolean isSpillToHdfsEnabled)
     {
         requireNonNull(pagesIndexFactory, "pagesIndexFactory is null");
 
@@ -235,6 +242,7 @@ public class OrderByOperator
         if (!isSecondarySpillEnabled) {
             spill2InProgress = SettableFuture.create();
         }
+        this.isSpillToHdfsEnabled = isSpillToHdfsEnabled;
     }
 
     @Override
@@ -410,7 +418,9 @@ public class OrderByOperator
             spiller = Optional.of(spillerFactory.get().create(
                     sourceTypes,
                     operatorContext.getSpillContext(),
-                    operatorContext.newAggregateSystemMemoryContext()));
+                    operatorContext.newAggregateSystemMemoryContext(),
+                    operatorContext.isSnapshotEnabled(),
+                    operatorContext.getDriverContext().getTaskId().getQueryId().toString(), isSpillToHdfsEnabled));
         }
         primarySpillRunning = true;
         pageIndex.sort(sortChannels, sortOrder);
@@ -512,11 +522,29 @@ public class OrderByOperator
         return spiller.isPresent();
     }
 
+    @Override
+    public boolean isSpillToHdfsEnabled()
+    {
+        return isSpillToHdfsEnabled;
+    }
+
     @Override
     public List<Path> getSpilledFilePaths()
     {
         if (isSpilled()) {
-            return spiller.get().getSpilledFilePaths();
+            if (isSpillToHdfsEnabled) {
+                return spiller.get().getSpilledFilePaths(true);
+            }
+            return spiller.get().getSpilledFilePaths(false);
+        }
+        return ImmutableList.of();
+    }
+
+    @Override
+    public List<Pair<Path, Long>> getSpilledFileInfo()
+    {
+        if (isSpilled()) {
+            return spiller.get().getSpilledFileInfo();
         }
         return ImmutableList.of();
     }
@@ -554,6 +582,7 @@ public class OrderByOperator
         if (spiller.isPresent()) {
             myState.spiller = spiller.get().capture(serdeProvider);
         }
+        myState.isSpillToHdfsEnabled = isSpillToHdfsEnabled;
         return myState;
     }
 
@@ -576,10 +605,13 @@ public class OrderByOperator
                 spiller = Optional.of(spillerFactory.get().create(
                         sourceTypes,
                         operatorContext.getSpillContext(),
-                        operatorContext.newAggregateSystemMemoryContext()));
+                        operatorContext.newAggregateSystemMemoryContext(),
+                        operatorContext.isSnapshotEnabled(),
+                        operatorContext.getDriverContext().getTaskId().getQueryId().toString(), isSpillToHdfsEnabled));
             }
             this.spiller.get().restore(myState.spiller, serdeProvider);
         }
+        this.isSpillToHdfsEnabled = myState.isSpillToHdfsEnabled;
     }
 
     @Override
@@ -601,5 +633,6 @@ public class OrderByOperator
         private Object spiller;
         private boolean primarySpillRunning;
         private boolean secondarySpillRunning;
+        private boolean isSpillToHdfsEnabled;
     }
 }

presto-main/src/main/java/io/prestosql/operator/PartitionedLookupSourceFactory.java

@@ -33,6 +33,7 @@ import javax.annotation.concurrent.NotThreadSafe;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.HashMap;
+import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.OptionalInt;
@@ -797,6 +798,12 @@ public final class PartitionedLookupSourceFactory
         }
     }
 
+    @Override
+    public Set<Integer> getSpilledPartitions()
+    {
+        return new HashSet<>(spilledPartitions.keySet());
+    }
+
     @Override
     public Object captureJoinPositions()
     {

presto-main/src/main/java/io/prestosql/operator/TableScanOperator.java

@@ -68,6 +68,7 @@ import static com.google.common.collect.ImmutableList.toImmutableList;
 import static com.google.common.util.concurrent.Futures.immediateFuture;
 import static io.airlift.concurrent.MoreFutures.toListenableFuture;
 import static io.prestosql.SystemSessionProperties.isCrossRegionDynamicFilterEnabled;
+import static io.prestosql.SystemSessionProperties.isSpillToHdfsEnabled;
 import static io.prestosql.spi.StandardErrorCode.GENERIC_INTERNAL_ERROR;
 import static io.prestosql.spi.operator.ReuseExchangeOperator.STRATEGY.REUSE_STRATEGY_CONSUMER;
 import static io.prestosql.spi.operator.ReuseExchangeOperator.STRATEGY.REUSE_STRATEGY_DEFAULT;
@@ -528,7 +529,8 @@ public class TableScanOperator
                     if (totalPageSize(pageSpilledList) >= (spillThreshold / 2)) {
                         if (!reuseExchangeTableScanMappingIdState.getSpiller().isPresent()) {
                             Optional<Spiller> spillObject = Optional.of(spillerFactory.get().create(types, operatorContext.getSpillContext(),
-                                    operatorContext.newAggregateSystemMemoryContext()));
+                                    operatorContext.newAggregateSystemMemoryContext(), operatorContext.isSnapshotEnabled(),
+                                    operatorContext.getDriverContext().getTaskId().getQueryId().toString(), isSpillToHdfsEnabled(operatorContext.getSession())));
                             reuseExchangeTableScanMappingIdState.setSpiller(spillObject);
                         }
 

presto-main/src/main/java/io/prestosql/operator/WindowOperator.java

@@ -44,6 +44,7 @@ import io.prestosql.spi.type.Type;
 import io.prestosql.spiller.Spiller;
 import io.prestosql.spiller.SpillerFactory;
 import io.prestosql.sql.gen.OrderingCompiler;
+import org.apache.commons.lang3.tuple.Pair;
 
 import javax.annotation.Nullable;
 import javax.validation.constraints.NotNull;
@@ -98,6 +99,7 @@ public class WindowOperator
         private final boolean spillEnabled;
         private final SpillerFactory spillerFactory;
         private final OrderingCompiler orderingCompiler;
+        private final boolean isSpillToHdfsEnabled;
 
         public WindowOperatorFactory(
                 int operatorId,
@@ -114,7 +116,8 @@ public class WindowOperator
                 PagesIndex.Factory pagesIndexFactory,
                 boolean spillEnabled,
                 SpillerFactory spillerFactory,
-                OrderingCompiler orderingCompiler)
+                OrderingCompiler orderingCompiler,
+                boolean isSpillToHdfsEnabled)
         {
             requireNonNull(sourceTypes, "sourceTypes is null");
             requireNonNull(planNodeId, "planNodeId is null");
@@ -147,6 +150,7 @@ public class WindowOperator
             this.spillEnabled = spillEnabled;
             this.spillerFactory = spillerFactory;
             this.orderingCompiler = orderingCompiler;
+            this.isSpillToHdfsEnabled = isSpillToHdfsEnabled;
         }
 
         @Override
@@ -169,7 +173,8 @@ public class WindowOperator
                     pagesIndexFactory,
                     spillEnabled,
                     spillerFactory,
-                    orderingCompiler);
+                    orderingCompiler,
+                    isSpillToHdfsEnabled);
         }
 
         @Override
@@ -196,7 +201,8 @@ public class WindowOperator
                     pagesIndexFactory,
                     spillEnabled,
                     spillerFactory,
-                    orderingCompiler);
+                    orderingCompiler,
+                    isSpillToHdfsEnabled);
         }
     }
 
@@ -216,6 +222,7 @@ public class WindowOperator
     private final PagesIndexWithHashStrategies inMemoryPagesIndexWithHashStrategies;
 
     private final SingleInputSnapshotState snapshotState;
+    private boolean isSpillToHdfsEnabled;
 
     public WindowOperator(
             OperatorContext operatorContext,
@@ -231,7 +238,8 @@ public class WindowOperator
             PagesIndex.Factory pagesIndexFactory,
             boolean spillEnabled,
             SpillerFactory spillerFactory,
-            OrderingCompiler orderingCompiler)
+            OrderingCompiler orderingCompiler,
+            boolean isSpillToHdfsEnabled)
     {
         requireNonNull(operatorContext, "operatorContext is null");
         requireNonNull(outputChannels, "outputChannels is null");
@@ -329,6 +337,7 @@ public class WindowOperator
         }
 
         windowInfo = new WindowInfo.DriverWindowInfoBuilder();
+        this.isSpillToHdfsEnabled = isSpillToHdfsEnabled;
         operatorContext.setInfoSupplier(this::getWindowInfo);
     }
 
@@ -899,7 +908,9 @@ public class WindowOperator
                 spiller = Optional.of(spillerFactory.create(
                         sourceTypes,
                         operatorContext.getSpillContext(),
-                        operatorContext.newAggregateSystemMemoryContext()));
+                        operatorContext.newAggregateSystemMemoryContext(),
+                        operatorContext.isSnapshotEnabled(),
+                        operatorContext.getDriverContext().getTaskId().getQueryId().toString(), isSpillToHdfsEnabled));
             }
 
             verify(inMemoryPagesIndexWithHashStrategies.pagesIndex.getPositionCount() > 0);
@@ -1086,7 +1097,9 @@ public class WindowOperator
                     this.spiller = Optional.of(spillerFactory.create(
                             sourceTypes,
                             operatorContext.getSpillContext(),
-                            operatorContext.newAggregateSystemMemoryContext()));
+                            operatorContext.newAggregateSystemMemoryContext(),
+                            operatorContext.isSnapshotEnabled(),
+                            operatorContext.getDriverContext().getTaskId().getQueryId().toString(), isSpillToHdfsEnabled));
                 }
                 this.spiller.get().restore(myState.spiller, serdeProvider);
             }
@@ -1262,11 +1275,29 @@ public class WindowOperator
         return spillEnabled && spillablePagesToPagesIndexes.get().spiller.isPresent();
     }
 
+    @Override
+    public boolean isSpillToHdfsEnabled()
+    {
+        return isSpillToHdfsEnabled;
+    }
+
     @Override
     public List<Path> getSpilledFilePaths()
     {
         if (isSpilled()) {
-            return spillablePagesToPagesIndexes.get().spiller.get().getSpilledFilePaths();
+            if (isSpillToHdfsEnabled) {
+                return spillablePagesToPagesIndexes.get().spiller.get().getSpilledFilePaths(true);
+            }
+            return spillablePagesToPagesIndexes.get().spiller.get().getSpilledFilePaths(false);
+        }
+        return ImmutableList.of();
+    }
+
+    @Override
+    public List<Pair<Path, Long>> getSpilledFileInfo()
+    {
+        if (isSpilled()) {
+            return spillablePagesToPagesIndexes.get().spiller.get().getSpilledFileInfo();
         }
         return ImmutableList.of();
     }
@@ -1293,6 +1324,7 @@ public class WindowOperator
         for (int i = 0; i < windowFunctions.size(); i++) {
             myState.windowFunctions[i] = windowFunctions.get(i).capture(serdeProvider);
         }
+        myState.isSpillToHdfsEnabled = isSpillToHdfsEnabled;
         return myState;
     }
 
@@ -1308,6 +1340,7 @@ public class WindowOperator
         for (int i = 0; i < myState.windowFunctions.length; i++) {
             this.windowFunctions.get(i).restore(myState.windowFunctions[i], serdeProvider);
         }
+        this.isSpillToHdfsEnabled = myState.isSpillToHdfsEnabled;
     }
 
     @Override
@@ -1325,5 +1358,6 @@ public class WindowOperator
         private Object windowInfo;
         private Object inMemoryPagesIndexWithHashStrategies;
         private Object[] windowFunctions;
+        private boolean isSpillToHdfsEnabled;
     }
 }

presto-main/src/main/java/io/prestosql/operator/WorkProcessorSourceOperatorAdapter.java

@@ -41,6 +41,7 @@ import java.util.function.Supplier;
 
 import static com.google.common.collect.ImmutableList.toImmutableList;
 import static com.google.common.util.concurrent.Futures.immediateFuture;
+import static io.prestosql.SystemSessionProperties.isSpillToHdfsEnabled;
 import static io.prestosql.operator.WorkProcessor.ProcessState.blocked;
 import static io.prestosql.operator.WorkProcessor.ProcessState.finished;
 import static io.prestosql.operator.WorkProcessor.ProcessState.ofResult;
@@ -419,7 +420,9 @@ public class WorkProcessorSourceOperatorAdapter
                     if (totalPageSize(pageSpilledList) >= (spillThreshold / 2)) {
                         if (!reuseExchangeTableScanMappingIdState.getSpiller().isPresent()) {
                             Optional<Spiller> spillObject = Optional.of(spillerFactory.get().create(projectionTypes, operatorContext.getSpillContext(),
-                                    operatorContext.newAggregateSystemMemoryContext()));
+                                    operatorContext.newAggregateSystemMemoryContext(),
+                                    operatorContext.isSnapshotEnabled(),
+                                    operatorContext.getDriverContext().getTaskId().getQueryId().toString(), isSpillToHdfsEnabled(operatorContext.getSession())));
                             reuseExchangeTableScanMappingIdState.setSpiller(spillObject);
                         }
 

presto-main/src/main/java/io/prestosql/operator/aggregation/builder/SpillableHashAggregationBuilder.java

@@ -45,6 +45,7 @@ import static com.google.common.base.Verify.verify;
 import static com.google.common.collect.ImmutableList.toImmutableList;
 import static com.google.common.util.concurrent.Futures.immediateFuture;
 import static io.airlift.concurrent.MoreFutures.getFutureValue;
+import static io.prestosql.SystemSessionProperties.isSpillToHdfsEnabled;
 import static io.prestosql.operator.Operator.NOT_BLOCKED;
 import static java.lang.Math.max;
 
@@ -260,7 +261,9 @@ public class SpillableHashAggregationBuilder
             spiller = Optional.of(spillerFactory.create(
                     hashAggregationBuilder.buildTypes(),
                     operatorContext.getSpillContext(),
-                    operatorContext.newAggregateSystemMemoryContext()));
+                    operatorContext.newAggregateSystemMemoryContext(),
+                    operatorContext.isSnapshotEnabled(),
+                    operatorContext.getDriverContext().getTaskId().getQueryId().toString(), isSpillToHdfsEnabled(operatorContext.getSession())));
         }
 
         // start spilling process with current content of the hashAggregationBuilder builder...
@@ -393,7 +396,9 @@ public class SpillableHashAggregationBuilder
                 spiller = Optional.of(spillerFactory.create(
                         hashAggregationBuilder.buildTypes(),
                         operatorContext.getSpillContext(),
-                        operatorContext.newAggregateSystemMemoryContext()));
+                        operatorContext.newAggregateSystemMemoryContext(),
+                        operatorContext.isSnapshotEnabled(),
+                        operatorContext.getDriverContext().getTaskId().getQueryId().toString(), isSpillToHdfsEnabled(operatorContext.getSession())));
             }
             this.spiller.get().restore(myState.spiller, serdeProvider);
         }

presto-main/src/main/java/io/prestosql/operator/scalar/RowToRowCast.java

@@ -42,6 +42,7 @@ import io.prestosql.sql.gen.CachedInstanceBinder;
 import io.prestosql.sql.gen.CallSiteBinder;
 
 import java.lang.invoke.MethodHandle;
+import java.nio.charset.StandardCharsets;
 import java.util.List;
 
 import static com.google.common.base.Preconditions.checkArgument;
@@ -102,7 +103,7 @@ public class RowToRowCast
 
         // Embed the MD5 hash code of input and output types into the generated class name instead of the raw type names,
         // which could prevent the class name from hitting the length limitation and invalid characters.
-        byte[] md5Suffix = Hashing.md5().hashBytes((fromType + "$" + toType).getBytes()).asBytes();
+        byte[] md5Suffix = Hashing.md5().hashBytes((fromType + "$" + toType).getBytes(StandardCharsets.UTF_8)).asBytes();
 
         ClassDefinition definition = new ClassDefinition(
                 a(PUBLIC, FINAL),

presto-main/src/main/java/io/prestosql/queryeditorui/resources/AuditLogResource.java

@@ -78,6 +78,18 @@ public class AuditLogResource
         this.eventListenerManager = requireNonNull(eventListenerManager, "eventListenerManager is null");
     }
 
+    @GET
+    @Path("/isAdmin")
+    public String isAdmin(@Context HttpServletResponse response,
+                          @Context HttpServletRequest servletRequest)
+    {
+        Optional<String> filterUser = AccessControlUtil.getUserForFilter(accessControl, serverConfig, servletRequest, groupProvider);
+        if (filterUser.isPresent()) {
+            return filterUser.get();
+        }
+        return "admin";
+    }
+
     @POST
     @Path("/{type}")
     public void getAuditLog(
@@ -92,15 +104,15 @@ public class AuditLogResource
         //if the user is admin, don't filter results by user.
         Optional<String> filterUser = AccessControlUtil.getUserForFilter(accessControl, serverConfig, servletRequest, groupProvider);
 
-        if (filterUser.isPresent()) {
-            response.setStatus(Response.SC_FORBIDDEN);
-            return;
-        }
         String inputUsername = emptyToNull(username);
         String inputBeginTime = emptyToNull(beginTime);
         String inputEndTime = emptyToNull(endTime);
         String inputLevel = emptyToNull(level);
 
+        if (filterUser.isPresent()) {  // if the user is not admin, only search himself auditLog
+            inputUsername = filterUser.get();
+        }
+
         List<String> logFiles = getLogFiles(type, inputBeginTime, inputEndTime, inputUsername, inputLevel);
         if (logFiles.isEmpty()) {
             response.setStatus(Response.SC_NOT_FOUND);
@@ -148,16 +160,15 @@ public class AuditLogResource
         Optional<String> filterUser = AccessControlUtil.getUserForFilter(accessControl, serverConfig, servletRequest, groupProvider);
 
         String sessionUser = AccessControlUtil.getUser(accessControl, new HttpRequestSessionContext(servletRequest, groupProvider));
-
-        if (filterUser.isPresent()) {
-            response.setStatus(Response.SC_FORBIDDEN);
-            return;
-        }
         String inputUsername = emptyToNull(username);
         String inputBeginTime = emptyToNull(beginTime);
         String inputEndTime = emptyToNull(endTime);
         String inputLevel = emptyToNull(level);
 
+        if (filterUser.isPresent()) {  // if the user is not admin, only search himself auditLog
+            inputUsername = sessionUser;
+        }
+
         List<String> logFiles = getLogFiles(type, inputBeginTime, inputEndTime, inputUsername, inputLevel);
         if (logFiles.isEmpty()) {
             response.setStatus(Response.SC_NOT_FOUND);

presto-main/src/main/java/io/prestosql/queryhistory/QueryHistoryService.java

@@ -318,6 +318,7 @@ public class QueryHistoryService
                 ImmutableSet.of(),
                 Optional.empty(),
                 true,
-                info.getResourceGroupId(), false);
+                info.getResourceGroupId(), false,
+                false);
     }
 }

presto-main/src/main/java/io/prestosql/queryhistory/collectionsql/CollectionSqlService.java

@@ -41,7 +41,7 @@ public class CollectionSqlService
         String query = favoriteInfo.getQuery();
         String catalog = favoriteInfo.getCatalog();
         String schema = favoriteInfo.getSchemata();
-        if (hetuMetaStoreManager.getHetuMetastore().getFavorite(1, 1, user).getTotal() == queryHistoryConfig.getMaxCollectionSqlCount()) {
+        if (hetuMetaStoreManager.getHetuMetastore().getFavorite(1, 1, user).getTotal() >= queryHistoryConfig.getMaxCollectionSqlCount()) {
             return false;
         }
         hetuMetaStoreManager.getHetuMetastore().insertFavorite(new FavoriteEntity(user, query, catalog, schema));

presto-main/src/main/java/io/prestosql/server/BasicQueryInfo.java

@@ -56,6 +56,7 @@ public class BasicQueryInfo
     private final BasicQueryStats queryStats;
     private final ErrorType errorType;
     private final ErrorCode errorCode;
+    private final boolean recoveryEnabled;
 
     @JsonCreator
     public BasicQueryInfo(
@@ -70,7 +71,8 @@ public class BasicQueryInfo
             @JsonProperty("preparedQuery") Optional<String> preparedQuery,
             @JsonProperty("queryStats") BasicQueryStats queryStats,
             @JsonProperty("errorType") ErrorType errorType,
-            @JsonProperty("errorCode") ErrorCode errorCode)
+            @JsonProperty("errorCode") ErrorCode errorCode,
+            @JsonProperty("recoveryEnabled") boolean recoveryEnabled)
     {
         this.queryId = requireNonNull(queryId, "queryId is null");
         this.session = requireNonNull(session, "session is null");
@@ -84,6 +86,7 @@ public class BasicQueryInfo
         this.query = requireNonNull(query, "query is null");
         this.preparedQuery = requireNonNull(preparedQuery, "preparedQuery is null");
         this.queryStats = requireNonNull(queryStats, "queryStats is null");
+        this.recoveryEnabled = recoveryEnabled;
     }
 
     public BasicQueryInfo(QueryInfo queryInfo)
@@ -99,7 +102,8 @@ public class BasicQueryInfo
                 queryInfo.getPreparedQuery(),
                 new BasicQueryStats(queryInfo.getQueryStats()),
                 queryInfo.getErrorType(),
-                queryInfo.getErrorCode());
+                queryInfo.getErrorCode(),
+                queryInfo.isRecoveryEnabled());
     }
 
     public static BasicQueryInfo immediateFailureQueryInfo(Session session, String query, URI self, Optional<ResourceGroupId> resourceGroupId, ErrorCode errorCode)
@@ -116,7 +120,8 @@ public class BasicQueryInfo
                 Optional.empty(),
                 immediateFailureQueryStats(),
                 errorCode == null ? null : errorCode.getType(),
-                errorCode);
+                errorCode,
+                false);
     }
 
     @JsonProperty
@@ -193,6 +198,12 @@ public class BasicQueryInfo
         return errorCode;
     }
 
+    @JsonProperty
+    public boolean isRecoveryEnabled()
+    {
+        return recoveryEnabled;
+    }
+
     @Override
     public String toString()
     {

presto-main/src/main/java/io/prestosql/server/QueryResource.java

@@ -506,7 +506,7 @@ public class QueryResource
                 ImmutableSet.of(),
                 Optional.empty(),
                 true,
-                info.getResourceGroupId(), false);
+                info.getResourceGroupId(), false, false);
     }
 
     private BasicQueryInfo getBasicQueryInfo(QueryId queryId)

presto-main/src/main/java/io/prestosql/server/TaskResource.java

@@ -258,7 +258,7 @@ public class TaskResource
         return taskStatus;
     }
 
-    @GET
+    @DELETE
     @Path("{taskId}/suspend")
     @Produces({MediaType.APPLICATION_JSON, APPLICATION_JACKSON_SMILE})
     public TaskInfo suspendTask(
@@ -282,7 +282,7 @@ public class TaskResource
         return taskInfo;
     }
 
-    @GET
+    @DELETE
     @Path("{taskId}/resume")
     @Produces({MediaType.APPLICATION_JSON, APPLICATION_JACKSON_SMILE})
     public TaskInfo resumeTask(

presto-main/src/main/java/io/prestosql/server/remotetask/Backoff.java

@@ -72,11 +72,13 @@ public class Backoff
                 .toArray();
     }
 
+    @Override
     public synchronized long getFailureCount()
     {
         return failureCount;
     }
 
+    @Override
     public synchronized Duration getFailureDuration()
     {
         if (firstFailureTime == 0) {
@@ -86,16 +88,19 @@ public class Backoff
         return new Duration(value, NANOSECONDS);
     }
 
+    @Override
     public synchronized Duration getFailureRequestTimeTotal()
     {
         return new Duration(max(0, failureRequestTimeTotal), NANOSECONDS);
     }
 
+    @Override
     public synchronized void startRequest()
     {
         lastRequestStart = ticker.read();
     }
 
+    @Override
     public synchronized void success()
     {
         lastRequestStart = 0;
@@ -118,6 +123,7 @@ public class Backoff
      * @return true if the failure is considered permanent
      * min retried and maxErrorDuration is passed.
      */
+    @Override
     public synchronized boolean failure()
     {
         long now = ticker.read();
@@ -143,6 +149,7 @@ public class Backoff
         return failureDuration >= maxFailureIntervalNanos;
     }
 
+    @Override
     public synchronized long getBackoffDelayNanos()
     {
         int tmpFailureCount = (int) min(backoffDelayIntervalsNanos.length, getFailureCount());

presto-main/src/main/java/io/prestosql/server/remotetask/HttpRemoteTask.java

@@ -176,6 +176,8 @@ public final class HttpRemoteTask
     private final AtomicBoolean aborting = new AtomicBoolean(false);
     private final AtomicBoolean abandoned = new AtomicBoolean(false);
     private final AtomicBoolean cancelledToResume = new AtomicBoolean(false);
+
+    private final AtomicBoolean suspending = new AtomicBoolean(false);
     private final boolean isBinaryEncoding;
     private Optional<PlanNodeId> parent;
 
@@ -678,7 +680,7 @@ public final class HttpRemoteTask
                 .setUri(uriBuilder.build())
                 .addHeader(PRESTO_TASK_INSTANCE_ID, instanceId)
                 .build();
-        scheduleAsyncCleanupRequest(createCleanupBackoff(), request, targetState.toString()); //todo(nitin) check if separate response handler needed here?
+        scheduleAsyncSuspendRequest(createCleanupBackoff(), request, targetState.toString()); //todo(nitin) check if separate response handler needed here?
     }
 
     @Override
@@ -704,7 +706,7 @@ public final class HttpRemoteTask
                 .setUri(uriBuilder.build())
                 .addHeader(PRESTO_TASK_INSTANCE_ID, instanceId)
                 .build();
-        scheduleAsyncCleanupRequest(createCleanupBackoff(), request, targetState.toString()); //todo(nitin) check if separate response handler needed here?
+        scheduleAsyncSuspendRequest(createCleanupBackoff(), request, targetState.toString()); //todo(nitin) check if separate response handler needed here?
     }
 
     private synchronized void cleanUpTask(TaskState newState)
@@ -767,6 +769,16 @@ public final class HttpRemoteTask
         doScheduleAsyncCleanupRequest(cleanupBackoff, request, action);
     }
 
+    private void scheduleAsyncSuspendRequest(Backoff cleanupBackoff, Request request, String action)
+    {
+        if (!suspending.compareAndSet(false, true)) {
+            // Do not initiate another round of cleanup requests if one had been initiated.
+            // Otherwise, we can get into an asynchronous recursion here. For example, when aborting a task after REMOTE_TASK_MISMATCH.
+            return;
+        }
+        doScheduleAsyncCleanupRequest(cleanupBackoff, request, action);
+    }
+
     private void doScheduleAsyncCleanupRequest(Backoff cleanupBackoff, Request request, String action)
     {
         ResponseHandler responseHandler;
@@ -849,6 +861,8 @@ public final class HttpRemoteTask
                     // Check for task state is in QueryInfo#areAllStagesDone.
                     taskStatus = TaskStatus.failWith(taskStatus, CANCELED_TO_RESUME, ImmutableList.of());
                 }
+
+                suspending.compareAndSet(true, false);
                 updateTaskInfo(getTaskInfo().withTaskStatus(taskStatus));
             }
         }, executor);

presto-main/src/main/java/io/prestosql/snapshot/RecoveryConfig.java

@@ -41,10 +41,12 @@ public class RecoveryConfig
     public static final String SNAPSHOT_USE_KRYO_SERIALIZATION = "hetu.snapshot.useKryoSerialization";
     public static final String SPILLER_SPILL_PROFILE = "experimental.spiller-spill-profile";
     public static final String SPILLER_SPILL_TO_HDFS = "experimental.spiller-spill-to-hdfs";
+    public static final String ELIMINATE_DUPLICATE_SPILL_FILES = "experimental.eliminate-duplicate-spill-files";
 
     private String snapshotProfile;
     private String spillProfile;
     private boolean spillToHdfs;
+    private boolean eliminateDuplicateSpillFilesEnabled;
 
     private long recoveryMaxRetries = 10;
     private Duration recoveryRetryTimeout = new Duration(10, TimeUnit.MINUTES);
@@ -190,4 +192,17 @@ public class RecoveryConfig
         this.spillToHdfs = spillToHdfs;
         return this;
     }
+
+    public boolean isEliminateDuplicateSpillFilesEnabled()
+    {
+        return eliminateDuplicateSpillFilesEnabled;
+    }
+
+    @Config(ELIMINATE_DUPLICATE_SPILL_FILES)
+    @ConfigDescription("eliminate back up of duplicate spill files")
+    public RecoveryConfig setEliminateDuplicateSpillFilesEnabled(boolean eliminateDuplicateSpillFilesEnabled)
+    {
+        this.eliminateDuplicateSpillFilesEnabled = eliminateDuplicateSpillFilesEnabled;
+        return this;
+    }
 }

presto-main/src/main/java/io/prestosql/snapshot/RecoveryUtils.java

@@ -44,6 +44,7 @@ import java.io.OutputStream;
 import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.util.Collection;
+import java.util.List;
 import java.util.Map;
 import java.util.Optional;
 import java.util.Set;
@@ -171,13 +172,13 @@ public class RecoveryUtils
         return snapshotStoreClient.loadState(snapshotStateId, dataCollector);
     }
 
-    public void storeFile(SnapshotStateId snapshotStateId, Path sourceFile, SnapshotDataCollector dataCollector)
+    public void storeFile(SnapshotStateId snapshotStateId, Path sourceFile, SnapshotDataCollector dataCollector, long skipBytes)
             throws Exception
     {
         requireNonNull(snapshotStoreClient);
         requireNonNull(sourceFile);
 
-        snapshotStoreClient.storeFile(snapshotStateId, sourceFile, dataCollector);
+        snapshotStoreClient.storeFile(snapshotStateId, sourceFile, dataCollector, skipBytes);
     }
 
     public Boolean loadFile(SnapshotStateId snapshotStateId, Path targetFile, SnapshotDataCollector dataCollector)
@@ -189,6 +190,15 @@ public class RecoveryUtils
         return snapshotStoreClient.loadFile(snapshotStateId, targetFile, dataCollector);
     }
 
+    public Boolean loadFiles(Map<Path, List<SnapshotStateId>> snapshotSpillMap, SnapshotDataCollector dataCollector)
+            throws Exception
+    {
+        requireNonNull(snapshotStoreClient);
+        requireNonNull(snapshotSpillMap);
+
+        return snapshotStoreClient.loadFiles(snapshotSpillMap, dataCollector);
+    }
+
     public void storeSnapshotResult(String queryId, Map<Long, SnapshotInfo> result)
             throws Exception
     {
@@ -366,4 +376,16 @@ public class RecoveryUtils
     {
         recoveryManagers.remove(queryId);
     }
+
+    public void storeSpilledPathInfo(SnapshotStateId spillId, Object snapshotSpillPaths)
+            throws IOException
+    {
+        snapshotStoreClient.storeSpilledPathInfo(spillId, snapshotSpillPaths);
+    }
+
+    public Map<Long, List<String>> loadSpilledPathInfo(SnapshotStateId spillId)
+            throws IOException, ClassNotFoundException
+    {
+        return snapshotStoreClient.loadSpilledPathInfo(spillId);
+    }
 }

presto-main/src/main/java/io/prestosql/snapshot/SingleInputSnapshotState.java

@@ -23,15 +23,21 @@ import io.prestosql.operator.OperatorContext;
 import io.prestosql.spi.Page;
 import io.prestosql.spi.snapshot.MarkerPage;
 import io.prestosql.spi.snapshot.Restorable;
+import org.apache.commons.lang3.tuple.Pair;
 
 import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
 import java.util.LinkedList;
 import java.util.List;
+import java.util.Map;
 import java.util.Optional;
 import java.util.Queue;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.TimeUnit;
 import java.util.function.Function;
 
+import static io.prestosql.SystemSessionProperties.isEliminateDuplicateSpillFilesEnabled;
 import static java.util.Objects.requireNonNull;
 
 /**
@@ -58,6 +64,10 @@ public class SingleInputSnapshotState
     private final Queue<MarkerPage> markers = new LinkedList<>();
     // For recording snapshot memory usage
     private final LocalMemoryContext snapshotMemoryContext;
+    private Map<Path, Long> spillFileSizeMap = new ConcurrentHashMap<>();
+    Map<Long, List<String>> snapshotSpillPaths = new LinkedHashMap<>();
+    private final boolean isEliminateDuplicateSpillFilesEnabled;
+    long lastSnapshotId = -1;
 
     public static SingleInputSnapshotState forOperator(Operator operator, OperatorContext operatorContext)
     {
@@ -67,7 +77,8 @@ public class SingleInputSnapshotState
                 operatorContext.getDriverContext().getSerde(),
                 snapshotId -> SnapshotStateId.forOperator(snapshotId, operatorContext),
                 snapshotId -> SnapshotStateId.forDriverComponent(snapshotId, operatorContext, operatorContext.getOperatorId() + "-spill"),
-                operatorContext.newLocalUserMemoryContext(SingleInputSnapshotState.class.getSimpleName()));
+                operatorContext.newLocalUserMemoryContext(SingleInputSnapshotState.class.getSimpleName()),
+                isEliminateDuplicateSpillFilesEnabled(operatorContext.getDriverContext().getSession()));
     }
 
     SingleInputSnapshotState(Restorable restorable,
@@ -75,7 +86,8 @@ public class SingleInputSnapshotState
                              PagesSerde pagesSerde,
                              Function<Long, SnapshotStateId> snapshotStateIdGenerator,
                              Function<Long, SnapshotStateId> spillStateIdGenerator,
-                             LocalMemoryContext snapshotMemoryContext)
+                             LocalMemoryContext snapshotMemoryContext,
+                             boolean isEliminateDuplicateSpillFilesEnabled)
     {
         this.restorable = requireNonNull(restorable, "restorable is null");
         this.restorableId = String.format("%s (%s)", restorable.getClass().getSimpleName(), snapshotStateIdGenerator.apply(0L).getId());
@@ -84,6 +96,7 @@ public class SingleInputSnapshotState
         this.spillStateIdGenerator = requireNonNull(spillStateIdGenerator, "spillStateIdGenerator is null");
         this.pagesSerde = pagesSerde;
         this.snapshotMemoryContext = snapshotMemoryContext;
+        this.isEliminateDuplicateSpillFilesEnabled = isEliminateDuplicateSpillFilesEnabled;
     }
 
     public void close()
@@ -128,7 +141,7 @@ public class SingleInputSnapshotState
                     restorable.restore(state.get(), pagesSerde);
                     timer.stop();
                     boolean successful = true;
-                    if (restorable instanceof Spillable && ((Spillable) restorable).isSpilled()) {
+                    if (restorable instanceof Spillable && ((Spillable) restorable).isSpilled() && !((Spillable) restorable).isSpillToHdfsEnabled()) {
                         Boolean result = loadSpilledFiles(snapshotId, (Spillable) restorable);
                         if (result == null) {
                             snapshotManager.failedToRestore(componentId, true);
@@ -178,8 +191,13 @@ public class SingleInputSnapshotState
         }
         try {
             storeState(componentId);
-            if (restorable instanceof Spillable && ((Spillable) restorable).isSpilled()) {
+            if (restorable instanceof Spillable && !((Spillable) restorable).isSpillToHdfsEnabled()) {
-                storeSpilledFiles(snapshotId, (Spillable) restorable);
+                if (((Spillable) restorable).isSpilled()) {
+                    storeSpilledFiles(snapshotId, (Spillable) restorable, true);
+                }
+                else {
+                    storeSpilledFiles(snapshotId, (Spillable) restorable, false);
+                }
             }
             if (record) {
                 snapshotManager.succeededToCapture(componentId);
@@ -233,13 +251,39 @@ public class SingleInputSnapshotState
         return marker;
     }
 
-    private void storeSpilledFiles(long snapshotId, Spillable spillable)
+    private void storeSpilledFiles(long snapshotId, Spillable spillable, boolean isSpilled)
             throws Exception
     {
-        List<Path> filePaths = spillable.getSpilledFilePaths();
+        if (!isEliminateDuplicateSpillFilesEnabled) {
-        SnapshotStateId spillId = spillStateIdGenerator.apply(snapshotId);
+            List<Path> filePaths = spillable.getSpilledFilePaths();
-        for (Path path : filePaths) {
+            SnapshotStateId spillId = spillStateIdGenerator.apply(snapshotId);
-            snapshotManager.storeFile(spillId, path);
+            for (Path path : filePaths) {
+                snapshotManager.storeFile(spillId, path, 0);
+            }
+        }
+        else {
+            SnapshotStateId spillId = spillStateIdGenerator.apply(snapshotId);
+            snapshotSpillPaths.putIfAbsent(snapshotId, new ArrayList<>());
+            if (isSpilled) {
+                List<Pair<Path, Long>> spilledFilesInfo = spillable.getSpilledFileInfo();
+                for (Pair<Path, Long> spillFileInfo : spilledFilesInfo) {
+                    if (spillFileSizeMap.size() != 0) {
+                        if (spillFileSizeMap.containsKey(spillFileInfo.getLeft())) {
+                            long delta = spillFileInfo.getRight() - spillFileSizeMap.get(spillFileInfo.getLeft());
+                            if (delta != 0) {
+                                snapshotManager.storeFile(spillId, spillFileInfo.getLeft(), spillFileSizeMap.get(spillFileInfo.getLeft()).longValue());
+                                spillFileSizeMap.put(spillFileInfo.getLeft(), spillFileInfo.getRight());
+                                snapshotSpillPaths.get(snapshotId).add(spillFileInfo.getLeft().toString());
+                            }
+                            continue;
+                        }
+                    }
+                    snapshotManager.storeFile(spillId, spillFileInfo.getLeft(), 0);
+                    spillFileSizeMap.put(spillFileInfo.getLeft(), spillFileInfo.getRight());
+                    snapshotSpillPaths.get(snapshotId).add(spillFileInfo.getLeft().toString());
+                }
+            }
+            snapshotManager.storeSpilledPathInfo(spillId, snapshotSpillPaths);
         }
     }
 
@@ -249,10 +293,59 @@ public class SingleInputSnapshotState
     private Boolean loadSpilledFiles(long snapshotId, Spillable spillable)
             throws Exception
     {
-        List<Path> filePaths = spillable.getSpilledFilePaths();
+        if (!isEliminateDuplicateSpillFilesEnabled) {
-        SnapshotStateId spillId = spillStateIdGenerator.apply(snapshotId);
+            List<Path> filePaths = spillable.getSpilledFilePaths();
-        for (Path path : filePaths) {
+            SnapshotStateId spillId = spillStateIdGenerator.apply(snapshotId);
-            Boolean result = snapshotManager.loadFile(spillId, path);
+            for (Path path : filePaths) {
+                Boolean result = snapshotManager.loadFile(spillId, path);
+                if (result == null || !result) {
+                    return result;
+                }
+            }
+        }
+        else {
+            SnapshotStateId lastSpillId = spillStateIdGenerator.apply(snapshotId);
+            List<Path> filePaths = spillable.getSpilledFilePaths();
+            if (filePaths.size() == 0) {
+                return true;
+            }
+            List<Path> loadPaths = new ArrayList<>();
+            if (snapshotId != lastSnapshotId || snapshotSpillPaths.size() == 0) {
+                snapshotSpillPaths = snapshotManager.loadSpilledPathInfo(lastSpillId);
+                lastSnapshotId = snapshotId;
+            }
+            if (snapshotSpillPaths == null) {
+                return null;
+            }
+            Map<Path, List<SnapshotStateId>> snapshotSpillMap = new LinkedHashMap<>();
+
+            for (Long snapshot : snapshotSpillPaths.keySet()) {
+                SnapshotStateId spillId = spillStateIdGenerator.apply(snapshot.longValue());
+                if (snapshotSpillPaths.get(snapshot).size() == 0) {
+                    continue;
+                }
+                for (Path path : filePaths) {
+                    if (snapshotSpillPaths.get(snapshot).contains(path.toString())) {
+                        if (loadPaths.contains(path)) {
+                            SnapshotStateId loadedSpillId = snapshotManager.loadSpilledFile(spillId);
+                            if (loadedSpillId == null) {
+                                return null;
+                            }
+                            snapshotSpillMap.get(path).add(loadedSpillId);
+                        }
+                        else {
+                            SnapshotStateId loadedSpillId = snapshotManager.loadSpilledFile(spillId);
+                            if (loadedSpillId == null) {
+                                return null;
+                            }
+                            snapshotSpillMap.put(path, new LinkedList<>());
+                            snapshotSpillMap.get(path).add(loadedSpillId);
+                            loadPaths.add(path);
+                        }
+                    }
+                }
+            }
+            Boolean result = snapshotManager.loadFiles(snapshotSpillMap);
             if (result == null || !result) {
                 return result;
             }

presto-main/src/main/java/io/prestosql/snapshot/SnapshotFileBasedClient.java

@@ -107,7 +107,7 @@ public class SnapshotFileBasedClient
     }
 
     @Override
-    public void storeFile(SnapshotStateId snapshotStateId, Path sourceFile, SnapshotDataCollector dataCollector)
+    public void storeFile(SnapshotStateId snapshotStateId, Path sourceFile, SnapshotDataCollector dataCollector, long skipBytes)
             throws IOException
     {
         Stopwatch timer = Stopwatch.createStarted();
@@ -120,6 +120,7 @@ public class SnapshotFileBasedClient
         try (OutputStream outputStream = fsClient.newOutputStream(file);
                 HetuFileSystemClient spillFs = getSpillerFileSystemClient(sourceFile);
                 InputStream inputStream = spillFs.newInputStream(sourceFile)) {
+            ByteStreams.skipFully(inputStream, skipBytes);
             ByteStreams.copy(inputStream, outputStream);
         }
         timer.stop();
@@ -164,6 +165,48 @@ public class SnapshotFileBasedClient
         return true;
     }
 
+    @Override
+    public boolean loadFiles(Map<Path, List<SnapshotStateId>> snapshotSpillMap, SnapshotDataCollector dataCollector)
+            throws Exception
+    {
+        Stopwatch timer = Stopwatch.createStarted();
+
+        for (Map.Entry<Path, List<SnapshotStateId>> snapshotSpillMapEntry : snapshotSpillMap.entrySet()) {
+            Path targetPath = snapshotSpillMapEntry.getKey();
+
+            targetPath.getParent().toFile().mkdirs();
+
+            try (HetuFileSystemClient spillFs = getSpillerFileSystemClient(targetPath);
+                    OutputStream outputStream = spillFs.newOutputStream(targetPath)) {
+                for (SnapshotStateId snapshotStateId : snapshotSpillMapEntry.getValue()) {
+                    List<String> hierarchy = new ArrayList<>(snapshotStateId.getHierarchy());
+                    String fileName = targetPath.getFileName().toString();
+                    hierarchy.add(fileName);
+                    Path file = RecoveryUtils.createStatePath(rootPath, hierarchy);
+
+                    if (!fsClient.exists(file)) {
+                        LOG.warn("File: %s does not exist under %s", targetPath.getFileName().toString(), snapshotStateId);
+                        return false;
+                    }
+
+                    try (InputStream inputStream = fsClient.newInputStream(file)) {
+                        ByteStreams.copy(inputStream, outputStream);
+                    }
+                    Long size;
+                    if (dataCollector != null) {
+                        size = (Long) fsClient.getAttribute(file, "size");
+                        dataCollector.updateSnapshotRestoreSize(size.longValue());
+                    }
+                }
+            }
+        }
+        timer.stop();
+        if (dataCollector != null) {
+            dataCollector.updateSnapshotRestoreCpuTime(timer.elapsed(TimeUnit.MILLISECONDS));
+        }
+        return true;
+    }
+
     @Override
     public void deleteAll(String queryId)
             throws IOException
@@ -228,8 +271,40 @@ public class SnapshotFileBasedClient
         }
     }
 
+    @Override
+    public void storeSpilledPathInfo(SnapshotStateId spillId, Object snapshotSpillPaths)
+            throws IOException
+    {
+        List<String> hierarchy = new ArrayList<>(spillId.getHierarchy());
+        hierarchy.add("snapshotSpillPaths");
+        Path file = RecoveryUtils.createStatePath(rootPath, hierarchy);
+
+        fsClient.createDirectories(file.getParent());
+
+        try (ObjectOutputStream oos = new ObjectOutputStream(fsClient.newOutputStream(file))) {
+            oos.writeObject(snapshotSpillPaths);
+        }
+    }
+
+    @Override
+    public Map<Long, List<String>> loadSpilledPathInfo(SnapshotStateId spillId)
+            throws IOException, ClassNotFoundException
+    {
+        List<String> hierarchy = new ArrayList<>(spillId.getHierarchy());
+        hierarchy.add("snapshotSpillPaths");
+        Path file = RecoveryUtils.createStatePath(rootPath, hierarchy);
+
+        if (!fsClient.exists(file)) {
+            return null;
+        }
+
+        try (ObjectInputStream ois = new ObjectInputStream(fsClient.newInputStream(file))) {
+            return (Map<Long, List<String>>) ois.readObject();
+        }
+    }
+
     private HetuFileSystemClient getSpillerFileSystemClient(Path filePath) throws IOException
     {
-        return !spillToHdfs && spillProfile == null ? fileSystemClientManager.getFileSystemClient(filePath) : fileSystemClientManager.getFileSystemClient(spillProfile, filePath);
+        return !spillToHdfs || spillProfile == null ? fileSystemClientManager.getFileSystemClient(filePath) : fileSystemClientManager.getFileSystemClient(spillProfile, filePath);
     }
 }

presto-main/src/main/java/io/prestosql/snapshot/SnapshotStoreClient.java

@@ -14,7 +14,9 @@
  */
 package io.prestosql.snapshot;
 
+import java.io.IOException;
 import java.nio.file.Path;
+import java.util.List;
 import java.util.Map;
 import java.util.Optional;
 import java.util.Set;
@@ -39,7 +41,7 @@ public interface SnapshotStoreClient
     /**
      * Store file from sourcePath to snapshotStateId of snapshot store
      */
-    void storeFile(SnapshotStateId snapshotStateId, Path sourcePath, SnapshotDataCollector dataCollector)
+    void storeFile(SnapshotStateId snapshotStateId, Path sourcePath, SnapshotDataCollector dataCollector, long skipBytes)
             throws Exception;
 
     /**
@@ -48,6 +50,12 @@ public interface SnapshotStoreClient
     boolean loadFile(SnapshotStateId snapshotStateId, Path targetPath, SnapshotDataCollector dataCollector)
             throws Exception;
 
+    /**
+     * Load Spilled Files for all SnapshotStateIds
+     */
+    boolean loadFiles(Map<Path, List<SnapshotStateId>> snapshotSpillMap, SnapshotDataCollector dataCollector)
+            throws Exception;
+
     /**
      * Delete everything under query
      */
@@ -77,4 +85,10 @@ public interface SnapshotStoreClient
      */
     Set<String> loadConsolidatedFiles(String queryId)
             throws Exception;
+
+    void storeSpilledPathInfo(SnapshotStateId spillId, Object snapshotSpillPaths)
+            throws IOException;
+
+    Map<Long, List<String>> loadSpilledPathInfo(SnapshotStateId spillId)
+            throws IOException, ClassNotFoundException;
 }

presto-main/src/main/java/io/prestosql/snapshot/Spillable.java

@@ -14,6 +14,8 @@
  */
 package io.prestosql.snapshot;
 
+import org.apache.commons.lang3.tuple.Pair;
+
 import java.nio.file.Path;
 import java.util.ArrayList;
 import java.util.List;
@@ -33,6 +35,16 @@ public interface Spillable
         return false;
     }
 
+    /**
+     * Determine if spilling to HDFS is enabled.
+     *
+     * @return true if enabled
+     */
+    default boolean isSpillToHdfsEnabled()
+    {
+        return false;
+    }
+
     /**
      * Get spilled file paths of an object
      *
@@ -42,4 +54,9 @@ public interface Spillable
     {
         return new ArrayList<>();
     }
+
+    default List<Pair<Path, Long>> getSpilledFileInfo()
+    {
+        return new ArrayList<>();
+    }
 }

presto-main/src/main/java/io/prestosql/snapshot/TaskSnapshotManager.java

@@ -77,6 +77,11 @@ public class TaskSnapshotManager
         return resumeCount;
     }
 
+    public TaskId getTaskId()
+    {
+        return taskId;
+    }
+
     public QuerySnapshotManager getQuerySnapshotManager()
     {
         return recoveryUtils.getQuerySnapshotManager(taskId.getQueryId());
@@ -195,10 +200,10 @@ public class TaskSnapshotManager
         return loadedValue;
     }
 
-    public void storeFile(SnapshotStateId snapshotStateId, Path sourceFile)
+    public void storeFile(SnapshotStateId snapshotStateId, Path sourceFile, long skipBytes)
             throws Exception
     {
-        recoveryUtils.storeFile(snapshotStateId, sourceFile, this);
+        recoveryUtils.storeFile(snapshotStateId, sourceFile, this, skipBytes);
         // store dummy value
         Map<String, Object> map = storeCache.computeIfAbsent(snapshotStateId.getSnapshotId(), (x) -> Collections.synchronizedMap(new HashMap<>()));
         map.put(snapshotStateId.toString(), snapshotStateId.toString());
@@ -219,6 +224,22 @@ public class TaskSnapshotManager
         return recoveryUtils.loadFile(SnapshotStateId.fromString((String) loadedValue.get()), targetFile, this);
     }
 
+    public Boolean loadFiles(Map<Path, List<SnapshotStateId>> snapshotSpillMap)
+            throws Exception
+    {
+        return recoveryUtils.loadFiles(snapshotSpillMap, this);
+    }
+
+    public SnapshotStateId loadSpilledFile(SnapshotStateId snapshotStateId)
+            throws Exception
+    {
+        Optional<Object> loadedValue = loadWithBacktrack(snapshotStateId);
+        if (!loadedValue.isPresent() || loadedValue.get() == NO_STATE) {
+            return null;
+        }
+        return SnapshotStateId.fromString((String) loadedValue.get());
+    }
+
     private OptionalLong getPreviousSnapshotIdIfComplete(Map<Long, SnapshotInfo> snapshotToSnapshotResultMap, long snapshotId)
     {
         try {
@@ -460,4 +481,24 @@ public class TaskSnapshotManager
     {
         return String.format(Locale.ENGLISH, "%s, with total component %d", taskId, totalComponents);
     }
+
+    public void storeSpilledPathInfo(SnapshotStateId spillId, Object snapshotSpillPaths)
+            throws Exception
+    {
+        recoveryUtils.storeSpilledPathInfo(spillId, snapshotSpillPaths);
+        Map<String, Object> map = storeCache.computeIfAbsent(spillId.getSnapshotId(), (x) -> Collections.synchronizedMap(new HashMap<>()));
+        map.put(spillId.toString(), spillId.toString());
+    }
+
+    public Map<Long, List<String>> loadSpilledPathInfo(SnapshotStateId spillId)
+            throws Exception
+    {
+        Optional<Object> loadedValue = loadWithBacktrack(spillId);
+        if (!loadedValue.isPresent() || loadedValue.get() == NO_STATE) {
+            return null;
+        }
+        else {
+            return recoveryUtils.loadSpilledPathInfo(SnapshotStateId.fromString((String) loadedValue.get()));
+        }
+    }
 }

presto-main/src/main/java/io/prestosql/spiller/FileHolder.java

@@ -35,14 +35,16 @@ final class FileHolder
 {
     private final Path filePath;
     private final HetuFileSystemClient fileSystemClient;
+    private final boolean isSpillToHdfsAndSnapshotEnabled;
 
     @GuardedBy("this")
     private boolean deleted;
 
-    public FileHolder(Path filePath, HetuFileSystemClient fileSystemClient)
+    public FileHolder(Path filePath, HetuFileSystemClient fileSystemClient, boolean isSpillToHdfsAndSnapshotEnabled)
     {
         this.filePath = requireNonNull(filePath, "filePath is null");
         this.fileSystemClient = requireNonNull(fileSystemClient, "fileSystemClient is null");
+        this.isSpillToHdfsAndSnapshotEnabled = isSpillToHdfsAndSnapshotEnabled;
     }
 
     public synchronized OutputStream newOutputStream(OpenOption... options)
@@ -68,7 +70,9 @@ final class FileHolder
         deleted = true;
 
         try {
-            fileSystemClient.deleteIfExists(filePath);
+            if (!isSpillToHdfsAndSnapshotEnabled) {
+                fileSystemClient.deleteIfExists(filePath);
+            }
         }
         catch (IOException e) {
             throw new UncheckedIOException(e);
@@ -79,4 +83,10 @@ final class FileHolder
     {
         return filePath;
     }
+
+    public long getFileSize()
+            throws IOException
+    {
+        return (Long) fileSystemClient.getAttribute(filePath, "size");
+    }
 }

presto-main/src/main/java/io/prestosql/spiller/FileSingleSessionStreamSpiller.java

@@ -0,0 +1,218 @@
+/*
+ * Copyright (C) 2018-2022. Huawei Technologies Co., Ltd. All rights reserved.
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.prestosql.spiller;
+
+import com.esotericsoftware.kryo.io.Output;
+import com.google.common.base.Stopwatch;
+import io.airlift.slice.OutputStreamSliceOutput;
+import io.airlift.slice.SliceOutput;
+import io.hetu.core.transport.execution.buffer.PagesSerde;
+import io.hetu.core.transport.execution.buffer.SerializedPage;
+import io.prestosql.spi.Page;
+import io.prestosql.spi.PrestoException;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import static com.google.common.base.Preconditions.checkState;
+import static io.hetu.core.transport.execution.buffer.PagesSerdeUtil.writeSerializedPage;
+import static io.prestosql.spi.StandardErrorCode.GENERIC_INTERNAL_ERROR;
+
+public class FileSingleSessionStreamSpiller
+{
+    private SliceOutput output;
+    private OutputStream outputStream;
+    private FileHolder sessionTargetFile;
+    private boolean useSessionDirectSerde;
+    private boolean closed;
+
+    void setTargetFile(FileHolder targetFile, boolean useDirectSerde)
+    {
+        this.sessionTargetFile = targetFile;
+        this.useSessionDirectSerde = useDirectSerde;
+    }
+
+    void setOutput(SliceOutput output)
+    {
+        this.output = output;
+    }
+
+    void setOutputStream(OutputStream outputStream)
+    {
+        this.outputStream = outputStream;
+    }
+
+    void closeSession()
+    {
+        if (!closed && (output != null || outputStream != null)) {
+            try {
+                if (useSessionDirectSerde) {
+                    checkState(outputStream != null, "output stream is null");
+                    outputStream.close();
+                    outputStream = null;
+                }
+                else {
+                    checkState(output != null, "output is null");
+                    output.close();
+                    output = null;
+                }
+                closed = true;
+            }
+            catch (UncheckedIOException | IOException e) {
+                throw new PrestoException(GENERIC_INTERNAL_ERROR, "Failed to close outputStream", e);
+            }
+        }
+    }
+
+    protected Stats writePages(boolean writable, Iterator<Page> pageIterator, PagesSerde serde)
+    {
+        long totalSpilledPagesInMemorySize = 0;
+        long totalBytesWritten = 0;
+        long totalSpilledBytes = 0;
+        long totalWriteTime;
+        List<Long> pageSizesList = new LinkedList<>();
+
+        synchronized (output) {
+            checkState(writable, "Spilling no longer allowed. The spiller has been made non-writable on first read for subsequent reads to be consistent");
+            Stopwatch timer = Stopwatch.createStarted();
+            while (pageIterator.hasNext()) {
+                Page page = pageIterator.next();
+                totalSpilledPagesInMemorySize += page.getSizeInBytes();
+                SerializedPage serializedPage = serde.serialize(page);
+                long pageSize = serializedPage.getSizeInBytes();
+                totalBytesWritten += pageSize;
+                totalSpilledBytes += pageSize;
+                pageSizesList.add(pageSize);
+                writeSerializedPage(output, serializedPage);
+            }
+            try {
+                output.flush();
+            }
+            catch (IOException ioException) {
+                throw new PrestoException(GENERIC_INTERNAL_ERROR, "Failed to flush output", ioException);
+            }
+            timer.stop();
+            totalWriteTime = timer.elapsed(TimeUnit.MILLISECONDS);
+            return new Stats(totalSpilledPagesInMemorySize, totalBytesWritten, totalSpilledBytes, totalWriteTime, pageSizesList);
+        }
+    }
+
+    protected Stats writePagesDirect(boolean writable, Iterator<Page> pageIterator, PagesSerde serde)
+    {
+        long totalSpilledPagesInMemorySize = 0;
+        long totalBytesWritten = 0;
+        long totalSpilledBytes = 0;
+        long totalWriteTime;
+        List<Long> pageSizesList = new LinkedList<>();
+
+        synchronized (outputStream) {
+            checkState(writable, "Spilling no longer allowed. The spiller has been made non-writable on first read for subsequent reads to be consistent");
+            Stopwatch timer = Stopwatch.createStarted();
+            while (pageIterator.hasNext()) {
+                Page page = pageIterator.next();
+                long pageSize = page.getSizeInBytes();
+                totalSpilledPagesInMemorySize += pageSize;
+                totalBytesWritten += pageSize;
+                totalSpilledBytes += pageSize;
+                pageSizesList.add(pageSize);
+                serde.serialize(outputStream, page);
+            }
+            try {
+                outputStream.flush();
+            }
+            catch (IOException ioException) {
+                throw new PrestoException(GENERIC_INTERNAL_ERROR, "Failed to flush outputStream", ioException);
+            }
+            timer.stop();
+            totalWriteTime = timer.elapsed(TimeUnit.MILLISECONDS);
+            return new Stats(totalSpilledPagesInMemorySize, totalBytesWritten, totalSpilledBytes, totalWriteTime, pageSizesList);
+        }
+    }
+
+    long getTargetFileSize()
+    {
+        if (useSessionDirectSerde) {
+            try {
+                outputStream.flush();
+            }
+            catch (IOException e) {
+                throw new PrestoException(GENERIC_INTERNAL_ERROR, "Failed to flush outputStream", e);
+            }
+            if (outputStream instanceof Output) {
+                return ((Output) outputStream).position();
+            }
+            else {
+                return ((OutputStreamSliceOutput) outputStream).size();
+            }
+        }
+        else {
+            try {
+                output.flush();
+            }
+            catch (IOException e) {
+                throw new PrestoException(GENERIC_INTERNAL_ERROR, "Failed to flush output", e);
+            }
+            return output.size();
+        }
+    }
+
+    protected class Stats
+    {
+        private final long totalSpilledPagesInMemorySize;
+        private final long totalBytesWritten;
+        private final long totalSpilledBytes;
+        private final long totalWriteTime;
+        private final List<Long> pageSizesList;
+
+        Stats(long totalSpilledPagesInMemorySize, long totalBytesWritten, long totalSpilledBytes, long totalWriteTime, List<Long> pageSizesList)
+        {
+            this.totalSpilledPagesInMemorySize = totalSpilledPagesInMemorySize;
+            this.totalBytesWritten = totalBytesWritten;
+            this.totalSpilledBytes = totalSpilledBytes;
+            this.totalWriteTime = totalWriteTime;
+            this.pageSizesList = pageSizesList;
+        }
+
+        public long getTotalSpilledPagesInMemorySize()
+        {
+            return totalSpilledPagesInMemorySize;
+        }
+
+        public long getTotalBytesWritten()
+        {
+            return totalBytesWritten;
+        }
+
+        public long getTotalSpilledBytes()
+        {
+            return totalSpilledBytes;
+        }
+
+        public long getTotalWriteTime()
+        {
+            return totalWriteTime;
+        }
+
+        public List<Long> getPageSizesList()
+        {
+            return pageSizesList;
+        }
+    }
+}

presto-main/src/main/java/io/prestosql/spiller/FileSingleStreamSpiller.java

@@ -40,6 +40,8 @@ import io.prestosql.spi.filesystem.HetuFileSystemClient;
 import io.prestosql.spi.snapshot.BlockEncodingSerdeProvider;
 import io.prestosql.spi.snapshot.RestorableConfig;
 import io.prestosql.spi.spiller.SpillCipher;
+import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.commons.lang3.tuple.Pair;
 
 import javax.annotation.concurrent.NotThreadSafe;
 import javax.crypto.Cipher;
@@ -73,8 +75,10 @@ import static java.util.Objects.requireNonNull;
 @NotThreadSafe
 
 @RestorableConfig(uncapturedFields = {"closer", "serde",
-        "spillerStats", "localSpillContext", "memoryContext", "executor", "spillInProgress", "cipherIV", "spillCipher", "fileSystemClientManager", "fileSystemClient", "spillPath"})
+        "spillerStats", "localSpillContext", "memoryContext", "executor", "spillInProgress", "cipherIV", "spillCipher", "fileSystemClientManager", "fileSystemClient",
+        "spillPath", "output", "oldState", "closed", "outputStream", "sessionTargetFile", "useSessionDirectSerde"})
 public class FileSingleStreamSpiller
+        extends FileSingleSessionStreamSpiller
         implements SingleStreamSpiller
 {
     @VisibleForTesting
@@ -106,6 +110,10 @@ public class FileSingleStreamSpiller
     // Snapshot: capture page sizes that are used to update localSpillContext and spillerStats during resume
     private List<Long> pageSizeList = new LinkedList<>();
     private final int spillPrefetchReadPages;
+    private FileSingleStreamSpillerState oldState;
+    private boolean isSingleSessionSpiller;
+    private long targetFileSize = Long.MAX_VALUE;
+    private final boolean isSnapshotEnabled;
 
     public FileSingleStreamSpiller(
             PagesSerde serde,
@@ -121,15 +129,24 @@ public class FileSingleStreamSpiller
             boolean useKryo,
             boolean spillToHdfs,
             String spillProfile,
-            FileSystemClientManager fileSystemClientManager)
+            FileSystemClientManager fileSystemClientManager,
+            boolean isSingleSessionSpiller,
+            boolean isSnapshotEnabled,
+            String queryId)
     {
         this.serde = requireNonNull(serde, "serde is null");
         this.executor = requireNonNull(executor, "executor is null");
         this.spillerStats = requireNonNull(spillerStats, "spillerStats is null");
-        this.spillPath = spillPath;
+        if (spillToHdfs) {
+            this.spillPath = Paths.get(spillPath.toString(), queryId);
+        }
+        else {
+            this.spillPath = spillPath;
+        }
         this.spillProfile = spillProfile;
         this.fileSystemClientManager = requireNonNull(fileSystemClientManager, "fileSystemClient is null");
         this.spillToHdfs = spillToHdfs;
+        this.isSnapshotEnabled = isSnapshotEnabled;
         this.localSpillContext = spillContext.newLocalSpillContext();
         this.memoryContext = requireNonNull(memoryContext, "memoryContext is null");
         if (requireNonNull(spillCipher, "spillCipher is null").isPresent()) {
@@ -163,6 +180,21 @@ public class FileSingleStreamSpiller
         this.useKryo = useKryo;
         this.compressionEnabled = compressionEnabled;
         this.spillCipher = spillCipher;
+        this.isSingleSessionSpiller = isSingleSessionSpiller;
+        if (isSingleSessionSpiller) {
+            super.setTargetFile(getTargetFile(), useDirectSerde);
+            try {
+                if (useDirectSerde) {
+                    super.setOutputStream(getStreamForWriting(getOutputStreamBasedOnSpillLocation(), BUFFER_SIZE));
+                }
+                else {
+                    super.setOutput(new OutputStreamSliceOutput(getOutputStreamBasedOnSpillLocation(), BUFFER_SIZE));
+                }
+            }
+            catch (IOException e) {
+                throw new PrestoException(GENERIC_INTERNAL_ERROR, "Failed to create output Stream", e);
+            }
+        }
     }
 
     @Override
@@ -171,14 +203,39 @@ public class FileSingleStreamSpiller
         requireNonNull(pageIterator, "pageIterator is null");
         checkNoSpillInProgress();
         if (useDirectSerde) {
-            spillInProgress = executor.submit(() -> writePagesDirect(pageIterator));
+            spillInProgress = executor.submit(() -> {
+                if (isSingleSessionSpiller) {
+                    Stats stats = super.writePagesDirect(writable, pageIterator, serde);
+                    setStats(stats);
+                }
+                else {
+                    writePagesDirect(pageIterator);
+                }
+            });
         }
         else {
-            spillInProgress = executor.submit(() -> writePages(pageIterator));
+            spillInProgress = executor.submit(() -> {
+                if (isSingleSessionSpiller) {
+                    Stats stats = super.writePages(writable, pageIterator, serde);
+                    setStats(stats);
+                }
+                else {
+                    writePages(pageIterator);
+                }
+            });
         }
         return spillInProgress;
     }
 
+    private void setStats(Stats stats)
+    {
+        spilledPagesInMemorySize += stats.getTotalSpilledPagesInMemorySize();
+        localSpillContext.updateBytes(stats.getTotalBytesWritten());
+        spillerStats.addToTotalSpilledBytes(stats.getTotalSpilledBytes());
+        pageSizeList.addAll(stats.getPageSizesList());
+        localSpillContext.updateWriteTime(stats.getTotalWriteTime());
+    }
+
     @Override
     public long getSpilledPagesInMemorySize()
     {
@@ -198,7 +255,7 @@ public class FileSingleStreamSpiller
         return executor.submit(() -> ImmutableList.copyOf(getSpilledPages()));
     }
 
-    private void writePages(Iterator<Page> pageIterator)
+    void writePages(Iterator<Page> pageIterator)
     {
         checkState(writable, "Spilling no longer allowed. The spiller has been made non-writable on first read for subsequent reads to be consistent");
         try (SliceOutput output = new OutputStreamSliceOutput(getOutputStreamBasedOnSpillLocation(), BUFFER_SIZE)) {
@@ -221,7 +278,7 @@ public class FileSingleStreamSpiller
         }
     }
 
-    private OutputStream getStreamForWriting(OutputStream outputStream, int bufferSize) throws IOException
+    protected OutputStream getStreamForWriting(OutputStream outputStream, int bufferSize) throws IOException
     {
         OutputStream tmpOutputStream = outputStream;
         if (spillCipher.isPresent()) {
@@ -290,9 +347,23 @@ public class FileSingleStreamSpiller
         return (input) -> !((InputStreamSliceInput) input).isReadable();
     }
 
+    private Predicate<InputStream> getSessionDirectStreamEndOfData()
+    {
+        if (useKryo) {
+            return (input) -> (((Input) input).end() || ((Input) input).position() >= targetFileSize);
+        }
+
+        return (input) -> (!((InputStreamSliceInput) input).isReadable() || ((InputStreamSliceInput) input).position() >= targetFileSize);
+    }
+
+    private Predicate<InputStream> getSessionStreamEndOfData()
+    {
+        return (input) -> (!((InputStreamSliceInput) input).isReadable() || ((InputStreamSliceInput) input).position() >= targetFileSize);
+    }
+
     private Iterator<Page> readPages()
     {
-        checkState(writable, "Repeated reads are disallowed to prevent potential resource leaks");
+        checkState(writable, "Repeated reads are disallowed to prevent potential resource leaks ");
         writable = false;
 
         try {
@@ -300,10 +371,20 @@ public class FileSingleStreamSpiller
             Iterator<Page> pages;
 
             if (useDirectSerde) {
-                pages = PagesSerdeUtil.readPagesDirect(serde, getStreamForReading(input, BUFFER_SIZE), getStreamEndOfData(), spillPrefetchReadPages);
+                if (spillToHdfs) {
+                    pages = PagesSerdeUtil.readPagesDirect(serde, getStreamForReading(input, BUFFER_SIZE), getSessionDirectStreamEndOfData(), spillPrefetchReadPages);
+                }
+                else {
+                    pages = PagesSerdeUtil.readPagesDirect(serde, getStreamForReading(input, BUFFER_SIZE), getStreamEndOfData(), spillPrefetchReadPages);
+                }
             }
             else {
-                pages = PagesSerdeUtil.readPages(serde, new InputStreamSliceInput(input, BUFFER_SIZE), spillPrefetchReadPages);
+                if (spillToHdfs) {
+                    pages = PagesSerdeUtil.readPages(serde, new InputStreamSliceInput(input, BUFFER_SIZE), spillPrefetchReadPages, getSessionStreamEndOfData());
+                }
+                else {
+                    pages = PagesSerdeUtil.readPages(serde, new InputStreamSliceInput(input, BUFFER_SIZE), spillPrefetchReadPages);
+                }
             }
             return closeWhenExhausted(pages, input, localSpillContext);
         }
@@ -377,9 +458,26 @@ public class FileSingleStreamSpiller
         return targetFile.getFilePath();
     }
 
+    @Override
+    public Pair<Path, Long> getSpilledFileInfo()
+    {
+        return ImmutablePair.of(targetFile.getFilePath(), oldState.targetFileSize);
+    }
+
+    @Override
+    public void closeSessionSpiller()
+    {
+        super.closeSession();
+    }
+
+    private FileHolder getTargetFile()
+    {
+        return targetFile;
+    }
+
     private void createSpillFiles() throws IOException
     {
-        this.targetFile = closer.register(new FileHolder(fileSystemClient.createTemporaryFile(spillPath, SPILL_FILE_PREFIX, SPILL_FILE_SUFFIX), fileSystemClient));
+        this.targetFile = closer.register(new FileHolder(fileSystemClient.createTemporaryFile(spillPath, SPILL_FILE_PREFIX, SPILL_FILE_SUFFIX), fileSystemClient, (spillToHdfs && isSnapshotEnabled)));
     }
 
     private OutputStream getOutputStreamBasedOnSpillLocation() throws IOException
@@ -405,6 +503,22 @@ public class FileSingleStreamSpiller
         state.useKryo = this.useKryo;
         state.spillToHdfs = this.spillToHdfs;
         state.spillProfile = this.spillProfile;
+        while (!spillInProgress.isDone()) {
+            // wait till spilling is completed
+        }
+        if (isSingleSessionSpiller) {
+            state.targetFileSize = super.getTargetFileSize();
+        }
+        else {
+            try {
+                state.targetFileSize = this.targetFile.getFileSize();
+            }
+            catch (IOException e) {
+                throw new PrestoException(GENERIC_INTERNAL_ERROR, "Failed to get target file size", e);
+            }
+        }
+        this.oldState = state;
+        state.isSingleSessionSpiller = this.isSingleSessionSpiller;
         return state;
     }
 
@@ -417,12 +531,19 @@ public class FileSingleStreamSpiller
             this.spilledPagesInMemorySize = myState.spilledPagesInMemorySize;
             this.pageSizeList = myState.pageSizeList;
             this.targetFile.close();
+            this.isSingleSessionSpiller = myState.isSingleSessionSpiller;
+            this.spillToHdfs = myState.spillToHdfs;
             this.spillProfile = myState.spillProfile;
             Path path = Paths.get(myState.targetFile);
             this.fileSystemClient = getFileSystem(path, spillToHdfs, spillProfile, fileSystemClientManager);
-            // Actual file content is restored after this returns, in SingleInputSnapshotState.loadSpilledFiles
+            if (spillToHdfs) {
-            fileSystemClient.deleteIfExists(path);
+                this.targetFile = closer.register(new FileHolder(path, fileSystemClient, true));
-            this.targetFile = closer.register(new FileHolder(fileSystemClient.createFile(path), fileSystemClient));
+            }
+            else {
+                // Actual file content is restored after this returns, in SingleInputSnapshotState.loadSpilledFiles
+                fileSystemClient.deleteIfExists(path);
+                this.targetFile = closer.register(new FileHolder(fileSystemClient.createFile(path), fileSystemClient, false));
+            }
             for (Long pageSize : pageSizeList) {
                 // restore localSpillContext and spillerStats
                 this.localSpillContext.updateBytes(pageSize);
@@ -431,7 +552,9 @@ public class FileSingleStreamSpiller
             this.compressionEnabled = myState.compressionEnabled;
             this.useDirectSerde = myState.useDirectSerde;
             this.useKryo = myState.useKryo;
-            this.spillToHdfs = myState.spillToHdfs;
+            if (spillToHdfs) {
+                this.targetFileSize = myState.targetFileSize;
+            }
         }
         catch (IOException e) {
             throw new UncheckedIOException(e);
@@ -445,10 +568,12 @@ public class FileSingleStreamSpiller
         private long spilledPagesInMemorySize;
         private List<Long> pageSizeList;
         private String targetFile;
+        private long targetFileSize;
         private boolean compressionEnabled;
         private boolean useDirectSerde;
         private boolean useKryo;
         private boolean spillToHdfs;
         private String spillProfile;
+        private boolean isSingleSessionSpiller;
     }
 }

presto-main/src/main/java/io/prestosql/spiller/FileSingleStreamSpillerFactory.java

@@ -37,10 +37,8 @@ import javax.annotation.PreDestroy;
 import java.io.IOException;
 import java.nio.file.Path;
 import java.nio.file.Paths;
-import java.util.ArrayList;
 import java.util.List;
 import java.util.Optional;
-import java.util.UUID;
 import java.util.stream.Stream;
 
 import static com.google.common.base.Preconditions.checkArgument;
@@ -153,17 +151,7 @@ public class FileSingleStreamSpillerFactory
         this.spillPrefetchReadPages = spillPrefetchReadPages;
         this.useKryo = useKryo;
         this.fileSystemClientManager = fileSystemClientManager;
-        if (spillToHdfs) {
+        this.spillPaths = ImmutableList.copyOf(spillPaths);
-            String uuid = UUID.randomUUID().toString();
-            List<Path> hdfsPaths = new ArrayList<>();
-            for (Path path : spillPaths) {
-                hdfsPaths.add(Paths.get(path.toString(), uuid));
-            }
-            this.spillPaths = ImmutableList.copyOf(hdfsPaths);
-        }
-        else {
-            this.spillPaths = ImmutableList.copyOf(spillPaths);
-        }
     }
 
     public synchronized void cleanupOldSpillFiles()
@@ -197,7 +185,12 @@ public class FileSingleStreamSpillerFactory
     }
 
     @Override
-    public synchronized SingleStreamSpiller create(List<Type> types, SpillContext spillContext, LocalMemoryContext memoryContext)
+    public synchronized SingleStreamSpiller create(List<Type> types, SpillContext spillContext, LocalMemoryContext memoryContext, boolean isSingleSessionSpiller, boolean isSnapshotEnabled, String queryId, boolean isSpillToHdfs)
+    {
+        return create(types, spillContext, memoryContext, isSingleSessionSpiller, isSnapshotEnabled, queryId, isSpillToHdfs, null);
+    }
+
+    public synchronized SingleStreamSpiller create(List<Type> types, SpillContext spillContext, LocalMemoryContext memoryContext, boolean isSingleSessionSpiller, boolean isSnapshotEnabled, String queryId, boolean isSpillToHdfs, Path spillToHdfsPath)
     {
         createSpillDirectories();
         Optional<SpillCipher> spillCipher = Optional.empty();
@@ -205,7 +198,22 @@ public class FileSingleStreamSpillerFactory
             spillCipher = Optional.of(new AesSpillCipher());
         }
         PagesSerde serde = serdeFactory.createPagesSerdeForSpill(spillCipher, spillDirectSerdeEnabled, useKryo);
-        return new FileSingleStreamSpiller(serde, executor, getNextSpillPath(), spillerStats, spillContext, memoryContext, spillCipher, spillCompressionEnabled, spillDirectSerdeEnabled, spillPrefetchReadPages, useKryo, spillToHdfs, spillProfile, fileSystemClientManager);
+        Path spillPath;
+        if (spillToHdfsPath != null) {
+            spillPath = getHdfsSpillPath(spillToHdfsPath);
+        }
+        else {
+            spillPath = isSpillToHdfs ? Paths.get("/tmp/hetu/snapshot") : getNextSpillPath();
+        }
+        return new FileSingleStreamSpiller(serde, executor, spillPath, spillerStats, spillContext, memoryContext, spillCipher, spillCompressionEnabled, spillDirectSerdeEnabled, spillPrefetchReadPages, useKryo, isSpillToHdfs, spillProfile, fileSystemClientManager, isSingleSessionSpiller, isSnapshotEnabled, queryId);
+    }
+
+    private synchronized Path getHdfsSpillPath(Path spillToHdfsPath)
+    {
+        if (hasEnoughDiskSpace(spillToHdfsPath)) {
+            return spillToHdfsPath;
+        }
+        throw new PrestoException(OUT_OF_SPILL_SPACE, "No free space available for spill");
     }
 
     private synchronized Path getNextSpillPath()

presto-main/src/main/java/io/prestosql/spiller/GenericPartitioningSpiller.java

@@ -14,6 +14,7 @@
 package io.prestosql.spiller;
 
 import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.AbstractIterator;
 import com.google.common.collect.ImmutableList;
 import com.google.common.io.Closer;
 import com.google.common.util.concurrent.Futures;
@@ -27,6 +28,7 @@ import io.prestosql.spi.snapshot.BlockEncodingSerdeProvider;
 import io.prestosql.spi.snapshot.RestorableConfig;
 import io.prestosql.spi.type.Type;
 import it.unimi.dsi.fastutil.ints.IntArrayList;
+import org.apache.commons.lang3.tuple.Pair;
 
 import javax.annotation.concurrent.ThreadSafe;
 
@@ -34,11 +36,14 @@ import java.io.IOException;
 import java.io.Serializable;
 import java.nio.file.Path;
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.Collections;
 import java.util.HashSet;
 import java.util.Iterator;
+import java.util.LinkedList;
 import java.util.List;
 import java.util.Optional;
+import java.util.Queue;
 import java.util.Set;
 import java.util.function.BiFunction;
 import java.util.function.BiPredicate;
@@ -54,7 +59,7 @@ import static java.util.Objects.requireNonNull;
 
 @ThreadSafe
 @RestorableConfig(uncapturedFields = {"types", "partitionFunction", "closer",
-        "spillerFactory", "spillContext", "memoryContext", "pageBuilders", "getRawHash"})
+        "spillerFactory", "spillContext", "memoryContext", "getRawHash"})
 public class GenericPartitioningSpiller
         implements PartitioningSpiller
 {
@@ -67,10 +72,14 @@ public class GenericPartitioningSpiller
 
     private final List<PageBuilder> pageBuilders;
     private final List<Optional<SingleStreamSpiller>> spillers;
+    private List<List<Optional<SingleStreamSpiller>>> backUpSpillers;
     private final BiFunction<Integer, Page, Long> getRawHash;
 
     private boolean readingStarted;
     private final Set<Integer> spilledPartitions = new HashSet<>();
+    private boolean isSingleSessionSpiller;
+    private final boolean isSnapshotEnabled;
+    private final String queryId;
 
     public GenericPartitioningSpiller(
             List<Type> types,
@@ -78,7 +87,10 @@ public class GenericPartitioningSpiller
             SpillContext spillContext,
             AggregatedMemoryContext memoryContext,
             SingleStreamSpillerFactory spillerFactory,
-            BiFunction<Integer, Page, Long> getRawHash)
+            BiFunction<Integer, Page, Long> getRawHash,
+            boolean isSingleSessionSpiller,
+            boolean isSnapshotEnabled,
+            String queryId)
     {
         requireNonNull(spillContext, "spillContext is null");
 
@@ -94,12 +106,17 @@ public class GenericPartitioningSpiller
 
         ImmutableList.Builder<PageBuilder> tmpPageBuilders = ImmutableList.builder();
         spillers = new ArrayList<>(partitionCount);
+        backUpSpillers = new ArrayList<>(partitionCount);
         for (int partition = 0; partition < partitionCount; partition++) {
             tmpPageBuilders.add(new PageBuilder(types));
             spillers.add(Optional.empty());
+            backUpSpillers.add(null);
         }
         this.pageBuilders = tmpPageBuilders.build();
         this.getRawHash = getRawHash;
+        this.isSingleSessionSpiller = isSingleSessionSpiller;
+        this.isSnapshotEnabled = isSnapshotEnabled;
+        this.queryId = queryId;
     }
 
     @Override
@@ -108,7 +125,45 @@ public class GenericPartitioningSpiller
         readingStarted = true;
         getFutureValue(flush(partition));
         spilledPartitions.remove(partition);
-        return getSpiller(partition).getSpilledPages();
+        if (isSingleSessionSpiller) {
+            List<Iterator<Page>> pageIteratorList = new ArrayList<>();
+            if (backUpSpillers.get(partition) != null) {
+                for (Optional<SingleStreamSpiller> singleStreamSpiller : backUpSpillers.get(partition)) {
+                    if (singleStreamSpiller.isPresent()) {
+                        pageIteratorList.add(singleStreamSpiller.get().getSpilledPages());
+                    }
+                }
+            }
+            pageIteratorList.add(getSpiller(partition).getSpilledPages());
+            return new AbstractIterator<Page>() {
+                Iterator<Page> pageIterator;
+                private Queue<Iterator<Page>> queue = new LinkedList<>(pageIteratorList);
+
+                @Override
+                protected Page computeNext()
+                {
+                    if (pageIterator == null && !queue.isEmpty()) {
+                        pageIterator = queue.poll();
+                    }
+                    while (pageIterator != null) {
+                        if (pageIterator.hasNext()) {
+                            Page page = pageIterator.next();
+                            return page;
+                        }
+                        if (!queue.isEmpty()) {
+                            pageIterator = queue.poll();
+                        }
+                        else {
+                            pageIterator = null;
+                        }
+                    }
+                    return endOfData();
+                }
+            };
+        }
+        else {
+            return getSpiller(partition).getSpilledPages();
+        }
     }
 
     @Override
@@ -144,7 +199,7 @@ public class GenericPartitioningSpiller
         for (int position = 0; position < page.getPositionCount(); position++) {
             int partition = partitionFunction.getPartition(page, position);
 
-            if (!spillPartitionMask.test(partition)) {
+            if (!spillPartitionMask.test(partition) && !spilledPartitions.contains(partition)) {
                 unspilledPositions.add(position);
                 continue;
             }
@@ -206,12 +261,18 @@ public class GenericPartitioningSpiller
     {
         Optional<SingleStreamSpiller> spiller = spillers.get(partition);
         if (!spiller.isPresent()) {
-            spiller = Optional.of(closer.register(spillerFactory.create(types, spillContext, memoryContext.newLocalMemoryContext(GenericPartitioningSpiller.class.getSimpleName()))));
+            spiller = Optional.of(closer.register(spillerFactory.create(types, spillContext, memoryContext.newLocalMemoryContext(GenericPartitioningSpiller.class.getSimpleName()), isSingleSessionSpiller, isSnapshotEnabled, queryId, isSingleSessionSpiller)));
             spillers.set(partition, spiller);
         }
         return spiller.get();
     }
 
+    @Override
+    public Set<Integer> getSpilledPartitions()
+    {
+        return spilledPartitions;
+    }
+
     @Override
     public synchronized void close()
             throws IOException
@@ -220,9 +281,44 @@ public class GenericPartitioningSpiller
     }
 
     @Override
-    public List<Path> getSpilledFilePaths()
+    public List<Path> getSpilledFilePaths(boolean isStoreSpillFiles)
     {
-        return spillers.stream().filter(spiller -> spiller.isPresent()).map(spiller -> spiller.get().getFile()).collect(Collectors.toList());
+        if (isSingleSessionSpiller) {
+            return Arrays.asList(backUpSpillers.stream()
+                            .filter(list -> list != null)
+                            .flatMap(list -> list.stream()
+                                    .filter(spiller -> spiller.isPresent())
+                                    .map(spiller -> spiller.get().getFile()))
+                            .collect(Collectors.toList()),
+                    spillers.stream()
+                            .filter(spiller -> spiller.isPresent())
+                            .map(spiller -> spiller.get().getFile())
+                            .collect(Collectors.toList()))
+                    .stream()
+                    .flatMap(list -> list.stream())
+                    .collect(Collectors.toList());
+        }
+        else {
+            return spillers.stream().filter(spiller -> spiller.isPresent()).map(spiller -> spiller.get().getFile()).collect(Collectors.toList());
+        }
+    }
+
+    @Override
+    public List<Pair<Path, Long>> getSpilledFileInfo()
+    {
+        return spillers.stream().filter(spiller -> spiller.isPresent()).map(spiller -> spiller.get().getSpilledFileInfo()).collect(Collectors.toList());
+    }
+
+    @Override
+    public void closeSessionSpiller(int partition)
+    {
+        spillers.get(partition).ifPresent(spiller -> spiller.closeSessionSpiller());
+    }
+
+    @Override
+    public ListenableFuture<?> flushPartition(int partition)
+    {
+        return flush(partition);
     }
 
     @Override
@@ -237,10 +333,22 @@ public class GenericPartitioningSpiller
                 myState.spillers.set(i, spillers.get(i).get().capture(serdeProvider));
             }
         }
+        myState.backUpSpillers = new ArrayList<>(Collections.nCopies(backUpSpillers.size(), null));
+        for (int i = 0; i < backUpSpillers.size(); i++) {
+            if (backUpSpillers.get(i) != null) {
+                myState.backUpSpillers.set(i, new LinkedList<>());
+                for (Optional<SingleStreamSpiller> singleStreamSpiller : backUpSpillers.get(i)) {
+                    if (singleStreamSpiller.isPresent()) {
+                        myState.backUpSpillers.get(i).add(singleStreamSpiller.get().capture(serdeProvider));
+                    }
+                }
+            }
+        }
         myState.pageBuilders = new ArrayList<>(Collections.nCopies(pageBuilders.size(), null));
         for (int pageBuilderCount = 0; pageBuilderCount < pageBuilders.size(); pageBuilderCount++) {
             myState.pageBuilders.set(pageBuilderCount, pageBuilders.get(pageBuilderCount).capture(serdeProvider));
         }
+        myState.isSingleSessionSpiller = isSingleSessionSpiller;
         return myState;
     }
 
@@ -252,11 +360,34 @@ public class GenericPartitioningSpiller
         for (int partition : myState.spilledPartitions) {
             this.spilledPartitions.add(partition);
         }
+        this.isSingleSessionSpiller = myState.isSingleSessionSpiller;
+
+        for (int i = 0; i < backUpSpillers.size(); i++) {
+            if (myState.backUpSpillers.get(i) != null) {
+                backUpSpillers.set(i, new LinkedList<>());
+                for (Object singleStreamSpiller : myState.backUpSpillers.get(i)) {
+                    SingleStreamSpiller spiller = spillerFactory.create(types, spillContext, memoryContext.newLocalMemoryContext(GenericPartitioningSpiller.class.getSimpleName()), isSingleSessionSpiller, isSnapshotEnabled, queryId, isSingleSessionSpiller);
+                    spiller.restore(singleStreamSpiller, serdeProvider);
+                    backUpSpillers.get(i).add(Optional.of(closer.register(spiller)));
+                }
+            }
+        }
+
         for (int i = 0; i < spillers.size(); i++) {
             if (myState.spillers.get(i) != null) {
-                SingleStreamSpiller spiller = spillerFactory.create(types, spillContext, memoryContext.newLocalMemoryContext(GenericPartitioningSpiller.class.getSimpleName()));
+                SingleStreamSpiller spiller = spillerFactory.create(types, spillContext, memoryContext.newLocalMemoryContext(GenericPartitioningSpiller.class.getSimpleName()), isSingleSessionSpiller, isSnapshotEnabled, queryId, isSingleSessionSpiller);
                 spiller.restore(myState.spillers.get(i), serdeProvider);
-                this.spillers.set(i, Optional.of(closer.register(spiller)));
+                if (isSingleSessionSpiller) {
+                    if (backUpSpillers.get(i) == null) {
+                        backUpSpillers.set(i, new LinkedList<>());
+                    }
+                    backUpSpillers.get(i).add(Optional.of(closer.register(spiller)));
+                    SingleStreamSpiller newSpiller = spillerFactory.create(types, spillContext, memoryContext.newLocalMemoryContext(GenericPartitioningSpiller.class.getSimpleName()), isSingleSessionSpiller, isSnapshotEnabled, queryId, isSingleSessionSpiller);
+                    this.spillers.set(i, Optional.of(closer.register(newSpiller)));
+                }
+                else {
+                    this.spillers.set(i, Optional.of(closer.register(spiller)));
+                }
             }
         }
         for (int pageBuilderCount = 0; pageBuilderCount < pageBuilders.size(); pageBuilderCount++) {
@@ -271,5 +402,7 @@ public class GenericPartitioningSpiller
         boolean readingStarted;
         List<Object> spillers;
         List<Object> pageBuilders;
+        List<List<Object>> backUpSpillers;
+        boolean isSingleSessionSpiller;
     }
 }

presto-main/src/main/java/io/prestosql/spiller/GenericPartitioningSpillerFactory.java

@@ -41,14 +41,17 @@ public class GenericPartitioningSpillerFactory
             List<Type> types,
             PartitionFunction partitionFunction,
             SpillContext spillContext,
-            AggregatedMemoryContext memoryContext)
+            AggregatedMemoryContext memoryContext,
+            boolean isSingleSessionSpiller,
+            boolean isSnapshotEnabled,
+            String queryId)
     {
-        return new GenericPartitioningSpiller(types, partitionFunction, spillContext, memoryContext, singleStreamSpillerFactory, null);
+        return new GenericPartitioningSpiller(types, partitionFunction, spillContext, memoryContext, singleStreamSpillerFactory, null, isSingleSessionSpiller, isSnapshotEnabled, queryId);
     }
 
     @Override
-    public PartitioningSpiller create(List<Type> types, PartitionFunction partitionFunction, SpillContext spillContext, AggregatedMemoryContext memoryContext, BiFunction<Integer, Page, Long> getRawHash)
+    public PartitioningSpiller create(List<Type> types, PartitionFunction partitionFunction, SpillContext spillContext, AggregatedMemoryContext memoryContext, BiFunction<Integer, Page, Long> getRawHash, boolean isSingleSessionSpiller, boolean isSnapshotEnabled, String queryId)
     {
-        return new GenericPartitioningSpiller(types, partitionFunction, spillContext, memoryContext, singleStreamSpillerFactory, getRawHash);
+        return new GenericPartitioningSpiller(types, partitionFunction, spillContext, memoryContext, singleStreamSpillerFactory, getRawHash, isSingleSessionSpiller, isSnapshotEnabled, queryId);
     }
 }

presto-main/src/main/java/io/prestosql/spiller/GenericSpiller.java

@@ -34,9 +34,13 @@ import java.util.ArrayList;
 import java.util.Iterator;
 import java.util.List;
 import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Function;
+import java.util.stream.Collectors;
 import java.util.stream.IntStream;
 
 import static com.google.common.base.Preconditions.checkState;
+import static com.google.common.util.concurrent.MoreExecutors.directExecutor;
+import static io.airlift.concurrent.MoreFutures.getDone;
 import static java.util.Objects.requireNonNull;
 import static java.util.stream.Collectors.toList;
 
@@ -53,23 +57,33 @@ public class GenericSpiller
     private ListenableFuture<?> previousSpill = Futures.immediateFuture(null);
     private final List<SingleStreamSpiller> singleStreamSpillers = new ArrayList<>();
     private final List<AtomicBoolean> spillCommitted = new ArrayList<>();
+    private boolean isSessionSpiller;
+    private final boolean isSnapshotEnabled;
+    private final String queryId;
+    private final boolean isSpillToHdfs;
 
     public GenericSpiller(
             List<Type> types,
             SpillContext spillContext,
             AggregatedMemoryContext aggregatedMemoryContext,
-            SingleStreamSpillerFactory singleStreamSpillerFactory)
+            SingleStreamSpillerFactory singleStreamSpillerFactory,
+            boolean isSnapshotEnabled,
+            String queryId,
+            boolean isSpillToHdfs)
     {
         this.types = requireNonNull(types, "types can not be null");
         this.spillContext = requireNonNull(spillContext, "spillContext can not be null");
         this.aggregatedMemoryContext = requireNonNull(aggregatedMemoryContext, "aggregatedMemoryContext can not be null");
         this.singleStreamSpillerFactory = requireNonNull(singleStreamSpillerFactory, "singleStreamSpillerFactory can not be null");
+        this.isSnapshotEnabled = isSnapshotEnabled;
+        this.queryId = queryId;
+        this.isSpillToHdfs = isSpillToHdfs;
     }
 
     @Override
     public ListenableFuture<?> spill(Iterator<Page> pageIterator)
     {
-        SingleStreamSpiller singleStreamSpiller = singleStreamSpillerFactory.create(types, spillContext, aggregatedMemoryContext.newLocalMemoryContext(GenericSpiller.class.getSimpleName()));
+        SingleStreamSpiller singleStreamSpiller = singleStreamSpillerFactory.create(types, spillContext, aggregatedMemoryContext.newLocalMemoryContext(GenericSpiller.class.getSimpleName()), false, isSnapshotEnabled, queryId, isSpillToHdfs);
         closer.register(singleStreamSpiller);
         singleStreamSpillers.add(singleStreamSpiller);
         spillCommitted.add(new AtomicBoolean(true));
@@ -77,6 +91,17 @@ public class GenericSpiller
         return previousSpill;
     }
 
+    @Override
+    public SingleStreamSpiller createSessionSpiller()
+    {
+        SingleStreamSpiller singleStreamSpiller = singleStreamSpillerFactory.create(types, spillContext, aggregatedMemoryContext.newLocalMemoryContext(GenericSpiller.class.getSimpleName()), true, isSnapshotEnabled, queryId, isSpillToHdfs);
+        closer.register(singleStreamSpiller);
+        singleStreamSpillers.add(singleStreamSpiller);
+        spillCommitted.add(new AtomicBoolean(true));
+        isSessionSpiller = true;
+        return singleStreamSpiller;
+    }
+
     /**
      * Initiate spilling of pages stream. Returns completed future once spilling has finished with commit function.
      *
@@ -85,7 +110,7 @@ public class GenericSpiller
     @Override
     public Pair<ListenableFuture<?>, Runnable> spillUnCommit(Iterator<Page> pageIterator)
     {
-        SingleStreamSpiller singleStreamSpiller = singleStreamSpillerFactory.create(types, spillContext, aggregatedMemoryContext.newLocalMemoryContext(GenericSpiller.class.getSimpleName()));
+        SingleStreamSpiller singleStreamSpiller = singleStreamSpillerFactory.create(types, spillContext, aggregatedMemoryContext.newLocalMemoryContext(GenericSpiller.class.getSimpleName()), false, isSnapshotEnabled, queryId, isSpillToHdfs);
         closer.register(singleStreamSpiller);
         singleStreamSpillers.add(singleStreamSpiller);
         AtomicBoolean isCommitted = new AtomicBoolean(false);
@@ -130,11 +155,39 @@ public class GenericSpiller
     }
 
     @Override
-    public List<Path> getSpilledFilePaths()
+    public ListenableFuture<Integer> getAllSpilledPages(Function<List<Page>, Integer> processor)
+    {
+        List<ListenableFuture<List<Page>>> listenableFutureList = singleStreamSpillers.stream().map(SingleStreamSpiller::getAllSpilledPages).collect(Collectors.toList());
+        return Futures.whenAllComplete(listenableFutureList).call(() -> getAllSpilledPages(listenableFutureList, processor), directExecutor());
+    }
+
+    private int getAllSpilledPages(List<ListenableFuture<List<Page>>> listenableFutureList, Function<List<Page>, Integer> processor)
+    {
+        int count = 0;
+        for (ListenableFuture<List<Page>> listenableFuture : listenableFutureList) {
+            processor.apply(getDone(listenableFuture));
+        }
+        return count;
+    }
+
+    @Override
+    public long getSpilledPagesInMemorySize()
+    {
+        return singleStreamSpillers.stream().mapToLong(SingleStreamSpiller::getSpilledPagesInMemorySize).sum();
+    }
+
+    @Override
+    public List<Path> getSpilledFilePaths(boolean isStoreSpillFiles)
     {
         return IntStream.range(0, spillCommitted.size()).filter(i -> spillCommitted.get(i).get()).mapToObj(o -> singleStreamSpillers.get(o).getFile()).collect(toList());
     }
 
+    @Override
+    public List<Pair<Path, Long>> getSpilledFileInfo()
+    {
+        return IntStream.range(0, spillCommitted.size()).filter(i -> spillCommitted.get(i).get()).mapToObj(o -> singleStreamSpillers.get(o).getSpilledFileInfo()).collect(toList());
+    }
+
     @Override
     public Object capture(BlockEncodingSerdeProvider serdeProvider)
     {
@@ -145,6 +198,7 @@ public class GenericSpiller
                 myState.singleStreamSpillers.add(s.capture(serdeProvider));
             }
         }
+        myState.isSessionSpiller = isSessionSpiller;
         return myState;
     }
 
@@ -152,8 +206,9 @@ public class GenericSpiller
     public void restore(Object state, BlockEncodingSerdeProvider serdeProvider)
     {
         GenericSpillerState myState = (GenericSpillerState) state;
+        isSessionSpiller = myState.isSessionSpiller;
         for (Object s : myState.singleStreamSpillers) {
-            SingleStreamSpiller singleStreamSpiller = singleStreamSpillerFactory.create(types, spillContext, aggregatedMemoryContext.newLocalMemoryContext(GenericSpiller.class.getSimpleName()));
+            SingleStreamSpiller singleStreamSpiller = singleStreamSpillerFactory.create(types, spillContext, aggregatedMemoryContext.newLocalMemoryContext(GenericSpiller.class.getSimpleName()), isSessionSpiller, isSnapshotEnabled, queryId, isSpillToHdfs);
             singleStreamSpiller.restore(s, serdeProvider);
             this.singleStreamSpillers.add(singleStreamSpiller);
             this.spillCommitted.add(new AtomicBoolean(true));
@@ -165,5 +220,6 @@ public class GenericSpiller
             implements Serializable
     {
         List<Object> singleStreamSpillers = new ArrayList<>();
+        private boolean isSessionSpiller;
     }
 }

presto-main/src/main/java/io/prestosql/spiller/GenericSpillerFactory.java

@@ -34,8 +34,8 @@ public class GenericSpillerFactory
     }
 
     @Override
-    public Spiller create(List<Type> types, SpillContext localSpillContext, AggregatedMemoryContext aggregatedMemoryContext)
+    public Spiller create(List<Type> types, SpillContext localSpillContext, AggregatedMemoryContext aggregatedMemoryContext, boolean isSnapshotEnabled, String queryId, boolean isSpillToHdfs)
     {
-        return new GenericSpiller(types, localSpillContext, aggregatedMemoryContext, singleStreamSpillerFactory);
+        return new GenericSpiller(types, localSpillContext, aggregatedMemoryContext, singleStreamSpillerFactory, isSnapshotEnabled, queryId, isSpillToHdfs);
     }
 }

presto-main/src/main/java/io/prestosql/spiller/PartitioningSpiller.java

@@ -14,15 +14,18 @@
 package io.prestosql.spiller;
 
 import com.google.common.collect.ImmutableList;
+import com.google.common.util.concurrent.Futures;
 import com.google.common.util.concurrent.ListenableFuture;
 import io.prestosql.spi.Page;
 import io.prestosql.spi.snapshot.Restorable;
+import org.apache.commons.lang3.tuple.Pair;
 
 import java.io.Closeable;
 import java.io.IOException;
 import java.nio.file.Path;
 import java.util.Iterator;
 import java.util.List;
+import java.util.Set;
 import java.util.function.BiPredicate;
 import java.util.function.IntPredicate;
 
@@ -52,6 +55,13 @@ public interface PartitioningSpiller
     // TODO getSpilledPages should not need flush last buffer to disk
     Iterator<Page> getSpilledPages(int partition);
 
+    Set<Integer> getSpilledPartitions();
+
+    default ListenableFuture<?> flushPartition(int partition)
+    {
+        return Futures.immediateFuture(null);
+    }
+
     void verifyAllPartitionsRead();
 
     /**
@@ -61,7 +71,12 @@ public interface PartitioningSpiller
     void close()
             throws IOException;
 
-    default List<Path> getSpilledFilePaths()
+    default List<Path> getSpilledFilePaths(boolean isStoreSpillFiles)
+    {
+        return ImmutableList.of();
+    }
+
+    default List<Pair<Path, Long>> getSpilledFileInfo()
     {
         return ImmutableList.of();
     }
@@ -87,4 +102,8 @@ public interface PartitioningSpiller
             return retained;
         }
     }
+
+    default void closeSessionSpiller(int partition)
+    {
+    }
 }

presto-main/src/main/java/io/prestosql/spiller/PartitioningSpillerFactory.java

@@ -28,21 +28,27 @@ public interface PartitioningSpillerFactory
             List<Type> types,
             PartitionFunction partitionFunction,
             SpillContext spillContext,
-            AggregatedMemoryContext memoryContext);
+            AggregatedMemoryContext memoryContext,
+            boolean isSingleSessionSpiller,
+            boolean isSnapshotEnabled,
+            String queryId);
 
     default PartitioningSpiller create(
             List<Type> types,
             PartitionFunction partitionFunction,
             SpillContext spillContext,
             AggregatedMemoryContext memoryContext,
-            BiFunction<Integer, Page, Long> getRawHash)
+            BiFunction<Integer, Page, Long> getRawHash,
+            boolean isSingleSessionSpiller,
+            boolean isSnapshotEnabled,
+            String queryId)
     {
-        return create(types, partitionFunction, spillContext, memoryContext);
+        return create(types, partitionFunction, spillContext, memoryContext, isSingleSessionSpiller, isSnapshotEnabled, queryId);
     }
 
     static PartitioningSpillerFactory unsupportedPartitioningSpillerFactory()
     {
-        return (types, partitionFunction, spillContext, memoryContext) -> {
+        return (types, partitionFunction, spillContext, memoryContext, isSingleSessionSpiller, isSnapshotEnabled, queryId) -> {
             throw new UnsupportedOperationException();
         };
     }

presto-main/src/main/java/io/prestosql/spiller/SingleStreamSpiller.java

@@ -16,6 +16,7 @@ package io.prestosql.spiller;
 import com.google.common.util.concurrent.ListenableFuture;
 import io.prestosql.spi.Page;
 import io.prestosql.spi.snapshot.Restorable;
+import org.apache.commons.lang3.tuple.Pair;
 
 import java.io.Closeable;
 import java.nio.file.Path;
@@ -70,6 +71,8 @@ public interface SingleStreamSpiller
 
     Path getFile();
 
+    Pair<Path, Long> getSpilledFileInfo();
+
     default void commit()
     {
         /* do nothing */
@@ -79,4 +82,9 @@ public interface SingleStreamSpiller
     {
         return true;
     }
+
+    default void closeSessionSpiller()
+    {
+        /* do nothing */
+    }
 }

presto-main/src/main/java/io/prestosql/spiller/SingleStreamSpillerFactory.java

@@ -21,11 +21,11 @@ import java.util.List;
 
 public interface SingleStreamSpillerFactory
 {
-    SingleStreamSpiller create(List<Type> types, SpillContext spillContext, LocalMemoryContext memoryContext);
+    SingleStreamSpiller create(List<Type> types, SpillContext spillContext, LocalMemoryContext memoryContext, boolean isSingleSessionSpiller, boolean isSnapshotEnabled, String queryId, boolean isSpillToHdfs);
 
     static SingleStreamSpillerFactory unsupportedSingleStreamSpillerFactory()
     {
-        return (types, spillContext, memoryContext) -> {
+        return (types, spillContext, memoryContext, isSingleSessionSpiller, isSnapshotEnabled, queryId, isSpillToHdfs) -> {
             throw new UnsupportedOperationException();
         };
     }

presto-main/src/main/java/io/prestosql/spiller/Spiller.java

@@ -14,6 +14,7 @@
 package io.prestosql.spiller;
 
 import com.google.common.collect.ImmutableList;
+import com.google.common.util.concurrent.Futures;
 import com.google.common.util.concurrent.ListenableFuture;
 import io.prestosql.spi.Page;
 import io.prestosql.spi.snapshot.Restorable;
@@ -24,6 +25,8 @@ import java.io.Closeable;
 import java.nio.file.Path;
 import java.util.Iterator;
 import java.util.List;
+import java.util.concurrent.Future;
+import java.util.function.Function;
 
 public interface Spiller
         extends Closeable, Restorable
@@ -48,8 +51,19 @@ public interface Spiller
 
     /**
      * Returns spilled files paths
+     * @param isStoreSpillFiles
      */
-    default List<Path> getSpilledFilePaths()
+    default List<Path> getSpilledFilePaths(boolean isStoreSpillFiles)
+    {
+        return ImmutableList.of();
+    }
+
+    default List<Path> getSpilledFilePathsToStore()
+    {
+        return ImmutableList.of();
+    }
+
+    default List<Pair<Path, Long>> getSpilledFileInfo()
     {
         return ImmutableList.of();
     }
@@ -60,4 +74,22 @@ public interface Spiller
      */
     @Override
     void close();
+
+    default long getSpilledPagesInMemorySize()
+    {
+        return 0;
+    }
+
+    /**
+     * Initiates read of previously spilled pages. The returned {@link Future} will be complete once all pages are read.
+     */
+    default ListenableFuture<Integer> getAllSpilledPages(Function<List<Page>, Integer> processor)
+    {
+        return Futures.immediateFuture(null);
+    }
+
+    default SingleStreamSpiller createSessionSpiller()
+    {
+        return null;
+    }
 }

presto-main/src/main/java/io/prestosql/spiller/SpillerFactory.java

@@ -21,5 +21,5 @@ import java.util.List;
 
 public interface SpillerFactory
 {
-    Spiller create(List<Type> types, SpillContext localSpillContext, AggregatedMemoryContext aggregatedMemoryContext);
+    Spiller create(List<Type> types, SpillContext localSpillContext, AggregatedMemoryContext aggregatedMemoryContext, boolean isSnapshotEnabled, String queryId, boolean isSpillToHdfs);
 }

presto-main/src/main/java/io/prestosql/sql/analyzer/FeaturesConfig.java

@@ -19,6 +19,7 @@ import com.google.common.collect.ImmutableList;
 import io.airlift.configuration.Config;
 import io.airlift.configuration.ConfigDescription;
 import io.airlift.configuration.DefunctConfig;
+import io.airlift.log.Logger;
 import io.airlift.units.DataSize;
 import io.airlift.units.Duration;
 import io.airlift.units.MaxDataSize;
@@ -906,10 +907,14 @@ public class FeaturesConfig
         return this;
     }
 
-    @AssertTrue(message = SPILLER_SPILL_PATH + " must be only contain single path when " + SPILLER_SPILL_TO_HDFS + " is set to true")
+    @AssertTrue(message = SPILLER_SPILL_PATH + " must be only contain single path when " + SPILLER_SPILL_TO_HDFS)
     public boolean isSpillerSpillPathConfiguredIfSpillToHdfsEnabled()
     {
-        return !isSpillToHdfs() || spillerSpillPaths.size() == 1;
+        Logger log = Logger.get(FeaturesConfig.class);
+        if (spillToHdfs) {
+            log.warn("Spiller path configured by user is ignored and /tmp/hetu/snapshot is used by default");
+        }
+        return !isSpillToHdfs() || (spillerSpillPaths.size() == 1);
     }
 
     @AssertTrue(message = SPILLER_SPILL_PROFILE + " must be configured when " + SPILLER_SPILL_TO_HDFS + " is set to true")

presto-main/src/main/java/io/prestosql/sql/planner/LocalExecutionPlanner.java

@@ -276,6 +276,7 @@ import static io.prestosql.SystemSessionProperties.isSpillEnabled;
 import static io.prestosql.SystemSessionProperties.isSpillForOuterJoinEnabled;
 import static io.prestosql.SystemSessionProperties.isSpillOrderBy;
 import static io.prestosql.SystemSessionProperties.isSpillReuseExchange;
+import static io.prestosql.SystemSessionProperties.isSpillToHdfsEnabled;
 import static io.prestosql.SystemSessionProperties.isSpillWindowOperator;
 import static io.prestosql.dynamicfilter.DynamicFilterCacheManager.createCacheKey;
 import static io.prestosql.expressions.LogicalRowExpressions.TRUE_CONSTANT;
@@ -1352,7 +1353,8 @@ public class LocalExecutionPlanner
                     pagesIndexFactory,
                     isSpillEnabled(session) && isSpillWindowOperator(session),
                     spillerFactory,
-                    orderingCompiler);
+                    orderingCompiler,
+                    isSpillToHdfsEnabled(session));
 
             return new PhysicalOperation(operatorFactory, outputMappings.build(), context, source);
         }
@@ -1403,6 +1405,7 @@ public class LocalExecutionPlanner
 
             boolean spillEnabled = isSpillEnabled(context.getSession()) && isSpillOrderBy(context.getSession());
             boolean spillNonBlocking = isNonBlockingSpillOrderby(context.getSession());
+            boolean isSpillToHdfsEnabled = isSpillToHdfsEnabled(context.getSession());
 
             OperatorFactory operator = new OrderByOperatorFactory(
                     context.getNextOperatorId(),
@@ -1416,7 +1419,8 @@ public class LocalExecutionPlanner
                     spillEnabled,
                     Optional.of(spillerFactory),
                     orderingCompiler,
-                    spillNonBlocking);
+                    spillNonBlocking,
+                    isSpillToHdfsEnabled);
 
             return new PhysicalOperation(operator, source.getLayout(), context, source);
         }
@@ -2773,7 +2777,9 @@ public class LocalExecutionPlanner
                     10_000,
                     pagesIndexFactory,
                     spillAllowed && taskCount > 1,
-                    singleStreamSpillerFactory);
+                    singleStreamSpillerFactory,
+                    spillerFactory,
+                    isSpillToHdfsEnabled(context.getSession()));
 
             factoriesBuilder.add(hashBuilderOperatorFactory);
 

presto-main/src/main/java/io/prestosql/sql/planner/iterative/rule/AddExchangeAboveCTENode.java

@@ -25,6 +25,7 @@ import io.prestosql.sql.planner.iterative.Rule;
 import java.util.Optional;
 
 import static io.prestosql.SystemSessionProperties.isCTEReuseEnabled;
+import static io.prestosql.SystemSessionProperties.isSnapshotEnabled;
 import static io.prestosql.sql.planner.plan.ExchangeNode.Scope.REMOTE;
 import static io.prestosql.sql.planner.plan.ExchangeNode.partitionedExchange;
 import static io.prestosql.sql.planner.plan.Patterns.cteScan;
@@ -40,7 +41,7 @@ public class AddExchangeAboveCTENode
     @Override
     public boolean isEnabled(Session session)
     {
-        return isCTEReuseEnabled(session);
+        return isCTEReuseEnabled(session) && !isSnapshotEnabled(session);
     }
 
     @Override

presto-main/src/main/java/io/prestosql/sql/planner/optimizations/AddReuseExchange.java

@@ -50,6 +50,7 @@ import java.util.stream.Collectors;
 import static io.prestosql.SystemSessionProperties.getSpillOperatorThresholdReuseExchange;
 import static io.prestosql.SystemSessionProperties.isColocatedJoinEnabled;
 import static io.prestosql.SystemSessionProperties.isReuseTableScanEnabled;
+import static io.prestosql.SystemSessionProperties.isSnapshotEnabled;
 import static io.prestosql.spi.operator.ReuseExchangeOperator.STRATEGY.REUSE_STRATEGY_CONSUMER;
 import static io.prestosql.spi.operator.ReuseExchangeOperator.STRATEGY.REUSE_STRATEGY_DEFAULT;
 import static io.prestosql.spi.operator.ReuseExchangeOperator.STRATEGY.REUSE_STRATEGY_PRODUCER;
@@ -78,7 +79,7 @@ public class AddReuseExchange
         requireNonNull(planSymbolAllocator, "symbolAllocator is null");
         requireNonNull(idAllocator, "idAllocator is null");
 
-        if (!isReuseTableScanEnabled(session) || isColocatedJoinEnabled(session)) {
+        if (!isReuseTableScanEnabled(session) || isColocatedJoinEnabled(session) || isSnapshotEnabled(session)) {
             return plan;
         }
         else {

presto-main/src/main/java/io/prestosql/sql/planner/optimizations/AddSortBasedAggregation.java

@@ -55,6 +55,7 @@ import java.util.Optional;
 import java.util.stream.Collectors;
 import java.util.stream.Stream;
 
+import static io.prestosql.SystemSessionProperties.isSnapshotEnabled;
 import static io.prestosql.SystemSessionProperties.isSortBasedAggregationEnabled;
 import static io.prestosql.spi.plan.TableScanNode.getActualColName;
 import static io.prestosql.sql.planner.plan.ChildReplacer.replaceChildren;
@@ -80,7 +81,7 @@ public class AddSortBasedAggregation
     @Override
     public PlanNode optimize(PlanNode plan, Session session, TypeProvider types, PlanSymbolAllocator planSymbolAllocator, PlanNodeIdAllocator idAllocator, WarningCollector warningCollector)
     {
-        if (!isSortBasedAggregationEnabled(session)) {
+        if (!isSortBasedAggregationEnabled(session) || isSnapshotEnabled(session)) {
             return plan;
         }
 

presto-main/src/main/java/io/prestosql/statestore/StateFetcher.java

@@ -302,7 +302,8 @@ public class StateFetcher
                 oldQueryInfo.getPreparedQuery(),
                 oldQueryInfo.getQueryStats(),
                 ErrorType.INTERNAL_ERROR,
-                QUERY_EXPIRE.toErrorCode());
+                QUERY_EXPIRE.toErrorCode(),
+                oldQueryInfo.isRecoveryEnabled());
 
         SharedQueryState newState = new SharedQueryState(
                 newQueryInfo,

presto-main/src/main/resources/webapp/assets/presto.css

@@ -480,6 +480,7 @@ pre {
     white-space: pre-line;
     max-height: 500px;
     max-width: 780px;
+    overflow: auto;
 }
 
 /** Query Monitor Toolbar **/

presto-main/src/main/resources/webapp/auditlog.html

@@ -22,7 +22,7 @@
     <![endif]-->
 
     <!-- jQuery -->
-    <script type="text/javascript" src="vendor/jquery/jquery-2.2.3.min.js"></script>
+    <script type="text/javascript" src="vendor/jquery/jquery-3.5.1.min.js"></script>
 
     <!-- Bootstrap JS -->
     <script type="text/javascript" src="vendor/bootstrap/js/bootstrap.js"></script>

presto-main/src/main/resources/webapp/querymonitor.html

@@ -21,7 +21,7 @@
 
 
     <!-- jQuery -->
-    <script type="text/javascript" src="vendor/jquery/jquery-2.2.3.min.js"></script>
+    <script type="text/javascript" src="vendor/jquery/jquery-3.5.1.min.js"></script>
 
     <!-- Bootstrap JS -->
     <script type="text/javascript" src="vendor/bootstrap/js/bootstrap.js"></script>

presto-main/src/main/resources/webapp/src/components/AuditLog.jsx

@@ -26,11 +26,14 @@ import MultiSelect from 'react-simple-multi-select';
 import Select from "rc-select";
 import Pagination from "rc-pagination";
 import localeInfo from "../node_modules/rc-pagination/es/locale/en_US";
+import UserStore from "../queryeditor/stores/UserStore";
+import UserActions from "../queryeditor/actions/UserActions";
 
 export class AuditLog extends React.Component {
     constructor(props) {
         super(props);
         this.state = {
+            user: '',
             auditlog:{},
             searchStringUser: '',
             searchStringDate:'',
@@ -135,7 +138,17 @@ export class AuditLog extends React.Component {
             });
             this.resetTimer();
         }.bind(this))
-            .error(function () {
+            .fail(function () {
+                this.resetTimer();
+            }.bind(this));
+
+        $.get(`../v1/audit/isAdmin`, function (user) {
+            this.setState({
+                user: user
+            });
+            this.resetTimer();
+        }.bind(this))
+            .fail(function () {
                 this.resetTimer();
             }.bind(this));
 
@@ -365,7 +378,6 @@ export class AuditLog extends React.Component {
         })
         let downloadString = downloadArray.join("&&");
 
-
         return (
             <div>
                 <div className='flex flex-row flex-initial header'>
@@ -377,8 +389,13 @@ export class AuditLog extends React.Component {
                         <div className="log toolbar-log">
                             <div className="col-xs-12 toolbar-audit">
                                 <div className="input-group-audit">
-                                    <input type="text" className="form-control form-control-auditlog search-bar" placeholder="User" name="User"
+                                    {this.state.user == "admin" ?
-                                           onChange={this.handleSearchStringChange} value={this.state.searchStringUser} />
+                                        <input type="text" className="form-control form-control-auditlog search-bar" placeholder="User" name="User"
+                                               onChange={this.handleSearchStringChange} value={this.state.searchStringUser}/>
+                                        :
+                                        <input type="text" className="form-control form-control-auditlog search-bar" placeholder="User" name="User"
+                                               onChange={this.handleSearchStringChange} value={this.state.user} readOnly="readOnly"/>
+                                    }
                                     <input type="text" id="date-picker" className="form-control form-control-auditlog search-bar" placeholder="Date" name="Date"
                                            onFocus={this.renderDateRangePicker}  onChange={this.handleSearchStringChange} value={this.state.searchStringDate}/>
                                     <select id="loglevel" className="selectpicker" data-style="select" onChange={this.handleSearchStringChange} value={this.state.searchStringLogLevel} name="LogLevel" >
@@ -389,6 +406,8 @@ export class AuditLog extends React.Component {
                                         <option value="Sql">Sql</option>
                                         <option value="WebUi">WebUI</option>
                                         <option value="Cluster">Cluster</option>
+
+
                                     </select>
                                     <a href={"../v1/audit/download?"+downloadString} target="_blank" className="btn"> Download <i className="icon fa fa-download"/> </a>
                                 </div>