Commit Graph

253 Commits

Author SHA1 Message Date
rajeevrastogi 412d70c7f0 [I3VFJ1] CTE Optimizer scheduling issue fix. CTE Consumer to schedule on same nodes as feeder 2021-06-14 22:07:39 +05:30
sandeep f5f40b3f95 [I3UQM6] Sort aggregation review fix 2021-06-10 22:12:11 +05:30
i-robot 86f272a01d !901 [I3FHGK] Improve snapshot stability
Merge pull request !901 from sandygao/improve-snapshot-stability
2021-06-10 01:13:56 +08:00
NeerajUnnikrishnan 1d6207faba Fix for timestamp semantics 2021-06-07 11:21:38 -04:00
sandeep b86843774e [I3T90] sort aggregation 2021-06-06 22:39:25 +05:30
Sandy Gao a8ca0ee314 Improve snapshot handling of transactional tables
- Merged file list contains folders for transactional tables
- Version file may be empty sometimes
2021-06-04 10:25:01 -04:00
nitin.kashyap c434bdf365 [I3U39U]fix access to stats data when empty stats comes from metastore 2021-06-03 16:16:52 +05:30
nitin.kashyap 0de42742d7 [I3SI4C] added config for batch size in config 2021-05-28 19:15:20 +05:30
i-robot 8d8bd63e36 !882 [#I3SFMD] Concurrent Query Optimization
Merge pull request !882 from RajeevRastogi/physical.plan.schedule.opt
2021-05-26 23:52:29 +08:00
rajeevrastogi a8b2e85340 [#I3SFMD] Concurrent query optimization 2021-05-26 20:25:45 +05:30
i-robot b9b092a65a !874 [I3FHGK] Improve snapshot stability
Merge pull request !874 from sandygao/improve-snapshot-stability
2021-05-26 21:32:29 +08:00
i-robot c6a3412e76 !881 Support Hudi MOR table Snapshot/Read-Optimized queries
Merge pull request !881 from Maxiaoqi/hudi_mor_support
2021-05-26 09:35:58 +08:00
NeerajUnnikrishnan 01b224af0c Predicate pushdown for CTE 2021-05-25 03:24:40 -04:00
maxiaoqi2020 e3eb3a6ac6 Support Hudi COW/MOR table Snapshot/Read-Optimized queries
Update Presto-hive UT tests
2021-05-24 20:20:41 +08:00
i-robot 0dcb194614 !877 Query Concurrency Code Optimizations
Merge pull request !877 from Surya Sumanth/concurrency_optimizations
2021-05-24 02:23:16 +08:00
SURYA SUMANTH N d379529ac3 Query Concurrency Code Optimizations 2021-05-23 23:28:45 +05:30
pradeep 23645d9b85 [I3SI4C] Delete empty staging directory and subdirectories through a single api call with recursive option 2021-05-22 20:32:51 +05:30
Sandy Gao 0e4683dc60 Improve snapshot stability
- Remove hive files from old tasks
- Recover from unexpected results from tasks
- Handle "create table if not exists" statements
- Improve mechanism to verify query rerun completeness
- Handle case where table-scan task finishes after resume is done
2021-05-20 16:05:19 -04:00
Sandy Gao 3aea36f7cc Improve snapshot stability:
- Avoid sharing memory context between old and new tasks
- Union sources produce same number of markers
- Exchange operator will be blocked waiting for noMoreSplits to be set
- Made noMoreSplits call happen early by scheduling all tasks with marker 0
- QuerySnapshotManager record snapshot initiated for accurate cleanup of snapshot subfiles
- Fix typo in driver logs
- Update HashBuilder comment
- Consider restored pages in ExchangeOperator isBlocked()
- Remove special handling in MergeOperator isBlocked()
- Fix issue where shared marker is updated with different origins
- Avoid too frequent resumes
2021-05-04 19:33:15 -04:00
i-robot e120e86739 !848 [I3MZ4W] Support for Hive ORC Stripe Filtering using predicate
Merge pull request !848 from Surya Sumanth/stripefiltering
2021-04-22 22:48:48 +08:00
SURYA SUMANTH N eba9235039 Support for Hive ORC Stripe Filtering using predicate 2021-04-22 11:12:31 +05:30
Han Weng 2b3e3851b6 Disable unstable UT 2021-04-21 15:43:54 -04:00
Sandy Gao 9764938d3f Improve snapshot stability
- Avoid index exception when renaming snapshot files
- Query may be in rescheduling state when scheduler finishes
- Ensure cancel-to-resume finishes so query can be retried
2021-04-20 11:12:05 -04:00
RaymondXu 027874cb2b - Added parallelSourceCount to MarkerSplitSource to handle multiple table scan to one ExchangeOperator
- multiple table writer pipeline error commiting
- Avoid snapshot deadlock situation
2021-04-16 17:09:54 -04:00
Han Weng 23de567d2f Update index cache UTs 2021-04-14 17:29:18 -04:00
i-robot 60c13ed4f1 !835 [I3E3LE] matching positions updated for all nulls case
Merge pull request !835 from Nitin-Kashyap/selectReadFix
2021-04-14 14:25:59 +08:00
nitin.kashyap 89ffaf3c44 [I3E3LE] matching positions updated for all nulls case 2021-04-14 10:51:02 +05:30
sandygao 3d9d6b200a !838 [I3FHGK] Improve snapshot stability
* Update file names before moving them to table folder
2021-04-14 09:23:16 +08:00
Sandy Gao 92be9893ee Improve snapshot stability:
- Fix issue where hash builder causes "too many components" error
- Do not delete task result if task instance id is not available
- Do not delete files when table-writer is cancelled or restored
  - Instead remove files before resume
- Do not move cancelled tasks to aborted, and do not close outputBuffer
- Improve synchronization when marking drivers as finished
- Fix potential NPE when a task error is received
- Improved logging
2021-04-13 17:08:15 -04:00
sandygao f154d8f117 !813 [I3FHGK] Improve snapshot stability
* Various fixes to snapshot feature:
2021-04-10 10:16:19 +08:00
i-robot 6129948175 !814 [I3C2KY] URI Special Character Escape Fix
Merge pull request !814 from Surya Sumanth/UriHashSymEscapeFix
2021-04-08 16:14:44 +08:00
SURYA SUMANTH N d0225487ef URI Special Character Escape Fix 2021-04-08 10:32:02 +05:30
i-robot 1950cc8a51 !796 UT reuse Exchange
Merge pull request !796 from sandeep/bugfix-ReuseExchangeUT
2021-04-05 13:21:17 +08:00
sandeep 62a4602692 !796 [I3E0XB] UT reuse exchange 2021-04-02 13:15:20 +05:30
nitin.kashyap 8aae9cf253 [I3E3LE] handle all null rows in longSelective reader 2021-04-01 18:34:25 +05:30
Raghunandan 34b34313b3 [maven-release-plugin] prepare for next development iteration 2021-03-31 11:15:11 +05:30
Daniel Zhang c9dbed474f TestIndexCache additional try to get actualSplitsIndex for correct assertion 2021-03-26 17:01:26 -04:00
Sandy Gao fff4183888 Revert "fix resource leak problems"
- This reverts commit a997fdfe
- Added fix for potential NPE when rolling back hive writes
- Fix MarkerSplitSource's sentFinalMarker for resume marker
- Avoid failing query if only 1 worker is left
2021-03-24 23:46:01 -04:00
tushengxia a997fdfe3d fix resource leak problems
add docs for hbase insert

fix conflicts
2021-03-24 11:34:36 +08:00
i-robot 8138e6aee4 !758 Improve index record lookup performance
Merge pull request !758 from Han_Weng/fix-index-metastore-jdbc
2021-03-24 07:57:49 +08:00
Han Weng 543a7e6de8 Improve index lookup performance and optimize temp folder usage 2021-03-23 18:01:14 -04:00
i-robot b5f3222690 !745 Add last modified time as part of orc cache key
Merge pull request !745 from Jiahao/QA-693
2021-03-24 03:17:50 +08:00
Jessica Surya 199c62313b - I2D63N Update snapshot configurations and documentation
- I2D63H Detect unsupported scenarios and disable snapshot
2021-03-23 10:47:58 -04:00
jiahao 81c93fbd44 add last modified time as part of OrcCacheKey 2021-03-20 17:29:30 -07:00
Sundar Annamalai e2093f1e89 Fixes intermittent star tree index test failures 2021-03-20 13:55:14 -04:00
Sandy Gao 61b7419999 Improvements and fixes to distributed snapshot:
- Fix issue with updating Hive transactional tables
- When there aren't enough worker available, retry earlier snapshots
- Move state save and load code to TaskSnapshotManager
- Reliably delete snapshots for finished queries
- Recognize restore failures on first error
- Fix issue with MultiChannelGroupByHash capture/restore
- Handle error when cleaning up sub files
- Handle case where no file system profile is available
2021-03-19 12:34:50 -04:00
Neerajunnikrishnan 5feccafb74 Handle CVEs for libraries and carbondata module 2021-03-17 19:27:39 +05:30
Sandy Gao 38e1d751b5 Address review comments 2021-03-15 13:08:24 -04:00
Sandy Gao f919f7f832 Snapshot - Updated and new unit tests 2021-03-15 13:08:23 -04:00
Sandy Gao 4eff6fc753 Snapshot - various other changes
- Support snapshot in Hive connector (ORC only)
- Update query editor to support new query states
- Update TPC-H and TPC-DS connectors to support rescheduling of splits
2021-03-15 13:08:23 -04:00