Steve Atherton
|
2ce967bd77
|
Merge branch 'release-6.3' of github.com:apple/foundationdb into feature-redwood
|
2021-01-05 21:34:26 -08:00 |
|
Steve Atherton
|
4c28341351
|
Performance bug fix: Every page cache hit was calling forwardError() unnecessarily.
|
2021-01-05 21:34:16 -08:00 |
|
Steve Atherton
|
d7ebae07f5
|
Added yield to remap cleanup loop to prevent a stack overflow if too many queue entries are processed without waiting on IO.
|
2020-12-12 02:40:06 -08:00 |
|
Steve Atherton
|
0b46af2925
|
Added a simulation-only check for pager remap cleanup writing the same page twice in a cleanup cycle, which should never happen, but a bug leading to this was fixed recently. Adjusted some buggify logic to widen edge case coverage around remap cleanup parameters.
|
2020-10-14 21:48:03 -07:00 |
|
Steve Atherton
|
dc35f2b4f5
|
Bug fix: In page remap cleanup, if a page update's next update was at exactly the oldest retained version then the earlier update would still choose to copy the updated page over top of the original (but it shouldn't) which would race with the later update's copy if the same remap cleanup cycle pops both updates from the queue.
|
2020-10-13 02:26:46 -07:00 |
|
Steve Atherton
|
89ae0aaf39
|
In Redwood unit test, if the verify actor falls far behind the latest commit version it will unnecessarily verify the latest version repeatedly which extends the test's run time needlessly.
|
2020-08-29 23:47:54 -07:00 |
|
Steve Atherton
|
167d3c5a8f
|
To avoid running too long, the Redwood correctness unit test now limits the size of the KV verification map. This map only grows as the test runs, as it contains a record for every key set or clear committed during the test.
|
2020-08-29 16:44:38 -07:00 |
|
Steve Atherton
|
d53e6cf687
|
Bug fix in KV byte counts.
|
2020-08-18 17:58:57 -07:00 |
|
Steve Atherton
|
090241f0df
|
Count BTreeCursor page reads that come from the cursor's cache of pages as cache hits in RedwoodMetrics. BTree Unit Test now verifies all data at the initial version after recovery from disk.
|
2020-08-15 14:30:49 -07:00 |
|
Steve Atherton
|
051ec7066a
|
BTree unit test duration now targets a number of page read/writes so it should time out less often.
|
2020-08-13 01:50:51 -07:00 |
|
Steve Atherton
|
a817c9abbe
|
Redwood metrics was adding details with empty names.
|
2020-08-12 03:24:23 -07:00 |
|
Steve Atherton
|
c1ccbc3d27
|
Bug fixes. Internal page update would be missed if child page ID updates were expected but none were done. When rebuilding internal pages, child page ID updates were being written to the original page memory instead of the rebuild record set.
|
2020-08-12 01:33:07 -07:00 |
|
Steve Atherton
|
c18d16ad30
|
Multiple sibling page updates within the remap cleanup window can now avoid the second update step, copying updated contents over the original page, by instead updating the parent page to point to the new sibling locations.
|
2020-08-09 03:04:38 -07:00 |
|
Steve Atherton
|
5268dbe1ec
|
Use knob for remap cleanup window in write speed test.
|
2020-07-21 00:42:08 -07:00 |
|
Steve Atherton
|
d05b7ee785
|
Pager remap remover now accumulates a configurable version interval of page updates behind the oldest retained version which are used to coalesce updates to the same original page ID to reduce write amplification for many workloads.
|
2020-07-20 04:08:33 -07:00 |
|
Steve Atherton
|
43d23bccb0
|
Applied clang-format.
|
2020-06-16 17:24:32 -07:00 |
|
Steve Atherton
|
ad68e44419
|
Change Redwood read concurrency lock to a reference because it must be able to live longer than the storage engine itself due to destruction order of actors if the storage engine is shut down while read operations are in progress.
|
2020-06-16 03:38:51 -07:00 |
|
A.J. Beamon
|
9888fa7a10
|
Use a reference for m_commitReadLock to avoid invalid destruction.
|
2020-06-15 14:46:16 -07:00 |
|
Steve Atherton
|
c3c7db6a9d
|
Added flowlock around reads done on the write path so that those reads cannot starve reads to support reads done by the storage server.
|
2020-06-09 17:00:21 -07:00 |
|
Steve Atherton
|
8fdb81b48d
|
Tweaked BTree test random parameter limits to avoid test runs which take too long.
|
2020-06-06 21:10:52 -07:00 |
|
Steve Atherton
|
a28b5f0a8b
|
Possible bug fix, flow locks should be taken after initializing BTree cursor in KVStoreRedwood read functions otherwise it might be possible for the BTree to be closed before the flow lock wait returns, depending on destruction order of some things.
|
2020-06-06 21:10:13 -07:00 |
|
Steve Atherton
|
949fbd9145
|
New low level BTree cursor class which is designed for short lifetimes, does less memory allocation, and can be used to perform getRange() operations with less CPU overhead.
|
2020-06-04 23:23:14 -07:00 |
|
Evan Tschannen
|
91f3b1c476
|
Merge pull request #3160 from satherton/feature-redwood
Update Redwood.
|
2020-05-20 15:04:01 -07:00 |
|
Steve Atherton
|
8c3cddc792
|
Metrics bug fix, page build count was being updated in an incorrect place.
|
2020-05-19 12:45:32 -07:00 |
|
Steve Atherton
|
53c4500694
|
Applied clang-format.
|
2020-05-19 04:11:39 -07:00 |
|
Steve Atherton
|
0d6f226938
|
Renamed some metrics to be clearer. Instead of "write" and "update" the terms are now "build" and "modify" which are less confusing and better represent what they mean in terms of DeltaTree operations.
|
2020-05-19 03:31:03 -07:00 |
|
Steve Atherton
|
7765168346
|
Redwood metrics now includes average item count, fill percentage, and record stored percentage for pages written or modified during the metrics interval.
|
2020-05-19 03:04:39 -07:00 |
|
Steve Atherton
|
f502236c31
|
Bug fix: BTree page header kvByte count was not being updated when pages are modified without rebuild.
|
2020-05-18 01:34:57 -07:00 |
|
Steve Atherton
|
ca8464881f
|
Applied clang-format.
|
2020-05-17 00:16:35 -07:00 |
|
Steve Atherton
|
d11fd75105
|
Bug fixes: No-hit page reads would count the access as a hit but would still move the page to the back of the eviction order. Page writes to uncached pages are no longer considered cache misses.
|
2020-05-17 00:16:10 -07:00 |
|
Steve Atherton
|
6347f76192
|
More Redwood metrics work. Reworked metrics standard output for unit tests.
|
2020-05-16 22:21:55 -07:00 |
|
Steve Atherton
|
e8626724f9
|
More refactor of Redwood metrics. Added per-level BTree stats. Added Pager total physical disk reads/writes, and uncacheable read hit/miss counts. Added logging to TraceEvents.
|
2020-05-16 02:51:57 -07:00 |
|
Steve Atherton
|
7cc8e281f2
|
Accidental override on non inherited function, which strangely only broke the Windows build.
|
2020-05-15 06:47:24 -07:00 |
|
Steve Atherton
|
0538edf7e7
|
Lazy / deferred subtree deletion had a few different names in the code, normalized everything to LazyClear. Removed accidental printfs.
|
2020-05-15 02:15:02 -07:00 |
|
Steve Atherton
|
32f4639168
|
Refactored page remap removal. The process is now called remap cleanup, does batches of remap entries in parallel, coalesces remaps of the same page ID to skip unnecessary writes, and has some knobs for controlling it. FIFOQueue now has peek() to support remap cleanup version lag limits. Added counters for remap cleanup and lazy subtree deletion. Refactored Redwood counters, normalized and grouped their names.
|
2020-05-15 02:10:51 -07:00 |
|
Steve Atherton
|
f51c1b352e
|
Applied clang-format.
|
2020-05-14 01:57:54 -07:00 |
|
tclinken
|
7d84e1b7af
|
Eliminate undefined behaviour
|
2020-05-13 19:59:34 -07:00 |
|
Steve Atherton
|
421a6581c1
|
Lazy delete cycles now run more often and not just when the btree is having mutations actively merged into it. Specifically, it is launched upon recovery, then stopped and relaunched after every commit. This will make better use of I/O in between calls to commit(). Also added knobs to control lazy delete cycle parallelism and work limits.
|
2020-05-13 02:27:03 -07:00 |
|
Steve Atherton
|
3aef977af4
|
Removed a duplicative mutation buffer lookup when merging mutation buffer into internal page.
|
2020-05-12 20:48:31 -07:00 |
|
Steve Atherton
|
43f9e4dfad
|
Implemented concurrent read limit in IKeyValueStore interface for Redwood. Added knobs for Redwood page size, concurrent read limit, and page fill factor. Changed commitSubtree() recursion back to use a vector and waitForAll() because it seems to be lower overhead than ActorCollection.
|
2020-05-10 16:13:22 -07:00 |
|
Steve Atherton
|
ffbed1d84c
|
Use actor collection for subtree recursions.
|
2020-05-10 11:44:07 -07:00 |
|
Steve Atherton
|
f306666e21
|
Added commonPrefixLength for StringRefs that takes a prefix skip length.
|
2020-05-10 04:24:11 -07:00 |
|
Steve Atherton
|
5bb5bd2fd5
|
Improvements to Redwood set benchmark to improve readability and limit commit sizes to a record count or byte limit.
|
2020-05-10 04:21:25 -07:00 |
|
Steve Atherton
|
2ed43ebdf7
|
Added a non-persistent, memory-only option to DWALPager.
|
2020-05-10 02:19:19 -07:00 |
|
Steve Atherton
|
e4006c4b63
|
Apply clang-format.
|
2020-05-09 02:21:26 -07:00 |
|
Evan Tschannen
|
cec63f2429
|
Merge pull request #3101 from satherton/feature-redwood
Redwood Update - Internal page incremental updates and efficient cleared/unchanged subtree skipping
|
2020-05-08 19:53:42 -07:00 |
|
Steve Atherton
|
163b291785
|
CommitSubtree() now skips recursing into adjacent subtrees that are uniformly cleared or unchanged very efficiently. This will be a huge reduction in CPU usage for random workloads.
|
2020-05-08 15:24:40 -07:00 |
|
Steve Atherton
|
1966dbb576
|
Checkpointing large refactor of how commitSubtree() works regarding internal pages such that now they can be incrementally modified and detecting changes involves far fewer comparison and staging of records.
|
2020-05-08 07:11:51 -07:00 |
|
Markus Pilman
|
92e672e2f0
|
Fixed compiler errors
|
2020-05-04 10:40:31 -07:00 |
|
Steve Atherton
|
dda0993d16
|
Apply clang-format to Redwood source.
|
2020-04-24 14:12:40 -07:00 |