a49cb41de7
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# cmake/ConfigureCompiler.cmake
# fdbserver/Knobs.cpp
# fdbserver/StorageCache.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/ThreadHelper.actor.h
# flow/serialize.h
# tests/CMakeLists.txt
2020-07-29 00:31:55 -07:00
5268dbe1ec
Use knob for remap cleanup window in write speed test.
2020-07-21 00:42:08 -07:00
d05b7ee785
Pager remap remover now accumulates a configurable version interval of page updates behind the oldest retained version which are used to coalesce updates to the same original page ID to reduce write amplification for many workloads.
2020-07-20 04:08:33 -07:00
f470ba8316
Remove using namespace std::rel_ops
...
This causes the following to not compile anymore
\#include <utility>
\#include <vector>
using namespace std::rel_ops;
int main() {
std::vector<int> xs;
return xs.rbegin() != xs.rend();
}
See https://godbolt.org/z/s1977n
2020-07-10 22:58:15 +00:00
1b68665228
Merge branch 'master' into release-6.3
2020-07-08 18:52:05 -07:00
1b55d75896
Remove TRIVIALLY_DESTRUCTIBLE macro
2020-07-04 19:28:10 -07:00
22f7f804b8
Merge branch 'release-6.3' into mengxu/merge-6.3-PR
2020-06-28 11:19:39 -07:00
8de70432a6
Fix memory leaks in VectorRef
...
VectorRef<T> relies on T being trivially destructible, but this was not
yet enforced. By statically asserting that T is trivially destructible,
we avoid leaking memory by not calling the destructor for non-trivially
destructible types
2020-06-24 10:54:19 -07:00
4216112e78
Fixed getWriteVersion and getLastCommittedVersion signatures
2020-06-17 22:38:54 -07:00
77e3b314f6
Mark IKeyValueStore::getStorageBytes const
2020-06-17 18:20:43 -07:00
2a393633b6
Improved const-correctness
...
Using the recently added IndexedSet::const_iterator
(https://github.com/apple/foundationdb/pull/3185 ), we can improve the
const-correctness of many functions. In this PR const is added where
applicable. Also, wherever I came across the following while adding
const, I made the following changes:
- virtual function overrides are marked as override
- NULL is replaced with nullptr
- git clang-format is applied
2020-06-17 18:20:43 -07:00
43d23bccb0
Applied clang-format.
2020-06-16 17:24:32 -07:00
ad68e44419
Change Redwood read concurrency lock to a reference because it must be able to live longer than the storage engine itself due to destruction order of actors if the storage engine is shut down while read operations are in progress.
2020-06-16 03:38:51 -07:00
9888fa7a10
Use a reference for m_commitReadLock to avoid invalid destruction.
2020-06-15 14:46:16 -07:00
c3c7db6a9d
Added flowlock around reads done on the write path so that those reads cannot starve reads to support reads done by the storage server.
2020-06-09 17:00:21 -07:00
8fdb81b48d
Tweaked BTree test random parameter limits to avoid test runs which take too long.
2020-06-06 21:10:52 -07:00
a28b5f0a8b
Possible bug fix, flow locks should be taken after initializing BTree cursor in KVStoreRedwood read functions otherwise it might be possible for the BTree to be closed before the flow lock wait returns, depending on destruction order of some things.
2020-06-06 21:10:13 -07:00
949fbd9145
New low level BTree cursor class which is designed for short lifetimes, does less memory allocation, and can be used to perform getRange() operations with less CPU overhead.
2020-06-04 23:23:14 -07:00
91f3b1c476
Merge pull request #3160 from satherton/feature-redwood
...
Update Redwood.
2020-05-20 15:04:01 -07:00
8c3cddc792
Metrics bug fix, page build count was being updated in an incorrect place.
2020-05-19 12:45:32 -07:00
53c4500694
Applied clang-format.
2020-05-19 04:11:39 -07:00
0d6f226938
Renamed some metrics to be clearer. Instead of "write" and "update" the terms are now "build" and "modify" which are less confusing and better represent what they mean in terms of DeltaTree operations.
2020-05-19 03:31:03 -07:00
7765168346
Redwood metrics now includes average item count, fill percentage, and record stored percentage for pages written or modified during the metrics interval.
2020-05-19 03:04:39 -07:00
f502236c31
Bug fix: BTree page header kvByte count was not being updated when pages are modified without rebuild.
2020-05-18 01:34:57 -07:00
ca8464881f
Applied clang-format.
2020-05-17 00:16:35 -07:00
d11fd75105
Bug fixes: No-hit page reads would count the access as a hit but would still move the page to the back of the eviction order. Page writes to uncached pages are no longer considered cache misses.
2020-05-17 00:16:10 -07:00
6347f76192
More Redwood metrics work. Reworked metrics standard output for unit tests.
2020-05-16 22:21:55 -07:00
e8626724f9
More refactor of Redwood metrics. Added per-level BTree stats. Added Pager total physical disk reads/writes, and uncacheable read hit/miss counts. Added logging to TraceEvents.
2020-05-16 02:51:57 -07:00
7cc8e281f2
Accidental override on non inherited function, which strangely only broke the Windows build.
2020-05-15 06:47:24 -07:00
0538edf7e7
Lazy / deferred subtree deletion had a few different names in the code, normalized everything to LazyClear. Removed accidental printfs.
2020-05-15 02:15:02 -07:00
32f4639168
Refactored page remap removal. The process is now called remap cleanup, does batches of remap entries in parallel, coalesces remaps of the same page ID to skip unnecessary writes, and has some knobs for controlling it. FIFOQueue now has peek() to support remap cleanup version lag limits. Added counters for remap cleanup and lazy subtree deletion. Refactored Redwood counters, normalized and grouped their names.
2020-05-15 02:10:51 -07:00
f51c1b352e
Applied clang-format.
2020-05-14 01:57:54 -07:00
7d84e1b7af
Eliminate undefined behaviour
2020-05-13 19:59:34 -07:00
421a6581c1
Lazy delete cycles now run more often and not just when the btree is having mutations actively merged into it. Specifically, it is launched upon recovery, then stopped and relaunched after every commit. This will make better use of I/O in between calls to commit(). Also added knobs to control lazy delete cycle parallelism and work limits.
2020-05-13 02:27:03 -07:00
3aef977af4
Removed a duplicative mutation buffer lookup when merging mutation buffer into internal page.
2020-05-12 20:48:31 -07:00
43f9e4dfad
Implemented concurrent read limit in IKeyValueStore interface for Redwood. Added knobs for Redwood page size, concurrent read limit, and page fill factor. Changed commitSubtree() recursion back to use a vector and waitForAll() because it seems to be lower overhead than ActorCollection.
2020-05-10 16:13:22 -07:00
ffbed1d84c
Use actor collection for subtree recursions.
2020-05-10 11:44:07 -07:00
f306666e21
Added commonPrefixLength for StringRefs that takes a prefix skip length.
2020-05-10 04:24:11 -07:00
5bb5bd2fd5
Improvements to Redwood set benchmark to improve readability and limit commit sizes to a record count or byte limit.
2020-05-10 04:21:25 -07:00
2ed43ebdf7
Added a non-persistent, memory-only option to DWALPager.
2020-05-10 02:19:19 -07:00
e4006c4b63
Apply clang-format.
2020-05-09 02:21:26 -07:00
cec63f2429
Merge pull request #3101 from satherton/feature-redwood
...
Redwood Update - Internal page incremental updates and efficient cleared/unchanged subtree skipping
2020-05-08 19:53:42 -07:00
163b291785
CommitSubtree() now skips recursing into adjacent subtrees that are uniformly cleared or unchanged very efficiently. This will be a huge reduction in CPU usage for random workloads.
2020-05-08 15:24:40 -07:00
1966dbb576
Checkpointing large refactor of how commitSubtree() works regarding internal pages such that now they can be incrementally modified and detecting changes involves far fewer comparison and staging of records.
2020-05-08 07:11:51 -07:00
92e672e2f0
Fixed compiler errors
2020-05-04 10:40:31 -07:00
dda0993d16
Apply clang-format to Redwood source.
2020-04-24 14:12:40 -07:00
ab46405bf4
Improvement to an edge case where writePages() would unexpectedly return only 1 page but not attempt to reuse the existing BTreePageID for it.
2020-04-23 04:23:05 -07:00
c80d28ac7e
Added several assertions and removed some checks for situations in commitSubtree() which should no longer be possible due to recent changes. This is mainly to test assumptions being made in the commitSubtree() refactor in progress. Set simulation to always choose Redwood, will revert for PR.
2020-04-21 04:12:39 -07:00
549ce29bdb
VersionDelta size options are now (0, 4, 6, 8), having removed the 2-byte option as it is not very useful.
2020-04-09 19:12:24 -07:00
6e5e7eea8d
Fixed a comment.
2020-04-09 17:14:25 -07:00
Evan Tschannen