dda0993d16
Apply clang-format to Redwood source.
2020-04-24 14:12:40 -07:00
ab46405bf4
Improvement to an edge case where writePages() would unexpectedly return only 1 page but not attempt to reuse the existing BTreePageID for it.
2020-04-23 04:23:05 -07:00
c80d28ac7e
Added several assertions and removed some checks for situations in commitSubtree() which should no longer be possible due to recent changes. This is mainly to test assumptions being made in the commitSubtree() refactor in progress. Set simulation to always choose Redwood, will revert for PR.
2020-04-21 04:12:39 -07:00
549ce29bdb
VersionDelta size options are now (0, 4, 6, 8), having removed the 2-byte option as it is not very useful.
2020-04-09 19:12:24 -07:00
6e5e7eea8d
Fixed a comment.
2020-04-09 17:14:25 -07:00
8de2bbb2dc
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2020-04-09 09:39:49 -07:00
d4334256f1
Bug fix: BTree::writePages() was not accounting for the small amount of unpredictability in node overhead sizes which could lead to serializing a tree that doesn't fit into its destination page buffer. WritePages() now skips the prefix common to all records in the input set when calling deltaSize() to estimate resulting DeltaTree serialized sizes.
2020-04-08 20:18:41 -07:00
9e82788c28
Bug fix, BTree header wasn't large enough to hold the largest possible root page references.
2020-04-08 03:38:37 -07:00
d979200799
Bump format version.
2020-04-08 03:29:12 -07:00
31fd4cd35a
Improved RedwoodRecord encoding/decoding tests, which found bugs in the version field scheme. Simplified it to select between different fixed size native integer formats.
2020-04-08 03:25:04 -07:00
7a4817b8bb
Re-enable ART mutation buffer in Redwood.
2020-04-07 11:47:19 -07:00
d7ca823a8d
Refactored RedwoodRecordRef::Delta to fix at least one bug and stop using variable length integers with extension bits. Instead, two (recently available) header bits now indicate which of four struct definitions the Delta is using. The resulting code is a bit repetitive but also easier to read and 15% faster in the RedwoodRecord Delta speed test vs the old version.
2020-04-07 04:47:08 -07:00
ab6e8e9b09
Bug fix, expanded page size calculation when the small page limit is reached was not correct in a couple of ways.
2020-04-06 23:37:32 -07:00
a354d7a037
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2020-04-06 20:29:12 -07:00
9bf0b5e71f
RedwoodRecord simplified to remove chunked key-value pair concept. Cursor and InternalCursor updated to be correct (but not optimal) for new record definition. DeltaTree::seek* refactored to support more seek types and with minor code duplication. Lots of debug output improvements. Refactored how bulk pages are written and introduced fill factor percentage (though since internal pages are not yet incrementally modified this is temporarily worse for those).
2020-04-06 20:27:30 -07:00
28c7899b76
Disable ART mutation buffer in Redwood until windows build issue is fixed.
2020-04-03 13:04:32 -07:00
1ed425233a
DeltaTree now has two size modes, 2 byte integer offers for <= 65k and 4 byte integer offsets for larger.
2020-04-01 04:24:40 -07:00
b10c6dad41
Added benchmark to DeltaTree RedwoodRecordRef test that does random seeks using a pool of cursors with different mirrors to test the speed of having to decode DeltaTree nodes much more often.
2020-04-01 04:12:18 -07:00
98061c80bb
Applying initial feedback from Steve
...
Removing iterator inheritance, and hence adding conversion operator
Using typedef to decide which MB implementation to use
2020-03-31 09:28:33 +02:00
59eede009c
Adding ART-based MutationBuffer
2020-03-30 13:10:44 +02:00
087c6fa33d
Merge branch 'master' into feature-redwood
2020-02-26 12:25:04 -08:00
636c256606
Simplified return value of CommitSubtree to a single ChildLinkSetRef, which doesn't contain a version. Removed unnecessary record copying in several places in CommitSubtree.
2020-02-24 15:18:34 -08:00
72a549df14
Added basic mutation buffer insert/lookup test.
2020-02-24 00:31:09 -08:00
1e45bb2be3
Further refactoring of MutationBuffer to make upcoming ART implementation easier to plug in. Key and mutation for an iterator position are now accessed via member functions which gives the implementation an opportunity to decode or reconstitute keys on-demand.
2020-02-23 23:44:15 -08:00
eb7016a09a
Some more refactoring of MutationBuffer so it will be easier to have multiple implementations.
2020-02-23 22:48:43 -08:00
710bc3ecb1
Added optimization for single key clears to avoid an additional mutation buffer boundary key insertion. Made mutation buffer members private and created additional member functions to avoid needing access to the private members.
2020-02-23 22:24:16 -08:00
3618dff07b
Corrected statement in test output to reflect new logic.
2020-02-23 00:21:39 -08:00
078f85fea7
Bug fix. Pager could return an an oldest version that is no longer readable because it is being expired in the current commit cycle.
2020-02-23 00:15:35 -08:00
648040b070
Removed versioned records from the mutation buffer and commit path because support for this was incomplete and has high overhead costs when only exact committed versions are being read. The BTree could still contain versioned records in the future as the read cursors still support this, but some optimizations were added for when all internal records are at version 0 which is the case and has been for quite some time. Mutation buffer now stores keys and values in one arena per buffer.
2020-02-22 20:21:06 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
f1ec780b31
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-redwood
2020-02-21 17:43:11 -08:00
5a1884d054
Leaf page modifications are now made incrementally until the DeltaTree is out of space or the tree is too imbalanced, at which point it falls back to a linear merge and page(s) rebuild.
2020-02-21 16:26:44 -08:00
986e533c8c
Bug fix, DeltaTree was not updating numItems after erase / insert operations which only change the deleted flag of an existing equivalent entry.
2020-02-20 22:49:36 -08:00
893729e3a8
Bug fix, Redwood was not using direct IO where available.
2020-02-20 10:29:20 -08:00
0c7c815396
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# tests/CMakeLists.txt
2020-02-12 16:12:57 -08:00
93e3e36d52
Changed RedwoodRecordRef::compare() to include value and updated VersionedBTree to adapt to this change. This fixes an (uncommitted) bug where DeltaTree inserts of a record matching a deleted record except for value would simply unhide the deleted record. For DeltaTree delete/insert sequences to work correctly compare() must only return 0 when the records are fully equivalent.
2020-02-12 01:18:35 -08:00
151f070809
Track failed evictions in ObjectCache.
2020-02-10 12:00:40 -08:00
9f4e39f1ef
DeltaTree now supports inserting a deleted item that is still present as a deleted node by simply clearing the node's deleted flag. Insertion of duplicates (determined by T::compare() == 0) is no longer allowed.
2020-02-07 01:00:11 -08:00
df2b0452b4
Step 3 of fixing storage server range reads: change return type of readRange from VectorRef<KeyValueRef> to RangeResultRef.
2020-02-06 13:19:24 -08:00
9cf9c46668
DeltaTree now tracks internal used/free/deleted space, and insert will fail safely if a new item does not fit.
2020-02-05 15:30:04 -08:00
9dbb9e4cec
DeltaTree now supports node deletion by flagging nodes as deleted (a state which the T::Delta must be able to store and return). DeltaTree::Cursor will hide deleted nodes from view.
2020-02-04 01:22:27 -08:00
460c9b78d8
DeltaTree improvements. DecodedNodes store non-parent ancestor pointer which enables moveNext() and movePrev() operations to be O(1) from any leaf node. This also enables seekLessThanOrEqual to accept a hint in the form of a cursor with a position close to the target to avoid traversal from the root when the destination is close to the hint. DeltaTree T::compare() now must take a skipLen argument which is used by insert() and seek operations to skip over bytes known to be shared between the new item or query and every node in the tree.
2020-01-31 00:32:48 -08:00
da73164eda
Move crc32c from fdbrpc to flow
...
So that we can use it from a piece of flow code without breaking module
boundaries.
Also rename generated-constants to crc32c-generated-constants so that
it's more apparent that they're related files.
2020-01-13 18:19:30 -08:00
560c1da805
Fix UBSAN report
...
/home/anoyes/workspace/foundationdb/fdbserver/VersionedBTree.actor.cpp:1606:10: runtime error: null pointer passed as argument 2, which is declared to never be null
2019-12-04 09:42:34 -08:00
04b2338e60
Added sequential insert speed test.
2019-11-23 00:40:58 -08:00
5f1644f293
DeltaTree::Reader is now DeltaTree::Mirror and supports insertion into a DeltaTree. DeltaTrees now support an item count, so BTreePage no longer has an item count, so the VersionedBTree format version has been bumped.
2019-11-23 00:09:11 -08:00
a9af2de1d2
Added option in redwood prefixed set test to use randomly generated prefixes in sorted order.
2019-11-20 15:55:59 -08:00
d91d744fd7
Typedefs to simplify ObjectCache a bit.
2019-11-20 03:20:23 -08:00
8d973ce762
Bug fix: Any time the least recently used ObjectCache entry is not evictable the effective size of the cache would grow by one entry for each read and will never shrink even once there are enough evictable pages to return the cache to its configured size. This has probably never actually happened because evictability of Redwood pages is currently based on having no pending IO, but it would be more of a problem if evictability were redefined to require a reference count of 1.
2019-11-19 23:45:13 -08:00
9e1e0d731d
Incremental subtree deletion now processes pages in parallel.
2019-11-18 02:34:37 -08:00
Steve Atherton