Commit Graph

1736 Commits

Author SHA1 Message Date
A.J. Beamon adb8d3ec7a Fix a memory issue when getting key range locations that manifests in particular when getting range split points 2023-01-23 11:01:37 -08:00
A.J. Beamon b10d1f227b Remove tenant name from the TenantInfo object 2023-01-20 14:04:43 -08:00
A.J. Beamon d47a2ab60f
Merge pull request #9128 from sfc-gh-ajbeamon/transactions-use-tenant-object
Transactions take a tenant object rather than a tenant name
2023-01-20 12:06:36 -08:00
He Liu ec6716ff2e
Restore kv (#9030)
* Allow multiple keyranges in CheckpointRequest.
Include DataMove ID in CheckpointMetaData.

* Use UID dataMoveId instead of Optional<UID>.

* Implemented ShardedRocks::checkpoint().

* Implementing createCheckpoint().

* Attempted to change getCheckpointMetaData*() for a single keyrange.

* Added getCheckpointMetaDataForRange.

* Minor fixes for NativeAPI.actor.cpp.

* Replace UID CheckpointMetaData::ssId with std::vector<UID>
CheckpointMetaData::src;

* Implemented getCheckpointMetaData() and completed checkpoint creation
and fetch in test.

* Refactoring CheckpointRequest and CheckpointMetaData

rename `dataMoveId` as `actionId` and make it Optional.

* Fixed ctor of CheckpointMetaData.

* Implemented ShardedRocksDB::restore().

* Tested checkpoint restore, and added range check for restore, so that
the target ranges can be a subset of the checkpoint ranges.

* Added test to partially restore a checkpoint.

* Refactor: added checkpointRestore().

* Sort ranges for comparison.

* Cleanups.

* Check restore ranges are empty; Add ranges in main thread.

* Resolved comments.

* Fixed GetCheckpointMetaData range check issue.

* Refactor CheckpointReader for CF checkpoint.

* Added CheckpointAsKeyValues as a parameter for newCheckpointReader.

* PhysicalShard::restoreKvs().

* Added `ranges` in fetchCheckpoint.

* Added RocksDBCheckpointKeyValues::ranges.

* Added ICheckpointIterator and implemented for RocksDBCheckpointReader.

* Refactored OpenAction for CheckpointReader, handled failure cases.

* Use RocksDBCheckpointIterator::end() in readRange.

* Set CheckpointReader timout and other Rocks read options.

* Implementing fetchCheckpointRange().

* Added more CheckpointReader tests.

* Cleanup.

* More cleanup.

* Added fetchCheckpointRanges test.

* Implemented restore of kv-based checkpoint.

* Improved CheckpointRestore test for kv, non-kv, as well as partial
restore.

* Added test of merge.

* Fixed merge test.

* Cleanup.

* Resolved comments.

Co-authored-by: He Liu <heliu@apple.com>
2023-01-20 09:58:34 -08:00
A.J. Beamon 258198ca20 Merge branch 'main' into transactions-use-tenant-object
# Conflicts:
#	fdbclient/include/fdbclient/DatabaseContext.h
#	fdbserver/workloads/BulkLoadWithTenants.actor.cpp
2023-01-20 09:06:39 -08:00
Hui Liu 36e8e5a3bb
Merge pull request #9176 from sfc-gh-huliu/restoreversion
Restore to a previous version
2023-01-19 17:37:23 -08:00
Hui Liu c85b984c3a blobrestore to previous point of time 2023-01-19 14:45:01 -08:00
Nim Wijetunga 82c92abca2
Change Noisy Sev30s to Sev20s (#9180)
change noisy sev30 traces to sev20
2023-01-19 14:06:34 -08:00
A.J. Beamon e1d48d28bf Remove tenant name from the TenantInfo object 2023-01-13 08:58:15 -08:00
Josh Slocum 0fc5c6c1e2
tenant entry was removed, changed to fixme (#9132) 2023-01-12 18:39:22 -06:00
Josh Slocum b6450f9eaa
More ss cf perf fixes main (#9109)
* changing future version logic for change feed fetch

* Optimizing change feed data structures and accesses

* coalescing change feed request ranges for merge cursor if they're to the same team

* fixing over-read of memory mutations for change feeds

* feed filter mutations common prefix cpu optimiation

* fix formatting
2023-01-12 16:15:05 -06:00
Xiaoxi Wang 07b37245dd merge upstream/main; solve merge conflicts 2023-01-12 12:29:03 -08:00
A.J. Beamon f999623bb1 Add a tenant lookup interface and use it when starting transactions 2023-01-06 15:51:12 -08:00
Xiaoxi Wang e3d09a5cbd Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/tenantCheck 2023-01-04 16:16:15 -08:00
Xiaoxi Wang 8266f52dea
Merge pull request #9012 from sfc-gh-xwang/feature/main/wiggleDelay
Persist accumulated wiggle delay
2023-01-04 16:14:09 -08:00
Hui Liu e3bf79cf71 Add correctness test for blob restore 2023-01-04 11:10:34 -08:00
Xiaoxi Wang bbcb3cc018 extract KeyBackedConfig, StorageWiggleData class; solve template resolution problem; solve MV txn and native api conflict by splitting RunTransaction file 2023-01-02 23:34:39 -08:00
Xiaoxi Wang 8ff6d6c7ee merge upstream/main; solve conflicts 2022-12-17 15:57:07 -08:00
Xiaoxi Wang 61e3917d58 check tenant access in applyMetadataToCommittedTransactions 2022-12-14 13:27:50 -08:00
Xiaoxi Wang c12de23824 Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/wiggleDelay 2022-12-11 14:27:22 -08:00
Jingyu Zhou fee561921a Fix assertion failure from changeFeedTSSValidator() 2022-12-10 10:02:14 -08:00
sfc-gh-tclinkenbeard 68f14f017c Fix clang 15 compiler warnings 2022-12-08 13:59:37 -08:00
Xiaoxi Wang ccc494319c perpetual wiggle key functions 2022-12-08 16:46:05 -05:00
Kevin Hoxha 3cea754ba3 metrics: Add OTEL metric definitions 2022-12-08 10:07:11 -08:00
Kevin Hoxha 188e635b2e metrics: Avoid name conflicts in MetricCollection
- In Counter constructor, make sure that we combine CounterCollection name
- In IMetric constructor, add a trace event and assertion in case of collision
- Rename ClusterRecoveryData CounterCollection name from master to ClusterRecoveryData
2022-12-08 10:07:11 -08:00
FoundationDB CI 86d6106dc1
format source code after switch to clang 15 2022-12-08 17:26:45 +00:00
Hao Fu d11b5b44e0
Remove boundaryAndExist (#8994)
* Remove boundaryAndExist

Client knows whether an entry is a boundary, and client also can
see whether the record for that entry exist by checking whether
the rangeResult is empty. This field is useless.

This field breaks backward compatibility, need to remove it.
2022-12-07 18:08:15 -08:00
Marian Dvorsky 085fce8478
Distributed tracing related improvements (#8942)
Several fixes/improvements related to distributed traces.

Remove "key" attributes and the TRACING_SPAN_ATTRIBUTES_ENABLED knob: we almost never want to log actual keys (as they can contain private data), however, we do want to use other span attributes.
In Transaction::setTransactionID, properly propagate spanContext flags, and set all copies of spancontext
2022-12-07 17:58:52 +01:00
A.J. Beamon b1fb8e8c7b Resolve merge issue by renaming namespace (combining with existing TenantAPI namespace) 2022-12-05 15:47:58 -08:00
A.J. Beamon ce4a2a55b2 Remove the usage of tenant names (mostly) from the storage server 2022-12-05 15:44:41 -08:00
Trevor Clinkenbeard 6f70fc243b
Add rare code probe annotations (#8948) 2022-12-05 09:23:47 -08:00
sfc-gh-tclinkenbeard 994fd4aa41 Remove rare annotation from some code probes 2022-12-01 09:59:08 -08:00
Xiaoge Su 7a0829a14e fixup! Remove comments 2022-11-30 14:31:09 -08:00
Xiaoge Su 98d3357c59 fixup! Reformat source 2022-11-30 14:31:09 -08:00
Xiaoge Su a0fc3685c9 fixup! Update code per comments 2022-11-30 14:31:09 -08:00
Xiaoge Su a355289d63 Remove cx->clearWatchMetadata() when connection file changed
In NativeAPI.actor.cpp, ::watch ACTOR will call cx->clearWatchMetadata()
when the connectionFileChanged event is triggered. After that, it will
create a new watch metadata for itself.

If there are multiple watches, each of them that receives the
cnnectionFileChanged will clear *all* watch data and create *one* watch
for itself. This does not makes sense. A watch should only clear the
metadata, and then create one, for itself.

cx->clearWatchMetadata() is only used there, thus removed.
2022-11-30 14:31:09 -08:00
Xiaoge Su b35d1b614e Revert "Merge pull request #8602 from apple/revert-8498-mmmm"
This reverts commit 4d789b2fd9, reversing
changes made to 9625efd5b9.
2022-11-30 14:31:09 -08:00
He Liu 3d2124df80
Checkpoint restore sharded rocks (#8758)
* Allow multiple keyranges in CheckpointRequest.
Include DataMove ID in CheckpointMetaData.

* Use UID dataMoveId instead of Optional<UID>.

* Implemented ShardedRocks::checkpoint().

* Implementing createCheckpoint().

* Attempted to change getCheckpointMetaData*() for a single keyrange.

* Added getCheckpointMetaDataForRange.

* Minor fixes for NativeAPI.actor.cpp.

* Replace UID CheckpointMetaData::ssId with std::vector<UID>
CheckpointMetaData::src;

* Implemented getCheckpointMetaData() and completed checkpoint creation
and fetch in test.

* Refactoring CheckpointRequest and CheckpointMetaData

rename `dataMoveId` as `actionId` and make it Optional.

* Fixed ctor of CheckpointMetaData.

* Implemented ShardedRocksDB::restore().

* Tested checkpoint restore, and added range check for restore, so that
the target ranges can be a subset of the checkpoint ranges.

* Added test to partially restore a checkpoint.

* Refactor: added checkpointRestore().

* Sort ranges for comparison.

* Cleanups.

* Check restore ranges are empty; Add ranges in main thread.

* Resolved comments.

* Fixed GetCheckpointMetaData range check issue.

* Fixed error description.

Co-authored-by: He Liu <heliu@apple.com>
2022-11-30 08:22:14 -08:00
Dennis Zhou f10247aa25
blob: purge range fix (#8867)
* blob: adjacent purge unit test

Ensure [A, B), [B, C) can be purged independently.

* blob: fix purge alignment condition

We only need to check if the first range starts before the end.
Additionally, an empty range is problematic if passed to getRange() and
we're expecting additional values past the end of the range.
2022-11-21 12:26:19 -06:00
Lukas Joswiak 750669c9d8 Remove unnecessary buggify 2022-11-21 09:03:22 -08:00
Lukas Joswiak 7d73d52a91 Enable tracing in simulation
This will help test the flow of span IDs from the client all the way to
the storage servers.
2022-11-21 09:03:22 -08:00
sfc-gh-tclinkenbeard 3c97f43138 Change Histogram::Unit::microseconds to milliseconds 2022-11-21 08:03:56 -08:00
Hui Liu de8285f11d
Merge pull request #8834 from sfc-gh-huliu/status
Show blob restore in fdbcli status command
2022-11-18 16:03:30 -08:00
Hui Liu 2e62822183 Show blob restore in fdbcli status command 2022-11-17 14:22:59 -08:00
Steve Atherton 8e8c4b4489
Merge pull request #8170 from sfc-gh-sgwydir/ddsketch
Use DDSketch for sample data
2022-11-17 10:38:12 -08:00
Ankita Kejriwal 3036f2458f
Merge pull request #8803 from sfc-gh-akejriwal/commitproxies
Add a transaction option to bypass storage quota enforcement
2022-11-15 15:22:11 -08:00
sfc-gh-tclinkenbeard 82db9415fb Add const qualifier to more methods
This commit mainly targets get* and is* methods
2022-11-15 14:57:32 -08:00
sfc-gh-tclinkenbeard ec615181ce Mark several more methods const 2022-11-15 14:57:32 -08:00
sfc-gh-tclinkenbeard c03f60c618 Update rare code probe annotations 2022-11-15 13:21:25 -08:00
Ankita Kejriwal 05c914d85f Incorporate code review suggestions 2022-11-14 13:06:27 -08:00