Commit Graph

1777 Commits

Author SHA1 Message Date
Josh Slocum 6187811f71
Reworking getBlobGranuleRanges to also use commit proxy rpc for authz, and adding test (#9470) 2023-02-24 17:15:32 -06:00
Markus Pilman 8695fc15fc Merge remote-tracking branch 'origin/main' into features/tenant-lock2 2023-02-22 13:12:23 -07:00
Steve Atherton 23df46773d
Merge pull request #9422 from sfc-gh-satherton/client-read-options
Add transaction option definitions for read priority and read cache
2023-02-22 09:00:25 -08:00
Josh Slocum bf97c3dbce
adding java tenant blob management test and fixing bug it found (#9428) 2023-02-22 10:52:26 -06:00
Steve Atherton a21b2fe9f9 Simplified read priority option to three separate options for normal/low/high priority. 2023-02-21 22:48:38 -08:00
Steve Atherton 5969616af8 Merge commit '6de85e7cd8e9dd74a571de9e04679e669bcbb5b6' into client-read-options 2023-02-21 20:46:20 -08:00
Markus Pilman 15d8548c0e Merge remote-tracking branch 'origin/main' into features/tenant-lock2
# Conflicts:
#	fdbserver/ApplyMetadataMutation.cpp
#	fdbserver/storageserver.actor.cpp
2023-02-21 13:39:35 -07:00
Josh Slocum 958f3b531b
Plumbing blob worker mapping through commit proxy like storage server (#9401)
* Plumbing blob worker mapping through commit proxy like storage server mapping

* review comments

* formatting
2023-02-21 13:21:44 -06:00
Steve Atherton 246fd1dd4e Remove auto cache option since there is no meaningful implementation of this yet. Change places using trState in a native Transaction to set cache mode or Low/Normal/High priority to use the new transaction options instead. 2023-02-21 02:50:30 -08:00
Steve Atherton 9bf28899d4 Add transaction option definitions for read priority and read server side cache mode. 2023-02-21 00:54:35 -08:00
sfc-gh-tclinkenbeard 398079db3a Merge remote-tracking branch 'origin/main' into expose-tag-throttled-duration 2023-02-20 17:54:06 -08:00
Josh Slocum bfb3ffc509
added c and java apis for granule flush (#9412) 2023-02-20 10:28:11 -06:00
sfc-gh-tclinkenbeard 1aef6cb5f7 Merge remote-tracking branch 'origin/main' into expose-tag-throttled-duration 2023-02-17 20:41:59 -08:00
Josh Slocum 6c2fb13173
adding wait parameter to blobbify api (#9360)
* adding wait parameter to blobbify api

* formatting

* fixing comment style

* fixing bug and adding debugging

* adding blob ranges unit test

* testing both blobbify cases in cancel

* formatting

* switch to explicit blocking api instead of boolean flag

* remove comments

* format
2023-02-17 12:20:53 -06:00
Junhyun Shim d9c126a2d9
Introduce WipedString for Arena block holding AuthZ tokens (#9381)
* Enable secure allocation mode in Arena

This mode allows zeroing out blocks holding sensitive data after use

* Introduce WipedString to all token-holding memory

Also introduce a option flag "sensitive"

* Make pointer equivalency a hard requirement for non-ASAN builds

So that we can detect when Arena/malloc/memory-wipe behavior changes
2023-02-16 10:44:32 +01:00
Markus Pilman 3017e15448 Merge remote-tracking branch 'origin/main' into features/tenant-lock2
# Conflicts:
#	fdbclient/include/fdbclient/TenantManagement.actor.h
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
2023-02-13 11:43:07 +01:00
Josh Slocum 1e5bac6238
fixing fault injection stalling change feed fetch (#9316) 2023-02-08 15:49:56 -06:00
Ankita Kejriwal b9630e57d9
Merge pull request #9291 from sfc-gh-akejriwal/debug2
Update storage metrics functions to use the version at which the tenant was read
2023-02-08 11:33:33 -08:00
sfc-gh-tclinkenbeard 09ad864eb5 Merge remote-tracking branch 'origin/main' into expose-tag-throttled-duration 2023-02-08 11:25:14 -08:00
Junhyun Shim d32d3dd085 Make token option NOT survive transaction hard resets 2023-02-08 18:12:22 +01:00
Junhyun Shim be225acd2a Merge remote-tracking branch 'origin/main' into authz-tenant-name-to-tenant-id 2023-02-06 23:13:43 +01:00
Junhyun Shim 6993e391f2 Extend authz test to include with/without GRV caching and forced MVC 2023-02-06 13:18:10 +01:00
Markus Pilman 1fb3c33b3f Merge remote-tracking branch 'origin/main' into features/tenant-lock2
# Conflicts:
#	fdbserver/CommitProxyServer.actor.cpp
2023-02-06 11:58:49 +01:00
Ankita Kejriwal 1a240a0eab Merge branch 'main' of github.com:apple/foundationdb into debug2 2023-02-03 12:26:19 -08:00
Ankita Kejriwal 913bc9114d Handle errors that can't be sent in reply; fix formatting 2023-02-02 19:05:05 -08:00
Hui Liu 774446d3a0 Support pagination for StorageServer splitMetrics API 2023-02-02 14:21:38 -08:00
Vaidas Gasiunas f8b1da8bc6
An option to initialize client tracing in setupNetwork (#9209)
* client_config_tester: use a generic mechanism to set specific network options

* trace_initialize_on_setup option to initialize client traces on network setup without local IP address

* trace_initialize_on_setup: Addressing review comments

* Restore correct formatting

* trace_initialize_on_setup: Update go bindings

* Include PID for identification into trace file names by default

* Use the same naming pattern for trace files in all configurations

* Empty commit
2023-02-02 10:00:51 +01:00
Ankita Kejriwal 9529f360ee Update storage metrics functions to use the version at which tenant was read 2023-02-01 19:44:31 -08:00
A.J. Beamon b8592c4e09
Merge pull request #9219 from sfc-gh-jfu/tenant-object-binding-tests
Add tenant getId to java/python binding tester
2023-01-31 16:22:38 -08:00
Hui Liu 6fbabab6aa Fix misc blob restore error messages 2023-01-30 13:01:27 -08:00
Markus Pilman 64c11c1f34 Implemented tenant lock and basic test 2023-01-26 15:47:39 +01:00
Jon Fu 6493da877a add method to return idfuture 2023-01-25 12:45:35 -08:00
Sreenath Bodagala fe4a839600
- Do not add fdbserver processes to the client list. (#9225)
Note: Server processes started getting reported as clients since 7.1.0
(not sure if this change in behavior was intentional or not), and this
breaks the operator upgrade logic.
2023-01-25 14:39:32 -05:00
Jingyu Zhou 27e1b76b68
Merge pull request #9226 from xis19/main-bench
Extract boost/asio.hpp out from flow/network.h
2023-01-25 09:14:52 -08:00
Xiaoge Su e2cc90c18f Extract GetServerDBInfoRequest, ClientDBInfo and OpenDatabaseRequest
Wall time: 909.3
clang time: 12747.718300
2023-01-24 14:40:11 -08:00
Evan Tschannen a1bf0829a2
Encrypt change feeds (#9164)
* encrypted change feeds, still missing re-encryption of mutations which have been modified

* removed inverted filtering and reduced the number of comparisons for calculating clear intersection

* fix merge conflict

* fixed another merge conflict

* encrypt mutations which have been modified

* renamed encrypted to encryptedMutation

* fix formatting

* format nativeAPI

* format code

* more formatting fixes
2023-01-24 14:20:54 -08:00
A.J. Beamon adb8d3ec7a Fix a memory issue when getting key range locations that manifests in particular when getting range split points 2023-01-23 11:01:37 -08:00
A.J. Beamon b10d1f227b Remove tenant name from the TenantInfo object 2023-01-20 14:04:43 -08:00
A.J. Beamon d47a2ab60f
Merge pull request #9128 from sfc-gh-ajbeamon/transactions-use-tenant-object
Transactions take a tenant object rather than a tenant name
2023-01-20 12:06:36 -08:00
He Liu ec6716ff2e
Restore kv (#9030)
* Allow multiple keyranges in CheckpointRequest.
Include DataMove ID in CheckpointMetaData.

* Use UID dataMoveId instead of Optional<UID>.

* Implemented ShardedRocks::checkpoint().

* Implementing createCheckpoint().

* Attempted to change getCheckpointMetaData*() for a single keyrange.

* Added getCheckpointMetaDataForRange.

* Minor fixes for NativeAPI.actor.cpp.

* Replace UID CheckpointMetaData::ssId with std::vector<UID>
CheckpointMetaData::src;

* Implemented getCheckpointMetaData() and completed checkpoint creation
and fetch in test.

* Refactoring CheckpointRequest and CheckpointMetaData

rename `dataMoveId` as `actionId` and make it Optional.

* Fixed ctor of CheckpointMetaData.

* Implemented ShardedRocksDB::restore().

* Tested checkpoint restore, and added range check for restore, so that
the target ranges can be a subset of the checkpoint ranges.

* Added test to partially restore a checkpoint.

* Refactor: added checkpointRestore().

* Sort ranges for comparison.

* Cleanups.

* Check restore ranges are empty; Add ranges in main thread.

* Resolved comments.

* Fixed GetCheckpointMetaData range check issue.

* Refactor CheckpointReader for CF checkpoint.

* Added CheckpointAsKeyValues as a parameter for newCheckpointReader.

* PhysicalShard::restoreKvs().

* Added `ranges` in fetchCheckpoint.

* Added RocksDBCheckpointKeyValues::ranges.

* Added ICheckpointIterator and implemented for RocksDBCheckpointReader.

* Refactored OpenAction for CheckpointReader, handled failure cases.

* Use RocksDBCheckpointIterator::end() in readRange.

* Set CheckpointReader timout and other Rocks read options.

* Implementing fetchCheckpointRange().

* Added more CheckpointReader tests.

* Cleanup.

* More cleanup.

* Added fetchCheckpointRanges test.

* Implemented restore of kv-based checkpoint.

* Improved CheckpointRestore test for kv, non-kv, as well as partial
restore.

* Added test of merge.

* Fixed merge test.

* Cleanup.

* Resolved comments.

Co-authored-by: He Liu <heliu@apple.com>
2023-01-20 09:58:34 -08:00
A.J. Beamon 258198ca20 Merge branch 'main' into transactions-use-tenant-object
# Conflicts:
#	fdbclient/include/fdbclient/DatabaseContext.h
#	fdbserver/workloads/BulkLoadWithTenants.actor.cpp
2023-01-20 09:06:39 -08:00
Hui Liu 36e8e5a3bb
Merge pull request #9176 from sfc-gh-huliu/restoreversion
Restore to a previous version
2023-01-19 17:37:23 -08:00
Hui Liu c85b984c3a blobrestore to previous point of time 2023-01-19 14:45:01 -08:00
Nim Wijetunga 82c92abca2
Change Noisy Sev30s to Sev20s (#9180)
change noisy sev30 traces to sev20
2023-01-19 14:06:34 -08:00
A.J. Beamon e1d48d28bf Remove tenant name from the TenantInfo object 2023-01-13 08:58:15 -08:00
Josh Slocum 0fc5c6c1e2
tenant entry was removed, changed to fixme (#9132) 2023-01-12 18:39:22 -06:00
Josh Slocum b6450f9eaa
More ss cf perf fixes main (#9109)
* changing future version logic for change feed fetch

* Optimizing change feed data structures and accesses

* coalescing change feed request ranges for merge cursor if they're to the same team

* fixing over-read of memory mutations for change feeds

* feed filter mutations common prefix cpu optimiation

* fix formatting
2023-01-12 16:15:05 -06:00
Xiaoxi Wang 07b37245dd merge upstream/main; solve merge conflicts 2023-01-12 12:29:03 -08:00
A.J. Beamon f999623bb1 Add a tenant lookup interface and use it when starting transactions 2023-01-06 15:51:12 -08:00
sfc-gh-tclinkenbeard 0f14647bbf Merge remote-tracking branch 'origin/main' into expose-tag-throttled-duration 2023-01-05 08:10:49 -08:00