wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
21,750 Commits 35 Branches 335 Tags 299 MiB
Commit Graph

103 Commits

Author SHA1 Message Date
Josh Slocum ddbf32d35e
adding knob and fixing blobrange cli check bug with too many granules (#7793) 2022-08-05 11:19:17 -05:00
A.J. Beamon ff23d5994e
Merge pull request #7729 from sfc-gh-ajbeamon/feature-metacluster 
Metacluster
 2022-08-04 15:29:44 -07:00
A.J. Beamon a2e66c9695 Don't buggify the assignment availability timeout too low or it will become impossible to make an assignment 2022-08-03 19:39:20 -07:00
He Liu fa418fd784
Change SHARD_ENCODE_LOCATION_METADATA to a server knob. (#7770) 
Co-authored-by: He Liu <heliu@apple.com>
 2022-08-03 13:51:40 -07:00
A.J. Beamon a69164d160 Merge branch 'main' into feature-metacluster 2022-07-29 11:53:06 -07:00
Josh Slocum cfc13e7018
Adding more blobrange cli commands and a couple other tweaks (#7727) 2022-07-29 08:20:45 -07:00
A.J. Beamon 7f53b340f5 Merge branch 'main' into feature-metacluster 
# Conflicts:
#	bindings/python/tests/fdbcli_tests.py
#	fdbcli/TenantCommands.actor.cpp
#	fdbclient/ClientKnobs.cpp
#	fdbclient/include/fdbclient/ClientKnobs.h
#	fdbclient/include/fdbclient/Tenant.h
#	fdbserver/BlobManager.actor.cpp
#	fdbserver/BlobWorker.actor.cpp
#	fdbserver/TenantCache.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
#	flow/include/flow/error_definitions.h
 2022-07-28 11:43:21 -07:00
A.J. Beamon 7c6b3fb0b8 Merge branch 'main' into feature-metacluster 2022-07-27 08:55:10 -07:00
A.J. Beamon d39c0b773a Add a limit to the number of tenants that can be created in a cluster 2022-07-27 08:21:03 -07:00
Renxuan Wang dc9599f2e9
Get network interfaces ready for https proxy. (#7556) 
* Move HTTP from fdbclient/ to fdbrpc/.

* Move md5 and libb64 to contrib/.

* Get network interfaces ready for https proxy.

* Rebase
 2022-07-25 17:08:32 -07:00
A.J. Beamon 077003c4f5 Cleanup tenant tombstones periodically 2022-07-24 18:24:19 -07:00
A.J. Beamon 41c1bef3bf Merge branch 'feature-tenant-groups' into feature-metacluster 2022-07-22 14:26:02 -07:00
Lukas Joswiak 703aa1d279 Mess with timeout values 2022-07-22 10:37:29 -07:00
Lukas Joswiak 40d403ed5f Reduce global configuration system key reads from proxy 
Clients now poll the proxy for the latest global config for a specific
version. The proxy now periodically requests the latest global
configuration data and stores it in memory, enabling it to respond
immediately to clients with the appropriate version.
 2022-07-22 10:37:29 -07:00
A.J. Beamon 279296c29f Merge branch 'tenant-metadata-change' into feature-metacluster 
# Conflicts:
#	fdbclient/SystemData.cpp
#	fdbclient/Tenant.cpp
#	fdbclient/include/fdbclient/SystemData.h
#	fdbclient/include/fdbclient/Tenant.h
#	fdbclient/include/fdbclient/TenantManagement.actor.h
#	fdbserver/TenantCache.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
 2022-07-20 09:18:27 -07:00
A.J. Beamon 537ceff8ac Remove the ability to configure a tenant subspace. Rename the prefixes used for tenant metadata. 2022-07-19 14:32:05 -07:00
A.J. Beamon 70bc67ce34 Update tenant assignment to check data cluster availability. Cleanup errors when performing an invalid operation on a non-metacluster and remove redundant function. 2022-07-18 16:15:33 -07:00
A.J. Beamon cb499fa5db Merge branch 'main' into feature-metacluster 2022-07-13 15:28:34 -07:00
He Liu bc5bfaffda
Shard based move (#6981) 
* Shard based move.

* Clean up.

* Clear results on retry in getInitialDataDistribution.

* Remove assertion on SHARD_ENCODE_LOCATION_METADATA for compatibility.

* Resolved comments.

Co-authored-by: He Liu <heliu@apple.com>
 2022-07-07 20:49:16 -07:00
A.J. Beamon e1a93988ef Merge branch 'main' into feature-metacluster 2022-06-28 14:58:07 -07:00
A.J. Beamon 4b69723a2f Cleanup metadata associated with a cluster when force removing it. Move some metacluster metadata into the normal key-space. Cleanup some todos. 2022-06-24 14:55:57 -07:00
A.J. Beamon 96fad987eb Add new command to get total metacluster capacity. Fix bug where removing a data cluster could make it impossible to create new tenants on it. Add knobs for max data clusters and max tenants. 2022-06-17 10:42:03 -07:00
Xiaoge Su e493f1c3cd fixup! Add a retry mechanism in changeQuorumChecker and changeQuorum 
This is to fix an issue when recovery and change coordinator key happens
together. The issue will occur when:

1. Recovery starts
2. Coordinator key change transaction started
3. During the recovery the coordinator key is read from cluster file and
   stored in the storage server
4. The cluster controller received `ChangeCoordiatorsRequest`, and
   updated the cluster name with the new value.

at this stage, the value related to coordinator key in storage server and
the worker is inconsistent.

5. changeQuorumChecker is called, which will verify such consistency.
   Since they are different, the call is returning failure and the
   caller, which could be a TEST_CASE, fails.

This is a rare race issue, and it is also noticed that when the
recovery/coordinator key change process is done, the database is in a
proper state which allows changeQuorumChecker behave properly. In this
case, a retry mechanism should be sufficiently fix corresponding test
failures.
 2022-06-14 10:43:13 -07:00
Zhanwei Wang da346d718f Fix compatibility issue of s3 backup 
1. For v4 signature, http request path should be encoded, currently only parameters are encoded.
2. For v2 signature, http request date header should be in form of either the http date header or x-amz-date, currently in form of ISO 8601
3. Empty response header value cause http_bad_response error.
4. Skip verifying MD5 for range get request and if http response code 206 is returned.
 2022-05-25 09:48:54 -05:00
Josh Slocum 674e6a8fdc Merge branch 'main' into min_shard_bytes_main 2022-04-29 16:00:27 -05:00
A.J. Beamon 79063d5fc4
Better accounting for tenant prefixes when working with key size limits (#6973) 
* Consider tenant prefix sizes when doing key size limit checks

* Fix formatting
 2022-04-27 14:38:59 -07:00
Josh Slocum ca68fefca2 Decreasing MIN_SHARD_BYTES knob 2022-04-20 18:46:11 -05:00
Dan Lambright 5bdc525353
Merge branch 'main' into vv 2022-04-08 13:16:04 -04:00
Dan Lambright 60c55e0785 Merge remote-tracking branch 'origin/version-vector-prototype' into vv 2022-04-05 11:17:39 -04:00
Renxuan Wang 465ff712b6
Move Hostname to its own files. (#6759) 
* Change DNS cache to use std::map.

Revert commit 90c259d84e, because if we use unordered_map, toString() can be inconsistent.

* Move ClientKnob::COORDINATOR_HOSTNAME_RESOLVE_DELAY to FlowKnob::HOSTNAME_RESOLVE_DELAY.

* Move Hostname to its own files.

Also, add resolve-related variables and functions in Hostname.
 2022-04-04 19:04:51 -07:00
Josh Slocum 268caa5ac8 fixing shard size knobs outside of simulation 2022-04-04 11:38:18 -05:00
Jingyu Zhou 4fd414a8ed Merge branch 'main' into vv 
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
 2022-03-31 09:38:36 -07:00
Jingyu Zhou cfcf0f152c Merge branch 'main-4a085fc84' into vv 
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
	fdbserver/ClusterRecovery.actor.cpp
	fdbserver/MasterInterface.h
	fdbserver/masterserver.actor.cpp
	flow/error_definitions.h
 2022-03-30 22:28:06 -07:00
Jingyu Zhou b34f4052cd Merge branch 'main-f28dfc12b' into vv 
Fix Conflicts:
	fdbclient/MultiVersionTransaction.actor.cpp
	fdbclient/NativeAPI.actor.cpp
	fdbclient/NativeAPI.actor.h
	fdbserver/storageserver.actor.cpp
 2022-03-30 21:01:25 -07:00
Jingyu Zhou 00b57d4cce Merge branch 'main-67eba5ec7' into vv 
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbclient/StorageServerInterface.h
	fdbserver/CommitProxyServer.actor.cpp
	fdbserver/storageserver.actor.cpp
 2022-03-30 20:05:55 -07:00
Jingyu Zhou e9659b5dd4 Merge branch 'master-PR-6500' into vv 
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbserver/masterserver.actor.cpp
 2022-03-30 14:53:49 -07:00
Evan Tschannen 90983990df
Merge pull request #6680 from sfc-gh-jslocum/bg_client_bindings_improvements 
granule parallelism, metrics, and other improvements to bg client bindings
 2022-03-25 13:44:43 -07:00
Renxuan Wang 2a59c5fd4e
Workers should monitor coordinators in submitCandidacy(). (#6655) 
* Workers should monitor coordinators in submitCandidacy().

* Change re-resolve delay to a knob.
 2022-03-24 19:20:42 -07:00
Josh Slocum b92c6ea92c granule parallelism, metrics, and other improvements to bg client bindings 2022-03-24 13:44:09 -05:00
Josh Slocum f27475e2f4 Merge branch 'main' into blob_integration 2022-03-22 11:41:58 -05:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Josh Slocum 37e7c80f26 Merge branch 'main' into blob_integration 2022-03-17 18:45:42 -05:00
A.J. Beamon 1e1098ca9a Don't check whether tenants are enabled when watch is called. Support unknown_tenant error. 2022-03-15 09:23:30 -07:00
A.J. Beamon ecccfd0868 Add cache invalidation to tenant cache. Send tenant ID along with tenant name in requests to validate that the tenant hasn't changed. Fix a few bugs. 2022-03-15 09:23:30 -07:00
A.J. Beamon 2a21126028 Don't apply read prefixes on the client. Cache tenant data locally. 2022-03-15 09:23:30 -07:00
Sreenath Bodagala bb1a8bd7bd - Set default value of SEND_ENTIRE_VERSION_VECTOR to false. 2022-03-09 19:21:29 +00:00
Josh Slocum e71b3533f9 Merge branch 'main' into blob_integration 2022-03-09 08:59:56 -06:00
Sreenath Bodagala 91b75ae014 - Intoduce a new knob "CLIENT_KNOBS::SEND_ENTIRE_VERSION_VECTOR" 
- Send the entire version vector on the read version code path
(sequencer -> grv proxy and grv proxy -> client)
 2022-03-08 21:21:10 +00:00
Josh Slocum 665e182bcc TSS Metrics improvements 2022-03-07 11:20:24 -06:00
A.J. Beamon cdebda35ab
Merge pull request #5725 from sfc-gh-jfu/jfu-grv-cache 
Add transaction option for clients to use cached read versions
 2022-03-04 09:17:27 -08:00