b247f565b7
cancel durable change feed actors in DatabaseContext destructor
2023-06-27 09:22:47 -07:00
75ec56bffb
When redoing a key location request, wait until after we've checked whether we've satisfied our min rows
2023-06-20 16:02:12 -07:00
88eed268c3
added a knob for how many bytes are read from disk
2023-06-11 16:10:20 -07:00
a8ceadd917
actor cancellation still needs to unset storage
2023-06-11 14:55:05 -07:00
359e178dcd
Merge branch 'main' into feature-durable-change-feed
...
# Conflicts:
# fdbclient/ClientKnobs.cpp
# fdbserver/BlobManager.actor.cpp
# fdbserver/worker.actor.cpp
2023-06-11 13:58:35 -07:00
f69f4c73ad
addressed review comments
2023-06-11 13:54:38 -07:00
7322e21e23
fixed compiler error
2023-06-11 09:25:05 -07:00
334a868dfe
fix: respect end when reading from disk; update the starting version when leaving a hole on disk
2023-06-11 09:24:09 -07:00
d03f08f914
fix: not all mutations were being made durable
2023-06-10 18:36:02 -07:00
be8d8a8f72
fix: popping the cache was removing too many versions
2023-06-09 16:20:48 -07:00
33a7f57da5
fix: clear the cache when popping change feeds; do not insert versions into the cache that are already durable
2023-06-09 13:49:33 -07:00
197c39b552
cache change feeds using a storage engine to avoid reading them for the server on startup
2023-06-07 08:41:31 -07:00
60753b5b57
Fix a couple thread-safety issues ( #10359 )
...
* Make CodeProbeImpl::_hitCount atomic
* Structure access to TraceLog::logTraceEventMetrics so that it is written before a trace log is opened and only read from one thread after it is opened.
* Fix condition in assert
* Rename TraceLog::log to logMetrics and move initialization of trace log metrics into TraceLog::open
---------
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2023-05-26 19:36:02 +02:00
8ad7ec6fdf
Psm ss ( #9817 )
...
* Update NativeAPI getCheckpointForRange().
* Implemented checkpoint in SS.
* clean up.
* Disabled StorageServerCheckpointTest.
* Serialized checkpoint creation and deletion.
Simplified checkpoint GC, via deleting CheckpointMetaData::dir.
* Fixed PhysicalShardMove test. Where fetchCheckpoint target range is misset.
* Minor improvements on CheckpointMetaData and DataMoveMetaData.
* fmt.
* Optimized PhysicalShardMove test
cleanup.
* Refactored ShardedRocks checkpoint/restore for psm.
* Complete ShardedRocks::restore.
* dismiss operation_obsolete, and throw actor_cancelled.
* Validate checkpoint when !asKeyValues.
* fmt.
* Don't read from uninitialized physical shard.
* Resolved commments.
* cleanup.
* Added verify_checksum_before_restore for ShardedRocks.
* Added ShardedRocksDB checkpoint/restore unit test.
* Populate CheckpointMetaData::dir in RocksDB.
* Rename MovingIn as Adding.
* Added StorageServerUtils.
* Added physical shard move in SS.
* Fix on ApplyMetaData, doFetchFile error handling etc.
* Debugging incorrect shard size.
* Create/delete checkpoints only when Physical shard move is enabled.
* Added back SHARD_ENCODE_LOCATION_METADATA.
* Fixed bytesSample incorrect issue.
Essentially dedicated CheckpointRocksDBCF as key-value based checkpoint, will need to add a new format for the file-based checkpoint.
* Cleanup.
* Cleanup & compile rocksdb with 8.1 branch.
* clean up.
* clean up.
* Allowed request_maybe_delivered error type in FetchShard.
* Added FDBRocksDBVersion.h.
* Fixed stuck fetchShard.
* Don't create checkpoint on TSS.
* Upgrade to RocksDB 8.1.1
* Cleanup.
* Fixed accidently deleted db_path and name fields.
* Improved trace event.
* Removed redundants from previuos ShardedrocksDB.
* Cleanup.
* cleanup.
* cleanup.
* reanme `state`.
* Cleanup.
* Removed excessive TraceEvent.
* * Fixed shardMap race condition on different threads
* Added *Stats, logging data move rates.
* Added `DD_PHYSICAL_SHARD_MOVE_PROBABILITY` to support hybrid data move.
* Resolved comments.
* fmt.
* Use physical shard move in PhysicalShardMoveTest.
* Enforce physical-shard-move for PhysicalShardMoveTest.
* fmt
2023-05-23 11:18:35 -07:00
7ef66ab356
Add OutstandingWatches and WatchMapSize to TransactionMetrics
2023-05-22 12:07:10 -07:00
7ca13d8f9c
support blob restore in fdbrestore ( #10248 )
2023-05-19 14:45:14 -07:00
712fefd59f
Merge pull request #10213 from sfc-gh-ajbeamon/tenant-code-probes
...
Add code probes for tenant and metacluster code
2023-05-15 12:13:00 -07:00
6c16875c34
Add networkoption to disable non-TLS connections ( #9984 )
...
* Add networkoption to disable non-TLS connections
* add disable plaintext connection to fdbserver
* python doc
* Formatting
* Add tls disable plaintext connection to client api test
* review
* fix negative test
* formatting
* add TLS support to c client config tests
Adds support for TLS in the client and server separately
* add tests for disable_plaintext_connections
Test TLS and Plaintext Clusters and Clients
* Fix documentation
* Rename option to indicate it is client-only
* clearer formatting
* default to allowing plaintext connections
* add SetTLSDisablePlaintextConnection to go bindings
2023-05-13 00:14:11 +02:00
d8141c049d
Add code probes for tenant code
2023-05-10 20:44:39 -07:00
9a2365daa8
fixing bugs with tenant_mode required on external clients and changin… ( #10183 )
...
* fixing bugs with tenant_mode required on external clients and changing test to find them
* Update fdbcli/BlobKeyCommand.actor.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
---------
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2023-05-09 13:41:58 -05:00
e69d54fbc0
Block unblobbify ( #10182 )
...
* stregthening check for not merging consecutive blob ranges
* implementing expanded unblobbify and changing tests to account
2023-05-09 11:43:11 -05:00
6be0c74d5b
Adding explicit blob range mutation log to handle large number of ranges ( #10174 )
...
* Adding explicit blob range mutation log to handle large number of ranges
* fixing ide build
2023-05-09 11:30:04 -05:00
0ab691b707
Merge pull request #10002 from sfc-gh-jazhuang/readThrough
...
Fix RangeResult.readThrough misuse
2023-04-27 09:59:11 -07:00
a05e078c4a
Remove locations.size() == expectedShardCount assertion and add comments
2023-04-26 14:23:09 -07:00
7f6d5f296a
Merge commit 'e318fc260070ba6ba604930b8f259c9b655938ea' into keybackedrangemap
...
# Conflicts:
# flow/include/flow/error_definitions.h
2023-04-25 14:21:23 -07:00
6b15d67928
Merge pull request #10010 from jzhou77/main
...
Properly handle proxy_memory_limit_exceeded error for GetKeyServerLocationsRequest
2023-04-25 11:18:03 -07:00
74bb659f71
Simplify backoff calls per comment
2023-04-25 09:15:16 -07:00
c544985fe5
Add trace events and adjust backoff
...
For each success, half the backoff until less than initial backoff value, then
set the backoff to 0.
2023-04-20 15:56:06 -07:00
a83295e3bd
Add backoff to lookupTenantImpl for commit_proxy_memory_limit_exceeded error
2023-04-19 16:55:46 -07:00
3bfd353a22
Add backoff to getKeyLocation_internal as well
2023-04-19 16:45:50 -07:00
ad1c7bba74
Make commit_proxy_memory_limit_exceeded error a database level backoff
...
Since this error means the database is overloaded.
2023-04-19 15:52:16 -07:00
b49625d45b
Properly handle proxy_memory_limit_exceeded error for GetKeyServerLocationsRequest
...
Also add buggify to inject the error in simulation.
2023-04-19 09:35:09 -07:00
53ee26d758
Changed KeyBackedTypes to an actor file. Added TypedKeySelectors for Map and Set classes and getRange() keySelector methods. Added debug macro for KeyBackedTypes. Rewrote KeyBackedRangeMap using keyselectors on KeyBackedMap.
2023-04-18 22:21:19 -07:00
6b8e35fd19
Merge pull request #9964 from sfc-gh-xwang/fix/main/comments
...
fix HealthMetrics update bug and correct comments about readLoadKSecond()
2023-04-18 16:42:42 -07:00
09ab501cb7
Improve logging for test ( #9966 )
...
In clearData() used by simulation test to clean up, we re-use the same debugID for multiple transactions,
making it less clear when grepping for the debugID. This PR assigns a new debugID for each new transaction.
Some of the commented-out tracing code are already obsolete, I updated several of them, but am not sure if
the tracing should be enabled when enabling debug transaction.
Finally, use different but similar with the same prefix error messages for different call stacks.
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2023-04-18 15:25:06 -07:00
b7da2ed16c
Fix RangeResult.readThrough misuse
...
Fix `RangeResult.readThrough` misuses:
1. KeyValueStores do not need to set readThrough, as it will not be
serialized and return. Also setting it to the last key of the result
is not right, it should at least be the keyAfter of the last key;
2. Fix NativeAPI doesn't set `RangeResult.more` in a few places;
3. Avoid `tryGetRange()` setting `readThrough` when `more` is false,
which was a workaround for the above item 2;
4. `tryGetRangeFromBlob()` doesn't set `more` but set `readThrough` to
indicate it is end, which was following the same above workaround I
think. Fixed that.
5. `getRangeStream()` is going to set `more` to true and then let the
`readThrough` be it's boundary.
Also added readThrough getter/setter function to validate it's usage.
2023-04-17 21:37:51 -07:00
bdab07cbc8
fix detailed HealthMetrics update bug
...
add omitted return value
2023-04-17 12:24:14 -07:00
29161b2fda
Revert matchIndex feature
...
It is not protocol compatible, revert it to avoid deployment issue.
Will have a new PR to have the feature if moving forward.
2023-04-17 09:39:45 -07:00
370feaa3c9
refactoring and adding future compatibility to blob range metadata ( #9955 )
...
* refactoring and adding future compatibility to blob range metadata
* formatting
2023-04-13 15:06:50 -05:00
67b737b44d
add getStorageStats method to DatabaseContext
2023-04-12 09:33:05 -07:00
d903d0cc8d
checkSafeExclusion should always create new ExclusionSafetyCheckRequest
2023-03-31 16:27:26 -07:00
64b6a5d257
Allow boolean parameters to be nested inside of namespaces or classes
2023-03-30 15:09:59 -07:00
894f555e95
Test loop profiler using API Tester ( #7174 )
...
* Api Tester: Specify knobs in the toml file; Test loop profiler
* Gracefully stop the loop profiler thread
* Protect loop profiler thread by mutex
* Create loop profiler thread only if is not stopped
2023-03-30 23:00:23 +02:00
c7fcf1e8f2
handling multiple in-flight rollbacks for tss change feed stream comparison ( #9816 )
2023-03-28 09:39:38 -05:00
81c3cb8c50
Psm checkpoint ( #9636 )
...
* Update NativeAPI getCheckpointForRange().
* Implemented checkpoint in SS.
* clean up.
* Disabled StorageServerCheckpointTest.
* Serialized checkpoint creation and deletion.
Simplified checkpoint GC, via deleting CheckpointMetaData::dir.
* Fixed PhysicalShardMove test. Where fetchCheckpoint target range is misset.
* Minor improvements on CheckpointMetaData and DataMoveMetaData.
* fmt.
* Optimized PhysicalShardMove test
cleanup.
* dismiss operation_obsolete, and throw actor_cancelled.
* fmt.
* Resolved commments.
2023-03-21 09:14:10 -07:00
216d0be2cf
Add processID, networkAddress, and locality to layer status JSON for Backup Agents. ( #9736 )
...
* Add processID, networkAddress, and locality to layer status JSON for Backup Agents.
* Backup/dr agent determines network address to report in Layer Status only once, when the status updater loop begins, since it is a blocking call which connects to the cluster. And lots of code cleanup.
2023-03-17 18:07:03 -07:00
52c0dc56cc
fixing 2 bugs related to high delta file waitCommitted latency
2023-03-15 08:39:42 -05:00
c43f8b3fdc
Refactor - introduce BlobRestoreController for APIs to manage restore state ( #9616 )
2023-03-08 07:50:30 -08:00
7273723a43
Add the hotrange fdbcli command ( #9570 )
...
* Add the hotrange fdbcli command
* Remove the unnecessary state
* Add the doc about the hotrange command
2023-03-06 14:46:52 -08:00
40ed365303
fixing checkBlobSubRange to not increase version every retry
2023-03-02 09:34:55 -06:00
Evan Tschannen