6e4e6ab2f4
Revert "Revert "Refactor GetEncryptCipherKeys ( #9600 )"" ( #9903 )
...
* Revert "Revert "Refactor GetEncryptCipherKeys (#9600 )" (#9708 )"
2023-04-05 10:03:48 -07:00
dbcab0b1bd
Revert "Refactor GetEncryptCipherKeys ( #9600 )" ( #9708 )
...
This reverts commit 2702665e35
2023-03-15 12:10:08 -07:00
2702665e35
Refactor GetEncryptCipherKeys ( #9600 )
...
* inital commit
* address pr comments
2023-03-08 17:05:03 -08:00
99b23ac04d
EaR: Configurable encryption support for Tlog mutations ( #9394 )
...
* EaR: Configurable encryption support for TLog mutations
Description
diff-1 : Address review comments
Major changes includes:
1. Update the code involved in ensuring Tlog mutation encryption to be
compliant with "configurable encryption" feature.
2. Update ENABLE_CONFIGURABLE_ENCRYPTION flag to be 'true' by default
and BUGGIFY it.
Testing
devRunCorrectness - 100K
2023-02-16 19:01:59 -08:00
fe18c87ac6
EaR: commit proxy fetch additional cipher keys post-resolution ( #9308 )
...
Commit proxy needs to fetch additional cipher keys post-resolution, since tenant ids for raw access requests and cross-tenant clear ranges are calculated after resolution.
2023-02-14 13:05:51 -08:00
982ce105e5
Reduce logging level for verbose events
...
From one of nightly failure due to too many log lines, these are top 3:
60100 FastRestoreLoaderDispatchRequests
79655 FastRestoreGetVersionSize
93888 FastRestoreSplitMutation
2023-02-01 09:51:16 -08:00
05a8a90830
Snapshot Backup Tenant Deletion Support ( #9145 )
...
Tenant deletion support for snapshot backups
2023-01-26 17:46:14 -08:00
86d6106dc1
format source code after switch to clang 15
2022-12-08 17:26:45 +00:00
80e0810bcc
Fix stack overflows
2022-11-10 12:10:15 -08:00
bb0236433c
Merge pull request #8540 from sfc-gh-xwang/feature/main/storageMetrics
...
Make MockStorageServer serve StorageMetrics related request
2022-10-25 17:29:21 -07:00
a8f821e152
Fix stack overflows
...
The loop is transformed by actor compiler into recursions that may cause stack
overflows. Thus, I added yield() to unwind stack and refactor the parsing code
so that the subsequent files are blocked until previous ones have finished.
2022-10-24 11:13:11 -07:00
3c67b7df39
extract serveStorageMetricsRequests template function
2022-10-24 09:58:41 -07:00
232e23c87e
Snapshot Backup Encryption ( #8095 )
...
* inital commit
* add encryption support
* refactor passing encryption keys
* disable encryption codepath
* pass enable encryption cli arg for fdbbackup
* change encryption key passing
* only enable encryption for backup if tenant mode is required
* revert server knobs
* ignore json trace files
* --amend
* integrate ekp
* add comments
* comments
* comments
* revert knobs
* fix bug
* modify encrypted backup logic
* use common prefix
* check boundaries between begin and end key
* return system encrypt domain
* modify
* add tenant entry cache to backup
* slight revert
* format
* Trigger Build
* format
* use tenant name to fetch keys
* Trigger Build
* Trigger Build
* merge
* fix getting ekp
* fix bug
* address pr comments
* Trigger Build
* Trigger Build
* remove faulty check
2022-09-29 14:45:47 -07:00
fda0d7223d
Update backup to include system key ranges needed for tenants. Run simulated backup tests with tenants.
2022-09-22 10:00:13 -07:00
eadb769cfa
Encrypt Backup Mutation Log ( #8159 )
...
* encrypt backup mutation log
* format
* address pr comments
* format
* fix bug
* revert knobs
* address pr comments
2022-09-20 15:43:39 -07:00
509067016f
Make encrypted mutation serialize as normal mutations
2022-08-09 19:16:26 -07:00
364644673f
Support TLog encryption in commit proxy ( #6942 )
...
This PR add support for TLog encryption through commit proxy. The encryption is done on per-mutation basis. As CP writes mutations to TLog, it inserts encryption header alongside encrypted mutations. Storage server (and other consumers of TLog such as storage cache and backup worker) decrypts the mutations as they peek TLog.
2022-06-29 14:21:05 -07:00
1e17f32469
Move StorageMetrics unit tests into cpp file
2022-05-10 13:21:23 -07:00
0a332ee1c1
Add proxy option to backup and restore params.
2022-03-28 17:10:49 -07:00
a71099471b
Update copyright header dates
2022-03-21 13:36:23 -07:00
8dcac2f76d
Fix typos
2022-03-13 10:02:11 -03:00
250a88e682
Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement.
2022-02-24 12:25:52 -08:00
59503a397e
Fix duplicate trace field.
2022-01-05 11:30:21 -08:00
abf73047ca
Enforce std:: specifier rather than using namespace
2021-09-16 19:40:28 -07:00
3922fa8cdd
Fix duplicate TraceEvent detail name.
2021-08-10 17:41:31 -07:00
eb5c8eb78d
Added Label field for keys and ranges in MutationTracking.
2021-08-10 17:32:32 -07:00
faa4154a56
MutationTracking now uses a vector of keys to track. Removed "Mutation" detail from DEBUG_MUTATION() events because they are duplicates of the fields already logged in the returned MutationTracking event, which are now renamed and combined into "Mutation". Removed more toString() calls in TraceEvent detail values.
2021-08-09 23:30:45 -07:00
df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
79cec09255
Apply clang-tidy's performance-inefficient-vector-operation fix
...
I ran this command in my build directory after compiling with
OPEN_FOR_IDE. It took a few small tweaks to get it to compile, which is
outside the scope of this commit.
$ python run-clang-tidy.py -j $(nproc) -checks='-*,performance-inefficient-vector-operation' -fix
2021-03-04 03:58:25 +00:00
0ef44739ea
Fix OPEN_FOR_IDE build in preparation for using clang-tidy
2021-01-26 02:04:11 +00:00
c77d9e4abe
merge conflicts
2020-12-02 21:53:19 +00:00
4669f837fa
Add uses of makeReference
2020-11-07 22:10:18 -08:00
41843f07e6
add simulator support for different process versions and ProtocolVersion test
2020-10-12 18:19:31 +00:00
a3c861b59d
Merge pull request #3777 from apple/release-6.3
...
Merge Release 6.3 into master
2020-09-14 10:32:50 -07:00
2febbe74ce
FastRestoreApplier:Fix conflict range inverted due to invalid memory access
...
Rerpot error on loader and applier if not error_code_operation_cancelled error ever happens
2020-09-09 14:40:21 -07:00
e69ee87f33
Refactor blob failure to simulateBlobFailure function
2020-09-09 10:54:21 -07:00
5880bd9fa7
BackupContainer:Fix getSnapshotFileKeyRange_impl on blob http error
2020-09-09 08:47:43 -07:00
83d1350d8d
FastRestore:Handle retriable blob error
2020-09-09 07:29:17 -07:00
6d58064964
Buggify _parsePartitionedLogFileOnLoader with blob http error
2020-09-09 06:46:08 -07:00
f3986d201d
Buggify backup decodeRangeFileBlock with blob connection error
2020-09-09 06:40:41 -07:00
12edadd059
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# fdbclient/Knobs.cpp
# fdbclient/MasterProxyInterface.h
# fdbrpc/simulator.h
# fdbserver/MasterProxyServer.actor.cpp
# tests/fast/CycleAndLock.txt
# tests/fast/TxnStateStoreCycleTest.txt
# tests/fast/VersionStamp.txt
# tests/slow/ParallelRestoreOldBackupApiCorrectnessAtomicRestore.txt
# tests/slow/ParallelRestoreOldBackupCorrectnessCycle.txt
# versions.target
2020-08-31 19:33:34 -07:00
7a29a3157f
FastRestore:Fix various bugs discovered by enhanced simulation
...
1. sendMutation request can be dispatched when the version batch has finished and its data has been deleted;
2. Request scheduler on loader may get into infinite loop because FASTRESTORE_SCHED_LOAD_REQ_BATCHSIZE knob can be set to 0 in simulation
2020-08-26 15:53:18 -07:00
f1bd2a18ed
Resovle review comments: No functional change
2020-08-26 10:30:31 -07:00
bd7c07436b
FastRestore:Add batchIndex to RestoreAsset for better performance tracking
2020-08-25 09:34:18 -07:00
778daf20c0
FastRestore:Fix incorrect assert
2020-08-24 19:59:56 -07:00
996ba2374c
FastREstore:Fix:Incorrect condition in printing out FastRestoreLoaderSendMutationToApplierDoneTooLate
2020-08-24 17:27:40 -07:00
e94261efe5
FastRestore:LoaderScheduler:Add validation on too old requests
2020-08-24 16:32:16 -07:00
6e3e36c8fc
FastRestore:RequestScheduler:Minor code style improvement
2020-08-24 10:45:46 -07:00
d9ea14ea6c
FastRestore:fix:loader can receive reply from vb that has been processed and deleted
2020-08-19 10:39:49 -07:00
88b94a3e59
FastRestore:Fix:Delayed request can be skipped from processing
...
When a request of sending mutations to appliers is delayed and the request for future version batches have been processed,
loader will increase its counter. The previous implementation will use the counter to decide if a request is too old and
have been processed. If the counter is larger than the batch index of the request, the request will be ignored although
it may not have been processed.
This causes correctness test failure for fast restore.
2020-08-18 23:48:06 -07:00
Nim Wijetunga