76d514bf56
Update shared rocksdb knobs. #11936 ( #11938 )
2025-02-11 15:41:02 -08:00
382cd9c744
Close and delete DB when checkpoint reader gets an error. #11925
2025-02-05 16:30:11 -08:00
fff5439ef2
Close DB properly in unit tests. #11915
2025-01-30 08:49:55 -08:00
786e2a6093
Add custom compaction policy based on number of range deletions in file
...
* compaction policy
* fix build error
2025-01-17 14:14:14 -08:00
9ca82b2fda
Add knob for direct IO
2025-01-17 14:11:35 -08:00
f4c3565aff
Rocksdb manual flush code changes ( #11849 )
2025-01-17 12:44:44 -08:00
d735250798
Fix the scope of sharded rocks checkpoint determinism flag ( #11893 )
2025-01-16 22:44:48 -08:00
cf7c8f41b2
BulkLoad Job Framework and Co-Testing BulkLoad and BulkDump ( #11865 )
...
* add bulkload job framework and fix bugs
* add BulkLoadChecksum, fix CI issue
* nits
* nits
* address comments
* mitigate perpetual wiggle to make sure DD can select a valid team to inject data
* fix submitBulkDumpJob and submitBulkLoadJob
* change remoteRoot to jobRoot
* add comments
2025-01-14 11:28:42 -08:00
122cb96b82
Make sharded rocks deterministic in simulation (phase 1) ( #11841 )
2024-12-18 16:05:04 -08:00
df0fc7d3b8
Fix issues with clang 19 ( #11834 )
...
* Fix issues with clang 19
* Fix format
* Ignore --undefined-version for gcc
2024-12-13 14:20:45 -08:00
1f7a9f4768
Manual flush if the rocksdb flush does not happen within a time interval. ( #11791 )
2024-11-19 11:57:32 -08:00
5be60d50d3
Histogram sample rate updated to 1. ( #11793 )
2024-11-19 11:53:16 -08:00
ca4ab1eca9
Fix traceTooManyEvents and externalTimeouts in BulkLoad test ( #11769 )
2024-11-11 11:05:43 -08:00
7290369aac
Use a single iterator pool for all physical shards. ( #11699 )
...
* Rewrite iterator pool.
* simulation fix
2024-10-15 17:28:54 -07:00
83dd1f202e
Fix block cache size error and improve logging. ( #11681 )
2024-09-24 13:08:31 -07:00
289d02899f
Add knobs for caching index blocks. ( #11650 )
2024-09-11 22:58:26 -07:00
5ee0db13e6
Fix external timeout with ShardedRocksDB and re-enable ShardedRocksDB in simulation tests ( #11638 )
...
* speedup sharded rocksdb in simulation
* re-enable shardedrocksdb and disable physical shard move
2024-09-08 10:57:55 -07:00
74990e44bd
Bulk Loading Framework ( #11369 )
2024-07-23 14:57:28 -07:00
1791d07be1
Improvements ( #11363 )
2024-05-15 09:04:50 -07:00
fa15b9df49
RocksDB memtable max range deletions knob update. ( #11386 )
2024-05-13 15:54:43 -07:00
9789c7f4ff
async io ( #11325 )
2024-04-22 14:20:11 -07:00
ed7a275231
Rocksdb caching knob options. ( #11282 )
2024-04-17 10:09:14 -07:00
be3dcbde62
Sharded RocksDB knob changes. ( #11291 )
2024-04-16 11:15:08 -07:00
de5cc85c28
block cache usage ( #11251 )
2024-03-14 16:15:38 -07:00
19e3f3e2dd
Disable compaction compaction for newly added shard. ( #11238 )
...
* Disable compaction compaction for newly added shard.
2024-03-07 14:41:53 -08:00
7db980e185
Rocksdb in-memory data structures protection checksums. ( #11206 )
2024-02-19 16:46:12 -08:00
13efe686f1
Compaction knobs & metrics ( #11187 )
2024-02-09 11:34:13 -08:00
73e40985b1
Throttle commit on rocksdb flush state and some rocksdb knob changes. ( #11177 )
2024-02-08 10:14:04 -08:00
a88114c222
Fix typos
2024-02-07 01:16:00 +02:00
02182d556a
add omitted checksum for rocksdb ( #11169 )
2024-02-01 18:33:14 -08:00
55d88c17c9
Rename knobs and clean up some trace events. ( #11135 )
2024-01-17 16:26:26 -08:00
af106e5bda
Update rocksdb knobs. ( #11069 )
2023-11-16 13:53:12 -08:00
e7aa0333a9
Update RocksDB options.
2023-11-08 14:09:31 -08:00
220ad87cc4
Rocksdb new options configuration ( #11048 )
2023-11-03 13:45:03 -07:00
33a29ddd85
Upgrade RocksDB version and disable CF range deletion optimization. ( #11045 )
...
* Upgrade RocksDB version and disable CF range deletion optimization.wq
* Disable iterator.
2023-11-02 17:25:11 -07:00
1896e5cd46
Fix an unitialized variable
...
Valgrind complains this for RecentRocksDBBackgroundWorkStats event.
2023-10-09 15:13:02 -07:00
b20dcf23a9
Support periodic compaction for sharded rocksdb. ( #10815 )
2023-08-25 15:38:01 -07:00
c63ee571e5
Export file metrics and add knob for file size multiplier. ( #10785 )
2023-08-16 11:27:33 -07:00
5868173a3e
nits
2023-08-10 16:51:28 -05:00
81db1da5a9
address comments
2023-08-10 16:51:28 -05:00
5598f4f28f
trace shardedrocks manifest size
2023-08-10 16:51:28 -05:00
ae7fe844d2
Merge pull request #10689 from xis19/rocksdbflag
...
Add a flag for enabling/disabling RocksDB storage engine in simulations
2023-08-01 10:50:09 -07:00
d0742c79ac
Improving visibility to debug sharded rocksdb ( #10694 )
...
* logging storage commit stats
* add rocks flush and compaction listener
* remove used field in FlushStats and fix CI error
* reduce LOGGING_ROCKSDB_BG_WORK_PROBABILITY
* merge rocks event listeners
* avoid using mutex/spinloop in rocksdb event listener
* code clean
* fix OnCompactionBegin and OnFlushBegin
* add logReason to RecentRocksDBBackgroundWorkStats
* add error listener back
2023-07-31 14:45:26 -07:00
0036bab8a4
Rename SSD_ROCKSDB_EXPERIMENTAL to WITH_ROCKSDB
2023-07-27 14:04:10 -07:00
24968674ce
Merge pull request #10643 from yao-xiao-github/write-buffer
...
Add knob.
2023-07-21 08:31:14 -07:00
e1aa6d3dd1
Log counters. ( #10626 )
2023-07-20 13:01:40 -07:00
391c2041d1
Add logs for write stalls. ( #10640 )
2023-07-20 13:00:05 -07:00
ff2f752bf9
Add knob.
2023-07-18 16:54:06 -07:00
ab72951034
Add knob for manifest file size and log rocksdb status. ( #10567 )
2023-07-05 10:40:43 -07:00
6337125712
Several minor improvements for ShardedRocksDB ( #10520 )
...
* Terminate DD if SHARD_ENCODE_LOCATION_METADATA is not enabled and storage_engine_type is ShardedRocksDB.
* Fixed Error in non-main thread.
* Minor improvements.
2023-06-24 16:07:14 -07:00
Yao Xiao