67e783acf8
Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3
...
# Conflicts:
# cmake/CompileBoost.cmake
# cmake/FDBComponents.cmake
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/simulator.h
# fdbserver/KeyValueStoreSQLite.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/storageserver.actor.cpp
# flow/Knobs.h
# flow/network.h
2021-02-08 09:20:28 -08:00
8547abbbe4
Bug fix: In simulation, KVStoreSQLite can outlive its process during a simulated reboot so its file references must be cleared explicitly during shutdown instead of implicitly during destruction. Re-enabled a useful trace event. Some cleanup, removed unnecessary member var initializer and removed unnecessary method.
2021-02-05 21:26:39 -08:00
f6342376ab
Update the knob name
2021-02-02 13:31:00 -08:00
17671604c0
Add option to define I/O size
2021-02-02 12:39:05 -08:00
e4b953ab38
Merge pull request #4229 from sfc-gh-satherton/afcache-write-limit
...
AsyncFileCached RateControl support and bug fixes
2021-01-31 15:13:03 -07:00
d607ba3ac1
Removed merge artifact which calls functions not present in release-6.2.
2021-01-31 03:59:36 -08:00
15aa877388
Use VFSAsyncFile::checkInjectedError() to replace old injected fault detection, see pr#4212
2021-01-30 18:25:43 -08:00
bb6608ccd8
Another bug fix in SQLite file instance lifetimes. Writer can be destroyed before doClose() or stopOnError() have a chance to react, so Writer can't be used to get access to file handles to remove the rate limiter and trigger pending requests to complete. New logic is for Writer initialization to place opened file handles in KeyValueStoreSQLite instance and move disableRateControl() there, to be called from doClose().
2021-01-30 18:23:50 -08:00
4982787a80
Another attempt to fix SQLite lifetime issues. KeyValueStoreSQLite::doClose() can now access the Writer instance and call disableRateControl() on its SQLiteDB instance prior to stopping the reader and writer thread pools. This should prevent rate limiting from interfering with SQLite shutdown.
2021-01-30 18:18:36 -08:00
b0fd6bfbcf
More bug fixes in SQLite file handle lifetimes after storage server restart (including simulated process reboot) when a write rate limit is used. Added several debug trace events, renamed some VFSAsync* events to be more consistent.
2021-01-30 18:18:16 -08:00
52c2695f59
Bug fixes involving use of AFCached rate control in simulation.
2021-01-30 18:13:44 -08:00
e56fe02a25
Merge branch 'add-throttling-on-AsyncfileCached' of github.com:sfc-gh-clin/foundationdb into afcache-write-limit
2021-01-30 18:11:54 -08:00
14af5857ed
Another refactor of SQLite injected error handling to address failures of the previous attempt. Some code cleanup. Added documentation of injected error handling works and why.
2021-01-30 18:06:05 -08:00
e72d486215
Refactored how injected fault state is managed to make debugging injected fault handling easier. Moved injected error tracking for open() into an INetwork global since it should be unique per simulated process. Fixed bug where injected error in WAL file was not recognized as injected because it occurred before SQLiteDB's vfsWAL pointer was initialized. Simplified logic in SQLiteDB. Added comments.
2021-01-30 18:06:04 -08:00
1a5c40fcef
Bug fixes: Check for null pager and wal pointers, and initialize vfsWAL after SQLite has opened the WAL.
2021-01-30 18:06:04 -08:00
e1f8023577
Rewrote how injected faults are handled in SQLite to be more reliable and work with an upcoming write throttling feature in AsyncFileCached.
2021-01-30 18:06:04 -08:00
4c4d1ec6b0
Use VFSAsyncFile::checkInjectedError() to replace old injected fault detection, see pr#4212
2021-01-29 15:28:08 -08:00
14e39a8747
Another bug fix in SQLite file instance lifetimes. Writer can be destroyed before doClose() or stopOnError() have a chance to react, so Writer can't be used to get access to file handles to remove the rate limiter and trigger pending requests to complete. New logic is for Writer initialization to place opened file handles in KeyValueStoreSQLite instance and move disableRateControl() there, to be called from doClose().
2021-01-28 00:59:05 -08:00
e3a6531477
Another attempt to fix SQLite lifetime issues. KeyValueStoreSQLite::doClose() can now access the Writer instance and call disableRateControl() on its SQLiteDB instance prior to stopping the reader and writer thread pools. This should prevent rate limiting from interfering with SQLite shutdown.
2021-01-27 05:31:02 -08:00
989fee908a
More bug fixes in SQLite file handle lifetimes after storage server restart (including simulated process reboot) when a write rate limit is used. Added several debug trace events, renamed some VFSAsync* events to be more consistent.
2021-01-27 00:01:33 -08:00
0d32bd0c44
Bug fixes involving use of AFCached rate control in simulation.
2021-01-24 01:50:09 -08:00
523a8c08db
Merge branch 'add-throttling-on-AsyncfileCached' of github.com:sfc-gh-clin/foundationdb into afcache-write-limit
...
# Conflicts:
# flow/Knobs.h
2021-01-17 05:30:55 -08:00
11927f6fff
Another refactor of SQLite injected error handling to address failures of the previous attempt. Some code cleanup. Added documentation of injected error handling works and why.
2021-01-17 04:17:13 -08:00
f1adafaf8c
Refactored how injected fault state is managed to make debugging injected fault handling easier. Moved injected error tracking for open() into an INetwork global since it should be unique per simulated process. Fixed bug where injected error in WAL file was not recognized as injected because it occurred before SQLiteDB's vfsWAL pointer was initialized. Simplified logic in SQLiteDB. Added comments.
2021-01-16 05:04:30 -08:00
8df6874417
Bug fixes: Check for null pager and wal pointers, and initialize vfsWAL after SQLite has opened the WAL.
2021-01-15 21:13:31 -08:00
6da78b1b19
Rewrote how injected faults are handled in SQLite to be more reliable and work with an upcoming write throttling feature in AsyncFileCached.
2021-01-15 19:29:14 -08:00
0613a149a6
Add rate control on both db file and log file
2020-12-11 00:22:26 -08:00
9156d38216
Add knob SQLITE_READER_THREADS to control the number of read threads
2020-12-09 12:09:07 -08:00
5b55216252
Add rate control to AsyncFileCached
2020-12-09 12:03:32 -08:00
aa07df6a91
Backport to 6.3 the ability to read xxhash3 checksum for sqlite pages
2020-11-24 00:28:49 -08:00
1c6aef76b5
When one of the sqlite reader or writer thread pools fail, fail the other with the same error.
2020-02-24 12:39:04 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
df2b0452b4
Step 3 of fixing storage server range reads: change return type of readRange from VectorRef<KeyValueRef> to RangeResultRef.
2020-02-06 13:19:24 -08:00
d09e07f1f5
Merge remote-tracking branch 'upstream/master' into features/icc
2020-02-04 10:26:18 -08:00
da73164eda
Move crc32c from fdbrpc to flow
...
So that we can use it from a piece of flow code without breaking module
boundaries.
Also rename generated-constants to crc32c-generated-constants so that
it's more apparent that they're related files.
2020-01-13 18:19:30 -08:00
41583aa576
Guard unlikely indexCode calculation per Steve
2019-12-03 21:48:30 -08:00
36e9f40fc2
Fix negative shift exponent
...
fdbserver/KeyValueStoreSQLite.actor.cpp:438:11: runtime error: shift exponent -1 is negative
2019-12-02 12:51:33 -08:00
7a500cd37f
A giant translation of TaskFooPriority -> TaskPriority::Foo
...
This is so that APIs that take priorities don't take ints, which are
common and easy to accidentally pass the wrong thing.
2019-06-25 02:47:35 -07:00
844dd60202
FDB compiling with intel compiler
2019-06-20 09:29:01 -07:00
ba52623637
Merge pull request #1582 from tclinken/features/sqlite-crc32c
...
Use crc32 for sqlite page checksums
2019-06-17 14:20:41 -07:00
603721e125
Merge branch 'master' into thread-safe-random-number-generation
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/genericactors.actor.cpp
# fdbrpc/sim2.actor.cpp
# fdbserver/DiskQueue.actor.cpp
# fdbserver/workloads/BulkSetup.actor.h
# flow/ActorCollection.actor.cpp
# flow/Net2.actor.cpp
# flow/Trace.cpp
# flow/flow.cpp
2019-05-23 08:35:47 -07:00
f4fbaac6b0
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-05-19 10:27:59 -07:00
12ff747e6a
Avoid tracing in PageChecksumCodec::checksum if silent flag is set
2019-05-17 10:49:53 -07:00
3fac380b90
Avoid tracing in PageChecksumCodec::checksum if silent flag is set
2019-05-17 10:43:28 -07:00
20e93c67ea
Allow sqlite pages to be checked for CRC32 checksum
...
Future versions of FDB will write sqlite pages with CRC32 checksums. In
order to roll back to this version from a version that writes CRC32
checksums, this version must be able to verify those checksums.
2019-05-17 01:05:06 -07:00
d7bcbe1210
Refactored PageChecksumCodec::checksum
2019-05-16 16:07:35 -07:00
90d886df95
Trace both hashlittle2 and crc32 checksums for SQLitePageChecksumFailure
2019-05-16 15:51:21 -07:00
04a72bdad6
Eliminate duplicate code in PageChecksumCodec::checksum
2019-05-16 11:09:37 -07:00
aca90cd4e2
Don't use memcpy in PageChecksumCodec::checksum
2019-05-16 07:25:58 -07:00
601c38ad82
Use crc32 for sqlite page checksums
2019-05-14 13:43:55 -07:00
A.J. Beamon