e72d486215
Refactored how injected fault state is managed to make debugging injected fault handling easier. Moved injected error tracking for open() into an INetwork global since it should be unique per simulated process. Fixed bug where injected error in WAL file was not recognized as injected because it occurred before SQLiteDB's vfsWAL pointer was initialized. Simplified logic in SQLiteDB. Added comments.
2021-01-30 18:06:04 -08:00
1a5c40fcef
Bug fixes: Check for null pager and wal pointers, and initialize vfsWAL after SQLite has opened the WAL.
2021-01-30 18:06:04 -08:00
e1f8023577
Rewrote how injected faults are handled in SQLite to be more reliable and work with an upcoming write throttling feature in AsyncFileCached.
2021-01-30 18:06:04 -08:00
4c4d1ec6b0
Use VFSAsyncFile::checkInjectedError() to replace old injected fault detection, see pr#4212
2021-01-29 15:28:08 -08:00
14e39a8747
Another bug fix in SQLite file instance lifetimes. Writer can be destroyed before doClose() or stopOnError() have a chance to react, so Writer can't be used to get access to file handles to remove the rate limiter and trigger pending requests to complete. New logic is for Writer initialization to place opened file handles in KeyValueStoreSQLite instance and move disableRateControl() there, to be called from doClose().
2021-01-28 00:59:05 -08:00
e3a6531477
Another attempt to fix SQLite lifetime issues. KeyValueStoreSQLite::doClose() can now access the Writer instance and call disableRateControl() on its SQLiteDB instance prior to stopping the reader and writer thread pools. This should prevent rate limiting from interfering with SQLite shutdown.
2021-01-27 05:31:02 -08:00
989fee908a
More bug fixes in SQLite file handle lifetimes after storage server restart (including simulated process reboot) when a write rate limit is used. Added several debug trace events, renamed some VFSAsync* events to be more consistent.
2021-01-27 00:01:33 -08:00
564c70acb4
Merge branch 'master' into anoyes/modernize-use-override
2021-01-25 18:26:18 -08:00
4ee97c0784
Use clang-tidy to automatically fix missing overrides
...
Use `clang-tidy -p . $file -checks='-*,modernize-use-override' -header-filter='.*' -fix`
to fix missing overrides, and then use git clang-format to reformat just
those changes. This went pretty well for most files.
Formatting the following files went off the rails, so I'm going to
follow up with a commit that's just clang-tidy and no clang-format.
- fdbclient/DatabaseBackupAgent.actor.cpp
- fdbclient/FileBackupAgent.actor.cpp
- fdbserver/OldTLogServer_4_6.actor.cpp
- fdbmonitor/SimpleIni.h
- fdbserver/workloads/ClientTransactionProfileCorrectness.actor.cpp
2021-01-26 02:04:12 +00:00
adde95bea5
Merge branch 'release-6.3' into anoyes/merge-6.3-to-master
...
Commit merge conflicts for now
2021-01-25 18:55:24 +00:00
0d32bd0c44
Bug fixes involving use of AFCached rate control in simulation.
2021-01-24 01:50:09 -08:00
523a8c08db
Merge branch 'add-throttling-on-AsyncfileCached' of github.com:sfc-gh-clin/foundationdb into afcache-write-limit
...
# Conflicts:
# flow/Knobs.h
2021-01-17 05:30:55 -08:00
11927f6fff
Another refactor of SQLite injected error handling to address failures of the previous attempt. Some code cleanup. Added documentation of injected error handling works and why.
2021-01-17 04:17:13 -08:00
f1adafaf8c
Refactored how injected fault state is managed to make debugging injected fault handling easier. Moved injected error tracking for open() into an INetwork global since it should be unique per simulated process. Fixed bug where injected error in WAL file was not recognized as injected because it occurred before SQLiteDB's vfsWAL pointer was initialized. Simplified logic in SQLiteDB. Added comments.
2021-01-16 05:04:30 -08:00
8df6874417
Bug fixes: Check for null pager and wal pointers, and initialize vfsWAL after SQLite has opened the WAL.
2021-01-15 21:13:31 -08:00
6da78b1b19
Rewrote how injected faults are handled in SQLite to be more reliable and work with an upcoming write throttling feature in AsyncFileCached.
2021-01-15 19:29:14 -08:00
0613a149a6
Add rate control on both db file and log file
2020-12-11 00:22:26 -08:00
9156d38216
Add knob SQLITE_READER_THREADS to control the number of read threads
2020-12-09 12:09:07 -08:00
5b55216252
Add rate control to AsyncFileCached
2020-12-09 12:03:32 -08:00
7fbc4d7391
Resolve conflicts
2020-12-04 23:58:42 +00:00
877997632d
Merge branch 'release-6.3' into anoyes/merge-release-6.3-master
...
Include conflict markers for review purposes
2020-12-04 01:38:07 +00:00
aa07df6a91
Backport to 6.3 the ability to read xxhash3 checksum for sqlite pages
2020-11-24 00:28:49 -08:00
4029217c2f
Update fdbserver/KeyValueStoreSQLite.actor.cpp
...
Co-authored-by: Andrew Noyes <63815641+sfc-gh-anoyes@users.noreply.github.com>
2020-11-23 10:07:08 -08:00
156a617ed8
Replace xxhash64 with xxhash3
...
The goal here is to improve performance
2020-11-21 18:25:35 -08:00
bcda617b80
Benchmark hashing algorithms in flowbench
2020-11-17 23:14:55 -08:00
e9bceda8ca
Add code coverage macros to PageChecksumCodec::checksum
2020-11-16 19:13:18 -08:00
287ab51502
Fix some bugs in PageChecksumCodec::checksum
2020-11-15 20:35:05 -08:00
145cddbb99
Clear the first 8 bits of xxHash64 checksum of SQLite pages
...
When upgrading a cluster to 7.0 which has stale data from 6.2, we don't
want to calculate an xxHash64 checksum for every page that's read. By
clearing the first 8 bits of all checksums written using xxHash64, the
server can skip this checksum in 255/256 cases when reading pages
written from version <=6.2.
2020-11-15 17:48:44 -08:00
c5689c4a72
Use xxhash64 for checksumming SQLite pages
2020-11-15 17:48:44 -08:00
4669f837fa
Add uses of makeReference
2020-11-07 22:10:18 -08:00
410d418c3e
Revert "Merge pull request #3770 from dongxinEric/feature/ss-commit-pipelining"
...
This reverts commit e258dffc650e96233015
2020-10-19 09:28:38 -07:00
dc4612712d
Merge remote-tracking branch 'origin/master' into use-override-more
2020-10-09 07:12:41 -07:00
e258dffc65
Merge pull request #3770 from dongxinEric/feature/ss-commit-pipelining
...
Pipeline the commit process in SS.
2020-10-07 20:34:57 -07:00
8571dcfe28
Use override where applicable in fdbserver
2020-10-07 18:41:19 -07:00
34d1d04904
Commit pipelining attempt #2
2020-09-23 20:03:04 -07:00
0814841827
Replace NULL with nullptr in fdbserver
2020-09-20 11:31:49 -07:00
77e3b314f6
Mark IKeyValueStore::getStorageBytes const
2020-06-17 18:20:43 -07:00
2a393633b6
Improved const-correctness
...
Using the recently added IndexedSet::const_iterator
(https://github.com/apple/foundationdb/pull/3185 ), we can improve the
const-correctness of many functions. In this PR const is added where
applicable. Also, wherever I came across the following while adding
const, I made the following changes:
- virtual function overrides are marked as override
- NULL is replaced with nullptr
- git clang-format is applied
2020-06-17 18:20:43 -07:00
1c6aef76b5
When one of the sqlite reader or writer thread pools fail, fail the other with the same error.
2020-02-24 12:39:04 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
df2b0452b4
Step 3 of fixing storage server range reads: change return type of readRange from VectorRef<KeyValueRef> to RangeResultRef.
2020-02-06 13:19:24 -08:00
d09e07f1f5
Merge remote-tracking branch 'upstream/master' into features/icc
2020-02-04 10:26:18 -08:00
da73164eda
Move crc32c from fdbrpc to flow
...
So that we can use it from a piece of flow code without breaking module
boundaries.
Also rename generated-constants to crc32c-generated-constants so that
it's more apparent that they're related files.
2020-01-13 18:19:30 -08:00
41583aa576
Guard unlikely indexCode calculation per Steve
2019-12-03 21:48:30 -08:00
36e9f40fc2
Fix negative shift exponent
...
fdbserver/KeyValueStoreSQLite.actor.cpp:438:11: runtime error: shift exponent -1 is negative
2019-12-02 12:51:33 -08:00
7a500cd37f
A giant translation of TaskFooPriority -> TaskPriority::Foo
...
This is so that APIs that take priorities don't take ints, which are
common and easy to accidentally pass the wrong thing.
2019-06-25 02:47:35 -07:00
844dd60202
FDB compiling with intel compiler
2019-06-20 09:29:01 -07:00
ba52623637
Merge pull request #1582 from tclinken/features/sqlite-crc32c
...
Use crc32 for sqlite page checksums
2019-06-17 14:20:41 -07:00
603721e125
Merge branch 'master' into thread-safe-random-number-generation
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/genericactors.actor.cpp
# fdbrpc/sim2.actor.cpp
# fdbserver/DiskQueue.actor.cpp
# fdbserver/workloads/BulkSetup.actor.h
# flow/ActorCollection.actor.cpp
# flow/Net2.actor.cpp
# flow/Trace.cpp
# flow/flow.cpp
2019-05-23 08:35:47 -07:00
f4fbaac6b0
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-05-19 10:27:59 -07:00
12ff747e6a
Avoid tracing in PageChecksumCodec::checksum if silent flag is set
2019-05-17 10:49:53 -07:00
3fac380b90
Avoid tracing in PageChecksumCodec::checksum if silent flag is set
2019-05-17 10:43:28 -07:00
20e93c67ea
Allow sqlite pages to be checked for CRC32 checksum
...
Future versions of FDB will write sqlite pages with CRC32 checksums. In
order to roll back to this version from a version that writes CRC32
checksums, this version must be able to verify those checksums.
2019-05-17 01:05:06 -07:00
d7bcbe1210
Refactored PageChecksumCodec::checksum
2019-05-16 16:07:35 -07:00
90d886df95
Trace both hashlittle2 and crc32 checksums for SQLitePageChecksumFailure
2019-05-16 15:51:21 -07:00
04a72bdad6
Eliminate duplicate code in PageChecksumCodec::checksum
2019-05-16 11:09:37 -07:00
aca90cd4e2
Don't use memcpy in PageChecksumCodec::checksum
2019-05-16 07:25:58 -07:00
601c38ad82
Use crc32 for sqlite page checksums
2019-05-14 13:43:55 -07:00
5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
2d5043c665
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-04-30 18:27:04 -07:00
9ff8aca1da
Increased the SQLITE_CHUNK_SIZE to 100MB (left at 4MB for simulation)
2019-04-26 13:53:56 -07:00
253d2400ef
Merge branch 'release-6.1' into speed-up-and-parameterize-spring-cleaning
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2019-04-23 14:38:52 -07:00
ea7abff9df
Clean up from review
2019-04-23 14:16:52 -07:00
4ad0496b39
Increase the frequency that lazy deletes are run. Add more parameters for better control over the spring cleaning process.
2019-04-23 14:01:51 -07:00
83db547306
Implemented the chunk size and db size hint fileControl options in our SQLite VFS implementation. KeyValueStoreSQLite now sets file chunk size based on a new knob, SQLITE_CHUNK_SIZE_PAGES.
2019-04-23 04:50:58 -07:00
6220a5ce0f
Merge pull request #1370 from jzhou77/fix-unreferenced
...
Remove unused functions
2019-04-09 11:49:45 -07:00
1c16f87a4e
Remove trace-calls to printable (in non-workloads)
2019-04-05 13:12:19 -07:00
bb82f8560a
process all volatile ints correctly in traces
2019-04-05 13:12:19 -07:00
c008e16c81
Defer formatting in traces to make them cheaper
...
This is the first part of making `TraceEvent` cheaper. The main idea is
to defer calls to any code that formats string. These are the main
changes:
- TraceEvent::detail now takes a c-string instead of std::string for
literals. This prevents unnecessary allocations if the trace is not
going to be printed in the first place (for example for SevDebug).
Before that `detail` expected a `std::string` as key, which mean that
any string literal would be copied on each call.
- Templates Traceable and SpecialTraceMetricType. These templates can be
specialized for any type that needs to be printed. The actual
formatting will be deferred to after the `enabled` check. This
provides two benefits: (1) if a TraceEvent is disabled, we don't pay
for the formatting and (2) TraceEvent can trace types that it doesn't
know about.
- TraceEvent::enabled will be set in the constructor if the Severity is
passed. This will make sure that `TraceEvent::init` is not called.
- `TraceEvent::detail` will be inlined. So for disabled TraceEvent
calls, a call to detail will only introduce a if-branch which is much
cheaper than a function call.
2019-04-05 13:12:19 -07:00
f7f8ddd894
Fix warnings on unused variables
...
Found by -Wunused-variable flag.
2019-04-01 14:00:20 -07:00
067a445e06
Replace unused _ variables with wait(success(...))
2019-02-12 17:30:30 -08:00
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
3922e477a5
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/ManagementAPI.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/LogSystemDiskQueueAdapter.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
2018-10-03 16:57:18 -07:00
48e620c680
Change the first of two trace events named "BTreeIntegrityCheck" to have the name "BTreeIntegrityCheckResults"
2018-09-24 08:40:18 -07:00
b727f0475c
Fix warnings about ACTORs not having waits. Fix shadowing of future variable in KVFileIntegrityCheck.
2018-09-10 10:51:41 -07:00
86dbe1f0e9
Fix more instances of actorcompiler.h being in the wrong place.
2018-08-14 15:50:26 -07:00
fb31a6999f
Rewrite all files to have #include actorcompiler.h as the last include.
2018-08-14 15:50:26 -07:00
535b5701e5
Rewrite all `Void _ = wait(...)` -> `wait(...)`.
...
This takes advantage of the new actorcompiler functionality to avoid
having duplicate definitions of `Void _` when trying to feed the
un-actorompiled source through clang.
2018-08-14 15:50:26 -07:00
1c29275672
call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details.
2018-08-01 14:30:57 -07:00
99c9958db7
Some more trace event normalization
2018-06-08 13:57:00 -07:00
e5488419cc
Attempt to normalize trace events:
...
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.
Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.
This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
0bae9880f1
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-21 10:25:11 -08:00
c7a120c59d
Rename IAsyncFile::incrementalDelete -> IAsyncFileSystem::incrementalDeleteFile.
...
`deleteFile` existed in IAsyncFileSystem, so an incremental delete function
seems to belong more as a virtual method on IAsyncFileSystem than a static
method on IAsyncFile, and the naming should match.
As long as we're here, change IAsyncFile to declare a virtual destructor, so
that it has good and proper C++ behavior. I presume this is what was vaguely
intended by the default constructor definition that previously existed?
2017-11-30 17:19:10 -08:00
2bf042a559
fix: file_corrupt was not checking for fault injection
...
latency threshold was too long
2017-09-25 17:22:41 -07:00
d8e308c18f
Enable use of incremental delete when deleting disk queue and sqlite KVS sqlite files.
2017-07-26 14:11:11 -07:00
f405c8d88e
Merge branch 'release-4.6' into release-5.0
...
# Conflicts:
# fdbrpc/AsyncFileKAIO.actor.h
# fdbrpc/sim2.actor.cpp
# fdbserver/optimisttest.actor.cpp
# versions.target
2017-06-15 17:40:19 -07:00
4bdcd8fc12
Merge branch 'release-4.6' into release-5.0
...
# Conflicts:
# bindings/bindingtester/run_binding_tester.sh
# fdbrpc/AsyncFileKAIO.actor.h
2017-06-14 16:43:53 -07:00
fa4fdb1f1d
Merge branch 'fix-io-timeout-handling' into release-5.0
...
# Conflicts:
# fdbserver/optimisttest.actor.cpp
2017-05-31 17:03:15 -07:00
98604d33a0
Merge branch 'fix-io-timeout-handling'
...
# Conflicts:
# fdbrpc/AsyncFileKAIO.actor.h
# fdbrpc/sim2.actor.cpp
# fdbserver/KeyValueStoreSQLite.actor.cpp
# fdbserver/optimisttest.actor.cpp
# fdbserver/worker.actor.cpp
# fdbserver/workloads/MachineAttrition.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2017-05-26 18:43:08 -07:00
a674cb4ef4
Initial repository commit
2017-05-25 13:48:44 -07:00
Steve Atherton