5967ef5eab
Added back the changes that report trace log flush failures and fix the random crash
2020-03-12 14:34:19 -07:00
a9136f3f72
Add waitForUnreliableExtraStoreReboot to wait for extra store to reboot
2020-03-12 10:18:31 -07:00
303df197cf
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogRouter.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/fdbserver.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/flow.vcxproj
# flow/flow.vcxproj.filters
# versions.target
2020-03-06 18:22:46 -08:00
1128666840
added additional logging on the log router
2020-03-05 18:17:06 -08:00
39610d15f8
Revert this change since it somehow introduced a random crash detected on circus
2020-03-04 16:14:38 -08:00
f20619c9fb
Resolve review comments. Changed how issues got cleared
2020-02-25 15:39:51 -08:00
fce71e4516
Added a TODO for the usage of 'issues' in 'monitorServerDBInfo'
2020-02-25 15:39:38 -08:00
090c89e90a
Addressed review comments. Fix the bug where issues on a worker may be wrongly cleared by subsequent GetDBinfo request.
2020-02-25 15:39:38 -08:00
288e95c7e1
Reallocate the issues set after each get. Changed an issues name to be accurate
2020-02-25 15:39:09 -08:00
f4f860bfa8
Changed issue reporting to be thread safe. Also changed the liveness ping to be thread safe.
2020-02-25 15:38:14 -08:00
a6580dc15f
Added the ability to ping a trace log writer thread and the monitoring in worker.actor.cpp. The current solution is simple a loose check. We can change this to be accurate check by using 'pthread_kill(writer_thread, 0)'
2020-02-25 15:37:53 -08:00
0b0414fb94
Addressded review comments. Change the issue reporting from 'ITraceLogWriter' to be a more generic way.
2020-02-25 15:37:53 -08:00
034dfe5e42
Now the inability to flush trace logs will be reported to both 'stderr' and also the status json object.
...
- Since the first flush failure, if the accumulated consecutive failure count exceeds the value defined in knobs, it will trigger the current worker process to report this issue via the 'GetServerDBInfo' interface of the cluster controler
- A successful flush will reset the accumulated counter.
Notice that the current solution does not take the time into consideration. The assumption is that flush failures tend to only happen in a clustered manner. The intermittent, but short, periods of flush failures are not considered as a problem since the memory pressure built by them should be negligible.
2020-02-25 15:37:32 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
59ff782927
fix: only delete the processId file on binaryReader errors
2020-02-20 23:04:39 -08:00
5586e6f6d8
Merge pull request #2697 from etschannen/feature-correctness-fixes
...
A variety of correctness fixes
2020-02-20 13:32:18 -08:00
9b3254d5f4
A corrupted processId file should be deleted in simulation, as that is the manual operation that would fix the problem in the real world
2020-02-19 15:21:42 -08:00
94d799552e
FastRestore:Apply clang-format against master
2020-02-18 16:41:59 -08:00
132f5aa9ba
FastRestore:Improve trace name and cosmetic change
2020-02-18 16:41:19 -08:00
31a6ec34b7
Merge branch 'master' into mengxu/fast-restore-agent-PR
2020-02-18 16:17:59 -08:00
1be6915a38
Fix an incorrect if else check.
2020-02-17 17:31:41 -08:00
1d9140d874
Removed TLogVersion logging.
...
Added logging of SharedTLog ID for each TLog.
Switched ID logged for TLogRejoining event to the TLog instead of the SharedTLog.
Made some parameters to startRole passed by reference.
2020-02-14 12:33:43 -08:00
32165c506f
Fix a one line bug where the if check comparison was wrong.
2020-02-12 16:55:33 -08:00
56053c565b
Improve TLog "Role" event by adding the worker ID, the TLog version, and under what circumstances the TLog is being started (Restored, Recruited, or Recovered).
...
The SharedTLog role was being started and stopped twice, so remove one instance of it.
2020-02-12 15:11:38 -08:00
3b57bf1781
Merge branch 'master' into mengxu/fast-restore-agent-PR
2020-02-03 17:23:54 -08:00
4524831456
Merge pull request #2518 from vishesh/task/failmon-remove-server
...
FailureMonitoring: Server processes no longer need to talk to ClusterController
2020-02-03 17:22:50 -08:00
ca3b6135d0
FastRestore:Add debug to see why restore role is not connected
...
Reason: restore is a fdbserver who does not register with CC.
The new failure monitor changes how connection works for client and server.
For client, it does not connect to CC to get connected.
For server, it has to connect to CC to get connected.
Restore worker becomes the special role that behaves like a client but is a server.
2020-02-03 17:19:52 -08:00
9c2046b11b
FastRestore:Minic fdbd to monitor coordintors
...
Before we start a fdb restore process.
2020-02-03 14:48:31 -08:00
559b95c61a
FastRestore:RestoreRole:Mimic how fdbd starts
2020-02-01 10:23:48 -08:00
ee6490c9d1
Merge pull request #2314 from mengranwo/memory-engine
...
New Radix-Tree based Memory Storage Engine
2020-01-30 16:20:13 -08:00
1eaea91cb3
Address review comments
2020-01-22 19:42:13 -08:00
85c4a4e422
Address review comments for PR #1625
2020-01-22 19:38:45 -08:00
116608a0a7
Set backup workers w.r.t. the correct epoch
...
For backup workers created for previous epoch, we need to associate them with
the correct epoch so that later peekLogRouter can get the correct peek cursor.
Otherwise, the workers can never peek the missing range of mutations.
2020-01-22 19:38:45 -08:00
de8d953865
Add backup role, class, and worker skeleton
2020-01-22 19:35:30 -08:00
daef5f011a
Merge remote-tracking branch 'apple/master' into task/failmon-remove-server
2020-01-21 13:20:15 -08:00
b0a1af1288
Added the actual read hot detection algorithm and logging machanism.
...
- When a shard has a read bandwidth larger than a threshold value(configurable via knob), and it's read-bandwidth/byte-size ratio is also larger than a threshold(configurable via knob), the corresponding shard tracker will run the algorithm
- The algorithm will divide the shard into 10MB(configurable via knob) chunks and try to find the chunk(s) that has large aforementioned ratio
- Then those ranges will be logged into TraceEvents. This will later do more like actually cache them.
2020-01-21 11:19:52 -08:00
3f9d9d8b84
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# cmake/FlowCommands.cmake
# documentation/sphinx/source/release-notes.rst
# fdbclient/StorageServerInterface.h
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# flow/Knobs.h
# flow/Platform.cpp
# versions.target
2020-01-16 18:37:47 -08:00
f597aa7e18
WIP : deployable/stable version since Nov 3. Start rebase to master branch
2020-01-15 13:49:45 -08:00
da1be272cb
fix: servers which opened the database would use the full list of proxies
2020-01-10 12:20:30 -08:00
6e6cfaff16
Cleanup old Failure Monitoring code
2020-01-07 15:53:32 -08:00
ebcb2f79ed
Merge branch 'master' of github.com:apple/foundationdb
2019-11-22 15:34:49 -08:00
8d3ef89540
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# fdbclient/MutationList.h
# fdbserver/MasterProxyServer.actor.cpp
# versions.target
2019-11-14 15:49:56 -08:00
a4a0bf18f9
Merging with Master.
2019-11-12 13:01:29 -08:00
1e5677b55a
increase the priority of reboot and recruitment requests
2019-11-11 15:17:11 -08:00
103cc37a35
added datahall kill and option to target a specific datahall/dc/machine id
2019-10-23 14:19:17 -07:00
1eb3a70b96
Spill SharedTLog when there's more than one.
...
When switching between spill_type or log_version, a new instance of a
SharedTLog is created in the transaction log processes. If this is done
in a saturated database, then doubling the amount of memory to hold
mutations in memory can cause TLogs to be uncomfortably close to the 8GB
OOM limit.
Instead, we now thread which UID of a SharedTLog is active, and the
other TLog spill out the majority of their mutations.
This is a backport of #2213 (fef89aa1
2019-10-17 01:24:50 -07:00
77c72de176
Comment variable and code style fix
...
Co-Authored-By: Jingyu Zhou <jingyuzhou@gmail.com>
2019-10-07 18:08:27 -07:00
a34a009bf0
Shuffle member initialization in constructor.
2019-10-07 18:08:27 -07:00
71af24dff3
Fix a bug that would cause active logs to spill aggressively
...
And add some useful logging about when things do or do not spill.
2019-10-07 18:08:27 -07:00
1d8a7e5af7
Spill SharedTLog when there's more than one.
...
When switching between spill_type or log_version, a new instance of a
SharedTLog is created in the transaction log processes. If this is done
in a saturated database, then doubling the amount of memory to hold
mutations in memory can cause TLogs to be uncomfortably close to the 8GB
OOM limit.
Instead, we now thread which UID of a SharedTLog is active, and the
other TLog spill out the majority of their mutations.
2019-10-07 18:08:27 -07:00
9f9d2dff42
Point both spill types at the same SharedTLog for >=V5
...
This actually results in TLog generations of different spill types in
the same disk queue. We also drop the LS_ parameter from the filename
to signify that there is no log spill configuration per file anymore.
2019-10-03 01:45:50 -07:00
6742222084
Make TLogServer able to spill by value and by reference
...
...and test it in simulation, but not combined yet.
It turns out that because of txsTag, we basically had to support
spill-by-value anyway. Thus, if we treat all tags like txsTag when
spilling and peeking, then we have an easy way to bring the two spilling
types back into one implementation.
2019-10-03 01:45:10 -07:00
24c46337e1
Advance TLogVersions for 7.0 while adding a V5 that is TLogServer
...
Advancing the MIN_RECRUITABLE and DEFAULT is just following the standard
progression for 7.0. It was convenient to do while adding the V5 so
that we can hook TLogServer back into being used.
2019-10-03 01:43:01 -07:00
60fb04ca68
Fork TLogServer into TLogServer_6_2
...
This prepares us for incoming modifications to the TLog that can't
easily coexist with our current on-disk state.
2019-10-03 01:41:25 -07:00
b495cc697b
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-09-13 09:25:08 -07:00
945cff1e5b
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-10 14:27:22 -07:00
41fd3d9467
Merge branch 'master' into remove-unused-ssi-get-version
...
# Conflicts:
# fdbclient/StorageServerInterface.h
# fdbserver/storageserver.actor.cpp
2019-09-10 08:11:43 -07:00
24aad14f06
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-08-30 17:23:58 -07:00
b3277f2982
Fix #2009 posix compliant args for snapshot binary
2019-08-30 12:54:09 -07:00
3d0bfbd66e
Revert a change moving promise out as state variable
...
The "recovery" is constructed in a loop, so moving out is wrong.
2019-08-27 11:34:43 -07:00
64ce0c3285
Remove the unused getVersion from StorageServerInterface.
2019-08-26 13:53:54 -07:00
17e2630b21
Move state variable to the head of the function
2019-08-22 14:21:13 -07:00
26444cad95
Fix another vector destructor caused broken promise
2019-08-22 14:12:05 -07:00
085706c70c
Fix a clang vector destruction issue.
...
This addresses issue #1862 .
2019-08-22 10:21:13 -07:00
14648e20f9
Merge pull request #1901 from ajbeamon/data-distribution-receives-bytes-input-rate
...
Send bytes input rate to data distribution
2019-07-30 15:01:36 -07:00
9afd162e2f
remove snap v1 related code
2019-07-25 17:29:31 -07:00
b91795d288
Send bytes input rate to DD.
2019-07-25 16:27:32 -07:00
a65c9f92ed
get rid of all timeouts and other changes
2019-07-24 15:36:28 -07:00
31c010b393
few minor fixes
2019-07-24 15:36:28 -07:00
080b3da322
includeCancelled for workerSnapCreate
2019-07-24 15:36:28 -07:00
8f1f0c0435
snap v2: worker and other helper related changes
2019-07-24 15:36:28 -07:00
7ad0d1a12b
Merge branch 'master' into feature-proxy-forward
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
2019-07-09 17:26:15 -07:00
cce00bb413
Merge pull request #1808 from ajbeamon/improved-transaction-metrics
...
Improve TransactionMetrics
2019-07-09 16:46:17 -07:00
d032d7fcf9
fix: if we get a broken_promise from the actor, wait to get the real error from the store
2019-07-09 16:37:54 -07:00
eabc610daa
Merge pull request #1813 from alexmiller-apple/log-version-4
...
Add a TLogVersion::V4
2019-07-09 08:42:20 -07:00
d2ef84a8f9
Add a TLogVersion::V4
...
And refactor some code to make adding more TLogVersions easier.
2019-07-08 22:22:45 -07:00
a5a6f8431c
Add a random UID to TransactionMetrics in case a client opens multiple connections and also a field to indicate whether the connection is internal. Convert some of the metrics to our Counter object instead of running totals.
2019-07-08 14:01:04 -07:00
ea6898144d
Merge remote-tracking branch 'upstream/master' into flowlock-api
2019-07-03 20:44:15 -07:00
8c10d832a1
Add coordinator role in trace events
2019-07-03 11:09:36 -07:00
0b9cd18fb4
checking cluster is healthy or not during recovery process(for storage engine), if healthy, delete data files and join as new
2019-07-01 16:09:51 -07:00
bf883d7055
Merge remote-tracking branch 'upstream/master' into flowlock-api
2019-06-25 14:26:50 -07:00
7a500cd37f
A giant translation of TaskFooPriority -> TaskPriority::Foo
...
This is so that APIs that take priorities don't take ints, which are
common and easy to accidentally pass the wrong thing.
2019-06-25 02:47:35 -07:00
37c1df2491
Merge pull request #1705 from bnamasivayam/suspend-process
...
Extend RebootRequest API to include time to suspend the process befor…
2019-06-20 17:36:25 -07:00
5eb833759e
Extend RebootRequest API to include time to suspend the process before reboot. This is intended to be used for testing purposes to simulate failures.
2019-06-14 11:35:38 -07:00
773f52d0a1
Merge remote-tracking branch 'upstream/master' into cloexec
2019-06-03 15:43:32 -07:00
d3a179b6f9
Multiple bug fixes
...
- wait for snapTLogFailKeys in a loop, otherwise in some race
condition it can cause a false assert
- in single region, there does not seem to be a guarantee of
tagLocalityListKey for a given DC ID, avoiding that assert for now
- to find the workers that are coordinators, looking up by primary
address is not sufficient in some cases, hence looking by both
primary and secondary address
- test make files to reflect the location of the new test cases
2019-05-28 22:07:46 -07:00
936ffc2dde
rebase related changes
2019-05-28 22:07:46 -07:00
ec7834e2f7
code re-orgnaization and address comments
2019-05-28 22:07:46 -07:00
b6e037ffbc
Replace fork with boost::process::child
2019-05-28 22:07:46 -07:00
61e93a9304
Address review comments and minor fixes
2019-05-28 22:07:46 -07:00
9e3104c2d4
Fix: races in async exec leading to bad backup
2019-05-28 22:07:46 -07:00
cfdad0c5e6
tlog to snapshot exactly at exec version
2019-05-28 22:07:46 -07:00
17ecba8313
trace cleanup and other indentation changes
2019-05-28 22:07:46 -07:00
898bed66c1
Allow only whitelisted binary path for exec op
2019-05-28 22:07:46 -07:00
aa79480d69
changes to make fdbfork asynchronous
2019-05-28 22:07:46 -07:00
72dd067173
Trace message changes and fix few FIXMEs
2019-05-28 22:07:46 -07:00
69edefe68b
Snapshot based backup and resotre implementation
2019-05-28 22:07:46 -07:00
69fb852ee0
Add more CLOEXEC-like things.
...
From missed call sites found during/after code review.
2019-05-14 20:30:58 -10:00
5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
1a4c1759a4
Merge pull request #1429 from jzhou77/pprof
...
Dump heap profiler when memory usage is high
2019-04-29 16:31:44 -07:00
Xin Dong