7bc55448aa
fix iterator bug
2021-05-24 19:11:28 +00:00
85cd2b9945
add perpetualStorageWiggler
2021-05-20 23:31:08 +00:00
3f3a81b3d9
add pid2server_info to maintain Process id set
2021-05-20 03:32:15 +00:00
f28ac955c3
Remove unnecessary temporary objects while growing objects of type std::vector<std::pair<A, B>>
2021-05-10 16:32:50 -07:00
5c2d7b6080
Create RangeResult type alias
2021-05-03 13:14:16 -07:00
0db28f6ea0
Merge pull request #4535 from jzhou77/fix-dd
...
Fix DD Assertion failed in canBeSet
2021-03-24 10:50:04 -07:00
0c3bc09524
Remove the shuttingDown flag
2021-03-21 20:12:37 -07:00
cb26576b95
Fix DD assertion failure
...
This fixes #4493 , where DDTeamCollection::~DDTeamCollection creates new teams
that hold pointer to the DDTeamCollection, thus later causes assertion failure
because the memory is invalid.
The fix is to cancel teamBuilder at the begining of the ~DDTeamCollection.
2021-03-21 19:54:44 -07:00
d2f9bf7eb6
added comments and fixed style
2021-03-16 15:44:49 -07:00
edefcff3ac
do not kill the data distributor after removing a failed server, completely remove the failed server
2021-03-15 16:48:08 -07:00
c0a1362478
fixed a bug where DD was shutdown while still in a callback from trackExcludedServers
2021-03-15 16:26:57 -07:00
c570a7b718
added trace events
2021-03-15 15:55:02 -07:00
403d933329
fixed trace event
2021-03-15 10:51:53 -07:00
831224df99
fixed complier error
2021-03-15 10:48:48 -07:00
4e4149b070
exclude failed shuts down data distribution while the server is being removed to avoid two processes making changes to the key servers at the same time
2021-03-15 10:43:06 -07:00
df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
2bb4f2e59f
Merge branch 'release-6.3-pre-format' into master-format
...
This merges release-6.3 branch right before it was fully formatted.
There were quite a few conflicts that are resolved here. CoroFlow had
a check for OOM errors introduced in 6.3, but didn't seem applicable in
the new implmentation which seems to use boost.
2021-03-10 09:37:41 -08:00
9645f489e6
Fix base trace event name inconsistency
2021-03-08 15:20:50 -08:00
59181245c1
Change SSVersionDiffLarge event log level to warning
2021-03-03 23:33:48 -08:00
79cec09255
Apply clang-tidy's performance-inefficient-vector-operation fix
...
I ran this command in my build directory after compiling with
OPEN_FOR_IDE. It took a few small tweaks to get it to compile, which is
outside the scope of this commit.
$ python run-clang-tidy.py -j $(nproc) -checks='-*,performance-inefficient-vector-operation' -fix
2021-03-04 03:58:25 +00:00
346a4e3ecd
Merge branch 'release-6.3'
...
# Conflicts:
# fdbcli/fdbcli.actor.cpp
# fdbrpc/LoadBalance.actor.h
# fdbrpc/MultiInterface.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/masterserver.actor.cpp
2021-03-01 18:52:06 -08:00
32486a2785
Reenable tlog pops in ddSnapCreateCore even if some disable requests fail
...
If some tlogs successfully disable pops but others fail to, we do not
want to wait TLOG_IGNORE_POP_AUTO_ENABLE_DELAY seconds before reenabling
pops
2021-02-20 18:24:21 -08:00
8025cc4571
Merge remote-tracking branch 'upstream/release-6.3' into merge-6.3-to-master-1-26-21
2021-01-26 19:46:47 +00:00
0ef44739ea
Fix OPEN_FOR_IDE build in preparation for using clang-tidy
2021-01-26 02:04:11 +00:00
0cde3cc48f
Make trace event happy
2021-01-25 14:04:16 -08:00
dce8af5b63
Change back to SevWarnAlways for now.
2021-01-25 13:54:24 -08:00
bdf58d0b2f
Add assertion to DDTeamCollection::overlappingMembers
2021-01-21 14:39:39 -08:00
ad99bf0471
Merge remote-tracking branch 'origin' into misc-changes
2021-01-21 10:03:07 -08:00
83506cda87
Log SevError instead of SevWarnAlways when all replicas of some data are lost.
2021-01-15 15:00:48 -08:00
ff7d306b09
Merge branch 'release-6.3' into anoyes/merge-6.3-to-master
...
Include conflict markers for now. Will resolve.
2021-01-15 18:04:09 +00:00
5b2e88b187
Use structured bindings in for loops
2020-12-27 01:46:20 -04:00
0d4e81e6b4
Use unique_ptr in DataDistribution.actor.cpp
2020-12-26 23:40:54 -04:00
19816ccdbf
Improve DataDistribution const-correctness
2020-12-26 20:22:27 -04:00
26a4884eef
Mark TCMachineTeamInfo::size const
2020-12-26 19:23:01 -04:00
bbb56e4089
Merge branch 'release-6.2' of https://github.com/apple/foundationdb into release-6.3
2020-12-23 14:26:59 -08:00
877997632d
Merge branch 'release-6.3' into anoyes/merge-release-6.3-master
...
Include conflict markers for review purposes
2020-12-04 01:38:07 +00:00
78503db523
Reset and retry transaction errors
2020-12-03 14:42:30 -08:00
ac02329d7d
Added a command in fdbcli to allow user to manually trigger the detailed teams info loggings in data distributor
2020-12-03 14:42:30 -08:00
b8a9807336
Move trackerCancelled higher in catch block
2020-11-24 20:34:06 +00:00
dc2bac5670
Resolve conflicts
2020-11-24 19:09:42 +00:00
1f541f02be
Merge branch 'anoyes/merge-6.2-to-6.3' into anoyes/release-6.3-merge
...
Merge, leaving conflict markers for now
2020-11-24 16:55:34 +00:00
fc9b78737f
Fix some merge bugs
2020-11-17 14:53:02 -08:00
d64cf8b9e3
Merge branch 6.3 into master
2020-11-17 11:22:45 -08:00
489ba20641
Fix several merge issues
2020-11-16 14:46:36 -08:00
d0391db862
Merge branch 'release-6.2' into release-6.3
2020-11-16 10:15:23 -08:00
ca8ea3b6ff
Fix memory issues caused by cancelling data distribution tracker
2020-11-15 23:52:36 -08:00
1343f40117
don't allow empty coments
2020-11-11 14:07:54 -07:00
bdd3dbfa7d
remove duplicates
2020-11-10 14:01:07 -07:00
4669f837fa
Add uses of makeReference
2020-11-07 22:10:18 -08:00
3ae611d668
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-pause-backup-snapshot
2020-11-04 14:26:49 -05:00
bda72d9a3d
first draft at changing snapshot backup behaviour
2020-11-02 17:12:30 -05:00
cf4c8e375f
Merge remote-tracking branch 'origin/release-6.3' into merge
2020-10-29 22:15:41 -07:00
99c1880a83
Merge commit 'f9581de2005e6b085776e81b9fcaa16442b32589' into merge-6.2-to-6.3
...
# Conflicts:
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
2020-10-27 12:21:26 -07:00
be7944773f
Fix a typo
2020-10-26 16:44:52 -07:00
9ef29d0cea
Changed getTeamID() to return a string instead of UID as suggested by reviews.
2020-10-26 16:44:52 -07:00
bec2cfb167
Fix typos.
2020-10-26 16:44:52 -07:00
0bc51bb780
Resolve review comments
2020-10-26 16:44:50 -07:00
7ebb2e5c09
Piggy back this PR to polish more TraceEvent by:
...
- Making it clear that it's tracking machine team info or server team info
- Added ID to both machine team and server team for better trackability
- Attach distributor id to some trace events.
2020-10-26 16:44:09 -07:00
c037bfd001
Added detailed logging when there is no servers left in a server team, because that may indicate a data loss incident.
2020-10-26 16:44:07 -07:00
6395b76d8c
Address more review comments
2020-10-23 15:29:08 -07:00
f757cae786
Address review comments
2020-10-23 14:01:53 -07:00
e0b1f95740
Merge remote-tracking branch 'origin/master' into remove-global-ddenabled-flag
2020-10-21 18:22:08 -07:00
8cc3e4d3c6
Merge release-6.3 into master
2020-10-19 22:51:56 -07:00
44c62b2d51
Merge pull request #3922 from jzhou77/release-6.3
...
Merge Release 6.2 to Release 6.3
2020-10-19 14:38:36 -07:00
652d753daf
Remove global ddEnabled flag
2020-10-17 11:23:52 -07:00
70c1ac2131
Use TraceEvent::error
2020-10-16 16:55:09 -07:00
8d0aa02a63
Do not periodically print detailed DD teams info
2020-10-16 16:11:14 -07:00
9dec4bc46a
Add ErrorCode to StorageServerTrackerCancelled trace event
2020-10-16 15:40:14 -07:00
8f17a1a5d6
Merge branch 'release-6.2' into release-6.3
2020-10-16 15:25:39 -07:00
30488df5ea
Fix build
2020-10-16 14:26:40 -07:00
81193a9226
Move TraceEvent upward
2020-10-16 14:23:27 -07:00
1e0e800751
Fix build
2020-10-16 12:10:07 -07:00
2b87627d1b
Check for cancellation after errorOut.sendError(e)
2020-10-16 12:10:07 -07:00
92e31dd338
Address review comments
2020-10-15 15:25:00 -07:00
1d43729cc9
Added a way to print detailed information about team collection for debugging.
2020-10-15 10:01:56 -07:00
a1e868a569
Merge pull request #3862 from sfc-gh-tclinkenbeard/use-override-more
...
Add uses of override keyword, remove unnecessary uses of virtual
2020-10-14 15:06:45 -07:00
3b66a1f2d4
Fix a couple places where we were creating vectors with default elements rather than reserving space.
2020-10-09 10:51:06 -07:00
a9607bdcec
Explicitly seal classes that inherit but aren't inherited from
2020-10-07 21:58:24 -07:00
8571dcfe28
Use override where applicable in fdbserver
2020-10-07 18:41:19 -07:00
b4ad989252
use stack transaction instead of heap
2020-10-05 16:51:01 -04:00
52a6496a54
fix compiler errors
2020-10-04 16:50:54 -07:00
91a8367acb
Avoid slow task in ~DataDistributionTracker
2020-10-01 11:44:55 -07:00
69580593dd
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-snapshot-record-version
2020-09-23 15:35:05 -04:00
0814841827
Replace NULL with nullptr in fdbserver
2020-09-20 11:31:49 -07:00
260c8d9568
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-snapshot-record-version
2020-09-11 15:05:58 -04:00
ae7bf24353
Merge pull request #3549 from yliucode/grv-proxy
...
Separate out a new role GrvProxy to serve GRVs.
2020-09-03 19:03:45 -07:00
87693cae81
merge master branch and resolve conflicts
2020-09-02 13:44:33 -07:00
b4c96cadc7
Merge branch 'release-6.3' into merge-release-6.3-into-master
2020-09-02 12:45:57 -07:00
d334b6484e
attempt to write to system keys with snapshot
2020-09-02 15:17:54 -04:00
0443ea7a9b
fix: prioritize marking a region as fully replicated over removing machine teams
2020-09-01 15:55:33 -07:00
12edadd059
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# fdbclient/Knobs.cpp
# fdbclient/MasterProxyInterface.h
# fdbrpc/simulator.h
# fdbserver/MasterProxyServer.actor.cpp
# tests/fast/CycleAndLock.txt
# tests/fast/TxnStateStoreCycleTest.txt
# tests/fast/VersionStamp.txt
# tests/slow/ParallelRestoreOldBackupApiCorrectnessAtomicRestore.txt
# tests/slow/ParallelRestoreOldBackupCorrectnessCycle.txt
# versions.target
2020-08-31 19:33:34 -07:00
8994719e46
Merge branch 'master' into grv-proxy
2020-08-31 10:21:32 -07:00
e87327b33b
Merge master branch and keep master proxy reporting txn cost estimation to ratekeeper
2020-08-29 12:47:35 -07:00
ca9b1f5b34
Merge branch 'release-6.3' into mengxu/fr-sched-PR
...
Resolve conflict at BackupContainer.actor.cpp
2020-08-27 16:54:00 -07:00
c3991262cf
Add nullptr check to traceAllInfo
2020-08-27 09:40:42 -07:00
63b3612ad5
Merge master branch and resolve conflicts
2020-08-24 16:42:31 -07:00
3afdb44c7a
merge master
2020-08-23 17:09:04 +00:00
e1b7dd0c7d
Merge remote-tracking branch 'upstream/release-6.3' into dyoungworth/fixMerge1
2020-08-22 12:25:19 -07:00
3b63d8b01b
remove FIXME; remote tagSet.reset(); trivial changes
2020-08-21 19:17:16 +00:00
f864606d8d
Don't block the data distributor when getting a GetDataDistributorMetricsRequest.
2020-08-21 18:16:07 +00:00
6380b92b10
Don't block the data distributor when getting a GetDataDistributorMetricsRequest.
2020-08-21 09:26:18 -07:00
599675cba8
modify some details to get better performance
2020-08-19 04:23:23 +00:00
1e571a5a1a
FastRestore:Loader:Kick off scheduler when loader starts to have new requests
2020-08-15 21:57:00 -07:00
f3ecf14601
change midShardSize type and other details
2020-08-12 17:49:12 +00:00
79ce16650d
merge master branch
2020-08-11 19:22:10 -07:00
0cceda9908
solve distributor present bug
2020-08-11 21:54:52 +00:00
97e49f2f70
Resolve throttling events
2020-08-10 22:01:12 -07:00
696e77c94e
query midShardSize from proxy
2020-08-10 20:13:44 +00:00
df9149fea4
ignore transaction tag of immediate transactions
2020-08-07 23:36:17 +00:00
13307679c5
use median shard size"
2020-08-05 03:57:25 +00:00
b903e60cb7
fix monitorDDMetricsChanges bugs
2020-08-03 17:12:36 +00:00
d1cc87452c
merge with master; solve conflicts; solve initialization;
2020-08-02 22:44:07 +00:00
c3a629588f
add client transaction tag sample
2020-07-31 19:08:42 +00:00
30ea639666
Remove debug traces
2020-07-29 07:55:05 -07:00
f7b76a92af
pass joshua
2020-07-29 07:26:55 -07:00
a49cb41de7
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# cmake/ConfigureCompiler.cmake
# fdbserver/Knobs.cpp
# fdbserver/StorageCache.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/ThreadHelper.actor.h
# flow/serialize.h
# tests/CMakeLists.txt
2020-07-29 00:31:55 -07:00
819e3ab3e8
Merge branch 'ratekeeper'
2020-07-28 16:48:50 +00:00
48a0fb5154
ask DD for shard info
2020-07-25 04:08:12 +00:00
525f10e30c
Merge master branch
2020-07-22 16:08:49 -07:00
ab0d8b0626
Merge pull request #3509 from sfc-gh-anoyes/anoyes/remove-using-relops
...
Remove using namespace std::rel_ops
2020-07-22 11:58:25 -07:00
638f586f78
Remove unnecessary override
2020-07-21 11:05:46 -07:00
83c5a30f62
Add encapsulation to TCTeamInfo and ParallelTCInfo
2020-07-21 11:05:41 -07:00
9a2ce4c981
Make IDataDistributionTeam const-correct
2020-07-21 11:05:34 -07:00
b2a3b4fd83
Merge branch 'master' into mengxu/merge-6.3-PR
2020-07-20 11:34:18 -07:00
1ba9b6b07f
DD:Change SendRelocateToDDQx100 to SendRelocateToDDQueue
2020-07-17 14:10:17 -07:00
098cdfb558
Replace actor_cancelled error with dd_cancelled
2020-07-16 20:26:07 -07:00
ba3c631350
Remove spammy trace
2020-07-16 10:33:24 -07:00
638e612a97
Improve coding style and trace events
2020-07-16 10:25:42 -07:00
acbb389862
Debug and fix very rare crash in TeamTracker
...
teamTracker only works when all DDTeamCollections are valid.
However, teamTracker can be triggered by zeroTeamSignalling event
after a DDTeamCollection is destructed and the other DDTeamCollection has not been
destructed yet.
This causes teamTracker to uses a pointer to the destructed DDTeamCollection and thus
has mysterious failure.
2020-07-16 10:23:02 -07:00
5b06d69d25
Pass watches test
2020-07-15 00:37:41 -07:00
47ae66bd61
Merge branch 'master' into mengxu/tmp-minor-comment-PR
...
Resolve conflict at waitFailureClient
2020-07-13 16:17:50 -07:00
ef8c1060a2
Merge branch 'master' into mengxu/tmp-merge-6.3
2020-07-13 10:15:56 -07:00
6f2e12be42
Minor improvement on comments
2020-07-12 18:32:47 -07:00
f470ba8316
Remove using namespace std::rel_ops
...
This causes the following to not compile anymore
\#include <utility>
\#include <vector>
using namespace std::rel_ops;
int main() {
std::vector<int> xs;
return xs.rbegin() != xs.rend();
}
See https://godbolt.org/z/s1977n
2020-07-10 22:58:15 +00:00
b09dddc07e
Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3
...
# Conflicts:
# cmake/ConfigureCompiler.cmake
# documentation/sphinx/source/downloads.rst
# fdbrpc/FlowTransport.actor.cpp
# fdbrpc/fdbrpc.vcxproj
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/Status.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/flow.vcxproj
2020-07-10 15:06:34 -07:00
5e02fd490e
fix: the check for if a teamCollection was tracking a source server was unreliable, leading to scenarios where we would temporarily replicate a shard less than teamSIze
2020-06-29 10:02:27 -07:00
cf13e00a8f
Merge remote-tracking branch 'origin/release-6.3' into fdb_cache_wo_allocator
2020-06-01 17:38:31 -07:00
6ce574f5ad
Merge remote-tracking branch 'upstream/release-6.3' into add-data-distribution-metrics
2020-05-17 23:36:52 -07:00
c2bc75516f
Merge branch 'release-6.3' of github.com:apple/foundationdb into features/trace-roles
2020-05-14 10:34:53 -07:00
48b1b20f67
Fixed a crash related to destruction order in data distribution
2020-05-10 23:14:19 -07:00
ef724bf939
Merge remote-tracking branch 'upstream/master' into add-data-distribution-metrics
2020-05-08 18:39:28 -07:00
e8b62e48f4
Rename DDMetrics to DDMetricsRef
2020-05-08 17:17:27 -07:00
5f9b127e56
Emit traces regularly about role assignment
...
We are currently emitting Role transition traces when a role starts and
when it ends. While this is useful for debugging, it doesn't work well
with tools that inject data and might potentially miss some trace lines.
We do decorate each trace lines with the roles assigned to that
particular process, however, this is not sufficient for tools that can
make use of the UID -> Role mapping
2020-05-08 16:27:57 -07:00
f4d30f8dce
Fix the compilation error.
2020-05-06 19:09:40 -07:00
94570ea590
Data distributor now waitfails caches
2020-05-06 10:35:56 -07:00
aed2d34bcb
Merge branch 'master' into feature-proxy-load-balance
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# flow/Knobs.cpp
2020-05-01 09:19:39 -07:00
b7f5f3be48
merge in master
2020-04-28 13:11:47 -07:00
0c84ad4bc6
Merge pull request #2917 from bnamasivayam/fail-slow-ss
...
Mark the storage servers that are continually lagging as unhealthy
2020-04-22 23:18:35 -07:00
d5bef6fc32
Update fdbserver/DataDistribution.actor.cpp
2020-04-22 09:45:56 -07:00
ba3e2af473
Merge commit '5288033bcfe40c3ade97c8bf2d04cf31b3f16cb1' into feature-tree-broadcast
2020-04-17 15:17:37 -07:00
Xiaoxi Wang