7e26b4ef0d
Address comments from PR
2019-02-07 15:37:04 -08:00
8a94d80deb
fdbservice and fdbrpc now compiling
2019-02-07 15:37:04 -08:00
455024b3fe
SimulationTest: Test the number of teams
...
Magnify the possibility that the number of created machine teams is
larger than the number of desired machine teams if we do NOT try to remove the surplus machine teams.
This help test the upgrade to machine team in FDB 6.1
2019-02-06 11:04:41 -08:00
8bfde8c571
fix: increased the rate of ssl tests too much
2019-02-04 11:39:49 -08:00
5b471699df
fix: restarting simulation tests looked for directories in the wrong location
2019-02-04 11:39:06 -08:00
e9ddd94e27
The failure monitor is given a list of all IP addresses associated with a process
...
The connect packet includes the correct remote address
Did a lot of code cleanup
Simulation test mixed TLS and non-TLS listeners on the same process
2019-01-31 18:20:14 -08:00
699f8dd617
fix: coordinators auto could put two coordinators in the same zone
...
simulation now tests two machines in the same zone
2019-01-18 15:42:48 -08:00
51b89ae083
WIP
2019-01-09 07:41:02 -08:00
e04abf25f7
simulator: Support multiple listeners on single process
...
Sim2Listener can now take the network address to listen on. This is
used to listen to multiple ports in simulator and test the patch
which added multiple network addresses to single endpoint.
2018-12-13 13:36:52 -08:00
487f8b5530
Increase the simulation timeout value to avoid operation_cancelled error
...
The default timeout value is 5400 simulation seconds when buggify is off.
Because we are restoring database sequentially, we are super slow.
We may have a correct restore longer than 5400 simulation seconds.
Increase the timeout threshold to avoid false positive error.
2018-12-07 16:12:50 -08:00
4b5d0b4e2c
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/AsyncFileBlobStore.actor.cpp
# fdbclient/AsyncFileBlobStore.actor.h
# fdbclient/BlobStore.actor.cpp
# fdbclient/BlobStore.h
# fdbclient/HTTP.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbrpc/LoadBalance.actor.h
# fdbrpc/batcher.actor.h
# fdbrpc/fdbrpc.vcxproj
# fdbrpc/sim2.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/masterserver.actor.cpp
2018-11-10 13:04:24 -08:00
a654183f63
Merge pull request #791 from ajbeamon/remove-cluster-from-iclientapi
...
Remove cluster from IClientApi (phase 2 of removing DB names)
2018-11-10 10:16:18 -08:00
45c8f2dfcb
restarting tests will sometimes configure to a fearless configuration on startup if possible
2018-11-02 14:16:47 -07:00
0277dab747
Removed accidental config change.
2018-10-25 04:01:42 -07:00
32b43cc02b
Added 'simple' flag in simulation config generation. Defaults to false, must be changed in code.
2018-10-25 01:25:41 -07:00
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
3922e477a5
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/ManagementAPI.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/LogSystemDiskQueueAdapter.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
2018-10-03 16:57:18 -07:00
c831051474
This removes the idea of clusters from IClientApi.
2018-09-21 15:58:14 -07:00
270b1b24a6
fix: we have to use durableKnownCommittedVersion, because the is the true lower bound on the recovery version of the remote logs
...
fixed a compiler error
2018-09-18 16:29:03 -07:00
200e65fe61
added a workload which tests killing an entire region, and recovering from the failure with data loss.
...
fix: we cannot pop the txs tag from remote logs until they have a full copy of the txnStateStore
fix: we have to modify all of history, we cannot stop after finding a local remote
2018-09-17 18:32:39 -07:00
2a97139d5d
This is the first step in eliminating the usage of database names in our code. The C API remains the same, but underneath that all usage of database names is eliminated.
2018-08-16 10:24:12 -07:00
63b1e85338
Ban `Void _ = wait(...)` constructions, and require just `wait(...)`.
...
There's never any reason to save the value of a Void return, and it's
the easiest source of redefined variable bugs that will creep back in
over time. So just `wait(...)`, it's cleaner that way.
2018-08-14 15:50:26 -07:00
fb31a6999f
Rewrite all files to have #include actorcompiler.h as the last include.
2018-08-14 15:50:26 -07:00
535b5701e5
Rewrite all `Void _ = wait(...)` -> `wait(...)`.
...
This takes advantage of the new actorcompiler functionality to avoid
having duplicate definitions of `Void _` when trying to feed the
un-actorompiled source through clang.
2018-08-14 15:50:26 -07:00
1c29275672
call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details.
2018-08-01 14:30:57 -07:00
f72a9f60c0
only disable fearless if a datacenter has actually been killed
...
fix: we must prevent recovery into the dead datacenter while reducing usable_regions
2018-07-16 10:06:57 -07:00
6c6eaedf09
set allowLogSetKills to false in configurations that could not kill log sets anyways to prevent unnecessary reconfiguration
2018-07-09 22:24:43 -07:00
82cc30be62
added testing for two_satellite_fast and two_satellite_safe
2018-07-09 22:01:46 -07:00
ed7c744a8e
fix: we cannot lower max_read_transaction_life_versions too low when configured with remote logs, because it causes the log routers to fall too far behind
2018-07-09 16:55:33 -07:00
a288d5b9a9
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-28 23:15:32 -07:00
0123627d67
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-06-22 10:43:07 -07:00
8a8914f046
re-added the ability to configure the number of log routers. Many log routers are needed to get a sufficient number of sockets involved in copying data across the WAN
2018-06-22 00:04:00 -07:00
1dce97f28c
Merge branch 'release-5.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/SimulatedCluster.actor.cpp
# packaging/msi/FDBInstaller.wxs
# versions.target
2018-06-21 17:05:11 -07:00
361e335730
Disable cert validation in simulation
2018-06-20 16:26:11 -07:00
1ccfb3a0f4
fix: log_anti_quorum was always 0 in simulation
...
removed durableStorageQuorum, because it is no longer a useful configuration parameter
2018-06-18 10:24:57 -07:00
e8c462882b
re-added remote_logs as a parameter, because it could be useful to have a different number of logs between when recruited as primary and remote
2018-06-18 10:22:34 -07:00
0913368651
added usable_regions to specify if we will replicate into a remote region
...
remote replication defaults to the primary replication
removed remote_logs, because they should be specified as an override in the regions object
2018-06-17 19:31:15 -07:00
3bdbf89f24
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# fdbclient/DatabaseConfiguration.cpp
# fdbserver/SimulatedCluster.actor.cpp
2018-06-13 18:35:28 -07:00
889889323e
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there.
...
The cluster controller will not switch to the other data center if remote logs are too far behind.
We will not recruit in DCs with negative priority.
2018-06-13 18:14:14 -07:00
0042294566
MultiDC: Serialized JSON represents DC and Satellites in same array.
...
With a `"satellite": 1` property that differentiates them. "satellite" will
likely also be deprecated in the future, but this is closer to what the final
serialized form will look like.
2018-06-13 17:55:55 -07:00
ac1fc3660d
MultiDC: Regions JSON should represent datacenters as an array.
...
This is so that the serialized format released in 6.0 will already include a
required change, which is to support >1 DC per region.
2018-06-12 16:18:54 -07:00
372ed67497
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# fdbserver/DataDistribution.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/TagPartitionedLogSystem.actor.cpp
2018-06-11 11:34:10 -07:00
64e0260085
fix: assert did not properly handle default constructed policies
2018-06-10 21:51:59 -07:00
2407e3774b
fix: we cannot run with less storage replication than log replication because it breaks recruitment logic
2018-06-10 20:22:58 -07:00
0bc7274d0e
fix: hasSatelliteReplication was set incorrectly
2018-06-10 20:20:41 -07:00
99c9958db7
Some more trace event normalization
2018-06-08 13:57:00 -07:00
e5488419cc
Attempt to normalize trace events:
...
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.
Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.
This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
4677789b38
fix: low latency tests need 4 machines per datacenter to support triple replication after 1 machine has failed
2018-05-07 11:28:25 -07:00
e27531d39e
Merge branch 'master' into feature-remote-logs
2018-04-30 22:55:46 -07:00
10d25927cd
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# fdbserver/DataDistribution.actor.cpp
2018-04-30 22:15:39 -07:00
bc8e6acbe8
Fix the other half of simulation requiring a TLS Plugin.
...
This commit:
1. Restores --tls_plugin as a way to provide the path to the TLS plugin when running in simulation.
2. Removes the TLS Plugin as being required for 5% of tests.
3. Standardizes on 'sslEnabled' as a variable name.
And is a fix/improvement upon commit f7733d1bf7733d1b
2018-04-30 18:26:29 -07:00
883f2318a0
test fearless configurations
2018-04-30 13:17:29 -07:00
69e831d522
Merge remote-tracking branch 'upstream/release-5.2' into merge-release-5.2
2018-04-28 17:44:52 -07:00
c721ab6854
Fix review comments
2018-04-27 13:54:34 -07:00
6bddf8aefa
Upgrade DR from 5.1 to 5.2
2018-04-26 17:24:40 -07:00
f7733d1bd0
Do not require the TLS Plugin for simulation.
...
It appears that explicit calls to TLS-related things had snuck in over time,
which meant that simulation runs that weren't even configured to use SSL still
wanted and required the TLS plugin.
This commit instead threads through the understanding of if any TLS-related
options were provided, and if not, then don't call anything TLS-related so that
we don't require the TLS plugin.
Hopefully this makes life easier for the opensource folk. :)
2018-04-24 16:53:30 -07:00
d46d5487bd
Merge branch 'release-5.2'
2018-04-18 20:46:03 -07:00
57d650062a
merge 5.1 into 5.2
2018-04-18 20:44:31 -07:00
224621be04
fix: extraDB==0 must leave g_simulator.extraDB as null, so that non-DR tests do not attempt to use a DR database
2018-04-18 19:34:35 -07:00
c6229e443c
fix: do not use resolution class when using regions
2018-04-11 21:22:53 -07:00
19762b847d
Merge branch 'release-5.2'
...
# Conflicts:
# fdbserver/DatabaseConfiguration.cpp
# fdbserver/SimulatedCluster.actor.cpp
2018-04-10 17:02:43 -07:00
c1ba16b3c8
Merge branch 'release-5.1' into release-5.2
...
# Conflicts:
# bindings/java/src/test/com/apple/foundationdb/test/AbstractTester.java
# bindings/java/src/test/com/apple/foundationdb/test/VersionstampSmokeTest.java
# bindings/nodejs/lib/fdb.js
# bindings/nodejs/src/Version.h
# bindings/nodejs/tests/tuple_test.js
2018-04-10 16:50:47 -07:00
b0a88001cc
Merge pull request #132 from yichic/support-dr-upgrade-test
...
Support DR upgrade test
2018-04-10 16:30:19 -07:00
d0230d4d13
Support DR upgrade test in 5.1
2018-04-10 15:19:53 -07:00
0136a01c18
Fix "Not enough physical servers available" error due to incorrect server calculation.
2018-04-05 15:13:21 -07:00
4fb2b99341
fix: using only one region still means we need 3 machines per datacenter, the other machines in the other datacenters just won’t be used
2018-03-30 19:26:22 -07:00
b36e08f08f
first version of non-copying recovery. Upgrades are broken, and it has not been tested using fearless configurations yet
2018-03-29 15:12:38 -07:00
82ed956c65
renamed the multi_dc configuration to three_datacenter. The old three_datacenter configuration was not a useful configuration.
2018-03-26 18:31:26 -07:00
5db52ab081
Merge pull request #87 from etschannen/feature-remote-logs
...
Feature remote logs
2018-03-23 12:55:17 -07:00
7c48e1d31c
Update SimulatedCluster.actor.cpp
2018-03-23 12:54:44 -07:00
ede5cab192
Merge pull request #89 from yichic/share-log-mutations-5.2
...
Share log mutations 5.2
2018-03-19 12:01:26 -07:00
d6559b144f
Share log mutations between backups and DRs which have the same backup range
2018-03-19 11:32:50 -07:00
26b93ff920
Share log mutations between backups and DRs which have the same backup range
2018-03-16 18:09:23 -07:00
a42205eb8e
test running with only one region
2018-03-15 15:40:58 -07:00
65b532658f
added support for single region configurations
2018-03-15 10:59:30 -07:00
f2c804e14f
Reverting changes from merge of master into release-5.2 ( b25810711c). Note that we never intend to release master into release-5.2, but if we did we would need to revert this commit.
2018-03-06 10:15:04 -08:00
1194e3a361
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-05 19:27:46 -08:00
470f5c01f3
changed remoteDcId to a vector of ids, to support future configurations where there are multiple remote databases
2018-02-26 17:09:09 -08:00
a67296b373
do not test fearless configurations to merge with master
2018-02-26 13:31:06 -08:00
8e966fdf9c
simulated cluster tests all configurations. Still needs to randomize the remote and satellite replication, along with them number of remote tlogs, log routers, and satellite tlogs
2018-02-26 13:15:44 -08:00
37a6a81634
Merge commit '7f6fc3e039c911cd84b8540f7f799fc38a1c1822' into feature-remote-logs
...
# Conflicts:
# fdbserver/workloads/RestartRecovery.actor.cpp
2018-02-23 12:33:28 -08:00
0bae9880f1
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-21 10:25:11 -08:00
1b5628d2c5
testing a single configured fearless setup in simulated cluster
...
consolidated simulation connection disablers into one call in the tester
automatically reconfigure from a fearless setup in simulation
2018-02-18 12:59:43 -08:00
cb25564d38
simulated cluster supports fearless configurations
...
removed unused simulation variables
run the simulation with only 1 coordinator most of the time, since we protect the coordinator from being killed, and protecting too many things is bad for simulation
2018-02-15 18:32:39 -08:00
ad19d3926b
fix: make sure there are enough machines in each dc to support triple replication for the configure workload
2018-02-14 17:06:22 -08:00
a52ea4eb78
restored 5.1 functionality of simulated cluster. Will test assigned primary and remote data centers. Does not test remote replication or satellite logs
2018-02-10 13:27:51 -08:00
ebd94bb654
removed a separately configurable storage team size for the remote data center, because it did not make sense
...
fix: the master did not monitor for the failure of remote logs
stop merge attempts when a data center is failed
fixed a variety of other problems with data distribution when a data center is failed
2018-02-02 11:46:04 -08:00
5ac4f73978
Merge branch 'release-5.1' into feature-remote-logs
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
# fdbrpc/Locality.h
# fdbrpc/simulator.h
# fdbserver/ApplyMetadataMutation.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/TagPartitionedLogSystem.actor.cpp
# fdbserver/WorkerInterface.h
# fdbserver/masterserver.actor.cpp
# flow/Net2.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-01-05 11:33:42 -08:00
8c51bc4ac4
fixed low latency tests in a way that gives us better test coverage
2017-11-28 18:20:29 -08:00
5a4a5985fd
Merge branch 'release-5.0'
2017-10-30 08:31:23 -07:00
87321f5017
Merge branch 'release-4.6' into release-5.0
2017-10-30 08:31:01 -07:00
7a36fd2134
disabled a variety of simulation tests to get correctness clean
2017-10-19 15:49:54 -07:00
e0be1ef1e0
Merge branch 'release-5.0'
2017-10-16 10:08:11 -07:00
432726ba2d
Merge branch 'release-4.6' into release-5.0
2017-10-16 09:54:21 -07:00
567d663afd
Fix SimulationConfig never generating a custom config.
...
A 0 was changed to a 1 when rewriting code, and `case 0:` was never being hit. :(
Thankfully, it looks like nothing was broken by this in the meantime.
2017-09-18 17:29:36 -07:00
34f987f56d
added a test in simulation which ensures that a recovery after a single failure takes less than 15 seconds
2017-09-15 17:55:01 -07:00
9a0a3b6329
Merge commit '66528becb82d826e81fa644bb378212584ab580e'
2017-08-28 16:47:59 -07:00
44e0df78c5
Added support for tracking roles for simulation workers
...
Fixed the exclusion and inclusion address simulation API and integration within workloads
Added more information within trace events for simulation
2017-08-28 11:25:37 -07:00
300b5a17ed
Merge branch 'release-5.0'
2017-08-25 18:55:33 -07:00
272b4b984c
fix: fixed a rare bug where we do not wait for a file in the process of being deleted to shutdown before rebooting a machine
2017-08-25 10:12:58 -07:00
7b78035365
Have SimulationConfig wrap DatabaseConfiguration to reduce code duplication.
...
This effectively turns initializing SimulationConfig into the equivalent of
building a config string and calling buildConfiguration on it.
2017-08-22 10:13:57 -07:00
9b25c72971
Pull database config and cluster config into one struct.
...
This will allow us to specify custom situations to be chosen more frequently,
and in particular control machines and processes.
2017-08-21 22:35:44 -07:00
9fd5955e92
Merge branch 'master' into removing-old-dc-code
2017-06-26 16:27:10 -07:00
9553458b78
Updated simulation to support managing exclusion and inclusion address
...
Added method for identifying acceptable availability process classes
Extended cluster availability function to ensure coordinators can be auto configured
Fixed availability function to allow protected processes to be considered as dead if not available
Added debug trace events for providing machine state when considering availability
Added trace event for protected coordinators
2017-06-19 16:48:15 -07:00
766dc23e26
fix: do not use TLS in protectedAddresses
2017-06-02 13:52:21 -07:00
98604d33a0
Merge branch 'fix-io-timeout-handling'
...
# Conflicts:
# fdbrpc/AsyncFileKAIO.actor.h
# fdbrpc/sim2.actor.cpp
# fdbserver/KeyValueStoreSQLite.actor.cpp
# fdbserver/optimisttest.actor.cpp
# fdbserver/worker.actor.cpp
# fdbserver/workloads/MachineAttrition.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2017-05-26 18:43:08 -07:00
0b9ed67e12
Fixed support for RemoveServers Workload
...
Added availability functions to simulation
2017-05-26 14:20:11 -07:00
a674cb4ef4
Initial repository commit
2017-05-25 13:48:44 -07:00
mpilman