toKill1 and toKill2 are a random subset of all processes. If simply kill all processes in toKill1 or toKill2,
we may kill too many processes to make the cluster unavailable and stuck.
Similar as what toKill2 were modified if it can cause cluster unavailable,
we should do the same thing for toKill1
Which fixes a compilation error due to a circular dependency between
flow.a and fdbrpc.a. However, this is now done at the cost of newNet2
users have to remember to add Net2FileSystem::stop() as a callback.
This requires the certificate chain to load successfully, otherwise
fdbcli will error out at an earlier point due to Net2 not being able to
configure TLS.
With TLS, a worker (or process) can have a TLS address and non-TLS address.
When a process is created in simulation, the primary address is TLS by default.
The non-TLS one is the TLS address port plus one.
In a connection between two workers, if their primary addresses do not enable
or disable TLS together, one worker will swap its primary address and secondary address
so that the TLS config of the two endpoints can match.
The swap can make the primary address no longer the TLS one that was created
when the process is created. And the swap only happens for worker instead of
process struct in simulation.
This swap can cause worker->address != process->address.
In checkForExtraDataStores actor, we use worker->address to check if a process
is killable and use the process->address to kill the process. The inconsistency
can cause simulation to kill a protected process that is not killable and leads
to simulation failure.
The idea being that we keep around a TLSConfig that the configuration
that the user has provided, and then when we want to intialize an SSL
context, we ask the TLSConfig to load all certificates and return us a
LoadedTLSConfig that is a concrete set of certificate bytes in memory.
initTLS now just takes the in-memory bytes and applies them to the ssl
context.
This is a large refactor to lead up into certificate refeshing, where we
will periodically check for changes to the certificates, and then
re-load them and apply them to a new SSL context.
trackLeakedConnection actor should give server enough time to
close its connection due to idle connection.
The current logic waits for at least 24 seconds to detect and close
an idle connection.
The current trackLeakedConnection actor waits for about 30 seconds
to claim LeakedConnection error.
We increase the delay in trackLeakedConnection actor to avoid
false positive error in simulation test.
Co-authored by: Vishesh Yadav
Meng Xu