- Distinguish corrupt/incomplete cases; latter case only refuses uncertain data
- Ensures quorum durability before marking own log faulty
- Refuse unsafe operations more selectively once log marked faulty
- Simplify and merge logic with standard bootstrap
- fix race conditions when updating Task.info (between running and Exclusive methods)
- do not run consequences of actions that fail and do not become durable (e.g. applying to data store/post-applying a transaction that we failed to complete the PreApplied/Applying step for)
Fairness: tasks are grouped by kind of work, and for each group we track work arrival and service via decaying counters. If work for some group(s) begin to be served at a much lower rate than some other group (e.g. because that group has a large backlog, that by simple priority comes first), then we begin processing some fraction of the work by service-ratio rather than priority. This means that we cannot starve request processing because, e.g., a sync point has produced thousands of state save tasks.
Incremental processing: tasks that process many keys may be declared INCR or ASYNC. ASYNC indicates the keys are needed (so should be loaded), but may be used in follow-up work so the task may be executed if the data is not yet loaded. An ASYNC task may be followed by an INCR task that processes keys as they are loaded, with limits to the batch sizes. This limits the latency impact of processing larger transactions such as sync points. INCR tasks may be declared to be processed in either an arbitrary order with no isolation, or “atomically” with its parent task. In the latter case, the complete unit of work appears to execute as a single execution to external observers (blocking progress on unprocessed keys).
Key-level ordering: AccordCacheEntry can inflate a special Queue object that can be used to impose ordering constraints: FIFO for atomic work (once it has been part processed, or submitted as part of a parent task); prioritised for tasks that have a key-level priority to impose (i.e. PreAccept/Accept/Commit/Stable/Apply want to be ordered by TxnId); and unsequenced tasks for those that have no sequencing restrictions.
Additional improvements:
- AccordCacheEntry are now explicitly LOCKED for the duration of their usage, which may span multiple executions for long running INCR tasks.
- ExclusiveExecutor releases its ownership lock immediately, since AccordCacheEntry locks guarantee safety (as they will not be released until the task is cleaned up)
- CassandraThread tracks active and locked AccordExecutor, so that we may safely and more aggressively use executeMaybeImmediately, safe in the knowledge it will never permit cross-executor executions or multiple executor locks to be held.
Also:
- Rename PreLoadContext -> ExecutionContext
- Break out AccordExecutor/AccordTask into separate package
A 32-bit length field read from the wire by CBUtil.readValue (and its
siblings) flowed directly into new byte[length] with no upper bound,
allowing an unauthenticated client to drive the JVM into
OutOfMemoryError: 'Requested array size exceeds VM limit' — and, with
the default -XX:OnOutOfMemoryError=kill -9 %p, terminate the process —
by declaring Integer.MAX_VALUE as the SASL-token length in AUTH_RESPONSE.
Guard the allocation in the single private readRawBytes(ByteBuf, int)
that all int32-length readers funnel through, rejecting lengths that
exceed the buffer's readable bytes with a ProtocolException.
patch by Francisco Guerrero; reviewed by Stefan Miklosovic for CASSANDRA-21521
Use a single one-time scheduled task with two tick-tock buffers to recycle metric IDs after a sufficiently long delay.
Use phantom references for ThreadLocalMetrics cleanup ony if it is needed to reduce the references processing overhead.
patch by Dmitry Konstantinov; reviewed by Benedict Elliott Smith for CASSANDRA-21475
nodetool getendpoints is deprecated. Also, all StorageServiceMBean.getNaturalEndpoints* methods are
deprecated and they call their replica counterparts. The deprecated methods are also not used directly anywhere
in the code (nor tests).
patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-17665
Cassandra resolves pluggable extensions by class name from configuration, schema, and tooling
inputs. These names were loaded with an initializing Class.forName(name) and type-checked only
afterward, so the named class ran its static initializer before its type was confirmed. After this
change such classes will be loaded without initialization, verified against the expected interface or
base class, and initialized only through normal use after validation.
A shared FBUtilities.classForNameWithoutInitialization helper and typed
instanceOrConstruct/construct overloads apply this to the configurable extension points loaded by
class name: the authentication, authorization, role-management, network and
internode-authenticator backends, the partitioner, audit logger, configuration loader, seed provider,
snitch, abstract types, secondary and custom indexes, compaction strategy, compressor, replication
strategy, SASI analyzers, key and cache providers, query handler, storage and stream hooks, tracing,
the JMX authorization proxy, MBeans, the monotonic clock, nodetool Sjk, triggers, the
sstableloader and stress class options, and diagnostic event classes (loaded without initialization
and checked against DiagnosticEvent, preserving the InvalidClassException contract and the existing
package restriction).
Regression tests confirm that an invalid-type load is rejected without initializing the target
class, and that valid implementations still resolve.
Hadoop client integration and hard-coded JDK and internal class probes are left unchanged.
patch by Jeremiah Jordan; reviewed by Stefan Miklosovic for CASSANDRA-21525
Restore deleteCDCRawFiles logic which is present in 6.0 and trunk but probably lost in 5.0 due to merges
Files may be concurrently removed by the CDC management thread, so we tolerate a file that has already been deleted rather than failing the test.
Fix IndexOutOfBoundsException at commitlog.AbstractCommitLogSegmentManager.forceRecycleAll - it is already fixed in 6.0/trunk.
Cleanup test data after run to reduce disk usage.
Patch by Dmitry Konstantinov; reviewed by Michael Semb Wever for CASSANDRA-20091
SEPExecutor.maybeExecuteImmediately() runs a task synchronously on the
calling worker thread, nested within the task the worker is already
running. Such immediate tasks were invisible in system_views.queries, which only exposed each worker's primary running task.
This is common on the coordinator path, where a local read or mutation is executed immediately within the enclosing QUERY task.
Each SEPWorker now also tracks an immediate current task, set around
maybeExecuteImmediately(), and exposes it as an additional
DebuggableTaskRunner, so the queries table reports both the enclosing
task and the immediate one as separate rows.
patch by Dmitry Konstantinov; reviewed by Caleb Rackliffe for CASSANDRA-21471
Patch by Sam Tunnicliffe and Dmitry Konstantinov; reviewed by Sam
Tunnicliffe and Dmitry Konstantinov for CASSANDRA-21384
Co-authored-by: Dmitry Konstantinov <netudima@gmail.com>
patch by Caleb Rackliffe; reviewed by David Capwell for CASSANDRA-21160
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: David Capwell <dcapwell@apache.org>
Benedict Elliott Smith