Transient Replication and Cheap Quorums

Patch by Blake Eggleston, Benedict Elliott Smith, Marcus Eriksson, Alex Petrov, Ariel Weisberg; Reviewed by Blake Eggleston, Marcus Eriksson, Benedict Elliott Smith, Alex Petrov, Ariel Weisberg for CASSANDRA-14404

Co-authored-by: Blake Eggleston <bdeggleston@gmail.com>
Co-authored-by: Benedict Elliott Smith <benedict@apache.org>
Co-authored-by: Marcus Eriksson <marcuse@apache.org>
Co-authored-by: Alex Petrov <oleksandr.petrov@gmail.com>

CHANGES.txt

@@ -1,4 +1,5 @@
 4.0
+ * Transient Replication and Cheap Quorums (CASSANDRA-14404)
  * Log server-generated timestamp and nowInSeconds used by queries in FQL (CASSANDRA-14675)
  * Add diagnostic events for read repairs (CASSANDRA-14668)
  * Use consistent nowInSeconds and timestamps values within a request (CASSANDRA-14671)

NEWS.txt

@@ -38,6 +38,10 @@ using the provided 'sstableupgrade' tool.
 
 New features
 ------------
+   - *Experimental* support for Transient Replication and Cheap Quorums introduced by CASSANDRA-14404
+     The intended audience for this functionality is expert users of Cassandra who are prepared
+     to validate every aspect of the database for their application and deployment practices. Future
+     releases of Cassandra will make this feature suitable for a wider audience.
    - *Experimental* support for Java 11 has been added. JVM options that differ between or are
      specific for Java 8 and 11 have been moved from jvm.options into jvm8.options and jvm11.options.
      IMPORTANT: Running C* on Java 11 is *experimental* and do it at your own risk.

conf/cassandra.yaml

@@ -1052,6 +1052,10 @@ enable_scripted_user_defined_functions: false
 # Materialized views are considered experimental and are not recommended for production use.
 enable_materialized_views: true
 
+# Enables creation of transiently replicated keyspaces on this node.
+# Transient replication is experimental and is not recommended for production use.
+#enable_transient_replication: true
+
 # The default Windows kernel timer and scheduling resolution is 15.6ms for power conservation.
 # Lowering this value on Windows can provide much tighter latency and better throughput, however
 # some virtualized environments may see a negative performance impact from changing this setting

doc/source/architecture/dynamo.rst

@@ -74,6 +74,35 @@ nodes in each rack, the data load on the smallest rack may be much higher.  Simi
 into a new rack, it will be considered a replica for the entire ring.  For this reason, many operators choose to
 configure all nodes on a single "rack".
 
+.. _transient-replication:
+
+Transient Replication
+~~~~~~~~~~~~~~~~~~~~~
+
+Transient replication allows you to configure a subset of replicas to only replicate data that hasn't been incrementally
+repaired. This allows you to decouple data redundancy from availability. For instance, if you have a keyspace replicated
+at rf 3, and alter it to rf 5 with 2 transient replicas, you go from being able to tolerate one failed replica to being
+able to tolerate two, without corresponding increase in storage usage. This is because 3 nodes will replicate all the data
+for a given token range, and the other 2 will only replicate data that hasn't been incrementally repaired.
+
+To use transient replication, you first need to enable it in ``cassandra.yaml``. Once enabled, both SimpleStrategy and
+NetworkTopologyStrategy can be configured to transiently replicate data. You configure it by specifying replication factor
+as ``<total_replicas>/<transient_replicas`` Both SimpleStrategy and NetworkTopologyStrategy support configuring transient
+replication.
+
+Transiently replicated keyspaces only support tables created with read_repair set to NONE and monotonic reads are not currently supported.
+You also can't use LWT, logged batches, and counters in 4.0. You will possibly never be able to use materialized views
+with transiently replicated keyspaces and probably never be able to use 2i with them.
+
+Transient replication is an experimental feature that may not be ready for production use. The expected audienced is experienced
+users of Cassandra capable of fully validating a deployment of their particular application. That means being able check
+that operations like reads, writes, decommission, remove, rebuild, repair, and replace all work with your queries, data,
+configuration, operational practices, and availability requirements.
+
+It is anticipated that 4.next will support monotonic reads with transient replication as well as LWT, logged batches, and
+counters.
+
+
 Tunable Consistency
 ^^^^^^^^^^^^^^^^^^^
 

doc/source/cql/ddl.rst

@@ -105,6 +105,14 @@ strategy is used. By default, Cassandra support the following ``'class'``:
 Attempting to create a keyspace that already exists will return an error unless the ``IF NOT EXISTS`` option is used. If
 it is used, the statement will be a no-op if the keyspace already exists.
 
+If :ref:`transient replication <transient-replication>` has been enabled, transient replicas can be configured for both
+SimpleStrategy and NetworkTopologyStrategy by defining replication factors in the format ``'<total_replicas>/<transient_replicas>'``
+
+For instance, this keyspace will have 3 replicas in DC1, 1 of which is transient, and 5 replicas in DC2, 2 of which are transient::
+
+    CREATE KEYSPACE some_keysopace
+               WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1' : '3/1'', 'DC2' : '5/2'};
+
 .. _use-statement:
 
 USE
@@ -455,6 +463,9 @@ A table supports the following options:
 | ``speculative_retry``          | *simple* | 99PERCENTILE| :ref:`Speculative retry options                           |
 |                                |          |             | <speculative-retry-options>`.                             |
 +--------------------------------+----------+-------------+-----------------------------------------------------------+
+| ``speculative_write_threshold``| *simple* | 99PERCENTILE| :ref:`Speculative retry options                           |
+|                                |          |             | <speculative-retry-options>`.                             |
++--------------------------------+----------+-------------+-----------------------------------------------------------+
 | ``gc_grace_seconds``           | *simple* | 864000      | Time to wait before garbage collecting tombstones         |
 |                                |          |             | (deletion markers).                                       |
 +--------------------------------+----------+-------------+-----------------------------------------------------------+
@@ -485,7 +496,8 @@ Speculative retry options
 By default, Cassandra read coordinators only query as many replicas as necessary to satisfy
 consistency levels: one for consistency level ``ONE``, a quorum for ``QUORUM``, and so on.
 ``speculative_retry`` determines when coordinators may query additional replicas, which is useful
-when replicas are slow or unresponsive.  The following are legal values (case-insensitive):
+when replicas are slow or unresponsive.  ``speculative_write_threshold`` specifies the threshold at which
+a cheap quorum write will be upgraded to include transient replicas.  The following are legal values (case-insensitive):
 
 ============================ ======================== =============================================================================
  Format                       Example                  Description

pylib/cqlshlib/cql3handling.py

@@ -49,6 +49,7 @@ class Cql3ParsingRuleSet(CqlParsingRuleSet):
         ('max_index_interval', None),
         ('default_time_to_live', None),
         ('speculative_retry', None),
+        ('speculative_write_threshold', None),
         ('memtable_flush_period_in_ms', None),
         ('cdc', None),
         ('read_repair', None),

pylib/cqlshlib/cqlshhandling.py

@@ -112,6 +112,7 @@ cqlsh_consistency_level_syntax_rules = r'''
                      | "SERIAL"
                      | "LOCAL_SERIAL"
                      | "LOCAL_ONE"
+                     | "NODE_LOCAL"
                      ;
 '''
 

pylib/cqlshlib/test/test_cqlsh_completion.py

@@ -594,7 +594,7 @@ class TestCqlshCompletion(CqlshCompletionCase):
                                      'memtable_flush_period_in_ms',
                                      'CLUSTERING',
                                      'COMPACT', 'caching', 'comment',
-                                     'min_index_interval', 'speculative_retry', 'cdc'])
+                                     'min_index_interval', 'speculative_retry', 'speculative_write_threshold', 'cdc'])
         self.trycompletions(prefix + ' new_table (col_a int PRIMARY KEY) WITH ',
                             choices=['bloom_filter_fp_chance', 'compaction',
                                      'compression',
@@ -603,7 +603,7 @@ class TestCqlshCompletion(CqlshCompletionCase):
                                      'memtable_flush_period_in_ms',
                                      'CLUSTERING',
                                      'COMPACT', 'caching', 'comment',
-                                     'min_index_interval', 'speculative_retry', 'cdc'])
+                                     'min_index_interval', 'speculative_retry', 'speculative_write_threshold', 'cdc'])
         self.trycompletions(prefix + ' new_table (col_a int PRIMARY KEY) WITH bloom_filter_fp_chance ',
                             immediate='= ')
         self.trycompletions(prefix + ' new_table (col_a int PRIMARY KEY) WITH bloom_filter_fp_chance = ',
@@ -650,7 +650,7 @@ class TestCqlshCompletion(CqlshCompletionCase):
                                      'memtable_flush_period_in_ms',
                                      'CLUSTERING',
                                      'COMPACT', 'caching', 'comment',
-                                     'min_index_interval', 'speculative_retry', 'cdc'])
+                                     'min_index_interval', 'speculative_retry', 'speculative_write_threshold', 'cdc'])
         self.trycompletions(prefix + " new_table (col_a int PRIMARY KEY) WITH compaction = "
                             + "{'class': 'DateTieredCompactionStrategy', '",
                             choices=['base_time_seconds', 'max_sstable_age_days',

pylib/cqlshlib/test/test_cqlsh_output.py

@@ -622,7 +622,8 @@ class TestCqlshOutput(BaseTestCase):
                 AND max_index_interval = 2048
                 AND memtable_flush_period_in_ms = 0
                 AND min_index_interval = 128
-                AND speculative_retry = '99PERCENTILE';
+                AND speculative_retry = '99PERCENTILE'
+                AND speculative_write_threshold = '99PERCENTILE';
 
         """ % quote_name(get_keyspace()))
 

src/java/org/apache/cassandra/batchlog/BatchlogManager.java

@@ -26,14 +26,20 @@ import java.util.concurrent.*;
 import javax.management.MBeanServer;
 import javax.management.ObjectName;
 import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Predicates;
 import com.google.common.collect.*;
 import com.google.common.util.concurrent.RateLimiter;
+
+import org.apache.cassandra.locator.ReplicaLayout;
+import org.apache.cassandra.locator.EndpointsForToken;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.locator.Replicas;
 import org.apache.cassandra.schema.SchemaConstants;
 import org.apache.cassandra.cql3.UntypedResultSet;
 import org.apache.cassandra.db.*;
@@ -419,7 +425,7 @@ public class BatchlogManager implements BatchlogManagerMBean
                 if (handler != null)
                 {
                     hintedNodes.addAll(handler.undelivered);
-                    HintsService.instance.write(transform(handler.undelivered, StorageService.instance::getHostIdForEndpoint),
+                    HintsService.instance.write(Collections2.transform(handler.undelivered, StorageService.instance::getHostIdForEndpoint),
                                                 Hint.create(undeliveredMutation, writtenAt));
                 }
             }
@@ -449,35 +455,41 @@ public class BatchlogManager implements BatchlogManagerMBean
                                                                                      long writtenAt,
                                                                                      Set<InetAddressAndPort> hintedNodes)
         {
-            Set<InetAddressAndPort> liveEndpoints = new HashSet<>();
             String ks = mutation.getKeyspaceName();
+            Keyspace keyspace = Keyspace.open(ks);
             Token tk = mutation.key().getToken();
 
-            for (InetAddressAndPort endpoint : StorageService.instance.getNaturalAndPendingEndpoints(ks, tk))
+            EndpointsForToken replicas = StorageService.instance.getNaturalAndPendingReplicasForToken(ks, tk);
+            Replicas.temporaryAssertFull(replicas); // TODO in CASSANDRA-14549
+
+            EndpointsForToken.Builder liveReplicasBuilder = EndpointsForToken.builder(tk);
+            for (Replica replica : replicas)
             {
-                if (endpoint.equals(FBUtilities.getBroadcastAddressAndPort()))
+                if (replica.isLocal())
                 {
                     mutation.apply();
                 }
-                else if (FailureDetector.instance.isAlive(endpoint))
+                else if (FailureDetector.instance.isAlive(replica.endpoint()))
                 {
-                    liveEndpoints.add(endpoint); // will try delivering directly instead of writing a hint.
+                    liveReplicasBuilder.add(replica); // will try delivering directly instead of writing a hint.
                 }
                 else
                 {
-                    hintedNodes.add(endpoint);
+                    hintedNodes.add(replica.endpoint());
-                    HintsService.instance.write(StorageService.instance.getHostIdForEndpoint(endpoint),
+                    HintsService.instance.write(StorageService.instance.getHostIdForEndpoint(replica.endpoint()),
                                                 Hint.create(mutation, writtenAt));
                 }
             }
 
-            if (liveEndpoints.isEmpty())
+            EndpointsForToken liveReplicas = liveReplicasBuilder.build();
+            if (liveReplicas.isEmpty())
                 return null;
 
-            ReplayWriteResponseHandler<Mutation> handler = new ReplayWriteResponseHandler<>(liveEndpoints, System.nanoTime());
+            Replicas.temporaryAssertFull(liveReplicas);
+            ReplayWriteResponseHandler<Mutation> handler = new ReplayWriteResponseHandler<>(keyspace, liveReplicas, System.nanoTime());
             MessageOut<Mutation> message = mutation.createMessage();
-            for (InetAddressAndPort endpoint : liveEndpoints)
+            for (Replica replica : liveReplicas)
-                MessagingService.instance().sendRR(message, endpoint, handler, false);
+                MessagingService.instance().sendWriteRR(message, replica, handler, false);
             return handler;
         }
 
@@ -497,16 +509,17 @@ public class BatchlogManager implements BatchlogManagerMBean
         {
             private final Set<InetAddressAndPort> undelivered = Collections.newSetFromMap(new ConcurrentHashMap<>());
 
-            ReplayWriteResponseHandler(Collection<InetAddressAndPort> writeEndpoints, long queryStartNanoTime)
+            ReplayWriteResponseHandler(Keyspace keyspace, EndpointsForToken writeReplicas, long queryStartNanoTime)
             {
-                super(writeEndpoints, Collections.<InetAddressAndPort>emptySet(), null, null, null, WriteType.UNLOGGED_BATCH, queryStartNanoTime);
+                super(ReplicaLayout.forWriteWithDownNodes(keyspace, null, writeReplicas.token(), writeReplicas, EndpointsForToken.empty(writeReplicas.token())),
-                undelivered.addAll(writeEndpoints);
+                      null, WriteType.UNLOGGED_BATCH, queryStartNanoTime);
+                Iterables.addAll(undelivered, writeReplicas.endpoints());
             }
 
             @Override
             protected int totalBlockFor()
             {
-                return this.naturalEndpoints.size();
+                return this.replicaLayout.selected().size();
             }
 
             @Override

src/java/org/apache/cassandra/config/Config.java

@@ -339,6 +339,8 @@ public class Config
 
     public boolean enable_materialized_views = true;
 
+    public boolean enable_transient_replication = false;
+
     /**
      * Optionally disable asynchronous UDF execution.
      * Disabling asynchronous UDF execution also implicitly disables the security-manager!

src/java/org/apache/cassandra/config/DatabaseDescriptor.java

@@ -59,6 +59,7 @@ import org.apache.cassandra.locator.DynamicEndpointSnitch;
 import org.apache.cassandra.locator.EndpointSnitchInfo;
 import org.apache.cassandra.locator.IEndpointSnitch;
 import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.Replica;
 import org.apache.cassandra.locator.SeedProvider;
 import org.apache.cassandra.net.BackPressureStrategy;
 import org.apache.cassandra.net.RateBasedBackPressure;
@@ -122,7 +123,7 @@ public class DatabaseDescriptor
     private static long indexSummaryCapacityInMB;
 
     private static String localDC;
-    private static Comparator<InetAddressAndPort> localComparator;
+    private static Comparator<Replica> localComparator;
     private static EncryptionContext encryptionContext;
     private static boolean hasLoggedConfig;
 
@@ -991,18 +992,14 @@ public class DatabaseDescriptor
         EndpointSnitchInfo.create();
 
         localDC = snitch.getDatacenter(FBUtilities.getBroadcastAddressAndPort());
-        localComparator = new Comparator<InetAddressAndPort>()
+        localComparator = (replica1, replica2) -> {
-        {
+            boolean local1 = localDC.equals(snitch.getDatacenter(replica1));
-            public int compare(InetAddressAndPort endpoint1, InetAddressAndPort endpoint2)
+            boolean local2 = localDC.equals(snitch.getDatacenter(replica2));
-            {
+            if (local1 && !local2)
-                boolean local1 = localDC.equals(snitch.getDatacenter(endpoint1));
+                return -1;
-                boolean local2 = localDC.equals(snitch.getDatacenter(endpoint2));
+            if (local2 && !local1)
-                if (local1 && !local2)
+                return 1;
-                    return -1;
+            return 0;
-                if (local2 && !local1)
-                    return 1;
-                return 0;
-            }
         };
     }
 
@@ -2308,7 +2305,7 @@ public class DatabaseDescriptor
         return localDC;
     }
 
-    public static Comparator<InetAddressAndPort> getLocalComparator()
+    public static Comparator<Replica> getLocalComparator()
     {
         return localComparator;
     }
@@ -2459,6 +2456,16 @@ public class DatabaseDescriptor
         return conf.enable_materialized_views;
     }
 
+    public static boolean isTransientReplicationEnabled()
+    {
+        return conf.enable_transient_replication;
+    }
+
+    public static void setTransientReplicationEnabledUnsafe(boolean enabled)
+    {
+        conf.enable_transient_replication = enabled;
+    }
+
     public static long getUserDefinedFunctionFailTimeout()
     {
         return conf.user_defined_function_fail_timeout;

src/java/org/apache/cassandra/cql3/QueryProcessor.java

@@ -202,7 +202,18 @@ public class QueryProcessor implements QueryHandler
         statement.authorize(clientState);
         statement.validate(clientState);
 
-        ResultMessage result = statement.execute(queryState, options, queryStartNanoTime);
+        ResultMessage result;
+        if (options.getConsistency() == ConsistencyLevel.NODE_LOCAL)
+        {
+            assert Boolean.getBoolean("cassandra.enable_nodelocal_queries") : "Node local consistency level is highly dangerous and should be used only for debugging purposes";
+            assert statement instanceof SelectStatement : "Only SELECT statements are permitted for node-local execution";
+            logger.info("Statement {} executed with NODE_LOCAL consistency level.", statement);
+            result = statement.executeLocally(queryState, options);
+        }
+        else
+        {
+            result = statement.execute(queryState, options, queryStartNanoTime);
+        }
         return result == null ? new ResultMessage.Void() : result;
     }
 

src/java/org/apache/cassandra/cql3/statements/BatchStatement.java

@@ -261,7 +261,7 @@ public class BatchStatement implements CQLStatement
         return statements;
     }
 
-    private Collection<? extends IMutation> getMutations(BatchQueryOptions options,
+    private List<? extends IMutation> getMutations(BatchQueryOptions options,
                                                          boolean local,
                                                          long batchTimestamp,
                                                          int nowInSeconds,
@@ -401,7 +401,7 @@ public class BatchStatement implements CQLStatement
         return new ResultMessage.Void();
     }
 
-    private void executeWithoutConditions(Collection<? extends IMutation> mutations, ConsistencyLevel cl, long queryStartNanoTime) throws RequestExecutionException, RequestValidationException
+    private void executeWithoutConditions(List<? extends IMutation> mutations, ConsistencyLevel cl, long queryStartNanoTime) throws RequestExecutionException, RequestValidationException
     {
         if (mutations.isEmpty())
             return;

src/java/org/apache/cassandra/cql3/statements/BatchUpdatesCollector.java

@@ -104,7 +104,7 @@ final class BatchUpdatesCollector implements UpdatesCollector
      * Returns a collection containing all the mutations.
      * @return a collection containing all the mutations.
      */
-    public Collection<IMutation> toMutations()
+    public List<IMutation> toMutations()
     {
         //TODO: The case where all statement where on the same keyspace is pretty common, optimize for that?
         List<IMutation> ms = new ArrayList<>();

src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java

@@ -465,7 +465,7 @@ public abstract class ModificationStatement implements CQLStatement
         else
             cl.validateForWrite(metadata.keyspace);
 
-        Collection<? extends IMutation> mutations =
+        List<? extends IMutation> mutations =
             getMutations(options,
                          false,
                          options.getTimestamp(queryState),
@@ -676,7 +676,7 @@ public abstract class ModificationStatement implements CQLStatement
      *
      * @return list of the mutations
      */
-    private Collection<? extends IMutation> getMutations(QueryOptions options,
+    private List<? extends IMutation> getMutations(QueryOptions options,
                                                          boolean local,
                                                          long timestamp,
                                                          int nowInSeconds,

src/java/org/apache/cassandra/cql3/statements/SingleTableUpdatesCollector.java

@@ -82,7 +82,7 @@ final class SingleTableUpdatesCollector implements UpdatesCollector
      * Returns a collection containing all the mutations.
      * @return a collection containing all the mutations.
      */
-    public Collection<IMutation> toMutations()
+    public List<IMutation> toMutations()
     {
         List<IMutation> ms = new ArrayList<>();
         for (PartitionUpdate.Builder builder : puBuilders.values())

src/java/org/apache/cassandra/cql3/statements/UpdatesCollector.java

@@ -18,17 +18,16 @@
 
 package org.apache.cassandra.cql3.statements;
 
-import java.util.Collection;
+import java.util.List;
 
 import org.apache.cassandra.db.ConsistencyLevel;
 import org.apache.cassandra.db.DecoratedKey;
 import org.apache.cassandra.db.IMutation;
-import org.apache.cassandra.db.Keyspace;
 import org.apache.cassandra.db.partitions.PartitionUpdate;
 import org.apache.cassandra.schema.TableMetadata;
 
 public interface UpdatesCollector
 {
     PartitionUpdate.Builder getPartitionUpdateBuilder(TableMetadata metadata, DecoratedKey dk, ConsistencyLevel consistency);
-    Collection<IMutation> toMutations();
+    List<IMutation> toMutations();
 }

src/java/org/apache/cassandra/cql3/statements/schema/AlterKeyspaceStatement.java

@@ -17,16 +17,27 @@
  */
 package org.apache.cassandra.cql3.statements.schema;
 
+import java.util.List;
 import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
 
 import com.google.common.collect.ImmutableSet;
 
 import org.apache.cassandra.audit.AuditLogContext;
 import org.apache.cassandra.audit.AuditLogEntryType;
 import org.apache.cassandra.auth.Permission;
+import org.apache.cassandra.config.Config;
+import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.cql3.CQLStatement;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.exceptions.ConfigurationException;
+import org.apache.cassandra.gms.Gossiper;
 import org.apache.cassandra.locator.AbstractReplicationStrategy;
+import org.apache.cassandra.locator.InetAddressAndPort;
 import org.apache.cassandra.locator.LocalStrategy;
+import org.apache.cassandra.locator.ReplicationFactor;
 import org.apache.cassandra.schema.KeyspaceMetadata;
 import org.apache.cassandra.schema.KeyspaceMetadata.KeyspaceDiff;
 import org.apache.cassandra.schema.Keyspaces;
@@ -34,9 +45,13 @@ import org.apache.cassandra.schema.Keyspaces.KeyspacesDiff;
 import org.apache.cassandra.service.ClientState;
 import org.apache.cassandra.transport.Event.SchemaChange;
 import org.apache.cassandra.transport.Event.SchemaChange.Change;
+import org.apache.cassandra.utils.FBUtilities;
 
 public final class AlterKeyspaceStatement extends AlterSchemaStatement
 {
+    private static final boolean allow_alter_rf_during_range_movement = Boolean.getBoolean(Config.PROPERTY_PREFIX + "allow_alter_rf_during_range_movement");
+    private static final boolean allow_unsafe_transient_changes = Boolean.getBoolean(Config.PROPERTY_PREFIX + "allow_unsafe_transient_changes");
+
     private final KeyspaceAttributes attrs;
 
     public AlterKeyspaceStatement(String keyspaceName, KeyspaceAttributes attrs)
@@ -60,6 +75,9 @@ public final class AlterKeyspaceStatement extends AlterSchemaStatement
 
         newKeyspace.params.validate(keyspaceName);
 
+        validateNoRangeMovements();
+        validateTransientReplication(keyspace.createReplicationStrategy(), newKeyspace.createReplicationStrategy());
+
         return schema.withAddedOrUpdated(newKeyspace);
     }
 
@@ -84,11 +102,77 @@ public final class AlterKeyspaceStatement extends AlterSchemaStatement
         AbstractReplicationStrategy before = keyspaceDiff.before.createReplicationStrategy();
         AbstractReplicationStrategy after = keyspaceDiff.after.createReplicationStrategy();
 
-        return before.getReplicationFactor() < after.getReplicationFactor()
+        return before.getReplicationFactor().fullReplicas < after.getReplicationFactor().fullReplicas
              ? ImmutableSet.of("When increasing replication factor you need to run a full (-full) repair to distribute the data.")
              : ImmutableSet.of();
     }
 
+    private void validateNoRangeMovements()
+    {
+        if (allow_alter_rf_during_range_movement)
+            return;
+
+        Stream<InetAddressAndPort> endpoints = Stream.concat(Gossiper.instance.getLiveMembers().stream(), Gossiper.instance.getUnreachableMembers().stream());
+        List<InetAddressAndPort> notNormalEndpoints = endpoints.filter(endpoint -> !FBUtilities.getBroadcastAddressAndPort().equals(endpoint) &&
+                                                                                   !Gossiper.instance.getEndpointStateForEndpoint(endpoint).isNormalState())
+                                                               .collect(Collectors.toList());
+        if (!notNormalEndpoints.isEmpty())
+        {
+            throw new ConfigurationException("Cannot alter RF while some endpoints are not in normal state (no range movements): " + notNormalEndpoints);
+        }
+    }
+
+    private void validateTransientReplication(AbstractReplicationStrategy oldStrategy, AbstractReplicationStrategy newStrategy)
+    {
+        //If there is no read traffic there are some extra alterations you can safely make, but this is so atypical
+        //that a good default is to not allow unsafe changes
+        if (allow_unsafe_transient_changes)
+            return;
+
+        ReplicationFactor oldRF = oldStrategy.getReplicationFactor();
+        ReplicationFactor newRF = newStrategy.getReplicationFactor();
+
+        int oldTrans = oldRF.transientReplicas();
+        int oldFull = oldRF.fullReplicas;
+        int newTrans = newRF.transientReplicas();
+        int newFull = newRF.fullReplicas;
+
+        if (newTrans > 0)
+        {
+            if (DatabaseDescriptor.getNumTokens() > 1)
+                throw new ConfigurationException(String.format("Transient replication is not supported with vnodes yet"));
+
+            Keyspace ks = Keyspace.open(keyspaceName);
+            for (ColumnFamilyStore cfs : ks.getColumnFamilyStores())
+            {
+                if (cfs.viewManager.hasViews())
+                {
+                    throw new ConfigurationException("Cannot use transient replication on keyspaces using materialized views");
+                }
+
+                if (cfs.indexManager.hasIndexes())
+                {
+                    throw new ConfigurationException("Cannot use transient replication on keyspaces using secondary indexes");
+                }
+            }
+        }
+
+        //This is true right now because the transition from transient -> full lacks the pending state
+        //necessary for correctness. What would happen if we allowed this is that we would attempt
+        //to read from a transient replica as if it were a full replica.
+        if (oldFull > newFull && oldTrans > 0)
+            throw new ConfigurationException("Can't add full replicas if there are any transient replicas. You must first remove all transient replicas, then change the # of full replicas, then add back the transient replicas");
+
+        //Don't increase transient replication factor by more than one at a time if changing number of replicas
+        //Just like with changing full replicas it's not safe to do this as you could read from too many replicas
+        //that don't have the necessary data. W/O transient replication this alteration was allowed and it's not clear
+        //if it should be.
+        //This is structured so you can convert as many full replicas to transient replicas as you want.
+        boolean numReplicasChanged = oldTrans + oldFull != newTrans + newFull;
+        if (numReplicasChanged && (newTrans > oldTrans && newTrans != oldTrans + 1))
+            throw new ConfigurationException("Can only safely increase number of transients one at a time with incremental repair run in between each time");
+    }
+
     @Override
     public AuditLogContext getAuditLogContext()
     {

src/java/org/apache/cassandra/cql3/statements/schema/AlterTableStatement.java

@@ -28,6 +28,7 @@ import org.apache.cassandra.db.marshal.AbstractType;
 import org.apache.cassandra.schema.*;
 import org.apache.cassandra.schema.Keyspaces.KeyspacesDiff;
 import org.apache.cassandra.service.ClientState;
+import org.apache.cassandra.service.reads.repair.ReadRepairStrategy;
 import org.apache.cassandra.transport.Event.SchemaChange;
 import org.apache.cassandra.transport.Event.SchemaChange.Change;
 import org.apache.cassandra.transport.Event.SchemaChange.Target;
@@ -360,6 +361,12 @@ public abstract class AlterTableStatement extends AlterSchemaStatement
                           "before being replayed.");
             }
 
+            if (keyspace.createReplicationStrategy().hasTransientReplicas()
+                && params.readRepair != ReadRepairStrategy.NONE)
+            {
+                throw ire("read_repair must be set to 'NONE' for transiently replicated keyspaces");
+            }
+
             return keyspace.withSwapped(keyspace.tables.withSwapped(table.withSwapped(params)));
         }
     }

src/java/org/apache/cassandra/cql3/statements/schema/CreateIndexStatement.java

@@ -28,7 +28,9 @@ import org.apache.cassandra.cql3.CQLStatement;
 import org.apache.cassandra.cql3.ColumnIdentifier;
 import org.apache.cassandra.cql3.QualifiedName;
 import org.apache.cassandra.cql3.statements.schema.IndexTarget.Type;
+import org.apache.cassandra.db.Keyspace;
 import org.apache.cassandra.db.marshal.MapType;
+import org.apache.cassandra.exceptions.InvalidRequestException;
 import org.apache.cassandra.schema.*;
 import org.apache.cassandra.schema.Keyspaces.KeyspacesDiff;
 import org.apache.cassandra.service.ClientState;
@@ -88,6 +90,9 @@ public final class CreateIndexStatement extends AlterSchemaStatement
         if (table.isView())
             throw ire("Secondary indexes on materialized views aren't supported");
 
+        if (Keyspace.open(table.keyspace).getReplicationStrategy().hasTransientReplicas())
+            throw new InvalidRequestException("Secondary indexes are not supported on transiently replicated keyspaces");
+
         List<IndexTarget> indexTargets = Lists.newArrayList(transform(rawIndexTargets, t -> t.prepare(table)));
 
         if (indexTargets.isEmpty() && !attrs.isCustom)

src/java/org/apache/cassandra/cql3/statements/schema/CreateTableStatement.java

@@ -27,11 +27,14 @@ import org.apache.cassandra.auth.DataResource;
 import org.apache.cassandra.auth.IResource;
 import org.apache.cassandra.auth.Permission;
 import org.apache.cassandra.cql3.*;
+import org.apache.cassandra.db.Keyspace;
 import org.apache.cassandra.db.marshal.*;
 import org.apache.cassandra.exceptions.AlreadyExistsException;
+import org.apache.cassandra.exceptions.ConfigurationException;
 import org.apache.cassandra.schema.*;
 import org.apache.cassandra.schema.Keyspaces.KeyspacesDiff;
 import org.apache.cassandra.service.ClientState;
+import org.apache.cassandra.service.reads.repair.ReadRepairStrategy;
 import org.apache.cassandra.transport.Event.SchemaChange;
 import org.apache.cassandra.transport.Event.SchemaChange.Change;
 import org.apache.cassandra.transport.Event.SchemaChange.Target;
@@ -98,6 +101,12 @@ public final class CreateTableStatement extends AlterSchemaStatement
         TableMetadata table = builder(keyspace.types).build();
         table.validate();
 
+        if (keyspace.createReplicationStrategy().hasTransientReplicas()
+            && table.params.readRepair != ReadRepairStrategy.NONE)
+        {
+            throw ire("read_repair must be set to 'NONE' for transiently replicated keyspaces");
+        }
+
         return schema.withAddedOrUpdated(keyspace.withSwapped(keyspace.tables.with(table)));
     }
 

src/java/org/apache/cassandra/cql3/statements/schema/CreateViewStatement.java

@@ -31,9 +31,11 @@ import org.apache.cassandra.cql3.restrictions.StatementRestrictions;
 import org.apache.cassandra.cql3.selection.RawSelector;
 import org.apache.cassandra.cql3.selection.Selectable;
 import org.apache.cassandra.cql3.statements.StatementType;
+import org.apache.cassandra.db.Keyspace;
 import org.apache.cassandra.db.marshal.AbstractType;
 import org.apache.cassandra.db.marshal.ReversedType;
 import org.apache.cassandra.exceptions.AlreadyExistsException;
+import org.apache.cassandra.exceptions.InvalidRequestException;
 import org.apache.cassandra.schema.*;
 import org.apache.cassandra.schema.Keyspaces.KeyspacesDiff;
 import org.apache.cassandra.service.ClientState;
@@ -107,6 +109,9 @@ public final class CreateViewStatement extends AlterSchemaStatement
         if (null == keyspace)
             throw ire("Keyspace '%s' doesn't exist", keyspaceName);
 
+        if (keyspace.createReplicationStrategy().hasTransientReplicas())
+            throw new InvalidRequestException("Materialized views are not supported on transiently replicated keyspaces");
+
         TableMetadata table = keyspace.tables.getNullable(tableName);
         if (null == table)
             throw ire("Base table '%s' doesn't exist", tableName);

src/java/org/apache/cassandra/cql3/statements/schema/TableAttributes.java

@@ -128,6 +128,9 @@ public final class TableAttributes extends PropertyDefinitions
         if (hasOption(Option.SPECULATIVE_RETRY))
             builder.speculativeRetry(SpeculativeRetryPolicy.fromString(getString(Option.SPECULATIVE_RETRY)));
 
+        if (hasOption(Option.SPECULATIVE_WRITE_THRESHOLD))
+            builder.speculativeWriteThreshold(SpeculativeRetryPolicy.fromString(getString(Option.SPECULATIVE_WRITE_THRESHOLD)));
+
         if (hasOption(Option.CRC_CHECK_CHANCE))
             builder.crcCheckChance(getDouble(Option.CRC_CHECK_CHANCE));
 

src/java/org/apache/cassandra/db/ColumnFamilyStore.java

@@ -30,7 +30,6 @@ import java.util.concurrent.*;
 import java.util.concurrent.atomic.AtomicInteger;
 import java.util.concurrent.atomic.AtomicReference;
 import java.util.regex.Pattern;
-import javax.annotation.Nullable;
 import javax.management.*;
 import javax.management.openmbean.*;
 
@@ -68,7 +67,6 @@ import org.apache.cassandra.io.FSReadError;
 import org.apache.cassandra.io.FSWriteError;
 import org.apache.cassandra.io.sstable.Component;
 import org.apache.cassandra.io.sstable.Descriptor;
-import org.apache.cassandra.io.sstable.KeyIterator;
 import org.apache.cassandra.io.sstable.SSTableMultiWriter;
 import org.apache.cassandra.io.sstable.format.*;
 import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
@@ -81,7 +79,6 @@ import org.apache.cassandra.metrics.TableMetrics;
 import org.apache.cassandra.repair.TableRepairManager;
 import org.apache.cassandra.schema.*;
 import org.apache.cassandra.schema.CompactionParams.TombstoneOption;
-import org.apache.cassandra.service.ActiveRepairService;
 import org.apache.cassandra.service.CacheService;
 import org.apache.cassandra.service.StorageService;
 import org.apache.cassandra.streaming.TableStreamManager;
@@ -205,7 +202,8 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
     private final Directories directories;
 
     public final TableMetrics metric;
-    public volatile long sampleLatencyNanos;
+    public volatile long sampleReadLatencyNanos;
+    public volatile long transientWriteLatencyNanos;
 
     private final CassandraTableWriteHandler writeHandler;
     private final CassandraStreamManager streamManager;
@@ -384,7 +382,8 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
         viewManager = keyspace.viewManager.forTable(metadata.id);
         metric = new TableMetrics(this);
         fileIndexGenerator.set(generation);
-        sampleLatencyNanos = TimeUnit.MILLISECONDS.toNanos(DatabaseDescriptor.getReadRpcTimeout() / 2);
+        sampleReadLatencyNanos = TimeUnit.MILLISECONDS.toNanos(DatabaseDescriptor.getReadRpcTimeout() / 2);
+        transientWriteLatencyNanos = TimeUnit.MILLISECONDS.toNanos(DatabaseDescriptor.getWriteRpcTimeout() / 2);
 
         logger.info("Initializing {}.{}", keyspace.getName(), name);
 
@@ -454,7 +453,8 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
     {
         try
         {
-            sampleLatencyNanos = metadata().params.speculativeRetry.calculateThreshold(metric.coordinatorReadLatency);
+            sampleReadLatencyNanos = metadata().params.speculativeRetry.calculateThreshold(metric.coordinatorReadLatency);
+            transientWriteLatencyNanos = metadata().params.speculativeWriteThreshold.calculateThreshold(metric.coordinatorWriteLatency);
         }
         catch (Throwable e)
         {
@@ -487,15 +487,15 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
         return directories;
     }
 
-    public SSTableMultiWriter createSSTableMultiWriter(Descriptor descriptor, long keyCount, long repairedAt, UUID pendingRepair, int sstableLevel, SerializationHeader header, LifecycleTransaction txn)
+    public SSTableMultiWriter createSSTableMultiWriter(Descriptor descriptor, long keyCount, long repairedAt, UUID pendingRepair, boolean isTransient, int sstableLevel, SerializationHeader header, LifecycleTransaction txn)
     {
         MetadataCollector collector = new MetadataCollector(metadata().comparator).sstableLevel(sstableLevel);
-        return createSSTableMultiWriter(descriptor, keyCount, repairedAt, pendingRepair, collector, header, txn);
+        return createSSTableMultiWriter(descriptor, keyCount, repairedAt, pendingRepair, isTransient, collector, header, txn);
     }
 
-    public SSTableMultiWriter createSSTableMultiWriter(Descriptor descriptor, long keyCount, long repairedAt, UUID pendingRepair, MetadataCollector metadataCollector, SerializationHeader header, LifecycleTransaction txn)
+    public SSTableMultiWriter createSSTableMultiWriter(Descriptor descriptor, long keyCount, long repairedAt, UUID pendingRepair, boolean isTransient, MetadataCollector metadataCollector, SerializationHeader header, LifecycleTransaction txn)
     {
-        return getCompactionStrategyManager().createSSTableMultiWriter(descriptor, keyCount, repairedAt, pendingRepair, metadataCollector, header, indexManager.listIndexes(), txn);
+        return getCompactionStrategyManager().createSSTableMultiWriter(descriptor, keyCount, repairedAt, pendingRepair, isTransient, metadataCollector, header, indexManager.listIndexes(), txn);
     }
 
     public boolean supportsEarlyOpen()
@@ -1402,7 +1402,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
 
         // cleanup size estimation only counts bytes for keys local to this node
         long expectedFileSize = 0;
-        Collection<Range<Token>> ranges = StorageService.instance.getLocalRanges(keyspace.getName());
+        Collection<Range<Token>> ranges = StorageService.instance.getLocalReplicas(keyspace.getName()).ranges();
         for (SSTableReader sstable : sstables)
         {
             List<SSTableReader.PartitionPositionBounds> positions = sstable.getPositionsForRanges(ranges);
@@ -1677,7 +1677,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
 
     public void cleanupCache()
     {
-        Collection<Range<Token>> ranges = StorageService.instance.getLocalRanges(keyspace.getName());
+        Collection<Range<Token>> ranges = StorageService.instance.getLocalReplicas(keyspace.getName()).ranges();
 
         for (Iterator<RowCacheKey> keyIter = CacheService.instance.rowCache.keyIterator();
              keyIter.hasNext(); )

src/java/org/apache/cassandra/db/ConsistencyLevel.java

@@ -17,16 +17,18 @@
  */
 package org.apache.cassandra.db;
 
-import java.util.ArrayList;
 import java.util.HashMap;
-import java.util.List;
 import java.util.Map;
 
 import com.google.common.collect.Iterables;
+import org.apache.cassandra.locator.Endpoints;
+import org.apache.cassandra.locator.ReplicaCollection;
+import org.apache.cassandra.locator.Replicas;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.Replica;
 import org.apache.cassandra.schema.TableMetadata;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.exceptions.InvalidRequestException;
@@ -47,7 +49,8 @@ public enum ConsistencyLevel
     EACH_QUORUM (7),
     SERIAL      (8),
     LOCAL_SERIAL(9),
-    LOCAL_ONE   (10, true);
+    LOCAL_ONE   (10, true),
+    NODE_LOCAL  (11, true);
 
     private static final Logger logger = LoggerFactory.getLogger(ConsistencyLevel.class);
 
@@ -89,13 +92,13 @@ public enum ConsistencyLevel
 
     private int quorumFor(Keyspace keyspace)
     {
-        return (keyspace.getReplicationStrategy().getReplicationFactor() / 2) + 1;
+        return (keyspace.getReplicationStrategy().getReplicationFactor().allReplicas / 2) + 1;
     }
 
     private int localQuorumFor(Keyspace keyspace, String dc)
     {
         return (keyspace.getReplicationStrategy() instanceof NetworkTopologyStrategy)
-             ? (((NetworkTopologyStrategy) keyspace.getReplicationStrategy()).getReplicationFactor(dc) / 2) + 1
+             ? (((NetworkTopologyStrategy) keyspace.getReplicationStrategy()).getReplicationFactor(dc).allReplicas / 2) + 1
              : quorumFor(keyspace);
     }
 
@@ -116,7 +119,7 @@ public enum ConsistencyLevel
             case SERIAL:
                 return quorumFor(keyspace);
             case ALL:
-                return keyspace.getReplicationStrategy().getReplicationFactor();
+                return keyspace.getReplicationStrategy().getReplicationFactor().allReplicas;
             case LOCAL_QUORUM:
             case LOCAL_SERIAL:
                 return localQuorumFor(keyspace, DatabaseDescriptor.getLocalDataCenter());
@@ -138,6 +141,28 @@ public enum ConsistencyLevel
         }
     }
 
+    public int blockForWrite(Keyspace keyspace, Endpoints<?> pending)
+    {
+        assert pending != null;
+
+        int blockFor = blockFor(keyspace);
+        switch (this)
+        {
+            case ANY:
+                break;
+            case LOCAL_ONE: case LOCAL_QUORUM: case LOCAL_SERIAL:
+                // we will only count local replicas towards our response count, as these queries only care about local guarantees
+                blockFor += countDCLocalReplicas(pending).allReplicas();
+                break;
+            case ONE: case TWO: case THREE:
+            case QUORUM: case EACH_QUORUM:
+            case SERIAL:
+            case ALL:
+                blockFor += pending.size();
+        }
+        return blockFor;
+    }
+
     /**
      * Determine if this consistency level meets or exceeds the consistency requirements of the given cl for the given keyspace
      */
@@ -156,40 +181,75 @@ public enum ConsistencyLevel
         return DatabaseDescriptor.getLocalDataCenter().equals(DatabaseDescriptor.getEndpointSnitch().getDatacenter(endpoint));
     }
 
-    public int countLocalEndpoints(Iterable<InetAddressAndPort> liveEndpoints)
+    public static boolean isLocal(Replica replica)
     {
-        int count = 0;
+        return isLocal(replica.endpoint());
-        for (InetAddressAndPort endpoint : liveEndpoints)
+    }
-            if (isLocal(endpoint))
+
-                count++;
+    private static ReplicaCount countDCLocalReplicas(ReplicaCollection<?> liveReplicas)
+    {
+        ReplicaCount count = new ReplicaCount();
+        for (Replica replica : liveReplicas)
+            if (isLocal(replica))
+                count.increment(replica);
         return count;
     }
 
-    private Map<String, Integer> countPerDCEndpoints(Keyspace keyspace, Iterable<InetAddressAndPort> liveEndpoints)
+    private static class ReplicaCount
+    {
+        int fullReplicas;
+        int transientReplicas;
+
+        int allReplicas()
+        {
+            return fullReplicas + transientReplicas;
+        }
+
+        void increment(Replica replica)
+        {
+            if (replica.isFull()) ++fullReplicas;
+            else ++transientReplicas;
+        }
+
+        boolean isSufficient(int allReplicas, int fullReplicas)
+        {
+            return this.fullReplicas >= fullReplicas
+                    && this.allReplicas() >= allReplicas;
+        }
+    }
+
+    private static Map<String, ReplicaCount> countPerDCEndpoints(Keyspace keyspace, Iterable<Replica> liveReplicas)
     {
         NetworkTopologyStrategy strategy = (NetworkTopologyStrategy) keyspace.getReplicationStrategy();
 
-        Map<String, Integer> dcEndpoints = new HashMap<String, Integer>();
+        Map<String, ReplicaCount> dcEndpoints = new HashMap<>();
         for (String dc: strategy.getDatacenters())
-            dcEndpoints.put(dc, 0);
+            dcEndpoints.put(dc, new ReplicaCount());
 
-        for (InetAddressAndPort endpoint : liveEndpoints)
+        for (Replica replica : liveReplicas)
         {
-            String dc = DatabaseDescriptor.getEndpointSnitch().getDatacenter(endpoint);
+            String dc = DatabaseDescriptor.getEndpointSnitch().getDatacenter(replica);
-            dcEndpoints.put(dc, dcEndpoints.get(dc) + 1);
+            dcEndpoints.get(dc).increment(replica);
         }
         return dcEndpoints;
     }
 
-    public List<InetAddressAndPort> filterForQuery(Keyspace keyspace, List<InetAddressAndPort> liveEndpoints)
+    public <E extends Endpoints<E>> E filterForQuery(Keyspace keyspace, E liveReplicas)
+    {
+        return filterForQuery(keyspace, liveReplicas, false);
+    }
+
+    public <E extends Endpoints<E>> E filterForQuery(Keyspace keyspace, E liveReplicas, boolean alwaysSpeculate)
     {
         /*
          * If we are doing an each quorum query, we have to make sure that the endpoints we select
          * provide a quorum for each data center. If we are not using a NetworkTopologyStrategy,
          * we should fall through and grab a quorum in the replication strategy.
+         *
+         * We do not speculate for EACH_QUORUM.
          */
         if (this == EACH_QUORUM && keyspace.getReplicationStrategy() instanceof NetworkTopologyStrategy)
-            return filterForEachQuorum(keyspace, liveEndpoints);
+            return filterForEachQuorum(keyspace, liveReplicas);
 
         /*
          * Endpoints are expected to be restricted to live replicas, sorted by snitch preference.
@@ -198,36 +258,34 @@ public enum ConsistencyLevel
          * the blockFor first ones).
          */
         if (isDCLocal)
-            liveEndpoints.sort(DatabaseDescriptor.getLocalComparator());
+            liveReplicas = liveReplicas.sorted(DatabaseDescriptor.getLocalComparator());
 
-        return liveEndpoints.subList(0, Math.min(liveEndpoints.size(), blockFor(keyspace)));
+        return liveReplicas.subList(0, Math.min(liveReplicas.size(), blockFor(keyspace) + (alwaysSpeculate ? 1 : 0)));
     }
 
-    private List<InetAddressAndPort> filterForEachQuorum(Keyspace keyspace, List<InetAddressAndPort> liveEndpoints)
+    private <E extends Endpoints<E>> E filterForEachQuorum(Keyspace keyspace, E liveReplicas)
     {
         NetworkTopologyStrategy strategy = (NetworkTopologyStrategy) keyspace.getReplicationStrategy();
-
+        Map<String, Integer> dcsReplicas = new HashMap<>();
-        Map<String, List<InetAddressAndPort>> dcsEndpoints = new HashMap<>();
+        for (String dc : strategy.getDatacenters())
-        for (String dc: strategy.getDatacenters())
-            dcsEndpoints.put(dc, new ArrayList<>());
-
-        for (InetAddressAndPort add : liveEndpoints)
         {
-            String dc = DatabaseDescriptor.getEndpointSnitch().getDatacenter(add);
+            // we put _up to_ dc replicas only
-            dcsEndpoints.get(dc).add(add);
+            dcsReplicas.put(dc, localQuorumFor(keyspace, dc));
         }
 
-        List<InetAddressAndPort> waitSet = new ArrayList<>();
+        return liveReplicas.filter((replica) -> {
-        for (Map.Entry<String, List<InetAddressAndPort>> dcEndpoints : dcsEndpoints.entrySet())
+            String dc = DatabaseDescriptor.getEndpointSnitch().getDatacenter(replica);
-        {
+            int replicas = dcsReplicas.get(dc);
-            List<InetAddressAndPort> dcEndpoint = dcEndpoints.getValue();
+            if (replicas > 0)
-            waitSet.addAll(dcEndpoint.subList(0, Math.min(localQuorumFor(keyspace, dcEndpoints.getKey()), dcEndpoint.size())));
+            {
-        }
+                dcsReplicas.put(dc, --replicas);
-
+                return true;
-        return waitSet;
+            }
+            return false;
+        });
     }
 
-    public boolean isSufficientLiveNodes(Keyspace keyspace, Iterable<InetAddressAndPort> liveEndpoints)
+    public boolean isSufficientLiveNodesForRead(Keyspace keyspace, Endpoints<?> liveReplicas)
     {
         switch (this)
         {
@@ -235,75 +293,92 @@ public enum ConsistencyLevel
                 // local hint is acceptable, and local node is always live
                 return true;
             case LOCAL_ONE:
-                return countLocalEndpoints(liveEndpoints) >= 1;
+                return countDCLocalReplicas(liveReplicas).isSufficient(1, 1);
             case LOCAL_QUORUM:
-                return countLocalEndpoints(liveEndpoints) >= blockFor(keyspace);
+                return countDCLocalReplicas(liveReplicas).isSufficient(blockFor(keyspace), 1);
             case EACH_QUORUM:
                 if (keyspace.getReplicationStrategy() instanceof NetworkTopologyStrategy)
                 {
-                    for (Map.Entry<String, Integer> entry : countPerDCEndpoints(keyspace, liveEndpoints).entrySet())
+                    int fullCount = 0;
+                    for (Map.Entry<String, ReplicaCount> entry : countPerDCEndpoints(keyspace, liveReplicas).entrySet())
                     {
-                        if (entry.getValue() < localQuorumFor(keyspace, entry.getKey()))
+                        ReplicaCount count = entry.getValue();
+                        if (!count.isSufficient(localQuorumFor(keyspace, entry.getKey()), 0))
                             return false;
+                        fullCount += count.fullReplicas;
                     }
-                    return true;
+                    return fullCount > 0;
                 }
                 // Fallthough on purpose for SimpleStrategy
             default:
-                return Iterables.size(liveEndpoints) >= blockFor(keyspace);
+                return liveReplicas.size() >= blockFor(keyspace)
+                        && Replicas.countFull(liveReplicas) > 0;
         }
     }
 
-    public void assureSufficientLiveNodes(Keyspace keyspace, Iterable<InetAddressAndPort> liveEndpoints) throws UnavailableException
+    public void assureSufficientLiveNodesForRead(Keyspace keyspace, Endpoints<?> liveReplicas) throws UnavailableException
+    {
+        assureSufficientLiveNodes(keyspace, liveReplicas, blockFor(keyspace), 1);
+    }
+    public void assureSufficientLiveNodesForWrite(Keyspace keyspace, Endpoints<?> allLive, Endpoints<?> pendingWithDown) throws UnavailableException
+    {
+        assureSufficientLiveNodes(keyspace, allLive, blockForWrite(keyspace, pendingWithDown), 0);
+    }
+    public void assureSufficientLiveNodes(Keyspace keyspace, Endpoints<?> allLive, int blockFor, int blockForFullReplicas) throws UnavailableException
     {
-        int blockFor = blockFor(keyspace);
         switch (this)
         {
             case ANY:
                 // local hint is acceptable, and local node is always live
                 break;
             case LOCAL_ONE:
-                if (countLocalEndpoints(liveEndpoints) == 0)
+            {
-                    throw new UnavailableException(this, 1, 0);
+                ReplicaCount localLive = countDCLocalReplicas(allLive);
+                if (!localLive.isSufficient(blockFor, blockForFullReplicas))
+                    throw UnavailableException.create(this, 1, blockForFullReplicas, localLive.allReplicas(), localLive.fullReplicas);
                 break;
+            }
             case LOCAL_QUORUM:
-                int localLive = countLocalEndpoints(liveEndpoints);
+            {
-                if (localLive < blockFor)
+                ReplicaCount localLive = countDCLocalReplicas(allLive);
+                if (!localLive.isSufficient(blockFor, blockForFullReplicas))
                 {
                     if (logger.isTraceEnabled())
                     {
-                        StringBuilder builder = new StringBuilder("Local replicas [");
+                        logger.trace(String.format("Local replicas %s are insufficient to satisfy LOCAL_QUORUM requirement of %d live replicas and %d full replicas in '%s'",
-                        for (InetAddressAndPort endpoint : liveEndpoints)
+                                allLive.filter(ConsistencyLevel::isLocal), blockFor, blockForFullReplicas, DatabaseDescriptor.getLocalDataCenter()));
-                        {
-                            if (isLocal(endpoint))
-                                builder.append(endpoint).append(",");
-                        }
-                        builder.append("] are insufficient to satisfy LOCAL_QUORUM requirement of ").append(blockFor).append(" live nodes in '").append(DatabaseDescriptor.getLocalDataCenter()).append("'");
-                        logger.trace(builder.toString());
                     }
-                    throw new UnavailableException(this, blockFor, localLive);
+                    throw UnavailableException.create(this, blockFor, blockForFullReplicas, localLive.allReplicas(), localLive.fullReplicas);
                 }
                 break;
+            }
             case EACH_QUORUM:
                 if (keyspace.getReplicationStrategy() instanceof NetworkTopologyStrategy)
                 {
-                    for (Map.Entry<String, Integer> entry : countPerDCEndpoints(keyspace, liveEndpoints).entrySet())
+                    int total = 0;
+                    int totalFull = 0;
+                    for (Map.Entry<String, ReplicaCount> entry : countPerDCEndpoints(keyspace, allLive).entrySet())
                     {
                         int dcBlockFor = localQuorumFor(keyspace, entry.getKey());
-                        int dcLive = entry.getValue();
+                        ReplicaCount dcCount = entry.getValue();
-                        if (dcLive < dcBlockFor)
+                        if (!dcCount.isSufficient(dcBlockFor, 0))
-                            throw new UnavailableException(this, entry.getKey(), dcBlockFor, dcLive);
+                            throw UnavailableException.create(this, entry.getKey(), dcBlockFor, dcCount.allReplicas(), 0, dcCount.fullReplicas);
+                        totalFull += dcCount.fullReplicas;
+                        total += dcCount.allReplicas();
                     }
+                    if (totalFull < blockForFullReplicas)
+                        throw UnavailableException.create(this, blockFor, total, blockForFullReplicas, totalFull);
                     break;
                 }
                 // Fallthough on purpose for SimpleStrategy
             default:
-                int live = Iterables.size(liveEndpoints);
+                int live = allLive.size();
-                if (live < blockFor)
+                int full = Replicas.countFull(allLive);
+                if (live < blockFor || full < blockForFullReplicas)
                 {
                     if (logger.isTraceEnabled())
-                        logger.trace("Live nodes {} do not satisfy ConsistencyLevel ({} required)", Iterables.toString(liveEndpoints), blockFor);
+                        logger.trace("Live nodes {} do not satisfy ConsistencyLevel ({} required)", Iterables.toString(allLive), blockFor);
-                    throw new UnavailableException(this, blockFor, live);
+                    throw UnavailableException.create(this, blockFor, blockForFullReplicas, live, full);
                 }
                 break;
         }

src/java/org/apache/cassandra/db/DiskBoundaryManager.java

@@ -19,8 +19,9 @@
 package org.apache.cassandra.db;
 
 import java.util.ArrayList;
-import java.util.Collection;
+import java.util.Comparator;
 import java.util.List;
+import java.util.stream.Collectors;
 
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -30,6 +31,8 @@ import org.apache.cassandra.dht.IPartitioner;
 import org.apache.cassandra.dht.Range;
 import org.apache.cassandra.dht.Splitter;
 import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
 import org.apache.cassandra.locator.TokenMetadata;
 import org.apache.cassandra.service.PendingRangeCalculatorService;
 import org.apache.cassandra.service.StorageService;
@@ -68,7 +71,7 @@ public class DiskBoundaryManager
 
     private static DiskBoundaries getDiskBoundaryValue(ColumnFamilyStore cfs)
     {
-        Collection<Range<Token>> localRanges;
+        RangesAtEndpoint localRanges;
 
         long ringVersion;
         TokenMetadata tmd;
@@ -87,7 +90,7 @@ public class DiskBoundaryManager
                 // Reason we use use the future settled TMD is that if we decommission a node, we want to stream
                 // from that node to the correct location on disk, if we didn't, we would put new files in the wrong places.
                 // We do this to minimize the amount of data we need to move in rebalancedisks once everything settled
-                localRanges = cfs.keyspace.getReplicationStrategy().getAddressRanges(tmd.cloneAfterAllSettled()).get(FBUtilities.getBroadcastAddressAndPort());
+                localRanges = cfs.keyspace.getReplicationStrategy().getAddressReplicas(tmd.cloneAfterAllSettled(), FBUtilities.getBroadcastAddressAndPort());
             }
             logger.debug("Got local ranges {} (ringVersion = {})", localRanges, ringVersion);
         }
@@ -106,9 +109,18 @@ public class DiskBoundaryManager
         if (localRanges == null || localRanges.isEmpty())
             return new DiskBoundaries(dirs, null, ringVersion, directoriesVersion);
 
-        List<Range<Token>> sortedLocalRanges = Range.sort(localRanges);
+        // note that Range.sort unwraps any wraparound ranges, so we need to sort them here
+        List<Range<Token>> fullLocalRanges = Range.sort(localRanges.stream()
+                                                                   .filter(Replica::isFull)
+                                                                   .map(Replica::range)
+                                                                   .collect(Collectors.toList()));
+        List<Range<Token>> transientLocalRanges = Range.sort(localRanges.stream()
+                                                                        .filter(Replica::isTransient)
+                                                                        .map(Replica::range)
+                                                                        .collect(Collectors.toList()));
+
+        List<PartitionPosition> positions = getDiskBoundaries(fullLocalRanges, transientLocalRanges, cfs.getPartitioner(), dirs);
 
-        List<PartitionPosition> positions = getDiskBoundaries(sortedLocalRanges, cfs.getPartitioner(), dirs);
         return new DiskBoundaries(dirs, positions, ringVersion, directoriesVersion);
     }
 
@@ -121,15 +133,26 @@ public class DiskBoundaryManager
      *
      * The final entry in the returned list will always be the partitioner maximum tokens upper key bound
      */
-    private static List<PartitionPosition> getDiskBoundaries(List<Range<Token>> sortedLocalRanges, IPartitioner partitioner, Directories.DataDirectory[] dataDirectories)
+    private static List<PartitionPosition> getDiskBoundaries(List<Range<Token>> fullRanges, List<Range<Token>> transientRanges, IPartitioner partitioner, Directories.DataDirectory[] dataDirectories)
     {
         assert partitioner.splitter().isPresent();
+
         Splitter splitter = partitioner.splitter().get();
         boolean dontSplitRanges = DatabaseDescriptor.getNumTokens() > 1;
-        List<Token> boundaries = splitter.splitOwnedRanges(dataDirectories.length, sortedLocalRanges, dontSplitRanges);
+
+        List<Splitter.WeightedRange> weightedRanges = new ArrayList<>(fullRanges.size() + transientRanges.size());
+        for (Range<Token> r : fullRanges)
+            weightedRanges.add(new Splitter.WeightedRange(1.0, r));
+
+        for (Range<Token> r : transientRanges)
+            weightedRanges.add(new Splitter.WeightedRange(0.1, r));
+
+        weightedRanges.sort(Comparator.comparing(Splitter.WeightedRange::left));
+
+        List<Token> boundaries = splitter.splitOwnedRanges(dataDirectories.length, weightedRanges, dontSplitRanges);
         // If we can't split by ranges, split evenly to ensure utilisation of all disks
         if (dontSplitRanges && boundaries.size() < dataDirectories.length)
-            boundaries = splitter.splitOwnedRanges(dataDirectories.length, sortedLocalRanges, false);
+            boundaries = splitter.splitOwnedRanges(dataDirectories.length, weightedRanges, false);
 
         List<PartitionPosition> diskBoundaries = new ArrayList<>();
         for (int i = 0; i < boundaries.size() - 1; i++)

src/java/org/apache/cassandra/db/Memtable.java

@@ -503,6 +503,7 @@ public class Memtable implements Comparable<Memtable>
                                                 toFlush.size(),
                                                 ActiveRepairService.UNREPAIRED_SSTABLE,
                                                 ActiveRepairService.NO_PENDING_REPAIR,
+                                                false,
                                                 sstableMetadataCollector,
                                                 new SerializationHeader(true, cfs.metadata(), columns, stats), txn);
         }

src/java/org/apache/cassandra/db/MutationVerbHandler.java

@@ -17,7 +17,6 @@
  */
 package org.apache.cassandra.db;
 
-import java.io.IOException;
 import java.util.Iterator;
 
 import org.apache.cassandra.exceptions.WriteTimeoutException;
@@ -38,7 +37,7 @@ public class MutationVerbHandler implements IVerbHandler<Mutation>
         Tracing.trace("Payload application resulted in WriteTimeout, not replying");
     }
 
-    public void doVerb(MessageIn<Mutation> message, int id)  throws IOException
+    public void doVerb(MessageIn<Mutation> message, int id)
     {
         // Check if there were any forwarding headers in this message
         InetAddressAndPort from = (InetAddressAndPort)message.parameters.get(ParameterType.FORWARD_FROM);
@@ -69,7 +68,7 @@ public class MutationVerbHandler implements IVerbHandler<Mutation>
         }
     }
 
-    private static void forwardToLocalNodes(Mutation mutation, MessagingService.Verb verb, ForwardToContainer forwardTo, InetAddressAndPort from) throws IOException
+    private static void forwardToLocalNodes(Mutation mutation, MessagingService.Verb verb, ForwardToContainer forwardTo, InetAddressAndPort from)
     {
         // tell the recipients who to send their ack to
         MessageOut<Mutation> message = new MessageOut<>(verb, mutation, Mutation.serializer).withParameter(ParameterType.FORWARD_FROM, from);

src/java/org/apache/cassandra/db/PartitionRangeReadCommand.java

@@ -24,7 +24,6 @@ import java.util.List;
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.collect.Iterables;
 
-import org.apache.cassandra.dht.Token;
 import org.apache.cassandra.schema.TableMetadata;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.db.filter.*;
@@ -61,6 +60,7 @@ public class PartitionRangeReadCommand extends ReadCommand implements PartitionR
 
     private PartitionRangeReadCommand(boolean isDigest,
                                      int digestVersion,
+                                     boolean acceptsTransient,
                                      TableMetadata metadata,
                                      int nowInSec,
                                      ColumnFilter columnFilter,
@@ -69,7 +69,7 @@ public class PartitionRangeReadCommand extends ReadCommand implements PartitionR
                                      DataRange dataRange,
                                      IndexMetadata index)
     {
-        super(Kind.PARTITION_RANGE, isDigest, digestVersion, metadata, nowInSec, columnFilter, rowFilter, limits, index);
+        super(Kind.PARTITION_RANGE, isDigest, digestVersion, acceptsTransient, metadata, nowInSec, columnFilter, rowFilter, limits, index);
         this.dataRange = dataRange;
     }
 
@@ -82,6 +82,7 @@ public class PartitionRangeReadCommand extends ReadCommand implements PartitionR
     {
         return new PartitionRangeReadCommand(false,
                                              0,
+                                             false,
                                              metadata,
                                              nowInSec,
                                              columnFilter,
@@ -103,6 +104,7 @@ public class PartitionRangeReadCommand extends ReadCommand implements PartitionR
     {
         return new PartitionRangeReadCommand(false,
                                              0,
+                                             false,
                                              metadata,
                                              nowInSec,
                                              ColumnFilter.all(metadata),
@@ -151,6 +153,7 @@ public class PartitionRangeReadCommand extends ReadCommand implements PartitionR
         // on the ring.
         return new PartitionRangeReadCommand(isDigestQuery(),
                                              digestVersion(),
+                                             acceptsTransient(),
                                              metadata(),
                                              nowInSec(),
                                              columnFilter(),
@@ -164,6 +167,7 @@ public class PartitionRangeReadCommand extends ReadCommand implements PartitionR
     {
         return new PartitionRangeReadCommand(isDigestQuery(),
                                              digestVersion(),
+                                             acceptsTransient(),
                                              metadata(),
                                              nowInSec(),
                                              columnFilter(),
@@ -177,6 +181,21 @@ public class PartitionRangeReadCommand extends ReadCommand implements PartitionR
     {
         return new PartitionRangeReadCommand(true,
                                              digestVersion(),
+                                             false,
+                                             metadata(),
+                                             nowInSec(),
+                                             columnFilter(),
+                                             rowFilter(),
+                                             limits(),
+                                             dataRange(),
+                                             indexMetadata());
+    }
+
+    public PartitionRangeReadCommand copyAsTransientQuery()
+    {
+        return new PartitionRangeReadCommand(false,
+                                             0,
+                                             true,
                                              metadata(),
                                              nowInSec(),
                                              columnFilter(),
@@ -191,6 +210,7 @@ public class PartitionRangeReadCommand extends ReadCommand implements PartitionR
     {
         return new PartitionRangeReadCommand(isDigestQuery(),
                                              digestVersion(),
+                                             acceptsTransient(),
                                              metadata(),
                                              nowInSec(),
                                              columnFilter(),
@@ -205,6 +225,7 @@ public class PartitionRangeReadCommand extends ReadCommand implements PartitionR
     {
         return new PartitionRangeReadCommand(isDigestQuery(),
                                              digestVersion(),
+                                             acceptsTransient(),
                                              metadata(),
                                              nowInSec(),
                                              columnFilter(),
@@ -406,6 +427,7 @@ public class PartitionRangeReadCommand extends ReadCommand implements PartitionR
                                        int version,
                                        boolean isDigest,
                                        int digestVersion,
+                                       boolean acceptsTransient,
                                        TableMetadata metadata,
                                        int nowInSec,
                                        ColumnFilter columnFilter,
@@ -415,7 +437,7 @@ public class PartitionRangeReadCommand extends ReadCommand implements PartitionR
         throws IOException
         {
             DataRange range = DataRange.serializer.deserialize(in, version, metadata);
-            return new PartitionRangeReadCommand(isDigest, digestVersion, metadata, nowInSec, columnFilter, rowFilter, limits, range, index);
+            return new PartitionRangeReadCommand(isDigest, digestVersion, acceptsTransient, metadata, nowInSec, columnFilter, rowFilter, limits, range, index);
         }
     }
 }

src/java/org/apache/cassandra/db/ReadCommand.java

@@ -34,7 +34,6 @@ import org.apache.cassandra.db.transform.RTBoundCloser;
 import org.apache.cassandra.db.transform.RTBoundValidator;
 import org.apache.cassandra.db.transform.StoppingTransformation;
 import org.apache.cassandra.db.transform.Transformation;
-import org.apache.cassandra.dht.Token;
 import org.apache.cassandra.exceptions.UnknownIndexException;
 import org.apache.cassandra.index.Index;
 import org.apache.cassandra.index.IndexNotAvailableException;
@@ -68,6 +67,7 @@ public abstract class ReadCommand extends AbstractReadQuery
     private final Kind kind;
 
     private final boolean isDigestQuery;
+    private final boolean acceptsTransient;
     // if a digest query, the version for which the digest is expected. Ignored if not a digest.
     private int digestVersion;
 
@@ -80,6 +80,7 @@ public abstract class ReadCommand extends AbstractReadQuery
                                                 int version,
                                                 boolean isDigest,
                                                 int digestVersion,
+                                                boolean acceptsTransient,
                                                 TableMetadata metadata,
                                                 int nowInSec,
                                                 ColumnFilter columnFilter,
@@ -104,6 +105,7 @@ public abstract class ReadCommand extends AbstractReadQuery
     protected ReadCommand(Kind kind,
                           boolean isDigestQuery,
                           int digestVersion,
+                          boolean acceptsTransient,
                           TableMetadata metadata,
                           int nowInSec,
                           ColumnFilter columnFilter,
@@ -115,6 +117,7 @@ public abstract class ReadCommand extends AbstractReadQuery
         this.kind = kind;
         this.isDigestQuery = isDigestQuery;
         this.digestVersion = digestVersion;
+        this.acceptsTransient = acceptsTransient;
         this.index = index;
     }
 
@@ -175,6 +178,14 @@ public abstract class ReadCommand extends AbstractReadQuery
         return this;
     }
 
+    /**
+     * @return Whether this query expects only a transient data response, or a full response
+     */
+    public boolean acceptsTransient()
+    {
+        return acceptsTransient;
+    }
+
     /**
      * Index (metadata) chosen for this query. Can be null.
      *
@@ -210,6 +221,7 @@ public abstract class ReadCommand extends AbstractReadQuery
      * Returns a copy of this command with isDigestQuery set to true.
      */
     public abstract ReadCommand copyAsDigestQuery();
+    public abstract ReadCommand copyAsTransientQuery();
 
     protected abstract UnfilteredPartitionIterator queryStorage(ColumnFamilyStore cfs, ReadExecutionController executionController);
 
@@ -569,6 +581,16 @@ public abstract class ReadCommand extends AbstractReadQuery
             return (flags & 0x01) != 0;
         }
 
+        private static boolean acceptsTransient(int flags)
+        {
+            return (flags & 0x08) != 0;
+        }
+
+        private static int acceptsTransientFlag(boolean acceptsTransient)
+        {
+            return acceptsTransient ? 0x08 : 0;
+        }
+
         // We don't set this flag anymore, but still look if we receive a
         // command with it set in case someone is using thrift a mixed 3.0/4.0+
         // cluster (which is unsupported). This is also a reminder for not
@@ -592,7 +614,11 @@ public abstract class ReadCommand extends AbstractReadQuery
         public void serialize(ReadCommand command, DataOutputPlus out, int version) throws IOException
         {
             out.writeByte(command.kind.ordinal());
-            out.writeByte(digestFlag(command.isDigestQuery()) | indexFlag(null != command.indexMetadata()));
+            out.writeByte(
+                    digestFlag(command.isDigestQuery())
+                    | indexFlag(null != command.indexMetadata())
+                    | acceptsTransientFlag(command.acceptsTransient())
+            );
             if (command.isDigestQuery())
                 out.writeUnsignedVInt(command.digestVersion());
             command.metadata().id.serialize(out);
@@ -611,6 +637,7 @@ public abstract class ReadCommand extends AbstractReadQuery
             Kind kind = Kind.values()[in.readByte()];
             int flags = in.readByte();
             boolean isDigest = isDigest(flags);
+            boolean acceptsTransient = acceptsTransient(flags);
             // Shouldn't happen or it's a user error (see comment above) but
             // better complain loudly than doing the wrong thing.
             if (isForThrift(flags))
@@ -628,7 +655,7 @@ public abstract class ReadCommand extends AbstractReadQuery
             DataLimits limits = DataLimits.serializer.deserialize(in, version,  metadata.comparator);
             IndexMetadata index = hasIndex ? deserializeIndexMetadata(in, version, metadata) : null;
 
-            return kind.selectionDeserializer.deserialize(in, version, isDigest, digestVersion, metadata, nowInSec, columnFilter, rowFilter, limits, index);
+            return kind.selectionDeserializer.deserialize(in, version, isDigest, digestVersion, acceptsTransient, metadata, nowInSec, columnFilter, rowFilter, limits, index);
         }
 
         private IndexMetadata deserializeIndexMetadata(DataInputPlus in, int version, TableMetadata metadata) throws IOException

src/java/org/apache/cassandra/db/SSTableImporter.java

@@ -349,9 +349,9 @@ public class SSTableImporter
             }
             if (options.clearRepaired)
             {
-                descriptor.getMetadataSerializer().mutateRepaired(descriptor,
+                descriptor.getMetadataSerializer().mutateRepairMetadata(descriptor, ActiveRepairService.UNREPAIRED_SSTABLE,
-                                                                  ActiveRepairService.UNREPAIRED_SSTABLE,
+                                                                        null,
-                                                                  null);
+                                                                        false);
             }
         }
     }

src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java

@@ -36,7 +36,6 @@ import org.apache.cassandra.db.lifecycle.*;
 import org.apache.cassandra.db.partitions.*;
 import org.apache.cassandra.db.rows.*;
 import org.apache.cassandra.db.transform.Transformation;
-import org.apache.cassandra.dht.Token;
 import org.apache.cassandra.exceptions.RequestExecutionException;
 import org.apache.cassandra.io.sstable.format.SSTableReader;
 import org.apache.cassandra.io.sstable.format.SSTableReadsListener;
@@ -71,6 +70,7 @@ public class SinglePartitionReadCommand extends ReadCommand implements SinglePar
     @VisibleForTesting
     protected SinglePartitionReadCommand(boolean isDigest,
                                          int digestVersion,
+                                         boolean acceptsTransient,
                                          TableMetadata metadata,
                                          int nowInSec,
                                          ColumnFilter columnFilter,
@@ -80,7 +80,7 @@ public class SinglePartitionReadCommand extends ReadCommand implements SinglePar
                                          ClusteringIndexFilter clusteringIndexFilter,
                                          IndexMetadata index)
     {
-        super(Kind.SINGLE_PARTITION, isDigest, digestVersion, metadata, nowInSec, columnFilter, rowFilter, limits, index);
+        super(Kind.SINGLE_PARTITION, isDigest, digestVersion, acceptsTransient, metadata, nowInSec, columnFilter, rowFilter, limits, index);
         assert partitionKey.getPartitioner() == metadata.partitioner;
         this.partitionKey = partitionKey;
         this.clusteringIndexFilter = clusteringIndexFilter;
@@ -111,6 +111,7 @@ public class SinglePartitionReadCommand extends ReadCommand implements SinglePar
     {
         return new SinglePartitionReadCommand(false,
                                               0,
+                                              false,
                                               metadata,
                                               nowInSec,
                                               columnFilter,
@@ -286,6 +287,7 @@ public class SinglePartitionReadCommand extends ReadCommand implements SinglePar
     {
         return new SinglePartitionReadCommand(isDigestQuery(),
                                               digestVersion(),
+                                              acceptsTransient(),
                                               metadata(),
                                               nowInSec(),
                                               columnFilter(),
@@ -300,6 +302,22 @@ public class SinglePartitionReadCommand extends ReadCommand implements SinglePar
     {
         return new SinglePartitionReadCommand(true,
                                               digestVersion(),
+                                              acceptsTransient(),
+                                              metadata(),
+                                              nowInSec(),
+                                              columnFilter(),
+                                              rowFilter(),
+                                              limits(),
+                                              partitionKey(),
+                                              clusteringIndexFilter(),
+                                              indexMetadata());
+    }
+
+    public SinglePartitionReadCommand copyAsTransientQuery()
+    {
+        return new SinglePartitionReadCommand(false,
+                                              0,
+                                              true,
                                               metadata(),
                                               nowInSec(),
                                               columnFilter(),
@@ -315,6 +333,7 @@ public class SinglePartitionReadCommand extends ReadCommand implements SinglePar
     {
         return new SinglePartitionReadCommand(isDigestQuery(),
                                               digestVersion(),
+                                              acceptsTransient(),
                                               metadata(),
                                               nowInSec(),
                                               columnFilter(),
@@ -1064,6 +1083,7 @@ public class SinglePartitionReadCommand extends ReadCommand implements SinglePar
                                        int version,
                                        boolean isDigest,
                                        int digestVersion,
+                                       boolean acceptsTransient,
                                        TableMetadata metadata,
                                        int nowInSec,
                                        ColumnFilter columnFilter,
@@ -1074,7 +1094,7 @@ public class SinglePartitionReadCommand extends ReadCommand implements SinglePar
         {
             DecoratedKey key = metadata.partitioner.decorateKey(metadata.partitionKeyType.readValue(in, DatabaseDescriptor.getMaxValueSize()));
             ClusteringIndexFilter filter = ClusteringIndexFilter.serializer.deserialize(in, version, metadata);
-            return new SinglePartitionReadCommand(isDigest, digestVersion, metadata, nowInSec, columnFilter, rowFilter, limits, key, filter, index);
+            return new SinglePartitionReadCommand(isDigest, digestVersion, acceptsTransient, metadata, nowInSec, columnFilter, rowFilter, limits, key, filter, index);
         }
     }
 

src/java/org/apache/cassandra/db/SystemKeyspace.java

@@ -29,13 +29,15 @@ import java.util.stream.StreamSupport;
 import javax.management.openmbean.OpenDataException;
 import javax.management.openmbean.TabularData;
 
+import com.google.common.annotations.VisibleForTesting;
 import com.google.common.collect.HashMultimap;
 import com.google.common.collect.ImmutableMap;
-import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.SetMultimap;
 import com.google.common.collect.Sets;
 import com.google.common.io.ByteStreams;
 import com.google.common.util.concurrent.ListenableFuture;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -71,6 +73,8 @@ import static java.util.Collections.singletonMap;
 
 import static org.apache.cassandra.cql3.QueryProcessor.executeInternal;
 import static org.apache.cassandra.cql3.QueryProcessor.executeOnceInternal;
+import static org.apache.cassandra.locator.Replica.fullReplica;
+import static org.apache.cassandra.locator.Replica.transientReplica;
 
 public final class SystemKeyspace
 {
@@ -95,12 +99,10 @@ public final class SystemKeyspace
     public static final String LOCAL = "local";
     public static final String PEERS_V2 = "peers_v2";
     public static final String PEER_EVENTS_V2 = "peer_events_v2";
-    public static final String RANGE_XFERS = "range_xfers";
     public static final String COMPACTION_HISTORY = "compaction_history";
     public static final String SSTABLE_ACTIVITY = "sstable_activity";
     public static final String SIZE_ESTIMATES = "size_estimates";
-    public static final String AVAILABLE_RANGES = "available_ranges";
+    public static final String AVAILABLE_RANGES_V2 = "available_ranges_v2";
-    public static final String TRANSFERRED_RANGES = "transferred_ranges";
     public static final String TRANSFERRED_RANGES_V2 = "transferred_ranges_v2";
     public static final String VIEW_BUILDS_IN_PROGRESS = "view_builds_in_progress";
     public static final String BUILT_VIEWS = "built_views";
@@ -110,6 +112,8 @@ public final class SystemKeyspace
     @Deprecated public static final String LEGACY_PEERS = "peers";
     @Deprecated public static final String LEGACY_PEER_EVENTS = "peer_events";
     @Deprecated public static final String LEGACY_TRANSFERRED_RANGES = "transferred_ranges";
+    @Deprecated public static final String LEGACY_AVAILABLE_RANGES = "available_ranges";
+
 
     public static final TableMetadata Batches =
         parse(BATCHES,
@@ -207,15 +211,6 @@ public final class SystemKeyspace
                 + "PRIMARY KEY ((peer), peer_port))")
                 .build();
 
-    private static final TableMetadata RangeXfers =
-        parse(RANGE_XFERS,
-                "ranges requested for transfer",
-                "CREATE TABLE %s ("
-                + "token_bytes blob,"
-                + "requested_at timestamp,"
-                + "PRIMARY KEY ((token_bytes)))")
-                .build();
-
     private static final TableMetadata CompactionHistory =
         parse(COMPACTION_HISTORY,
                 "week-long compaction history",
@@ -256,14 +251,15 @@ public final class SystemKeyspace
                 + "PRIMARY KEY ((keyspace_name), table_name, range_start, range_end))")
                 .build();
 
-    private static final TableMetadata AvailableRanges =
+    private static final TableMetadata AvailableRangesV2 =
-        parse(AVAILABLE_RANGES,
+    parse(AVAILABLE_RANGES_V2,
-                "available keyspace/ranges during bootstrap/replace that are ready to be served",
+          "available keyspace/ranges during bootstrap/replace that are ready to be served",
-                "CREATE TABLE %s ("
+          "CREATE TABLE %s ("
-                + "keyspace_name text,"
+          + "keyspace_name text,"
-                + "ranges set<blob>,"
+          + "full_ranges set<blob>,"
-                + "PRIMARY KEY ((keyspace_name)))")
+          + "transient_ranges set<blob>,"
-                .build();
+          + "PRIMARY KEY ((keyspace_name)))")
+    .build();
 
     private static final TableMetadata TransferredRangesV2 =
         parse(TRANSFERRED_RANGES_V2,
@@ -366,6 +362,16 @@ public final class SystemKeyspace
             + "PRIMARY KEY ((operation, keyspace_name), peer))")
             .build();
 
+    @Deprecated
+    private static final TableMetadata LegacyAvailableRanges =
+        parse(LEGACY_AVAILABLE_RANGES,
+              "available keyspace/ranges during bootstrap/replace that are ready to be served",
+              "CREATE TABLE %s ("
+              + "keyspace_name text,"
+              + "ranges set<blob>,"
+              + "PRIMARY KEY ((keyspace_name)))")
+        .build();
+
     private static TableMetadata.Builder parse(String table, String description, String cql)
     {
         return CreateTableStatement.parse(format(cql, table), SchemaConstants.SYSTEM_KEYSPACE_NAME)
@@ -390,11 +396,11 @@ public final class SystemKeyspace
                          LegacyPeers,
                          PeerEventsV2,
                          LegacyPeerEvents,
-                         RangeXfers,
                          CompactionHistory,
                          SSTableActivity,
                          SizeEstimates,
-                         AvailableRanges,
+                         AvailableRangesV2,
+                         LegacyAvailableRanges,
                          TransferredRangesV2,
                          LegacyTransferredRanges,
                          ViewBuildsInProgress,
@@ -1270,36 +1276,38 @@ public final class SystemKeyspace
         executeInternal(cql, keyspace, table);
     }
 
-    public static synchronized void updateAvailableRanges(String keyspace, Collection<Range<Token>> completedRanges)
+    public static synchronized void updateAvailableRanges(String keyspace, Collection<Range<Token>> completedFullRanges, Collection<Range<Token>> completedTransientRanges)
     {
-        String cql = "UPDATE system.%s SET ranges = ranges + ? WHERE keyspace_name = ?";
+        String cql = "UPDATE system.%s SET full_ranges = full_ranges + ?, transient_ranges = transient_ranges + ? WHERE keyspace_name = ?";
-        Set<ByteBuffer> rangesToUpdate = new HashSet<>(completedRanges.size());
+        executeInternal(format(cql, AVAILABLE_RANGES_V2),
-        for (Range<Token> range : completedRanges)
+                        completedFullRanges.stream().map(SystemKeyspace::rangeToBytes).collect(Collectors.toSet()),
-        {
+                        completedTransientRanges.stream().map(SystemKeyspace::rangeToBytes).collect(Collectors.toSet()),
-            rangesToUpdate.add(rangeToBytes(range));
+                        keyspace);
-        }
-        executeInternal(format(cql, AVAILABLE_RANGES), rangesToUpdate, keyspace);
     }
 
-    public static synchronized Set<Range<Token>> getAvailableRanges(String keyspace, IPartitioner partitioner)
+    public static synchronized RangesAtEndpoint getAvailableRanges(String keyspace, IPartitioner partitioner)
     {
-        Set<Range<Token>> result = new HashSet<>();
         String query = "SELECT * FROM system.%s WHERE keyspace_name=?";
-        UntypedResultSet rs = executeInternal(format(query, AVAILABLE_RANGES), keyspace);
+        UntypedResultSet rs = executeInternal(format(query, AVAILABLE_RANGES_V2), keyspace);
+        InetAddressAndPort endpoint = InetAddressAndPort.getLocalHost();
+        RangesAtEndpoint.Builder builder = RangesAtEndpoint.builder(endpoint);
         for (UntypedResultSet.Row row : rs)
         {
-            Set<ByteBuffer> rawRanges = row.getSet("ranges", BytesType.instance);
+            Optional.ofNullable(row.getSet("full_ranges", BytesType.instance))
-            for (ByteBuffer rawRange : rawRanges)
+                    .ifPresent(full_ranges -> full_ranges.stream()
-            {
+                            .map(buf -> byteBufferToRange(buf, partitioner))
-                result.add(byteBufferToRange(rawRange, partitioner));
+                            .forEach(range -> builder.add(fullReplica(endpoint, range))));
-            }
+            Optional.ofNullable(row.getSet("transient_ranges", BytesType.instance))
+                    .ifPresent(transient_ranges -> transient_ranges.stream()
+                            .map(buf -> byteBufferToRange(buf, partitioner))
+                            .forEach(range -> builder.add(transientReplica(endpoint, range))));
         }
-        return ImmutableSet.copyOf(result);
+        return builder.build();
     }
 
     public static void resetAvailableRanges()
     {
-        ColumnFamilyStore availableRanges = Keyspace.open(SchemaConstants.SYSTEM_KEYSPACE_NAME).getColumnFamilyStore(AVAILABLE_RANGES);
+        ColumnFamilyStore availableRanges = Keyspace.open(SchemaConstants.SYSTEM_KEYSPACE_NAME).getColumnFamilyStore(AVAILABLE_RANGES_V2);
         availableRanges.truncateBlocking();
     }
 
@@ -1405,7 +1413,13 @@ public final class SystemKeyspace
         return result.one().getString("release_version");
     }
 
-    private static ByteBuffer rangeToBytes(Range<Token> range)
+    @VisibleForTesting
+    public static Set<Range<Token>> rawRangesToRangeSet(Set<ByteBuffer> rawRanges, IPartitioner partitioner)
+    {
+        return rawRanges.stream().map(buf -> byteBufferToRange(buf, partitioner)).collect(Collectors.toSet());
+    }
+
+    static ByteBuffer rangeToBytes(Range<Token> range)
     {
         try (DataOutputBuffer out = new DataOutputBuffer())
         {

src/java/org/apache/cassandra/db/SystemKeyspaceMigrator40.java

@@ -18,6 +18,11 @@
 
 package org.apache.cassandra.db;
 
+import java.nio.ByteBuffer;
+import java.util.Collections;
+import java.util.Optional;
+import java.util.Set;
+
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -45,6 +50,9 @@ public class SystemKeyspaceMigrator40
     static final String peerEventsName = String.format("%s.%s", SchemaConstants.SYSTEM_KEYSPACE_NAME, SystemKeyspace.PEER_EVENTS_V2);
     static final String legacyTransferredRangesName = String.format("%s.%s", SchemaConstants.SYSTEM_KEYSPACE_NAME, SystemKeyspace.LEGACY_TRANSFERRED_RANGES);
     static final String transferredRangesName = String.format("%s.%s", SchemaConstants.SYSTEM_KEYSPACE_NAME, SystemKeyspace.TRANSFERRED_RANGES_V2);
+    static final String legacyAvailableRangesName = String.format("%s.%s", SchemaConstants.SYSTEM_KEYSPACE_NAME, SystemKeyspace.LEGACY_AVAILABLE_RANGES);
+    static final String availableRangesName = String.format("%s.%s", SchemaConstants.SYSTEM_KEYSPACE_NAME, SystemKeyspace.AVAILABLE_RANGES_V2);
+
 
     private static final Logger logger = LoggerFactory.getLogger(SystemKeyspaceMigrator40.class);
 
@@ -55,6 +63,7 @@ public class SystemKeyspaceMigrator40
         migratePeers();
         migratePeerEvents();
         migrateTransferredRanges();
+        migrateAvailableRanges();
     }
 
     private static void migratePeers()
@@ -181,4 +190,40 @@ public class SystemKeyspaceMigrator40
         logger.info("Migrated {} rows from legacy {} to {}", transferred, legacyTransferredRangesName, transferredRangesName);
     }
 
+    static void migrateAvailableRanges()
+    {
+        ColumnFamilyStore newAvailableRanges = Keyspace.open(SchemaConstants.SYSTEM_KEYSPACE_NAME).getColumnFamilyStore(SystemKeyspace.AVAILABLE_RANGES_V2);
+
+        if (!newAvailableRanges.isEmpty())
+            return;
+
+        logger.info("{} table was empty, migrating legacy {} to {}", availableRangesName, legacyAvailableRangesName, availableRangesName);
+
+        String query = String.format("SELECT * FROM %s",
+                                     legacyAvailableRangesName);
+
+        String insert = String.format("INSERT INTO %s ("
+                                      + "keyspace_name, "
+                                      + "full_ranges, "
+                                      + "transient_ranges) "
+                                      + " values ( ?, ?, ? )",
+                                      availableRangesName);
+
+        UntypedResultSet rows = QueryProcessor.executeInternalWithPaging(query, 1000);
+        int transferred = 0;
+        for (UntypedResultSet.Row row : rows)
+        {
+            logger.debug("Transferring row {}", transferred);
+            String keyspace = row.getString("keyspace_name");
+            Set<ByteBuffer> ranges = Optional.ofNullable(row.getSet("ranges", BytesType.instance)).orElse(Collections.emptySet());
+            QueryProcessor.executeInternal(insert,
+                                           keyspace,
+                                           ranges,
+                                           Collections.emptySet());
+            transferred++;
+        }
+
+        logger.info("Migrated {} rows from legacy {} to {}", transferred, legacyAvailableRangesName, availableRangesName);
+    }
+
 }

src/java/org/apache/cassandra/db/TableCQLHelper.java

@@ -310,6 +310,7 @@ public class TableCQLHelper
         builder.append("\n\tAND max_index_interval = ").append(tableParams.maxIndexInterval);
         builder.append("\n\tAND memtable_flush_period_in_ms = ").append(tableParams.memtableFlushPeriodInMs);
         builder.append("\n\tAND speculative_retry = '").append(tableParams.speculativeRetry).append("'");
+        builder.append("\n\tAND speculative_write_threshold = '").append(tableParams.speculativeWriteThreshold).append("'");
         builder.append("\n\tAND comment = ").append(singleQuote(tableParams.comment));
         builder.append("\n\tAND caching = ").append(toCQL(tableParams.caching.asMap()));
         builder.append("\n\tAND compaction = ").append(toCQL(tableParams.compaction.asMap()));

src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java

@@ -530,12 +530,13 @@ public abstract class AbstractCompactionStrategy
                                                        long keyCount,
                                                        long repairedAt,
                                                        UUID pendingRepair,
+                                                       boolean isTransient,
                                                        MetadataCollector meta,
                                                        SerializationHeader header,
                                                        Collection<Index> indexes,
                                                        LifecycleTransaction txn)
     {
-        return SimpleSSTableMultiWriter.create(descriptor, keyCount, repairedAt, pendingRepair, cfs.metadata, meta, header, indexes, txn);
+        return SimpleSSTableMultiWriter.create(descriptor, keyCount, repairedAt, pendingRepair, isTransient, cfs.metadata, meta, header, indexes, txn);
     }
 
     public boolean supportsEarlyOpen()

src/java/org/apache/cassandra/db/compaction/AbstractStrategyHolder.java

@@ -158,11 +158,11 @@ public abstract class AbstractStrategyHolder
      * groups they deal with. IOW, if one holder returns true for a given isRepaired/isPendingRepair combo,
      * none of the others should.
      */
-    public abstract boolean managesRepairedGroup(boolean isRepaired, boolean isPendingRepair);
+    public abstract boolean managesRepairedGroup(boolean isRepaired, boolean isPendingRepair, boolean isTransient);
 
     public boolean managesSSTable(SSTableReader sstable)
     {
-        return managesRepairedGroup(sstable.isRepaired(), sstable.isPendingRepair());
+        return managesRepairedGroup(sstable.isRepaired(), sstable.isPendingRepair(), sstable.isTransient());
     }
 
     public abstract AbstractCompactionStrategy getStrategyFor(SSTableReader sstable);
@@ -193,6 +193,7 @@ public abstract class AbstractStrategyHolder
                                                                 long keyCount,
                                                                 long repairedAt,
                                                                 UUID pendingRepair,
+                                                                boolean isTransient,
                                                                 MetadataCollector collector,
                                                                 SerializationHeader header,
                                                                 Collection<Index> indexes,
@@ -203,4 +204,6 @@ public abstract class AbstractStrategyHolder
      * if it's not held by this holder
      */
     public abstract int getStrategyIndex(AbstractCompactionStrategy strategy);
+
+    public abstract boolean containsSSTable(SSTableReader sstable);
 }

src/java/org/apache/cassandra/db/compaction/CompactionManager.java

@@ -23,7 +23,7 @@ import java.lang.management.ManagementFactory;
 import java.util.*;
 import java.util.concurrent.*;
 import java.util.concurrent.atomic.AtomicInteger;
-import java.util.function.LongPredicate;
+import java.util.function.Predicate;
 import java.util.stream.Collectors;
 import javax.management.MBeanServer;
 import javax.management.ObjectName;
@@ -34,6 +34,9 @@ import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Preconditions;
 import com.google.common.collect.*;
 import com.google.common.util.concurrent.*;
+
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -43,7 +46,6 @@ import org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor;
 import org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor;
 import org.apache.cassandra.concurrent.NamedThreadFactory;
 import org.apache.cassandra.exceptions.ConfigurationException;
-import org.apache.cassandra.repair.ValidationPartitionIterator;
 import org.apache.cassandra.schema.TableMetadata;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.schema.Schema;
@@ -71,7 +73,6 @@ import org.apache.cassandra.io.sstable.metadata.StatsMetadata;
 import org.apache.cassandra.io.util.FileUtils;
 import org.apache.cassandra.metrics.CompactionMetrics;
 import org.apache.cassandra.metrics.TableMetrics;
-import org.apache.cassandra.repair.Validator;
 import org.apache.cassandra.schema.CompactionParams.TombstoneOption;
 import org.apache.cassandra.service.ActiveRepairService;
 import org.apache.cassandra.service.StorageService;
@@ -81,6 +82,8 @@ import org.apache.cassandra.utils.Throwables;
 import org.apache.cassandra.utils.concurrent.Refs;
 
 import static java.util.Collections.singleton;
+import static org.apache.cassandra.service.ActiveRepairService.NO_PENDING_REPAIR;
+import static org.apache.cassandra.service.ActiveRepairService.UNREPAIRED_SSTABLE;
 
 /**
  * <p>
@@ -509,7 +512,10 @@ public class CompactionManager implements CompactionManagerMBean
             return AllSSTableOpStatus.ABORTED;
         }
         // if local ranges is empty, it means no data should remain
-        final Collection<Range<Token>> ranges = StorageService.instance.getLocalRanges(keyspace.getName());
+        final RangesAtEndpoint replicas = StorageService.instance.getLocalReplicas(keyspace.getName());
+        final Set<Range<Token>> allRanges = replicas.ranges();
+        final Set<Range<Token>> transientRanges = replicas.filter(Replica::isTransient).ranges();
+        final Set<Range<Token>> fullRanges = replicas.filter(Replica::isFull).ranges();
         final boolean hasIndexes = cfStore.indexManager.hasIndexes();
 
         return parallelAllSSTableOperation(cfStore, new OneSSTableOperation()
@@ -525,8 +531,8 @@ public class CompactionManager implements CompactionManagerMBean
             @Override
             public void execute(LifecycleTransaction txn) throws IOException
             {
-                CleanupStrategy cleanupStrategy = CleanupStrategy.get(cfStore, ranges, FBUtilities.nowInSeconds());
+                CleanupStrategy cleanupStrategy = CleanupStrategy.get(cfStore, allRanges, transientRanges, txn.onlyOne().isRepaired(), FBUtilities.nowInSeconds());
-                doCleanupOne(cfStore, txn, cleanupStrategy, ranges, hasIndexes);
+                doCleanupOne(cfStore, txn, cleanupStrategy, replicas.ranges(), fullRanges, transientRanges, hasIndexes);
             }
         }, jobs, OperationType.CLEANUP);
     }
@@ -574,9 +580,8 @@ public class CompactionManager implements CompactionManagerMBean
             logger.info("Partitioner does not support splitting");
             return AllSSTableOpStatus.ABORTED;
         }
-        final Collection<Range<Token>> r = StorageService.instance.getLocalRanges(cfs.keyspace.getName());
 
-        if (r.isEmpty())
+        if (StorageService.instance.getLocalReplicas(cfs.keyspace.getName()).isEmpty())
         {
             logger.info("Relocate cannot run before a node has joined the ring");
             return AllSSTableOpStatus.ABORTED;
@@ -643,7 +648,11 @@ public class CompactionManager implements CompactionManagerMBean
     /**
      * Splits the given token ranges of the given sstables into a pending repair silo
      */
-    public ListenableFuture<?> submitPendingAntiCompaction(ColumnFamilyStore cfs, Collection<Range<Token>> ranges, Refs<SSTableReader> sstables, LifecycleTransaction txn, UUID sessionId)
+    public ListenableFuture<?> submitPendingAntiCompaction(ColumnFamilyStore cfs,
+                                                           RangesAtEndpoint tokenRanges,
+                                                           Refs<SSTableReader> sstables,
+                                                           LifecycleTransaction txn,
+                                                           UUID sessionId)
     {
         Runnable runnable = new WrappedRunnable()
         {
@@ -651,7 +660,7 @@ public class CompactionManager implements CompactionManagerMBean
             {
                 try (TableMetrics.TableTimer.Context ctx = cfs.metric.anticompactionTime.time())
                 {
-                    performAnticompaction(cfs, ranges, sstables, txn, ActiveRepairService.UNREPAIRED_SSTABLE, sessionId, sessionId);
+                    performAnticompaction(cfs, tokenRanges, sstables, txn, sessionId);
                 }
             }
         };
@@ -672,49 +681,70 @@ public class CompactionManager implements CompactionManagerMBean
         }
     }
 
+    /**
+     * for sstables that are fully contained in the given ranges, just rewrite their metadata with
+     * the pending repair id and remove them from the transaction
+     */
+    private static void mutateFullyContainedSSTables(ColumnFamilyStore cfs,
+                                                     Refs<SSTableReader> refs,
+                                                     Iterator<SSTableReader> sstableIterator,
+                                                     Collection<Range<Token>> ranges,
+                                                     LifecycleTransaction txn,
+                                                     UUID sessionID,
+                                                     boolean isTransient) throws IOException
+    {
+        if (ranges.isEmpty())
+            return;
+
+        List<Range<Token>> normalizedRanges = Range.normalize(ranges);
+
+        Set<SSTableReader> fullyContainedSSTables = findSSTablesToAnticompact(sstableIterator, normalizedRanges, sessionID);
+
+        cfs.metric.bytesMutatedAnticompaction.inc(SSTableReader.getTotalBytes(fullyContainedSSTables));
+        cfs.getCompactionStrategyManager().mutateRepaired(fullyContainedSSTables, UNREPAIRED_SSTABLE, sessionID, isTransient);
+        // since we're just re-writing the sstable metdata for the fully contained sstables, we don't want
+        // them obsoleted when the anti-compaction is complete. So they're removed from the transaction here
+        txn.cancel(fullyContainedSSTables);
+        refs.release(fullyContainedSSTables);
+    }
+
     /**
      * Make sure the {validatedForRepair} are marked for compaction before calling this.
      *
      * Caller must reference the validatedForRepair sstables (via ParentRepairSession.getActiveRepairedSSTableRefs(..)).
      *
      * @param cfs
-     * @param ranges Ranges that the repair was carried out on
+     * @param ranges token ranges to be repaired
      * @param validatedForRepair SSTables containing the repaired ranges. Should be referenced before passing them.
-     * @param parentRepairSession parent repair session ID
+     * @param sessionID the repair session we're anti-compacting for
      * @throws InterruptedException
      * @throws IOException
      */
     public void performAnticompaction(ColumnFamilyStore cfs,
-                                      Collection<Range<Token>> ranges,
+                                      RangesAtEndpoint ranges,
                                       Refs<SSTableReader> validatedForRepair,
                                       LifecycleTransaction txn,
-                                      long repairedAt,
+                                      UUID sessionID) throws IOException
-                                      UUID pendingRepair,
-                                      UUID parentRepairSession) throws InterruptedException, IOException
     {
         try
         {
-            ActiveRepairService.ParentRepairSession prs = ActiveRepairService.instance.getParentRepairSession(parentRepairSession);
+            ActiveRepairService.ParentRepairSession prs = ActiveRepairService.instance.getParentRepairSession(sessionID);
             Preconditions.checkArgument(!prs.isPreview(), "Cannot anticompact for previews");
+            Preconditions.checkArgument(!ranges.isEmpty(), "No ranges to anti-compact");
 
             if (logger.isInfoEnabled())
-                logger.info("{} Starting anticompaction for {}.{} on {}/{} sstables", PreviewKind.NONE.logPrefix(parentRepairSession), cfs.keyspace.getName(), cfs.getTableName(), validatedForRepair.size(), cfs.getLiveSSTables().size());
+                logger.info("{} Starting anticompaction for {}.{} on {}/{} sstables", PreviewKind.NONE.logPrefix(sessionID), cfs.keyspace.getName(), cfs.getTableName(), validatedForRepair.size(), cfs.getLiveSSTables().size());
             if (logger.isTraceEnabled())
-                logger.trace("{} Starting anticompaction for ranges {}", PreviewKind.NONE.logPrefix(parentRepairSession), ranges);
+                logger.trace("{} Starting anticompaction for ranges {}", PreviewKind.NONE.logPrefix(sessionID), ranges);
+
             Set<SSTableReader> sstables = new HashSet<>(validatedForRepair);
+            validateSSTableBoundsForAnticompaction(sessionID, sstables, ranges);
+            mutateFullyContainedSSTables(cfs, validatedForRepair, sstables.iterator(), ranges.fullRanges(), txn, sessionID, false);
+            mutateFullyContainedSSTables(cfs, validatedForRepair, sstables.iterator(), ranges.transientRanges(), txn, sessionID, true);
 
-            Iterator<SSTableReader> sstableIterator = sstables.iterator();
-            List<Range<Token>> normalizedRanges = Range.normalize(ranges);
-
-            Set<SSTableReader> fullyContainedSSTables = findSSTablesToAnticompact(sstableIterator, normalizedRanges, parentRepairSession);
-
-            cfs.metric.bytesMutatedAnticompaction.inc(SSTableReader.getTotalBytes(fullyContainedSSTables));
-            cfs.getCompactionStrategyManager().mutateRepaired(fullyContainedSSTables, repairedAt, pendingRepair);
-            txn.cancel(fullyContainedSSTables);
-            validatedForRepair.release(fullyContainedSSTables);
             assert txn.originals().equals(sstables);
             if (!sstables.isEmpty())
-                doAntiCompaction(cfs, ranges, txn, repairedAt, pendingRepair);
+                doAntiCompaction(cfs, ranges, txn, sessionID);
             txn.finish();
         }
         finally
@@ -723,7 +753,28 @@ public class CompactionManager implements CompactionManagerMBean
             txn.close();
         }
 
-        logger.info("{} Completed anticompaction successfully", PreviewKind.NONE.logPrefix(parentRepairSession));
+        logger.info("{} Completed anticompaction successfully", PreviewKind.NONE.logPrefix(sessionID));
+    }
+
+    static void validateSSTableBoundsForAnticompaction(UUID sessionID,
+                                                       Collection<SSTableReader> sstables,
+                                                       RangesAtEndpoint ranges)
+    {
+        List<Range<Token>> normalizedRanges = Range.normalize(ranges.ranges());
+        for (SSTableReader sstable : sstables)
+        {
+            Bounds<Token> bounds = new Bounds<>(sstable.first.getToken(), sstable.last.getToken());
+
+            if (!Iterables.any(normalizedRanges, r -> (r.contains(bounds.left) && r.contains(bounds.right)) || r.intersects(bounds)))
+            {
+                // this should never happen - in PendingAntiCompaction#getSSTables we select all sstables that intersect the repaired ranges, that can't have changed here
+                String message = String.format("%s SSTable %s (%s) does not intersect repaired ranges %s, this sstable should not have been included.",
+                                               PreviewKind.NONE.logPrefix(sessionID), sstable, bounds, normalizedRanges);
+                logger.error(message);
+                throw new IllegalStateException(message);
+            }
+        }
+
     }
 
     @VisibleForTesting
@@ -736,8 +787,6 @@ public class CompactionManager implements CompactionManagerMBean
 
             Bounds<Token> sstableBounds = new Bounds<>(sstable.first.getToken(), sstable.last.getToken());
 
-            boolean shouldAnticompact = false;
-
             for (Range<Token> r : normalizedRanges)
             {
                 // ranges are normalized - no wrap around - if first and last are contained we know that all tokens are contained in the range
@@ -746,23 +795,13 @@ public class CompactionManager implements CompactionManagerMBean
                     logger.info("{} SSTable {} fully contained in range {}, mutating repairedAt instead of anticompacting", PreviewKind.NONE.logPrefix(parentRepairSession), sstable, r);
                     fullyContainedSSTables.add(sstable);
                     sstableIterator.remove();
-                    shouldAnticompact = true;
                     break;
                 }
                 else if (r.intersects(sstableBounds))
                 {
                     logger.info("{} SSTable {} ({}) will be anticompacted on range {}", PreviewKind.NONE.logPrefix(parentRepairSession), sstable, sstableBounds, r);
-                    shouldAnticompact = true;
                 }
             }
-
-            if (!shouldAnticompact)
-            {
-                // this should never happen - in PendingAntiCompaction#getSSTables we select all sstables that intersect the repaired ranges, that can't have changed here
-                String message = String.format("%s SSTable %s (%s) does not intersect repaired ranges %s, this sstable should not have been included.", PreviewKind.NONE.logPrefix(parentRepairSession), sstable, sstableBounds, normalizedRanges);
-                logger.error(message);
-                throw new IllegalStateException(message);
-            }
         }
         return fullyContainedSSTables;
     }
@@ -914,7 +953,10 @@ public class CompactionManager implements CompactionManagerMBean
         {
             ColumnFamilyStore cfs = entry.getKey();
             Keyspace keyspace = cfs.keyspace;
-            Collection<Range<Token>> ranges = StorageService.instance.getLocalRanges(keyspace.getName());
+            final RangesAtEndpoint replicas = StorageService.instance.getLocalReplicas(keyspace.getName());
+            final Set<Range<Token>> allRanges = replicas.ranges();
+            final Set<Range<Token>> transientRanges = replicas.filter(Replica::isTransient).ranges();
+            final Set<Range<Token>> fullRanges = replicas.filter(Replica::isFull).ranges();
             boolean hasIndexes = cfs.indexManager.hasIndexes();
             SSTableReader sstable = lookupSSTable(cfs, entry.getValue());
 
@@ -924,10 +966,10 @@ public class CompactionManager implements CompactionManagerMBean
             }
             else
             {
-                CleanupStrategy cleanupStrategy = CleanupStrategy.get(cfs, ranges, FBUtilities.nowInSeconds());
+                CleanupStrategy cleanupStrategy = CleanupStrategy.get(cfs, allRanges, transientRanges, sstable.isRepaired(), FBUtilities.nowInSeconds());
                 try (LifecycleTransaction txn = cfs.getTracker().tryModify(sstable, OperationType.CLEANUP))
                 {
-                    doCleanupOne(cfs, txn, cleanupStrategy, ranges, hasIndexes);
+                    doCleanupOne(cfs, txn, cleanupStrategy, allRanges, fullRanges, transientRanges, hasIndexes);
                 }
                 catch (IOException e)
                 {
@@ -1104,22 +1146,33 @@ public class CompactionManager implements CompactionManagerMBean
      *
      * @throws IOException
      */
-    private void doCleanupOne(final ColumnFamilyStore cfs, LifecycleTransaction txn, CleanupStrategy cleanupStrategy, Collection<Range<Token>> ranges, boolean hasIndexes) throws IOException
+    private void doCleanupOne(final ColumnFamilyStore cfs,
+                              LifecycleTransaction txn,
+                              CleanupStrategy cleanupStrategy,
+                              Collection<Range<Token>> allRanges,
+                              Collection<Range<Token>> fullRanges,
+                              Collection<Range<Token>> transientRanges,
+                              boolean hasIndexes) throws IOException
     {
         assert !cfs.isIndex();
 
         SSTableReader sstable = txn.onlyOne();
 
         // if ranges is empty and no index, entire sstable is discarded
-        if (!hasIndexes && !new Bounds<>(sstable.first.getToken(), sstable.last.getToken()).intersects(ranges))
+        if (!hasIndexes && !new Bounds<>(sstable.first.getToken(), sstable.last.getToken()).intersects(allRanges))
         {
             txn.obsoleteOriginals();
             txn.finish();
             return;
         }
-        if (!needsCleanup(sstable, ranges))
+
+        boolean needsCleanupFull = needsCleanup(sstable, fullRanges);
+        boolean needsCleanupTransient = needsCleanup(sstable, transientRanges);
+        //If there are no ranges for which the table needs cleanup either due to lack of intersection or lack
+        //of the table being repaired.
+        if (!needsCleanupFull && (!needsCleanupTransient || !sstable.isRepaired()))
         {
-            logger.trace("Skipping {} for cleanup; all rows should be kept", sstable);
+            logger.trace("Skipping {} for cleanup; all rows should be kept. Needs cleanup full ranges: {} Needs cleanup transient ranges: {} Repaired: {}", sstable, needsCleanupFull, needsCleanupTransient, sstable.isRepaired());
             return;
         }
 
@@ -1150,7 +1203,7 @@ public class CompactionManager implements CompactionManagerMBean
              CompactionIterator ci = new CompactionIterator(OperationType.CLEANUP, Collections.singletonList(scanner), controller, nowInSec, UUIDGen.getTimeUUID(), metrics))
         {
             StatsMetadata metadata = sstable.getSSTableMetadata();
-            writer.switchWriter(createWriter(cfs, compactionFileLocation, expectedBloomFilterSize, metadata.repairedAt, metadata.pendingRepair, sstable, txn));
+            writer.switchWriter(createWriter(cfs, compactionFileLocation, expectedBloomFilterSize, metadata.repairedAt, metadata.pendingRepair, metadata.isTransient, sstable, txn));
             long lastBytesScanned = 0;
 
             while (ci.hasNext())
@@ -1218,11 +1271,18 @@ public class CompactionManager implements CompactionManagerMBean
             this.nowInSec = nowInSec;
         }
 
-        public static CleanupStrategy get(ColumnFamilyStore cfs, Collection<Range<Token>> ranges, int nowInSec)
+        public static CleanupStrategy get(ColumnFamilyStore cfs, Collection<Range<Token>> ranges, Collection<Range<Token>> transientRanges, boolean isRepaired, int nowInSec)
         {
-            return cfs.indexManager.hasIndexes()
+            if (cfs.indexManager.hasIndexes())
-                 ? new Full(cfs, ranges, nowInSec)
+            {
-                 : new Bounded(cfs, ranges, nowInSec);
+                if (!transientRanges.isEmpty())
+                {
+                    //Shouldn't have been possible to create this situation
+                    throw new AssertionError("Can't have indexes and transient ranges");
+                }
+                return new Full(cfs, ranges, nowInSec);
+            }
+            return new Bounded(cfs, ranges, transientRanges, isRepaired, nowInSec);
         }
 
         public abstract ISSTableScanner getScanner(SSTableReader sstable);
@@ -1230,7 +1290,10 @@ public class CompactionManager implements CompactionManagerMBean
 
         private static final class Bounded extends CleanupStrategy
         {
-            public Bounded(final ColumnFamilyStore cfs, Collection<Range<Token>> ranges, int nowInSec)
+            private final Collection<Range<Token>> transientRanges;
+            private final boolean isRepaired;
+
+            public Bounded(final ColumnFamilyStore cfs, Collection<Range<Token>> ranges, Collection<Range<Token>> transientRanges, boolean isRepaired, int nowInSec)
             {
                 super(ranges, nowInSec);
                 cacheCleanupExecutor.submit(new Runnable()
@@ -1241,12 +1304,23 @@ public class CompactionManager implements CompactionManagerMBean
                         cfs.cleanupCache();
                     }
                 });
+                this.transientRanges = transientRanges;
+                this.isRepaired = isRepaired;
             }
 
             @Override
             public ISSTableScanner getScanner(SSTableReader sstable)
             {
-                return sstable.getScanner(ranges);
+                //If transient replication is enabled and there are transient ranges
+                //then cleanup should remove any partitions that are repaired and in the transient range
+                //as they should already be synchronized at other full replicas.
+                //So just don't scan the portion of the table containing the repaired transient ranges
+                Collection<Range<Token>> rangesToScan = ranges;
+                if (isRepaired)
+                {
+                    rangesToScan = Collections2.filter(ranges, range -> !transientRanges.contains(range));
+                }
+                return sstable.getScanner(rangesToScan);
             }
 
             @Override
@@ -1291,6 +1365,7 @@ public class CompactionManager implements CompactionManagerMBean
                                              long expectedBloomFilterSize,
                                              long repairedAt,
                                              UUID pendingRepair,
+                                             boolean isTransient,
                                              SSTableReader sstable,
                                              LifecycleTransaction txn)
     {
@@ -1301,6 +1376,7 @@ public class CompactionManager implements CompactionManagerMBean
                                     expectedBloomFilterSize,
                                     repairedAt,
                                     pendingRepair,
+                                    isTransient,
                                     sstable.getSSTableLevel(),
                                     sstable.header,
                                     cfs.indexManager.listIndexes(),
@@ -1312,6 +1388,7 @@ public class CompactionManager implements CompactionManagerMBean
                                                               int expectedBloomFilterSize,
                                                               long repairedAt,
                                                               UUID pendingRepair,
+                                                              boolean isTransient,
                                                               Collection<SSTableReader> sstables,
                                                               LifecycleTransaction txn)
     {
@@ -1335,6 +1412,7 @@ public class CompactionManager implements CompactionManagerMBean
                                     (long) expectedBloomFilterSize,
                                     repairedAt,
                                     pendingRepair,
+                                    isTransient,
                                     cfs.metadata,
                                     new MetadataCollector(sstables, cfs.metadata().comparator, minLevel),
                                     SerializationHeader.make(cfs.metadata(), sstables),
@@ -1347,16 +1425,19 @@ public class CompactionManager implements CompactionManagerMBean
      * will store the non-repaired ranges. Once anticompation is completed, the original sstable is marked as compacted
      * and subsequently deleted.
      * @param cfs
-     * @param repaired a transaction over the repaired sstables to anticompacy
+     * @param txn a transaction over the repaired sstables to anticompact
-     * @param ranges Repaired ranges to be placed into one of the new sstables. The repaired table will be tracked via
+     * @param ranges full and transient ranges to be placed into one of the new sstables. The repaired table will be tracked via
-     * the {@link org.apache.cassandra.io.sstable.metadata.StatsMetadata#repairedAt} field.
+     *   the {@link org.apache.cassandra.io.sstable.metadata.StatsMetadata#pendingRepair} field.
      */
-    private void doAntiCompaction(ColumnFamilyStore cfs, Collection<Range<Token>> ranges, LifecycleTransaction repaired, long repairedAt, UUID pendingRepair)
+    private void doAntiCompaction(ColumnFamilyStore cfs,
+                                  RangesAtEndpoint ranges,
+                                  LifecycleTransaction txn,
+                                  UUID pendingRepair)
     {
-        logger.info("Performing anticompaction on {} sstables", repaired.originals().size());
+        logger.info("Performing anticompaction on {} sstables", txn.originals().size());
 
         //Group SSTables
-        Set<SSTableReader> sstables = repaired.originals();
+        Set<SSTableReader> sstables = txn.originals();
 
         // Repairs can take place on both unrepaired (incremental + full) and repaired (full) data.
         // Although anti-compaction could work on repaired sstables as well and would result in having more accurate
@@ -1366,101 +1447,111 @@ public class CompactionManager implements CompactionManagerMBean
         cfs.metric.bytesAnticompacted.inc(SSTableReader.getTotalBytes(unrepairedSSTables));
         Collection<Collection<SSTableReader>> groupedSSTables = cfs.getCompactionStrategyManager().groupSSTablesForAntiCompaction(unrepairedSSTables);
 
-        // iterate over sstables to check if the repaired / unrepaired ranges intersect them.
+        // iterate over sstables to check if the full / transient / unrepaired ranges intersect them.
         int antiCompactedSSTableCount = 0;
         for (Collection<SSTableReader> sstableGroup : groupedSSTables)
         {
-            try (LifecycleTransaction txn = repaired.split(sstableGroup))
+            try (LifecycleTransaction groupTxn = txn.split(sstableGroup))
             {
-                int antiCompacted = antiCompactGroup(cfs, ranges, txn, repairedAt, pendingRepair);
+                int antiCompacted = antiCompactGroup(cfs, ranges, groupTxn, pendingRepair);
                 antiCompactedSSTableCount += antiCompacted;
             }
         }
 
         String format = "Anticompaction completed successfully, anticompacted from {} to {} sstable(s).";
-        logger.info(format, repaired.originals().size(), antiCompactedSSTableCount);
+        logger.info(format, txn.originals().size(), antiCompactedSSTableCount);
     }
 
-    private int antiCompactGroup(ColumnFamilyStore cfs, Collection<Range<Token>> ranges,
+    private int antiCompactGroup(ColumnFamilyStore cfs,
-                             LifecycleTransaction anticompactionGroup, long repairedAt, UUID pendingRepair)
+                                 RangesAtEndpoint ranges,
+                                 LifecycleTransaction txn,
+                                 UUID pendingRepair)
     {
+        Preconditions.checkArgument(!ranges.isEmpty(), "need at least one full or transient range");
         long groupMaxDataAge = -1;
 
-        for (Iterator<SSTableReader> i = anticompactionGroup.originals().iterator(); i.hasNext();)
+        for (Iterator<SSTableReader> i = txn.originals().iterator(); i.hasNext();)
         {
             SSTableReader sstable = i.next();
             if (groupMaxDataAge < sstable.maxDataAge)
                 groupMaxDataAge = sstable.maxDataAge;
         }
 
-        if (anticompactionGroup.originals().size() == 0)
+        if (txn.originals().size() == 0)
         {
             logger.info("No valid anticompactions for this group, All sstables were compacted and are no longer available");
             return 0;
         }
 
-        logger.info("Anticompacting {}", anticompactionGroup);
+        logger.info("Anticompacting {}", txn);
-        Set<SSTableReader> sstableAsSet = anticompactionGroup.originals();
+        Set<SSTableReader> sstableAsSet = txn.originals();
 
         File destination = cfs.getDirectories().getWriteableLocationAsFile(cfs.getExpectedCompactedFileSize(sstableAsSet, OperationType.ANTICOMPACTION));
-        long repairedKeyCount = 0;
-        long unrepairedKeyCount = 0;
         int nowInSec = FBUtilities.nowInSeconds();
 
         CompactionStrategyManager strategy = cfs.getCompactionStrategyManager();
-        try (SSTableRewriter repairedSSTableWriter = SSTableRewriter.constructWithoutEarlyOpening(anticompactionGroup, false, groupMaxDataAge);
+        try (SSTableRewriter fullWriter = SSTableRewriter.constructWithoutEarlyOpening(txn, false, groupMaxDataAge);
-             SSTableRewriter unRepairedSSTableWriter = SSTableRewriter.constructWithoutEarlyOpening(anticompactionGroup, false, groupMaxDataAge);
+             SSTableRewriter transWriter = SSTableRewriter.constructWithoutEarlyOpening(txn, false, groupMaxDataAge);
-             AbstractCompactionStrategy.ScannerList scanners = strategy.getScanners(anticompactionGroup.originals());
+             SSTableRewriter unrepairedWriter = SSTableRewriter.constructWithoutEarlyOpening(txn, false, groupMaxDataAge);
+
+             AbstractCompactionStrategy.ScannerList scanners = strategy.getScanners(txn.originals());
              CompactionController controller = new CompactionController(cfs, sstableAsSet, getDefaultGcBefore(cfs, nowInSec));
              CompactionIterator ci = new CompactionIterator(OperationType.ANTICOMPACTION, scanners.scanners, controller, nowInSec, UUIDGen.getTimeUUID(), metrics))
         {
             int expectedBloomFilterSize = Math.max(cfs.metadata().params.minIndexInterval, (int)(SSTableReader.getApproximateKeyCount(sstableAsSet)));
 
-            repairedSSTableWriter.switchWriter(CompactionManager.createWriterForAntiCompaction(cfs, destination, expectedBloomFilterSize, repairedAt, pendingRepair, sstableAsSet, anticompactionGroup));
+            fullWriter.switchWriter(CompactionManager.createWriterForAntiCompaction(cfs, destination, expectedBloomFilterSize, UNREPAIRED_SSTABLE, pendingRepair, false, sstableAsSet, txn));
-            unRepairedSSTableWriter.switchWriter(CompactionManager.createWriterForAntiCompaction(cfs, destination, expectedBloomFilterSize, ActiveRepairService.UNREPAIRED_SSTABLE, null, sstableAsSet, anticompactionGroup));
+            transWriter.switchWriter(CompactionManager.createWriterForAntiCompaction(cfs, destination, expectedBloomFilterSize, UNREPAIRED_SSTABLE, pendingRepair, true, sstableAsSet, txn));
-            Range.OrderedRangeContainmentChecker containmentChecker = new Range.OrderedRangeContainmentChecker(ranges);
+            unrepairedWriter.switchWriter(CompactionManager.createWriterForAntiCompaction(cfs, destination, expectedBloomFilterSize, UNREPAIRED_SSTABLE, NO_PENDING_REPAIR, false, sstableAsSet, txn));
+
+            Predicate<Token> fullChecker = !ranges.fullRanges().isEmpty() ? new Range.OrderedRangeContainmentChecker(ranges.fullRanges()) : t -> false;
+            Predicate<Token> transChecker = !ranges.transientRanges().isEmpty() ? new Range.OrderedRangeContainmentChecker(ranges.transientRanges()) : t -> false;
             while (ci.hasNext())
             {
                 try (UnfilteredRowIterator partition = ci.next())
                 {
-                    // if current range from sstable is repaired, save it into the new repaired sstable
+                    Token token = partition.partitionKey().getToken();
-                    if (containmentChecker.contains(partition.partitionKey().getToken()))
+                    // if this row is contained in the full or transient ranges, append it to the appropriate sstable
+                    if (fullChecker.test(token))
                     {
-                        repairedSSTableWriter.append(partition);
+                        fullWriter.append(partition);
-                        repairedKeyCount++;
+                    }
+                    else if (transChecker.test(token))
+                    {
+                        transWriter.append(partition);
                     }
-                    // otherwise save into the new 'non-repaired' table
                     else
                     {
-                        unRepairedSSTableWriter.append(partition);
+                        // otherwise, append it to the unrepaired sstable
-                        unrepairedKeyCount++;
+                        unrepairedWriter.append(partition);
                     }
                 }
             }
 
             List<SSTableReader> anticompactedSSTables = new ArrayList<>();
-            // since both writers are operating over the same Transaction, we cannot use the convenience Transactional.finish() method,
+            // since all writers are operating over the same Transaction, we cannot use the convenience Transactional.finish() method,
             // as on the second finish() we would prepareToCommit() on a Transaction that has already been committed, which is forbidden by the API
             // (since it indicates misuse). We call permitRedundantTransitions so that calls that transition to a state already occupied are permitted.
-            anticompactionGroup.permitRedundantTransitions();
+            txn.permitRedundantTransitions();
-            repairedSSTableWriter.setRepairedAt(repairedAt).prepareToCommit();
+
-            unRepairedSSTableWriter.prepareToCommit();
+            fullWriter.prepareToCommit();
-            anticompactedSSTables.addAll(repairedSSTableWriter.finished());
+            transWriter.prepareToCommit();
-            anticompactedSSTables.addAll(unRepairedSSTableWriter.finished());
+            unrepairedWriter.prepareToCommit();
-            repairedSSTableWriter.commit();
+
-            unRepairedSSTableWriter.commit();
+            anticompactedSSTables.addAll(fullWriter.finished());
+            anticompactedSSTables.addAll(transWriter.finished());
+            anticompactedSSTables.addAll(unrepairedWriter.finished());
+
+            fullWriter.commit();
+            transWriter.commit();
+            unrepairedWriter.commit();
 
-            logger.trace("Repaired {} keys out of {} for {}/{} in {}", repairedKeyCount,
-                                                                       repairedKeyCount + unrepairedKeyCount,
-                                                                       cfs.keyspace.getName(),
-                                                                       cfs.getTableName(),
-                                                                       anticompactionGroup);
             return anticompactedSSTables.size();
         }
         catch (Throwable e)
         {
             JVMStabilityInspector.inspectThrowable(e);
-            logger.error("Error anticompacting " + anticompactionGroup, e);
+            logger.error("Error anticompacting " + txn, e);
         }
         return 0;
     }

src/java/org/apache/cassandra/db/compaction/CompactionStrategyHolder.java

@@ -24,6 +24,7 @@ import java.util.List;
 import java.util.UUID;
 
 import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterables;
 
 import org.apache.cassandra.db.ColumnFamilyStore;
 import org.apache.cassandra.db.SerializationHeader;
@@ -71,11 +72,19 @@ public class CompactionStrategyHolder extends AbstractStrategyHolder
     }
 
     @Override
-    public boolean managesRepairedGroup(boolean isRepaired, boolean isPendingRepair)
+    public boolean managesRepairedGroup(boolean isRepaired, boolean isPendingRepair, boolean isTransient)
     {
-        Preconditions.checkArgument(!isPendingRepair || !isRepaired,
+        if (!isPendingRepair)
-                                    "SSTables cannot be both repaired and pending repair");
+        {
-        return !isPendingRepair && (isRepaired == this.isRepaired);
+            Preconditions.checkArgument(!isTransient, "isTransient can only be true for sstables pending repairs");
+            return this.isRepaired == isRepaired;
+        }
+        else
+        {
+            Preconditions.checkArgument(!isRepaired, "SSTables cannot be both repaired and pending repair");
+            return false;
+
+        }
     }
 
     @Override
@@ -206,7 +215,15 @@ public class CompactionStrategyHolder extends AbstractStrategyHolder
     }
 
     @Override
-    public SSTableMultiWriter createSSTableMultiWriter(Descriptor descriptor, long keyCount, long repairedAt, UUID pendingRepair, MetadataCollector collector, SerializationHeader header, Collection<Index> indexes, LifecycleTransaction txn)
+    public SSTableMultiWriter createSSTableMultiWriter(Descriptor descriptor,
+                                                       long keyCount,
+                                                       long repairedAt,
+                                                       UUID pendingRepair,
+                                                       boolean isTransient,
+                                                       MetadataCollector collector,
+                                                       SerializationHeader header,
+                                                       Collection<Index> indexes,
+                                                       LifecycleTransaction txn)
     {
         if (isRepaired)
         {
@@ -226,6 +243,7 @@ public class CompactionStrategyHolder extends AbstractStrategyHolder
                                                  keyCount,
                                                  repairedAt,
                                                  pendingRepair,
+                                                 isTransient,
                                                  collector,
                                                  header,
                                                  indexes,
@@ -237,4 +255,10 @@ public class CompactionStrategyHolder extends AbstractStrategyHolder
     {
         return strategies.indexOf(strategy);
     }
+
+    @Override
+    public boolean containsSSTable(SSTableReader sstable)
+    {
+        return Iterables.any(strategies, acs -> acs.getSSTables().contains(sstable));
+    }
 }

src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java

@@ -56,6 +56,7 @@ import org.apache.cassandra.index.Index;
 import org.apache.cassandra.io.sstable.Component;
 import org.apache.cassandra.io.sstable.Descriptor;
 import org.apache.cassandra.io.sstable.ISSTableScanner;
+import org.apache.cassandra.io.sstable.SSTable;
 import org.apache.cassandra.io.sstable.SSTableMultiWriter;
 import org.apache.cassandra.io.sstable.format.SSTableReader;
 import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
@@ -112,6 +113,7 @@ public class CompactionStrategyManager implements INotificationConsumer
     /**
      * Variables guarded by read and write lock above
      */
+    private final PendingRepairHolder transientRepairs;
     private final PendingRepairHolder pendingRepairs;
     private final CompactionStrategyHolder repaired;
     private final CompactionStrategyHolder unrepaired;
@@ -156,10 +158,11 @@ public class CompactionStrategyManager implements INotificationConsumer
                 return compactionStrategyIndexForDirectory(descriptor);
             }
         };
-        pendingRepairs = new PendingRepairHolder(cfs, router);
+        transientRepairs = new PendingRepairHolder(cfs, router, true);
+        pendingRepairs = new PendingRepairHolder(cfs, router, false);
         repaired = new CompactionStrategyHolder(cfs, router, true);
         unrepaired = new CompactionStrategyHolder(cfs, router, false);
-        holders = ImmutableList.of(pendingRepairs, repaired, unrepaired);
+        holders = ImmutableList.of(transientRepairs, pendingRepairs, repaired, unrepaired);
 
         cfs.getTracker().subscribe(this);
         logger.trace("{} subscribed to the data tracker.", this);
@@ -176,7 +179,6 @@ public class CompactionStrategyManager implements INotificationConsumer
      * Return the next background task
      *
      * Returns a task for the compaction strategy that needs it the most (most estimated remaining tasks)
-     *
      */
     public AbstractCompactionTask getNextBackgroundTask(int gcBefore)
     {
@@ -188,18 +190,16 @@ public class CompactionStrategyManager implements INotificationConsumer
                 return null;
 
             int numPartitions = getNumTokenPartitions();
+
             // first try to promote/demote sstables from completed repairs
-            List<TaskSupplier> repairFinishedSuppliers = pendingRepairs.getRepairFinishedTaskSuppliers();
+            AbstractCompactionTask repairFinishedTask;
-            if (!repairFinishedSuppliers.isEmpty())
+            repairFinishedTask = pendingRepairs.getNextRepairFinishedTask();
-            {
+            if (repairFinishedTask != null)
-                Collections.sort(repairFinishedSuppliers);
+                return repairFinishedTask;
-                for (TaskSupplier supplier : repairFinishedSuppliers)
+
-                {
+            repairFinishedTask = transientRepairs.getNextRepairFinishedTask();
-                    AbstractCompactionTask task = supplier.getTask();
+            if (repairFinishedTask != null)
-                    if (task != null)
+                return repairFinishedTask;
-                        return task;
-                }
-            }
 
             // sort compaction task suppliers by remaining tasks descending
             List<TaskSupplier> suppliers = new ArrayList<>(numPartitions * holders.size());
@@ -393,64 +393,28 @@ public class CompactionStrategyManager implements INotificationConsumer
         }
     }
 
-
-
     @VisibleForTesting
-    List<AbstractCompactionStrategy> getRepaired()
+    CompactionStrategyHolder getRepairedUnsafe()
     {
-        readLock.lock();
+        return repaired;
-        try
-        {
-            return Lists.newArrayList(repaired.allStrategies());
-        }
-        finally
-        {
-            readLock.unlock();
-        }
     }
 
     @VisibleForTesting
-    List<AbstractCompactionStrategy> getUnrepaired()
+    CompactionStrategyHolder getUnrepairedUnsafe()
     {
-        readLock.lock();
+        return unrepaired;
-        try
-        {
-            return Lists.newArrayList(unrepaired.allStrategies());
-        }
-        finally
-        {
-            readLock.unlock();
-        }
     }
 
     @VisibleForTesting
-    Iterable<AbstractCompactionStrategy> getForPendingRepair(UUID sessionID)
+    PendingRepairHolder getPendingRepairsUnsafe()
     {
-        readLock.lock();
+        return pendingRepairs;
-        try
-        {
-            return pendingRepairs.getStrategiesFor(sessionID);
-        }
-        finally
-        {
-            readLock.unlock();
-        }
     }
 
     @VisibleForTesting
-    Set<UUID> pendingRepairs()
+    PendingRepairHolder getTransientRepairsUnsafe()
     {
-        readLock.lock();
+        return transientRepairs;
-        try
-        {
-            Set<UUID> ids = new HashSet<>();
-            pendingRepairs.getManagers().forEach(p -> ids.addAll(p.getSessions()));
-            return ids;
-        }
-        finally
-        {
-            readLock.unlock();
-        }
     }
 
     public boolean hasDataForPendingRepair(UUID sessionID)
@@ -458,8 +422,7 @@ public class CompactionStrategyManager implements INotificationConsumer
         readLock.lock();
         try
         {
-            return Iterables.any(pendingRepairs.getManagers(),
+            return pendingRepairs.hasDataForSession(sessionID) || transientRepairs.hasDataForSession(sessionID);
-                                 prm -> prm.hasDataForSession(sessionID));
         }
         finally
         {
@@ -682,18 +645,19 @@ public class CompactionStrategyManager implements INotificationConsumer
         throw new IllegalStateException("No holder claimed " + sstable);
     }
 
-    private AbstractStrategyHolder getHolder(long repairedAt, UUID pendingRepair)
+    private AbstractStrategyHolder getHolder(long repairedAt, UUID pendingRepair, boolean isTransient)
     {
         return getHolder(repairedAt != ActiveRepairService.UNREPAIRED_SSTABLE,
-                         pendingRepair != ActiveRepairService.NO_PENDING_REPAIR);
+                         pendingRepair != ActiveRepairService.NO_PENDING_REPAIR,
+                         isTransient);
     }
 
     @VisibleForTesting
-    AbstractStrategyHolder getHolder(boolean isRepaired, boolean isPendingRepair)
+    AbstractStrategyHolder getHolder(boolean isRepaired, boolean isPendingRepair, boolean isTransient)
     {
         for (AbstractStrategyHolder holder : holders)
         {
-            if (holder.managesRepairedGroup(isRepaired, isPendingRepair))
+            if (holder.managesRepairedGroup(isRepaired, isPendingRepair, isTransient))
                 return holder;
         }
 
@@ -1146,16 +1110,26 @@ public class CompactionStrategyManager implements INotificationConsumer
                                                        long keyCount,
                                                        long repairedAt,
                                                        UUID pendingRepair,
+                                                       boolean isTransient,
                                                        MetadataCollector collector,
                                                        SerializationHeader header,
                                                        Collection<Index> indexes,
                                                        LifecycleTransaction txn)
     {
+        SSTable.validateRepairedMetadata(repairedAt, pendingRepair, isTransient);
         maybeReloadDiskBoundaries();
         readLock.lock();
         try
         {
-            return getHolder(repairedAt, pendingRepair).createSSTableMultiWriter(descriptor, keyCount, repairedAt, pendingRepair, collector, header, indexes, txn);
+            return getHolder(repairedAt, pendingRepair, isTransient).createSSTableMultiWriter(descriptor,
+                                                                                              keyCount,
+                                                                                              repairedAt,
+                                                                                              pendingRepair,
+                                                                                              isTransient,
+                                                                                              collector,
+                                                                                              header,
+                                                                                              indexes,
+                                                                                              txn);
         }
         finally
         {
@@ -1220,7 +1194,7 @@ public class CompactionStrategyManager implements INotificationConsumer
      * Mutates sstable repairedAt times and notifies listeners of the change with the writeLock held. Prevents races
      * with other processes between when the metadata is changed and when sstables are moved between strategies.
       */
-    public void mutateRepaired(Collection<SSTableReader> sstables, long repairedAt, UUID pendingRepair) throws IOException
+    public void mutateRepaired(Collection<SSTableReader> sstables, long repairedAt, UUID pendingRepair, boolean isTransient) throws IOException
     {
         Set<SSTableReader> changed = new HashSet<>();
 
@@ -1229,7 +1203,7 @@ public class CompactionStrategyManager implements INotificationConsumer
         {
             for (SSTableReader sstable: sstables)
             {
-                sstable.descriptor.getMetadataSerializer().mutateRepaired(sstable.descriptor, repairedAt, pendingRepair);
+                sstable.descriptor.getMetadataSerializer().mutateRepairMetadata(sstable.descriptor, repairedAt, pendingRepair, isTransient);
                 sstable.reloadSSTableMetadata();
                 changed.add(sstable);
             }

src/java/org/apache/cassandra/db/compaction/CompactionTask.java

@@ -29,20 +29,19 @@ import com.google.common.base.Predicate;
 import com.google.common.collect.Iterables;
 import com.google.common.collect.Sets;
 import com.google.common.util.concurrent.RateLimiter;
-
-import org.apache.cassandra.db.Directories;
-import org.apache.cassandra.db.compaction.writers.CompactionAwareWriter;
-import org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter;
-import org.apache.cassandra.io.sstable.format.SSTableReader;
 import org.apache.commons.lang3.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Directories;
 import org.apache.cassandra.db.SystemKeyspace;
 import org.apache.cassandra.db.compaction.CompactionManager.CompactionExecutorStatsCollector;
+import org.apache.cassandra.db.compaction.writers.CompactionAwareWriter;
+import org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter;
 import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
+import org.apache.cassandra.io.sstable.format.SSTableReader;
 import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
 import org.apache.cassandra.service.ActiveRepairService;
 import org.apache.cassandra.utils.FBUtilities;
@@ -339,6 +338,23 @@ public class CompactionTask extends AbstractCompactionTask
         return ids.iterator().next();
     }
 
+    public static boolean getIsTransient(Set<SSTableReader> sstables)
+    {
+        if (sstables.isEmpty())
+        {
+            return false;
+        }
+
+        boolean isTransient = sstables.iterator().next().isTransient();
+
+        if (!Iterables.all(sstables, sstable -> sstable.isTransient() == isTransient))
+        {
+            throw new RuntimeException("Attempting to compact transient sstables with non transient sstables");
+        }
+
+        return isTransient;
+    }
+
 
     /*
      * Checks if we have enough disk space to execute the compaction.  Drops the largest sstable out of the Task until

src/java/org/apache/cassandra/db/compaction/PendingRepairHolder.java

@@ -20,6 +20,7 @@ package org.apache.cassandra.db.compaction;
 
 import java.util.ArrayList;
 import java.util.Collection;
+import java.util.Collections;
 import java.util.List;
 import java.util.UUID;
 
@@ -43,10 +44,12 @@ import org.apache.cassandra.service.ActiveRepairService;
 public class PendingRepairHolder extends AbstractStrategyHolder
 {
     private final List<PendingRepairManager> managers = new ArrayList<>();
+    private final boolean isTransient;
 
-    public PendingRepairHolder(ColumnFamilyStore cfs, DestinationRouter router)
+    public PendingRepairHolder(ColumnFamilyStore cfs, DestinationRouter router, boolean isTransient)
     {
         super(cfs, router);
+        this.isTransient = isTransient;
     }
 
     @Override
@@ -66,15 +69,15 @@ public class PendingRepairHolder extends AbstractStrategyHolder
     {
         managers.clear();
         for (int i = 0; i < numTokenPartitions; i++)
-            managers.add(new PendingRepairManager(cfs, params));
+            managers.add(new PendingRepairManager(cfs, params, isTransient));
     }
 
     @Override
-    public boolean managesRepairedGroup(boolean isRepaired, boolean isPendingRepair)
+    public boolean managesRepairedGroup(boolean isRepaired, boolean isPendingRepair, boolean isTransient)
     {
         Preconditions.checkArgument(!isPendingRepair || !isRepaired,
                                     "SSTables cannot be both repaired and pending repair");
-        return isPendingRepair;
+        return isPendingRepair && (this.isTransient == isTransient);
     }
 
     @Override
@@ -145,7 +148,23 @@ public class PendingRepairHolder extends AbstractStrategyHolder
         return tasks;
     }
 
-    public ArrayList<TaskSupplier> getRepairFinishedTaskSuppliers()
+    AbstractCompactionTask getNextRepairFinishedTask()
+    {
+        List<TaskSupplier> repairFinishedSuppliers = getRepairFinishedTaskSuppliers();
+        if (!repairFinishedSuppliers.isEmpty())
+        {
+            Collections.sort(repairFinishedSuppliers);
+            for (TaskSupplier supplier : repairFinishedSuppliers)
+            {
+                AbstractCompactionTask task = supplier.getTask();
+                if (task != null)
+                    return task;
+            }
+        }
+        return null;
+    }
+
+    private ArrayList<TaskSupplier> getRepairFinishedTaskSuppliers()
     {
         ArrayList<TaskSupplier> suppliers = new ArrayList<>(managers.size());
         for (PendingRepairManager manager : managers)
@@ -218,6 +237,7 @@ public class PendingRepairHolder extends AbstractStrategyHolder
                                                        long keyCount,
                                                        long repairedAt,
                                                        UUID pendingRepair,
+                                                       boolean isTransient,
                                                        MetadataCollector collector,
                                                        SerializationHeader header,
                                                        Collection<Index> indexes,
@@ -233,6 +253,7 @@ public class PendingRepairHolder extends AbstractStrategyHolder
                                                  keyCount,
                                                  repairedAt,
                                                  pendingRepair,
+                                                 isTransient,
                                                  collector,
                                                  header,
                                                  indexes,
@@ -249,4 +270,15 @@ public class PendingRepairHolder extends AbstractStrategyHolder
         }
         return -1;
     }
+
+    public boolean hasDataForSession(UUID sessionID)
+    {
+        return Iterables.any(managers, prm -> prm.hasDataForSession(sessionID));
+    }
+
+    @Override
+    public boolean containsSSTable(SSTableReader sstable)
+    {
+        return Iterables.any(managers, prm -> prm.containsSSTable(sstable));
+    }
 }

src/java/org/apache/cassandra/db/compaction/PendingRepairManager.java

@@ -30,7 +30,9 @@ import java.util.UUID;
 import java.util.stream.Collectors;
 
 import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
 import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
 import com.google.common.collect.Maps;
 
 import org.slf4j.Logger;
@@ -62,6 +64,7 @@ class PendingRepairManager
 
     private final ColumnFamilyStore cfs;
     private final CompactionParams params;
+    private final boolean isTransient;
     private volatile ImmutableMap<UUID, AbstractCompactionStrategy> strategies = ImmutableMap.of();
 
     /**
@@ -75,10 +78,11 @@ class PendingRepairManager
         }
     }
 
-    PendingRepairManager(ColumnFamilyStore cfs, CompactionParams params)
+    PendingRepairManager(ColumnFamilyStore cfs, CompactionParams params, boolean isTransient)
     {
         this.cfs = cfs;
         this.params = params;
+        this.isTransient = isTransient;
     }
 
     private ImmutableMap.Builder<UUID, AbstractCompactionStrategy> mapBuilder()
@@ -156,6 +160,7 @@ class PendingRepairManager
 
     synchronized void addSSTable(SSTableReader sstable)
     {
+        Preconditions.checkArgument(sstable.isTransient() == isTransient);
         getOrCreate(sstable).addSSTable(sstable);
     }
 
@@ -389,6 +394,15 @@ class PendingRepairManager
         return strategies.keySet().contains(sessionID);
     }
 
+    boolean containsSSTable(SSTableReader sstable)
+    {
+        if (!sstable.isPendingRepair())
+            return false;
+
+        AbstractCompactionStrategy strategy = strategies.get(sstable.getPendingRepair());
+        return strategy != null && strategy.getSSTables().contains(sstable);
+    }
+
     public Collection<AbstractCompactionTask> createUserDefinedTasks(Collection<SSTableReader> sstables, int gcBefore)
     {
         Map<UUID, List<SSTableReader>> group = sstables.stream().collect(Collectors.groupingBy(s -> s.getSSTableMetadata().pendingRepair));
@@ -419,18 +433,35 @@ class PendingRepairManager
         protected void runMayThrow() throws Exception
         {
             boolean completed = false;
+            boolean obsoleteSSTables = isTransient && repairedAt > 0;
             try
             {
-                logger.debug("Setting repairedAt to {} on {} for {}", repairedAt, transaction.originals(), sessionID);
+                if (obsoleteSSTables)
-                cfs.getCompactionStrategyManager().mutateRepaired(transaction.originals(), repairedAt, ActiveRepairService.NO_PENDING_REPAIR);
+                {
+                    logger.info("Obsoleting transient repaired ssatbles");
+                    Preconditions.checkState(Iterables.all(transaction.originals(), SSTableReader::isTransient));
+                    transaction.obsoleteOriginals();
+                }
+                else
+                {
+                    logger.debug("Setting repairedAt to {} on {} for {}", repairedAt, transaction.originals(), sessionID);
+                    cfs.getCompactionStrategyManager().mutateRepaired(transaction.originals(), repairedAt, ActiveRepairService.NO_PENDING_REPAIR, false);
+                }
                 completed = true;
             }
             finally
             {
-                // we always abort because mutating metadata isn't guarded by LifecycleTransaction, so this won't roll
+                if (obsoleteSSTables)
-                // anything back. Also, we don't want to obsolete the originals. We're only using it to prevent other
+                {
-                // compactions from marking these sstables compacting, and unmarking them when we're done
+                    transaction.finish();
-                transaction.abort();
+                }
+                else
+                {
+                    // we abort here because mutating metadata isn't guarded by LifecycleTransaction, so this won't roll
+                    // anything back. Also, we don't want to obsolete the originals. We're only using it to prevent other
+                    // compactions from marking these sstables compacting, and unmarking them when we're done
+                    transaction.abort();
+                }
                 if (completed)
                 {
                     removeSession(sessionID);

src/java/org/apache/cassandra/db/compaction/Scrubber.java

@@ -170,7 +170,7 @@ public class Scrubber implements Closeable
             }
 
             StatsMetadata metadata = sstable.getSSTableMetadata();
-            writer.switchWriter(CompactionManager.createWriter(cfs, destination, expectedBloomFilterSize, metadata.repairedAt, metadata.pendingRepair, sstable, transaction));
+            writer.switchWriter(CompactionManager.createWriter(cfs, destination, expectedBloomFilterSize, metadata.repairedAt, metadata.pendingRepair, metadata.isTransient, sstable, transaction));
 
             DecoratedKey prevKey = null;
 
@@ -277,7 +277,7 @@ public class Scrubber implements Closeable
                 // out of order rows, but no bad rows found - we can keep our repairedAt time
                 long repairedAt = badRows > 0 ? ActiveRepairService.UNREPAIRED_SSTABLE : metadata.repairedAt;
                 SSTableReader newInOrderSstable;
-                try (SSTableWriter inOrderWriter = CompactionManager.createWriter(cfs, destination, expectedBloomFilterSize, repairedAt, metadata.pendingRepair, sstable, transaction))
+                try (SSTableWriter inOrderWriter = CompactionManager.createWriter(cfs, destination, expectedBloomFilterSize, repairedAt, metadata.pendingRepair, metadata.isTransient, sstable, transaction))
                 {
                     for (Partition partition : outOfOrder)
                         inOrderWriter.append(partition.unfilteredIterator());

src/java/org/apache/cassandra/db/compaction/Upgrader.java

@@ -68,14 +68,15 @@ public class Upgrader
         this.estimatedRows = (long) Math.ceil((double) estimatedTotalKeys / estimatedSSTables);
     }
 
-    private SSTableWriter createCompactionWriter(long repairedAt, UUID parentRepair)
+    private SSTableWriter createCompactionWriter(StatsMetadata metadata)
     {
         MetadataCollector sstableMetadataCollector = new MetadataCollector(cfs.getComparator());
         sstableMetadataCollector.sstableLevel(sstable.getSSTableLevel());
         return SSTableWriter.create(cfs.newSSTableDescriptor(directory),
                                     estimatedRows,
-                                    repairedAt,
+                                    metadata.repairedAt,
-                                    parentRepair,
+                                    metadata.pendingRepair,
+                                    metadata.isTransient,
                                     cfs.metadata,
                                     sstableMetadataCollector,
                                     SerializationHeader.make(cfs.metadata(), Sets.newHashSet(sstable)),
@@ -91,8 +92,7 @@ public class Upgrader
              AbstractCompactionStrategy.ScannerList scanners = strategyManager.getScanners(transaction.originals());
              CompactionIterator iter = new CompactionIterator(transaction.opType(), scanners.scanners, controller, nowInSec, UUIDGen.getTimeUUID()))
         {
-            StatsMetadata metadata = sstable.getSSTableMetadata();
+            writer.switchWriter(createCompactionWriter(sstable.getSSTableMetadata()));
-            writer.switchWriter(createCompactionWriter(metadata.repairedAt, metadata.pendingRepair));
             while (iter.hasNext())
                 writer.append(iter.next());
 

src/java/org/apache/cassandra/db/compaction/Verifier.java

@@ -350,6 +350,7 @@ public class Verifier implements Closeable
         public RangeOwnHelper(List<Range<Token>> normalizedRanges)
         {
             this.normalizedRanges = normalizedRanges;
+            Range.assertNormalized(normalizedRanges);
         }
 
         /**
@@ -457,7 +458,7 @@ public class Verifier implements Closeable
         {
             try
             {
-                sstable.descriptor.getMetadataSerializer().mutateRepaired(sstable.descriptor, ActiveRepairService.UNREPAIRED_SSTABLE, sstable.getSSTableMetadata().pendingRepair);
+                sstable.descriptor.getMetadataSerializer().mutateRepairMetadata(sstable.descriptor, ActiveRepairService.UNREPAIRED_SSTABLE, sstable.getPendingRepair(), sstable.isTransient());
                 sstable.reloadSSTableMetadata();
                 cfs.getTracker().notifySSTableRepairedStatusChanged(Collections.singleton(sstable));
             }

src/java/org/apache/cassandra/db/compaction/writers/CompactionAwareWriter.java

@@ -57,6 +57,7 @@ public abstract class CompactionAwareWriter extends Transactional.AbstractTransa
     protected final long maxAge;
     protected final long minRepairedAt;
     protected final UUID pendingRepair;
+    protected final boolean isTransient;
 
     protected final SSTableRewriter sstableWriter;
     protected final LifecycleTransaction txn;
@@ -91,6 +92,7 @@ public abstract class CompactionAwareWriter extends Transactional.AbstractTransa
         sstableWriter = SSTableRewriter.construct(cfs, txn, keepOriginals, maxAge);
         minRepairedAt = CompactionTask.getMinRepairedAt(nonExpiredSSTables);
         pendingRepair = CompactionTask.getPendingRepair(nonExpiredSSTables);
+        isTransient = CompactionTask.getIsTransient(nonExpiredSSTables);
         DiskBoundaries db = cfs.getDiskBoundaries();
         diskBoundaries = db.positions;
         locations = db.directories;

src/java/org/apache/cassandra/db/compaction/writers/DefaultCompactionWriter.java

@@ -72,6 +72,7 @@ public class DefaultCompactionWriter extends CompactionAwareWriter
                                                     estimatedTotalKeys,
                                                     minRepairedAt,
                                                     pendingRepair,
+                                                    isTransient,
                                                     cfs.metadata,
                                                     new MetadataCollector(txn.originals(), cfs.metadata().comparator, sstableLevel),
                                                     SerializationHeader.make(cfs.metadata(), nonExpiredSSTables),

src/java/org/apache/cassandra/db/compaction/writers/MajorLeveledCompactionWriter.java

@@ -108,6 +108,7 @@ public class MajorLeveledCompactionWriter extends CompactionAwareWriter
                 keysPerSSTable,
                 minRepairedAt,
                 pendingRepair,
+                isTransient,
                 cfs.metadata,
                 new MetadataCollector(txn.originals(), cfs.metadata().comparator, currentLevel),
                 SerializationHeader.make(cfs.metadata(), txn.originals()),

src/java/org/apache/cassandra/db/compaction/writers/MaxSSTableSizeWriter.java

@@ -111,6 +111,7 @@ public class MaxSSTableSizeWriter extends CompactionAwareWriter
                                                     estimatedTotalKeys / estimatedSSTables,
                                                     minRepairedAt,
                                                     pendingRepair,
+                                                    isTransient,
                                                     cfs.metadata,
                                                     new MetadataCollector(allSSTables, cfs.metadata().comparator, level),
                                                     SerializationHeader.make(cfs.metadata(), nonExpiredSSTables),

src/java/org/apache/cassandra/db/compaction/writers/SplittingSizeTieredCompactionWriter.java

@@ -107,6 +107,7 @@ public class SplittingSizeTieredCompactionWriter extends CompactionAwareWriter
                                                     currentPartitionsToWrite,
                                                     minRepairedAt,
                                                     pendingRepair,
+                                                    isTransient,
                                                     cfs.metadata,
                                                     new MetadataCollector(allSSTables, cfs.metadata().comparator, 0),
                                                     SerializationHeader.make(cfs.metadata(), nonExpiredSSTables),

src/java/org/apache/cassandra/db/partitions/PartitionIterators.java

@@ -82,18 +82,6 @@ public abstract class PartitionIterators
         return new SingletonPartitionIterator(iterator);
     }
 
-    public static void consume(PartitionIterator iterator)
-    {
-        while (iterator.hasNext())
-        {
-            try (RowIterator partition = iterator.next())
-            {
-                while (partition.hasNext())
-                    partition.next();
-            }
-        }
-    }
-
     /**
      * Wraps the provided iterator so it logs the returned rows for debugging purposes.
      * <p>

src/java/org/apache/cassandra/db/repair/CassandraKeyspaceRepairManager.java

@@ -26,8 +26,7 @@ import com.google.common.util.concurrent.ListenableFuture;
 
 import org.apache.cassandra.db.ColumnFamilyStore;
 import org.apache.cassandra.db.Keyspace;
-import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.locator.RangesAtEndpoint;
-import org.apache.cassandra.dht.Token;
 import org.apache.cassandra.repair.KeyspaceRepairManager;
 
 public class CassandraKeyspaceRepairManager implements KeyspaceRepairManager
@@ -40,9 +39,12 @@ public class CassandraKeyspaceRepairManager implements KeyspaceRepairManager
     }
 
     @Override
-    public ListenableFuture prepareIncrementalRepair(UUID sessionID, Collection<ColumnFamilyStore> tables, Collection<Range<Token>> ranges, ExecutorService executor)
+    public ListenableFuture prepareIncrementalRepair(UUID sessionID,
+                                                     Collection<ColumnFamilyStore> tables,
+                                                     RangesAtEndpoint tokenRanges,
+                                                     ExecutorService executor)
     {
-        PendingAntiCompaction pac = new PendingAntiCompaction(sessionID, tables, ranges, executor);
+        PendingAntiCompaction pac = new PendingAntiCompaction(sessionID, tables, tokenRanges, executor);
         return pac.run();
     }
 }

src/java/org/apache/cassandra/db/repair/PendingAntiCompaction.java

@@ -43,6 +43,7 @@ import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
 import org.apache.cassandra.dht.Range;
 import org.apache.cassandra.dht.Token;
 import org.apache.cassandra.io.sstable.format.SSTableReader;
+import org.apache.cassandra.locator.RangesAtEndpoint;
 import org.apache.cassandra.utils.concurrent.Refs;
 
 /**
@@ -126,17 +127,17 @@ public class PendingAntiCompaction
     static class AcquisitionCallback implements AsyncFunction<List<AcquireResult>, Object>
     {
         private final UUID parentRepairSession;
-        private final Collection<Range<Token>> ranges;
+        private final RangesAtEndpoint tokenRanges;
 
-        public AcquisitionCallback(UUID parentRepairSession, Collection<Range<Token>> ranges)
+        public AcquisitionCallback(UUID parentRepairSession, RangesAtEndpoint tokenRanges)
         {
             this.parentRepairSession = parentRepairSession;
-            this.ranges = ranges;
+            this.tokenRanges = tokenRanges;
         }
 
         ListenableFuture<?> submitPendingAntiCompaction(AcquireResult result)
         {
-            return CompactionManager.instance.submitPendingAntiCompaction(result.cfs, ranges, result.refs, result.txn, parentRepairSession);
+            return CompactionManager.instance.submitPendingAntiCompaction(result.cfs, tokenRanges, result.refs, result.txn, parentRepairSession);
         }
 
         public ListenableFuture apply(List<AcquireResult> results) throws Exception
@@ -177,14 +178,17 @@ public class PendingAntiCompaction
 
     private final UUID prsId;
     private final Collection<ColumnFamilyStore> tables;
-    private final Collection<Range<Token>> ranges;
+    private final RangesAtEndpoint tokenRanges;
     private final ExecutorService executor;
 
-    public PendingAntiCompaction(UUID prsId, Collection<ColumnFamilyStore> tables, Collection<Range<Token>> ranges, ExecutorService executor)
+    public PendingAntiCompaction(UUID prsId,
+                                 Collection<ColumnFamilyStore> tables,
+                                 RangesAtEndpoint tokenRanges,
+                                 ExecutorService executor)
     {
         this.prsId = prsId;
         this.tables = tables;
-        this.ranges = ranges;
+        this.tokenRanges = tokenRanges;
         this.executor = executor;
     }
 
@@ -194,12 +198,12 @@ public class PendingAntiCompaction
         for (ColumnFamilyStore cfs : tables)
         {
             cfs.forceBlockingFlush();
-            ListenableFutureTask<AcquireResult> task = ListenableFutureTask.create(new AcquisitionCallable(cfs, ranges, prsId));
+            ListenableFutureTask<AcquireResult> task = ListenableFutureTask.create(new AcquisitionCallable(cfs, tokenRanges.ranges(), prsId));
             executor.submit(task);
             tasks.add(task);
         }
         ListenableFuture<List<AcquireResult>> acquisitionResults = Futures.successfulAsList(tasks);
-        ListenableFuture compactionResult = Futures.transformAsync(acquisitionResults, new AcquisitionCallback(prsId, ranges), MoreExecutors.directExecutor());
+        ListenableFuture compactionResult = Futures.transformAsync(acquisitionResults, new AcquisitionCallback(prsId, tokenRanges), MoreExecutors.directExecutor());
         return compactionResult;
     }
 }

src/java/org/apache/cassandra/db/streaming/CassandraOutgoingFile.java

@@ -68,17 +68,18 @@ public class CassandraOutgoingFile implements OutgoingStream
     private final ComponentManifest manifest;
     private Boolean isFullyContained;
 
-    private final List<Range<Token>> ranges;
+    private final List<Range<Token>> normalizedRanges;
 
     public CassandraOutgoingFile(StreamOperation operation, Ref<SSTableReader> ref,
-                                 List<SSTableReader.PartitionPositionBounds> sections, Collection<Range<Token>> ranges,
+                                 List<SSTableReader.PartitionPositionBounds> sections, List<Range<Token>> normalizedRanges,
                                  long estimatedKeys)
     {
         Preconditions.checkNotNull(ref.get());
+        Range.assertNormalized(normalizedRanges);
         this.ref = ref;
         this.estimatedKeys = estimatedKeys;
         this.sections = sections;
-        this.ranges = ImmutableList.copyOf(ranges);
+        this.normalizedRanges = ImmutableList.copyOf(normalizedRanges);
         this.filename = ref.get().getFilename();
         this.manifest = getComponentManifest(ref.get());
 
@@ -194,7 +195,7 @@ public class CassandraOutgoingFile implements OutgoingStream
                                                            .getCompactionStrategyFor(ref.get());
 
         if (compactionStrategy instanceof LeveledCompactionStrategy)
-            return contained(ranges, ref.get());
+            return contained(normalizedRanges, ref.get());
 
         return false;
     }
@@ -251,6 +252,6 @@ public class CassandraOutgoingFile implements OutgoingStream
     @Override
     public String toString()
     {
-        return "CassandraOutgoingFile{" + ref.get().getFilename() + '}';
+        return "CassandraOutgoingFile{" + filename + '}';
     }
 }

src/java/org/apache/cassandra/db/streaming/CassandraStreamManager.java

@@ -18,19 +18,10 @@
 
 package org.apache.cassandra.db.streaming;
 
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.List;
-import java.util.Set;
-import java.util.UUID;
-
 import com.google.common.base.Predicate;
 import com.google.common.base.Predicates;
 import com.google.common.collect.Iterables;
 import com.google.common.collect.Sets;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
 import org.apache.cassandra.db.ColumnFamilyStore;
 import org.apache.cassandra.db.PartitionPosition;
 import org.apache.cassandra.db.lifecycle.SSTableIntervalTree;
@@ -39,6 +30,8 @@ import org.apache.cassandra.db.lifecycle.View;
 import org.apache.cassandra.dht.Range;
 import org.apache.cassandra.dht.Token;
 import org.apache.cassandra.io.sstable.format.SSTableReader;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replica;
 import org.apache.cassandra.service.ActiveRepairService;
 import org.apache.cassandra.streaming.IncomingStream;
 import org.apache.cassandra.streaming.OutgoingStream;
@@ -49,6 +42,14 @@ import org.apache.cassandra.streaming.TableStreamManager;
 import org.apache.cassandra.streaming.messages.StreamMessageHeader;
 import org.apache.cassandra.utils.concurrent.Ref;
 import org.apache.cassandra.utils.concurrent.Refs;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
 
 /**
  * Implements the streaming interface for the native cassandra storage engine.
@@ -96,14 +97,14 @@ public class CassandraStreamManager implements TableStreamManager
     }
 
     @Override
-    public Collection<OutgoingStream> createOutgoingStreams(StreamSession session, Collection<Range<Token>> ranges, UUID pendingRepair, PreviewKind previewKind)
+    public Collection<OutgoingStream> createOutgoingStreams(StreamSession session, RangesAtEndpoint replicas, UUID pendingRepair, PreviewKind previewKind)
     {
         Refs<SSTableReader> refs = new Refs<>();
         try
         {
-            final List<Range<PartitionPosition>> keyRanges = new ArrayList<>(ranges.size());
+            final List<Range<PartitionPosition>> keyRanges = new ArrayList<>(replicas.size());
-            for (Range<Token> range : ranges)
+            for (Replica replica : replicas)
-                keyRanges.add(Range.makeRowRange(range));
+                keyRanges.add(Range.makeRowRange(replica.range()));
             refs.addAll(cfs.selectAndReference(view -> {
                 Set<SSTableReader> sstables = Sets.newHashSet();
                 SSTableIntervalTree intervalTree = SSTableIntervalTree.build(view.select(SSTableSet.CANONICAL));
@@ -141,11 +142,16 @@ public class CassandraStreamManager implements TableStreamManager
             }).refs);
 
 
+            List<Range<Token>> normalizedFullRanges = Range.normalize(replicas.filter(Replica::isFull).ranges());
+            List<Range<Token>> normalizedAllRanges = Range.normalize(replicas.ranges());
+            //Create outgoing file streams for ranges possibly skipping repaired ranges in sstables
             List<OutgoingStream> streams = new ArrayList<>(refs.size());
-            for (SSTableReader sstable: refs)
+            for (SSTableReader sstable : refs)
             {
-                Ref<SSTableReader> ref = refs.get(sstable);
+                List<Range<Token>> ranges = sstable.isRepaired() ? normalizedFullRanges : normalizedAllRanges;
                 List<SSTableReader.PartitionPositionBounds> sections = sstable.getPositionsForRanges(ranges);
+
+                Ref<SSTableReader> ref = refs.get(sstable);
                 if (sections.isEmpty())
                 {
                     ref.release();

src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java

@@ -156,7 +156,7 @@ public class CassandraStreamReader implements IStreamReader
         Preconditions.checkState(streamReceiver instanceof CassandraStreamReceiver);
         LifecycleTransaction txn = CassandraStreamReceiver.fromReceiver(session.getAggregator(tableId)).getTransaction();
 
-        RangeAwareSSTableWriter writer = new RangeAwareSSTableWriter(cfs, estimatedKeys, repairedAt, pendingRepair, format, sstableLevel, totalSize, txn, getHeader(cfs.metadata()));
+        RangeAwareSSTableWriter writer = new RangeAwareSSTableWriter(cfs, estimatedKeys, repairedAt, pendingRepair, false, format, sstableLevel, totalSize, txn, getHeader(cfs.metadata()));
         return writer;
     }
 

src/java/org/apache/cassandra/db/view/TableViews.java

@@ -60,6 +60,11 @@ public class TableViews extends AbstractCollection<View>
         baseTableMetadata = Schema.instance.getTableMetadataRef(id);
     }
 
+    public boolean hasViews()
+    {
+        return !views.isEmpty();
+    }
+
     public int size()
     {
         return views.size();

src/java/org/apache/cassandra/db/view/ViewBuilder.java

@@ -43,6 +43,8 @@ import org.apache.cassandra.db.compaction.CompactionInterruptedException;
 import org.apache.cassandra.db.compaction.CompactionManager;
 import org.apache.cassandra.dht.Range;
 import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.Replicas;
 import org.apache.cassandra.repair.SystemDistributedKeyspace;
 import org.apache.cassandra.service.StorageService;
 import org.apache.cassandra.utils.FBUtilities;
@@ -135,14 +137,15 @@ class ViewBuilder
         }
 
         // Get the local ranges for which the view hasn't already been built nor it's building
-        Set<Range<Token>> newRanges = StorageService.instance.getLocalRanges(ksName)
+        RangesAtEndpoint replicatedRanges = StorageService.instance.getLocalReplicas(ksName);
-                                                             .stream()
+        Replicas.temporaryAssertFull(replicatedRanges);
-                                                             .map(r -> r.subtractAll(builtRanges))
+        Set<Range<Token>> newRanges = replicatedRanges.ranges()
-                                                             .flatMap(Set::stream)
+                                                      .stream()
-                                                             .map(r -> r.subtractAll(pendingRanges.keySet()))
+                                                      .map(r -> r.subtractAll(builtRanges))
-                                                             .flatMap(Set::stream)
+                                                      .flatMap(Set::stream)
-                                                             .collect(Collectors.toSet());
+                                                      .map(r -> r.subtractAll(pendingRanges.keySet()))
-
+                                                      .flatMap(Set::stream)
+                                                      .collect(Collectors.toSet());
         // If there are no new nor pending ranges we should finish the build
         if (newRanges.isEmpty() && pendingRanges.isEmpty())
         {

src/java/org/apache/cassandra/db/view/ViewManager.java

@@ -79,7 +79,7 @@ public class ViewManager
             {
                 assert keyspace.getName().equals(update.metadata().keyspace);
 
-                if (coordinatorBatchlog && keyspace.getReplicationStrategy().getReplicationFactor() == 1)
+                if (coordinatorBatchlog && keyspace.getReplicationStrategy().getReplicationFactor().allReplicas == 1)
                     continue;
 
                 if (!forTable(update.metadata().id).updatedViews(update).isEmpty())

src/java/org/apache/cassandra/db/view/ViewUtils.java

@@ -18,16 +18,17 @@
 
 package org.apache.cassandra.db.view;
 
-import java.util.ArrayList;
-import java.util.List;
 import java.util.Optional;
+import java.util.function.Predicate;
 
+import com.google.common.collect.Iterables;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.db.Keyspace;
 import org.apache.cassandra.dht.Token;
 import org.apache.cassandra.locator.AbstractReplicationStrategy;
-import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.EndpointsForToken;
 import org.apache.cassandra.locator.NetworkTopologyStrategy;
+import org.apache.cassandra.locator.Replica;
 import org.apache.cassandra.utils.FBUtilities;
 
 public final class ViewUtils
@@ -58,46 +59,51 @@ public final class ViewUtils
      *
      * @return Optional.empty() if this method is called using a base token which does not belong to this replica
      */
-    public static Optional<InetAddressAndPort> getViewNaturalEndpoint(String keyspaceName, Token baseToken, Token viewToken)
+    public static Optional<Replica> getViewNaturalEndpoint(String keyspaceName, Token baseToken, Token viewToken)
     {
         AbstractReplicationStrategy replicationStrategy = Keyspace.open(keyspaceName).getReplicationStrategy();
 
         String localDataCenter = DatabaseDescriptor.getEndpointSnitch().getDatacenter(FBUtilities.getBroadcastAddressAndPort());
-        List<InetAddressAndPort> baseEndpoints = new ArrayList<>();
+        EndpointsForToken naturalBaseReplicas = replicationStrategy.getNaturalReplicasForToken(baseToken);
-        List<InetAddressAndPort> viewEndpoints = new ArrayList<>();
+        EndpointsForToken naturalViewReplicas = replicationStrategy.getNaturalReplicasForToken(viewToken);
-        for (InetAddressAndPort baseEndpoint : replicationStrategy.getNaturalEndpoints(baseToken))
-        {
-            // An endpoint is local if we're not using Net
-            if (!(replicationStrategy instanceof NetworkTopologyStrategy) ||
-                DatabaseDescriptor.getEndpointSnitch().getDatacenter(baseEndpoint).equals(localDataCenter))
-                baseEndpoints.add(baseEndpoint);
-        }
 
-        for (InetAddressAndPort viewEndpoint : replicationStrategy.getNaturalEndpoints(viewToken))
+        Optional<Replica> localReplica = Iterables.tryFind(naturalViewReplicas, Replica::isLocal).toJavaUtil();
-        {
+        if (localReplica.isPresent())
-            // If we are a base endpoint which is also a view replica, we use ourselves as our view replica
+            return localReplica;
-            if (viewEndpoint.equals(FBUtilities.getBroadcastAddressAndPort()))
-                return Optional.of(viewEndpoint);
 
-            // We have to remove any endpoint which is shared between the base and the view, as it will select itself
+        // We only select replicas from our own DC
-            // and throw off the counts otherwise.
+        // TODO: this is poor encapsulation, leaking implementation details of replication strategy
-            if (baseEndpoints.contains(viewEndpoint))
+        Predicate<Replica> isLocalDC = r -> !(replicationStrategy instanceof NetworkTopologyStrategy)
-                baseEndpoints.remove(viewEndpoint);
+                || DatabaseDescriptor.getEndpointSnitch().getDatacenter(r).equals(localDataCenter);
-            else if (!(replicationStrategy instanceof NetworkTopologyStrategy) ||
+
-                     DatabaseDescriptor.getEndpointSnitch().getDatacenter(viewEndpoint).equals(localDataCenter))
+        // We have to remove any endpoint which is shared between the base and the view, as it will select itself
-                viewEndpoints.add(viewEndpoint);
+        // and throw off the counts otherwise.
-        }
+        EndpointsForToken baseReplicas = naturalBaseReplicas.filter(
+                r -> !naturalViewReplicas.endpoints().contains(r.endpoint()) && isLocalDC.test(r)
+        );
+        EndpointsForToken viewReplicas = naturalViewReplicas.filter(
+                r -> !naturalBaseReplicas.endpoints().contains(r.endpoint()) && isLocalDC.test(r)
+        );
 
         // The replication strategy will be the same for the base and the view, as they must belong to the same keyspace.
         // Since the same replication strategy is used, the same placement should be used and we should get the same
         // number of replicas for all of the tokens in the ring.
-        assert baseEndpoints.size() == viewEndpoints.size() : "Replication strategy should have the same number of endpoints for the base and the view";
+        assert baseReplicas.size() == viewReplicas.size() : "Replication strategy should have the same number of endpoints for the base and the view";
-        int baseIdx = baseEndpoints.indexOf(FBUtilities.getBroadcastAddressAndPort());
+
+        int baseIdx = -1;
+        for (int i=0; i<baseReplicas.size(); i++)
+        {
+            if (baseReplicas.get(i).isLocal())
+            {
+                baseIdx = i;
+                break;
+            }
+        }
 
         if (baseIdx < 0)
             //This node is not a base replica of this key, so we return empty
             return Optional.empty();
 
-        return Optional.of(viewEndpoints.get(baseIdx));
+        return Optional.of(viewReplicas.get(baseIdx));
     }
 }

src/java/org/apache/cassandra/dht/Range.java

@@ -19,6 +19,7 @@ package org.apache.cassandra.dht;
 
 import java.io.Serializable;
 import java.util.*;
+import java.util.function.Predicate;
 
 import org.apache.commons.lang3.ObjectUtils;
 
@@ -529,7 +530,7 @@ public class Range<T extends RingPosition<T>> extends AbstractBounds<T> implemen
     /**
      * Helper class to check if a token is contained within a given collection of ranges
      */
-    public static class OrderedRangeContainmentChecker
+    public static class OrderedRangeContainmentChecker implements Predicate<Token>
     {
         private final Iterator<Range<Token>> normalizedRangesIterator;
         private Token lastToken = null;
@@ -550,7 +551,8 @@ public class Range<T extends RingPosition<T>> extends AbstractBounds<T> implemen
          * @param t token to check, must be larger than or equal to the last token passed
          * @return true if the token is contained within the ranges given to the constructor.
          */
-        public boolean contains(Token t)
+        @Override
+        public boolean test(Token t)
         {
             assert lastToken == null || lastToken.compareTo(t) <= 0;
             lastToken = t;
@@ -567,4 +569,25 @@ public class Range<T extends RingPosition<T>> extends AbstractBounds<T> implemen
             }
         }
     }
+
+    public static <T extends RingPosition<T>> void assertNormalized(List<Range<T>> ranges)
+    {
+        Range<T> lastRange = null;
+        for (Range<T> range : ranges)
+        {
+            if (lastRange == null)
+            {
+                lastRange = range;
+            }
+            else if (lastRange.left.compareTo(range.left) >= 0 || lastRange.intersects(range))
+            {
+                throw new AssertionError(String.format("Ranges aren't properly normalized. lastRange %s, range %s, compareTo %d, intersects %b, all ranges %s%n",
+                                                       lastRange,
+                                                       range,
+                                                       lastRange.compareTo(range),
+                                                       lastRange.intersects(range),
+                                                       ranges));
+            }
+        }
+    }
 }

src/java/org/apache/cassandra/dht/RangeFetchMapCalculator.java

@@ -19,25 +19,27 @@
 package org.apache.cassandra.dht;
 
 import java.math.BigInteger;
-import java.net.InetAddress;
-import java.util.ArrayList;
 import java.util.Collection;
-import java.util.Collections;
 import java.util.Comparator;
-import java.util.List;
 import java.util.Set;
 import java.util.stream.Collectors;
 
 import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Predicate;
+import com.google.common.base.Predicates;
 import com.google.common.collect.HashMultimap;
 import com.google.common.collect.Multimap;
 
+import org.apache.cassandra.locator.EndpointsByRange;
+import org.apache.cassandra.locator.EndpointsForRange;
 import org.apache.cassandra.locator.InetAddressAndPort;
-import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.locator.Replica;
+
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.locator.Replicas;
 import org.psjava.algo.graph.flownetwork.FordFulkersonAlgorithm;
 import org.psjava.algo.graph.flownetwork.MaximumFlowAlgorithm;
 import org.psjava.algo.graph.flownetwork.MaximumFlowAlgorithmResult;
@@ -73,20 +75,20 @@ public class RangeFetchMapCalculator
 {
     private static final Logger logger = LoggerFactory.getLogger(RangeFetchMapCalculator.class);
     private static final long TRIVIAL_RANGE_LIMIT = 1000;
-    private final Multimap<Range<Token>, InetAddressAndPort> rangesWithSources;
+    private final EndpointsByRange rangesWithSources;
-    private final Collection<RangeStreamer.ISourceFilter> sourceFilters;
+    private final Predicate<Replica> sourceFilters;
     private final String keyspace;
     //We need two Vertices to act as source and destination in the algorithm
     private final Vertex sourceVertex = OuterVertex.getSourceVertex();
     private final Vertex destinationVertex = OuterVertex.getDestinationVertex();
     private final Set<Range<Token>> trivialRanges;
 
-    public RangeFetchMapCalculator(Multimap<Range<Token>, InetAddressAndPort> rangesWithSources,
+    public RangeFetchMapCalculator(EndpointsByRange rangesWithSources,
-                                   Collection<RangeStreamer.ISourceFilter> sourceFilters,
+                                   Collection<Predicate<Replica>> sourceFilters,
                                    String keyspace)
     {
         this.rangesWithSources = rangesWithSources;
-        this.sourceFilters = sourceFilters;
+        this.sourceFilters = Predicates.and(sourceFilters);
         this.keyspace = keyspace;
         this.trivialRanges = rangesWithSources.keySet()
                                               .stream()
@@ -158,14 +160,15 @@ public class RangeFetchMapCalculator
             boolean localDCCheck = true;
             while (!added)
             {
-                List<InetAddressAndPort> srcs = new ArrayList<>(rangesWithSources.get(trivialRange));
                 // sort with the endpoint having the least number of streams first:
-                srcs.sort(Comparator.comparingInt(o -> optimisedMap.get(o).size()));
+                EndpointsForRange replicas = rangesWithSources.get(trivialRange)
-                for (InetAddressAndPort src : srcs)
+                        .sorted(Comparator.comparingInt(o -> optimisedMap.get(o.endpoint()).size()));
+                Replicas.temporaryAssertFull(replicas);
+                for (Replica replica : replicas)
                 {
-                    if (passFilters(src, localDCCheck))
+                    if (passFilters(replica, localDCCheck))
                     {
-                        fetchMap.put(src, trivialRange);
+                        fetchMap.put(replica.endpoint(), trivialRange);
                         added = true;
                         break;
                     }
@@ -347,15 +350,16 @@ public class RangeFetchMapCalculator
     private boolean addEndpoints(MutableCapacityGraph<Vertex, Integer> capacityGraph, RangeVertex rangeVertex, boolean localDCCheck)
     {
         boolean sourceFound = false;
-        for (InetAddressAndPort endpoint : rangesWithSources.get(rangeVertex.getRange()))
+        Replicas.temporaryAssertFull(rangesWithSources.get(rangeVertex.getRange()));
+        for (Replica replica : rangesWithSources.get(rangeVertex.getRange()))
         {
-            if (passFilters(endpoint, localDCCheck))
+            if (passFilters(replica, localDCCheck))
             {
                 sourceFound = true;
                 // if we pass filters, it means that we don't filter away localhost and we can count it as a source:
-                if (endpoint.equals(FBUtilities.getBroadcastAddressAndPort()))
+                if (replica.isLocal())
                     continue; // but don't add localhost to the graph to avoid streaming locally
-                final Vertex endpointVertex = new EndpointVertex(endpoint);
+                final Vertex endpointVertex = new EndpointVertex(replica.endpoint());
                 capacityGraph.insertVertex(rangeVertex);
                 capacityGraph.insertVertex(endpointVertex);
                 capacityGraph.addEdge(rangeVertex, endpointVertex, Integer.MAX_VALUE);
@@ -364,26 +368,20 @@ public class RangeFetchMapCalculator
         return sourceFound;
     }
 
-    private boolean isInLocalDC(InetAddressAndPort endpoint)
+    private boolean isInLocalDC(Replica replica)
     {
-        return DatabaseDescriptor.getLocalDataCenter().equals(DatabaseDescriptor.getEndpointSnitch().getDatacenter(endpoint));
+        return DatabaseDescriptor.getLocalDataCenter().equals(DatabaseDescriptor.getEndpointSnitch().getDatacenter(replica));
     }
 
     /**
      *
-     * @param endpoint   Endpoint to check
+     * @param replica   Replica to check
      * @param localDCCheck Allow endpoints with local DC
      * @return   True if filters pass this endpoint
      */
-    private boolean passFilters(final InetAddressAndPort endpoint, boolean localDCCheck)
+    private boolean passFilters(final Replica replica, boolean localDCCheck)
     {
-        for (RangeStreamer.ISourceFilter filter : sourceFilters)
+        return sourceFilters.apply(replica) && (!localDCCheck || isInLocalDC(replica));
-        {
-            if (!filter.shouldInclude(endpoint))
-                return false;
-        }
-
-        return !localDCCheck || isInLocalDC(endpoint);
     }
 
     private static abstract class Vertex

src/java/org/apache/cassandra/dht/RangeStreamer.java

@@ -18,27 +18,40 @@
 package org.apache.cassandra.dht;
 
 import java.util.*;
+import java.util.function.BiFunction;
+import java.util.function.Function;
+import java.util.stream.Collectors;
 
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Preconditions;
-import com.google.common.collect.ArrayListMultimap;
+import com.google.common.base.Predicate;
 import com.google.common.collect.HashMultimap;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Iterators;
 import com.google.common.collect.Multimap;
-import com.google.common.collect.Sets;
 
+import org.apache.cassandra.gms.FailureDetector;
+import org.apache.cassandra.locator.Endpoints;
+import org.apache.cassandra.locator.EndpointsByReplica;
 import org.apache.cassandra.locator.InetAddressAndPort;
 import org.apache.cassandra.locator.LocalStrategy;
 
+import org.apache.cassandra.locator.EndpointsByRange;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.locator.ReplicaCollection.Mutable.Conflict;
 import org.apache.commons.lang3.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import org.apache.cassandra.db.Keyspace;
-import org.apache.cassandra.gms.EndpointState;
 import org.apache.cassandra.gms.Gossiper;
 import org.apache.cassandra.gms.IFailureDetector;
 import org.apache.cassandra.locator.AbstractReplicationStrategy;
 import org.apache.cassandra.locator.IEndpointSnitch;
+import org.apache.cassandra.locator.Replica;
+import org.apache.cassandra.locator.ReplicaCollection;
+import org.apache.cassandra.locator.Replicas;
 import org.apache.cassandra.locator.TokenMetadata;
 import org.apache.cassandra.service.StorageService;
 import org.apache.cassandra.streaming.PreviewKind;
@@ -47,13 +60,25 @@ import org.apache.cassandra.streaming.StreamResultFuture;
 import org.apache.cassandra.streaming.StreamOperation;
 import org.apache.cassandra.utils.FBUtilities;
 
+import static com.google.common.base.Predicates.and;
+import static com.google.common.base.Predicates.not;
+import static com.google.common.collect.Iterables.all;
+import static com.google.common.collect.Iterables.any;
+import static org.apache.cassandra.locator.Replica.fullReplica;
+
 /**
- * Assists in streaming ranges to a node.
+ * Assists in streaming ranges to this node.
  */
 public class RangeStreamer
 {
     private static final Logger logger = LoggerFactory.getLogger(RangeStreamer.class);
 
+    public static Predicate<Replica> ALIVE_PREDICATE = replica ->
+                                                             (!Gossiper.instance.isEnabled() ||
+                                                              (Gossiper.instance.getEndpointStateForEndpoint(replica.endpoint()) == null ||
+                                                               Gossiper.instance.getEndpointStateForEndpoint(replica.endpoint()).isAlive())) &&
+                                                             FailureDetector.instance.isAlive(replica.endpoint());
+
     /* bootstrap tokens. can be null if replacing the node. */
     private final Collection<Token> tokens;
     /* current token ring */
@@ -62,26 +87,59 @@ public class RangeStreamer
     private final InetAddressAndPort address;
     /* streaming description */
     private final String description;
-    private final Multimap<String, Map.Entry<InetAddressAndPort, Collection<Range<Token>>>> toFetch = HashMultimap.create();
+    private final Multimap<String, Multimap<InetAddressAndPort, FetchReplica>> toFetch = HashMultimap.create();
-    private final Set<ISourceFilter> sourceFilters = new HashSet<>();
+    private final Set<Predicate<Replica>> sourceFilters = new HashSet<>();
     private final StreamPlan streamPlan;
     private final boolean useStrictConsistency;
     private final IEndpointSnitch snitch;
     private final StreamStateStore stateStore;
 
-    /**
+    public static class FetchReplica
-     * A filter applied to sources to stream from when constructing a fetch map.
-     */
-    public static interface ISourceFilter
     {
-        public boolean shouldInclude(InetAddressAndPort endpoint);
+        public final Replica local;
+        public final Replica remote;
+
+        public FetchReplica(Replica local, Replica remote)
+        {
+            Preconditions.checkNotNull(local);
+            Preconditions.checkNotNull(remote);
+            assert local.isLocal() && !remote.isLocal();
+            this.local = local;
+            this.remote = remote;
+        }
+
+        public String toString()
+        {
+            return "FetchReplica{" +
+                   "local=" + local +
+                   ", remote=" + remote +
+                   '}';
+        }
+
+        public boolean equals(Object o)
+        {
+            if (this == o) return true;
+            if (o == null || getClass() != o.getClass()) return false;
+
+            FetchReplica that = (FetchReplica) o;
+
+            if (!local.equals(that.local)) return false;
+            return remote.equals(that.remote);
+        }
+
+        public int hashCode()
+        {
+            int result = local.hashCode();
+            result = 31 * result + remote.hashCode();
+            return result;
+        }
     }
 
     /**
      * Source filter which excludes any endpoints that are not alive according to a
      * failure detector.
      */
-    public static class FailureDetectorSourceFilter implements ISourceFilter
+    public static class FailureDetectorSourceFilter implements Predicate<Replica>
     {
         private final IFailureDetector fd;
 
@@ -90,16 +148,16 @@ public class RangeStreamer
             this.fd = fd;
         }
 
-        public boolean shouldInclude(InetAddressAndPort endpoint)
+        public boolean apply(Replica replica)
         {
-            return fd.isAlive(endpoint);
+            return fd.isAlive(replica.endpoint());
         }
     }
 
     /**
      * Source filter which excludes any endpoints that are not in a specific data center.
      */
-    public static class SingleDatacenterFilter implements ISourceFilter
+    public static class SingleDatacenterFilter implements Predicate<Replica>
     {
         private final String sourceDc;
         private final IEndpointSnitch snitch;
@@ -110,27 +168,27 @@ public class RangeStreamer
             this.snitch = snitch;
         }
 
-        public boolean shouldInclude(InetAddressAndPort endpoint)
+        public boolean apply(Replica replica)
         {
-            return snitch.getDatacenter(endpoint).equals(sourceDc);
+            return snitch.getDatacenter(replica).equals(sourceDc);
         }
     }
 
     /**
      * Source filter which excludes the current node from source calculations
      */
-    public static class ExcludeLocalNodeFilter implements ISourceFilter
+    public static class ExcludeLocalNodeFilter implements Predicate<Replica>
     {
-        public boolean shouldInclude(InetAddressAndPort endpoint)
+        public boolean apply(Replica replica)
         {
-            return !FBUtilities.getBroadcastAddressAndPort().equals(endpoint);
+            return !replica.isLocal();
         }
     }
 
     /**
      * Source filter which only includes endpoints contained within a provided set.
      */
-    public static class WhitelistedSourcesFilter implements ISourceFilter
+    public static class WhitelistedSourcesFilter implements Predicate<Replica>
     {
         private final Set<InetAddressAndPort> whitelistedSources;
 
@@ -139,9 +197,9 @@ public class RangeStreamer
             this.whitelistedSources = whitelistedSources;
         }
 
-        public boolean shouldInclude(InetAddressAndPort endpoint)
+        public boolean apply(Replica replica)
         {
-            return whitelistedSources.contains(endpoint);
+            return whitelistedSources.contains(replica.endpoint());
         }
     }
 
@@ -167,7 +225,7 @@ public class RangeStreamer
         streamPlan.listeners(this.stateStore);
     }
 
-    public void addSourceFilter(ISourceFilter filter)
+    public void addSourceFilter(Predicate<Replica> filter)
     {
         sourceFilters.add(filter);
     }
@@ -176,80 +234,95 @@ public class RangeStreamer
      * Add ranges to be streamed for given keyspace.
      *
      * @param keyspaceName keyspace name
-     * @param ranges ranges to be streamed
+     * @param replicas ranges to be streamed
      */
-    public void addRanges(String keyspaceName, Collection<Range<Token>> ranges)
+    public void addRanges(String keyspaceName, ReplicaCollection<?> replicas)
     {
-        if(Keyspace.open(keyspaceName).getReplicationStrategy() instanceof LocalStrategy)
+        Keyspace keyspace = Keyspace.open(keyspaceName);
+        AbstractReplicationStrategy strat = keyspace.getReplicationStrategy();
+        if(strat instanceof LocalStrategy)
         {
             logger.info("Not adding ranges for Local Strategy keyspace={}", keyspaceName);
             return;
         }
 
-        boolean useStrictSource = useStrictSourcesForRanges(keyspaceName);
+        boolean useStrictSource = useStrictSourcesForRanges(strat);
-        Multimap<Range<Token>, InetAddressAndPort> rangesForKeyspace = useStrictSource
+        EndpointsByReplica fetchMap = calculateRangesToFetchWithPreferredEndpoints(replicas, keyspace, useStrictSource);
-                ? getAllRangesWithStrictSourcesFor(keyspaceName, ranges) : getAllRangesWithSourcesFor(keyspaceName, ranges);
 
-        for (Map.Entry<Range<Token>, InetAddressAndPort> entry : rangesForKeyspace.entries())
+        for (Map.Entry<Replica, Replica> entry : fetchMap.flattenEntries())
             logger.info("{}: range {} exists on {} for keyspace {}", description, entry.getKey(), entry.getValue(), keyspaceName);
 
-        AbstractReplicationStrategy strat = Keyspace.open(keyspaceName).getReplicationStrategy();
-        Multimap<InetAddressAndPort, Range<Token>> rangeFetchMap = useStrictSource || strat == null || strat.getReplicationFactor() == 1
-                                                            ? getRangeFetchMap(rangesForKeyspace, sourceFilters, keyspaceName, useStrictConsistency)
-                                                            : getOptimizedRangeFetchMap(rangesForKeyspace, sourceFilters, keyspaceName);
 
-        for (Map.Entry<InetAddressAndPort, Collection<Range<Token>>> entry : rangeFetchMap.asMap().entrySet())
+        Multimap<InetAddressAndPort, FetchReplica> workMap;
+        //Only use the optimized strategy if we don't care about strict sources, have a replication factor > 1, and no
+        //transient replicas.
+        if (useStrictSource || strat == null || strat.getReplicationFactor().allReplicas == 1 || strat.getReplicationFactor().hasTransientReplicas())
+        {
+            workMap = convertPreferredEndpointsToWorkMap(fetchMap);
+        }
+        else
+        {
+            workMap = getOptimizedWorkMap(fetchMap, sourceFilters, keyspaceName);
+        }
+
+        toFetch.put(keyspaceName, workMap);
+        for (Map.Entry<InetAddressAndPort, Collection<FetchReplica>> entry : workMap.asMap().entrySet())
         {
             if (logger.isTraceEnabled())
             {
-                for (Range<Token> r : entry.getValue())
+                for (FetchReplica r : entry.getValue())
-                    logger.trace("{}: range {} from source {} for keyspace {}", description, r, entry.getKey(), keyspaceName);
+                    logger.trace("{}: range source {} local range {} for keyspace {}", description, r.remote, r.local, keyspaceName);
             }
-            toFetch.put(keyspaceName, entry);
         }
     }
 
     /**
-     * @param keyspaceName keyspace name to check
+     * @param strat AbstractReplicationStrategy of keyspace to check
      * @return true when the node is bootstrapping, useStrictConsistency is true and # of nodes in the cluster is more than # of replica
      */
-    private boolean useStrictSourcesForRanges(String keyspaceName)
+    private boolean useStrictSourcesForRanges(AbstractReplicationStrategy strat)
     {
-        AbstractReplicationStrategy strat = Keyspace.open(keyspaceName).getReplicationStrategy();
         return useStrictConsistency
                 && tokens != null
-                && metadata.getSizeOfAllEndpoints() != strat.getReplicationFactor();
+                && metadata.getSizeOfAllEndpoints() != strat.getReplicationFactor().allReplicas;
     }
 
     /**
-     * Get a map of all ranges and their respective sources that are candidates for streaming the given ranges
+     * Wrapper method to assemble the arguments for invoking the implementation with RangeStreamer's parameters
-     * to us. For each range, the list of sources is sorted by proximity relative to the given destAddress.
+     * @param fetchRanges
-     *
+     * @param keyspace
-     * @throws java.lang.IllegalStateException when there is no source to get data streamed
+     * @param useStrictConsistency
+     * @return
      */
-    private Multimap<Range<Token>, InetAddressAndPort> getAllRangesWithSourcesFor(String keyspaceName, Collection<Range<Token>> desiredRanges)
+    private EndpointsByReplica calculateRangesToFetchWithPreferredEndpoints(ReplicaCollection<?> fetchRanges, Keyspace keyspace, boolean useStrictConsistency)
     {
-        AbstractReplicationStrategy strat = Keyspace.open(keyspaceName).getReplicationStrategy();
+        AbstractReplicationStrategy strat = keyspace.getReplicationStrategy();
-        Multimap<Range<Token>, InetAddressAndPort> rangeAddresses = strat.getRangeAddresses(metadata.cloneOnlyTokenMap());
 
-        Multimap<Range<Token>, InetAddressAndPort> rangeSources = ArrayListMultimap.create();
+        TokenMetadata tmd = metadata.cloneOnlyTokenMap();
-        for (Range<Token> desiredRange : desiredRanges)
+
+        TokenMetadata tmdAfter = null;
+
+        if (tokens != null)
         {
-            for (Range<Token> range : rangeAddresses.keySet())
+            // Pending ranges
-            {
+            tmdAfter =  tmd.cloneOnlyTokenMap();
-                if (range.contains(desiredRange))
+            tmdAfter.updateNormalTokens(tokens, address);
-                {
+        }
-                    List<InetAddressAndPort> preferred = snitch.getSortedListByProximity(address, rangeAddresses.get(range));
+        else if (useStrictConsistency)
-                    rangeSources.putAll(desiredRange, preferred);
+        {
-                    break;
+            throw new IllegalArgumentException("Can't ask for strict consistency and not supply tokens");
-                }
-            }
-
-            if (!rangeSources.keySet().contains(desiredRange))
-                throw new IllegalStateException("No sources found for " + desiredRange);
         }
 
-        return rangeSources;
+        return RangeStreamer.calculateRangesToFetchWithPreferredEndpoints(snitch::sortedByProximity,
+                                                                           strat,
+                                                                           fetchRanges,
+                                                                           useStrictConsistency,
+                                                                           tmd,
+                                                                           tmdAfter,
+                                                                           ALIVE_PREDICATE,
+                                                                           keyspace.getName(),
+                                                                           sourceFilters);
+
     }
 
     /**
@@ -257,129 +330,234 @@ public class RangeStreamer
      * For each range, the list should only contain a single source. This allows us to consistently migrate data without violating
      * consistency.
      *
-     * @throws java.lang.IllegalStateException when there is no source to get data streamed, or more than 1 source found.
+     **/
-     */
+     public static EndpointsByReplica
-    private Multimap<Range<Token>, InetAddressAndPort> getAllRangesWithStrictSourcesFor(String keyspace, Collection<Range<Token>> desiredRanges)
+     calculateRangesToFetchWithPreferredEndpoints(BiFunction<InetAddressAndPort, EndpointsForRange, EndpointsForRange> snitchGetSortedListByProximity,
+                                                  AbstractReplicationStrategy strat,
+                                                  ReplicaCollection<?> fetchRanges,
+                                                  boolean useStrictConsistency,
+                                                  TokenMetadata tmdBefore,
+                                                  TokenMetadata tmdAfter,
+                                                  Predicate<Replica> isAlive,
+                                                  String keyspace,
+                                                  Collection<Predicate<Replica>> sourceFilters)
     {
-        assert tokens != null;
+        EndpointsByRange rangeAddresses = strat.getRangeAddresses(tmdBefore);
-        AbstractReplicationStrategy strat = Keyspace.open(keyspace).getReplicationStrategy();
 
-        // Active ranges
+        InetAddressAndPort localAddress = FBUtilities.getBroadcastAddressAndPort();
-        TokenMetadata metadataClone = metadata.cloneOnlyTokenMap();
+        logger.debug ("Keyspace: {}", keyspace);
-        Multimap<Range<Token>, InetAddressAndPort> addressRanges = strat.getRangeAddresses(metadataClone);
+        logger.debug("To fetch RN: {}", fetchRanges);
+        logger.debug("Fetch ranges: {}", rangeAddresses);
 
-        // Pending ranges
+        Predicate<Replica> testSourceFilters = and(sourceFilters);
-        metadataClone.updateNormalTokens(tokens, address);
+        Function<EndpointsForRange, EndpointsForRange> sorted =
-        Multimap<Range<Token>, InetAddressAndPort> pendingRangeAddresses = strat.getRangeAddresses(metadataClone);
+                endpoints -> snitchGetSortedListByProximity.apply(localAddress, endpoints);
 
-        // Collects the source that will have its range moved to the new node
+        //This list of replicas is just candidates. With strict consistency it's going to be a narrow list.
-        Multimap<Range<Token>, InetAddressAndPort> rangeSources = ArrayListMultimap.create();
+        EndpointsByReplica.Mutable rangesToFetchWithPreferredEndpoints = new EndpointsByReplica.Mutable();
-
+        for (Replica toFetch : fetchRanges)
-        for (Range<Token> desiredRange : desiredRanges)
         {
-            for (Map.Entry<Range<Token>, Collection<InetAddressAndPort>> preEntry : addressRanges.asMap().entrySet())
+            //Replica that is sufficient to provide the data we need
-            {
+            //With strict consistency and transient replication we may end up with multiple types
-                if (preEntry.getKey().contains(desiredRange))
+            //so this isn't used with strict consistency
-                {
+            Predicate<Replica> isSufficient = r -> (toFetch.isTransient() || r.isFull());
-                    Set<InetAddressAndPort> oldEndpoints = Sets.newHashSet(preEntry.getValue());
+            Predicate<Replica> accept = r ->
-                    Set<InetAddressAndPort> newEndpoints = Sets.newHashSet(pendingRangeAddresses.get(desiredRange));
+                       isSufficient.test(r)                 // is sufficient
+                    && !r.endpoint().equals(localAddress)   // is not self
+                    && isAlive.test(r);                     // is alive
 
-                    // Due to CASSANDRA-5953 we can have a higher RF then we have endpoints.
+            logger.debug("To fetch {}", toFetch);
-                    // So we need to be careful to only be strict when endpoints == RF
+            for (Range<Token> range : rangeAddresses.keySet())
-                    if (oldEndpoints.size() == strat.getReplicationFactor())
+            {
+                if (range.contains(toFetch.range()))
+                {
+                    EndpointsForRange oldEndpoints = rangeAddresses.get(range);
+
+                    //Ultimately we populate this with whatever is going to be fetched from to satisfy toFetch
+                    //It could be multiple endpoints and we must fetch from all of them if they are there
+                    //With transient replication and strict consistency this is to get the full data from a full replica and
+                    //transient data from the transient replica losing data
+                    EndpointsForRange sources;
+                    if (useStrictConsistency)
                     {
-                        oldEndpoints.removeAll(newEndpoints);
+                        //Start with two sets of who replicates the range before and who replicates it after
-                        assert oldEndpoints.size() == 1 : "Expected 1 endpoint but found " + oldEndpoints.size();
+                        EndpointsForRange newEndpoints = strat.calculateNaturalReplicas(toFetch.range().right, tmdAfter);
+                        logger.debug("Old endpoints {}", oldEndpoints);
+                        logger.debug("New endpoints {}", newEndpoints);
+
+                        //Due to CASSANDRA-5953 we can have a higher RF then we have endpoints.
+                        //So we need to be careful to only be strict when endpoints == RF
+                        if (oldEndpoints.size() == strat.getReplicationFactor().allReplicas)
+                        {
+                            Set<InetAddressAndPort> endpointsStillReplicated = newEndpoints.endpoints();
+                            // Remove new endpoints from old endpoints based on address
+                            oldEndpoints = oldEndpoints.filter(r -> !endpointsStillReplicated.contains(r.endpoint()));
+
+                            if (!all(oldEndpoints, isAlive))
+                                throw new IllegalStateException("A node required to move the data consistently is down: "
+                                                                + oldEndpoints.filter(not(isAlive)));
+
+                            if (oldEndpoints.size() > 1)
+                                throw new AssertionError("Expected <= 1 endpoint but found " + oldEndpoints);
+
+                            //If we are transitioning from transient to full and and the set of replicas for the range is not changing
+                            //we might end up with no endpoints to fetch from by address. In that case we can pick any full replica safely
+                            //since we are already a transient replica and the existing replica remains.
+                            //The old behavior where we might be asked to fetch ranges we don't need shouldn't occur anymore.
+                            //So it's an error if we don't find what we need.
+                            if (oldEndpoints.isEmpty() && toFetch.isTransient())
+                            {
+                                throw new AssertionError("If there are no endpoints to fetch from then we must be transitioning from transient to full for range " + toFetch);
+                            }
+
+                            if (!any(oldEndpoints, isSufficient))
+                            {
+                                // need an additional replica
+                                EndpointsForRange endpointsForRange = sorted.apply(rangeAddresses.get(range));
+                                // include all our filters, to ensure we include a matching node
+                                Optional<Replica> fullReplica = Iterables.<Replica>tryFind(endpointsForRange, and(accept, testSourceFilters)).toJavaUtil();
+                                if (fullReplica.isPresent())
+                                    oldEndpoints = Endpoints.concat(oldEndpoints, EndpointsForRange.of(fullReplica.get()));
+                                else
+                                    throw new IllegalStateException("Couldn't find any matching sufficient replica out of " + endpointsForRange);
+                            }
+
+                            //We have to check the source filters here to see if they will remove any replicas
+                            //required for strict consistency
+                            if (!all(oldEndpoints, testSourceFilters))
+                                throw new IllegalStateException("Necessary replicas for strict consistency were removed by source filters: " + oldEndpoints.filter(not(testSourceFilters)));
+                        }
+                        else
+                        {
+                            oldEndpoints = sorted.apply(oldEndpoints.filter(accept));
+                        }
+
+                        //Apply testSourceFilters that were given to us, and establish everything remaining is alive for the strict case
+                        sources = oldEndpoints.filter(testSourceFilters);
+                    }
+                    else
+                    {
+                        //Without strict consistency we have given up on correctness so no point in fetching from
+                        //a random full + transient replica since it's also likely to lose data
+                        //Also apply testSourceFilters that were given to us so we can safely select a single source
+                        sources = sorted.apply(rangeAddresses.get(range).filter(and(accept, testSourceFilters)));
+                        //Limit it to just the first possible source, we don't need more than one and downstream
+                        //will fetch from every source we supply
+                        sources = sources.size() > 0 ? sources.subList(0, 1) : sources;
                     }
 
-                    rangeSources.put(desiredRange, oldEndpoints.iterator().next());
+                    // storing range and preferred endpoint set
+                    rangesToFetchWithPreferredEndpoints.putAll(toFetch, sources, Conflict.NONE);
+                    logger.debug("Endpoints to fetch for {} are {}", toFetch, sources);
                 }
             }
 
-            // Validate
+            EndpointsForRange addressList = rangesToFetchWithPreferredEndpoints.getIfPresent(toFetch);
-            Collection<InetAddressAndPort> addressList = rangeSources.get(desiredRange);
+            if (addressList == null)
-            if (addressList == null || addressList.isEmpty())
+                throw new IllegalStateException("Failed to find endpoints to fetch " + toFetch);
-                throw new IllegalStateException("No sources found for " + desiredRange);
 
-            if (addressList.size() > 1)
+            /*
-                throw new IllegalStateException("Multiple endpoints found for " + desiredRange);
+             * When we move forwards (shrink our bucket) we are the one losing a range and no one else loses
+             * from that action (we also don't gain). When we move backwards there are two people losing a range. One is a full replica
+             * and the other is a transient replica. So we must need fetch from two places in that case for the full range we gain.
+             * For a transient range we only need to fetch from one.
+             */
+            if (useStrictConsistency && addressList.size() > 1 && (addressList.filter(Replica::isFull).size() > 1 || addressList.filter(Replica::isTransient).size() > 1))
+                throw new IllegalStateException(String.format("Multiple strict sources found for %s, sources: %s", toFetch, addressList));
 
-            InetAddressAndPort sourceIp = addressList.iterator().next();
+            //We must have enough stuff to fetch from
-            EndpointState sourceState = Gossiper.instance.getEndpointStateForEndpoint(sourceIp);
+            if ((toFetch.isFull() && !any(addressList, Replica::isFull)) || addressList.isEmpty())
-            if (Gossiper.instance.isEnabled() && (sourceState == null || !sourceState.isAlive()))
+            {
-                throw new RuntimeException("A node required to move the data consistently is down (" + sourceIp + "). " +
+                if (strat.getReplicationFactor().allReplicas == 1)
-                                           "If you wish to move the data from a potentially inconsistent replica, restart the node with -Dcassandra.consistent.rangemovement=false");
+                {
+                    if (useStrictConsistency)
+                    {
+                        logger.warn("A node required to move the data consistently is down");
+                        throw new IllegalStateException("Unable to find sufficient sources for streaming range " + toFetch + " in keyspace " + keyspace + " with RF=1. " +
+                                                        "Ensure this keyspace contains replicas in the source datacenter.");
+                    }
+                    else
+                        logger.warn("Unable to find sufficient sources for streaming range {} in keyspace {} with RF=1. " +
+                                    "Keyspace might be missing data.", toFetch, keyspace);
+
+                }
+                else
+                {
+                    if (useStrictConsistency)
+                        logger.warn("A node required to move the data consistently is down");
+                    throw new IllegalStateException("Unable to find sufficient sources for streaming range " + toFetch + " in keyspace " + keyspace);
+                }
+            }
         }
-
+        return rangesToFetchWithPreferredEndpoints.asImmutableView();
-        return rangeSources;
     }
 
     /**
-     * @param rangesWithSources The ranges we want to fetch (key) and their potential sources (value)
+     * The preferred endpoint list is the wrong format because it is keyed by Replica (this node) rather than the source
-     * @param sourceFilters A (possibly empty) collection of source filters to apply. In addition to any filters given
+     * endpoint we will fetch from which streaming wants.
-     *                      here, we always exclude ourselves.
+     * @param preferredEndpoints
-     * @param keyspace keyspace name
+     * @return
-     * @return Map of source endpoint to collection of ranges
      */
-    private static Multimap<InetAddressAndPort, Range<Token>> getRangeFetchMap(Multimap<Range<Token>, InetAddressAndPort> rangesWithSources,
+    public static Multimap<InetAddressAndPort, FetchReplica> convertPreferredEndpointsToWorkMap(EndpointsByReplica preferredEndpoints)
-                                                                               Collection<ISourceFilter> sourceFilters, String keyspace,
-                                                                               boolean useStrictConsistency)
     {
-        Multimap<InetAddressAndPort, Range<Token>> rangeFetchMapMap = HashMultimap.create();
+        Multimap<InetAddressAndPort, FetchReplica> workMap = HashMultimap.create();
-        for (Range<Token> range : rangesWithSources.keySet())
+        for (Map.Entry<Replica, EndpointsForRange> e : preferredEndpoints.entrySet())
         {
-            boolean foundSource = false;
+            for (Replica source : e.getValue())
-
-            outer:
-            for (InetAddressAndPort address : rangesWithSources.get(range))
             {
-                for (ISourceFilter filter : sourceFilters)
+                assert (e.getKey()).isLocal();
-                {
+                assert !source.isLocal();
-                    if (!filter.shouldInclude(address))
+                workMap.put(source.endpoint(), new FetchReplica(e.getKey(), source));
-                        continue outer;
-                }
-
-                if (address.equals(FBUtilities.getBroadcastAddressAndPort()))
-                {
-                    // If localhost is a source, we have found one, but we don't add it to the map to avoid streaming locally
-                    foundSource = true;
-                    continue;
-                }
-
-                rangeFetchMapMap.put(address, range);
-                foundSource = true;
-                break; // ensure we only stream from one other node for each range
-            }
-
-            if (!foundSource)
-            {
-                AbstractReplicationStrategy strat = Keyspace.open(keyspace).getReplicationStrategy();
-                if (strat != null && strat.getReplicationFactor() == 1)
-                {
-                    if (useStrictConsistency)
-                        throw new IllegalStateException("Unable to find sufficient sources for streaming range " + range + " in keyspace " + keyspace + " with RF=1. " +
-                                                        "Ensure this keyspace contains replicas in the source datacenter.");
-                    else
-                        logger.warn("Unable to find sufficient sources for streaming range {} in keyspace {} with RF=1. " +
-                                    "Keyspace might be missing data.", range, keyspace);
-                }
-                else
-                    throw new IllegalStateException("Unable to find sufficient sources for streaming range " + range + " in keyspace " + keyspace);
             }
         }
-
+        logger.debug("Work map {}", workMap);
-        return rangeFetchMapMap;
+        return workMap;
     }
 
-
+    /**
-    private static Multimap<InetAddressAndPort, Range<Token>> getOptimizedRangeFetchMap(Multimap<Range<Token>, InetAddressAndPort> rangesWithSources,
+     * Optimized version that also outputs the final work map
-                                                                                        Collection<ISourceFilter> sourceFilters, String keyspace)
+     */
+    private static Multimap<InetAddressAndPort, FetchReplica> getOptimizedWorkMap(EndpointsByReplica rangesWithSources,
+                                                                                  Collection<Predicate<Replica>> sourceFilters, String keyspace)
     {
-        RangeFetchMapCalculator calculator = new RangeFetchMapCalculator(rangesWithSources, sourceFilters, keyspace);
+        //For now we just aren't going to use the optimized range fetch map with transient replication to shrink
+        //the surface area to test and introduce bugs.
+        //In the future it's possible we could run it twice once for full ranges with only full replicas
+        //and once with transient ranges and all replicas. Then merge the result.
+        EndpointsByRange.Mutable unwrapped = new EndpointsByRange.Mutable();
+        for (Map.Entry<Replica, Replica> entry : rangesWithSources.flattenEntries())
+        {
+            Replicas.temporaryAssertFull(entry.getValue());
+            unwrapped.put(entry.getKey().range(), entry.getValue());
+        }
+
+        RangeFetchMapCalculator calculator = new RangeFetchMapCalculator(unwrapped.asImmutableView(), sourceFilters, keyspace);
         Multimap<InetAddressAndPort, Range<Token>> rangeFetchMapMap = calculator.getRangeFetchMap();
         logger.info("Output from RangeFetchMapCalculator for keyspace {}", keyspace);
-        validateRangeFetchMap(rangesWithSources, rangeFetchMapMap, keyspace);
+        validateRangeFetchMap(unwrapped.asImmutableView(), rangeFetchMapMap, keyspace);
-        return rangeFetchMapMap;
+
+        //Need to rewrap as Replicas
+        Multimap<InetAddressAndPort, FetchReplica> wrapped = HashMultimap.create();
+        for (Map.Entry<InetAddressAndPort, Range<Token>> entry : rangeFetchMapMap.entries())
+        {
+            Replica toFetch = null;
+            for (Replica r : rangesWithSources.keySet())
+            {
+                if (r.range().equals(entry.getValue()))
+                {
+                    if (toFetch != null)
+                        throw new AssertionError(String.format("There shouldn't be multiple replicas for range %s, replica %s and %s here", r.range(), r, toFetch));
+                    toFetch = r;
+                }
+            }
+            if (toFetch == null)
+                throw new AssertionError("Shouldn't be possible for the Replica we fetch to be null here");
+            //Committing the cardinal sin of synthesizing a Replica, but it's ok because we assert earlier all of them
+            //are full and optimized range fetch map doesn't support transient replication yet.
+            wrapped.put(entry.getKey(), new FetchReplica(toFetch, fullReplica(entry.getKey(), entry.getValue())));
+        }
+
+        return wrapped;
     }
 
     /**
@@ -388,7 +566,7 @@ public class RangeStreamer
      * @param rangeFetchMapMap
      * @param keyspace
      */
-    private static void validateRangeFetchMap(Multimap<Range<Token>, InetAddressAndPort> rangesWithSources, Multimap<InetAddressAndPort, Range<Token>> rangeFetchMapMap, String keyspace)
+    private static void validateRangeFetchMap(EndpointsByRange rangesWithSources, Multimap<InetAddressAndPort, Range<Token>> rangeFetchMapMap, String keyspace)
     {
         for (Map.Entry<InetAddressAndPort, Range<Token>> entry : rangeFetchMapMap.entries())
         {
@@ -398,7 +576,7 @@ public class RangeStreamer
                                         + " in keyspace " + keyspace);
             }
 
-            if (!rangesWithSources.get(entry.getValue()).contains(entry.getKey()))
+            if (!rangesWithSources.get(entry.getValue()).endpoints().contains(entry.getKey()))
             {
                 throw new IllegalStateException("Trying to stream from wrong endpoint. Range: " + entry.getValue()
                                                 + " in keyspace " + keyspace + " from endpoint: " + entry.getKey());
@@ -408,39 +586,70 @@ public class RangeStreamer
         }
     }
 
-    public static Multimap<InetAddressAndPort, Range<Token>> getWorkMap(Multimap<Range<Token>, InetAddressAndPort> rangesWithSourceTarget, String keyspace,
-                                                                        IFailureDetector fd, boolean useStrictConsistency)
-    {
-        return getRangeFetchMap(rangesWithSourceTarget, Collections.<ISourceFilter>singleton(new FailureDetectorSourceFilter(fd)), keyspace, useStrictConsistency);
-    }
-
     // For testing purposes
     @VisibleForTesting
-    Multimap<String, Map.Entry<InetAddressAndPort, Collection<Range<Token>>>> toFetch()
+    Multimap<String, Multimap<InetAddressAndPort, FetchReplica>> toFetch()
     {
         return toFetch;
     }
 
     public StreamResultFuture fetchAsync()
     {
-        for (Map.Entry<String, Map.Entry<InetAddressAndPort, Collection<Range<Token>>>> entry : toFetch.entries())
+        toFetch.forEach((keyspace, sources) -> {
-        {
+            logger.debug("Keyspace {} Sources {}", keyspace, sources);
-            String keyspace = entry.getKey();
+            sources.asMap().forEach((source, fetchReplicas) -> {
-            InetAddressAndPort source = entry.getValue().getKey();
-            Collection<Range<Token>> ranges = entry.getValue().getValue();
 
-            // filter out already streamed ranges
+                // filter out already streamed ranges
-            Set<Range<Token>> availableRanges = stateStore.getAvailableRanges(keyspace, StorageService.instance.getTokenMetadata().partitioner);
+                RangesAtEndpoint available = stateStore.getAvailableRanges(keyspace, StorageService.instance.getTokenMetadata().partitioner);
-            if (ranges.removeAll(availableRanges))
-            {
-                logger.info("Some ranges of {} are already available. Skipping streaming those ranges.", availableRanges);
-            }
 
-            if (logger.isTraceEnabled())
+                Predicate<FetchReplica> isAvailable = fetch -> {
-                logger.trace("{}ing from {} ranges {}", description, source, StringUtils.join(ranges, ", "));
+                    Replica availableRange =  available.byRange().get(fetch.local.range());
-            /* Send messages to respective folks to stream data over to me */
+                    if (availableRange == null)
-            streamPlan.requestRanges(source, keyspace, ranges);
+                        //Range is unavailable
-        }
+                        return false;
+                    if (fetch.local.isFull())
+                        //For full, pick only replicas with matching transientness
+                        return availableRange.isFull() == fetch.remote.isFull();
+
+                    // Any transient or full will do
+                    return true;
+                };
+
+                List<FetchReplica> remaining = fetchReplicas.stream().filter(not(isAvailable)).collect(Collectors.toList());
+
+                if (remaining.size() < available.size())
+                {
+                    List<FetchReplica> skipped = fetchReplicas.stream().filter(isAvailable).collect(Collectors.toList());
+                    logger.info("Some ranges of {} are already available. Skipping streaming those ranges. Skipping {}. Fully available {} Transiently available {}",
+                                fetchReplicas, skipped, available.filter(Replica::isFull).ranges(), available.filter(Replica::isTransient).ranges());
+                }
+
+                if (logger.isTraceEnabled())
+                    logger.trace("{}ing from {} ranges {}", description, source, StringUtils.join(remaining, ", "));
+
+                //At the other end the distinction between full and transient is ignored it just used the transient status
+                //of the Replica objects we send to determine what to send. The real reason we have this split down to
+                //StreamRequest is that on completion StreamRequest is used to write to the system table tracking
+                //what has already been streamed. At that point since we only have the local Replica instances so we don't
+                //know what we got from the remote. We preserve that here by splitting based on the remotes transient
+                //status.
+                InetAddressAndPort self = FBUtilities.getBroadcastAddressAndPort();
+                RangesAtEndpoint full = remaining.stream()
+                        .filter(pair -> pair.remote.isFull())
+                        .map(pair -> pair.local)
+                        .collect(RangesAtEndpoint.collector(self));
+                RangesAtEndpoint transientReplicas = remaining.stream()
+                        .filter(pair -> pair.remote.isTransient())
+                        .map(pair -> pair.local)
+                        .collect(RangesAtEndpoint.collector(self));
+
+                logger.debug("Source and our replicas {}", fetchReplicas);
+                logger.debug("Source {} Keyspace {}  streaming full {} transient {}", source, keyspace, full, transientReplicas);
+
+                /* Send messages to respective folks to stream data over to me */
+                streamPlan.requestRanges(source, keyspace, full, transientReplicas);
+            });
+        });
 
         return streamPlan.execute();
     }

src/java/org/apache/cassandra/dht/Splitter.java

@@ -25,6 +25,7 @@ import java.util.Collection;
 import java.util.Collections;
 import java.util.HashSet;
 import java.util.List;
+import java.util.Objects;
 import java.util.Set;
 
 import com.google.common.annotations.VisibleForTesting;
@@ -117,32 +118,31 @@ public abstract class Splitter
         return new BigDecimal(elapsedTokens(token, range)).divide(new BigDecimal(tokensInRange(range)), 3, BigDecimal.ROUND_HALF_EVEN).doubleValue();
     }
 
-    public List<Token> splitOwnedRanges(int parts, List<Range<Token>> localRanges, boolean dontSplitRanges)
+    public List<Token> splitOwnedRanges(int parts, List<WeightedRange> weightedRanges, boolean dontSplitRanges)
     {
-        if (localRanges.isEmpty() || parts == 1)
+        if (weightedRanges.isEmpty() || parts == 1)
             return Collections.singletonList(partitioner.getMaximumToken());
 
         BigInteger totalTokens = BigInteger.ZERO;
-        for (Range<Token> r : localRanges)
+        for (WeightedRange weightedRange : weightedRanges)
         {
-            BigInteger right = valueForToken(token(r.right));
+            totalTokens = totalTokens.add(weightedRange.totalTokens(this));
-            totalTokens = totalTokens.add(right.subtract(valueForToken(r.left)));
         }
+
         BigInteger perPart = totalTokens.divide(BigInteger.valueOf(parts));
         // the range owned is so tiny we can't split it:
         if (perPart.equals(BigInteger.ZERO))
             return Collections.singletonList(partitioner.getMaximumToken());
 
         if (dontSplitRanges)
-            return splitOwnedRangesNoPartialRanges(localRanges, perPart, parts);
+            return splitOwnedRangesNoPartialRanges(weightedRanges, perPart, parts);
 
         List<Token> boundaries = new ArrayList<>();
         BigInteger sum = BigInteger.ZERO;
-        for (Range<Token> r : localRanges)
+        for (WeightedRange weightedRange : weightedRanges)
         {
-            Token right = token(r.right);
+            BigInteger currentRangeWidth = weightedRange.totalTokens(this);
-            BigInteger currentRangeWidth = valueForToken(right).subtract(valueForToken(r.left)).abs();
+            BigInteger left = valueForToken(weightedRange.left());
-            BigInteger left = valueForToken(r.left);
             while (sum.add(currentRangeWidth).compareTo(perPart) >= 0)
             {
                 BigInteger withinRangeBoundary = perPart.subtract(sum);
@@ -155,26 +155,24 @@ public abstract class Splitter
         }
         boundaries.set(boundaries.size() - 1, partitioner.getMaximumToken());
 
-        assert boundaries.size() == parts : boundaries.size() + "!=" + parts + " " + boundaries + ":" + localRanges;
+        assert boundaries.size() == parts : boundaries.size() + "!=" + parts + " " + boundaries + ":" + weightedRanges;
         return boundaries;
     }
 
-    private List<Token> splitOwnedRangesNoPartialRanges(List<Range<Token>> localRanges, BigInteger perPart, int parts)
+    private List<Token> splitOwnedRangesNoPartialRanges(List<WeightedRange> weightedRanges, BigInteger perPart, int parts)
     {
         List<Token> boundaries = new ArrayList<>(parts);
         BigInteger sum = BigInteger.ZERO;
 
         int i = 0;
-        final int rangesCount = localRanges.size();
+        final int rangesCount = weightedRanges.size();
         while (boundaries.size() < parts - 1 && i < rangesCount - 1)
         {
-            Range<Token> r = localRanges.get(i);
+            WeightedRange r = weightedRanges.get(i);
-            Range<Token> nextRange = localRanges.get(i + 1);
+            WeightedRange nextRange = weightedRanges.get(i + 1);
-            Token right = token(r.right);
-            Token nextRight = token(nextRange.right);
 
-            BigInteger currentRangeWidth = valueForToken(right).subtract(valueForToken(r.left));
+            BigInteger currentRangeWidth = r.totalTokens(this);
-            BigInteger nextRangeWidth = valueForToken(nextRight).subtract(valueForToken(nextRange.left));
+            BigInteger nextRangeWidth = nextRange.totalTokens(this);
             sum = sum.add(currentRangeWidth);
 
             // does this or next range take us beyond the per part limit?
@@ -187,7 +185,7 @@ public abstract class Splitter
                 if (diffNext.compareTo(diffCurrent) >= 0)
                 {
                     sum = BigInteger.ZERO;
-                    boundaries.add(right);
+                    boundaries.add(token(r.right()));
                 }
             }
             i++;
@@ -256,4 +254,61 @@ public abstract class Splitter
         }
         return subranges;
     }
+
+    public static class WeightedRange
+    {
+        private final double weight;
+        private final Range<Token> range;
+
+        public WeightedRange(double weight, Range<Token> range)
+        {
+            this.weight = weight;
+            this.range = range;
+        }
+
+        public BigInteger totalTokens(Splitter splitter)
+        {
+            BigInteger right = splitter.valueForToken(splitter.token(range.right));
+            BigInteger left = splitter.valueForToken(range.left);
+            BigInteger factor = BigInteger.valueOf(Math.max(1, (long) (1 / weight)));
+            BigInteger size = right.subtract(left);
+            return size.abs().divide(factor);
+        }
+
+        public Token left()
+        {
+            return range.left;
+        }
+
+        public Token right()
+        {
+            return range.right;
+        }
+
+        public Range<Token> range()
+        {
+            return range;
+        }
+
+        public String toString()
+        {
+            return "WeightedRange{" +
+                   "weight=" + weight +
+                   ", range=" + range +
+                   '}';
+        }
+
+        public boolean equals(Object o)
+        {
+            if (this == o) return true;
+            if (!(o instanceof WeightedRange)) return false;
+            WeightedRange that = (WeightedRange) o;
+            return Objects.equals(range, that.range);
+        }
+
+        public int hashCode()
+        {
+            return Objects.hash(weight, range);
+        }
+    }
 }

src/java/org/apache/cassandra/dht/StreamStateStore.java

@@ -19,38 +19,43 @@ package org.apache.cassandra.dht;
 
 import java.util.Set;
 
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
 import org.apache.cassandra.db.SystemKeyspace;
 import org.apache.cassandra.streaming.StreamEvent;
 import org.apache.cassandra.streaming.StreamEventHandler;
 import org.apache.cassandra.streaming.StreamRequest;
 import org.apache.cassandra.streaming.StreamState;
+import org.apache.cassandra.utils.Pair;
 
 /**
  * Store and update available ranges (data already received) to system keyspace.
  */
 public class StreamStateStore implements StreamEventHandler
 {
-    public Set<Range<Token>> getAvailableRanges(String keyspace, IPartitioner partitioner)
+    private static final Logger logger = LoggerFactory.getLogger(StreamStateStore.class);
+
+    public RangesAtEndpoint getAvailableRanges(String keyspace, IPartitioner partitioner)
     {
         return SystemKeyspace.getAvailableRanges(keyspace, partitioner);
     }
 
     /**
-     * Check if given token's data is available in this node.
+     * Check if given token's data is available in this node. This doesn't handle transientness in a useful way
+     * so it's only used by a legacy test
      *
      * @param keyspace keyspace name
      * @param token token to check
      * @return true if given token in the keyspace is already streamed and ready to be served.
      */
+    @VisibleForTesting
     public boolean isDataAvailable(String keyspace, Token token)
     {
-        Set<Range<Token>> availableRanges = getAvailableRanges(keyspace, token.getPartitioner());
+        RangesAtEndpoint availableRanges = getAvailableRanges(keyspace, token.getPartitioner());
-        for (Range<Token> range : availableRanges)
+        return availableRanges.ranges().stream().anyMatch(range -> range.contains(token));
-        {
-            if (range.contains(token))
-                return true;
-        }
-        return false;
     }
 
     /**
@@ -73,7 +78,7 @@ public class StreamStateStore implements StreamEventHandler
                 }
                 for (StreamRequest request : se.requests)
                 {
-                    SystemKeyspace.updateAvailableRanges(request.keyspace, request.ranges);
+                    SystemKeyspace.updateAvailableRanges(request.keyspace, request.full.ranges(), request.transientReplicas.ranges());
                 }
             }
         }

src/java/org/apache/cassandra/dht/tokenallocator/ReplicationAwareTokenAllocator.java

@@ -73,7 +73,7 @@ class ReplicationAwareTokenAllocator<Unit> extends TokenAllocatorBase<Unit>
         Map<Unit, UnitInfo<Unit>> unitInfos = createUnitInfos(groups);
         if (groups.size() < replicas)
         {
-            // We need at least replicas groups to do allocation correctly. If there aren't enough, 
+            // We need at least replicas groups to do allocation correctly. If there aren't enough,
             // use random allocation.
             // This part of the code should only be reached via the RATATest. StrategyAdapter should disallow
             // token allocation in this case as the algorithm is not able to cover the behavior of NetworkTopologyStrategy.

src/java/org/apache/cassandra/dht/tokenallocator/TokenAllocation.java

@@ -113,7 +113,7 @@ public class TokenAllocation
     {
         double size = current.size(next);
         Token representative = current.getPartitioner().midpoint(current, next);
-        for (InetAddressAndPort n : rs.calculateNaturalEndpoints(representative, tokenMetadata))
+        for (InetAddressAndPort n : rs.calculateNaturalReplicas(representative, tokenMetadata).endpoints())
         {
             Double v = ownership.get(n);
             ownership.put(n, v != null ? v + size : size);
@@ -169,7 +169,7 @@ public class TokenAllocation
 
     static StrategyAdapter getStrategy(final TokenMetadata tokenMetadata, final SimpleStrategy rs, final InetAddressAndPort endpoint)
     {
-        final int replicas = rs.getReplicationFactor();
+        final int replicas = rs.getReplicationFactor().allReplicas;
 
         return new StrategyAdapter()
         {
@@ -196,7 +196,7 @@ public class TokenAllocation
     static StrategyAdapter getStrategy(final TokenMetadata tokenMetadata, final NetworkTopologyStrategy rs, final IEndpointSnitch snitch, final InetAddressAndPort endpoint)
     {
         final String dc = snitch.getDatacenter(endpoint);
-        final int replicas = rs.getReplicationFactor(dc);
+        final int replicas = rs.getReplicationFactor(dc).allReplicas;
 
         if (replicas == 0 || replicas == 1)
         {

src/java/org/apache/cassandra/exceptions/UnavailableException.java

@@ -25,14 +25,26 @@ public class UnavailableException extends RequestExecutionException
     public final int required;
     public final int alive;
 
-    public UnavailableException(ConsistencyLevel consistency, int required, int alive)
+    public static UnavailableException create(ConsistencyLevel consistency, int required, int alive)
     {
-        this("Cannot achieve consistency level " + consistency, consistency, required, alive);
+        assert alive < required;
+        return create(consistency, required, 0, alive, 0);
     }
 
-    public UnavailableException(ConsistencyLevel consistency, String dc, int required, int alive)
+    public static UnavailableException create(ConsistencyLevel consistency, int required, int requiredFull, int alive, int aliveFull)
     {
-        this("Cannot achieve consistency level " + consistency + " in DC " + dc, consistency, required, alive);
+        if (required > alive)
+            return new UnavailableException("Cannot achieve consistency level " + consistency, consistency, required, alive);
+        assert requiredFull < aliveFull;
+        return new UnavailableException("Insufficient full replicas", consistency, required, alive);
+    }
+
+    public static UnavailableException create(ConsistencyLevel consistency, String dc, int required, int requiredFull, int alive, int aliveFull)
+    {
+        if (required > alive)
+            return new UnavailableException("Cannot achieve consistency level " + consistency + " in DC " + dc, consistency, required, alive);
+        assert requiredFull < aliveFull;
+        return new UnavailableException("Insufficient full replicas in DC " + dc, consistency, required, alive);
     }
 
     public UnavailableException(String msg, ConsistencyLevel consistency, int required, int alive)

src/java/org/apache/cassandra/gms/EndpointState.java

@@ -144,6 +144,11 @@ public class EndpointState
         return rpcState != null && Boolean.parseBoolean(rpcState.value);
     }
 
+    public boolean isNormalState()
+    {
+        return getStatus().equals(VersionedValue.STATUS_NORMAL);
+    }
+
     public String getStatus()
     {
         VersionedValue status = getApplicationState(ApplicationState.STATUS_WITH_PORT);

src/java/org/apache/cassandra/hints/HintsService.java

@@ -20,11 +20,14 @@ package org.apache.cassandra.hints;
 import java.io.File;
 import java.lang.management.ManagementFactory;
 import java.net.UnknownHostException;
+import java.util.Collection;
 import java.util.Collections;
+import java.util.List;
 import java.util.UUID;
 import java.util.concurrent.*;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.function.Supplier;
+import java.util.stream.Collectors;
 
 import javax.management.MBeanServer;
 import javax.management.ObjectName;
@@ -39,6 +42,7 @@ import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.config.ParameterizedClass;
 import org.apache.cassandra.gms.FailureDetector;
 import org.apache.cassandra.gms.IFailureDetector;
+import org.apache.cassandra.locator.EndpointsForToken;
 import org.apache.cassandra.locator.InetAddressAndPort;
 import org.apache.cassandra.metrics.HintedHandoffMetrics;
 import org.apache.cassandra.metrics.StorageMetrics;
@@ -46,9 +50,7 @@ import org.apache.cassandra.dht.Token;
 import org.apache.cassandra.service.StorageProxy;
 import org.apache.cassandra.service.StorageService;
 
-import static com.google.common.collect.Iterables.filter;
 import static com.google.common.collect.Iterables.transform;
-import static com.google.common.collect.Iterables.size;
 
 /**
  * A singleton-ish wrapper over various hints components:
@@ -151,7 +153,7 @@ public final class HintsService implements HintsServiceMBean
      * @param hostIds host ids of the hint's target nodes
      * @param hint the hint to store
      */
-    public void write(Iterable<UUID> hostIds, Hint hint)
+    public void write(Collection<UUID> hostIds, Hint hint)
     {
         if (isShutDown)
             throw new IllegalStateException("HintsService is shut down and can't accept new hints");
@@ -161,7 +163,7 @@ public final class HintsService implements HintsServiceMBean
 
         bufferPool.write(hostIds, hint);
 
-        StorageMetrics.totalHints.inc(size(hostIds));
+        StorageMetrics.totalHints.inc(hostIds.size());
     }
 
     /**
@@ -183,9 +185,14 @@ public final class HintsService implements HintsServiceMBean
         String keyspaceName = hint.mutation.getKeyspaceName();
         Token token = hint.mutation.key().getToken();
 
-        Iterable<UUID> hostIds =
+        EndpointsForToken replicas = StorageService.instance.getNaturalAndPendingReplicasForToken(keyspaceName, token);
-        transform(filter(StorageService.instance.getNaturalAndPendingEndpoints(keyspaceName, token), StorageProxy::shouldHint),
+
-                  StorageService.instance::getHostIdForEndpoint);
+        // judicious use of streams: eagerly materializing probably cheaper
+        // than performing filters / translations 2x extra via Iterables.filter/transform
+        List<UUID> hostIds = replicas.stream()
+                .filter(StorageProxy::shouldHint)
+                .map(replica -> StorageService.instance.getHostIdForEndpoint(replica.endpoint()))
+                .collect(Collectors.toList());
 
         write(hostIds, hint);
     }

src/java/org/apache/cassandra/io/sstable/AbstractSSTableSimpleWriter.java

@@ -69,13 +69,14 @@ abstract class AbstractSSTableSimpleWriter implements Closeable
         SerializationHeader header = new SerializationHeader(true, metadata.get(), columns, EncodingStats.NO_STATS);
 
         if (makeRangeAware)
-            return SSTableTxnWriter.createRangeAware(metadata, 0,  ActiveRepairService.UNREPAIRED_SSTABLE, ActiveRepairService.NO_PENDING_REPAIR, formatType, 0, header);
+            return SSTableTxnWriter.createRangeAware(metadata, 0,  ActiveRepairService.UNREPAIRED_SSTABLE, ActiveRepairService.NO_PENDING_REPAIR, false, formatType, 0, header);
 
         return SSTableTxnWriter.create(metadata,
                                        createDescriptor(directory, metadata.keyspace, metadata.name, formatType),
                                        0,
                                        ActiveRepairService.UNREPAIRED_SSTABLE,
                                        ActiveRepairService.NO_PENDING_REPAIR,
+                                       false,
                                        0,
                                        header,
                                        Collections.emptySet());

src/java/org/apache/cassandra/io/sstable/SSTable.java

@@ -23,6 +23,7 @@ import java.nio.charset.Charset;
 import java.util.*;
 import java.util.concurrent.CopyOnWriteArraySet;
 
+import com.google.common.base.Preconditions;
 import com.google.common.base.Predicates;
 import com.google.common.collect.Collections2;
 import com.google.common.collect.Sets;
@@ -46,6 +47,9 @@ import org.apache.cassandra.utils.ByteBufferUtil;
 import org.apache.cassandra.utils.Pair;
 import org.apache.cassandra.utils.memory.HeapAllocator;
 
+import static org.apache.cassandra.service.ActiveRepairService.NO_PENDING_REPAIR;
+import static org.apache.cassandra.service.ActiveRepairService.UNREPAIRED_SSTABLE;
+
 /**
  * This class is built on top of the SequenceFile. It stores
  * data on disk in sorted fashion. However the sorting is upto
@@ -350,4 +354,13 @@ public abstract class SSTable
     {
         return AbstractBounds.bounds(first.getToken(), true, last.getToken(), true);
     }
+
+    public static void validateRepairedMetadata(long repairedAt, UUID pendingRepair, boolean isTransient)
+    {
+        Preconditions.checkArgument((pendingRepair == NO_PENDING_REPAIR) || (repairedAt == UNREPAIRED_SSTABLE),
+                                    "pendingRepair cannot be set on a repaired sstable");
+        Preconditions.checkArgument(!isTransient || (pendingRepair != NO_PENDING_REPAIR),
+                                    "isTransient can only be true for sstables pending repair");
+
+    }
 }

src/java/org/apache/cassandra/io/sstable/SSTableLoader.java

@@ -126,7 +126,7 @@ public class SSTableLoader implements StreamEventHandler
                                               for (Map.Entry<InetAddressAndPort, Collection<Range<Token>>> entry : ranges.entrySet())
                                               {
                                                   InetAddressAndPort endpoint = entry.getKey();
-                                                  Collection<Range<Token>> tokenRanges = entry.getValue();
+                                                  List<Range<Token>> tokenRanges = Range.normalize(entry.getValue());
 
                                                   List<SSTableReader.PartitionPositionBounds> sstableSections = sstable.getPositionsForRanges(tokenRanges);
                                                   long estimatedKeys = sstable.estimatedKeysForRanges(tokenRanges);

src/java/org/apache/cassandra/io/sstable/SSTableTxnWriter.java

@@ -99,10 +99,10 @@ public class SSTableTxnWriter extends Transactional.AbstractTransactional implem
     }
 
     @SuppressWarnings("resource") // log and writer closed during doPostCleanup
-    public static SSTableTxnWriter create(ColumnFamilyStore cfs, Descriptor descriptor, long keyCount, long repairedAt, UUID pendingRepair, int sstableLevel, SerializationHeader header)
+    public static SSTableTxnWriter create(ColumnFamilyStore cfs, Descriptor descriptor, long keyCount, long repairedAt, UUID pendingRepair, boolean isTransient, int sstableLevel, SerializationHeader header)
     {
         LifecycleTransaction txn = LifecycleTransaction.offline(OperationType.WRITE);
-        SSTableMultiWriter writer = cfs.createSSTableMultiWriter(descriptor, keyCount, repairedAt, pendingRepair, sstableLevel, header, txn);
+        SSTableMultiWriter writer = cfs.createSSTableMultiWriter(descriptor, keyCount, repairedAt, pendingRepair, isTransient, sstableLevel, header, txn);
         return new SSTableTxnWriter(txn, writer);
     }
 
@@ -112,6 +112,7 @@ public class SSTableTxnWriter extends Transactional.AbstractTransactional implem
                                                     long keyCount,
                                                     long repairedAt,
                                                     UUID pendingRepair,
+                                                    boolean isTransient,
                                                     SSTableFormat.Type type,
                                                     int sstableLevel,
                                                     SerializationHeader header)
@@ -122,7 +123,7 @@ public class SSTableTxnWriter extends Transactional.AbstractTransactional implem
         SSTableMultiWriter writer;
         try
         {
-            writer = new RangeAwareSSTableWriter(cfs, keyCount, repairedAt, pendingRepair, type, sstableLevel, 0, txn, header);
+            writer = new RangeAwareSSTableWriter(cfs, keyCount, repairedAt, pendingRepair, isTransient, type, sstableLevel, 0, txn, header);
         }
         catch (IOException e)
         {
@@ -140,6 +141,7 @@ public class SSTableTxnWriter extends Transactional.AbstractTransactional implem
                                           long keyCount,
                                           long repairedAt,
                                           UUID pendingRepair,
+                                          boolean isTransient,
                                           int sstableLevel,
                                           SerializationHeader header,
                                           Collection<Index> indexes)
@@ -147,12 +149,12 @@ public class SSTableTxnWriter extends Transactional.AbstractTransactional implem
         // if the column family store does not exist, we create a new default SSTableMultiWriter to use:
         LifecycleTransaction txn = LifecycleTransaction.offline(OperationType.WRITE);
         MetadataCollector collector = new MetadataCollector(metadata.get().comparator).sstableLevel(sstableLevel);
-        SSTableMultiWriter writer = SimpleSSTableMultiWriter.create(descriptor, keyCount, repairedAt, pendingRepair, metadata, collector, header, indexes, txn);
+        SSTableMultiWriter writer = SimpleSSTableMultiWriter.create(descriptor, keyCount, repairedAt, pendingRepair, isTransient, metadata, collector, header, indexes, txn);
         return new SSTableTxnWriter(txn, writer);
     }
 
-    public static SSTableTxnWriter create(ColumnFamilyStore cfs, Descriptor desc, long keyCount, long repairedAt, UUID pendingRepair, SerializationHeader header)
+    public static SSTableTxnWriter create(ColumnFamilyStore cfs, Descriptor desc, long keyCount, long repairedAt, UUID pendingRepair, boolean isTransient, SerializationHeader header)
     {
-        return create(cfs, desc, keyCount, repairedAt, pendingRepair, 0, header);
+        return create(cfs, desc, keyCount, repairedAt, pendingRepair, isTransient, 0, header);
     }
 }

src/java/org/apache/cassandra/io/sstable/SimpleSSTableMultiWriter.java

@@ -111,13 +111,14 @@ public class SimpleSSTableMultiWriter implements SSTableMultiWriter
                                             long keyCount,
                                             long repairedAt,
                                             UUID pendingRepair,
+                                            boolean isTransient,
                                             TableMetadataRef metadata,
                                             MetadataCollector metadataCollector,
                                             SerializationHeader header,
                                             Collection<Index> indexes,
                                             LifecycleTransaction txn)
     {
-        SSTableWriter writer = SSTableWriter.create(descriptor, keyCount, repairedAt, pendingRepair, metadata, metadataCollector, header, indexes, txn);
+        SSTableWriter writer = SSTableWriter.create(descriptor, keyCount, repairedAt, pendingRepair, isTransient, metadata, metadataCollector, header, indexes, txn);
         return new SimpleSSTableMultiWriter(writer, txn);
     }
 }

src/java/org/apache/cassandra/io/sstable/format/RangeAwareSSTableWriter.java

@@ -44,6 +44,7 @@ public class RangeAwareSSTableWriter implements SSTableMultiWriter
     private final long estimatedKeys;
     private final long repairedAt;
     private final UUID pendingRepair;
+    private final boolean isTransient;
     private final SSTableFormat.Type format;
     private final SerializationHeader header;
     private final LifecycleTransaction txn;
@@ -53,7 +54,7 @@ public class RangeAwareSSTableWriter implements SSTableMultiWriter
     private final List<SSTableReader> finishedReaders = new ArrayList<>();
     private SSTableMultiWriter currentWriter = null;
 
-    public RangeAwareSSTableWriter(ColumnFamilyStore cfs, long estimatedKeys, long repairedAt, UUID pendingRepair, SSTableFormat.Type format, int sstableLevel, long totalSize, LifecycleTransaction txn, SerializationHeader header) throws IOException
+    public RangeAwareSSTableWriter(ColumnFamilyStore cfs, long estimatedKeys, long repairedAt, UUID pendingRepair, boolean isTransient, SSTableFormat.Type format, int sstableLevel, long totalSize, LifecycleTransaction txn, SerializationHeader header) throws IOException
     {
         DiskBoundaries db = cfs.getDiskBoundaries();
         directories = db.directories;
@@ -62,6 +63,7 @@ public class RangeAwareSSTableWriter implements SSTableMultiWriter
         this.estimatedKeys = estimatedKeys / directories.size();
         this.repairedAt = repairedAt;
         this.pendingRepair = pendingRepair;
+        this.isTransient = isTransient;
         this.format = format;
         this.txn = txn;
         this.header = header;
@@ -73,7 +75,7 @@ public class RangeAwareSSTableWriter implements SSTableMultiWriter
                 throw new IOException(String.format("Insufficient disk space to store %s",
                                                     FBUtilities.prettyPrintMemory(totalSize)));
             Descriptor desc = cfs.newSSTableDescriptor(cfs.getDirectories().getLocationForDisk(localDir), format);
-            currentWriter = cfs.createSSTableMultiWriter(desc, estimatedKeys, repairedAt, pendingRepair, sstableLevel, header, txn);
+            currentWriter = cfs.createSSTableMultiWriter(desc, estimatedKeys, repairedAt, pendingRepair, isTransient, sstableLevel, header, txn);
         }
     }
 
@@ -95,7 +97,7 @@ public class RangeAwareSSTableWriter implements SSTableMultiWriter
                 finishedWriters.add(currentWriter);
 
             Descriptor desc = cfs.newSSTableDescriptor(cfs.getDirectories().getLocationForDisk(directories.get(currentIndex)), format);
-            currentWriter = cfs.createSSTableMultiWriter(desc, estimatedKeys, repairedAt, pendingRepair, sstableLevel, header, txn);
+            currentWriter = cfs.createSSTableMultiWriter(desc, estimatedKeys, repairedAt, pendingRepair, isTransient, sstableLevel, header, txn);
         }
     }
 

src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java

@@ -1852,6 +1852,11 @@ public abstract class SSTableReader extends SSTable implements SelfRefCounted<SS
         return sstableMetadata.repairedAt;
     }
 
+    public boolean isTransient()
+    {
+        return sstableMetadata.isTransient;
+    }
+
     public boolean intersects(Collection<Range<Token>> ranges)
     {
         Bounds<Token> range = new Bounds<>(first.getToken(), last.getToken());

src/java/org/apache/cassandra/io/sstable/format/SSTableWriter.java

@@ -55,6 +55,7 @@ public abstract class SSTableWriter extends SSTable implements Transactional
 {
     protected long repairedAt;
     protected UUID pendingRepair;
+    protected boolean isTransient;
     protected long maxDataAge = -1;
     protected final long keyCount;
     protected final MetadataCollector metadataCollector;
@@ -77,6 +78,7 @@ public abstract class SSTableWriter extends SSTable implements Transactional
                             long keyCount,
                             long repairedAt,
                             UUID pendingRepair,
+                            boolean isTransient,
                             TableMetadataRef metadata,
                             MetadataCollector metadataCollector,
                             SerializationHeader header,
@@ -86,6 +88,7 @@ public abstract class SSTableWriter extends SSTable implements Transactional
         this.keyCount = keyCount;
         this.repairedAt = repairedAt;
         this.pendingRepair = pendingRepair;
+        this.isTransient = isTransient;
         this.metadataCollector = metadataCollector;
         this.header = header;
         this.rowIndexEntrySerializer = descriptor.version.getSSTableFormat().getIndexSerializer(metadata.get(), descriptor.version, header);
@@ -96,6 +99,7 @@ public abstract class SSTableWriter extends SSTable implements Transactional
                                        Long keyCount,
                                        Long repairedAt,
                                        UUID pendingRepair,
+                                       boolean isTransient,
                                        TableMetadataRef metadata,
                                        MetadataCollector metadataCollector,
                                        SerializationHeader header,
@@ -103,20 +107,21 @@ public abstract class SSTableWriter extends SSTable implements Transactional
                                        LifecycleTransaction txn)
     {
         Factory writerFactory = descriptor.getFormat().getWriterFactory();
-        return writerFactory.open(descriptor, keyCount, repairedAt, pendingRepair, metadata, metadataCollector, header, observers(descriptor, indexes, txn.opType()), txn);
+        return writerFactory.open(descriptor, keyCount, repairedAt, pendingRepair, isTransient, metadata, metadataCollector, header, observers(descriptor, indexes, txn.opType()), txn);
     }
 
     public static SSTableWriter create(Descriptor descriptor,
                                        long keyCount,
                                        long repairedAt,
                                        UUID pendingRepair,
+                                       boolean isTransient,
                                        int sstableLevel,
                                        SerializationHeader header,
                                        Collection<Index> indexes,
                                        LifecycleTransaction txn)
     {
         TableMetadataRef metadata = Schema.instance.getTableMetadataRef(descriptor);
-        return create(metadata, descriptor, keyCount, repairedAt, pendingRepair, sstableLevel, header, indexes, txn);
+        return create(metadata, descriptor, keyCount, repairedAt, pendingRepair, isTransient, sstableLevel, header, indexes, txn);
     }
 
     public static SSTableWriter create(TableMetadataRef metadata,
@@ -124,13 +129,14 @@ public abstract class SSTableWriter extends SSTable implements Transactional
                                        long keyCount,
                                        long repairedAt,
                                        UUID pendingRepair,
+                                       boolean isTransient,
                                        int sstableLevel,
                                        SerializationHeader header,
                                        Collection<Index> indexes,
                                        LifecycleTransaction txn)
     {
         MetadataCollector collector = new MetadataCollector(metadata.get().comparator).sstableLevel(sstableLevel);
-        return create(descriptor, keyCount, repairedAt, pendingRepair, metadata, collector, header, indexes, txn);
+        return create(descriptor, keyCount, repairedAt, pendingRepair, isTransient, metadata, collector, header, indexes, txn);
     }
 
     @VisibleForTesting
@@ -138,11 +144,12 @@ public abstract class SSTableWriter extends SSTable implements Transactional
                                        long keyCount,
                                        long repairedAt,
                                        UUID pendingRepair,
+                                       boolean isTransient,
                                        SerializationHeader header,
                                        Collection<Index> indexes,
                                        LifecycleTransaction txn)
     {
-        return create(descriptor, keyCount, repairedAt, pendingRepair, 0, header, indexes, txn);
+        return create(descriptor, keyCount, repairedAt, pendingRepair, isTransient, 0, header, indexes, txn);
     }
 
     private static Set<Component> components(TableMetadata metadata)
@@ -309,6 +316,7 @@ public abstract class SSTableWriter extends SSTable implements Transactional
                                                   metadata().params.bloomFilterFpChance,
                                                   repairedAt,
                                                   pendingRepair,
+                                                  isTransient,
                                                   header);
     }
 
@@ -338,6 +346,7 @@ public abstract class SSTableWriter extends SSTable implements Transactional
                                            long keyCount,
                                            long repairedAt,
                                            UUID pendingRepair,
+                                           boolean isTransient,
                                            TableMetadataRef metadata,
                                            MetadataCollector metadataCollector,
                                            SerializationHeader header,

src/java/org/apache/cassandra/io/sstable/format/Version.java

@@ -55,6 +55,8 @@ public abstract class Version
 
     public abstract boolean hasPendingRepair();
 
+    public abstract boolean hasIsTransient();
+
     public abstract boolean hasMetadataChecksum();
 
     /**

src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java

@@ -21,6 +21,9 @@ import java.util.Collection;
 import java.util.Set;
 import java.util.UUID;
 
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.io.sstable.SSTable;
 import org.apache.cassandra.schema.TableMetadata;
 import org.apache.cassandra.schema.TableMetadataRef;
 import org.apache.cassandra.db.RowIndexEntry;
@@ -85,13 +88,15 @@ public class BigFormat implements SSTableFormat
                                   long keyCount,
                                   long repairedAt,
                                   UUID pendingRepair,
+                                  boolean isTransient,
                                   TableMetadataRef metadata,
                                   MetadataCollector metadataCollector,
                                   SerializationHeader header,
                                   Collection<SSTableFlushObserver> observers,
                                   LifecycleTransaction txn)
         {
-            return new BigTableWriter(descriptor, keyCount, repairedAt, pendingRepair, metadata, metadataCollector, header, observers, txn);
+            SSTable.validateRepairedMetadata(repairedAt, pendingRepair, isTransient);
+            return new BigTableWriter(descriptor, keyCount, repairedAt, pendingRepair, isTransient, metadata, metadataCollector, header, observers, txn);
         }
     }
 
@@ -120,7 +125,7 @@ public class BigFormat implements SSTableFormat
         // mb (3.0.7, 3.7): commit log lower bound included
         // mc (3.0.8, 3.9): commit log intervals included
 
-        // na (4.0.0): uncompressed chunks, pending repair session, checksummed sstable metadata file, new Bloomfilter format
+        // na (4.0.0): uncompressed chunks, pending repair session, isTransient, checksummed sstable metadata file, new Bloomfilter format
         //
         // NOTE: when adding a new version, please add that to LegacySSTableTest, too.
 
@@ -131,6 +136,7 @@ public class BigFormat implements SSTableFormat
         public final boolean hasMaxCompressedLength;
         private final boolean hasPendingRepair;
         private final boolean hasMetadataChecksum;
+        private final boolean hasIsTransient;
         /**
          * CASSANDRA-9067: 4.0 bloom filter representation changed (two longs just swapped)
          * have no 'static' bits caused by using the same upper bits for both bloom filter and token distribution.
@@ -148,6 +154,7 @@ public class BigFormat implements SSTableFormat
             hasCommitLogIntervals = version.compareTo("mc") >= 0;
             hasMaxCompressedLength = version.compareTo("na") >= 0;
             hasPendingRepair = version.compareTo("na") >= 0;
+            hasIsTransient = version.compareTo("na") >= 0;
             hasMetadataChecksum = version.compareTo("na") >= 0;
             hasOldBfFormat = version.compareTo("na") < 0;
         }
@@ -175,6 +182,12 @@ public class BigFormat implements SSTableFormat
             return hasPendingRepair;
         }
 
+        @Override
+        public boolean hasIsTransient()
+        {
+            return hasIsTransient;
+        }
+
         @Override
         public int correspondingMessagingVersion()
         {

src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java

@@ -68,13 +68,14 @@ public class BigTableWriter extends SSTableWriter
                           long keyCount,
                           long repairedAt,
                           UUID pendingRepair,
+                          boolean isTransient,
                           TableMetadataRef metadata,
                           MetadataCollector metadataCollector, 
                           SerializationHeader header,
                           Collection<SSTableFlushObserver> observers,
                           LifecycleTransaction txn)
     {
-        super(descriptor, keyCount, repairedAt, pendingRepair, metadata, metadataCollector, header, observers);
+        super(descriptor, keyCount, repairedAt, pendingRepair, isTransient, metadata, metadataCollector, header, observers);
         txn.trackNew(this); // must track before any files are created
 
         if (compression)

src/java/org/apache/cassandra/io/sstable/metadata/IMetadataSerializer.java

@@ -71,7 +71,7 @@ public interface IMetadataSerializer
     void mutateLevel(Descriptor descriptor, int newLevel) throws IOException;
 
     /**
-     * Mutate the repairedAt time and pendingRepair ID
+     * Mutate the repairedAt time, pendingRepair ID, and transient status
      */
-    void mutateRepaired(Descriptor descriptor, long newRepairedAt, UUID newPendingRepair) throws IOException;
+    public void mutateRepairMetadata(Descriptor descriptor, long newRepairedAt, UUID newPendingRepair, boolean isTransient) throws IOException;
 }

src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java

@@ -83,7 +83,8 @@ public class MetadataCollector implements PartitionStatisticsCollector
                                  ActiveRepairService.UNREPAIRED_SSTABLE,
                                  -1,
                                  -1,
-                                 null);
+                                 null,
+                                 false);
     }
 
     protected EstimatedHistogram estimatedPartitionSize = defaultPartitionSizeHistogram();
@@ -272,7 +273,7 @@ public class MetadataCollector implements PartitionStatisticsCollector
         this.hasLegacyCounterShards = this.hasLegacyCounterShards || hasLegacyCounterShards;
     }
 
-    public Map<MetadataType, MetadataComponent> finalizeMetadata(String partitioner, double bloomFilterFPChance, long repairedAt, UUID pendingRepair, SerializationHeader header)
+    public Map<MetadataType, MetadataComponent> finalizeMetadata(String partitioner, double bloomFilterFPChance, long repairedAt, UUID pendingRepair, boolean isTransient, SerializationHeader header)
     {
         Map<MetadataType, MetadataComponent> components = new EnumMap<>(MetadataType.class);
         components.put(MetadataType.VALIDATION, new ValidationMetadata(partitioner, bloomFilterFPChance));
@@ -294,7 +295,8 @@ public class MetadataCollector implements PartitionStatisticsCollector
                                                              repairedAt,
                                                              totalColumnsSet,
                                                              totalRows,
-                                                             pendingRepair));
+                                                             pendingRepair,
+                                                             isTransient));
         components.put(MetadataType.COMPACTION, new CompactionMetadata(cardinality));
         components.put(MetadataType.HEADER, header.toComponent());
         return components;

src/java/org/apache/cassandra/io/sstable/metadata/MetadataSerializer.java

@@ -230,7 +230,7 @@ public class MetadataSerializer implements IMetadataSerializer
         rewriteSSTableMetadata(descriptor, currentComponents);
     }
 
-    public void mutateRepaired(Descriptor descriptor, long newRepairedAt, UUID newPendingRepair) throws IOException
+    public void mutateRepairMetadata(Descriptor descriptor, long newRepairedAt, UUID newPendingRepair, boolean isTransient) throws IOException
     {
         if (logger.isTraceEnabled())
             logger.trace("Mutating {} to repairedAt time {} and pendingRepair {}",
@@ -238,7 +238,7 @@ public class MetadataSerializer implements IMetadataSerializer
         Map<MetadataType, MetadataComponent> currentComponents = deserialize(descriptor, EnumSet.allOf(MetadataType.class));
         StatsMetadata stats = (StatsMetadata) currentComponents.remove(MetadataType.STATS);
         // mutate time & id
-        currentComponents.put(MetadataType.STATS, stats.mutateRepairedAt(newRepairedAt).mutatePendingRepair(newPendingRepair));
+        currentComponents.put(MetadataType.STATS, stats.mutateRepairedMetadata(newRepairedAt, newPendingRepair, isTransient));
         rewriteSSTableMetadata(descriptor, currentComponents);
     }
 

src/java/org/apache/cassandra/io/sstable/metadata/StatsMetadata.java

@@ -64,6 +64,7 @@ public class StatsMetadata extends MetadataComponent
     public final long totalColumnsSet;
     public final long totalRows;
     public final UUID pendingRepair;
+    public final boolean isTransient;
 
     public StatsMetadata(EstimatedHistogram estimatedPartitionSize,
                          EstimatedHistogram estimatedColumnCount,
@@ -83,7 +84,8 @@ public class StatsMetadata extends MetadataComponent
                          long repairedAt,
                          long totalColumnsSet,
                          long totalRows,
-                         UUID pendingRepair)
+                         UUID pendingRepair,
+                         boolean isTransient)
     {
         this.estimatedPartitionSize = estimatedPartitionSize;
         this.estimatedColumnCount = estimatedColumnCount;
@@ -104,6 +106,7 @@ public class StatsMetadata extends MetadataComponent
         this.totalColumnsSet = totalColumnsSet;
         this.totalRows = totalRows;
         this.pendingRepair = pendingRepair;
+        this.isTransient = isTransient;
     }
 
     public MetadataType getType()
@@ -155,10 +158,11 @@ public class StatsMetadata extends MetadataComponent
                                  repairedAt,
                                  totalColumnsSet,
                                  totalRows,
-                                 pendingRepair);
+                                 pendingRepair,
+                                 isTransient);
     }
 
-    public StatsMetadata mutateRepairedAt(long newRepairedAt)
+    public StatsMetadata mutateRepairedMetadata(long newRepairedAt, UUID newPendingRepair, boolean newIsTransient)
     {
         return new StatsMetadata(estimatedPartitionSize,
                                  estimatedColumnCount,
@@ -178,30 +182,8 @@ public class StatsMetadata extends MetadataComponent
                                  newRepairedAt,
                                  totalColumnsSet,
                                  totalRows,
-                                 pendingRepair);
+                                 newPendingRepair,
-    }
+                                 newIsTransient);
-
-    public StatsMetadata mutatePendingRepair(UUID newPendingRepair)
-    {
-        return new StatsMetadata(estimatedPartitionSize,
-                                 estimatedColumnCount,
-                                 commitLogIntervals,
-                                 minTimestamp,
-                                 maxTimestamp,
-                                 minLocalDeletionTime,
-                                 maxLocalDeletionTime,
-                                 minTTL,
-                                 maxTTL,
-                                 compressionRatio,
-                                 estimatedTombstoneDropTime,
-                                 sstableLevel,
-                                 minClusteringValues,
-                                 maxClusteringValues,
-                                 hasLegacyCounterShards,
-                                 repairedAt,
-                                 totalColumnsSet,
-                                 totalRows,
-                                 newPendingRepair);
     }
 
     @Override
@@ -292,6 +274,12 @@ public class StatsMetadata extends MetadataComponent
                 if (component.pendingRepair != null)
                     size += UUIDSerializer.serializer.serializedSize(component.pendingRepair, 0);
             }
+
+            if (version.hasIsTransient())
+            {
+                size += TypeSizes.sizeof(component.isTransient);
+            }
+
             return size;
         }
 
@@ -338,6 +326,11 @@ public class StatsMetadata extends MetadataComponent
                     out.writeByte(0);
                 }
             }
+
+            if (version.hasIsTransient())
+            {
+                out.writeBoolean(component.isTransient);
+            }
         }
 
         public StatsMetadata deserialize(Version version, DataInputPlus in) throws IOException
@@ -386,6 +379,8 @@ public class StatsMetadata extends MetadataComponent
                 pendingRepair = UUIDSerializer.serializer.deserialize(in, 0);
             }
 
+            boolean isTransient = version.hasIsTransient() && in.readBoolean();
+
             return new StatsMetadata(partitionSizes,
                                      columnCounts,
                                      commitLogIntervals,
@@ -404,7 +399,8 @@ public class StatsMetadata extends MetadataComponent
                                      repairedAt,
                                      totalColumnsSet,
                                      totalRows,
-                                     pendingRepair);
+                                     pendingRepair,
+                                     isTransient);
         }
     }
 }

src/java/org/apache/cassandra/locator/AbstractEndpointSnitch.java

@@ -17,13 +17,12 @@
  */
 package org.apache.cassandra.locator;
 
-import java.util.*;
+import com.google.common.collect.Iterables;
-
 import org.apache.cassandra.config.DatabaseDescriptor;
 
 public abstract class AbstractEndpointSnitch implements IEndpointSnitch
 {
-    public abstract int compareEndpoints(InetAddressAndPort target, InetAddressAndPort a1, InetAddressAndPort a2);
+    public abstract int compareEndpoints(InetAddressAndPort target, Replica r1, Replica r2);
 
     /**
      * Sorts the <tt>Collection</tt> of node addresses by proximity to the given address
@@ -31,27 +30,9 @@ public abstract class AbstractEndpointSnitch implements IEndpointSnitch
      * @param unsortedAddress the nodes to sort
      * @return a new sorted <tt>List</tt>
      */
-    public List<InetAddressAndPort> getSortedListByProximity(InetAddressAndPort address, Collection<InetAddressAndPort> unsortedAddress)
+    public <C extends ReplicaCollection<? extends C>> C sortedByProximity(final InetAddressAndPort address, C unsortedAddress)
     {
-        List<InetAddressAndPort> preferred = new ArrayList<>(unsortedAddress);
+        return unsortedAddress.sorted((r1, r2) -> compareEndpoints(address, r1, r2));
-        sortByProximity(address, preferred);
-        return preferred;
-    }
-
-    /**
-     * Sorts the <tt>List</tt> of node addresses, in-place, by proximity to the given address
-     * @param address the address to sort the proximity by
-     * @param addresses the nodes to sort
-     */
-    public void sortByProximity(final InetAddressAndPort address, List<InetAddressAndPort> addresses)
-    {
-        Collections.sort(addresses, new Comparator<InetAddressAndPort>()
-        {
-            public int compare(InetAddressAndPort a1, InetAddressAndPort a2)
-            {
-                return compareEndpoints(address, a1, a2);
-            }
-        });
     }
 
     public void gossiperStarting()
@@ -59,7 +40,7 @@ public abstract class AbstractEndpointSnitch implements IEndpointSnitch
         // noop by default
     }
 
-    public boolean isWorthMergingForRangeQuery(List<InetAddressAndPort> merged, List<InetAddressAndPort> l1, List<InetAddressAndPort> l2)
+    public boolean isWorthMergingForRangeQuery(ReplicaCollection<?> merged, ReplicaCollection<?> l1, ReplicaCollection<?> l2)
     {
         // Querying remote DC is likely to be an order of magnitude slower than
         // querying locally, so 2 queries to local nodes is likely to still be
@@ -70,14 +51,9 @@ public abstract class AbstractEndpointSnitch implements IEndpointSnitch
              : true;
     }
 
-    private boolean hasRemoteNode(List<InetAddressAndPort> l)
+    private boolean hasRemoteNode(ReplicaCollection<?> l)
     {
         String localDc = DatabaseDescriptor.getLocalDataCenter();
-        for (InetAddressAndPort ep : l)
+        return Iterables.any(l, replica -> !localDc.equals(getDatacenter(replica)));
-        {
-            if (!localDc.equals(getDatacenter(ep)))
-                return true;
-        }
-        return false;
     }
 }

src/java/org/apache/cassandra/locator/AbstractNetworkTopologySnitch.java

@@ -37,8 +37,11 @@ public abstract class AbstractNetworkTopologySnitch extends AbstractEndpointSnit
      */
     abstract public String getDatacenter(InetAddressAndPort endpoint);
 
-    public int compareEndpoints(InetAddressAndPort address, InetAddressAndPort a1, InetAddressAndPort a2)
+    @Override
+    public int compareEndpoints(InetAddressAndPort address, Replica r1, Replica r2)
     {
+        InetAddressAndPort a1 = r1.endpoint();
+        InetAddressAndPort a2 = r2.endpoint();
         if (address.equals(a1) && !address.equals(a2))
             return -1;
         if (address.equals(a2) && !address.equals(a1))

src/java/org/apache/cassandra/locator/AbstractReplicaCollection.java

@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.locator;
+
+import com.google.common.collect.Iterables;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.function.BiConsumer;
+import java.util.function.BinaryOperator;
+import java.util.function.Function;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+import java.util.stream.Collector;
+import java.util.stream.Stream;
+
+/**
+ * A collection like class for Replica objects. Since the Replica class contains inetaddress, range, and
+ * transient replication status, basic contains and remove methods can be ambiguous. Replicas forces you
+ * to be explicit about what you're checking the container for, or removing from it.
+ */
+public abstract class AbstractReplicaCollection<C extends AbstractReplicaCollection<C>> implements ReplicaCollection<C>
+{
+    protected static final List<Replica> EMPTY_LIST = new ArrayList<>(); // since immutable, can safely return this to avoid megamorphic callsites
+
+    public static <C extends ReplicaCollection<C>, B extends Builder<C, ?, B>> Collector<Replica, B, C> collector(Set<Collector.Characteristics> characteristics, Supplier<B> supplier)
+    {
+        return new Collector<Replica, B, C>()
+        {
+            private final BiConsumer<B, Replica> accumulator = Builder::add;
+            private final BinaryOperator<B> combiner = (a, b) -> { a.addAll(b.mutable); return a; };
+            private final Function<B, C> finisher = Builder::build;
+            public Supplier<B> supplier() { return supplier; }
+            public BiConsumer<B, Replica> accumulator() { return accumulator; }
+            public BinaryOperator<B> combiner() { return combiner; }
+            public Function<B, C> finisher() { return finisher; }
+            public Set<Characteristics> characteristics() { return characteristics; }
+        };
+    }
+
+    protected final List<Replica> list;
+    protected final boolean isSnapshot;
+    protected AbstractReplicaCollection(List<Replica> list, boolean isSnapshot)
+    {
+        this.list = list;
+        this.isSnapshot = isSnapshot;
+    }
+
+    // if subList == null, should return self (or a clone thereof)
+    protected abstract C snapshot(List<Replica> subList);
+    protected abstract C self();
+    /**
+     * construct a new Mutable of our own type, so that we can concatenate
+     * TODO: this isn't terribly pretty, but we need sometimes to select / merge two Endpoints of unknown type;
+     */
+    public abstract Mutable<C> newMutable(int initialCapacity);
+
+
+    public C snapshot()
+    {
+        return isSnapshot ? self()
+                          : snapshot(list.isEmpty() ? EMPTY_LIST
+                                                    : new ArrayList<>(list));
+    }
+
+    public final C subList(int start, int end)
+    {
+        List<Replica> subList;
+        if (isSnapshot)
+        {
+            if (start == 0 && end == size()) return self();
+            else if (start == end) subList = EMPTY_LIST;
+            else subList = list.subList(start, end);
+        }
+        else
+        {
+            if (start == end) subList = EMPTY_LIST;
+            else subList = new ArrayList<>(list.subList(start, end)); // TODO: we could take a subList here, but comodification checks stop us
+        }
+        return snapshot(subList);
+    }
+
+    public final C filter(Predicate<Replica> predicate)
+    {
+        return filter(predicate, Integer.MAX_VALUE);
+    }
+
+    public final C filter(Predicate<Replica> predicate, int limit)
+    {
+        if (isEmpty())
+            return snapshot();
+
+        List<Replica> copy = null;
+        int beginRun = -1, endRun = -1;
+        int i = 0;
+        for (; i < list.size() ; ++i)
+        {
+            Replica replica = list.get(i);
+            if (predicate.test(replica))
+            {
+                if (copy != null)
+                    copy.add(replica);
+                else if (beginRun < 0)
+                    beginRun = i;
+                else if (endRun > 0)
+                {
+                    copy = new ArrayList<>(Math.min(limit, (list.size() - i) + (endRun - beginRun)));
+                    for (int j = beginRun ; j < endRun ; ++j)
+                        copy.add(list.get(j));
+                    copy.add(list.get(i));
+                }
+                if (--limit == 0)
+                {
+                    ++i;
+                    break;
+                }
+            }
+            else if (beginRun >= 0 && endRun < 0)
+                endRun = i;
+        }
+
+        if (beginRun < 0)
+            beginRun = endRun = 0;
+        if (endRun < 0)
+            endRun = i;
+        if (copy == null)
+            return subList(beginRun, endRun);
+        return snapshot(copy);
+    }
+
+    public final class Select
+    {
+        private final List<Replica> result;
+        public Select(int expectedSize)
+        {
+            this.result = new ArrayList<>(expectedSize);
+        }
+
+        /**
+         * Add matching replica to the result; this predicate should be mutually exclusive with all prior predicates.
+         * Stop once we have targetSize replicas in total, including preceding calls
+         */
+        public Select add(Predicate<Replica> predicate, int targetSize)
+        {
+            assert !Iterables.any(result, predicate::test);
+            for (int i = 0 ; result.size() < targetSize && i < list.size() ; ++i)
+                if (predicate.test(list.get(i)))
+                    result.add(list.get(i));
+            return this;
+        }
+        public Select add(Predicate<Replica> predicate)
+        {
+            return add(predicate, Integer.MAX_VALUE);
+        }
+        public C get()
+        {
+            return snapshot(result);
+        }
+    }
+
+    /**
+     * An efficient method for selecting a subset of replica via a sequence of filters.
+     *
+     * Example: select().add(filter1).add(filter2, 3).get();
+     *
+     * @return a Select object
+     */
+    public final Select select()
+    {
+        return select(list.size());
+    }
+    public final Select select(int expectedSize)
+    {
+        return new Select(expectedSize);
+    }
+
+    public final C sorted(Comparator<Replica> comparator)
+    {
+        List<Replica> copy = new ArrayList<>(list);
+        copy.sort(comparator);
+        return snapshot(copy);
+    }
+
+    public final Replica get(int i)
+    {
+        return list.get(i);
+    }
+
+    public final int size()
+    {
+        return list.size();
+    }
+
+    public final boolean isEmpty()
+    {
+        return list.isEmpty();
+    }
+
+    public final Iterator<Replica> iterator()
+    {
+        return list.iterator();
+    }
+
+    public final Stream<Replica> stream() { return list.stream(); }
+
+    public final boolean equals(Object o)
+    {
+        if (this == o) return true;
+        if (!(o instanceof AbstractReplicaCollection<?>))
+        {
+            if (!(o instanceof ReplicaCollection<?>))
+                return false;
+
+            ReplicaCollection<?> that = (ReplicaCollection<?>) o;
+            return Iterables.elementsEqual(this, that);
+        }
+        AbstractReplicaCollection<?> that = (AbstractReplicaCollection<?>) o;
+        return Objects.equals(list, that.list);
+    }
+
+    public final int hashCode()
+    {
+        return list.hashCode();
+    }
+
+    @Override
+    public final String toString()
+    {
+        return list.toString();
+    }
+
+    static <C extends AbstractReplicaCollection<C>> C concat(C replicas, C extraReplicas, Mutable.Conflict ignoreConflicts)
+    {
+        if (extraReplicas.isEmpty())
+            return replicas;
+        if (replicas.isEmpty())
+            return extraReplicas;
+        Mutable<C> mutable = replicas.newMutable(replicas.size() + extraReplicas.size());
+        mutable.addAll(replicas);
+        mutable.addAll(extraReplicas, ignoreConflicts);
+        return mutable.asImmutableView();
+    }
+
+}

src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java

@@ -22,8 +22,8 @@ import java.lang.reflect.InvocationTargetException;
 import java.util.*;
 
 import com.google.common.annotations.VisibleForTesting;
-import com.google.common.collect.HashMultimap;
+import com.google.common.base.Preconditions;
-import com.google.common.collect.Multimap;
+import org.apache.cassandra.locator.ReplicaCollection.Mutable.Conflict;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -73,9 +73,9 @@ public abstract class AbstractReplicationStrategy
         // lazy-initialize keyspace itself since we don't create them until after the replication strategies
     }
 
-    private final Map<Token, ArrayList<InetAddressAndPort>> cachedEndpoints = new NonBlockingHashMap<Token, ArrayList<InetAddressAndPort>>();
+    private final Map<Token, EndpointsForRange> cachedReplicas = new NonBlockingHashMap<>();
 
-    public ArrayList<InetAddressAndPort> getCachedEndpoints(Token t)
+    public EndpointsForRange getCachedReplicas(Token t)
     {
         long lastVersion = tokenMetadata.getRingVersion();
 
@@ -86,13 +86,13 @@ public abstract class AbstractReplicationStrategy
                 if (lastVersion > lastInvalidatedVersion)
                 {
                     logger.trace("clearing cached endpoints");
-                    cachedEndpoints.clear();
+                    cachedReplicas.clear();
                     lastInvalidatedVersion = lastVersion;
                 }
             }
         }
 
-        return cachedEndpoints.get(t);
+        return cachedReplicas.get(t);
     }
 
     /**
@@ -102,64 +102,65 @@ public abstract class AbstractReplicationStrategy
      * @param searchPosition the position the natural endpoints are requested for
      * @return a copy of the natural endpoints for the given token
      */
-    public ArrayList<InetAddressAndPort> getNaturalEndpoints(RingPosition searchPosition)
+    public EndpointsForToken getNaturalReplicasForToken(RingPosition searchPosition)
+    {
+        return getNaturalReplicas(searchPosition).forToken(searchPosition.getToken());
+    }
+
+    public EndpointsForRange getNaturalReplicas(RingPosition searchPosition)
     {
         Token searchToken = searchPosition.getToken();
         Token keyToken = TokenMetadata.firstToken(tokenMetadata.sortedTokens(), searchToken);
-        ArrayList<InetAddressAndPort> endpoints = getCachedEndpoints(keyToken);
+        EndpointsForRange endpoints = getCachedReplicas(keyToken);
         if (endpoints == null)
         {
             TokenMetadata tm = tokenMetadata.cachedOnlyTokenMap();
             // if our cache got invalidated, it's possible there is a new token to account for too
             keyToken = TokenMetadata.firstToken(tm.sortedTokens(), searchToken);
-            endpoints = new ArrayList<InetAddressAndPort>(calculateNaturalEndpoints(searchToken, tm));
+            endpoints = calculateNaturalReplicas(searchToken, tm);
-            cachedEndpoints.put(keyToken, endpoints);
+            cachedReplicas.put(keyToken, endpoints);
         }
 
-        return new ArrayList<InetAddressAndPort>(endpoints);
+        return endpoints;
     }
 
     /**
      * calculate the natural endpoints for the given token
      *
-     * @see #getNaturalEndpoints(org.apache.cassandra.dht.RingPosition)
+     * @see #getNaturalReplicasForToken(org.apache.cassandra.dht.RingPosition)
      *
      * @param searchToken the token the natural endpoints are requested for
      * @return a copy of the natural endpoints for the given token
      */
-    public abstract List<InetAddressAndPort> calculateNaturalEndpoints(Token searchToken, TokenMetadata tokenMetadata);
+    public abstract EndpointsForRange calculateNaturalReplicas(Token searchToken, TokenMetadata tokenMetadata);
 
-    public <T> AbstractWriteResponseHandler<T> getWriteResponseHandler(Collection<InetAddressAndPort> naturalEndpoints,
+    public <T> AbstractWriteResponseHandler<T> getWriteResponseHandler(ReplicaLayout.ForToken replicaLayout,
-                                                                       Collection<InetAddressAndPort> pendingEndpoints,
-                                                                       ConsistencyLevel consistency_level,
                                                                        Runnable callback,
                                                                        WriteType writeType,
                                                                        long queryStartNanoTime)
     {
-        return getWriteResponseHandler(naturalEndpoints, pendingEndpoints, consistency_level, callback, writeType, queryStartNanoTime, DatabaseDescriptor.getIdealConsistencyLevel());
+        return getWriteResponseHandler(replicaLayout, callback, writeType, queryStartNanoTime, DatabaseDescriptor.getIdealConsistencyLevel());
     }
 
-    public <T> AbstractWriteResponseHandler<T> getWriteResponseHandler(Collection<InetAddressAndPort> naturalEndpoints,
+    public <T> AbstractWriteResponseHandler<T> getWriteResponseHandler(ReplicaLayout.ForToken replicaLayout,
-                                                                       Collection<InetAddressAndPort> pendingEndpoints,
-                                                                       ConsistencyLevel consistency_level,
                                                                        Runnable callback,
                                                                        WriteType writeType,
                                                                        long queryStartNanoTime,
                                                                        ConsistencyLevel idealConsistencyLevel)
     {
         AbstractWriteResponseHandler resultResponseHandler;
-        if (consistency_level.isDatacenterLocal())
+        if (replicaLayout.consistencyLevel.isDatacenterLocal())
         {
             // block for in this context will be localnodes block.
-            resultResponseHandler = new DatacenterWriteResponseHandler<T>(naturalEndpoints, pendingEndpoints, consistency_level, getKeyspace(), callback, writeType, queryStartNanoTime);
+            resultResponseHandler = new DatacenterWriteResponseHandler<T>(replicaLayout, callback, writeType, queryStartNanoTime);
         }
-        else if (consistency_level == ConsistencyLevel.EACH_QUORUM && (this instanceof NetworkTopologyStrategy))
+        else if (replicaLayout.consistencyLevel == ConsistencyLevel.EACH_QUORUM && (this instanceof NetworkTopologyStrategy))
         {
-            resultResponseHandler = new DatacenterSyncWriteResponseHandler<T>(naturalEndpoints, pendingEndpoints, consistency_level, getKeyspace(), callback, writeType, queryStartNanoTime);
+            resultResponseHandler = new DatacenterSyncWriteResponseHandler<T>(replicaLayout, callback, writeType, queryStartNanoTime);
         }
         else
         {
-            resultResponseHandler = new WriteResponseHandler<T>(naturalEndpoints, pendingEndpoints, consistency_level, getKeyspace(), callback, writeType, queryStartNanoTime);
+            resultResponseHandler = new WriteResponseHandler<T>(replicaLayout, callback, writeType, queryStartNanoTime);
         }
 
         //Check if tracking the ideal consistency level is configured
@@ -168,16 +169,14 @@ public abstract class AbstractReplicationStrategy
             //If ideal and requested are the same just use this handler to track the ideal consistency level
             //This is also used so that the ideal consistency level handler when constructed knows it is the ideal
             //one for tracking purposes
-            if (idealConsistencyLevel == consistency_level)
+            if (idealConsistencyLevel == replicaLayout.consistencyLevel)
             {
                 resultResponseHandler.setIdealCLResponseHandler(resultResponseHandler);
             }
             else
             {
                 //Construct a delegate response handler to use to track the ideal consistency level
-                AbstractWriteResponseHandler idealHandler = getWriteResponseHandler(naturalEndpoints,
+                AbstractWriteResponseHandler idealHandler = getWriteResponseHandler(replicaLayout.withConsistencyLevel(idealConsistencyLevel),
-                                                                                    pendingEndpoints,
-                                                                                    idealConsistencyLevel,
                                                                                     callback,
                                                                                     writeType,
                                                                                     queryStartNanoTime,
@@ -202,7 +201,12 @@ public abstract class AbstractReplicationStrategy
      *
      * @return the replication factor
      */
-    public abstract int getReplicationFactor();
+    public abstract ReplicationFactor getReplicationFactor();
+
+    public boolean hasTransientReplicas()
+    {
+        return getReplicationFactor().hasTransientReplicas();
+    }
 
     /*
      * NOTE: this is pretty inefficient. also the inverse (getRangeAddresses) below.
@@ -210,53 +214,81 @@ public abstract class AbstractReplicationStrategy
      * (fixing this would probably require merging tokenmetadata into replicationstrategy,
      * so we could cache/invalidate cleanly.)
      */
-    public Multimap<InetAddressAndPort, Range<Token>> getAddressRanges(TokenMetadata metadata)
+    public RangesByEndpoint getAddressReplicas(TokenMetadata metadata)
     {
-        Multimap<InetAddressAndPort, Range<Token>> map = HashMultimap.create();
+        RangesByEndpoint.Mutable map = new RangesByEndpoint.Mutable();
 
         for (Token token : metadata.sortedTokens())
         {
             Range<Token> range = metadata.getPrimaryRangeFor(token);
-            for (InetAddressAndPort ep : calculateNaturalEndpoints(token, metadata))
+            for (Replica replica : calculateNaturalReplicas(token, metadata))
             {
-                map.put(ep, range);
+                // LocalStrategy always returns (min, min] ranges for it's replicas, so we skip the check here
+                Preconditions.checkState(range.equals(replica.range()) || this instanceof LocalStrategy);
+                map.put(replica.endpoint(), replica);
             }
         }
 
-        return map;
+        return map.asImmutableView();
     }
 
-    public Multimap<Range<Token>, InetAddressAndPort> getRangeAddresses(TokenMetadata metadata)
+    public RangesAtEndpoint getAddressReplicas(TokenMetadata metadata, InetAddressAndPort endpoint)
     {
-        Multimap<Range<Token>, InetAddressAndPort> map = HashMultimap.create();
+        RangesAtEndpoint.Builder builder = RangesAtEndpoint.builder(endpoint);
+        for (Token token : metadata.sortedTokens())
+        {
+            Range<Token> range = metadata.getPrimaryRangeFor(token);
+            Replica replica = calculateNaturalReplicas(token, metadata)
+                    .byEndpoint().get(endpoint);
+            if (replica != null)
+            {
+                // LocalStrategy always returns (min, min] ranges for it's replicas, so we skip the check here
+                Preconditions.checkState(range.equals(replica.range()) || this instanceof LocalStrategy);
+                builder.add(replica, Conflict.DUPLICATE);
+            }
+        }
+        return builder.build();
+    }
+
+
+    public EndpointsByRange getRangeAddresses(TokenMetadata metadata)
+    {
+        EndpointsByRange.Mutable map = new EndpointsByRange.Mutable();
 
         for (Token token : metadata.sortedTokens())
         {
             Range<Token> range = metadata.getPrimaryRangeFor(token);
-            for (InetAddressAndPort ep : calculateNaturalEndpoints(token, metadata))
+            for (Replica replica : calculateNaturalReplicas(token, metadata))
             {
-                map.put(range, ep);
+                // LocalStrategy always returns (min, min] ranges for it's replicas, so we skip the check here
+                Preconditions.checkState(range.equals(replica.range()) || this instanceof LocalStrategy);
+                map.put(range, replica);
             }
         }
 
-        return map;
+        return map.asImmutableView();
     }
 
-    public Multimap<InetAddressAndPort, Range<Token>> getAddressRanges()
+    public RangesByEndpoint getAddressReplicas()
     {
-        return getAddressRanges(tokenMetadata.cloneOnlyTokenMap());
+        return getAddressReplicas(tokenMetadata.cloneOnlyTokenMap());
     }
 
-    public Collection<Range<Token>> getPendingAddressRanges(TokenMetadata metadata, Token pendingToken, InetAddressAndPort pendingAddress)
+    public RangesAtEndpoint getAddressReplicas(InetAddressAndPort endpoint)
     {
-        return getPendingAddressRanges(metadata, Arrays.asList(pendingToken), pendingAddress);
+        return getAddressReplicas(tokenMetadata.cloneOnlyTokenMap(), endpoint);
     }
 
-    public Collection<Range<Token>> getPendingAddressRanges(TokenMetadata metadata, Collection<Token> pendingTokens, InetAddressAndPort pendingAddress)
+    public RangesAtEndpoint getPendingAddressRanges(TokenMetadata metadata, Token pendingToken, InetAddressAndPort pendingAddress)
+    {
+        return getPendingAddressRanges(metadata, Collections.singleton(pendingToken), pendingAddress);
+    }
+
+    public RangesAtEndpoint getPendingAddressRanges(TokenMetadata metadata, Collection<Token> pendingTokens, InetAddressAndPort pendingAddress)
     {
         TokenMetadata temp = metadata.cloneOnlyTokenMap();
         temp.updateNormalTokens(pendingTokens, pendingAddress);
-        return getAddressRanges(temp).get(pendingAddress);
+        return getAddressReplicas(temp, pendingAddress);
     }
 
     public abstract void validateOptions() throws ConfigurationException;
@@ -329,6 +361,10 @@ public abstract class AbstractReplicationStrategy
         AbstractReplicationStrategy strategy = createInternal(keyspaceName, strategyClass, tokenMetadata, snitch, strategyOptions);
         strategy.validateExpectedOptions();
         strategy.validateOptions();
+        if (strategy.hasTransientReplicas() && !DatabaseDescriptor.isTransientReplicationEnabled())
+        {
+            throw new ConfigurationException("Transient replication is disabled. Enable in cassandra.yaml to use.");
+        }
     }
 
     public static Class<AbstractReplicationStrategy> getClass(String cls) throws ConfigurationException
@@ -344,21 +380,23 @@ public abstract class AbstractReplicationStrategy
 
     public boolean hasSameSettings(AbstractReplicationStrategy other)
     {
-        return getClass().equals(other.getClass()) && getReplicationFactor() == other.getReplicationFactor();
+        return getClass().equals(other.getClass()) && getReplicationFactor().equals(other.getReplicationFactor());
     }
 
-    protected void validateReplicationFactor(String rf) throws ConfigurationException
+    protected void validateReplicationFactor(String s) throws ConfigurationException
     {
         try
         {
-            if (Integer.parseInt(rf) < 0)
+            ReplicationFactor rf = ReplicationFactor.fromString(s);
+            if (rf.hasTransientReplicas())
             {
-                throw new ConfigurationException("Replication factor must be non-negative; found " + rf);
+                if (DatabaseDescriptor.getNumTokens() > 1)
+                    throw new ConfigurationException(String.format("Transient replication is not supported with vnodes yet"));
             }
         }
-        catch (NumberFormatException e2)
+        catch (IllegalArgumentException e)
         {
-            throw new ConfigurationException("Replication factor must be numeric; found " + rf);
+            throw new ConfigurationException(e.getMessage());
         }
     }
 

src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java

@@ -42,7 +42,6 @@ import org.apache.cassandra.net.MessagingService;
 import org.apache.cassandra.service.StorageService;
 import org.apache.cassandra.utils.FBUtilities;
 
-
 /**
  * A dynamic snitch that sorts endpoints by latency with an adapted phi failure detector
  */
@@ -185,55 +184,38 @@ public class DynamicEndpointSnitch extends AbstractEndpointSnitch implements ILa
         return subsnitch.getDatacenter(endpoint);
     }
 
-    public List<InetAddressAndPort> getSortedListByProximity(final InetAddressAndPort address, Collection<InetAddressAndPort> addresses)
-    {
-        List<InetAddressAndPort> list = new ArrayList<>(addresses);
-        sortByProximity(address, list);
-        return list;
-    }
-
     @Override
-    public void sortByProximity(final InetAddressAndPort address, List<InetAddressAndPort> addresses)
+    public <C extends ReplicaCollection<? extends C>> C sortedByProximity(final InetAddressAndPort address, C unsortedAddresses)
     {
         assert address.equals(FBUtilities.getBroadcastAddressAndPort()); // we only know about ourself
-        if (dynamicBadnessThreshold == 0)
+        return dynamicBadnessThreshold == 0
-        {
+                ? sortedByProximityWithScore(address, unsortedAddresses)
-            sortByProximityWithScore(address, addresses);
+                : sortedByProximityWithBadness(address, unsortedAddresses);
-        }
-        else
-        {
-            sortByProximityWithBadness(address, addresses);
-        }
     }
 
-    private void sortByProximityWithScore(final InetAddressAndPort address, List<InetAddressAndPort> addresses)
+    private <C extends ReplicaCollection<? extends C>> C sortedByProximityWithScore(final InetAddressAndPort address, C unsortedAddresses)
     {
         // Scores can change concurrently from a call to this method. But Collections.sort() expects
         // its comparator to be "stable", that is 2 endpoint should compare the same way for the duration
         // of the sort() call. As we copy the scores map on write, it is thus enough to alias the current
         // version of it during this call.
         final HashMap<InetAddressAndPort, Double> scores = this.scores;
-        Collections.sort(addresses, new Comparator<InetAddressAndPort>()
+        return unsortedAddresses.sorted((r1, r2) -> compareEndpoints(address, r1, r2, scores));
-        {
-            public int compare(InetAddressAndPort a1, InetAddressAndPort a2)
-            {
-                return compareEndpoints(address, a1, a2, scores);
-            }
-        });
     }
 
-    private void sortByProximityWithBadness(final InetAddressAndPort address, List<InetAddressAndPort> addresses)
+    private <C extends ReplicaCollection<? extends C>> C sortedByProximityWithBadness(final InetAddressAndPort address, C replicas)
     {
-        if (addresses.size() < 2)
+        if (replicas.size() < 2)
-            return;
+            return replicas;
 
-        subsnitch.sortByProximity(address, addresses);
+        // TODO: avoid copy
+        replicas = subsnitch.sortedByProximity(address, replicas);
         HashMap<InetAddressAndPort, Double> scores = this.scores; // Make sure the score don't change in the middle of the loop below
                                                            // (which wouldn't really matter here but its cleaner that way).
-        ArrayList<Double> subsnitchOrderedScores = new ArrayList<>(addresses.size());
+        ArrayList<Double> subsnitchOrderedScores = new ArrayList<>(replicas.size());
-        for (InetAddressAndPort inet : addresses)
+        for (Replica replica : replicas)
         {
-            Double score = scores.get(inet);
+            Double score = scores.get(replica.endpoint());
             if (score == null)
                 score = 0.0;
             subsnitchOrderedScores.add(score);
@@ -250,17 +232,18 @@ public class DynamicEndpointSnitch extends AbstractEndpointSnitch implements ILa
         {
             if (subsnitchScore > (sortedScoreIterator.next() * (1.0 + dynamicBadnessThreshold)))
             {
-                sortByProximityWithScore(address, addresses);
+                return sortedByProximityWithScore(address, replicas);
-                return;
             }
         }
+
+        return replicas;
     }
 
     // Compare endpoints given an immutable snapshot of the scores
-    private int compareEndpoints(InetAddressAndPort target, InetAddressAndPort a1, InetAddressAndPort a2, Map<InetAddressAndPort, Double> scores)
+    private int compareEndpoints(InetAddressAndPort target, Replica a1, Replica a2, Map<InetAddressAndPort, Double> scores)
     {
-        Double scored1 = scores.get(a1);
+        Double scored1 = scores.get(a1.endpoint());
-        Double scored2 = scores.get(a2);
+        Double scored2 = scores.get(a2.endpoint());
         
         if (scored1 == null)
         {
@@ -280,7 +263,7 @@ public class DynamicEndpointSnitch extends AbstractEndpointSnitch implements ILa
             return 1;
     }
 
-    public int compareEndpoints(InetAddressAndPort target, InetAddressAndPort a1, InetAddressAndPort a2)
+    public int compareEndpoints(InetAddressAndPort target, Replica a1, Replica a2)
     {
         // That function is fundamentally unsafe because the scores can change at any time and so the result of that
         // method is not stable for identical arguments. This is why we don't rely on super.sortByProximity() in
@@ -414,7 +397,7 @@ public class DynamicEndpointSnitch extends AbstractEndpointSnitch implements ILa
         return getSeverity(FBUtilities.getBroadcastAddressAndPort());
     }
 
-    public boolean isWorthMergingForRangeQuery(List<InetAddressAndPort> merged, List<InetAddressAndPort> l1, List<InetAddressAndPort> l2)
+    public boolean isWorthMergingForRangeQuery(ReplicaCollection<?> merged, ReplicaCollection<?> l1, ReplicaCollection<?> l2)
     {
         if (!subsnitch.isWorthMergingForRangeQuery(merged, l1, l2))
             return false;
@@ -434,12 +417,12 @@ public class DynamicEndpointSnitch extends AbstractEndpointSnitch implements ILa
     }
 
     // Return the max score for the endpoint in the provided list, or -1.0 if no node have a score.
-    private double maxScore(List<InetAddressAndPort> endpoints)
+    private double maxScore(ReplicaCollection<?> endpoints)
     {
         double maxScore = -1.0;
-        for (InetAddressAndPort endpoint : endpoints)
+        for (Replica replica : endpoints)
         {
-            Double score = scores.get(endpoint);
+            Double score = scores.get(replica.endpoint());
             if (score == null)
                 continue;
 

src/java/org/apache/cassandra/locator/Ec2Snitch.java

@@ -68,7 +68,7 @@ public class Ec2Snitch extends AbstractNetworkTopologySnitch
     {
         String az = awsApiCall(ZONE_NAME_QUERY_URL);
 
-        // if using the full naming scheme, region name is created by removing letters from the 
+        // if using the full naming scheme, region name is created by removing letters from the
         // end of the availability zone and zone is the full zone name
         usingLegacyNaming = isUsingLegacyNaming(props);
         String region;

src/java/org/apache/cassandra/locator/Endpoints.java

@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.locator;
+
+import org.apache.cassandra.locator.ReplicaCollection.Mutable.Conflict;
+import org.apache.cassandra.utils.FBUtilities;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+
+public abstract class Endpoints<E extends Endpoints<E>> extends AbstractReplicaCollection<E>
+{
+    static final Map<InetAddressAndPort, Replica> EMPTY_MAP = Collections.unmodifiableMap(new LinkedHashMap<>());
+
+    volatile Map<InetAddressAndPort, Replica> byEndpoint;
+
+    Endpoints(List<Replica> list, boolean isSnapshot, Map<InetAddressAndPort, Replica> byEndpoint)
+    {
+        super(list, isSnapshot);
+        this.byEndpoint = byEndpoint;
+    }
+
+    @Override
+    public Set<InetAddressAndPort> endpoints()
+    {
+        return byEndpoint().keySet();
+    }
+
+    public Map<InetAddressAndPort, Replica> byEndpoint()
+    {
+        Map<InetAddressAndPort, Replica> map = byEndpoint;
+        if (map == null)
+            byEndpoint = map = buildByEndpoint(list);
+        return map;
+    }
+
+    public boolean contains(InetAddressAndPort endpoint, boolean isFull)
+    {
+        Replica replica = byEndpoint().get(endpoint);
+        return replica != null && replica.isFull() == isFull;
+    }
+
+    @Override
+    public boolean contains(Replica replica)
+    {
+        return replica != null
+                && Objects.equals(
+                        byEndpoint().get(replica.endpoint()),
+                        replica);
+    }
+
+    private static Map<InetAddressAndPort, Replica> buildByEndpoint(List<Replica> list)
+    {
+        // TODO: implement a delegating map that uses our superclass' list, and is immutable
+        Map<InetAddressAndPort, Replica> byEndpoint = new LinkedHashMap<>(list.size());
+        for (Replica replica : list)
+        {
+            Replica prev = byEndpoint.put(replica.endpoint(), replica);
+            assert prev == null : "duplicate endpoint in EndpointsForRange: " + prev + " and " + replica;
+        }
+
+        return Collections.unmodifiableMap(byEndpoint);
+    }
+
+    public E withoutSelf()
+    {
+        InetAddressAndPort self = FBUtilities.getBroadcastAddressAndPort();
+        return filter(r -> !self.equals(r.endpoint()));
+    }
+
+    public E without(Set<InetAddressAndPort> remove)
+    {
+        return filter(r -> !remove.contains(r.endpoint()));
+    }
+
+    public E keep(Set<InetAddressAndPort> keep)
+    {
+        return filter(r -> keep.contains(r.endpoint()));
+    }
+
+    public E keep(Iterable<InetAddressAndPort> endpoints)
+    {
+        ReplicaCollection.Mutable<E> copy = newMutable(
+                endpoints instanceof Collection<?>
+                        ? ((Collection<InetAddressAndPort>) endpoints).size()
+                        : size()
+        );
+        Map<InetAddressAndPort, Replica> byEndpoint = byEndpoint();
+        for (InetAddressAndPort endpoint : endpoints)
+        {
+            Replica keep = byEndpoint.get(endpoint);
+            if (keep == null)
+                continue;
+            copy.add(keep, ReplicaCollection.Mutable.Conflict.DUPLICATE);
+        }
+        return copy.asSnapshot();
+    }
+
+    /**
+     * Care must be taken to ensure no conflicting ranges occur in pending and natural.
+     * Conflicts can occur for two reasons:
+     *   1) due to lack of isolation when reading pending/natural
+     *   2) because a movement that changes the type of replication from transient to full must be handled
+     *      differently for reads and writes (with the reader treating it as transient, and writer as full)
+     *
+     * The method haveConflicts() below, and resolveConflictsInX, are used to detect and resolve any issues
+     */
+    public static <E extends Endpoints<E>> E concat(E natural, E pending)
+    {
+        return AbstractReplicaCollection.concat(natural, pending, Conflict.NONE);
+    }
+
+    public static <E extends Endpoints<E>> boolean haveConflicts(E natural, E pending)
+    {
+        Set<InetAddressAndPort> naturalEndpoints = natural.endpoints();
+        for (InetAddressAndPort pendingEndpoint : pending.endpoints())
+        {
+            if (naturalEndpoints.contains(pendingEndpoint))
+                return true;
+        }
+        return false;
+    }
+
+    // must apply first
+    public static <E extends Endpoints<E>> E resolveConflictsInNatural(E natural, E pending)
+    {
+        return natural.filter(r -> !r.isTransient() || !pending.contains(r.endpoint(), true));
+    }
+
+    // must apply second
+    public static <E extends Endpoints<E>> E resolveConflictsInPending(E natural, E pending)
+    {
+        return pending.without(natural.endpoints());
+    }
+
+}

src/java/org/apache/cassandra/locator/EndpointsByRange.java

@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.locator;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Maps;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.locator.ReplicaCollection.Mutable.Conflict;
+
+import java.util.Collections;
+import java.util.Map;
+
+public class EndpointsByRange extends ReplicaMultimap<Range<Token>, EndpointsForRange>
+{
+    public EndpointsByRange(Map<Range<Token>, EndpointsForRange> map)
+    {
+        super(map);
+    }
+
+    public EndpointsForRange get(Range<Token> range)
+    {
+        Preconditions.checkNotNull(range);
+        return map.getOrDefault(range, EndpointsForRange.empty(range));
+    }
+
+    public static class Mutable extends ReplicaMultimap.Mutable<Range<Token>, EndpointsForRange.Mutable>
+    {
+        @Override
+        protected EndpointsForRange.Mutable newMutable(Range<Token> range)
+        {
+            return new EndpointsForRange.Mutable(range);
+        }
+
+        // TODO: consider all ignoreDuplicates cases
+        public void putAll(Range<Token> range, EndpointsForRange replicas, Conflict ignoreConflicts)
+        {
+            get(range).addAll(replicas, ignoreConflicts);
+        }
+
+        public EndpointsByRange asImmutableView()
+        {
+            return new EndpointsByRange(Collections.unmodifiableMap(Maps.transformValues(map, EndpointsForRange.Mutable::asImmutableView)));
+        }
+    }
+
+}

src/java/org/apache/cassandra/locator/EndpointsByReplica.java

@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.locator;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Maps;
+import org.apache.cassandra.locator.ReplicaCollection.Mutable.Conflict;
+
+import java.util.Collections;
+import java.util.Map;
+
+public class EndpointsByReplica extends ReplicaMultimap<Replica, EndpointsForRange>
+{
+    public EndpointsByReplica(Map<Replica, EndpointsForRange> map)
+    {
+        super(map);
+    }
+
+    public EndpointsForRange get(Replica range)
+    {
+        Preconditions.checkNotNull(range);
+        return map.getOrDefault(range, EndpointsForRange.empty(range.range()));
+    }
+
+    public static class Mutable extends ReplicaMultimap.Mutable<Replica, EndpointsForRange.Mutable>
+    {
+        @Override
+        protected EndpointsForRange.Mutable newMutable(Replica replica)
+        {
+            return new EndpointsForRange.Mutable(replica.range());
+        }
+
+        // TODO: consider all ignoreDuplicates cases
+        public void putAll(Replica range, EndpointsForRange replicas, Conflict ignoreConflicts)
+        {
+            map.computeIfAbsent(range, r -> newMutable(r)).addAll(replicas, ignoreConflicts);
+        }
+
+        public EndpointsByReplica asImmutableView()
+        {
+            return new EndpointsByReplica(Collections.unmodifiableMap(Maps.transformValues(map, EndpointsForRange.Mutable::asImmutableView)));
+        }
+    }
+
+}

src/java/org/apache/cassandra/locator/EndpointsForRange.java

@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.locator;
+
+import com.google.common.base.Preconditions;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import static com.google.common.collect.Iterables.all;
+
+/**
+ * A ReplicaCollection where all Replica are required to cover a range that fully contains the range() defined in the builder().
+ * Endpoints are guaranteed to be unique; on construction, this is enforced unless optionally silenced (in which case
+ * only the first occurrence makes the cut).
+ */
+public class EndpointsForRange extends Endpoints<EndpointsForRange>
+{
+    private final Range<Token> range;
+    private EndpointsForRange(Range<Token> range, List<Replica> list, boolean isSnapshot)
+    {
+        this(range, list, isSnapshot, null);
+    }
+    private EndpointsForRange(Range<Token> range, List<Replica> list, boolean isSnapshot, Map<InetAddressAndPort, Replica> byEndpoint)
+    {
+        super(list, isSnapshot, byEndpoint);
+        this.range = range;
+        assert range != null;
+    }
+
+    public Range<Token> range()
+    {
+        return range;
+    }
+
+    @Override
+    public Mutable newMutable(int initialCapacity)
+    {
+        return new Mutable(range, initialCapacity);
+    }
+
+    public EndpointsForToken forToken(Token token)
+    {
+        if (!range.contains(token))
+            throw new IllegalArgumentException(token + " is not contained within " + range);
+        return new EndpointsForToken(token, list, isSnapshot, byEndpoint);
+    }
+
+    @Override
+    public EndpointsForRange self()
+    {
+        return this;
+    }
+
+    @Override
+    protected EndpointsForRange snapshot(List<Replica> snapshot)
+    {
+        if (snapshot.isEmpty()) return empty(range);
+        return new EndpointsForRange(range, snapshot, true);
+    }
+
+    public static class Mutable extends EndpointsForRange implements ReplicaCollection.Mutable<EndpointsForRange>
+    {
+        boolean hasSnapshot;
+        public Mutable(Range<Token> range) { this(range, 0); }
+        public Mutable(Range<Token> range, int capacity) { super(range, new ArrayList<>(capacity), false, new LinkedHashMap<>()); }
+
+        public void add(Replica replica, Conflict ignoreConflict)
+        {
+            if (hasSnapshot) throw new IllegalStateException();
+            Preconditions.checkNotNull(replica);
+            if (!replica.range().contains(super.range))
+                throw new IllegalArgumentException("Replica " + replica + " does not contain " + super.range);
+
+            Replica prev = super.byEndpoint.put(replica.endpoint(), replica);
+            if (prev != null)
+            {
+                super.byEndpoint.put(replica.endpoint(), prev); // restore prev
+                switch (ignoreConflict)
+                {
+                    case DUPLICATE:
+                        if (prev.equals(replica))
+                            break;
+                    case NONE:
+                        throw new IllegalArgumentException("Conflicting replica added (expected unique endpoints): " + replica + "; existing: " + prev);
+                    case ALL:
+                }
+                return;
+            }
+
+            list.add(replica);
+        }
+
+        @Override
+        public Map<InetAddressAndPort, Replica> byEndpoint()
+        {
+            // our internal map is modifiable, but it is unsafe to modify the map externally
+            // it would be possible to implement a safe modifiable map, but it is probably not valuable
+            return Collections.unmodifiableMap(super.byEndpoint());
+        }
+
+        private EndpointsForRange get(boolean isSnapshot)
+        {
+            return new EndpointsForRange(super.range, super.list, isSnapshot, Collections.unmodifiableMap(super.byEndpoint));
+        }
+
+        public EndpointsForRange asImmutableView()
+        {
+            return get(false);
+        }
+
+        public EndpointsForRange asSnapshot()
+        {
+            hasSnapshot = true;
+            return get(true);
+        }
+    }
+
+    public static class Builder extends ReplicaCollection.Builder<EndpointsForRange, Mutable, EndpointsForRange.Builder>
+    {
+        public Builder(Range<Token> range) { this(range, 0); }
+        public Builder(Range<Token> range, int capacity) { super (new Mutable(range, capacity)); }
+        public boolean containsEndpoint(InetAddressAndPort endpoint)
+        {
+            return mutable.asImmutableView().byEndpoint.containsKey(endpoint);
+        }
+    }
+
+    public static Builder builder(Range<Token> range)
+    {
+        return new Builder(range);
+    }
+    public static Builder builder(Range<Token> range, int capacity)
+    {
+        return new Builder(range, capacity);
+    }
+
+    public static EndpointsForRange empty(Range<Token> range)
+    {
+        return new EndpointsForRange(range, EMPTY_LIST, true, EMPTY_MAP);
+    }
+
+    public static EndpointsForRange of(Replica replica)
+    {
+        // we only use ArrayList or ArrayList.SubList, to ensure callsites are bimorphic
+        ArrayList<Replica> one = new ArrayList<>(1);
+        one.add(replica);
+        // we can safely use singletonMap, as we only otherwise use LinkedHashMap
+        return new EndpointsForRange(replica.range(), one, true, Collections.unmodifiableMap(Collections.singletonMap(replica.endpoint(), replica)));
+    }
+
+    public static EndpointsForRange of(Replica ... replicas)
+    {
+        return copyOf(Arrays.asList(replicas));
+    }
+
+    public static EndpointsForRange copyOf(Collection<Replica> replicas)
+    {
+        if (replicas.isEmpty())
+            throw new IllegalArgumentException("Collection must be non-empty to copy");
+        Range<Token> range = replicas.iterator().next().range();
+        assert all(replicas, r -> range.equals(r.range()));
+        return builder(range, replicas.size()).addAll(replicas).build();
+    }
+}

src/java/org/apache/cassandra/locator/EndpointsForToken.java

@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.locator;
+
+import com.google.common.base.Preconditions;
+import org.apache.cassandra.dht.Token;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * A ReplicaCollection where all Replica are required to cover a range that fully contains the token() defined in the builder().
+ * Endpoints are guaranteed to be unique; on construction, this is enforced unless optionally silenced (in which case
+ * only the first occurrence makes the cut).
+ */
+public class EndpointsForToken extends Endpoints<EndpointsForToken>
+{
+    private final Token token;
+    private EndpointsForToken(Token token, List<Replica> list, boolean isSnapshot)
+    {
+        this(token, list, isSnapshot, null);
+    }
+
+    EndpointsForToken(Token token, List<Replica> list, boolean isSnapshot, Map<InetAddressAndPort, Replica> byEndpoint)
+    {
+        super(list, isSnapshot, byEndpoint);
+        this.token = token;
+        assert token != null;
+    }
+
+    public Token token()
+    {
+        return token;
+    }
+
+    @Override
+    public Mutable newMutable(int initialCapacity)
+    {
+        return new Mutable(token, initialCapacity);
+    }
+
+    @Override
+    public EndpointsForToken self()
+    {
+        return this;
+    }
+
+    @Override
+    protected EndpointsForToken snapshot(List<Replica> subList)
+    {
+        if (subList.isEmpty()) return empty(token);
+        return new EndpointsForToken(token, subList, true);
+    }
+
+    public static class Mutable extends EndpointsForToken implements ReplicaCollection.Mutable<EndpointsForToken>
+    {
+        boolean hasSnapshot;
+        public Mutable(Token token) { this(token, 0); }
+        public Mutable(Token token, int capacity) { super(token, new ArrayList<>(capacity), false, new LinkedHashMap<>()); }
+
+        public void add(Replica replica, Conflict ignoreConflict)
+        {
+            if (hasSnapshot) throw new IllegalStateException();
+            Preconditions.checkNotNull(replica);
+            if (!replica.range().contains(super.token))
+                throw new IllegalArgumentException("Replica " + replica + " does not contain " + super.token);
+
+            Replica prev = super.byEndpoint.put(replica.endpoint(), replica);
+            if (prev != null)
+            {
+                super.byEndpoint.put(replica.endpoint(), prev); // restore prev
+                switch (ignoreConflict)
+                {
+                    case DUPLICATE:
+                        if (prev.equals(replica))
+                            break;
+                    case NONE:
+                        throw new IllegalArgumentException("Conflicting replica added (expected unique endpoints): " + replica + "; existing: " + prev);
+                    case ALL:
+                }
+                return;
+            }
+
+            list.add(replica);
+        }
+
+        @Override
+        public Map<InetAddressAndPort, Replica> byEndpoint()
+        {
+            // our internal map is modifiable, but it is unsafe to modify the map externally
+            // it would be possible to implement a safe modifiable map, but it is probably not valuable
+            return Collections.unmodifiableMap(super.byEndpoint());
+        }
+
+        private EndpointsForToken get(boolean isSnapshot)
+        {
+            return new EndpointsForToken(super.token, super.list, isSnapshot, Collections.unmodifiableMap(super.byEndpoint));
+        }
+
+        public EndpointsForToken asImmutableView()
+        {
+            return get(false);
+        }
+
+        public EndpointsForToken asSnapshot()
+        {
+            hasSnapshot = true;
+            return get(true);
+        }
+    }
+
+    public static class Builder extends ReplicaCollection.Builder<EndpointsForToken, Mutable, EndpointsForToken.Builder>
+    {
+        public Builder(Token token) { this(token, 0); }
+        public Builder(Token token, int capacity) { super (new Mutable(token, capacity)); }
+    }
+
+    public static Builder builder(Token token)
+    {
+        return new Builder(token);
+    }
+    public static Builder builder(Token token, int capacity)
+    {
+        return new Builder(token, capacity);
+    }
+
+    public static EndpointsForToken empty(Token token)
+    {
+        return new EndpointsForToken(token, EMPTY_LIST, true, EMPTY_MAP);
+    }
+
+    public static EndpointsForToken of(Token token, Replica replica)
+    {
+        // we only use ArrayList or ArrayList.SubList, to ensure callsites are bimorphic
+        ArrayList<Replica> one = new ArrayList<>(1);
+        one.add(replica);
+        // we can safely use singletonMap, as we only otherwise use LinkedHashMap
+        return new EndpointsForToken(token, one, true, Collections.unmodifiableMap(Collections.singletonMap(replica.endpoint(), replica)));
+    }
+
+    public static EndpointsForToken of(Token token, Replica ... replicas)
+    {
+        return copyOf(token, Arrays.asList(replicas));
+    }
+
+    public static EndpointsForToken copyOf(Token token, Collection<Replica> replicas)
+    {
+        if (replicas.isEmpty()) return empty(token);
+        return builder(token, replicas.size()).addAll(replicas).build();
+    }
+}

src/java/org/apache/cassandra/locator/IEndpointSnitch.java

@@ -17,8 +17,6 @@
  */
 package org.apache.cassandra.locator;
 
-import java.util.Collection;
-import java.util.List;
 import java.util.Set;
 
 /**
@@ -39,20 +37,20 @@ public interface IEndpointSnitch
      */
     public String getDatacenter(InetAddressAndPort endpoint);
 
+    default public String getDatacenter(Replica replica)
+    {
+        return getDatacenter(replica.endpoint());
+    }
+
     /**
      * returns a new <tt>List</tt> sorted by proximity to the given endpoint
      */
-    public List<InetAddressAndPort> getSortedListByProximity(InetAddressAndPort address, Collection<InetAddressAndPort> unsortedAddress);
+    public <C extends ReplicaCollection<? extends C>> C sortedByProximity(final InetAddressAndPort address, C addresses);
-
-    /**
-     * This method will sort the <tt>List</tt> by proximity to the given address.
-     */
-    public void sortByProximity(InetAddressAndPort address, List<InetAddressAndPort> addresses);
 
     /**
      * compares two endpoints in relation to the target endpoint, returning as Comparator.compare would
      */
-    public int compareEndpoints(InetAddressAndPort target, InetAddressAndPort a1, InetAddressAndPort a2);
+    public int compareEndpoints(InetAddressAndPort target, Replica r1, Replica r2);
 
     /**
      * called after Gossiper instance exists immediately before it starts gossiping
@@ -63,7 +61,7 @@ public interface IEndpointSnitch
      * Returns whether for a range query doing a query against merged is likely
      * to be faster than 2 sequential queries, one against l1 followed by one against l2.
      */
-    public boolean isWorthMergingForRangeQuery(List<InetAddressAndPort> merged, List<InetAddressAndPort> l1, List<InetAddressAndPort> l2);
+    public boolean isWorthMergingForRangeQuery(ReplicaCollection<?> merged, ReplicaCollection<?> l1, ReplicaCollection<?> l2);
 
     /**
      * Determine if the datacenter or rack values in the current node's snitch conflict with those passed in parameters.