natas_hw
/
cassandra
mirror of https://github.com/apache/cassandra
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Snitch re-implementation 

Deprecate IEndpointSnitch entirely, to be replaced with new interfaces:
* Locator for endpoint -> location mapping
* InitialLocationProvider to supply the DC & rack for registration
* NodeProximity for sorting endpoints and replicas at query time

For migration/upgrade/deprecation, nodes can still be configured with
an IEndpointSnitch implementation via endpoint_snitch in config, but
we hide this with a facade and only present the new interfaces.

Patch by Sam Tunnicliffe and Marcus Eriksson; reviewed by Sam
Tunnicliffe and Marcus Eriksson for CASSANDRA-19488

Co-authored-by: Marcus Eriksson <marcuse@apache.org>
Co-authored-by: Sam Tunnicliffe <samt@apache.org>
This commit is contained in:
Sam Tunnicliffe 2024-06-18 17:29:46 +01:00
parent 2ff41551a6
commit 48dcf5e092
180 changed files with 3897 additions and 2284 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

43
NEWS.txt
View File

@ -108,7 +108,6 @@ New features
the respective configuration sections and by commenting out `configure_jmx` function call in cassandra-env.sh.
Enabling both ways of configuring JMX will result in a node failing to start.
Upgrading
---------
[The following is a placeholder, to be revised asap]
@ -119,12 +118,12 @@ Upgrading
2. The Cluster Metadata Service (CMS) has been enabled
The first step is just a regular upgrade, there are no changes to external APIs, SSTable formats to consider.
Step 2 requires an operator to run a new nodetool subcommand, intializecms, on one node in the cluster.
- > nodetool initializecms
Doing so creates the CMS with the local node as its only member. The initializecms command cannot be executed until
- > nodetool cms initialize
Doing so creates the CMS with the local node as its only member. The initialize command cannot be executed until
step 1 is complete and all nodes are running 5.1 as migrating metadata management over to the new TCM system
requires all nodes to be in agreement over the current state of the cluster. Essentially this means agreement on
schema and topology. Once the upgrade has started, but before initializecms is run, metadata-changing operations are
not permitted and if attempted from an upgraded node will be rejected.
schema and topology. Once the upgrade has started, but before the initialize command is run, metadata-changing
operations are not permitted and if attempted from an upgraded node will be rejected.
Prohibited operations include:
- schema changes
- node replacement
@ -136,22 +135,22 @@ Upgrading
a previous version, though this is planned before release. Any automation which can trigger these operations should
be disabled for the cluster prior to starting the upgrade.
Should any of the prohibited operations be executed (i.e. on a node that is still running a pre-5.1 version) before
the CMS migration, nodes which are DOWN or which have been upgraded will not process the metadata changes. However,
nodes still UP and running the old version will. This will eventually cause the migration to fail, as the cluster
will not be in agreement.
- > nodetool initializecms
the CMS intialization, nodes which are DOWN or which have been upgraded will not process the metadata changes.
However, nodes still UP and running the old version will. This will eventually cause the migration to fail, as the
cluster will not be in agreement.
- > nodetool cms initialize
Got mismatching cluster metadatas from [/x.x.x.x:7000] aborting migration
See 'nodetool help' or 'nodetool help <command>'.
If the initializecms command fails, it will indicate which nodes’ current metadata does not agree with the node
If the cms initialize command fails, it will indicate which nodes’ current metadata does not agree with the node
where the command was executed. To mitigate this situation, bring any mismatching nodes DOWN and rerun the
initializecms command with the additional —ignore flag.
- nodetool intializecms -ignore x.x.x.x
cms initialize command with the additional —ignore flag.
- nodetool cms intialize -ignore x.x.x.x
Once the command has run successfully the ignored nodes can be restarted but any metadata changes that they
accepted/and or applied whilst the cluster was in mixed mode will be lost. We plan to improve this before beta, but
in the meantime operators should ensure no schema or membership/ownership changes are performed during the upgrade.
Although the restrictions on metadata-changing operations will be removed as soon as the initial CMS migration is
complete, at that point the CMS will only contain a single member, which is not suitable for real clusters. To
modify the membership of the CMS a second nodetool subcommand, reconfigurecms, should be run. This command allows
modify the membership of the CMS a second nodetool subcommand, cms reconfigure, should be run. This command allows
the number of required number of CMS members to be specified for each datacenter. Consensus for metadata operations
by the CMS is obtained via Paxos, operating at SERIAL/QUORUM consistency, so the minimum safe size for the CMS is 3
nodes. In multi-DC deployments, the CMS members may be distributed across DCs to ensure resilience in the case of a
@ -173,7 +172,23 @@ Upgrading
Deprecation
-----------
- `use_deterministic_table_id` is no longer supported and should be removed from cassandra.yaml. Table IDs may still be supplied explicitly on CREATE.
- IEndpointSnitch has been deprecated as ClusterMetadata is now the source of truth regarding topology information.
The responsibilities of the snitch have been broken out to a handful of new classes:
* o.a.c.locator.Locator provides datacenter and rack info for endpoints. This is not configurable as topology is
always sourced from ClusterMetadata.
* o.a.c.locator.InitialLocationProvider supplies the local datacenter and rack for a new node joining a cluster
for the first time. This will be used exactly once to register the Location of the node in ClusterMetadata and
is configurable using the `initial_location_provider` yaml setting.
* o.a.c.locator.NodeProximity handles sorting and ranking of replica lists. It is configurable via the
`node_proximity` yaml setting.
* o.a.c.locator.NodeAddressConfig exists mainly to support the functionality of ReconnectableSnitchHelper and
Ec2MultiRegionSnitch which dynamically configures the broadcast address based on cloud metadata. Optionally
configurable using the `address_config` yaml setting.
For migration and to allow us to deprecate snitches in a controlled way, use of endpoint_snitch in yaml remains
fully supported and all existing IEndpointSnitch implementations should continue to work seamlessly and no action
is required by operators at upgrade time. The in-tree snitches have been refactored to extract implementations of
the new interfaces so that their functionality can also be used via the new `initial_location_provider`,
`node_proximity` and `address_config` settings.
5.0
===

94
conf/cassandra.yaml
View File

@ -1465,6 +1465,9 @@ slow_query_log_timeout: 500ms
# most users should never need to adjust this.
# phi_convict_threshold: 8
# IEndpointSnitch has been deprecated in Cassandra 5.1
# Configuring a cluster with an IEndpointSnitch implementation using the endpoint_snitch setting remains supported,
# but is superceded by the new settings detailed below.
# endpoint_snitch -- Set this to a class that implements
# IEndpointSnitch. The snitch has two functions:
#
@ -1550,6 +1553,97 @@ slow_query_log_timeout: 500ms
# of the snitch, which will be assumed to be on your classpath.
endpoint_snitch: SimpleSnitch
# The settings in the following section are intended to supercede the use of endpoint_snitch:
# initial_location_provider
# node_proximity
# addresses_config (optional)
# prefer_local_connections (optional, defaults to false)
# The initial location provider supplies the datacenter and rack for a new node joining the
# cluster for the first time. This DC/rack is used to register the node with the cluster and
# is then persisted in and propagated by cluster metadata. The initial location provider is
# not used after this initial phase of the node lifecycle is complete. Cassandra provides the
# following implementations:
#
# SimpleLocationProvider:
# Hardcoded to supply a static location of `datacenter1/rack1`. Replicates the behaviour of
# SimpleSnitch.
#
# RackDCFileLocationProvider:
# The rack and datacenter for the local node are defined in cassandra-rackdc.properties,
# and is backwards compatible with the configuration of GossipingPropertyFileSnitch.
#
# TopologyFileLocationProvider:
# The rack and datacenter for the local node are defined in cassandra-topology.properties,
# and is backwards compatible with the configuration of PropertyFileSnitch. Only the location
# info for the local node is relevant now, rack/dc specifications for other endpoints are
# ignored.
#
# AlibabaCloudLocationProvider:
# Obtains datacenter and rack of the node from the metadata service of Alibaba cloud. This
# maps the ECS region to datacenter and the ECS availability zone to the rack. Intended to
# replace use of AlibabaCloudSnitch.
#
# AzureCloudLocationProvider:
# Maps datacenter to `location` and rack to `zone` fields of the `compute` object obtained
# from the Azure metadata service. If the availablity zone is not enabled, it will use the
# failure domain and get its respective value. Intended to replace use of AzureSnitch.
#
# CloudstackLocationProvider:
# Assumes a Cloudstack Zone follows the typical convention country-location-az, using a
# country/location tuple as a datacenter and the availability zone as a rack. Intended to
# replace use of CloudstackSnitch.
# WARNING: This legacy snitch has been deprecated and is scheduled to be removed in a future
# version of Cassandra.
#
# Ec2LocationProvider:
# Loads Region and Availability Zone information from the EC2 API. The Region is treated as the
# datacenter and the AvailablilityZone as the rack. Intended to replace use of Ec2Snitch and
# Ec2MultiRegionSnitch.
#
# GoogleCloudLocationProvider:
# Fetches source data from the metadata service of Google Cloud. This provider maps the GCE
# region to datacenter and the availibility zone to the rack. Intended to replace use of
# GoogleCloudSnitch.
#initial_location_provider: SimpleLocationProvider
# Node proximity controls sorting and ranking of replica lists in order to route requests efficiently.
# Dynamic snitch (not configured here) provides an additional level of sorting on top of whatever is
# specified here of sorting based on recorded latencies between peers.
#
# Out of the box Cassandra provides:
#
# NoOpProximity:
# Performs no reordering of replica lists and assigns every endpoint equal proximity, regardless of
# actual topology. This replicates the behaviour of SimpleSnitch.
#
# NetworkTopologyProximity:
# Ranks proximity of nodes based on datacenter and rack. Nodes in the same rack are considered
# closest, followed by those in the same datacenter. Nodes in different datacenters are
# considered furthest apart. Replicates the behaviour of topology aware snitches such as
# GossipingPropertyFileSnitch, PropertyFileSnitch and the multiple cloud platform snitches.
#node_proximity: NetworkTopologyProximity
# Address configuration (public and private endpoint addresses) is almost always derived directly
# from this configuration file using broadcast_address. In certain deployments, both the public and
# private addresses may be configured from an external source. For instance, in EC2 a node will have
# discover both of its own addresses at startup by querying the appropriate cloud metadata service.
# In multi-region EC2 deployments, this should be set to Ec2MultiRegionAddressConfig, to replicate the
# behaviour of Ec2MultiRegionSnitch.
#addresses_config: Ec2MultiRegionAddressConfig
# Controls whether or not to ensure that connections to peers in the same datacenter are established
# using private addresses. Typically, this is for situations like EC2 where a node will have a public
# address and a private address. We may initially connect on the public, then discover the private, and
# reconnect on the private.
# In Ec2MultiRegionSnitch (now deprecated), this behaviour was hard coded so if migrating from that
# snitch to modern config, set this to true.
# In GossipingPropertyFileSnitch (now deprecated), this behaviour was configured by the `prefer_local`
# property so if migrating from that snitch to modern config, set this accordingly.
# Note that all of the deprecated in-tree snitches can still be used in configuration, so any config
# migration is currently optional.
#prefer_local_connections: false
# controls how often to perform the more expensive part of host score
# calculation
# Min unit: ms

150
conf/cassandra_latest.yaml
View File

@ -1445,90 +1445,94 @@ slow_query_log_timeout: 500ms
# most users should never need to adjust this.
# phi_convict_threshold: 8
# endpoint_snitch -- Set this to a class that implements
# IEndpointSnitch. The snitch has two functions:
# The settings in the following section are intended to supercede the use of endpoint_snitch:
# initial_location_provider
# node_proximity
# addresses_config (optional)
# prefer_local_connections (optional, defaults to false)
# The initial location provider supplies the datacenter and rack for a new node joining the
# cluster for the first time. This DC/rack is used to register the node with the cluster and
# is then persisted in and propagated by cluster metadata. The initial location provider is
# not used after this initial phase of the node lifecycle is complete. Cassandra provides the
# following implementations:
#
# - it teaches Cassandra enough about your network topology to route
# requests efficiently
# - it allows Cassandra to spread replicas around your cluster to avoid
# correlated failures. It does this by grouping machines into
# "datacenters" and "racks." Cassandra will do its best not to have
# more than one replica on the same "rack" (which may not actually
# be a physical location)
# SimpleLocationProvider:
# Hardcoded to supply a static location of `datacenter1/rack1`. Replicates the behaviour of
# SimpleSnitch.
#
# CASSANDRA WILL NOT ALLOW YOU TO SWITCH TO AN INCOMPATIBLE SNITCH
# ONCE DATA IS INSERTED INTO THE CLUSTER. This would cause data loss.
# This means that if you start with the default SimpleSnitch, which
# locates every node on "rack1" in "datacenter1", your only options
# if you need to add another datacenter are GossipingPropertyFileSnitch
# (and the older PFS). From there, if you want to migrate to an
# incompatible snitch like Ec2Snitch you can do it by adding new nodes
# under Ec2Snitch (which will locate them in a new "datacenter") and
# decommissioning the old ones.
# RackDCFileLocationProvider:
# The rack and datacenter for the local node are defined in cassandra-rackdc.properties,
# and is backwards compatible with the configuration of GossipingPropertyFileSnitch.
#
# Out of the box, Cassandra provides:
# TopologyFileLocationProvider:
# The rack and datacenter for the local node are defined in cassandra-topology.properties,
# and is backwards compatible with the configuration of PropertyFileSnitch. Only the location
# info for the local node is relevant now, rack/dc specifications for other endpoints are
# ignored.
#
# SimpleSnitch:
# Treats Strategy order as proximity. This can improve cache
# locality when disabling read repair. Only appropriate for
# single-datacenter deployments.
# AlibabaCloudLocationProvider:
# Obtains datacenter and rack of the node from the metadata service of Alibaba cloud. This
# maps the ECS region to datacenter and the ECS availability zone to the rack. Intended to
# replace use of AlibabaCloudSnitch.
#
# GossipingPropertyFileSnitch
# This should be your go-to snitch for production use. The rack
# and datacenter for the local node are defined in
# cassandra-rackdc.properties and propagated to other nodes via
# gossip. If cassandra-topology.properties exists, it is used as a
# fallback, allowing migration from the PropertyFileSnitch.
# AzureCloudLocationProvider:
# Maps datacenter to `location` and rack to `zone` fields of the `compute` object obtained
# from the Azure metadata service. If the availablity zone is not enabled, it will use the
# failure domain and get its respective value. Intended to replace use of AzureSnitch.
#
# PropertyFileSnitch:
# Proximity is determined by rack and data center, which are
# explicitly configured in cassandra-topology.properties.
# CloudstackLocationProvider:
# Assumes a Cloudstack Zone follows the typical convention country-location-az, using a
# country/location tuple as a datacenter and the availability zone as a rack. Intended to
# replace use of CloudstackSnitch.
# WARNING: This legacy snitch has been deprecated and is scheduled to be removed in a future
# version of Cassandra.
#
# AlibabaCloudSnitch:
# Snitch for getting dc and rack of a node from metadata service of Alibaba cloud.
# This snitch that assumes an ECS region is a DC and an ECS availability_zone is a rack.
# Ec2LocationProvider:
# Loads Region and Availability Zone information from the EC2 API. The Region is treated as the
# datacenter and the AvailablilityZone as the rack. Intended to replace use of Ec2Snitch and
# Ec2MultiRegionSnitch.
#
# AzureSnitch:
# Gets datacenter from 'location' and rack from 'zone' fields of 'compute' object
# from instance metadata service. If the availability zone is not enabled, it will use the fault
# domain and get its respective value.
# GoogleCloudLocationProvider:
# Fetches source data from the metadata service of Google Cloud. This provider maps the GCE
# region to datacenter and the availibility zone to the rack. Intended to replace use of
# GoogleCloudSnitch.
initial_location_provider: SimpleLocationProvider
# Node proximity controls sorting and ranking of replica lists in order to route requests efficiently.
# Dynamic snitch (not configured here) provides an additional level of sorting on top of whatever is
# specified here of sorting based on recorded latencies between peers.
#
# CloudstackSnitch:
# A snitch that assumes a Cloudstack Zone follows the typical convention
# country-location-az and uses a country/location tuple as a datacenter
# and the availability zone as a rack.
# WARNING: This snitch is deprecated and it is scheduled to be removed
# in the next major version of Cassandra.
# Out of the box Cassandra provides:
#
# Ec2Snitch:
# Appropriate for EC2 deployments in a single Region. Loads Region
# and Availability Zone information from the EC2 API. The Region is
# treated as the datacenter, and the Availability Zone as the rack.
# Only private IPs are used, so this will not work across multiple
# Regions.
# NoOpProximity:
# Performs no reordering of replica lists and assigns every endpoint equal proximity, regardless of
# actual topology. This replicates the behaviour of SimpleSnitch.
#
# Ec2MultiRegionSnitch:
# Uses public IPs as broadcast_address to allow cross-region
# connectivity. (Thus, you should set seed addresses to the public
# IP as well.) You will need to open the storage_port or
# ssl_storage_port on the public IP firewall. (For intra-Region
# traffic, Cassandra will switch to the private IP after
# establishing a connection.)
#
# GoogleCloudSnitch:
# Snitch for getting dc and rack of a node from metadata service of Google cloud.
# This snitch that assumes an GCE region is a DC and an GCE availability_zone is a rack.
#
# RackInferringSnitch:
# Proximity is determined by rack and data center, which are
# assumed to correspond to the 3rd and 2nd octet of each node's IP
# address, respectively. Unless this happens to match your
# deployment conventions, this is best used as an example of
# writing a custom Snitch class and is provided in that spirit.
#
# You can use a custom Snitch by setting this to the full class name
# of the snitch, which will be assumed to be on your classpath.
endpoint_snitch: SimpleSnitch
# NetworkTopologyProximity:
# Ranks proximity of nodes based on datacenter and rack. Nodes in the same rack are considered
# closest, followed by those in the same datacenter. Nodes in different datacenters are
# considered furthest apart. Replicates the behaviour of topology aware snitches such as
# GossipingPropertyFileSnitch, PropertyFileSnitch and the multiple cloud platform snitches.
node_proximity: NetworkTopologyProximity
# Address configuration (public and private endpoint addresses) is almost always derived directly
# from this configuration file using broadcast_address. In certain deployments, both the public and
# private addresses may be configured from an external source. For instance, in EC2 a node will have
# discover both of its own addresses at startup by querying the appropriate cloud metadata service.
# In multi-region EC2 deployments, this should be set to Ec2MultiRegionAddressConfig, to replicate the
# behaviour of Ec2MultiRegionSnitch.
#addresses_config: Ec2MultiRegionAddressConfig
# Controls whether or not to ensure that connections to peers in the same datacenter are established
# using private addresses. Typically, this is for situations like EC2 where a node will have a public
# address and a private address. We may initially connect on the public, then discover the private, and
# reconnect on the private.
# In Ec2MultiRegionSnitch (now deprecated), this behaviour was hard coded so if migrating from that
# snitch to modern config, set this to true.
# In GossipingPropertyFileSnitch (now deprecated), this behaviour was configured by the `prefer_local`
# property so if migrating from that snitch to modern config, set this accordingly.
#prefer_local_connections: false
# controls how often to perform the more expensive part of host score
# calculation

5
src/java/org/apache/cassandra/config/Config.java
View File

@ -430,6 +430,11 @@ public class Config
public DurationSpec.IntMillisecondsBound dynamic_snitch_reset_interval = new DurationSpec.IntMillisecondsBound("10m");
public double dynamic_snitch_badness_threshold = 1.0;
public String node_proximity;
public String initial_location_provider;
public String addresses_config;
public boolean prefer_local_connections = false;
public String failure_detector = "FailureDetector";
public EncryptionOptions.ServerEncryptionOptions server_encryption_options = new EncryptionOptions.ServerEncryptionOptions();

169
src/java/org/apache/cassandra/config/DatabaseDescriptor.java
View File

@ -32,7 +32,6 @@ import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.Comparator;
import java.util.Enumeration;
import java.util.HashMap;
import java.util.HashSet;
@ -90,6 +89,7 @@ import org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard;
import org.apache.cassandra.dht.IPartitioner;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.fql.FullQueryLoggerOptions;
import org.apache.cassandra.gms.IEndpointStateChangeSubscriber;
import org.apache.cassandra.gms.IFailureDetector;
import org.apache.cassandra.gms.VersionedValue;
import org.apache.cassandra.io.FSWriteError;
@ -105,8 +105,14 @@ import org.apache.cassandra.locator.DynamicEndpointSnitch;
import org.apache.cassandra.locator.EndpointSnitchInfo;
import org.apache.cassandra.locator.IEndpointSnitch;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.locator.Replica;
import org.apache.cassandra.locator.Locator;
import org.apache.cassandra.locator.LocationInfo;
import org.apache.cassandra.locator.InitialLocationProvider;
import org.apache.cassandra.locator.NodeAddressConfig;
import org.apache.cassandra.locator.ReconnectableSnitchHelper;
import org.apache.cassandra.locator.SeedProvider;
import org.apache.cassandra.locator.NodeProximity;
import org.apache.cassandra.locator.SnitchAdapter;
import org.apache.cassandra.security.AbstractCryptoProvider;
import org.apache.cassandra.security.EncryptionContext;
import org.apache.cassandra.security.JREProvider;
@ -114,6 +120,7 @@ import org.apache.cassandra.security.SSLFactory;
import org.apache.cassandra.service.CacheService.CacheType;
import org.apache.cassandra.service.StorageService;
import org.apache.cassandra.service.paxos.Paxos;
import org.apache.cassandra.tcm.RegistrationStatus;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.MBeanWrapper;
import org.apache.cassandra.utils.Pair;
@ -185,7 +192,10 @@ public class DatabaseDescriptor
static final DurationSpec.LongMillisecondsBound LOWEST_ACCEPTED_TIMEOUT = new DurationSpec.LongMillisecondsBound(10L);
private static Supplier<IFailureDetector> newFailureDetector;
private static IEndpointSnitch snitch;
private static NodeProximity nodeProximity;
private static Locator initializationLocator;
private static InitialLocationProvider initialLocationProvider;
private static IEndpointStateChangeSubscriber localAddressReconnector;
private static InetAddress listenAddress; // leave null so we can fall through to getLocalHost
private static InetAddress broadcastAddress;
private static InetAddress rpcAddress;
@ -218,8 +228,6 @@ public class DatabaseDescriptor
private static long counterCacheSizeInMiB;
private static long indexSummaryCapacityInMiB;
private static String localDC;
private static Comparator<Replica> localComparator;
private static EncryptionContext encryptionContext;
private static boolean hasLoggedConfig;
@ -292,6 +300,7 @@ public class DatabaseDescriptor
sstableFormats = null;
clearMBean("org.apache.cassandra.db:type=DynamicEndpointSnitch");
clearMBean("org.apache.cassandra.db:type=EndpointSnitchInfo");
clearMBean("org.apache.cassandra.db:type=LocationInfo");
}
private static void clearMBean(String name)
@ -352,6 +361,8 @@ public class DatabaseDescriptor
applySnitch();
applyFailureDetector();
applyEncryptionContext();
}
@ -518,6 +529,8 @@ public class DatabaseDescriptor
applySnitch();
applyFailureDetector();
applyTokensConfig();
applySeedProvider();
@ -1484,24 +1497,50 @@ public class DatabaseDescriptor
// definitely not safe for tools + clients - implicitly instantiates StorageService
public static void applySnitch()
{
/* end point snitch */
if (conf.endpoint_snitch == null)
{
throw new ConfigurationException("Missing endpoint_snitch directive", false);
}
snitch = createEndpointSnitch(conf.dynamic_snitch, conf.endpoint_snitch);
EndpointSnitchInfo.create();
boolean hasLegacyConfig = conf.endpoint_snitch != null;
boolean hasModernConfig = conf.initial_location_provider != null && conf.node_proximity != null;
localDC = snitch.getLocalDatacenter();
localComparator = (replica1, replica2) -> {
boolean local1 = localDC.equals(snitch.getDatacenter(replica1));
boolean local2 = localDC.equals(snitch.getDatacenter(replica2));
if (local1 && !local2)
return -1;
if (local2 && !local1)
return 1;
return 0;
};
if (hasLegacyConfig == hasModernConfig)
throw new ConfigurationException("Configuration must specify either node_proximity and " +
"initial_location_provider or endpoint_snitch but not both. ");
NodeProximity proximity;
NodeAddressConfig addressConfig;
if (hasLegacyConfig)
{
logger.info("Use of endpoint_snitch in configuration is deprecated and should be replaced by " +
"initial_location_provider and node_proximity");
SnitchAdapter adapter = new SnitchAdapter(createEndpointSnitch(conf.endpoint_snitch));
proximity = adapter;
initialLocationProvider = adapter;
addressConfig = adapter;
}
else
{
proximity = createProximityImpl(conf.node_proximity);
initialLocationProvider = createInitialLocationProvider(conf.initial_location_provider);
addressConfig = conf.addresses_config != null
? createAddressConfig(conf.addresses_config)
: NodeAddressConfig.DEFAULT;
}
// this is done here and not in applyAddressConfig as historically, Ec2MultiRegionSnitch is
// responsible for querying the cloud metadata service to get the public IP used for
// broadcast_address and we only want to instantiate the snitch here.
addressConfig.configureAddresses();
initializationLocator = new Locator(RegistrationStatus.instance,
FBUtilities.getBroadcastAddressAndPort(),
initialLocationProvider);
nodeProximity = conf.dynamic_snitch ? new DynamicEndpointSnitch(proximity) : proximity;
localAddressReconnector = addressConfig.preferLocalConnections()
? new ReconnectableSnitchHelper(initializationLocator, true)
: new IEndpointStateChangeSubscriber() { /* NO-OP */ };
EndpointSnitchInfo.create();
LocationInfo.create();
}
public static void applyFailureDetector()
{
newFailureDetector = () -> createFailureDetector(conf.failure_detector);
}
@ -1717,12 +1756,36 @@ public class DatabaseDescriptor
});
}
public static IEndpointSnitch createEndpointSnitch(boolean dynamic, String snitchClassName) throws ConfigurationException
public static IEndpointSnitch createEndpointSnitch(String snitchClassName) throws ConfigurationException
{
if (!snitchClassName.contains("."))
snitchClassName = "org.apache.cassandra.locator." + snitchClassName;
IEndpointSnitch snitch = FBUtilities.construct(snitchClassName, "snitch");
return dynamic ? new DynamicEndpointSnitch(snitch) : snitch;
return snitch;
}
public static NodeProximity createProximityImpl(String className) throws ConfigurationException
{
if (!className.contains("."))
className = "org.apache.cassandra.locator." + className;
NodeProximity sorter = FBUtilities.construct(className, "node proximity measurement");
return sorter;
}
public static InitialLocationProvider createInitialLocationProvider(String className) throws ConfigurationException
{
if (!className.contains("."))
className = "org.apache.cassandra.locator." + className;
InitialLocationProvider provider = FBUtilities.construct(className, "initial location provider");
return provider;
}
public static NodeAddressConfig createAddressConfig(String className) throws ConfigurationException
{
if (!className.contains("."))
className = "org.apache.cassandra.locator." + className;
NodeAddressConfig config = FBUtilities.construct(className, "node address config");
return config;
}
private static IFailureDetector createFailureDetector(String detectorClassName) throws ConfigurationException
@ -2083,14 +2146,49 @@ public class DatabaseDescriptor
return old;
}
public static IEndpointSnitch getEndpointSnitch()
public static IEndpointStateChangeSubscriber getLocalAddressReconnectionHelper()
{
return snitch;
return localAddressReconnector;
}
public static void setEndpointSnitch(IEndpointSnitch eps)
public static boolean preferLocalConnections()
{
snitch = eps;
return conf.prefer_local_connections;
}
/**
* Return a Locator that can be used from the moment a node begins its initial startup.
* It is not initialized with a ClusterMetadata instance (as there may not be one yet), so
* instead it always performs lookups using the current metadata. This means it is possible
* for the value of a lookup for the same endpoint to change between calls e.g. if a peer
* were to register or change broadcast address between the two calls being made.
* @return Locator
*/
public static Locator getLocator()
{
if (initializationLocator == null && isClientInitialized())
return Locator.forClients();
return initializationLocator;
}
/**
* Used to provide the location (dc/rack) of the local node during its initial startup
* for the purpose of registering it with ClusterMetadata.
* See: org.apache.cassandra.locator.Locator
*/
public static InitialLocationProvider getInitialLocationProvider()
{
return initialLocationProvider;
}
public static NodeProximity getNodeProximity()
{
return nodeProximity;
}
public static void setNodeProximity(NodeProximity proximity)
{
nodeProximity = proximity;
}
public static IFailureDetector newFailureDetector()
@ -3708,7 +3806,7 @@ public class DatabaseDescriptor
public static boolean isDynamicEndpointSnitch()
{
// not using config.dynamic_snitch because snitch can be changed via JMX
return snitch instanceof DynamicEndpointSnitch;
return nodeProximity instanceof DynamicEndpointSnitch;
}
public static Config.BatchlogEndpointStrategy getBatchlogEndpointStrategy()
@ -4038,18 +4136,7 @@ public class DatabaseDescriptor
public static String getLocalDataCenter()
{
return localDC;
}
@VisibleForTesting
public static void setLocalDataCenter(String value)
{
localDC = value;
}
public static Comparator<Replica> getLocalComparator()
{
return localComparator;
return initializationLocator == null ? null : initializationLocator.local().datacenter;
}
public static Config.InternodeCompression internodeCompression()

20
src/java/org/apache/cassandra/config/EncryptionOptions.java
View File

@ -32,10 +32,11 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.locator.IEndpointSnitch;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.locator.Locator;
import org.apache.cassandra.security.DisableSslContextFactory;
import org.apache.cassandra.security.ISslContextFactory;
import org.apache.cassandra.tcm.membership.Location;
import org.apache.cassandra.utils.FBUtilities;
import static org.apache.cassandra.utils.LocalizeString.toLowerCaseLocalized;
@ -786,7 +787,15 @@ public class EncryptionOptions
public boolean shouldEncrypt(InetAddressAndPort endpoint)
{
IEndpointSnitch snitch = DatabaseDescriptor.getEndpointSnitch();
// When a node is started for the very first time, it has no way to determine whether the seed nodes
// it makes its initial connections to are in a local or remote datacenter and/or rack. When the node is
// in this specific state, Locator will return the constant Location.UNKNOWN for any lookup of a peer's
// location. This is intended to ensure that _all_ peers are treated as remote during this initial phase and
// that the most strict encryption settings allowable by the internode_encryption settings are applied.
// Any connections established during this phase, of which there should be few, will be dropped and
// re-established as soon as the node initialises its local ClusterMetadata and so is able to get accurate
// topology information for peers.
Locator locator = DatabaseDescriptor.getLocator();
switch (internode_encryption)
{
case none:
@ -794,13 +803,14 @@ public class EncryptionOptions
case all:
break;
case dc:
if (snitch.getDatacenter(endpoint).equals(snitch.getLocalDatacenter()))
if (locator.location(endpoint).datacenter.equals(locator.local().datacenter))
return false;
break;
case rack:
// for rack then check if the DC's are the same.
if (snitch.getRack(endpoint).equals(snitch.getLocalRack())
&& snitch.getDatacenter(endpoint).equals(snitch.getLocalDatacenter()))
Location remote = locator.location(endpoint);
Location local = locator.local();
if (remote.rack.equals(local.rack) && remote.datacenter.equals(local.datacenter))
return false;
break;
}

9
src/java/org/apache/cassandra/cql3/statements/DescribeStatement.java
View File

@ -41,6 +41,8 @@ import org.apache.cassandra.exceptions.RequestExecutionException;
import org.apache.cassandra.exceptions.RequestValidationException;
import org.apache.cassandra.io.util.DataInputBuffer;
import org.apache.cassandra.io.util.DataOutputBuffer;
import org.apache.cassandra.locator.NodeProximity;
import org.apache.cassandra.locator.SnitchAdapter;
import org.apache.cassandra.schema.*;
import org.apache.cassandra.service.ClientState;
import org.apache.cassandra.service.QueryState;
@ -674,8 +676,11 @@ public abstract class DescribeStatement<T> extends CQLStatement.Raw implements C
List<Object> list = new ArrayList<Object>();
list.add(DatabaseDescriptor.getClusterName());
list.add(trimIfPresent(DatabaseDescriptor.getPartitionerName(), "org.apache.cassandra.dht."));
list.add(trimIfPresent(DatabaseDescriptor.getEndpointSnitch().getClass().getName(),
"org.apache.cassandra.locator."));
NodeProximity proximity = DatabaseDescriptor.getNodeProximity();
String nodeProximityClassName = proximity instanceof SnitchAdapter ? ((SnitchAdapter) proximity).snitch.getClass().getName()
: proximity.getClass().getName();
list.add(trimIfPresent(nodeProximityClassName,
"org.apache.cassandra.locator."));
String useKs = state.getRawKeyspace();
if (mustReturnsRangeOwnerships(useKs))

11
src/java/org/apache/cassandra/db/ConsistencyLevel.java
View File

@ -18,13 +18,14 @@
package org.apache.cassandra.db;
import com.carrotsearch.hppc.ObjectIntHashMap;
import org.apache.cassandra.locator.Endpoints;
import org.apache.cassandra.locator.InOurDc;
import org.apache.cassandra.schema.TableMetadata;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.locator.AbstractReplicationStrategy;
import org.apache.cassandra.locator.Endpoints;
import org.apache.cassandra.locator.InOurDc;
import org.apache.cassandra.locator.Locator;
import org.apache.cassandra.locator.NetworkTopologyStrategy;
import org.apache.cassandra.schema.TableMetadata;
import org.apache.cassandra.transport.ProtocolException;
import static org.apache.cassandra.locator.Replicas.addToCountPerDc;
@ -121,10 +122,10 @@ public enum ConsistencyLevel
}
}
public static ObjectIntHashMap<String> eachQuorumForWrite(AbstractReplicationStrategy replicationStrategy, Endpoints<?> pendingWithDown)
public static ObjectIntHashMap<String> eachQuorumForWrite(Locator locator, AbstractReplicationStrategy replicationStrategy, Endpoints<?> pendingWithDown)
{
ObjectIntHashMap<String> perDc = eachQuorumForRead(replicationStrategy);
addToCountPerDc(perDc, pendingWithDown, 1);
addToCountPerDc(locator, perDc, pendingWithDown, 1);
return perDc;
}

2
src/java/org/apache/cassandra/db/CounterMutationVerbHandler.java
View File

@ -38,7 +38,7 @@ public class CounterMutationVerbHandler extends AbstractMutationVerbHandler<Coun
final CounterMutation cm = message.payload;
logger.trace("Applying forwarded {}", cm);
String localDataCenter = DatabaseDescriptor.getEndpointSnitch().getLocalDatacenter();
String localDataCenter = DatabaseDescriptor.getLocator().local().datacenter;
// We should not wait for the result of the write in this thread,
// otherwise we could have a distributed deadlock between replicas
// running this VerbHandler (see #4578).

22
src/java/org/apache/cassandra/db/SystemKeyspace.java
View File

@ -82,7 +82,6 @@ import org.apache.cassandra.io.util.DataInputBuffer;
import org.apache.cassandra.io.util.DataOutputBuffer;
import org.apache.cassandra.io.util.File;
import org.apache.cassandra.io.util.RebufferingInputStream;
import org.apache.cassandra.locator.IEndpointSnitch;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.locator.MetaStrategy;
import org.apache.cassandra.metrics.RestorableMeter;
@ -115,6 +114,7 @@ import org.apache.cassandra.service.snapshot.SnapshotType;
import org.apache.cassandra.streaming.StreamOperation;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.Epoch;
import org.apache.cassandra.tcm.membership.Location;
import org.apache.cassandra.tcm.membership.NodeState;
import org.apache.cassandra.transport.ProtocolVersion;
import org.apache.cassandra.utils.ByteBufferUtil;
@ -653,15 +653,23 @@ public final class SystemKeyspace
"listen_address," +
"listen_port" +
") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)";
IEndpointSnitch snitch = DatabaseDescriptor.getEndpointSnitch();
// If this node has not yet been registered in cluster metadata, record the initialization location provided
// by the configured Locator, this will also be used when registering an new node in cluster metadata during its
// intial startup.
// If the node is present in cluster metadata (i.e. has been registered already and we have replayed the
// metadata log), then get the location from there (via the Locator).
// We do this in case either the Locator config or location in metadata has been modified in any way, as cluster
// metadata is the ultimate source of truth.
Location location = DatabaseDescriptor.getLocator().local();
executeOnceInternal(format(req, LOCAL),
LOCAL,
DatabaseDescriptor.getClusterName(),
FBUtilities.getReleaseVersionString(),
QueryProcessor.CQL_VERSION.toString(),
String.valueOf(ProtocolVersion.CURRENT.asInt()),
snitch.getLocalDatacenter(),
snitch.getLocalRack(),
location.datacenter,
location.rack,
DatabaseDescriptor.getPartitioner().getClass().getName(),
FBUtilities.getJustBroadcastNativeAddress(),
DatabaseDescriptor.getNativeTransportPort(),
@ -950,6 +958,12 @@ public final class SystemKeyspace
executeInternal(format(req, LOCAL, LOCAL), version);
}
public static synchronized void updateRack(String rack)
{
String req = "INSERT INTO system.%s (key, rack) VALUES ('%s', ?)";
executeInternal(format(req, LOCAL, LOCAL), rack);
}
public static Set<String> tokensAsSet(Collection<Token> tokens)
{
if (tokens.isEmpty())

6
src/java/org/apache/cassandra/db/view/ViewUtils.java
View File

@ -22,13 +22,13 @@ import java.util.Optional;
import java.util.function.Predicate;
import com.google.common.collect.Iterables;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.dht.Token;
import org.apache.cassandra.locator.EndpointsForToken;
import org.apache.cassandra.locator.NetworkTopologyStrategy;
import org.apache.cassandra.locator.Replica;
import org.apache.cassandra.schema.KeyspaceMetadata;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.membership.Location;
public final class ViewUtils
{
@ -60,7 +60,7 @@ public final class ViewUtils
*/
public static Optional<Replica> getViewNaturalEndpoint(ClusterMetadata metadata, String keyspace, Token baseToken, Token viewToken)
{
String localDataCenter = DatabaseDescriptor.getEndpointSnitch().getLocalDatacenter();
Location local = metadata.locator.local();
KeyspaceMetadata keyspaceMetadata = metadata.schema.getKeyspaces().getNullable(keyspace);
EndpointsForToken naturalBaseReplicas = metadata.placements.get(keyspaceMetadata.params.replication).reads.forToken(baseToken).get();
@ -73,7 +73,7 @@ public final class ViewUtils
// We only select replicas from our own DC
// TODO: this is poor encapsulation, leaking implementation details of replication strategy
Predicate<Replica> isLocalDC = r -> !(keyspaceMetadata.replicationStrategy instanceof NetworkTopologyStrategy)
|| DatabaseDescriptor.getEndpointSnitch().getDatacenter(r).equals(localDataCenter);
|| metadata.locator.location(r.endpoint()).sameDatacenter(local);
// We have to remove any endpoint which is shared between the base and the view, as it will select itself
// and throw off the counts otherwise.

6
src/java/org/apache/cassandra/db/virtual/InternodeInboundTable.java
View File

@ -32,6 +32,7 @@ import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.net.InboundMessageHandlers;
import org.apache.cassandra.schema.TableMetadata;
import org.apache.cassandra.tcm.membership.Location;
public final class InternodeInboundTable extends AbstractVirtualTable
{
@ -112,9 +113,8 @@ public final class InternodeInboundTable extends AbstractVirtualTable
private void addRow(SimpleDataSet dataSet, InetAddressAndPort addressAndPort, InboundMessageHandlers handlers)
{
String dc = DatabaseDescriptor.getEndpointSnitch().getDatacenter(addressAndPort);
String rack = DatabaseDescriptor.getEndpointSnitch().getRack(addressAndPort);
dataSet.row(addressAndPort.getAddress(), addressAndPort.getPort(), dc, rack)
Location location = DatabaseDescriptor.getLocator().location(addressAndPort);
dataSet.row(addressAndPort.getAddress(), addressAndPort.getPort(), location.datacenter, location.rack)
.column(USING_BYTES, handlers.usingCapacity())
.column(USING_RESERVE_BYTES, handlers.usingEndpointReserveCapacity())
.column(CORRUPT_FRAMES_RECOVERED, handlers.corruptFramesRecovered())

6
src/java/org/apache/cassandra/db/virtual/InternodeOutboundTable.java
View File

@ -34,6 +34,7 @@ import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.net.OutboundConnection;
import org.apache.cassandra.net.OutboundConnections;
import org.apache.cassandra.schema.TableMetadata;
import org.apache.cassandra.tcm.membership.Location;
public final class InternodeOutboundTable extends AbstractVirtualTable
{
@ -112,10 +113,9 @@ public final class InternodeOutboundTable extends AbstractVirtualTable
private void addRow(SimpleDataSet dataSet, InetAddressAndPort addressAndPort, OutboundConnections connections)
{
String dc = DatabaseDescriptor.getEndpointSnitch().getDatacenter(addressAndPort);
String rack = DatabaseDescriptor.getEndpointSnitch().getRack(addressAndPort);
Location location = DatabaseDescriptor.getLocator().location(addressAndPort);
long pendingBytes = sum(connections, OutboundConnection::pendingBytes);
dataSet.row(addressAndPort.getAddress(), addressAndPort.getPort(), dc, rack)
dataSet.row(addressAndPort.getAddress(), addressAndPort.getPort(), location.datacenter, location.rack)
.column(USING_BYTES, pendingBytes)
.column(USING_RESERVE_BYTES, connections.usingReserveBytes())
.column(PENDING_COUNT, sum(connections, OutboundConnection::pendingCount))

11
src/java/org/apache/cassandra/db/virtual/PendingHintsTable.java
View File

@ -36,10 +36,11 @@ import org.apache.cassandra.gms.FailureDetector;
import org.apache.cassandra.gms.FailureDetectorMBean;
import org.apache.cassandra.hints.HintsService;
import org.apache.cassandra.hints.PendingHintsInfo;
import org.apache.cassandra.locator.IEndpointSnitch;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.locator.Locator;
import org.apache.cassandra.schema.TableMetadata;
import org.apache.cassandra.service.StorageService;
import org.apache.cassandra.tcm.membership.Location;
public final class PendingHintsTable extends AbstractVirtualTable
{
@ -81,7 +82,7 @@ public final class PendingHintsTable extends AbstractVirtualTable
public DataSet data()
{
List<PendingHintsInfo> pendingHints = HintsService.instance.getPendingHintsInfo();
IEndpointSnitch snitch = DatabaseDescriptor.getEndpointSnitch();
Locator locator = DatabaseDescriptor.getLocator();
SimpleDataSet result = new SimpleDataSet(metadata());
@ -91,6 +92,7 @@ public final class PendingHintsTable extends AbstractVirtualTable
else
simpleStates = Collections.emptyMap();
Location location = Location.UNKNOWN;
for (PendingHintsInfo info : pendingHints)
{
InetAddressAndPort addressAndPort = StorageService.instance.getEndpointForHostId(info.hostId);
@ -101,10 +103,11 @@ public final class PendingHintsTable extends AbstractVirtualTable
String status = "Unknown";
if (addressAndPort != null)
{
location = locator.location(addressAndPort);
address = addressAndPort.getAddress();
port = addressAndPort.getPort();
rack = snitch.getRack(addressAndPort);
dc = snitch.getDatacenter(addressAndPort);
rack = location.rack;
dc = location.datacenter;
status = simpleStates.getOrDefault(addressAndPort.toString(), status);
}
result.row(info.hostId)

2
src/java/org/apache/cassandra/dht/BootStrapper.java
View File

@ -121,7 +121,7 @@ public class BootStrapper extends ProgressEventNotifierSupport
RangeStreamer streamer = new RangeStreamer(metadata,
StreamOperation.BOOTSTRAP,
useStrictConsistency,
DatabaseDescriptor.getEndpointSnitch(),
DatabaseDescriptor.getNodeProximity(),
stateStore,
true,
DatabaseDescriptor.getStreamingConnectionsPerHost(),

2
src/java/org/apache/cassandra/dht/Datacenters.java
View File

@ -28,7 +28,7 @@ public class Datacenters
{
private static class DCHandle
{
private static final String thisDc = DatabaseDescriptor.getEndpointSnitch().getLocalDatacenter();
private static final String thisDc = DatabaseDescriptor.getLocator().local().datacenter;
}
public static String thisDatacenter()

9
src/java/org/apache/cassandra/dht/RangeFetchMapCalculator.java
View File

@ -33,12 +33,12 @@ import com.google.common.collect.Multimap;
import org.apache.cassandra.locator.EndpointsByRange;
import org.apache.cassandra.locator.EndpointsForRange;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.locator.Locator;
import org.apache.cassandra.locator.Replica;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.locator.Replicas;
import org.psjava.algo.graph.flownetwork.FordFulkersonAlgorithm;
import org.psjava.algo.graph.flownetwork.MaximumFlowAlgorithm;
@ -82,14 +82,17 @@ public class RangeFetchMapCalculator
private final Vertex sourceVertex = OuterVertex.getSourceVertex();
private final Vertex destinationVertex = OuterVertex.getDestinationVertex();
private final Set<Range<Token>> trivialRanges;
private final Locator locator;
public RangeFetchMapCalculator(EndpointsByRange rangesWithSources,
Collection<RangeStreamer.SourceFilter> sourceFilters,
String keyspace)
String keyspace,
Locator locator)
{
this.rangesWithSources = rangesWithSources;
this.sourceFilters = Predicates.and(sourceFilters);
this.keyspace = keyspace;
this.locator = locator;
this.trivialRanges = rangesWithSources.keySet()
.stream()
.filter(RangeFetchMapCalculator::isTrivial)
@ -374,7 +377,7 @@ public class RangeFetchMapCalculator
private boolean isInLocalDC(Replica replica)
{
return DatabaseDescriptor.getLocalDataCenter().equals(DatabaseDescriptor.getEndpointSnitch().getDatacenter(replica));
return locator.local().sameDatacenter(locator.location(replica.endpoint()));
}
/**

44
src/java/org/apache/cassandra/dht/RangeStreamer.java
View File

@ -48,15 +48,16 @@ import org.apache.cassandra.locator.AbstractReplicationStrategy;
import org.apache.cassandra.locator.EndpointsByRange;
import org.apache.cassandra.locator.EndpointsByReplica;
import org.apache.cassandra.locator.EndpointsForRange;
import org.apache.cassandra.locator.IEndpointSnitch;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.locator.LocalStrategy;
import org.apache.cassandra.locator.Locator;
import org.apache.cassandra.locator.NetworkTopologyStrategy;
import org.apache.cassandra.locator.RangesAtEndpoint;
import org.apache.cassandra.locator.Replica;
import org.apache.cassandra.locator.ReplicaCollection;
import org.apache.cassandra.locator.ReplicaCollection.Builder.Conflict;
import org.apache.cassandra.locator.Replicas;
import org.apache.cassandra.locator.NodeProximity;
import org.apache.cassandra.schema.ReplicationParams;
import org.apache.cassandra.streaming.PreviewKind;
import org.apache.cassandra.streaming.StreamOperation;
@ -93,7 +94,7 @@ public class RangeStreamer
private final List<SourceFilter> sourceFilters = new ArrayList<>();
private final StreamPlan streamPlan;
private final boolean useStrictConsistency;
private final IEndpointSnitch snitch;
private final NodeProximity proximity;
private final StreamStateStore stateStore;
private final MovementMap movements;
private final MovementMap strictMovements;
@ -178,18 +179,18 @@ public class RangeStreamer
public static class SingleDatacenterFilter implements SourceFilter
{
private final String sourceDc;
private final IEndpointSnitch snitch;
private final Locator locator;
public SingleDatacenterFilter(IEndpointSnitch snitch, String sourceDc)
public SingleDatacenterFilter(Locator locator, String sourceDc)
{
this.sourceDc = sourceDc;
this.snitch = snitch;
this.locator = locator;
}
@Override
public boolean apply(Replica replica)
{
return snitch.getDatacenter(replica).equals(sourceDc);
return locator.location(replica.endpoint()).datacenter.equals(sourceDc);
}
@Override
@ -204,19 +205,19 @@ public class RangeStreamer
*/
public static class ExcludeLocalDatacenterFilter implements SourceFilter
{
private final IEndpointSnitch snitch;
private final Locator locator;
private final String localDc;
public ExcludeLocalDatacenterFilter(IEndpointSnitch snitch)
public ExcludeLocalDatacenterFilter(Locator locator)
{
this.snitch = snitch;
this.localDc = snitch.getLocalDatacenter();
this.locator = locator;
this.localDc = locator.local().datacenter;
}
@Override
public boolean apply(Replica replica)
{
return !snitch.getDatacenter(replica).equals(localDc);
return !locator.location(replica.endpoint()).datacenter.equals(localDc);
}
@Override
@ -292,21 +293,21 @@ public class RangeStreamer
public RangeStreamer(ClusterMetadata metadata,
StreamOperation streamOperation,
boolean useStrictConsistency,
IEndpointSnitch snitch,
NodeProximity proximity,
StreamStateStore stateStore,
boolean connectSequentially,
int connectionsPerHost,
MovementMap movements,
MovementMap strictMovements)
{
this(metadata, streamOperation, useStrictConsistency, snitch, stateStore,
this(metadata, streamOperation, useStrictConsistency, proximity, stateStore,
FailureDetector.instance, connectSequentially, connectionsPerHost, movements, strictMovements);
}
RangeStreamer(ClusterMetadata metadata,
StreamOperation streamOperation,
boolean useStrictConsistency,
IEndpointSnitch snitch,
NodeProximity proximity,
StreamStateStore stateStore,
IFailureDetector failureDetector,
boolean connectSequentially,
@ -319,7 +320,7 @@ public class RangeStreamer
this.description = streamOperation.getDescription();
this.streamPlan = new StreamPlan(streamOperation, connectionsPerHost, connectSequentially, null, PreviewKind.NONE);
this.useStrictConsistency = useStrictConsistency;
this.snitch = snitch;
this.proximity = proximity;
this.stateStore = stateStore;
this.movements = movements;
this.strictMovements = strictMovements;
@ -368,7 +369,7 @@ public class RangeStreamer
}
boolean useStrictSource = useStrictSourcesForRanges(keyspace.getMetadata().params.replication, strat);
EndpointsByReplica fetchMap = calculateRangesToFetchWithPreferredEndpoints(snitch::sortedByProximity,
EndpointsByReplica fetchMap = calculateRangesToFetchWithPreferredEndpoints(proximity::sortedByProximity,
keyspace.getReplicationStrategy(),
useStrictConsistency,
metadata,
@ -389,7 +390,7 @@ public class RangeStreamer
}
else
{
workMap = getOptimizedWorkMap(fetchMap, sourceFilters, keyspaceName);
workMap = getOptimizedWorkMap(fetchMap, sourceFilters, keyspaceName, metadata.locator);
}
if (toFetch.put(keyspaceName, workMap) != null)
@ -458,7 +459,7 @@ public class RangeStreamer
* consistency.
**/
public static EndpointsByReplica
calculateRangesToFetchWithPreferredEndpoints(BiFunction<InetAddressAndPort, EndpointsForRange, EndpointsForRange> snitchGetSortedListByProximity,
calculateRangesToFetchWithPreferredEndpoints(BiFunction<InetAddressAndPort, EndpointsForRange, EndpointsForRange> sortByProximity,
AbstractReplicationStrategy strat,
boolean useStrictConsistency,
ClusterMetadata metadata,
@ -473,7 +474,7 @@ public class RangeStreamer
logger.debug("To fetch RN: {}", movements.get(params).keySet());
Predicate<Replica> testSourceFilters = and(sourceFilters);
Function<EndpointsForRange, EndpointsForRange> sorted = endpoints -> snitchGetSortedListByProximity.apply(localAddress, endpoints);
Function<EndpointsForRange, EndpointsForRange> sorted = endpoints -> sortByProximity.apply(localAddress, endpoints);
//This list of replicas is just candidates. With strict consistency it's going to be a narrow list.
EndpointsByReplica.Builder rangesToFetchWithPreferredEndpoints = new EndpointsByReplica.Builder();
@ -599,7 +600,8 @@ public class RangeStreamer
*/
private static Multimap<InetAddressAndPort, FetchReplica> getOptimizedWorkMap(EndpointsByReplica rangesWithSources,
Collection<SourceFilter> sourceFilters,
String keyspace)
String keyspace,
Locator locator)
{
//For now we just aren't going to use the optimized range fetch map with transient replication to shrink
//the surface area to test and introduce bugs.
@ -613,7 +615,7 @@ public class RangeStreamer
}
EndpointsByRange unwrappedView = unwrapped.build();
RangeFetchMapCalculator calculator = new RangeFetchMapCalculator(unwrappedView, sourceFilters, keyspace);
RangeFetchMapCalculator calculator = new RangeFetchMapCalculator(unwrappedView, sourceFilters, keyspace, locator);
Multimap<InetAddressAndPort, Range<Token>> rangeFetchMapMap = calculator.getRangeFetchMap();
logger.info("Output from RangeFetchMapCalculator for keyspace {}", keyspace);
validateRangeFetchMap(unwrappedView, rangeFetchMapMap, keyspace);

33
src/java/org/apache/cassandra/dht/tokenallocator/OfflineTokenAllocator.java
View File

@ -23,7 +23,6 @@ import java.net.UnknownHostException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.TreeSet;
@ -32,13 +31,17 @@ import com.google.common.base.Preconditions;
import com.google.common.collect.Maps;
import org.apache.commons.math3.stat.descriptive.SummaryStatistics;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.dht.IPartitioner;
import org.apache.cassandra.dht.Token;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.locator.SimpleSnitch;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.RegistrationStatus;
import org.apache.cassandra.tcm.membership.Location;
import org.apache.cassandra.utils.OutputHandler;
import static org.apache.cassandra.locator.SimpleLocationProvider.LOCATION;
public class OfflineTokenAllocator
{
public static List<FakeNode> allocate(int rf, int numTokens, int[] nodesPerRack, OutputHandler logger, IPartitioner partitioner)
@ -54,6 +57,9 @@ public class OfflineTokenAllocator
Preconditions.checkArgument(nodes >= rf,
"not enough nodes %s for rf %s in %s", Arrays.stream(nodesPerRack).sum(), rf, Arrays.toString(nodesPerRack));
DatabaseDescriptor.setPartitionerUnsafe(partitioner);
// Set RegistrationStatus to REGISTERED so that Locator works exclusively from ClusterMetadata
RegistrationStatus.instance.onRegistration();
List<FakeNode> fakeNodes = new ArrayList<>(nodes);
MultinodeAllocator allocator = new MultinodeAllocator(rf, numTokens, logger, partitioner);
@ -114,27 +120,21 @@ public class OfflineTokenAllocator
private static class MultinodeAllocator
{
private final FakeSnitch fakeSnitch;
private final TokenAllocation allocation;
private final Map<Integer, SummaryStatistics> lastCheckPoint = Maps.newHashMap();
private final OutputHandler logger;
private MultinodeAllocator(int rf, int numTokens, OutputHandler logger, IPartitioner partitioner)
{
this.fakeSnitch = new FakeSnitch();
this.allocation = TokenAllocation.create(fakeSnitch, new ClusterMetadata(partitioner), rf, numTokens);
this.allocation = TokenAllocation.create(LOCATION.datacenter, new ClusterMetadata(partitioner), rf, numTokens);
this.logger = logger;
}
private FakeNode allocateTokensForNode(int nodeId, Integer rackId)
{
// Update snitch and token metadata info
InetAddressAndPort fakeNodeAddressAndPort = getLoopbackAddressWithPort(nodeId);
fakeSnitch.nodeByRack.put(fakeNodeAddressAndPort, rackId);
// todo:
//fakeMetadata.updateTopology(fakeNodeAddressAndPort);
// Allocate tokens
allocation.addNodeToMetadata(fakeNodeAddressAndPort, new Location(LOCATION.datacenter, rackId.toString()));
Collection<Token> tokens = allocation.allocate(fakeNodeAddressAndPort);
// Validate ownership stats
@ -145,7 +145,7 @@ public class OfflineTokenAllocator
private void validateAllocation(int nodeId, int rackId)
{
SummaryStatistics newOwnership = allocation.getAllocationRingOwnership(SimpleSnitch.DATA_CENTER_NAME, Integer.toString(rackId));
SummaryStatistics newOwnership = allocation.getAllocationRingOwnership(LOCATION.datacenter, Integer.toString(rackId));
SummaryStatistics oldOwnership = lastCheckPoint.put(rackId, newOwnership);
if (oldOwnership != null)
logger.debug(String.format("Replicated node load in rack=%d before allocating node %d: %s.", rackId, nodeId,
@ -164,17 +164,6 @@ public class OfflineTokenAllocator
}
}
private static class FakeSnitch extends SimpleSnitch
{
final Map<InetAddressAndPort, Integer> nodeByRack = new HashMap<>();
@Override
public String getRack(InetAddressAndPort endpoint)
{
return Integer.toString(nodeByRack.get(endpoint));
}
}
private static InetAddressAndPort getLoopbackAddressWithPort(int port)
{
try

73
src/java/org/apache/cassandra/dht/tokenallocator/TokenAllocation.java
View File

@ -24,7 +24,9 @@ import java.util.List;
import java.util.Map;
import java.util.NavigableMap;
import java.util.TreeMap;
import java.util.function.Supplier;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import com.google.common.collect.Multimap;
@ -34,23 +36,25 @@ import org.apache.commons.math3.stat.descriptive.SummaryStatistics;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.dht.Token;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.locator.AbstractReplicationStrategy;
import org.apache.cassandra.locator.IEndpointSnitch;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.locator.Locator;
import org.apache.cassandra.locator.NetworkTopologyStrategy;
import org.apache.cassandra.locator.SimpleStrategy;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.membership.Location;
import org.apache.cassandra.tcm.membership.NodeAddresses;
import org.apache.cassandra.tcm.membership.NodeId;
import org.apache.cassandra.tcm.membership.NodeVersion;
public class TokenAllocation
{
public static final double WARN_STDEV_GROWTH = 0.05;
private static final Logger logger = LoggerFactory.getLogger(TokenAllocation.class);
final ClusterMetadata metadata;
ClusterMetadata metadata;
final AbstractReplicationStrategy replicationStrategy;
final int numTokens;
final Map<String, Map<String, StrategyAdapter>> strategyByRackDc = new HashMap<>();
@ -75,14 +79,14 @@ public class TokenAllocation
final InetAddressAndPort endpoint,
int numTokens)
{
return create(DatabaseDescriptor.getEndpointSnitch(), metadata, replicas, numTokens).allocate(endpoint);
return create(metadata.locator.local().datacenter, metadata, replicas, numTokens).allocate(endpoint);
}
static TokenAllocation create(IEndpointSnitch snitch, ClusterMetadata metadata, int replicas, int numTokens)
static TokenAllocation create(String localDatacenter, ClusterMetadata metadata, int replicas, int numTokens)
{
// We create a fake NTS replication strategy with the specified RF in the local DC
HashMap<String, String> options = new HashMap<>();
options.put(snitch.getLocalDatacenter(), Integer.toString(replicas));
options.put(localDatacenter, Integer.toString(replicas));
NetworkTopologyStrategy fakeReplicationStrategy = new NetworkTopologyStrategy(null, options);
return new TokenAllocation(metadata, fakeReplicationStrategy, numTokens);
@ -93,6 +97,23 @@ public class TokenAllocation
return new TokenAllocation(metadata, rs, numTokens);
}
@VisibleForTesting
void updateTokensForNode(NodeId id, Collection<Token> tokens)
{
metadata = metadata.transformer()
.proposeToken(id, tokens)
.addToRackAndDC(id) // needed by NetworkTopologyStrategy
.build().metadata;
}
// For use by OfflineTokenAllocator
void addNodeToMetadata(InetAddressAndPort endpoint, Location location)
{
metadata = metadata.transformer()
.register(new NodeAddresses(endpoint), location, NodeVersion.CURRENT)
.build().metadata;
}
Collection<Token> allocate(InetAddressAndPort endpoint)
{
StrategyAdapter strategy = getOrCreateStrategy(endpoint);
@ -100,6 +121,8 @@ public class TokenAllocation
tokens = strategy.adjustForCrossDatacenterClashes(tokens);
SummaryStatistics os = strategy.replicatedOwnershipStats();
NodeId nodeId = metadata.directory.peerId(endpoint);
updateTokensForNode(nodeId, tokens);
SummaryStatistics ns = strategy.replicatedOwnershipStats();
logger.info("Selected tokens {}", tokens);
@ -136,7 +159,7 @@ public class TokenAllocation
// return true iff the provided endpoint occurs in the same virtual token-ring we are allocating for
// i.e. the set of the nodes that share ownership with the node we are allocating
// alternatively: return false if the endpoint's ownership is independent of the node we are allocating tokens for
abstract boolean inAllocationRing(InetAddressAndPort other);
abstract boolean inAllocationRing(Locator locator, InetAddressAndPort other);
final TokenAllocator<InetAddressAndPort> createAllocator()
{
@ -145,7 +168,7 @@ public class TokenAllocation
for (Map.Entry<Token, NodeId> en : metadata.tokenMap.asMap().entrySet())
{
InetAddressAndPort endpoint = metadata.directory.endpoint(en.getValue());
if (inAllocationRing(endpoint))
if (inAllocationRing(metadata.locator, endpoint))
sortedTokens.put(en.getKey(), endpoint);
}
return TokenAllocatorFactory.createTokenAllocator(sortedTokens, this, metadata.tokenMap.partitioner());
@ -161,7 +184,7 @@ public class TokenAllocation
{
NodeId nodeId = metadata.tokenMap.owner(t);
InetAddressAndPort other = metadata.directory.endpoint(nodeId);
if (inAllocationRing(other))
if (inAllocationRing(metadata.locator, other))
throw new ConfigurationException(String.format("Allocated token %s already assigned to node %s. Is another node also allocating tokens?", t, other));
t = t.nextValidToken();
}
@ -176,7 +199,7 @@ public class TokenAllocation
for (Map.Entry<InetAddressAndPort, Double> en : evaluateReplicatedOwnership().entrySet())
{
// Filter only in the same allocation ring
if (inAllocationRing(en.getKey()))
if (inAllocationRing(metadata.locator, en.getKey()))
{
NodeId nodeId = metadata.directory.peerId(en.getKey());
stat.addValue(en.getValue() / metadata.tokenMap.tokens(nodeId).size());
@ -220,10 +243,8 @@ public class TokenAllocation
private StrategyAdapter getOrCreateStrategy(InetAddressAndPort endpoint)
{
IEndpointSnitch snitch = DatabaseDescriptor.getEndpointSnitch();
String dc = snitch.getDatacenter(endpoint);
String rack = snitch.getRack(endpoint);
return getOrCreateStrategy(dc, rack);
Location location = metadata.locator.location(endpoint);
return getOrCreateStrategy(location.datacenter, location.rack);
}
private StrategyAdapter getOrCreateStrategy(String dc, String rack)
@ -236,13 +257,13 @@ public class TokenAllocation
if (replicationStrategy instanceof NetworkTopologyStrategy)
return createStrategy(metadata, (NetworkTopologyStrategy) replicationStrategy, dc, rack);
if (replicationStrategy instanceof SimpleStrategy)
return createStrategy((SimpleStrategy) replicationStrategy);
return createStrategy(metadata, (SimpleStrategy) replicationStrategy);
throw new ConfigurationException("Token allocation does not support replication strategy " + replicationStrategy.getClass().getSimpleName());
}
private StrategyAdapter createStrategy(final SimpleStrategy rs)
private StrategyAdapter createStrategy(ClusterMetadata metadata, final SimpleStrategy rs)
{
return createStrategy(DatabaseDescriptor.getEndpointSnitch(), null, null, rs.getReplicationFactor().allReplicas, false);
return createStrategy(() -> metadata.locator, null, null, rs.getReplicationFactor().allReplicas, false);
}
private StrategyAdapter createStrategy(ClusterMetadata metadata, NetworkTopologyStrategy strategy, String dc, String rack)
@ -251,6 +272,7 @@ public class TokenAllocation
// if topology hasn't been setup yet for this dc+rack then treat it as a separate unit
Multimap<String, InetAddressAndPort> datacenterRacks = metadata.directory.datacenterRacks(dc);
Supplier<Locator> locator = () -> metadata.locator;
int racks = datacenterRacks != null && datacenterRacks.containsKey(rack)
? datacenterRacks.asMap().size()
: 1;
@ -258,21 +280,21 @@ public class TokenAllocation
if (replicas <= 1)
{
// each node is treated as separate and replicates once
return createStrategy(DatabaseDescriptor.getEndpointSnitch(), dc, null, 1, false);
return createStrategy(locator, dc, null, 1, false);
}
else if (racks == replicas)
{
// each node is treated as separate and replicates once, with separate allocation rings for each rack
return createStrategy(DatabaseDescriptor.getEndpointSnitch(), dc, rack, 1, false);
return createStrategy(locator, dc, rack, 1, false);
}
else if (racks > replicas)
{
// group by rack
return createStrategy(DatabaseDescriptor.getEndpointSnitch(), dc, null, replicas, true);
return createStrategy(locator, dc, null, replicas, true);
}
else if (racks == 1)
{
return createStrategy(DatabaseDescriptor.getEndpointSnitch(), dc, null, replicas, false);
return createStrategy(locator, dc, null, replicas, false);
}
throw new ConfigurationException(String.format("Token allocation failed: the number of racks %d in datacenter %s is lower than its replication factor %d.",
@ -281,7 +303,7 @@ public class TokenAllocation
// a null dc will always return true for inAllocationRing(..)
// a null rack will return true for inAllocationRing(..) for all nodes in the same dc
private StrategyAdapter createStrategy(IEndpointSnitch snitch, String dc, String rack, int replicas, boolean groupByRack)
private StrategyAdapter createStrategy(Supplier<Locator> locator, String dc, String rack, int replicas, boolean groupByRack)
{
return new StrategyAdapter()
{
@ -294,13 +316,14 @@ public class TokenAllocation
@Override
public Object getGroup(InetAddressAndPort unit)
{
return groupByRack ? snitch.getRack(unit) : unit;
return groupByRack ? locator.get().location(unit).rack : unit;
}
@Override
public boolean inAllocationRing(InetAddressAndPort other)
public boolean inAllocationRing(Locator locator, InetAddressAndPort other)
{
return (dc == null || dc.equals(snitch.getDatacenter(other))) && (rack == null || rack.equals(snitch.getRack(other)));
Location location = locator.location(other);
return (dc == null || dc.equals(location.datacenter)) && (rack == null || rack.equals(location.rack));
}
};
}

5
src/java/org/apache/cassandra/gms/Gossiper.java
View File

@ -1648,13 +1648,12 @@ public class Gossiper implements IFailureDetectionEventListener, GossiperMBean,
buildSeedsList();
/* initialize the heartbeat state for this localEndpoint */
maybeInitializeLocalState(generationNbr);
register(DatabaseDescriptor.getLocalAddressReconnectionHelper());
ClusterMetadata metadata = ClusterMetadata.current();
if (mergeLocalStates && metadata.myNodeId() != null)
mergeNodeToGossip(metadata.myNodeId(), metadata);
//notify snitches that Gossiper is about to start
DatabaseDescriptor.getEndpointSnitch().gossiperStarting();
shutdownAnnounced.set(false);
scheduledGossipTask = executor.scheduleWithFixedDelay(new GossipTask(),
Gossiper.intervalInMillis,

8
src/java/org/apache/cassandra/locator/AbstractCloudMetadataServiceConnector.java
View File

@ -36,11 +36,11 @@ import org.apache.cassandra.exceptions.ConfigurationException;
import static java.lang.String.format;
import static java.nio.charset.StandardCharsets.UTF_8;
abstract class AbstractCloudMetadataServiceConnector
public abstract class AbstractCloudMetadataServiceConnector
{
static final String METADATA_URL_PROPERTY = "metadata_url";
static final String METADATA_REQUEST_TIMEOUT_PROPERTY = "metadata_request_timeout";
static final String DEFAULT_METADATA_REQUEST_TIMEOUT = "30s";
public static final String METADATA_URL_PROPERTY = "metadata_url";
public static final String METADATA_REQUEST_TIMEOUT_PROPERTY = "metadata_request_timeout";
public static final String DEFAULT_METADATA_REQUEST_TIMEOUT = "30s";
protected final String metadataServiceUrl;
protected final int requestTimeoutMs;

60
src/java/org/apache/cassandra/locator/AbstractCloudMetadataServiceSnitch.java
View File

@ -23,70 +23,32 @@ import java.util.Map;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.membership.NodeId;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.Pair;
import static java.lang.String.format;
/**
* @deprecated See CASSANDRA-19488
*/
@Deprecated(since = "CEP-21")
abstract class AbstractCloudMetadataServiceSnitch extends AbstractNetworkTopologySnitch
{
static final Logger logger = LoggerFactory.getLogger(AbstractCloudMetadataServiceSnitch.class);
static final String DEFAULT_DC = "UNKNOWN-DC";
static final String DEFAULT_RACK = "UNKNOWN-RACK";
protected final AbstractCloudMetadataServiceConnector connector;
private final String localRack;
private final String localDc;
protected final CloudMetadataLocationProvider locationProvider;
private Map<InetAddressAndPort, Map<String, String>> savedEndpoints;
public AbstractCloudMetadataServiceSnitch(AbstractCloudMetadataServiceConnector connector, Pair<String, String> dcAndRack)
public AbstractCloudMetadataServiceSnitch(CloudMetadataLocationProvider locationProvider)
{
this.connector = connector;
this.localDc = dcAndRack.left;
this.localRack = dcAndRack.right;
logger.info(format("%s using datacenter: %s, rack: %s, connector: %s, properties: %s",
getClass().getName(), getLocalDatacenter(), getLocalRack(), connector, connector.getProperties()));
this.locationProvider = locationProvider;
}
@Override
public final String getLocalRack()
public String getLocalRack()
{
return localRack;
return locationProvider.initialLocation().rack;
}
@Override
public final String getLocalDatacenter()
public String getLocalDatacenter()
{
return localDc;
}
@Override
public final String getRack(InetAddressAndPort endpoint)
{
if (endpoint.equals(FBUtilities.getBroadcastAddressAndPort()))
return getLocalRack();
ClusterMetadata metadata = ClusterMetadata.current();
NodeId nodeId = metadata.directory.peerId(endpoint);
if (nodeId == null)
return DEFAULT_RACK;
return metadata.directory.location(nodeId).rack;
}
@Override
public final String getDatacenter(InetAddressAndPort endpoint)
{
if (endpoint.equals(FBUtilities.getBroadcastAddressAndPort()))
return getLocalDatacenter();
ClusterMetadata metadata = ClusterMetadata.current();
NodeId nodeId = metadata.directory.peerId(endpoint);
if (nodeId == null)
return DEFAULT_DC;
return metadata.directory.location(nodeId).datacenter;
return locationProvider.initialLocation().datacenter;
}
}

25
src/java/org/apache/cassandra/locator/AbstractEndpointSnitch.java
View File

@ -18,12 +18,30 @@
package org.apache.cassandra.locator;
import com.google.common.collect.Iterables;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.tcm.membership.Location;
/**
* @deprecated
*/
@Deprecated(since = "CEP-21")
public abstract class AbstractEndpointSnitch implements IEndpointSnitch
{
public abstract int compareEndpoints(InetAddressAndPort target, Replica r1, Replica r2);
@Override
public String getRack(InetAddressAndPort endpoint)
{
throw new UnsupportedOperationException("IEndpointSnitch has been deprecated and is no longer in use");
}
@Override
public String getDatacenter(InetAddressAndPort endpoint)
{
throw new UnsupportedOperationException("IEndpointSnitch has been deprecated and is no longer in use");
}
/**
* Sorts the <tt>Collection</tt> of node addresses by proximity to the given address
* @param address the address to sort by proximity to
@ -53,7 +71,8 @@ public abstract class AbstractEndpointSnitch implements IEndpointSnitch
private boolean hasRemoteNode(ReplicaCollection<?> l)
{
String localDc = DatabaseDescriptor.getLocalDataCenter();
return Iterables.any(l, replica -> !localDc.equals(getDatacenter(replica)));
Locator locator = DatabaseDescriptor.getLocator();
Location local = locator.local();
return Iterables.any(l, replica -> !local.sameDatacenter(locator.location(replica.endpoint())));
}
}

41
src/java/org/apache/cassandra/locator/AbstractNetworkTopologySnitch.java
View File

@ -20,48 +20,15 @@ package org.apache.cassandra.locator;
/**
* An endpoint snitch tells Cassandra information about network topology that it can use to route
* requests more efficiently.
* @deprecated See CASSANDRA-19488
*/
@Deprecated(since = "CEP-21")
public abstract class AbstractNetworkTopologySnitch extends AbstractEndpointSnitch
{
/**
* Return the rack for which an endpoint resides in
* @param endpoint a specified endpoint
* @return string of rack
*/
abstract public String getRack(InetAddressAndPort endpoint);
/**
* Return the data center for which an endpoint resides in
* @param endpoint a specified endpoint
* @return string of data center
*/
abstract public String getDatacenter(InetAddressAndPort endpoint);
private static final NodeProximity proximity = new NetworkTopologyProximity();
@Override
public int compareEndpoints(InetAddressAndPort address, Replica r1, Replica r2)
{
InetAddressAndPort a1 = r1.endpoint();
InetAddressAndPort a2 = r2.endpoint();
if (address.equals(a1) && !address.equals(a2))
return -1;
if (address.equals(a2) && !address.equals(a1))
return 1;
String addressDatacenter = getDatacenter(address);
String a1Datacenter = getDatacenter(a1);
String a2Datacenter = getDatacenter(a2);
if (addressDatacenter.equals(a1Datacenter) && !addressDatacenter.equals(a2Datacenter))
return -1;
if (addressDatacenter.equals(a2Datacenter) && !addressDatacenter.equals(a1Datacenter))
return 1;
String addressRack = getRack(address);
String a1Rack = getRack(a1);
String a2Rack = getRack(a2);
if (addressRack.equals(a1Rack) && !addressRack.equals(a2Rack))
return -1;
if (addressRack.equals(a2Rack) && !addressRack.equals(a1Rack))
return 1;
return 0;
return proximity.compareEndpoints(address, r1, r2);
}
}

50
src/java/org/apache/cassandra/locator/AlibabaCloudLocationProvider.java Normal file
View File

@ -0,0 +1,50 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import java.io.IOException;
import org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector.DefaultCloudMetadataServiceConnector;
import static org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector.METADATA_URL_PROPERTY;
public class AlibabaCloudLocationProvider extends CloudMetadataLocationProvider
{
static final String DEFAULT_METADATA_SERVICE_URL = "http://100.100.100.200";
static final String ZONE_NAME_QUERY_URL = "/latest/meta-data/zone-id";
/**
* Used via reflection by DatabaseDescriptor::createInitialLocationProvider
*/
public AlibabaCloudLocationProvider() throws IOException
{
this(new SnitchProperties());
}
public AlibabaCloudLocationProvider(SnitchProperties properties) throws IOException
{
this(new DefaultCloudMetadataServiceConnector(properties.putIfAbsent(METADATA_URL_PROPERTY,
DEFAULT_METADATA_SERVICE_URL)));
}
public AlibabaCloudLocationProvider(AbstractCloudMetadataServiceConnector connector) throws IOException
{
super(connector, c -> SnitchUtils.parseLocation(c.apiCall(ZONE_NAME_QUERY_URL), c.getProperties().getDcSuffix()));
}
}

9
src/java/org/apache/cassandra/locator/AlibabaCloudSnitch.java
View File

@ -22,6 +22,7 @@ import java.io.IOException;
import org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector.DefaultCloudMetadataServiceConnector;
import static org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector.METADATA_URL_PROPERTY;
import static org.apache.cassandra.locator.AlibabaCloudLocationProvider.DEFAULT_METADATA_SERVICE_URL;
/**
* A snitch that assumes an ECS region is a DC and an ECS availability_zone
@ -29,12 +30,11 @@ import static org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector
* format of the zone-id is like 'cn-hangzhou-a' where cn means china, hangzhou
* means the hangzhou region, a means the az id. We use 'cn-hangzhou' as the dc,
* and 'a' as the zone-id.
* @deprecated See CASSANDRA-19488
*/
@Deprecated(since = "CEP-21")
public class AlibabaCloudSnitch extends AbstractCloudMetadataServiceSnitch
{
static final String DEFAULT_METADATA_SERVICE_URL = "http://100.100.100.200";
static final String ZONE_NAME_QUERY_URL = "/latest/meta-data/zone-id";
public AlibabaCloudSnitch() throws IOException
{
this(new SnitchProperties());
@ -48,7 +48,6 @@ public class AlibabaCloudSnitch extends AbstractCloudMetadataServiceSnitch
public AlibabaCloudSnitch(AbstractCloudMetadataServiceConnector connector) throws IOException
{
super(connector, SnitchUtils.parseDcAndRack(connector.apiCall(ZONE_NAME_QUERY_URL),
connector.getProperties().getDcSuffix()));
super(new AlibabaCloudLocationProvider(connector));
}
}

100
src/java/org/apache/cassandra/locator/AzureCloudLocationProvider.java Normal file
View File

@ -0,0 +1,100 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import java.io.IOException;
import com.google.common.collect.ImmutableMap;
import com.fasterxml.jackson.databind.JsonNode;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector.DefaultCloudMetadataServiceConnector;
import org.apache.cassandra.tcm.membership.Location;
import org.apache.cassandra.utils.JsonUtils;
import static java.lang.String.format;
import static org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector.METADATA_URL_PROPERTY;
public class AzureCloudLocationProvider extends CloudMetadataLocationProvider
{
static final String DEFAULT_METADATA_SERVICE_URL = "http://169.254.169.254";
static final String METADATA_QUERY_TEMPLATE = "/metadata/instance/compute?api-version=%s&format=json";
static final String METADATA_HEADER = "Metadata";
static final String API_VERSION_PROPERTY_KEY = "azure_api_version";
static final String DEFAULT_API_VERSION = "2021-12-13";
/**
* Used via reflection by DatabaseDescriptor::createInitialLocationProvider
*/
public AzureCloudLocationProvider() throws IOException
{
this(new SnitchProperties());
}
public AzureCloudLocationProvider(SnitchProperties properties) throws IOException
{
this(new DefaultCloudMetadataServiceConnector(properties.putIfAbsent(METADATA_URL_PROPERTY,
DEFAULT_METADATA_SERVICE_URL)));
}
public AzureCloudLocationProvider(AbstractCloudMetadataServiceConnector connector) throws IOException
{
super(connector, AzureCloudLocationProvider::resolveLocation);
}
static Location resolveLocation(AbstractCloudMetadataServiceConnector connector) throws IOException
{
String apiVersion = connector.getProperties().get(API_VERSION_PROPERTY_KEY, DEFAULT_API_VERSION);
String response = connector.apiCall(format(METADATA_QUERY_TEMPLATE, apiVersion),
ImmutableMap.of(METADATA_HEADER, "true"));
JsonNode jsonNode = JsonUtils.JSON_OBJECT_MAPPER.readTree(response);
JsonNode location = jsonNode.get("location");
JsonNode zone = jsonNode.get("zone");
JsonNode platformFaultDomain = jsonNode.get("platformFaultDomain");
String datacenter;
String rack;
if (location == null || location.isNull() || location.asText().isEmpty())
throw new ConfigurationException("Unable to retrieve initial location from cloud metadata service. " +
"This is required for registration, please check configuration");
else
datacenter = location.asText();
if (zone == null || zone.isNull() || zone.asText().isEmpty())
{
if (platformFaultDomain == null || platformFaultDomain.isNull() || platformFaultDomain.asText().isEmpty())
{
throw new ConfigurationException("Unable to retrieve initial zone or platform fault domain from cloud metadata service. " +
"This is required for registration, please check configuration");
}
else
{
rack = platformFaultDomain.asText();
}
}
else
{
rack = zone.asText();
}
return new Location(datacenter + connector.getProperties().getDcSuffix(), "rack-" + rack);
}
}

54
src/java/org/apache/cassandra/locator/AzureSnitch.java
View File

@ -20,15 +20,10 @@ package org.apache.cassandra.locator;
import java.io.IOException;
import com.google.common.collect.ImmutableMap;
import com.fasterxml.jackson.databind.JsonNode;
import org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector.DefaultCloudMetadataServiceConnector;
import org.apache.cassandra.utils.JsonUtils;
import org.apache.cassandra.utils.Pair;
import static java.lang.String.format;
import static org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector.METADATA_URL_PROPERTY;
import static org.apache.cassandra.locator.AzureCloudLocationProvider.DEFAULT_METADATA_SERVICE_URL;
/**
* AzureSnitch will resolve datacenter and rack by calling {@code /metadata/instance/compute} endpoint returning
@ -38,15 +33,11 @@ import static org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector
* A datacenter is resolved from {@code location} field and a rack is resolved by looking into {@code zone} field first.
* When zone is not set, or it is empty string, it will look into {@code platformFaultDomain} field. Such resolved
* value is prepended by {@code rack-} string.
* @deprecated See CASSANDRA-19488
*/
@Deprecated(since = "CEP-21")
public class AzureSnitch extends AbstractCloudMetadataServiceSnitch
{
static final String DEFAULT_METADATA_SERVICE_URL = "http://169.254.169.254";
static final String METADATA_QUERY_TEMPLATE = "/metadata/instance/compute?api-version=%s&format=json";
static final String METADATA_HEADER = "Metadata";
static final String API_VERSION_PROPERTY_KEY = "azure_api_version";
static final String DEFAULT_API_VERSION = "2021-12-13";
public AzureSnitch() throws IOException
{
this(new SnitchProperties());
@ -60,43 +51,6 @@ public class AzureSnitch extends AbstractCloudMetadataServiceSnitch
public AzureSnitch(AbstractCloudMetadataServiceConnector connector) throws IOException
{
super(connector, resolveDcAndRack(connector));
}
private static Pair<String, String> resolveDcAndRack(AbstractCloudMetadataServiceConnector connector) throws IOException
{
String apiVersion = connector.getProperties().get(API_VERSION_PROPERTY_KEY, DEFAULT_API_VERSION);
String response = connector.apiCall(format(METADATA_QUERY_TEMPLATE, apiVersion), ImmutableMap.of(METADATA_HEADER, "true"));
JsonNode jsonNode = JsonUtils.JSON_OBJECT_MAPPER.readTree(response);
JsonNode location = jsonNode.get("location");
JsonNode zone = jsonNode.get("zone");
JsonNode platformFaultDomain = jsonNode.get("platformFaultDomain");
String datacenter;
String rack;
if (location == null || location.isNull() || location.asText().isEmpty())
datacenter = DEFAULT_DC;
else
datacenter = location.asText();
if (zone == null || zone.isNull() || zone.asText().isEmpty())
{
if (platformFaultDomain == null || platformFaultDomain.isNull() || platformFaultDomain.asText().isEmpty())
{
rack = DEFAULT_RACK;
}
else
{
rack = platformFaultDomain.asText();
}
}
else
{
rack = zone.asText();
}
return Pair.create(datacenter + connector.getProperties().getDcSuffix(), "rack-" + rack);
super(new AzureCloudLocationProvider(connector));
}
}

58
src/java/org/apache/cassandra/locator/BaseProximity.java Normal file
View File

@ -0,0 +1,58 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import com.google.common.collect.Iterables;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.tcm.membership.Location;
public abstract class BaseProximity implements NodeProximity
{
public abstract int compareEndpoints(InetAddressAndPort target, Replica r1, Replica r2);
/**
* Sorts the <tt>Collection</tt> of node addresses by proximity to the given address
* @param address the address to sort by proximity to
* @param unsortedAddress the nodes to sort
* @return a new sorted <tt>List</tt>
*/
public <C extends ReplicaCollection<? extends C>> C sortedByProximity(final InetAddressAndPort address, C unsortedAddress)
{
return unsortedAddress.sorted((r1, r2) -> compareEndpoints(address, r1, r2));
}
public boolean isWorthMergingForRangeQuery(ReplicaCollection<?> merged, ReplicaCollection<?> l1, ReplicaCollection<?> l2)
{
// Querying remote DC is likely to be an order of magnitude slower than
// querying locally, so 2 queries to local nodes is likely to still be
// faster than 1 query involving remote ones
boolean mergedHasRemote = hasRemoteNode(merged);
return mergedHasRemote
? hasRemoteNode(l1) || hasRemoteNode(l2)
: true;
}
private boolean hasRemoteNode(ReplicaCollection<?> l)
{
Locator locator = DatabaseDescriptor.getLocator();
Location local = locator.local();
return Iterables.any(l, replica -> !local.sameDatacenter(locator.location(replica.endpoint())));
}
}

68
src/java/org/apache/cassandra/locator/CloudMetadataLocationProvider.java Normal file
View File

@ -0,0 +1,68 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import java.io.IOException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.tcm.membership.Location;
import static java.lang.String.format;
public class CloudMetadataLocationProvider implements InitialLocationProvider
{
static final Logger logger = LoggerFactory.getLogger(CloudMetadataLocationProvider.class);
protected final AbstractCloudMetadataServiceConnector connector;
private final LocationResolver locationResolver;
private volatile Location location;
public CloudMetadataLocationProvider(AbstractCloudMetadataServiceConnector connector, LocationResolver locationResolver)
{
this.connector = connector;
this.locationResolver = locationResolver;
}
@Override
public final Location initialLocation()
{
if (location == null)
{
try
{
location = locationResolver.resolve(connector);
logger.info(format("%s using datacenter: %s, rack: %s, connector: %s, properties: %s",
getClass().getName(), location.datacenter, location.rack, connector, connector.getProperties()));
}
catch (IOException e)
{
throw new ConfigurationException("Unable to resolve initial location using cloud metadata service connector", e);
}
}
return location;
}
public interface LocationResolver
{
Location resolve(AbstractCloudMetadataServiceConnector connector) throws IOException;
}
}

135
src/java/org/apache/cassandra/locator/CloudstackLocationProvider.java Normal file
View File

@ -0,0 +1,135 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import java.io.BufferedReader;
import java.io.IOException;
import java.net.URI;
import java.util.Properties;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.io.util.File;
import org.apache.cassandra.io.util.FileReader;
import org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector.DefaultCloudMetadataServiceConnector;
import org.apache.cassandra.tcm.membership.Location;
import org.apache.cassandra.utils.JVMStabilityInspector;
import static org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector.METADATA_URL_PROPERTY;
public class CloudstackLocationProvider extends CloudMetadataLocationProvider
{
private static final Logger logger = LoggerFactory.getLogger(CloudstackLocationProvider.class);
static final String ZONE_NAME_QUERY_URI = "/latest/meta-data/availability-zone";
private static final String[] LEASE_FILES =
{
"file:///var/lib/dhcp/dhclient.eth0.leases",
"file:///var/lib/dhclient/dhclient.eth0.leases"
};
/**
* Used via reflection by DatabaseDescriptor::createInitialLocationProvider
*/
public CloudstackLocationProvider() throws IOException
{
this(new SnitchProperties(new Properties()));
}
public CloudstackLocationProvider(SnitchProperties snitchProperties) throws IOException
{
this(new DefaultCloudMetadataServiceConnector(snitchProperties.putIfAbsent(METADATA_URL_PROPERTY, csMetadataEndpoint())));
}
public CloudstackLocationProvider(AbstractCloudMetadataServiceConnector connector) throws IOException
{
super(connector, CloudstackLocationProvider::resolveLocation);
logger.warn("{} is deprecated and not actively maintained. It will be removed in the next " +
"major version of Cassandra.", CloudstackSnitch.class.getName());
}
private static Location resolveLocation(AbstractCloudMetadataServiceConnector connector) throws IOException
{
String zone = connector.apiCall(ZONE_NAME_QUERY_URI);
String[] zoneParts = zone.split("-");
if (zoneParts.length != 3)
throw new ConfigurationException("CloudstackSnitch cannot handle invalid zone format: " + zone);
return new Location(zoneParts[0] + '-' + zoneParts[1], zoneParts[2]);
}
private static String csMetadataEndpoint() throws ConfigurationException
{
for (String lease_uri : LEASE_FILES)
{
try
{
File lease_file = new File(new URI(lease_uri));
if (lease_file.exists())
{
return csEndpointFromLease(lease_file);
}
}
catch (Exception e)
{
JVMStabilityInspector.inspectThrowable(e);
}
}
throw new ConfigurationException("No valid DHCP lease file could be found.");
}
private static String csEndpointFromLease(File lease) throws ConfigurationException
{
String line;
String endpoint = null;
Pattern identifierPattern = Pattern.compile("^[ \t]*option dhcp-server-identifier (.*);$");
try (BufferedReader reader = new BufferedReader(new FileReader(lease)))
{
while ((line = reader.readLine()) != null)
{
Matcher matcher = identifierPattern.matcher(line);
if (matcher.find())
{
endpoint = matcher.group(1);
break;
}
}
}
catch (Exception e)
{
throw new ConfigurationException("CloudstackSnitch cannot access lease file.");
}
if (endpoint == null)
{
throw new ConfigurationException("No metadata server could be found in lease file.");
}
return "http://" + endpoint;
}
}

94
src/java/org/apache/cassandra/locator/CloudstackSnitch.java
View File

@ -17,21 +17,8 @@
*/
package org.apache.cassandra.locator;
import java.io.BufferedReader;
import java.io.IOException;
import java.net.URI;
import java.util.Properties;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.io.util.File;
import org.apache.cassandra.io.util.FileReader;
import org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector.DefaultCloudMetadataServiceConnector;
import org.apache.cassandra.utils.JVMStabilityInspector;
import org.apache.cassandra.utils.Pair;
import static org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector.METADATA_URL_PROPERTY;
/**
* A snitch that assumes a Cloudstack Zone follows the typical convention
@ -45,14 +32,6 @@ import static org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector
@Deprecated(since = "5.0")
public class CloudstackSnitch extends AbstractCloudMetadataServiceSnitch
{
static final String ZONE_NAME_QUERY_URI = "/latest/meta-data/availability-zone";
private static final String[] LEASE_FILES =
{
"file:///var/lib/dhcp/dhclient.eth0.leases",
"file:///var/lib/dhclient/dhclient.eth0.leases"
};
public CloudstackSnitch() throws IOException
{
this(new SnitchProperties(new Properties()));
@ -60,78 +39,15 @@ public class CloudstackSnitch extends AbstractCloudMetadataServiceSnitch
public CloudstackSnitch(SnitchProperties snitchProperties) throws IOException
{
this(new DefaultCloudMetadataServiceConnector(snitchProperties.putIfAbsent(METADATA_URL_PROPERTY, csMetadataEndpoint())));
}
public CloudstackSnitch(AbstractCloudMetadataServiceConnector connector) throws IOException
{
super(connector, resolveDcAndRack(connector));
super( new CloudstackLocationProvider(snitchProperties));
logger.warn("{} is deprecated and not actively maintained. It will be removed in the next " +
"major version of Cassandra.", CloudstackSnitch.class.getName());
}
private static Pair<String, String> resolveDcAndRack(AbstractCloudMetadataServiceConnector connector) throws IOException
public CloudstackSnitch(AbstractCloudMetadataServiceConnector connector) throws IOException
{
String zone = connector.apiCall(ZONE_NAME_QUERY_URI);
String[] zoneParts = zone.split("-");
if (zoneParts.length != 3)
throw new ConfigurationException("CloudstackSnitch cannot handle invalid zone format: " + zone);
return Pair.create(zoneParts[0] + '-' + zoneParts[1], zoneParts[2]);
}
private static String csMetadataEndpoint() throws ConfigurationException
{
for (String lease_uri : LEASE_FILES)
{
try
{
File lease_file = new File(new URI(lease_uri));
if (lease_file.exists())
{
return csEndpointFromLease(lease_file);
}
}
catch (Exception e)
{
JVMStabilityInspector.inspectThrowable(e);
}
}
throw new ConfigurationException("No valid DHCP lease file could be found.");
}
private static String csEndpointFromLease(File lease) throws ConfigurationException
{
String line;
String endpoint = null;
Pattern identifierPattern = Pattern.compile("^[ \t]*option dhcp-server-identifier (.*);$");
try (BufferedReader reader = new BufferedReader(new FileReader(lease)))
{
while ((line = reader.readLine()) != null)
{
Matcher matcher = identifierPattern.matcher(line);
if (matcher.find())
{
endpoint = matcher.group(1);
break;
}
}
}
catch (Exception e)
{
throw new ConfigurationException("CloudstackSnitch cannot access lease file.");
}
if (endpoint == null)
{
throw new ConfigurationException("No metadata server could be found in lease file.");
}
return "http://" + endpoint;
super(new CloudstackLocationProvider(connector));
logger.warn("{} is deprecated and not actively maintained. It will be removed in the next " +
"major version of Cassandra.", CloudstackSnitch.class.getName());
}
}

65
src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
View File

@ -47,8 +47,9 @@ import static org.apache.cassandra.config.CassandraRelevantProperties.IGNORE_DYN
/**
* A dynamic snitch that sorts endpoints by latency with an adapted phi failure detector
* // TODO rename to DynamicNodeProximity or similar - but take care with jmx interface
*/
public class DynamicEndpointSnitch extends AbstractEndpointSnitch implements LatencySubscribers.Subscriber, DynamicEndpointSnitchMBean
public class DynamicEndpointSnitch implements NodeProximity, LatencySubscribers.Subscriber, DynamicEndpointSnitchMBean
{
private static final boolean USE_SEVERITY = !IGNORE_DYNAMIC_SNITCH_SEVERITY.getBoolean();
@ -69,7 +70,7 @@ public class DynamicEndpointSnitch extends AbstractEndpointSnitch implements Lat
private volatile HashMap<InetAddressAndPort, Double> scores = new HashMap<>();
private final ConcurrentHashMap<InetAddressAndPort, ExponentiallyDecayingReservoir> samples = new ConcurrentHashMap<>();
public final IEndpointSnitch subsnitch;
public final NodeProximity delegate;
private volatile ScheduledFuture<?> updateSchedular;
private volatile ScheduledFuture<?> resetSchedular;
@ -77,33 +78,21 @@ public class DynamicEndpointSnitch extends AbstractEndpointSnitch implements Lat
private final Runnable update;
private final Runnable reset;
public DynamicEndpointSnitch(IEndpointSnitch snitch)
public DynamicEndpointSnitch(NodeProximity delegate)
{
this(snitch, null);
this(delegate, null);
}
public DynamicEndpointSnitch(IEndpointSnitch snitch, String instance)
public DynamicEndpointSnitch(NodeProximity delegate, String instance)
{
mbeanName = "org.apache.cassandra.db:type=DynamicEndpointSnitch";
if (instance != null)
mbeanName += ",instance=" + instance;
subsnitch = snitch;
update = new Runnable()
{
public void run()
{
updateScores();
}
};
reset = new Runnable()
{
public void run()
{
// we do this so that a host considered bad has a chance to recover, otherwise would we never try
// to read from it, which would cause its score to never change
reset();
}
};
this.delegate = delegate;
update = this::updateScores;
// we do this so that a host considered bad has a chance to recover, otherwise would we never try
// to read from it, which would cause its score to never change
reset = this::reset;
if (DatabaseDescriptor.isDaemonInitialized())
{
@ -155,22 +144,6 @@ public class DynamicEndpointSnitch extends AbstractEndpointSnitch implements Lat
MBeanWrapper.instance.unregisterMBean(mbeanName);
}
@Override
public void gossiperStarting()
{
subsnitch.gossiperStarting();
}
public String getRack(InetAddressAndPort endpoint)
{
return subsnitch.getRack(endpoint);
}
public String getDatacenter(InetAddressAndPort endpoint)
{
return subsnitch.getDatacenter(endpoint);
}
@Override
public <C extends ReplicaCollection<? extends C>> C sortedByProximity(final InetAddressAndPort address, C unsortedAddresses)
{
@ -196,7 +169,7 @@ public class DynamicEndpointSnitch extends AbstractEndpointSnitch implements Lat
return replicas;
// TODO: avoid copy
replicas = subsnitch.sortedByProximity(address, replicas);
replicas = delegate.sortedByProximity(address, replicas);
HashMap<InetAddressAndPort, Double> scores = this.scores; // Make sure the score don't change in the middle of the loop below
// (which wouldn't really matter here but its cleaner that way).
ArrayList<Double> subsnitchOrderedScores = new ArrayList<>(replicas.size());
@ -250,7 +223,7 @@ public class DynamicEndpointSnitch extends AbstractEndpointSnitch implements Lat
}
if (scored1.equals(scored2))
return subsnitch.compareEndpoints(target, a1, a2);
return delegate.compareEndpoints(target, a1, a2);
if (scored1 < scored2)
return -1;
else
@ -353,7 +326,10 @@ public class DynamicEndpointSnitch extends AbstractEndpointSnitch implements Lat
public String getSubsnitchClassName()
{
return subsnitch.getClass().getName();
Class<?> clazz = delegate.getClass();
if (delegate instanceof SnitchAdapter)
clazz = ((SnitchAdapter)delegate).snitch.getClass();
return clazz.getName();
}
public List<Double> dumpTimings(String hostname) throws UnknownHostException
@ -401,7 +377,7 @@ public class DynamicEndpointSnitch extends AbstractEndpointSnitch implements Lat
public boolean isWorthMergingForRangeQuery(ReplicaCollection<?> merged, ReplicaCollection<?> l1, ReplicaCollection<?> l2)
{
if (!subsnitch.isWorthMergingForRangeQuery(merged, l1, l2))
if (!delegate.isWorthMergingForRangeQuery(merged, l1, l2))
return false;
// skip checking scores in the single-node case
@ -433,9 +409,4 @@ public class DynamicEndpointSnitch extends AbstractEndpointSnitch implements Lat
}
return maxScore;
}
public boolean validate(Set<String> datacenters, Set<String> racks)
{
return subsnitch.validate(datacenters, racks);
}
}

162
src/java/org/apache/cassandra/locator/Ec2LocationProvider.java Normal file
View File

@ -0,0 +1,162 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import java.io.IOException;
import java.util.HashSet;
import java.util.Set;
import com.google.common.annotations.VisibleForTesting;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.membership.Location;
public class Ec2LocationProvider extends CloudMetadataLocationProvider
{
static final String SNITCH_PROP_NAMING_SCHEME = "ec2_naming_scheme";
static final String EC2_NAMING_LEGACY = "legacy";
static final String EC2_NAMING_STANDARD = "standard";
@VisibleForTesting
public static final String ZONE_NAME_QUERY = "/latest/meta-data/placement/availability-zone";
private final boolean usingLegacyNaming;
/**
* Used via reflection by DatabaseDescriptor::createInitialLocationProvider
*/
public Ec2LocationProvider() throws IOException, ConfigurationException
{
this(new SnitchProperties());
}
public Ec2LocationProvider(SnitchProperties props) throws IOException, ConfigurationException
{
this(Ec2MetadataServiceConnector.create(props));
}
Ec2LocationProvider(AbstractCloudMetadataServiceConnector connector) throws IOException
{
super(connector, Ec2LocationProvider::getLocation);
usingLegacyNaming = isUsingLegacyNaming(connector.getProperties());
}
private static Location getLocation(AbstractCloudMetadataServiceConnector connector) throws IOException
{
// if using the full naming scheme, region name is created by removing letters from the
// end of the availability zone and zone is the full zone name
boolean usingLegacyNaming = isUsingLegacyNaming(connector.getProperties());
String az = connector.apiCall(ZONE_NAME_QUERY);
String region;
String localDc;
String localRack;
if (usingLegacyNaming)
{
// Split "us-east-1a" or "asia-1a" into "us-east"/"1a" and "asia"/"1a".
String[] splits = az.split("-");
localRack = splits[splits.length - 1];
// hack for CASSANDRA-4026
region = az.substring(0, az.length() - 1);
if (region.endsWith("1"))
region = az.substring(0, az.length() - 3);
}
else
{
// grab the region name, which is embedded in the availability zone name.
// thus an AZ of "us-east-1a" yields the region name "us-east-1"
region = az.replaceFirst("[a-z]+$","");
localRack = az;
}
localDc = region.concat(connector.getProperties().getDcSuffix());
return new Location(localDc, localRack);
}
private static boolean isUsingLegacyNaming(SnitchProperties props)
{
return props.get(SNITCH_PROP_NAMING_SCHEME, EC2_NAMING_STANDARD).equalsIgnoreCase(EC2_NAMING_LEGACY);
}
public void validate(ClusterMetadata metadata)
{
// Validate that the settings here match what was used to locate
// and register other nodes in the cluster (if there are any)
Set<String> datacenters = metadata.directory.allDatacenterRacks().keySet();
Set<String> racks = new HashSet<>();
for (String dc : datacenters)
racks.addAll(metadata.directory.datacenterRacks(dc).keySet());
validate(datacenters, racks);
}
public boolean validate(Set<String> datacenters, Set<String> racks)
{
boolean valid = true;
for (String dc : datacenters)
{
// predicated on the late-2017 AWS naming 'convention' that all region names end with a digit.
// Unfortunately, life isn't that simple. Since we allow custom datacenter suffixes (CASSANDRA-5155),
// an operator could conceiveably make the suffix "a", and thus create a region name that looks just like
// one of the region's availability zones. (for example, "us-east-1a").
// Further, we can't make any assumptions of what that suffix might be by looking at this node's
// datacenterSuffix as conceivably their could be many different suffixes in play for a given region.
//
// It is impossible to distinguish standard and legacy names for datacenters in some cases
// as the format didn't change for some regions (us-west-2 for example).
// We can still identify as legacy the dc names without a number as a suffix like us-east"
boolean dcUsesLegacyFormat = dc.matches("^[a-z]+-[a-z]+$");
if (dcUsesLegacyFormat && !usingLegacyNaming)
{
valid = false;
break;
}
}
for (String rack : racks)
{
// predicated on late-2017 AWS naming 'convention' that AZs do not have a digit as the first char -
// we had that in our legacy AZ (rack) names. Thus we test to see if the rack is in the legacy format.
//
// NOTE: the allowed custom suffix only applies to datacenter (region) names, not availability zones.
boolean rackUsesLegacyFormat = rack.matches("[\\d][a-z]");
if (rackUsesLegacyFormat != usingLegacyNaming)
{
valid = false;
break;
}
}
if (!valid)
{
throw new ConfigurationException(String.format("This ec2-enabled location provider appears to be using the " +
"%s naming scheme for regions, but existing nodes in cluster " +
"are using the opposite: " +
"region(s) = %s, availability zone(s) = %s. " +
"Please check the %s property in the %s configuration file " +
"for more details.",
usingLegacyNaming ? "legacy" : "standard", datacenters, racks,
SNITCH_PROP_NAMING_SCHEME, SnitchProperties.RACKDC_PROPERTY_FILENAME));
}
return true;
}
}

97
src/java/org/apache/cassandra/locator/Ec2MultiRegionAddressConfig.java Normal file
View File

@ -0,0 +1,97 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import java.io.IOException;
import java.net.InetAddress;
import java.net.UnknownHostException;
import com.google.common.annotations.VisibleForTesting;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.utils.FBUtilities;
public class Ec2MultiRegionAddressConfig implements NodeAddressConfig
{
private static final Logger logger = LoggerFactory.getLogger(Ec2MultiRegionAddressConfig.class);
@VisibleForTesting
public static final String PUBLIC_IP_QUERY = "/latest/meta-data/public-ipv4";
@VisibleForTesting
public static final String PRIVATE_IP_QUERY = "/latest/meta-data/local-ipv4";
private final String localPublicAddress;
private final String localPrivateAddress;
/**
* Used via reflection by DatabaseDescriptor::createAddressConfig
*/
public Ec2MultiRegionAddressConfig() throws IOException
{
this(Ec2MetadataServiceConnector.create(new SnitchProperties()));
}
@VisibleForTesting
public Ec2MultiRegionAddressConfig(AbstractCloudMetadataServiceConnector connector) throws IOException
{
this.localPublicAddress = connector.apiCall(PUBLIC_IP_QUERY);
logger.info("EC2 multi region address config using publicIP as identifier: {}", localPublicAddress);
this.localPrivateAddress = connector.apiCall(PRIVATE_IP_QUERY);
}
@Override
public void configureAddresses()
{
// use the Public IP to broadcast Address to other nodes.
try
{
InetAddress broadcastAddress = InetAddress.getByName(localPublicAddress);
DatabaseDescriptor.setBroadcastAddress(broadcastAddress);
if (DatabaseDescriptor.getBroadcastRpcAddress() == null)
{
logger.info("broadcast_rpc_address unset, broadcasting public IP as rpc_address: {}", localPublicAddress);
DatabaseDescriptor.setBroadcastRpcAddress(broadcastAddress);
}
}
catch (UnknownHostException e)
{
throw new ConfigurationException("Unable to obtain public address for node from cloud metadata service", e);
}
try
{
InetAddress privateAddress = InetAddress.getByName(localPrivateAddress);
FBUtilities.setLocalAddress(privateAddress);
}
catch (UnknownHostException e)
{
throw new ConfigurationException("Unable to obtain private address for node from cloud metadata service", e);
}
}
@Override
public boolean preferLocalConnections()
{
// Always prefer re-connecting on private addresses if in the same datacenter (i.e. region)
return true;
}
}

47
src/java/org/apache/cassandra/locator/Ec2MultiRegionSnitch.java
View File

@ -18,16 +18,10 @@
package org.apache.cassandra.locator;
import java.io.IOException;
import java.net.InetAddress;
import java.net.UnknownHostException;
import com.google.common.annotations.VisibleForTesting;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.gms.ApplicationState;
import org.apache.cassandra.gms.Gossiper;
import org.apache.cassandra.service.StorageService;
/**
* 1) Snitch will automatically set the public IP by querying the AWS API
@ -39,14 +33,17 @@ import org.apache.cassandra.service.StorageService;
*
* Operational: All the nodes in this cluster needs to be able to (modify the
* Security group settings in AWS) communicate via Public IP's.
* @deprecated See CASSANDRA-19488
*/
@Deprecated(since = "CEP-21")
public class Ec2MultiRegionSnitch extends Ec2Snitch
{
private final Ec2MultiRegionAddressConfig addressConfig;
@VisibleForTesting
static final String PUBLIC_IP_QUERY = "/latest/meta-data/public-ipv4";
@VisibleForTesting
static final String PRIVATE_IP_QUERY = "/latest/meta-data/local-ipv4";
private final String localPrivateAddress;
public Ec2MultiRegionSnitch() throws IOException, ConfigurationException
{
@ -58,36 +55,22 @@ public class Ec2MultiRegionSnitch extends Ec2Snitch
this(Ec2MetadataServiceConnector.create(props));
}
Ec2MultiRegionSnitch(AbstractCloudMetadataServiceConnector connector) throws IOException
@VisibleForTesting
public Ec2MultiRegionSnitch(AbstractCloudMetadataServiceConnector connector) throws IOException
{
super(connector);
InetAddress localPublicAddress = InetAddress.getByName(connector.apiCall(PUBLIC_IP_QUERY));
logger.info("EC2Snitch using publicIP as identifier: {}", localPublicAddress);
localPrivateAddress = connector.apiCall(PRIVATE_IP_QUERY);
// use the Public IP to broadcast Address to other nodes.
DatabaseDescriptor.setBroadcastAddress(localPublicAddress);
if (DatabaseDescriptor.getBroadcastRpcAddress() == null)
{
logger.info("broadcast_rpc_address unset, broadcasting public IP as rpc_address: {}", localPublicAddress);
DatabaseDescriptor.setBroadcastRpcAddress(localPublicAddress);
}
addressConfig = new Ec2MultiRegionAddressConfig(connector);
}
@Override
public void gossiperStarting()
public void configureAddresses()
{
super.gossiperStarting();
InetAddressAndPort address;
try
{
address = InetAddressAndPort.getByName(localPrivateAddress);
}
catch (UnknownHostException e)
{
throw new RuntimeException(e);
}
Gossiper.instance.addLocalApplicationState(ApplicationState.INTERNAL_ADDRESS_AND_PORT, StorageService.instance.valueFactory.internalAddressAndPort(address));
Gossiper.instance.addLocalApplicationState(ApplicationState.INTERNAL_IP, StorageService.instance.valueFactory.internalIP(address.getAddress()));
Gossiper.instance.register(new ReconnectableSnitchHelper(this, getLocalDatacenter(), true));
addressConfig.configureAddresses();
}
@Override
public boolean preferLocalConnections()
{
return addressConfig.preferLocalConnections();
}
}

100
src/java/org/apache/cassandra/locator/Ec2Snitch.java
View File

@ -23,7 +23,6 @@ import java.util.Set;
import com.google.common.annotations.VisibleForTesting;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.utils.Pair;
/**
* A snitch that assumes an EC2 region is a DC and an EC2 availability_zone
@ -44,7 +43,9 @@ import org.apache.cassandra.utils.Pair;
* to a user is {@link Ec2MetadataServiceConnector.V2Connector#AWS_EC2_METADATA_TOKEN_TTL_SECONDS_HEADER_PROPERTY}
* which is by default set to {@link Ec2MetadataServiceConnector.V2Connector#MAX_TOKEN_TIME_IN_SECONDS}. TTL has
* to be an integer from the range [30, 21600].
* @deprecated See CASSANDRA-19488
*/
@Deprecated(since = "CEP-21")
public class Ec2Snitch extends AbstractCloudMetadataServiceSnitch
{
private static final String SNITCH_PROP_NAMING_SCHEME = "ec2_naming_scheme";
@ -54,8 +55,6 @@ public class Ec2Snitch extends AbstractCloudMetadataServiceSnitch
@VisibleForTesting
public static final String ZONE_NAME_QUERY = "/latest/meta-data/placement/availability-zone";
private final boolean usingLegacyNaming;
public Ec2Snitch() throws IOException, ConfigurationException
{
this(new SnitchProperties());
@ -68,103 +67,12 @@ public class Ec2Snitch extends AbstractCloudMetadataServiceSnitch
Ec2Snitch(AbstractCloudMetadataServiceConnector connector) throws IOException
{
super(connector, getDcAndRack(connector));
usingLegacyNaming = isUsingLegacyNaming(connector.getProperties());
}
private static Pair<String, String> getDcAndRack(AbstractCloudMetadataServiceConnector connector) throws IOException
{
String az = connector.apiCall(ZONE_NAME_QUERY);
// if using the full naming scheme, region name is created by removing letters from the
// end of the availability zone and zone is the full zone name
boolean usingLegacyNaming = isUsingLegacyNaming(connector.getProperties());
String region;
String localDc;
String localRack;
if (usingLegacyNaming)
{
// Split "us-east-1a" or "asia-1a" into "us-east"/"1a" and "asia"/"1a".
String[] splits = az.split("-");
localRack = splits[splits.length - 1];
// hack for CASSANDRA-4026
region = az.substring(0, az.length() - 1);
if (region.endsWith("1"))
region = az.substring(0, az.length() - 3);
}
else
{
// grab the region name, which is embedded in the availability zone name.
// thus an AZ of "us-east-1a" yields the region name "us-east-1"
region = az.replaceFirst("[a-z]+$","");
localRack = az;
}
localDc = region.concat(connector.getProperties().getDcSuffix());
return Pair.create(localDc, localRack);
}
private static boolean isUsingLegacyNaming(SnitchProperties props)
{
return props.get(SNITCH_PROP_NAMING_SCHEME, EC2_NAMING_STANDARD).equalsIgnoreCase(EC2_NAMING_LEGACY);
super(new Ec2LocationProvider(connector));
}
@Override
public boolean validate(Set<String> datacenters, Set<String> racks)
{
return validate(datacenters, racks, usingLegacyNaming);
}
@VisibleForTesting
static boolean validate(Set<String> datacenters, Set<String> racks, boolean usingLegacyNaming)
{
boolean valid = true;
for (String dc : datacenters)
{
// predicated on the late-2017 AWS naming 'convention' that all region names end with a digit.
// Unfortunately, life isn't that simple. Since we allow custom datacenter suffixes (CASSANDRA-5155),
// an operator could conceiveably make the suffix "a", and thus create a region name that looks just like
// one of the region's availability zones. (for example, "us-east-1a").
// Further, we can't make any assumptions of what that suffix might be by looking at this node's
// datacenterSuffix as conceivably their could be many different suffixes in play for a given region.
//
// It is impossible to distinguish standard and legacy names for datacenters in some cases
// as the format didn't change for some regions (us-west-2 for example).
// We can still identify as legacy the dc names without a number as a suffix like us-east"
boolean dcUsesLegacyFormat = dc.matches("^[a-z]+-[a-z]+$");
if (dcUsesLegacyFormat && !usingLegacyNaming)
{
valid = false;
break;
}
}
for (String rack : racks)
{
// predicated on late-2017 AWS naming 'convention' that AZs do not have a digit as the first char -
// we had that in our legacy AZ (rack) names. Thus we test to see if the rack is in the legacy format.
//
// NOTE: the allowed custom suffix only applies to datacenter (region) names, not availability zones.
boolean rackUsesLegacyFormat = rack.matches("[\\d][a-z]");
if (rackUsesLegacyFormat != usingLegacyNaming)
{
valid = false;
break;
}
}
if (!valid)
{
logger.error("This ec2-enabled snitch appears to be using the {} naming scheme for regions, " +
"but existing nodes in cluster are using the opposite: region(s) = {}, availability zone(s) = {}. " +
"Please check the {} property in the {} configuration file for more details.",
usingLegacyNaming ? "legacy" : "standard", datacenters, racks,
SNITCH_PROP_NAMING_SCHEME, SnitchProperties.RACKDC_PROPERTY_FILENAME);
}
return valid;
return ((Ec2LocationProvider)locationProvider).validate(datacenters, racks);
}
}

13
src/java/org/apache/cassandra/locator/EndpointSnitchInfo.java
View File

@ -22,6 +22,7 @@ import java.net.UnknownHostException;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.utils.MBeanWrapper;
@Deprecated(since = "CEP-21")
public class EndpointSnitchInfo implements EndpointSnitchInfoMBean
{
public static void create()
@ -31,26 +32,28 @@ public class EndpointSnitchInfo implements EndpointSnitchInfoMBean
public String getDatacenter(String host) throws UnknownHostException
{
return DatabaseDescriptor.getEndpointSnitch().getDatacenter(InetAddressAndPort.getByName(host));
return DatabaseDescriptor.getLocator().location(InetAddressAndPort.getByName(host)).datacenter;
}
public String getRack(String host) throws UnknownHostException
{
return DatabaseDescriptor.getEndpointSnitch().getRack(InetAddressAndPort.getByName(host));
return DatabaseDescriptor.getLocator().location(InetAddressAndPort.getByName(host)).rack;
}
public String getDatacenter()
{
return DatabaseDescriptor.getEndpointSnitch().getLocalDatacenter();
return DatabaseDescriptor.getLocator().local().datacenter;
}
public String getRack()
{
return DatabaseDescriptor.getEndpointSnitch().getLocalRack();
return DatabaseDescriptor.getLocator().local().rack;
}
public String getSnitchName()
{
return DatabaseDescriptor.getEndpointSnitch().getClass().getName();
NodeProximity proximity = DatabaseDescriptor.getNodeProximity();
Class<?> clazz = proximity instanceof SnitchAdapter ? ((SnitchAdapter)proximity).snitch.getClass() : proximity.getClass();
return clazz.getName();
}
}

3
src/java/org/apache/cassandra/locator/EndpointSnitchInfoMBean.java
View File

@ -21,7 +21,10 @@ import java.net.UnknownHostException;
/**
* MBean exposing standard Snitch info
*
* @deprecated see CASSANDRA-19488
*/
@Deprecated(since = "CEP-21")
public interface EndpointSnitchInfoMBean
{
/**

54
src/java/org/apache/cassandra/locator/GoogleCloudLocationProvider.java Normal file
View File

@ -0,0 +1,54 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import java.io.IOException;
import com.google.common.collect.ImmutableMap;
import org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector.DefaultCloudMetadataServiceConnector;
import static org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector.METADATA_URL_PROPERTY;
public class GoogleCloudLocationProvider extends CloudMetadataLocationProvider
{
static final String DEFAULT_METADATA_SERVICE_URL = "http://metadata.google.internal";
static final String ZONE_NAME_QUERY_URL = "/computeMetadata/v1/instance/zone";
static final ImmutableMap<String, String> HEADERS = ImmutableMap.of("Metadata-Flavor", "Google");
/**
* Used via reflection by DatabaseDescriptor::createInitialLocationProvider
*/
public GoogleCloudLocationProvider() throws IOException
{
this(new SnitchProperties());
}
public GoogleCloudLocationProvider(SnitchProperties properties) throws IOException
{
this(new DefaultCloudMetadataServiceConnector(properties.putIfAbsent(METADATA_URL_PROPERTY,
DEFAULT_METADATA_SERVICE_URL)));
}
public GoogleCloudLocationProvider(AbstractCloudMetadataServiceConnector connector) throws IOException
{
super(connector, c -> SnitchUtils.parseLocation(c.apiCall(ZONE_NAME_QUERY_URL, HEADERS),
c.getProperties().getDcSuffix()));
}
}

18
src/java/org/apache/cassandra/locator/GoogleCloudSnitch.java
View File

@ -19,21 +19,14 @@ package org.apache.cassandra.locator;
import java.io.IOException;
import com.google.common.collect.ImmutableMap;
import org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector.DefaultCloudMetadataServiceConnector;
import static org.apache.cassandra.locator.AbstractCloudMetadataServiceConnector.METADATA_URL_PROPERTY;
/**
* A snitch that assumes an GCE region is a DC and an GCE availability_zone
* is a rack. This information is available in the config for the node.
* @deprecated See CASSANDRA-19488
*/
@Deprecated(since = "CEP-21")
public class GoogleCloudSnitch extends AbstractCloudMetadataServiceSnitch
{
static final String DEFAULT_METADATA_SERVICE_URL = "http://metadata.google.internal";
static final String ZONE_NAME_QUERY_URL = "/computeMetadata/v1/instance/zone";
public GoogleCloudSnitch() throws IOException
{
this(new SnitchProperties());
@ -41,14 +34,11 @@ public class GoogleCloudSnitch extends AbstractCloudMetadataServiceSnitch
public GoogleCloudSnitch(SnitchProperties properties) throws IOException
{
this(new DefaultCloudMetadataServiceConnector(properties.putIfAbsent(METADATA_URL_PROPERTY,
DEFAULT_METADATA_SERVICE_URL)));
super(new GoogleCloudLocationProvider(properties));
}
public GoogleCloudSnitch(AbstractCloudMetadataServiceConnector connector) throws IOException
{
super(connector, SnitchUtils.parseDcAndRack(connector.apiCall(ZONE_NAME_QUERY_URL,
ImmutableMap.of("Metadata-Flavor", "Google")),
connector.getProperties().getDcSuffix()));
super(new GoogleCloudLocationProvider(connector));
}
}

105
src/java/org/apache/cassandra/locator/GossipingPropertyFileSnitch.java
View File

@ -18,107 +18,40 @@
package org.apache.cassandra.locator;
import java.util.concurrent.atomic.AtomicReference;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.gms.ApplicationState;
import org.apache.cassandra.gms.Gossiper;
import org.apache.cassandra.service.StorageService;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.membership.NodeId;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.tcm.membership.Location;
public class GossipingPropertyFileSnitch extends AbstractNetworkTopologySnitch// implements IEndpointStateChangeSubscriber
/**
* @deprecated See CASSANDRA-19488
*/
@Deprecated(since = "CEP-21")
public class GossipingPropertyFileSnitch extends AbstractNetworkTopologySnitch
{
private static final Logger logger = LoggerFactory.getLogger(GossipingPropertyFileSnitch.class);
private final String myDC;
private final String myRack;
private final boolean preferLocal;
private final AtomicReference<ReconnectableSnitchHelper> snitchHelperReference;
private static final String DEFAULT_DC = "UNKNOWN_DC";
private static final String DEFAULT_RACK = "UNKNOWN_RACK";
private final Location fromConfig;
public final boolean preferLocal;
public GossipingPropertyFileSnitch() throws ConfigurationException
{
SnitchProperties properties = loadConfiguration();
myDC = properties.get("dc", DEFAULT_DC).trim();
myRack = properties.get("rack", DEFAULT_RACK).trim();
SnitchProperties properties = RackDCFileLocationProvider.loadConfiguration();
fromConfig = new RackDCFileLocationProvider(properties).initialLocation();
preferLocal = Boolean.parseBoolean(properties.get("prefer_local", "false"));
snitchHelperReference = new AtomicReference<>();
}
private static SnitchProperties loadConfiguration() throws ConfigurationException
@Override
public String getLocalRack()
{
final SnitchProperties properties = new SnitchProperties();
if (!properties.contains("dc") || !properties.contains("rack"))
throw new ConfigurationException("DC or rack not found in snitch properties, check your configuration in: " + SnitchProperties.RACKDC_PROPERTY_FILENAME);
return properties;
return fromConfig.rack;
}
/**
* Return the data center for which an endpoint resides in
*
* @param endpoint the endpoint to process
* @return string of data center
*/
public String getDatacenter(InetAddressAndPort endpoint)
@Override
public String getLocalDatacenter()
{
if (endpoint.equals(FBUtilities.getBroadcastAddressAndPort()))
return myDC;
ClusterMetadata metadata = ClusterMetadata.current();
NodeId nodeId = metadata.directory.peerId(endpoint);
if (nodeId == null)
return DEFAULT_DC;
return metadata.directory.location(nodeId).datacenter;
return fromConfig.datacenter;
}
/**
* Return the rack for which an endpoint resides in
*
* @param endpoint the endpoint to process
* @return string of rack
*/
public String getRack(InetAddressAndPort endpoint)
@Override
public boolean preferLocalConnections()
{
if (endpoint.equals(FBUtilities.getBroadcastAddressAndPort()))
return myRack;
ClusterMetadata metadata = ClusterMetadata.current();
NodeId nodeId = metadata.directory.peerId(endpoint);
if (nodeId == null)
return DEFAULT_RACK;
return metadata.directory.location(nodeId).rack;
}
public void gossiperStarting()
{
super.gossiperStarting();
Gossiper.instance.addLocalApplicationState(ApplicationState.INTERNAL_ADDRESS_AND_PORT,
StorageService.instance.valueFactory.internalAddressAndPort(FBUtilities.getLocalAddressAndPort()));
Gossiper.instance.addLocalApplicationState(ApplicationState.INTERNAL_IP,
StorageService.instance.valueFactory.internalIP(FBUtilities.getJustLocalAddress()));
loadGossiperState();
}
private void loadGossiperState()
{
assert Gossiper.instance != null;
ReconnectableSnitchHelper pendingHelper = new ReconnectableSnitchHelper(this, myDC, preferLocal);
Gossiper.instance.register(pendingHelper);
pendingHelper = snitchHelperReference.getAndSet(pendingHelper);
if (pendingHelper != null)
Gossiper.instance.unregister(pendingHelper);
return preferLocal;
}
}

10
src/java/org/apache/cassandra/locator/IEndpointSnitch.java
View File

@ -26,8 +26,9 @@ import org.apache.cassandra.utils.FBUtilities;
* This interface helps determine location of node in the datacenter relative to another node.
* Give a node A and another node B it can tell if A and B are on the same rack or in the same
* datacenter.
* @deprecated Replaced by Locator and NodeProximity; see CASSANDRA-19488
*/
@Deprecated(since="CEP-21")
public interface IEndpointSnitch
{
/**
@ -94,4 +95,11 @@ public interface IEndpointSnitch
{
return true;
}
default void configureAddresses() {}
default boolean preferLocalConnections()
{
return false;
}
}

31
src/java/org/apache/cassandra/locator/InOurDc.java
View File

@ -20,8 +20,8 @@ package org.apache.cassandra.locator;
import java.util.function.Predicate;
import static org.apache.cassandra.config.DatabaseDescriptor.getEndpointSnitch;
import static org.apache.cassandra.config.DatabaseDescriptor.getLocalDataCenter;
import static org.apache.cassandra.config.DatabaseDescriptor.getLocator;
public class InOurDc
{
@ -29,49 +29,44 @@ public class InOurDc
private static EndpointTester endpoints;
final String dc;
final IEndpointSnitch snitch;
final Locator locator;
private InOurDc(String dc, IEndpointSnitch snitch)
private InOurDc(String dc, Locator locator)
{
this.dc = dc;
this.snitch = snitch;
this.locator = locator;
}
boolean stale()
{
return dc != getLocalDataCenter()
|| snitch != getEndpointSnitch()
// this final clause checks if somehow the snitch/localDc have got out of whack;
// presently, this is possible but very unlikely, but this check will also help
// resolve races on these global fields as well
|| !dc.equals(snitch.getLocalDatacenter());
return !dc.equals(getLocalDataCenter());
}
private static final class ReplicaTester extends InOurDc implements Predicate<Replica>
{
private ReplicaTester(String dc, IEndpointSnitch snitch)
private ReplicaTester(String dc, Locator locator)
{
super(dc, snitch);
super(dc, locator);
}
@Override
public boolean test(Replica replica)
{
return dc.equals(snitch.getDatacenter(replica.endpoint()));
return dc.equals(locator.location(replica.endpoint()).datacenter);
}
}
private static final class EndpointTester extends InOurDc implements Predicate<InetAddressAndPort>
{
private EndpointTester(String dc, IEndpointSnitch snitch)
private EndpointTester(String dc, Locator locator)
{
super(dc, snitch);
super(dc, locator);
}
@Override
public boolean test(InetAddressAndPort endpoint)
{
return dc.equals(snitch.getDatacenter(endpoint));
return dc.equals(locator.location(endpoint).datacenter);
}
}
@ -79,7 +74,7 @@ public class InOurDc
{
ReplicaTester cur = replicas;
if (cur == null || cur.stale())
replicas = cur = new ReplicaTester(getLocalDataCenter(), getEndpointSnitch());
replicas = cur = new ReplicaTester(getLocalDataCenter(), getLocator());
return cur;
}
@ -87,7 +82,7 @@ public class InOurDc
{
EndpointTester cur = endpoints;
if (cur == null || cur.stale())
endpoints = cur = new EndpointTester(getLocalDataCenter(), getEndpointSnitch());
endpoints = cur = new EndpointTester(getLocalDataCenter(), getLocator());
return cur;
}

44
src/java/org/apache/cassandra/locator/InitialLocationProvider.java Normal file
View File

@ -0,0 +1,44 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.membership.Location;
public interface InitialLocationProvider
{
/**
* Provides the Location with which a new node should register itself with ClusterMetadata.
* After registration, this is no longer necessary and Location is always sourced from
* ClusterMetadata
* @return the datacenter and rack to register with
*/
Location initialLocation();
/**
* Validate that the locations of previously registered peers are considered valid by the locally configured
* InitialLocationProvider. In practice, of the in-tree implementations only Ec2LocationProvider overrides this and
* uses it to ensure that the same ec2 naming scheme is used across all peers.
* See CASSANDRA-7839 for origins.
* @param metadata ClusterMetadata at the time of registering
* @return true if the implementation considers the locations of existing nodes compatible with its own
* configuration, false otherwise
*/
default void validate(ClusterMetadata metadata) {}
}

66
src/java/org/apache/cassandra/locator/LocationInfo.java Normal file
View File

@ -0,0 +1,66 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import java.net.UnknownHostException;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.utils.MBeanWrapper;
public class LocationInfo implements LocationInfoMBean
{
public static void create()
{
MBeanWrapper.instance.registerMBean(new LocationInfo(), "org.apache.cassandra.db:type=LocationInfo");
}
public String getDatacenter(String host) throws UnknownHostException
{
return DatabaseDescriptor.getLocator().location(InetAddressAndPort.getByName(host)).datacenter;
}
public String getRack(String host) throws UnknownHostException
{
return DatabaseDescriptor.getLocator().location(InetAddressAndPort.getByName(host)).rack;
}
public String getDatacenter()
{
return DatabaseDescriptor.getLocator().local().datacenter;
}
public String getRack()
{
return DatabaseDescriptor.getLocator().local().rack;
}
public String getNodeProximityName()
{
NodeProximity proximity = DatabaseDescriptor.getNodeProximity();
Class<?> clazz = proximity instanceof SnitchAdapter ? ((SnitchAdapter)proximity).snitch.getClass() : proximity.getClass();
return clazz.getName();
}
public boolean hasLegacySnitchAdapter()
{
NodeProximity proximity = DatabaseDescriptor.getNodeProximity();
if (proximity instanceof DynamicEndpointSnitch)
return ((DynamicEndpointSnitch)proximity).delegate instanceof SnitchAdapter;
return proximity instanceof SnitchAdapter;
}
}

51
src/java/org/apache/cassandra/locator/LocationInfoMBean.java Normal file
View File

@ -0,0 +1,51 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import java.net.UnknownHostException;
public interface LocationInfoMBean
{
/**
* Provides the Rack name depending on the respective snitch used, given the host name/ip
* @param host
* @throws UnknownHostException
*/
public String getRack(String host) throws UnknownHostException;
/**
* Provides the Datacenter name depending on the respective snitch used, given the hostname/ip
* @param host
* @throws UnknownHostException
*/
public String getDatacenter(String host) throws UnknownHostException;
/**
* Provides the Rack name depending on the respective snitch used for this node
*/
public String getRack();
/**
* Provides the Datacenter name depending on the respective snitch used for this node
*/
public String getDatacenter();
public String getNodeProximityName();
public boolean hasLegacySnitchAdapter();
}

195
src/java/org/apache/cassandra/locator/Locator.java Normal file
View File

@ -0,0 +1,195 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.Epoch;
import org.apache.cassandra.tcm.RegistrationStatus;
import org.apache.cassandra.tcm.membership.Directory;
import org.apache.cassandra.tcm.membership.Location;
import org.apache.cassandra.tcm.membership.NodeId;
import org.apache.cassandra.utils.FBUtilities;
/**
* Provides Location (datacenter & rack) information for endpoints. Usually this is obtained directly
* from ClusterMetadata.directory, using either a specific directory instance or the most current
* published one. During initial startup, location info for the local node may be derived from some
* other source, such as a config file or cloud metadata api. This is then used to register the node
* and its location in ClusterMetadata, which then becomes the ultimate source of truth.
*/
public class Locator
{
private final InetAddressAndPort localEndpoint;
// Indicates whether the *local node* is yet to register itself with ClusterMetadata.
// This is relevant because once a node is registered, it's location is always derived
// from ClusterMetadata. However, before registering the location is obtained from
// configuration. This pre-registration location is what the node supplies to ClusterMetadata
// when registering and is also used to determine DC locality of peers, for instance when
// establishing initial internode connections during the discovery phase.
private final RegistrationStatus state;
// Source of truth for location lookups. This may be null, in which case every lookup will
// use the most up to date version from ClusterMetadata.current().directory
private final Directory directory;
// Supplies the Location for this node only during its initial startup. This location will be
// used to register the node with ClusterMetadata and is not used after that has occurred.
// See DatabaseDescriptor::getInitialLocationProvider
private final InitialLocationProvider locationProvider;
// This is the Location used to register this node during its initial startup. It is lazily initialized
// using the supplied InitialLocationProvider and memoized here as that may be a non-trivial operation.
// Some providers fetch location metadata from remote services etc. It should usually be unnecessary to
// access the initialization location after a node's first startup.
private volatile Location initializationLocation;
// Set from ClusterMetadata. After initial registration has happened, location of this node itself
// is always taken from ClusterMetadata.
private volatile VersionedLocation local;
private static class VersionedLocation
{
final Epoch epoch;
final Location location;
private VersionedLocation(Epoch epoch, Location location)
{
this.epoch = epoch;
this.location = location;
}
}
public static Locator usingDirectory(Directory directory)
{
return new Locator(RegistrationStatus.instance,
FBUtilities.getBroadcastAddressAndPort(),
DatabaseDescriptor.getInitialLocationProvider(),
directory);
}
public static Locator forClients()
{
return new Locator(RegistrationStatus.instance,
FBUtilities.getBroadcastAddressAndPort(),
() -> Location.UNKNOWN,
null);
}
/**
* Creates a Locator instance which always uses the Directory from the most current ClusterMetadata.
* This means that the values returned from {@link this#location(InetAddressAndPort)} can change between
* invocations, if interleaved with the publication of updated cluster metadata.
*/
public Locator(RegistrationStatus state,
InetAddressAndPort localEndpoint,
InitialLocationProvider provider)
{
this(state, localEndpoint, provider, null);
}
/**
* Creates a Locator instance which returns consistent results based on the supplied Directory instance.
* Changes to RegistrationStatus could still have an effect i.e. the node transitioned from UNREGISTERED to
* REGISTERED between two calls to local(), the first would return the Location according to
* DatabaseDescriptor.getInitialLocationProvider, but the second would consult the supplied Directory.
*/
public Locator(RegistrationStatus state,
InetAddressAndPort localEndpoint,
InitialLocationProvider provider,
Directory directory)
{
this.state = state;
this.localEndpoint = localEndpoint;
this.locationProvider = provider;
this.directory = directory;
this.local = new VersionedLocation(Epoch.EMPTY, Location.UNKNOWN);
}
public Location location(InetAddressAndPort endpoint)
{
switch (state.getCurrent())
{
case INITIAL:
return endpoint.equals(localEndpoint) ? initialLocation() : Location.UNKNOWN;
case UNREGISTERED:
return endpoint.equals(localEndpoint) ? initialLocation() : fromDirectory(endpoint);
default:
return fromDirectory(endpoint);
}
}
public Location local()
{
switch (state.getCurrent())
{
case INITIAL:
case UNREGISTERED:
return initialLocation();
default:
// For now, local location is immutable and once registered with cluster metadata, it cannot be
// changed. Revisit this if that assumption changes.
VersionedLocation location = local;
if (location.epoch.isAfter(Epoch.EMPTY))
return location.location;
local = versionedFromDirectory(localEndpoint);
return local.location;
}
}
// The distinction between versioned and unversioned may be removed if/when we allow
// a node's Location to be modified after registration. This duplication should not
// be necessary then.
private VersionedLocation versionedFromDirectory(InetAddressAndPort endpoint)
{
Directory source = directory;
if (source == null)
{
ClusterMetadata metadata = ClusterMetadata.currentNullable();
if (metadata == null)
return new VersionedLocation(Epoch.EMPTY, Location.UNKNOWN);
source = metadata.directory;
}
NodeId nodeId = source.peerId(endpoint);
Location location = nodeId != null ? source.location(nodeId) : Location.UNKNOWN;
return new VersionedLocation(source.lastModified(), location);
}
private Location fromDirectory(InetAddressAndPort endpoint)
{
Directory source = directory;
if (source == null)
{
ClusterMetadata metadata = ClusterMetadata.currentNullable();
if (metadata == null)
return Location.UNKNOWN;
source = metadata.directory;
}
NodeId nodeId = source.peerId(endpoint);
return nodeId != null ? source.location(nodeId) : Location.UNKNOWN;
}
private Location initialLocation()
{
if (initializationLocation == null)
initializationLocation = locationProvider.initialLocation();
return initializationLocation;
}
}

51
src/java/org/apache/cassandra/locator/NetworkTopologyProximity.java Normal file
View File

@ -0,0 +1,51 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.tcm.membership.Location;
public class NetworkTopologyProximity extends BaseProximity
{
public int compareEndpoints(InetAddressAndPort address, Replica r1, Replica r2)
{
InetAddressAndPort a1 = r1.endpoint();
InetAddressAndPort a2 = r2.endpoint();
if (address.equals(a1) && !address.equals(a2))
return -1;
if (address.equals(a2) && !address.equals(a1))
return 1;
Locator locator = DatabaseDescriptor.getLocator();
Location l1 = locator.location(a1);
Location l2 = locator.location(a2);
Location location = locator.location(address);
if (location.datacenter.equals(l1.datacenter) && !location.datacenter.equals(l2.datacenter))
return -1;
if (location.datacenter.equals(l2.datacenter) && !location.datacenter.equals(l1.datacenter))
return 1;
if (location.rack.equals(l1.rack) && !location.rack.equals(l2.rack))
return -1;
if (location.rack.equals(l2.rack) && !location.rack.equals(l1.rack))
return 1;
return 0;
}
}

37
src/java/org/apache/cassandra/locator/NoOpProximity.java Normal file
View File

@ -0,0 +1,37 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
public class NoOpProximity extends BaseProximity
{
@Override
public <C extends ReplicaCollection<? extends C>> C sortedByProximity(final InetAddressAndPort address, C unsortedAddress)
{
// Optimization to avoid walking the list
return unsortedAddress;
}
@Override
public int compareEndpoints(InetAddressAndPort target, Replica r1, Replica r2)
{
// Making all endpoints equal ensures we won't change the original ordering (since
// Collections.sort is guaranteed to be stable)
return 0;
}
}

47
src/java/org/apache/cassandra/locator/NodeAddressConfig.java Normal file
View File

@ -0,0 +1,47 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import org.apache.cassandra.config.DatabaseDescriptor;
public interface NodeAddressConfig
{
void configureAddresses();
boolean preferLocalConnections();
NodeAddressConfig DEFAULT = new NodeAddressConfig()
{
@Override
public void configureAddresses()
{
// Default is to use whatever addresses are specified in node config already, so this is a no-op
}
@Override
public boolean preferLocalConnections()
{
// Previously the equivalent config was hard coded for Ec2MultiRegionSnitch and read from
// cassandra-rackdc.properties in the case of GossipingPropertyFileSnitch. In legacy config
// mode, where one of those IEndpointSnitch impls is still specified the original behaviour
// is preserved. For modern config the option can be specified in the main yaml, distinct
// from the NodeProximity and InitialLocationProvider options.
return DatabaseDescriptor.preferLocalConnections();
}
};
}

38
src/java/org/apache/cassandra/locator/NodeProximity.java Normal file
View File

@ -0,0 +1,38 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
public interface NodeProximity
{
/**
* returns a new <tt>List</tt> sorted by proximity to the given endpoint
*/
public <C extends ReplicaCollection<? extends C>> C sortedByProximity(final InetAddressAndPort address, C addresses);
/**
* compares two endpoints in relation to the target endpoint, returning as Comparator.compare would
*/
public int compareEndpoints(InetAddressAndPort target, Replica r1, Replica r2);
/**
* Returns whether for a range query doing a query against merged is likely
* to be faster than 2 sequential queries, one against l1 followed by one against l2.
*/
public boolean isWorthMergingForRangeQuery(ReplicaCollection<?> merged, ReplicaCollection<?> l1, ReplicaCollection<?> l2);
}

139
src/java/org/apache/cassandra/locator/PropertyFileSnitch.java
View File

@ -17,22 +17,8 @@
*/
package org.apache.cassandra.locator;
import java.io.InputStream;
import java.net.UnknownHostException;
import java.util.Map;
import java.util.Properties;
import com.google.common.annotations.VisibleForTesting;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.membership.Location;
import org.apache.cassandra.tcm.membership.NodeId;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.commons.lang3.StringUtils;
/**
* <p>
@ -50,133 +36,28 @@ import org.apache.commons.lang3.StringUtils;
* this is done automatically with location derived from gossip state (ultimately from system.local).
* Once registered, the Rack & DC should not be changed but currently the only safeguards against this are the
* StartupChecks which validate the snitch against system.local.
* @deprecated See CASSANDRA-19488
*/
@Deprecated(since = "CEP-21")
public class PropertyFileSnitch extends AbstractNetworkTopologySnitch
{
private static final Logger logger = LoggerFactory.getLogger(PropertyFileSnitch.class);
public static final String SNITCH_PROPERTIES_FILENAME = "cassandra-topology.properties";
// All the defaults
private static final String DEFAULT_PROPERTY = "default";
@VisibleForTesting
public static final String DEFAULT_DC = "default";
@VisibleForTesting
public static final String DEFAULT_RACK = "default";
// Used only during initialization of a new node. This provides the location it will register in cluster metadata
private final Location local;
public PropertyFileSnitch() throws ConfigurationException
{
local = loadConfiguration();
local = new TopologyFileLocationProvider().initialLocation();
}
public String getDatacenter(InetAddressAndPort endpoint)
@Override
public String getLocalRack()
{
if (endpoint.equals(FBUtilities.getBroadcastAddressAndPort()))
return local.datacenter;
ClusterMetadata metadata = ClusterMetadata.current();
NodeId nodeId = metadata.directory.peerId(endpoint);
if (nodeId == null)
return DEFAULT_DC;
return metadata.directory.location(nodeId).datacenter;
return local.rack;
}
/**
* Return the rack for which an endpoint resides in
*
* @param endpoint the endpoint to process
* @return string of rack
*/
public String getRack(InetAddressAndPort endpoint)
@Override
public String getLocalDatacenter()
{
if (endpoint.equals(FBUtilities.getBroadcastAddressAndPort()))
return local.rack;
ClusterMetadata metadata = ClusterMetadata.current();
NodeId nodeId = metadata.directory.peerId(endpoint);
if (nodeId == null)
return DEFAULT_RACK;
return metadata.directory.location(nodeId).rack;
}
private Location makeLocation(String value)
{
if (value == null || value.isEmpty())
return null;
String[] parts = value.split(":");
if (parts.length < 2)
{
return new Location(DEFAULT_DC, DEFAULT_RACK);
}
else
{
return new Location(parts[0].trim(), parts[1].trim());
}
}
private Location loadConfiguration() throws ConfigurationException
{
Properties properties = new Properties();
try (InputStream stream = getClass().getClassLoader().getResourceAsStream(SNITCH_PROPERTIES_FILENAME))
{
properties.load(stream);
}
catch (Exception e)
{
throw new ConfigurationException("Unable to read " + SNITCH_PROPERTIES_FILENAME, e);
}
// may be null, which is ok unless config doesn't contain the location of the local node
Location defaultLocation = makeLocation(properties.getProperty(DEFAULT_PROPERTY));
Location local = null;
InetAddressAndPort broadcastAddress = FBUtilities.getBroadcastAddressAndPort();
for (Map.Entry<Object, Object> entry : properties.entrySet())
{
String key = (String) entry.getKey();
String value = (String) entry.getValue();
if (DEFAULT_PROPERTY.equals(key))
continue;
String hostString = StringUtils.remove(key, '/');
try
{
InetAddressAndPort host = InetAddressAndPort.getByName(hostString);
if (host.equals(broadcastAddress))
{
local = makeLocation(value);
break;
}
}
catch (UnknownHostException e)
{
throw new ConfigurationException("Unknown host " + hostString, e);
}
}
if (local == null)
{
if (defaultLocation == null)
{
throw new ConfigurationException(String.format("Snitch definitions at %s do not define a location for " +
"this node's broadcast address %s, nor does it provides a default",
SNITCH_PROPERTIES_FILENAME, broadcastAddress));
}
else
{
logger.debug("Broadcast address {} was not present in snitch config, using default location {}. " +
"This only matters on first boot, before registering with the cluster metadata service",
broadcastAddress, defaultLocation);
return defaultLocation;
}
}
logger.debug("Loaded location {} for broadcast address {} from property file. " +
"This only matters on first boot, before registering with the cluster metadata service",
local, broadcastAddress);
return local;
return local.datacenter;
}
}

72
src/java/org/apache/cassandra/locator/RackDCFileLocationProvider.java Normal file
View File

@ -0,0 +1,72 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.tcm.membership.Location;
/**
* Emulates the previous behaviour of GossipingPropertyFileSnitch; reads cassandra-rackdc.properties
* to identify the local datacenter and rack for one time use during node registration.
*
* Based on a properties file in the following format:
*
* dc=DC1
* rack=RAC1
*/
public class RackDCFileLocationProvider implements InitialLocationProvider
{
private final Location local;
private static final Location DEFAULT = new Location("UNKNOWN_DC", "UNKNOWN_RACK");
/**
* Used via reflection by DatabaseDescriptor::createInitialLocationProvider
*/
public RackDCFileLocationProvider()
{
this(loadConfiguration());
}
/**
* Used in legacy compatibility mode by GossipingPropertyFileSnitch
* @param properties
*/
RackDCFileLocationProvider(SnitchProperties properties)
{
local = new Location(properties.get("dc", DEFAULT.datacenter).trim(),
properties.get("rack", DEFAULT.rack).trim());
}
@Override
public Location initialLocation()
{
return local;
}
public static SnitchProperties loadConfiguration() throws ConfigurationException
{
final SnitchProperties properties = new SnitchProperties();
if (!properties.contains("dc") || !properties.contains("rack"))
throw new ConfigurationException("DC or rack not found in snitch properties, check your configuration in: " + SnitchProperties.RACKDC_PROPERTY_FILENAME);
return properties;
}
}

32
src/java/org/apache/cassandra/locator/RackInferringSnitch.java
View File

@ -17,9 +17,7 @@
*/
package org.apache.cassandra.locator;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.membership.Location;
import org.apache.cassandra.tcm.membership.NodeId;
import org.apache.cassandra.utils.FBUtilities;
/**
@ -29,7 +27,9 @@ import org.apache.cassandra.utils.FBUtilities;
* Local location is derived from (broadcast) ip address and added to ClusterMetadata during node
* registration. Every member of the cluster is required to do this, hence remote peers' Location
* can always be retrieved, consistently.
* @deprecated See CASSANDRA-19488
*/
@Deprecated(since = "CEP-21")
public class RackInferringSnitch extends AbstractNetworkTopologySnitch
{
final Location local;
@ -37,31 +37,25 @@ public class RackInferringSnitch extends AbstractNetworkTopologySnitch
public RackInferringSnitch()
{
InetAddressAndPort localAddress = FBUtilities.getBroadcastAddressAndPort();
local = new Location(Integer.toString(localAddress.getAddress().getAddress()[1] & 0xFF, 10),
Integer.toString(localAddress.getAddress().getAddress()[2] & 0xFF, 10));
local = inferLocation(localAddress);
}
public String getDatacenter(InetAddressAndPort endpoint)
public static Location inferLocation(InetAddressAndPort address)
{
if (endpoint.equals(FBUtilities.getBroadcastAddressAndPort()))
return local.datacenter;
return new Location(Integer.toString(address.getAddress().getAddress()[1] & 0xFF, 10),
Integer.toString(address.getAddress().getAddress()[2] & 0xFF, 10));
ClusterMetadata metadata = ClusterMetadata.current();
NodeId nodeId = metadata.directory.peerId(endpoint);
if (nodeId == null)
return Integer.toString(endpoint.getAddress().getAddress()[1] & 0xFF, 10);
return metadata.directory.location(nodeId).datacenter;
}
@Override
public String getRack(InetAddressAndPort endpoint)
{
if (endpoint.equals(FBUtilities.getBroadcastAddressAndPort()))
return local.rack;
return inferLocation(endpoint).rack;
}
ClusterMetadata metadata = ClusterMetadata.current();
NodeId nodeId = metadata.directory.peerId(endpoint);
if (nodeId == null)
return Integer.toString(endpoint.getAddress().getAddress()[2] & 0xFF, 10);
return metadata.directory.location(nodeId).rack;
@Override
public String getDatacenter(InetAddressAndPort endpoint)
{
return inferLocation(endpoint).datacenter;
}
}

34
src/java/org/apache/cassandra/locator/ReconnectableSnitchHelper.java
View File

@ -40,14 +40,12 @@ import static org.apache.cassandra.auth.IInternodeAuthenticator.InternodeConnect
public class ReconnectableSnitchHelper implements IEndpointStateChangeSubscriber
{
private static final Logger logger = LoggerFactory.getLogger(ReconnectableSnitchHelper.class);
private final IEndpointSnitch snitch;
private final String localDc;
private final Locator locator;
private final boolean preferLocal;
public ReconnectableSnitchHelper(IEndpointSnitch snitch, String localDc, boolean preferLocal)
public ReconnectableSnitchHelper(Locator locator, boolean preferLocal)
{
this.snitch = snitch;
this.localDc = localDc;
this.locator = locator;
this.preferLocal = preferLocal;
}
@ -55,7 +53,7 @@ public class ReconnectableSnitchHelper implements IEndpointStateChangeSubscriber
{
try
{
reconnect(publicAddress, InetAddressAndPort.getByName(localAddressValue.value), snitch, localDc);
reconnect(publicAddress, InetAddressAndPort.getByName(localAddressValue.value), locator);
}
catch (UnknownHostException e)
{
@ -64,7 +62,7 @@ public class ReconnectableSnitchHelper implements IEndpointStateChangeSubscriber
}
@VisibleForTesting
static void reconnect(InetAddressAndPort publicAddress, InetAddressAndPort localAddress, IEndpointSnitch snitch, String localDc)
static void reconnect(InetAddressAndPort publicAddress, InetAddressAndPort localAddress, Locator locator)
{
final OutboundConnectionSettings settings = new OutboundConnectionSettings(publicAddress, localAddress).withDefaults(ConnectionCategory.MESSAGING);
if (!settings.authenticator().authenticate(settings.to.getAddress(), settings.to.getPort(), null, OUTBOUND_PRECONNECT))
@ -73,18 +71,13 @@ public class ReconnectableSnitchHelper implements IEndpointStateChangeSubscriber
return;
}
if (snitch.getDatacenter(publicAddress).equals(localDc))
if (locator.local().sameDatacenter(locator.location(publicAddress)))
{
MessagingService.instance().maybeReconnectWithNewIp(publicAddress, localAddress);
logger.debug("Initiated reconnect to an Internal IP {} for the {}", localAddress, publicAddress);
}
}
public void beforeChange(InetAddressAndPort endpoint, EndpointState currentState, ApplicationState newStateKey, VersionedValue newValue)
{
// no-op
}
public void onJoin(InetAddressAndPort endpoint, EndpointState epState)
{
if (preferLocal && !Gossiper.instance.isDeadState(epState))
@ -127,19 +120,4 @@ public class ReconnectableSnitchHelper implements IEndpointStateChangeSubscriber
if (preferLocal && internalIP != null)
reconnect(endpoint, internalIPAndPorts != null ? internalIPAndPorts : internalIP);
}
public void onDead(InetAddressAndPort endpoint, EndpointState state)
{
// do nothing.
}
public void onRemove(InetAddressAndPort endpoint)
{
// do nothing.
}
public void onRestart(InetAddressAndPort endpoint, EndpointState state)
{
// do nothing.
}
}

4
src/java/org/apache/cassandra/locator/ReplicaLayout.java
View File

@ -360,7 +360,7 @@ public abstract class ReplicaLayout<E extends Endpoints<E>>
EndpointsForToken replicas = keyspace.getMetadata().params.replication.isLocal()
? forLocalStrategyToken(metadata, replicationStrategy, token)
: forNonLocalStrategyTokenRead(metadata, keyspace.getMetadata(), token);
replicas = DatabaseDescriptor.getEndpointSnitch().sortedByProximity(FBUtilities.getBroadcastAddressAndPort(), replicas);
replicas = DatabaseDescriptor.getNodeProximity().sortedByProximity(FBUtilities.getBroadcastAddressAndPort(), replicas);
replicas = replicas.filter(FailureDetector.isReplicaAlive);
return new ReplicaLayout.ForTokenRead(replicationStrategy, replicas);
}
@ -376,7 +376,7 @@ public abstract class ReplicaLayout<E extends Endpoints<E>>
? forLocalStrategyRange(metadata, replicationStrategy, range)
: forNonLocalStategyRangeRead(metadata, keyspace.getMetadata(), range);
replicas = DatabaseDescriptor.getEndpointSnitch().sortedByProximity(FBUtilities.getBroadcastAddressAndPort(), replicas);
replicas = DatabaseDescriptor.getNodeProximity().sortedByProximity(FBUtilities.getBroadcastAddressAndPort(), replicas);
replicas = replicas.filter(FailureDetector.isReplicaAlive);
return new ReplicaLayout.ForRangeRead(replicationStrategy, range, replicas);
}

91
src/java/org/apache/cassandra/locator/ReplicaPlans.java
View File

@ -68,6 +68,8 @@ import org.apache.cassandra.service.reads.AlwaysSpeculativeRetryPolicy;
import org.apache.cassandra.service.reads.SpeculativeRetryPolicy;
import org.apache.cassandra.tcm.Epoch;
import org.apache.cassandra.tcm.membership.Directory;
import org.apache.cassandra.tcm.membership.Location;
import org.apache.cassandra.tcm.membership.NodeState;
import org.apache.cassandra.utils.FBUtilities;
@ -98,7 +100,7 @@ public class ReplicaPlans
logger.warn("System property {} was set to {} but must be 1 or 2. Running with {}", CassandraRelevantProperties.REQUIRED_BATCHLOG_REPLICA_COUNT.getKey(), batchlogReplicaCount, REQUIRED_BATCHLOG_REPLICA_COUNT);
}
public static boolean isSufficientLiveReplicasForRead(AbstractReplicationStrategy replicationStrategy, ConsistencyLevel consistencyLevel, Endpoints<?> liveReplicas)
public static boolean isSufficientLiveReplicasForRead(Locator locator, AbstractReplicationStrategy replicationStrategy, ConsistencyLevel consistencyLevel, Endpoints<?> liveReplicas)
{
switch (consistencyLevel)
{
@ -114,7 +116,7 @@ public class ReplicaPlans
{
int fullCount = 0;
Collection<String> dcs = ((NetworkTopologyStrategy) replicationStrategy).getDatacenters();
for (ObjectObjectCursor<String, Replicas.ReplicaCount> entry : countPerDc(dcs, liveReplicas))
for (ObjectObjectCursor<String, Replicas.ReplicaCount> entry : countPerDc(locator, dcs, liveReplicas))
{
Replicas.ReplicaCount count = entry.value;
if (!count.hasAtleast(localQuorumFor(replicationStrategy, entry.key), 0))
@ -130,17 +132,17 @@ public class ReplicaPlans
}
}
static void assureSufficientLiveReplicasForRead(AbstractReplicationStrategy replicationStrategy, ConsistencyLevel consistencyLevel, Endpoints<?> liveReplicas) throws UnavailableException
static void assureSufficientLiveReplicasForRead(Locator locator, AbstractReplicationStrategy replicationStrategy, ConsistencyLevel consistencyLevel, Endpoints<?> liveReplicas) throws UnavailableException
{
assureSufficientLiveReplicas(replicationStrategy, consistencyLevel, liveReplicas, consistencyLevel.blockFor(replicationStrategy), 1);
assureSufficientLiveReplicas(locator, replicationStrategy, consistencyLevel, liveReplicas, consistencyLevel.blockFor(replicationStrategy), 1);
}
static void assureSufficientLiveReplicasForWrite(AbstractReplicationStrategy replicationStrategy, ConsistencyLevel consistencyLevel, Endpoints<?> allLive, Endpoints<?> pendingWithDown) throws UnavailableException
static void assureSufficientLiveReplicasForWrite(Locator locator, AbstractReplicationStrategy replicationStrategy, ConsistencyLevel consistencyLevel, Endpoints<?> allLive, Endpoints<?> pendingWithDown) throws UnavailableException
{
assureSufficientLiveReplicas(replicationStrategy, consistencyLevel, allLive, consistencyLevel.blockForWrite(replicationStrategy, pendingWithDown), 0);
assureSufficientLiveReplicas(locator, replicationStrategy, consistencyLevel, allLive, consistencyLevel.blockForWrite(replicationStrategy, pendingWithDown), 0);
}
static void assureSufficientLiveReplicas(AbstractReplicationStrategy replicationStrategy, ConsistencyLevel consistencyLevel, Endpoints<?> allLive, int blockFor, int blockForFullReplicas) throws UnavailableException
static void assureSufficientLiveReplicas(Locator locator, AbstractReplicationStrategy replicationStrategy, ConsistencyLevel consistencyLevel, Endpoints<?> allLive, int blockFor, int blockForFullReplicas) throws UnavailableException
{
switch (consistencyLevel)
{
@ -172,7 +174,7 @@ public class ReplicaPlans
int total = 0;
int totalFull = 0;
Collection<String> dcs = ((NetworkTopologyStrategy) replicationStrategy).getDatacenters();
for (ObjectObjectCursor<String, Replicas.ReplicaCount> entry : countPerDc(dcs, allLive))
for (ObjectObjectCursor<String, Replicas.ReplicaCount> entry : countPerDc(locator, dcs, allLive))
{
int dcBlockFor = localQuorumFor(replicationStrategy, entry.key);
Replicas.ReplicaCount dcCount = entry.value;
@ -226,7 +228,7 @@ public class ReplicaPlans
public static Replica findCounterLeaderReplica(ClusterMetadata metadata, String keyspaceName, DecoratedKey key, String localDataCenter, ConsistencyLevel cl) throws UnavailableException
{
Keyspace keyspace = Keyspace.open(keyspaceName);
IEndpointSnitch snitch = DatabaseDescriptor.getEndpointSnitch();
NodeProximity proximity = DatabaseDescriptor.getNodeProximity();
AbstractReplicationStrategy replicationStrategy = keyspace.getReplicationStrategy();
EndpointsForToken replicas = metadata.placements.get(keyspace.getMetadata().params.replication).reads.forToken(key.getToken()).get();
@ -243,7 +245,7 @@ public class ReplicaPlans
List<Replica> localReplicas = new ArrayList<>(replicas.size());
for (Replica replica : replicas)
if (snitch.getDatacenter(replica).equals(localDataCenter))
if (metadata.locator.location(replica.endpoint()).datacenter.equals(localDataCenter))
localReplicas.add(replica);
if (localReplicas.isEmpty())
@ -252,8 +254,8 @@ public class ReplicaPlans
if (cl.isDatacenterLocal())
throw UnavailableException.create(cl, cl.blockFor(replicationStrategy), 0);
// No endpoint in local DC, pick the closest endpoint according to the snitch
replicas = snitch.sortedByProximity(FBUtilities.getBroadcastAddressAndPort(), replicas);
// No endpoint in local DC, pick the closest endpoint according to the configured proximity measures
replicas = proximity.sortedByProximity(FBUtilities.getBroadcastAddressAndPort(), replicas);
return replicas.get(0);
}
@ -296,16 +298,17 @@ public class ReplicaPlans
private static ReplicaLayout.ForTokenWrite liveAndDownForBatchlogWrite(Token token, ClusterMetadata metadata, boolean isAny)
{
IEndpointSnitch snitch = DatabaseDescriptor.getEndpointSnitch();
Multimap<String, InetAddressAndPort> localEndpoints = HashMultimap.create(metadata.directory.allDatacenterRacks()
.get(snitch.getLocalDatacenter()));
Directory directory = metadata.directory;
Location local = metadata.locator.local();
Multimap<String, InetAddressAndPort> localEndpoints = HashMultimap.create(directory.allDatacenterRacks()
.get(local.datacenter));
// Replicas are picked manually:
// - replicas should be alive according to the failure detector
// - replicas should be in the local datacenter
// - choose min(2, number of qualifying candiates above)
// - allow the local node to be the only replica only if it's a single-node DC
Collection<InetAddressAndPort> chosenEndpoints = filterBatchlogEndpoints(false,
snitch.getLocalRack(),
local.rack,
localEndpoints,
Collections::shuffle,
(r) -> FailureDetector.isEndpointAlive.test(r) && metadata.directory.peerState(r) == NodeState.JOINED,
@ -343,7 +346,7 @@ public class ReplicaPlans
AbstractReplicationStrategy replicationStrategy = liveAndDown.replicationStrategy();
EndpointsForToken contacts = writeAll.select(consistencyLevel, liveAndDown, liveAndDown);
assureSufficientLiveReplicasForWrite(replicationStrategy, consistencyLevel, liveAndDown.all(), liveAndDown.pending());
assureSufficientLiveReplicasForWrite(metadata.locator, replicationStrategy, consistencyLevel, liveAndDown.all(), liveAndDown.pending());
return new ReplicaPlan.ForWrite(systemKeyspace,
replicationStrategy,
consistencyLevel,
@ -506,7 +509,7 @@ public class ReplicaPlans
if (racks.isEmpty())
racks.addAll(validated.keySet());
String rack = DatabaseDescriptor.getEndpointSnitch().getRack(endpoint);
String rack = DatabaseDescriptor.getLocator().location(endpoint).rack;
if (!racks.remove(rack))
continue;
if (result.contains(endpoint))
@ -523,7 +526,7 @@ public class ReplicaPlans
public static List<InetAddressAndPort> sortByProximity(Collection<InetAddressAndPort> endpoints)
{
EndpointsForRange endpointsForRange = SystemReplicas.getSystemReplicas(endpoints);
return DatabaseDescriptor.getEndpointSnitch()
return DatabaseDescriptor.getNodeProximity()
.sortedByProximity(FBUtilities.getBroadcastAddressAndPort(), endpointsForRange)
.endpointList();
}
@ -538,7 +541,7 @@ public class ReplicaPlans
ReplicaLayout.ForTokenWrite live = liveAndDown.filter(isAlive);
EndpointsForToken contacts = selector.select(consistencyLevel, liveAndDown, live);
assureSufficientLiveReplicasForWrite(replicationStrategy, consistencyLevel, live.all(), liveAndDown.pending());
assureSufficientLiveReplicasForWrite(metadata.locator, replicationStrategy, consistencyLevel, live.all(), liveAndDown.pending());
return new ReplicaPlan.ForWrite(keyspace,
replicationStrategy,
consistencyLevel,
@ -588,7 +591,7 @@ public class ReplicaPlans
AbstractReplicationStrategy replicationStrategy = liveAndDown.replicationStrategy();
EndpointsForToken contacts = selector.select(consistencyLevel, liveAndDown, live);
assureSufficientLiveReplicasForWrite(replicationStrategy, consistencyLevel, live.all(), liveAndDown.pending());
assureSufficientLiveReplicasForWrite(metadata.locator, replicationStrategy, consistencyLevel, live.all(), liveAndDown.pending());
return new ReplicaPlan.ForWrite(keyspace,
replicationStrategy,
@ -655,14 +658,15 @@ public class ReplicaPlans
* soft-ensure that we reach QUORUM in all DCs we are able to, by writing to every node;
* even if we don't wait for ACK, we have in both cases sent sufficient messages.
*/
ObjectIntHashMap<String> requiredPerDc = eachQuorumForWrite(liveAndDown.replicationStrategy(), liveAndDown.pending());
addToCountPerDc(requiredPerDc, live.natural().filter(Replica::isFull), -1);
addToCountPerDc(requiredPerDc, live.pending(), -1);
Locator locator = ClusterMetadata.current().locator;
ObjectIntHashMap<String> requiredPerDc = eachQuorumForWrite(locator, liveAndDown.replicationStrategy(), liveAndDown.pending());
addToCountPerDc(locator, requiredPerDc, live.natural().filter(Replica::isFull), -1);
addToCountPerDc(locator, requiredPerDc, live.pending(), -1);
IEndpointSnitch snitch = DatabaseDescriptor.getEndpointSnitch();
for (Replica replica : filter(live.natural(), Replica::isTransient))
{
String dc = snitch.getDatacenter(replica);
String dc = locator.location(replica.endpoint()).datacenter;
if (requiredPerDc.addTo(dc, -1) >= 0)
contacts.add(replica);
}
@ -713,12 +717,12 @@ public class ReplicaPlans
}
else
{
ObjectIntHashMap<String> requiredPerDc = eachQuorumForWrite(liveAndDown.replicationStrategy(), liveAndDown.pending());
addToCountPerDc(requiredPerDc, contacts.snapshot(), -1);
IEndpointSnitch snitch = DatabaseDescriptor.getEndpointSnitch();
Locator locator = ClusterMetadata.current().locator;
ObjectIntHashMap<String> requiredPerDc = eachQuorumForWrite(locator, liveAndDown.replicationStrategy(), liveAndDown.pending());
addToCountPerDc(locator, requiredPerDc, contacts.snapshot(), -1);
for (Replica replica : filter(live.all(), r -> !contacts.contains(r)))
{
String dc = snitch.getDatacenter(replica);
String dc = locator.location(replica.endpoint()).datacenter;
if (requiredPerDc.addTo(dc, -1) >= 0)
contacts.add(replica);
}
@ -796,18 +800,16 @@ public class ReplicaPlans
return indexQueryPlan != null ? IndexStatusManager.instance.filterForQuery(replicas, keyspace, indexQueryPlan, consistencyLevel) : replicas;
}
private static <E extends Endpoints<E>> E contactForEachQuorumRead(NetworkTopologyStrategy replicationStrategy, E candidates)
private static <E extends Endpoints<E>> E contactForEachQuorumRead(Locator locator, NetworkTopologyStrategy replicationStrategy, E candidates)
{
ObjectIntHashMap<String> perDc = eachQuorumForRead(replicationStrategy);
final IEndpointSnitch snitch = DatabaseDescriptor.getEndpointSnitch();
return candidates.filter(replica -> {
String dc = snitch.getDatacenter(replica);
String dc = locator.location(replica.endpoint()).datacenter;
return perDc.addTo(dc, -1) >= 0;
});
}
private static <E extends Endpoints<E>> E contactForRead(AbstractReplicationStrategy replicationStrategy, ConsistencyLevel consistencyLevel, boolean alwaysSpeculate, E candidates)
private static <E extends Endpoints<E>> E contactForRead(Locator locator, AbstractReplicationStrategy replicationStrategy, ConsistencyLevel consistencyLevel, boolean alwaysSpeculate, E candidates)
{
/*
* If we are doing an each quorum query, we have to make sure that the endpoints we select
@ -819,7 +821,7 @@ public class ReplicaPlans
* TODO: this is still very inconistently managed between {LOCAL,EACH}_QUORUM and other consistency levels - should address this in a follow-up
*/
if (consistencyLevel == EACH_QUORUM && replicationStrategy instanceof NetworkTopologyStrategy)
return contactForEachQuorumRead((NetworkTopologyStrategy) replicationStrategy, candidates);
return contactForEachQuorumRead(locator, (NetworkTopologyStrategy) replicationStrategy, candidates);
int count = consistencyLevel.blockFor(replicationStrategy) + (alwaysSpeculate ? 1 : 0);
return candidates.subList(0, Math.min(count, candidates.size()));
@ -903,10 +905,10 @@ public class ReplicaPlans
AbstractReplicationStrategy replicationStrategy = keyspace.getReplicationStrategy();
ReplicaLayout.ForTokenRead forTokenRead = ReplicaLayout.forTokenReadLiveSorted(metadata, keyspace, replicationStrategy, token);
EndpointsForToken candidates = candidatesForRead(keyspace, indexQueryPlan, consistencyLevel, forTokenRead.natural());
EndpointsForToken contacts = contactForRead(replicationStrategy, consistencyLevel, retry.equals(AlwaysSpeculativeRetryPolicy.INSTANCE), candidates);
EndpointsForToken contacts = contactForRead(metadata.locator, replicationStrategy, consistencyLevel, retry.equals(AlwaysSpeculativeRetryPolicy.INSTANCE), candidates);
if (throwOnInsufficientLiveReplicas)
assureSufficientLiveReplicasForRead(replicationStrategy, consistencyLevel, contacts);
assureSufficientLiveReplicasForRead(metadata.locator, replicationStrategy, consistencyLevel, contacts);
return new ReplicaPlan.ForTokenRead(keyspace, replicationStrategy, consistencyLevel, candidates, contacts,
(newClusterMetadata) -> forRead(newClusterMetadata, keyspace, token, indexQueryPlan, consistencyLevel, retry, false),
@ -941,10 +943,10 @@ public class ReplicaPlans
AbstractReplicationStrategy replicationStrategy = keyspace.getReplicationStrategy();
ReplicaLayout.ForRangeRead forRangeRead = ReplicaLayout.forRangeReadLiveSorted(metadata, keyspace, replicationStrategy, range);
EndpointsForRange candidates = candidatesForRead(keyspace, indexQueryPlan, consistencyLevel, forRangeRead.natural());
EndpointsForRange contacts = contactForRead(replicationStrategy, consistencyLevel, false, candidates);
EndpointsForRange contacts = contactForRead(metadata.locator, replicationStrategy, consistencyLevel, false, candidates);
if (throwOnInsufficientLiveReplicas)
assureSufficientLiveReplicasForRead(replicationStrategy, consistencyLevel, contacts);
assureSufficientLiveReplicasForRead(metadata.locator, replicationStrategy, consistencyLevel, contacts);
return new ReplicaPlan.ForRangeRead(keyspace,
replicationStrategy,
@ -986,7 +988,8 @@ public class ReplicaPlans
/**
* Take two range read plans for adjacent ranges, and check if it is OK (and worthwhile) to combine them into a single plan
*/
public static ReplicaPlan.ForRangeRead maybeMerge(Keyspace keyspace,
public static ReplicaPlan.ForRangeRead maybeMerge(ClusterMetadata metadata,
Keyspace keyspace,
ConsistencyLevel consistencyLevel,
ReplicaPlan.ForRangeRead left,
ReplicaPlan.ForRangeRead right)
@ -998,16 +1001,16 @@ public class ReplicaPlans
EndpointsForRange mergedCandidates = left.readCandidates().keep(right.readCandidates().endpoints());
AbstractReplicationStrategy replicationStrategy = keyspace.getReplicationStrategy();
EndpointsForRange contacts = contactForRead(replicationStrategy, consistencyLevel, false, mergedCandidates);
EndpointsForRange contacts = contactForRead(metadata.locator, replicationStrategy, consistencyLevel, false, mergedCandidates);
// Estimate whether merging will be a win or not
if (!DatabaseDescriptor.getEndpointSnitch().isWorthMergingForRangeQuery(contacts, left.contacts(), right.contacts()))
if (!DatabaseDescriptor.getNodeProximity().isWorthMergingForRangeQuery(contacts, left.contacts(), right.contacts()))
return null;
AbstractBounds<PartitionPosition> newRange = left.range().withNewRight(right.range().right);
// Check if there are enough shared endpoints for the merge to be possible.
if (!isSufficientLiveReplicasForRead(replicationStrategy, consistencyLevel, mergedCandidates))
if (!isSufficientLiveReplicasForRead(metadata.locator, replicationStrategy, consistencyLevel, mergedCandidates))
return null;
int newVnodeCount = left.vnodeCount() + right.vnodeCount();

12
src/java/org/apache/cassandra/locator/Replicas.java
View File

@ -25,8 +25,8 @@ import java.util.function.Predicate;
import com.carrotsearch.hppc.ObjectIntHashMap;
import com.carrotsearch.hppc.ObjectObjectHashMap;
import com.google.common.collect.Iterables;
import org.apache.cassandra.config.DatabaseDescriptor;
import static com.google.common.collect.Iterables.all;
@ -88,16 +88,15 @@ public class Replicas
/**
* count the number of full and transient replicas, separately, for each DC
*/
public static ObjectObjectHashMap<String, ReplicaCount> countPerDc(Collection<String> dataCenters, Iterable<Replica> replicas)
public static ObjectObjectHashMap<String, ReplicaCount> countPerDc(Locator locator, Collection<String> dataCenters, Iterable<Replica> replicas)
{
ObjectObjectHashMap<String, ReplicaCount> perDc = new ObjectObjectHashMap<>(dataCenters.size());
for (String dc: dataCenters)
perDc.put(dc, new ReplicaCount());
IEndpointSnitch snitch = DatabaseDescriptor.getEndpointSnitch();
for (Replica replica : replicas)
{
String dc = snitch.getDatacenter(replica);
String dc = locator.location(replica.endpoint()).datacenter;
perDc.get(dc).increment(replica);
}
return perDc;
@ -106,12 +105,11 @@ public class Replicas
/**
* increment each of the map's DC entries for each matching replica provided
*/
public static void addToCountPerDc(ObjectIntHashMap<String> perDc, Iterable<Replica> replicas, int add)
public static void addToCountPerDc(Locator locator, ObjectIntHashMap<String> perDc, Iterable<Replica> replicas, int add)
{
IEndpointSnitch snitch = DatabaseDescriptor.getEndpointSnitch();
for (Replica replica : replicas)
{
String dc = snitch.getDatacenter(replica);
String dc = locator.location(replica.endpoint()).datacenter;
perDc.addTo(dc, add);
}
}

32
src/java/org/apache/cassandra/locator/SimpleLocationProvider.java Normal file
View File

@ -0,0 +1,32 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import org.apache.cassandra.tcm.membership.Location;
public class SimpleLocationProvider implements InitialLocationProvider
{
public static final Location LOCATION = new Location("datacenter1", "rack1");
@Override
public Location initialLocation()
{
return LOCATION;
}
}

37
src/java/org/apache/cassandra/locator/SimpleSnitch.java
View File

@ -21,34 +21,41 @@ package org.apache.cassandra.locator;
* A simple endpoint snitch implementation that treats Strategy order as proximity,
* allowing non-read-repaired reads to prefer a single endpoint, which improves
* cache locality.
* @deprecated See CASSANDRA-19488
*/
@Deprecated(since = "CEP-21")
public class SimpleSnitch extends AbstractEndpointSnitch
{
public static final String DATA_CENTER_NAME = "datacenter1";
public static final String RACK_NAME = "rack1";
private static final NodeProximity sorter = new NoOpProximity();
private static final SimpleLocationProvider provider = new SimpleLocationProvider();
public String getRack(InetAddressAndPort endpoint)
@Override
public String getLocalRack()
{
return RACK_NAME;
}
public String getDatacenter(InetAddressAndPort endpoint)
{
return DATA_CENTER_NAME;
return provider.initialLocation().rack;
}
@Override
public <C extends ReplicaCollection<? extends C>> C sortedByProximity(final InetAddressAndPort address, C unsortedAddress)
public String getLocalDatacenter()
{
// Optimization to avoid walking the list
return unsortedAddress;
return provider.initialLocation().datacenter;
}
@Override
public <C extends ReplicaCollection<? extends C>> C sortedByProximity(InetAddressAndPort address, C unsortedAddress)
{
return sorter.sortedByProximity(address, unsortedAddress);
}
@Override
public int compareEndpoints(InetAddressAndPort target, Replica r1, Replica r2)
{
// Making all endpoints equal ensures we won't change the original ordering (since
// Collections.sort is guaranteed to be stable)
return 0;
return sorter.compareEndpoints(target, r1, r2);
}
@Override
public boolean isWorthMergingForRangeQuery(ReplicaCollection<?> merged, ReplicaCollection<?> l1, ReplicaCollection<?> l2)
{
return sorter.isWorthMergingForRangeQuery(merged, l1, l2);
}
}

84
src/java/org/apache/cassandra/locator/SnitchAdapter.java Normal file
View File

@ -0,0 +1,84 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import java.util.HashSet;
import java.util.Set;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.membership.Location;
public class SnitchAdapter implements InitialLocationProvider, NodeProximity, NodeAddressConfig
{
public final IEndpointSnitch snitch;
public SnitchAdapter(IEndpointSnitch snitch)
{
this.snitch = snitch;
}
@Override
public Location initialLocation()
{
return new Location(snitch.getLocalDatacenter(), snitch.getLocalRack());
}
@Override
public void validate(ClusterMetadata metadata)
{
Set<String> datacenters = metadata.directory.allDatacenterRacks().keySet();
Set<String> racks = new HashSet<>();
for (String dc : datacenters)
racks.addAll(metadata.directory.datacenterRacks(dc).keySet());
if (!snitch.validate(datacenters, racks))
throw new ConfigurationException("Initial location provider rejected registration location, " +
"please check the system log for errors");
}
@Override
public <C extends ReplicaCollection<? extends C>> C sortedByProximity(InetAddressAndPort address, C addresses)
{
return snitch.sortedByProximity(address, addresses);
}
@Override
public int compareEndpoints(InetAddressAndPort target, Replica r1, Replica r2)
{
return snitch.compareEndpoints(target, r1, r2);
}
@Override
public boolean isWorthMergingForRangeQuery(ReplicaCollection<?> merged, ReplicaCollection<?> l1, ReplicaCollection<?> l2)
{
return snitch.isWorthMergingForRangeQuery(merged, l1, l2);
}
@Override
public void configureAddresses()
{
snitch.configureAddresses();
}
@Override
public boolean preferLocalConnections()
{
return snitch.preferLocalConnections();
}
}

9
src/java/org/apache/cassandra/locator/SnitchUtils.java
View File

@ -18,6 +18,7 @@
package org.apache.cassandra.locator;
import org.apache.cassandra.tcm.membership.Location;
import org.apache.cassandra.utils.Pair;
import static java.lang.String.format;
@ -26,7 +27,13 @@ public class SnitchUtils
{
private SnitchUtils() {}
static Pair<String, String> parseDcAndRack(String response, String dcSuffix)
public static Location parseLocation(String response, String dcSuffix)
{
Pair<String, String> pair = parseDcAndRack(response, dcSuffix);
return new Location(pair.left, pair.right);
}
public static Pair<String, String> parseDcAndRack(String response, String dcSuffix)
{
String[] splits = response.split("/");
String az = splits[splits.length - 1];

160
src/java/org/apache/cassandra/locator/TopologyFileLocationProvider.java Normal file
View File

@ -0,0 +1,160 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.locator;
import java.io.InputStream;
import java.net.UnknownHostException;
import java.util.Map;
import java.util.Properties;
import com.google.common.annotations.VisibleForTesting;
import org.apache.commons.lang3.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.tcm.membership.Location;
import org.apache.cassandra.utils.FBUtilities;
/**
* Emulates the previous behaviour of PropertyFileSnitch; reads cassandra-topology.properties
* to identify the local datacenter and rack for one time use during node registration.
*
* Based on a properties file in the following format:
*
* 10.0.0.13=DC1:RAC2
* 10.21.119.14=DC3:RAC2
* 10.20.114.15=DC2:RAC2
* default=DC1:r1
*
* Post CEP-21, only the local rack and DC are loaded from file. Each peer in the cluster is required to register
* itself with the Cluster Metadata Service and provide its Location (Rack + DC) before joining. During upgrades,
* this is done automatically with location derived from gossip state (ultimately from system.local).
* Once registered, the Rack & DC should not be changed but currently the only safeguards against this are the
* StartupChecks which validate the snitch against system.local.
*/
public class TopologyFileLocationProvider implements InitialLocationProvider
{
private static final Logger logger = LoggerFactory.getLogger(TopologyFileLocationProvider.class);
public static final String PROPERTIES_FILENAME = "cassandra-topology.properties";
// All the defaults
private static final String DEFAULT_PROPERTY = "default";
@VisibleForTesting
public static final String DEFAULT_DC = "default";
@VisibleForTesting
public static final String DEFAULT_RACK = "default";
// Used only during initialization of a new node. This provides the location it will register in cluster metadata
private final Location local;
public TopologyFileLocationProvider()
{
local = loadConfiguration();
}
@Override
public Location initialLocation()
{
return local;
}
private Location makeLocation(String value)
{
if (value == null || value.isEmpty())
return null;
String[] parts = value.split(":");
if (parts.length < 2)
{
return new Location(DEFAULT_DC, DEFAULT_RACK);
}
else
{
return new Location(parts[0].trim(), parts[1].trim());
}
}
private Location loadConfiguration() throws ConfigurationException
{
Properties properties = new Properties();
try (InputStream stream = getClass().getClassLoader().getResourceAsStream(PROPERTIES_FILENAME))
{
properties.load(stream);
}
catch (Exception e)
{
throw new ConfigurationException("Unable to read " + PROPERTIES_FILENAME, e);
}
Location local = null;
InetAddressAndPort broadcastAddress = FBUtilities.getBroadcastAddressAndPort();
for (Map.Entry<Object, Object> entry : properties.entrySet())
{
String key = (String) entry.getKey();
String value = (String) entry.getValue();
if (DEFAULT_PROPERTY.equals(key))
continue;
String hostString = StringUtils.remove(key, '/');
try
{
InetAddressAndPort host = InetAddressAndPort.getByName(hostString);
if (host.equals(broadcastAddress))
{
local = makeLocation(value);
break;
}
}
catch (UnknownHostException e)
{
throw new ConfigurationException("Unknown host " + hostString, e);
}
}
// This may be null, which is ok unless config doesn't contain the location of the local node
Location defaultLocation = makeLocation(properties.getProperty(DEFAULT_PROPERTY));
if (local == null)
{
if (defaultLocation == null)
{
throw new ConfigurationException(String.format("Snitch definitions at %s do not define a location for " +
"this node's broadcast address %s, nor does it provides a default",
PROPERTIES_FILENAME, broadcastAddress));
}
else
{
logger.debug("Broadcast address {} was not present in snitch config, using default location {}. " +
"This only matters on first boot, before registering with the cluster metadata service",
broadcastAddress, defaultLocation);
local = defaultLocation;
}
}
logger.debug("Loaded location {} for broadcast address {} from property file. " +
"This only matters on first boot, before registering with the cluster metadata service",
local, broadcastAddress);
return local;
}
}

2
src/java/org/apache/cassandra/metrics/MessagingMetrics.java
View File

@ -114,7 +114,7 @@ public class MessagingMetrics implements InboundMessageHandlers.GlobalMetricCall
public DCLatencyRecorder internodeLatencyRecorder(InetAddressAndPort from)
{
String dcName = DatabaseDescriptor.getEndpointSnitch().getDatacenter(from);
String dcName = DatabaseDescriptor.getLocator().location(from).datacenter;
DCLatencyRecorder dcUpdater = dcLatency.get(dcName);
if (dcUpdater == null)
dcUpdater = dcLatency.computeIfAbsent(dcName, k -> new DCLatencyRecorder(Metrics.timer(factory.createMetricName(dcName + "-Latency")), allLatency));

18
src/java/org/apache/cassandra/net/OutboundConnectionSettings.java
View File

@ -28,11 +28,10 @@ import org.apache.cassandra.config.Config;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.config.EncryptionOptions.ServerEncryptionOptions;
import org.apache.cassandra.db.SystemKeyspace;
import org.apache.cassandra.locator.IEndpointSnitch;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.locator.Locator;
import org.apache.cassandra.utils.FBUtilities;
import static org.apache.cassandra.config.DatabaseDescriptor.getEndpointSnitch;
import static org.apache.cassandra.net.MessagingService.instance;
import static org.apache.cassandra.utils.FBUtilities.getBroadcastAddressAndPort;
@ -420,7 +419,7 @@ public class OutboundConnectionSettings
if (tcpNoDelay != null)
return tcpNoDelay;
if (DatabaseDescriptor.isClientOrToolInitialized() || isInLocalDC(getEndpointSnitch(), getBroadcastAddressAndPort(), to))
if (DatabaseDescriptor.isClientOrToolInitialized() || isInLocalDC(getBroadcastAddressAndPort(), to))
return INTRADC_TCP_NODELAY;
return DatabaseDescriptor.getInterDCTcpNoDelay();
@ -459,7 +458,7 @@ public class OutboundConnectionSettings
if (category.isStreaming())
return Framing.UNPROTECTED;
return shouldCompressConnection(getEndpointSnitch(), getBroadcastAddressAndPort(), to)
return shouldCompressConnection(getBroadcastAddressAndPort(), to)
? Framing.LZ4 : Framing.CRC;
}
@ -479,10 +478,11 @@ public class OutboundConnectionSettings
from(), socketFactory(), callbacks(), debug(), endpointToVersion());
}
private static boolean isInLocalDC(IEndpointSnitch snitch, InetAddressAndPort localHost, InetAddressAndPort remoteHost)
private static boolean isInLocalDC(InetAddressAndPort localHost, InetAddressAndPort remoteHost)
{
String remoteDC = snitch.getDatacenter(remoteHost);
String localDC = snitch.getDatacenter(localHost);
Locator locator = DatabaseDescriptor.getLocator();
String remoteDC = locator.location(remoteHost).datacenter;
String localDC = locator.location(localHost).datacenter;
return remoteDC != null && remoteDC.equals(localDC);
}
@ -494,10 +494,10 @@ public class OutboundConnectionSettings
}
@VisibleForTesting
static boolean shouldCompressConnection(IEndpointSnitch snitch, InetAddressAndPort localHost, InetAddressAndPort remoteHost)
static boolean shouldCompressConnection(InetAddressAndPort localHost, InetAddressAndPort remoteHost)
{
return (DatabaseDescriptor.internodeCompression() == Config.InternodeCompression.all)
|| ((DatabaseDescriptor.internodeCompression() == Config.InternodeCompression.dc) && !isInLocalDC(snitch, localHost, remoteHost));
|| ((DatabaseDescriptor.internodeCompression() == Config.InternodeCompression.dc) && !isInLocalDC(localHost, remoteHost));
}
}

4
src/java/org/apache/cassandra/repair/RepairJob.java
View File

@ -488,7 +488,7 @@ public class RepairJob extends AsyncFuture<RepairResult> implements Runnable
private String getDC(InetAddressAndPort address)
{
return ctx.snitch().getDatacenter(address);
return ctx.locator().location(address).datacenter;
}
/**
@ -573,7 +573,7 @@ public class RepairJob extends AsyncFuture<RepairResult> implements Runnable
Map<String, Queue<InetAddressAndPort>> requestsByDatacenter = new HashMap<>();
for (InetAddressAndPort endpoint : endpoints)
{
String dc = DatabaseDescriptor.getEndpointSnitch().getDatacenter(endpoint);
String dc = DatabaseDescriptor.getLocator().location(endpoint).datacenter;
Queue<InetAddressAndPort> queue = requestsByDatacenter.computeIfAbsent(dc, k -> new LinkedList<>());
queue.add(endpoint);
}

12
src/java/org/apache/cassandra/repair/SharedContext.java
View File

@ -32,8 +32,8 @@ import org.apache.cassandra.gms.FailureDetector;
import org.apache.cassandra.gms.Gossiper;
import org.apache.cassandra.gms.IFailureDetector;
import org.apache.cassandra.gms.IGossiper;
import org.apache.cassandra.locator.IEndpointSnitch;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.locator.Locator;
import org.apache.cassandra.net.MessageDelivery;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.service.ActiveRepairService;
@ -73,7 +73,7 @@ public interface SharedContext
};
}
IFailureDetector failureDetector();
IEndpointSnitch snitch();
Locator locator();
IGossiper gossiper();
ICompactionManager compactionManager();
ActiveRepairService repair();
@ -147,9 +147,9 @@ public interface SharedContext
}
@Override
public IEndpointSnitch snitch()
public Locator locator()
{
return DatabaseDescriptor.getEndpointSnitch();
return DatabaseDescriptor.getLocator();
}
@Override
@ -270,9 +270,9 @@ public interface SharedContext
}
@Override
public IEndpointSnitch snitch()
public Locator locator()
{
return delegate().snitch();
return delegate().locator();
}
@Override

6
src/java/org/apache/cassandra/schema/DistributedMetadataLogKeyspace.java
View File

@ -20,6 +20,7 @@ package org.apache.cassandra.schema;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.Collections;
import java.util.Set;
import java.util.function.Supplier;
@ -224,4 +225,9 @@ public final class DistributedMetadataLogKeyspace
{
return KeyspaceMetadata.create(SchemaConstants.METADATA_KEYSPACE_NAME, new KeyspaceParams(true, ReplicationParams.simpleMeta(1, knownDatacenters)), Tables.of(Log));
}
public static KeyspaceMetadata initialMetadata(String datacenter)
{
return initialMetadata(Collections.singleton(datacenter));
}
}

21
src/java/org/apache/cassandra/schema/DistributedSchema.java
View File

@ -62,13 +62,20 @@ public class DistributedSchema implements MetadataValue<DistributedSchema>
public static DistributedSchema first(Set<String> knownDatacenters)
{
// During upgrades from pre-5.1 versions, the replication params of the system_cluster_metadata
// keyspace using one of the existing DCs. This is so that this keyspace does not cause issues
// for tooling, clients or control plane systems which may inspect schema and have specific
// expectations about DC layout. This keyspace is unused until the CMS is initialized.
// For new clusters which start out on 5.1 or later, this is not necessary to the initial
// replication params use a empty string for the placeholder DC name.
// During CMS initialization, the replication of this keyspace will be set for real using
// the DC of the first node to become a CMS member. This happens in the PreInitialize
// transformation when executed on the first CMS member.
if (knownDatacenters.isEmpty())
{
if (DatabaseDescriptor.getLocalDataCenter() != null)
knownDatacenters = Collections.singleton(DatabaseDescriptor.getLocalDataCenter());
else
knownDatacenters = Collections.singleton("DC1");
}
knownDatacenters = Collections.singleton("");
return new DistributedSchema(Keyspaces.of(DistributedMetadataLogKeyspace.initialMetadata(knownDatacenters)), Epoch.FIRST);
}
@ -216,7 +223,7 @@ public class DistributedSchema implements MetadataValue<DistributedSchema>
});
// Avoid system table side effects during initialization
if (epoch.isEqualOrAfter(Epoch.FIRST))
if (epoch.isEqualOrAfter(Epoch.FIRST) && !DatabaseDescriptor.isClientOrToolInitialized())
{
Collection<Mutation> mutations = SchemaKeyspace.convertSchemaDiffToMutations(ksDiff, FBUtilities.timestampMicros());
SchemaKeyspace.applyChanges(mutations);

6
src/java/org/apache/cassandra/schema/ReplicationParams.java
View File

@ -18,6 +18,7 @@
package org.apache.cassandra.schema;
import java.io.IOException;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashMap;
import java.util.Map;
@ -121,6 +122,11 @@ public final class ReplicationParams
return new ReplicationParams(SimpleStrategy.class, ImmutableMap.of("replication_factor", replicationFactor));
}
public static ReplicationParams simpleMeta(int replicationFactor, String datacenter)
{
return simpleMeta(replicationFactor, Collections.singleton(datacenter));
}
public static ReplicationParams simpleMeta(int replicationFactor, Set<String> knownDatacenters)
{
if (replicationFactor <= 0)

24
src/java/org/apache/cassandra/service/CassandraDaemon.java
View File

@ -65,6 +65,11 @@ import org.apache.cassandra.exceptions.StartupException;
import org.apache.cassandra.io.util.File;
import org.apache.cassandra.io.util.FileUtils;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.locator.Locator;
import org.apache.cassandra.tcm.CMSOperations;
import org.apache.cassandra.tcm.ClusterMetadataService;
import org.apache.cassandra.tcm.RegistrationStatus;
import org.apache.cassandra.tcm.Startup;
import org.apache.cassandra.metrics.CassandraMetricsRegistry;
import org.apache.cassandra.metrics.DefaultNameFactory;
import org.apache.cassandra.net.StartupClusterConnectivityChecker;
@ -74,11 +79,8 @@ import org.apache.cassandra.security.ThreadAwareSecurityManager;
import org.apache.cassandra.service.paxos.PaxosState;
import org.apache.cassandra.service.snapshot.SnapshotManager;
import org.apache.cassandra.streaming.StreamManager;
import org.apache.cassandra.tcm.CMSOperations;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.ClusterMetadataService;
import org.apache.cassandra.tcm.MultiStepOperation;
import org.apache.cassandra.tcm.Startup;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.JMXServerUtils;
import org.apache.cassandra.utils.JVMStabilityInspector;
@ -265,6 +267,8 @@ public class CassandraDaemon
Startup.initialize(DatabaseDescriptor.getSeeds());
disableAutoCompaction(Schema.instance.distributedKeyspaces().names());
CMSOperations.initJmx();
if (ClusterMetadata.current().myNodeId() != null)
RegistrationStatus.instance.onRegistration();
}
catch (InterruptedException | ExecutionException | IOException e)
{
@ -369,9 +373,13 @@ public class CassandraDaemon
exitOrFail(1, "Fatal configuration error", e);
}
// The local rack may have been changed at some point, which will now be reflected in cluster metadata. Update
// the system.local table just in case the actual value doesn't match what the configured location provided
// reported when the earlier call to SystemKeyspace::persistLocalMetadata was made prior to initialising cluster
// metadata.
SystemKeyspace.updateRack(ClusterMetadata.current().locator.local().rack);
ScheduledExecutors.optionalTasks.execute(() -> ClusterMetadataService.instance().processor().fetchLogAndWait());
// TODO: (TM/alexp), this can be made time-dependent
// Because we are writing to the system_distributed keyspace, this should happen after that is created, which
// happens in StorageService.instance.initServer()
Runnable viewRebuild = () -> {
@ -383,11 +391,6 @@ public class CassandraDaemon
};
ScheduledExecutors.optionalTasks.schedule(viewRebuild, StorageService.RING_DELAY_MILLIS, TimeUnit.MILLISECONDS);
// TODO: (TM/alexp), we do not need to wait for gossip settlement anymore
// if (!FBUtilities.getBroadcastAddressAndPort().equals(InetAddressAndPort.getLoopbackAddress()))
// Gossiper.waitToSettle();
StorageService.instance.doAuthSetup();
// re-enable auto-compaction after replay, so correct disk boundaries are used
@ -637,8 +640,9 @@ public class CassandraDaemon
{
StartupClusterConnectivityChecker connectivityChecker = StartupClusterConnectivityChecker.create(DatabaseDescriptor.getBlockForPeersTimeoutInSeconds(),
DatabaseDescriptor.getBlockForPeersInRemoteDatacenters());
Locator locator = DatabaseDescriptor.getLocator();
Set<InetAddressAndPort> peers = new HashSet<>(ClusterMetadata.current().directory.allJoinedEndpoints());
connectivityChecker.execute(peers, DatabaseDescriptor.getEndpointSnitch()::getDatacenter);
connectivityChecker.execute(peers, ep -> locator.location(ep).datacenter);
// check to see if transports may start else return without starting. This is needed when in survey mode or
// when bootstrap has not completed.

12
src/java/org/apache/cassandra/service/DatacenterSyncWriteResponseHandler.java
View File

@ -24,7 +24,7 @@ import java.util.function.Supplier;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.Mutation;
import org.apache.cassandra.locator.IEndpointSnitch;
import org.apache.cassandra.locator.Locator;
import org.apache.cassandra.locator.NetworkTopologyStrategy;
import org.apache.cassandra.locator.Replica;
import org.apache.cassandra.locator.ReplicaPlan;
@ -38,7 +38,7 @@ import org.apache.cassandra.transport.Dispatcher;
*/
public class DatacenterSyncWriteResponseHandler<T> extends AbstractWriteResponseHandler<T>
{
private static final IEndpointSnitch snitch = DatabaseDescriptor.getEndpointSnitch();
private static final Locator locator = DatabaseDescriptor.getLocator();
private final Map<String, AtomicInteger> responses = new HashMap<String, AtomicInteger>();
private final AtomicInteger acks = new AtomicInteger(0);
@ -64,14 +64,14 @@ public class DatacenterSyncWriteResponseHandler<T> extends AbstractWriteResponse
}
else
{
responses.put(DatabaseDescriptor.getLocalDataCenter(), new AtomicInteger(ConsistencyLevel.quorumFor(replicaPlan.replicationStrategy())));
responses.put(locator.local().datacenter, new AtomicInteger(ConsistencyLevel.quorumFor(replicaPlan.replicationStrategy())));
}
// During bootstrap, we have to include the pending endpoints or we may fail the consistency level
// guarantees (see #833)
for (Replica pending : replicaPlan.pending())
{
responses.get(snitch.getDatacenter(pending)).incrementAndGet();
responses.get(locator.location(pending.endpoint()).datacenter).incrementAndGet();
}
}
@ -80,8 +80,8 @@ public class DatacenterSyncWriteResponseHandler<T> extends AbstractWriteResponse
try
{
String dataCenter = message == null
? DatabaseDescriptor.getLocalDataCenter()
: snitch.getDatacenter(message.from());
? locator.local().datacenter
: locator.location(message.from()).datacenter;
responses.get(dataCenter).getAndDecrement();
acks.incrementAndGet();

6
src/java/org/apache/cassandra/service/Rebuild.java
View File

@ -104,17 +104,17 @@ public class Rebuild
RangeStreamer streamer = new RangeStreamer(metadata,
StreamOperation.REBUILD,
false, // no strict consistency when rebuilding
DatabaseDescriptor.getEndpointSnitch(),
DatabaseDescriptor.getNodeProximity(),
StorageService.instance.streamStateStore,
false,
DatabaseDescriptor.getStreamingConnectionsPerHost(),
rebuildMovements,
null);
if (sourceDc != null)
streamer.addSourceFilter(new RangeStreamer.SingleDatacenterFilter(DatabaseDescriptor.getEndpointSnitch(), sourceDc));
streamer.addSourceFilter(new RangeStreamer.SingleDatacenterFilter(metadata.locator, sourceDc));
if (excludeLocalDatacenterNodes)
streamer.addSourceFilter(new RangeStreamer.ExcludeLocalDatacenterFilter(DatabaseDescriptor.getEndpointSnitch()));
streamer.addSourceFilter(new RangeStreamer.ExcludeLocalDatacenterFilter(metadata.locator));
if (keyspace == null)
{

42
src/java/org/apache/cassandra/service/StartupChecks.java
View File

@ -144,8 +144,6 @@ public class StartupChecks
checkDataDirs,
checkSSTablesFormat,
checkSystemKeyspaceState,
checkDatacenter,
checkRack,
checkLegacyAuthTables,
new DataResurrectionCheck());
@ -738,46 +736,6 @@ public class StartupChecks
}
};
public static final StartupCheck checkDatacenter = new StartupCheck()
{
@Override
public void execute(StartupChecksOptions options) throws StartupException
{
String storedDc = SystemKeyspace.getDatacenter();
if (storedDc != null)
{
String currentDc = DatabaseDescriptor.getEndpointSnitch().getLocalDatacenter();
if (!storedDc.equals(currentDc))
{
String formatMessage = "Cannot start node if snitch's data center (%s) differs from previous data center (%s). " +
"Please fix the snitch configuration, decommission and rebootstrap this node";
throw new StartupException(StartupException.ERR_WRONG_CONFIG, String.format(formatMessage, currentDc, storedDc));
}
}
}
};
public static final StartupCheck checkRack = new StartupCheck()
{
@Override
public void execute(StartupChecksOptions options) throws StartupException
{
String storedRack = SystemKeyspace.getRack();
if (storedRack != null)
{
String currentRack = DatabaseDescriptor.getEndpointSnitch().getLocalRack();
if (!storedRack.equals(currentRack))
{
String formatMessage = "Cannot start node if snitch's rack (%s) differs from previous rack (%s). " +
"Please fix the snitch configuration, decommission and rebootstrap this node";
throw new StartupException(StartupException.ERR_WRONG_CONFIG, String.format(formatMessage, currentRack, storedRack));
}
}
}
};
public static final StartupCheck checkLegacyAuthTables = new StartupCheck()
{
@Override

10
src/java/org/apache/cassandra/service/StorageProxy.java
View File

@ -877,7 +877,7 @@ public class StorageProxy implements StorageProxyMBean
throws UnavailableException, OverloadedException, WriteTimeoutException, WriteFailureException
{
Tracing.trace("Determining replicas for mutation");
final String localDataCenter = DatabaseDescriptor.getEndpointSnitch().getLocalDatacenter();
final String localDataCenter = DatabaseDescriptor.getLocator().local().datacenter;
List<AbstractWriteResponseHandler<IMutation>> responseHandlers = new ArrayList<>(mutations.size());
WriteType plainWriteType = mutations.size() <= 1 ? WriteType.SIMPLE : WriteType.UNLOGGED_BATCH;
@ -1011,7 +1011,7 @@ public class StorageProxy implements StorageProxyMBean
throws UnavailableException, OverloadedException, WriteTimeoutException
{
Tracing.trace("Determining replicas for mutation");
final String localDataCenter = DatabaseDescriptor.getEndpointSnitch().getLocalDatacenter();
final String localDataCenter = DatabaseDescriptor.getLocator().local().datacenter;
long startTime = nanoTime();
@ -1350,7 +1350,7 @@ public class StorageProxy implements StorageProxyMBean
private static void syncWriteBatchedMutations(List<WriteResponseHandlerWrapper> wrappers, Stage stage, Dispatcher.RequestTime requestTime)
throws WriteTimeoutException, OverloadedException
{
String localDataCenter = DatabaseDescriptor.getEndpointSnitch().getLocalDatacenter();
String localDataCenter = DatabaseDescriptor.getLocator().local().datacenter;
for (WriteResponseHandlerWrapper wrapper : wrappers)
{
@ -1528,7 +1528,7 @@ public class StorageProxy implements StorageProxyMBean
Collections.singletonList(MessageFlag.CALL_BACK_ON_FAILURE));
}
String dc = DatabaseDescriptor.getEndpointSnitch().getDatacenter(destination);
String dc = DatabaseDescriptor.getLocator().location(destination.endpoint()).datacenter;
// direct writes to local DC or old Cassandra versions
// (1.1 knows how to forward old-style String message IDs; updated to int in 2.0)
@ -2403,7 +2403,7 @@ public class StorageProxy implements StorageProxyMBean
Set<String> disabledDCs = DatabaseDescriptor.hintedHandoffDisabledDCs();
if (!disabledDCs.isEmpty())
{
final String dc = DatabaseDescriptor.getEndpointSnitch().getDatacenter(replica);
final String dc = DatabaseDescriptor.getLocator().location(replica.endpoint()).datacenter;
if (disabledDCs.contains(dc))
{
Tracing.trace("Not hinting {} since its data center {} has been disabled {}", replica, dc, disabledDCs);

58
src/java/org/apache/cassandra/service/StorageService.java
View File

@ -47,6 +47,7 @@ import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.Supplier;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import javax.annotation.Nullable;
@ -137,7 +138,6 @@ import org.apache.cassandra.locator.DynamicEndpointSnitch;
import org.apache.cassandra.locator.EndpointsByRange;
import org.apache.cassandra.locator.EndpointsForRange;
import org.apache.cassandra.locator.EndpointsForToken;
import org.apache.cassandra.locator.IEndpointSnitch;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.locator.LocalStrategy;
import org.apache.cassandra.locator.MetaStrategy;
@ -145,6 +145,8 @@ import org.apache.cassandra.locator.RangesAtEndpoint;
import org.apache.cassandra.locator.RangesByEndpoint;
import org.apache.cassandra.locator.Replica;
import org.apache.cassandra.locator.Replicas;
import org.apache.cassandra.locator.NodeProximity;
import org.apache.cassandra.locator.SnitchAdapter;
import org.apache.cassandra.locator.SystemReplicas;
import org.apache.cassandra.metrics.Sampler;
import org.apache.cassandra.metrics.SamplingManager;
@ -178,6 +180,7 @@ import org.apache.cassandra.streaming.StreamState;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.ClusterMetadataService;
import org.apache.cassandra.tcm.MultiStepOperation;
import org.apache.cassandra.tcm.RegistrationStatus;
import org.apache.cassandra.tcm.Transformation;
import org.apache.cassandra.tcm.compatibility.GossipHelper;
import org.apache.cassandra.tcm.compatibility.TokenRingUtils;
@ -821,6 +824,7 @@ public class StorageService extends NotificationBroadcasterSupport implements IE
self = Register.maybeRegister();
}
RegistrationStatus.instance.onRegistration();
Startup.maybeExecuteStartupTransformation(self);
try
@ -3429,8 +3433,8 @@ public class StorageService extends NotificationBroadcasterSupport implements IE
}
else
{
// stream to the closest peer as chosen by the snitch
candidates = DatabaseDescriptor.getEndpointSnitch().sortedByProximity(getBroadcastAddressAndPort(), candidates);
// stream to the closest peer as chosen by the configured proximity measures
candidates = DatabaseDescriptor.getNodeProximity().sortedByProximity(getBroadcastAddressAndPort(), candidates);
InetAddressAndPort hintsDestinationHost = candidates.get(0).endpoint();
return ClusterMetadata.current().directory.peerId(hintsDestinationHost).toUUID();
}
@ -4144,12 +4148,12 @@ public class StorageService extends NotificationBroadcasterSupport implements IE
public void setDynamicUpdateInterval(int dynamicUpdateInterval)
{
if (DatabaseDescriptor.getEndpointSnitch() instanceof DynamicEndpointSnitch)
if (DatabaseDescriptor.getNodeProximity() instanceof DynamicEndpointSnitch)
{
try
{
updateSnitch(null, true, dynamicUpdateInterval, null, null);
updateProximityInternal(null, true, dynamicUpdateInterval, null, null);
}
catch (ClassNotFoundException e)
{
@ -4164,6 +4168,18 @@ public class StorageService extends NotificationBroadcasterSupport implements IE
}
public void updateSnitch(String epSnitchClassName, Boolean dynamic, Integer dynamicUpdateInterval, Integer dynamicResetInterval, Double dynamicBadnessThreshold) throws ClassNotFoundException
{
Supplier<NodeProximity> factory = () -> new SnitchAdapter(DatabaseDescriptor.createEndpointSnitch(epSnitchClassName));
updateProximityInternal(factory, dynamic, dynamicUpdateInterval, dynamicResetInterval, dynamicBadnessThreshold);
}
public void updateNodeProximity(String npsClassName, Boolean dynamic, Integer dynamicUpdateInterval, Integer dynamicResetInterval, Double dynamicBadnessThreshold) throws ClassNotFoundException
{
Supplier<NodeProximity> factory = () -> DatabaseDescriptor.createProximityImpl(npsClassName);
updateProximityInternal(factory, dynamic, dynamicUpdateInterval, dynamicResetInterval, dynamicBadnessThreshold);
}
private void updateProximityInternal(Supplier<NodeProximity> implSupplier, Boolean dynamic, Integer dynamicUpdateInterval, Integer dynamicResetInterval, Double dynamicBadnessThreshold) throws ClassNotFoundException
{
// apply dynamic snitch configuration
if (dynamicUpdateInterval != null)
@ -4173,51 +4189,53 @@ public class StorageService extends NotificationBroadcasterSupport implements IE
if (dynamicBadnessThreshold != null)
DatabaseDescriptor.setDynamicBadnessThreshold(dynamicBadnessThreshold);
IEndpointSnitch oldSnitch = DatabaseDescriptor.getEndpointSnitch();
NodeProximity oldProximity = DatabaseDescriptor.getNodeProximity();
// new snitch registers mbean during construction
if(epSnitchClassName != null)
if(implSupplier != null)
{
// need to unregister the mbean _before_ the new dynamic snitch is instantiated (and implicitly initialized
// and its mbean registered)
if (oldSnitch instanceof DynamicEndpointSnitch)
((DynamicEndpointSnitch)oldSnitch).close();
if (oldProximity instanceof DynamicEndpointSnitch)
((DynamicEndpointSnitch)oldProximity).close();
IEndpointSnitch newSnitch;
NodeProximity newProximity;
try
{
newSnitch = DatabaseDescriptor.createEndpointSnitch(dynamic != null && dynamic, epSnitchClassName);
newProximity = implSupplier.get();
if (dynamic != null && dynamic)
newProximity = new DynamicEndpointSnitch(newProximity);
}
catch (ConfigurationException e)
{
throw new ClassNotFoundException(e.getMessage());
}
if (newSnitch instanceof DynamicEndpointSnitch)
if (newProximity instanceof DynamicEndpointSnitch)
{
logger.info("Created new dynamic snitch {} with update-interval={}, reset-interval={}, badness-threshold={}",
((DynamicEndpointSnitch)newSnitch).subsnitch.getClass().getName(), DatabaseDescriptor.getDynamicUpdateInterval(),
((DynamicEndpointSnitch)newProximity).delegate.getClass().getName(), DatabaseDescriptor.getDynamicUpdateInterval(),
DatabaseDescriptor.getDynamicResetInterval(), DatabaseDescriptor.getDynamicBadnessThreshold());
}
else
{
logger.info("Created new non-dynamic snitch {}", newSnitch.getClass().getName());
logger.info("Created new non-dynamic snitch {}", newProximity.getClass().getName());
}
// point snitch references to the new instance
DatabaseDescriptor.setEndpointSnitch(newSnitch);
// point reference to the new instance
DatabaseDescriptor.setNodeProximity(newProximity);
}
else
{
if (oldSnitch instanceof DynamicEndpointSnitch)
if (oldProximity instanceof DynamicEndpointSnitch)
{
logger.info("Applying config change to dynamic snitch {} with update-interval={}, reset-interval={}, badness-threshold={}",
((DynamicEndpointSnitch)oldSnitch).subsnitch.getClass().getName(), DatabaseDescriptor.getDynamicUpdateInterval(),
((DynamicEndpointSnitch)oldProximity).delegate.getClass().getName(), DatabaseDescriptor.getDynamicUpdateInterval(),
DatabaseDescriptor.getDynamicResetInterval(), DatabaseDescriptor.getDynamicBadnessThreshold());
DynamicEndpointSnitch snitch = (DynamicEndpointSnitch)oldSnitch;
snitch.applyConfigChanges();
DynamicEndpointSnitch proximity = (DynamicEndpointSnitch)oldProximity;
proximity.applyConfigChanges();
}
}
}

22
src/java/org/apache/cassandra/service/StorageServiceMBean.java
View File

@ -652,15 +652,35 @@ public interface StorageServiceMBean extends NotificationEmitter
*
* The parameters {@code dynamicUpdateInterval}, {@code dynamicResetInterval} and {@code dynamicBadnessThreshold}
* can be specified individually to update the parameters of the dynamic snitch during runtime.
*
* @param epSnitchClassName the canonical path name for a class implementing IEndpointSnitch
* @param dynamic boolean that decides whether dynamicsnitch is used or not - only valid, if {@code epSnitchClassName} is specified
* @param dynamicUpdateInterval integer, in ms (defaults to the value configured in cassandra.yaml, which defaults to 100)
* @param dynamicResetInterval integer, in ms (defaults to the value configured in cassandra.yaml, which defaults to 600,000)
* @param dynamicBadnessThreshold double, (defaults to the value configured in cassandra.yaml, which defaults to 0.0)
* @deprecated See CASSANDRA-19488
*/
@Deprecated(since = "5.1")
public void updateSnitch(String epSnitchClassName, Boolean dynamic, Integer dynamicUpdateInterval, Integer dynamicResetInterval, Double dynamicBadnessThreshold) throws ClassNotFoundException;
/**
* Change NodeProximity class and dynamic-ness (and dynamic attributes) at runtime.
* This method supercedes the deprecated updateSnitch method.
*
* This method is used to change the proximity implementation and/or dynamic proximity parameters.
* If {@code proximityClassName} is specified, it will configure a new instance and make it a
* 'dynamic snitch' if {@code dynamic} is specified and {@code true}.
*
* The parameters {@code dynamicUpdateInterval}, {@code dynamicResetInterval} and {@code dynamicBadnessThreshold}
* can be specified individually to update the parameters of the dynamic snitch during runtime.
*
* @param proximityClassName the canonical path name for a class implementing NodeProximity
* @param dynamic boolean that decides whether dynamicsnitch is used or not - only valid, if {@code epSnitchClassName} is specified
* @param dynamicUpdateInterval integer, in ms (defaults to the value configured in cassandra.yaml, which defaults to 100)
* @param dynamicResetInterval integer, in ms (defaults to the value configured in cassandra.yaml, which defaults to 600,000)
* @param dynamicBadnessThreshold double, (defaults to the value configured in cassandra.yaml, which defaults to 0.0)
*/
public void updateNodeProximity(String proximityClassName, Boolean dynamic, Integer dynamicUpdateInterval, Integer dynamicResetInterval, Double dynamicBadnessThreshold) throws ClassNotFoundException;
/*
Update dynamic_snitch_update_interval in ms
*/

12
src/java/org/apache/cassandra/service/TokenRange.java
View File

@ -22,8 +22,9 @@ import java.util.*;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.dht.Range;
import org.apache.cassandra.dht.Token;
import org.apache.cassandra.locator.IEndpointSnitch;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.locator.Locator;
import org.apache.cassandra.tcm.membership.Location;
/**
* Holds token range informations for the sake of {@link StorageService#describeRing}.
@ -57,12 +58,15 @@ public class TokenRange
public static TokenRange create(Token.TokenFactory tokenFactory, Range<Token> range, List<InetAddressAndPort> endpoints, boolean withPorts)
{
List<EndpointDetails> details = new ArrayList<>(endpoints.size());
IEndpointSnitch snitch = DatabaseDescriptor.getEndpointSnitch();
Locator locator = DatabaseDescriptor.getLocator();
for (InetAddressAndPort ep : endpoints)
{
Location location = locator.location(ep);
details.add(new EndpointDetails(ep,
StorageService.instance.getNativeaddress(ep, withPorts),
snitch.getDatacenter(ep),
snitch.getRack(ep)));
location.datacenter,
location.rack));
}
return new TokenRange(tokenFactory, range, details);
}

4
src/java/org/apache/cassandra/service/paxos/PaxosCommit.java
View File

@ -33,6 +33,7 @@ import org.apache.cassandra.exceptions.RequestFailureReason;
import org.apache.cassandra.locator.EndpointsForToken;
import org.apache.cassandra.locator.InOurDc;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.locator.Locator;
import org.apache.cassandra.locator.Replica;
import org.apache.cassandra.net.IVerbHandler;
import org.apache.cassandra.net.Message;
@ -214,7 +215,8 @@ public class PaxosCommit<OnDone extends Consumer<? super PaxosCommit.Status>> ex
private static boolean isInLocalDc(InetAddressAndPort destination)
{
return DatabaseDescriptor.getLocalDataCenter().equals(DatabaseDescriptor.getEndpointSnitch().getDatacenter(destination));
Locator locator = DatabaseDescriptor.getLocator();
return locator.local().sameDatacenter(locator.location(destination));
}
/**

7
src/java/org/apache/cassandra/service/paxos/PaxosRepair.java
View File

@ -550,12 +550,13 @@ public class PaxosRepair extends AbstractPaxosRepair
ReplicationParams replication = keyspace.getMetadata().params.replication;
// Special case meta keyspace as it uses a custom partitioner/tokens, but the paxos table and repairs
// are based on the system partitioner
ClusterMetadata metadata = ClusterMetadata.current();
Collection<InetAddressAndPort> allEndpoints = replication.isMeta()
? ClusterMetadata.current().fullCMSMembers()
: ClusterMetadata.current().placements.get(replication).reads.forRange(range).endpoints();
? metadata.fullCMSMembers()
: metadata.placements.get(replication).reads.forRange(range).endpoints();
return hasSufficientLiveNodesForTopologyChange(allEndpoints,
liveEndpoints,
DatabaseDescriptor.getEndpointSnitch()::getDatacenter,
ep -> metadata.locator.location(ep).datacenter,
DatabaseDescriptor.paxoTopologyRepairNoDcChecks(),
DatabaseDescriptor.paxoTopologyRepairStrictEachQuorum());
}

2
src/java/org/apache/cassandra/service/reads/ReadCallback.java
View File

@ -258,7 +258,7 @@ public class ReadCallback<E extends Endpoints<E>, P extends ReplicaPlan.ForRead<
private void assertWaitingFor(InetAddressAndPort from)
{
assert !replicaPlan().consistencyLevel().isDatacenterLocal()
|| DatabaseDescriptor.getLocalDataCenter().equals(DatabaseDescriptor.getEndpointSnitch().getDatacenter(from))
|| DatabaseDescriptor.getLocator().local().sameDatacenter(DatabaseDescriptor.getLocator().location(from))
: "Received read response from unexpected replica: " + from;
}
}

4
src/java/org/apache/cassandra/service/reads/range/ReplicaPlanMerger.java
View File

@ -27,6 +27,7 @@ import org.apache.cassandra.db.ConsistencyLevel;
import org.apache.cassandra.db.Keyspace;
import org.apache.cassandra.locator.ReplicaPlan;
import org.apache.cassandra.locator.ReplicaPlans;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.utils.AbstractIterator;
class ReplicaPlanMerger extends AbstractIterator<ReplicaPlan.ForRangeRead>
@ -49,6 +50,7 @@ class ReplicaPlanMerger extends AbstractIterator<ReplicaPlan.ForRangeRead>
return endOfData();
ReplicaPlan.ForRangeRead current = ranges.next();
ClusterMetadata metadata = ClusterMetadata.current();
// getRestrictedRange has broken the queried range into per-[vnode] token ranges, but this doesn't take
// the replication factor into account. If the intersection of live endpoints for 2 consecutive ranges
@ -64,7 +66,7 @@ class ReplicaPlanMerger extends AbstractIterator<ReplicaPlan.ForRangeRead>
break;
ReplicaPlan.ForRangeRead next = ranges.peek();
ReplicaPlan.ForRangeRead merged = ReplicaPlans.maybeMerge(keyspace, consistency, current, next);
ReplicaPlan.ForRangeRead merged = ReplicaPlans.maybeMerge(metadata, keyspace, consistency, current, next);
if (merged == null)
break;

4
src/java/org/apache/cassandra/streaming/DataMovementVerbHandler.java
View File

@ -57,7 +57,9 @@ public class DataMovementVerbHandler implements IVerbHandler<DataMovement>
assert local.isSelf();
boolean transientAdded = false;
boolean fullAdded = false;
for (Replica remote : DatabaseDescriptor.getEndpointSnitch().sortedByProximity(local.endpoint(), endpoints).filter(ep -> FailureDetector.instance.isAlive(ep.endpoint())))
for (Replica remote : DatabaseDescriptor.getNodeProximity()
.sortedByProximity(local.endpoint(), endpoints)
.filter(ep -> FailureDetector.instance.isAlive(ep.endpoint())))
{
assert !remote.isSelf();
if (remote.isFull() && !fullAdded)

7
src/java/org/apache/cassandra/streaming/StreamManager.java
View File

@ -114,9 +114,10 @@ public class StreamManager implements StreamManagerMBean
this.interDCLimiter = interDCLimiter;
this.throughput = throughput;
this.interDCThroughput = interDCThroughput;
if (DatabaseDescriptor.getLocalDataCenter() != null && DatabaseDescriptor.getEndpointSnitch() != null)
isLocalDC = DatabaseDescriptor.getLocalDataCenter().equals(
DatabaseDescriptor.getEndpointSnitch().getDatacenter(peer));
if (DatabaseDescriptor.getLocalDataCenter() != null && DatabaseDescriptor.getLocator() != null)
isLocalDC = DatabaseDescriptor.getLocator()
.local()
.sameDatacenter(DatabaseDescriptor.getLocator().location(peer));
else
isLocalDC = true;
}

5
src/java/org/apache/cassandra/tcm/ClusterMetadata.java
View File

@ -46,6 +46,7 @@ import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.locator.EndpointsForRange;
import org.apache.cassandra.locator.EndpointsForToken;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.locator.Locator;
import org.apache.cassandra.locator.MetaStrategy;
import org.apache.cassandra.locator.Replica;
import org.apache.cassandra.net.CMSIdentifierMismatchException;
@ -96,6 +97,9 @@ public class ClusterMetadata
public final InProgressSequences inProgressSequences;
public final ImmutableMap<ExtensionKey<?,?>, ExtensionValue<?>> extensions;
// This isn't serialized as part of ClusterMetadata it's really just a view over the Directory.
public final Locator locator;
// These fields are lazy but only for the test purposes, since their computation requires initialization of the log ks
private EndpointsForRange fullCMSReplicas;
private Set<InetAddressAndPort> fullCMSEndpoints;
@ -174,6 +178,7 @@ public class ClusterMetadata
this.lockedRanges = lockedRanges;
this.inProgressSequences = inProgressSequences;
this.extensions = ImmutableMap.copyOf(extensions);
this.locator = Locator.usingDirectory(directory);
}
public Set<InetAddressAndPort> fullCMSMembers()

10
src/java/org/apache/cassandra/tcm/ClusterMetadataService.java
View File

@ -98,6 +98,9 @@ public class ClusterMetadataService
throw new IllegalStateException(String.format("Cluster metadata is already initialized to %s.", instance),
trace);
instance = newInstance;
RegistrationStatus.instance.onInitialized();
if (newInstance.metadata().myNodeId() != null)
RegistrationStatus.instance.onRegistration();
trace = new RuntimeException("Previously initialized trace");
}
@ -105,6 +108,7 @@ public class ClusterMetadataService
public static ClusterMetadataService unsetInstance()
{
ClusterMetadataService tmp = instance();
RegistrationStatus.instance.resetState();
instance = null;
return tmp;
}
@ -245,7 +249,8 @@ public class ClusterMetadataService
{
if (instance != null)
return;
ClusterMetadata emptyFromSystemTables = emptyWithSchemaFromSystemTables(Collections.singleton("DC1"))
String localDC = DatabaseDescriptor.getLocalDataCenter();
ClusterMetadata emptyFromSystemTables = emptyWithSchemaFromSystemTables(Collections.singleton(localDC))
.forceEpoch(Epoch.EMPTY);
LocalLog.LogSpec logSpec = LocalLog.logSpec()
@ -274,7 +279,7 @@ public class ClusterMetadataService
new PeerLogFetcher(log));
log.readyUnchecked();
log.bootstrap(FBUtilities.getBroadcastAddressAndPort());
log.bootstrap(FBUtilities.getBroadcastAddressAndPort(), localDC);
ClusterMetadataService.setInstance(cms);
}
@ -292,6 +297,7 @@ public class ClusterMetadataService
if (instance != null)
return;
ClusterMetadataService.setInstance(StubClusterMetadataService.forClientTools(initialSchema));
}

67
src/java/org/apache/cassandra/tcm/RegistrationStatus.java Normal file
View File

@ -0,0 +1,67 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.tcm;
import java.util.concurrent.atomic.AtomicReference;
import com.google.common.annotations.VisibleForTesting;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.net.MessagingService;
public class RegistrationStatus
{
public enum State { INITIAL, UNREGISTERED, REGISTERED }
private static final Logger logger = LoggerFactory.getLogger(RegistrationStatus.class);
public static final RegistrationStatus instance = new RegistrationStatus();
private final AtomicReference<RegistrationStatus.State> state = new AtomicReference<>(State.INITIAL);
public RegistrationStatus.State getCurrent()
{
return state.get();
}
@VisibleForTesting
public void resetState()
{
state.set(State.INITIAL);
}
public void onInitialized()
{
logger.info("Node is initialized, moving to UNREGISTERED state");
if (!state.compareAndSet(State.INITIAL, State.UNREGISTERED))
throw new IllegalStateException(String.format("Cannot move to UNREGISTERED state (%s)", state.get()));
}
public void onRegistration()
{
// This may have been done already if the metadata log replay at start up included our registration
RegistrationStatus.State current = state.get();
if (current == State.REGISTERED)
return;
logger.info("This node is registered, moving state to REGISTERED and interrupting any previously established peer connections");
state.getAndSet(RegistrationStatus.State.REGISTERED);
MessagingService.instance().channelManagers.keySet().forEach(MessagingService.instance()::interruptOutbound);
}
}

3
src/java/org/apache/cassandra/tcm/Startup.java
View File

@ -135,7 +135,8 @@ import static org.apache.cassandra.utils.FBUtilities.getBroadcastAddressAndPort;
public static void initializeAsFirstCMSNode()
{
InetAddressAndPort addr = FBUtilities.getBroadcastAddressAndPort();
ClusterMetadataService.instance().log().bootstrap(addr);
String datacenter = DatabaseDescriptor.getLocator().local().datacenter;
ClusterMetadataService.instance().log().bootstrap(addr, datacenter);
ClusterMetadata metadata = ClusterMetadata.current();
assert ClusterMetadataService.state() == LOCAL : String.format("Can't initialize as node hasn't transitioned to CMS state. State: %s.\n%s", ClusterMetadataService.state(), metadata);

4
src/java/org/apache/cassandra/tcm/StubClusterMetadataService.java
View File

@ -44,7 +44,6 @@ public class StubClusterMetadataService extends ClusterMetadataService
public static StubClusterMetadataService forClientTools()
{
DatabaseDescriptor.setLocalDataCenter("DC1");
KeyspaceMetadata ks = DistributedMetadataLogKeyspace.initialMetadata(Collections.singleton("DC1"));
DistributedSchema schema = new DistributedSchema(Keyspaces.of(ks));
return new StubClusterMetadataService(new ClusterMetadata(DatabaseDescriptor.getPartitioner(),
@ -54,7 +53,6 @@ public class StubClusterMetadataService extends ClusterMetadataService
public static StubClusterMetadataService forClientTools(DistributedSchema initialSchema)
{
DatabaseDescriptor.setLocalDataCenter("DC1");
ClusterMetadata metadata = new ClusterMetadata(DatabaseDescriptor.getPartitioner());
metadata = metadata.transformer().with(initialSchema).build().metadata.forceEpoch(Epoch.EMPTY);
return new StubClusterMetadataService(metadata);
@ -167,6 +165,7 @@ public class StubClusterMetadataService extends ClusterMetadataService
public StubClusterMetadataService build()
{
if (initial == null)
{
initial = new ClusterMetadata(Epoch.EMPTY,
partitioner,
DistributedSchema.empty(),
@ -176,6 +175,7 @@ public class StubClusterMetadataService extends ClusterMetadataService
LockedRanges.EMPTY,
InProgressSequences.EMPTY,
ImmutableMap.of());
}
return new StubClusterMetadataService(new UniformRangePlacement(),
snapshots != null ? snapshots : MetadataSnapshots.NO_OP,
LocalLog.logSpec().withInitialState(initial).createLog(),

2
src/java/org/apache/cassandra/tcm/compatibility/TokenRingUtils.java
View File

@ -157,7 +157,7 @@ public class TokenRingUtils
public static Collection<Range<Token>> getPrimaryRangeForEndpointWithinDC(String keyspace, InetAddressAndPort referenceEndpoint)
{
ClusterMetadata metadata = ClusterMetadata.current();
String localDC = DatabaseDescriptor.getEndpointSnitch().getDatacenter(referenceEndpoint);
String localDC = DatabaseDescriptor.getLocator().location(referenceEndpoint).datacenter;
Collection<InetAddressAndPort> localDcNodes = metadata.directory.datacenterEndpoints(localDC);
AbstractReplicationStrategy strategy = Keyspace.open(keyspace).getReplicationStrategy();

12
src/java/org/apache/cassandra/tcm/listeners/LegacyStateListener.java
View File

@ -34,6 +34,7 @@ import org.apache.cassandra.db.virtual.PeersTable;
import org.apache.cassandra.dht.Token;
import org.apache.cassandra.gms.Gossiper;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.schema.Schema;
import org.apache.cassandra.service.StorageService;
import org.apache.cassandra.tcm.ClusterMetadata;
@ -50,6 +51,7 @@ import static org.apache.cassandra.tcm.membership.NodeState.BOOTSTRAPPING;
import static org.apache.cassandra.tcm.membership.NodeState.BOOT_REPLACING;
import static org.apache.cassandra.tcm.membership.NodeState.LEFT;
import static org.apache.cassandra.tcm.membership.NodeState.MOVING;
import static org.apache.cassandra.tcm.membership.NodeState.REGISTERED;
public class LegacyStateListener implements ChangeListener.Async
{
@ -115,8 +117,14 @@ public class LegacyStateListener implements ChangeListener.Async
Gossiper.instance.addLocalApplicationState(SCHEMA, StorageService.instance.valueFactory.schema(next.schema.getVersion()));
}
if (next.directory.peerState(change) == LEFT)
if (next.directory.peerState(change) == REGISTERED)
{
// Re-establish any connections made prior to this node registering
InetAddressAndPort endpoint = next.directory.endpoint(change);
logger.info("Peer with address {} has registered, interrupting any previously established connections", endpoint);
MessagingService.instance().interruptOutbound(endpoint);
}
else if (next.directory.peerState(change) == LEFT)
{
Gossiper.instance.mergeNodeToGossip(change, next, prev.tokenMap.tokens(change));
InetAddressAndPort endpoint = prev.directory.endpoint(change);

3
src/java/org/apache/cassandra/tcm/listeners/SchemaListener.java
View File

@ -26,6 +26,7 @@ import org.apache.cassandra.schema.Schema;
import org.apache.cassandra.schema.SchemaDiagnostics;
import org.apache.cassandra.service.StorageService;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.Epoch;
public class SchemaListener implements ChangeListener
{
@ -44,7 +45,7 @@ public class SchemaListener implements ChangeListener
protected void notifyInternal(ClusterMetadata prev, ClusterMetadata next, boolean fromSnapshot, boolean loadSSTables)
{
if (!fromSnapshot && next.schema.lastModified().equals(prev.schema.lastModified()))
if (next.epoch.isAfter(Epoch.FIRST) && !fromSnapshot && next.schema.lastModified().equals(prev.schema.lastModified()))
return;
next.schema.initializeKeyspaceInstances(prev.schema, loadSSTables);
}

18
src/java/org/apache/cassandra/tcm/log/LocalLog.java
View File

@ -279,14 +279,24 @@ public abstract class LocalLog implements Closeable
entryFilters = Lists.newCopyOnWriteArrayList();
}
public void bootstrap(InetAddressAndPort addr)
@VisibleForTesting
public void unsafeBootstrapForTesting(InetAddressAndPort addr)
{
bootstrap(addr, "");
}
/**
*
* @param addr
* @param datacenter
*/
public void bootstrap(InetAddressAndPort addr, String datacenter)
{
ClusterMetadata metadata = metadata();
assert metadata.epoch.isBefore(FIRST) : String.format("Metadata epoch %s should be before first", metadata.epoch);
Transformation transform = PreInitialize.withFirstCMS(addr);
Transformation transform = PreInitialize.withFirstCMS(addr, datacenter);
append(new Entry(Entry.Id.NONE, FIRST, transform));
waitForHighestConsecutive();
metadata = metadata();
metadata = waitForHighestConsecutive();
assert metadata.epoch.is(Epoch.FIRST) : String.format("Epoch: %s. CMS: %s", metadata.epoch, metadata.fullCMSMembers());
}

1
src/java/org/apache/cassandra/tcm/membership/Directory.java
View File

@ -38,6 +38,7 @@ import org.apache.cassandra.db.TypeSizes;
import org.apache.cassandra.io.util.DataInputPlus;
import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.tcm.Epoch;
import org.apache.cassandra.tcm.MetadataValue;
import org.apache.cassandra.tcm.serialization.MetadataSerializer;

6
src/java/org/apache/cassandra/tcm/membership/Location.java
View File

@ -30,6 +30,7 @@ import org.apache.cassandra.tcm.serialization.Version;
public class Location
{
public static final Serializer serializer = new Serializer();
public static final Location UNKNOWN = new Location("UNKNOWN_DC", "UNKNOWN_RACK");
public final String datacenter;
public final String rack;
@ -40,6 +41,11 @@ public class Location
this.rack = rack;
}
public boolean sameDatacenter(Location other)
{
return datacenter.equals(other.datacenter);
}
@Override
public boolean equals(Object o)
{

1
src/java/org/apache/cassandra/tcm/membership/NodeAddresses.java
View File

@ -84,7 +84,6 @@ public class NodeAddresses
public boolean conflictsWith(NodeAddresses other)
{
return broadcastAddress.equals(other.broadcastAddress) ||
localAddress.equals(other.localAddress) ||
nativeAddress.equals(other.nativeAddress);
}

5
src/java/org/apache/cassandra/tcm/sequences/Move.java
View File

@ -367,8 +367,9 @@ public class Move extends MultiStepOperation<Epoch>
// if we are not running with strict consistency, try to find other sources for streaming
if (needsRelaxedSources.get())
{
for (Replica source : DatabaseDescriptor.getEndpointSnitch().sortedByProximity(FBUtilities.getBroadcastAddressAndPort(),
oldOwners.forRange(destination.range()).get()))
for (Replica source : DatabaseDescriptor.getNodeProximity()
.sortedByProximity(FBUtilities.getBroadcastAddressAndPort(),
oldOwners.forRange(destination.range()).get()))
{
if (fd.isAlive(source.endpoint()) && !source.endpoint().equals(destination.endpoint()))
{

1
src/java/org/apache/cassandra/tcm/serialization/Version.java
View File

@ -48,6 +48,7 @@ public enum Version
V4(4),
/**
* - AlterSchema includes execution timestamp
* - PreInitialize includes datacenter (affects local serialization on first CMS node only)
*/
V5(5),

Some files were not shown because too many files have changed in this diff Show More