mirror of https://github.com/apache/cassandra
SSTable format API
Summary of the changes:
Format, reader and writer
---------------------------
There are a lot of refactorings around sstable related classes aiming to extract the most generic functionality to the top-level entities and push down implementation-specific stuff to the actual implementation. In Particular, the top-level, implementation agnostic classes/interfaces are SSTableFormat interface, SSTable, SSTableReader, SSTableWriter, IVerifier, and IScrubber. The rest of the codebase has been reviewed for explicit usages of big table format-specific usages of sstable classes and refactored. SSTable, SSTableReader, and SSTableWriter have their builders. Builders make a hierarchy that follows the same inheritance structure as readers and writers.
There are also partial implementations that add support for some features and may or may not be used by the custom implementations. They include:
- AbstractSSTableFormat - adds an implementation of some initialization methods - in practice, all of the format implementations should extend this class
- SSTableReaderWithFilter - add support for Bloom filter to the reader
- SortedTableWriter - generic implementation for a writer which writes partitions in the default order to the data file, supports Bloom filter and some index of partitions
- IndexSummarySupport - interface implemented by the readers using index summaries
- KeyCacheSupport - interface implemented by the readers using row key cache
Descriptor
---------------------------
Refactored the Descriptor class so that:
- All paths are created from the base directory File rather than from a String
- All the methods named *filename* producing full paths were made private; their current implementations are returning file names rather than paths (the naming was inconsistent)
- The usages of the `filenameFor` method were refactored to use the `fileFor` method
- The usages of the `fromFilename` method were refactored to use a `fromFileWithComponent(..., false).left` expression
In essence, the Descriptor class is no longer working on String-based paths.
Index summaries
---------------------------
Removed the index summary from the generic SSTableReader class and created an interface IndexSummarySupport to be implemented by the readers that need it. Methods in related classes that refer back to the reader were refactored to support just readers of the SSTableReader & IndexSummarySupport type. Therefore, we will no longer need to assume that the generic SSTableReader has anything to do with an index summary.
A new IndexSummaryComponent class encloses data fields from the index summary file (note that aside from the index summary itself, the file includes the first and last partition of the sstable). The class has been extracted to deal with those fields and have that logic in a single place.
Filter
---------------------------
Refactored IFilter and its serialization - in particular, added the `serialize` method to the IFilter interface and moved loading/saving logic to a separate utility class FilterComponent.
Extracted the SSTableReaderWithFilter abstract reader extending the generic SSTableReader with filter support.
Extracted bloom filter metrics into separate entities allowing to plug them in if the implementation uses a filter.
Cache
---------------------------
Refactored CacheService to support different key-cache values. CacheService now supports arbitrary IRowIndexEntry implementation as a key-cache value. A new version of the auto-saving cache was created ("g") because some information about the type of serialized row index entry needs to be known before it is deserialized (or skipped). Therefore, the SSTableFormat type ordinal number is stored, which is sufficient because the IRowIndexEntry serializer is specific to the sstable format type.
Similarly to the IndexSummarySupport, a new KeyCacheSupport interface has to be implemented to mark the reader as supporting key-cache. It contains the default implementation of several methods the rest of the system relies on when the key-cache is supported.
Other changes
---------------------------
- Fixed disabling chunk cache - enable(boolean) method in ChunkCache does not make any sense - it makes a false impression it can disable chunk cache once enabled, while in fact, it only clears it. Added setFileCacheEnabled to DatabaseDescriptor
- Made WrappingUnfilteredRowIterator an interface
- DataInputStreamPlus extends InputStream - this makes it possible for input stream-based inheritors of DataInputPlus to extend DataInputStreamPlus. It simplifies coding because sometimes we want to get DataInputPlus implementation extending InputStream as an argument.
- Table and keyspace metrics were made pluggable - in particular, added the ability for a certain format to register gauges that are specific only to that format and make no sense for others
- Implemented mmapped region extension for compressed data
- Refactored FileHandle so that it is no longer closable
- Implemented WrappingRebufferer
- Introduced the SSTable.Owner interface to make SSTable implementation not reference higher-level entities directly. SSTable accepts passing null as the owner when there is no owner (like sometimes in offline tools) or passing a mock when needed in tests.
Individual commits
---------------------------
[4a87cd36fe] Fix disabling chunk cache
[c84c75ccf3] Made WrappingUnfilteredRowIterator an interface
[253d2b828e] Add getType to SSTableFormat
[3f169dcc20] Remove getIndexSerializer from SSTableFormat
[05bae1833b] Pull down rowIndexEntrySerializer field
[da675f2809] Moved RowIndexEntry
[673f0c5c39] Reduce usages of RowIndexEntry
[c72538be91] Refactor CacheService to support for different key cache values
[54d33ee656] Minor refactoring of ColumnIndex
[93862df967] Just moved AbstractSSTableIterator to o.a.c.io.sstable.format
[9e4566a1de] Refactored AbstractSSTableIterator
[a4e61e80bb] Extracted IScrubber and IVerifier interfaces
[20f78c7419] Push down implementation of SSTableReader.firstKeyBeyond
[f2c24e5774] Moved SSTableReader.getSampleIndexesForRanges to IndexSummary
[b6c3a6c1ea] Moved SSTableReader.getKeySamples implementation to IndexSummary
[c4b90ebb33] Refactor InstanceTidier so that it is more generic
[918d5a9e74] Refactor dropping page cache
[a52fb4d558] Refactor sstable metrics
[f6d10f930f] NEW (fix up) - DataInputStreamPlus extends InputStream
[8f6a56d972] Getting rid of index summary in SSTableReader
[4a918bf725] Removed direct usages of primary index from SSTableReader
[358fa32602] Refactor KeyIterator so that it is sstable format agnostic
[14c09d89c2] Remove explicit usage of Components outside of format specific classes
[feff14e137] Move clone methods implementation from SSTableReader to BigTableReader
[64e9787b10] Move saveIndexSummary and saveBloomFilter to SSTableReaderBuilder
[ae71fe6ed8] Moved indexSummary field to BigTableReader and made it private
[df9fd8c4b9] Moved ifile field to BigTableReader and made it private
[2be6ea9ecf] Moved static open methods for BigTableReader to the reader factory
[bc0e55ac48] Minor refactoring around IFilter and its serialization
[5b95704beb] Minor refactorings around IndexSummary
[87812335e8] Extracted TOCComponent class to deal with TOC file
[fdad092a6a] Extracted CompressionInfoComponent class
[39b47e388d] Extracted StatsComponent as a helper for elements of SSTable metadata
[cdb55bff47] Fix SSTable.getMinimalKey
[b99c6d5805] Refactor FileHandle so that it is no longer closable
[77b7f7ace5] Implement WrappingRebufferer
[b6868914dd] Add progressPercentage to ProgressInfo
[7fd4956e5b] Moved copy/rename/hardLink methods from SSTableWriter to SSTable
[1ccc6bf148] Create generic SSTableBuilder and IOOptions
[da58a81102] Refactor SSTableReaderBuilder
[4501ddba1c] Refactor ColumnIndex
[d4f9e1a64b] Extracted non-big-table-specific functionality from BigTableWriter to SortedTableWriter
[379525d01e] Refactor BigTableZeroCopyWriter to SSTableZeroCopyWriter as it is not specific to big format
[8ac37f83bc] Extract EmptySSTableScanner out from BigTableScanner
[ee6673f1cf] Implement SSTableWriterBuilder
[bb26629235] Refactor opening early / final
[a327595015] Refactored SSTableWriter factory
[16ffd7334b] Extract non-big-format-specific logic from scrubber and verifier
[75e02db6af] Allow to specify the default SSTableFormat via system property
[a7b9d0d628] Small fixes around streaming
[407f977c36] Move guard collection size
[0529e57d2f] Remove explicit references to big format
[61509963ec] Unclassified minor changes
[da28d1af3a] Replaced getCreationTimeFor(Component) with getDataCreationTime()
[e99c834de6] !!! Reformatting
[882b7baa5a] Rename SSTableReader.maybePresent and fix its redundant usages
[b70c983bea] Implement mmapped region extension for compressed data
[d7ff3970de] Introduce SSTable.Owner interface
[e9feb9c462] Replaced getCreationTimeFor(Component) with getDataCreationTime()
[ee8082fb07] Created SSTableFormat.deleteOrphanedComponents
[e62950fd3d] Refactor metrics further
[cefa5b3814] Extract key cache support into separate entity
[dd55101ca1] Extracted SSTableReaderWithFilter
[510b651824] Implement customizable component types
[2be512d9fa] Pluggable SSTableFormat by making SSTableFormat.Type not an enum
[670836b55d] Refactor CRC and digest validators
[00c91103bc] Extract delete method to delete SSTables and purge row cache entries
[0819dc9fc2] Extracted trySkipFileCacheBefore(key) to SSTableReader
[732f841750] Added missing overrides in ForwardingSSTableReader
[db623218fd] Update DatabaseDescriptorRefTest
[c018c468e5] Cleanup
[eafc836242] Add @SuppressWarnings("resource") where needed
[3b7c911dd6] Documentation
patch by Jacek Lewandowski, reviewed by Branimir Lambov for CASSANDRA-17056
Co-authored-by: @jacek-lewandowski
Co-authored-by: @blambov
This commit is contained in:
Jacek Lewandowski
Jacek Lewandowski
parent
0a0e06847b
commit
b7e1e44a90
|
|
@ -1,4 +1,5 @@
|
|||
4.2
|
||||
* CEP-17: SSTable API (CASSANDRA-17056)
|
||||
* Gossip stateMapOrdering does not have correct ordering when both EndpointState are in the bootstrapping set (CASSANDRA-18292)
|
||||
* Snapshot only sstables containing mismatching ranges on preview repair mismatch (CASSANDRA-17561)
|
||||
* More accurate skipping of sstables in read path (CASSANDRA-18134)
|
||||
|
|
|
|||
4
NEWS.txt
4
NEWS.txt
|
|
@ -125,7 +125,8 @@ New features
|
|||
- On virtual tables, it is not strictly necessary to specify `ALLOW FILTERING` for select statements which would
|
||||
normally require it, except `system_views.system_logs`.
|
||||
- More accurate skipping of sstables in read path due to better handling of min/max clustering and lower bound;
|
||||
SSTable format has been bumped to 'nc' because there are new fields in stats metadata
|
||||
SSTable format has been bumped to 'nc' because there are new fields in stats metadata\
|
||||
- SSTal
|
||||
|
||||
Upgrading
|
||||
---------
|
||||
|
|
@ -140,6 +141,7 @@ Upgrading
|
|||
upgrades involving 3.x and 4.x nodes. The fix for that issue makes it can now appear during rolling upgrades from
|
||||
4.1.0 or 4.0.0-4.0.7. If that is your case, please use protocol v4 or higher in your driver. See CASSANDRA-17507
|
||||
for further details.
|
||||
- Added API for alternative sstable implementations. For details, see src/java/org/apache/cassandra/io/sstable/SSTable_API.md
|
||||
|
||||
Deprecation
|
||||
-----------
|
||||
|
|
|
|||
|
|
@ -1910,3 +1910,14 @@ drop_compact_storage_enabled: false
|
|||
# excluded_keyspaces: # comma separated list of keyspaces to exclude from the check
|
||||
# excluded_tables: # comma separated list of keyspace.table pairs to exclude from the check
|
||||
|
||||
# Supported sstable formats
|
||||
# This is a list of elements consisting of class_name and parameters, where class_name should point to the class
|
||||
# implementing org.apache.cassandra.io.sstable.format.SSTableFormat. Parameters must include unique 'id' integer
|
||||
# which is used in some serialization to denote the format type in a compact way (such as local key cache); and 'name'
|
||||
# which will be used to recognize the format type - in particular that name will be used in sstable file names and in
|
||||
# stream headers so the name has to be the same for the same format across all the nodes in the cluster.
|
||||
sstable_formats:
|
||||
- class_name: org.apache.cassandra.io.sstable.format.big.BigFormat
|
||||
parameters:
|
||||
id: 0
|
||||
name: big
|
||||
|
|
|
|||
|
|
@ -17,13 +17,13 @@
|
|||
*/
|
||||
package org.apache.cassandra.cache;
|
||||
|
||||
import java.io.BufferedInputStream;
|
||||
import java.io.FileNotFoundException;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.OutputStream;
|
||||
import java.nio.file.NoSuchFileException;
|
||||
import java.util.*;
|
||||
import java.util.ArrayDeque;
|
||||
import java.util.Iterator;
|
||||
import java.util.Set;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.ScheduledFuture;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
|
|
@ -33,19 +33,28 @@ import org.slf4j.LoggerFactory;
|
|||
|
||||
import org.apache.cassandra.concurrent.ExecutorPlus;
|
||||
import org.apache.cassandra.concurrent.ScheduledExecutors;
|
||||
import org.apache.cassandra.schema.TableId;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.config.DatabaseDescriptor;
|
||||
import org.apache.cassandra.schema.Schema;
|
||||
import org.apache.cassandra.schema.SchemaConstants;
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.compaction.CompactionInfo;
|
||||
import org.apache.cassandra.db.compaction.CompactionInfo.Unit;
|
||||
import org.apache.cassandra.db.compaction.CompactionManager;
|
||||
import org.apache.cassandra.db.compaction.OperationType;
|
||||
import org.apache.cassandra.db.compaction.CompactionInfo.Unit;
|
||||
import org.apache.cassandra.io.FSWriteError;
|
||||
import org.apache.cassandra.io.util.*;
|
||||
import org.apache.cassandra.io.util.ChecksummedRandomAccessReader;
|
||||
import org.apache.cassandra.io.util.ChecksummedSequentialWriter;
|
||||
import org.apache.cassandra.io.util.CorruptFileException;
|
||||
import org.apache.cassandra.io.util.DataInputPlus;
|
||||
import org.apache.cassandra.io.util.DataInputPlus.DataInputStreamPlus;
|
||||
import org.apache.cassandra.io.util.DataOutputPlus;
|
||||
import org.apache.cassandra.io.util.DataOutputStreamPlus;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.io.util.FileUtils;
|
||||
import org.apache.cassandra.io.util.SequentialWriterOption;
|
||||
import org.apache.cassandra.io.util.WrappedDataOutputStreamPlus;
|
||||
import org.apache.cassandra.schema.Schema;
|
||||
import org.apache.cassandra.schema.SchemaConstants;
|
||||
import org.apache.cassandra.schema.TableId;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.service.CacheService;
|
||||
import org.apache.cassandra.utils.JVMStabilityInspector;
|
||||
import org.apache.cassandra.utils.Pair;
|
||||
|
|
@ -59,8 +68,9 @@ public class AutoSavingCache<K extends CacheKey, V> extends InstrumentingCache<K
|
|||
{
|
||||
public interface IStreamFactory
|
||||
{
|
||||
InputStream getInputStream(File dataPath, File crcPath) throws IOException;
|
||||
OutputStream getOutputStream(File dataPath, File crcPath);
|
||||
DataInputStreamPlus getInputStream(File dataPath, File crcPath) throws IOException;
|
||||
|
||||
DataOutputStreamPlus getOutputStream(File dataPath, File crcPath);
|
||||
}
|
||||
|
||||
private static final Logger logger = LoggerFactory.getLogger(AutoSavingCache.class);
|
||||
|
|
@ -85,8 +95,10 @@ public class AutoSavingCache<K extends CacheKey, V> extends InstrumentingCache<K
|
|||
* "e" introduced with CASSANDRA-11206, omits IndexInfo from key-cache, stores offset into index-file
|
||||
*
|
||||
* "f" introduced with CASSANDRA-9425, changes "keyspace.table.index" in cache keys to TableMetadata.id+TableMetadata.indexName
|
||||
*
|
||||
* "g" introduced an explicit sstable format type ordinal number so that the entry can be skipped regardless of the actual implementation and used serializer
|
||||
*/
|
||||
private static final String CURRENT_VERSION = "f";
|
||||
private static final String CURRENT_VERSION = "g";
|
||||
|
||||
private static volatile IStreamFactory streamFactory = new IStreamFactory()
|
||||
{
|
||||
|
|
@ -95,12 +107,12 @@ public class AutoSavingCache<K extends CacheKey, V> extends InstrumentingCache<K
|
|||
.trickleFsyncByteInterval(DatabaseDescriptor.getTrickleFsyncIntervalInKiB() * 1024)
|
||||
.finishOnClose(true).build();
|
||||
|
||||
public InputStream getInputStream(File dataPath, File crcPath) throws IOException
|
||||
public DataInputStreamPlus getInputStream(File dataPath, File crcPath) throws IOException
|
||||
{
|
||||
return ChecksummedRandomAccessReader.open(dataPath, crcPath);
|
||||
}
|
||||
|
||||
public OutputStream getOutputStream(File dataPath, File crcPath)
|
||||
public DataOutputStreamPlus getOutputStream(File dataPath, File crcPath)
|
||||
{
|
||||
return new ChecksummedSequentialWriter(dataPath, crcPath, null, writerOption);
|
||||
}
|
||||
|
|
@ -175,6 +187,7 @@ public class AutoSavingCache<K extends CacheKey, V> extends InstrumentingCache<K
|
|||
return cacheLoad;
|
||||
}
|
||||
|
||||
@SuppressWarnings("resource")
|
||||
public int loadSaved()
|
||||
{
|
||||
int count = 0;
|
||||
|
|
@ -189,15 +202,15 @@ public class AutoSavingCache<K extends CacheKey, V> extends InstrumentingCache<K
|
|||
try
|
||||
{
|
||||
logger.info("reading saved cache {}", dataPath);
|
||||
in = new DataInputStreamPlus(new LengthAvailableInputStream(new BufferedInputStream(streamFactory.getInputStream(dataPath, crcPath)), dataPath.length()));
|
||||
in = streamFactory.getInputStream(dataPath, crcPath);
|
||||
|
||||
//Check the schema has not changed since CFs are looked up by name which is ambiguous
|
||||
UUID schemaVersion = new UUID(in.readLong(), in.readLong());
|
||||
if (!schemaVersion.equals(Schema.instance.getVersion()))
|
||||
throw new RuntimeException("Cache schema version "
|
||||
+ schemaVersion
|
||||
+ " does not match current schema version "
|
||||
+ Schema.instance.getVersion());
|
||||
+ schemaVersion
|
||||
+ " does not match current schema version "
|
||||
+ Schema.instance.getVersion());
|
||||
|
||||
ArrayDeque<Future<Pair<K, V>>> futures = new ArrayDeque<>();
|
||||
long loadByNanos = start + TimeUnit.SECONDS.toNanos(DatabaseDescriptor.getCacheLoadTimeout());
|
||||
|
|
@ -268,7 +281,7 @@ public class AutoSavingCache<K extends CacheKey, V> extends InstrumentingCache<K
|
|||
}
|
||||
if (logger.isTraceEnabled())
|
||||
logger.trace("completed reading ({} ms; {} keys) saved cache {}",
|
||||
TimeUnit.NANOSECONDS.toMillis(nanoTime() - start), count, dataPath);
|
||||
TimeUnit.NANOSECONDS.toMillis(nanoTime() - start), count, dataPath);
|
||||
return count;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -23,15 +23,22 @@ package org.apache.cassandra.cache;
|
|||
import java.nio.ByteBuffer;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import com.google.common.base.Throwables;
|
||||
import com.google.common.collect.Iterables;
|
||||
|
||||
import com.github.benmanes.caffeine.cache.*;
|
||||
import com.github.benmanes.caffeine.cache.CacheLoader;
|
||||
import com.github.benmanes.caffeine.cache.Caffeine;
|
||||
import com.github.benmanes.caffeine.cache.LoadingCache;
|
||||
import com.github.benmanes.caffeine.cache.RemovalCause;
|
||||
import com.github.benmanes.caffeine.cache.RemovalListener;
|
||||
import org.apache.cassandra.concurrent.ImmediateExecutor;
|
||||
import org.apache.cassandra.config.DatabaseDescriptor;
|
||||
import org.apache.cassandra.io.sstable.CorruptSSTableException;
|
||||
import org.apache.cassandra.io.util.*;
|
||||
import org.apache.cassandra.io.util.ChannelProxy;
|
||||
import org.apache.cassandra.io.util.ChunkReader;
|
||||
import org.apache.cassandra.io.util.FileHandle;
|
||||
import org.apache.cassandra.io.util.Rebufferer;
|
||||
import org.apache.cassandra.io.util.RebuffererFactory;
|
||||
import org.apache.cassandra.metrics.ChunkCacheMetrics;
|
||||
import org.apache.cassandra.utils.memory.BufferPool;
|
||||
import org.apache.cassandra.utils.memory.BufferPools;
|
||||
|
|
@ -170,7 +177,7 @@ public class ChunkCache
|
|||
cache.invalidateAll();
|
||||
}
|
||||
|
||||
private RebuffererFactory wrap(ChunkReader file)
|
||||
public RebuffererFactory wrap(ChunkReader file)
|
||||
{
|
||||
return new CachingRebufferer(file);
|
||||
}
|
||||
|
|
@ -196,14 +203,6 @@ public class ChunkCache
|
|||
cache.invalidateAll(Iterables.filter(cache.asMap().keySet(), x -> x.path.equals(fileName)));
|
||||
}
|
||||
|
||||
@VisibleForTesting
|
||||
public void enable(boolean enabled)
|
||||
{
|
||||
ChunkCache.enabled = enabled;
|
||||
cache.invalidateAll();
|
||||
metrics.reset();
|
||||
}
|
||||
|
||||
// TODO: Invalidate caches for obsoleted/MOVED_START tables?
|
||||
|
||||
/**
|
||||
|
|
@ -319,4 +318,4 @@ public class ChunkCache
|
|||
.map(policy -> policy.weightedSize().orElseGet(cache::estimatedSize))
|
||||
.orElseGet(cache::estimatedSize);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -341,6 +341,9 @@ public enum CassandraRelevantProperties
|
|||
SEED_COUNT_WARN_THRESHOLD("cassandra.seed_count_warn_threshold"),
|
||||
|
||||
|
||||
SSTABLE_FORMAT_DEFAULT("cassandra.sstable.format.default"),
|
||||
|
||||
|
||||
/** When enabled, recursive directory deletion will be executed using a unix command `rm -rf` instead of traversing
|
||||
* and removing individual files. This is now used only tests, but eventually we will make it true by default.*/
|
||||
USE_NIX_RECURSIVE_DELETE("cassandra.use_nix_recursive_delete"),
|
||||
|
|
@ -356,6 +359,7 @@ public enum CassandraRelevantProperties
|
|||
* can be also done manually for that particular case: {@code flush(SchemaConstants.SCHEMA_KEYSPACE_NAME);}. */
|
||||
FLUSH_LOCAL_SCHEMA_CHANGES("cassandra.test.flush_local_schema_changes", "true"),
|
||||
|
||||
TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS("cassandra.streaminghistogram.roundseconds", "60"),
|
||||
;
|
||||
|
||||
CassandraRelevantProperties(String key, String defaultVal)
|
||||
|
|
@ -561,6 +565,21 @@ public enum CassandraRelevantProperties
|
|||
return Enum.valueOf(defaultValue.getDeclaringClass(), toUppercase ? value.toUpperCase() : value);
|
||||
}
|
||||
|
||||
/**
|
||||
* Gets the value of a system property as an enum, optionally calling {@link String#toUpperCase()} first.
|
||||
* If the value is missing, the default value for this property is used
|
||||
*
|
||||
* @param toUppercase before converting to enum
|
||||
* @param enumClass enumeration class
|
||||
* @param <T> type
|
||||
* @return enum value
|
||||
*/
|
||||
public <T extends Enum<T>> T getEnum(boolean toUppercase, Class<T> enumClass)
|
||||
{
|
||||
String value = System.getProperty(key, defaultVal);
|
||||
return Enum.valueOf(enumClass, toUppercase ? value.toUpperCase() : value);
|
||||
}
|
||||
|
||||
/**
|
||||
* Sets the value into system properties.
|
||||
* @param value to set
|
||||
|
|
|
|||
|
|
@ -21,25 +21,27 @@ import java.lang.reflect.Field;
|
|||
import java.lang.reflect.Modifier;
|
||||
import java.util.Collections;
|
||||
import java.util.HashMap;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.HashSet;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.TreeMap;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
import javax.annotation.Nullable;
|
||||
|
||||
import com.google.common.base.Joiner;
|
||||
import com.google.common.collect.ImmutableList;
|
||||
import com.google.common.collect.ImmutableMap;
|
||||
import com.google.common.collect.ImmutableSet;
|
||||
import com.google.common.collect.Sets;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.apache.cassandra.audit.AuditLogOptions;
|
||||
import org.apache.cassandra.db.ConsistencyLevel;
|
||||
import org.apache.cassandra.fql.FullQueryLoggerOptions;
|
||||
import org.apache.cassandra.io.sstable.format.big.BigFormat;
|
||||
import org.apache.cassandra.service.StartupChecks.StartupCheckType;
|
||||
|
||||
/**
|
||||
|
|
@ -70,6 +72,9 @@ public class Config
|
|||
*/
|
||||
public static final String PROPERTY_PREFIX = "cassandra.";
|
||||
|
||||
public static final String SSTABLE_FORMAT_ID = "id";
|
||||
public static final String SSTABLE_FORMAT_NAME = "name";
|
||||
|
||||
public String cluster_name = "Test Cluster";
|
||||
public String authenticator;
|
||||
public String authorizer;
|
||||
|
|
@ -352,6 +357,10 @@ public class Config
|
|||
|
||||
public String[] data_file_directories = new String[0];
|
||||
|
||||
public List<ParameterizedClass> sstable_formats = ImmutableList.of(new ParameterizedClass(BigFormat.class.getName(),// "org.apache.cassandra.io.sstable.format.big.BigFormat",
|
||||
ImmutableMap.of(SSTABLE_FORMAT_ID, "0",
|
||||
SSTABLE_FORMAT_NAME, "big")));
|
||||
|
||||
/**
|
||||
* The directory to use for storing the system keyspaces data.
|
||||
* If unspecified the data will be stored in the first of the data_file_directories.
|
||||
|
|
|
|||
|
|
@ -33,6 +33,8 @@ import java.util.ArrayList;
|
|||
import java.util.Collection;
|
||||
import java.util.Comparator;
|
||||
import java.util.Enumeration;
|
||||
import java.util.HashMap;
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
|
|
@ -43,12 +45,14 @@ import java.util.concurrent.TimeUnit;
|
|||
import java.util.function.Function;
|
||||
import java.util.function.Supplier;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
import java.util.stream.Collectors;
|
||||
import javax.annotation.Nullable;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import com.google.common.base.Preconditions;
|
||||
import com.google.common.collect.ImmutableMap;
|
||||
import com.google.common.collect.ImmutableSet;
|
||||
import com.google.common.collect.Lists;
|
||||
import com.google.common.primitives.Ints;
|
||||
import com.google.common.primitives.Longs;
|
||||
import com.google.common.util.concurrent.RateLimiter;
|
||||
|
|
@ -78,6 +82,7 @@ import org.apache.cassandra.exceptions.ConfigurationException;
|
|||
import org.apache.cassandra.fql.FullQueryLoggerOptions;
|
||||
import org.apache.cassandra.gms.IFailureDetector;
|
||||
import org.apache.cassandra.io.FSWriteError;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat;
|
||||
import org.apache.cassandra.io.util.DiskOptimizationStrategy;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.io.util.FileUtils;
|
||||
|
|
@ -185,6 +190,8 @@ public class DatabaseDescriptor
|
|||
private static GuardrailsOptions guardrails;
|
||||
private static StartupChecksOptions startupChecksOptions;
|
||||
|
||||
private static Map<String, Supplier<SSTableFormat<?, ?>>> sstableFormatFactories;
|
||||
|
||||
private static Function<CommitLog, AbstractCommitLogSegmentManager> commitLogSegmentMgrProvider = c -> DatabaseDescriptor.isCDCEnabled()
|
||||
? new CommitLogSegmentManagerCDC(c, DatabaseDescriptor.getCommitLogLocation())
|
||||
: new CommitLogSegmentManagerStandard(c, DatabaseDescriptor.getCommitLogLocation());
|
||||
|
|
@ -247,6 +254,8 @@ public class DatabaseDescriptor
|
|||
|
||||
setConfig(loadConfig());
|
||||
|
||||
applySSTableFormats();
|
||||
|
||||
applySimpleConfig();
|
||||
|
||||
applyPartitioner();
|
||||
|
|
@ -264,6 +273,14 @@ public class DatabaseDescriptor
|
|||
clientInitialization(true);
|
||||
}
|
||||
|
||||
/**
|
||||
* Equivalent to {@link #clientInitialization(boolean) clientInitialization(true, Config::new)}.
|
||||
*/
|
||||
public static void clientInitialization(boolean failIfDaemonOrTool)
|
||||
{
|
||||
clientInitialization(failIfDaemonOrTool, Config::new);
|
||||
}
|
||||
|
||||
/**
|
||||
* Initializes this class as a client, which means that just an empty configuration will
|
||||
* be used.
|
||||
|
|
@ -272,7 +289,7 @@ public class DatabaseDescriptor
|
|||
* {@link #toolInitialization()} has been performed before, an
|
||||
* {@link AssertionError} will be thrown.
|
||||
*/
|
||||
public static void clientInitialization(boolean failIfDaemonOrTool)
|
||||
public static void clientInitialization(boolean failIfDaemonOrTool, Supplier<Config> configSupplier)
|
||||
{
|
||||
if (!failIfDaemonOrTool && (daemonInitialized || toolInitialized))
|
||||
{
|
||||
|
|
@ -291,8 +308,9 @@ public class DatabaseDescriptor
|
|||
clientInitialized = true;
|
||||
setDefaultFailureDetector();
|
||||
Config.setClientMode(true);
|
||||
conf = new Config();
|
||||
conf = configSupplier.get();
|
||||
diskOptimizationStrategy = new SpinningDiskOptimizationStrategy();
|
||||
applySSTableFormats();
|
||||
}
|
||||
|
||||
public static boolean isClientInitialized()
|
||||
|
|
@ -379,6 +397,8 @@ public class DatabaseDescriptor
|
|||
private static void applyAll() throws ConfigurationException
|
||||
{
|
||||
//InetAddressAndPort cares that applySimpleConfig runs first
|
||||
applySSTableFormats();
|
||||
|
||||
applySimpleConfig();
|
||||
|
||||
applyPartitioner();
|
||||
|
|
@ -1330,6 +1350,74 @@ public class DatabaseDescriptor
|
|||
paritionerName = partitioner.getClass().getCanonicalName();
|
||||
}
|
||||
|
||||
@VisibleForTesting
|
||||
public static Map<String, Supplier<SSTableFormat<?, ?>>> loadSSTableFormatFactories(List<ParameterizedClass> configuredFormats)
|
||||
{
|
||||
ImmutableMap.Builder<String, Supplier<SSTableFormat<?, ?>>> sstableFormatFactories = ImmutableMap.builderWithExpectedSize(configuredFormats.size());
|
||||
Set<String> names = new HashSet<>(configuredFormats.size());
|
||||
Set<Integer> ids = new HashSet<>(configuredFormats.size());
|
||||
|
||||
for (ParameterizedClass formatConfig : configuredFormats)
|
||||
{
|
||||
assert formatConfig.parameters != null;
|
||||
Map<String, String> params = new HashMap<>(formatConfig.parameters);
|
||||
|
||||
String name = params.get(Config.SSTABLE_FORMAT_NAME);
|
||||
if (name == null)
|
||||
throw new ConfigurationException("Missing 'name' parameter in sstable format configuration for " + formatConfig.class_name);
|
||||
if (!name.matches("^[a-z]+$"))
|
||||
throw new ConfigurationException("'name' parameter in sstable format configuration for " + formatConfig.class_name + " must be non-empty, lower-case letters only string");
|
||||
if (names.contains(name))
|
||||
throw new ConfigurationException("Name '" + name + "' of sstable format " + formatConfig.class_name + " is already defined for another sstable format");
|
||||
params.remove(Config.SSTABLE_FORMAT_NAME);
|
||||
|
||||
String idString = params.get(Config.SSTABLE_FORMAT_ID);
|
||||
if (idString == null)
|
||||
throw new ConfigurationException("Missing 'id' parameter in sstable format configuration for " + formatConfig.class_name);
|
||||
int id;
|
||||
try
|
||||
{
|
||||
id = Integer.parseInt(idString);
|
||||
}
|
||||
catch (RuntimeException ex)
|
||||
{
|
||||
throw new ConfigurationException("'id' parameter in sstable format configuration for " + formatConfig.class_name + " must be an integer");
|
||||
}
|
||||
if (id < 0 || id > 127)
|
||||
throw new ConfigurationException("'id' parameter in sstable format configuration for " + formatConfig.class_name + " must be within bounds [0..127] range");
|
||||
if (ids.contains(id))
|
||||
throw new ConfigurationException("ID '" + id + "' of sstable format " + formatConfig.class_name + " is already defined for another sstable format");
|
||||
params.remove(Config.SSTABLE_FORMAT_ID);
|
||||
|
||||
Supplier<SSTableFormat<?, ?>> factory = () -> {
|
||||
Class<SSTableFormat<?, ?>> cls = FBUtilities.classForName(formatConfig.class_name, "sstable format");
|
||||
if (!SSTableFormat.class.isAssignableFrom(cls))
|
||||
throw new ConfigurationException(String.format("Class %s for sstable format %s does not implement %s", formatConfig.class_name, name, SSTableFormat.class.getName()));
|
||||
|
||||
SSTableFormat<?, ?> sstableFormat = FBUtilities.instanceOrConstruct(cls.getName(), "sstable format");
|
||||
sstableFormat.setup(id, name, params);
|
||||
return sstableFormat;
|
||||
};
|
||||
sstableFormatFactories.put(name, factory);
|
||||
names.add(name);
|
||||
ids.add(id);
|
||||
}
|
||||
|
||||
return sstableFormatFactories.build();
|
||||
}
|
||||
|
||||
private static void applySSTableFormats()
|
||||
{
|
||||
if (sstableFormatFactories != null)
|
||||
logger.warn("Reinitializing SSTableFactories - this should happen only in tests");
|
||||
|
||||
sstableFormatFactories = loadSSTableFormatFactories(conf.sstable_formats);
|
||||
|
||||
Iterable<SSTableFormat.Type> types = SSTableFormat.Type.values(); // make sure we know where those types get initialized
|
||||
types.forEach(t -> t.info.allComponents()); // make sure to reach all supported components for a type so that we know all of them are registered
|
||||
logger.info("Supported sstable formats are: {}", Lists.newArrayList(types).stream().map(f -> f.name + " -> " + f.info.getClass().getName() + " with singleton components: " + f.info.allComponents()).collect(Collectors.joining(", ")));
|
||||
}
|
||||
|
||||
/**
|
||||
* Computes the sum of the 2 specified positive values returning {@code Long.MAX_VALUE} if the sum overflow.
|
||||
*
|
||||
|
|
@ -3208,6 +3296,12 @@ public class DatabaseDescriptor
|
|||
return conf.file_cache_enabled;
|
||||
}
|
||||
|
||||
@VisibleForTesting
|
||||
public static void setFileCacheEnabled(boolean enabled)
|
||||
{
|
||||
conf.file_cache_enabled = enabled;
|
||||
}
|
||||
|
||||
public static int getFileCacheSizeInMiB()
|
||||
{
|
||||
if (conf.file_cache_size == null)
|
||||
|
|
@ -4578,4 +4672,9 @@ public class DatabaseDescriptor
|
|||
{
|
||||
conf.client_request_size_metrics_enabled = enabled;
|
||||
}
|
||||
|
||||
public static Map<String, Supplier<SSTableFormat<?, ?>>> getSSTableFormatFactories()
|
||||
{
|
||||
return Objects.requireNonNull(sstableFormatFactories, "Forgot to initialize DatabaseDescriptor?");
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -88,17 +88,16 @@ import org.apache.cassandra.db.compaction.CompactionInfo;
|
|||
import org.apache.cassandra.db.compaction.CompactionManager;
|
||||
import org.apache.cassandra.db.compaction.CompactionStrategyManager;
|
||||
import org.apache.cassandra.db.compaction.OperationType;
|
||||
import org.apache.cassandra.db.compaction.Verifier;
|
||||
import org.apache.cassandra.db.filter.ClusteringIndexFilter;
|
||||
import org.apache.cassandra.db.filter.DataLimits;
|
||||
import org.apache.cassandra.db.memtable.Flushing;
|
||||
import org.apache.cassandra.db.memtable.Memtable;
|
||||
import org.apache.cassandra.db.memtable.ShardBoundaries;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleNewTracker;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
|
||||
import org.apache.cassandra.db.lifecycle.SSTableSet;
|
||||
import org.apache.cassandra.db.lifecycle.Tracker;
|
||||
import org.apache.cassandra.db.lifecycle.View;
|
||||
import org.apache.cassandra.db.memtable.Flushing;
|
||||
import org.apache.cassandra.db.memtable.Memtable;
|
||||
import org.apache.cassandra.db.memtable.ShardBoundaries;
|
||||
import org.apache.cassandra.db.partitions.CachedPartition;
|
||||
import org.apache.cassandra.db.partitions.PartitionUpdate;
|
||||
import org.apache.cassandra.db.repair.CassandraTableRepairManager;
|
||||
|
|
@ -120,17 +119,19 @@ import org.apache.cassandra.io.FSReadError;
|
|||
import org.apache.cassandra.io.FSWriteError;
|
||||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.IScrubber;
|
||||
import org.apache.cassandra.io.sstable.IVerifier;
|
||||
import org.apache.cassandra.io.sstable.SSTable;
|
||||
import org.apache.cassandra.io.sstable.SSTableId;
|
||||
import org.apache.cassandra.io.sstable.SSTableIdFactory;
|
||||
import org.apache.cassandra.io.sstable.SSTableMultiWriter;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat.Components;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.Version;
|
||||
import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.io.util.FileOutputStreamPlus;
|
||||
import org.apache.cassandra.io.util.FileUtils;
|
||||
import org.apache.cassandra.metrics.Sampler;
|
||||
import org.apache.cassandra.metrics.Sampler.Sample;
|
||||
import org.apache.cassandra.metrics.Sampler.SamplerType;
|
||||
|
|
@ -186,7 +187,7 @@ import static org.apache.cassandra.utils.Throwables.maybeFail;
|
|||
import static org.apache.cassandra.utils.Throwables.merge;
|
||||
import static org.apache.cassandra.utils.concurrent.CountDownLatch.newCountDownLatch;
|
||||
|
||||
public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
|
||||
public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner, SSTable.Owner
|
||||
{
|
||||
private static final Logger logger = LoggerFactory.getLogger(ColumnFamilyStore.class);
|
||||
|
||||
|
|
@ -507,7 +508,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
|
|||
if (data.loadsstables)
|
||||
{
|
||||
Directories.SSTableLister sstableFiles = directories.sstableLister(Directories.OnTxnErr.IGNORE).skipTemporary(true);
|
||||
sstables = SSTableReader.openAll(sstableFiles.list().entrySet(), metadata);
|
||||
sstables = SSTableReader.openAll(this, sstableFiles.list().entrySet(), metadata);
|
||||
data.addInitialSSTablesWithoutUpdatingSize(sstables);
|
||||
}
|
||||
|
||||
|
|
@ -795,19 +796,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
|
|||
}
|
||||
}
|
||||
|
||||
File dataFile = new File(desc.filenameFor(Component.DATA));
|
||||
if (components.contains(Component.DATA) && dataFile.length() > 0)
|
||||
// everything appears to be in order... moving on.
|
||||
continue;
|
||||
|
||||
// missing the DATA file! all components are orphaned
|
||||
logger.warn("Removing orphans for {}: {}", desc, components);
|
||||
for (Component component : components)
|
||||
{
|
||||
File file = new File(desc.filenameFor(component));
|
||||
if (file.exists())
|
||||
FileUtils.deleteWithConfirm(desc.filenameFor(component));
|
||||
}
|
||||
desc.getFormat().deleteOrphanedComponents(desc, components);
|
||||
}
|
||||
|
||||
// cleanup incomplete saved caches
|
||||
|
|
@ -889,7 +878,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
|
|||
sstableIdGenerator.get(),
|
||||
descriptor.formatType);
|
||||
}
|
||||
while (newDescriptor.fileFor(Component.DATA).exists());
|
||||
while (newDescriptor.fileFor(Components.DATA).exists());
|
||||
return newDescriptor;
|
||||
}
|
||||
|
||||
|
|
@ -949,7 +938,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
|
|||
name,
|
||||
sstableIdGenerator.get(),
|
||||
format);
|
||||
assert !newDescriptor.fileFor(Component.DATA).exists();
|
||||
assert !newDescriptor.fileFor(Components.DATA).exists();
|
||||
return newDescriptor;
|
||||
}
|
||||
|
||||
|
|
@ -1530,7 +1519,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
|
|||
View view = data.getView();
|
||||
|
||||
List<SSTableReader> sortedByFirst = Lists.newArrayList(sstables);
|
||||
Collections.sort(sortedByFirst, (o1, o2) -> o1.first.compareTo(o2.first));
|
||||
sortedByFirst.sort(SSTableReader.sstableComparator);
|
||||
|
||||
List<AbstractBounds<PartitionPosition>> bounds = new ArrayList<>();
|
||||
DecoratedKey first = null, last = null;
|
||||
|
|
@ -1547,21 +1536,21 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
|
|||
{
|
||||
if (first == null)
|
||||
{
|
||||
first = sstable.first;
|
||||
last = sstable.last;
|
||||
first = sstable.getFirst();
|
||||
last = sstable.getLast();
|
||||
}
|
||||
else
|
||||
{
|
||||
if (sstable.first.compareTo(last) <= 0) // we do overlap
|
||||
if (sstable.getFirst().compareTo(last) <= 0) // we do overlap
|
||||
{
|
||||
if (sstable.last.compareTo(last) > 0)
|
||||
last = sstable.last;
|
||||
if (sstable.getLast().compareTo(last) > 0)
|
||||
last = sstable.getLast();
|
||||
}
|
||||
else
|
||||
{
|
||||
bounds.add(AbstractBounds.bounds(first, true, last, true));
|
||||
first = sstable.first;
|
||||
last = sstable.last;
|
||||
first = sstable.getFirst();
|
||||
last = sstable.getLast();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1669,13 +1658,13 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
|
|||
return CompactionManager.instance.performCleanup(ColumnFamilyStore.this, jobs);
|
||||
}
|
||||
|
||||
public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, boolean skipCorrupted, boolean checkData, boolean reinsertOverflowedTTL, int jobs) throws ExecutionException, InterruptedException
|
||||
public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, IScrubber.Options options, int jobs) throws ExecutionException, InterruptedException
|
||||
{
|
||||
return scrub(disableSnapshot, skipCorrupted, reinsertOverflowedTTL, false, checkData, jobs);
|
||||
return scrub(disableSnapshot, false, options, jobs);
|
||||
}
|
||||
|
||||
@VisibleForTesting
|
||||
public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, boolean skipCorrupted, boolean reinsertOverflowedTTL, boolean alwaysFail, boolean checkData, int jobs) throws ExecutionException, InterruptedException
|
||||
public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, boolean alwaysFail, IScrubber.Options options, int jobs) throws ExecutionException, InterruptedException
|
||||
{
|
||||
// skip snapshot creation during scrub, SEE JIRA 5891
|
||||
if(!disableSnapshot)
|
||||
|
|
@ -1687,7 +1676,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
|
|||
|
||||
try
|
||||
{
|
||||
return CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted, checkData, reinsertOverflowedTTL, jobs);
|
||||
return CompactionManager.instance.performScrub(ColumnFamilyStore.this, options, jobs);
|
||||
}
|
||||
catch(Throwable t)
|
||||
{
|
||||
|
|
@ -1722,7 +1711,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
|
|||
return true;
|
||||
}
|
||||
|
||||
public CompactionManager.AllSSTableOpStatus verify(Verifier.Options options) throws ExecutionException, InterruptedException
|
||||
public CompactionManager.AllSSTableOpStatus verify(IVerifier.Options options) throws ExecutionException, InterruptedException
|
||||
{
|
||||
return CompactionManager.instance.performVerify(ColumnFamilyStore.this, options);
|
||||
}
|
||||
|
|
@ -1730,7 +1719,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
|
|||
/**
|
||||
* Rewrites all SSTables according to specified parameters
|
||||
*
|
||||
* @param skipIfCurrentVersion - if {@link true}, will rewrite only SSTables that have version older than the current one ({@link org.apache.cassandra.io.sstable.format.big.BigFormat#latestVersion})
|
||||
* @param skipIfCurrentVersion - if {@link true}, will rewrite only SSTables that have version older than the current one ({@link SSTableFormat#getLatestVersion()})
|
||||
* @param skipIfNewerThanTimestamp - max timestamp (local creation time) for SSTable; SSTables created _after_ this timestamp will be excluded from compaction
|
||||
* @param skipIfCompressionMatches - if {@link true}, will rewrite only SSTables whose compression parameters are different from {@link TableMetadata#params#getCompressionParameters()} ()}
|
||||
* @param jobs number of jobs for parallel execution
|
||||
|
|
@ -1973,7 +1962,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
|
|||
for (SSTableReader sstr : select(View.select(SSTableSet.LIVE, dk)).sstables)
|
||||
{
|
||||
// check if the key actually exists in this sstable, without updating cache and stats
|
||||
if (sstr.getPosition(dk, SSTableReader.Operator.EQ, false) != null)
|
||||
if (sstr.getPosition(dk, SSTableReader.Operator.EQ, false) >= 0)
|
||||
mapped.add(mapper.apply(sstr));
|
||||
}
|
||||
return mapped;
|
||||
|
|
@ -2136,7 +2125,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
|
|||
|
||||
private List<String> mapToDataFilenames(Collection<SSTableReader> sstables)
|
||||
{
|
||||
return sstables.stream().map(s -> s.descriptor.relativeFilenameFor(Component.DATA)).collect(Collectors.toList());
|
||||
return sstables.stream().map(s -> s.descriptor.relativeFilenameFor(Components.DATA)).collect(Collectors.toList());
|
||||
}
|
||||
|
||||
private void writeSnapshotSchema(File schemaFile)
|
||||
|
|
@ -2194,7 +2183,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
|
|||
if (logger.isTraceEnabled())
|
||||
logger.trace("using snapshot sstable {}", entries.getKey());
|
||||
// open offline so we don't modify components or track hotness.
|
||||
sstable = SSTableReader.open(entries.getKey(), entries.getValue(), metadata, true, true);
|
||||
sstable = SSTableReader.open(this, entries.getKey(), entries.getValue(), metadata, true, true);
|
||||
refs.tryRef(sstable);
|
||||
// release the self ref as we never add the snapshot sstable to DataTracker where it is otherwise released
|
||||
sstable.selfRef().release();
|
||||
|
|
@ -2944,6 +2933,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
|
|||
}
|
||||
|
||||
|
||||
@Override
|
||||
public Double getCrcCheckChance()
|
||||
{
|
||||
return crcCheckChance.value();
|
||||
|
|
@ -3289,10 +3279,10 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
|
|||
DecoratedKey last = null;
|
||||
for (SSTableReader sstable : sstables)
|
||||
{
|
||||
if (first == null || first.compareTo(sstable.first) > 0)
|
||||
first = sstable.first;
|
||||
if (last == null || last.compareTo(sstable.last) < 0)
|
||||
last = sstable.last;
|
||||
if (first == null || first.compareTo(sstable.getFirst()) > 0)
|
||||
first = sstable.getFirst();
|
||||
if (last == null || last.compareTo(sstable.getLast()) < 0)
|
||||
last = sstable.getLast();
|
||||
}
|
||||
|
||||
DiskBoundaries diskBoundaries = getDiskBoundaries();
|
||||
|
|
@ -3476,4 +3466,16 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
|
|||
return null;
|
||||
return topPartitions.topTombstones().lastUpdate;
|
||||
}
|
||||
|
||||
@Override
|
||||
public OpOrder.Barrier newReadOrderingBarrier()
|
||||
{
|
||||
return readOrdering.newBarrier();
|
||||
}
|
||||
|
||||
@Override
|
||||
public TableMetrics getMetrics()
|
||||
{
|
||||
return metric;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -30,11 +30,12 @@ import org.apache.cassandra.dht.Token.KeyBound;
|
|||
import org.apache.cassandra.schema.ColumnMetadata;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.utils.ByteBufferUtil;
|
||||
import org.apache.cassandra.utils.IFilter.FilterKey;
|
||||
import org.apache.cassandra.utils.MurmurHash;
|
||||
import org.apache.cassandra.utils.bytecomparable.ByteComparable;
|
||||
import org.apache.cassandra.utils.bytecomparable.ByteSource;
|
||||
import org.apache.cassandra.utils.bytecomparable.ByteSourceInverse;
|
||||
import org.apache.cassandra.utils.IFilter.FilterKey;
|
||||
import org.apache.cassandra.utils.MurmurHash;
|
||||
import org.apache.cassandra.utils.memory.HeapCloner;
|
||||
|
||||
/**
|
||||
* Represents a decorated key, handy for certain operations
|
||||
|
|
@ -47,13 +48,7 @@ import org.apache.cassandra.utils.MurmurHash;
|
|||
*/
|
||||
public abstract class DecoratedKey implements PartitionPosition, FilterKey
|
||||
{
|
||||
public static final Comparator<DecoratedKey> comparator = new Comparator<DecoratedKey>()
|
||||
{
|
||||
public int compare(DecoratedKey o1, DecoratedKey o2)
|
||||
{
|
||||
return o1.compareTo(o2);
|
||||
}
|
||||
};
|
||||
public static final Comparator<DecoratedKey> comparator = DecoratedKey::compareTo;
|
||||
|
||||
private final Token token;
|
||||
|
||||
|
|
@ -200,6 +195,17 @@ public abstract class DecoratedKey implements PartitionPosition, FilterKey
|
|||
public abstract ByteBuffer getKey();
|
||||
public abstract int getKeyLength();
|
||||
|
||||
/**
|
||||
* If this key occupies only part of a larger buffer, allocate a new buffer that is only as large as necessary.
|
||||
* Otherwise, it returns this key.
|
||||
*/
|
||||
public DecoratedKey retainable()
|
||||
{
|
||||
return ByteBufferUtil.canMinimize(getKey())
|
||||
? new BufferDecoratedKey(getToken(), HeapCloner.instance.clone(getKey()))
|
||||
: this;
|
||||
}
|
||||
|
||||
public void filterHash(long[] dest)
|
||||
{
|
||||
ByteBuffer key = getKey();
|
||||
|
|
@ -240,4 +246,4 @@ public abstract class DecoratedKey implements PartitionPosition, FilterKey
|
|||
assert terminator == ByteSource.TERMINATOR : "Decorated key encoding must end in terminator.";
|
||||
return keyBytes;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -17,46 +17,59 @@
|
|||
*/
|
||||
package org.apache.cassandra.db;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.*;
|
||||
import java.util.concurrent.ThreadLocalRandom;
|
||||
import java.util.function.BiPredicate;
|
||||
import java.util.function.Function;
|
||||
import java.util.function.Supplier;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
|
||||
import java.io.IOError;
|
||||
import java.io.IOException;
|
||||
import java.nio.file.FileStore;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.Collection;
|
||||
import java.util.Collections;
|
||||
import java.util.Comparator;
|
||||
import java.util.HashMap;
|
||||
import java.util.HashSet;
|
||||
import java.util.Iterator;
|
||||
import java.util.LinkedHashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.Set;
|
||||
import java.util.Spliterator;
|
||||
import java.util.concurrent.ThreadLocalRandom;
|
||||
import java.util.function.BiPredicate;
|
||||
import java.util.function.Function;
|
||||
import java.util.function.Supplier;
|
||||
import java.util.stream.Collectors;
|
||||
import java.util.stream.Stream;
|
||||
import java.util.stream.StreamSupport;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import com.google.common.collect.ImmutableMap;
|
||||
import com.google.common.collect.Iterables;
|
||||
import com.google.common.collect.Maps;
|
||||
import com.google.common.util.concurrent.RateLimiter;
|
||||
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.apache.cassandra.config.*;
|
||||
import org.apache.cassandra.config.DatabaseDescriptor;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
|
||||
import org.apache.cassandra.io.FSDiskFullWriteError;
|
||||
import org.apache.cassandra.io.FSError;
|
||||
import org.apache.cassandra.io.FSNoDiskAvailableForWriteError;
|
||||
import org.apache.cassandra.io.FSReadError;
|
||||
import org.apache.cassandra.io.FSWriteError;
|
||||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.SSTable;
|
||||
import org.apache.cassandra.io.sstable.SSTableId;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.io.util.FileStoreUtils;
|
||||
import org.apache.cassandra.io.util.FileUtils;
|
||||
import org.apache.cassandra.io.sstable.*;
|
||||
import org.apache.cassandra.schema.SchemaConstants;
|
||||
import org.apache.cassandra.io.util.PathUtils;
|
||||
import org.apache.cassandra.schema.SchemaConstants;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.service.snapshot.SnapshotManifest;
|
||||
import org.apache.cassandra.service.snapshot.TableSnapshot;
|
||||
|
|
@ -275,7 +288,7 @@ public class Directories
|
|||
if (file.isDirectory())
|
||||
return false;
|
||||
|
||||
Descriptor desc = SSTable.tryDescriptorFromFilename(file);
|
||||
Descriptor desc = SSTable.tryDescriptorFromFile(file);
|
||||
return desc != null && desc.ksname.equals(metadata.keyspace) && desc.cfname.equals(metadata.name);
|
||||
});
|
||||
for (File indexFile : indexFiles)
|
||||
|
|
@ -322,7 +335,7 @@ public class Directories
|
|||
{
|
||||
File file = new File(dir, filename);
|
||||
if (file.exists())
|
||||
return Descriptor.fromFilename(file);
|
||||
return Descriptor.fromFileWithComponent(file, false).left;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
|
@ -969,7 +982,7 @@ public class Directories
|
|||
{
|
||||
for (Component c : entry.getValue())
|
||||
{
|
||||
l.add(new File(entry.getKey().filenameFor(c)));
|
||||
l.add(entry.getKey().fileFor(c));
|
||||
}
|
||||
}
|
||||
return l;
|
||||
|
|
@ -1310,7 +1323,7 @@ public class Directories
|
|||
public boolean isAcceptable(Path path)
|
||||
{
|
||||
File file = new File(path);
|
||||
Descriptor desc = SSTable.tryDescriptorFromFilename(file);
|
||||
Descriptor desc = SSTable.tryDescriptorFromFile(file);
|
||||
return desc != null
|
||||
&& desc.ksname.equals(metadata.keyspace)
|
||||
&& desc.cfname.equals(metadata.name)
|
||||
|
|
|
|||
|
|
@ -23,9 +23,6 @@ import java.util.concurrent.TimeUnit;
|
|||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.db.virtual.VirtualKeyspaceRegistry;
|
||||
import org.apache.cassandra.db.virtual.VirtualTable;
|
||||
import org.apache.cassandra.config.DatabaseDescriptor;
|
||||
import org.apache.cassandra.db.filter.ClusteringIndexFilter;
|
||||
import org.apache.cassandra.db.filter.ColumnFilter;
|
||||
|
|
@ -38,14 +35,17 @@ import org.apache.cassandra.db.partitions.PartitionIterator;
|
|||
import org.apache.cassandra.db.partitions.UnfilteredPartitionIterator;
|
||||
import org.apache.cassandra.db.partitions.UnfilteredPartitionIterators;
|
||||
import org.apache.cassandra.db.rows.BaseRowIterator;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.db.transform.RTBoundValidator;
|
||||
import org.apache.cassandra.db.transform.Transformation;
|
||||
import org.apache.cassandra.db.virtual.VirtualKeyspaceRegistry;
|
||||
import org.apache.cassandra.db.virtual.VirtualTable;
|
||||
import org.apache.cassandra.dht.AbstractBounds;
|
||||
import org.apache.cassandra.dht.Bounds;
|
||||
import org.apache.cassandra.exceptions.RequestExecutionException;
|
||||
import org.apache.cassandra.index.Index;
|
||||
import org.apache.cassandra.io.sstable.SSTableReadsListener;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReadsListener;
|
||||
import org.apache.cassandra.io.util.DataInputPlus;
|
||||
import org.apache.cassandra.io.util.DataOutputPlus;
|
||||
import org.apache.cassandra.metrics.TableMetrics;
|
||||
|
|
|
|||
|
|
@ -28,19 +28,20 @@ import java.util.Set;
|
|||
import java.util.UUID;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.apache.cassandra.db.compaction.Verifier;
|
||||
import org.apache.cassandra.db.lifecycle.SSTableSet;
|
||||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.IVerifier;
|
||||
import org.apache.cassandra.io.sstable.KeyIterator;
|
||||
import org.apache.cassandra.io.sstable.SSTable;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat.Components;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableWriter;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.service.ActiveRepairService;
|
||||
import org.apache.cassandra.utils.OutputHandler;
|
||||
import org.apache.cassandra.utils.Pair;
|
||||
import org.apache.cassandra.utils.concurrent.Refs;
|
||||
|
||||
|
|
@ -185,7 +186,7 @@ public class SSTableImporter
|
|||
for (SSTableReader reader : newSSTables)
|
||||
{
|
||||
if (options.invalidateCaches && cfs.isRowCacheEnabled())
|
||||
invalidateCachesForSSTable(reader.descriptor);
|
||||
invalidateCachesForSSTable(reader);
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -209,8 +210,8 @@ public class SSTableImporter
|
|||
private static String formatMetadata(SSTableReader sstable)
|
||||
{
|
||||
return String.format("{[%s, %s], %d, %s, %d}",
|
||||
sstable.first.getToken(),
|
||||
sstable.last.getToken(),
|
||||
sstable.getFirst().getToken(),
|
||||
sstable.getLast().getToken(),
|
||||
sstable.getSSTableLevel(),
|
||||
sstable.isRepaired(),
|
||||
sstable.onDiskLength());
|
||||
|
|
@ -233,7 +234,7 @@ public class SSTableImporter
|
|||
SSTableReader sstable = null;
|
||||
try
|
||||
{
|
||||
sstable = SSTableReader.open(descriptor, components, cfs.metadata);
|
||||
sstable = SSTableReader.open(cfs, descriptor, components, cfs.metadata);
|
||||
targetDirectory = cfs.getDirectories().getLocationForDisk(cfs.diskBoundaryManager.getDiskBoundaries(cfs).getCorrectDiskForSSTable(sstable));
|
||||
}
|
||||
finally
|
||||
|
|
@ -241,7 +242,7 @@ public class SSTableImporter
|
|||
if (sstable != null)
|
||||
sstable.selfRef().release();
|
||||
}
|
||||
return targetDirectory == null ? cfs.getDirectories().getWriteableLocationToLoadFile(new File(descriptor.baseFilename())) : targetDirectory;
|
||||
return targetDirectory == null ? cfs.getDirectories().getWriteableLocationToLoadFile(descriptor.baseFile()) : targetDirectory;
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -304,11 +305,11 @@ public class SSTableImporter
|
|||
{
|
||||
for (MovedSSTable movedSSTable : movedSSTables)
|
||||
{
|
||||
if (new File(movedSSTable.newDescriptor.filenameFor(Component.DATA)).exists())
|
||||
if (movedSSTable.newDescriptor.fileFor(Components.DATA).exists())
|
||||
{
|
||||
logger.debug("Moving sstable {} back to {}", movedSSTable.newDescriptor.filenameFor(Component.DATA)
|
||||
, movedSSTable.oldDescriptor.filenameFor(Component.DATA));
|
||||
SSTableWriter.rename(movedSSTable.newDescriptor, movedSSTable.oldDescriptor, movedSSTable.components);
|
||||
logger.debug("Moving sstable {} back to {}", movedSSTable.newDescriptor.fileFor(Components.DATA)
|
||||
, movedSSTable.oldDescriptor.fileFor(Components.DATA));
|
||||
SSTable.rename(movedSSTable.newDescriptor, movedSSTable.oldDescriptor, movedSSTable.components);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -324,11 +325,8 @@ public class SSTableImporter
|
|||
logger.debug("Removing copied SSTables which were left in data directories after failed SSTable import.");
|
||||
for (MovedSSTable movedSSTable : movedSSTables)
|
||||
{
|
||||
if (new File(movedSSTable.newDescriptor.filenameFor(Component.DATA)).exists())
|
||||
{
|
||||
// no logging here as for moveSSTablesBack case above as logging is done in delete method
|
||||
SSTableWriter.delete(movedSSTable.newDescriptor, movedSSTable.components);
|
||||
}
|
||||
// no logging here as for moveSSTablesBack case above as logging is done in delete method
|
||||
movedSSTable.newDescriptor.getFormat().delete(movedSSTable.newDescriptor);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -336,9 +334,9 @@ public class SSTableImporter
|
|||
* Iterates over all keys in the sstable index and invalidates the row cache
|
||||
*/
|
||||
@VisibleForTesting
|
||||
void invalidateCachesForSSTable(Descriptor desc)
|
||||
void invalidateCachesForSSTable(SSTableReader reader)
|
||||
{
|
||||
try (KeyIterator iter = new KeyIterator(desc, cfs.metadata()))
|
||||
try (KeyIterator iter = reader.keyIterator())
|
||||
{
|
||||
while (iter.hasNext())
|
||||
{
|
||||
|
|
@ -346,6 +344,10 @@ public class SSTableImporter
|
|||
cfs.invalidateCachedPartition(decoratedKey);
|
||||
}
|
||||
}
|
||||
catch (IOException ex)
|
||||
{
|
||||
throw new RuntimeException("Failed to import sstable " + reader.getFilename(), ex);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -360,14 +362,15 @@ public class SSTableImporter
|
|||
SSTableReader reader = null;
|
||||
try
|
||||
{
|
||||
reader = SSTableReader.open(descriptor, components, cfs.metadata);
|
||||
Verifier.Options verifierOptions = Verifier.options()
|
||||
.extendedVerification(extendedVerify)
|
||||
.checkOwnsTokens(verifyTokens)
|
||||
.quick(!verifySSTables)
|
||||
.invokeDiskFailurePolicy(false)
|
||||
.mutateRepairStatus(false).build();
|
||||
try (Verifier verifier = new Verifier(cfs, reader, false, verifierOptions))
|
||||
reader = SSTableReader.open(cfs, descriptor, components, cfs.metadata);
|
||||
IVerifier.Options verifierOptions = IVerifier.options()
|
||||
.extendedVerification(extendedVerify)
|
||||
.checkOwnsTokens(verifyTokens)
|
||||
.quick(!verifySSTables)
|
||||
.invokeDiskFailurePolicy(false)
|
||||
.mutateRepairStatus(false).build();
|
||||
|
||||
try (IVerifier verifier = reader.getVerifier(cfs, new OutputHandler.LogOutput(), false, verifierOptions))
|
||||
{
|
||||
verifier.verify();
|
||||
}
|
||||
|
|
@ -389,7 +392,7 @@ public class SSTableImporter
|
|||
*/
|
||||
private void maybeMutateMetadata(Descriptor descriptor, Options options) throws IOException
|
||||
{
|
||||
if (new File(descriptor.filenameFor(Component.STATS)).exists())
|
||||
if (descriptor.fileFor(Components.STATS).exists())
|
||||
{
|
||||
if (options.resetLevel)
|
||||
{
|
||||
|
|
|
|||
|
|
@ -19,7 +19,12 @@ package org.apache.cassandra.db;
|
|||
|
||||
import java.io.IOException;
|
||||
import java.nio.ByteBuffer;
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
import java.util.NavigableSet;
|
||||
import java.util.TreeSet;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
|
|
@ -30,18 +35,37 @@ import org.apache.cassandra.cache.IRowCacheEntry;
|
|||
import org.apache.cassandra.cache.RowCacheKey;
|
||||
import org.apache.cassandra.cache.RowCacheSentinel;
|
||||
import org.apache.cassandra.config.DatabaseDescriptor;
|
||||
import org.apache.cassandra.db.filter.*;
|
||||
import org.apache.cassandra.db.lifecycle.*;
|
||||
import org.apache.cassandra.db.filter.ClusteringIndexFilter;
|
||||
import org.apache.cassandra.db.filter.ClusteringIndexNamesFilter;
|
||||
import org.apache.cassandra.db.filter.ClusteringIndexSliceFilter;
|
||||
import org.apache.cassandra.db.filter.ColumnFilter;
|
||||
import org.apache.cassandra.db.filter.DataLimits;
|
||||
import org.apache.cassandra.db.filter.RowFilter;
|
||||
import org.apache.cassandra.db.lifecycle.SSTableSet;
|
||||
import org.apache.cassandra.db.lifecycle.View;
|
||||
import org.apache.cassandra.db.memtable.Memtable;
|
||||
import org.apache.cassandra.db.partitions.*;
|
||||
import org.apache.cassandra.db.rows.*;
|
||||
import org.apache.cassandra.db.partitions.CachedBTreePartition;
|
||||
import org.apache.cassandra.db.partitions.CachedPartition;
|
||||
import org.apache.cassandra.db.partitions.ImmutableBTreePartition;
|
||||
import org.apache.cassandra.db.partitions.PartitionIterator;
|
||||
import org.apache.cassandra.db.partitions.PartitionIterators;
|
||||
import org.apache.cassandra.db.partitions.SingletonUnfilteredPartitionIterator;
|
||||
import org.apache.cassandra.db.partitions.UnfilteredPartitionIterator;
|
||||
import org.apache.cassandra.db.rows.Cell;
|
||||
import org.apache.cassandra.db.rows.Row;
|
||||
import org.apache.cassandra.db.rows.Rows;
|
||||
import org.apache.cassandra.db.rows.Unfiltered;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterators;
|
||||
import org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator;
|
||||
import org.apache.cassandra.db.transform.RTBoundValidator;
|
||||
import org.apache.cassandra.db.transform.Transformation;
|
||||
import org.apache.cassandra.db.virtual.VirtualKeyspaceRegistry;
|
||||
import org.apache.cassandra.db.virtual.VirtualTable;
|
||||
import org.apache.cassandra.exceptions.RequestExecutionException;
|
||||
import org.apache.cassandra.io.sstable.SSTableReadsListener;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReadsListener;
|
||||
import org.apache.cassandra.io.util.DataInputPlus;
|
||||
import org.apache.cassandra.io.util.DataOutputPlus;
|
||||
import org.apache.cassandra.metrics.TableMetrics;
|
||||
|
|
@ -49,7 +73,9 @@ import org.apache.cassandra.net.Verb;
|
|||
import org.apache.cassandra.schema.ColumnMetadata;
|
||||
import org.apache.cassandra.schema.IndexMetadata;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.service.*;
|
||||
import org.apache.cassandra.service.CacheService;
|
||||
import org.apache.cassandra.service.ClientState;
|
||||
import org.apache.cassandra.service.StorageProxy;
|
||||
import org.apache.cassandra.tracing.Tracing;
|
||||
import org.apache.cassandra.utils.FBUtilities;
|
||||
import org.apache.cassandra.utils.btree.BTreeSet;
|
||||
|
|
@ -538,20 +564,26 @@ public class SinglePartitionReadCommand extends ReadCommand implements SinglePar
|
|||
try
|
||||
{
|
||||
// Use a custom iterator instead of DataLimits to avoid stopping the original iterator
|
||||
UnfilteredRowIterator toCacheIterator = new WrappingUnfilteredRowIterator(iter)
|
||||
UnfilteredRowIterator toCacheIterator = new WrappingUnfilteredRowIterator()
|
||||
{
|
||||
private int rowsCounted = 0;
|
||||
|
||||
@Override
|
||||
public UnfilteredRowIterator wrapped()
|
||||
{
|
||||
return iter;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean hasNext()
|
||||
{
|
||||
return rowsCounted < rowsToCache && super.hasNext();
|
||||
return rowsCounted < rowsToCache && iter.hasNext();
|
||||
}
|
||||
|
||||
@Override
|
||||
public Unfiltered next()
|
||||
{
|
||||
Unfiltered unfiltered = super.next();
|
||||
Unfiltered unfiltered = iter.next();
|
||||
if (unfiltered.isRow())
|
||||
{
|
||||
Row row = (Row) unfiltered;
|
||||
|
|
@ -1307,7 +1339,7 @@ public class SinglePartitionReadCommand extends ReadCommand implements SinglePar
|
|||
private int mergedSSTables;
|
||||
|
||||
@Override
|
||||
public void onSSTableSelected(SSTableReader sstable, RowIndexEntry<?> indexEntry, SelectionReason reason)
|
||||
public void onSSTableSelected(SSTableReader sstable, SelectionReason reason)
|
||||
{
|
||||
sstable.incrementReadCount();
|
||||
mergedSSTables++;
|
||||
|
|
|
|||
|
|
@ -23,8 +23,8 @@ import org.apache.cassandra.db.filter.ColumnFilter;
|
|||
import org.apache.cassandra.db.partitions.PartitionUpdate;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound;
|
||||
import org.apache.cassandra.io.sstable.SSTableReadsListener;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReadsListener;
|
||||
import org.apache.cassandra.schema.TableId;
|
||||
import org.apache.cassandra.utils.FBUtilities;
|
||||
|
||||
|
|
|
|||
|
|
@ -17,34 +17,39 @@
|
|||
*/
|
||||
package org.apache.cassandra.db.compaction;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.Collections;
|
||||
import java.util.HashMap;
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import com.google.common.collect.ImmutableMap;
|
||||
|
||||
import org.apache.cassandra.db.Directories;
|
||||
import org.apache.cassandra.db.SerializationHeader;
|
||||
import org.apache.cassandra.index.Index;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleNewTracker;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.SSTableMultiWriter;
|
||||
import org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.Directories;
|
||||
import org.apache.cassandra.db.SerializationHeader;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleNewTracker;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
|
||||
import org.apache.cassandra.dht.Range;
|
||||
import org.apache.cassandra.dht.Token;
|
||||
import org.apache.cassandra.exceptions.ConfigurationException;
|
||||
import org.apache.cassandra.index.Index;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.ISSTableScanner;
|
||||
import org.apache.cassandra.io.sstable.SSTableMultiWriter;
|
||||
import org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
|
||||
import org.apache.cassandra.io.sstable.metadata.StatsMetadata;
|
||||
import org.apache.cassandra.schema.CompactionParams;
|
||||
import org.apache.cassandra.utils.TimeUUID;
|
||||
|
||||
import static org.apache.cassandra.io.sstable.Component.DATA;
|
||||
import static org.apache.cassandra.utils.Clock.Global.currentTimeMillis;
|
||||
|
||||
/**
|
||||
|
|
@ -399,7 +404,7 @@ public abstract class AbstractCompactionStrategy
|
|||
// since we use estimations to calculate, there is a chance that compaction will not drop tombstones actually.
|
||||
// if that happens we will end up in infinite compaction loop, so first we check enough if enough time has
|
||||
// elapsed since SSTable created.
|
||||
if (currentTimeMillis() < sstable.getCreationTimeFor(DATA) + tombstoneCompactionInterval * 1000)
|
||||
if (currentTimeMillis() < sstable.getDataCreationTime() + tombstoneCompactionInterval * 1000)
|
||||
return false;
|
||||
|
||||
double droppableRatio = sstable.getEstimatedDroppableTombstoneRatio(gcBefore);
|
||||
|
|
@ -423,16 +428,16 @@ public abstract class AbstractCompactionStrategy
|
|||
else
|
||||
{
|
||||
// what percentage of columns do we expect to compact outside of overlap?
|
||||
if (sstable.getIndexSummarySize() < 2)
|
||||
if (!sstable.isEstimationInformative())
|
||||
{
|
||||
// we have too few samples to estimate correct percentage
|
||||
return false;
|
||||
}
|
||||
// first, calculate estimated keys that do not overlap
|
||||
long keys = sstable.estimatedKeys();
|
||||
Set<Range<Token>> ranges = new HashSet<Range<Token>>(overlaps.size());
|
||||
Set<Range<Token>> ranges = new HashSet<>(overlaps.size());
|
||||
for (SSTableReader overlap : overlaps)
|
||||
ranges.add(new Range<>(overlap.first.getToken(), overlap.last.getToken()));
|
||||
ranges.add(new Range<>(overlap.getFirst().getToken(), overlap.getLast().getToken()));
|
||||
long remainingKeys = keys - sstable.estimatedKeysForRanges(ranges);
|
||||
// next, calculate what percentage of columns we have within those keys
|
||||
long columns = sstable.getEstimatedCellPerPartitionCount().mean() * remainingKeys;
|
||||
|
|
@ -560,7 +565,7 @@ public abstract class AbstractCompactionStrategy
|
|||
Collection<Index> indexes,
|
||||
LifecycleNewTracker lifecycleNewTracker)
|
||||
{
|
||||
return SimpleSSTableMultiWriter.create(descriptor, keyCount, repairedAt, pendingRepair, isTransient, cfs.metadata, meta, header, indexes, lifecycleNewTracker);
|
||||
return SimpleSSTableMultiWriter.create(descriptor, keyCount, repairedAt, pendingRepair, isTransient, cfs.metadata, meta, header, indexes, lifecycleNewTracker, cfs);
|
||||
}
|
||||
|
||||
public boolean supportsEarlyOpen()
|
||||
|
|
|
|||
|
|
@ -17,7 +17,14 @@
|
|||
*/
|
||||
package org.apache.cassandra.db.compaction;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.HashMap;
|
||||
import java.util.HashSet;
|
||||
import java.util.Iterator;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.function.LongPredicate;
|
||||
|
||||
import com.google.common.base.Predicates;
|
||||
|
|
@ -27,7 +34,10 @@ import org.slf4j.Logger;
|
|||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.apache.cassandra.config.Config;
|
||||
import org.apache.cassandra.db.*;
|
||||
import org.apache.cassandra.db.AbstractCompactionController;
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.PartitionPosition;
|
||||
import org.apache.cassandra.db.memtable.Memtable;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
|
|
@ -251,7 +261,7 @@ public class CompactionController extends AbstractCompactionController
|
|||
|
||||
for (SSTableReader sstable: filteredSSTables)
|
||||
{
|
||||
if (sstable.maybePresent(key))
|
||||
if (sstable.mayContainAssumingKeyIsInRange(key))
|
||||
{
|
||||
minTimestampSeen = Math.min(minTimestampSeen, sstable.getMinTimestamp());
|
||||
hasTimestamp = true;
|
||||
|
|
@ -316,8 +326,8 @@ public class CompactionController extends AbstractCompactionController
|
|||
reader.getMaxTimestamp() <= minTimestamp ||
|
||||
tombstoneOnly && !reader.mayHaveTombstones())
|
||||
return null;
|
||||
RowIndexEntry<?> position = reader.getPosition(key, SSTableReader.Operator.EQ);
|
||||
if (position == null)
|
||||
long position = reader.getPosition(key, SSTableReader.Operator.EQ);
|
||||
if (position < 0)
|
||||
return null;
|
||||
FileDataInput dfile = openDataFiles.computeIfAbsent(reader, this::openDataFile);
|
||||
return reader.simpleIterator(dfile, key, position, tombstoneOnly);
|
||||
|
|
|
|||
|
|
@ -57,7 +57,7 @@ public final class CompactionInfo
|
|||
private final ImmutableSet<SSTableReader> sstables;
|
||||
private final String targetDirectory;
|
||||
|
||||
public CompactionInfo(TableMetadata metadata, OperationType tasktype, long completed, long total, Unit unit, TimeUUID compactionId, Collection<SSTableReader> sstables, String targetDirectory)
|
||||
public CompactionInfo(TableMetadata metadata, OperationType tasktype, long completed, long total, Unit unit, TimeUUID compactionId, Collection<? extends SSTableReader> sstables, String targetDirectory)
|
||||
{
|
||||
this.tasktype = tasktype;
|
||||
this.completed = completed;
|
||||
|
|
@ -74,7 +74,7 @@ public final class CompactionInfo
|
|||
this(metadata, tasktype, completed, total, Unit.BYTES, compactionId, sstables, targetDirectory);
|
||||
}
|
||||
|
||||
public CompactionInfo(TableMetadata metadata, OperationType tasktype, long completed, long total, TimeUUID compactionId, Collection<SSTableReader> sstables)
|
||||
public CompactionInfo(TableMetadata metadata, OperationType tasktype, long completed, long total, TimeUUID compactionId, Collection<? extends SSTableReader> sstables)
|
||||
{
|
||||
this(metadata, tasktype, completed, total, Unit.BYTES, compactionId, sstables, null);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,33 +17,51 @@
|
|||
*/
|
||||
package org.apache.cassandra.db.compaction;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.function.LongPredicate;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.function.LongPredicate;
|
||||
|
||||
import com.google.common.collect.ImmutableSet;
|
||||
import com.google.common.collect.Ordering;
|
||||
|
||||
import org.apache.cassandra.config.DatabaseDescriptor;
|
||||
import org.apache.cassandra.dht.Token;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.schema.Schema;
|
||||
import org.apache.cassandra.schema.SchemaConstants;
|
||||
import org.apache.cassandra.schema.TableId;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
|
||||
import org.apache.cassandra.db.transform.DuplicateRowChecker;
|
||||
import org.apache.cassandra.db.*;
|
||||
import org.apache.cassandra.db.AbstractCompactionController;
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.Columns;
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.DeletionTime;
|
||||
import org.apache.cassandra.db.EmptyIterators;
|
||||
import org.apache.cassandra.db.Keyspace;
|
||||
import org.apache.cassandra.db.RegularAndStaticColumns;
|
||||
import org.apache.cassandra.db.SystemKeyspace;
|
||||
import org.apache.cassandra.db.filter.ColumnFilter;
|
||||
import org.apache.cassandra.db.partitions.PurgeFunction;
|
||||
import org.apache.cassandra.db.partitions.UnfilteredPartitionIterator;
|
||||
import org.apache.cassandra.db.partitions.UnfilteredPartitionIterators;
|
||||
import org.apache.cassandra.db.rows.*;
|
||||
import org.apache.cassandra.db.rows.RangeTombstoneBoundMarker;
|
||||
import org.apache.cassandra.db.rows.RangeTombstoneMarker;
|
||||
import org.apache.cassandra.db.rows.Row;
|
||||
import org.apache.cassandra.db.rows.Rows;
|
||||
import org.apache.cassandra.db.rows.Unfiltered;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterators;
|
||||
import org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator;
|
||||
import org.apache.cassandra.db.transform.DuplicateRowChecker;
|
||||
import org.apache.cassandra.db.transform.Transformation;
|
||||
import org.apache.cassandra.dht.Token;
|
||||
import org.apache.cassandra.index.transactions.CompactionTransaction;
|
||||
import org.apache.cassandra.io.sstable.ISSTableScanner;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.metrics.TopPartitionTracker;
|
||||
import org.apache.cassandra.schema.CompactionParams.TombstoneOption;
|
||||
import org.apache.cassandra.schema.Schema;
|
||||
import org.apache.cassandra.schema.SchemaConstants;
|
||||
import org.apache.cassandra.schema.TableId;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.service.paxos.PaxosRepairHistory;
|
||||
import org.apache.cassandra.service.paxos.uncommitted.PaxosRows;
|
||||
import org.apache.cassandra.utils.TimeUUID;
|
||||
|
|
@ -385,8 +403,10 @@ public class CompactionIterator extends CompactionInfo.Holder implements Unfilte
|
|||
* The result produced by this iterator is such that when merged with tombSource it produces the same output
|
||||
* as the merge of dataSource and tombSource.
|
||||
*/
|
||||
private static class GarbageSkippingUnfilteredRowIterator extends WrappingUnfilteredRowIterator
|
||||
private static class GarbageSkippingUnfilteredRowIterator implements WrappingUnfilteredRowIterator
|
||||
{
|
||||
private final UnfilteredRowIterator wrapped;
|
||||
|
||||
final UnfilteredRowIterator tombSource;
|
||||
final DeletionTime partitionLevelDeletion;
|
||||
final Row staticRow;
|
||||
|
|
@ -413,7 +433,7 @@ public class CompactionIterator extends CompactionInfo.Holder implements Unfilte
|
|||
*/
|
||||
protected GarbageSkippingUnfilteredRowIterator(UnfilteredRowIterator dataSource, UnfilteredRowIterator tombSource, boolean cellLevelGC)
|
||||
{
|
||||
super(dataSource);
|
||||
this.wrapped = dataSource;
|
||||
this.tombSource = tombSource;
|
||||
this.cellLevelGC = cellLevelGC;
|
||||
metadata = dataSource.metadata();
|
||||
|
|
@ -433,6 +453,12 @@ public class CompactionIterator extends CompactionInfo.Holder implements Unfilte
|
|||
dataNext = advance(dataSource);
|
||||
}
|
||||
|
||||
@Override
|
||||
public UnfilteredRowIterator wrapped()
|
||||
{
|
||||
return wrapped;
|
||||
}
|
||||
|
||||
private static Unfiltered advance(UnfilteredRowIterator source)
|
||||
{
|
||||
return source.hasNext() ? source.next() : null;
|
||||
|
|
@ -446,7 +472,7 @@ public class CompactionIterator extends CompactionInfo.Holder implements Unfilte
|
|||
|
||||
public void close()
|
||||
{
|
||||
super.close();
|
||||
wrapped.close();
|
||||
tombSource.close();
|
||||
}
|
||||
|
||||
|
|
@ -719,4 +745,4 @@ public class CompactionIterator extends CompactionInfo.Holder implements Unfilte
|
|||
{
|
||||
return cfs.name.equals(SystemKeyspace.PAXOS) && cfs.keyspace.getName().equals(SchemaConstants.SYSTEM_KEYSPACE_NAME);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -18,7 +18,16 @@
|
|||
package org.apache.cassandra.db.compaction;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.Collection;
|
||||
import java.util.Collections;
|
||||
import java.util.HashMap;
|
||||
import java.util.HashSet;
|
||||
import java.util.Iterator;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.Callable;
|
||||
import java.util.concurrent.ExecutionException;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
|
|
@ -34,25 +43,33 @@ import javax.management.openmbean.TabularData;
|
|||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import com.google.common.base.Preconditions;
|
||||
import com.google.common.collect.*;
|
||||
import com.google.common.base.Predicates;
|
||||
import com.google.common.collect.ArrayListMultimap;
|
||||
import com.google.common.collect.Collections2;
|
||||
import com.google.common.collect.ConcurrentHashMultiset;
|
||||
import com.google.common.collect.Iterables;
|
||||
import com.google.common.collect.Lists;
|
||||
import com.google.common.collect.Maps;
|
||||
import com.google.common.collect.Multimap;
|
||||
import com.google.common.collect.Multiset;
|
||||
import com.google.common.collect.Sets;
|
||||
import com.google.common.util.concurrent.RateLimiter;
|
||||
import com.google.common.util.concurrent.Uninterruptibles;
|
||||
|
||||
import org.apache.cassandra.concurrent.ExecutorFactory;
|
||||
import org.apache.cassandra.concurrent.WrappedExecutorPlus;
|
||||
import org.apache.cassandra.dht.AbstractBounds;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.locator.RangesAtEndpoint;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import net.openhft.chronicle.core.util.ThrowingSupplier;
|
||||
import org.apache.cassandra.cache.AutoSavingCache;
|
||||
import org.apache.cassandra.exceptions.ConfigurationException;
|
||||
import org.apache.cassandra.repair.NoSuchRepairSessionException;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.concurrent.ExecutorFactory;
|
||||
import org.apache.cassandra.concurrent.WrappedExecutorPlus;
|
||||
import org.apache.cassandra.config.DatabaseDescriptor;
|
||||
import org.apache.cassandra.schema.Schema;
|
||||
import org.apache.cassandra.db.*;
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.Directories;
|
||||
import org.apache.cassandra.db.DiskBoundaries;
|
||||
import org.apache.cassandra.db.Keyspace;
|
||||
import org.apache.cassandra.db.SerializationHeader;
|
||||
import org.apache.cassandra.db.SystemKeyspace;
|
||||
import org.apache.cassandra.db.compaction.CompactionInfo.Holder;
|
||||
import org.apache.cassandra.db.lifecycle.ILifecycleTransaction;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
|
||||
|
|
@ -62,27 +79,40 @@ import org.apache.cassandra.db.lifecycle.View;
|
|||
import org.apache.cassandra.db.lifecycle.WrappedLifecycleTransaction;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.db.view.ViewBuilderTask;
|
||||
import org.apache.cassandra.dht.Bounds;
|
||||
import org.apache.cassandra.dht.AbstractBounds;
|
||||
import org.apache.cassandra.dht.Range;
|
||||
import org.apache.cassandra.dht.Token;
|
||||
import org.apache.cassandra.exceptions.ConfigurationException;
|
||||
import org.apache.cassandra.index.SecondaryIndexBuilder;
|
||||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.ISSTableScanner;
|
||||
import org.apache.cassandra.io.sstable.IndexSummaryRedistribution;
|
||||
import org.apache.cassandra.io.sstable.IScrubber;
|
||||
import org.apache.cassandra.io.sstable.IVerifier;
|
||||
import org.apache.cassandra.io.sstable.SSTableRewriter;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableWriter;
|
||||
import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
|
||||
import org.apache.cassandra.io.sstable.metadata.StatsMetadata;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.io.util.FileUtils;
|
||||
import org.apache.cassandra.locator.RangesAtEndpoint;
|
||||
import org.apache.cassandra.metrics.CompactionMetrics;
|
||||
import org.apache.cassandra.metrics.TableMetrics;
|
||||
import org.apache.cassandra.repair.NoSuchRepairSessionException;
|
||||
import org.apache.cassandra.schema.CompactionParams.TombstoneOption;
|
||||
import org.apache.cassandra.schema.Schema;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.service.ActiveRepairService;
|
||||
import org.apache.cassandra.service.StorageService;
|
||||
import org.apache.cassandra.streaming.PreviewKind;
|
||||
import org.apache.cassandra.utils.*;
|
||||
import org.apache.cassandra.utils.FBUtilities;
|
||||
import org.apache.cassandra.utils.JVMStabilityInspector;
|
||||
import org.apache.cassandra.utils.MBeanWrapper;
|
||||
import org.apache.cassandra.utils.OutputHandler;
|
||||
import org.apache.cassandra.utils.Throwables;
|
||||
import org.apache.cassandra.utils.TimeUUID;
|
||||
import org.apache.cassandra.utils.WrappedRunnable;
|
||||
import org.apache.cassandra.utils.concurrent.Future;
|
||||
import org.apache.cassandra.utils.concurrent.ImmediateFuture;
|
||||
import org.apache.cassandra.utils.concurrent.Refs;
|
||||
|
|
@ -455,16 +485,7 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
}
|
||||
}
|
||||
|
||||
public AllSSTableOpStatus performScrub(final ColumnFamilyStore cfs, final boolean skipCorrupted, final boolean checkData,
|
||||
int jobs)
|
||||
throws InterruptedException, ExecutionException
|
||||
{
|
||||
return performScrub(cfs, skipCorrupted, checkData, false, jobs);
|
||||
}
|
||||
|
||||
public AllSSTableOpStatus performScrub(final ColumnFamilyStore cfs, final boolean skipCorrupted, final boolean checkData,
|
||||
final boolean reinsertOverflowedTTL, int jobs)
|
||||
throws InterruptedException, ExecutionException
|
||||
public AllSSTableOpStatus performScrub(ColumnFamilyStore cfs, IScrubber.Options options, int jobs)
|
||||
{
|
||||
return parallelAllSSTableOperation(cfs, new OneSSTableOperation()
|
||||
{
|
||||
|
|
@ -477,12 +498,12 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
@Override
|
||||
public void execute(LifecycleTransaction input)
|
||||
{
|
||||
scrubOne(cfs, input, skipCorrupted, checkData, reinsertOverflowedTTL, active);
|
||||
scrubOne(cfs, input, options, active);
|
||||
}
|
||||
}, jobs, OperationType.SCRUB);
|
||||
}
|
||||
|
||||
public AllSSTableOpStatus performVerify(ColumnFamilyStore cfs, Verifier.Options options) throws InterruptedException, ExecutionException
|
||||
public AllSSTableOpStatus performVerify(ColumnFamilyStore cfs, IVerifier.Options options) throws InterruptedException, ExecutionException
|
||||
{
|
||||
assert !cfs.isIndex();
|
||||
return parallelAllSSTableOperation(cfs, new OneSSTableOperation()
|
||||
|
|
@ -513,7 +534,7 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
return false;
|
||||
|
||||
// Skip if SSTable creation time is past given timestamp
|
||||
if (sstable.getCreationTimeFor(Component.DATA) > skipIfOlderThanTimestamp)
|
||||
if (sstable.getDataCreationTime() > skipIfOlderThanTimestamp)
|
||||
return false;
|
||||
|
||||
TableMetadata metadata = cfs.metadata.get();
|
||||
|
|
@ -598,8 +619,8 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
{
|
||||
logger.debug("Skipping {} ([{}, {}]) for cleanup; all rows should be kept. Needs cleanup full ranges: {} Needs cleanup transient ranges: {} Repaired: {}",
|
||||
sstable,
|
||||
sstable.first.getToken(),
|
||||
sstable.last.getToken(),
|
||||
sstable.getFirst().getToken(),
|
||||
sstable.getLast().getToken(),
|
||||
needsCleanupFull,
|
||||
needsCleanupTransient,
|
||||
sstable.isRepaired());
|
||||
|
|
@ -862,7 +883,7 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
List<Range<Token>> normalizedRanges = Range.normalize(ranges.ranges());
|
||||
for (SSTableReader sstable : sstables)
|
||||
{
|
||||
Bounds<Token> bounds = new Bounds<>(sstable.first.getToken(), sstable.last.getToken());
|
||||
AbstractBounds<Token> bounds = sstable.getBounds();
|
||||
|
||||
if (!Iterables.any(normalizedRanges, r -> (r.contains(bounds.left) && r.contains(bounds.right)) || r.intersects(bounds)))
|
||||
{
|
||||
|
|
@ -884,12 +905,12 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
{
|
||||
SSTableReader sstable = sstableIterator.next();
|
||||
|
||||
Bounds<Token> sstableBounds = new Bounds<>(sstable.first.getToken(), sstable.last.getToken());
|
||||
AbstractBounds<Token> sstableBounds = sstable.getBounds();
|
||||
|
||||
for (Range<Token> r : normalizedRanges)
|
||||
{
|
||||
// ranges are normalized - no wrap around - if first and last are contained we know that all tokens are contained in the range
|
||||
if (r.contains(sstable.first.getToken()) && r.contains(sstable.last.getToken()))
|
||||
if (r.contains(sstable.getFirst().getToken()) && r.contains(sstable.getLast().getToken()))
|
||||
{
|
||||
logger.info("{} SSTable {} fully contained in range {}, mutating repairedAt instead of anticompacting", PreviewKind.NONE.logPrefix(parentRepairSession), sstable, r);
|
||||
fullyContainedSSTables.add(sstable);
|
||||
|
|
@ -1005,7 +1026,7 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
{
|
||||
forceCompaction(cfStore,
|
||||
() -> sstablesInBounds(cfStore, ranges),
|
||||
sstable -> new Bounds<>(sstable.first.getToken(), sstable.last.getToken()).intersects(ranges));
|
||||
sstable -> sstable.getBounds().intersects(ranges));
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -1043,24 +1064,12 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
|
||||
public void forceCompactionForKey(ColumnFamilyStore cfStore, DecoratedKey key)
|
||||
{
|
||||
forceCompaction(cfStore, () -> sstablesWithKey(cfStore, key), sstable -> sstable.maybePresent(key));
|
||||
forceCompaction(cfStore, () -> sstablesWithKey(cfStore, key), Predicates.alwaysTrue());
|
||||
}
|
||||
|
||||
public void forceCompactionForKeys(ColumnFamilyStore cfStore, Collection<DecoratedKey> keys)
|
||||
{
|
||||
com.google.common.base.Predicate<SSTableReader> predicate = sstable -> {
|
||||
for (DecoratedKey key : keys)
|
||||
{
|
||||
if(sstable.maybePresent(key))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
};
|
||||
|
||||
forceCompaction(cfStore, () -> sstablesWithKeys(cfStore, keys), predicate);
|
||||
forceCompaction(cfStore, () -> sstablesWithKeys(cfStore, keys), Predicates.alwaysTrue());
|
||||
}
|
||||
|
||||
private static Collection<SSTableReader> sstablesWithKey(ColumnFamilyStore cfs, DecoratedKey key)
|
||||
|
|
@ -1070,7 +1079,7 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
key.getToken().maxKeyBound());
|
||||
for (SSTableReader sstable : liveTables)
|
||||
{
|
||||
if (sstable.maybePresent(key))
|
||||
if (sstable.mayContainAssumingKeyIsInRange(key))
|
||||
sstables.add(sstable);
|
||||
}
|
||||
return sstables.isEmpty() ? Collections.emptyList() : sstables;
|
||||
|
|
@ -1096,7 +1105,7 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
for (String filename : filenames)
|
||||
{
|
||||
// extract keyspace and columnfamily name from filename
|
||||
Descriptor desc = Descriptor.fromFilename(filename.trim());
|
||||
Descriptor desc = Descriptor.fromFileWithComponent(new File(filename.trim()), false).left;
|
||||
if (Schema.instance.getTableMetadataRef(desc) == null)
|
||||
{
|
||||
logger.warn("Schema does not exist for file {}. Skipping.", filename);
|
||||
|
|
@ -1122,7 +1131,7 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
for (String filename : filenames)
|
||||
{
|
||||
// extract keyspace and columnfamily name from filename
|
||||
Descriptor desc = Descriptor.fromFilename(filename.trim());
|
||||
Descriptor desc = Descriptor.fromFileWithComponent(new File(filename.trim()), false).left;
|
||||
if (Schema.instance.getTableMetadataRef(desc) == null)
|
||||
{
|
||||
logger.warn("Schema does not exist for file {}. Skipping.", filename);
|
||||
|
|
@ -1243,11 +1252,11 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
}
|
||||
|
||||
@VisibleForTesting
|
||||
void scrubOne(ColumnFamilyStore cfs, LifecycleTransaction modifier, boolean skipCorrupted, boolean checkData, boolean reinsertOverflowedTTL, ActiveCompactionsTracker activeCompactions)
|
||||
void scrubOne(ColumnFamilyStore cfs, LifecycleTransaction modifier, IScrubber.Options options, ActiveCompactionsTracker activeCompactions)
|
||||
{
|
||||
CompactionInfo.Holder scrubInfo = null;
|
||||
|
||||
try (Scrubber scrubber = new Scrubber(cfs, modifier, skipCorrupted, checkData, reinsertOverflowedTTL))
|
||||
SSTableFormat format = modifier.onlyOne().descriptor.getFormat();
|
||||
try (IScrubber scrubber = format.getScrubber(cfs, modifier, new OutputHandler.LogOutput(), options))
|
||||
{
|
||||
scrubInfo = scrubber.getScrubInfo();
|
||||
activeCompactions.beginCompaction(scrubInfo);
|
||||
|
|
@ -1261,11 +1270,10 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
}
|
||||
|
||||
@VisibleForTesting
|
||||
void verifyOne(ColumnFamilyStore cfs, SSTableReader sstable, Verifier.Options options, ActiveCompactionsTracker activeCompactions)
|
||||
void verifyOne(ColumnFamilyStore cfs, SSTableReader sstable, IVerifier.Options options, ActiveCompactionsTracker activeCompactions)
|
||||
{
|
||||
CompactionInfo.Holder verifyInfo = null;
|
||||
|
||||
try (Verifier verifier = new Verifier(cfs, sstable, false, options))
|
||||
try (IVerifier verifier = sstable.getVerifier(cfs, new OutputHandler.LogOutput(), false, options))
|
||||
{
|
||||
verifyInfo = verifier.getVerifyInfo();
|
||||
activeCompactions.beginCompaction(verifyInfo);
|
||||
|
|
@ -1296,7 +1304,7 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
// see if there are any keys LTE the token for the start of the first range
|
||||
// (token range ownership is exclusive on the LHS.)
|
||||
Range<Token> firstRange = sortedRanges.get(0);
|
||||
if (sstable.first.getToken().compareTo(firstRange.left) <= 0)
|
||||
if (sstable.getFirst().getToken().compareTo(firstRange.left) <= 0)
|
||||
return true;
|
||||
|
||||
// then, iterate over all owned ranges and see if the next key beyond the end of the owned
|
||||
|
|
@ -1352,11 +1360,11 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
SSTableReader sstable = txn.onlyOne();
|
||||
|
||||
// if ranges is empty and no index, entire sstable is discarded
|
||||
if (!hasIndexes && !new Bounds<>(sstable.first.getToken(), sstable.last.getToken()).intersects(allRanges))
|
||||
if (!hasIndexes && !sstable.getBounds().intersects(allRanges))
|
||||
{
|
||||
txn.obsoleteOriginals();
|
||||
txn.finish();
|
||||
logger.info("SSTable {} ([{}, {}]) does not intersect the owned ranges ({}), dropping it", sstable, sstable.first.getToken(), sstable.last.getToken(), allRanges);
|
||||
logger.info("SSTable {} ([{}, {}]) does not intersect the owned ranges ({}), dropping it", sstable, sstable.getFirst().getToken(), sstable.getLast().getToken(), allRanges);
|
||||
return;
|
||||
}
|
||||
|
||||
|
|
@ -1556,16 +1564,18 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
{
|
||||
FileUtils.createDirectory(compactionFileLocation);
|
||||
|
||||
return SSTableWriter.create(cfs.metadata,
|
||||
cfs.newSSTableDescriptor(compactionFileLocation),
|
||||
expectedBloomFilterSize,
|
||||
repairedAt,
|
||||
pendingRepair,
|
||||
isTransient,
|
||||
sstable.getSSTableLevel(),
|
||||
sstable.header,
|
||||
cfs.indexManager.listIndexes(),
|
||||
txn);
|
||||
Descriptor descriptor = cfs.newSSTableDescriptor(compactionFileLocation);
|
||||
return descriptor.getFormat().getWriterFactory().builder(descriptor)
|
||||
.setKeyCount(expectedBloomFilterSize)
|
||||
.setRepairedAt(repairedAt)
|
||||
.setPendingRepair(pendingRepair)
|
||||
.setTransientSSTable(isTransient)
|
||||
.setTableMetadataRef(cfs.metadata)
|
||||
.setMetadataCollector(new MetadataCollector(cfs.metadata().comparator).sstableLevel(sstable.getSSTableLevel()))
|
||||
.setSerializationHeader(sstable.header)
|
||||
.addDefaultComponents()
|
||||
.addFlushObserversForSecondaryIndexes(cfs.indexManager.listIndexes(), txn.opType())
|
||||
.build(txn, cfs);
|
||||
}
|
||||
|
||||
public static SSTableWriter createWriterForAntiCompaction(ColumnFamilyStore cfs,
|
||||
|
|
@ -1593,16 +1603,19 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
break;
|
||||
}
|
||||
}
|
||||
return SSTableWriter.create(cfs.newSSTableDescriptor(compactionFileLocation),
|
||||
(long) expectedBloomFilterSize,
|
||||
repairedAt,
|
||||
pendingRepair,
|
||||
isTransient,
|
||||
cfs.metadata,
|
||||
new MetadataCollector(sstables, cfs.metadata().comparator, minLevel),
|
||||
SerializationHeader.make(cfs.metadata(), sstables),
|
||||
cfs.indexManager.listIndexes(),
|
||||
txn);
|
||||
|
||||
Descriptor descriptor = cfs.newSSTableDescriptor(compactionFileLocation);
|
||||
return descriptor.getFormat().getWriterFactory().builder(descriptor)
|
||||
.setKeyCount(expectedBloomFilterSize)
|
||||
.setRepairedAt(repairedAt)
|
||||
.setPendingRepair(pendingRepair)
|
||||
.setTransientSSTable(isTransient)
|
||||
.setTableMetadataRef(cfs.metadata)
|
||||
.setMetadataCollector(new MetadataCollector(sstables, cfs.metadata().comparator, minLevel))
|
||||
.setSerializationHeader(SerializationHeader.make(cfs.metadata(), sstables))
|
||||
.addDefaultComponents()
|
||||
.addFlushObserversForSecondaryIndexes(cfs.indexManager.listIndexes(), txn.opType())
|
||||
.build(txn, cfs);
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -1894,22 +1907,16 @@ public class CompactionManager implements CompactionManagerMBean
|
|||
return executor.submitIfRunning(runnable, "cache write");
|
||||
}
|
||||
|
||||
public List<SSTableReader> runIndexSummaryRedistribution(IndexSummaryRedistribution redistribution) throws IOException
|
||||
public <T, E extends Throwable> T runAsActiveCompaction(Holder activeCompactionInfo, ThrowingSupplier<T, E> callable) throws E
|
||||
{
|
||||
return runIndexSummaryRedistribution(redistribution, active);
|
||||
}
|
||||
|
||||
@VisibleForTesting
|
||||
List<SSTableReader> runIndexSummaryRedistribution(IndexSummaryRedistribution redistribution, ActiveCompactionsTracker activeCompactions) throws IOException
|
||||
{
|
||||
activeCompactions.beginCompaction(redistribution);
|
||||
active.beginCompaction(activeCompactionInfo);
|
||||
try
|
||||
{
|
||||
return redistribution.redistributeSummaries();
|
||||
return callable.get();
|
||||
}
|
||||
finally
|
||||
{
|
||||
activeCompactions.finishCompaction(redistribution);
|
||||
active.finishCompaction(activeCompactionInfo);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -58,11 +58,11 @@ import org.apache.cassandra.db.lifecycle.SSTableSet;
|
|||
import org.apache.cassandra.dht.Range;
|
||||
import org.apache.cassandra.dht.Token;
|
||||
import org.apache.cassandra.index.Index;
|
||||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.ISSTableScanner;
|
||||
import org.apache.cassandra.io.sstable.SSTable;
|
||||
import org.apache.cassandra.io.sstable.SSTableMultiWriter;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat.Components;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
|
||||
import org.apache.cassandra.io.sstable.metadata.StatsMetadata;
|
||||
|
|
@ -252,8 +252,8 @@ public class CompactionStrategyManager implements INotificationConsumer
|
|||
.stream()
|
||||
.filter(s -> !compacting.contains(s) && !s.descriptor.version.isLatestVersion())
|
||||
.sorted((o1, o2) -> {
|
||||
File f1 = new File(o1.descriptor.filenameFor(Component.DATA));
|
||||
File f2 = new File(o2.descriptor.filenameFor(Component.DATA));
|
||||
File f1 = o1.descriptor.fileFor(Components.DATA);
|
||||
File f2 = o2.descriptor.fileFor(Components.DATA);
|
||||
return Longs.compare(f1.lastModified(), f2.lastModified());
|
||||
}).collect(Collectors.toList());
|
||||
for (SSTableReader sstable : potentialUpgrade)
|
||||
|
|
|
|||
|
|
@ -241,7 +241,7 @@ public class CompactionTask extends AbstractCompactionTask
|
|||
|
||||
StringBuilder newSSTableNames = new StringBuilder();
|
||||
for (SSTableReader reader : newSStables)
|
||||
newSSTableNames.append(reader.descriptor.baseFilename()).append(",");
|
||||
newSSTableNames.append(reader.descriptor.baseFile()).append(",");
|
||||
long totalSourceRows = 0;
|
||||
for (int i = 0; i < mergedRowCounts.length; i++)
|
||||
totalSourceRows += mergedRowCounts[i] * (i + 1);
|
||||
|
|
|
|||
|
|
@ -17,7 +17,16 @@
|
|||
*/
|
||||
package org.apache.cassandra.db.compaction;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.Arrays;
|
||||
import java.util.Collection;
|
||||
import java.util.Collections;
|
||||
import java.util.Comparator;
|
||||
import java.util.HashMap;
|
||||
import java.util.HashSet;
|
||||
import java.util.Iterator;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.function.Function;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
|
|
@ -27,18 +36,16 @@ import com.google.common.collect.ImmutableSet;
|
|||
import com.google.common.collect.Iterables;
|
||||
import com.google.common.collect.Sets;
|
||||
import com.google.common.primitives.Ints;
|
||||
|
||||
import org.apache.cassandra.db.PartitionPosition;
|
||||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.apache.cassandra.config.DatabaseDescriptor;
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.PartitionPosition;
|
||||
import org.apache.cassandra.dht.Bounds;
|
||||
import org.apache.cassandra.dht.Range;
|
||||
import org.apache.cassandra.dht.Token;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.service.StorageService;
|
||||
import org.apache.cassandra.utils.Pair;
|
||||
|
||||
|
|
@ -115,7 +122,7 @@ public class LeveledManifest
|
|||
long maxModificationTime = Long.MIN_VALUE;
|
||||
for (SSTableReader ssTableReader : level)
|
||||
{
|
||||
long modificationTime = ssTableReader.getCreationTimeFor(Component.DATA);
|
||||
long modificationTime = ssTableReader.getDataCreationTime();
|
||||
if (modificationTime >= maxModificationTime)
|
||||
{
|
||||
sstableWithMaxModificationTime = ssTableReader;
|
||||
|
|
|
|||
|
|
@ -1,864 +0,0 @@
|
|||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.apache.cassandra.db.compaction;
|
||||
|
||||
import java.io.IOError;
|
||||
import java.io.IOException;
|
||||
import java.nio.ByteBuffer;
|
||||
|
||||
import java.nio.file.Paths;
|
||||
import java.util.*;
|
||||
import java.util.concurrent.locks.Lock;
|
||||
import java.util.concurrent.locks.ReadWriteLock;
|
||||
import java.util.concurrent.locks.ReentrantReadWriteLock;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import com.google.common.collect.ImmutableSet;
|
||||
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.db.*;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
|
||||
import org.apache.cassandra.db.rows.*;
|
||||
import org.apache.cassandra.db.partitions.*;
|
||||
import org.apache.cassandra.io.sstable.*;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableWriter;
|
||||
import org.apache.cassandra.io.sstable.metadata.StatsMetadata;
|
||||
import org.apache.cassandra.io.util.FileUtils;
|
||||
import org.apache.cassandra.io.util.RandomAccessReader;
|
||||
import org.apache.cassandra.service.ActiveRepairService;
|
||||
import org.apache.cassandra.utils.*;
|
||||
import org.apache.cassandra.utils.concurrent.Refs;
|
||||
import org.apache.cassandra.utils.memory.HeapCloner;
|
||||
|
||||
import static org.apache.cassandra.utils.TimeUUID.Generator.nextTimeUUID;
|
||||
|
||||
public class Scrubber implements Closeable
|
||||
{
|
||||
private final ColumnFamilyStore cfs;
|
||||
private final SSTableReader sstable;
|
||||
private final LifecycleTransaction transaction;
|
||||
private final File destination;
|
||||
private final boolean skipCorrupted;
|
||||
private final boolean reinsertOverflowedTTLRows;
|
||||
|
||||
private final boolean isCommutative;
|
||||
private final boolean isIndex;
|
||||
private final boolean checkData;
|
||||
private final long expectedBloomFilterSize;
|
||||
|
||||
private final ReadWriteLock fileAccessLock;
|
||||
private final RandomAccessReader dataFile;
|
||||
private final RandomAccessReader indexFile;
|
||||
private final ScrubInfo scrubInfo;
|
||||
private final RowIndexEntry.IndexSerializer rowIndexEntrySerializer;
|
||||
|
||||
private int goodPartitions;
|
||||
private int badPartitions;
|
||||
private int emptyPartitions;
|
||||
|
||||
private ByteBuffer currentIndexKey;
|
||||
private ByteBuffer nextIndexKey;
|
||||
private long currentPartitionPositionFromIndex;
|
||||
private long nextPartitionPositionFromIndex;
|
||||
|
||||
private NegativeLocalDeletionInfoMetrics negativeLocalDeletionInfoMetrics = new NegativeLocalDeletionInfoMetrics();
|
||||
|
||||
private final OutputHandler outputHandler;
|
||||
|
||||
private static final Comparator<Partition> partitionComparator = Comparator.comparing(Partition::partitionKey);
|
||||
private final SortedSet<Partition> outOfOrder = new TreeSet<>(partitionComparator);
|
||||
|
||||
public Scrubber(ColumnFamilyStore cfs, LifecycleTransaction transaction, boolean skipCorrupted, boolean checkData)
|
||||
{
|
||||
this(cfs, transaction, skipCorrupted, checkData, false);
|
||||
}
|
||||
|
||||
public Scrubber(ColumnFamilyStore cfs, LifecycleTransaction transaction, boolean skipCorrupted, boolean checkData,
|
||||
boolean reinsertOverflowedTTLRows)
|
||||
{
|
||||
this(cfs, transaction, skipCorrupted, new OutputHandler.LogOutput(), checkData, reinsertOverflowedTTLRows);
|
||||
}
|
||||
|
||||
@SuppressWarnings("resource")
|
||||
public Scrubber(ColumnFamilyStore cfs,
|
||||
LifecycleTransaction transaction,
|
||||
boolean skipCorrupted,
|
||||
OutputHandler outputHandler,
|
||||
boolean checkData,
|
||||
boolean reinsertOverflowedTTLRows)
|
||||
{
|
||||
this.cfs = cfs;
|
||||
this.transaction = transaction;
|
||||
this.sstable = transaction.onlyOne();
|
||||
this.outputHandler = outputHandler;
|
||||
this.skipCorrupted = skipCorrupted;
|
||||
this.reinsertOverflowedTTLRows = reinsertOverflowedTTLRows;
|
||||
this.rowIndexEntrySerializer = sstable.descriptor.version.getSSTableFormat().getIndexSerializer(cfs.metadata(),
|
||||
sstable.descriptor.version,
|
||||
sstable.header);
|
||||
|
||||
List<SSTableReader> toScrub = Collections.singletonList(sstable);
|
||||
|
||||
this.destination = cfs.getDirectories().getLocationForDisk(cfs.getDiskBoundaries().getCorrectDiskForSSTable(sstable));
|
||||
this.isCommutative = cfs.metadata().isCounter();
|
||||
|
||||
boolean hasIndexFile = (new File(sstable.descriptor.filenameFor(Component.PRIMARY_INDEX))).exists();
|
||||
this.isIndex = cfs.isIndex();
|
||||
if (!hasIndexFile)
|
||||
{
|
||||
// if there's any corruption in the -Data.db then partitions can't be skipped over. but it's worth a shot.
|
||||
outputHandler.warn("Missing component: " + sstable.descriptor.filenameFor(Component.PRIMARY_INDEX));
|
||||
}
|
||||
this.checkData = checkData && !this.isIndex; //LocalByPartitionerType does not support validation
|
||||
this.expectedBloomFilterSize = Math.max(
|
||||
cfs.metadata().params.minIndexInterval,
|
||||
hasIndexFile ? SSTableReader.getApproximateKeyCount(toScrub) : 0);
|
||||
|
||||
this.fileAccessLock = new ReentrantReadWriteLock();
|
||||
// loop through each partition, deserializing to check for damage.
|
||||
// We'll also loop through the index at the same time, using the position from the index to recover if the
|
||||
// partition header (key or data size) is corrupt. (This means our position in the index file will be one
|
||||
// partition "ahead" of the data file.)
|
||||
this.dataFile = transaction.isOffline()
|
||||
? sstable.openDataReader()
|
||||
: sstable.openDataReader(CompactionManager.instance.getRateLimiter());
|
||||
|
||||
this.indexFile = hasIndexFile
|
||||
? RandomAccessReader.open(new File(sstable.descriptor.filenameFor(Component.PRIMARY_INDEX)))
|
||||
: null;
|
||||
|
||||
this.scrubInfo = new ScrubInfo(dataFile, sstable, fileAccessLock.readLock());
|
||||
|
||||
this.currentPartitionPositionFromIndex = 0;
|
||||
this.nextPartitionPositionFromIndex = 0;
|
||||
|
||||
if (reinsertOverflowedTTLRows)
|
||||
outputHandler.output("Starting scrub with reinsert overflowed TTL option");
|
||||
}
|
||||
|
||||
private UnfilteredRowIterator withValidation(UnfilteredRowIterator iter, String filename)
|
||||
{
|
||||
return checkData ? UnfilteredRowIterators.withValidation(iter, filename) : iter;
|
||||
}
|
||||
|
||||
private String keyString(DecoratedKey key)
|
||||
{
|
||||
if (key == null)
|
||||
return "(unknown)";
|
||||
|
||||
try
|
||||
{
|
||||
return cfs.metadata().partitionKeyType.getString(key.getKey());
|
||||
}
|
||||
catch (Exception e)
|
||||
{
|
||||
return String.format("(corrupted; hex value: %s)", ByteBufferUtil.bytesToHex(key.getKey()));
|
||||
}
|
||||
}
|
||||
|
||||
public void scrub()
|
||||
{
|
||||
List<SSTableReader> finished = new ArrayList<>();
|
||||
outputHandler.output(String.format("Scrubbing %s (%s)", sstable, FBUtilities.prettyPrintMemory(dataFile.length())));
|
||||
try (SSTableRewriter writer = SSTableRewriter.construct(cfs, transaction, false, sstable.maxDataAge);
|
||||
Refs<SSTableReader> refs = Refs.ref(Collections.singleton(sstable)))
|
||||
{
|
||||
nextIndexKey = indexAvailable() ? ByteBufferUtil.readWithShortLength(indexFile) : null;
|
||||
if (indexAvailable())
|
||||
{
|
||||
// throw away variable so we don't have a side effect in the assert
|
||||
long firstRowPositionFromIndex = rowIndexEntrySerializer.deserializePositionAndSkip(indexFile);
|
||||
assert firstRowPositionFromIndex == 0 : firstRowPositionFromIndex;
|
||||
}
|
||||
|
||||
StatsMetadata metadata = sstable.getSSTableMetadata();
|
||||
writer.switchWriter(CompactionManager.createWriter(cfs, destination, expectedBloomFilterSize, metadata.repairedAt, metadata.pendingRepair, metadata.isTransient, sstable, transaction));
|
||||
|
||||
DecoratedKey prevKey = null;
|
||||
|
||||
while (!dataFile.isEOF())
|
||||
{
|
||||
if (scrubInfo.isStopRequested())
|
||||
throw new CompactionInterruptedException(scrubInfo.getCompactionInfo());
|
||||
|
||||
long partitionStart = dataFile.getFilePointer();
|
||||
outputHandler.debug("Reading row at " + partitionStart);
|
||||
|
||||
DecoratedKey key = null;
|
||||
try
|
||||
{
|
||||
ByteBuffer raw = ByteBufferUtil.readWithShortLength(dataFile);
|
||||
if (!cfs.metadata.getLocal().isIndex())
|
||||
cfs.metadata.getLocal().partitionKeyType.validate(raw);
|
||||
key = sstable.decorateKey(raw);
|
||||
}
|
||||
catch (Throwable th)
|
||||
{
|
||||
throwIfFatal(th);
|
||||
// check for null key below
|
||||
}
|
||||
|
||||
updateIndexKey();
|
||||
|
||||
long dataStart = dataFile.getFilePointer();
|
||||
|
||||
long dataStartFromIndex = -1;
|
||||
long dataSizeFromIndex = -1;
|
||||
if (currentIndexKey != null)
|
||||
{
|
||||
dataStartFromIndex = currentPartitionPositionFromIndex + 2 + currentIndexKey.remaining();
|
||||
dataSizeFromIndex = nextPartitionPositionFromIndex - dataStartFromIndex;
|
||||
}
|
||||
|
||||
String keyName = key == null ? "(unreadable key)" : keyString(key);
|
||||
outputHandler.debug(String.format("partition %s is %s", keyName, FBUtilities.prettyPrintMemory(dataSizeFromIndex)));
|
||||
assert currentIndexKey != null || !indexAvailable();
|
||||
|
||||
try
|
||||
{
|
||||
if (key == null)
|
||||
throw new IOError(new IOException("Unable to read partition key from data file"));
|
||||
|
||||
if (currentIndexKey != null && !key.getKey().equals(currentIndexKey))
|
||||
{
|
||||
throw new IOError(new IOException(String.format("Key from data file (%s) does not match key from index file (%s)",
|
||||
//ByteBufferUtil.bytesToHex(key.getKey()), ByteBufferUtil.bytesToHex(currentIndexKey))));
|
||||
"_too big_", ByteBufferUtil.bytesToHex(currentIndexKey))));
|
||||
}
|
||||
|
||||
if (indexFile != null && dataSizeFromIndex > dataFile.length())
|
||||
throw new IOError(new IOException("Impossible partition size (greater than file length): " + dataSizeFromIndex));
|
||||
|
||||
if (indexFile != null && dataStart != dataStartFromIndex)
|
||||
outputHandler.warn(String.format("Data file partition position %d differs from index file row position %d", dataStart, dataStartFromIndex));
|
||||
|
||||
if (tryAppend(prevKey, key, writer))
|
||||
prevKey = key;
|
||||
}
|
||||
catch (Throwable th)
|
||||
{
|
||||
throwIfFatal(th);
|
||||
outputHandler.warn(String.format("Error reading partition %s (stacktrace follows):", keyName), th);
|
||||
|
||||
if (currentIndexKey != null
|
||||
&& (key == null || !key.getKey().equals(currentIndexKey) || dataStart != dataStartFromIndex))
|
||||
{
|
||||
|
||||
outputHandler.output(String.format("Retrying from partition index; data is %s bytes starting at %s",
|
||||
dataSizeFromIndex, dataStartFromIndex));
|
||||
key = sstable.decorateKey(currentIndexKey);
|
||||
try
|
||||
{
|
||||
if (!cfs.metadata.getLocal().isIndex())
|
||||
cfs.metadata.getLocal().partitionKeyType.validate(key.getKey());
|
||||
dataFile.seek(dataStartFromIndex);
|
||||
|
||||
if (tryAppend(prevKey, key, writer))
|
||||
prevKey = key;
|
||||
}
|
||||
catch (Throwable th2)
|
||||
{
|
||||
throwIfFatal(th2);
|
||||
throwIfCannotContinue(key, th2);
|
||||
|
||||
outputHandler.warn("Retry failed too. Skipping to next partition (retry's stacktrace follows)", th2);
|
||||
badPartitions++;
|
||||
if (!seekToNextPartition())
|
||||
break;
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
throwIfCannotContinue(key, th);
|
||||
|
||||
outputHandler.warn("Partition starting at position " + dataStart + " is unreadable; skipping to next");
|
||||
badPartitions++;
|
||||
if (currentIndexKey != null)
|
||||
if (!seekToNextPartition())
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!outOfOrder.isEmpty())
|
||||
{
|
||||
// out of order partitions/rows, but no bad partition found - we can keep our repairedAt time
|
||||
long repairedAt = badPartitions > 0 ? ActiveRepairService.UNREPAIRED_SSTABLE : sstable.getSSTableMetadata().repairedAt;
|
||||
SSTableReader newInOrderSstable;
|
||||
try (SSTableWriter inOrderWriter = CompactionManager.createWriter(cfs, destination, expectedBloomFilterSize, repairedAt, metadata.pendingRepair, metadata.isTransient, sstable, transaction))
|
||||
{
|
||||
for (Partition partition : outOfOrder)
|
||||
inOrderWriter.append(partition.unfilteredIterator());
|
||||
newInOrderSstable = inOrderWriter.finish(-1, sstable.maxDataAge, true);
|
||||
}
|
||||
transaction.update(newInOrderSstable, false);
|
||||
finished.add(newInOrderSstable);
|
||||
outputHandler.warn(String.format("%d out of order partition (or partitions with out of order rows) found while scrubbing %s; " +
|
||||
"Those have been written (in order) to a new sstable (%s)", outOfOrder.size(), sstable, newInOrderSstable));
|
||||
}
|
||||
|
||||
// finish obsoletes the old sstable
|
||||
finished.addAll(writer.setRepairedAt(badPartitions > 0 ? ActiveRepairService.UNREPAIRED_SSTABLE : sstable.getSSTableMetadata().repairedAt).finish());
|
||||
}
|
||||
catch (IOException e)
|
||||
{
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (transaction.isOffline())
|
||||
finished.forEach(sstable -> sstable.selfRef().release());
|
||||
}
|
||||
|
||||
if (!finished.isEmpty())
|
||||
{
|
||||
outputHandler.output("Scrub of " + sstable + " complete: " + goodPartitions + " partitions in new sstable and " + emptyPartitions + " empty (tombstoned) partitions dropped");
|
||||
if (negativeLocalDeletionInfoMetrics.fixedRows > 0)
|
||||
outputHandler.output("Fixed " + negativeLocalDeletionInfoMetrics.fixedRows + " rows with overflowed local deletion time.");
|
||||
if (badPartitions > 0)
|
||||
outputHandler.warn("Unable to recover " + badPartitions + " partitions that were skipped. You can attempt manual recovery from the pre-scrub snapshot. You can also run nodetool repair to transfer the data from a healthy replica, if any");
|
||||
}
|
||||
else
|
||||
{
|
||||
if (badPartitions > 0)
|
||||
outputHandler.warn("No valid partitions found while scrubbing " + sstable + "; it is marked for deletion now. If you want to attempt manual recovery, you can find a copy in the pre-scrub snapshot");
|
||||
else
|
||||
outputHandler.output("Scrub of " + sstable + " complete; looks like all " + emptyPartitions + " partitions were tombstoned");
|
||||
}
|
||||
}
|
||||
|
||||
@SuppressWarnings("resource")
|
||||
private boolean tryAppend(DecoratedKey prevKey, DecoratedKey key, SSTableRewriter writer)
|
||||
{
|
||||
// OrderCheckerIterator will check, at iteration time, that the rows are in the proper order. If it detects
|
||||
// that one row is out of order, it will stop returning them. The remaining rows will be sorted and added
|
||||
// to the outOfOrder set that will be later written to a new SSTable.
|
||||
OrderCheckerIterator sstableIterator = new OrderCheckerIterator(getIterator(key),
|
||||
cfs.metadata().comparator);
|
||||
|
||||
try (UnfilteredRowIterator iterator = withValidation(sstableIterator, dataFile.getPath()))
|
||||
{
|
||||
if (prevKey != null && prevKey.compareTo(key) > 0)
|
||||
{
|
||||
saveOutOfOrderPartition(prevKey, key, iterator);
|
||||
return false;
|
||||
}
|
||||
|
||||
if (writer.tryAppend(iterator) == null)
|
||||
emptyPartitions++;
|
||||
else
|
||||
goodPartitions++;
|
||||
}
|
||||
|
||||
if (sstableIterator.hasRowsOutOfOrder())
|
||||
{
|
||||
outputHandler.warn(String.format("Out of order rows found in partition: %s", keyString(key)));
|
||||
outOfOrder.add(sstableIterator.getRowsOutOfOrder());
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Only wrap with {@link FixNegativeLocalDeletionTimeIterator} if {@link #reinsertOverflowedTTLRows} option
|
||||
* is specified
|
||||
*/
|
||||
@SuppressWarnings("resource")
|
||||
private UnfilteredRowIterator getIterator(DecoratedKey key)
|
||||
{
|
||||
RowMergingSSTableIterator rowMergingIterator = new RowMergingSSTableIterator(SSTableIdentityIterator.create(sstable, dataFile, key), outputHandler);
|
||||
return reinsertOverflowedTTLRows ? new FixNegativeLocalDeletionTimeIterator(rowMergingIterator,
|
||||
outputHandler,
|
||||
negativeLocalDeletionInfoMetrics) : rowMergingIterator;
|
||||
}
|
||||
|
||||
private void updateIndexKey()
|
||||
{
|
||||
currentIndexKey = nextIndexKey;
|
||||
currentPartitionPositionFromIndex = nextPartitionPositionFromIndex;
|
||||
try
|
||||
{
|
||||
nextIndexKey = !indexAvailable() ? null : ByteBufferUtil.readWithShortLength(indexFile);
|
||||
|
||||
nextPartitionPositionFromIndex = !indexAvailable()
|
||||
? dataFile.length()
|
||||
: rowIndexEntrySerializer.deserializePositionAndSkip(indexFile);
|
||||
}
|
||||
catch (Throwable th)
|
||||
{
|
||||
JVMStabilityInspector.inspectThrowable(th);
|
||||
outputHandler.warn("Error reading index file", th);
|
||||
nextIndexKey = null;
|
||||
nextPartitionPositionFromIndex = dataFile.length();
|
||||
}
|
||||
}
|
||||
|
||||
private boolean indexAvailable()
|
||||
{
|
||||
return indexFile != null && !indexFile.isEOF();
|
||||
}
|
||||
|
||||
private boolean seekToNextPartition()
|
||||
{
|
||||
while(nextPartitionPositionFromIndex < dataFile.length())
|
||||
{
|
||||
try
|
||||
{
|
||||
dataFile.seek(nextPartitionPositionFromIndex);
|
||||
return true;
|
||||
}
|
||||
catch (Throwable th)
|
||||
{
|
||||
throwIfFatal(th);
|
||||
outputHandler.warn(String.format("Failed to seek to next partition position %d", nextPartitionPositionFromIndex), th);
|
||||
badPartitions++;
|
||||
}
|
||||
|
||||
updateIndexKey();
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
private void saveOutOfOrderPartition(DecoratedKey prevKey, DecoratedKey key, UnfilteredRowIterator iterator)
|
||||
{
|
||||
// TODO bitch if the row is too large? if it is there's not much we can do ...
|
||||
outputHandler.warn(String.format("Out of order partition detected (%s found after %s)",
|
||||
keyString(key), keyString(prevKey)));
|
||||
outOfOrder.add(ImmutableBTreePartition.create(iterator));
|
||||
}
|
||||
|
||||
private void throwIfFatal(Throwable th)
|
||||
{
|
||||
if (th instanceof Error && !(th instanceof AssertionError || th instanceof IOError))
|
||||
throw (Error) th;
|
||||
}
|
||||
|
||||
private void throwIfCannotContinue(DecoratedKey key, Throwable th)
|
||||
{
|
||||
if (isIndex)
|
||||
{
|
||||
outputHandler.warn(String.format("An error occurred while scrubbing the partition with key '%s' for an index table. " +
|
||||
"Scrubbing will abort for this table and the index will be rebuilt.", keyString(key)));
|
||||
throw new IOError(th);
|
||||
}
|
||||
|
||||
if (isCommutative && !skipCorrupted)
|
||||
{
|
||||
outputHandler.warn(String.format("An error occurred while scrubbing the partition with key '%s'. Skipping corrupt " +
|
||||
"data in counter tables will result in undercounts for the affected " +
|
||||
"counters (see CASSANDRA-2759 for more details), so by default the scrub will " +
|
||||
"stop at this point. If you would like to skip the row anyway and continue " +
|
||||
"scrubbing, re-run the scrub with the --skip-corrupted option.",
|
||||
keyString(key)));
|
||||
throw new IOError(th);
|
||||
}
|
||||
}
|
||||
|
||||
public void close()
|
||||
{
|
||||
fileAccessLock.writeLock().lock();
|
||||
try
|
||||
{
|
||||
FileUtils.closeQuietly(dataFile);
|
||||
FileUtils.closeQuietly(indexFile);
|
||||
}
|
||||
finally
|
||||
{
|
||||
fileAccessLock.writeLock().unlock();
|
||||
}
|
||||
}
|
||||
|
||||
public CompactionInfo.Holder getScrubInfo()
|
||||
{
|
||||
return scrubInfo;
|
||||
}
|
||||
|
||||
private static class ScrubInfo extends CompactionInfo.Holder
|
||||
{
|
||||
private final RandomAccessReader dataFile;
|
||||
private final SSTableReader sstable;
|
||||
private final TimeUUID scrubCompactionId;
|
||||
private final Lock fileReadLock;
|
||||
|
||||
public ScrubInfo(RandomAccessReader dataFile, SSTableReader sstable, Lock fileReadLock)
|
||||
{
|
||||
this.dataFile = dataFile;
|
||||
this.sstable = sstable;
|
||||
this.fileReadLock = fileReadLock;
|
||||
scrubCompactionId = nextTimeUUID();
|
||||
}
|
||||
|
||||
public CompactionInfo getCompactionInfo()
|
||||
{
|
||||
fileReadLock.lock();
|
||||
try
|
||||
{
|
||||
return new CompactionInfo(sstable.metadata(),
|
||||
OperationType.SCRUB,
|
||||
dataFile.getFilePointer(),
|
||||
dataFile.length(),
|
||||
scrubCompactionId,
|
||||
ImmutableSet.of(sstable),
|
||||
Paths.get(sstable.getFilename()).getParent().toString());
|
||||
}
|
||||
catch (Exception e)
|
||||
{
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
finally
|
||||
{
|
||||
fileReadLock.unlock();
|
||||
}
|
||||
}
|
||||
|
||||
public boolean isGlobal()
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
@VisibleForTesting
|
||||
public ScrubResult scrubWithResult()
|
||||
{
|
||||
scrub();
|
||||
return new ScrubResult(this);
|
||||
}
|
||||
|
||||
public static final class ScrubResult
|
||||
{
|
||||
public final int goodPartitions;
|
||||
public final int badPartitions;
|
||||
public final int emptyPartitions;
|
||||
|
||||
public ScrubResult(Scrubber scrubber)
|
||||
{
|
||||
this.goodPartitions = scrubber.goodPartitions;
|
||||
this.badPartitions = scrubber.badPartitions;
|
||||
this.emptyPartitions = scrubber.emptyPartitions;
|
||||
}
|
||||
}
|
||||
|
||||
public class NegativeLocalDeletionInfoMetrics
|
||||
{
|
||||
public volatile int fixedRows = 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* During 2.x migration, under some circumstances rows might have gotten duplicated.
|
||||
* Merging iterator merges rows with same clustering.
|
||||
*
|
||||
* For more details, refer to CASSANDRA-12144.
|
||||
*/
|
||||
private static class RowMergingSSTableIterator extends WrappingUnfilteredRowIterator
|
||||
{
|
||||
Unfiltered nextToOffer = null;
|
||||
private final OutputHandler output;
|
||||
|
||||
RowMergingSSTableIterator(UnfilteredRowIterator source, OutputHandler output)
|
||||
{
|
||||
super(source);
|
||||
this.output = output;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean hasNext()
|
||||
{
|
||||
return nextToOffer != null || wrapped.hasNext();
|
||||
}
|
||||
|
||||
@Override
|
||||
public Unfiltered next()
|
||||
{
|
||||
Unfiltered next = nextToOffer != null ? nextToOffer : wrapped.next();
|
||||
|
||||
if (next.isRow())
|
||||
{
|
||||
boolean logged = false;
|
||||
while (wrapped.hasNext())
|
||||
{
|
||||
Unfiltered peek = wrapped.next();
|
||||
if (!peek.isRow() || !next.clustering().equals(peek.clustering()))
|
||||
{
|
||||
nextToOffer = peek; // Offer peek in next call
|
||||
return next;
|
||||
}
|
||||
|
||||
// Duplicate row, merge it.
|
||||
next = Rows.merge((Row) next, (Row) peek);
|
||||
|
||||
if (!logged)
|
||||
{
|
||||
String partitionKey = metadata().partitionKeyType.getString(partitionKey().getKey());
|
||||
output.warn("Duplicate row detected in " + metadata().keyspace + '.' + metadata().name + ": " + partitionKey + " " + next.clustering().toString(metadata()));
|
||||
logged = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
nextToOffer = null;
|
||||
return next;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* In some case like CASSANDRA-12127 the cells might have been stored in the wrong order. This decorator check the
|
||||
* cells order and collect the out of order cells to correct the problem.
|
||||
*/
|
||||
private static final class OrderCheckerIterator extends AbstractIterator<Unfiltered> implements UnfilteredRowIterator
|
||||
{
|
||||
/**
|
||||
* The decorated iterator.
|
||||
*/
|
||||
private final UnfilteredRowIterator iterator;
|
||||
|
||||
private final ClusteringComparator comparator;
|
||||
|
||||
private Unfiltered previous;
|
||||
|
||||
/**
|
||||
* The partition containing the rows which are out of order.
|
||||
*/
|
||||
private Partition rowsOutOfOrder;
|
||||
|
||||
public OrderCheckerIterator(UnfilteredRowIterator iterator, ClusteringComparator comparator)
|
||||
{
|
||||
this.iterator = iterator;
|
||||
this.comparator = comparator;
|
||||
}
|
||||
|
||||
public TableMetadata metadata()
|
||||
{
|
||||
return iterator.metadata();
|
||||
}
|
||||
|
||||
public boolean isReverseOrder()
|
||||
{
|
||||
return iterator.isReverseOrder();
|
||||
}
|
||||
|
||||
public RegularAndStaticColumns columns()
|
||||
{
|
||||
return iterator.columns();
|
||||
}
|
||||
|
||||
public DecoratedKey partitionKey()
|
||||
{
|
||||
return iterator.partitionKey();
|
||||
}
|
||||
|
||||
public Row staticRow()
|
||||
{
|
||||
return iterator.staticRow();
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isEmpty()
|
||||
{
|
||||
return iterator.isEmpty();
|
||||
}
|
||||
|
||||
public void close()
|
||||
{
|
||||
iterator.close();
|
||||
}
|
||||
|
||||
public DeletionTime partitionLevelDeletion()
|
||||
{
|
||||
return iterator.partitionLevelDeletion();
|
||||
}
|
||||
|
||||
public EncodingStats stats()
|
||||
{
|
||||
return iterator.stats();
|
||||
}
|
||||
|
||||
public boolean hasRowsOutOfOrder()
|
||||
{
|
||||
return rowsOutOfOrder != null;
|
||||
}
|
||||
|
||||
public Partition getRowsOutOfOrder()
|
||||
{
|
||||
return rowsOutOfOrder;
|
||||
}
|
||||
|
||||
protected Unfiltered computeNext()
|
||||
{
|
||||
if (!iterator.hasNext())
|
||||
return endOfData();
|
||||
|
||||
Unfiltered next = iterator.next();
|
||||
|
||||
// If we detect that some rows are out of order we will store and sort the remaining ones to insert them
|
||||
// in a separate SSTable.
|
||||
if (previous != null && comparator.compare(next, previous) < 0)
|
||||
{
|
||||
rowsOutOfOrder = ImmutableBTreePartition.create(UnfilteredRowIterators.concat(next, iterator), false);
|
||||
return endOfData();
|
||||
}
|
||||
previous = next;
|
||||
return next;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* This iterator converts negative {@link AbstractCell#localDeletionTime()} into {@link AbstractCell#MAX_DELETION_TIME}
|
||||
*
|
||||
* This is to recover entries with overflowed localExpirationTime due to CASSANDRA-14092
|
||||
*/
|
||||
private static final class FixNegativeLocalDeletionTimeIterator extends AbstractIterator<Unfiltered> implements UnfilteredRowIterator
|
||||
{
|
||||
/**
|
||||
* The decorated iterator.
|
||||
*/
|
||||
private final UnfilteredRowIterator iterator;
|
||||
|
||||
private final OutputHandler outputHandler;
|
||||
private final NegativeLocalDeletionInfoMetrics negativeLocalExpirationTimeMetrics;
|
||||
|
||||
public FixNegativeLocalDeletionTimeIterator(UnfilteredRowIterator iterator, OutputHandler outputHandler,
|
||||
NegativeLocalDeletionInfoMetrics negativeLocalDeletionInfoMetrics)
|
||||
{
|
||||
this.iterator = iterator;
|
||||
this.outputHandler = outputHandler;
|
||||
this.negativeLocalExpirationTimeMetrics = negativeLocalDeletionInfoMetrics;
|
||||
}
|
||||
|
||||
public TableMetadata metadata()
|
||||
{
|
||||
return iterator.metadata();
|
||||
}
|
||||
|
||||
public boolean isReverseOrder()
|
||||
{
|
||||
return iterator.isReverseOrder();
|
||||
}
|
||||
|
||||
public RegularAndStaticColumns columns()
|
||||
{
|
||||
return iterator.columns();
|
||||
}
|
||||
|
||||
public DecoratedKey partitionKey()
|
||||
{
|
||||
return iterator.partitionKey();
|
||||
}
|
||||
|
||||
public Row staticRow()
|
||||
{
|
||||
return iterator.staticRow();
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isEmpty()
|
||||
{
|
||||
return iterator.isEmpty();
|
||||
}
|
||||
|
||||
public void close()
|
||||
{
|
||||
iterator.close();
|
||||
}
|
||||
|
||||
public DeletionTime partitionLevelDeletion()
|
||||
{
|
||||
return iterator.partitionLevelDeletion();
|
||||
}
|
||||
|
||||
public EncodingStats stats()
|
||||
{
|
||||
return iterator.stats();
|
||||
}
|
||||
|
||||
protected Unfiltered computeNext()
|
||||
{
|
||||
if (!iterator.hasNext())
|
||||
return endOfData();
|
||||
|
||||
Unfiltered next = iterator.next();
|
||||
if (!next.isRow())
|
||||
return next;
|
||||
|
||||
if (hasNegativeLocalExpirationTime((Row) next))
|
||||
{
|
||||
outputHandler.debug(String.format("Found row with negative local expiration time: %s", next.toString(metadata(), false)));
|
||||
negativeLocalExpirationTimeMetrics.fixedRows++;
|
||||
return fixNegativeLocalExpirationTime((Row) next);
|
||||
}
|
||||
|
||||
return next;
|
||||
}
|
||||
|
||||
private boolean hasNegativeLocalExpirationTime(Row next)
|
||||
{
|
||||
Row row = next;
|
||||
if (row.primaryKeyLivenessInfo().isExpiring() && row.primaryKeyLivenessInfo().localExpirationTime() < 0)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
for (ColumnData cd : row)
|
||||
{
|
||||
if (cd.column().isSimple())
|
||||
{
|
||||
Cell<?> cell = (Cell<?>)cd;
|
||||
if (cell.isExpiring() && cell.localDeletionTime() < 0)
|
||||
return true;
|
||||
}
|
||||
else
|
||||
{
|
||||
ComplexColumnData complexData = (ComplexColumnData)cd;
|
||||
for (Cell<?> cell : complexData)
|
||||
{
|
||||
if (cell.isExpiring() && cell.localDeletionTime() < 0)
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
private Unfiltered fixNegativeLocalExpirationTime(Row row)
|
||||
{
|
||||
LivenessInfo livenessInfo = row.primaryKeyLivenessInfo();
|
||||
if (livenessInfo.isExpiring() && livenessInfo.localExpirationTime() < 0)
|
||||
livenessInfo = livenessInfo.withUpdatedTimestampAndLocalDeletionTime(livenessInfo.timestamp() + 1, AbstractCell.MAX_DELETION_TIME);
|
||||
|
||||
return row.transformAndFilter(livenessInfo, row.deletion(), cd -> {
|
||||
if (cd.column().isSimple())
|
||||
{
|
||||
Cell cell = (Cell)cd;
|
||||
return cell.isExpiring() && cell.localDeletionTime() < 0
|
||||
? cell.withUpdatedTimestampAndLocalDeletionTime(cell.timestamp() + 1, AbstractCell.MAX_DELETION_TIME)
|
||||
: cell;
|
||||
}
|
||||
else
|
||||
{
|
||||
ComplexColumnData complexData = (ComplexColumnData)cd;
|
||||
return complexData.transformAndFilter(cell -> cell.isExpiring() && cell.localDeletionTime() < 0
|
||||
? cell.withUpdatedTimestampAndLocalDeletionTime(cell.timestamp() + 1, AbstractCell.MAX_DELETION_TIME)
|
||||
: cell);
|
||||
}
|
||||
}).clone(HeapCloner.instance);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -27,8 +27,8 @@ import org.apache.cassandra.db.ColumnFamilyStore;
|
|||
import org.apache.cassandra.db.Directories;
|
||||
import org.apache.cassandra.db.compaction.writers.CompactionAwareWriter;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
|
||||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.CorruptSSTableException;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat.Components;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.metadata.StatsMetadata;
|
||||
|
||||
|
|
@ -82,7 +82,7 @@ public class SingleSSTableLCSTask extends AbstractCompactionTask
|
|||
catch (Throwable t)
|
||||
{
|
||||
transaction.abort();
|
||||
throw new CorruptSSTableException(t, sstable.descriptor.filenameFor(Component.DATA));
|
||||
throw new CorruptSSTableException(t, sstable.descriptor.fileFor(Components.DATA));
|
||||
}
|
||||
cfs.getTracker().notifySSTableMetadataChanged(sstable, metadataBefore);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@
|
|||
*/
|
||||
package org.apache.cassandra.db.compaction;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.Arrays;
|
||||
import java.util.function.LongPredicate;
|
||||
|
||||
import com.google.common.base.Throwables;
|
||||
|
|
@ -25,9 +25,10 @@ import com.google.common.collect.Sets;
|
|||
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
|
||||
import org.apache.cassandra.db.SerializationHeader;
|
||||
import org.apache.cassandra.io.sstable.*;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.SSTableRewriter;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableWriter;
|
||||
import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
|
||||
|
|
@ -72,16 +73,19 @@ public class Upgrader
|
|||
{
|
||||
MetadataCollector sstableMetadataCollector = new MetadataCollector(cfs.getComparator());
|
||||
sstableMetadataCollector.sstableLevel(sstable.getSSTableLevel());
|
||||
return SSTableWriter.create(cfs.newSSTableDescriptor(directory),
|
||||
estimatedRows,
|
||||
metadata.repairedAt,
|
||||
metadata.pendingRepair,
|
||||
metadata.isTransient,
|
||||
cfs.metadata,
|
||||
sstableMetadataCollector,
|
||||
SerializationHeader.make(cfs.metadata(), Sets.newHashSet(sstable)),
|
||||
cfs.indexManager.listIndexes(),
|
||||
transaction);
|
||||
|
||||
Descriptor descriptor = cfs.newSSTableDescriptor(directory);
|
||||
return descriptor.getFormat().getWriterFactory().builder(descriptor)
|
||||
.setKeyCount(estimatedRows)
|
||||
.setRepairedAt(metadata.repairedAt)
|
||||
.setPendingRepair(metadata.pendingRepair)
|
||||
.setTransientSSTable(metadata.isTransient)
|
||||
.setTableMetadataRef(cfs.metadata)
|
||||
.setMetadataCollector(sstableMetadataCollector)
|
||||
.setSerializationHeader(SerializationHeader.make(cfs.metadata(), Sets.newHashSet(sstable)))
|
||||
.addDefaultComponents()
|
||||
.addFlushObserversForSecondaryIndexes(cfs.indexManager.listIndexes(), transaction.opType())
|
||||
.build(transaction, cfs);
|
||||
}
|
||||
|
||||
public void upgrade(boolean keepOriginals)
|
||||
|
|
|
|||
|
|
@ -1,726 +0,0 @@
|
|||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.apache.cassandra.db.compaction;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import com.google.common.base.Throwables;
|
||||
import com.google.common.collect.ImmutableSet;
|
||||
|
||||
import org.apache.cassandra.db.*;
|
||||
import org.apache.cassandra.db.rows.Cell;
|
||||
import org.apache.cassandra.db.rows.Row;
|
||||
import org.apache.cassandra.db.rows.Unfiltered;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
|
||||
import org.apache.cassandra.dht.LocalPartitioner;
|
||||
import org.apache.cassandra.dht.Range;
|
||||
import org.apache.cassandra.dht.Token;
|
||||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.CorruptSSTableException;
|
||||
import org.apache.cassandra.io.sstable.IndexSummary;
|
||||
import org.apache.cassandra.io.sstable.KeyIterator;
|
||||
import org.apache.cassandra.io.sstable.SSTableIdentityIterator;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.metadata.MetadataComponent;
|
||||
import org.apache.cassandra.io.sstable.metadata.MetadataType;
|
||||
import org.apache.cassandra.io.sstable.metadata.ValidationMetadata;
|
||||
import org.apache.cassandra.io.util.DataIntegrityMetadata;
|
||||
import org.apache.cassandra.io.util.DataIntegrityMetadata.FileDigestValidator;
|
||||
import org.apache.cassandra.io.util.FileInputStreamPlus;
|
||||
import org.apache.cassandra.io.util.FileUtils;
|
||||
import org.apache.cassandra.io.util.RandomAccessReader;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.service.ActiveRepairService;
|
||||
import org.apache.cassandra.service.StorageService;
|
||||
import org.apache.cassandra.utils.BloomFilterSerializer;
|
||||
import org.apache.cassandra.utils.ByteBufferUtil;
|
||||
import org.apache.cassandra.utils.FBUtilities;
|
||||
import org.apache.cassandra.utils.IFilter;
|
||||
import org.apache.cassandra.utils.OutputHandler;
|
||||
import org.apache.cassandra.utils.TimeUUID;
|
||||
|
||||
import java.io.Closeable;
|
||||
import java.io.DataInputStream;
|
||||
import java.io.IOError;
|
||||
import java.io.IOException;
|
||||
import java.nio.ByteBuffer;
|
||||
import java.nio.file.Files;
|
||||
import java.time.Instant;
|
||||
import java.util.*;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.locks.Lock;
|
||||
import java.util.concurrent.locks.ReadWriteLock;
|
||||
import java.util.concurrent.locks.ReentrantReadWriteLock;
|
||||
import java.util.function.Function;
|
||||
import java.util.function.LongPredicate;
|
||||
|
||||
import org.apache.cassandra.io.util.File;
|
||||
|
||||
import static org.apache.cassandra.utils.TimeUUID.Generator.nextTimeUUID;
|
||||
|
||||
public class Verifier implements Closeable
|
||||
{
|
||||
private final ColumnFamilyStore cfs;
|
||||
private final SSTableReader sstable;
|
||||
|
||||
private final CompactionController controller;
|
||||
|
||||
private final ReadWriteLock fileAccessLock;
|
||||
private final RandomAccessReader dataFile;
|
||||
private final RandomAccessReader indexFile;
|
||||
private final VerifyInfo verifyInfo;
|
||||
private final RowIndexEntry.IndexSerializer rowIndexEntrySerializer;
|
||||
private final Options options;
|
||||
private final boolean isOffline;
|
||||
/**
|
||||
* Given a keyspace, return the set of local and pending token ranges. By default {@link StorageService#getLocalAndPendingRanges(String)}
|
||||
* is expected, but for the standalone verifier case we can't use that, so this is here to allow the CLI to provide
|
||||
* the token ranges.
|
||||
*/
|
||||
private final Function<String, ? extends Collection<Range<Token>>> tokenLookup;
|
||||
|
||||
private int goodRows;
|
||||
|
||||
private final OutputHandler outputHandler;
|
||||
private FileDigestValidator validator;
|
||||
|
||||
public Verifier(ColumnFamilyStore cfs, SSTableReader sstable, boolean isOffline, Options options)
|
||||
{
|
||||
this(cfs, sstable, new OutputHandler.LogOutput(), isOffline, options);
|
||||
}
|
||||
|
||||
public Verifier(ColumnFamilyStore cfs, SSTableReader sstable, OutputHandler outputHandler, boolean isOffline, Options options)
|
||||
{
|
||||
this.cfs = cfs;
|
||||
this.sstable = sstable;
|
||||
this.outputHandler = outputHandler;
|
||||
this.rowIndexEntrySerializer = sstable.descriptor.version.getSSTableFormat().getIndexSerializer(cfs.metadata(), sstable.descriptor.version, sstable.header);
|
||||
|
||||
this.controller = new VerifyController(cfs);
|
||||
|
||||
this.fileAccessLock = new ReentrantReadWriteLock();
|
||||
this.dataFile = isOffline
|
||||
? sstable.openDataReader()
|
||||
: sstable.openDataReader(CompactionManager.instance.getRateLimiter());
|
||||
this.indexFile = RandomAccessReader.open(new File(sstable.descriptor.filenameFor(Component.PRIMARY_INDEX)));
|
||||
this.verifyInfo = new VerifyInfo(dataFile, sstable, fileAccessLock.readLock());
|
||||
this.options = options;
|
||||
this.isOffline = isOffline;
|
||||
this.tokenLookup = options.tokenLookup;
|
||||
}
|
||||
|
||||
public void verify()
|
||||
{
|
||||
boolean extended = options.extendedVerification;
|
||||
long rowStart = 0;
|
||||
|
||||
outputHandler.output(String.format("Verifying %s (%s)", sstable, FBUtilities.prettyPrintMemory(dataFile.length())));
|
||||
if (options.checkVersion && !sstable.descriptor.version.isLatestVersion())
|
||||
{
|
||||
String msg = String.format("%s is not the latest version, run upgradesstables", sstable);
|
||||
outputHandler.output(msg);
|
||||
// don't use markAndThrow here because we don't want a CorruptSSTableException for this.
|
||||
throw new RuntimeException(msg);
|
||||
}
|
||||
|
||||
outputHandler.output(String.format("Deserializing sstable metadata for %s ", sstable));
|
||||
try
|
||||
{
|
||||
EnumSet<MetadataType> types = EnumSet.of(MetadataType.VALIDATION, MetadataType.STATS, MetadataType.HEADER);
|
||||
Map<MetadataType, MetadataComponent> sstableMetadata = sstable.descriptor.getMetadataSerializer().deserialize(sstable.descriptor, types);
|
||||
if (sstableMetadata.containsKey(MetadataType.VALIDATION) &&
|
||||
!((ValidationMetadata)sstableMetadata.get(MetadataType.VALIDATION)).partitioner.equals(sstable.getPartitioner().getClass().getCanonicalName()))
|
||||
throw new IOException("Partitioner does not match validation metadata");
|
||||
}
|
||||
catch (Throwable t)
|
||||
{
|
||||
outputHandler.warn(t);
|
||||
markAndThrow(t, false);
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
outputHandler.debug("Deserializing index for "+sstable);
|
||||
deserializeIndex(sstable);
|
||||
}
|
||||
catch (Throwable t)
|
||||
{
|
||||
outputHandler.warn(t);
|
||||
markAndThrow(t);
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
outputHandler.debug("Deserializing index summary for "+sstable);
|
||||
deserializeIndexSummary(sstable);
|
||||
}
|
||||
catch (Throwable t)
|
||||
{
|
||||
outputHandler.output("Index summary is corrupt - if it is removed it will get rebuilt on startup "+sstable.descriptor.filenameFor(Component.SUMMARY));
|
||||
outputHandler.warn(t);
|
||||
markAndThrow(t, false);
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
outputHandler.debug("Deserializing bloom filter for "+sstable);
|
||||
deserializeBloomFilter(sstable);
|
||||
|
||||
}
|
||||
catch (Throwable t)
|
||||
{
|
||||
outputHandler.warn(t);
|
||||
markAndThrow(t);
|
||||
}
|
||||
|
||||
if (options.checkOwnsTokens && !isOffline && !(cfs.getPartitioner() instanceof LocalPartitioner))
|
||||
{
|
||||
outputHandler.debug("Checking that all tokens are owned by the current node");
|
||||
try (KeyIterator iter = new KeyIterator(sstable.descriptor, sstable.metadata()))
|
||||
{
|
||||
List<Range<Token>> ownedRanges = Range.normalize(tokenLookup.apply(cfs.metadata.keyspace));
|
||||
if (ownedRanges.isEmpty())
|
||||
return;
|
||||
RangeOwnHelper rangeOwnHelper = new RangeOwnHelper(ownedRanges);
|
||||
while (iter.hasNext())
|
||||
{
|
||||
DecoratedKey key = iter.next();
|
||||
rangeOwnHelper.validate(key);
|
||||
}
|
||||
}
|
||||
catch (Throwable t)
|
||||
{
|
||||
outputHandler.warn(t);
|
||||
markAndThrow(t);
|
||||
}
|
||||
}
|
||||
|
||||
if (options.quick)
|
||||
return;
|
||||
|
||||
// Verify will use the Digest files, which works for both compressed and uncompressed sstables
|
||||
outputHandler.output(String.format("Checking computed hash of %s ", sstable));
|
||||
try
|
||||
{
|
||||
validator = null;
|
||||
|
||||
if (new File(sstable.descriptor.filenameFor(Component.DIGEST)).exists())
|
||||
{
|
||||
validator = DataIntegrityMetadata.fileDigestValidator(sstable.descriptor);
|
||||
validator.validate();
|
||||
}
|
||||
else
|
||||
{
|
||||
outputHandler.output("Data digest missing, assuming extended verification of disk values");
|
||||
extended = true;
|
||||
}
|
||||
}
|
||||
catch (IOException e)
|
||||
{
|
||||
outputHandler.warn(e);
|
||||
markAndThrow(e);
|
||||
}
|
||||
finally
|
||||
{
|
||||
FileUtils.closeQuietly(validator);
|
||||
}
|
||||
|
||||
if (!extended)
|
||||
return;
|
||||
|
||||
outputHandler.output("Extended Verify requested, proceeding to inspect values");
|
||||
|
||||
try
|
||||
{
|
||||
ByteBuffer nextIndexKey = ByteBufferUtil.readWithShortLength(indexFile);
|
||||
{
|
||||
long firstRowPositionFromIndex = rowIndexEntrySerializer.deserializePositionAndSkip(indexFile);
|
||||
if (firstRowPositionFromIndex != 0)
|
||||
markAndThrow(new RuntimeException("firstRowPositionFromIndex != 0: "+firstRowPositionFromIndex));
|
||||
}
|
||||
|
||||
List<Range<Token>> ownedRanges = isOffline ? Collections.emptyList() : Range.normalize(tokenLookup.apply(cfs.metadata().keyspace));
|
||||
RangeOwnHelper rangeOwnHelper = new RangeOwnHelper(ownedRanges);
|
||||
DecoratedKey prevKey = null;
|
||||
|
||||
while (!dataFile.isEOF())
|
||||
{
|
||||
|
||||
if (verifyInfo.isStopRequested())
|
||||
throw new CompactionInterruptedException(verifyInfo.getCompactionInfo());
|
||||
|
||||
rowStart = dataFile.getFilePointer();
|
||||
outputHandler.debug("Reading row at " + rowStart);
|
||||
|
||||
DecoratedKey key = null;
|
||||
try
|
||||
{
|
||||
key = sstable.decorateKey(ByteBufferUtil.readWithShortLength(dataFile));
|
||||
}
|
||||
catch (Throwable th)
|
||||
{
|
||||
throwIfFatal(th);
|
||||
// check for null key below
|
||||
}
|
||||
|
||||
if (options.checkOwnsTokens && ownedRanges.size() > 0 && !(cfs.getPartitioner() instanceof LocalPartitioner))
|
||||
{
|
||||
try
|
||||
{
|
||||
rangeOwnHelper.validate(key);
|
||||
}
|
||||
catch (Throwable t)
|
||||
{
|
||||
outputHandler.warn(String.format("Key %s in sstable %s not owned by local ranges %s", key, sstable, ownedRanges), t);
|
||||
markAndThrow(t);
|
||||
}
|
||||
}
|
||||
|
||||
ByteBuffer currentIndexKey = nextIndexKey;
|
||||
long nextRowPositionFromIndex = 0;
|
||||
try
|
||||
{
|
||||
nextIndexKey = indexFile.isEOF() ? null : ByteBufferUtil.readWithShortLength(indexFile);
|
||||
nextRowPositionFromIndex = indexFile.isEOF()
|
||||
? dataFile.length()
|
||||
: rowIndexEntrySerializer.deserializePositionAndSkip(indexFile);
|
||||
}
|
||||
catch (Throwable th)
|
||||
{
|
||||
markAndThrow(th);
|
||||
}
|
||||
|
||||
long dataStart = dataFile.getFilePointer();
|
||||
long dataStartFromIndex = currentIndexKey == null
|
||||
? -1
|
||||
: rowStart + 2 + currentIndexKey.remaining();
|
||||
|
||||
long dataSize = nextRowPositionFromIndex - dataStartFromIndex;
|
||||
// avoid an NPE if key is null
|
||||
String keyName = key == null ? "(unreadable key)" : ByteBufferUtil.bytesToHex(key.getKey());
|
||||
outputHandler.debug(String.format("row %s is %s", keyName, FBUtilities.prettyPrintMemory(dataSize)));
|
||||
|
||||
assert currentIndexKey != null || indexFile.isEOF();
|
||||
|
||||
try
|
||||
{
|
||||
if (key == null || dataSize > dataFile.length())
|
||||
markAndThrow(new RuntimeException(String.format("key = %s, dataSize=%d, dataFile.length() = %d", key, dataSize, dataFile.length())));
|
||||
|
||||
try (UnfilteredRowIterator iterator = SSTableIdentityIterator.create(sstable, dataFile, key))
|
||||
{
|
||||
Row first = null;
|
||||
int duplicateRows = 0;
|
||||
long minTimestamp = Long.MAX_VALUE;
|
||||
long maxTimestamp = Long.MIN_VALUE;
|
||||
while (iterator.hasNext())
|
||||
{
|
||||
Unfiltered uf = iterator.next();
|
||||
if (uf.isRow())
|
||||
{
|
||||
Row row = (Row) uf;
|
||||
if (first != null && first.clustering().equals(row.clustering()))
|
||||
{
|
||||
duplicateRows++;
|
||||
for (Cell cell : row.cells())
|
||||
{
|
||||
maxTimestamp = Math.max(cell.timestamp(), maxTimestamp);
|
||||
minTimestamp = Math.min(cell.timestamp(), minTimestamp);
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
if (duplicateRows > 0)
|
||||
logDuplicates(key, first, duplicateRows, minTimestamp, maxTimestamp);
|
||||
duplicateRows = 0;
|
||||
first = row;
|
||||
maxTimestamp = Long.MIN_VALUE;
|
||||
minTimestamp = Long.MAX_VALUE;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (duplicateRows > 0)
|
||||
logDuplicates(key, first, duplicateRows, minTimestamp, maxTimestamp);
|
||||
}
|
||||
|
||||
if ( (prevKey != null && prevKey.compareTo(key) > 0) || !key.getKey().equals(currentIndexKey) || dataStart != dataStartFromIndex )
|
||||
markAndThrow(new RuntimeException("Key out of order: previous = "+prevKey + " : current = " + key));
|
||||
|
||||
goodRows++;
|
||||
prevKey = key;
|
||||
|
||||
|
||||
outputHandler.debug(String.format("Row %s at %s valid, moving to next row at %s ", goodRows, rowStart, nextRowPositionFromIndex));
|
||||
dataFile.seek(nextRowPositionFromIndex);
|
||||
}
|
||||
catch (Throwable th)
|
||||
{
|
||||
markAndThrow(th);
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (Throwable t)
|
||||
{
|
||||
Throwables.throwIfUnchecked(t);
|
||||
throw new RuntimeException(t);
|
||||
}
|
||||
finally
|
||||
{
|
||||
controller.close();
|
||||
}
|
||||
|
||||
outputHandler.output("Verify of " + sstable + " succeeded. All " + goodRows + " rows read successfully");
|
||||
}
|
||||
|
||||
private void logDuplicates(DecoratedKey key, Row first, int duplicateRows, long minTimestamp, long maxTimestamp)
|
||||
{
|
||||
String keyString = sstable.metadata().partitionKeyType.getString(key.getKey());
|
||||
long firstMaxTs = Long.MIN_VALUE;
|
||||
long firstMinTs = Long.MAX_VALUE;
|
||||
for (Cell cell : first.cells())
|
||||
{
|
||||
firstMaxTs = Math.max(firstMaxTs, cell.timestamp());
|
||||
firstMinTs = Math.min(firstMinTs, cell.timestamp());
|
||||
}
|
||||
outputHandler.output(String.format("%d duplicate rows found for [%s %s] in %s.%s (%s), timestamps: [first row (%s, %s)], [duplicates (%s, %s, eq:%b)]",
|
||||
duplicateRows,
|
||||
keyString, first.clustering().toString(sstable.metadata()),
|
||||
sstable.metadata().keyspace,
|
||||
sstable.metadata().name,
|
||||
sstable,
|
||||
dateString(firstMinTs), dateString(firstMaxTs),
|
||||
dateString(minTimestamp), dateString(maxTimestamp), minTimestamp == maxTimestamp));
|
||||
}
|
||||
|
||||
private String dateString(long time)
|
||||
{
|
||||
return Instant.ofEpochMilli(TimeUnit.MICROSECONDS.toMillis(time)).toString();
|
||||
}
|
||||
|
||||
/**
|
||||
* Use the fact that check(..) is called with sorted tokens - we keep a pointer in to the normalized ranges
|
||||
* and only bump the pointer if the key given is out of range. This is done to avoid calling .contains(..) many
|
||||
* times for each key (with vnodes for example)
|
||||
*/
|
||||
@VisibleForTesting
|
||||
public static class RangeOwnHelper
|
||||
{
|
||||
private final List<Range<Token>> normalizedRanges;
|
||||
private int rangeIndex = 0;
|
||||
private DecoratedKey lastKey;
|
||||
|
||||
public RangeOwnHelper(List<Range<Token>> normalizedRanges)
|
||||
{
|
||||
this.normalizedRanges = normalizedRanges;
|
||||
Range.assertNormalized(normalizedRanges);
|
||||
}
|
||||
|
||||
/**
|
||||
* check if the given key is contained in any of the given ranges
|
||||
*
|
||||
* Must be called in sorted order - key should be increasing
|
||||
*
|
||||
* @param key the key
|
||||
* @throws RuntimeException if the key is not contained
|
||||
*/
|
||||
public void validate(DecoratedKey key)
|
||||
{
|
||||
if (!check(key))
|
||||
throw new RuntimeException("Key " + key + " is not contained in the given ranges");
|
||||
}
|
||||
|
||||
/**
|
||||
* check if the given key is contained in any of the given ranges
|
||||
*
|
||||
* Must be called in sorted order - key should be increasing
|
||||
*
|
||||
* @param key the key
|
||||
* @return boolean
|
||||
*/
|
||||
public boolean check(DecoratedKey key)
|
||||
{
|
||||
assert lastKey == null || key.compareTo(lastKey) > 0;
|
||||
lastKey = key;
|
||||
|
||||
if (normalizedRanges.isEmpty()) // handle tests etc where we don't have any ranges
|
||||
return true;
|
||||
|
||||
if (rangeIndex > normalizedRanges.size() - 1)
|
||||
throw new IllegalStateException("RangeOwnHelper can only be used to find the first out-of-range-token");
|
||||
|
||||
while (!normalizedRanges.get(rangeIndex).contains(key.getToken()))
|
||||
{
|
||||
rangeIndex++;
|
||||
if (rangeIndex > normalizedRanges.size() - 1)
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
private void deserializeIndex(SSTableReader sstable) throws IOException
|
||||
{
|
||||
try (RandomAccessReader primaryIndex = RandomAccessReader.open(new File(sstable.descriptor.filenameFor(Component.PRIMARY_INDEX))))
|
||||
{
|
||||
long indexSize = primaryIndex.length();
|
||||
|
||||
while ((primaryIndex.getFilePointer()) != indexSize)
|
||||
{
|
||||
ByteBuffer key = ByteBufferUtil.readWithShortLength(primaryIndex);
|
||||
RowIndexEntry.Serializer.skip(primaryIndex, sstable.descriptor.version);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private void deserializeIndexSummary(SSTableReader sstable) throws IOException
|
||||
{
|
||||
File file = new File(sstable.descriptor.filenameFor(Component.SUMMARY));
|
||||
TableMetadata metadata = cfs.metadata();
|
||||
try (DataInputStream iStream = new DataInputStream(Files.newInputStream(file.toPath())))
|
||||
{
|
||||
try (IndexSummary indexSummary = IndexSummary.serializer.deserialize(iStream,
|
||||
cfs.getPartitioner(),
|
||||
metadata.params.minIndexInterval,
|
||||
metadata.params.maxIndexInterval))
|
||||
{
|
||||
ByteBufferUtil.readWithLength(iStream);
|
||||
ByteBufferUtil.readWithLength(iStream);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private void deserializeBloomFilter(SSTableReader sstable) throws IOException
|
||||
{
|
||||
File bfPath = new File(sstable.descriptor.filenameFor(Component.FILTER));
|
||||
if (bfPath.exists())
|
||||
{
|
||||
try (FileInputStreamPlus stream = bfPath.newInputStream();
|
||||
IFilter bf = BloomFilterSerializer.deserialize(stream, sstable.descriptor.version.hasOldBfFormat()))
|
||||
{
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public void close()
|
||||
{
|
||||
fileAccessLock.writeLock().lock();
|
||||
try
|
||||
{
|
||||
FileUtils.closeQuietly(dataFile);
|
||||
FileUtils.closeQuietly(indexFile);
|
||||
}
|
||||
finally
|
||||
{
|
||||
fileAccessLock.writeLock().unlock();
|
||||
}
|
||||
}
|
||||
|
||||
private void throwIfFatal(Throwable th)
|
||||
{
|
||||
if (th instanceof Error && !(th instanceof AssertionError || th instanceof IOError))
|
||||
throw (Error) th;
|
||||
}
|
||||
|
||||
private void markAndThrow(Throwable cause)
|
||||
{
|
||||
markAndThrow(cause, true);
|
||||
}
|
||||
|
||||
private void markAndThrow(Throwable cause, boolean mutateRepaired)
|
||||
{
|
||||
if (mutateRepaired && options.mutateRepairStatus) // if we are able to mutate repaired flag, an incremental repair should be enough
|
||||
{
|
||||
try
|
||||
{
|
||||
sstable.mutateRepairedAndReload(ActiveRepairService.UNREPAIRED_SSTABLE, sstable.getPendingRepair(), sstable.isTransient());
|
||||
cfs.getTracker().notifySSTableRepairedStatusChanged(Collections.singleton(sstable));
|
||||
}
|
||||
catch(IOException ioe)
|
||||
{
|
||||
outputHandler.output("Error mutating repairedAt for SSTable " + sstable.getFilename() + ", as part of markAndThrow");
|
||||
}
|
||||
}
|
||||
Exception e = new Exception(String.format("Invalid SSTable %s, please force %srepair", sstable.getFilename(), (mutateRepaired && options.mutateRepairStatus) ? "" : "a full "), cause);
|
||||
if (options.invokeDiskFailurePolicy)
|
||||
throw new CorruptSSTableException(e, sstable.getFilename());
|
||||
else
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
|
||||
public CompactionInfo.Holder getVerifyInfo()
|
||||
{
|
||||
return verifyInfo;
|
||||
}
|
||||
|
||||
private static class VerifyInfo extends CompactionInfo.Holder
|
||||
{
|
||||
private final RandomAccessReader dataFile;
|
||||
private final SSTableReader sstable;
|
||||
private final TimeUUID verificationCompactionId;
|
||||
private final Lock fileReadLock;
|
||||
|
||||
public VerifyInfo(RandomAccessReader dataFile, SSTableReader sstable, Lock fileReadLock)
|
||||
{
|
||||
this.dataFile = dataFile;
|
||||
this.sstable = sstable;
|
||||
this.fileReadLock = fileReadLock;
|
||||
verificationCompactionId = nextTimeUUID();
|
||||
}
|
||||
|
||||
public CompactionInfo getCompactionInfo()
|
||||
{
|
||||
fileReadLock.lock();
|
||||
try
|
||||
{
|
||||
return new CompactionInfo(sstable.metadata(),
|
||||
OperationType.VERIFY,
|
||||
dataFile.getFilePointer(),
|
||||
dataFile.length(),
|
||||
verificationCompactionId,
|
||||
ImmutableSet.of(sstable));
|
||||
}
|
||||
catch (Exception e)
|
||||
{
|
||||
throw new RuntimeException();
|
||||
}
|
||||
finally
|
||||
{
|
||||
fileReadLock.unlock();
|
||||
}
|
||||
}
|
||||
|
||||
public boolean isGlobal()
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
private static class VerifyController extends CompactionController
|
||||
{
|
||||
public VerifyController(ColumnFamilyStore cfs)
|
||||
{
|
||||
super(cfs, Integer.MAX_VALUE);
|
||||
}
|
||||
|
||||
@Override
|
||||
public LongPredicate getPurgeEvaluator(DecoratedKey key)
|
||||
{
|
||||
return time -> false;
|
||||
}
|
||||
}
|
||||
|
||||
public static Options.Builder options()
|
||||
{
|
||||
return new Options.Builder();
|
||||
}
|
||||
|
||||
public static class Options
|
||||
{
|
||||
public final boolean invokeDiskFailurePolicy;
|
||||
public final boolean extendedVerification;
|
||||
public final boolean checkVersion;
|
||||
public final boolean mutateRepairStatus;
|
||||
public final boolean checkOwnsTokens;
|
||||
public final boolean quick;
|
||||
public final Function<String, ? extends Collection<Range<Token>>> tokenLookup;
|
||||
|
||||
private Options(boolean invokeDiskFailurePolicy, boolean extendedVerification, boolean checkVersion, boolean mutateRepairStatus, boolean checkOwnsTokens, boolean quick, Function<String, ? extends Collection<Range<Token>>> tokenLookup)
|
||||
{
|
||||
this.invokeDiskFailurePolicy = invokeDiskFailurePolicy;
|
||||
this.extendedVerification = extendedVerification;
|
||||
this.checkVersion = checkVersion;
|
||||
this.mutateRepairStatus = mutateRepairStatus;
|
||||
this.checkOwnsTokens = checkOwnsTokens;
|
||||
this.quick = quick;
|
||||
this.tokenLookup = tokenLookup;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString()
|
||||
{
|
||||
return "Options{" +
|
||||
"invokeDiskFailurePolicy=" + invokeDiskFailurePolicy +
|
||||
", extendedVerification=" + extendedVerification +
|
||||
", checkVersion=" + checkVersion +
|
||||
", mutateRepairStatus=" + mutateRepairStatus +
|
||||
", checkOwnsTokens=" + checkOwnsTokens +
|
||||
", quick=" + quick +
|
||||
'}';
|
||||
}
|
||||
|
||||
public static class Builder
|
||||
{
|
||||
private boolean invokeDiskFailurePolicy = false; // invoking disk failure policy can stop the node if we find a corrupt stable
|
||||
private boolean extendedVerification = false;
|
||||
private boolean checkVersion = false;
|
||||
private boolean mutateRepairStatus = false; // mutating repair status can be dangerous
|
||||
private boolean checkOwnsTokens = false;
|
||||
private boolean quick = false;
|
||||
private Function<String, ? extends Collection<Range<Token>>> tokenLookup = StorageService.instance::getLocalAndPendingRanges;
|
||||
|
||||
public Builder invokeDiskFailurePolicy(boolean param)
|
||||
{
|
||||
this.invokeDiskFailurePolicy = param;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder extendedVerification(boolean param)
|
||||
{
|
||||
this.extendedVerification = param;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder checkVersion(boolean param)
|
||||
{
|
||||
this.checkVersion = param;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder mutateRepairStatus(boolean param)
|
||||
{
|
||||
this.mutateRepairStatus = param;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder checkOwnsTokens(boolean param)
|
||||
{
|
||||
this.checkOwnsTokens = param;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder quick(boolean param)
|
||||
{
|
||||
this.quick = param;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder tokenLookup(Function<String, ? extends Collection<Range<Token>>> tokenLookup)
|
||||
{
|
||||
this.tokenLookup = tokenLookup;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Options build()
|
||||
{
|
||||
return new Options(invokeDiskFailurePolicy, extendedVerification, checkVersion, mutateRepairStatus, checkOwnsTokens, quick, tokenLookup);
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -31,16 +31,19 @@ import org.apache.cassandra.db.DecoratedKey;
|
|||
import org.apache.cassandra.db.Directories;
|
||||
import org.apache.cassandra.db.DiskBoundaries;
|
||||
import org.apache.cassandra.db.PartitionPosition;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.db.SerializationHeader;
|
||||
import org.apache.cassandra.db.compaction.CompactionTask;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.SSTableRewriter;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableWriter;
|
||||
import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.utils.FBUtilities;
|
||||
import org.apache.cassandra.utils.TimeUUID;
|
||||
import org.apache.cassandra.utils.concurrent.Transactional;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
|
||||
|
||||
/**
|
||||
|
|
@ -246,4 +249,24 @@ public abstract class CompactionAwareWriter extends Transactional.AbstractTransa
|
|||
{
|
||||
return cfs.getExpectedCompactedFileSize(nonExpiredSSTables, txn.opType());
|
||||
}
|
||||
|
||||
/**
|
||||
* It is up to the caller to set the following fields:
|
||||
* - {@link SSTableWriter.Builder#setKeyCount(long)},
|
||||
* - {@link SSTableWriter.Builder#setSerializationHeader(SerializationHeader)} and,
|
||||
* - {@link SSTableWriter.Builder#setMetadataCollector(MetadataCollector)}
|
||||
*
|
||||
* @param descriptor
|
||||
* @return
|
||||
*/
|
||||
protected SSTableWriter.Builder<?, ?> newWriterBuilder(Descriptor descriptor)
|
||||
{
|
||||
return descriptor.getFormat().getWriterFactory().builder(descriptor)
|
||||
.setTableMetadataRef(cfs.metadata)
|
||||
.setTransientSSTable(isTransient)
|
||||
.setRepairedAt(minRepairedAt)
|
||||
.setPendingRepair(pendingRepair)
|
||||
.addFlushObserversForSecondaryIndexes(cfs.indexManager.listIndexes(), txn.opType())
|
||||
.addDefaultComponents();
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -26,8 +26,9 @@ import org.slf4j.LoggerFactory;
|
|||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.Directories;
|
||||
import org.apache.cassandra.db.SerializationHeader;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableWriter;
|
||||
import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
|
||||
|
|
@ -69,17 +70,15 @@ public class DefaultCompactionWriter extends CompactionAwareWriter
|
|||
{
|
||||
sstableDirectory = directory;
|
||||
|
||||
Descriptor descriptor = cfs.newSSTableDescriptor(getDirectories().getLocationForDisk(directory));
|
||||
MetadataCollector collector = new MetadataCollector(txn.originals(), cfs.metadata().comparator, sstableLevel);
|
||||
SerializationHeader header = SerializationHeader.make(cfs.metadata(), nonExpiredSSTables);
|
||||
|
||||
@SuppressWarnings("resource")
|
||||
SSTableWriter writer = SSTableWriter.create(cfs.newSSTableDescriptor(getDirectories().getLocationForDisk(directory)),
|
||||
estimatedTotalKeys,
|
||||
minRepairedAt,
|
||||
pendingRepair,
|
||||
isTransient,
|
||||
cfs.metadata,
|
||||
new MetadataCollector(txn.originals(), cfs.metadata().comparator, sstableLevel),
|
||||
SerializationHeader.make(cfs.metadata(), nonExpiredSSTables),
|
||||
cfs.indexManager.listIndexes(),
|
||||
txn);
|
||||
SSTableWriter writer = newWriterBuilder(descriptor).setMetadataCollector(collector)
|
||||
.setSerializationHeader(header)
|
||||
.setKeyCount(estimatedTotalKeys)
|
||||
.build(txn, cfs);
|
||||
sstableWriter.switchWriter(writer);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -21,11 +21,12 @@ import java.util.Set;
|
|||
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.Directories;
|
||||
import org.apache.cassandra.db.RowIndexEntry;
|
||||
import org.apache.cassandra.db.SerializationHeader;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.db.compaction.LeveledManifest;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.io.sstable.AbstractRowIndexEntry;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableWriter;
|
||||
import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
|
||||
|
|
@ -69,7 +70,7 @@ public class MajorLeveledCompactionWriter extends CompactionAwareWriter
|
|||
@SuppressWarnings("resource")
|
||||
public boolean realAppend(UnfilteredRowIterator partition)
|
||||
{
|
||||
RowIndexEntry rie = sstableWriter.append(partition);
|
||||
AbstractRowIndexEntry rie = sstableWriter.append(partition);
|
||||
partitionsWritten++;
|
||||
long totalWrittenInCurrentWriter = sstableWriter.currentWriter().getEstimatedOnDiskBytesWritten();
|
||||
if (totalWrittenInCurrentWriter > maxSSTableSize)
|
||||
|
|
@ -91,16 +92,18 @@ public class MajorLeveledCompactionWriter extends CompactionAwareWriter
|
|||
{
|
||||
sstableDirectory = location;
|
||||
averageEstimatedKeysPerSSTable = Math.round(((double) averageEstimatedKeysPerSSTable * sstablesWritten + partitionsWritten) / (sstablesWritten + 1));
|
||||
sstableWriter.switchWriter(SSTableWriter.create(cfs.newSSTableDescriptor(getDirectories().getLocationForDisk(sstableDirectory)),
|
||||
keysPerSSTable,
|
||||
minRepairedAt,
|
||||
pendingRepair,
|
||||
isTransient,
|
||||
cfs.metadata,
|
||||
new MetadataCollector(txn.originals(), cfs.metadata().comparator, currentLevel),
|
||||
SerializationHeader.make(cfs.metadata(), txn.originals()),
|
||||
cfs.indexManager.listIndexes(),
|
||||
txn));
|
||||
|
||||
Descriptor descriptor = cfs.newSSTableDescriptor(getDirectories().getLocationForDisk(sstableDirectory));
|
||||
MetadataCollector collector = new MetadataCollector(txn.originals(), cfs.metadata().comparator, currentLevel);
|
||||
SerializationHeader serializationHeader = SerializationHeader.make(cfs.metadata(), txn.originals());
|
||||
|
||||
@SuppressWarnings("resource")
|
||||
SSTableWriter writer = newWriterBuilder(descriptor).setKeyCount(keysPerSSTable)
|
||||
.setSerializationHeader(serializationHeader)
|
||||
.setMetadataCollector(collector)
|
||||
.build(txn, cfs);
|
||||
|
||||
sstableWriter.switchWriter(writer);
|
||||
partitionsWritten = 0;
|
||||
sstablesWritten = 0;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -21,11 +21,12 @@ import java.util.Set;
|
|||
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.Directories;
|
||||
import org.apache.cassandra.db.RowIndexEntry;
|
||||
import org.apache.cassandra.db.SerializationHeader;
|
||||
import org.apache.cassandra.db.compaction.OperationType;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.io.sstable.AbstractRowIndexEntry;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableWriter;
|
||||
import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
|
||||
|
|
@ -80,7 +81,7 @@ public class MaxSSTableSizeWriter extends CompactionAwareWriter
|
|||
|
||||
protected boolean realAppend(UnfilteredRowIterator partition)
|
||||
{
|
||||
RowIndexEntry rie = sstableWriter.append(partition);
|
||||
AbstractRowIndexEntry rie = sstableWriter.append(partition);
|
||||
if (sstableWriter.currentWriter().getEstimatedOnDiskBytesWritten() > maxSSTableSize)
|
||||
{
|
||||
switchCompactionLocation(sstableDirectory);
|
||||
|
|
@ -92,18 +93,16 @@ public class MaxSSTableSizeWriter extends CompactionAwareWriter
|
|||
public void switchCompactionLocation(Directories.DataDirectory location)
|
||||
{
|
||||
sstableDirectory = location;
|
||||
@SuppressWarnings("resource")
|
||||
SSTableWriter writer = SSTableWriter.create(cfs.newSSTableDescriptor(getDirectories().getLocationForDisk(sstableDirectory)),
|
||||
estimatedTotalKeys / estimatedSSTables,
|
||||
minRepairedAt,
|
||||
pendingRepair,
|
||||
isTransient,
|
||||
cfs.metadata,
|
||||
new MetadataCollector(allSSTables, cfs.metadata().comparator, level),
|
||||
SerializationHeader.make(cfs.metadata(), nonExpiredSSTables),
|
||||
cfs.indexManager.listIndexes(),
|
||||
txn);
|
||||
|
||||
Descriptor descriptor = cfs.newSSTableDescriptor(getDirectories().getLocationForDisk(sstableDirectory));
|
||||
MetadataCollector collector = new MetadataCollector(allSSTables, cfs.metadata().comparator, level);
|
||||
SerializationHeader header = SerializationHeader.make(cfs.metadata(), nonExpiredSSTables);
|
||||
|
||||
@SuppressWarnings("resource")
|
||||
SSTableWriter writer = newWriterBuilder(descriptor).setKeyCount(estimatedTotalKeys / estimatedSSTables)
|
||||
.setMetadataCollector(collector)
|
||||
.setSerializationHeader(header)
|
||||
.build(txn, cfs);
|
||||
sstableWriter.switchWriter(writer);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -25,10 +25,11 @@ import org.slf4j.LoggerFactory;
|
|||
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.Directories;
|
||||
import org.apache.cassandra.db.RowIndexEntry;
|
||||
import org.apache.cassandra.db.SerializationHeader;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.io.sstable.AbstractRowIndexEntry;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableWriter;
|
||||
import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
|
||||
|
|
@ -85,7 +86,7 @@ public class SplittingSizeTieredCompactionWriter extends CompactionAwareWriter
|
|||
@Override
|
||||
public boolean realAppend(UnfilteredRowIterator partition)
|
||||
{
|
||||
RowIndexEntry rie = sstableWriter.append(partition);
|
||||
AbstractRowIndexEntry rie = sstableWriter.append(partition);
|
||||
if (sstableWriter.currentWriter().getEstimatedOnDiskBytesWritten() > currentBytesToWrite && currentRatioIndex < ratios.length - 1) // if we underestimate how many keys we have, the last sstable might get more than we expect
|
||||
{
|
||||
currentRatioIndex++;
|
||||
|
|
@ -102,17 +103,15 @@ public class SplittingSizeTieredCompactionWriter extends CompactionAwareWriter
|
|||
{
|
||||
sstableDirectory = location;
|
||||
long currentPartitionsToWrite = Math.round(ratios[currentRatioIndex] * estimatedTotalKeys);
|
||||
Descriptor descriptor = cfs.newSSTableDescriptor(getDirectories().getLocationForDisk(location));
|
||||
MetadataCollector collector = new MetadataCollector(allSSTables, cfs.metadata().comparator, 0);
|
||||
SerializationHeader header = SerializationHeader.make(cfs.metadata(), nonExpiredSSTables);
|
||||
|
||||
@SuppressWarnings("resource")
|
||||
SSTableWriter writer = SSTableWriter.create(cfs.newSSTableDescriptor(getDirectories().getLocationForDisk(location)),
|
||||
currentPartitionsToWrite,
|
||||
minRepairedAt,
|
||||
pendingRepair,
|
||||
isTransient,
|
||||
cfs.metadata,
|
||||
new MetadataCollector(allSSTables, cfs.metadata().comparator, 0),
|
||||
SerializationHeader.make(cfs.metadata(), nonExpiredSSTables),
|
||||
cfs.indexManager.listIndexes(),
|
||||
txn);
|
||||
SSTableWriter writer = newWriterBuilder(descriptor).setKeyCount(currentPartitionsToWrite)
|
||||
.setMetadataCollector(collector)
|
||||
.setSerializationHeader(header)
|
||||
.build(txn, cfs);
|
||||
logger.trace("Switching writer, currentPartitionsToWrite = {}", currentPartitionsToWrite);
|
||||
sstableWriter.switchWriter(writer);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,16 +17,27 @@
|
|||
*/
|
||||
package org.apache.cassandra.db.lifecycle;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
|
||||
import com.google.common.base.Predicate;
|
||||
import com.google.common.collect.*;
|
||||
import com.google.common.collect.ImmutableMap;
|
||||
import com.google.common.collect.ImmutableSet;
|
||||
import com.google.common.collect.Iterables;
|
||||
|
||||
import org.apache.cassandra.io.sstable.SSTable;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.utils.Throwables;
|
||||
|
||||
import static com.google.common.base.Predicates.*;
|
||||
import static com.google.common.base.Predicates.and;
|
||||
import static com.google.common.base.Predicates.equalTo;
|
||||
import static com.google.common.base.Predicates.in;
|
||||
import static com.google.common.base.Predicates.not;
|
||||
import static com.google.common.base.Predicates.or;
|
||||
import static com.google.common.collect.Iterables.any;
|
||||
import static com.google.common.collect.Iterables.concat;
|
||||
import static com.google.common.collect.Iterables.filter;
|
||||
|
|
@ -50,7 +61,7 @@ class Helpers
|
|||
* really present, and that the items to add are not (unless we're also removing them)
|
||||
* @return a new identity map with the contents of the provided one modified
|
||||
*/
|
||||
static <T> Map<T, T> replace(Map<T, T> original, Set<T> remove, Iterable<T> add)
|
||||
static <T> Map<T, T> replace(Map<T, T> original, Set<? extends T> remove, Iterable<? extends T> add)
|
||||
{
|
||||
// ensure the ones being removed are the exact same ones present
|
||||
for (T reader : remove)
|
||||
|
|
|
|||
|
|
@ -18,33 +18,54 @@
|
|||
package org.apache.cassandra.db.lifecycle;
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.Collections;
|
||||
import java.util.HashSet;
|
||||
import java.util.IdentityHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
import java.util.function.BiPredicate;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import com.google.common.base.Predicate;
|
||||
import com.google.common.collect.*;
|
||||
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import com.google.common.collect.ImmutableList;
|
||||
import com.google.common.collect.Iterables;
|
||||
import com.google.common.collect.Lists;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.Directories;
|
||||
import org.apache.cassandra.db.compaction.OperationType;
|
||||
import org.apache.cassandra.io.sstable.SSTable;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader.UniqueIdentifier;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.utils.Throwables;
|
||||
import org.apache.cassandra.utils.TimeUUID;
|
||||
import org.apache.cassandra.utils.concurrent.Transactional;
|
||||
|
||||
import static com.google.common.base.Functions.compose;
|
||||
import static com.google.common.base.Predicates.*;
|
||||
import static com.google.common.base.Predicates.in;
|
||||
import static com.google.common.collect.ImmutableSet.copyOf;
|
||||
import static com.google.common.collect.Iterables.*;
|
||||
import static com.google.common.collect.Iterables.concat;
|
||||
import static com.google.common.collect.Iterables.getFirst;
|
||||
import static com.google.common.collect.Iterables.transform;
|
||||
import static java.util.Collections.singleton;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.*;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.abortObsoletion;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.checkNotReplaced;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.concatUniq;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.emptySet;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.filterIn;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.filterOut;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.markObsolete;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.orIn;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.prepareForObsoletion;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.select;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.selectFirst;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.setReplaced;
|
||||
import static org.apache.cassandra.db.lifecycle.View.updateCompacting;
|
||||
import static org.apache.cassandra.db.lifecycle.View.updateLiveSet;
|
||||
import static org.apache.cassandra.utils.Throwables.maybeFail;
|
||||
|
|
@ -160,12 +181,12 @@ public class LifecycleTransaction extends Transactional.AbstractTransactional im
|
|||
}
|
||||
|
||||
@SuppressWarnings("resource") // log closed during postCleanup
|
||||
LifecycleTransaction(Tracker tracker, OperationType operationType, Iterable<SSTableReader> readers)
|
||||
LifecycleTransaction(Tracker tracker, OperationType operationType, Iterable<? extends SSTableReader> readers)
|
||||
{
|
||||
this(tracker, new LogTransaction(operationType, tracker), readers);
|
||||
}
|
||||
|
||||
LifecycleTransaction(Tracker tracker, LogTransaction log, Iterable<SSTableReader> readers)
|
||||
LifecycleTransaction(Tracker tracker, LogTransaction log, Iterable<? extends SSTableReader> readers)
|
||||
{
|
||||
this.tracker = tracker;
|
||||
this.log = log;
|
||||
|
|
@ -456,7 +477,7 @@ public class LifecycleTransaction extends Transactional.AbstractTransactional im
|
|||
private List<SSTableReader> restoreUpdatedOriginals()
|
||||
{
|
||||
Iterable<SSTableReader> torestore = filterIn(originals, logged.update, logged.obsolete);
|
||||
return ImmutableList.copyOf(transform(torestore, (reader) -> current(reader).cloneWithRestoredStart(reader.first)));
|
||||
return ImmutableList.copyOf(transform(torestore, (reader) -> current(reader).cloneWithRestoredStart(reader.getFirst())));
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -21,15 +21,22 @@
|
|||
package org.apache.cassandra.db.lifecycle;
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.util.*;
|
||||
import java.util.Collections;
|
||||
import java.util.Comparator;
|
||||
import java.util.HashMap;
|
||||
import java.util.HashSet;
|
||||
import java.util.LinkedHashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.NavigableSet;
|
||||
import java.util.Optional;
|
||||
import java.util.Set;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import com.google.common.collect.Iterables;
|
||||
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
|
@ -37,8 +44,9 @@ import org.slf4j.LoggerFactory;
|
|||
import org.apache.cassandra.db.compaction.OperationType;
|
||||
import org.apache.cassandra.db.lifecycle.LogRecord.Type;
|
||||
import org.apache.cassandra.io.sstable.SSTable;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.big.BigFormat;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.utils.Throwables;
|
||||
import org.apache.cassandra.utils.TimeUUID;
|
||||
|
||||
|
|
@ -62,7 +70,7 @@ final class LogFile implements AutoCloseable
|
|||
static String EXT = ".log";
|
||||
static char SEP = '_';
|
||||
// cc_txn_opname_id.log (where cc is one of the sstable versions defined in BigVersion)
|
||||
static Pattern FILE_REGEX = Pattern.compile(String.format("^(.{2})_txn_(.*)_(.*)%s$", EXT));
|
||||
static Pattern FILE_REGEX = Pattern.compile(String.format("^(.{2}_)?txn_(.*)_(.*)%s$", EXT));
|
||||
|
||||
// A set of physical files on disk, each file is an identical replica
|
||||
private final LogReplicaSet replicas = new LogReplicaSet();
|
||||
|
|
@ -496,18 +504,14 @@ final class LogFile implements AutoCloseable
|
|||
|
||||
private String getFileName()
|
||||
{
|
||||
return StringUtils.join(BigFormat.latestVersion,
|
||||
LogFile.SEP,
|
||||
"txn",
|
||||
LogFile.SEP,
|
||||
type.fileName,
|
||||
LogFile.SEP,
|
||||
id.toString(),
|
||||
LogFile.EXT);
|
||||
return StringUtils.join(SSTableFormat.Type.current().info.getLatestVersion(), LogFile.SEP, // remove version and separator when downgrading to 4.x is becomes unsupported
|
||||
"txn", LogFile.SEP,
|
||||
type.fileName, LogFile.SEP,
|
||||
id.toString(), LogFile.EXT);
|
||||
}
|
||||
|
||||
public boolean isEmpty()
|
||||
{
|
||||
return records.isEmpty();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -22,7 +22,16 @@ package org.apache.cassandra.db.lifecycle;
|
|||
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.Collections;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import java.util.Set;
|
||||
import java.util.TreeSet;
|
||||
import java.util.function.BiPredicate;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
|
|
@ -152,7 +161,7 @@ final class LogRecord
|
|||
|
||||
public static LogRecord make(Type type, SSTable table)
|
||||
{
|
||||
String absoluteTablePath = absolutePath(table.descriptor.baseFilename());
|
||||
String absoluteTablePath = absolutePath(table.descriptor.baseFile());
|
||||
return make(type, getExistingFiles(absoluteTablePath), table.getAllFilePaths().size(), absoluteTablePath);
|
||||
}
|
||||
|
||||
|
|
@ -161,7 +170,7 @@ final class LogRecord
|
|||
// contains a mapping from sstable absolute path (everything up until the 'Data'/'Index'/etc part of the filename) to the sstable
|
||||
Map<String, SSTable> absolutePaths = new HashMap<>();
|
||||
for (SSTableReader table : tables)
|
||||
absolutePaths.put(absolutePath(table.descriptor.baseFilename()), table);
|
||||
absolutePaths.put(absolutePath(table.descriptor.baseFile()), table);
|
||||
|
||||
// maps sstable base file name to the actual files on disk
|
||||
Map<String, List<File>> existingFiles = getExistingFiles(absolutePaths.keySet());
|
||||
|
|
@ -176,9 +185,9 @@ final class LogRecord
|
|||
return records;
|
||||
}
|
||||
|
||||
private static String absolutePath(String baseFilename)
|
||||
private static String absolutePath(File baseFile)
|
||||
{
|
||||
return FileUtils.getCanonicalPath(baseFilename + Component.separator);
|
||||
return baseFile.withSuffix(String.valueOf(Component.separator)).canonicalPath();
|
||||
}
|
||||
|
||||
public LogRecord withExistingFiles(List<File> existingFiles)
|
||||
|
|
|
|||
|
|
@ -22,18 +22,22 @@ import java.io.IOException;
|
|||
import java.io.PrintStream;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.NoSuchFileException;
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Queue;
|
||||
import java.util.concurrent.ConcurrentLinkedQueue;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.function.Predicate;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import com.google.common.util.concurrent.Runnables;
|
||||
|
||||
import com.codahale.metrics.Counter;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import com.codahale.metrics.Counter;
|
||||
import org.apache.cassandra.concurrent.ScheduledExecutors;
|
||||
import org.apache.cassandra.config.DatabaseDescriptor;
|
||||
import org.apache.cassandra.db.Directories;
|
||||
|
|
@ -41,15 +45,17 @@ import org.apache.cassandra.db.SystemKeyspace;
|
|||
import org.apache.cassandra.db.compaction.OperationType;
|
||||
import org.apache.cassandra.db.lifecycle.LogRecord.Type;
|
||||
import org.apache.cassandra.io.FSWriteError;
|
||||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.SSTable;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat.Components;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.io.util.FileUtils;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.service.StorageService;
|
||||
import org.apache.cassandra.utils.*;
|
||||
import org.apache.cassandra.utils.FBUtilities;
|
||||
import org.apache.cassandra.utils.Throwables;
|
||||
import org.apache.cassandra.utils.TimeUUID;
|
||||
import org.apache.cassandra.utils.concurrent.Ref;
|
||||
import org.apache.cassandra.utils.concurrent.RefCounted;
|
||||
import org.apache.cassandra.utils.concurrent.Transactional;
|
||||
|
|
@ -387,18 +393,14 @@ class LogTransaction extends Transactional.AbstractTransactional implements Tran
|
|||
try
|
||||
{
|
||||
// If we can't successfully delete the DATA component, set the task to be retried later: see TransactionTidier
|
||||
File datafile = new File(desc.filenameFor(Component.DATA));
|
||||
|
||||
if (logger.isTraceEnabled())
|
||||
logger.trace("Tidier running for old sstable {}", desc.baseFilename());
|
||||
logger.trace("Tidier running for old sstable {}", desc);
|
||||
|
||||
if (datafile.exists())
|
||||
delete(datafile);
|
||||
else if (!wasNew)
|
||||
if (!desc.fileFor(Components.DATA).exists() && !wasNew)
|
||||
logger.error("SSTableTidier ran with no existing data file for an sstable that was not new");
|
||||
|
||||
// let the remainder be cleaned up by delete
|
||||
SSTable.delete(desc, SSTable.discoverComponentsFor(desc));
|
||||
desc.getFormat().delete(desc);
|
||||
}
|
||||
catch (Throwable t)
|
||||
{
|
||||
|
|
|
|||
|
|
@ -17,7 +17,11 @@
|
|||
*/
|
||||
package org.apache.cassandra.db.lifecycle;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.CopyOnWriteArrayList;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
|
||||
|
|
@ -25,23 +29,34 @@ import com.google.common.annotations.VisibleForTesting;
|
|||
import com.google.common.base.Function;
|
||||
import com.google.common.base.Predicate;
|
||||
import com.google.common.base.Predicates;
|
||||
import com.google.common.collect.*;
|
||||
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.Directories;
|
||||
import org.apache.cassandra.db.memtable.Memtable;
|
||||
import org.apache.cassandra.db.commitlog.CommitLogPosition;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import com.google.common.collect.Iterables;
|
||||
import com.google.common.collect.Sets;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.apache.cassandra.config.DatabaseDescriptor;
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.Directories;
|
||||
import org.apache.cassandra.db.commitlog.CommitLogPosition;
|
||||
import org.apache.cassandra.db.compaction.OperationType;
|
||||
import org.apache.cassandra.db.memtable.Memtable;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.metadata.StatsMetadata;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.io.util.FileUtils;
|
||||
import org.apache.cassandra.metrics.StorageMetrics;
|
||||
import org.apache.cassandra.notifications.*;
|
||||
import org.apache.cassandra.notifications.INotification;
|
||||
import org.apache.cassandra.notifications.INotificationConsumer;
|
||||
import org.apache.cassandra.notifications.InitialSSTableAddedNotification;
|
||||
import org.apache.cassandra.notifications.MemtableDiscardedNotification;
|
||||
import org.apache.cassandra.notifications.MemtableRenewedNotification;
|
||||
import org.apache.cassandra.notifications.MemtableSwitchedNotification;
|
||||
import org.apache.cassandra.notifications.SSTableAddedNotification;
|
||||
import org.apache.cassandra.notifications.SSTableDeletingNotification;
|
||||
import org.apache.cassandra.notifications.SSTableListChangedNotification;
|
||||
import org.apache.cassandra.notifications.SSTableMetadataChanged;
|
||||
import org.apache.cassandra.notifications.SSTableRepairStatusChanged;
|
||||
import org.apache.cassandra.notifications.TruncationNotification;
|
||||
import org.apache.cassandra.utils.Pair;
|
||||
import org.apache.cassandra.utils.Throwables;
|
||||
import org.apache.cassandra.utils.concurrent.OpOrder;
|
||||
|
|
@ -51,7 +66,11 @@ import static com.google.common.collect.ImmutableSet.copyOf;
|
|||
import static com.google.common.collect.Iterables.filter;
|
||||
import static java.util.Collections.singleton;
|
||||
import static java.util.Collections.singletonList;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.*;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.abortObsoletion;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.markObsolete;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.notIn;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.prepareForObsoletion;
|
||||
import static org.apache.cassandra.db.lifecycle.Helpers.setupOnline;
|
||||
import static org.apache.cassandra.db.lifecycle.View.permitCompacting;
|
||||
import static org.apache.cassandra.db.lifecycle.View.updateCompacting;
|
||||
import static org.apache.cassandra.db.lifecycle.View.updateLiveSet;
|
||||
|
|
@ -99,7 +118,7 @@ public class Tracker
|
|||
/**
|
||||
* @return a Transaction over the provided sstables if we are able to mark the given @param sstables as compacted, before anyone else
|
||||
*/
|
||||
public LifecycleTransaction tryModify(Iterable<SSTableReader> sstables, OperationType operationType)
|
||||
public LifecycleTransaction tryModify(Iterable<? extends SSTableReader> sstables, OperationType operationType)
|
||||
{
|
||||
if (Iterables.isEmpty(sstables))
|
||||
return new LifecycleTransaction(this, operationType, sstables);
|
||||
|
|
|
|||
|
|
@ -17,17 +17,22 @@
|
|||
*/
|
||||
package org.apache.cassandra.db.lifecycle;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.Collections;
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import com.google.common.base.Function;
|
||||
import com.google.common.base.Functions;
|
||||
import com.google.common.base.Predicate;
|
||||
import com.google.common.collect.*;
|
||||
import com.google.common.collect.ImmutableList;
|
||||
import com.google.common.collect.Iterables;
|
||||
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.memtable.Memtable;
|
||||
import org.apache.cassandra.db.PartitionPosition;
|
||||
import org.apache.cassandra.db.memtable.Memtable;
|
||||
import org.apache.cassandra.dht.AbstractBounds;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.utils.Interval;
|
||||
|
|
@ -136,13 +141,30 @@ public class View
|
|||
case NONCOMPACTING:
|
||||
return filter(sstables, (s) -> !compacting.contains(s));
|
||||
case CANONICAL:
|
||||
// When early open is not in play, the LIVE and CANONICAL sets are the same.
|
||||
// However, when we do have early-open sstables, we will have some unfinished sources in the live set.
|
||||
// For these sources we need to extract the originals, in their non-moved-start versions, from the
|
||||
// compacting set.
|
||||
// This creates a problem when the compaction completes, as then both:
|
||||
// - the source is in the compacting set
|
||||
// - the result is in the live set
|
||||
// This currently causes the CANONICAL set to return both source and result when early-open is disabled,
|
||||
// and is otherwise worked around by opening early the last sstable in the result set (which pushes it
|
||||
// in the compacting set with EARLY openReason) and the !compacting.contains(sstable) check in the
|
||||
// second loop below.
|
||||
// Unfortunately there does not appear to be a way to avoid this workaround. Filtering the compacting
|
||||
// set through having an early-open version in live does not work because sources are fully removed from
|
||||
// the live set when they are completely exhausted.
|
||||
|
||||
// Add the compacting versions first because they will be the canonical versions of compaction sources.
|
||||
Set<SSTableReader> canonicalSSTables = new HashSet<>(sstables.size() + compacting.size());
|
||||
for (SSTableReader sstable : compacting)
|
||||
if (sstable.openReason != SSTableReader.OpenReason.EARLY)
|
||||
canonicalSSTables.add(sstable);
|
||||
// reason for checking if compacting contains the sstable is that if compacting has an EARLY version
|
||||
// of a NORMAL sstable, we still have the canonical version of that sstable in sstables.
|
||||
// note that the EARLY version is equal, but not == since it is a different instance of the same sstable.
|
||||
// Add anything that is not compacting, removing any compaction result where we still have the
|
||||
// compaction sources.
|
||||
// note that the EARLY version is equal to the original, i.e. the set itself can guarantee early-open
|
||||
// versions of sstables in compacting won't be added, but we also want to remove the results.
|
||||
for (SSTableReader sstable : sstables)
|
||||
if (!compacting.contains(sstable) && sstable.openReason != SSTableReader.OpenReason.EARLY)
|
||||
canonicalSSTables.add(sstable);
|
||||
|
|
@ -241,7 +263,7 @@ public class View
|
|||
// METHODS TO CONSTRUCT FUNCTIONS FOR MODIFYING A VIEW:
|
||||
|
||||
// return a function to un/mark the provided readers compacting in a view
|
||||
static Function<View, View> updateCompacting(final Set<SSTableReader> unmark, final Iterable<SSTableReader> mark)
|
||||
static Function<View, View> updateCompacting(final Set<? extends SSTableReader> unmark, final Iterable<? extends SSTableReader> mark)
|
||||
{
|
||||
if (unmark.isEmpty() && Iterables.isEmpty(mark))
|
||||
return Functions.identity();
|
||||
|
|
@ -259,7 +281,7 @@ public class View
|
|||
|
||||
// construct a predicate to reject views that do not permit us to mark these readers compacting;
|
||||
// i.e. one of them is either already compacting, has been compacted, or has been replaced
|
||||
static Predicate<View> permitCompacting(final Iterable<SSTableReader> readers)
|
||||
static Predicate<View> permitCompacting(final Iterable<? extends SSTableReader> readers)
|
||||
{
|
||||
return new Predicate<View>()
|
||||
{
|
||||
|
|
@ -353,4 +375,4 @@ public class View
|
|||
}
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -52,7 +52,7 @@ import org.apache.cassandra.dht.Bounds;
|
|||
import org.apache.cassandra.dht.IncludingExcludingBounds;
|
||||
import org.apache.cassandra.dht.Range;
|
||||
import org.apache.cassandra.index.transactions.UpdateTransaction;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReadsListener;
|
||||
import org.apache.cassandra.io.sstable.SSTableReadsListener;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.schema.TableMetadataRef;
|
||||
import org.apache.cassandra.utils.concurrent.OpOrder;
|
||||
|
|
|
|||
|
|
@ -51,7 +51,7 @@ import org.apache.cassandra.dht.IncludingExcludingBounds;
|
|||
import org.apache.cassandra.dht.Murmur3Partitioner.LongToken;
|
||||
import org.apache.cassandra.dht.Range;
|
||||
import org.apache.cassandra.index.transactions.UpdateTransaction;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReadsListener;
|
||||
import org.apache.cassandra.io.sstable.SSTableReadsListener;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.schema.TableMetadataRef;
|
||||
import org.apache.cassandra.utils.ObjectSizes;
|
||||
|
|
|
|||
|
|
@ -63,7 +63,7 @@ import org.apache.cassandra.dht.IncludingExcludingBounds;
|
|||
import org.apache.cassandra.dht.Range;
|
||||
import org.apache.cassandra.index.transactions.UpdateTransaction;
|
||||
import org.apache.cassandra.io.compress.BufferType;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReadsListener;
|
||||
import org.apache.cassandra.io.sstable.SSTableReadsListener;
|
||||
import org.apache.cassandra.metrics.TableMetrics;
|
||||
import org.apache.cassandra.metrics.TrieMemtableMetricsView;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
|
|
|
|||
|
|
@ -21,13 +21,13 @@ import java.util.Arrays;
|
|||
import java.util.Collections;
|
||||
import java.util.Iterator;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
|
||||
import org.apache.cassandra.db.DeletionTime;
|
||||
import org.apache.cassandra.db.partitions.UnfilteredPartitionIterator;
|
||||
import org.apache.cassandra.utils.AbstractIterator;
|
||||
import org.apache.cassandra.utils.CloseableIterator;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
|
||||
/**
|
||||
* A utility class to split the given {@link#UnfilteredRowIterator} into smaller chunks each
|
||||
* having at most {@link #throttle} + 1 unfiltereds.
|
||||
|
|
@ -83,7 +83,7 @@ public class ThrottledUnfilteredIterator extends AbstractIterator<UnfilteredRowI
|
|||
return throttledItr = origin;
|
||||
}
|
||||
|
||||
throttledItr = new WrappingUnfilteredRowIterator(origin)
|
||||
throttledItr = new WrappingUnfilteredRowIterator()
|
||||
{
|
||||
private int count = 0;
|
||||
private boolean isFirst = throttledItr == null;
|
||||
|
|
@ -96,10 +96,16 @@ public class ThrottledUnfilteredIterator extends AbstractIterator<UnfilteredRowI
|
|||
// it must be consumed as last element of current batch
|
||||
private RangeTombstoneMarker closeMarker = null;
|
||||
|
||||
@Override
|
||||
public UnfilteredRowIterator wrapped()
|
||||
{
|
||||
return origin;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean hasNext()
|
||||
{
|
||||
return (withinLimit() && wrapped.hasNext()) || closeMarker != null;
|
||||
return (withinLimit() && origin.hasNext()) || closeMarker != null;
|
||||
}
|
||||
|
||||
@Override
|
||||
|
|
@ -119,7 +125,7 @@ public class ThrottledUnfilteredIterator extends AbstractIterator<UnfilteredRowI
|
|||
if (overflowed.hasNext())
|
||||
next = overflowed.next();
|
||||
else
|
||||
next = wrapped.next();
|
||||
next = origin.next();
|
||||
recordNext(next);
|
||||
return next;
|
||||
}
|
||||
|
|
@ -132,8 +138,8 @@ public class ThrottledUnfilteredIterator extends AbstractIterator<UnfilteredRowI
|
|||
// when reach throttle with a remaining openMarker, we need to create corresponding closeMarker.
|
||||
if (count == throttle && openMarker != null)
|
||||
{
|
||||
assert wrapped.hasNext();
|
||||
closeOpenMarker(wrapped.next());
|
||||
assert origin.hasNext();
|
||||
closeOpenMarker(origin.next());
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -191,13 +197,13 @@ public class ThrottledUnfilteredIterator extends AbstractIterator<UnfilteredRowI
|
|||
@Override
|
||||
public DeletionTime partitionLevelDeletion()
|
||||
{
|
||||
return isFirst ? wrapped.partitionLevelDeletion() : DeletionTime.LIVE;
|
||||
return isFirst ? origin.partitionLevelDeletion() : DeletionTime.LIVE;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Row staticRow()
|
||||
{
|
||||
return isFirst ? wrapped.staticRow() : Rows.EMPTY_STATIC_ROW;
|
||||
return isFirst ? origin.staticRow() : Rows.EMPTY_STATIC_ROW;
|
||||
}
|
||||
|
||||
@Override
|
||||
|
|
@ -258,4 +264,4 @@ public class ThrottledUnfilteredIterator extends AbstractIterator<UnfilteredRowI
|
|||
}
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -17,15 +17,21 @@
|
|||
*/
|
||||
package org.apache.cassandra.db.rows;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.IOError;
|
||||
import java.io.IOException;
|
||||
import java.nio.BufferOverflowException;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.apache.cassandra.db.*;
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.DeletionTime;
|
||||
import org.apache.cassandra.db.EmptyIterators;
|
||||
import org.apache.cassandra.db.RegularAndStaticColumns;
|
||||
import org.apache.cassandra.db.SerializationHeader;
|
||||
import org.apache.cassandra.db.TypeSizes;
|
||||
import org.apache.cassandra.db.filter.ColumnFilter;
|
||||
import org.apache.cassandra.io.sstable.format.big.BigFormatPartitionWriter;
|
||||
import org.apache.cassandra.io.util.DataInputPlus;
|
||||
import org.apache.cassandra.io.util.DataOutputPlus;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
|
|
@ -60,7 +66,7 @@ import org.apache.cassandra.utils.ByteBufferUtil;
|
|||
*
|
||||
* Please note that the format described above is the on-wire format. On-disk, the format is basically the
|
||||
* same, but the header is written once per sstable, not once per-partition. Further, the actual row and
|
||||
* range tombstones are not written using this class, but rather by {@link ColumnIndex}.
|
||||
* range tombstones are not written using this class, but rather by {@link BigFormatPartitionWriter}.
|
||||
*/
|
||||
public class UnfilteredRowIteratorSerializer
|
||||
{
|
||||
|
|
@ -280,4 +286,4 @@ public class UnfilteredRowIteratorSerializer
|
|||
sHeader, key, isReversed, isEmpty, partitionDeletion, staticRow, rowEstimate);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -20,7 +20,6 @@
|
|||
*/
|
||||
package org.apache.cassandra.db.rows;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.Comparator;
|
||||
import java.util.Optional;
|
||||
|
||||
|
|
@ -32,14 +31,14 @@ import org.apache.cassandra.db.ClusteringPrefix;
|
|||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.DeletionTime;
|
||||
import org.apache.cassandra.db.RegularAndStaticColumns;
|
||||
import org.apache.cassandra.db.RowIndexEntry;
|
||||
import org.apache.cassandra.db.Slices;
|
||||
import org.apache.cassandra.db.filter.ClusteringIndexFilter;
|
||||
import org.apache.cassandra.db.filter.ColumnFilter;
|
||||
import org.apache.cassandra.db.transform.RTBoundValidator;
|
||||
import org.apache.cassandra.io.sstable.IndexInfo;
|
||||
import org.apache.cassandra.io.sstable.SSTable;
|
||||
import org.apache.cassandra.io.sstable.SSTableReadsListener;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReadsListener;
|
||||
import org.apache.cassandra.io.sstable.keycache.KeyCacheSupport;
|
||||
import org.apache.cassandra.io.sstable.metadata.StatsMetadata;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.utils.IteratorWithLowerBound;
|
||||
|
|
@ -197,21 +196,10 @@ public class UnfilteredRowIteratorWithLowerBound extends LazilyInitializedUnfilt
|
|||
*/
|
||||
private ClusteringBound<?> maybeGetLowerBoundFromKeyCache()
|
||||
{
|
||||
RowIndexEntry<?> rowIndexEntry = sstable.getCachedPosition(partitionKey(), false);
|
||||
if (rowIndexEntry == null || !rowIndexEntry.indexOnHeap())
|
||||
return null;
|
||||
if (sstable instanceof KeyCacheSupport<?>)
|
||||
return ((KeyCacheSupport<?>) sstable).getLowerBoundPrefixFromCache(partitionKey(), isReverseOrder);
|
||||
|
||||
try (RowIndexEntry.IndexInfoRetriever onHeapRetriever = rowIndexEntry.openWithIndex(null))
|
||||
{
|
||||
IndexInfo columns = onHeapRetriever.columnsIndex(isReverseOrder() ? rowIndexEntry.columnsIndexCount() - 1 : 0);
|
||||
ClusteringBound<?> bound = isReverseOrder() ? columns.lastName.asEndBound() : columns.firstName.asStartBound();
|
||||
assertBoundSize(bound);
|
||||
return bound.artificialLowerBound(isReverseOrder());
|
||||
}
|
||||
catch (IOException e)
|
||||
{
|
||||
throw new RuntimeException("should never occur", e);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -253,16 +241,16 @@ public class UnfilteredRowIteratorWithLowerBound extends LazilyInitializedUnfilt
|
|||
|
||||
final StatsMetadata m = sstable.getSSTableMetadata();
|
||||
ClusteringBound<?> bound = m.coveredClustering.open(isReverseOrder);
|
||||
assertBoundSize(bound);
|
||||
assertBoundSize(bound, sstable);
|
||||
return bound.artificialLowerBound(isReverseOrder);
|
||||
}
|
||||
|
||||
private void assertBoundSize(ClusteringPrefix<?> lowerBound)
|
||||
public static void assertBoundSize(ClusteringPrefix<?> lowerBound, SSTable sstable)
|
||||
{
|
||||
assert lowerBound.size() <= metadata().comparator.size() :
|
||||
assert lowerBound.size() <= sstable.metadata().comparator.size() :
|
||||
String.format("Unexpected number of clustering values %d, expected %d or fewer for %s",
|
||||
lowerBound.size(),
|
||||
metadata().comparator.size(),
|
||||
sstable.metadata().comparator.size(),
|
||||
sstable.getFilename());
|
||||
}
|
||||
}
|
||||
|
|
@ -17,12 +17,17 @@
|
|||
*/
|
||||
package org.apache.cassandra.db.rows;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.List;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.apache.cassandra.db.*;
|
||||
import org.apache.cassandra.db.Columns;
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.DeletionTime;
|
||||
import org.apache.cassandra.db.Digest;
|
||||
import org.apache.cassandra.db.EmptyIterators;
|
||||
import org.apache.cassandra.db.RegularAndStaticColumns;
|
||||
import org.apache.cassandra.db.filter.ColumnFilter;
|
||||
import org.apache.cassandra.db.transform.FilteredRows;
|
||||
import org.apache.cassandra.db.transform.MoreRows;
|
||||
|
|
@ -263,16 +268,22 @@ public abstract class UnfilteredRowIterators
|
|||
/**
|
||||
* Returns an iterator that concatenate the specified atom with the iterator.
|
||||
*/
|
||||
public static UnfilteredRowIterator concat(final Unfiltered first, final UnfilteredRowIterator rest)
|
||||
public static UnfilteredRowIterator concat(final Unfiltered first, final UnfilteredRowIterator wrapped)
|
||||
{
|
||||
return new WrappingUnfilteredRowIterator(rest)
|
||||
return new WrappingUnfilteredRowIterator()
|
||||
{
|
||||
private boolean hasReturnedFirst;
|
||||
|
||||
@Override
|
||||
public UnfilteredRowIterator wrapped()
|
||||
{
|
||||
return wrapped;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean hasNext()
|
||||
{
|
||||
return hasReturnedFirst ? super.hasNext() : true;
|
||||
return hasReturnedFirst ? wrapped.hasNext() : true;
|
||||
}
|
||||
|
||||
@Override
|
||||
|
|
@ -283,7 +294,7 @@ public abstract class UnfilteredRowIterators
|
|||
hasReturnedFirst = true;
|
||||
return first;
|
||||
}
|
||||
return super.next();
|
||||
return wrapped.next();
|
||||
}
|
||||
};
|
||||
}
|
||||
|
|
@ -604,4 +615,4 @@ public abstract class UnfilteredRowIterators
|
|||
}
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
|
|
@ -23,7 +23,7 @@ import org.apache.cassandra.db.DecoratedKey;
|
|||
import org.apache.cassandra.db.Slices;
|
||||
import org.apache.cassandra.db.filter.ColumnFilter;
|
||||
import org.apache.cassandra.db.partitions.UnfilteredPartitionIterator;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReadsListener;
|
||||
import org.apache.cassandra.io.sstable.SSTableReadsListener;
|
||||
|
||||
/**
|
||||
* Common data access interface for sstables and memtables.
|
||||
|
|
|
|||
|
|
@ -17,76 +17,72 @@
|
|||
*/
|
||||
package org.apache.cassandra.db.rows;
|
||||
|
||||
import com.google.common.collect.UnmodifiableIterator;
|
||||
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.DeletionTime;
|
||||
import org.apache.cassandra.db.RegularAndStaticColumns;
|
||||
import org.apache.cassandra.db.transform.Transformation;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.db.*;
|
||||
|
||||
/**
|
||||
* Abstract class to make writing unfiltered iterators that wrap another iterator
|
||||
* easier. By default, the wrapping iterator simply delegate every call to
|
||||
* the wrapped iterator so concrete implementations will have to override
|
||||
* some of the methods.
|
||||
* the wrapped iterator so concrete implementations will have to override some methods.
|
||||
* <p>
|
||||
* Note that if most of what you want to do is modifying/filtering the returned
|
||||
* {@code Unfiltered}, {@link org.apache.cassandra.db.transform.Transformation#merge(UnfilteredRowIterator,Transformation)} can be a simpler option.
|
||||
* {@code Unfiltered}, {@link org.apache.cassandra.db.transform.Transformation#apply(UnfilteredRowIterator, Transformation)}
|
||||
* can be a simpler option.
|
||||
*/
|
||||
public abstract class WrappingUnfilteredRowIterator extends UnmodifiableIterator<Unfiltered> implements UnfilteredRowIterator
|
||||
public interface WrappingUnfilteredRowIterator extends UnfilteredRowIterator
|
||||
{
|
||||
protected final UnfilteredRowIterator wrapped;
|
||||
UnfilteredRowIterator wrapped();
|
||||
|
||||
protected WrappingUnfilteredRowIterator(UnfilteredRowIterator wrapped)
|
||||
default TableMetadata metadata()
|
||||
{
|
||||
this.wrapped = wrapped;
|
||||
return wrapped().metadata();
|
||||
}
|
||||
|
||||
public TableMetadata metadata()
|
||||
default RegularAndStaticColumns columns()
|
||||
{
|
||||
return wrapped.metadata();
|
||||
return wrapped().columns();
|
||||
}
|
||||
|
||||
public RegularAndStaticColumns columns()
|
||||
default boolean isReverseOrder()
|
||||
{
|
||||
return wrapped.columns();
|
||||
return wrapped().isReverseOrder();
|
||||
}
|
||||
|
||||
public boolean isReverseOrder()
|
||||
default DecoratedKey partitionKey()
|
||||
{
|
||||
return wrapped.isReverseOrder();
|
||||
return wrapped().partitionKey();
|
||||
}
|
||||
|
||||
public DecoratedKey partitionKey()
|
||||
default DeletionTime partitionLevelDeletion()
|
||||
{
|
||||
return wrapped.partitionKey();
|
||||
return wrapped().partitionLevelDeletion();
|
||||
}
|
||||
|
||||
public DeletionTime partitionLevelDeletion()
|
||||
default Row staticRow()
|
||||
{
|
||||
return wrapped.partitionLevelDeletion();
|
||||
return wrapped().staticRow();
|
||||
}
|
||||
|
||||
public Row staticRow()
|
||||
default EncodingStats stats()
|
||||
{
|
||||
return wrapped.staticRow();
|
||||
return wrapped().stats();
|
||||
}
|
||||
|
||||
public EncodingStats stats()
|
||||
default boolean hasNext()
|
||||
{
|
||||
return wrapped.stats();
|
||||
return wrapped().hasNext();
|
||||
}
|
||||
|
||||
public boolean hasNext()
|
||||
default Unfiltered next()
|
||||
{
|
||||
return wrapped.hasNext();
|
||||
return wrapped().next();
|
||||
}
|
||||
|
||||
public Unfiltered next()
|
||||
default void close()
|
||||
{
|
||||
return wrapped.next();
|
||||
wrapped().close();
|
||||
}
|
||||
|
||||
public void close()
|
||||
{
|
||||
wrapped.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -27,12 +27,12 @@ import org.slf4j.Logger;
|
|||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.apache.cassandra.io.compress.CompressionMetadata;
|
||||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat.Components;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.util.ChannelProxy;
|
||||
import org.apache.cassandra.streaming.ProgressInfo;
|
||||
import org.apache.cassandra.streaming.StreamingDataOutputPlus;
|
||||
import org.apache.cassandra.streaming.StreamSession;
|
||||
import org.apache.cassandra.streaming.StreamingDataOutputPlus;
|
||||
import org.apache.cassandra.utils.FBUtilities;
|
||||
|
||||
/**
|
||||
|
|
@ -71,7 +71,7 @@ public class CassandraCompressedStreamWriter extends CassandraStreamWriter
|
|||
int sectionIdx = 0;
|
||||
|
||||
// stream each of the required sections of the file
|
||||
String filename = sstable.descriptor.filenameFor(Component.DATA);
|
||||
String filename = sstable.descriptor.fileFor(Components.DATA).toString();
|
||||
for (Section section : sections)
|
||||
{
|
||||
// length of the section to stream
|
||||
|
|
|
|||
|
|
@ -25,17 +25,20 @@ import java.util.function.UnaryOperator;
|
|||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.apache.cassandra.config.DatabaseDescriptor;
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.Directories;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleNewTracker;
|
||||
import org.apache.cassandra.io.compress.BufferType;
|
||||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.IOOptions;
|
||||
import org.apache.cassandra.io.sstable.SSTableMultiWriter;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat;
|
||||
import org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter;
|
||||
import org.apache.cassandra.io.sstable.SSTableZeroCopyWriter;
|
||||
import org.apache.cassandra.io.sstable.metadata.StatsMetadata;
|
||||
import org.apache.cassandra.io.util.DataInputPlus;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.io.util.SequentialWriterOption;
|
||||
import org.apache.cassandra.schema.TableId;
|
||||
import org.apache.cassandra.streaming.ProgressInfo;
|
||||
import org.apache.cassandra.streaming.StreamReceiver;
|
||||
|
|
@ -60,9 +63,6 @@ public class CassandraEntireSSTableStreamReader implements IStreamReader
|
|||
|
||||
public CassandraEntireSSTableStreamReader(StreamMessageHeader messageHeader, CassandraStreamHeader streamHeader, StreamSession session)
|
||||
{
|
||||
if (streamHeader.format != SSTableFormat.Type.BIG)
|
||||
throw new AssertionError("Unsupported SSTable format " + streamHeader.format);
|
||||
|
||||
if (session.getPendingRepair() != null)
|
||||
{
|
||||
// we should only ever be streaming pending repair sstables if the session has a pending repair id
|
||||
|
|
@ -84,7 +84,7 @@ public class CassandraEntireSSTableStreamReader implements IStreamReader
|
|||
*/
|
||||
@SuppressWarnings("resource") // input needs to remain open, streams on top of it can't be closed
|
||||
@Override
|
||||
public SSTableMultiWriter read(DataInputPlus in) throws Throwable
|
||||
public SSTableMultiWriter read(DataInputPlus in) throws IOException
|
||||
{
|
||||
ColumnFamilyStore cfs = ColumnFamilyStore.getIfExists(tableId);
|
||||
if (cfs == null)
|
||||
|
|
@ -103,7 +103,7 @@ public class CassandraEntireSSTableStreamReader implements IStreamReader
|
|||
prettyPrintMemory(totalSize),
|
||||
cfs.metadata());
|
||||
|
||||
BigTableZeroCopyWriter writer = null;
|
||||
SSTableZeroCopyWriter writer = null;
|
||||
|
||||
try
|
||||
{
|
||||
|
|
@ -122,7 +122,7 @@ public class CassandraEntireSSTableStreamReader implements IStreamReader
|
|||
prettyPrintMemory(totalSize));
|
||||
|
||||
writer.writeComponent(component.type, in, length);
|
||||
session.progress(writer.descriptor.filenameFor(component), ProgressInfo.Direction.IN, length, length, length);
|
||||
session.progress(writer.descriptor.fileFor(component).toString(), ProgressInfo.Direction.IN, length, length, length);
|
||||
bytesRead += length;
|
||||
|
||||
logger.debug("[Stream #{}] Finished receiving {} component from {}, componentSize = {}, readBytes = {}, totalSize = {}",
|
||||
|
|
@ -145,7 +145,11 @@ public class CassandraEntireSSTableStreamReader implements IStreamReader
|
|||
{
|
||||
logger.error("[Stream {}] Error while reading sstable from stream for table = {}", session.planId(), cfs.metadata(), e);
|
||||
if (writer != null)
|
||||
e = writer.abort(e);
|
||||
{
|
||||
Throwable e2 = writer.abort(null);
|
||||
if (e2 != null)
|
||||
e.addSuppressed(e2);
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
|
@ -165,7 +169,7 @@ public class CassandraEntireSSTableStreamReader implements IStreamReader
|
|||
}
|
||||
|
||||
@SuppressWarnings("resource")
|
||||
protected BigTableZeroCopyWriter createWriter(ColumnFamilyStore cfs, long totalSize, Collection<Component> components) throws IOException
|
||||
protected SSTableZeroCopyWriter createWriter(ColumnFamilyStore cfs, long totalSize, Collection<Component> components) throws IOException
|
||||
{
|
||||
File dataDir = getDataDir(cfs, totalSize);
|
||||
|
||||
|
|
@ -176,8 +180,24 @@ public class CassandraEntireSSTableStreamReader implements IStreamReader
|
|||
|
||||
Descriptor desc = cfs.newSSTableDescriptor(dataDir, header.version, header.format);
|
||||
|
||||
logger.debug("[Table #{}] {} Components to write: {}", cfs.metadata(), desc.filenameFor(Component.DATA), components);
|
||||
IOOptions ioOptions = new IOOptions(DatabaseDescriptor.getDiskOptimizationStrategy(),
|
||||
DatabaseDescriptor.getDiskAccessMode(),
|
||||
DatabaseDescriptor.getIndexAccessMode(),
|
||||
DatabaseDescriptor.getDiskOptimizationEstimatePercentile(),
|
||||
SequentialWriterOption.newBuilder()
|
||||
.trickleFsync(false)
|
||||
.bufferSize(2 << 20)
|
||||
.bufferType(BufferType.OFF_HEAP)
|
||||
.build(),
|
||||
DatabaseDescriptor.getFlushCompression());
|
||||
|
||||
return new BigTableZeroCopyWriter(desc, cfs.metadata, lifecycleNewTracker, components);
|
||||
logger.debug("[Table #{}] {} Components to write: {}", cfs.metadata(), desc, components);
|
||||
return desc.getFormat()
|
||||
.getWriterFactory()
|
||||
.builder(desc)
|
||||
.setComponents(components)
|
||||
.setTableMetadataRef(cfs.metadata)
|
||||
.setIOOptions(ioOptions)
|
||||
.createZeroCopyWriter(lifecycleNewTracker, cfs);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -27,9 +27,9 @@ import org.slf4j.LoggerFactory;
|
|||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.streaming.ProgressInfo;
|
||||
import org.apache.cassandra.streaming.StreamingDataOutputPlus;
|
||||
import org.apache.cassandra.streaming.StreamManager;
|
||||
import org.apache.cassandra.streaming.StreamSession;
|
||||
import org.apache.cassandra.streaming.StreamingDataOutputPlus;
|
||||
|
||||
import static org.apache.cassandra.streaming.StreamManager.StreamRateLimiter;
|
||||
import static org.apache.cassandra.utils.FBUtilities.prettyPrintMemory;
|
||||
|
|
@ -93,7 +93,7 @@ public class CassandraEntireSSTableStreamWriter
|
|||
long bytesWritten = out.writeFileToChannel(channel, limiter);
|
||||
progress += bytesWritten;
|
||||
|
||||
session.progress(sstable.descriptor.filenameFor(component), ProgressInfo.Direction.OUT, bytesWritten, bytesWritten, length);
|
||||
session.progress(sstable.descriptor.fileFor(component).toString(), ProgressInfo.Direction.OUT, bytesWritten, bytesWritten, length);
|
||||
|
||||
logger.debug("[Stream #{}] Finished streaming {}.{} gen {} component {} to {}, xfered = {}, length = {}, totalSize = {}",
|
||||
session.planId(),
|
||||
|
|
|
|||
|
|
@ -182,7 +182,7 @@ public class CassandraStreamHeader
|
|||
|
||||
if (header.isEntireSSTable)
|
||||
{
|
||||
ComponentManifest.serializer.serialize(header.componentManifest, out, version);
|
||||
ComponentManifest.serializers.get(header.format).serialize(header.componentManifest, out, version);
|
||||
ByteBufferUtil.writeWithVIntLength(header.firstKey.getKey(), out);
|
||||
}
|
||||
}
|
||||
|
|
@ -201,8 +201,9 @@ public class CassandraStreamHeader
|
|||
@VisibleForTesting
|
||||
public CassandraStreamHeader deserialize(DataInputPlus in, int version, Function<TableId, IPartitioner> partitionerMapper) throws IOException
|
||||
{
|
||||
Version sstableVersion = SSTableFormat.Type.current().info.getVersion(in.readUTF());
|
||||
SSTableFormat.Type format = SSTableFormat.Type.validate(in.readUTF());
|
||||
String sstableVersionString = in.readUTF();
|
||||
SSTableFormat.Type format = SSTableFormat.Type.getByName(in.readUTF());
|
||||
Version sstableVersion = format.info.getVersion(sstableVersionString);
|
||||
|
||||
long estimatedKeys = in.readLong();
|
||||
int count = in.readInt();
|
||||
|
|
@ -221,7 +222,7 @@ public class CassandraStreamHeader
|
|||
|
||||
if (isEntireSSTable)
|
||||
{
|
||||
manifest = ComponentManifest.serializer.deserialize(in, version);
|
||||
manifest = ComponentManifest.serializers.get(format).deserialize(in, version);
|
||||
ByteBuffer keyBuf = ByteBufferUtil.readWithVIntLength(in);
|
||||
IPartitioner partitioner = partitionerMapper.apply(tableId);
|
||||
if (partitioner == null)
|
||||
|
|
@ -267,7 +268,7 @@ public class CassandraStreamHeader
|
|||
|
||||
if (header.isEntireSSTable)
|
||||
{
|
||||
size += ComponentManifest.serializer.serializedSize(header.componentManifest, version);
|
||||
size += ComponentManifest.serializers.get(header.format).serializedSize(header.componentManifest, version);
|
||||
size += ByteBufferUtil.serializedSizeWithVIntLength(header.firstKey.getKey());
|
||||
}
|
||||
return size;
|
||||
|
|
|
|||
|
|
@ -39,9 +39,9 @@ import org.apache.cassandra.db.rows.Row;
|
|||
import org.apache.cassandra.db.rows.Unfiltered;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.exceptions.UnknownColumnException;
|
||||
import org.apache.cassandra.io.sstable.RangeAwareSSTableWriter;
|
||||
import org.apache.cassandra.io.sstable.SSTableMultiWriter;
|
||||
import org.apache.cassandra.io.sstable.SSTableSimpleIterator;
|
||||
import org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.Version;
|
||||
|
|
|
|||
|
|
@ -21,23 +21,21 @@ import java.io.IOException;
|
|||
import java.nio.ByteBuffer;
|
||||
import java.util.Collection;
|
||||
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import net.jpountz.lz4.LZ4Compressor;
|
||||
import net.jpountz.lz4.LZ4Factory;
|
||||
import org.apache.cassandra.io.compress.BufferType;
|
||||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat.Components;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.util.ChannelProxy;
|
||||
import org.apache.cassandra.io.util.DataIntegrityMetadata;
|
||||
import org.apache.cassandra.io.util.DataIntegrityMetadata.ChecksumValidator;
|
||||
import org.apache.cassandra.streaming.ProgressInfo;
|
||||
import org.apache.cassandra.streaming.StreamingDataOutputPlus;
|
||||
import org.apache.cassandra.streaming.StreamManager;
|
||||
import org.apache.cassandra.streaming.StreamManager.StreamRateLimiter;
|
||||
import org.apache.cassandra.streaming.StreamSession;
|
||||
import org.apache.cassandra.streaming.StreamingDataOutputPlus;
|
||||
import org.apache.cassandra.streaming.async.StreamCompressionSerializer;
|
||||
import org.apache.cassandra.utils.FBUtilities;
|
||||
import org.apache.cassandra.utils.memory.BufferPools;
|
||||
|
|
@ -84,9 +82,7 @@ public class CassandraStreamWriter
|
|||
sstable.getFilename(), session.peer, sstable.getSSTableMetadata().repairedAt, totalSize);
|
||||
|
||||
try(ChannelProxy proxy = sstable.getDataChannel().newChannel();
|
||||
ChecksumValidator validator = new File(sstable.descriptor.filenameFor(Component.CRC)).exists()
|
||||
? DataIntegrityMetadata.checksumValidator(sstable.descriptor)
|
||||
: null)
|
||||
ChecksumValidator validator = sstable.maybeGetChecksumValidator())
|
||||
{
|
||||
int bufferSize = validator == null ? DEFAULT_CHUNK_SIZE: validator.chunkSize;
|
||||
|
||||
|
|
@ -94,7 +90,7 @@ public class CassandraStreamWriter
|
|||
long progress = 0L;
|
||||
|
||||
// stream each of the required sections of the file
|
||||
String filename = sstable.descriptor.filenameFor(Component.DATA);
|
||||
String filename = sstable.descriptor.fileFor(Components.DATA).toString();
|
||||
for (SSTableReader.PartitionPositionBounds section : sections)
|
||||
{
|
||||
long start = validator == null ? section.lowerPosition : validator.chunkStart(section.lowerPosition);
|
||||
|
|
|
|||
|
|
@ -18,32 +18,23 @@
|
|||
|
||||
package org.apache.cassandra.db.streaming;
|
||||
|
||||
import com.google.common.collect.ImmutableSet;
|
||||
import java.io.IOException;
|
||||
import java.nio.channels.FileChannel;
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.util.FileUtils;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.channels.FileChannel;
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
|
||||
/**
|
||||
* Mutable SSTable components and their hardlinks to avoid concurrent sstable component modification
|
||||
* during entire-sstable-streaming.
|
||||
*/
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.io.util.FileUtils;
|
||||
|
||||
public class ComponentContext implements AutoCloseable
|
||||
{
|
||||
private static final Logger logger = LoggerFactory.getLogger(ComponentContext.class);
|
||||
|
||||
private static final Set<Component> MUTABLE_COMPONENTS = ImmutableSet.of(Component.STATS, Component.SUMMARY);
|
||||
|
||||
private final Map<Component, File> hardLinks;
|
||||
private final ComponentManifest manifest;
|
||||
|
||||
|
|
@ -57,13 +48,13 @@ public class ComponentContext implements AutoCloseable
|
|||
{
|
||||
Map<Component, File> hardLinks = new HashMap<>(1);
|
||||
|
||||
for (Component component : MUTABLE_COMPONENTS)
|
||||
for (Component component : descriptor.getFormat().mutableComponents())
|
||||
{
|
||||
File file = new File(descriptor.filenameFor(component));
|
||||
File file = descriptor.fileFor(component);
|
||||
if (!file.exists())
|
||||
continue;
|
||||
|
||||
File hardlink = new File(descriptor.tmpFilenameForStreaming(component));
|
||||
File hardlink = descriptor.tmpFileForStreaming(component);
|
||||
FileUtils.createHardLink(file, hardlink);
|
||||
hardLinks.put(component, hardlink);
|
||||
}
|
||||
|
|
@ -81,9 +72,9 @@ public class ComponentContext implements AutoCloseable
|
|||
*/
|
||||
public FileChannel channel(Descriptor descriptor, Component component, long size) throws IOException
|
||||
{
|
||||
String toTransfer = hardLinks.containsKey(component) ? hardLinks.get(component).path() : descriptor.filenameFor(component);
|
||||
File toTransfer = hardLinks.containsKey(component) ? hardLinks.get(component) : descriptor.fileFor(component);
|
||||
@SuppressWarnings("resource") // file channel will be closed by Caller
|
||||
FileChannel channel = new File(toTransfer).newReadChannel();
|
||||
FileChannel channel = toTransfer.newReadChannel();
|
||||
|
||||
assert size == channel.size() : String.format("Entire sstable streaming expects %s file size to be %s but got %s.",
|
||||
component, size, channel.size());
|
||||
|
|
|
|||
|
|
@ -19,26 +19,30 @@
|
|||
package org.apache.cassandra.db.streaming;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Iterator;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import com.google.common.collect.ImmutableList;
|
||||
import com.google.common.collect.ImmutableMap;
|
||||
import com.google.common.collect.Iterators;
|
||||
|
||||
import org.apache.cassandra.db.TypeSizes;
|
||||
import org.apache.cassandra.io.IVersionedSerializer;
|
||||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat;
|
||||
import org.apache.cassandra.io.util.DataInputPlus;
|
||||
import org.apache.cassandra.io.util.DataOutputPlus;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
|
||||
/**
|
||||
* SSTable components and their sizes to be tranfered via entire-sstable-streaming
|
||||
*/
|
||||
public final class ComponentManifest implements Iterable<Component>
|
||||
{
|
||||
private static final List<Component> STREAM_COMPONENTS = ImmutableList.of(Component.DATA, Component.PRIMARY_INDEX, Component.STATS,
|
||||
Component.COMPRESSION_INFO, Component.FILTER, Component.SUMMARY,
|
||||
Component.DIGEST, Component.CRC);
|
||||
|
||||
private final LinkedHashMap<Component, Long> components;
|
||||
|
||||
public ComponentManifest(Map<Component, Long> components)
|
||||
|
|
@ -49,11 +53,11 @@ public final class ComponentManifest implements Iterable<Component>
|
|||
@VisibleForTesting
|
||||
public static ComponentManifest create(Descriptor descriptor)
|
||||
{
|
||||
LinkedHashMap<Component, Long> components = new LinkedHashMap<>(STREAM_COMPONENTS.size());
|
||||
LinkedHashMap<Component, Long> components = new LinkedHashMap<>(descriptor.getFormat().streamingComponents().size());
|
||||
|
||||
for (Component component : STREAM_COMPONENTS)
|
||||
for (Component component : descriptor.getFormat().streamingComponents())
|
||||
{
|
||||
File file = new File(descriptor.filenameFor(component));
|
||||
File file = descriptor.fileFor(component);
|
||||
if (!file.exists())
|
||||
continue;
|
||||
|
||||
|
|
@ -111,45 +115,58 @@ public final class ComponentManifest implements Iterable<Component>
|
|||
'}';
|
||||
}
|
||||
|
||||
public static final IVersionedSerializer<ComponentManifest> serializer = new IVersionedSerializer<ComponentManifest>()
|
||||
public static final Map<SSTableFormat.Type, IVersionedSerializer<ComponentManifest>> serializers;
|
||||
|
||||
static
|
||||
{
|
||||
public void serialize(ComponentManifest manifest, DataOutputPlus out, int version) throws IOException
|
||||
ImmutableMap.Builder<SSTableFormat.Type, IVersionedSerializer<ComponentManifest>> b = ImmutableMap.builder();
|
||||
for (SSTableFormat.Type formatType : SSTableFormat.Type.values())
|
||||
{
|
||||
out.writeUnsignedVInt32(manifest.components.size());
|
||||
for (Map.Entry<Component, Long> entry : manifest.components.entrySet())
|
||||
IVersionedSerializer<ComponentManifest> serializer = new IVersionedSerializer<ComponentManifest>()
|
||||
{
|
||||
out.writeUTF(entry.getKey().name);
|
||||
out.writeUnsignedVInt(entry.getValue());
|
||||
}
|
||||
|
||||
public void serialize(ComponentManifest manifest, DataOutputPlus out, int version) throws IOException
|
||||
{
|
||||
out.writeUnsignedVInt32(manifest.components.size());
|
||||
for (Map.Entry<Component, Long> entry : manifest.components.entrySet())
|
||||
{
|
||||
out.writeUTF(entry.getKey().name);
|
||||
out.writeUnsignedVInt(entry.getValue());
|
||||
}
|
||||
}
|
||||
|
||||
public ComponentManifest deserialize(DataInputPlus in, int version) throws IOException
|
||||
{
|
||||
int size = in.readUnsignedVInt32();
|
||||
|
||||
LinkedHashMap<Component, Long> components = new LinkedHashMap<>(size);
|
||||
|
||||
for (int i = 0; i < size; i++)
|
||||
{
|
||||
Component component = Component.parse(in.readUTF(), formatType);
|
||||
long length = in.readUnsignedVInt();
|
||||
components.put(component, length);
|
||||
}
|
||||
|
||||
return new ComponentManifest(components);
|
||||
}
|
||||
|
||||
public long serializedSize(ComponentManifest manifest, int version)
|
||||
{
|
||||
long size = TypeSizes.sizeofUnsignedVInt(manifest.components.size());
|
||||
for (Map.Entry<Component, Long> entry : manifest.components.entrySet())
|
||||
{
|
||||
size += TypeSizes.sizeof(entry.getKey().name);
|
||||
size += TypeSizes.sizeofUnsignedVInt(entry.getValue());
|
||||
}
|
||||
return size;
|
||||
}
|
||||
};
|
||||
|
||||
b.put(formatType, serializer);
|
||||
}
|
||||
|
||||
public ComponentManifest deserialize(DataInputPlus in, int version) throws IOException
|
||||
{
|
||||
int size = in.readUnsignedVInt32();
|
||||
|
||||
LinkedHashMap<Component, Long> components = new LinkedHashMap<>(size);
|
||||
|
||||
for (int i = 0; i < size; i++)
|
||||
{
|
||||
Component component = Component.parse(in.readUTF());
|
||||
long length = in.readUnsignedVInt();
|
||||
components.put(component, length);
|
||||
}
|
||||
|
||||
return new ComponentManifest(components);
|
||||
}
|
||||
|
||||
public long serializedSize(ComponentManifest manifest, int version)
|
||||
{
|
||||
long size = TypeSizes.sizeofUnsignedVInt(manifest.components.size());
|
||||
for (Map.Entry<Component, Long> entry : manifest.components.entrySet())
|
||||
{
|
||||
size += TypeSizes.sizeof(entry.getKey().name);
|
||||
size += TypeSizes.sizeofUnsignedVInt(entry.getValue());
|
||||
}
|
||||
return size;
|
||||
}
|
||||
};
|
||||
serializers = b.build();
|
||||
}
|
||||
|
||||
@Override
|
||||
public Iterator<Component> iterator()
|
||||
|
|
|
|||
|
|
@ -20,8 +20,6 @@
|
|||
*/
|
||||
package org.apache.cassandra.db.transform;
|
||||
|
||||
import java.util.Collections;
|
||||
|
||||
import org.apache.cassandra.db.partitions.BasePartitionIterator;
|
||||
import org.apache.cassandra.db.rows.BaseRowIterator;
|
||||
import org.apache.cassandra.utils.Throwables;
|
||||
|
|
@ -112,7 +110,7 @@ implements BasePartitionIterator<R>
|
|||
catch (Throwable t)
|
||||
{
|
||||
if (next != null)
|
||||
Throwables.close(t, Collections.singleton(next));
|
||||
Throwables.close(t, next);
|
||||
throw t;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -23,11 +23,17 @@ import java.net.InetAddress;
|
|||
import java.net.InetSocketAddress;
|
||||
import java.net.UnknownHostException;
|
||||
import java.nio.ByteBuffer;
|
||||
import java.util.*;
|
||||
import java.util.concurrent.*;
|
||||
import java.util.Collection;
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.ExecutionException;
|
||||
import java.util.concurrent.Future;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.TimeoutException;
|
||||
|
||||
import com.google.common.net.HostAndPort;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
|
|
@ -40,6 +46,7 @@ import org.apache.cassandra.hadoop.ConfigHelper;
|
|||
import org.apache.cassandra.hadoop.HadoopCompat;
|
||||
import org.apache.cassandra.io.sstable.CQLSSTableWriter;
|
||||
import org.apache.cassandra.io.sstable.SSTableLoader;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.io.util.FileUtils;
|
||||
import org.apache.cassandra.locator.InetAddressAndPort;
|
||||
import org.apache.cassandra.schema.TableMetadataRef;
|
||||
|
|
@ -321,6 +328,6 @@ public class CqlBulkRecordWriter extends RecordWriter<Object, List<ByteBuffer>>
|
|||
public void output(String msg) {}
|
||||
public void debug(String msg) {}
|
||||
public void warn(String msg) {}
|
||||
public void warn(String msg, Throwable th) {}
|
||||
public void warn(Throwable th, String msg) {}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -26,13 +26,19 @@ import java.util.Set;
|
|||
import java.util.concurrent.Callable;
|
||||
import java.util.function.BiFunction;
|
||||
|
||||
import org.apache.cassandra.db.memtable.Memtable;
|
||||
import org.apache.cassandra.schema.ColumnMetadata;
|
||||
import org.apache.cassandra.cql3.Operator;
|
||||
import org.apache.cassandra.db.*;
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.DeletionTime;
|
||||
import org.apache.cassandra.db.RangeTombstone;
|
||||
import org.apache.cassandra.db.ReadCommand;
|
||||
import org.apache.cassandra.db.ReadExecutionController;
|
||||
import org.apache.cassandra.db.RegularAndStaticColumns;
|
||||
import org.apache.cassandra.db.WriteContext;
|
||||
import org.apache.cassandra.db.compaction.OperationType;
|
||||
import org.apache.cassandra.db.filter.RowFilter;
|
||||
import org.apache.cassandra.db.marshal.AbstractType;
|
||||
import org.apache.cassandra.db.memtable.Memtable;
|
||||
import org.apache.cassandra.db.partitions.PartitionIterator;
|
||||
import org.apache.cassandra.db.partitions.PartitionUpdate;
|
||||
import org.apache.cassandra.db.partitions.UnfilteredPartitionIterator;
|
||||
|
|
@ -42,8 +48,9 @@ import org.apache.cassandra.index.internal.CollatedViewIndexBuilder;
|
|||
import org.apache.cassandra.index.transactions.IndexTransaction;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.ReducingKeyIterator;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFlushObserver;
|
||||
import org.apache.cassandra.io.sstable.SSTableFlushObserver;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.schema.ColumnMetadata;
|
||||
import org.apache.cassandra.schema.IndexMetadata;
|
||||
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -17,16 +17,30 @@
|
|||
*/
|
||||
package org.apache.cassandra.index.sasi;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.Collection;
|
||||
import java.util.Collections;
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
import java.util.NavigableMap;
|
||||
import java.util.Optional;
|
||||
import java.util.Set;
|
||||
import java.util.SortedMap;
|
||||
import java.util.TreeMap;
|
||||
import java.util.concurrent.Callable;
|
||||
import java.util.function.BiFunction;
|
||||
|
||||
import com.googlecode.concurrenttrees.common.Iterables;
|
||||
|
||||
import org.apache.cassandra.config.*;
|
||||
import org.apache.cassandra.config.DatabaseDescriptor;
|
||||
import org.apache.cassandra.cql3.Operator;
|
||||
import org.apache.cassandra.cql3.statements.schema.IndexTarget;
|
||||
import org.apache.cassandra.db.*;
|
||||
import org.apache.cassandra.db.CassandraWriteContext;
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.DeletionTime;
|
||||
import org.apache.cassandra.db.RangeTombstone;
|
||||
import org.apache.cassandra.db.ReadCommand;
|
||||
import org.apache.cassandra.db.RegularAndStaticColumns;
|
||||
import org.apache.cassandra.db.WriteContext;
|
||||
import org.apache.cassandra.db.compaction.CompactionManager;
|
||||
import org.apache.cassandra.db.compaction.OperationType;
|
||||
import org.apache.cassandra.db.filter.RowFilter;
|
||||
|
|
@ -49,9 +63,15 @@ import org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter;
|
|||
import org.apache.cassandra.index.sasi.plan.QueryPlan;
|
||||
import org.apache.cassandra.index.transactions.IndexTransaction;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFlushObserver;
|
||||
import org.apache.cassandra.io.sstable.SSTableFlushObserver;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.notifications.*;
|
||||
import org.apache.cassandra.notifications.INotification;
|
||||
import org.apache.cassandra.notifications.INotificationConsumer;
|
||||
import org.apache.cassandra.notifications.MemtableDiscardedNotification;
|
||||
import org.apache.cassandra.notifications.MemtableRenewedNotification;
|
||||
import org.apache.cassandra.notifications.MemtableSwitchedNotification;
|
||||
import org.apache.cassandra.notifications.SSTableAddedNotification;
|
||||
import org.apache.cassandra.notifications.SSTableListChangedNotification;
|
||||
import org.apache.cassandra.schema.ColumnMetadata;
|
||||
import org.apache.cassandra.schema.IndexMetadata;
|
||||
import org.apache.cassandra.schema.Schema;
|
||||
|
|
|
|||
|
|
@ -21,13 +21,13 @@
|
|||
package org.apache.cassandra.index.sasi;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.*;
|
||||
import java.util.Collection;
|
||||
import java.util.Collections;
|
||||
import java.util.Map;
|
||||
import java.util.SortedMap;
|
||||
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.schema.ColumnMetadata;
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.RowIndexEntry;
|
||||
import org.apache.cassandra.db.compaction.CompactionInfo;
|
||||
import org.apache.cassandra.db.compaction.CompactionInterruptedException;
|
||||
import org.apache.cassandra.db.compaction.OperationType;
|
||||
|
|
@ -36,11 +36,12 @@ import org.apache.cassandra.index.SecondaryIndexBuilder;
|
|||
import org.apache.cassandra.index.sasi.conf.ColumnIndex;
|
||||
import org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter;
|
||||
import org.apache.cassandra.io.FSReadError;
|
||||
import org.apache.cassandra.io.sstable.KeyIterator;
|
||||
import org.apache.cassandra.io.sstable.SSTable;
|
||||
import org.apache.cassandra.io.sstable.KeyReader;
|
||||
import org.apache.cassandra.io.sstable.SSTableIdentityIterator;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.io.util.RandomAccessReader;
|
||||
import org.apache.cassandra.schema.ColumnMetadata;
|
||||
import org.apache.cassandra.utils.ByteBufferUtil;
|
||||
import org.apache.cassandra.utils.TimeUUID;
|
||||
|
||||
|
|
@ -57,22 +58,23 @@ class SASIIndexBuilder extends SecondaryIndexBuilder
|
|||
private final SortedMap<SSTableReader, Map<ColumnMetadata, ColumnIndex>> sstables;
|
||||
|
||||
private long bytesProcessed = 0;
|
||||
private final long totalSizeInBytes;
|
||||
private final long totalBytesToProcess;
|
||||
|
||||
public SASIIndexBuilder(ColumnFamilyStore cfs, SortedMap<SSTableReader, Map<ColumnMetadata, ColumnIndex>> sstables)
|
||||
{
|
||||
long totalIndexBytes = 0;
|
||||
long totalBytesToProcess = 0;
|
||||
for (SSTableReader sstable : sstables.keySet())
|
||||
totalIndexBytes += getPrimaryIndexLength(sstable);
|
||||
totalBytesToProcess += sstable.uncompressedLength();
|
||||
|
||||
this.cfs = cfs;
|
||||
this.sstables = sstables;
|
||||
this.totalSizeInBytes = totalIndexBytes;
|
||||
this.totalBytesToProcess = totalBytesToProcess;
|
||||
}
|
||||
|
||||
public void build()
|
||||
{
|
||||
AbstractType<?> keyValidator = cfs.metadata().partitionKeyType;
|
||||
long processedBytesInFinishedSSTables = 0;
|
||||
for (Map.Entry<SSTableReader, Map<ColumnMetadata, ColumnIndex>> e : sstables.entrySet())
|
||||
{
|
||||
SSTableReader sstable = e.getKey();
|
||||
|
|
@ -83,46 +85,45 @@ class SASIIndexBuilder extends SecondaryIndexBuilder
|
|||
PerSSTableIndexWriter indexWriter = SASIIndex.newWriter(keyValidator, sstable.descriptor, indexes, OperationType.COMPACTION);
|
||||
targetDirectory = indexWriter.getDescriptor().directory.path();
|
||||
|
||||
long previousKeyPosition = 0;
|
||||
try (KeyIterator keys = new KeyIterator(sstable.descriptor, cfs.metadata()))
|
||||
try (KeyReader keys = sstable.keyReader())
|
||||
{
|
||||
while (keys.hasNext())
|
||||
while (!keys.isExhausted())
|
||||
{
|
||||
if (isStopRequested())
|
||||
throw new CompactionInterruptedException(getCompactionInfo());
|
||||
|
||||
final DecoratedKey key = keys.next();
|
||||
final long keyPosition = keys.getKeyPosition();
|
||||
final DecoratedKey key = sstable.decorateKey(keys.key());
|
||||
final long keyPosition = keys.keyPositionForSecondaryIndex();
|
||||
|
||||
indexWriter.startPartition(key, keyPosition);
|
||||
indexWriter.startPartition(key, keys.dataPosition(), keyPosition);
|
||||
|
||||
try
|
||||
dataFile.seek(keys.dataPosition());
|
||||
ByteBufferUtil.readWithShortLength(dataFile); // key
|
||||
|
||||
try (SSTableIdentityIterator partition = SSTableIdentityIterator.create(sstable, dataFile, key))
|
||||
{
|
||||
RowIndexEntry indexEntry = sstable.getPosition(key, SSTableReader.Operator.EQ);
|
||||
dataFile.seek(indexEntry.position);
|
||||
ByteBufferUtil.readWithShortLength(dataFile); // key
|
||||
|
||||
try (SSTableIdentityIterator partition = SSTableIdentityIterator.create(sstable, dataFile, key))
|
||||
// if the row has statics attached, it has to be indexed separately
|
||||
if (cfs.metadata().hasStaticColumns())
|
||||
{
|
||||
// if the row has statics attached, it has to be indexed separately
|
||||
if (cfs.metadata().hasStaticColumns())
|
||||
indexWriter.nextUnfilteredCluster(partition.staticRow());
|
||||
|
||||
while (partition.hasNext())
|
||||
indexWriter.nextUnfilteredCluster(partition.next());
|
||||
indexWriter.nextUnfilteredCluster(partition.staticRow());
|
||||
}
|
||||
}
|
||||
catch (IOException ex)
|
||||
{
|
||||
throw new FSReadError(ex, sstable.getFilename());
|
||||
|
||||
while (partition.hasNext())
|
||||
indexWriter.nextUnfilteredCluster(partition.next());
|
||||
}
|
||||
|
||||
bytesProcessed += keyPosition - previousKeyPosition;
|
||||
previousKeyPosition = keyPosition;
|
||||
keys.advance();
|
||||
long dataPosition = keys.isExhausted() ? sstable.uncompressedLength() : keys.dataPosition();
|
||||
bytesProcessed = processedBytesInFinishedSSTables + dataPosition;
|
||||
}
|
||||
|
||||
completeSSTable(indexWriter, sstable, indexes.values());
|
||||
}
|
||||
catch (IOException ex)
|
||||
{
|
||||
throw new FSReadError(ex, sstable.getFilename());
|
||||
}
|
||||
processedBytesInFinishedSSTables += sstable.uncompressedLength();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -132,25 +133,19 @@ class SASIIndexBuilder extends SecondaryIndexBuilder
|
|||
return new CompactionInfo(cfs.metadata(),
|
||||
OperationType.INDEX_BUILD,
|
||||
bytesProcessed,
|
||||
totalSizeInBytes,
|
||||
totalBytesToProcess,
|
||||
compactionId,
|
||||
sstables.keySet(),
|
||||
targetDirectory);
|
||||
}
|
||||
|
||||
private long getPrimaryIndexLength(SSTable sstable)
|
||||
{
|
||||
File primaryIndex = new File(sstable.getIndexFilename());
|
||||
return primaryIndex.exists() ? primaryIndex.length() : 0;
|
||||
}
|
||||
|
||||
private void completeSSTable(PerSSTableIndexWriter indexWriter, SSTableReader sstable, Collection<ColumnIndex> indexes)
|
||||
{
|
||||
indexWriter.complete();
|
||||
|
||||
for (ColumnIndex index : indexes)
|
||||
{
|
||||
File tmpIndex = new File(sstable.descriptor.filenameFor(index.getComponent()));
|
||||
File tmpIndex = sstable.descriptor.fileFor(index.getComponent());
|
||||
if (!tmpIndex.exists()) // no data was inserted into the index for given sstable
|
||||
continue;
|
||||
|
||||
|
|
|
|||
|
|
@ -22,6 +22,9 @@ import java.nio.ByteBuffer;
|
|||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
|
||||
import com.google.common.base.Function;
|
||||
import org.apache.commons.lang3.builder.HashCodeBuilder;
|
||||
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.marshal.AbstractType;
|
||||
import org.apache.cassandra.index.sasi.conf.ColumnIndex;
|
||||
|
|
@ -36,10 +39,6 @@ import org.apache.cassandra.io.util.File;
|
|||
import org.apache.cassandra.io.util.FileUtils;
|
||||
import org.apache.cassandra.utils.concurrent.Ref;
|
||||
|
||||
import org.apache.commons.lang3.builder.HashCodeBuilder;
|
||||
|
||||
import com.google.common.base.Function;
|
||||
|
||||
public class SSTableIndex
|
||||
{
|
||||
private final ColumnIndex columnIndex;
|
||||
|
|
@ -176,7 +175,7 @@ public class SSTableIndex
|
|||
{
|
||||
try
|
||||
{
|
||||
return sstable.keyAt(offset);
|
||||
return sstable.keyAtPositionFromSecondaryIndex(offset);
|
||||
}
|
||||
catch (IOException e)
|
||||
{
|
||||
|
|
|
|||
|
|
@ -28,13 +28,12 @@ import java.util.concurrent.atomic.AtomicReference;
|
|||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
|
||||
import org.apache.cassandra.schema.ColumnMetadata;
|
||||
import org.apache.cassandra.cql3.Operator;
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.memtable.Memtable;
|
||||
import org.apache.cassandra.db.marshal.AbstractType;
|
||||
import org.apache.cassandra.db.marshal.AsciiType;
|
||||
import org.apache.cassandra.db.marshal.UTF8Type;
|
||||
import org.apache.cassandra.db.memtable.Memtable;
|
||||
import org.apache.cassandra.db.rows.Cell;
|
||||
import org.apache.cassandra.db.rows.Row;
|
||||
import org.apache.cassandra.index.sasi.analyzer.AbstractAnalyzer;
|
||||
|
|
@ -47,7 +46,9 @@ import org.apache.cassandra.index.sasi.plan.Expression.Op;
|
|||
import org.apache.cassandra.index.sasi.utils.RangeIterator;
|
||||
import org.apache.cassandra.index.sasi.utils.RangeUnionIterator;
|
||||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat.Components;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.schema.ColumnMetadata;
|
||||
import org.apache.cassandra.schema.IndexMetadata;
|
||||
import org.apache.cassandra.utils.FBUtilities;
|
||||
|
||||
|
|
@ -78,7 +79,7 @@ public class ColumnIndex
|
|||
this.mode = IndexMode.getMode(column, config);
|
||||
this.memtable = new AtomicReference<>(new IndexMemtable(this));
|
||||
this.tracker = new DataTracker(keyValidator, this);
|
||||
this.component = new Component(Component.Type.SECONDARY_INDEX, String.format(FILE_NAME_FORMAT, getIndexName()));
|
||||
this.component = Components.Types.SECONDARY_INDEX.createComponent(String.format(FILE_NAME_FORMAT, getIndexName()));
|
||||
this.isTokenized = getAnalyzer().isTokenizing();
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -24,6 +24,9 @@ import java.util.Set;
|
|||
import java.util.concurrent.atomic.AtomicReference;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.apache.cassandra.db.marshal.AbstractType;
|
||||
import org.apache.cassandra.index.sasi.SSTableIndex;
|
||||
import org.apache.cassandra.index.sasi.conf.view.View;
|
||||
|
|
@ -31,9 +34,6 @@ import org.apache.cassandra.io.sstable.format.SSTableReader;
|
|||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.utils.Pair;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
/** a pared-down version of DataTracker and DT.View. need one for each index of each column family */
|
||||
public class DataTracker
|
||||
{
|
||||
|
|
@ -145,7 +145,7 @@ public class DataTracker
|
|||
if (sstable.isMarkedCompacted())
|
||||
continue;
|
||||
|
||||
File indexFile = new File(sstable.descriptor.filenameFor(columnIndex.getComponent()));
|
||||
File indexFile = sstable.descriptor.fileFor(columnIndex.getComponent());
|
||||
if (!indexFile.exists())
|
||||
continue;
|
||||
|
||||
|
|
|
|||
|
|
@ -17,21 +17,33 @@
|
|||
*/
|
||||
package org.apache.cassandra.index.sasi.disk;
|
||||
|
||||
import java.io.*;
|
||||
import java.io.Closeable;
|
||||
import java.io.IOException;
|
||||
import java.nio.ByteBuffer;
|
||||
import java.nio.channels.FileChannel;
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.Iterator;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.NavigableMap;
|
||||
import java.util.TreeMap;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import com.google.common.base.Function;
|
||||
import com.google.common.collect.Iterables;
|
||||
import com.google.common.collect.Iterators;
|
||||
import com.google.common.collect.PeekingIterator;
|
||||
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.marshal.AbstractType;
|
||||
import org.apache.cassandra.index.sasi.Term;
|
||||
import org.apache.cassandra.index.sasi.plan.Expression;
|
||||
import org.apache.cassandra.index.sasi.plan.Expression.Op;
|
||||
import org.apache.cassandra.index.sasi.utils.MappedBuffer;
|
||||
import org.apache.cassandra.index.sasi.utils.RangeUnionIterator;
|
||||
import org.apache.cassandra.index.sasi.utils.AbstractIterator;
|
||||
import org.apache.cassandra.index.sasi.utils.MappedBuffer;
|
||||
import org.apache.cassandra.index.sasi.utils.RangeIterator;
|
||||
import org.apache.cassandra.db.marshal.AbstractType;
|
||||
import org.apache.cassandra.index.sasi.utils.RangeUnionIterator;
|
||||
import org.apache.cassandra.io.FSReadError;
|
||||
import org.apache.cassandra.io.util.ChannelProxy;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
|
|
@ -40,11 +52,6 @@ import org.apache.cassandra.io.util.FileUtils;
|
|||
import org.apache.cassandra.utils.ByteBufferUtil;
|
||||
import org.apache.cassandra.utils.FBUtilities;
|
||||
|
||||
import com.google.common.base.Function;
|
||||
import com.google.common.collect.Iterables;
|
||||
import com.google.common.collect.Iterators;
|
||||
import com.google.common.collect.PeekingIterator;
|
||||
|
||||
import static org.apache.cassandra.index.sasi.disk.OnDiskBlock.SearchResult;
|
||||
|
||||
public class OnDiskIndex implements Iterable<OnDiskIndex.DataTerm>, Closeable
|
||||
|
|
@ -144,7 +151,7 @@ public class OnDiskIndex implements Iterable<OnDiskIndex.DataTerm>, Closeable
|
|||
|
||||
FileChannel channel = index.newReadChannel();
|
||||
indexSize = channel.size();
|
||||
indexFile = new MappedBuffer(new ChannelProxy(indexPath, channel));
|
||||
indexFile = new MappedBuffer(new ChannelProxy(index, channel));
|
||||
}
|
||||
catch (IOException e)
|
||||
{
|
||||
|
|
|
|||
|
|
@ -21,35 +21,37 @@ import java.nio.ByteBuffer;
|
|||
import java.util.HashSet;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.*;
|
||||
import java.util.concurrent.Callable;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Future;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import com.google.common.collect.Maps;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.concurrent.ExecutorPlus;
|
||||
import org.apache.cassandra.schema.ColumnMetadata;
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.compaction.OperationType;
|
||||
import org.apache.cassandra.db.marshal.AbstractType;
|
||||
import org.apache.cassandra.db.rows.Row;
|
||||
import org.apache.cassandra.db.rows.Unfiltered;
|
||||
import org.apache.cassandra.index.sasi.analyzer.AbstractAnalyzer;
|
||||
import org.apache.cassandra.index.sasi.conf.ColumnIndex;
|
||||
import org.apache.cassandra.index.sasi.utils.CombinedTermIterator;
|
||||
import org.apache.cassandra.index.sasi.utils.TypeUtil;
|
||||
import org.apache.cassandra.db.marshal.AbstractType;
|
||||
import org.apache.cassandra.io.FSError;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFlushObserver;
|
||||
import org.apache.cassandra.io.sstable.SSTableFlushObserver;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.io.util.FileUtils;
|
||||
import org.apache.cassandra.schema.ColumnMetadata;
|
||||
import org.apache.cassandra.utils.FBUtilities;
|
||||
import org.apache.cassandra.utils.Pair;
|
||||
import org.apache.cassandra.utils.concurrent.CountDownLatch;
|
||||
import org.apache.cassandra.utils.concurrent.ImmediateFuture;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import com.google.common.collect.Maps;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import static org.apache.cassandra.concurrent.ExecutorFactory.Global.executorFactory;
|
||||
import static org.apache.cassandra.utils.Clock.Global.nanoTime;
|
||||
import static org.apache.cassandra.utils.concurrent.CountDownLatch.newCountDownLatch;
|
||||
|
|
@ -98,15 +100,24 @@ public class PerSSTableIndexWriter implements SSTableFlushObserver
|
|||
indexes.put(entry.getKey(), newIndex(entry.getValue()));
|
||||
}
|
||||
|
||||
@Override
|
||||
public void begin()
|
||||
{}
|
||||
|
||||
public void startPartition(DecoratedKey key, long curPosition)
|
||||
@Override
|
||||
public void startPartition(DecoratedKey key, long keyPosition, long KeyPositionForSASI)
|
||||
{
|
||||
currentKey = key;
|
||||
currentKeyPosition = curPosition;
|
||||
currentKeyPosition = KeyPositionForSASI;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void staticRow(Row staticRow)
|
||||
{
|
||||
nextUnfilteredCluster(staticRow);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void nextUnfilteredCluster(Unfiltered unfiltered)
|
||||
{
|
||||
if (!unfiltered.isRow())
|
||||
|
|
@ -126,6 +137,7 @@ public class PerSSTableIndexWriter implements SSTableFlushObserver
|
|||
});
|
||||
}
|
||||
|
||||
@Override
|
||||
public void complete()
|
||||
{
|
||||
if (isComplete)
|
||||
|
|
@ -168,7 +180,7 @@ public class PerSSTableIndexWriter implements SSTableFlushObserver
|
|||
protected class Index
|
||||
{
|
||||
@VisibleForTesting
|
||||
protected final String outputFile;
|
||||
protected final File outputFile;
|
||||
|
||||
private final ColumnIndex columnIndex;
|
||||
private final AbstractAnalyzer analyzer;
|
||||
|
|
@ -183,7 +195,7 @@ public class PerSSTableIndexWriter implements SSTableFlushObserver
|
|||
public Index(ColumnIndex columnIndex)
|
||||
{
|
||||
this.columnIndex = columnIndex;
|
||||
this.outputFile = descriptor.filenameFor(columnIndex.getComponent());
|
||||
this.outputFile = descriptor.fileFor(columnIndex.getComponent());
|
||||
this.analyzer = columnIndex.getAnalyzer();
|
||||
this.segments = new HashSet<>();
|
||||
this.maxMemorySize = maxMemorySize(columnIndex);
|
||||
|
|
@ -244,15 +256,14 @@ public class PerSSTableIndexWriter implements SSTableFlushObserver
|
|||
final OnDiskIndexBuilder builder = currentBuilder;
|
||||
currentBuilder = newIndexBuilder();
|
||||
|
||||
final String segmentFile = filename(isFinal);
|
||||
final File segmentFile = file(isFinal);
|
||||
|
||||
return () -> {
|
||||
long start = nanoTime();
|
||||
|
||||
try
|
||||
{
|
||||
File index = new File(segmentFile);
|
||||
return builder.finish(index) ? new OnDiskIndex(index, columnIndex.getValidator(), null) : null;
|
||||
return builder.finish(segmentFile) ? new OnDiskIndex(segmentFile, columnIndex.getValidator(), null) : null;
|
||||
}
|
||||
catch (Exception | FSError e)
|
||||
{
|
||||
|
|
@ -310,13 +321,13 @@ public class PerSSTableIndexWriter implements SSTableFlushObserver
|
|||
|
||||
OnDiskIndexBuilder builder = newIndexBuilder();
|
||||
builder.finish(Pair.create(combinedMin, combinedMax),
|
||||
new File(outputFile),
|
||||
outputFile,
|
||||
new CombinedTermIterator(parts));
|
||||
}
|
||||
catch (Exception | FSError e)
|
||||
{
|
||||
logger.error("Failed to flush index {}.", outputFile, e);
|
||||
FileUtils.delete(outputFile);
|
||||
outputFile.tryDelete();
|
||||
}
|
||||
finally
|
||||
{
|
||||
|
|
@ -330,7 +341,7 @@ public class PerSSTableIndexWriter implements SSTableFlushObserver
|
|||
if (part != null)
|
||||
FileUtils.closeQuietly(part);
|
||||
|
||||
FileUtils.delete(outputFile + "_" + segment);
|
||||
outputFile.withSuffix("_" + segment).tryDelete();
|
||||
}
|
||||
|
||||
latch.decrement();
|
||||
|
|
@ -348,9 +359,9 @@ public class PerSSTableIndexWriter implements SSTableFlushObserver
|
|||
return new OnDiskIndexBuilder(keyValidator, columnIndex.getValidator(), columnIndex.getMode().mode);
|
||||
}
|
||||
|
||||
public String filename(boolean isFinal)
|
||||
public File file(boolean isFinal)
|
||||
{
|
||||
return outputFile + (isFinal ? "" : "_" + segmentNumber++);
|
||||
return isFinal ? outputFile : outputFile.withSuffix("_" + segmentNumber++);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,32 @@
|
|||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.cassandra.io;
|
||||
|
||||
import java.io.DataInput;
|
||||
import java.io.DataOutput;
|
||||
import java.io.IOException;
|
||||
|
||||
public interface IGenericSerializer<T, I extends DataInput, O extends DataOutput>
|
||||
{
|
||||
void serialize(T t, O out) throws IOException;
|
||||
|
||||
T deserialize(I in) throws IOException;
|
||||
|
||||
long serializedSize(T t);
|
||||
}
|
||||
|
|
@ -22,7 +22,7 @@ import java.io.IOException;
|
|||
import org.apache.cassandra.io.util.DataInputPlus;
|
||||
import org.apache.cassandra.io.util.DataOutputPlus;
|
||||
|
||||
public interface ISerializer<T>
|
||||
public interface ISerializer<T> extends IGenericSerializer<T, DataInputPlus, DataOutputPlus>
|
||||
{
|
||||
/**
|
||||
* Serialize the specified type into the specified DataOutput instance.
|
||||
|
|
@ -32,7 +32,8 @@ public interface ISerializer<T>
|
|||
* @param out DataOutput into which serialization needs to happen.
|
||||
* @throws java.io.IOException
|
||||
*/
|
||||
public void serialize(T t, DataOutputPlus out) throws IOException;
|
||||
@Override
|
||||
void serialize(T t, DataOutputPlus out) throws IOException;
|
||||
|
||||
/**
|
||||
* Deserialize from the specified DataInput instance.
|
||||
|
|
@ -40,11 +41,13 @@ public interface ISerializer<T>
|
|||
* @throws IOException
|
||||
* @return the type that was deserialized
|
||||
*/
|
||||
public T deserialize(DataInputPlus in) throws IOException;
|
||||
@Override
|
||||
T deserialize(DataInputPlus in) throws IOException;
|
||||
|
||||
public long serializedSize(T t);
|
||||
@Override
|
||||
long serializedSize(T t);
|
||||
|
||||
public default void skip(DataInputPlus in) throws IOException
|
||||
default void skip(DataInputPlus in) throws IOException
|
||||
{
|
||||
deserialize(in);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -29,7 +29,12 @@ import org.apache.cassandra.io.FSReadError;
|
|||
import org.apache.cassandra.io.FSWriteError;
|
||||
import org.apache.cassandra.io.sstable.CorruptSSTableException;
|
||||
import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
|
||||
import org.apache.cassandra.io.util.*;
|
||||
import org.apache.cassandra.io.util.ChecksumWriter;
|
||||
import org.apache.cassandra.io.util.DataPosition;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.io.util.FileUtils;
|
||||
import org.apache.cassandra.io.util.SequentialWriter;
|
||||
import org.apache.cassandra.io.util.SequentialWriterOption;
|
||||
import org.apache.cassandra.schema.CompressionParams;
|
||||
import org.apache.cassandra.utils.ByteBufferUtil;
|
||||
|
||||
|
|
@ -66,14 +71,14 @@ public class CompressedSequentialWriter extends SequentialWriter
|
|||
* Create CompressedSequentialWriter without digest file.
|
||||
*
|
||||
* @param file File to write
|
||||
* @param offsetsPath File name to write compression metadata
|
||||
* @param offsetsFile File to write compression metadata
|
||||
* @param digestFile File to write digest
|
||||
* @param option Write option (buffer size and type will be set the same as compression params)
|
||||
* @param parameters Compression mparameters
|
||||
* @param sstableMetadataCollector Metadata collector
|
||||
*/
|
||||
public CompressedSequentialWriter(File file,
|
||||
String offsetsPath,
|
||||
File offsetsFile,
|
||||
File digestFile,
|
||||
SequentialWriterOption option,
|
||||
CompressionParams parameters,
|
||||
|
|
@ -95,7 +100,7 @@ public class CompressedSequentialWriter extends SequentialWriter
|
|||
maxCompressedLength = parameters.maxCompressedLength();
|
||||
|
||||
/* Index File (-CompressionInfo.db component) and it's header */
|
||||
metadataWriter = CompressionMetadata.Writer.open(parameters, offsetsPath);
|
||||
metadataWriter = CompressionMetadata.Writer.open(parameters, offsetsFile);
|
||||
|
||||
this.sstableMetadataCollector = sstableMetadataCollector;
|
||||
crcMetadata = new ChecksumWriter(new DataOutputStream(Channels.newOutputStream(channel)));
|
||||
|
|
@ -397,4 +402,4 @@ public class CompressedSequentialWriter extends SequentialWriter
|
|||
this.nextChunkIndex = nextChunkIndex;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -17,14 +17,11 @@
|
|||
*/
|
||||
package org.apache.cassandra.io.compress;
|
||||
|
||||
import java.nio.file.NoSuchFileException;
|
||||
import java.io.DataInput;
|
||||
import java.io.DataOutput;
|
||||
import java.io.EOFException;
|
||||
|
||||
import org.apache.cassandra.io.util.*;
|
||||
import java.io.FileNotFoundException;
|
||||
import java.io.IOException;
|
||||
import java.nio.file.NoSuchFileException;
|
||||
import java.util.Collection;
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
|
|
@ -39,18 +36,25 @@ import org.apache.cassandra.exceptions.ConfigurationException;
|
|||
import org.apache.cassandra.io.FSReadError;
|
||||
import org.apache.cassandra.io.FSWriteError;
|
||||
import org.apache.cassandra.io.IVersionedSerializer;
|
||||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.CorruptSSTableException;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.util.DataInputPlus;
|
||||
import org.apache.cassandra.io.util.DataOutputPlus;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.io.util.FileInputStreamPlus;
|
||||
import org.apache.cassandra.io.util.FileOutputStreamPlus;
|
||||
import org.apache.cassandra.io.util.Memory;
|
||||
import org.apache.cassandra.io.util.SafeMemory;
|
||||
import org.apache.cassandra.schema.CompressionParams;
|
||||
import org.apache.cassandra.utils.concurrent.Transactional;
|
||||
import org.apache.cassandra.utils.concurrent.Ref;
|
||||
import org.apache.cassandra.utils.concurrent.Transactional;
|
||||
import org.apache.cassandra.utils.concurrent.WrappedSharedCloseable;
|
||||
|
||||
/**
|
||||
* Holds metadata about compressed file
|
||||
* TODO extract interface ICompressionMetadata which will just provide non-resource properties
|
||||
*/
|
||||
public class CompressionMetadata
|
||||
public class CompressionMetadata extends WrappedSharedCloseable
|
||||
{
|
||||
// dataLength can represent either the true length of the file
|
||||
// or some shorter value, in the case we want to impose a shorter limit on readers
|
||||
|
|
@ -59,42 +63,18 @@ public class CompressionMetadata
|
|||
public final long compressedFileLength;
|
||||
private final Memory chunkOffsets;
|
||||
private final long chunkOffsetsSize;
|
||||
public final String indexFilePath;
|
||||
public final File chunksIndexFile;
|
||||
public final CompressionParams parameters;
|
||||
|
||||
/**
|
||||
* Create metadata about given compressed file including uncompressed data length, chunk size
|
||||
* and list of the chunk offsets of the compressed data.
|
||||
*
|
||||
* This is an expensive operation! Don't create more than one for each
|
||||
* sstable.
|
||||
*
|
||||
* @param dataFilePath Path to the compressed file
|
||||
*
|
||||
* @return metadata about given compressed file.
|
||||
*/
|
||||
public static CompressionMetadata create(String dataFilePath)
|
||||
{
|
||||
return createWithLength(dataFilePath, new File(dataFilePath).length());
|
||||
}
|
||||
|
||||
public static CompressionMetadata createWithLength(String dataFilePath, long compressedLength)
|
||||
{
|
||||
return new CompressionMetadata(Descriptor.fromFilename(dataFilePath), compressedLength);
|
||||
}
|
||||
|
||||
@VisibleForTesting
|
||||
public CompressionMetadata(Descriptor desc, long compressedLength)
|
||||
@SuppressWarnings("resource")
|
||||
public static CompressionMetadata open(File chunksIndexFile, long compressedLength, boolean hasMaxCompressedSize)
|
||||
{
|
||||
this(desc.filenameFor(Component.COMPRESSION_INFO), compressedLength, desc.version.hasMaxCompressedLength());
|
||||
}
|
||||
CompressionParams parameters;
|
||||
long dataLength;
|
||||
Memory chunkOffsets;
|
||||
|
||||
@VisibleForTesting
|
||||
public CompressionMetadata(String indexFilePath, long compressedLength, boolean hasMaxCompressedSize)
|
||||
{
|
||||
this.indexFilePath = indexFilePath;
|
||||
|
||||
try (FileInputStreamPlus stream = new File(indexFilePath).newInputStream())
|
||||
try (FileInputStreamPlus stream = chunksIndexFile.newInputStream())
|
||||
{
|
||||
String compressorName = stream.readUTF();
|
||||
int optionCount = stream.readInt();
|
||||
|
|
@ -119,7 +99,6 @@ public class CompressionMetadata
|
|||
}
|
||||
|
||||
dataLength = stream.readLong();
|
||||
compressedFileLength = compressedLength;
|
||||
chunkOffsets = readChunkOffsets(stream);
|
||||
}
|
||||
catch (FileNotFoundException | NoSuchFileException e)
|
||||
|
|
@ -128,22 +107,39 @@ public class CompressionMetadata
|
|||
}
|
||||
catch (IOException e)
|
||||
{
|
||||
throw new CorruptSSTableException(e, indexFilePath);
|
||||
throw new CorruptSSTableException(e, chunksIndexFile);
|
||||
}
|
||||
|
||||
this.chunkOffsetsSize = chunkOffsets.size();
|
||||
return new CompressionMetadata(chunksIndexFile, parameters, chunkOffsets, chunkOffsets.size(), dataLength, compressedLength);
|
||||
}
|
||||
|
||||
// do not call this constructor directly, unless used in testing
|
||||
@VisibleForTesting
|
||||
public CompressionMetadata(String filePath, CompressionParams parameters, Memory offsets, long offsetsSize, long dataLength, long compressedLength)
|
||||
public CompressionMetadata(File chunksIndexFile,
|
||||
CompressionParams parameters,
|
||||
Memory chunkOffsets,
|
||||
long chunkOffsetsSize,
|
||||
long dataLength,
|
||||
long compressedFileLength)
|
||||
{
|
||||
this.indexFilePath = filePath;
|
||||
super(chunkOffsets);
|
||||
this.chunksIndexFile = chunksIndexFile;
|
||||
this.parameters = parameters;
|
||||
this.dataLength = dataLength;
|
||||
this.compressedFileLength = compressedLength;
|
||||
this.chunkOffsets = offsets;
|
||||
this.chunkOffsetsSize = offsetsSize;
|
||||
this.compressedFileLength = compressedFileLength;
|
||||
this.chunkOffsets = chunkOffsets;
|
||||
this.chunkOffsetsSize = chunkOffsetsSize;
|
||||
}
|
||||
|
||||
private CompressionMetadata(CompressionMetadata copy)
|
||||
{
|
||||
super(copy);
|
||||
this.chunksIndexFile = copy.chunksIndexFile;
|
||||
this.parameters = copy.parameters;
|
||||
this.dataLength = copy.dataLength;
|
||||
this.compressedFileLength = copy.compressedFileLength;
|
||||
this.chunkOffsets = copy.chunkOffsets;
|
||||
this.chunkOffsetsSize = copy.chunkOffsetsSize;
|
||||
}
|
||||
|
||||
public ICompressor compressor()
|
||||
|
|
@ -170,11 +166,19 @@ public class CompressionMetadata
|
|||
return chunkOffsets.size();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void addTo(Ref.IdentityCollection identities)
|
||||
{
|
||||
super.addTo(identities);
|
||||
identities.add(chunkOffsets);
|
||||
}
|
||||
|
||||
@Override
|
||||
public CompressionMetadata sharedCopy()
|
||||
{
|
||||
return new CompressionMetadata(this);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read offsets of the individual chunks from the given input.
|
||||
*
|
||||
|
|
@ -182,7 +186,7 @@ public class CompressionMetadata
|
|||
*
|
||||
* @return collection of the chunk offsets.
|
||||
*/
|
||||
private Memory readChunkOffsets(DataInput input)
|
||||
private static Memory readChunkOffsets(FileInputStreamPlus input)
|
||||
{
|
||||
final int chunkCount;
|
||||
try
|
||||
|
|
@ -193,7 +197,7 @@ public class CompressionMetadata
|
|||
}
|
||||
catch (IOException e)
|
||||
{
|
||||
throw new FSReadError(e, indexFilePath);
|
||||
throw new FSReadError(e, input.file);
|
||||
}
|
||||
|
||||
@SuppressWarnings("resource")
|
||||
|
|
@ -217,10 +221,10 @@ public class CompressionMetadata
|
|||
if (e instanceof EOFException)
|
||||
{
|
||||
String msg = String.format("Corrupted Index File %s: read %d but expected %d chunks.",
|
||||
indexFilePath, i, chunkCount);
|
||||
throw new CorruptSSTableException(new IOException(msg, e), indexFilePath);
|
||||
input.file.path(), i, chunkCount);
|
||||
throw new CorruptSSTableException(new IOException(msg, e), input.file);
|
||||
}
|
||||
throw new FSReadError(e, indexFilePath);
|
||||
throw new FSReadError(e, input.file);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -236,11 +240,11 @@ public class CompressionMetadata
|
|||
long idx = 8 * (position / parameters.chunkLength());
|
||||
|
||||
if (idx >= chunkOffsetsSize)
|
||||
throw new CorruptSSTableException(new EOFException(), indexFilePath);
|
||||
throw new CorruptSSTableException(new EOFException(), chunksIndexFile);
|
||||
|
||||
if (idx < 0)
|
||||
throw new CorruptSSTableException(new IllegalArgumentException(String.format("Invalid negative chunk index %d with position %d", idx, position)),
|
||||
indexFilePath);
|
||||
chunksIndexFile);
|
||||
|
||||
long chunkOffset = chunkOffsets.getLong(idx);
|
||||
long nextChunkOffset = (idx + 8 == chunkOffsetsSize)
|
||||
|
|
@ -250,6 +254,28 @@ public class CompressionMetadata
|
|||
return new Chunk(chunkOffset, (int) (nextChunkOffset - chunkOffset - 4)); // "4" bytes reserved for checksum
|
||||
}
|
||||
|
||||
public long getDataOffsetForChunkOffset(long chunkOffset)
|
||||
{
|
||||
long l = 0;
|
||||
long h = (chunkOffsetsSize >> 3) - 1;
|
||||
long idx, offset;
|
||||
|
||||
while (l <= h)
|
||||
{
|
||||
idx = (l + h) >>> 1;
|
||||
offset = chunkOffsets.getLong(idx << 3);
|
||||
|
||||
if (offset < chunkOffset)
|
||||
l = idx + 1;
|
||||
else if (offset > chunkOffset)
|
||||
h = idx - 1;
|
||||
else
|
||||
return idx * parameters.chunkLength();
|
||||
}
|
||||
|
||||
throw new IllegalArgumentException("No chunk with offset " + chunkOffset);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param sections Collection of sections in uncompressed file. Should not contain sections that overlap each other.
|
||||
* @return Total chunk size in bytes for given sections including checksum.
|
||||
|
|
@ -314,16 +340,11 @@ public class CompressionMetadata
|
|||
return offsets.toArray(new Chunk[offsets.size()]);
|
||||
}
|
||||
|
||||
public void close()
|
||||
{
|
||||
chunkOffsets.close();
|
||||
}
|
||||
|
||||
public static class Writer extends Transactional.AbstractTransactional implements Transactional
|
||||
{
|
||||
// path to the file
|
||||
private final CompressionParams parameters;
|
||||
private final String filePath;
|
||||
private final File file;
|
||||
private int maxCount = 100;
|
||||
private SafeMemory offsets = new SafeMemory(maxCount * 8L);
|
||||
private int count = 0;
|
||||
|
|
@ -331,15 +352,15 @@ public class CompressionMetadata
|
|||
// provided by user when setDescriptor
|
||||
private long dataLength, chunkCount;
|
||||
|
||||
private Writer(CompressionParams parameters, String path)
|
||||
private Writer(CompressionParams parameters, File file)
|
||||
{
|
||||
this.parameters = parameters;
|
||||
filePath = path;
|
||||
this.file = file;
|
||||
}
|
||||
|
||||
public static Writer open(CompressionParams parameters, String path)
|
||||
public static Writer open(CompressionParams parameters, File file)
|
||||
{
|
||||
return new Writer(parameters, path);
|
||||
return new Writer(parameters, file);
|
||||
}
|
||||
|
||||
public void addOffset(long offset)
|
||||
|
|
@ -374,7 +395,7 @@ public class CompressionMetadata
|
|||
}
|
||||
catch (IOException e)
|
||||
{
|
||||
throw new FSWriteError(e, filePath);
|
||||
throw new FSWriteError(e, file);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -386,6 +407,7 @@ public class CompressionMetadata
|
|||
return this;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void doPrepare()
|
||||
{
|
||||
assert chunkCount == count;
|
||||
|
|
@ -400,7 +422,7 @@ public class CompressionMetadata
|
|||
}
|
||||
|
||||
// flush the data to disk
|
||||
try (FileOutputStreamPlus out = new FileOutputStreamPlus(filePath))
|
||||
try (FileOutputStreamPlus out = file.newOutputStream(File.WriteMode.OVERWRITE))
|
||||
{
|
||||
writeHeader(out, dataLength, count);
|
||||
for (int i = 0; i < count; i++)
|
||||
|
|
@ -415,7 +437,7 @@ public class CompressionMetadata
|
|||
}
|
||||
catch (IOException e)
|
||||
{
|
||||
throw new FSWriteError(e, filePath);
|
||||
throw new FSWriteError(e, file);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -434,7 +456,7 @@ public class CompressionMetadata
|
|||
if (tCount < this.count)
|
||||
compressedLength = tOffsets.getLong(tCount * 8L);
|
||||
|
||||
return new CompressionMetadata(filePath, parameters, tOffsets, tCount * 8L, dataLength, compressedLength);
|
||||
return new CompressionMetadata(file, parameters, tOffsets, tCount * 8L, dataLength, compressedLength);
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -460,16 +482,19 @@ public class CompressionMetadata
|
|||
count = chunkIndex;
|
||||
}
|
||||
|
||||
@Override
|
||||
protected Throwable doPostCleanup(Throwable failed)
|
||||
{
|
||||
return offsets.close(failed);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected Throwable doCommit(Throwable accumulate)
|
||||
{
|
||||
return accumulate;
|
||||
}
|
||||
|
||||
@Override
|
||||
protected Throwable doAbort(Throwable accumulate)
|
||||
{
|
||||
return accumulate;
|
||||
|
|
@ -494,6 +519,7 @@ public class CompressionMetadata
|
|||
this.length = length;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean equals(Object o)
|
||||
{
|
||||
if (this == o) return true;
|
||||
|
|
@ -503,6 +529,7 @@ public class CompressionMetadata
|
|||
return length == chunk.length && offset == chunk.offset;
|
||||
}
|
||||
|
||||
@Override
|
||||
public int hashCode()
|
||||
{
|
||||
int result = (int) (offset ^ (offset >>> 32));
|
||||
|
|
@ -510,6 +537,7 @@ public class CompressionMetadata
|
|||
return result;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString()
|
||||
{
|
||||
return String.format("Chunk<offset: %d, length: %d>", offset, length);
|
||||
|
|
@ -518,17 +546,20 @@ public class CompressionMetadata
|
|||
|
||||
static class ChunkSerializer implements IVersionedSerializer<Chunk>
|
||||
{
|
||||
@Override
|
||||
public void serialize(Chunk chunk, DataOutputPlus out, int version) throws IOException
|
||||
{
|
||||
out.writeLong(chunk.offset);
|
||||
out.writeInt(chunk.length);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Chunk deserialize(DataInputPlus in, int version) throws IOException
|
||||
{
|
||||
return new Chunk(in.readLong(), in.readInt());
|
||||
}
|
||||
|
||||
@Override
|
||||
public long serializedSize(Chunk chunk, int version)
|
||||
{
|
||||
long size = TypeSizes.sizeof(chunk.offset);
|
||||
|
|
|
|||
|
|
@ -0,0 +1,44 @@
|
|||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import java.util.function.BiFunction;
|
||||
import java.util.function.Function;
|
||||
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
|
||||
public abstract class AbstractMetricsProviders<R extends SSTableReader> implements MetricsProviders
|
||||
{
|
||||
protected final <T extends Number> GaugeProvider<T> newGaugeProvider(String name, Function<Iterable<R>, T> combiner)
|
||||
{
|
||||
return new SimpleGaugeProvider<>(this::map, name, combiner);
|
||||
}
|
||||
|
||||
protected final <T extends Number> GaugeProvider<T> newGaugeProvider(String name, T neutralValue, Function<R, T> extractor, BiFunction<T, T, T> combiner)
|
||||
{
|
||||
return new SimpleGaugeProvider<>(this::map, name, readers -> {
|
||||
T total = neutralValue;
|
||||
for (R reader : readers)
|
||||
total = combiner.apply(total, extractor.apply(reader));
|
||||
return total;
|
||||
});
|
||||
}
|
||||
|
||||
protected abstract R map(SSTableReader r);
|
||||
}
|
||||
|
|
@ -0,0 +1,75 @@
|
|||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import java.io.IOException;
|
||||
|
||||
import org.apache.cassandra.cache.IMeasurableMemory;
|
||||
import org.apache.cassandra.db.DeletionTime;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat;
|
||||
import org.apache.cassandra.io.util.DataOutputPlus;
|
||||
|
||||
/**
|
||||
* The base RowIndexEntry is not stored on disk, only specifies a position in the data file
|
||||
*/
|
||||
public abstract class AbstractRowIndexEntry implements IMeasurableMemory
|
||||
{
|
||||
public final long position;
|
||||
|
||||
public AbstractRowIndexEntry(long position)
|
||||
{
|
||||
this.position = position;
|
||||
}
|
||||
|
||||
/**
|
||||
* Row position in a data file
|
||||
*/
|
||||
public long getPosition()
|
||||
{
|
||||
return position;
|
||||
}
|
||||
|
||||
/**
|
||||
* @return true if this index entry contains the row-level tombstone and column summary. Otherwise,
|
||||
* caller should fetch these from the row header.
|
||||
*/
|
||||
public boolean isIndexed()
|
||||
{
|
||||
return columnsIndexCount() > 1;
|
||||
}
|
||||
|
||||
public DeletionTime deletionTime()
|
||||
{
|
||||
throw new UnsupportedOperationException();
|
||||
}
|
||||
|
||||
public int columnsIndexCount()
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
public abstract SSTableFormat<?, ?> getSSTableFormat();
|
||||
|
||||
/**
|
||||
* Serialize this entry for key cache
|
||||
*
|
||||
* @param out the output stream for serialized entry
|
||||
*/
|
||||
public abstract void serializeForCache(DataOutputPlus out) throws IOException;
|
||||
}
|
||||
|
|
@ -15,20 +15,34 @@
|
|||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.apache.cassandra.db.columniterator;
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import java.io.Closeable;
|
||||
import java.io.IOException;
|
||||
import java.util.Comparator;
|
||||
import java.util.Iterator;
|
||||
import java.util.NoSuchElementException;
|
||||
|
||||
import org.apache.cassandra.db.*;
|
||||
import org.apache.cassandra.db.BufferClusteringBound;
|
||||
import org.apache.cassandra.db.ClusteringBound;
|
||||
import org.apache.cassandra.db.Columns;
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.DeletionTime;
|
||||
import org.apache.cassandra.db.RegularAndStaticColumns;
|
||||
import org.apache.cassandra.db.Slice;
|
||||
import org.apache.cassandra.db.Slices;
|
||||
import org.apache.cassandra.db.UnfilteredDeserializer;
|
||||
import org.apache.cassandra.db.UnfilteredValidation;
|
||||
import org.apache.cassandra.db.filter.ColumnFilter;
|
||||
import org.apache.cassandra.db.rows.*;
|
||||
import org.apache.cassandra.io.sstable.CorruptSSTableException;
|
||||
import org.apache.cassandra.io.sstable.IndexInfo;
|
||||
import org.apache.cassandra.db.rows.DeserializationHelper;
|
||||
import org.apache.cassandra.db.rows.EncodingStats;
|
||||
import org.apache.cassandra.db.rows.RangeTombstoneBoundMarker;
|
||||
import org.apache.cassandra.db.rows.RangeTombstoneMarker;
|
||||
import org.apache.cassandra.db.rows.Row;
|
||||
import org.apache.cassandra.db.rows.Rows;
|
||||
import org.apache.cassandra.db.rows.Unfiltered;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.db.rows.UnfilteredSerializer;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.util.DataPosition;
|
||||
import org.apache.cassandra.io.util.FileDataInput;
|
||||
import org.apache.cassandra.io.util.FileHandle;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
|
|
@ -37,7 +51,7 @@ import org.apache.cassandra.utils.ByteBufferUtil;
|
|||
import static org.apache.cassandra.utils.vint.VIntCoding.VIntOutOfRangeException;
|
||||
|
||||
|
||||
public abstract class AbstractSSTableIterator implements UnfilteredRowIterator
|
||||
public abstract class AbstractSSTableIterator<RIE extends AbstractRowIndexEntry> implements UnfilteredRowIterator
|
||||
{
|
||||
protected final SSTableReader sstable;
|
||||
// We could use sstable.metadata(), but that can change during execution so it's good hygiene to grab an immutable instance
|
||||
|
|
@ -62,7 +76,7 @@ public abstract class AbstractSSTableIterator implements UnfilteredRowIterator
|
|||
protected AbstractSSTableIterator(SSTableReader sstable,
|
||||
FileDataInput file,
|
||||
DecoratedKey key,
|
||||
RowIndexEntry indexEntry,
|
||||
RIE indexEntry,
|
||||
Slices slices,
|
||||
ColumnFilter columnFilter,
|
||||
FileHandle ifile)
|
||||
|
|
@ -179,9 +193,9 @@ public abstract class AbstractSSTableIterator implements UnfilteredRowIterator
|
|||
}
|
||||
}
|
||||
|
||||
protected abstract Reader createReaderInternal(RowIndexEntry indexEntry, FileDataInput file, boolean shouldCloseFile);
|
||||
protected abstract Reader createReaderInternal(RIE indexEntry, FileDataInput file, boolean shouldCloseFile);
|
||||
|
||||
private Reader createReader(RowIndexEntry indexEntry, FileDataInput file, boolean shouldCloseFile)
|
||||
private Reader createReader(RIE indexEntry, FileDataInput file, boolean shouldCloseFile)
|
||||
{
|
||||
return slices.isEmpty() ? new NoRowsReader(file, shouldCloseFile)
|
||||
: createReaderInternal(indexEntry, file, shouldCloseFile);
|
||||
|
|
@ -258,7 +272,7 @@ public abstract class AbstractSSTableIterator implements UnfilteredRowIterator
|
|||
e.addSuppressed(suppressed);
|
||||
}
|
||||
sstable.markSuspect();
|
||||
throw new CorruptSSTableException(e, reader.file.getPath());
|
||||
throw new CorruptSSTableException(e, reader.toString());
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -289,21 +303,28 @@ public abstract class AbstractSSTableIterator implements UnfilteredRowIterator
|
|||
catch (IOException e)
|
||||
{
|
||||
sstable.markSuspect();
|
||||
throw new CorruptSSTableException(e, reader.file.getPath());
|
||||
throw new CorruptSSTableException(e, reader.toString());
|
||||
}
|
||||
}
|
||||
|
||||
protected abstract class Reader implements Iterator<Unfiltered>
|
||||
public interface Reader extends Iterator<Unfiltered>, Closeable {
|
||||
|
||||
void setForSlice(Slice slice) throws IOException;
|
||||
|
||||
void seekToPosition(long columnOffset) throws IOException;
|
||||
}
|
||||
|
||||
public abstract class AbstractReader implements Reader
|
||||
{
|
||||
private final boolean shouldCloseFile;
|
||||
public FileDataInput file;
|
||||
|
||||
protected UnfilteredDeserializer deserializer;
|
||||
public UnfilteredDeserializer deserializer;
|
||||
|
||||
// Records the currently open range tombstone (if any)
|
||||
protected DeletionTime openMarker = null;
|
||||
public DeletionTime openMarker;
|
||||
|
||||
protected Reader(FileDataInput file, boolean shouldCloseFile)
|
||||
protected AbstractReader(FileDataInput file, boolean shouldCloseFile)
|
||||
{
|
||||
this.file = file;
|
||||
this.shouldCloseFile = shouldCloseFile;
|
||||
|
|
@ -318,7 +339,7 @@ public abstract class AbstractSSTableIterator implements UnfilteredRowIterator
|
|||
deserializer = UnfilteredDeserializer.create(metadata, file, sstable.header, helper);
|
||||
}
|
||||
|
||||
protected void seekToPosition(long position) throws IOException
|
||||
public void seekToPosition(long position) throws IOException
|
||||
{
|
||||
// This may be the first time we're actually looking into the file
|
||||
if (file == null)
|
||||
|
|
@ -329,6 +350,7 @@ public abstract class AbstractSSTableIterator implements UnfilteredRowIterator
|
|||
else
|
||||
{
|
||||
file.seek(position);
|
||||
deserializer.clearState();
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -355,7 +377,7 @@ public abstract class AbstractSSTableIterator implements UnfilteredRowIterator
|
|||
e.addSuppressed(suppressed);
|
||||
}
|
||||
sstable.markSuspect();
|
||||
throw new CorruptSSTableException(e, reader.file.getPath());
|
||||
throw new CorruptSSTableException(e, toString());
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -376,7 +398,7 @@ public abstract class AbstractSSTableIterator implements UnfilteredRowIterator
|
|||
e.addSuppressed(suppressed);
|
||||
}
|
||||
sstable.markSuspect();
|
||||
throw new CorruptSSTableException(e, reader.file.getPath());
|
||||
throw new CorruptSSTableException(e, toString());
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -386,229 +408,178 @@ public abstract class AbstractSSTableIterator implements UnfilteredRowIterator
|
|||
protected abstract boolean hasNextInternal() throws IOException;
|
||||
protected abstract Unfiltered nextInternal() throws IOException;
|
||||
|
||||
@Override
|
||||
public void close() throws IOException
|
||||
{
|
||||
if (shouldCloseFile && file != null)
|
||||
file.close();
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString()
|
||||
{
|
||||
return file != null ? file.toString() : "null";
|
||||
}
|
||||
}
|
||||
|
||||
// Reader for when we have Slices.NONE but need to read static row or partition level deletion
|
||||
private class NoRowsReader extends AbstractSSTableIterator.Reader
|
||||
protected class ForwardReader extends AbstractReader
|
||||
{
|
||||
private NoRowsReader(FileDataInput file, boolean shouldCloseFile)
|
||||
// The start of the current slice. This will be null as soon as we know we've passed that bound.
|
||||
protected ClusteringBound<?> start;
|
||||
// The end of the current slice. Will never be null.
|
||||
protected ClusteringBound<?> end = BufferClusteringBound.TOP;
|
||||
|
||||
protected Unfiltered next; // the next element to return: this is computed by hasNextInternal().
|
||||
|
||||
protected boolean sliceDone; // set to true once we know we have no more result for the slice. This is in particular
|
||||
// used by the indexed reader when we know we can't have results based on the index.
|
||||
|
||||
public ForwardReader(FileDataInput file, boolean shouldCloseFile)
|
||||
{
|
||||
super(file, shouldCloseFile);
|
||||
}
|
||||
|
||||
public void setForSlice(Slice slice) throws IOException
|
||||
{
|
||||
return;
|
||||
start = slice.start().isBottom() ? null : slice.start();
|
||||
end = slice.end();
|
||||
|
||||
sliceDone = false;
|
||||
next = null;
|
||||
}
|
||||
|
||||
// Skip all data that comes before the currently set slice.
|
||||
// Return what should be returned at the end of this, or null if nothing should.
|
||||
private Unfiltered handlePreSliceData() throws IOException
|
||||
{
|
||||
assert deserializer != null;
|
||||
|
||||
// Note that the following comparison is not strict. The reason is that the only cases
|
||||
// where it can be == is if the "next" is a RT start marker (either a '[' of a ')[' boundary),
|
||||
// and if we had a strict inequality and an open RT marker before this, we would issue
|
||||
// the open marker first, and then return then next later, which would send in the
|
||||
// stream both '[' (or '(') and then ')[' for the same clustering value, which is wrong.
|
||||
// By using a non-strict inequality, we avoid that problem (if we do get ')[' for the same
|
||||
// clustering value than the slice, we'll simply record it in 'openMarker').
|
||||
while (deserializer.hasNext() && deserializer.compareNextTo(start) <= 0)
|
||||
{
|
||||
if (deserializer.nextIsRow())
|
||||
deserializer.skipNext();
|
||||
else
|
||||
updateOpenMarker((RangeTombstoneMarker)deserializer.readNext());
|
||||
}
|
||||
|
||||
ClusteringBound<?> sliceStart = start;
|
||||
start = null;
|
||||
|
||||
// We've reached the beginning of our queried slice. If we have an open marker
|
||||
// we should return that first.
|
||||
if (openMarker != null)
|
||||
return new RangeTombstoneBoundMarker(sliceStart, openMarker);
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
// Compute the next element to return, assuming we're in the middle to the slice
|
||||
// and the next element is either in the slice, or just after it. Returns null
|
||||
// if we're done with the slice.
|
||||
protected Unfiltered computeNext() throws IOException
|
||||
{
|
||||
assert deserializer != null;
|
||||
|
||||
while (true)
|
||||
{
|
||||
// We use a same reasoning as in handlePreSliceData regarding the strictness of the inequality below.
|
||||
// We want to exclude deserialized unfiltered equal to end, because 1) we won't miss any rows since those
|
||||
// woudn't be equal to a slice bound and 2) a end bound can be equal to a start bound
|
||||
// (EXCL_END(x) == INCL_START(x) for instance) and in that case we don't want to return start bound because
|
||||
// it's fundamentally excluded. And if the bound is a end (for a range tombstone), it means it's exactly
|
||||
// our slice end, but in that case we will properly close the range tombstone anyway as part of our "close
|
||||
// an open marker" code in hasNextInterna
|
||||
if (!deserializer.hasNext() || deserializer.compareNextTo(end) >= 0)
|
||||
return null;
|
||||
|
||||
Unfiltered next = deserializer.readNext();
|
||||
UnfilteredValidation.maybeValidateUnfiltered(next, metadata(), key, sstable);
|
||||
// We may get empty row for the same reason expressed on UnfilteredSerializer.deserializeOne.
|
||||
if (next.isEmpty())
|
||||
continue;
|
||||
|
||||
if (next.kind() == Unfiltered.Kind.RANGE_TOMBSTONE_MARKER)
|
||||
updateOpenMarker((RangeTombstoneMarker) next);
|
||||
return next;
|
||||
}
|
||||
}
|
||||
|
||||
protected boolean hasNextInternal() throws IOException
|
||||
{
|
||||
if (next != null)
|
||||
return true;
|
||||
|
||||
if (sliceDone)
|
||||
return false;
|
||||
|
||||
if (start != null)
|
||||
{
|
||||
Unfiltered unfiltered = handlePreSliceData();
|
||||
if (unfiltered != null)
|
||||
{
|
||||
next = unfiltered;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
next = computeNext();
|
||||
if (next != null)
|
||||
return true;
|
||||
|
||||
// for current slice, no data read from deserialization
|
||||
sliceDone = true;
|
||||
// If we have an open marker, we should not close it, there could be more slices
|
||||
if (openMarker != null)
|
||||
{
|
||||
next = new RangeTombstoneBoundMarker(end, openMarker);
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
protected Unfiltered nextInternal() throws IOException
|
||||
{
|
||||
if (!hasNextInternal())
|
||||
throw new NoSuchElementException();
|
||||
|
||||
Unfiltered toReturn = next;
|
||||
next = null;
|
||||
return toReturn;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// Reader for when we have Slices.NONE but need to read static row or partition level deletion
|
||||
private class NoRowsReader extends AbstractReader
|
||||
{
|
||||
private NoRowsReader(FileDataInput file, boolean shouldCloseFile)
|
||||
{
|
||||
super(file, shouldCloseFile);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setForSlice(Slice slice)
|
||||
{
|
||||
// no-op
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean hasNextInternal()
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
@Override
|
||||
protected Unfiltered nextInternal() throws IOException
|
||||
{
|
||||
throw new NoSuchElementException();
|
||||
}
|
||||
}
|
||||
|
||||
// Used by indexed readers to store where they are of the index.
|
||||
public static class IndexState implements AutoCloseable
|
||||
{
|
||||
private final Reader reader;
|
||||
private final ClusteringComparator comparator;
|
||||
|
||||
private final RowIndexEntry indexEntry;
|
||||
private final RowIndexEntry.IndexInfoRetriever indexInfoRetriever;
|
||||
private final boolean reversed;
|
||||
|
||||
private int currentIndexIdx;
|
||||
|
||||
// Marks the beginning of the block corresponding to currentIndexIdx.
|
||||
private DataPosition mark;
|
||||
|
||||
public IndexState(Reader reader, ClusteringComparator comparator, RowIndexEntry indexEntry, boolean reversed, FileHandle indexFile)
|
||||
{
|
||||
this.reader = reader;
|
||||
this.comparator = comparator;
|
||||
this.indexEntry = indexEntry;
|
||||
this.indexInfoRetriever = indexEntry.openWithIndex(indexFile);
|
||||
this.reversed = reversed;
|
||||
this.currentIndexIdx = reversed ? indexEntry.columnsIndexCount() : -1;
|
||||
}
|
||||
|
||||
public boolean isDone()
|
||||
{
|
||||
return reversed ? currentIndexIdx < 0 : currentIndexIdx >= indexEntry.columnsIndexCount();
|
||||
}
|
||||
|
||||
// Sets the reader to the beginning of blockIdx.
|
||||
public void setToBlock(int blockIdx) throws IOException
|
||||
{
|
||||
if (blockIdx >= 0 && blockIdx < indexEntry.columnsIndexCount())
|
||||
{
|
||||
reader.seekToPosition(columnOffset(blockIdx));
|
||||
mark = reader.file.mark();
|
||||
reader.deserializer.clearState();
|
||||
}
|
||||
|
||||
currentIndexIdx = blockIdx;
|
||||
reader.openMarker = blockIdx > 0 ? index(blockIdx - 1).endOpenMarker : null;
|
||||
}
|
||||
|
||||
private long columnOffset(int i) throws IOException
|
||||
{
|
||||
return indexEntry.position + index(i).offset;
|
||||
}
|
||||
|
||||
public int blocksCount()
|
||||
{
|
||||
return indexEntry.columnsIndexCount();
|
||||
}
|
||||
|
||||
// Update the block idx based on the current reader position if we're past the current block.
|
||||
// This only makes sense for forward iteration (for reverse ones, when we reach the end of a block we
|
||||
// should seek to the previous one, not update the index state and continue).
|
||||
public void updateBlock() throws IOException
|
||||
{
|
||||
assert !reversed;
|
||||
|
||||
// If we get here with currentBlockIdx < 0, it means setToBlock() has never been called, so it means
|
||||
// we're about to read from the beginning of the partition, but haven't "prepared" the IndexState yet.
|
||||
// Do so by setting us on the first block.
|
||||
if (currentIndexIdx < 0)
|
||||
{
|
||||
setToBlock(0);
|
||||
return;
|
||||
}
|
||||
|
||||
while (currentIndexIdx + 1 < indexEntry.columnsIndexCount() && isPastCurrentBlock())
|
||||
{
|
||||
reader.openMarker = currentIndex().endOpenMarker;
|
||||
++currentIndexIdx;
|
||||
|
||||
// We have to set the mark, and we have to set it at the beginning of the block. So if we're not at the beginning of the block, this forces us to a weird seek dance.
|
||||
// This can only happen when reading old file however.
|
||||
long startOfBlock = columnOffset(currentIndexIdx);
|
||||
long currentFilePointer = reader.file.getFilePointer();
|
||||
if (startOfBlock == currentFilePointer)
|
||||
{
|
||||
mark = reader.file.mark();
|
||||
}
|
||||
else
|
||||
{
|
||||
reader.seekToPosition(startOfBlock);
|
||||
mark = reader.file.mark();
|
||||
reader.seekToPosition(currentFilePointer);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Check if we've crossed an index boundary (based on the mark on the beginning of the index block).
|
||||
public boolean isPastCurrentBlock() throws IOException
|
||||
{
|
||||
assert reader.deserializer != null;
|
||||
return reader.file.bytesPastMark(mark) >= currentIndex().width;
|
||||
}
|
||||
|
||||
public int currentBlockIdx()
|
||||
{
|
||||
return currentIndexIdx;
|
||||
}
|
||||
|
||||
public IndexInfo currentIndex() throws IOException
|
||||
{
|
||||
return index(currentIndexIdx);
|
||||
}
|
||||
|
||||
public IndexInfo index(int i) throws IOException
|
||||
{
|
||||
return indexInfoRetriever.columnsIndex(i);
|
||||
}
|
||||
|
||||
// Finds the index of the first block containing the provided bound, starting at the provided index.
|
||||
// Will be -1 if the bound is before any block, and blocksCount() if it is after every block.
|
||||
public int findBlockIndex(ClusteringBound<?> bound, int fromIdx) throws IOException
|
||||
{
|
||||
if (bound.isBottom())
|
||||
return -1;
|
||||
if (bound.isTop())
|
||||
return blocksCount();
|
||||
|
||||
return indexFor(bound, fromIdx);
|
||||
}
|
||||
|
||||
public int indexFor(ClusteringPrefix<?> name, int lastIndex) throws IOException
|
||||
{
|
||||
IndexInfo target = new IndexInfo(name, name, 0, 0, null);
|
||||
/*
|
||||
Take the example from the unit test, and say your index looks like this:
|
||||
[0..5][10..15][20..25]
|
||||
and you look for the slice [13..17].
|
||||
|
||||
When doing forward slice, we are doing a binary search comparing 13 (the start of the query)
|
||||
to the lastName part of the index slot. You'll end up with the "first" slot, going from left to right,
|
||||
that may contain the start.
|
||||
|
||||
When doing a reverse slice, we do the same thing, only using as a start column the end of the query,
|
||||
i.e. 17 in this example, compared to the firstName part of the index slots. bsearch will give us the
|
||||
first slot where firstName > start ([20..25] here), so we subtract an extra one to get the slot just before.
|
||||
*/
|
||||
int startIdx = 0;
|
||||
int endIdx = indexEntry.columnsIndexCount() - 1;
|
||||
|
||||
if (reversed)
|
||||
{
|
||||
if (lastIndex < endIdx)
|
||||
{
|
||||
endIdx = lastIndex;
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
if (lastIndex > 0)
|
||||
{
|
||||
startIdx = lastIndex;
|
||||
}
|
||||
}
|
||||
|
||||
int index = binarySearch(target, comparator.indexComparator(reversed), startIdx, endIdx);
|
||||
return (index < 0 ? -index - (reversed ? 2 : 1) : index);
|
||||
}
|
||||
|
||||
private int binarySearch(IndexInfo key, Comparator<IndexInfo> c, int low, int high) throws IOException
|
||||
{
|
||||
while (low <= high)
|
||||
{
|
||||
int mid = (low + high) >>> 1;
|
||||
IndexInfo midVal = index(mid);
|
||||
int cmp = c.compare(midVal, key);
|
||||
|
||||
if (cmp < 0)
|
||||
low = mid + 1;
|
||||
else if (cmp > 0)
|
||||
high = mid - 1;
|
||||
else
|
||||
return mid;
|
||||
}
|
||||
return -(low + 1);
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString()
|
||||
{
|
||||
return String.format("IndexState(indexSize=%d, currentBlock=%d, reversed=%b)", indexEntry.columnsIndexCount(), currentIndexIdx, reversed);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() throws IOException
|
||||
{
|
||||
indexInfoRetriever.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -18,8 +18,8 @@
|
|||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.Closeable;
|
||||
import java.io.IOException;
|
||||
import java.nio.ByteBuffer;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
|
|
@ -27,7 +27,9 @@ import java.util.Collections;
|
|||
import java.util.concurrent.atomic.AtomicReference;
|
||||
import java.util.stream.Stream;
|
||||
|
||||
import org.apache.cassandra.db.*;
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.RegularAndStaticColumns;
|
||||
import org.apache.cassandra.db.SerializationHeader;
|
||||
import org.apache.cassandra.db.partitions.PartitionUpdate;
|
||||
import org.apache.cassandra.db.rows.EncodingStats;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat;
|
||||
|
|
@ -64,7 +66,7 @@ abstract class AbstractSSTableSimpleWriter implements Closeable
|
|||
this.makeRangeAware = makeRangeAware;
|
||||
}
|
||||
|
||||
protected SSTableTxnWriter createWriter() throws IOException
|
||||
protected SSTableTxnWriter createWriter(SSTable.Owner owner) throws IOException
|
||||
{
|
||||
SerializationHeader header = new SerializationHeader(true, metadata.get(), columns, EncodingStats.NO_STATS);
|
||||
|
||||
|
|
@ -79,7 +81,8 @@ abstract class AbstractSSTableSimpleWriter implements Closeable
|
|||
false,
|
||||
0,
|
||||
header,
|
||||
Collections.emptySet());
|
||||
Collections.emptySet(),
|
||||
owner);
|
||||
}
|
||||
|
||||
private static Descriptor createDescriptor(File directory, final String keyspace, final String columnFamily, final SSTableFormat.Type fmt) throws IOException
|
||||
|
|
@ -95,7 +98,7 @@ abstract class AbstractSSTableSimpleWriter implements Closeable
|
|||
try (Stream<Path> existingPaths = Files.list(directory.toPath()))
|
||||
{
|
||||
Stream<SSTableId> existingIds = existingPaths.map(File::new)
|
||||
.map(SSTable::tryDescriptorFromFilename)
|
||||
.map(SSTable::tryDescriptorFromFile)
|
||||
.filter(d -> d != null && d.cfname.equals(columnFamily))
|
||||
.map(d -> d.id);
|
||||
|
||||
|
|
|
|||
|
|
@ -17,11 +17,18 @@
|
|||
*/
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import java.util.EnumSet;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
import java.util.Objects;
|
||||
import java.util.concurrent.CopyOnWriteArrayList;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import com.google.common.base.Objects;
|
||||
import com.google.common.base.Preconditions;
|
||||
import com.google.common.collect.Iterables;
|
||||
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat.Components.Types;
|
||||
|
||||
/**
|
||||
* SSTables are made up of multiple components in separate files. Components are
|
||||
|
|
@ -32,84 +39,149 @@ public class Component
|
|||
{
|
||||
public static final char separator = '-';
|
||||
|
||||
final static EnumSet<Type> TYPES = EnumSet.allOf(Type.class);
|
||||
|
||||
/**
|
||||
* WARNING: Be careful while changing the names or string representation of the enum
|
||||
* members. Streaming code depends on the names during streaming (Ref: CASSANDRA-14556).
|
||||
*/
|
||||
public enum Type
|
||||
public final static class Type
|
||||
{
|
||||
// the base data for an sstable: the remaining components can be regenerated
|
||||
// based on the data component
|
||||
DATA("Data.db"),
|
||||
// index of the row keys with pointers to their positions in the data file
|
||||
PRIMARY_INDEX("Index.db"),
|
||||
// serialized bloom filter for the row keys in the sstable
|
||||
FILTER("Filter.db"),
|
||||
// file to hold information about uncompressed data length, chunk offsets etc.
|
||||
COMPRESSION_INFO("CompressionInfo.db"),
|
||||
// statistical metadata about the content of the sstable
|
||||
STATS("Statistics.db"),
|
||||
// holds CRC32 checksum of the data file
|
||||
DIGEST("Digest.crc32"),
|
||||
// holds the CRC32 for chunks in an a uncompressed file.
|
||||
CRC("CRC.db"),
|
||||
// holds SSTable Index Summary (sampling of Index component)
|
||||
SUMMARY("Summary.db"),
|
||||
// table of contents, stores the list of all components for the sstable
|
||||
TOC("TOC.txt"),
|
||||
// built-in secondary index (may be multiple per sstable)
|
||||
SECONDARY_INDEX("SI_.*.db"),
|
||||
// custom component, used by e.g. custom compaction strategy
|
||||
CUSTOM(null);
|
||||
private final static CopyOnWriteArrayList<Type> typesCollector = new CopyOnWriteArrayList<>();
|
||||
|
||||
final String repr;
|
||||
public static final List<Type> all = Collections.unmodifiableList(typesCollector);
|
||||
|
||||
Type(String repr)
|
||||
public final int id;
|
||||
public final String name;
|
||||
public final String repr;
|
||||
private final Component singleton;
|
||||
|
||||
@SuppressWarnings("rawtypes")
|
||||
public final Class<? extends SSTableFormat> formatClass;
|
||||
|
||||
/**
|
||||
* Creates a new non-singleton type and registers it a global type registry - see {@link #registerType(Type)}.
|
||||
*
|
||||
* @param name type name, must be unique for this and all parent formats
|
||||
* @param repr the regular expression to be used to recognize a name represents this type
|
||||
* @param formatClass format class for which this type is defined for
|
||||
*/
|
||||
public static Type create(String name, String repr, Class<? extends SSTableFormat<?, ?>> formatClass)
|
||||
{
|
||||
return new Type(name, repr, false, formatClass);
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a new singleton type and registers it in a global type registry - see {@link #registerType(Type)}.
|
||||
*
|
||||
* @param name type name, must be unique for this and all parent formats
|
||||
* @param repr the regular expression to be used to recognize a name represents this type
|
||||
* @param formatClass format class for which this type is defined for
|
||||
*/
|
||||
public static Type createSingleton(String name, String repr, Class<? extends SSTableFormat<?, ?>> formatClass)
|
||||
{
|
||||
return new Type(name, repr, true, formatClass);
|
||||
}
|
||||
|
||||
private Type(String name, String repr, boolean isSingleton, Class<? extends SSTableFormat<?, ?>> formatClass)
|
||||
{
|
||||
this.name = Objects.requireNonNull(name);
|
||||
this.repr = repr;
|
||||
this.id = typesCollector.size();
|
||||
this.formatClass = formatClass == null ? SSTableFormat.class : formatClass;
|
||||
this.singleton = isSingleton ? new Component(this) : null;
|
||||
|
||||
registerType(this);
|
||||
}
|
||||
|
||||
/**
|
||||
* If you have two formats registered, they may both define a type say `INDEX`. It is allowed even though
|
||||
* they have the same name because they are not in the same branch. Though, we cannot let a custom type
|
||||
* define a type `TOC` which is declared on the top level.
|
||||
* So, e.g. given we have `TOC@SSTableFormat`, and `BigFormat` tries to define `TOC@BigFormat`, we should
|
||||
* forbid that; but, given we have `INDEX@BigFormat`, we should allow to define `INDEX@TrieFormat` as those
|
||||
* types are be distinguishable via format type.
|
||||
*
|
||||
* @param type a type to be registered
|
||||
*/
|
||||
private static void registerType(Type type)
|
||||
{
|
||||
synchronized (typesCollector)
|
||||
{
|
||||
if (typesCollector.stream().anyMatch(t -> (Objects.equals(t.name, type.name) || Objects.equals(t.repr, type.repr)) && (t.formatClass.isAssignableFrom(type.formatClass))))
|
||||
throw new AssertionError("Type named " + type.name + " is already registered");
|
||||
|
||||
typesCollector.add(type);
|
||||
}
|
||||
}
|
||||
|
||||
@VisibleForTesting
|
||||
public static Type fromRepresentation(String repr)
|
||||
public static Type fromRepresentation(String repr, SSTableFormat.Type formatType)
|
||||
{
|
||||
for (Type type : TYPES)
|
||||
for (Type type : Type.all)
|
||||
{
|
||||
if (type.repr != null && Pattern.matches(type.repr, repr))
|
||||
if (type.repr != null && Pattern.matches(type.repr, repr) && type.formatClass.isAssignableFrom(formatType.info.getClass()))
|
||||
return type;
|
||||
}
|
||||
return CUSTOM;
|
||||
return Types.CUSTOM;
|
||||
}
|
||||
|
||||
public static Component createComponent(String repr, SSTableFormat.Type formatType)
|
||||
{
|
||||
Type type = fromRepresentation(repr, formatType);
|
||||
if (type.singleton != null)
|
||||
return type.singleton;
|
||||
else
|
||||
return new Component(type, repr);
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean equals(Object o)
|
||||
{
|
||||
if (this == o) return true;
|
||||
if (o == null || getClass() != o.getClass()) return false;
|
||||
Type type = (Type) o;
|
||||
return id == type.id;
|
||||
}
|
||||
|
||||
@Override
|
||||
public int hashCode()
|
||||
{
|
||||
return id;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString()
|
||||
{
|
||||
return name;
|
||||
}
|
||||
|
||||
public Component getSingleton()
|
||||
{
|
||||
return Objects.requireNonNull(singleton);
|
||||
}
|
||||
|
||||
public Component createComponent(String repr)
|
||||
{
|
||||
Preconditions.checkArgument(singleton == null);
|
||||
return new Component(this, repr);
|
||||
}
|
||||
}
|
||||
|
||||
// singleton components for types that don't need ids
|
||||
public final static Component DATA = new Component(Type.DATA);
|
||||
public final static Component PRIMARY_INDEX = new Component(Type.PRIMARY_INDEX);
|
||||
public final static Component FILTER = new Component(Type.FILTER);
|
||||
public final static Component COMPRESSION_INFO = new Component(Type.COMPRESSION_INFO);
|
||||
public final static Component STATS = new Component(Type.STATS);
|
||||
public final static Component DIGEST = new Component(Type.DIGEST);
|
||||
public final static Component CRC = new Component(Type.CRC);
|
||||
public final static Component SUMMARY = new Component(Type.SUMMARY);
|
||||
public final static Component TOC = new Component(Type.TOC);
|
||||
|
||||
public final Type type;
|
||||
public final String name;
|
||||
public final int hashCode;
|
||||
|
||||
public Component(Type type)
|
||||
private Component(Type type)
|
||||
{
|
||||
this(type, type.repr);
|
||||
assert type != Type.CUSTOM;
|
||||
}
|
||||
|
||||
public Component(Type type, String name)
|
||||
private Component(Type type, String name)
|
||||
{
|
||||
assert name != null : "Component name cannot be null";
|
||||
|
||||
this.type = type;
|
||||
this.name = name;
|
||||
this.hashCode = Objects.hashCode(type, name);
|
||||
this.hashCode = Objects.hash(type, name);
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -127,26 +199,24 @@ public class Component
|
|||
* @return the component corresponding to {@code name}. Note that this always return a component as an unrecognized
|
||||
* name is parsed into a CUSTOM component.
|
||||
*/
|
||||
public static Component parse(String name)
|
||||
public static Component parse(String name, SSTableFormat.Type formatType)
|
||||
{
|
||||
Type type = Type.fromRepresentation(name);
|
||||
return Type.createComponent(name, formatType);
|
||||
}
|
||||
|
||||
// Build (or retrieve singleton for) the component object
|
||||
switch (type)
|
||||
{
|
||||
case DATA: return Component.DATA;
|
||||
case PRIMARY_INDEX: return Component.PRIMARY_INDEX;
|
||||
case FILTER: return Component.FILTER;
|
||||
case COMPRESSION_INFO: return Component.COMPRESSION_INFO;
|
||||
case STATS: return Component.STATS;
|
||||
case DIGEST: return Component.DIGEST;
|
||||
case CRC: return Component.CRC;
|
||||
case SUMMARY: return Component.SUMMARY;
|
||||
case TOC: return Component.TOC;
|
||||
case SECONDARY_INDEX: return new Component(Type.SECONDARY_INDEX, name);
|
||||
case CUSTOM: return new Component(Type.CUSTOM, name);
|
||||
default: throw new AssertionError();
|
||||
}
|
||||
public static Iterable<Component> getSingletonsFor(SSTableFormat<?, ?> format)
|
||||
{
|
||||
return Iterables.transform(Iterables.filter(Type.all, t -> t.singleton != null && t.formatClass.isAssignableFrom(format.getClass())), t -> t.singleton);
|
||||
}
|
||||
|
||||
public static Iterable<Component> getSingletonsFor(Class<? extends SSTableFormat<?, ?>> formatClass)
|
||||
{
|
||||
return Iterables.transform(Iterables.filter(Type.all, t -> t.singleton != null && t.formatClass.isAssignableFrom(formatClass)), t -> t.singleton);
|
||||
}
|
||||
|
||||
public boolean isValidFor(Descriptor descriptor)
|
||||
{
|
||||
return type.formatClass.isAssignableFrom(descriptor.formatType.info.getClass());
|
||||
}
|
||||
|
||||
@Override
|
||||
|
|
@ -162,8 +232,8 @@ public class Component
|
|||
return true;
|
||||
if (!(o instanceof Component))
|
||||
return false;
|
||||
Component that = (Component)o;
|
||||
return this.type == that.type && this.name.equals(that.name);
|
||||
Component that = (Component) o;
|
||||
return this.hashCode == that.hashCode && this.type == that.type && this.name.equals(that.name);
|
||||
}
|
||||
|
||||
@Override
|
||||
|
|
|
|||
|
|
@ -17,14 +17,17 @@
|
|||
*/
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import com.google.common.base.Objects;
|
||||
import com.google.common.base.Splitter;
|
||||
|
||||
import com.google.common.collect.ImmutableSet;
|
||||
import com.google.common.collect.Sets;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
|
|
@ -36,6 +39,8 @@ import org.apache.cassandra.io.sstable.metadata.MetadataSerializer;
|
|||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.utils.Pair;
|
||||
|
||||
import static com.google.common.base.Preconditions.checkArgument;
|
||||
import static com.google.common.base.Preconditions.checkNotNull;
|
||||
import static org.apache.cassandra.io.sstable.Component.separator;
|
||||
import static org.apache.cassandra.utils.TimeUUID.Generator.nextTimeUUID;
|
||||
|
||||
|
|
@ -84,6 +89,8 @@ public class Descriptor
|
|||
public final SSTableId id;
|
||||
public final SSTableFormat.Type formatType;
|
||||
private final int hashCode;
|
||||
private final String prefix;
|
||||
private final File baseFile;
|
||||
|
||||
/**
|
||||
* A descriptor that assumes CURRENT_VERSION.
|
||||
|
|
@ -110,7 +117,13 @@ public class Descriptor
|
|||
|
||||
public Descriptor(Version version, File directory, String ksname, String cfname, SSTableId id, SSTableFormat.Type formatType)
|
||||
{
|
||||
assert version != null && directory != null && ksname != null && cfname != null && formatType.info.getLatestVersion().getClass().equals(version.getClass());
|
||||
checkNotNull(version);
|
||||
checkNotNull(directory);
|
||||
checkNotNull(ksname);
|
||||
checkNotNull(cfname);
|
||||
checkNotNull(formatType);
|
||||
checkArgument(version.getSSTableFormat() == formatType.info);
|
||||
|
||||
this.version = version;
|
||||
this.directory = directory.toCanonical();
|
||||
this.ksname = ksname;
|
||||
|
|
@ -118,41 +131,53 @@ public class Descriptor
|
|||
this.id = id;
|
||||
this.formatType = formatType;
|
||||
|
||||
StringBuilder buf = new StringBuilder();
|
||||
appendFileName(buf);
|
||||
this.prefix = buf.toString();
|
||||
this.baseFile = new File(directory.toPath().resolve(prefix));
|
||||
|
||||
// directory is unnecessary for hashCode, and for simulator consistency we do not include it
|
||||
hashCode = Objects.hashCode(version, id, ksname, cfname, formatType);
|
||||
}
|
||||
|
||||
public String tmpFilenameFor(Component component)
|
||||
private String tmpFilenameFor(Component component)
|
||||
{
|
||||
return filenameFor(component) + TMP_EXT;
|
||||
return fileFor(component) + TMP_EXT;
|
||||
}
|
||||
|
||||
/**
|
||||
* @return a unique temporary file name for given component during entire-sstable-streaming.
|
||||
*/
|
||||
public String tmpFilenameForStreaming(Component component)
|
||||
public File tmpFileFor(Component component)
|
||||
{
|
||||
return new File(directory.toPath().resolve(tmpFilenameFor(component)));
|
||||
}
|
||||
|
||||
private String tmpFilenameForStreaming(Component component)
|
||||
{
|
||||
// Use UUID to handle concurrent streamings on the same sstable.
|
||||
// TMP_EXT allows temp file to be removed by {@link ColumnFamilyStore#scrubDataDirectories}
|
||||
return String.format("%s.%s%s", filenameFor(component), nextTimeUUID(), TMP_EXT);
|
||||
}
|
||||
|
||||
public String filenameFor(Component component)
|
||||
/**
|
||||
* @return a unique temporary file name for given component during entire-sstable-streaming.
|
||||
*/
|
||||
public File tmpFileForStreaming(Component component)
|
||||
{
|
||||
return baseFilename() + separator + component.name();
|
||||
return new File(directory.toPath().resolve(tmpFilenameForStreaming(component)));
|
||||
}
|
||||
|
||||
private String filenameFor(Component component)
|
||||
{
|
||||
return prefix + separator + component.name();
|
||||
}
|
||||
|
||||
public File fileFor(Component component)
|
||||
{
|
||||
return new File(filenameFor(component));
|
||||
return new File(directory.toPath().resolve(filenameFor(component)));
|
||||
}
|
||||
|
||||
public String baseFilename()
|
||||
public File baseFile()
|
||||
{
|
||||
StringBuilder buff = new StringBuilder();
|
||||
buff.append(directory).append(File.pathSeparator());
|
||||
appendFileName(buff);
|
||||
return buff.toString();
|
||||
return baseFile;
|
||||
}
|
||||
|
||||
private void appendFileName(StringBuilder buff)
|
||||
|
|
@ -175,7 +200,7 @@ public class Descriptor
|
|||
return buff.toString();
|
||||
}
|
||||
|
||||
public SSTableFormat getFormat()
|
||||
public SSTableFormat<?, ?> getFormat()
|
||||
{
|
||||
return formatType.info;
|
||||
}
|
||||
|
|
@ -193,29 +218,28 @@ public class Descriptor
|
|||
return ret;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the set of components consisting of the provided mandatory components and those optional components
|
||||
* for which the corresponding file exists.
|
||||
*/
|
||||
public Set<Component> getComponents(Set<Component> mandatory, Set<Component> optional)
|
||||
{
|
||||
ImmutableSet.Builder<Component> builder = ImmutableSet.builder();
|
||||
builder.addAll(mandatory);
|
||||
for (Component component : optional)
|
||||
{
|
||||
if (fileFor(component).exists())
|
||||
builder.add(component);
|
||||
}
|
||||
return builder.build();
|
||||
}
|
||||
|
||||
public static boolean isValidFile(File file)
|
||||
{
|
||||
String filename = file.name();
|
||||
return filename.endsWith(".db") && !LEGACY_TMP_REGEX.matcher(filename).matches();
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a sstable filename into a Descriptor.
|
||||
* <p>
|
||||
* This is a shortcut for {@code fromFilename(new File(filename))}.
|
||||
*
|
||||
* @param filename the filename to a sstable component.
|
||||
* @return the descriptor for the parsed file.
|
||||
*
|
||||
* @throws IllegalArgumentException if the provided {@code file} does point to a valid sstable filename. This could
|
||||
* mean either that the filename doesn't look like a sstable file, or that it is for an old and unsupported
|
||||
* versions.
|
||||
*/
|
||||
public static Descriptor fromFilename(String filename)
|
||||
{
|
||||
return fromFilenameWithComponent(new File(filename), false).left;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a sstable filename into a Descriptor.
|
||||
* <p>
|
||||
|
|
@ -233,9 +257,9 @@ public class Descriptor
|
|||
* mean either that the filename doesn't look like a sstable file, or that it is for an old and unsupported
|
||||
* versions.
|
||||
*/
|
||||
public static Descriptor fromFilename(File file)
|
||||
public static Descriptor fromFile(File file)
|
||||
{
|
||||
return fromFilenameWithComponent(file).left;
|
||||
return fromFileWithComponent(file).left;
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -249,12 +273,12 @@ public class Descriptor
|
|||
* mean either that the filename doesn't look like a sstable file, or that it is for an old and unsupported
|
||||
* versions.
|
||||
*/
|
||||
public static Pair<Descriptor, Component> fromFilenameWithComponent(File file)
|
||||
public static Pair<Descriptor, Component> fromFileWithComponent(File file)
|
||||
{
|
||||
return fromFilenameWithComponent(file, true);
|
||||
return fromFileWithComponent(file, true);
|
||||
}
|
||||
|
||||
public static Pair<Descriptor, Component> fromFilenameWithComponent(File file, boolean validateDirs)
|
||||
public static Pair<Descriptor, Component> fromFileWithComponent(File file, boolean validateDirs)
|
||||
{
|
||||
// We need to extract the keyspace and table names from the parent directories, so make sure we deal with the
|
||||
// absolute path.
|
||||
|
|
@ -307,11 +331,11 @@ public class Descriptor
|
|||
* mean either that the filename doesn't look like a sstable file, or that it is for an old and unsupported
|
||||
* versions.
|
||||
*/
|
||||
public static Pair<Descriptor, Component> fromFilenameWithComponent(File file, String keyspace, String table)
|
||||
public static Pair<Descriptor, Component> fromFileWithComponent(File file, String keyspace, String table)
|
||||
{
|
||||
if (null == keyspace || null == table)
|
||||
{
|
||||
return fromFilenameWithComponent(file);
|
||||
return fromFileWithComponent(file);
|
||||
}
|
||||
|
||||
SSTableInfo info = validateAndExtractInfo(file);
|
||||
|
|
@ -354,14 +378,14 @@ public class Descriptor
|
|||
SSTableFormat.Type format;
|
||||
try
|
||||
{
|
||||
format = SSTableFormat.Type.validate(formatString);
|
||||
format = SSTableFormat.Type.getByName(formatString);
|
||||
}
|
||||
catch (RuntimeException e)
|
||||
{
|
||||
throw invalidSSTable(name, "unknown 'format' part (%s)", formatString);
|
||||
}
|
||||
|
||||
Component component = Component.parse(tokens.get(3));
|
||||
Component component = Component.parse(tokens.get(3), format);
|
||||
|
||||
Version version = format.info.getVersion(versionString);
|
||||
if (!version.isCompatible())
|
||||
|
|
@ -412,10 +436,21 @@ public class Descriptor
|
|||
return version.isCompatible();
|
||||
}
|
||||
|
||||
public Set<Component> discoverComponents()
|
||||
{
|
||||
Set<Component> components = Sets.newHashSetWithExpectedSize(Component.Type.all.size());
|
||||
for (Component component : Component.getSingletonsFor(formatType.info))
|
||||
{
|
||||
if (fileFor(component).exists())
|
||||
components.add(component);
|
||||
}
|
||||
return components;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString()
|
||||
{
|
||||
return baseFilename();
|
||||
return baseFile().absolutePath();
|
||||
}
|
||||
|
||||
@Override
|
||||
|
|
@ -426,6 +461,8 @@ public class Descriptor
|
|||
if (!(o instanceof Descriptor))
|
||||
return false;
|
||||
Descriptor that = (Descriptor)o;
|
||||
if (this.hashCode != that.hashCode)
|
||||
return false;
|
||||
return that.directory.equals(this.directory)
|
||||
&& that.id.equals(this.id)
|
||||
&& that.ksname.equals(this.ksname)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,78 @@
|
|||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import java.util.Set;
|
||||
|
||||
import com.google.common.collect.ImmutableSet;
|
||||
|
||||
import org.apache.cassandra.db.partitions.AbstractUnfilteredPartitionIterator;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
|
||||
public class EmptySSTableScanner extends AbstractUnfilteredPartitionIterator implements ISSTableScanner
|
||||
{
|
||||
private final SSTableReader sstable;
|
||||
|
||||
public EmptySSTableScanner(SSTableReader sstable)
|
||||
{
|
||||
this.sstable = sstable;
|
||||
}
|
||||
|
||||
public long getBytesScanned()
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
public long getLengthInBytes()
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
public long getCompressedLengthInBytes()
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
public Set<SSTableReader> getBackingSSTables()
|
||||
{
|
||||
return ImmutableSet.of(sstable);
|
||||
}
|
||||
|
||||
public long getCurrentPosition()
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
public TableMetadata metadata()
|
||||
{
|
||||
return sstable.metadata();
|
||||
}
|
||||
|
||||
public boolean hasNext()
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
public UnfilteredRowIterator next()
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import com.codahale.metrics.Gauge;
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.Keyspace;
|
||||
|
||||
public abstract class GaugeProvider<T extends Number>
|
||||
{
|
||||
public final String name;
|
||||
|
||||
public GaugeProvider(String name)
|
||||
{
|
||||
this.name = name;
|
||||
}
|
||||
|
||||
public abstract Gauge<T> getTableGauge(ColumnFamilyStore cfs);
|
||||
|
||||
public abstract Gauge<T> getKeyspaceGauge(Keyspace keyspace);
|
||||
|
||||
public abstract Gauge<T> getGlobalGauge();
|
||||
}
|
||||
|
|
@ -0,0 +1,62 @@
|
|||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import org.apache.cassandra.config.Config;
|
||||
import org.apache.cassandra.config.DatabaseDescriptor;
|
||||
import org.apache.cassandra.io.util.DiskOptimizationStrategy;
|
||||
import org.apache.cassandra.io.util.SequentialWriterOption;
|
||||
|
||||
public class IOOptions
|
||||
{
|
||||
public static IOOptions fromDatabaseDescriptor()
|
||||
{
|
||||
return new IOOptions(DatabaseDescriptor.getDiskOptimizationStrategy(),
|
||||
DatabaseDescriptor.getDiskAccessMode(),
|
||||
DatabaseDescriptor.getIndexAccessMode(),
|
||||
DatabaseDescriptor.getDiskOptimizationEstimatePercentile(),
|
||||
SequentialWriterOption.newBuilder()
|
||||
.trickleFsync(DatabaseDescriptor.getTrickleFsync())
|
||||
.trickleFsyncByteInterval(DatabaseDescriptor.getTrickleFsyncIntervalInKiB() * 1024)
|
||||
.build(),
|
||||
DatabaseDescriptor.getFlushCompression());
|
||||
}
|
||||
|
||||
public final DiskOptimizationStrategy diskOptimizationStrategy;
|
||||
public final Config.DiskAccessMode defaultDiskAccessMode;
|
||||
public final Config.DiskAccessMode indexDiskAccessMode;
|
||||
public final double diskOptimizationEstimatePercentile;
|
||||
public final SequentialWriterOption writerOptions;
|
||||
public final Config.FlushCompression flushCompression;
|
||||
|
||||
public IOOptions(DiskOptimizationStrategy diskOptimizationStrategy,
|
||||
Config.DiskAccessMode defaultDiskAccessMode,
|
||||
Config.DiskAccessMode indexDiskAccessMode,
|
||||
double diskOptimizationEstimatePercentile,
|
||||
SequentialWriterOption writerOptions,
|
||||
Config.FlushCompression flushCompression)
|
||||
{
|
||||
this.diskOptimizationStrategy = diskOptimizationStrategy;
|
||||
this.defaultDiskAccessMode = defaultDiskAccessMode;
|
||||
this.indexDiskAccessMode = indexDiskAccessMode;
|
||||
this.diskOptimizationEstimatePercentile = diskOptimizationEstimatePercentile;
|
||||
this.writerOptions = writerOptions;
|
||||
this.flushCompression = flushCompression;
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,129 @@
|
|||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import java.util.StringJoiner;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
|
||||
import org.apache.cassandra.db.compaction.CompactionInfo;
|
||||
import org.apache.cassandra.utils.Closeable;
|
||||
|
||||
public interface IScrubber extends Closeable
|
||||
{
|
||||
void scrub();
|
||||
|
||||
void close();
|
||||
|
||||
CompactionInfo.Holder getScrubInfo();
|
||||
|
||||
@VisibleForTesting
|
||||
ScrubResult scrubWithResult();
|
||||
|
||||
static Options.Builder options()
|
||||
{
|
||||
return new Options.Builder();
|
||||
}
|
||||
|
||||
final class ScrubResult
|
||||
{
|
||||
public final int goodPartitions;
|
||||
public final int badPartitions;
|
||||
public final int emptyPartitions;
|
||||
|
||||
public ScrubResult(int goodPartitions, int badPartitions, int emptyPartitions)
|
||||
{
|
||||
this.goodPartitions = goodPartitions;
|
||||
this.badPartitions = badPartitions;
|
||||
this.emptyPartitions = emptyPartitions;
|
||||
}
|
||||
}
|
||||
|
||||
class Options
|
||||
{
|
||||
public final boolean checkData;
|
||||
public final boolean reinsertOverflowedTTLRows;
|
||||
public final boolean skipCorrupted;
|
||||
|
||||
private Options(boolean checkData, boolean reinsertOverflowedTTLRows, boolean skipCorrupted)
|
||||
{
|
||||
this.checkData = checkData;
|
||||
this.reinsertOverflowedTTLRows = reinsertOverflowedTTLRows;
|
||||
this.skipCorrupted = skipCorrupted;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString()
|
||||
{
|
||||
return new StringJoiner(", ", Options.class.getSimpleName() + "[", "]")
|
||||
.add("checkData=" + checkData)
|
||||
.add("reinsertOverflowedTTLRows=" + reinsertOverflowedTTLRows)
|
||||
.add("skipCorrupted=" + skipCorrupted)
|
||||
.toString();
|
||||
}
|
||||
|
||||
public static class Builder
|
||||
{
|
||||
private boolean checkData = false;
|
||||
private boolean reinsertOverflowedTTLRows = false;
|
||||
private boolean skipCorrupted = false;
|
||||
|
||||
public Builder checkData()
|
||||
{
|
||||
this.checkData = true;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder checkData(boolean checkData)
|
||||
{
|
||||
this.checkData = checkData;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder reinsertOverflowedTTLRows()
|
||||
{
|
||||
this.reinsertOverflowedTTLRows = true;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder reinsertOverflowedTTLRows(boolean reinsertOverflowedTTLRows)
|
||||
{
|
||||
this.reinsertOverflowedTTLRows = reinsertOverflowedTTLRows;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder skipCorrupted()
|
||||
{
|
||||
this.skipCorrupted = true;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder skipCorrupted(boolean skipCorrupted)
|
||||
{
|
||||
this.skipCorrupted = skipCorrupted;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Options build()
|
||||
{
|
||||
return new Options(checkData, reinsertOverflowedTTLRows, skipCorrupted);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,153 @@
|
|||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import java.io.Closeable;
|
||||
import java.util.Collection;
|
||||
import java.util.function.Function;
|
||||
|
||||
import org.apache.cassandra.db.compaction.CompactionInfo;
|
||||
import org.apache.cassandra.dht.Range;
|
||||
import org.apache.cassandra.dht.Token;
|
||||
import org.apache.cassandra.service.StorageService;
|
||||
|
||||
public interface IVerifier extends Closeable
|
||||
{
|
||||
static Options.Builder options()
|
||||
{
|
||||
return new Options.Builder();
|
||||
}
|
||||
|
||||
void verify();
|
||||
|
||||
@Override
|
||||
void close();
|
||||
|
||||
CompactionInfo.Holder getVerifyInfo();
|
||||
|
||||
class Options
|
||||
{
|
||||
public final boolean invokeDiskFailurePolicy;
|
||||
|
||||
/**
|
||||
* Force extended verification - unless it is enabled, extended verificiation will be done only
|
||||
* if there is no digest present. Setting it along with quick makes no sense.
|
||||
*/
|
||||
public final boolean extendedVerification;
|
||||
|
||||
public final boolean checkVersion;
|
||||
public final boolean mutateRepairStatus;
|
||||
public final boolean checkOwnsTokens;
|
||||
|
||||
/**
|
||||
* Quick check which does not include sstable data verification.
|
||||
*/
|
||||
public final boolean quick;
|
||||
|
||||
public final Function<String, ? extends Collection<Range<Token>>> tokenLookup;
|
||||
|
||||
private Options(boolean invokeDiskFailurePolicy,
|
||||
boolean extendedVerification,
|
||||
boolean checkVersion,
|
||||
boolean mutateRepairStatus,
|
||||
boolean checkOwnsTokens,
|
||||
boolean quick,
|
||||
Function<String, ? extends Collection<Range<Token>>> tokenLookup)
|
||||
{
|
||||
this.invokeDiskFailurePolicy = invokeDiskFailurePolicy;
|
||||
this.extendedVerification = extendedVerification;
|
||||
this.checkVersion = checkVersion;
|
||||
this.mutateRepairStatus = mutateRepairStatus;
|
||||
this.checkOwnsTokens = checkOwnsTokens;
|
||||
this.quick = quick;
|
||||
this.tokenLookup = tokenLookup;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString()
|
||||
{
|
||||
return "Options{" +
|
||||
"invokeDiskFailurePolicy=" + invokeDiskFailurePolicy +
|
||||
", extendedVerification=" + extendedVerification +
|
||||
", checkVersion=" + checkVersion +
|
||||
", mutateRepairStatus=" + mutateRepairStatus +
|
||||
", checkOwnsTokens=" + checkOwnsTokens +
|
||||
", quick=" + quick +
|
||||
'}';
|
||||
}
|
||||
|
||||
public static class Builder
|
||||
{
|
||||
private boolean invokeDiskFailurePolicy = false; // invoking disk failure policy can stop the node if we find a corrupt stable
|
||||
private boolean extendedVerification = false;
|
||||
private boolean checkVersion = false;
|
||||
private boolean mutateRepairStatus = false; // mutating repair status can be dangerous
|
||||
private boolean checkOwnsTokens = false;
|
||||
private boolean quick = false;
|
||||
private Function<String, ? extends Collection<Range<Token>>> tokenLookup = StorageService.instance::getLocalAndPendingRanges;
|
||||
|
||||
public Builder invokeDiskFailurePolicy(boolean param)
|
||||
{
|
||||
this.invokeDiskFailurePolicy = param;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder extendedVerification(boolean param)
|
||||
{
|
||||
this.extendedVerification = param;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder checkVersion(boolean param)
|
||||
{
|
||||
this.checkVersion = param;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder mutateRepairStatus(boolean param)
|
||||
{
|
||||
this.mutateRepairStatus = param;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder checkOwnsTokens(boolean param)
|
||||
{
|
||||
this.checkOwnsTokens = param;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder quick(boolean param)
|
||||
{
|
||||
this.quick = param;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder tokenLookup(Function<String, ? extends Collection<Range<Token>>> tokenLookup)
|
||||
{
|
||||
this.tokenLookup = tokenLookup;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Options build()
|
||||
{
|
||||
return new Options(invokeDiskFailurePolicy, extendedVerification, checkVersion, mutateRepairStatus, checkOwnsTokens, quick, tokenLookup);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -18,17 +18,21 @@
|
|||
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import org.apache.cassandra.db.*;
|
||||
import java.io.IOException;
|
||||
import java.util.List;
|
||||
|
||||
import org.apache.cassandra.db.ClusteringPrefix;
|
||||
import org.apache.cassandra.db.DeletionTime;
|
||||
import org.apache.cassandra.db.SerializationHeader;
|
||||
import org.apache.cassandra.db.TypeSizes;
|
||||
import org.apache.cassandra.db.marshal.AbstractType;
|
||||
import org.apache.cassandra.io.ISerializer;
|
||||
import org.apache.cassandra.io.sstable.format.Version;
|
||||
import org.apache.cassandra.io.sstable.format.big.RowIndexEntry;
|
||||
import org.apache.cassandra.io.util.DataInputPlus;
|
||||
import org.apache.cassandra.io.util.DataOutputPlus;
|
||||
import org.apache.cassandra.utils.ObjectSizes;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* {@code IndexInfo} is embedded in the indexed version of {@link RowIndexEntry}.
|
||||
* Each instance roughly covers a range of {@link org.apache.cassandra.config.Config#column_index_size column_index_size} KiB
|
||||
|
|
|
|||
|
|
@ -19,103 +19,48 @@ package org.apache.cassandra.io.sstable;
|
|||
|
||||
import java.io.IOException;
|
||||
import java.util.concurrent.locks.ReadWriteLock;
|
||||
import java.util.concurrent.locks.ReentrantReadWriteLock;
|
||||
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.RowIndexEntry;
|
||||
import org.apache.cassandra.dht.IPartitioner;
|
||||
import org.apache.cassandra.io.util.DataInputPlus;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.io.util.RandomAccessReader;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.utils.AbstractIterator;
|
||||
import org.apache.cassandra.utils.ByteBufferUtil;
|
||||
import org.apache.cassandra.utils.CloseableIterator;
|
||||
|
||||
public class KeyIterator extends AbstractIterator<DecoratedKey> implements CloseableIterator<DecoratedKey>
|
||||
{
|
||||
private final static class In
|
||||
{
|
||||
private final File path;
|
||||
private volatile RandomAccessReader in;
|
||||
|
||||
public In(File path)
|
||||
{
|
||||
this.path = path;
|
||||
}
|
||||
|
||||
private void maybeInit()
|
||||
{
|
||||
if (in != null)
|
||||
return;
|
||||
|
||||
synchronized (this)
|
||||
{
|
||||
if (in == null)
|
||||
{
|
||||
in = RandomAccessReader.open(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public DataInputPlus get()
|
||||
{
|
||||
maybeInit();
|
||||
return in;
|
||||
}
|
||||
|
||||
public boolean isEOF()
|
||||
{
|
||||
maybeInit();
|
||||
return in.isEOF();
|
||||
}
|
||||
|
||||
public void close()
|
||||
{
|
||||
if (in != null)
|
||||
in.close();
|
||||
}
|
||||
|
||||
public long getFilePointer()
|
||||
{
|
||||
maybeInit();
|
||||
return in.getFilePointer();
|
||||
}
|
||||
|
||||
public long length()
|
||||
{
|
||||
maybeInit();
|
||||
return in.length();
|
||||
}
|
||||
}
|
||||
|
||||
private final Descriptor desc;
|
||||
private final In in;
|
||||
private final IPartitioner partitioner;
|
||||
private final KeyReader it;
|
||||
private final ReadWriteLock fileAccessLock;
|
||||
private final long totalBytes;
|
||||
|
||||
private long keyPosition;
|
||||
private boolean initialized = false;
|
||||
|
||||
public KeyIterator(Descriptor desc, TableMetadata metadata)
|
||||
public KeyIterator(KeyReader it, IPartitioner partitioner, long totalBytes, ReadWriteLock fileAccessLock)
|
||||
{
|
||||
this.desc = desc;
|
||||
in = new In(new File(desc.filenameFor(Component.PRIMARY_INDEX)));
|
||||
partitioner = metadata.partitioner;
|
||||
fileAccessLock = new ReentrantReadWriteLock();
|
||||
this.it = it;
|
||||
this.partitioner = partitioner;
|
||||
this.totalBytes = totalBytes;
|
||||
this.fileAccessLock = fileAccessLock;
|
||||
}
|
||||
|
||||
protected DecoratedKey computeNext()
|
||||
{
|
||||
fileAccessLock.readLock().lock();
|
||||
if (fileAccessLock != null)
|
||||
fileAccessLock.readLock().lock();
|
||||
try
|
||||
{
|
||||
if (in.isEOF())
|
||||
return endOfData();
|
||||
|
||||
keyPosition = in.getFilePointer();
|
||||
DecoratedKey key = partitioner.decorateKey(ByteBufferUtil.readWithShortLength(in.get()));
|
||||
RowIndexEntry.Serializer.skip(in.get(), desc.version); // skip remainder of the entry
|
||||
return key;
|
||||
if (!initialized)
|
||||
{
|
||||
initialized = true;
|
||||
return it.isExhausted()
|
||||
? endOfData()
|
||||
: partitioner.decorateKey(it.key());
|
||||
}
|
||||
else
|
||||
{
|
||||
return it.advance()
|
||||
? partitioner.decorateKey(it.key())
|
||||
: endOfData();
|
||||
}
|
||||
}
|
||||
catch (IOException e)
|
||||
{
|
||||
|
|
@ -123,45 +68,44 @@ public class KeyIterator extends AbstractIterator<DecoratedKey> implements Close
|
|||
}
|
||||
finally
|
||||
{
|
||||
fileAccessLock.readLock().unlock();
|
||||
if (fileAccessLock != null)
|
||||
fileAccessLock.readLock().unlock();
|
||||
}
|
||||
}
|
||||
|
||||
public void close()
|
||||
{
|
||||
fileAccessLock.writeLock().lock();
|
||||
if (fileAccessLock != null)
|
||||
fileAccessLock.writeLock().lock();
|
||||
try
|
||||
{
|
||||
in.close();
|
||||
it.close();
|
||||
}
|
||||
finally
|
||||
{
|
||||
fileAccessLock.writeLock().unlock();
|
||||
if (fileAccessLock != null)
|
||||
fileAccessLock.writeLock().unlock();
|
||||
}
|
||||
}
|
||||
|
||||
public long getBytesRead()
|
||||
{
|
||||
fileAccessLock.readLock().lock();
|
||||
if (fileAccessLock != null)
|
||||
fileAccessLock.readLock().lock();
|
||||
try
|
||||
{
|
||||
return in.getFilePointer();
|
||||
return it.isExhausted() ? totalBytes : it.dataPosition();
|
||||
}
|
||||
finally
|
||||
{
|
||||
fileAccessLock.readLock().unlock();
|
||||
if (fileAccessLock != null)
|
||||
fileAccessLock.readLock().unlock();
|
||||
}
|
||||
}
|
||||
|
||||
public long getTotalBytes()
|
||||
{
|
||||
// length is final in the referenced object.
|
||||
// no need to acquire the lock
|
||||
return in.length();
|
||||
return totalBytes;
|
||||
}
|
||||
|
||||
public long getKeyPosition()
|
||||
{
|
||||
return keyPosition;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,74 @@
|
|||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import java.io.Closeable;
|
||||
import java.io.IOException;
|
||||
import java.nio.ByteBuffer;
|
||||
|
||||
/**
|
||||
* Reads keys from an SSTable.
|
||||
* <p/>
|
||||
* It is specific to SSTable format how the keys are read but in general the assumption is that it will read all the
|
||||
* keys in the order as they are placed in data file.
|
||||
* <p/>
|
||||
* After creating it, it should be at the first key. Unless the SSTable is empty, {@link #key()},
|
||||
* {@link #keyPositionForSecondaryIndex()} and {@link #dataPosition()} should return approriate values. If there is
|
||||
* no data, {@link #isExhausted()} returns {@code true}. In order to move to the next key, {@link #advance()} should be
|
||||
* called. It returns {@code true} if the reader moved to the next key; otherwise, there is no more data to read and
|
||||
* the reader is exhausted. When the reader is exhausted, return values of {@link #key()},
|
||||
* {@link #keyPositionForSecondaryIndex()} and {@link #dataPosition()} are undefined.
|
||||
*/
|
||||
public interface KeyReader extends Closeable
|
||||
{
|
||||
/**
|
||||
* Current key
|
||||
*/
|
||||
ByteBuffer key();
|
||||
|
||||
/**
|
||||
* Position in the component preferred for reading keys. This is specific to SSTable implementation
|
||||
*/
|
||||
long keyPositionForSecondaryIndex();
|
||||
|
||||
/**
|
||||
* Position in the data file where the associated content resides
|
||||
*/
|
||||
long dataPosition();
|
||||
|
||||
/**
|
||||
* Moves the iterator forward. Returns false if we reach EOF and there nothing more to read
|
||||
*/
|
||||
boolean advance() throws IOException;
|
||||
|
||||
/**
|
||||
* Returns true if we reach EOF
|
||||
*/
|
||||
boolean isExhausted();
|
||||
|
||||
/**
|
||||
* Resets the iterator to the initial position
|
||||
*/
|
||||
void reset() throws IOException;
|
||||
|
||||
/**
|
||||
* Closes the iterator quietly
|
||||
*/
|
||||
@Override
|
||||
void close();
|
||||
}
|
||||
|
|
@ -16,14 +16,9 @@
|
|||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.cassandra.db.filter;
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import org.apache.cassandra.db.RejectException;
|
||||
|
||||
public class RowIndexEntryReadSizeTooLargeException extends RejectException
|
||||
public interface MetricsProviders
|
||||
{
|
||||
public RowIndexEntryReadSizeTooLargeException(String message)
|
||||
{
|
||||
super(message);
|
||||
}
|
||||
Iterable<GaugeProvider<?>> getGaugeProviders();
|
||||
}
|
||||
|
|
@ -15,7 +15,7 @@
|
|||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.apache.cassandra.io.sstable.format;
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.ArrayList;
|
||||
|
|
@ -30,8 +30,8 @@ import org.apache.cassandra.db.PartitionPosition;
|
|||
import org.apache.cassandra.db.SerializationHeader;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleNewTracker;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.SSTableMultiWriter;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.schema.TableId;
|
||||
import org.apache.cassandra.utils.FBUtilities;
|
||||
import org.apache.cassandra.utils.TimeUUID;
|
||||
|
|
@ -17,15 +17,18 @@
|
|||
*/
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.Iterator;
|
||||
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.util.FileUtils;
|
||||
import org.apache.cassandra.utils.CloseableIterator;
|
||||
import org.apache.cassandra.utils.IMergeIterator;
|
||||
import org.apache.cassandra.utils.MergeIterator;
|
||||
import org.apache.cassandra.utils.Throwables;
|
||||
|
||||
/**
|
||||
* Caller must acquire and release references to the sstables used here.
|
||||
|
|
@ -39,7 +42,17 @@ public class ReducingKeyIterator implements CloseableIterator<DecoratedKey>
|
|||
{
|
||||
iters = new ArrayList<>(sstables.size());
|
||||
for (SSTableReader sstable : sstables)
|
||||
iters.add(new KeyIterator(sstable.descriptor, sstable.metadata()));
|
||||
{
|
||||
try
|
||||
{
|
||||
iters.add(sstable.keyIterator());
|
||||
}
|
||||
catch (IOException ex)
|
||||
{
|
||||
iters.forEach(FileUtils::closeQuietly);
|
||||
throw new RuntimeException("Failed to create a key iterator for sstable " + sstable.getFilename());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private void maybeInit()
|
||||
|
|
@ -78,7 +91,17 @@ public class ReducingKeyIterator implements CloseableIterator<DecoratedKey>
|
|||
public void close()
|
||||
{
|
||||
if (mi != null)
|
||||
{
|
||||
mi.close();
|
||||
}
|
||||
else
|
||||
{
|
||||
// if merging iterator was not initialized before this reducing iterator is closed, we need to close the
|
||||
// underlying iterators manually
|
||||
Throwable err = Throwables.close(null, iters);
|
||||
if (err != null)
|
||||
throw Throwables.unchecked(err);
|
||||
}
|
||||
}
|
||||
|
||||
public long getTotalBytes()
|
||||
|
|
@ -121,4 +144,4 @@ public class ReducingKeyIterator implements CloseableIterator<DecoratedKey>
|
|||
{
|
||||
throw new UnsupportedOperationException();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -17,16 +17,15 @@
|
|||
*/
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
|
||||
import java.io.FileNotFoundException;
|
||||
import java.io.IOError;
|
||||
import java.io.IOException;
|
||||
import java.io.PrintWriter;
|
||||
import java.lang.ref.WeakReference;
|
||||
import java.nio.ByteBuffer;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.NoSuchFileException;
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.CopyOnWriteArraySet;
|
||||
import javax.annotation.Nullable;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import com.google.common.base.Preconditions;
|
||||
|
|
@ -34,43 +33,35 @@ import com.google.common.base.Predicates;
|
|||
import com.google.common.collect.Collections2;
|
||||
import com.google.common.collect.ImmutableSet;
|
||||
import com.google.common.collect.Sets;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.apache.cassandra.db.BufferDecoratedKey;
|
||||
import org.apache.cassandra.cache.ChunkCache;
|
||||
import org.apache.cassandra.config.CassandraRelevantProperties;
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.RowIndexEntry;
|
||||
import org.apache.cassandra.dht.AbstractBounds;
|
||||
import org.apache.cassandra.dht.IPartitioner;
|
||||
import org.apache.cassandra.dht.Token;
|
||||
import org.apache.cassandra.io.FSWriteError;
|
||||
import org.apache.cassandra.io.util.DiskOptimizationStrategy;
|
||||
import org.apache.cassandra.io.util.FileOutputStreamPlus;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat.Components;
|
||||
import org.apache.cassandra.io.sstable.format.TOCComponent;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.io.util.FileUtils;
|
||||
import org.apache.cassandra.io.util.RandomAccessReader;
|
||||
import org.apache.cassandra.metrics.TableMetrics;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.schema.TableMetadataRef;
|
||||
import org.apache.cassandra.utils.ByteBufferUtil;
|
||||
import org.apache.cassandra.utils.Pair;
|
||||
import org.apache.cassandra.utils.TimeUUID;
|
||||
import org.apache.cassandra.utils.memory.HeapCloner;
|
||||
import org.apache.cassandra.utils.concurrent.OpOrder;
|
||||
import org.apache.cassandra.utils.concurrent.SharedCloseable;
|
||||
|
||||
import static org.apache.cassandra.io.util.File.WriteMode.APPEND;
|
||||
import static com.google.common.base.Preconditions.checkNotNull;
|
||||
import static org.apache.cassandra.service.ActiveRepairService.NO_PENDING_REPAIR;
|
||||
import static org.apache.cassandra.service.ActiveRepairService.UNREPAIRED_SSTABLE;
|
||||
|
||||
/**
|
||||
* This class is built on top of the SequenceFile. It stores
|
||||
* data on disk in sorted fashion. However the sorting is upto
|
||||
* the application. This class expects keys to be handed to it
|
||||
* in sorted order.
|
||||
*
|
||||
* A separate index file is maintained as well, containing the
|
||||
* SSTable keys and the offset into the SSTable at which they are found.
|
||||
* Every 1/indexInterval key is read into memory when the SSTable is opened.
|
||||
*
|
||||
* Finally, a bloom filter file is also kept for the keys in each SSTable.
|
||||
* This class represents an abstract sstable on disk whose keys and corresponding partitions are stored in
|
||||
* a {@link SSTableFormat.Components#DATA} file in order as imposed by {@link DecoratedKey#comparator}.
|
||||
*/
|
||||
public abstract class SSTable
|
||||
{
|
||||
|
|
@ -78,70 +69,101 @@ public abstract class SSTable
|
|||
|
||||
public static final int TOMBSTONE_HISTOGRAM_BIN_SIZE = 100;
|
||||
public static final int TOMBSTONE_HISTOGRAM_SPOOL_SIZE = 100000;
|
||||
public static final int TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS = Integer.valueOf(System.getProperty("cassandra.streaminghistogram.roundseconds", "60"));
|
||||
public static final int TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS = CassandraRelevantProperties.TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS.getInt();
|
||||
|
||||
public final Descriptor descriptor;
|
||||
protected final Set<Component> components;
|
||||
public final boolean compression;
|
||||
|
||||
public DecoratedKey first;
|
||||
public DecoratedKey last;
|
||||
|
||||
protected final DiskOptimizationStrategy optimizationStrategy;
|
||||
protected final TableMetadataRef metadata;
|
||||
|
||||
protected SSTable(Descriptor descriptor, Set<Component> components, TableMetadataRef metadata, DiskOptimizationStrategy optimizationStrategy)
|
||||
{
|
||||
// In almost all cases, metadata shouldn't be null, but allowing null allows to create a mostly functional SSTable without
|
||||
// full schema definition. SSTableLoader use that ability
|
||||
assert descriptor != null;
|
||||
assert components != null;
|
||||
public final ChunkCache chunkCache;
|
||||
public final IOOptions ioOptions;
|
||||
|
||||
this.descriptor = descriptor;
|
||||
Set<Component> dataComponents = new HashSet<>(components);
|
||||
this.compression = dataComponents.contains(Component.COMPRESSION_INFO);
|
||||
this.components = new CopyOnWriteArraySet<>(dataComponents);
|
||||
this.metadata = metadata;
|
||||
this.optimizationStrategy = Objects.requireNonNull(optimizationStrategy);
|
||||
@Nullable
|
||||
private final WeakReference<Owner> owner;
|
||||
|
||||
public SSTable(Builder<?, ?> builder, Owner owner)
|
||||
{
|
||||
this.owner = new WeakReference<>(owner);
|
||||
checkNotNull(builder.descriptor);
|
||||
checkNotNull(builder.getComponents());
|
||||
|
||||
this.descriptor = builder.descriptor;
|
||||
this.ioOptions = builder.getIOOptions();
|
||||
this.components = new CopyOnWriteArraySet<>(builder.getComponents());
|
||||
this.compression = components.contains(Components.COMPRESSION_INFO);
|
||||
this.metadata = builder.getTableMetadataRef();
|
||||
this.chunkCache = builder.getChunkCache();
|
||||
}
|
||||
|
||||
public final Optional<Owner> owner()
|
||||
{
|
||||
if (owner == null)
|
||||
return Optional.empty();
|
||||
return Optional.ofNullable(owner.get());
|
||||
}
|
||||
|
||||
public static void rename(Descriptor tmpdesc, Descriptor newdesc, Set<Component> components)
|
||||
{
|
||||
components.stream()
|
||||
.filter(c -> !newdesc.getFormat().generatedOnLoadComponents().contains(c))
|
||||
.filter(c -> !c.equals(Components.DATA))
|
||||
.forEach(c -> tmpdesc.fileFor(c).move(newdesc.fileFor(c)));
|
||||
|
||||
// do -Data last because -Data present should mean the sstable was completely renamed before crash
|
||||
tmpdesc.fileFor(Components.DATA).move(newdesc.fileFor(Components.DATA));
|
||||
|
||||
// rename it without confirmation because summary can be available for loadNewSSTables but not for closeAndOpenReader
|
||||
components.stream()
|
||||
.filter(c -> newdesc.getFormat().generatedOnLoadComponents().contains(c))
|
||||
.forEach(c -> tmpdesc.fileFor(c).tryMove(newdesc.fileFor(c)));
|
||||
}
|
||||
|
||||
public static void copy(Descriptor tmpdesc, Descriptor newdesc, Set<Component> components)
|
||||
{
|
||||
components.stream()
|
||||
.filter(c -> !newdesc.getFormat().generatedOnLoadComponents().contains(c))
|
||||
.filter(c -> !c.equals(Components.DATA))
|
||||
.forEach(c -> FileUtils.copyWithConfirm(tmpdesc.fileFor(c), newdesc.fileFor(c)));
|
||||
|
||||
// do -Data last because -Data present should mean the sstable was completely copied before crash
|
||||
FileUtils.copyWithConfirm(tmpdesc.fileFor(Components.DATA), newdesc.fileFor(Components.DATA));
|
||||
|
||||
// copy it without confirmation because summary can be available for loadNewSSTables but not for closeAndOpenReader
|
||||
components.stream()
|
||||
.filter(c -> newdesc.getFormat().generatedOnLoadComponents().contains(c))
|
||||
.forEach(c -> FileUtils.copyWithOutConfirm(tmpdesc.fileFor(c), newdesc.fileFor(c)));
|
||||
}
|
||||
|
||||
public static void hardlink(Descriptor tmpdesc, Descriptor newdesc, Set<Component> components)
|
||||
{
|
||||
components.stream()
|
||||
.filter(c -> !newdesc.getFormat().generatedOnLoadComponents().contains(c))
|
||||
.filter(c -> !c.equals(Components.DATA))
|
||||
.forEach(c -> FileUtils.createHardLinkWithConfirm(tmpdesc.fileFor(c), newdesc.fileFor(c)));
|
||||
|
||||
// do -Data last because -Data present should mean the sstable was completely copied before crash
|
||||
FileUtils.createHardLinkWithConfirm(tmpdesc.fileFor(Components.DATA), newdesc.fileFor(Components.DATA));
|
||||
|
||||
// copy it without confirmation because summary can be available for loadNewSSTables but not for closeAndOpenReader
|
||||
components.stream()
|
||||
.filter(c -> newdesc.getFormat().generatedOnLoadComponents().contains(c))
|
||||
.forEach(c -> FileUtils.createHardLinkWithoutConfirm(tmpdesc.fileFor(c), newdesc.fileFor(c)));
|
||||
}
|
||||
|
||||
public abstract DecoratedKey getFirst();
|
||||
|
||||
public abstract DecoratedKey getLast();
|
||||
|
||||
public abstract AbstractBounds<Token> getBounds();
|
||||
|
||||
@VisibleForTesting
|
||||
public Set<Component> getComponents()
|
||||
{
|
||||
return ImmutableSet.copyOf(components);
|
||||
}
|
||||
|
||||
/**
|
||||
* We use a ReferenceQueue to manage deleting files that have been compacted
|
||||
* and for which no more SSTable references exist. But this is not guaranteed
|
||||
* to run for each such file because of the semantics of the JVM gc. So,
|
||||
* we write a marker to `compactedFilename` when a file is compacted;
|
||||
* if such a marker exists on startup, the file should be removed.
|
||||
*
|
||||
* This method will also remove SSTables that are marked as temporary.
|
||||
*
|
||||
* @return true if the file was deleted
|
||||
*/
|
||||
public static boolean delete(Descriptor desc, Set<Component> components)
|
||||
{
|
||||
logger.info("Deleting sstable: {}", desc);
|
||||
// remove the DATA component first if it exists
|
||||
if (components.contains(Component.DATA))
|
||||
FileUtils.deleteWithConfirm(desc.filenameFor(Component.DATA));
|
||||
for (Component component : components)
|
||||
{
|
||||
if (component.equals(Component.DATA) || component.equals(Component.SUMMARY))
|
||||
continue;
|
||||
|
||||
FileUtils.deleteWithConfirm(desc.filenameFor(component));
|
||||
}
|
||||
|
||||
if (components.contains(Component.SUMMARY))
|
||||
FileUtils.delete(desc.filenameFor(Component.SUMMARY));
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
public TableMetadata metadata()
|
||||
{
|
||||
return metadata.get();
|
||||
|
|
@ -157,25 +179,9 @@ public abstract class SSTable
|
|||
return getPartitioner().decorateKey(key);
|
||||
}
|
||||
|
||||
/**
|
||||
* If the given @param key occupies only part of a larger buffer, allocate a new buffer that is only
|
||||
* as large as necessary.
|
||||
*/
|
||||
public static DecoratedKey getMinimalKey(DecoratedKey key)
|
||||
{
|
||||
return key.getKey().position() > 0 || key.getKey().hasRemaining() || !key.getKey().hasArray()
|
||||
? new BufferDecoratedKey(key.getToken(), HeapCloner.instance.clone(key.getKey()))
|
||||
: key;
|
||||
}
|
||||
|
||||
public String getFilename()
|
||||
{
|
||||
return descriptor.filenameFor(Component.DATA);
|
||||
}
|
||||
|
||||
public String getIndexFilename()
|
||||
{
|
||||
return descriptor.filenameFor(Component.PRIMARY_INDEX);
|
||||
return descriptor.fileFor(Components.DATA).absolutePath();
|
||||
}
|
||||
|
||||
public String getColumnFamilyName()
|
||||
|
|
@ -192,10 +198,31 @@ public abstract class SSTable
|
|||
{
|
||||
List<String> ret = new ArrayList<>(components.size());
|
||||
for (Component component : components)
|
||||
ret.add(descriptor.filenameFor(component));
|
||||
ret.add(descriptor.fileFor(component).absolutePath());
|
||||
return ret;
|
||||
}
|
||||
|
||||
/**
|
||||
* The method sets fields for this sstable representation on the provided {@link Builder}. The method is intended
|
||||
* to be called from the overloaded {@code unbuildTo} method in subclasses.
|
||||
*
|
||||
* @param builder the builder on which the fields should be set
|
||||
* @param sharedCopy whether the {@link SharedCloseable} resources should be passed as shared copies or directly;
|
||||
* note that the method will overwrite the fields representing {@link SharedCloseable} only if
|
||||
* they are not set in the builder yet (the relevant fields in the builder are {@code null}).
|
||||
* Although {@link SSTable} does not keep any references to resources, the parameters is added
|
||||
* for the possible future fields and for consistency with the overloaded implementations in
|
||||
* subclasses
|
||||
* @return the same instance of builder as provided
|
||||
*/
|
||||
protected final <B extends Builder<?, B>> B unbuildTo(B builder, boolean sharedCopy)
|
||||
{
|
||||
return builder.setTableMetadataRef(metadata)
|
||||
.setComponents(components)
|
||||
.setChunkCache(chunkCache)
|
||||
.setIOOptions(ioOptions);
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a sstable filename into both a {@link Descriptor} and {@code Component} object.
|
||||
*
|
||||
|
|
@ -208,7 +235,7 @@ public abstract class SSTable
|
|||
{
|
||||
try
|
||||
{
|
||||
return Descriptor.fromFilenameWithComponent(file);
|
||||
return Descriptor.fromFileWithComponent(file);
|
||||
}
|
||||
catch (Throwable e)
|
||||
{
|
||||
|
|
@ -230,7 +257,7 @@ public abstract class SSTable
|
|||
{
|
||||
try
|
||||
{
|
||||
return Descriptor.fromFilenameWithComponent(file, keyspace, table);
|
||||
return Descriptor.fromFileWithComponent(file, keyspace, table);
|
||||
}
|
||||
catch (Throwable e)
|
||||
{
|
||||
|
|
@ -248,11 +275,11 @@ public abstract class SSTable
|
|||
* @return the {@code Descriptor} corresponding to {@code file} if it corresponds to a valid and supported sstable
|
||||
* filename, {@code null} otherwise.
|
||||
*/
|
||||
public static Descriptor tryDescriptorFromFilename(File file)
|
||||
public static Descriptor tryDescriptorFromFile(File file)
|
||||
{
|
||||
try
|
||||
{
|
||||
return Descriptor.fromFilename(file);
|
||||
return Descriptor.fromFile(file);
|
||||
}
|
||||
catch (Throwable e)
|
||||
{
|
||||
|
|
@ -260,140 +287,10 @@ public abstract class SSTable
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Discovers existing components for the descriptor. Slow: only intended for use outside the critical path.
|
||||
*/
|
||||
public static Set<Component> componentsFor(final Descriptor desc)
|
||||
{
|
||||
try
|
||||
{
|
||||
try
|
||||
{
|
||||
return readTOC(desc);
|
||||
}
|
||||
catch (FileNotFoundException | NoSuchFileException e)
|
||||
{
|
||||
Set<Component> components = discoverComponentsFor(desc);
|
||||
if (components.isEmpty())
|
||||
return components; // sstable doesn't exist yet
|
||||
|
||||
if (!components.contains(Component.TOC))
|
||||
components.add(Component.TOC);
|
||||
appendTOC(desc, components);
|
||||
return components;
|
||||
}
|
||||
}
|
||||
catch (IOException e)
|
||||
{
|
||||
throw new IOError(e);
|
||||
}
|
||||
}
|
||||
|
||||
public static Set<Component> discoverComponentsFor(Descriptor desc)
|
||||
{
|
||||
Set<Component.Type> knownTypes = Sets.difference(Component.TYPES, Collections.singleton(Component.Type.CUSTOM));
|
||||
Set<Component> components = Sets.newHashSetWithExpectedSize(knownTypes.size());
|
||||
for (Component.Type componentType : knownTypes)
|
||||
{
|
||||
Component component = new Component(componentType);
|
||||
if (new File(desc.filenameFor(component)).exists())
|
||||
components.add(component);
|
||||
}
|
||||
return components;
|
||||
}
|
||||
|
||||
/** @return An estimate of the number of keys contained in the given index file. */
|
||||
public static long estimateRowsFromIndex(RandomAccessReader ifile, Descriptor descriptor) throws IOException
|
||||
{
|
||||
// collect sizes for the first 10000 keys, or first 10 mebibytes of data
|
||||
final int SAMPLES_CAP = 10000, BYTES_CAP = (int)Math.min(10000000, ifile.length());
|
||||
int keys = 0;
|
||||
while (ifile.getFilePointer() < BYTES_CAP && keys < SAMPLES_CAP)
|
||||
{
|
||||
ByteBufferUtil.skipShortLength(ifile);
|
||||
RowIndexEntry.Serializer.skip(ifile, descriptor.version);
|
||||
keys++;
|
||||
}
|
||||
assert keys > 0 && ifile.getFilePointer() > 0 && ifile.length() > 0 : "Unexpected empty index file: " + ifile;
|
||||
long estimatedRows = ifile.length() / (ifile.getFilePointer() / keys);
|
||||
ifile.seek(0);
|
||||
return estimatedRows;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString()
|
||||
{
|
||||
return getClass().getSimpleName() + "(" +
|
||||
"path='" + getFilename() + '\'' +
|
||||
')';
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads the list of components from the TOC component.
|
||||
* @return set of components found in the TOC
|
||||
*/
|
||||
protected static Set<Component> readTOC(Descriptor descriptor) throws IOException
|
||||
{
|
||||
return readTOC(descriptor, true);
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads the list of components from the TOC component.
|
||||
* @param skipMissing, skip adding the component to the returned set if the corresponding file is missing.
|
||||
* @return set of components found in the TOC
|
||||
*/
|
||||
protected static Set<Component> readTOC(Descriptor descriptor, boolean skipMissing) throws IOException
|
||||
{
|
||||
File tocFile = new File(descriptor.filenameFor(Component.TOC));
|
||||
List<String> componentNames = Files.readAllLines(tocFile.toPath());
|
||||
Set<Component> components = Sets.newHashSetWithExpectedSize(componentNames.size());
|
||||
for (String componentName : componentNames)
|
||||
{
|
||||
Component component = new Component(Component.Type.fromRepresentation(componentName), componentName);
|
||||
if (skipMissing && !new File(descriptor.filenameFor(component)).exists())
|
||||
logger.error("Missing component: {}", descriptor.filenameFor(component));
|
||||
else
|
||||
components.add(component);
|
||||
}
|
||||
return components;
|
||||
}
|
||||
|
||||
/**
|
||||
* Appends new component names to the TOC component.
|
||||
*/
|
||||
protected static void appendTOC(Descriptor descriptor, Collection<Component> components)
|
||||
{
|
||||
File tocFile = new File(descriptor.filenameFor(Component.TOC));
|
||||
try (FileOutputStreamPlus out = tocFile.newOutputStream(APPEND);
|
||||
PrintWriter w = new PrintWriter(out))
|
||||
{
|
||||
for (Component component : components)
|
||||
w.println(component.name);
|
||||
w.flush();
|
||||
out.sync();
|
||||
}
|
||||
catch (IOException e)
|
||||
{
|
||||
throw new FSWriteError(e, tocFile);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Registers new custom components. Used by custom compaction strategies.
|
||||
* Adding a component for the second time is a no-op.
|
||||
* Don't remove this - this method is a part of the public API, intended for use by custom compaction strategies.
|
||||
* @param newComponents collection of components to be added
|
||||
*/
|
||||
public synchronized void addComponents(Collection<Component> newComponents)
|
||||
{
|
||||
Collection<Component> componentsToAdd = Collections2.filter(newComponents, Predicates.not(Predicates.in(components)));
|
||||
appendTOC(descriptor, componentsToAdd);
|
||||
components.addAll(componentsToAdd);
|
||||
}
|
||||
|
||||
public AbstractBounds<Token> getBounds()
|
||||
{
|
||||
return AbstractBounds.bounds(first.getToken(), true, last.getToken(), true);
|
||||
return String.format("%s:%s(path='%s')", getClass().getSimpleName(), descriptor.formatType.name, getFilename());
|
||||
}
|
||||
|
||||
public static void validateRepairedMetadata(long repairedAt, TimeUUID pendingRepair, boolean isTransient)
|
||||
|
|
@ -402,6 +299,118 @@ public abstract class SSTable
|
|||
"pendingRepair cannot be set on a repaired sstable");
|
||||
Preconditions.checkArgument(!isTransient || (pendingRepair != NO_PENDING_REPAIR),
|
||||
"isTransient can only be true for sstables pending repair");
|
||||
}
|
||||
|
||||
/**
|
||||
* Registers new custom components. Used by custom compaction strategies.
|
||||
* Adding a component for the second time is a no-op.
|
||||
* Don't remove this - this method is a part of the public API, intended for use by custom compaction strategies.
|
||||
*
|
||||
* @param newComponents collection of components to be added
|
||||
*/
|
||||
public synchronized void addComponents(Collection<Component> newComponents)
|
||||
{
|
||||
Collection<Component> componentsToAdd = Collections2.filter(newComponents, Predicates.not(Predicates.in(components)));
|
||||
TOCComponent.appendTOC(descriptor, componentsToAdd);
|
||||
components.addAll(componentsToAdd);
|
||||
}
|
||||
|
||||
public interface Owner
|
||||
{
|
||||
Double getCrcCheckChance();
|
||||
|
||||
OpOrder.Barrier newReadOrderingBarrier();
|
||||
|
||||
TableMetrics getMetrics();
|
||||
}
|
||||
|
||||
/**
|
||||
* A builder of this sstable representation. It should be extended for each implementation with the specific fields.
|
||||
*
|
||||
* @param <S> type of the sstable representation to be build with this builder
|
||||
* @param <B> type of this builder
|
||||
*/
|
||||
public static class Builder<S extends SSTable, B extends Builder<S, B>>
|
||||
{
|
||||
public final Descriptor descriptor;
|
||||
|
||||
private Set<Component> components;
|
||||
private TableMetadataRef tableMetadataRef;
|
||||
private ChunkCache chunkCache = ChunkCache.instance;
|
||||
private IOOptions ioOptions = IOOptions.fromDatabaseDescriptor();
|
||||
|
||||
public Builder(Descriptor descriptor)
|
||||
{
|
||||
checkNotNull(descriptor);
|
||||
this.descriptor = descriptor;
|
||||
}
|
||||
|
||||
public B setComponents(Collection<Component> components)
|
||||
{
|
||||
if (components != null)
|
||||
{
|
||||
components.forEach(c -> Preconditions.checkState(c.isValidFor(descriptor), "Invalid component type for sstable format " + descriptor.formatType.name));
|
||||
this.components = ImmutableSet.copyOf(components);
|
||||
}
|
||||
else
|
||||
{
|
||||
this.components = null;
|
||||
}
|
||||
return (B) this;
|
||||
}
|
||||
|
||||
public B addComponents(Collection<Component> components)
|
||||
{
|
||||
if (components == null || components.isEmpty())
|
||||
return (B) this;
|
||||
|
||||
if (this.components == null)
|
||||
return setComponents(components);
|
||||
|
||||
return setComponents(Sets.union(this.components, ImmutableSet.copyOf(components)));
|
||||
}
|
||||
|
||||
public B setTableMetadataRef(TableMetadataRef ref)
|
||||
{
|
||||
this.tableMetadataRef = ref;
|
||||
return (B) this;
|
||||
}
|
||||
|
||||
public B setChunkCache(ChunkCache chunkCache)
|
||||
{
|
||||
this.chunkCache = chunkCache;
|
||||
return (B) this;
|
||||
}
|
||||
|
||||
public B setIOOptions(IOOptions ioOptions)
|
||||
{
|
||||
this.ioOptions = ioOptions;
|
||||
return (B) this;
|
||||
}
|
||||
|
||||
public Descriptor getDescriptor()
|
||||
{
|
||||
return descriptor;
|
||||
}
|
||||
|
||||
public Set<Component> getComponents()
|
||||
{
|
||||
return components;
|
||||
}
|
||||
|
||||
public TableMetadataRef getTableMetadataRef()
|
||||
{
|
||||
return tableMetadataRef;
|
||||
}
|
||||
|
||||
public ChunkCache getChunkCache()
|
||||
{
|
||||
return chunkCache;
|
||||
}
|
||||
|
||||
public IOOptions getIOOptions()
|
||||
{
|
||||
return ioOptions;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -15,10 +15,12 @@
|
|||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.apache.cassandra.io.sstable.format;
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.rows.Row;
|
||||
import org.apache.cassandra.db.rows.Unfiltered;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
|
||||
/**
|
||||
* Observer for events in the lifecycle of writing out an sstable.
|
||||
|
|
@ -34,10 +36,22 @@ public interface SSTableFlushObserver
|
|||
* Called when a new partition in being written to the sstable,
|
||||
* but before any cells are processed (see {@link #nextUnfilteredCluster(Unfiltered)}).
|
||||
*
|
||||
* @param key The key being appended to SSTable.
|
||||
* @param indexPosition The position of the key in the SSTable PRIMARY_INDEX file.
|
||||
* @param key the key being appended to SSTable.
|
||||
* @param keyPosition the position of the key in the SSTable data file
|
||||
* @param KeyPositionForSASI SSTable format specific key position for storage attached indexes, it can be
|
||||
* in data file or in some index file. It is the same position as returned by
|
||||
* {@link KeyReader#keyPositionForSecondaryIndex()} for the same format, and the same
|
||||
* position as expected by {@link SSTableReader#keyAtPositionFromSecondaryIndex(long)}.
|
||||
*/
|
||||
void startPartition(DecoratedKey key, long indexPosition);
|
||||
void startPartition(DecoratedKey key, long keyPosition, long KeyPositionForSASI);
|
||||
|
||||
/**
|
||||
* Called when a static row is being written to the sstable. If static columns are present in the table, it is called
|
||||
* after {@link #startPartition(DecoratedKey, long, long)} and before any calls to {@link #nextUnfilteredCluster(Unfiltered)}.
|
||||
*
|
||||
* @param staticRow static row appended to the sstable, can be empty, may not be {@code null}
|
||||
*/
|
||||
void staticRow(Row staticRow);
|
||||
|
||||
/**
|
||||
* Called after the unfiltered cluster is written to the sstable.
|
||||
|
|
@ -52,4 +66,4 @@ public interface SSTableFlushObserver
|
|||
* Called when all data is written to the file and it's ready to be finished up.
|
||||
*/
|
||||
void complete();
|
||||
}
|
||||
}
|
||||
|
|
@ -35,7 +35,6 @@ import java.util.function.Supplier;
|
|||
import java.util.stream.Collectors;
|
||||
import java.util.stream.Stream;
|
||||
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
|
|
@ -54,8 +53,11 @@ import org.apache.cassandra.db.marshal.MapType;
|
|||
import org.apache.cassandra.db.marshal.SetType;
|
||||
import org.apache.cassandra.db.marshal.TupleType;
|
||||
import org.apache.cassandra.db.marshal.UserType;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat.Components;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat.Components.Types;
|
||||
import org.apache.cassandra.io.sstable.metadata.MetadataComponent;
|
||||
import org.apache.cassandra.io.sstable.metadata.MetadataType;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.schema.ColumnMetadata;
|
||||
import org.apache.cassandra.schema.IndexMetadata;
|
||||
import org.apache.cassandra.schema.Schema;
|
||||
|
|
@ -301,7 +303,7 @@ public abstract class SSTableHeaderFix
|
|||
.filter(p -> {
|
||||
try
|
||||
{
|
||||
return Descriptor.fromFilenameWithComponent(new File(p)).right.type == Component.Type.DATA;
|
||||
return Descriptor.fromFileWithComponent(new File(p)).right.type == Types.DATA;
|
||||
}
|
||||
catch (IllegalArgumentException t)
|
||||
{
|
||||
|
|
@ -309,8 +311,8 @@ public abstract class SSTableHeaderFix
|
|||
return false;
|
||||
}
|
||||
})
|
||||
.map(Path::toString)
|
||||
.map(Descriptor::fromFilename)
|
||||
.map(File::new)
|
||||
.map(file -> Descriptor.fromFileWithComponent(file, false).left)
|
||||
.forEach(descriptors::add);
|
||||
}
|
||||
|
||||
|
|
@ -342,8 +344,8 @@ public abstract class SSTableHeaderFix
|
|||
return;
|
||||
}
|
||||
|
||||
Set<Component> components = SSTable.discoverComponentsFor(desc);
|
||||
if (components.stream().noneMatch(c -> c.type == Component.Type.STATS))
|
||||
Set<Component> components = desc.discoverComponents();
|
||||
if (components.stream().noneMatch(c -> c.type == Types.STATS))
|
||||
{
|
||||
error("sstable %s has no -Statistics.db component.", desc);
|
||||
return;
|
||||
|
|
@ -845,7 +847,7 @@ public abstract class SSTableHeaderFix
|
|||
|
||||
private void writeNewMetadata(Descriptor desc, Map<MetadataType, MetadataComponent> newMetadata)
|
||||
{
|
||||
String file = desc.filenameFor(Component.STATS);
|
||||
File file = desc.fileFor(Components.STATS);
|
||||
info.accept(String.format(" Writing new metadata file %s", file));
|
||||
try
|
||||
{
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@ import org.apache.cassandra.io.util.File;
|
|||
* A new implementation must adhere to the following invariants:
|
||||
* - Must be locally sortable - that is, the comparison must reflect the comparison of generation times of identifiers
|
||||
* generated on the same node
|
||||
* - String representation must *not* include the {@link Descriptor#FILENAME_SEPARATOR} character, see {@link Descriptor#fromFilenameWithComponent(File)}
|
||||
* - String representation must *not* include the {@link Descriptor#FILENAME_SEPARATOR} character, see {@link Descriptor#fromFileWithComponent(File)}
|
||||
* - must be case-insensitive because the sstables can be stored on case-insensitive file system
|
||||
* <p>
|
||||
*/
|
||||
|
|
@ -50,7 +50,7 @@ public interface SSTableId
|
|||
* {@link Builder#fromString(String)}
|
||||
* <p>
|
||||
* Must not contain any {@link Descriptor#FILENAME_SEPARATOR} character as it is used in the Descriptor
|
||||
* see {@link Descriptor#fromFilenameWithComponent(File)}
|
||||
* see {@link Descriptor#fromFileWithComponent(File)}
|
||||
*/
|
||||
@Override
|
||||
String toString();
|
||||
|
|
|
|||
|
|
@ -17,17 +17,24 @@
|
|||
*/
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import org.apache.cassandra.db.*;
|
||||
import org.apache.cassandra.db.rows.*;
|
||||
import java.io.IOError;
|
||||
import java.io.IOException;
|
||||
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.DeletionTime;
|
||||
import org.apache.cassandra.db.RegularAndStaticColumns;
|
||||
import org.apache.cassandra.db.UnfilteredValidation;
|
||||
import org.apache.cassandra.db.rows.DeserializationHelper;
|
||||
import org.apache.cassandra.db.rows.EncodingStats;
|
||||
import org.apache.cassandra.db.rows.Row;
|
||||
import org.apache.cassandra.db.rows.Unfiltered;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.util.FileDataInput;
|
||||
import org.apache.cassandra.io.util.RandomAccessReader;
|
||||
import org.apache.cassandra.schema.TableMetadata;
|
||||
import org.apache.cassandra.utils.ByteBufferUtil;
|
||||
|
||||
import java.io.IOError;
|
||||
import java.io.IOException;
|
||||
|
||||
import static org.apache.cassandra.utils.vint.VIntCoding.VIntOutOfRangeException;
|
||||
|
||||
public class SSTableIdentityIterator implements Comparable<SSTableIdentityIterator>, UnfilteredRowIterator
|
||||
|
|
@ -72,13 +79,16 @@ public class SSTableIdentityIterator implements Comparable<SSTableIdentityIterat
|
|||
}
|
||||
|
||||
@SuppressWarnings("resource")
|
||||
public static SSTableIdentityIterator create(SSTableReader sstable, FileDataInput dfile, RowIndexEntry<?> indexEntry, DecoratedKey key, boolean tombstoneOnly)
|
||||
public static SSTableIdentityIterator create(SSTableReader sstable, FileDataInput dfile, long dataPosition, DecoratedKey key, boolean tombstoneOnly)
|
||||
{
|
||||
try
|
||||
{
|
||||
dfile.seek(indexEntry.position);
|
||||
dfile.seek(dataPosition);
|
||||
ByteBufferUtil.skipShortLength(dfile); // Skip partition key
|
||||
DeletionTime partitionLevelDeletion = DeletionTime.serializer.deserialize(dfile);
|
||||
if (!partitionLevelDeletion.validate())
|
||||
UnfilteredValidation.handleInvalid(sstable.metadata(), key, sstable, "partitionLevelDeletion="+partitionLevelDeletion.toString());
|
||||
|
||||
DeserializationHelper helper = new DeserializationHelper(sstable.metadata(), sstable.descriptor.version.correspondingMessagingVersion(), DeserializationHelper.Flag.LOCAL);
|
||||
SSTableSimpleIterator iterator = tombstoneOnly
|
||||
? SSTableSimpleIterator.createTombstoneOnly(sstable.metadata(), dfile, sstable.header, helper, partitionLevelDeletion)
|
||||
|
|
|
|||
|
|
@ -17,25 +17,43 @@
|
|||
*/
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.Collections;
|
||||
import java.util.HashMap;
|
||||
import java.util.HashSet;
|
||||
import java.util.Iterator;
|
||||
import java.util.LinkedList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import com.google.common.collect.HashMultimap;
|
||||
import com.google.common.collect.Multimap;
|
||||
|
||||
import org.apache.cassandra.db.streaming.CassandraOutgoingFile;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.locator.InetAddressAndPort;
|
||||
import org.apache.cassandra.io.FSError;
|
||||
import org.apache.cassandra.schema.TableMetadataRef;
|
||||
import org.apache.cassandra.db.Directories;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
|
||||
import org.apache.cassandra.db.streaming.CassandraOutgoingFile;
|
||||
import org.apache.cassandra.dht.Range;
|
||||
import org.apache.cassandra.dht.Token;
|
||||
import org.apache.cassandra.io.FSError;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat.Components;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.streaming.*;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.locator.InetAddressAndPort;
|
||||
import org.apache.cassandra.schema.TableMetadataRef;
|
||||
import org.apache.cassandra.streaming.OutgoingStream;
|
||||
import org.apache.cassandra.streaming.PreviewKind;
|
||||
import org.apache.cassandra.streaming.StreamEvent;
|
||||
import org.apache.cassandra.streaming.StreamEventHandler;
|
||||
import org.apache.cassandra.streaming.StreamOperation;
|
||||
import org.apache.cassandra.streaming.StreamPlan;
|
||||
import org.apache.cassandra.streaming.StreamResultFuture;
|
||||
import org.apache.cassandra.streaming.StreamState;
|
||||
import org.apache.cassandra.streaming.StreamingChannel;
|
||||
import org.apache.cassandra.utils.OutputHandler;
|
||||
import org.apache.cassandra.utils.Pair;
|
||||
|
||||
import org.apache.cassandra.utils.concurrent.Ref;
|
||||
|
||||
import static org.apache.cassandra.streaming.StreamingChannel.Factory.Global.streamingFactory;
|
||||
|
|
@ -55,7 +73,6 @@ public class SSTableLoader implements StreamEventHandler
|
|||
private final Set<InetAddressAndPort> failedHosts = new HashSet<>();
|
||||
|
||||
private final List<SSTableReader> sstables = new ArrayList<>();
|
||||
private final Multimap<InetAddressAndPort, OutgoingStream> streamingDetails = HashMultimap.create();
|
||||
|
||||
public SSTableLoader(File directory, Client client, OutputHandler outputHandler)
|
||||
{
|
||||
|
|
@ -78,10 +95,11 @@ public class SSTableLoader implements StreamEventHandler
|
|||
}
|
||||
|
||||
@SuppressWarnings("resource")
|
||||
protected Collection<SSTableReader> openSSTables(final Map<InetAddressAndPort, Collection<Range<Token>>> ranges)
|
||||
private Multimap<InetAddressAndPort, CassandraOutgoingFile> openSSTables(final Map<InetAddressAndPort, Collection<Range<Token>>> ranges)
|
||||
{
|
||||
outputHandler.output("Opening sstables and calculating sections to stream");
|
||||
|
||||
Multimap<InetAddressAndPort, CassandraOutgoingFile> streamingDetails = HashMultimap.create();
|
||||
LifecycleTransaction.getFiles(directory.toPath(),
|
||||
(file, type) ->
|
||||
{
|
||||
|
|
@ -105,13 +123,16 @@ public class SSTableLoader implements StreamEventHandler
|
|||
}
|
||||
|
||||
Descriptor desc = p == null ? null : p.left;
|
||||
if (p == null || !p.right.equals(Component.DATA))
|
||||
if (p == null || !p.right.equals(Components.DATA))
|
||||
return false;
|
||||
|
||||
if (!new File(desc.filenameFor(Component.PRIMARY_INDEX)).exists())
|
||||
for (Component c : desc.getFormat().primaryComponents())
|
||||
{
|
||||
outputHandler.output(String.format("Skipping file %s because index is missing", name));
|
||||
return false;
|
||||
if (!desc.fileFor(c).exists())
|
||||
{
|
||||
outputHandler.output(String.format("Skipping file %s because %s is missing", name, c.name));
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
TableMetadataRef metadata = client.getTableMetadata(desc.cfname);
|
||||
|
|
@ -121,22 +142,14 @@ public class SSTableLoader implements StreamEventHandler
|
|||
return false;
|
||||
}
|
||||
|
||||
Set<Component> components = new HashSet<>();
|
||||
components.add(Component.DATA);
|
||||
components.add(Component.PRIMARY_INDEX);
|
||||
if (new File(desc.filenameFor(Component.SUMMARY)).exists())
|
||||
components.add(Component.SUMMARY);
|
||||
if (new File(desc.filenameFor(Component.COMPRESSION_INFO)).exists())
|
||||
components.add(Component.COMPRESSION_INFO);
|
||||
if (new File(desc.filenameFor(Component.STATS)).exists())
|
||||
components.add(Component.STATS);
|
||||
Set<Component> components = desc.getComponents(desc.getFormat().primaryComponents(), desc.getFormat().uploadComponents());
|
||||
|
||||
try
|
||||
{
|
||||
// To conserve memory, open SSTableReaders without bloom filters and discard
|
||||
// the index summary after calculating the file sections to stream and the estimated
|
||||
// number of keys for each endpoint. See CASSANDRA-5555 for details.
|
||||
SSTableReader sstable = SSTableReader.openForBatch(desc, components, metadata);
|
||||
SSTableReader sstable = SSTableReader.openForBatch(null, desc, components, metadata);
|
||||
sstables.add(sstable);
|
||||
|
||||
// calculate the sstable sections to stream as well as the estimated number of
|
||||
|
|
@ -154,12 +167,12 @@ public class SSTableLoader implements StreamEventHandler
|
|||
|
||||
long estimatedKeys = sstable.estimatedKeysForRanges(tokenRanges);
|
||||
Ref<SSTableReader> ref = sstable.ref();
|
||||
OutgoingStream stream = new CassandraOutgoingFile(StreamOperation.BULK_LOAD, ref, sstableSections, tokenRanges, estimatedKeys);
|
||||
CassandraOutgoingFile stream = new CassandraOutgoingFile(StreamOperation.BULK_LOAD, ref, sstableSections, tokenRanges, estimatedKeys);
|
||||
streamingDetails.put(endpoint, stream);
|
||||
}
|
||||
|
||||
// to conserve heap space when bulk loading
|
||||
sstable.releaseSummary();
|
||||
sstable.releaseInMemoryComponents();
|
||||
}
|
||||
catch (FSError e)
|
||||
{
|
||||
|
|
@ -170,7 +183,7 @@ public class SSTableLoader implements StreamEventHandler
|
|||
},
|
||||
Directories.OnTxnErr.IGNORE);
|
||||
|
||||
return sstables;
|
||||
return streamingDetails;
|
||||
}
|
||||
|
||||
public StreamResultFuture stream()
|
||||
|
|
@ -186,14 +199,14 @@ public class SSTableLoader implements StreamEventHandler
|
|||
StreamPlan plan = new StreamPlan(StreamOperation.BULK_LOAD, connectionsPerHost, false, null, PreviewKind.NONE).connectionFactory(client.getConnectionFactory());
|
||||
|
||||
Map<InetAddressAndPort, Collection<Range<Token>>> endpointToRanges = client.getEndpointToRangesMap();
|
||||
openSSTables(endpointToRanges);
|
||||
if (sstables.isEmpty())
|
||||
Multimap<InetAddressAndPort, CassandraOutgoingFile> streamingDetails = openSSTables(endpointToRanges);
|
||||
if (streamingDetails.isEmpty())
|
||||
{
|
||||
// return empty result
|
||||
return plan.execute();
|
||||
}
|
||||
|
||||
outputHandler.output(String.format("Streaming relevant part of %s to %s", names(sstables), endpointToRanges.keySet()));
|
||||
outputHandler.output(String.format("Streaming relevant part of %s to %s", names(streamingDetails.values()), endpointToRanges.keySet()));
|
||||
|
||||
for (Map.Entry<InetAddressAndPort, Collection<Range<Token>>> entry : endpointToRanges.entrySet())
|
||||
{
|
||||
|
|
@ -201,13 +214,8 @@ public class SSTableLoader implements StreamEventHandler
|
|||
if (toIgnore.contains(remote))
|
||||
continue;
|
||||
|
||||
List<OutgoingStream> streams = new LinkedList<>();
|
||||
|
||||
// references are acquired when constructing the SSTableStreamingSections above
|
||||
for (OutgoingStream stream : streamingDetails.get(remote))
|
||||
{
|
||||
streams.add(stream);
|
||||
}
|
||||
List<OutgoingStream> streams = new LinkedList<>(streamingDetails.get(remote));
|
||||
|
||||
plan.transferStreams(remote, streams);
|
||||
}
|
||||
|
|
@ -229,10 +237,13 @@ public class SSTableLoader implements StreamEventHandler
|
|||
*/
|
||||
private void releaseReferences()
|
||||
{
|
||||
for (SSTableReader sstable : sstables)
|
||||
Iterator<SSTableReader> it = sstables.iterator();
|
||||
while (it.hasNext())
|
||||
{
|
||||
SSTableReader sstable = it.next();
|
||||
sstable.selfRef().release();
|
||||
assert sstable.selfRef().globalCount() == 0 : String.format("for sstable = %s, ref count = %d", sstable, sstable.selfRef().globalCount());
|
||||
it.remove();
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -246,12 +257,9 @@ public class SSTableLoader implements StreamEventHandler
|
|||
}
|
||||
}
|
||||
|
||||
private String names(Collection<SSTableReader> sstables)
|
||||
private String names(Collection<CassandraOutgoingFile> sstables)
|
||||
{
|
||||
StringBuilder builder = new StringBuilder();
|
||||
for (SSTableReader sstable : sstables)
|
||||
builder.append(sstable.descriptor.filenameFor(Component.DATA)).append(" ");
|
||||
return builder.toString();
|
||||
return sstables.stream().map(CassandraOutgoingFile::getName).distinct().collect(Collectors.joining(" "));
|
||||
}
|
||||
|
||||
public Set<InetAddressAndPort> getFailedHosts()
|
||||
|
|
|
|||
|
|
@ -15,9 +15,10 @@
|
|||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.apache.cassandra.io.sstable.format;
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import org.apache.cassandra.db.RowIndexEntry;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.tracing.Tracing;
|
||||
|
||||
/**
|
||||
* Listener for receiving notifications associated with reading SSTables.
|
||||
|
|
@ -29,10 +30,22 @@ public interface SSTableReadsListener
|
|||
*/
|
||||
enum SkippingReason
|
||||
{
|
||||
BLOOM_FILTER,
|
||||
MIN_MAX_KEYS,
|
||||
PARTITION_INDEX_LOOKUP,
|
||||
INDEX_ENTRY_NOT_FOUND;
|
||||
BLOOM_FILTER("Bloom filter allows skipping sstable {}"),
|
||||
MIN_MAX_KEYS("Check against min and max keys allows skipping sstable {}"),
|
||||
PARTITION_INDEX_LOOKUP("Partition index lookup allows skipping sstable {}"),
|
||||
INDEX_ENTRY_NOT_FOUND("Partition index lookup complete (bloom filter false positive) for sstable {}");
|
||||
|
||||
private final String message;
|
||||
|
||||
SkippingReason(String message)
|
||||
{
|
||||
this.message = message;
|
||||
}
|
||||
|
||||
public void trace(Descriptor descriptor)
|
||||
{
|
||||
Tracing.trace(message, descriptor.id);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -40,8 +53,20 @@ public interface SSTableReadsListener
|
|||
*/
|
||||
enum SelectionReason
|
||||
{
|
||||
KEY_CACHE_HIT,
|
||||
INDEX_ENTRY_FOUND;
|
||||
KEY_CACHE_HIT("Key cache hit for sstable {}, size = {}"),
|
||||
INDEX_ENTRY_FOUND("Partition index found for sstable {}, size = {}");
|
||||
|
||||
private final String message;
|
||||
|
||||
SelectionReason(String message)
|
||||
{
|
||||
this.message = message;
|
||||
}
|
||||
|
||||
public void trace(Descriptor descriptor, AbstractRowIndexEntry entry)
|
||||
{
|
||||
Tracing.trace(message, descriptor.id, entry.columnsIndexCount());
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -66,7 +91,7 @@ public interface SSTableReadsListener
|
|||
* @param indexEntry the index entry
|
||||
* @param reason the reason for which the SSTable has been selected
|
||||
*/
|
||||
default void onSSTableSelected(SSTableReader sstable, RowIndexEntry<?> indexEntry, SelectionReason reason)
|
||||
default void onSSTableSelected(SSTableReader sstable, SelectionReason reason)
|
||||
{
|
||||
}
|
||||
|
||||
|
|
@ -17,22 +17,18 @@
|
|||
*/
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import java.lang.ref.WeakReference;
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
|
||||
import org.apache.cassandra.cache.InstrumentingCache;
|
||||
import org.apache.cassandra.cache.KeyCacheKey;
|
||||
import org.apache.cassandra.config.DatabaseDescriptor;
|
||||
import org.apache.cassandra.db.ColumnFamilyStore;
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.RowIndexEntry;
|
||||
import org.apache.cassandra.db.lifecycle.ILifecycleTransaction;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableWriter;
|
||||
import org.apache.cassandra.utils.NativeLibrary;
|
||||
import org.apache.cassandra.utils.concurrent.Transactional;
|
||||
|
||||
/**
|
||||
|
|
@ -43,12 +39,12 @@ import org.apache.cassandra.utils.concurrent.Transactional;
|
|||
* for on-close (i.e. when all references expire) that drops the page cache prior to that key position
|
||||
*
|
||||
* hard-links are created for each partially written sstable so that readers opened against them continue to work past
|
||||
* the rename of the temporary file, which is deleted once all readers against the hard-link have been closed.
|
||||
* renaming of the temporary file, which is deleted once all readers against the hard-link have been closed.
|
||||
* If for any reason the writer is rolled over, we immediately rename and fully expose the completed file in the Tracker.
|
||||
*
|
||||
* On abort we restore the original lower bounds to the existing readers and delete any temporary files we had in progress,
|
||||
* but leave any hard-links in place for the readers we opened to cleanup when they're finished as we would had we finished
|
||||
* successfully.
|
||||
* On abort, we restore the original lower bounds to the existing readers and delete any temporary files we had in progress,
|
||||
* but leave any hard-links in place for the readers we opened, and clean-up when the readers finish as we would do
|
||||
* if we had finished successfully.
|
||||
*/
|
||||
public class SSTableRewriter extends Transactional.AbstractTransactional implements Transactional
|
||||
{
|
||||
|
|
@ -69,7 +65,6 @@ public class SSTableRewriter extends Transactional.AbstractTransactional impleme
|
|||
private final boolean eagerWriterMetaRelease; // true if the writer metadata should be released when switch is called
|
||||
|
||||
private SSTableWriter writer;
|
||||
private Map<DecoratedKey, RowIndexEntry> cachedKeys = new HashMap<>();
|
||||
|
||||
// for testing (TODO: remove when have byteman setup)
|
||||
private boolean throwEarly, throwLate;
|
||||
|
|
@ -117,31 +112,16 @@ public class SSTableRewriter extends Transactional.AbstractTransactional impleme
|
|||
return writer;
|
||||
}
|
||||
|
||||
public RowIndexEntry append(UnfilteredRowIterator partition)
|
||||
public AbstractRowIndexEntry append(UnfilteredRowIterator partition)
|
||||
{
|
||||
// we do this before appending to ensure we can resetAndTruncate() safely if the append fails
|
||||
// we do this before appending to ensure we can resetAndTruncate() safely if appending fails
|
||||
DecoratedKey key = partition.partitionKey();
|
||||
maybeReopenEarly(key);
|
||||
RowIndexEntry index = writer.append(partition);
|
||||
if (DatabaseDescriptor.shouldMigrateKeycacheOnCompaction())
|
||||
{
|
||||
if (!transaction.isOffline() && index != null)
|
||||
{
|
||||
for (SSTableReader reader : transaction.originals())
|
||||
{
|
||||
if (reader.getCachedPosition(key, false) != null)
|
||||
{
|
||||
cachedKeys.put(key, index);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return index;
|
||||
return writer.append(partition);
|
||||
}
|
||||
|
||||
// attempts to append the row, if fails resets the writer position
|
||||
public RowIndexEntry tryAppend(UnfilteredRowIterator partition)
|
||||
public AbstractRowIndexEntry tryAppend(UnfilteredRowIterator partition)
|
||||
{
|
||||
writer.mark();
|
||||
try
|
||||
|
|
@ -163,20 +143,18 @@ public class SSTableRewriter extends Transactional.AbstractTransactional impleme
|
|||
{
|
||||
for (SSTableReader reader : transaction.originals())
|
||||
{
|
||||
RowIndexEntry index = reader.getPosition(key, SSTableReader.Operator.GE);
|
||||
NativeLibrary.trySkipCache(reader.getFilename(), 0, index == null ? 0 : index.position);
|
||||
reader.trySkipFileCacheBefore(key);
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
SSTableReader reader = writer.setMaxDataAge(maxAge).openEarly();
|
||||
if (reader != null)
|
||||
{
|
||||
writer.setMaxDataAge(maxAge);
|
||||
writer.openEarly(reader -> {
|
||||
transaction.update(reader, false);
|
||||
currentlyOpenedEarlyAt = writer.getFilePointer();
|
||||
moveStarts(reader, reader.last);
|
||||
moveStarts(reader.last);
|
||||
transaction.checkpoint();
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -210,27 +188,13 @@ public class SSTableRewriter extends Transactional.AbstractTransactional impleme
|
|||
* one, the old *instance* will have reference count == 0 and if we were to start a new compaction with that old
|
||||
* instance, we would get exceptions.
|
||||
*
|
||||
* @param newReader the rewritten reader that replaces them for this region
|
||||
* @param lowerbound if !reset, must be non-null, and marks the exclusive lowerbound of the start for each sstable
|
||||
*/
|
||||
private void moveStarts(SSTableReader newReader, DecoratedKey lowerbound)
|
||||
private void moveStarts(DecoratedKey lowerbound)
|
||||
{
|
||||
if (transaction.isOffline() || preemptiveOpenInterval == Long.MAX_VALUE)
|
||||
return;
|
||||
|
||||
newReader.setupOnline();
|
||||
List<DecoratedKey> invalidateKeys = null;
|
||||
if (!cachedKeys.isEmpty())
|
||||
{
|
||||
invalidateKeys = new ArrayList<>(cachedKeys.size());
|
||||
for (Map.Entry<DecoratedKey, RowIndexEntry> cacheKey : cachedKeys.entrySet())
|
||||
{
|
||||
invalidateKeys.add(cacheKey.getKey());
|
||||
newReader.cacheKey(cacheKey.getKey(), cacheKey.getValue());
|
||||
}
|
||||
}
|
||||
|
||||
cachedKeys.clear();
|
||||
for (SSTableReader sstable : transaction.originals())
|
||||
{
|
||||
// we call getCurrentReplacement() to support multiple rewriters operating over the same source readers at once.
|
||||
|
|
@ -241,49 +205,19 @@ public class SSTableRewriter extends Transactional.AbstractTransactional impleme
|
|||
if (latest.first.compareTo(lowerbound) > 0)
|
||||
continue;
|
||||
|
||||
Runnable runOnClose = invalidateKeys != null ? new InvalidateKeys(latest, invalidateKeys) : null;
|
||||
if (lowerbound.compareTo(latest.last) >= 0)
|
||||
{
|
||||
if (!transaction.isObsolete(latest))
|
||||
{
|
||||
if (runOnClose != null)
|
||||
{
|
||||
latest.runOnClose(runOnClose);
|
||||
}
|
||||
transaction.obsolete(latest);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
DecoratedKey newStart = latest.firstKeyBeyond(lowerbound);
|
||||
assert newStart != null;
|
||||
SSTableReader replacement = latest.cloneWithNewStart(newStart, runOnClose);
|
||||
transaction.update(replacement, true);
|
||||
}
|
||||
}
|
||||
|
||||
private static final class InvalidateKeys implements Runnable
|
||||
{
|
||||
final List<KeyCacheKey> cacheKeys = new ArrayList<>();
|
||||
final WeakReference<InstrumentingCache<KeyCacheKey, ?>> cacheRef;
|
||||
|
||||
private InvalidateKeys(SSTableReader reader, Collection<DecoratedKey> invalidate)
|
||||
{
|
||||
this.cacheRef = new WeakReference<>(reader.getKeyCache());
|
||||
if (cacheRef.get() != null)
|
||||
if (!transaction.isObsolete(latest))
|
||||
{
|
||||
for (DecoratedKey key : invalidate)
|
||||
cacheKeys.add(reader.getCacheKey(key));
|
||||
}
|
||||
}
|
||||
|
||||
public void run()
|
||||
{
|
||||
for (KeyCacheKey key : cacheKeys)
|
||||
{
|
||||
InstrumentingCache<KeyCacheKey, ?> cache = cacheRef.get();
|
||||
if (cache != null)
|
||||
cache.remove(key);
|
||||
DecoratedKey newStart = latest.firstKeyBeyond(lowerbound);
|
||||
assert newStart != null;
|
||||
SSTableReader replacement = latest.cloneWithNewStart(newStart);
|
||||
transaction.update(replacement, true);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -291,7 +225,10 @@ public class SSTableRewriter extends Transactional.AbstractTransactional impleme
|
|||
public void switchWriter(SSTableWriter newWriter)
|
||||
{
|
||||
if (newWriter != null)
|
||||
writers.add(newWriter.setMaxDataAge(maxAge));
|
||||
{
|
||||
newWriter.setMaxDataAge(maxAge);
|
||||
writers.add(newWriter);
|
||||
}
|
||||
|
||||
if (eagerWriterMetaRelease && writer != null)
|
||||
writer.releaseMetadataOverhead();
|
||||
|
|
@ -310,12 +247,15 @@ public class SSTableRewriter extends Transactional.AbstractTransactional impleme
|
|||
return;
|
||||
}
|
||||
|
||||
// Open fully completed sstables early. This is also required for the final sstable in a set (where newWriter
|
||||
// is null) to permit the compilation of a canonical set of sstables (see View.select).
|
||||
if (preemptiveOpenInterval != Long.MAX_VALUE)
|
||||
{
|
||||
// we leave it as a tmp file, but we open it and add it to the Tracker
|
||||
SSTableReader reader = writer.setMaxDataAge(maxAge).openFinalEarly();
|
||||
writer.setMaxDataAge(maxAge);
|
||||
SSTableReader reader = writer.openFinalEarly();
|
||||
transaction.update(reader, false);
|
||||
moveStarts(reader, reader.last);
|
||||
moveStarts(reader.last);
|
||||
transaction.checkpoint();
|
||||
}
|
||||
|
||||
|
|
@ -370,7 +310,9 @@ public class SSTableRewriter extends Transactional.AbstractTransactional impleme
|
|||
for (SSTableWriter writer : writers)
|
||||
{
|
||||
assert writer.getFilePointer() > 0;
|
||||
writer.setRepairedAt(repairedAt).setOpenResult(true).prepareToCommit();
|
||||
writer.setRepairedAt(repairedAt);
|
||||
writer.setOpenResult(true);
|
||||
writer.prepareToCommit();
|
||||
SSTableReader reader = writer.finished();
|
||||
transaction.update(reader, false);
|
||||
preparedForCommit.add(reader);
|
||||
|
|
|
|||
|
|
@ -214,7 +214,7 @@ class SSTableSimpleUnsortedWriter extends AbstractSSTableSimpleWriter
|
|||
if (b == SENTINEL)
|
||||
return;
|
||||
|
||||
try (SSTableTxnWriter writer = createWriter())
|
||||
try (SSTableTxnWriter writer = createWriter(null))
|
||||
{
|
||||
for (Map.Entry<DecoratedKey, PartitionUpdate.Builder> entry : b.entrySet())
|
||||
writer.append(entry.getValue().build().unfilteredIterator());
|
||||
|
|
|
|||
|
|
@ -21,7 +21,8 @@ import java.io.IOException;
|
|||
|
||||
import com.google.common.base.Throwables;
|
||||
|
||||
import org.apache.cassandra.db.*;
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.RegularAndStaticColumns;
|
||||
import org.apache.cassandra.db.partitions.PartitionUpdate;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.apache.cassandra.schema.TableMetadataRef;
|
||||
|
|
@ -51,7 +52,7 @@ class SSTableSimpleWriter extends AbstractSSTableSimpleWriter
|
|||
private SSTableTxnWriter getOrCreateWriter() throws IOException
|
||||
{
|
||||
if (writer == null)
|
||||
writer = createWriter();
|
||||
writer = createWriter(null);
|
||||
|
||||
return writer;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -28,7 +28,6 @@ import org.apache.cassandra.db.compaction.OperationType;
|
|||
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.index.Index;
|
||||
import org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableFormat;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
|
||||
|
|
@ -144,12 +143,13 @@ public class SSTableTxnWriter extends Transactional.AbstractTransactional implem
|
|||
boolean isTransient,
|
||||
int sstableLevel,
|
||||
SerializationHeader header,
|
||||
Collection<Index> indexes)
|
||||
Collection<Index> indexes,
|
||||
SSTable.Owner owner)
|
||||
{
|
||||
// if the column family store does not exist, we create a new default SSTableMultiWriter to use:
|
||||
LifecycleTransaction txn = LifecycleTransaction.offline(OperationType.WRITE);
|
||||
MetadataCollector collector = new MetadataCollector(metadata.get().comparator).sstableLevel(sstableLevel);
|
||||
SSTableMultiWriter writer = SimpleSSTableMultiWriter.create(descriptor, keyCount, repairedAt, pendingRepair, isTransient, metadata, collector, header, indexes, txn);
|
||||
SSTableMultiWriter writer = SimpleSSTableMultiWriter.create(descriptor, keyCount, repairedAt, pendingRepair, isTransient, metadata, collector, header, indexes, txn, owner);
|
||||
return new SSTableTxnWriter(txn, writer);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -15,89 +15,81 @@
|
|||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.apache.cassandra.io.sstable.format.big;
|
||||
package org.apache.cassandra.io.sstable;
|
||||
|
||||
import java.io.EOFException;
|
||||
import java.io.IOException;
|
||||
import java.nio.channels.ClosedChannelException;
|
||||
import java.util.Collection;
|
||||
import java.util.EnumMap;
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
|
||||
import com.google.common.collect.ImmutableList;
|
||||
import com.google.common.collect.ImmutableSet;
|
||||
import com.google.common.collect.Sets;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleNewTracker;
|
||||
import org.apache.cassandra.io.util.File;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import org.apache.cassandra.config.DatabaseDescriptor;
|
||||
import org.apache.cassandra.db.DecoratedKey;
|
||||
import org.apache.cassandra.db.lifecycle.LifecycleNewTracker;
|
||||
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
|
||||
import org.apache.cassandra.dht.AbstractBounds;
|
||||
import org.apache.cassandra.dht.Token;
|
||||
import org.apache.cassandra.io.FSWriteError;
|
||||
import org.apache.cassandra.io.compress.BufferType;
|
||||
import org.apache.cassandra.io.sstable.Component;
|
||||
import org.apache.cassandra.io.sstable.Descriptor;
|
||||
import org.apache.cassandra.io.sstable.SSTable;
|
||||
import org.apache.cassandra.io.sstable.SSTableMultiWriter;
|
||||
import org.apache.cassandra.io.sstable.format.SSTableReader;
|
||||
import org.apache.cassandra.io.util.DataInputPlus;
|
||||
import org.apache.cassandra.io.util.SequentialWriter;
|
||||
import org.apache.cassandra.io.util.SequentialWriterOption;
|
||||
import org.apache.cassandra.net.AsyncStreamingInputPlus;
|
||||
import org.apache.cassandra.schema.TableId;
|
||||
import org.apache.cassandra.schema.TableMetadataRef;
|
||||
|
||||
import static java.lang.String.format;
|
||||
import static org.apache.cassandra.utils.FBUtilities.prettyPrintMemory;
|
||||
|
||||
public class BigTableZeroCopyWriter extends SSTable implements SSTableMultiWriter
|
||||
public class SSTableZeroCopyWriter extends SSTable implements SSTableMultiWriter
|
||||
{
|
||||
private static final Logger logger = LoggerFactory.getLogger(BigTableZeroCopyWriter.class);
|
||||
private static final Logger logger = LoggerFactory.getLogger(SSTableZeroCopyWriter.class);
|
||||
|
||||
private final TableMetadataRef metadata;
|
||||
private volatile SSTableReader finalReader;
|
||||
private final Map<Component.Type, SequentialWriter> componentWriters;
|
||||
|
||||
private static final SequentialWriterOption WRITER_OPTION =
|
||||
SequentialWriterOption.newBuilder()
|
||||
.trickleFsync(false)
|
||||
.bufferSize(2 << 20)
|
||||
.bufferType(BufferType.OFF_HEAP)
|
||||
.build();
|
||||
|
||||
private static final ImmutableSet<Component> SUPPORTED_COMPONENTS =
|
||||
ImmutableSet.of(Component.DATA,
|
||||
Component.PRIMARY_INDEX,
|
||||
Component.SUMMARY,
|
||||
Component.STATS,
|
||||
Component.COMPRESSION_INFO,
|
||||
Component.FILTER,
|
||||
Component.DIGEST,
|
||||
Component.CRC);
|
||||
|
||||
public BigTableZeroCopyWriter(Descriptor descriptor,
|
||||
TableMetadataRef metadata,
|
||||
LifecycleNewTracker lifecycleNewTracker,
|
||||
final Collection<Component> components)
|
||||
public SSTableZeroCopyWriter(Builder<?, ?> builder,
|
||||
LifecycleNewTracker lifecycleNewTracker,
|
||||
SSTable.Owner owner)
|
||||
{
|
||||
super(descriptor, ImmutableSet.copyOf(components), metadata, DatabaseDescriptor.getDiskOptimizationStrategy());
|
||||
super(builder, owner);
|
||||
|
||||
lifecycleNewTracker.trackNew(this);
|
||||
this.metadata = metadata;
|
||||
this.componentWriters = new EnumMap<>(Component.Type.class);
|
||||
this.componentWriters = new HashMap<>();
|
||||
|
||||
if (!SUPPORTED_COMPONENTS.containsAll(components))
|
||||
if (!descriptor.getFormat().streamingComponents().containsAll(components))
|
||||
throw new AssertionError(format("Unsupported streaming component detected %s",
|
||||
Sets.difference(ImmutableSet.copyOf(components), SUPPORTED_COMPONENTS)));
|
||||
Sets.difference(ImmutableSet.copyOf(components), descriptor.getFormat().streamingComponents())));
|
||||
|
||||
for (Component c : components)
|
||||
componentWriters.put(c.type, makeWriter(descriptor, c));
|
||||
}
|
||||
|
||||
private static SequentialWriter makeWriter(Descriptor descriptor, Component component)
|
||||
@Override
|
||||
public DecoratedKey getFirst()
|
||||
{
|
||||
return new SequentialWriter(new File(descriptor.filenameFor(component)), WRITER_OPTION, false);
|
||||
throw new UnsupportedOperationException();
|
||||
}
|
||||
|
||||
@Override
|
||||
public DecoratedKey getLast()
|
||||
{
|
||||
throw new UnsupportedOperationException();
|
||||
}
|
||||
|
||||
@Override
|
||||
public AbstractBounds<Token> getBounds()
|
||||
{
|
||||
throw new UnsupportedOperationException();
|
||||
}
|
||||
|
||||
private SequentialWriter makeWriter(Descriptor descriptor, Component component)
|
||||
{
|
||||
return new SequentialWriter(descriptor.fileFor(component), ioOptions.writerOptions, false);
|
||||
}
|
||||
|
||||
private void write(DataInputPlus in, long size, SequentialWriter out) throws FSWriteError
|
||||
|
|
@ -119,14 +111,14 @@ public class BigTableZeroCopyWriter extends SSTable implements SSTableMultiWrite
|
|||
}
|
||||
catch (IOException e)
|
||||
{
|
||||
throw new FSWriteError(e, out.getPath());
|
||||
throw new FSWriteError(e, out.getFile());
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean append(UnfilteredRowIterator partition)
|
||||
{
|
||||
throw new UnsupportedOperationException("Operation not supported by BigTableBlockWriter");
|
||||
throw new UnsupportedOperationException();
|
||||
}
|
||||
|
||||
@Override
|
||||
|
|
@ -150,7 +142,7 @@ public class BigTableZeroCopyWriter extends SSTable implements SSTableMultiWrite
|
|||
public Collection<SSTableReader> finished()
|
||||
{
|
||||
if (finalReader == null)
|
||||
finalReader = SSTableReader.open(descriptor, components, metadata);
|
||||
finalReader = SSTableReader.open(owner().orElse(null), descriptor, components, metadata);
|
||||
|
||||
return ImmutableList.of(finalReader);
|
||||
}
|
||||
|
|
@ -0,0 +1,267 @@
|
|||
# SSTable API
|
||||
|
||||
[CEP-17](https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-17%3A+SSTable+format+API)
|
||||
/ [CASSANDRA-17056](https://issues.apache.org/jira/browse/CASSANDRA-17056)
|
||||
|
||||
## Configuration specification
|
||||
|
||||
SSTable formats and options are specified under the `sstable_formats` key in the _cassandra.yaml_ configuration file.
|
||||
By default, the first sstable format implementation specified is the default one, which means that newly produced
|
||||
sstables will use that implementation. Other implementations are used to read the sstables.
|
||||
|
||||
```yaml
|
||||
sstable_formats:
|
||||
- class_name: 〈class name of the default SSTableFormat implementation〉
|
||||
parameters:
|
||||
id: 〈unique integer identifier of the implementation〉
|
||||
name: 〈unique string identifier of the implementation〉
|
||||
param1: 〈format specific parameter 1〉
|
||||
param2: 〈format specific parameter 2〉
|
||||
# ...
|
||||
- class_name: 〈class name of additional SSTableFormat implementation〉
|
||||
parameters:
|
||||
id: 〈unique integer identifier of the implementation〉
|
||||
name: 〈unique string identifier of the implementation〉
|
||||
param1: 〈format specific parameter 1〉
|
||||
param2: 〈format specific parameter 2〉
|
||||
# ...
|
||||
# ...
|
||||
```
|
||||
|
||||
Each sstable format definition includes the class name of the class implementing [`SSTableFormat`](format/SSTableFormat.java)
|
||||
interface and a map of parameters. Two parameters are mandatory - _id_ and _name_, and they have to be unique across
|
||||
all the defined formats. Additionally, they need to obey the following rules:
|
||||
- _id_ must be an integer between 0 and 127
|
||||
- _name_ must contain only lowercase ASCII letters
|
||||
|
||||
The _name_ parameter is the string put in the sstable file name. Therefore, it is the way Cassandra chooses the sstable
|
||||
format implementation when loading an sstable. The _id_ parameter is used in saved cache files. Both parameters should
|
||||
remain the same for the same format when upgrading. In particular, previous Cassandra versions have only
|
||||
[_big_ format](format/big/BigFormat.java) implementation, thus when upgrading from the old version, the configuration
|
||||
should keep _id_ set to _0_ and _name_ set to _big_ for that implementation (this is the default configuration), and any
|
||||
new implementations should use different values for those parameters.
|
||||
|
||||
The remaining parameters are optional and passed as a map to the `setup` method of a class implementing `SSTableFormat`.
|
||||
|
||||
Default configuration:
|
||||
```yaml
|
||||
sstable_formats:
|
||||
- class_name: org.apache.cassandra.io.sstable.format.big.BigFormat
|
||||
parameters:
|
||||
id: 0
|
||||
name: big
|
||||
```
|
||||
|
||||
Example configuration which uses `BtiFormat` as the default one keeping the right parameters for the `BigFormat`
|
||||
implementation.
|
||||
```yaml
|
||||
sstable_formats:
|
||||
- class_name: org.apache.cassandra.io.sstable.format.bti.BtiFormat
|
||||
parameters:
|
||||
id: 1
|
||||
name: bti
|
||||
- class_name: org.apache.cassandra.io.sstable.format.big.BigFormat
|
||||
parameters:
|
||||
id: 0
|
||||
name: big
|
||||
```
|
||||
|
||||
## Components
|
||||
|
||||
Each sstable consists of a set of components - required and optional. A component constitutes an identifier required
|
||||
to obtain the exact file with an sstable descriptor. Components are grouped by type. A type may define either
|
||||
a singleton component (for example, _stats_ component) or a non-singleton component (for example, _secondary index_
|
||||
component).
|
||||
|
||||
A set of generic types of components that are thought of as common to all the sstable implementations is defined in the
|
||||
[`SSTableFormat.Components`](format/SSTableFormat.java) class. They include singleton types like `DATA`,
|
||||
`COMPRESSION_INFO`, `STATS`, `FILTER`, `DIGEST`, `CRC`, and `TOC`, which comes with predefined singleton component
|
||||
instances, as well as non-singleton types like `SECONDARY_INDEX` and `CUSTOM`.
|
||||
|
||||
Apart from the generic components, each sstable format implementation may describe its specific component types.
|
||||
For example, the _big table_ format describes additionally `PRIMARY_INDEX` and `SUMMARY` singleton types and
|
||||
the corresponding singleton components (see [`BigFormat.Components`](format/big/BigFormat.java)).
|
||||
|
||||
Custom types can be created with one of the `Component.Type.create(name, repr, formatClass)`,
|
||||
`Component.Type.createSingleton(name, repr, formatClass)` methods. Each created type is registered in a global types'
|
||||
registry. Types registry is hierarchical which means that an sstable implementation may use types defined for its
|
||||
format class and for all parent format classes (for example, the types defined for the `BigFormat` class extend the set
|
||||
of types defined for the `SSTableFormat` interface).
|
||||
|
||||
For example, types defined for `BigFormat`:
|
||||
|
||||
```java
|
||||
public static class Types extends SSTableFormat.Components.Types
|
||||
{
|
||||
public static final Component.Type PRIMARY_INDEX = Component.Type.createSingleton("PRIMARY_INDEX", "Index.db", BigFormat.class);
|
||||
public static final Component.Type SUMMARY = Component.Type.createSingleton("SUMMARY", "Summary.db", BigFormat.class);
|
||||
}
|
||||
```
|
||||
|
||||
Singleton components are immediately associated with the singleton types and retrieved with the `<type>.getSingleton()`
|
||||
method:
|
||||
|
||||
```java
|
||||
public static class Components extends AbstractSSTableFormat.Components
|
||||
{
|
||||
public final static Component PRIMARY_INDEX = Types.PRIMARY_INDEX.getSingleton();
|
||||
public final static Component SUMMARY = Types.SUMMARY.getSingleton();
|
||||
}
|
||||
```
|
||||
|
||||
Non-singleton components are created explicitly as follows:
|
||||
|
||||
```java
|
||||
Component idx1 = Types.SECONDARY_INDEX.createComponent("SI_idx1.db");
|
||||
```
|
||||
|
||||
## Implementation
|
||||
|
||||
We strongly suggest the main format class to extend [`AbstractSSTableFormat`](format/AbstractSSTableFormat.java) because
|
||||
it includes the expected implementation of a couple of methods that should not be reimplemented differently.
|
||||
|
||||
### Initialization
|
||||
|
||||
Cassandra either initializes the sstable format class as a singleton by calling its constructor or obtains the instance
|
||||
by accessing a static field called `instance` in the class. As a part of the initialization, Cassandra calls the `setup`
|
||||
method and provides the configuration parameters. Right after initialization, Cassandra calls the `allComponents` method
|
||||
to confirm all the components defined for the format are initialized and usable.
|
||||
|
||||
### Predefined sets of components
|
||||
|
||||
SSTable format defines a couple of collections of components. You should declare those collections as constant and
|
||||
immutable sets.
|
||||
|
||||
### Reader
|
||||
|
||||
#### Construction
|
||||
|
||||
An sstable reader ([`SSTableReader`](format/SSTableReader.java)) is responsible for reading the data from an sstable.
|
||||
It is created by a _simple builder_ ([`SSTableReader.Builder`](format/SSTableReader.java)) or a _loading
|
||||
builder_ ([`SSTableReaderLoadingBuilder`](format/SSTableReaderLoadingBuilder.java)). The builders are supplied by
|
||||
a _reader factory_ ([`SSTableFormat.SSTableReaderFactory`](format/SSTableFormat.java)).
|
||||
|
||||
The constructor of a particular `SSTableReader` implementation should accept two parameters - one is the format-specific
|
||||
_simple builder_ and the other one is an sstable owner (usually a `ColumnFamilyStore` instance, but it can be null
|
||||
either). The constructor should be simple and not do anything but assign internal fields with values from the builder.
|
||||
|
||||
A simple builder does not perform any logic except basic validation - it only stores the provided values the reader
|
||||
constructor can access. A new reader implementation should include a public static simple builder inner class that
|
||||
extends the `SSTableReader.Builder` generic reader builder (or `SSTableReaderWithFilter.Builder`, see [below](#filter)).
|
||||
|
||||
In contrast to the simple builder, a loading builder can perform additional operations like more complex validation,
|
||||
opening resources, loading caches, indexes, filters, etc. It internally creates a simple builder and eventually
|
||||
instantiates a reader.
|
||||
|
||||
#### General notes
|
||||
|
||||
Note that if the builder carries some closeable resources to the reader, they should be returned by the `setupInstance`
|
||||
method.
|
||||
|
||||
You will find some `cloneXXX` methods to implement - remember to create a reader clone in a lambda passed to
|
||||
the `runWithLock()` method.
|
||||
|
||||
#### Unbuilding
|
||||
It is convenient to implement the `unbuildTo` method, which takes a _simple builder_ and initializes it so that
|
||||
the builder can produce the same reader. The method should also take the `sharedCopy` boolean argument denoting whether
|
||||
it should copy the fields referencing closeable resources to the builder directly or as (shared) copies. The convention
|
||||
also requires copying the resources only if they are unset in the builder (the field is null). The method should call
|
||||
the `super.unbuildTo` method as a first step so that all the fields managed by the parent class are copied and in
|
||||
the actual implementation only the fields specific to this format have to be assigned.
|
||||
|
||||
For example, the implementation of that method in a reader for the _big table_ format is as follows:
|
||||
|
||||
```java
|
||||
protected final Builder unbuildTo(Builder builder, boolean sharedCopy)
|
||||
{
|
||||
Builder b = super.unbuildTo(builder, sharedCopy);
|
||||
if (builder.getIndexFile() == null)
|
||||
b.setIndexFile(sharedCopy ? sharedCopyOrNull(ifile) : ifile);
|
||||
if (builder.getIndexSummary() == null)
|
||||
b.setIndexSummary(sharedCopy ? sharedCopyOrNull(indexSummary) : indexSummary);
|
||||
|
||||
b.setKeyCache(keyCache);
|
||||
|
||||
return b;
|
||||
}
|
||||
```
|
||||
|
||||
#### Filter
|
||||
|
||||
If the sstable includes a _filter_, the reader class should extend
|
||||
the [`SSTableReaderWithFilter`](format/SSTableReaderWithFilter.java) abstract reader (and its _simple builder_ should
|
||||
extend the `SSTableReaderWithFilter.SSTableReaderWithFilterBuilder` builder).
|
||||
|
||||
The reader with filter provides the `isPresentInFilter` method for extending implementation. It also implements other
|
||||
filter-specific methods the system relies on if the implemented reader extends that class.
|
||||
|
||||
Note that if the implemented reader extends the `SSTableReaderWithFilter` class, it should include the `FILTER`
|
||||
component in the appropriate component sets.
|
||||
|
||||
The reader with filter implementation comes with additional [metrics](filter/BloomFilterMetrics.java) - read more about custom
|
||||
metrics support [here](#metrics).
|
||||
|
||||
#### Index summary
|
||||
|
||||
Some sstable format implementations, such as _big table_ format, may use _index summaries_. If a reader uses _index
|
||||
summaries_ it should implement the [`IndexSummarySupport`](indexsummary/IndexSummarySupport.java) interface.
|
||||
|
||||
The support for _index summaries_ comes with additional [metrics](indexsummary/IndexSummaryMetrics.java) - read more
|
||||
about custom metrics support [here](#metrics).
|
||||
|
||||
#### Key cache
|
||||
|
||||
If an sstable format implementation uses row key cache, it should implement
|
||||
the[`KeyCacheSupport`](keycache/KeyCacheSupport.java) interface. In particular, it should store a `KeyCache` instance
|
||||
and return it with the `getKeyCache()` method. The interface has the default implementations of several methods
|
||||
the system relies on if the reader implements the `KeyCacheSupport` interface.
|
||||
|
||||
The interface comes with additional [metrics](keycache/KeyCacheMetrics.java) - read more about custom metrics support
|
||||
[here](#metrics).
|
||||
|
||||
#### Metrics
|
||||
|
||||
A custom sstable format implementation may provide additional metrics on a table, keyspace, and global level. Those
|
||||
metrics are accessible via JMX. The `SSTableFormat` implementation exposes the additional metrics by implementing the
|
||||
`SSTableFormat.getFormatSpecificMetricsProviders` method. The method should return a singleton object implementing the
|
||||
[`MetricsProviders`](MetricsProviders.java) interface. Currently, there is only support for custom gauges, but it can be
|
||||
extended when needed.
|
||||
|
||||
Each custom metric (gauge) is an implementation of the [GaugeProvider](GaugeProvider.java) abstract class. Although the
|
||||
class expects the implementation to provide a gauge for each level of aggregation, there is a helper class -
|
||||
[SimpleGaugeProvider](SimpleGaugeProvider.java) - which does that automatically with a supplied reduction lambda. There
|
||||
is [`AbstractMetricsProviders`](AbstractMetricsProviders.java) class which is a partial implementation of the
|
||||
`MetricsProviders` interface and leverages `SimpleGaugeProvider` in the offered methods.
|
||||
|
||||
Example - additional metrics for sstables supporting index summaries (see
|
||||
[`IndexSummaryMetrics`](indexsummary/IndexSummaryMetrics.java) for a full example):
|
||||
```java
|
||||
private final GaugeProvider<Long> indexSummaryOffHeapMemoryUsed = newGaugeProvider("IndexSummaryOffHeapMemoryUsed",
|
||||
0L,
|
||||
r -> r.getIndexSummary().getOffHeapSize(),
|
||||
Long::sum);
|
||||
```
|
||||
|
||||
### Writer
|
||||
|
||||
#### Construction
|
||||
|
||||
An sstable writer ([`SSTableWriter`](format/SSTableWriter.java)) is responsible for writing the data to sstable files.
|
||||
It is created by a _builder_ ([`SSTableWriter.Builder`](format/SSTableWriter.java)). The builder is supplied by
|
||||
a _writer factory_ ([`SSTableFormat.SSTableWriterFactory`](format/SSTableFormat.java)).
|
||||
|
||||
#### SortedTableWriter
|
||||
|
||||
There are not many methods to be implemented in the writer. The most notable one is `append` which should write
|
||||
the provided partition to disk. However, there is a generic default implementation
|
||||
[`SortedTableWriter`](format/SortedTableWriter.java) which handles things like writing to the data file using
|
||||
the default serializers, generic support for a partition index, notifications, metadata collection, and building
|
||||
a filter. The writer triggers fine-grained events when data are added and those methods can be overridden in
|
||||
the subclasses to apply specific behaviours (for example, `onPartitionStart`, `onRow`, `onStaticRow`, etc.).
|
||||
Eventually it calls an abstract `createRowIndexEntry` method which should be implemented in the subclass.
|
||||
|
||||
### Scrubber and verifier
|
||||
|
||||
A custom sstable format should also come with its own verifier and scrubber implementing [`IVerifier`](IVerifier.java)
|
||||
and [`IScrubber`](IScrubber.java) interfaces correspondingly. A generic partial implementation is provided in
|
||||
[`SortedTableVerifier`](format/SortedTableVerifier.java) and [`SortedTableScrubber`](format/SortedTableScrubber.java).
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue